Skip to content
For website owners

vennbot, the crawler of venn

vennbot reads public documentation and specialist pages, mainly from manufacturers of warehouse technology. The venn team adds these websites as knowledge sources for Evi, the AI agent in venn. Customers of venn cannot use vennbot to crawl websites.

User agent
vennbot/1.0 (+https://app.even-logistics.de/bot)
robots.txt token
vennbot
Requests

What vennbot fetches

  • Websites the venn team has added as knowledge sources, for example product documentation and technical articles
  • HTML pages and linked PDFs, only within the area the venn team sets for the website
  • Only what anyone can see without logging in: GET requests without cookies or credentials, no logins, no forms
  • By default, no paths such as /login, /search, /cart, /contact or /privacy

vennbot also reads single pages, RSS feeds and sitemaps that the venn team follows for a technology, at most 30 pages per source and visit. Your robots.txt applies there too. When a customer sets up single sign-on, venn uses the same user agent to load the SAML metadata from the address the customer provides, and loads it again at a sign-in once the stored copy is older than one day. This request is part of the customer's setup and does not read robots.txt.

Behaviour

How vennbot crawls a website

This applies to websites the venn team has added as a knowledge source. For single pages, feeds and sitemaps that the venn team follows for a technology, only your robots.txt counts so far: the pace, the pauses, noindex and noai from this section do not apply to them yet.

robots.txt

vennbot follows the group for User-agent: vennbot, otherwise the group for *. Allow and Disallow apply by the longest matching rule, including * and $. If your robots.txt cannot be reached or returns a server error, vennbot fetches nothing from your host.

Pace

vennbot fetches pages one after another, at most one per second from your host. If your robots.txt sets a Crawl-delay, vennbot waits that long. If it is longer than 30 seconds, vennbot does not crawl the website.

429 and 503

If your server answers with 429 or 503, vennbot pauses your host for as long as Retry-After says: at least one minute, at most one hour. Without Retry-After, the pause is five minutes.

Error series

After ten errors within one hour, such as timeouts, connection or server errors, vennbot pauses your host for one hour.

noindex and noai

noindex, none, noai or noimageai in the robots or vennbot meta tag or in the X-Robots-Tag header keep the page out of Evi's knowledge. With nofollow, vennbot does not follow the page's links, and it always skips links marked rel="nofollow".

Sitemaps

To find pages, vennbot reads the sitemaps listed in your robots.txt as well as /sitemap.xml and /sitemap_index.xml, also gzipped.

Conditional requests

If your server sends an ETag or Last-Modified header, vennbot makes a conditional request next time. If the page is unchanged, a 304 response is enough.

Schedule

By default, vennbot crawls a website once a week. The venn team can also set a website to daily, monthly or manual runs.

Opt-out

Block vennbot

Your robots.txt decides what vennbot may fetch.

The whole website

vennbot then reads only your robots.txt. The exception is SAML metadata that a customer of venn has us load for single sign-on.

User-agent: vennbot
Disallow: /

One area only

Once there is a group for vennbot, the rules under * no longer apply to it. Repeat there whatever should stay blocked.

User-agent: vennbot
Disallow: /internal/

Single pages

When vennbot crawls a website, in the head of the page or as a response header:

<meta name="vennbot" content="noindex">
X-Robots-Tag: vennbot: noindex

vennbot caches your robots.txt for at most 24 hours, and every request after that follows the new rules. When vennbot crawls a website, pages it has already fetched and that your rules now block leave Evi's knowledge on the next crawl. For single pages, feeds and sitemaps that the venn team follows for a technology, only robots.txt works so far, and content already taken stays until we remove it at your request.

Contact

Questions or removal

Do you have questions about vennbot, or would you like us to remove content it has already fetched? Write to us and tell us the domain concerned.