SpesPublicaBot

SpesPublicaBot is the crawler for Spes Publica's own search index. It fetches public web pages so they can be found in Spes Publica search. Its user agent is:

SpesPublicaBot/0.1 (+https://spespublica.com/bot)

What it does

• It follows your robots.txt, using the rules for SpesPublicaBot, or for * if none name it. If your robots.txt can't be reached, or answers with a server error, it doesn't crawl your site.
• It waits at least 2 seconds between requests to a site, and makes one request at a time. It follows a Crawl-delay of up to 30 seconds; if you ask for longer, it doesn't crawl your site.
• It follows noindex and nofollow in a page's robots meta tag or X-Robots-Tag header, and skips links marked rel="nofollow".
• It only fetches web pages, not images, video or downloads. It keeps a page's title, description and the start of its text, for search results that link to your page.
• It works from its own lists of sites and the links it finds. It never fetches a page because someone searched for something.

Blocking it

To keep SpesPublicaBot off your whole site, add this to your robots.txt:

User-agent: SpesPublicaBot
Disallow: /

It reads your robots.txt again at least once a day, so a change takes effect within a day.

Questions

If SpesPublicaBot is causing you trouble, email dan@spesp.com with your site's address, and we'll look into it.