About the Ferrik crawler

What it is

Ferrik is a desktop SEO crawler. Every crawl runs on the computer of the person who started it, when they start it, against a site they are auditing. It is not a search engine and it does not train AI models.

How it identifies itself

It identifies itself with the user-agent Ferrik (+https://ferrik.io/bot) on every request it makes to the site being audited, including the pages it renders in a browser. The name and the address do not change between versions.

Requests come from that person's own computer, not from any server of ours, so there is no fixed address range to allow or block.

What it fetches, and how fast

It requests the HTML pages of the site it was pointed at, the site's robots.txt and sitemaps, and the files those pages reference on the same site: images, scripts, stylesheets, fonts, media and embedded frames, up to 1,000 of them per crawl. In its default mode it also loads each page in a hidden browser window, so the site sees whatever that page itself loads when it runs: fonts, media, embedded frames, and the requests its own scripts make.

A crawl stops at 500 pages on the free tier and has no page limit on a paid one. In its default mode it fetches up to 3 pages at a time, or up to 10 when it is told not to render pages. When a server answers 429 or 503 it drops to one request at a time and waits at least 2 seconds between requests, doubling the wait on each further refusal up to 5 seconds; it honours a Retry-After header up to 30 seconds; and it stops after five consecutive bot-check pages.

robots.txt, and how to block it

It reads robots.txt before crawling and follows the rules in a group addressed to it by name, User-agent: Ferrik, or otherwise the rules for all crawlers, User-agent: *. Case does not matter.

To block Ferrik from the whole site:

User-agent: Ferrik
Disallow: /

Naming Ferrik replaces your general rules for Ferrik — it doesn't add to them. If you have Disallow rules under User-agent: * that you want Ferrik to keep following, repeat them inside the Ferrik group. This is how robots.txt works for every crawler, not something specific to us.

Write the one word Ferrik. A group named with the full user-agent string, or with FerrikBot or Ferrik-Crawler, is not applied.

If robots.txt cannot be read at all, the crawl proceeds without rules. A 404 counts as a file with no rules.

Contact

To ask for a slower pace, an exemption, or to report a problem, email support@ferrik.io with the URL and the time you saw the request.