seo

robots.txt Tester

Fetch robots.txt and show disallows for User-agent *.

How to use

  1. Enter a site origin (https://example.com).
  2. Fetch robots.txt.
  3. Review rules.

FAQ

Is crawling polite?

Single GET with short timeout and rate limits.