North SEO logo
Founded 2025 - 8+ years experience

Robots.txt Tester

Fetches a site's robots.txt, lists the rules and sitemaps, and tests whether a given URL is allowed for Googlebot.

Why it matters

One wrong line in robots.txt can hide a whole site from Google. It happens often after a site launch when the staging rules come along.

What you get

Fetches the file, lists every rule by user-agent, confirms a sitemap is declared, flags a site-wide block, and tests whether a path you enter is allowed for Googlebot.

Part of the SEO Toolkit

47 tools, one subscription, no per-tool pricing. Monthly plan.

  • ✓ Content, on-page and technical checks
  • ✓ Keyword, SERP and rank tools on live Google data
  • ✓ Image tools and AI writing assistants
  • ✓ 1,500 runs a month, cancel any time
Create your account

Already have one? Open the tool

More Technical tools

  • Page Speed CheckerGoogle PageSpeed Insights scores for mobile and desktop, Core Web Vitals, and the top opportunities.
  • HTTP Status & Redirect CheckerStatus code, the full redirect chain hop by hop, response headers and server timing for any URL.
  • SSL Certificate CheckerCertificate issuer, validity dates, days until expiry, covered domains and whether the chain validates.
  • Robots.txt GeneratorBuild a correct robots.txt: pick what to block, add your sitemap, and copy the file.
  • XML Sitemap GeneratorCrawls a site (up to 150 pages) following internal links and produces a sitemap.xml you can upload.

All 47 tools →

Frequently asked questions