robots.txt Validator
Paste any robots.txt and instantly validate every rule, grouped by User-agent. It catches malformed Allow/Disallow paths, duplicate lines, bad Sitemap URLs and directives Google ignores.
Your robots.txt
Paste robots.txt contents
Comments (#…), directives (User-agent, Allow, Disallow, Sitemap, Crawl-delay) are all supported.
Validation Results
Rule groups
Sitemap directives
- inforobots.txt looks clean: ready to deploy.
How to Audit a robots.txt File
Four quick checks that catch the mistakes most often missed in production robots.txt files.
Paste the live file
Always audit the live https://yourdomain.com/robots.txt: not the staging version. CMS builds often rewrite rules at deploy time.
Check groups per User-agent
Each User-agent: block is a group. Googlebot, Bingbot and Googlebot-Image all read their own group only: generic * rules are a fallback.
Confirm every Sitemap URL
Sitemap: lines must be absolute URLs. Always add them: they're the fastest way for a crawler to discover your full URL list.
Remove Google-ignored directives
Crawl-delay, Host, Clean-param, and noindex in robots.txt are ignored by Googlebot. Either remove them or use the correct alternative (Search Console, meta robots).
robots.txt Directive Reference
What each directive actually does, and which crawlers respect it.
User-agent
Names the crawler a group applies to. Use * for all crawlers, or specific bot names (Googlebot, Bingbot, etc.).
Disallow
Blocks crawling of a URL path. Disallow: / blocks everything, Disallow: /admin/ blocks one folder.
Allow
Whitelists paths inside a Disallowed folder. More specific Allow rules override general Disallow.
Sitemap
Points crawlers to your sitemap.xml. Must be an absolute URL. Multiple Sitemap lines are allowed.
Crawl-delay
Seconds between requests. Honoured by Bing and Yandex: ignored by Google. Use Search Console for Google rate.
Comments (#)
Lines starting with # are comments. Document every non-obvious rule: future you will thank you.
Wildcards (*)
Match any sequence of characters in a path. Disallow: /*?filter= blocks all filtered URLs.
End anchor ($)
Matches end of URL. Disallow: /*.pdf$ blocks only PDF files, not pages that contain .pdf in the path.
Common robots.txt Mistakes
These slip into production regularly, and each one can tank indexing in days.
Indexing disasters
Noindex confusion
Crawl budget waste
Security leaks
robots.txt FAQ
Will Disallow remove a URL from Google?
No: Disallow only blocks crawling. The URL can still appear in Google results (without a snippet) if other sites link to it. Use meta noindex for true removal.
Should I block JavaScript and CSS?
Never. Google needs to render your pages to evaluate them. Blocking JS/CSS usually drops rankings: the page renders broken in Google's eyes.
Is my robots.txt sent anywhere?
No. Everything runs in your browser. Nothing is uploaded or stored.
Does my robots.txt need a BOM?
No. Save as plain UTF-8 text without a byte order mark. Many crawlers misread files that start with a BOM.
Want a Full Crawl & Indexability Audit?
Our Australian-led SEO team audits robots.txt, canonicals, noindex tags, redirects and sitemap coverage: then hands you a fix list in priority order.
- robots + canonical audit
- GSC coverage analysis
- No lock-in commitment
No long-term commitment. Cancel anytime. 100% satisfaction guaranteed.
Embed this Robots.txt Validator on your site
Free to embed: paste this where you want it to appear. Please keep the attribution link; that's all we ask in return.
<iframe src="https://workspacein.com/embed/robots-txt-validator" title="Robots.txt Validator by workspacein.com" width="100%" height="1200" loading="lazy" style="border:1px solid #e6e6e6;border-radius:12px;max-width:680px;"></iframe>
<p style="font:14px/1.5 sans-serif;color:#555">Powered by <a href="https://workspacein.com/tools/robots-txt-validator">Robots.txt Validator</a> from workspacein.com</p>