How to Read a robots.txt File
robots.txt controls what search engines can and cannot crawl. Here is how to read one without guessing.
What robots.txt does
The robots.txt file sits at the root of every website. It tells crawlers which paths they are allowed to access and which they should skip.
It is not access control — a determined crawler can ignore it. But legitimate search engines respect it per the robots exclusion protocol, which makes it one of the most powerful tools in technical SEO.
The syntax
User-agent
Specifies which crawler the rules apply to. means all crawlers. means only Google.
Allow and Disallow
means do not crawl anything under /admin. creates an exception. Rules are matched by prefix.
Sitemap
tells crawlers where to find the sitemap. This line is how many crawlers discover your sitemap in the first place.
Common mistakes
- Blocking CSS and JavaScript. Modern crawlers need these to render your page. Blocking them means Google sees a blank page.
- Blocking the entire site. blocks everything. This happens more often than you would think, usually from a staging environment left in place after launch.
- Forgetting the trailing slash. blocks , , and . If you meant only the directory, be specific.
How to check yours
Visit in a browser. Or use ColorGears Web Insights to analyze it alongside your sitemap, headers, and meta tags in one pass.
Related: XML Sitemaps: The Complete Guide covers the other half of crawler discovery — what your sitemap tells search engines to find.