Docs SEO
robots.txt
Checks that your site has a robots.txt file, where you tell crawlers what to skip and where your sitemap is.
What it checks
We request /robots.txt on the address your homepage lands on. It passes when the answer is a success and is not an HTML page. The rules inside are not judged here; the AI crawler rules are checked in AI crawlers allowed in robots.txt, and the Sitemap line is used by XML sitemap.
- PassA robots.txt file loads.
- WarningIt is missing, returns an error, or the server answers with an HTML page instead.
Why it matters
Crawlers ask for robots.txt before anything else. Without it they crawl everything, which is fine for many sites, so this is only a warning. It is still the standard place to list your sitemap and keep crawlers out of areas such as internal search results.
How to fix it
Publish a plain text file at the root that allows crawling by default and has a Sitemap line with the full address of your sitemap. Do not use it to hide private pages: anyone can read it, and it is not access control.
Platform notes
WordPress serves a basic robots.txt by itself, and SEO plugins let you edit it.
Shopify serves one by default. To change it, add a robots.txt.liquid template to the theme.
Both have a robots.txt editor in the SEO settings.
Squarespace creates the file for you.
Add a robots.ts file in the app folder, or a static robots.txt in the public folder.
Cloudflare can manage robots.txt for you and add rules to it. If you see rules you did not write, check that setting.
Your report shows what this check found on your site. Pro shows the exact fix for your site. See pricing.
Free, about 30 seconds, with every other check included.