Docs SEO

robots.txt

Checks that your site has a robots.txt file, where you tell crawlers what to skip and where your sitemap is.

SEOWeight 1In reports as robots

What it checks

We request /robots.txt on the address your homepage lands on. It passes when the answer is a success and is not an HTML page. The rules inside are not judged here; the AI crawler rules are checked in AI crawlers allowed in robots.txt, and the Sitemap line is used by XML sitemap.

Why it matters

Crawlers ask for robots.txt before anything else. Without it they crawl everything, which is fine for many sites, so this is only a warning. It is still the standard place to list your sitemap and keep crawlers out of areas such as internal search results.

How to fix it

Publish a plain text file at the root that allows crawling by default and has a Sitemap line with the full address of your sitemap. Do not use it to hide private pages: anyone can read it, and it is not access control.

Platform notes

WordPress

WordPress serves a basic robots.txt by itself, and SEO plugins let you edit it.

Shopify

Shopify serves one by default. To change it, add a robots.txt.liquid template to the theme.

Webflow and Wix

Both have a robots.txt editor in the SEO settings.

Squarespace

Squarespace creates the file for you.

Next.js, Vercel and Netlify

Add a robots.ts file in the app folder, or a static robots.txt in the public folder.

Cloudflare

Cloudflare can manage robots.txt for you and add rules to it. If you see rules you did not write, check that setting.

Your report shows what this check found on your site. Pro shows the exact fix for your site. See pricing.

See what this check finds on your site

Free, about 30 seconds, with every other check included.