Robots.txt in the AI Era: Choose Which Crawlers to Allow

Robots.txt still matters even with AI

Robots.txt is a simple text file that tells well-behaved crawlers which URLs they may fetch. Search engines, AI training services, and on-demand assistants all read it before they request a page. The file does not stop a crawler that ignores the request, but most major services respect it as a polite guideline. Keeping it up to date gives you control over what content is easily discoverable.

Beyond indexing, the file influences how language models may pull facts when a user asks a question. If a page is blocked, the model is less likely to cite it, even if the content is valuable. Conversely, allowing a page signals that you are comfortable sharing the information for both search results and AI-generated answers. Updating robots.txt is a low-cost habit that protects privacy and shapes visibility.

Three crawler categories you need to know

Search indexers are the bots that build the public catalog of web pages. They visit your site, follow links, read visible text, and store the results for keyword queries. Their goal is to match a user’s search phrase with the most relevant pages. They respect robots.txt, sitemap hints, and canonical tags, and they favor pages that load quickly and have clear structure.

AI training crawlers collect large volumes of text to teach large language models. They often crawl broadly, ignoring some SEO signals, but many still read robots.txt as a courtesy. The material they ingest may later appear in generated answers, sometimes without a visible source. Allowing a page to be crawled does not guarantee citation, but blocking it reduces the chance that the model ever sees the content.

Allowing search crawlers: practical rules

For search bots, aim to let them reach every page that you want to appear in keyword results. Keep the main navigation simple and use internal links that point to important content. Add a sitemap file and reference it in robots.txt so the bots can discover pages quickly. Use canonical tags on duplicate pages to point to the preferred version, and avoid blocking essential assets like CSS or JavaScript that help render the page.

Create a robots.txt that disallows only the folders you do not need indexed, such as admin panels or private uploads. Use a line like “Disallow: /admin/” and keep the rest open. Test the file with a fetch tool to see which URLs are blocked. Then monitor search console reports for crawl errors and adjust the rules if you notice important pages are missing.

Deciding on AI training crawlers

AI training crawlers are less predictable, but you can still guide them with robots.txt. If a page contains proprietary data, confidential plans, or early-stage product details, consider blocking it to keep the information out of future model outputs. If the page is public-facing, well-written, and you want it cited, leave it open. Remember that blocking does not hide the page from the public web, it only signals a request not to be used for training.

Add a specific rule for known AI training agents if you can identify their user-agent strings, for example by disallowing “/private/”. If the agents are unknown, a broader “Disallow: /secret/” can reduce accidental exposure. Keep a short note in the file explaining why each block exists; this helps future reviewers understand the intent. Review the rule set quarterly and adjust as your product roadmap changes.

On-demand agents: when to allow immediate fetch

On-demand agents are services that retrieve a page at the moment a user asks a question. They often respect robots.txt, but they also look for clear metadata that tells them the page is safe to quote. If a page contains up-to-date contact info, pricing tables, or FAQ answers, allowing immediate fetch helps the assistant provide accurate snippets. Blocking such pages can lead to vague or missing answers for your visitors.

To support on-demand agents, keep the URL accessible and avoid disallowing it in robots.txt. Add structured data that marks the content type, such as FAQPage or Product, so the agent can extract the answer quickly. Test a query with a popular assistant and note whether the page appears in the response; if not, consider removing any block that might be hiding it. Refresh the metadata whenever the page content changes.

Questions people ask

How can I find out which AI training crawlers visit my site?

Check your server logs for unusual user-agent strings that mention AI, language model, or training. Some services publish a list of their agents, so you can match them. If you see repeated requests from unknown agents, add a temporary block and monitor the impact. Over time you will build a small inventory of the crawlers that actually reach your site.

Is it safe to block a page from AI training crawlers but still let search bots see it?

Robots.txt rules apply per crawler, so you can write a disallow line that matches only the user-agent of known training bots while keeping the page open for standard search agents. Be aware that if you use a generic block, search bots will also be prevented from indexing the page. Test with both types of agents to confirm the intended behavior.

What structured data should I add to help on-demand agents quote my content?

Add schema markup that matches the content purpose. FAQPage works well for question-answer lists, Product for item details, Article for blog posts, and Event for date-specific information. Use JSON-LD format in the page head. The markup gives the assistant clear facts to extract, increasing the chance the answer will include a citation to your page.