---
title: "What is robots.txt for AI crawlers? - LeapScope"
url: https://leap-scope.com/glossary/robots-txt-ai-crawlers
description: "AI crawler rules in robots.txt are the User-agent groups that allow or block the named bots AI vendors use for training, search indexing and user-triggered…"
---

# robots.txt for AI crawlers

AI crawler rules in robots.txt are the User-agent groups that allow or block the named bots AI vendors use for training, search indexing and user-triggered fetches.

Each vendor runs several crawlers with different jobs. OpenAI's GPTBot collects training data while OAI-SearchBot indexes pages for ChatGPT search; Anthropic has ClaudeBot and Claude-SearchBot; Google-Extended governs Gemini training but not Search. Blocking the training bot does not remove a site from answers; blocking the search bot does.

A site that wants to be cited allows at least the search and user-triggered crawlers. Blocks added hastily in 2023 often still cover both.

robots.txt is a request, not enforcement: a firewall or bot manager can block a crawler the file allows.

## Related terms

- [AI crawler](https://leap-scope.com/glossary/ai-crawler)
- [GPTBot](https://leap-scope.com/glossary/gptbot)
- [llms.txt](https://leap-scope.com/glossary/llms-txt)

[All glossary terms](https://leap-scope.com/glossary)

## Measure robots.txt for ai crawlers for your brand

LeapScope tracks the prompts your buyers ask on the AI engines every day and reports mentions, citations and share of voice against your competitors.