---
name: ai-bot-rules
description: Declare explicit User-agent rules for known AI crawlers
---

# AI bot rules in robots.txt

Add explicit `User-agent` blocks in `/robots.txt` for the major AI crawlers so your access stance is unambiguous.

## Requirements

- Add a `User-agent` block per AI crawler you want to address
- Use `Allow: /` or `Disallow: /` to declare your stance

## Known AI bots to consider

- `GPTBot` (OpenAI training)
- `ClaudeBot`, `Claude-Web`, `anthropic-ai` (Anthropic)
- `Google-Extended` (Gemini training)
- `PerplexityBot` (Perplexity)
- `CCBot` (Common Crawl)
- `Applebot-Extended` (Apple training)

## Example

```
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: PerplexityBot
Allow: /
```

## References

- [OpenAI bots overview](https://platform.openai.com/docs/bots/overview-of-openai-crawlers)
- [Anthropic ClaudeBot](https://docs.anthropic.com/en/docs/agents-and-tools/claude-bot)