---
title: "What is AI crawler? - LeapScope"
url: https://leap-scope.com/glossary/ai-crawler
description: "An AI crawler is a bot an AI company runs to fetch web pages, for training models, for indexing a search product, or to read a page a user asked about."
---

# AI crawler

An AI crawler is a bot an AI company runs to fetch web pages, for training models, for indexing a search product, or to read a page a user asked about.

The three purposes matter because they are controlled separately in robots.txt and have different consequences. Training crawlers feed future models; search crawlers decide whether a page can be cited today; user-triggered fetchers load a page on demand.

Most AI crawlers do not execute JavaScript. Content that appears only after scripts run is invisible to them, which is why AI readiness checks read the raw HTML.

## Related terms

- [robots.txt for AI crawlers](https://leap-scope.com/glossary/robots-txt-ai-crawlers)
- [GPTBot](https://leap-scope.com/glossary/gptbot)
- [AI readiness (of a page)](https://leap-scope.com/glossary/ai-readiness)

[All glossary terms](https://leap-scope.com/glossary)

## Measure ai crawler for your brand

LeapScope tracks the prompts your buyers ask on the AI engines every day and reports mentions, citations and share of voice against your competitors.