What Problems Can AI Crawlers Cause?
Content is being "harvested for free"
AI crawlers are bulk scraping your articles, product descriptions, and original data to train large language models. The content you worked hard to create is being turned into someone else's training data for free.
Loss of Referral Traffic
Although some AI summaries may provide links to the original content, the likelihood of users visiting those sites significantly decreases once they have a brief summary. Advertising revenue, subscription conversions, product views—all business models reliant on traffic are being eroded.
Server is being "overwhelmed"
High-frequency AI crawlers put a heavy load on the source site, causing slower page loads, significantly degrading the real user experience, and directly impacting business conversion.
AI Crawler Identification and Control
When AI crawler control is enabled, EdgeOne will identify and control requests from the following crawlers
 | |
|---|---|
| Amazon Kendra(Amazon) | |
| Anchor Browser(Anchor) | |
| ClaudeBot(Anthropic-AI) | |
| atlassian-bot(atlassian) | |
| AwarioSmartBot(Awario) | |
| bigsur.ai(Big Sur AI) | |
| Cotoyogi(Cotoyogi) | |
| Factset_spyderbot(Factset) | |
| GoogleOther(Google) | |
| Google-CloudVertexBot(Google) | |
| Google NotebookLM(Google) | |
| Google-Extended(Google) | |
| pangu(Huawei) | |
| Liner Bot(Liner Bot) | |
| Meta-ExternalAgent(Meta) | |
| Novellum AI Crawl(Novellum) | |
| GPTBot(OpenAI) | |
| ShapBot(Parallel) | |
| PerplexityBot(Perplexity) | |
| QualifiedBot(Qualified.com, Inc.) | |
| WARDBot(WEBSPARK) |
