← Agent Data Hub · Search endpoints · /llms.txt

Robots.txt AI Crawler Permission Analyzer (GPTBot, ClaudeBot, PerplexityBot)

Evaluates robots.txt rules for AI user-agents (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot).

Call it (no API key)

GET https://api.agentdatahub.de/api/v1/call/web-robots-txt-ai-analyzer?compact=1

Full metadata: without compact=1 · Markdown: format=llm · Only some fields: add &fields=a,b.c · MCP: tool call_api with {"apiId":"web-robots-txt-ai-analyzer"}

Parameters

Example response (live, compact)

{
  "apiId": "web-robots-txt-ai-analyzer",
  "data_freshness": "live",
  "data": {
    "evaluated_bots": [
      "GPTBot",
      "OAI-SearchBot",
      "ClaudeBot",
      "PerplexityBot",
      "Google-Extended"
    ],
    "default_policy": "ALLOW"
  },
  "source": "100% Platform AI Crawler Policy Engine",
  "license": "100% Platform Deterministic Compute / Open Reference"
}

Source & license