How to Build an AI Sitemap for Agentic Crawlers: A Technical Guide to Signaling Content Structure Beyond Google

Your XML sitemap tells Googlebot where to go. Your robots.txt tells bots what to avoid. Neither was built for PerplexityBot, OAI-SearchBot, Claude-SearchBot, or the agentic fetchers that now determine whether your content gets cited in AI-generated answers. This technical guide covers the full stack: robots.txt bot-tier separation, llms.txt implementation with honest impact assessment, AI-specific XML sitemaps, Markdown endpoint companions, schema markup for AI parsability, HTTP header differentiation, and server log monitoring — with working code for Next.js, Express, WordPress, and Nginx.