ChatGPT Can't Audit Your Internal Links. Here's Why That Matters and When It Might Change.

A client came to us in the middle of a content audit. They had identified 139 pages on their site that were candidates for deletion or rework — aged content, thin posts, outdated service pages, old campaign landing pages that had outlived their purpose. Before pulling the trigger on any of it, they wanted to do the responsible thing: check whether any other pages on the site were internally linking to those 139 URLs. Because if a page you're about to delete or redirect is receiving internal links from twenty other pages, you need to know that before you act. You need to update or redirect those links, or at minimum factor them into your decision about what to do with the content.

Reasonable, careful, exactly the right instinct.

Before reaching out to us, they'd tried to handle it themselves. They went to ChatGPT, gave it the list of 139 URLs, explained what they needed, and asked it to go through the site and identify which internal pages were pointing to any of those URLs.

ChatGPT couldn't do it.

It gave them something — it usually does — but what it gave them was not a reliable answer. It either acknowledged it couldn't access the live site, produced a generic response about how internal linking works, or in some cases confidently returned information that was partially fabricated. None of it was usable for an actual audit decision about 139 pages of real content.

This is not a ChatGPT failure in the sense of a bug or a product shortcoming that's going to be patched in the next update. It reflects a fundamental architectural reality about what large language models can and cannot do right now. Understanding that reality matters if you're making content decisions based on what you think AI can help you with.

What the Client Was Actually Asking ChatGPT to Do

To understand why this request failed, it helps to be precise about what answering it correctly would actually require.

Finding every internal page that links to a specific set of 139 URLs is not a research question. It is not a knowledge question. It is not something that can be answered by a model with a strong grasp of SEO principles and a broad training dataset. It is a data retrieval problem that requires real-time access to the live state of a specific website.

To answer that question correctly, a tool needs to do the following: make an HTTP request to every crawlable URL on the domain, receive and parse the HTML response from each one, extract every anchor tag and its href attribute from each response, resolve relative URLs to absolute ones, and then cross-reference the discovered link destinations against the client's list of 139 target URLs to find matches. That process has to run across the entire reachable URL space of the site — not a sample, not a guess, not a recollection from training data — because a link to one of those 139 pages could be sitting on any page on the domain, including pages that were published last week.

That is a crawl. It requires persistent outbound HTTP connectivity to a live server, the ability to parse and interpret actual HTML from actual HTTP responses, stateful URL queue management across potentially hundreds or thousands of requests, and the ability to hold and query a relational dataset of source URLs mapped to destination URLs as the crawl progresses.

ChatGPT has none of that infrastructure during a standard conversation. It is a token prediction model. It does not maintain an active HTTP client. It does not have a mechanism to initiate, queue, and recursively execute a crawl session against a live domain. It does not have access to the current state of your website. When it appears to answer questions about a specific site's content, it is either drawing on something from its training data — which has a knowledge cutoff, may be incomplete, and almost certainly does not reflect the current state of a site that's been actively edited — or it is generating plausible-sounding output that has no grounding in the actual live content.

For a question like "which of my pages link to this URL," a hallucinated or training-data-derived answer is worse than no answer. You cannot make deletion decisions based on it.

What About ChatGPT's Browsing Tool?

This is where it gets worth unpacking carefully, because the existence of ChatGPT's web browsing feature creates a reasonable-sounding expectation that doesn't hold up for this use case.

The browsing tool can fetch a URL and read the content of that page. It can tell you what's on a given page at a point in time. That is genuinely useful for a narrow set of tasks — checking what a specific page says, reading a piece of content, retrieving structured data from a single URL.

But there is a significant gap between "can fetch a single URL on request" and "can systematically crawl an entire domain, parse every page's link graph, and cross-reference 139 destination URLs against that complete dataset."

The browsing tool does not run recursive crawls. It does not maintain a queue of discovered URLs and systematically process them. It does not build a persistent relational map of internal links across an entire domain. When you ask it to check which pages link to a set of URLs, it cannot do that comprehensively — it can at best fetch a small number of pages you specifically direct it to, which for a site of any meaningful size is nowhere near sufficient to answer the question reliably.

A site with 500 pages, 1,000 pages, 2,000 pages — you cannot manually direct a browsing tool through every one of those and have it maintain a coherent picture of what it found. The context window fills. The session ends. The state is lost. And at no point in that process do you have a reliable, complete answer to "which pages link to these 139 URLs."

What AI Can Actually Help With in a Content Audit Right Now

To be fair about where the current capability boundary sits: AI tools are genuinely useful for certain parts of a content audit workflow. It's worth being specific about where that usefulness begins and ends.

If you have a small, bounded set of pages — five to ten service pages, a handful of industry landing pages, a core navigation structure that maps to a limited number of URLs — you can manually feed that content to an AI tool and get useful analysis back. You can paste in the HTML or text of a page and ask it to identify the internal links present. You can give it a small crawl export and ask it to find patterns. For limited, manually-scoped inputs where you are providing the data rather than asking the model to retrieve it, AI tools can do meaningful analytical work.

What they cannot do is the retrieval step. They cannot go get the data. They cannot crawl the live site, discover the full URL space, parse the link graph, and return a complete picture of internal linking across a domain. That boundary — between analysis of data you provide versus retrieval of data from a live system — is the line that matters for this kind of audit work.

For 139 pages on a real site of any meaningful size, you are squarely in retrieval territory. You need a tool that can actually crawl.

What You Should Actually Use Instead

For the specific task the client was trying to do — identify every internal page linking to any of 139 target URLs — the correct toolset is a dedicated crawler. Screaming Frog is the most common choice for this kind of targeted audit work and handles it directly. Run a full crawl of the domain, let it complete, and then use the Inlinks report filtered against your list of 139 target URLs to see exactly which source pages are linking to each one, what the anchor text is, and whether the destination is returning a 200, a redirect, or an error.

Sitebulb handles the same workflow with a more visual output. Ahrefs Site Audit and Semrush's site audit tool both produce internal link data as part of their standard crawl output, though for this specific use case — cross-referencing against a custom list of URLs — Screaming Frog's filtering capabilities give you more direct control.

The crawl for a site of a few hundred to a few thousand pages takes minutes. The output is a complete, accurate, filterable dataset of every internal link on the domain. That is the tool for this job. It has been the tool for this job for over a decade. The fact that AI exists does not change what the job requires.

When Will AI Actually Be Able to Do This?

This is the honest answer: not soon, and the timeline is longer than most people assume.

The barriers are not primarily about model intelligence. The models are capable of sophisticated analysis and would handle the cross-referencing task well if given accurate input data. The barriers are architectural and infrastructural.

Real-time web access at crawl scale requires an AI system that can initiate and manage persistent HTTP sessions, maintain crawl state across potentially thousands of requests, handle rate limiting, respect robots.txt, process diverse HTML structures reliably, and do all of this within a session that doesn't time out, lose context, or hit token limits before the crawl completes. Current AI assistant architectures are not built for this. The browsing tools that exist are designed for single-page retrieval on demand, not systematic domain-wide crawling.

Stateful crawl management is a different engineering problem from conversational AI. A crawler needs to maintain a visited URL set, a queue of discovered URLs, a growing dataset of source-to-destination link mappings, and enough context to know when the crawl is complete. Doing that within a language model's context window is either impractical at scale or requires infrastructure that sits outside the model entirely — at which point you are essentially building a traditional crawler with an AI interface bolted on, which is a very different product from what currently exists.

Reliability and verifiability are the hardest long-term problem. For a content audit that's going to result in 139 pages being deleted, redirected, or reworked, you need to be able to trust the output completely. A missed link — a page that links to one of those 139 URLs that the tool didn't catch — means a broken internal link after the content change goes live. The tolerance for error in this kind of audit is very low, and AI systems that generate plausible outputs rather than retrieved data cannot currently meet that bar for high-stakes SEO decisions.

Realistically, for AI to reliably handle live site crawls and internal link audits as a native capability — not as a thin interface over a traditional crawler, but as something a model can do autonomously with verifiable accuracy — you are looking at a five to ten year window at a minimum. That accounts for continued development in agentic AI systems, persistent memory and state management, real-time web integration at scale, and the kind of reliability benchmarking that would make it appropriate to trust the output for decisions with real consequences.

In the meantime, the tooling that exists for this job is mature, fast, and reliable. It just isn't AI.

The Practical Takeaway

If you are in the middle of a content audit and you need to know which internal pages are linking to a set of URLs before you make any deletion or redirect decisions — do not ask an AI tool to retrieve that data from your live site. It cannot do it reliably, and for a decision that affects the link structure of your entire domain, an unreliable answer is more dangerous than no answer.

What you need is a crawl. What you also need, depending on the size of the audit and the complexity of your site's internal linking, is someone who knows how to run it, interpret the output, and help you make the right call on each of those 139 pages — not just which ones have inbound internal links, but what to do about those links, how to handle the redirects, and how to restructure the content that's worth keeping rather than just deleting.

That's the kind of audit work we do. If you're sitting on a list of pages you're not sure what to do with, get in touch and let's talk through it.

Frequently Asked Questions

If ChatGPT can't crawl a live site, why does it sometimes return what looks like internal link data when you ask it about a specific website?

Because it's drawing on its training data, not the live site. Large language models are trained on web crawls that include indexed content from across the internet. If your site has been around long enough to have been included in that training data, the model may have some representation of your content — but that snapshot is frozen at the training cutoff date, incomplete by definition since no training crawl captures everything, and almost certainly out of sync with the current state of the site. When ChatGPT returns something that looks like internal link data for a real domain, it is pattern-matching against what it absorbed during training, not retrieving anything from the live server. For a content audit where the decisions hinge on the current state of your internal link structure, training-derived output is not just unreliable — it's actively misleading, because it may reflect a version of the site that no longer exists.

What's the minimum site size where this becomes a real problem versus something you could manually check?

Roughly fifty to seventy-five pages is where manual checking becomes impractical for an internal link audit of this kind. Below that threshold, you could theoretically open each page, use browser find-in-page to search for the target URLs, and track your findings in a spreadsheet — tedious but doable. Above it, the combinatorial problem grows fast. If you have 300 pages on your site and 50 URLs in your audit list, you are looking at potentially checking 300 source pages for links to any of 50 destinations. That's not a manual job. The client in this post had 139 pages in their audit list on a site large enough that manual checking was never a realistic option, which is exactly why they went looking for an automated solution and ended up at ChatGPT.

Does Screaming Frog or a similar crawler tell you specifically which pages link to a target URL, or do you have to figure that out yourself from the raw crawl data?

Screaming Frog handles this directly. Once a crawl is complete, you can use the Bulk Export → All Inlinks function or filter the Inlinks tab on any individual URL to see every source page linking to that destination, along with the anchor text, link type, and HTTP status of both the source and destination. For an audit against a specific list of URLs, the most efficient workflow is to use the crawl data as a database — either filter within Screaming Frog's interface or export the full inlinks dataset and run a VLOOKUP or filtering operation in Excel or Google Sheets against your list of 139 target URLs. The output is a clean, filterable table of every internal link pointing to every page in your audit list, which is exactly what you need before making any deletion or redirect decisions.

If a page in the audit has no internal links pointing to it, does that mean it's safe to delete without any redirect?

Not necessarily, and this is an important distinction. No internal links pointing to a page means you don't have an internal link cleanup job to do after the deletion — but it doesn't mean the page has no inbound signals worth preserving. Before deleting any page with no internal links, you should check three additional things: whether it has external backlinks pointing to it from other domains (check GSC's Links report or your backlink tool of choice), whether it appears in any XML sitemap files that would leave a dead URL in your sitemap after deletion, and whether it has any direct traffic history in GA that suggests users have it bookmarked or linked from sources outside your control. A page with zero internal links but meaningful external backlinks or direct traffic warrants a redirect to the most relevant live destination, not a hard delete. No internal links just means your internal cleanup is minimal — it doesn't clear the full checklist.

When agentic AI systems do eventually get reliable live crawl capability, will that replace tools like Screaming Frog for this kind of work?

For some use cases, probably yes — eventually. The more likely near-term outcome is that AI agents will wrap around existing crawl infrastructure rather than replace it. The engineering required to build a reliable, scalable web crawler from scratch is substantial and largely solved by tools that have been refined over many years. What AI adds is the analytical and decision-making layer on top of that data: interpreting the crawl output, prioritizing which orphaned pages represent the biggest risk, recommending redirect targets based on semantic similarity, flagging which pages in an audit list are worth reworking versus which are genuinely redundant. That combination — purpose-built crawler for data retrieval, AI for analysis and recommendation — is where the near-term trajectory points. Full replacement of dedicated crawl tooling by a native AI capability is a longer horizon, and for high-stakes audit work where the output drives real decisions, the reliability bar is high enough that adoption will lag capability even after the technical pieces are in place.