Finding the urls nothing links to before a relaunch

Any page that no longer has a path of links leading to it from the homepage. They accumulate quietly, in a handful of recognisable shapes:

• old blog posts that fell off the last page of the archive, or whose category page was removed in a previous redesign.
• campaign landing pages that were only ever reached from an ad, an email or a printed qr code — never linked from the site at all.
• discontinued products pulled from the catalogue but left online, which is common in shops where the url is also the invoice reference.
• pdf files — datasheets, price lists, manuals. They rank, they get linked, and no crawler configuration finds them if the page that linked to them is gone.
• language variants that survive only as an hreflang alternate after the language switcher was rebuilt.

What they have in common is the reason they matter: a page nobody links to internally is often precisely the page other people link to externally. Age is what created both.

The sitemap is generated by the cms rather than by the navigation, so it keeps listing pages long after the last link to them was removed. It is the single most productive source, and it comes in three shapes you have to handle:

a plain sitemap — one file, a list of <loc> entries. Nothing to do but read it.

a sitemap index — a file that only points at other sitemap files, usually because the site went over the 50 000-url limit or splits by content type. Each of the referenced files has to be fetched in turn, and forgetting to follow the index is the classic way to end up with an empty list.

gzipped files — sitemap.xml.gz. Large sites serve them compressed; a tool that only reads .xml silently finds nothing.

One more thing sitemaps hide: hreflang alternates. An entry can carry <xhtml:link rel="alternate"> elements pointing at the same page in other languages, and those urls often appear nowhere else in the file. They need a redirect just as much as the url they hang off.

Before guessing at /sitemap.xml, read /robots.txt. It may carry several Sitemap: lines, and they are absolute urls — so they can point at a file with a name you would never have guessed, or at another host entirely, such as a shop system on a subdomain.

The disallow rules are worth a glance too, for a different reason: a directory blocked from crawling still exists, still answers, and still needs a target if it was ever public.

This is the only source that reports what is in the index and how much traffic each url gets, which makes it both a url source and a priority list. Export the pages report for the longest period available and you have the urls that would actually cost you something if they broke.

The limitation to plan around: the export in the interface caps at 1 000 rows. For a site with more indexed pages than that, the export is a sample of the best-performing urls rather than the full list — useful for prioritising, not sufficient as the url source on its own. The Search Console API returns up to 25 000 rows per request, which is the way to get past it.

Cross-check the export against the sitemap in both directions. Urls in Search Console that are not in the sitemap are usually orphans. Urls in the sitemap that Search Console never reports may simply not be indexed, which is a decision waiting to be made about whether they deserve a redirect at all.

The access log is the only source that records requests nobody planned for. Filter for responses with status 200 over the last twelve months, strip out assets and bots, and you have a list that includes things no crawler and no sitemap will ever produce: pdf downloads, feed urls, endpoints that an old app still calls, campaign urls with parameters.

Twelve months is the useful window because seasonal pages — a christmas landing page, an annual report — only appear once a year and are exactly the kind of url that gets forgotten in a spring relaunch.

It happens more often than it should: the domain gets switched over before anybody thinks about redirects, and there is nothing left to crawl. The urls still exist as records elsewhere.

• the Wayback Machine — archived copies of the site, including, very often, /sitemap.xml and /robots.txt as they were. Fetch the archived sitemap and you have the old url list back.
• Search Console — the old property keeps its data after the site goes down, so the page export still works.
• the server logs — the hosting was usually still running last month even if the site was not.
• the old cms — a database export or a backup that still holds the slug of every page.

Silentfrog has a mode for this: old site already offline? takes the old site’s pages from a sitemap file or a url list and crawls only the new site (when the old site is gone). Several sitemap files at once are fine, hreflang alternates inside them are read, and a sitemap index is rejected with a message rather than silently producing nothing — those referenced files cannot be fetched from a site that no longer answers, so you have to supply them yourself.

What it costs: a url list carries no titles and no headlines, so matching runs on the url alone. Expect more rows in needs review and read them rather than trusting them.

The sources overlap, which is the point — you want the union, deduplicated on a normalized form of the url so that www., a trailing slash, index.html and case differences do not turn one page into four rows.

In Silentfrog the sitemap sources are read alongside the crawl by default, following sitemap indexes and unpacking .xml.gz, and you can paste or upload anything else on top: one url per line, or a csv with the url in the first column, so a crawler export, a Search Console export or a log extract goes in unchanged. Bare paths like /blog/post work too (starting a run). If the old site sits on an intranet or behind an ip allowlist, the desktop app reaches what your own machine reaches.

The last check comes after the map exists. Import the Search Console export as traffic data and the urls that get traffic but were never found by any source are listed separately, so they can be added as rows instead of being forgotten (traffic data). If that list is long, a source is missing.

• An orphan url is a live, indexed page nothing on your site links to any more — and a crawler cannot reach it.
• Read robots.txt first: its Sitemap lines point at files, and hosts, you would not have guessed.
• Follow sitemap indexes, unpack .xml.gz, and pick up hreflang alternates inside sitemap entries.
• The Search Console page export caps at 1 000 rows in the interface; the api returns up to 25 000 per request.
• Server logs are the only source for pdfs, feeds and seasonal campaign urls — use a twelve-month window.
• If the site is already offline, the Wayback Machine usually still has its sitemap.xml.
• Merge the sources on a normalized url, then check which trafficked urls no source found. A long list means one is missing.