Redirect mapping with AI: ChatGPT prompts, embeddings and scripts compared

They are not rivals so much as layers. Most of a typical redirect map is boring — same slug under a new folder, same title with a new brand suffix — and anything clever spent on those rows is wasted. The interesting part is the minority of pages that were renamed, merged or rewritten, and that is where the methods really differ.

The common first attempt: export the old urls, export the new ones, paste both into a chat and ask for a csv of pairs. For a small site with readable slugs the result looks convincing, and much of it is right. The problems are in the part that looks just as convincing:

invented targets — a language model writes the most plausible continuation, not a lookup result. Asked where /team/jobs-2021 should go, it can answer /careers/open-positions because that is what such a page is usually called — whether or not the new site has it. Kalai et al., why language models hallucinate, argue that training and evaluation reward guessing over acknowledging uncertainty, which is exactly the behaviour a redirect map cannot afford. Imported, an invented target is a redirect to a 404.

long lists — the lists have to fit into the model’s context, and fitting is not the same as being used well. Liu et al., lost in the middle, found performance highest when the relevant information sits at the beginning or end of a long input and markedly worse in the middle. In practice: rows quietly skipped, output cut off partway through, a pair from the middle of the list matched worse than one from the top.

repeatability — the same prompt can return a different map tomorrow. That makes a change hard to review: you cannot tell which rows changed because the input changed and which because the model did.

what it sees — two url lists carry no content. A model mapping /p/417 to anything is guessing from nothing, and so would a person.

None of this rules a chat model out. It rules out importing its output unchecked. For twenty urls you will check by eye anyway, a chat is a fine way to get a first draft.

Embeddings turn the text of each page into a vector, and pages with similar meaning end up close together; the redirect target for an old page is its nearest neighbour among the new pages. Because the comparison is on content rather than strings, it finds pairs no slug comparison could: /leistungen/webdesign and /services/websites with the same text, translated or rewritten.

Screaming Frog’s vector embeddings tutorial is the most complete public walkthrough: connect OpenAI, Gemini or Ollama, store the html of both crawls, generate embeddings from the main content (navigation and footer excluded), and read the closest semantically similar address for every old url. It requires a paid licence and an api key, and it is candid about its limits — it calls itself not a purpose-built redirect mapping feature, says the results should not be trusted blindly, and recommends reviewing every result, lowest similarity first. Daniel Emery’s Automated Redirect Matchmaker does a lighter version in a free Colab notebook: two crawl exports, embeddings of the columns you pick (titles, meta descriptions, headings) with sentence-transformers, and the nearest new url with a similarity score — no api key, and its author asks you to check everything.

Embeddings do not invent urls — the nearest neighbour is always a real page from the new crawl — but they always return some neighbour. An old page with no counterpart still gets a closest match, just with a lower score, so a threshold and a person are part of the method. And the page text goes to the embedding provider unless you run a local model such as Ollama, which matters for a client’s unreleased site.

The middle ground: string similarity on urls (and sometimes titles), in a notebook. Libraries such as RapidFuzz score every old url against every new one; the MLforSEO fuzzy matching tutorial compares several of them for 404 and redirect mapping. Free, transparent, repeatable — and only as good as the strings you give it. On urls alone it pairs /blog/seo-tips with /blog/seo-tools quite happily. More options in redirect mapping tools compared.

This is how Silentfrog matches, and it uses no language model. Both sites are crawled, so every page brings its path, title and h1. Pages with the same normalised path are exact matches. Every other old page is scored against every new one: slug similarity (Jaro-Winkler and path-token overlap) weighs 0.5, title similarity 0.3 with brand suffixes like “| Acme” stripped, h1 similarity 0.2, and the weight of a missing signal moves to the slug. A score of 0.85 or more is accepted, 0.60 to 0.85 goes to review, below that the page is unmatched.

What that buys: every target is a page the new crawl actually found, so an invented url is impossible; the same crawl gives the same map every time; and nothing about either site is sent to an ai provider. What it cannot do: pair a page whose slug, title and h1 all changed beyond recognition. Those land in review or unmatched, sorted to the top of the table, where a person — or a carefully constrained model — picks the target.

If you want a language model in the process, put it where it helps and fence it in:

• match deterministically first; exact and high-confidence pairs never go near the model;
• send only the review and unmatched rows, a few at a time, never the whole site in one prompt;
• give each old url a short list of candidate new urls with ids — the top scorers, or the pages in the same section — plus the titles of both sides;
• ask for a candidate id or the word none, never a free-text url;
• validate in code: an id that is not in the list is rejected, not corrected;
• review the model's picks like any other uncertain row before export.

Old page: /team/jobs-2021 — "Jobs at Acme 2021"
Candidates:
  c1 /careers — "Careers"
  c2 /about/team — "Our team"
  c3 /blog/hiring-update — "How we hire"
Answer with one candidate id, or "none" if no candidate
is the same page or its clear successor. No other text.

# reject anything the model did not pick from the list
import csv

new_urls = {row["path"] for row in csv.DictReader(open("new-site.csv"))}
with open("ai-pairs.csv") as f, open("checked.csv", "w", newline="") as out:
    writer = csv.writer(out)
    for row in csv.DictReader(f):
        ok = row["target"] in new_urls
        writer.writerow([row["source"], row["target"], "ok" if ok else "REJECT: not on new site"])

The validation step is the part that matters most, and it needs no ai at all: a set of the new site’s real paths and a membership test. Run it on any map that a model touched, however it was produced.

More and more mapping tools advertise ai. The label says little; these questions say whether its output can go into an import file:

• can a target be a url the new site does not have? If the tool generates targets rather than choosing them from a crawl or a list, every row needs checking.
• what happens when nothing fits? A method that always returns a closest match needs a threshold and an unmatched bucket, or the weakest guesses look exactly like the best pairs.
• does the same input give the same map? If not, a re-run after a content change produces a diff full of noise.
• can you see why a pair was made? A score, or the signals behind it, is what lets a person review a hundred rows in minutes instead of reopening every page.
• where does the content go? Page text sent to a model provider is a question for the client’s contract before launch, not after.

Whichever method builds the map, two things do not change: the uncertain pairs need a person, and the live redirects need a crawl after go-live (the relaunch checklist). Silentfrog does the deterministic layer — both crawls, the matching, the review queue and the file your platform imports — free for the first 50 pages of the old site.

• Pasting two url lists into a chat model works for small, obvious maps and fails silently on large ones.
• The typical failures: invented target urls, rows lost in long lists, different answers on each run.
• Embeddings are the strongest AI method for pages renamed beyond recognition; they still need a threshold and review.
• Deterministic slug, title and h1 matching cannot invent urls and gives the same map every time.
• Use a model only on the leftovers, choosing from a closed candidate list, and validate every target in code.