Vector Search Alone Is Not Retrieval

Embedding search is genuinely good at the thing it is for: finding text that means the same as the query while using different words. A question about “cancelling a subscription” matches a passage about “terminating a recurring plan,” which no keyword index would ever return.

It is correspondingly bad at the opposite case, and the opposite case is extremely common in technical products.

What vectors reliably miss

A user searches ERR_CONN_4021. There is exactly one document containing that string, and it is the answer. Semantic search returns five passages about connection errors in general, because that is what the query means, and the exact token carries almost no semantic weight — it may not even be in the model's vocabulary in a useful way.

The same failure covers most identifiers:

Function and class names — resolveTenantScope
Error codes, HTTP statuses, exception types
Product SKUs, order numbers, version strings
Configuration keys and environment variable names
People and proper nouns that were rare in training data

The trap is that these are the queries where the user is most certain the answer exists, because they have the exact string in front of them. Returning vaguely related material is worse than returning nothing — it reads as a system that did not even try.

No amount of chunking work or model upgrading fixes this. The information needed is lexical, and you threw the lexical representation away when you converted text into a single dense vector.

Running both, because they fail differently

The fix is unglamorous: keep a keyword index alongside the vector index and query both. In Postgres that is full-text search on the same table, so it costs one more column and one more index rather than another system:

ALTER TABLE chunks
  ADD COLUMN tsv tsvector
  GENERATED ALWAYS AS (to_tsvector('english', content)) STORED;

CREATE INDEX ON chunks USING gin (tsv);

A generated column means the index maintains itself as content changes — no trigger, no application code, no chance of the two disagreeing.

Now run both retrievals for every query and combine them. The reason this works is that the two methods fail on disjoint query types: keyword search is exact and literal, vector search is fuzzy and semantic, and the union covers both. It is not an ensemble for marginal accuracy — it is coverage of a gap.

One tuning note specific to identifiers: default text-search configurations stem words and drop punctuation, which can mangle ERR_CONN_4021 or useEffect. If exact identifiers matter, add a `simple` configuration or a trigram index alongside, so the literal string is findable as written.

Combining two ranked lists

The two searches return scores that are not comparable — cosine distance and a text-search rank live on different scales with different distributions. Normalising them into a weighted sum is possible and fragile, because the normalisation depends on the score distribution of each query.

The robust standard is reciprocal rank fusion, which throws the scores away and uses only positions:

// score = sum over lists of 1 / (k + rank), k ~ 60
function fuse(lists: string[][], k = 60) {
  const scores = new Map<string, number>();
  for (const list of lists) {
    list.forEach((id, i) => {
      scores.set(id, (scores.get(id) ?? 0) + 1 / (k + i + 1));
    });
  }
  return [...scores].sort((a, b) => b[1] - a[1]).map(([id]) => id);
}

Because it only reads ranks, it needs no calibration and is unbothered by one retriever producing wildly different score ranges than the other. A document appearing in both lists rises above one that ranked first in only one — which is exactly the behaviour you want, since agreement between two independent methods is real signal.

The constant dampens the advantage of top positions. Leave it near 60 unless you have measured a reason to move it.

Reranking, where the quality actually comes from

Fusion gives a decent candidate set. Reranking is what makes the top three genuinely good, and it is the highest-leverage component after chunking.

The reason it works is architectural. An embedding is computed for the chunk before any query exists, so it has to summarise the whole chunk into one vector with no idea what will be asked. A cross-encoder reads the query and the chunk together and scores that specific pair, which lets it judge relevance rather than general similarity.

That is far more accurate and far too slow to run over a corpus — which gives the standard two-stage shape: retrieve broadly and cheaply, then rerank a small candidate set expensively.

const candidates = fuse([vectorHits, keywordHits]).slice(0, 50);
const ranked = await rerank(query, candidates);   // cross-encoder
const context = ranked.slice(0, 5);               // what the model actually sees

Retrieving fifty and keeping five is a reasonable starting point. The gain comes from the reranker having enough material to choose from, so being generous at the retrieval stage and strict at the end beats being strict at both.

Reranking also gives you something retrieval scores cannot: a usable confidence signal. Cosine distances are not calibrated and hover in a narrow band whether or not anything relevant exists. Cross-encoder scores separate much more cleanly, which means you can set a threshold and answer “I do not have information about that” instead of feeding the model the five least-bad chunks and letting it improvise.

The shape it converges on

Filter first — tenant, permissions, document status. Never rank content the user may not see, and never rely on filtering after the fact.
Retrieve twice — vector and keyword, generously, maybe fifty each.
Fuse by reciprocal rank.
Rerank the merged candidates with a cross-encoder.
Threshold, and return nothing rather than noise.
Expand the survivors with surrounding context before handing them to the model.

Latency is the obvious objection. Steps two and three are milliseconds; the reranker is the cost, typically tens to low hundreds of milliseconds for fifty candidates. Against a generation step measured in seconds, that is affordable — and it is the difference between an assistant that cites the right paragraph and one that cites something adjacent.

Measuring, so you know which stage to fix

Evaluate the stages separately or you cannot tell what improved. Build a set of real queries with the chunk that should be returned for each, then track two numbers: how often the correct chunk appears in the retrieved candidates at all, and how often it survives into the final few.

The first number is a retrieval problem — chunking, filters, the keyword index. The second is a ranking problem — fusion and the reranker. Fixing the wrong one is the most common way to spend a week without moving anything, and the aggregate “did the answer look good” metric cannot distinguish them.

The short version

Dense vectors cannot find the exact string a user is holding, and that is a structural gap rather than a model deficiency. Run keyword search beside it, merge with reciprocal rank fusion so no score calibration is needed, then rerank with a cross-encoder that sees query and chunk together. Use the reranker's score to decline to answer. Filter before ranking, never after. And measure retrieval and ranking as separate numbers.