The RAG System With No Embeddings

I have written four posts that assume you have embeddings. That chunking decides what can ever be retrieved, that Postgres and pgvector are usually enough storage, that vectors alone miss the exact string a user is holding, and that a moving corpus makes the index a liability. I still believe all four.

The most recent thing I built has no embedding model in it at all. Retrieval is Postgres full-text search, and the language-model budget that would have gone into an embedding pipeline goes into rewriting the query and picking the chunks. It is a multi-tenant service, and it is faster and considerably less operational work than the version with vectors would have been.

An embedding pipeline is a standing obligation

The reason to consider this is not that vector search works badly. It is that “add embeddings” is not one decision. It is a system you now own, permanently:

• Every document change means re-embedding. Deletions are the dangerous case, and getting invalidation right is its own project.
• Every embedding-model upgrade means re-embedding the entire corpus. You cannot mix vector spaces, so there is no incremental migration — it is a full rebuild with a cutover.
• The index has parameters, and the wrong ones silently cost recall rather than failing loudly.
• Filtering interacts badly with approximate indexes, which is the trap that eats a week the first time.
• You have a hard dependency on an inference provider on the write path. If embeddings fail, ingestion stalls.

For a single-tenant product over a corpus that changes weekly, that bill is fine, and the pgvector post stands. For a multi-tenant service where every tenant is adding documents continuously, it is a background system that has to be healthy forever, and its unhealthiness is invisible — stale vectors do not throw, they just answer slightly wrong.

Postgres full-text as the entire retriever

The retrieval layer is the lexical half of the hybrid setup, kept and nothing else added:

ALTER TABLE chunks
  ADD COLUMN tsv tsvector
  GENERATED ALWAYS AS (to_tsvector('english', content)) STORED;

CREATE INDEX ON chunks USING gin (tsv);

-- ranked, tenant-scoped, one round trip
SELECT id, content, ts_rank_cd(tsv, query) AS rank
FROM chunks, websearch_to_tsquery('english', $1) query
WHERE tsv @@ query AND tenant_id = $2
ORDER BY rank DESC
LIMIT 50;

A generated column keeps itself current as content changes. There is no pipeline to run, no provider to be down, and no second store to keep consistent. A document is searchable the moment the row commits, in the same transaction.

And it has the well-known hole. Someone asks how to stop being billed; the document says “terminating a recurring plan.” No token overlap, no match. That gap is exactly why the earlier post said to run vectors alongside.

Spend the model call on the query, not the corpus

The alternative is to close that gap at query time with a cheap model call instead of at index time with an embedding pipeline. Before searching, expand the question into the vocabulary the corpus might actually use:

// "how do I stop being billed"
{
  terms: [
    "cancel subscription",
    "terminate recurring plan",
    "billing cancellation",
    "end membership",
    "refund policy"
  ],
  headings: ["Cancellation", "Billing"]
}

Run full-text search for each variant and fuse the ranked lists by reciprocal rank — the same fusion described in the hybrid retrieval post, doing the same job, over a different set of inputs. There it merged two retrievers. Here it merges one retriever run several ways.

This is the whole trick, and it is worth naming plainly: an embedding does the semantic work implicitly, at write time, in vector space, for every document you own. Query expansion does it explicitly, at read time, in words, for the one question actually being asked. You are paying for the same bridge, at a different moment, in a different currency.

Letting the model pick the chunks

Fusion leaves fifty candidates that need narrowing to about five. The standard answer is a cross-encoder reranker, and its advantage is that it reads the query and the chunk together rather than comparing two vectors computed in ignorance of each other.

A language model reading the candidates has that same advantage, and you are already paying for a model. Hand it the titles and opening lines of the fifty and ask which are worth reading in full:

const shortlist = await selectChunks({
  question,
  candidates: fused.slice(0, 50).map((c) => ({
    id: c.id,
    heading: c.heading,
    preview: c.content.slice(0, 200),
  })),
  keep: 5,
});

Previews rather than full chunks keep this affordable, and the judgement is mostly about topic rather than detail, so previews are usually enough. A hosted reranker will be cheaper per call at high volume; the point here is that it is one fewer model, one fewer vendor, and one fewer thing to deploy.

The honest bill, and why it is payable

This design puts two extra model calls on the read path — expansion and selection — where a vector system has none. On a cold query it is slower. Pretending otherwise would be dishonest.

What makes it work is that both calls are far more cacheable than generation. Expansion depends only on the question and the corpus version, not on the user, the tenant, or the conversation. Three levels, each invalidated by a version stamp that ingestion bumps:

• Expansion, keyed by the normalised question. Shared across every tenant, because the expansion of “how do I cancel” is not tenant-specific — only the search results are.
• Retrieval, keyed by question plus tenant plus corpus version.
• Answer, keyed the same way and invalidated the same way.

Version-stamping rather than deleting matters: ingestion bumps a counter, and every entry beneath the old version becomes unreachable without a scan for keys to evict. It also makes pre-warming natural — when ingestion finishes, replay the tenant's common questions so the first real one after a document upload is already warm. Cached retrieval lands in tens of milliseconds, which is well under what the vector path would have cost.

The tenant boundary underneath all of this is row-level security, and the caching rules are the ones from the post on the rest of the AI stack — a cache key that omits the tenant is a data leak, not a performance bug.

The part I did not expect

The operational simplification was the goal. The thing I would actually keep this design for is that retrieval failures became readable.

When a vector system returns the wrong chunk, the explanation is a cosine distance. There is nothing to inspect. You re-chunk, swap the model, tune the index, and hope. When this system returns the wrong chunk, the log contains the expansion terms it searched for, and the answer is usually sitting in plain text: it expanded “plan” into subscription vocabulary when the user meant a financial plan. That is a fixable, describable bug.

It also composes with exposing retrieval as a tool. A model that can read why its own search failed can write a better one, which a cosine distance never allowed it to do.

Where this falls apart

• Cross-lingual retrieval. A question in Hindi against English documents has no lexical overlap to expand into. Multilingual embeddings handle this natively. This is the clearest loss.
• Genuine vocabulary chasms. Patients describing symptoms against clinical notes, or lay users against legal text. Expansion helps and does not close it; embeddings trained on the domain do better.
• High query volume with low cache hit rates. If every question is unique, you pay two model calls every time and the economics invert.
• Typos and fuzzy matching. Trigram indexes patch this, but it is real work that vectors partly absorb for free.
• A small, stable corpus. If your documents rarely change, the standing obligation this whole post is about barely exists. Use pgvector; the earlier post is right.

The short version

Embeddings buy you a semantic bridge and charge you a permanent pipeline: re-embedding on every change, full rebuilds on every model upgrade, an index to tune, and an inference dependency on the write path. If your corpus moves constantly and your users mostly share vocabulary with your documents, you can buy the same bridge at read time instead — expand the query into corpus vocabulary with a cheap model call, run full-text search over the variants, fuse by reciprocal rank, and let the model shortlist the survivors. Cache the expansion and the retrieval behind a version stamp that ingestion bumps, and the common path is faster than the vector one. Go back to embeddings the moment you need to cross a language.