Cache Invalidation with Debezium: Let the Database Tell You

Caching a read-heavy endpoint is easy. Keeping that cache honest is where the work is, and the usual first attempt looks like this:

async updateProduct(id: string, dto: UpdateProductDto) {
  const product = await this.products.update(id, dto);
  await this.redis.del(`product:${id}`);      // invalidate
  return product;
}

This works exactly as long as one assumption holds: that this function is the only way a product row ever changes.

I ended up replacing this pattern with a change data capture pipeline on a Postgres and Redis backend, and what follows is the version I would set up again — including the parts that only show themselves in production.

Why invalidation rots

That assumption has a short life. The row also changes from:

a second service that writes to the same database
the nightly importer, which uses bulk upserts and skips the ORM
a cascade — updating a category rewrites the denormalised column on every product under it
the data-fix script someone ran against production at 2am to unblock a customer
a migration that backfills a column across the whole table

None of those call redis.del. The cache is now wrong, and it stays wrong until the TTL expires or somebody notices. The failure is also the worst possible shape for debugging: it does not reproduce locally, it does not throw, nothing appears in the logs, and by the time support escalates it, the cache has quietly healed itself.

The root cause is the same one behind most cache bugs. The invalidation trigger lives in one write path, while the truth lives in the database, which has many.

TTL is a dial, not a fix

The standard response is to lower the TTL until staleness stops generating tickets. That is a dial between two things you do not want: long TTL means users see wrong data, short TTL means you are hammering the database you built the cache to protect. At 30 seconds on a hot product page you have mostly built a rate limiter with extra steps.

Change data capture removes the dial. Instead of guessing how long the cached value stays true, you subscribe to the database telling you the moment it stops being true.

What Debezium actually does

Postgres already writes every committed change to the write-ahead log, because replication and crash recovery depend on it. Logical decoding exposes that log as a structured stream rather than raw pages. Debezium is a connector that reads that stream and publishes one message per row change.

Each message carries more than most people expect:

{
  "op": "u",                        // c=create, u=update, d=delete, r=snapshot
  "before": { "id": 42, "price": 899,  "stock": 3 },
  "after":  { "id": 42, "price": 1099, "stock": 3 },
  "source": {
    "table": "products",
    "lsn": 3894102384,              // position in the WAL: total order
    "txId": 55219,                  // transaction boundary
    "ts_ms": 1755248042113
  }
}

Three properties matter for invalidation. It is post-commit, so you can never invalidate on a change that rolled back. It is totally ordered by LSN. And it is complete — there is no code path that writes to the table without passing through the WAL, which is exactly the guarantee the application-level redis.del could not give you.

The usual runtime is Kafka Connect, but Debezium Server and the embedded engine both let you run a connector that sinks somewhere else entirely. If Kafka is not already in your stack, do not let this decision drag Kafka in with it.

The Postgres side

Three pieces on the database, and one of them is the setting people miss until it bites.

-- 1. postgresql.conf — requires a restart
wal_level = logical

-- 2. what to publish
CREATE PUBLICATION cache_cdc FOR TABLE products, categories, inventory;

-- 3. what the "before" image contains
ALTER TABLE products REPLICA IDENTITY FULL;

That third statement is the one to think about. REPLICA IDENTITY DEFAULT puts only the primary key in before, which is fine if every cache key is derived from the primary key. The moment a cache is keyed by something else — products:by-slug:blue-widget, products:by-category:8 — you need the old value to know which key to drop when that column changes. Without FULL, an update that moves a product from category 8 to category 9 tells you to invalidate category 9 and leaves category 8 stale forever.

FULL is not free: it writes the entire previous row into the WAL on every update, which costs write throughput and log volume on the primary. Set it per table, on the ones whose non-key columns actually participate in cache keys, not database-wide out of caution.

Delete, never set

The consumer is smaller than the setup that precedes it, and the temptation is to make it clever:

// Wrong, even though it looks like an optimisation
await redis.set(`product:${after.id}`, JSON.stringify(after));

Writing the new value into the cache from a CDC consumer is a trap.

The shapes do not match. What you cached is almost never one row — it is a row joined to its category, its media, and a computed availability flag. The CDC payload is a row. Reconstructing the cached shape inside the consumer means duplicating query logic that will drift from the real one within a month.
Sets are order-sensitive, deletes are not. Two updates redelivered out of order leave the older value in the cache permanently. Two deletes redelivered in any order leave the same correct result: the key is gone and the next read repopulates it. Since delivery is at-least-once, idempotent-under-reordering is the property you want.

So the consumer stays boring, which is the point:

async onChange(event: ChangeEvent) {
  const row = event.after ?? event.before;      // deletes only have "before"
  const keys = cacheKeysFor(event.table, row, event.before);
  if (keys.length) await this.redis.del(...keys);
}

Deletes are worth one note: Debezium emits the delete event and then a tombstone — a message with a null value — so Kafka log compaction can drop the key's history. Consumers that assume every message has a payload will throw on the tombstone. Skip nulls explicitly.

The actually hard part

The pipeline is a weekend. Knowing which cache keys a changed row participates in is the real design problem, and it is where this approach either stays maintainable or does not.

A hand-written cacheKeysFor that lists every key pattern per table works, and it rots the same way the original redis.del calls did — someone adds a cached query and forgets the mapping. Two approaches age better:

Tag sets. When a read populates a cache entry, record which entities it touched: SADD tag:product:42 → query:xyz. CDC on product 42 reads the tag set and deletes every key in it. Precise, handles joins and list queries, costs one extra set per cached read and needs the tag sets cleaned up as keys expire.

Generation counters. Keep gen:product:42, embed its value in every derived key, and have CDC do a single INCR. Old keys are instantly unreachable and die on their own TTL. One write per change regardless of how many derived entries exist and no bookkeeping; the trade is an extra read to fetch the generation before building a key, and stale entries occupying memory until they expire.

Generation counters are the better default for anything list- or join-shaped. Tag sets are worth it when cache memory is tight enough that dead entries matter.

The race you cannot remove

Here is the part that decides whether you can turn TTLs off. You cannot.

A read misses the cache and queries the database. It gets the old row.
A write commits. Debezium publishes. The consumer deletes the key.
The read from step 1, still in flight, writes what it fetched into the cache.

The cache now holds a stale value that no future event will invalidate, because the event already happened. CDC narrows this window compared to TTL-only invalidation — it does not close it, and no invalidation scheme triggered after a commit can.

What actually helps, in increasing order of effort:

Keep a TTL anyway. Not as the invalidation strategy, as the backstop that bounds how long any missed invalidation can survive. Minutes instead of seconds, because CDC is doing the real work.
Write the row version into the cache entry — an updated_at or the source LSN — and have the populating write refuse to overwrite a newer entry. This is a compare-and-set on the version, and it removes the common case.
Single-flight your cache fills so only one read per key is ever in flight. Fewer racing writers, smaller window, and it fixes the stampede on expiry as a side effect.

Operational edges

The pipeline is well-behaved right up until it is not, and the failure modes are specific enough to list.

A stopped connector fills the primary's disk. This is the one that causes real incidents. Postgres retains WAL segments for an inactive replication slot forever, on the assumption the consumer will return. A connector that has been down over a weekend can take the database with it. Alert on pg_replication_slots lag from day one, set max_slot_wal_keep_size so the database drops a hopeless slot instead of dying with it, and delete slots for connectors you have retired.
The first snapshot is not free. On startup the connector reads existing rows to establish a baseline. On a large table that is a long, heavy scan. Use incremental snapshots — driven by a signal table, chunked, resumable — rather than discovering the blocking version at the worst moment.
Schema changes reach the consumer. Dropping a column that appears in a cache key breaks invalidation silently — the mapping function stops producing that key and nothing errors. Treat cache key mappings as code that migrations have to be reviewed against.
Partition by primary key. Per-key ordering is only guaranteed if all changes to a row land in the same partition. Debezium keys messages by primary key by default; do not override that without knowing why.
One more moving part in every environment. Local development needs an answer that is not “run Kafka.” A flag that falls back to write-through invalidation outside production is usually the pragmatic call.

CDC is not a domain event

Once this is running, the pipeline looks like a free event bus, and the next idea is always to hang notifications or downstream workflows off it. That is the mistake this whole approach is one step away from.

A CDC message says status changed from pending to paid. It does not say the order was paid. You can reverse-engineer intent from the diff, and then a refund flow that briefly touches the same column fires a payment confirmation at a customer. The other half is coupling: your column names become the payload of a product feature, and a routine rename breaks something three teams away.

The split that works is boring. CDC for derived state that must follow the database — caches, search indexes, read models. Explicit domain events, written by the code that knows what happened, for anything a human will read. I wrote about that second half in why notifications need an integrated event source.

The two combine well, and this is the detail worth stealing: write your domain events into an outbox table inside the same transaction as the state change, then let Debezium tail that table. The outbox gives you deliberate, well-named events; CDC gives you delivery that cannot miss a commit. You end up needing exactly one pipeline instead of two.

When not to bother

If one service owns its database, every write goes through one ORM, and there are no importers, no cross-service writes and no data-fix scripts, then redis.del next to the update is correct and you should keep it. The connector, the slot monitoring and the snapshot planning buy nothing.

It starts paying when:

more than one service or job writes the same tables
bulk paths bypass the ORM
cached values are joins or aggregates, so “which key changed” is not obvious from the write
stale reads are a correctness problem — pricing, stock, permissions — rather than a cosmetic one
you are already running Debezium for a search index or a read model, and the marginal cost is one more consumer

The short version

Cache invalidation is unreliable when the trigger lives in one write path and the truth lives in a database with many. Change data capture moves the trigger to where the truth changes, which is the only place that sees every writer. Keep the consumer dumb, delete instead of set, keep a TTL as a backstop, watch your replication slot, and do not let the row diffs pretend to be domain events.