Skip to content
SJ
All writing

·11 min read

Ranking Is Not a Prompt

Asking a model to rank a list gives you an order with no explanation, no stability and no dial to turn. Use it for the judgements no formula can make, then do the ordering in code.

AI / LLM / Ranking / Architecture / Backend


A personalised feed works the same way everywhere. Several hundred candidates arrived since the last run, there is room for twelve, and something has to choose. The obvious move, in 2026, is to paste the candidates into a prompt and ask for the best twelve, with reasons.

It demos beautifully. Then you run it daily for a month and discover you have built a component with no dial on it: you cannot say why a story was picked, cannot reproduce yesterday's order, cannot make recency count for slightly more without rewriting English and hoping, and cannot tell whether this morning's feed is worse than last Tuesday's or just different.

The fix is to keep the model and be precise about which part of the problem is a judgement and which part is arithmetic.

What asking the model to rank costs

Five specific problems, in roughly the order they bite.

It does not survive the candidate set growing. Twenty items fit comfortably. Four hundred is a context-window problem, and long before you hit the limit, quality degrades with position: material in the middle of a long list gets less weight than material at either end. The position bias that makes LLM judges unreliable is the same effect, and it means your ordering partly encodes the order the candidates happened to arrive in.

It is not stable. Same inputs, same prompt, different day, different order. Some of that is real disagreement at the margin and some of it is sampling, and you cannot tell which. A feed that reshuffles for no reason is a product problem before it is an engineering one.

There is no dial. “Weight freshness a little more heavily” is not a change you can make. You can add a sentence and run the eval set, and the sentence may be worth 2% or 40%, in a direction you did not specify. Every tuning request becomes a prompt experiment with an unknown coefficient.

The explanation is post-hoc. Ask for reasons alongside the ranking and you get plausible sentences generated to accompany a decision already made. They read well and they are not a causal account of anything, which you discover the first time a reason cites a property the item does not have.

It costs per candidate, on every run, forever. Ranking is the one stage that by definition touches everything, so it is the worst possible place to put your most expensive operation.

What the model is for here

There are questions in this problem that no formula can answer. Is this an original report or a rewrite of someone else's? Is it substantive, or an announcement with a headline? Does it actually concern the thing this reader said they cared about, as opposed to sharing keywords with it? Those are judgements, they are what a language model is good at, and none of them are orderings.

So the model runs per item, not per list, and returns small structured values under a schema, borrowing the discipline from getting reliable JSON out of a model, applied to a narrow question:

const Assessment = z.object({
  substance: z.enum(["original", "derivative", "announcement"]),
  topicMatch: z.number().min(0).max(1),     // against this reader's topics
  claimSupport: z.enum(["cited", "asserted", "speculative"]),
  oneLine: z.string().max(180),
});

Three properties follow from scoring one item at a time, and they are the whole argument.

A judgement about one item does not change when another item arrives. That is what makes it cacheable, and a story assessed yesterday does not need assessing again today. A list ranking has to be redone in full every time anything changes.

It parallelises. Four hundred independent small calls are a concurrency problem, which is an ordinary one. One call with four hundred items in it is a context problem, which is not.

It is evaluable. “Is this an announcement?” has an answer a human can check on fifty examples. “Is this the right twelve?” does not, which is why ranking prompts are so often shipped with no test at all.

And because the expensive stage is now per item, you can afford to run it on fewer items: cheap deterministic filters first (deduplication, hard exclusions, anything already seen), and the model only on what survives.

The score is arithmetic, and it belongs to you

With judgements available as numbers, the ordering is a weighted sum. Six signals, three of them model judgements and three computed, plus two penalties the next section explains:

const WEIGHTS = {
  topicMatch:   0.30,   // model judgement, 0..1
  substance:    0.20,   // model judgement, mapped from the enum
  claimSupport: 0.10,   // model judgement, mapped from the enum
  recency:      0.20,   // computed: exponential decay, 36h half-life
  corroboration:0.10,   // computed: independent sources on this cluster
  sourceQuality:0.10,   // computed: per-source prior, hand-maintained
} as const;

export function score(item: Candidate, reader: Reader): Scored {
  const terms = {
    topicMatch:    WEIGHTS.topicMatch    * item.assessment.topicMatch,
    substance:     WEIGHTS.substance     * SUBSTANCE[item.assessment.substance],
    claimSupport:  WEIGHTS.claimSupport  * SUPPORT[item.assessment.claimSupport],
    recency:       WEIGHTS.recency       * decay(item.publishedAt, 36),
    corroboration: WEIGHTS.corroboration * saturate(item.sourceCount, 4),
    sourceQuality: WEIGHTS.sourceQuality * priorFor(item.source),
    novelty:      -NOVELTY_WEIGHT        * seenBefore(item, reader),
    avoided:      -AVOID_WEIGHT          * matchesAvoidList(item, reader),
  };

  // keep the terms, not just the total: everything below needs them
  return { ...item, terms, total: sum(Object.values(terms)) };
}

Nothing here is clever, and that is the point. The weights are eight numbers in version control: they can be reviewed, diffed, set to zero to answer “does this signal matter at all”, and changed by someone who is not the person who wrote the prompt. The model contributed the three things that needed reading comprehension and contributed nothing to the ordering.

This is the same division as keeping figures out of the model in a portfolio assistant, where every number a user sees is computed in TypeScript and handed back as a tool result. There the arithmetic was an XIRR. Here it is a rank. The principle does not change: use the model for the judgement, and compute the number.

Preferences are subtraction, not instruction

Two requirements defeat ranking prompts specifically, and both become trivial as terms.

“Stop showing me things I have already seen.” As prompt text this is a sentence competing with every other sentence, and it degrades as the history grows. As a term it is a penalty proportional to similarity against what this reader has been shown, bounded so it suppresses rather than eliminates. A new development in a story the reader followed last week should still be able to win.

“I do not care about this topic.” A declared avoid list as a penalty is enforced at every run with the same strength, and it can be audited: the reader can be told that this item scored lower because it matched something they asked to avoid. A prompt instruction gives you neither guarantee nor receipt.

Both are deliberately penalties rather than filters. A hard filter on “seen” makes follow-up coverage unreachable, and a hard filter on an avoid list means one over-broad preference silently removes a category forever. Subtraction degrades; exclusion is permanent.

The last decision is about the set, not the item

Take the top twelve by score and you will frequently get twelve angles on the same story, because if one item about an event scores well, its near-duplicates score well too. No per-item score can fix this, because the property you want, variety, is not a property of an item.

It is, however, one loop:

const picked: Scored[] = [];
const pool = [...scored].sort((a, b) => b.total - a.total);

while (picked.length < SLOTS && pool.length) {
  // penalise each remaining candidate by its similarity to what is
  // already in the feed, then take the best of what is left
  const next = pool.reduce((best, c) => {
    const adjusted = c.total - LAMBDA * maxSimilarity(c, picked);
    return adjusted > best.adjusted ? { c, adjusted } : best;
  }, { c: pool[0], adjusted: -Infinity });

  picked.push(next.c);
  pool.splice(pool.indexOf(next.c), 1);
}

One constant, LAMBDA, controls how much the feed prefers breadth over raw score, and you can move it and look at the result. Asking a prompt for “a good mix” gives you a mix whose breadth you cannot set, measure or explain.

A score the interface can show

Because the total is a sum of named terms, the two or three largest contributors are available for free, which pays off in two directions.

For the reader, “matched your interest in inference hardware, three independent sources, published four hours ago” is a true statement about the computation, not a sentence a model wrote about a decision it had already made. It is also correctable: a person who disagrees with the reason has told you something specific.

For you, a complaint becomes a query. Store the term vector alongside the run (the trace table is already there) and “why was this at the top” is answered by reading eight numbers. The equivalent investigation for a ranking prompt is rerunning it and hoping it does the same thing again.

Tuning without spending a model call

The property I did not anticipate, and now would not give up: because the signals are stored and the ordering is a pure function of them, you can replay. Pull last week's candidates with their assessments, change the weights, recompute every feed offline, and diff the output against what shipped. No model calls, seconds to run, and the answer to “what would doubling the recency weight have done” is a list of the fourteen items that would have changed places.

That is not available in any form when the ranking lives in a prompt. There, the same question costs a full rerun against the model, produces a different answer each time, and still does not isolate the change.

Two caveats. Replay is only as good as the stored signals: change the assessment schema and the history stops being comparable, so version it. And a weight change still needs the eval set: a feed that looks better in a diff of fourteen items can be worse across a hundred readers, which is the case for scoring aggregate quality rather than trusting an inspection.

Where a model should do the ranking

This is a trade, and it loses in real cases.

  • Small sets and comparative judgements. Choosing between five candidate answers, or picking which three of eight paragraphs belong in a summary, is a comparison a model makes well and a formula makes badly. The embedding-free retrieval system spends its model budget this way, on selecting among a short list of chunks.
  • Reranking a retrieval's top-k. A cross-encoder or model reranker over fifty results is standard and correct, but notice it scores one query-document pair at a time. That is a judgement per item, which is the argument of this post, not a counterexample to it.
  • Cold start. With no interaction data and no intuition about weights, a prompt is a reasonable v0 that gets you to a product. The mistake is leaving it there once you have signals worth weighting.
  • No evaluation harness. Eight weights you cannot measure is worse than one prompt you cannot measure, because it is eight things to guess at. Transparency helps only if you are looking.

And the weights themselves are a guess at the start. The difference is that a transparent guess can be argued with, moved a little, replayed against last week, and eventually fitted to real feedback. An opaque one can only be re-prompted.

The short version

Handing a model a list and asking for an order buys you a component with no stability, no explanation, no dial and a cost proportional to everything you own. Split the problem: let the model make the per-item judgements a formula cannot (is this substantive, is it on topic, is the claim supported) under a narrow schema, cached and parallel, and run it only on candidates that survived the cheap filters. Then compute the order yourself as a weighted sum of named terms, with preferences as bounded penalties rather than sentences, and a diversification pass for the property that belongs to the set rather than the item. Keep the terms alongside the total so the interface can explain a placement and you can debug one with a query. Replay weight changes against stored signals instead of re-prompting. And when the set is small and the call depends on comparing the items, hand it back to the model. That is a judgement, and judgement is what you were paying for.

Written by Saumya Jain

GenAI & Full Stack Engineer working on agentic RAG systems, NestJS microservices, and real-time systems. Currently open to remote work.

LinkedInGitHubResume