System One in production

One call, sixteen questions

· 6 min read

The shape of a typed decision model is easy to miss if you arrive from a chat API. You do not send a prompt and get a completion. You send one state and a dictionary of questions, and you get one answer per question, evaluated together. Once that lands, the unit of design stops being the question and starts being the decision point.

A turn is one call

Every message a shopper sends produces exactly one model request in our pipeline. That request carries the store configuration, the previous search state, the last eight messages and the new message, and it asks between eight and sixteen typed questions about all of it.

questions = {
  'intent':       choice(...),   # search, support, other, unclear
  'context':      choice(...),   # continue or new
  'focus':        choice(...),   # which text represents the request now
  'adult':        choice(...),   # recipient is an adult
  'budget':       choice(...),   # which number is the budget
  'currency':     choice(...),   # TRY, USD, EUR, none
  'alternatives': choice(...),   # exclude what we already showed
  'price_sort':   choice(...),   # cheaper than before
}
for i, term in enumerate(terms[:8]):
    questions['exclude_' + str(i)] = choice(...)   # does this exclusion still apply

answers = meter.ask(state, questions)   # one request

Eight of those are fixed. The rest are generated: one Choice per active exclusion, up to eight, each asking whether that term still applies in the current search. A conversation where nobody has ruled anything out asks eight questions. A conversation four corrections deep asks twelve.

The questions do not depend on each other's answers, which is what makes this work. They are all readings of the same state, taken at the same moment. If one of them needed the answer to another you would be back to sequential calls, and the design pressure is to find a formulation where they do not.

What you would have written instead

With a chat model you have two options and both are bad. Send eight separate requests and pay eight round trips, or write one prompt that asks for all eight answers as JSON and accept that the model now has eight chances to produce something you cannot parse, with no way to tell which part it got confused about.

The second option is what most people ship, and it degrades in a specific way: a prompt that asks for nine things answers the first three well. Attention is finite and the later fields get the leftovers. You end up reordering your JSON schema by importance, which is a sentence that should make anyone uncomfortable.

Typed questions do not have that gradient, because each one is scored against its own option set rather than generated as the tail of a sequence. That is the part worth wanting, more than the speed.

Then cap the number of calls

One call per turn is the design. Three calls per conversation is the limit the code enforces, and it exists because designs drift. The moment a second call becomes acceptable, a third is an easy argument, and an assistant that makes an unbounded number of model calls per message is a cost incident waiting for a chatty user.

The cap is not a performance tuning knob. It is a structural assertion: if you find yourself needing a fourth call, something about the decomposition is wrong and you should fix that rather than raise the number.

The call belongs next to the data, not at the edge

Our frontend runs on Cloudflare Workers. The model call does not, and the reason is worth stating because edge inference is the fashionable answer.

The state we send is assembled, not collected. It needs the previous search state, the recent conversation and the store configuration, all of which live in PostgreSQL. The retrieval that produces ranking candidates is a pgvector similarity query against an index in that same database, with a local embedding model in front of it. Running the typed decision at the edge would mean shipping all of that to the edge first, and then shipping the result back to where the catalog is.

So the Worker serves the site and proxies the API, and the decision sits next to the vector index. Latency is dominated by the retrieval and the model, not by the hop, and end to end a turn lands around 1.5 seconds. Put the inference where the state already is.

Where the batching stops helping

Two places, and both are worth knowing before you design around this.

The first is that a batch shares a fate. Our validation rejects the whole response if any single answer fails its criteria check, which is the right call for correctness and does mean one bad answer costs you the turn rather than one field. If you have a question that is allowed to fail independently, it does not belong in the batch.

The second is that state size is shared too. Every question in the call sees the same state, so a question that needs a large payload makes that payload part of the bill for all the others. That is why our ranking candidates are budgeted to a token ceiling and stripped of fields no decision depends on. The batch is cheap per question, not free per byte.

Navlu is a conversational product discovery assistant for ecommerce stores. One Jev call per turn, capped at three per conversation, running next to the pgvector index rather than at the edge.

← All posts

Built by Naylalabs