Routing AI so most queries never reach a model

The expensive part is deciding when to call a model

Natural-language search makes an easy demo. Type “cozy Italian place for date night” and a large model will return something plausible. The hard part comes later, when that model sits in the path of every query a platform serves: every request pays for inference, waits for it, and depends on a provider being up.

We delivered a multi-tenant restaurant discovery microservice that had to handle both kinds of query: the short ones people type most of the time, like “sushi” or “pizza”, and the descriptive ones, like “romantic spot with live music near me”. The design question was not which model to use. It was when a model should run at all.

The answer was a router that classifies every query before any paid inference happens and sends it down one of three lanes. Only one of them calls a model.

Classify first, pay second

The classifier is deterministic. It is ordinary code, not a model, so choosing a lane costs nothing and gives the same answer for the same input every time. That matters for debugging: when a query lands in the wrong lane, we can reproduce it and write a test for it.

The lanes split the traffic unevenly on purpose. The architecture sizes the keyword lane for ~80% of traffictarget and the AI lane for ~20% of traffictarget. Both shares are design targets, not measured splits, and they are labelled that way everywhere we publish them.

The lane a query took is also part of its cache key, so classification decides what counts as the same query, not just where it goes.

async function search(tenant: TenantId, q: SearchQuery): Promise<Result<Venues, SearchError>> {
  const lane = classify(q);                       // rules, no model call
  const key = hash(tenant, q.text, q.geo, q.filters, lane);

  const cached = await resultCache.get(key);      // the cache lane
  if (cached) return ok(cached);

  const venues = lane === "keyword"
    ? await keywordSearch(tenant, q)              // ILIKE + PostGIS
    : await aiSearch(tenant, q);                  // Haiku -> JSON intent -> pgvector

  await resultCache.set(key, venues);             // short TTL
  return ok(venues);
}

Three lanes, three budgets

Each lane has its own latency budget and its own cost profile. The router exists to keep a query in the cheapest lane that can answer it.

The one latency figure we label shipped belongs to the fast path: the majority of requests stay on a <100ms pathshipped. The per-lane latency budgets behind it are targets.

• Keyword lane: simple intents, such as a cuisine or a dish. PostgreSQL ILIKE with PostGIS geo filters and no model call, so each query has a $0 marginal model costshipped.
• AI lane: descriptive intent. Claude Haiku turns the query into structured JSON, then OpenAI embeddings and pgvector cosine similarity rank venues inside Postgres. It is the only lane that pays for inference, and its budget is ~$0.001 per AI querytarget on average.
• Cache lane: repeat demand. Results live in Redis under a hashed key built from the tenant, the query, the location, the filters and the classification, with a short TTL, so a popular query is computed once and then served from memory.

The model is a parser, not a search engine

In the AI lane the model never sees the venue data and never ranks anything. Claude Haiku has one job: read the query and return structured intent as JSON, such as a cuisine, a vibe or a place. From there, retrieval is ordinary database work: filters in SQL, geography in PostGIS, and semantic similarity through pgvector, all inside Postgres.

That split has three consequences. The paid call stays small, because the model reads a sentence and writes a small object. The output has a known vocabulary, because each tenant’s configuration lists its valid vibes and cuisines. And ranking stays in our code: the hybrid ranker’s weights are per-tenant configuration, not a prompt.

A second Redis cache holds parsed intents, so a query the service has just interpreted does not go back to the model. Venue embeddings are built when venues change, by an embedding saga triggered from the indexing side of the service, never while someone is searching.

The smallest model that does the job

Intent parsing does not need a frontier model, so it does not get one. The split is lopsided: ~90% of AI calls are routed to Claude Haikushipped, and Claude Sonnet answers menu questions only, which is a different job from parsing a search.

Model calls are rate-limited per tenant inside the service, so one tenant’s burst of descriptive queries cannot starve every other tenant of model capacity.

Both model providers sit behind adapters in the service’s shared kernel, next to the Prisma and Redis services. The search code depends on an interface, not on a vendor’s SDK, which keeps a model change contained.

Tenancy runs through every lane

The service is multi-tenant, and every shortcut above has to respect that. The tenant is part of the cache key, so two brands never share a cached result. Tenant data is isolated in Postgres with row-level security, and the tenant travels through each request in an AsyncLocalStorage context rather than as a parameter someone can forget to pass.

Each tenant also brings its own configuration: its valid vibes and cuisines, its ranker weights and its embedding templates, held in an in-memory registry warmed from Redis. Adding a brand is a configuration change, not a code change.

What the figures say, and what they don’t

Every figure in this article carries its label. Three are shipped: the fast path, the share of AI calls routed to Haiku, and the keyword lane’s zero model cost, which is true by construction. The rest are targets. The traffic split and the cost per AI query were set as budgets when the architecture was designed, and nothing we have recorded measures them yet. We would rather print that than round a budget up into a result.

The service ships with metrics, Pino logging and explicit result types for failures. That is what makes the lane split something an operator can watch, rather than something the design document promises.

When this pattern fits

Routing before inference pays off when three things hold: a large share of queries is simple enough to answer without a model, a cheap deterministic classifier can tell the two kinds apart, and a query in the wrong lane is recoverable. Search usually meets all three. A support assistant that has to understand every message usually does not.

If you are adding a model to an existing product, start by collecting what users actually type and sorting a sample of it by hand. The shape of that list tells you whether you need a router, and roughly where its line should sit.

The rule we took from this build: set the budget per query before you choose the model, and make the model’s job as narrow as the problem allows.