Model routing, caching and cost gateway
One door for every model call: routing, caching, budgets and fallbacks.
Use it when
As soon as more than one team or feature calls models, or when cost and latency need to be engineered rather than hoped for.
The parts, top to bottom
across every level
Hover or tap any part to see what it does. The light shows the order a request moves through.
What happens, in order
- 1Request with intent & budget
- 2Semantic / exact cache check
- 3Route: small model first, escalate if needed
- 4Prompt caching & batching
- 5Fallback provider on error
- 6Meter cost per team & feature
What we typically build it with
Trade-offs
A few milliseconds of added latency and one more component to run, in exchange for control over the two numbers everyone asks about: routing alone can cut cost by over 85% at 95% of frontier quality (RouteLLM), prompt caching makes repeated context 90% cheaper, and batching halves the price of anything non-urgent.
Further reading
Problems this architecture solves
Generic problem statements with the flow and the outcomes the industry has documented.
Support queues that never shrink
Tier-1 questions consume most of the team's day, answers vary by agent, and customers wait. Generic chatbots deflect badly because they do not know your policies or your systems.
How the system works
- Customer message
- Intent & policy check
- Retrieve from your docs & orders
- Answer or act via tools
- Escalate with context
Outcome: Klarna's assistant handled two thirds of customer chats in its first month and cut resolution time from 11 minutes to under two; Intercom reports Fin resolving 76% of conversations on average. Klarna press release
ArchitectureEvery question needs an analyst
Leaders wait days for a report, dashboards disagree with each other, and the data team is a queue. Raw data sits in the warehouse without becoming a decision.
How the system works
- Raw sources
- Lakehouse with partitions & quality checks
- Semantic layer of metrics
- Text-to-SQL agent
- Dashboards & alerts
Outcome: In dbt's own test, questions answered through a governed semantic layer reached 83% accuracy, against a 16.7% raw-SQL baseline in the data.world benchmark it replicated; vendors report time-to-insight falling by over 99% with 62% adoption within a year. dbt: the semantic layer as the data interface for LLMs
ArchitectureLLM spend that grows faster than usage
Every team calls model APIs directly, nobody can attribute cost to a feature, and a model upgrade or a traffic spike doubles the bill without warning.
How the system works
- One gateway endpoint
- Budgets & keys per team
- Route: small model first
- Prompt & semantic cache
- Batch the non-urgent
- Cost traced per feature
Outcome: Query routing alone cuts cost by more than 85% while retaining 95% of frontier quality; prompt caching bills repeated context at a tenth of the price and batch APIs halve everything non-urgent. RouteLLM (ICLR 2025)
ArchitectureOther platform patterns
Evaluation and observability loop
Golden sets, judges, traces and alerts, wrapped around every other architecture.
Use it when: Always. It is how we know whether a change made the system better, and how you know before your customers do.
Flow
- 1Collect real inputs
- 2Label expected outputs
- 3Run evals in CI on every change
- 4Trace production calls
- 5Sample & score online
Let's build intelligent systems that drive growth
Tachyon is the engineering partner for teams that need AI in production, not in a deck. Start with a free 60-minute discovery call.
Tell us the problem, we will map it to the architecture
Every great partnership begins with a conversation. Whether you are exploring possibilities or ready to scale, tell us what you are actually trying to build.
- ubheshubham.37@gmail.com
- +91 84592 96471
- Clients worldwide · English
- Pune, India · Headquarters