01/work/routing-run
routing.run
A stable OpenAI-compatible endpoint over a model market that changes every week.
- role
- Co-founder & CTO
- period
- Feb — Jun 2026
- project
- visit routing.run
I cofounded Routing and built its backend and platform from zero: provider adapters, route policy, health-aware fallback, authentication, usage metering, billing, and the production path through acquisition and handover.
the problem
Model providers fail differently. One times out, another returns an incompatible error body, and a third begins streaming before failing halfway through the response.
If every product owns that complexity, changing a model or fallback chain becomes an application deployment. The gateway needed to make provider churn an operational decision instead.
constraints
- Keep the client surface OpenAI-compatible so an existing integration can move by changing its base URL and key.
- Preserve streaming while normalizing provider errors and retry behavior.
- Meter usage without persisting prompts or completions.
- Let operators change route aliases and fallback chains without shipping client code.
architecture
- clientOpenAI-compatible request
- edgeAPI key, limits, request identity
- policyroute alias, model choice, fallback chain
- healthlatency and provider failure signals
- providernormalized streaming adapter
- ledgerusage and billing metadata only
stack · Python · FastAPI · PostgreSQL · Redis · Docker
decisions
- A route is configuration, not code
Applications call a stable alias. Provider selection and fallback order live in the control plane, so an operator can move traffic without touching every caller.
- Health is more than a ping
A provider can return 200 and still be unusably slow. Routing decisions need recent latency and request failures as well as simple availability checks.
- Privacy changes the data model
The proxy holds request bytes only long enough to forward them. Accounting is built from bounded metadata rather than a convenient archive of user prompts.
- Retries belong beside billing
A retry or fallback must not become a second customer charge. Provider attempts and billable requests are separate events with explicit ownership.
results
Current product behavior and privacy language are documented on routing.run. Ownership and the acquisition handover reflect my role from February through June 2026.
failures & lessons
- The unhappy path is the product
The simple proxy was never the hard part. Streaming failures, mismatched error formats, partial responses, and retry boundaries determined whether the gateway was trustworthy.
- Fallback needs a stop condition
Blindly trying every provider increases latency and cost. Policy has to know which errors are retryable, how much time remains, and when returning the original failure is safer.
- Operations need an audit trail
Moving traffic without a deploy is useful only when the team can explain which policy changed, who changed it, and how the request path reacted.