LLM Gateway
The shared OpenAI-compatible API that Coinbase's internal AI applications and coding agents use to reach models from multiple providers across multiple clouds. Because so much of the company shares it, one provider outage or one ambiguous error reaches everyone at once, so the design goal is graceful degradation and failures that explain themselves.
Technical lead and on-call owner since the team took over the service in May 2026. Led a reliability review at handover, wrote the reliability and routing design, and led failover, stalled-stream handling, release gating, model onboarding, access control, caching, and cost attribution.
- Clients call a standard OpenAI-style API with one credential and name a model or routing alias; the gateway chooses the provider and deployment, retries, and fails over.
- Restricted models use group-based access control.
- Every failure carries a stable error class in the body and a header, so a caller can tell a spend limit from a provider throttle or an open circuit breaker. Usage is attributed to the calling team, user, and repository.
- Owned policy on an open-source core. The team owns the policy layer on top of an open-source proxy: authentication, quotas, routing, privacy controls, and audit.
- Failover without retry storms. Traffic for each model spreads across providers and regions, unhealthy deployments are taken out of rotation, and failover respects access rules.
- Cache affinity. Session affinity keeps repeat calls on one deployment, so provider prompt caching stays effective.
- Stalled streams. A watchdog tracks time to first token and idle time between chunks. It shipped observe-only first, then with per-model abort windows, instead of a lower global timeout that would kill long agent runs.
- Release safety. The pre-production gate discovers models at runtime, requires one canary per provider family to pass, and separates provider gaps from gateway defects; production rolls out in canary waves against a baseline.