AI model gateway
Model Proxy for model traffic you can route, cap, and explain.
Point Claude Code, Codex, OpenAI SDK, Anthropic SDK, or an AI application at one gateway. Model Proxy gives the request a virtual key, resolves the route, checks budget and provider health, records metrics-only observability, and sends the call to the right upstream model.
Setup workbench
Waiting for provider keyWhat Model Proxy is
A control layer between AI clients and model providers.
Model Proxy is an AI model gateway for teams that use more than one model provider or more than one AI client. A request arrives with a virtual key and a requested model name. The gateway checks the key scope, resolves the alias to an upstream provider model, applies budget and rate rules, avoids disabled providers, and records request metadata for monitoring. The client only needs a gateway URL and a virtual key. Teams can rotate upstream provider credentials, change fallbacks, test cheaper routes, or stop a failing provider without editing every application.
The first version focuses on a short path to readiness: generate client configuration, create provider profiles without exposing raw keys, simulate routes before production, test provider capability only when a key is supplied, and keep request logging conservative by default.
Current infrastructure signals
Gateway buyers now expect routing, budgets, and observability together.
OpenAI-compatible requests still center on a chat-completions style endpoint, while Claude Code documents an LLM gateway setup that can point ANTHROPIC_BASE_URL at a gateway. Cloudflare AI Gateway documents spend limits that can return 429 when a configured budget is exceeded, and its dynamic routing docs describe versioned routes with conditions, quotas, model nodes, and fallbacks. LiteLLM documents proxy spend tracking and rate limits for multi-provider deployments.
- Client compatibility: keep a stable base URL and move routing decisions into the gateway.
- Budget timing: enforce cost policy in the request path instead of discovering spend only in a bill.
- Provider health: treat fallback, kill switch, and rollback as production controls, not incident notes.
- Search provenance: when a search bridge runs, show source URLs and cost separately from the model call.
Request path
Model Proxy makes each request explainable before it leaves the gateway.
A production request should not be a blind pass-through. Model Proxy treats every call as a decision that can be inspected: which client sent it, which virtual key was used, which model alias the client named, which route version matched, whether the selected provider is enabled, whether the upstream model has known pricing, and whether the workspace budget still allows the call. If any of those checks is unresolved, the safer answer is to stop with a clear status rather than sending traffic to the wrong provider.
This matters most when a team has several AI entry points. Claude Code may need an Anthropic-compatible base URL, an internal app may use the OpenAI SDK, a background job may need embeddings, and a support tool may call web search before the model request. Model Proxy keeps those clients from growing separate credential files, separate budget logic, and separate incident playbooks. A route can move from one upstream model to another while the client keeps the same requested model name.
- 1. Identify the caller: client type, session, project label, requested endpoint, and virtual key scope.
- 2. Resolve the alias: match the requested model to an enabled route version and upstream model.
- 3. Check policy: apply provider health, known price, RPM, TPM, monthly budget, and retention rules before dispatch.
- 4. Record the outcome: monitor the decision, cost status, provider, latency, and failure boundary without storing bodies by default.
Console scope
The console starts where the request will fail.
Most gateway trouble is not the proxy listener. It is an alias that resolves to the wrong model, a provider key that cannot call the endpoint, a budget rule that silently treats an unknown price as free, or a log policy that saves more body data than the team intended. The Model Proxy console keeps those controls in the first working area.
The console is built around the sequence an operator actually follows. First, generate the client setup so the application points at the gateway. Next, add a provider profile with its base URL, upstream model, and known input/output prices. Then map the public model alias to that provider, run the route simulator, and inspect whether the selected provider, upstream model, fallback, and cost status match the intended rollout. Only after that should a virtual key scope be created for a project or endpoint.
| Control | Why it is first-version work |
|---|---|
| Setup Wizard | Generate client-specific configuration and copy commands from one place. |
| Route Simulator | Resolve alias, provider, upstream model, fallback, and budget outcome before a real request. |
| Capability Probe | Check chat, streaming, tool, embedding, and search-bridge readiness when a provider key is supplied. |
| Traffic Monitor | Show real console events and proxy checks produced during the current workspace session. |
| Retention Settings | Keep body retention off by default and make any broader logging choice explicit. |
Operator checklist
Use Model Proxy when routing policy needs to survive client changes.
Model Proxy is useful when the team expects route changes, provider incidents, cost pressure, or audit questions after launch. A single-provider hobby project may not need it. A product team with agents, coding tools, batch jobs, and support workflows usually does, because each client otherwise becomes a separate place to rotate keys, patch model names, and explain spend. The gateway gives the team one surface for the request policy while still letting each client keep the protocol it already speaks.
Before moving real production traffic, check the route table, provider profile, key scope, and retention policy together. The same alias that appears in client setup should appear in the route simulator. The provider saved in the profile should be the provider resolved by the simulator. A missing provider key should produce a clear readiness error, not a false healthy status. Retention should stay metrics-only unless the team has approved body logging. These small checks prevent expensive mistakes: a fallback that never fires, a disabled provider that still receives calls, or a budget rule that looks configured but never runs in the request path.
- Use it for: multi-client gateway setup, provider failover, project-level virtual keys, route testing, cost guardrails, and audit-friendly request metadata.
- Be careful with: raw provider keys, prompt/response body retention, and unknown model prices. Those should stay explicit operator decisions.
- Do not use it for: bypassing provider terms, hiding unsupported models, or treating missing billing configuration as a healthy route.
- Good rollout signal: the monitor shows setup, provider, route, simulator, key, search, and retention events tied to the same workspace journey.
Model Proxy questions
Does Model Proxy store provider keys?
The public console does not store raw provider keys. Provider tests and capability probes require a key for that action and send it only to the selected upstream endpoint. Team deployments should store upstream keys in encrypted server-side secret storage.
Can it replace a normal HTTP proxy?
No. A normal proxy moves network traffic. Model Proxy understands model aliases, provider capability, token usage, cost policy, virtual keys, search bridge provenance, and retention rules.
What happens when model pricing is unknown?
The route simulator marks the cost as unresolved and does not treat the request as free. A production gateway should block, require review, or route through a model with a known price table.
When should a team create a virtual key?
Create the virtual key after the route simulation resolves to the intended provider and upstream model. The key should name the project, allowed models, endpoint scope, RPM and TPM limits, monthly budget, and retention level. That sequence keeps the key tied to a real route instead of becoming another unmanaged credential.
How does Model Proxy handle search provenance?
The search bridge is separate from the model request. A gateway should record search provider, query, result count, source URLs, cache status, and search cost before those results become context for a model. If the search provider key is missing, the bridge should stop with a clear missing-key state.