AI Gateway Evaluation: Routing, Failure and Cost Evidence

Create a route manifest for each application and data class. Record the preferred model, allowed alternatives, region constraints, permitted features and the person allowed to change the route. Fallback is a policy decision as well as an availability mechanism: the alternate provider may differ in data handling, supported output format, tool-call behavior or commercial terms. The cheaper or healthier endpoint is not automatically an acceptable replacement.

The current LiteLLM documentation describes retries followed by fallback between model groups and distinguishes ordinary failures, context-window failures and content-policy failures. Portkey’s documentation also describes configured fallback paths. These are descriptions of particular implementations. Our recommendation is to inspect each error category separately and decide whether your application should retry, choose an approved alternative or stop. Do not turn a content-policy refusal into permission to search for a less restrictive model.

Test configuration in the form actually deployed. LiteLLM currently documents that certain mock fallback flags no longer affect Proxy requests from version 1.85.0, although direct Router tests retain them. A passing library mock can therefore say little about the live proxy path. In an authorized non-production environment, produce the intended upstream error and inspect the normal caller request, route decision and resulting attempt record.

Separate provider credentials, application keys and human administration. A team-scoped key should not silently inherit access to every configured model or another tenant’s budget. Ask the operator to demonstrate creation, restriction, revocation and rotation using the intended role. Inspect the configuration change record as well as a successful request, then send a request that should be denied.

Budgets need an explicit operating definition. Find out whether a number is a reporting threshold, an alert, a rate limit or a hard spending boundary. Ask how concurrent requests, delayed usage reporting, cached responses and retries are handled. Use a small synthetic quota in the pilot; do not assume that a dashboard total explains enforcement timing or reconciles exactly with a provider invoice.

Test sensitive-data handling with markers rather than real secrets. Inspect the gateway log, tracing backend, evaluator requests and exports for those markers. A configuration described as disabling body storage may still emit metadata or forward content to another service. Record each destination, retention rule and operator. Keep redaction effectiveness and access isolation as observations requiring evidence, not automatic conclusions from a product feature label.

Interrupt telemetry delivery separately from the gateway request path. Can the application keep operating while the operator knows evidence is missing? If the gateway itself is unavailable, the team needs a defined response: queue, reject, degrade or use an explicitly approved alternate path. An emergency direct-provider bypass can also bypass policy and logging, so it requires its own authorization and test rather than an undocumented workaround.

Export the configuration and evidence needed to investigate an incident after the trial ends. Keep key secrets out of the export, but retain model aliases, policy versions, attempt IDs, evaluator references and ownership decisions. Review whether another operator can reproduce the investigation without access to a salesperson’s demo environment. Include deletion, credential rotation and support escalation in the exit exercise.

Finish with a decision record tied to the tested version and use case. State what was observed, what remains unknown, who accepts the unresolved issues and which change triggers a repeat review. New models, providers, routing rules, SDK versions and deployment modes can invalidate an earlier assumption. A useful pilot produces an operating agreement and inspectable evidence, not a permanent platform-wide endorsement.