Routing, aliases, translation, fallback and errors: what an Apigee AI gateway does so that clients can swap Gemini for Claude by changing the model field.
Part 2 of 6 in Apigee as an AI gateway: common problems teams hit when they put LLMs into production, and how to solve them with Apigee, using open-source templates as examples.
Here’s the promise an AI gateway makes to developers: write your code against one API, and switch models by changing one string.
curl "https://$GATEWAY_HOST/v1/chat/completions" \
-H "x-api-key: $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "google/gemini-flash-latest",
"stream": true,
"messages": [
{"role": "user", "content": "Summarize our refund policy in three bullets."}
]
}'
Swap google/gemini-flash-latest for anthropic/claude-sonnet-5–5 and nothing else in the request changes. Behind that one field, the gateway has to work out which model the client means, which provider serves it and how to speak that provider’s API. And it has to keep the promise when something fails.
The promise is easy to make and surprisingly easy to break. Here’s where it bends, and how the two example projects keep it.
Routing as code
The simplest router is a set of route rules. The llm-routing feature in gsjurseth/apigee-yaml-configs sends anything that starts with google/ to Gemini and everything else to OpenAI:
endpoints:
- name: default
basePath: "{BASE_PATH}"
routes:
- name: gemini
condition: llm.model StartsWith "google/"
target: gemini-target
- name: openai
target: openai-target
Apigee uses the first route rule that matches. The interesting part is the default, and the feature’s description tells a story worth learning from. An earlier version matched gpt- for OpenAI, which silently sent every other OpenAI model family, such as o1-* and chatgpt-*, to Gemini, where the requests failed. Inverting the default means new model families work without a change, as long as Gemini requests carry the google/ prefix.
That’s a good habit in general: make the rules you control specific, and let the default absorb other people’s naming decisions.
Routing as data
Route rules need a redeploy for every change. tyayers/apigee-completions-template moves the decision into data. A key-value map (KVM) policy reads one entry, ModelRoutingText, and a JavaScript step turns it into a target.route variable that the route rules match on:
{
"models": {
"google/": "googlecloud-oai",
"anthropic/": "googlecloud",
"openai/": "openai"
},
"mappings": {
"google/gemini-flash-latest": "google/gemini-3.8-flash"
}
}
models maps a provider prefix to a route family. mappings holds aliases: clients ask for gemini-flash-latest, and the platform team decides what that points to this month. The template caches the entry for ten seconds, so moving an alias to a new model version takes effect almost immediately, with no redeploy and no client change. That’s the platform team’s version of changing one string.
Speaking Claude
Gemini is the easy case. The OpenAI-compatible endpoint of Gemini Enterprise Agent Platform, formerly Vertex AI, accepts the client’s request as it is, apart from the model name.
Claude on Agent Platform speaks Anthropic’s Messages API through the rawPredict and streamRawPredict methods, so the gateway has to translate. tyayers’ template is a good reference. Its Claude target URL ends in publishers/{ai.provider}/models/{ai.model}:{ai.method}, and a JavaScript step sets ai.method to streamRawPredict or rawPredict, depending on whether the client asked for a stream.
On the way in, the translation moves system messages into Anthropic’s top-level system field, adds anthropic_version, fills in max_tokens (Anthropic requires it) and turns image data URLs into base64 image blocks. On the way out, it rebuilds an OpenAI chat.completion.
Streams are the interesting part. Anthropic sends typed events such as message_start, content_block_delta and message_stop, while OpenAI clients expect chat.completion.chunk objects and a final data: [DONE]. Apigee’s EventFlow runs policies on each server-sent event as it arrives, so the gateway can rewrite every event without buffering the answer.
Write down what your translation covers. Tool calls are the usual gap: if OpenAI-style tools aren’t mapped to Anthropic’s format, an agent that relies on them quietly gets plain text back. A clear 400 is kinder than a silent downgrade.
When a provider fails
One string also implies that the gateway, not the client, deals with a provider having a bad minute. Stage 4 of gsjurseth’s example adds cross-provider fallback with one feature. When the primary target answers with a 4xx or 5xx, the target’s DefaultFaultRule sends the prompt to the other provider through a ServiceCallout and returns that answer, with an x-apigee-fallback header so callers can tell (trimmed to the Gemini target):
targets:
- name: gemini-target
flows:
- name: PostFlow
mode: Response
steps:
- name: RF-TargetErrorToFault
condition: response.status.code >= 400
defaultFaultRule:
name: FallbackToOpenAI
steps:
- name: KVM-GetOpenAIKey
- name: AM-BuildOpenAIFallbackRequest
- name: SC-FallbackToOpenAI
- name: AM-CopyOpenAIFallbackResponse
It’s a teaching example, and its own comments say what production needs: translate the model name with a mapping instead of hard-coding the fallback model. I’d add two more refinements. Carry the whole conversation, not just the last prompt. And fall back only on errors another provider can fix, such as 429s, 5xx responses and timeouts. A malformed request fails at the second provider too, and you pay twice.
Errors are part of the contract
Everything the gateway can’t recover from still has to fail in a way OpenAI clients understand. By default, Apigee reports policy errors in its own envelope. A prompt blocked by Model Armor comes back as a 400 with a body like this:
{"fault": {"faultstring": "...", "detail": {"errorcode": "steps.sanitize.user.prompt.response.FilterMatched"}}}
An OpenAI SDK looks for error.message, error.type and error.code, and finds none of them. So the last feature in the chain should rewrite every error, whether it comes from upstream, from an Apigee policy or from the gateway itself, into OpenAI’s error envelope with a stable string code. For example:
- invalid_api_key for a 401
- model_not_found for a model the gateway doesn’t know
- rate_limit_exceeded for a 429, with a Retry-After header
- content_policy_violation when Model Armor blocks a prompt, with a neutral message
Then test the shape of your failures, not only their status codes. One jq check in every negative test catches most regressions:
jq -e '.error | (.message | type == "string") and has("type") and has("code")'
What I’d tell you
- Keep aliases and provider routing in data, with a short cache, so model changes don’t need a redeploy.
- Make the routing rules you control specific, and let the default absorb new model names.
- A compatibility promise covers requests, streams and errors. Errors are the part teams forget.
- Fall back on errors another provider can fix, and say so in a header.
One string for the client means careful plumbing in the gateway. That’s exactly where it belongs.
Next — Part 3: My favorite guardrail test is a polite trap. What it catches, what it can’t, and how masking and Model Armor fit together in Apigee.
Switching models should take one string was originally published in Google Cloud – Community on Medium, where people are continuing the conversation by highlighting and responding to this story.
Source Credit: https://medium.com/google-cloud/switching-models-should-take-one-string-213b15578009?source=rss—-e52cf94d98af—4
About the Author
