Never let a request decide who pays

Token quotas, API products and cost reports in an Apigee AI gateway, and the one rule that keeps them honest: only the credential decides.

Part 4 of 6 in Apigee as an AI gateway: common problems teams hit when they put LLMs into production, and how to solve them with Apigee, using open-source templates as examples.

Here’s a test I like to run against an AI gateway. It sends a free-plan API key, a large prompt and three extra headers:

curl "https://$GATEWAY_HOST/v1/chat/completions" \
  -H "x-api-key: $FREE_PLAN_KEY" \
  -H "x-user-id: someone-else" \
  -H "x-tier: enterprise" \
  -H "x-requested-tokens: 1" \
  -H "Content-Type: application/json" \
  -d @large-prompt.json

The test passes only if the gateway ignores all three. The usage must land on the key’s own app and developer, the free plan’s limits must apply, and the gateway must measure the prompt itself.

Each header is a tempting shortcut: a quick way to test plans, a hook to force a 429 in a test, a fallback for clients that don’t send an ID. Each one also lets the caller decide who pays. If a header can name the user, anyone can put someone else’s name on the bill. If it can choose the plan, everyone is on the biggest one. And if a fresh ID on every request means a fresh quota, nobody ever runs out.

Why tokens

For LLM traffic, two requests can differ in cost by a factor of a thousand. A requests-per-minute limit protects the gateway, but it does little for your budget, and model providers enforce their own token quotas anyway. So an AI gateway counts tokens per consumer, and consumers get plans: a small one for experiments, bigger ones for production apps.

The limits are rarely the hard part. The hard part is deciding who the consumer is and which plan they’re on.

Only the credential decides

Apigee’s AI policies are deliberately flexible. Their elements accept references to flow variables, and the LLMTokenQuota reference lists headers and query parameters as possible sources, next to API products, apps, developers and key-value maps. That flexibility is useful, but it means a policy can’t tell a header the caller typed from an attribute your platform team set. That choice is yours, and there’s one safe answer:

Identity comes from the credential. VerifyAPIKey maps the key to an app, its developer and the API products it may use.
Plan comes from the API product, or from custom attributes on the app or developer that only the platform team can change.
Size is measured by the gateway: prompt tokens before the call, and actual usage from the model’s response after it.

The LLMTokenQuota reference shows the pattern with per-developer attributes. Here it is as an enforcement policy:

<LLMTokenQuota name="LTQ-Enforce" type="rollingwindow">
  <SharedName>developer-tokens</SharedName>
  <EnforceOnly>true</EnforceOnly>
  <Identifier ref="verifyapikey.verify-api-key.client_id"/>
  <Interval ref="verifyapikey.verify-api-key.developer.timeInterval">1</Interval>
  <TimeUnit ref="verifyapikey.verify-api-key.developer.timeUnit">minute</TimeUnit>
  <Allow count="10000" countRef="verifyapikey.verify-api-key.developer.limit"/>
</LLMTokenQuota>

Each app gets its own counter, keyed on the verified client ID, and the limits come from the developer’s attributes. The values inside the elements are fallbacks. When a reference doesn’t resolve, Apigee uses the value instead, so a developer without attributes gets the smallest plan.

Never let a request decide who pays — *Figure 1 — Headers are ignored. The verified key decides identity and plan, and the gateway measures the tokens itself.*

Plans belong in API products

Attributes work, but the natural home for a plan is an API product. Products have LLM operations, which list the proxies, paths and models a plan may use, each with an optional token quota. VerifyAPIKey matches the key to its product and exposes the product’s LLM quota settings as flow variables. LLMTokenQuota reads them through references such as verifyapikey.verify-api-key.apiproduct.developer.llmQuota.limit, or all at once with UseQuotaConfigInAPIProduct.

Plans are configuration, so keep them in version control next to the gateway. apigee-templater has a YAML format for products (trimmed):

name: ai-standard-plan
displayName: AI Standard Plan
type: product
approvalType: manual
environments:
  - prod
proxies:
  - llm-gateway
llmOperations:
  - apiSource: llm-gateway
    operations:
      - name: /chat/completions
        model: google/gemini-3.8-flash
        methods:
          - POST

Two details are worth a second look. With approvalType: manual, someone on the platform team approves every key request for the plan, so nobody upgrades themselves. And LLMTokenQuota reads the model from the request’s model field by default and matches it against the product’s LLM operations, so make sure the names agree, especially if the gateway resolves aliases (Part 2). apigee-templater can also export existing products to YAML and deploy products together with a template, which makes plan changes as reviewable as code.

Enforce before, count after

Three policies cover three jobs:

PromptTokenLimit works like SpikeArrest for prompt tokens. It stops a burst of huge prompts before they reach the model.
LLMTokenQuota holds the budget per minute, day or month.
A classic Quota policy still caps requests, which protects the gateway itself.

You can’t know how many tokens a model will produce until it has produced them, so LLMTokenQuota runs twice on one shared counter. In the request flow, EnforceOnly rejects a request when the budget is spent, but never adds to the counter. In the response flow, CountOnly adds the tokens the model actually used, but never rejects. SharedName ties the two together. Point the counting policy’s LLMTokenUsageSource at the response’s usage data: the reference examples read Gemini’s native usageMetadata, while an OpenAI-compatible gateway reads usage.

*Figure 2 — Requests are admitted while budget remains, and actual usage is added afterwards. A consumer can overshoot once within a window and is then blocked until the window moves on.*

One large answer can push a consumer past the limit, and the next request has to wait. That’s a reasonable trade-off. Reserving the worst-case output for every request would waste most of the budget.

Two more details:

Streams. Put the counting policy in the EventFlow. It counts when an event carries usage data and does nothing for the others, so make sure your streams include usage.
The 429. Both token policies answer a violation with a 429. Map it to OpenAI’s rate_limit_exceeded with a Retry-After header (Part 2), so SDK retry logic works unchanged.

Telling finance

Limits protect the budget. Reports explain it. tyayers’ template turns usage into money after every response. A JavaScript step reads the token counts from the response, or from whichever stream event carries them, and multiplies them by per-model prices from a key-value map. Then a DataCapture policy writes the results to custom data collectors (trimmed):

- name: DC-TokenAnalytics
  type: DataCapture
  content:
    DataCapture:
      Capture:
        - Collect:
            metadata:
              ref: ai.model
              default: unknown
          DataCollector: dc_ai_model
        - Collect:
            metadata:
              ref: ai.totalTokenCount
              default: "0"
          DataCollector: dc_ai_total_token_count
        - Collect:
            metadata:
              ref: ai.totalCost
              default: "0"
          DataCollector: dc_ai_total_cost

The template’s initialize.sh creates data collectors and three custom reports. The cost report sums cost by two dimensions, dc_ai_model and developer_email.

developer_email is the interesting one, because no client sends it. According to the analytics reference, Apigee fills it in only when the proxy checks an API key or OAuth token for an app in an API product. Otherwise every row says (not set). The template’s README treats API key verification as optional, which is the right call for a quick start. Turn it on before you show anyone a cost report. It’s the difference between cost per developer and one big total.

*Figure 3 — Token counts and prices become cost figures in custom reports. The verified key supplies the developer dimension.*

Two caveats. These figures are estimates, not invoices: prices change, and a price list doesn’t know about discounts, so reconcile them with Cloud Billing. And streams report usage at the end, if at all, so check that each provider you route to actually sends it.

What I’d tell you

Take identity, plan and size from the verified credential and the gateway’s own measurements, never from the request.
Keep plans in API products, review them like code, and make upgrades a human decision.
Enforce before the call and count after it, on one shared counter.
Turn on API key verification before you build a single cost report.

Every header is a suggestion from the caller. Budgets, limits and invoices deserve more than suggestions.

Next — Part 5: A semantic cache is easy to demo and hard to trust. What the cache key really is, why a threshold has a direction, and how the two cache policies reach Vector Search.

Never let a request decide who pays was originally published in Google Cloud – Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

Source Credit: https://medium.com/google-cloud/never-let-a-request-decide-who-pays-1090ee69fa1e?source=rss—-e52cf94d98af—4