Tokens are the currency of large language models. Every prompt you send is split into tokens — small chunk of text, and every token the model reads or writes is counted, billed, and adds to how long you wait for an answer. A modern LLM app sends a lot of tokens. A request rarely contains just the user’s question. It usually also carries:
- System Prompt — telling the model how to behave
- Instructions and examples for good answers
- Documents pulled from a knowledge base (policies, FAQs, product catalogs)
- Tool definitions for agents
The catch is that most of that surrounding context is the same on every request: the same system prompt, the same policy document and the same tool list, sent again and again. The model rereads all of it every time, and you pay for every one of those tokens every single time. That’s the problem context caching solves. In this blog we will learn how it works, using Gemini on Google Cloud Agent Platform and a real retail example, with real example on token counts and real costs. First, let’s see where the money actually goes.
Token Economics: Where the Money Goes
LLM APIs don’t bill per request. They bill per token, and there are two kinds:
- Input tokens: everything you send (system prompt, documents, chat history, the question)
- Output tokens: everything the model writes back
Output tokens cost more per token than input tokens.
Caching in an LLM
When Gemini answers a request, it works in two phases:
- Prefill: It reads every input token and builds an internal representation of the prompt, called the KV cache (key–value cache). This is where the work on your input tokens happens.
- Decode: it writes the answer one token at a time, using that representation.
Ways to Cache on Agent Platform
Agent Platform gives you two ways to do this:
- Implicit caching: automatic, free, best effort
- Explicit caching: you create and manage the cache yourself, and the discount is guaranteed
In this blog we will walk through both with a real grocery-retail example, real token counts from real API calls, the dollar cost before and after, the context-window rules that decide whether something gets cached at all, and the commands to inspect your caches.
Scenario
FreshMarket, a grocery and pharmacy chain, runs an AI support assistant on Gemini that answers product return and refund questions from shoppers and store agents. Every answer depends on the same Return & Refund Policy policy document which is approx 4,650-token, so each request sends that full policy plus a short customer query/questions which approx 15-token question. We will run this setup with implicit and explicit caching on Agent Platform to see how many tokens each one actually caches, and how much money each one saves.
The policy document (approx 4650 tokens):
FRESHMARKET GROCERY — RETURN & REFUND POLICY
(A grocery, pharmacy & essentials retailer — Albertsons-style)
1. GENERAL RETURNS
We want you to be happy with every purchase. Most non-perishable items can be returned within 30 days with a receipt or loyalty account lookup, for a full refund to your original payment method.
2. GROCERY & PERISHABLE ITEMS
Fresh produce, meat, dairy, bakery, and deli items are covered by our Freshness Guarantee: if you're not satisfied with the quality, bring back the item (or a photo of it, for delivery orders) within 7 days for a full refund or replacement — no receipt needed if purchased with a loyalty account.
3. PHARMACY ITEMS
Prescription medications cannot be returned once dispensed, per state and federal law. Over-the-counter health items (vitamins, cold medicine, etc.) can be returned unopened within 30 days.
4. ALCOHOL, TOBACCO & GIFT CARDS
These items are final sale and cannot be returned or refunded, in accordance with state regulations.
5. ONLINE ORDERS, DELIVERY & CURBSIDE PICKUP
If an online, delivery, or curbside order arrives damaged, missing items, or incorrect, report it within 48 hours through the app or website for a full refund or free redelivery — no need to return the item.
6. LOYALTY PROGRAM MEMBERS
Members of our rewards program get:
- Extended return window: 45 days instead of 30 on non-perishable items
- Refunds processed as instant loyalty account credit, available immediately
- No receipt needed — purchases are automatically looked up via phone number or member ID
7. WIC & SNAP/EBT PURCHASES
Items purchased with WIC or SNAP/EBT benefits can be exchanged for the same or a similar item of equal value, per federal program rules. Cash refunds are not permitted on these purchases.
8. HOW TO RETURN AN ITEM
- In-store: Bring the item and receipt (or use loyalty lookup) to the Customer Service desk.
- Online orders: Start a return from the "My Orders" page in the app or website.
- Delivery issues: Use the "Report a Problem" button within 48 hours of delivery.
9. QUESTIONS?
Contact Customer Care at 1-800-555-0142 or support@freshmarketgrocery.com, daily 7am–10pm.
This policy may change at any time; the version in effect at the time of purchase applies to that order.
APPENDIX A — DEPARTMENT-SPECIFIC RULES
A1. MEAT & SEAFOOD
Fresh meat, poultry, and seafood are covered by the Freshness Guarantee for 7 days from purchase or until the "use by" date, whichever comes first. Items that were frozen after purchase remain eligible if the customer reports a quality issue on first thaw. Live or whole seafood (lobster, crab, oysters) must be reported within 24 hours of purchase. Custom butcher cuts prepared to order are refundable only for quality defects, not for change of mind. Store associates should never ask a customer to bring back raw meat that has been opened at home; a photo or verbal description is enough to issue a refund or replacement.
A2. BAKERY & CUSTOM CAKES
Packaged bakery items (bread, rolls, muffins) follow the standard 7-day Freshness Guarantee. Custom-decorated cakes and special-order platters can be cancelled for a full refund up to 48 hours before the scheduled pickup time. Cancellations within 48 hours receive a 50% refund, because the ingredients and labor have already been committed. If a custom cake is incorrect at pickup (wrong name, wrong flavor, wrong design), the customer may choose a full refund or a free remake, and receives a $10 loyalty credit for the inconvenience.
A3. DELI & PREPARED FOODS
Hot foods, rotisserie chicken, sushi, salad bar, and prepared meals are refundable for quality issues on the same day of purchase only. Deli-sliced meats and cheeses follow the 7-day Freshness Guarantee. Catering trays ordered for events can be cancelled for a full refund up to 72 hours in advance; after that, a 25% cancellation fee applies. Food safety concerns (foreign objects, suspected contamination, undercooked items) must always be escalated to the Store Manager, who will file an incident report and issue a full refund regardless of timing.
A4. FLORAL
Fresh-cut flowers and bouquets are guaranteed for 7 days. If they wilt early, the customer may return them or show a photo for a full refund or replacement. Potted plants carry a 30-day guarantee against plant failure when care instructions were followed. Balloons and seasonal decorations are final sale once inflated or opened.
A5. HOUSEHOLD, GENERAL MERCHANDISE & SMALL APPLIANCES
Cleaning supplies, paper goods, kitchenware, and seasonal items follow the standard 30-day policy (45 days for loyalty members). Small electrical appliances (coffee makers, blenders, toasters) must be returned with all original packaging and accessories within 30 days. After 30 days, the customer should be directed to the manufacturer's warranty. Opened hazardous products (lighter fluid, pool chemicals, aerosol pesticides) cannot be returned for safety reasons unless defective.
A6. BABY PRODUCTS
Unopened infant formula, diapers, and baby food can be returned within 30 days with a receipt or loyalty lookup. Opened infant formula cannot be returned for safety reasons, except in the case of a manufacturer recall. Formula purchased with WIC benefits is subject to Section 7 and may only be exchanged for an equivalent WIC-approved product.
A7. HEALTH, BEAUTY & PERSONAL CARE
Cosmetics, skincare, and hair care products can be returned within 30 days even if opened, provided at least half of the product remains and the customer experienced an adverse reaction or was dissatisfied with results. Razors, toothbrushes, and other personal hygiene devices can only be returned unopened. Hearing aids, blood pressure monitors, and other medical devices sold at the pharmacy counter follow the manufacturer's return policy.
A8. PHARMACY — ADDITIONAL DETAIL
Prescription medications cannot be returned or exchanged once they leave the pharmacy, even if unopened, due to federal and state safety regulations. If a prescription was filled incorrectly (wrong drug, wrong strength, wrong patient), the pharmacist on duty must be notified immediately; the customer receives a corrected fill at no cost and any copay is refunded. Unused medications can be disposed of safely using the drug take-back kiosk located near the pharmacy counter; this is a disposal service and does not generate a refund. Vaccination services, health screenings, and consultation fees are non-refundable once the service is performed.
APPENDIX B — PAYMENT & REFUND METHODS
B1. ORIGINAL PAYMENT METHOD
Refunds are issued to the original form of payment whenever possible. Credit and debit card refunds typically appear within 3–5 business days, depending on the bank. Cash purchases are refunded in cash at the Customer Service desk for amounts up to $200; larger cash refunds are issued as a check mailed within 10 business days.
B2. SPLIT TENDER
When a purchase was paid with more than one method (for example, part gift card and part credit card), the refund is applied in the following order: first to any WIC or SNAP/EBT benefit (as an exchange only, per Section 7), then back to the credit or debit card, and finally to a new gift card for any remaining balance originally paid by gift card.
B3. DIGITAL COUPONS & LOYALTY REWARDS
If a returned item was purchased using a digital coupon or loyalty reward, the refund equals the amount the customer actually paid after discounts. Redeemed loyalty points used on a returned item are restored to the customer's account within 24 hours. Fuel rewards earned on a returned purchase are deducted from the customer's fuel rewards balance.
B4. NO RECEIPT & NO LOYALTY ACCOUNT
Customers without a receipt and without a loyalty account may receive store credit for the item's lowest selling price in the past 60 days, limited to three no-receipt returns per 90-day period. A valid government-issued photo ID is required for all no-receipt returns and is used only to enforce this limit.
B5. PRICE ADJUSTMENTS
If an item's shelf price drops within 7 days of purchase, loyalty members may request a one-time price adjustment for the difference. Price adjustments do not apply to clearance, "Manager's Special," or limited-time flash deals.
APPENDIX C — SPECIAL SITUATIONS
C1. PRODUCT RECALLS
Any item subject to a manufacturer or government recall can be returned for a full refund at any time, with or without a receipt, even if opened or partially consumed. Loyalty members who purchased a recalled item are notified automatically by email and app notification.
C2. HOLIDAY RETURN WINDOW
Non-perishable items purchased between November 15 and December 24 can be returned until January 31 of the following year. This extension does not apply to perishable items, alcohol, tobacco, gift cards, or prescription medications.
C3. THIRD-PARTY DELIVERY PARTNERS
Orders placed through third-party delivery apps (not the FreshMarket app or website) must be reported to the third-party provider first. FreshMarket will honor the Freshness Guarantee for quality issues on items from those orders when the customer brings the item or receipt to a store, but delivery fees and tips can only be refunded by the third-party provider.
C4. BULK & BUSINESS PURCHASES
Purchases of more than $1,000 in a single transaction, or more than 10 units of the same item, may require Store Manager approval for return. This helps prevent resale fraud and does not affect the customer's rights under the Freshness Guarantee or a product recall.
C5. DAMAGED IN STORE
If a customer is injured by or reports a hazard involving a product while in the store, associates must stop, ensure the customer's safety, notify the Store Manager, and complete an incident report before processing any refund.
C6. ABUSE OF POLICY
FreshMarket reserves the right to decline a return when there is evidence of fraud, repeated abuse of the Freshness Guarantee, or returns of stolen merchandise. Associates should never accuse a customer directly; instead, politely explain that the return requires Store Manager review.
APPENDIX D — ASSOCIATE GUIDELINES FOR SUPPORT AGENTS
D1. TONE
Always greet the customer warmly, acknowledge the issue, and apologize for the inconvenience before explaining the policy. Avoid internal jargon. Use the customer's name when it is known.
D2. ESCALATION
Escalate to a Store Manager or Customer Care supervisor when: the refund exceeds $250; the issue involves food safety, injury, or a pharmacy error; the customer requests an exception to the written policy; or the customer has been unable to resolve the issue after two contacts.
D3. GOODWILL CREDITS
Support agents may issue a goodwill loyalty credit of up to $10 per contact, and up to $25 with supervisor approval, for service failures such as late deliveries, repeated missing items, or incorrect custom orders. Goodwill credits cannot be converted to cash.
D4. DOCUMENTATION
Every refund, replacement, or goodwill credit must be recorded in the customer's account notes with the reason code, the policy section applied, and the associate's ID. This ensures consistent treatment if the customer contacts support again.
D5. WHEN THE POLICY IS UNCLEAR
If a situation is not clearly covered by this policy, the agent should say so honestly, avoid guessing, and escalate to a supervisor. Never promise an outcome that the policy does not support.
APPENDIX E — FREQUENTLY ASKED QUESTIONS
E1. I lost my receipt. Can I still return something?
Yes. If you shopped with your loyalty account, we can find the purchase using your phone number or member ID and process the return as if you had the receipt. If you did not use a loyalty account, you may receive store credit for the item's lowest selling price in the past 60 days, with a valid photo ID, up to three times in any 90-day period (see Section B4).
E2. Can I return an item to a different FreshMarket store than the one I bought it from?
Yes. Any FreshMarket store can accept returns for items bought at any other FreshMarket location, including online orders picked up at a different store. Store credit and loyalty credits are valid across all locations. Returns for items bought at partner banner stores are accepted only when the item is also carried at the FreshMarket store where it is being returned.
E3. How long does a refund take to show up?
Loyalty account credits are instant. Card refunds usually appear within 3–5 business days, but some banks take up to 10 business days to post the credit. Mailed checks for large cash refunds arrive within 10 business days. If a customer has not received a card refund after 10 business days, the agent should provide the refund transaction reference number and suggest the customer contact their bank.
E4. Can I get a refund on a product I just didn't like the taste of?
Yes, for grocery items. Our Freshness Guarantee and our Satisfaction Promise cover taste and quality complaints on food items, including our store-brand products. Store-brand items (FreshMarket Signature, FreshMarket Organics, and Value Corner) carry a "Love It or It's Free" promise: if you don't love it, bring it back within 30 days, even if opened or partially used, for a full refund or an exchange for a national brand item. Alcohol is excluded (see Section 4).
E5. Are sale and clearance items returnable?
Sale items follow the same policy as regular-priced items; the refund equals the price actually paid. Items marked "Final Clearance" or "As-Is" on the shelf tag or receipt are final sale and cannot be returned unless defective or recalled.
E6. Can I exchange an item instead of getting a refund?
Yes. Exchanges for the same item (for example, a different size, flavor, or an undamaged replacement) can be made within the applicable return window. If the replacement costs more, the customer pays the difference; if it costs less, the difference is refunded to the original payment method or as loyalty credit.
E7. What if my item was damaged before I opened it?
Items found damaged, broken, or spoiled on first opening are always eligible for a full refund or replacement within the applicable return window, even when the category is otherwise final sale (for example, a sealed bottle of wine that is found to be leaking at the register can be exchanged before leaving the store). Once alcohol has left the store, state regulations still apply.
E8. Can I return a gift?
Gifts with a gift receipt can be returned for store credit or loyalty credit in the recipient's name, for the price paid by the gift giver. Without a gift receipt, gifts follow the no-receipt policy in Section B4.
E9. Do returns affect my loyalty points?
Yes. Points earned on a returned item are removed from the account when the return is processed. If this would make the points balance negative, the balance is set to zero rather than going negative; the customer is never charged for points.
E10. What about fuel purchases at FreshMarket fuel stations?
Fuel purchases are non-refundable once dispensed. Pump malfunctions, overcharges, or duplicate charges should be reported to the fuel station attendant or to Customer Care within 7 days, with the pump number and approximate time of purchase, so the transaction can be reviewed and corrected.
APPENDIX F — WORKED EXAMPLES FOR SUPPORT AGENTS
F1. Opened wine the customer didn't like
Situation: A customer bought a bottle of wine, opened it at home, didn't like it, and wants a refund.
Correct answer: Alcohol is final sale per Section 4 and state regulations. Politely explain that we cannot refund or exchange alcohol once it leaves the store. The agent may offer a goodwill loyalty credit of up to $10 only if there was a service failure (for example, an associate incorrectly told the customer the wine was returnable); dislike of taste alone is not a service failure.
F2. Rotisserie chicken bought yesterday
Situation: A customer says yesterday's rotisserie chicken was dry and wants a refund today.
Correct answer: Hot prepared foods are refundable for quality issues on the same day only (Section A3). Because the complaint is a day late, the agent should explain the policy kindly. If the customer reports a food-safety concern (for example, undercooked meat or illness), this changes the situation: escalate to the Store Manager, file an incident report, and issue a full refund regardless of timing.
F3. Loyalty member, no receipt, 40-day-old blender
Situation: A loyalty member wants to return a blender bought 40 days ago, with no receipt, in the original box.
Correct answer: Loyalty members get 45 days on non-perishable items (Section 6), and the purchase can be looked up by phone number. However, small appliances require return within 30 days (Section A5), which takes priority because it is the more specific rule. Direct the customer to the manufacturer's warranty. If unsure, escalate per Section D5.
F4. SNAP-purchased baby formula, customer asks for cash
Situation: A customer bought infant formula with SNAP/EBT benefits and wants cash back.
Correct answer: Cash refunds are not permitted on SNAP/EBT or WIC purchases (Section 7). Unopened formula can be exchanged for the same or a similar item of equal value. Opened formula cannot be returned for safety reasons unless it is part of a recall (Section A6).
F5. Missing items in a delivery order, reported 3 days later
Situation: A customer reports a missing item from a delivery order 3 days after delivery.
Correct answer: Delivery issues must be reported within 48 hours (Section 5). Since the report is late, the agent should explain the window. For a first-time occurrence and a small amount, the agent may use judgment and issue a goodwill loyalty credit up to $10 (Section D3). Record the reason code in the account notes (Section D4).
F6. Wrong prescription strength
Situation: A customer discovers their prescription was filled at the wrong strength.
Correct answer: This is a pharmacy error, not a return. Notify the pharmacist on duty immediately; the customer receives a corrected fill at no cost and any copay is refunded (Section A8). Escalate as a pharmacy error per Section D2.
F7. Custom birthday cake cancelled the day before
Situation: A customer cancels a custom cake 24 hours before pickup.
Correct answer: Cancellations within 48 hours of pickup receive a 50% refund (Section A2). Explain that the ingredients and decorator time have already been scheduled.
F8. Recalled peanut butter, jar half eaten, no receipt
Situation: A customer brings in a half-eaten jar of peanut butter that is on a recall list, with no receipt.
Correct answer: Recalled items are refundable at any time, with or without a receipt, even if opened or partially consumed (Section C1). Issue a full refund for the current shelf price.
F9. Gift card purchased by mistake
Situation: A customer bought a $100 gift card by mistake and wants the money back.
Correct answer: Gift cards are final sale (Section 4). If the card was purchased in the current visit and has not been activated or used, the Store Manager may void the transaction at the register; otherwise, it cannot be refunded.
F10. Bulk return of 20 cases of water
Situation: A customer wants to return 20 cases of bottled water bought for a cancelled event.
Correct answer: Returns of more than 10 units of the same item require Store Manager approval (Section C4). The water is otherwise returnable within the standard window with a receipt or loyalty lookup.
APPENDIX G — STORE SERVICES & HOURS
G1. CUSTOMER SERVICE DESK HOURS
The Customer Service desk is open from 7am to 10pm daily in most stores. Returns cannot be processed at self-checkout lanes. During holidays, desk hours may be shortened; holiday hours are posted in the store and on the website at least one week in advance.
G2. PHARMACY HOURS
Pharmacy hours vary by location, typically 9am–9pm Monday through Friday, 9am–6pm Saturday, and 10am–5pm Sunday. Pharmacy-related returns and errors must be handled at the pharmacy counter, not at the Customer Service desk.
G3. ONLINE AND APP SUPPORT
Online and delivery issues can be reported 24/7 through the app or website. Chat support with a live agent is available daily from 7am to 10pm local time. Outside these hours, the virtual assistant can process standard refunds for missing or damaged delivery items automatically, using the rules in Section 5.
G4. ACCESSIBILITY
Customers who cannot visit a store due to mobility, health, or distance may request a return pickup through Customer Care for orders placed online. Associates should offer this option proactively when a customer mentions difficulty returning an item in person.
G5. LANGUAGE SUPPORT
Customer Care offers support in English, Spanish, Vietnamese, Chinese (Mandarin and Cantonese), Tagalog, and Korean. In-store associates can connect customers to phone interpretation services in over 150 languages at the Customer Service desk.
The Context Window
Before looking at code, we need to understand the concept of context window, because it decides what can be cached. The context window is everything the model reads in one request: system instruction, documents, chat history, and your question. It also has to leave room for the answer. Caching works on the beginning of the context window, called the prefix.
Cache Hit Logic:
The model reads a prompt left to right, one token at a time. For each token it computes internal data (the “KV cache”), and that data depends on the token itself plus every token before it. That leads to one rule: Saved work can only be reused if everything before it is exactly the same. So caching can only reuse the beginning of a prompt (the prefix), up to the first token that differs. Everything after that is computed fresh and billed at full price.
✅ Policy first, question last:
Call 1: [Policy policy policy ... policy] [Can I return produce?]
Call 2: [Policy policy policy ... policy] [Do you refund prescriptions?]
└──────── identical ────────┘ └── differs from here ──┘
reused ✅ (4,600+ tokens) computed fresh (~15 tokens)
❌ Question first, policy last:
Call 1: [Can I return produce?] [Policy policy ... policy]
Call 2: [Do you refund prescriptions?] [Policy policy ... policy]
└── differs at token 1 ──┘
nothing reusable ❌
Implicit vs Explicit Caching
How implicit caching decides on a hit: Nobody sets up an implicit cache. You send the full prompt (policy plus question) every time, and Google checks whether it recently processed a prompt that started the same way. It compares the start of the prompt in fixed-size blocks, so a partial block at the end is never cached. A hit also isn’t guaranteed: it misses if the start of the prompt differs, if too much time has passed, or if the request lands on a server that doesn’t have the saved work.
How explicit caching decides on a hit: We create the cache up front. The policy and system instruction are processed once and stored as a fixed starting point. After that, each request sends only the question plus a reference to the cache, and Agent Platform puts the question after the stored policy. The starting point is always identical, so every call is an exact match. It only fails if the cache has expired or been deleted, or if you call it with a different model or location.
A simple mind map on how caching decision can be taken for a model.
With this, lets deep dive with the actual demo.
Implicit Caching (Automatic)
This is the easy one. You don’t write any caching code. You put the big, unchanging content first and the short, unique question last. If Agent Platform notices the same prefix again shortly after, it reuses the processed prefix and bills those tokens at a 90% discount.
"""
Implicit Context Caching Demo — Vertex AI Gemini
Use case: FreshMarket Grocery support chatbot answering shopper
questions using the same Return & Refund Policy document as context.
Before running:
- Make sure store_return_policy.txt is in the same folder.
- Make sure you are authenticated: `gcloud auth application-default login`
- The shared prefix must be at least 2,048 tokens (2.5 Flash) or it is never cached.
"""
from google import genai
from google.genai.types import HttpOptions, Part
PROJECT_ID = "GCP_PROJECT_ID"
LOCATION = "us-central1"
MODEL = "gemini-2.5-flash"
client = genai.Client(
vertexai=True,
project=PROJECT_ID,
location=LOCATION,
http_options=HttpOptions(api_version="v1"),
)
# Load the big, unchanging policy document once
with open("store_return_policy.txt", "r", encoding="utf-8") as f:
return_policy = f.read()
def ask_support_bot(question: str):
response = client.models.generate_content(
model=MODEL,
contents=[
Part.from_text(text=return_policy), # same content every time
Part.from_text(text=question), # only this changes
],
)
usage = response.usage_metadata
print(f"Q: {question}")
print(f"A: {response.text}\n")
print(f"Total input tokens: {usage.prompt_token_count}")
# None means no cache hit on this call
print(f"Cached (discounted) tokens: {usage.cached_content_token_count or 0}\n")
print("-" * 70)
if __name__ == "__main__":
# A shopper asks a few questions in the same chat session.
# The first call is billed normally; the following calls,
# sent shortly after with the same policy prefix, should
# automatically get a cache-hit discount (best effort, not guaranteed;
# in testing the first hit sometimes only showed up after 5–6 calls).
ask_support_bot("Can I return spoiled produce without a receipt?")
ask_support_bot("Do you refund prescription medications?")
ask_support_bot("What happens if my delivery order arrives with a missing item?")
ask_support_bot("Can I return a gift without a gift receipt?")
ask_support_bot("I cancelled my custom cake the day before pickup. What refund do I get?")
ask_support_bot("Is fuel refundable if the pump overcharged me?")
Output:
Q: Can I return spoiled produce without a receipt?
A: Yes, you can return spoiled produce.
FreshMarket's Freshness Guarantee (Section 2) covers fresh produce:
* If you're not satisfied with the quality, you can bring the item back (or a photo of it, for delivery orders) within **7 days** for a full refund or replacement.
* **No receipt is needed if you purchased it with a loyalty account.** They can look up your purchase using your phone number or member ID (Section 2, E1).
If you do **not** have a loyalty account and **no receipt**, the general policy for no-receipt returns (Section B4, E1) would typically apply. This means you might receive store credit for the item's lowest selling price in the past 60 days, provided you have a valid photo ID.
Given it's a "Freshness Guarantee" and the item is spoiled, FreshMarket is generally inclined to resolve quality issues for food items.
Total input tokens: 4654
Cached (discounted) tokens: 0
----------------------------------------------------------------------
Q: Do you refund prescription medications?
A: No, prescription medications cannot be returned or refunded once they have been dispensed or leave the pharmacy, due to state and federal safety regulations (Sections 3 and A8).
However, if a prescription was filled incorrectly (wrong drug, wrong strength, or wrong patient), you should notify the pharmacist on duty immediately. In such cases, you will receive a corrected fill at no cost, and any copay will be refunded (Section A8).
For unused medications you wish to dispose of safely, you can use the drug take-back kiosk located near the pharmacy counter, but this is a disposal service and does not generate a refund (Section A8).
Total input tokens: 4651
Cached (discounted) tokens: 4075
----------------------------------------------------------------------
Q: What happens if my delivery order arrives with a missing item?
A: If your FreshMarket delivery order arrives with a missing item, you should **report it within 48 hours of delivery** through the FreshMarket app or website.
Upon reporting, you will be offered a **full refund** for the missing item or a **free redelivery** of that item. There is **no need to return** the item that *wasn't* delivered.
(As noted in example F5, if you report the missing item after the 48-hour window, for a first-time occurrence and a small amount, a goodwill loyalty credit of up to $10 might be issued.)
Total input tokens: 4657
Cached (discounted) tokens: 0
----------------------------------------------------------------------
Q: Can I return a gift without a gift receipt?
A: Yes, you can return a gift without a gift receipt at FreshMarket Grocery.
According to Section E8 and Section B4 of the policy:
* **What you get:** You will receive **store credit** for the item's lowest selling price in the past 60 days.
* **Requirements:** You will need a valid government-issued photo ID.
* **Limitations:** This type of return is limited to three no-receipt returns per 90-day period.
Total input tokens: 4655
Cached (discounted) tokens: 0
----------------------------------------------------------------------
Q: I cancelled my custom cake the day before pickup. What refund do I get?
A: According to Section A2 (Bakery & Custom Cakes) of the FreshMarket Grocery return policy, if you cancel a custom cake within 48 hours of the scheduled pickup time (which "the day before pickup" falls under), you will receive a **50% refund**. This is because the ingredients and labor would have already been committed.
Total input tokens: 4661
Cached (discounted) tokens: 0
----------------------------------------------------------------------
Q: Is fuel refundable if the pump overcharged me?
A: According to Appendix E, Section E10 of the policy:
While fuel purchases are generally non-refundable once dispensed, if you were **overcharged** due to a pump malfunction or duplicate charge, you should **report it to the fuel station attendant or to Customer Care within 7 days.**
You'll need to provide the pump number and approximate time of purchase so the transaction can be reviewed and corrected.
Total input tokens: 4655
Cached (discounted) tokens: 0
----------------------------------------------------------------------
Only call 2 was a cache hit (4,075 tokens), so this run got 1 of 6 cache hits.
- Call 1 — Processes the policy and saves it on whichever server handled it. It’s a miss unless an earlier run already saved it.
- Call 2 — Arrives right after, while that saved copy is freshest and most likely to be on the server the request lands on
- Call 3–6 — Traffic spreads across other servers, and the saved copy may be evicted or simply not on the server that gets the request
Agent Platform doesnt run Gemini on one machine. Each request goes to one of many servers, and the saved prefix work is kept on the server that did it, not shared across all of them. Google Cloud decides where each request goes, based on load. You can’t pin a request to a server. So even an identical prompt sent a second later can land somewhere with nothing saved.
Explicit Caching
Now imagine FreshMarket’s support team handles hundreds of tickets during a shift, spread over hours. Implicit caching would miss many of them. You want a guaranteed discount on tokens, so you create the cache yourself, reuse it all shift, and delete it at the end.
"""
Explicit Context Caching Demo — Vertex AI Gemini
--------------------------------------------------
Use case: FreshMarket Grocery support team handling hundreds of
tickets during a shift, all referencing the same Return & Refund
Policy document.
Before running:
- Make sure store_return_policy.txt is in the same folder.
- Make sure you're authenticated: `gcloud auth application-default login`
- The cached content must be at least 4,096 tokens for Gemini 3.8 Flash
(2,048 for 2.5 models) or caches.create() fails.
"""
from google import genai
from google.genai.types import (
CreateCachedContentConfig,
GenerateContentConfig,
HttpOptions,
Part,
)
from datetime import timedelta
PROJECT_ID = "GCP_PROJECT_ID"
# Gemini 3.8 Flash is not in single regions (us-central1 etc.), only "global" and
# the "us"/"eu" multi-regions. Our org policy (gcp.resourceLocations) blocks
# creating caches in "global", so we use "us".
LOCATION = "us"
MODEL = "gemini-3.8-flash"
client = genai.Client(
vertexai=True,
project=PROJECT_ID,
location=LOCATION,
http_options=HttpOptions(api_version="v1"),
)
# Step 1: Load the policy document
with open("store_return_policy.txt", "r", encoding="utf-8") as f:
return_policy = f.read()
# Step 2: Create the cache once, valid for the length of a support shift
cache = client.caches.create(
model=MODEL,
config=CreateCachedContentConfig(
contents=[Part.from_text(text=return_policy)],
system_instruction=(
"You are a FreshMarket customer support assistant. "
"Answer using only the attached return policy."
),
display_name="freshmarket_return_policy_cache",
ttl=f"{int(timedelta(hours=2).total_seconds())}s", # "7200s", covers a typical shift
),
)
print(f"Cache created: {cache.name}")
print(f"Cached tokens: {cache.usage_metadata.total_token_count}")
print(f"Expires at: {cache.expire_time}\n")
# Step 3: Every ticket references the cache by name — only the question is sent
def handle_ticket(customer_question: str):
response = client.models.generate_content(
model=MODEL,
contents=[Part.from_text(text=customer_question)],
config=GenerateContentConfig(cached_content=cache.name),
)
usage = response.usage_metadata
print(f"Total input tokens: {usage.prompt_token_count}")
print(f"Cached (discounted) tokens: {usage.cached_content_token_count or 0}\n")
return response.text
if __name__ == "__main__":
tickets = [
"A customer wants to return alcohol they didn't like. Is that allowed?",
"A loyalty member wants a refund without a receipt — is that OK?",
"Customer used SNAP benefits and wants a cash refund on baby formula.",
]
try:
for question in tickets:
print(f"Q: {question}")
print(f"A: {handle_ticket(question)}\n")
print("-" * 70)
finally:
# Step 4: Delete the cache when the shift ends, to stop paying storage
client.caches.delete(name=cache.name)
print(f"\nCache {cache.name} deleted.")
Output:
Cache created: projects/12345678/locations/us/cachedContents/6602244618
Cached tokens: 4663
Expires at: 2026-09-26 12:56:58.695406+00:00
Q: A customer wants to return alcohol they didn't like. Is that allowed?
Direct use of automatic function calling (AFC) in Models.generate_content is not recommended. Instead, we recommend to use AFC in Chat.send_message. Similarly, direct use of AFC in Models.generate_content_stream is not recommended. Instead, we recommend to use AFC in Chat.send_message_stream.
Total input tokens: 4678
Cached (discounted) tokens: 4663
A: No, that is not allowed.
According to our policy, alcohol is final sale and cannot be returned, refunded, or exchanged once it leaves the store, in accordance with state regulations (Section 4 and Appendix F1). Dislike of taste alone does not qualify for a refund or credit.
----------------------------------------------------------------------
Q: A loyalty member wants a refund without a receipt — is that OK?
Total input tokens: 4676
Cached (discounted) tokens: 4663
A: Yes, that is completely okay! As a loyalty program member, you do not need a physical receipt.
Your purchase can be automatically looked up in the system using your phone number or member ID, and the return will be processed just as if you had the receipt (as long as the item is within the applicable return window, such as 45 days for non-perishable items or 7 days for perishable items).
----------------------------------------------------------------------
Q: Customer used SNAP benefits and wants a cash refund on baby formula.
Total input tokens: 4675
Cached (discounted) tokens: 4663
A: Hello! Thank you for contacting FreshMarket Customer Care.
Per federal program regulations and our store policy, **cash refunds are not permitted on purchases made with SNAP/EBT benefits**.
Here are the options available for baby formula:
* **If the formula is unopened:** You may exchange it for the same or a similar item of equal value within 30 days with your receipt or loyalty account lookup.
* **If the formula has been opened:** For health and safety reasons, opened infant formula cannot be returned or exchanged unless it is subject to a manufacturer recall.
Please let us know if you need assistance with an exchange at your local store!
----------------------------------------------------------------------
Cache projects/12345678/locations/us/cachedContents/6602244618 deleted.
Explicit caching behaves the same every time. Compared with implicit caching: implicit got 1 hit in 6 calls. Explicit got 3 of 3 here, each covering all 4,663. All answers were correct and cited the right sections. The cache is stored on GCP side, in Agent Platform, not on your machine. It’s a CachedContent resource that belongs to your GCP project and sits in the location you created it in. Cache is set with ttl or expire_time, and it defaults to 1 hour. You can extend it with caches.update().
When it expires, Google deletes it automatically and calls that reference it fail.
Conclusion:
We built a FreshMarket support assistant and ran it both ways. Implicit caching was free but it is hit or miss. Explicit caching cached all 4,663 tokens every time. It also showed that caching only works when the static content comes first and is at least the model’s minimum size. This matters because the repeated context around each question, not the question itself, is where most of an LLM bill goes. Caching turns that repeated cost into a one-time cost, and makes responses faster too. For projects, start with implicit caching as the free default, and structure every prompt with static content first. When a large document is reused many times over hours (support shifts, batch jobs, RAG over knowledge bases), switch to explicit caching, set the TTL to the length of the job, and always delete the cache when you are done. Thats it from this blog of mine. Suggestions are welcomed.
Implicit vs Explicit Context Caching in Google Cloud Agent Platform: Explained with Real Numbers was originally published in Google Cloud – Community on Medium, where people are continuing the conversation by highlighting and responding to this story.
Source Credit: https://medium.com/google-cloud/implicit-vs-explicit-context-caching-in-google-cloud-agent-platform-explained-with-real-numbers-db35d67f9b18?source=rss—-e52cf94d98af—4
About the Author
