Make groundbreaking improvements in your application’s contextual search with AlloyDB AI! | by Abirami Sukumaran | Google Cloud – Community | Jun, 2025

With Reranking, AI Powered SQL Operators & Multimodal Embeddings…

If you’ve been following my work, you’d remember one of my recent projects that I shared earlier this year, a Toystore ecommerce application with AlloyDB, Gemini and Vertex AI, which was highly AI-centric but still involved a lot of application coding. But NOW!!!! Yesss, you guessed it! We’ve made it agentic but most importantly, we also want to help businesses thrive by giving users the experience that will strive to fetch the finest matches for their search contexts. This is the new Toystore Agent using …

💯 Reranking
🥇 Contextual Filter Operators
🗃️ Multimodal to Text Embedding compatibility
🖼️ Multimodal to Multimodal Embedding Vector Search
(Image to Text and Image to Image distance methods)

* using SQL queries only!!!

You were always looking for highly refined and precise responses for user searches. For example: when they say “black belt and no leather” your app should know not to pick matches with leather in the matching belts.

So we introduced, RE-RANKING, DIRECT TEXT-to-IMAGE distance methods for Vector Search and finally for the cherry on top, use the IF operator to make human level filtering on the matched vectors without having to invoke another complex application on top of our already sophisticated system.

The highlight is that there is no other database out there that can possibly do all of this in individual SQL statements. Yes you heard it right!!! Don’t need a stored proc, not even a function but just QUERIES!

I have implemented this as an AI agent that is capable of doing these sophisticated search tasks without changing anything in the source code of the web application itself.

Using AlloyDB, ADK and MCP Toolbox!

1. MCP Toolbox for Databases(AlloyDB in this case): I created a tools.yaml file with queries to carryout these advanced AI tasks and just put together a one — page single agent to execute that for me.

2. ADK for building the AI AGENT: I am using Agent Development Kit with its 3 step process for creating a single agent: MODEL, INSTRUCTION, TOOLS.

3. AlloyDB AI: I incorporated the search queries in the Tools.yaml file of MCP Toolbox to implement the precision improved search using reranking, text to multimodal comparison and AI filtering features.

AlloyDB has made it possible to implement such powerful advanced AI features with only SQL statements. But MCP Toolbox and ADK have made it so much easier to put this together as an actionable agent in not more than 10 lines of code.

Here is the detailed 3 part process of this agentic application:

PART 1: AlloyDB

I have detailed the steps to create cluster, instance, table, data, extensions and other settings in this codelab. For this implementation, a few updates are required:

alter table toys add column image_embeddings vector(1408);
alter table toys add column gcs_url VARCHAR(1000);
update toys set gcs_url = 
replace(image_url, 'https://storage.googleapis.com/img_public_test/', 'gs://img_public_test/') 
where gcs_url is null;
update toys set image_embeddings = ai.image_embedding(
model_id => 'multimodalembedding@001',
image => gcs_url,
mimetype => 'image/png')
where gcs_url is not null;

The other settings required for enabling these advanced AI features in AlloyDB can be referenced here:

Reranking
AI Powered SQL Operators
Multimodal Embeddings

We’ll look at the queries we have used, to cover these three in the Tools.yaml file in the next section.

PART 2: MCP TOOLBOX

I have detailed the steps to create and set up MCP Toolbox for AlloyDB and deploy it in Cloud Run in this codelab. Remember to use the latest Toolbox version and accordingly update the Clould Run deployment params based on the product documentation here.

1. Set up MCP toolbox for AlloyDB
2. Create Tools.yaml to contain the queries we need for this Toystore Agent:

...
tools:
search_toys_by_context:
kind: postgres-sql
source: alloydb
description: Get matching toys based on the search text.
parameters:
- name: search_text
type: string
description: Description of a toy that the user wants to find database matches for.
statement: |
select description, quantity, price, image_url, ref_number from (
SELECT name, description, quantity, price, image_url,
ROW_NUMBER() OVER () AS ref_number
FROM   toys 
ORDER BY image_embeddings <=> ai.text_embedding('multimodalembedding@001',$1)::vector 
limit 10) as x order by ref_number;
search_toys_by_image:
kind: postgres-sql
source: alloydb
description: Get matching toys based on the image that the user uploads.
parameters:
- name: image
type: string
description: Base64 string of the image of a toy that the user uploads in order to find matching toys.
statement: |
select description, quantity, price, image_url, ref_number from (
SELECT name, description, quantity, price, image_url,
ROW_NUMBER() OVER () AS ref_number
FROM   toys 
ORDER BY image_embeddings <=> ai.text_embedding('multimodalembedding@001',$1)::vector 
limit 10) as x order by ref_number;
search_toys_by_image_url:
kind: postgres-sql
source: alloydb
description: Get matching toys based on the image that the user uploads.
parameters:
- name: imageurl
type: string
description: Base64 string of the image of a toy that the user uploads in order to find matching toys.
statement: |
select description, quantity, price, image_url, ref_number from (
SELECT name, description, quantity, price, image_url,
ROW_NUMBER() OVER () AS ref_number
FROM   toys 
ORDER BY image_embeddings <=> ai.image_embedding('multimodalembedding@001',$1)::vector 
limit 10) as x order by ref_number;  
rerank_and_filter:
kind: postgres-sql
source: alloydb
description: Get more precise and deterministic matches for toys based on the user search text using reranking and filter operator methods.
parameters:
- name: search_text
type: string
description: Description of a toy that the user wants to find database matches for.
statement: |
WITH
initial_ranking AS (
SELECT *,
ROW_NUMBER() OVER () AS ref_number
FROM toys 
ORDER BY image_embeddings <=> ai.text_embedding(
model_id => 'multimodalembedding@001', content =>   $1 )::vector
LIMIT 10
),
reranked_results AS (
SELECT index, score from 
ai.rank(
model_id => 'semantic-ranker-default-003',
search_string => $1,
documents => (SELECT ARRAY_AGG(description ORDER BY ref_number) FROM initial_ranking)
-- ,top_n => 15
)
)
SELECT name,description,quantity,price, image_url,score
FROM initial_ranking, reranked_results 
WHERE initial_ranking.ref_number = reranked_results.index
and google_ml.if(
prompt => 'Here are descriptions of toys matching the user request, return the ones that are reasonably precise matches. Here is the user request: ' || $1 || ' Here are the matches from database.' ||description)
ORDER BY reranked_results.score DESC;
...

Take the rerank-and-filter query for example:

WITH
initial_ranking AS (
SELECT *,
ROW_NUMBER() OVER () AS ref_number
FROM toys 
ORDER BY image_embeddings <=> ai.text_embedding(
model_id => 'multimodalembedding@001', content =>   $1 )::vector
LIMIT 10
),
reranked_results AS (
SELECT index, score from 
ai.rank(
model_id => 'semantic-ranker-default-003',
search_string => $1,
documents => (SELECT ARRAY_AGG(description ORDER BY ref_number) FROM initial_ranking)
)
)
SELECT name,description,quantity,price, image_url,score
FROM initial_ranking, reranked_results 
WHERE initial_ranking.ref_number = reranked_results.index
and google_ml.if(
prompt => 'Here are descriptions of toys matching the user request, return the ones that are reasonably precise matches. Here is the user request: ' || $1 || ' Here are the matches from database.' ||description)
ORDER BY reranked_results.score DESC;

Goal: Perform a sophisticated AI-powered search for toys based on aadvanced multi-modal user search:

1. Initial Vector Search (multimodal): Uses ai.text_embeddingto find the 10 toys with image_embeddings closest to the text input, creating initial_ranking.
2. Row Numbering: Assigns a unique number to each initial result (ref_number) for joining later.
3. Reranking: ai.rank reranks the top 10 toys based on description and search_string, creating reranked_results.
4. Array Aggregation: The query ARRAY_AGG combines the descriptions into a single array of text documents.
5. Joining: Joins initial results with reranked scores based on the assigned row number and index to merge them.
6. Precise Match Filtering: Uses google_ml.if as an AI-powered filter to ensure results are precise matches to the text query.
7. Prompt Engineering: Engineering a prompt with the “google_ml.if” to help the AI be precise.
8. Final Selection: Selects relevant toy information (name, description, etc.) along with the reranked score.
9. Ordering: Orders final results by the reranked score for optimal relevance, with descending scores.

That’s it. We are able to drive multimodal and context filtered matches in just one query.

Now that you have set up the Tools.yaml & MCP Toolbox for this use case, make sure to deploy it in Cloud Run following the steps here.

PART 3: THE AGENT

I have used Agent Development Kit ADK Python sdk.

1. Enabled project ID, Vertex AI and Location in my .env file

GOOGLE_GENAI_USE_VERTEXAI=1
GOOGLE_CLOUD_PROJECT=abis-*****
GOOGLE_CLOUD_LOCATION=us-central1

2. Added dependencies in requirements.txt

google-adk
toolbox-core
deprecated

3. Created the 3-part Agent (with Model, Instruction and Tools):

root_agent = Agent(
name="toystore_agent",
model="gemini-2.0-flash",
description=(" Toystore agent that can assist with searching for matching toys and can perform 5 tasks: 1. Search by natural language text. 2. Search for Reranked and filtered results 3. Search by image 4. Search  by image url 5. Place order."
),
instruction=(
""" 
**********************************************************************************************************
**********************************************************************************************************
You are a helpful toystore agent that can assist with 5 main activities: 1. Search for matching toys by natural language text. 2. Fine tune search results with Re-ranked and Filtered matches  3. Search matching toys by image 4. Search matching toys by image url. 5. Place order"
When a user uploads an image, their primary intent is an image based search.
Follow these steps based on user input:
1.  **Search by natural language text:**
If the user provides search text (and no image is uploaded), pass this text as the "search_text" parameter to the "search_toys_by_context" tool.
Deliver the results in a list.
2.  **Search for finest results using Rerank and filter:**
If the user provides search text and explicitly asks for reranked or filtered results (and no image is uploaded), pass this text as the "search_text" parameter to the "rerank_and_filter" tool.
Deliver the results in a list. If no items are returned, inform the user and suggest other broader search options.
3.  **Search by image (png or jpeg):**
If a user uploads an image (e.g., a PNG or JPEG file):
a.  Convert that into text description precisely covering all the important details from the image and this MUST be passed as the "image" argument to the "search_toys_by_image" tool.
b.  Deliver the results from "search_toys_by_image" to the user in a list.
Inform the user that images should be in JPEG or PNG format, if the uploaded file is in any other format.
4.  **Search by image url:**
If a user enter a URL for an image (gsutil uri):
a.  The gsutil URI MUST be passed as the "imageurl" argument to the "search_toys_by_image_url" tool.
b.  Deliver the results from "search_toys_by_image_url" to the user in a list.
Inform the user that url should be of Google Cloud Storage gsutil URI.
5.  **Place order:**
If the user explicitly asks to place an order:
a. Make sure to get a confirmation of which item the user wants to place order for by using its description or price of imageurl as the inputs from the user.
b. Send a decorative "ITEM ADDED TO CART: <>" message to the user.
**General Instructions:**
*   Always respond in a friendly and helpful tone.
*   If a search returns no results, inform the user and suggest trying a different search query or image.
*   Do not make assumptions about user intent.
*   If the user asks for something outside of the agents capabilities, politely inform them that you cannot fulfill the request.
* Ask follow-up questions only when strictly necessary. Be concise.
**********************************************************************************************************
**********************************************************************************************************
""",
tools = tools

Test you agent with the adk web UI from outside the agent folder:

adk web

Deploy your agent on Cloud Run using adk deploy command:

adk deploy cloud_run \
--project=YOUR_PROJECT_ID \
--region=us-central1 \
--service_name=toystore-agent \
--app_name=toystore-agent-app \
--with_ui \
./toystore-agent-app

Try your agent for the new human-like AI-powered contextual searches with AlloyDB: