How I built a dual-engine AI Support Assistant with Gemini Agent Search and Google Developer…

How I built a dual-engine AI Support Assistant with Gemini Agent Search and Google Developer Knowledge API

Build a production-ready support assistant that combines Google Developer Knowledge API with private runbooks in Vertex AI Search to generate policy-compliant troubleshooting guides.

Edited by:

Evelyn Camacho Soberon Technical Writer, Google Cloud

As a support engineer racing to meet SLOs, tab-switching is your worst enemy, especially when balancing official Google Cloud documentation with your company’s strict internal playbooks. Relying on generic AI to resolve production tickets often forces a difficult trade-off: getting an answer that is technically correct, but operationally wrong.

A dual-engine RAG approach solves this issue. Think of it like finding a five-star recipe online, but having a personal nutritionist instantly adapt it to fit your strict dietary restrictions. By querying public documentation and private runbooks in parallel, it synthesizes technical answers with internal constraints to deliver policy-compliant and actionable resolution guides.

Key Takeaways

• Dual-engine RAG architecture: Build an assistant that queries two distinct data sources simultaneously to balance generic technical advice with strict internal policies.
• Vertex AI Search private grounding: Securely index and retrieve private operational constraints and business rules directly from your company’s runbooks.
• Developer Knowledge API retrieval: Fetch grounded, official technical recommendations and troubleshooting steps directly from Google Cloud’s documentation.
• Gemini policy orchestration: Use Gemini to act as a reasoning engine that automatically filters public technical solutions against your private business rules to generate a policy-compliant resolution guide.

The danger of generic AI advice

Let’s understand a scenario where a support engineer receives the following ticket:

Client Acme is complaining about a 5-second cold start latency on their Cloud Run frontend service. How can we fix it?

A standard RAG assistant relying on public documentation will immediately recommend setting min-instances to 1 as the primary fix.

While this is technically correct, it creates massive operational problems:

• Budget violations: Acme operates under a strict budget cap that restricts their frontend service to a maximum of 1 instance.
• Process bypass: Their company runbook explicitly disallows modifying min-instances without written approval from their Lead Architect, Sarah Connor.

To handle this multi-pronged issue, we need a system that can retrieve the public documentation on how to optimize Cloud Run startup latency, and automatically filter those recommendations against Acme’s private constraints. Only after this analysis, should the system alert the engineer of any conflict and propose a compliant workaround.

Let’s get started with creating the support assistant!

Prerequisites

Before you begin, make sure you have:

1. A Google Cloud Project with billing enabled.
2. The gcloud CLI installed and authenticated on your local machine.
3. Python 3.10+ installed.

Billing & quota details

This tutorial is designed to run within the free tiers and developer quotas of each service:

• Google Cloud Storage: Storage of our single runbook file falls within the Cloud Storage Always Free tier. Billing only applies if you exceed free tier storage limits or incur egress charges.
• Gemini Agent Search (Vertex AI Search): Vertex AI Search offers a free trial tier for new accounts. For full details on query costs, see Vertex AI Search pricing. Regular queries are charged on a pay-as-you-go basis per 1,000 queries once trial credits are exhausted.
• Developer Knowledge API: Free to enable and use under standard developer limits. To monitor and manage request allocations, see Google Cloud Quotas & System Limits.
• Gemini 3.5 Flash API: Free to use within standard developer limits in Google AI Studio. For details on rate limits and pay-as-you-go options, see Google AI Studio Pricing. Paid billing only applies if you opt to upgrade to increase concurrency and rate limits.

Automated Google Cloud environment setup

Before building our support assistant, we need to prepare the underlying Google Cloud infrastructure. This automated setup script handles three key requirements:

• Enables Required APIs: Enables Cloud Storage, Vertex AI Agent Builder (discoveryengine), Developer Knowledge, and Generative Language APIs so our orchestrator can interact with all required endpoints.
• Provisions the Discovery Engine Service Identity: Pre-creates the Google-managed Discovery Engine Service Agent (service-PROJECT_NUMBER@gcp-sa-discoveryengine.iam.gserviceaccount.com). This ensures the background indexing service can be granted IAM permissions immediately without requiring an initial interactive console activation.
• Configures Storage & IAM Access: Creates the Cloud Storage bucket to host our customer runbook and grants the roles/storage.objectViewer role to both the background search agent for indexing and your active user account for console verification.

To configure your environment programmatically, copy the bash script below and paste it directly into your terminal window:

# 1. Initialize variables from active gcloud config
export PROJECT_ID=$(gcloud config get-value project)
export PROJECT_NUMBER=$(gcloud projects describe ${PROJECT_ID} --format="value(projectNumber)")
export BUCKET_NAME="${PROJECT_ID}-internal-kb"
export USER_EMAIL=$(gcloud config get-value account)

# 2. Enable all required GCP APIs
gcloud services enable \
  storage-api.googleapis.com \
  discoveryengine.googleapis.com \
  developerknowledge.googleapis.com \
  generativelanguage.googleapis.com \
  --project=${PROJECT_ID}

# 3. Provision the Discovery Engine Service Identity programmatically
# (This creates the background service account without requiring a manual Console click)
gcloud beta services identity create \
  --service=discoveryengine.googleapis.com \
  --project=${PROJECT_ID}

# 4. Create the GCS bucket
gcloud storage buckets create gs://${BUCKET_NAME} --project=${PROJECT_ID} --location=us-central1

# 5. Grant Storage Object Viewer to the Agent Search Service Agent (for background indexing)
gcloud storage buckets add-iam-policy-binding gs://${BUCKET_NAME} \
  --member="serviceAccount:@gcp-sa-discoveryengine.iam.gserviceaccount.com">service-${PROJECT_NUMBER}@gcp-sa-discoveryengine.iam.gserviceaccount.com" \
  --role="roles/storage.objectViewer"

# 6. Grant Storage Object Viewer to your active Console user (for frontend validation)
gcloud storage buckets add-iam-policy-binding gs://${BUCKET_NAME} \
  --member="user:${USER_EMAIL}" \

  --role="roles/storage.objectViewer"

echo "Setup complete! All APIs enabled and permissions configured successfully."

Replace the following:

• PROJECT_ID: your Google Cloud project ID. If you already configured your active project in the gcloud CLI, run the $(gcloud config get-value project) command to retrieve it automatically.
• PROJECT_NUMBER: your unique Google Cloud project number.
• BUCKET_NAME: the name of the Cloud Storage bucket to store your runbooks. By default, this is set to ${PROJECT_ID}-internal-kb.
• USER_EMAIL: the email address of your active Google Cloud account used to grant your account access to view the bucket in the Google Cloud console.

Set up the Private Knowledge Engine (Agent Search)

We will set up a private search engine containing an example customer runbook.

Create and upload the customer runbook

First, create the runbook file locally, and then upload it to your Cloud Storage bucket:

# Create the client runbook file with .txt extension
cat << 'EOF' > client-acme-runbook.txt
# Client Acme Corporation - Cloud Run Deployment Runbook
* **Client ID:** ACME-404
* **SLA Tier:** Platinum (4-hour resolution target)
* **Infrastructure Specifications:**
  * Frontend Service Name: `acme-frontend-prod`
  * Regional Deployment: `us-central1`
  * Max Instances: `1` (Hard cap imposed due to strict budget constraints)
  * CPU Allocation: `CPU allocated only during request processing`
  * Backend Database: Cloud SQL PostgreSQL (`acme-db-prod`)
  *   Networking: Serverless VPC (Virtual Private Cloud) Access connector (`acme-vpc-connector`)
* **Deployment Constraints:**
  * Deployed strictly via GitHub Actions workflow (`.github/workflows/deploy.yml`). Manual console updates are forbidden.
  * Direct access to the production DB requires using the Bastion Host `acme-bastion-prod`.
* **Escalation Path:**
  * Primary Contact: Sarah Connor (Lead Architect, sarah@acme-corp.com)
  * Secondary Contact: PagerDuty escalation policy `acme-prod-critical`
* **Common Issues & Custom Rules:**
  * **High Cold Starts:** High cold-start latency due to CPU throttling during scale-up. Do not modify the `min-instances` count without approval from Sarah Connor, as it breaches billing thresholds.
  * **Database Connection Timeouts:** If timeouts occur, verify that the Serverless VPC Access connector is in a `READY` state before attempting database restarts.
EOF

# Upload the file to your bucket
gcloud storage cp client-acme-runbook.txt gs://${BUCKET_NAME}/

Replace the following:

• BUCKET_NAME: the name of the Cloud Storage bucket that you created in the automated setup step (defaults to ${PROJECT_ID}-internal-kb).

Create an Agent Search Data Store

Configure the data store that will power your private knowledge search.

1. In the Google Cloud console, go to the Agent Builder (or Vertex AI Search) page.
2. In the navigation menu, click Data Stores, and then click Create data store.
3. On the Source page, select Cloud Storage.
4. In the Folder path field, enter gs://PROJECT_ID-internal-kb/*, replacing PROJECT_ID with your Google Cloud project ID.
5. Under Select the kind of data you are importing, select Unstructured documents (such as PDF, HTML, TXT), and click Continue.
6. In the Data store name field, enter acme-internal-kb.
7. Click Create to create your data store.
8. On the Data Stores list page, copy and save your Data Store ID.

Create a Search App (Serving Layer)

To query your data store programmatically, you must associate it with a Search App (Engine), which acts as the query serving layer.

1. In the navigation menu of the Agent Builder console, click Apps.
2. Click Create app.
3. On the Create app page, under Search, click Create.
4. In the App name field, enter acme-search-app.
5. In the External name of your company or organization field, enter Acme Corporation.
6. Under Location, keep the default global (Global), and click Continue.
7. On the Data Stores page, select the checkbox next to your data store acme-internal-kb.
8. Click Create.
9. Once created, click on your app name in the Apps list. Locate and copy your Engine ID (or App ID) from the App details page, such as acme-search-app_1234567890.

Set up the Google Developer Knowledge API

The Google Developer Knowledge API provides grounded access to official Google Cloud documentation and developer guides. In this section, you create and restrict an API key to authenticate requests, and confirm access by sending a sample curl query.

Create an API key

To call the Developer Knowledge API, you need an API key:

1. In the Google Cloud console, go to the Credentials page.
2. Click Create credentials > API key.
3. (Recommended) Restrict the key to only authorize the Developer Knowledge API.
4. Save the generated key as an environment variable:

export DEVELOPERKNOWLEDGE_API_KEY="DEVELOPERKNOWLEDGE_API_KEY

Replace DEVELOPERKNOWLEDGE_API_KEY with your generated API key.

Test the API with curl

Verify that you can query Google’s developer docs:

curl -X POST "https://developerknowledge.googleapis.com/v1:answerQuery?key=$DEVELOPERKNOWLEDGE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"query": "How do I reduce Cloud Run cold starts?"}'

You should receive a structured JSON response containing a grounded answer directly from Google’s official documentation.

Write the Python orchestration script

Now, we will write a script that queries both engines and uses Gemini to generate the final response.

Install required libraries

Install the required Google Cloud and AI libraries:

pip install google-cloud-discoveryengine google-generativeai requests

Create the orchestrator script

Run the following command in your terminal to create the Python application file app.py. The script dynamically passes the input query to both search engines, formats the prompt template, and sends it to Gemini:

cat << 'EOF' > app.py
import os
import sys
import requests
import google.generativeai as genai
from google.cloud import discoveryengine_v1beta as discoveryengine
# Configuration - Replace with your details
PROJECT_ID = os.environ.get("PROJECT_ID")
ENGINE_ID = os.environ.get("ENGINE_ID")
LOCATION = "global"
MODEL_NAME = os.environ.get("GEMINI_MODEL", "gemini-3.5-flash")
# Initialize Gemini Client (Uses GEMINI_API_KEY env variable)
genai.configure(api_key=os.environ.get("GEMINI_API_KEY"))
model = genai.GenerativeModel(MODEL_NAME)

# 1. Helper to query the Private KB (Agent Search)
def query_private_kb(query_text: str):
client = discoveryengine.SearchServiceClient()
serving_config = f"projects/{PROJECT_ID}/locations/{LOCATION}/collections/default_collection/engines/{ENGINE_ID}/servingConfigs/default_search"
request = discoveryengine.SearchRequest(
serving_config=serving_config,
query=query_text,
page_size=3,
)
response = client.search(request)
results = []
for result in response.results:
doc_data = result.document.derived_struct_data
if "snippets" in doc_data:
for s in doc_data["snippets"]:
results.append(s.get("snippet", ""))
return "\n".join(results)

# 2. Helper to query the Public KB (Developer Knowledge API)
def query_public_gcp_kb(query_text: str):
api_key = os.environ.get("DEVELOPERKNOWLEDGE_API_KEY")
url = f"https://developerknowledge.googleapis.com/v1:answerQuery?key={api_key}"
headers = {"Content-Type": "application/json"}
payload = {"query": query_text}
try:
response = requests.post(url, json=payload, headers=headers)
response.raise_for_status()
data = response.json()
return data.get("answer", {}).get("answerText", "No answer found.")
except Exception as e:
return f"Error querying Developer Knowledge API: {e}"

# 3. Main Orchestration
def run_support_assistant(ticket_query: str):
print(f"\n[Ticket Query]: {ticket_query}")
# Dynamically search private client specs using the ticket query
print("Searching private database…")
private_context = query_private_kb(ticket_query)
# Dynamically search public GCP advice using the ticket query
print("Searching public Google Cloud docs…")
public_context = query_public_gcp_kb(ticket_query)
# Format system prompt template
prompt = f"""
You are a Senior Cloud Support Assistant. Your task is to resolve the support ticket.
CRITICAL INSTRUCTIONS:
1. You MUST read the [Private Runbook Context] for specific client configurations, limits, and rules.
2. You MUST cross-reference all generic GCP recommendations against the client's private constraints. If a generic recommendation (e.g., increasing min-instances) violates a client limit or rule (e.g., budget caps or requiring specific manager approvals), you MUST call this out explicitly and provide the compliant fallback option.
3. Refer to specific client contacts (e.g., Sarah Connor), services, databases, and network connectors mentioned in the runbook when explaining the steps.
[Private Runbook Context]:
{private_context}
[Official GCP Guidance]:
{public_context}
[Support Ticket Query]:
{ticket_query}
Provide a precise troubleshooting guide for the support engineer.
"""
response = model.generate_content(prompt)
print("\n[AI Assistant Resolution Guide]:")
print(response.text)
if __name__ == "__main__":
if not all([PROJECT_ID, ENGINE_ID, os.environ.get("DEVELOPERKNOWLEDGE_API_KEY"), os.environ.get("GEMINI_API_KEY")]):
print("Please set your environment variables: PROJECT_ID, ENGINE_ID, DEVELOPERKNOWLEDGE_API_KEY, and GEMINI_API_KEY.")
else:
# Allow passing the ticket query as a command line argument
default_query = "Client Acme is complaining about 5-second cold start latency on their frontend service. How can we fix it?"
query = sys.argv[1] if len(sys.argv) > 1 else default_query
run_support_assistant(ticket_query=query)
EOF

Run the Test

Set your environment variables and run the script:

export PROJECT_ID="PROJECT_ID"
export ENGINE_ID="ENGINE_ID"
export DEVELOPERKNOWLEDGE_API_KEY="DEVELOPERKNOWLEDGE_API_KEY"
export GEMINI_API_KEY="GEMINI_API_KEY"
# Run with the default query
python3 app.py
# Or pass a custom query as a command line argument
python3 app.py "Client Acme is complaining about database connection timeouts on their Cloud SQL backend."

Replace the following:

• PROJECT_ID: your Google Cloud project ID.
• ENGINE_ID: the Engine/App ID of your Vertex AI Search app.
• DEVELOPERKNOWLEDGE_API_KEY: the API key you created for the Developer Knowledge API.
• GEMINI_API_KEY: your Gemini API key from Google AI Studio. To obtain a Gemini API key, see Get a Gemini API key.

Expected output

Your output should look similar to this:

[AI Assistant Resolution Guide]:
### **Acme Support Ticket: Resolution & Troubleshooting Guide**
**Ticket ID:** #ACM-98431
**Service:** `acme-frontend` (Cloud Run)
**Issue:** 5-second cold start latency
**Assigned To:** Cloud Support Engineering Team
 - -
### **CRITICAL COMPLIANCE WARNING (Read Before Acting)**
According to Acme's operational rules:
* **The Conflict:** Standard GCP documentation suggests setting `min_instances` (e.g., to 1 or more) to eliminate cold starts. However, **Acme operates under a strict budget cap policy that prohibits idle/always-on instances** for the frontend service without explicit business justification.
* **The Rule:** You **MUST NOT** increase `min_instances` on `acme-frontend` without prior written approval from **Sarah Connor** (Acme's Technical Lead & Budget Approver).
* **Compliant Fallback:** If Sarah Connor does not approve the budget increase for `min_instances`, you must resolve the latency strictly through **code-level optimizations, CPU adjustments, startup probes, and connection tuning** as detailed below.
 - -
### **Step-by-Step Troubleshooting & Resolution Guide**
#### **Phase 1: Triage and Governance Check**
1. **Contact Sarah Connor:** Reach out to Sarah Connor to request a budget exception for setting `min_instances = 1`.
2. **Fallback implementation:** If denied, proceed strictly with the budget-compliant fallback steps below to optimize the cold start duration within the 0-instance scaling limit.
#### **Phase 2: Network & Connection Optimization**
1. **Inspect VPC Connector Latency:** `acme-frontend` routes traffic to the database via the **`acme-vpc-connector`** (Serverless VPC Access). Ensure that the connector throughput is not bottlenecked.
2. **Database Connection Lazy-Loading:** The frontend connects to **`acme-production-db`** (Cloud SQL). Do not open the connection pool until the first actual handler request arrives to minimize container boot time.
…

The result: AI that understands your business rules”

Notice how the AI successfully blended the two data sources:

1. GCP Public Documentation (DKP API): Provides technical recommendations. For example, enabling cpu-boost, setting concurrency to 80, or configuring a /healthz startup probe.
2. Private Knowledge Base (Agent Search): Injected critical business logic, security constraints, and operational context:

• It flagged that standard GCP advice (min_instances = 1) violates Acme’s strict budget rules.
• It correctly identified Sarah Connor as the authority who must approve any deviations.
• It referenced their actual backend database (acme-production-db) and VPC connector (acme-vpc-connector).

Without this dual-engine approach, a generic LLM would simply output standard documentation advice to increase min_instances, which a junior support engineer might apply immediately. In an enterprise environment, this would violate budget limits, trigger billing alerts, and bypass architectural approvals. Grounding the AI in both public and private contexts ensures generated solutions are not just technically sound, but organizational-policy-compliant.

Best practices for production deployment

To prepare your AI support assistant for production, consider the following best practices:

• Secure secrets management: Store API keys in Secret Manager and load them dynamically at runtime rather than storing them in plain-text environment variables.
• Apply least-privilege IAM: Run production workloads using a dedicated service account granted only roles/discoveryengine.viewer and roles/storage.objectViewer. Restrict API keys to specific APIs and authorized network IP addresses.
• Automate document ingestion: Connect external sources such as Jira, Confluence, SharePoint, or Google Drive directly to your data store by using Vertex AI Search connectors instead of manual file uploads.
• Use structured metadata filters: Include client IDs and system tags in your SearchRequest filter parameters to narrow search results to specific customer environments.
• Cache frequent queries: Cache common documentation responses in Memorystore for Redis to reduce latency and API quota usage.

Clean up resources

To avoid incurring ongoing charges to your Google Cloud account for the resources used in this tutorial, clean up the deployed components.

Delete the search app and data store

Note: You cannot delete a data store while it is connected to an app. You must delete the search app first.

To delete the app and data store in the Google Cloud console:

1. In the Google Cloud console, go to the Agent Builder > Apps page.
2. In the apps list, locate acme-search-app, click More actions (three vertical dots), and then click Delete.
3. In the confirmation dialog, enter the app name, and then click Confirm.
4. In the navigation menu, click Data Stores.
5. Locate acme-internal-kb, click More actions (three vertical dots), and then click Delete.
6. In the confirmation dialog, click Delete.

Delete the Cloud Storage bucket

To delete the Cloud Storage bucket and all ingested runbook files, run the following commands:

export BUCKET_NAME="${PROJECT_ID}-internal-kb"
gcloud storage rm - recursive gs://${BUCKET_NAME}/

Delete the API keys

1. In the Google Cloud console, go to the APIs & Services > Credentials page.
2. Under API Keys, locate the key created for the Developer Knowledge API.
3. Click Delete (trash icon) next to the key, and confirm deletion.

Remove local project files

To remove the local scripts and configuration files, run:

rm -f app.py app_test.py client-acme-runbook.txt

Related Resources

• Gemini Enterprise Connectors Guide
• Google Cloud Secret Manager
• Google Cloud Memorystore for Redis

How I built a dual-engine AI Support Assistant with Gemini Agent Search and Google Developer… was originally published in Google Cloud – Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

Source Credit: https://medium.com/google-cloud/how-i-built-a-dual-engine-ai-support-assistant-with-gemini-agent-search-and-google-developer-e29d7c6e78a2?source=rss—-e52cf94d98af—4

About the Author

Deven Goratela

Administrator

Visit Website View All Posts