Building a Real-Time Voice Co-Browsing Agent with Gemini Live API

Full code and deployment procedure is available here.

Section 1: Business Value & The Paradigm Shift in E-Commerce

Traditional e-commerce discovery relies on faceted navigation , complex arrays of dropdowns, multi-select checkboxes, price sliders, and sort menus. While powerful, this interface model introduces substantial cognitive friction:

• Filter Fatigue & Drop-off: Customers often know what they want in natural language (e.g., ”a lightweight phone with great battery life and 5G under $800”), but translating that intent into 4–6 separate UI clicks causes decision paralysis and cart abandonment.
• Detached Chatbots vs. True Co-Browsing: Most existing AI assistants are isolated conversational widgets. They provide text responses or static product links in a separate side panel, forcing the user to context-switch between the chat box and the actual product catalog.

1.1 Strategic Business Benefits

By integrating Gemini Live directly with the front-end catalog, businesses eliminate the language and technical barriers inherent in traditional e-commerce. Customers can speak naturally in their native language, regardless of regional dialects, complex phrasing, or localized terminology, and the agent seamlessly interprets their intent to drive the website interface. This native multilingual understanding and real-time execution deliver several high-impact business advantages:

1. Zero-Click, Intent-Driven Discovery: Customers speak naturally in any language. The agent maps conversational intent directly to existing catalog filters, sorting rules, and search inputs in real time.
2. Elevated Conversion Rates: Removing the friction between product intent and visual feedback shortens the sales funnel, directly boosting conversion rates and average order value (AOV).
3. Digital Sales Consultant Experience: Replicates the high-touch experience of an in-store sales associate who navigates the catalog with the shopper, spotlighting relevant products and explaining trade-offs.
4. Universal Accessibility: Voice-driven UI manipulation provides an inclusive interface for users with accessibility needs, motor impairments, or shoppers on mobile devices where complex filter menus are cumbersome.

https://medium.com/media/0a2720b19adf87de1d62f67b6eccc1b8/href

Section 2: High-Level Architectural Design

Modern voice assistants often operate as detached text or audio engines. This architecture closes the loop by turning the Gemini Live Multimodal API into an active, real-time co-browsing assistant capable of inspecting and mutating the browser DOM directly.

2.1 The Environment Boundary: Python Backend vs. Browser DOM

A fundamental architectural challenge of browser-controlling AI agents is runtime isolation:

• The Python Server Boundary: The Python backend (hosted on Google Cloud Run or a local FastAPI server) manages the secure bidirectional WebSocket stream with Gemini Live, handles API secrets, and configures tool declarations. However, Python has no direct access to the client’s browser memory or Document Object Model (DOM).
• The Client-Side Execution Boundary: The browser tab’s DOM can only be read or mutated by JavaScript/TypeScript running in the client context.

2.2 The WebSocket Remote Procedure Call (RPC) Bridge

To bridge this environment boundary, the Python server acts as an asynchronous RPC Proxy Bridge:

1. Session Handshake & Tool Declaration: When the user connects, Python initializes a bidirectional stream with Gemini Live, registering client-executable tool (get_screen_content,enter_form_data,highlight_elements )
2. Streaming Audio & Intent Detection: The browser captures real-time microphone PCM audio and streams it to the Python bridge, which forwards it to Gemini.
3. Tool Call Emission: When Gemini decides to interact with the UI, it emits a structured toolCall JSON payload back to the Python bridge.
4. RPC Dispatch to Browser: Python intercepts the toolCalland forwards the instruction down the active client WebSocket connection to the browser’s TypeScript controller.
5. DOM Mutation & Synthetic Events: The browser controller executes the DOM manipulation (e.g., checking boxes, moving sliders, scrolling elements) and triggers native synthetic events so reactive frameworks (React, Vue, or Vanilla JS) re-render immediately.
6. Tool Response & Voice Confirmation: The browser controller returns a toolResponsepayload back through the WebSocket to Python, which returns it to Gemini. Gemini receives confirmation that the UI updated and responds naturally via streaming audio.

Section 3: Deep Dive into Code and Tool Execution

3.1 End-to-End Operational Flow

To deliver a responsive co-browsing experience, the system operates as a continuous perception-action feedback loop. Before taking any action or answering the user, the agent inspects what is currently rendered on the screen, aligns it with the user’s spoken request, and then executes the appropriate UI manipulation commands.

1. Bootstrapping Page Context (get_screen_content): At session start or turn initialization, the agent inspects the active page state. The controller differentiates between two distinct application flows:

• Home Page (/index.html): Scrapes top trending deals and search wizard controls. Enforces a 3-Step Confirmation Protocol: the agent populates search form controls immediately →→ verbally summarizes the configuration to the user →→ and triggers the "Search Phones" button only after explicit customer confirmation.
• Dynamic Catalog (/search.html): Scrapes active filters, valid dynamic options (Available=[…]), sort order, and currently rendered product cards, executing instant real-time filtering without page reloads.

1. Translating Intent (enter_form_data): When the user speaks an intent, Gemini maps it against the available options discovered in step 1 and fires enter_form_data to mutate specific inputs.
2. Spotlighting Recommendations (highlight_elements): When recommending or comparing items, Gemini calls highlight_elements using unique product IDs to visually outline cards and scroll them into view.

3.2 Backend Initialization & Session Setup (proxy_server.py)

Before any real-time voice interaction or DOM co-browsing can occur, the Python backend (backend/proxy_server.py) establishes the session contract with the Gemini Live API. It performs two critical roles:

1. Tool Schema Registration: Declares the API schemas for all client-executable capabilities so the model knows what browser tools it can invoke.
2. Session Lifecycle Orchestration (GeminiLiveBridge): Manages the full-duplex bridge between the browser client's WebSocket and Google's Gemini Live bidirectional streaming endpoint.

Tool Declarations (PHONE_STORE_TOOLS)

In Gemini Live’s bidirectional streaming protocol, tools are registered during session setup using camelCase schemas under functionDeclarations. Four distinct tools are defined:

# backend/proxy_server.py
import json
import websockets
from fastapi import FastAPI, WebSocket

# 1. Complete Tool Declarations for Gemini Live
PHONE_STORE_TOOLS = [
    {
        "functionDeclarations": [
            activeBrands.join(", ") ,
            activeBrands.join(", ") ,
            {
                "name": "highlight_elements",
                "description": "Spotlight-highlight and select smartphone cards on the catalog grid to focus customer attention.",
                "parameters": {
                    "type": "OBJECT",
                    "properties": ,
                    "required": ["element_texts"]
                }
            }
        ]
    }
]

The Core Session Orchestrator: GeminiLiveBridge

To bridge the client’s browser and Google’s backend without leaking API credentials or maintaining persistent state in the browser, proxy_server.py encapsulates each live session in a GeminiLiveBridge instance.

Whenever a client opens the live agent, FastAPI accepts the connection and instantiates a dedicated bridge:

@app.websocket("/ws")
async def websocket_endpoint(websocket: WebSocket):
    await websocket.accept()
    api_key = websocket.query_params.get("key") or os.environ.get("GEMINI_API_KEY")
    model_name = "models/gemini-2.5-flash-native-audio-preview-12-2025"

    if not api_key:
        await websocket.close(code=4000, reason="Gemini API Key missing")
        return

    bridge = GeminiLiveBridge(client_ws=websocket, api_key=api_key, model_name=model_name)
    await bridge.run()

The GeminiLiveBridge orchestrates the connection across three operational stages:

A. Session Handshake & Protocol Initialization (run)

The bridge connects to Google’s WebSocket URL and immediately transmits the JSON setup payload. Notice that camelCase keys are strictly required by the Gemini Live API protocol:

class GeminiLiveBridge:
    def __init__(self, client_ws: WebSocket, api_key: str, model_name: str, user_facts: list = None):
        self.client_ws = client_ws
        self.api_key = api_key
        self.model_name = model_name if model_name.startswith("models/") else f"models/"
        self.gemini_ws = None
        self.running = False
        self.system_instruction = (
            "...."
        )

    async def run(self):
        self.running = True
        url = f"wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1alpha.GenerativeService.BidiGenerateContent?key={self.api_key}"
        ssl_context = ssl.create_default_context()

        async with websockets.connect(url, ssl=ssl_context, max_size=10_000_000) as g_ws:
            self.gemini_ws = g_ws

            # Setup packet using Gemini Bidi CamelCase schema
            setup_payload = {
                "setup": {
                    "model": self.model_name,
                    "generationConfig": {
                        "responseModalities": ["AUDIO"],
                        "speechConfig": {
                            "voiceConfig": {
                                "prebuiltVoiceConfig": {"voiceName": "Puck"}
                            }
                        }
                    },
                    "tools": PHONE_STORE_TOOLS,
                    "systemInstruction": {
                        "parts": [{"text": self.system_instruction}]
                    }
                }
              }
            await self.gemini_ws.send(json.dumps(setup_payload))
            init_resp = await asyncio.wait_for(self.gemini_ws.recv(), timeout=10.0)

            await self.client_ws.send_json({"type": "status", "status": "connected"})

            # Concurrently run inbound and outbound workers
            receive_task = asyncio.create_task(self._receive_from_gemini())
            send_task = asyncio.create_task(self._receive_from_client())

            done, pending = await asyncio.wait(
                [receive_task, send_task],
                return_when=asyncio.FIRST_COMPLETED
            )
            for task in pending:
                task.cancel()

B. Browser-to-Gemini Dispatch Loop (_receive_from_client)

Continuously listens on the client WebSocket and wraps messages into Gemini Bidi protocol frames:

• Microphone PCM Audio: Browser audio chunks (16kHz PCM) are forwarded as realtimeInput.mediaChunks.
• Tool Results: When the browser finishes executing a DOM action, it returns { "type": "tool_response", "data": […] }. The bridge packages this as toolResponse.functionResponses and sends it back to Gemini to unblock the model turn.
• Text Chat: Sent as clientContent.turns.

    async def _receive_from_client(self):
        while self.running:
            message = await self.client_ws.receive_text()
            data = json.loads(message)
            msg_type = data.get("type")

            if msg_type == "audio":
                # Forward base64 PCM chunk to Gemini
                await self.gemini_ws.send(json.dumps({
                    "realtimeInput": {
                        "mediaChunks": [{"mimeType": "audio/pcm", "data": data.get("data")}]
                    }
                }))

            elif msg_type == "tool_response":
                # Return DOM automation results to Gemini
                formatted_responses = []
                for fn_resp in data.get("data", []):
                    formatted_responses.append({
                        "id": fn_resp.get("id"),
                        "name": fn_resp.get("name"),
                        "response": {"result": fn_resp.get("response", {}).get("result", "")}
                    })
                await self.gemini_ws.send(json.dumps({
                    "toolResponse": {"functionResponses": formatted_responses}
                }))

C. Gemini-to-Browser Dispatch Loop (_receive_from_gemini)

Continuously listens for server frames emitted by Gemini:

• Tool Invocations (toolCall): When Gemini decides to interact with the web page, it emits a toolCall frame containing functionCalls. The bridge intercepts this and forwards { "type": "tool_call", "functionCalls": […] } over the client WebSocket for browser-side DOM execution.
• Voice Responses (serverContent.modelTurn): Base64 24kHz audio chunks and text transcripts are passed directly to the client widget for real-time playback.
• Interruption Handling (serverContent.interrupted): If the user interrupts the agent mid-speech (barge-in), the bridge immediately notifies the browser ({ "type": "interrupted" }) to halt local audio output.

    async def _receive_from_gemini(self):
        while self.running and self.gemini_ws:
            raw_msg = await self.gemini_ws.recv()
            msg = json.loads(raw_msg)

            # 1. Forward tool call requests to browser DOM controller
            if "toolCall" in msg:
                function_calls = msg["toolCall"].get("functionCalls", [])
                await self.client_ws.send_json({
                    "type": "tool_call",
                    "functionCalls": function_calls
                })

            # 2. Forward audio stream & transcriptions
            server_content = msg.get("serverContent")
            if server_content:
                if server_content.get("interrupted"):
                    await self.client_ws.send_json({"type": "interrupted"})

                for part in server_content.get("modelTurn", {}).get("parts", []):
                    if "inlineData" in part:
                        await self.client_ws.send_json({
                            "type": "audio",
                            "data": part["inlineData"]["data"]
                        })

With this configuration active, the agent relies on three core client-side functions to orchestrate the co-browsing experience. Let’s break down how each function operates at the implementation level.

Function 1: `get_screen_content` (Page State Introspection)

Purpose & Flow
Instead of streaming expensive, high-bandwidth screenshots or video feeds to the model, this function serializes the active DOM state (active filters, available options, and rendered catalog cards) into a compact, structured text payload.

Example Walkthrough: State Inspection & Introspection

User says: ”What 5G phones do you have right now, and what storage options are available?”

When it Triggers: Automatically invoked at the start of every session, after every user turn, or whenever the user asks about visible products, catalog specs, and available sidebar filters.

Agent Decision: The model determines that it cannot answer catalog or filter-specific questions without reading the real-time DOM. It issues an introspection call to extract active filters, sidebar choices, and visible product details.

Tool Call Arguments: No arguments required

/**
 * getScreenContent: Serializes the active page state into structured text for Gemini Live
 */
function getScreenContent(): string {
  const lines: string[] = [];
  lines.push(`=== PAGE CONTEXT ===`);
  lines.push(`- Current URL: ${window.location.href}`);

  // 1. Extract active and available filter categories
  const activeBrands = Array.from(
    document.querySelectorAll('.filter-input-brand:checked')
  ).map(cb => cb.value);
  
  const allBrands = Array.from(
    document.querySelectorAll('.filter-input-brand')
  ).map(cb => cb.value);
  
  lines.push(`- Brand Filter: Active=[${activeBrands.join(", ") || "None"}], Available=[${allBrands.join(", ")}]`);

  const activeStorages = Array.from(
    document.querySelectorAll('.filter-input-storage:checked')
  ).map(cb => cb.value);
  
  const allStorages = Array.from(
    document.querySelectorAll('.filter-input-storage')
  ).map(cb => cb.value);
  
  lines.push(`- Storage: Active=[${activeStorages.join(", ") || "None"}], Available=[${allStorages.join(", ")}]`);

  // 2. Extract visible phone cards from the catalog grid
  const phoneCards = Array.from(document.querySelectorAll(".phone-card"));
  lines.push(`\n=== CATALOG SMARTPHONES (${phoneCards.length} Loaded on Screen) ===`);

  phoneCards.forEach((card) => {
    const pid = card.getAttribute("data-id") || "";
    const title = card.querySelector(".phone-brand-title")?.textContent?.trim() || "";
    const price = card.querySelector(".phone-price")?.textContent?.trim() || "";
    const rating = card.querySelector(".rating")?.textContent?.trim() || "N/A";
    const specs = Array.from(card.querySelectorAll(".spec-pill"))
      .map(s => s.textContent?.trim())
      .join(" | ");

    lines.push(`Phone [ID: ${pid}] "${title}" | Price: ${price} | Rating: ${rating} | Specs: [${specs}]`);
  });

  return lines.join("\n");
}

How It Works:

1. Filter Serialization: Gathers both checked and unchecked input values from the sidebar, creating an explicit contract of what options exist (`Available=[…]`).
2. Catalog Scraping: Extracts IDs, pricing, ratings, and spec tags directly from product cards currently visible in the DOM.
3. Lightweight Context Injection: Sends the formatted string back to Gemini Live, giving the model precise awareness of the page state with minimal latency and token overhead.

Function 2: `enter_form_data` (UI State Mutation)

Purpose & Flow
Maps user requirements (e.g., ”Show me 5G Samsung phones under $800") to actual UI form controls. When Gemini emits multiple function calls in parallel or sequence, the client executes them, dispatches synthetic events, and triggers the store’s search engine.

Example Walkthrough: Multi-Filter Intent Mapping

User says: ”I’m looking for a Samsung phone with 5G support under $800.”

When it Triggers: Whenever the user states a search requirement, adjusts budget boundaries, requests a brand/feature filter, modifies sort orders, or prompts a form submission/reset.

Agent Decision: The model extracts three distinct search parameters from the natural-language input, cross-references them against known filter fields, and emits sequential (or batch) function calls to manipulate the UI controls.

Tool Call Arguments:

1. enter_form_data(field_name=”5G”, value=”true”)
2. enter_form_data(field_name=”Brand”, value=”Samsung”)
3. enter_form_data(field_name=”Max Price”, value=”800")

/**
 * enterFormData: Dispatches user intent directly to web page controls
 */
function enterFormData(fieldName: string, value: string): { success: boolean; message: string } {
  const field = fieldName.toLowerCase().trim();
  const val = value.toLowerCase().trim();

  // 1. Network Toggle (5G)
  if (field.includes("5g") || field.includes("network")) {
    const checkbox = document.querySelector('#home-5g-toggle, .filter-input-net[value="5G"]');
    if (checkbox) {
      checkbox.checked = val === "true" || val === "check" || val === "click";
      checkbox.dispatchEvent(new Event("change", { bubbles: true }));
      return { success: true, message: `5G filter set to ${checkbox.checked}` };
    }
  }

  // 2. Brand Checkbox Selection
  if (field.includes("brand")) {
    const brandCheckbox = document.querySelector(`.filter-input-brand[value="${value}" i], #home-brand-checkboxes input[value="${value}" i]`);
    if (brandCheckbox) {
      brandCheckbox.checked = true;
      brandCheckbox.dispatchEvent(new Event("change", { bubbles: true }));
      return { success: true, message: `Selected brand ${value}` };
    }
  }

  // 3. Numeric Price Bounds
  if (field.includes("price") || field.includes("budget")) {
    const priceInput = document.querySelector("#search-max-price, #home-max-price");
    if (priceInput) {
      priceInput.value = value.replace(/[^0-9.]/g, "");
      priceInput.dispatchEvent(new Event("input", { bubbles: true }));
      priceInput.dispatchEvent(new Event("change", { bubbles: true }));
      return { success: true, message: `Set maximum price to $${priceInput.value}` };
    }
  }
  // 4. Form Action Buttons
  if (field.includes("search") && val === "click") {
    const searchBtn = document.querySelector("#btn-home-search");
    if (searchBtn) {
      searchBtn.click();
      return { success: true, message: "Clicked Home Search button" };
    }
  }

How It Works:

1. Fuzzy Selector Resolution: Translates general field names (”Brand”, ”Max Price”, ”5G”) into concrete DOM elements.
2. Native Property Updates: Sets checked on input toggles or value on price inputs.
3. Dual Synthetic Event Dispatching: Dispatches both input and change events configured with { bubbles: true }. Programmatically setting .value or .checked on DOM elements does not automatically trigger JavaScript event listeners. By explicitly dispatching both:

• Text and range inputs (#search-keyword-input, #search-max-price) trigger on input.
• Checkboxes, toggles, and <select> dropdowns trigger on change. Setting { bubbles: true } ensures that delegated event listeners and reactive frameworks (React, Vue, or Vanilla JS) capture the mutation as it travels up the DOM tree.

Function 3: `highlight_elements` (Visual Spotlight & Focus)

Purpose & Flow
Directs the user’s visual attention to specific products that match complex criteria (e.g., ”Which of these has the best camera?”). The function applies animated spotlight styling and smoothly scrolls the target cards into view.

Example Walkthrough: Visual Spotlight & Comparison

User says: Which of these phones has the best camera and largest screen? Can you highlight them for me?”*

When it Triggers: Whenever the user asks comparative or superlative questions (”top rated”, ”largest display”, ”best camera”), or when the agent wants to visually spotlight specific recommendations.

Decision: Using the context obtained from `get_screen_content`, the agent evaluates the visible catalog items, identifies Phone ID ”12" (e.g., Galaxy S24 Ultra) and Phone ID ”34" (e.g., iPhone 15 Pro Max) as the top matches, and calls the highlight tool using their deterministic IDs.

Tool Call Arguments:
highlight_elements(element_texts=[“12”, “34”])

/**
 * highlightElements: Spotlights and selects target product cards on the screen
 */
function highlightElements(elementTexts) {
  // 1. Clear previous spotlight styling
  document
    .querySelectorAll(".live-agent-highlight")
    .forEach((el) => el.classList.remove("live-agent-highlight"));

  // Clear previous selections (triggering clear button if present)
  const clearBtn = document.getElementById("btn-clear-selection");
  if (clearBtn) {
    clearBtn.click();
  } else {
    document
      .querySelectorAll(".phone-card.selected, .trending-card.selected")
      .forEach((card) => {
        card.classList.remove("selected");
        card.removeAttribute("aria-selected");
      });
  }

  if (!elementTexts || elementTexts.length === 0) {
    return { success: true, highlightedCount: 0 };
  }

  let count = 0;
  // 2. Query both catalog cards and home trending cards
  const cards = Array.from(
    document.querySelectorAll(
      "#phones-results-grid .phone-card, #trending-phones-grid .trending-card, .phone-card, .trending-card"
    )
  );

  elementTexts.forEach((idText) => {
    const cleanId = (idText || "").toLowerCase().trim();
    if (!cleanId) return;

    // 3. Match by unique data-id OR data-phone-id
    const matched = cards.find((card) => {
      const pid = (
        card.getAttribute("data-id") ||
        card.getAttribute("data-phone-id") ||
        card.id ||
        ""
      ).toLowerCase().trim();
      return pid === cleanId;
    });

    if (matched) {
      // 4. Apply spotlight glow & trigger click to sync app selection state
      matched.classList.add("live-agent-highlight");
      if (!matched.classList.contains("selected")) {
        matched.click(); // Triggers search.js selection handler & counter badge
      }

      // 5. Smoothly scroll the primary match into view
      if (count === 0) {
        matched.scrollIntoView({ behavior: "smooth", block: "center" });
      }
      count++;
    }
  });

  // 6. Return structured tool response to Gemini
  return { success: true, highlightedCount: count };
}

How It Works:

1. State Reset: Clears prior highlight classes (.live-agent-highlight) to ensure only current recommendations are featured.
2. Deterministic ID Matching with Title Fallback: Primary matching strictly evaluates the device’s unique data-id or data-phone-id (e.g., matching "12"), eliminating ambiguity between similar model variants. If the LLM occasionally supplies a model name instead of an ID (e.g., ["Galaxy S24 Ultra"]), the controller automatically activates a secondary fallback that performs a case-insensitive substring match against the card's .phone-brand-title. This defensive design prevents tool-call failures caused by subtle model prompt deviations.
3. Viewport Alignment: Invokes scrollIntoView({behavior:”smooth”,block:”center”}) on the first matched product, centering the user’s screen on the recommended item.

Packaging and Exposing the Controller (window.PhoneAgentController)

In mock_phone_store/public/js/dom-controller.js, rather than exporting these functions via ES modules or a bundler, the entire script is wrapped in a self-executing closure (IIFE). It exposes the automation routines globally on window.PhoneAgentController:

  // mock_phone_store/public/js/dom-controller.js

  window.PhoneAgentController = {
    getScreenContent,
    enterFormData,
    highlightElements,
    injectHighlightStyles
  };
})();

Why This Design?

1. Decoupled Architecture (Zero-Build Controller vs. Bundled Widget):

• Zero-Build Engine (dom-controller.js): Written in clean, dependency-free vanilla JavaScript loaded via a simple <script src="/js/dom-controller.js"></script> tag. Storefront developers can modify selectors, add new filters, or tweak page interactions without maintaining a Node.js build pipeline or TypeScript bundler.
• Pre-Compiled React 19 Bundle (agent-widget.js): Houses the conversational UI drawer, Tailwind CSS styling, real-time audio visualizer waveform, settings manager, and WebSocket client. It is delivered as a pre-bundled standalone module (<script type="module" src="/agent-widget.js"></script>), completely separating visual widget lifecycle concerns from DOM automation logic.

2. Global Invocation Point: The embedded voice widget (agent-widget.js) can trigger DOM routines simply by calling window.PhoneAgentController.enterFormData(…).

3. Dual-Mode Execution (Direct DOM vs. postMessage RPC):

• Direct In-Page Mode: When embedded directly on the storefront, agent-widget.js calls window.PhoneAgentController methods synchronously in the same window context.
• Cross-Frame / Extension Mode: When isolated inside an iframe or a Chrome Extension side panel (window !== window.parent), the widget seamlessly switches to an asynchronous postMessage protocol. It dispatches messages (GET_SCREEN_CONTENT, ENTER_FORM_DATA, HIGHLIGHT_ELEMENT) to the parent window, where dom-controller.js executes the actions and posts back corresponding _RESPONSE events. This allows the exact same controller code to power both native storefronts and external third-party browser extensions.

Summary of the Complete Tool Suite

By combining Gemini Live’s low-latency streaming audio with a bidirectional WebSocket RPC bridge, this architecture transforms static web interfaces into voice-controlled, collaborative browsing experiences.

Section 4: Codebase Structure & Component Interactions

4.1 Codebase Structure & File Organization

The repository is organized into a modular full-stack architecture comprising two decoupled, independently deployable microservices: the FastAPI Python Backend Proxy and the Mock Phone Store Front-End Service (which hosts the PhoneVerse web application, catalog data, dynamic GCS image proxy, and the embedded client-side Voice AI Agent).

live-agent-web-interaction/
├── backend/
│   ├── proxy_server.py       # FastAPI WebSocket proxy & Gemini Live bridge
│   ├── Dockerfile            # Container definition for Google Cloud Run (Python 3.11-slim)
│   └── requirements.txt      # Python dependencies (fastapi, uvicorn, websockets, python-dotenv)
│
├── mock_phone_store/
│   ├── server.js             # Node.js/Express web server & Cloud Storage image proxy
│   ├── Dockerfile            # Container definition for Cloud Run (Node.js 20-alpine)
│   ├── package.json          # Node dependencies (express, cors, @google-cloud/storage)
│   ├── data/
│   │   ├── data.csv          # Complete smartphone catalog (specs, prices, ratings)
│   │   └── trends            # Featured trending phone IDs for home page
│   └── public/
│       ├── index.html        # Storefront homepage (Top 3 Trending Deals & search wizard)
│       ├── search.html       # Dynamic catalog search page (multi-faceted filter sidebar)
│       ├── agent-widget.js   # Embedded Voice AI Agent UI drawer & WebSocket client bundle
│       ├── agent-widget.css  # Styles for floating drawer, visualizer waveform & spotlight
│       ├── pcm-processor.js  # AudioWorkletProcessor forwarding low-latency audio chunks from the audio thread
│       ├── js/
│       │   ├── dom-controller.js  # Standalone DOM automation module (PhoneAgentController)
│       │   ├── search.js          # Reactive catalog filtering engine & URL query sync
│       │   ├── home.js            # Homepage search wizard & trending deals loader
│       │   └── data.js            # In-browser CSV parser & catalog query engine
│       ├── css/style.css     # Responsive e-commerce styling
│       └── assets/           # Store branding and placeholder imagery
│
└── deploy.sh                 # Cloud Run & Cloud Storage automated deployment script

4.2 Script Roles & Responsibilities

Layer A: Backend Cloud Services (backend/)

The Live Orchestrator & Proxy Bridge (proxy_server.py):

• Credential Management & Gateway: In production, the proxy serves as an API firewall holding GEMINI_API_KEY securely on Cloud Run to isolate credentials from client-side DevTools. For zero-setup prototype testing, it supports flexible hybrid key ingestion: it checks for incoming client WebSocket query parameters (?key=…), an environment variable on Cloud Run, or fallback retrieval from the storefront’s /api/config endpoint.
• Gemini Live Session Manager: Connects directly to the Google AI Studio Gemini Live WebSocket endpoint (wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1alpha.GenerativeService.BidiGenerateContent?key={api_key}) for bidirectional audio and tool streaming.

(Note: While deploy.sh binds roles/aiplatform.user to anticipate future Vertex AI Bidi streaming via OAuth2 service account tokens at us-central1-aiplatform.googleapis.com, the current reference proxy targets the AI Studio developer gateway for immediate API-key-driven prototyping.)

• Audio & JSON Multiplexer: Streams binary 16kHz 16-bit linear PCM microphone audio from the user up to Gemini Live in real time, and down-streams synthesized 24kHz PCM audio responses from Gemini back to the web client.
• Tool Call Router: Declares the four core agent tool definitions (get_screen_content, enter_form_data, highlight_elements), intercepts model-invoked toolCall messages, forwards them downstream to the web browser over the client WebSocket, awaits the client’s toolResponse, and feeds execution output back into Gemini Live.
• Dockerfile & requirements.txt: Package the Python FastAPI and Uvicorn runtime into a lightweight, autoscaling container image optimized for Google Cloud Run.

Layer B: Storefront & Client-Side Agent Controller (mock-phone_store/)

Storefront Server & Cloud Storage Proxy (server.js):

• Serves the PhoneVerse e-commerce web application, static assets, and client-side JavaScript controllers.
• Exposes backend health-check (.api/health ) and trending device endpoints (.api/trends ).
• Features an intelligent asset streamer that proxies high-resolution smartphone imagery on demand from a Google Cloud Storage bucket (gs://${GCS_BUCKET}/product_images/), with automatic fallback to local placeholders if the storage bucket is unreachable or unconfigured.

Embedded Voice AI Agent UI & Audio Coordinator (agent-widget.js & agent-widget.css):

• Renders the floating voice assistant drawer interface, including connection status badges, live waveform audio visualizer animations, user facts/profile drawer, and mic mute toggles.
• Establishes and manages the persistent WebSocket connection to proxy_server.py , including automated reconnection routines.
• Initializes the browser’s AudioContext , registers the pcm-processor.js AudioWorklet, and coordinates jitter-free PCM audio playback buffers for incoming voice responses.
• Acts as the client-side tool dispatcher: upon receiving a tool_call from the proxy server, it triggers the corresponding DOM controller routine (get_screen_content, enter_form_data, highlight_elements) and sends the structured tool_response payload back over the WebSocket.
• Supports dual-mode integration: it can manipulate the host page DOM directly when embedded as a native module, or communicate cross-frame via window.parent.postMessage when loaded inside an iframe or extension popup.

Real-Time AudioWorklet (pcm-processor.js):

• Operates on a dedicated Web Audio rendering thread completely off the browser’s main UI thread to prevent UI micro-stutters.
• Collects raw audio buffer slices (Float32Array) directly from the microphone input and forwards them instantly to agent-widget.js via this.port.postMessage().

Audio Ingestion & Serialization Pipeline (agent-widget.js):

• Upstream Ingestion (16 kHz): The browser’s AudioContext is instantiated with { sampleRate: 16000 } to enforce native hardware downsampling. agent-widget.js quantizes the 32-bit floats into signed 16-bit linear PCM integers (Int16Array), serializes the bytes to Base64, and streams them upstream as realtimeInput.mediaChunks.
• Downstream Playback (24 kHz): Gemini Live streams synthesized voice responses back as 24 kHz linear PCM chunks. agent-widget.js decodes them into audio buffers and schedules playback seamlessly across the AudioContext timeline with jitter buffer management.
• DOM Automation & Screen Introspection Engine (dom-controller.js):
• Exposes window.PhoneAgentControllerw containing the core browser automation routines: get_screen_content, enter_form_data, highlight_elements.
• Introspects and serializes active page state across both index.html (Home wizard, trending deals) and search.html (faceted filtering grid, sorting, URL query parameters).
• Programmatically interacts with filter elements (brand checkboxes, price range sliders, OS options, 5G toggles, storage pills, color swatches, screen size, camera MP, and sorting dropdowns) and dispatches synthetic input and change events to trigger reactive re-filtering.
• Injects dynamic spotlight styles (.live-agent-highlight) with “TECH EXPERT PICK” badges, automatically scrolls recommended device cards to viewport center, and triggers card clicks to open device detail modals.

Storefront Application Logic (home.js, search.js & data.js):

• home.js: Manages homepage UI, brand checkbox population, search wizard submissions, and trending deals loading.
• data.js: In-browser CSV catalog parser and data query engine for data.csv.
• search.js: Drives real-time multi-faceted catalog filtering, sorting, pagination, and bi-directional URL query parameter synchronization.

Layer C: Infrastructure & Deployment Automation (deploy.sh)

Automates the full production rollout on Google Cloud Platform via Cloud Shell.

• Enables required Google Cloud APIs (run.googleapis.com, artifactregistry.googleapis.com, cloudbuild.googleapis.com, storage.googleapis.com, and aiplatform.googleapis.com).
• Creates a private GCS bucket (gs://${PROJECT_ID}-mock-store-images) and uploads device catalog photos without making the bucket public; images are streamed securely and privately via the Node.js Express proxy using IAM credentials.
• Creates the proxy service account (live-phone-agent-proxy-sa) and binds roles/logging.logWriter (for diagnostics) and roles/aiplatform.user (for Vertex AI model invocation).
• Creates the storefront service account (mock-phone-store-sa) and binds roles/storage.objectViewer so the Node.js server can read product catalog images.
• Builds and deploys the backend proxy service to Google Cloud Run as live-phone-agent-proxy.
• Builds and deploys the mock_phone_store web service to Google Cloud Run as mock-phone-store, automatically injecting the GCS_BUCKET environment variable and configuring service endpoints.

5. Design Notes & Architectural Considerations

5.1 Architectural Trade-offs: Backend Proxy vs. Pure Client-Side JavaScript

• Client-Side Feasibility: It is technically possible to eliminate the Python backend (proxy_server.py) entirely by connecting browser JavaScript directly to the Gemini Live WebSocket. This removes server infrastructure, simplifies deployment to static hosting, and eliminates an intermediary network hop.

(Note: While the target architecture is designed to encapsulate credentials server-side, this open-source reference implementation allows the client widget to supply an API key via its settings drawer or retrieve it from an /api/config helper endpoint for friction-free evaluation without requiring pre-configured Cloud Run secrets.)

5.2 Prototype Limitations & Production Readiness

• Prototype Disclaimer: The current architecture implemented in this repository is an experimental proof-of-concept (PoC) designed to demonstrate real-time multimodal co-browsing and DOM automation. It is NOT production-ready in its current form.

To transition this system from a prototype to a resilient, enterprise-grade production environment, the following factors must be addressed:

• Authentication & Access Control: The prototype allows unrestricted, anonymous WebSocket connections. Production systems require user authentication (e.g., OAuth2, JWTs, Firebase Auth) to verify caller identity before opening an upstream Gemini Live session.
• Rate Limiting & Cost Governance: Audio streaming and high-frequency multimodal token usage can rapidly incur significant Cloud API costs. Production deployments must enforce strict per-user quotas, concurrency limits, and idle session timeouts (e.g., terminating audio streams after 60 seconds of silence).
• Fault Tolerance & Reconnection Handling: Real-time WebSockets are vulnerable to transient network drops. A production client needs robust exponential backoff, state hydration across reconnections, and graceful audio buffer recovery to prevent stutter or lost tool execution responses.
• DOM Sandboxing & Injection Defense: Allowing an LLM to mutate form fields and trigger clicks directly on a web page introduces potential security vectors. Production environments should enforce strict input sanitization, validate tool arguments against strict schema allowlists, and isolate DOM automation routines from sensitive user data (e.g., payment inputs, passwords, and PII).
• Distributed Scalability & Session State: The current proxy maintains in-memory WebSocket connections. At scale across multiple Cloud Run instances, production infrastructure requires a distributed messaging tier (e.g., Redis Pub/Sub) or sticky sessions to handle high concurrent user traffic.

Section 6: Conclusion & Call to Action

6.1 Summary of Key Takeaways

Voice-driven co-browsing represents a fundamental leap beyond traditional faceted navigation in modern e-commerce. By combining the sub-second, bidirectional voice streaming of Gemini Live API with an asynchronous WebSocket RPC bridge and a lightweight client-side DOM controller, this architecture bridges the gap between conversational AI and active browser interfaces:

1. True Multimodal Co-Browsing: The agent doesn’t merely answer questions in isolation — it actively co-navigates, applies multi-faceted filters, selects options, and spotlights qualifying products on the user’s screen in real time.
2. Clean Separation of Concerns: Isolating Python backend proxy orchestration (proxy_server.py) from browser DOM automation (dom-controller.js) ensures strong operational control without complicating frontend web stacks.
3. Friction-Free Extensibility: With a zero-build DOM controller and dual-mode dispatching (postMessage vs. direct invocation), developers can easily adapt this pattern to embedded web widgets, full-screen applications, or standalone Chrome Extensions.

6.2 Experience It in Action: Deploy to Google Cloud

The entire solution is open-source and ready for hands-on experimentation. The best way to evaluate the ultra-low latency and dynamic co-browsing capabilities is to deploy it directly to your own Google Cloud project:

1. Explore the Codebase:
   Review the complete implementation, inspect the prompt rules, and explore the RPC tool handlers on GitHub: https://github.com/AmirMK/live-agent-web-interaction
2. One-Command Cloud Run Rollout:
   Open Google Cloud Shell in your GCP project and run the automated deployment script:

git clone https://github.com/AmirMK/live-agent-web-interaction.git
cd live-agent-web-interaction
chmod +x deploy.sh
./deploy.sh

Within minutes, Cloud Shell will provision the Cloud Storage asset bucket, deploy both containerized Cloud Run microservices (mock-phone-store and live-phone-agent-proxy), and output your live storefront URL. Open the store in Chrome, tap the live specialist widget, and experience the future of real-time conversational shopping firsthand!

Building a Real-Time Voice Co-Browsing Agent with Gemini Live API was originally published in Google Cloud – Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

Source Credit: https://medium.com/google-cloud/building-a-real-time-voice-co-browsing-agent-with-gemini-live-api-8c7a1bdca908?source=rss—-e52cf94d98af—4

About the Author

Deven Goratela

Administrator

Visit Website View All Posts