Transitioning from Batch Logging to Systems of Action
The transition from experimental LLM chats to production-grade autonomous agents represents a fundamental shift from “Systems of Record” to “Systems of Action.” In this new paradigm, traditional batch logging is an operational liability. Why? Because when an agent hallucinates, fails to trigger a tool, or enters an infinite loop, post-mortem analysis is insufficient.
Architecting for immediate intervention requires real-time telemetry that treats BigQuery not just as a passive data warehouse, but as a high-velocity engine for agentic triggers.
Most operational systems that rely on scheduled batch queries suffer from massive operational delays in scenarios like credit card fraud detection or supply chain bottlenecks. Human investigators piece together log entries and run ad-hoc SQL. Conversely, pushing raw data streams straight into LLMs is cost-prohibitive due to token burn. We need a hybrid model where high-speed, stateful SQL filters out the noise so AI agents only process targeted, enriched anomaly context.
The 3-Pillar Hybrid Architecture: Detect, Route, Resolve
To solve this, Google Cloud provides a three-pillar event-driven architecture-
- Detection (BigQuery Continuous Queries): Persistent SQL runs continuously over streaming ingestion tables (via the Storage Write API) to evaluate rules and windowed heuristics. Rather than polling, BigQuery uses stateful operations like tumbling windows ( TUMBLE ) and stream-to-stream JOINs to compute rolling metrics and detect complex conditions, such as a customer making transactions in two distant countries within minutes (‘Impossible Travel’). When an anomaly condition is met, EXPORT DATA immediately streams the structured alert payload to a Pub/Sub topic without leaving the data warehouse
- Routing (Pub/Sub & Single Message Transforms): Pub/Sub intercepts exported events, using lightweight JavaScript UDFs to reshape payloads into the exact schema expected by Vertex AI Agent Engine, and pushes it directly to the agent’s webhook endpoint.
- Resolution (ADK & Agent Engine): Autonomous agents hosted on Vertex AI Agent Engine receive the event, execute reasoning loops using custom tools, and either resolve the issue or escalate to human-in-the-loop workflows.
Real-World Scenario: Cymbal Bank Fraud Detection
Architecting for high-stakes environments such as fraud detection requires an event-driven loop where detection triggers autonomous action. In this pattern, BigQuery Continuous Queries monitor a stream of incoming data (e.g., retail transactions) for anomalies like “Impossible Travel.”
When an anomaly is detected, the query uses the EXPORT DATA statement to push the event to Pub/Sub in real-time. This message triggers an ADK Agent via Vertex AI Agent Engine, which can then use its tools to evaluate historical context and resolve the event or escalate it to a human.
Triggering an Agent via Continuous Stream
The following SQL demonstrates the Reverse ETL pattern used to trigger an agent from a continuous stream:
EXPORT DATA OPTIONS (
format = 'CLOUD_PUBSUB',
uri = 'https://pubsub.googleapis.com/projects/[PROJECT_ID]/topics/fraud_investigator_trigger'
) AS (
SELECT
customer_id,
transaction_id,
'Anomaly Detected: Impossible Travel' AS alert_reason
FROM
APPENDS(TABLE `project.dataset.transactions`)
WHERE
is_anomaly = TRUE
);
The Challenge: Black-Box of Production Agents
When you deploy such autonomous AI agents, traditional logging falls short because of the black box execution, unexpected token burn, silent hallucinations, and unmapped multi-agent decisions, etc.
Agents can make nondeterministic choices: they invoke nested tools, hand off control to sub-agents, pause for human approvals, and retry failed prompts. Without structured tracing, debugging a failed customer interaction feels like looking into a black box. Additionally, without token-level budgeting and latency tracking per tool, API costs and latency spikes can easily spiral out of control. We need an analytics engine that treats agent execution traces as queryable, structured data.
The BigQuery Agent Analytics plugin for the ADK bridges the observability gap. It provides high-throughput, low-latency telemetry essential for debugging complex multi-agent fleets. By leveraging a Reverse ETL pattern, it allows BigQuery to push insights directly into the agentic workflow, transforming raw logs into actionable intelligence. It logs your AI agents’ operational telemetry, lifecycle events, tool executions, and LLM interactions directly into BigQuery tables.
Before we jump into how BigQuery Agent Analytics gives data practitioners total visbility into agent behavior, lets briefly look at the set up.
The Set Up
>> To set up the BQ Agent Analytics plugin, you have to follow these config options and understand the schema and production setup.
>> The attributes.adk envelope — In the multi-agent workflows (agents that transfer control, checkpoint their state, and compact long histories) and long-running tools that pause and resume across turns, a small metadata envelope, attributes.adkis used to build observability by tying the rows back to the ADK event that produced them.
>> Event Lifecycle Coverage: The plugin automatically provisions a partitioned, clustered agent_events table that records 24 canonical event types. There are default event types and payload generated to trace LLM interactions. These events track the execution of tools by the agent and changes to agent’s state typically triggered by tools.
Unboxing the Black-Box of Agent Analytics
BigQuery Agent Analytics Unboxes Agent Behavior into 5 Observable Pillars:
- Muti-Agent Tracing & Execution Graphs: Every single action carries a persistent session_id and OpenTelemetry (trace_id, span_id). The plugin stamps an attributes.adk envelope on log rows to reconstruct multi-agent execution graphs (DAG), handle agent transfers, and track long-running pauses.
- Multimodal Analysis: Automatically offloads large files (such as images and audio) to Google Cloud Storage and exposes them to BigQuery ML via Object Tables.
- Tool Provenance & Human-in-the-Loop (HITL) Tracing: Tracks the exact origin of tools with explicit provenance tags (LOCAL, MCP, SUB_AGENT, A2A, TRANSFER_AGENT) and records Human-in-the-Loop (HITL) confirmation prompts, credential requests, and user inputs.
- Fleet Observability to Manage Costs and Performance- Monitoring an enterprise agent fleet requires granular tracking of token consumption and responsiveness. The plugin captures detailed metrics, including usage_prompt_tokens and usage_completion_tokens, allowing for precise cost-attribution strategies. Performance is measured via total_ms and ttft_ms (Time-to-First-Token)
- Business Decision Intelligence: The Context Graph feature (enabled via the BigQuery Agent Analytics SDK and the bqaa context-graph CLI tool) bridges the gap between raw, unstructured agent event logs and high-level business decision intelligence. The logged events can be materialized into a property graph, allowing operators to query decision trees using Graph Query Language (GQL)
Let’s look at each of these in further detail.
1. Multi-Agent Tracing & Execution Graphs
BigQuery Agent Analytics reconstructs agent sessions using distributed tracing and execution graphs. For the failed sessions, it runs Root Cause Analysis (RCA) using BigQuery AI functions.
- Granular Telemetry Streaming & Immutable Event Capture: As an agent executes, the plugin captures every operational event (such as AGENT_STARTING, LLM_REQUEST, TOOL_STARTING, and TOOL_ERROR) and streams them asynchronously into BigQuery tables via the high-throughput Storage Write API. This guarantees a complete, chronological record of every step the agent took.
- Distributed Tracing via trace_id, span_id, and the attributes.adk envelope: Every logged row is stamped with OpenTelemetry-compatible identifiers (trace_id, span_id) and an attributes.adk metadata envelope. parent_span_idis a 16-char HEX span id to reconstruct the hierarchical Directed Acyclic Graph (DAG). For multi-agent workflows, the plugin links agent transfers, state checkpoints (AGENT_STATE_CHECKPOINT), and long-running tool suspensions (TOOL_PAUSED) back to the exact conversation turn that triggered them.
- Automated Diagnostics via BigQuery AI Functions (AI.GENERATE): The plugin identifies the failed session and reconstructs the full conversation context. Then asks Gemini to diagnose the issue using AI.GENERATE. Instead of manual hunting, Supervisor agents automatically run AI.GENERATE over the aggregated history of a failed session to self-diagnose why an agent hit an API error or hallucinated.
DECLARE failed_session_id STRING;
-- Find a recent failed session
SET failed_session_id = (
SELECT session_id
FROM `your-gcp-project-id.your-dataset-id.agent_events`
WHERE error_message IS NOT NULL
ORDER BY timestamp DESC
LIMIT 1
);
-- Reconstruct the full conversation context
WITH SessionContext AS (
SELECT
session_id,
STRING_AGG(CONCAT(event_type, ': ', COALESCE(TO_JSON_STRING(content), '')), '\n' ORDER BY timestamp) as full_history
FROM `your-gcp-project-id.your-dataset-id.agent_events`
WHERE session_id = failed_session_id
GROUP BY session_id
)
-- Ask Gemini to diagnose the issue
SELECT
session_id,
AI.GENERATE(
('Analyze this conversation log and explain the root cause of the failure. Log: ', full_history),
endpoint => 'gemini-flash-latest'
).result AS root_cause_explanation
FROM SessionContext;
2. Multimodal Analysis
Multimodal GCS Offloading in the BigQuery Agent Analytics plugin solves a core infrastructure challenge: how to capture massive text prompts or rich binary media (such as customer-submitted identity documents, transaction receipts, or voice verification audio) without bloating your analytical database or hitting row-size limits.
- Generating Secure Signed URLs — Large text prompts (>500KB) or binary media (images, audio) are automatically offloaded to Google Cloud Storage. BigQuery stores an ObjectRef reference in content_parts, allowing operators to inspect images via BigQuery Object Tables or generate signed URLs with OBJ.GET_ACCESS_URL.If an engineer or fraud analyst needs to view a specific image or listen to an audio clip associated with a flagged session, BigQuery can dynamically generate time-bound, secure access URLs using the object reference.
- Linking Object Tables for Direct SQL Inspection– By connecting your GCS bucket to BigQuery as an Object Table, you can query multimodal assets like traditional relational data — inspecting file sizes, update times, and mime types alongside your agent execution logs.
- Multimodal Auditing with BigQuery Remote Models (Gemini) — Because the assets are securely referenced via GCS URIs, you can pass offloaded images or documents straight into BigQuery ML remote models (such as Gemini) to perform automated visual audits, OCR on receipts, or anomaly classification at scale
SELECT
ml_generate_text_result.candidates[0].content.parts[0].text AS ai_image_analysis
FROM
ML.GENERATE_TEXT(
MODEL `your-project.your_dataset.gemini_multimodal_model`,
(
SELECT
CONCAT('Analyze this transaction document for signs of tampering: ',
ML.SET_DOCK_IMAGE(JSON_VALUE(content_part, '$.object_ref.uri'))) AS prompt
FROM `your-project.your_dataset.agent_events`,
UNNEST(JSON_EXTRACT_ARRAY(payload, '$.content_parts')) AS content_part
WHERE trace_id = 'target_suspicious_trace_id'
)
);
3. Tool Provenance & HITL Tracing
BigQuery Agent Analytics plugin provides deep visibility into who executed a tool, where it came from, and when a human had to step in.
When an agent calls a tool, the plugin logs its execution payload and stamps its exact origin classification (origin). This prevents "black-box" confusion when multiple agent tiers or external servers interact. To query tool provenance in BigQuery, query v_tool_completed to audit which tool origins are consuming resources or failing most frequently.
SELECT
JSON_VALUE(payload, '$.tool_name') AS tool_name,
JSON_VALUE(payload, '$.origin') AS tool_origin,
COUNT(1) AS total_invocations,
AVG(CAST(JSON_VALUE(payload, '$.latency_ms') AS FLOAT64)) AS avg_latency_ms
FROM
`cymbal-fraud.analytics.v_tool_completed`
GROUP BY
tool_name,
tool_origin
ORDER BY
total_invocations DESC;
3.1 HITL Tracing
In high-stakes financial pipelines like a $2,500 cross-border fraud check, automation must occasionally yield to a human expert. When an agent encounters an ambiguous risk threshold, it pauses execution and triggers a Human-in-the-Loop event . The plugin records dedicated event types for these interactions:— HITL_CREDENTIAL_REQUEST, HITL_CONFIRMATION_REQUEST, HITL_INPUT_REQUEST, *_COMPLETED. Under ADK 2.0 multi-agent workflows, when an HITL request is emitted, the plugin logs a TOOL_PAUSED event and stamps an attributes.adk envelope on the row.
4. Monitoring Fleet Health in Real-Time
Fleet observability shifts the focus from debugging a single isolated agent run to monitoring an entire production ecosystem of enterprise AI agents. In high-volume environments like Cymbal Bank’s fraud prevention pipeline — which processes thousands of concurrent transaction alerts — platform operators need microscopic visibility into two critical pillars: cost attribution (token burn) and responsiveness (latency SLAs).
The BigQuery Agent Analytics plugin streams these vital telemetry markers directly into BigQuery tables, giving engineering and finance teams the granular data needed to optimize performance and control operational spend.
4.1 Granular Token Consumption & Cost-Attribution
LLM API costs scale directly with token volume. Rather than receiving a generic, aggregated monthly bill from your AI provider, the plugin captures detailed usage attributes on every model call:
- usage_prompt_tokens: Measures the exact size of the incoming context window (system instructions, user prompts, retrieved customer profile data, and event payloads).
- usage_completion_tokens: Measures the length of the model's generated response (reasoning steps, risk scoring rationale, and final decisions).
By storing these metrics natively in BigQuery alongside metadata like trace_id and agent names, users can run precise cost-attribution models such as breakdown token burn by specific departments, identifying which agents or instruction sets are bloated & allowing engineers to optimize system prompts, or set up automated BigQuery alerting thresholds when token consumption spikes.
4.2 Responsiveness & Latency Tracking (total_ms & TTFT)
In real-time fraud detection, seconds matter. If an agent takes too long to analyze a transaction, downstream systems time out or user experience degrades. The plugin tracks performance through two core latency metrics:
- total_ms (Total Execution Latency): Measures the complete end-to-end duration of an LLM request or a tool execution from start to finish.
- ttft_ms (Time-to-First-Token): Measures how quickly the model begins streaming its response back. This is crucial for real-time applications where perceived responsiveness depends on getting that first token instantly.
Users can query the v_llm_response view to track rolling averages, spot performance degradation across regions, and ensure SLAs are met.
SELECT
JSON_VALUE(payload, '$.model') AS model_name,
DATE(timestamp) AS execution_date,
COUNT(1) AS total_requests,
ROUND(AVG(CAST(JSON_VALUE(payload, '$.latency_ms') AS FLOAT64)), 2) AS avg_total_latency_ms,
ROUND(AVG(CAST(JSON_VALUE(payload, '$.ttft_ms') AS FLOAT64)), 2) AS avg_ttft_ms,
MAX(CAST(JSON_VALUE(payload, '$.latency_ms') AS FLOAT64)) AS max_total_latency_ms
FROM
`cymbal-fraud.analytics.v_llm_response`
GROUP BY
model_name,
execution_date
ORDER BY
execution_date DESC;
4.3 Combining Token Burn and Latency in Fleet Dashboards
Because all fleet metrics reside natively in BigQuery, teams can easily connect BI tools (like Looker or Looker Studio) to build comprehensive executive dashboards showing:
- Cost vs. Latency Trade-offs: Comparing cheaper, faster models (e.g., flash tiers) against heavier reasoning models to find the optimal balance for fraud classification.
- Spike Detection: Correlating sudden surges in usage_prompt_tokens with unexpected data bloat or recursive tool loops.
5. Business Decision Intelligence
Instead of treating agent traces as isolated lists of API requests, responses, and tool errors, the Context Graph transforms them into a queryable BigQuery property graph that maps out how decisions are made.
Using a scheduled refresh (bqaa context-graph) or event-driven pipeline, the SDK reads raw agent telemetry and distills session logs into structured decision components: DecisionRequest, DecisionOption, DecisionOutcome
The extracted components are mapped into a native BigQuery property graph schema using declarative table DDL and a CREATE PROPERTY GRAPH statement. Crucially, no separate external graph database or complex ETL pipeline is required—everything lives and runs natively within BigQuery.
Once materialized, you can use standard Graph Query Language (GQL) inside BigQuery via GRAPH_TABLE to traverse the decision network. For example, auditors or engineers can trace the exact chain of reasoning and confidence scores for a transaction review.
SELECT * FROM GRAPH_TABLE (
cymbal_fraud_graph
MATCH (req:DecisionRequest) -[eo:evaluatesOption]-> (opt:DecisionOption),
(req) -[ri:resultedIn]-> (out:DecisionOutcome)
COLUMNS (
req.request_id AS transaction_id,
req.request_text AS alert_summary,
opt.option_label AS risk_strategy_considered,
opt.confidence AS confidence_score,
out.status AS final_decision,
out.rationale AS audit_rationale
)
);
The Future of Autonomous Operations
The shift toward autonomous operations demands that we view telemetry not as an afterthought, but as a core agentic skill. The combination of the BigQuery Agent Analytics SDK and Looker dashboards turns raw operational traces into competitive business intelligence. By materializing execution graphs and utilizing AI-powered RCA, organizations can finally peer inside the “black box” of agentic AI, ensuring that every autonomous action is observable, accountable, and optimized for performance.
Hope this was a helpful read.
Thank you.
Engineering Agent Observability with BigQuery was originally published in Google Cloud – Community on Medium, where people are continuing the conversation by highlighting and responding to this story.
Source Credit: https://medium.com/google-cloud/mastering-agent-observability-with-bigquery-f6de99cd18ce?source=rss—-e52cf94d98af—4
About the Author
