Pub/Sub AI Bytes: Part2 — Real-time model telemetry ingestion at scale

Pub/Sub AI Bytes — Part2: Real-time model telemetry ingestion at scale

Damien Contreras, Prateek Duble

Welcome to Part 2 of Pub/Sub AI Bytes! In our first post, we explored how Pub/Sub AI Inference Single Message Transforms (SMTs) bring zero-ETL AI intelligence directly into streaming pipelines.

This week, we shift our focus to an emerging AI Architecture Pattern: how frontier model builders and AI-native enterprises ingest, sanitize, and analyze massive volumes of real-time model telemetry at scale. This pattern is also applicable to generic real-time telemetry ingestion.

The challenge: Telemetry at scale

AI models and agentic applications generate exceptionally rich, high-velocity telemetry. Beyond basic request/response logs, modern AI systems emit granular events across every step of an interaction:

• Session lifecycles: Initialization, user interrupts, context-window compressions, and multi-turn durations.
• Token & performance metrics: Prompt token counts, cached token reads, completion tokens, time-to-first-token (TTFT).
• Agent tool execution: Tool call definitions, execution durations, retry loops, and code execution outputs.
• Output quality & safety signals: Guardrail classifications, user feedback ratings (thumbs up/down), exception classifications.

When you have dozens of engineering teams rapidly shipping features across developer CLIs, desktop applications, web interfaces, and backend microservices, building, maintaining and deploying telemetry ingestion becomes difficult at scale. Many times every team has to file infrastructure tickets, configure databases, or write custom stream-processing pipelines to ingest new telemetry data to be used downstream for analytics, model training or other high ROI use cases.

To keep pace with the velocity of AI development, users need a simple, yet highly scalable self-service telemetry platform where any engineer can define an event contract and have enterprise-grade ingestion infrastructure provisioned automatically.

Here is how you can solve this challenge using Cloud Pub/Sub and BigQuery Subscriptions.

Proto as the single source of truth

In this architecture, infrastructure is never provisioned by hand. Instead, a central Protocol Buffer repository acts as the single source of truth:

1. Self-service event definition: An engineer creates or updates a .proto file representing their telemetry event and decorates it with a custom metadata annotation.
2. Automated CI/CD provisioning: On Git merge, the CI pipeline automatically provisions the required Google Cloud infrastructure:

• A dedicated Pub/Sub Topic (e.g. {env}-event-{name})
• A native BigQuery Subscription configured for telemetry ingestion
• A Dead Letter Queue (DLQ) topic for unroutable or malformed payloads

3. Publishers: The CI pipeline simultaneously compiles and publishes typed client libraries to the internal schema repo. Your GKE publisher code matches the message to the correct topic and BigQuery table according to its schema.

// Protobuf message definition with automated provisioning metadata options.
syntax = "proto3";

package telemetry.ai.v1;

import "google/protobuf/timestamp.proto";

// Custom option to drive automated infrastructure provisioning
option (event_metadata) = {
  project_id: "proj-telemetry-prod"
  dataset_id: "telemetry_events"
};

message ModelEvent {
  string event_id = 1;
  string session_id = 2;
  string model_name = 3;            
  google.protobuf.Timestamp timestamp = 4;
  
  int32 prompt_tokens = 5;
  int32 completion_tokens = 6;
  int64 latency_ms = 7;
  
  string client_type = 8;             // e.g., "cli", "browser", "desktop-ide"
  repeated string tool_names = 9;     // e.g., ["file_editor", "bash", "web_search"]
  bool has_error = 10;
  string error_category = 11;
}

Engineers write zero Terraform and file zero infrastructure tickets. They define a protobuf message with one annotation, and the pipeline sets up end-to-end ingestion in minutes for them to ingest telemetry data.

Telemetry Ingestion at Scale with Pub/Sub

At runtime, telemetry flows through a completely serverless, zero-ETL pipeline:

How the components interact:

1. Publishers: When a client application emits an event, it imports the compiled protobuf bindings directly. These events are sent to a Google Kubernetes Engine (GKE) cluster.
2. Pub/Sub topic ingestion: GKE is responsible for publishing events into the dedicated Pub/Sub topic over gRPC or HTTP.
3. Single Message Transform (SMT): In real-world environments, client versions drift. A legacy CLI tool might omit newly required fields, or a desktop client may send timestamps in a non-standard format. An SMT executes lightweight, in-flight JavaScript normalization — fixing malformed keys, stripping accidental metadata, or populating fallback defaults before messages reach storage.
4. Direct BigQuery subscriptions (Zero-ETL): Rather than routing messages through intermediate custom services, Pub/Sub writes directly to BigQuery using BigQuery subscriptions. Pub/Sub maps protobuf fields directly to BigQuery table columns, handling automatic table partitioning (e.g., partitioned by event timestamp) with zero operational toil.
5. Dead-letter routing: Any message that fails schema validation or cannot be written to BigQuery is routed automatically to a Dead Letter (DLQ) topic, preventing unparseable payloads from backing up the subscription.

Simplified Architecture for Telemetry Ingestion

Before native BigQuery subscriptions and SMTs, ingesting streaming telemetry required a custom streaming pipeline:

Key takeaways

If you are designing telemetry pipelines for AI systems or any other event-generating system, consider these three principles:

1. Contracts over infrastructure: By embedding infrastructure metadata inside .proto files, your CI/CD pipeline can provision topics and tables without manual intervention.
2. Simplified transforms at scale: Use Pub/Sub SMTs to catch and repair message payloads before telemetry data hits your analytical warehouse.
3. Embrace Zero-ETL streaming: For 1:1 ingestion into BigQuery, skip the overhead of dedicated engines or custom services. Pub/Sub BigQuery subscriptions deliver lower latency with zero maintenance.

Try it yourself

Ready to modernize your streaming telemetry?

• Learn how to set up Pub/Sub BigQuery subscriptions for direct, zero-ETL ingestion.
• Check out Single Message Transforms (SMTs) for in-flight payload normalization.

Tune in next Friday for Part 3 of Pub/Sub AI Bytes, where we explore Gemini in Pub/Sub: Generating and editing SMT code in seconds!

Pub/Sub AI Bytes: Part2 — Real-time model telemetry ingestion at scale was originally published in Google Cloud – Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

Source Credit: https://medium.com/google-cloud/pub-sub-ai-bytes-part2-real-time-model-telemetry-ingestion-at-scale-63b702206094?source=rss—-e52cf94d98af—4

About the Author

Deven Goratela

Administrator

Visit Website View All Posts