Pub/Sub AI Bytes: Part 7 — Three ways to stream real-time data into your Apache Iceberg Lakehouse

Welcome to Part 7 of Pub/Sub AI Bytes! Last week in Part 6, we explored how to eliminate stale context in generative AI applications by building real-time Streaming RAG pipelines with Cloud Pub/Sub AI Inference SMTs, Cloud Bigtable, and Cloud Dataflow.

This week, we shift our focus to one of the highest architectural priorities for Data and AI leaders: Lakehouse! In this blog, we share how you can stream real-time data directly into an open Apache Iceberg Lakehouse.

The Real-Time Lakehouse Dilemma

To maximize the value of data without creating silos, organizations are standardizing on the Open Lakehouse architecture powered by Apache Iceberg. By storing data once as open Apache Parquet files in customer-owned Cloud Storage (GCS) buckets with Iceberg metadata, multiple engines can work off the exact same tables without copying data:

• BigQuery can run sub-second SQL analytics and executive dashboards.
• Apache Spark can run large-scale feature engineering and data science pipelines.

So why is streaming into an Iceberg Lakehouse traditionally so difficult?

Data lakes were originally designed for hourly or nightly batch loads. When teams try to stream high-velocity events directly into Apache Iceberg, they quickly run into the “small file problem”: continuous micro-commits generate thousands of tiny Parquet files and rapid metadata snapshots that degrade query performance, trigger commit contention, and force engineering teams to build and maintain complex background compaction and garbage-collection jobs.

Cloud Pub/Sub removes this friction. Acting as the serverless, globally scalable ingestion backbone in front of your Lakehouse — where you can optionally attach Single Message Transforms (SMTs) to redact PII or normalize schemas in flight — Pub/Sub gives you three architectural paths to land real-time streams into Apache Iceberg based on your team’s workflow and preference:

1. Path 1 — The Zero-ETL Serverless Path (Pub/Sub BigQuery Subscriptions + BigQuery Managed Icerberg Tables): A 100% no-code pipeline that streams directly from Pub/Sub into Iceberg tables managed by BigQuery, with automated background compaction.
2. Path 2 — The Composable Stream Processing Path (Pub/Sub + Cloud Dataflow + Iceberg): A managed Apache Beam pipeline for teams that need stateful in-flight transformations, windowing, or dynamic multi-table routing.
3. Path 3 — The Spark-Native Lakehouse Path (Pub/Sub + Apache Spark Structured Streaming + Iceberg — Coming in Q1 2027): A native Spark connector designed for organizations standardized on Apache Spark for their Lakehouse ETL.

Path 1: The Zero-ETL Serverless Path (Pub/Sub BigQuery Subscriptions + Managed Iceberg Tables)

Suppose your AI platform team wants every live agent interaction and telemetry event published to an agent-telemetry Pub/Sub topic to land immediately in an open Apache Iceberg table in your own Cloud Storage bucket — without writing, deploying, or managing a single line of streaming pipeline code.

You can achieve this out of the box by combining Pub/Sub BigQuery Subscriptions with BigQuery tables for Apache Iceberg (also known as BigQuery Managed Iceberg Tables).

How the architecture works

1. Create a BigQuery Iceberg table on customer-owned storage: You define an Iceberg table in BigQuery linked to your Cloud Storage bucket (storage_uri = ‘gs://my-ai-lakehouse/agent_telemetry’) configured with file_format = ‘PARQUET’ and table_format = ‘ICEBERG’.
2. Attach a Pub/Sub BigQuery Subscription: You point a standard Pub/Sub BigQuery subscription directly at your Iceberg table ( -bigquery-table=my-project:lakehouse_dataset.agent_telemetry -use-table-schema). No additional changes are required — the subscription writes to a BigQuery Managed Iceberg Table seamlessly.
3. Zero-ops streaming & automated compaction: Under the hood, Pub/Sub writes incoming messages via the high-throughput BigQuery Storage Write API. BigQuery makes the streaming events immediately queryable while automatically converting rows into optimized Apache Parquet files in your GCS bucket, updating the Iceberg metadata manifest, and running serverless background compaction, clustering, and garbage collection.
4. Built-in failure isolation: If a malformed event fails schema validation, Pub/Sub automatically routes the undeliverable message (along with the CloudPubSubDeadLetterSourceDeliveryErrorMessage attribute) to your configured Dead-Letter Topic without stalling ingestion.

When to choose Path 1: Choose this path when you want the fastest, zero-code route from a live event stream to an open Iceberg table — letting BigQuery completely eliminate the “small file problem” and operational overhead on your behalf.

Path 2: The Composable Stream Processing Path (Pub/Sub + Cloud Dataflow + Lakehouse)

What if your raw telemetry stream requires stateful processing before landing in the lakehouse — such as joining prompt events with downstream user conversion events across a 10-minute session window, enriching records with external lookups, or dynamically routing a single multiplexed Pub/Sub topic into dozens of different Iceberg tables?

For these workloads, Cloud Dataflow provides a turnkey Managed I/O connector for Apache Iceberg that integrates directly with the Lakehouse REST Catalog.

How the architecture works

1. Ingest and transform in flight: A streaming Dataflow pipeline reads continuously from your Pub/Sub subscription and applies Apache Beam transforms (in Java, Python, or declarative Dataflow YAML) to window, aggregate, deduplicate, filter, or enrich the stream.
2. Stream directly to the Lakhouse REST Catalog: Using Beam’s Managed.write(Managed.ICEBERG) connector, Dataflow commits streaming data directly to your Iceberg warehouse on Cloud Storage through the standard Iceberg REST Catalog endpoint, with tunable commit intervals to balance data freshness and Parquet file size.
3. Dynamic Destinations & Dynamic Table Creation: Instead of maintaining a separate pipeline for every event type, Dataflow’s Managed Iceberg I/O can inspect fields inside each incoming Pub/Sub message — such as prompt_trace, tool_execution, or guardrail_alert — and dynamically route records to the appropriate Iceberg table (e.g., ai_lakehouse.events_${event_type}), even creating new Iceberg tables on the fly as new event categories appear.

When to choose Path 2: Choose Pub/Sub + Cloud Dataflow when you need in-flight stateful stream processing, dynamic multi-table routing and auto-table creation.

Path 3: The Spark-Native Lakehouse Path (Pub/Sub + Apache Spark Structured Streaming — Coming in Q1 2027)

Many enterprise data and ML teams standardize their lakehouse pipelines around Apache Spark on Google Cloud Managed Service for Apache Spark (across Serverless and Cluster runtimes). Historically, connecting Spark Structured Streaming to Pub/Sub required custom receiver code, unmaintained third-party libraries, or extra staging hops through Cloud Storage — because Spark expects partitioned logs with replayable offsets, whereas Pub/Sub distributes messages dynamically across workers using expiring leases (ackDeadline) and at-least-once delivery.

In Q1 2027, Google Cloud is introducing native Pub/Sub-to-Lakehouse streaming on Managed Service for Apache Spark, coordinating Pub/Sub lease management and Spark checkpoints directly in the runtime.

When to choose Path 3: Choose this path when your organization is standardized on Managed Service for Apache Spark and wants to drain Pub/Sub subscription backlogs directly into Apache Iceberg tables on schedule or on demand — with zero custom JARs, or staging hops.

Get started with your Real-Time Iceberg Lakehouse today

Whether your team prefers a zero-code BigQuery subscription, a composable Cloud Dataflow streaming pipeline, or Apache Spark Structured Streaming, Cloud Pub/Sub makes it effortless to turn live event streams into open Apache Iceberg tables:

• Zero-ETL Path: Check out the documentation for Pub/Sub BigQuery Subscriptions with Apache Iceberg managed tables and Creating BigQuery tables for Apache Iceberg.
• Composable Dataflow Path: Follow the step-by-step guide for Streaming Writes to Apache Iceberg with Lakehouse Catalog in Cloud Dataflow and explore Dataflow Managed I/O for Apache Iceberg.
• In-Flight Data Prep: Learn how Pub/Sub Single Message Transforms (SMTs) can clean, filter, and enrich your events before they land in your lakehouse.

Looking for previous blogs in the Pub/Sub AI Bytes series?

• Part 1: In-flight AI Inference with SMTs
• Part 2: Real-time model telemetry ingestion at scale
• Part 3: Gemini in Pub/Sub — Generate/Edit SMT code in seconds
• Part 4: High-Throughput ingestion for Large-scale Model Training
• Part 5: Zero-ETL streaming — Direct to Database with Bigtable subscriptions (GA)
• Part 6: Build a Real-time streaming RAG (Retrieval-Augmented Generation) Pipeline

Stay tuned for Part 8 of Pub/Sub AI Bytes next Friday! We will showcase how to simplify stream processing with SMTs, and when to use alternatives such as Dataflow andContinuous Queries for stream processing.

Pub/Sub AI Bytes: Part 7 — Three ways to stream real-time data into your Apache Iceberg Lakehouse was originally published in Google Cloud – Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

Source Credit: https://medium.com/google-cloud/pub-sub-ai-bytes-part-7-three-ways-to-stream-real-time-data-into-your-apache-iceberg-lakehouse-5d781c1fbca0?source=rss—-e52cf94d98af—4

About the Author

Deven Goratela

Administrator

Visit Website View All Posts