A battle-tested decision framework with concrete cost/latency trade-offs for Chief Architects and Lead Data Engineers building on Google Cloud Iceberg Lakehouse, BigLake, and BigQuery Omni.
Architecture GuideGoogle Cloud ReferenceReading Time: 14 minTarget: BigQuery Omni • Apache Iceberg • BigLake • GCS
Executive Summary
The Apache Iceberg table format has fundamentally rewritten enterprise data architecture. By decoupling physical columnar storage formats from proprietary query engines, Iceberg turned the multi-cloud lakehouse from marketing fiction into standard engineering reality.
Yet for Chief Architects and Data Platform Leads, Iceberg introduces a high-stakes fork in the road:
When your petabytes of analytical data live in AWS S3 or Azure ADLS, but your enterprise analytics, AI/ML, and business intelligence ecosystems center on Google Cloud (BigQuery, Vertex AI, Looker) — should you physically migrate your Iceberg tables into Google Cloud Storage (GCS), or federate queries in-place using BigQuery Omni?
Make the wrong architectural bet, and you face either runaway multi-cloud egress bills and multi-quarter migration delays, or sluggish query SLAs and brittle cross-cloud shuffle bottlenecks.
This article delivers a battle-tested decision framework grounded in official Google Cloud reference architectures. We dissect the precise mechanics of both paradigms, establish concrete mathematical cost and latency formulas, walk through production step-by-step implementations, examine five critical engineering traps, provide an executive FAQ, and compile official Google Cloud documentation citations.
1. Architectural Paradigms: How They Actually Work Under the Hood
Before evaluating trade-offs, we must understand the physical and control plane mechanics of both architectures across the Google Cloud ecosystem.
Strategy A: Zero-Copy In-Place Federation (BigQuery Omni)
BigQuery Omni is Google Cloud’s distributed analytics solution that brings BigQuery’s execution engine directly into foreign clouds (AWS S3 and Azure Blob/ADLS Gen2).
- Compute Colocation: Google deploys and manages BigQuery compute clusters running inside Google-managed VPCs physically located within the target provider’s regions (e.g., AWS us-east-1 or Azure eastus2).
- Identity & Metadata Federation: BigQuery connects to the foreign catalog (AWS Glue Data Catalog, an Iceberg REST catalog like Apache Polaris, or direct Iceberg metadata.json pointers on S3) through a secure BigLake Connection backed by cross-cloud IAM role federation (AWS IAM sts:AssumeRoleWithWebIdentity using Google's OIDC issuer).
- Intra-Cloud Query Execution: When a user in GCP submits a query targeting an Omni table, the BigQuery control plane in GCP decomposes the execution DAG:
- File scans, partition pruning, column projection, and localized aggregations (GROUP BY, SUM, COUNT) are pushed down to the Omni cluster running inside AWS/Azure.
- Remote worker nodes read S3 objects over local AWS availability zone networks without touching the public internet.
- Intermediate & Result Egress:
- For isolated queries, only the finalized, filtered result rows stream across the WAN back to the GCP control plane or write out to an S3/GCS destination.
- For cross-cloud queries (JOIN between an AWS Omni table and a GCP-native table), BigQuery Omni executes the remote subplan in AWS, materializes an intermediate dataset, and transfers it over Google's global private network to the GCP host region to execute the final join.
Strategy B: Physical Migration to Native GCS Iceberg Lakehouse
Physical migration relocates raw Parquet objects and real-time ingestion pipelines to Google Cloud Storage (GCS), registering tables with BigLake Metastore or an Iceberg REST catalog.
- Bulk Ingestion via Storage Transfer Service (STS): GCP STS streams petabyte-scale Parquet files from S3/ADLS to GCS over the Google high-speed global backbone or dedicated Cross-Cloud Interconnect (CCI).
- Metadata Recreation vs. In-Place Path Rewriting:
- Option B1 (Fast Metadata Migration): The Parquet data files are transferred to GCS, and an Apache Spark system.migrate or metadata re-pointing script rewrites Iceberg manifest lists to reference gs:// paths.
- Option B2 (Managed Iceberg Tables): Ingestion pipelines (Kafka, Flink, Spark) are flipped to write natively into GCS as BigQuery Managed Iceberg tables or BigLake tables.
- Local BigLake Integration: BigLake acts as the unified storage engine, unlocking BigQuery native slots, BI Engine in-memory acceleration, and automated metadata caching (metadata_cache_mode = 'AUTOMATIC').
2. The Decision Framework: Cost and Latency Trade-Offs
Choosing between Migration and Federation is not a philosophical debate; it is an optimization problem governed by Network Egress, Storage Duplication, Query Frequency, Data Scan Cardinality, and P99 Latency SLAs.
The Mathematical Cost Model
Total cost of ownership (TCO) over a given operational horizon $T$ (months):
TCO (Federation) = ∑ [ RemoteStorage + OmniCompute + EgressResult ]
TCO (Migration) = OneTimeEgress + ∑ [ GCSStorage + DeltaSyncEgress + GCPCompute + PipelineOps ]
Google Cloud Cross-Cloud Interconnect (CCI) Impact: When establishing dedicated 10G/100G CCI links between AWS and Google Cloud, egress fees drop from standard internet rates (~$0.08–$0.09/GB) down to ~$0.02–$0.05/GB, significantly lowering the barrier for ongoing federation and bulk migration.
The Breakeven Formula
At what query frequency and data scan volume does Physical Migration become cheaper than Federation?
Monthly Breakeven Condition:
N_queries × ( Cost_Omni_Execution + Cost_WAN_Shuffle_Egress ) > ( Cost_GCS_Storage + Cost_Sync_Pipelines )
If queries perform cross-cloud joins, BigQuery Omni must shuffle intermediate partitions across the WAN. If an un-aggregated intermediate shuffle of 10 GB occurs per query, the egress per query escalates dramatically from $0.004 to $0.80.
Latency Profiles: The Speed of Light vs. Colocated NVMe
- Light Aggregation (SUM, AVG, COUNT with partition pruning): Federation is excellent. Omni executes locally in AWS. Only the 2 KB summary crosses the WAN. Latency delta vs native is negligible (< 500ms overhead for control-plane handshakes).
- Cross-Cloud Fact-to-Fact Joins (S3 transactions JOIN GCS customers): Federation suffers severe latency degradation. BigQuery must materialize one side of the join and transmit it across the WAN over TLS. Latency spikes from 12 seconds to 100+ seconds. Local GCS migration resolves this via Google’s high-bandwidth Jupiter network fabric.
- Interactive BI (Looker / Tableau / PowerBI): Federation cannot leverage BigQuery BI Engine in-memory acceleration. Migration enables full BI Engine acceleration for sub-second P95 dashboard responses.
The Executive Decision Matrix
3. Step-by-Step Engineering Implementation
Track A: Configuring BigQuery Omni over AWS S3 Iceberg
Step 1: Establish Secure IAM Federation (Zero Secret Keys)
Never use static AWS access keys. BigQuery Omni uses Google Cloud Service Accounts federated with AWS IAM through OpenID Connect (OIDC).
1. In Google Cloud, create a BigQuery Omni AWS connection:
# Set environment variables
export PROJECT_ID="corp-lakehouse-prod"
export REGION="aws-us-east-1"
export CONNECTION_ID="omni-aws-s3-lakehouse"
# Create the Omni Connection in the corresponding AWS region
bq mk --connection \
--connection_type=AWS \
--location=${REGION} \
--project_id=${PROJECT_ID} \
${CONNECTION_ID}
2. Retrieve the Google Identity created for this connection:
bq show --format=prettyjson --connection ${PROJECT_ID}:${REGION}:${CONNECTION_ID}
Note the identity field in the output, e.g.: "identity": "104928374910293847592"
3. In AWS, create an IAM Role (BigQueryOmniAccessRole) with a Trust Policy trusting Google's OIDC issuer:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "accounts.google.com"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"accounts.google.com:sub": "104928374910293847592"
}
}
}
]
}
4. Attach S3 and Glue permissions to the role and bind the role ARN back to BigQuery:
export AWS_ROLE_ARN="arn:aws:iam::123456789012:role/BigQueryOmniAccessRole"
bq update --connection \
--location=${REGION} \
--aws_role_arn=${AWS_ROLE_ARN} \
${PROJECT_ID}:${REGION}:${CONNECTION_ID}
Step 2: Register the External Iceberg Table in BigQuery
-- Create an external BigLake dataset located in the AWS Omni region
CREATE SCHEMA IF NOT EXISTS `corp-lakehouse-prod.aws_omni_ecommerce`
OPTIONS (
location = 'aws-us-east-1'
);
-- Register the AWS S3 Apache Iceberg table using the BigLake Connection
CREATE OR REPLACE EXTERNAL TABLE `corp-lakehouse-prod.aws_omni_ecommerce.customer_orders`
WITH CONNECTION `aws-us-east-1.omni-aws-s3-lakehouse`
OPTIONS (
format = 'ICEBERG',
uris = ['arn:aws:glue:us-east-1:123456789012:table/ecommerce_db/customer_orders']
-- Alternatively, provide direct metadata file pointers:
-- uris = ['s3://enterprise-iceberg-lakehouse-us-east-1/customer_orders/metadata/v482.metadata.json']
);
Step 3: Execute Federated Analytics
-- Query runs locally in AWS with near-zero egress toll
SELECT
order_date,
region,
COUNT(order_id) AS total_orders,
SUM(order_amount_usd) AS gross_revenue
FROM `corp-lakehouse-prod.aws_omni_ecommerce.customer_orders`
WHERE order_date >= '2026-09-01'
GROUP BY 1, 2
ORDER BY gross_revenue DESC;
Track B: High-Throughput Physical Migration Pipeline
Step 1: Automate Mass Data Transfer with Storage Transfer Service (STS)
# Terraform configuration for continuous S3 to GCS transfer
resource "google_storage_transfer_job" "s3_to_gcs_iceberg_sync" {
description = "Continuous multi-cloud sync from AWS S3 Iceberg to GCS Lakehouse"
project = var.gcp_project_id
transfer_spec {
aws_s3_data_source {
bucket_name = "enterprise-iceberg-lakehouse-us-east-1"
role_arn = "arn:aws:iam::123456789012:role/GcpStorageTransferServiceRole"
}
gcs_data_sink {
bucket_name = "gcp-iceberg-lakehouse-us-central1"
path = "lakehouse_tables/customer_orders/"
}
transfer_options {
overwrite_when = "DIFFERENT"
delete_objects_unique_in_sink = false
}
}
schedule {
schedule_start_date {
year = 2026
month = 10
day = 1
}
}
}
Step 2: Fixing Iceberg Metadata Paths (s3:// to gs://)
Use Apache Spark on Cloud Dataproc Serverless to rewrite manifests cleanly:
# rewrite_iceberg_metadata.py (Cloud Dataproc Serverless)
from pyspark.sql import SparkSession
spark = SparkSession.builder \
.appName("IcebergMetadataPathRewrite") \
.config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \
.config("spark.sql.catalog.gcs_catalog", "org.apache.iceberg.spark.SparkCatalog") \
.config("spark.sql.catalog.gcs_catalog.type", "hadoop") \
.config("spark.sql.catalog.gcs_catalog.warehouse", "gs://gcp-iceberg-lakehouse-us-central1/warehouse") \
.getOrCreate()
# Call Iceberg's migrate procedure to re-point manifests to GCS paths
spark.sql("""
CALL gcs_catalog.system.migrate(
table => 'gs://gcp-iceberg-lakehouse-us-central1/lakehouse_tables/customer_orders'
)
""")
Step 3: Registering GCS Iceberg Tables with BigLake & Cache Acceleration
CREATE OR REPLACE EXTERNAL TABLE `corp-lakehouse-prod.gcp_lakehouse_ecommerce.customer_orders`
WITH CONNECTION `us-central1.biglake-gcs-connection`
OPTIONS (
format = 'ICEBERG',
uris = ['gs://gcp-iceberg-lakehouse-us-central1/lakehouse_tables/customer_orders/metadata/v482.metadata.json'],
max_staleness = INTERVAL 30 MINUTE, -- Enables internal BigQuery metadata caching
metadata_cache_mode = 'AUTOMATIC'
);
4. Key Engineering Challenges and Battle-Tested Solutions
Challenge 1: Metadata Drift and Synchronization Lag
The Problem: When upstream pipelines write new commits or compact files on S3, BigQuery’s catalog can quickly become out of sync, returning stale snapshots or failing on deleted manifests.
The Solution: If using direct S3 URIs, set up Amazon EventBridge on S3 ObjectCreated events in the metadata/ prefix, routed via SNS to a Cloud Function invoking ALTER TABLE … REFRESH METADATA. Better yet, integrate with an Iceberg REST Catalog (e.g., Apache Polaris) to provide immediate commit visibility across both clouds.
Challenge 2: Cross-Cloud Join Data Gravity Trap
The Problem: Joining an AWS S3 Omni table with a GCS BigQuery table can cause an uncompressed multi-terabyte cross-cloud shuffle across the WAN.
The Solution: Always pre-aggregate and filter inside Common Table Expressions (CTEs) within the Omni subquery before executing the join, and enforce strict maximum_bytes_billed guardrails on analytical queries.
Challenge 3: Multi-Engine ACID Maintenance & Compaction Collisions
The Problem: Concurrently running OPTIMIZE, VACUUM, or orphan file cleanup on AWS while BigQuery Omni is scanning manifests can cause FileNotFoundException errors.
The Solution: Enforce single-writer governance for table maintenance. Configure Iceberg table properties: history.expire.min-snapshots-to-keep = 10 and retain snapshots for at least 24 hours to protect in-flight read queries.
Challenge 4: Cross-Cloud Security and Column Masking
The Problem: Duplicating access policies across AWS IAM/Lake Formation and GCP IAM/Dataplex leads to governance drift.
The Solution: Centralize governance in Google Cloud Dataplex. BigLake connections enforce BigQuery Row-Level and Column-Level Security policies even when the compute executes over BigQuery Omni inside AWS S3, masking data before results egress to GCP.
Challenge 5: Partition Pruning Failures Across Catalogs
The Problem: Hidden partition transforms (e.g., days(timestamp)) can fail to translate between BigQuery and foreign catalog indices, triggering full-table scans.
The Solution: Always inspect the execution plan (bq show -j <job_id>) to verify partition pruning. Prefer Iceberg REST Catalogs, which natively interpret Iceberg V2 partition specifications.
5. Frequently Asked Questions (FAQ)
Q1: Does BigQuery Omni require me to manage Kubernetes or Anthos clusters inside my AWS VPC?
No. BigQuery Omni runs as a fully managed SaaS service operated by Google inside Google-managed AWS accounts located in the same AWS region as your S3 buckets. You manage no EC2 instances, VPC peering, or Anthos clusters. Connectivity is purely via AWS IAM role federation.
Q2: Is there an egress cost when BigQuery Omni reads data from S3?
No, not for scanning data. The Omni compute cluster is located inside the same AWS region. S3 reads stay within the AWS local region network and incur zero egress. You only pay AWS egress fees for the final result rows returned to GCP or written to an external destination.
Q3: What happens if an Iceberg table undergoes an ACID commit while BigQuery Omni is scanning it?
Apache Iceberg provides snapshot isolation. BigQuery Omni reads the snapshot ID determined at the start of the query planning phase. Even if concurrent writes append new data files or compact existing ones, the in-flight query continues reading immutable manifest files without locking or errors.
Q4: Can I use BigQuery BI Engine to accelerate federated Iceberg tables?
No. BI Engine’s in-memory execution subsystem relies on low-latency memory buses physically colocated within Google Cloud data centers. It cannot cache remote AWS S3 tables. If sub-second P95 dashboard responsiveness is mandatory for Looker or Tableau, physical migration to GCS is necessary.
Q5: How should we choose between an Iceberg REST Catalog and AWS Glue when using BigQuery Omni?
If your architecture involves multi-engine interoperability (BigQuery Omni, Snowflake, Databricks, Spark), choose an Iceberg REST Catalog (e.g., Apache Polaris or Project Nessie). The open specification avoids proprietary catalog translation quirks and provides unified ACID visibility across all engines.
Q6: Can Storage Transfer Service (STS) sync only the delta changes of an Iceberg table?
Yes. STS supports object-level change detection, timestamp filtering, and manifest-driven file transfers. By coupling a Cloud Function to Iceberg commit events, you can feed STS a list of newly generated Parquet and metadata files, syncing only the deltas rather than re-scanning petabytes of historical objects.
6. The Architect’s Playbook: Summary & Next Steps
To choose between Migrate and Federate, run this 5-day assessment within your data platform team:
The Rule of Thumb:
- Federate via BigQuery Omni if:
- Your data changes rapidly in AWS and is primarily produced and consumed there.
- Your GCP queries are analytical, aggregate-heavy, or ad-hoc.
- You face regulatory data residency constraints or lack the migration engineering bandwidth.
- Migrate physically to GCS Lakehouse if:
- You require sub-second interactive BI Engine acceleration in Looker.
- Queries constantly perform heavy joins between multi-cloud data and GCP-resident data warehouses.
- High query frequency will cause cumulative egress costs to exceed one-time migration costs within 3 to 6 months.
7. Official Google Cloud References & Architecture Center Citations
For architecture review boards and deep technical verification, reference these official Google Cloud documentation guides:
Introduction to BigQuery Omni
Google Cloud documentation detailing the multi-cloud Anthos architecture, regional compute workers, and zero-egress local processing.
Configure AWS S3 Cross-Cloud Connections
Step-by-step security guide for configuring IAM trust policies (AssumeRoleWithWebIdentity) between Google BigQuery and AWS S3.
Run Cross-Cloud Queries with BigQuery Omni
Documentation on executing federated queries, join constraints, WAN data transfers, and writing query results to cross-cloud destination tables.
Introduction to BigLake Storage Engine
Detailed architecture of BigLake, decoupling storage from compute while enforcing row/column security and metadata caching over Apache Iceberg.
Query Apache Iceberg External & Managed Tables
Official guide to creating BigLake Iceberg tables, registering external metadata from S3/GCS, and managing full-lifecycle Iceberg tables.
BigLake Metastore Overview
Unified, open REST-compatible metastore service enabling seamless interoperability between BigQuery, Spark, Presto, and Trino.
Google Cloud Storage Transfer Service (STS)
Architecture and best practices for automated, petabyte-scale transfers from Amazon S3 and Azure Blob to Google Cloud Storage.
Google Cloud Cross-Cloud Interconnect (CCI)
High-speed physical interconnect establishing private 10 Gbps / 100 Gbps dedicated links between Google Cloud and AWS/Azure.
Google Cloud Dataplex Universal Governance
Intelligent data fabric for unified metadata discovery, automated data quality, and policy tag enforcement across multi-cloud lakehouses.
Migrate vs. Federate: How to Choose the Right Strategy for Your Multi-Cloud Iceberg Data was originally published in Google Cloud – Community on Medium, where people are continuing the conversation by highlighting and responding to this story.
Source Credit: https://medium.com/google-cloud/migrate-vs-federate-how-to-choose-the-right-strategy-for-your-multi-cloud-iceberg-data-f43373142812?source=rss—-e52cf94d98af—4
About the Author
