BigQuery isn’t just a data warehouse anymore. Most teams know how to run analytics queries, build dashboards, or use BigQuery for BI.
But a surprisingly underused feature — BigQuery Sharing — lets you build live data exchanges across internal teams, regions, and even external partners without copying data. It’s stable, scalable, and solves real modern data distribution problems, yet few teams adopt it.
As data usage shifts from reporting to operational and AI-driven workloads, sharing data becomes a platform problem, not an export task.
In this article we’ll cover:
- Why traditional sharing patterns are painful
- What BigQuery Sharing actually is
- How governance and cost models work
- VPC-SC rules for secure sharing
- Streaming data sharing
- Real world use cases
- Terraform examples
The Problem With Traditional Data Sharing
Many organizations still share data by:
- Exporting CSV dumps
- Copying datasets to partner projects
- Using SFTP folders or email attachments
- Maintaining fragile ETL pipelines
These approaches create:
- Data duplication
- Version mismatch
- Security blind spots
- High maintenance work
What if you could share live data in a governed way, without moving it? That’s what BigQuery Sharing enables.
What BigQuery Sharing Actually Is
BigQuery Sharing (formerly Analytics Hub) is a built-in way to:
- Share datasets or views
- Publish them as listings
- Group them in data exchanges
- Enable internal or external consumers to subscribe and query them
Despite being available for a while, it’s still underrated — likely because few architects realize it exists or how much it simplifies data collaboration.
Core Concepts
- Data Exchange: A catalog of shared assets
- Listing: A specific dataset/view advertised in an exchange
- Subscriber: A consumer project linked to the listing
- Shared Dataset: Stays in the publisher’s project — not copiedThese concepts allow you to publish and subscribe to shared data without moving raw data around.
Where It Fits in Modern Architectures
This works especially well in:
Internal Enterprise Data Hubs
Imagine each region or line of business owns its own BigQuery project. They produce data independently but need a central analytics function:
- Each unit publishes curated datasets/views
- The central analytics team subscribes
- No ETL copies needed
External Partner Data Sharing
Retailers, manufacturers, and service providers can share relevant data with vendors or partners:
- Product performance metrics
- Aggregated sales trends
- Regional insights
Your partners query the shared listings directly, paying for compute while you retain ownership and storage of data.
Pricing & Governance
BigQuery separates storage and compute:
- Data stays in the publisher’s project
- Consumers pay for the compute they use
That means:
- Publishers manage storage costs
- Consumers pay for query costs
- Nothing is physically moved or duplicated
Governance stays with you:
- IAM controls dataset visibility
- Authorized views shape what consumers see
- Audit logs track activity
This provides a clean security and billing boundary that traditional file sharing can’t match.
VPC-SC Support for BigQuery Sharing (Security Perimeters)
Many enterprises operate in confidential or regulated environments.
Google Cloud’s VPC Service Controls (VPC-SC) let you restrict data access to defined perimeters.
The good news is BigQuery Sharing (Analytics Hub) works with VPC-SC. You can share datasets between projects that are inside the same perimeterand still enforce perimeter policies. This ensures:
- Shared data doesn’t leak outside your security boundary
- Shared datasets respect your compliance posture
- All BigQuery queries stay within scoped networks
This makes BigQuery Sharing suitable for regulated industries like finance, healthcare, and government where perimeter controls are required.
In practice you only need to ensure both the publisher and consumer projects are in the same VPC-SC perimeter.
Real-Time Data Sharing: BigQuery Stream Sharing
Static datasets are useful — but many applications today depend on real-time data delivery. BigQuery’s Stream Sharing enhances BigQuery Sharing by enabling real-time streams to be shared.
This is a powerful pattern. Instead of waiting for batch loads:
- Data is appended in real time
- Subscribers see new rows as they arrive
- You don’t need separate brokers or sync jobs
Streaming Use Cases
Financial Services
- Share rapidly changing instrument prices, quotes, and orders
- Detect money-laundering or payment fraud with near-real-time data
- Support trading risk computations
Retail and CPG
- Manage store inventory in real time
- Personalize marketing and customer interaction
- Adjust prices dynamically
- Monitor social signals (e.g., social media feeds)
- Optimize physical store layouts with live operational feeds
Stream sharing gives consumers current data while publishers retain ownership and governance.
Terraform Snippets
- Create a Data Exchange
resource "google_bigquery_analytics_hub_data_exchange" "hub" {
project = var.hub_project_id
location = var.location
data_exchange_id = "global_data_exchange"
display_name = "Global Data Exchange"
description = "Shared resources for internal and external data consumers"
}
2. Publish a Listing
resource "google_bigquery_analytics_hub_listing" "events_listing" {
project = var.hub_project_id
location = var.location
data_exchange_id = google_bigquery_analytics_hub_data_exchange.hub.data_exchange_id
listing_id = "events_shared"
display_name = "Shared Events"
description = "Curated view of events data for partners"
bigquery_dataset {
dataset = "projects/${var.hub_project_id}/datasets/${var.shared_dataset_id}"
}
}
Conclusion
BigQuery Sharing, including VPC-SC support and real-time stream sharing is one of those hidden but practical features that solves real collaboration problems.
It lets you:
- Share data across internal teams without copying
- Securely expose curated data to external partners
- Stream real-time data with governance
- Separate storage and compute billing cleanly
- Stay within security perimeters
BigQuery Sharing: An Underrated Data Exchange Platform You Should Know was originally published in Google Cloud – Community on Medium, where people are continuing the conversation by highlighting and responding to this story.
Source Credit: https://medium.com/google-cloud/bigquery-sharing-an-underrated-data-exchange-platform-you-should-know-3a440101383b?source=rss—-e52cf94d98af—4
About the Author
