BigQuery Search innovations: Unify structured & unstructured data

Modern enterprises possess a vast amount of unstructured data, yet they frequently encounter significant challenges in managing and extracting value from it. Historically, unlocking the insights hidden within PDFs, audio files, images, and unstructured text required a fragmented architecture: moving data out of your warehouse, stitching together complex LLM pipelines, and managing disparate search indexes.

BigQuery has worked with many enterprises to make sense of their unstructured data sources. For example, consider an advanced healthcare company managing thousands of clinical trial documents in PDF form. BigQuery helps unlock insights from these documents through a simple, five-step lifecycle: Access, Process, Ground, Relate, and Activate.

In this post, we are highlighting three major milestones focused heavily on the “Ground” phase of this framework:

1. General Availability (GA) of Autonomous Embedding Generation

2. General Availability (GA) of AI.SEARCH with massive single-query performance gains

3. Public Preview of Hybrid Search

Let’s dive into how these features work together to simplify your AI architecture, using a real-world clinical trial research platform as an example.

Simplify Pipelines with Autonomous Embedding Generation (GA)

Building a retrieval-augmented generation (RAG) pipeline or search application usually requires managing complex, asynchronous embedding infrastructure. You have to handle retries, error logging, and pipeline orchestration every time a new record arrives.

With the General Availability of Autonomous Embedding Generation, BigQuery manages this entirely for you. By simply defining a column in your schema, BigQuery asynchronously and continuously generates embeddings as new data is ingested. You have the flexibility to choose external models (like Vertex AI text-embeddings) or natively utilize Gemma embedding models directly within BigQuery.

How it works in practice:

Imagine you are building a research platform analyzing clinical trial PDFs stored in Google Cloud Storage. After extracting the study titles and disease areas into a table, you can automatically embed those titles: