“Only 10%-30% of most data in enterprises is structured. The rest is unstructured.”
That’s what studies from internet are saying in 2026.Because it’s a reality, I’ve decided to study myself about what we can do we those unstructured data.
Assuming you have different kind of files as PDF, PPT, PPTX …those files could be very heavy and complex to query.
I have imagined a workflow for unstructured data to answer business questions.
It start with data ingestion, the classic way, serverless with components that are event based :
- Event-Driven Cloud Function (Gen2):
Automatically triggered via Eventarc (object.v1.finalized) upon any file upload to the GCS bucket.
Parallel orchestration using a worker pool that concurrently feeds all 5 targets. A failure in one engine does not interrupt the remaining four thanks to strict error isolation.
- RAM-Safe Streaming Extraction:
To prevent out-of-memory (OOM) errors on large Office files (particularly media-heavy PowerPoint decks), a range-based HTTP reader was introduced. It selectively reads only the compressed XML entries from the ZIP archive without loading hundreds of megabytes of embedded media into RAM.
Then it’s all about extraction, embeddings generations, storing them into a database and serve this data to be requested later.
The platform implements five fully autonomous, production-grade RAG pipelines running concurrently. Rather than sharing a common vector representation, each engine operates entirely end-to-end within its own dedicated stack — handling its own document parsing, chunking strategy, native embedding model, and in-database vector retrieval.
These five architectures are particularly compelling because each relies on distinct components, exhibits different latencies, and carries different operational costs — ultimately delivering unique trade-offs depending on your specific requirements.
After that you have to be able to manipulate and query the data :
The Flask application (deployed on App Engine and secured via Identity-Aware Proxy) provides three primary modules:
- Catalog & Coverage: Document-by-document visibility into ingestion status and chunk counts per engine (demonstrating why token-based, layout-aware splitting yields fewer, denser chunks than raw character-based chunking).
- Comparative Benchmark:
Concurrent querying across all 5 engines for any user prompt.
Side-by-side display of generated answers, retrieval latencies, and selected source chunks.
Automated comparative synthesis authored by Gemini.
Qualitative evaluation powered by RAG Triad metrics (Faithfulness, Context Relevancy, Answer Relevancy).
- GraphRAG Explorer: Interactive visualization of the Neo4j knowledge graph, paired with a read-only Text-to-Cypher interface allowing users to explore relational data structures in natural language.
And to finish I thought I need a neutral analysis to override my personal flavor :
When a user submits a query in the Benchmark tab:
- Concurrent execution across all 5 pipelines: The query is dispatched simultaneously to all 5 engines using an asynchronous execution pool.
- Dedicated vector spaces: For each engine, the user query is embedded on the fly using its own native embedding model, ensuring an honest end-to-end comparison of the entire pipeline (rather than a simple database retrieval benchmark over a shared vector space).
- Strict fault isolation: If an individual engine throws an exception or hits an API rate limit, its result degrades gracefully without impacting or blocking the other 4 engines.
- Parallel answer generation: The LLM (Gemini 3.7 Flash) generates an isolated synthesis in parallel for each engine, grounded strictly on the context chunks retrieved by that specific engine.
- Dual-layer evaluation:
Business & Domain Comparative Analysis: Gemini serves as an impartial domain and scientific judge to establish the latency podium ranking, identify facts, figures, or claims captured by certain versions and missed by others, and deliver a final operational recommendation.
Automated RAG Triad Metrics: Automated scoring across 3 fundamental criteria: Context Faithfulness, Context Relevancy, and Answer Relevancy.
The conclusions of this article is :
This not in the hands of Gemini only, don’t worry because even if it feel impartial the very last important milestone is the human in the loop and I have shared this app with other colleagues and we all agree :
- The performance is important and interesting to know
- The most important one is the quality of the embeddings
- The second most important thing is the way the data is analysed and explained, very different from one to another. (Again depending your need)
Just for your eyes only I’m sharing with you the average latency per solution when you do a request from the front :
— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — –
🕸️ Neo4j AuraDB [████████] 1,383 ms 🥇
🍃 MongoDB Atlas [█████████] 1,500 ms (+117 ms)
⚡ BigQuery AI [██████████████] 2,200 ms (+817 ms)
🔷 Azure AI Search [████████████████] 2,800 ms (+1,417 ms)
🧱 Databricks [███████████████████████] 3,600 ms (+2,217 ms)
— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — –
Don’t judge only by the numbers, but more by the facts..
RAG into the Wild: 5 Engines, 5 Truths about your unstructured Data was originally published in Google Cloud – Community on Medium, where people are continuing the conversation by highlighting and responding to this story.
Source Credit: https://medium.com/google-cloud/rag-into-the-wild-5-engines-5-truths-about-your-unstructured-data-8ee9b8c3fdc1?source=rss—-e52cf94d98af—4
About the Author
