New Firestore query engine enables pipelines

A hallmark of a valuable database is how easy it is to query the data inside, so that developers can build tailored and complex user experiences in an application. Last week marked a significant evolution for Firestore, Google Cloud’s enterprise-grade, scalable document database, with the debut of an advanced query engine designed to help you build more sophisticated applications. 

Available as part of Firestore in Native mode, this powerful engine introduces over a hundred new query capabilities, called pipeline operations, available in preview, which streamline complex queries directly within the database. Alongside this, we’re launching precise indexing controls and refreshed observability tools like query explain and query insights, giving you granular control over performance. All these robust capabilities are now available in the Firestore Enterprise edition, which also offers a more transparent pricing model for potential cost savings. This is all in the service of building highly expressive, performant applications that can query, transform and filter data across many dimensions, with less operational overhead. At the same time, you’re benefiting from Firestore’s unique serverless foundation, multi-region replication, and virtually unlimited scalability, freeing you from database management complexities, so you can truly focus on innovation.

Used by a vibrant community of over 600,000 developers, Firestore has long been appreciated for its simplicity. In 2019, Firestore Standard edition in Native mode streamlined the development of collaborative applications with a straightforward query interface that guaranteed high performance through the use of automatically generated indexes. However, this simplified query engine has a strong dependence on indexing for query execution, often demanding upfront planning throughout the application lifecycle. Now, with the introduction of the advanced query engine in Enterprise edition, developers can construct highly expressive applications, regardless of the explicit presence of indexes — particularly for demanding solutions like e-commerce, interactive gaming, content management, and sophisticated user personalization. The refined query engine makes it easier to create pipeline operations, complete with sophisticated new stages and expressions, including support for complex aggregations, querying directly over arrays, advanced string matching capabilities, and granular filtering options.

A new query engine and pipeline operations experience

To enable this, we’ve updated Firestore’s existing SDKs with expanded support for pipeline operations. Now, you can elegantly chain together numerous stages for essential tasks such as aggregations, grouping, and filtering. Queries now run without mandatory indexes, giving you complete autonomy over when you want to create indexes to optimize performance. Let’s take a look at an illustration of pipeline operations.

Note: This example assumes you’re familiar with Firestore’s data model and existing query methods.

Suppose you want to identify the top trending hashtags on an existing food recipe application that allows users to add hashtags to recipes. For essential data (like the recipe text itself), you might represent a recipe as a document with some fields. Since a hashtag can be represented with just a string, you could add hashtags directly to the recipe document as an array of strings: