Boost Performance and Lower Costs with AlloyDB AI Functions

Here is a sample result of summarized reviews for two gaming console products: 

The power of LLMs on your data: now significantly faster and cheaper

We now have achieved unprecedented performance and cost breakthroughs in AI function processing. Previously, running a foundation model call for every single row in a massive database introduced cost and latency constraints. We have shattered these barriers by introducing two breakthrough capabilities:

Smart Batching for AI Functions: This AI Function Acceleration capability provides intelligent batching of AI function calls for optimal performance and quality. This efficiency is achieved by deduplicating prompt overhead; the LLM’s boilerplate instructions are transmitted once per batch rather than repeated across every individual row. A question you may have is – “Why not do this in my own application layer?”. That’s because, AlloyDB intelligently determines the right batch size for optimal results – if you underestimate the batch size, you won’t reap gains for cost and latency, and if you overestimate the batch size, the prompt to the LLM could get bloated and lead to hallucinations, or you could exceed the model’s token limits. In addition to calculating the perfect batch size for every request, AlloyDB also handles retries automatically out of the box, ensuring your pipeline stays resilient. We did some testing internally and saw massive gains; for example, an up to  2,400x performance boost (processing 10,000 rows/sec) over traditional row-at-a-time LLM calls. This is currently available for the ai.if and ai.rank functions, with support for additional functions coming in the future.

Let’s look at an example of using Smart Batching / Acceleration with ai.if to solve this use case: Imagine a customer on a gadget retail site searching for a camera that can handle an underwater depth of ’60 meters or deeper.’ Traditional hybrid search will pull the closest semantic and full-text matches, but it misses the hard constraints of numerical data—meaning it might serve up a camera that works only at 20 meters depth. By using AlloyDB’s ai.if-based intelligent filtering, the database actually understands the nuance of depth and makes the query return products that meet or exceed that 60-meter depth criteria. Notice how, in the example below, you don’t need to specify the batch size – AlloyDB handles all the optimizations under the hood when using ai.if.