Introduction
If you operate large-scale data pipelines, you know the frustration of watching cluster runtimes drag while your cloud infrastructure bill climbs. Traditional Spark execution relies heavily on the Java Virtual Machine (JVM). While Project Tungsten and Whole-Stage Code Generation have pushed the boundaries of optimization, CPU-bound workloads eventually hit an unavoidable ceiling.
To bypass these JVM limits, Spark workloads are shifting towards native acceleration. In this model, Spark acts as the coordinator and scheduler. It delegates the heavy row-by-row data processing to high-performance, native query engines written in C++, Rust, or CUDA. By running execution in native libraries and using vectorized data formats, you can run Spark jobs faster, lower your cloud infrastructure costs, and eliminate garbage collection pauses.
To understand why native engines excel, you must examine the specific limits of the JVM when processing analytical queries at scale.
The Volcano Iterator Model
When standard database engines or Spark plans execute, they use the Volcano Iterator model. Each operator in the physical plan exposes a next() method that pulls ina single row at a time. For datasets with billions of rows, this model triggers billions of virtual function dispatches through Java vtables. This ruins CPU instruction cache locality and the CPU cannot retain intermediate results in high-speed registers. Instead, it must constantly push and pull data from the JVM function call stack.
JVM Garbage Collection (GC) Overhead
To aggregate or join large datasets, Spark creates millions of temporary objects on the JVM heap. Even when you use off-heap memory, these objects trigger frequent garbage collection pauses that freeze executor threads. On datasets scaling into tens of terabytes, garbage collection overhead regularly consumes a large chunk of overall Spark task execution time. This overhead wastes cluster resources and inflates your infrastructure bills.
JIT Compiler Limits and SIMD Inefficiencies
To bypass the Volcano model, Spark’s Tungsten engine compiles physical operators into single Java methods at runtime. The JVM JIT compiler then optimizes these methods. However, the JVM limits method bytecode size to 64KB. Furthermore, the JIT compiler refuses to optimize or inline any method that exceeds 8KB of bytecode. When you run complex queries with wide tables or nested CASE WHEN expressions, Spark breaches these limits. It then falls back to the slow Volcano iterator mode.
Additionally, the JVM JIT compiler fails to auto-vectorize standard Java code. It cannot leverage modern CPU Single Instruction Multiple Data (SIMD) registers, such as AVX-2 and AVX-512. These registers can process multiple data points in a single clock cycle. Because Java lacks fine-grained, assembly-level hooks, you cannot enforce SIMD vectorization manually.
The Solution: The Columnar Native Shift
Columnar native execution addresses these bottlenecks by offloading physical execution to native query engines while keeping Spark as the central orchestrator.
In this architecture, Spark still handles query planning, table catalog coordination, task scheduling, and partition distribution. However, when the Spark executors run their physical tasks, they bypass JVM execution. Instead, they pass entire sub-graphs of the physical plan to native C++, Rust, or CUDA libraries.
To make this transition efficient, native accelerators rely on Apache Arrow and vectorized memory layouts. Apache Arrow stores data in memory as contiguous columns rather than rows. This alignment enables native engines to run SIMD instructions directly on the columns.
To prevent performance losses from JNI (Java Native Interface) overhead, accelerators do not make JNI calls for individual rows. Instead, they offload entire physical plan pipelines. The JVM and the native engine exchange large blocks of data as zero-copy Apache Arrow ColumnarBatch payloads. This approach minimizes JNI transitions and keeps data processing inside native memory space.
Apache Gluten + Velox (C++ CPU Engine)
Apache Gluten intercepts the physical plan after Spark’s Catalyst Optimizer finishes. It translates the plan into Substrait IR (Intermediate Representation). This format uses Google Protocol Buffers to serialize an open, cross-language relational algebra format. Gluten passes this serialized Substrait plan over JNI to Velox, a vectorized C++ query engine from Meta. For jobs that Gluten can’t effectively move to Velox, it will fall those jobs back to the JVM.
Velox processes the execution plan natively. It uses a unified off-heap memory allocator. To prevent out-of-memory (OOM) crashes, Velox manages its own integrated disk spilling natively in C++ when off-heap allocations exceed your configured limits.
The Apache Software Foundation graduated Apache Gluten to a Top-Level Project in March 2026.
To set up and deploy Gluten with the Velox backend in your Spark environment, follow the instructions in the official Apache Gluten Documentation. The guide covers prerequisite environment configuration, pre-compiled bundle JAR downloads, off-heap memory sizing, and instructions for building custom bundles for x86_64 and ARM64 architectures.
Real-world Case Study: Palantir Foundry
Palantir Foundry integrated Apache Gluten + Velox directly into Palantir Foundry as its native acceleration feature for Python transforms and Pipeline Builder.
The Foundry query engine applies optimization rules to identify Spark physical operators that Velox can execute. In the Spark physical plan, the engine labels these native nodes with a caret symbol (^), such as ^ProjectExecTransformer and ^HashAggregate. Transition nodes called RowToVeloxColumnar and VeloxColumnarToRowExec sandwich these native blocks. These transition nodes serialize JVM Spark data into Arrow format and deserialize native outputs back into Spark UnsafeRow format.
To optimize performance, you must monitor serialization overhead. If your physical plan contains too many transition nodes, the CPU cost of moving data across the JNI boundary can exceed the compute savings of the Velox C++ engine.
Performance Benchmarks
On a single-node setup running a 3TB TPC-H dataset, Apache Gluten achieved a 3.34x overall speedup compared to standard Spark, with individual queries achieving speedups up to 23.45x.
Apache DataFusion Comet (Rust CPU Engine)
Apple originally developed Apache DataFusion Comet, which joined the Apache Software Foundation as a DataFusion subproject in 2024. Comet focuses on modularity, safety, and an extremely simple setup.
Architecture and Arrow-Native Execution
Comet leverages the Rust-based Apache DataFusion query engine. It keeps Spark queries Arrow-native end-to-end. Operators, expressions, shuffle exchanges, and broadcast variables all remain in the Apache Arrow columnar format.
Like Gluten, Comet translates the JVM physical plan using Protocol Buffers and deserializes it into native DataFusion execution nodes in Rust.
To eliminate JVM-based transition steps, Comet version 0.14.0 and newer uses a native Rust-based Columnar-to-Row (C2R) converter by default. This converter reduces memory overhead when native query output flows back into row-based Spark operators.
Additionally, Comet manages threads efficiently. All Spark tasks on a Comet executor share a single multi-threaded Tokio runtime in Rust. The executor threads communicate with native DataFusion streams via asynchronous channels (mpsc) and park themselves in blocking_recv() to avoid CPU busy-polling.
To add Comet to your Spark cluster without building native binaries from source, review the Apache DataFusion Comet Installation and Quickstart Guide. The guide details how to download the pre-compiled JAR, configure the Comet Spark plugin, and enable native Parquet scanning and columnar shuffle managers.
Real-world Case Studies
Apple developed and open-sourced Comet to run petabyte-scale transformations on their Apache Iceberg data lakehouse.
LiveRamp adopted Comet with Apache Arrow to accelerate Spark-based Clean Room workloads. By moving from JVM heap memory to Arrow-based off-heap memory, LiveRamp reduced garbage collection overhead, lowered heap pressure, minimized memory spills, and optimized CPU utilization with zero code changes.
Performance Benchmarks
On a 1TB TPC-DS dataset running on commodity hardware, Apache DataFusion Comet delivered a 2x overall speedup, which translated to a 50% reduction in cloud infrastructure costs without requiring GPU acceleration.
NVIDIA RAPIDS Accelerator (GPU Engine)
The RAPIDS Accelerator translates Spark physical plans into GPU execution graphs. These graphs run on the C++/CUDA-based cuDF library.
By executing directly on GPUs, RAPIDS exploits High Bandwidth Memory (HBM). This memory achieves a massive bandwidth of 1.5 to 3.0+ TB/s, compared to only 100–300 GB/s on traditional CPUs.
To bypass network and CPU bottlenecks during data exchanges, RAPIDS includes an accelerated shuffle manager that utilizes Unified Communication X (UCX). UCX enables GPUDirect RDMA and NVLink GPU-to-GPU communications. This technology transfers shuffle blocks directly between GPU memories across the network without involving host CPUs or system RAM.
Additionally, RAPIDS manages memory latency. To avoid the high latency of dynamically allocating GPU memory, the accelerator initializes a pre-allocated RAPIDS Memory Manager (RMM) pool. This pool can expand dynamically, but it remains isolated from standard JVM memory.
To deploy the RAPIDS Accelerator across on-premise GPU nodes, Kubernetes clusters, or cloud platforms, consult the official NVIDIA RAPIDS Accelerator User Guide. The documentation provides comprehensive walkthroughs for configuring GPU memory manager (RMM) pools, enabling UCX accelerated shuffle, and leveraging the RAPIDS Qualification Tool to identify which pipelines will benefit most from GPU acceleration.
Real-world Case Studies
Commonwealth Bank of Australia (CBA) deployed the RAPIDS Accelerator on GPU infrastructure to run credit card transaction fraud models. The GPU acceleration delivered a massive 640x performance boost, processing 6.3 billion transactions in just five days and dropping daily inference time to 46 minutes.
Capital One deployed the NVIDIA RAPIDS Accelerator alongside Dask-RAPIDS on AWS EMR to run credit card fraud detection. The workflow achieved a 14x faster end-to-end processing, an 8x reduction in infrastructure costs, and accelerated model iteration cycles by 100x.
Performance Benchmarks
On standard analytical and machine learning pipelines, the RAPIDS Accelerator delivered a 5x overall speedup and reduced compute infrastructure costs by up to 4x.
Google Cloud Lightning Engine (Managed C++ CPU Engine)
Lightning Engine is Google Cloud’s proprietary, fully managed query acceleration engine. Google integrates it directly into the Managed Service for Apache Spark (which encompasses both serverless batch workloads and cluster-based Compute Engine VM deployments).
Lightning Engine utilizes Google-engineered extensions of Apache Gluten and Velox. Google optimized these C++ libraries to integrate with Compute Engine VMs and hardware layers.
To eliminate manual memory tuning, Lightning Engine includes an intelligent, unified memory manager. This manager dynamically shifts memory allocations between off-heap C++ and JVM on-heap spaces. This automation prevents manual tuning errors and completely eliminates OOM memory spills.
Additionally, Lightning Engine incorporates custom optimization rules from Google’s internal database systems, F1 and Procella:
- Single HashTable Caching: In standard Spark, broadcast joins repeatedly build join hash tables across separate tasks on the same executor. Lightning Engine builds this hash table once per executor and caches it. This lowers CPU utilization and memory footprint.
- Aggregation Pushdown: This rule pushes partial aggregations downstream of join shuffles to reduce network transfer volumes.
- Auto Shuffle Partitioning: This mechanism adaptively determines the optimal number of shuffle partitions at runtime based on stage statistics. It prevents OOM spills on wide stages without over-partitioning.
Cloud-Native Storage and Connectors
To eliminate the hidden “listing tax” of reading partitioned Cloud Storage tables, the driver utilizes lexicographical listing to cache Google Cloud Storage (GCS) metadata. It broadcasts this metadata directly to executors. This minimizes GCS API listing calls and lowers cloud costs.
To accelerate scan speeds, the native BigQuery connector consumes BigQuery data directly in the native Apache Arrow format. By bypassing JVM row-oriented transition (UnsafeRow serialization), this connector eliminates CPU overhead.
To enable native query acceleration in Google Cloud, you do not need code changes or manual installation steps. You can activate Lightning Engine with simple configuration flags when submitting batch jobs or creating clusters in Managed Service for Apache Spark.
Real-world Case Studies
Lowe’s migrated its compute-intensive analytical and data warehousing pipelines on Managed Service for Apache Spark (serverless mode) to the Lightning Engine. This migration delivered major reductions in aggregate Compute Engine runtime hours, lowered overall Google Cloud spend, and accelerated data-to-insight timelines with zero code modifications.
Performance Benchmarks
On a 10TB TPC-DS dataset, Lightning Engine achieved up to a 4.9x overall speedup.
Try Lightning Engine Today: Follow the step-by-step walkthrough in the Google Cloud Lightning Engine Codelab to test native query acceleration on your own Spark pipelines.
Conclusion
Spark native accelerators provide a viable path to bypass JVM performance ceilings. By offloading CPU-bound tasks to vectorized, native query engines in C++, Rust, and CUDA, you can dramatically accelerate execution speeds and lower cloud infrastructure costs.
What native accelerator are you excited to test in your Spark pipelines? Share your thoughts in the comments section!
Moving Apache Spark beyond the JVM with Native Acceleration was originally published in Google Cloud – Community on Medium, where people are continuing the conversation by highlighting and responding to this story.
Source Credit: https://medium.com/google-cloud/moving-apache-spark-beyond-the-jvm-with-native-acceleration-0835ffcdc461?source=rss—-e52cf94d98af—4
About the Author
