DataFusion¶

Setup and registration¶

Add the dependency:

[dependencies]
vortex-datafusion = "<version>"

Register the Vortex format with a SessionContext:

use std::sync::Arc;

use datafusion::datasource::provider::DefaultTableFactory;
use datafusion::execution::SessionStateBuilder;
use datafusion::prelude::SessionContext;
use datafusion_common::GetExt;
use object_store::memory::InMemory;

use crate::VortexFormatFactory;

let factory = Arc::new(VortexFormatFactory::new());
let state = SessionStateBuilder::new()
    .with_default_features()
    .with_table_factory(
        factory.get_ext().to_uppercase(),
        Arc::new(DefaultTableFactory::new()),
    )
    .with_file_formats(vec![factory])
    .build();
let ctx = SessionContext::new_with_state(state).enable_url_table();

Create an external table and query it:

ctx.sql(
    "CREATE EXTERNAL TABLE my_table \
            (name VARCHAR NOT NULL, age INT NOT NULL) \
        STORED AS vortex \
        LOCATION '/demo/'",
)
.await?;

You can also register a ListingTable directly:

let ctx = SessionContext::new();
let format = Arc::new(VortexFormat::new(session));
let table_url = ListingTableUrl::parse(
    filepath
        .to_str()
        .ok_or_else(|| vortex_err!("Path is not valid UTF-8"))?,
)?;
let config = ListingTableConfig::new(table_url)
    .with_listing_options(ListingOptions::new(format))
    .infer_schema(&ctx.state())
    .await?;

let listing_table = Arc::new(ListingTable::try_new(config)?);
ctx.register_table("vortex_tbl", listing_table as _)?;

Write query results to Vortex using INSERT INTO:

ctx.sql(
    "INSERT INTO my_table VALUES \
            ('Alice', 30), ('Bob', 25), ('Charlie', 35), ('Diana', 28)",
)
.await?
.collect()
.await?;

Registration¶

Vortex registers itself through DataFusion’s FileFormatFactory interface. Once registered, DataFusion can create ListingTable providers that automatically discover .vortex files in a directory or object store prefix. The format factory creates a VortexFormat instance that carries the Vortex session and its associated options (segment cache, read sizes, etc.).

Schema inference reads the footer of one of the discovered files to extract the Arrow schema. DataFusion then uses this schema when planning queries against the table.

Multiple Files¶

DataFusion handles multi-file scans through its own file listing and partitioning layer. Each discovered file becomes a PartitionedFile that DataFusion assigns to execution partitions. Vortex implements the FileOpener trait to open individual files on demand as DataFusion’s executor schedules them.

Layout readers are cached across partitions using a shared concurrent map keyed by file path. This avoids redundant footer parsing when the same file is accessed by multiple partitions or repeated queries.

Filter and Projection Pushdown¶

The integration converts DataFusion physical expressions into Vortex expressions using an ExpressionConverter trait. Supported predicates (comparisons, LIKE, NOT, IS NULL, IN lists, casts) are pushed into the Vortex scan where they participate in pruning and filter evaluation at the layout level. Unsupported predicates remain in the DataFusion plan and are evaluated after the scan.

Filter pushdown operates at two levels. The full predicate is used to prune entire files before they are opened, using file-level statistics. The subset of predicates that Vortex can evaluate efficiently is pushed into the per-file scan for row-level filtering.

Fully supported projections are evaluated in the scan. If any expression is unsupported, the complete projection runs in DataFusion over the required raw columns.

Searched CASE WHEN expressions can be pushed down when all their children can be converted, including branches that may fail. Conditions and values execute only on the rows that reach them, so guards such as CASE WHEN divisor <> 0 THEN value / divisor ELSE 0 END remain safe. Simple CASE value WHEN ... expressions currently remain in DataFusion.

The integration supports pluggable expression conversion via a custom ExpressionConverter, allowing engine-specific rewrites or schema adaptation when file schemas diverge from the table schema.

Data Export¶

Vortex arrays produced by the scan are converted to Arrow RecordBatches for consumption by DataFusion. Batches are sliced to respect DataFusion’s configured batch size preference.