Package dev.vortex.spark.write

Class VortexOutputWriter

java.lang.Object

org.apache.spark.sql.execution.datasources.OutputWriter

dev.vortex.spark.write.VortexOutputWriter

public final class VortexOutputWriter
extends org.apache.spark.sql.execution.datasources.OutputWriter

Writes Spark InternalRow data to a Vortex file.

Each row is appended straight into a batch of Arrow vectors as it arrives, and a full batch is handed to the Vortex writer.

Field Summary

Fields

Modifier and Type	Field	Description
static final String	BATCH_SIZE_OPTION	Option sizing the row batch converted to Arrow and handed to Vortex at a time.
static final String	WRITE_BATCH_SIZE_OPTION	Option sizing the write batch for Vortex alone, overriding "batch.size".

Constructor Summary

Constructors

Constructor	Description
VortexOutputWriter(String filePath, org.apache.spark.sql.types.StructType schema, VortexOptions options, dev.vortex.io.NativeWritable writable)	Creates a writer for the task path assigned by Spark's commit protocol.

Method Summary

Modifier and Type	Method	Description
void	close()	Flushes the last batch and finalizes the file.
String	path()
void	write(org.apache.spark.sql.catalyst.InternalRow row)	Writes a single row to the Vortex file.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

BATCH_SIZE_OPTION

public static final String BATCH_SIZE_OPTION

Option sizing the row batch converted to Arrow and handed to Vortex at a time.

See Also:

• Constant Field Values

WRITE_BATCH_SIZE_OPTION

public static final String WRITE_BATCH_SIZE_OPTION

Option sizing the write batch for Vortex alone, overriding "batch.size".

See Also:

• Constant Field Values

Constructor Details

VortexOutputWriter

public VortexOutputWriter(String filePath,
 org.apache.spark.sql.types.StructType schema,
 VortexOptions options,
 dev.vortex.io.NativeWritable writable)

Creates a writer for the task path assigned by Spark's commit protocol.

Parameters:

filePath - the path where the Vortex file will be written

schema - the schema of the data to write

options - additional write options

Method Details

write

public void write(org.apache.spark.sql.catalyst.InternalRow row)

Writes a single row to the Vortex file.

The row is consumed before this method returns: Spark reuses the row object for the next call.

Specified by:

write in class org.apache.spark.sql.execution.datasources.OutputWriter

Parameters:

row - the row to write

close

public void close()

Flushes the last batch and finalizes the file.

After a failed write(org.apache.spark.sql.catalyst.InternalRow), or in a task that has already failed, the last batch is dropped because the file is never committed. In both cases this releases everything and logs what fails instead of throwing, because Spark's abort closes a task's writers one after another and stops at the first that throws.

Specified by:

close in class org.apache.spark.sql.execution.datasources.OutputWriter

path

public String path()

Specified by:

path in class org.apache.spark.sql.execution.datasources.OutputWriter