Package org.elasticsearch.xpack.esql.datasources.spi

Interface SegmentableFormatReader

All Superinterfaces:

AutoCloseable, Closeable, FormatReader

public interface SegmentableFormatReader
extends FormatReader

Extension of FormatReader for line-oriented text formats (CSV, NDJSON) that support intra-file parallel parsing.

Formats that implement this interface declare they can find record boundaries within an arbitrary byte stream, enabling the framework to split a single file into byte-range segments and parse them concurrently on multiple threads.

Columnar formats (Parquet, ORC) should not implement this interface — they have row-group-level parallelism instead.

Nested Class Summary

Nested classes/interfaces inherited from interface org.elasticsearch.xpack.esql.datasources.spi.FormatReader

FormatReader.SchemaResolution

Field Summary

Fields

Modifier and Type	Field	Description
static final int	DEFAULT_MAX_RECORD_BYTES	Default cap on the bytes a single record may occupy; the streaming splitter fails the query rather than buffering past this when a scanner cannot find a boundary.

Fields inherited from interface org.elasticsearch.xpack.esql.datasources.spi.FormatReader

DEFAULT_SCHEMA_RESOLUTION, NO_LIMIT

Method Summary

Modifier and Type	Method	Description
default long	minimumSegmentSize()	Returns the minimum segment size in bytes below which splitting is not worthwhile.
default RecordSplitter	recordSplitter()	Returns the record-boundary splitter for this reader.
RecordSplitter	recordSplitter(int maxRecordBytes)	Returns the record-boundary splitter with a caller-supplied record-size cap.

Methods inherited from interface java.io.Closeable

close

Methods inherited from interface org.elasticsearch.xpack.esql.datasources.spi.FormatReader

aggregatePushdownSupport, defaultErrorPolicy, defaultSchemaResolution, fileExtensions, filterPushdownSupport, formatName, metadata, metadataAsync, read, read, readAsync, rowPositionStrategy, schema, statusSnapshot, supportsNativeAsync, supportsWholeFileCompression, withConfig, withConfigTrackingConsumedKeys, withDeclaredDateFormats, withDeclaredTypeColumns, withPushedFilter, withSchema

Field Details

DEFAULT_MAX_RECORD_BYTES

static final int DEFAULT_MAX_RECORD_BYTES

Default cap on the bytes a single record may occupy; the streaming splitter fails the query rather than buffering past this when a scanner cannot find a boundary. Overridable via the max_record_size pragma.

See Also:

• Constant Field Values

Method Details

recordSplitter

default RecordSplitter recordSplitter()

Returns the record-boundary splitter for this reader.

recordSplitter

RecordSplitter recordSplitter(int maxRecordBytes)

Returns the record-boundary splitter with a caller-supplied record-size cap. Implementations report RecordSplitter.RECORD_TOO_LARGE when a record exceeds maxRecordBytes.

minimumSegmentSize

default long minimumSegmentSize()

Returns the minimum segment size in bytes below which splitting is not worthwhile. Segments smaller than this will be merged with an adjacent segment.

Defaults to 1 MiB. ClickHouse benchmarks show 1 MiB chunks are optimal for parallel parsing — 100 KB chunks are ~40% slower due to per-chunk overhead, while 10 MiB chunks offer only marginal improvement. Implementations may override to reflect their parsing overhead.