Package org.elasticsearch.xpack.esql.datasources.spi

Interface RangeAwareFormatReader

All Superinterfaces:

AutoCloseable, Closeable, FormatReader

public interface RangeAwareFormatReader
extends FormatReader

Extension of FormatReader for columnar formats (Parquet, ORC) that support row-group-level split parallelism.

Unlike line-oriented formats that use SegmentableFormatReader with byte-range views, columnar formats need full-file access (e.g. Parquet's footer is at EOF) but can selectively read row groups within a byte range. This interface separates the range specification from the file access, allowing the reader to open the full file and use format-native range filtering (e.g. ParquetReadOptions.withRange()).

The split discovery flow:

1. discoverSplitRanges(org.elasticsearch.xpack.esql.datasources.spi.StorageObject) reads file metadata (e.g. Parquet footer) and returns byte ranges for each independently readable unit (e.g. row group).
2. The framework creates one split per range, distributed across drivers/nodes.
3. At read time, readRange(org.elasticsearch.xpack.esql.datasources.spi.StorageObject, org.elasticsearch.xpack.esql.datasources.spi.RangeReadContext) receives the full-file object and the assigned byte range, reading only the relevant row groups.

Nested Class Summary

Nested Classes

Modifier and Type	Interface	Description
static final record	RangeAwareFormatReader.SplitRange	A byte range within a file with optional per-range statistics (e.g.
static final record	RangeAwareFormatReader.SplitRef	Per-split descriptor passed to readAll(java.util.List<org.elasticsearch.xpack.esql.datasources.spi.RangeAwareFormatReader.SplitRef>, java.util.List<java.lang.String>, int).

Nested classes/interfaces inherited from interface org.elasticsearch.xpack.esql.datasources.spi.FormatReader

FormatReader.SchemaResolution

Field Summary

Fields inherited from interface org.elasticsearch.xpack.esql.datasources.spi.FormatReader

DEFAULT_SCHEMA_RESOLUTION, NO_LIMIT

Method Summary

Modifier and Type	Method	Description
List<RangeAwareFormatReader.SplitRange>	discoverSplitRanges(StorageObject object)	Discovers independently readable byte ranges within a file by reading its metadata.
default CloseableIterator<Page>	readAll(List<RangeAwareFormatReader.SplitRef> splits, List<String> projectedColumns, int batchSize)	Reads all given objects in a single batched call, returning a unified page iterator.
CloseableIterator<Page>	readRange(StorageObject object, RangeReadContext context)	Reads only the row groups / stripes that fall within the given byte range.
default boolean	supportsBatchRead()	Returns true if this reader supports batch multi-file reads via readAll(java.util.List<org.elasticsearch.xpack.esql.datasources.spi.RangeAwareFormatReader.SplitRef>, java.util.List<java.lang.String>, int).

Methods inherited from interface java.io.Closeable

close

Methods inherited from interface org.elasticsearch.xpack.esql.datasources.spi.FormatReader

aggregatePushdownSupport, defaultErrorPolicy, defaultSchemaResolution, fileExtensions, filterPushdownSupport, formatName, metadata, metadataAsync, read, read, readAsync, rowPositionStrategy, schema, statusSnapshot, supportsNativeAsync, supportsWholeFileCompression, withConfig, withConfigTrackingConsumedKeys, withDeclaredDateFormats, withDeclaredTypeColumns, withPushedFilter, withSchema

Method Details

discoverSplitRanges

List<RangeAwareFormatReader.SplitRange> discoverSplitRanges(StorageObject object)
                                                     throws IOException

Discovers independently readable byte ranges within a file by reading its metadata. Each range typically corresponds to one row group (Parquet) or stripe (ORC).

Returns a list of RangeAwareFormatReader.SplitRange objects. An empty list means the file cannot be split (e.g. single row group) and should be read as a whole.

Parameters:

object - the storage object representing the full file

Returns:

list of split ranges with optional per-range statistics

Throws:

IOException

readRange

CloseableIterator<Page> readRange(StorageObject object,
 RangeReadContext context)
                           throws IOException

Reads only the row groups / stripes that fall within the given byte range. The storage object must represent the full file (not a range-limited view), because columnar formats need access to file-level metadata (e.g. footer).

Parameters:

object - the full-file storage object

context - per-split read parameters and optional cross-split file context

Returns:

an iterator that yields pages from the matching row groups

Throws:

IOException

supportsBatchRead

default boolean supportsBatchRead()

Returns true if this reader supports batch multi-file reads via readAll(java.util.List<org.elasticsearch.xpack.esql.datasources.spi.RangeAwareFormatReader.SplitRef>, java.util.List<java.lang.String>, int). When supported the execution framework calls readAll(java.util.List<org.elasticsearch.xpack.esql.datasources.spi.RangeAwareFormatReader.SplitRef>, java.util.List<java.lang.String>, int) for a batch of files instead of calling readRange(org.elasticsearch.xpack.esql.datasources.spi.StorageObject, org.elasticsearch.xpack.esql.datasources.spi.RangeReadContext) once per split, allowing the reader to process multiple files concurrently in a single call.

Only enabled when there are no per-file virtual partition columns (those require per-split injection that is incompatible with a unified batch iterator). The default implementation returns false.

readAll

default CloseableIterator<Page> readAll(List<RangeAwareFormatReader.SplitRef> splits,
 List<String> projectedColumns,
 int batchSize)
                                 throws IOException

Reads all given objects in a single batched call, returning a unified page iterator. Called by the framework instead of readRange(org.elasticsearch.xpack.esql.datasources.spi.StorageObject, org.elasticsearch.xpack.esql.datasources.spi.RangeReadContext) when supportsBatchRead() returns true.

The reader is responsible for applying any pushed filters and projections internally. The returned iterator may interleave pages from different files.

Parameters:

splits - per-split descriptors; each carries a storage object plus the byte range [offset, offset+length) that identifies which row groups within the file belong to this split

projectedColumns - columns to project, or null for all columns

batchSize - target page size in rows

Throws:

IOException