Package org.elasticsearch.xpack.esql.datasources.spi

Interface ColumnExtractor

All Superinterfaces:
AutoCloseable, Closeable, Releasable
public interface ColumnExtractor extends Releasable
Loads one projected column by physical row identity.

Deferred extraction keeps the forward scan narrow (sort keys, predicates, and the synthetic routing column) and materializes wide projection-only columns only after a per-driver TopN has chosen surviving rows.

Addressing space (physical row identity)

The positions accepted by extract(java.lang.String[], org.elasticsearch.xpack.esql.core.type.DataType[], long[], org.elasticsearch.compute.data.BlockFactory) are row identities: stable, opaque longs the iterator that emits ROW_POSITION_COLUMN attached to each row. The extractor and the iterator must agree on the encoding — for the Parquet implementation, it is the file-global row index in footer iteration order, which is filter-, late-materialization-, and page-skip-invariant. This makes the handshake robust to whatever survives between scan and extract: the same row coming back later carries the same identity that pointed at it during the forward scan, no matter how many predicate / page-skipping transforms ran in between.

The handshake between an iterator and its matching extractor is expressed by ColumnExtractorProducer: an iterator that emits ROW_POSITION_COLUMN implements that interface and produces an extractor whose addressing space matches its own.

Composition across multiple sources opened in the same driver — assigning extractor ids, encoding references, and dispatching extractions — is handled by SourceExtractors (org.elasticsearch.xpack.esql.datasources.SourceExtractors), not by this SPI.

Threading: a single driver thread owns an instance; implementations need not be thread-safe.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final int
    LOCAL_POSITION_BITS
    Bit position of the extractor id within an encoded _rowPosition value.
    static final String
    ROW_POSITION_COLUMN
    Reserved name of the synthetic row-reference column emitted by source readers that support deferred materialization.

  • Method Summary

    Modifier and Type
    Method
    Description
    Block[]
    extract(String[] columnNames, DataType[] targetTypes, long[] positions, BlockFactory blockFactory)
    Materializes one or more columns at the given row identities in a single I/O batch.
    default Block
    extract(String columnName, long[] positions, BlockFactory blockFactory)
    Convenience overload for the common single-column case.
    long
    rowCount()
    Number of distinct row identities addressable by this extractor.

    Methods inherited from interface org.elasticsearch.core.Releasable

    close

  • Field Details

    • LOCAL_POSITION_BITS

      static final int LOCAL_POSITION_BITS
      Bit position of the extractor id within an encoded _rowPosition value. The low LOCAL_POSITION_BITS hold the per-extractor physical row identity; the high bits hold the registry-assigned extractor id. This is the wire-format the iterator implementing ColumnExtractorProducer must respect when it pre-encodes its emitted values.

      The framework's SourceExtractors owns the canonical encoding/decoding helpers and pins this same value; the constant is mirrored here so format readers depend only on the SPI when implementing the encoding handshake.

      See Also:

    • ROW_POSITION_COLUMN

      static final String ROW_POSITION_COLUMN
      Reserved name of the synthetic row-reference column emitted by source readers that support deferred materialization. Values are opaque encoded references combining an extractor id with a physical row identity — see org.elasticsearch.xpack.esql.datasources.SourceExtractors for the bit layout.

      Encoding lives at the iterator boundary: the source factory wires an extractor id into the iterator via ColumnExtractorProducer.setExtractorId(int), and from then on every _rowPosition value the iterator emits is already encoded as (id << 48) | physicalRowOffset. Downstream operators decode without a separate encoding pass. The extract operator removes this channel from its output.

      Historically the encoding ran in a wrapper iterator that rebuilt the _rowPosition block per page. That wrapper was removed once iterators began materialising the values themselves; if a future format reader cannot pre-encode in-line, re-introducing a similar wrapper is a viable (if slower) fallback — the contract surfaces here so the option remains discoverable.

      See Also:

  • Method Details

    • rowCount

      long rowCount()
      Number of distinct row identities addressable by this extractor.

      The valid range for extract(java.lang.String[], org.elasticsearch.xpack.esql.core.type.DataType[], long[], org.elasticsearch.compute.data.BlockFactory)'s positions entries is [0, rowCount()). For Parquet this is the file's total row count.

      Implementations should return a pure in-memory value without triggering I/O.

      Returns:
      number of rows addressable by this extractor

    • extract

      Block[] extract(String[] columnNames, @Nullable DataType[] targetTypes, long[] positions, BlockFactory blockFactory) throws IOException
      Materializes one or more columns at the given row identities in a single I/O batch.

      Callers invoke this after TopN with only the identities that survived pruning. Identities may be unsorted and may repeat; the returned blocks must align row i with positions[i] so operators can zip deferred values with encoded references without an extra reordering pass. Each entry must lie in [0, rowCount()).

      Implementations should fetch every requested column's bytes in one coalesced batch so that latency and per-request overhead are amortised across the projection. For object-store backends like S3 the win is significant: the cost dominator is the per-request RTT/TTFB, so issuing one fan-out batch covering all columns instead of N sequential batches turns a O(N) round-trip wait into O(1). For physical layouts where columns within a row group are contiguous (e.g. Parquet column chunks), the bytes for multiple columns also coalesce naturally during merging.

      Returned blocks are owned by the caller; on partial failure, the implementation must release any blocks it already built before propagating the exception.

      targetTypes carries the planner/declared type per column so extraction honors the same declared-type coercion the eager decode paths run (DeclaredTypeCoercions): a column whose file type differs from its target coerces value-by-value (per-value failures null the cell and emit a response Warning header, bulk-API style). null — the whole array or an entry — means "emit the file's own type" (no coercion).

      Parameters:
      columnNames - logical columns to load, in output order; must not contain ROW_POSITION_COLUMN
      targetTypes - planner/declared type per column, aligned with columnNames; null (array or entry) disables coercion for that column
      positions - row identities; every element is read
      blockFactory - factory for breaker-aware allocation
      Returns:
      one block per requested column, each with exactly positions.length values in caller order
      Throws:
      IOException - if the format reader fails while satisfying the request

    • extract

      default Block extract(String columnName, long[] positions, BlockFactory blockFactory) throws IOException
      Convenience overload for the common single-column case. Delegates to extract(String[], DataType[], long[], BlockFactory) so implementations only need to provide one code path.
      Throws:
      IOException