Package org.elasticsearch.xpack.esql.datasources.spi

Record Class FormatReadContext

java.lang.Object
java.lang.Record
org.elasticsearch.xpack.esql.datasources.spi.FormatReadContext
Record Components:
projectedColumns - columns to read. null means "no projection info available — read every column" (backward compatibility default). An empty list means "the optimizer pruned every column" (e.g. COUNT(*)); format readers may take a fast path that skips type conversion and emits row- count-only Pages.
batchSize - target number of rows per page
rowLimit - maximum total rows to return (FormatReader.NO_LIMIT for unlimited)
errorPolicy - how to handle malformed rows
firstSplit - whether this is the first split for the file (consistent with lastSplit; format-agnostic replacement for the legacy skipFirstLine parameter)
lastSplit - whether this is the last split for the file (affects trailing-record handling)
recordAligned - whether the split starts at a record boundary (no leading partial record). When false, line-oriented readers may need to skip a leading partial line (e.g. bzip2 / zstd-indexed macro-splits). When true, the split is known to start exactly on a record boundary (e.g. streaming-parallel chunks sliced on \n). Has no effect on the first split.
readSchema - optional planner-resolved positional column layout. When non-null, format readers use it as the authoritative typed schema; when null, readers fall back to per-file inference. Distinct from FormatReader.withSchema(java.util.List<org.elasticsearch.xpack.esql.core.expression.Attribute>), which carries the projection. Empty list and null both mean "no schema"; the compact constructor collapses empty to null so readers do one check.
splitStartByte - file-global byte offset at which this split begins (i.e. FileSplit.offset()). Text readers add the bytes they consume to this anchor to emit a file-global, split-invariant start byte per record for the _rowPosition channel (the substrate of _file.record_ref / _id). 0 for the whole-file (non-split) case and for columnar formats, which derive a file-global row index from their own footer/stripe metadata rather than from a byte anchor.

Note: this carries the SAME VALUE as statsBaseOffset at every current call site, but they are distinct CONTRACTS, not one quantity: splitStartByte may be a COMPRESSED coordinate under compressed-offset splits (COMPRESSED_OFFSET_SPLIT_KEY), whereas statsBaseOffset is decompressed-stream only. Stripe capture is disabled exactly where the two would diverge, so they must stay separate — a blind unification would mis-address stripes on compressed splits.

maxRecordBytes - maximum bytes a single text record may occupy while split/trim code scans for a record boundary.
statsBaseOffset - the DECOMPRESSED-stream byte offset of this read's first byte, used by the reader to address records to canonical stripes (ordinal = floor((statsBaseOffset + recordOffsetInRead) / statsStripeSize)). Ignored when statsStripeSize <= 0. Same value as splitStartByte at every current call site, but a distinct contract under compressed-offset splits (see its note).
statsStripeSize - canonical-stripe grid in bytes for per-stripe stats attribution, or <= 0 to disable stripe addressing (the reader then emits no stripe-addressed contributions and the warm short-circuit safe-misses). A pure stats overlay — it never affects how the read is chunked or split.
statsFileFinal - whether this read reaches the file's true end (the segmentator's EOF chunk, or the segmented coordinator's trailing segment). Only the file-final read may mark its last stripe complete-on-the-right (atStripeEnd) and terminal (eof); a non-final chunk ends mid-stripe at a chunk boundary, so its trailing stripe is a partial right fragment the next chunk continues. Marking a non-final chunk's trailing stripe complete would silently undercount.
statsColumnScope - how much per-stripe statistics the read harvests while it scans (row count only / row count + projected columns / row count + all file columns / nothing). Orthogonal to statsStripeSize: the grid decides which stripe a record lands in, this decides what is summarised per stripe. null defaults to StripeColumnScope.PROJECTED (back-compat for call sites that predate the setting); the compact constructor collapses null to that default so readers do one check.
informationalWarningSink - optional relay for client-visible lenient-policy warnings (see SkipWarnings) raised while reading. null means the reader should fall back to emitting warnings directly via HeaderWarning, which is only correct when the read runs on the request/driver thread. Callers that dispatch reads to a background thread (e.g. AsyncExternalSourceOperatorFactory) must set this to a sink — typically AsyncExternalSourceBuffer::recordInformationalWarning, preserving these warnings' pre-existing behavior of never flipping the response's is_partial flag (see AsyncExternalSourceBuffer#recordWarning for the one warning that does) — so the warning is relayed back and re-emitted on the correct thread instead of being silently dropped.
public record FormatReadContext(List<String> projectedColumns, int batchSize, int rowLimit, ErrorPolicy errorPolicy, boolean firstSplit, boolean lastSplit, boolean recordAligned, List<Attribute> readSchema, long splitStartByte, int maxRecordBytes, long statsBaseOffset, long statsStripeSize, boolean statsFileFinal, StripeColumnScope statsColumnScope, Consumer<String> informationalWarningSink) extends Record
Immutable context for a single FormatReader.read(org.elasticsearch.xpack.esql.datasources.spi.StorageObject, org.elasticsearch.xpack.esql.datasources.spi.FormatReadContext) or FormatReader.readAsync(org.elasticsearch.xpack.esql.datasources.spi.StorageObject, org.elasticsearch.xpack.esql.datasources.spi.FormatReadContext, java.util.concurrent.Executor, org.elasticsearch.action.ActionListener<org.elasticsearch.compute.operator.CloseableIterator<org.elasticsearch.compute.data.Page>>) call. Bundles all per-read execution parameters that were previously spread across 12+ method overloads.

Format-specific configuration (delimiter, encoding, etc.) lives on the reader instance via FormatReader.withConfig(java.util.Map<java.lang.String, java.lang.Object>). Per-query optimizer hints (pushed filters) live on the reader instance via FormatReader.withPushedFilter(java.lang.Object). This context carries only the parameters that may vary per file or per split within a single query execution.

  • Constructor Details

    • FormatReadContext

      public FormatReadContext(List<String> projectedColumns, int batchSize, int rowLimit, ErrorPolicy errorPolicy, boolean firstSplit, boolean lastSplit, boolean recordAligned, @Nullable List<Attribute> readSchema, long splitStartByte, int maxRecordBytes, long statsBaseOffset, long statsStripeSize, boolean statsFileFinal, StripeColumnScope statsColumnScope, @Nullable Consumer<String> informationalWarningSink)
      Creates an instance of a FormatReadContext record class.
      Parameters:
      projectedColumns - the value for the projectedColumns record component
      batchSize - the value for the batchSize record component
      rowLimit - the value for the rowLimit record component
      errorPolicy - the value for the errorPolicy record component
      firstSplit - the value for the firstSplit record component
      lastSplit - the value for the lastSplit record component
      recordAligned - the value for the recordAligned record component
      readSchema - the value for the readSchema record component
      splitStartByte - the value for the splitStartByte record component
      maxRecordBytes - the value for the maxRecordBytes record component
      statsBaseOffset - the value for the statsBaseOffset record component
      statsStripeSize - the value for the statsStripeSize record component
      statsFileFinal - the value for the statsFileFinal record component
      statsColumnScope - the value for the statsColumnScope record component
      informationalWarningSink - the value for the informationalWarningSink record component

  • Method Details

    • of

      public static FormatReadContext of(List<String> projectedColumns, int batchSize)
      Creates a minimal context for the common non-split case. Leaves errorPolicy as null so the reader falls back to its own default — typically the policy resolved from the user's WITH options via FormatReader.withConfig(java.util.Map<java.lang.String, java.lang.Object>), or the FormatReader.defaultErrorPolicy() when no user options are set. Callers that need to override the policy should use builder() or withErrorPolicy(ErrorPolicy).

    • withRowLimit

      public FormatReadContext withRowLimit(int limit)
      Returns a copy with a different row limit.

    • withErrorPolicy

      public FormatReadContext withErrorPolicy(ErrorPolicy policy)
      Returns a copy with a different error policy.

    • withSplit

      public FormatReadContext withSplit(boolean first, boolean last)
      Returns a copy configured for a split-based read.

    • builder

      public static FormatReadContext.Builder builder()

    • toString

      public final String toString()
      Returns a string representation of this record class. The representation contains the name of the class, followed by the name and value of each of the record components.
      Specified by:
      toString in class Record
      Returns:
      a string representation of this object

    • hashCode

      public final int hashCode()
      Returns a hash code value for this object. The value is derived from the hash code of each of the record components.
      Specified by:
      hashCode in class Record
      Returns:
      a hash code value for this object

    • equals

      public final boolean equals(Object o)
      Indicates whether some other object is "equal to" this one. The objects are equal if the other object is of the same class and if all the record components are equal. Reference components are compared with Objects::equals(Object,Object); primitive components are compared with '=='.
      Specified by:
      equals in class Record
      Parameters:
      o - the object with which to compare
      Returns:
      true if this object is the same as the o argument; false otherwise.

    • projectedColumns

      public List<String> projectedColumns()
      Returns the value of the projectedColumns record component.
      Returns:
      the value of the projectedColumns record component

    • batchSize

      public int batchSize()
      Returns the value of the batchSize record component.
      Returns:
      the value of the batchSize record component

    • rowLimit

      public int rowLimit()
      Returns the value of the rowLimit record component.
      Returns:
      the value of the rowLimit record component

    • errorPolicy

      public ErrorPolicy errorPolicy()
      Returns the value of the errorPolicy record component.
      Returns:
      the value of the errorPolicy record component

    • firstSplit

      public boolean firstSplit()
      Returns the value of the firstSplit record component.
      Returns:
      the value of the firstSplit record component

    • lastSplit

      public boolean lastSplit()
      Returns the value of the lastSplit record component.
      Returns:
      the value of the lastSplit record component

    • recordAligned

      public boolean recordAligned()
      Returns the value of the recordAligned record component.
      Returns:
      the value of the recordAligned record component

    • readSchema

      @Nullable public List<Attribute> readSchema()
      Returns the value of the readSchema record component.
      Returns:
      the value of the readSchema record component

    • splitStartByte

      public long splitStartByte()
      Returns the value of the splitStartByte record component.
      Returns:
      the value of the splitStartByte record component

    • maxRecordBytes

      public int maxRecordBytes()
      Returns the value of the maxRecordBytes record component.
      Returns:
      the value of the maxRecordBytes record component

    • statsBaseOffset

      public long statsBaseOffset()
      Returns the value of the statsBaseOffset record component.
      Returns:
      the value of the statsBaseOffset record component

    • statsStripeSize

      public long statsStripeSize()
      Returns the value of the statsStripeSize record component.
      Returns:
      the value of the statsStripeSize record component

    • statsFileFinal

      public boolean statsFileFinal()
      Returns the value of the statsFileFinal record component.
      Returns:
      the value of the statsFileFinal record component

    • statsColumnScope

      public StripeColumnScope statsColumnScope()
      Returns the value of the statsColumnScope record component.
      Returns:
      the value of the statsColumnScope record component

    • informationalWarningSink

      @Nullable public Consumer<String> informationalWarningSink()
      Returns the value of the informationalWarningSink record component.
      Returns:
      the value of the informationalWarningSink record component