Package org.elasticsearch.xpack.esql.datasources.spi

Interface RecordSplitter

public interface RecordSplitter
Finds format-specific record boundaries for splittable row-oriented external data sources.

Implementations are used in two different contexts: stream-based planning code that skips forward to the next complete record, and byte-array chunking code that slices an already-read buffer at the last complete record.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final long
    AMBIGUOUS
    Returned only by findProvenRecordBoundary(InputStream) when a bounded probe cannot prove a record start within its convergence window (e.g.
    static final long
    RECORD_TOO_LARGE
    Returned by findNextRecordBoundary(InputStream) when a scanner exceeds maxRecordBytes() before finding a record boundary.

  • Method Summary

    Modifier and Type
    Method
    Description
    default int
    findLastRecordBoundary(byte[] buf, int length)
    Returns the index of the byte that terminates the last complete record within buf[0..length), or -1 if no complete record terminates inside that range.
    int
    findLastRecordBoundary(byte[] buf, int offset, int length)
    Returns the index of the byte that terminates the last complete record within buf[offset..offset + length), or -1 if no complete record terminates inside that range.
    long
    findNextRecordBoundary(InputStream stream)
    Scans forward from the current stream position until the next complete record boundary.
    default long
    findProvenRecordBoundary(InputStream stream)
    Probes for a proven record start starting from the current stream position (an arbitrary mid-file stride offset).
    default long
    findRecordStartAtOrAfter(InputStream stream, long minSkip, BooleanSupplier isCancelled)
    Exact forward walk from a stream opened at a known record start, keeping cross-record parse state (e.g.
    int
    maxRecordBytes()
    Maximum bytes a single record may occupy before the splitter reports RECORD_TOO_LARGE.
    default boolean
    supportsProvenProbing()
    Whether this splitter can prove a record start at an arbitrary mid-file offset even though it is not strided (i.e.
    default boolean
    supportsStridedProbing()
    Whether it is safe to begin a boundary probe at an arbitrary mid-file offset (as the stride-based segmentation in FileSplitProvider and ParallelParsingCoordinator does).

  • Field Details

  • Method Details

    • findNextRecordBoundary

      long findNextRecordBoundary(InputStream stream) throws IOException
      Scans forward from the current stream position until the next complete record boundary.
      Returns:
      the number of bytes consumed, -1 if EOF is reached before a boundary, or RECORD_TOO_LARGE if the next record exceeds maxRecordBytes()
      Throws:
      IOException

    • findLastRecordBoundary

      int findLastRecordBoundary(byte[] buf, int offset, int length) throws IOException
      Returns the index of the byte that terminates the last complete record within buf[offset..offset + length), or -1 if no complete record terminates inside that range.

      When the range ends inside an open record, implementations must return the previous complete record boundary, not a byte inside the open tail. Implementations that enforce maxRecordBytes() may return (int) RECORD_TOO_LARGE when no boundary was found and the open record already exceeds the cap, so callers fail fast instead of growing the buffer further.

      Throws:
      IOException

    • findLastRecordBoundary

      default int findLastRecordBoundary(byte[] buf, int length) throws IOException
      Returns the index of the byte that terminates the last complete record within buf[0..length), or -1 if no complete record terminates inside that range.
      Throws:
      IOException

    • maxRecordBytes

      int maxRecordBytes()
      Maximum bytes a single record may occupy before the splitter reports RECORD_TOO_LARGE.

    • supportsStridedProbing

      default boolean supportsStridedProbing()
      Whether it is safe to begin a boundary probe at an arbitrary mid-file offset (as the stride-based segmentation in FileSplitProvider and ParallelParsingCoordinator does).

      This holds only when every record terminator is unambiguous from the byte immediately at it, i.e. a raw newline is always a true record boundary regardless of the state the scan started in. It is false for splitters whose grammar can carry a record across a raw newline (quoted fields with embedded newlines, bracketed multi-value cells): starting a probe inside such a construct misreads an in-construct newline as a terminator and desyncs the parse. Callers that get false must not split at arbitrary offsets; they read the file as one sequential stream instead.

    • supportsProvenProbing

      default boolean supportsProvenProbing()
      Whether this splitter can prove a record start at an arbitrary mid-file offset even though it is not strided (i.e. supportsStridedProbing() is false because its grammar can carry a record across a raw newline). This is the second capability boolean on the SPI, orthogonal to supportsStridedProbing(): a splitter that returns true here restores cross-node macro splitting for quoted/escaped text by running a bounded findProvenRecordBoundary(InputStream) probe and falling back to a monotonic findRecordStartAtOrAfter(InputStream, long, BooleanSupplier) exact walk, both of which yield only true record starts, so a split boundary is never cut inside a quoted field or an escaped newline.

      Callers must check this before calling findProvenRecordBoundary(InputStream) or findRecordStartAtOrAfter(InputStream, long, BooleanSupplier), exactly as they check supportsStridedProbing() before findNextRecordBoundary(InputStream) on the strided path.

    • findProvenRecordBoundary

      default long findProvenRecordBoundary(InputStream stream) throws IOException
      Probes for a proven record start starting from the current stream position (an arbitrary mid-file stride offset). Returns the byte count consumed through the record terminator at which the probe proved a record start (mirroring findNextRecordBoundary(InputStream)'s consumed return, so the caller adds it to the stream's base offset), or AMBIGUOUS if it could not prove one within its bounded convergence window. Never returns RECORD_TOO_LARGE: window exhaustion is always AMBIGUOUS, deferring the cap verdict to findRecordStartAtOrAfter(InputStream, long, BooleanSupplier).

      The default implementation throws so a splitter that advertises supportsProvenProbing() but forgets to override this fails loud rather than silently delegating to a quote-unaware scan.

      Throws:
      IOException

    • findRecordStartAtOrAfter

      default long findRecordStartAtOrAfter(InputStream stream, long minSkip, BooleanSupplier isCancelled) throws IOException
      Exact forward walk from a stream opened at a known record start, keeping cross-record parse state (e.g. in-quote) across records in a single pass, returning the stream-relative offset of the first record start at or after minSkip. minSkip is stream-relative and always > 0, so the returned offset is strictly greater than the stream's base record start (never 0). Returns RECORD_TOO_LARGE if a single record exceeds maxRecordBytes() before a boundary, or -1 at EOF before reaching minSkip.

      isCancelled is polled inside the byte loop so a long walk (a record up to maxRecordBytes()) aborts promptly; a cancelled walk throws TaskCancelledException. Read-time callers that do not carry a cancellable task pass () -> false.

      The default implementation throws so a splitter that advertises supportsProvenProbing() but forgets to override this fails loud.

      Throws:
      IOException