Package org.elasticsearch.xpack.esql.optimizer.rules.physical.local

Class InjectRowPositionForExternalId

java.lang.Object
org.elasticsearch.xpack.esql.rule.Rule<ExternalSourceExec,PhysicalPlan>
org.elasticsearch.xpack.esql.rule.ParameterizedRule<ExternalSourceExec,PhysicalPlan,LocalPhysicalOptimizerContext>
org.elasticsearch.xpack.esql.optimizer.PhysicalOptimizerRules.ParameterizedOptimizerRule<ExternalSourceExec,LocalPhysicalOptimizerContext>
org.elasticsearch.xpack.esql.optimizer.rules.physical.local.InjectRowPositionForExternalId
public class InjectRowPositionForExternalId extends PhysicalOptimizerRules.ParameterizedOptimizerRule<ExternalSourceExec,LocalPhysicalOptimizerContext>
Inject the synthetic ColumnExtractor.ROW_POSITION_COLUMN into an ExternalSourceExec whose bound output references the standard _id metadata column or the _file.record_ref virtual column. The reader-emitted _rowPosition channel carries each record's file-global, split-invariant position; the producer pipeline composes _id as the opaque (location, mtime, rowPosition) hash per row and exposes the masked physical position directly as _file.record_ref (see ExternalRowIdentity / VirtualColumnIterator).

Every file reader materializes _rowPosition: Parquet (Java) and ORC emit a file-global row index from footer/stripe metadata (Parquet-Java encodes an extractor id into the high bits for the deferred-extraction path, which the composition path masks off); Parquet-RS splices an all-null block so _id renders null per the disclosed carve-out; NDJSON and CSV both emit a file-global byte anchor (CSV: the record's start byte; NDJSON: the parser position just past the record's opening token — exact anchor is opaque, intrinsic position is the contract), so the same physical record carries the same _rowPosition regardless of split layout.

Sibling of InsertExternalFieldExtraction, which also injects _rowPosition but for the deferred-extraction late-materialization path (it additionally requires a TopN above the source) and only for ColumnExtractorAware (Parquet-class) readers, which encode an extractor id into the high bits; the composition path masks that off, so a raw (unencoded) position from a non-deferred reader composes identically.

No-ops when the source's output already contains a _rowPosition attribute (deferred extraction got there first, or another query rewrote the plan twice). Idempotent.