Package org.elasticsearch.xpack.esql.optimizer.rules.physical.local

Class InsertExternalFieldExtraction

java.lang.Object
org.elasticsearch.xpack.esql.rule.Rule<TopNExec,PhysicalPlan>
org.elasticsearch.xpack.esql.rule.ParameterizedRule<TopNExec,PhysicalPlan,LocalPhysicalOptimizerContext>
org.elasticsearch.xpack.esql.optimizer.PhysicalOptimizerRules.ParameterizedOptimizerRule<TopNExec,LocalPhysicalOptimizerContext>
org.elasticsearch.xpack.esql.optimizer.rules.physical.local.InsertExternalFieldExtraction
public class InsertExternalFieldExtraction extends PhysicalOptimizerRules.ParameterizedOptimizerRule<TopNExec,LocalPhysicalOptimizerContext>
Inserts a deferred field-extraction step above a per-driver TopNExec that sits on top of an ExternalSourceExec. The forward scan is narrowed to the columns the TopN actually uses (sort keys and any predicate / eval inputs traversed between TopN and the source) plus a synthetic _rowPosition key. The remaining ("wide") columns are loaded later, for the at-most-LIMIT surviving rows, by an ExternalFieldExtractExec placed immediately above the TopN.

Net effect: a TopN that previously pulled every projected column out of the file now only reads the sort key column up front and pays the I/O for the wide columns only for the rows that survive — analogous to Lucene's late materialization for SORT … | LIMIT N.

Bail-out conditions (rule returns the plan unchanged):

  • No ExternalSourceExec reachable below the TopN through a UnaryExec spine.
  • supportsDeferredExtraction(String, ExternalOptimizerContext) returns false for the source's format — the underlying reader does not implement ColumnExtractorAware.
  • The TopN's limit is unknown or exceeds TOPN_EXTRACT_LIMIT_MAX.
  • Fewer than DEFERRED_COLUMN_MIN columns would actually be deferred.
  • One of the source's columns is already named "_rowPosition" — we refuse to shadow user data rather than try to rename.

Production filter invariant: any ExternalSourceExec reaching this rule with a ExternalSourceExec.pushedFilter() also carries non-empty ExternalSourceExec.pushedExpressions(), because the only production code path that installs a pushed filter is PushFiltersToSource, which always populates both via ExternalSourceExec.withPushedFilterAndExpressions(Object, java.util.List). The rule uses the expressions to keep filter-referenced columns eager so the narrowed projection still has every input the filter reads.

  • Field Details

    • DEFERRED_COLUMN_MIN

      public static final int DEFERRED_COLUMN_MIN
      Minimum number of columns that need to be deferred before the optimization is worth applying. Below this, the per-row extraction overhead (random access reads, page assembly) outweighs the I/O saved by not loading the columns up front.
      See Also:

    • TOPN_EXTRACT_LIMIT_MAX

      public static final int TOPN_EXTRACT_LIMIT_MAX
      Maximum TopN limit for which the optimization fires. Above this, the extraction phase risks doing more random-access I/O than the streaming forward scan it replaces.
      See Also:

  • Constructor Details

    • InsertExternalFieldExtraction

      public InsertExternalFieldExtraction()

  • Method Details