Package org.elasticsearch.xpack.esql.datasources

Class ExternalStatsRequirementExtractor

java.lang.Object
org.elasticsearch.xpack.esql.datasources.ExternalStatsRequirementExtractor
public final class ExternalStatsRequirementExtractor extends Object
Detects which external relations are read by a query shape that needs eager global statistics aggregated across all files of a multi-file glob during planning.

Eager global stats exist for exactly one optimization: the metadata-only fast path for an ungrouped aggregate (COUNT/MIN/MAX with no BY) directly over an external relation (see ComputeService#canSkipSplitDiscovery). Every other consumer (filter/aggregate pushdown, split-filter classification) runs after split discovery and prefers per-split stats over the global sourceMetadata. Therefore grouped STATS ... BY and INLINESTATS never benefit from the eager global aggregation and must not pay its per-file footer-read cost.

The result of pathsRequiringEagerStats(LogicalPlan) is used by ExternalSourceResolver to gate the FIRST_FILE_WINS eager all-file stats aggregation: a path absent from the set defers the N footer reads (keeping STATS_FILE_COUNT, marking stats partial), so wide globs no longer block planning on thousands of object-store GETs for queries that do not consume the global stats (e.g. LIMIT, SELECT *, grouped STATS ... BY, INLINESTATS).

Deliberate safety bias

An ungrouped aggregate is matched anywhere above the relation, not only as its direct parent. A false negative would turn a metadata-only COUNT(*) over a huge glob into a full N-file scan (a severe regression); a false positive (e.g. ... | WHERE x > 5 | STATS COUNT(*), which usually will not actually skip split discovery) only costs the over-read we already tolerate today. We bias toward eager for ungrouped aggregates.

Per-path conservatism for mixed branches

The result is a Set: if the same path appears under an ungrouped aggregate in one branch (e.g. a FORK arm) and under LIMIT in another, the union marks it as requiring eager stats. The single resolution of that path stays eager, which is correct because one resolution feeds both branches.

  • Method Details

    • pathsRequiringEagerStats

      public static Set<String> pathsRequiringEagerStats(LogicalPlan unresolvedPlan)
      Returns the literal tablePath of every UnresolvedExternalRelation that has an ungrouped Aggregate ancestor. The path-key derivation is identical to PreAnalyzer and EsqlSession#extractExternalConfigs (BytesRefs.toString(literal.value())), so the keys match the resolver's icebergPaths by construction.
      Parameters:
      unresolvedPlan - the root of the unresolved logical plan
      Returns:
      the set of literal path strings whose resolution must eagerly aggregate global stats