Package org.elasticsearch.xpack.esql.datasources.cache

Class ExternalStats

java.lang.Object
org.elasticsearch.xpack.esql.datasources.cache.ExternalStats
public final class ExternalStats extends Object
Serialization vocabulary and value types for external-text aggregate metadata. The cache itself is the unified SchemaCacheEntry: captured stats flow data-node → coordinator via DriverCompletionInfo.capturedSourceMetadata, get reconciled into a SchemaCacheEntry's safeMetadata as the well-known _stats.* keys below, and the optimizer (PushStatsToExternalSource) short-circuits COUNT(*) / COUNT(col) / MIN(col) / MAX(col) on warm queries. This type holds only the key names and the ExternalStats.Stats / ExternalStats.ColumnStats records shared across the capture, reconcile, and lookup sites.

  • Field Details

    • MTIME_MILLIS_KEY

      public static final String MTIME_MILLIS_KEY
      mtime (epoch millis) published into SourceMetadata.sourceMetadata() so the warm-path lookup can match the cached entry without a storage round-trip.
      See Also:

    • CONFIG_FINGERPRINT_KEY

      public static final String CONFIG_FINGERPRINT_KEY
      Node-stable fingerprint of the row-interpretation-affecting config (see SchemaCacheKey.buildFormatConfig(java.util.Map<java.lang.String, java.lang.Object>)). Distinct fingerprints scope distinct entries so a same-file re-query under different WITH options does not serve stale stats.
      See Also:

    • PARTIAL_CHUNK_KEY

      public static final String PARTIAL_CHUNK_KEY
      Set on per-chunk/per-segment contributions to mark them as a partial cover of the file (as opposed to a whole-file read). SourceStatsContribution.classify routes a partial to the stripe-fragment path: a stripe-addressed partial (carries STRIPE_SIZE_KEY etc.) folds through the reconciler's per-stripe interval cover; a partial WITHOUT stripe addressing is un-addressable and is a deterministic safe miss (never cached, never wrong). Whole-file reads carry no marker and stay on the authoritative dedup path.
      See Also:

    • COVERAGE_START_KEY

      public static final String COVERAGE_START_KEY
      Coverage-addressing keys. A stripe fragment describes the half-open byte sub-range [COVERAGE_START_KEY, COVERAGE_END_KEY) of its canonical stripe that it observed, in the path's own read coordinate system (decompressed-stream offset for stream codecs like gzip/zstd; raw file offset for uncompressed or block-splittable inputs — a single file is read in exactly one coordinate system per (path, config), so ranges are always comparable). The range is the fragment's intrinsic identity within its stripe: the reconciler's per-stripe interval cover counts a range observed more than once — the two branches of a FORK each re-scanning the source, a schema-probe pass plus the data scan, a retry, a redelivery — exactly once, while sibling chunkings of one stripe tile to the same folded stripe stats.

      COVERAGE_IS_LAST_KEY marks the fragment that observed the end of the input — the file's last stripe — which drives the whole-file completeness marker (stripes 0..K all covered + EOF known ⇒ the fold is cacheable as a file-level statistic). The keys ride inside the opaque _stats.* map, so there is no transport-version impact; an older node emits no coverage and its contribution is treated as un-addressable (never cached).

      See Also:

    • COVERAGE_END_KEY

      public static final String COVERAGE_END_KEY
      See Also:

    • COVERAGE_IS_LAST_KEY

      public static final String COVERAGE_IS_LAST_KEY
      See Also:

    • CHUNK_HAD_ERRORS_KEY

      public static final String CHUNK_HAD_ERRORS_KEY
      Published by the parallel coordinators when a scan did NOT complete cleanly — an error mid-scan, a truncated/cancelled read, a chunk cut short — i.e. the scan's extent is not deterministic. Classified as Poison: the coordinator discards every contribution for the file rather than commit stats whose extent another scan would not reproduce. A row DROPPED by the error policy (skip_row, or a structural malformed row under null_field) is deliberately NOT an error here: which rows survive is a deterministic function of the file bytes and the policy (pinned by the config fingerprint), so a clean-completing scan commits exact stats over the survivors.
      See Also:

    • STRIPE_SIZE_KEY

      public static final String STRIPE_SIZE_KEY
      Canonical-stripe addressing — orthogonal model. A file's stripe grid divides its byte stream (decompressed-stream offset for stream codecs, raw file offset for seekable inputs) into stripes of ExternalSourceCacheSettings.STRIPE_SIZE bytes. Stripes are a pure ADDRESSING grid, not a partitioning unit: the producing READER attributes each record to the stripe its start offset falls in (ordinal = floor(recordStartOffset / stripe_size)) as it parses, independently of how the read was chunked, split, or distributed. Chunks are never cut to align with stripes; a chunk spanning stripes contributes one fragment per stripe, and a chunk boundary landing mid-stripe splits that stripe across two chunks' fragments.

      Because attribution is by record-start offset, a stripe's content is a pure function of the file — identical across any two scans regardless of their chunking — which is what makes the reconciler's per-stripe interval-cover dedup exact (a FORK branch covering a stripe whole and a sibling splitting it differently fold to the same stripe stats).

      • STRIPE_SIZE_KEY — the grid B (bytes); a grid-consistency check, also identifies a fragment as stripe-addressed (absent ⇒ not cacheable, a safe miss).
      • STRIPE_ORDINAL_KEY — the reader-assigned stripe ordinal k this fragment belongs to (NOT inferred from the byte offset; the reader knows it record-canonically).
      • COVERAGE_START_KEY/COVERAGE_END_KEY — the record-canonical byte sub-range of stripe k this fragment covered, for the interval-cover tiling.
      • STRIPE_AT_START_KEY — this chunk covers the stripe's left grid line (splitStartByte <= k*B), so this fragment anchors the stripe's true (grid) start. This is a byte-range-cover predicate, NOT "holds the stripe's first record" — a stripe whose first record lands in the next chunk is still anchored here by the chunk that owns its left grid line.
      • STRIPE_AT_END_KEY — this chunk covers the stripe's right grid line (chunkAbsEnd >= (k+1)*B) or is the file-final chunk: the stripe's true (grid) end.
      • COVERAGE_IS_LAST_KEY — this fragment observed end-of-input: the file's last stripe (drives the whole-file completeness marker).
      See Also:

    • STRIPE_ORDINAL_KEY

      public static final String STRIPE_ORDINAL_KEY
      See Also:

    • STRIPE_AT_START_KEY

      public static final String STRIPE_AT_START_KEY
      See Also:

    • STRIPE_AT_END_KEY

      public static final String STRIPE_AT_END_KEY
      See Also:

    • STRIPE_ENTRY_PREFIX

      public static final String STRIPE_ENTRY_PREFIX
      Coordinator-cache keys for per-stripe committed stats, stored inside a SchemaCacheEntry's safeMetadata alongside the whole-file _stats.* fold. _stats.stripe.<k>. prefixes one committed stripe's flat stats map (row_count, columns.*, plus its span); the whole-file fold is written only when stripes 0..K are all committed and the marker (the file-EOF stripe ordinal, STRIPE_LAST_INDEX_KEY) is known. Commits are idempotent: re-committing a stripe overwrites with identical content.
      See Also:

    • STRIPE_LAST_INDEX_KEY

      public static final String STRIPE_LAST_INDEX_KEY
      See Also:

    • STRIPE_GRID_KEY

      public static final String STRIPE_GRID_KEY
      The stripe grid (bytes) the entry's committed _stats.stripe.<k> state was accumulated on — the entry-level twin of the fragment-level STRIPE_SIZE_KEY. Ordinals are only comparable within one grid: without this stamp, commits from data nodes running different esql.source.cache.stripe.size values (rolling restart, config drift) would interleave ordinals from different grids into one entry and the 0..K fold would serve a silently wrong count over a "complete" cover. A delta whose grid differs from the stamp CLEARS the entry's stripe state and re-stamps — accumulation restarts on the new grid (safe-miss, never mixed).
      See Also:

  • Method Details

    • isStripeBookkeeping

      public static boolean isStripeBookkeeping(String key)
      True for the coordinator-cache per-stripe bookkeeping keys (STRIPE_ENTRY_PREFIX sub-entries, STRIPE_LAST_INDEX_KEY, STRIPE_GRID_KEY) — the accumulation state behind the 0..K whole-file fold. They have no consumer outside ExternalSourceCacheService: the resolver strips them off the plan wire and the cache clears them once they can no longer contribute (complete fold, stale grid). This predicate is the single definition of that key set.