Package org.elasticsearch.xpack.esql.datasources.cache

Class ExternalSourceCacheSettings

java.lang.Object
org.elasticsearch.xpack.esql.datasources.cache.ExternalSourceCacheSettings
public final class ExternalSourceCacheSettings extends Object
Cluster settings for ESQL external source caching. Everything here is restart-only (NodeScope) except the enabled flag, which is dynamic and wired to a live consumer in EsqlPlugin.createComponents. A setting must not be declared Dynamic unless a ClusterSettings.addSettingsUpdateConsumer actually observes updates — a Dynamic flag without a consumer accepts runtime updates and silently ignores them.

  • Field Details

    • CACHE_SIZE

      public static final Setting<ByteSizeValue> CACHE_SIZE

    • CACHE_ENABLED

      public static final Setting<Boolean> CACHE_ENABLED

    • SCHEMA_TTL

      public static final Setting<TimeValue> SCHEMA_TTL

    • LISTING_TTL

      public static final Setting<TimeValue> LISTING_TTL

    • STRIPE_SIZE

      public static final Setting<ByteSizeValue> STRIPE_SIZE
      Canonical stripe size for row-format external-source statistics, in file/decompressed-stream bytes. A stripe is a pure ADDRESSING grid over file content: the reader attributes each record to stripe floor(recordStartOffset / B) as it parses, and stats are captured, deduplicated, and cached per stripe (see ExternalSourceCacheService). It is orthogonal to partitioning — chunk dispatch, macro-splits, and parallelism are unaffected; the grid only determines which stripe a record's stats land in. The value participates in stripe identity, so it is restart-only and cluster-uniform: changing it simply makes previously cached stripe entries unmatchable (a clean invalidation, never a mixed grid).

      Default 8 MB, derived (not arbitrary) from the ClickBench text-format file-size distribution against the schema-cache budget: a representative ~1.8 GB shard yields ~231 stripes (ample pruning resolution), and a 500-hot-file working set consumes ~11 MB — 42% of the ~26 MB schema budget on a 32 GB heap. Smaller grids (≤1 MB) overflow the budget on realistic working sets; larger grids (≥32 MB) coarsen a representative shard to <60 stripes, blunting per-stripe min/max pruning. 8 MB is the knee.

    • STRIPE_COLUMNS

      public static final Setting<StripeColumnScope> STRIPE_COLUMNS
      How much per-stripe statistics a row-format external read harvests while it scans. Orthogonal to STRIPE_SIZE: the grid size decides which stripe a record lands in, this scope decides what is summarised per stripe. Modes (StripeColumnScope):
      • none — harvest nothing; a warm aggregate always re-scans.
      • count — per-stripe row count only (no per-column min/max/null). Enough to serve a warm COUNT(*).
      • projected — row count plus min/max/null for the query's projected columns. The default.
      • all — row count plus min/max/null for every column in the file's schema (the reader materialises the unprojected columns for the stats pass).
      Row count is harvested in count, projected, and all — everything except none. This is what lets a COUNT(*) read (zero projected columns) still record each stripe's row count.

      Unlike STRIPE_SIZE, this does NOT participate in stripe identity — it only changes how much a fresh scan harvests, never which stripe a record belongs to — so it COULD safely become dynamic. It is restart-only today because its one consumer (FileSourceFactory) reads it from the node's startup Settings; declaring it Dynamic without a live settings consumer would accept a runtime update and silently never observe it.

  • Method Details