Package org.elasticsearch.xpack.esql.datasources.cache

Class ParsedFooterCache<T>

java.lang.Object
org.elasticsearch.xpack.esql.datasources.cache.ParsedFooterCache<T>
Type Parameters:
T - the parsed metadata type held by this cache (e.g. ParquetMetadata).
public final class ParsedFooterCache<T> extends Object
JVM-wide cache for parsed file metadata (e.g. Parquet ParquetMetadata, ORC OrcTail). Sits at the same architectural layer as FooterByteCache but stores the result of the format-specific footer parse rather than its raw bytes, so the (typically Thrift/protobuf) deserialization runs at most once per (path, fileLength) key across:
  • concurrent splits of the same file taken by N producer threads;
  • back-to-back queries against the same file within the access TTL.

Why parsed metadata and not just raw bytes

FooterByteCache eliminates redundant tail-byte reads from object storage, but the parse still runs every time a reader is opened: N producers fanning out over a wide file each pay the full deserialization cost, even though the cached bytes are identical. Caching the parsed result collapses that into a single deserialization. FooterByteCache stays in place because it also serves opportunistic partial-tail reads (page indexes, dictionary tails) that are not full-footer parses, and because format readers fall back to byte reads on a cold cache.

Sharing keys with FooterByteCache

The cache is keyed by FooterByteCache.Key ((path, fileLength)) so that the same key construction logic used to hit the byte cache also hits this cache; both caches stay aligned without an extra key type. Per-format singletons (one for Parquet, one for ORC, etc.) keep the value type concrete and the cache's ownership unambiguous.

Lifecycle

  • Created as a singleton per format reader; no SPI plumbing is required to share entries across producers since every code path that needs a parsed footer already constructs a FooterByteCache.Key via the storage-object adapter.
  • Access-based TTL — sourced from FooterByteCache.EXPIRE_AFTER_ACCESS_SECONDS so the two caches age out together; covers a single query's fan-out (where concurrent splits keep the entry alive) while ensuring that file modifications between queries trigger a fresh parse.
  • Count-based LRU eviction — parsed metadata structures (e.g. Parquet ParquetMetadata or ORC OrcTail) do not expose a cheap byte size, so the cache caps the number of entries rather than total bytes. Worst-case heap budget: a single parsed footer for an extreme file (e.g. 100 columns × 200 row groups → ~20k column-chunk entries) can occupy ~10–20 MiB of heap. With DEFAULT_MAX_ENTRIES = 32<T> the absolute worst case is in the hundreds of MiB if every cached entry is for such an extreme file and all live concurrently inside the TTL window. Typical workloads (≤10 columns, ≤50 row groups, ≈200 KiB per entry) sit at a few MiB total. Adding a real weigher would require an estimator pass over the parsed structure on every put — not implemented here, but tracked for follow-up if heap pressure shows up in production. Note that the byte and parsed caches evict independently — TTL alignment keeps them timing-consistent but does not synchronize eviction events.

Cached values must be treated as immutable by all callers — callers that need to derive a filtered view (e.g. only the row groups for a specific byte range) should build a new value from the cached one rather than mutating the cached structure.