Package org.elasticsearch.xpack.esql.datasources.cache

Class ExternalSourceCacheService

java.lang.Object

org.elasticsearch.xpack.esql.datasources.cache.ExternalSourceCacheService

All Implemented Interfaces:

Closeable, AutoCloseable

public class ExternalSourceCacheService
extends Object
implements Closeable

Coordinator-only, in-memory cache service for external source metadata. Maintains two independent caches:

• Schema cache (20% of budget, 5m TTL) — shared across users
• Listing cache (80% of budget, 30s TTL) — isolated by credential hash

Uses hard TTL via setExpireAfterWrite for the initial implementation. Lazy TTL with ETag revalidation is deferred to a follow-up PR.

Constructor Summary

Constructors

Constructor	Description
ExternalSourceCacheService(Settings settings)

Method Summary

Modifier and Type	Method	Description
void	clearAll()
void	close()
FileList	getOrComputeListing(ListingCacheKey key, CacheLoader<ListingCacheKey,FileList> loader)	Returns a cached file listing or stores the provided one.
SchemaCacheEntry	getOrComputeSchema(SchemaCacheKey key, CacheLoader<SchemaCacheKey,SchemaCacheEntry> loader)	Returns a cached schema entry or computes it via the loader.
SchemaCacheEntry	getSchemaIfPresent(SchemaCacheKey key)	Returns a cached schema entry, or null on a miss (or when the cache is disabled).
boolean	isEnabled()
void	putSchema(SchemaCacheKey key, SchemaCacheEntry entry)	Stores a schema entry.
void	reconcileSourceStats(Map<String,Map<String,Object>> mergedStatsPerFile)	Reconciles already-merged data-node-captured source stats into the schema cache.
void	reconcileSourceStatsFromContributions(Map<String,List<Map<String,Object>>> contributionsPerFile)	Coordinator-side entry point.
void	setEnabled(boolean enabled)
Map<String,Object>	usageStats()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

ExternalSourceCacheService

public ExternalSourceCacheService(Settings settings)

Method Details

getOrComputeSchema

public SchemaCacheEntry getOrComputeSchema(SchemaCacheKey key,
 CacheLoader<SchemaCacheKey,SchemaCacheEntry> loader)
                                    throws Exception

Returns a cached schema entry or computes it via the loader. The loader is only invoked on a cache miss. When the cache is disabled, the loader is called directly (bypassing the cache).

Throws:

Exception

getSchemaIfPresent

public SchemaCacheEntry getSchemaIfPresent(SchemaCacheKey key)

Returns a cached schema entry, or null on a miss (or when the cache is disabled). Unlike getOrComputeSchema(org.elasticsearch.xpack.esql.datasources.cache.SchemaCacheKey, org.elasticsearch.common.cache.CacheLoader<org.elasticsearch.xpack.esql.datasources.cache.SchemaCacheKey, org.elasticsearch.xpack.esql.datasources.cache.SchemaCacheEntry>), this never invokes a loader — it is the peek half of the async resolve path, which fetches on a miss without holding an executor thread and then stores the result via putSchema(org.elasticsearch.xpack.esql.datasources.cache.SchemaCacheKey, org.elasticsearch.xpack.esql.datasources.cache.SchemaCacheEntry). This trades strict thundering-herd coalescing (two concurrent misses for the same key may both fetch) for the ability to resolve asynchronously.

putSchema

public void putSchema(SchemaCacheKey key,
 SchemaCacheEntry entry)

Stores a schema entry. No-op when the cache is disabled. Pairs with getSchemaIfPresent(org.elasticsearch.xpack.esql.datasources.cache.SchemaCacheKey).

getOrComputeListing

public FileList getOrComputeListing(ListingCacheKey key,
 CacheLoader<ListingCacheKey,FileList> loader)
                             throws Exception

Returns a cached file listing or stores the provided one. The loader is only invoked on a cache miss. When the cache is disabled, the loader is called directly (bypassing the cache).

Throws:

Exception

reconcileSourceStatsFromContributions

public void reconcileSourceStatsFromContributions(Map<String,List<Map<String,Object>>> contributionsPerFile)

Coordinator-side entry point. Takes the DriverCompletionInfo.capturedSourceMetadata payload — raw per-file contribution lists shipped back from every data node — merges each list via SourceStatisticsSerializer.mergeStatistics (Parquet's existing multi-row- group merge algorithm), then enriches the matching SchemaCacheEntry so the next query's planning-time lookup short-circuits on the merged stats.

reconcileSourceStats

public void reconcileSourceStats(Map<String,Map<String,Object>> mergedStatsPerFile)

Reconciles already-merged data-node-captured source stats into the schema cache. For each (path, mergedStats) entry, finds the cached SchemaCacheEntry whose location and mtime match and replaces it with a new entry whose safeMetadata folds in the merged _stats.* keys. Entries with no cache match are ignored (the warm path will just trigger a fresh metadata() call on the next query).

setEnabled

public void setEnabled(boolean enabled)

isEnabled

public boolean isEnabled()

clearAll

public void clearAll()

usageStats

public Map<String,Object> usageStats()

close

public void close()

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable