Package org.elasticsearch.xpack.esql.datasources

Class ExternalSourceResolver

java.lang.Object

org.elasticsearch.xpack.esql.datasources.ExternalSourceResolver

public class ExternalSourceResolver
extends Object

Resolver for external data sources (Iceberg tables, Parquet files, etc.). This runs in parallel with IndexResolver to resolve external source metadata.

Following the same pattern as IndexResolver, this resolver:

• Takes a list of external source paths to resolve
• Performs I/O operations to fetch metadata (from S3/Iceberg catalogs)
• Returns ExternalSourceResolution containing resolved metadata
• Runs asynchronously to avoid blocking

Registry-based resolution: This resolver iterates the ExternalSourceFactory instances collected by DataSourceModule to find the first factory that can handle a given path. File-based sources (Parquet, CSV) are handled by the framework-internal FileSourceFactory registered as a catch-all fallback.

Configuration handling: Query parameters are converted to a generic Map<String, Object> instead of source-specific classes like S3Configuration. This allows the SPI to remain generic while source-specific implementations can interpret the configuration as needed.

Field Summary

Fields

Modifier and Type	Field	Description
static final Set<String>	CONFIG_KEYS
static final String	CONFIG_SCHEMA_RESOLUTION
static final String	DATASOURCE_CONFIG_KEY	Config key under which DatasetRewriter stores data-source-level settings (auth credentials, region, etc.) when building the merged config for a dataset query.

Constructor Summary

Constructors

Constructor	Description
ExternalSourceResolver(Executor executor, DataSourceModule dataSourceModule)
ExternalSourceResolver(Executor executor, DataSourceModule dataSourceModule, Settings settings)
ExternalSourceResolver(Executor executor, DataSourceModule dataSourceModule, Settings settings, ExternalSourceCacheService cacheService)
ExternalSourceResolver(Executor executor, DataSourceModule dataSourceModule, Settings settings, ExternalSourceCacheService cacheService, int metadataReadConcurrency)
ExternalSourceResolver(Executor executor, DataSourceModule dataSourceModule, Settings settings, ExternalSourceCacheService cacheService, BooleanSupplier isCancelled)
ExternalSourceResolver(Executor executor, DataSourceModule dataSourceModule, Settings settings, ExternalSourceCacheService cacheService, BooleanSupplier isCancelled, int metadataReadConcurrency)
ExternalSourceResolver(Executor executor, DataSourceModule dataSourceModule, Settings settings, ExternalSourceCacheService cacheService, BooleanSupplier isCancelled, int metadataReadConcurrency, ThreadContext threadContext)

Method Summary

Modifier and Type	Method	Description
ExternalSourceCacheService	cacheService()	Coordinator-side accessor used by EsqlSession to reconcile data-node-captured source stats post-query.
Executor	executor()	Executor the discovery fan-out runs on.
int	metadataReadConcurrency()	Maximum in-flight per-file metadata reads for a multi-file discovery.
static Map<String,Object>	planConfig(Map<String,Object> config)	Returns a config with the DATASOURCE_CONFIG_KEY sub-map removed.
void	resolve(List<String> paths, Map<String,Map<String,Object>> pathConfigs, Map<String,List<PartitionFilterHintExtractor.PartitionFilterHint>> filterHints, Map<String,DatasetMapping> declaredMappings, Set<String> pathsRequiringStats, ActionListener<ExternalSourceResolution> listener)	Resolves external sources.
void	resolve(List<String> paths, Map<String,Map<String,Object>> pathConfigs, Map<String,List<PartitionFilterHintExtractor.PartitionFilterHint>> filterHints, ActionListener<ExternalSourceResolution> listener)
void	resolve(List<String> paths, Map<String,Map<String,Object>> pathConfigs, ActionListener<ExternalSourceResolution> listener)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

CONFIG_SCHEMA_RESOLUTION

public static final String CONFIG_SCHEMA_RESOLUTION

See Also:

• Constant Field Values

DATASOURCE_CONFIG_KEY

public static final String DATASOURCE_CONFIG_KEY

Config key under which DatasetRewriter stores data-source-level settings (auth credentials, region, etc.) when building the merged config for a dataset query. These are kept separate from the dataset format settings so that file-format factories can validate only the keys they own, and so that credential values are not embedded in the serialized plan sent to data nodes.

See Also:

• Constant Field Values

CONFIG_KEYS

public static final Set<String> CONFIG_KEYS

Constructor Details

ExternalSourceResolver

public ExternalSourceResolver(Executor executor,
 DataSourceModule dataSourceModule)

ExternalSourceResolver

public ExternalSourceResolver(Executor executor,
 DataSourceModule dataSourceModule,
 Settings settings)

ExternalSourceResolver

public ExternalSourceResolver(Executor executor,
 DataSourceModule dataSourceModule,
 Settings settings,
 @Nullable
 ExternalSourceCacheService cacheService)

ExternalSourceResolver

public ExternalSourceResolver(Executor executor,
 DataSourceModule dataSourceModule,
 Settings settings,
 @Nullable
 ExternalSourceCacheService cacheService,
 @Nullable
 BooleanSupplier isCancelled)

ExternalSourceResolver

public ExternalSourceResolver(Executor executor,
 DataSourceModule dataSourceModule,
 Settings settings,
 @Nullable
 ExternalSourceCacheService cacheService,
 int metadataReadConcurrency)

Parameters:

metadataReadConcurrency - maximum number of in-flight per-file metadata reads during a multi-file discovery. Production passes the esql_worker pool size.

ExternalSourceResolver

public ExternalSourceResolver(Executor executor,
 DataSourceModule dataSourceModule,
 Settings settings,
 @Nullable
 ExternalSourceCacheService cacheService,
 @Nullable
 BooleanSupplier isCancelled,
 int metadataReadConcurrency)

Parameters:

isCancelled - consulted before each per-file footer read so a wide-glob discovery aborts promptly on cancellation; null means "never cancelled".

metadataReadConcurrency - maximum number of in-flight per-file metadata reads during a multi-file discovery. Production passes the esql_worker pool size.

ExternalSourceResolver

public ExternalSourceResolver(Executor executor,
 DataSourceModule dataSourceModule,
 Settings settings,
 @Nullable
 ExternalSourceCacheService cacheService,
 @Nullable
 BooleanSupplier isCancelled,
 int metadataReadConcurrency,
 @Nullable
 ThreadContext threadContext)

Parameters:

isCancelled - consulted before each per-file footer read so a wide-glob discovery aborts promptly on cancellation; null means "never cancelled".

metadataReadConcurrency - maximum number of in-flight per-file metadata reads during a multi-file discovery. Production passes the esql_worker pool size.

threadContext - the calling request's transport ThreadContext, captured while still on the authenticated calling thread; restored around the outward resolve(java.util.List<java.lang.String>, java.util.Map<java.lang.String, java.util.Map<java.lang.String, java.lang.Object>>, org.elasticsearch.action.ActionListener<org.elasticsearch.xpack.esql.datasources.ExternalSourceResolution>) completion listener so that async completions on non-ES threads don't lose the request's security context. null when there is no context to preserve (tests, non-request call sites).

Method Details

planConfig

public static Map<String,Object> planConfig(Map<String,Object> config)

Returns a config with the DATASOURCE_CONFIG_KEY sub-map removed. Used as the fallback when serializing a plan to a data node whose transport version predates the encrypted-secret carrier (see ExternalSourceExec.writeTo): such a node cannot deserialize the carrier, so the credentials are stripped and it reverts to pre-credential-forwarding behavior.

cacheService

public ExternalSourceCacheService cacheService()

Coordinator-side accessor used by EsqlSession to reconcile data-node-captured source stats post-query.

metadataReadConcurrency

public int metadataReadConcurrency()

Maximum in-flight per-file metadata reads for a multi-file discovery. Visible for wiring tests.

executor

public Executor executor()

Executor the discovery fan-out runs on. Visible for wiring/isolation tests.

resolve

public void resolve(List<String> paths,
 Map<String,Map<String,Object>> pathConfigs,
 ActionListener<ExternalSourceResolution> listener)

resolve

public void resolve(List<String> paths,
 Map<String,Map<String,Object>> pathConfigs,
 @Nullable
 Map<String,List<PartitionFilterHintExtractor.PartitionFilterHint>> filterHints,
 ActionListener<ExternalSourceResolution> listener)

resolve

public void resolve(List<String> paths,
 Map<String,Map<String,Object>> pathConfigs,
 @Nullable
 Map<String,List<PartitionFilterHintExtractor.PartitionFilterHint>> filterHints,
 @Nullable
 Map<String,DatasetMapping> declaredMappings,
 @Nullable
 Set<String> pathsRequiringStats,
 ActionListener<ExternalSourceResolution> listener)

Resolves external sources. A per-path declared mapping drives strict/non-strict schema resolution and carries column renames; pathsRequiringStats gates the FIRST_FILE_WINS eager all-file stats aggregation.

Parameters:

declaredMappings - per-path declared mapping — strict skips inference, non-strict overlays it, and its derived read-instructions (renames, _id.path, date formats) ride a typed DeclaredReadSpec to the reader boundary; null when no path declares a mapping.

pathsRequiringStats - paths whose multi-file FFW resolution must eagerly aggregate global statistics across all files (the ungrouped-aggregate metadata fast path). A null value selects legacy behavior — every path resolves eagerly — preserving existing call sites and tests. When non-null, a path absent from the set defers the per-file footer reads (keeping STATS_FILE_COUNT, marking stats partial). See ExternalStatsRequirementExtractor.pathsRequiringEagerStats(org.elasticsearch.xpack.esql.plan.logical.LogicalPlan).