Package org.elasticsearch.xpack.esql.datasources

Class StorageProviderRegistry

java.lang.Object

org.elasticsearch.xpack.esql.datasources.StorageProviderRegistry

All Implemented Interfaces:

Closeable, AutoCloseable

public class StorageProviderRegistry
extends Object
implements Closeable

Registry for StorageProvider implementations, keyed by URI scheme. Allows pluggable discovery of storage providers based on the scheme portion of a StoragePath (e.g., "http", "https", "s3", "file").

Providers are created lazily on first access rather than eagerly at startup, so heavy dependencies (S3 client, HTTP client, etc.) are only loaded when an EXTERNAL query actually targets that backend.

All non-file providers are automatically wrapped with per-scheme concurrency limiting and retry logic for transient storage failures (503, 429, connection resets, timeouts). Wrap order: caller → Retryable(with adaptive backoff) → ConcurrencyLimited → raw provider

Concurrency limiters are shared per-scheme and adaptive backoff state is shared per-throttle-scope across all providers (including per-query config providers), because cloud API rate limits are per account/IP, not per client instance.

Registration methods are intended for single-threaded initialization only (called from the DataSourceModule constructor).

This registry implements Closeable to properly close all created providers when the registry is no longer needed.

Constructor Summary

Constructors

Constructor	Description
StorageProviderRegistry(Settings settings)
StorageProviderRegistry(Settings settings, DataSourceCredentials credentials)	Test-only convenience constructor.
StorageProviderRegistry(Settings settings, DataSourceCredentials credentials, BooleanSupplier managedIdentityEnabled, org.elasticsearch.xpack.esql.datasources.RetryScheduler retryScheduler)
StorageProviderRegistry(Settings settings, DataSourceCredentials credentials, BooleanSupplier managedIdentityEnabled, org.elasticsearch.xpack.esql.datasources.RetryScheduler retryScheduler, LocalFileAccess localFileAccess)

Method Summary

Modifier and Type	Method	Description
org.elasticsearch.xpack.esql.datasources.ConcurrencyBudgetAllocator	allocatorForScheme(String scheme)	Per-query concurrency budget allocator for the given scheme, or null when per-query budgeting does not apply (file scheme, or permit limiting disabled via max_concurrent_requests=0).
void	close()
StorageProvider	createProvider(String scheme, Settings settings, Map<String,Object> config)	Convenience: returns the StorageProvider only.
Configured<StorageProvider>	createProviderTrackingConsumedKeys(String scheme, Settings settings, Map<String,Object> config)
boolean	hasProvider(String scheme)
StorageProvider	provider(StoragePath path)
void	registerFactory(String scheme, StorageProviderFactory factory)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

StorageProviderRegistry

public StorageProviderRegistry(Settings settings)

StorageProviderRegistry

public StorageProviderRegistry(Settings settings,
 @Nullable
 DataSourceCredentials credentials)

Test-only convenience constructor. The default managedIdentityEnabled supplier reads the cluster setting directly and does not apply the stateless gate that production wiring enforces in EsqlPlugin (where the boolean is forced to false when DiscoveryNode.isStateless). Similarly, localFileAccess defaults to LocalFileAccess.UNRESTRICTED and does not apply the stateless gate or the allowlist from ExternalSourceSettings#LOCAL_ALLOWED_PATHS. Production always goes through the five-argument constructor via DataSourceModule.

StorageProviderRegistry

public StorageProviderRegistry(Settings settings,
 @Nullable
 DataSourceCredentials credentials,
 BooleanSupplier managedIdentityEnabled,
 org.elasticsearch.xpack.esql.datasources.RetryScheduler retryScheduler)

StorageProviderRegistry

public StorageProviderRegistry(Settings settings,
 @Nullable
 DataSourceCredentials credentials,
 BooleanSupplier managedIdentityEnabled,
 org.elasticsearch.xpack.esql.datasources.RetryScheduler retryScheduler,
 LocalFileAccess localFileAccess)

Method Details

registerFactory

public void registerFactory(String scheme,
 StorageProviderFactory factory)

provider

public StorageProvider provider(StoragePath path)

hasProvider

public boolean hasProvider(String scheme)

createProvider

public StorageProvider createProvider(String scheme,
 Settings settings,
 Map<String,Object> config)

Convenience: returns the StorageProvider only. Use createProviderTrackingConsumedKeys(String, Settings, Map) when the consumed-keys set is needed.

createProviderTrackingConsumedKeys

public Configured<StorageProvider> createProviderTrackingConsumedKeys(String scheme,
 Settings settings,
 Map<String,Object> config)

allocatorForScheme

public org.elasticsearch.xpack.esql.datasources.ConcurrencyBudgetAllocator allocatorForScheme(String scheme)

Per-query concurrency budget allocator for the given scheme, or null when per-query budgeting does not apply (file scheme, or permit limiting disabled via max_concurrent_requests=0). Shared per-scheme so a single query cannot starve others on the same backend.

close

public void close()
           throws IOException

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Throws:

IOException