Package org.elasticsearch.xpack.esql.datasources.spi

Interface DataSourcePlugin

public interface DataSourcePlugin

Extension point for data source implementations. Plugins implementing this interface will be discovered by ESQL at startup via Elasticsearch's plugin discovery mechanism.

This interface allows plugins to provide:

• Storage providers (S3, GCS, Azure, HTTP) for accessing data - keyed by URI scheme
• Format readers (Parquet, CSV, ORC) for parsing data files - keyed by format name
• Table catalog connectors (Iceberg, Delta Lake) for table metadata - keyed by catalog type
• Custom operator factories for complex datasources - keyed by source type
• Decompression codecs for compound extensions (e.g. .csv.gz) - via decompressionCodecs(Settings) or decompressionCodecs(Settings, ExecutorService)

All methods have default implementations returning empty maps/lists, allowing plugins to implement only the capabilities they provide.

Note: Method names follow the project convention of omitting the "get" prefix since there are no corresponding setters.

Method Summary

Modifier and Type	Method	Description
default Map<String,ConnectorFactory>	connectors(Settings settings)
default Map<String,DataSourceValidator>	datasourceValidators(Settings settings)	CRUD-time validators for datasource and dataset settings, keyed by type name.
default List<DecompressionCodec>	decompressionCodecs(Settings settings)	Decompression codecs this plugin provides (e.g.
default List<DecompressionCodec>	decompressionCodecs(Settings settings, ExecutorService executor)	Decompression codecs with access to the datasource module's Elasticsearch-managed ExecutorService (typically the generic thread pool).
default Map<String,FormatReaderFactory>	formatReaders(Settings settings)
default Set<FormatSpec>	formatSpecs()	Format descriptors this plugin provides.
default List<NamedWriteableRegistry.Entry>	namedWriteables()
default Map<String,SourceOperatorFactoryProvider>	operatorFactories(Settings settings)
default Map<String,ExternalSourceFactory>	sourceFactories(Settings settings)
default Map<String,StorageProviderFactory>	storageProviders(Settings settings)
default Map<String,StorageProviderFactory>	storageProviders(Settings settings, ExecutorService executor)
default Map<String,StorageProviderFactory>	storageProviders(StorageProviderServices services)	Storage providers with access to node-level services (StorageProviderServices).
default Set<String>	supportedCatalogs()	Catalog types this plugin provides (e.g.
default Set<String>	supportedConnectorSchemes()	URI schemes handled by this plugin's connectors (e.g.
default Set<String>	supportedSchemes()	URI schemes whose storage providers this plugin supplies (e.g.
default Map<String,TableCatalogFactory>	tableCatalogs(Settings settings)

Method Details

supportedSchemes

default Set<String> supportedSchemes()

URI schemes whose storage providers this plugin supplies (e.g. "s3", "gs", "http"). Called once at registration time; must be cheap (no I/O, no heavy deps). Connector plugins should use supportedConnectorSchemes() instead.

supportedConnectorSchemes

default Set<String> supportedConnectorSchemes()

URI schemes handled by this plugin's connectors (e.g. "flight", "grpc"). Separate from supportedSchemes() which is for storage providers.

formatSpecs

default Set<FormatSpec> formatSpecs()

Format descriptors this plugin provides. Each FormatSpec pairs a logical format name with the file extensions that select it. Format names must match the keys returned by formatReaders(Settings).

supportedCatalogs

default Set<String> supportedCatalogs()

Catalog types this plugin provides (e.g. "iceberg"). Keys must match tableCatalogs(Settings) keys.

storageProviders

default Map<String,StorageProviderFactory> storageProviders(Settings settings)

storageProviders

default Map<String,StorageProviderFactory> storageProviders(Settings settings,
 ExecutorService executor)

storageProviders

default Map<String,StorageProviderFactory> storageProviders(StorageProviderServices services)

Storage providers with access to node-level services (StorageProviderServices). Plugins that need the node Environment or ResourceWatcherService — e.g. to resolve or watch operator-managed token symlinks under ${ES_PATH_CONF} — should override this method.

The default delegates to storageProviders(Settings, ExecutorService), so plugins with no node-context needs can keep overriding the simpler signatures unchanged.

Lifecycle: this method may allocate node-level resources (file watchers, clients). The DataSourceModule closes every Closeable DataSourcePlugin when it shuts down, so implementations must release those resources in close() and tolerate a close() that runs without any prior storageProviders call.

formatReaders

default Map<String,FormatReaderFactory> formatReaders(Settings settings)

decompressionCodecs

default List<DecompressionCodec> decompressionCodecs(Settings settings)

Decompression codecs this plugin provides (e.g. gzip for .gz, .gzip). Used for compound extensions like .csv.gz or .ndjson.gz.

decompressionCodecs

default List<DecompressionCodec> decompressionCodecs(Settings settings,
 ExecutorService executor)

Decompression codecs with access to the datasource module's Elasticsearch-managed ExecutorService (typically the generic thread pool). Codecs that parallelize work (e.g. scanning compressed block boundaries) should use this executor instead of creating ad-hoc thread pools.

sourceFactories

default Map<String,ExternalSourceFactory> sourceFactories(Settings settings)

tableCatalogs

default Map<String,TableCatalogFactory> tableCatalogs(Settings settings)

operatorFactories

default Map<String,SourceOperatorFactoryProvider> operatorFactories(Settings settings)

connectors

default Map<String,ConnectorFactory> connectors(Settings settings)

namedWriteables

default List<NamedWriteableRegistry.Entry> namedWriteables()

datasourceValidators

default Map<String,DataSourceValidator> datasourceValidators(Settings settings)

CRUD-time validators for datasource and dataset settings, keyed by type name. Receives node Settings so implementations can read cluster admin overrides (e.g. limits, allowed auth modes, default field values) when constructing validators.