Package org.elasticsearch.xpack.esql.datasources

package org.elasticsearch.xpack.esql.datasources

Related Packages

Package	Description
org.elasticsearch.xpack.esql	The ES|QL query language.
org.elasticsearch.xpack.esql.datasources.arrow
org.elasticsearch.xpack.esql.datasources.cache
org.elasticsearch.xpack.esql.datasources.dataset
org.elasticsearch.xpack.esql.datasources.datasource
org.elasticsearch.xpack.esql.datasources.glob	Glob pattern matching and expansion for ES|QL external datasources.
org.elasticsearch.xpack.esql.datasources.metadata
org.elasticsearch.xpack.esql.datasources.pushdown
org.elasticsearch.xpack.esql.datasources.spi
org.elasticsearch.xpack.esql.datasources.utils

Class	Description
AsyncConnectorSourceOperatorFactory	Source-operator factory that executes a connector query on a background thread and feeds pages into an AsyncExternalSourceBuffer for the driver to consume.
AsyncExternalSourceBuffer	Thread-safe buffer for async external source data.
AsyncExternalSourceOperator	Source operator that retrieves data from external sources (Iceberg tables, Parquet files, etc.).
AsyncExternalSourceOperator.Status
AsyncExternalSourceOperatorFactory	Dual-mode async factory for creating source operators that read from external storage.
AsyncExternalSourceOperatorFactory.Builder	Fluent builder for AsyncExternalSourceOperatorFactory.
AutoPartitionDetector	Auto-detecting partition detector that tries Hive-style detection first, then falls back to template-based detection if a path template is configured.
CoalescedSplit	A composite split that groups multiple child splits into a single scheduling unit.
ColumnMapping	Per-file value describing how this file's blocks become the query's output blocks: for each output column, the source position in the file's natural-order page (or -1 when the file lacks that column), and the cast (if any) the coordinator chose during reconciliation.
ColumnStatTypeSupport	The single source of truth for how a column's ESQL DataType participates in the external-source per-column statistics pipeline (harvest, cache classification, warm-path serving, and block construction).
ColumnStatTypeSupport.StatBlockKind	The block flavour a served MIN/MAX extremum is materialized into.
ColumnStatTypeSupport.StatCoercion	How a stat Number is normalized to the column's resolved type before it is folded/served.
DatasetResolver	Owns the security round-trip for FROM <dataset>, the companion of DatasetRewriter: each FROM relation the query reads is dispatched — one EsqlResolveDatasetAction per relation, with that relation's RAW patterns — so the authorization engine read-authorizes the names (index read privilege) and rejects DLS/FLS-restricted datasets, then narrows wildcard expansion to the authorized abstractions.
DatasetRewriter	Rewrites FROM <dataset> into the same UnresolvedExternalRelation the EXTERNAL command produces, so both paths converge at the existing resolver + analyzer.
DatasetRewriter.DatasetResolution	Per-relation result of DatasetRewriter.resolve(java.lang.String[], java.lang.String[], org.elasticsearch.cluster.metadata.ProjectMetadata, org.elasticsearch.cluster.metadata.IndexNameExpressionResolver): the authorized concrete dataset names the relation targets, the concrete non-dataset names resolved from the same pattern (drives heterogeneous-FROM UnionAll building), and the explicitly-named datasets absent from the authorized set (surfaced by DatasetRewriter.rewriteOne(org.elasticsearch.xpack.esql.plan.logical.UnresolvedRelation, org.elasticsearch.cluster.metadata.DatasetMetadata, org.elasticsearch.xpack.esql.datasources.metadata.DataSourceMetadata, org.elasticsearch.xpack.esql.datasources.DatasetRewriter.DatasetResolution, boolean) as Unknown index).
DataSourceCapabilities	Aggregated capability declarations from all registered DataSourcePlugin instances.
DataSourceCredentials	Decrypts secret values at the connector boundary.
DataSourceEncryptedDataHandler	Re-encrypts every stored data-source secret under the active project encryption key on rotation — without it, retiring a key would strand secrets undecryptable.
DataSourceModule	Module that collects all data source implementations from plugins.
DeclaredReadSpec	The read-instructions a declared dataset mapping derives for the data node.
DeclaredSchemaResolver	Bridges a DatasetMapping (the String-typed, server-side declaration) into the ES|QL currency the external-source resolver speaks: Attributes typed with DataType, plus the logical→physical rename map.
DeclaredSchemaResolver.Overlaid	Result of a non-strict overlay: the user-facing output (declared columns renamed to their logical name and retyped, undeclared columns untouched) and the per-file fileSchema the reader resolves against the file.
DeclaredSchemaValidator	Shape-only validation of a dataset's DatasetMapping at PUT time — no file I/O.
DecompressionCodecRegistry	Registry for DecompressionCodec implementations, keyed by extension.
DeferredExtractionCapable	Capability marker for a SourceOperatorFactory that participates in deferred (post-TopN) field extraction.
DirectByteBufferCopies	Utilities for copying streaming ByteBuffer chunks into a pre-sized direct destination.
EsqlDataSourcesCapabilities	REST-handler capability constants for the ES|QL data source + dataset CRUD API.
EsqlEncryptedDataHandlerProvider	Contributes the DataSourceEncryptedDataHandler to the project encryption key rotation coordinator.
ExternalFailures	Classifies a failure raised while reading an external data source into the exception the AsyncExternalSourceOperator should surface, so that it maps to the right HTTP status.
ExternalFieldExtractOperator	Page-mapping operator that materializes deferred ("wide") columns for the rows surviving a per-driver TopN.
ExternalFieldExtractOperator.Factory	Builds ExternalFieldExtractOperators for each driver.
ExternalFieldExtractOperator.Status	Per-driver counters for ExternalFieldExtractOperator, surfaced as the operator's status in the profile: pages processed, rows whose deferred columns were materialized, and wall time in materialize(...).
ExternalMetadataColumns	Registry of the standard ES index metadata names (_id, _index, _version, ...) that the external-source pipeline knows how to materialise on external datasets.
ExternalRowIdentity	Per-page composition of the _id metadata column for external datasets.
ExternalSchema	Coordinator-only carrier for the four-schema-model units (File, Unified, Query, Per-file Query — see SchemaReconciliation) inside the datasources subsystem.
ExternalSliceQueue	Thread-safe queue of ExternalSplit instances for parallel external source execution.
ExternalSourceDrainUtils	Utility for draining pages from a CloseableIterator into an AsyncExternalSourceBuffer with non-blocking backpressure.
ExternalSourceMetadata	Extended interface for external data source metadata.
ExternalSourceOperatorFactory	Deprecated. retained for test fixtures only; new code should use AsyncExternalSourceOperatorFactory.
ExternalSourceResolution	Holds the result of external source resolution (Iceberg/Parquet metadata).
ExternalSourceResolution.ResolvedSource
ExternalSourceResolver	Resolver for external data sources (Iceberg tables, Parquet files, etc.).
ExternalSourceSettings	Cluster settings for controlling ESQL external source behavior; all node-scoped.
ExternalStatsRequirementExtractor	Detects which external relations are read by a query shape that needs eager global statistics aggregated across all files of a multi-file glob during planning.
FileMetadataColumns	Registry of well-known file metadata virtual columns for external data sources.
FileSplit	Represents a byte range within a file for a file-based external source.
FileSplitProvider	Default SplitProvider for file-based sources.
FilterAdaptation	Format-agnostic utility for adapting pushed filter expressions to a specific file's column set.
FilterEvaluationOrderEstimator	Reorders filter conjuncts by estimated selectivity and cost using column statistics from source metadata.
FormatNameResolver	Single source of truth for resolving format names from EXTERNAL query configuration.
FormatReaderRegistry	Registry for FormatReader implementations, keyed by format name and file extension.
HivePartitionDetector	Detects Hive-style partition columns from file paths (e.g., /year=2024/month=06/file.parquet).
LocalFileAccess	Enforces the esql.datasource.local_allowed_paths allowlist gate for file:// external sources.
MergedSplitStats	Lazily merges statistics from a list of child SplitStats instances.
OperatorFactoryRegistry	Registry for source operator factories.
ParallelParsingCoordinator	Coordinates parallel parsing of a single file by splitting it into byte-range segments and dispatching each segment to a parser thread.
PartitionConfig	Configuration for partition detection strategy, parsed from WITH clause parameters.
PartitionConfig.Strategy
PartitionDetector	Pluggable strategy for detecting partition columns from file paths.
PartitionFilterHintExtractor	Walks an unresolved logical plan extracting simple filter predicates from Filter nodes above UnresolvedExternalRelation nodes.
PartitionFilterHintExtractor.Operator
PartitionFilterHintExtractor.PartitionFilterHint
PartitionMetadata	Holds partition information detected from file paths.
PhysicalNames	Single source of truth for the declared-schema logical→physical column rename (path).
SchemaReconciliation	Schema reconciliation algorithms for multi-file external sources.
SchemaReconciliation.FileSchemaInfo	Per-file schema information collected during reconciliation.
SchemaReconciliation.Result	Result of schema reconciliation during planning.
SourceExtractors	Driver-scoped lookup table of per-file ColumnExtractors, addressed by a compact int id allocated at registration time.
SourceStatisticsSerializer	Serializes and deserializes SourceStatistics to/from a flat Map<String, Object> using well-known keys.
SplitCoalescer	Groups many small ExternalSplits into CoalescedSplits to reduce scheduling overhead.
SplitDiscoveryPhase	Walks the physical plan tree, discovers splits for each ExternalSourceExec, and replaces them with split-enriched copies via ExternalSourceExec.withSplits(java.util.List<org.elasticsearch.xpack.esql.datasources.spi.ExternalSplit>).
SplitDiscoveryPhase.Result	Post-prune "scanned" accounting collected while resolving splits, surfaced at the root of the query profile.
SplitStats	Compact representation of per-split (row-group/stripe) statistics with path-segment dictionary compression.
SplitStats.Builder	Builder for constructing SplitStats instances.
StatValueComparator	Type-normalizing comparator for column-statistic values (min/max) against query literals.
StorageEntry	Metadata about an object returned from directory listing.
StorageIterator	Iterator over objects in a directory.
StorageManager	Format-agnostic manager for creating StorageObject instances from paths.
StorageProviderRegistry	Registry for StorageProvider implementations, keyed by URI scheme.
StreamingParallelParsingCoordinator	Coordinates parallel parsing of a sequential (non-seekable) decompressed stream by chunking it at record boundaries and dispatching each chunk to a parser thread.
StreamingParallelParsingCoordinator.WarningSinks	Bundles the two independent, easily-confused warning relays a streaming parallel read may need to thread onto worker threads, as a single typed parameter — two bare adjacent Consumer<String> parameters/fields invite silent transposition at a call site.
SynthesizeExternalSource	Producer-side composition of the _source metadata column for external datasets.
SyntheticColumns	Helpers for reader-synthesized internal channels — columns with no backing source field, filled by the format reader from a per-format value source.
SyntheticColumns.Kind	Registry of reader-synthesized internal channels.
TemplatePartitionDetector	Detects partition columns from bare directory segments using a path template.
TextAggregatePushdownSupport	Aggregate-pushdown opt-in for the line-oriented text-format readers (CSV / TSV / NDJSON).