Package org.elasticsearch.xpack.esql.datasources

Class AsyncExternalSourceOperatorFactory.Builder

java.lang.Object

org.elasticsearch.xpack.esql.datasources.AsyncExternalSourceOperatorFactory.Builder

Enclosing class:

AsyncExternalSourceOperatorFactory

public static final class AsyncExternalSourceOperatorFactory.Builder
extends Object

Fluent builder for AsyncExternalSourceOperatorFactory. Required parameters are captured via AsyncExternalSourceOperatorFactory.builder(StorageProvider, FormatReader, StoragePath, List, int, int, Executor). Optional parameters default to: FormatReader.NO_LIMIT for rowLimit, empty collections for partition/pushed-expression lists, null for opt-in hooks (sliceQueue, pushdownSupport, etc.), and 1 for parsingParallelism.

Method Summary

Modifier and Type	Method	Description
AsyncExternalSourceOperatorFactory	build()
AsyncExternalSourceOperatorFactory.Builder	datasetName(String datasetName)	Sets the dataset name surfaced to query rows via the _index metadata column.
AsyncExternalSourceOperatorFactory.Builder	deferredExtraction(boolean deferredExtraction)	Opt into deferred field extraction: the source emits a synthetic ColumnExtractor.ROW_POSITION_COLUMN column in encoded form (extractor id packed with file-local position) and registers a ColumnExtractor per opened file.
AsyncExternalSourceOperatorFactory.Builder	errorPolicy(ErrorPolicy errorPolicy)
AsyncExternalSourceOperatorFactory.Builder	externalSourceMetrics(ExternalSourceMetrics externalSourceMetrics)	Node telemetry sink for object-store read metrics; defaults to ExternalSourceMetrics.NOOP.
AsyncExternalSourceOperatorFactory.Builder	fileList(FileList fileList)
AsyncExternalSourceOperatorFactory.Builder	idPath(String idPath)	Declared _id.path (the logical name of the data column that supplies each row's _id), or null when the dataset declares no mappings._id.path.
AsyncExternalSourceOperatorFactory.Builder	lastModifiedMillis(Long lastModifiedMillis)	Sets the file last-modified epoch-millis used to materialise _version on the single-file producer paths.
AsyncExternalSourceOperatorFactory.Builder	maxConcurrentOpenSegments(int maxConcurrentOpenSegments)
AsyncExternalSourceOperatorFactory.Builder	maxRecordBytes(int maxRecordBytes)
AsyncExternalSourceOperatorFactory.Builder	onClose(Closeable onClose)
AsyncExternalSourceOperatorFactory.Builder	parallelism(int parallelism)
AsyncExternalSourceOperatorFactory.Builder	parsingParallelism(int parsingParallelism)
AsyncExternalSourceOperatorFactory.Builder	partitionColumnNames(Set<String> partitionColumnNames)
AsyncExternalSourceOperatorFactory.Builder	partitionValues(Map<String,Object> partitionValues)
AsyncExternalSourceOperatorFactory.Builder	producerBlockFactory(BlockFactory producerBlockFactory)	Sets the BlockFactory used by VirtualColumnIterator when materialising constant blocks for virtual partition / _file.* columns.
AsyncExternalSourceOperatorFactory.Builder	producerExecutor(Executor producerExecutor)	Executor for the page consumer (producer loop + drain), distinct from the read/parse executor passed to AsyncExternalSourceOperatorFactory.builder(org.elasticsearch.xpack.esql.datasources.spi.StorageProvider, org.elasticsearch.xpack.esql.datasources.spi.FormatReader, org.elasticsearch.xpack.esql.datasources.spi.StoragePath, java.util.List<org.elasticsearch.xpack.esql.core.expression.Attribute>, int, int, java.util.concurrent.Executor).
AsyncExternalSourceOperatorFactory.Builder	pushdownSupport(FilterPushdownSupport pushdownSupport)
AsyncExternalSourceOperatorFactory.Builder	pushedExpressions(List<Expression> pushedExpressions)
AsyncExternalSourceOperatorFactory.Builder	renames(Map<String,String> renames)	Declared logical->physical column renames; applied to reader-facing names at the last mile.
AsyncExternalSourceOperatorFactory.Builder	rowLimit(int rowLimit)
AsyncExternalSourceOperatorFactory.Builder	schemaMap(Map<StoragePath,SchemaReconciliation.FileSchemaInfo> schemaMap)
AsyncExternalSourceOperatorFactory.Builder	sliceQueue(ExternalSliceQueue sliceQueue)
AsyncExternalSourceOperatorFactory.Builder	statsColumnScope(StripeColumnScope statsColumnScope)	How much per-stripe statistics a fresh scan harvests (see ExternalSourceCacheSettings#STRIPE_COLUMNS and StripeColumnScope).
AsyncExternalSourceOperatorFactory.Builder	statsStripeSize(long statsStripeSize)	Canonical-stripe grid for per-stripe stats accounting, in decompressed-stream bytes (<= 0 disables — the default).
AsyncExternalSourceOperatorFactory.Builder	streamingSegmentatorAdmission(org.elasticsearch.xpack.esql.datasources.StreamingSegmentatorAdmission streamingSegmentatorAdmission)	Node-level gate bounding concurrent streaming segmentators on the shared esql_external_io pool (see StreamingSegmentatorAdmission).

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

producerExecutor

public AsyncExternalSourceOperatorFactory.Builder producerExecutor(@Nullable
 Executor producerExecutor)

Executor for the page consumer (producer loop + drain), distinct from the read/parse executor passed to AsyncExternalSourceOperatorFactory.builder(org.elasticsearch.xpack.esql.datasources.spi.StorageProvider, org.elasticsearch.xpack.esql.datasources.spi.FormatReader, org.elasticsearch.xpack.esql.datasources.spi.StoragePath, java.util.List<org.elasticsearch.xpack.esql.core.expression.Attribute>, int, int, java.util.concurrent.Executor). Production wires this to esql_worker while the read/parse executor is the esql_external_io pool, so a full I/O pool of blocked parser workers cannot starve the drain that consumes their pages. When unset (or null), the consumer shares the read/parse executor — the prior single-pool behavior, retained for tests and callers that pass one executor.

rowLimit

public AsyncExternalSourceOperatorFactory.Builder rowLimit(int rowLimit)

fileList

public AsyncExternalSourceOperatorFactory.Builder fileList(@Nullable
 FileList fileList)

schemaMap

public AsyncExternalSourceOperatorFactory.Builder schemaMap(@Nullable
 Map<StoragePath,SchemaReconciliation.FileSchemaInfo> schemaMap)

partitionColumnNames

public AsyncExternalSourceOperatorFactory.Builder partitionColumnNames(@Nullable
 Set<String> partitionColumnNames)

partitionValues

public AsyncExternalSourceOperatorFactory.Builder partitionValues(@Nullable
 Map<String,Object> partitionValues)

datasetName

public AsyncExternalSourceOperatorFactory.Builder datasetName(@Nullable
 String datasetName)

Sets the dataset name surfaced to query rows via the _index metadata column. null when the FROM did not resolve to a single registered dataset (e.g. bare-glob FROM "s3://bucket/*.parquet"); the _index column then renders as SQL NULL. Only consulted when the bound attributes include an ExternalMetadataAttribute named _index.

renames

public AsyncExternalSourceOperatorFactory.Builder renames(Map<String,String> renames)

Declared logical->physical column renames; applied to reader-facing names at the last mile.

idPath

public AsyncExternalSourceOperatorFactory.Builder idPath(@Nullable
 String idPath)

Declared _id.path (the logical name of the data column that supplies each row's _id), or null when the dataset declares no mappings._id.path. When set, VirtualColumnIterator stamps _id from that column instead of the synthetic (file+row-position) identity.

lastModifiedMillis

public AsyncExternalSourceOperatorFactory.Builder lastModifiedMillis(@Nullable
 Long lastModifiedMillis)

Sets the file last-modified epoch-millis used to materialise _version on the single-file producer paths. null (the default) leaves _version as SQL NULL on those paths. Only consulted when the source has no per-file mtime source — the slice-queue path reads _file.modified out of the FileSplit's partition values, and the multi-file path reads it off the FileList entry; both ignore this builder value.

producerBlockFactory

public AsyncExternalSourceOperatorFactory.Builder producerBlockFactory(@Nullable
 BlockFactory producerBlockFactory)

Sets the BlockFactory used by VirtualColumnIterator when materialising constant blocks for virtual partition / _file.* columns. Production should pass the node-level (root) factory so producer-thread allocations route through the global request circuit breaker rather than the driver-local breaker (which asserts single-thread access during the driver's run loop). Tests may leave this null to fall back to the driver context's factory.

sliceQueue

public AsyncExternalSourceOperatorFactory.Builder sliceQueue(@Nullable
 ExternalSliceQueue sliceQueue)

errorPolicy

public AsyncExternalSourceOperatorFactory.Builder errorPolicy(@Nullable
 ErrorPolicy errorPolicy)

parsingParallelism

public AsyncExternalSourceOperatorFactory.Builder parsingParallelism(int parsingParallelism)

maxConcurrentOpenSegments

public AsyncExternalSourceOperatorFactory.Builder maxConcurrentOpenSegments(int maxConcurrentOpenSegments)

maxRecordBytes

public AsyncExternalSourceOperatorFactory.Builder maxRecordBytes(int maxRecordBytes)

statsStripeSize

public AsyncExternalSourceOperatorFactory.Builder statsStripeSize(long statsStripeSize)

Canonical-stripe grid for per-stripe stats accounting, in decompressed-stream bytes (<= 0 disables — the default). See ExternalSourceCacheSettings#STRIPE_SIZE and StreamingParallelParsingCoordinator; stats-accounting overlay only, never a partitioning or scheduling input.

statsColumnScope

public AsyncExternalSourceOperatorFactory.Builder statsColumnScope(@Nullable
 StripeColumnScope statsColumnScope)

How much per-stripe statistics a fresh scan harvests (see ExternalSourceCacheSettings#STRIPE_COLUMNS and StripeColumnScope). null restores the StripeColumnScope.PROJECTED default. Orthogonal to statsStripeSize.

pushedExpressions

public AsyncExternalSourceOperatorFactory.Builder pushedExpressions(@Nullable
 List<Expression> pushedExpressions)

pushdownSupport

public AsyncExternalSourceOperatorFactory.Builder pushdownSupport(@Nullable
 FilterPushdownSupport pushdownSupport)

onClose

public AsyncExternalSourceOperatorFactory.Builder onClose(@Nullable
 Closeable onClose)

Parameters:

onClose - lifecycle callback owned by this factory, invoked exactly once when the last operator created by AsyncExternalSourceOperatorFactory.get(org.elasticsearch.compute.operator.DriverContext) completes (ref count drops to zero). Used by the per-source concurrency budget to deregister from the allocator. May be null when no per-source cleanup is needed. Callers must ensure that get() is called at least once; otherwise the callback never fires and the resource it guards leaks.

parallelism

public AsyncExternalSourceOperatorFactory.Builder parallelism(int parallelism)

deferredExtraction

public AsyncExternalSourceOperatorFactory.Builder deferredExtraction(boolean deferredExtraction)

Opt into deferred field extraction: the source emits a synthetic ColumnExtractor.ROW_POSITION_COLUMN column in encoded form (extractor id packed with file-local position) and registers a ColumnExtractor per opened file. The caller (typically InsertExternalFieldExtraction via LocalExecutionPlanner) pairs this with an ExternalFieldExtractOperator that resolves the extractor by id and materializes the deferred columns post-TopN.

The configured FormatReader must implement ColumnExtractorAware — verified at build() time.

externalSourceMetrics

public AsyncExternalSourceOperatorFactory.Builder externalSourceMetrics(ExternalSourceMetrics externalSourceMetrics)

Node telemetry sink for object-store read metrics; defaults to ExternalSourceMetrics.NOOP.

streamingSegmentatorAdmission

public AsyncExternalSourceOperatorFactory.Builder streamingSegmentatorAdmission(org.elasticsearch.xpack.esql.datasources.StreamingSegmentatorAdmission streamingSegmentatorAdmission)

Node-level gate bounding concurrent streaming segmentators on the shared esql_external_io pool (see StreamingSegmentatorAdmission). Set by FileSourceFactory from node settings; when unset the builder keeps its StreamingSegmentatorAdmission.unbounded() default.

build

public AsyncExternalSourceOperatorFactory build()