Package org.elasticsearch.xpack.esql.datasources

Class AsyncExternalSourceOperatorFactory

java.lang.Object
org.elasticsearch.xpack.esql.datasources.AsyncExternalSourceOperatorFactory
All Implemented Interfaces:
Describable, Operator.OperatorFactory, SourceOperator.SourceOperatorFactory, DeferredExtractionCapable
public class AsyncExternalSourceOperatorFactory extends Object implements SourceOperator.SourceOperatorFactory, DeferredExtractionCapable
Dual-mode async factory for creating source operators that read from external storage.

This factory automatically selects the optimal execution mode based on the FormatReader's capabilities:

Key design principles:

  • Simple things stay simple - CSV/JSON readers just implement sync read()
  • Async when beneficial - Parquet can override readAsync() for parallel I/O
  • ES ThreadPool integration - All executors come from ES, not standalone threads
  • Backpressure via buffer - Uses AsyncExternalSourceBuffer with waitForSpace()

Two executors, deliberately on different pools so the parser workers and the consumer that drains their pages never contend for the same threads:

  • executor — the read/parse pool. Runs the blocking file opens (length(), computeSegments) and the segment parser workers. Sourced from SourceOperatorContext.fileReadExecutor — the dedicated esql_external_io pool — falling back to context.executor() when unset.
  • producerExecutor — the consumer pool. Runs the (non-blocking) producer/drain loop that consumes the parser workers' pages into the buffer. Sourced from context.executor() — esql_worker, the same compute pool whose drivers AsyncExternalSourceBuffer.pollPage() — falling back to executor when unset (single-pool test callers).
A full read/parse pool of blocked parser workers therefore can never starve the drain that must consume their pages (the multi-file parallel-parse stall). The drain is non-blocking: it runs synchronously while the buffer has space and yields when full, resuming via producerExecutor when space is freed.
See Also: