Package org.elasticsearch.xpack.esql.plan.physical

Class ExternalSourceExec

java.lang.Object
org.elasticsearch.xpack.esql.core.tree.Node<PhysicalPlan>
org.elasticsearch.xpack.esql.plan.QueryPlan<PhysicalPlan>
org.elasticsearch.xpack.esql.plan.physical.PhysicalPlan
org.elasticsearch.xpack.esql.plan.physical.LeafExec
org.elasticsearch.xpack.esql.plan.physical.ExternalSourceExec
All Implemented Interfaces:
NamedWriteable, Writeable, DataSourceExec, EstimatesRowSize
public class ExternalSourceExec extends LeafExec implements EstimatesRowSize, DataSourceExec
Generic physical plan node for reading from external data sources (e.g., Iceberg tables, Parquet files).

This is the unified physical plan node for all external sources, replacing source-specific nodes. It uses generic maps for configuration and metadata to avoid leaking source-specific types (like S3Configuration) into core ESQL code.

Key design principles:

  • Generic configuration: Uses Map<String, Object> for config instead of source-specific classes like S3Configuration
  • Opaque metadata: Source-specific data (native schema, etc.) is stored in sourceMetadata() and passed through without core understanding it
  • Coordinator plan never holds it: the coordinator Mapper leaves the source as FragmentExec(ExternalRelation); this exec is materialized per-node by LocalMapper from ExternalRelation.toPhysicalExec() during PlannerUtils.localPlan(), on whichever node runs the compute. It is a serializable NamedWriteable (see writeTo(org.elasticsearch.common.io.stream.StreamOutput) / readFrom(org.elasticsearch.common.io.stream.StreamInput)) so it can ride the wire, but only its execution inputs survive that round trip; see the field categories below.

Field categories. The fields fall into three groups, and only the first is serialized:

  • Execution inputs (serialized): sourcePath, sourceType, attributes, config, sourceMetadata, estimatedRowSize, splits, datasetName. These are what a data-node operator needs to read; in the distributed path the data node reads the files via the serialized splits.
  • Build-time state (not serialized): fileList, schemaMap, unifiedSchema. These are coordinator-resolved and read off the exec by split discovery and planExternalSource. fileList and schemaMap originate on ExternalRelation and are coordinator-only (a data node deserializes the relation with FileList.UNRESOLVED / an empty schemaMap, so toPhysicalExec() does not rebuild them there); the data node relies on the serialized splits instead. unifiedSchema is the exception: toPhysicalExec() rebuilds it on every node from the serialized metadata.schema().
  • Local-execution pushdown hints (not serialized): pushedFilter, pushedExpressions, pushedLimit, pushedTopN, deferredExtraction. These are decisions produced by LocalPhysicalPlanOptimizer rules that run after this exec is created, independently on each node. They are intentionally re-derived per-node and must not be serialized.
When adding a field: if a data-node operator consumes it directly, it is an execution input and must be serialized; if it derives from the relation or from local physical optimization, it is transient and stays out of writeTo(org.elasticsearch.common.io.stream.StreamOutput).