Package org.elasticsearch.xpack.esql.datasources

Class SchemaReconciliation

java.lang.Object
org.elasticsearch.xpack.esql.datasources.SchemaReconciliation
public final class SchemaReconciliation extends Object
Schema reconciliation algorithms for multi-file external sources.

Supports three strategies:

  • FIRST_FILE_WINS — use the first file's schema (existing behavior, no reconciliation)
  • STRICT — validate all files share the exact same schema
  • UNION_BY_NAME — merge schemas by column name with safe type widening

Type widening is intentionally conservative: only lossless promotions are allowed. This is NOT EsqlDataTypeConverter.commonType(), which allows LONG→DOUBLE (lossy above 2^53).

Under UNION_BY_NAME, any pair the lossless table cannot widen falls back to DataType.KEYWORD (the cross-type join): values from numerically-typed files are stringified via ColumnMapping's per-block cast and a single response Warning header per affected column tells the user what happened. This matches the industry baseline (DuckDB, ClickHouse, Spark all widen to string as the cross-type floor) and turns "samplers disagreed" — the normal steady state for sampling-based readers — from a hard error into a benign widening. Users who want the strict-mismatch error can opt into schema_resolution = "strict" which still throws.

The lossy LONG + DOUBLE pair is *not* covered by the lossless table on purpose (precision loss above 2^53). Under UBN it goes to KEYWORD, which is louder and safer than silent precision loss; the lossless table itself stays unchanged.

The four schemas in an external-source query

Four distinct schemas exist in every external-source query. In simpler modes (single file, FFW, STRICT) some collapse onto each other; under UNION_BY_NAME all four are genuinely distinct. Code touching FileSplit.readSchema(), ExternalSourceExec.attributes, or ColumnMapping reads much more clearly with these names in mind:
File schema (per-file, file shape)
What's literally in one file. Parquet/ORC: read from the file footer. CSV/NDJSON: inferred from a byte sample. Carried per-file on FileSplit.readSchema().
Unified schema (one for the whole table)
The cross-file harmonized schema. Produced here as SchemaReconciliation.Result.unifiedSchema(): FFW takes the anchor file's schema, STRICT validates a common schema, UBN takes the column-name union with type widening. Becomes ExternalSourceExec.attributes at first, before the optimizer's projection pruning rewrites that field.
Query schema (unified shape; same for every file in the query)
The subset of unified schema the query actually materializes after projection pruning. Lives on ExternalSourceExec.attributes on the wire. Drives the per-file ColumnMapping after ColumnMapping.pruneToPerFileQuery(org.elasticsearch.xpack.esql.datasources.ExternalSchema, org.elasticsearch.xpack.esql.datasources.ExternalSchema, org.elasticsearch.xpack.esql.datasources.ExternalSchema).
Per-file query schema (per-file, file shape — what the reader actually produces)
Query schema ∩ this file's columns, ordered to match the file's natural layout. Derived per file at split-construction time and at read time. Under FFW and STRICT it collapses to the Query schema because every file has every projected column.

Worked example (UNION_BY_NAME)

   a.csv = [name:keyword, age:int]
   b.csv = [age:long, name:keyword, city:keyword]
   query: EXTERNAL "*.csv" WITH {"schema_resolution": "union_by_name"}
          | KEEP name, city
          | SORT name

   File schema:           a → [name:keyword, age:int]
                          b → [age:long, name:keyword, city:keyword]
   Unified schema:        [name:keyword, age:long, city:keyword]  (age widens int → long)
   Query schema:          [name:keyword, city:keyword]            (KEEP drops age)
   Per-file query schema: a → [name]                              (no city in a)
                          b → [name, city]                        (in b's natural order)

  • Method Details

    • schemaWiden

      @Nullable public static DataType schemaWiden(DataType a, DataType b)
      Safe type widening for schema reconciliation. Only lossless promotions are allowed; returns null if no safe supertype exists.

      Widening rules:

      • INTEGER + LONG → LONG (lossless: int32 ⊆ int64)
      • INTEGER + DOUBLE → DOUBLE (lossless: int32 ≤ 2^31 < 2^53)
      • DATETIME + DATE_NANOS → DATE_NANOS (more precise type wins)
      All other cross-type pairs return null (no lossless supertype). UBN reconciliation additionally falls back to DataType.KEYWORD for those — see widenToCommonOrKeyword(org.elasticsearch.xpack.esql.core.type.DataType, org.elasticsearch.xpack.esql.core.type.DataType) and reconcileUnionByName(java.util.Map<org.elasticsearch.xpack.esql.datasources.spi.StoragePath, org.elasticsearch.xpack.esql.datasources.spi.SourceMetadata>). LONG + DOUBLE deliberately stays out of this table (precision loss above 2^53) and is therefore one of the pairs that goes to KEYWORD under UBN.
      Returns:
      the widened type, or null if no safe supertype exists

    • reconcileStrict

      public static SchemaReconciliation.Result reconcileStrict(StoragePath referenceFile, Map<StoragePath,SourceMetadata> fileMetadata)
      STRICT reconciliation: validate all files share the exact same schema. Nullability differences are tolerated; all other differences produce an error.
      Parameters:
      referenceFile - path of the first (reference) file
      fileMetadata - ordered map of file path → metadata (first entry is the reference)
      Returns:
      reconciliation result with the reference schema and per-file info
      Throws:
      IllegalArgumentException - if any file's schema doesn't match

    • reconcileUnionByName

      public static SchemaReconciliation.Result reconcileUnionByName(Map<StoragePath,SourceMetadata> fileMetadata)
      UNION_BY_NAME reconciliation: merge schemas from all files into a superset. Missing columns are NULL-filled; type differences are resolved by safe widening or, when no lossless supertype exists, by falling back to DataType.KEYWORD with a per-column Warning response header. See the class javadoc for the rationale and the lattice picture.
      Parameters:
      fileMetadata - ordered map of file path → metadata (insertion order = file sort order)
      Returns:
      reconciliation result with unified schema and per-file mappings