Package org.elasticsearch.xpack.esql.datasources

Class SourceStatisticsSerializer

java.lang.Object
org.elasticsearch.xpack.esql.datasources.SourceStatisticsSerializer
public final class SourceStatisticsSerializer extends Object
Serializes and deserializes SourceStatistics to/from a flat Map<String, Object> using well-known keys. This allows statistics to flow through the opaque sourceMetadata map in ExternalSourceExec without requiring new fields or serialization format changes.

  • Field Details

    • STATS_KEY_PREFIX

      public static final String STATS_KEY_PREFIX
      Common prefix of every flat statistics key (row count, size, per-column stats, partial flag).
      See Also:

    • STATS_ROW_COUNT

      public static final String STATS_ROW_COUNT
      See Also:

    • STATS_SIZE_BYTES

      public static final String STATS_SIZE_BYTES
      See Also:

    • STATS_PARTIAL

      public static final String STATS_PARTIAL
      When set to true in sourceMetadata, indicates that the statistics are derived from a single anchor file in a multi-file glob query (FIRST_FILE_WINS schema resolution) and do not represent the full dataset. This flag is set by ExternalSourceResolver.markStatsAsPartial when a glob matches more than one file.

      The aggregate pushdown rule (PushStatsToExternalSource) checks this flag via SplitStats.resolveEffectiveStats and bails out when set. Note that once per-split statistics are available (populated during split discovery), the merged per-split stats take precedence and this flag is not consulted.

      See Also:

    • STATS_FILE_COUNT

      public static final String STATS_FILE_COUNT
      Number of files matched by the glob pattern; useful for observability and debugging.
      See Also:

    • STATS_COL_PREFIX

      public static final String STATS_COL_PREFIX
      See Also:

    • PARTITION_COLUMNS_KEY

      public static final String PARTITION_COLUMNS_KEY
      Names of the Hive-partition (path-derived) columns, stamped here at resolution so the DATA-NODE fold can recognize them: the coordinator-only FileList (which carries PartitionMetadata) deserializes to UNRESOLVED on a remote node, but sourceMetadata travels with the serialized relation. A partition column is absent from every file's column stats, so without this signal a data-node COUNT(partition_col) would serve rowCount - rowCount = 0. Deliberately NOT under the _stats. prefix so the FirstFileWins / union-by-name stat merges preserve it. Value: List<String>.
      See Also:

  • Method Details

    • embedStatistics

      public static Map<String,Object> embedStatistics(Map<String,Object> sourceMetadata, SourceStatistics statistics)
      Merges statistics entries into a new map that includes both the original sourceMetadata entries and the serialized statistics. Returns the original map if statistics are absent.

    • extractStatistics

      public static Optional<SourceStatistics> extractStatistics(Map<String,Object> sourceMetadata)
      Extracts statistics from the sourceMetadata map. Returns empty if no statistics keys are present.

    • extractRowCount

      @Nullable public static Long extractRowCount(Map<String,Object> sourceMetadata)
      Extracts the row count directly from the sourceMetadata map. Returns null if the metadata is null or the row count is absent/non-numeric.

    • extractColumnNullCount

      @Nullable public static Long extractColumnNullCount(Map<String,Object> sourceMetadata, String columnName)
      Extracts the null count for a specific column directly from the sourceMetadata map. Returns null if the metadata is null or the null count is absent/non-numeric.

    • extractColumnValueCount

      @Nullable public static Long extractColumnValueCount(Map<String,Object> sourceMetadata, String columnName)
      Extracts the value count (count of non-null values) for a specific column directly from the sourceMetadata map. Returns null if the metadata is null or the value count is absent/non-numeric.

    • extractColumnMin

      @Nullable public static Object extractColumnMin(Map<String,Object> sourceMetadata, String columnName)
      Extracts the min value for a specific column directly from the sourceMetadata map. Returns null if the metadata is null or the min is absent.

    • extractColumnMax

      @Nullable public static Object extractColumnMax(Map<String,Object> sourceMetadata, String columnName)
      Extracts the max value for a specific column directly from the sourceMetadata map. Returns null if the metadata is null or the max is absent.

    • columnNullCountKey

      public static String columnNullCountKey(String columnName)
      Returns the flat key used for a column's null count statistic.

    • columnValueCountKey

      public static String columnValueCountKey(String columnName)
      Returns the flat key used for a column's value count (non-null values) statistic.

    • columnMinKey

      public static String columnMinKey(String columnName)
      Returns the flat key used for a column's min statistic.

    • columnMaxKey

      public static String columnMaxKey(String columnName)
      Returns the flat key used for a column's max statistic.

    • columnSizeBytesKey

      public static String columnSizeBytesKey(String columnName)
      Returns the flat key used for a column's size in bytes statistic.

    • columnMinUnservableKey

      public static String columnMinUnservableKey(String columnName)
      Returns the flat key that marks a column's min statistic unservable.

    • columnMaxUnservableKey

      public static String columnMaxUnservableKey(String columnName)
      Returns the flat key that marks a column's max statistic unservable.

    • poisonColumnExtrema

      public static void poisonColumnExtrema(Map<String,Object> statsMap, String columnName)
      Poisons a column's min/max in-place: drops the extremum values and writes the unservable markers so the column safe-misses to a scan. Count stats (row/null/value counts) are left intact. Used when the extremum cannot be trusted — e.g. the FIRST_FILE_WINS fold detects a column whose physical type diverges across files, so both the unit-blind fold AND the anchor-schema misread of the divergent file make a warm extremum unable to match a scan.

    • overlayDeclaredSchemaOnStats

      public static Map<String,Object> overlayDeclaredSchemaOnStats(Map<String,Object> statsMap, Map<String,String> physicalToLogical, Set<String> poisonColumns)
      The declared-schema overlay's stats boundary — the fourth, after reconciliation-normalize, FFW-divergence poison, and commit-time coercion. Stats are produced keyed by physical (file) column names holding inferred-type values; the declared overlay renames/retypes the plan afterwards, so without this the warm path serves physical-keyed stats under logical names (a renamed COUNT(col) serves 0) and inferred-type extrema/counts a coerced scan never produces. This (1) REKEYS every per-column stat family physical→logical for each path rename — a pure move changes no value, so the rekeyed stats stay exactly correct and warm serving survives the rename; (2) POISONS the extrema and DROPS value_count/null_count for poisonColumns (declared retype or declared date format — read-time coercion can null cells and re-represent values, so no pre-coercion stat is trustworthy). row_count/file_count/size_bytes and non-column keys are untouched, so COUNT(*) stays warm and the cost estimator keeps its byte signal. Returns the input instance unchanged when there is nothing to do; otherwise a new map (inputs are routinely Map.copyOf-immutable).
      Parameters:
      physicalToLogical - physical→logical name moves (the inverse of DeclaredReadSpec#renames; 1:1 by validation, so the inverse is well-defined)
      poisonColumns - LOGICAL names whose extrema + counts must safe-miss

    • compareKeywordUtf8

      public static int compareKeywordUtf8(Object a, Object b)
      Compares two keyword/text stat extrema in UTF-8 byte order — the SAME order the runtime keyword MIN/MAX aggregators and comparisons use (BytesRef unsigned-byte order) — NOT String.compareTo(java.lang.String)'s UTF-16 code-unit order, which disagrees for supplementary (astral) chars vs BMP chars in [U+E000..U+FFFF]. Accepts either representation a stat value takes: String (parquet footer) or BytesRef (text harvest). Single owner of keyword-stat ordering: the cross-file fold (SplitStats.mergedMin/mergedMax), the split-filter classifier (StatValueComparator), and the parquet cross-row-group fold (ParquetFormatReader) all delegate here.

    • normalizeStatsToReconciled

      public static Map<String,Object> normalizeStatsToReconciled(Map<String,Object> statsMap, Map<String,DataType> fileTypes, Map<String,DataType> reconciledTypes)
      Normalizes a per-file stat map's min/max to the RECONCILED column type, at the boundary where BOTH the file's own type and the multi-file reconciled type are known (multi-file discovery / split construction). Per-file stats are stored in the file's LOCAL unit/representation, but every warm consumer — the split-filter classifier, the filtered/whole-file merge, the source-level fold, and the MIN/MAX serve — reads the value AS the reconciled type (af.dataType()) with no further rescale. Normalizing here, once, is what makes those consumers correct instead of comparing file-local units unit-blind. Two cases need it (the numeric Long/Double flap within one representation is handled separately by the cache-path coerceColumnStatsToResolvedTypes and the poison fold):
      • Temporal widening — a DATETIME (epoch-millis) file column reconciled to DATE_NANOS (epoch-nanos) has its min/max rescaled ×1e6 (Math.multiplyExact(int, int)); on overflow the value is dropped and the unservable marker written (safe-miss), never a wrong nanos value.
      • Representation change — a numeric/temporal file column reconciled to KEYWORD/TEXT (SchemaReconciliation's non-widenable fallback) would be served under lexicographic/stringified order, not numeric, so its numeric min/max is dropped and the marker written (safe-miss).
      Count stats (value_count/null_count/row_count) are unit- and representation-independent and pass through. The unservable marker (not a bare removal) is written so marker-wins normalization in SplitStats.of forces a safe-miss even if a later overlay would otherwise resurrect a stale value. Returns the input unchanged when no column needed normalization.

    • extractColumnSizeBytes

      @Nullable public static Long extractColumnSizeBytes(Map<String,Object> sourceMetadata, String columnName)
      Extracts the size in bytes for a specific column directly from the sourceMetadata map. Returns null if the metadata is null or the size is absent/non-numeric.

    • mergeStatistics

      public static Map<String,Object> mergeStatistics(List<Map<String,Object>> splitStats)
      Merges per-file _stats.* maps into a single dataset-wide map.

      Implements the "implicit nulls" contract for UNION_BY_NAME aggregate pushdown: a column absent from a per-file map (no _stats.columns.<col>.* keys at all) means the column is physically absent from that file, so its entire row count is folded into the merged null_count accumulator for that column. This makes Count(col) = totalRowCount - mergedNullCount correct downstream in PushStatsToExternalSource.

      Format-reader ground truth: Parquet always writes size_bytes for present columns and ORC always writes null_count, so any column-family key in a per-file map is sufficient to mark the column as physically present in that file. The rare exception is Parquet writing a column with stats disabled — present, with size_bytes, but no null_count. We refuse to fabricate a null count in that case: the cross-file fold marks the column's null_count poisoned and drops the entry, so downstream consumers see "unknown" and fall back rather than under-count.

      Min/max/size_bytes accumulators are unchanged: they only sum across files where the column is present, which is the correct semantics regardless of implicit nulls.

    • mergeStatistics

      public static Map<String,Object> mergeStatistics(List<Map<String,Object>> splitStats, boolean implicitNullsForAbsentColumn)
      Parameters:
      implicitNullsForAbsentColumn - when true (footer formats), a column absent from a per-file map is treated as physically absent and its rows fold into the merged null_count (UNION_BY_NAME semantics). When false (text formats under partial harvest), a column absent from any file's stats is "not harvested" -- it may be physically present -- so the merged column is dropped entirely, forcing downstream COUNT/MIN/MAX to safe-miss (re-scan) rather than undercount or serve a subset extremum.