Module org.elasticsearch.compute
Package org.elasticsearch.compute.data

Class Utf8Sanitizer

java.lang.Object
org.elasticsearch.compute.data.Utf8Sanitizer
public final class Utf8Sanitizer extends Object
Validates and, if necessary, repairs UTF-8 byte sequences that enter ES|QL as KEYWORD values from external sources (Parquet, ORC, Arrow).

ES|QL assumes every KEYWORD/TEXT BytesRef holds well-formed UTF-8: for example Utf8AscTopNEncoder/Utf8DescTopNEncoder index a 248-entry code-length table by the raw lead byte, so a byte in 0xF8..0xFF throws ArrayIndexOutOfBoundsException on the sort path. External writers (notably Spark) can emit malformed UTF-8 in string-annotated columns, so the byte hygiene ES|QL relies on cannot be assumed at the source boundary.

sanitize(BytesRef) returns the input unchanged when it is already well-formed (the common, fast path with no allocation) and otherwise returns a copy in which each maximal ill-formed subsequence is replaced by a single Unicode replacement character U+FFFD (EF BF BD). The "substitution of maximal subparts" behaviour matches the Unicode recommendation and the output of ICU / Spark, so downstream KEYWORD operations become total without silently dropping or merging distinct malformed inputs.

  • Method Summary

    Modifier and Type
    Method
    Description
    static boolean
    isWellFormed(byte[] bytes, int off, int len)
    Allocation-free scan reporting whether [off, off+len) is well-formed UTF-8.
    static org.apache.lucene.util.BytesRef
    sanitize(org.apache.lucene.util.BytesRef in)
    Returns in unchanged when it is well-formed UTF-8, otherwise a new BytesRef (offset 0) with every maximal ill-formed subsequence replaced by U+FFFD.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Method Details

    • isWellFormed

      public static boolean isWellFormed(byte[] bytes, int off, int len)
      Allocation-free scan reporting whether [off, off+len) is well-formed UTF-8.

    • sanitize

      public static org.apache.lucene.util.BytesRef sanitize(org.apache.lucene.util.BytesRef in)
      Returns in unchanged when it is well-formed UTF-8, otherwise a new BytesRef (offset 0) with every maximal ill-formed subsequence replaced by U+FFFD.