Package org.elasticsearch.xpack.esql.datasources

Class ExternalRowIdentity

java.lang.Object
org.elasticsearch.xpack.esql.datasources.ExternalRowIdentity
public final class ExternalRowIdentity extends Object
Per-page composition of the _id metadata column for external datasets. The composed value is opaque: base64url(murmur3_128(location).h1 | mtime | rowPosition) — 24 identity bytes rendered as a fixed 32-character URL-safe string. The location (storage path) is hashed, never rendered: file URIs are an implementation detail of the dataset layout and must not leak through _id into Kibana row keys, Security alert hashes, or logs. Same composition as the house precedent for synthesized ids, TsidExtractingIdFieldMapper#createId: murmur3-128 the high-entropy variable part, pack the scalar parts alongside, base64url-encode without padding. Only the high 64 bits (h1) of the 128-bit location hash are packed, so cross-file _id uniqueness is probabilistic at 64 bits of location entropy (plus mtime and row position) — the same birthday-bound trade-off the TSID precedent accepts, not a guarantee.

The packed mtime is the identity salt: a file replaced in place under the same name produces ids distinct from its predecessor's — without it, a consumer caching by _id would silently conflate rows from two different file generations. mtime == 0 is the FileList convention for "storage layer reported none" and packs as zero; _id stays well-formed and the honest unknown surfaces through _version's null.

The row position is masked off the optional ColumnExtractor.LOCAL_POSITION_BITS-encoded extractor id used by the deferred-extraction path before it enters the identity bytes, so the same physical row composes the same _id regardless of extraction strategy.

Allocation discipline: one 16-byte BytesRef per file for the identity prefix; one exactly-sized byte[] plus one int[] of offsets per page; per-row packing and rendering run on two stack-resident scratch buffers, zero per-row allocation.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final int
    RENDERED_LENGTH
    Rendered length: base64url of 24 bytes — no padding because IDENTITY_BYTES % 3 == 0 — is exactly 32 characters.

  • Method Summary

    Modifier and Type
    Method
    Description
    static BytesRefBlock
    composePage(org.apache.lucene.util.BytesRef prefix, LongBlock rowPositionBlock, BlockFactory factory)
    Compose an _id block for one page.
    static org.apache.lucene.util.BytesRef
    prefix(StoragePath path, long mtimeMillis)
    Build the per-file identity prefix: [murmur3_128(locationUtf8).h1 BE | mtimeMillis BE], 16 bytes.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • RENDERED_LENGTH

      public static final int RENDERED_LENGTH
      Rendered length: base64url of 24 bytes — no padding because IDENTITY_BYTES % 3 == 0 — is exactly 32 characters.
      See Also:

  • Method Details

    • prefix

      public static org.apache.lucene.util.BytesRef prefix(StoragePath path, long mtimeMillis)
      Build the per-file identity prefix: [murmur3_128(locationUtf8).h1 BE | mtimeMillis BE], 16 bytes. One allocation per file; the returned BytesRef is held by the producer iterator for the lifetime of the file and reused across every page. Only the first 8 hash bytes are kept — the truncation TsidExtractingIdFieldMapper applies to the tsid hash; the packed mtime and row position keep same-file rows distinct regardless. mtimeMillis is the file's last-modified epoch millis; callers pass 0 when the storage layer reports none (the FileList convention for a missing mtime).

    • composePage

      public static BytesRefBlock composePage(org.apache.lucene.util.BytesRef prefix, LongBlock rowPositionBlock, BlockFactory factory)
      Compose an _id block for one page. Output position count matches rowPositionBlock.getPositionCount(); null row-positions yield null _id. The block allocates against factory so its breaker bytes follow the producer-thread accounting path used by other constant-block allocations. Two-pass design: pass 1 walks the row-position block, packs prefix + physicalPosition into the identity scratch and base64url-renders each row directly into a single producer-side backing array while recording per-row offsets; pass 2 walks offsets[] and feeds an (offset, length) scratch view per row to the vector / block builder.