Package org.elasticsearch.xpack.esql.plan.logical.join

Class AbstractSubqueryJoin

java.lang.Object
org.elasticsearch.xpack.esql.core.tree.Node<LogicalPlan>
org.elasticsearch.xpack.esql.plan.QueryPlan<LogicalPlan>
org.elasticsearch.xpack.esql.plan.logical.LogicalPlan
org.elasticsearch.xpack.esql.plan.logical.BinaryPlan
org.elasticsearch.xpack.esql.plan.logical.join.Join
org.elasticsearch.xpack.esql.plan.logical.join.AbstractSubqueryJoin
All Implemented Interfaces:
NamedWriteable, Writeable, PostAnalysisVerificationAware, PostOptimizationVerificationAware, Resolvable, ExecutesOn, ExecutesOn.Coordinator, SortAgnostic, SortPreserving
Direct Known Subclasses:
AntiJoin, MarkJoin, SemiJoin
public abstract class AbstractSubqueryJoin extends Join implements SortPreserving, ExecutesOn.Coordinator
Shared base for the joins that implement field IN (subquery) and field NOT IN (subquery): SemiJoin (SEMI), AntiJoin (ANTI) and MarkJoin (MARK). The three are siblings — they share this dedup pipeline but differ in the terminal plan they produce.

The right side is an independent subquery that must be executed first. Once the subquery result is available as a LocalRelation, inlineData(org.elasticsearch.xpack.esql.plan.logical.join.AbstractSubqueryJoin, org.elasticsearch.xpack.esql.plan.logical.local.LocalRelation, int, org.elasticsearch.compute.data.BlockFactory, java.util.concurrent.atomic.AtomicReference<org.elasticsearch.compute.data.Page>) converts the node into an In list predicate or a hash join. The SEMI / ANTI / MARK differences are isolated into the small set of protected hooks below; the default implementations here encode the SEMI (IN) behavior, AntiJoin flips a couple of hooks for NOT IN, and MarkJoin returns Eval-based plans that preserve every left row and produce a boolean mark attribute with three-valued logic.

  • Constructor Details

  • Method Details

    • writeTo

      public void writeTo(StreamOutput out) throws IOException
      Specified by:
      writeTo in interface Writeable
      Overrides:
      writeTo in class Join
      Throws:
      IOException

    • getWriteableName

      public String getWriteableName()
      Specified by:
      getWriteableName in interface NamedWriteable
      Overrides:
      getWriteableName in class Join

    • computeOutputExpressions

      public List<NamedExpression> computeOutputExpressions(List<? extends NamedExpression> left, List<? extends NamedExpression> right)
      Overrides:
      computeOutputExpressions in class Join

    • expressionsResolved

      public boolean expressionsResolved()
      Overrides:
      expressionsResolved in class Join

    • buildEmptyRightSidePlan

      protected abstract LogicalPlan buildEmptyRightSidePlan(Source source)
      Build the terminal plan when the right side has zero rows. x IN () ≡ FALSE for SEMI / MARK; x NOT IN () ≡ TRUE for ANTI. MarkJoin produces an Eval that sets the mark attribute to FALSE.

    • buildShortCircuitPlan

      protected LogicalPlan buildShortCircuitPlan(Source source, boolean allRightNull)
      Build the terminal plan when the right side contains a NULL value that forces the predicate to a constant for every left row. For SEMI this only fires when every right value is NULL (allRightNull == true) and produces Filter(FALSE). For ANTI any NULL on the right is fatal, so it overrides shortCircuitOnAnyRightNull() and likewise produces Filter(FALSE). MarkJoin does not short-circuit on any NULL — it keeps the row counts unchanged and emits Eval($m = NULL) when every right value is NULL.

    • shortCircuitOnAnyRightNull

      protected boolean shortCircuitOnAnyRightNull()
      Whether any NULL on the right side makes the predicate non-TRUE for every left row. For ANTI (x NOT IN (..., NULL, ...)) this is always the case, so once inlineData(org.elasticsearch.xpack.esql.plan.logical.join.AbstractSubqueryJoin, org.elasticsearch.xpack.esql.plan.logical.local.LocalRelation, int, org.elasticsearch.compute.data.BlockFactory, java.util.concurrent.atomic.AtomicReference<org.elasticsearch.compute.data.Page>) detects a NULL position in the BlockHash dedup output it short-circuits to Filter(FALSE). For SEMI the NULL stays in the dedup output for the filter path (x IN (a, b, NULL) ≡ x IN (a, b) under WHERE semantics) and is stripped before constructing the LocalRelation on the hash-join path. For MARK the rightHadNulls flag is forwarded to the hash-join path's CASE expression; the filter path uses the NULL position in the dedup output directly.

    • buildFilterPathPlan

      protected LogicalPlan buildFilterPathPlan(Block dedupKeys, DataType keyType, Attribute leftField, Source source, boolean rightHadNulls)
      Build the terminal plan for the small-dedup "filter" path. dedupKeys contains the BlockHash-deduplicated distinct values from the right side and may include a single NULL position at index 0 if any input position was NULL or multi-valued (BlockHash collapses all NULL inputs into its reserved group 0). rightHadNulls is equivalent to dedupKeys.isNull(0) and is passed through for documentation / hash-join-path symmetry. SEMI returns Filter(In(...)), ANTI returns Filter(NOT In(...)) (ANTI never reaches this path with rightHadNulls = true — it short-circuits earlier), and MarkJoin builds an Eval whose IN list is the dedup positions verbatim, relying on In's three-valued semantics to compute the mark.

    • inListFromDedupKeys

      protected static In inListFromDedupKeys(Source source, Attribute leftField, Block dedupKeys, DataType keyType)
      Build the In expression for the filter path by turning each BlockHash dedup position into a Literal. A NULL position (BlockHash's reserved group 0) becomes a NULL literal naturally, so In's three-valued semantics carry the correct predicate/mark value into the surrounding Filter (SEMI/ANTI) or Eval (MARK).

    • buildHashJoinPathPlan

      protected LogicalPlan buildHashJoinPathPlan(LogicalPlan leftSide, LocalRelation deduplicatedData, JoinConfig leftJoinConfig, Attribute sentinelAttr, Source source, boolean rightHadNulls)
      Build the terminal plan for the large-dedup "hash-join" path. The deduplicated key column has already been wrapped in a LocalRelation alongside a constant TRUE sentinel column. SEMI/ANTI add a sentinel filter and Project that drops the right-side column; MarkJoin instead computes the mark via a CASE expression and projects the mark column out alongside the original left output.

    • wrapInExpression

      protected Expression wrapInExpression(Source source, Expression in)
      Wraps the In expression built from the dedup keys for the inline-filter path. SEMI returns it as-is; ANTI wraps it in Not.

    • sentinelFilterCondition

      protected Expression sentinelFilterCondition(Source source, Attribute sentinel)
      Sentinel-column filter that drives the hash-join path. SEMI keeps matched rows (IS NOT NULL on the sentinel); ANTI keeps unmatched rows (IS NULL).

    • filterNullLeftKeysBeforeHashJoin

      protected boolean filterNullLeftKeysBeforeHashJoin()
      Whether the hash-join path needs to drop NULL-keyed left rows before the join. SEMI/ANTI filter on the sentinel after the join; ANTI in particular would otherwise keep NULL-keyed rows (sentinel NULL → "no match") even though NULL NOT IN (...) should yield NULL. MARK keeps every left row and handles the NULL-key case explicitly inside the CASE expression that produces the mark, so it suppresses the filter.

    • postAnalysisVerification

      public void postAnalysisVerification(Failures failures)
      Description copied from interface: PostAnalysisVerificationAware
      Allows the implementer to validate itself. This usually involves checking its internal setup, which often means checking the parameters it received on construction: their data or syntactic type, class, their count, expressions' structure etc. The discovered failures are added to the given Failures object.

      It is often more useful to perform the checks as extended as it makes sense, over stopping at the first failure. This will allow the author to progress faster to a correct query.

      Example: the Filter class, which models the WHERE command, checks that the expression it filters on - condition - is of a Boolean or NULL type: 

           
     @Override
     void postAnalysisVerification(Failures failures) {
          if (condition.dataType() != NULL && condition.dataType() != BOOLEAN) {
              failures.add(fail(condition, "Condition expression needs to be boolean, found [{}]", condition.dataType()));
          }
     }
      Specified by:
      postAnalysisVerification in interface PostAnalysisVerificationAware
      Overrides:
      postAnalysisVerification in class Join
      Parameters:
      failures - the object to add failures to.

    • isSubqueryJoinUnsupported

      public static boolean isSubqueryJoinUnsupported(DataType type)
      Same as Join.UNSUPPORTED_TYPES but TEXT and VERSION are allowed in IN/NOT IN subquery joins because IN/NOT IN can compare these types via equality.

    • inlineData

      public static LogicalPlan inlineData(AbstractSubqueryJoin subqueryJoin, LocalRelation data, int hashJoinThreshold, BlockFactory blockFactory, AtomicReference<Page> pageHolder)
      Converts this subquery join (SEMI / ANTI / MARK) into an executable plan once the subquery result is available.

      The subquery key column is always run through a BlockHash (backed by the supplied blockFactory, so the dedup blocks are tracked by the request circuit breaker) before deciding the output shape. Doing dedup up-front lets duplicate-heavy subqueries collapse to the cheaper inline-filter path even when their raw position count exceeds the threshold, and it shrinks the IN list (and therefore both the planning and runtime cost) for the filter path itself.

      Once dedup is done:

      • If the deduplicated key count is <= hashJoinThreshold, the dedup keys are converted to Literals and the join becomes Filter(IN(...)) (or Filter(NOT IN(...)) for ANTI). The dedup blocks are released immediately.
      • Otherwise, the dedup keys plus a constant TRUE sentinel become a LocalRelation on the right of a LEFT Join; a Filter on the sentinel (IS NOT NULL for SEMI, IS NULL for ANTI) and a Project that drops the right-side column complete the rewrite. The mapper turns this into a HashJoinExec.

      SQL NULL / MV semantics for IN / NOT IN. Under WHERE, x IN (...) returns NULL (filtered out) when x is NULL or when x matches no non-null value and the list contains a NULL; x NOT IN (..., NULL, ...) is never TRUE for any row. Multi-valued operands on either side fold to NULL with the standard "single-value function encountered multi-value" warning, matching the In operator's MV behavior. The filter path inherits these semantics from In directly. The hash-join path needs extra work because its runtime BlockHash (inside RowInTableLookupOperator) treats null = null as a match, and a non-match always yields sentinel NULL (kept by ANTI). We close every gap as part of dedup canonicalization and join construction:

      • Right side: convert MV positions to NULL via convertMvPositionsToNull(org.elasticsearch.compute.data.Block, org.elasticsearch.compute.data.BlockFactory) (mirrors the left-side MvSingleValueOrNull guard), then hand the result to BlockHash. BlockHash collapses every NULL — original or MV-derived — into its reserved group 0, which surfaces as a single NULL position at index 0 of the dedup output. The post-dedup logic derives rightHadNulls from that position and drives the short-circuit decisions:
        • SEMI: x IN (a, b, NULL) ≡ x IN (a, b) once WHERE drops NULL results, so the filter path keeps the NULL literal (it's harmless) and the hash-join path strips it from the LocalRelation.
        • ANTI: any NULL on the right makes the predicate non-TRUE for every row, so we short-circuit to Filter(FALSE) immediately after the dedup.
        • MARK: keeps rightHadNulls only as a hint for the hash-join path's CASE expression — the filter path uses the NULL position in the dedup output directly to drive In's three-valued mark.
      • Left side (hash-join path only): insert Eval(svKey = MvSingleValueOrNull(leftField)) and (for SEMI/ANTI) Filter(IsNotNull(svKey)) above subqueryJoin.left() so NULL-keyed and MV-keyed left rows can never reach the lookup. MARK keeps the SV guard but skips the IsNotNull filter so MV/NULL left rows survive into the output with the CASE-driven mark.

      Page lifecycle: when a non-null pageHolder is supplied, the source page held there is released eagerly (the dedup step copied every distinct value into a fresh, CB-tracked block), and the holder is either cleared (filter / empty / short-circuit paths) or swapped to point at the new dedup page (hash-join path) so the caller's existing cleanup releases the right page at end of plan execution.

      Parameters:
      blockFactory - CB-tracked factory used to allocate the dedup key column and sentinel
      pageHolder - optional holder that initially references the source page from data.supplier().get(); updated to reference the new dedup page (hash-join path) or cleared (filter / empty path) so the caller's existing cleanup releases the correct page. May be null for callers that do not need this swap (e.g. unit tests).

    • firstSubPlan

      public static AbstractSubqueryJoin.LogicalPlanTuple firstSubPlan(LogicalPlan optimizedPlan, Set<LocalRelation> subPlansResults)
      Finds the first subquery join in the plan whose right side has not yet been replaced with results. Unlike InlineJoin, the right side is an independent subquery that doesn't use StubRelation, no replaceStub is needed, deep copy is not required.

    • newMainPlan

      public static LogicalPlan newMainPlan(LogicalPlan optimizedPlan, AbstractSubqueryJoin.LogicalPlanTuple subPlans, LocalRelation resultWrapper, int hashJoinThreshold, BlockFactory blockFactory, AtomicReference<Page> pageHolder)