Package org.elasticsearch.xpack.esql.expression.function.fulltext

Class FullTextFunction

java.lang.Object
org.elasticsearch.xpack.esql.core.tree.Node<Expression>
org.elasticsearch.xpack.esql.core.expression.Expression
org.elasticsearch.xpack.esql.core.expression.function.Function
org.elasticsearch.xpack.esql.expression.function.fulltext.FullTextFunction
All Implemented Interfaces:
NamedWriteable, Writeable, PostAnalysisPlanVerificationAware, PostOptimizationPlanVerificationAware, PostOptimizationVerificationAware, RewriteableAware, TranslationAware, Resolvable, EvaluatorMapper, ExpressionScoreMapper
Direct Known Subclasses:
Kql, MultiMatch, QueryString, SingleFieldFullTextFunction, Term
public abstract class FullTextFunction extends Function implements TranslationAware, PostAnalysisPlanVerificationAware, EvaluatorMapper, ExpressionScoreMapper, PostOptimizationVerificationAware, RewriteableAware, PostOptimizationPlanVerificationAware
Base class for full-text functions that use ES queries to match documents. These functions needs to be pushed down to Lucene queries to be executed - there’s no Evaluator for them, but depend on LocalPhysicalPlanOptimizer to rewrite them into Lucene queries.

  • Constructor Details

  • Method Details

    • dataType

      public DataType dataType()
      Specified by:
      dataType in class Expression

    • resolveType

      protected final Expression.TypeResolution resolveType()
      Overrides:
      resolveType in class Expression

    • resolveParams

      protected Expression.TypeResolution resolveParams()
      Resolves the type for the function parameters, as part of the type resolution for the function
      Returns:
      type resolution for the function parameters

    • resolveQuery

      protected Expression.TypeResolution resolveQuery(TypeResolutions.ParamOrdinal queryOrdinal)
      Resolves the type for the query parameter, as part of the type resolution for the function
      Returns:
      type resolution for the query parameter

    • query

      public Expression query()

    • nullable

      public Nullability nullable()
      Overrides:
      nullable in class Function

    • functionType

      public String functionType()
      Used to differentiate error messages between functions and operators
      Returns:
      function type for error messages

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class Function

    • equals

      public boolean equals(Object obj)
      Overrides:
      equals in class Function

    • translatable

      public TranslationAware.Translatable translatable(LucenePushdownPredicates pushdownPredicates)
      Description copied from interface: TranslationAware
      Can this instance be translated or not? Usually checks whether the expression arguments are actual fields that exist in Lucene. See TranslationAware.Translatable for precisely what can be signaled from this method.
      Specified by:
      translatable in interface TranslationAware

    • asQuery

      public Query asQuery(LucenePushdownPredicates pushdownPredicates, TranslatorHandler handler)
      Description copied from interface: TranslationAware
      Translates the implementing expression into a Query. If during translation a child needs to be translated first, the handler needs to be used even if the child implements this interface as well. This is to ensure that the child is wrapped in a SingleValueQuery if necessary.

      So use this:

      Query childQuery = handler.asQuery(child);

      and not this:

      Query childQuery = child.asQuery(handler);

      Specified by:
      asQuery in interface TranslationAware

    • queryBuilder

      public QueryBuilder queryBuilder()
      Specified by:
      queryBuilder in interface RewriteableAware
      Returns:
      The current active query builder.

    • translate

      protected abstract Query translate(LucenePushdownPredicates pushdownPredicates, TranslatorHandler handler)

    • postAnalysisPlanVerification

      public BiConsumer<LogicalPlan,Failures> postAnalysisPlanVerification()
      Description copied from interface: PostAnalysisPlanVerificationAware
      Allows the implementer to return a consumer that will perform self-validation in the context of the tree structure the implementer is part of. This usually involves checking the type and configuration of the children or that of the parent.

      It is often more useful to perform the checks as extended as it makes sense, over stopping at the first failure. This will allow the author to progress faster to a correct query.

      Example: a GroupingFunction instance, which models a function to group documents to aggregate over, can only be used in the context of the STATS command, modeled by the Aggregate class. This is how this verification is performed: 

           
      @Override
      public BiConsumer<LogicalPlan, Failures> postAnalysisPlanVerification() {
          return (p, failures) -> {
              if (p instanceof Aggregate == false) {
                  p.forEachExpression(
                      GroupingFunction.class,
                      gf -> failures.add(fail(gf, "cannot use grouping function [{}] outside of a STATS command", gf.sourceText()))
                  );
              }
          };
      }
      Specified by:
      postAnalysisPlanVerification in interface PostAnalysisPlanVerificationAware
      Returns:
      a consumer that will receive a tree to check and an accumulator of failures found during inspection.

    • fieldVerifier

      protected void fieldVerifier(LogicalPlan plan, FullTextFunction function, Expression field, Failures failures)

    • toEvaluator

      public EvalOperator.ExpressionEvaluator.Factory toEvaluator(EvaluatorMapper.ToEvaluator toEvaluator)
      Description copied from interface: EvaluatorMapper
      Convert this into an EvalOperator.ExpressionEvaluator.

      Note for implementors: If you are implementing this function, you should call the passed-in lambda on your children, after doing any other manipulation (casting, etc.) necessary.

      Note for Callers: If you are attempting to call this method, and you have an Expression and a Layout, you likely want to call EvalMapper.toEvaluator(org.elasticsearch.xpack.esql.core.expression.FoldContext, org.elasticsearch.xpack.esql.core.expression.Expression, org.elasticsearch.xpack.esql.planner.Layout) instead. On the other hand, if you already have something that looks like the parameter for this method, you should call this method with that function.

      Build an EvalOperator.ExpressionEvaluator.Factory for the tree of expressions rooted at this node. This is only guaranteed to return a sensible evaluator if this node has a valid type. If this node is a subclass of Expression then "valid type" means that Expression.typeResolved() returns a non-error resolution. If Expression.typeResolved() returns an error then this method may throw. Or return an evaluator that produces garbage. Or return an evaluator that throws when run.

      Specified by:
      toEvaluator in interface EvaluatorMapper

    • evaluatorQueryBuilder

      protected QueryBuilder evaluatorQueryBuilder()
      Returns the query builder to be used when the function cannot be pushed down to Lucene, but uses a LuceneQueryEvaluator instead
      Returns:
      the query builder to be used in the LuceneQueryEvaluator

    • toScorer

      public ScoreOperator.ExpressionScorer.Factory toScorer(ExpressionScoreMapper.ToScorer toScorer)
      Specified by:
      toScorer in interface ExpressionScoreMapper

    • getNameFromFieldAttribute

      protected String getNameFromFieldAttribute(FieldAttribute fieldAttribute)

    • fieldAsFieldAttribute

      protected FieldAttribute fieldAsFieldAttribute(Expression field)

    • postOptimizationVerification

      public void postOptimizationVerification(Failures failures)
      Description copied from interface: PostOptimizationVerificationAware
      Validates the implementing expression - discovered failures are reported to the given Failures class.

      Example: the Bucket function, which produces buckets over a numerical or date field, based on a number of literal arguments needs to check if its arguments are all indeed literals. This is how this verification is performed: 

           

      @Override
      public void postOptimizationVerification(Failures failures) {
          String operation = sourceText();

          failures.add(isFoldable(buckets, operation, SECOND))
              .add(from != null ? isFoldable(from, operation, THIRD) : null)
              .add(to != null ? isFoldable(to, operation, FOURTH) : null);
      }
      Specified by:
      postOptimizationVerification in interface PostOptimizationVerificationAware

    • postOptimizationPlanVerification

      public BiConsumer<LogicalPlan,Failures> postOptimizationPlanVerification()
      Description copied from interface: PostOptimizationPlanVerificationAware
      Validates the implementing expression - discovered failures are reported to the given Failures class.

      Example: the SORT command, OrderBy, can only be executed currently if it can be associated with a LIMIT Limit and together transformed into a TopN (which is executable). The replacement of the LIMIT+SORT into a TopN is done at the end of the optimization phase. This means that any SORT still existing in the plan post optimization is an error. However, there can be a LIMIT in the plan, but separated from the SORT by an INLINE STATS; in this case, the LIMIT cannot be pushed down near the SORT. To inform the user how they need to modify the query so it can be run, we implement this: 

           

      @Override
      public BiConsumer<LogicalPlan, Failures> postOptimizationPlanVerification() {
          return (p, failures) -> {
              if (p instanceof InlineJoin inlineJoin) {
                  inlineJoin.forEachUp(OrderBy.class, orderBy -> {
                      failures.add(
                          fail(
                              inlineJoin,
                              "unbounded SORT [{}] not supported before INLINE STATS [{}], move the sort after the INLINE STATS",
                              orderBy.sourceText(),
                              inlineJoin.sourceText()
                          )
                      );
                  });
              } else if (p instanceof OrderBy) {
                  failures.add(fail(p, "Unbounded SORT not supported yet [{}] please add a LIMIT", p.sourceText()));
              }
          };
      }

      If we didn't need to check the structure of the plan, it would have sufficed to implement the PostOptimizationVerificationAware interface, which would simply check if there is an instance of OrderBy in the plan.

      Specified by:
      postOptimizationPlanVerification in interface PostOptimizationPlanVerificationAware
      Returns:
      a consumer that will receive a tree to check and an accumulator of failures found during inspection.