Package org.elasticsearch.xpack.esql.expression.function.fulltext

Class FullTextFunction

java.lang.Object

org.elasticsearch.xpack.esql.core.tree.Node<Expression>

org.elasticsearch.xpack.esql.core.expression.Expression

org.elasticsearch.xpack.esql.core.expression.function.Function

org.elasticsearch.xpack.esql.expression.function.fulltext.FullTextFunction

All Implemented Interfaces:

NamedWriteable, Writeable, PostAnalysisPlanVerificationAware, PostOptimizationPlanVerificationAware, PostOptimizationVerificationAware, RewriteableAware, TranslationAware, Resolvable, EvaluatorMapper, ExpressionScoreMapper

Direct Known Subclasses:

Kql, QueryString, SingleFieldFullTextFunction

public abstract class FullTextFunction
extends Function
implements TranslationAware, PostAnalysisPlanVerificationAware, EvaluatorMapper, ExpressionScoreMapper, PostOptimizationVerificationAware, RewriteableAware, PostOptimizationPlanVerificationAware

Base class for full-text functions that use ES queries to match documents. These functions needs to be pushed down to Lucene queries to be executed - there’s no Evaluator for them, but depend on LocalPhysicalPlanOptimizer to rewrite them into Lucene queries.

Nested Class Summary

Nested classes/interfaces inherited from class org.elasticsearch.xpack.esql.core.expression.Expression

Expression.TypeResolution

Nested classes/interfaces inherited from class org.elasticsearch.xpack.esql.core.tree.Node

Node.NodeStringFormat

Nested classes/interfaces inherited from interface org.elasticsearch.xpack.esql.evaluator.mapper.EvaluatorMapper

EvaluatorMapper.ToEvaluator

Nested classes/interfaces inherited from interface org.elasticsearch.xpack.esql.score.ExpressionScoreMapper

ExpressionScoreMapper.ToScorer

Nested classes/interfaces inherited from interface org.elasticsearch.xpack.esql.capabilities.PostOptimizationVerificationAware

PostOptimizationVerificationAware.CoordinatorOnly

Nested classes/interfaces inherited from interface org.elasticsearch.xpack.esql.capabilities.TranslationAware

TranslationAware.FinishedTranslatable, TranslationAware.SingleValueTranslationAware, TranslationAware.Translatable

Nested classes/interfaces inherited from interface org.elasticsearch.common.io.stream.Writeable

Writeable.Reader<V>, Writeable.Writer<V>

Field Summary

Fields inherited from class org.elasticsearch.xpack.esql.core.tree.Node

TO_STRING_MAX_LINES, TO_STRING_MAX_WIDTH

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	FullTextFunction(Source source, Expression query, List<Expression> children, QueryBuilder queryBuilder)

Method Summary

Modifier and Type	Method	Description
Query	asQuery(LucenePushdownPredicates pushdownPredicates, TranslatorHandler handler)	Translates the implementing expression into a Query.
DataType	dataType()	The DataType returned by executing the tree rooted at this expression.
boolean	equals(Object obj)
protected QueryBuilder	evaluatorQueryBuilder()	Returns the query builder to be used when the function cannot be pushed down to Lucene, but uses a LuceneQueryEvaluator instead
protected FieldAttribute	fieldAsFieldAttribute(Expression field)
protected void	fieldVerifier(LogicalPlan plan, FullTextFunction function, Expression field, Failures failures)
String	functionType()	Used to differentiate error messages between functions and operators
protected String	getNameFromFieldAttribute(FieldAttribute fieldAttribute)
int	hashCode()
protected boolean	isRuntimeSearch()
Nullability	nullable()
BiConsumer<LogicalPlan,Failures>	postAnalysisPlanVerification()	Allows the implementer to return a consumer that will perform self-validation in the context of the tree structure the implementer is part of.
BiConsumer<LogicalPlan,Failures>	postOptimizationPlanVerification()	Validates the implementing expression - discovered failures are reported to the given Failures class.
void	postOptimizationVerification(Failures failures)	Validates the implementing expression - discovered failures are reported to the given Failures class.
Expression	query()
QueryBuilder	queryBuilder()
boolean	requiresQueryBuilderRewrite()
protected Expression.TypeResolution	resolveParams()	Resolves the type for the function parameters, as part of the type resolution for the function
protected Expression.TypeResolution	resolveQuery(TypeResolutions.ParamOrdinal queryOrdinal)	Resolves the type for the query parameter, as part of the type resolution for the function
protected final Expression.TypeResolution	resolveType()	The implementation of Expression.typeResolved(), which is just a caching wrapper around this method.
ExpressionEvaluator.Factory	toEvaluator(EvaluatorMapper.ToEvaluator toEvaluator)	Convert this into an ExpressionEvaluator.
ScoreOperator.ExpressionScorer.Factory	toScorer(ExpressionScoreMapper.ToScorer toScorer)
TranslationAware.Translatable	translatable(LucenePushdownPredicates pushdownPredicates)	Can this instance be translated or not? Usually checks whether the expression arguments are actual fields that exist in Lucene.
protected abstract Query	translate(LucenePushdownPredicates pushdownPredicates, TranslatorHandler handler)

Methods inherited from class org.elasticsearch.xpack.esql.core.expression.function.Function

arguments, functionName, nodeString

Methods inherited from class org.elasticsearch.xpack.esql.core.expression.Expression

canonical, canonicalize, childrenResolved, fold, foldable, propertiesToString, references, resolved, semanticEquals, semanticHash, toString, toString, typeResolved

Methods inherited from class org.elasticsearch.xpack.esql.core.tree.Node

allMatch, anyMatch, children, collect, collect, collect, collectFirstChildren, collectLeaves, doCollectFirst, forEachDown, forEachDown, forEachDownMayReturnEarly, forEachProperty, forEachPropertyDown, forEachPropertyOnly, forEachPropertyUp, forEachUp, forEachUp, info, nodeName, nodeProperties, nodeString, replaceChildren, replaceChildrenSameSize, source, sourceLocation, sourceText, toString, transformChildren, transformChildren, transformDown, transformDown, transformDown, transformDown, transformDownSkipBranch, transformNodeProps, transformPropertiesDown, transformPropertiesOnly, transformPropertiesUp, transformUp, transformUp, transformUp

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.elasticsearch.xpack.esql.evaluator.mapper.EvaluatorMapper

fold

Methods inherited from interface org.elasticsearch.xpack.esql.score.ExpressionScoreMapper

contributesToScore

Methods inherited from interface org.elasticsearch.common.io.stream.NamedWriteable

getWriteableName

Methods inherited from interface org.elasticsearch.xpack.esql.capabilities.RewriteableAware

replaceQueryBuilder

Methods inherited from interface org.elasticsearch.xpack.esql.capabilities.TranslationAware

asLuceneQuery

Methods inherited from interface org.elasticsearch.common.io.stream.Writeable

writeTo

Constructor Details

FullTextFunction

protected FullTextFunction(Source source,
 Expression query,
 List<Expression> children,
 QueryBuilder queryBuilder)

Method Details

dataType

public DataType dataType()

Description copied from class: Expression

The DataType returned by executing the tree rooted at this expression. If Expression.typeResolved() returns an error then the behavior of this method is undefined. It may return a valid type. Or it may throw an exception. Or it may return a totally nonsensical type.

Specified by:

dataType in class Expression

resolveType

protected final Expression.TypeResolution resolveType()

Description copied from class: Expression

The implementation of Expression.typeResolved(), which is just a caching wrapper around this method. See it's javadoc for what this method should return.

Implementations will rarely interact with the Expression.TypeResolution class directly, instead usually calling the utility methods on TypeResolutions.

Implementations should fail if Expression.childrenResolved() returns false.

Overrides:

resolveType in class Expression

resolveParams

protected Expression.TypeResolution resolveParams()

Resolves the type for the function parameters, as part of the type resolution for the function

Returns:

type resolution for the function parameters

resolveQuery

protected Expression.TypeResolution resolveQuery(TypeResolutions.ParamOrdinal queryOrdinal)

Resolves the type for the query parameter, as part of the type resolution for the function

Returns:

type resolution for the query parameter

query

public Expression query()

nullable

public Nullability nullable()

Overrides:

nullable in class Function

functionType

public String functionType()

Used to differentiate error messages between functions and operators

Returns:

function type for error messages

hashCode

public int hashCode()

Overrides:

hashCode in class Function

equals

public boolean equals(Object obj)

Overrides:

equals in class Function

translatable

public TranslationAware.Translatable translatable(LucenePushdownPredicates pushdownPredicates)

Description copied from interface: TranslationAware

Can this instance be translated or not? Usually checks whether the expression arguments are actual fields that exist in Lucene. See TranslationAware.Translatable for precisely what can be signaled from this method.

Specified by:

translatable in interface TranslationAware

asQuery

public Query asQuery(LucenePushdownPredicates pushdownPredicates,
 TranslatorHandler handler)

Description copied from interface: TranslationAware

Translates the implementing expression into a Query. If during translation a child needs to be translated first, the handler needs to be used even if the child implements this interface as well. This is to ensure that the child is wrapped in a SingleValueQuery if necessary.

So use this:

Query childQuery = handler.asQuery(child);

and not this:

Query childQuery = child.asQuery(handler);

Specified by:

asQuery in interface TranslationAware

queryBuilder

public QueryBuilder queryBuilder()

Specified by:

queryBuilder in interface RewriteableAware

Returns:

The current active query builder.

translate

protected abstract Query translate(LucenePushdownPredicates pushdownPredicates,
 TranslatorHandler handler)

postAnalysisPlanVerification

public BiConsumer<LogicalPlan,Failures> postAnalysisPlanVerification()

Description copied from interface: PostAnalysisPlanVerificationAware

Allows the implementer to return a consumer that will perform self-validation in the context of the tree structure the implementer is part of. This usually involves checking the type and configuration of the children or that of the parent.

It is often more useful to perform the checks as extended as it makes sense, over stopping at the first failure. This will allow the author to progress faster to a correct query.

Example: a GroupingFunction instance, which models a function to group documents to aggregate over, can only be used in the context of the STATS command, modeled by the Aggregate class. This is how this verification is performed:

     
      @Override
      public BiConsumer<LogicalPlan, Failures> postAnalysisPlanVerification() {
          return (p, failures) -> {
              if (p instanceof Aggregate == false) {
                  p.forEachExpression(
                      GroupingFunction.class,
                      gf -> failures.add(fail(gf, "cannot use grouping function [{}] outside of a STATS command", gf.sourceText()))
                  );
              }
          };
      }
     
     

Specified by:

postAnalysisPlanVerification in interface PostAnalysisPlanVerificationAware

Returns:

a consumer that will receive a tree to check and an accumulator of failures found during inspection.

isRuntimeSearch

protected boolean isRuntimeSearch()

fieldVerifier

protected void fieldVerifier(LogicalPlan plan,
 FullTextFunction function,
 Expression field,
 Failures failures)

toEvaluator

public ExpressionEvaluator.Factory toEvaluator(EvaluatorMapper.ToEvaluator toEvaluator)

Description copied from interface: EvaluatorMapper

Convert this into an ExpressionEvaluator.

Note for implementors: If you are implementing this function, you should call the passed-in lambda on your children, after doing any other manipulation (casting, etc.) necessary.

Note for Callers: If you are attempting to call this method, and you have an Expression and a Layout, you likely want to call EvalMapper.toEvaluator(org.elasticsearch.xpack.esql.core.expression.FoldContext, org.elasticsearch.xpack.esql.core.expression.Expression, org.elasticsearch.xpack.esql.planner.Layout) instead. On the other hand, if you already have something that looks like the parameter for this method, you should call this method with that function.

Build an ExpressionEvaluator.Factory for the tree of expressions rooted at this node. This is only guaranteed to return a sensible evaluator if this node has a valid type. If this node is a subclass of Expression then "valid type" means that Expression.typeResolved() returns a non-error resolution. If Expression.typeResolved() returns an error then this method may throw. Or return an evaluator that produces garbage. Or return an evaluator that throws when run.

Specified by:

toEvaluator in interface EvaluatorMapper

evaluatorQueryBuilder

protected QueryBuilder evaluatorQueryBuilder()

Returns the query builder to be used when the function cannot be pushed down to Lucene, but uses a LuceneQueryEvaluator instead

Returns:

the query builder to be used in the LuceneQueryEvaluator

toScorer

public ScoreOperator.ExpressionScorer.Factory toScorer(ExpressionScoreMapper.ToScorer toScorer)

Specified by:

toScorer in interface ExpressionScoreMapper

getNameFromFieldAttribute

protected String getNameFromFieldAttribute(FieldAttribute fieldAttribute)

fieldAsFieldAttribute

protected FieldAttribute fieldAsFieldAttribute(Expression field)

postOptimizationVerification

public void postOptimizationVerification(Failures failures)

Description copied from interface: PostOptimizationVerificationAware

Validates the implementing expression - discovered failures are reported to the given Failures class.

Example: the Bucket function, which produces buckets over a numerical or date field, based on a number of literal arguments needs to check if its arguments are all indeed literals. This is how this verification is performed:

     

      @Override
      public void postOptimizationVerification(Failures failures) {
          String operation = sourceText();

          failures.add(isFoldable(buckets, operation, SECOND))
              .add(from != null ? isFoldable(from, operation, THIRD) : null)
              .add(to != null ? isFoldable(to, operation, FOURTH) : null);
      }
     
     

Specified by:

postOptimizationVerification in interface PostOptimizationVerificationAware

postOptimizationPlanVerification

public BiConsumer<LogicalPlan,Failures> postOptimizationPlanVerification()

Description copied from interface: PostOptimizationPlanVerificationAware

Validates the implementing expression - discovered failures are reported to the given Failures class.

Example: the SORT command, OrderBy, can only be executed currently if it can be associated with a LIMIT Limit and together transformed into a TopN (which is executable). The replacement of the LIMIT+SORT into a TopN is done at the end of the optimization phase. This means that any SORT still existing in the plan post optimization is an error. However, there can be a LIMIT in the plan, but separated from the SORT by an INLINE STATS; in this case, the LIMIT cannot be pushed down near the SORT. To inform the user how they need to modify the query so it can be run, we implement this:

     

      @Override
      public BiConsumer<LogicalPlan, Failures> postOptimizationPlanVerification() {
          return (p, failures) -> {
              if (p instanceof InlineJoin inlineJoin) {
                  inlineJoin.forEachUp(OrderBy.class, orderBy -> {
                      failures.add(
                          fail(
                              inlineJoin,
                              "unbounded SORT [{}] not supported before INLINE STATS [{}], move the sort after the INLINE STATS",
                              orderBy.sourceText(),
                              inlineJoin.sourceText()
                          )
                      );
                  });
              } else if (p instanceof OrderBy) {
                  failures.add(fail(p, "Unbounded SORT not supported yet [{}] please add a LIMIT", p.sourceText()));
              }
          };
      }
     
     

If we didn't need to check the structure of the plan, it would have sufficed to implement the PostOptimizationVerificationAware interface, which would simply check if there is an instance of OrderBy in the plan.

Specified by:

postOptimizationPlanVerification in interface PostOptimizationPlanVerificationAware

Returns:

a consumer that will receive a tree to check and an accumulator of failures found during inspection.

requiresQueryBuilderRewrite

public boolean requiresQueryBuilderRewrite()

Specified by:

requiresQueryBuilderRewrite in interface RewriteableAware

Returns:

true if the `QueryBuilder` needs to be rewritten, false otherwise. If no rewrite is required, this instance will be skipped during the rewrite phase in QueryBuilderResolver.resolveQueryBuilders(org.elasticsearch.xpack.esql.plan.logical.LogicalPlan, org.elasticsearch.xpack.esql.plugin.TransportActionServices, org.elasticsearch.action.ActionListener<org.elasticsearch.xpack.esql.plan.logical.LogicalPlan>).