Package org.elasticsearch.xpack.esql.expression.function.scalar
ScalarFunction
subclass to link into the ESQL core infrastructure and the EvalOperator.ExpressionEvaluator
implementation to run the actual function.
Guide to adding new function
Adding functions is fairly easy and should be fun! This is a step by step list of how to do it.
- Fork the Elasticsearch repo.
- Clone your fork locally.
- Add Elastic’s remote, it should look a little like:
[remote "elastic"] url = git@github.com:elastic/elasticsearch.git fetch = +refs/heads/*:refs/remotes/elastic/* [remote "nik9000"] url = git@github.com:nik9000/elasticsearch.git fetch = +refs/heads/*:refs/remotes/nik9000/* -
Feel free to use
gitas a scratch pad. We're going to squash all commits before merging and will only keep the PR subject line and description in the commit message. - Open Elasticsearch in IntelliJ.
-
Run the csv tests (see
x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/CsvTests.java) from within Intellij or, alternatively, via Gradle:./gradlew :x-pack:plugin:esql:test --tests "org.elasticsearch.xpack.esql.CsvTests"IntelliJ will take a few minutes to compile everything but the test itself should take only a few seconds. This is a fast path to running ESQL’s integration tests. -
Pick one of the csv-spec files in
x-pack/plugin/esql/qa/testFixtures/src/main/resources/and add a test for the function you want to write. These files are roughly themed but there isn’t a strong guiding principle in the organization. -
Rerun the
CsvTestsand watch your new test fail. Yay, TDD doing it’s job. -
Find a function in this package similar to the one you are working on and copy it to build
yours. There’s some ceremony required in each function class to make it constant foldable,
and return the right types. Take a stab at these, but don’t worry too much about getting
it right. Your function might extend from one of several abstract base classes, all of
those are fine for this guide, but might have special instructions called out later.
Known good base classes:
AbstractConvertFunctionAbstractMultivalueFunctionUnaryScalarFunctionor any subclass likeAbstractTrigonometricFunctionEsqlScalarFunctionDoubleConstantFunction
-
There are also methods annotated with
Evaluatorthat contain the actual inner implementation of the function. They are usually named "process" or "processInts" or "processBar". Modify those to look right and run theCsvTestsagain. This should generate anEvalOperator.ExpressionEvaluatorimplementation calling the method annotated withEvaluator. . To make it work with IntelliJ, also clickBuild->Recompile 'FunctionName.java'. Please commit the generated evaluator before submitting your PR.NOTE 1: The function you copied may have a method annotated with
ConvertEvaluatororMvEvaluatorinstead ofEvaluator. Those do similar things and the instructions should still work for you regardless. If your function contains an implementation ofEvalOperator.ExpressionEvaluatorwritten by hand then please stop and ask for help. This is not a good first function.NOTE 2: Regardless of which annotation is on your "process" method you can learn more about the options for generating code from the javadocs on those annotations.
-
Once your evaluator is generated you can have your function return it,
generally by implementing
EvaluatorMapper.toEvaluator(org.elasticsearch.xpack.esql.evaluator.mapper.EvaluatorMapper.ToEvaluator). It’s possible that your abstract base class implements that function and will need you to implement something else:AbstractConvertFunction:factoriesAbstractMultivalueFunction:evaluatorAbstractTrigonometricFunction:doubleEvaluatorDoubleConstantFunction: nothing!
-
Add your function to
EsqlFunctionRegistry. This links it into the language andMETA FUNCTIONS. -
Implement serialization for your function by implementing
NamedWriteable.getWriteableName(),Writeable.writeTo(org.elasticsearch.common.io.stream.StreamOutput), and a deserializing constructor. Then add anNamedWriteableRegistry.Entryconstant and register it. To register it, look for a method likeScalarFunctionWritables.getNamedWriteables()in your function’s class hierarchy. Keep going up until you hit a function with that name. Then add your new "ENTRY" constant to the list it returns. -
Rerun the
CsvTests. They should find your function and maybe even pass. Add a few more tests in the csv-spec tests. They run quickly so it isn’t a big deal having half a dozen of them per function. In fact, it’s useful to add more complex combinations of things here, just to catch any accidental strange interactions. For example, have your function take its input from an index likeFROM employees | EVAL foo=MY_FUNCTION(emp_no). It’s probably a good idea to have your function passed as a parameter to another function likeEVAL foo=MOST(0, MY_FUNCTION(emp_no)). And likely useful to try the reverse likeEVAL foo=MY_FUNCTION(MOST(languages + 10000, emp_no). -
Now it’s time to make a unit test! The infrastructure for these is under some flux at
the moment, but it’s good to extend
AbstractScalarFunctionTestCase. All of these tests are parameterized and expect to spend some time finding good parameters. Also add serialization tests that extendAbstractExpressionSerializationTests<>. And also add type error tests that extendsErrorsForCasesWithoutExamplesTestCase. -
Once you are happy with the tests run the auto formatter:
./gradlew -p x-pack/plugin/esql/ spotlessApply -
Now you can run all of the ESQL tests like CI:
./gradlew -p x-pack/plugin/esql/ test -
We need to tag to what release the function applies to so we can generate docs in the next step!
On the constructor of your function class you very likely have an annotation
@FunctionInfo. Add the attributeappliesTowith availability information. For example a GA function available in 9.2.0 would be tagged as{ @FunctionAppliesTo(lifeCycle = FunctionAppliesToLifecycle.GA, version = "9.2.0") } -
Now it’s time to generate some docs!
Actually, running the tests in the example above should have done it for you.
Make sure to commit the following generated files
docs/reference/query-languages/esql/_snippets/functions/description/myfunction.mddocs/reference/query-languages/esql/_snippets/functions/examples/myfunction.mddocs/reference/query-languages/esql/_snippets/functions/layout/myfunction.mddocs/reference/query-languages/esql/_snippets/functions/parameters/myfunction.mddocs/reference/query-languages/esql/_snippets/functions/types/myfunction.mddocs/reference/query-languages/esql/kibana/definition/functions/myfunction.jsondocs/reference/query-languages/esql/kibana/docs/functions/myfunction.mddocs/reference/query-languages/esql/images/functions/myfunction.svg
docs/reference/query-languages/esql/functions-operatorsanddocs/reference/query-languages/esql/_snippets/listsdirectories.
For example, if you are writing a Math function, you will want to add it indocs/reference/query-languages/esql/functions-operators/math-functions.mdand indocs/reference/query-languages/esql/_snippets/lists/math-functions.md.
You can generate the docs for just your function by running
./gradlew :x-pack:plugin:esql:test -Dtests.class='*SinTests'. It’s just running your new unit test. You should see something like:> Task :x-pack:plugin:esql:test ESQL Docs: Only files related to [sin.md], patching them into place -
Install the docs-builder binary from https://github.com/elastic/docs-builder,
then build the docs locally by running the command below from the elasticsearch directory:
docs-builder serve - You can now browse the docs at http://localhost:3000. Or you can go directly to ES|QL functions and operators to see your function in the list and follow its link to get to the page you built. Make sure it looks ok.
-
Let’s finish up the code by making the tests backwards compatible. Since this is a new
feature we just have to convince the tests not to run in a cluster that includes older
versions of Elasticsearch. We do that with a
capabilityon the REST handler. ESQL has a ton of capabilities so we list them all inEsqlCapabilities. Add a new one for your function. Now add something likerequired_capability: my_functionto all of your csv-spec tests. Run those csv-spec tests as integration tests to double check that they run on the main branch.
NOTE: You may notice tests gated based on Elasticsearch version. This was the old way of doing things. Now, we use specific capabilities for each function. -
Sometimes we want to implement a function without releasing it. For example, there's a probability its implementation might
need to change because we're not sure how to handle some edge cases, or we just require further feedback on it. In such case,
we can still ship it as a snapshot function, and ensure the following:
- The new function doesn't show up in the docs. Committing the 8 generated docs files is ok.
- The new function is marked as a snapshot function in
EsqlCapabilities. - The new function is grouped with other snapshot functions in
EsqlFunctionRegistry. - The class that implements your new functions has the right annotations.
x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/esql/60_usage.ymlaroundnot_exists: esql.functions.delay. The release build will fail if this function is available. Check out the instructions for running a release build intesting.asciidocif you want to test this, but it's generally enough to let CI do it. -
Open the PR. The subject and description of the PR are important because those'll turn
into the commit message we see in the commit history. Good PR descriptions make me very
happy. But functions don’t need an essay.
Add the
>enhancementand:Analytics/ES|QLtags if you are able. Request a review if you can, probably from one of the folks that github proposes to you. -
CI might fail for random looking reasons. The first thing you should do is merge
maininto your PR branch. That’s usually just:
Don’t worry about the commit message. It'll get squashed away in the merge.git checkout main && git pull elastic main && git checkout mybranch && git merge main
-
ClassesClassDescriptionA
ScalarFunctionis aFunctionthat takes values from some operation and converts each to another value.