Module org.elasticsearch.compute
Package org.elasticsearch.compute.lucene.read

Class ValuesSourceReaderOperator

java.lang.Object
org.elasticsearch.compute.operator.AbstractPageMappingToIteratorOperator
org.elasticsearch.compute.lucene.read.ValuesSourceReaderOperator
All Implemented Interfaces:
Closeable, AutoCloseable, Operator, Releasable
public class ValuesSourceReaderOperator extends AbstractPageMappingToIteratorOperator
Loads values from Lucene.

Input pages must contain a DocVector which describes the location of lucene document. We represent documents as a triple of integers that we all "shard", "segment", and "doc". "Shard" point to the lucene index. "Segment" points to the segment inside the index. And "doc" is the offset within that segment. Input look like 

┌───────────────────────┐
│          doc          │
├───────┬─────────┬─────┤
│ shard │ segment │ doc │
├───────┼─────────┼─────┤
│     0 │       0 │   0 │
│     0 │       0 │   1 │
│     0 │       0 │   2 │
│     0 │       0 │   3 │
│     0 │       0 │  12 │
└───────┴─────────┴─────┘

And output pages have the loaded fields appended: 

┌───────────────────────┬──────┐
│          doc          │      │
├───────┬─────────┬─────┤ name │
│ shard │ segment │ doc │      │
├───────┼─────────┼─────┼──────┤
│     0 │       0 │   0 │ foo  │
│     0 │       0 │   1 │ bar  │
│     0 │       0 │   2 │ baz  │
│     0 │       0 │   3 │ foo  │
│     0 │       0 │  12 │ bar  │
└───────┴─────────┴─────┴──────┘

Are we loading from one segment?

If we load from one segment then we can load more efficiently using ValuesFromSingleReader. Otherwise, we use ValuesFromManyReader. Loading from one shard looks like the example above. And it's super common. LuceneSourceOperator always loads fields this way and that's "high performance" way of reading documents. But after a sort then you are likely to be loading from many segments. Which looks like: 

┌───────────────────────┐
│          doc          │
├───────┬─────────┬─────┤
│ shard │ segment │ doc │
├───────┼─────────┼─────┤
│     0 │       0 │   0 │
│     0 │       1 │   0 │
│     0 │       1 │   1 │
│     1 │       0 │   1 │
│     1 │       1 │  12 │
└───────┴─────────┴─────┘

row-by-row vs column-at-a-time

Lucene can be configured to score data in two kinds of ways: rows and columns. All fields configured for "row" style storage and concatenated together and compressed with something like zstd. "Column" fields are stored as a dense array of values on disk and compressed using math tricks.

The "row" style fields need to be loaded together, one row at a time.

┌─────┬────────┐   ┌─────┬────────┐   ┌─────┬────────┐   ┌─────┬────────┐   ┌─────┬────────┐   ┌─────┬────────┐
│ ref │ class  │   │ ref │ class  │   │ ref │ class  │   │ ref │ class  │   │ ref │ class  │   │ ref │ class  │
├─────┼────────┤   ├─────┼────────┤   ├─────┼────────┤   ├─────┼────────┤   ├─────┼────────┤   ├─────┼────────┤
│     │        │   │ 173 │ Euclid │   │ 173 │ Euclid │   │ 173 │ Euclid │   │ 173 │ Euclid │   │ 173 │ Euclid │
│     │        │ ⟶ │     │        │ ⟶ │ 049 │ Euclid │ ⟶ │ 049 │ Euclid │ ⟶ │ 049 │ Euclid │ ⟶ │ 049 │ Euclid │
│     │        │   │     │        │   │     │        │   │ 096 │ Euclid │   │ 096 │ Euclid │   │ 096 │ Euclid │
│     │        │   │     │        │   │     │        │   │     │        │   │ 682 │ Keter  │   │ 682 │ Keter  │
│     │        │   │     │        │   │     │        │   │     │        │   │     │        │   │ 055 │ Keter  │
└─────┴────────┘   └─────┴────────┘   └─────┴────────┘   └─────┴────────┘   └─────┴────────┘   └─────┴────────┘

The "column" style fields need to be loaded one at a time:

┌─────┐   ┌─────┬────────┐   ┌─────┬────────┬──────┐   ┌─────┬────────┬──────┬────────────┐
│ ref │   │ ref │ class  │   │ ref │ class  │ site │   │ ref │ class  │ site │ casualties │
├─────┤   ├─────┼────────┤   ├─────┼────────┼──────┤   ├─────┼────────┼──────┼────────────┤
│ 173 │   │ 173 │ Euclid │   │ 173 │ Euclid │ 19   │   │ 173 │ Euclid │ 19   │ 0          │
│ 049 │   │ 049 │ Euclid │   │ 049 │ Euclid │ 19   │   │ 049 │ Euclid │ 19   │ 1          │
│ 096 │ ⟶ │ 096 │ Euclid │ ⟶ │ 096 │ Euclid │ ██   │ ⟶ │ 096 │ Euclid │ ██   │ ██         │
│ 682 │   │ 682 │ Keter  │   │ 682 │ Keter  │ ██   │   │ 682 │ Keter  │ ██   │ 34         │
│ 055 │   │ 055 │ Keter  │   │ 055 │ Keter  │ 19   │   │ 055 │ Keter  │ 19   │ null       │
└─────┘   └─────┴────────┘   └─────┴────────┴──────┘   └─────┴────────┴──────┴────────────┘

Oh, no! Giant strings

It's important to keep the Blocks we build from being giant. A couple of mb is ok, but 100mb is not usually great for the query. The most surefire way to do this is to load fields in the order they appear in the input page and then stop if you load too much. ValuesFromSingleReader and ValuesFromManyReader don't do that because they are trying to be more efficient when loading small things. But when we load big things we use ValuesFromDocSequence to load them in the order they appear so we can always bail early.