Class LuceneSliceQueue

java.lang.Object
org.elasticsearch.compute.lucene.LuceneSliceQueue

public final class LuceneSliceQueue extends Object
Shared Lucene slices between Lucene operators.

Each shard is built with a list of queries to run and tags to add to the queries (List<QueryAndTags>). Some examples:

  • For queries like FROM foo we'll use a one element list containing match_all, []. It loads all documents in the index and append no extra fields to the loaded documents.
  • For queries like FROM foo | WHERE a > 10 we'll use a one element list containing +single_value(a) +(a > 10), []. It loads all documents where a is single valued and greater than 10.
  • For queries like FROM foo | STATS MAX(b) BY ROUND_TO(a, 0, 100) we'll use a two element list containing
    • +single_value(a) +(a < 100), [0]
    • +single_value(a) +(a >= 100), [100]
    It loads all documents in the index where a is single valued and adds a constant 0 to the documents where a < 100 and the constant 100 to the documents where a >= 100.

IMPORTANT: Runners make no effort to deduplicate the results from multiple queries. If you need to only see each document one time then make sure the queries are mutually exclusive.

  • Field Details

  • Method Details

    • nextSlice

      @Nullable public LuceneSlice nextSlice(LuceneSlice prev)
      Retrieves the next available LuceneSlice for processing.

      This method implements a three-tiered strategy to minimize the overhead of switching between segments: 1. If a previous slice is provided, it first attempts to return the next sequential slice. This keeps a thread working on the same segments, minimizing the overhead of segment switching. 2. If affinity fails, it returns a slice from the sliceHeads queue, which is an entry point for a new, independent group of segments, allowing the calling Driver to work on a fresh set of segments. 3. If the sliceHeads queue is exhausted, it "steals" a slice from the stealableSlices queue. This fallback ensures all threads remain utilized.

      Parameters:
      prev - the previously returned LuceneSlice, or null if starting
      Returns:
      the next available LuceneSlice, or null if exhausted
    • totalSlices

      public int totalSlices()
    • partitioningStrategies

      public Map<String,LuceneSliceQueue.PartitioningStrategy> partitioningStrategies()
      Strategy used to partition each shard in this queue.
    • remainingShardsIdentifiers

      public Collection<String> remainingShardsIdentifiers()
    • create

      public static LuceneSliceQueue create(List<? extends ShardContext> contexts, Function<ShardContext,List<LuceneSliceQueue.QueryAndTags>> queryFunction, DataPartitioning dataPartitioning, Function<org.apache.lucene.search.Query,LuceneSliceQueue.PartitioningStrategy> autoStrategy, int taskConcurrency, Function<ShardContext,org.apache.lucene.search.ScoreMode> scoreModeFunction)