Module org.elasticsearch.server

Package org.elasticsearch.index.codec.vectors.diskbbq.next

Class ESNextDiskBBQVectorsWriter

java.lang.Object

org.apache.lucene.codecs.KnnVectorsWriter

org.elasticsearch.index.codec.vectors.diskbbq.IVFVectorsWriter

org.elasticsearch.index.codec.vectors.diskbbq.next.ESNextDiskBBQVectorsWriter

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.lucene.util.Accountable

public class ESNextDiskBBQVectorsWriter
extends IVFVectorsWriter

Default implementation of IVFVectorsWriter. It uses HierarchicalKMeans algorithm to partition the vector space, and then stores the centroids and posting list in a sequential fashion.

Nested Class Summary

Nested classes/interfaces inherited from class org.elasticsearch.index.codec.vectors.diskbbq.IVFVectorsWriter

IVFVectorsWriter.CentroidOffsetAndLength

Nested classes/interfaces inherited from class org.apache.lucene.codecs.KnnVectorsWriter

org.apache.lucene.codecs.KnnVectorsWriter.MergedVectorValues

Field Summary

Fields inherited from interface org.apache.lucene.util.Accountable

NULL_ACCOUNTABLE

Constructor Summary

Constructors

Constructor	Description
ESNextDiskBBQVectorsWriter(org.apache.lucene.index.SegmentWriteState state, String rawVectorFormatName, boolean useDirectIOReads, org.apache.lucene.codecs.hnsw.FlatVectorsWriter rawVectorDelegate, ESNextDiskBBQVectorsFormat.QuantEncoding encoding, int vectorPerCluster, int centroidsPerParentCluster)

Method Summary

Modifier and Type	Method	Description
IVFVectorsWriter.CentroidOffsetAndLength	buildAndWritePostingsLists(org.apache.lucene.index.FieldInfo fieldInfo, CentroidSupplier centroidSupplier, org.apache.lucene.index.FloatVectorValues floatVectorValues, org.apache.lucene.store.IndexOutput postingsOutput, long fileOffset, int[] assignments, int[] overspillAssignments)
IVFVectorsWriter.CentroidOffsetAndLength	buildAndWritePostingsLists(org.apache.lucene.index.FieldInfo fieldInfo, CentroidSupplier centroidSupplier, org.apache.lucene.index.FloatVectorValues floatVectorValues, org.apache.lucene.store.IndexOutput postingsOutput, long fileOffset, org.apache.lucene.index.MergeState mergeState, int[] assignments, int[] overspillAssignments)
CentroidAssignments	calculateCentroids(org.apache.lucene.index.FieldInfo fieldInfo, org.apache.lucene.index.FloatVectorValues floatVectorValues)	Calculate the centroids for the given field.
CentroidAssignments	calculateCentroids(org.apache.lucene.index.FieldInfo fieldInfo, org.apache.lucene.index.FloatVectorValues floatVectorValues, org.apache.lucene.index.MergeState mergeState)
CentroidSupplier	createCentroidSupplier(org.apache.lucene.store.IndexInput centroidsInput, int numCentroids, org.apache.lucene.index.FieldInfo fieldInfo, float[] globalCentroid)
protected void	doWriteMeta(org.apache.lucene.store.IndexOutput metaOutput, org.apache.lucene.index.FieldInfo field, int numCentroids)
void	writeCentroids(org.apache.lucene.index.FieldInfo fieldInfo, CentroidSupplier centroidSupplier, int[] centroidAssignments, float[] globalCentroid, IVFVectorsWriter.CentroidOffsetAndLength centroidOffsetAndLength, org.apache.lucene.store.IndexOutput centroidOutput)

Methods inherited from class org.elasticsearch.index.codec.vectors.diskbbq.IVFVectorsWriter

addField, close, finish, flush, mergeOneField, ramBytesUsed

Methods inherited from class org.apache.lucene.codecs.KnnVectorsWriter

mapOldOrdToNewOrd, merge

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.lucene.util.Accountable

getChildResources

Constructor Details

ESNextDiskBBQVectorsWriter

public ESNextDiskBBQVectorsWriter(org.apache.lucene.index.SegmentWriteState state,
 String rawVectorFormatName,
 boolean useDirectIOReads,
 org.apache.lucene.codecs.hnsw.FlatVectorsWriter rawVectorDelegate,
 ESNextDiskBBQVectorsFormat.QuantEncoding encoding,
 int vectorPerCluster,
 int centroidsPerParentCluster)
                           throws IOException

Throws:

IOException

Method Details

buildAndWritePostingsLists

public IVFVectorsWriter.CentroidOffsetAndLength buildAndWritePostingsLists(org.apache.lucene.index.FieldInfo fieldInfo,
 CentroidSupplier centroidSupplier,
 org.apache.lucene.index.FloatVectorValues floatVectorValues,
 org.apache.lucene.store.IndexOutput postingsOutput,
 long fileOffset,
 int[] assignments,
 int[] overspillAssignments)
                                                                    throws IOException

Specified by:

buildAndWritePostingsLists in class IVFVectorsWriter

Throws:

IOException

buildAndWritePostingsLists

public IVFVectorsWriter.CentroidOffsetAndLength buildAndWritePostingsLists(org.apache.lucene.index.FieldInfo fieldInfo,
 CentroidSupplier centroidSupplier,
 org.apache.lucene.index.FloatVectorValues floatVectorValues,
 org.apache.lucene.store.IndexOutput postingsOutput,
 long fileOffset,
 org.apache.lucene.index.MergeState mergeState,
 int[] assignments,
 int[] overspillAssignments)
                                                                    throws IOException

Specified by:

buildAndWritePostingsLists in class IVFVectorsWriter

Throws:

IOException

createCentroidSupplier

public CentroidSupplier createCentroidSupplier(org.apache.lucene.store.IndexInput centroidsInput,
 int numCentroids,
 org.apache.lucene.index.FieldInfo fieldInfo,
 float[] globalCentroid)

Specified by:

createCentroidSupplier in class IVFVectorsWriter

doWriteMeta

protected void doWriteMeta(org.apache.lucene.store.IndexOutput metaOutput,
 org.apache.lucene.index.FieldInfo field,
 int numCentroids)
                    throws IOException

Specified by:

doWriteMeta in class IVFVectorsWriter

Throws:

IOException

writeCentroids

public void writeCentroids(org.apache.lucene.index.FieldInfo fieldInfo,
 CentroidSupplier centroidSupplier,
 int[] centroidAssignments,
 float[] globalCentroid,
 IVFVectorsWriter.CentroidOffsetAndLength centroidOffsetAndLength,
 org.apache.lucene.store.IndexOutput centroidOutput)
                    throws IOException

Specified by:

writeCentroids in class IVFVectorsWriter

Throws:

IOException

calculateCentroids

public CentroidAssignments calculateCentroids(org.apache.lucene.index.FieldInfo fieldInfo,
 org.apache.lucene.index.FloatVectorValues floatVectorValues,
 org.apache.lucene.index.MergeState mergeState)
                                       throws IOException

Specified by:

calculateCentroids in class IVFVectorsWriter

Throws:

IOException

calculateCentroids

public CentroidAssignments calculateCentroids(org.apache.lucene.index.FieldInfo fieldInfo,
 org.apache.lucene.index.FloatVectorValues floatVectorValues)
                                       throws IOException

Calculate the centroids for the given field. We use the HierarchicalKMeans algorithm to partition the space of all vectors across merging segments

Specified by:

calculateCentroids in class IVFVectorsWriter

Parameters:

fieldInfo - merging field info

floatVectorValues - the float vector values to merge

Returns:

the vector assignments, soar assignments, and if asked the centroids themselves that were computed

Throws:

IOException - if an I/O error occurs