Module org.elasticsearch.compute

Package org.elasticsearch.compute.data.arrow

Class BytesRefArrowBufVector

java.lang.Object

org.elasticsearch.compute.data.AbstractNonThreadSafeRefCounted

org.elasticsearch.compute.data.arrow.AbstractArrowBufVector<BytesRefVector,BytesRefBlock>

org.elasticsearch.compute.data.arrow.BytesRefArrowBufVector

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.lucene.util.Accountable, BytesRefVector, Vector, RefCounted, Releasable

public final class BytesRefArrowBufVector
extends AbstractArrowBufVector<BytesRefVector,BytesRefBlock>
implements BytesRefVector

Arrow buffer backed BytesRefVector. Variable-length values use an additional offsets buffer that maps each value index to a byte range in the data buffer.

Nested Class Summary

Nested classes/interfaces inherited from interface org.elasticsearch.compute.data.BytesRefVector

BytesRefVector.Builder

Nested classes/interfaces inherited from interface org.elasticsearch.compute.data.Vector

Vector.Builder

Field Summary

Fields inherited from class org.elasticsearch.compute.data.arrow.AbstractArrowBufVector

blockFactory, closed, positionCount, valueBuffer

Fields inherited from interface org.apache.lucene.util.Accountable

NULL_ACCOUNTABLE

Fields inherited from interface org.elasticsearch.core.RefCounted

ALWAYS_REFERENCED

Fields inherited from interface org.elasticsearch.compute.data.Vector

SERIALIZE_VECTOR_ARRAY, SERIALIZE_VECTOR_BIG_ARRAY, SERIALIZE_VECTOR_CONSTANT, SERIALIZE_VECTOR_ORDINAL, SERIALIZE_VECTOR_VALUES

Constructor Summary

Constructors

Constructor	Description
BytesRefArrowBufVector(org.apache.arrow.memory.ArrowBuf valueBuffer, org.apache.arrow.memory.ArrowBuf valueOffsetsBuffer, int positionCount, BlockFactory blockFactory)

Method Summary

Modifier and Type	Method	Description
BytesRefBlock	asBlock()	Returns a new Block containing this vector.
OrdinalBytesRefVector	asOrdinals()	Returns an ordinal BytesRef vector if this vector is backed by a dictionary and ordinals; otherwise, returns null.
protected ArrowBufBlockConstructor<BytesRefBlock>	blockConstructor()
protected int	byteSize()
ElementType	elementType()	Returns the element type of this vector.
boolean	equals(Object obj)	Compares the given object with this vector for equality.
BytesRefVector	filter(boolean mayContainDuplicates, int[] positions, int offset, int length)	Creates a new vector that only exposes the positions provided.
PagedBytesCursor	get(int position, PagedBytesCursor scratch)	Retrieves the bytes value stored at the given value index using a PagedBytesCursor for zero-copy access to the underlying paged byte storage.
org.apache.lucene.util.BytesRef	getBytesRef(int position, org.apache.lucene.util.BytesRef dest)	Build a contiguous array of bytes for the value stored at the given position.
int	hashCode()	Returns the hash code of this vector, as defined by BytesRefVector.hash(BytesRefVector).
ReleasableIterator<BytesRefBlock>	lookup(IntBlock positions, ByteSizeValue targetBlockSize)	Builds an Iterator of new Blocks with the same Vector.elementType() as this Vector whose values are copied from positions in this Vector.
static BytesRefArrowBufVector	of(org.apache.arrow.vector.VarCharVector arrowVec, BlockFactory blockFactory)
long	ramBytesUsed()
void	releaseBuffers()	Releases (decrements the reference count) this vector's Arrow buffers
void	retainBuffers()	Retains (increments the reference count) this vector's Arrow buffers
BytesRefVector	slice(int beginInclusive, int endExclusive)	Return a subset of this vector from beginInclusive to endExclusive.
int	valueMaxByteSize()	The maximum size in bytes of any single value stored in this vector, or 0 if there are no values.
protected ArrowBufVectorConstructor<BytesRefVector>	vectorConstructor()
void	writeTo(StreamOutput out)	Serializes this Vector to the given stream output.

Methods inherited from class org.elasticsearch.compute.data.arrow.AbstractArrowBufVector

allowPassingToDifferentDriver, blockFactory, closeInternal, getPositionCount, isConstant, keepMask, valuesSegment

Methods inherited from class org.elasticsearch.compute.data.AbstractNonThreadSafeRefCounted

attachReleasable, close, decRef, hasReferences, incRef, isReleased, tryIncRef

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.lucene.util.Accountable

getChildResources

Methods inherited from interface org.elasticsearch.compute.data.BytesRefVector

deepCopy, filter, keepMask

Methods inherited from interface org.elasticsearch.core.RefCounted

decRef, hasReferences, incRef, mustIncRef, tryIncRef

Methods inherited from interface org.elasticsearch.core.Releasable

close

Methods inherited from interface org.elasticsearch.compute.data.Vector

allowPassingToDifferentDriver, attachReleasable, blockFactory, getPositionCount, isConstant, isReleased

Constructor Details

BytesRefArrowBufVector

public BytesRefArrowBufVector(org.apache.arrow.memory.ArrowBuf valueBuffer,
 org.apache.arrow.memory.ArrowBuf valueOffsetsBuffer,
 int positionCount,
 BlockFactory blockFactory)

Method Details

of

public static BytesRefArrowBufVector of(org.apache.arrow.vector.VarCharVector arrowVec,
 BlockFactory blockFactory)

vectorConstructor

protected ArrowBufVectorConstructor<BytesRefVector> vectorConstructor()

Specified by:

vectorConstructor in class AbstractArrowBufVector<BytesRefVector,BytesRefBlock>

blockConstructor

protected ArrowBufBlockConstructor<BytesRefBlock> blockConstructor()

Specified by:

blockConstructor in class AbstractArrowBufVector<BytesRefVector,BytesRefBlock>

retainBuffers

public void retainBuffers()

Description copied from class: AbstractArrowBufVector

Retains (increments the reference count) this vector's Arrow buffers

Overrides:

retainBuffers in class AbstractArrowBufVector<BytesRefVector,BytesRefBlock>

releaseBuffers

public void releaseBuffers()

Description copied from class: AbstractArrowBufVector

Releases (decrements the reference count) this vector's Arrow buffers

Overrides:

releaseBuffers in class AbstractArrowBufVector<BytesRefVector,BytesRefBlock>

getBytesRef

public org.apache.lucene.util.BytesRef getBytesRef(int position,
 org.apache.lucene.util.BytesRef dest)

Description copied from interface: BytesRefVector

Build a contiguous array of bytes for the value stored at the given position. The underlying data is generally stored in pages that look like byte[][] with some data spanning more than one of the inner byte[] arrays. In that case, this builds a byte[] in the BytesRef, copies the bytes, and returns it. Otherwise, this returns a zero-copy snapshot of the underlying data. Except arrow. Arrow always copies.

If possible, use BytesRefVector.get(int, org.elasticsearch.common.bytes.PagedBytesCursor) because it only needs to copy in the arrow implementation.

Specified by:

getBytesRef in interface BytesRefVector

Parameters:

position - the position index

dest - the destination

Returns:

the data value (as a BytesRef)

get

public PagedBytesCursor get(int position,
 PagedBytesCursor scratch)

Description copied from interface: BytesRefVector

Retrieves the bytes value stored at the given value index using a PagedBytesCursor for zero-copy access to the underlying paged byte storage. Except arrow. Arrow always copies.

Specified by:

get in interface BytesRefVector

Parameters:

position - the position index

scratch - the cursor to initialize and return

Returns:

the initialized cursor pointing to the value's bytes

byteSize

protected int byteSize()

Specified by:

byteSize in class AbstractArrowBufVector<BytesRefVector,BytesRefBlock>

elementType

public ElementType elementType()

Description copied from interface: Vector

Returns the element type of this vector.

Specified by:

elementType in interface Vector

Returns:

the element type of this vector

valueMaxByteSize

public int valueMaxByteSize()

Description copied from interface: BytesRefVector

The maximum size in bytes of any single value stored in this vector, or 0 if there are no values. Scans all positions to find the maximum.

Specified by:

valueMaxByteSize in interface BytesRefVector

Specified by:

valueMaxByteSize in interface Vector

Returns:

the maximum byte size of any single value in this vector

asOrdinals

public OrdinalBytesRefVector asOrdinals()

Description copied from interface: BytesRefVector

Returns an ordinal BytesRef vector if this vector is backed by a dictionary and ordinals; otherwise, returns null. Callers must not release the returned vector as no extra reference is retained by this method.

Specified by:

asOrdinals in interface BytesRefVector

asBlock

public BytesRefBlock asBlock()

Description copied from interface: Vector

Returns a new Block containing this vector.

Specified by:

asBlock in interface BytesRefVector

Specified by:

asBlock in interface Vector

Overrides:

asBlock in class AbstractArrowBufVector<BytesRefVector,BytesRefBlock>

Returns:

a new Block containing this vector

ramBytesUsed

public long ramBytesUsed()

Specified by:

ramBytesUsed in interface org.apache.lucene.util.Accountable

Overrides:

ramBytesUsed in class AbstractArrowBufVector<BytesRefVector,BytesRefBlock>

slice

public BytesRefVector slice(int beginInclusive,
 int endExclusive)

Description copied from interface: BytesRefVector

Return a subset of this vector from beginInclusive to endExclusive. This may return the same instance if the range covers all positions, but if it does it will RefCounted.incRef() it.

Specified by:

slice in interface BytesRefVector

Specified by:

slice in interface Vector

lookup

public ReleasableIterator<BytesRefBlock> lookup(IntBlock positions,
 ByteSizeValue targetBlockSize)

Description copied from interface: Vector

Builds an Iterator of new Blocks with the same Vector.elementType() as this Vector whose values are copied from positions in this Vector. It has the same number of positions as the positions parameter.

For example, if this vector contained [a, b, c] and were called with the block [0, 1, 1, [1, 2]] then the result would be [a, b, b, [b, c]].

This process produces count(positions) values per positions which could be quite large. Instead of returning a single Block, this returns an Iterator of Blocks containing all of the promised values.

The returned ReleasableIterator may retain a reference to the positions parameter. Close it to release those references.

This block is built using the same BlockFactory as was used to build the positions parameter.

Specified by:

lookup in interface BytesRefVector

Specified by:

lookup in interface Vector

filter

public BytesRefVector filter(boolean mayContainDuplicates,
 int[] positions,
 int offset,
 int length)

Description copied from interface: Vector

Creates a new vector that only exposes the positions provided. Materialization of the selected positions is avoided.

Specified by:

filter in interface BytesRefVector

Specified by:

filter in interface Vector

Overrides:

filter in class AbstractArrowBufVector<BytesRefVector,BytesRefBlock>

Parameters:

mayContainDuplicates - may the positions array contain duplicate positions?

positions - the positions array

offset - the start index in the positions array

length - the number of positions to use from the array

Returns:

a filtered vector

writeTo

public void writeTo(StreamOutput out)
             throws IOException

Description copied from interface: BytesRefVector

Serializes this Vector to the given stream output.

Specified by:

writeTo in interface BytesRefVector

Throws:

IOException

equals

public boolean equals(Object obj)

Description copied from interface: BytesRefVector

Compares the given object with this vector for equality. Returns true if and only if the given object is a BytesRefVector, and both vectors are equal.

Specified by:

equals in interface BytesRefVector

Overrides:

equals in class Object

hashCode

public int hashCode()

Description copied from interface: BytesRefVector

Returns the hash code of this vector, as defined by BytesRefVector.hash(BytesRefVector).

Specified by:

hashCode in interface BytesRefVector

Overrides:

hashCode in class Object