Module org.elasticsearch.compute

Package org.elasticsearch.compute.data.arrow

Class BytesRefArrowBufBlock

java.lang.Object

org.elasticsearch.compute.data.AbstractNonThreadSafeRefCounted

org.elasticsearch.compute.data.arrow.AbstractArrowBufBlock<BytesRefVector,BytesRefBlock>

org.elasticsearch.compute.data.arrow.BytesRefArrowBufBlock

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.lucene.util.Accountable, Writeable, Block, BytesRefBlock, RefCounted, Releasable, BlockLoader.Block

public final class BytesRefArrowBufBlock
extends AbstractArrowBufBlock<BytesRefVector,BytesRefBlock>
implements BytesRefBlock

Arrow buffer backed BytesRefBlock. Variable-length values use an additional offsets buffer that maps each value index to a byte range in the data buffer, following the Arrow variable-length binary layout. This is separate from the position-to-value-index offsets used for multi-valued fields.

Nested Class Summary

Nested classes/interfaces inherited from interface org.elasticsearch.compute.data.Block

Block.Builder, Block.MvOrdering

Nested classes/interfaces inherited from interface org.elasticsearch.compute.data.BytesRefBlock

BytesRefBlock.Builder

Nested classes/interfaces inherited from interface org.elasticsearch.common.io.stream.Writeable

Writeable.Reader<V>, Writeable.Writer<V>

Field Summary

Fields inherited from class org.elasticsearch.compute.data.arrow.AbstractArrowBufBlock

blockFactory, closed, offsetBuffer, offsetCount, validityBuffer, valueBuffer, valueCount

Fields inherited from interface org.apache.lucene.util.Accountable

NULL_ACCOUNTABLE

Fields inherited from interface org.elasticsearch.compute.data.Block

ESQL_AGGREGATE_METRIC_DOUBLE_BLOCK, MAX_LOOKUP, PAGE_MEM_OVERHEAD_PER_BLOCK, SERIALIZE_BLOCK_ARRAY, SERIALIZE_BLOCK_BIG_ARRAY, SERIALIZE_BLOCK_ORDINAL, SERIALIZE_BLOCK_VALUES, SERIALIZE_BLOCK_VECTOR

Fields inherited from interface org.elasticsearch.compute.data.BytesRefBlock

NULL_VALUE

Fields inherited from interface org.elasticsearch.core.RefCounted

ALWAYS_REFERENCED

Constructor Summary

Constructors

Constructor	Description
BytesRefArrowBufBlock(org.apache.arrow.memory.ArrowBuf valueBuffer, org.apache.arrow.memory.ArrowBuf valueOffsetsBuffer, org.apache.arrow.memory.ArrowBuf validityBuffer, org.apache.arrow.memory.ArrowBuf offsetBuffer, int valueCount, int offsetCount, BlockFactory blockFactory)

Method Summary

Modifier and Type	Method	Description
OrdinalBytesRefBlock	asOrdinals()	Returns an ordinal bytesref block if this block is backed by a dictionary and ordinals; otherwise, returns null.
BytesRefVector	asVector()	Returns an efficient dense single-value view of this block.
protected ArrowBufBlockConstructor<BytesRefBlock>	blockConstructor()
protected int	byteSize()
ElementType	elementType()	Returns the element type of this block.
boolean	equals(Object obj)	Compares the given object with this block for equality.
BytesRefBlock	expand()	Expand multivalued fields into one row per value.
BytesRefBlock	filter(boolean mayContainDuplicates, int[] positions, int offset, int length)	Creates a new block that only exposes the positions provided.
PagedBytesCursor	get(int valueIndex, PagedBytesCursor scratch)	Retrieves the bytes value stored at the given value index using a PagedBytesCursor for zero-copy access to the underlying paged byte storage.
org.apache.lucene.util.BytesRef	getBytesRef(int valueIndex, org.apache.lucene.util.BytesRef dest)	Build a contiguous array of bytes for the value stored at the given position.
int	hashCode()	Returns the hash code of this block, as defined by BytesRefBlock.hash(BytesRefBlock).
BytesRefBlock	keepMask(BooleanVector mask)	Build a Block with the same values as this Block, but replacing all values for which mask.getBooleanValue(position) returns false with null.
ReleasableIterator<? extends BytesRefBlock>	lookup(IntBlock positions, ByteSizeValue targetBlockSize)	Builds an Iterator of new Blocks with the same Block.elementType() as this Block whose values are copied from positions in this Block.
static BytesRefBlock	of(org.apache.arrow.vector.ValueVector arrowVector, BlockFactory blockFactory)
long	ramBytesUsed()
void	releaseBuffers()	Releases (decrements the reference count) this block's Arrow buffers
void	retainBuffers()	Retains (increments the reference count) this block's Arrow buffers
int	valueMaxByteSize()	The maximum size in bytes of any single value stored in this block, or 0 if there are no values.
protected ArrowBufVectorConstructor<BytesRefVector>	vectorConstructor()

Methods inherited from class org.elasticsearch.compute.data.arrow.AbstractArrowBufBlock

allowPassingToDifferentDriver, areAllValuesNull, blockFactory, closeInternal, doesHaveMultivaluedFields, getFirstValueIndex, getPositionCount, getTotalValueCount, getValueCount, isNull, mayHaveMultivaluedFields, mayHaveNulls, mvOrdering, setValidityBit, validityBufferLength

Methods inherited from class org.elasticsearch.compute.data.AbstractNonThreadSafeRefCounted

attachReleasable, close, decRef, hasReferences, incRef, isReleased, tryIncRef

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.lucene.util.Accountable

getChildResources

Methods inherited from interface org.elasticsearch.compute.data.Block

allowPassingToDifferentDriver, areAllValuesNull, attachReleasable, blockFactory, doesHaveMultivaluedFields, getFirstValueIndex, getPositionCount, getTotalValueCount, getValueCount, insertNulls, isNull, isReleased, mayHaveMultivaluedFields, mayHaveNulls, mvDeduplicated, mvOrdering, mvSortedAscending

Methods inherited from interface org.elasticsearch.compute.data.BytesRefBlock

deepCopy, filter, hasValue, slice, writeTo

Methods inherited from interface org.elasticsearch.core.RefCounted

decRef, hasReferences, incRef, mustIncRef, tryIncRef

Methods inherited from interface org.elasticsearch.core.Releasable

close

Constructor Details

BytesRefArrowBufBlock

public BytesRefArrowBufBlock(org.apache.arrow.memory.ArrowBuf valueBuffer,
 org.apache.arrow.memory.ArrowBuf valueOffsetsBuffer,
 @Nullable
 org.apache.arrow.memory.ArrowBuf validityBuffer,
 @Nullable
 org.apache.arrow.memory.ArrowBuf offsetBuffer,
 int valueCount,
 int offsetCount,
 BlockFactory blockFactory)

Parameters:

valueBuffer - raw byte data

valueOffsetsBuffer - int32 offsets into valueBuffer for each value (Arrow variable-length layout)

validityBuffer - null bitmap (nullable)

offsetBuffer - position-to-value-index mapping for multi-valued fields (nullable)

valueCount - number of positions

offsetCount - number of position offset entries

blockFactory - block factory

Method Details

of

public static BytesRefBlock of(org.apache.arrow.vector.ValueVector arrowVector,
 BlockFactory blockFactory)

retainBuffers

public void retainBuffers()

Description copied from class: AbstractArrowBufBlock

Retains (increments the reference count) this block's Arrow buffers

Overrides:

retainBuffers in class AbstractArrowBufBlock<BytesRefVector,BytesRefBlock>

releaseBuffers

public void releaseBuffers()

Description copied from class: AbstractArrowBufBlock

Releases (decrements the reference count) this block's Arrow buffers

Overrides:

releaseBuffers in class AbstractArrowBufBlock<BytesRefVector,BytesRefBlock>

byteSize

protected int byteSize()

Specified by:

byteSize in class AbstractArrowBufBlock<BytesRefVector,BytesRefBlock>

vectorConstructor

protected ArrowBufVectorConstructor<BytesRefVector> vectorConstructor()

Specified by:

vectorConstructor in class AbstractArrowBufBlock<BytesRefVector,BytesRefBlock>

blockConstructor

protected ArrowBufBlockConstructor<BytesRefBlock> blockConstructor()

Specified by:

blockConstructor in class AbstractArrowBufBlock<BytesRefVector,BytesRefBlock>

getBytesRef

public org.apache.lucene.util.BytesRef getBytesRef(int valueIndex,
 org.apache.lucene.util.BytesRef dest)

Description copied from interface: BytesRefBlock

Build a contiguous array of bytes for the value stored at the given position. The underlying data is generally stored in pages that look like byte[][] with some data spanning more than one of the inner byte[] arrays. In that case, this builds a byte[] in the BytesRef, copies the bytes, and returns it. Otherwise, this returns a zero-copy snapshot of the underlying data. Except arrow. Arrow always copies.

If possible, use BytesRefBlock.get(int, org.elasticsearch.common.bytes.PagedBytesCursor) because it only needs to copy in the arrow implementation.

There are Block.getValueCount(int) values in each position. You can access them all with something like:

   int start = getFirstValueIndex(position);
   int end = start + getValueCount(position);
   for (int i = start; i < end; i++) {
      BytesRef v = getBytesRef(i, scratch);
      // do stuff
   }

Specified by:

getBytesRef in interface BytesRefBlock

Parameters:

valueIndex - the value index

Returns:

the data value (as a BytesRef)

get

public PagedBytesCursor get(int valueIndex,
 PagedBytesCursor scratch)

Description copied from interface: BytesRefBlock

Retrieves the bytes value stored at the given value index using a PagedBytesCursor for zero-copy access to the underlying paged byte storage. Except arrow. Arrow always copies.

There are Block.getValueCount(int) values in each position. You can access them all with something like:

   int start = getFirstValueIndex(position);
   int end = start + getValueCount(position);
   for (int i = start; i < end; i++) {
      PagedBytesCursor v = get(i, scratch);
      // do stuff
   }

Specified by:

get in interface BytesRefBlock

Parameters:

valueIndex - the value index

scratch - the cursor to initialize and return

Returns:

the initialized cursor pointing to the value's bytes

elementType

public ElementType elementType()

Description copied from interface: Block

Returns the element type of this block.

Specified by:

elementType in interface Block

Returns:

the element type of this block

valueMaxByteSize

public int valueMaxByteSize()

Description copied from interface: BytesRefBlock

The maximum size in bytes of any single value stored in this block, or 0 if there are no values. Scans all value indices to find the maximum.

Specified by:

valueMaxByteSize in interface Block

Specified by:

valueMaxByteSize in interface BytesRefBlock

Returns:

the maximum byte size of any single value in this block

asOrdinals

public OrdinalBytesRefBlock asOrdinals()

Description copied from interface: BytesRefBlock

Returns an ordinal bytesref block if this block is backed by a dictionary and ordinals; otherwise, returns null. Callers must not release the returned block as no extra reference is retained by this method.

Specified by:

asOrdinals in interface BytesRefBlock

asVector

public BytesRefVector asVector()

Description copied from interface: Block

Returns an efficient dense single-value view of this block. Null, if the block is not dense single-valued. That is, if mayHaveNulls returns true, or getTotalValueCount is not equal to getPositionCount.

Specified by:

asVector in interface Block

Specified by:

asVector in interface BytesRefBlock

Overrides:

asVector in class AbstractArrowBufBlock<BytesRefVector,BytesRefBlock>

Returns:

an efficient dense single-value view of this block

ramBytesUsed

public long ramBytesUsed()

Specified by:

ramBytesUsed in interface org.apache.lucene.util.Accountable

Overrides:

ramBytesUsed in class AbstractArrowBufBlock<BytesRefVector,BytesRefBlock>

filter

public BytesRefBlock filter(boolean mayContainDuplicates,
 int[] positions,
 int offset,
 int length)

Description copied from interface: Block

Creates a new block that only exposes the positions provided.

Specified by:

filter in interface Block

Specified by:

filter in interface BytesRefBlock

Overrides:

filter in class AbstractArrowBufBlock<BytesRefVector,BytesRefBlock>

Parameters:

mayContainDuplicates - may the positions array contain duplicate positions?

positions - the positions to retain

offset - the start index in the positions array

length - the number of positions to use from the array

Returns:

a filtered block

keepMask

public BytesRefBlock keepMask(BooleanVector mask)

Description copied from interface: Block

Build a Block with the same values as this Block, but replacing all values for which mask.getBooleanValue(position) returns false with null. The mask vector must be at least as long as this Block.

Specified by:

keepMask in interface Block

Specified by:

keepMask in interface BytesRefBlock

Overrides:

keepMask in class AbstractArrowBufBlock<BytesRefVector,BytesRefBlock>

expand

public BytesRefBlock expand()

Description copied from interface: Block

Expand multivalued fields into one row per value. Returns the same block if there aren't any multivalued fields to expand. The returned block needs to be closed by the caller to release the block's resources.

Specified by:

expand in interface Block

Specified by:

expand in interface BytesRefBlock

Overrides:

expand in class AbstractArrowBufBlock<BytesRefVector,BytesRefBlock>

lookup

public ReleasableIterator<? extends BytesRefBlock> lookup(IntBlock positions,
 ByteSizeValue targetBlockSize)

Description copied from interface: Block

Builds an Iterator of new Blocks with the same Block.elementType() as this Block whose values are copied from positions in this Block. It has the same number of positions as the positions parameter.

For example, if this block contained [a, b, [b, c]] and were called with the block [0, 1, 1, [1, 2]] then the result would be [a, b, b, [b, b, c]].

This process produces count(this) * count(positions) values per positions which could be quite large. Instead of returning a single Block, this returns an Iterator of Blocks containing all of the promised values.

The returned ReleasableIterator may retain a reference to the positions parameter. Close it to release those references.

This block is built using the same BlockFactory as was used to build the positions parameter.

Specified by:

lookup in interface Block

Specified by:

lookup in interface BytesRefBlock

Overrides:

lookup in class AbstractArrowBufBlock<BytesRefVector,BytesRefBlock>

equals

public boolean equals(Object obj)

Description copied from interface: BytesRefBlock

Compares the given object with this block for equality. Returns true if and only if the given object is a BytesRefBlock, and both blocks are equal.

Specified by:

equals in interface BytesRefBlock

Overrides:

equals in class Object

hashCode

public int hashCode()

Description copied from interface: BytesRefBlock

Returns the hash code of this block, as defined by BytesRefBlock.hash(BytesRefBlock).

Specified by:

hashCode in interface BytesRefBlock

Overrides:

hashCode in class Object