Package org.elasticsearch.xpack.esql.datasources.spi

Class DirectReadBuffer

java.lang.Object
org.elasticsearch.xpack.esql.datasources.spi.DirectReadBuffer
All Implemented Interfaces:
Closeable, AutoCloseable, Releasable
public final class DirectReadBuffer extends Object implements Releasable
Result of StorageObject.readBytesAsync(long, long, DirectBufferFactory, java.util.concurrent.Executor, org.elasticsearch.action.ActionListener): the bytes plus a Releasable that frees the native memory backing them.

The buffer is a direct ByteBuffer view of an ArrowBuf obtained from the caller-supplied DirectBufferFactory (via DirectBufferFactory.allocate(int)). The caller must invoke close() once the bytes have been consumed: closing decrements the ArrowBuf's reference count, which is what actually returns the memory to the underlying allocator. Closing the allocator alone is not enough — Arrow's BaseAllocator.close() treats outstanding ArrowBufs as a leak.

Using buffer() after close() reads dangling memory; the underlying chunk may have been recycled into another allocation.

Use-after-free / double-free detection (assertions only)

The bytes are exposed as a detached ByteBuffer (via ArrowBuf.nioBuffer(...)), so reads through that view bypass Arrow's reference-count tracking entirely — the Arrow debug allocator can detect a double-free or a leak, but it is blind to a read that aliases this buffer after close() has freed it. That is the failure mode behind the nondeterministic zstd "Src size is incorrect" / "Destination buffer is too small" corruption: a slice handed out before close is read after the backing ArrowBuf was returned to the allocator and recycled.

To make that class of bug deterministic and self-locating, when assertions are enabled this type:

  • captures the allocation stack trace at construction and the free stack trace at close() ("who deallocated this"),
  • throws on a second close() (double-free) and on any buffer() access after close (use-after-free), attaching both stack traces, and
  • poisons the direct memory with a recognizable pattern immediately before releasing it, so any surviving alias that reads the freed region fails the same way on every run instead of occasionally seeing still-intact bytes.
All of this compiles out (no allocation, no poisoning) when assertions are disabled, so the production read path is unchanged.

  • Constructor Details

  • Method Details

    • allocate

      public static DirectReadBuffer allocate(org.apache.arrow.memory.BufferAllocator allocator, int length)
      Bridge used by DirectBufferFactory.forAllocator(BufferAllocator): allocates an ArrowBuf of length bytes from allocator and wraps it as a DirectReadBuffer. Backends should call DirectBufferFactory.allocate(int) instead of this method directly.

      The returned buffer's contents are uninitialized; the caller is responsible for filling buffer() before delivering it downstream and for calling close() once consumption is complete (or on the failure path).

      The intermediate ArrowBuf is released on every failure path. Allocator failures (breaker trip, OutOfMemoryError, Arrow runtime exceptions) propagate as-is so callers can distinguish a circuit-breaker rejection (eligible for a 429 response) from an I/O error.

    • buffer

      public ByteBuffer buffer()
      The bytes. Must not be accessed after close(); doing so reads freed (and possibly recycled) native memory. When es.arrow.debug_buffers is on, a post-close access throws with the allocation and free stack traces attached.

    • release

      public Releasable release()
      The underlying Releasable. Prefer close(), which adds lifecycle checks.

    • close

      public void close()
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Specified by:
      close in interface Releasable