Package org.elasticsearch.xpack.esql.datasources.spi

Interface StorageObject

All Known Implementing Classes:

AbstractMeteredStorageObject

public interface StorageObject

Unified interface for storage object access.

Simple providers: implement sync methods - async wrapping is automatic. Async-capable providers (HTTP, S3): override async methods for native non-blocking I/O.

Provides metadata access and methods to open streams for reading. Uses standard Java InputStream for compatibility with existing Elasticsearch code. Random access is handled via range-based reads (like BlobContainer pattern).

StorageObject instances themselves do not hold open resources and do not need closing. The InputStream instances returned by newStream() and newStream(long, long) are the resources that callers must close.

Field Summary

Fields

Modifier and Type	Field	Description
static final long	READ_TO_END	Sentinel length for newStream(long, long) meaning "read from position to the end of the object" — the open-ended form.
static final int	TRANSFER_BUFFER_SIZE	Transfer buffer size used by the default readBytes(long, ByteBuffer) when reading into a direct ByteBuffer via an InputStream.

Method Summary

Modifier and Type	Method	Description
default void	abortStream(InputStream stream)	Closes a stream opened by this object, discarding any unread bytes without blocking.
default void	attachMetrics(ExternalSourceMetrics metrics, String scheme)	Attaches the node telemetry sink and the storage scheme dimension so that this object's read/retry events are published to ExternalSourceMetrics (in addition to the profile counters surfaced by metrics()).
boolean	exists()	Checks if the object exists.
Instant	lastModified()	Returns the last modification time, or null if not available.
long	length()	Returns the object size in bytes.
default StorageObjectMetrics	metrics()	Returns cumulative I/O counters for reads against this object.
default InputStream	newStream()	Opens an input stream for sequential reading of the whole object, from the beginning to the end.
InputStream	newStream(long position, long length)	Opens an input stream for reading a byte range.
StoragePath	path()	Returns the path of this object.
default int	readBytes(long position, ByteBuffer target)	Reads bytes from a specific position directly into a ByteBuffer.
default void	readBytesAsync(long position, long length, DirectBufferFactory factory, Executor executor, ActionListener<DirectReadBuffer> listener)	Async byte read with ActionListener callback.
default void	readBytesAsync(long position, ByteBuffer target, Executor executor, ActionListener<Integer> listener)	Async byte read into a caller-provided ByteBuffer.
default boolean	supportsNativeAsync()	Returns true if this object has native async support.

Field Details

TRANSFER_BUFFER_SIZE

static final int TRANSFER_BUFFER_SIZE

Transfer buffer size used by the default readBytes(long, ByteBuffer) when reading into a direct ByteBuffer via an InputStream. Kept small to avoid large stack allocations while still being efficient for typical I/O page sizes.

See Also:

• Constant Field Values

READ_TO_END

static final long READ_TO_END

Sentinel length for newStream(long, long) meaning "read from position to the end of the object" — the open-ended form. It exists because the total length is not always knowable up front (e.g. a compressed object has no addressable length), so the open-ended read must signal "to the end" with this marker rather than by computing length() - position.

See Also:

• Constant Field Values

Method Details

newStream

default InputStream newStream()
                       throws IOException

Opens an input stream for sequential reading of the whole object, from the beginning to the end.

The default is newStream(0, READ_TO_END), so the whole-object read shares the open-ended code path (and, through the decorator chain, its resilience). A provider may override this for a plain whole-object GET.

Throws:

IOException

newStream

InputStream newStream(long position,
 long length)
               throws IOException

Opens an input stream for reading a byte range. A length of READ_TO_END reads from position to the end of the object (the open-ended form) — essential for streams whose total length is not available. Providers MUST translate READ_TO_END into a native open-ended read (e.g. HTTP Range: bytes=position-) and MUST NOT require length() to serve it; an empty object (or position at/after the end) yields an empty stream.

Critical for columnar formats like Parquet that read specific column chunks. For reading object footers, use a bounded length: newStream(length() - footerSize, footerSize).

Throws:

IOException

length

long length()
     throws IOException

Returns the object size in bytes.

Throws:

IOException

lastModified

Instant lastModified()
              throws IOException

Returns the last modification time, or null if not available.

Throws:

IOException

exists

boolean exists()
        throws IOException

Checks if the object exists.

Throws:

IOException

path

StoragePath path()

Returns the path of this object.

abortStream

default void abortStream(InputStream stream)
                  throws IOException

Closes a stream opened by this object, discarding any unread bytes without blocking.

The default implementation calls InputStream.close(), which is correct for local files and most providers (they simply close the connection). Override this for providers whose close() drains remaining bytes to reuse the connection pool — where closing a partially-read stream would block for the full remaining object transfer time.

Use this instead of closing directly when the caller intentionally reads only a prefix of the stream (e.g., schema detection), and connection reuse is not required.

Contract: stream must be the exact InputStream instance returned by newStream() or newStream(long, long) on this object — not a wrapper around it. Passing a wrapped stream (e.g. a BufferedInputStream layered on top) causes provider-specific overrides (e.g. the S3 Abortable cast) to fall back to a draining close(), silently defeating the abort.

Throws:

IOException

readBytesAsync

default void readBytesAsync(long position,
 long length,
 DirectBufferFactory factory,
 Executor executor,
 ActionListener<DirectReadBuffer> listener)

Async byte read with ActionListener callback.

Default implementation wraps the sync readBytes(long, ByteBuffer) method in an executor. Override this method for native async I/O (e.g., HTTP sendAsync, S3AsyncClient).

Columnar formats (Parquet) can use this for parallel chunk reads when supportsNativeAsync() returns true.

Returned buffer contract: the DirectReadBuffer.buffer() delivered to the listener has capacity() == length (the requested length) and remaining() equal to the number of bytes actually read. On a short read these differ — consumers must use remaining() (or limit() - position()) to size their work, never capacity(). The buffer is direct.

On end-of-content at position the buffer is delivered with remaining() == 0.

Buffer ownership: the storage object obtains exactly one DirectReadBuffer of length bytes from factory. The caller must invoke DirectReadBuffer.close() once the bytes have been consumed; closing releases the buffer back to its underlying allocator. See DirectReadBuffer for the contract.

Implementation contract: if the read fails, implementations must close the DirectReadBuffer before calling listener.onFailure(). Callers that fan out reads across multiple merged ranges rely on this invariant to avoid double-releasing buffers from the successfully-completed sibling ranges on the failure path.

Parameters:

position - the starting byte position

length - the number of bytes to read

factory - produces the DirectReadBuffer the bytes are read into; the storage object calls DirectBufferFactory.allocate(int) exactly once with length

executor - executor for running the async operation

listener - callback for the result or failure

readBytesAsync

default void readBytesAsync(long position,
 ByteBuffer target,
 Executor executor,
 ActionListener<Integer> listener)

Async byte read into a caller-provided ByteBuffer.

Avoids per-call allocation by reading directly into the target buffer. For heap-backed buffers, reads directly into the backing array. For direct buffers, falls back to allocating a temporary array.

The buffer's position is advanced by the number of bytes read. The listener receives the number of bytes actually read.

Parameters:

position - the starting byte position in the storage object

target - the ByteBuffer to read into; bytes are written starting at target.position()

executor - executor for running the async operation

listener - callback with the number of bytes read, or failure

readBytes

default int readBytes(long position,
 ByteBuffer target)
               throws IOException

Reads bytes from a specific position directly into a ByteBuffer.

This is a positional, stateless read: each call specifies the position explicitly, so no mutable cursor state is needed. Providers override this to enable zero-copy I/O:

• Local files: FileChannel.read(target, position) reads from OS page cache directly into both heap and direct buffers with no intermediate copies.
• GCS: ReadChannel.read(target) reads natively into the ByteBuffer.
• S3/HTTP/Azure: The default stream-based implementation is used; for heap-backed buffers it reads directly into the backing array, for direct buffers it uses a small chunked transfer buffer (8 KB) instead of allocating a full-size temporary array.

The buffer's position is advanced by the number of bytes read.

Parameters:

position - the starting byte position in the storage object

target - the ByteBuffer to read into; bytes are written starting at target.position()

Returns:

the number of bytes actually read, or -1 if the position is at or past end of content

Throws:

IOException - if an I/O error occurs

supportsNativeAsync

default boolean supportsNativeAsync()

Returns true if this object has native async support.

Columnar formats (Parquet) can use this to determine whether to use readBytesAsync(long, long, org.elasticsearch.xpack.esql.datasources.spi.DirectBufferFactory, java.util.concurrent.Executor, org.elasticsearch.action.ActionListener<org.elasticsearch.xpack.esql.datasources.spi.DirectReadBuffer>) for parallel chunk reads instead of sequential stream-based reads.

Returns:

true if readBytesAsync(long, long, org.elasticsearch.xpack.esql.datasources.spi.DirectBufferFactory, java.util.concurrent.Executor, org.elasticsearch.action.ActionListener<org.elasticsearch.xpack.esql.datasources.spi.DirectReadBuffer>) has a native implementation, false if it uses the default sync wrapper

metrics

default StorageObjectMetrics metrics()

Returns cumulative I/O counters for reads against this object.

Implementations that don't track I/O return StorageObjectMetrics.ZERO. Decorator wrappers must delegate to the wrapped object so counters are attributed to the underlying store, not the wrapper layer.

attachMetrics

default void attachMetrics(ExternalSourceMetrics metrics,
 String scheme)

Attaches the node telemetry sink and the storage scheme dimension so that this object's read/retry events are published to ExternalSourceMetrics (in addition to the profile counters surfaced by metrics()). Called once by the operator wiring when it opens the object. The default is a no-op for objects that don't track I/O; decorator wrappers must forward to the wrapped object so the metrics attach to the underlying store, not the wrapper layer.