Package org.elasticsearch.xpack.esql.datasources

Class FileSplitProvider

java.lang.Object

org.elasticsearch.xpack.esql.datasources.FileSplitProvider

All Implemented Interfaces:

SplitProvider

public class FileSplitProvider
extends Object
implements SplitProvider

Default SplitProvider for file-based sources. Converts each file in the FileList into a FileSplit, applying L1 partition pruning when filter hints and partition metadata are available.

When filter hints contain resolved Expression objects, evaluates them against each file's partition values to prune files that cannot match the filter.

Splitting modes. This provider supports two distinct splitting strategies. The downstream reader's behaviour (partial-line skip vs. no skip) differs between them, gated by FormatReadContext.recordAligned().

• Record-aligned macro splits — for uncompressed line-oriented formats (NDJSON/JSONL/JSON, CSV/TSV). RecordSplitter.findNextRecordBoundary(java.io.InputStream) probes near target_split_size strides so each FileSplit starts on a record boundary. Splits are tagged with RECORD_ALIGNED_MACRO_SPLIT_KEY and readers receive recordAligned=true, so they must not drop any leading bytes. See tryNewlineAlignedMacroSplits(org.elasticsearch.xpack.esql.datasources.spi.StoragePath, long, java.lang.String, java.util.Map<java.lang.String, java.lang.Object>, java.util.Map<java.lang.String, java.lang.Object>, org.elasticsearch.xpack.esql.datasources.ColumnMapping, java.util.List<org.elasticsearch.xpack.esql.core.expression.Attribute>, long, int, java.util.List<org.elasticsearch.xpack.esql.datasources.spi.ExternalSplit>, org.elasticsearch.xpack.esql.datasources.spi.StorageProvider, org.elasticsearch.xpack.esql.datasources.spi.FormatReader, java.util.function.BooleanSupplier).
• Block-aligned splits — for splittable compressed formats (e.g. bzip2) via SplittableDecompressionCodec.findBlockBoundaries(org.elasticsearch.xpack.esql.datasources.spi.StorageObject, long, long). Splits land on compression block boundaries, not record boundaries. Readers receive recordAligned=false and must skip a leading partial record on every non-first split. See tryBlockAlignedSplits(org.elasticsearch.xpack.esql.datasources.spi.StoragePath, long, java.lang.String, java.util.Map<java.lang.String, java.lang.Object>, java.util.Map<java.lang.String, java.lang.Object>, org.elasticsearch.xpack.esql.datasources.ColumnMapping, java.util.List<org.elasticsearch.xpack.esql.core.expression.Attribute>, java.util.List<org.elasticsearch.xpack.esql.datasources.spi.ExternalSplit>, org.elasticsearch.xpack.esql.datasources.spi.StorageProvider).

Field Summary

Fields

Modifier and Type	Field	Description
static final Set<String>	CONFIG_KEYS	Configuration keys this splitter consumes from a query-time configuration map.
static final String	CONFIG_TARGET_SPLIT_SIZE

Fields inherited from interface org.elasticsearch.xpack.esql.datasources.spi.SplitProvider

SINGLE

Constructor Summary

Constructors

Constructor	Description
FileSplitProvider()
FileSplitProvider(long targetSplitSizeBytes)
FileSplitProvider(long targetSplitSizeBytes, DecompressionCodecRegistry codecRegistry, StorageProviderRegistry storageRegistry, Settings settings)
FileSplitProvider(long targetSplitSizeBytes, DecompressionCodecRegistry codecRegistry, StorageProviderRegistry storageRegistry, FormatReaderRegistry formatRegistry, Settings settings)
FileSplitProvider(long targetSplitSizeBytes, DecompressionCodecRegistry codecRegistry, StorageProviderRegistry storageRegistry, FormatReaderRegistry formatRegistry, Settings settings, Executor executor)

Method Summary

Modifier and Type	Method	Description
SplitDiscoveryResult	discoverSplits(SplitDiscoveryContext context)
static boolean	isRecordAlignedMacroSplit(FileSplit split)	Whether this leaf split came from tryNewlineAlignedMacroSplits(org.elasticsearch.xpack.esql.datasources.spi.StoragePath, long, java.lang.String, java.util.Map<java.lang.String, java.lang.Object>, java.util.Map<java.lang.String, java.lang.Object>, org.elasticsearch.xpack.esql.datasources.ColumnMapping, java.util.List<org.elasticsearch.xpack.esql.core.expression.Attribute>, long, int, java.util.List<org.elasticsearch.xpack.esql.datasources.spi.ExternalSplit>, org.elasticsearch.xpack.esql.datasources.spi.StorageProvider, org.elasticsearch.xpack.esql.datasources.spi.FormatReader, java.util.function.BooleanSupplier).
static StorageObject	storageObjectForSplit(StorageProvider storageProvider, FileSplit fileSplit)	Builds a StorageObject that exposes only the bytes for the given FileSplit.
static long	validateTargetSplitSize(String value)	Parses and validates an already-trimmed target_split_size value, returning the size in bytes.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

CONFIG_TARGET_SPLIT_SIZE

public static final String CONFIG_TARGET_SPLIT_SIZE

See Also:

• Constant Field Values

CONFIG_KEYS

public static final Set<String> CONFIG_KEYS

Configuration keys this splitter consumes from a query-time configuration map. Aggregated by FileSourceFactory.COORDINATOR_KEYS. New keys read by this class via config.get(...) must be added here so the ConfigKeyValidator recognises them — pinned by FileSourceFactoryValidationTests.

Constructor Details

FileSplitProvider

public FileSplitProvider()

FileSplitProvider

public FileSplitProvider(long targetSplitSizeBytes)

FileSplitProvider

public FileSplitProvider(long targetSplitSizeBytes,
 DecompressionCodecRegistry codecRegistry,
 StorageProviderRegistry storageRegistry,
 Settings settings)

FileSplitProvider

public FileSplitProvider(long targetSplitSizeBytes,
 DecompressionCodecRegistry codecRegistry,
 StorageProviderRegistry storageRegistry,
 FormatReaderRegistry formatRegistry,
 Settings settings)

FileSplitProvider

public FileSplitProvider(long targetSplitSizeBytes,
 DecompressionCodecRegistry codecRegistry,
 StorageProviderRegistry storageRegistry,
 FormatReaderRegistry formatRegistry,
 Settings settings,
 @Nullable
 Executor executor)

Method Details

discoverSplits

public SplitDiscoveryResult discoverSplits(SplitDiscoveryContext context)

Specified by:

discoverSplits in interface SplitProvider

storageObjectForSplit

public static StorageObject storageObjectForSplit(StorageProvider storageProvider,
 FileSplit fileSplit)

Builds a StorageObject that exposes only the bytes for the given FileSplit. Always wraps the provider's base object in RangeStorageObject so format readers and splittable decompressors only see the split's compressed byte span (including offset 0).

isRecordAlignedMacroSplit

public static boolean isRecordAlignedMacroSplit(FileSplit split)

Whether this leaf split came from tryNewlineAlignedMacroSplits(org.elasticsearch.xpack.esql.datasources.spi.StoragePath, long, java.lang.String, java.util.Map<java.lang.String, java.lang.Object>, java.util.Map<java.lang.String, java.lang.Object>, org.elasticsearch.xpack.esql.datasources.ColumnMapping, java.util.List<org.elasticsearch.xpack.esql.core.expression.Attribute>, long, int, java.util.List<org.elasticsearch.xpack.esql.datasources.spi.ExternalSplit>, org.elasticsearch.xpack.esql.datasources.spi.StorageProvider, org.elasticsearch.xpack.esql.datasources.spi.FormatReader, java.util.function.BooleanSupplier).

validateTargetSplitSize

public static long validateTargetSplitSize(String value)

Parses and validates an already-trimmed target_split_size value, returning the size in bytes. Shared by the query path (resolveTargetSplitSize(java.util.Map<java.lang.String, java.lang.Object>)) and the dataset CRUD validator so both accept exactly the same inputs. The caller owns trimming and the null/empty fallback to a default; this method always parses.

Throws:

ElasticsearchParseException - if the unit suffix is missing or malformed

IllegalArgumentException - if the resulting size is not positive