Package org.elasticsearch.xpack.esql.datasources.glob

package org.elasticsearch.xpack.esql.datasources.glob
Glob pattern matching and expansion for ES|QL external datasources.

This package handles path resolution for external data sources (S3, GCS, Azure, HTTP, local filesystem) by expanding user-supplied patterns into concrete object paths.

Supported Patterns

Glob pattern syntax reference
Pattern Description Example Matches
* Matches zero or more characters within a single path segment (does not cross /). data/*.parquet data/sales.parquet, data/orders.parquet
** Matches zero or more path segments, including nested directories. data/**/*.parquet data/2024/01/file.parquet, data/file.parquet
? Matches exactly one character (not /). file-?.csv file-A.csv, file-1.csv
[abc] Matches one character from the set. Supports ranges ([0-9]) and negation ([!abc] or [^abc]). file-[0-9].csv file-0.csv through file-9.csv
{a,b,c} Expands into one candidate per comma-separated alternative. Multiple groups produce the Cartesian product. {sales,orders}/year={2024,2025}/*.parquet sales/year=2024/*.parquet, sales/year=2025/*.parquet, orders/year=2024/*.parquet, orders/year=2025/*.parquet
{N..M} Numeric range expansion. Generates one candidate per integer in the range (inclusive, ascending or descending). Leading zeros on either operand enable zero-padded output using the wider operand's width (bash semantics). shard-{000..099}.parquet shard-000.parquet, shard-001.parquet, … shard-099.parquet

Resolution Strategy

Patterns are resolved through one of two paths, chosen automatically:

  • Brace-only fast path — When the pattern contains only brace groups ({a,b} or {N..M}) and no other metacharacters, each expanded candidate is checked individually via exists() (HEAD request on cloud storage), avoiding a potentially expensive listing operation.
  • Listing path — When the pattern contains *, ?, or [...], objects are listed under the longest non-pattern prefix and filtered through regex-based matching.

Both paths respect the esql.external.max_glob_expansion cluster setting (default 100) which caps the number of expanded candidates from brace/range groups before falling back to listing, and the esql.external.max_discovered_files setting which caps total discovered files.

See Also: