Package org.elasticsearch.xpack.esql.datasources

Class ExternalSourceSettings

java.lang.Object
org.elasticsearch.xpack.esql.datasources.ExternalSourceSettings
public final class ExternalSourceSettings extends Object
Cluster settings for controlling ESQL external source behavior; all node-scoped. The external-read concurrency bound (MAX_CONCURRENT_REQUESTS) and the throttle retry budget are read at node startup — the concurrency bound sizes both the per-scheme permit semaphores and the backend SDK HTTP connection pools when they are built, the retry budget is read when the storage-provider registry initializes — so a change to either takes effect after a node restart.

Covers three areas: the external-read concurrency bound (the single MAX_CONCURRENT_REQUESTS knob, which sizes the in-flight-read permit semaphore and the SDK connection pools below it); reactive throttle handling for object stores (the retry duration budget — throttling is handled by backoff, not a concurrency cap); and glob/listing safety limits (max discovered files, max brace expansion) to prevent degenerate queries from overwhelming storage backends.

  • Field Details

    • MAX_CONCURRENT_REQUESTS

      public static final Setting<Integer> MAX_CONCURRENT_REQUESTS
      The single external-read concurrency knob, per scheme, per node. It sizes both the per-scheme permit semaphore that bounds in-flight data reads (StorageProviderRegistry) and the backend SDK HTTP connection pools (S3 Netty maxConcurrency, Azure reactor-netty ConnectionProvider), and — via blobStoreConcurrency(Settings) — the metadata-discovery fan-out. Set to 0 to disable permit-based concurrency limiting entirely.

      The default is CPU-bound rather than a fixed literal: defaultBlobStoreConcurrency(Settings) — the snapshot_meta sizing shape (allocatedProcessors * 3) clamped to [16, 100]. That scales in-flight reads with node size so a wide fan-out over many small blobs is not self-throttled by a low fixed cap, while the floor keeps small nodes from collapsing to a handful of permits and the ceiling bounds a single store's load. Operators can raise it (up to 500) for high-throughput clusters or lower it when a store throttles.

      Static (Setting.Property.NodeScope): the value sizes the per-scheme semaphores and SDK pools when they are built and there is no settings-update consumer to resize a live Semaphore or connection pool, so a change takes effect after a node restart.

    • MAX_CONCURRENT_SEGMENTATORS

      public static final Setting<Integer> MAX_CONCURRENT_SEGMENTATORS
      Upper bound on how many stream-only-compressed (gzip/zstd) segmentators may occupy the esql_external_io pool at once, across all queries and operators on the node. Each open streaming read runs one long-lived segmentator that pins a pool thread while it blocks on its chunk/buffer queues and the upstream decompress stream; its per-chunk parser tasks run on the same pool. Without a cap, enough concurrently-open reads pin every pool thread and their parser tasks — queued behind the segmentators — never get a thread, deadlocking the reads (elastic/esql-planning #1093, item 4). Bounding concurrent segmentators below the pool size guarantees threads remain free for parser tasks.

      Default 0 means "derive": the effective cap tracks the esql_external_io pool size (externalIoThreads(Settings)), clamped to poolSize - 1 so at least one pool thread always remains for parser tasks. An explicit positive value overrides the derivation (still clamped below the pool size). Node-scoped: the pool is sized at startup.

    • THROTTLE_MAX_RETRY_DURATION

      public static final Setting<Integer> THROTTLE_MAX_RETRY_DURATION
      Maximum total time (in seconds) to spend retrying throttled cloud API requests before giving up. Bounds the cumulative retry duration regardless of the retry count, ensuring queries fail cleanly when throttling is persistent rather than blocking until the HTTP request timeout fires. Default: 30 seconds. Set to 0 to disable the duration budget (retry count only).

    • MAX_DISCOVERED_FILES

      public static final Setting<Integer> MAX_DISCOVERED_FILES
      Hard cap on the number of files that glob expansion will collect before aborting. Protects against degenerate globs (e.g. s3://bucket/*) on large buckets. Default: 10,000 — generous for legitimate use, catches truly degenerate cases.

    • MAX_GLOB_EXPANSION

      public static final Setting<Integer> MAX_GLOB_EXPANSION
      Cap on the number of concrete paths generated by brace expansion before falling back to listing. Prevents patterns like {1,2,...,10000} from generating excessive HeadObject calls. Default: 100 — above this, the system uses normal listing instead of per-key checks.

    • WORKLOAD_IDENTITY_ENABLED

      public static final Setting<Boolean> WORKLOAD_IDENTITY_ENABLED
      Deprecated former name for MANAGED_IDENTITY_ENABLED. Still honored for backwards compatibility — it is the fallback source for the new key, so an operator's existing esql.datasource.workload_identity.enabled config keeps working — and emits a deprecation warning when set. Prefer MANAGED_IDENTITY_ENABLED.

    • MANAGED_IDENTITY_ENABLED

      public static final Setting<Boolean> MANAGED_IDENTITY_ENABLED
      Enables auth=managed_identity for external data source reads, which resolves credentials from the node's ambient cloud identity (IAM instance profile / IMDS on AWS and Azure, GCE metadata server on GCP) rather than requiring explicit credentials in the query or datasource.

      Disabled by default. Must be explicitly enabled by an operator on self-hosted, single-cloud, single-tenant deployments where the node's ambient identity is the intended credential. Never enable in serverless or multi-tenant deployments: ambient credentials bypass tenant isolation.

      This is an operator-dynamic setting: changes take effect immediately without a node restart. When this key is not set, it falls back to the deprecated WORKLOAD_IDENTITY_ENABLED key's value, so reads through this setting see an operator's pre-rename configuration.

    • FEDERATED_IDENTITY_ENABLED

      public static final Setting<Boolean> FEDERATED_IDENTITY_ENABLED
      Gates provisioning data sources that use workload-identity federation (e.g. S3 role_arn, GCS sts_audience, Azure tenant_id/client_id). A single setting covers every file-based provider, since they all funnel through FileDataSourceValidator and share the DataSourceConfiguration#hasFederatedAuth() mechanism.

      Disabled by default. This is an operator-dynamic setting: changes take effect immediately without a node restart, and it can only be changed by an operator (e.g. via file-based settings), not by an end user through the public _cluster/settings API — so it can be enabled fleet-wide (or per-deployment) via gitops without exposing a customer-facing toggle.

    • LOCAL_ALLOWED_PATHS

      public static final Setting<List<String>> LOCAL_ALLOWED_PATHS
      Allowlist of local filesystem root paths from which ES|QL file:// external sources are permitted to read. Mirrors path.repo: the list is the enable — an empty list (the default) disables local-disk reads entirely. When non-empty, a file:// path is allowed only if it normalizes to a location under one of the listed roots; ..-escapes and anything outside every root are rejected.

      This is a node-scope setting; a node restart is required for changes to take effect.

  • Method Details

    • defaultBlobStoreConcurrency

      public static int defaultBlobStoreConcurrency(int allocatedProcessors)
      The default per-node concurrency for accessing an external blob store, derived from the node's allocated processors using the snapshot_meta thread pool's sizing shape (processors * 3), clamped to [16, 100]. This is the single source of truth for blob-store access concurrency so metadata discovery and data retrieval stay consistent: both are latency-bound I/O against object stores and should scale the same way with node size rather than each picking an ad-hoc constant. The floor keeps small nodes from self-throttling (and from sizing the esql_external_io pool too small to run the parse pipeline); the ceiling bounds a single store's load.

    • defaultBlobStoreConcurrency

      public static int defaultBlobStoreConcurrency(Settings settings)
      Convenience overload resolving allocated processors from the given settings.

    • blobStoreConcurrency

      public static int blobStoreConcurrency(Settings settings)
      The effective per-node blob-store access concurrency that every external access path reads, so one knob governs metadata discovery and data reads alike: the operator's MAX_CONCURRENT_REQUESTS value when set, otherwise the CPU-bound defaultBlobStoreConcurrency(Settings) default. The data-read path bounds in-flight reads with a per-scheme permit semaphore sized by this value (StorageProviderRegistry), and the metadata-discovery fan-out (TransportEsqlQueryAction.externalSourceConcurrency()) uses the same value — so an operator override reaches both paths.

    • externalIoThreads

      public static int externalIoThreads(Settings settings)
      Thread count for the dedicated esql_external_io pool (EsqlPlugin). Sized to exactly the single concurrency knob blobStoreConcurrency(Settings) — no headroom — because every blocking task that lands on this pool is a permit-gated reader (remote schemes) or is bounded by the pool itself (file://, which is not permit-wrapped), so the pool never needs more than N threads. The parse pipeline's page consumer runs on esql_worker, not here (see AsyncExternalSourceOperatorFactory), so a full pool of readers can never starve its own drain; that separation is what makes pool == permits safe rather than deadlock-prone. One exception to tracking the knob: max_concurrent_requests=0 disables the permit limiter (unbounded in-flight reads), but the I/O pool still needs threads to run the reads and parse pipeline, so it falls back to the CPU-scaled defaultBlobStoreConcurrency(Settings) default rather than a zero-thread pool. Always >= 1.

    • maxConcurrentSegmentators

      public static int maxConcurrentSegmentators(Settings settings)
      Effective cap on concurrent stream-only-compressed segmentators for a node, resolving MAX_CONCURRENT_SEGMENTATORS (0 = derive) and clamping the result to [1, poolSize - 1] so at least one esql_external_io thread always remains for the one-shot parser tasks that a segmentator depends on. The pool size is externalIoThreads(Settings), which sizes that pool; the derived default tracks it, so the cap is poolSize - 1 unless an operator sets a lower explicit value. An explicit value above poolSize - 1 is clamped and logged at WARN so the operator sees it was ineffective.

      Degenerate case: a poolSize == 1 pool (only reachable by explicitly setting esql.external.max_concurrent_requests=1) floors the cap at 1, which cannot leave a free parser thread — parallel streaming needs poolSize >= 2 to be deadlock-free. At that size the deadlock-free path is the single-pass fallback (parallelism <= 1), which runs no segmentator; a parallel streaming read on a one-thread pool is a misconfiguration this cap cannot rescue.

    • settings

      public static List<Setting<?>> settings()