Package org.elasticsearch.xpack.esql.datasources.spi

Class FileDataSourceValidator

java.lang.Object
org.elasticsearch.xpack.esql.datasources.spi.FileDataSourceValidator
All Implemented Interfaces:
DataSourceValidator
public class FileDataSourceValidator extends Object implements DataSourceValidator
DataSourceValidator for file-based external sources (S3, GCS, Azure).

Plugins pass their supportedSchemes() (e.g. Set.of("s3", "s3a", "s3n")) directly to the constructor — the validator appends "://" internally when matching resource URIs, so plugins do not need to duplicate the scheme list with URI suffixes.

The format dataset setting selects a file format explicitly (any registered format name, or FormatNameResolver.FORMAT_AUTO / absent to infer from the resource extension). Format-specific dataset fields (e.g. CSV's delimiter, Parquet's optimized_reader) are validated against the resolved format: explicit format → resource extension → unknown. Both require a FileDataSourceValidator.FormatConfigKeyResolver set via withFormatConfigKeyResolver(org.elasticsearch.xpack.esql.datasources.spi.FileDataSourceValidator.FormatConfigKeyResolver, java.util.Set<java.lang.String>). Without a resolver the validator cannot know any format names, so format and all format-specific fields are rejected and only the base dataset fields are accepted, preserving backward compatibility.

  • Field Details

    • FEDERATED_IDENTITY_DISABLED_MESSAGE

      public static final String FEDERATED_IDENTITY_DISABLED_MESSAGE
      Error shown when a data source is provisioned with federated authentication settings while the esql.datasource.federated_identity.enabled cluster setting is disabled. The federated fields themselves remain registered on each configuration regardless of the setting, so a PUT carrying them produces this explicit message rather than an "unknown setting" error.
      See Also:

    • COORDINATOR_DATASET_KEYS

      public static final Set<String> COORDINATOR_DATASET_KEYS
      Coordinator-level data-shape keys accepted on a dataset, sourced from each owning component's own CONFIG_KEYS plus the format selector. This is exactly FileSourceFactory.COORDINATOR_KEYS minus the EXTERNAL-only knob (reader) and the internal _datasource envelope, a relationship pinned by FileSourceFactoryValidationTests. Keeping it sourced from the components' constants holds the dataset vocabulary in lockstep with the query path, so a new coordinator key cannot silently regress to EXTERNAL-only.

  • Constructor Details

  • Method Details

    • withFormatConfigKeyResolver

      public FileDataSourceValidator withFormatConfigKeyResolver(FileDataSourceValidator.FormatConfigKeyResolver resolver, Set<String> compressionExtensions)
      Returns a new validator that resolves a dataset's file format (from an explicit format setting or the resource extension) and validates format-specific fields against it. The resolver maps a format name to its config keys, an extension to its format name, and enumerates the known format names for error messages.

      The compressionExtensions set restricts compound-extension fallback (e.g. data.csv.gz) to only known compression suffixes, mirroring the runtime resolution in FormatReaderRegistry/DecompressionCodecRegistry.

    • withManagedIdentityEnabled

      public FileDataSourceValidator withManagedIdentityEnabled(BooleanSupplier supplier)
      Returns a new validator that gates auth=managed_identity on the supplied boolean supplier. The supplier is called on each validation. Pass a live supplier (e.g. backed by an AtomicBoolean updated via ClusterSettings.addSettingsUpdateConsumer) so that operator changes to esql.datasource.managed_identity.enabled take effect without a node restart.

    • withFederatedIdentityEnabled

      public FileDataSourceValidator withFederatedIdentityEnabled(BooleanSupplier supplier)
      Returns a new validator that gates federated workload-identity authentication on the supplied boolean supplier. The supplier is called on each validation. Wire it to ExternalSourceSettings#FEDERATED_IDENTITY_ENABLED in production; tests pass a fixed supplier to exercise both states without depending on a live cluster setting.

    • type

      public String type()
      Description copied from interface: DataSourceValidator
      Type identifier, e.g. "s3", "gcs", "azure".
      Specified by:
      type in interface DataSourceValidator

    • validateDatasource

      public Map<String,DataSourceSetting> validateDatasource(Map<String,Object> datasourceSettings)
      Description copied from interface: DataSourceValidator
      Validates settings and wraps them in DataSourceSetting so secret classification travels into cluster state. Throws ValidationException if invalid.
      Specified by:
      validateDatasource in interface DataSourceValidator

    • validateDatasource

      public Map<String,DataSourceSetting> validateDatasource(Map<String,Object> datasourceSettings, Set<String> existingSecretKeys)
      Description copied from interface: DataSourceValidator
      Merge-aware variant used when a PUT replaces an existing data source: existingSecretKeys names secret fields already stored for it, so a request that omits one still satisfies credential-completeness checks. The returned map, like DataSourceValidator.validateDatasource(Map), contains only the fields present in datasourceSettings; carrying the omitted secret's stored value forward is the caller's job. The default ignores existingSecretKeys and delegates to the single-arg overload.
      Specified by:
      validateDatasource in interface DataSourceValidator

    • validateDataset

      public Map<String,Object> validateDataset(Map<String,DataSourceSetting> datasourceSettings, String resource, Map<String,Object> datasetSettings)
      Description copied from interface: DataSourceValidator
      Validates dataset settings. Returns plain values — datasets carry no secrets. Parent passed for cross-checks. Throws ValidationException if invalid.
      Specified by:
      validateDataset in interface DataSourceValidator

    • unknownFormatError

      public static String unknownFormatError(String format, Set<String> knownFormats)
      The validation error reported when an explicit format value is not a registered format. Exposed so tests assert the exact message against the same literal production emits, rather than a brittle substring. knownFormats is sorted for a deterministic message.

    • cannotDetermineFormatError

      public static String cannotDetermineFormatError(String resource, Set<String> formatSpecificKeys)
      The validation error reported when a format-specific setting is present but the resource's format cannot be determined (no usable format value and an unknown or absent extension). Exposed so tests assert the exact message against the same literal production emits. formatSpecificKeys is sorted for a deterministic message.