Package org.elasticsearch.xpack.esql.datasources.spi

Record Class ErrorPolicy

java.lang.Object
java.lang.Record
org.elasticsearch.xpack.esql.datasources.spi.ErrorPolicy
Record Components:
mode - how to handle rows with parse errors
maxErrors - maximum number of errors to tolerate before failing (only meaningful for SKIP_ROW and NULL_FIELD)
maxErrorRatio - maximum fraction of rows (0.0–1.0) that may be errors before failing; 0.0 disables ratio-based tolerance (only meaningful for SKIP_ROW and NULL_FIELD)
logErrors - whether to log each skipped/null-filled row at WARN level
public record ErrorPolicy(ErrorPolicy.Mode mode, long maxErrors, double maxErrorRatio, boolean logErrors) extends Record
Configures how format readers handle malformed or unparseable rows.

Error modes

Error mode comparison across engines
ES/ESQLSparkDuckDBClickHouseBehaviour
FAIL_FASTFAILFAST(default) errors_num=0Abort on first error
SKIP_ROWDROPMALFORMEDignore_errors errors_num>0Drop the entire bad row
NULL_FIELDPERMISSIVE— —Null-fill unparseable fields, keep the row

Error budget

maxErrors and maxErrorRatio only apply to ErrorPolicy.Mode.SKIP_ROW and ErrorPolicy.Mode.NULL_FIELD. They are ignored in ErrorPolicy.Mode.FAIL_FAST (which always aborts on the first error). When both are set, whichever limit is hit first triggers failure.
Error budget comparison across engines
ES/ESQLDuckDBClickHouse
max_errorsrejects_limit input_format_allow_errors_num
max_error_ratio— input_format_allow_errors_ratio

Usage

  FROM s3://bucket/data.csv WITH {"max_errors": 100}
  FROM s3://bucket/data.csv WITH {"error_mode": "skip_row", "max_error_ratio": 0.1}
  FROM s3://bucket/data.csv WITH {"error_mode": "null_field"}

Client-visible warnings

Whenever the non-strict modes (ErrorPolicy.Mode.SKIP_ROW and ErrorPolicy.Mode.NULL_FIELD) cause a row to be dropped or a field to be null-filled, format readers emit response Warning headers via SkipWarnings: a one-time summary identifying the file, plus per-event details capped at SkipWarnings.MAX_ADDED_WARNINGS entries (further events collapse into a single "further warnings suppressed" line). This is independent of logErrors, which still only controls server-side WARN logging.

  • Field Details

  • Constructor Details

    • ErrorPolicy

      public ErrorPolicy(ErrorPolicy.Mode mode, long maxErrors, double maxErrorRatio, boolean logErrors)
      Creates an instance of a ErrorPolicy record class.
      Parameters:
      mode - the value for the mode record component
      maxErrors - the value for the maxErrors record component
      maxErrorRatio - the value for the maxErrorRatio record component
      logErrors - the value for the logErrors record component

    • ErrorPolicy

      public ErrorPolicy(long maxErrors, boolean logErrors)
      Convenience constructor for ErrorPolicy.Mode.SKIP_ROW without ratio-based tolerance.

    • ErrorPolicy

      public ErrorPolicy(long maxErrors, double maxErrorRatio, boolean logErrors)
      Convenience constructor for ErrorPolicy.Mode.SKIP_ROW with ratio-based tolerance.

  • Method Details

    • isStrict

      public boolean isStrict()

    • isPermissive

      public boolean isPermissive()

    • modeName

      public String modeName()
      Lowercase, locale-insensitive name of mode(), suitable for user-facing messages (e.g. "skip_row", "null_field"). Matches the parser input accepted by ErrorPolicy.Mode.parse(String).

    • isBudgetExceeded

      public boolean isBudgetExceeded(long errorsSoFar, long rowsSoFar)
      Checks whether the error budget has been exceeded given the current counts. Always returns false for ErrorPolicy.Mode.FAIL_FAST (which never reaches this check — it throws immediately in the caller).
      Parameters:
      errorsSoFar - total errors encountered so far
      rowsSoFar - total rows processed so far (including errors)
      Returns:
      true if the budget is exceeded and processing should stop

    • fromConfig

      public static ErrorPolicy fromConfig(Map<String,Object> config, ErrorPolicy defaultPolicy)
      Resolves an ErrorPolicy from the user's WITH options. Returns defaultPolicy when none of CONFIG_ERROR_MODE, CONFIG_MAX_ERRORS, or CONFIG_MAX_ERROR_RATIO are set.

      Validation matches what FileSourceFactory applied historically: invalid mode strings, non-numeric budgets, and FAIL_FAST combined with budget keys are all rejected with IllegalArgumentException.

    • toString

      public final String toString()
      Returns a string representation of this record class. The representation contains the name of the class, followed by the name and value of each of the record components.
      Specified by:
      toString in class Record
      Returns:
      a string representation of this object

    • hashCode

      public final int hashCode()
      Returns a hash code value for this object. The value is derived from the hash code of each of the record components.
      Specified by:
      hashCode in class Record
      Returns:
      a hash code value for this object

    • equals

      public final boolean equals(Object o)
      Indicates whether some other object is "equal to" this one. The objects are equal if the other object is of the same class and if all the record components are equal. Reference components are compared with Objects::equals(Object,Object); primitive components are compared with '=='.
      Specified by:
      equals in class Record
      Parameters:
      o - the object with which to compare
      Returns:
      true if this object is the same as the o argument; false otherwise.

    • mode

      public ErrorPolicy.Mode mode()
      Returns the value of the mode record component.
      Returns:
      the value of the mode record component

    • maxErrors

      public long maxErrors()
      Returns the value of the maxErrors record component.
      Returns:
      the value of the maxErrors record component

    • maxErrorRatio

      public double maxErrorRatio()
      Returns the value of the maxErrorRatio record component.
      Returns:
      the value of the maxErrorRatio record component

    • logErrors

      public boolean logErrors()
      Returns the value of the logErrors record component.
      Returns:
      the value of the logErrors record component