Various Elasticsearch features such as Security or Watcher keep their state in their own indices. We keep these indices separate from the user index space for a few reasons. In some cases, the indices contain information that should be hidden from users. But, more generally, we want to protect these indices and data streams from being inadvertently modified or deleted.
The system resources are grouped by feature, using the SystemIndices.Feature class. Most features will be loaded from
instances of SystemIndexPlugin; any other features will be described in this class. Features may be retrieved by name or
iterated over (see getFeature(String) and getFeatures()). Each Feature provides collections of
SystemIndexDescriptors or SystemDataStreamDescriptors. These descriptors define their resources by means of patterns.
Any index name or data stream name that matches one of these patterns is considered a system resource for * that descriptor, and user
access to it will be restricted. These patterns are gathered and validated so that the SystemIndices object can provide information
about system resources: for example, whether a particular string will be considered a “system name” (see isSystemIndex(String)).
For more information about the expected behavior of system indices, see SystemIndexDescriptor. For more information about
the expected behavior of system data streams, see SystemDataStreamDescriptor.
The SystemIndices object is constructed during Node startup, and is not modified after construction.
In other words, the set of system resources will be consistent over the lifetime of a node.
System resources will specify thread pools for reads, writes, and searches. This can ensure that system-critical operations, such
as user authentication, are not blocked by heavy thread contention from user activities. getExecutorSelector() provides an
object for convenient look-ups of these thread pools.
There are a few requirements for system features, collectively:
- The feature names must be distinct.
- System index patterns must not overlap.
- Aliases for system indices must be distinct.
- Feature names must not be reserved names. Right now, the only reserved name is “none”.
System index access is currently controlled by Security role index permissions. However, we have deprecated general rest access to
system indices. This class provides checks for system index “access levels” (see getSystemIndexAccessLevel(ThreadContext)).
If a request has the wrong access level for a system index it is targeting, then we will issue a deprecation warning. In the future,
we will block access. The non-deprecated way to access certain external system indices is to use the correct request headers. This
behavior is already in place in SystemDataStreamDescriptor and “net-new” system indices (see
SystemIndexDescriptor.isNetNew()).
The implementation of the system index name checks makes heavy use of the Lucene Automaton class. At a high level, an
automaton is a kind of matcher that can be created from a regex. Lucene Automata give us the ability to check for overlapping
patterns, and to create efficient unions of patterns.
-
Nested Class Summary
Nested ClassesModifier and TypeClassDescriptionstatic classDescribes an Elasticsearch system feature that keeps state in protected indices and data streams.static enumIn a future release, these access levels will be used to allow or deny requests for system resources. -
Field Summary
FieldsModifier and TypeFieldDescriptionstatic final Stringstatic final Stringstatic final IndexVersionstatic final VersionThese versions should be set to current major and current major's index versionstatic final Map<String, SystemIndexDescriptor.MappingsVersion> static final Stringstatic final String -
Constructor Summary
ConstructorsConstructorDescriptionSystemIndices(List<SystemIndices.Feature> pluginAndModuleFeatures) Initialize the SystemIndices object -
Method Summary
Modifier and TypeMethodDescriptionstatic IllegalArgumentExceptiondataStreamAccessException(ThreadContext threadContext, Collection<String> names) Finds a single matchingSystemDataStreamDescriptor, if any, for the given DataStream name.findMatchingDescriptor(String name) Finds a single matchingSystemIndexDescriptor, if any, for the given index name.Used to determine which executor should be used for operations on this index.getFeature(String name) Get a feature by name.Get a set of feature names.Get a collection of the Features this SystemIndices object is managing.getProductSystemIndexNamePredicate(ThreadContext threadContext) Builds a predicate that tests whether a system index should be accessible for a given ThreadContext.getSystemIndexAccessLevel(ThreadContext threadContext) Determines what level of system index access should be allowed in the current context.org.apache.lucene.util.automaton.AutomatonThe Elasticsearch security plugin can use the automaton that matches all system resource names to efficiently authorize requests.booleanisNetNewSystemIndex(String indexName) Checks whether an index is a net-new system index, meaning we can apply non-BWC behavior to it.booleanisSystemDataStream(String name) Determines whether the provided name matches that of a system data stream that has been defined by aSystemDataStreamDescriptorbooleanisSystemIndex(String indexName) Determines whether a given index is a system index by comparing its name to the collection of loadedSystemIndexDescriptors.booleanisSystemIndex(Index index) Determines whether a given index is a system index by comparing its name to the collection of loadedSystemIndexDescriptorsbooleanDetermines whether the provided name matches that of an index that backs a system data stream.booleanisSystemName(String name) Checks whether the given name matches a reserved name or pattern that is intended for use by a system component.static IllegalArgumentExceptionnetNewSystemIndexAccessException(ThreadContext threadContext, Collection<String> names) validateDataStreamAccess(String dataStreamName, ThreadContext threadContext) static voidvalidateFeatureName(String name, String plugin) Check that a feature name is not reserved
-
Field Details
-
SYSTEM_INDEX_ACCESS_CONTROL_HEADER_KEY
- See Also:
-
EXTERNAL_SYSTEM_INDEX_ACCESS_CONTROL_HEADER_KEY
- See Also:
-
NO_UPGRADE_REQUIRED_VERSION
These versions should be set to current major and current major's index version -
NO_UPGRADE_REQUIRED_INDEX_VERSION
-
MIGRATE_SYSTEM_INDEX_CAUSE
- See Also:
-
UPGRADED_INDEX_SUFFIX
-
SERVER_SYSTEM_MAPPINGS_VERSIONS
public static final Map<String,SystemIndexDescriptor.MappingsVersion> SERVER_SYSTEM_MAPPINGS_VERSIONS
-
-
Constructor Details
-
SystemIndices
Initialize the SystemIndices object- Parameters:
pluginAndModuleFeatures- A list of features from which we will load system indices. These features come from plugins and modules. Non-plugin system features such as Tasks will be added automatically.
-
-
Method Details
-
isSystemName
Checks whether the given name matches a reserved name or pattern that is intended for use by a system component. The name is checked against index names, aliases, data stream names, and the names of indices that back a system data stream. -
isSystemIndex
Determines whether a given index is a system index by comparing its name to the collection of loadedSystemIndexDescriptors- Parameters:
index- theIndexobject to check against loadedSystemIndexDescriptors- Returns:
- true if the
Index's name matches a pattern from aSystemIndexDescriptor
-
isSystemIndex
Determines whether a given index is a system index by comparing its name to the collection of loadedSystemIndexDescriptors. This will also match alias names that belong to system indices.- Parameters:
indexName- the index name to check against loadedSystemIndexDescriptors- Returns:
- true if the index name matches a pattern from a
SystemIndexDescriptor
-
isSystemDataStream
Determines whether the provided name matches that of a system data stream that has been defined by aSystemDataStreamDescriptor -
isSystemIndexBackingDataStream
Determines whether the provided name matches that of an index that backs a system data stream. Backing indices for system data streams are marked as "system" in their metadata (seeSystemIndexMetadataUpgradeService) and receive the same protections as the system data stream. -
getSystemNameAutomaton
public org.apache.lucene.util.automaton.Automaton getSystemNameAutomaton()The Elasticsearch security plugin can use the automaton that matches all system resource names to efficiently authorize requests.- Returns:
- An
Automatonthat tests whether strings are names of system indices, aliases, or data streams.
-
isNetNewSystemIndex
Checks whether an index is a net-new system index, meaning we can apply non-BWC behavior to it. SeeSystemIndexDescriptor.isNetNew().- Parameters:
indexName- The index name to check.- Returns:
trueif the given index is covered by a net-new system index descriptor,falseotherwise.
-
getExecutorSelector
Used to determine which executor should be used for operations on this index. SeeExecutorSelectordocs for details. -
findMatchingDescriptor
Finds a single matchingSystemIndexDescriptor, if any, for the given index name. Does not take into account system data streams and their backing indices.- Parameters:
name- the name of the index- Returns:
- The matching
SystemIndexDescriptorornullif no descriptor is found
-
findMatchingDataStreamDescriptor
Finds a single matchingSystemDataStreamDescriptor, if any, for the given DataStream name.- Parameters:
name- the name of the DataStream- Returns:
- The matching
SystemDataStreamDescriptorornullif no descriptor is found
-
getProductSystemIndexNamePredicate
Builds a predicate that tests whether a system index should be accessible for a given ThreadContext. We allow guaranteed (that is, non-deprecated) external access to system indices based on special request headers in addition to security roles. If those headers are present and provide a product name with access to the index, they will be added to the thread context and checked here. Without these headers, we will add deprecation warnings to the response. In future versions, we will deny access altogether.- Parameters:
threadContext- the threadContext containing headers used for system index access- Returns:
- Predicate to check external system index names with
-
getFeatureNames
Get a set of feature names. This is useful for checking whether particular features are present on the node.- Returns:
- A set of all feature names
-
getFeature
Get a feature by name.- Parameters:
name- Name of a feature.- Returns:
- The corresponding feature if it exists on this node, null otherwise.
-
getFeatures
Get a collection of the Features this SystemIndices object is managing.- Returns:
- A collection of Features.
-
validateDataStreamAccess
public SystemDataStreamDescriptor validateDataStreamAccess(String dataStreamName, ThreadContext threadContext) -
dataStreamAccessException
public static IllegalArgumentException dataStreamAccessException(ThreadContext threadContext, Collection<String> names) -
netNewSystemIndexAccessException
public static IllegalArgumentException netNewSystemIndexAccessException(ThreadContext threadContext, Collection<String> names) -
getSystemIndexAccessLevel
public static SystemIndices.SystemIndexAccessLevel getSystemIndexAccessLevel(ThreadContext threadContext) Determines what level of system index access should be allowed in the current context. For system data streams and "net-new" system indices (seeSystemIndexDescriptor.isNetNew()), access levels should be used to reject requests entirely. For non-net-new, backwards-compatible system indices, these access levels should be used for deprecation warnings.- Parameters:
threadContext- the current thread context that has headers associated with the current request- Returns:
SystemIndices.SystemIndexAccessLevel.ALLif unrestricted system index access should be allowed,SystemIndices.SystemIndexAccessLevel.RESTRICTEDif a subset of system index access should be allowed, orSystemIndices.SystemIndexAccessLevel.NONEif no system index access should be allowed.
-
getSystemIndexDescriptors
-
getMappingsVersions
-
validateFeatureName
Check that a feature name is not reserved- Parameters:
name- Name of featureplugin- Name of plugin providing the feature
-