- All Implemented Interfaces:
ClusterStateApplier,ReportingService<IngestInfo>
- Direct Known Subclasses:
SimulateIngestService
-
Nested Class Summary
Nested ClassesModifier and TypeClassDescriptionstatic classUsed by this class andReservedPipelineActionstatic classSpecialized cluster state update task specifically for ingest pipeline operations.static final recordstatic classUsed in this class and externally by theReservedPipelineActionNested classes/interfaces inherited from interface org.elasticsearch.node.ReportingService
ReportingService.Info -
Field Summary
Fields -
Constructor Summary
ConstructorsConstructorDescriptionIngestService(ClusterService clusterService, ThreadPool threadPool, Environment env, ScriptService scriptService, AnalysisRegistry analysisRegistry, List<IngestPlugin> ingestPlugins, Client client, MatcherWatchdog matcherWatchdog, FailureStoreMetrics failureStoreMetrics, ProjectResolver projectResolver, FeatureService featureService) -
Method Summary
Modifier and TypeMethodDescriptionvoidaddIngestClusterStateListener(Consumer<ClusterState> listener) Adds a listener that gets invoked with the current cluster state before processor factories get invoked.voidCalled when a new cluster state (ClusterChangedEvent.state()needs to be applied.static MatcherWatchdogcreateGrokThreadWatchdog(Environment env, ThreadPool threadPool) voiddelete(ProjectId projectId, DeletePipelineRequest request, ActionListener<AcknowledgedResponse> listener) Deletes the pipeline specified by id in the request.voidexecuteBulkRequest(ProjectId projectId, int numberOfActionRequests, Iterable<DocWriteRequest<?>> actionRequests, IntConsumer onDropped, Function<String, Boolean> resolveFailureStore, TriConsumer<Integer, String, Exception> onStoreFailure, TriConsumer<Integer, Exception, IndexDocFailureStoreStatus> onFailure, BiConsumer<Thread, Exception> onCompletion, Executor executor) Executes all applicable pipelines for a collection of documents.getPipeline(String id) This method is to be used exclusively by thePipelineProcessor.getPipeline(ProjectId projectId, String id) Returns the pipeline by the specified idstatic List<PipelineConfiguration> getPipelines(ProjectMetadata project, String... ids) <P extends Processor>
Collection<String> getPipelineWithProcessorType(ProjectId projectId, Class<P> clazz, Predicate<P> predicate) getProcessorsInPipeline(ProjectId projectId, String pipelineId, Class<P> clazz) Gets all the Processors of the given type from within a Pipeline.static booleanhasPipeline(IndexRequest indexRequest) Checks whether an IndexRequest has at least one pipeline defined.info()static booleanisNoOpPipelineUpdate(ProjectMetadata metadata, PutPipelineRequest request) voidputPipeline(ProjectId projectId, PutPipelineRequest request, ActionListener<AcknowledgedResponse> listener, Consumer<ActionListener<NodesInfoResponse>> nodeInfoListener) Stores the specified pipeline definition in the request.voidreloadPipeline(ProjectId projectId, String id) static IngestService.PipelinesresolvePipelines(DocWriteRequest<?> originalRequest, IndexRequest indexRequest, ProjectMetadata projectMetadata, long epochMillis) Resolve the default and final pipelines from the cluster state metadata or index templates.static voidresolvePipelinesAndUpdateIndexRequest(DocWriteRequest<?> originalRequest, IndexRequest indexRequest, ProjectMetadata projectMetadata) Resolves the potential pipelines (default and final) from the requests or templates associated to the index and then **mutates** theIndexRequestpassed object with the pipeline information.static voidsetPipelineOnRequest(IndexRequest indexRequest, IngestService.Pipelines resolvedPipelines) Set the request pipeline on the index request if present, otherwise set the default pipeline.stats()voidvalidatePipelineRequest(ProjectId projectId, PutPipelineRequest request, NodesInfoResponse nodeInfos)
-
Field Details
-
NOOP_PIPELINE_NAME
- See Also:
-
INGEST_ORIGIN
- See Also:
-
-
Constructor Details
-
IngestService
public IngestService(ClusterService clusterService, ThreadPool threadPool, Environment env, ScriptService scriptService, AnalysisRegistry analysisRegistry, List<IngestPlugin> ingestPlugins, Client client, MatcherWatchdog matcherWatchdog, FailureStoreMetrics failureStoreMetrics, ProjectResolver projectResolver, FeatureService featureService)
-
-
Method Details
-
createGrokThreadWatchdog
-
resolvePipelinesAndUpdateIndexRequest
public static void resolvePipelinesAndUpdateIndexRequest(DocWriteRequest<?> originalRequest, IndexRequest indexRequest, ProjectMetadata projectMetadata) Resolves the potential pipelines (default and final) from the requests or templates associated to the index and then **mutates** theIndexRequestpassed object with the pipeline information.Also, this method marks the request as `isPipelinesResolved = true`: Due to the request could be rerouted from a coordinating node to an ingest node, we have to be able to avoid double resolving the pipelines and also able to distinguish that either the pipeline comes as part of the request or resolved from this method. All this is made to later be able to reject the request in case the pipeline was set by a required pipeline **and** the request also has a pipeline request too.
- Parameters:
originalRequest- Original write request received.indexRequest- TheIndexRequestobject to update.projectMetadata- Project metadata from the cluster state from where the pipeline information is derived.
-
resolvePipelines
public static IngestService.Pipelines resolvePipelines(DocWriteRequest<?> originalRequest, IndexRequest indexRequest, ProjectMetadata projectMetadata, long epochMillis) Resolve the default and final pipelines from the cluster state metadata or index templates.- Parameters:
originalRequest- initial requestindexRequest- the index request, which could be different from the initial request if reroutedprojectMetadata- cluster data metadataepochMillis- current time for index name resolution- Returns:
- the resolved pipelines
-
setPipelineOnRequest
public static void setPipelineOnRequest(IndexRequest indexRequest, IngestService.Pipelines resolvedPipelines) Set the request pipeline on the index request if present, otherwise set the default pipeline. Always set the final pipeline.- Parameters:
indexRequest- the index requestresolvedPipelines- default and final pipelines resolved from metadata and templates
-
getClusterService
-
getScriptService
-
getProjectResolver
-
delete
public void delete(ProjectId projectId, DeletePipelineRequest request, ActionListener<AcknowledgedResponse> listener) Deletes the pipeline specified by id in the request. -
getPipelines
- Returns:
- pipeline configuration specified by id. If multiple ids or wildcards are specified multiple pipelines may be returned
-
putPipeline
public void putPipeline(ProjectId projectId, PutPipelineRequest request, ActionListener<AcknowledgedResponse> listener, Consumer<ActionListener<NodesInfoResponse>> nodeInfoListener) throws Exception Stores the specified pipeline definition in the request.- Throws:
Exception
-
validatePipelineRequest
public void validatePipelineRequest(ProjectId projectId, PutPipelineRequest request, NodesInfoResponse nodeInfos) throws Exception - Throws:
Exception
-
isNoOpPipelineUpdate
-
getPipeline
This method is to be used exclusively by thePipelineProcessor. It solely exists for the pipeline processor to be able to retrieve the pipeline that it needs to execute. Processors will exclusively be executed in a context where the project id is set in the thread context, which allows us to use the project resolver here. -
getPipeline
Returns the pipeline by the specified id -
getProcessorFactories
-
info
- Specified by:
infoin interfaceReportingService<IngestInfo>
-
executeBulkRequest
public void executeBulkRequest(ProjectId projectId, int numberOfActionRequests, Iterable<DocWriteRequest<?>> actionRequests, IntConsumer onDropped, Function<String, Boolean> resolveFailureStore, TriConsumer<Integer, String, Exception> onStoreFailure, TriConsumer<Integer, Exception, IndexDocFailureStoreStatus> onFailure, BiConsumer<Thread, Exception> onCompletion, Executor executor) Executes all applicable pipelines for a collection of documents.- Parameters:
projectId- The ID of the project we are indexing into.numberOfActionRequests- The total number of requests to process.actionRequests- The collection of requests to be processed.onDropped- A callback executed when a document is dropped by a pipeline. Accepts the slot in the collection of requests that the document occupies.resolveFailureStore- A function executed on each ingest failure to determine if the failure should be stored somewhere.onStoreFailure- A callback executed when a document fails ingest but the failure should be persisted elsewhere. Accepts the slot in the collection of requests that the document occupies, the index name that the request was targeting at the time of failure, and the exception that the document encountered.onFailure- A callback executed when a document fails ingestion and does not need to be persisted. Accepts the slot in the collection of requests that the document occupies, and the exception that the document encountered.onCompletion- A callback executed once all documents have been processed. Accepts the thread that ingestion completed on or an exception in the event that the entire operation has failed.executor- Which executor the bulk request should be executed on.
-
stats
-
addIngestClusterStateListener
Adds a listener that gets invoked with the current cluster state before processor factories get invoked.This is useful for components that are used by ingest processors, so that they have the opportunity to update before these components get used by the ingest processor factory.
-
applyClusterState
Description copied from interface:ClusterStateApplierCalled when a new cluster state (ClusterChangedEvent.state()needs to be applied. The cluster state to be applied is already committed when this method is called, so an applier must therefore be prepared to deal with any state it receives without throwing an exception. Throwing an exception from an applier is very bad because it will stop the application of this state before it has reached all the other appliers, and will likely result in another attempt to apply the same (or very similar) cluster state which might continue until this node is removed from the cluster.Cluster states are applied one-by-one which means they can be a performance bottleneck. Implementations of this method should therefore be fast, so please consider forking work into the background rather than doing everything inline.
- Specified by:
applyClusterStatein interfaceClusterStateApplier
-
getProcessorsInPipeline
public <P extends Processor> List<P> getProcessorsInPipeline(ProjectId projectId, String pipelineId, Class<P> clazz) Gets all the Processors of the given type from within a Pipeline.- Parameters:
projectId-pipelineId- the pipeline to inspectclazz- the Processor class to look for- Returns:
- True if the pipeline contains an instance of the Processor class passed in
-
getPipelineWithProcessorType
-
reloadPipeline
- Throws:
Exception
-
hasPipeline
Checks whether an IndexRequest has at least one pipeline defined.This method assumes that the pipelines are beforehand resolved.
-