- All Implemented Interfaces:
Closeable,AutoCloseable
-
Nested Class Summary
Nested ClassesModifier and TypeClassDescriptionprotected static final classprotected static final classNested classes/interfaces inherited from class org.elasticsearch.index.engine.Engine
Engine.Delete, Engine.DeleteResult, Engine.EventListener, Engine.FlushResult, Engine.Get, Engine.GetResult, Engine.Index, Engine.IndexCommitListener, Engine.IndexCommitRef, Engine.IndexResult, Engine.IndexThrottle, Engine.NoOp, Engine.NoOpLock, Engine.NoOpResult, Engine.Operation, Engine.RefreshResult, Engine.Result, Engine.Searcher, Engine.SearcherScope, Engine.SearcherSupplier, Engine.TranslogRecoveryRunner, Engine.Warmer -
Field Summary
FieldsModifier and TypeFieldDescriptionprotected static final Stringfinal AtomicLongfinal AtomicLongprotected static final StringFields inherited from class org.elasticsearch.index.engine.Engine
CAN_MATCH_SEARCH_SOURCE, DOC_STATS_SOURCE, enableRecoverySource, engineConfig, ES_VERSION, eventListener, failedEngine, failEngineLock, FORCE_MERGE_UUID_KEY, HISTORY_UUID_KEY, isClosed, lastWriteNanos, logger, MAX_UNSAFE_AUTO_ID_TIMESTAMP_COMMIT_ID, MIN_RETAINED_SEQNO, pauseIndexingOnThrottle, ROOT_DOC_FIELD_NAME, SEARCH_SOURCE, SEGMENTS_STATS_SOURCE, shardId, store, UNKNOWN_PRIMARY_TERM -
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionAcquires a lock on Lucene soft-deleted documents to prevent them from being trimmedacquireLastIndexCommit(boolean flushFirst) Snapshots the most recent index and returns a handle to it.Snapshots the most recent safe index commit from the engine.voidRequest that this engine throttle incoming indexing requests to one thread.voidaddFlushListener(Translog.Location location, ActionListener<Long> listener) Add a listener for the translog location.protected voidadvanceMaxSeqNoOfDeletesOnPrimary(long seqNo) protected voidadvanceMaxSeqNoOfUpdatesOnPrimary(long seqNo) voidadvanceMaxSeqNoOfUpdatesOrDeletes(long maxSeqNoOfUpdatesOnPrimary) A replica shard receives a new max_seq_no_of_updates from its primary shard, then calls this method to advance this marker to at least the given sequence number.protected voidafterFlush(long generation) protected booleanassertNonPrimaryOrigin(Engine.Operation operation) protected booleanprotected booleanassertPrimaryIncomingSequenceNumber(Engine.Operation.Origin origin, long seqNo) voidasyncEnsureGlobalCheckpointSynced(long globalCheckpoint, Consumer<Exception> listener) Ensures that the global checkpoint has been persisted to the underlying storage.voidasyncEnsureTranslogSynced(Translog.Location location, Consumer<Exception> listener) Ensures that the location has been written to the underlying storage.protected final voidcloseNoLock(String reason, CountDownLatch closedLatch) Closes the engine without acquiring any refs or locks.protected voidcommitIndexWriter(org.apache.lucene.index.IndexWriter writer, Translog translog) Commits the specified index writer.completionStats(String... fieldNamePatterns) Returns theCompletionStatsfor this engineintcountChanges(String source, long fromSeqNo, long toSeqNo) Counts the number of operations in the range of the given sequence numbers.protected ElasticsearchReaderManagercreateInternalReaderManager(ElasticsearchDirectoryReader directoryReader) protected LiveVersionMapArchiveprotected ElasticsearchMergeSchedulercreateMergeScheduler(ShardId shardId, IndexSettings indexSettings, ThreadPoolMergeExecutorService threadPoolMergeExecutorService, MergeMetrics mergeMetrics) protected org.apache.lucene.index.IndexWritercreateWriter(org.apache.lucene.store.Directory directory, org.apache.lucene.index.IndexWriterConfig iwc) voidReverses a previousEngine.activateThrottling()call.delete(Engine.Delete delete) Perform document delete operation on the engineprotected voiddeleteByQuery(ShardSplittingQuery query) protected InternalEngine.DeletionStrategyprotected longestimateMergeBytes(org.apache.lucene.index.MergePolicy.OneMerge merge) intfillSeqNoGaps(long primaryTerm) Fills up the local checkpoints history with no-ops until the local checkpoint and the max seen sequence ID are identical.protected voidflushHoldingLock(boolean force, boolean waitIfOngoing, ActionListener<Engine.FlushResult> listener) The actual implementation ofEngine.flush(boolean, boolean, ActionListener), to be called either when holding a ref that ensures the engine remains open, or holdingIndexShard#engineMutexwhile closing the engine.voidforceMerge(boolean flush, int maxNumSegments, boolean onlyExpungeDeletes, String forceMergeUUID) Triggers a forced merge on this engineprotected longget(Engine.Get get, MappingLookup mappingLookup, DocumentParser documentParser, Function<Engine.Searcher, Engine.Searcher> searcherWrapper) getCommitExtraUserData(long localCheckpoint) Allows InternalEngine extenders to return custom key-value pairs which will be included in the Lucene commit user-data.returns the force merge uuid for the enginegetFromTranslog(Engine.Get get, MappingLookup mappingLookup, DocumentParser documentParser, Function<Engine.Searcher, Engine.Searcher> searcherWrapper) Similar toEngine.get(org.elasticsearch.index.engine.Engine.Get, org.elasticsearch.index.mapper.MappingLookup, org.elasticsearch.index.mapper.DocumentParser, java.util.function.Function<org.elasticsearch.index.engine.Engine.Searcher, org.elasticsearch.index.engine.Engine.Searcher>), but it only attempts to serve the get from the translog.returns the history uuid for the enginelongHow much heap is used that would be freed by a refresh.longReturns the number of milliseconds this engine was under index throttling.org.apache.lucene.index.SegmentInfoslongReturns the latest global checkpoint value that has been persisted in the underlying storage (i.e.longprotected LiveVersionMapArchiveprotected LocalCheckpointTrackerfinal longReturns the maximum auto_id_timestamp of all append-only index requests have been processed by this engine or the auto_id_timestamp received from its primary shard viaEngine.updateMaxUnsafeAutoIdTimestamp(long).longlongReturns the maximum sequence number of either update or delete operations have been processed in this engine or the sequence number fromEngine.advanceMaxSeqNoOfUpdatesOrDeletes(long).final longReturns the minimum seqno that is retained in the Lucene index.longprotected longlonggetRawFieldRange(String field) protected final org.apache.lucene.search.ReferenceManager<ElasticsearchDirectoryReader> getSeqNoStats(long globalCheckpoint) longReturns the total time flushes have been executed excluding waiting on locks.Returns the last location that the translog of this engine has written into.longReturns how many bytes we are currently moving from indexing buffer to segments on diskprotected final booleanChecks if the given operation has been processed in this engine or not.booleanhasCompleteOperationHistory(String reason, long startingSeqNo) Checks if this engine has every operations sincestartingSeqNo(inclusive) in its history (either Lucene or translog)protected booleanindex(Engine.Index index) Perform document index operation on the engineprotected InternalEngine.IndexingStrategybooleanisDocumentInLiveVersionMap(org.apache.lucene.util.BytesRef uid) Whether the document is in the live version map or not.protected final booleanbooleanReturns thetrueiff this engine is currently under index throttling.booleanChecks if the underlying storage sync is required.protected final voidmarkSeqNoAsSeen(long seqNo) Marks the given seq_no as seen and advances the max_seq_no of this engine to at least that value.protected booleanmaybeFailEngine(String source, Exception e) Check whether the engine should be failedvoidTries to prune buffered deletes from the version map.voidmaybeRefresh(String source, ActionListener<Engine.RefreshResult> listener) Asynchronously refreshes the engine for new search operations to reflect the latest changes unless another thread is already refreshing or reseting the engine concurrently.protected voidmergeException(Throwable exc) newChangesSnapshot(String source, long fromSeqNo, long toSeqNo, boolean requiredFullRange, boolean singleConsumer, boolean accessStats, long maxChunkSize) Creates a new history snapshot from Lucene for reading operations whose seqno in the requesting seqno range (both inclusive).protected ElasticsearchIndexDeletionPolicynewIndexDeletionPolicy(EngineConfig engineConfig, org.apache.logging.log4j.Logger logger, Translog translog, org.elasticsearch.index.engine.SoftDeletesPolicy softDeletesPolicy) protected Translog.SnapshotnewTranslogSnapshot(long fromSeqNo, long toSeqNo) noOp(Engine.NoOp noOp) voidprotected final InternalEngine.DeletionStrategyprotected final InternalEngine.IndexingStrategyExecutes a pre-flight check for a given NoOp.protected Engine.GetResultrealtimeGetUnderLock(Engine.Get get, MappingLookup mappingLookup, DocumentParser documentParser, Function<Engine.Searcher, Engine.Searcher> searcherWrapper, boolean getFromSearcher) protected voidvoidrecoverFromTranslog(Engine.TranslogRecoveryRunner translogRecoveryRunner, long recoverUpToSeqNo, ActionListener<Void> listener) Performs recovery from the transaction log up torecoverUpToSeqNo(inclusive).Synchronously refreshes the engine for new search operations to reflect the latest changes.protected final Engine.RefreshResultrefresh(String source, Engine.SearcherScope scope, boolean block) protected final voidrefreshIfNeeded(String source, long requestingSeqNo) Refresh this engine **internally** iff the requesting seq_no is greater than the last refreshed checkpoint.protected Engine.RefreshResultrefreshInternalSearcher(String source, boolean block) intrestoreLocalHistoryFromTranslog(Engine.TranslogRecoveryRunner translogRecoveryRunner) This method replays translog to restore the Lucene index which might be reverted previously.voidReverses a previousEngine.suspendThrottling()call.voidRolls the translog generation and cleans unneeded.segments()The list of segments in the engine.segments(boolean includeVectorFormatsInfo) booleanChecks if this engine should be flushed periodically.booleanTests whether or not the translog generation should be rolled to a new generation.voidDo not replay translog operations, but make the engine be ready.voidIf indexing is throttled to the point where it is paused completely, another task trying to get indexing permits might want to pause throttling by letting one thread pass at a time so that it does not get starved.voidprotected LongConsumervoidtrimOperationsFromTranslog(long belowTerm, long aboveSeqNo) Trims translog for terms belowbelowTermand seq# aboveaboveSeqNovoidchecks and removes translog files that no longer need to be retained.final voidupdateMaxUnsafeAutoIdTimestamp(long newTimestamp) Forces this engine to advance its max_unsafe_auto_id_timestamp marker to at least the given timestamp.protected voidwaitForCommitDurability(long generation, ActionListener<Void> listener) This method is meant to be overridden by subclasses that require special logic to ensure commit durability.voidCalled when our engine is using too much heap and should move buffered indexed/deleted documents to disk.protected final voidwriterSegmentStats(SegmentsStats stats) Methods inherited from class org.elasticsearch.index.engine.Engine
acquireEnsureOpenRef, acquireIndexCommitForSnapshot, acquireSearcher, acquireSearcher, acquireSearcher, acquireSearcherSupplier, acquireSearcherSupplier, addPrimaryTermAndGenerationListener, addSegmentGenerationListener, allowSearchIdleOptimization, awaitPendingClose, buildSeqNoStats, close, commitStats, config, denseVectorStats, denseVectorStats, docsStats, docStats, ensureOpen, ensureOpen, externalRefresh, failEngine, fillSegmentStats, flush, flush, flush, flushAndClose, getEngineConfig, getFromSearcher, getLastWriteNanos, isClosing, isDrainedForClose, onSearcherCreation, prepareForEngineReset, readIndexVersion, recoverFromTranslog, refreshNeeded, segmentsStats, shardFieldInfos, shardFieldStats, shardFieldStats, sparseVectorStats, sparseVectorStats, verifyEngineBeforeIndexClosing, wrapForAssertions
-
Field Details
-
REAL_TIME_GET_REFRESH_SOURCE
- See Also:
-
UNSAFE_VERSION_MAP_REFRESH_SOURCE
- See Also:
-
translogGetCount
-
translogInMemorySegmentsCount
-
-
Constructor Details
-
InternalEngine
-
-
Method Details
-
translogPersistedSeqNoConsumer
-
newIndexDeletionPolicy
protected ElasticsearchIndexDeletionPolicy newIndexDeletionPolicy(EngineConfig engineConfig, org.apache.logging.log4j.Logger logger, Translog translog, org.elasticsearch.index.engine.SoftDeletesPolicy softDeletesPolicy) -
completionStats
Description copied from class:EngineReturns theCompletionStatsfor this engine- Specified by:
completionStatsin classEngine
-
restoreLocalHistoryFromTranslog
public int restoreLocalHistoryFromTranslog(Engine.TranslogRecoveryRunner translogRecoveryRunner) throws IOException Description copied from class:EngineThis method replays translog to restore the Lucene index which might be reverted previously. This ensures that all acknowledged writes are restored correctly when this engine is promoted.- Specified by:
restoreLocalHistoryFromTranslogin classEngine- Returns:
- the number of translog operations have been recovered
- Throws:
IOException
-
fillSeqNoGaps
Description copied from class:EngineFills up the local checkpoints history with no-ops until the local checkpoint and the max seen sequence ID are identical.- Specified by:
fillSeqNoGapsin classEngine- Parameters:
primaryTerm- the shards primary term this engine was created for- Returns:
- the number of no-ops added
- Throws:
IOException
-
recoverFromTranslog
public void recoverFromTranslog(Engine.TranslogRecoveryRunner translogRecoveryRunner, long recoverUpToSeqNo, ActionListener<Void> listener) Description copied from class:EnginePerforms recovery from the transaction log up torecoverUpToSeqNo(inclusive). This operation will close the engine if the recovery fails.- Specified by:
recoverFromTranslogin classEngine- Parameters:
translogRecoveryRunner- the translog recovery runnerrecoverUpToSeqNo- the upper bound, inclusive, of sequence number to be recoveredlistener- listener notified on completion of the recovery, whether successful or otherwise
-
skipTranslogRecovery
public void skipTranslogRecovery()Description copied from class:EngineDo not replay translog operations, but make the engine be ready.- Specified by:
skipTranslogRecoveryin classEngine
-
newTranslogSnapshot
- Throws:
IOException
-
isTranslogSyncNeeded
public boolean isTranslogSyncNeeded()Description copied from class:EngineChecks if the underlying storage sync is required.- Specified by:
isTranslogSyncNeededin classEngine
-
asyncEnsureTranslogSynced
Description copied from class:EngineEnsures that the location has been written to the underlying storage.- Specified by:
asyncEnsureTranslogSyncedin classEngine
-
asyncEnsureGlobalCheckpointSynced
Description copied from class:EngineEnsures that the global checkpoint has been persisted to the underlying storage.- Specified by:
asyncEnsureGlobalCheckpointSyncedin classEngine
-
syncTranslog
- Specified by:
syncTranslogin classEngine- Throws:
IOException
-
getTranslogStats
- Specified by:
getTranslogStatsin classEngine
-
getTranslogLastWriteLocation
Description copied from class:EngineReturns the last location that the translog of this engine has written into.- Specified by:
getTranslogLastWriteLocationin classEngine
-
getHistoryUUID
Description copied from class:Enginereturns the history uuid for the engine- Specified by:
getHistoryUUIDin classEngine
-
getForceMergeUUID
returns the force merge uuid for the engine -
getWritingBytes
public long getWritingBytes()Returns how many bytes we are currently moving from indexing buffer to segments on disk- Specified by:
getWritingBytesin classEngine
-
createInternalReaderManager
protected ElasticsearchReaderManager createInternalReaderManager(ElasticsearchDirectoryReader directoryReader) -
isDocumentInLiveVersionMap
public boolean isDocumentInLiveVersionMap(org.apache.lucene.util.BytesRef uid) Description copied from class:EngineWhether the document is in the live version map or not. This is used in stateless so that theEnsureDocsSearchableActioncan judge whether a requested document needs to be refreshed to the search shards before executing the term vector information API on the search shards.- Overrides:
isDocumentInLiveVersionMapin classEngine
-
get
public Engine.GetResult get(Engine.Get get, MappingLookup mappingLookup, DocumentParser documentParser, Function<Engine.Searcher, Engine.Searcher> searcherWrapper) -
getFromTranslog
public Engine.GetResult getFromTranslog(Engine.Get get, MappingLookup mappingLookup, DocumentParser documentParser, Function<Engine.Searcher, Engine.Searcher> searcherWrapper) Description copied from class:EngineSimilar toEngine.get(org.elasticsearch.index.engine.Engine.Get, org.elasticsearch.index.mapper.MappingLookup, org.elasticsearch.index.mapper.DocumentParser, java.util.function.Function<org.elasticsearch.index.engine.Engine.Searcher, org.elasticsearch.index.engine.Engine.Searcher>), but it only attempts to serve the get from the translog. If not found in translog, it returns null, asEngine.GetResult.NOT_EXISTScould mean deletion.- Overrides:
getFromTranslogin classEngine
-
realtimeGetUnderLock
protected Engine.GetResult realtimeGetUnderLock(Engine.Get get, MappingLookup mappingLookup, DocumentParser documentParser, Function<Engine.Searcher, Engine.Searcher> searcherWrapper, boolean getFromSearcher) - Parameters:
getFromSearcher- indicates whether we also try the internal searcher if not found in translog. In the case where we just started tracking locations in the translog, we always use the internal searcher.
-
assertPrimaryCanOptimizeAddDocument
-
assertPrimaryIncomingSequenceNumber
-
generateSeqNoForOperationOnPrimary
-
advanceMaxSeqNoOfUpdatesOnPrimary
protected void advanceMaxSeqNoOfUpdatesOnPrimary(long seqNo) -
advanceMaxSeqNoOfDeletesOnPrimary
protected void advanceMaxSeqNoOfDeletesOnPrimary(long seqNo) -
index
Description copied from class:EnginePerform document index operation on the engine- Specified by:
indexin classEngine- Parameters:
index- operation to perform- Returns:
Engine.IndexResultcontaining updated translog location, version and document specific failures Note: engine level failures (i.e. persistent engine failures) are thrown- Throws:
IOException
-
planIndexingAsNonPrimary
protected final InternalEngine.IndexingStrategy planIndexingAsNonPrimary(Engine.Index index) throws IOException - Throws:
IOException
-
indexingStrategyForOperation
protected InternalEngine.IndexingStrategy indexingStrategyForOperation(Engine.Index index) throws IOException - Throws:
IOException
-
delete
Description copied from class:EnginePerform document delete operation on the engine- Specified by:
deletein classEngine- Parameters:
delete- operation to perform- Returns:
Engine.DeleteResultcontaining updated translog location, version and document specific failures Note: engine level failures (i.e. persistent engine failures) are thrown- Throws:
IOException
-
deletionStrategyForOperation
protected InternalEngine.DeletionStrategy deletionStrategyForOperation(Engine.Delete delete) throws IOException - Throws:
IOException
-
planDeletionAsNonPrimary
protected final InternalEngine.DeletionStrategy planDeletionAsNonPrimary(Engine.Delete delete) throws IOException - Throws:
IOException
-
assertNonPrimaryOrigin
-
maybePruneDeletes
public void maybePruneDeletes()Description copied from class:EngineTries to prune buffered deletes from the version map.- Specified by:
maybePruneDeletesin classEngine
-
noOp
- Specified by:
noOpin classEngine- Throws:
IOException
-
preFlightCheckForNoOp
Executes a pre-flight check for a given NoOp. If this method returns a non-empty result, the engine won't process this NoOp and returns a failure.- Throws:
IOException
-
refresh
Description copied from class:EngineSynchronously refreshes the engine for new search operations to reflect the latest changes.- Specified by:
refreshin classEngine- Throws:
EngineException
-
maybeRefresh
public void maybeRefresh(String source, ActionListener<Engine.RefreshResult> listener) throws EngineException Description copied from class:EngineAsynchronously refreshes the engine for new search operations to reflect the latest changes unless another thread is already refreshing or reseting the engine concurrently.- Specified by:
maybeRefreshin classEngine- Throws:
EngineException
-
refreshInternalSearcher
protected Engine.RefreshResult refreshInternalSearcher(String source, boolean block) throws EngineException - Throws:
EngineException
-
refresh
protected final Engine.RefreshResult refresh(String source, Engine.SearcherScope scope, boolean block) throws EngineException - Throws:
EngineException
-
writeIndexingBuffer
Description copied from class:EngineCalled when our engine is using too much heap and should move buffered indexed/deleted documents to disk.- Specified by:
writeIndexingBufferin classEngine- Throws:
IOException
-
reclaimVersionMapMemory
protected void reclaimVersionMapMemory() -
shouldPeriodicallyFlush
public boolean shouldPeriodicallyFlush()Description copied from class:EngineChecks if this engine should be flushed periodically. This check is mainly based on the uncommitted translog size and the translog flush threshold setting.- Specified by:
shouldPeriodicallyFlushin classEngine
-
flushHoldingLock
protected void flushHoldingLock(boolean force, boolean waitIfOngoing, ActionListener<Engine.FlushResult> listener) throws EngineException Description copied from class:EngineThe actual implementation ofEngine.flush(boolean, boolean, ActionListener), to be called either when holding a ref that ensures the engine remains open, or holdingIndexShard#engineMutexwhile closing the engine.- Specified by:
flushHoldingLockin classEngine- Throws:
EngineException
-
isFlushLockIsHeldByCurrentThread
protected final boolean isFlushLockIsHeldByCurrentThread() -
hasUncommittedChanges
protected boolean hasUncommittedChanges() -
afterFlush
protected void afterFlush(long generation) -
rollTranslogGeneration
Description copied from class:EngineRolls the translog generation and cleans unneeded.- Specified by:
rollTranslogGenerationin classEngine- Throws:
EngineException
-
trimUnreferencedTranslogFiles
Description copied from class:Enginechecks and removes translog files that no longer need to be retained. SeeTranslogDeletionPolicyfor details- Specified by:
trimUnreferencedTranslogFilesin classEngine- Throws:
EngineException
-
shouldRollTranslogGeneration
public boolean shouldRollTranslogGeneration()Description copied from class:EngineTests whether or not the translog generation should be rolled to a new generation. This test is based on the size of the current generation compared to the configured generation threshold size.- Specified by:
shouldRollTranslogGenerationin classEngine- Returns:
trueif the current generation should be rolled to a new generation
-
trimOperationsFromTranslog
Description copied from class:EngineTrims translog for terms belowbelowTermand seq# aboveaboveSeqNo- Specified by:
trimOperationsFromTranslogin classEngine- Throws:
EngineException- See Also:
-
forceMerge
public void forceMerge(boolean flush, int maxNumSegments, boolean onlyExpungeDeletes, String forceMergeUUID) throws EngineException, IOException Description copied from class:EngineTriggers a forced merge on this engine- Specified by:
forceMergein classEngine- Throws:
EngineExceptionIOException
-
acquireLastIndexCommit
Description copied from class:EngineSnapshots the most recent index and returns a handle to it. If needed will try and "commit" the lucene index to make sure we have a "fresh" copy of the files to snapshot.- Specified by:
acquireLastIndexCommitin classEngine- Parameters:
flushFirst- indicates whether the engine should flush before returning the snapshot- Throws:
EngineException
-
acquireSafeIndexCommit
Description copied from class:EngineSnapshots the most recent safe index commit from the engine.- Specified by:
acquireSafeIndexCommitin classEngine- Throws:
EngineException
-
getSafeCommitInfo
- Specified by:
getSafeCommitInfoin classEngine- Returns:
- a summary of the contents of the current safe commit
-
maybeFailEngine
Description copied from class:EngineCheck whether the engine should be failed- Overrides:
maybeFailEnginein classEngine
-
getLastCommittedSegmentInfos
public org.apache.lucene.index.SegmentInfos getLastCommittedSegmentInfos()- Specified by:
getLastCommittedSegmentInfosin classEngine
-
writerSegmentStats
- Overrides:
writerSegmentStatsin classEngine
-
getIndexBufferRAMBytesUsed
public long getIndexBufferRAMBytesUsed()Description copied from class:EngineHow much heap is used that would be freed by a refresh. This includes both the current memory being freed and any remaining memory usage that could be freed, e.g., by refreshing. Note that this may throwAlreadyClosedException.- Specified by:
getIndexBufferRAMBytesUsedin classEngine
-
segments
Description copied from class:EngineThe list of segments in the engine. -
segments
-
closeNoLock
Description copied from class:EngineCloses the engine without acquiring any refs or locks. The caller should either have changedEngine.isClosingfromfalsetotrueor else must hold theEngine.failEngineLock. The implementation must decrement the supplied latch when done.- Specified by:
closeNoLockin classEngine
-
getReferenceManager
protected final org.apache.lucene.search.ReferenceManager<ElasticsearchDirectoryReader> getReferenceManager(Engine.SearcherScope scope) - Specified by:
getReferenceManagerin classEngine
-
createWriter
protected org.apache.lucene.index.IndexWriter createWriter(org.apache.lucene.store.Directory directory, org.apache.lucene.index.IndexWriterConfig iwc) throws IOException - Throws:
IOException
-
activateThrottling
public void activateThrottling()Description copied from class:EngineRequest that this engine throttle incoming indexing requests to one thread. Must be matched by a later call toEngine.deactivateThrottling().- Specified by:
activateThrottlingin classEngine
-
deactivateThrottling
public void deactivateThrottling()Description copied from class:EngineReverses a previousEngine.activateThrottling()call.- Specified by:
deactivateThrottlingin classEngine
-
suspendThrottling
public void suspendThrottling()Description copied from class:EngineIf indexing is throttled to the point where it is paused completely, another task trying to get indexing permits might want to pause throttling by letting one thread pass at a time so that it does not get starved.- Specified by:
suspendThrottlingin classEngine
-
resumeThrottling
public void resumeThrottling()Description copied from class:EngineReverses a previousEngine.suspendThrottling()call.- Specified by:
resumeThrottlingin classEngine
-
isThrottled
public boolean isThrottled()Description copied from class:EngineReturns thetrueiff this engine is currently under index throttling.- Specified by:
isThrottledin classEngine- See Also:
-
getIndexThrottleTimeInMillis
public long getIndexThrottleTimeInMillis()Description copied from class:EngineReturns the number of milliseconds this engine was under index throttling.- Specified by:
getIndexThrottleTimeInMillisin classEngine
-
createMergeScheduler
protected ElasticsearchMergeScheduler createMergeScheduler(ShardId shardId, IndexSettings indexSettings, @Nullable ThreadPoolMergeExecutorService threadPoolMergeExecutorService, MergeMetrics mergeMetrics) -
mergeException
-
commitIndexWriter
protected void commitIndexWriter(org.apache.lucene.index.IndexWriter writer, Translog translog) throws IOException Commits the specified index writer.- Parameters:
writer- the index writer to committranslog- the translog- Throws:
IOException
-
getCommitExtraUserData
Allows InternalEngine extenders to return custom key-value pairs which will be included in the Lucene commit user-data. Custom user data keys can be overwritten by if their keys conflict keys used by InternalEngine.- Parameters:
localCheckpoint- the local checkpoint of the commit
-
onSettingsChanged
public void onSettingsChanged()- Overrides:
onSettingsChangedin classEngine
-
getMergeStats
- Overrides:
getMergeStatsin classEngine
-
getLocalCheckpointTracker
-
getLastSyncedGlobalCheckpoint
public long getLastSyncedGlobalCheckpoint()Description copied from class:EngineReturns the latest global checkpoint value that has been persisted in the underlying storage (i.e. translog's checkpoint)- Specified by:
getLastSyncedGlobalCheckpointin classEngine
-
getMaxSeqNo
public long getMaxSeqNo()- Specified by:
getMaxSeqNoin classEngine- Returns:
- the max issued or seen seqNo for this Engine
-
getProcessedLocalCheckpoint
public long getProcessedLocalCheckpoint()- Specified by:
getProcessedLocalCheckpointin classEngine- Returns:
- the processed local checkpoint for this Engine
-
getPersistedLocalCheckpoint
public long getPersistedLocalCheckpoint()- Specified by:
getPersistedLocalCheckpointin classEngine- Returns:
- the persisted local checkpoint for this Engine
-
markSeqNoAsSeen
protected final void markSeqNoAsSeen(long seqNo) Marks the given seq_no as seen and advances the max_seq_no of this engine to at least that value. -
hasBeenProcessedBefore
Checks if the given operation has been processed in this engine or not.- Returns:
- true if the given operation was processed; otherwise false.
-
getSeqNoStats
- Specified by:
getSeqNoStatsin classEngine- Returns:
- a
SeqNoStatsobject, using local state and the supplied global checkpoint
-
getTotalFlushTimeExcludingWaitingOnLockInMillis
public long getTotalFlushTimeExcludingWaitingOnLockInMillis()Description copied from class:EngineReturns the total time flushes have been executed excluding waiting on locks.- Overrides:
getTotalFlushTimeExcludingWaitingOnLockInMillisin classEngine
-
countChanges
Description copied from class:EngineCounts the number of operations in the range of the given sequence numbers.- Specified by:
countChangesin classEngine- Parameters:
source- the source of the requestfromSeqNo- the start sequence number (inclusive)toSeqNo- the end sequence number (inclusive)- Throws:
IOException- See Also:
-
newChangesSnapshot
public Translog.Snapshot newChangesSnapshot(String source, long fromSeqNo, long toSeqNo, boolean requiredFullRange, boolean singleConsumer, boolean accessStats, long maxChunkSize) throws IOException Description copied from class:EngineCreates a new history snapshot from Lucene for reading operations whose seqno in the requesting seqno range (both inclusive). This feature requires soft-deletes enabled. If soft-deletes are disabled, this method will throw anIllegalStateException.- Specified by:
newChangesSnapshotin classEngine- Throws:
IOException
-
hasCompleteOperationHistory
Description copied from class:EngineChecks if this engine has every operations sincestartingSeqNo(inclusive) in its history (either Lucene or translog)- Specified by:
hasCompleteOperationHistoryin classEngine
-
getMinRetainedSeqNo
public final long getMinRetainedSeqNo()Returns the minimum seqno that is retained in the Lucene index. Operations whose seq# are at least this value should exist in the Lucene index.- Specified by:
getMinRetainedSeqNoin classEngine- Returns:
- the minimum retained sequence number
-
acquireHistoryRetentionLock
Description copied from class:EngineAcquires a lock on Lucene soft-deleted documents to prevent them from being trimmed- Specified by:
acquireHistoryRetentionLockin classEngine
-
refreshIfNeeded
Refresh this engine **internally** iff the requesting seq_no is greater than the last refreshed checkpoint. -
getMaxSeenAutoIdTimestamp
public final long getMaxSeenAutoIdTimestamp()Description copied from class:EngineReturns the maximum auto_id_timestamp of all append-only index requests have been processed by this engine or the auto_id_timestamp received from its primary shard viaEngine.updateMaxUnsafeAutoIdTimestamp(long). Notes this method returns the auto_id_timestamp of all append-only requests, not max_unsafe_auto_id_timestamp.- Overrides:
getMaxSeenAutoIdTimestampin classEngine
-
updateMaxUnsafeAutoIdTimestamp
public final void updateMaxUnsafeAutoIdTimestamp(long newTimestamp) Description copied from class:EngineForces this engine to advance its max_unsafe_auto_id_timestamp marker to at least the given timestamp. The engine will disable optimization for all append-only whose timestamp at mostnewTimestamp.- Specified by:
updateMaxUnsafeAutoIdTimestampin classEngine
-
getMaxSeqNoOfUpdatesOrDeletes
public long getMaxSeqNoOfUpdatesOrDeletes()Description copied from class:EngineReturns the maximum sequence number of either update or delete operations have been processed in this engine or the sequence number fromEngine.advanceMaxSeqNoOfUpdatesOrDeletes(long). An index request is considered as an update operation if it overwrites the existing documents in Lucene index with the same document id.A note on the optimization using max_seq_no_of_updates_or_deletes: For each operation O, the key invariants are:
- I1: There is no operation on docID(O) with seqno that is > MSU(O) and < seqno(O)
- I2: If MSU(O) < seqno(O) then docID(O) did not exist when O was applied; more precisely, if there is any O' with seqno(O') < seqno(O) and docID(O') = docID(O) then the one with the greatest seqno is a delete.
When a receiving shard (either a replica or a follower) receives an operation O, it must first ensure its own MSU at least MSU(O), and then compares its MSU to its local checkpoint (LCP). If LCP < MSU then there's a gap: there may be some operations that act on docID(O) about which we do not yet know, so we cannot perform an add. Note this also covers the case where a future operation O' with seqNo(O') > seqNo(O) and docId(O') = docID(O) is processed before O. In that case MSU(O') is at least seqno(O') and this means MSU >= seqNo(O') > seqNo(O) > LCP (because O wasn't processed yet).
However, if MSU <= LCP then there is no gap: we have processed every operation <= LCP, and no operation O' with seqno(O') > LCP and seqno(O') < seqno(O) also has docID(O') = docID(O), because such an operation would have seqno(O') > LCP >= MSU >= MSU(O) which contradicts the first invariant. Furthermore in this case we immediately know that docID(O) has been deleted (or never existed) without needing to check Lucene for the following reason. If there's no earlier operation on docID(O) then this is clear, so suppose instead that the preceding operation on docID(O) is O': 1. The first invariant above tells us that seqno(O') <= MSU(O) <= LCP so we have already applied O' to Lucene. 2. Also MSU(O) <= MSU <= LCP < seqno(O) (we discard O if seqno(O) <= LCP) so the second invariant applies, meaning that the O' was a delete.
Therefore, if MSU <= LCP < seqno(O) we know that O can safely be optimized with and added to lucene with addDocument. Moreover, operations that are optimized using the MSU optimization must not be processed twice as this will create duplicates in Lucene. To avoid this we check the local checkpoint tracker to see if an operation was already processed.
- Specified by:
getMaxSeqNoOfUpdatesOrDeletesin classEngine- See Also:
-
advanceMaxSeqNoOfUpdatesOrDeletes
public void advanceMaxSeqNoOfUpdatesOrDeletes(long maxSeqNoOfUpdatesOnPrimary) Description copied from class:EngineA replica shard receives a new max_seq_no_of_updates from its primary shard, then calls this method to advance this marker to at least the given sequence number.- Specified by:
advanceMaxSeqNoOfUpdatesOrDeletesin classEngine
-
getRawFieldRange
- Specified by:
getRawFieldRangein classEngine- Returns:
- a
ShardLongFieldRangecontaining the min and max raw values of the given field for this shard if the engine guarantees these values never to change, orShardLongFieldRange.EMPTYif this field is empty, orShardLongFieldRange.UNKNOWNif this field's value range may change in future.
-
addFlushListener
Add a listener for the translog location. The listener is notified when a commit covering the translog location is flushed (see alsoflushHoldingLock(boolean, boolean, org.elasticsearch.action.ActionListener<org.elasticsearch.index.engine.Engine.FlushResult>)). NOTE that flush does _not_ imply commit durability. If commit durability is desired, the caller must handle it separately.- Overrides:
addFlushListenerin classEngine
-
waitForCommitDurability
This method is meant to be overridden by subclasses that require special logic to ensure commit durability. The default implementation is basically a NOOP. -
getLastUnsafeSegmentGenerationForGets
public long getLastUnsafeSegmentGenerationForGets()- Overrides:
getLastUnsafeSegmentGenerationForGetsin classEngine
-
createLiveVersionMapArchive
-
getLiveVersionMapArchive
-
getLiveVersionMap
-
getPreCommitSegmentGeneration
protected long getPreCommitSegmentGeneration() -
estimateMergeBytes
protected long estimateMergeBytes(org.apache.lucene.index.MergePolicy.OneMerge merge) -
deleteByQuery
- Throws:
Exception
-