Class BlobStoreRepository
- All Implemented Interfaces:
Closeable,AutoCloseable,LifecycleComponent,Releasable,Repository
- Direct Known Subclasses:
FsRepository,MeteredBlobStoreRepository
This repository works with any BlobStore implementation. The blobStore could be (and is preferably) lazily initialized in
createBlobStore().
org.elasticsearch.repositories.blobstore.-
Nested Class Summary
Nested classes/interfaces inherited from interface org.elasticsearch.repositories.Repository
Repository.Factory -
Field Summary
FieldsModifier and TypeFieldDescriptionprotected final BigArraysstatic final Setting<ByteSizeValue> Size hint for the IO buffer size to use when reading from and writing to the repository.protected final intIO buffer size hint for reading and writing to the underlying blob store.Setting to disable caching of the latest repository data.When set to true metadata files are stored in compressed format.static final ChecksumBlobStoreFormat<Metadata> static final StringPrefix for the name of the rootRepositoryDatablob.static final StringName of the blob which points to the rootRepositoryDatablob.static final ChecksumBlobStoreFormat<IndexMetadata> static final ChecksumBlobStoreFormat<BlobStoreIndexShardSnapshot> static final ChecksumBlobStoreFormat<BlobStoreIndexShardSnapshots> static final Setting<ByteSizeValue> static final Setting<ByteSizeValue> Setting that defines the maximum number of snapshots to which the repository may grow.protected RepositoryMetadatastatic final StringName suffix for allmeta-*.datandsnap-*.datblobs.static final StringBlob name format for globalMetadataandIndexMetadatablobs.static final StringName prefix for the blobs that hold the globalMetadatafor each snapshot, and theIndexMetadatafor its indices.static final Stringstatic final Stringstatic final StringAllBlobStoreRepositoryimplementations can be made read-only by setting this key totruein their settings.static final Stringstatic final Stringstatic final ChecksumBlobStoreFormat<SnapshotInfo> static final StringName format for shard-levelBlobStoreIndexShardSnapshotsblobs.static final StringName prefix for shard-levelBlobStoreIndexShardSnapshotsblobs.static final StringBlob name format for top-levelSnapshotInfoand shard-levelBlobStoreIndexShardSnapshotblobs.static final StringName prefix for the blobs that hold top-levelSnapshotInfoand shard-levelBlobStoreIndexShardSnapshotmetadata.static final Stringstatic final Stringstatic final Stringstatic final Stringstatic final StringSetting to disable writing theindex.latestblob which enables the contents of this repository to be used with a url-repository.protected final booleanprotected final ThreadPoolstatic final StringName prefix for blobs holding the actual shard data.static final StringSetting that defines if the repository should be used to recover index files during peer recoveries.Fields inherited from class org.elasticsearch.common.component.AbstractLifecycleComponent
lifecycle -
Constructor Summary
ConstructorsModifierConstructorDescriptionprotectedBlobStoreRepository(RepositoryMetadata metadata, NamedXContentRegistry namedXContentRegistry, ClusterService clusterService, BigArrays bigArrays, RecoverySettings recoverySettings, BlobPath basePath) Constructs new BlobStoreRepository -
Method Summary
Modifier and TypeMethodDescriptionvoidBlock until all in-flight operations for this repository have completed.basePath()Returns base path of the repository Public for testing.protected BlobContainermaintains single lazy instance ofBlobContainerMaintains single lazy instance ofBlobStore.booleancanUpdateInPlace(Settings updatedSettings, Set<String> ignoredSettings) Check if this instancesSettingscan be changed to the provided updated settings without recreating the repository.protected ByteSizeValueReturns data file chunk size.voidcleanup(long repositoryDataGeneration, IndexVersion repositoryFormatIndexVersion, ActionListener<DeleteResult> listener) Runs cleanup actions on the repository.voidcloneShardSnapshot(SnapshotId source, SnapshotId target, RepositoryShardId shardId, ShardGeneration shardGeneration, ActionListener<ShardSnapshotResult> listener) Clones a shard snapshot.protected abstract BlobStoreCreates new BlobStore to read and write data.voiddeleteSnapshots(Collection<SnapshotId> snapshotIds, long repositoryDataGeneration, IndexVersion minimumNodeVersion, ActionListener<RepositoryData> repositoryDataUpdateListener, Runnable onCompletion) Deletes snapshotsprotected voiddoClose()Close this component.protected voiddoStart()Start this component.protected voiddoStop()Stop this component.voidendVerification(String seed) Called at the end of repository verification process.voidfinalizeSnapshot(FinalizeSnapshotContext finalizeSnapshotContext) Finalizes snapshotting processprotected BlobStoregetBlobStoreIndexShardSnapshots(IndexId indexId, int shardId, ShardGeneration shardGen) Loads all available snapshots in the repository using the givengenerationfor a shard.All blob-store repositories include the counts of read-only and read-write repositories in their telemetry.Returns metadata about this repository.intvoidgetRepositoryData(Executor responseExecutor, ActionListener<RepositoryData> listener) Fetches theRepositoryDataand passes it into the listener.static StringgetRepositoryDataBlobName(long repositoryGeneration) longReturns restore throttle time in nanosecondsgetShardSnapshotStatus(SnapshotId snapshotId, IndexId indexId, ShardId shardId) Retrieve shard snapshot status for the stored snapshotgetSnapshotGlobalMetadata(SnapshotId snapshotId) Returns global metadata associated with the snapshot.getSnapshotIndexMetaData(RepositoryData repositoryData, SnapshotId snapshotId, IndexId index) Returns the index metadata associated with the snapshot.voidgetSnapshotInfo(Collection<SnapshotId> snapshotIds, boolean abortOnFailure, BooleanSupplier isCancelled, CheckedConsumer<SnapshotInfo, Exception> consumer, ActionListener<Void> listener) Reads a collection ofSnapshotInfoinstances from the repository.longReturns snapshot throttle time in nanosecondsbooleanprotected final booleanReturns true if metadata and snapshot files should be compressedbooleanReturns true if the repository supports only read operationsloadShardSnapshot(BlobContainer shardContainer, SnapshotId snapshotId) Loads information about shard snapshotmaybeRateLimitRestores(InputStream stream) Wrap the restore rate limiter (controlled by the repository setting `max_restore_bytes_per_sec` and the cluster setting `indices.recovery.max_bytes_per_sec`) around the given stream.maybeRateLimitRestores(InputStream stream, RateLimitingInputStream.Listener throttleListener) Wrap the restore rate limiter (controlled by the repository setting `max_restore_bytes_per_sec` and the cluster setting `indices.recovery.max_bytes_per_sec`) around the given stream.maybeRateLimitSnapshots(InputStream stream) Wrap the snapshot rate limiter around the given stream.maybeRateLimitSnapshots(InputStream stream, RateLimitingInputStream.Listener throttleListener) Wrap the snapshot rate limiter around the given stream.voidrestoreShard(Store store, SnapshotId snapshotId, IndexId indexId, ShardId snapshotShardId, RecoveryState recoveryState, ActionListener<Void> listener) Restores snapshot of the shard.shardContainer(IndexId indexId, int shardId) protected voidsnapshotFile(SnapshotShardContext context, BlobStoreIndexShardSnapshot.FileInfo fileInfo) Snapshot individual fileprotected voidsnapshotFiles(SnapshotShardContext context, BlockingQueue<BlobStoreIndexShardSnapshot.FileInfo> filesToSnapshot, ActionListener<Collection<Void>> allFilesUploadedListener) voidsnapshotShard(SnapshotShardContext context) Creates a snapshot of the shard referenced by the givenSnapshotShardContext.Verifies repository on the master node and returns the verification token.stats()Returns stats on the repository usagebooleantoString()voidupdateState(ClusterState state) Update the repository with the incoming cluster state.voidverify(String seed, DiscoveryNode localNode) Verifies repository settings on data node.protected ActionListener<RepositoryData> Some repositories (i.e.protected voidwriteIndexGen(RepositoryData repositoryData, long expectedGen, IndexVersion version, Function<ClusterState, ClusterState> stateFilter, ActionListener<RepositoryData> listener) Writing a new index generation (root) blob is a three-step process.Methods inherited from class org.elasticsearch.common.component.AbstractLifecycleComponent
addLifecycleListener, close, lifecycleState, start, stopMethods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, waitMethods inherited from interface org.elasticsearch.common.component.LifecycleComponent
addLifecycleListener, lifecycleState, start, stopMethods inherited from interface org.elasticsearch.core.Releasable
closeMethods inherited from interface org.elasticsearch.repositories.Repository
getSnapshotInfo
-
Field Details
-
metadata
-
threadPool
-
STATELESS_SHARD_READ_THREAD_NAME
- See Also:
-
STATELESS_TRANSLOG_THREAD_NAME
- See Also:
-
STATELESS_SHARD_WRITE_THREAD_NAME
- See Also:
-
STATELESS_CLUSTER_STATE_READ_WRITE_THREAD_NAME
- See Also:
-
STATELESS_SHARD_PREWARMING_THREAD_NAME
- See Also:
-
SEARCHABLE_SNAPSHOTS_CACHE_FETCH_ASYNC_THREAD_NAME
- See Also:
-
SEARCHABLE_SNAPSHOTS_CACHE_PREWARMING_THREAD_NAME
- See Also:
-
INDEX_FILE_PREFIX
Prefix for the name of the rootRepositoryDatablob.- See Also:
-
INDEX_LATEST_BLOB
Name of the blob which points to the rootRepositoryDatablob.- See Also:
-
METADATA_PREFIX
Name prefix for the blobs that hold the globalMetadatafor each snapshot, and theIndexMetadatafor its indices.- See Also:
-
SNAPSHOT_PREFIX
Name prefix for the blobs that hold top-levelSnapshotInfoand shard-levelBlobStoreIndexShardSnapshotmetadata.- See Also:
-
METADATA_BLOB_NAME_SUFFIX
Name suffix for allmeta-*.datandsnap-*.datblobs.- See Also:
-
METADATA_NAME_FORMAT
Blob name format for globalMetadataandIndexMetadatablobs.- See Also:
-
SNAPSHOT_NAME_FORMAT
Blob name format for top-levelSnapshotInfoand shard-levelBlobStoreIndexShardSnapshotblobs.- See Also:
-
SNAPSHOT_INDEX_PREFIX
Name prefix for shard-levelBlobStoreIndexShardSnapshotsblobs.- See Also:
-
SNAPSHOT_INDEX_NAME_FORMAT
Name format for shard-levelBlobStoreIndexShardSnapshotsblobs.- See Also:
-
UPLOADED_DATA_BLOB_PREFIX
Name prefix for blobs holding the actual shard data.- See Also:
-
URL_REPOSITORY_TYPE
- See Also:
-
READONLY_SETTING_KEY
AllBlobStoreRepositoryimplementations can be made read-only by setting this key totruein their settings.- See Also:
-
COMPRESS_SETTING
When set to true metadata files are stored in compressed format. This setting doesn’t affect index files that are already compressed by default. Changing the setting does not invalidate existing files since reads do not observe the setting, instead they examine the file to see if it is compressed or not. -
CACHE_REPOSITORY_DATA
Setting to disable caching of the latest repository data. -
BUFFER_SIZE_SETTING
Size hint for the IO buffer size to use when reading from and writing to the repository. -
SUPPORT_URL_REPO
Setting to disable writing theindex.latestblob which enables the contents of this repository to be used with a url-repository. -
MAX_SNAPSHOTS_SETTING
Setting that defines the maximum number of snapshots to which the repository may grow. Trying to create a snapshot into the repository that would move it above this size will throw an exception. -
USE_FOR_PEER_RECOVERY_SETTING
Setting that defines if the repository should be used to recover index files during peer recoveries. -
supportURLRepo
protected final boolean supportURLRepo -
GLOBAL_METADATA_FORMAT
-
INDEX_METADATA_FORMAT
-
SNAPSHOT_FORMAT
-
INDEX_SHARD_SNAPSHOT_FORMAT
public static final ChecksumBlobStoreFormat<BlobStoreIndexShardSnapshot> INDEX_SHARD_SNAPSHOT_FORMAT -
INDEX_SHARD_SNAPSHOTS_FORMAT
public static final ChecksumBlobStoreFormat<BlobStoreIndexShardSnapshots> INDEX_SHARD_SNAPSHOTS_FORMAT -
MAX_SNAPSHOT_BYTES_PER_SEC
-
MAX_RESTORE_BYTES_PER_SEC
-
bigArrays
-
bufferSize
protected final int bufferSizeIO buffer size hint for reading and writing to the underlying blob store. -
READ_ONLY_USAGE_STATS_NAME
- See Also:
-
READ_WRITE_USAGE_STATS_NAME
- See Also:
-
-
Constructor Details
-
BlobStoreRepository
protected BlobStoreRepository(RepositoryMetadata metadata, NamedXContentRegistry namedXContentRegistry, ClusterService clusterService, BigArrays bigArrays, RecoverySettings recoverySettings, BlobPath basePath) Constructs new BlobStoreRepository- Parameters:
metadata- The metadata for this repository including name and settingsclusterService- ClusterService
-
-
Method Details
-
getRepositoryDataBlobName
- Parameters:
repositoryGeneration- The numeric generation of theRepositoryDatablob.- Returns:
- The name of the blob holding the corresponding
RepositoryData.
-
doStart
protected void doStart()Description copied from class:AbstractLifecycleComponentStart this component. Typically that means doing things like launching background processes and registering listeners on other components. Other components have been initialized by this point, but may not yet be started.If this method throws an exception then the startup process will fail, but this component will not be stopped before it is closed.
This method is called while synchronized on
AbstractLifecycleComponent.lifecycle. It is only called once in the lifetime of a component, although it may not be called at all if the startup process encountered some kind of fatal error, such as the failure of some other component to initialize or start.- Specified by:
doStartin classAbstractLifecycleComponent
-
doStop
protected void doStop()Description copied from class:AbstractLifecycleComponentStop this component. Typically that means doing the reverse of whateverAbstractLifecycleComponent.doStart()does.This method is called while synchronized on
AbstractLifecycleComponent.lifecycle. It is only called once in the lifetime of a component, after callingAbstractLifecycleComponent.doStart(), although it will not be called at all if this component did not successfully start.- Specified by:
doStopin classAbstractLifecycleComponent
-
doClose
protected void doClose()Description copied from class:AbstractLifecycleComponentClose this component. Typically that means doing the reverse of whatever happened during initialization, such as releasing resources acquired there.This method is called while synchronized on
AbstractLifecycleComponent.lifecycle. It is called once in the lifetime of a component. If the component was started then it will be stopped before it is closed, and once it is closed it will not be started or stopped.- Specified by:
doClosein classAbstractLifecycleComponent
-
awaitIdle
public void awaitIdle()Description copied from interface:RepositoryBlock until all in-flight operations for this repository have completed. Must only be called after this instance has been closed by a call to stopReleasable.close(). Waiting for ongoing operations should be implemented here instead of inLifecycleComponent.stop()orReleasable.close()hooks of this interface as these are expected to be called on the cluster state applier thread (which must not block) if a repository is removed from the cluster. This method is intended to be called on node shutdown instead as a means to ensure no repository operations are leaked.- Specified by:
awaitIdlein interfaceRepository
-
cloneShardSnapshot
public void cloneShardSnapshot(SnapshotId source, SnapshotId target, RepositoryShardId shardId, @Nullable ShardGeneration shardGeneration, ActionListener<ShardSnapshotResult> listener) Description copied from interface:RepositoryClones a shard snapshot.- Specified by:
cloneShardSnapshotin interfaceRepository- Parameters:
source- source snapshottarget- target snapshotshardId- shard idshardGeneration- shard generation in repolistener- listener to complete with new shard generation once clone has completed
-
canUpdateInPlace
Description copied from interface:RepositoryCheck if this instancesSettingscan be changed to the provided updated settings without recreating the repository.- Specified by:
canUpdateInPlacein interfaceRepository- Parameters:
updatedSettings- new repository settingsignoredSettings- setting names to ignore even if changed- Returns:
- true if the repository can be updated in place
-
updateState
Description copied from interface:RepositoryUpdate the repository with the incoming cluster state. This method is invoked fromRepositoriesService.applyClusterState(org.elasticsearch.cluster.ClusterChangedEvent)and thus the same semantics as withClusterStateApplier.applyClusterState(org.elasticsearch.cluster.ClusterChangedEvent)apply for theClusterStatethat is passed here.- Specified by:
updateStatein interfaceRepository- Parameters:
state- new cluster state
-
threadPool
-
getBlobStore
-
blobContainer
maintains single lazy instance ofBlobContainer -
blobStore
Maintains single lazy instance ofBlobStore. Public for testing. -
createBlobStore
Creates new BlobStore to read and write data.- Throws:
Exception
-
basePath
Returns base path of the repository Public for testing. -
isCompress
protected final boolean isCompress()Returns true if metadata and snapshot files should be compressed- Returns:
- true if compression is needed
-
chunkSize
Returns data file chunk size.This method should return null if no chunking is needed.
- Returns:
- chunk size
-
getMetadata
Description copied from interface:RepositoryReturns metadata about this repository.- Specified by:
getMetadatain interfaceRepository
-
stats
Description copied from interface:RepositoryReturns stats on the repository usage- Specified by:
statsin interfaceRepository
-
wrapWithWeakConsistencyProtection
protected ActionListener<RepositoryData> wrapWithWeakConsistencyProtection(ActionListener<RepositoryData> listener) Some repositories (i.e. S3) run at extra risk of corruption when using the pre-7.6.0 repository format, against which we try and protect by adding some delays in between operations so that things have a chance to settle down. This method is the hook that allows the delete process to add this protection when necessary. -
deleteSnapshots
public void deleteSnapshots(Collection<SnapshotId> snapshotIds, long repositoryDataGeneration, IndexVersion minimumNodeVersion, ActionListener<RepositoryData> repositoryDataUpdateListener, Runnable onCompletion) Description copied from interface:RepositoryDeletes snapshots- Specified by:
deleteSnapshotsin interfaceRepository- Parameters:
snapshotIds- snapshot ids to deleterepositoryDataGeneration- the generation of theRepositoryDatain the repository at the start of the deletionminimumNodeVersion- the minimumIndexVersionacross the nodes in the cluster, with which the repository format must remain compatiblerepositoryDataUpdateListener- listener completed when theRepositoryDatais updated, or when the process fails without changing the repository contents - in either case, it is now safe for the next operation on this repository to proceed.onCompletion- action executed on completion of the cleanup actions that follow a successfulRepositoryDataupdate; not called ifrepositoryDataUpdateListenercompletes exceptionally.
-
cleanup
public void cleanup(long repositoryDataGeneration, IndexVersion repositoryFormatIndexVersion, ActionListener<DeleteResult> listener) Runs cleanup actions on the repository. Increments the repository state id by one before executing any modifications on the repository. TODO: Add shard level cleanups TODO: Add unreferenced index metadata cleanup- Deleting stale indices
- Deleting unreferenced root level blobs
- Parameters:
repositoryDataGeneration- Generation ofRepositoryDataat start of processrepositoryFormatIndexVersion- Repository format versionlistener- Listener to complete when done
-
finalizeSnapshot
Description copied from interface:RepositoryFinalizes snapshotting processThis method is called on master after all shards are snapshotted.
- Specified by:
finalizeSnapshotin interfaceRepository- Parameters:
finalizeSnapshotContext- finalization context
-
getSnapshotInfo
public void getSnapshotInfo(Collection<SnapshotId> snapshotIds, boolean abortOnFailure, BooleanSupplier isCancelled, CheckedConsumer<SnapshotInfo, Exception> consumer, ActionListener<Void> listener) Description copied from interface:RepositoryReads a collection ofSnapshotInfoinstances from the repository.- Specified by:
getSnapshotInfoin interfaceRepository- Parameters:
snapshotIds- The IDs of the snapshots whoseSnapshotInfoinstances should be retrieved.abortOnFailure- Whether to stop fetching furtherSnapshotInfoinstances if a single fetch fails.isCancelled- Supplies whether the enclosing task is cancelled, which should stop fetchingSnapshotInfoinstances.consumer- A consumer for eachSnapshotInforetrieved. Called concurrently from multiple threads. If the consumer throws an exception andabortOnFailureistruethen the fetching will stop.listener- IfabortOnFailureistrueand any operation fails then the failure is passed to this listener. Also completed exceptionally on cancellation. Otherwise, completed once all requestedSnapshotInfoinstances have been processed by theconsumer.
-
getSnapshotGlobalMetadata
Description copied from interface:RepositoryReturns global metadata associated with the snapshot.- Specified by:
getSnapshotGlobalMetadatain interfaceRepository- Parameters:
snapshotId- the snapshot id to load the global metadata from- Returns:
- the global metadata about the snapshot
-
getSnapshotIndexMetaData
public IndexMetadata getSnapshotIndexMetaData(RepositoryData repositoryData, SnapshotId snapshotId, IndexId index) throws IOException Description copied from interface:RepositoryReturns the index metadata associated with the snapshot.- Specified by:
getSnapshotIndexMetaDatain interfaceRepository- Parameters:
repositoryData- currentRepositoryDatasnapshotId- the snapshot id to load the index metadata fromindex- theIndexIdto load the metadata from- Returns:
- the index metadata about the given index for the given snapshot
- Throws:
IOException
-
shardContainer
-
getSnapshotThrottleTimeInNanos
public long getSnapshotThrottleTimeInNanos()Description copied from interface:RepositoryReturns snapshot throttle time in nanoseconds- Specified by:
getSnapshotThrottleTimeInNanosin interfaceRepository
-
getRestoreThrottleTimeInNanos
public long getRestoreThrottleTimeInNanos()Description copied from interface:RepositoryReturns restore throttle time in nanoseconds- Specified by:
getRestoreThrottleTimeInNanosin interfaceRepository
-
startVerification
Description copied from interface:RepositoryVerifies repository on the master node and returns the verification token.If the verification token is not null, it's passed to all data nodes for verification. If it's null - no additional verification is required
- Specified by:
startVerificationin interfaceRepository- Returns:
- verification token that should be passed to all Index Shard Repositories for additional verification or null
-
endVerification
Description copied from interface:RepositoryCalled at the end of repository verification process.This method should perform all necessary cleanup of the temporary files created in the repository
- Specified by:
endVerificationin interfaceRepository- Parameters:
seed- verification request generated byRepository.startVerification()command
-
getRepositoryData
Description copied from interface:RepositoryFetches theRepositoryDataand passes it into the listener. May completes the listener with aRepositoryExceptionif there is an error in reading the repository data.- Specified by:
getRepositoryDatain interfaceRepository- Parameters:
responseExecutor- Executor to use to complete the listener if not using the calling thread. UsingEsExecutors.DIRECT_EXECUTOR_SERVICEmeans to complete the listener on the thread which ultimately resolved theRepositoryData, which might be a low-latency transport or cluster applier thread so make sure not to do anything slow or expensive in that case.listener- Listener which is either completed on the calling thread (if theRepositoryDatais immediately available, e.g. from an in-memory cache), otherwise it is completed usingresponseExecutor.
-
isReadOnly
public boolean isReadOnly()Description copied from interface:RepositoryReturns true if the repository supports only read operations- Specified by:
isReadOnlyin interfaceRepository- Returns:
- true if the repository is read/only
-
writeIndexGen
protected void writeIndexGen(RepositoryData repositoryData, long expectedGen, IndexVersion version, Function<ClusterState, ClusterState> stateFilter, ActionListener<RepositoryData> listener) Writing a new index generation (root) blob is a three-step process. Typically, it starts from a stable state where the pending generationRepositoryMetadata.pendingGeneration()is equal to the safe generationRepositoryMetadata.generation(), but after a failure it may be that the pending generation starts out greater than the safe generation.-
We reserve ourselves a new root blob generation
G, greater thanRepositoryMetadata.pendingGeneration(), via a cluster state update which edits theRepositoryMetadataentry for this repository, increasing its pending generation toGwithout changing its safe generation. -
We write the updated
RepositoryDatato a new root blob with generationG. -
We mark the successful end of the update of the repository data with a cluster state update which edits the
RepositoryMetadataentry for this repository again, increasing its safe generation to equal to its pending generationG.
Gthen we will fail to update the safe repository generation in the final step, and meanwhile the new master will choose a generation greater thanGfor all subsequent root blobs so there is no risk that we will clobber its writes. See the package level documentation fororg.elasticsearch.repositories.blobstorefor more details.Note that a failure here does not imply that the process was unsuccessful or the repository is unchanged. Once we have written the new root blob the repository is updated from the point of view of any other clusters reading from it, and if we performed a full cluster restart at that point then we would also pick up the new root blob. Writing the root blob may succeed without us receiving a successful response from the repository, leading us to report that the write operation failed. Updating the safe generation may likewise succeed on a majority of master-eligible nodes which does not include this one, again leading to an apparent failure.
We therefore cannot safely clean up apparently-dangling blobs after a failure here. Instead, we defer any cleanup until after the next successful root-blob write, which may happen on a different master node or possibly even in a different cluster.
- Parameters:
repositoryData- RepositoryData to writeexpectedGen- expected repository generation at the start of the operationversion- version of the repository metadata to writestateFilter- filter for the last cluster state update executed by this methodlistener- completion listener
-
We reserve ourselves a new root blob generation
-
snapshotShard
Description copied from interface:RepositoryCreates a snapshot of the shard referenced by the givenSnapshotShardContext.As snapshot process progresses, implementation of this method should update
IndexShardSnapshotStatusobject returned bySnapshotShardContext.status()and callIndexShardSnapshotStatus.ensureNotAborted()to see if the snapshot process should be aborted.- Specified by:
snapshotShardin interfaceRepository- Parameters:
context- snapshot shard context that must be completed viaSnapshotShardContext.onResponse(org.elasticsearch.repositories.ShardSnapshotResult)orDelegatingActionListener.onFailure(java.lang.Exception)
-
snapshotFiles
protected void snapshotFiles(SnapshotShardContext context, BlockingQueue<BlobStoreIndexShardSnapshot.FileInfo> filesToSnapshot, ActionListener<Collection<Void>> allFilesUploadedListener) -
restoreShard
public void restoreShard(Store store, SnapshotId snapshotId, IndexId indexId, ShardId snapshotShardId, RecoveryState recoveryState, ActionListener<Void> listener) Description copied from interface:RepositoryRestores snapshot of the shard.The index can be renamed on restore, hence different
shardIdandsnapshotShardIdare supplied.- Specified by:
restoreShardin interfaceRepository- Parameters:
store- the store to restore the index intosnapshotId- snapshot idindexId- id of the index in the repository from which the restore is occurringsnapshotShardId- shard id (in the snapshot)recoveryState- recovery statelistener- listener to invoke once done
-
maybeRateLimitRestores
Wrap the restore rate limiter (controlled by the repository setting `max_restore_bytes_per_sec` and the cluster setting `indices.recovery.max_bytes_per_sec`) around the given stream. Any throttling is reported to the given listener and not otherwise recorded in the value returned bygetRestoreThrottleTimeInNanos(). -
maybeRateLimitRestores
public InputStream maybeRateLimitRestores(InputStream stream, RateLimitingInputStream.Listener throttleListener) Wrap the restore rate limiter (controlled by the repository setting `max_restore_bytes_per_sec` and the cluster setting `indices.recovery.max_bytes_per_sec`) around the given stream. Any throttling is recorded in the value returned bygetRestoreThrottleTimeInNanos(). -
maybeRateLimitSnapshots
Wrap the snapshot rate limiter around the given stream. Any throttling is recorded in the value returned bygetSnapshotThrottleTimeInNanos(). Note that speed is throttled by the repository setting `max_snapshot_bytes_per_sec` and, if recovery node bandwidth settings have been set, additionally by the `indices.recovery.max_bytes_per_sec` speed. -
maybeRateLimitSnapshots
public InputStream maybeRateLimitSnapshots(InputStream stream, RateLimitingInputStream.Listener throttleListener) Wrap the snapshot rate limiter around the given stream. Any throttling is recorded in the value returned bygetSnapshotThrottleTimeInNanos(). Note that speed is throttled by the repository setting `max_snapshot_bytes_per_sec` and, if recovery node bandwidth settings have been set, additionally by the `indices.recovery.max_bytes_per_sec` speed. -
getShardSnapshotStatus
public IndexShardSnapshotStatus.Copy getShardSnapshotStatus(SnapshotId snapshotId, IndexId indexId, ShardId shardId) Description copied from interface:RepositoryRetrieve shard snapshot status for the stored snapshot- Specified by:
getShardSnapshotStatusin interfaceRepository- Parameters:
snapshotId- snapshot idindexId- the snapshotted index id for the shard to get status forshardId- shard id- Returns:
- snapshot status
-
verify
Description copied from interface:RepositoryVerifies repository settings on data node.- Specified by:
verifyin interfaceRepository- Parameters:
seed- value returned byRepository.startVerification()localNode- the local node information, for inclusion in verification errors
-
toString
-
loadShardSnapshot
public BlobStoreIndexShardSnapshot loadShardSnapshot(BlobContainer shardContainer, SnapshotId snapshotId) Loads information about shard snapshot -
getBlobStoreIndexShardSnapshots
public BlobStoreIndexShardSnapshots getBlobStoreIndexShardSnapshots(IndexId indexId, int shardId, @Nullable ShardGeneration shardGen) throws IOException Loads all available snapshots in the repository using the givengenerationfor a shard. WhenshardGenis null it tries to load it using the BwC mode, listing the available index- blobs in the shard container.- Throws:
IOException
-
snapshotFile
protected void snapshotFile(SnapshotShardContext context, BlobStoreIndexShardSnapshot.FileInfo fileInfo) throws IOException Snapshot individual file- Parameters:
fileInfo- file to snapshot- Throws:
IOException
-
supportURLRepo
public boolean supportURLRepo() -
hasAtomicOverwrites
public boolean hasAtomicOverwrites()- Returns:
- whether this repository performs overwrites atomically. In practice we only overwrite the `index.latest` blob so this is not very important, but the repository analyzer does test that overwrites happen atomically. It will skip those tests if the repository overrides this method to indicate that it does not support atomic overwrites.
-
getReadBufferSizeInBytes
public int getReadBufferSizeInBytes() -
getAnalysisFailureExtraDetail
- Returns:
- extra information to be included in the exception message emitted on failure of a repository analysis.
-
getUsageFeatures
- Specified by:
getUsageFeaturesin interfaceRepository- Returns:
- a set of the names of the features that this repository instance uses, for reporting in the cluster stats for telemetry collection.
-
getExtraUsageFeatures
All blob-store repositories include the counts of read-only and read-write repositories in their telemetry. This method returns other features of the repositories in use.- Returns:
- a set of the names of the extra features that this repository instance uses, for reporting in the cluster stats for telemetry collection.
-