Batch wraps an external storage instance to a batched version.

Create creates ExternalStorage.

DefineFlags adds flags to the flag set corresponding to all backend options.

ExtractQueryParameters moves the query parameters of the URL into the options using reflection.

FormatBackendURL obtains the raw URL which can be used the reconstruct the backend.

GetActiveUploadWorkerCount returns the active upload worker count.

Different from `http.DefaultTransport`, set the `MaxIdleConns` and `MaxIdleConnsPerHost` to the actual request concurrency to reuse tcp connection as much as possible.

HiddenFlagsForStream hidden flags for stream cmd.

InterceptDecompressReader intercepts the reader and wraps it with a decompress reader on the given ExternalFileReader.

IsLocal returns true if the URL is a local file path.

IsLocalPath returns true if the path is a local file path.

JSONEffects converts a slices of effects into json.

MakeLockMeta creates a LockMeta by the current node's metadata.

New creates an ExternalStorage with options.

NewBufferWriter creates a Writer that simply writes to a buffer (useful for testing).

NewFromURL creates an ExternalStorage from URL.

NewGCSStorage creates a GCS external storage implementation.

NewGCSWriter returns a GCSWriter which uses GCS multipart upload API behind the scene.

NewHDFSStorage creates a new HDFS storage.

NewKS3Storage initialize a new s3 storage for metadata.

NewLocalStorage return a LocalStorage at directory `base`.

NewMemStorage creates a new in-memory storage.

NewS3Storage initialize a new s3 storage for metadata.

NewS3StorageForTest creates a new S3Storage for testing only.

NewUploaderWriter wraps the Writer interface over an uploader.

NewWithDefaultOpt creates ExternalStorage with default options.

ParseBackend constructs a structured backend description from the storage URL.

ParseBackendFromURL constructs a structured backend description from the *url.URL.

ParseRangeInfo parses the Content-Range header and returns the offsets.

ParseRawURL parse raw url to url object.

PutAndDeleteObjectCheck checks the permission of putObject S3 API doesn't provide a way to check the permission, we have to put an object to check the permission.

ReadDataInRange reads data from storage in range [start, start+len(p)).

TryLockRemote tries to create a "lock file" at the external storage.

UnmarshalDir iterates over a prefix, then "unmarshal" the content of each file it met with the unmarshal function.

ValidateCloudStorageURI makes validation for tidb_cloud_storage_uri.

WithCompression returns an ExternalStorage with compress option.

AccessBuckets represents bucket access permission it replace the origin skip-check-path.

GetObject represents GetObject permission.

Gzip will compress given bytes in gzip format.

ListObjects represents listObjects permission.

LocalURIPrefix represents the local storage prefix.

NoCompression won't compress given bytes.

PutAndDeleteObject represents PutAndDeleteObject permission we cannot check DeleteObject permission alone, so we use PutAndDeleteObject instead.

PutObject represents PutObject permission.

Snappy will compress given bytes in snappy format.

Zstd will compress given bytes in zstd format.

WriteBufferSize is the size of the buffer used for writing.

AzblobBackendOptions is the options for Azure Blob storage.

AzureBlobStorage is a storage engine that stores data in Azure Blob Storage.

BackendOptions further configures the storage backend not expressed by the storage URL.

Batched is a wrapper of an external storage that suspends all write operations ("effects").

BytesWriter is a Writer implementation on top of bytes.Buffer that is useful for testing.

DecompressConfig is the config used for decompression.

EffDeleteFiles is the side effect of a call to `DeleteFiles`.

EffPut is the side effect of a call to `WriteFile`.

EffRename is the side effect of a call to `Rename`.

ErrLocked is the error returned when the lock is held by others.

ExternalStorageOptions are backend-independent options provided to New.

GCSBackendOptions are options for configuration the GCS storage.

GCSStorage defines some standard operations for BR/Lightning on the GCS storage.

GCSWriter uses XML multipart upload API to upload a single file.

HDFSStorage represents HDFS storage.

KS3Storage acts almost same as S3Storage except it's used for kingsoft s3.

KS3Uploader does multi-part upload to s3.

LocalStorage represents local file system storage.

LockMeta is the meta information of a lock.

MemStorage represents a in-memory storage.

NoopWriter is a writer that does nothing.

RangeInfo represents the an HTTP Content-Range header value of the form `bytes [Start]-[End]/[Size]`.

S3BackendOptions contains options for s3 storage.

S3Storage defines some standard operations for BR/Lightning on the S3 storage.

S3Uploader does multi-part upload to s3.

WalkOption is the option of storage.WalkDir.

ClientBuilder provides common method to build a service client.

ExternalFileReader represents the streaming external file reader.

ExternalFileWriter represents the streaming external file writer.

ExternalStorage represents a kind of file system storage.

ReadSeekCloser is the interface that groups the basic Read, Seek and Close methods.

StrongConsistency is a marker interface that indicates the storage is strong consistent over its `Read`, `Write` and `WalkDir` APIs.

Uploader upload file with chunks.

Writer is like io.Writer but with Context, create a new writer on top of Uploader with NewUploaderWriter.

CompressType represents the type of compression.

EffDeleteFile is the side effect of a call to `DeleteFile`.

Effect is an side effect that happens in the batch storage.

Permission represents the permission we need to check in create storage.