Drivers

Chunked storage drivers

json ChunkedTensorStoreKvStoreAdapter : object

Common options supported by all chunked storage drivers.

Extends:
Subtypes:
Required members:
driver : string

Driver identifier

Specifies the TensorStore driver.

kvstore : KvStore | KvStoreUrl

Base key-value store for the TensorStore.

Optional members:
context : Context

Specifies context resources that augment/override the parent context.

dtype : dtype

Specifies the data type.

rank : integer[0, 32]

Specifies the rank of the TensorStore.

If transform is also specified, the input rank must match. Otherwise, the rank constraint applies to the driver directly.

transform : IndexTransform

Specifies a transform.

schema : Schema

Specifies constraints on the schema.

When opening an existing array, specifies constraints on the existing schema; opening will fail if the constraints do not match. Any soft constraints specified in the chunk_layout are ignored. When creating a new array, a suitable schema will be selected automatically based on the specified schema constraints in combination with any driver-specific constraints.

path : string = ""

Additional path relative to kvstore.

This is joined as an additional "/"-separated path component after any path member directly within kvstore. This is supported for backwards compatibility only; the KvStore.path member should be used instead.

Example

"path/to/data"
cache_pool : ContextResource = "cache_pool"

Cache pool for data.

Specifies or references a previously defined Context.cache_pool. It is normally more convenient to specify a default cache_pool in the context.

data_copy_concurrency : ContextResource = "data_copy_concurrency"

Specifies or references a previously defined Context.data_copy_concurrency. It is normally more convenient to specify a default data_copy_concurrency in the context.

recheck_cached_data = true

Time after which cached data is assumed to be fresh. Cached data older than the specified time is revalidated prior to being returned from a read operation. Partial chunk writes are always consistent regardless of the value of this option.

The default value of true means that cached data is revalidated on every read. To enable in-memory data caching, you must both specify a cache_pool with a non-zero total_bytes_limit and also specify false, "open", or an explicit time bound for recheck_cached_data.

open : boolean

Open an existing TensorStore. If neither open nor create is specified, defaults to true.

create : boolean = false

Create a new TensorStore. Specify true for both open and create to permit either opening an existing TensorStore or creating a new TensorStore if it does not already exist.

delete_existing : boolean = false

Delete any existing data at the specified path before creating a new TensorStore. Requires that create is true, and that open is false.

assume_metadata : boolean = false

Neither read nor write stored metadata. Instead, just assume any necessary metadata based on constraints in the spec, using the same defaults for any unspecified metadata as when creating a new TensorStore. The stored metadata need not even exist. Operations such as resizing that modify the stored metadata are not supported. Requires that open is true and delete_existing is false. This option takes precedence over assume_cached_metadata if that option is also specified.

Warning

This option can lead to data corruption if the assumed metadata does not match the stored metadata, or multiple concurrent writers use different assumed metadata.

assume_cached_metadata : boolean = false

Skip reading the metadata when opening. Instead, just assume any necessary metadata based on constraints in the spec, using the same defaults for any unspecified metadata as when creating a new TensorStore. The stored metadata may still be accessed by subsequent operations that need to re-validate or modify the metadata. Requires that open is true and delete_existing is false. The assume_metadata option takes precedence if also specified.

Note

Unlike the assume_metadata option, operations such as resizing that modify the stored metadata are supported (and access the stored metadata).

Warning

This option can lead to data corruption if the assumed metadata does not match the stored metadata, or multiple concurrent writers use different assumed metadata.

metadata_cache_pool : ContextResource

Cache pool for metadata only.

Specifies or references a previously defined Context.cache_pool. If not specified, defaults to the value of cache_pool.

recheck_cached_metadata : CacheRevalidationBound = "open"

Time after which cached metadata is assumed to be fresh. Cached metadata older than the specified time is revalidated prior to use. The metadata is used to check the bounds of every read or write operation.

Specifying true means that the metadata will be revalidated prior to every read or write operation. With the default value of "open", any cached metadata is revalidated when the TensorStore is opened but is not rechecked for each read or write operation.

fill_missing_data_reads = true

Replace missing chunks with the fill value when reading.

If disabled, reading a missing chunk will result in an error. Note that the fill value may still be used when writing a partial chunk. Typically this should only be set to false in the case that store_data_equal_to_fill_value was enabled when writing.

store_data_equal_to_fill_value = false

Store all explicitly written data, even if it is equal to the fill value.

This ensures that explicitly written data, even if it is equal to the fill value, can be distinguished from missing data. If disabled, chunks equal to the fill value may be represented as missing chunks.

json CacheRevalidationBound : true | false | "open" | number

Determines under what circumstances cached data is revalidated.

One of:
true

Revalidate cached data at every operation.

false

Assume cached data is always fresh and never revalidate.

"open"

Revalidate cached data older than the time at which the TensorStore was opened.

number

Revalidate cached data older than the specified time in seconds since the unix epoch.

Single file drivers

json TensorStoreKvStoreAdapter : object

Specifies a TensorStore stored using a base key-value store.

Extends:

  • TensorStore — Specifies a TensorStore to open/create.

Subtypes:
Required members:
driver : string

Driver identifier

Specifies the TensorStore driver.

kvstore : KvStore | KvStoreUrl

Base key-value store for the TensorStore.

Optional members:
context : Context

Specifies context resources that augment/override the parent context.

dtype : dtype

Specifies the data type.

rank : integer[0, 32]

Specifies the rank of the TensorStore.

If transform is also specified, the input rank must match. Otherwise, the rank constraint applies to the driver directly.

transform : IndexTransform

Specifies a transform.

schema : Schema

Specifies constraints on the schema.

When opening an existing array, specifies constraints on the existing schema; opening will fail if the constraints do not match. Any soft constraints specified in the chunk_layout are ignored. When creating a new array, a suitable schema will be selected automatically based on the specified schema constraints in combination with any driver-specific constraints.

path : string = ""

Additional path relative to kvstore.

This is joined as an additional "/"-separated path component after any path member directly within kvstore. This is supported for backwards compatibility only; the KvStore.path member should be used instead.

Example

"path/to/data"
cache_pool : ContextResource = "cache_pool"

Cache pool for data.

Specifies or references a previously defined Context.cache_pool. It is normally more convenient to specify a default cache_pool in the context.

data_copy_concurrency : ContextResource = "data_copy_concurrency"

Specifies or references a previously defined Context.data_copy_concurrency. It is normally more convenient to specify a default data_copy_concurrency in the context.

recheck_cached_data : CacheRevalidationBound = "open"

Time after which cached data is assumed to be fresh.

Cached data older than the specified time is revalidated prior to being returned from a read operation. Writes are always consistent regardless of the value of this option.

Specifying true means that the data will be revalidated prior to every read operation. With a value of "open", any cached data is revalidated when the TensorStore is opened but is not rechecked for each read operation.

Note

To enable in-memory data caching, you must both specify a cache_pool with a non-zero total_bytes_limit and also specify false, "open", or an explicit time bound for recheck_cached_data.

Adapters

json TensorStoreAdapter : object

Specifies a TensorStore that adapts a base TensorStore.

Extends:

  • TensorStore — Specifies a TensorStore to open/create.

Subtypes:
Required members:
driver : string

Driver identifier

Specifies the TensorStore driver.

base : TensorStore | KvStore | TensorStoreUrl

Base TensorStore to adapt.

If a KvStore or KvStoreUrl is specified, the auto driver is used implicitly to auto-detect the format.

Optional members:
context : Context

Specifies context resources that augment/override the parent context.

dtype : dtype

Specifies the data type.

rank : integer[0, 32]

Specifies the rank of the TensorStore.

If transform is also specified, the input rank must match. Otherwise, the rank constraint applies to the driver directly.

transform : IndexTransform

Specifies a transform.

schema : Schema

Specifies constraints on the schema.

When opening an existing array, specifies constraints on the existing schema; opening will fail if the constraints do not match. Any soft constraints specified in the chunk_layout are ignored. When creating a new array, a suitable schema will be selected automatically based on the specified schema constraints in combination with any driver-specific constraints.