Drivers

Single file drivers

Chunked storage drivers

Chunked storage driver JSON Schema

Common options supported by all chunked storage drivers.

object with members:

driver
Required
string
Driver identifier

Specifies the TensorStore driver.

context

Specifies context resources that augment/override the parent context.

dtype

Specifies the data type.

rank
integer [0, ∞)
transform

Specifies a transform.

kvstore
Required

Specifies the underlying storage mechanism.

path
string (default is “”)

Path within the Key-Value Store specified by kvstore.

Example
"path/to/data"
open
boolean

Open an existing TensorStore. If neither open nor create is specified, defaults to true.

create
boolean (default is false)

Create a new TensorStore. Specify true for both open and create to permit either opening an existing TensorStore or creating a new TensorStore if it does not already exist.

delete_existing
boolean (default is false)

Delete any existing data at the specified path before creating a new TensorStore. Requires that create is true, and that open is false.

allow_metadata_mismatch
boolean (default is false)

Allow a mismatch between the existing metadata and the metadata specified for creating a new TensorStore. Requires that create is true.

cache_pool
Context resource (default is “cache_pool”)

Specifies or references a previously defined cache_pool context resource. It is normally more convenient to specify a default cache_pool in the context.

data_copy_concurrency
Context resource (default is “data_copy_concurrency”)

Specifies or references a previously defined data_copy_concurrency context resource. It is normally more convenient to specify a default data_copy_concurrency in the context.

recheck_cached_metadata
Cache revalidation bound (default is “open”)

Time after which cached metadata is assumed to be fresh. Cached metadata older than the specified time is revalidated prior to use. The metadata is used to check the bounds of every read or write operation.

Specifying true means that the metadata will be revalidated prior to every read or write operation. With the default value of “open”, any cached metadata is revalidated when the TensorStore is opened but is not rechecked for each read or write operation.

recheck_cached_data
Cache revalidation bound (default is true)

Time after which cached data is assumed to be fresh. Cached data older than the specified time is revalidated prior to being returned from a read operation. Partial chunk writes are always consistent regardless of the value of this option.

The default value of true means that cached data is revalidated on every read. To enable in-memory data caching, you must both specify a cache_pool context resource with a non-zero total_bytes_limit and also specify false, “open”, or an explicit time bound for recheck_cached_data.

Cache revalidation bound JSON Schema

true | false | “open” | number (-∞, ∞)

Determines under what circumstances cached data is revalidated.

One of:
  • true

    Revalidate cached data at every operation.

  • false

    Assume cached data is always fresh and never revalidate.

  • “open”

    Revalidate cached data older than the time at which the TensorStore was opened.

  • number (-∞, ∞)

    Revalidate cached data older than the specified time in seconds since the unix epoch.