n5 Driver

The n5 driver provides access to N5 arrays backed by any supported Key-Value Storage Layer. It supports reading, writing, creating new datasets, and resizing datasets.

JSON Schema

object with members:

driver
Required
“n5”
context

Specifies context resources that augment/override the parent context.

dtype

Specifies the data type.

rank
integer [0, ∞)
transform

Specifies a transform.

kvstore
Required

Specifies the underlying storage mechanism.

path
string (default is “”)

Path within the Key-Value Store specified by kvstore.

Example
"path/to/data"
open
boolean

Open an existing TensorStore. If neither open nor create is specified, defaults to true.

create
boolean (default is false)

Create a new TensorStore. Specify true for both open and create to permit either opening an existing TensorStore or creating a new TensorStore if it does not already exist.

delete_existing
boolean (default is false)

Delete any existing data at the specified path before creating a new TensorStore. Requires that create is true, and that open is false.

allow_metadata_mismatch
boolean (default is false)

Allow a mismatch between the existing metadata and the metadata specified for creating a new TensorStore. Requires that create is true.

cache_pool
Context resource (default is “cache_pool”)

Specifies or references a previously defined cache_pool context resource. It is normally more convenient to specify a default cache_pool in the context.

data_copy_concurrency
Context resource (default is “data_copy_concurrency”)

Specifies or references a previously defined data_copy_concurrency context resource. It is normally more convenient to specify a default data_copy_concurrency in the context.

recheck_cached_metadata
Cache revalidation bound (default is “open”)

Time after which cached metadata is assumed to be fresh. Cached metadata older than the specified time is revalidated prior to use. The metadata is used to check the bounds of every read or write operation.

Specifying true means that the metadata will be revalidated prior to every read or write operation. With the default value of “open”, any cached metadata is revalidated when the TensorStore is opened but is not rechecked for each read or write operation.

recheck_cached_data
Cache revalidation bound (default is true)

Time after which cached data is assumed to be fresh. Cached data older than the specified time is revalidated prior to being returned from a read operation. Partial chunk writes are always consistent regardless of the value of this option.

The default value of true means that cached data is revalidated on every read. To enable in-memory data caching, you must both specify a cache_pool context resource with a non-zero total_bytes_limit and also specify false, “open”, or an explicit time bound for recheck_cached_data.

metadata N5 array metadata.

Specifies the metdata of a dataset exactly as in the attributes.json file. Required when creating a new dataset. When opening an existing dataset, specifies constraints on the existing metadata.

object with members:

dimensions
array of integer [0, ∞)
Dimensions of the dataset.

Required when creating a new array.

Example
[500, 500, 500]
blockSize
array of integer [1, ∞)
Chunk dimensions.

Required when creating a new dataset. Must have the same length as dimensions.

Example
[64, 64, 64]
dataType
“uint8” | “uint16” | “uint32” | “uint64” | “int8” | “int16” | “int32” | “int64” | “float32” | “float64”
Specifies the data type.

Required when creating a new dataset.

compression Specifies the chunk compression method.
axes
array of string
Specifies a label for each dimension of the dataset.

Optional. If not specified when creating a new dataset, all dimensions are unlabeled (equivalent to specifying an empty string for each dimension). Labels are specified in the same order as the dimensions and blockSize properties. Note that this specifies the stored dimension labels. As with any TensorStore driver, dimension labels may also be overridden by specifying an Index transform via the transform property.

Example
["x", "y", "z"]

Compression

Compression JSON Schema

The type member identifies the compression method. The remaining members are specific to the compression method.

object with members:

type
Required
string

Identifies the compressor.

The following compression methods are supported:

raw compression JSON Schema

Chunks are encoded directly as big endian values without compression.

object with members:

type
Required
“raw”

gzip compression JSON Schema

Specifies zlib compression with a gzip or zlib header.

object with members:

type
Required
“gzip”
level
integer [-1, 9] (default is -1)
Specifies the zlib compression level to use.

Level 0 indicates no compression (fastest), while level 9 indicaets the best compression ratio (slowest). The default value of -1 indicates to use the zlib default compression level (equal to 6).

useZlib
boolean (default is false)

If true, use a zlib header. Otherwise, use a gzip header.

bzip2 compression JSON Schema

Specifies bzip2 compression.

object with members:

type
Required
“bzip2”
blockSize
integer [1, 9] (default is 9)
Specifies the bzip2 block size to use (in units of 100KB), which also determine the compression level.

xz compression JSON Schema

Specifies xz compression.

object with members:

type
Required
“xz”
preset
integer [0, 9] (default is 6)

Specifies the XZ preset level. Preset 0 corresponds to the fastest compression with the worst compression ratio, while preset 9 corresponds to the slowest compression with the best compression ratio.

blosc compression JSON Schema

Specifies Blosc compression.

object with members:

type
Required
“blosc”
cname
Required
“blosclz” | “lz4” | “lz4hc” | “snappy” | “zlib” | “zstd”

Specifies the compression method used by Blosc.

clevel
Required
integer [0, 9]
Specifies the Blosc compression level to use.

Higher values are slower but achieve a higher compression ratio.

shuffle
Required
0 | 1 | 2
One of:
  • 0
    No shuffle
  • 1
    Byte-wise shuffle
  • 2
    Bit-wise shuffle
Example
{"type": "blosc", "cname": "blosclz", "clevel": 9, "shuffle": 2}

Limitations

Datasets with varlength chunks are not supported.

The N5 specification does not define a fill value, but TensorStore assumes a fill value of 0.