neuroglancer_precomputed
Driver¶
The neuroglancer_precomputed
driver provides access to Neuroglancer Precomputed format
volumes backed by any supported Key-Value Storage Layer. It supports reading,
writing, and creating new volumes.
Multiscale volumes are supported, but each scale must be opened individually.
- json driver/neuroglancer_precomputed : object¶
- Extends:¶
- Required members:¶
-
driver :
"neuroglancer_precomputed"
¶
- kvstore : KvStore | KvStoreUrl¶
Specifies the underlying storage mechanism.
-
driver :
- Optional members:¶
-
-
rank : integer[
0
,32
]¶ Specifies the rank of the TensorStore.
If
transform
is also specified, the input rank must match. Otherwise, the rank constraint applies to the driver directly.
- transform : IndexTransform¶
Specifies a transform.
- schema : Schema¶
Specifies constraints on the schema.
When opening an existing array, specifies constraints on the existing schema; opening will fail if the constraints do not match. Any soft constraints specified in the
chunk_layout
are ignored. When creating a new array, a suitable schema will be selected automatically based on the specified schema constraints in combination with any driver-specific constraints.
-
path : string =
""
¶ -
This is joined as an additional
"/"
-separated path component after anypath
member directly withinkvstore
. This is supported for backwards compatibility only; theKvStore.path
member should be used instead.Example
"path/to/data"
- open : boolean¶
Open an existing TensorStore. If neither
open
norcreate
is specified, defaults totrue
.
-
create : boolean =
false
¶ Create a new TensorStore. Specify
true
for bothopen
andcreate
to permit either opening an existing TensorStore or creating a new TensorStore if it does not already exist.
-
delete_existing : boolean =
false
¶ Delete any existing data at the specified path before creating a new TensorStore. Requires that
create
istrue
, and thatopen
isfalse
.
-
assume_metadata : boolean =
false
¶ Neither read nor write stored metadata. Instead, just assume any necessary metadata based on constraints in the spec, using the same defaults for any unspecified metadata as when creating a new TensorStore. The stored metadata need not even exist. Operations such as resizing that modify the stored metadata are not supported. Requires that
open
istrue
anddelete_existing
isfalse
. This option takes precedence overassume_cached_metadata
if that option is also specified.Warning
This option can lead to data corruption if the assumed metadata does not match the stored metadata, or multiple concurrent writers use different assumed metadata.
-
assume_cached_metadata : boolean =
false
¶ Skip reading the metadata when opening. Instead, just assume any necessary metadata based on constraints in the spec, using the same defaults for any unspecified metadata as when creating a new TensorStore. The stored metadata may still be accessed by subsequent operations that need to re-validate or modify the metadata. Requires that
open
istrue
anddelete_existing
isfalse
. Theassume_metadata
option takes precedence if also specified.Note
Unlike the
assume_metadata
option, operations such as resizing that modify the stored metadata are supported (and access the stored metadata).Warning
This option can lead to data corruption if the assumed metadata does not match the stored metadata, or multiple concurrent writers use different assumed metadata.
-
cache_pool : ContextResource =
"cache_pool"
¶ Cache pool for data.
Specifies or references a previously defined
Context.cache_pool
. It is normally more convenient to specify a defaultcache_pool
in thecontext
.
- metadata_cache_pool : ContextResource¶
Cache pool for metadata only.
Specifies or references a previously defined
Context.cache_pool
. If not specified, defaults to the value ofcache_pool
.
-
data_copy_concurrency : ContextResource =
"data_copy_concurrency"
¶ Specifies or references a previously defined
Context.data_copy_concurrency
. It is normally more convenient to specify a defaultdata_copy_concurrency
in thecontext
.
-
recheck_cached_metadata : CacheRevalidationBound =
"open"
¶ Time after which cached metadata is assumed to be fresh. Cached metadata older than the specified time is revalidated prior to use. The metadata is used to check the bounds of every read or write operation.
Specifying
true
means that the metadata will be revalidated prior to every read or write operation. With the default value of"open"
, any cached metadata is revalidated when the TensorStore is opened but is not rechecked for each read or write operation.
-
recheck_cached_data : CacheRevalidationBound =
true
¶ Time after which cached data is assumed to be fresh. Cached data older than the specified time is revalidated prior to being returned from a read operation. Partial chunk writes are always consistent regardless of the value of this option.
The default value of
true
means that cached data is revalidated on every read. To enable in-memory data caching, you must both specify acache_pool
with a non-zerototal_bytes_limit
and also specifyfalse
,"open"
, or an explicit time bound forrecheck_cached_data
.
-
scale_index : integer[
0
, +∞)¶ Zero-based index of the scale to open or create.
When opening an existing volume, this member may be specified to explicitly indicate the scale to open. Otherwise, the first scale matching the
resolution
constraint, is chosen. To create a new scale, this must either be left unspecified or equal the number of existing scales (which is also the index that will be assigned to the new scale).
- multiscale_metadata : object¶
Scale-independent metadata.
Specifies the scale-independent metadata of a new volume exactly as in the info file, except that not all members are required. Required when creating a new multiscale volume. When opening an existing volume or creating a new scale within an existing multiscale volume, specifies constraints on the existing metadata.
- Optional members:¶
-
type :
"image"
|"segmentation"
¶ Specifying the type of volume.
This has no bearing on TensorStore, but is used by Neuroglancer to pick the default layer type to use. Required when creating a new multiscale volume.
-
data_type :
"uint8"
|"uint16"
|"uint32"
|"uint64"
|"float32"
¶ Specifies the data type.
Required when creating a new multiscale volume.
- num_channels : integer¶
Number of channels.
Required when creating a new volume.
-
type :
- scale_metadata : object¶
Per-scale metadata.
Specifies the per-scale metadata of a new volume as in the info file, except that not all members are required, and the
chunk_size
member for specifying a single chunk size takes the place of thechunk_sizes
member in theinfo
file. Required when creating a new scale. When opening an existing scale, serves to select the scale to open and constrains the existing metadata.- Optional members:¶
- key : string¶
Specifies the scale key (relative to
path
).When opening an existing scale, may be specified to select the scale by
key
. When creating a new scale, if not specified, defaults to"<xres>_<yres>_<zres>"
, where[xres, yres, zres]
is theresolution
.
-
size : array[
3
] of integer[0
, +∞)¶ Voxel dimensions of the volume (XYZ order).
Required when creating a new scale if
domain
is not specified.Example
[500, 500, 500]
-
voxel_offset : array[
3
] of integer =[0, 0, 0]
¶ Voxel origin of the volume (XYZ order).
If specified,
size
must also be specified.
-
chunk_size : array[
3
] of integer[1
, +∞)¶ Chunk dimensions (XYZ order).
When creating a new scale, if not specified explicitly, will be chosen automatically based on the
chunk_layout
. When opening an existing scale with multiple supportedchunk_sizes
, may be specified to select the chunk size to use. Otherwise, the first supported chunk size is used.Example
[500, 500, 500]
-
resolution : array[
3
] of number¶ Voxel size in nanometers (XYZ order).
When opening an existing scale, may be specified to select the scale by
resolution
.
-
encoding :
"raw"
|"jpeg"
|"png"
|"compressed_segmentation"
¶ Specifies the chunk encoding.
Required when creating a new scale.
-
jpeg_quality : integer[
0
,100
] =75
¶ JPEG encoding quality.
Only applies if
encoding
is"jpeg"
. The quality is specified using the IJG (Independent JPEG Group) [0, 100] recommended scale, with 0 having the worst quality (smallest file size) and 100 the best quality (largest file size).
-
png_level : integer[
0
,9
]¶ PNG compression level.
Only applies if
encoding
is"png"
. Specifies the zlib compression level between [0, 9], where 0 is uncompressed, with 1 having the fastest compression (largest file size), and 9 the slowest compression (smallest file size). When unset, the library default compression level is used.
-
compressed_segmentation_block_size : array[
3
] of number¶ Block size for compressed segmentation encoding.
If
encoding
is"compressed_segmentation"
, required when creating a new scale, and serves as a constraint when opening an existing scale. Otherwise, must not be specified.
-
sharding : kvstore/neuroglancer_uint64_sharded/ShardingSpec |
null
¶ Specifies to use the unsharded (indicated by
null
) or sharded format. When creating a new scale, if not specified, the unsharded format is used.
-
rank : integer[
Example JSON specifications¶
{
"driver": "neuroglancer_precomputed",
"kvstore": "gs://my-bucket/path/to/volume/",
}
{
"driver": "neuroglancer_precomputed",
"kvstore": "gs://my-bucket/path/to/volume/",
"scale_index": 1
}
{
"driver": "neuroglancer_precomputed",
"kvstore": "gs://my-bucket/path/to/volume/",
"scale_metadata": {
"resolution": [4, 4, 40]
}
}
{
"driver": "neuroglancer_precomputed",
"kvstore": "gs://my-bucket/path/to/volume/",
"schema": {
"dimension_units": ["4nm", "4nm", "40nm", null]
}
}
{
"driver": "neuroglancer_precomputed",
"kvstore": "gs://my-bucket/path/to/volume/",
"scale_metadata": {
"key": "4_4_40"
}
}
{
"driver": "neuroglancer_precomputed",
"kvstore": "gs://my-bucket/path/to/volume/",
"scale_metadata": {
"size": [40000, 50000, 10000],
"encoding": "compressed_segmentation",
"compressed_segmentation_block_size": [8, 8, 8],
"chunk_size": [64, 64, 64],
"resolution": [8, 8, 40]
}
}
{
"driver": "neuroglancer_precomputed",
"kvstore": "gs://my-bucket/path/to/volume/",
"multiscale_metadata": {
"type": "segmentation",
"data_type": "uint64",
"num_channels": 1
},
"scale_metadata": {
"size": [40000, 50000, 10000],
"encoding": "compressed_segmentation",
"compressed_segmentation_block_size": [8, 8, 8],
"chunk_size": [64, 64, 64],
"resolution": [8, 8, 40]
}
}
Mapping to TensorStore Schema¶
Example with unsharded raw encoding
For the following info metadata file:
{
"@type": "neuroglancer_multiscale_volume",
"data_type": "uint8",
"num_channels": 2,
"scales": [ {
"chunk_sizes": [[100, 200, 300]],
"encoding": "raw",
"key": "8_8_8",
"resolution": [8.0, 8.0, 8.0],
"size": [1000, 2000, 3000],
"voxel_offset": [20, 30, 40]
}],
"type": "image"
}
the corresponding Schema
is:
{
"chunk_layout": {
"grid_origin": [20, 30, 40, 0],
"inner_order": [3, 2, 1, 0],
"read_chunk": {"shape": [100, 200, 300, 2]},
"write_chunk": {"shape": [100, 200, 300, 2]}
},
"codec": {"driver": "neuroglancer_precomputed", "encoding": "raw"},
"dimension_units": [[8.0, "nm"], [8.0, "nm"], [8.0, "nm"], null],
"domain": {
"exclusive_max": [1020, 2030, 3040, 2],
"inclusive_min": [20, 30, 40, 0],
"labels": ["x", "y", "z", "channel"]
},
"dtype": "uint8",
"rank": 4
}
Example with unsharded compressed segmentation encoding
For the following info metadata file:
{
"@type": "neuroglancer_multiscale_volume",
"data_type": "uint64",
"num_channels": 2,
"scales": [ {
"chunk_sizes": [[100, 200, 300]],
"compressed_segmentation_block_size": [8, 8, 8],
"encoding": "compressed_segmentation",
"key": "8_8_8",
"resolution": [8.0, 8.0, 8.0],
"size": [1000, 2000, 3000],
"voxel_offset": [20, 30, 40]
}],
"type": "segmentation"
}
the corresponding Schema
is:
{
"chunk_layout": {
"codec_chunk": {"shape": [8, 8, 8, 1]},
"grid_origin": [20, 30, 40, 0],
"inner_order": [3, 2, 1, 0],
"read_chunk": {"shape": [100, 200, 300, 2]},
"write_chunk": {"shape": [100, 200, 300, 2]}
},
"codec": {"driver": "neuroglancer_precomputed", "encoding": "compressed_segmentation"},
"dimension_units": [[8.0, "nm"], [8.0, "nm"], [8.0, "nm"], null],
"domain": {
"exclusive_max": [1020, 2030, 3040, 2],
"inclusive_min": [20, 30, 40, 0],
"labels": ["x", "y", "z", "channel"]
},
"dtype": "uint64",
"rank": 4
}
Example with sharded raw encoding
For the following info metadata file:
{
"@type": "neuroglancer_multiscale_volume",
"data_type": "uint8",
"num_channels": 2,
"scales": [ {
"chunk_sizes": [[64, 64, 64]],
"encoding": "raw",
"key": "8_8_8",
"resolution": [8.0, 8.0, 8.0],
"sharding": {
"@type": "neuroglancer_uint64_sharded_v1",
"data_encoding": "gzip",
"hash": "identity",
"minishard_bits": 6,
"minishard_index_encoding": "gzip",
"preshift_bits": 9,
"shard_bits": 15
},
"size": [34432, 39552, 51508],
"voxel_offset": [20, 30, 40]
}],
"type": "image"
}
the corresponding Schema
is:
{
"chunk_layout": {
"grid_origin": [20, 30, 40, 0],
"inner_order": [3, 2, 1, 0],
"read_chunk": {"shape": [64, 64, 64, 2]},
"write_chunk": {"shape": [2048, 2048, 2048, 2]}
},
"codec": {
"driver": "neuroglancer_precomputed",
"encoding": "raw",
"shard_data_encoding": "gzip"
},
"dimension_units": [[8.0, "nm"], [8.0, "nm"], [8.0, "nm"], null],
"domain": {
"exclusive_max": [34452, 39582, 51548, 2],
"inclusive_min": [20, 30, 40, 0],
"labels": ["x", "y", "z", "channel"]
},
"dtype": "uint8",
"rank": 4
}
Data type¶
The neuroglancer_precomputed
data types map to TensorStore data types of the
same name:
Note that internally the neuroglancer_precomputed
format always uses little
endian encoding.
Domain¶
The Schema.domain
has a rank of 4 and includes both the chunked
dimensions as well as the channel dimension. The
IndexDomain.shape
of the Schema.domain
corresponds
to driver/neuroglancer_precomputed.scale_metadata.size
;
IndexDomain.inclusive_min
corresponds to
driver/neuroglancer_precomputed.scale_metadata.voxel_offset
. The
channel dimension always has an origin of 0.
The dimension labels are always ["x", "y", "z",
"channel"]
.
The bounds are not resizable.
Example
For the following info metadata file:
{
"@type": "neuroglancer_multiscale_volume",
"data_type": "uint8",
"num_channels": 2,
"scales": [ {
"chunk_sizes": [[64, 64, 64]],
"encoding": "raw",
"key": "8_8_8",
"resolution": [8.0, 8.0, 8.0],
"size": [1000, 2000, 3000],
"voxel_offset": [20, 30, 40]
}],
"type": "image"
}
the corresponding IndexDomain
is:
{
"exclusive_max": [1020, 2030, 3040, 2],
"inclusive_min": [20, 30, 40, 0],
"labels": ["x", "y", "z", "channel"]
}
Chunk layout¶
The ChunkLayout.read_chunk
shape, specifying the granularity at
which reads may be performed efficiently, corresponds to
driver/neuroglancer_precomputed.scale_metadata.chunk_size
. The
channel
dimension is always unchunked, i.e. the chunk size is equal to the
number of channels.
The ChunkLayout.grid_origin
always matches the origin of the
domain
.
With respect to the dimension order of [x, y, z, channel]
, when using the
"raw"
driver/neuroglancer_precomputed/Codec.encoding
, the
neuroglancer_precomputed
format always stores the data within chunks in
colexicographic order (i.e. Fortran order).
When using the unsharded format, the ChunkLayout.write_chunk
shape is equal to the ChunkLayout.read_chunk
shape.
When using the sharded format, if the sharding parameters are such that each
shard corresponds to a rectangular region, then the
ChunkLayout.write_chunk
shape corresponds to the shape of each
shard. Otherwise, the write_chunk
shape corresponds
to the shape of the entire volume, rounded up to a multiple of the
read_chunk
shape.
When using the "raw"
, "png"
, or "jpeg"
driver/neuroglancer_precomputed/Codec.encoding
, hard constraints
on the ChunkLayout.codec_chunk
must not be specified.
When using the "compressed_segmentation"
driver/neuroglancer_precomputed/Codec.encoding
, the
ChunkLayout.codec_chunk
shape corresponds to the
compressed_segmentation_block_size
.
Note that the codec chunk size along the channel dimension is always 1.
Selection of chunk layout when creating a new array¶
When creating a new array, the read chunk shape may be constrained explicitly
via ChunkLayout/Grid.shape
or implicitly via
ChunkLayout/Grid.aspect_ratio
and
ChunkLayout/Grid.elements
. If
ChunkLayout/Grid.elements
is not specified for the
read_chunk
, the default is 1 million elements per
chunk. A suitable base (read) chunk shape will be chosen based on these
constraints.
The write chunk shape may also be constrained separately, either explicitly via
ChunkLayout/Grid.shape
or implicitly via
ChunkLayout/Grid.elements
. If no constraints on the write chunk
shape are specified, it will be set to match the read chunk shape and the
unsharded format will be used. Otherwise, suitable sharding parameters will be
chosen automatically to satisfy the constraints. If
ChunkLayout/Grid.elements
is not specified for the write chunk,
the unsharded format is used. Due to the limitations of the sharding format,
any ChunkLayout/Grid.aspect_ratio
constraints on the write chunk
shape are ignored.
When using the "compressed_segmentation"
driver/neuroglancer_precomputed/Codec.encoding
, the compressed
segmentation block size is chosen automatically based on the
ChunkLayout.codec_chunk
constraints. Both explicit constraints
via ChunkLayout/Grid.shape
and implicit constraints via
ChunkLayout/Grid.aspect_ratio
and
ChunkLayout/Grid.elements
are supported. If
ChunkLayout/Grid.elements
is not specified, the default is 512
elements per chunk.
Example of unconstrained chunk layout
>>> ts.open(
... {
... 'driver': 'neuroglancer_precomputed',
... 'kvstore': {
... 'driver': 'memory'
... }
... },
... create=True,
... dtype=ts.uint16,
... domain=ts.IndexDomain(
... inclusive_min=[20, 30, 40, 0],
... shape=[1000, 2000, 3000, 2],
... )).result().chunk_layout
ChunkLayout({
'grid_origin': [20, 30, 40, 0],
'inner_order': [3, 2, 1, 0],
'read_chunk': {'shape': [80, 80, 80, 2]},
'write_chunk': {'shape': [80, 80, 80, 2]},
})
Example of unconstrained chunk layout with compressed segmentation encoding
>>> ts.open(
... {
... 'driver': 'neuroglancer_precomputed',
... 'kvstore': {
... 'driver': 'memory'
... }
... },
... create=True,
... dtype=ts.uint32,
... codec=ts.CodecSpec({
... 'driver': 'neuroglancer_precomputed',
... 'encoding': 'compressed_segmentation',
... }),
... domain=ts.IndexDomain(
... inclusive_min=[20, 30, 40, 0],
... shape=[1000, 2000, 3000, 2],
... )).result().chunk_layout
ChunkLayout({
'codec_chunk': {'shape': [8, 8, 8, 1]},
'grid_origin': [20, 30, 40, 0],
'inner_order': [3, 2, 1, 0],
'read_chunk': {'shape': [80, 80, 80, 2]},
'write_chunk': {'shape': [80, 80, 80, 2]},
})
Example of chunk layout with separate read and write chunk constraints
>>> ts.open(
... {
... 'driver': 'neuroglancer_precomputed',
... 'kvstore': {
... 'driver': 'memory'
... }
... },
... create=True,
... dtype=ts.uint16,
... chunk_layout=ts.ChunkLayout(
... chunk_aspect_ratio=[2, 1, 1, 0],
... read_chunk_elements=2000000,
... write_chunk_elements=1000000000,
... ),
... domain=ts.IndexDomain(
... inclusive_min=[20, 30, 40, 0],
... shape=[1000, 2000, 3000, 2],
... )).result().chunk_layout
ChunkLayout({
'grid_origin': [20, 30, 40, 0],
'inner_order': [3, 2, 1, 0],
'read_chunk': {'shape': [159, 79, 79, 2]},
'write_chunk': {'shape': [1113, 1264, 632, 2]},
})
Example of chunk layout with explicit chunk shapes
>>> ts.open(
... {
... 'driver': 'neuroglancer_precomputed',
... 'kvstore': {
... 'driver': 'memory'
... }
... },
... create=True,
... dtype=ts.uint16,
... chunk_layout=ts.ChunkLayout(
... read_chunk_shape=[64, 64, 64, 2],
... write_chunk_shape=[512, 512, 512, 2],
... ),
... domain=ts.IndexDomain(
... inclusive_min=[20, 30, 40, 0],
... shape=[1000, 2000, 3000, 2],
... )).result().chunk_layout
ChunkLayout({
'grid_origin': [20, 30, 40, 0],
'inner_order': [3, 2, 1, 0],
'read_chunk': {'shape': [64, 64, 64, 2]},
'write_chunk': {'shape': [512, 512, 512, 2]},
})
Codec¶
Within the Schema.codec
, the compression parameters are
represented in the same way as in the
scale_metadata
:
- json driver/neuroglancer_precomputed/Codec : object¶
Neuroglancer Precomputed Codec
- Optional members:¶
-
encoding :
"raw"
|"jpeg"
|"png"
|"compressed_segmentation"
¶ Specifies the chunk encoding.
Required when creating a new scale.
-
jpeg_quality : integer[
0
,100
] =75
¶ JPEG encoding quality.
Only applies if
encoding
is"jpeg"
. The quality is specified using the IJG (Independent JPEG Group) [0, 100] recommended scale, with 0 having the worst quality (smallest file size) and 100 the best quality (largest file size).
-
png_level : integer[
0
,9
]¶ PNG compression level.
Only applies if
encoding
is"png"
. Specifies the zlib compression level between [0, 9], where 0 is uncompressed, with 1 having the fastest compression (largest file size), and 9 the slowest compression (smallest file size). When unset, the library default compression level is used.
-
shard_data_encoding :
"raw"
|"gzip"
¶ Additional data compression when using the sharded format.
This specifies the value of
kvstore/neuroglancer_uint64_sharded/ShardingSpec.data_encoding
. If not specified, defaults to"gzip"
if theencoding
is equal to"raw"
or"compressed_segmentation"
, and to"raw"
ifencoding
is equal to"jpeg"
.
-
encoding :
It is an error to specify any other Codec.driver
.
Fill value¶
The neuroglancer_precomputed
format does not support specifying a fill
value. TensorStore always assumes a fill value of 0
.
Dimension Units¶
The dimension units of the first three (x
, y
, and z
) dimensions
always have a base unit of "nm"
; the multiplier corresponds to the
resolution
. It
is an error to specify a base unit other than "nm"
for these dimensions.
The final (channel
) dimension always has an unspecified base unit. It is an
error to specify a unit for this dimension.
When creating a new scale, if neither dimension_units
nor
resolution
is
specified, a unit of "1nm"
is used by default.
When opening an existing scale, the scale to open may be selected based on the
specified dimension_units
.
Limitations¶
Resizing is not supported.
Warning
Writing to volumes in the sharded format is supported, but because updating a shard requires rewriting it entirely, write operations may be very inefficient unless special care is taken:
It is most efficient to group writes by shard (i.e. according to the
ChunkLayout.write_chunk
shape).The temporary memory required to write a shard is 2 to 3 times the size of the shard. It is therefore advised that the shards be kept as small as possible (while still avoiding an excess number of objects in the underlying key-value store).