#include "tensorstore/chunk_layout.h"
class tensorstore::ChunkLayout;

Specifies a precise chunk layout or constraints on a chunk layout.

A chunk layout is a hierarchical, regular grid defined over an index space.

At the coarsest level, a chunk layout specifies a chunk grid to which writes should be aligned. Each write chunk may then be optionally further subdivided by a grid of read chunks. Each read chunk may then be optionally further subdivided by a grid of codec chunks. Refer to the Chunk layout documentation for details.

A precise chunk layout is defined by:

  • The grid_origin, specifying the origin of the top-level (write) grid. There is no separate origin for the read grid; it is assumed to be aligned to the start of each write chunk. There is also no separate origin for the codec grid; while it may not be aligned to the start of each read - chunk, the offset, if any, is not specified by the chunk layout.

  • The write_chunk_shape, and read_chunk_shape, and codec_chunk_shape.

  • The optional inner_order permutation specifying the data storage order (e.g. C order or Fortran order) within read chunks.

The precise chunk layout may also be left unspecified, and several types of hard and soft constraints on a chunk layout may be specified instead. When creating a new TensorStore, the driver determines the precise chunk layout automatically based on the specified constraints as well as any driver-specific requirements:

  • Each dimension of grid_origin may be left unspecified, or specified as a soft constraint rather than a hard constraint.

  • Each dimension of the write_chunk_shape, read_chunk_shape, or codec_chunk_shape may be left unspecified, or specified as a soft constraint.

  • Instead of specifying a precise write/read/codec chunk shape, an aspect ratio and target number of elements per chunk may be specified.

  • The inner_order permutation may be specified as a soft constraint rather than a hard constraint.

Hard and soft constraints compose as follows:

  • When specifying a new soft constraint:

    • If there is no existing hard or soft constraint, the specified value is set as a soft constraint.

    • If there is already a hard or soft constraint, the new value is ignored.

  • When specifying a new hard constraint:

    • If there is no existing value, or an existing soft constraint, the specified value is set as a hard constraint.

    • If there is an existing hard constraint, it remains unchanged, and an error is returned if it does not match the new hard constraint.

The inner_order constraint is set as an atomic value, even though it is represented as a permutation vector.

Constraints are set using the Set overloads. For example:

tensorstore::ChunkLayout layout;
TENSORSTORE_RETURN_IF_ERROR(layout.Set(
    tensorstore::ChunkLayout::InnerOrder({0, 2, 1})));
TENSORSTORE_RETURN_IF_ERROR(layout.Set(
    tensorstore::ChunkLayout::InnerOrder(
        {0, 2, 1}, /*hard_constraint=*/false)}));
TENSORSTORE_RETURN_IF_ERROR(layout.Set(
    tensorstore::ChunkLayout::ReadChunkElements(5)));
TENSORSTORE_RETURN_IF_ERROR(layout.Set(
    tensorstore::ChunkLayout::ReadChunkShape(
        {100, 200, 300})));

Types

enum Usage;

Specifies the type of operation to which a chunk grid applies.

struct ChunkElementsBase : public MaybeHardConstraintIndex;

Base type for specifying the target number of elements for a write/read/codec chunk.

struct ChunkElementsFor<U> : public ChunkElementsBase;

Target number of elements in a chunk for the specified usage U.

using ChunkElements = ChunkElementsFor<kUnspecifiedUsage>;

Target number of elements for both write and read chunks.

using WriteChunkElements = ChunkElementsFor<Usage::kWrite>;

Target number of elements for write chunks.

using ReadChunkElements = ChunkElementsFor<Usage::kRead>;

Target number of elements for read chunks.

using CodecChunkElements = ChunkElementsFor<Usage::kCodec>;

Target number of elements for codec chunks.

struct ChunkShapeBase : public MaybeHardConstraintSpan<Index>;

Base type for specifying explicit constraints on the write/read/codec chunk shape.

struct ChunkShapeFor<U> : public ChunkShapeBase;

Constrains the chunk shape for the specified usage U.

using ChunkShape = ChunkShapeFor<kUnspecifiedUsage>;

Constrains the shape for both write and read chunks.

using WriteChunkShape = ChunkShapeFor<Usage::kWrite>;

Constrains the shape for write chunks.

using ReadChunkShape = ChunkShapeFor<Usage::kRead>;

Constrains the shape for read chunks.

using CodecChunkShape = ChunkShapeFor<Usage::kCodec>;

Constrains the shape for codec chunks.

struct ChunkAspectRatioBase : public MaybeHardConstraintSpan<double>;

Base type for specifying constraints on the aspect ratio of the write/read/codec chunk shape.

struct ChunkAspectRatioFor<U> : public ChunkAspectRatioBase;

Constrains the aspect ratio of the chunk shape for the specified usage U.

class Grid;

Owned mutable representation of shape, aspect_ratio and elements constraints for a write/read/codec grid.

class GridView;

Base type for constraints applicable to a read/write/codec chunk grid.

class GridViewFor<U> : public GridView;

Strongly-typed view that provides access to parameters specific to a particular Usage.

using Chunk = GridViewFor<kUnspecifiedUsage>;
using WriteChunk = GridViewFor<Usage::kWrite>;
using ReadChunk = GridViewFor<Usage::kRead>;
using CodecChunk = GridViewFor<Usage::kCodec>;

Aliases of GridViewFor that provide access to parameters specific to a particular Usage.

struct InnerOrder : public ContiguousLayoutPermutation<>;

Specifies the data storage order within innermost chunks as a permutation of [0, ..., rank-1].

struct GridOrigin : public MaybeHardConstraintSpan<Index>;

Specifies the base origin/offset of the chunk grid.

using ChunkAspectRatio = ChunkAspectRatioFor<kUnspecifiedUsage>;
using WriteChunkAspectRatio = ChunkAspectRatioFor<Usage::kWrite>;
using ReadChunkAspectRatio = ChunkAspectRatioFor<Usage::kRead>;
using CodecChunkAspectRatio = ChunkAspectRatioFor<Usage::kCodec>;

Specifies the desired aspect ratio for all chunk shapes (codec, read, and write).

class GridViewFor<ChunkLayout::kUnspecifiedUsage> : public GridView;

Specifies chunk layout constraints for a single usage specified at run time.

Methods

DimensionIndex rank() const;

Returns the rank constraint, or dynamic_rank if unspecified.

bool HasHardConstraints() const;

Returns true if any hard constraints, other than rank, are specified.

absl::Status
GetChunkTemplate(Usage usage, MutableBoxView<> box) const;

Sets box to the precise write/read chunk template.

InnerOrder inner_order() const;
explicit operator InnerOrder() const;

Returns the inner order constraint.

absl::Status Set(InnerOrder value);

Sets the inner order constraint.

GridOrigin grid_origin() const;
explicit operator GridOrigin() const;

Returns the grid origin constraint.

absl::Status Set(GridOrigin value);

Sets/updates the grid origin constraint.

WriteChunk write_chunk() const;
explicit operator WriteChunk() const;

Returns the write chunk constraints.

ReadChunk read_chunk() const;
explicit operator ReadChunk() const;

Returns the read chunk constraints.

CodecChunk codec_chunk() const;
explicit operator CodecChunk() const;

Returns the read chunk constraints.

GridView operator[](Usage usage) const;

Returns the chunk constraints for the given usage.

absl::Status Set(const GridViewFor<U>& value);

Sets the grid constraints for the specified usage.

WriteChunkShape write_chunk_shape() const;
explicit operator WriteChunkShape() const;

Returns the write chunk shape constraints.

ReadChunkShape read_chunk_shape() const;
explicit operator ReadChunkShape() const;

Returns the read chunk shape constraints.

CodecChunkShape codec_chunk_shape() const;
explicit operator CodecChunkShape() const;

Returns the codec chunk shape constraints.

WriteChunkAspectRatio write_chunk_aspect_ratio() const;
explicit operator WriteChunkAspectRatio() const;

Returns the write chunk aspect ratio constraints.

ReadChunkAspectRatio read_chunk_aspect_ratio() const;
explicit operator ReadChunkAspectRatio() const;

Returns the read chunk aspect ratio constraints.

CodecChunkAspectRatio codec_chunk_aspect_ratio() const;
explicit operator CodecChunkAspectRatio() const;

Returns the codec chunk aspect ratio constraints.

WriteChunkElements write_chunk_elements() const;
explicit operator WriteChunkElements() const;

Returns the write chunk target number of elements constraint.

ReadChunkElements read_chunk_elements() const;
explicit operator ReadChunkElements() const;

Returns the read chunk target number of elements constraint.

CodecChunkElements codec_chunk_elements() const;
explicit operator CodecChunkElements() const;

Returns the codec chunk target number of elements constraint.

absl::Status Set(ChunkShapeFor<U> value);

Sets/updates the chunk shape constraints for the given usage U.

absl::Status Set(ChunkAspectRatioFor<U> value);

Sets/updates the chunk aspect ratio constraints for the given usage U.

absl::Status Set(ChunkElementsFor<U> value);

Sets/updates the chunk target number of elements constraint for the given usage U.

absl::Status Set(ChunkLayout value);

Merges in additional chunk layout constraints. Soft constraints of *this take precedence over soft constraints of value.

absl::Status Set(RankConstraint value);

Sets the rank.

absl::Status Finalize();

Validates and converts this layout into a precise chunk layout.

Friend functions

friend Result<ChunkLayout>
ApplyIndexTransform(IndexTransformView<> transform,
                    ChunkLayout output_constraints);

Transforms a chunk layout for the output space of a transform to a corresponding chunk layout for the input space of the transform.

friend Result<ChunkLayout>
ApplyInverseIndexTransform(IndexTransformView<> transform,
                           ChunkLayout input_constraints);

Transforms a chunk layout for the input space of a transform to a corresponding chunk layout for the output space of the transform.

friend Result<ChunkLayout>
ApplyIndexTransform(Expr&& expr, ChunkLayout constraints);

Transforms a ChunkLayout object by a DimExpression.

friend PipelineResultType<ChunkLayout, Func>
operator|(ChunkLayout layout, Func&& func);

“Pipeline” operator.

Data members

static constexpr bool IsOption<T>;

Evaluates to true for option types compatible with Set. Supported types are: