Filesystem and Metadata Structure

A Sedpack dataset is stored in a dedicated root directory containing a top-level dataset_info.json file and one subdirectory per populated split (train, test, holdout). Both metadata and data files are organized as a tree to scale to many examples without overwhelming the underlying filesystem or requiring all metadata to be loaded into memory at once.

Directory Layout

File systems degrade in performance when a single directory contains too many entries. When writing shards, PathGenerator (with defaults levels=4, max_branching=1_000, name_length=10) creates 3 nested subdirectory levels of 10-character hexadecimal names (at most 1,000 entries per directory) and a 32-character hexadecimal shard filename at the leaf (when using write_multiprocessing, each worker process also writes under its own 32-character UUID subdirectory inside the split):

tiny_aes_sedpack/
├── dataset_info.json
├── train/
│   ├── shards_list.json
│   └── 8a745b3853/
│       ├── shards_list.json
│       └── 412a5873e4/
│           ├── shards_list.json
│           └── 20fd9888f6/
│               ├── shards_list.json
│               ├── 062203904722443e82655d5b65650f68.fb
│               └── 62b497f95ff84c1184fc7f85316967e8.fb
├── test/
│   └── ...
└── holdout/
    └── ...

All file paths stored inside JSON metadata files are relative to the dataset root directory and are validated to prevent directory traversal (..).

Metadata Hierarchy

Sedpack separates dataset-wide configuration from per-directory shard listings using Pydantic models serialized as JSON. Custom JSON-serializable metadata (custom_metadata) can be attached at three levels: the dataset (Metadata), individual attributes (Attribute), and individual shards (ShardInfo).

1. Root Configuration (`dataset_info.json`)

The root dataset_info.json file (represented by DatasetInfo) is the entry point when loading a dataset and always serializes three top-level sections:

metadata (Metadata): General information about the dataset:
- description: Human-readable description (default "").
- dataset_license: License string or URL.
- dataset_version: Semantic version of the dataset (default "1.0.0").
- download_from: Source URL for downloading the dataset (default "").
- custom_metadata: Arbitrary JSON-serializable dictionary (for example, citation info or target implementation in the TinyAES dataset).
- sedpack_version: Version of Sedpack used to create the dataset (checked on load to ensure the reader is not older than the dataset).
dataset_structure (DatasetStructure): Schema and storage parameters:
- saved_data_description: List of Attribute definitions (name, dtype, shape, and custom_metadata such as global "min" and "max" trace values in TinyAES). When dtype is "bytes" or "str" and shape is (), the attribute has variable size (returned as bytes or str). When dtype is "bytes" and shape is a 1D tuple (N,), the attribute has fixed length N bytes (can be written as bytes of length N or a 1D integer array/sequence of shape (N,) with values in [0, 255], and is returned as bytes of length N). When shard_file_type is "fb", attributes are stored in this exact order.
- compression: Compression algorithm applied to shard files ("", "BZ2", "GZIP", "LZMA", "LZ4", "ZIP", "ZLIB", "ZSTD").
- examples_per_shard: Maximum number of examples written to a single shard file before opening a new one (default 256).
- shard_file_type: Storage format and file extension for shards ("fb", "npz", or "tfrec").
- hash_checksum_algorithms: Tuple of hash algorithms (default ("sha256",), also supporting md5, sha1, sha224, sha384, sha512, sha3_*, xxh32, xxh64, xxh128) computed for every file.
splits (dict[SplitT, ShardListInfo]): Maps each populated split (train, test, holdout) to a ShardListInfo pointer referencing <split>/shards_list.json.

Example dataset_info.json (abbreviated from the TinyAES tutorial dataset):

{
  "metadata": {
    "description": "SCAAML AES side-channel attacks tutorial",
    "dataset_license": "Apache License 2.0",
    "dataset_version": "1.0.0",
    "download_from": "",
    "custom_metadata": {
      "implementation": "TinyAES"
    },
    "sedpack_version": "0.1.4"
  },
  "dataset_structure": {
    "saved_data_description": [
      {
        "name": "trace1",
        "dtype": "float16",
        "shape": [80000],
        "custom_metadata": {"min": -0.42, "max": 0.39}
      },
      {
        "name": "key",
        "dtype": "uint8",
        "shape": [16],
        "custom_metadata": {}
      }
    ],
    "compression": "LZ4",
    "examples_per_shard": 256,
    "shard_file_type": "fb",
    "hash_checksum_algorithms": ["sha256"]
  },
  "splits": {
    "train": {
      "shard_list_info_file": {
        "file_path": "train/shards_list.json",
        "hash_checksums": ["201a27e3..."]
      },
      "number_of_examples": 512,
      "number_of_shards": 2
    }
  }
}

2. Shard List Pointers (`ShardListInfo`)

A ShardListInfo references a shards_list.json file without requiring it to be read immediately:

shard_list_info_file (FileInfo): Relative file_path (always named shards_list.json) and its hash_checksums.
number_of_examples: Total number of examples in the subtree rooted at that shards_list.json.
number_of_shards: Total number of shard files in the subtree.

Storing number_of_examples and number_of_shards in the pointer allows fast counting and indexing without traversing or downloading child metadata files.

3. Directory Shard Lists (`shards_list.json`)

Each directory in a split’s hierarchy contains a shards_list.json file (represented by ShardsList and serialized with exclude_defaults=True, so empty fields are omitted):

relative_path_self: Path to this shards_list.json relative to the dataset root.
number_of_examples: Sum of examples across all shards in this directory and all its subdirectories.
shard_files: List of ShardInfo objects for shard files located directly in this directory (omitted in intermediate directories that only contain subdirectories).
children_shard_lists: List of ShardListInfo pointers to shards_list.json files in immediate subdirectories (omitted in leaf directories).

Example intermediate train/shards_list.json:

{
  "relative_path_self": "train/shards_list.json",
  "number_of_examples": 512,
  "children_shard_lists": [
    {
      "shard_list_info_file": {
        "file_path": "train/8a745b3853/shards_list.json",
        "hash_checksums": ["c909e2cc..."]
      },
      "number_of_examples": 512,
      "number_of_shards": 2
    }
  ]
}

When writing examples concurrently or across multiple processes, each writer writes into its own directory subtree (<split>/<relative_path_from_split>/...) and merges its local shards_list.json files up to <split>/<relative_path_from_split>/shards_list.json on exit. Calling dataset.write_config(updated_infos=...) on the coordinator then runs merge_shard_infos to merge each worker subtree up to <split>/shards_list.json and dataset_info.json (see Parallel and Distributed Writing below).

4. Individual Shard Metadata (`ShardInfo`)

Each ShardInfo entry in a leaf shards_list.json describes one shard:

file_infos: Tuple of FileInfo objects (file_path relative to the dataset root and hash_checksums) for the shard file(s).
number_of_examples: Number of examples stored in this shard.
custom_metadata: Optional JSON-serializable dictionary attached to the shard (omitted when empty). Passing a different custom_metadata to write_example automatically closes the current shard and opens a new one. In TinyAES, each shard stores 256 traces with the same AES key and records {"key": [...]} in custom_metadata, enabling shard_filter and balanced shard iteration without reading the shard files themselves.

Example leaf train/8a745b3853/412a5873e4/20fd9888f6/shards_list.json:

{
  "relative_path_self": "train/8a745b3853/412a5873e4/20fd9888f6/shards_list.json",
  "number_of_examples": 512,
  "shard_files": [
    {
      "file_infos": [
        {
          "file_path": "train/8a745b3853/412a5873e4/20fd9888f6/062203904722443e82655d5b65650f68.fb",
          "hash_checksums": ["00fb1da6..."]
        }
      ],
      "number_of_examples": 256,
      "custom_metadata": {
        "key": [43, 126, 21, 22, 40, 174, 210, 166, 171, 247, 21, 136, 9, 207, 79, 60]
      }
    }
  ]
}

Parallel and Distributed Writing

DatasetFiller decouples in-memory example buffering from shard serialization, compression, hashing, and metadata merging:

Single-coordinator writing (dataset.filler(concurrency=4)): Each full shard is serialized, compressed, and hashed asynchronously in a ProcessPoolExecutor (concurrency=4 worker processes by default) across all shard formats ("fb", "npz", "tfrec"). Because relative_path_from_split defaults to Path(".") (empty), exiting the with block writes the leaf shards_list.json files and automatically calls dataset.write_config(...) to update all the way up to <split>/shards_list.json and dataset_info.json.
Built-in multi-process writing (dataset.write_multiprocessing): When generating examples in parallel across multiple CPU processes on one machine, dataset.write_multiprocessing assigns each worker a DatasetFiller with a unique non-empty relative_path_from_split (Path(uuid.uuid4().hex)). The user-supplied feed_writer callable receives dataset_filler: DatasetFiller as its first positional argument, followed by arbitrary *args and **kwargs unpacked from that worker’s entry in custom_arguments and custom_kwarguments. On exit, each worker updates metadata up to its own <split>/<worker_uuid>/shards_list.json, and write_multiprocessing then merges all worker ShardListInfo objects into <split>/shards_list.json and dataset_info.json:
```
def feed_worker(
    dataset_filler: DatasetFiller,
    chunk: list[ExampleT],
    split: SplitT = "train",
) -> int:
    # First argument is `dataset_filler`; remaining `*args` and `**kwargs`
    # are unpacked from `custom_arguments` and `custom_kwarguments`.
    with dataset_filler:
        for values in chunk:
            dataset_filler.write_example(values=values, split=split)
    return len(chunk)

counts = dataset.write_multiprocessing(
    feed_writer=feed_worker,
    custom_arguments=[(chunk_0,), (chunk_1,)],
    custom_kwarguments=[{"split": "train"}, {"split": "test"}],
)
```
Custom multi-worker / distributed writing (DatasetFiller with non-empty relative_path_from_split): When running custom worker processes or distributed jobs, pass a unique, non-empty relative_path_from_split (e.g., Path(f"worker_{worker_id}")) to each worker’s DatasetFiller. Exiting the with block merges and hashes that worker’s shard lists up to <split>/<relative_path_from_split>/shards_list.json without touching <split>/shards_list.json or dataset_info.json. Once all workers finish, collect filler.get_updated_infos() on the coordinator and call dataset.write_config(updated_infos=...):
```
def run_worker(
    dataset_path: Path,
    worker_id: int,
    items: list[ExampleT],
) -> list[ShardListInfo]:
    ds = Dataset(dataset_path)
    filler = DatasetFiller(
        dataset=ds,
        relative_path_from_split=Path(f"worker_{worker_id}"),
    )
    with filler:
        for item in items:
            filler.write_example(values=item, split="train")
    return filler.get_updated_infos()

# Coordinator: merge all worker ShardListInfo updates into the dataset.
updated_infos: list[ShardListInfo] = []
for worker_id, chunk in enumerate(chunks):
    updated_infos.extend(run_worker(dataset.path, worker_id, chunk))
dataset.write_config(updated_infos=updated_infos)
```
Caveats for custom multi-worker / distributed writing:
- relative_path_from_split must be non-empty for every concurrent worker even when each worker writes to a different split (e.g., one worker writing only "train" and another writing only "test"). If relative_path_from_split is left empty (Path(".")), exiting the with block calls dataset.write_config(...) and overwrites the shared root dataset_info.json, racing with other workers and losing their split metadata.
- Concurrent workers must use disjoint relative_path_from_split paths where neither path is a parent prefix of another (e.g., do not run Path("a") and Path("a/b") concurrently, since the Path("a") worker writes <split>/a/shards_list.json on exit). Sibling paths such as Path("job_1/worker_0") and Path("job_1/worker_1") are safe.
- When relative_path_from_split is non-empty, the written shards are not visible in <split>/shards_list.json or dataset_info.json until the coordinator calls dataset.write_config(updated_infos=...).

Integrity and Merkle Tree

Every FileInfo stores the hash checksums configured in dataset_structure.hash_checksum_algorithms:

Each shard file’s checksum is stored in its leaf shards_list.json.
Each child shards_list.json checksum is stored in its parent’s shards_list.json.
Each split’s root shards_list.json checksum is stored in dataset_info.json.

This forms a Merkle tree over all metadata and data files in the dataset. Recording expected = dataset.current_metadata_checksums() and calling dataset.check(hash_checksums_values=expected) verifies dataset_info.json, every shards_list.json in the tree, and every shard file on disk.

Shard File Formats

Each shard file stores up to examples_per_shard examples compressed with dataset_structure.compression in one of three formats:

"fb" (FlatBuffers): Default format. Provides zero-copy-friendly binary serialization and high-throughput concurrent iteration via the Rust backend.
"npz" (NumPy): Standard NumPy archive format.
"tfrec" (TFRecord): Protocol Buffer format used by TensorFlow (requires TensorFlow to be installed, e.g., pip install "sedpack[tf]").