gcs_grpc Key-Value Store driver

The gcs_grpc driver provides access to Google Cloud Storage using the Cloud Storage gRPC API. Keys directly correspond to paths within a Google Cloud Storage bucket.

Compared to the gcs driver, which uses the JSON/HTTP API, this driver uses gRPC, which generally offers lower latency and higher throughput, and is required in order to write to RAPID storage class buckets.

Conditional writes are used to safely allow concurrent access from multiple machines.

json kvstore/gcs_grpc : object

Read/write access to Google Cloud Storage (GCS) via gRPC.

JSON specification of the key-value store.

Extends:

  • KvStore — Key-value store specification.

Required members:
driver : "gcs_grpc"
bucket : string

Google Cloud Storage bucket to use.

The Google Cloud account that is used must have appropriate permissions on the bucket. If the bucket has Requester Pays enabled, either additional permissions are required or a separate billing project must be specified using Context.gcs_user_project.

Optional members:
path : string

Key prefix within the key-value store.

If the prefix is intended to correspond to a Unix-style directory path, it should end with "/".

context : Context

Specifies context resources that augment/override the parent context.

endpoint : string

Endpoint to use for the gRPC connection.

When unset, a default Google Cloud Storage gRPC endpoint is used. Setting this is normally only useful for testing against a fake server.

Example

"localhost:8080"

Example

"dns:///storage.googleapis.com"
num_channels : integer[0, +∞)

Number of gRPC channels to use.

Each channel corresponds to an independent connection to the service. When unset or 0, a default is chosen based on the endpoint: a single channel is used for DirectPath endpoints, which multiplex internally, and a larger number derived from the number of available CPU cores is used otherwise. Increasing this may improve throughput for workloads that issue many concurrent requests.

timeout : string

Timeout applied to each individual gRPC call.

Specified as a duration string, e.g. "5s". This should generally not be set; values less than 250ms are ignored. When unset, no per-call deadline is applied.

wait_for_connection : string

Maximum duration to wait for the gRPC channel to become ready.

Specified as a duration string, e.g. "1s". When unset, requests do not wait for the channel to become connected before being issued.

experimental_mode : "standard" | "zonal" | "non_atomic"

Experimental write mode for Google Cloud Storage objects.

Warning

This option is experimental and may change or be removed without backward compatibility.

Selects how objects are written to the bucket:

  • When omitted, TensorStore starts with standard non-appendable writes and automatically switches to atomic zonal writes if the bucket requires appendable objects.

  • "standard": Always uses direct non-appendable writes without fallback. Writes to zonal (RAPID) buckets will fail.

  • "zonal": Always uses atomic zonal writes by uploading an appendable object to a temporary key and renaming it via MoveObject. Writes to non-zonal buckets will fail.

  • "non_atomic": Exclusively for zonal (RAPID) buckets. Writes directly to the destination key as an appendable object without a temporary object or MoveObject. Concurrent readers may observe partial data during an in-flight write.

Zonal buckets do not support Requester Pays, so specifying a non-empty Context.gcs_user_project together with "zonal" or "non_atomic" is an error when the store is opened. See RAPID storage class buckets.

gcs_user_project : ContextResource

Specifies or references a previously defined Context.gcs_user_project.

gcs_request_retries : ContextResource

Specifies or references a previously defined Context.gcs_request_retries.

data_copy_concurrency : ContextResource

Specifies or references a previously defined Context.data_copy_concurrency.

json Context.gcs_user_project : object

Specifies a Google Cloud project to bill for Google Cloud Storage requests. If a project_id is not specified, requests are billed to the project that owns the bucket by default. For Requester Pays buckets, however, requests without a project_id specified will fail unless the Google Cloud account has additional permissions.

Optional members:
project_id : string

Google Cloud project id, e.g. "my-project". The Google Cloud account that is used must have appropriate permissions to bill to the specified project.

json Context.gcs_request_retries : object

Specifies retry parameters for handling transient network errors. An exponential delay is added between consecutive retry attempts. The default values are appropriate for GCS.

Optional members:
max_retries : integer[0, +∞) = 32

Maximum number of retry attempts in the case of transient errors.

initial_delay : string = "1s"

Initial backoff delay for transient errors.

max_delay : string = "32s"

Maximum backoff delay for transient errors.

json Context.data_copy_concurrency : object

Specifies a limit on the number of CPU cores used concurrently for data copying/encoding/decoding.

Optional members:
limit : integer[1, +∞) | "shared" = "shared"

The maximum number of CPU cores that may be used. If the special value of "shared" is specified, a shared global limit equal to the number of CPU cores/threads available applies.

json KvStoreUrl/gcs_grpc : string

gcs_grpc:// KvStore URL scheme

Google Cloud Storage key-value stores accessed via gRPC may be specified using the gcs_grpc://bucket/path URL syntax.

The URL syntax provides no way to specify experimental_mode, and converting a store back to a URL drops it. This only costs one extra round trip on the first write: the bucket type is detected automatically, so the zonal write path is still used when needed. See RAPID storage class buckets.

Examples

URL representation

JSON representation

"gcs_grpc://my-bucket"

 
{"driver": "gcs_grpc",
 "bucket": "my-bucket"}

"gcs_grpc://my-bucket/path/to/dataset"

 
{"driver": "gcs_grpc",
 "bucket": "my-bucket",
 "path": "path/to/dataset"}
Extends:

  • KvStoreUrl — URL representation of a key-value store.

RAPID storage class buckets

Google Cloud Storage RAPID storage class buckets (also known as zonal buckets) only support appendable objects. Because appendable objects become immediately visible and readable before finalization, writing directly to the destination key would violate single-key write atomicity. To retain atomic writes, TensorStore writes to a temporary object path within the same directory and then atomically renames the temporary object to the final destination key once the data is persisted and finalized.

Warning

The experimental_mode option is experimental and may be modified or removed in a future release without backward compatibility guarantees.

Set experimental_mode to "zonal" to select this atomic zonal write path upfront:

{
  "driver": "gcs_grpc",
  "bucket": "my-rapid-bucket",
  "experimental_mode": "zonal"
}

For zonal bucket workloads that do not require atomic writes or lack storage.objects.move and storage.objects.delete permissions, set experimental_mode to "non_atomic" to write appendable objects directly to the destination key without a temporary object or rename:

{
  "driver": "gcs_grpc",
  "bucket": "my-rapid-bucket",
  "experimental_mode": "non_atomic"
}

If experimental_mode is set to "standard", TensorStore forces the direct non-appendable write path as a hard constraint without fallback; writes to buckets requiring appendable objects will fail.

If experimental_mode is omitted, TensorStore learns the bucket type automatically on the first write without requiring bucket-level permissions: if a direct write fails because appendable objects are required, the driver automatically switches to the atomic zonal write path and remembers the mode for subsequent writes. Likewise, if the direct write succeeds, the driver remembers that the bucket is not zonal so that subsequent writes do not re-probe. An explicit mode setting is never overridden by detection.

Behavior of the atomic zonal write path

  • While a write is in progress, its temporary object (named <key>.__tmp_<hex>) is visible to List and may be deleted by a concurrent DeleteRange covering its prefix.

  • Cancelling a write does not interrupt a temporary upload or rename that is already in progress. If the write is cancelled during the upload, the temporary object is deleted once the upload completes; a cancelled write whose rename has already started may still take effect.

  • Uploads interrupted by a transient error are restarted from the beginning rather than resumed.

  • RAPID buckets do not support Requester Pays; see experimental_mode.

Additional permissions

The atomic zonal write path ("zonal" or auto-detected) needs more than the storage.objects.create permission required by the standard write path:

  • storage.objects.move, to rename the temporary object;

  • storage.objects.delete, both to overwrite an existing destination key and to remove a temporary object that was not renamed.

Workloads using "non_atomic" write directly to the target object and require only storage.objects.create. A credential scoped only to storage.objects.create and storage.objects.get works on regional buckets and with "non_atomic", but fails on the atomic zonal write path.

Orphaned temporary objects

If a process crashes between writing the temporary object and renaming it, or if deleting a temporary object fails, the temporary object remains in the bucket. Because temporary objects use the dynamic naming pattern <key>.__tmp_<hex>, Google Cloud Storage object lifecycle rules cannot match them using suffix filters. Orphaned objects can be cleaned up using DeleteRange across the enclosing range, out-of-band cleanup scripts, or bucket-level TTL expiration rules on ephemeral buckets.

Authentication

See the gcs driver authentication section for how to configure Google Cloud credentials.

Note

Unlike the HTTP-based gcs driver, which permits unauthenticated access to publicly readable buckets, the gcs_grpc driver connecting to Google Cloud Storage endpoints requires valid Google Cloud credentials even for public buckets. Insecure connections without credentials are only supported for local test endpoints (e.g. emulators).

Testing

To test the gcs_grpc driver against a local fake server or test emulator, such as storage-testbench, specify the endpoint property:

{
  "driver": "gcs_grpc",
  "bucket": "my-test-bucket",
  "endpoint": "localhost:9000"
}

When a non-Google endpoint (not ending in .googleapis.com) is specified, the driver automatically connects using insecure channel credentials.