gcs Key-Value Store driver

The gcs driver provides access to Google Cloud Storage. Keys directly correspond to paths within a Google Cloud Storage bucket.

Conditional writes are used to safely allow concurrent access from multiple machines.

gcs Key-Value Store JSON Schema

object with members:

driver
Required
“gcs”
context

Specifies context resources that augment/override the parent context.

bucket
Required
string
Google Cloud Storage bucket to use.

The Google Cloud account that is used must have appropriate permissions on the bucket. If the bucket has Requestor Pays enabled, either additional permissions are required or a separate billing project must be specified using gcs_user_project context resource.

gcs_request_concurrency

Specifies or references a previously defined gcs_request_concurrency context resource.

gcs_user_project

Specifies or references a previously defined gcs_user_project context resource.

gcs_request_retries

Specifies or references a previously defined gcs_request_retries context resource.

gcs_user_project context resource JSON Schema

Specifies a Google Cloud project to bill for Google Cloud Storage requests. If a project_id is not specified, requests are billed to the project that owns the bucket by default. For Requestor Pays buckets, however, requests without a project_id specified will fail unless the Google Cloud account has additional permissions.

object with members:

project_id
string
Google Cloud project id, e.g. “my-project”. The Google Cloud account that is used must have appropriate permissions to bill to the specified project.

gcs_request_concurrency context resource JSON Schema

Specifies a limit on the number of concurrent requests to Google Cloud Storage.

object with members:

limit
integer [1, ∞) | “shared” (default is “shared”)

The maximum number of concurrent requests. If the special value of “shared” is specified, a shared global limit of 32 applies.

gcs_request_retries context resource JSON Schema

Specifies retry parameters for handling transient network errors.

object with members:

max_retries
integer [1, ∞) (default is 32)

Maximum number of attempts in the case of transient errors.

Authentication

To use the gcs driver, you must specify service account credentials, which can be done through one of the following methods:

  1. Set the GOOGLE_APPLICATION_CREDENTIALS environment variable to the local path to a Google Cloud JSON credentials file.

  2. Set up Google Cloud SDK application default credentials. Install the Google Cloud SDK and run:

    gcloud auth application-default login
    

    This stores Google Cloud credentials in ~/.config/gcloud/application_default_credentials.json or $CLOUDSDK_CONFIG/application_default_credentials.json.

    This is often the most convenient method to use on a development machine.

  3. On Google Compute Engine (GCE), the default service account credentials are retrieved automatically from the metadata service if credentials are not otherwise specified.

TLS CA certificates

TensorStore connects to the Google Cloud Storage API using HTTP and depends on the system certificate authority (CA) store to secure connections. In many cases it will work by default without any additional configuration, but if you receive an error like:

CURL error[77] Problem with the SSL CA cert (path? access rights?):
error setting certificate verify locations:
  CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none

refer to the HTTP request-related environment variables section for information on how to specify the path to the system certificate store at runtime.