gcs Key-Value Store driver

The gcs driver provides access to Google Cloud Storage. Keys directly correspond to paths within a Google Cloud Storage bucket.

Conditional writes are used to safely allow concurrent access from multiple machines.

json kvstore/gcs : object

Extends

KeyValueStore

Required members

driver : "gcs"
bucket : string

Google Cloud Storage bucket to use.

The Google Cloud account that is used must have appropriate permissions on the bucket. If the bucket has Requestor Pays enabled, either additional permissions are required or a separate billing project must be specified using Context.gcs_user_project.

Optional members

context : Context

Specifies context resources that augment/override the parent context.

gcs_request_concurrency : ContextResource

Specifies or references a previously defined Context.gcs_request_concurrency.

gcs_user_project : ContextResource

Specifies or references a previously defined Context.gcs_user_project.

gcs_request_retries : ContextResource

Specifies or references a previously defined Context.gcs_request_retries.

json Context.gcs_user_project : object

Specifies a Google Cloud project to bill for Google Cloud Storage requests. If a project_id is not specified, requests are billed to the project that owns the bucket by default. For Requestor Pays buckets, however, requests without a project_id specified will fail unless the Google Cloud account has additional permissions.

Optional members

project_id : string

Google Cloud project id, e.g. "my-project". The Google Cloud account that is used must have appropriate permissions to bill to the specified project.

json Context.gcs_request_concurrency : object

Specifies a limit on the number of concurrent requests to Google Cloud Storage.

Optional members

limit : integer[1, +∞) | "shared" = "shared"

The maximum number of concurrent requests. If the special value of "shared" is specified, a shared global limit of 32 applies.

json Context.gcs_request_retries : object

Specifies retry parameters for handling transient network errors.

Optional members

max_retries : integer[1, +∞) = 32

Maximum number of attempts in the case of transient errors.

Authentication

To use the gcs driver, you can access buckets that allow public access (i.e. access by allUsers) without credentials. In order to access non-public buckets, you must specify service account credentials, which can be done through one of the following methods:

  1. Set the GOOGLE_APPLICATION_CREDENTIALS environment variable to the local path to a Google Cloud JSON credentials file.

  2. Set up Google Cloud SDK application default credentials. Install the Google Cloud SDK and run:

    gcloud auth application-default login
    

    This stores Google Cloud credentials in ~/.config/gcloud/application_default_credentials.json or $CLOUDSDK_CONFIG/application_default_credentials.json.

    This is often the most convenient method to use on a development machine.

  3. On Google Compute Engine (GCE), the default service account credentials are retrieved automatically from the metadata service if credentials are not otherwise specified.

TLS CA certificates

TensorStore connects to the Google Cloud Storage API using HTTP and depends on the system certificate authority (CA) store to secure connections. In many cases it will work by default without any additional configuration, but if you receive an error like:

CURL error[77] Problem with the SSL CA cert (path? access rights?):
error setting certificate verify locations:
  CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none

refer to the HTTP request-related environment variables section for information on how to specify the path to the system certificate store at runtime.

Testing

To test the gcs driver with a fake Google Cloud Storage server, such as fake-gcs-server, you can set the TENSORSTORE_GCS_HTTP_URL environment variable to e.g. http://localhost:4443.