Building and Installing

There are several ways to build and install TensorStore, depending on the intended use case.

Python API

The TensorStore Python API requires Python 3.8 or later (Python 2 is not supported).

Installation from PyPI package

The Python bindings can be installed directly from the tensorstore PyPI package using pip. It is recommended to first create a virtual environment.

To install the latest published version, use:

# Use -vv option to show progress
python3 -m pip install tensorstore -vv

Note

On Windows, you may have to use instead:

py -3 -m pip install tensorstore -vv

This is the simplest and fastest way to install the TensorStore Python bindings if you aren’t intending to make changes to the TensorStore source code.

If a pre-built binary package is available for your specific platform and Python version, it will be used and no additional build tools are required. Otherwise, the package will be built from the source distribution and the normal build dependencies are required.

Installation from local checkout

If you intend to make changes to the TensorStore source code while simultaneously using TensorStore as a dependency, you can create a virtual environment and then install from a local checkout of the git repository:

git clone https://github.com/google/tensorstore
cd tensorstore
python3 setup.py develop

This invokes Bazel to build the TensorStore C++ extension module. You must have the required build dependencies.

After making changes to the C++ source code, you must re-run:

python3 setup.py develop

to rebuild the extension module. Rebuilds are incremental and will be much faster than the initial build.

Note that while it also works to invoke python3 -m pip install -e . or python3 -m pip install ., that will result in Bazel being invoked from a temporary copy of the source tree, which prevents incremental rebuilds.

The build is affected by the following environment variables:

TENSORSTORE_BAZELISK

Path to Bazelisk script that is invoked in order to execute the build. By default the bundled bazelisk.py is used, but this environment variable allows that to be overridden in order to pass additional options, etc.

BAZELISK_HOME

Path to cache directory used by Bazelisk for downloaded Bazel versions. Defaults to a platform-specific cache directory.

TENSORSTORE_BAZEL_COMPILATION_MODE

Bazel compilation mode to use. Defaults to opt (optimized build).

TENSORSTORE_BAZEL_STARTUP_OPTIONS

Additional Bazel startup options to specify when building. Multiple options may be separated by spaces; options containing spaces or other special characters should be encoded according to Posix shell escaping rules as implemented by shlex.split().

This may be used to specify a non-standard cache directory:

TENSORSTORE_BAZEL_STARTUP_OPTIONS="--output_user_root /path/to/bazel_cache"
TENSORSTORE_BAZEL_BUILD_OPTIONS

Additional Bazel build options to specify when building. The encoding is the same as for TENSORSTORE_BAZEL_STARTUP_OPTIONS.

ARCHFLAGS

macOS only. Specifies the CPU architecture to target for cross-compilation. May be -arch x86_64 or -arch arm64. Universal2 builds (specified by -arch arm64 -arch x86_64 are not supported).

MACOSX_DEPLOYMENT_TARGET

macOS only. Specifies the minimum required macOS version to target. Must not be earlier than 10.14. If not specified, defaults to the same macOS version required by the Python binary itself, or 10.14 if later.

TENSORSTORE_PREBUILT_DIR

If specified, building is skipped, and instead setup.py expects to find the pre-built extension module in the specified directory, from a prior invocation of build_ext:

python3 setup.py build_ext -b /tmp/prebuilt
TENSORSTORE_PREBUILT_DIR=/tmp/prebuilt pip wheel .

IPython shell without installing

python3 bazelisk.py run -c opt //python/tensorstore:shell

Publishing a PyPI package

To build a source package:

python3 setup.py sdist

To build a binary package:

python3 setup.py bdist_wheel

The packages are written to the dist/ sub-directory.

C++ API

The C++ API is supported for both Bazel and CMake projects. In either case, it must be added as a dependency so that it is built from source and statically linked as part of the overall build.

Bazel integration

To add TensorStore as a dependency to an existing Bazel workspace:

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
load("@bazel_tools//tools/build_defs/repo:utils.bzl", "maybe")

maybe(
    http_archive,
    name = "com_google_tensorstore",
    strip_prefix = "tensorstore-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
    url = "https://github.com/google/tensorstore/archive/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.tar.gz",
    sha256 = "YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY",
)

Additionally, TensorStore must be built in C++17 mode. You should add the compiler flags specified in the .bazelrc file in the TensorStore repository to your dependent project’s .bazelrc.

The supported C++ toolchains are listed below.

CMake integration

To add TensorStore as a dependency to an existing CMake project:

include(FetchContent)

FetchContent_Declare(
  tensorstore
  URL "https://github.com/google/tensorstore/archive/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.tar.gz"
  URL_HASH SHA256=YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
)

# Additional FetchContent_Declare calls as needed...

FetchContent_MakeAvailable(tensorstore)

# Define a target that depends on TensorStore...

target_link_libraries(
  my_target
  PRIVATE
    tensorstore::tensorstore tensorstore::all_drivers
)

TensorStore requires that the project is built in C++17 mode.

The supported C++ toolchains and additional system requirements are listed below.

Note

Python is used to generate the CMake build rules automatically from the Bazel build rules as part of the CMake configuration step.

Supported generators

The following CMake generators are supported:

  • Ninja and Ninja Multi-Config

  • Makefile generators

  • Visual Studio generators

  • Xcode (targetting arm64 only)

The Ninja generator is recommended because it provides the fastest builds.

Third-party dependencies

By default, TensorStore’s CMake build also pulls in all of its dependencies via FetchContent and statically links them. This behavior may be overridden on a per-package basis via TENSORSTORE_USE_SYSTEM_<PACKAGE> options, which may be set on the CMake command line with the syntax -DTENSORSTORE_USE_SYSTEM_<PACKAGE>=ON.

Warning

Some combinations of system-provided and vendored dependencies can lead to symbol collisions, which can result in crashes or incorrect behavior at runtime. For example, if you specify -DTENSORSTORE_USE_SYSTEM_CURL=ON to use a system-provided CURL, which links with a system-provided ZLIB, then you should also specify -DTENSORSTORE_USE_SYSTEM_ZLIB=ON as well to ensure more than one copy of zlib is not linked into the binary.

In general it is safest to use either all system-provided dependencies, or all vendored dependencies.

Build caching

When using CMake, it is often helpful to use a build caching tool like sccache or to speed up re-builds. To enable sccache, specify -DCMAKE_{C,CXX}_COMPILER_LAUNCHER=ccache when invoking CMake.

Note

Caching is only supported by the Ninja and Makefile generators.

Development

For development of TensorStore, ensure that you have the required build dependencies.

Building the documentation

python3 bazelisk.py run //tools/docs:build_docs -- --output /tmp/tensorstore-docs

Running tests

python3 bazelisk.py test //...

Build dependencies

TensorStore is written in C++ and is compatible with the following C++ compilers:

  • GCC 10 or later (Linux)

  • Clang 8 or later (Linux)

  • Microsoft Visual Studio 2019 version 16.10 (MSVC 14.29.30037) or later

  • Clang-cl 9 or later (Windows)

  • Mingw64 GCC 12 or later (Windows); CMake only, Bazel is not supported, and lld is recommended over ld for speed.

  • Apple Xcode 11.3.1 or later (earlier versions of XCode 11 have a code generation bug related to stack alignment)

Bazel build

The Bazel build system is used automatically when building the Python API, and may also be used to build the C++ API and command-line tools. You don’t need to install Bazel manually; the included copy of bazelisk automatically downloads a suitable version for your operating system. Bazelisk requires Python 2.7 or later to run.

Note

On macOS, starting with Python 3.6, installing Python using the installer from python.org does not automatically set up Python with the SSL/TLS certificates needed by bazelisk.

If you have not already done so, you need to run the /Applications/Python 3.x/Install Certificates.command script in your Python installation directory. Refer to the documentation at /Applications/Python 3.x/ReadMe.rtf for more information.

TensorStore depends on a number of third-party libraries. By default, these dependencies are fetched and built automatically as part of the TensorStore Bazel build, which requires no additional effort.

On Linux and macOS, however, it is possible to override this behavior for a subset of these libraries and instead link to a system-provided version. This reduces the binary size, and if your system packages are kept up to date, ensures TensorStore uses up-to-date versions of these dependencies.

TENSORSTORE_SYSTEM_LIBS

To use system-provided libraries, set the TENSORSTORE_SYSTEM_LIBS environment variable to a comma-separated list of the following identifiers prior to invoking Bazel:

Required third-party libraries

Identifier

Bundled library

Version

blake3

BLAKE3

64747d48ffe9

com_google_boringssl

boringssl

098695591f3a

com_google_brotli

brotli

6d03dfbedda1

org_sourceware_bzip2

bzip2

1.0.8

com_github_cares_cares

c-ares

6654436a307a

org_blosc_cblosc

c-blosc

1.21.1

net_zlib

chromium-zlib

2d44c51ada6d325b85b53427b02dabf44648bca4

se_curl

curl

7.86.0

org_aomedia_avif

libavif

88d98ec7a85d

jpeg

libjpeg-turbo

2.1.4

png

libpng

1.6.37

libwebp

libwebp

1.2.4

org_lz4

lz4

1.9.4

nasm

nasm

2.15.05

org_nghttp2

nghttp2

1.50.0

com_github_nlohmann_json

nlohmann/json

3.10.5

com_github_pybind_pybind11

pybind11

2.10.1

com_google_snappy

snappy

984b191f0fef

libtiff

tiff

4.4.0

org_tukaani_xz

xz

5.3.3alpha

net_zstd

zstd

1.5.5

For example, to run the tests using the system-provided curl, jpeg, and SSL libraries:

export TENSORSTORE_SYSTEM_LIBS=se_curl,jpeg,com_google_boringssl
python3 bazelisk.py test //...

CMake build

In addition to a supported C++ toolchain, the following system dependencies are also required for the CMake build:

  • Python 3.8 or later

  • CMake 3.24 or later

  • Perl, for building libaom from source (default). Must be in PATH. Not required if -DTENSORSTORE_USE_SYSTEM_LIBAOM=ON is specified.

  • NASM, for building libjpeg-turbo, libaom, and dav1d from source (default). Must be in PATH.Not required if -DTENSORSTORE_USE_SYSTEM_{JPEG,LIBAOM,DAV1D}=ON is specified.

  • GNU Patch or equivalent. Must be in PATH.