This directory contains a library, database, and tools for cloning transparency logs. The core library and database is log-agnostic, and each tool tailors this generic library to a specific log.
The core library attempts to balance optimization of the following goals:
This is achieved by:
The goal is to download data as quickly as possible from the log, and then persist verified data locally. A single download session looks like this:
N
M
M
is 0[M, N)
from the log
leaves
table of the database from this memory pool, strictly in order of their indexN
leaves have been written to the database, calculate the Merkle root of all of these leavescheckpoints
table of the database
Note that this means that until a download session completes successfully, the database may contain unverified leaves with an index greater than that stored in the latest checkpoint. Leaves must not be trusted if their index is greater or equal to the size of the latest checkpoint.
This library was designed to form the first part of a local data pipeline, i.e. downstream tooling can be written that reads from this local mirror of the log. Such tooling MUST only trust leaves that are committed to by a checkpoint; reading leaves with an index greater than the current checkpoint size is possible, but such data is unverified and using this defeats the purpose of using verifiable data structures.
The leaves
table records the leaf data as blobs; this accurately reflects what the log has committed to, but does not enable efficient SQL queries into the data.
A common usage pattern for a specific log ecosystem would be to have the first stage of the local pipeline parse the leaf data and break out the contents into a table with an appropriate schema for the parsed data.
In MariaDB, create a database and user. Below is an example of doing this for MariaDB 10.6, creating a database google_xenon2022
, with a user clonetool
with password letmein
.
MariaDB [(none)]> CREATE DATABASE google_xenon2022;
MariaDB [(none)]> CREATE USER 'clonetool'@localhost IDENTIFIED BY 'letmein';
MariaDB [(none)]> GRANT ALL PRIVILEGES ON google_xenon2022.* TO 'clonetool'@localhost;
MariaDB [(none)]> FLUSH PRIVILEGES;
The clone library logs information as it runs using glog
.
Providing --alsologtostderr
is passed to any tool using the library, you should see output such as the following during the cloning process:
I0824 11:09:23.517796 2881011 clone.go:71] Fetching [4459168, 95054738): Remote leaves: 95054738. Local leaves: 4459168 (0 verified).
I0824 11:09:28.519257 2881011 clone.go:177] 1202.8 leaves/s, last leaf=4459168 (remaining: 90595569, ETA: 20h55m21s), time working=24.2%
I0824 11:09:33.518444 2881011 clone.go:177] 1049.6 leaves/s, last leaf=4465183 (remaining: 90589554, ETA: 23h58m29s), time working=23.0%
I0824 11:09:38.518542 2881011 clone.go:177] 1024.0 leaves/s, last leaf=4470430 (remaining: 90584307, ETA: 24h34m21s), time working=23.3%
When tuning, it is recommended to also provide --v=1
to see more verbose output.
In particular, this will allow you to see if the tool is encountering errors (such as being told to back off) by the log server, e.g. if you see lines such as Retryable error getting data
in the output.
Assuming you aren’t being rate limited, then optimization goes as follows:
working
percentage regularly around 100%: this measures how much time is being spent writing to the database. To do this, increase the number of workers
to ensure that data is always available to the database writerworking %
is around 100, then increasing the DB write batch size will increase throughput, to a pointThis process is somewhat iterative to find what works for your setup. It depends on many variables such as log latency and rate limiting, the database, the machine running the clone tool, etc.
Download clients are provided for:
See the documentation for these for specifics of each tool.
The data is stored in a simple database table named leaves
, where each leaf in the log
is identified by its position in the log (column id
) and the data at that position is a
blob (column data
). The expectation is that clients will write custom tooling that reads
from this SQL table in order to perform custom tasks, e.g. verification of data, searching
to find relevant records, etc.
An example query that returns the first 5 leaves in the DB:
select * from leaves where id < 5;
On a Mac with Homebrew already installed, getting MariaDB installed
and connected to it in order to run the setup above is simple.
At the time of writing, this installed 10.8.3-MariaDB Homebrew
which works well for
the cloning tool.
brew install mariadb
brew services restart mariadb
mysql