Litmus CLI
A command-line interface for deploying and managing Litmus, a tool for quickly building and testing LLMs.
Prerequisites
- Google Cloud SDK (gcloud): Ensure you have the Google Cloud SDK installed and authenticated.
- Install: https://cloud.google.com/sdk/docs/install
- Authenticate:
gcloud auth login
- Go 1.18 or higher: Required for building and running the CLI.
Installation
Fast installation
Make sure you have the Google Cloud SDK installed and configured with the correct project.
Install binary:
- Linux:
curl https://storage.googleapis.com/litmus-cloud/install/linux.sh | sudo sh
- sha256: https://storage.googleapis.com/litmus-cloud/prod/linux/litmus.sha256
- OSX:
curl https://storage.googleapis.com/litmus-cloud/install/osx.sh | sudo sh
- sha256: https://storage.googleapis.com/litmus-cloud/prod/osx/litmus.sha256
Manual Build
Clone this repository to your local machine:
bashgit clone https://github.com/google/litmus.git
Navigate to the project directory:
bashcd litmus/cli
Install Go dependencies:
bashgo mod download
Build the CLI:
bashgo build
Usage
Usage: litmus <command> [options] [flags]
Commands:
open Open the Litmus dashboard
deploy Deploy the application
destroy Destroy Litmus resources
update Update the application
status Show the status of the Litmus deployment
version Display the version of the Litmus CLI
execute Execute a payload against the Litmus application
ls List all runs
run Open a specific Litmus run
start Starts a new Litmus run
analytics Manage Litmus analytics (deploy or destroy)
proxy Manage Litmus proxy (deploy, list, destroy, destroy-all)
tunnel Create a tunnel to the Litmus UI
Options:
--project <project-id>: Specify the project ID (overrides default)
--region <region>: Specify the region (defaults to 'us-central1')
--quiet Suppress verbose output
--preserve-data Preserve data in Cloud Storage, Firestore, and BigQuery
Tunnel Options:
--port <port>: Specify the local port for the tunnel (default: 8081)
Examples
Deploy Litmus:
bashlitmus deploy
This command deploys the Litmus core services (API and Worker) to your default GCP project in the
us-central1
region. During deployment it will create required service accounts, grant permissions and deploy the services to Cloud Run. You can use the--quiet
flag to suppress verbose output.Deploy to a specific project and region:
bashlitmus deploy --project my-project-id --region us-east1
This command deploys the Litmus core services (API and Worker) to the specified GCP project (
my-project-id
) in the specified region (us-east1
).Deploy to a specific environment:
bashlitmus deploy dev
This command deploys the Litmus core services to the
dev
environment. This will pull and deploy the latestdev
images.Destroy the Litmus deployment:
bashlitmus destroy
This command deletes all Litmus resources in your default project and
us-central1
region. It removes the API and worker service deployments, deletes secrets from secret manager, service accounts, and the Cloud Storage bucket. You can use the--quiet
flag to suppress verbose output.Destroy the Litmus deployment and preserve data:
bashlitmus destroy --preserve-data
This command deletes all Litmus resources in your default project and
us-central1
region but keeps the data in Cloud Storage, Firestore and BigQuery.Update the Litmus deployment:
bashlitmus update
This command updates your Litmus deployment to the latest version available. It updates both the API and the Worker deployments. You can use the
--quiet
flag to suppress verbose output.Update the Litmus deployment to a specific environment:
bashlitmus update dev
This command updates your Litmus deployment to the latest
dev
version available. It updates both the API and the Worker deployments.Get deployment status:
bashlitmus status
This command retrieves and displays the status of your Litmus deployment. This includes the service URL, username and password.
Display CLI version:
bashlitmus version
This command displays the version of the installed Litmus CLI.
Execute a payload:
bashlitmus execute "Hello, world!"
This is a placeholder command, there is no implementation yet.
List all runs:
bashlitmus ls
This command retrieves and displays a list of all the test runs that have been submitted, including their status and other details.
Open a specific run:
bashlitmus run <runID>
This command opens the details page for a specific Litmus run in your default browser. You can get the run ID by running the
litmus ls
command.Start a new Litmus Test Run:
bashlitmus start $TEMPLATE_ID $RUN_ID
This command submits a new test run using the provided template ID and run ID. Make sure that the template exists before running the command. The
$RUN_ID
can be generated automatically by runninguuidgen
.Deploy Litmus Analytics:
bashlitmus analytics deploy
This command sets up the analytics components for Litmus, including a BigQuery dataset for storing logs and log sinks to route logs from the proxy and API to BigQuery.
Destroy the Litmus Analytics deployment:
bashlitmus analytics destroy
This command removes the analytics components for Litmus, including the BigQuery dataset and the log sinks.
Deploy Litmus Proxy:
bashlitmus proxy deploy
This command deploys the Litmus proxy service. It will prompt the user to select from a list of available regions and platforms. The Proxy is used for logging and analyzing your LLM interactions.
Deploy Litmus Proxy for specific upstream URL:
bashlitmus proxy deploy --upstreamURL <your_upstream_url>
This command deploys the Litmus proxy service for a specific upstream URL. Replace
<your_upstream_url>
with the desired upstream endpoint (e.g.,europe-west1-aiplatform.googleapis.com
).List all deployed Litmus Proxies:
bashlitmus proxy list
This command lists all Litmus proxy services that are currently deployed in your GCP project. It displays the name and URL of each proxy.
Destroy a Litmus Proxy deployment:
bashlitmus proxy destroy <service_name>
This command destroys the specified Litmus proxy deployment. Replace
<service_name>
with the name of the deployed proxy (e.g.,us-central1-aiplatform-litmus-abcd
).Destroy all Litmus Proxy deployments:
bashlitmus proxy destroy-all
This command destroys all Litmus proxy deployments in your current project and region.
Create a tunnel to the Litmus UI:
bashlitmus tunnel --port 8081
This command creates an SSH tunnel to the Litmus UI, making it accessible on your local machine at
http://localhost:8081
.
Configuration
- The CLI uses your default gcloud project configuration.
- You can use the
--project
flag to specify a different project for all commands. - You can use the
--region
flag to specify a different region for thedeploy
anddestroy
commands.