Skip to content

Getting Started

This guide will walk you through the initial steps of setting up Litmus and running your first tests.

Prerequisites

Before you start, ensure you have the following:

Quick Deployment with Litmus CLI

The easiest way to set up Litmus is using the Litmus CLI.

  1. Install the Litmus CLI:

  2. Deploy Litmus:

    bash
    litmus deploy
    • This will deploy the Litmus core services to your default GCP project.
    • The deployment script creates required service accounts, grants permissions, and deploys the worker and API services to Cloud Run.
    • You can customize the project and region using flags (see CLI Usage).
  3. Access Litmus:

    • Run litmus status to retrieve the Litmus web interface URL and credentials.
    • Open the provided URL in your browser and log in using the displayed username and password.

Next Steps

  • Create Test Templates: Define and manage templates specifying the structure and parameters of your HTTP request tests.

  • Submit Test Runs: Use the Litmus UI to submit test runs using pre-defined templates and provide test data.

  • (Optional) Deploy the Proxy Service: For enhanced LLM monitoring, deploy the Litmus proxy service:

    bash
    litmus proxy deploy
    • This captures and logs interactions with your LLM (Vertex AI or other providers) to BigQuery for detailed analysis.

Explore and Analyze

  • Web Interface: Utilize the Litmus UI to monitor test run progress, view detailed results, filter responses for in-depth analysis, and gain insights into proxy logs and aggregated LLM usage metrics.

Need Help?

  • Litmus CLI Usage: Refer to the Litmus CLI documentation: CLI Usage
  • Proxy Service: Learn about deploying and using the Litmus proxy service: Proxy Usage
  • Contribute: We welcome your contributions! See Contribution Guide for details.

Let's get started with testing and evaluating your LLMs with Litmus!

Disclaimer: This is not an officially supported Google product.