4.1 Developer Usage Guide (`esmeralda-platform`)¶

From Local Workstation to Zero-Trust Production Without Changing a Line of Code¶

When a developer on the CX or AI CoE team builds a mortgage underwriting agent for applicants like Julian Sterling, they need two seemingly contradictory things:

Frictionless local inner-loop speed: Running esmeralda query or esmeralda serve --local on a laptop against local MCP servers (http://localhost:8001/mcp) with zero certificates, proxies, or load balancers in the way.
Zero-trust enterprise production rigor: Running the exact same code inside Vertex AI Agent Runtime behind the Central Agent Gateway, authenticating to https://legacy-dms.esmeralda.internal/mcp with internal Root CAs, SPIFFE identities (X-Agent-Spiffe-Id), 3LO user tokens (User-Auth-Token), and structured FinOps telemetry.

This guide shows how esmeralda (esmeralda-platform) bridges both worlds cleanly, and which building block to reach for in every scenario. For the complete module-by-module specification, see the package reference.

Why We Built It This Way (Key Decisions & Rationale)¶

Decision	Alternative Considered	Why We Chose This
Declarative `agent.yaml` (`framework: google-adk \\| a2a`, `env`, `local_env`)	Separate startup scripts and environment files per framework	Lets `esmeralda serve` and `esmeralda query` run ADK orchestrators and A2A specialists with the exact same CLI commands locally and in containers.
`esmeralda.mcp.toolset` & `esmeralda.a2a.remote_agent` Wrappers	Constructing raw `McpToolset` / `RemoteA2aAgent` instances in each agent	Automatically attaches `Authorization: Bearer` (ID or access token), `User-Auth-Token`, `X-Agent-Spiffe-Id`, `X-API-Key`, and pins A2A RPC URLs to the private gateway host.
Zero-Latency Local JWT Verification (`esmeralda.mcp_authz`)	Calling `oauth2.googleapis.com/tokeninfo` on every MCP tool call	Synchronous `tokeninfo` calls add latency and fail inside VPC Service Controls perimeters; `mcp_authz.verify_request` validates user claims and blocks service-account masquerading locally.
Read-Only `esmeralda mcp` Server for Coding Agents	Giving AI assistants raw shell access to `terraform apply` or `destroy`	Exposes structured, compact JSON tools (`status`, `list_units`, `plan_unit`, `query_agent`, `test_gateway`, `agent_logs`, `describe_contracts`) with zero mutation risk.

Which piece do I need?¶

Situation	Use
Any agent container on Agent Runtime	`ENTRYPOINT ["esmeralda", "run", "--"]` in the Dockerfile
Any agent package	`esmeralda.prepare()` at the top of `agent/__init__.py`, before importing the agent definition
Every agent package	`app = esmeralda.create_app(root_agent)` in `agent/__init__.py`
Start the agent's server (container or workstation)	`esmeralda serve` / `make serve AGENT=<agent>`
Agent served over A2A	`framework: a2a` and `agent_card` in `agent.yaml`; the package exports `app` like any agent, and `esmeralda serve` does the rest
A runner you build yourself	`EsmeraldaTelemetryPlugin()` in the runner's `plugins`, and `esmeralda.finalize()` once the tracer provider is set up
Agent that calls MCP tool servers through the gateway	`esmeralda.mcp.toolset(url, prefix=...)`
Agent that calls another agent over A2A	`esmeralda.a2a.remote_agent(name, url=...)` as a sub-agent
Client or test calling an agent through the Agent Runtime API	`"state_delta": {"temp:caller_context": {...}}` in the query input
Agent-specific plugins (BigQuery analytics, ...)	`esmeralda.create_app(root_agent, plugins=[...])`
Agent-specific process setup	`esmeralda.prepare([*esmeralda.lifecycle.PREPARE, my_step])`
Run or test an agent, locally or deployed	`esmeralda query`, or `make query AGENT=<agent>` (add `TARGET=remote` or `TARGET=<url>`)
Inspect, plan, apply, verify, or check logs of an environment	`uv run esmeralda status / doctor / units / plan / apply / verify / logs / contracts / guard`
Let a coding agent inspect and check a deployment (read-only)	`esmeralda mcp` as an MCP server: Operating a checkout from an agent

Add the library to an agent¶

These steps apply to a new agent and to an existing one. The CX mortgage orchestrator is the reference.

1. Depend on it from the workspace¶

In the agent's pyproject.toml:

[project]
dependencies = [
    "esmeralda",
    # ...
]

[tool.uv.sources]
esmeralda = { workspace = true }

Then regenerate the lock with make lock and commit uv.lock (see Contributing: Dependencies).

2. Prepare the process and export the app¶

agent/__init__.py:

import esmeralda

esmeralda.prepare()

from .agent import root_agent  # noqa: E402

app = esmeralda.create_app(root_agent)

Order matters: prepare() patches the google-genai and aiohttp clients, so it must run before agent.py creates the agent and its clients.

create_app returns an ADK App named agent, with EsmeraldaTelemetryPlugin first. The ADK loader picks up app before root_agent. The name agent must match the package directory and the --gemini_enterprise_app_name passed to adk api_server.

Note

Don't wire telemetry callbacks on the agent itself (after_model_callback=..., after_tool_callback=...). The plugin already emits the events for every agent in the app, and doing both would emit them twice.

3. Start the container through `esmeralda run` and `esmeralda serve`¶

Dockerfile:

COPY requirements.lock .
RUN pip install --no-cache-dir --require-hashes -r requirements.lock

# The esmeralda wheel, built by `make build-*` into dist/. Its dependencies are in requirements.lock.
COPY dist/ /tmp/dist/
RUN pip install --no-cache-dir --no-deps /tmp/dist/esmeralda-*.whl && rm -rf /tmp/dist

COPY . .

ENV GRPC_DEFAULT_SSL_ROOTS_FILE_PATH=/etc/ssl/certs/ca-certificates.crt
ENV REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
ENV SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt

ENTRYPOINT ["esmeralda", "run", "--"]
CMD ["esmeralda", "serve", "--otel-to-cloud"]

The same CMD works for ADK and A2A agents: esmeralda serve reads framework from agent.yaml (see Serving an agent). The base image needs ca-certificates, so that update-ca-certificates is available.

4. Build the wheel into the image¶

requirements.lock is exported with --no-emit-workspace, so it doesn't contain esmeralda. Pass the library as the fourth argument of build_image in the Makefile, and the build will put its wheel in dist/:

build-my-agent: ## Build and push my agent image
    $(call build_image,apps/agents/my-agent,my-agent,my-agent,esmeralda)

Serving an agent¶

esmeralda serve starts the agent's server, the same way in the container and on a workstation. It reads framework from agent.yaml:

Framework	Server
`google-adk`	`adk api_server` for the agent directory, with the Agent Runtime endpoints (`--gemini_enterprise_app_name agent`). `--web` starts the ADK dev UI instead
`a2a`	An A2A REST server for the package's `app`, with the agent card from `agent.yaml` `agent_card` (`A2A_AGENT_URL` overrides its `url`). Mounted at `/`, `/a2a` and `/api/a2a`

Option	Container	Workstation
`--otel-to-cloud`	Yes: exports traces and logs to Google Cloud	Usually not
`--local`	No	Yes: applies `local_env` from `agent.yaml`
`--port`	`$PORT` (default 8080)	Any

Locally, make serve adds --local and starts the local MCP servers:

make serve AGENT=ai-coe-mortgage-specialist            # A2A server on :8080
make serve AGENT=cx-mortgage-orchestrator PORT=8081    # ADK API server on :8081
make serve AGENT=cx-mortgage-orchestrator WEB=1        # ADK dev UI
make query AGENT=cx-mortgage-orchestrator TARGET=http://localhost:8081

Note

The first time ADK runs in a terminal, it asks once whether to enable ADK usage telemetry. It never asks in the container, which has no terminal.

What an A2A agent package provides¶

An A2A agent is written like any other agent: the package exports app (esmeralda.create_app). esmeralda serve builds the A2A executor, the session service and the task store around it:

Concern	Default	Override
Sessions	Vertex AI managed sessions on Agent Runtime (`GOOGLE_CLOUD_AGENT_ENGINE_ID`); in memory locally	`USE_IN_MEMORY_SESSIONS=1`
A2A task store	In memory (`InMemoryTaskStore`; requires a single worker with `max_instances: 1`)	Export `a2a_task_store_builder` from the package (a callable that returns an A2A `TaskStore`) when scaling beyond `max_instances: 1` with `tasks/get` polling or stream resubscribe. Multi-turn context is already durable in `VertexAiSessionService` keyed by `context_id`
Plugins	`EsmeraldaTelemetryPlugin`	The app's other plugins (for example BigQuery analytics)

finalize() runs once the tracer provider is set up.

Calling MCP servers and other agents¶

Calls through the gateway need the same things every time: a bearer token for the target (an ID token or an access token, see Token kinds), the end user's token when there is one, and the agent's API key, which an adapter can use for rate limiting. The library adds them for you.

MCP tool servers¶

import os

from esmeralda import mcp

dms_toolset = mcp.toolset(os.environ.get("DMS_MCP_URL", "https://legacy-dms.esmeralda.internal/mcp"), prefix="dms")

root_agent = Agent(..., tools=[dms_toolset])

Each request carries:

Authorization: Bearer <token>: by default an ID token for the server's origin (see Token kinds). Not sent to localhost, so local MCP servers work as-is.
the delegated user credential, only if the server's host is listed in ESMERALDA_USER_CREDENTIAL_TARGETS, in the header configured there (default X-User-Token). See Delegated user credential.
X-API-Key: <AGENT_NAME>.

Other agents (A2A)¶

from esmeralda import a2a

specialist = a2a.remote_agent(
    "mortgage_tools_agent",
    url=os.environ["A2A_AGENT_URL"],  # the callee's gateway address, e.g. https://<agent>.esmeralda.internal
    description="Delegate all mortgage-related queries to this agent.",
)

root_agent = Agent(..., sub_agents=[specialist])

The remote agent fetches the card from <url>/v1/card, and pins every RPC address on it to url, so calls always go through the configured gateway address (the card of an agent on Agent Runtime advertises the engine's own aiplatform.googleapis.com URL). ADK's own checks still apply: https, or http on a loopback host. Each request carries a bearer token for url (an access token for an agent host served by the entry point's direct backend, see Token kinds) and the API key. Each message carries this agent's caller context in A2A metadata, plus the user token (user_auth_token) when the callee's host is listed in ESMERALDA_USER_CREDENTIAL_TARGETS.

Token kinds¶

The entry point's targets take different credentials: Cloud Run MCP servers (and hosts routed to an adapter such as Envoy) take an ID token for the host, while agents on Agent Runtime served by the built-in direct backend take an OAuth2 access token (see Entry point → token kinds).

esmeralda.auth.token_kind(url) picks the kind: "access" for hosts listed in ACCESS_TOKEN_HOSTS (comma-separated hostnames; Terraform sets it on each agent from the service catalog and the routing) and for *.googleapis.com, "id" otherwise. esmeralda.auth.bearer_token(url, kind=None) returns the matching token. mcp.toolset(..., token=...) and a2a.remote_agent(..., token=...) take "id" or "access" to override the kind.

ID tokens and access tokens¶

ID tokens and access tokens come from esmeralda.auth. By default they are minted for the runtime identity: on Agent Runtime that is the agent's own Agent Identity (from the metadata server), elsewhere Application Default Credentials. Each is cached (ID tokens per audience) and refreshed shortly before it expires, so tool calls don't pay for IAM round trips.

On Agent Runtime, google-auth (2.61 and later) mints certificate-bound ID tokens by default, and Cloud Run rejects them once the call has crossed the Agent Gateway and the entry point (both terminate TLS). The library therefore requests every runtime-identity ID token unbound, with the documented per-call opt-out fetch_id_token_credentials(audience, bind_id_token=False). Access tokens keep the library default (bound); the container-wide GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false is never set. Unbound tokens are bearer tokens valid for hours; see Outbound tokens and the Agent Gateway for the trade-off.

SERVICE_ACCOUNT_EMAIL is a deprecated opt-in: when set, tokens are minted for that service account through impersonation instead. If impersonation fails, the library logs a warning and falls back to the runtime identity.

The user's token¶

When Gemini Enterprise calls an agent with a user authorization, the runtime stores the user's OAuth token as temp:<authorization id> session state. It is ephemeral and never written to the session. Set USER_AUTH_ID to the authorization id configured in Gemini Enterprise (default user_auth_token).

esmeralda.context.user_token(ctx) finds the token for the current invocation, either in that state or in the A2A metadata of a calling agent. Toolsets and remote agents forward it on their own, but only to the targets that accept it (see below). Agent code can call it from a tool or a callback.

Delegated user credential¶

The user's credential is a secret. It is forwarded only where a target asked for it, and never ends up in telemetry.

Which targets receive it. Only the hosts listed in ESMERALDA_USER_CREDENTIAL_TARGETS, a JSON object that maps a host (optionally host:port) to the header the target reads:

export ESMERALDA_USER_CREDENTIAL_TARGETS='{"legacy-dms.esmeralda.internal": "X-User-Token", "income-verification.esmeralda.internal": null}'

null, true or "" mean the default header, X-User-Token; false leaves the host out.
A host that isn't listed never receives the credential: no header for MCP servers, no user_auth_token in the A2A metadata of remote agents. The variable unset, malformed or not an object means no target at all (fail closed).
Wildcards aren't supported, and the header can't be one that carries other credentials, routing or tracing (Authorization, Cookie, Host, X-API-Key, X-Agent-Spiffe-Id, traceparent, baggage, ...).
For A2A targets, the credential travels as user_auth_token A2A metadata (what the callee's runtime reads), not as a header.

The variable will be generated from the catalog: every target whose entry has user_credential: {accept: true, header: <name>} becomes "<endpoint host>": "<name>" (no header means X-User-Token), and every other target is left out.

Which credential. For MCP servers, esmeralda.auth_manager.resolve_user_token picks it:

When Auth Manager / 3LO delegation is configured (auth_provider on the toolset, or AGENT_AUTH_PROVIDER), the delegated token it brokers is preferred over the raw token. If Auth Manager fails, the call goes out without a credential rather than falling back to the raw token.
Otherwise, the invocation's own token (context.user_token).

Auth Manager isn't called for a host that isn't listed. Remote agents only ever receive the invocation's own token: a brokered token is scoped to this agent's tools, so the callee brokers its own.

Never in telemetry. The credential never goes into OpenTelemetry baggage, span attributes, log records, usage records, exception messages or persisted session state; it lives only in temp: state for one invocation. On an A2A agent, ADK would copy the request metadata (token included) into every stored event; esmeralda serve therefore uses esmeralda.a2a.request_converter, which moves user_auth_token from the metadata into temp: state before the run starts (a hand-built A2aAgentExecutor should pass A2aAgentExecutorConfig(request_converter=esmeralda.a2a.request_converter)). Code that has to log a value that might contain it masks it with esmeralda.user_credential.redact(value, [token]) (or describe_error(exc, [token]) for exceptions); esmeralda query does this for --verbose output and errors. Two tests enforce the rules: packages/esmeralda/tests/test_user_credential_leak.py runs an agent with a canary token and checks that it reaches only the listed targets and nowhere in spans, baggage, logs, usage records, stdout or the stored session; test_user_credential_a2a_leak.py does the same over A2A (callee and RemoteA2aAgent caller, stored sessions on both sides); tests/guards/test_user_credential_guards.py fails on code that puts the credential into baggage or span attributes.

On the MCP server. Read the credential from the configured header (USER_CREDENTIAL_HEADER, default X-User-Token; esmeralda.mcp_authz also accepts the legacy User-Auth-Token), and never log its value. apps/services/legacy-dms shows the pattern: it logs that a user context was received, never the value.

Agent Identity Auth Manager (3LO Delegation) & End-User Token Verification¶

In Gemini Enterprise Agent Platform (GEAP), Agent Identity Auth Manager (agentidentitycredentials.googleapis.com) implements the Three-Identity Delegation Framework (User identity + Agent SPIFFE identity = Delegated identity):

Brokering 3LO/2LO credentials (esmeralda.auth_manager): When an MCP toolset is configured with auth_provider (or $AGENT_AUTH_PROVIDER), esmeralda.auth_manager.resolve_user_token(ctx, auth_provider=...) calls {auth_provider}:acquireToken on Agent Identity Auth Manager using the agent's runtime identity and X-Agent-Spiffe-Id ($AGENT_SPIFFE_ID or workload X.509 certificate). The brokered token is preferred over the invocation's own token, and is sent only to hosts listed in ESMERALDA_USER_CREDENTIAL_TARGETS.

from esmeralda import mcp

dms_toolset = mcp.toolset(
    os.environ["DMS_MCP_URL"],
    prefix="dms",
    auth_provider=os.environ.get("AGENT_AUTH_PROVIDER"),
)

Zero-latency end-user & delegation verification on MCP servers (esmeralda.mcp_authz): Downstream MCP servers can inspect the user credential header (X-User-Token, or $USER_CREDENTIAL_HEADER; the legacy User-Auth-Token is still read) and X-Agent-Spiffe-Id locally without making synchronous calls to oauth2.googleapis.com/tokeninfo (which fail inside VPC Service Controls perimeters) and without hardcoding fictional user allowlists:
Always rejects service account tokens (*.gserviceaccount.com) presented as the user credential, preventing workload service accounts from masquerading as human end users.
Fails closed when MCP_ENFORCE_USER_AUTH=true and the user credential is missing, expired, or lacks user identity claims.

from esmeralda import mcp_authz

caller = mcp_authz.verify_request(request.headers)

Caller context¶

The caller context says which project and agent made a call. The plugin puts it in OpenTelemetry baggage for the run, and every span created during the run gets the attributes caller.project_id and caller.agent_name. It is for telemetry and cost attribution, never for authorization.

Agent-to-agent calls (A2A)¶

esmeralda.a2a.remote_agent sends it automatically: the calling agent identifies itself from GOOGLE_CLOUD_PROJECT and AGENT_NAME. A hand-built RemoteA2aAgent can pass a2a_request_meta_provider=esmeralda.a2a.request_metadata_for(url) (the user token is added only when url's host accepts it).

On the receiving side, ADK exposes the metadata as run_config.custom_metadata["a2a_metadata"], where the plugin reads it.

Direct calls through the Agent Runtime API¶

Send the context as temporary session state. The temp: prefix keeps it for the current invocation only; it is never written to the session:

{
  "class_method": "async_stream_query",
  "input": {
    "message": "Verify Julian Sterling's income",
    "user_id": "user-1",
    "session_id": "1234",
    "state_delta": {"temp:caller_context": {"project_id": "team-a-project", "agent_name": "team_a_agent"}}
  }
}

Warning

Don't pass caller_context as a top-level query argument. The runtime forwards unknown arguments to the ADK runner, which rejects them.

Agent-specific plugins and setup¶

Extra ADK plugins go after EsmeraldaTelemetryPlugin:

app = esmeralda.create_app(root_agent, plugins=[create_bq_plugin()])

Extra process setup is a function without arguments, added to the default steps. Make it idempotent (safe to run twice) and let it raise on failure. A half-configured agent should not start.

def enable_my_sdk_proxy() -> None:
    ...

esmeralda.prepare([*esmeralda.lifecycle.PREPARE, enable_my_sdk_proxy])

The same applies to finalize, with esmeralda.lifecycle.FINALIZE. When you pass custom finalize steps, give them to the plugin too: EsmeraldaTelemetryPlugin(finalize_steps=...).

Querying an agent: locally, on a server, or deployed¶

esmeralda query calls any agent, ADK or A2A, with one command and one report. It reads framework from the agent's agent.yaml to choose how to talk to it:

Target	Command	How it calls the agent
In-process (default)	`esmeralda query --agent-dir apps/agents/<agent> "message"`	Loads the package's `app` and runs it through the ADK runner. No server or port needed, and it works in a debugger
A running server	`... --url http://localhost:8080 "message"`	ADK: `POST /api/stream_reasoning_engine` (`async_stream_query`). A2A: agent card, then `message:send`
The deployed agent	`... --engine projects/P/locations/L/reasoningEngines/ID "message"`	The same calls through the Agent Runtime API, with your gcloud credentials

Every run prints the same report: the events as they arrive (agent transfers, tool calls and results, text), then the answer, tool and token counts, and a verdict. The exit code is 0 on success, and 1 if the agent reported an error, returned no answer, or (A2A) the task didn't complete. Add --fail-on-tool-error to also fail when a tool call returned an error.

The caller context is always sent, as <project>/esmeralda_cli by default (override with --caller project/agent). Pass --user-token <token> (or set $ESMERALDA_USER_TOKEN) to forward an end-user OAuth token as temp:user_auth_token (ADK) or user_auth_token in A2A metadata. Other options: --user, --session (continue a conversation), --timeout, --verbose (raw events).

From the repository root, make query resolves the agent directory and the deployed engine for you:

make query AGENT=ai-coe-mortgage-specialist                           # in-process, local MCP servers started for you
make query AGENT=cx-mortgage-orchestrator QUERY="Verify Julian Sterling's income"
make query AGENT=cx-mortgage-orchestrator TARGET=remote ENV=dev       # the engine deployed in dev
make query AGENT=cx-mortgage-orchestrator TARGET=http://localhost:8080
make test-e2e ENV=dev                                                 # remote query: specialist, then orchestrator

Local runs and `local_env`¶

A local run applies the agent's agent.yaml env (the same variables the deployed engine gets), then its local_env, without overriding variables already set in your shell. local_env holds what differs on a workstation:

local_env:
  LOCAL_MODE: "true"                               # orchestrator: load the specialist in-process
  EMAIL_MCP_URL: "http://localhost:8001/mcp"       # local MCP servers started by make query
  INCOME_VERIFICATION_URL: "http://localhost:8002/mcp"
  DMS_MCP_URL: "http://localhost:8003/mcp"

Local runs call Gemini on Vertex AI with your Application Default Credentials. Run gcloud auth application-default login with an account that can use Vertex AI, and set GOOGLE_CLOUD_PROJECT to a project where it is enabled.

Operating a checkout from an agent (`esmeralda mcp`)¶

esmeralda mcp is a read-only MCP server (stdio transport) for coding agents and MCP clients. Its tools answer the usual operator questions (what is deployed, what would a plan change, does the agent answer, do the routes work, what do the logs say) with compact JSON instead of raw Terraform output or log dumps.

Add it to the client's MCP configuration. Start the client from the repository root, or set ESMERALDA_REPO:

{
  "mcpServers": {
    "esmeralda": {
      "command": "uv",
      "args": ["run", "--frozen", "--all-packages", "esmeralda", "mcp"],
      "env": {"ESMERALDA_REPO": "/path/to/esmeralda", "TG_PROVIDER_CACHE": "1"}
    }
  }
}

--all-packages runs the server in the workspace environment, which has fastmcp (locked through the MCP services) and google-cloud-logging (locked through the agents). Without fastmcp, the command exits with that hint.

Tool	Returns	Notes
`status(env, unit=None)`	Deployed unit count, projects, entry point IP/domain/routing, gateways and their routes, agent engine ids and URLs, MCP service URIs, Agent Gateway, test VM. With `unit`: that unit's non-sensitive outputs, long values replaced by their size	Reads the local state files when the env uses the local backend (instant), else `terragrunt output -json` per unit
`list_units(env)`	Every Terragrunt unit with `deployed` and its managed resource count
`plan_unit(env, unit)`	`changes`, `counts` (add/change/destroy/replace/read), one line per changed resource (`+`, `~`, `-/+`, `-`, `<=`) with the changed attribute names and replacement causes, changed outputs, drift	`terragrunt plan -lock=false`; the plan file goes to `.esmeralda/plans/` (gitignored). Nothing is applied
`query_agent(env, agent, message)`	`passed`, the answer, agents involved, tool calls, tool errors, tokens, seconds	The same path as `make query AGENT=<agent> TARGET=remote` (`esmeralda.query.run` with the unit's `engine_id`)
`test_gateway(env)`	`passed`, entry point, one entry per host: probe kind, HTTP code, `ok`, stale DNS, error	Runs `scripts/test_gateway.sh` (IAP SSH to the test VM)
`agent_logs(env, agent, since="30m", severity="WARNING", limit=50)`	Entries `{time, severity, message, count?}`, newest first	Cloud Logging for the agent's Reasoning Engine. Tracebacks are collapsed to their exception line, repeated messages (numbers and ids masked) are merged with a `count`, messages are cut to 300 characters
`describe_contracts()`	The gateway adapter contract and the `gateways` block (from the ingress gateways guide), plus each env's current `gateways` block and gateway units

Design rules:

Read-only. There is no apply or destroy tool. Changes go through make targets and review.
Per-tool timeouts (plan 15 min, query and gateway test 5 min, logs 2 min, status 3 min).
Actionable errors. Failures come back as tool errors with a hint, for example gcloud auth expired: run gcloud auth login and gcloud auth application-default login, an unknown unit with the closest match, or a missing ESMERALDA_BILLING_ACCOUNT.
The server's environment is the client's. Terragrunt reads ESMERALDA_BILLING_ACCOUNT (and ESMERALDA_ORG_ID) from env.yaml; the server fills them from the user-local env file when the client's environment lacks them, or pass them in the client's env block. Prefer TG_PROVIDER_CACHE=1 over TF_PLUGIN_CACHE_DIR: the Terraform plugin cache isn't safe for the parallel terraform init runs Terragrunt does to read dependency outputs, and fails with does not match any of the checksums.

Terragrunt variables: user-local env file¶

Every env.yaml reads ESMERALDA_BILLING_ACCOUNT (required) and ESMERALDA_ORG_ID (optional) from the environment. Instead of exporting them from an interactive shell profile, put them in a file outside the repository:

mkdir -p ~/.config/esmeralda
cat > ~/.config/esmeralda/env <<'EOF'
ESMERALDA_BILLING_ACCOUNT=<billing-account-id>
ESMERALDA_ORG_ID=<org-id>
EOF
chmod 600 ~/.config/esmeralda/env

Path: $ESMERALDA_ENV_FILE if set, else ~/.config/esmeralda/env.
Who reads it: every esmeralda command that runs Terragrunt or the repo scripts (status, units, plan, apply, verify, doctor, mcp), and the Makefile (-include), so make targets get the same values.
Format: plain KEY=VALUE lines and # comments, so the CLI and make read it the same way. Only ESMERALDA_* keys. The CLI also tolerates export and quotes, but make would keep the quotes, so esmeralda doctor warns about them and about other keys.
Precedence: a variable already set in the environment (or on the make command line) wins over the file.
Privacy: values are never printed. esmeralda doctor reports where each variable comes from (env, env-file or missing) and warns if the file is accessible by group or others.

Warning

Never commit this file or copy it into the repository: billing account and organization ids are environment specifics.

Note

The make targets for plan, status and environment checks cover similar ground for humans. esmeralda mcp is self-contained in the package and doesn't call them, so both can evolve independently; if they converge, the tools can call the shared code.

Running and testing locally¶

esmeralda query (above) is the standard way to run an agent locally.
adk web / adk api_server load app exactly as in production. Without AGENT_GATEWAY_ROOT_CERTIFICATES, the entrypoint only logs a warning, and you don't need it locally.
Scripts can wrap the app with the Agent Engine template: AdkApp(app=app).
Unit tests that mock ADK or google-genai (as the agents' conftest.py do) should stub prepare, because it patches the real clients. The library's own tests cover the patches:

import esmeralda

esmeralda.prepare = lambda *args, **kwargs: None

Run commands through make, or set UV_FROZEN=1. A bare uv run re-locks uv.lock through the corporate package proxy. The pre-push lock check catches it, but it costs a make lock to fix.

Library tests:

make test-agents   # agents and esmeralda
uv run --frozen --package esmeralda --extra dev pytest packages/esmeralda/tests

Troubleshooting¶

Symptom	Likely cause	Check
`CERTIFICATE_VERIFY_FAILED` on egress	Root CA not installed	Agent logs show `✅ Installed N Agent Gateway root certificate(s)`; the engine has `AGENT_GATEWAY_ROOT_CERTIFICATES`; the Dockerfile `ENTRYPOINT` is `esmeralda run`
Container exits right away with `contains no PEM certificate`	The env var is set but malformed	The Layer 4 output `agw_root_ca_bundle`
`Network is unreachable` from Gemini or sessions	`prepare()` ran after the clients were created	`esmeralda.prepare()` is the first statement in `agent/__init__.py`
No `mcp_tool_execution` / `genai_token_consumption` events	The plugin isn't in the runner	The package exports `app` (`create_app`), or the custom runner lists `EsmeraldaTelemetryPlugin()`
Events appear twice	Telemetry callbacks also wired on the agent	Remove `after_model_callback` / `after_tool_callback` telemetry from the agent
Spans without `caller.*` attributes	The caller didn't send the context, or no SDK tracer provider	`state_delta` / A2A metadata on the caller side; a one-time warning `No OpenTelemetry SDK tracer provider` in the logs

4.1 Developer Usage Guide (esmeralda-platform)¶

From Local Workstation to Zero-Trust Production Without Changing a Line of Code¶

Why We Built It This Way (Key Decisions & Rationale)¶

Which piece do I need?¶

Add the library to an agent¶

1. Depend on it from the workspace¶

2. Prepare the process and export the app¶

3. Start the container through esmeralda run and esmeralda serve¶

4. Build the wheel into the image¶

Serving an agent¶

What an A2A agent package provides¶

Calling MCP servers and other agents¶

MCP tool servers¶

Other agents (A2A)¶

Token kinds¶

ID tokens and access tokens¶

The user's token¶

Delegated user credential¶

Agent Identity Auth Manager (3LO Delegation) & End-User Token Verification¶

Caller context¶

Agent-to-agent calls (A2A)¶

Direct calls through the Agent Runtime API¶

Agent-specific plugins and setup¶

Querying an agent: locally, on a server, or deployed¶

Local runs and local_env¶

Operating a checkout from an agent (esmeralda mcp)¶

Terragrunt variables: user-local env file¶

Running and testing locally¶

Troubleshooting¶

4.1 Developer Usage Guide (`esmeralda-platform`)¶

3. Start the container through `esmeralda run` and `esmeralda serve`¶

Local runs and `local_env`¶

Operating a checkout from an agent (`esmeralda mcp`)¶