agentmesh-control-plane

agentmesh-control-plane admits nodes, mints credentials, holds the mesh policy and tracks routers. It is an HTTP server over SQLite or PostgreSQL.

agentmesh-control-plane [flags]                     run the server
agentmesh-control-plane admin ban   --peer <id>     ban a node directly in the database
agentmesh-control-plane admin unban --peer <id>     lift a ban

Flags

HTTP API

Every request and response is a message in api/agentmesh.proto, in one of two encodings. Routes that mesh components call use binary protobuf (application/x-protobuf). Routes for operators and the console use protojson of the same messages, with proto field names; unknown fields are rejected. POST /policies accepts either and answers in the encoding of the request.

Responses that carry credentials are sent with Cache-Control: no-store.

Unauthenticated

Enrollment, refresh, and STS

Every request below carries a challenge_unix_ms and a challenge_signature. The enrollee signs mesh:<endpoint>:<peer_id>:<challenge_unix_ms> with its key, and the control plane accepts the signature within five minutes of that instant. This proves that the caller holds the key behind the peer ID it names. Every instant in a response (expire_time and the like) is a google.protobuf.Timestamp, an RFC 3339 string in JSON.

Enrollment is admitted at 10 requests a second for the whole mesh, with a burst of 20. A request over that is answered 429 with a Retry-After header. agentmesh-node waits for it and sends the request again, with jitter, for up to three minutes before it reports the failure, so a fleet of up to about 1500 members that starts at once joins over its first minutes instead of losing the members the limiter turned away. A larger fleet starts in waves; the scalability guide has the arithmetic. A client of your own should retry the same way.

Admin

All routes require Authorization: Bearer <admin-token>. Request and response bodies are protojson of the messages named below, with proto field names; unknown fields are rejected.

User

For a human OIDC user acting on their own nodes, authenticated with an ID token from a configured human issuer (tokens matching --workload-issuer are refused with 403). The console uses these routes.

Notes

• A new control plane grants nothing. With no policy in the database, enrollment succeeds only for identities bound to the requested role, and with no bindings there are none. A policy must be posted first.
• Bootstrap tokens are spent atomically. max_usages holds under concurrent enrollments, and a pending request is resolved exactly once.
• An approved peer that enrolls again gets a new credential without approval, as long as the token is valid, the role matches the record and the peer is not banned. This is how a bootstrap node recovers from a retired signing key.
• Bans propagate in two ways. /info lists banned peer IDs for nodes and routers that start or restart. A signed mesh event reaches the ones that are already running.
• The rotation deadline is in the database. The first rotation is due one --key-rotation-interval after the first start, each rotation sets the next, and every replica checks the deadline at least once a minute. A control plane that restarts more often than it rotates still rotates on time, and one that was down when a rotation was due rotates on its first check. agentmesh_control_plane_next_key_rotation_timestamp_seconds is the deadline; a value in the past is a rotation that is not happening.
• Node records are garbage-collected. A node whose session has expired is deleted after --node-retention. A node enrolled with a bootstrap token has no session; its record is deleted once it has not refreshed its credential for --node-retention, or for --ephemeral-node-retention when the token was ephemeral, and a node that comes back after that enrolls again. A node without a persistent data directory enrolls a new identity on every restart, so without this the node table would only grow. agentmesh_control_plane_nodes_deleted_total{reason} counts the deletions: session, silent, ephemeral.
• The records around an enrollment are garbage-collected with it. The same sweep deletes enrollment requests resolved, or left pending, for --node-retention (for --ephemeral-node-retention when an ephemeral token made them), bootstrap tokens that expired --node-retention ago and that no remaining request refers to, and router leases that expired --node-retention ago. It also drops the service report held in memory for every node that is no longer admitted. agentmesh_control_plane_storage_rows_deleted_total{table} counts these deletions; agentmesh_control_plane_storage_rows{table} is the size of each table, and a table that keeps growing past its retention is a sweep that is failing, which the log says.

Metrics

/metrics exposes agentmesh_control_plane_* counters and gauges: enrolled nodes by role and state, active routers, mesh-connected peers as reported by router leases, rows per store table, service reports held in memory, per-route request counts and latencies, and the Go runtime.

A member whose request on a mesh endpoint (/policies, /egress, /revocations, /token/exchange, /sts/token, …) is answered 401 is counted in agentmesh_control_plane_member_auth_refusals_total{reason} and logged with the same reason: malformed (not a base64 credential), invalid_biscuit (not signed by a key this control plane trusts, or expired), unknown_node (the credential verifies but the store has no enrollment record for the node: the store was reset, or the node was removed), banned, session_expired. unknown_node is the one to look at after a control plane restart on an empty store: every node then refused is a node that needs a bootstrap token (--bootstrap-token-path) to enroll again on its own.