QA overview

The private QA stack is meant to exercise OpenClaw in a more realistic, channel-shaped way than a single unit test can.

Current pieces:

• text

  Copy

  extensions/qa-channel
  : synthetic message channel with DM, channel, thread, reaction, edit, and delete surfaces.
• text

  Copy

  extensions/qa-lab
  : debugger UI and QA bus for observing the transcript, injecting inbound messages, and exporting a Markdown report.
• text

  Copy

  extensions/qa-matrix
  , future runner plugins: live-transport adapters that drive a real channel inside a child QA gateway.
• text

  Copy

  qa/
  : repo-backed seed assets for the kickoff task and baseline QA scenarios.

Command surface

Every QA flow runs under

. Many have script aliases; both forms are supported.

Operator flow

The current QA operator flow is a two-pane QA site:

• Left: Gateway dashboard (Control UI) with the agent.
• Right: QA Lab, showing the Slack-ish transcript and scenario plan.

Run it with:

bash
Copy
pnpm qa:lab:up

That builds the QA site, starts the Docker-backed gateway lane, and exposes the QA Lab page where an operator or automation loop can give the agent a QA mission, observe real channel behavior, and record what worked, failed, or stayed blocked.

For faster QA Lab UI iteration without rebuilding the Docker image each time, start the stack with a bind-mounted QA Lab bundle:

bash
Copy
pnpm openclaw qa docker-build-image
pnpm qa:lab:build
pnpm qa:lab:up:fast
pnpm qa:lab:watch

keeps the Docker services on a prebuilt image and bind-mounts into the container. rebuilds that bundle on change, and the browser auto-reloads when the QA Lab asset hash changes.

For a local OpenTelemetry trace smoke, run:

bash
Copy
pnpm qa:otel:smoke

That script starts a local OTLP/HTTP trace receiver, runs the

QA scenario with the plugin enabled, then decodes the exported protobuf spans and asserts the release-critical shape: , , , , and must be present; model calls must not export on successful turns; raw diagnostic IDs and attributes must stay out of the trace. It writes next to the QA suite artifacts.

Observability QA stays source-checkout only. The npm tarball intentionally omits QA Lab, so package Docker release lanes do not run

commands. Use from a built source checkout when changing diagnostics instrumentation.

For a transport-real Matrix smoke lane, run:

bash
Copy
pnpm openclaw qa matrix --profile fast --fail-fast

The full CLI reference, profile/scenario catalog, env vars, and artifact layout for this lane live in Matrix QA. At a glance: it provisions a disposable Tuwunel homeserver in Docker, registers temporary driver/SUT/observer users, runs the real Matrix plugin inside a child QA gateway scoped to that transport (no

), then writes a Markdown report, JSON summary, observed-events artifact, and combined output log under .

For transport-real Telegram and Discord smoke lanes:

bash
Copy
pnpm openclaw qa telegram
pnpm openclaw qa discord

Both target a pre-existing real channel with two bots (driver + SUT). Required env vars, scenario lists, output artifacts, and the Convex credential pool are documented in Telegram and Discord QA reference below.

Before using pooled live credentials, run:

bash
Copy
pnpm openclaw qa credentials doctor

The doctor checks Convex broker env, validates endpoint settings, and verifies admin/list reachability when the maintainer secret is present. It reports only set/missing status for secrets.

Live transport coverage

Live transport lanes share one contract instead of each inventing their own scenario list shape.

is the broad synthetic product-behavior suite and is not part of the live transport coverage matrix.

This keeps

as the broad product-behavior suite while Matrix, Telegram, and future live transports share one explicit transport-contract checklist.

For a disposable Linux VM lane without bringing Docker into the QA path, run:

bash
Copy
pnpm openclaw qa suite --runner multipass --scenario channel-chat-baseline

This boots a fresh Multipass guest, installs dependencies, builds OpenClaw inside the guest, runs

, then copies the normal QA report and summary back into on the host. It reuses the same scenario-selection behavior as on the host. Host and Multipass suite runs execute multiple selected scenarios in parallel with isolated gateway workers by default. defaults to concurrency 4, capped by the selected scenario count. Use to tune the worker count, or for serial execution. The command exits non-zero when any scenario fails. Use when you want artifacts without a failing exit code. Live runs forward the supported QA auth inputs that are practical for the guest: env-based provider keys, the QA live provider config path, and when present. Keep under the repo root so the guest can write back through the mounted workspace.

Telegram and Discord QA reference

Matrix has a dedicated page because of its scenario count and Docker-backed homeserver provisioning. Telegram and Discord are smaller — a handful of scenarios each, no profile system, against pre-existing real channels — so their reference lives here.

Shared CLI flags

Both lanes register through

and accept the same flags:

Both exit non-zero on any failed scenario.

writes artifacts without setting a failing exit code.

Telegram QA

bash
Copy
pnpm openclaw qa telegram

Targets one real private Telegram group with two distinct bots (driver + SUT). The SUT bot must have a Telegram username; bot-to-bot observation works best when both bots have Bot-to-Bot Communication Mode enabled in

.

Required env when

:

• text

  Copy

  OPENCLAW_QA_TELEGRAM_GROUP_ID
  — numeric chat id (string).
• text

  Copy

  OPENCLAW_QA_TELEGRAM_DRIVER_BOT_TOKEN
• text

  Copy

  OPENCLAW_QA_TELEGRAM_SUT_BOT_TOKEN

Optional:

• text

  Copy

  OPENCLAW_QA_TELEGRAM_CAPTURE_CONTENT=1
  keeps message bodies in observed-message artifacts (default redacts).

Scenarios (

):

• text

  Copy

  telegram-canary
• text

  Copy

  telegram-mention-gating
• text

  Copy

  telegram-mentioned-message-reply
• text

  Copy

  telegram-help-command
• text

  Copy

  telegram-commands-command
• text

  Copy

  telegram-tools-compact-command
• text

  Copy

  telegram-whoami-command
• text

  Copy

  telegram-context-command

Output artifacts:

• text

  Copy

  telegram-qa-report.md
• text

  Copy

  telegram-qa-summary.json
  — includes per-reply RTT (driver send → observed SUT reply) starting with the canary.
• text

  Copy

  telegram-qa-observed-messages.json
  — bodies redacted unless
  text

  Copy

  OPENCLAW_QA_TELEGRAM_CAPTURE_CONTENT=1
  .

Discord QA

bash
Copy
pnpm openclaw qa discord

Targets one real private Discord guild channel with two bots: a driver bot controlled by the harness and a SUT bot started by the child OpenClaw gateway through the bundled Discord plugin. Verifies channel mention handling and that the SUT bot has registered the native

command with Discord.

Required env when

:

• text

  Copy

  OPENCLAW_QA_DISCORD_GUILD_ID
• text

  Copy

  OPENCLAW_QA_DISCORD_CHANNEL_ID
• text

  Copy

  OPENCLAW_QA_DISCORD_DRIVER_BOT_TOKEN
• text

  Copy

  OPENCLAW_QA_DISCORD_SUT_BOT_TOKEN
• text

  Copy

  OPENCLAW_QA_DISCORD_SUT_APPLICATION_ID
  — must match the SUT bot user id returned by Discord (the lane fails fast otherwise).

Optional:

• text

  Copy

  OPENCLAW_QA_DISCORD_CAPTURE_CONTENT=1
  keeps message bodies in observed-message artifacts.

Scenarios (

):

• text

  Copy

  discord-canary
• text

  Copy

  discord-mention-gating
• text

  Copy

  discord-native-help-command-registration

Output artifacts:

• text

  Copy

  discord-qa-report.md
• text

  Copy

  discord-qa-summary.json
• text

  Copy

  discord-qa-observed-messages.json
  — bodies redacted unless
  text

  Copy

  OPENCLAW_QA_DISCORD_CAPTURE_CONTENT=1
  .

Convex credential pool

Both Telegram and Discord lanes can lease credentials from a shared Convex pool instead of reading the env vars above. Pass

(or set ); QA Lab acquires an exclusive lease, heartbeats it for the duration of the run, and releases it on shutdown. Pool kinds are and .

Payload shapes the broker validates on

:

• Telegram (
  text

  Copy

  kind: "telegram"
  ):
  text

  Copy

  { groupId: string, driverToken: string, sutToken: string }
  —
  text

  Copy

  groupId
  must be a numeric chat-id string.
• Discord (
  text

  Copy

  kind: "discord"
  ):
  text

  Copy

  { guildId: string, channelId: string, driverBotToken: string, sutBotToken: string, sutApplicationId: string }
  .

Operational env vars and the Convex broker endpoint contract live in Testing → Shared Telegram credentials via Convex (the section name predates Discord support; the broker semantics are identical for both kinds).

Repo-backed seeds

Seed assets live in

:

• text

  Copy

  qa/scenarios/index.md
• text

  Copy

  qa/scenarios/<theme>/*.md

These are intentionally in git so the QA plan is visible to both humans and the agent.

should stay a generic markdown runner. Each scenario markdown file is the source of truth for one test run and should define:

• scenario metadata
• optional category, capability, lane, and risk metadata
• docs and code refs
• optional plugin requirements
• optional gateway config patch
• the executable
  text

  Copy

  qa-flow

The reusable runtime surface that backs

is allowed to stay generic and cross-cutting. For example, markdown scenarios can combine transport-side helpers with browser-side helpers that drive the embedded Control UI through the Gateway seam without adding a special-case runner.

Scenario files should be grouped by product capability rather than source tree folder. Keep scenario IDs stable when files move; use

and for implementation traceability.

The baseline list should stay broad enough to cover:

• DM and channel chat
• thread behavior
• message action lifecycle
• cron callbacks
• memory recall
• model switching
• subagent handoff
• repo-reading and docs-reading
• one small build task such as Lobster Invaders

Provider mock lanes

has two local provider mock lanes:

• text

  Copy

  mock-openai
  is the scenario-aware OpenClaw mock. It remains the default deterministic mock lane for repo-backed QA and parity gates.
• text

  Copy

  aimock
  starts an AIMock-backed provider server for experimental protocol, fixture, record/replay, and chaos coverage. It is additive and does not replace the
  text

  Copy

  mock-openai
  scenario dispatcher.

Provider-lane implementation lives under

. Each provider owns its defaults, local server startup, gateway model config, auth-profile staging needs, and live/mock capability flags. Shared suite and gateway code should route through the provider registry instead of branching on provider names.

Transport adapters

owns a generic transport seam for markdown QA scenarios. is the first adapter on that seam, but the design target is wider: future real or synthetic channels should plug into the same suite runner instead of adding a transport-specific QA runner.

At the architecture level, the split is:

• text

  Copy

  qa-lab
  owns generic scenario execution, worker concurrency, artifact writing, and reporting.
• The transport adapter owns gateway config, readiness, inbound and outbound observation, transport actions, and normalized transport state.
• Markdown scenario files under
  text

  Copy

  qa/scenarios/
  define the test run;
  text

  Copy

  qa-lab
  provides the reusable runtime surface that executes them.

Adding a channel

Adding a channel to the markdown QA system requires exactly two things:

1. A transport adapter for the channel.
2. A scenario pack that exercises the channel contract.

Do not add a new top-level QA command root when the shared

host can own the flow.

owns the shared host mechanics:

• the
  text

  Copy

  openclaw qa
  command root
• suite startup and teardown
• worker concurrency
• artifact writing
• report generation
• scenario execution
• compatibility aliases for older
  text

  Copy

  qa-channel
  scenarios

Runner plugins own the transport contract:

• how
  text

  Copy

  openclaw qa <runner>
  is mounted beneath the shared
  text

  Copy

  qa
  root
• how the gateway is configured for that transport
• how readiness is checked
• how inbound events are injected
• how outbound messages are observed
• how transcripts and normalized transport state are exposed
• how transport-backed actions are executed
• how transport-specific reset or cleanup is handled

The minimum adoption bar for a new channel:

1. Keep
   text

   Copy

   qa-lab
   as the owner of the shared
   text

   Copy

   qa
   root.
2. Implement the transport runner on the shared
   text

   Copy

   qa-lab
   host seam.
3. Keep transport-specific mechanics inside the runner plugin or channel harness.
4. Mount the runner as
   text

   Copy

   openclaw qa <runner>
   instead of registering a competing root command. Runner plugins should declare
   text

   Copy

   qaRunners
   in
   text

   Copy

   openclaw.plugin.json
   and export a matching
   text

   Copy

   qaRunnerCliRegistrations
   array from
   text

   Copy

   runtime-api.ts
   . Keep
   text

   Copy

   runtime-api.ts
   light; lazy CLI and runner execution should stay behind separate entrypoints.
5. Author or adapt markdown scenarios under the themed
   text

   Copy

   qa/scenarios/
   directories.
6. Use the generic scenario helpers for new scenarios.
7. Keep existing compatibility aliases working unless the repo is doing an intentional migration.

The decision rule is strict:

• If behavior can be expressed once in
  text

  Copy

  qa-lab
  , put it in
  text

  Copy

  qa-lab
  .
• If behavior depends on one channel transport, keep it in that runner plugin or plugin harness.
• If a scenario needs a new capability that more than one channel can use, add a generic helper instead of a channel-specific branch in
  text

  Copy

  suite.ts
  .
• If a behavior is only meaningful for one transport, keep the scenario transport-specific and make that explicit in the scenario contract.

Scenario helper names

Preferred generic helpers for new scenarios:

• text

  Copy

  waitForTransportReady
• text

  Copy

  waitForChannelReady
• text

  Copy

  injectInboundMessage
• text

  Copy

  injectOutboundMessage
• text

  Copy

  waitForTransportOutboundMessage
• text

  Copy

  waitForChannelOutboundMessage
• text

  Copy

  waitForNoTransportOutbound
• text

  Copy

  getTransportSnapshot
• text

  Copy

  readTransportMessage
• text

  Copy

  readTransportTranscript
• text

  Copy

  formatTransportTranscript
• text

  Copy

  resetTransport

Compatibility aliases remain available for existing scenarios —

, , , , — but new scenario authoring should use the generic names. The aliases exist to avoid a flag-day migration, not as the model going forward.

Reporting

exports a Markdown protocol report from the observed bus timeline. The report should answer:

• What worked
• What failed
• What stayed blocked
• What follow-up scenarios are worth adding

For the inventory of available scenarios — useful when sizing follow-up work or wiring a new transport — run

(add for machine-readable output).

For character and style checks, run the same scenario across multiple live model refs and write a judged Markdown report:

bash
Copy
pnpm openclaw qa character-eval \
  --model openai/gpt-5.5,thinking=medium,fast \
  --model openai/gpt-5.2,thinking=xhigh \
  --model openai/gpt-5,thinking=xhigh \
  --model anthropic/claude-opus-4-6,thinking=high \
  --model anthropic/claude-sonnet-4-6,thinking=high \
  --model zai/glm-5.1,thinking=high \
  --model moonshot/kimi-k2.5,thinking=high \
  --model google/gemini-3.1-pro-preview,thinking=high \
  --judge-model openai/gpt-5.5,thinking=xhigh,fast \
  --judge-model anthropic/claude-opus-4-6,thinking=high \
  --blind-judge-models \
  --concurrency 16 \
  --judge-concurrency 16

The command runs local QA gateway child processes, not Docker. Character eval scenarios should set the persona through

, then run ordinary user turns such as chat, workspace help, and small file tasks. The candidate model should not be told that it is being evaluated. The command preserves each full transcript, records basic run stats, then asks the judge models in fast mode with reasoning where supported to rank the runs by naturalness, vibe, and humor. Use when comparing providers: the judge prompt still gets every transcript and run status, but candidate refs are replaced with neutral labels such as ; the report maps rankings back to real refs after parsing. Candidate runs default to thinking, with for GPT-5.5 and for older OpenAI eval refs that support it. Override a specific candidate inline with . still sets a global fallback, and the older form is kept for compatibility. OpenAI candidate refs default to fast mode so priority processing is used where the provider supports it. Add , , or inline when a single candidate or judge needs an override. Pass only when you want to force fast mode on for every candidate model. Candidate and judge durations are recorded in the report for benchmark analysis, but judge prompts explicitly say not to rank by speed. Candidate and judge model runs both default to concurrency 16. Lower or when provider limits or local gateway pressure make a run too noisy. When no candidate is passed, the character eval defaults to , , , , , , , and when no is passed. When no is passed, the judges default to and .

Related docs

• Matrix QA
• QA Channel
• Testing
• Dashboard