note

**QA stack (qa-lab, qa-channel, live transport lanes)** is documented separately:

• QA overview — architecture, command surface, scenario authoring.
• Matrix QA — reference for
  text

  Copy

  pnpm openclaw qa matrix
  .
• QA channel — the synthetic transport plugin used by repo-backed scenarios.

This page covers running the regular test suites and Docker/Parallels runners. The QA-specific runners section below (QA-specific runners) lists the concrete

invocations and points back at the references above.

Quick start

Most days:

• Full gate (expected before push):
  text

  Copy

  pnpm build && pnpm check && pnpm check:test-types && pnpm test
• Faster local full-suite run on a roomy machine:
  text

  Copy

  pnpm test:max
• Direct Vitest watch loop:
  text

  Copy

  pnpm test:watch
• Direct file targeting now routes extension/channel paths too:
  text

  Copy

  pnpm test extensions/discord/src/monitor/message-handler.preflight.test.ts
• Prefer targeted runs first when you are iterating on a single failure.
• Docker-backed QA site:
  text

  Copy

  pnpm qa:lab:up
• Linux VM-backed QA lane:
  text

  Copy

  pnpm openclaw qa suite --runner multipass --scenario channel-chat-baseline

When you touch tests or want extra confidence:

• Coverage gate:
  text

  Copy

  pnpm test:coverage
• E2E suite:
  text

  Copy

  pnpm test:e2e

When debugging real providers/models (requires real creds):

• Live suite (models + gateway tool/image probes):
  text

  Copy

  pnpm test:live
• Target one live file quietly:
  text

  Copy

  pnpm test:live -- src/agents/models.profiles.live.test.ts
• Docker live model sweep:
  text

  Copy

  pnpm test:docker:live-models
  • Each selected model now runs a text turn plus a small file-read-style probe. Models whose metadata advertises
    text

    Copy

    image
    input also run a tiny image turn. Disable the extra probes with
    text

    Copy

    OPENCLAW_LIVE_MODEL_FILE_PROBE=0
    or
    text

    Copy

    OPENCLAW_LIVE_MODEL_IMAGE_PROBE=0
    when isolating provider failures.
  • CI coverage: daily
    text

    Copy

    OpenClaw Scheduled Live And E2E Checks
    and manual
    text

    Copy

    OpenClaw Release Checks
    both call the reusable live/E2E workflow with
    text

    Copy

    include_live_suites: true
    , which includes separate Docker live model matrix jobs sharded by provider.
  • For focused CI reruns, dispatch
    text

    Copy

    OpenClaw Live And E2E Checks (Reusable)
    with
    text

    Copy

    include_live_suites: true
    and
    text

    Copy

    live_models_only: true
    .
  • Add new high-signal provider secrets to
    text

    Copy

    scripts/ci-hydrate-live-auth.sh
    plus
    text

    Copy

    .github/workflows/openclaw-live-and-e2e-checks-reusable.yml
    and its scheduled/release callers.
• Native Codex bound-chat smoke:
  text

  Copy

  pnpm test:docker:live-codex-bind
  • Runs a Docker live lane against the Codex app-server path, binds a synthetic Slack DM with
    text

    Copy

    /codex bind
    , exercises
    text

    Copy

    /codex fast
    and
    text

    Copy

    /codex permissions
    , then verifies a plain reply and an image attachment route through the native plugin binding instead of ACP.
• Codex app-server harness smoke:
  text

  Copy

  pnpm test:docker:live-codex-harness
  • Runs gateway agent turns through the plugin-owned Codex app-server harness, verifies
    text

    Copy

    /codex status
    and
    text

    Copy

    /codex models
    , and by default exercises image, cron MCP, sub-agent, and Guardian probes. Disable the sub-agent probe with
    text

    Copy

    OPENCLAW_LIVE_CODEX_HARNESS_SUBAGENT_PROBE=0
    when isolating other Codex app-server failures. For a focused sub-agent check, disable the other probes:
    text

    Copy

    OPENCLAW_LIVE_CODEX_HARNESS_IMAGE_PROBE=0 OPENCLAW_LIVE_CODEX_HARNESS_MCP_PROBE=0 OPENCLAW_LIVE_CODEX_HARNESS_GUARDIAN_PROBE=0 OPENCLAW_LIVE_CODEX_HARNESS_SUBAGENT_PROBE=1 pnpm test:docker:live-codex-harness
    . This exits after the sub-agent probe unless
    text

    Copy

    OPENCLAW_LIVE_CODEX_HARNESS_SUBAGENT_ONLY=0
    is set.
• Crestodian rescue command smoke:
  text

  Copy

  pnpm test:live:crestodian-rescue-channel
  • Opt-in belt-and-suspenders check for the message-channel rescue command surface. It exercises
    text

    Copy

    /crestodian status
    , queues a persistent model change, replies
    text

    Copy

    /crestodian yes
    , and verifies the audit/config write path.
• Crestodian planner Docker smoke:
  text

  Copy

  pnpm test:docker:crestodian-planner
  • Runs Crestodian in a configless container with a fake Claude CLI on
    text

    Copy

    PATH
    and verifies the fuzzy planner fallback translates into an audited typed config write.
• Crestodian first-run Docker smoke:
  text

  Copy

  pnpm test:docker:crestodian-first-run
  • Starts from an empty OpenClaw state dir, routes bare
    text

    Copy

    openclaw
    to Crestodian, applies setup/model/agent/Discord plugin + SecretRef writes, validates config, and verifies audit entries. The same Ring 0 setup path is also covered in QA Lab by
    text

    Copy

    pnpm openclaw qa suite --scenario crestodian-ring-zero-setup
    .
• Moonshot/Kimi cost smoke: with
  text

  Copy

  MOONSHOT_API_KEY
  set, run
  text

  Copy

  openclaw models list --provider moonshot --json
  , then run an isolated
  text

  Copy

  openclaw agent --local --session-id live-kimi-cost --message 'Reply exactly: KIMI_LIVE_OK' --thinking off --json
  against
  text

  Copy

  moonshot/kimi-k2.6
  . Verify the JSON reports Moonshot/K2.6 and the assistant transcript stores normalized
  text

  Copy

  usage.cost
  .

QA-specific runners

These commands sit beside the main test suites when you need QA-lab realism:

CI runs QA Lab in dedicated workflows.

runs on matching PRs and from manual dispatch with mock providers. runs nightly on and from manual dispatch with the mock parity gate, live Matrix lane, Convex-managed live Telegram lane, and Convex-managed live Discord lane as parallel jobs. Scheduled QA and release checks pass Matrix explicitly, while the Matrix CLI and manual workflow input default remain ; manual dispatch can shard into , , , , and jobs. runs parity plus the fast Matrix and Telegram lanes before release approval, using for release transport checks so they stay deterministic and avoid normal provider-plugin startup. These live transport gateways disable memory search; memory behavior stays covered by the QA parity suites.

Full release live media shards use

, which already has and . Docker live model/backend shards use the shared image built once per selected commit, then pull it with instead of rebuilding inside every shard.

• text

  Copy

  pnpm openclaw qa suite
  • Runs repo-backed QA scenarios directly on the host.
  • Runs multiple selected scenarios in parallel by default with isolated gateway workers.
    text

    Copy

    qa-channel
    defaults to concurrency 4 (bounded by the selected scenario count). Use
    text

    Copy

    --concurrency <count>
    to tune the worker count, or
    text

    Copy

    --concurrency 1
    for the older serial lane.
  • Exits non-zero when any scenario fails. Use
    text

    Copy

    --allow-failures
    when you want artifacts without a failing exit code.
  • Supports provider modes
    text

    Copy

    live-frontier
    ,
    text

    Copy

    mock-openai
    , and
    text

    Copy

    aimock
    .
    text

    Copy

    aimock
    starts a local AIMock-backed provider server for experimental fixture and protocol-mock coverage without replacing the scenario-aware
    text

    Copy

    mock-openai
    lane.
• text

  Copy

  pnpm test:gateway:cpu-scenarios
  • Runs the gateway startup bench plus a small mock QA Lab scenario pack (
    text

    Copy

    channel-chat-baseline
    ,
    text

    Copy

    memory-failure-fallback
    ,
    text

    Copy

    gateway-restart-inflight-run
    ) and writes a combined CPU observation summary under
    text

    Copy

    .artifacts/gateway-cpu-scenarios/
    .
  • Flags only sustained hot CPU observations by default (
    text

    Copy

    --cpu-core-warn
    plus
    text

    Copy

    --hot-wall-warn-ms
    ), so short startup bursts are recorded as metrics without looking like the minutes-long gateway peg regression.
  • Uses built
    text

    Copy

    dist
    artifacts; run a build first when the checkout does not already have fresh runtime output.
• text

  Copy

  pnpm openclaw qa suite --runner multipass
  • Runs the same QA suite inside a disposable Multipass Linux VM.
  • Keeps the same scenario-selection behavior as
    text

    Copy

    qa suite
    on the host.
  • Reuses the same provider/model selection flags as
    text

    Copy

    qa suite
    .
  • Live runs forward the supported QA auth inputs that are practical for the guest: env-based provider keys, the QA live provider config path, and
    text

    Copy

    CODEX_HOME
    when present.
  • Output dirs must stay under the repo root so the guest can write back through the mounted workspace.
  • Writes the normal QA report + summary plus Multipass logs under
    text

    Copy

    .artifacts/qa-e2e/...
    .
• text

  Copy

  pnpm qa:lab:up
  • Starts the Docker-backed QA site for operator-style QA work.
• text

  Copy

  pnpm test:docker:npm-onboard-channel-agent
  • Builds an npm tarball from the current checkout, installs it globally in Docker, runs non-interactive OpenAI API-key onboarding, configures Telegram by default, verifies enabling the plugin installs runtime dependencies on demand, runs doctor, and runs one local agent turn against a mocked OpenAI endpoint.
  • Use
    text

    Copy

    OPENCLAW_NPM_ONBOARD_CHANNEL=discord
    to run the same packaged-install lane with Discord.
• text

  Copy

  pnpm test:docker:session-runtime-context
  • Runs a deterministic built-app Docker smoke for embedded runtime context transcripts. It verifies hidden OpenClaw runtime context is persisted as a non-display custom message instead of leaking into the visible user turn, then seeds an affected broken session JSONL and verifies
    text

    Copy

    openclaw doctor --fix
    rewrites it to the active branch with a backup.
• text

  Copy

  pnpm test:docker:npm-telegram-live
  • Installs an OpenClaw package candidate in Docker, runs installed-package onboarding, configures Telegram through the installed CLI, then reuses the live Telegram QA lane with that installed package as the SUT Gateway.
  • Defaults to
    text

    Copy

    OPENCLAW_NPM_TELEGRAM_PACKAGE_SPEC=openclaw@beta
    ; set
    text

    Copy

    OPENCLAW_NPM_TELEGRAM_PACKAGE_TGZ=/path/to/openclaw-current.tgz
    or
    text

    Copy

    OPENCLAW_CURRENT_PACKAGE_TGZ
    to test a resolved local tarball instead of installing from the registry.
  • Uses the same Telegram env credentials or Convex credential source as
    text

    Copy

    pnpm openclaw qa telegram
    . For CI/release automation, set
    text

    Copy

    OPENCLAW_NPM_TELEGRAM_CREDENTIAL_SOURCE=convex
    plus
    text

    Copy

    OPENCLAW_QA_CONVEX_SITE_URL
    and the role secret. If
    text

    Copy

    OPENCLAW_QA_CONVEX_SITE_URL
    and a Convex role secret are present in CI, the Docker wrapper selects Convex automatically.
  • text

    Copy

    OPENCLAW_NPM_TELEGRAM_CREDENTIAL_ROLE=ci|maintainer
    overrides the shared
    text

    Copy

    OPENCLAW_QA_CREDENTIAL_ROLE
    for this lane only.
  • GitHub Actions exposes this lane as the manual maintainer workflow
    text

    Copy

    NPM Telegram Beta E2E
    . It does not run on merge. The workflow uses the
    text

    Copy

    qa-live-shared
    environment and Convex CI credential leases.
• GitHub Actions also exposes
  text

  Copy

  Package Acceptance
  for side-run product proof against one candidate package. It accepts a trusted ref, published npm spec, HTTPS tarball URL plus SHA-256, or tarball artifact from another run, uploads the normalized
  text

  Copy

  openclaw-current.tgz
  as
  text

  Copy

  package-under-test
  , then runs the existing Docker E2E scheduler with smoke, package, product, full, or custom lane profiles. Set
  text

  Copy

  telegram_mode=mock-openai
  or
  text

  Copy

  live-frontier
  to run the Telegram QA workflow against the same
  text

  Copy

  package-under-test
  artifact.
  • Latest beta product proof:

bash
Copy
gh workflow run package-acceptance.yml --ref main \
  -f source=npm \
  -f package_spec=openclaw@beta \
  -f suite_profile=product \
  -f telegram_mode=mock-openai

• Exact tarball URL proof requires a digest:

bash
Copy
gh workflow run package-acceptance.yml --ref main \
  -f source=url \
  -f package_url=https://registry.npmjs.org/openclaw/-/openclaw-VERSION.tgz \
  -f package_sha256=<sha256> \
  -f suite_profile=package

• Artifact proof downloads a tarball artifact from another Actions run:

bash
Copy
gh workflow run package-acceptance.yml --ref main \
  -f source=artifact \
  -f artifact_run_id=<run-id> \
  -f artifact_name=<artifact-name> \
  -f suite_profile=smoke

• text

  Copy

  pnpm test:docker:bundled-channel-deps
  • Packs and installs the current OpenClaw build in Docker, starts the Gateway with OpenAI configured, then enables bundled channel/plugins via config edits.
  • Verifies setup discovery leaves unconfigured plugin runtime dependencies absent, the first configured Gateway or doctor run installs each bundled plugin's runtime dependencies on demand, and a second restart does not reinstall dependencies that were already activated.
  • Also installs a known older npm baseline, enables Telegram before running
    text

    Copy

    openclaw update --tag <candidate>
    , and verifies the candidate's post-update doctor repairs bundled channel runtime dependencies without a harness-side postinstall repair.

• text

  Copy

  pnpm test:parallels:npm-update

  • Runs the native packaged-install update smoke across Parallels guests. Each selected platform first installs the requested baseline package, then runs the installed

    text

    Copy

    openclaw update
    command in the same guest and verifies the installed version, update status, gateway readiness, and one local agent turn.

  • Use

    text

    Copy

    --platform macos
    ,
    text

    Copy

    --platform windows
    , or
    text

    Copy

    --platform linux
    while iterating on one guest. Use
    text

    Copy

    --json
    for the summary artifact path and per-lane status.

  • The OpenAI lane uses

    text

    Copy

    openai/gpt-5.5
    for the live agent-turn proof by default. Pass
    text

    Copy

    --model <provider/model>
    or set
    text

    Copy

    OPENCLAW_PARALLELS_OPENAI_MODEL
    when deliberately validating another OpenAI model.

  • Wrap long local runs in a host timeout so Parallels transport stalls cannot consume the rest of the testing window:

    bash
    Copy
    timeout --foreground 150m pnpm test:parallels:npm-update -- --json
    timeout --foreground 90m pnpm test:parallels:npm-update -- --platform windows --json
    

  • The script writes nested lane logs under

    text

    Copy

    /tmp/openclaw-parallels-npm-update.*
    . Inspect
    text

    Copy

    windows-update.log
    ,
    text

    Copy

    macos-update.log
    , or
    text

    Copy

    linux-update.log
    before assuming the outer wrapper is hung.

  • Windows update can spend 10 to 15 minutes in post-update doctor/runtime dependency repair on a cold guest; that is still healthy when the nested npm debug log is advancing.

  • Do not run this aggregate wrapper in parallel with individual Parallels macOS, Windows, or Linux smoke lanes. They share VM state and can collide on snapshot restore, package serving, or guest gateway state.

  • The post-update proof runs the normal bundled plugin surface because capability facades such as speech, image generation, and media understanding are loaded through bundled runtime APIs even when the agent turn itself only checks a simple text response.

• text

  Copy

  pnpm openclaw qa aimock
  • Starts only the local AIMock provider server for direct protocol smoke testing.

• text

  Copy

  pnpm openclaw qa matrix
  • Runs the Matrix live QA lane against a disposable Docker-backed Tuwunel homeserver. Source-checkout only — packaged installs do not ship
    text

    Copy

    qa-lab
    .
  • Full CLI, profile/scenario catalog, env vars, and artifact layout: Matrix QA.

• text

  Copy

  pnpm openclaw qa telegram
  • Runs the Telegram live QA lane against a real private group using the driver and SUT bot tokens from env.
  • Requires
    text

    Copy

    OPENCLAW_QA_TELEGRAM_GROUP_ID
    ,
    text

    Copy

    OPENCLAW_QA_TELEGRAM_DRIVER_BOT_TOKEN
    , and
    text

    Copy

    OPENCLAW_QA_TELEGRAM_SUT_BOT_TOKEN
    . The group id must be the numeric Telegram chat id.
  • Supports
    text

    Copy

    --credential-source convex
    for shared pooled credentials. Use env mode by default, or set
    text

    Copy

    OPENCLAW_QA_CREDENTIAL_SOURCE=convex
    to opt into pooled leases.
  • Exits non-zero when any scenario fails. Use
    text

    Copy

    --allow-failures
    when you want artifacts without a failing exit code.
  • Requires two distinct bots in the same private group, with the SUT bot exposing a Telegram username.
  • For stable bot-to-bot observation, enable Bot-to-Bot Communication Mode in
    text

    Copy

    @BotFather
    for both bots and ensure the driver bot can observe group bot traffic.
  • Writes a Telegram QA report, summary, and observed-messages artifact under
    text

    Copy

    .artifacts/qa-e2e/...
    . Replying scenarios include RTT from driver send request to observed SUT reply.

Live transport lanes share one standard contract so new transports do not drift; the per-lane coverage matrix lives in QA overview → Live transport coverage.

is the broad synthetic suite and is not part of that matrix.

Shared Telegram credentials via Convex (v1)

When

(or ) is enabled for , QA lab acquires an exclusive lease from a Convex-backed pool, heartbeats that lease while the lane is running, and releases the lease on shutdown.

Reference Convex project scaffold:

• text

  Copy

  qa/convex-credential-broker/

Required env vars:

• text

  Copy

  OPENCLAW_QA_CONVEX_SITE_URL
  (for example
  text

  Copy

  https://your-deployment.convex.site
  )
• One secret for the selected role:
  • text

    Copy

    OPENCLAW_QA_CONVEX_SECRET_MAINTAINER
    for
    text

    Copy

    maintainer
  • text

    Copy

    OPENCLAW_QA_CONVEX_SECRET_CI
    for
    text

    Copy

    ci
• Credential role selection:
  • CLI:
    text

    Copy

    --credential-role maintainer|ci
  • Env default:
    text

    Copy

    OPENCLAW_QA_CREDENTIAL_ROLE
    (defaults to
    text

    Copy

    ci
    in CI,
    text

    Copy

    maintainer
    otherwise)

Optional env vars:

• text

  Copy

  OPENCLAW_QA_CREDENTIAL_LEASE_TTL_MS
  (default
  text

  Copy

  1200000
  )
• text

  Copy

  OPENCLAW_QA_CREDENTIAL_HEARTBEAT_INTERVAL_MS
  (default
  text

  Copy

  30000
  )
• text

  Copy

  OPENCLAW_QA_CREDENTIAL_ACQUIRE_TIMEOUT_MS
  (default
  text

  Copy

  90000
  )
• text

  Copy

  OPENCLAW_QA_CREDENTIAL_HTTP_TIMEOUT_MS
  (default
  text

  Copy

  15000
  )
• text

  Copy

  OPENCLAW_QA_CONVEX_ENDPOINT_PREFIX
  (default
  text

  Copy

  /qa-credentials/v1
  )
• text

  Copy

  OPENCLAW_QA_CREDENTIAL_OWNER_ID
  (optional trace id)
• text

  Copy

  OPENCLAW_QA_ALLOW_INSECURE_HTTP=1
  allows loopback
  text

  Copy

  http://
  Convex URLs for local-only development.

should use in normal operation.

Maintainer admin commands (pool add/remove/list) require

specifically.

CLI helpers for maintainers:

bash
Copy
pnpm openclaw qa credentials doctor
pnpm openclaw qa credentials add --kind telegram --payload-file qa/telegram-credential.json
pnpm openclaw qa credentials list --kind telegram
pnpm openclaw qa credentials remove --credential-id <credential-id>

Use

before live runs to check the Convex site URL, broker secrets, endpoint prefix, HTTP timeout, and admin/list reachability without printing secret values. Use for machine-readable output in scripts and CI utilities.

Default endpoint contract (

+ ):

• text

  Copy

  POST /acquire
  • Request:
    text

    Copy

    { kind, ownerId, actorRole, leaseTtlMs, heartbeatIntervalMs }
  • Success:
    text

    Copy

    { status: "ok", credentialId, leaseToken, payload, leaseTtlMs?, heartbeatIntervalMs? }
  • Exhausted/retryable:
    text

    Copy

    { status: "error", code: "POOL_EXHAUSTED" | "NO_CREDENTIAL_AVAILABLE", ... }
• text

  Copy

  POST /heartbeat
  • Request:
    text

    Copy

    { kind, ownerId, actorRole, credentialId, leaseToken, leaseTtlMs }
  • Success:
    text

    Copy

    { status: "ok" }
    (or empty
    text

    Copy

    2xx
    )
• text

  Copy

  POST /release
  • Request:
    text

    Copy

    { kind, ownerId, actorRole, credentialId, leaseToken }
  • Success:
    text

    Copy

    { status: "ok" }
    (or empty
    text

    Copy

    2xx
    )
• text

  Copy

  POST /admin/add
  (maintainer secret only)
  • Request:
    text

    Copy

    { kind, actorId, payload, note?, status? }
  • Success:
    text

    Copy

    { status: "ok", credential }
• text

  Copy

  POST /admin/remove
  (maintainer secret only)
  • Request:
    text

    Copy

    { credentialId, actorId }
  • Success:
    text

    Copy

    { status: "ok", changed, credential }
  • Active lease guard:
    text

    Copy

    { status: "error", code: "LEASE_ACTIVE", ... }
• text

  Copy

  POST /admin/list
  (maintainer secret only)
  • Request:
    text

    Copy

    { kind?, status?, includePayload?, limit? }
  • Success:
    text

    Copy

    { status: "ok", credentials, count }

Payload shape for Telegram kind:

• text

  Copy

  { groupId: string, driverToken: string, sutToken: string }
• text

  Copy

  groupId
  must be a numeric Telegram chat id string.
• text

  Copy

  admin/add
  validates this shape for
  text

  Copy

  kind: "telegram"
  and rejects malformed payloads.

Adding a channel to QA

The architecture and scenario-helper names for new channel adapters live in QA overview → Adding a channel. The minimum bar: implement the transport runner on the shared

host seam, declare in the plugin manifest, mount as , and author scenarios under .

Test suites (what runs where)

Think of the suites as “increasing realism” (and increasing flakiness/cost):

Unit / integration (default)

• Command:
  text

  Copy

  pnpm test
• Config: untargeted runs use the
  text

  Copy

  vitest.full-*.config.ts
  shard set and may expand multi-project shards into per-project configs for parallel scheduling
• Files: core/unit inventories under
  text

  Copy

  src/**/*.test.ts
  ,
  text

  Copy

  packages/**/*.test.ts
  , and
  text

  Copy

  test/**/*.test.ts
  ; UI unit tests run in the dedicated
  text

  Copy

  unit-ui
  shard
• Scope:
  • Pure unit tests
  • In-process integration tests (gateway auth, routing, tooling, parsing, config)
  • Deterministic regressions for known bugs
• Expectations:
  • Runs in CI
  • No real keys required
  • Should be fast and stable
  • Resolver and public-surface loader tests must prove broad
    text

    Copy

    api.js
    and
    text

    Copy

    runtime-api.js
    fallback behavior with generated tiny plugin fixtures, not real bundled plugin source APIs. Real plugin API loads belong in plugin-owned contract/integration suites.

Projects, shards, and scoped lanes

Embedded runner coverage

Vitest pool and isolation defaults

Fast local iteration

Perf debugging

Stability (gateway)

• Command:
  text

  Copy

  pnpm test:stability:gateway
• Config:
  text

  Copy

  vitest.gateway.config.ts
  , forced to one worker
• Scope:
  • Starts a real loopback Gateway with diagnostics enabled by default
  • Drives synthetic gateway message, memory, and large-payload churn through the diagnostic event path
  • Queries
    text

    Copy

    diagnostics.stability
    over the Gateway WS RPC
  • Covers diagnostic stability bundle persistence helpers
  • Asserts the recorder remains bounded, synthetic RSS samples stay under the pressure budget, and per-session queue depths drain back to zero
• Expectations:
  • CI-safe and keyless
  • Narrow lane for stability-regression follow-up, not a substitute for the full Gateway suite

E2E (gateway smoke)

• Command:
  text

  Copy

  pnpm test:e2e
• Config:
  text

  Copy

  vitest.e2e.config.ts
• Files:
  text

  Copy

  src/**/*.e2e.test.ts
  ,
  text

  Copy

  test/**/*.e2e.test.ts
  , and bundled-plugin E2E tests under
  text

  Copy

  extensions/
• Runtime defaults:
  • Uses Vitest
    text

    Copy

    threads
    with
    text

    Copy

    isolate: false
    , matching the rest of the repo.
  • Uses adaptive workers (CI: up to 2, local: 1 by default).
  • Runs in silent mode by default to reduce console I/O overhead.
• Useful overrides:
  • text

    Copy

    OPENCLAW_E2E_WORKERS=<n>
    to force worker count (capped at 16).
  • text

    Copy

    OPENCLAW_E2E_VERBOSE=1
    to re-enable verbose console output.
• Scope:
  • Multi-instance gateway end-to-end behavior
  • WebSocket/HTTP surfaces, node pairing, and heavier networking
• Expectations:
  • Runs in CI (when enabled in the pipeline)
  • No real keys required
  • More moving parts than unit tests (can be slower)

E2E: OpenShell backend smoke

• Command:
  text

  Copy

  pnpm test:e2e:openshell
• File:
  text

  Copy

  extensions/openshell/src/backend.e2e.test.ts
• Scope:
  • Starts an isolated OpenShell gateway on the host via Docker
  • Creates a sandbox from a temporary local Dockerfile
  • Exercises OpenClaw's OpenShell backend over real
    text

    Copy

    sandbox ssh-config
    + SSH exec
  • Verifies remote-canonical filesystem behavior through the sandbox fs bridge
• Expectations:
  • Opt-in only; not part of the default
    text

    Copy

    pnpm test:e2e
    run
  • Requires a local
    text

    Copy

    openshell
    CLI plus a working Docker daemon
  • Uses isolated
    text

    Copy

    HOME
    /
    text

    Copy

    XDG_CONFIG_HOME
    , then destroys the test gateway and sandbox
• Useful overrides:
  • text

    Copy

    OPENCLAW_E2E_OPENSHELL=1
    to enable the test when running the broader e2e suite manually
  • text

    Copy

    OPENCLAW_E2E_OPENSHELL_COMMAND=/path/to/openshell
    to point at a non-default CLI binary or wrapper script

Live (real providers + real models)

• Command:
  text

  Copy

  pnpm test:live
• Config:
  text

  Copy

  vitest.live.config.ts
• Files:
  text

  Copy

  src/**/*.live.test.ts
  ,
  text

  Copy

  test/**/*.live.test.ts
  , and bundled-plugin live tests under
  text

  Copy

  extensions/
• Default: enabled by
  text

  Copy

  pnpm test:live
  (sets
  text

  Copy

  OPENCLAW_LIVE_TEST=1
  )
• Scope:
  • “Does this provider/model actually work today with real creds?”
  • Catch provider format changes, tool-calling quirks, auth issues, and rate limit behavior
• Expectations:
  • Not CI-stable by design (real networks, real provider policies, quotas, outages)
  • Costs money / uses rate limits
  • Prefer running narrowed subsets instead of “everything”
• Live runs source
  text

  Copy

  ~/.profile
  to pick up missing API keys.
• By default, live runs still isolate
  text

  Copy

  HOME
  and copy config/auth material into a temp test home so unit fixtures cannot mutate your real
  text

  Copy

  ~/.openclaw
  .
• Set
  text

  Copy

  OPENCLAW_LIVE_USE_REAL_HOME=1
  only when you intentionally need live tests to use your real home directory.
• text

  Copy

  pnpm test:live
  now defaults to a quieter mode: it keeps
  text

  Copy

  [live] ...
  progress output, but suppresses the extra
  text

  Copy

  ~/.profile
  notice and mutes gateway bootstrap logs/Bonjour chatter. Set
  text

  Copy

  OPENCLAW_LIVE_TEST_QUIET=0
  if you want the full startup logs back.
• API key rotation (provider-specific): set
  text

  Copy

  *_API_KEYS
  with comma/semicolon format or
  text

  Copy

  *_API_KEY_1
  ,
  text

  Copy

  *_API_KEY_2
  (for example
  text

  Copy

  OPENAI_API_KEYS
  ,
  text

  Copy

  ANTHROPIC_API_KEYS
  ,
  text

  Copy

  GEMINI_API_KEYS
  ) or per-live override via
  text

  Copy

  OPENCLAW_LIVE_*_KEY
  ; tests retry on rate limit responses.
• Progress/heartbeat output:
  • Live suites now emit progress lines to stderr so long provider calls are visibly active even when Vitest console capture is quiet.
  • text

    Copy

    vitest.live.config.ts
    disables Vitest console interception so provider/gateway progress lines stream immediately during live runs.
  • Tune direct-model heartbeats with
    text

    Copy

    OPENCLAW_LIVE_HEARTBEAT_MS
    .
  • Tune gateway/probe heartbeats with
    text

    Copy

    OPENCLAW_LIVE_GATEWAY_HEARTBEAT_MS
    .

Which suite should I run?

Use this decision table:

• Editing logic/tests: run
  text

  Copy

  pnpm test
  (and
  text

  Copy

  pnpm test:coverage
  if you changed a lot)
• Touching gateway networking / WS protocol / pairing: add
  text

  Copy

  pnpm test:e2e
• Debugging “my bot is down” / provider-specific failures / tool calling: run a narrowed
  text

  Copy

  pnpm test:live

Live (network-touching) tests

For the live model matrix, CLI backend smokes, ACP smokes, Codex app-server harness, and all media-provider live tests (Deepgram, BytePlus, ComfyUI, image, music, video, media harness) — plus credential handling for live runs — see Testing — live suites.

Docker runners (optional "works in Linux" checks)

These Docker runners split into two buckets:

• Live-model runners:
  text

  Copy

  test:docker:live-models
  and
  text

  Copy

  test:docker:live-gateway
  run only their matching profile-key live file inside the repo Docker image (
  text

  Copy

  src/agents/models.profiles.live.test.ts
  and
  text

  Copy

  src/gateway/gateway-models.profiles.live.test.ts
  ), mounting your local config dir and workspace (and sourcing
  text

  Copy

  ~/.profile
  if mounted). The matching local entrypoints are
  text

  Copy

  test:live:models-profiles
  and
  text

  Copy

  test:live:gateway-profiles
  .
• Docker live runners default to a smaller smoke cap so a full Docker sweep stays practical:
  text

  Copy

  test:docker:live-models
  defaults to
  text

  Copy

  OPENCLAW_LIVE_MAX_MODELS=12
  , and
  text

  Copy

  test:docker:live-gateway
  defaults to
  text

  Copy

  OPENCLAW_LIVE_GATEWAY_SMOKE=1
  ,
  text

  Copy

  OPENCLAW_LIVE_GATEWAY_MAX_MODELS=8
  ,
  text

  Copy

  OPENCLAW_LIVE_GATEWAY_STEP_TIMEOUT_MS=45000
  , and
  text

  Copy

  OPENCLAW_LIVE_GATEWAY_MODEL_TIMEOUT_MS=90000
  . Override those env vars when you explicitly want the larger exhaustive scan.
• text

  Copy

  test:docker:all
  builds the live Docker image once via
  text

  Copy

  test:docker:live-build
  , packs OpenClaw once as an npm tarball through
  text

  Copy

  scripts/package-openclaw-for-docker.mjs
  , then builds/reuses two
  text

  Copy

  scripts/e2e/Dockerfile
  images. The bare image is only the Node/Git runner for install/update/plugin-dependency lanes; those lanes mount the prebuilt tarball. The functional image installs the same tarball into
  text

  Copy

  /app
  for built-app functionality lanes. Docker lane definitions live in
  text

  Copy

  scripts/lib/docker-e2e-scenarios.mjs
  ; planner logic lives in
  text

  Copy

  scripts/lib/docker-e2e-plan.mjs
  ;
  text

  Copy

  scripts/test-docker-all.mjs
  executes the selected plan. The aggregate uses a weighted local scheduler:
  text

  Copy

  OPENCLAW_DOCKER_ALL_PARALLELISM
  controls process slots, while resource caps keep heavy live, npm-install, and multi-service lanes from all starting at once. If a single lane is heavier than the active caps, the scheduler can still start it when the pool is empty and then keeps it running alone until capacity is available again. Defaults are 10 slots,
  text

  Copy

  OPENCLAW_DOCKER_ALL_LIVE_LIMIT=9
  ,
  text

  Copy

  OPENCLAW_DOCKER_ALL_NPM_LIMIT=10
  , and
  text

  Copy

  OPENCLAW_DOCKER_ALL_SERVICE_LIMIT=7
  ; tune
  text

  Copy

  OPENCLAW_DOCKER_ALL_WEIGHT_LIMIT
  or
  text

  Copy

  OPENCLAW_DOCKER_ALL_DOCKER_LIMIT
  only when the Docker host has more headroom. The runner performs a Docker preflight by default, removes stale OpenClaw E2E containers, prints status every 30 seconds, stores successful lane timings in
  text

  Copy

  .artifacts/docker-tests/lane-timings.json
  , and uses those timings to start longer lanes first on later runs. Use
  text

  Copy

  OPENCLAW_DOCKER_ALL_DRY_RUN=1
  to print the weighted lane manifest without building or running Docker, or
  text

  Copy

  node scripts/test-docker-all.mjs --plan-json
  to print the CI plan for selected lanes, package/image needs, and credentials.
• text

  Copy

  Package Acceptance
  is the GitHub-native package gate for "does this installable tarball work as a product?" It resolves one candidate package from
  text

  Copy

  source=npm
  ,
  text

  Copy

  source=ref
  ,
  text

  Copy

  source=url
  , or
  text

  Copy

  source=artifact
  , uploads it as
  text

  Copy

  package-under-test
  , then runs the reusable Docker E2E lanes against that exact tarball instead of repacking the selected ref.
  text

  Copy

  workflow_ref
  selects the trusted workflow/harness scripts, while
  text

  Copy

  package_ref
  selects the source commit/branch/tag to pack when
  text

  Copy

  source=ref
  ; this lets current acceptance logic validate older trusted commits. Profiles are ordered by breadth:
  text

  Copy

  smoke
  is quick install/channel/agent plus gateway/config,
  text

  Copy

  package
  is the package/update/plugin contract plus the keyless upgrade-survivor fixture, the published-baseline upgrade survivor lane, and the default native replacement for most Parallels package/update coverage,
  text

  Copy

  product
  adds MCP channels, cron/subagent cleanup, OpenAI web search, and OpenWebUI, and
  text

  Copy

  full
  runs the release-path Docker chunks with OpenWebUI. For
  text

  Copy

  published-upgrade-survivor
  , Package Acceptance always uses
  text

  Copy

  package-under-test
  as the candidate and
  text

  Copy

  published_upgrade_survivor_baseline
  as the published baseline, defaulting to
  text

  Copy

  openclaw@latest
  ; shard broader coverage by dispatching multiple runs with exact baseline values. The published lane configures its baseline with a baked
  text

  Copy

  openclaw config set
  command recipe, then records recipe steps in the lane summary. Release validation runs a custom package delta (
  text

  Copy

  bundled-channel-deps-compat plugins-offline
  ) plus Telegram package QA because the release-path Docker chunks already cover the overlapping package/update/plugin lanes. Targeted GitHub Docker rerun commands generated from artifacts include prior package artifact, prepared image inputs, and the published upgrade-survivor baseline when available, so failed lanes can avoid rebuilding the package and images.
• Build and release checks run
  text

  Copy

  scripts/check-cli-bootstrap-imports.mjs
  after tsdown. The guard walks the static built graph from
  text

  Copy

  dist/entry.js
  and
  text

  Copy

  dist/cli/run-main.js
  and fails if pre-dispatch startup imports package dependencies such as Commander, prompt UI, undici, or logging before command dispatch; it also keeps the bundled gateway run chunk under budget and rejects static imports of known cold gateway paths. Packaged CLI smoke also covers root help, onboard help, doctor help, status, config schema, and a model-list command.
• Package Acceptance legacy compatibility is capped at
  text

  Copy

  2026.4.25
  (
  text

  Copy

  2026.4.25-beta.*
  included). Through that cutoff, the harness tolerates only shipped-package metadata gaps: omitted private QA inventory entries, missing
  text

  Copy

  gateway install --wrapper
  , missing patch files in the tarball-derived git fixture, missing persisted
  text

  Copy

  update.channel
  , legacy plugin install-record locations, missing marketplace install-record persistence, and config metadata migration during
  text

  Copy

  plugins update
  . For packages after
  text

  Copy

  2026.4.25
  , those paths are strict failures.
• Container smoke runners:
  text

  Copy

  test:docker:openwebui
  ,
  text

  Copy

  test:docker:onboard
  ,
  text

  Copy

  test:docker:npm-onboard-channel-agent
  ,
  text

  Copy

  test:docker:update-channel-switch
  ,
  text

  Copy

  test:docker:upgrade-survivor
  ,
  text

  Copy

  test:docker:published-upgrade-survivor
  ,
  text

  Copy

  test:docker:session-runtime-context
  ,
  text

  Copy

  test:docker:agents-delete-shared-workspace
  ,
  text

  Copy

  test:docker:gateway-network
  ,
  text

  Copy

  test:docker:browser-cdp-snapshot
  ,
  text

  Copy

  test:docker:mcp-channels
  ,
  text

  Copy

  test:docker:pi-bundle-mcp-tools
  ,
  text

  Copy

  test:docker:cron-mcp-cleanup
  ,
  text

  Copy

  test:docker:plugins
  ,
  text

  Copy

  test:docker:plugin-update
  , and
  text

  Copy

  test:docker:config-reload
  boot one or more real containers and verify higher-level integration paths.

The live-model Docker runners also bind-mount only the needed CLI auth homes (or all supported ones when the run is not narrowed), then copy them into the container home before the run so external-CLI OAuth can refresh tokens without mutating the host auth store:

• Direct models:
  text

  Copy

  pnpm test:docker:live-models
  (script:
  text

  Copy

  scripts/test-live-models-docker.sh
  )
• ACP bind smoke:
  text

  Copy

  pnpm test:docker:live-acp-bind
  (script:
  text

  Copy

  scripts/test-live-acp-bind-docker.sh
  ; covers Claude, Codex, and Gemini by default, with strict Droid/OpenCode coverage via
  text

  Copy

  pnpm test:docker:live-acp-bind:droid
  and
  text

  Copy

  pnpm test:docker:live-acp-bind:opencode
  )
• CLI backend smoke:
  text

  Copy

  pnpm test:docker:live-cli-backend
  (script:
  text

  Copy

  scripts/test-live-cli-backend-docker.sh
  )
• Codex app-server harness smoke:
  text

  Copy

  pnpm test:docker:live-codex-harness
  (script:
  text

  Copy

  scripts/test-live-codex-harness-docker.sh
  )
• Gateway + dev agent:
  text

  Copy

  pnpm test:docker:live-gateway
  (script:
  text

  Copy

  scripts/test-live-gateway-models-docker.sh
  )
• Observability smoke:
  text

  Copy

  pnpm qa:otel:smoke
  is a private QA source-checkout lane. It is intentionally not part of package Docker release lanes because the npm tarball omits QA Lab.
• Open WebUI live smoke:
  text

  Copy

  pnpm test:docker:openwebui
  (script:
  text

  Copy

  scripts/e2e/openwebui-docker.sh
  )
• Onboarding wizard (TTY, full scaffolding):
  text

  Copy

  pnpm test:docker:onboard
  (script:
  text

  Copy

  scripts/e2e/onboard-docker.sh
  )
• Npm tarball onboarding/channel/agent smoke:
  text

  Copy

  pnpm test:docker:npm-onboard-channel-agent
  installs the packed OpenClaw tarball globally in Docker, configures OpenAI via env-ref onboarding plus Telegram by default, verifies doctor repairs activated plugin runtime deps, and runs one mocked OpenAI agent turn. Reuse a prebuilt tarball with
  text

  Copy

  OPENCLAW_CURRENT_PACKAGE_TGZ=/path/to/openclaw-*.tgz
  , skip the host rebuild with
  text

  Copy

  OPENCLAW_NPM_ONBOARD_HOST_BUILD=0
  , or switch channel with
  text

  Copy

  OPENCLAW_NPM_ONBOARD_CHANNEL=discord
  .
• Update channel switch smoke:
  text

  Copy

  pnpm test:docker:update-channel-switch
  installs the packed OpenClaw tarball globally in Docker, switches from package
  text

  Copy

  stable
  to git
  text

  Copy

  dev
  , verifies the persisted channel and plugin post-update work, then switches back to package
  text

  Copy

  stable
  and checks update status.
• Upgrade survivor smoke:
  text

  Copy

  pnpm test:docker:upgrade-survivor
  installs the packed OpenClaw tarball over a dirty old-user fixture with agents, channel config, plugin allowlists, stale plugin runtime-deps state, and existing workspace/session files. It runs package update plus non-interactive doctor without live provider or channel keys, then starts a loopback Gateway and checks config/state preservation plus startup/status budgets.
• Published upgrade survivor smoke:
  text

  Copy

  pnpm test:docker:published-upgrade-survivor
  installs
  text

  Copy

  openclaw@latest
  by default, seeds realistic existing-user files, configures that baseline with a baked command recipe, validates the resulting config, updates that published install to the candidate tarball, runs non-interactive doctor, writes
  text

  Copy

  .artifacts/upgrade-survivor/summary.json
  , then starts a loopback Gateway and checks configured intents, state preservation, startup, and status budgets. Override the baseline with
  text

  Copy

  OPENCLAW_UPGRADE_SURVIVOR_BASELINE_SPEC
  ; Package Acceptance exposes the same value as
  text

  Copy

  published_upgrade_survivor_baseline
  .
• Session runtime context smoke:
  text

  Copy

  pnpm test:docker:session-runtime-context
  verifies hidden runtime context transcript persistence plus doctor repair of affected duplicated prompt-rewrite branches.
• Bun global install smoke:
  text

  Copy

  bash scripts/e2e/bun-global-install-smoke.sh
  packs the current tree, installs it with
  text

  Copy

  bun install -g
  in an isolated home, and verifies
  text

  Copy

  openclaw infer image providers --json
  returns bundled image providers instead of hanging. Reuse a prebuilt tarball with
  text

  Copy

  OPENCLAW_BUN_GLOBAL_SMOKE_PACKAGE_TGZ=/path/to/openclaw-*.tgz
  , skip the host build with
  text

  Copy

  OPENCLAW_BUN_GLOBAL_SMOKE_HOST_BUILD=0
  , or copy
  text

  Copy

  dist/
  from a built Docker image with
  text

  Copy

  OPENCLAW_BUN_GLOBAL_SMOKE_DIST_IMAGE=openclaw-dockerfile-smoke:local
  .
• Installer Docker smoke:
  text

  Copy

  bash scripts/test-install-sh-docker.sh
  shares one npm cache across its root, update, and direct-npm containers. Update smoke defaults to npm
  text

  Copy

  latest
  as the stable baseline before upgrading to the candidate tarball. Override with
  text

  Copy

  OPENCLAW_INSTALL_SMOKE_UPDATE_BASELINE=2026.4.22
  locally, or with the Install Smoke workflow's
  text

  Copy

  update_baseline_version
  input on GitHub. Non-root installer checks keep an isolated npm cache so root-owned cache entries do not mask user-local install behavior. Set
  text

  Copy

  OPENCLAW_INSTALL_SMOKE_NPM_CACHE_DIR=/path/to/cache
  to reuse the root/update/direct-npm cache across local reruns.
• Install Smoke CI skips the duplicate direct-npm global update with
  text

  Copy

  OPENCLAW_INSTALL_SMOKE_SKIP_NPM_GLOBAL=1
  ; run the script locally without that env when direct
  text

  Copy

  npm install -g
  coverage is needed.
• Agents delete shared workspace CLI smoke:
  text

  Copy

  pnpm test:docker:agents-delete-shared-workspace
  (script:
  text

  Copy

  scripts/e2e/agents-delete-shared-workspace-docker.sh
  ) builds the root Dockerfile image by default, seeds two agents with one workspace in an isolated container home, runs
  text

  Copy

  agents delete --json
  , and verifies valid JSON plus retained workspace behavior. Reuse the install-smoke image with
  text

  Copy

  OPENCLAW_AGENTS_DELETE_SHARED_WORKSPACE_E2E_IMAGE=openclaw-dockerfile-smoke:local OPENCLAW_AGENTS_DELETE_SHARED_WORKSPACE_E2E_SKIP_BUILD=1
  .
• Gateway networking (two containers, WS auth + health):
  text

  Copy

  pnpm test:docker:gateway-network
  (script:
  text

  Copy

  scripts/e2e/gateway-network-docker.sh
  )
• Browser CDP snapshot smoke:
  text

  Copy

  pnpm test:docker:browser-cdp-snapshot
  (script:
  text

  Copy

  scripts/e2e/browser-cdp-snapshot-docker.sh
  ) builds the source E2E image plus a Chromium layer, starts Chromium with raw CDP, runs
  text

  Copy

  browser doctor --deep
  , and verifies CDP role snapshots cover link URLs, cursor-promoted clickables, iframe refs, and frame metadata.
• OpenAI Responses web_search minimal reasoning regression:
  text

  Copy

  pnpm test:docker:openai-web-search-minimal
  (script:
  text

  Copy

  scripts/e2e/openai-web-search-minimal-docker.sh
  ) runs a mocked OpenAI server through Gateway, verifies
  text

  Copy

  web_search
  raises
  text

  Copy

  reasoning.effort
  from
  text

  Copy

  minimal
  to
  text

  Copy

  low
  , then forces the provider schema reject and checks the raw detail appears in Gateway logs.
• MCP channel bridge (seeded Gateway + stdio bridge + raw Claude notification-frame smoke):
  text

  Copy

  pnpm test:docker:mcp-channels
  (script:
  text

  Copy

  scripts/e2e/mcp-channels-docker.sh
  )
• Pi bundle MCP tools (real stdio MCP server + embedded Pi profile allow/deny smoke):
  text

  Copy

  pnpm test:docker:pi-bundle-mcp-tools
  (script:
  text

  Copy

  scripts/e2e/pi-bundle-mcp-tools-docker.sh
  )
• Cron/subagent MCP cleanup (real Gateway + stdio MCP child teardown after isolated cron and one-shot subagent runs):
  text

  Copy

  pnpm test:docker:cron-mcp-cleanup
  (script:
  text

  Copy

  scripts/e2e/cron-mcp-cleanup-docker.sh
  )
• Plugins (install smoke, ClawHub kitchen-sink install/uninstall, marketplace updates, and Claude-bundle enable/inspect):
  text

  Copy

  pnpm test:docker:plugins
  (script:
  text

  Copy

  scripts/e2e/plugins-docker.sh
  ) Set
  text

  Copy

  OPENCLAW_PLUGINS_E2E_CLAWHUB=0
  to skip the ClawHub block, or override the default kitchen-sink package/runtime pair with
  text

  Copy

  OPENCLAW_PLUGINS_E2E_CLAWHUB_SPEC
  and
  text

  Copy

  OPENCLAW_PLUGINS_E2E_CLAWHUB_ID
  . Without
  text

  Copy

  OPENCLAW_CLAWHUB_URL
  /
  text

  Copy

  CLAWHUB_URL
  , the test uses a hermetic local ClawHub fixture server.
• Plugin update unchanged smoke:
  text

  Copy

  pnpm test:docker:plugin-update
  (script:
  text

  Copy

  scripts/e2e/plugin-update-unchanged-docker.sh
  )
• Config reload metadata smoke:
  text

  Copy

  pnpm test:docker:config-reload
  (script:
  text

  Copy

  scripts/e2e/config-reload-source-docker.sh
  )
• Bundled plugin runtime deps:
  text

  Copy

  pnpm test:docker:bundled-channel-deps
  builds a small Docker runner image by default, builds and packs OpenClaw once on the host, then mounts that tarball into each Linux install scenario. Reuse the image with
  text

  Copy

  OPENCLAW_SKIP_DOCKER_BUILD=1
  , skip the host rebuild after a fresh local build with
  text

  Copy

  OPENCLAW_BUNDLED_CHANNEL_HOST_BUILD=0
  , or point at an existing tarball with
  text

  Copy

  OPENCLAW_CURRENT_PACKAGE_TGZ=/path/to/openclaw-*.tgz
  . The full Docker aggregate and release-path bundled-channel chunks pre-pack this tarball once, then shard bundled channel checks into independent lanes, including separate update lanes for Telegram, Discord, Slack, Feishu, memory-lancedb, and ACPX. Release chunks split channel smokes, update targets, and setup/runtime contracts into
  text

  Copy

  bundled-channels-core
  ,
  text

  Copy

  bundled-channels-update-a
  ,
  text

  Copy

  bundled-channels-update-b
  , and
  text

  Copy

  bundled-channels-contracts
  ; the aggregate
  text

  Copy

  bundled-channels
  chunk remains available for manual reruns. The release workflow also splits provider installer chunks and bundled plugin install/uninstall chunks; legacy
  text

  Copy

  package-update
  ,
  text

  Copy

  plugins-runtime
  , and
  text

  Copy

  plugins-integrations
  chunks remain aggregate aliases for manual reruns. Use
  text

  Copy

  OPENCLAW_BUNDLED_CHANNELS=telegram,slack
  to narrow the channel matrix when running the bundled lane directly, or
  text

  Copy

  OPENCLAW_BUNDLED_CHANNEL_UPDATE_TARGETS=telegram,acpx
  to narrow the update scenario. Per-scenario Docker runs default to
  text

  Copy

  OPENCLAW_BUNDLED_CHANNEL_DOCKER_RUN_TIMEOUT=900s
  ; the multi-target update scenario defaults to
  text

  Copy

  OPENCLAW_BUNDLED_CHANNEL_UPDATE_DOCKER_RUN_TIMEOUT=2400s
  . The lane also verifies that
  text

  Copy

  channels.<id>.enabled=false
  and
  text

  Copy

  plugins.entries.<id>.enabled=false
  suppress doctor/runtime-dependency repair.
• Narrow bundled plugin runtime deps while iterating by disabling unrelated scenarios, for example:
  text

  Copy

  OPENCLAW_BUNDLED_CHANNEL_SCENARIOS=0 OPENCLAW_BUNDLED_CHANNEL_UPDATE_SCENARIO=0 OPENCLAW_BUNDLED_CHANNEL_ROOT_OWNED_SCENARIO=0 OPENCLAW_BUNDLED_CHANNEL_SETUP_ENTRY_SCENARIO=0 pnpm test:docker:bundled-channel-deps
  .

To prebuild and reuse the shared functional image manually:

bash
Copy
OPENCLAW_DOCKER_E2E_IMAGE=openclaw-docker-e2e-functional:local pnpm test:docker:e2e-build
OPENCLAW_DOCKER_E2E_IMAGE=openclaw-docker-e2e-functional:local OPENCLAW_SKIP_DOCKER_BUILD=1 pnpm test:docker:mcp-channels

Suite-specific image overrides such as

still win when set. When points at a remote shared image, the scripts pull it if it is not already local. The QR and installer Docker tests keep their own Dockerfiles because they validate package/install behavior rather than the shared built-app runtime.

The live-model Docker runners also bind-mount the current checkout read-only and stage it into a temporary workdir inside the container. This keeps the runtime image slim while still running Vitest against your exact local source/config. The staging step skips large local-only caches and app build outputs such as

, , , and app-local or Gradle output directories so Docker live runs do not spend minutes copying machine-specific artifacts. They also set so gateway live probes do not start real Telegram/Discord/etc. channel workers inside the container. still runs , so pass through as well when you need to narrow or exclude gateway live coverage from that Docker lane. is a higher-level compatibility smoke: it starts an OpenClaw gateway container with the OpenAI-compatible HTTP endpoints enabled, starts a pinned Open WebUI container against that gateway, signs in through Open WebUI, verifies exposes , then sends a real chat request through Open WebUI's proxy. The first run can be noticeably slower because Docker may need to pull the Open WebUI image and Open WebUI may need to finish its own cold-start setup. This lane expects a usable live model key, and ( by default) is the primary way to provide it in Dockerized runs. Successful runs print a small JSON payload like . is intentionally deterministic and does not need a real Telegram, Discord, or iMessage account. It boots a seeded Gateway container, starts a second container that spawns , then verifies routed conversation discovery, transcript reads, attachment metadata, live event queue behavior, outbound send routing, and Claude-style channel + permission notifications over the real stdio MCP bridge. The notification check inspects the raw stdio MCP frames directly so the smoke validates what the bridge actually emits, not just what a specific client SDK happens to surface. is deterministic and does not need a live model key. It builds the repo Docker image, starts a real stdio MCP probe server inside the container, materializes that server through the embedded Pi bundle MCP runtime, executes the tool, then verifies and keep tools while and filter them. is deterministic and does not need a live model key. It starts a seeded Gateway with a real stdio MCP probe server, runs an isolated cron turn and a one-shot child turn, then verifies the MCP child process exits after each run.

Manual ACP plain-language thread smoke (not CI):

• text

  Copy

  bun scripts/dev/discord-acp-plain-language-smoke.ts --channel <discord-channel-id> ...
• Keep this script for regression/debug workflows. It may be needed again for ACP thread routing validation, so do not delete it.

Useful env vars:

• text

  Copy

  OPENCLAW_CONFIG_DIR=...
  (default:
  text

  Copy

  ~/.openclaw
  ) mounted to
  text

  Copy

  /home/node/.openclaw
• text

  Copy

  OPENCLAW_WORKSPACE_DIR=...
  (default:
  text

  Copy

  ~/.openclaw/workspace
  ) mounted to
  text

  Copy

  /home/node/.openclaw/workspace
• text

  Copy

  OPENCLAW_PROFILE_FILE=...
  (default:
  text

  Copy

  ~/.profile
  ) mounted to
  text

  Copy

  /home/node/.profile
  and sourced before running tests
• text

  Copy

  OPENCLAW_DOCKER_PROFILE_ENV_ONLY=1
  to verify only env vars sourced from
  text

  Copy

  OPENCLAW_PROFILE_FILE
  , using temporary config/workspace dirs and no external CLI auth mounts
• text

  Copy

  OPENCLAW_DOCKER_CLI_TOOLS_DIR=...
  (default:
  text

  Copy

  ~/.cache/openclaw/docker-cli-tools
  ) mounted to
  text

  Copy

  /home/node/.npm-global
  for cached CLI installs inside Docker
• External CLI auth dirs/files under
  text

  Copy

  $HOME
  are mounted read-only under
  text

  Copy

  /host-auth...
  , then copied into
  text

  Copy

  /home/node/...
  before tests start
  • Default dirs:
    text

    Copy

    .minimax
  • Default files:
    text

    Copy

    ~/.codex/auth.json
    ,
    text

    Copy

    ~/.codex/config.toml
    ,
    text

    Copy

    .claude.json
    ,
    text

    Copy

    ~/.claude/.credentials.json
    ,
    text

    Copy

    ~/.claude/settings.json
    ,
    text

    Copy

    ~/.claude/settings.local.json
  • Narrowed provider runs mount only the needed dirs/files inferred from
    text

    Copy

    OPENCLAW_LIVE_PROVIDERS
    /
    text

    Copy

    OPENCLAW_LIVE_GATEWAY_PROVIDERS
  • Override manually with
    text

    Copy

    OPENCLAW_DOCKER_AUTH_DIRS=all
    ,
    text

    Copy

    OPENCLAW_DOCKER_AUTH_DIRS=none
    , or a comma list like
    text

    Copy

    OPENCLAW_DOCKER_AUTH_DIRS=.claude,.codex
• text

  Copy

  OPENCLAW_LIVE_GATEWAY_MODELS=...
  /
  text

  Copy

  OPENCLAW_LIVE_MODELS=...
  to narrow the run
• text

  Copy

  OPENCLAW_LIVE_GATEWAY_PROVIDERS=...
  /
  text

  Copy

  OPENCLAW_LIVE_PROVIDERS=...
  to filter providers in-container
• text

  Copy

  OPENCLAW_SKIP_DOCKER_BUILD=1
  to reuse an existing
  text

  Copy

  openclaw:local-live
  image for reruns that do not need a rebuild
• text

  Copy

  OPENCLAW_LIVE_REQUIRE_PROFILE_KEYS=1
  to ensure creds come from the profile store (not env)
• text

  Copy

  OPENCLAW_OPENWEBUI_MODEL=...
  to choose the model exposed by the gateway for the Open WebUI smoke
• text

  Copy

  OPENCLAW_OPENWEBUI_PROMPT=...
  to override the nonce-check prompt used by the Open WebUI smoke
• text

  Copy

  OPENWEBUI_IMAGE=...
  to override the pinned Open WebUI image tag

Docs sanity

Run docs checks after doc edits:

. Run full Mintlify anchor validation when you need in-page heading checks too: .

Offline regression (CI-safe)

These are “real pipeline” regressions without real providers:

• Gateway tool calling (mock OpenAI, real gateway + agent loop):
  text

  Copy

  src/gateway/gateway.test.ts
  (case: "runs a mock OpenAI tool call end-to-end via gateway agent loop")
• Gateway wizard (WS
  text

  Copy

  wizard.start
  /
  text

  Copy

  wizard.next
  , writes config + auth enforced):
  text

  Copy

  src/gateway/gateway.test.ts
  (case: "runs wizard over ws and writes auth token config")

Agent reliability evals (skills)

We already have a few CI-safe tests that behave like “agent reliability evals”:

• Mock tool-calling through the real gateway + agent loop (
  text

  Copy

  src/gateway/gateway.test.ts
  ).
• End-to-end wizard flows that validate session wiring and config effects (
  text

  Copy

  src/gateway/gateway.test.ts
  ).

What’s still missing for skills (see Skills):

• Decisioning: when skills are listed in the prompt, does the agent pick the right skill (or avoid irrelevant ones)?
• Compliance: does the agent read
  text

  Copy

  SKILL.md
  before use and follow required steps/args?
• Workflow contracts: multi-turn scenarios that assert tool order, session history carryover, and sandbox boundaries.

Future evals should stay deterministic first:

• A scenario runner using mock providers to assert tool calls + order, skill file reads, and session wiring.
• A small suite of skill-focused scenarios (use vs avoid, gating, prompt injection).
• Optional live evals (opt-in, env-gated) only after the CI-safe suite is in place.

Contract tests (plugin and channel shape)

Contract tests verify that every registered plugin and channel conforms to its interface contract. They iterate over all discovered plugins and run a suite of shape and behavior assertions. The default

unit lane intentionally skips these shared seam and smoke files; run the contract commands explicitly when you touch shared channel or provider surfaces.

Commands

• All contracts:
  text

  Copy

  pnpm test:contracts
• Channel contracts only:
  text

  Copy

  pnpm test:contracts:channels
• Provider contracts only:
  text

  Copy

  pnpm test:contracts:plugins

Channel contracts

Located in

:

• plugin - Basic plugin shape (id, name, capabilities)
• setup - Setup wizard contract
• session-binding - Session binding behavior
• outbound-payload - Message payload structure
• inbound - Inbound message handling
• actions - Channel action handlers
• threading - Thread ID handling
• directory - Directory/roster API
• group-policy - Group policy enforcement

Provider status contracts

Located in

.

• status - Channel status probes
• registry - Plugin registry shape

Provider contracts

Located in

:

• auth - Auth flow contract
• auth-choice - Auth choice/selection
• catalog - Model catalog API
• discovery - Plugin discovery
• loader - Plugin loading
• runtime - Provider runtime
• shape - Plugin shape/interface
• wizard - Setup wizard

When to run

• After changing plugin-sdk exports or subpaths
• After adding or modifying a channel or provider plugin
• After refactoring plugin registration or discovery

Contract tests run in CI and do not require real API keys.

Adding regressions (guidance)

When you fix a provider/model issue discovered in live:

• Add a CI-safe regression if possible (mock/stub provider, or capture the exact request-shape transformation)
• If it’s inherently live-only (rate limits, auth policies), keep the live test narrow and opt-in via env vars
• Prefer targeting the smallest layer that catches the bug:
  • provider request conversion/replay bug → direct models test
  • gateway session/history/tool pipeline bug → gateway live smoke or CI-safe gateway mock test
• SecretRef traversal guardrail:
  • text

    Copy

    src/secrets/exec-secret-ref-id-parity.test.ts
    derives one sampled target per SecretRef class from registry metadata (
    text

    Copy

    listSecretTargetRegistryEntries()
    ), then asserts traversal-segment exec ids are rejected.
  • If you add a new
    text

    Copy

    includeInPlan
    SecretRef target family in
    text

    Copy

    src/secrets/target-registry-data.ts
    , update
    text

    Copy

    classifyTargetClass
    in that test. The test intentionally fails on unclassified target ids so new classes cannot be skipped silently.

Related

• Testing live
• CI