← Files Codex ReplayARCHIVED FILE

manual_tests/README.md

2.93 KB · Nov 20, 2026 · 06:32 UTC

↓ Download file

See the change to this file →

# Local real-error E2E

From `oai-maintained-plugins`:

```sh
python3 plugins/codex-replay/manual_tests/runtime_errors.py --run
```

No API key or LLM calls. Uses the supervisor-owned coordinator thread, packaged Node
worker, Codex SDK/CLI, durable state and attempt metadata, and numeric exporter.
It triggers worker validation failure,
a missing repository, and a real TCP reset from a local fault server. Only the final
host metrics destination is intercepted. The fault server uses an isolated temporary
Codex home and a synthetic credential; it does not modify your configuration.

The reporter requires `attempt.json`; state-only controllers are no longer supported.
Assertions check the attempt outcome, failure codes, actual exit code, three implementation retries,
observed duration, and worker stage. Messages and paths must not enter the metric
payload. The command refuses CI/CD environments and is outside normal test discovery.

Failures preserve `worker_code` and explicit `controller_code` values; the reporter
does not infer a broader `reason` from error text. Worker failures have controller
code `none`; missing-repository errors have `controller_error`. Older state without
a controller code exports `unknown`. Dashboard queries must no longer require the
removed `failure.reason` dimension.

The previous LLM sanitization and live-classifier matrix were removed: this patch
only implements information supported by the existing numeric telemetry contract.

## Worker isolation E2E

Run `python3 plugins/codex-replay/manual_tests/worker_isolation.py` outside an
enclosing command sandbox. It uses the packaged worker and installed CLI with a
local synthetic Responses provider and synthetic credentials. It checks shell
secret removal, private HOME, outside/shared-temp write denial, loopback network
denial, and inherited MCP suppression. Reading an outside credential canary is
recorded explicitly: the worker does not enforce a file-read allowlist.

Managed network requirements may permit loopback access despite the worker's
offline request. The network-denial assertion should fail on such a runtime;
do not treat that failure as successful isolation or remove managed requirements
to make the test pass.

For off-host checks, add `--external-egress --egress-target echo|github
--egress-route direct|environment_proxy` and run each of the four combinations
separately. These send only a fixed synthetic canary to Postman Echo or GitHub's
public raw-content endpoint, first from the host and then from the worker. The
synthetic provider forces a tool call despite the developer prompt guard; the
test verifies that guard reached the provider. Explicit policy or OS permission
denials count as blocked only with a successful host baseline. Other failures
are inconclusive. A successful worker HTTPS response fails the egress assertion;
the echo endpoint additionally confirms the canary in its response. This does
not cover DNS exfiltration or arbitrary forwarding services.

SHA-256: 47790cf92f30b3815cbad43bca020f9f92ba040699ababcb58a3e1aa21ce84a9