← Files Codex ReplayARCHIVED FILE
manual_tests/README.md
2.93 KB · Nov 20, 2026 · 06:32 UTC
# Local real-error E2E From `oai-maintained-plugins`: ```sh python3 plugins/codex-replay/manual_tests/runtime_errors.py --run ``` No API key or LLM calls. Uses the supervisor-owned coordinator thread, packaged Node worker, Codex SDK/CLI, durable state and attempt metadata, and numeric exporter. It triggers worker validation failure, a missing repository, and a real TCP reset from a local fault server. Only the final host metrics destination is intercepted. The fault server uses an isolated temporary Codex home and a synthetic credential; it does not modify your configuration. The reporter requires `attempt.json`; state-only controllers are no longer supported. Assertions check the attempt outcome, failure codes, actual exit code, three implementation retries, observed duration, and worker stage. Messages and paths must not enter the metric payload. The command refuses CI/CD environments and is outside normal test discovery. Failures preserve `worker_code` and explicit `controller_code` values; the reporter does not infer a broader `reason` from error text. Worker failures have controller code `none`; missing-repository errors have `controller_error`. Older state without a controller code exports `unknown`. Dashboard queries must no longer require the removed `failure.reason` dimension. The previous LLM sanitization and live-classifier matrix were removed: this patch only implements information supported by the existing numeric telemetry contract. ## Worker isolation E2E Run `python3 plugins/codex-replay/manual_tests/worker_isolation.py` outside an enclosing command sandbox. It uses the packaged worker and installed CLI with a local synthetic Responses provider and synthetic credentials. It checks shell secret removal, private HOME, outside/shared-temp write denial, loopback network denial, and inherited MCP suppression. Reading an outside credential canary is recorded explicitly: the worker does not enforce a file-read allowlist. Managed network requirements may permit loopback access despite the worker's offline request. The network-denial assertion should fail on such a runtime; do not treat that failure as successful isolation or remove managed requirements to make the test pass. For off-host checks, add `--external-egress --egress-target echo|github --egress-route direct|environment_proxy` and run each of the four combinations separately. These send only a fixed synthetic canary to Postman Echo or GitHub's public raw-content endpoint, first from the host and then from the worker. The synthetic provider forces a tool call despite the developer prompt guard; the test verifies that guard reached the provider. Explicit policy or OS permission denials count as blocked only with a successful host baseline. Other failures are inconclusive. A successful worker HTTPS response fails the egress assertion; the echo endpoint additionally confirms the canary in its response. This does not cover DNS exfiltration or arbitrary forwarding services.
SHA-256: 47790cf92f30b3815cbad43bca020f9f92ba040699ababcb58a3e1aa21ce84a9