← Files Compound EngineeringARCHIVED FILE
skills/ce-pov/scripts/peer-job-runner.py
102 KB · Oct 4, 2026 · 12:33 UTC
#!/usr/bin/env python3
"""Detached peer-job runner: start/status/wait/result/reap for delegated work.
Some harnesses enforce a hard ceiling on a single shell tool call and kill the
supervising shell mid-run, so no tool call may span a peer worker's runtime.
This runner splits the lifecycle so every call is short and all durable state
lives on disk:
start claim a job dir, preflight the worker, detach it into its own
session (double fork with os.setsid between the forks), print ONLY
the job id, return fast. The detached process supervises the worker
and writes ONE atomic terminal record. Also sweeps sibling run roots
older than 24 hours (best-effort, owner-checked).
status print each job's state word without blocking.
wait bounded poll (~1s cadence, never longer than --max-secs) that
returns early once every watched job has settled.
result ownership-checked bounded read of a done job's published artifact.
reap ask the detached supervisor to terminate the job now; returns fast.
If the supervisor itself is gone, reap kills the worker tree and
writes the terminal record itself. Reaping a terminal job is a
safe no-op.
Job directory (durable state, the source of truth):
<root>/<skill>/<run-id>/jobs/<job-id>/
meta.json identity: skill, run id, label, input digest, start time,
worker argv, result path (written at start, before detach)
pid supervisor pid + worker pid (written by the supervisor before
start returns; its presence marks "detached"). Platform-
conditional fields — consumers must use .get(): POSIX adds
supervisor_pgid; Windows adds job_name (its job object) and
supervisor_identity / worker_identity (GetProcessTimes guards
so a recycled pid is not treated as the original process).
out.log worker's combined stdout+stderr (byte growth = liveness)
reason terminal detail, written before the status rename so the
status file is always the LAST record to land
status exactly one word, published atomically (tmp + os.replace):
done | failed | timeout | died-without-result
States reported by status/wait:
running detached, no terminal record yet
done worker exited 0 (and, when --result-path was declared,
the result file exists non-empty)
failed nonzero exit, byte-cap kill, or exit 0 without the
declared result
timeout supervisor idle/hard window fired, or a requested reap
died-without-result worker killed by an external signal with no result
evidence (or vanished together with its supervisor)
never-started meta exists but nothing was ever detached (preflight
failure)
unreadable an ownership or sanity check failed; content withheld
Supervision (runs inside the detached session, never in a tool call): poll
~2s; liveness is out.log byte growth; idle window with no growth reaps the
worker tree; a hard cap reaps it regardless; byte caps on out.log and the
published result classify as failed with a recorded reason. Reaping is TERM
to the worker's own process group (the worker is started as a session/group
leader), a grace period, then KILL — with a deepest-first tree walk as the
fallback when the group kill is unavailable. The supervisor classifies the
outcome exactly once; when both the worker's internal cap and the
supervisor's window fire, the supervisor's record wins.
Environment overrides (defaults in parentheses):
CE_PEER_JOBS_ROOT base dir (/tmp/compound-engineering-<effective-uid>,
or $TMPDIR/compound-engineering-<effective-uid> when
/tmp cannot host a writable private root, e.g. under
a sandbox that only allowlists $TMPDIR)
CE_WORK_RUNS_ROOT parent CE Work dir containing all <run-id>/ dirs
CE_PEER_IDLE_SECS idle window, no out.log growth (240)
CE_PEER_HARD_SECS hard cap on worker wall clock
(default: max(1230, CROSS_MODEL_HARD_SECS+30);
an explicit value always wins)
CROSS_MODEL_HARD_SECS when CE_PEER_HARD_SECS is unset, widens the
supervisor hard window (see above)
CE_PEER_LOG_MAX_BYTES out.log byte cap (10485760)
CE_PEER_RESULT_MAX_BYTES result byte cap, supervise + read (5242880)
CE_PEER_POLL_SECS supervisor poll interval (2)
CE_PEER_GRACE_SECS TERM-to-KILL grace during reap (5)
CE_PEER_BASH Windows: absolute bash.exe for peer workers
(preferred over PATH / WSL System32 bash)
CLAUDE_CODE_GIT_BASH_PATH Claude Code Git Bash path; used on Windows when
CE_PEER_BASH is unset (#1268)
Security posture: the job root is a predictable, owner-private directory under
world-shared /tmp. Every read of job state opens the file first (no-follow) and
verifies the descriptor's owner (os.fstat st_uid == os.geteuid, guarded where
geteuid is unavailable) before any content is emitted; a mismatch reports
"unreadable", never content. Reads are bounded by size caps — out.log is never
slurped. Directory/file creation uses 0700/0600 modes, exclusive no-follow
creation, owner/type verification on path components, exact 0700 verification
on the top-level root, and atomic rename for every publish. The worker argv is
exec'd directly (argv list, never a shell); job
ids are minted internally; --skill/--run-id/--label are restricted to
[A-Za-z0-9._-]. Nothing here ever prompts: headless/CI-safe by design.
Platform (#1243): the mechanisms above describe POSIX. Native Windows Python
has no fork/setsid, uid, mode bits, or process groups, so the same contract is
met by win32 equivalents, all behind `sys.platform == "win32"` branches so the
POSIX path is behaviorally unchanged:
detach re-invoke this script as a DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP
child (CREATE_BREAKAWAY_FROM_JOB where the job allows) running the
internal `__supervise` entrypoint; the pid file is the ack.
reap cmd_reap drops a `.reap` marker the supervisor polls for (no directed
signal to a detached, console-less process).
teardown the worker tree lives in a named Job Object -- the real killpg
analog, since it reaches descendants of an already-exited leader,
which taskkill /T cannot (it walks parent->child from a LIVE
parent). Windows releases a named object's name once the last
handle closes, so a cmd_reap running after the supervisor died
falls back to a Toolhelp32 snapshot walk; that works because
Windows never reparents orphans, so a dead pid still appears as
th32ParentProcessID on its live children. A recycled pid that is
now this process (or whose GetProcessTimes identity does not
match the pid file) is not the original leader: sweep stale-PPID
descendants, do not TerminateProcess the live reused process.
ownership st_uid == geteuid becomes: the object's owner SID is one this token
creates objects as (user or default-owner SID), checked on the opened
handle (GetSecurityInfo) exactly like the POSIX fstat-by-fd check.
privacy 0700/0600 modes become a hardened ACL (icacls: break inheritance,
grant only the user + SYSTEM + Administrators — the root-equivalents).
jobs root defaults under %LOCALAPPDATA%\\compound-engineering-jobs (then the
user temp dir), owner-private, since there is no shared /tmp.
Pure stdlib. No third-party dependencies.
"""
import argparse
import glob
import json
import os
import re
import shutil
import signal
import stat
import subprocess
import sys
import tempfile
import time
# Identifier charset for --skill/--run-id/--label and bare job refs. The dot is
# allowed (model/date tokens use it) but an all-dot value (".", "..") would be a
# path component that escapes the jobs root, so it is rejected separately below.
SAFE_RE = re.compile(r"^[A-Za-z0-9._-]+$")
def _is_safe_token(value: str) -> bool:
return bool(SAFE_RE.match(value)) and value.strip(".") != ""
TERMINAL_STATES = ("done", "failed", "timeout", "died-without-result")
IS_WINDOWS = sys.platform == "win32"
_uid_getter = getattr(os, "geteuid", None) or getattr(os, "getuid", None)
_EFFECTIVE_UID = _uid_getter() if _uid_getter is not None else None
if IS_WINDOWS:
# No geteuid on Windows; the current-user SID is the ownership identity
# (see the Windows security section below), and the per-user jobs root lives
# under LOCALAPPDATA (falling back to the user temp dir) with a hardened ACL
# so R6 has a working default rather than a required override.
_WIN_ROOT_BASE = os.environ.get("LOCALAPPDATA") or tempfile.gettempdir()
DEFAULT_ROOT = os.path.join(_WIN_ROOT_BASE, "compound-engineering-jobs")
elif _EFFECTIVE_UID is not None:
DEFAULT_ROOT = os.path.join("/tmp", f"compound-engineering-{_EFFECTIVE_UID}")
else:
DEFAULT_ROOT = None
O_NOFOLLOW = getattr(os, "O_NOFOLLOW", 0)
# Windows CPython opens os.open() descriptors in CRT *text* mode by default:
# writes expand \n -> \r\n and reads stop at the first 0x1A (Ctrl-Z EOF), which
# would silently corrupt and truncate a peer's result artifact and desync the
# out.log byte caps from st_size. O_BINARY is 0 on POSIX, so this is a no-op
# there and every os.open below stays byte-exact on both platforms.
O_BINARY = getattr(os, "O_BINARY", 0)
SWEEP_AGE_SECS = 24 * 3600
CLAIM_ATTEMPTS = 16
STATUS_READ_CAP = 256
REASON_READ_CAP = 1024
META_READ_CAP = 64 * 1024
EXIT_CODES_DOC = """\
exit codes:
0 the command itself succeeded. For status/wait this means the query ran;
it says nothing about job outcomes — parse stdout (or --json) for states.
For `result` it means a done job's artifact (or a --path file) was emitted;
for reap it includes the safe no-op on an already-terminal job.
1 runtime error (preflight failure, unknown job, detach failure)
2 usage error; for `result`: the job is still running
3 for `result`: job settled but not done (failed / timeout /
died-without-result / never-started), or the result file is missing
4 the read was refused, so content is never emitted: the ownership check
failed (job state or result not owned by the current user), or the path
is there but unreadable (a symlink rejected by O_NOFOLLOW, a non-regular
file, a byte-cap overrun). Only a genuinely absent file is 3.
environment overrides: CE_PEER_JOBS_ROOT, CE_WORK_RUNS_ROOT, CE_PEER_IDLE_SECS,
CE_PEER_HARD_SECS, CROSS_MODEL_HARD_SECS, CE_PEER_LOG_MAX_BYTES,
CE_PEER_RESULT_MAX_BYTES, CE_PEER_POLL_SECS, CE_PEER_GRACE_SECS (defaults in
the module docstring).
"""
class RunnerError(Exception):
"""Actionable operational error: message to stderr, exit 1."""
class Unreadable(Exception):
"""Job state failed an ownership or sanity check; content withheld."""
# --- configuration -----------------------------------------------------------
# Supervisor hard-window floor: clears the highest cross-model worker default
# (review skills use CROSS_MODEL_HARD_SECS:-1200) so an unset knob still nests
# worker < deadline < runner without orchestrator arithmetic. Grace matches the
# historical prose +30s so a raised knob widens the runner the same way.
_RUNNER_HARD_FLOOR = 1230.0
_RUNNER_HARD_GRACE = 30.0
def _private_root_usable(path: str) -> bool:
"""True when `path` is (or can now be) a directory we own and can write into.
Creation is the probe: a sandbox that denies writes under /tmp refuses the
mkdir, and one that lets a pre-existing root stand still fails the access
check, so both land on the fallback instead of failing at the first job.
"""
try:
os.mkdir(path, 0o700)
except FileExistsError:
pass
except OSError:
return False
try:
_check_owned_dir(path)
except (OSError, RunnerError):
return False
return os.access(path, os.W_OK)
def _fallback_root() -> str:
return os.path.join(os.environ.get("TMPDIR") or "/tmp", f"compound-engineering-{_EFFECTIVE_UID}")
def jobs_root_base() -> str:
configured = os.environ.get("CE_PEER_JOBS_ROOT")
if configured:
return os.path.abspath(configured)
if DEFAULT_ROOT is None:
raise RunnerError("effective user ID is unavailable; cannot derive the jobs root")
if IS_WINDOWS or _private_root_usable(DEFAULT_ROOT):
return os.path.abspath(DEFAULT_ROOT)
# Same order and candidates as the skills' shell preamble, so a job started
# there is found here.
return os.path.abspath(_fallback_root())
def candidate_jobs_root_bases() -> list:
"""Every root an existing job may live under: the configured root alone, or
both the /tmp root and the $TMPDIR fallback (deduplicated, primary first).
Creation uses jobs_root_base(); lookup of an already-started job must not
depend on which root *this* invocation would create under, because a
sandboxed session and a later unsandboxed one resolve different roots.
"""
configured = os.environ.get("CE_PEER_JOBS_ROOT")
if configured:
return [os.path.abspath(configured)]
if DEFAULT_ROOT is None:
raise RunnerError("effective user ID is unavailable; cannot derive the jobs root")
bases = [os.path.abspath(DEFAULT_ROOT)]
if not IS_WINDOWS:
fallback = os.path.abspath(_fallback_root())
if fallback not in bases:
bases.append(fallback)
return bases
def skill_runs_root(skill: str) -> str:
if skill == "ce-work" and os.environ.get("CE_WORK_RUNS_ROOT"):
return os.path.abspath(os.environ["CE_WORK_RUNS_ROOT"])
return os.path.join(jobs_root_base(), skill)
def candidate_skill_runs_roots(skill: str) -> list:
if skill == "ce-work" and os.environ.get("CE_WORK_RUNS_ROOT"):
return [os.path.abspath(os.environ["CE_WORK_RUNS_ROOT"])]
return [os.path.join(base, skill) for base in candidate_jobs_root_bases()]
def _env_num(name: str, default: float, conv, *, allow_zero: bool = False):
raw = os.environ.get(name)
if not raw:
return default
try:
val = conv(raw)
except ValueError:
return default
if allow_zero and val == 0:
return None
return val if val > 0 else default
def _derived_hard_default() -> float:
"""Outermost supervisor hard window when CE_PEER_HARD_SECS is unset.
Reads ambient CROSS_MODEL_HARD_SECS (the runner already forwards os.environ
to the worker, so a user-set knob is present here). Explicit CE_PEER_HARD_SECS
still wins via cfg() — ce-work and elevation paths keep their own windows.
"""
cross = _env_num("CROSS_MODEL_HARD_SECS", 0.0, float)
return max(_RUNNER_HARD_FLOOR, cross + _RUNNER_HARD_GRACE)
def cfg(skill=None) -> dict:
return {
"idle": _env_num("CE_PEER_IDLE_SECS", 240.0, float, allow_zero=skill == "ce-work"),
"hard": _env_num("CE_PEER_HARD_SECS", _derived_hard_default(), float),
"log_max": int(_env_num("CE_PEER_LOG_MAX_BYTES", 10 * 1024 * 1024, int)),
"result_max": int(_env_num("CE_PEER_RESULT_MAX_BYTES", 5 * 1024 * 1024, int)),
"poll": _env_num("CE_PEER_POLL_SECS", 2.0, float),
"grace": _env_num("CE_PEER_GRACE_SECS", 5.0, float),
}
# --- Windows security + process primitives ------------------------------------
#
# POSIX ownership is `fstat().st_uid == geteuid()` plus mode 0700/0600. Windows
# has neither uids nor mode bits, so the equivalent identity is the current
# user's SID: a job dir/file is "ours" when its owner SID is one this process's
# token creates objects as (the user SID or the token's default owner SID -- an
# elevated process defaults new objects to Administrators). A foreign user's
# planted dir carries neither SID and is rejected, exactly as a uid mismatch is
# on POSIX. The DACL is hardened to user+SYSTEM+Administrators (root-equivalents,
# mirroring how root still reaches a 0700 dir) with inheritance broken, so no
# world/Users grant survives. Pure stdlib via ctypes -- no pywin32.
if IS_WINDOWS:
import ctypes
import msvcrt
from ctypes import wintypes
_advapi32 = ctypes.WinDLL("advapi32", use_last_error=True)
_kernel32 = ctypes.WinDLL("kernel32", use_last_error=True)
_SE_FILE_OBJECT = 1
_OWNER_SECURITY_INFORMATION = 0x00000001
_TOKEN_QUERY = 0x0008
_TOKEN_USER_CLASS = 1
_TOKEN_OWNER_CLASS = 4
_STILL_ACTIVE = 259
_WAIT_TIMEOUT = 0x00000102
_PROCESS_QUERY_LIMITED_INFORMATION = 0x1000
_SYNCHRONIZE = 0x00100000
# A detached, console-less parent still gives its children a NEW console
# unless this is set, so every job would flash a window on the user's
# desktop. Applied to the worker and to every helper tool we shell out to.
_WIN_NO_WINDOW = getattr(subprocess, "CREATE_NO_WINDOW", 0x08000000)
def _win_tool(name: str) -> str:
"""Absolute path to a System32 tool. CreateProcess searches the
application and current directories before System32, so invoking
`icacls`/`taskkill` by bare name from an untrusted CWD is a binary-
hijack surface. Falls back to the bare name only if System32 is
unresolvable, which is strictly better than never running."""
root = os.environ.get("SystemRoot") or r"C:\Windows"
candidate = os.path.join(root, "System32", name + ".exe")
return candidate if os.path.isfile(candidate) else name
_advapi32.ConvertSidToStringSidW.argtypes = [
ctypes.c_void_p, ctypes.POINTER(ctypes.c_wchar_p)]
_advapi32.ConvertSidToStringSidW.restype = wintypes.BOOL
_advapi32.OpenProcessToken.argtypes = [
wintypes.HANDLE, wintypes.DWORD, ctypes.POINTER(wintypes.HANDLE)]
_advapi32.OpenProcessToken.restype = wintypes.BOOL
_advapi32.GetTokenInformation.argtypes = [
wintypes.HANDLE, ctypes.c_int, ctypes.c_void_p, wintypes.DWORD,
ctypes.POINTER(wintypes.DWORD)]
_advapi32.GetTokenInformation.restype = wintypes.BOOL
_advapi32.GetSecurityInfo.argtypes = [
wintypes.HANDLE, ctypes.c_int, wintypes.DWORD,
ctypes.POINTER(ctypes.c_void_p), ctypes.c_void_p, ctypes.c_void_p,
ctypes.c_void_p, ctypes.POINTER(ctypes.c_void_p)]
_advapi32.GetSecurityInfo.restype = wintypes.DWORD
_advapi32.GetNamedSecurityInfoW.argtypes = [
wintypes.LPCWSTR, ctypes.c_int, wintypes.DWORD,
ctypes.POINTER(ctypes.c_void_p), ctypes.c_void_p, ctypes.c_void_p,
ctypes.c_void_p, ctypes.POINTER(ctypes.c_void_p)]
_advapi32.GetNamedSecurityInfoW.restype = wintypes.DWORD
_kernel32.GetCurrentProcess.restype = wintypes.HANDLE
_kernel32.OpenProcess.argtypes = [
wintypes.DWORD, wintypes.BOOL, wintypes.DWORD]
_kernel32.OpenProcess.restype = wintypes.HANDLE
_kernel32.GetProcessTimes.argtypes = [
wintypes.HANDLE,
ctypes.POINTER(wintypes.FILETIME),
ctypes.POINTER(wintypes.FILETIME),
ctypes.POINTER(wintypes.FILETIME),
ctypes.POINTER(wintypes.FILETIME),
]
_kernel32.GetProcessTimes.restype = wintypes.BOOL
_kernel32.QueryFullProcessImageNameW.argtypes = [
wintypes.HANDLE, wintypes.DWORD, wintypes.LPWSTR,
ctypes.POINTER(wintypes.DWORD)]
_kernel32.QueryFullProcessImageNameW.restype = wintypes.BOOL
_kernel32.WaitForSingleObject.argtypes = [wintypes.HANDLE, wintypes.DWORD]
_kernel32.WaitForSingleObject.restype = wintypes.DWORD
_kernel32.CloseHandle.argtypes = [wintypes.HANDLE]
_kernel32.CloseHandle.restype = wintypes.BOOL
_kernel32.LocalFree.argtypes = [wintypes.HGLOBAL]
_kernel32.LocalFree.restype = wintypes.HGLOBAL
_kernel32.CreateJobObjectW.argtypes = [ctypes.c_void_p, wintypes.LPCWSTR]
_kernel32.CreateJobObjectW.restype = wintypes.HANDLE
_kernel32.OpenJobObjectW.argtypes = [
wintypes.DWORD, wintypes.BOOL, wintypes.LPCWSTR]
_kernel32.OpenJobObjectW.restype = wintypes.HANDLE
_kernel32.AssignProcessToJobObject.argtypes = [
wintypes.HANDLE, wintypes.HANDLE]
_kernel32.AssignProcessToJobObject.restype = wintypes.BOOL
_kernel32.TerminateJobObject.argtypes = [wintypes.HANDLE, wintypes.UINT]
_kernel32.TerminateJobObject.restype = wintypes.BOOL
_JOB_OBJECT_TERMINATE = 0x0008
_PROCESS_TERMINATE = 0x0001
_PROCESS_SET_QUOTA = 0x0100
_TH32CS_SNAPPROCESS = 0x00000002
_TH32CS_SNAPTHREAD = 0x00000004
_THREAD_SUSPEND_RESUME = 0x0002
# CreateProcess CREATE_SUSPENDED: primary thread starts frozen so we can
# AssignProcessToJobObject before any user code (or child spawn) runs.
_CREATE_SUSPENDED = 0x00000004
class _PROCESSENTRY32W(ctypes.Structure):
_fields_ = [
("dwSize", wintypes.DWORD),
("cntUsage", wintypes.DWORD),
("th32ProcessID", wintypes.DWORD),
("th32DefaultHeapID", ctypes.POINTER(ctypes.c_ulong)),
("th32ModuleID", wintypes.DWORD),
("cntThreads", wintypes.DWORD),
("th32ParentProcessID", wintypes.DWORD),
("pcPriClassBase", ctypes.c_long),
("dwFlags", wintypes.DWORD),
("szExeFile", ctypes.c_wchar * 260),
]
class _THREADENTRY32(ctypes.Structure):
_fields_ = [
("dwSize", wintypes.DWORD),
("cntUsage", wintypes.DWORD),
("th32ThreadID", wintypes.DWORD),
("th32OwnerProcessID", wintypes.DWORD),
("tpBasePri", ctypes.c_long),
("tpDeltaPri", ctypes.c_long),
("dwFlags", wintypes.DWORD),
]
_kernel32.CreateToolhelp32Snapshot.argtypes = [wintypes.DWORD, wintypes.DWORD]
_kernel32.CreateToolhelp32Snapshot.restype = ctypes.c_void_p
_kernel32.Process32FirstW.argtypes = [
ctypes.c_void_p, ctypes.POINTER(_PROCESSENTRY32W)]
_kernel32.Process32FirstW.restype = wintypes.BOOL
_kernel32.Process32NextW.argtypes = [
ctypes.c_void_p, ctypes.POINTER(_PROCESSENTRY32W)]
_kernel32.Process32NextW.restype = wintypes.BOOL
_kernel32.Thread32First.argtypes = [
ctypes.c_void_p, ctypes.POINTER(_THREADENTRY32)]
_kernel32.Thread32First.restype = wintypes.BOOL
_kernel32.Thread32Next.argtypes = [
ctypes.c_void_p, ctypes.POINTER(_THREADENTRY32)]
_kernel32.Thread32Next.restype = wintypes.BOOL
_kernel32.OpenThread.argtypes = [
wintypes.DWORD, wintypes.BOOL, wintypes.DWORD]
_kernel32.OpenThread.restype = wintypes.HANDLE
_kernel32.ResumeThread.argtypes = [wintypes.HANDLE]
_kernel32.ResumeThread.restype = wintypes.DWORD
_kernel32.TerminateProcess.argtypes = [wintypes.HANDLE, wintypes.UINT]
_kernel32.TerminateProcess.restype = wintypes.BOOL
def _win_process_children_map():
"""th32ParentProcessID -> [child pids] from one Toolhelp snapshot."""
snap = _kernel32.CreateToolhelp32Snapshot(_TH32CS_SNAPPROCESS, 0)
if not snap or snap == ctypes.c_void_p(-1).value:
return {}
children = {}
try:
entry = _PROCESSENTRY32W()
entry.dwSize = ctypes.sizeof(_PROCESSENTRY32W)
more = _kernel32.Process32FirstW(snap, ctypes.byref(entry))
while more:
children.setdefault(entry.th32ParentProcessID, []).append(
entry.th32ProcessID)
more = _kernel32.Process32NextW(snap, ctypes.byref(entry))
finally:
_kernel32.CloseHandle(ctypes.c_void_p(snap))
return children
def _win_descendants_deepest_first(root_pid: int, children=None):
"""Children before parents, via a process snapshot. This is the direct
analog of the POSIX `ps`-based walk and carries the same pid-reuse
exposure. It works on an EXITED leader because Windows never reparents
orphans: a dead pid still appears as th32ParentProcessID on its live
children (unlike POSIX, where orphans are reparented to init)."""
if children is None:
children = _win_process_children_map()
order, queue = [], [root_pid]
while queue:
for child in children.get(queue.pop(0), []):
order.append(child)
queue.append(child)
return list(reversed(order))
def _win_process_identity(pid: int):
"""Creation time plus image path — analog of `ps -o lstart= -o command=`.
Creation time is the PID-reuse guard: Windows recycles PIDs aggressively,
and a recycled pid always carries a later FILETIME than the worker we
recorded. None means unproven (gone or unopenable)."""
handle = _kernel32.OpenProcess(
_PROCESS_QUERY_LIMITED_INFORMATION, False, pid)
if not handle:
return None
try:
created, exited, kernel, user = (wintypes.FILETIME() for _ in range(4))
if not _kernel32.GetProcessTimes(
handle, ctypes.byref(created), ctypes.byref(exited),
ctypes.byref(kernel), ctypes.byref(user)):
return None
started = (created.dwHighDateTime << 32) | created.dwLowDateTime
if not started:
return None
size = wintypes.DWORD(32768)
buf = ctypes.create_unicode_buffer(size.value)
image = (buf.value if _kernel32.QueryFullProcessImageNameW(
handle, 0, buf, ctypes.byref(size)) else "")
finally:
_kernel32.CloseHandle(handle)
return "{} {}".format(started, image)
def _win_process_start_time(pid: int):
handle = _kernel32.OpenProcess(
_PROCESS_QUERY_LIMITED_INFORMATION, False, pid)
if not handle:
return None
try:
created, exited, kernel, user = (wintypes.FILETIME() for _ in range(4))
if not _kernel32.GetProcessTimes(
handle, ctypes.byref(created), ctypes.byref(exited),
ctypes.byref(kernel), ctypes.byref(user)):
return None
started = (created.dwHighDateTime << 32) | created.dwLowDateTime
return started or None
finally:
_kernel32.CloseHandle(handle)
def _win_process_identity_matches(pid: int, recorded) -> bool:
if not recorded:
return True
current = _win_process_identity(pid)
return current is not None and current == recorded
def _win_terminate_pid(pid: int) -> bool:
if pid <= 0 or pid == os.getpid():
return False
handle = _kernel32.OpenProcess(_PROCESS_TERMINATE, False, pid)
if not handle:
return False
try:
return bool(_kernel32.TerminateProcess(handle, 1))
finally:
_kernel32.CloseHandle(handle)
def _win_job_name(job_dir: str) -> str:
"""A per-job named kernel object. Naming it is what makes this a real
pgid analog: a DIFFERENT process (cmd_reap, after the supervisor is
gone) can reopen it by name and terminate the whole tree."""
return "Local\\ce-peer-job-" + os.path.basename(job_dir.rstrip("\\/"))
def _win_create_job(name: str):
"""Create the job the worker tree will live in. Deliberately WITHOUT
JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE: like a POSIX process group, the job
must outlive the supervisor so a dead-supervisor reap still finds a
live tree to classify and sweep (matching the POSIX lifecycle tests)."""
handle = _kernel32.CreateJobObjectW(None, name)
return handle or None
def _win_assign_to_job(job_handle, pid: int) -> bool:
proc = _kernel32.OpenProcess(
_PROCESS_SET_QUOTA | _PROCESS_TERMINATE, False, pid)
if not proc:
return False
try:
return bool(_kernel32.AssignProcessToJobObject(job_handle, proc))
finally:
_kernel32.CloseHandle(proc)
def _win_resume_process(pid: int) -> bool:
"""Resume every thread of a CREATE_SUSPENDED process. subprocess.Popen
does not expose hThread from PROCESS_INFORMATION, so walk the thread
snapshot. CREATE_SUSPENDED only freezes the primary thread; resuming
all owned threads is still correct and idempotent for running ones."""
snap = _kernel32.CreateToolhelp32Snapshot(_TH32CS_SNAPTHREAD, 0)
if not snap or snap == ctypes.c_void_p(-1).value:
return False
resumed = False
try:
entry = _THREADENTRY32()
entry.dwSize = ctypes.sizeof(_THREADENTRY32)
more = _kernel32.Thread32First(snap, ctypes.byref(entry))
while more:
if entry.th32OwnerProcessID == pid:
handle = _kernel32.OpenThread(
_THREAD_SUSPEND_RESUME, False, entry.th32ThreadID)
if handle:
try:
# (DWORD)-1 == failure; 0xFFFFFFFF as unsigned.
if _kernel32.ResumeThread(handle) != 0xFFFFFFFF:
resumed = True
finally:
_kernel32.CloseHandle(handle)
more = _kernel32.Thread32Next(snap, ctypes.byref(entry))
finally:
_kernel32.CloseHandle(ctypes.c_void_p(snap))
return resumed
def _win_terminate_job(name: str) -> bool:
"""Terminate every process in the named job, whatever the tree shape.
This is the piece taskkill /T cannot do: it reaches descendants whose
parent has already exited, because job membership is inherited and
does not depend on a live parent to walk from."""
handle = _kernel32.OpenJobObjectW(_JOB_OBJECT_TERMINATE, False, name)
if not handle:
return False
try:
return bool(_kernel32.TerminateJobObject(handle, 1))
finally:
_kernel32.CloseHandle(handle)
_WIN_IDENTITY_SIDS = None
def _win_sid_to_string(psid) -> str:
strp = ctypes.c_wchar_p()
if not _advapi32.ConvertSidToStringSidW(psid, ctypes.byref(strp)):
raise OSError(f"ConvertSidToStringSid failed: {ctypes.get_last_error()}")
try:
return strp.value
finally:
_kernel32.LocalFree(ctypes.cast(strp, wintypes.HGLOBAL))
def _win_token_sid(token, info_class) -> str:
size = wintypes.DWORD(0)
_advapi32.GetTokenInformation(token, info_class, None, 0, ctypes.byref(size))
buf = (ctypes.c_byte * size.value)()
if not _advapi32.GetTokenInformation(
token, info_class, buf, size, ctypes.byref(size)
):
raise OSError(f"GetTokenInformation failed: {ctypes.get_last_error()}")
# TOKEN_USER / TOKEN_OWNER both begin with a PSID at offset 0.
sid_ptr = ctypes.cast(buf, ctypes.POINTER(ctypes.c_void_p))[0]
return _win_sid_to_string(ctypes.c_void_p(sid_ptr))
def _win_identity_sids() -> frozenset:
"""The SID strings this process's token creates objects as: the user SID
and the default-owner SID (they differ for an elevated process). Cached;
an object owned by any of these is treated as ours."""
global _WIN_IDENTITY_SIDS
if _WIN_IDENTITY_SIDS is not None:
return _WIN_IDENTITY_SIDS
token = wintypes.HANDLE()
if not _advapi32.OpenProcessToken(
_kernel32.GetCurrentProcess(), _TOKEN_QUERY, ctypes.byref(token)
):
raise OSError(f"OpenProcessToken failed: {ctypes.get_last_error()}")
try:
sids = {
_win_token_sid(token, _TOKEN_USER_CLASS),
_win_token_sid(token, _TOKEN_OWNER_CLASS),
}
finally:
_kernel32.CloseHandle(token)
_WIN_IDENTITY_SIDS = frozenset(s for s in sids if s)
return _WIN_IDENTITY_SIDS
def _win_owner_sid(api, target) -> str:
"""Shared GetSecurityInfo / GetNamedSecurityInfoW shape: read the OWNER
SID into a freshly allocated security descriptor and stringify it. The
SID points INSIDE that descriptor, so freeing the descriptor is the only
(and required) cleanup -- never free the SID separately."""
psid = ctypes.c_void_p()
psd = ctypes.c_void_p()
err = api(target, _SE_FILE_OBJECT, _OWNER_SECURITY_INFORMATION,
ctypes.byref(psid), None, None, None, ctypes.byref(psd))
if err != 0:
raise OSError(f"{api.__name__} failed: {err}")
try:
return _win_sid_to_string(psid)
finally:
_kernel32.LocalFree(ctypes.cast(psd, wintypes.HGLOBAL))
def _win_owner_sid_from_handle(handle: int) -> str:
return _win_owner_sid(_advapi32.GetSecurityInfo, wintypes.HANDLE(handle))
def _win_owner_sid_from_path(path: str) -> str:
return _win_owner_sid(_advapi32.GetNamedSecurityInfoW, path)
def _win_owns_path(path: str) -> bool:
return _win_owner_sid_from_path(path) in _win_identity_sids()
def _win_owns_handle(handle: int) -> bool:
return _win_owner_sid_from_handle(handle) in _win_identity_sids()
def _win_run_quiet(cmd) -> bool:
"""Fire-and-forget a Windows tool (icacls/taskkill): output suppressed,
exit status returned but never raised. check=False suppresses a NONZERO
exit, NOT a missing executable -- Popen still raises FileNotFoundError
when the tool is absent from PATH, which would otherwise escape the
supervisor's teardown and turn an already-classified `done` job into
`failed`. Returns True only when the tool ran and exited 0."""
try:
return subprocess.run(
cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
check=False, creationflags=_WIN_NO_WINDOW).returncode == 0
except OSError:
return False
def _win_harden_acl(path: str) -> None:
"""Break inheritance and grant only the current user plus the
root-equivalents (SYSTEM, Administrators), so no world/Users grant
survives -- the 0700 analog. Best-effort: the owner check and the
O_EXCL/O_CREAT claim remain the hard gates if icacls is unavailable."""
sids = _win_identity_sids()
if not sids:
return False
inherit = "(OI)(CI)" if os.path.isdir(path) else ""
# Grant EVERY identity SID, not an arbitrary one from the set: an
# elevated token carries two (user + default owner), and picking one
# nondeterministically could grant the wrong principal.
grants = []
for sid in sorted(sids) + ["S-1-5-18", "S-1-5-32-544"]:
grants += ["/grant:r", f"*{sid}:{inherit}F"]
return _win_run_quiet([_win_tool("icacls"), path, "/inheritance:r"] + grants)
def _win_pid_alive(pid: int) -> bool:
handle = _kernel32.OpenProcess(
_PROCESS_QUERY_LIMITED_INFORMATION | _SYNCHRONIZE, False, pid)
if not handle:
return False
try:
return _kernel32.WaitForSingleObject(handle, 0) == _WAIT_TIMEOUT
finally:
_kernel32.CloseHandle(handle)
def _win_kill_tree(
root_pid: int, grace: float, job_name=None, expected_identity=None,
) -> bool:
"""Terminate the worker tree (KTD3). Returns whether the LEADER was
alive when the kill began -- the reap classification signal -- which is
independent of how much of the tree we then sweep.
The job object is the primary mechanism and the true killpg analog: it
reaches descendants even after the leader has exited. taskkill /T can
NOT -- it walks parent->child from a live parent, so against an exited
pid it returns "process not found" and silently leaves grandchildren
running forever. That is why the sweep is attempted whenever a job name
exists, regardless of leader liveness.
No graceful phase: a console-less worker cannot receive taskkill's
WM_CLOSE (it reports "can only be terminated forcefully"), so the grace
window was pure latency that also widened the cmd_reap race.
The job is only reachable by the process that created it: Windows
releases a named object's NAME once the last handle closes, even while
member processes keep the object alive (verified: OpenJobObject then
fails with ERROR_FILE_NOT_FOUND). So a cmd_reap running after the
supervisor died cannot use it, and falls back to the snapshot walk --
which is exactly the dead-leader case, hence deepest-first descendants
BEFORE the leader, and never gated on leader liveness.
A live pid at root_pid is not automatically the original leader:
Windows recycles PIDs, and cmd_reap is often the next python.exe after
the worker exits (the orphan-grandchild smoke). Never terminate this
process, and never terminate a live pid whose GetProcessTimes identity
does not match the recorded worker. Stale-PPID orphans still show the
dead leader as parent. When the pid was reused, the start-time cutoff
applies only to *direct* children of that pid (the new process's own
children vs stale-PPID orphans). A pre-reuse child's full subtree is
still original-tree work, including descendants spawned after reuse."""
self_pid = os.getpid()
is_self = root_pid == self_pid
alive = (not is_self) and _win_pid_alive(root_pid)
recorded_leader = alive and _win_process_identity_matches(
root_pid, expected_identity)
if job_name:
_win_terminate_job(job_name)
# Always Toolhelp-sweep after (or without) the job terminate: children
# that raced outside the job before AssignProcessToJobObject completed
# are not members, and TerminateJobObject alone would leave them.
# CREATE_SUSPENDED closes that spawn race; this remains the belt.
children_map = _win_process_children_map()
reuse_cutoff = None
if not recorded_leader and (is_self or alive):
reuse_cutoff = _win_process_start_time(root_pid)
if reuse_cutoff is not None:
def _predates_reuse(pid):
started = _win_process_start_time(pid)
return started is None or started < reuse_cutoff
kill_set = _pre_reuse_descendant_pids(
root_pid, children_map, _predates_reuse, self_pid)
else:
kill_set = None
for pid in _win_descendants_deepest_first(root_pid, children_map):
if pid == self_pid:
continue
if kill_set is not None and pid not in kill_set:
continue
_win_terminate_pid(pid)
if recorded_leader:
_win_terminate_pid(root_pid)
return recorded_leader
# --- hardened I/O primitives --------------------------------------------------
def _euid():
return _EFFECTIVE_UID
def _check_owned_dir(path: str, require_private: bool = False) -> None:
st = os.lstat(path)
if not stat.S_ISDIR(st.st_mode):
raise RunnerError(f"{path}: not a real directory (symlink or file planted?)")
if IS_WINDOWS:
# SID ownership stands in for st_uid; the hardened ACL (not a mode bit)
# provides privacy, so there is no separate require_private gate.
if not _win_owns_path(path):
raise RunnerError(f"{path}: not owned by the current user")
return
euid = _euid()
if euid is not None and st.st_uid != euid:
raise RunnerError(f"{path}: not owned by the current user")
if require_private:
mode = stat.S_IMODE(st.st_mode)
if mode != 0o700:
raise RunnerError(f"{path}: must have mode 0700, found {mode:04o}")
def ensure_owned_dirs(base: str, path: str) -> None:
"""mkdir -p `path` (mode 0700) verifying owner and type on every component
from `base` down — a planted symlink or foreign dir aborts, never traversed."""
rel = os.path.relpath(path, base)
comps = [] if rel == "." else rel.split(os.sep)
cur = base
created_base = True
try:
os.mkdir(cur, 0o700)
except FileExistsError:
created_base = False
_check_owned_dir(cur)
if IS_WINDOWS:
# `icacls /inheritance:r` is destructive and irreversible in a way
# POSIX's chmod 0700 is not: it permanently drops inherited ACEs. So
# only re-ACL a root this runner owns -- one we just created, or the
# managed default (repairing a default left non-private, which is what
# the POSIX unconditional chmod is for). A pre-existing user-supplied
# CE_PEER_JOBS_ROOT keeps its ACLs and rests on the owner check.
default_root = os.path.abspath(DEFAULT_ROOT) if DEFAULT_ROOT else None
ours = created_base or (
default_root is not None
and os.path.normcase(cur) == os.path.normcase(default_root))
if ours and not _win_harden_acl(cur):
# Never proceed as if hardened: an unverified root is the one case
# where the privacy half of the model would silently be missing.
raise RunnerError(
f"{cur}: could not harden the jobs-root ACL (icacls failed or "
"is unavailable); refusing to use a root whose privacy is "
"unverified"
)
else:
os.chmod(cur, 0o700)
_check_owned_dir(cur, require_private=True)
for comp in comps:
cur = os.path.join(cur, comp)
created = False
try:
os.mkdir(cur, 0o700)
created = True
except FileExistsError:
pass
if created:
if IS_WINDOWS:
_win_harden_acl(cur)
else:
os.chmod(cur, 0o700)
_check_owned_dir(cur)
def read_owned(path: str, cap: int) -> bytes:
"""Open no-follow, verify the OPENED descriptor's owner via fstat, enforce
the size cap, and return content. Raises Unreadable on any trust failure."""
fd = os.open(path, os.O_RDONLY | O_NOFOLLOW | O_BINARY)
try:
st = os.fstat(fd)
if IS_WINDOWS:
# Verify the OPENED handle's owner SID (TOCTOU-safe, like the POSIX
# fstat-by-fd check) before emitting a byte.
if not _win_owns_handle(msvcrt.get_osfhandle(fd)):
raise Unreadable(f"{path}: not owned by the current user; refusing to read")
else:
euid = _euid()
if euid is not None and st.st_uid != euid:
raise Unreadable(f"{path}: not owned by the current user; refusing to read")
if not stat.S_ISREG(st.st_mode):
raise Unreadable(f"{path}: not a regular file")
if st.st_size > cap:
raise Unreadable(f"{path}: {st.st_size} bytes exceeds the {cap}-byte read cap")
chunks = []
got = 0
while got <= cap:
chunk = os.read(fd, 65536)
if not chunk:
break
chunks.append(chunk)
got += len(chunk)
if got > cap:
raise Unreadable(f"{path}: grew past the {cap}-byte read cap during read")
return b"".join(chunks)
finally:
os.close(fd)
def create_exclusive(path: str, data: bytes = b"", mode: int = 0o600) -> None:
fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL | O_NOFOLLOW | O_BINARY, mode)
try:
if data:
os.write(fd, data)
finally:
os.close(fd)
def write_atomic(path: str, data: bytes) -> None:
fd, tmp = tempfile.mkstemp(dir=os.path.dirname(path), prefix=".tmp-")
try:
with os.fdopen(fd, "wb") as f:
f.write(data)
os.replace(tmp, path)
except BaseException:
try:
os.unlink(tmp)
except OSError:
pass
raise
def write_terminal(job_dir: str, state: str, reason: str, overwrite: bool = True) -> None:
"""Publish the single terminal record. The reason detail lands FIRST so the
atomic status rename is always the last record; a reason write failure never
blocks the status."""
status_path = os.path.join(job_dir, "status")
if not overwrite and os.path.lexists(status_path):
return
try:
write_atomic(os.path.join(job_dir, "reason"), (reason.rstrip("\n") + "\n").encode())
except OSError:
pass
write_atomic(status_path, (state + "\n").encode())
# --- job identity and resolution ----------------------------------------------
def mint_job_id() -> str:
return f"{time.strftime('%Y%m%dT%H%M%SZ', time.gmtime())}-{os.urandom(4).hex()}"
def claim_job_dir(jobs_root: str):
"""Atomically claim a fresh job dir: os.mkdir (no -p) fails on collision,
so the id is regenerated rather than a dir ever being shared."""
for _ in range(CLAIM_ATTEMPTS):
job_id = mint_job_id()
job_dir = os.path.join(jobs_root, job_id)
try:
os.mkdir(job_dir, 0o700)
return job_id, job_dir
except FileExistsError:
continue
raise RunnerError(f"could not claim a unique job dir after {CLAIM_ATTEMPTS} attempts")
def resolve_job_dir(ref: str, skill=None) -> str:
if os.sep in ref:
p = os.path.abspath(ref)
if os.path.isdir(p):
return p
raise RunnerError(f"no such job dir: {ref}")
if not _is_safe_token(ref):
raise RunnerError(f"invalid job ref: {ref!r}")
if skill is not None:
if not _is_safe_token(skill):
raise RunnerError(f"invalid skill: {skill!r}")
search_roots = candidate_skill_runs_roots(skill)
patterns = [os.path.join(root, "*", "jobs", ref) for root in search_roots]
else:
search_roots = candidate_jobs_root_bases()
patterns = [os.path.join(root, "*", "*", "jobs", ref) for root in search_roots]
matches = sorted({match for pattern in patterns for match in glob.glob(pattern)})
if not matches:
raise RunnerError(f"job not found under {', '.join(search_roots)}: {ref}")
if len(matches) > 1:
raise RunnerError(f"ambiguous job id {ref}: {len(matches)} matches; pass the job dir path")
return matches[0]
def job_state(job_dir: str) -> str:
try:
_check_owned_dir(job_dir)
except (RunnerError, OSError):
return "unreadable"
try:
word = read_owned(os.path.join(job_dir, "status"), STATUS_READ_CAP)
word = word.decode("utf-8", "replace").strip()
return word if word in TERMINAL_STATES else "unreadable"
except FileNotFoundError:
pass
except (Unreadable, OSError):
return "unreadable"
if os.path.lexists(os.path.join(job_dir, "pid")):
return "running"
return "never-started"
def job_reason(job_dir: str) -> str:
"""The terminal record's detail line, or "" when unavailable. Decorative
context for a message; never load-bearing, so every failure reads as ""."""
try:
raw = read_owned(os.path.join(job_dir, "reason"), REASON_READ_CAP)
except (Unreadable, OSError):
return ""
return raw.decode("utf-8", "replace").strip()
# --- process-tree control -----------------------------------------------------
def _pid_alive(pid: int) -> bool:
if IS_WINDOWS:
return _win_pid_alive(pid)
try:
os.kill(pid, 0)
return True
except ProcessLookupError:
return False
except OSError:
return True
def _pid_running(pid: int) -> bool:
"""True only for a live process, NOT a <defunct> zombie. os.kill(pid, 0)
succeeds for a zombie (the process exited but has not been reaped), which
must not count as a live worker when classifying a reap: a zombie leader
means the worker is gone (died-without-result), not still running (timeout).
Falls back to the kill -0 result when process state is unavailable."""
if IS_WINDOWS:
# Windows has no <defunct> zombie state -- a terminated process's handle
# is signaled and OpenProcess-based liveness already reports it dead.
return _win_pid_alive(pid)
if not _pid_alive(pid):
return False
try:
out = subprocess.run(
["ps", "-o", "state=", "-p", str(pid)],
capture_output=True, text=True, check=False,
).stdout.strip()
except OSError:
return True
if not out:
return False
return not out.startswith("Z")
def _kill_quiet(pid: int, sig: int) -> bool:
try:
os.kill(pid, sig)
return True
except OSError:
return False
def _killpg_quiet(pgid: int, sig: int) -> bool:
try:
os.killpg(pgid, sig)
return True
except OSError:
return False
def _descendants_deepest_first(root_pid: int):
"""Fallback tree enumeration via ps when a process-group kill is not
available: children die before their parents can respawn or orphan them."""
try:
out = subprocess.run(
["ps", "-eo", "pid=,ppid="], capture_output=True, text=True, check=False
).stdout
except OSError:
return []
children = {}
for line in out.splitlines():
parts = line.split()
if len(parts) != 2:
continue
try:
pid, ppid = int(parts[0]), int(parts[1])
except ValueError:
continue
children.setdefault(ppid, []).append(pid)
order, queue = [], [root_pid]
while queue:
for child in children.get(queue.pop(0), []):
order.append(child)
queue.append(child)
return list(reversed(order))
def _signal_group_or_tree(pid: int, sig: int) -> None:
"""Signal the pid's process group, falling back to a deepest-first tree
walk when the group kill is unavailable."""
if not _killpg_quiet(pid, sig):
for descendant in _descendants_deepest_first(pid):
_kill_quiet(descendant, sig)
_kill_quiet(pid, sig)
def _pre_reuse_descendant_pids(root_pid, children, predates_reuse, skip_pid=None):
"""Direct children that predate a recycled leader pid, plus each of those
children's full subtree.
Toolhelp still lists the original tree under a dead pid as parent, mixed
with the new process's own children. The start-time cutoff applies only to
direct children. A pre-reuse child's later descendants stay original-tree
work even if they started after the reuse.
"""
keep = set()
queue = []
for child in children.get(root_pid, []):
if skip_pid is not None and child == skip_pid:
continue
if not predates_reuse(child):
continue
queue.append(child)
while queue:
pid = queue.pop(0)
if skip_pid is not None and pid == skip_pid:
continue
if pid in keep:
continue
keep.add(pid)
queue.extend(children.get(pid, []))
return keep
def kill_tree(root_pid: int, grace: float, job_name=None, expected_identity=None) -> bool:
"""TERM the pid's process group (workers are started as group leaders),
falling back to a deepest-first tree walk; grace, then KILL survivors.
`job_name` is Windows-only (the worker's job object, the pgid analog) and
is ignored on POSIX, where the pgid is derived from the pid itself.
`expected_identity` is Windows-only (GetProcessTimes identity recorded at
start) and is ignored on POSIX."""
if IS_WINDOWS:
return _win_kill_tree(root_pid, grace, job_name, expected_identity)
# Do NOT early-return just because the leader pid is dead: killpg targets
# the pgid, which persists while any group member lives even after the
# leader exits, so a dead leader can still front a live group we must sweep.
# Use _pid_running (zombie-aware), not _pid_alive: a just-exited leader is
# briefly a <defunct> zombie for which kill -0 still succeeds, and counting
# that as alive would misclassify the reap as timeout instead of
# died-without-result (and make the dead-leader sweep test timing-dependent).
leader_alive = _pid_running(root_pid)
# Snapshot the descendant set BEFORE any KILL: once the group leader is
# reaped its children reparent to init and drop out of the tree, so a set
# enumerated after the kill would miss them and leak orphans.
survivors = _descendants_deepest_first(root_pid)
_signal_group_or_tree(root_pid, signal.SIGTERM)
deadline = time.monotonic() + grace
while time.monotonic() < deadline:
if leader_alive and not _pid_alive(root_pid):
break
time.sleep(0.1)
_killpg_quiet(root_pid, signal.SIGKILL)
for pid in survivors:
_kill_quiet(pid, signal.SIGKILL)
_kill_quiet(root_pid, signal.SIGKILL)
return leader_alive
# --- the supervisor (runs inside the detached session) -------------------------
def classify_exit(rc: int, result_path, conf: dict):
result_size = None
if result_path:
try:
st = os.lstat(result_path)
if stat.S_ISREG(st.st_mode) and st.st_size > 0:
result_size = st.st_size
except OSError:
pass
if result_size is not None and result_size > conf["result_max"]:
return "failed", (
f"result exceeded byte cap ({result_size} > {conf['result_max']} bytes)"
)
if rc == 0:
if result_path is None or result_size is not None:
return "done", "worker exited 0"
return "failed", "worker exited 0 without publishing a non-empty result"
if rc < 0:
if result_size is not None:
return "done", f"worker killed by signal {-rc} after publishing its result"
return "died-without-result", (
f"worker killed by signal {-rc} with no result evidence"
)
return "failed", f"worker exited {rc}"
def classify_exit_with_pending_reap(rc: int, result_path, conf: dict, reap_pending: bool):
"""Classify a worker that already exited, optionally under a pending reap.
When reap is pending (Windows `.reap` or POSIX SIGTERM flag) and the worker
was killed by the fallback path, classify_exit would record "failed" for a
non-zero kill exit — prefer timeout. When the worker already completed
successfully (done + result), keep that: a late reap must not rewrite a
finished peer run.
"""
state, reason = classify_exit(rc, result_path, conf)
if reap_pending and state != "done":
return "timeout", "reaped on request before completion"
return state, reason
def _reap_worker(proc, conf: dict, job_name=None) -> None:
# Deliberately parallel to kill_tree but driven by proc.poll(): an unreaped
# Popen child is a zombie that os.kill(pid, 0) still reports alive, so the
# pid-based liveness check would burn the whole grace window.
if proc.poll() is not None:
return
if IS_WINDOWS:
_win_kill_tree(proc.pid, conf["grace"], job_name)
try:
proc.wait(timeout=5)
except Exception:
pass
return
_signal_group_or_tree(proc.pid, signal.SIGTERM)
deadline = time.monotonic() + conf["grace"]
while time.monotonic() < deadline:
if proc.poll() is not None:
break
time.sleep(0.1)
if proc.poll() is None:
_killpg_quiet(proc.pid, signal.SIGKILL)
for pid in _descendants_deepest_first(proc.pid):
_kill_quiet(pid, signal.SIGKILL)
try:
proc.wait(timeout=5)
except Exception:
pass
def _reap_requested(flag: dict, job_dir: str) -> bool:
"""POSIX delivers the reap as SIGTERM (sets flag). Windows has no reliable
directed-signal path to a detached, console-less supervisor, so cmd_reap
drops a `.reap` marker in the job dir and the loop polls for it."""
if flag["reap"]:
return True
if IS_WINDOWS and os.path.lexists(os.path.join(job_dir, ".reap")):
return True
return False
def _interruptible_sleep(secs: float, flag: dict, job_dir: str) -> None:
end = time.monotonic() + secs
while time.monotonic() < end:
# On POSIX _reap_requested reduces to flag["reap"] (the IS_WINDOWS
# branch never fires), so this is the original signal-driven behavior.
if _reap_requested(flag, job_dir):
return
time.sleep(min(0.1, max(0.01, end - time.monotonic())))
def _is_system32_wsl_bash(path: str) -> bool:
"""True for Windows System32 WSL launchers, including Sysnative aliases."""
if not path:
return False
base = os.path.basename(path).lower()
if base not in ("bash", "bash.exe", "sh", "sh.exe"):
return False
system_root = os.environ.get("SystemRoot") or r"C:\Windows"
windows_root = os.path.abspath(system_root)
blocked_parents = {
os.path.normcase(os.path.join(windows_root, name))
for name in ("System32", "Sysnative")
}
parent = os.path.normcase(os.path.dirname(os.path.abspath(path)))
return parent in blocked_parents
def _git_bash_well_known_paths():
"""Standard Git for Windows bash.exe locations."""
pf64 = os.environ.get("ProgramW6432") or ""
pf = os.environ.get("ProgramFiles") or r"C:\Program Files"
pf86 = os.environ.get("ProgramFiles(x86)") or r"C:\Program Files (x86)"
local = os.environ.get("LOCALAPPDATA") or ""
roots = []
seen = set()
for root in (pf64, pf, pf86):
if not root:
continue
key = os.path.normcase(os.path.abspath(root))
if key in seen:
continue
seen.add(key)
roots.append(root)
paths = []
for root in roots:
paths.extend([
os.path.join(root, "Git", "bin", "bash.exe"),
os.path.join(root, "Git", "usr", "bin", "bash.exe"),
])
if local:
paths.extend([
os.path.join(local, "Programs", "Git", "bin", "bash.exe"),
os.path.join(local, "Programs", "Git", "usr", "bin", "bash.exe"),
])
return paths
def _windows_path_shell_candidates():
"""Every bash/sh on PATH in PATH order (not only shutil.which's first hit)."""
path_env = os.environ.get("PATH") or ""
names = ("bash.exe", "bash", "sh.exe", "sh")
found = []
seen = set()
for directory in path_env.split(os.pathsep):
if not directory:
continue
for name in names:
candidate = os.path.join(directory, name)
try:
if not os.path.isfile(candidate):
continue
except OSError:
continue
key = os.path.normcase(os.path.abspath(candidate))
if key in seen:
continue
seen.add(key)
found.append(candidate)
return found
def _env_assignment_token(token: str, allow_option_like: bool = False) -> bool:
"""True for env(1) NAME=value operands (not options or the command)."""
if (
not token
or (token.startswith("-") and not allow_option_like)
or "=" not in token
):
return False
return True
def _env_option_advance(tok: str) -> int:
"""How many argv slots an env(1) option occupies (incl. the option itself).
GNU env options that take a separate operand: -u/--unset, -C/--chdir.
Attached `--name=value` forms are a single slot.
Short options may be clustered. No-operand flags (-i/-v and their exact
long aliases and signal-handling options) advance one slot; -u/-C consume
the rest of the token as an
attached operand or the next argv slot. Unsupported options fail closed
before worker detach.
(#1292 Codex P2)
"""
if tok in ("-u", "--unset", "-C", "--chdir"):
return 2
if tok.startswith(("--unset=", "--chdir=")):
return 1
if tok in ("--ignore-environment", "--debug"):
return 1
if tok == "--list-signal-handling" or tok in (
"--block-signal",
"--default-signal",
"--ignore-signal",
) or tok.startswith((
"--block-signal=",
"--default-signal=",
"--ignore-signal=",
)):
return 1
if tok == "--null":
raise RunnerError(
"env -0/--null cannot be used with a command by native Windows "
"peer workers; remove the null-output option"
)
if not tok.startswith("-"):
return 1
if tok.startswith("--"):
raise RunnerError(
f"unsupported env long option {tok!r} for native Windows peer "
"workers; use an exact supported option or pass -- before "
"option-like assignments"
)
cluster = tok[1:]
for index, option in enumerate(cluster):
if option in "iv":
continue
if option == "0":
raise RunnerError(
"env -0/--null cannot be used with a command by native "
"Windows peer workers; remove the null-output option"
)
if option == "S":
raise RunnerError(
"env -S/--split-string is unsupported for native Windows "
"peer workers; pass env assignments and the command as "
"separate arguments"
)
if option in "uC":
return 1 if index + 1 < len(cluster) else 2
raise RunnerError(
f"unsupported env short-option cluster {tok!r} for native "
"Windows peer workers; pass env options separately"
)
return 1
def _env_bash_index(argv):
"""Locate the env(1)-launched bash/sh command, for #1268/#1292 rewriting.
Matches the production cross-model shape `env VAR=… bash script.sh …`
(#1268). Operand-taking options (-u/-C and long forms) consume their
arguments before the command token is sought (#1292). Split-string forms
fail closed because Python shlex does not match Git env.exe semantics.
Returns (argv_index, None), or (-1, None) when no bash/sh command is
present.
"""
if not argv:
return -1, None
if os.path.basename(argv[0]).lower() not in ("env", "env.exe"):
return -1, None
i = 1
options_done = False
while i < len(argv):
tok = argv[i]
if tok in ("-", "--") and not options_done:
options_done = True
i += 1
continue
if _env_assignment_token(tok, allow_option_like=options_done):
options_done = True
i += 1
continue
if not options_done and (
tok in ("-S", "--split-string") or tok.startswith(
("-S", "--split-string=")
)
):
raise RunnerError(
"env -S/--split-string is unsupported for native Windows "
"peer workers; pass env assignments and the command as "
"separate arguments"
)
if not options_done and tok.startswith("-"):
span = _env_option_advance(tok)
if span > 1 and i + 1 >= len(argv):
return -1, None
i += span
continue
base = os.path.basename(tok).lower()
if base in ("bash", "bash.exe", "sh", "sh.exe"):
return i, None
return -1, None
return -1, None
def _windows_path_is_absolute(path: str) -> bool:
"""True for Windows absolute paths (drive letter or path separator)."""
return os.sep in path or (len(path) >= 2 and path[1] == ":")
def _prefer_windows_posix_shell(token: str) -> str:
"""Absolute non-WSL bash/sh kept; bare names and System32 go through resolve.
Explicit absolute paths (portable Git, custom installs) must not be
substituted by the preferred resolver (#1292 Codex P2). Bare `bash`/`sh`
and System32 WSL launchers still use `_resolve_windows_posix_shell()`.
"""
if _windows_path_is_absolute(token):
path = os.path.abspath(token)
if not os.path.isfile(path):
raise RunnerError(
f"peer worker shell does not exist or is not a regular file: {token}"
)
if _is_system32_wsl_bash(path):
return _resolve_windows_posix_shell()
return path
return _resolve_windows_posix_shell()
def _rewrite_windows_env_bash_argv(argv):
"""Rewrite bare bash/sh inside an env-prefixed argv.
Returns (argv, resolved_shell_or_None). Raises RunnerError when a bash/sh
token is present but no usable non-WSL shell can be resolved. Absolute
non-WSL bash tokens are kept unchanged (#1292 P2). Split-string options
are rejected before detach because their parser semantics are not safely
reproduced here (#1292).
"""
idx, split_prefix = _env_bash_index(argv)
if idx < 0:
return list(argv), None
out = list(argv)
assert split_prefix is None
shell = _prefer_windows_posix_shell(out[idx])
if os.path.normcase(os.path.abspath(out[idx])) != os.path.normcase(shell):
out[idx] = shell
return out, shell
def _resolve_windows_posix_shell() -> str:
"""Absolute path to a non-WSL POSIX shell for native Windows peer workers.
Order: CE_PEER_BASH, CLAUDE_CODE_GIT_BASH_PATH, well-known Git Bash
installs, then every PATH bash/sh excluding System32 WSL. Fail closed when
nothing usable remains — never select System32\\bash.exe (#1268).
"""
candidates = []
for key in ("CE_PEER_BASH", "CLAUDE_CODE_GIT_BASH_PATH"):
val = (os.environ.get(key) or "").strip()
if val:
candidates.append(val)
candidates.extend(_git_bash_well_known_paths())
candidates.extend(_windows_path_shell_candidates())
seen = set()
for raw in candidates:
path = os.path.abspath(raw)
key = os.path.normcase(path)
if key in seen:
continue
seen.add(key)
if not os.path.isfile(path):
continue
if _is_system32_wsl_bash(path):
continue
return path
raise RunnerError(
"no usable Git Bash (or other non-WSL POSIX shell) for native Windows "
"peer workers; install Git for Windows or set CE_PEER_BASH / "
"CLAUDE_CODE_GIT_BASH_PATH to an absolute bash.exe path "
"(System32\\bash.exe / WSL is not used)"
)
def _popen_argv(argv):
"""Argv for subprocess.Popen.
On Windows, CreateProcess does not honor shebang, so a bare *.sh / *.bash
worker must be launched through bash/sh. Prefer Git Bash over System32
WSL bash (#1268). Bare `bash`/`sh` prefixes (review skills) and bare
`bash`/`sh` tokens after `env VAR=…` (cross-model) are rewritten to that
absolute path. Explicit absolute non-WSL bash/sh paths are kept (#1292 P2).
meta.json still records the caller argv for authorize-dispatch contracts
that forbid a shell prefix on ce-work.
"""
if not IS_WINDOWS or not argv:
return list(argv)
head = argv[0]
base = os.path.basename(head).lower()
if base in ("env", "env.exe"):
rewritten, _shell = _rewrite_windows_env_bash_argv(argv)
return rewritten
if base in ("bash", "bash.exe", "sh", "sh.exe"):
shell = _prefer_windows_posix_shell(head)
if os.path.normcase(os.path.abspath(head)) == os.path.normcase(shell):
return list(argv)
return [shell] + list(argv[1:])
lower = head.lower()
if not (lower.endswith(".sh") or lower.endswith(".bash")):
return list(argv)
shell = _resolve_windows_posix_shell()
return [shell, head] + list(argv[1:])
def supervise(job_dir: str, argv, result_path, conf: dict, ack_fd: int) -> None:
"""The watchdog around the worker child. Owns liveness (out.log growth),
the idle/hard windows, byte caps, reap-on-request, and the single terminal
classification."""
flag = {"reap": False}
def on_term(signum, frame):
flag["reap"] = True
if not IS_WINDOWS:
signal.signal(signal.SIGTERM, on_term)
signal.signal(signal.SIGHUP, signal.SIG_IGN)
# On Windows there is no SIGHUP and no reliable directed SIGTERM to a
# detached supervisor; reap arrives via the `.reap` marker polled below.
acked = False
def ack():
nonlocal acked
if acked:
return
acked = True
# Windows detach has no ack pipe (ack_fd is None): the pid file written
# just above IS the ack, and the parent polls for it.
if ack_fd is None:
return
try:
os.write(ack_fd, b"ok")
os.close(ack_fd)
except OSError:
pass
log_fd = None
job_name = None
job_handle = None
try:
log_fd = os.open(
os.path.join(job_dir, "out.log"),
os.O_WRONLY | os.O_APPEND | O_NOFOLLOW | O_BINARY)
if IS_WINDOWS:
# Created BEFORE the worker so the tree can never start outside it.
# The handle is held for the supervisor's lifetime; the job is what
# makes teardown reach descendants of an exited leader.
job_name = _win_job_name(job_dir)
job_handle = _win_create_job(job_name)
devnull = os.open(os.devnull, os.O_RDONLY)
try:
# Export the interpreter running this supervisor so Windows workers
# (and any adapter that honors it) do not re-resolve to the Store
# python3 stub — see resolve-python convention / #1247.
worker_env = {
**os.environ,
"CE_PEER_JOB_ID": os.path.basename(job_dir),
"CE_PEER_PYTHON": sys.executable,
}
popen_kwargs = dict(
stdin=devnull,
stdout=log_fd,
stderr=log_fd,
env=worker_env,
close_fds=True,
)
if IS_WINDOWS:
# New process group so the worker's own tree is isolated; reap =
# job terminate + Toolhelp walk (there is no killpg on Windows).
# CREATE_NO_WINDOW because the supervisor is console-less, so
# without it Windows allocates a NEW console per worker and the
# user sees a window flash for every job. (It is mutually
# exclusive with DETACHED_PROCESS, which is why the supervisor
# itself uses DETACHED_PROCESS and only the worker uses this.)
# CREATE_SUSPENDED: assign to the Job Object before any worker
# code runs, so early child spawns inherit membership.
popen_kwargs["creationflags"] = (
subprocess.CREATE_NEW_PROCESS_GROUP
| _WIN_NO_WINDOW
| _CREATE_SUSPENDED)
else:
popen_kwargs["start_new_session"] = True # worker leads its own group
# Wrap bare *.sh on Windows at spawn time only — meta still has the
# caller argv (see _popen_argv).
proc = subprocess.Popen(_popen_argv(argv), **popen_kwargs)
finally:
os.close(devnull)
pid_doc = {
"supervisor_pid": os.getpid(),
"worker_pid": proc.pid,
}
if IS_WINDOWS:
sup_ident = _win_process_identity(os.getpid())
worker_ident = _win_process_identity(proc.pid)
if sup_ident:
pid_doc["supervisor_identity"] = sup_ident
if worker_ident:
pid_doc["worker_identity"] = worker_ident
# Assign while still suspended, then resume. Record the job only
# once the worker is actually a member, so a later reap never trusts
# a name that owns nothing. If assignment fails (job creation denied,
# or a nested-job restriction), leave it unset and let teardown fall
# back to the Toolhelp walk.
if job_handle is not None and _win_assign_to_job(job_handle, proc.pid):
pid_doc["job_name"] = job_name
else:
job_name = None
if not _win_resume_process(proc.pid):
raise RunnerError(
f"could not resume suspended Windows worker pid {proc.pid}"
)
else:
# pgid drives POSIX group kills; Windows reaps by job object.
pid_doc["supervisor_pgid"] = os.getpgid(0)
# The pid file lands before the parent is acked, so a returned `start`
# guarantees the detach marker exists (status never mis-reads a fresh
# job as never-started).
write_atomic(os.path.join(job_dir, "pid"), (json.dumps(pid_doc) + "\n").encode())
except Exception as exc:
write_terminal(job_dir, "failed", f"could not launch worker: {exc}")
ack()
return
ack()
start_t = time.monotonic()
last_growth = start_t
last_size = 0
while True:
rc = proc.poll()
if rc is not None:
state, reason = classify_exit_with_pending_reap(
rc, result_path, conf, _reap_requested(flag, job_dir),
)
break
if _reap_requested(flag, job_dir):
# Classification is fixed BEFORE the kill: even if the worker
# publishes and exits 0 during the grace window, the supervisor's
# record wins (R3).
_reap_worker(proc, conf, job_name)
state, reason = "timeout", "reaped on request before completion"
break
try:
size = os.fstat(log_fd).st_size
except OSError:
size = last_size
now = time.monotonic()
if size > last_size:
last_size, last_growth = size, now
if size > conf["log_max"]:
_reap_worker(proc, conf, job_name)
state, reason = "failed", (
f"out.log exceeded byte cap ({size} > {conf['log_max']} bytes)"
)
break
if conf["idle"] is not None and now - last_growth >= conf["idle"]:
_reap_worker(proc, conf, job_name)
state, reason = "timeout", f"no output for {conf['idle']:g}s (idle window)"
break
if now - start_t >= conf["hard"]:
_reap_worker(proc, conf, job_name)
state, reason = "timeout", f"hard cap {conf['hard']:g}s exceeded"
break
_interruptible_sleep(conf["poll"], flag, job_dir)
# An externally killed worker can leave group members behind (its shell's
# children); sweep the group before publishing so no orphan outlives the
# terminal record. A pgid cannot be recycled while members remain.
if IS_WINDOWS:
# Job-object sweep: unlike taskkill this still reaches descendants when
# the worker leader has already exited, which is the orphan case the
# POSIX killpg pair below covers.
_win_kill_tree(proc.pid, min(conf["grace"], 1.0), job_name)
else:
_killpg_quiet(proc.pid, signal.SIGTERM)
_killpg_quiet(proc.pid, signal.SIGKILL)
write_terminal(job_dir, state, reason)
def detach_supervisor(job_dir: str, argv, result_path, conf: dict) -> bool:
"""setsid double-fork. The grandchild (new session, stdio on /dev/null,
reparented to init) runs the supervisor; the parent returns once the
supervisor acks that the pid file exists."""
if IS_WINDOWS:
return detach_supervisor_windows(job_dir, argv, result_path, conf)
sys.stdout.flush()
sys.stderr.flush()
read_fd, write_fd = os.pipe()
pid1 = os.fork()
if pid1 == 0:
os.close(read_fd)
os.setsid()
if os.fork() > 0:
os._exit(0)
rc = 0
try:
devnull = os.open(os.devnull, os.O_RDWR)
os.dup2(devnull, 0)
os.dup2(devnull, 1)
os.dup2(devnull, 2)
if devnull > 2:
os.close(devnull)
supervise(job_dir, argv, result_path, conf, write_fd)
except BaseException:
rc = 1
try:
write_terminal(
job_dir, "failed", "supervisor crashed before classification",
overwrite=False,
)
except BaseException:
pass
os._exit(rc)
os.close(write_fd)
os.waitpid(pid1, 0)
ack = b""
try:
while len(ack) < 2:
chunk = os.read(read_fd, 2 - len(ack))
if not chunk:
break
ack += chunk
finally:
os.close(read_fd)
return ack == b"ok"
def detach_supervisor_windows(job_dir: str, argv, result_path, conf: dict) -> bool:
"""Windows detach: there is no fork/setsid, so re-invoke this script as a
fresh DETACHED_PROCESS running the internal `__supervise` entrypoint. The
spawn spec travels through an owner-private file in the job dir; the parent
returns once the supervisor has left its ack marker (the pid file, or a
terminal status if the worker could not launch). CREATE_BREAKAWAY_FROM_JOB
is the analog of setsid's reparent-to-init: it lets the supervisor outlive a
launching harness that runs inside a kill-on-close Job Object, falling back
when the job forbids breakaway."""
spec = {"argv": list(argv), "result_path": result_path, "conf": conf}
create_exclusive(
os.path.join(job_dir, ".spawn.json"),
(json.dumps(spec) + "\n").encode(),
)
cmd = [sys.executable, os.path.abspath(__file__), "__supervise", job_dir]
base_flags = subprocess.DETACHED_PROCESS | subprocess.CREATE_NEW_PROCESS_GROUP
devnull = os.open(os.devnull, os.O_RDWR)
proc = None
try:
for flags in (base_flags | subprocess.CREATE_BREAKAWAY_FROM_JOB, base_flags):
try:
proc = subprocess.Popen(
cmd,
stdin=devnull, stdout=devnull, stderr=devnull,
close_fds=True, creationflags=flags,
)
break
except OSError:
proc = None
if proc is None:
return False
finally:
os.close(devnull)
pid_path = os.path.join(job_dir, "pid")
status_path = os.path.join(job_dir, "status")
deadline = time.monotonic() + 15.0
while time.monotonic() < deadline:
if os.path.lexists(pid_path) or os.path.lexists(status_path):
return True
if proc.poll() is not None:
# Supervisor process exited without leaving a marker: detach failed.
return os.path.lexists(pid_path) or os.path.lexists(status_path)
time.sleep(0.05)
# Deadline with the supervisor still running. Do not abandon it: reporting
# a detach failure while leaving a live, unreachable supervisor/worker pair
# behind is exactly the orphan this runner exists to prevent. The snapshot
# walk reaches the worker as the supervisor's child.
try:
if proc.poll() is None:
_win_kill_tree(proc.pid, 0.0)
except Exception:
pass
return False
def _win_supervise_from_spec(job_dir: str) -> int:
"""Internal `__supervise` entrypoint: the detached Windows supervisor. Drops
its console-less std handles onto NUL, reads the owner-checked spawn spec,
and runs the shared supervisor loop with a file-based (not fd) ack."""
rc = 0
# Redundant with detach_supervisor_windows, which already binds this
# process's stdio to NUL via Popen -- kept deliberately so the entrypoint is
# self-contained: a supervisor is long-lived, and any future/manual
# invocation that inherited a real pipe could block forever once it filled.
try:
devnull = os.open(os.devnull, os.O_RDWR)
for fd in (0, 1, 2):
try:
os.dup2(devnull, fd)
except OSError:
pass
if devnull > 2:
os.close(devnull)
except OSError:
pass
try:
_check_owned_dir(job_dir)
spec = json.loads(read_owned(os.path.join(job_dir, ".spawn.json"), META_READ_CAP))
argv = spec["argv"]
result_path = spec.get("result_path")
conf = spec["conf"]
try:
os.unlink(os.path.join(job_dir, ".spawn.json"))
except OSError:
pass
supervise(job_dir, argv, result_path, conf, None)
except BaseException:
rc = 1
try:
write_terminal(
job_dir, "failed", "supervisor crashed before classification",
overwrite=False,
)
except BaseException:
pass
return rc
# --- subcommands ---------------------------------------------------------------
def sweep_stale_runs(skill_dir: str, keep: str) -> None:
"""Best-effort retention (R14): remove sibling run roots older than 24h.
Owner-checked via lstat; never raises, never touches the current run."""
try:
entries = list(os.scandir(skill_dir))
except OSError:
return
now = time.time()
euid = _euid()
keep_abs = os.path.abspath(keep)
for entry in entries:
if os.path.abspath(entry.path) == keep_abs:
continue
try:
st = entry.stat(follow_symlinks=False)
except OSError:
continue
if not stat.S_ISDIR(st.st_mode):
continue
if IS_WINDOWS:
try:
if not _win_owns_path(entry.path):
continue
except OSError:
continue
elif euid is not None and st.st_uid != euid:
continue
if now - st.st_mtime <= SWEEP_AGE_SECS:
continue
shutil.rmtree(entry.path, ignore_errors=True)
def _require_detach_support() -> None:
"""Detached peer jobs need a supported detach path: os.fork/os.setsid on
POSIX, or the native Windows DETACHED_PROCESS path (#1243). Checked first,
before jobs_root_base()/geteuid, so an unsupported host fails with this clear
message instead of jobs_root_base()'s unrelated "effective user ID is
unavailable" error or an AttributeError mid-detach. Native Windows is now
supported; only a non-win32 Python missing fork/setsid (some embedded
builds) is rejected here."""
if IS_WINDOWS:
return
if not hasattr(os, "fork") or not hasattr(os, "setsid"):
raise RunnerError(
"detached peer jobs require os.fork/os.setsid on this platform; no "
"job was started. Run under a POSIX Python, or on native Windows use "
"a Windows Python 3 build (see "
"EveryInc/compound-engineering-plugin#1243)."
)
def cmd_start(args, worker_argv) -> int:
_require_detach_support()
for flag, value in (("--skill", args.skill), ("--run-id", args.run_id)):
if not _is_safe_token(value):
raise RunnerError(f"{flag} must match [A-Za-z0-9._-]+ and not be all dots (got {value!r})")
if args.label is not None and not _is_safe_token(args.label):
raise RunnerError(f"--label must match [A-Za-z0-9._-]+ and not be all dots (got {args.label!r})")
if not worker_argv:
raise RunnerError("no worker argv; place it after `--`")
base = jobs_root_base()
skill_dir = skill_runs_root(args.skill)
run_dir = os.path.join(skill_dir, args.run_id)
jobs_root = os.path.join(run_dir, "jobs")
ensure_owned_dirs(skill_dir if skill_dir != os.path.join(base, args.skill) else base, jobs_root)
if not args.no_sweep:
sweep_stale_runs(skill_dir, keep=run_dir)
job_id, job_dir = claim_job_dir(jobs_root)
result_path = os.path.abspath(args.result_path) if args.result_path else None
argv0 = worker_argv[0]
problem = None
windows_posix_shell = None
base0 = os.path.basename(argv0).lower()
if IS_WINDOWS and base0 in ("bash", "bash.exe", "sh", "sh.exe"):
# Prefer Git Bash over PATH/System32 WSL before meta + detach (#1268).
# Keep an explicit absolute non-WSL bash (portable Git) (#1292 P2).
try:
resolved = _prefer_windows_posix_shell(argv0)
windows_posix_shell = resolved
except RunnerError as exc:
problem = str(exc)
resolved = argv0
elif IS_WINDOWS and base0 in ("env", "env.exe"):
# Production cross-model: env VAR=… bash script.sh — rewrite bash
# before detach so env cannot PATH-resolve System32 WSL (#1268).
if os.sep in argv0 or (len(argv0) >= 2 and argv0[1] == ":"):
resolved = os.path.abspath(argv0)
if not os.path.isfile(resolved):
problem = "does not exist or is not a regular file"
else:
resolved = shutil.which(argv0)
if resolved is None:
problem = "was not found on PATH"
resolved = argv0
try:
rewritten, shell = _rewrite_windows_env_bash_argv(list(worker_argv))
if shell is not None:
windows_posix_shell = shell
worker_argv = rewritten
except RunnerError as exc:
problem = str(exc) if problem is None else f"{problem}; {exc}"
elif os.sep in argv0 or (IS_WINDOWS and len(argv0) >= 2 and argv0[1] == ":"):
resolved = os.path.abspath(argv0)
if not os.path.isfile(resolved):
problem = "does not exist or is not a regular file"
elif IS_WINDOWS and resolved.lower().endswith((".sh", ".bash")):
# CreateProcess cannot run shebang scripts; _popen_argv wraps with
# Git Bash. Require that shell now so start fails closed, not after
# detach. Skip the X_OK check — Windows often marks .sh non-exec.
try:
windows_posix_shell = _resolve_windows_posix_shell()
except RunnerError as exc:
problem = str(exc)
elif not os.access(resolved, os.X_OK):
problem = "is not executable"
else:
resolved = shutil.which(argv0)
if resolved is None:
problem = "was not found on PATH"
resolved = argv0
elif IS_WINDOWS and resolved.lower().endswith((".sh", ".bash")):
try:
windows_posix_shell = _resolve_windows_posix_shell()
except RunnerError as exc:
problem = str(exc)
elif IS_WINDOWS and os.path.basename(resolved).lower() in (
"bash", "bash.exe", "sh", "sh.exe",
):
# which() may have returned System32 WSL — rewrite now.
try:
resolved = _resolve_windows_posix_shell()
windows_posix_shell = resolved
except RunnerError as exc:
problem = str(exc)
argv = [resolved] + list(worker_argv[1:])
conf = cfg(args.skill)
meta = {
"job_id": job_id,
"skill": args.skill,
"run_id": args.run_id,
"label": args.label,
"input_digest": args.input_digest,
"started_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
"worker_argv": argv,
"result_path": result_path,
"sweep_enabled": not args.no_sweep,
"supervision": conf,
}
if windows_posix_shell:
meta["windows_posix_shell"] = windows_posix_shell
try:
create_exclusive(
os.path.join(job_dir, "meta.json"),
(json.dumps(meta, indent=2) + "\n").encode(),
)
except OSError as exc:
raise RunnerError(f"cannot write job metadata for {job_id}: {exc}")
if problem is not None:
raise RunnerError(
f"preflight failed for job {job_id}: worker {argv0!r} {problem}; "
f"nothing was detached (job left never-started at {job_dir})"
)
try:
create_exclusive(os.path.join(job_dir, "out.log"))
except OSError as exc:
raise RunnerError(
f"preflight failed for job {job_id}: job dir not writable ({exc}); "
"nothing was detached"
)
if not detach_supervisor(job_dir, argv, result_path, conf):
raise RunnerError(
f"detach failed for job {job_id}: supervisor did not acknowledge; "
f"inspect {job_dir}"
)
print(job_id)
return 0
def _emit_states(rows, as_json: bool) -> None:
if as_json:
print(json.dumps(
[{"ref": r, "job_dir": d, "state": s} for r, d, s in rows]
))
elif len(rows) == 1:
print(rows[0][2])
else:
for ref, _, state in rows:
print(f"{ref}\t{state}")
def cmd_status(args) -> int:
rows = []
for ref in args.jobs:
job_dir = resolve_job_dir(ref, args.skill)
rows.append((ref, job_dir, job_state(job_dir)))
_emit_states(rows, args.json)
return 0
def cmd_wait(args) -> int:
dirs = [(ref, resolve_job_dir(ref, args.skill)) for ref in args.jobs]
deadline = time.monotonic() + max(0.0, args.max_secs)
rows = [(ref, d, "running") for ref, d in dirs]
while True:
# Settled states are final; only still-running jobs get re-read.
rows = [
(ref, d, state if state != "running" else job_state(d))
for ref, d, state in rows
]
if all(state != "running" for _, _, state in rows):
break
remaining = deadline - time.monotonic()
if remaining <= 0:
break
time.sleep(min(1.0, remaining))
_emit_states(rows, args.json)
return 0
def _emit_bytes(data: bytes) -> None:
buffer = getattr(sys.stdout, "buffer", None)
if buffer is not None:
buffer.write(data)
buffer.flush()
else:
sys.stdout.write(data.decode("utf-8", "replace"))
def _report_absent_artifact(target: str, args) -> int:
"""An absent --path artifact is an outcome, not a read error: a peer that
skipped its gate exits 0 and writes nothing, so the file is legitimately
missing on the most common fold-in path. Name that outcome, and when the
caller also passed the job id, name the job's state -- otherwise "still
running" and "ran, produced nothing" arrive as one errno the caller cannot
act on. Each outcome keeps the exit code the job-result contract already
assigns it, so a trust or lookup failure never reads as the routine skip:
2 running, 4 ownership, 1 unknown job, 3 settled with no artifact."""
sys.stderr.write(f"peer-job-runner: no artifact at {target}\n")
if not args.job:
return 3
try:
job_dir = resolve_job_dir(args.job, args.skill)
except RunnerError as exc:
sys.stderr.write(f"peer-job-runner: {exc}\n")
return 1
state = job_state(job_dir)
if state == "unreadable":
# Do not read `reason` here: job_dir already failed its owner check, and
# O_NOFOLLOW guards only the final component, so a swapped directory
# could redirect that read.
sys.stderr.write(
f"peer-job-runner: job state unreadable (ownership or corruption): {job_dir}\n"
)
return 4
if state == "running":
sys.stderr.write(f"peer-job-runner: job {args.job} is still running\n")
return 2
reason = job_reason(job_dir)
sys.stderr.write(
f"peer-job-runner: job {args.job}: {state}" + (f" ({reason})" if reason else "") + "\n"
)
return 3
def cmd_result(args) -> int:
if not getattr(args, "path", None) and not args.job:
sys.stderr.write("peer-job-runner: result needs a job id or --path FILE\n")
return 2
if getattr(args, "path", None):
# Verified read of an arbitrary artifact: same fd-ownership check and
# bounded read as job results. Exists because fold-in filenames can embed
# values unknown at start time (so no --result-path was declared), yet the
# consumer must never read a predictable /tmp path unchecked.
target = os.path.abspath(args.path)
try:
data = read_owned(target, cfg()["result_max"])
except Unreadable as exc:
sys.stderr.write(f"peer-job-runner: unreadable: {exc}\n")
return 4
except FileNotFoundError:
return _report_absent_artifact(target, args)
except OSError as exc:
# Only a genuine ENOENT is "the peer produced nothing". Every other
# read failure means the path is there but was refused -- a planted
# symlink rejected by O_NOFOLLOW is the case this guard exists for --
# so it takes the trust-failure code, never the routine one.
sys.stderr.write(f"peer-job-runner: refused to read {target}: {exc}\n")
return 4
_emit_bytes(data)
return 0
job_dir = resolve_job_dir(args.job, args.skill)
state = job_state(job_dir)
if state == "unreadable":
sys.stderr.write(
f"peer-job-runner: job state unreadable (ownership or corruption): {job_dir}\n"
)
return 4
if state == "running":
sys.stderr.write("peer-job-runner: running\n")
return 2
if state != "done":
sys.stderr.write(f"peer-job-runner: {state}\n")
return 3
conf = cfg()
try:
meta = json.loads(read_owned(os.path.join(job_dir, "meta.json"), META_READ_CAP))
except Unreadable as exc:
sys.stderr.write(f"peer-job-runner: unreadable: {exc}\n")
return 4
except (OSError, ValueError) as exc:
sys.stderr.write(f"peer-job-runner: cannot read job metadata: {exc}\n")
return 4
result_path = meta.get("result_path") if isinstance(meta, dict) else None
if not result_path:
sys.stderr.write("peer-job-runner: job declared no result path; nothing to emit\n")
return 0
try:
data = read_owned(result_path, conf["result_max"])
except Unreadable as exc:
sys.stderr.write(f"peer-job-runner: unreadable: {exc}\n")
return 4
except OSError as exc:
sys.stderr.write(f"peer-job-runner: result missing or unreadable: {exc}\n")
return 3
_emit_bytes(data)
return 0
def cmd_reap(args) -> int:
job_dir = resolve_job_dir(args.job, args.skill)
state = job_state(job_dir)
if state in TERMINAL_STATES or state == "never-started":
return 0
if state == "unreadable":
sys.stderr.write(
f"peer-job-runner: job state unreadable (ownership or corruption): {job_dir}\n"
)
return 4
conf = cfg()
pid_doc = None
try:
pid_doc = json.loads(read_owned(os.path.join(job_dir, "pid"), META_READ_CAP))
except (Unreadable, OSError, ValueError):
pid_doc = None
if not isinstance(pid_doc, dict):
pid_doc = {}
sup_pid = pid_doc.get("supervisor_pid")
sup_pgid = pid_doc.get("supervisor_pgid")
worker_pid = pid_doc.get("worker_pid")
# Windows-only: the worker tree's job object. Named precisely so this
# process -- which never held the supervisor's handle -- can reopen and
# terminate the tree even after the worker leader has exited.
job_name = pid_doc.get("job_name")
worker_identity = pid_doc.get("worker_identity")
supervisor_identity = pid_doc.get("supervisor_identity")
supervisor_ours = (
isinstance(sup_pid, int)
and not (IS_WINDOWS and sup_pid == os.getpid())
and _pid_alive(sup_pid)
and (
not IS_WINDOWS
or _win_process_identity_matches(sup_pid, supervisor_identity)
)
)
if supervisor_ours:
# The supervisor owns TERM-grace-KILL and the terminal classification.
# POSIX signals it (SIGTERM to the group or pid); Windows drops the
# `.reap` marker the supervisor's loop polls for.
if IS_WINDOWS:
try:
with open(os.path.join(job_dir, ".reap"), "w") as f:
f.write("reap\n")
signaled = True
except OSError:
signaled = False
else:
signaled = (isinstance(sup_pgid, int) and _killpg_quiet(sup_pgid, signal.SIGTERM)) \
or _kill_quiet(sup_pid, signal.SIGTERM)
if signaled:
# kill -0 is true for a zombie, so confirm the classification landed
# rather than trusting the signal; fall through to self-cleanup if not.
# Windows: the supervisor only notices `.reap` on its next poll tick
# (default 2s), so min(grace, 1.0) alone is shorter than one poll and
# races into the fallback self-classify path.
wait_budget = min(conf["grace"], 1.0)
if IS_WINDOWS:
wait_budget = max(wait_budget, conf["poll"] + 0.25)
deadline = time.monotonic() + wait_budget
while time.monotonic() < deadline:
if job_state(job_dir) in TERMINAL_STATES:
return 0
time.sleep(0.05)
# Supervisor gone: perform the tree kill and classification ourselves,
# with a short grace so reap still returns quickly. Sweep whenever we have a
# worker pid, NOT only when its leader is still alive: a child can survive in
# the worker's process group after the leader exits, and kill_tree targets
# the pgid precisely so that orphan is swept instead of leaked. Guarding this
# on _pid_alive would re-defeat kill_tree's dead-leader-safe path. kill_tree
# returns whether the leader was alive, which is the reap classification.
worker_leader_alive = False
if isinstance(worker_pid, int):
worker_leader_alive = kill_tree(
worker_pid, min(conf["grace"], 1.0), job_name, worker_identity)
# A worker can publish its declared result and exit before this fallback runs
# (e.g. the supervisor died mid-run, then the worker completed cleanly). Honor
# that result instead of discarding it as died-without-result: read the
# declared result_path and classify from the artifact, mirroring
# classify_exit. Only with no usable result do we fall back to timeout (leader
# was alive) / died-without-result (leader gone).
result_path = None
try:
meta = json.loads(read_owned(os.path.join(job_dir, "meta.json"), META_READ_CAP))
result_path = meta.get("result_path") if isinstance(meta, dict) else None
except (Unreadable, OSError, ValueError):
result_path = None
result_size = None
if result_path:
try:
st = os.lstat(result_path)
if stat.S_ISREG(st.st_mode) and st.st_size > 0:
result_size = st.st_size
except OSError:
pass
if result_size is not None and result_size > conf["result_max"]:
word, reason = "failed", (
f"result exceeded byte cap ({result_size} > {conf['result_max']} bytes)"
)
elif result_size is not None:
word, reason = "done", "worker published its result before reap (supervisor was gone)"
elif worker_leader_alive:
word, reason = "timeout", (
"reaped by request; supervisor was gone, worker tree killed by reap"
)
else:
word, reason = "died-without-result", (
"supervisor and worker both gone without a terminal record"
)
write_terminal(job_dir, word, reason, overwrite=False)
return 0
# --- CLI -----------------------------------------------------------------------
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(
prog="peer-job-runner.py",
description=(
"Detached, supervised job lifecycle for delegated peer work: "
"no call here ever spans the worker's runtime."
),
epilog=EXIT_CODES_DOC,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
sub = parser.add_subparsers(dest="cmd", required=True)
p_start = sub.add_parser(
"start",
help="claim a job, preflight, detach the worker, print the job id",
)
p_start.add_argument("--skill", required=True)
p_start.add_argument("--run-id", required=True, dest="run_id")
p_start.add_argument("--label", default=None)
p_start.add_argument("--input-digest", default=None, dest="input_digest")
p_start.add_argument(
"--result-path", default=None, dest="result_path",
help="worker's expected result file; done then requires it non-empty",
)
p_start.add_argument(
"--no-sweep", action="store_true",
help="retain old sibling run roots (ce-work durable recovery)",
)
p_status = sub.add_parser("status", help="print each job's state word")
p_status.add_argument("--skill", default=None, help="limit job-id lookup to this skill")
p_status.add_argument("--json", action="store_true")
p_status.add_argument("jobs", nargs="+", help="job ids or job dir paths")
p_wait = sub.add_parser(
"wait", help="bounded poll until all watched jobs settle (or the cap)"
)
p_wait.add_argument("--skill", default=None, help="limit job-id lookup to this skill")
p_wait.add_argument("--max-secs", type=float, default=30.0, dest="max_secs")
p_wait.add_argument("--json", action="store_true")
p_wait.add_argument("jobs", nargs="+", help="job ids or job dir paths")
p_result = sub.add_parser(
"result",
help="emit a done job's artifact (exit: 0 done, 2 running, 3 other, 4 unreadable)",
)
p_result.add_argument("--skill", default=None, help="limit job-id lookup to this skill")
p_result.add_argument("job", nargs="?", default=None)
p_result.add_argument(
"--path",
default=None,
help=(
"ownership-checked bounded read of this file instead of a job's "
"declared result; pass the job id too so an absent file reports "
"that job's state"
),
)
p_reap = sub.add_parser(
"reap", help="terminate a running job now; no-op if already terminal"
)
p_reap.add_argument("--skill", default=None, help="limit job-id lookup to this skill")
p_reap.add_argument("job")
return parser
def main(argv) -> int:
# Internal Windows detach re-invocation (not a user-facing subcommand): the
# detached supervisor process runs `__supervise <job_dir>`. Gated on
# IS_WINDOWS so POSIX keeps its previous behavior exactly (argparse usage
# error), and so a non-win32 Python without geteuid -- where the ownership
# checks degrade -- can never be steered into exec'ing argv from a planted
# .spawn.json. Only the Windows detach path ever emits this argv.
if IS_WINDOWS and argv and argv[0] == "__supervise":
if len(argv) < 2:
return 2
return _win_supervise_from_spec(argv[1])
worker_argv = []
if "--" in argv:
split = argv.index("--")
argv, worker_argv = argv[:split], argv[split + 1:]
args = build_parser().parse_args(argv)
try:
if args.cmd == "start":
return cmd_start(args, worker_argv)
if args.cmd == "status":
return cmd_status(args)
if args.cmd == "wait":
return cmd_wait(args)
if args.cmd == "result":
return cmd_result(args)
if args.cmd == "reap":
return cmd_reap(args)
return 2
except RunnerError as exc:
sys.stderr.write(f"peer-job-runner: {exc}\n")
return 1
except Unreadable as exc:
sys.stderr.write(f"peer-job-runner: unreadable: {exc}\n")
return 4
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
SHA-256: aac862a60af97eb4b9da6c90a1c688d27618ccb830ea5b1ee816b2ae653bfb0a