skills/esmfold2-binder-design/references/modal.md

# Modal binder-design execution

Primary sources: [official binder-design example](https://modal.com/docs/examples/esmfold2_binder_design), [Modal Function API](https://modal.com/docs/sdk/py/latest/modal.Function), [Modal FunctionCall API](https://modal.com/docs/sdk/py/latest/modal.FunctionCall), [Modal SDK changelog](https://modal.com/docs/sdk/py/changelog), [Modal authentication](https://modal.com/docs/sdk/py/latest/modal.config), and [Modal scaling limits](https://modal.com/docs/guide/scale).

The official example:

- builds a pinned Biohub ESM image with deterministic CUDA settings
- caches roughly 50 GB of ESMC/ESMFold2/critic weights on a Volume
- stores result tables on a separate persistent Volume
- runs one `design` call per seed/template/target
- fans out with `spawn()` and gathers with `FunctionCall.gather`
- ranks candidates with combined interface/distogram evidence

Update upstream pins to the revisions validated by this plugin. Do not execute a mutable `main` or silently reuse stale cached weights.

## Durable control plane

The helper's `modal-jobs` command targets a deployed function that accepts keyword payloads. It is control-plane-only: every spawn, gather, and cancel command requires the same explicit non-secret Modal workspace name, environment name, app/function, positive immutable function deployment version, and state path. Freeze those values in the reviewed execution contract; never infer them by reading token or profile files. Before every manager action, the helper resolves the active credential-bound workspace through `Workspace.from_context().hydrate()`, fails closed if resolution fails or the verified name differs from `--workspace-name`, and persists only that verified non-secret name. The helper requires the reviewed stable Modal Python SDK `1.5.2`, passes both the deployment version and `environment_name` to `Function.from_name(...)`, and records the complete identity with the SDK version in state. A state without the workspace/environment binding is intentionally rejected as ambiguous rather than upgraded. This matters because the workspace, environment, deployed function, SDK `FunctionCall` methods, and exception taxonomy determine terminal-versus-resumable outcomes. The command rejects empty/non-string call IDs and persists each accepted ID before gather; it does not infer scientific provenance from arbitrary return values.

During gather, a built-in polling `TimeoutError` and non-terminal `modal.exception.Error` control-plane failures keep the durable call ID resumable. Explicit terminal Modal result errors, cancellation, and ordinary Python exceptions deserialized from remote user code are persisted as terminal outcomes. This distinction follows the pinned SDK's `FunctionCall.get()` result-decoding path; treating an arbitrary remote exception as a transport retry would otherwise leave a completed failed call pending forever.

Before `spawn`, refresh current pricing and account/payment readiness, show the immutable deployment version, app/function, payload count, GPU type, timeout, concurrency, persistent-volume plan, and cost ceiling, then obtain separate explicit current-turn confirmation. The hidden execution token is supplied only after that review; it is a post-consent backstop rather than authorization.

Use the same state file and repeat the exact frozen `--workspace-name`, `--environment-name`, `--app-name`, `--function-name`, and `--function-version` for `gather` or `cancel`. The helper validates all five values before resolving or acting on a persisted `FunctionCall` ID. Cancellation remains `cancellation-requested` until a later gather reconciles a late result or provider-accepted cancellation. Never pass Modal tokens on the command line, persist profile contents, or treat a workspace name as a credential; the native profile or token environment pair remains outside job state.

There is one unavoidable RPC ambiguity: a client can die after Modal accepts a spawn but before the call ID is persisted. On reload, a durable `spawning` entry becomes `submission-indeterminate`, records workspace, environment, app/function, deployment and SDK versions, and payload digest for manual dashboard reconciliation, and must never be auto-resubmitted. This prevents a retry from silently duplicating GPU work when acceptance is unknown. A timeout, disconnect, or other unclassified exception returned by `spawn()` is treated the same way because it can also occur after provider acceptance; it is not mislabeled as a definitive failure.

For a call to become `completed`, the deployed function must return exactly a compact `{ "submission_sha256": ..., "result": ..., "provenance": ... }` envelope. Both the envelope and provenance input digest must match the submitted payload; the provenance endpoint must match the deployed app/function, and `provenance.parameters.modal_provider_identity` must exactly repeat the frozen workspace, environment, app, function, function version, and SDK version. It must also pass the shared schema, include the `modal` route, applicable pinned model/code/HF revisions and timestamps, and at least one artifact metadata record with path, size, media type, and a syntactically valid SHA-256 field. Binder envelopes must record the complete critic model revision map.

The control plane validates that provider-declared artifact metadata; it does **not** retrieve the remote artifact or independently recompute its checksum. Consumers requiring byte-level verification must materialize the artifact through an authenticated, bounded transfer and compare the downloaded bytes with the declared size and digest. Large arrays and structures should remain in durable storage and be referenced by metadata rather than embedded in state. A result envelope is limited to 512 KiB, 50,000 JSON nodes, and 32 nesting levels. Bare JSON, stale/wrong-input results, incomplete provenance, non-JSON values, cycles, and oversized envelopes fail closed; the durable call stays auditable rather than becoming false scientific evidence.

Modal documents up to 1,000 concurrent inputs per map invocation and larger limits for spawned async calls, but these are service limits, not a campaign target. This plugin deliberately caps one `modal-jobs` state at 128 payloads, reads at most 16 MiB of input JSON, and caps the complete durable state at 72 MiB. `--max-jobs` is a second, caller-selected ceiling and cannot raise those hard limits. Split larger campaigns into independently identified bounded shards and inspect current account/provider limits. A single-seed smoke is an integration check; launch full campaigns only from a pinned, bounded campaign contract.

## Bounded smoke

The internal harness verifies the official Modal examples checkout and source hashes, creates a temporary copy, then pins Transformers, ESMC-6B, and every experimental inversion/critic checkpoint. The direct integration smoke uses one H100, batch size one, seed zero, and a one-hour timeout. It writes evidence to a fresh output directory, uses ticket-scoped model/results Volumes capped at 55 GiB combined, and removes both after every outcome.

Use the official `main` entry point for this smoke. Do not use the `sweep` entry point: even with one seed, its GPU orchestrator can overlap a second H100 design worker and would no longer be a one-worker integration check.