docs/PLATFORM-ASKS.md

# Platform asks — capabilities the film pipeline needs from Picsart

Findings from the 2026 best-practices research and the `pa-ai-agent-skills`
analysis. These are relays, not plugin work: each exists in the industry (or
already inside Picsart's agent platform) and is missing from the MCP surfaces
the plugin can call.

1. **Model-level identity lock** (LoRA / PuLID-class character fine-tune).
   The studio-grade consistency tier: encode a lead's face into weights once,
   hold it across every generation. Prompt+reference discipline (what the
   pipeline does today) is the ceiling without it.

2. **Video upscale / frame interpolation on the gen-ai surface.** The
   practitioner post chain is upscale → grain → grade. **Done 2026-09-09**:
   video upscalers are on the gen-ai surface; `film-finishing` picks one from
   the live catalog at the time of the pass and names none, because the tools
   change.

3b. **Word-level speech alignment on the film plugin's own surface.** The rough
    cut (default-on, `film-edit`) times every dialogue shot from its line's word
    boundaries — first word, last word, where the tail silence really starts —
    and that is also how "the line reads" gets checked against the script. The
    capability exists twice on the platform and neither is reachable here:
    `video-transcribe-audio` ships only with the **classic** Picsart API plugin,
    and `pcp/v1/whisperx-stt` (word segments, forced alignment — what
    `video2clip` uses) is an agent-platform workflow with no MCP surface. Note
    the ask is narrower than transcription: **we already know the words.** A
    forced-alignment call taking audio plus the expected text would be smaller,
    cheaper and more accurate than open transcription, and it is the shape this
    pipeline actually wants. Without either, every shot's timing falls back to
    its planned beat clock — a real rough cut, but blind to what the take did.
    **Parked with the automatic rough cut (2026-09-04)**: a run arrives cut by
    the model, so no trim is derived any more. Still wanted for "the line
    reads" — checking a take's dialogue against the script.

3. **Scene-detect + audio beat analysis as MCP tools.** Both already run on
   the agent platform (`media-platform/v1/videos/scene-detect` — PySceneDetect
   with cut lists/EDL; BPM/beat/key analysis). On the MCP surface they would
   let the film flow detect unintended model-inserted cuts inside takes and
   cut trailers on beats. **Since 2026-09-03 this is load-bearing rather than
   nice-to-have**: scenes now generate as one multi-shot take (a *run*,
   `film-scenes`) and the delivered file has to be split back into shots where
   the model actually cut — near the declared times, not on them. Today that is
   contact-sheet frames around each planned time plus a look; a cut list from
   scene-detect would make the split exact, free of charged frames, and would
   catch a wrong cut count before anyone watches.

4. **Subject-tracked reframe** (`autoframe_reframe`-class): measured
   per-moment subject centers → a crop track. The finishing stage promises
   9:16/1:1 promo reframes and currently judges crops manually per shot.

5. **Video-LLM "watch"** (`deep_analyze_media`-class): one-pass "what
   happens, who is on screen, where the beats are" over a clip — replaces
   several contact-sheet + vision round trips in take QC and join QC.

6. **Durable upload grant for the MCP service principal** (tracked from the
   Drive durability fix): `type=web`-class uploads currently reject the
   service token, so Drive saves of ephemeral URLs fall back to honest
   `durability: "expiring"` warnings.

6b. **A horizontal camera angle on the shot card (server repo, small).** The
    card records shot size and camera height but not which *side* of the subject
    the camera is on — so the 30° rule (two shots of one subject need ≥30° of
    camera movement between them, or the cut reads as a glitch) cannot be
    checked, because there is no angle to compare. The size-step form is what we
    enforce instead (`film-scenes/references/SEAMS.md`) and it catches the ugly
    cases, but it is a proxy: a camera that stays put while the lens changes
    passes the size check and still stutters. What it needs is a per-shot
    horizontal angle — clock positions around the subject would match the
    vocabulary the lighting presets already use — and the shotlist board is
    where it belongs, beside size and height.

7. **Widget follow-up (server repo, small):** the Scene Review board's
   thumbnails and expanded player assume video; still images (the animatic
   boards from `film-scenes` step 3) should render via inferred media type.

8. **Voice registration — the second half of a flow whose first half already
   ships.** The one missing link
   in the whole voice chain, and the reason a *designed* voice cannot be kept in
   sync. The sync-preserving repair is speech-to-speech on a take's own dialogue
   (`eleven-sts-v2`: ±3ms over 5.3s, envelope correlation 0.852 at zero lag), and
   both sts models take `audioUrl` + **`voiceId`**. A described voice's identity
   is a sample file, and nothing on the surface converts one into an id —
   ElevenLabs' own instant/professional voice cloning does exactly this. With it,
   every designed voice becomes swappable onto finished takes and the dub route
   disappears. Without it, describing a voice forecloses lip-sync, which is a
   trap in a feature we ship.

   **The gap is narrower than "add voice cloning".** Checked 2026-08-27 across
   all 23 audio models and the classic REST specs (a search for "voice" in the
   OpenAPI specs returns nothing — the classic surface has no voice management at
   all). What IS exposed: `eleven-voice-create` — *"generate voice previews from
   a description to audition before committing"* — whose schema is a bare
   `prompt` and whose output is audio, plus `eleven-voice-design-v3`/`-v2`, same
   shape. Upstream, that previews call returns generated-voice ids and a second
   call turns the chosen preview into a permanent voice. **We surface the
   audition and not the commit** — the model's own description says "before
   committing" and there is nothing to commit to. Wiring that second call (and
   returning the id on the passport) is the whole ask; `/v1/voices/add`-class
   registration from an arbitrary sample would cover the rest.

9. **Speaker separation (diarization) on an audio track.** `eleven-audio-isolation`
   splits voice from background, not speaker A from speaker B. A two-hander shot
   comes back as one mixed track, so per-character voice work on it is only
   possible where the lines do not overlap (cut by transcript timings first).
   Overlapping dialogue currently cannot be re-voiced at all.

10. **A lip-sync pass: video + new audio → the mouth re-animated.** The single
    tool that would make the voice problem disappear, and the standard 2026
    practitioner answer to it (LatentSync, MuseTalk, Wav2Lip; commercially Sync).
    With it the flow is one rule — shoot normally, replace the dialogue with the
    approved voice, re-drive the mouth — and nothing has to be decided before the
    take. Without it there is no way to put an approved voice on a finished shot
    whose mouth is on screen, except regenerating that shot from the audio
    (`ltx-2.3-a2v`), which changes generators mid-scene.
    Checked the surface: no v2v model accepts an audio input. `ltx-v2.3-retake`
    advertises "replace audio, video, or both" but its exposed schema is
    `prompt` + `duration` + `videoUrl` only — no audio parameter — so the
    capability is either not wired through or not real. Worth confirming with the
    model owners before building on the description.

11. **Run grouping on the shotlist board (server repo, medium).** The unit of
    generation is the run — one multi-shot generation for as much of the film
    as fits the model's ceiling, scenes included (`film-scenes`, 2026-09-03) —
    and the board has no way to show it. The user's experience of the board is
    meant to stay exactly as it is — short cards, presets per shot — with the
    grouping visible on top. Three things it needs, in order of value:
    a **run marker** spanning the rows that share a generation (and a way to move
    the split), so the user can see what one re-roll costs; **`genSec` and the
    video commit per run** rather than per row — the board's per-shot `genSec`
    is now read as "what this shot would cost alone" and ignored inside a run;
    and **free-form `sec` inside a run** — the pills offer 3|5|7|8|10|12, but a
    shot inside a run keeps its exact cut length (a 5.3s line is a 6.3s shot,
    no ladder) because only the run's total snaps to the model. Until it lands,
    the grouping is said in chat when the list is approved and lives only in
    `film.json.scenes[].runs`.

12. **Let `picsart_save_asset` lock on sheet approval (server repo, small).**
    **Done 2026-09-05** on `film-ux-polish`: `lockBlockers` no longer asks for
    a battery — an absent `stressTest` blocks nothing, a recorded battery with
    failures still does, a waiver still clears it. The same change disabled
    `picsart_asset_sheet` (decorator swapped for a no-op, source kept) and
    rewrote the `picsart_save_asset` / `picsart_asset_review` descriptions to
    the one-sheet, approval-is-the-lock flow, because the old tool text was
    pulling the model back into per-angle stills and a stress test.

13. **Auto-emit one crop per declared state at save (server repo, medium).**
    A sheet is how identity is generated; a single-panel **view** is how it
    travels, and today the plugin cuts those views itself with
    `picsart_media_export` and files them in `referenceImages[]`. The ask is for
    `picsart_save_asset` to do it: given the sheet, its grid and the panel map
    it already receives, emit and store one crop per declared state, addressable
    as `@tag.state` (`@alina.adult_formal`), and return those URLs on the
    passport. Then a prompt references a state instead of a file the assistant
    had to remember to cut, the library holds a view per state for every project
    by construction, and the sc04a/sc04b failure — a panel pointer resolved
    differently on each generation — has no way back in. Two dependencies of
    ours worth stating: the crops must be **byte-stable** across reads (the same
    state must hand back the same file, not a re-render), and each must carry
    its box, so a wrong crop is diagnosable without re-deriving it.

14. **A server-side verify gate on `picsart_generate` (server repo, the only
    thing that makes this certain).** The plugin now has a prompt-verify step
    (`film-scenes/references/PROMPT-VERIFY.md`) and a deterministic checker for
    its mechanical half (`film-scenes/scripts/check_dispatch_intent.py`), and
    both are mandatory in the skill text. **Skill text cannot guarantee
    execution.** Every safeguard here is an instruction an assistant follows or
    forgets, and the failure this step exists to fix was committed by an
    assistant that could quote the rules correctly minutes beforehand. Only the
    server can refuse. The ask, smallest version first: `picsart_generate`
    accepts an optional `verify` object — `{ sha256, kind }` — and when the
    project's `film.json` carries `verify.required: true`, a call whose
    `sha256` does not match a fingerprint of the arguments actually being
    dispatched is **rejected before the charge**, with the verdict's own error
    text returned. The fingerprint is already computed and returned by the
    checker (`requestSha256`), so the plugin side exists; what is missing is
    something that says no. A film that opts in then cannot spend credits on an
    unverified prompt, in any client, by any assistant, including a future one
    that never read this file. Second, larger version: the checks themselves
    move server-side, so the rules are enforced for every user of the film
    surface rather than every user who happens to be running these skills.