Update to LemonSlice

{
  "description": "Add a LemonSlice video avatar to a LiveKit Agents app, or build one from scratch. LemonSlice turns any single image into a real-time, lip-synced talking avatar that joins the LiveKit room as a participant. Use when: (1) the user wants a talking avatar, a face on a voice agent, a lip-synced character, or a video agent on LiveKit — even if they do not name LemonSlice; (2) they mention lemonslice.AvatarSession, livekit-agents[lemonslice], @livekit/agents-plugin-lemonslice, or the lemonslice-examples repo; (3) they want the avatar to move and emote on its own or on cue (CWM-1, Character World Model, emotion engine, waving, smiling, laughing, leaning in), or to change its image or demeanor mid-call (update-image); (4) they are moving from another LiveKit avatar plugin (Tavus, HeyGen, Hedra, Beyond Presence, Anam, Synthesia); (5) the avatar never shows up, shows a black frame, drops after a fixed number of minutes, or the audio doubles. Works for Python and Node.js agents.\n",
  "included_files": [
    {
      "relative_path": "references/new-app.md",
      "size_in_bytes": 7217
    },
    {
      "relative_path": "references/reference.md",
      "size_in_bytes": 18489
    }
  ],
  "name": "lemonslice-livekit-app",
  "skill_md_contents": "---\nname: lemonslice-livekit-app\ndescription: |\n  Add a LemonSlice video avatar to a LiveKit Agents app, or build one from scratch. LemonSlice turns any single image into a real-time, lip-synced talking avatar that joins the LiveKit room as a participant. Use when: (1) the user wants a talking avatar, a face on a voice agent, a lip-synced character, or a video agent on LiveKit — even if they do not name LemonSlice; (2) they mention lemonslice.AvatarSession, livekit-agents[lemonslice], @livekit/agents-plugin-lemonslice, or the lemonslice-examples repo; (3) they want the avatar to move and emote on its own or on cue (CWM-1, Character World Model, emotion engine, waving, smiling, laughing, leaning in), or to change its image or demeanor mid-call (update-image); (4) they are moving from another LiveKit avatar plugin (Tavus, HeyGen, Hedra, Beyond Presence, Anam, Synthesia); (5) the avatar never shows up, shows a black frame, drops after a fixed number of minutes, or the audio doubles. Works for Python and Node.js agents.\nlicense: MIT\nmetadata:\n  author: lemonslice\n  version: \"1.0.0\"\n---\n\n# LemonSlice avatar on LiveKit\n\nLemonSlice is an avatar plugin for LiveKit Agents, in Python and Node.js. You give it one image (any face, any art style, no training step) and it joins the room as a second participant that publishes lip-synced video. Your `AgentSession` keeps doing STT, LLM, and TTS; the plugin reroutes the session's audio to LemonSlice, which streams synced audio and video back into the room.\n\nThis skill works out what the user already has, picks one route, and implements it.\n\n## Facts that shape every decision\n\n- **Keys.** `LEMONSLICE_API_KEY` from https://lemonslice.com/developers, plus `LIVEKIT_URL`, `LIVEKIT_API_KEY`, `LIVEKIT_API_SECRET` from a LiveKit Cloud project. The LemonSlice key needs an active subscription; a key without one gets a `401`/`402` from the session API. Nothing runs without these, so confirm them before writing code.\n- **Source image.** Exactly one of `agent_image_url` (public `https://` URL), `agent_image` (local file, uploaded by the plugin), or `agent_id` (an avatar saved in the LemonSlice dashboard). Default to `agent_image_url`; the dashboard is optional. Site paths like `/avatar.png` and `localhost` URLs fail because LemonSlice's servers fetch the image.\n- **Both languages.** Python: `livekit-agents[lemonslice]`. Node: `@livekit/agents-plugin-lemonslice`. The frontend is just a LiveKit client.\n- **Attach before start.** `await avatar.start(session, room)` must finish before `await session.start(...)`.\n- **The avatar owns the audio.** Keep `audio_output=False` in `RoomOptions` (Python) or the equivalent in Node, and do not give the session another audio output afterwards. Otherwise the user hears the reply twice, or lip-sync dies.\n- **Sessions time out.** Idle timeout defaults to 60 s and resets while the avatar talks (`idle_timeout`, `-1` disables). Hard cap is 30 minutes per session unless LemonSlice raises it.\n- **Mid-call control is a REST call, not a plugin method.** Image and demeanor prompts change via `POST /api/liveai/sessions/{session_id}/control`. `avatar.start()` returns the `session_id` you need.\n- **CWM-1 is the behavior model.** `model=\"cwm-1\"` (or `\"cwm-1-lite\"`) switches the avatar to Character World Model 1, whose emotion engine decides what the character does and feels: waving, smiling, laughing, leaning in, glancing away. It can run fully automatically (`action_engine=\"natural\"`) or take cues from your code. Under CWM-1, `agent_prompt` and `agent_idle_prompt` are ignored.\n- **Plan-gated.** CWM-1 and `model=\"flash\"` (fastest response) are on the Ultra and Enterprise tiers. Offer them; do not build on them until the user confirms their plan.\n\n## Step 1: Find the starting point\n\nScan the codebase before asking anything.\n\n| Look for | Where | Means |\n| --- | --- | --- |\n| `livekit-agents` / `livekit.agents` | `pyproject.toml`, imports | Python LiveKit agent → **Add** |\n| `@livekit/agents` | `package.json` | Node LiveKit agent → **Add** |\n| `livekit.plugins.lemonslice` / `agents-plugin-lemonslice` | imports | Already integrated → **Fix**, read `references/reference.md` |\n| `livekit.plugins.tavus` / `bey` / `hedra` / `heygen` / `anam` / `synthesia` | imports | Other avatar plugin → **Replace** |\n| `AgentSession(llm=<realtime model>)` | entrypoint | Speech-to-speech agent; avatar still attaches the same way |\n| `AgentSession(stt=…, llm=…, tts=…)` | entrypoint | Cascade agent |\n| `LEMONSLICE_API_KEY`, `LIVEKIT_*` | `.env*` | Which keys exist |\n| `@lemonsliceai/avatar`, `LiveKitRoom`, `useVoiceAssistant` | frontend | A LiveKit frontend exists; the avatar will render there |\n| `livekit-agents` version | lockfile | `~=1.8` for the current LiveKit Inference models and `TurnDetector` |\n\nThen ask only what is still unknown, as one list:\n\n```\nTo add the LemonSlice avatar I need to know:\n1. Do you have a LemonSlice API key (https://lemonslice.com/developers) and a LiveKit Cloud project?\n2. Is there an existing LiveKit agent (Python or Node), or are we starting fresh?\n3. Which image should the avatar use? A public https URL is easiest; I can use a placeholder until you have one.\n```\n\n## Step 2: Pick one route\n\n```\nPlugin already imported and something is wrong?    → FIX      (references/reference.md, troubleshooting)\nWorking LiveKit agent, Python or Node?              → ADD      (below)\nAgent uses another avatar plugin?                   → REPLACE  (below)\nNo agent at all?                                    → NEW APP  (references/new-app.md)\n```\n\nSay which route you picked and why in two sentences, then implement. Do not present a menu.\n\n## Step 3: Implement\n\n### ADD — existing agent\n\n1. Install.\n   - Python: `uv add \"livekit-agents[lemonslice]~=1.8\"` (or pip). Raise an older `livekit-agents` pin to `~=1.8` first.\n   - Node: `pnpm add @livekit/agents-plugin-lemonslice`\n2. Add `LEMONSLICE_API_KEY` to the agent's environment. It never goes to the frontend.\n3. Between building the `AgentSession` and starting it:\n\nPython:\n\n```python\nfrom livekit.plugins import lemonslice\n\navatar = lemonslice.AvatarSession(\n    agent_image_url=AGENT_IMAGE_URL,        # public https URL\n    agent_prompt=\"A friendly person talking.\",  # demeanor while speaking, optional\n)\nsession_id = await avatar.start(session, room=ctx.room)   # keep this for control calls\n\nawait session.start(\n    room=ctx.room,\n    agent=Assistant(),\n    room_options=room_io.RoomOptions(\n        audio_input=room_io.AudioInputOptions(\n            noise_cancellation=ai_coustics.audio_enhancement(\n                model=ai_coustics.EnhancerModel.QUAIL_VF_S\n            ),\n        ),\n        audio_output=False,   # the avatar publishes the audio\n    ),\n)\n```\n\nNode:\n\n```ts\nimport * as lemonslice from '@livekit/agents-plugin-lemonslice';\n\nconst avatar = new lemonslice.AvatarSession({\n  agentImageUrl: AGENT_IMAGE_URL,\n  agentPrompt: 'A friendly person talking.',\n});\nconst sessionId = await avatar.start(session, ctx.room);\n\nawait session.start({ agent, room: ctx.room, outputOptions: { audioEnabled: false } });\n```\n\n4. Wait for the avatar before the first line, so it does not speak into an empty room:\n\n```python\nawait utils.wait_for_agent(ctx.room)        # Python\nawait session.generate_reply(instructions=\"Greet the user briefly.\")\n```\n\n```ts\nawait waitForParticipant({ room: ctx.room, identity: 'lemonslice-avatar-agent' });  // Node\nawait session.generateReply({ instructions: 'Greet the user briefly.' });\n```\n\nNo frontend change is required to see the avatar: it is a normal LiveKit participant with a video track. For a good experience, though, do the frontend pattern in Step 4.\n\n### REPLACE — from another avatar plugin\n\nLiveKit's avatar plugins share the `AvatarSession` shape, so this is a small diff:\n\n| Theirs | Ours |\n| --- | --- |\n| `from livekit.plugins import tavus` (`bey`, `hedra`, `heygen`, `anam`, `synthesia`) | `from livekit.plugins import lemonslice` |\n| `tavus.AvatarSession(replica_id=…)`, `synthesia.AvatarSession(AvatarConfig(avatar_ids=[…]))`, etc. | `lemonslice.AvatarSession(agent_image_url=\"https://…\")` |\n| `TAVUS_API_KEY` / `SYNTHESIA_API_KEY` / … | `LEMONSLICE_API_KEY` |\n| Provider-specific avatar ids | Any public image URL |\n\nKeep their `await avatar.start(session, room=…)` call where it is. If they swapped avatars mid-call with a plugin method, port that to a `update-image` control call (Step 5). Remove the old plugin extra from dependencies.\n\n### NEW APP\n\nRead `references/new-app.md`. It covers the two starters (`03-livekit-app-python`, `04-livekit-app-nodejs`), the env file, the current LiveKit Inference cascade, dispatch, and the frontend.\n\n## Step 4: Frontend — ring, then show\n\nAvatar video takes a few seconds to arrive after the participant joins. Showing the video element on `ParticipantConnected` gives the user a black frame. Instead:\n\n1. Play a ringing state while connecting.\n2. Switch to the call UI when the first video frame renders. Use `@lemonsliceai/avatar`'s `LiveKitAvatarReadyWatcher` (React), or listen for `bot_ready` on the `lemonslice` data topic.\n3. Return to an idle state on `ParticipantDisconnected` for the avatar identity (default `lemonslice-avatar-agent`), so the user can call again.\n\n```tsx\nimport { LiveKitAvatarReadyWatcher } from \"@lemonsliceai/avatar/livekit-react\";\n\n<LiveKitRoom serverUrl={serverUrl} token={token} connect>\n  <LiveKitAvatarReadyWatcher onReady={() => setAvatarReady(true)} />\n  {avatarReady ? <CallUI /> : <RingingUI />}\n</LiveKitRoom>\n```\n\nMint tokens server-side. If the worker sets `agent_name`, the token's room config must dispatch it (`RoomAgentDispatch`) or the agent never joins; see `references/new-app.md`.\n\n## Step 5: Bring the character to life with CWM-1 (optional, Ultra/Enterprise)\n\nWithout CWM-1 the avatar already talks with natural hand gestures. CWM-1 adds a character that reacts: it smiles at good news, crosses its arms, glances away while thinking, waves goodbye. It works from the same single image, with no per-character tuning.\n\nPick one of three levels, from hands-off to fully scripted:\n\n**1. Automatic (start here).** LemonSlice picks contextually appropriate behavior from the conversation.\n\n```python\navatar = lemonslice.AvatarSession(\n    agent_image_url=AGENT_IMAGE_URL,\n    model=\"cwm-1\",              # or \"cwm-1-lite\"\n    action_engine=\"natural\",\n)\n```\n\nOn LiveKit, also publish the user's speaking state, so the character visibly listens instead of only switching between idle and responding:\n\n```python\n@session.on(\"user_state_changed\")\ndef _on_user_state(ev: UserStateChangedEvent) -> None:\n    state = {\"speaking\": \"speaking\", \"listening\": \"idle\"}.get(ev.new_state)\n    if state:\n        asyncio.create_task(ctx.room.local_participant.publish_data(\n            json.dumps({\"state\": state}).encode(),\n            reliable=True,\n            topic=\"ls.user-state\",\n            destination_identities=[\"lemonslice-avatar-agent\"],\n        ))\n```\n\n**2. Cued from your code, over the LiveKit data channel.** This is the lowest-latency way to trigger a specific action: ~250 ms faster than REST. Publish `{\"action\": \"<name>\"}` on topic `ls.action` to the avatar participant:\n\n```python\nasync def perform_action(action: str) -> None:\n    await ctx.room.local_participant.publish_data(\n        json.dumps({\"action\": action}).encode(),\n        reliable=True,\n        topic=\"ls.action\",\n        destination_identities=[\"lemonslice-avatar-agent\"],\n    )\n```\n\nThe browser can publish the same message (`room.localParticipant.publishData(…, { topic: \"ls.action\", destinationIdentities: [...] })`), for example from buttons.\n\n**3. Cued over REST**, for non-LiveKit transports or calls from a backend: `POST …/control` with `{\"event\": \"action\", \"action\": \"<name>\"}`.\n\nLevels 2 and 3 work with or without `action_engine`. A cued action pre-empts whatever the automatic engine is doing; the engine resumes when it finishes. One action plays at a time, once, for a fixed length. Send it again to hold it, or `clear_action` to stop it.\n\nTypical wiring: wrap `perform_action` in LLM `@function_tool`s so the model waves on hello and goodbye, or smiles at good news. Trigger `wave` after `wait_for_agent` at call start. The full action list (`wave`, `smile`, `laugh`, `excited`, `angry`, `arms_crossed`, `lean_slow`, `listen`, `explain_finger_up`, and others), plus which ones read best during silence, is in `references/reference.md`.\n\n## Step 6: Session control (optional)\n\nEverything below is one `POST` with the `session_id` from `avatar.start()`:\n\n```\nPOST https://lemonslice.com/api/liveai/sessions/{session_id}/control\nX-API-Key: $LEMONSLICE_API_KEY\n```\n\n| `event` | Body | What it does |\n| --- | --- | --- |\n| `update-image` | `image_url` or `image_base64` (< 900 KB decoded) | Swap the avatar's look or whole identity mid-call, ~600 ms. Any audio playing is cut, so trigger during silence. Completion arrives as `image_change_complete` / `image_change_error` on the `lemonslice` data topic. |\n| `action` | `action` | Trigger a CWM-1 action (see Step 5). On LiveKit, prefer the `ls.action` data message. |\n| `update-agent-prompt` | `agent_prompt` | Change demeanor while speaking (\"look excited\"). Ignored under CWM-1. |\n| `update-idle-prompt` | `idle_prompt` | Change demeanor while silent. Ignored under CWM-1. |\n| `reset-idle-timeout` | — | Keep the session alive while the user is engaged but nobody is talking. |\n| `terminate` | — | End the avatar without ending the room. |\n\nTypical wiring: expose `update-image` as an LLM `@function_tool`, so the model changes outfit or scene when the conversation calls for it. Full payloads are in `references/reference.md`. The `09-realtime-image-change` example in `lemonslice-examples` is the end-to-end reference for image changes.\n\n## Step 7: Verify\n\n1. Start the worker in `dev` mode (`uv run python agent.py dev` / `pnpm dev:agent`). The terminal `console` mode is a local mock room; the avatar will never appear there and no error is raised. Test in a real room: the app's own frontend, or LiveKit Cloud → Agents → Console with the worker's `agent_name`.\n2. Confirm the sequence: agent joins → avatar participant joins → `bot_ready` → first frame → the agent greets.\n3. Speak. Lip-sync should match the reply audio with no echo or double audio.\n4. Leave the call. The avatar participant should leave within the idle timeout; if the room keeps running, call `ctx.room.disconnect()` or send `terminate`.\n5. Check that `LEMONSLICE_API_KEY` and `LIVEKIT_API_SECRET` are server-side only.\n\n## Rules that apply on every route\n\n- Keys first. Do not clone, install, or run until the user confirms their `.env`.\n- Public image URL. Keep the starter's image until the user supplies one; never block on a dashboard avatar.\n- Match the voice to the face. Pick a TTS voice whose gender and age fit the image and confirm with the user. On LiveKit Inference, Fish Audio `9a9cf47702da476aa4629e2506d4a857` (Hannah) is a solid female default; browse LiveKit's TTS catalog for others.\n- If the conversation feels slow, measure the LLM first. Avatar video is rarely the bottleneck; `model=\"flash\"` shaves 200–300 ms if the user has access.\n- Use `agent_prompt` for mood, not choreography. For specific gestures and emotions, use CWM-1.\n\n## References\n\n- `references/new-app.md` — greenfield scaffold from the LemonSlice starters: env, install, cascade, dispatch, frontend, deploy.\n- `references/reference.md` — `AvatarSession` parameters (Python/Node), CWM-1 action list and channels, events on the `lemonslice` topic, control endpoint payloads, model variants, errors, and the symptom → fix table.\n- https://lemonslice.com/docs/livekit — plugin docs.\n- https://lemonslice.com/docs/reference/cwm-1 — CWM-1 emotion engine, with example videos of every action.\n- https://lemonslice.com/docs/reference/production-checklist — what LemonSlice has learned shipping avatars to production.\n- https://github.com/lemonsliceai/lemonslice-examples — starters `02` (agent only), `03` (Python full app), `04` (Node full app), `07` (Zoom/Meet/Teams bot), `09` (real-time image change).\n- `npx skills add livekit/agent-skills` — LiveKit's own skill for current agent APIs and model catalogs.\n"
}