Update to Flyte

{
  "description": "Builds and serves Flyte 2 apps — FastAPI, Streamlit, vLLM, SGLang, WebSocket, and browser apps. Use when the user wants to serve a model, create a REST API, build a dashboard, deploy an LLM backend, or create a web app with Flyte. Trigger words: \"app\", \"serve\", \"deploy app\", \"FastAPI\", \"Streamlit\", \"vLLM\", \"SGLang\", \"REST API\", \"dashboard\", \"serving\", \"endpoint\", \"webhook\", \"WebSocket\".",
  "included_files": [],
  "name": "flyte-sdk-app",
  "skill_md_contents": "---\nname: flyte-sdk-app\ndescription: 'Builds and serves Flyte 2 apps — FastAPI, Streamlit, vLLM, SGLang, WebSocket, and browser apps. Use when the user wants to serve a model, create a REST API, build a dashboard, deploy an LLM backend, or create a web app with Flyte. Trigger words: \"app\", \"serve\", \"deploy app\", \"FastAPI\", \"Streamlit\", \"vLLM\", \"SGLang\", \"REST API\", \"dashboard\", \"serving\", \"endpoint\", \"webhook\", \"WebSocket\".'\n---\n\n# Flyte 2 SDK App Skill\n\nBuild and serve applications with Flyte 2.\n\n## Grounding References\n\n| Resource | URL |\n|---|---|\n| Official docs | https://www.union.ai/docs/v2/flyte |\n| Docs index (LLMs) | https://www.union.ai/docs/v2/flyte/llms.txt |\n| SDK API reference | https://www.union.ai/docs/v2/union/api-reference/flyte-sdk/ |\n| CLI API reference | https://www.union.ai/docs/v2/union/api-reference/flyte-cli/ |\n| flyte-sdk source | https://github.com/flyteorg/flyte-sdk |\n| Example code | https://github.com/unionai/unionai-examples |\n| Flyte MCP tools | Available via the `flyte-cluster` and `flyte-docs` MCP servers |\n\n## App Types\n\n| App Type | Use Case | Import |\n|---|---|---|\n| `FastAPIAppEnvironment` | REST APIs, model serving | `from flyte.app.extras import FastAPIAppEnvironment` |\n| `StreamlitAppEnvironment` | Dashboards, data apps | `from flyte.app.extras import StreamlitAppEnvironment` |\n| `vLLMAppEnvironment` | LLM serving | `from flyte.app.extras import vLLMAppEnvironment` |\n| `SGLangAppEnvironment` | Structured generation | `from flyte.app.extras import SGLangAppEnvironment` |\n| Custom (`AppEnvironment`) | Any HTTP server | `import flyte` |\n\n## FastAPI App — Model Serving\n\n### Basic FastAPI app\n\n```python\nfrom fastapi import FastAPI\nimport flyte\nfrom flyte.app.extras import FastAPIAppEnvironment\n\napp = FastAPI()\nenv = FastAPIAppEnvironment(\n    name=\"my-model\",\n    app=app,\n    image=flyte.Image.from_debian_base(python_version=(3, 12)).with_pip_packages(\n        \"fastapi\", \"uvicorn\", \"torch\",\n    ),\n)\n\n@app.get(\"/predict\")\nasync def predict(x: float) -> dict:\n    return {\"result\": x * 2 + 5}\n\nif __name__ == \"__main__\":\n    flyte.init_from_config()\n    flyte.serve(env)\n```\n\n### Model serving with loading\n\n```python\nfrom fastapi import FastAPI\nimport flyte\nfrom flyte.app.extras import FastAPIAppEnvironment\n\napp = FastAPI()\nenv = FastAPIAppEnvironment(\n    name=\"text-classifier\",\n    app=app,\n    image=flyte.Image.from_debian_base(python_version=(3, 12)).with_pip_packages(\n        \"fastapi\", \"uvicorn\", \"torch\", \"transformers\",\n    ),\n)\n\nmodel = None  # Loaded once at startup\n\n@app.on_event(\"startup\")\nasync def load_model():\n    global model\n    model = transformers.AutoModelForSequenceClassification.from_pretrained(\"bert-base\")\n\n@app.get(\"/predict\")\nasync def predict(text: str) -> dict:\n    assert model is not None\n    outputs = model(transformers.encode(text))\n    return {\"prediction\": outputs.argmax().item(), \"confidence\": outputs.softmax().max().item()}\n\nif __name__ == \"__main__\":\n    flyte.init_from_config()\n    flyte.serve(env)\n```\n\n### Multi-file FastAPI app\n\n```\napp/\n  __init__.py\n  main.py        # FastAPI app entry\n  routes/\n    __init__.py\n    predict.py\n    health.py\n  models/\n    __init__.py\n    classifier.py\n```\n\n```python\n# app/main.py\nfrom fastapi import FastAPI\nfrom .routes import predict, health\n\napp = FastAPI()\napp.include_router(predict.router, prefix=\"/api\")\napp.include_router(health.router, prefix=\"/health\")\n```\n\n## Streamlit App — Data Dashboards\n\n### Basic Streamlit app\n\n```python\nimport streamlit as st\nimport flyte\nfrom flyte.app.extras import StreamlitAppEnvironment\n\nst.title(\"Data Dashboard\")\n\ndf = st.dataframe(load_data())\n\nif st.button(\"Refresh\"):\n    st.rerun()\n\nenv = StreamlitAppEnvironment(\n    name=\"dashboard\",\n    script=\"app.py\",\n    image=flyte.Image.from_debian_base(python_version=(3, 12)).with_pip_packages(\n        \"streamlit\", \"pandas\", \"matplotlib\",\n    ),\n)\n\nif __name__ == \"__main__\":\n    flyte.init_from_config()\n    flyte.serve(env)\n```\n\n### Streamlit with upstream app dependency\n\n```python\nimport streamlit as st\nimport requests\nimport flyte\nfrom flyte.app.extras import StreamlitAppEnvironment\n\n# Access upstream app endpoint\nMODEL_ENDPOINT = flyte.app.AppEndpoint(app_name=\"model-serving\")\n\nst.title(\"Model Results\")\n\ntext = st.text_input(\"Enter text:\")\nif text:\n    response = requests.post(\n        f\"{MODEL_ENDPOINT.url}/predict\",\n        json={\"text\": text},\n    )\n    st.json(response.json())\n\nenv = StreamlitAppEnvironment(\n    name=\"results-dashboard\",\n    script=\"app.py\",\n    depends_on=[MODEL_ENDPOINT],\n)\n```\n\n## vLLM App — LLM Serving\n\n### Basic vLLM app\n\n```python\nimport flyte\nfrom flyte.app.extras import vLLMAppEnvironment\n\nenv = vLLMAppEnvironment(\n    name=\"llm-serving\",\n    model=\"meta-llama/Llama-3-8b-Instruct\",\n    image=flyte.Image.from_base(\"vllm/vllm-openai:latest\"),\n    resources=flyte.Resources(\n        cpu=\"8\",\n        memory=\"32Gi\",\n        gpu=\"1\",\n        gpu_model=\"nvidia-a10g\",\n    ),\n)\n\nif __name__ == \"__main__\":\n    flyte.init_from_config()\n    flyte.serve(env)\n```\n\n### vLLM with model prefetch\n\n```python\nenv = vLLMAppEnvironment(\n    name=\"llm-serving\",\n    model=\"meta-llama/Llama-3-8b-Instruct\",\n    prefetch=True,  # prefetch model weights at deploy time\n    image=flyte.Image.from_base(\"vllm/vllm-openai:latest\"),\n    resources=flyte.Resources(\n        cpu=\"8\",\n        memory=\"32Gi\",\n        gpu=\"1\",\n        gpu_model=\"nvidia-a10g\",\n    ),\n)\n```\n\n### vLLM multi-GPU\n\n```python\nenv = vLLMAppEnvironment(\n    name=\"llm-serving\",\n    model=\"meta-llama/Llama-3-70b-Instruct\",\n    tensor_parallel_size=4,  # shard across 4 GPUs\n    prefetch=True,\n    image=flyte.Image.from_base(\"vllm/vllm-openai:latest\"),\n    resources=flyte.Resources(\n        cpu=\"16\",\n        memory=\"128Gi\",\n        gpu=\"4\",\n        gpu_model=\"nvidia-a100\",\n    ),\n)\n```\n\n## SGLang App — Structured Generation\n\n### Basic SGLang app\n\n```python\nimport flyte\nfrom flyte.app.extras import SGLangAppEnvironment\n\nenv = SGLangEnvironment(\n    name=\"structured-gen\",\n    model=\"meta-llama/Llama-3-8b-Instruct\",\n    prefetch=True,\n    image=flyte.Image.from_base(\"sgl-project/sglang:latest\"),\n    resources=flyte.Resources(\n        cpu=\"4\",\n        memory=\"16Gi\",\n        gpu=\"1\",\n        gpu_model=\"nvidia-a10g\",\n    ),\n)\n\nif __name__ == \"__main__\":\n    flyte.init_from_config()\n    flyte.serve(env)\n```\n\n## WebSocket Apps\n\n```python\nimport asyncio\nimport flyte\nfrom flyte.app.extras import FastAPIAppEnvironment\nfrom fastapi import FastAPI, WebSocket\n\napp = FastAPI()\nenv = FastAPIAppEnvironment(\n    name=\"websocket-app\",\n    app=app,\n    image=flyte.Image.from_debian_base(python_version=(3, 12)).with_pip_packages(\n        \"fastapi\", \"uvicorn\", \"websockets\",\n    ),\n)\n\n@app.websocket(\"/ws\")\nasync def websocket_endpoint(websocket: WebSocket):\n    await websocket.accept()\n    try:\n        while True:\n            data = await websocket.receive_text()\n            result = process(data)\n            await websocket.send_text(result)\n    except WebSocketDisconnect:\n        pass\n\nif __name__ == \"__main__\":\n    flyte.init_from_config()\n    flyte.serve(env)\n```\n\n## Serving vs Deploying\n\n### Serve (ephemeral, for development)\n\n```bash\n# Serve an app locally\nflyte serve app.py env\n```\n\n```python\n# Serve programmatically\nresult = flyte.serve(env)\nprint(f\"App URL: {result.url}\")\n```\n\n### Deploy (persistent, for production)\n\n```bash\n# Deploy an app\nflyte deploy app.py env\n```\n\n```python\n# Deploy programmatically\nresult = flyte.deploy(env)\nprint(f\"App URL: {result.url}\")\n```\n\n### Activating and deactivating apps\n\n```bash\n# Activate a deployed app\nflyte update app <app_name> --activate --project flytesnacks --domain development\n\n# Deactivate\nflyte update app <app_name> --deactivate --project flytesnacks --domain development\n\n# Check status\nflyte get app <app_name> --project flytesnacks --domain development\n```\n\n### Using Flyte MCP for app management\n\nGetting an app's status, activating it, and deactivating it are all available as MCP\ntools, each taking the app name.\n\n\n## App Parameters\n\n### Passing parameters into apps\n\n```python\nenv = FastAPIAppEnvironment(\n    name=\"model-serving\",\n    app=app,\n    parameters={\n        \"model_name\": flyte.app.Parameter(name=\"model_name\", mount=\"/models/model.safetensors\"),\n        \"api_key\": flyte.app.Parameter(name=\"api_key\", env_var=\"API_KEY\"),\n    },\n)\n```\n\n### Overriding parameters at serve time\n\n```bash\nflyte serve app.py env --parameter model_name=/custom/path\n```\n\n## App Autoscaling\n\n### Auto-scaling apps\n\n```python\nfrom datetime import timedelta\n\nenv = FastAPIAppEnvironment(\n    name=\"auto-scaling-app\",\n    app=app,\n    scaling=flyte.app.Scaling(\n        replicas=(1, 10),  # autoscale between (min, max) replicas\n        scaledown_after=timedelta(minutes=10),\n    ),\n)\n```\n\n## App Dependencies (Serving Graphs)\n\n### Deploying multiple apps together\n\n```python\nmodel_env = FastAPIAppEnvironment(\n    name=\"model-serving\",\n    app=model_app,\n    image=model_image,\n)\n\ndashboard_env = StreamlitAppEnvironment(\n    name=\"results-dashboard\",\n    script=\"dashboard.py\",\n    depends_on=[model_env],  # upstream dependency\n    image=dashboard_image,\n)\n\n# Deploy both together\nflyte.deploy(model_env)\nflyte.deploy(dashboard_env)\n\n# Access upstream endpoint\nmodel_url = model_env.endpoint.url\n```\n\n### GPU/CPU split serving graph\n\n```python\n# GPU app: model inference\ngpu_env = FastAPIAppEnvironment(\n    name=\"model-gpu\",\n    app=gpu_app,\n    image=flyte.Image.from_base(\"nvidia/cuda:12.1-py3\").with_pip_packages(\n        \"torch\", \"fastapi\", \"uvicorn\",\n    ),\n    resources=flyte.Resources(\n        cpu=\"4\", memory=\"16Gi\", gpu=\"1\", gpu_model=\"nvidia-a10g\",\n    ),\n)\n\n# CPU app: pre/post processing\ncpu_env = FastAPIAppEnvironment(\n    name=\"preprocess-cpu\",\n    app=cpu_app,\n    image=flyte.Image.from_debian_base(python_version=(3, 12)).with_pip_packages(\n        \"fastapi\", \"uvicorn\", \"pillow\", \"numpy\",\n    ),\n    depends_on=[gpu_env],\n    resources=flyte.Resources(cpu=\"2\", memory=\"4Gi\"),\n)\n```\n\n## Webhook Apps\n\n### Basic webhook\n\n```python\nimport flyte\nfrom flyte.app.extras import FastAPIAppEnvironment\nfrom fastapi import FastAPI, Request\n\napp = FastAPI()\nenv = FastAPIAppEnvironment(\n    name=\"webhook-receiver\",\n    app=app,\n    image=flyte.Image.from_debian_base(python_version=(3, 12)).with_pip_packages(\n        \"fastapi\", \"uvicorn\",\n    ),\n)\n\n@app.post(\"/webhook\")\nasync def webhook(request: Request):\n    payload = await request.json()\n    # Trigger a Flyte workflow\n    flyte.run(process_webhook, inputs={\"payload\": payload})\n    return {\"status\": \"received\"}\n\nif __name__ == \"__main__\":\n    flyte.init_from_config()\n    flyte.serve(env)\n```\n\n## App Secrets\n\n### Secret-based authentication\n\n```python\n# Create a secret (via CLI or SDK)\n# flyte create secret my-api-key --value \"sk-xxx\"\n\nenv = FastAPIAppEnvironment(\n    name=\"authenticated-app\",\n    app=app,\n    image=flyte.Image.from_debian_base(python_version=(3, 12)).with_pip_packages(\n        \"fastapi\", \"uvicorn\",\n    ),\n    secrets={\"api_key\": flyte.Secret(key=\"my-api-key\", group=\"default\")},\n)\n\n# Access secret inside the app\napi_key = os.environ[\"FLYTE_SECRET_MY_API_KEY\"]\n```\n\n## Anti-Patterns\n\n1. **Don't use `flyte.run()` inside apps** — use `flyte.serve()` for apps, `flyte.run()` for workflows.\n2. **Don't forget `flyte.init_from_config()`** — required before `flyte.serve()`.\n3. **Don't hardcode model paths** — use `flyte.app.AppEndpoint` for upstream app URLs.\n4. **Don't use Union-only features** — avoid `ReusePolicy` and other Union-specific APIs.\n5. **Don't serve GPU apps without GPU resources** — always specify `gpu` and `gpu_model` in resources.\n"
}