Update to Pixeltable

{
  "description": "Build multimodal AI apps with Pixeltable. One application file (app.py) declares TableModel tables and FastAPIRouter routes. Create tables with pxt schema update. Start HTTP with pxt service update. Insert a row or POST to try the app. Use computed columns instead of LangChain, pandas-as-store, or a separate vector DB. Use when building RAG, processing images/video/audio/documents, or serving an API. Do NOT use for general Python or direct PostgreSQL administration.",
  "included_files": [
    {
      "relative_path": "agents/openai.yaml",
      "size_in_bytes": 203
    },
    {
      "relative_path": "references/anti-patterns.md",
      "size_in_bytes": 3134
    },
    {
      "relative_path": "references/cli.md",
      "size_in_bytes": 15005
    },
    {
      "relative_path": "references/core-api.md",
      "size_in_bytes": 16321
    },
    {
      "relative_path": "references/providers.md",
      "size_in_bytes": 4709
    },
    {
      "relative_path": "references/workflows.md",
      "size_in_bytes": 3383
    }
  ],
  "name": "pixeltable",
  "skill_md_contents": "---\nname: pixeltable\ndescription: Build multimodal AI apps with Pixeltable. One application file (app.py) declares TableModel tables and FastAPIRouter routes. Create tables with pxt schema update. Start HTTP with pxt service update. Insert a row or POST to try the app. Use computed columns instead of LangChain, pandas-as-store, or a separate vector DB. Use when building RAG, processing images/video/audio/documents, or serving an API. Do NOT use for general Python or direct PostgreSQL administration.\nlicense: Apache-2.0\nallowed-tools: []\n---\n\n## STOP\n\nIf you find yourself importing any of these, you are off-path:\n\n1. **Do not use LangChain / LlamaIndex / Haystack / LangGraph.** Chunking is `document_splitter`. Search is `.similarity()`. Tools are `pxt.tools()` + `invoke_tools()`.\n2. **Do not use pandas as a working store.** Tables are the store. `.collect().to_pandas()` is export only.\n3. **Do not write `for row in ...:` loops calling models.** Wrap the call in a computed column.\n4. **Do not install a separate vector database.** In an app, `__indexes__ = [pxt.EmbeddingIndex(...)]` on the model. In a notebook, `t.add_embedding_index(col, embedding=fn)`. Search with `.similarity(string=query)`.\n5. **Do not write `while not done:` agent loops.** Insert a row. The computed-column chain runs.\n\nSee [anti-patterns.md](references/anti-patterns.md) (6 macros).\n\n## What is Pixeltable?\n\nOne application file (`app.py`) is the backend.\n\n- `pxt schema update`: creates tables from `TableModel` classes. Does not start HTTP.\n- Insert a sample, `.select()`, `pxt dashboard`, or `pxt schema diff`. Compute runs on insert. After `pxt service update`, curl POST.\n- `pxt service update`: starts HTTP (local or `pxt://`). `pxt service list` prints the URL. This is the serving command; do not reach for `pxt service run`.\n\n`pxt db update` sets hosted image, secrets, and workers. It does not insert rows and does not start app HTTP.\n\nFirst run: [Quickstart](https://docs.pixeltable.com/overview/quick-start). Why: [Why Pixeltable](https://docs.pixeltable.com/overview/pixeltable).\n\n## Starting a new project\n\n```bash\npip install 'pixeltable[serve]'   # Python 3.11+\npxt init\npxt service example --out app.py\npxt schema check app.py           # validates the file; warns if 'app' is shadowed\npxt schema update app.py my_app\npxt service update app.py my_app\npxt service list                  # assigned port; do not hard-code :8000\n```\n\n`pxt service example` writes models plus a `FastAPIRouter`. Schema only (no HTTP): `pxt schema example --brief --out app.py`. Then edit `app.py` and run `pxt schema update` again. After a schema change, run `pxt service update` again if routes exist. Do not `python app.py`. Full flags: [cli.md](references/cli.md).\n\nThe last argument (`my_app`, or `pxt://org:db` on Cloud) is a catalog directory, not a folder on disk. `pxt init` marks the project root. Schema does not start HTTP. Service does not create tables. Non-interactive: `pxt service update ... -f`. Local handle: `pxt.get_table('my_app.docs')`.\n\nSame file on Cloud: set `PIXELTABLE_API_KEY`, add `[[pixeltable.database]]` with `name = 'pxt://org:db'`, then `pxt db update pxt://org:db -f`, then `pxt schema update app.py pxt://org:db -f`, then `pxt service update app.py pxt://org:db -f`. Cloud handle: `pxt.get_table('pxt://org:db/docs')`. Cloud databases store media in their managed home bucket by default; set a column `destination=` only to override it. On Cloud, try the app with dashboard insert plus `pxt schema diff`, and inspect failures with `pxt service logs` / `pxt db logs`. [Cloud](https://docs.pixeltable.com/howto/deployment/cloud).\n\n## The application file\n\n`pxt service example --out app.py` writes this shape. Edit it. Then `pxt schema update app.py my_app`.\n\n```python\nimport pixeltable as pxt\nimport pixeltable.functions as pxtf\nfrom pixeltable.serving import FastAPIRouter\n\nTableModel = pxt.model_base()\n\n\n@pxt.udf\ndef excerpt(text: str, n: int = 12) -> str:\n    return text if len(text) <= n else f'{text[:n]}...'\n\n\nclass Docs(TableModel, name='docs'):\n    id = pxt.Column(value=pxtf.uuid.uuid7(), primary_key=True)\n    title: pxt.String\n    body: pxt.String | None\n    title_upper = pxtf.string.upper(title)\n    summary = excerpt(title)\n\n\ningest = FastAPIRouter(name='ingest')\ningest.add_insert_route(\n    Docs, path='/docs', inputs=[Docs.title, Docs.body],\n    outputs=[Docs.id, Docs.title_upper, Docs.summary],\n)\ningest.add_update_route(\n    Docs, path='/docs/update', inputs=[Docs.title],\n    outputs=[Docs.id, Docs.title_upper],\n)\ningest.add_compute_route(Docs, path='/titles', inputs=[Docs.title], outputs=[Docs.title_upper])\n```\n\nAnnotation is a stored column. Assignment is a computed column. Optional is `T | None`. Primary key is `pxt.Column(..., primary_key=True)`. Indexes on the model: `__indexes__ = [pxt.EmbeddingIndex(...)]`. `from pixeltable.serving import FastAPIRouter`.\n\nAlready have FastAPI: after schema update, `ingest.bind('my_app')` then `app.include_router(ingest)`. Call `pxt.get_table()` inside custom handlers. [workflows.md](references/workflows.md).\n\nRAG, views, and search: [workflows.md](references/workflows.md). Do not add Hugging Face or spaCy unless the user asked.\n\n## Apps vs notebooks\n\n- **Apps:** `app.py` + `pxt schema update` + `pxt service update`. Indexes on the model.\n- **Notebooks / REPL:** `pxt.create_table()`, `add_computed_column()`, `add_embedding_index()`. The appendix below uses that form.\n\n## Where to look\n\n| Need | Open |\n|------|------|\n| `pxt schema`, `pxt service`, inspect | [cli.md](references/cli.md) |\n| Types, views, UDFs, UDAs | [core-api.md](references/core-api.md) |\n| Provider import and output shape | [providers.md](references/providers.md) |\n| Serving, FastAPIRouter, routes | [workflows.md](references/workflows.md) |\n| Wrong stack | [anti-patterns.md](references/anti-patterns.md) |\n\nAdd video, audio, agents, or a UI by editing `app.py`. A view is either a filter (`base=Docs.where(...)`) or an iterator (`frame_iterator`, `audio_splitter`, `document_splitter`, `video_splitter`, `string_splitter`, `list_iterator`, `tile_iterator`). Check `pixeltable.functions` before writing a UDF. Start from `pxt service example` or `pxt schema example`. Do not invent a second `pxt schema update` path.\n\n## API traps\n\n| Wrong | Correct |\n|-------|---------|\n| `openai.vision(...)` | Deprecated (the only deprecated function in `pixeltable.functions`). Use `chat_completions` with `image_url`, or `responses` |\n| `from pixeltable.iterators import ...` | The whole `pixeltable.iterators` package is a deprecated shim (`FrameIterator`, `VideoSplitter`, `DocumentSplitter`, `StringSplitter`, `AudioSplitter`, `TileIterator`). Import the function from `pixeltable.functions.*` -- e.g. `from pixeltable.functions.video import frame_iterator` |\n| `similarity(query)` | `similarity(string=query)`. Also `image=` / `audio=` / `video=` / `document=` / `vector=`; `idx=` picks among several indexes on one column |\n| Re-run with `if_exists='ignore'` to fix logic | Notebook: `add_computed_column(..., if_exists='replace')`. App: **rename** the column, then `pxt schema update --allow-destructive` |\n| Edit a computed column's expression in place, then `--allow-destructive` | Editing an existing column's expression is `UNSUPPORTED`; the flag does not help and the whole update applies nothing. Rename the column |\n| `t.summary_errortype` | `t.summary.errortype` / `t.summary.errormsg`, on stored computed or media columns. `t.<col>.fileurl` / `.localpath` for media |\n| `pxt.Required[pxt.String]` | Non-nullable by default. Optional: `T \\| None` |\n| `@pxt.udf def f(x: str)` fed a nullable column | A non-nullable parameter that receives `None` **skips the call**: the cell is `None` and `errormsg` is empty. Annotate `x: str \\| None` and handle `None` in the body |\n| `whisper.load_model(...)` inside a UDF body | Weights reload on every row. Use the shipped wrapper (`pxtf.whisper.transcribe`, `clip.using(...)`), or a module-scope cached loader |\n| `recompute_columns(columns=['summary'])` | `t.recompute_columns('summary', errors_only=True)` |\n| TOML routes or a retired serve CLI | `FastAPIRouter` + `pxt schema update` + `pxt service update` |\n| `add_embedding_index()` in `app.py` | `__indexes__` on the TableModel. Note the DSL names an index `name=`, the SDK `idx_name=` |\n| `make_video(order_by=...)` / `stitch_tiles(order_by=...)` | Both are `requires_order_by` UDAs: the ordering expression is the **first positional** argument -- `make_video(t.pos, t.frame, fps=25)`. `order_by=` raises |\n| `pxt.create_table()` / `get_table()` at import in `app.py` | `TableModel` + `pxt schema update`. Import must not mutate the catalog |\n| `EmbeddingIndex(frame, image_embed=clip)` | `embedding=clip` (covers text and image). Or both `string_embed=` and `image_embed=`. `image_embed=` alone cannot answer `similarity(string=...)` |\n| `uuid.astype(pxt.String)` | `uuid.to_string()` (`from pixeltable.functions.uuid import to_string`). `astype` is not UUID→String |\n\nExtract the field (`.text`, `.choices[0].message.content`). Cast Json with `.astype(pxt.String)` only before embedding or concatenating.\n\n## Notebook / REPL appendix\n\n```python\nimport pixeltable as pxt\n\npxt.create_dir('my_project', if_exists='ignore')\nt = pxt.create_table('my_project.documents', {\n    'title': pxt.String,\n    'content': pxt.String,\n    'image': pxt.Image,\n    'video': pxt.Video,\n    'audio': pxt.Audio,\n    'doc': pxt.Document,\n}, if_exists='ignore')\n```\n\nTypes are non-nullable by default. Optional is `T | None`. Do not use `pxt.Required`.\n\n```python\nfrom pixeltable.functions.uuid import uuid7\n\nt = pxt.create_table('my_project.items', {\n    'content': pxt.String,\n    'uuid': uuid7(),\n}, primary_key=['uuid'], if_exists='ignore')\n```\n\nInsert: `t.insert([{...}])`. Computed column:\n\n```python\nfrom pixeltable.functions.openai import chat_completions\n\nt.add_computed_column(\n    summary=chat_completions(\n        messages=[{'role': 'user', 'content': t.content}],\n        model='gpt-4o-mini',\n    ).choices[0].message.content,\n    if_exists='ignore',\n)\n```\n\nViews: `document_splitter`, `frame_iterator` (from `pixeltable.functions.video`), `string_splitter`, `audio_splitter`. Notebook indexes: `t.add_embedding_index('content', embedding=embed_fn, if_exists='ignore')`.\n\nQuery: `t.where(...).select(...).collect()`. Similarity: `t.content.similarity(string=query)`. In `@pxt.query`, alias as `score=sim`.\n\nUDFs are recorded as a module path relative to the project root (`app.excerpt`).\n\nAlways `if_exists='ignore'` on notebook `create_*` / `add_*`. Failed cells: `t.recompute_columns('summary', errors_only=True)`. `string_splitter` / `document_splitter(..., separators='sentence')` need spaCy. Embedding indexes need `.using(...)`.\n\n## pxt CLI\n\n```bash\npxt init\npxt service example --out app.py\npxt schema check app.py\npxt schema update app.py my_app\npxt service update app.py my_app\npxt service list\npxt ls -l\npxt errors my_app/docs\npxt dashboard\n```\n\n[cli.md](references/cli.md).\n\n## Resources\n\n- [Quickstart](https://docs.pixeltable.com/overview/quick-start)\n- [CLI](https://docs.pixeltable.com/platform/cli)\n- [MCP Server](https://github.com/pixeltable/mcp-server-pixeltable-developer)\n- [Docs](https://docs.pixeltable.com/llms-full.txt)\n"
}