Update to NVIDIA BioNeMo Agent Toolkit

{
  "name": "msa-search-nim",
  "description": "Generate multiple sequence alignments (MSAs) for protein sequences using the ColabFold MSA-Search NIM. Use for homolog search, UniRef30/ColabFold env searches, A3M or FASTA alignments, paired MSA search for complexes, PDB70 structural templates, hosted NVIDIA API calls, or local Docker deployment. For local deployment, download the databases in parallel with aria2c and launch via NIM_MODEL_NAME (the recommended default fast path, ~14 min vs >80 min for the built-in downloader); a plain docker run uses the slow built-in downloader.",
  "included_files": [
    {
      "relative_path": "references/api.md",
      "size_in_bytes": 6437
    },
    {
      "relative_path": "references/examples.md",
      "size_in_bytes": 2025
    },
    {
      "relative_path": "references/parameters.md",
      "size_in_bytes": 1830
    },
    {
      "relative_path": "references/science.md",
      "size_in_bytes": 1164
    },
    {
      "relative_path": "references/validation.md",
      "size_in_bytes": 1074
    }
  ],
  "skill_md_contents": "---\nname: msa-search-nim\ndescription: >\n  Generate multiple sequence alignments (MSAs) for protein sequences using the ColabFold MSA-Search NIM. Use for homolog search, UniRef30/ColabFold env searches, A3M or FASTA alignments, paired MSA search for complexes, PDB70 structural templates, hosted NVIDIA API calls, or local Docker deployment. For local deployment, download the databases in parallel with aria2c and launch via NIM_MODEL_NAME (the recommended default fast path, ~14 min vs >80 min for the built-in downloader); a plain docker run uses the slow built-in downloader.\nlicense: Apache-2.0 AND CC-BY-4.0\ncompatibility: \"requests>=2.28\"\nallowed-tools: Bash, Read, Write, AskUserQuestion\n---\n\n# MSA-Search NIM\n\nGenerate protein MSAs with GPU-accelerated MMSeqs2. Use this `SKILL.md` for\nfirst-pass hosted/local usage; load supplemental files only when needed:\n\n- `references/api.md`: exact endpoints, schemas, Docker flags, response fields.\n- `references/science.md`: MSA purpose, pairing/templates, limits, handoffs.\n- `references/parameters.md`: database, pairing, depth, and template tuning.\n- `references/validation.md`: alignment, template, and artifact checks.\n- `references/examples.md`: compact hosted/local request patterns.\n\n## Choose Mode And Endpoint\n\nAsk only when context is unclear:\n\n> Hosted NVIDIA API or local Docker NIM?\n\n- Hosted standard MSA: `https://health.api.nvidia.com/v1/biology/colabfold/msa-search/predict`\n- Hosted paired MSA: `https://health.api.nvidia.com/v1/biology/colabfold/msa-search/paired/predict`\n- Local standard MSA: `http://localhost:8000/biology/colabfold/msa-search/predict`\n- Local paired MSA: `http://localhost:8000/biology/colabfold/msa-search/paired/predict`\n- Local templates: `http://localhost:8000/biology/colabfold/msa-search/structure-templates/predict`\n\nLocal inference paths do not include `/v1/`. Hosted requests use `Authorization: Bearer $NGC_API_KEY`. Supported local Docker\nstartup uses `NGC_API_KEY` (or `NVIDIA_API_KEY` via the preflight) for\nregistry login, entitlement checks, and first-run model downloads; pass it\ninto the container with `-e NGC_API_KEY`. Local inference requests use no\nauth header after readiness. Warm-cache key-free startup varies by\nimage/version and should not be assumed.\nThe hosted template path returned HTTP 404 in validation, so use local Docker\nfor template search unless the hosted docs/service changes.\n\n## Local Docker\n\n**Default local deployment = parallel download + `NIM_MODEL_NAME`.** The first recipe below\nis the one to use for real workflows. It downloads the database(s) with a range-parallel\ndownloader (aria2c) and starts the NIM against those files — ~14 min for UniRef30 vs >80 min\nfor the NIM's built-in downloader (measured, H100). Do **not** reach for the plain `docker run`\n(the \"Fallback\" subsection) unless you only want a `databases:pdb70` smoke test or you\ndeliberately want the NIM to manage its own blob cache.\n\nLocal setup requires a GPU. Size the NVMe volume to the profile you pick (UniRef30 ~490 GB;\nfull set ~1.4 TB). For setup answers, include env preflight, `docker login`, the parallel\ndownload, `NIM_MODEL_NAME` launch, readiness, and then no-auth local inference. Do not invent a\ncache default or drop the `NVIDIA_API_KEY` fallback.\n\n```bash\n# --- env preflight (do not drop the NVIDIA_API_KEY fallback) ---\nset -a\n[ -f .env ] && . ./.env\nset +a\nif [ -z \"${NGC_API_KEY:-}\" ] && [ -n \"${NVIDIA_API_KEY:-}\" ]; then\n  export NGC_API_KEY=\"$NVIDIA_API_KEY\"\nfi\n: \"${NGC_API_KEY:?Set NGC_API_KEY or NVIDIA_API_KEY}\"\n: \"${DB_DIR:=/data/fast-db}\"          # where the parallel download lands\n\necho \"$NGC_API_KEY\" | docker login nvcr.io --username '$oauthtoken' --password-stdin\n\n# --- 1) pick the DB version(s) you need (paired/complex work = uniref30 only) ---\nDB_VERSION=uniref30_2302-m18v1\ncommand -v aria2c >/dev/null || sudo apt-get install -y aria2\nmkdir -p \"$DB_DIR\"\n\n# --- 2) parallel download from NGC (see \"Parallel Download\" section for the all-DB loop) ---\ncurl -fsS -H \"Authorization: Bearer $NGC_API_KEY\" \\\n  \"https://api.ngc.nvidia.com/v2/org/nim/team/colabfold/models/msa-search/${DB_VERSION}/files\" \\\n  -o /tmp/files.json\nDB_DIR=\"$DB_DIR\" python3 - <<'PY'\nimport json, os\nd = json.load(open(\"/tmp/files.json\")); dbdir = os.environ[\"DB_DIR\"]; lines = []\nfor url, path in zip(d[\"urls\"], d[\"filepath\"]):\n    lines += [url.strip(), f\"  dir={dbdir}\", f\"  out={path}\"]\nopen(\"/tmp/aria.in\", \"w\").write(\"\\n\".join(lines) + \"\\n\")\nPY\naria2c -i /tmp/aria.in --max-concurrent-downloads=4 --max-connection-per-server=16 \\\n  --split=16 --min-split-size=1M --continue=true --file-allocation=none\n\n# --- 3) launch the NIM against the downloaded files (skips the slow built-in download) ---\ndocker run -d --name msa-search --runtime=nvidia --gpus all \\\n  -e NGC_API_KEY \\\n  -e NIM_MODEL_NAME=/databases \\\n  -v \"${DB_DIR}:/databases\" \\\n  -p 8000:8000 \\\n  nvcr.io/nim/colabfold/msa-search:2\n```\n\nReadiness:\n\n```bash\nuntil curl -sf http://localhost:8000/v1/health/ready; do sleep 5; done\n```\n\nIf the DB is already present in `$DB_DIR`, skip steps 1-2 — the launch alone is a ~20 s warm\nstart. See \"Parallel Download For Any Database Set\" for the multi-database (`databases:all`)\nloop and the full rationale.\n\n### Fallback: Let The NIM Download Its Own Databases (slower)\n\nUse this only for a quick `databases:pdb70` smoke test, or when you specifically want the NIM\nto manage its own blob cache. It uses the built-in downloader, which is slow on large profiles\n(UniRef30 stalled past 80 min in testing). Pin the smallest profile with `NIM_MODEL_PROFILE`\n(see \"Faster Startup\") so it does not fetch the full 1.4 TB.\n\n```bash\n: \"${LOCAL_NIM_CACHE:?Set LOCAL_NIM_CACHE}\"\nmkdir -p \"${LOCAL_NIM_CACHE}\"; chmod 777 \"${LOCAL_NIM_CACHE}\"\ndocker run --rm --name msa-search \\\n  --runtime=nvidia --gpus all \\\n  -e NGC_API_KEY \\\n  -e NIM_MODEL_PROFILE=<hash-from-list-model-profiles> \\\n  -v \"${LOCAL_NIM_CACHE}:/opt/nim/.cache\" \\\n  -p 8000:8000 \\\n  nvcr.io/nim/colabfold/msa-search:2\n```\n\n\n## Faster Startup: Task-Specific Database Profiles\n\nThe full database download is ~1.4 TB and can take well over an hour on first launch. If you\nonly need some databases, select a **task-specific profile** so the NIM downloads just those.\nThis is the single biggest lever on local startup time.\n\nList the profiles your image actually ships (hashes change between releases — never hardcode\nthem):\n\n```bash\ndocker run --rm --entrypoint list-model-profiles nvcr.io/nim/colabfold/msa-search:2\n```\n\nThen pass the chosen hash with `NIM_MODEL_PROFILE`:\n\n```bash\ndocker run --rm --name msa-search \\\n  --runtime=nvidia --gpus all \\\n  -e NGC_API_KEY \\\n  -e NIM_MODEL_PROFILE=<hash-from-list-model-profiles> \\\n  -v \"${LOCAL_NIM_CACHE}:/opt/nim/.cache\" \\\n  -p 8000:8000 \\\n  nvcr.io/nim/colabfold/msa-search:2\n```\n\nProfiles available in this image (confirm hashes with `list-model-profiles`):\n\n| Profile tags | Databases | Best for | Storage |\n|---|---|---|---|\n| `databases:pdb70` | PDB70 | Quick testing / smoke check | ~100 MB |\n| `databases:uniref30` | UniRef30 | **Paired MSA search for complexes** — UniRef30 is the only DB used for species-based pairing | ~500 GB |\n| `databases:uniref30,pdb70,pdb` | UniRef30 + PDB70 + PDB structures | Structural template search | ~700 GB |\n| `databases:all` (default) | UniRef30 + ColabFold envdb + PDB70 + PDB100 + PDB structures | Full sensitivity, all databases | ~1.2 TB |\n\nVerify the loaded profile after readiness:\n\n```bash\ncurl -s localhost:8000/v1/metadata | jq\n```\n\nNotes:\n\n- The request-level `databases` parameter only selects among databases **already\n  downloaded**; it does NOT change what is fetched at startup. Startup footprint is set by\n  `NIM_MODEL_PROFILE` alone.\n- **Paired search needs UniRef30 only.** `colabfold_envdb_202108` has no taxonomy and cannot\n  be used for pairing, so `databases:uniref30` is the correct, smallest profile for\n  complex/paired workflows — it skips the envdb, the largest part of the full set.\n- For maximum monomer sensitivity (UniRef30 + envdb merged) you still need `databases:all`;\n  there is no envdb-inclusive profile smaller than the full set.\n\n### Custom Or Individual Databases\n\nTo use a single manually downloaded database (or your own MMSeqs2 DB), download it from NGC\nand point the NIM at the mount with `NIM_MODEL_NAME` instead of a profile:\n\n```bash\nngc registry model download-version nim/colabfold/msa-search:uniref30_2302-m18v1\n# then mount the directory and set -e NIM_MODEL_NAME=/databases\n```\n\n`NIM_MODEL_NAME` **replaces** the profile databases entirely — the NIM uses only what is\nunder that directory (discovered by scanning for `**/*.idx`). Mount multiple databases under\none parent to combine them. NGC-downloaded databases are pre-indexed for GPU Server; custom\ndatabases must be indexed with `mmseqs createindex` first. Individually downloadable NGC model\nversions: `uniref30_2302-m18v1`, `colabfold_envdb_202108-m18v1`, `pdb70_220313-m18v1`,\n`pdb100_230517-m18v1`, `pdb_20251028_zip-m18v1`.\n\n## Recommended: Parallel Download For Any Database Set (Fast Deployment)\n\n**This is the recommended way to download the databases at all — for any profile,\nincluding the full `databases:all` set.** Task-specific profiles cut *what* you\ndownload; this parallel downloader cuts *how long* that download takes. Use it whether\nyou need one database or all of them. The gain is largest for the `databases:uniref30` profile, which is ~490 GB\ndominated by two very large files (a ~241 GB GPU index and a ~134 GB sequence DB).\n\nThe NIM's built-in downloader parallelizes **across files** (`max_parallel_files=10`) but pulls\neach file over roughly one connection. The NGC CDN throttles a single connection to ~20–25\nMB/s, so while the downloader is fetching one of the two giant files, most of its parallel\nslots sit idle and throughput collapses to that single-flow rate. Measured on an H100 node,\nthe built-in path did not reach `/health/ready` in over 80 minutes.\n\nA range-parallel downloader splits **each file** into many byte-range segments (the NGC CDN\nadvertises `accept-ranges: bytes`), so a single 241 GB file is pulled over 16 connections at\nonce — ~15× the single-flow rate. Same node, `aria2c` fetched the full ~490 GB in **~13.5\nminutes**.\n\nWorkflow (download once with aria2, then start the NIM against the files via `NIM_MODEL_NAME`):\n\n```bash\n# 1) Get presigned file URLs for the individual database model version from NGC.\n#    (Requires NGC_API_KEY. The response arrays `urls` and `filepath` are positionally paired.)\ncurl -s -H \"Authorization: Bearer $NGC_API_KEY\" \\\n  'https://api.ngc.nvidia.com/v2/org/nim/team/colabfold/models/msa-search/uniref30_2302-m18v1/files' \\\n  -o files.json\n\n# 2) Build an aria2 input file (URL + target filename per entry) and download in parallel.\npython3 - <<'PY'\nimport json\nd = json.load(open(\"files.json\"))\nlines = []\nfor url, path in zip(d[\"urls\"], d[\"filepath\"]):\n    lines += [url.strip(), \"  dir=/data/fast-db\", f\"  out={path}\"]\nopen(\"aria.in\", \"w\").write(\"\\n\".join(lines) + \"\\n\")\nPY\naria2c -i aria.in \\\n  --max-concurrent-downloads=4 --max-connection-per-server=16 --split=16 \\\n  --min-split-size=1M --continue=true --file-allocation=none\n\n# 3) Start the NIM against the downloaded directory. NIM_MODEL_NAME makes the NIM discover\n#    databases by scanning for **/*.idx, bypassing the profile/blob cache entirely.\ndocker run -d --name msa-search --runtime=nvidia --gpus all \\\n  -e NGC_API_KEY \\\n  -e NIM_MODEL_NAME=/databases \\\n  -v /data/fast-db:/databases \\\n  -p 8000:8000 \\\n  nvcr.io/nim/colabfold/msa-search:2\n```\n\nFor **all databases** (equivalent to `databases:all`), repeat step 1 for each individual DB\nversion and download them into sibling directories under one parent, then point\n`NIM_MODEL_NAME` at that parent — the NIM discovers every DB by scanning `**/*.idx`:\n\n```bash\n# fetch each DB's file list into /data/all-db/<db>/ ... then one aria2c per list, e.g.:\nfor V in uniref30_2302-m18v1 colabfold_envdb_202108-m18v1 pdb70_220313-m18v1 \\\n         pdb100_230517-m18v1 pdb_20251028_zip-m18v1; do\n  curl -s -H \"Authorization: Bearer $NGC_API_KEY\" \\\n    \"https://api.ngc.nvidia.com/v2/org/nim/team/colabfold/models/msa-search/$V/files\" \\\n    -o \"files_$V.json\"\n  # build an aria2 input from files_$V.json (dir=/data/all-db) and run aria2c on it\ndone\n# then launch once against the parent:\n#   docker run -d ... -e NIM_MODEL_NAME=/databases -v /data/all-db:/databases ...\n```\n\nThe per-connection CDN throttle is the same for every database, so parallel download helps\nthe full set proportionally — the more you download, the more absolute time it saves.\n\nNotes:\n\n- The presigned URLs expire (typically within a day) — build the aria2 input and start the\n  download promptly after fetching `files.json`.\n- Keep the downloaded directory's internal layout intact (e.g. `uniref30_2302/…`); the\n  `filepath` values already encode it. The NIM needs the `.idx` file plus its companion files\n  and the small `.UNIREF30_READY` / `*.tar.gz.unpacked` markers.\n- The bottleneck is the CDN's per-connection cap, not local disk or CPU — a fast NVMe volume\n  writes far faster than the network delivers. Raising `--split` / `--max-connection-per-server`\n  helps only up to the node's aggregate egress ceiling.\n- Best of all: download the profile once, then **persist the cache volume** (or this\n  `fast-db` directory) and mount it on future nodes for a ~20 s warm start with no re-download.\n\n## Standard MSA Request\n\nUse exact case-sensitive database names and response keys.\n\n```python\nimport os\nimport requests\n\nHOSTED = True\nurl = (\n    \"https://health.api.nvidia.com/v1/biology/colabfold/msa-search/predict\"\n    if HOSTED else \"http://localhost:8000/biology/colabfold/msa-search/predict\"\n)\nheaders = {\"Content-Type\": \"application/json\"}\nif HOSTED:\n    headers[\"Authorization\"] = f\"Bearer {os.environ['NGC_API_KEY']}\"\n\npayload = {\n    \"sequence\": \"SGSMKTAISLPDETFDRVSRRASELGMSRSEFFTKAAQR\",\n    \"databases\": [\"Uniref30_2302\", \"colabfold_envdb_202108\"],\n    \"e_value\": 0.0001,\n    \"output_alignment_formats\": [\"a3m\"],\n}\nresponse = requests.post(url, headers=headers, json=payload, timeout=300)\nresponse.raise_for_status()\nresult = response.json()\n```\n\n## Paired MSA Request\n\nUse paired search for protein complexes; payload field is `sequences` plural,\nand output is `alignments_by_chain`.\n\n```python\nurl = (\n    \"https://health.api.nvidia.com/v1/biology/colabfold/msa-search/paired/predict\"\n    if HOSTED else \"http://localhost:8000/biology/colabfold/msa-search/paired/predict\"\n)\npayload = {\n    \"sequences\": [chain_a_sequence, chain_b_sequence],\n    \"e_value\": 0.0001,\n    \"output_alignment_formats\": [\"a3m\"],\n}\n```\n\n## Local Template Search\n\nUse local Docker for structural templates. Set `max_msa_sequences=500` unless\n`NIM_GLOBAL_MAX_MSA_DEPTH` was changed.\n\n```python\nurl = \"http://localhost:8000/biology/colabfold/msa-search/structure-templates/predict\"\nheaders = {\"Content-Type\": \"application/json\"}\npayload = {\n    \"sequence\": \"VLSPADKTNVKAAWGKVGAHAGEYGAEALERMFLSFPTTKTYFPHFDLSHGSAQVKGHGKKVADALTNAVA\",\n    \"structural_template_databases\": [\"pdb70_220313\"],\n    \"max_structures\": 20,\n    \"max_msa_sequences\": 500,\n}\n```\n\n## Save Outputs\n\n```python\n# Standard MSA: result[\"alignments\"][database][format][\"alignment\"]\nfor db_name, formats in result.get(\"alignments\", {}).items():\n    for fmt_name, data in formats.items():\n        with open(f\"msa_{db_name}.{fmt_name}\", \"w\", encoding=\"utf-8\") as handle:\n            handle.write(data[\"alignment\"])\n\n# Paired MSA: one alignment set per chain\nfor chain_id, chain_data in result.get(\"alignments_by_chain\", {}).items():\n    for db_name, formats in chain_data.items():\n        for fmt_name, data in formats.items():\n            with open(f\"msa_chain_{chain_id}_{db_name}.{fmt_name}\", \"w\", encoding=\"utf-8\") as handle:\n                handle.write(data[\"alignment\"])\n\n# Template search: save mmCIF structures and M8 hit tables\nfor name, cif in result.get(\"structures\", {}).items():\n    open(f\"template_{name}.cif\", \"w\", encoding=\"utf-8\").write(cif)\nfor name, hit_table in result.get(\"search_hits\", {}).items():\n    open(f\"template_hits_{name}.m8\", \"w\", encoding=\"utf-8\").write(hit_table)\n```\n\nA3M output can feed OpenFold3, AlphaFold2, or RoseTTAFold. For alignment depth,\ntemplate, and sequence sanity checks, read `references/validation.md`.\n\n## Limits And Troubleshooting\n\n- Sequence length: 1-4096 amino acids; `X` works since v2.3.0.\n- `max_msa_sequences`: 1-500; local GPU server default must match\n  `NIM_GLOBAL_MAX_MSA_DEPTH`.\n- Paired MSA requires at least two sequences.\n- Local URL 404 usually means an accidental `/v1/` prefix.\n- First local run can take hours while databases populate `LOCAL_NIM_CACHE`.\n"
}