Update to Domino Data Lab

{
  "description": "Work with Domino Volumes for NetApp ONTAP - enterprise-grade, multi-terabyte storage with near-instant snapshots. Covers volume creation, snapshot versioning with commit messages, cross-project sharing, mount paths (/mnt/netapp-volumes/ or /domino/netapp-volumes/), and the NetApp Volumes REST API. Use when managing large-scale data storage, needing fast no-copy snapshots, or integrating existing NetApp ONTAP infrastructure with Domino.",
  "included_files": [],
  "name": "netapp-volumes",
  "skill_md_contents": "---\nname: netapp-volumes\ndescription: Work with Domino Volumes for NetApp ONTAP - enterprise-grade, multi-terabyte storage with near-instant snapshots. Covers volume creation, snapshot versioning with commit messages, cross-project sharing, mount paths (/mnt/netapp-volumes/ or /domino/netapp-volumes/), and the NetApp Volumes REST API. Use when managing large-scale data storage, needing fast no-copy snapshots, or integrating existing NetApp ONTAP infrastructure with Domino.\n---\n\n# Domino NetApp Volumes Skill\n\n## Description\nThis skill helps users work with Domino Volumes for NetApp ONTAP — enterprise-grade storage that mounts NetApp ONTAP filesystems directly into Domino workloads, enabling multi-terabyte scale with near-instant snapshotting.\n\n## Activation\nActivate this skill when users want to:\n- Create or manage NetApp Volumes in Domino\n- Work with NetApp volume snapshots and versioning\n- Share large-scale data across projects or teams\n- Access existing NetApp ONTAP storage from Domino workspaces, jobs, or apps\n- Use the NetApp Volumes REST API programmatically\n- Understand mount paths for NetApp volumes\n- Choose between NetApp Volumes and Domino Datasets\n\n## What is a Domino NetApp Volume?\n\nA Domino NetApp Volume is:\n- **Enterprise-grade storage**: Backed by NetApp ONTAP via Kubernetes PVCs with the `netapp-storage` storage class\n- **Multi-terabyte scale**: No practical upper limit — suitable for very large datasets\n- **Near-instant snapshots**: ~3 seconds even for 100+ GB volumes using redirect-on-write (no extra storage consumed at snapshot time)\n- **Versioned with commit messages**: Snapshots support human-readable tags and commit messages, unlike Datasets\n- **Shareable**: Attach a single volume to multiple projects and teams\n- **Persistent**: Data persists across executions\n\n### NetApp Volumes vs. Domino Datasets\n\n| Feature | NetApp Volumes | Domino Datasets |\n|---------|---------------|-----------------|\n| Storage backend | External NetApp ONTAP | Domino-managed NFS/EFS |\n| Data scale | Multi-terabyte and beyond | Up to ~1 TB |\n| Snapshot speed | ~3 seconds (any size) | Scales with data size |\n| Snapshot storage cost | No extra space (redirect-on-write) | Duplicates physical data |\n| Commit messages on snapshots | Supported | Not supported |\n| Create new volume from snapshot | Read-only clones | Can create editable dataset |\n| Cross-project sharing | Yes | Yes |\n| Admin prerequisite | Admin must register filesystems | None |\n| REST API | Full dedicated API | Via Domino Python SDK |\n\n**Use NetApp Volumes when:**\n- Data exceeds ~1 TB\n- Near-instant snapshotting is required\n- Your organization already has NetApp ONTAP infrastructure\n- You need snapshot commit messages for audit trails\n- High-performance shared storage across many teams\n\n**Use Domino Datasets when:**\n- Data is Domino-managed with no external infrastructure dependency\n- You need to create editable new versions from snapshots\n- Data is under ~1 TB\n\n---\n\n## Prerequisites\n\nAn admin must register at least one NetApp filesystem (Kubernetes PVC with the `netapp-storage` label) before users can create volumes. Contact your Domino administrator if no filesystems are available.\n\n---\n\n## Creating a NetApp Volume\n\n### From the Domino Home Page\n1. Navigate to **Data > NetApp Volumes** in the toolbar\n2. Click **Add NetApp Volume > Create Volume**\n3. Fill in the form:\n   - **Name**: Letters, numbers, underscores, hyphens only\n   - **Description**: Brief overview of the data\n   - **Data Plane**: Select from available options\n   - **NetApp Filesystem**: Choose a registered filesystem\n   - **Capacity**: Set maximum storage allocation\n4. Click **Next** to configure permissions\n5. Assign users with **Reader / Editor / Owner** roles\n6. Click **Finish**\n7. Manually add the volume to projects afterward\n\n### From within a Project (recommended — auto-associates)\n1. Open a project → **Data > NetApp Volumes** (left panel)\n2. Click **Add NetApp Volume > Create Volume**\n3. Fill in the same form fields as above\n4. The volume is automatically associated with the current project on creation\n\n### Via REST API\n```python\nimport os\nimport requests\n\n# Auth token from in-cluster token service\ntoken = requests.get(\"http://localhost:8899/access-token\").text.strip()\nheaders = {\n    \"Authorization\": f\"Bearer {token}\",\n    \"Content-Type\": \"application/json\"\n}\n\napi_url = os.environ[\"DOMINO_API_HOST\"]\nremotefs_url = os.environ[\"DOMINO_REMOTE_FILE_SYSTEM_HOSTPORT\"]\n\n# Look up your user ID (username is available in the DOMINO_USER_NAME env var)\nuser_resp = requests.get(\n    f\"{api_url}/v4/users?userName={os.environ['DOMINO_USER_NAME']}\",\n    headers=headers\n).json()\nuser_id = user_resp[0][\"id\"]\n\n# Create a volume (capacity is in bytes; grants is required)\nresponse = requests.post(\n    f\"{remotefs_url}/remotefs/v1/volumes\",\n    headers=headers,\n    json={\n        \"name\": \"large-training-data\",\n        \"description\": \"Multi-TB training dataset for vision models\",\n        \"filesystemId\": \"<filesystem-id>\",\n        \"capacity\": 5_000_000_000_000,  # 5 TB in bytes\n        \"grants\": [\n            {\"targetId\": user_id, \"targetRole\": \"VolumeOwner\"}\n        ]\n    }\n)\nvolume = response.json()\nprint(f\"Created volume ID: {volume['id']}\")\n```\n\n---\n\n## Adding a Volume to a Project\n\n### Via Domino UI\n1. Go to project → **Data > NetApp Volumes**\n2. Click **Add NetApp Volume > Add Existing Volume**\n3. Select the volume from the list\n4. Configure access level for the project\n\n### Via REST API\n```python\n# Attach a volume to a project\nrequests.post(\n    f\"{remotefs_url}/remotefs/v1/rpc/attach-volume-to-project\",\n    headers=headers,\n    json={\n        \"volumeId\": \"<volume-id>\",\n        \"projectId\": \"<project-id>\"\n    }\n)\n```\n\n---\n\n## Mount Paths\n\nMount paths depend on your **project type**. Check which exists in your execution to determine your project type.\n\n### Git-Based Projects\n\n| Volume Type | Mount Path |\n|-------------|-----------|\n| Live volume | `/mnt/netapp-volumes/<volume-name>/` |\n| Snapshot by number | `/mnt/netapp-volumes/snapshots/<volume-name>/<snapshot-number>/` |\n| Snapshot by tag | `/mnt/netapp-volumes/snapshot-tags/<volume-name>/<tag-name>/` |\n\n### DFS (Domino File System) Projects\n\n| Volume Type | Mount Path |\n|-------------|-----------|\n| Live volume | `/domino/netapp-volumes/<volume-name>/` |\n| Snapshot by number | `/domino/netapp-volumes/snapshots/<volume-name>/<snapshot-number>/` |\n| Snapshot by tag | `/domino/netapp-volumes/snapshot-tags/<volume-name>/<tag-name>/` |\n\n### Snapshot Access Behavior\n\nThere are two ways to access snapshots, with an important behavioral difference:\n\n- **`/snapshots/<volume-name>/<number>/`** — Accessed by snapshot number. When a new snapshot is taken while a workspace is running, the new numbered directory appears **immediately** in that workspace without a restart.\n- **`/snapshot-tags/<volume-name>/<tag-name>/`** — Accessed by tag name. Each snapshot has at most one active tag path — a symlink to its numbered snapshot directory. If you apply multiple tags to the same snapshot, only the most recently applied tag creates a path; earlier tags for that snapshot are not accessible by path. Tag paths for new snapshots are also **not** visible in a running workspace — you must **restart the workspace** for them to appear.\n\nUse the numbered path when you need to access a fresh snapshot from within a live workspace. Use the tagged path for stable, named references in reproducible runs.\n\n> **Important:** Each snapshot exposes only one tag path at a time — the most recently applied tag. Older tags on the same snapshot do not have accessible paths.\n\n> **Important:** Renaming a volume changes its mount path. Update any hardcoded paths in your code after renaming.\n\n### Identify Your Project Type\n\n```python\nimport os\n\nif os.path.exists(\"/domino/netapp-volumes\"):\n    print(\"DFS Project\")\n    netapp_root = \"/domino/netapp-volumes\"\nelif os.path.exists(\"/mnt/netapp-volumes\"):\n    print(\"Git-Based Project\")\n    netapp_root = \"/mnt/netapp-volumes\"\n```\n\n### Permissions\n\n- **Owners/Editors**: Read-write access to the live volume\n- **Readers**: Read-only access\n\n### Example: Reading Data\n\n```python\nimport pandas as pd\n\n# Git-Based Project\ndf = pd.read_parquet(\"/mnt/netapp-volumes/large-training-data/features.parquet\")\n\n# DFS Project\ndf = pd.read_parquet(\"/domino/netapp-volumes/large-training-data/features.parquet\")\n\n# Read from a specific snapshot tag\ndf = pd.read_parquet(\"/mnt/netapp-volumes/snapshot-tags/large-training-data/v2.0/features.parquet\")\n```\n\n### Example: Writing Data\n\n```python\n# Write directly to the live volume\ndf.to_parquet(\"/mnt/netapp-volumes/large-training-data/processed/output.parquet\", index=False)\n\n# List files\nimport os\nfiles = os.listdir(\"/mnt/netapp-volumes/large-training-data/\")\n```\n\n---\n\n## Snapshots and Versioning\n\n### What is a Snapshot?\nA snapshot is a read-only, immutable record of the volume's data at a specific point in time. NetApp snapshots use redirect-on-write — no additional storage is consumed at creation time.\n\n**Advantages over Domino Dataset snapshots:**\n- Near-instant for any volume size (~3 seconds)\n- Support commit messages for documentation and audit trails\n- No storage overhead at creation time\n\n### Create a Snapshot via UI\n1. Go to project → **Data > NetApp Volumes**\n2. Select the volume → click **Take Snapshot**\n3. Add an optional **commit message** (e.g., \"Added Q4 customer records\")\n4. Add an optional **tag name** (e.g., `v2.0`, `production`, `2024-Q4`)\n5. Click **Confirm**\n\n### Create a Snapshot via REST API\n```python\n# Create snapshot with description and one or more tags (tagNames is an array)\nresponse = requests.post(\n    f\"{remotefs_url}/remotefs/v1/snapshots\",\n    headers=headers,\n    json={\n        \"volumeId\": \"<volume-id>\",\n        \"description\": \"Added Q4 customer records — 50M new rows\",\n        \"tagNames\": [\"v2.0\"]\n    }\n)\nsnapshot = response.json()\nprint(f\"Snapshot ID: {snapshot['id']}\")\n```\n\n### Add a Tag to an Existing Snapshot\n```python\nrequests.post(\n    f\"{remotefs_url}/remotefs/v1/snapshots/{snapshot_id}/tags\",\n    headers=headers,\n    json={\"name\": \"production\"}\n)\n```\n\n### Create Snapshot After a Job Run\n```python\n# Snapshot tied to a specific Domino run for reproducibility\nrequests.post(\n    f\"{remotefs_url}/remotefs/v1/rpc/create-snapshot-from-run\",\n    headers=headers,\n    json={\n        \"volumeId\": \"<volume-id>\",\n        \"runId\": \"<domino-run-id>\",\n        \"userId\": \"<user-id>\",\n        \"description\": \"post-training-run-42\"\n    }\n)\n```\n\n### Restore a Snapshot\n```python\nrequests.post(\n    f\"{remotefs_url}/remotefs/v1/rpc/restore-snapshot\",\n    headers=headers,\n    json={\"snapshotId\": \"<snapshot-id>\"}\n)\n```\n\n---\n\n## Roles and Permissions\n\n| Role | API Value | Capabilities |\n|------|-----------|-------------|\n| **Reader** | `VolumeReader` | View files and snapshots; mount as read-only |\n| **Editor** | `VolumeEditor` | All Reader capabilities + modify description, create/delete snapshots, manage shared access, manage users |\n| **Owner** | `VolumeOwner` | All Editor capabilities + update volume grants, request deletion |\n\n### Update Permissions via UI\n1. Go to **Data > NetApp Volumes** → select volume\n2. Click the three-dot menu → **Edit permissions**\n3. Add/remove users and assign roles\n\n### Update Permissions via REST API\n```python\n# targetRole values: \"VolumeOwner\", \"VolumeEditor\", \"VolumeReader\"\nrequests.put(\n    f\"{remotefs_url}/remotefs/v1/volumes/{volume_id}/grants\",\n    headers=headers,\n    json=[\n        {\"targetId\": \"<user-id>\", \"targetRole\": \"VolumeEditor\"},\n        {\"targetId\": \"<other-user-id>\", \"targetRole\": \"VolumeReader\"}\n    ]\n)\n```\n\n---\n\n## Using NetApp Volumes in Jobs\n\n### Via Domino REST API\n```python\nrequests.post(\n    f\"{api_url}/api/jobs/v1/jobs\",\n    headers=headers,\n    json={\n        \"projectId\": \"<project-id>\",\n        \"runCommand\": \"python train.py\",\n        \"hardwareTierId\": \"<hardware-tier-id>\",\n        \"environmentId\": \"<environment-id>\",  # required — use DOMINO_ENVIRONMENT_ID env var\n        \"netAppVolumeIds\": [\"<volume-id>\"],\n        \"snapshotNetAppVolumesOnCompletion\": True  # auto-snapshot mounted volumes when job finishes\n    }\n)\n```\n\nSet `snapshotNetAppVolumesOnCompletion: true` to automatically take a snapshot of all mounted NetApp volumes when the job completes. This is the recommended approach for training jobs — it captures the exact state of the volume at the end of the run without requiring a separate API call.\n\n---\n\n## Listing Volumes and Snapshots\n\n```python\n# List all volumes accessible to you\nvolumes = requests.get(\n    f\"{remotefs_url}/remotefs/v1/volumes\",\n    headers=headers\n).json()\n\nfor v in volumes[\"data\"]:\n    capacity_tb = v[\"capacity\"] / 1_000_000_000_000\n    print(f\"{v['name']} — {capacity_tb:.1f} TB — ID: {v['id']}\")\n\n# List snapshots for a volume\nsnapshots = requests.get(\n    f\"{remotefs_url}/remotefs/v1/snapshots\",\n    headers=headers,\n    params={\"volumeId\": \"<volume-id>\"}\n).json()\n\nfor s in snapshots[\"data\"]:\n    print(f\"Snapshot {s['id']} v{s['version']} — {s.get('description', '')} — tags: {[t['name'] for t in s.get('tags', [])]}\")\n```\n\n---\n\n## Best Practices\n\n### 1. Use Appropriate Storage\n| Data Type | Storage |\n|-----------|---------|\n| Large data of any file type where the entire filesystem should be versioned as a unit, with intentional snapshots | NetApp Volume |\n| Small output files — charts, reports, model binaries | Artifacts / DFS (auto-versioned per file, not suitable for large files) |\n| Code | Git / Project files |\n\n### 2. Snapshot Before Changes\n```python\n# Always snapshot before modifying large volumes\nrequests.post(\n    f\"{remotefs_url}/remotefs/v1/snapshots\",\n    headers=headers,\n    json={\n        \"volumeId\": \"<volume-id>\",\n        \"description\": \"Pre-processing baseline snapshot\",\n        \"tagNames\": [\"pre-processing-2024-01\"]\n    }\n)\n\n# Then run your data transformation\nprocess_data()\n```\n\n### 3. Use Efficient File Formats\n```python\n# Parquet for tabular data (faster reads, smaller storage)\ndf.to_parquet(\"/mnt/netapp-volumes/dataset/data.parquet\")\n\n# Feather for fast pandas I/O\ndf.to_feather(\"/mnt/netapp-volumes/dataset/data.feather\")\n\n# HDF5 for large numerical arrays\nimport h5py\nwith h5py.File(\"/mnt/netapp-volumes/dataset/arrays.h5\", \"w\") as f:\n    f.create_dataset(\"features\", data=features_array)\n```\n\n### 4. Organize Data\n```\n/mnt/netapp-volumes/my-volume/\n├── raw/\n│   ├── 2024-Q1/\n│   └── 2024-Q2/\n├── processed/\n│   ├── features.parquet\n│   └── labels.parquet\n└── metadata/\n    └── schema.json\n```\n\n### 5. Use Snapshot Tags for Reproducibility\nReference snapshot tags (not snapshot IDs) in your training scripts so that tagged paths remain stable across runs:\n```python\n# Reproducible reference using a tag\nTRAINING_DATA = \"/mnt/netapp-volumes/snapshot-tags/dataset/v2.0/\"\ndf = pd.read_parquet(f\"{TRAINING_DATA}/features.parquet\")\n```\n\n### 6. Reading Large Volumes Efficiently\n```python\nimport dask.dataframe as dd\n\n# Lazy read — no data loaded until .compute()\ndf = dd.read_parquet(\"/mnt/netapp-volumes/dataset/large_data.parquet\")\nresult = df.groupby(\"category\").mean().compute()\n```\n\n---\n\n## Troubleshooting\n\n### Volume Not Found at Mount Path\n- Verify the volume is added to the project (**Data > NetApp Volumes**)\n- Confirm the volume name matches the path exactly (case-sensitive)\n- Check that the workspace/job was started after the volume was added to the project\n- Restart the workspace to pick up newly added volumes\n\n### Permission Denied\n- Check your role on the volume (need Editor or Owner to write)\n- Verify you have been granted access by the volume owner\n- For read-only snapshots, use the snapshot tag path instead of the live volume path\n\n### Snapshot Tag Not Mounted\n- Each snapshot has at most **one** tag path — the most recently applied tag. Earlier tags on the same snapshot are never surfaced as directories and cannot be accessed by path.\n- Tag paths for new snapshots do **not** appear in a running workspace. The numbered snapshot directory (`/snapshots/<name>/<number>/`) appears immediately, but the tag symlink path only becomes visible after restarting the workspace.\n- Verify the tag was created successfully via the UI or API\n\n### No NetApp Filesystems Available\n- Contact your Domino administrator — they must register NetApp ONTAP filesystems before volumes can be created\n- Admins can register filesystems via **Admin > NetApp Volumes**\n\n### Slow Read Performance\n- Use columnar formats (Parquet, Feather) instead of CSV for tabular data\n- Read only needed columns: `pd.read_parquet(path, columns=[\"col1\", \"col2\"])`\n- Use Dask or chunked reading for files larger than available RAM\n\n---\n\n## Documentation Reference\n\nBefore writing or verifying any API call, use the cluster swagger to confirm current endpoint paths and field names. Use public docs for workflow context and field explanations.\n\n**Get the cluster base URL:** `$DOMINO_API_HOST` (injected by Domino into every workspace, job, and app).\n\nFetch the NetApp Volumes swagger spec (requires bearer token):\n```bash\nTOKEN=$(curl -s http://localhost:8899/access-token)\n# The swagger UI is only accessible via the external cluster URL (not $DOMINO_API_HOST).\n# Derive it from the JWT iss claim — works in any workspace type.\nCLUSTER_URL=$(echo $TOKEN | cut -d'.' -f2 | python3 -c \"\nimport sys, base64, json, re\np = sys.stdin.read().strip()\np += '=' * (-len(p) % 4)\nprint(re.sub(r'/auth/realms/.*', '', json.loads(base64.b64decode(p))['iss']))\n\")\ncurl -H \"Authorization: Bearer $TOKEN\" \"$CLUSTER_URL/domino-netapp-volumes/swagger/doc.json\"\n# Browser UI (must be logged in): $CLUSTER_URL/domino-netapp-volumes/swagger/index.html\n```\n\n**Public docs (workflow context and field explanations):**\n- [NetApp Volumes REST API Reference](https://docs.dominodatalab.com/en/cloud/api_guide/b3b2a1/domino-netapp-volumes-api/)\n- [Work with NetApp Volumes](https://docs.dominodatalab.com/en/cloud/user_guide/06da1b/work-with-netapp-volumes/)\n- [Create NetApp Volumes](https://docs.dominodatalab.com/en/cloud/user_guide/e6887f/create-netapp-volumes-from-domino-or-a-project/)\n- [Add or Remove NetApp Volumes on Projects](https://docs.dominodatalab.com/en/cloud/user_guide/306570/add-or-remove-netapp-volumes-on-projects/)\n- [View and Edit NetApp Volumes](https://docs.dominodatalab.com/en/cloud/user_guide/93fa12/view-and-edit-netapp-volumes/)\n- [Version Data with Snapshots](https://docs.dominodatalab.com/en/cloud/user_guide/dbdbff/version-data-with-snapshots/)\n- [Access Data in Domino (comparison guide)](https://docs.dominodatalab.com/en/cloud/user_guide/16d9c1/access-data-in-domino/)\n"
}