{"id":53686,"plugin_id":"plugins_6ac012b82680819195f8a26cbe32c246","kind":"skill","collection_source":"skill_api","comparison_source":null,"observed_at":"2026-10-24T13:54:51.050Z","digest":"9abc0c796b27b2de2ed5cb8d15c2e22aa85e4e8a876b7c99173283229b811f08","against":null,"payload":{"description":"Work with Domino Datasets - high-performance, versioned filesystem storage. Covers dataset creation, snapshots for versioning, sharing across projects, mounting paths (/domino/datasets/), and performance optimization. Use when managing data storage, creating reproducible data versions, or sharing data between projects.","included_files":[],"name":"domino-datasets","skill_md_contents":"---\nname: domino-datasets\ndescription: Work with Domino Datasets - high-performance, versioned filesystem storage. Covers dataset creation, snapshots for versioning, sharing across projects, mounting paths (/domino/datasets/), and performance optimization. Use when managing data storage, creating reproducible data versions, or sharing data between projects.\n---\n\n# Domino Datasets Skill\n\n## Description\nThis skill helps users work with Domino Datasets - high-performance, versioned filesystem storage for data science projects.\n\n## Activation\nActivate this skill when users want to:\n- Create or manage Domino Datasets\n- Work with dataset snapshots and versioning\n- Share data between projects\n- Access large datasets efficiently\n- Understand dataset paths and mounting\n\n## What is a Domino Dataset?\n\nA Domino Dataset is:\n- **High-performance storage**: Network filesystem optimized for data science\n- **Versioned**: Create snapshots for reproducibility\n- **Shareable**: Access across projects\n- **Scalable**: No file size or count limits\n- **Persistent**: Data persists across executions\n\n## Creating a Dataset\n\n### Via Domino UI\n1. Navigate to your project\n2. Go to **Data** > **Domino Datasets**\n3. Click **Create New Dataset**\n4. Enter:\n   - **Name**: Dataset name (e.g., `training-data`)\n   - **Description**: What the dataset contains\n5. Click **Create**\n\n### Via Python SDK\n```python\nfrom domino import Domino\n\ndomino = Domino(\"project-owner/project-name\")\n\n# Create a new dataset\ndataset = domino.datasets_create(\n    name=\"training-data\",\n    description=\"Training data for classification model\"\n)\n```\n\n## Dataset Paths\n\nDataset paths differ based on your **project type**. Domino has two project types with different mount structures.\n\n### DFS (Domino File System) Projects\n\nDFS projects use `/domino` as the root:\n\n```\n/domino\n   |--/datasets\n      |--/local               <== Local datasets and snapshots\n         |--/clapton          <== Read-write dataset for owner and editor, read-only for reader\n         |--/mingus           <== Read-write dataset for owner and editor, read-only for reader\n         |--/snapshots        <== Snapshot folder organized by dataset\n            |--/clapton       <== Read-write for owner and editor, read-only for reader\n               |--/tag1          <== Mounted under latest tag\n               |--/1             <== Always mounted under the snapshot number\n               |--/2\n            |--/mingus\n               |--/tag2\n               |--/1\n               |--/2\n      |--/ella                <== Read-write shared dataset for owner and editor, Read-only for reader\n      |--/davis               <== Read-write shared dataset for owner and editor, Read-only for reader\n      |--/snapshots           <== Shared datasets snapshots organized by dataset\n         |--/ella             <== Read-write for owner and editor, read-only for reader\n            |--/tag3          <== Mounted under latest tag\n            |--/1             <== Always mounted under the snapshot number\n            |--/2\n         |--/davis\n            |--/tag4\n            |--/1\n            |--/2\n```\n\n| Dataset Type | Path |\n|-------------|------|\n| Local datasets | `/domino/datasets/local/{dataset-name}/` |\n| Local snapshots | `/domino/datasets/local/snapshots/{dataset-name}/{tag-or-number}/` |\n| Shared datasets | `/domino/datasets/{dataset-name}/` |\n| Shared snapshots | `/domino/datasets/snapshots/{dataset-name}/{tag-or-number}/` |\n\n### Git-Based Projects\n\nGit-based projects use `/mnt` as the root:\n\n```\n/mnt\n   |--/data                  <== Local datasets and snapshots\n     |--/clapton             <== Read-write dataset for owner and editor, read-only for reader\n     |--/mingus              <== Read-write dataset for owner and editor, read-only for reader\n     |--/snapshots           <== Snapshot folder organized by dataset\n        |--/clapton          <== Read-write for owner and editor, read-only for reader\n           |--/tag1          <== Mounted under latest tag\n           |--/1             <== Always mounted under the snapshot number\n           |--/2\n        |--/mingus\n           |--/tag2\n           |--/1\n           |--/2\n   |--/imported\n     |--/data\n        |--/ella             <== Read-write shared dataset for owner and editor, read-only for reader\n        |--/davis            <== Read-write shared dataset for owner and editor, read-only for reader\n        |--/snapshots        <== Shared dataset snapshots organized by dataset\n           |--/ella          <== Read-write for owner and editor, read-only for reader\n              |--/tag3       <== Mounted under latest tag\n              |--/1          <== Always mounted under the snapshot number\n              |--/2\n           |--/davis\n              |--/tag4\n              |--/1\n              |--/2\n```\n\n| Dataset Type | Path |\n|-------------|------|\n| Local datasets | `/mnt/data/{dataset-name}/` |\n| Local snapshots | `/mnt/data/snapshots/{dataset-name}/{tag-or-number}/` |\n| Shared datasets | `/mnt/imported/data/{dataset-name}/` |\n| Shared snapshots | `/mnt/imported/data/snapshots/{dataset-name}/{tag-or-number}/` |\n\n### How to Identify Your Project Type\n\nCheck which paths exist in your execution:\n```python\nimport os\n\nif os.path.exists(\"/domino/datasets\"):\n    print(\"DFS Project\")\n    dataset_root = \"/domino/datasets/local\"\nelif os.path.exists(\"/mnt/data\"):\n    print(\"Git-Based Project\")\n    dataset_root = \"/mnt/data\"\n```\n\n### Permissions\n\nBoth project types follow the same permission model:\n- **Owners/Editors**: Read-write access to datasets\n- **Readers**: Read-only access\n\n### Example: Reading Data\n\n```python\nimport pandas as pd\n\n# Git-Based Project\ndf = pd.read_csv(\"/mnt/data/training-data/customers.csv\")\n\n# DFS Project\ndf = pd.read_csv(\"/domino/datasets/local/training-data/customers.csv\")\n\n# List files\nimport os\nfiles = os.listdir(\"/mnt/data/training-data/\")  # Git-Based\nfiles = os.listdir(\"/domino/datasets/local/training-data/\")  # DFS\n```\n\n## Uploading Data\n\n### Via Domino UI\n1. Go to dataset page\n2. Click **Upload**\n3. Select files (up to 50GB or 50,000 files via UI)\n4. Click **Upload**\n\n### Via Domino CLI (Large Uploads)\n```bash\n# For large uploads, use CLI\ndomino upload /local/path/to/data /mnt/data/training-data/\n```\n\n### Via Code in Workspace\n```python\nimport shutil\n\n# Copy from local to dataset\nshutil.copy(\"local_file.csv\", \"/mnt/data/training-data/\")\n\n# Write directly\ndf.to_csv(\"/mnt/data/training-data/processed.csv\", index=False)\n```\n\n## Snapshots\n\n### What is a Snapshot?\nA snapshot is a read-only, immutable version of your dataset at a point in time. Use snapshots for:\n- Reproducibility\n- Versioning training data\n- Rolling back to previous states\n\n### Create a Snapshot\n```python\n# Via Python SDK\nsnapshot = domino.datasets_snapshot(\n    dataset_name=\"training-data\",\n    tag=\"v1.0\"\n)\n```\n\nOr via UI:\n1. Go to dataset page\n2. Click **Create Snapshot**\n3. Add optional tag (e.g., `v1.0`, `production`)\n\n### Access Snapshots\n```python\n# Latest snapshot\ndf = pd.read_csv(\"/mnt/data/training-data/data.csv\")\n\n# Specific tagged snapshot\ndf = pd.read_csv(\"/mnt/data/training-data@v1.0/data.csv\")\n```\n\n### Snapshot Limits\n- Default limit: 20 snapshots per dataset\n- Configurable by admins\n- Oldest snapshots auto-deleted when limit reached\n\n## Tags\n\n### What are Tags?\nTags provide friendly names for snapshots:\n- `production`: Current production data\n- `v1.0`, `v2.0`: Version numbers\n- `2024-01-15`: Date-based tags\n\n### Move Tags\nTags can be moved to different snapshots:\n```python\n# Move 'production' tag to latest snapshot\ndomino.datasets_tag(\n    dataset_name=\"training-data\",\n    snapshot_id=\"snapshot-123\",\n    tag=\"production\"\n)\n```\n\n## Sharing Datasets\n\n### Within Organization\n1. Go to dataset settings\n2. Set visibility to **Organization**\n3. Other projects can mount the dataset\n\n### Cross-Project Access\n```python\n# Import dataset from another project\n# Configured in project settings\ndf = pd.read_csv(\"/mnt/data/shared-dataset/data.csv\")\n```\n\n## Best Practices\n\n### 1. Use Appropriate Storage\n| Data Type | Storage |\n|-----------|---------|\n| Large training data | Domino Dataset |\n| Model artifacts | `/mnt/artifacts/` |\n| Code | Git/Project files |\n| Temporary files | `/tmp/` |\n\n### 2. Organize Data\n```\n/mnt/data/my-dataset/\n├── raw/\n│   ├── customers.csv\n│   └── transactions.csv\n├── processed/\n│   ├── features.parquet\n│   └── labels.parquet\n└── metadata/\n    └── schema.json\n```\n\n### 3. Use Efficient Formats\n```python\n# Parquet for tabular data (faster, smaller)\ndf.to_parquet(\"/mnt/data/dataset/data.parquet\")\n\n# Feather for pandas DataFrames\ndf.to_feather(\"/mnt/data/dataset/data.feather\")\n\n# HDF5 for numerical arrays\nimport h5py\nwith h5py.File(\"/mnt/data/dataset/data.h5\", \"w\") as f:\n    f.create_dataset(\"features\", data=features)\n```\n\n### 4. Document Data\nInclude README and schema:\n```python\n# Write metadata\nmetadata = {\n    \"created\": \"2024-01-15\",\n    \"source\": \"Customer database\",\n    \"columns\": {\"id\": \"int\", \"name\": \"string\", \"value\": \"float\"}\n}\n\nwith open(\"/mnt/data/dataset/metadata.json\", \"w\") as f:\n    json.dump(metadata, f)\n```\n\n### 5. Snapshot Before Changes\n```python\n# Create snapshot before processing\ndomino.datasets_snapshot(\n    dataset_name=\"training-data\",\n    tag=\"pre-processing\"\n)\n\n# Then modify data\nprocess_data()\n```\n\n## Reading Large Datasets\n\n### Chunked Reading\n```python\n# Read in chunks\nchunks = pd.read_csv(\n    \"/mnt/data/dataset/large_file.csv\",\n    chunksize=100000\n)\n\nfor chunk in chunks:\n    process(chunk)\n```\n\n### Lazy Loading with Dask\n```python\nimport dask.dataframe as dd\n\n# Read without loading into memory\ndf = dd.read_parquet(\"/mnt/data/dataset/large_data.parquet\")\n\n# Process lazily\nresult = df.groupby(\"category\").mean().compute()\n```\n\n### Memory Mapping\n```python\nimport numpy as np\n\n# Memory-map large arrays\ndata = np.memmap(\n    \"/mnt/data/dataset/features.dat\",\n    dtype='float32',\n    mode='r',\n    shape=(1000000, 100)\n)\n```\n\n## Troubleshooting\n\n### Dataset Not Found\n- Verify dataset name is correct\n- Check dataset is mounted to project\n- Confirm you have access permissions\n\n### Permission Denied\n- Check project role (need Contributor+)\n- Verify dataset sharing settings\n- Contact dataset owner\n\n### Slow Performance\n- Use efficient file formats (Parquet > CSV)\n- Read only needed columns\n- Use chunked/lazy loading for large files\n\n### Snapshot Failed\n- Check disk quota\n- Verify no files are open/locked\n- Check snapshot limit not reached\n\n## Documentation Reference\n- [Create and manage Datasets](https://docs.dominodatalab.com/en/latest/user_guide/0a8d11/create-and-manage-datasets/)\n- [Version data with Snapshots](https://docs.dominodatalab.com/en/cloud/user_guide/dbdbff/version-data-with-snapshots/)\n- [Datasets best practices](https://archive.docs.dominodatalab.com/en/4.4/user_guide/a222c9/datasets-best-practices/)\n"},"changes":[],"summary":"First saved snapshot. No earlier version is available for comparison.","summary_kind":"deterministic","summary_metadata":{}}