{"id":53660,"plugin_id":"plugins_6ac012b82680819195f8a26cbe32c246","kind":"skill","collection_source":"skill_api","comparison_source":null,"observed_at":"2026-10-24T13:54:47.090Z","digest":"a52082eb8d945920ae4fe0d29595df36c049783ac7dc24d07a47beb942a0e2df","against":null,"payload":{"description":"Access external LLM providers through Domino AI Gateway - a secure proxy with centralized API key management, usage monitoring, and compliance. Supports OpenAI, AWS Bedrock, Azure OpenAI, Anthropic, and more. Use when calling LLMs from Domino, configuring AI Gateway endpoints, or monitoring LLM usage and costs.","included_files":[],"name":"domino-ai-gateway","skill_md_contents":"---\nname: domino-ai-gateway\ndescription: Access external LLM providers through Domino AI Gateway - a secure proxy with centralized API key management, usage monitoring, and compliance. Supports OpenAI, AWS Bedrock, Azure OpenAI, Anthropic, and more. Use when calling LLMs from Domino, configuring AI Gateway endpoints, or monitoring LLM usage and costs.\n---\n\n# Domino AI Gateway Skill\n\n## Description\nThis skill helps users work with Domino AI Gateway - a secure proxy for accessing external Large Language Model (LLM) providers with centralized management, monitoring, and compliance.\n\n## Activation\nActivate this skill when users want to:\n- Access LLM providers (OpenAI, AWS Bedrock, etc.) in Domino\n- Configure AI Gateway endpoints\n- Monitor LLM usage and costs\n- Understand secure API key management\n- Use LLMs in workspaces and jobs\n\n## What is AI Gateway?\n\nDomino AI Gateway provides:\n- **Secure LLM Access**: Proxy to external LLM providers\n- **Centralized API Keys**: Keys stored securely, never exposed to users\n- **Usage Monitoring**: Track all LLM interactions\n- **Access Control**: Granular permissions per endpoint\n- **Audit Logging**: Compliance-ready logs\n\n## Supported LLM Providers\n\n| Provider | Models |\n|----------|--------|\n| OpenAI | GPT-4, GPT-4 Turbo, GPT-3.5 |\n| AWS Bedrock | Claude, Titan, Llama 2 |\n| Azure OpenAI | GPT-4, GPT-3.5 |\n| Anthropic | Claude 3, Claude 2 |\n| Google Vertex AI | PaLM, Gemini |\n| Cohere | Command, Embed |\n\n## Creating an AI Gateway Endpoint\n\n### Via Domino UI\n1. Go to **Endpoints** > **Gateway LLMs**\n2. Click **Create Endpoint**\n3. Configure:\n   - **Name**: Endpoint name (e.g., `openai-gpt4`)\n   - **Provider**: Select LLM provider\n   - **Model**: Specific model to use\n   - **API Key**: Provider API key (stored securely)\n   - **Access**: Who can use this endpoint\n4. Click **Create**\n\n### Via API\n```python\n# Create endpoint via Domino API\nimport requests, os\n\nTOKEN = requests.get(\"http://localhost:8899/access-token\").text.strip()\nBASE = os.environ[\"DOMINO_API_HOST\"]\n\nresponse = requests.post(\n    f\"{BASE}/api/aigateway/v1/endpoints\",\n    headers={\"Authorization\": f\"Bearer {TOKEN}\"},\n    json={\n        \"name\": \"openai-gpt4\",\n        \"provider\": \"openai\",\n        \"model\": \"gpt-4\",\n        \"providerApiKey\": \"sk-...\"\n    }\n)\n```\n\n## Using AI Gateway in Code\n\n### OpenAI-Compatible Interface\nAI Gateway provides an OpenAI-compatible interface:\n\n```python\nfrom openai import OpenAI\n\n# Configure client to use AI Gateway\nclient = OpenAI(\n    api_key=\"not-needed\",  # Handled by AI Gateway\n    base_url=\"https://your-domino.com/api/aigateway/v1/openai\"\n)\n\n# Use like standard OpenAI\nresponse = client.chat.completions.create(\n    model=\"openai-gpt4\",  # Your endpoint name\n    messages=[\n        {\"role\": \"user\", \"content\": \"Hello, how are you?\"}\n    ]\n)\n\nprint(response.choices[0].message.content)\n```\n\n### With LangChain\n```python\nfrom langchain_openai import ChatOpenAI\n\nllm = ChatOpenAI(\n    model=\"openai-gpt4\",  # Endpoint name\n    openai_api_key=\"not-needed\",\n    openai_api_base=\"https://your-domino.com/api/aigateway/v1/openai\"\n)\n\nresponse = llm.invoke(\"What is machine learning?\")\nprint(response.content)\n```\n\n### Direct API Call\n```python\nimport requests, os\n\nTOKEN = requests.get(\"http://localhost:8899/access-token\").text.strip()\nBASE = os.environ[\"DOMINO_API_HOST\"]\n\nresponse = requests.post(\n    f\"{BASE}/api/aigateway/v1/chat/completions\",\n    headers={\n        \"Content-Type\": \"application/json\",\n        \"Authorization\": f\"Bearer {TOKEN}\",\n    },\n    json={\n        \"model\": \"openai-gpt4\",\n        \"messages\": [{\"role\": \"user\", \"content\": \"Hello!\"}]\n    }\n)\n\nresult = response.json()\nprint(result[\"choices\"][0][\"message\"][\"content\"])\n```\n\n## Access Control\n\n### Endpoint Permissions\nConfigure who can use each endpoint:\n- **Everyone**: All Domino users\n- **Specific Users**: Named individuals\n- **Organizations**: Specific Domino organizations\n\n### Setting Permissions\n1. Go to endpoint settings\n2. Click **Access Control**\n3. Add users or organizations\n4. Save changes\n\n## Monitoring and Logging\n\n### View Usage\n1. Go to **Endpoints** > **Gateway LLMs**\n2. Click on endpoint name\n3. View metrics:\n   - Request count\n   - Token usage\n   - Response times\n   - Error rates\n\n### Download Logs\n```bash\n# Via UI: Endpoints > Gateway LLMs > Download logs\n\n# Logs include:\n# - Timestamp\n# - User\n# - Model\n# - Input/Output tokens\n# - Response time\n# - Status\n```\n\n### Log Format\n```json\n{\n  \"timestamp\": \"2024-01-15T10:30:00Z\",\n  \"user\": \"user@company.com\",\n  \"endpoint\": \"openai-gpt4\",\n  \"model\": \"gpt-4\",\n  \"inputTokens\": 150,\n  \"outputTokens\": 200,\n  \"durationMs\": 1500,\n  \"status\": \"success\"\n}\n```\n\n## Cost Management\n\n### Track Costs\nAI Gateway tracks token usage per:\n- User\n- Project\n- Endpoint\n- Time period\n\n### Set Limits (Admin)\nAdmins can configure:\n- Token limits per user/project\n- Request rate limits\n- Cost alerts\n\n## Best Practices\n\n### 1. Use Endpoint Names Consistently\n```python\n# Define endpoint once\nLLM_ENDPOINT = \"production-gpt4\"\n\n# Use throughout code\nresponse = client.chat.completions.create(\n    model=LLM_ENDPOINT,\n    messages=[...]\n)\n```\n\n### 2. Handle Rate Limits\n```python\nimport time\nfrom openai import RateLimitError\n\ndef call_llm_with_retry(messages, max_retries=3):\n    for attempt in range(max_retries):\n        try:\n            return client.chat.completions.create(\n                model=\"openai-gpt4\",\n                messages=messages\n            )\n        except RateLimitError:\n            if attempt < max_retries - 1:\n                time.sleep(2 ** attempt)\n            else:\n                raise\n```\n\n### 3. Log Important Calls\n```python\nimport logging\n\nlogger = logging.getLogger(__name__)\n\ndef query_llm(prompt):\n    logger.info(f\"Querying LLM with prompt length: {len(prompt)}\")\n    response = client.chat.completions.create(\n        model=\"openai-gpt4\",\n        messages=[{\"role\": \"user\", \"content\": prompt}]\n    )\n    logger.info(f\"Response tokens: {response.usage.total_tokens}\")\n    return response.choices[0].message.content\n```\n\n### 4. Use Streaming for Long Responses\n```python\n# Streaming response\nstream = client.chat.completions.create(\n    model=\"openai-gpt4\",\n    messages=[{\"role\": \"user\", \"content\": \"Write a long story\"}],\n    stream=True\n)\n\nfor chunk in stream:\n    if chunk.choices[0].delta.content:\n        print(chunk.choices[0].delta.content, end=\"\")\n```\n\n## Security\n\n### API Key Management\n- Keys stored in Domino's secure vault\n- Never exposed to end users\n- Rotatable without code changes\n\n### Data Privacy\n- Requests logged for audit\n- PII can be masked (admin config)\n- Data retention policies configurable\n\n## Troubleshooting\n\n### Authentication Error\n```\nError: 401 Unauthorized\n```\n- Verify Domino API key is valid\n- Check endpoint access permissions\n- Ensure correct base URL\n\n### Rate Limit Exceeded\n```\nError: 429 Too Many Requests\n```\n- Implement retry logic\n- Contact admin for limit increase\n- Use multiple endpoints for load distribution\n\n### Model Not Found\n```\nError: Model 'model-name' not found\n```\n- Verify endpoint name is correct\n- Check endpoint exists and is active\n- Confirm you have access to the endpoint\n\n## API Reference\n\nBefore writing or verifying any API call, use the cluster swagger to confirm current endpoint paths and field names. Use public docs for workflow context and field explanations.\n\n**Get the cluster base URL:** `$DOMINO_API_HOST` (injected by Domino into every workspace, job, and app).\n\nFetch the swagger spec:\n```bash\n# No authentication required for the public API spec\ncurl \"$DOMINO_API_HOST/assets/public-api.json\"\n# Browser UI: $DOMINO_API_HOST/assets/lib/swagger-ui/index.html?url=/assets/public-api.json#/\n```\n\n**Public docs (workflow context and field explanations):**\n- [API Guide](https://docs.dominodatalab.com/en/latest/api_guide/f35c19/api-guide/)\n- [AI Gateway](https://docs.dominodatalab.com/en/latest/user_guide/c9ac47/ai-gateway/)\n- [Monitor AI Gateway LLM logs](https://docs.dominodatalab.com/en/cloud/admin_guide/984c09/monitor-ai-gateway-large-language-model-llm-logs/)\n"},"changes":[],"summary":"First saved snapshot. No earlier version is available for comparison.","summary_kind":"deterministic","summary_metadata":{}}