Update to Twilio Developer Kit

{
  "name": "twilio-agent-connect",
  "description": "Use when building or integrating Twilio Agent Connect (TAC) to connect third-party LLM agent runtimes with Twilio Voice, Messaging, ConversationRelay, Conversation Memory, Conversation Orchestrator, or Enterprise Knowledge.",
  "included_files": [
    {
      "relative_path": "agents/openai.yaml",
      "size_in_bytes": 221
    }
  ],
  "skill_md_contents": "---\nname: twilio-agent-connect\ndescription: >\n  Use when building or integrating Twilio Agent Connect (TAC) to connect\n  third-party LLM agent runtimes with Twilio Voice, Messaging,\n  ConversationRelay, Conversation Memory, Conversation Orchestrator, or\n  Enterprise Knowledge.\n---\n\n# Twilio Agent Connect\n\n## Overview\n\nTwilio Agent Connect (TAC) is a Python and TypeScript SDK that integrates third-party LLM agentic applications with Twilio's communication technologies. TAC provides middleware for identity resolution, memory/context management (via Conversation Memory), conversation orchestration (via Conversation Orchestrator), and multi-channel handling (Voice, SMS, RCS, WhatsApp, Chat).\n\n**Key Architecture Principle**: TAC is not an agent runtime itself—it's middleware that enables existing LLM applications (OpenAI Agents SDK, Bedrock, LangChain, Microsoft Foundry, etc.) to leverage Twilio Conversations services.\n\n## Product Context\n\n### Core Twilio Conversations Services\n\nTAC integrates with three core Twilio Conversations services:\n\n1. **Conversation Memory (Memory Store)** - Persistent user context and memory management\n   - Profile storage with traits and attributes\n   - Observation and summary storage\n   - Session history with full conversation context\n   - Identity resolution (profile lookup by phone/email)\n\n2. **Conversation Orchestrator** - Multi-channel conversation lifecycle management\n   - Unified conversation API across all channels\n   - Participant management\n   - Communication routing\n   - Conversation grouping and configuration\n\n3. **Enterprise Knowledge** - Knowledge base integration\n   - Semantic search across knowledge bases\n   - RAG (Retrieval-Augmented Generation) support\n   - Knowledge chunk retrieval with relevance scoring\n\n### Supported Channels\n\nTAC provides built-in support for:\n- **Voice** - ConversationRelay (WebSocket-based real-time voice)\n- **SMS** - Text messaging\n- **RCS** - Rich Communication Services\n- **WhatsApp** - WhatsApp Business messaging\n- **Chat** - Web chat integrations\n\nAll channels support both inbound (customer-initiated) and outbound (agent-initiated) conversations.\n\n### ConversationRelay-Only Mode\n\nTAC supports a simplified \"ConversationRelay-only\" mode for getting started with voice conversations without requiring Conversation Orchestrator or Conversation Memory setup. This mode provides:\n- TwiML generation\n- WebSocket protocol handling\n- Voice conversation lifecycle management\n- Callback-based message processing\n\n## Installation\n\n### Python SDK\n\n**Requirements**: Python 3.10+\n\n```bash\n# Using uv (recommended)\nuv add git+https://github.com/twilio/twilio-agent-connect-python.git\n\n# With server support (includes FastAPI and uvicorn for TACFastAPIServer)\nuv add git+https://github.com/twilio/twilio-agent-connect-python.git --extra server\n\n# Using pip\npip install git+https://github.com/twilio/twilio-agent-connect-python.git\npip install \"git+https://github.com/twilio/twilio-agent-connect-python.git[server]\"\n```\n\n### TypeScript SDK\n\n**Requirements**: Node.js 22.13+\n\n```bash\n# Clone and build (not yet published to npm)\ngit clone https://github.com/twilio/twilio-agent-connect-typescript.git\ncd twilio-agent-connect-typescript\nnpm install\nnpm run build\n```\n\n## Quick Start\n\n### Multi-Channel Agent with OpenAI (Python)\n\n```python\nfrom dotenv import load_dotenv\nfrom openai import AsyncOpenAI\nfrom tac import TAC, TACConfig\nfrom tac.adapters.openai import with_tac_memory\nfrom tac.channels.sms import SMSChannel\nfrom tac.channels.voice import VoiceChannel\nfrom tac.server import TACFastAPIServer\n\nload_dotenv()\n\ntac = TAC(config=TACConfig.from_env())\nvoice_channel = VoiceChannel(tac)\nsms_channel = SMSChannel(tac)\nopenai_client = AsyncOpenAI()\n\nconversation_history = {}\nSYSTEM_INSTRUCTIONS = (\n    \"You are a customer service agent speaking with a user over voice or SMS. \"\n    \"Keep responses short and conversational — a sentence or two. \"\n    \"Do not use markdown, asterisks, bullets, or emojis; your words will be \"\n    \"spoken aloud or sent as plain text.\"\n)\n\nasync def handle_message_ready(user_message, context, memory_response):\n    conv_id = context.conversation_id\n\n    if conv_id not in conversation_history:\n        conversation_history[conv_id] = []\n    conversation_history[conv_id].append({\"role\": \"user\", \"content\": user_message})\n\n    # Inject conversation memory and profile into OpenAI client\n    client = with_tac_memory(openai_client, memory_response, context)\n\n    response = await client.responses.create(\n        model=\"gpt-5.4-mini\",\n        instructions=SYSTEM_INSTRUCTIONS,\n        input=conversation_history[conv_id]\n    )\n\n    llm_response = response.output_text\n    conversation_history[conv_id].append({\"role\": \"assistant\", \"content\": llm_response})\n\n    return llm_response\n\ntac.on_message_ready(handle_message_ready)\nTACFastAPIServer(tac=tac, voice_channel=voice_channel, messaging_channels=[sms_channel]).start()\n```\n\n### Multi-Channel Agent with OpenAI (TypeScript)\n\n```typescript\nimport { config } from 'dotenv';\nimport OpenAI from 'openai';\nimport {\n  TAC,\n  TACConfig,\n  VoiceChannel,\n  SMSChannel,\n  TACServer,\n  MemoryPromptBuilder,\n} from 'twilio-agent-connect';\n\nconfig();\n\nconst openai = new OpenAI();\nconst tac = await TAC.create({ config: TACConfig.fromEnv() });\nconst voiceChannel = new VoiceChannel(tac);\nconst smsChannel = new SMSChannel(tac);\n\ntac.registerChannel(voiceChannel);\ntac.registerChannel(smsChannel);\n\nconst conversationHistory: Record<string, OpenAI.Chat.ChatCompletionMessageParam[]> = {};\n\nconst SYSTEM_INSTRUCTIONS =\n  'You are a customer service agent speaking with a user over voice or SMS. ' +\n  'Keep responses short and conversational — a sentence or two. ' +\n  'Do not use markdown, asterisks, bullets, or emojis; your words will be ' +\n  'spoken aloud or sent as plain text.';\n\ntac.onMessageReady(async ({ conversationId, message, memory, session }) => {\n  const convId = conversationId as string;\n\n  if (!conversationHistory[convId]) {\n    conversationHistory[convId] = [];\n  }\n\n  const memoryContext = MemoryPromptBuilder.build(memory, session);\n  const systemPrompt = SYSTEM_INSTRUCTIONS + (memoryContext ? `\\n\\n${memoryContext}` : '');\n\n  conversationHistory[convId].push({ role: 'user', content: message });\n\n  const response = await openai.chat.completions.create({\n    model: 'gpt-4o-mini',\n    messages: [\n      { role: 'system', content: systemPrompt },\n      ...conversationHistory[convId],\n    ],\n  });\n\n  const llmResponse = response.choices[0]?.message?.content ?? '';\n  conversationHistory[convId].push({ role: 'assistant', content: llmResponse });\n\n  return llmResponse;\n});\n\nconst server = new TACServer(tac);\nawait server.start();\n```\n\n## Configuration\n\n### Required Environment Variables\n\n```bash\n# Twilio Account Credentials\nTWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nTWILIO_AUTH_TOKEN=your_auth_token\nTWILIO_API_KEY=SKxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nTWILIO_API_SECRET=your_api_key_secret\n\n# Conversation Configuration\nTWILIO_CONVERSATION_CONFIGURATION_ID=conv_configuration_xxxx\n\n# Phone Number\nTWILIO_PHONE_NUMBER=+1234567890\n\n# Server Configuration (for Voice)\nTWILIO_VOICE_PUBLIC_DOMAIN=your-domain.ngrok.io\n```\n\n### Optional Memory Configuration\n\n```bash\n# Conversation Memory (optional)\nTWILIO_MEMORY_STORE_ID=mem_service_xxxx\nTWILIO_TRAIT_GROUPS=Contact,Preferences\n```\n\n## Cloud Platform Integrations\n\n### AWS Integration\n\n**Package**: `twilio-agent-connect-aws`\n\nConnect AWS agent services to Twilio channels:\n\n```bash\n# With Strands SDK\npip install twilio-agent-connect-aws[strands,server]\n\n# With Bedrock Agents\npip install twilio-agent-connect-aws[bedrock,server]\n\n# With Bedrock AgentCore\npip install twilio-agent-connect-aws[agentcore,server]\n```\n\n**Features**:\n- **StrandsConnector** - AWS Strands SDK integration with per-conversation agent isolation\n- **BedrockConnector** - AWS Bedrock Agents (console-created agents)\n- **BedrockAgentCoreConnector** - AWS Bedrock AgentCore (custom agent code deployment)\n\n**Repository**: https://github.com/twilio/twilio-agent-connect-aws\n\n### Microsoft/Azure Integration\n\n**Package**: `twilio-agent-connect-microsoft` (formerly `tac-azure`)\n\nConnect Microsoft Foundry agents to Twilio channels:\n\n```bash\n# With Agent Framework\npip install twilio-agent-connect-microsoft[agent-framework,server]\n\n# With Voice Live\npip install twilio-agent-connect-microsoft[voice-live,server]\n```\n\n**Features**:\n- **AgentFrameworkConnector** - Microsoft Agent Framework integration\n  - Supports Foundry Hosted Agents, Foundry Prompt Agents, Azure OpenAI (Responses API, Chat Completions)\n  - Pluggable session persistence (in-memory, file, Cosmos DB)\n  - Memory context injection and lifecycle hooks\n- **VoiceLiveConnector** - Voice Live API integration\n  - Text-in / text-streaming-out over WebSocket\n  - STT and TTS handled by Twilio ConversationRelay\n  - Native interrupt handling via Voice Live `response.cancel`\n  - Server-side conversation state management\n  - Tool execution with async handlers\n\n**Repository**: https://github.com/twilio/twilio-agent-connect-microsoft\n\n## Key Features\n\n### Memory Management\n\nAutomatic integration with Twilio Conversation Memory for persistent user context:\n- Profile retrieval with traits\n- Observation and summary storage\n- Session history with full message context\n- Automatic profile lookup by phone/email\n\n### Conversation Lifecycle\n\nAutomatic tracking of conversation sessions and state:\n- Multi-channel conversation initialization\n- Participant management\n- Conversation status tracking\n- Graceful cleanup on conversation end\n\n### Message Flow\n\n1. **Webhook/Connection Received** - Twilio sends webhook (messaging) or WebSocket connection (voice)\n2. **Channel Processing** - Channel validates and processes the incoming event\n3. **Memory Retrieval** - TAC optionally retrieves user memories and profile from Conversation Memory\n4. **Callback Invoked** - Your `on_message_ready` callback receives user message, context, and optional memory response\n5. **Response Handling** - Your callback returns a response string that TAC routes to the appropriate channel\n\n### Outbound Conversations\n\nTAC supports agent-initiated conversations across all channels:\n- Programmatic conversation creation\n- Participant addition\n- Message sending\n- Full conversation lifecycle management\n\n## Voice-Specific Features\n\n### ConversationRelay Protocol\n\nTAC handles the full ConversationRelay WebSocket protocol:\n- TwiML generation for inbound calls\n- WebSocket connection management\n- Message parsing and validation\n- Automatic conversation initialization\n- Status callback handling\n\n### Voice Live API (Microsoft Integration)\n\nThe Voice Live connector provides:\n- Text-in / text-streaming-out interface\n- STT (Speech-to-Text) handled by Twilio\n- TTS (Text-to-Speech) handled by Twilio\n- Server-side interrupt handling\n- No local session management required\n\n## Messaging-Specific Features\n\n### SMS Channel\n\n- Idempotency-based deduplication using Twilio's `i-twilio-idempotency-token` header\n- Fire-and-forget webhook processing with immediate 200 response\n- Automatic conversation initialization\n- Profile retrieval per message\n\n### Multi-Channel Support\n\nTAC provides unified handling across SMS, RCS, WhatsApp, and Chat:\n- Single `on_message_ready` callback for all channels\n- Automatic channel detection and routing\n- Per-channel response formatting\n\n## Advanced Features\n\n### Conversation Intelligence Integration\n\nProcess Conversation Intelligence operator results to create observations and summaries:\n\n```python\nfrom tac.core.config import ConversationIntelligenceConfig\n\nconfig = TACConfig.from_env()\nconfig.conversation_intelligence_config = ConversationIntelligenceConfig(\n    configuration_id=\"your_ci_configuration_id\",\n    observation_operator_sid=\"LY...\",\n    summary_operator_sid=\"LY...\",\n)\n\n@app.post(\"/ci-webhook\")\nasync def ci_webhook_handler(request: Request):\n    payload = await request.json()\n    result = await tac.process_conversation_intelligence_event(payload)\n    return result.model_dump()\n```\n\n### Custom Tools\n\nTAC provides built-in tools for common operations:\n- Memory recall\n- Knowledge base search\n- Studio Flow handoff (human escalation)\n- Message sending\n\nYou can also create custom tools using the `@function_tool` decorator:\n\n```python\nfrom tac.tools import function_tool\n\n@function_tool()\ndef send_email(recipient: str, subject: str, body: str) -> bool:\n    \"\"\"\n    Sends an email to a recipient.\n\n    Args:\n        recipient: Email address\n        subject: Email subject\n        body: Email body\n\n    Returns:\n        True on success, False on failure\n    \"\"\"\n    # Implementation here\n    return True\n```\n\n### Adapter Pattern\n\nTAC provides adapters for automatic memory injection into LLM runtimes:\n\n**Python OpenAI Adapter**:\n```python\nfrom tac.adapters.openai import with_tac_memory\n\nclient = with_tac_memory(openai_client, memory_response, context)\n# Memory and profile automatically injected into system messages\n```\n\n**TypeScript Memory Prompt Builder**:\n```typescript\nimport { MemoryPromptBuilder } from 'twilio-agent-connect';\n\nconst memoryContext = MemoryPromptBuilder.build(memory, session);\nconst systemPrompt = SYSTEM_INSTRUCTIONS + `\\n\\n${memoryContext}`;\n```\n\n## Documentation Links\n\n- **Quickstart Guide**: https://www.twilio.com/docs/platform/tac/quickstart\n- **Overview Documentation**: https://www.twilio.com/docs/platform/tac/overview\n- **Python SDK**: https://github.com/twilio/twilio-agent-connect-python\n- **TypeScript SDK**: https://github.com/twilio/twilio-agent-connect-typescript\n- **AWS Integration**: https://github.com/twilio/twilio-agent-connect-aws\n- **Microsoft Integration**: https://github.com/twilio/twilio-agent-connect-microsoft\n\n## Setup Wizard\n\nTAC includes a web-based setup wizard to automatically create required Twilio services:\n\n```bash\n# Python SDK\ngit clone https://github.com/twilio/twilio-agent-connect-python.git\ncd twilio-agent-connect-python\nmake setup  # Opens http://localhost:8080\n```\n\nThe wizard creates:\n- Conversation Memory store\n- Conversation Configuration\n- Generates `.env` file with all required credentials\n\n## Common Use Cases\n\n### Customer Support Agent\n\nBuild an AI-powered customer support agent with:\n- Multi-channel support (voice, SMS, WhatsApp)\n- Persistent customer memory and context\n- Knowledge base integration\n- Human handoff capability\n\n### Outbound Campaign Agent\n\nCreate an agent that initiates conversations:\n- Schedule outbound calls or messages\n- Personalized messaging based on customer profile\n- Conversation tracking and analytics\n\n### Voice IVR Replacement\n\nReplace traditional IVR with conversational AI:\n- Natural language understanding\n- Context-aware responses\n- Seamless handoff to human agents\n\n### Multi-Language Support\n\nBuild globally accessible agents:\n- Automatic language detection\n- Multi-language conversation memory\n- Localized responses\n\n## Best Practices\n\n### Error Handling\n\nTAC provides lenient error handling:\n- Profile lookup failures fall back to Conversation Orchestrator API\n- Memory retrieval failures continue without exceptions\n- All errors logged with appropriate severity levels\n\n### Performance Optimization\n\n- Use immediate 200 responses for webhooks to prevent retries\n- Enable conversation deduplication for high-traffic applications\n- Leverage conversation grouping for related interactions\n\n### Security\n\n- Never commit API keys or tokens to version control\n- Use environment variables for all credentials\n- Implement webhook signature validation (Twilio SDK provides helpers)\n- Use HTTPS for all webhook endpoints\n\n### Testing\n\n- Use ngrok for local webhook testing\n- Test each channel independently before multi-channel deployment\n- Implement logging for debugging webhook processing\n- Use TAC's built-in logging with channel-specific logger names\n\n## Troubleshooting\n\n### Common Issues\n\n**Memory not retrieving**:\n- Verify `TWILIO_MEMORY_STORE_ID` is set\n- Check profile_id is present in webhook data\n- Enable DEBUG logging: `TWILIO_TAC_LOG_LEVEL=DEBUG`\n\n**Voice not connecting**:\n- Verify `TWILIO_VOICE_PUBLIC_DOMAIN` is accessible\n- Check TwiML endpoint returns valid XML\n- Ensure WebSocket endpoint is reachable\n- Verify Conversation Configuration is active\n\n**Duplicate messages**:\n- Ensure webhook returns 200 immediately\n- Verify idempotency token is passed to channel\n- Check deduplication capacity is sufficient\n\n**Channel isolation issues**:\n- Verify each channel has distinct conversation sessions\n- Check `configuration_id` filtering is enabled\n- Ensure conversation status is properly tracked\n\n## Version Requirements\n\n- **Python SDK**: Python 3.10+\n- **TypeScript SDK**: Node.js 22.13+\n- **Twilio SDK**: twilio>=9.8.3\n\n## License\n\nMIT License - see repository LICENSE files for details.\n"
}