Update to Twilio Developer Kit

{
  "name": "twilio-reliability-patterns",
  "description": "Handle rate limits, retries, and failures when building on Twilio at scale. Covers 429 exponential backoff with jitter, per-number throughput limits, StatusCallback resilience, thin-receiver pattern, and fallback chains. Use this skill whenever sending messages or making calls at volume, or when building production-grade Twilio integrations.",
  "included_files": [
    {
      "relative_path": "agents/openai.yaml",
      "size_in_bytes": 235
    }
  ],
  "skill_md_contents": "---\nname: twilio-reliability-patterns\ndescription: >\n  Handle rate limits, retries, and failures when building on Twilio at\n  scale. Covers 429 exponential backoff with jitter, per-number throughput\n  limits, StatusCallback resilience, thin-receiver pattern, and fallback\n  chains. Use this skill whenever sending messages or making calls at\n  volume, or when building production-grade Twilio integrations.\n---\n\n## Overview\n\nTwilio enforces per-resource rate limits. At scale, 429 errors are expected behavior — not bugs. This skill teaches the patterns that prevent production failures: exponential backoff, throughput management, and resilient callback handling.\n\n429 concurrency errors are not well documented — implement exponential backoff with ±10% jitter.\n\n---\n\n## Prerequisites\n\n- A working Twilio integration (any product)\n- Understanding of your expected volume (messages/sec, calls/sec)\n- StatusCallback URLs configured — see `twilio-messaging-services`, `twilio-sms-send-message`\n\n---\n\n## Key Patterns\n\n### 1. Exponential Backoff with Jitter\n\nWhen you receive a 429 (Too Many Requests), wait and retry. Naive fixed-interval retry creates thundering herds. Use exponential backoff with randomized jitter.\n\n**Python**\n```python\nimport time, random, requests\n\ndef send_with_backoff(client, to, body, messaging_service_sid, max_retries=5):\n    for attempt in range(max_retries):\n        try:\n            message = client.messages.create(\n                to=to,\n                body=body,\n                messaging_service_sid=messaging_service_sid,\n                status_callback=\"https://yourapp.com/status\"\n            )\n            return message\n        except Exception as e:\n            if hasattr(e, 'status') and e.status == 429:\n                # Exponential backoff: 100ms, 200ms, 400ms, 800ms, 1600ms\n                base_delay = 0.1 * (2 ** attempt)\n                # Add ±10% jitter to prevent thundering herd\n                jitter = base_delay * 0.1 * (2 * random.random() - 1)\n                delay = min(base_delay + jitter, 30)  # cap at 30 seconds\n                time.sleep(delay)\n            else:\n                raise  # Non-429 errors: don't retry, investigate\n    raise Exception(f\"Failed after {max_retries} retries\")\n```\n\n**Node.js**\n```node\nasync function sendWithBackoff(client, to, body, messagingServiceSid, maxRetries = 5) {\n    for (let attempt = 0; attempt < maxRetries; attempt++) {\n        try {\n            return await client.messages.create({\n                to,\n                body,\n                messagingServiceSid,\n                statusCallback: \"https://yourapp.com/status\",\n            });\n        } catch (err) {\n            if (err.status === 429) {\n                // Exponential backoff: 100ms, 200ms, 400ms, 800ms, 1600ms\n                const baseDelay = 100 * Math.pow(2, attempt);\n                // Add ±10% jitter\n                const jitter = baseDelay * 0.1 * (2 * Math.random() - 1);\n                const delay = Math.min(baseDelay + jitter, 30000); // cap at 30s\n                await new Promise(r => setTimeout(r, delay));\n            } else {\n                throw err; // Non-429: don't retry\n            }\n        }\n    }\n    throw new Error(`Failed after ${maxRetries} retries`);\n}\n```\n\n**Parameters:**\n- Initial delay: 100ms\n- Multiplier: 2x per attempt\n- Jitter: ±10% of base delay (randomized)\n- Max delay: 30 seconds\n- Max retries: 5 (covers up to ~3.2 second base delay)\n\n### 2. Per-Number Throughput Limits\n\nThese limits are not prominently documented:\n\n| Number type | SMS throughput | Voice throughput | Notes |\n|-------------|---------------|-----------------|-------|\n| Local (long code) | ~1 SMS/sec | 1 concurrent call | Lowest cost, lowest throughput |\n| Toll-free | ~3 SMS/sec | — | Faster verification (3-5 days) |\n| Short code | 10-100 SMS/sec | — | Highest throughput, 8-12 week provisioning, expensive |\n| Messaging Service (pool) | Sum of all numbers in pool | — | Multiply throughput by adding numbers |\n\n**Throughput opacity:** Sending velocity and queue depth are opaque — there is no dashboard showing messages per second. Use Messaging Services to multiply throughput by pooling numbers. A pool of 10 long codes = ~10 SMS/sec.\n\n### 3. Bulk Send Pattern\n\nFor sending to large lists, use a rate-limited dispatch loop:\n\n**Python**\n```python\nimport asyncio\nfrom collections import deque\n\nasync def bulk_send(client, recipients, body, messaging_service_sid, rate_per_second=10):\n    \"\"\"Send to a list of recipients with rate limiting and backoff.\"\"\"\n    queue = deque(recipients)\n    results = []\n    \n    while queue:\n        batch = []\n        for _ in range(min(rate_per_second, len(queue))):\n            batch.append(queue.popleft())\n        \n        for recipient in batch:\n            try:\n                msg = send_with_backoff(client, recipient, body, messaging_service_sid)\n                results.append({\"to\": recipient, \"sid\": msg.sid, \"status\": \"sent\"})\n            except Exception as e:\n                results.append({\"to\": recipient, \"error\": str(e), \"status\": \"failed\"})\n        \n        if queue:  # Don't sleep after last batch\n            await asyncio.sleep(1)  # 1 second between batches\n    \n    return results\n```\n\n**Key:** Set `rate_per_second` based on your number pool size, not your desired speed. Sending faster than your pool supports just generates 429s.\n\n> **Compliance:** Before bulk sending, verify recipient consent (opt-in records), respect quiet hours, and implement maximum batch size limits. Monitor for anomalous send patterns that could indicate abuse.\n\n### 4. StatusCallback Resilience\n\nAt scale, StatusCallbacks create their own load problem.\n\n**The math:** 50 concurrent calls × 6 status events per call = 300 webhook invocations per second. Twilio Functions allow 30 concurrent executions per service.\n\n**Thin-receiver pattern** — receive, queue, respond immediately:\n\n**Node.js (Express)**\n```node\nconst { Queue } = require(\"bullmq\");\nconst statusQueue = new Queue(\"twilio-status\");\n\n// Thin receiver: accept callback, queue it, respond 200 immediately\napp.post(\"/status\", async (req, res) => {\n    await statusQueue.add(\"status-event\", {\n        callSid: req.body.CallSid,\n        callStatus: req.body.CallStatus,\n        timestamp: Date.now(),\n    });\n    res.sendStatus(200);  // Respond FAST — Twilio will retry on timeout\n});\n\n// Process asynchronously\nconst worker = new Worker(\"twilio-status\", async (job) => {\n    const { callSid, callStatus } = job.data;\n    await updateDatabase(callSid, callStatus);\n});\n```\n\n**Python (Flask + Celery)**\n```python\n@app.route(\"/status\", methods=[\"POST\"])\ndef status_callback():\n    # Queue for async processing\n    process_status.delay(\n        call_sid=request.form[\"CallSid\"],\n        call_status=request.form[\"CallStatus\"]\n    )\n    return \"\", 200  # Respond FAST\n\n@celery.task\ndef process_status(call_sid, call_status):\n    update_database(call_sid, call_status)\n```\n\n**Idempotency key:** Use `{CallSid}-{CallStatus}` as a composite key. Twilio retries on timeout, which can cause duplicate callbacks. Deduplicate before processing.\n\n### 5. Fallback Chains\n\nWhen delivery on one channel fails, escalate to the next:\n\n**Python**\n```python\nasync def send_with_fallback(client, to, message, messaging_service_sid):\n    \"\"\"Try SMS → Voice → Email fallback chain.\"\"\"\n    \n    # Try SMS first\n    try:\n        msg = client.messages.create(\n            to=to, body=message, messaging_service_sid=messaging_service_sid,\n            status_callback=\"https://yourapp.com/status\"\n        )\n        # Wait for delivery confirmation via StatusCallback\n        # If undelivered after timeout, fall through to voice\n        return {\"channel\": \"sms\", \"sid\": msg.sid}\n    except Exception:\n        pass  # SMS failed, try voice\n    \n    # Fallback to voice\n    try:\n        call = client.calls.create(\n            to=to, from_=\"+15551234567\",\n            twiml=f\"<Response><Say>{message}</Say></Response>\",\n            status_callback=\"https://yourapp.com/call-status\"\n        )\n        return {\"channel\": \"voice\", \"sid\": call.sid}\n    except Exception:\n        pass  # Voice failed, try email\n    \n    # Last resort: email\n    # Use SendGrid — see twilio-sendgrid-email\n    return {\"channel\": \"email\", \"status\": \"queued\"}\n```\n\n### 6. Voice Concurrency Limits\n\n| Resource | Default limit | Notes |\n|----------|--------------|-------|\n| Concurrent calls per account | 1 (trial) / variable (paid) | Request increase via support |\n| Calls per second (CPS) | 1 CPS (default) | Increase via support for outbound campaigns |\n| Conference participants | 250 per conference | |\n| Twilio Functions concurrent | 30 per service | Use thin-receiver pattern above |\n\nFor outbound campaigns, request CPS increase before launch — not during.\n\n### 7. Webhook Timeout Handling\n\nTwilio expects a response within **15 seconds** for voice webhooks and **15 seconds** for messaging webhooks. If your endpoint doesn't respond:\n- Voice: Twilio hangs up or falls back to `voiceFallbackUrl`\n- Messaging: Twilio retries the callback\n\n**Always configure fallback URLs:**\n```python\n# On phone number configuration\nnumber = client.incoming_phone_numbers(phone_sid).update(\n    voice_url=\"https://yourapp.com/voice\",\n    voice_fallback_url=\"https://yourapp.com/voice-fallback\",  # backup endpoint\n    sms_url=\"https://yourapp.com/sms\",\n    sms_fallback_url=\"https://yourapp.com/sms-fallback\"\n)\n```\n\n---\n\n## Monitoring Checklist\n\nSet up these alerts before going to production:\n\n| Metric | Alert threshold | How to track |\n|--------|----------------|-------------|\n| 429 error rate | > 5% of requests | Count 429s in your backoff handler |\n| Delivery failure rate | > 2% of messages | StatusCallback `failed`/`undelivered` events |\n| Webhook response time | > 5 seconds p95 | Your APM tool (DataDog, New Relic) |\n| Queue depth | Growing over 5 minutes | Your message queue metrics |\n| Concurrent calls | > 80% of limit | Twilio Usage API or Event Streams |\n\nTwilio's built-in alerting systems are under-used — end-users often discover issues before developers do. Configure StatusCallbacks + Event Streams for delivery failure alerts on every integration.\n\n---\n\n## CANNOT\n\n- **Cannot avoid 429 errors on any Twilio API** — Backoff patterns apply to all APIs (Messaging, Voice, Verify, Lookup)\n- **Cannot increase per-number throughput** — Add more numbers via Messaging Services instead\n- **Cannot configure StatusCallback retry behavior** — Twilio retries on timeout automatically; not configurable\n- **Cannot exceed Twilio Functions limits** — 30 concurrent executions/service, 10-second timeout, 256 MB memory\n- **Cannot use a native Twilio rate limiting API** — You must implement rate limiting in your application\n\n---\n\n## Next Steps\n\n- **Messaging at scale:** `twilio-messaging-services`\n- **Monitor delivery:** `twilio-sms-send-message` (StatusCallbacks)\n- **Debug failures:** `twilio-debugging-observability`\n- **Compliance for bulk sends:** `twilio-compliance-traffic`\n"
}