Update to Duende Skills

{
  "description": "Guide for deploying Duende IdentityServer to production, covering reverse proxy configuration, data protection, health checks, distributed caching, multi-instance deployment, OpenTelemetry integration, logging, and common deployment pitfalls.",
  "included_files": [],
  "name": "identityserver-deployment",
  "skill_md_contents": "---\nname: identityserver-deployment\ndescription: \"Guide for deploying Duende IdentityServer to production, covering reverse proxy configuration, data protection, health checks, distributed caching, multi-instance deployment, OpenTelemetry integration, logging, and common deployment pitfalls.\"\ninvocable: false\n---\n\n# IdentityServer Deployment, Proxies, and Production Readiness\n\n## When to Use This Skill\n\n- Deploying IdentityServer behind a reverse proxy or load balancer\n- Configuring ASP.NET Core Data Protection for production persistence\n- Implementing health checks for monitoring IdentityServer instances\n- Setting up distributed caching for multi-instance deployments\n- Configuring OpenTelemetry for metrics, traces, and logs\n- Troubleshooting common deployment issues (HTTPS downgrade, cookie problems, key rotation failures)\n- Understanding the difference between Data Protection keys and IdentityServer signing keys\n- Setting up logging and events for production monitoring\n\nDocs: https://docs.duendesoftware.com/identityserver/deployment\n\n## Deployment Architecture\n\nIdentityServer is ASP.NET Core middleware. It can be hosted with the same diversity of technology as any ASP.NET Core application:\n\n- **Hosting**: On-premises, cloud (Azure, AWS, GCP), containers, Kubernetes\n- **Web servers**: Kestrel, IIS, Nginx, Apache\n- **Artifacts**: Files, containers (no Dockerfile needed with `dotnet publish /t:PublishContainer`)\n- **Scaling**: Horizontal with load balancers; requires shared state for multi-instance\n\n## Reverse Proxy and Load Balancer Configuration\n\n### The Problem\n\nWhen IdentityServer runs behind a proxy that terminates TLS or changes the originating IP, the middleware sees incorrect request information. This causes:\n\n- HTTPS requests downgraded to HTTP\n- HTTP issuer published in `.well-known/openid-configuration` instead of HTTPS\n- Incorrect host names in discovery document or redirects\n- Cookies missing the `Secure` attribute (breaks `SameSite` behavior)\n\n### Solution: ForwardedHeaders Middleware\n\nMost proxies set `X-Forwarded-For` and `X-Forwarded-Proto` headers. Configure ASP.NET Core to read them.\n\n#### Option 1: Environment Variable (Simplest)\n\nSet `ASPNETCORE_FORWARDEDHEADERS_ENABLED=true`. This automatically adds the middleware and accepts forwarded headers from any single proxy. Best for cloud-hosted environments and Kubernetes.\n\n#### Option 2: Explicit Configuration (More Control)\n\n```csharp\n// Program.cs\nbuilder.Services.Configure<ForwardedHeadersOptions>(options =>\n{\n    options.ForwardedHeaders = ForwardedHeaders.XForwardedHost |\n                                ForwardedHeaders.XForwardedProto;\n\n    // Add the IP address of your known proxy\n    options.KnownProxies.Add(IPAddress.Parse(\"203.0.113.42\"));\n\n    // Or use a network range\n    // var network = new IPNetwork(IPAddress.Parse(\"198.51.100.0\"), 24);\n    // options.KnownNetworks.Add(network);\n\n    // Number of proxies in front of the app\n    options.ForwardLimit = 1;\n});\n```\n\n**Important**: The ForwardedHeaders middleware must run **early** in the pipeline, before IdentityServer middleware and ASP.NET authentication middleware.\n\n### Default KnownNetworks\n\nBy default, `KnownNetworks` and `KnownProxies` support localhost (`127.0.0.1/8` and `::1`). This is useful for local development or when the proxy and .NET host are on the same machine. In production, configure the actual proxy addresses.\n\n## ASP.NET Core Data Protection\n\n> **Cross-cutting concern:** Data protection is critical for all Duende products — both IdentityServer and BFF. See [ASP.NET Core Data Protection](https://docs.duendesoftware.com/general/data-protection/) for comprehensive guidance covering all Duende SDKs.\n\n### Why It Matters\n\nData Protection is critical for IdentityServer. It encrypts and signs sensitive data including:\n\n- Signing keys at rest (when automatic key management is used)\n- Persisted grants at rest\n- Server-side session data at rest\n- State parameters for external OIDC providers\n- UI message payloads (logout context, error context)\n- Authentication session cookies\n- Anti-forgery tokens\n\n### Production Configuration\n\n```csharp\n// Program.cs\nbuilder.Services.AddDataProtection()\n    // Choose a persistence method\n    .PersistKeysToFoo()       // PersistKeysToFileSystem, PersistKeysToDbContext,\n                               // PersistKeysToAzureBlobStorage, PersistKeysToAWSSystemsManager,\n                               // PersistKeysToStackExchangeRedis\n    // Choose a key protection method\n    .ProtectKeysWithBar()     // ProtectKeysWithCertificate, ProtectKeysWithAzureKeyVault\n    // Set explicit application name\n    .SetApplicationName(\"My.IdentityServer\");\n```\n\n### Critical Rules\n\n1. **Always persist keys to durable storage** using a `.PersistKeysTo...()` method\n2. **Ensure the storage itself is durable** — e.g., if using Redis, configure Redis persistence (RDB/AOF)\n3. **Always set an explicit application name** with `.SetApplicationName()` to prevent key isolation issues\n4. **Share keys across all load-balanced instances**\n5. **Consider a key escrow sink** — for backup/restore of corrupted data protection keys, configure an `IXmlEncryptor`-based escrow\n\n### Data Protection Keys vs Signing Keys\n\n| Aspect       | Data Protection Keys                               | IdentityServer Signing Keys                                               |\n| ------------ | -------------------------------------------------- | ------------------------------------------------------------------------- |\n| Purpose      | Encrypt/sign sensitive data at rest and in cookies | Sign JWT tokens (id_tokens, access tokens)                                |\n| Cryptography | Symmetric (private key)                            | Asymmetric (public/private key pair)                                      |\n| Visibility   | Internal to the application                        | Public keys published via discovery/JWKS                                  |\n| Managed by   | ASP.NET Core framework                             | IdentityServer (automatic key management)                                 |\n| Storage      | Configured via `.PersistKeysTo...()`               | File system (default), EF operational store, or custom `ISigningKeyStore` |\n\nBoth are critical secrets. Losing either causes failures.\n\n### Common Data Protection Problems\n\n| Problem                                     | Symptom                                                 | Solution                                                  |\n| ------------------------------------------- | ------------------------------------------------------- | --------------------------------------------------------- |\n| No shared keys in load-balanced environment | `CryptographicException`: key not found in key ring     | Configure shared key persistence                          |\n| Keys generated in dev included in build     | Keys from wrong environment can't be read in production | Exclude `~/keys` directory from source control and builds |\n| Application name mismatch                   | Keys from one deployment can't be read by another       | Set explicit `SetApplicationName()` consistently          |\n| IIS lacking permissions                     | Ephemeral keys generated every restart                  | Follow Microsoft's IIS Data Protection configuration      |\n| .NET 6 path normalization change            | Keys break between .NET versions                        | Always set explicit application name (reverted in .NET 7+) |\n\n### Symptoms of Data Protection Failure\n\n- `CryptographicException` in logs\n- Error messages like \"Error unprotecting key with kid {Signing Key ID}\"\n- \"The key {Data Protection Key ID} was not found in the key ring\"\n- Automatic signing key management fails silently\n\n## IdentityServer Data Stores for Multi-Instance\n\n### Configuration Data\n\nFor multi-instance deployments, configuration data must be shared:\n\n| Scenario                      | Recommendation                                                        |\n| ----------------------------- | --------------------------------------------------------------------- |\n| Rarely changing configuration | In-memory stores loaded from config files (with redeploy for changes) |\n| Dynamic configuration (SaaS)  | Database via EF Core stores or custom stores                          |\n\n### Operational Data\n\nOperational data must always be shared in multi-instance deployments:\n\n- Authorization codes, tokens, consent — via persisted grant store\n- Signing keys — via `ISigningKeyStore` (EF operational store or custom)\n- Server-side sessions — via `IServerSideSessionStore`\n\nUse Entity Framework Core or a persistent cache like Redis.\n\n## Distributed Caching\n\nSome optional features require ASP.NET Core's `IDistributedCache`:\n\n| Feature                       | Why It Needs Distributed Cache                               |\n| ----------------------------- | ------------------------------------------------------------ |\n| OIDC state data formatter     | Stores external provider state server-side instead of in URL |\n| JWT replay cache              | Prevents JWT client credentials replay                       |\n| Device flow throttling        | Rate-limits polling across instances                         |\n| Authorization parameter store | Stores PAR request data                                      |\n\nConfigure a distributed cache for multi-instance deployments:\n\n```csharp\n// Program.cs — Example using Redis\nbuilder.Services.AddStackExchangeRedisCache(options =>\n{\n    options.Configuration = \"localhost:6379\";\n});\n```\n\n## Health Checks\n\n### Discovery Endpoint Health Check\n\nTests that IdentityServer can process requests and communicate with the configuration store:\n\n```csharp\npublic class DiscoveryHealthCheck : IHealthCheck\n{\n    private readonly IEnumerable<Hosting.Endpoint> _endpoints;\n    private readonly IHttpContextAccessor _httpContextAccessor;\n\n    public DiscoveryHealthCheck(IEnumerable<Hosting.Endpoint> endpoints,\n        IHttpContextAccessor httpContextAccessor)\n    {\n        _endpoints = endpoints;\n        _httpContextAccessor = httpContextAccessor;\n    }\n\n    public async Task<HealthCheckResult> CheckHealthAsync(\n        HealthCheckContext context,\n        CancellationToken cancellationToken = default)\n    {\n        try\n        {\n            var endpoint = _endpoints.FirstOrDefault(\n                x => x.Name == IdentityServerConstants.EndpointNames.Discovery);\n            if (endpoint != null)\n            {\n                var handler = _httpContextAccessor.HttpContext.RequestServices\n                    .GetRequiredService(endpoint.Handler) as IEndpointHandler;\n                if (handler != null)\n                {\n                    var result = await handler.ProcessAsync(\n                        _httpContextAccessor.HttpContext);\n                    if (result is DiscoveryDocumentResult)\n                    {\n                        return HealthCheckResult.Healthy();\n                    }\n                }\n            }\n        }\n        catch { }\n\n        return new HealthCheckResult(context.Registration.FailureStatus);\n    }\n}\n```\n\n### JWKS Health Check\n\nTests that IdentityServer can access its signing keys:\n\n```csharp\npublic class DiscoveryKeysHealthCheck : IHealthCheck\n{\n    private readonly IEnumerable<Hosting.Endpoint> _endpoints;\n    private readonly IHttpContextAccessor _httpContextAccessor;\n\n    public DiscoveryKeysHealthCheck(IEnumerable<Hosting.Endpoint> endpoints,\n        IHttpContextAccessor httpContextAccessor)\n    {\n        _endpoints = endpoints;\n        _httpContextAccessor = httpContextAccessor;\n    }\n\n    public async Task<HealthCheckResult> CheckHealthAsync(\n        HealthCheckContext context,\n        CancellationToken cancellationToken = default)\n    {\n        try\n        {\n            var endpoint = _endpoints.FirstOrDefault(\n                x => x.Name == IdentityServerConstants.EndpointNames.Jwks);\n            if (endpoint != null)\n            {\n                var handler = _httpContextAccessor.HttpContext.RequestServices\n                    .GetRequiredService(endpoint.Handler) as IEndpointHandler;\n                if (handler != null)\n                {\n                    var result = await handler.ProcessAsync(\n                        _httpContextAccessor.HttpContext);\n                    if (result is JsonWebKeysResult)\n                    {\n                        return HealthCheckResult.Healthy();\n                    }\n                }\n            }\n        }\n        catch { }\n\n        return new HealthCheckResult(context.Registration.FailureStatus);\n    }\n}\n```\n\n**Note**: Finding endpoints by name requires IdentityServer v6.3+.\n\n## OpenTelemetry Integration\n\nIdentityServer emits traces, metrics, and logs via the .NET OpenTelemetry SDK (added in v6.1, expanded in v7.0).\n\n### Setup\n\n```bash\ndotnet add package OpenTelemetry\ndotnet add package OpenTelemetry.Extensions.Hosting\ndotnet add package OpenTelemetry.Instrumentation.AspNetCore\ndotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol\n```\n\n```csharp\n// Program.cs\nusing OpenTelemetry.Resources;\n\n// Add OpenTelemetry logging to correlate logs with traces\nbuilder.Logging.AddOpenTelemetry();\n\nvar openTelemetry = builder.Services.AddOpenTelemetry();\n\nopenTelemetry.ConfigureResource(r => r\n    .AddService(builder.Environment.ApplicationName));\n\nopenTelemetry.WithMetrics(m => m\n    .AddMeter(\"Duende.IdentityServer\")   // Telemetry.ServiceName == \"Duende.IdentityServer\"\n    .AddPrometheusExporter());\n\nopenTelemetry.WithTracing(t => t\n    .AddSource(IdentityServerConstants.Tracing.Basic)\n    .AddSource(IdentityServerConstants.Tracing.Cache)\n    .AddSource(IdentityServerConstants.Tracing.Services)\n    .AddSource(IdentityServerConstants.Tracing.Stores)\n    .AddSource(IdentityServerConstants.Tracing.Validation)\n    .AddAspNetCoreInstrumentation()\n    .AddConsoleExporter());\n\n// Add Prometheus scraping endpoint\napp.UseOpenTelemetryPrometheusScrapingEndpoint();\n```\n\n### Tracing Sources\n\n| Source                                       | What It Traces                                                  |\n| -------------------------------------------- | --------------------------------------------------------------- |\n| `IdentityServerConstants.Tracing.Basic`      | High-level request processing (validators, response generators) |\n| `IdentityServerConstants.Tracing.Cache`      | Cache operations                                                |\n| `IdentityServerConstants.Tracing.Services`   | Service-layer operations                                        |\n| `IdentityServerConstants.Tracing.Stores`     | Store operations (database calls)                               |\n| `IdentityServerConstants.Tracing.Validation` | Detailed validation operations                                  |\n\nIn production, you may want only `Basic` tracing. Use all sources during development and troubleshooting.\n\n### Key Metrics (v7.0+)\n\nThe meter name is `Duende.IdentityServer` (accessible via `Telemetry.ServiceName`).\n\n| Metric          | Counter Name                            | Description                                      |\n| --------------- | --------------------------------------- | ------------------------------------------------ |\n| Operations      | `tokenservice.operation`                | Aggregated success/failure/internal_error counts |\n| Active Requests | `active_requests`                       | Current requests being processed by endpoints    |\n| Token Issuance  | `tokenservice.token_issued`             | Successful/failed token issuance attempts        |\n| Client Auth     | `tokenservice.client.secret_validation` | Client authentication success/failure            |\n| Introspection   | `tokenservice.introspection`            | Token introspection counts                       |\n| Revocation      | `tokenservice.revocation`               | Token revocation counts                          |\n\n### UI Metrics (From Quickstart)\n\n| Metric      | Counter Name              | Tags                                    |\n| ----------- | ------------------------- | --------------------------------------- |\n| User Login  | `tokenservice.user_login` | client, idp, error                      |\n| User Logout | `user_logout`             | idp                                     |\n| Consent     | `tokenservice.consent`    | client, scope, consent (granted/denied) |\n\n## Logging\n\nIdentityServer uses ASP.NET Core's standard `ILogger`. Logs are written under the `Duende.IdentityServer` category.\n\n### Log Levels\n\n| Level         | Usage                                               |\n| ------------- | --------------------------------------------------- |\n| `Trace`       | Sensitive data (tokens); never enable in production |\n| `Debug`       | Internal flow and decisions; short-term debugging   |\n| `Information` | General application flow; long-term value           |\n| `Warning`     | Abnormal or unexpected events                       |\n| `Error`       | Failed validation, unhandled exceptions             |\n| `Critical`    | Missing store implementations, invalid key material |\n\n### Configuration\n\n```json\n{\n  \"Logging\": {\n    \"LogLevel\": {\n      \"Default\": \"Information\",\n      \"Duende.IdentityServer\": \"Information\"\n    }\n  }\n}\n```\n\nIn production, default to `Warning` to avoid excessive log volume.\n\n### Filtering Exceptions\n\n```csharp\nbuilder.Services.AddIdentityServer(options =>\n{\n    options.Logging.UnhandledExceptionLoggingFilter = (ctx, ex) =>\n    {\n        // Return false to suppress, true to log\n        if (ctx.RequestAborted.IsCancellationRequested && ex is OperationCanceledException)\n            return false; // Already the default\n        return true;\n    };\n});\n```\n\n### OpenTelemetry Log Correlation\n\nLogs written to `ILogger` in .NET 8+ can be exported to OpenTelemetry traces. Add `builder.Logging.AddOpenTelemetry()` to correlate logs with trace IDs.\n\n## Events System\n\nEvents provide higher-level structured data about operations, suitable for APM integration.\n\n### Enabling Events\n\n```csharp\nbuilder.Services.AddIdentityServer(options =>\n{\n    options.Events.RaiseSuccessEvents = true;\n    options.Events.RaiseFailureEvents = true;\n    options.Events.RaiseErrorEvents = true;\n    options.Events.RaiseInformationEvents = true;\n});\n```\n\n### Raising Events\n\n```csharp\npublic async Task<IActionResult> Login(LoginInputModel model)\n{\n    if (_users.ValidateCredentials(model.Username, model.Password))\n    {\n        var user = _users.FindByUsername(model.Username);\n        await _events.RaiseAsync(\n            new UserLoginSuccessEvent(user.Username, user.SubjectId, user.Username));\n    }\n    else\n    {\n        await _events.RaiseAsync(\n            new UserLoginFailureEvent(model.Username, \"invalid credentials\"));\n    }\n}\n```\n\n### Custom Event Sink\n\n```csharp\npublic class SeqEventSink : IEventSink\n{\n    private readonly Logger _log;\n\n    public SeqEventSink()\n    {\n        _log = new LoggerConfiguration()\n            .WriteTo.Seq(\"http://localhost:5341\")\n            .CreateLogger();\n    }\n\n    public Task PersistAsync(Event evt)\n    {\n        if (evt.EventType == EventTypes.Success ||\n            evt.EventType == EventTypes.Information)\n        {\n            _log.Information(\"{Name} ({Id}), Details: {@details}\",\n                evt.Name, evt.Id, evt);\n        }\n        else\n        {\n            _log.Error(\"{Name} ({Id}), Details: {@details}\",\n                evt.Name, evt.Id, evt);\n        }\n        return Task.CompletedTask;\n    }\n}\n```\n\nEvents work well with structured logging stores like ELK, Seq, or Splunk.\n\n## Rate Limiting\n\nDuende IdentityServer has **no built-in rate limiting**. Assess it for public-facing or multi-tenant deployments. Three combinable approaches:\n\n### (a) Network Layer (first line of defense)\n\nReverse proxy / gateway (nginx, Azure Application Gateway, AWS API Gateway, Cloudflare). Partitions only by IP/path — coarse, but stops most volumetric abuse before it reaches the app.\n\n### (b) ASP.NET Core Rate Limiting Middleware\n\nRegister **before** `app.UseIdentityServer()`:\n\n```csharp\napp.UseRateLimiter();\napp.UseIdentityServer();\n```\n\n**Critical caveat:** IdentityServer matches its protocol endpoints (`/connect/authorize`, `/connect/token`, …) with its **own middleware, NOT ASP.NET Core endpoint routing**. You therefore **cannot attach a named per-endpoint policy** to protocol endpoints — only the **GLOBAL limiter** applies to them.\n\n- Approximate per-endpoint limits by **partitioning the global limiter on `context.Request.Path`**.\n- Named policies (`RequireRateLimiting(\"...\")`) still work on **your own routed Razor Pages** (login/consent).\n- For the token endpoint, prefer returning a **JSON error + `Retry-After` header** rather than an HTML 429.\n\n```csharp\nbuilder.Services.AddRateLimiter(options =>\n{\n    // Global limiter — the ONLY limiter that applies to protocol endpoints\n    options.GlobalLimiter = PartitionedRateLimiter.Create<HttpContext, string>(context =>\n    {\n        var ip = context.Connection.RemoteIpAddress?.ToString() ?? \"unknown\";\n        // Partition on path to approximate per-endpoint limits\n        return RateLimitPartition.GetSlidingWindowLimiter(\n            partitionKey: $\"{ip}:{context.Request.Path}\",\n            factory: _ => new SlidingWindowRateLimiterOptions\n            {\n                PermitLimit = 20,\n                Window = TimeSpan.FromMinutes(1),\n                SegmentsPerWindow = 4\n            });\n    });\n});\n```\n\n### (c) Identity-Aware Custom Validator\n\nImplement `ICustomTokenRequestValidator` — it runs **after** token request validation, so `ClientId`/user are known:\n\n```csharp\npublic class RateLimitingTokenRequestValidator : ICustomTokenRequestValidator\n{\n    // v8 added the CancellationToken parameter to this interface\n    public Task ValidateAsync(CustomTokenRequestValidationContext context, CancellationToken ct)\n    {\n        var clientId = context.Result.ValidatedRequest.ClientId;\n        if (IsOverLimit(clientId))\n        {\n            context.Result.IsError = true;\n            context.Result.Error = \"rate_limited\";\n            context.Result.ErrorDescription = \"Too many requests\";\n        }\n        return Task.CompletedTask;\n    }\n}\n\n// idsvrBuilder.AddCustomTokenRequestValidator<RateLimitingTokenRequestValidator>();\n```\n\n**Note:** It runs **after** client authentication, secret validation, and DB lookups — so pair it with a coarser layer (a or b) to shed load earlier.\n\n## Production Readiness Checklist\n\n| Item                                                  | Status                                 | Notes                                   |\n| ----------------------------------------------------- | -------------------------------------- | --------------------------------------- |\n| Data Protection keys persisted to durable storage     | Required                               | `.PersistKeysTo...()`                   |\n| Data Protection keys shared across instances          | Required for multi-instance            | Same storage for all instances          |\n| Explicit application name set                         | Required                               | `.SetApplicationName(\"My.IdentityServer\")` |\n| ForwardedHeaders configured (if behind proxy)         | Required                               | Match your proxy's headers              |\n| Operational store configured with durable persistence | Required                               | EF Core or custom store                 |\n| Token cleanup enabled                                 | Recommended                            | `EnableTokenCleanup = true`             |\n| Configuration store cache enabled                     | Recommended                            | `AddConfigurationStoreCache()`          |\n| Distributed cache configured (if multi-instance)      | Recommended                            | Redis, SQL, etc.                        |\n| Health checks implemented                             | Recommended                            | Discovery + JWKS endpoints              |\n| OpenTelemetry configured                              | Recommended                            | Metrics + traces for monitoring         |\n| Events enabled                                        | Recommended                            | For auditing and APM                    |\n| Signing key store uses durable storage                | Required for multi-instance            | EF operational store or custom          |\n| Logging level set to Warning+ for production          | Recommended                            | Avoid log bloat                         |\n| `~/keys` directory excluded from source control       | Required if using file-based key store | Prevent dev keys in production          |\n| HTTPS + ForwardedHeaders configured before IdentityServer | Required if behind proxy           | Discovery must publish HTTPS issuer     |\n| Signing keys shared by all instances + rotation plan  | Required for multi-instance            | Automatic Key Management where available |\n| DB schema changes applied before new app version starts | Required                             | Plus operational-store cleanup enabled  |\n| Same operational data / signing keys / DP keys / caches per instance | Required for multi-instance | Every instance shares all shared state  |\n| CORS allows only required client origins              | Required                               | Watch middleware order                  |\n| Token + session lifetimes match threat model          | Recommended                            | Tune per deployment                     |\n| Rate limiting assessed                                | Recommended                            | Public / multi-tenant deployments       |\n\n## Common Anti-Patterns\n\n- ❌ Deploying without configuring ForwardedHeaders behind a reverse proxy\n- ✅ Always configure ForwardedHeaders when behind a proxy; test by checking the discovery document's issuer URL\n\n- ❌ Using default (ephemeral) Data Protection keys in production\n- ✅ Always persist keys to durable, shared storage with `.PersistKeysTo...()`\n\n- ❌ Not setting `SetApplicationName()` causing key isolation between deployments\n- ✅ Always set an explicit, consistent application name\n\n- ❌ Using file-system signing key store in containerized/multi-instance deployments\n- ✅ Use EF operational store or a shared `ISigningKeyStore` implementation\n\n- ❌ Enabling `Trace` or `Debug` logging in production — exposes tokens and sensitive data\n- ✅ Use `Warning` level in production; use `Information` temporarily for troubleshooting\n\n- ❌ Not enabling token cleanup — database grows indefinitely\n- ✅ Enable `EnableTokenCleanup = true` and configure appropriate intervals\n\n## Common Pitfalls\n\n1. **Discovery document shows HTTP issuer**: The most common deployment issue. Always configure ForwardedHeaders or the `ASPNETCORE_FORWARDEDHEADERS_ENABLED` environment variable when behind a TLS-terminating proxy.\n\n2. **CryptographicException on startup**: Usually means Data Protection keys from one environment are being used in another. Check that keys are persisted correctly and the application name is consistent.\n\n3. **Signing keys not shared across instances**: The default file-system key store is per-instance. Use `AddOperationalStore()` which includes `ISigningKeyStore`, or configure a custom shared store.\n\n4. **Redis losing Data Protection keys on restart**: If using `PersistKeysToStackExchangeRedis`, configure Redis with persistence (RDB snapshots or AOF) to survive restarts.\n\n5. **IIS Data Protection permissions**: IIS may lack permissions to persist Data Protection keys. Follow Microsoft's IIS-specific Data Protection documentation.\n\n6. **Multiple proxies in chain**: If you have more than one proxy, set `ForwardLimit` to match the number of proxies, and add all proxy addresses to `KnownProxies` or `KnownNetworks`.\n\n7. **Cookie SameSite failures behind proxy**: If the proxy strips HTTPS, cookies won't get the `Secure` attribute, causing `SameSite=None` cookies to be rejected by browsers. Fix the proxy configuration first.\n\n8. **OpenTelemetry trace source selection**: In production, subscribing to all trace sources (`Stores`, `Validation`, etc.) can generate excessive trace data. Start with `Basic` and add more sources as needed for troubleshooting.\n\n9. **v8 license key format / runtime enforcement**: The v8 license key is a signed JWT with a `kid` header. A v7-format key still runs v8 core, but a v8 key fails on v7/earlier or the BFF runtime with `IDX10503: ... Token does not have a kid.` v8 also **throws at startup** when a configured license lacks the entitlement for Server-Side Sessions, Automatic Key Management, or SAML — run lower environments with the production key so gaps surface before production.\n\n---\n\n## Related Skills\n\n- `identityserver-hosting-setup` — DI registration and middleware pipeline\n- `identityserver-data-storage` — EF Core stores, migrations, token cleanup\n- `identityserver-aspire` — orchestrating IdentityServer in Aspire AppHost\n"
}