← Files Duende SkillsARCHIVED FILE
skills/identityserver-deployment/SKILL.md
28.5 KB · Oct 2, 2026 · 00:31 UTC
---
name: identityserver-deployment
description: "Guide for deploying Duende IdentityServer to production, covering reverse proxy configuration, data protection, health checks, distributed caching, multi-instance deployment, OpenTelemetry integration, logging, and common deployment pitfalls."
invocable: false
---
# IdentityServer Deployment, Proxies, and Production Readiness
## When to Use This Skill
- Deploying IdentityServer behind a reverse proxy or load balancer
- Configuring ASP.NET Core Data Protection for production persistence
- Implementing health checks for monitoring IdentityServer instances
- Setting up distributed caching for multi-instance deployments
- Configuring OpenTelemetry for metrics, traces, and logs
- Troubleshooting common deployment issues (HTTPS downgrade, cookie problems, key rotation failures)
- Understanding the difference between Data Protection keys and IdentityServer signing keys
- Setting up logging and events for production monitoring
Docs: https://docs.duendesoftware.com/identityserver/deployment
## Deployment Architecture
IdentityServer is ASP.NET Core middleware. It can be hosted with the same diversity of technology as any ASP.NET Core application:
- **Hosting**: On-premises, cloud (Azure, AWS, GCP), containers, Kubernetes
- **Web servers**: Kestrel, IIS, Nginx, Apache
- **Artifacts**: Files, containers (no Dockerfile needed with `dotnet publish /t:PublishContainer`)
- **Scaling**: Horizontal with load balancers; requires shared state for multi-instance
## Reverse Proxy and Load Balancer Configuration
### The Problem
When IdentityServer runs behind a proxy that terminates TLS or changes the originating IP, the middleware sees incorrect request information. This causes:
- HTTPS requests downgraded to HTTP
- HTTP issuer published in `.well-known/openid-configuration` instead of HTTPS
- Incorrect host names in discovery document or redirects
- Cookies missing the `Secure` attribute (breaks `SameSite` behavior)
### Solution: ForwardedHeaders Middleware
Most proxies set `X-Forwarded-For` and `X-Forwarded-Proto` headers. Configure ASP.NET Core to read them.
#### Option 1: Environment Variable (Simplest)
Set `ASPNETCORE_FORWARDEDHEADERS_ENABLED=true`. This automatically adds the middleware and accepts forwarded headers from any single proxy. Best for cloud-hosted environments and Kubernetes.
#### Option 2: Explicit Configuration (More Control)
```csharp
// Program.cs
builder.Services.Configure<ForwardedHeadersOptions>(options =>
{
options.ForwardedHeaders = ForwardedHeaders.XForwardedHost |
ForwardedHeaders.XForwardedProto;
// Add the IP address of your known proxy
options.KnownProxies.Add(IPAddress.Parse("203.0.113.42"));
// Or use a network range
// var network = new IPNetwork(IPAddress.Parse("198.51.100.0"), 24);
// options.KnownNetworks.Add(network);
// Number of proxies in front of the app
options.ForwardLimit = 1;
});
```
**Important**: The ForwardedHeaders middleware must run **early** in the pipeline, before IdentityServer middleware and ASP.NET authentication middleware.
### Default KnownNetworks
By default, `KnownNetworks` and `KnownProxies` support localhost (`127.0.0.1/8` and `::1`). This is useful for local development or when the proxy and .NET host are on the same machine. In production, configure the actual proxy addresses.
## ASP.NET Core Data Protection
> **Cross-cutting concern:** Data protection is critical for all Duende products — both IdentityServer and BFF. See [ASP.NET Core Data Protection](https://docs.duendesoftware.com/general/data-protection/) for comprehensive guidance covering all Duende SDKs.
### Why It Matters
Data Protection is critical for IdentityServer. It encrypts and signs sensitive data including:
- Signing keys at rest (when automatic key management is used)
- Persisted grants at rest
- Server-side session data at rest
- State parameters for external OIDC providers
- UI message payloads (logout context, error context)
- Authentication session cookies
- Anti-forgery tokens
### Production Configuration
```csharp
// Program.cs
builder.Services.AddDataProtection()
// Choose a persistence method
.PersistKeysToFoo() // PersistKeysToFileSystem, PersistKeysToDbContext,
// PersistKeysToAzureBlobStorage, PersistKeysToAWSSystemsManager,
// PersistKeysToStackExchangeRedis
// Choose a key protection method
.ProtectKeysWithBar() // ProtectKeysWithCertificate, ProtectKeysWithAzureKeyVault
// Set explicit application name
.SetApplicationName("My.IdentityServer");
```
### Critical Rules
1. **Always persist keys to durable storage** using a `.PersistKeysTo...()` method
2. **Ensure the storage itself is durable** — e.g., if using Redis, configure Redis persistence (RDB/AOF)
3. **Always set an explicit application name** with `.SetApplicationName()` to prevent key isolation issues
4. **Share keys across all load-balanced instances**
5. **Consider a key escrow sink** — for backup/restore of corrupted data protection keys, configure an `IXmlEncryptor`-based escrow
### Data Protection Keys vs Signing Keys
| Aspect | Data Protection Keys | IdentityServer Signing Keys |
| ------------ | -------------------------------------------------- | ------------------------------------------------------------------------- |
| Purpose | Encrypt/sign sensitive data at rest and in cookies | Sign JWT tokens (id_tokens, access tokens) |
| Cryptography | Symmetric (private key) | Asymmetric (public/private key pair) |
| Visibility | Internal to the application | Public keys published via discovery/JWKS |
| Managed by | ASP.NET Core framework | IdentityServer (automatic key management) |
| Storage | Configured via `.PersistKeysTo...()` | File system (default), EF operational store, or custom `ISigningKeyStore` |
Both are critical secrets. Losing either causes failures.
### Common Data Protection Problems
| Problem | Symptom | Solution |
| ------------------------------------------- | ------------------------------------------------------- | --------------------------------------------------------- |
| No shared keys in load-balanced environment | `CryptographicException`: key not found in key ring | Configure shared key persistence |
| Keys generated in dev included in build | Keys from wrong environment can't be read in production | Exclude `~/keys` directory from source control and builds |
| Application name mismatch | Keys from one deployment can't be read by another | Set explicit `SetApplicationName()` consistently |
| IIS lacking permissions | Ephemeral keys generated every restart | Follow Microsoft's IIS Data Protection configuration |
| .NET 6 path normalization change | Keys break between .NET versions | Always set explicit application name (reverted in .NET 7+) |
### Symptoms of Data Protection Failure
- `CryptographicException` in logs
- Error messages like "Error unprotecting key with kid {Signing Key ID}"
- "The key {Data Protection Key ID} was not found in the key ring"
- Automatic signing key management fails silently
## IdentityServer Data Stores for Multi-Instance
### Configuration Data
For multi-instance deployments, configuration data must be shared:
| Scenario | Recommendation |
| ----------------------------- | --------------------------------------------------------------------- |
| Rarely changing configuration | In-memory stores loaded from config files (with redeploy for changes) |
| Dynamic configuration (SaaS) | Database via EF Core stores or custom stores |
### Operational Data
Operational data must always be shared in multi-instance deployments:
- Authorization codes, tokens, consent — via persisted grant store
- Signing keys — via `ISigningKeyStore` (EF operational store or custom)
- Server-side sessions — via `IServerSideSessionStore`
Use Entity Framework Core or a persistent cache like Redis.
## Distributed Caching
Some optional features require ASP.NET Core's `IDistributedCache`:
| Feature | Why It Needs Distributed Cache |
| ----------------------------- | ------------------------------------------------------------ |
| OIDC state data formatter | Stores external provider state server-side instead of in URL |
| JWT replay cache | Prevents JWT client credentials replay |
| Device flow throttling | Rate-limits polling across instances |
| Authorization parameter store | Stores PAR request data |
Configure a distributed cache for multi-instance deployments:
```csharp
// Program.cs — Example using Redis
builder.Services.AddStackExchangeRedisCache(options =>
{
options.Configuration = "localhost:6379";
});
```
## Health Checks
### Discovery Endpoint Health Check
Tests that IdentityServer can process requests and communicate with the configuration store:
```csharp
public class DiscoveryHealthCheck : IHealthCheck
{
private readonly IEnumerable<Hosting.Endpoint> _endpoints;
private readonly IHttpContextAccessor _httpContextAccessor;
public DiscoveryHealthCheck(IEnumerable<Hosting.Endpoint> endpoints,
IHttpContextAccessor httpContextAccessor)
{
_endpoints = endpoints;
_httpContextAccessor = httpContextAccessor;
}
public async Task<HealthCheckResult> CheckHealthAsync(
HealthCheckContext context,
CancellationToken cancellationToken = default)
{
try
{
var endpoint = _endpoints.FirstOrDefault(
x => x.Name == IdentityServerConstants.EndpointNames.Discovery);
if (endpoint != null)
{
var handler = _httpContextAccessor.HttpContext.RequestServices
.GetRequiredService(endpoint.Handler) as IEndpointHandler;
if (handler != null)
{
var result = await handler.ProcessAsync(
_httpContextAccessor.HttpContext);
if (result is DiscoveryDocumentResult)
{
return HealthCheckResult.Healthy();
}
}
}
}
catch { }
return new HealthCheckResult(context.Registration.FailureStatus);
}
}
```
### JWKS Health Check
Tests that IdentityServer can access its signing keys:
```csharp
public class DiscoveryKeysHealthCheck : IHealthCheck
{
private readonly IEnumerable<Hosting.Endpoint> _endpoints;
private readonly IHttpContextAccessor _httpContextAccessor;
public DiscoveryKeysHealthCheck(IEnumerable<Hosting.Endpoint> endpoints,
IHttpContextAccessor httpContextAccessor)
{
_endpoints = endpoints;
_httpContextAccessor = httpContextAccessor;
}
public async Task<HealthCheckResult> CheckHealthAsync(
HealthCheckContext context,
CancellationToken cancellationToken = default)
{
try
{
var endpoint = _endpoints.FirstOrDefault(
x => x.Name == IdentityServerConstants.EndpointNames.Jwks);
if (endpoint != null)
{
var handler = _httpContextAccessor.HttpContext.RequestServices
.GetRequiredService(endpoint.Handler) as IEndpointHandler;
if (handler != null)
{
var result = await handler.ProcessAsync(
_httpContextAccessor.HttpContext);
if (result is JsonWebKeysResult)
{
return HealthCheckResult.Healthy();
}
}
}
}
catch { }
return new HealthCheckResult(context.Registration.FailureStatus);
}
}
```
**Note**: Finding endpoints by name requires IdentityServer v6.3+.
## OpenTelemetry Integration
IdentityServer emits traces, metrics, and logs via the .NET OpenTelemetry SDK (added in v6.1, expanded in v7.0).
### Setup
```bash
dotnet add package OpenTelemetry
dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol
```
```csharp
// Program.cs
using OpenTelemetry.Resources;
// Add OpenTelemetry logging to correlate logs with traces
builder.Logging.AddOpenTelemetry();
var openTelemetry = builder.Services.AddOpenTelemetry();
openTelemetry.ConfigureResource(r => r
.AddService(builder.Environment.ApplicationName));
openTelemetry.WithMetrics(m => m
.AddMeter("Duende.IdentityServer") // Telemetry.ServiceName == "Duende.IdentityServer"
.AddPrometheusExporter());
openTelemetry.WithTracing(t => t
.AddSource(IdentityServerConstants.Tracing.Basic)
.AddSource(IdentityServerConstants.Tracing.Cache)
.AddSource(IdentityServerConstants.Tracing.Services)
.AddSource(IdentityServerConstants.Tracing.Stores)
.AddSource(IdentityServerConstants.Tracing.Validation)
.AddAspNetCoreInstrumentation()
.AddConsoleExporter());
// Add Prometheus scraping endpoint
app.UseOpenTelemetryPrometheusScrapingEndpoint();
```
### Tracing Sources
| Source | What It Traces |
| -------------------------------------------- | --------------------------------------------------------------- |
| `IdentityServerConstants.Tracing.Basic` | High-level request processing (validators, response generators) |
| `IdentityServerConstants.Tracing.Cache` | Cache operations |
| `IdentityServerConstants.Tracing.Services` | Service-layer operations |
| `IdentityServerConstants.Tracing.Stores` | Store operations (database calls) |
| `IdentityServerConstants.Tracing.Validation` | Detailed validation operations |
In production, you may want only `Basic` tracing. Use all sources during development and troubleshooting.
### Key Metrics (v7.0+)
The meter name is `Duende.IdentityServer` (accessible via `Telemetry.ServiceName`).
| Metric | Counter Name | Description |
| --------------- | --------------------------------------- | ------------------------------------------------ |
| Operations | `tokenservice.operation` | Aggregated success/failure/internal_error counts |
| Active Requests | `active_requests` | Current requests being processed by endpoints |
| Token Issuance | `tokenservice.token_issued` | Successful/failed token issuance attempts |
| Client Auth | `tokenservice.client.secret_validation` | Client authentication success/failure |
| Introspection | `tokenservice.introspection` | Token introspection counts |
| Revocation | `tokenservice.revocation` | Token revocation counts |
### UI Metrics (From Quickstart)
| Metric | Counter Name | Tags |
| ----------- | ------------------------- | --------------------------------------- |
| User Login | `tokenservice.user_login` | client, idp, error |
| User Logout | `user_logout` | idp |
| Consent | `tokenservice.consent` | client, scope, consent (granted/denied) |
## Logging
IdentityServer uses ASP.NET Core's standard `ILogger`. Logs are written under the `Duende.IdentityServer` category.
### Log Levels
| Level | Usage |
| ------------- | --------------------------------------------------- |
| `Trace` | Sensitive data (tokens); never enable in production |
| `Debug` | Internal flow and decisions; short-term debugging |
| `Information` | General application flow; long-term value |
| `Warning` | Abnormal or unexpected events |
| `Error` | Failed validation, unhandled exceptions |
| `Critical` | Missing store implementations, invalid key material |
### Configuration
```json
{
"Logging": {
"LogLevel": {
"Default": "Information",
"Duende.IdentityServer": "Information"
}
}
}
```
In production, default to `Warning` to avoid excessive log volume.
### Filtering Exceptions
```csharp
builder.Services.AddIdentityServer(options =>
{
options.Logging.UnhandledExceptionLoggingFilter = (ctx, ex) =>
{
// Return false to suppress, true to log
if (ctx.RequestAborted.IsCancellationRequested && ex is OperationCanceledException)
return false; // Already the default
return true;
};
});
```
### OpenTelemetry Log Correlation
Logs written to `ILogger` in .NET 8+ can be exported to OpenTelemetry traces. Add `builder.Logging.AddOpenTelemetry()` to correlate logs with trace IDs.
## Events System
Events provide higher-level structured data about operations, suitable for APM integration.
### Enabling Events
```csharp
builder.Services.AddIdentityServer(options =>
{
options.Events.RaiseSuccessEvents = true;
options.Events.RaiseFailureEvents = true;
options.Events.RaiseErrorEvents = true;
options.Events.RaiseInformationEvents = true;
});
```
### Raising Events
```csharp
public async Task<IActionResult> Login(LoginInputModel model)
{
if (_users.ValidateCredentials(model.Username, model.Password))
{
var user = _users.FindByUsername(model.Username);
await _events.RaiseAsync(
new UserLoginSuccessEvent(user.Username, user.SubjectId, user.Username));
}
else
{
await _events.RaiseAsync(
new UserLoginFailureEvent(model.Username, "invalid credentials"));
}
}
```
### Custom Event Sink
```csharp
public class SeqEventSink : IEventSink
{
private readonly Logger _log;
public SeqEventSink()
{
_log = new LoggerConfiguration()
.WriteTo.Seq("http://localhost:5341")
.CreateLogger();
}
public Task PersistAsync(Event evt)
{
if (evt.EventType == EventTypes.Success ||
evt.EventType == EventTypes.Information)
{
_log.Information("{Name} ({Id}), Details: {@details}",
evt.Name, evt.Id, evt);
}
else
{
_log.Error("{Name} ({Id}), Details: {@details}",
evt.Name, evt.Id, evt);
}
return Task.CompletedTask;
}
}
```
Events work well with structured logging stores like ELK, Seq, or Splunk.
## Rate Limiting
Duende IdentityServer has **no built-in rate limiting**. Assess it for public-facing or multi-tenant deployments. Three combinable approaches:
### (a) Network Layer (first line of defense)
Reverse proxy / gateway (nginx, Azure Application Gateway, AWS API Gateway, Cloudflare). Partitions only by IP/path — coarse, but stops most volumetric abuse before it reaches the app.
### (b) ASP.NET Core Rate Limiting Middleware
Register **before** `app.UseIdentityServer()`:
```csharp
app.UseRateLimiter();
app.UseIdentityServer();
```
**Critical caveat:** IdentityServer matches its protocol endpoints (`/connect/authorize`, `/connect/token`, …) with its **own middleware, NOT ASP.NET Core endpoint routing**. You therefore **cannot attach a named per-endpoint policy** to protocol endpoints — only the **GLOBAL limiter** applies to them.
- Approximate per-endpoint limits by **partitioning the global limiter on `context.Request.Path`**.
- Named policies (`RequireRateLimiting("...")`) still work on **your own routed Razor Pages** (login/consent).
- For the token endpoint, prefer returning a **JSON error + `Retry-After` header** rather than an HTML 429.
```csharp
builder.Services.AddRateLimiter(options =>
{
// Global limiter — the ONLY limiter that applies to protocol endpoints
options.GlobalLimiter = PartitionedRateLimiter.Create<HttpContext, string>(context =>
{
var ip = context.Connection.RemoteIpAddress?.ToString() ?? "unknown";
// Partition on path to approximate per-endpoint limits
return RateLimitPartition.GetSlidingWindowLimiter(
partitionKey: $"{ip}:{context.Request.Path}",
factory: _ => new SlidingWindowRateLimiterOptions
{
PermitLimit = 20,
Window = TimeSpan.FromMinutes(1),
SegmentsPerWindow = 4
});
});
});
```
### (c) Identity-Aware Custom Validator
Implement `ICustomTokenRequestValidator` — it runs **after** token request validation, so `ClientId`/user are known:
```csharp
public class RateLimitingTokenRequestValidator : ICustomTokenRequestValidator
{
// v8 added the CancellationToken parameter to this interface
public Task ValidateAsync(CustomTokenRequestValidationContext context, CancellationToken ct)
{
var clientId = context.Result.ValidatedRequest.ClientId;
if (IsOverLimit(clientId))
{
context.Result.IsError = true;
context.Result.Error = "rate_limited";
context.Result.ErrorDescription = "Too many requests";
}
return Task.CompletedTask;
}
}
// idsvrBuilder.AddCustomTokenRequestValidator<RateLimitingTokenRequestValidator>();
```
**Note:** It runs **after** client authentication, secret validation, and DB lookups — so pair it with a coarser layer (a or b) to shed load earlier.
## Production Readiness Checklist
| Item | Status | Notes |
| ----------------------------------------------------- | -------------------------------------- | --------------------------------------- |
| Data Protection keys persisted to durable storage | Required | `.PersistKeysTo...()` |
| Data Protection keys shared across instances | Required for multi-instance | Same storage for all instances |
| Explicit application name set | Required | `.SetApplicationName("My.IdentityServer")` |
| ForwardedHeaders configured (if behind proxy) | Required | Match your proxy's headers |
| Operational store configured with durable persistence | Required | EF Core or custom store |
| Token cleanup enabled | Recommended | `EnableTokenCleanup = true` |
| Configuration store cache enabled | Recommended | `AddConfigurationStoreCache()` |
| Distributed cache configured (if multi-instance) | Recommended | Redis, SQL, etc. |
| Health checks implemented | Recommended | Discovery + JWKS endpoints |
| OpenTelemetry configured | Recommended | Metrics + traces for monitoring |
| Events enabled | Recommended | For auditing and APM |
| Signing key store uses durable storage | Required for multi-instance | EF operational store or custom |
| Logging level set to Warning+ for production | Recommended | Avoid log bloat |
| `~/keys` directory excluded from source control | Required if using file-based key store | Prevent dev keys in production |
| HTTPS + ForwardedHeaders configured before IdentityServer | Required if behind proxy | Discovery must publish HTTPS issuer |
| Signing keys shared by all instances + rotation plan | Required for multi-instance | Automatic Key Management where available |
| DB schema changes applied before new app version starts | Required | Plus operational-store cleanup enabled |
| Same operational data / signing keys / DP keys / caches per instance | Required for multi-instance | Every instance shares all shared state |
| CORS allows only required client origins | Required | Watch middleware order |
| Token + session lifetimes match threat model | Recommended | Tune per deployment |
| Rate limiting assessed | Recommended | Public / multi-tenant deployments |
## Common Anti-Patterns
- ❌ Deploying without configuring ForwardedHeaders behind a reverse proxy
- ✅ Always configure ForwardedHeaders when behind a proxy; test by checking the discovery document's issuer URL
- ❌ Using default (ephemeral) Data Protection keys in production
- ✅ Always persist keys to durable, shared storage with `.PersistKeysTo...()`
- ❌ Not setting `SetApplicationName()` causing key isolation between deployments
- ✅ Always set an explicit, consistent application name
- ❌ Using file-system signing key store in containerized/multi-instance deployments
- ✅ Use EF operational store or a shared `ISigningKeyStore` implementation
- ❌ Enabling `Trace` or `Debug` logging in production — exposes tokens and sensitive data
- ✅ Use `Warning` level in production; use `Information` temporarily for troubleshooting
- ❌ Not enabling token cleanup — database grows indefinitely
- ✅ Enable `EnableTokenCleanup = true` and configure appropriate intervals
## Common Pitfalls
1. **Discovery document shows HTTP issuer**: The most common deployment issue. Always configure ForwardedHeaders or the `ASPNETCORE_FORWARDEDHEADERS_ENABLED` environment variable when behind a TLS-terminating proxy.
2. **CryptographicException on startup**: Usually means Data Protection keys from one environment are being used in another. Check that keys are persisted correctly and the application name is consistent.
3. **Signing keys not shared across instances**: The default file-system key store is per-instance. Use `AddOperationalStore()` which includes `ISigningKeyStore`, or configure a custom shared store.
4. **Redis losing Data Protection keys on restart**: If using `PersistKeysToStackExchangeRedis`, configure Redis with persistence (RDB snapshots or AOF) to survive restarts.
5. **IIS Data Protection permissions**: IIS may lack permissions to persist Data Protection keys. Follow Microsoft's IIS-specific Data Protection documentation.
6. **Multiple proxies in chain**: If you have more than one proxy, set `ForwardLimit` to match the number of proxies, and add all proxy addresses to `KnownProxies` or `KnownNetworks`.
7. **Cookie SameSite failures behind proxy**: If the proxy strips HTTPS, cookies won't get the `Secure` attribute, causing `SameSite=None` cookies to be rejected by browsers. Fix the proxy configuration first.
8. **OpenTelemetry trace source selection**: In production, subscribing to all trace sources (`Stores`, `Validation`, etc.) can generate excessive trace data. Start with `Basic` and add more sources as needed for troubleshooting.
9. **v8 license key format / runtime enforcement**: The v8 license key is a signed JWT with a `kid` header. A v7-format key still runs v8 core, but a v8 key fails on v7/earlier or the BFF runtime with `IDX10503: ... Token does not have a kid.` v8 also **throws at startup** when a configured license lacks the entitlement for Server-Side Sessions, Automatic Key Management, or SAML — run lower environments with the production key so gaps surface before production.
---
## Related Skills
- `identityserver-hosting-setup` — DI registration and middleware pipeline
- `identityserver-data-storage` — EF Core stores, migrations, token cleanup
- `identityserver-aspire` — orchestrating IdentityServer in Aspire AppHost
SHA-256: 219df1cfb0bab8a198c1889f1fa8bd3ad08eec291f615ffaad6572af41105fbe