Disaster Recovery Plan — Experimently

This page describes what the AWS CDK app in infrastructure/cdk actually creates, and how to recover each part of it. Where the CDK does not create something a recovery would need (a second region, replicated secrets), the page says so.

The commands use ENV for the environment (prod, staging, dev). Most names follow from it: the ECS cluster is experimentation-$ENV, the API service is experimentation-backend-$ENV, the dashboard service is experimentation-dashboard-$ENV, and the Redis replication group is experimentation-redis-$ENV-redis. The Aurora cluster's identifier is generated by CloudFormation, so read it from the database stack:

CLUSTER=$(aws cloudformation describe-stacks --stack-name "experimentation-database-$ENV" \
  --query "Stacks[0].Outputs[?OutputKey=='ClusterIdentifier'].OutputValue" --output text)

Recovery Objectives

Objective	Target	Mechanism
RPO (Recovery Point Objective)	5 minutes	Aurora point-in-time recovery (PITR), within the backup retention period

Aurora backs up continuously. You can restore to any second between the start of the backup retention period and the cluster's LatestRestorableTime, which AWS documents as typically within 5 minutes of the present. The retention period is the limit: see the table below. If Redis fails, no data is lost: it holds only cache and rate-limit state.

Backup Strategy

Data	What protects it	Retention	How it is recovered
PostgreSQL (Aurora)	Continuous backup (PITR) and a daily automated snapshot. The CDK sets a backup retention of 35 days in prod and staging, and 1 day in every other environment. `deploy.yml` also takes a manual cluster snapshot before every deploy's migration (`pre-deploy-<env>-<tag>-<time>`), and deleting the prod stack leaves a final snapshot.	prod and staging 35 days, others 1 (automated); manual snapshots until you delete them	PITR or snapshot restore to a new cluster (Scenario 4)
Redis (ElastiCache)	A daily snapshot, 02:00–03:00 UTC. There is no append-only file. In prod, three nodes with automatic failover; in staging and dev, one node.	prod 7 days, staging 3, dev 1	Automatic failover (prod), or let it refill: it is a cache
Dashboard	Holds no data. It is the `frontend/Dockerfile` image, run as the ECS service `experimentation-dashboard-$ENV` behind the API's load balancer. The CDK creates no S3 bucket or CloudFront distribution for it.	Images in ECR (`experimentation-platform/web`) and, for releases, GHCR	Redeploy the image (Rollback Runbook, "The dashboard is the opposite case")
API	Holds no data. Its image is in ECR (`experimentation-platform/backend`) and, for releases, GHCR.	As above	Redeploy through CodeDeploy (Rollback Runbook)
ECS task definitions	Registered revisions, one per deployment	Until deregistered	Name the revision in a CodeDeploy deployment
Secrets Manager	The database credentials secret (`experimentation-database-$ENV-aurora-credentials`), in the stack's region only. The CDK configures no replica regions.	Until deleted	Within its recovery window after deletion, `aws secretsmanager restore-secret`. There is no copy in another region
Application logs	CloudWatch Logs only, with no export to S3. `/ecs/experimentation-backend-$ENV` and `/ecs/experimentation-dashboard-$ENV` are kept 90 days, and `/ecs/experimentation-migrate-$ENV` 30 days.	As listed	Not recoverable after expiry
S3 (full profile only)	The analytics data lake bucket (versioned), and the Glue ETL stack's Athena results bucket (objects expire after 30 days) and Glue scripts bucket. None has cross-region replication. The core profile creates no bucket.	Kept when the prod stack is deleted; deleted with it elsewhere	Versioning restores an overwritten or deleted object in the data lake; nothing else

infrastructure/tests/test_disaster_recovery_doc.py synthesises the prod app (and staging, for the Aurora retention) and checks the retention, replication and log figures in this table against it.

A cluster deployed from an earlier version of the CDK app, which set no retention, keeps CloudFormation's default of one day until its database stack is redeployed. Check what a cluster actually keeps before you rely on it:

aws rds describe-db-clusters --db-cluster-identifier "$CLUSTER" \
  --query 'DBClusters[].BackupRetentionPeriod'

Scenario 1: Single ECS Task Crash

Detection: ECS replaces the task automatically. While another task stays healthy, no alarm emails. Each of the API's two target groups has a healthy-task alarm, experimentation-api-healthy-blue-$ENV and experimentation-api-healthy-green-$ENV: it is in ALARM once its target group has had no healthy target for 3 minutes. Neither alarm has an action. The composite experimentation-api-no-healthy-task-$ENV emails ALARM_EMAIL when both are in ALARM at once. Between deployments one of blue and green is empty, so the idle colour's healthy alarm is always in ALARM, by design; that is not an incident. In an environment that runs one API task (demo), replacing that task leaves no healthy target for a few minutes, and the composite sends a true "no healthy task" email for it.

Response

No action is needed.

Verification

Verify replacement task is running:

aws ecs describe-services \
  --cluster "experimentation-$ENV" \
  --services "experimentation-backend-$ENV" \
  --query 'services[0].{Running:runningCount,Desired:desiredCount,Events:events[:3]}'

If the running count stays below the desired count (3 in prod) for 5 minutes, go to Scenario 2.

Scenario 2: All ECS Tasks Down

Detection: experimentation-api-no-healthy-task-$ENV emails ALARM_EMAIL once neither target group has had a healthy target for 3 minutes (Scenario 1 describes the two alarms it combines). The API's 5xx alarms (experimentation-api-5xx-blue-$ENV and -green-$ENV) do not fire for this: they count 5xx responses from the tasks (HTTPCode_Target_5XX_Count). With no task running, the load balancer answers 503 itself, and those minutes count as not breaching. The composite needs the alarm topic's email subscription to be confirmed, and CloudWatch to be reachable; an uptime check from outside AWS on /health covers both.

Immediate Response

Step 1: Check why tasks are failing:

aws ecs describe-services \
  --cluster "experimentation-$ENV" \
  --services "experimentation-backend-$ENV" \
  --query 'services[0].{Status:status,Running:runningCount,Events:events[:10]}'

Step 2: Check stopped task reasons:

STOPPED=$(aws ecs list-tasks \
  --cluster "experimentation-$ENV" \
  --desired-status STOPPED \
  --query 'taskArns[:3]' \
  --output text)

aws ecs describe-tasks \
  --cluster "experimentation-$ENV" \
  --tasks $STOPPED \
  --query 'tasks[*].{StopReason:stoppedReason,ContainerReason:containers[0].reason,ExitCode:containers[0].exitCode}'

Step 3: Check recent application logs for the crash reason. The API writes one JSON object per line with a lower-case level, so this matches its error and critical lines (in staging and prod; dev and demo write console lines, which it does not match):

aws logs filter-log-events \
  --log-group-name "/ecs/experimentation-backend-$ENV" \
  --filter-pattern '{ ($.level = "error") || ($.level = "critical") }' \
  --start-time $(( ($(date +%s) - 900) * 1000 )) \
  --limit 50

Step 4: restart the API. The API service uses the CodeDeploy deployment controller, so restart it with a CodeDeploy deployment that names the revision already serving, not with aws ecs update-service --force-new-deployment. The commands are in Restart the API on the revision it is serving in the Rollback Runbook.

Verification

curl -fsS https://app.example.com/health | jq '{status, database: .checks.database.status}'

It prints "status": "healthy" and "database": "healthy". /health answers 503 while the database is unreachable.

Escalation

If tasks keep failing after the restart, the cause is likely one of these: - The new container image is crashing → roll back to the previous task definition (see Rollback Runbook) - Aurora or Redis is unreachable → go to Scenario 3 or 5

If not resolved within 30 minutes: escalate to Engineering Lead.

Scenario 3: Aurora Database Primary Instance Failure

Detection: Application logs show could not connect to server errors, and /health answers 503, so experimentation-api-no-healthy-task-$ENV emails once no task passes its health check (Scenario 2). The one database alarm, AuroraHighCPU-$ENV, watches the writer's CPU (DBClusterIdentifier from the SSM parameter /experimentation/$ENV/database/aurora-cluster-identifier, Role=WRITER) and treats missing data as breaching, so a writer that stops publishing metrics also puts it in ALARM, after 15 minutes. In prod Aurora fails over to the reader. Other environments have one instance and no reader, so Aurora has to replace the instance.

Automatic Failover (prod: Aurora handles this)

In prod the cluster has a writer and a reader.

The API connects to the cluster (writer) endpoint. Its connection pool is configured to test each connection before use (pool_pre_ping in backend/app/db/session.py), so a connection to the old writer is replaced the next time it is used.

Verify Failover Completed

Check cluster status:

aws rds describe-db-clusters \
  --db-cluster-identifier "$CLUSTER" \
  --query 'DBClusters[0].{Status:Status,Endpoint:Endpoint,ReaderEndpoint:ReaderEndpoint,Members:DBClusterMembers[*].{Role:IsClusterWriter,ID:DBInstanceIdentifier}}'

Watch until status is "available" and a new writer is identified:

watch -n 10 "aws rds describe-db-clusters \
  --db-cluster-identifier $CLUSTER \
  --query 'DBClusters[0].{Status:Status,Writer:DBClusterMembers[?IsClusterWriter].DBInstanceIdentifier|[0]}'"

Restart the API (if it does not recover by itself)

Restart through CodeDeploy, as in Scenario 2, Step 4.

Escalation

If the failover has not completed within 5 minutes, or no instance is available: go to Scenario 4.

Scenario 4: Full Aurora Database Cluster Failure

Detection: aws rds describe-db-clusters returns status failed, and every database call in the application logs fails. RTO: in the staging rehearsal on 2026-10-11 (UTC), on a database holding little data, the restore took 11 min 41 s and the cutover took the API down for 12 min 45 s (#295). A larger database restores more slowly. RPO: 5 minutes (PITR), and only within the backup retention period (35 days in prod and staging, 1 elsewhere; see Backup Strategy). Beyond that, the newest manual snapshot (pre-deploy-... or pre-migration-...) is the recovery point.

A cluster in status failed has no decided path. The restore below renames the original cluster, and whether RDS can rename a failed one is not known: nobody has tried it, and it cannot be rehearsed. If the cluster is failed, do not start the restore; call the Engineering Lead and AWS Support, as in the last step of this scenario. The restore is the decided path when the cluster is available but its data is not: Scenario 8, or a migration to undo (Rollback Runbook, Step 3).

Immediate Response

Step 1: Confirm cluster is down:

aws rds describe-db-clusters \
  --db-cluster-identifier "$CLUSTER" \
  --query 'DBClusters[0].{Status:Status,LatestRestorableTime:LatestRestorableTime}'

Step 2: List the most recent snapshots, automated and manual:

aws rds describe-db-cluster-snapshots \
  --db-cluster-identifier "$CLUSTER" \
  --query 'sort_by(DBClusterSnapshots, &SnapshotCreateTime)[-5:].{ID:DBClusterSnapshotIdentifier,Time:SnapshotCreateTime,Type:SnapshotType,Status:Status}'

Restore from PITR (preferred)

Rehearsed on a staging account on 2026-10-11 (UTC) (#295): read, restore, cutover, and the cutover undone with rollback; readers, keep and start-api were not run. Prod's readers step has run only against the fake AWS in the test suite (backend/tests/unit/infrastructure/test_restore_repoint.py), on a two-instance cluster: staging runs one instance.

Both backend task definitions take the database host from the database stack's writer endpoint, which is the cluster identifier plus a suffix, and deploy.yml, the stack and its alarms all find the cluster by that identifier. A restore always creates a new cluster, under a new identifier, so scripts/restore_repoint.sh restores beside the original and then, with the API at 0 tasks, renames the original cluster and its instances out of the stack's names and the restored cluster and its writer onto them. The endpoint stays the same text, nothing is redeployed, and nothing is deleted: the original is kept, renamed <env>-db-failed-<time>.

Run it from a fresh clone of the release the environment serves, with administrator credentials for the environment's account and jq installed. Run each phase on its own, with the Engineering Lead's approval at the time it runs: restore adds a cluster and an instance that bill until they are deleted, and cutover and rollback take the API down. Every identifier, group, class and count is read from the stacks, the original cluster or the API service; you type only the environment and the restore time. Each phase adds what it read and did to the evidence directory read creates in the current directory (log.txt, probe.log, parity.diff). It names the account's resources: keep it out of the repository.

Step 1: Record the original cluster, the stacks and the API. This changes nothing:

EVID=$(scripts/restore_repoint.sh read "$ENV")

read refuses a cluster that is not available, a cluster whose parameter groups, subnet group, VPC groups or instances are not the database stack's, task definitions that do not take POSTGRES_SERVER from the cluster endpoint, a newest migration task revision that names an image tag rather than a digest (the message says how to register one), and an API that is already at 0 tasks.

Step 2: Restore beside the original. Replace 2026-03-01T14:55:00Z with the time to restore to, in UTC: just before the failure, or before the migration when you come from the Rollback Runbook. The application keeps running on the original meanwhile.

scripts/restore_repoint.sh restore "$EVID" 2026-03-01T14:55:00Z

The restored cluster gets the original's subnet group, VPC groups and cluster parameter group, and --copy-tags-to-snapshot; its writer gets the original writer's class, parameter group, promotion tier and tags. restore then refuses unless both match the original on every setting it compares (parity.diff): a restore given no parameter group gets the engine's default one, with no error, and loses the settings the database stack makes, the cluster's timezone, rds.force_ssl and shared_preload_libraries and the instances' logging and work_mem. Last, it runs two one-off tasks of the migration task definition in the API tasks' subnets, with the stack's database credentials. The first marks the restored database, which also proves those credentials open it; the second checks that the original reads as unmarked. password authentication failed in probe.log means the password was changed after the restore time: set the restored cluster's master password to the secret's value and run restore again with the same time, which carries on with the cluster it made. It refuses another time for a cluster it already made: to restore to another time, run read again for a new evidence directory.

Step 3: Check the data on the restored cluster before anything points at it: the rows you expect are there and the bad change is not. probe.log shows the restored database's alembic_version. It must equal the migration heads of the release the API runs. If it does not, the API would answer /health and fail requests: put the API on the release that matches first (Rollback Runbook).

Step 4: Point the stack's names at the restored cluster. The API is down from the start of this step to its end: 12 min 45 s in the staging rehearsal on 2026-10-11 (UTC), on a database holding little data (#295).

scripts/restore_repoint.sh cutover "$EVID"

cutover asks you to type the cluster identifier it shows, and refuses input from a pipe or a here-document, so run it on its own line. It refuses while a CodeDeploy deployment is in flight, scales the API to 0, and waits until no API or migration task is running. It then renames the clusters and instances with --apply-immediately, refuses unless the writer endpoint is exactly what it was, and starts the API only after two probes in a row, from the API tasks' subnets, have reached the marked database through that endpoint. The API gets back the task count and scaling Step 1 recorded.

During the cutover of the staging rehearsal on 2026-10-11 (UTC), /health answered 503 for 11 min 40 s (#295). The composite alarm experimentation-api-no-healthy-task-$ENV is expected to email ALARM_EMAIL after about three minutes, then clear; that has not been rehearsed on AWS as of v0.31.1. A green /health afterwards does not show which cluster answered, because it runs SELECT 1; the probes do. Writes made to the original between the restore time and the cutover stay on it, under its new name.

Step 5 (prod only): Add the reader, under the original reader's name and with its settings:

scripts/restore_repoint.sh readers "$EVID"

Step 6: Keep the original: a manual snapshot (<name>-kept) and deletion protection.

scripts/restore_repoint.sh keep "$EVID"

The original keeps billing for its instances and storage, and it holds the only point-in-time history from before the incident: the restored cluster's starts when it was created. Deleting it is a human decision, taken at that moment. It still uses the stack's parameter, subnet and VPC groups, so deleting the database stack fails until it is gone.

Before the cutover the restored cluster is not watched by AuroraHighCPU-$ENV either. The alarm reads the cluster identifier from the database stack's parameter /experimentation/$ENV/database/aurora-cluster-identifier, which names the stack's own cluster. The cutover gives the restored cluster that identifier, so from then on the alarm watches it with no redeploy, as it did in the staging rehearsal on 2026-10-11 (UTC) (#295). The original, once renamed, is watched by no alarm.

Undo a cutover

scripts/restore_repoint.sh rollback "$EVID"

rollback runs the same steps with the roles exchanged, and asks for the same confirmation: the restored cluster leaves the stack's names as <env>-db-abandoned-<time>, the original comes back onto them, and the probes check that the endpoint reaches the unmarked database. If the API never started on the restored cluster nothing is lost; otherwise the writes made after the cutover stay on the abandoned cluster. Run keep afterwards to keep that one. If keep already ran for the original, the original comes back onto the stack's names with deletion protection on, which the database stack does not set: deleting the database stack fails while it is on.

If the script stops part-way

When cutover or rollback stops after scaling the API to 0, the API stays at 0 and the script prints where every cluster of the restore is (log.txt has it too):

The stack's identifier still names the original and no <env>-db-failed-<time> exists: nothing was renamed. scripts/restore_repoint.sh start-api "$EVID" puts the API's task count and scaling back.
cutover stopped after the original was renamed, for instance because the endpoint changed (the message then names the rollback command) or the probes did not pass within 30 minutes (probe.log): wait until no cluster or instance of the restore is renaming, then run rollback. It also works when no cluster holds the stack's identifier yet, by moving the original back on its own.
rollback stopped part-way: there is no scripted way on. The Engineering Lead decides, with log.txt and the list the script printed.

A cluster in status failed is not covered: read refuses it (see the top of this scenario).

Verification

curl -fsS https://app.example.com/health | jq '{status, database: .checks.database.status}'

It prints "status": "healthy" and "database": "healthy". That shows the API reaches a database, not which one: the cutover's probes show that.

Escalation

If restore does not complete within 30 minutes: escalate to Engineering Lead and VP Engineering. Contact AWS Premium Support for assistance with the Aurora cluster.

Scenario 5: ElastiCache Redis Failure

Detection: Each API task logs rate limiter: Redis unavailable, counting requests per process once, at the first request after Redis fails, and rate limiter: Redis reachable again within 30 s of Redis recovering (query below). /health reports checks.redis.status as unhealthy but stays ready (200) unless REDIS_REQUIRED=true. The Redis alarms are one per node, RedisHighCPU-001-$ENV (and -002-, -003- in prod), on EngineCPUUtilization for experimentation-redis-$ENV-redis-001 and its siblings. They treat missing data as breaching, so a node that stops publishing metrics puts its alarm in ALARM after 15 minutes. No alarm watches whether a node accepts connections.

Application Behavior During Redis Failure

Without Redis, each of these pinned by the test named: - Rate limiting falls back to per-task counts, so per-IP limits are multiplied by the running task count, and retries Redis every 30 s (backend/tests/unit/middleware/test_rate_limiter.py). - Readiness does not fail unless REDIS_REQUIRED=true (backend/tests/unit/core/test_health_endpoints.py). - Readiness does not hold other requests. /health and /health/ready run in the thread pool, so /health/live keeps answering while a readiness check waits on Redis (TestReadinessRunsOffTheEventLoop in backend/tests/unit/core/test_health_endpoints.py). - One connect attempt. The readiness ping and the results cache try to connect once, with 1 s connect and socket timeouts, where redis-py's default is ten retries with backoff (test_the_client_tries_to_connect_once and test_the_client_gives_up_on_an_unreachable_redis_after_a_second in backend/tests/unit/core/test_redis_connection.py).

Which tasks are counting per process, and since when (CloudWatch Logs Insights on /ecs/experimentation-backend-$ENV):

fields @timestamp, @logStream, rate_limiter, reason, fallback_seconds, event
| filter logger = "backend.app.middleware.rate_limiter" and ispresent(rate_limiter)
| sort @timestamp asc

rate_limiter is per_process on the line that starts the fallback and redis on the line that ends it; fallback_seconds on the second says how long it lasted. reason and detail name the error, which is the place to look when a wrong REDIS_HOST or a TLS mismatch keeps a task in the fallback after every deploy.

Immediate Response

Step 1: Check ElastiCache replication group status:

aws elasticache describe-replication-groups \
  --replication-group-id "experimentation-redis-$ENV-redis" \
  --query 'ReplicationGroups[0].{Status:Status,MemberClusters:MemberClusters,AutomaticFailover:AutomaticFailover}'

In prod the replication group has three nodes with automatic failover and Multi-AZ, so ElastiCache promotes a replica by itself. In staging and dev there is one node and no replica: wait for ElastiCache to replace it, or recreate the stack.

Restart the API once Redis is back

Restart through CodeDeploy, as in Scenario 2, Step 4, so that every task reconnects to Redis.

Verification

curl -fsS https://app.example.com/health | jq '.checks.redis.status'

It prints "healthy".

Escalation

If Redis cannot be recovered within 30 minutes, keep running without it and open a critical ticket to recreate the Redis stack.

Scenario 6: Full AWS Region Failure

Detection: The AWS Health Dashboard shows a disruption in your region, and every health check fails. Regional failover is not built.

What the CDK does not create

A second region needs all of the following, and the CDK creates none of them:

Secrets Manager replicas in the second region
ECR repositories in the second region, holding the images (or pull them from GHCR)
S3 replication, for the full profile's data lake
An Aurora Global Database. Without one, cdk deploy in a new region creates a new, empty database, and the data comes back only by restoring a snapshot you have copied to that region (aws rds copy-db-cluster-snapshot).

Scenario 7: Security Breach

Detection: AWS GuardDuty alert, CloudWatch anomaly, or customer report.

See Responding to an incident on your deployment. If the cause may be a vulnerability in Experimently itself, report it privately as SECURITY.md describes.

Key immediate actions:

Disable suspected compromised account:

UPDATE experimentation.users SET is_active = false WHERE email = '<suspected>';

Revoke API keys:

UPDATE experimentation.api_keys SET is_active = false WHERE user_id = '<user-uuid>';

If AWS credentials are compromised, deactivate immediately:

aws iam update-access-key --access-key-id <key-id> --status Inactive

Take forensic snapshot before any remediation:

aws rds create-db-cluster-snapshot \
  --db-cluster-identifier "$CLUSTER" \
  --db-cluster-snapshot-identifier incident-$(date +%Y%m%d)-pre-remediation

Scenario 8: Complete Data Loss

Detection: Database tables empty or corrupted; results API returning empty data; operator error confirmed. RPO: PITR to 5 minutes before the loss, if the loss is found within the backup retention period (35 days in prod and staging, 1 elsewhere). After that, the newest manual snapshot is the recovery point.

Response

The restore is the one in Restore from PITR (Scenario 4). Its rehearsal on a staging account on 2026-10-11 (UTC) restored beside the running API, which was down only for the cutover (#295). Here the API is stopped from the start instead, so that nothing more is written to the damaged data, and the outage lasts from Step 2 until the cutover ends, restore included.

Step 1: Record the environment while the API still runs. read records the API's task count and scaling, which the cutover puts back at the end, so run it before Step 2; it refuses an API that is already at 0 tasks.

EVID=$(scripts/restore_repoint.sh read "$ENV")

Step 2: STOP the application immediately to prevent further writes to corrupted state. The API service auto-scales, with a floor equal to its task count (3 in prod), so lower the floor to 0 and suspend scaling before stopping the tasks; otherwise scaling starts them again:

aws application-autoscaling register-scalable-target \
  --service-namespace ecs \
  --scalable-dimension ecs:service:DesiredCount \
  --resource-id "service/experimentation-$ENV/experimentation-backend-$ENV" \
  --min-capacity 0 \
  --suspended-state DynamicScalingInSuspended=true,DynamicScalingOutSuspended=true,ScheduledScalingSuspended=true

aws ecs update-service \
  --cluster "experimentation-$ENV" \
  --service "experimentation-backend-$ENV" \
  --desired-count 0

Step 3: Confirm the extent of data loss. Connect to Aurora and check key tables:

SELECT 'experiments' AS table_name, COUNT(*) FROM experimentation.experiments
UNION ALL SELECT 'feature_flags', COUNT(*) FROM experimentation.feature_flags
UNION ALL SELECT 'assignments', COUNT(*) FROM experimentation.assignments
UNION ALL SELECT 'events', COUNT(*) FROM experimentation.events;

Step 4: Restore to just before the data loss, and check the restored data: Steps 2 and 3 of Restore from PITR, with the $EVID of Step 1.

Step 5: Point the application at the restored cluster: Step 4 of Restore from PITR (cutover), then its Steps 5 and 6. The cutover starts the API with the task count and scaling Step 1 recorded; nothing is typed.