Nov 2026Cloud · Terraform
Disaster recovery through infrastructure as code: Terraform rebuilds a small business's system in AWS after a typhoon, from verified backups, in under six minutes.
A disaster recovery proof of concept for Philippine small businesses. It protects a sample management system for a fictional farmers' cooperative in Naga City: members, inventory and a ledger in FastAPI and PostgreSQL 16 on Docker Compose. When a typhoon takes the office server offline, Terraform builds a temporary recovery site in AWS Singapore, restores a verified backup, and checks that the application works again.
Every hour the database is backed up with a snapshot-consistent pg_dump, restored into a throwaway test database and recounted before it counts as a backup. Verified copies go to an encrypted, versioned S3 bucket. In a recovery, a local job runner checks the backup and the Terraform plan, the operator approves, and the recovery host restores the data, checks its checksum, health and record counts, and reports the measured RTO and RPO before teardown.
- Final real AWS drill: recovered in 5 min 29 s with 26 s of data loss, 10 of 10 checks passed. Targets were 30 minutes and 1 hour.
- Standby costs cents a month: no servers run until a recovery is approved. Recorded compute spend for the whole project was about USD 0.01.
- No SSH keys, no IAM users and no long-lived access keys: SSO for the operator, Session Manager for the host.
- React and Vite dashboard live on GitHub Pages, with a simulated recovery walkthrough and the published drill results.
Infrastructure as code
The whole recovery site is Terraform
Nothing in AWS was clicked together. Five modules describe the recovery site, and one variable decides how much of it exists.
recovery_enabled = falseStandby. Backup bucket, IAM roles, log group and a budget alert. No servers; S3 storage is the only real cost.
recovery_enabled = trueRecovery. Adds the VPC, subnet, internet gateway, S3 endpoint, security group, EC2 host and a status alarm. Setting it back to false removes them and leaves the backups untouched.
- networkingVPC, one public subnet, internet gateway, free S3 gateway endpoint, and a security group with no inbound rules
- computet3.small on Amazon Linux 2023, IMDSv2 only, encrypted gp3, terminates on shutdown, bootstrapped from a template
- storagePrivate, encrypted, versioned backup bucket with lifecycle rules and a TLS-only policy
- identityHost role with Session Manager and read-only backups; an upload-only backup-writer role that cannot read or delete
- monitoringLog group with 7-day retention, host status alarm, AWS Budgets alerts at 50% and 100%
- Remote state in S3Versioned, encrypted bucket with native S3 lockfile locking and
prevent_destroy - Checked in CI
fmt,validate,terraform teston a mocked provider, TFLint and Trivy in GitHub Actions - Plan guardThe job runner fails closed on an unexpected plan, and the operator types an approval before
apply
terraform plan plan guard approval apply restore verify teardown
AWS services
- Amazon S3Encrypted, versioned off-site backups and the Terraform state
- Amazon EC2Temporary recovery host, created only during a recovery
- Amazon VPCRecovery network with no inbound rules and no NAT gateway
- AWS IAMUpload-only backup writer and a read-only recovery host role
- IAM Identity CenterSSO sign-in for the operator, no long-lived keys
- Systems ManagerSession Manager access and recovery progress, no SSH
- CloudWatchRecovery logs and the host status alarm
- AWS BudgetsSpending alerts so a forgotten drill cannot run up a bill
Recovery results
| Run | Time to recover (RTO) | Data loss (RPO) |
|---|---|---|
| Final disaster drill | 5 min 29 s | 26 s |
| First recovery drill | 7 min 15 s | 19 min 38 s |
| Project target | 30 min | 60 min |
Single test runs on a small synthetic dataset, not production guarantees. Off-site uploads need an operator session, so real data loss depends on the latest uploaded backup.










