Run on July 7th, 2023

• 5:40 pm Pacific: Working on cleaning up the CloudFormation deployment which failed during the 4.1.1 update.
• 10:05 pm: Shutting down the load balancer and backing up the database.
• 10:39 pm: CloudFormation template update complete. Re-enabled the load balancer and loaded the site, which reports that it is running v4.1.4.
• 11:00 pm: Missed updating the template to use db.t4g.small instance class instead of .micro, so the web and Sidekiq services are currently failing due to lack of db connections.  Revised the template and am re-running now.
• 11:45 pm: All services appear stable, and Sidekiq is about halfway through the tasks that queued up during downtime.  No further updates planned for tonight, July 7th, unless there are problems.

What did I learn?

The problem I was having with CloudFormation not rolling back was that I did manual changes to the ECS task configs.  When I reverted back to the original config for the web service and continued the rollback, it worked fine.  After that, updates also worked fine.

I have to keep my own branch of the mastodon-on-aws repo with tweaks specific to montereybay.social up to date.