AWS CloudFormation Drifted Resource (How to fix)
How I solved an AWS CloudFormation resource deletion issue by investigating and recreating resources
If you've ever worked with AWS CloudFormation, you know that it's a powerful tool for managing your AWS infrastructure as code. However, like any tool, it's not without its quirks and challenges. Recently, I ran into an issue while working on a CloudFormation stack that involved a resource deletion problem. In this blog post, I'll share how I solved this issue and hopefully help others who might encounter a similar issue.
It all started with an error message, like always..
The problem started with an error message that looked like this:
AWS::Events::Rule UPDATE_FAILED Unable to retrieve Arn attribute for AWS::Logs::LogGroup, with error message Resource of type 'AWS::Logs::LogGroup' with identifier '{"/properties/LogGroupName":"/aws/events/development"}' was not found.
This error message indicated that CloudFormation was unable to retrieve the ARN attribute for an AWS Logs LogGroup resource. The reason for this was that the LogGroup resource didn't exist, but CloudFormation claimed that it did and tried to find its ARN. So why was the LogGroup resource missing in the first place?
Root cause
The issue arose when an AWS CloudFormation LogGroup resource was deleted for some unknown reason.
I only found this out by running a Drift detection, and the drift status returned as DRIFTED.
It is likely that there were some other scripts that cleaned up resources not following naming conventions but I can not confirm. There is no event log in CloudTrail (maybe I'll find it eventually).
Solution
Clean up by CF
To solve this issue, I launched the CloudFormation stack without the LogGroup resource definition to clean up the CloudFormation stack and force it to delete the LogGroupForEvents resource from the eventbridge-rule's stack. After that, I launched the CloudFormation stack normally to recreate the resources. This time, CloudFormation correctly created the LogGroup resource with the new name without any issues.
Delete resource manually
Manual deletion of the resource would be quicker if you have access and know what you're doing. After deleting it don't forget to update the stack again.
Conclusion
In conclusion, this issue with CloudFormation and resource deletion can be confusing, but it's not insurmountable. With some careful investigation and the right approach, it's possible to solve this issue and get your CloudFormation stack up and running again.
This article is co-authored with ChatGPT
Cover image comes from Unsplash