Automated AWS Cost Report Generation and Email Notification for Daily and Monthly Insights
Introduction:
In today’s cloud-based infrastructure, managing costs is a crucial aspect for businesses. AWS offers a comprehensive set of tools and services to monitor and optimize costs, but keeping track of expenses can still be a daunting task. To simplify this process, automating the generation of cost reports and receiving regular email notifications can provide valuable insights and help businesses make informed decisions.
In this blog post, we will explore how to set up an automated AWS cost report generation and email notification system using AWS Lambda, Amazon EventBridge, and Amazon SES (Simple Email Service). This solution will enable you to receive daily and monthly cost insights directly in your inbox, empowering you with the information you need to manage your AWS costs efficiently.
Prerequisites:
Before diving into the implementation, there are a few prerequisites you should have in place:
1. AWS Account: You will need an active AWS account to access the required services and resources.
2. IAM Role: Create an IAM role with appropriate permissions to allow your Lambda function to access AWS Cost Explorer API and send emails using SES.
3. AWS Lambda: Familiarity with AWS Lambda, a serverless compute service that allows you to run code without provisioning or managing servers.
4. EventBridge (CloudWatch Events): Understand the basics of EventBridge, a serverless event bus service that allows you to schedule and trigger events based on time, application state changes, or custom events.
5. SES (Simple Email Service): Basic knowledge of SES, an email sending and receiving service, used to deliver cost reports and notifications to your email address.
Implementation Options:
In this blog post, we will cover only implementation options: manual setup through the AWS Management Console and the other approach is infrastructure-as-code using Terraform.
1. Manual Setup: We will guide you through the step-by-step configuration process using the AWS Management Console. This approach is suitable for users who prefer a hands-on approach and want to understand the individual components and their configurations.
2. Terraform: For users who prefer infrastructure-as-code and want to automate the provisioning of resources, we will provide instructions for implementing the solution using Terraform. This approach allows for repeatable and version-controlled deployments of the cost management system.
To follow along with the manual setup implementation, please refer to the detailed instructions provided in this blog post. If you prefer the Terraform approach, you can access the complete Terraform code and instructions on our GitHub repository: https://github.com/AmanPathak-DevOps/Terraform-for-AWS/tree/master/Non-Modularized/AWS-Cost-Reporting. The repository includes the necessary Terraform scripts and configuration files to set up the cost management system.
By following the upcoming sections of this blog, you’ll gain insights into both manual and Terraform-based implementations of the automated AWS cost report generation and email notification system. Whether you prefer a hands-on or automated approach, this solution will empower you to effortlessly stay informed about your AWS costs and optimize your cloud expenditure.
Let’s get started with the step-by-step implementation of this cost management solution.
- We have to add two types of SES here, first- sender, and second receiver.
Let’s create first by going on Amazon SES-> Verified Identities and clicking on Create an identity.
2. Choose Identity type as Email address, add your Email address, and click on Create identity.
Don’t forget to create the receiver’s SES and do the same thing to create the receiver’s SES.
3. After creating SES, you will get a notification on your given email address. Click on the link that highlighted the link in the Screenshots below.
After doing this, you will be able to receive or send mail.
4. To confirm if you have agreed to accept emails in SES, you can check the identity status, which should be “Verified”.
After doing this, SES configuration will be completed and we will configure the Lambda Function.
5. Click on “Create function”.
6. Take the details from the below screenshots.
About Runtime, I will be working with Python3.9 as this is not much older and quite stable.
And remember to Create a new role with basic Lambda permissions and click on “Create”.
7. You can see in the screenshot below that we have created the lambda function.
8. Now, We need to add some permissions for the Lambda function. As it will require the data from CostExplorer and after that Lambda function will help to send an email using Amazon SES service.
Go to the IAM role that is attached to the AWS-Cost-Reporting Lambda function.
9. Click on “Add permissions”.
10. We have added two new permissions(Cost-Explorer-GetUsage and AmazonSESFullAccess) in the role.
11. Now, we need some library packages(boto3, DateTime, SDKPandas, etc). To do that, we will add a layer in which all the libraries related to our code will be there.
Click on “Layers” highlighted in the screenshot below.
12. Click on “Add a layer”.
13. First layer is already given by AWS itself. Do the same according to the given screenshot below.
14. Now, we need the Python DateTime package, which is not added in the previous layer.
To do that, firstly we will install the DateTime package and move the DateTime package to the file layer/python/lib/python3.9/site-packages/. Make sure that the file sequence must be like that and to follow the entire process, refer to the given screenshot below.
15. Click on “Add a layer” by going to the “AWS-Cost-Reporting” lambda.
16. This time click on the highlighter “Layers” in the given screenshot below.
17. Now, Create the layer first by adding the zip file that you have created in step/screenshot 14 and keep the rest of things the same.
18. Now, Add a layer. We have a custom layer. So, click on “Custom layers” and add the created layers.
After doing this, main dependencies configurations have been completed for the Lambda function.
19. Now, we will add our Python code to fetch the Cost Usage from CostExplorer and send the mail with the Cost Usage details to the given mail ids.
Note: In the provided code, there are two places where you need to update your email addresses. Please make the necessary changes at those locations.
Python Code link: https://github.com/AmanPathak-DevOps/Terraform-for-AWS/blob/master/Non-Modularized/AWS-Cost-Reporting/costusage.zip
20. To run the lambda function. Click on Test and do the following things by referring to the given screenshot below.
21. Here, We have run the code and it has given the expected results related to the cost.
Now, we have to check whether the mail has been received or not.
22. In the below screenshot, you can see that the mail has been received.
23. Now, We can’t run the lambda function manually on a daily basis.
So, to automate this. We will add an Eventbridge rule that has a scheduler that will trigger the lambda function at the given time in the scheduler.
To do this, Click on Create rule by selecting EventBridge Rule.
24. Select Rule type as Schedule and click on Continue to create rule.
25. Here, We will schedule the time to trigger the lambda function.
In the below screenshot, We have provided the UTC time 11:20 IST which will be IST 4:50 PM and It will trigger on the 1st day of the month every year at 4:50 PM IST.
You can schedule the time according to you.
For testing purposes try to schedule time while you implement it.
26. Now, add the Lambda function as Target which will trigger the AWS-Cost-Reporting Lambda function, and click on Next.
Now, you can see the rule that we have created in the given screenshot below.
According to the EventBridge rule, the Lambda function will trigger at 4:50 PM IST.
Currently, it is 4:40 PM on my clock. Let’s wait for 10 minutes and see if we receive the expected email or not.
Hurray, we received the message at the expected time.
Conclusion: Automating AWS cost reporting and email notifications is a valuable step towards effective cost management in the cloud. In this blog post, we have discussed two implementation options: manual setup through the AWS Management Console and infrastructure-as-code using Terraform. You can choose the approach that best suits your needs and preferences.
Whether you opt for the manual setup or the Terraform implementation, following the instructions provided will enable you to automate the generation of cost reports and receive timely email notifications. This will empower you to make informed decisions and optimize your AWS costs efficiently.
Feel free to explore the blog post and the accompanying resources, and start implementing the automated AWS cost report generation and email notification system today.
Feel free to reach out to me if you have any doubts or queries.
LinkedIn- https://www.linkedin.com/in/aman-devops/
Github- https://github.com/AmanPathak-DevOps/
Happy Learning!
Aman Pathak
