Challenge Overview
Challenge Objectives
-
Create a script to generate CSV reports for AWS metadata API usage
Project Background
The usage of the AWS metadata APIs is being throttled because of so many requests, all information about API usage is stored in S3 using JSON files. The objective is to import those files into a tool, and generate reports that have aggregate totals of which credentials and APIs are used.
Technology Stack
-
AWS SDK / API
-
Python 2.7
-
Jenkins
-
AWS CloudTrail / Athena / S3
Individual Requirements
1. The usage data stored in the s3 bucket is generated from AWS Cloud Trail, and they will be imported into AWS Athena for analysis.
The configuration steps above should be well described in your submission.
2. You are also responsible for creating a python script to execute queries against AWS Athena to generate the following two reports:
-
Total API usage report by credentials
-
The report should contain the user identity and total API usage
-
-
Top-10 API endpoint being called
- The report should contain the API endpoint and total API usage
The script should be executed from command line, and should take two optional input arguments:
- startDate - if not provided, use current date
- endDate - if not provided, use current date
All data records with the startDate and endDate should be analyzed, and the date format should be like: yyyy-MM-dd. And if the startDate and endDate are the same, the script will just query the data records in that specific day.
After the execution, two reports should be generated in CSV file format, and saved to local file system. Logging should be added, and proper logs should be generated to indicate the job is successful or failed. Please configure it to log to console only for now. The script should follow OOP pattern, and it should be well documented as well.
The following properties should be configurable (please use an external json or YAML file for configuration):
-
Either AWS client id/key pair, or an IAM instance profile for authentication. Both should be supported
-
Local folder to store the generated CSV files. The filename should be like: api_usage_by_credentails_<yyyy-MM-dd-HH:mm:ss>.csv and api_usage_by_endpoints_<yyyy-mm-dd-HH:mm:ss>.csv.
Provides steps and screeshots to run the script from the Jenkins
Final Submission Guidelines
- Source Code- Deployment and Verification Guide