Automating the detection and remediation of exposed AWS keys
In today's cloud-powered world, protecting the AWS environment is paramount. A critical component of this security involves safeguarding access keys. When access keys are exposed on public source code management (SCM) platforms like GitHub, GitLab, or Bitbucket, they can be exploited by threat actors, leading to potentially catastrophic breaches of the AWS environment. Tools such as TruffleHog, GitRob, and Gitleaks are useful for identifying sensitive content on public SCM platforms. If threat actors gain access to these exposed keys, this could pose a risk to the AWS environment and potentially lead to security concerns.
This blog explores an automated approach to detect and remediate exposed AWS keys on public SCM platforms, including suspending the exposed IAM access keys, summarising recent API activity associated with these keys, and sending notifications to a Slack channel, by leveraging AWS Health Dashboard, AWS EventBridge, and AWS Lambda.
The Importance of Securing AWS Access Keys
AWS access keys are essential for securing the AWS cloud environment, providing programmatic access to AWS services and enabling automation and integration. If these keys are exposed, they can pose significant security risks.
Common situation where AWS keys might be exposed include:
- Accidental Exposure: Developers may accidentally push code with AWS keys to public SCM.
- Malicious Intent: Insiders or attackers might leak keys intentionally for personal gain.
- Phishing Attacks: Users could fall for phishing scams, revealing their keys to attackers.
Given these risks, it is important to implement automated mechanisms that can promptly detect and mitigate the exposure of AWS keys.
Implementing an Automated Workflow for Managing Exposed AWS Keys
AWS offers a range of services and tools that can be used to form an extensive automated remediation procedure, which addresses the problem of exposed AWS keys. The process for managing exposed AWS keys utilises a range of AWS tools and capabilities.The AWS Health Dashboard automatically identifies and monitors issues related to exposed keys. Amazon EventBridge is configured to trigger specific workflows based on these events, and AWS Lambda performs essential remediation tasks, such as suspending exposed keys, gathering pertinent logs from AWS CloudWatch Logs, and sending alerts via Slack.
AWS Health Dashboard : Detection
The AWS Health Dashboard is a central hub for alerts and notifications about the AWS environment’s health. It provides information on ongoing incidents, planned maintenance, and critical security events, such as exposed AWS keys. AWS continuously scans public repositories for leaked credentials, proactively identifying potential security risks before they can be exploited. When an exposed access key is detected, AWS creates an issue on the Health Dashboard within minutes. This timely notifications will trigger our automated remediation process.
EventBridge: The Orchestrator
EventBridge is a serverless event bus that simplifies the process of connecting applications and services through events. It orchestrates automated workflows by using preconfigured rules to monitor the AWS Health Dashboard for specific events. When it detects an event related to the exposed credentials, such as a "RISK" alert, it triggers a Lambda function to start the remediation process.
Lambda: The Remediation Engine
AWS Lambda is a serverless computing solution that allows code to run without provisioning or managing servers. When an exposed key event is detected, EventBridge triggers a Lambda function that performs these following actions:
- Suspend the Exposed Access Key: Using the iam:UpdateAccessKey permission, the Lambda function immediately suspends the exposed access key to stop any unauthorised access or potential misuse.
- Collect Logs: The function uses the logs:FilterLogEvents permission to gather logs associated with the exposed key from the last 24 hours, providing valuable information for post-incident analysis.
- Send Notifications: The function sends a Slack alert to notify the security team on the exposed key and includes specific details needed for further investigation.
Remediation Flow for exposed AWS Keys
Setting up this automated remediation flow involves several steps, such as creating IAM roles, defining EventBridge rules, and writing the Lambda function. This section offers a detailed, step-by-step guide to implement this solution.
- Creating IAM Role with Necessary Policies : Let’s start by creating an IAM role with policies that specifically allow the Lambda function to suspend access keys and retrieve logs. This ensures the function operates under least privilege principles to securely perform the remediation.
- Configuring EventBridge Rule: Creating an EventBridge rule that uses AWS Health Dashboard events to activate the Lambda function. The rule should be set up to detect AWS Health events linked to security issues. The following is the settings for the EventBridge rule:
- Setting up Lambda Function:The automatic remediation process relies on the Lambda function, which handles key actions to swiftly address exposed AWS keys and secure the AWS environment.
The Lambda function performs these following actions: - Suspending the exposed Access Key:
When the Lambda function is triggered, it begins by deactivating the exposed access key. This automation will immediately suspend the exposed access key to stop any unauthorised access or potential misuse. The function uses the iam:UpdateAccessKey permission to set the key’s status to Inactive. - Gathering Logs:After suspending the access key, the next step is to gather logs by using the logs:FilterLogEvents permission. These logs will provide the details on the actions performed with the exposed key, which are essential for further analysis and to assess if there is any indication of potential security breach.
The Lambda function retrieves logs from the past 24 hours, which will capture all relevant activity related to the exposed key. The collected logs include the information such as API requests made with the key, accessed resources, and any anomalies that might indicate the malicious behaviour.
- Sending Notification :Once the previous automation workflow is completed, the Lambda function sends a notification about the exposed key. The security team is promptly alerted via collaboration tools, in this case, Slack. The notification includes details such as the SCM URL, exposed access key ID, AWS account, and affected users, allowing the respective team to triage and evaluate the alert to determine if any further action is required.
Post Incident activities
After discovering an exposed access key and completing the automation workflow, we are following these activities as part of our precautionary measures:
- Conduct a Detailed Analysis :We will analyse the logs collected by the Lambda function to identify all actions performed using the affected exposed key to check if there are any unusual or suspicious activities.
- Generate New Access Keys : Generate new access keys to replace the affected exposed key. We will also update all the applications and scripts with the new access keys to ensure there are no availability issues in our environment after the key deactivated.
- Heighten Monitoring: As a further precautionary measure, we will heighten our SOC monitoring to be more vigilant and responsive to the potential threats related to the AWS cloud and applications.
Conclusion
Automating the detection and remediation of exposed AWS keys is essential for strengthening our organisation's security posture. By leveraging AWS Health Dashboard, EventBridge, and Lambda, we can develop an automated workflow to significantly enhance our ability to detect and respond to security incidents involving exposed AWS access keys to safeguard our digital assets from the potential security breaches.
Reference
Bug Bounty
Got what it takes to hack? Feel free to report a vulnerability in our assets and get yourself a reward through our bug bounty program. Find more details about policy and guidelines at https://www.halodoc.com/security
Join Us
Scalability, reliability, and maintainability are the three pillars that govern what we build at Halodoc Tech. We are actively looking for engineers at all levels and if solving hard problems with challenging requirements is your forte, please reach out to us with your resumé at careers.india@halodoc.com
About Halodoc
Halodoc is the number 1 all around Healthcare application in Indonesia. Our mission is to simplify and bring quality healthcare across Indonesia, from Sabang to Merauke. We connect 20,000+ doctors with patients in need through our Tele-consultation service. We partner with 3500+ pharmacies in 100+ cities to bring medicine to your doorstep. We've also partnered with Indonesia's largest lab provider to provide lab home services, and to top it off we have recently launched a premium appointment service that partners with 500+ hospitals that allow patients to book a doctor appointment inside our application. We are extremely fortunate to be trusted by our investors, such as the Bill & Melinda Gates Foundation, Singtel, UOB Ventures, Allianz, GoJek, Astra, Temasek, and many more. We recently closed our Series D round and in total have raised around USD$100+ million for our mission. Our team works tirelessly to make sure that we create the best healthcare solution personalised for all of our patient's needs, and are continuously on a path to simplify healthcare for Indonesia.