Detecting and Securing against Automated attacks

In the ever-evolving digital landscape, the prevalence of automated threats has become a persistent challenge. As technology advances, so does the sophistication of malicious actors armed with an array of tools. This blog unveils a strategic playbook to navigate and defend against these automated onslaughts. From understanding the threat landscape to implementing proactive detection measures.

Understanding Automated Attacks

Automated attacks are of different kinds depending on the intention of the attack. Broadly they all fall under the below categories:

• DoS/ DDoS attacks
• Force browsing / Aggressive recon
• Exploiting misconfigurations in infra

Note : For simplicity, We’ll refer AWS as cloud provider, you can assume equivalent services in GCP or others

1. DoS/DDoS attacks

Denial of service / Distributed Denial of Service are infamous attacks known for malicious attempt to bring down service by over-flooding requests usually one among these types :

1. Volumetric attack — Connectionless like UDP flood
2. Protocol attack — Connection based, like SYN flood
3. Application layer attacks —  Connection based, HTTP flood

While DoS attacks are typically carried out by 1 IP address, DDoS leverages an army of computers usually infected and controlled by a malware called botnet.

Detecting DoS attacks are fairly straightforward compared with DDoS, Notice the sharp spike in traffic in the below image :

Sudden bust in traffic under very short time is a primary indicator for a DoS/DDoS attack.

DoS Attack : Implementing a rate-limit would detect and also throttle the attack, if it breaches certain threshold

As the rate-limit threshold is breached at 08:55, you can see the decline in allowed requests in blue and increase in blocked requests in light pink.

Alternately, One can mitigate this adding that IP address in NACL

NACL : A network access control list (ACL) allows or denies specific inbound or outbound traffic at the subnet level

DDoS Attack : Detecting and mitigating DDoS is a tedious and complex process as it involves volumetric data analysis and composition of traffic based on the 30 day average baseline traffic.

Detection and mitigation through real time traffic analysis is offered by AWS Shield and AWS Advanced Shield services.

Shield Response Team (SRT) from AWS would also mitigate the DDoS attack in realtime by writing WAF rules based on the attack traffic observed, as they have 24x7 access to our Web Application Firewall through IAM role and lambda to protect our resources by writing shield rules on the fly, if required.

Indeed, the AWS WAF rate-limit is not impervious to DoS attacks. It periodically checks the rate-limit threshold every 30 seconds, providing a buffer window of the first 30 seconds of a 5-minute time frame to be exploited by attackers. This buffer is acceptable for DoS attacks but, if you have sensitive APIs like OTP generation etc., You can use code to construct a rate-limiter with microsecond granularity and take use of service-based rate-limits.

Here is the method we used : https://blogs.halodoc.io/taming-the-traffic-redis-and-lua-powered-sliding-window-rate-limiter-in-action/

2. Force browsing / Aggressive recon

Forced browsing is an attack where the aim is to enumerate and access resources that are not referenced by the application, but are still accessible.

In the spite of finding a hidden admin panel or backup file or some important file or internal application that should not be public, adversaries try to fuzz web applications with a huge wordlist based on the type of application.

Tools like ffuf and similar recon tools/ scripts are used for forced browsing and also for DOS attacks.

While rate-limit works greatly for force browsing too, it allows huge no of requests through as the rate-limit threshold looks for count instead of the malicious requests that carry a payload for XSS, SQL injection or similar vulnerabilities.

Use AWS managed WAF rules  — these rules cover the common payloads seen in  the wild. you can notice unwanted/restricted file extensions in the above image.

AWS also offers IP reputation detection from own threat intel feed to block. These are IPs with reputation for DDoS, Reconnaissance. One can leverage that intel through AWS Amazon IP reputation list.

Below are some of the AWS managed rules Halodoc one can use for protection against basic attacks :

Geo-restriction

Enabling geo-restriction would drastically reduce attacks from all other countries than mentioned.

3. Exploiting misconfigurations in infra

These could be misconfigurations in HTTP server, kubernetes, subdomains or even exposed API gateways.

Adversaries are constantly trying to monitor companies assets and look for change in the assets config or addition of new resources.  Shodan.io is one such example to track and monitor public assets of a company.

Here is the one among the most commonly exploited cloud vulnerabilities in the wild :

Well, it's hard to constantly assess and scan the ever changing cloud infra, that's why you need Cloud Security Posture Management tool (CSPM). There are few open source tools like prowler, service screener too.

We, at Halodoc, use Pingsafe as our Cloud-Native Application Protection Platform (CNAPP) which covers Cloud security posture management (CSPM), Kubernetes security posture management (KSPM), Cloud Detection and Response (CDR), Compliance Monitoring

Need for CSPM and threat detection

While CSPM tools help organisations identify misconfigurations, vulnerabilities, and deviations from security standards across cloud environment, Some key uses of CSPM include:

• Misconfiguration prevention
• Compliance assurance
• Risk Reduction
• Automation
• Continuous monitoring

The scans run on our infra for every 30 mins for critical misconfigurations, This gives our team an advantage in time against attackers to work on the spilled beans.

Here's another example where dropwizard metrics were made publicly available at http://host/metrics in a unfortunate incident. These go unnoticed usually as these are found after deployment and no checks to detect for infra misconfigs apart from pentest. In the below case, Pingsafe notifies within minutes for the exposed metrics soon after deployment as it checks for misconfigs every 30 mins.

Leveraging threat intel

Create a collection of threat feed and ingest into WAF to actively block IPs that are already marked as malicious by various trusted sources like shown below :

Threat feeds are updated continuously from a variety of sources, using information from honeypots, IOCs provided by reputable AV and EDR providers, and more. To ensure that your defences are always up to date with the newest threats, it is advised that you automate the process of fetching IPs from threat intelligence sources directly to WAF on a scheduled basis.

Below is the easy flow diagram to ingest IOCs to WAF.

Sample code looks like this

##Assume IAM role and add constants like URL feed, IPset ID

try:
    # Retrieve the list of IPs from the source
    response = requests.get(url)
    source_ips = [ip.strip() + '/32' for ip in response.text.split('\n') if ip.strip()] 
    
    # List IP sets
    response = waf_client.list_ip_sets(Scope='CLOUDFRONT', Limit=100)
    
    # Get the existing IPs in the specified IP set
    existing_ips = set()
    for ip_set_summary in response.get('IPSets', []):
        if ip_set_summary['Id'] == waf_ip_set_id:
            ip_set = waf_client.get_ip_set(Name='ipset-name', Id=waf_ip_set_id, Scope='CLOUDFRONT')
            existing_ips = set(ip_set['IPSet']['Addresses'])
            lock_token = ip_set['LockToken']
    
    # Find missing IPs and add them to the IP set
    missing_ips = set(source_ips) - existing_ips
    if missing_ips:
        update_response = waf_client.update_ip_set(
            Name='ipset-name',
            Scope='CLOUDFRONT',
            Id=waf_ip_set_id,
            Addresses=list(existing_ips.union(missing_ips)), 
            LockToken=lock_token
        )
        print(f"Updated IP set {waf_ip_set_id} with missing IPs.")
    else:
        print("No missing IPs to add.")

except Exception as e:
    print(f"An unexpected error occurred: {e}")

For wafv2 permissions, use a lambda with an IAM role or Jenkins, scheduling the job on a weekly or monthly basis depending on your threat feed update frequency. By banning these IPs, your WAF is automatically shielded from prevailing assaults and constantly guarded as and when new IOCs are gathered by the threat sources.

These IPsets are linked to a WAF rule that has specific actions to allow, count, and block. The security policies associated with webacls will go into effect immediately.

As shown above, you can pull IOCs from reliable sources and feed it to your IPsets.

Monitoring and Alerting

Having monitoring dashboard and alerting is very critical for any visibility into these threats and their mitigation.

Here are such logs that you could enable :

• VPC flow logs -  enables you to capture information about the IP traffic going to and from network interfaces in your VPC and AWS has managed services like guardduty that help in detecting any abnormalities in the inbound and outbound traffic like bruteforce attempt, DOS attack.
• Cloudfront logs - Primary logs that has record of IP address of the users and aides in investigating any attacks
• WAF logs - These are optional but comes in handy while investigating any WAF rules and their threshold breaches and analysing based on WAF rule through cloudwatch metrics
• Application logs - Necessary for debugging your application behavior. Primarily for 4xx, 5xx errors and circuitbreakers.

Below is our new relic dashboard for application logs. This helps in monitoring for 4xx errors and spike in any specific API and to continously improve the WAF based on these logs.

To get alerted for any abnormalities, in this case we can take an example of rate-limit threshold breach/ change in state of the ALARM.

Alerting

Architecture

Steps to enable alerts

1. Create a Rate Limiting WAF Rule for IPs (includes all requests)
2. AWS WAF will send a signal to CloudWatch metrics whenever there is a blocked/counted requests
3. This will show the rate-limit rule metrics.
4. Create an alarm for the rate limit breach threshold (In our case, set the alarm metric a bit higher than the actual threshold for alerts)
5. In the notification channel select one SNS for email and another SNS for triggering lambda.

This is an email alert from cloudwatch when change in state of ALARM

You'd get something similar for WAF metric when the threshold is breached for further analysis to block the IPs.

Conclusion

In conclusion, the battle against automated attacks necessitates a multifaceted defence strategy. By delving into the intricacies of DoS/DDoS attacks, forced browsing, and cloud misconfigurations, we've highlighted the efficacy of deploying AWS shield, managed rules, rate-limiting, CSPM, and geo-restriction for CloudFronts. This layered security approach forms a robust defense matrix, emphasizing the importance of constant monitoring, alerting, and proactive measures. As we navigate the dynamic digital landscape, the amalgamation of these measures not only fortifies our systems against existing threats but also positions us to adapt and repel emerging challenges, ensuring the resilience and security of our online environments.

Bug Bounty

Got what it takes to hack? Feel free to report a vulnerability in our assets and get yourself a reward through our bug bounty program. Find more details about policy and guidelines at https://www.halodoc.com/security

Join us

Scalability, reliability and maintainability are the three pillars that govern what we build at Halodoc Tech. We are actively looking for engineers at all levels and if solving hard problems with challenging requirements is your forte, please reach out to us with your resumé at careers.india@halodoc.com.

About Halodoc

Halodoc is the pioneer of a digital health ecosystem with a mission to simplify healthcare access by addressing users’ pain points through accessible, reliable, and quality services. The company’s commitment to promoting wellness is reflected in the comprehensive range of health solutions, covering preventive to curative approaches, all within a single application.Since 2016, Halodoc has been enhancing health literacy in Indonesia through user-friendly healthcare Communication, Education, and Information (KIE). Our expanding ecosystem offers a range of convenient healthcare services, including Home Lab for home care health test; My Insurance for seamless access to cashless outpatient service benefits; Chat with Doctor teleconsultation with 20,000+ licensed doctors and health workers; as well as Health Store for access to purchase medicines, supplements, and various health products from 4,900+ trusted partner pharmacies. Halodoc has been recognized as one of a telehealth platform making a positive impact on the healthcare sector and received the “supervised” status in the Regulatory Sandbox program by the Ministry of Health of the Republic of Indonesia. It reflects a strong commitment and partnership between Halodoc and the MoH, ensuring participatory supervision to safeguard Digital Health Innovation (IDK) organizers, consumers and healthcare workers as the partner of digital innovations. The platform has also received a string of prestigious awards, including being featured on CB Insights’ Digital Health 150 list in 2019–2020 and receiving the Indonesian government’s 2023 PPKM Award.Download Halodoc app via iOS and Android.