Managing the Log4j Zeroday vulnerability at Halodoc
At Halodoc, we take cybersecurity seriously. In this article, you will learn about Log4j zeroday vulnerability , Java Naming and Directory Interface (JNDI), its working and architecture, and remote code execution in Log4j functionality.
The Log4j vulnerability allows attackers to control java-based servers and launch remote code execution (RCE) attacks.
This issue only affects log4j versions between 2.0 and 2.15.0-rc1. In order to exploit this flaw you need:
- A remotely accessible public endpoint running any protocol (HTTP, TCP, etc) that allows an attacker to send arbitrary data.
- A log statement in the endpoint that logs the attacker-controlled data.
Due to the existence of JMS (Java Message Service) Appender which can use JNDI in the log4j 1.x, it is possible that log4j version 1.x is also affected by this vulnerability.
Before we start Log4j vulnerability, let's learn about the JNDI.
So what is JNDI and how it works?
The JNDI is an application programming interface (API) that provides naming and directory functionality to applications written using the Java™ programming language. It is defined to be independent of any specific directory service implementation. Thus a variety of directories services which are new can be deployed in a standard way.
Architecture
The JNDI architecture consists of an API and a service provider interface (SPI). Java applications use the JNDI API to access various naming and directory services. The SPI enables a variety of naming and directory services to be plugged in transparently, thereby allowing the Java application using the JNDI API to access its services. See the following figure:
Packaging
To use the JNDI, you must have the JNDI classes and one or more service providers. The JDK includes service providers for the following naming/directory services:
- Lightweight Directory Access Protocol (LDAP)
- Common Object Request Broker Architecture (CORBA)
- Common Object Services (COS) name service
- Java Remote Method Invocation (RMI) Registry
- Domain Name Service (DNS)
The normal user can conveniently and flexibly add values to the configuration at arbitrary places with the predesigned format by using this feature. In detail, when calling the log method in the application, log4j 2.x will call the format method to check the specific characters ${ in each log.
Should these characters be present, the Java method lookup will be called to find strings after the characters ${ and then replace the expression after the characters ${ with the real value found before.
So what is Log4j?
A Log4j, is an open-source logging library commonly used by apps and services across the internet. Attackers have identified a flaw that can break into systems, steal passwords and logins, extract data, and infect networks with malicious software.
How do exploits works?
For example, when calling the log function in the application to log the content the strings java:runtime, java:vm, and java:os after the characters ${ will be considered as the parameter of the lookup method and finally replaced with the corresponding values, such as Java(TM) SE Runtime Environment (build 1.7.0_67-b01) from Oracle Corporation, Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed-mode, and Windows 7 6.1 Service Pack 1, architecture: amd64-64.
How attacker performs an attack?
In the illustration below, we will learn how the attacker identifies and exploits the log4j vulnerability.
- An attacker inserts the JNDI lookup in a header field that is likely to be logged.
- The string is passed to log4j for logging.
- log4j interpolates the string and queries the malicious LDAP server.
- The LDAP server responds with directory information that contains the malicious Java class.
- JAVA deserializes (or downloads) the malicious java class and executes it.
How can we defend ourselves against this attack?
Apache foundation released a patch for this vulnerability and various OS vendors have released updated binaries with the fix and suggested that customers update their Log4j libraries versions. Halodoc has implemented the following approaches to prevent the Log4j vulnerability exploitation and reduce the security risk.
- Custom WAF Rules
- Upgraded/Replaced Log4j binaries for all the applications
- Disabled the loopback function for some versions of Log4j in use
Custom Web Application Fire (WAF) Rules
Using firewall rules on servers was a good mitigation technique to prevent attackers. If the server can make DNS lookups and attackers scan for vulnerable instances of log4j2 which will trigger the DNS lookup. Although attackers can easily bypass firewalls, having a firewall can block the outgoing connections of an actual attack and provide some degree of security.
As a hot fix AWS suggested that we implement WAF ACL rule “AWSManagedRulesKnownBadInputsRuleSet” but on the second day of the vulnerability the ACL rule got bypassed by the attacker.
To identify and block the WAF bypass mechanism of the Log4j exploit at Halodoc, we have added a custom set of rules in our WAF, which analyzes and determines the vulnerability pattern and stops any abnormal activity.
Upgrade/Replace Log4j binaries
The best fix against this vulnerability is to patch log4j library to version 2.17.0 or above:
Log4j 2.x mitigation: Implement one of the mitigation techniques below.
- Java 8 (or later) users should upgrade to release 2.17.0.
- Users requiring Java 7 should upgrade to release 2.12.2 when it becomes available (work in progress, expected to be available soon).
Other options will be, remove the JndiLookup class from the classpath: zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
** Note that only the log4j-core JAR file is impacted by this vulnerability.
Applications using only the log4j-api JAR file without the log4j-core JAR file are not impacted by this vulnerability.
Additionally, we have applied some patches on the application level where we have removed the vulnerable packages of the library and disabled all the unintended loopbacks of the library.
Disabled the loopback function
Simply removing the jar files will break logging via log4j 2, but this is probably the weakest remediation technique as it is intrusive and prone to error. We have removed vulnerable library packages from the applications and disabled all unintended callbacks from the library as a hotfix. Below is the example of the configuration.
FormatMsgNoLookups='true'Conclusion
At Halodoc, we strive to implement all standard security measures to avoid exploits. For Log4j, we have implemented patches on multiple levels of an application by disabling the callback and updating the version of the library. We also implemented, a set of custom rules in WAF. We have make sure that no information leakage occurred during this event and no data was extracted or got infected by malicious software.
About Halodoc
Halodoc is the number 1 all around Healthcare application in Indonesia. Our mission is to simplify and bring quality healthcare across Indonesia, from Sabang to Merauke. We connect 20,000+ doctors with patients in need through our Tele-consultation service. We partner with 3500+ pharmacies in 100+ cities to bring medicine to your doorstep. We've also partnered with Indonesia's largest lab provider to provide lab home services, and to top it off we have recently launched a premium appointment service that partners with 500+ hospitals that allow patients to book a doctor appointment inside our application. We are extremely fortunate to be trusted by our investors, such as the Bill & Melinda Gates Foundation, Singtel, UOB Ventures, Allianz, GoJek, Astra, Temasek and many more. We recently closed our Series C round and In total have raised around USD$180 million for our mission. Our team works tirelessly to make sure that we create the best healthcare solution personalised for all of our patient's needs, and are continuously on a path to simplify healthcare for Indonesia.
Join us
We are always looking out for top engineering talent across all roles for our tech team. If challenging problems that drive big impact enthral you, do reach out to us at careers.india@halodoc.com
We have a Security Responsible Disclosure Program and a dedicated security team that identify and report vulnerabilities in our products. If you feel you have identified any potential security threat our products or services, please feel free to write to us at security@halodoc.com.
Reference
https://en.wikipedia.org/wiki/Java_Naming_and_Directory_Interface
https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=Apache%20Log4J
https://www.prplbx.com/resources/blog/log4j/
https://aws.amazon.com/security/security-bulletins/AWS-2021-006/
https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-groups-baseline.html