Log4j: new software supply chain vulnerability unfolding as this holiday’s cyber nightmare

Nearly a year ago the security world worked through a first-of-its-kind supply chain vulnerability during the SolarWinds/SUNBURST incident response. Now at the end of 2021, a different type of supply chain vulnerability impacting our software supply chain is affecting systems across the internet — with exploitations identified in the wild. 

Updated 13 Dec: Added information on Mirai botnet usage of the vulnerability, more background information, and a warning regarding patching systems.

Updated 15 Dec: Added information on IronNet's deployment of Threat Intelligence Rules (TIRs).

The intent of this blog is to provide a short background of the event and the Log4j Remote Code Execution vulnerability.  This article is written by analysts with a cyber threat intelligence background and emphasizes network-based artifacts, but communicates at a broader level for those either leading/managing, developing, securing, or even using commercial, government or any Java software application that depends on open source code. Based on the scale and severity of this vulnerability, this article provides a very brief overview of the vulnerability itself, its impacts, potential attack vectors, and possible mitigation efforts as details of the extent of this incident unfold.

Background of the Log4j open-source logging API

On December 9th 2021, a remote code execution (RCE) vulnerability (CVE-2021-44228) with a CVSS score of 10.0 was identified in Log4J, an open source logging API for Java that “strives to be simple to understand and to use.” Although it was discovered through a bug bounty program for the popular game Minecraft by Chen Zhaojun of Alibaba Cloud Security Team, its effects reach far beyond the game.

Log4J is maintained by the Apache Software Foundation (ASF) and is commonly used in servers and development frameworks. In the last four months, Log4j has been downloaded more than 84 million times. Many of the Java projects maintained by the ASF use Log4J for logging, such as the popular Apache Struts project. Given the popularity of Log4J and Apache's projects, the list of organizations with vulnerable applications is staggering and includes organizations such as Apple, Twitter, Amazon, and ElasticSearch. Organizations may not even know they are running Log4j as it may be used by a third-party framework or toolset.

This particular vulnerability lies in Log4J's handling of JNDI (Java Naming and Directory Interface) and a service such as LDAP. JNDI allows Java to access naming and directory services, such as LDAP, DNS, and RMI. Most importantly, JNDI can be used in conjunction with LDAP to retrieve an object containing directory service information needed by a Java program, even if the object resides on a different machine. Log4J recently added support for JNDI property substitution in logging statements. This means that if an attacker can send a JNDI command that Log4J will log, it will be executed.

The challenge for the adversary then is to find a way to send a malicious JNDI payload to a Java application that Log4J will attempt to log. Note that simply sending the payload to an application that uses Log4J is insufficient. Log4J must attempt to log the payload for it to be executed. One common value that is logged directly is a browser's USER-AGENT, a value that can be easily crafted by an attacker.

The heart of the vulnerability, namely the fact that JNDI can be used to execute remote, malicious code was originally described in a BlackHat presentation in 2016: Twitter. While it did not describe using the vulnerability through Log4j, the vulnerability itself has been public for over 5 years.

Who is affected by the Log4j vulnerability and what's the impact?

The affected versions of the Log4j application are any versions from log4j v2.0-beta9 to v2.14.1. Java applications that rely on vulnerable versions of Log4j are widespread across a variety of verticals and range from mature enterprise services like Apple iCloud to common web frameworks like Apache Struts to online games like Minecraft. Apache has already released Log4j version 2.15.0, which patches the vulnerability. Any application that uses the updated version is no longer vulnerable to this exploit. A good resource that provides a sense for the scope of organizations and services that are confirmed to be affected can be found here: GitHub

The capabilities this vulnerability provides for attackers is not limited to simply dropping a second stage payload to a target. There have been POCs and observations of attackers using this capability to exfiltrate sensitive environment information from vulnerable servers without dropping any follow on payloads. This information can frequently include passwords which could potentially be used for additional remote access to a target network, separating the malicious JNDI exploitation from the follow-on attack. Given the combined scope of affected applications and the broad capabilities delivered, it is hard to overstate the severity of this vulnerability. 

Identifying in network traffic: What does it look like on the wire?

There are a variety of ways to trigger web applications to log the content of requests, and attackers can sometimes even target this vulnerability without having to directly send a malicious HTTP request as shown here: Twitter Thread. Given this, it is best just to assume that detections should be looking for any inbound requests to application servers that contain malicious JNDI strings in their content.

Crafting a functional JNDI string is relatively simple. There are now a huge number of variations, but below are two of the simplest and commonly observed protocols that are known to work for this exploit:

Then an attacker must append a malicious IP or domain name and path where they are hosting the second stage payload they wish to execute on the target server like so:

There are currently six Suricata signatures included in the Proofpoint ET Pro ruleset that are configured to look for the above two malicious patterns over HTTP, TCP, and UDP from any external sources. 

As always, attackers are clever and have already developed and implemented bypasses that do not allow for a simple regex detection of malicious JNDI strings. A good reference for the existing bypasses and how challenging this activity is to detect using regexm refer to this GitHub repo. Examples of functional JNDI strings that bypass most pattern-based detection techniques include the following:

To address the bypasses, ET Pro has released an out-of-band update to include some looser hunting-based signatures that are tolerant of variations in functional JNDI strings, such as these. Due to the less restrictive logic, these rules are likely to produce more false positives, but should still be enabled and monitored if environments are known to be vulnerable. 

Detecting simple attacks by the Mirai Botnet

One of the easiest ways to find a simple attack that does not use any bypass techniques, is to simply search HTTP user-agent strings for a malicious JNDI string.  In IronDome, this can be done with a query with the following syntax:

We executed the query for our internal infrastructure.  The results indicated that four sessions were attempting to exploit the vulnerability against the IronNet VPN.

After diving into each session's details, we were able to get more information about the HTTP requests.

The first session included the following malicious JNDI command into the user-agent field:

The base64 code in the path decodes to the following command:

The lh.sh script downloaded from the secondary server contains the following commands:

Using ReversingLabs, we determined that the three files are versions of the Mirai botnet.

The hash for each of the files is shown below:

IronNet’s response to the exploitation of the Log4j vulnerability

IronNet has employed a multi-pronged approach to address the ongoing exploitation of this vulnerability. First, hundreds of Threat Intelligence Rules (TIRs) have been deployed for the known malicious IPs attempting to gain RCE on targets. A great resource on IP addresses of malicious threat actors has been provided by GreyNoise and can be found here. IronNet has also observed a significant number of attacks against our own infrastructure, as well as that of our partners. We have used data from these attacks to produce unique intelligence comprising both the IPs used for scanning and the second stage domains and IPs used to deliver the malicious payloads. 

In addition to our Threat Intelligence Rules, IronNet has deployed all available Suricata signatures designed to detect the malicious JNDI strings, as well as any known bypass attempts, such as the ones outlined above. These should detect a majority of the standard exploit attempts, and even some of the more common evasive techniques, but should not be considered a complete detection capability for this activity.

Lastly and certainly not least, IronNet has prioritized behavioral detections surrounding the network traffic resulting from successful exploitations. There are a number of analytics that address this, but the Suspicious File Downloads and Domain Analysis HTTP analytics are best positioned to support detections of this activity. The majority of current attacks include an attempt to deliver to the target a second-stage payload, which frequently comes from a shady domain.

Additional analytics that are well aligned to detect the follow-on activity from a successful exploitation include DNS Tunneling and Consistent Beaconing. DNS Tunneling may not initially seem relevant to this threat, but these alerts should be considered high priority due to the ongoing exfiltration attempts observed where attackers are able to exfiltrate sensitive environment variables from servers by tunneling data within the DNS subdomain. This technique is detailed on Twitter here. Essentially, all an attacker has to do is insert the environment variable they wish to exfiltrate into the JNDI string as a subdomain. So as long as the attacker has an authoritative name server for the domain requested, they will be able to see the value of the exfiltrated environment variable. Here is an example below where the attacker is attempting to exfiltrate the value of the SENSITIVE_VAR environment variable:

What should you do now to secure your enterprise?

The best option to mitigate these attacks is upgrade Log4J to the newest version, as the vulnerability has been patched. However, that is not always an option as upgrading to the newest version may require upgrading Java itself to a newer version.

If upgrading Log4J is not an option, you can configure Log4j to not perform lookups or Java itself to not trust JNDI URLs.  If your infrastructure uses Log4j version 2.10 or newer, setting the system property log4j2.formatMsgNoLookups or the environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS to true will mitigate the vulnerability.  If your infrastructure uses Log4j version 2.0-beta9 to 2.10.0, simply using the following command to remove the JndiLookup class from the Java classpath will mitigate the vulnerability: zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class.  If none of these are possible, setting the com.sun.jndi.rmi.object.trustURLCodebase and com.sun.jndi.cosnaming.object.trustURLCodebase properties for Java itself to false will also address the problem.

Scoping affected applications/servers and confirming that the above patching strategies do, in fact, mitigate the vulnerability is a critical step in the process that cannot be skipped. The easiest way to begin scoping is to use a publicly accessible tool like the Log4Shell utility provided by Huntress. Utilities like this provide a functional JNDI string that will produce a callback on vulnerable servers, but only log the successful callback and stop.  They do not deliver a malicious payload. These utilities provide defenders with a rapid way to confirm that a once vulnerable endpoint has successfully been remediated, or even to discover if vulnerable endpoints exist within an organization.

If an organization finds that their installations of Log4j have been patched or the vulnerability has been mitigated, it is important to verify that it was done by that organization.  There are reports that threat actors have mitigated the vulnerability after establishing a foothold in an effort to better conceal their success.

From Log4J CVE disclosure to TIR deployment in 11 hours

IronNet has handled the Log4J vulnerability with the highest urgency. The response from our product team, threat hunters, researchers, and security operations professionals demonstrated world-class speed. For instance, 

• Our CyOC (threat intel and hunter team) deployed the first batch of 101 Threat Intelligence Rules (TIRs) within 24 hours. 
• As of December 14, 4:00pm EST, IronNet has deployed more than 326 TIRs specifically for this event. 

29 behavioral detections

IronNet deployed more than 29 behavioral analytics to detect various aspects of the exploitation behavior. From these analytics we identified more than 10,000 distinct exploitation attempts that correlated across our customer base using IronNet’s Collective Defense platform. Additionally from this output IronNet Identified additional  IOCs to create targeted rules to better protect our customers. IronNet’s threat hunters, combined with our NDR and collective defense capabilities, give IronNet the unique ability to investigate historical network traffic in this way.

IronNet hunters and analysts are monitoring both partner network infrastructures and findings from other researchers to stay up to date.  

To stay connected with the IronNet threat research team, visit our threat intelligence page.