Me puppet master: Behind the scenes of crawling P2P botnets

At Kleissner & Associates we run world-wide operations on detecting virus infections. We put quite some efforts in detecting viruses – even for elder threats such as ZeroAccess 1 which is from 2011 and was superseded by ZeroAccess 2 in 2012.

ZeroAccess 1

We finished the implementation of the ZeroAccess 1 crawler already in December 2013, however had no initial peers. The embedded peer lists in the samples are of course outdated. The crawler needs some initial peers to find it’s way into the p2p network. We tried reaching out to our arrogant colleagues from Dell SecureWorks which, however, never responded.

The solution to finding initial peers was port scanning. We scanned the entire internet (at least the IPv4 space) for IPs with open ZeroAccess 1 ports. The result list of this scan was then fed to the crawler which contacted every IP and asked for a peer list update. We used masscan, a great tool by Robert Graham to scan the internet. The scan time for the entire internet was about 18 hours (the network devices in the datacenter didn’t allow a faster speed even though it would have been possible with the internet line and the server itself).

For most ZA1 botnets (there are 6 ones with each a different port) we actually only scanned part of the internet, as we only require 1 IP that is hooked into the rest of the p2p network. After a couple of abuse complaints to our datacenter and scanning 6% of the internet we found 110.029 IPs with port 34354 open. Out of them, there were 178 real infections. They were more than enough to hook the crawler into the p2p network.

As of right now there are still 31.428 ZeroAccess 1 infected machines:

ZA1 2-18-2014

It is interesting to see that for 2 ZeroAccess 1 botnets (“ZAv1 25700” and “ZAv1 22292”) our colleagues last year discovered less bots than we did now (the picture is copypasta from the p2pwned report):

p2pwend

Initially, before we proceeded with massive port scanning we tried bruteforcing peers by just contacting IP after IP. That process, however, turned out to be way too slow and impracticable.

BgEILC3CAAACw0s

ZeroAccess 2

Since the criminals reportedly abandoned their botnet we see major peer decreases every week. While it was a little bit over 1 million at peak in December 2013, it reduced down to roughly 500k within few weeks and since last week it dropped down to 455.479 peers right now:

ZA2 2-18-2014

One thing that is easy to do on ZeroAccess (both versions) is peer list injection (/poisoning) and therewith potential ddos. It would be a piece of cake injecting for example cia.gov (198.81.129.107) as a peer into every peer list and letting the botnet do the rest (flooding/ddosing cia.gov). This type of attack is very much comparable to DNS amplification attacks, in which other peers unintentionally do the work and multiply the attack.

Peer list injection though is a feature by design. Peers share each other peer lists so that they maintain connectivity. In fact, we also use peer list injection to “send” one peer to another crawling server. We run distributed p2p sinkholing to maximize the chances of catching any peer and provide at the same time redundancy.

ZeuS Gameover

Thanks to our colleagues from Dell SecureWorks and Crowdstrike who launched multiple disruption attacks against different botnets including ZeroAccess and ZeuS Gameover, the criminals added quite a lot protections to their bot code in order to protect the botnet from crawling, disruption, forging requests and replay attacks.

While we appreciate any efforts on taking down criminals and making the IT world more secure, we ask anyone to take sustainable actions. Just like Microsoft taking down a huge list of domains (and not even checking if the domains are actually sinkholes by researchers) it makes no sense doing a PR stunt like disrupting a botnet for a week. What happens is what see with ZeuS Gameover: The criminals enhance their code and and ultimately any monitoring (and detecting the infections) on the botnet might be jeopardized.

Ultimately the criminals running the botnet have to be arrested and at the same time the bar of the technical and financial obstacle of running a botnet have to be raised to prevent others from even thinking to engage in such activity. Otherwise the criminals will just continue with their activity. And that’s why you shouldn’t disrupt botnets.

Here is how ZeuS Gameover secures itself (some information was collected out of other’s research papers, some from own research and reverse engineering):

  • To contact a peer you need to know the IP, port and the bot id (20 byte identifier). The bot id is a SHA-1 generated out of the computer name + volume GUID of the first hard disk.
  • ZeuS uses random ports, that’s why you cannot use port scanning to detect possible infections
  • Packets are encrypted (RC4) using the receivers bot id. Without the id you cannot decrypt the packet.
  • IDSes cannot make signatures over the packet, as contents, port and size always change. Each packet has a random amount of random bytes appended to randomize the packet.
  • If there was no updated within 7 days, ZeuS Gameover falls automatically back to the DGA, that was done against peer list poisoning
  • Internal peer list is limited to netmask 255.255.255.128 per entry (prevents poisoning with ips from the same range)
  • There is a blacklist of subnets on the configuration. There is also a dynamic blacklist to limit connections to 10 packets/minute.
  • The most ingenious feature in ZeuS Gameover to prevent crawling: Peers only return peers in the peer list reply that are xor-nearest to you (to the bot id). So no matter if you contact 10 peers, or 100.000 peers you are likely to *always* get about the same “neighbour” peers. Because every peer knows you with your bot ID and uses it to encrypt the packet, you cannot simply change it.
  • Annoyingly ZeuS Gameover stores and uses internally always only the RC4 keystate, which means you cannot (easily) retrieve the RC4 key itself. You can use, however, the keystate for encryption/decryption or try to find the point where the keystate is initially generated out of the key.

Of course we found a way around that limitations – that is, however, a company secret. We can reveal, however, that our crawler has the bot id “KLEISSNER ASSOCIATES” – how cool is that.

ZeuS Gameover technical analysis

Interested in how to analyze ZeuS Gameover? Here’s how, step-by-step:

First of all you need from somewhere the ZeuS Gameover sample. There are public malware sharing sites (for example http://virusshare.com/). Once you got the sample, use API Monitor to set a breakpoint on WriteProcessMemory. That function is together with CreateRemoteThread very often used to inject malicious code into other processes.

ZGO 0

ZeuS Gameover will (like the normal ZeuS versions) copy itself into the %AppData% directory and spawn a second process. When the breakpoint is hit you can use directly API Monitor to investigate the memory. You can quickly see that the source buffer contains the MZ signature (of the PE header). By using HxD (a free hex editor) and the values of lpBuffer and size you can easily manually extract the memory and store it to a file.

ZGO 2

ZGO 3

The exported exe is already relocated, so the gaps (between raw start of section and virtual one) have to be removed manually. It’s a little bit tricky as you also have to take care if virtual size > raw one (deleting the extra memory), but if you manage to do it properly the import directory is rebuilt nicely:

ZGO 4

The executable is now very easy to analyze. As there are no valid relocations though, you will have to tell IDA manually to load it at address 005A0000h:

ZGO 5

ZGO 6

Then.. tadaa! Easy to analyze:

ZGO 7

If you trace back “GetVolumeInformationW” (which is required for generating the bot it) you can easily find the function that gets the volume GUID:

ZGO 9

It is even more interesting to trace back recvfrom(), as the function is used to receive UDP packets.

ZGO 10

In OllyDbg you can attach to taskhost.exe (ZeuS Gameover always uses that process for I/O) and set directly a breakpoint on it:

ZGO 11

Using taskview you can quickly find out on what port ZeuS Gameover listens for incoming packets (and it’s also a good verification if it runs correctly):

ZGO 12

For testing you can use the Packet Sender and send a dummy packet to your local ZeuS Gameover infection. As you can see in the second picture on the below right frame the code receives the packet successfully. The base code addresses in the picture is different because the machine had to be restarted during analysis due to changed network adapters.

ZGO 13

If you trace the code back in IDA you can see the outer function, ReceivePacket2:

ZGO 14

A very important function called in DecryptPacket is VerifyPacket which uses heuristic checks to check whether the packet is valid after decryption or not.

ZGO 15

ZGO 16

If you patch this verification check you in memory you can force your ZeuS Gameover infection to accept old unecrypted messages – which means that you now only need to know the IP and port. No longer the bot id. That is ideal for getting an initial list of peers.

ZGO 17

Then, after little of this and that, there’s the full-featured ZeuS Gameover P2P crawler which is hooked into the p2p network:

ZGO 18