Drupal Cryptocurrency Mining: What I Learned from Getting Hacked

I personally had the bad luck of being one of the first servers attacked by cryptocurrency miners, and I want to share what I learned.

To begin, it’s important to note, I am not a security expert. I am a web developer who likes playing around with everything related to web applications, servers, services, etc.

I arrived home one hour after the patch was released and found that a server I manage for personal and learning purposes had already been hacked. My server was converted into a crypto mining resource for Monero.

Here is why I didn’t just nuke the server and how I ended up cleaning the server while learning a bit in the process.

The situation

There are many Drupal 6, 7, and 8 sites being hacked due to the two most recent critical vulnerabilities in Drupal, collectively known as Drupalgeddon2.

SA-CORE-2018-002 and the followup SA-CORE-2018-004 were released between March and April of 2018 and since then, hackers have been leveraging these vulnerabilities to inject malicious code into the server to gain control the control of the server via command line, to do nefarious activities like mining cryptocurrencies or sending bulk spam emails.

While organizations would likely be wise to take more rapid/radical steps to deal with a server that's been compromised, my goal was to learn, take some basic steps to tighten up security where I could, and share what I've found.

Is my server compromised?

This is a common question but very difficult to answer in some cases. One obvious way to tell is by asking the legitimate owner if they’ve noticed a different behavior on their website. However, this is often not enough to notice. Cryptocurrency miners have caught on to ways to mitigate detection by throttling.

So, before we can answer this question accurately, a number of recommendations, tools and procedures should be considered.

First of all, are you doing everything you can to embrace preventive security?

Reduce services and libraries used on the server as much as possible because the more services and applications you have, the more vectors for security holes you can have.
Collaborative software is usually more secure than owned code. Drupal has a very dedicated team of security experts that evaluates, fixes, and releases timely updates to thwart all the menaces they can. When you install Drupal or other open source software with a large community of users, you can be sure your server is going to be well protected.
Be restrictive, not permissive. For example, close all ports in a server and just open the ones you need (80, 443) is more secure. Also, switching to non-standard ports for some services like SSH can be helpful.
Permissions and user accounts on a server really matters. Write permissions for apache user in the whole web directory, or god forbid, running apache as root, giving sudo access to the Apache user. Please don’t do that! Take your time to properly configure user and file-system permissions on your servers.

The second step is having tools and methodologies to review your server, detect anomalies and to restore the server (or at least the data).

Munin is a highly recommended open source monitoring tool to analyze server resource usage trends. Choose a monitoring tool and take your time to learn about it and use it.
Make backups and test them. A malfunctioning backup system is the same as not having backups at all.
Store your backups on another server. Having backups sitting on a compromised server that you cannot access is the same as not having backups at all. Amazon S3, Google Drive, FTP. Drupal 7’s Backup & Migrate module now has integration with Amazon S3, and the configuration is straightforward, so there really is no excuse not to do this.
Have a disaster recovery plan. A guide, tools, scripts to rebuild your server and data.
Your own experience and your own tools.

With the above recommendations in place, we can now ask the question again: Is my server compromised? The answer should be verifiable by simply checking your tools and looking for anomalies. In my case, CPU increased to 400% suddenly. I noticed this because my websites became very slow to load and the graphs from Munin confirmed this.

How can I clean my server?

First, does your server house sensitive information? And can you disconnect the server from internet while you have access to it? DO IT! Then proceed to execute your disaster recovery plan.

If you can’t — or don’t want — to disconnect, turn off the machine and execute the disaster recovery plan using your off-site backups.

In my case (for learning purposes only!) I decided not to disconnect nor turn the machine off. I’ll reiterate here that for critical web applications with real customer and site visitor data, you must have a proper disaster recovery plan prepared.

Analyze the server

My problem was CPU consumption, therefore I executed:

ps auxf

And later

htop

These commands tell me what are the processes running, who is running it (root, www-data,...) and what is the resource consumption its have.

I found an anomaly with this command:

sh -c php-fpm /apps/web_taes/web/sites/default/files/article/files/cFGb #[and a lot of parameters]

This was the clue that lead me to understand that the server had been hacked. The attacker was able to execute PHP scripts using the shell. In a correctly configured web server, Drupal will disallow php script execution in the public files directory with these .htaccess directives which can work with Apache in either mod_php or php-fpm. Double check that you have such configuration in place.

In my case, the hacker’s file upload was an executable, but none of the files in the public files directory should be executable. I was able to quickly find other malicious files by searching for executable files in Drupal’s files directory:

find /apps/web_*/web/sites/default/files/ -executable -type f

I found 3 more files similar to cFGb and manually quarantined them by moving them to a temporary folder in my user account outside the public web directories for learning purposes:

find /apps/web_*/web/sites/default/files/ -executable -type f -exec mv {} /home/myuser/tmp/{}_back \;

I copied the exact command, get the ips and domains used, and blocked the IPs and domains of the mining pools with iptables:

iptables -A OUTPUT -s POOL-IP-ADDRESS -j DROP
service iptables save

Buying time with a temporary solution

I knew that blocking an IP is useless in the long term, but the short term goal here is to buy more time to learn how the attacker is entering the server. To do that I wrote a small custom php script that runs a blockade against the cryptocurrency mining tasks while also figuring out from where the attacker is entering.

I configured this script to run on a schedule via crontab -e

1 * * * * php /home/myuser/clean.php >> /var/log/clean.log 2>&1

After a few days, the clean.log file was full of traces of infected files. Also the log revealed that just before the infected file was reported, a strange connection was being established:

tcp 0 0 x.x.x.x:80 x.x.x.x:57214 ESTABLISHED 3927/varnishd

tcp 0 0 x.x.x.x:54887 x.x.x.x:8008 ESTABLISHED 3972/varnishd

tcp 0 0 x.x.x.x:8008 x.x.x.x:54887 ESTABLISHED 19862/apache2

tcp 0 0 x.x.x.x:3333 Y.Y.Y.Y:49455 ESTABLISHED 26432/054bf8

CLEAN[2018/05/07-08:44:01]: Infected file: /apps/web_taes/web/sites/default/files/styles/max_660px/054bf8

The IP Y.Y.Y.Y was a server running outdated nginx 0.9 that was connecting to my server on port 3333. Blocking this IP as well as port 3333 with iptables stopped the attack. Is this a definitive solution? Obviously not! Attackers typically use a network of zombie servers, like the Y.Y.Y.Y server, to do their bidding. Blocking one IP and one port will not protect me forever, just for some time.

Conclusions

Drupalgeddon2 may be among the worst exploits being leveraged by hackers because the code to reproduce the first part of the vulnerability was published, leading hackers to get a clue about other ways to exploit. There were two updates required within a couple weeks of each other so if you were not on-top of patching your sites or updating Drupal in both cases, your site may have easily been targeted.

I maintain this server as my personal lab with the ports opened and have a couple Wordpress and Drupal web applications. This server has been attacked a few times in the past, for a variety of reasons:

Apache process running as www-data had write permissions in the entire web directory, resulting in Drupal code being modified.
PHP scripts were allowed to be executable inside public web directories.
A Drupal installation running on an old version 7.19 *after* Drupalgeddon.

Here are a number of lessons I’ve learned along the way towards recovering the server:

Preventive security is a must.
Taking time to upgrade all applications and services helps to protect against attacks.
Being restrictive by blocking all ports except 80, 443, 22 probably would have prevented this attack. Please, don’t use port 22 for ssh. Close it, or at least change it.

Hardening file system permissions prevents from attacks that target modifying or injecting malicious code into Drupal code.
Using version control system like git for the application codebase on the server lets you quickly see if files are modified on the server and allows you to revert changes if needed.
Properly configured .htaccess files in the public web directories prevent attacks that allow remote execution of malicious PHP files uploaded to the server.
Public Zero-day vulnerabilities are published for free to metasploit almost on the same day.
In case that isn’t enough to scare you, there is a very lucrative business of for-pay zero-day vulnerabilities to be bought on the deep web that already exists but are not detected or used yet.
Once you’re server is compromised, your IP address is very likely to be stored “somewhere” accessible to hackers, heightening the likeliness of being among the first to be attacked in future exploits.
Rebuilding the application, nuking the server, and changing the IP address will always be the best solutions to recover from an attack and prevent being targeted in the future.
Using Platform-as-a-Service providers like Platform.sh, Pantheon, or Acquia, combined with Content Delivery Networks like CloudFlare, CloudFront, or Fastly will often have automated solutions in place within minutes of a security breach, for example, to sanitize URLs and mitigate distributed attacks. These service providers are a great investment for critical business applications.

Have you been hacked? Do you have your own tools or methodologies you want to share? We’d like to hear more. Please add a comment below or contact us at info@bluespark.com.