wp-targetSo I come in to work this morning and notice that one of my WordPress sites is having problems reliably coming up.  I had not received any alerts on my phone, but still, around 1/2 the time the main news-feed page wasn’t coming up. If you ever see a system that’s normally rock solid, suddenly running “a little funny”.. don’t put it off.  Jump on and do a bit of poking around. I did that today and this post is documenting what I found, what I learned and what I had to do to secure my system.

NOTE:   Don’t let the fear and dread of Linux security keep you from jumping in and learning something new! If you have a little Linux command line knowledge (or are willing to learn a little more), and love solving mysteries — then keeping your own dedicated WordPress system(s) isn’t that hard. Not only that, but investigating and fixing issues like this can actually be fairly educational, and kind of fun! (for geeks :)

The Problem:

xmlrpc-attacks

[1] WordPress XML-RPC attacks have skyrocketed in late 2015.Ref – https://goo.gl/uf6nJp

If your dedicated WordPress site suddenly seems sluggish, starts throwing errors like “Error establishing a database connection” or “Connection Refused”, you see the load levels (from the “uptime” command) spiking to 3-5 (or higher), you’re seeing tons of unexplained mysql and httpd processes on the system, or you’re getting complaints from your hoster about “DNS flooding”.. if you see any of these symptoms then there’s a good chance that your WordPress install is probably being used in a WP botnet attack of some type.  Specifically, if you’ve never “locked down” your WP site in any way it’s probably being used as a part of a xmlrpc.php “pingback” attack by one or more botnet attack farms.  These types of attacks have skyrocketed since 2015 as WordPress has become more and more popular, and fewer and fewer people running it are keeping their systems locked down and secured.[1]

If you think your WP site is being used as a botnet attack node, don’t worry, this does not mean your system is root or user-level compromised (i.e. you’re probably not “hacked”… yet).  Chances are your WordPress site is simply been recruited into a botnet army by a malicious third party (often from hosts in Russia, Korea or Africa) by way of a terrible bit of local WordPress code known as xmlrpc.php that comes on all default WP installs.

Side Note: Dynamic content security is one of the many reasons I personally don’t like running dynamic server side web-content on my production systems. But hey, that IS what WP is for. So if you’re going to run it, do so a dedicated cloud VM or something.. not a real physical box.

 

Solution:

First:
sudoGet logged in and do a “sudo netstant -antp” to see who’s remotely abusing your machine. The malicious system is often a single machine (IP address) from Korea, Russia or Africa that will stand out as 98% of your httpd traffic. Let’s start of by temporarily blocking the abusers with iptables to get your box back to a stable state like this:

NOTE: It depends on your flavor of Linux.. but one centOS/RHEL I just “sudo su -” and then put everything from the “-A” on in my /etc/sysconfig/iptables file and restarted iptables . Also, rule ordering is critical, and you may also need to switch from the “firewalld.service” back to using the more traditional iptables.service if you want to use iptables like this.

You’ll also probably want to remove this IP based block(s) after you perform the other fixes below.

After blocking the attackers with iptables, you’ll probably want to restart the httpd service (e.g. on CentOS7/Ubuntu, this would be with “systemctl restart httpd.service” ) as well as your mysqld.service or mariadb.service.

Give the system a minute to settle down and then check your system load with “uptime” command (any normal system load should drop back down under 1.0 after doing these things), and verify those abusive connections start bleeding off with “netstat -antp”. Once you’re stable again and no longer being abused, go on to the second step.. getting a real fix in place.

 

Second:  (not recommended, unless you’re in a real bind)
No-EntryOne quick and dirty fix to stop deviants from abusing your system is to locate the file “xmlrpc.php” and simply rename it something that can’t be used by the attacker, such as <path>/xmlrpc.php_BAK . Start off by fully “su -” or “sudo su -” ing to root. Then locate and take care to move the target xml-rpc.php file.

While this is rather quick and dirty (and can cause some WP pingback functionality problems), it is also very effective if you’re struggling to just stay above water, logged in and able to execute bash commands.  If after renaming the file, if you’re still really having problems staying logged in or feel like you’re fighting the system for resources, then at this point you can simply reboot to drop all connections and free up all your resources and log back in to a much more well behaved system.

 

Third:
Some people like to use this as an opportunity to see who’s using the xmlrpc calls (in their logs) and then either block those hosts, IPs and networks, or do a little security sleuthing.  If you’re in to that and want to run your system as a honeypot, that’s fine. Just put an .htaccess file to the same directory in which your xmlrpc.php resides and add this to the end:

and then your attackers will be both blocked and attempts recorded in your httpd log files (usually in /var/log/httpd/access* by default). Just grep your logs for xmlrpc.php and have fun! :)

Another, very powerful DIY method for blocking brute-force attacks on your /wp-admin login is to password protect the login screen. Yes, a login to get to the WP login.  It sounds like a pain, but it keeps thousands of bot-nets from hammering your site and doing “dictionary attacks” hitting your /wp-admin page over and over and over.  This is done old-school style by either creating or adding everything below (after the “…”) to your /var/www/html/wp-admin/.htaccess file:

Once you add this, simply create/populate the referenced  .htpasswd file by doing this from the command line as root:

No need to even restart apache. Just hit the /wp-admin page and you should be prompted with an http-basic-auth method.. showing that the hackers and bot-net farms will no longer automatically be able to brute force your site’s admin accounts.

 

xml-pingback

[2] The preferred fix. Just do a plugin search for XML-RPC and install this module.

Fourth:
The other more popular way is to address this at the WordPress layer by simply installing a WP DoS plugin such as “Disable XML-RPC Pingback“[2] (or one of the similar security plugins) and just be done with it.

 

 

bps-logo

[3] BPS and BPS-Pro are popular, all inclusive plugins that lock down against attacks, hacks, does alerts, logging, backups and more.

If you have not loaded any other, all inclusive security plugins, you might want to check one of the all inclusive security tools too such as “Bullet Proof Security” (PBS) or BPS-Pro. or similar site security plugins[3]

 

In Closing:
Running WordPress or any dynamic-content/CMS’ (Content Management Systems) website includes a measure of a risk.  Modern CMS’ are very powerful, easy to set up and run, but with great power comes.. well, you know.  In short, no on-line system should be set up with out-of-the-box settings, not hardened, and left up as a fire and forget resource.. but especially not these dynamic-content systems. If you’re going to run a production WordPress system or similar CMS, you still need to adhere to traditional best practices of:

  • Implimenting reliable tested backups
  • Appying regular system and application patches (automated or manual)
  • Subscribing to vendor security alerts
  • Active system monitoring

On top of that, watch for anything “funny” or unexpected behavior, and always be willing to log in and give things a sniff. Who knows, you just might learn a thing or two.

What have you learned when securing your own systems?  Share it!
When we share information, we all benefit.

Share and Enjoy

  • Facebook
  • Twitter
  • Delicious
  • LinkedIn
  • StumbleUpon
  • Add to favorites
  • Email
  • RSS