Remind me to never Tweet something like this again:
I fired that off around 7:30am on Wednesday, you know, trying to be all positive and everything. “Setting the tone for the day” as those bright and cheery people say you should do.
Three hours later, my email, phone, text messages and Twitter DM’s starting lighting up with things like, “Dude, your site is throwing malware warnings” and “Why am I getting redirected to some Russian site when I go to your blog?”
And here we are, a full 2.5 days later…
As you can see (assuming you are actually on the site and not looking at your feed reader right now), things are quite different here. We’re now running on a plain Jane Twenty Eleven theme. That’s because once I found out the site was hacked, I started purging any file that might hold rogue code.
Nothing I did got rid of the redirection problem.
A clean install of the latest version of WordPress.
Nuking every plugin and reinstalling them.
Peeking into the MySQL database looking for oddities. Noting the bloat from running a database for over six years harking all the way back to the WordPress 1.5 era, I decided to obliterate the whole thing and do a restore from a recent backup (you DO back up your database regularly, don’t you?).
That was fun.
I had to split the backup file up because phpMyAdmin chokes on big files. And 1600 some-odd posts with 22,000+ comments and a boat load of photos makes for a choke-worthy file. (Thank goodness I stumbled across BigDump – a Staggered MySQL Dump Importer. It worked like a charm).
Trust me, dinking around in the database makes me nervous as hell. It’s terrifying to delete a database that contains the last 6.5 years of your blogging life.
Several hours into this madness, I started seeing this in Google:
Isn’t that special? There’s my friend The Google telling the world that my site may harm their computers. That lovely little warning was on every search result returned for PhoenixRealEstateGuy.com…
My first thought was… well, I can’t print that here, kids stop by some times. My second thought was, “This is not good.” It was time to call in the professionals.
Enter Sucuri.net. They remove malware and monitor your site for future issues. These folks are good. Just a few hours after dropping $90 (the best $90 I’ve spent in a long time) they had located, isolated and fixed multiple compromised files.
Or at least so we thought.
Fast forward several hours, I forget now exactly how many. The nasty warning in Google results were gone, the site wasn’t redirecting to Siberia. All seemed well in Phoenix Real Estate Guy land. And then, it all came rushing back. Redirections, malware warnings, flags in the search results…
Enter Day 2 of the nightmare.
What Went Wrong
I got back with the wizards at Sucuri and they immediately asked if I had other sites running on the same server. With their direction and guidance, I purged a bunch of old stuff from the server and updated what was left. They then scrubbed everything again.
If you are a regular reader you know I try a lot of different “blogsperiments” – testing new ideas, new sites, themes, plugins, you name it. Most of these experiments fail, some miserably. Some just fade away. A lucky few actually work and get incorporated on this site or others.
The problem is, I am woefully poor at cleaning out these failed and faded experiments. I just leave them laying around on the server. After all, they can’t hurt anything, right?
When the warnings and such returned with a vengeance, the message was clear. The hackers still had a way into my site and were hell-bent on destruction.
By leaving old, and neglected, WordPress installs around, I was leaving the back door to the server wide open for these… sub-humans… to waltz right back in and do their nasty deeds. Some of these sites were running very old versions of WordPress. Versions that had no business being on a live site/server. You see, hackers love to attack WordPress installs, because WordPress powers 14.7% of the top million websites in the world, and 22 out of every 100 new active domains in the US run WordPress (source – State of the Word 2011).
Douchebags Hackers like to find vulnerabilities in software that has a large user base. WordPress qualifies. That’s why the WordPress development team is so diligent about releasing new versions – not only to add cool new features, but to thwart hackers.
It is not WordPress’ fault I had live sites that were FIFTY FOUR versions (I kid you not) behind the latest release. Right now, some WordPress geeks are saying, “For the love of God man, 54 versions? You deserve having this happen to you!”
And they’re right. But honestly, I had no idea a hacker could get in through one site and use that to infiltrate another one on the same server. Hey, I’m neither a hacker or a developer, I’m just some dude trying to sell real estate.
Lesson learned: Update EVERYTHING you run. Or just delete the old sites (don’t forget to delete the database too!) I used to think it was OK to be a couple of versions behind. (Clearly being 54 versions behind is just stupid.) Never again. I will forever more update ALL my sites within 24 hours of WordPress releasing a new version. Especially versions that are released as “security updates”. Thinking, “My site is inconsequential, no hacker would waste their time with it” is also stupid. Hackers are stupid, we don’t need to be practically opening the door and saying, “Come on in fellas!”
And at this moment in time, we are clean as a whistle. Google has just removed that ugly “This site may harm your computer” warning from the search results (it may take a few more hours for those to clear completely) and we are back in business.
What I Learned
The afore mentioned keeping the server clean of ancient sites is the primary lesson. Do that. Stop reading right now and see what all you are running. If you have an old site that’s been abandoned, kill it off. If you’re like me and you also try things like adding a forum, putting in a server based chat, putting separate WordPress installs on subdomains, get in there and update the software or purge the files.
You should scan your site for malware regularly. Pay Sucuri Security $90/year and they will run scans every 6 hours automatically. And they perform malware cleanup for you. They are outstanding to work with. Good, fast, and cheap with superior customer service. Trust me on this.
Recommended: Host your main site by itself. Host your toys, tests and experiments somewhere else. Keep the money site separate from the test sites.
There are other things you can do to help keep the nasty people out. I’ve read a TON about WordPress security in the past couple of days. While I am certainly no expert, I know far more than I did two days ago.
I’m going to lock this thing down tighter than the vault at Fort Knox. Here’s a great post on WordPress security – Top 5 WordPress Security Tips You Most Likely Don’t Follow. Read it and heed it.
Things are still a little goofy on the back end. Most notably I can’t get the Facebook Comments plugin to activate. I likely deleted something I shouldn’t have. So for now, Facebook comments are missing. I may or may not bring them back. But that’s another blog post.
You’ll have to deal with this rather drab theme for a couple of days. In a fortunate stroke of coincidental timing, I’m really excited to announce that for the past several weeks I’ve been working with the world class team at Copyblogger Media to do a wholesale redesign of the theme TPREG runs on.
If you’re not familiar with Copyblogger Media and names like Brian Clark, Sonia Simone, and Brian Gardner, well, you should be. These folks are internet studs. Their lead designer I’ve been working with, Rafal Tomal, is a freaking genius and is the man who has served up designs for a few little sites like Problogger.net, Copyblogger.com, ChrisBrogan.com, Jay Baer’s Convince & Convert, and real estate’s own Better Homes & Gardens blog Clean Slate blog (and others).
Just to be in the same design portfolio with these folks is a ridiculous honor, and I can’t wait to roll out the site. It’ll be a BIG change, but it will set the stage for some really cool stuff to come.
Any day now, we’ll be a full-blown custom Genesis Framework site. Due to this hacker mess, we’re bringing it online a little sooner than expected, so it may be missing some minor functionality here and there, but it’s going to be awesomeness. This will cause a little downtime as I’ll have to switch Domain Name Servers over to the new host, but that’s downtime I have no problem with. This may happen very soon.
Thanks for Bearing With Me
This has been an excruciating couple of days. (Having to take an online traffic school course in the middle of all this crap sure didn’t help!) I appreciate all the folks that let me know about the issues and things they were seeing. Special thanks to Jon and Inna Hardison at HA Media for a call and offer of their considerable experience and skills! Sucuri Security along with Brian, Rafal and Derick at Copyblogger have been a godsend. And the support of my wife and kids, our agents and lots of friends as I lost my mind over the past few days helped save me from finding a bridge to take a header from.
I’m sure there are other broken things. But most importantly, the scary career-limiting warnings are gone. The site is clean. We’re back online with bigger and better things to come!