Anatomy of a blog hack

So, last weekend I found out that my blogs had been hacked.

Actually, it wasn’t just my blogs, nothing personal involved or anything like that: the shared server space my sites were hosted on was compromised, and a good number of other sites and files were hacked as well. Based on what I can piece together, here’s what happened:

There were a number of sites on this hosting space that were running out-of-date versions of WordPress, and some that also had various other PHP code installed (NetOffice, Gallery 2, a few others). Any software that is outdated is potentially at risk to known exploits, but more worryingly, I found an old bit of PHP code on the server that was set up to run arbitrary PHP code for (I presume) some back-end admin processing, and ultimately I think this was what had been exploited.

And until I had found and killed this code, the exploit happened at least 3 times even as I was cleaning up the server.

The exploit itself, once I knew what to look for, was fairly simple:

  • In PHP files that were writable to the Apache webserver process, the code was altered so that any line containing an opening PHP tag (which tells the server to start executing the code after it as dynamic PHP until the closing tag is reached) looked something like this:
    From <?php .....
    To: <?php     eval(base64_decode('malicious code encoded here')); .........
  • When I copied this code to a sandboxed PHP environment and decoded it, it contained fairly simple instructions:
    • If the visitor to the site was coming from a Referrer—in other words, if they had clicked on a link from another site like Google search results, Facebook, someone else’s blog—they were redirected instead to a completely different site that presumably contained spam, or malware, or whatever.
    • If the visitor was coming to the site directly—they had typed the URL directly into the browser’s Location bar, or clicked on a bookmark—then they were passed on through to the site.
Because I normally type in URLs to my blogs directly, or click the “recently visited” link in Chrome’s list, I didn’t see the exploit at first. But as I was writing a blog post on The Brew Site on Friday the 20th, I was searching out a link to a previous blog post (gotta love Google for that) and when I clicked that link to pull up the earlier post, I was redirected to some site in Poland (or at least, with a Polish country code for the top-level domain).

Fortunately, I don’t believe this hack was in place for long, since I often search out links in this manner and would have noticed sooner: Sometime in the wee hours of the morning of January 19th was when the files were first modified is the earliest I can determine.

It took me a bit of time to figure out the exploit (at first I was thinking it was the Google 302 hijacking exploit), but once I did I was cleaning up files on my blogs by Saturday morning. I hadn’t yet had the chance to address the (many) other files and old sites on the server hosting space, so unfortunately my blogs got re-infected at least once more before I was able to kill the old files and update others. Most of my weekend (and part of the following week) was spent updating, fixing permissions, cleaning, and deleting files and sites.

For reference, a handy pattern for detecting this code in grep is:

grep -R -l 'eval(base64_decode(' *

(This should always work because you should never have similar PHP running in your legitimate code…)

Now, I keep my WordPress blog software (and installed plugins) up-to-date pretty religiously, and I try to keep permissions set appropriately. But a good number of files in each blog were infected even so—how? It turns out, even though a fair number of the core files that were originally installed (manually) had the correct Unix group (“<account>:users”) and permissions of 644 (rw- r– r–) and were untouched, I was also making liberal use of WordPress’s built-in auto-updating feature, along with automatic plugin installation, and at some point the files that WordPress were updating got set to the “nobody:users” group—the Apache webserver process. It was these files that were exploitable to the “nobody” Apache process that was being exploited by the other code on the server. (Along with the few files I had set to group-writeable as well.)

So, lesson learned. I’ve battened down the hatches, fixed the permissions on all the files in my sites, and have decided to forgo WordPress’s auto-installing and update features for now for good measure. And, I’ve finished up a (long overdue) move of my blogs to a new webhost with none of the legacy code possibilities that were extant on the original server. (Nothing against the original web hosting provider, I just needed a clean break with an affordable price.)

Of course, you all let me know if you still run into any problems, okay?

Tools of the trade

It’s been awhile since I’ve posted anything overly technical here, but it strikes me that a “snapshot” of what I do (for work) and how I do it (the tech) might be useful to some.

What I do is web development for Smart Solutions here in Bend. Smart Solutions is a web and software development company and the company essentially has three main divisions: custom software development, SEO (search engine optimization—I know, that’s another post), and web development. All these “divisions” work pretty closely with each other—there’s a lot of line-blurring, actually—but for the most part developing websites for clients is what I do.

The platform we develop for is Pixelsilk—the custom Content Management System (CMS) that Smart Solutions developed from the ground up (and is still developing). The marketing pitch is, it’s SEO-optimized, gives you full control of your HTML, gives you all the tools you need to interact with social media, etc. etc. etc. Move past all that and get to the meat of it, and the primary things I really like about Pixelsilk is that you interact with all of your content and data inside of the system (rather that working with offline files that need to be FTP’d to various places), there’s a powerful and comprehensive Javascript API (giving me the capability to extend the system in new ways), it gives you the ability to re-use code and libraries, and that it’s entirely web-based—-meaning I can work on a site from any browser.

I’m also the company’s defacto WordPress developer—yes, we host WordPress blogs in addition to Pixelsilk sites—and a few other PHP applications so I still get a chance to flex my PHP muscles every now and again. (Smart Solutions is otherwise a Microsoft and .NET shop.)

Of course, I use a number of additional tools to develop for the web, and that’s what this post is really about.

What I use is a mishmash of online and offline tools. In the “offline” category I make use of:

  • The GIMP, open-source graphics software. Free to download, and fairly powerful, there’s still a lot I’m learning about it, but I do most of the graphics work I need to accomplish with it. (Photoshop is the standard for the company, but I’m not versed in it.)
  • Microsoft Visual Studio, various flavors. Sometimes moving the HTML/Javascript/CSS into an editing tool is easier to deal with, and I frankly like the Visual Studio editing environment best of the various programs I’ve tried for these types of files.
  • PHP Designer. I actually use the (older) free version because, well, it’s free and does what I need, it’s fairly lightweight, and it has the same kind of keyboard mappings and editing environment as Visual Studio.
  • Notepad. Yes, a stripped-down plain text editor. You’d be amazed at how much I have this open.
  • FileZilla. Yes, sometimes you still need an FTP client, and FileZilla is a good free Windows client.
  • PuTTY. A great free SSH client, because I spend a non-insignificant amount of time on a *nix command line.
  • Apache/PHP/MySQL: Installed on my Windows boxen as test environments. Pretty critical especially when developing WordPress themes.

Online:

  • Google Chrome and Mozilla Firefox as my primary browsers. I actually use Chrome as my primary while at work and Firefox while at home; these are both highly standards-compliant web browsers and I know if I can get something to work properly in them, then that is in fact how it should work. Chrome has some great built-in development and inspection tools; in Firefox I employ a number of extensions.
  • Web Developer (Firefox plugin): A variety of pretty essential additions in toolbar format for all aspects of web development.
  • Firebug (Firefox plugin): Probably the #1 plugin I would recommend; it adds code inspection, network information, Javascript debugging and inspection, and all manner of incredibly useful tools—you can’t be a proper web developer without this installed. (Chrome’s built-in tools come pretty close to this.)
  • Page Speed (Firebug add-on): A fantastic add-on to Firebug that analyzes the overall page performance (using Google’s recommended benchmarks/tests) and gives you hints on what you can improve.
  • Header Spy (Firefox plugin): Shows HTTP headers on the status bar, useful for troubleshooting server information.
  • AFOM (Firefox plugin): Incredibly useful plugin for the Windows version of Firefox which fixes the memory leak prone to Windows Firefox.
  • Internet Explorer: Of course, you can’t develop for the web without checking your work in IE, and IE8 has a decent set of developer tools built-in—including the ability to switch between IE7, IE8, and Quirks modes.
  • W3C Validator: Because you want to make sure your site code validates and works properly, right?
  • jQuery: The best Javascript library out there. If I’m doing anything in Javascript these days, 99% of the time it’s using jQuery.

There is of course other tools I use that fall primarily under the heading of “my own sites” and are not necessarily web development per se: Google Analytics and Google AdSense are two examples. That’s probably another post.

This list is likely incomplete—I may have missed an item or two or three, and if I think of any I’ll update it. But this gives an idea of the various tools I’m employing currently and to a large extent what I’d consider the minimum number any good web developer should be using these days.

PHP contest: Texas Holdem

I thought this sounded interesting considering how popular poker is these days (you know who you are): PHP Editors is holding a PHP programming contest for a Texas Holdem game. I might try it out. It wouldn’t be anything like most commercial poker sites out there, but it would be an interesting programming project.

…Not unlike being back in school, writing a program for whatever computer course I’d be in. Those were the days; they were still teaching Pascal at the time. I remember writing a Hangman game (it mostly sucked), and an algorithm for storing shuffling a deck of cards (which might have been a precursor for a poker program).

Of course, handling and “shuffling” a deck of cards that only exists in a computer program is trivial. You simply need to have a structure representing the cards, and draw them randomly. (And a method for keeping track of what’s been drawn.) Each subsequent “shuffle” is simply a different random number set selecting the cards.

Two for the price of one

Just pointers to a couple of blog posts I enjoyed.

First, Chris reviews Burger King in a fun sort of anti-Bend Restaurants way. Plus, you gotta love it when someone puts so much effort into writing a review like this…

“Parked in lot?”, you ask, and yes I did because there are critical steps in eating your Whopper, and the first is, eat it when it’s hot and fresh. Yes, “Eat your Whopper while it’s hot and fresh,” is what momma always told me. Also the fact that the window view from the restaurant isn’t really much different helps too. The next tip I have to offer you is to put a few onion rings and some of the zesty onion ring sauce on your Whopper and then get ready for Whopper-Bliss.

The other pointer is to Jake and to his post about the PHP easter egg. I wasn’t aware of this particular quirk, either, but apparently PHP will output an image of a dog if you append a string to the URL appropriately. I do know of the phpinfo() “easter egg” that only appears on April 1—the PHP logo image is replaced with something goofy. But this other one is new to me.

Ning

By now the geeky part of the blogosphere and, er, web-o-sphere has been rocked by the announcement of Ning two days ago. Check it out. Their one-line description reads “Ning is a free online service (or, as we like to call it, a Playground) for building and using social applications.”

What this means is they’re hosting a service/platform that allows people to build their own social software applications… things like online voting/polls, dating services, bookmarks, review sites. In theory the level of complexity in creating these ranges from point-and-click Clone-N-Theme all the way up to Advanced PHP Developer.

Yup, PHP. It looks like they’re opening the doors to the system and letting you code the apps directly. “Uh-oh,” I thought. “I hope they have PHP sandboxed.” And sure enough, it says in the FAQ that they do. Sounds iffy to me (let’s just say I’m glad I’m not responsible for running this service!), though I’ll give them points for innovation and guts here.

Looks like they offer up an API for their developer environment, all running under PHP 5. And from what I’ve been seeing, they’ve assembled a team of some of the top PHP people out there to put this together, so that’s impressive.

It’s all very Web 2.0, especially with tags (and the annoying/clever convention of showing the relative popularity of various tags with different font sizes). I don’t know, it looks interesting, and it’ll be hot for awhile, but I gotta wonder just how valuable it is to have hundreds of crappy variations of “Which is cuter,” most of which were created as throwaway examples and abandoned by idly curious people (like me)…