So over this past weekend tumblr had a “minor” security breach. Sure, maybe none of their users’ info was leaked, but what was leaked was basically tumblr’s entire website configuration which is at least as salacious, at least to geeks. Alright, at least to web app development geeks like what I am.
I had a good look at their configuration underwear (I’d be happy to show them mine if they asked 
) and was happy to see some overlap in approach (the good stuff – yay!) and a bit dismayed to see other stuff in there that seemed a bit hackish. But hey, their config runs a 10 million UV/month site so, it works alright.
The interesting thought to me is what to do when your site is this painfully and righteously compromised – as in, what to do *immediately*, in what order. We’re talking the full blown klaxon’s blaring, lights flashing, people screaming, dogs and cats living together, mass hysteria filled moments immediately afterward.
My hit list to prepare for such an apocalypse:
- If you can, stop the bleeding – take your site down, redirect to your sorry page – this doesn’t actually do much beyond easing a tiny bit of embarrassment. The horse has already bolted.
- You’re going to need to change allll those passwords and keys – as you add services for your site, keep a list of the URLs you’d need to visit in order to change those passwords alongside the credentials themselves in the same file – this saves you valuable seconds. If you have an accomplice, you can readily say “I’ll take the top 5, you take the bottom 5″ and halve your time.
- Make sure you use a common email address to sign up for all your site’s accounts – some of them will likely require you to confirm the changes by clicking on links in email and you don’t want to be logging in and out of loads of different accounts when the poop is slapping that fan.
Of course the best move is not to play this game at all – by making it super duper unlikely in the first place. Those in the web app business that haven’t experienced the colonic gripping fear that goes along with a feckup of these proportions, yet, kudos to you, you get to plan ahead for when it does. For all the many many others, myself included, who’ve been bitten by this to at least some degree, see if you think my list passes muster:
- Store your web app’s configuration settings OUTSIDE the web root – i.e. somewhere where your web server will not send it out on request as if it were a regular text file.
- Restrict access to this file as much as possible. Chown and Chmod to be as aloof as you can – for some situations you can even make your web app configuration readable only by root!
- Store all sensitive system information in only ONE file, vs spread around your app. Compromised is compromised but you’ll be more likely to secure something well if you only have to concentrate on securing *one file*
- Don’t include this settings passwords file in your standard code repository. Yes, it’s very convenient to have it there, but every time your code is checked out somewhere, so are your passwords. It doesn’t have to be that way.
That’s it. There’s always more, just like there’s never such a thing as total security. Had tumblr done all this, it’s not a guarantee they wouldn’t have leaked their data, but it would have been way way way less likely and way way way harder than some poor schmoe fat fingering a vi command. Voosh. Shudder. I feel that pain.