Guide to Disaster: How The Tech Team Handled WordPress Security Flaw

By now, the news has spread rapidly in security circles and on mailing lists about an exploit to the WordPress software less than or equal to version 2.1.3. To give you some background, we had held off on upgrading to version 2.2 that came out last week due to bugs in the software that we felt were unacceptable to our company. Nothing critical, but as we are nearly finished rolling out new themes that are all widgetized to the network, I felt that lingering widget bugs were pretty critical to our platform. The decision was made on release day that we would not upgrade until WP 2.2.1 was released next month sometime.

That was the plan as recently as Monday evening. But something changed quickly and I want to give you a window into how the team worked together to avert a crisis. As this timeline is fairly raw, I hope it gives some perspective on how we are able to react and triage situations quickly and put issues to rest all the time. We don’t always have critical security flaws, but we do work together to problem solve on a daily basis. This is how we roll.


Monday, May 21 – 10:52PM EDT
An email is sent to the WordPress hackers mailing list alerting the community of a posted exploit to all versions of WordPress under version 2.1.3.

Monday, May 21 – 11:17PM EDT
Exercising caution as with all security alerts, I carefully setup a test and run proof of concept script against one of our blogs. Threat confirmed.

Tuesday, May 22 – 12:07AM EDT
I forward the notice to the tech team for them to digest in the morning.

Tuesday, May 22 – 8:21AM EDT
Brian Layman confirms threat and indicates that our upgrade timeframe decision has been made. I agree.

Tuesday, May 22 – 9:22AM EDT
Sean Walberg, our systems administrator, suggests we delay upgrade until peak traffic time is passed. Already, we were under a Digg storm and we did not need to exacerbate issues with an upgrade.

Tuesday, May 22 – 9:40AM EDT
Channel Editors notified of the problem and the impending upgrades and are given instructions to change passwords after the upgrade.

Tuesday, May 22 – 2:30PM EDT
Brian Layman and I work up more verification of the exploit by analyzing and executing the code against further targets on our next work. Re-confirmed.

Tuesday, May 22 – 4:30PM EDT
Upgrade script and subversion repositories prepped for switch to WordPress 2.2. We chose revision 5505 as most of the widget issues we were initially concerned with were addressed prior to this revision. Core plugin set re-evaluated by team. Eliminated one plugin due to security.

Tuesday, May 22 – 6:00PM EDT
Upgraded Tech channel and verified functionality of widgets, in particular.

Tuesday, May 22 – 9:00PM EDT
Upgraded entire network to r5505.

Tuesday, May 22 – 9:30PM EDT
Support, support, support. Reports roll in regarding broken this and that – mostly having to do with plugins and widgets. Solve almost all except a weird database error on one blog.

Tuesday, May 22 – 10:40PM EDT
Major bug discovered – well, not major for WordPress, but certainly for us from a user experience perspective.

Wednesday, May 23 – 12:35AM EDT
Reupgraded network to r5520 which included further fixes for widgets.

All in all, because we have created tools and standardized everything we do, we are able to avert problems before they become problems. We do it all the time for big problems and small. Folks who run networks, whether blog networks like b5media or simply groups of blogs that are maintained by the same person or group can choose to upgrade blogs by hand, one by one, or sit on the problem hoping to not be attacked “until the weekend”, or they can take attacks seriously, use tools that assist in upgrading (Brian’s upgrade script is very good too) and be done very quickly and efficiently.

Our upgrade of over 200 blogs was completed in 30 minutes and 6 seconds – a slowdown from earlier reported times based on instituting a pause between each upgrade. Our time of execution from problem introduction to problem solution? Less than 24 hours.

Published by

Aaron Brazell

Aaron Brazell is a Baltimore, MD-based WordPress developer, a co-founder at WP Engine, WordPress core contributor and author. He wrote the book WordPress Bible and has been publishing on the web since 2000. You can follow him on Twitter, on his personal blog and view his photography at The Aperture Filter.

8 thoughts on “Guide to Disaster: How The Tech Team Handled WordPress Security Flaw”

  1. I still don’t understand why WordPress would rather force all wordpress users to upgrade to 2.2 (which is brand new, and really untested compared to “older” versions) rather than supply a patch for their existing user base. I didn’t want to upgrade to 2.2, but they are in effect forcing me to by not supporting a “legacy” version of their product which is not even 2 MONTHS old. That is just horrible product support no matter how easy it is to upgrade to 2.2, or what new features it offers.


    Kevin
    http://technogeek.org/

  2. Kevin, moving to WP2.2 is nothing like previous major upgrades such as 1.5 -> 2.0 or 2.0 -> 2.1. Functionally, 2.2 is pretty similar to what 2.1.4 would look like. Don’t let the versioning scare you.

  3. I would have also preferred 2.1.4, though I hear Aaron in that 2.2 is more of an incremental upgrade with the new update timelines.

    Many kudos to Aaron and Brian — I ran that upgrade script and in just a few minutes I made the move from 2.1.3 -> 2.2. Less stress and less work than what I typically do. Nice work, guys! I owe you both beer.

  4. More of a “what is proper product support” than being scared. I’m also feeling a little grumpy after following the thread about WP-3/PHP5 on the mailing list ealier this week. The fact that many were not even willing to discuss it because “its so/to far away” and “lets just focus on the next release” just left a bad taste in my mouth.
    I completed the upgrade (to the release version, not the SVN) and it was relatively painless. This is inspite of my own wierd setup involving symlinks and such. Of course my site is also very “young” with few plugins and a very basic theme.


    Kevin
    http://technogeek.org/

  5. Kevin- My advice, don’t get personally attached to that conversation. It happens every few months. Convincing the mass of developers that PHP5 is a worthy jump is a marathon, not a sprint.

  6. And a good many of us do agree that php5 would be a good addition and would allow us to do some very good things natively. BUT getting a concensus that the switch WILL happen would be next to impossible on that list this far in advance. The list just doesn’t work that way.

    A more plausible goal would be to get group agreement that WP3 could potentially have a PHP5 requirement AND that enhancements implemented in a way that require PHP5 will not be thrown out but added to a WP3 tag.

    Then people might grudgingly agree…

  7. I’ve finally managed to use the InstantUpgrade plugin and that sure has made upgrading faster. Once comfortable with this auto-upgrading, I still intend to try one of these all-site-upgrade scripts!

    But I halted in the middle of upgrading, because I was quite disappointed with version 2.2. Firstly, because it makes my Iimage Browser Plugin not work properly. Actually, there is no Secondly. I mean, I hate WordPress’s built-in image loading – and was quite fond of Iimage Browser.

Comments are closed.