Solving the WordPress Traffic Overload Problem

Anyone who’s been around WordPress for a “Digg effect” or other massive influx of traffic knows that it can be a real problem. From a technical standpoint, the problem is that PHP is entirely loaded into memory for every pageload. That includes the 99% of PHP that is not being used to actually render the page.

On low traffic sites, this problem is not necessarily noticed. It doesn’t have a huge impact. However when there are hundreds of requests hitting a server in a single second, that kind of overhead builds up very fast.

There are solutions to this sort of thing and depending on what the scale of the environment is, some might be more excessive than necessary. The WP Super Cache plugin is a quick solution that will cause pages loaded on WordPress to be cached meaning that if subsequent page loads can pul the HTML from the cache without having to load the overhead of PHP as well, everyone wins. On the more extreme end, server configurations can be made to send requests for different types of content (for instance, images) to specialized servers optimized for that content type.

Very geeky stuff. It’s important to note that WordPress gets a black eye all the time for it’s caching mechanisms and ability to handle the load of a “Digg effect”, etc. In fact, Instapundit Glenn Reynolds is the latest to take a stinging swipe at WordPress and trust me when I say, we heard it loud and clear.

At b5media (where I’ll be leaving as the Director of Technology soon), we’ve had to deal with this as well and have managed to develop really sound solutions to some of these problems. However, for WordPress as a whole, it is a well recognized problem that not everyone can solve by following in our footsteps (or footsteps).

We’re going to do what we can to help solve this problem once and for all as two of our developers, Mark Jaquith and Brian Layman will be mentoring a Google Summer of Code intern to develop a robust caching engine for WordPress. We hope that this exercise will result in a more reliable (and sane!) caching mechanism.

Integrated Caching Solutions will improve WordPress’s speed and reliablity out of the box and allow people to “Digg Proof” their sites without the struggle of installing plugins on a site that is virtually unreachable. (Source: WordPress Google Summer of Code 2008)

Glenn, I hope that the work that Mark, Brian and our intern will be doing will improve the WordPress problem. In the meantime, let me know if I can help you with anything (though I believe you are using Movable Type). It is a known issue and it’s one that needs to be solved and hopefully some steps can be made toward that this summer.

Technosailor Maintenance Today

To give everyone a head’s up, at some point today, this blog will go into “maintenance mode” while I move to a different server. While Technosailor has been a part of the b5media network, I also announced my departure from b5 last week and so the blog is moving to a colocated server at Defender Hosting. They have been gracious enough to work out a fantastic arrangement with me, and I’m happy to have my own box to be able to fiddle around on.

So comments will be going off at some point to prevent data loss and mail might bounce sporadically. I hope to do this as smoothly and as transparently as possible though. Probably this evening.

Departing b5media

About two and a half years ago, I embarked on a journey of a lifetime. It was a breakthrough journey that took me from working in a cubicle at Northrop Grumman to living my dream building an internet startup. At some point, we took funding. I recruited Brian Layman and Mark Jaquith and the rest of the tech team. We moved from hosting a blog network on someone’s shared hosting provider to developing hard plans to encompass hundreds of servers in grid format.

We went from WordPress 1.5 and basic installs to over 350 installs of WordPress working in tandem with robust data sharing via APIs. We developed techniques to deploy advertising campaigns quickly and efficiently. We worked hard to build reporting tools, financial tools, management tools, and more – all in an effort to make this company work.

And it has and continues to grow at an astounding rate.

It’s been a great ride, but now it’s time to look forward and explore new territory. Recently, through my interactions with so many wonderful people in the DC technology community, the PodCamp community, the larger social media, business and technology communities, my appetite for something new and challenging has been overwhelming. I’ve had several conversations with Jeremy Wright, our CEO, over the past months exploring this stuff and he has been amazingly supportive, as has the rest of the b5media team.

An exact date has not been set, as I want to make sure a replacement is found and brought up to speed, however it will probably be within the next couple of months.

In the meantime, I’m looking forward to the next steps. At this point, I’m not sure what those next steps are but I have options. I’d like to develop Mokonji a bit more. I also have a few other venturistic ideas floating around in my head and I want to spend some time focusing more on the DC community that has sprung up around here. Obviously, I’ll continue to be blogging here at Technosailor, though in the short term, I’ll be exploring effective ways to monetize it – gotta pay for the server costs now (whew, forgot how that worked! ;-) )

I don’t know. The eager anticipation of the “what’s next” is crazy for me right now. It’s sort of like going out on a date for the first time. You don’t know what to expect but you know you really, really like the girl a lot.

Thanks for all your support for these years that I’ve been blogging. Looking forward to see what comes next! And if you want to be me (erm, the b5media Director of Technology), the company is actively looking for my replacement.