I’ve been running an experiment for the past three months. I wanted to see how well WordPress would do if I ran it by itself, without any sort of caching. So far, so good.

About four months ago, my web server kept getting pummelled into the ground almost daily, and I couldn’t figure out why it kept happening. After researching the issue, I found the prevailing opinion to side with the need for a caching plugin. People were complaining that it’s just not optimized well, and must be run with the aid of such a plugin, otherwise higher levels of traffic will bring the web server down. Trouble was, I already ran my WP install cached, using WP Super Cache, had been doing so for over a year, and my server still went down. (I should specify it had only recently started to go down.) What was I to do?

I posted a message in the WP forums asking why WordPress doesn’t generate static files. Were there any plans to do so in the future? To my surprise, Matt Mullenweg (WP’s founder) replied to my post, and told me that while there are caching plugins out there, WordPress.com doesn’t run any, and they’re doing just fine hosting millions of blogs. Others chimed in as well, and their replies got me to make the following changes:

1. Made the switch to a VPS (Virtual Private Server) with SliceHost. Four months later, I’m still very happy about that move.
2. Doubled the RAM on my web server (to 512MB from 256MB).
3. Turned off WP Super Cache and started running my site by itself.

Each step followed the other in succession. I wanted to make gradual changes so I could see why my server kept having issues. Switching to a VPS host was good, and it was needed, but for my traffic levels, it wasn’t enough. Doubling the RAM was good and it was needed, and while the new RAM is enough for now, I’d still be having problems if I didn’t also disable my caching plugin.

Here’s where I think the crux of the caching/non-caching issue lies: it has to do with the load placed on the server as cached versions of the pages get created. Normally, that’s a non-issue. But as I monitored my server carefully, I discovered that it went down only as it started to get indexed heavily by search engines. Their bots visited my site in spurts, with traffic peaking, then falling back down. They spawned multiple threads, over ten at times, following links and slurping up the content. It’s when bot traffic peaked that an incredible load was placed on the web server. It kept generating cached versions of pages it hadn’t already cached, RAM and CPU demand increased to unsustainable levels, and it went down.

No amount of tweaking the Apache and MySQL config files helped with this sort of scenario, or at least it didn’t help me. You see, the difference between peak traffic levels with search engines vs. people is that people will go to a single article or a group of articles that are in demand. A caching plugin works great for those sorts of situations. There’s a limited number of pages to worry about caching, and those pages get served up time and time again. The load is acceptable. When a search engine bot starts indexing your site, it’ll call up any and all available pages that it can find. That can place a huge load on the web server as it scrambles to serve up those pages and build static versions for the caching plugin. I believe that it’s too much for most medium-sized servers to handle, and they will usually go down.

In my case, disabling the caching plugin and making sure no traces were left in the .htaccess file were the only things that helped. Now, I might have up to four different search engine bots crawling my site, each spawning multiple threads, and my server will usually not go down. Sure, there are times when the server will get dangerously low on RAM, and will be unresponsive for 5-10 minutes, but that’s an acceptable scenario for me. And if I should all of a sudden get huge amounts of people traffic to a post, it’s possible that the web server will also become unresponsive, at least for a time. But the great thing about running WordPress by itself is that Apache will usually take care of itself. As the requests die down, Apache will kill the extra threads, the available RAM will go back up again, and the server will recover nicely. That wasn’t possible while I ran the caching plugin. When it went down, it stayed down, and that was a problem.

I realize that what works for me may not work for others. I have not tested what happens with WP Super Cache on a larger server, for example one with more RAM. It’s possible that the larger amount of RAM there will offset the greater demand placed on the server as it builds static versions of the pages, although I’m not sure what to say about the CPU usage. That also peaked as the caching plugin went crazy. Not sure how that’ll work on a more powerful server.

WP Super Cache has some options that allow you to cache more pages and keep them cached for longer periods of time. Perhaps fiddling with those options would have allowed me to keep running the plugin, but I wanted to see how things stood from the other side of the fence. Like I said, so far, so good. Caveats aside, running WordPress by itself was the cure for my persistent web server outages.