Hey all,
I just received our new Virtualmin.com server, and finished building the hardware. I'll be installing the OS, setting up Cloudmin, and installing Drupal 7 today. We'll be migrating to the new server, and hopefully installing it in the colo in Dallas within the next two weeks. This will, hopefully, resolve all of our performance problems here on the forums and issue tracker. The new box has two big, fast SSDs for the system and website and databases, as well as a couple of huge 3TB 7200 RPM spinning disks for backups and downloadable Cloudmin image storage.
Here's the picture: http://i.imgur.com/lUWwPTO.jpg
Looks humble, but it's a solid chunk of hardware. 32GB RAM, 3.4 GHz quad core, 1.8TB of SSD, and 6TB of traditional disk.
This will help resolve some of the issues we've had with the shopping cart (like renewals and upgrades being way more complicated and error prone than they should be), and I'm hoping to be able to enabled monthly subscription purchases, as well (they're broken in Ubercart with our payment processor and use case...we'll be switching to Drupal Commerce, most likely, and hopefully have working subscriptions).
Thanks for your patience with our slow forums and issue tracker. Me, and Eric, and Jamie have all spent way too much time trying to resolve the problems. We finally decided at the last weekly meeting to give up on fixing the old server, and just to throw new hardware at the problem.
I plan to swap out the old server for the new one, and drop some SSDs into the old server, as well, since it has plenty of life left in it (it's still a more powerful server than the new one, as we couldn't afford another server that size) but the hardware RAID disk subsystem in that box is broken in some sort of weird way that we could never figure out, so I'll need to get access to it so I can benchmark it and fix whatever is wrong about it. The old box is only using about 3% of its CPU, but disk IO is awful, despite having a bad ass hardware RAID system, so database writes slow down the website horribly. It's silly that a box with 16 cores and four spinning disks on a really expensive hardware LSI RAID controller can't keep up with one little website. But, we'll resolve it with SSD speed!
LSI RAID you say... I have those in my hosted root servers as well, and I happen to know that by default those controllers disable the write cache if they don't have a BBU (backup battery), indeed resulting in terrible write speed. You need to specifically enable the write-back cache using LSI management tools.
It's just a wild guess, but might that be the cause for you too?
Bingo! You may very well be correct. Seems likely, even. I remember even thinking about whether we have battery backup or not and whether I could safely enable that feature. I don't remember what the verdict was, but I'm sure I would have gone the conservative route (if I wasn't sure we had battery backup, I would have left it disabled). When the box comes back home with me in a week or two, I'll give it a look to see what the state of things is. It may be that we need to get that battery backup enabled and working. And we'll probably drop SSDs in there, anyway, as they've gotten so much cheaper and they outperform even the fastest RAID on spinning disks.
--
Check out the forum guidelines!
You might even be able to verify this before you get the box back. :) There's a command line tool from LSI called "MegaCLI", which is available for all kinds of operating systems (even including VMware ESXi), and which you can use to fully manage the controller in a shell.