How to check what caused Apache2 to completely crash? And possible to auto-restart if it happens again?

Well, I have a suspicion you're running into RAM issues (or lack-thereof :-) 256MB is a bit on the low-end...

If you run the "free" command on your server, what output do you get?

What I'd start off by doing, is removing any services you know you don't need.

However, in order to see what the problems are, you'll want to check out various error logs (which ones depend on what distro you're using). Looking in /var/log/messages and the Apache log, /var/log/apache2/error_log or /var/log/httpd/error_log would be good places to start.

I'd also check out the dmesg output, especially towards the end, as it may contain any kernel errors you're seeing.

Virtualmin has a "Status monitoring" feature in System Settings -> Features and Plugins. You can use that to have it email you when a site is down. I don't recall off the top of my head whether it auto-restarts services... but if not, I've had great luck with tools like Monit for doing that.

However, my main suggestion in your case is to free up whatever memory you can, and to make sure you have plenty of swap available.

-Eric

Okay, looking at your free output, you don't appear to have swap -- so it looks like you're using an OpenVZ-based VPS.

The problem is that with no swap space, there's absolutely no margin for error... no ability to have additional resources if/when your main RAM gets used up.

If there's any kind of usage spike -- whether it be several folks accessing your site at the same time, a few emails being spam checked, cron jobs running at night that require a lot of RAM, or maybe all those at the same time -- your system can be without RAM and things will crash.

To try and help you track down what's going on, you could try installing a monitoring tool such as Monit, which could help you keep an eye on your system. You can configure it to let you know if memory is running low, for example... as well as to restart Apache if it dies.

For pretty graphs and stuff and better usage information, you might also look into Munin.

However, I'll offer that the forums here are filled with folks who've tried to setup OpenVZ-based servers with no swap, and ran into problems when their RAM ran out.

It does look like you did a good job disabling unneeded services.

The best I can offer from here, barring a way to get some additional RAM, would be to use a tool like Monit to restart daemons if something goes awry.

-Eric

Monit actually isn't terribly bad. When you have it installed and done basic changes on the .conf.

Throw this in and adjust as per needed

check process sshd with pidfile /var/run/sshd.pid
   start program  "/etc/init.d/sshd start"
   stop program  "/etc/init.d/sshd stop"
   if failed port 22 protocol ssh then restart
   if 5 restarts within 5 cycles then timeout
 
check process mysql with pidfile /var/run/mysqld/mysqld.pid
   group mysql
   start program = "/etc/init.d/mysqld start"
   stop program = "/etc/init.d/mysqld stop"
   if failed host 127.0.0.1 port 3306 then restart
   if 5 restarts within 5 cycles then timeout
 
check process apache with pidfile /var/run/httpd.pid
   group apache
   start program = "/etc/init.d/httpd start"
   stop program  = "/etc/init.d/httpd stop"
   if failed host www.daworm.net port 80 protocol http
      and request "/monit/token" then restart
   if cpu is greater than 60% for 2 cycles then alert
   if cpu > 80% for 5 cycles then restart
   if totalmem > 500 MB for 5 cycles then restart
   if children > 250 then restart
   if loadavg(5min) greater than 10 for 8 cycles then stop
   if 3 restarts within 5 cycles then timeout
 
check process postfix with pidfile /var/spool/postfix/pid/master.pid
   group mail
   start program = "/etc/init.d/postfix start"
   stop  program = "/etc/init.d/postfix stop"
   if failed port 25 protocol smtp then restart
   if 5 restarts within 5 cycles then timeout

The main things will be the locations of your PID files and the group/user names of your applications. Where I got this from for example apache was in as "www" for the user/group. I have to change it to "apache".

In addition, throw this into a file chmod +x the file and then run it.

#!/bin/bash
echo "This is intended as a guideline only!"
if [ -e /etc/debian_version ]; then
    APACHE="apache2"
elif [ -e /etc/redhat-release ]; then
    APACHE="httpd"
fi
RSS=`ps -aylC $APACHE |grep "$APACHE" |awk '{print $8'} |sort -n |tail -n 1`
RSS=`expr $RSS / 1024`
echo "Stopping $APACHE to calculate free memory"
/etc/init.d/$APACHE stop &> /dev/null
MEM=`free -m |head -n 2 |tail -n 1 |awk '{free=($4); print free}'`
echo "Starting $APACHE again"
/etc/init.d/$APACHE start &> /dev/null
echo "MaxClients should be around" `expr $MEM / $RSS`

The above will give you a simple output based on your system resources.

After which you adjust your httpd.conf file appropriately for the MaxClients etc.

Here's what I changed my settings to based on the recommended above and Apache stablised for myself.

<IfModule prefork.c>
StartServers       4
MinSpareServers    3
MaxSpareServers   5
ServerLimit      19
MaxClients       19
MaxRequestsPerChild  4000
</IfModule>

You will have to adjust the MaxClients and ServerLimit to be the same and drop your StartServers down a few (I think default is like 8?.

If you are low traffic this would be more than enough. AND again, adjust your MinSpareServers and MaxSpareServers.

Yours will be lower than 19. 19 is based on my 2GB ram in the server I have.

And LAST last thing

Look in /var/log/virtualmin/xxx.xxx_error_log

Where xxx.xxx is your domain names. You may notice some php errors or similar showing up in there. I identified a script from that (which was a few years old and seriously out of date) and since disabling it, i've only had to worry about the server doing graceful restarts on log rotates.

:::EDIT::: Just noticed an entry in my apache portion for Monit... the "if mem greater than xxx" then restart. You may want to adjust that down to say 200? Seeing as you only have 256mb. 5 cycles is basically 5 checks, in my case I check every minute.