I've been running Virtualmin on my VPS for almost a week now and I think I have it set up properly and all that, I followed a few tutorials here on the site and so on.
But yesterday I noticed how all my sites (including Virtualmin) suddenly stopped working. It would instantly give me an error when I go to the web site URL. Logging in at the VPS Control panel I could see that only 20mb out of the 256mb memory was used, and usually it's around 50% when apache, mysql etc. runs.
So something caused Apache (and MySQL?) to just CRASH and don't come back again by itself. Obviously this is a HUGE problem if it happens right after I go to bed and my sites will be offline for 6-8 hours.
How can I check (what log files?) what caused apache to break down - and even better, can I do something in Virtualmin to make sure everything automatically comes back up? Is there a cronjob or something (?) that can check every 5 minutes if a local URL works and if not, it will restart a bunch of services?
Thanks in advance :)
Well, I have a suspicion you're running into RAM issues (or lack-thereof :-) 256MB is a bit on the low-end...
If you run the "free" command on your server, what output do you get?
What I'd start off by doing, is removing any services you know you don't need.
However, in order to see what the problems are, you'll want to check out various error logs (which ones depend on what distro you're using). Looking in /var/log/messages and the Apache log, /var/log/apache2/error_log or /var/log/httpd/error_log would be good places to start.
I'd also check out the dmesg output, especially towards the end, as it may contain any kernel errors you're seeing.
Virtualmin has a "Status monitoring" feature in System Settings -> Features and Plugins. You can use that to have it email you when a site is down. I don't recall off the top of my head whether it auto-restarts services... but if not, I've had great luck with tools like Monit for doing that.
However, my main suggestion in your case is to free up whatever memory you can, and to make sure you have plenty of swap available.
-Eric
Plesk Power Panels (VPS controlpanel) says 110MB used, Virtualmin says 140MB used and "free" says 112MB used:
(EDIT: Okay the copy/paste didn't look too good in here, so I took a screenshot instead and attached).
I reloaded them all at the same time, so I guess Virtualmin displays more than what is actually being used?
I also tried to disable all the services I don't need. I've attached a screenshot of my bootup & bootdown list within Virtualmin, to this post.
I'll try to keep an eye on /var/log/apache2/error.log and and /var/log/messages next time it crashes.
My dmesg files are all empty. (the ones in /var/log).
I looked for "Status monitoring" in System Settings > Features and Plugins, but it's not there. Maybe it's only for PRO users?
Regarding memory, I tried to free up as much as possible, but if you have suggestions on what else I can do, I would love to hear them. However, I'm just running a single site on it now with a few hundred pageviews a day, so I would be surprised if 256MB isn't enough for apache, mysql and virtualmin?
Okay, looking at your free output, you don't appear to have swap -- so it looks like you're using an OpenVZ-based VPS.
The problem is that with no swap space, there's absolutely no margin for error... no ability to have additional resources if/when your main RAM gets used up.
If there's any kind of usage spike -- whether it be several folks accessing your site at the same time, a few emails being spam checked, cron jobs running at night that require a lot of RAM, or maybe all those at the same time -- your system can be without RAM and things will crash.
To try and help you track down what's going on, you could try installing a monitoring tool such as Monit, which could help you keep an eye on your system. You can configure it to let you know if memory is running low, for example... as well as to restart Apache if it dies.
For pretty graphs and stuff and better usage information, you might also look into Munin.
However, I'll offer that the forums here are filled with folks who've tried to setup OpenVZ-based servers with no swap, and ran into problems when their RAM ran out.
It does look like you did a good job disabling unneeded services.
The best I can offer from here, barring a way to get some additional RAM, would be to use a tool like Monit to restart daemons if something goes awry.
-Eric
I might be using OpenVZ-based VPS, I don't know, sorry :) Is it not possible to have a swap file, then?
I'll take a look at Monit and see if it can help me anyway, but I'm not good at this linux stuff so if the installation/usage is too complicated, then I'll have no clue how to use it.
Thanks for the hint about OpenVZ though, I'll search the forum for more information and tips for setting it up.
Monit actually isn't terribly bad. When you have it installed and done basic changes on the .conf.
Throw this in and adjust as per needed
check process sshd with pidfile /var/run/sshd.pid start program "/etc/init.d/sshd start" stop program "/etc/init.d/sshd stop" if failed port 22 protocol ssh then restart if 5 restarts within 5 cycles then timeout check process mysql with pidfile /var/run/mysqld/mysqld.pid group mysql start program = "/etc/init.d/mysqld start" stop program = "/etc/init.d/mysqld stop" if failed host 127.0.0.1 port 3306 then restart if 5 restarts within 5 cycles then timeout check process apache with pidfile /var/run/httpd.pid group apache start program = "/etc/init.d/httpd start" stop program = "/etc/init.d/httpd stop" if failed host www.daworm.net port 80 protocol http and request "/monit/token" then restart if cpu is greater than 60% for 2 cycles then alert if cpu > 80% for 5 cycles then restart if totalmem > 500 MB for 5 cycles then restart if children > 250 then restart if loadavg(5min) greater than 10 for 8 cycles then stop if 3 restarts within 5 cycles then timeout check process postfix with pidfile /var/spool/postfix/pid/master.pid group mail start program = "/etc/init.d/postfix start" stop program = "/etc/init.d/postfix stop" if failed port 25 protocol smtp then restart if 5 restarts within 5 cycles then timeoutThe main things will be the locations of your PID files and the group/user names of your applications. Where I got this from for example apache was in as "www" for the user/group. I have to change it to "apache".
In addition, throw this into a file chmod +x the file and then run it.
#!/bin/bash echo "This is intended as a guideline only!" if [ -e /etc/debian_version ]; then APACHE="apache2" elif [ -e /etc/redhat-release ]; then APACHE="httpd" fi RSS=`ps -aylC $APACHE |grep "$APACHE" |awk '{print $8'} |sort -n |tail -n 1` RSS=`expr $RSS / 1024` echo "Stopping $APACHE to calculate free memory" /etc/init.d/$APACHE stop &> /dev/null MEM=`free -m |head -n 2 |tail -n 1 |awk '{free=($4); print free}'` echo "Starting $APACHE again" /etc/init.d/$APACHE start &> /dev/null echo "MaxClients should be around" `expr $MEM / $RSS`The above will give you a simple output based on your system resources.
After which you adjust your httpd.conf file appropriately for the MaxClients etc.
Here's what I changed my settings to based on the recommended above and Apache stablised for myself.
You will have to adjust the MaxClients and ServerLimit to be the same and drop your StartServers down a few (I think default is like 8?.
If you are low traffic this would be more than enough. AND again, adjust your MinSpareServers and MaxSpareServers.
Yours will be lower than 19. 19 is based on my 2GB ram in the server I have.
And LAST last thing
Look in /var/log/virtualmin/xxx.xxx_error_log
Where xxx.xxx is your domain names. You may notice some php errors or similar showing up in there. I identified a script from that (which was a few years old and seriously out of date) and since disabling it, i've only had to worry about the server doing graceful restarts on log rotates.
:::EDIT::: Just noticed an entry in my apache portion for Monit... the "if mem greater than xxx" then restart. You may want to adjust that down to say 200? Seeing as you only have 256mb. 5 cycles is basically 5 checks, in my case I check every minute.