I have Virtualmin GPL and it has been fine for the last couple of years, but now WebMin just seems to randomly shut down and takes all other services out except HTTPD. I have been monitoring the server (VPS 4 GB RAM, CentOS Linux 6.6, Webmin 1.801, Virtualmin 5.03) but am stumped. If anyone can suggest anything.
When the server stops responding, these services have to all be manually started:
/etc/init.d/webmin
/etc/init.d/usermin
/etc/init.d/saslauthd
/etc/rc.d/init.d/named
service rsyslog
/etc/init.d/spamd
/etc/init.d/crond
/etc/init.d/xinetd
service mailman
/etc/init.d/denyhosts
I've searched all logs and all I can find that makes no sense.
I found a folder with a huge number of files (over 2.3GB) in the folder /var/spool/postfix
Here are logs from webmin/webmin.debug from the 10 minutes prior to Webmin stopping at 21:55, until I manually restarted it at 22:52:
32410 [20/Jun/2016 21:46:08.631255] - - - STOP "runtime=1"
32409 [20/Jun/2016 21:46:08.654671] - - - STOP "runtime=1"
32484 [20/Jun/2016 21:47:07.173064] - - - START "script=webmincron.pl"
32516 [20/Jun/2016 21:50:01.226610] - - - START "script=monitor.pl"
32516 [20/Jun/2016 21:50:03.269023] - - status STOP "runtime=2"
32620 [20/Jun/2016 21:55:01.883755] - - - START "script=monitor.pl"
32620 [20/Jun/2016 21:55:03.401559] - - status STOP "runtime=2"
32710 [20/Jun/2016 22:00:02.152993] - - - START "script=monitor.pl"
32710 [20/Jun/2016 22:00:03.163123] - - status STOP "runtime=1"
1290 [20/Jun/2016 22:05:01.410616] - - - START "script=monitor.pl"
1290 [20/Jun/2016 22:05:03.077810] - - status STOP "runtime=2"
1402 [20/Jun/2016 22:10:01.541894] - - - START "script=monitor.pl"
1402 [20/Jun/2016 22:10:02.652265] - - status STOP "runtime=1"
1476 [20/Jun/2016 22:15:02.103055] - - - START "script=monitor.pl"
1476 [20/Jun/2016 22:15:03.545523] - - status STOP "runtime=1"
1566 [20/Jun/2016 22:20:02.002817] - - - START "script=monitor.pl"
1566 [20/Jun/2016 22:20:02.962014] - - status STOP "runtime=0"
1627 [20/Jun/2016 22:25:01.214166] - - - START "script=monitor.pl"
1627 [20/Jun/2016 22:25:02.653845] - - status STOP "runtime=1"
1682 [20/Jun/2016 22:26:01.786344] - - - START "script=awstats.pl"
1682 [20/Jun/2016 22:26:04.886002] - - virtualmin-awstats STOP "runtime=3"
1762 [20/Jun/2016 22:30:02.110633] - - - START "script=monitor.pl"
1762 [20/Jun/2016 22:30:03.481356] - - status STOP "runtime=1"
1868 [20/Jun/2016 22:35:01.512463] - - - START "script=monitor.pl"
1868 [20/Jun/2016 22:35:02.572846] - - status STOP "runtime=1"
1934 [20/Jun/2016 22:40:01.448326] - - - START "script=monitor.pl"
1934 [20/Jun/2016 22:40:03.000716] - - status STOP "runtime=2"
2033 [20/Jun/2016 22:45:01.682392] - - - START "script=monitor.pl"
2033 [20/Jun/2016 22:45:02.869104] - - status STOP "runtime=1"
3174 [20/Jun/2016 22:50:01.285520] - - - START "script=monitor.pl"
3174 [20/Jun/2016 22:50:02.772236] - - status STOP "runtime=1"
3274 [20/Jun/2016 22:51:12.859236] - - - START "script=webmincron.pl"
3274 [20/Jun/2016 22:51:12.944199] - - - STOP "runtime=0"
3276 [20/Jun/2016 22:51:12.862996] - - - START "script=webmincron.pl"
3276 [20/Jun/2016 22:51:15.074765] - - - STOP "runtime=3"
3374 [20/Jun/2016 22:52:02.872224] - - - START "script=webmincron.pl"
3825 [20/Jun/2016 22:52:16.151429] - - - START "script=lookup-domain.pl"
3829 [20/Jun/2016 22:52:16.202541] - - - START "script=lookup-domain.pl"
3829 [20/Jun/2016 22:52:18.683217] - - virtual-server STOP "runtime=2"
3825 [20/Jun/2016 22:52:18.726941] - - virtual-server STOP "runtime=2"
3834 [20/Jun/2016 22:52:16.269099] - - - START "script=lookup-domain.pl"
3834 [20/Jun/2016 22:52:18.867508] - - virtual-server STOP "runtime=2"
3375 [20/Jun/2016 22:52:32.430444] - - - STOP "runtime=30"
3374 [20/Jun/2016 22:52:32.468956] - - - STOP "runtime=30"
3946 [20/Jun/2016 22:52:34.254622] - hsd1.nj.comcast.net - START "script=session_login.cgi"
3946 [20/Jun/2016 22:52:34.456359] - .hsd1.nj.comcast.net - STOP "runtime=0"
3949 [20/Jun/2016 22:52:40.541553] - .hsd1.nj.comcast.net - START "script=index.cgi"
3947 [20/Jun/2016 22:52:41.043414] - .hsd1.nj.comcast.net - START "script=index.cgi"
3947 [20/Jun/2016 22:52:41.906140] root .hsd1.nj.comcast.net - STOP "runtime=0"
3956 [20/Jun/2016 22:52:44.306450] - .hsd1.nj.comcast.net - START "script=index.cgi"
From the messages log
Jun 20 06:04:32 ip-68-178-130-21 rsyslogd-2177: imuxsock begins to drop messages from pid 15492 due to rate-limiting
Jun 20 06:04:35 ip-68-178-130-21 rsyslogd-2177: imuxsock lost 62 messages from pid 15492 due to rate-limiting
Jun 20 09:59:56 ip-68-178-130-21 rsyslogd-2177: imuxsock begins to drop messages from pid 15492 due to rate-limiting
Jun 20 10:02:37 ip-68-178-130-21 rsyslogd-2177: imuxsock lost 40 messages from pid 15492 due to rate-limiting
Jun 20 12:47:36 ip-68-178-130-21 rsyslogd-2177: imuxsock begins to drop messages from pid 23766 due to rate-limiting
Jun 20 12:47:39 ip-68-178-130-21 rsyslogd-2177: imuxsock begins to drop messages from pid 23826 due to rate-limiting
Jun 20 12:47:45 ip-68-178-130-21 rsyslogd-2177: imuxsock lost 410 messages from pid 23826 due to rate-limiting
Jun 20 12:51:56 ip-68-178-130-21 rsyslogd-2177: imuxsock begins to drop messages from pid 24509 due to rate-limiting
Jun 20 12:52:00 ip-68-178-130-21 rsyslogd-2177: imuxsock begins to drop messages from pid 24562 due to rate-limiting
Jun 20 12:52:06 ip-68-178-130-21 rsyslogd-2177: imuxsock lost 437 messages from pid 24562 due to rate-limiting
Jun 20 12:54:17 ip-68-178-130-21 rsyslogd-2177: imuxsock begins to drop messages from pid 26444 due to rate-limiting
Jun 20 12:54:20 ip-68-178-130-21 rsyslogd-2177: imuxsock begins to drop messages from pid 26502 due to rate-limiting
Jun 20 12:54:26 ip-68-178-130-21 rsyslogd-2177: imuxsock lost 420 messages from pid 26502 due to rate-limiting
Jun 20 13:55:02 ip-68-178-130-21 rsyslogd-2177: imuxsock begins to drop messages from pid 19540 due to rate-limiting
Jun 20 13:55:36 ip-68-178-130-21 rsyslogd-2177: imuxsock lost 676 messages from pid 19540 due to rate-limiting
Jun 20 14:00:11 ip-68-178-130-21 rsyslogd-2177: imuxsock lost 244 messages from pid 26444 due to rate-limiting
Jun 20 15:18:47 ip-68-178-130-21 rsyslogd-2177: imuxsock begins to drop messages from pid 25694 due to rate-limiting
Jun 20 15:19:04 ip-68-178-130-21 rsyslogd-2177: imuxsock lost 561 messages from pid 25694 due to rate-limiting
Jun 20 16:10:01 ip-68-178-130-21 rsyslogd-2177: imuxsock lost 235 messages from pid 23766 due to rate-limiting
Jun 20 17:27:51 ip-68-178-130-21 rsyslogd-2177: imuxsock begins to drop messages from pid 28015 due to rate-limiting
Jun 20 17:27:56 ip-68-178-130-21 rsyslogd-2177: imuxsock begins to drop messages from pid 28070 due to rate-limiting
Jun 20 17:28:02 ip-68-178-130-21 rsyslogd-2177: imuxsock lost 422 messages from pid 28070 due to rate-limiting
Jun 20 17:28:32 ip-68-178-130-21 rsyslogd-2177: imuxsock begins to drop messages from pid 28566 due to rate-limiting
Jun 20 17:28:37 ip-68-178-130-21 rsyslogd-2177: imuxsock begins to drop messages from pid 28621 due to rate-limiting
Jun 20 17:28:43 ip-68-178-130-21 rsyslogd-2177: imuxsock lost 420 messages from pid 28621 due to rate-limiting
Jun 20 17:47:32 ip-68-178-130-21 rsyslogd-2177: imuxsock begins to drop messages from pid 5870 due to rate-limiting
Jun 20 17:47:37 ip-68-178-130-21 rsyslogd-2177: imuxsock begins to drop messages from pid 5925 due to rate-limiting
Jun 20 17:47:43 ip-68-178-130-21 rsyslogd-2177: imuxsock lost 422 messages from pid 5925 due to rate-limiting
Jun 20 17:48:38 ip-68-178-130-21 rsyslogd-2177: imuxsock begins to drop messages from pid 6072 due to rate-limiting
Jun 20 17:48:43 ip-68-178-130-21 rsyslogd-2177: imuxsock begins to drop messages from pid 6133 due to rate-limiting
Jun 20 17:48:49 ip-68-178-130-21 rsyslogd-2177: imuxsock lost 422 messages from pid 6133 due to rate-limiting
Jun 20 21:27:49 ip-68-178-130-21 rsyslogd-2177: imuxsock lost 239 messages from pid 24509 due to rate-limiting
reds of these in webmin/miniserv.error:
Use of uninitialized value in string eq at /usr/libexec/webmin/acl/edit_user.cgi line 345.
Use of uninitialized value in string eq at /usr/libexec/webmin/acl/edit_user.cgi line 345.
Use of uninitialized value in string eq at /usr/libexec/webmin/acl/edit_user.cgi line 345.
Use of uninitialized value in string eq at /usr/libexec/webmin/acl/edit_user.cgi line 345.
Use of uninitialized value in string eq at /usr/libexec/webmin/acl/edit_user.cgi line 345.
Subroutine list_users redefined at /usr/libexec/webmin/acl/acl-lib.pl line 42.
Subroutine get_user redefined at /usr/libexec/webmin/acl/acl-lib.pl line 191.
Subroutine list_groups redefined at /usr/libexec/webmin/acl/acl-lib.pl line 215.
Subroutine get_group redefined at /usr/libexec/webmin/acl/acl-lib.pl line 323.
Subroutine list_modules redefined at /usr/libexec/webmin/acl/acl-lib.pl line 336.
Subroutine list_module_infos redefined at /usr/libexec/webmin/acl/acl-lib.pl line 347.
Subroutine create_user redefined at /usr/libexec/webmin/acl/acl-lib.pl line 361.
Subroutine modify_user redefined at /usr/libexec/webmin/acl/acl-lib.pl line 529.