Hi there,
Recently, without any significant changes that I'm aware of, I found that my webmin instance was not accessible. It simply said "connection refused" in chrome on the default port 10000. After checking the systemctl status, it appeared that the webmin service was still running (and in fact, all of my websites were still accessible), but using "netstat -tulpn" showed that nothing was listening on port 10000. Usermin on port 20000 was working just fine.
I ended up restarting the service, which worked, but less than 30 minutes later while I was doing normal things, I got a popup in webmin that said "lost connection." Upon refresh, the same thing was happening; nothing was listening on port 10000, usermin works fine, websites work fine, etc.
It sounds to me like miniserv is crashing, however the miniserv.error logs show absolutely nothing around the time of the crash. You might also consider memory usage, but my server has 16GB and I rarely use more than 60%. dmesg shows nothing about killing any processes.
Where should I be looking to diagnose this problem?
Thanks
Howdy,
A common cause of that is resource issues... if your server is running out of RAM, it could start killing off processes such as Webmin in order to keep the server online.
What is the output of these commands:
free -muptime
dmesg | tail -30
Hey there, thanks for the response.
It just crashed again and it happened right after I went into the postfix config and clicked on the Relocated Mapping button. This error appeared on the page and then nothing was accessible anymore:
"Failed to query Postfix config command to get the current value of parameter relocated_maps:"
I have run the commands you mentioned, here are the outputs:
free -m
total used free shared buff/cache available
Mem: 16051 3224 12295 343 531 12223
Swap: 0 0 0
uptime
22:17:04 up 2:55, 2 users, load average: 0.25, 0.34, 0.35
dmesg | tail -30
[ 9666.701185] audit: type=1400 audit(1566705780.822:320): apparmor="ALLOWED" operation="open" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/dovecot.index" pid=23837 comm="imap" requested_mask="rw" denied_mask="rw" fsuid=1001 ouid=1001
[ 9666.762450] audit: type=1400 audit(1566705780.882:321): apparmor="ALLOWED" operation="open" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot.index.log" pid=23837 comm="imap" requested_mask="ra" denied_mask="ra" fsuid=1001 ouid=1001
[ 9666.763760] audit: type=1400 audit(1566705780.882:322): apparmor="ALLOWED" operation="open" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot.index" pid=23837 comm="imap" requested_mask="rw" denied_mask="rw" fsuid=1001 ouid=1001
[ 9666.765036] audit: type=1400 audit(1566705780.886:323): apparmor="ALLOWED" operation="mknod" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot-uidlist.lock" pid=23837 comm="imap" requested_mask="c" denied_mask="c" fsuid=1001 ouid=1001
[ 9666.767414] audit: type=1400 audit(1566705780.886:324): apparmor="ALLOWED" operation="open" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot-uidlist.lock" pid=23837 comm="imap" requested_mask="rwc" denied_mask="rwc" fsuid=1001 ouid=1001
[ 9666.767482] audit: type=1400 audit(1566705780.886:325): apparmor="ALLOWED" operation="open" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot-uidlist" pid=23837 comm="imap" requested_mask="rw" denied_mask="rw" fsuid=1001 ouid=1001
[ 9666.768074] audit: type=1400 audit(1566705780.886:326): apparmor="ALLOWED" operation="file_lock" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot.index.log" pid=23837 comm="imap" requested_mask="wk" denied_mask="wk" fsuid=1001 ouid=1001
[ 9666.768179] audit: type=1400 audit(1566705780.886:327): apparmor="ALLOWED" operation="file_lock" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot.index.log" pid=23837 comm="imap" requested_mask="k" denied_mask="k" fsuid=1001 ouid=1001
[ 9666.768203] audit: type=1400 audit(1566705780.886:328): apparmor="ALLOWED" operation="unlink" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot-uidlist.lock" pid=23837 comm="imap" requested_mask="d" denied_mask="d" fsuid=1001 ouid=1001
[10085.172472] audit: type=1400 audit(1566706199.295:329): apparmor="ALLOWED" operation="open" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/dovecot.index.log" pid=26086 comm="imap" requested_mask="ra" denied_mask="ra" fsuid=1001 ouid=1001
[10085.172831] audit: type=1400 audit(1566706199.295:330): apparmor="ALLOWED" operation="open" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/dovecot.index" pid=26086 comm="imap" requested_mask="rw" denied_mask="rw" fsuid=1001 ouid=1001
[10085.418278] audit: type=1400 audit(1566706199.539:331): apparmor="ALLOWED" operation="open" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot.index.log" pid=26086 comm="imap" requested_mask="ra" denied_mask="ra" fsuid=1001 ouid=1001
[10085.418613] audit: type=1400 audit(1566706199.539:332): apparmor="ALLOWED" operation="open" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot.index" pid=26086 comm="imap" requested_mask="rw" denied_mask="rw" fsuid=1001 ouid=1001
[10085.419518] audit: type=1400 audit(1566706199.539:333): apparmor="ALLOWED" operation="mknod" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot-uidlist.lock" pid=26086 comm="imap" requested_mask="c" denied_mask="c" fsuid=1001 ouid=1001
[10085.421396] audit: type=1400 audit(1566706199.543:334): apparmor="ALLOWED" operation="open" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot-uidlist.lock" pid=26086 comm="imap" requested_mask="rwc" denied_mask="rwc" fsuid=1001 ouid=1001
[10085.421479] audit: type=1400 audit(1566706199.543:335): apparmor="ALLOWED" operation="open" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot-uidlist" pid=26086 comm="imap" requested_mask="rw" denied_mask="rw" fsuid=1001 ouid=1001
[10085.422260] audit: type=1400 audit(1566706199.543:336): apparmor="ALLOWED" operation="file_lock" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot.index.log" pid=26086 comm="imap" requested_mask="wk" denied_mask="wk" fsuid=1001 ouid=1001
[10085.422322] audit: type=1400 audit(1566706199.543:337): apparmor="ALLOWED" operation="file_lock" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot.index.log" pid=26086 comm="imap" requested_mask="k" denied_mask="k" fsuid=1001 ouid=1001
[10085.422338] audit: type=1400 audit(1566706199.543:338): apparmor="ALLOWED" operation="unlink" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot-uidlist.lock" pid=26086 comm="imap" requested_mask="d" denied_mask="d" fsuid=1001 ouid=1001
[10386.739392] audit_printk_skb: 3 callbacks suppressed
[10386.739394] audit: type=1400 audit(1566706500.859:340): apparmor="ALLOWED" operation="open" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/dovecot.index.log" pid=27887 comm="imap" requested_mask="ra" denied_mask="ra" fsuid=1001 ouid=1001
[10386.739801] audit: type=1400 audit(1566706500.863:341): apparmor="ALLOWED" operation="open" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/dovecot.index" pid=27887 comm="imap" requested_mask="rw" denied_mask="rw" fsuid=1001 ouid=1001
[10386.802458] audit: type=1400 audit(1566706500.923:342): apparmor="ALLOWED" operation="open" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot.index.log" pid=27887 comm="imap" requested_mask="ra" denied_mask="ra" fsuid=1001 ouid=1001
[10386.803610] audit: type=1400 audit(1566706500.923:343): apparmor="ALLOWED" operation="open" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot.index" pid=27887 comm="imap" requested_mask="rw" denied_mask="rw" fsuid=1001 ouid=1001
[10386.806576] audit: type=1400 audit(1566706500.927:344): apparmor="ALLOWED" operation="mknod" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot-uidlist.lock" pid=27887 comm="imap" requested_mask="c" denied_mask="c" fsuid=1001 ouid=1001
[10386.808546] audit: type=1400 audit(1566706500.931:345): apparmor="ALLOWED" operation="open" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot-uidlist.lock" pid=27887 comm="imap" requested_mask="rwc" denied_mask="rwc" fsuid=1001 ouid=1001
[10386.808619] audit: type=1400 audit(1566706500.931:346): apparmor="ALLOWED" operation="open" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot-uidlist" pid=27887 comm="imap" requested_mask="rw" denied_mask="rw" fsuid=1001 ouid=1001
[10386.809092] audit: type=1400 audit(1566706500.931:347): apparmor="ALLOWED" operation="file_lock" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot.index.log" pid=27887 comm="imap" requested_mask="wk" denied_mask="wk" fsuid=1001 ouid=1001
[10386.809190] audit: type=1400 audit(1566706500.931:348): apparmor="ALLOWED" operation="file_lock" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot.index.log" pid=27887 comm="imap" requested_mask="k" denied_mask="k" fsuid=1001 ouid=1001
[10386.809218] audit: type=1400 audit(1566706500.931:349): apparmor="ALLOWED" operation="unlink" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot-uidlist.lock" pid=27887 comm="imap" requested_mask="d" denied_mask="d" fsuid=1001 ouid=1001
Not sure what these audit things are, but there's a whole lot of them in dmesg, even as far back as -100. Thanks!
MItch, did you get it figured out?
I am having the exact same issue. It's really annoying. I don't want to keep having to restart the instance. And i really hope it's not resources, as i have double the ram as my last vps.
No luck. It seems like it stays up until I start using it, then within 30 mins it will die randomly. It "feels" like a memory usage problem, but there are no traces of it being killed, I have 16GB of RAM, and this just randomly started happening without any changes to the server after it working for a year....so I have no idea
@Falcons34, you may wan to try what I mentioned in Comment #1 above to see if that shows anything.
@MutchTamadge, is this a dedicated server, or a VPS? If a VPS, what kind of VPS is it? I don't see any obvious resource issues in the above, but you're right that it does sound like a resource issue of some kind. I'm curious if it could be related to a VPS with certain types of resource restrictions, such as OpenVZ.
-Eric
Hi Eric, thanks for this
Yes, I ran those and it was showing out of memory errors and killing off clam processes. So I'm assuming clam is the problem. I knew it was a memory hog but was hoping 4G of RAM would be enough. Anyhow, is there any recommendation for a program to replace clam?
Maybe try boosting your php memory limit up some.
I'm having the same issue after doing to latest updates.
Collin Lynch
https://www.innovativeflare.com
I solved my issues this: https://www.virtualmin.com/comment/818100#comment-818100
2 things I disabled afterward that really saved a lot of ram were dovecot and postfix.
Collin Lynch
https://www.innovativeflare.com
I run dovecot and postfix on a 4GB RAM system and don't have any problems.
Normally I don't use Clamav in the foreground...however,
If I start clamav
-my RAM usage increases by about 20% (normally im running at about 22%, now 42%)...stats tell me /usr/sbin/clamd --foreground=true is using 1.01GB.
-CPU does jump for a while then drops back to normal.
https://ajecreative.com.au
@adamjedgar Nice site, what theme are you using?
Collin Lynch
https://www.innovativeflare.com
Collin
WP 5.2.3 theme maxhost and revslider version... apache 2.4.25 ... owl carousel, tweenmax , bootstrap 3.3.6 and so more you can read out of headers and source if you are meaning these things.
also some mixed content , on mail ports from "postfix" i guess The expiration date of this certificate is 2019-08-04 09:28:17 UTC.And Collin you are using WP 5.2.3 maybe raidiant theme parts, apache 2.4.29 Ubuntu , woocommerce 3.7.1 and so more
Trigger The server does not support any cipher suites that provide forward secrecy. on thgose 2 dashboard.innovativeflare.com 54.70.248.63 scan.innovativeflare.com 34.209.57.169on maindomain box : The server uses a 1024-bit Diffie-Hellman group., The server uses a 1024-bit RSA key. ,and on all still also TLS 1.0