URGENT Suddenly /etc/clamd.d/virtualmin.conf uses 99.8 % processor

Submitted by Dim Git on Sun, 10/23/2016 - 06:31

I have posted this here https://www.virtualmin.com/node/43119 but the situation is now very urgent. Sorry . :o(

Briefly :-

Suddenly clamd.virtualmin -c /etc/clamd.d/virtualmin.conf is using 99% of processor

Status page doesn't seem to update between visits or if refreshed never shows.

CentOS Linux 5.11 Webmin version 1.821 Virtualmin version 5.04 Kernel and CPU Linux 2.6.18-412.el5 on x86_64

I have another server of very similar spec showing the same.

Tried stopping virus scan using "System Settings" > "Features and Plugins" but it hasn't stopped the problem.

I now find that authentication fails on outgoing emails but incoming is ok.

Thanks, Tim CentOS Linux 5.11 Webmin version 1.810 Virtualmin version 5.04 Kernel and CPU Linux 2.6.18-412.el5 on x86_64

Status: 
Active

Comments

Submitted by Dim Git on Sun, 10/23/2016 - 07:02

UPDATE:

I have found why outgoing mail was not working, saslauth was not running. I did perform a reboot earlier and I found a few things were not running. Webmin had not updated the status shown in "bootup and shutdown actions", even after starting it manually, it still shows as "unknown".

Thanks for reading

Submitted by andreychek on Sun, 10/23/2016 - 10:26

Howdy -- thanks for contacting us?

Are things working properly for you now?

We've actually gotten a handful of other people using CentOS 5 who, all in the last few hours, indicates that they were seeing high CPU usage with ClamAV.

We were wondering if that was better for you now, or whether that was still high.

Submitted by Dim Git on Sun, 10/23/2016 - 15:08

No, both systems are struggling along with 98% or 99% processor use.

I have not been able to achieve anything more than I have reported. :o(

Thanks for getting back to me though.

Submitted by andreychek on Sun, 10/23/2016 - 16:32

Okay, my suggestion would be -- temporarily disable ClamAV, and we'll dig into this further to try and sort out how best to resolve it.

Submitted by Dim Git on Mon, 10/24/2016 - 02:25

Thanks for getting back to me.

ITried stopping virus scan using "System Settings" > "Features and Plugins" but it hasn't stopped it. Is there somewhere else I should do that?

Thanks, Tim

Submitted by Dim Git on Mon, 10/24/2016 - 02:52

I have found another couple of symptoms :-

The login to mail account via Virtualmin button no longer shows on the "Edit User" page.

Associated, I guess, is the timeout when trying to get into the Webmail facility using either webmail.domain.com or 123.123.123.123:20000.

This could of course be connected to the reboot and things not starting up. I will report back if I find anything.

Thanks, Tim

Submitted by andreychek on Mon, 10/24/2016 - 09:58

Those things aren't actually related. There appears to be a ClamAV bug occurring with the particular ClamAV version and CentOS 5. We're working on putting together a new ClamAV version in the hopes of fixing that.

If you're seeing any issues within Virtualmin itself, that's a separate issue. If it's causing a problem, feel free to open a separate request regarding that.

Submitted by Dim Git on Mon, 10/24/2016 - 12:57

Thanks Andreychek,

My initial thoughts were the same, "cant be related" but I thought further that since all the problems started at the same time and I had done a reboot, perhaps because the processor was fully occupied Webmin simply timed out getting everything running. Oh well, back to the grind.

Still can't stop virus scanning. I tried to stop it using "System Settings" > "Features and Plugins" but it hasn't stopped it. Is there somewhere else I should do that?

Submitted by andreychek on Mon, 10/24/2016 - 13:13

Was the virus scanner feature disabled from your various Virtual Servers first?

You can either do that manually, by going into Edit Virtual Servers -> Enabled Features for each of your domains -- or, you can go into List Virtual Servers, and bulk change that feature there.

You won't be able to disable it from the Features and Plugins section, until none of the domains are set to use it.

Submitted by Dim Git on Mon, 10/24/2016 - 15:29

Hi Andreychek, Thanks for your ongoing interest.

I really wish I could find a way to insert a screenshot here but alas, I can't see a way to do that.

If I look at "Running Processes" I see this :-
CPU load averages: 3.66 (1 mins) , 3.41 (5 mins) , 3.24 (15 mins)
CPU type: Intel(R) Core(TM)2 Quad CPU Q8400 @ 2.66GHz , 4 cores

3304 nobody 99.9 % clamd.virtualmin -c /etc/clamd.d/virtualmin.conf

If I look at "Features and Plugins" I see this :-
Feature or Plugin - Virus filtering
Source - Core
Version - 5.04
Domains - 0

There is NO check mark in the checkbox to the left of "Virus filtering".

So, on one hand virus filtering isn't running but on the other hand it is.

Or am I just wandering around in the darkness?

Lost!

Submitted by Blueforce on Mon, 10/24/2016 - 16:46 Pro Licensee

Hi Dim Git,

I had the same problem as you, and you can do a bulk change as Andreychek said. Click on "List virtual servers" then click on "Select all" and all virtual servers is checked click on "Update selected". Then click on "Enabled feature changes" and there select Disable for "Virus filtering enabled?". After that you can go to "System Settings" and "Features and Plugins" and uncheck "Virus filtering"

But you should look at my issue, and how I fixed the same problems that you have, it worked for me! It has the subject "URGENT! - LibClamAV Error: mpool_malloc()" and here is the link: https://www.virtualmin.com/node/43122

Regards, Leffe (Blueforce)

Submitted by Dim Git on Tue, 10/25/2016 - 00:19

Thanks for your input Leffe,

Your post and the referred thread gave me a nudge in the right direction.

I have an appointment that I cannot skip or change in a couple of hours so I really didn't want to risk another reboot as you have done in case something goes wrong. As I said in my last post some parts said that anti-virus was running while others said it was NOT running. "System Settings" > "Features and Plugins" said it was NOT running but "Running Processes" said it WAS. (I had already disabled it on all virtual servers)

Eventually I went into "Webmin" > "System" > "Bootup and Shutdown" and selected "clamd-virtualmin" where the edit action page showed "Started Now?" as "unknown". I shut it down and now the server is showing 100% idle.

Webmail and Log into Usermin etc. are now working too.

Thanks again Leffe and Andreychek for helping me get out of a hole until a fix is released.

Take care, Tim

Submitted by andreychek on Tue, 10/25/2016 - 00:40

Great, glad to hear the workaround is working!

Joe is working on putting together a new ClamAV package for CentOS 5.

Submitted by Dim Git on Wed, 10/26/2016 - 03:34

Hi Guys,

Given that it would seem that this issue appears to have stopped other things running in WM (following a reboot) I am wondering if the following could also have been affected by this.

The system is, as you can see a few years old and today's logwatch contained the following :-

smartd 5.42 2011-10-20 r3458 [x86_64-linux-2.6.18-412.el5] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net
Device: /dev/sda, type changed from 'scsi' to 'sat'
Device: /dev/sda [SAT], ST3250318AS, S/N:9VMVG611, WWN:5-000c50-02d51e0bd, FW:CC46,
250 GB
Device: /dev/sdb, type changed from 'scsi' to 'sat'
Device: /dev/sdb [SAT], ST3250318AS, S/N:9VMVG5A2, WWN:5-000c50-02d51cffa, FW:CC46,
250 GB

No previous logwatch reports had this so it is something new.
This is the first time since stopping AV that the logwatch has run automatically (I did run it manually)

Is smartmontools run periodically as a cron or similar (couldn't find it) ?

Should I worry?

Submitted by andreychek on Wed, 10/26/2016 - 09:52

I wouldn't worry unless you're experiencing a problem :-)

Those issues are unrelated though.

The problem you previously saw was due to a bug in ClamAV.

These other notices you're seeing appear to be changes in device names, likely due to changes in the kernel that is now active since you've rebooted.

I'm not seeing anything in the above that seems particular unusual, but I'd just verify that things are working as expected.

Submitted by Dim Git on Tue, 11/01/2016 - 02:11

Now there is another problem.

This is from "Running Processes"

7870 clamav 99.9 % /usr/bin/freshclam --quiet
18010 clamav 99.9 % /usr/bin/freshclam --quiet
29036 clamav 99.9 % /usr/bin/freshclam --quiet

Couldn't find how to stop this from loading/executing so I just killed the processes.

Guess I will have to do it again sometime.

Submitted by andreychek on Tue, 11/01/2016 - 08:12

I believe that's the freshclam cron job, probably trying to update the ClamAV signatures. You could always try disabling that cron job.

However, I'll followup with Joe again to try and figure out what the holdup is with the new packages... he may have run into some snags.

Submitted by Dim Git on Wed, 11/02/2016 - 02:46

Thanks Andreychek .

This morning a new symptom in Logwatch and just in case it helps Joe, here it is :-

WARNING: [LibClamAV] mpool_malloc(): Attempt to allocate 8388608 bytes. Please
report to http://bugs.clamav.net

There are seemingly thousands of the above lines ( didn't count 'em :o) ) To give you a clue of the number, the logwatch email is usually less than 200kB this morning it was 1.17 MB

Submitted by Dim Git on Wed, 11/09/2016 - 03:34

Hi All,

I see that there are some clamav updates available to Yum.

I have been reluctant to jump in and do those updates because I thought there might be an update to this thread if Joe had completed his work and these were the ones we should expect.

Please can someone comment on that? Are these the updates expected?

clamav End-user tools for the Clam Antivirus scanner New version 0.99.2-1.vm.el5.centos
clamav-data Virus signature data for the Clam Antivirus scanner New version 0.99.2-1.vm.el5.centos
clamav-filesystem Filesystem structure for clamav New version 0.99.2-1.vm.el5.centos
clamav-lib Dynamic libraries for the Clam Antivirus scanner New version 0.99.2-1.vm.el5.centos
clamav-server Clam Antivirus scanner server New version 0.99.2-1.vm.el5.centos
clamav-server-sysv SysV initscripts for clamav server New version 0.99.2-1.vm.el5.centos
clamav-update Auto-updater for the Clam Antivirus scanner data-files New version 0.99.2-1.vm.el5.centos

Thanks

Submitted by andreychek on Wed, 11/09/2016 - 09:25

Yup! Those updates should correct the problems you were seeing.

Submitted by Dim Git on Thu, 11/10/2016 - 05:10

Just in case anyone else comes here looking for a solution, here is what worked for me.

Did the updates from the link on the opening page.

Tried to start clamd-virtualmin from the "Bootup and Shutdown" page but got an error that the database was corrupt.

Deleted daily.cvd, main.cvd and mirrors.dat in the /var/lib/clamav/directory

Ran "freshclam" in "Running Processes" > Run.

Enabled "/usr/share/clamav/freshclam-sleep"  in Scheduled Cron Jobs.

Started clamd-virtualmin in "Bootup and Shutdown" page.

I rebooted the server and everything was working as expected even those things that were thought to be not connected with this issue.

Thanks for your help guys.