logrotate vhost and crash apache

Submitted by mtcocktail on Sun, 04/29/2012 - 02:40

Many post on this problem appear since one or two years. The postrotate script has changed of restart to graceful, and after a sleep 5; was added.

The problem is in logrotate.conf a bloc for each vhost who tells graceful restart in the same time. And the /etc/logrotate.d/apache2 who is also called to graceful apache.

Sometimes for me if server has a high load, (during backup), sleep 5 isn't sufficient and apache crash.

I've see maybe two solution :

One : logrotate has a features who has "sharedscripts", if you have many log process, the postrotate script is execute one time.

Virtualmin could make only one block directive far all vhost who have rotate feature. Replace

     /var/log/virtualmin/domaine1_access_log /var/log/virtualmin/domaine1_error_log {
         rotate 5
         weekly
         compress
         postrotate
         /usr/sbin/apache2ctl graceful ; sleep 5
         endscript
     }

     /var/log/virtualmin/domaine2_access_log /var/log/virtualmin/domaine2_error_log {
        rotate 5
        weekly
        compress
        postrotate
             /usr/sbin/apache2ctl graceful ; sleep 5
        endscript
     }

by :

     /var/log/virtualmin/domaine1_access_log /var/log/virtualmin/domaine1_error_log /var/log/virtualmin/domaine2_access_log /var/log/virtualmin/domaine2_error_log {
          rotate 5
          weekly
          compress
          sharedscripts
          postrotate
                 /usr/sbin/apache2ctl graceful
          endscript
        }

I'm agree that if you have more than 100 vhost is not relay readable. But it's work....

Another solution maybe is test pid file before run graceful (as in the logrotate of apache) for replace the sleep 5 :

    postrotate
            if [ -f "`. /etc/apache2/envvars ; echo ${APACHE_PID_FILE:-/var/run/apache2.pid}`" ]; then
                    /usr/sbin/apache2ctl graceful
            fi
    endscript

Regards,

Max

Status: 
Active

Comments

Submitted by andreychek on Sun, 04/29/2012 - 09:11

Jamie, we do still see this on occasion, even with the "sleep 5" as Max mentioned.

His ideas are both good suggestions, I'm particularly fond of the "sharedscript" idea -- giving up some readability for only doing one Apache restart.

Is something like that feasible?

Submitted by JamieCameron on Sun, 04/29/2012 - 18:23

Yes, that could be done ... in fact, the block could be simplified even more by taking advantage of the fact that all Virtualmin logs are now under /var/log/virtualmin . The only down-side is that you couldn't have a different logrotate config for each domain.

I will look into implementing a switch to a single logrotate config in the next Virtualmin release.

Submitted by clwheatley on Sun, 04/29/2012 - 19:26

Amazing! I've been researching a solution to this very problem we are facing and came to fill out a support request just now. I am so glad to hear of this possibility. We have well over 100 vhosts and Sunday mornings are becoming issues when clients go to their site and nothing comes up, something comes up slowely, or timeouts. I was all but ready to start staggering log rotation which was my last resort.

Jamie, any idea on a time frame?

Chris

Submitted by clwheatley on Sun, 04/29/2012 - 19:43

One question that came to mind was whether a change from single back to multiple would be possible if it was found staggering the weekly Apache log rotation to be more effective.

Will there be a mechanism to go back to the traditional Apache log rotation?

Submitted by JamieCameron on Sun, 04/29/2012 - 20:52

We will certainly support both modes. ETA for this feature is likely to be a couple of weeks.

Submitted by mtcocktail on Mon, 04/30/2012 - 02:21

Thanks in advance of this feature :)

Submitted by JamieCameron on Sat, 05/19/2012 - 00:40

I've been thinking about this, and it looks like it won't be practical to have a single logrotate block that matches all domains log files. The reason is that unless Apache is signaled to re-open its logs after each one, for a system with many virtual hosts log entries will be lost in the period between when rotation of the first one starts and when Apache finally gets restarted.

One improvement I can make though is to use the sharescripts directive in each domain's block, so that only a single Apache restart is needed for both the access and error logs.

Submitted by globerada on Sun, 01/21/2018 - 12:00

Hi. I can confirm that this "bug" is still active after almost 5 years. Clean install of Ubuntu 16.04 since 5 days ago.

I spent all the day searching over the internet why the server went down at 06:25 this Sunday. I only have 1 virtual host (+default).

error.log

[Sun Jan 21 05:28:35.216763 2018] [fcgid:warn] [pid 4291:tid 140418366719872] mod_fcgid: process 7939 graceful kill fail, sending SIGKILL
[Sun Jan 21 06:25:37.323358 2018] [mpm_event:notice] [pid 9658:tid 140418366719872] AH00493: SIGUSR1 received.  Doing graceful restart
[Sun Jan 21 06:25:37.439765 2018] [core:notice] [pid 9658] AH00060: seg fault or similar nasty error detected in the parent process

Syslog

Jan 21 06:25:37 vps502497 systemd[1]: Reloading LSB: Apache2 web server.
Jan 21 06:25:37 vps502497 apache2[11012]:  * Reloading Apache httpd web server apache2
Jan 21 06:25:37 vps502497 apache2[11012]:  *
Jan 21 06:25:37 vps502497 systemd[1]: Reloaded LSB: Apache2 web server.
Jan 21 06:25:43 vps502497 apache2[11041]:  * Stopping Apache httpd web server apache2
Jan 21 06:25:43 vps502497 apache2[11041]:  *

how can I fix this in order to avoid this crash?

For now, I just added systemctl restart apache2.service

Into "system and server status" , apache server /If monitor goes down, run command.

Also, I have removed the "Commands to run after rotation" /usr/sbin/apache2ctl graceful ; sleep 5

Submitted by JamieCameron on Sun, 01/21/2018 - 17:38

We have fixed this somewhat - in new virtualmin installs, all domains' log files are included in a single logrotate configuration block. However, this is not done for upgrades from installs that were originally setup years ago.

In your case, it looks like Apache crashed during the restart though.

Submitted by bombcar on Sun, 04/15/2018 - 06:59

I have the same exact problem, and only on Some Sundays and at 6:25 AM - it appears that something can cause apache2 to segfault - and the only culprit I can find is the logrotate settings for the only virtual domain I have running.

I've changed to:

/usr/sbin/apache2ctl restart ; sleep 10

but I'd rather not have to use a restart unless needed - what can I do to track it down further?

Submitted by andreychek on Sun, 04/15/2018 - 09:22

You can try using the single logrotate block technique that Jamie described in the comment above.

I don't have an example of that on-hand unfortunately, but you could always do a test install and copy the config from there.

Submitted by pnav on Tue, 04/17/2018 - 12:45 Pro Licensee

I'd like to 2nd and 3rd this. I admin for two different organizations with completely separate AWS accounts, but nearly identical setups. The first one was installed ~Feb 2018, the second was early April 2018. They both have this issue, with Apache crashing every 2-3 weeks. Unlike other commentators though, both systems only have a single vhost. The best response I've developed is to move the cron job to a time late at night when I'm still up, and use CloudWatch to monitor server response. Should things go down, at least I get an email about it so I can restart Apache within minutes.

I'd attach the syslog, but looks like it expired from a couple weeks ago. Will update should it happen again.

Submitted by bombcar on Tue, 04/17/2018 - 12:54

I almost wonder if the single vhost contributes to it - but if someone has an example of the "logrotate block" mentioned I'd love to experiment.

Submitted by JamieCameron on Tue, 04/17/2018 - 20:40

It should look like :

/var/log/virtualmin/domain1.com_access_log /var/log/virtualmin/domain1.com_error_log /var/log/virtualmin/domain2.com_access_log /var/log/virtualmin/domain2.com_error_log {
        rotate 5
        weekly
        compress
        postrotate
        service httpd graceful ; sleep 5
        endscript
        sharedscripts
}

Submitted by bombcar on Tue, 04/17/2018 - 20:43

That's basically already there (I only have one domain) - it's /var/log/apache2/*.log colliding with /var/log/virtualmin/domain.com_access_log as they seem to run at the same time and both try to graceful apache. (The files are /etc/logrotate.d/apache2 and /etc/logrotate.d/domain.com.conf respectively.

Submitted by JamieCameron on Wed, 04/18/2018 - 22:05

If you are only hosting Virtualmin domains, you can remove the entry for /var/log/apache2/*.log

Submitted by pnav on Thu, 04/19/2018 - 10:05 Pro Licensee

Jamie - that could help, I'll try that, thanks. Though what's "*.log" meant to capture if it isn't an error or access log? Want to make sure I'm not killing anything that's actually needed.

Thx.

Submitted by JamieCameron on Thu, 04/19/2018 - 22:07

Maybe it captures the default Apache logs, which shouldn't get any contents on a Virtualmin system.