Apache doesn't start / URGENT - Need support!

Submitted by Blueforce on Sun, 03/03/2019 - 08:23 Pro Licensee

Need urgent support!

Something happened on our server a few hours ago and now Apache and MariaDB wont start!

CentOS 7.6.1810, Webmin 1.900, Usermin 1.751, Virtualmin 6.06 Pro.

I get this error when re-checking config

(2)No such file or directory: AH02291: Cannot access directory '/etc/httpd/logs/' for main error log
(2)No such file or directory: AH02291: Cannot access directory '/etc/httpd/logs/' for error log of vhost defined at /etc/httpd/conf.d/ssl.conf:56
AH00014: Configuration check failed

Please suggest what to do or I'll pay you to help me out.

Best regards, Leffe

Status: 
Closed (fixed)

Comments

Submitted by andreychek on Sun, 03/03/2019 - 09:18

Howdy -- thanks for contacting us!

What is the output of these commands:

ls -l /etc/httpd/
ls -l /var/log
dmesg | tail -30
uptime
df -h
rpm -qa | grep httpd
ls /etc/yum.repos.d

The error you're seeing suggests that the normal Apache log directory may no longer exist.

Are you aware of anything that may have changed recently? And are there other admins with root access?

Submitted by Blueforce on Sun, 03/03/2019 - 09:34 Pro Licensee

Hi,

I yestarday tried a few "bug tracker" scripts and selected Mantis, but some hours after that our customers started to call... DB and Apache was down.

I'm the only admin(root) no customers have virtualmin access. SSH is normaly shut down.

If I cant find what to do can you log in an check, of course I'll pay for that! We are are customers since way back... 14 years or so...

Please advice!

[root@serverone ~]# ls -l /etc/httpd/
total 8
drwxr-xr-x. 2 root root   35 Mar  3 15:10 conf
drwxr-xr-x. 2 root root 4096 Feb 15 07:10 conf.d
drwxr-xr-x. 2 root root 4096 Feb 15 07:09 conf.modules.d
lrwxrwxrwx  1 root root   19 Jul 11  2018 logs -> ../../var/log/httpd
lrwxrwxrwx  1 root root   29 Jul 11  2018 modules -> ../../usr/lib64/httpd/modules
lrwxrwxrwx  1 root root   10 Jul 11  2018 run -> /run/httpd [root@serverone ~]# ls -l /var/log
total 2780
-rw-r--r-- 1 root root     836 Mar  3 16:00 UcliEvt.log
-rw-r--r-- 1 root root   11568 Mar  3 16:00 arcerror.txt
-rw------- 1 root root   19164 Mar  3 13:46 boot.log
-rw------- 1 root utmp    4224 Mar  3 16:14 btmp
-rw------- 1 root root   12366 Mar  3 16:20 cron
-rw-r--r-- 1 root root   57651 Mar  3 13:46 dmesg
-rw-r--r-- 1 root root   58013 Mar  3 10:09 dmesg.old
-rw-r--r-- 1 root root  370256 Mar  3 16:10 lastlog
-rw------- 1 root root 1388200 Mar  3 16:22 maillog
-rw------- 1 root root  628165 Mar  3 16:22 messages
-rw------- 1 root mail   36513 Mar  3 16:22 procmail.log
-rw------- 1 root root  389418 Mar  3 16:22 secure
drwxr-xr-x 2 root root      22 Mar  3 10:10 tuned
drwx--x--x 2 root root       6 Mar  3 14:16 virtualmin
-rw-rw-r-- 1 root utmp    3840 Mar  3 13:46 wtmp
-rw------- 1 root root       0 Mar  3 10:46 yum.log [root@serverone ~]# dmesg | tail -30
[    7.714466] intel_rapl: Found RAPL domain core
[    7.714470] intel_rapl: Found RAPL domain dram
[    7.719813] intel_pmc_core:  initialized
[    7.872524] iTCO_vendor_support: vendor-support=0
[    7.873726] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11
[    7.873811] iTCO_wdt: unable to reset NO_REBOOT flag, device disabled by hardware/BIOS
[    7.913112] XFS (sda1): Mounting V4 Filesystem
[    8.036349] Adding 8191996k swap on /dev/mapper/centos-swap.  Priority:-2 extents:1 across:8191996k FS
[    8.186756] XFS (dm-2): Mounting V4 Filesystem
[    8.358430] power_meter ACPI000D:00: Found ACPI power meter.
[    8.469441] XFS (dm-2): Ending clean mount
[    8.496916] power_meter ACPI000D:00: Found ACPI power meter.
[    8.496985] power_meter ACPI000D:00: Ignoring unsafe software power cap!
[   15.137114] XFS (sda1): Ending clean mount
[   15.518787] nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
[   15.562396] ip6_tables: (C) 2000-2006 Netfilter Core Team
[   16.046031] IPv6: ADDRCONF(NETDEV_UP): eno1: link is not ready
[   16.092808] IPv6: ADDRCONF(NETDEV_UP): eno1: link is not ready
[   16.094753] IPv6: ADDRCONF(NETDEV_UP): eno2: link is not ready
[   16.141493] IPv6: ADDRCONF(NETDEV_UP): eno2: link is not ready
[   17.724958] igb 0000:02:00.0 eno1: igb: eno1 NIC Link is Up 100 Mbps Full Duplex, Flow Control: RX
[   17.725230] IPv6: ADDRCONF(NETDEV_CHANGE): eno1: link becomes ready
[   24.435629] hcpdriver: loading out-of-tree module taints kernel.
[   24.435633] hcpdriver: module license 'Proprietary' taints kernel.
[   24.435634] Disabling lock debugging due to kernel taint
[   24.435774] hcpdriver: module verification failed: signature and/or required key missing - tainting kernel
[   24.436418] hcp: INFO: hcp driver loaded: 4.6.0 112, NR_CPUS: 5120
[   24.436470] hcp: INFO: hcp_watchdog: started.
[ 7320.270691] Netfilter messages via NETLINK v0.30.
[ 7320.294522] ip_set: protocol 6 [root@serverone ~]# uptime
16:24:14 up  2:38,  0 users,  load average: 0.04, 0.05, 0.05 [root@serverone ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   50G  9.5G   41G  19% /
devtmpfs                 7.7G     0  7.7G   0% /dev
tmpfs                    7.7G  4.0K  7.7G   1% /dev/shm
tmpfs                    7.7G   17M  7.7G   1% /run
tmpfs                    7.7G     0  7.7G   0% /sys/fs/cgroup
/dev/mapper/centos-home  873G   19G  854G   3% /home
/dev/sda1                497M  295M  202M  60% /boot
tmpfs                    1.6G     0  1.6G   0% /run/user/0 [root@serverone ~]# rpm -qa | grep httpd
httpd-tools-2.4.6-80.el7.vm.3.x86_64
httpd-2.4.6-80.el7.vm.3.x86_64 [root@serverone ~]# ls /etc/yum.repos.d
CentOS-Base.repo
CentOS-CR.repo
CentOS-Debuginfo.repo
CentOS-Media.repo
CentOS-SCLo-scl-rh.repo
CentOS-SCLo-scl.repo
CentOS-Sources.repo
CentOS-Vault.repo
CentOS-fasttrack.repo
r1soft.repo
virtualmin.repo
virtualmin.repo.rpmsave

Submitted by andreychek on Sun, 03/03/2019 - 09:53

Hmm, the output you shared above is normal, other than the Apache log directory missing. That's a simple fix though.

Try re-creating it with the following:

mkdir /var/log/httpd

Once you do that, are you then able to start Apache?

Submitted by andreychek on Sun, 03/03/2019 - 09:56

Title: URGENT - Need support! » Apache doesn't start / URGENT - Need support!

Submitted by Blueforce on Sun, 03/03/2019 - 10:01 Pro Licensee

Now Apache started!

The Mariadb log directory also was missing, I created that as well so now the DB is up to!

Do you have any idea what could have been causing this?

//Leffe

Submitted by andreychek on Sun, 03/03/2019 - 10:10

That's great, I'm glad that helped!

I've honestly never heard of that happening before, and I'm not sure what might cause that.

I had hoped some of the commands you can above would help identify the culprit, but they appear normal.

For the moment, I think the best we can offer is to monitor things, and see if it happens again.

You could of course try reviewing the various other logs to see if anything unusual came up around the time that happened, though I don't have anything specific to suggest looking for.

Submitted by Blueforce on Sun, 03/03/2019 - 10:20 Pro Licensee

Tanks!

And as usual... Excellent support!

I'll monitor everything for a while, and as I said, a few scrips from Virtualmins list tried, and then deleted when I selected Mantis... that is the only thing that had changed... but wait, I updated Own Cloud to the latest version in in your script list.

I'll try to figure out what was causing this!

A really BIG thanks!

Submitted by Blueforce on Sun, 03/03/2019 - 10:20 Pro Licensee

Status: Active » Fixed

Submitted by IssueBot on Sun, 03/17/2019 - 11:30

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.