Web pages not displaying [#40604]

Submitted by jmboyle58 on Tue, 05/03/2016 - 09:35

Email, FTP and access to web panels okay but no web pages showing, wordpress site etc. Example when I enter www.jmb01.com in browser address bar nothing happens, just a blank screen.

Status:

Active

Comments

Submitted by andreychek on Tue, 05/03/2016 - 09:51 Comment #1

Howdy -- hmm, the domain "jmb01.com" resolves to the IP "176.126.245.218" for me.

And I don't see to be able to connect to Apache, Webmin, or SSH on that IP address.

Is that the correct IP for your server?

Also, was this all working at one point, did something just stop recently? Or was this always a problem?

And is your server directly on the Internet, or is it behind a NAT router?

Submitted by jmboyle58 on Tue, 05/03/2016 - 10:11 Comment #2

Hi. Yeah that's the correct IP.

The server is directly on the Internet and everything was working the last time I checked about a week ago.

Postfix and Dovecot are still accessible and working at normal speed. FTP and SSH works also. The SSH port is 83.

Submitted by andreychek on Tue, 05/03/2016 - 10:22 Comment #3

Does it help to restart Apache?

What you may want to try is these commands:

service httpd stop
killall -9 httpd
service httpd start

After that, are you able to access any of your sites?

If not, are you seeing any errors in /var/log/httpd/error_log?

Submitted by jmboyle58 on Tue, 05/03/2016 - 10:50 Comment #4

Restarting Apache didn't help. Below is a dump from the error log. What is the MaxRequestWorkers setting mentioned at the end?

[Mon May 02 02:45:39.620272 2016] [auth_digest:notice] [pid 340] AH01757: generating secret for digest authentication ... [Mon May 02 02:45:39.632829 2016] [lbmethod_heartbeat:notice] [pid 340] AH02282: No slotmem from mod_heartmonitor [Mon May 02 02:45:39.755964 2016] [mpm_prefork:notice] [pid 340] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_fcgid/2.3.9 PHP/5.4.16 SVN/1.7.14 configured -- resuming normal operations [Mon May 02 02:45:39.756001 2016] [core:notice] [pid 340] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND' [Mon May 02 02:45:44.724607 2016] [mpm_prefork:notice] [pid 340] AH00171: Graceful restart requested, doing restart AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 176.126.245.218. Set the 'ServerName' directive globally to suppress this message [Mon May 02 02:45:57.198901 2016] [auth_digest:notice] [pid 340] AH01757: generating secret for digest authentication ... [Mon May 02 02:45:57.200071 2016] [lbmethod_heartbeat:notice] [pid 340] AH02282: No slotmem from mod_heartmonitor [Mon May 02 02:45:57.295539 2016] [mpm_prefork:notice] [pid 340] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_fcgid/2.3.9 PHP/5.4.16 SVN/1.7.14 configured -- resuming normal operations [Mon May 02 02:45:57.295574 2016] [core:notice] [pid 340] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND' [Mon May 02 02:46:00.425061 2016] [mpm_prefork:notice] [pid 340] AH00171: Graceful restart requested, doing restart AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 176.126.245.218. Set the 'ServerName' directive globally to suppress this message [Mon May 02 02:46:11.166763 2016] [auth_digest:notice] [pid 340] AH01757: generating secret for digest authentication ... [Mon May 02 02:46:11.167903 2016] [lbmethod_heartbeat:notice] [pid 340] AH02282: No slotmem from mod_heartmonitor [Mon May 02 02:46:11.251669 2016] [mpm_prefork:notice] [pid 340] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_fcgid/2.3.9 PHP/5.4.16 SVN/1.7.14 configured -- resuming normal operations [Mon May 02 02:46:11.251857 2016] [core:notice] [pid 340] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND' [Mon May 02 02:46:16.394400 2016] [mpm_prefork:notice] [pid 340] AH00171: Graceful restart requested, doing restart AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 176.126.245.218. Set the 'ServerName' directive globally to suppress this message [Mon May 02 02:46:29.861588 2016] [auth_digest:notice] [pid 340] AH01757: generating secret for digest authentication ... [Mon May 02 02:46:29.862725 2016] [lbmethod_heartbeat:notice] [pid 340] AH02282: No slotmem from mod_heartmonitor [Mon May 02 02:46:29.940547 2016] [mpm_prefork:notice] [pid 340] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_fcgid/2.3.9 PHP/5.4.16 SVN/1.7.14 configured -- resuming normal operations [Mon May 02 02:46:29.940585 2016] [core:notice] [pid 340] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND' [Mon May 02 19:43:06.345375 2016] [mpm_prefork:notice] [pid 340] AH00170: caught SIGWINCH, shutting down gracefully [Mon May 02 19:44:35.902517 2016] [suexec:notice] [pid 665] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Mon May 02 19:44:36.116070 2016] [auth_digest:notice] [pid 665] AH01757: generating secret for digest authentication ... [Mon May 02 19:44:36.117154 2016] [lbmethod_heartbeat:notice] [pid 665] AH02282: No slotmem from mod_heartmonitor [Mon May 02 19:44:36.541444 2016] [mpm_prefork:notice] [pid 665] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_fcgid/2.3.9 PHP/5.4.16 SVN/1.7.14 configured -- resuming normal operations [Mon May 02 19:44:36.541510 2016] [core:notice] [pid 665] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND' [Mon May 02 19:44:43.565754 2016] [mpm_prefork:error] [pid 665] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting [Mon May 02 19:55:25.283028 2016] [mpm_prefork:notice] [pid 665] AH00170: caught SIGWINCH, shutting down gracefully [Mon May 02 19:55:35.800709 2016] [suexec:notice] [pid 1875] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Mon May 02 19:55:35.882819 2016] [auth_digest:notice] [pid 1875] AH01757: generating secret for digest authentication ... [Mon May 02 19:55:35.883959 2016] [lbmethod_heartbeat:notice] [pid 1875] AH02282: No slotmem from mod_heartmonitor [Mon May 02 19:55:35.957091 2016] [mpm_prefork:notice] [pid 1875] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_fcgid/2.3.9 PHP/5.4.16 SVN/1.7.14 configured -- resuming normal operations [Mon May 02 19:55:35.957157 2016] [core:notice] [pid 1875] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND' [Mon May 02 19:55:42.977934 2016] [mpm_prefork:error] [pid 1875] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting [Tue May 03 11:24:19.185029 2016] [mpm_prefork:notice] [pid 1875] AH00170: caught SIGWINCH, shutting down gracefully [Tue May 03 11:25:17.668388 2016] [suexec:notice] [pid 9473] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 176.126.245.218. Set the 'ServerName' directive globally to suppress this message [Tue May 03 11:25:17.894207 2016] [auth_digest:notice] [pid 9473] AH01757: generating secret for digest authentication ... [Tue May 03 11:25:17.895317 2016] [lbmethod_heartbeat:notice] [pid 9473] AH02282: No slotmem from mod_heartmonitor [Tue May 03 11:25:18.013457 2016] [mpm_prefork:notice] [pid 9473] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_fcgid/2.3.9 PHP/5.4.16 SVN/1.7.14 configured -- resuming normal operations [Tue May 03 11:25:18.013533 2016] [core:notice] [pid 9473] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND' [Tue May 03 11:25:23.051686 2016] [mpm_prefork:error] [pid 9473] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting

Submitted by andreychek on Tue, 05/03/2016 - 12:07 Comment #5

Hmm, I'm seeing this error in your logs:

server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting

If your server has reached it's maximum allowed connections, that could cause the behavior you're seeing.

Is one of your websites perhaps getting hit particularly hard?

Also, what is the output of these two commands:

uptime
ps auxwf

Submitted by jmboyle58 on Tue, 05/03/2016 - 12:31 Comment #6

                      Last login: Tue May  3 13:15:20 2016 from cpc2-grth10-2-0-cust59.16-4.cable.virginm.net

[root@server ~]# uptime load average: 0.76, 0.84, 0.87 rel="nofollow">root@server ~]# ps auxwf STAT START TIME COMMAND Ss May02 0:15 init -z S May02 0:00 [kthreadd/6752] S May02 0:00 _ [khelper/6752] Ss May02 0:15 /usr/lib/systemd/systemd-journald Ss May02 0:00 /usr/lib/systemd/systemd-udevd Ss May02 0:02 /usr/lib/systemd/systemd-logind Ss May02 0:04 /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile - Ss May02 0:01 /usr/sbin/saslauthd -m /run/saslauthd -a pam -n 2 -r S May02 0:01 _ /usr/sbin/saslauthd -m /run/saslauthd -a pam -n 2 -r Ssl May02 0:05 /usr/sbin/rsyslogd -n Ss May02 0:00 /usr/sbin/sshd -D Ss 11:23 0:00 _ sshd: root@notty Ss 13:15 0:00 _ sshd: root@pts/0 Ss 13:27 0:00 _ -bash R+ 13:28 0:00 _ ps auxwf Ss May02 0:00 /usr/sbin/xinetd -stayalive -pidfile /var/run/xinetd.pid Ss May02 0:02 /usr/sbin/dovecot -F S May02 0:00 _ dovecot/anvil S May02 0:00 _ dovecot/log S 10:10 0:00 _ dovecot/imap S 10:39 0:00 _ dovecot/imap S 12:58 0:00 _ dovecot/imap S 13:00 0:00 _ dovecot/imap S 13:00 0:00 _ dovecot/imap S 13:00 0:00 _ dovecot/imap S 13:00 0:00 _ dovecot/imap S 13:15 0:00 _ dovecot/imap S 13:15 0:00 _ dovecot/imap S 13:15 0:00 _ dovecot/imap S 13:15 0:00 _ dovecot/imap S 13:15 0:00 _ dovecot/imap S 13:15 0:00 _ dovecot/imap S 13:15 0:00 _ dovecot/imap S 13:15 0:00 _ dovecot/imap S 13:15 0:00 _ dovecot/imap S 13:15 0:00 _ dovecot/imap S 13:15 0:00 _ dovecot/imap S 13:15 0:00 _ dovecot/imap S 13:15 0:00 _ dovecot/imap S 13:15 0:00 _ dovecot/imap S 13:15 0:00 _ dovecot/imap S 13:15 0:00 _ dovecot/imap S 13:18 0:00 _ dovecot/imap S 13:19 0:00 _ dovecot/imap S 13:19 0:00 _ dovecot/imap S 13:19 0:00 _ dovecot/imap S 13:19 0:00 _ dovecot/imap S 13:27 0:00 _ dovecot/config Ss+ May02 0:00 /sbin/agetty --noclear --keep-baud console 115200 38400 9600 vt220 Ss May02 0:00 /usr/sbin/crond -n Ss+ May02 0:00 /sbin/agetty --noclear tty2 linux Ss May02 0:00 /bin/sh /usr/bin/mysqld_safe --basedir=/usr Sl May02 2:54 _ /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --p Ssl May02 0:05 /usr/sbin/named -u named Ss May02 0:00 /usr/bin/python /usr/lib/mailman/bin/mailmanctl -s start S May02 0:11 _ /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=ArchRunne S May02 0:11 _ /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=BounceRun S May02 0:11 _ /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=CommandRu S May02 0:11 _ /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=IncomingR S May02 0:11 _ /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=NewsRunne S May02 0:11 _ /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=OutgoingR S May02 0:11 _ /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=VirginRun S May02 0:00 _ /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=RetryRunn Ss May02 0:15 /usr/bin/spamd --pidfile /var/run/spamd.pid -d -c -m5 -H S 11:16 0:50 _ spamd child S 12:03 0:04 _ spamd child Ss May02 0:01 /usr/libexec/postfix/master -w S 08:59 0:00 _ qmgr -l -t unix -u S 12:19 0:00 _ pickup -l -t unix -u S 13:18 0:00 _ anvil -l -t unix -u S 13:26 0:00 _ smtpd -n smtp -t inet -u -o stress= -s 2 -o smtpd_sasl_auth_ena S 13:26 0:00 _ trivial-rewrite -n rewrite -t unix -u Ss May02 0:01 /usr/bin/perl /usr/libexec/usermin/miniserv.pl /etc/usermin/miniser Ss May02 0:06 /usr/bin/perl /usr/libexec/webmin/miniserv.pl /etc/webmin/miniserv. Ss May02 0:01 proftpd: (accepting connections) Ss 09:15 0:01 /usr/libexec/webmin/virtual-server/lookup-domain-daemon.pl Ss 11:25 0:00 /usr/sbin/httpd -DFOREGROUND S 11:25 0:00 _ /usr/sbin/httpd -DFOREGROUND S 11:25 6:40 | _ /bin/php-cgi S 11:25 6:44 | _ /bin/php-cgi S 11:25 6:32 | _ /bin/php-cgi S 11:25 6:26 | _ /bin/php-cgi S 11:25 6:16 | _ /bin/php-cgi S 11:25 6:29 | _ /bin/php-cgi S 11:25 6:24 | _ /bin/php-cgi S 11:25 6:31 | _ /bin/php-cgi S 11:25 6:24 | _ /bin/php-cgi S 11:32 0:04 | _ /bin/php-cgi S 11:32 0:04 | _ /bin/php-cgi S 11:37 0:07 | _ /bin/php-cgi S 13:15 0:44 | _ /bin/php-cgi S 11:25 0:00 _ /usr/sbin/httpd -DFOREGROUND S 11:25 0:00 _ /usr/sbin/httpd -DFOREGROUND S 11:25 0:00 _ /usr/sbin/httpd -DFOREGROUND S 11:25 0:00 _ /usr/sbin/httpd -DFOREGROUND S 11:25 0:00 _ /usr/sbin/httpd -DFOREGROUND S 11:25 0:00 _ /usr/sbin/httpd -DFOREGROUND S 11:25 0:00 _ /usr/sbin/httpd -DFOREGROUND S 11:25 0:00 _ /usr/sbin/httpd -DFOREGROUND S 11:25 0:00 _ /usr/sbin/httpd -DFOREGROUND S 11:25 0:00 _ /usr/sbin/httpd -DFOREGROUND rel="nofollow">root@server ~]#

Submitted by andreychek on Tue, 05/03/2016 - 13:16 Comment #7

Okay, that all looks pretty normal.

It's been working a bit better for me over the last hour or so, but just to see if anything unusual shows up, what does this command show:

netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr | head

Submitted by jmboyle58 on Tue, 05/03/2016 - 13:48 Comment #8

It just says -bash: netstat: command not found

I see in the AWstats that there's been over 191,000 hits on jmb01.com today alone from a couple of Russian IP addresses. I'll block these in IP tables and see if that helps.

Thanks for your help. I'll let you know if I have any more problems.

Submitted by andreychek on Tue, 05/03/2016 - 13:53 Comment #9

You may also want to take a peek at $HOME/logs/access_log for that domain to see if any IP's are currently hammering away on certain parts of your site.

Also, it looks like netstat may not be installed -- you can install that by running "yum install net-tools".

Submitted by jmboyle58 on Tue, 05/03/2016 - 14:25 Comment #10

Okay. I installed net-tools. Here is the output.

[root@server ~]# netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr | head 74 185.103.252.170 41 185.130.4.120 27 185.103.252.3 23 77.103.236.60 17 185.130.4.197 12 104.25.19.33 6 104.25.18.33 5 143.95.251.2 4 142.4.26.236 3 2.126.104.206 [root@server ~]#

Also, here's a portion of the access log file as it's massive and the same all the way down,

185.103.252.170 - - [03/May/2016:15:01:42 -0400] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)" 185.130.4.120 - - [03/May/2016:15:01:43 -0400] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)" 185.103.252.170 - - [03/May/2016:15:01:33 -0400] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)" 185.103.252.170 - - [03/May/2016:15:01:33 -0400] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)" 185.103.252.170 - - [03/May/2016:15:01:44 -0400] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)" 185.130.4.120 - - [03/May/2016:15:01:38 -0400] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)" 185.130.4.197 - - [03/May/2016:15:01:47 -0400] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)" 185.103.252.170 - - [03/May/2016:15:01:38 -0400] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)" 185.103.252.170 - - [03/May/2016:15:01:39 -0400] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)" 185.130.4.120 - - [03/May/2016:15:01:40 -0400] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)" 185.130.4.197 - - [03/May/2016:15:01:50 -0400] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)" 185.130.4.197 - - [03/May/2016:15:01:51 -0400] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)" 185.130.4.120 - - [03/May/2016:15:01:43 -0400] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)" 185.103.252.3 - - [03/May/2016:15:01:53 -0400] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)" 185.130.4.120 - - [03/May/2016:15:01:44 -0400] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)" 185.103.252.170 - - [03/May/2016:15:01:45 -0400] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)" 185.130.4.197 - - [03/May/2016:15:01:45 -0400] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)" 185.103.252.170 - - [03/May/2016:15:01:55 -0400] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)" 185.103.252.170 - - [03/May/2016:15:01:45 -0400] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)" 185.103.252.170 - - [03/May/2016:15:01:45 -0400] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)" 185.103.252.170 - - [03/May/2016:15:01:56 -0400] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"

Submitted by andreychek on Tue, 05/03/2016 - 14:35 Comment #11

Ah goodness, yeah that looks like a lot bots hammering on your WordPress installation.

I'd probably start banning any IP address that's hitting "/xmlrpc.php" until thinks are back to normal on your server.

Submitted by jmboyle58 on Tue, 05/03/2016 - 14:46 Comment #12

Okay will do. Thank you again.