hi all -
i am trying to put together a script to read all activity in my log files from the last 15 minutes. the best answer i came up with is to use the unix 'tac' (cat backwards!) command embedded into a php script. then compare the timestamps to a specially formatted timestamp called 'earliest date'.
the code below works, but somehow it seems like a pretty clumsy way to do this.
any other ideas?
<div class="codeblock"><pre><code><span style="color: #000000"><span style="color: #0000BB"><?php<br /></span><span style="color: #007700">&</span><span style="color: #FF8000">#10; $earliest_date = date('Y-m-d H:i:s' , strtotime('-15 minutes')); $filename = '/var/log/virtualmin/marksdomain.com_access_log' ; $file = popen("/usr/bin/tac $filename 2>/dev/null;", 'r'); while ($line = fgets($file)) { ##example: 168.8.56.7 - - [26/Sep/2013:10:59:04 -0400] "GET /wp-content/ $regex = '|^(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}) - - \[(\d{2}/\w{3}/\d{4}:\d{2}:\d{2}:\d{2})|'; preg_match($regex, $line, $matches ) ; //$matches[1] is the IP number, while $matches[2] is the date. $activityDateTime = date_create_from_format('d/M/Y:H:i:s', $matches[2])->format('Y-m-d H:i:s') ; // reformat the date to a more php-friendly format if ( $activityDateTime >= $earliest_date ) { echo $line ; } else { pclose($file); break; } } <br /></span><span style="color: #0000BB">?></span></span></code></pre></div>
i am expanding it to include all log files. specifically i am looking for wordpress attacks:
<?php
// example: /var/log/virtualmin/marksdomain.com_access_log
$earliest_date = date('Y-m-d H:i:s' , strtotime('-15 minutes'));
define ( 'DIRECTORY' , '/var/log/virtualmin' );
if ( $handle = opendir(constant('DIRECTORY')) ) {
while ( false !== ($filename = readdir($handle))) {
if ( $filename != '.' && $filename != '..' ) {
$fullFileName = constant('DIRECTORY') . '/' . $filename ;
$path_parts = pathinfo($fullFileName);
if ( $path_parts['extension'] === 'com_access_log' ) {
$ipArray = array();
$p_openHandle = popen("tac $fullFileName 2>/dev/null",'r');
while ($line = fgets($p_openHandle)) {
##example: 168.8.56.7 - - [26/Sep/2013:10:59:04 -0400] "GET /wp-content/themes/english2/images/back.jpg HTTP/1.1" 200 389 "htt
$regex = '|^(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}) - - \[(\d{2}/\w{3}/\d{4}:\d{2}:\d{2}:\d{2}) -\d{4}|';
preg_match($regex, $line, $matches ) ;
$activityDateTime = date_create_from_format('d/M/Y:H:i:s', $matches[2])->format('Y-m-d H:i:s') ;
if ( $activityDateTime >= $earliest_date ) {
if ( preg_match('/wp-login.php/', $line ) ) {
if (!(array_key_exists($matches[1], $ipArray ))) {
$ipArray[$matches[1]] = 1 ;
} else {
$ipArray[$matches[1]]++;
}
}
} else {
pclose($p_openHandle);
break 1;
}
} // end while
if ( $ipArray ) {
echo $filename;
echo "\n";
arsort($ipArray);
print_r($ipArray);
}
} // end if
} // end if
} //end while
}
?>
hello - my problem all along is people trying to break into my wordpress installations. although all the WP sites have captcha and limit-login-lockdown, i still wish to be able to block anybody who even tries.
this script below runs frequently, and puts any offending IP number into the csf blacklist.
i put in parameters passed from crontab, and now loop through the entire virtualmin log file. currently it runs every 10 minutes on my server. if there is a breakin attempt, i am notified via email.
<?php
/* wordpressLoginTracker.php
program to scan virtualmin log files looking for wordpress login attempts
parameters:
1) max allowed count in time period
2) time period in minutes
2013-10-01 - initial writing
*/
// /var/log/virtualmin/englishwithoutaccent.com_access_log
define ( 'MAX_ALLOWED_ACCESSES' , $argv[1] );
define ( 'TIME_TO_CHECK' , '-' . $argv[2] . ' minutes' );
define ( 'DIRECTORY' , '/var/log/virtualmin' );
define ( 'VIRTUALMIN_LOG_SUFFIX' , 'com_access_log' );
define ( 'SEND_TO' , 'mark@markyboy.com' );
$earliest_date = date('Y-m-d H:i:s' , strtotime(constant('TIME_TO_CHECK')));
echo 'running on: ' . date("D M d, Y G:i a") . ' looking for ' . $earliest_date . "\n" ; // log this run
if ( $handle = opendir(constant('DIRECTORY')) ) {
while ( false !== ($filename = readdir($handle))) {
if ( $filename != '.' && $filename != '..' ) {
$fullFileName = constant('DIRECTORY') . '/' . $filename ;
$path_parts = pathinfo($fullFileName);
if ( $path_parts['extension'] === constant('VIRTUALMIN_LOG_SUFFIX') ) {
$ipArray = array();
$p_openHandle = popen("tac $fullFileName 2>/dev/null",'r');
while ($line = fgets($p_openHandle)) {
##example: 168.8.56.7 - - [26/Sep/2013:10:59:04 -0400] "GET /wp-content/themes/english2/images/back.jpg HTTP/1.1" 200 389 "htt
$regex = '|^(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}) - - \[(\d{2}/\w{3}/\d{4}:\d{2}:\d{2}:\d{2}) -\d{4}|';
preg_match($regex, $line, $matches ) ;
$activityDateTime = date_create_from_format('d/M/Y:H:i:s', $matches[2])->format('Y-m-d H:i:s') ;
if ( $activityDateTime >= $earliest_date ) {
if ( preg_match('/wp-login.php/', $line ) ) {
if (!(array_key_exists($matches[1], $ipArray ))) {
$ipArray[$matches[1]] = 1 ;
} else {
$ipArray[$matches[1]]++;
}
}
} else {
pclose($p_openHandle);
break 1;
}
} // end while we still have stuff to read in the file
if ( $ipArray ) {
//echo $filename;
arsort($ipArray);
foreach ($ipArray as $ipNbr => $ipCount ) {
if ( $ipCount > constant('MAX_ALLOWED_ACCESSES') ) {
exec("/usr/sbin/csf --deny $ipNbr") ;
$to = constant('SEND_TO') ;
$subject = 'WP attack attempt' ;
$headers = 'From: info@comptonpeslonline.com' . "\r\n"
. 'Reply-To: info@comptonpeslonline.com' . "\r\n"
. 'X-Mailer: PHP/' . phpversion()
;
$message = $filename . "\r\n"
. 'http://whatismyipaddress.com/ip/' . $ipNbr . "\r\n"
. 'Nbr of attempts: ' . $ipCount . "\r\n"
;
mail ($to, $subject, $message, $headers );
echo $message;
} else {
break 1;
}
} // end foreach
} // end if $ipArray has anything in it or not
} // end if file is a virtualmin log file
} // end if file is anything besides a single dot or a double dot
} //end while we still have files to process
}
?>
You should probably use a command like logtail to only process logfile lines that have been added since last run. Going thru the whole logfile each time can be a performance killer.
true, but i didn't think the "tac" command reads through the entire file. does anybody know ?
i just did a 'tac' on a couple of the largest text files i could find and the command executed near instantly.
tac /home/marksDomain.com/public_html/DebugLogs/eval.log | head ;
hi locutus -
i looked at logtail -- it looked very interesting, but it appears to be an ajax powered log reader. i guess i was looking for the easiest way to just look at the last 15 minutes of log activity, from inside a crontab bash script.
a month ago i was on the highway when i learned my server was under a horrible attack that was shutting our servers down! since then i have enacted synflood protection on CSF, and that seems to have slowed it down.
however, there is little one can do about it when you are driving in-between broadband-enabled cell towers!
Ajax powered? Not that I know of... It's a Linux shell/command line tool that outputs those lines of a given logfile (or any file really) that have been added since last call.
http://linux.die.net/man/8/logtail
ok i looked at the wrong one:
http://sourceforge.net/projects/logtail/
but you have to admit they are spelled similar ! maybe they are related somehow?
i still cant help but think there may be something to using the 'tac' command in conjunction with the 'head' command, but the command-line logtail command might be more fun.
mystery solved!
the logtail you mention is a perl script:
http://www.fourmilab.ch/webtools/logtail/
while the one that google found is some ajax-webpage tool.
does anybody have any example of using logtail? i cant seem to find anything out there.
i am concerned that logtail is pretty old. it does not look like it is actively maintained.
also, for fun i took a BIG log file and cat'ed it to a test log file in order to make it a two-gig file. then i issued a
tac temp.log | head -5 ;
and this command happened instantly ! this tells me that tac may be a suitable method to read off the bottom of a log file.
but again, if anybody has a working example of logtail i would appreciate it.
What's complicated about logtail that you need a "working example"? It just outputs everything in the logfile that's been added since the last run (for that, it stores the "current position" in an additional file).
Of course, if you feed it an artificial 2 GB logfile, it will initially output ALL of it, which of course takes VERY long! So question is, what do you need/want? If you just want to always read the last X lines from a file,
tail -nis the way to go. If you have to process ALL lines that have been added, so as not to miss any and also not process any multiple times, use "logtail".Doesn't matter if it's being maintained or something. Software that just works and has a very simple purpose doesn't need to be updated all the time. I'm using it in several places in my scripts, but nothing more than just
logtail FILENAMEand piping that to other commands. Nothing fancy about it.hi locutus -
you will recall the ferengi needed extra help navigating their starship, and Will Riker was more than happy to assist.
here is what i did so far:
this command works fine:
Usage: logtail [ options ] logfile Version 1.1 -- 2002 August 7. blah blah blah
create a small test log and feed it into logtail:
this sits there forever, no output.
this also sits there forever
same
if i run it in debug, it goes into a function at 564 check-dns, and then sits there in an endless loop on line 886: recv(SOCKDNS, $msg, 65535, 0);
this suggests to me its going over the network, by default.
i am excited about using logtail, assuming i can get it working!
I'm not sure that you're using the correct logtail. Mine is located in
/usr/sbin/logtail, no ".pl" extension, and I also don't need to run it through "perl". Ubuntu 12.04 here. I installed logtail from an Ubuntu repository package.It runs fine for me. The first call e.g.
logtail /var/log/syslogoutputs the whole file, no delays or anything. Creates/var/log/syslog.offsetto store its last processing position. Subsequent calls only output what was added to syslog.locutus - up front, let me once again thank you for helping me.
could i please trouble you to get me your logtail version? here is what my logtail did (oddly, whatever i have needs a dash-u to show help)
perl /usr/lib/perl/logtail.pl -u ; Usage: logtail [ options ] logfile ... Version 1.1 -- 2002 August 7. Options: -d Show full domain name for echo messages. -ehostname Echo to named host. Multiple -e options echo to multiple hosts. -lport Listen for echo on port (default 5741), 0 = don't listen. -n No forking for -r DNS lookups -pinterval Reopen log files every interval minutes (default 15), 0 = never. -q Quiet: don't print on local machine, just echo. -rtime Resolve: attempt to find host names from IP addresses in the log. Cache timeout: time secs. -stime Sleep time seconds between polls of files. -tport Transmit echo on port (default 5741). -u Print this message. -vlevel Verbose: generate debug output. -wcols Wrap lines at cols columns, 0 = no wrap. The latest version of this program is available from: http:/www.fourmilab.ch/webtools/logtail/ Report bugs to bugs@fourmilab.chthank you very much, locutus!
locutus - by any possible chance did you mean logcheck rather than logtail ?
now i know how the "Pakleds" felt when the Enterprise had to rescue them !
My logtail doesn't have the "-u" option. As I said, although it does seem to be a perl script as per the shebang, but it's named just "logtail", not "logtail.pl".
No, I didn't mean "logcheck" (which I also have in use), I did mean logtail.
Here's the package information from Ubuntu:
and that's ALL it took !
Locutus, i want to offer you my most sincere appreciation for your patience regaring this matter!
You're welcome, and good luck with your coding endeavor! :)