Sendmail & Mailscanner version 2

[warnesj]

I think I have a fix for this now too. Sorry about that. I must have an older version of Perl because I don't get those warnings. At any rate, see if this works.

Find,

Code: Select all

open(MAILLOG, "tail -n 0 -f /var/log/maillog|") or die "Cannot open maillog: $!.";
while(!$time_to_die){

And replace it with,

Code: Select all

open(MAILLOG, "tail -n 0 -f /var/log/maillog|") or die "Cannot open maillog: $!.";
my $line="";
while(!$time_to_die){

Essentially you need to add my $line=""; on a line before the "while...." statement to initialize the $line variable that is used in the pattern matching.

Thanks for finding this. I'll update the main post too.

[Avenger]

I'm sorry to have to say that but I still have the same error
Adding this new line had not change the error.
Anything else ?
(Thx for helping me)

[warnesj]

Okay I've been digging away again. Unfortunately I can't replicate the problem on my Fedora Core installation. But how about we try this,

Find,

Code: Select all

while(!$time_to_die){
        $line=<MAILLOG>;
        # Look for received messages where the sender is not from our domain(s)

And replace it with,

Code: Select all

while(!$time_to_die && ( $line = <MAILLOG> )){
        # Look for received messages where the sender is not from our domain(s)

If that works then I'll make the changes permanent on the main page and upload a new version of the script.

[RattleSn@ke]

Hi!

Graph template from Sven worked indeed for me! (FC6, Cacti 0.8.6i)

The other problem I have is that i'm running the watchmaillog.sh script remotely on our mailserver. (FC5, Plesk8.1). On this server we've setup SpamAssassin which is running fine. It produces output in the maillog like

Code: Select all

Feb 26 21:23:39 giant spamd[21797]: spamd: processing message <000b01c759e3$fb7e38b0$6601a8c0@kibernet-gw.corbina.net> for user@domain.com:110
Feb 26 21:23:51 giant spamd[21797]: spamd: clean message (3.1/7.0) for user@domain.com:110 in 12.0 seconds, 35663 bytes.
Feb 26 21:23:51 giant spamd[21797]: spamd: result: . 3 - EXTRA_MPART_TYPE,HTML_10_20,HTML_MESSAGE,RCVD_IN_BL_SPAMCOP_NET scantime=12.0,size=35663,user=user@domain.com,uid=110,required_score=7.0,rhost=localhost,raddr=127.0.0.1,rport=/tmp/spamd_full.sock,mid=<000b01c759e3$fb7e38b0$6601a8c0@kibernet-gw.corbina.net>,autolearn=no

So I changed this::

Code: Select all

	# Look for MailScanner spam scanning batch results
	if($line=~/Spam\ Checks\:\ Found/){
		$item="spam";
		$spam_count_pos = index($line,"Spam\ Checks\:\ Found");
		$spam_count_pos2 = index($line, "\ spam\ messages");
		$spam_count = substr($line,($spam_count_pos+19),($spam_count_pos2-($spam_count_pos+19)));
		&readcounterfile;
		$counter{$item}=$counter{$item}+$spam_count;
		if($debug){print("Found $spam_count SPAM in the MailScanner batch, incrementing the spam counter to $counter{$item}.\n");}
		&writecounterfile;
	}

to this::

Code: Select all

        # Look for MailScanner spam scanning batch results
        if($line=~/is\ spam/ || $line=~/spamd\: identified spam/){
                $item="spam";
                $spam_count_pos = index($line,"Spam\ Checks\:\ Found");
                $spam_count_pos2 = index($line, "\ spam\ messages");
                $spam_count = substr($line,($spam_count_pos+19),($spam_count_pos2-($spam_count_pos+19)));
                &readcounterfile;
                $counter{$item}=$counter{$item}+$spam_count;
                if($debug){print("Found $spam_count SPAM in the MailScanner batch, incrementing the spam counter to $counter{$item}.\n");}
                &writecounterfile;
        }

But still all my counters are 0!!! :s
At plesk the maillog is at /usr/local/psa/var/log/maillog, so I symlinked /var/log/maillog -> /usr/local/psa/var/log/maillog

I also did a

Code: Select all

cat /var/log/maillog | grep "spamd: identified spam"

which gave me around 50 lines of found spam entries.
So I seems to work ...

Anyone an idea??

[EDIT]
I'm using ClamAV as virus-scanner. It is reporting to /var/log/clamav/clamd.log Can I change that somewhere??

Thanks!
Onno.

[warnesj]

It might be that the script is expecting numbers at certain positions. The "Spam Checks: Found" that spamd generated was the SpamAssassin results from a batch scan. So if you modify that search to look for individual "spamd: identified spam" messages then the numbers that the script is looking for (because it thinks the results are from a batch scan) aren't there. Can you post a copy of a syslog message that has your "spamd: identified spam" message? I can offer some suggestions for a mod.

[RattleSn@ke]

It might be that the script is expecting numbers at certain positions. The "Spam Checks: Found" that spamd generated was the SpamAssassin results from a batch scan. So if you modify that search to look for individual "spamd: identified spam" messages then the numbers that the script is looking for (because it thinks the results are from a batch scan) aren't there. Can you post a copy of a syslog message that has your "spamd: identified spam" message? I can offer some suggestions for a mod.

Hi!

Thanks for your reply. Since the maillog is symlinked: /var/log/maillog -> /usr/local/psa/var/log/maillog
Here's the output of [cat /var/log/maillog |grep "spamd: identified spam"]:

Code: Select all

Mar  1 10:20:55 giant spamd[17402]: spamd: identified spam (7.8/7.0) for user1@domain1.com:110 in 22.3 seconds, 3385 bytes.
Mar  1 11:06:48 giant spamd[17402]: spamd: identified spam (24.3/7.0) for user2@domain1.com:110 in 20.4 seconds, 13965 bytes.
Mar  1 11:12:45 giant spamd[17402]: spamd: identified spam (19.8/7.0) for user2@domain1.com:110 in 20.3 seconds, 1202 bytes.
Mar  1 11:41:46 giant spamd[17402]: spamd: identified spam (11.7/7.0) for user3@domain2.com:110 in 20.6 seconds, 2856 bytes.
Mar  1 12:20:30 giant spamd[17402]: spamd: identified spam (29.6/7.0) for user2@domain1.com:110 in 21.3 seconds, 12643 bytes.
Mar  1 12:49:13 giant spamd[17402]: spamd: identified spam (11.2/7.0) for user3@domain2.com:110 in 20.3 seconds, 1548 bytes.

Thanks!!
But note, all other counters also remain 0, and last type after a day I had around 20 perl processes... is that ok??

Greetz.
Onno.

[RattleSn@ke]

Hi there!!

I'd made some changes by myself, and have at least a graph that's working I think. (see attachment)

I know it's not a really interesting image, this because I only had some of all domains added for testing... but it's looking good I think.

Any comments are welcome!

Thanks.
Onno.

[Peer]

Yep,
I have got it working with ClamAV and Spamassassin.
Here is the snippet of my watchmaillog.sh (i have the orriginal lines commented out)

#!/usr/bin/perl
#
# Daemon used to watch the maillog messages for certain messages and trigger events when
# certain messages occur
#
# By Jason Warnes
#
# Change Log
# ~~~~~~~~~~
# 2006-08-22: Initial release
# 2006-09-05: Fixed signal handling
# Added /var/run/watchmaillog.pid file for logrotate usage
# 2006-09-07: Added proper signal handling (Thanks pvenezia!)
# Added SpamAssassin spamd checking support for SPAM (Thanks raiten!)
# 2006-09-18: Added new item mess_waiting, which is the number of messages MailScanner
# detects when a new batch is started.
# 2006-11-02: Changed the way SPAM messages were detected so the script works
# on servers configured for less verbose logging. (Thanks sdetroch!)
# 2006-11-08: Added new item mess_rejected, which is the number of rejected
# messages by Sendmail.
# 2007-02-06: Fixed <MAILLOG> close statement at end of main program. (Thanks Avenger!)
# Fixed warning messages about uninitialized $line used in pattern
# matching (Thanks Avenger!)


$debug=0; # 1=Debug messages are displayed, 0=No debug messages are displayed
$daemon=1; # 1=Daemonize the program, 0=Run interactive
$syslog=1; # 1=Log stuff to syslog, 0=No logging to syslog
$self="/home/scripts/reliant/cacti/watchmaillog_scripts/watchmaillog.sh"; # Location of this script
$counterfile="/home/scripts/reliant/cacti/watchmaillog_scripts/watchmaillog_counters"; # Location to store the counter file
$resetfile="/home/scripts/reliant/cacti/watchmaillog_scripts/watchmaillog_reset"; # Location of the reset counter flag file
$pidfile="/var/run/watchmaillog.pid"; # Location of the running process ID file (used in logrotate)

use Sys::Syslog;
use POSIX;
use Time::HiRes qw( gettimeofday tv_interval );

$|=1;

my $sigset = POSIX::SigSet->new();
my $hupaction = POSIX::SigAction->new('hup_signal_handler',
$sigset,
&POSIX::SA_NODEFER);
my $osigaction = POSIX::SigAction->new('signal_handler',
$sigset,
&POSIX::SA_NODEFER);
POSIX::sigaction(&POSIX::SIGHUP, $hupaction);
POSIX::sigaction(&POSIX::SIGINT, $osigaction);
POSIX::sigaction(&POSIX::SIGTERM, $osigaction);


if($daemon){
$pid=fork;
if($pid) {
open(PID,">".$pidfile) or die "Cannot open PID file: $!.";
print PID ("$pid\n"); # Write the PID out to the PID file for logrotate
close(PID);
}
exit if $pid;
die "Couldn't fork : $!" unless defined($pid);
setsid() or die "Can't start a new session: $!";
$time_to_die=0;
}

sub signal_handler {
$time_to_die=1;
}

sub hup_signal_handler {
if($debug){print "got SIGHUP\n";}
exec($self) or die "Couldn't restart: $!\n";
}

if($syslog){openlog("watchmaillog","pid","daemon");}
if($syslog){syslog("notice","Starting.");}
if($debug){print("watchmaillog is starting.\n");}

# Main part of the program
open(MAILLOG, "tail -n 0 -f /var/log/maillog|") or die "Cannot open maillog: $!.";
my $line="";
while(!$time_to_die){
$line=<MAILLOG>;
# Look for received messages where the sender is not from our domain(s)
if(($line=~/from\=/) && ($line!~/\@sonnehoek.nl|\@home.org|\@reliant.home.org/)){
$item="mess_recv";
&readcounterfile;
$counter{$item}++;
if($debug){print("Found an inbound message, incrementing the message recieve counter to $counter{$item}.\n");}
&writecounterfile;
}
# Look for messages sent to our domain(s), indicates an inbound message relayed to an internal server
if(($line=~/stat\=Sent/) && ($line=~/\@sonnehoek.nl|\@home.org|\@reliant.home.org/)){
$item="mess_relay";
&readcounterfile;
$counter{$item}++;
if($debug){print("Found an clean inbound message, incrementing the clean message recieve counter to $counter{$item}.\n");}
&writecounterfile;
}
# Look for sent messages to NOT our email domain(s), indicates an outbound message
if(($line=~/stat\=Sent/) && ($line!~/\@sonnehoek.nl|\@home.org|\@reliant.home.org/)){
$item="mess_sent";
&readcounterfile;
$counter{$item}++;
if($debug){print("Found an outbound message, incrementing the message sent counter to $counter{$item}.\n");}
&writecounterfile;
}
# Look for rejected messages
if((($line=~/ruleset/) && ($line=~/reject\=/)) || ($line =~/rejecting/)){
$item="mess_rejected";
&readcounterfile;
$counter{$item}++;
if($debug){print("Found a rejected message, incrementing the message rejected counter to $counter{$item}.\n");}
&writecounterfile;
}
# Look for Spamassassin spam results
# if($line=~/Spam\ Checks\:\ Found/){
if($line=~/X\-Spam\-Status\:\ Yes/){
$item="spam";
# $spam_count_pos = index($line,"Spam\ Checks\:\ Found");
# $spam_count_pos = index($line,"Spam\ Status\:\ Yes");
# $spam_count_pos2 = index($line, "\ spam\ messages");
# $spam_count = substr($line,($spam_count_pos+19),($spam_count_pos2-($spam_count_pos+19)));
$spam_count = 1;
&readcounterfile;
$counter{$item}=$counter{$item}+$spam_count;
if($debug){print("Found $spam_count SPAM in the Maillog, incrementing the spam counter to $counter{$item}.\n");}
&writecounterfile;
}
# Look for ClamAV virus scanning results
# if($line=~/Virus\ Scanning\:\ Found/){
if($line=~/Blocked\ INFECTED/){
$item="virus";
# $virus_count_pos = index($line,"Virus\ Scanning\:\ Found");
# $virus_count_pos2 = index($line, "\ viruses");
# $virus_count = substr($line,($virus_count_pos+22),($virus_count_pos2-($virus_count_pos+22)));
$virus_count = 1;
&readcounterfile;
$counter{$item}=$counter{$item}+$virus_count;
if($debug){print("Found $virus_count viruses in the Maillog, incrementing the virus counter to $counter{$item}.\n");}
&writecounterfile;
}
# Look for MailScanner waiting messages
if($line=~/New\ Batch\:\ Found/){
$item="mess_waiting";
$mess_waiting_pos = index($line,"New\ Batch\:\ Found");
$mess_waiting_pos2 = index($line,"\ messages\ waiting");
$mess_waiting = substr($line,($mess_waiting_pos+17),($mess_waiting_pos2-($mess_waiting_pos+17)));
&readcounterfile;
$counter{$item}=$mess_waiting;
if($debug){print("Mailscanner found $mess_waiting messages waiting, setting the mess_waiting counter to $counter{$item}.\n");}
&writecounterfile;
}
}
close(MAILLOG);
if($debug){print("watchmaillog is ending.\n");}
if($syslog){syslog("notice","Ending.");}
unlink($pidfile);

# Subroutine to read the contents of the counter file
sub readcounterfile {
# Read the counter values from the file
if($debug){print("Reading contents of counter file.\n");}
open(COUNTER,$counterfile);
while($line=<COUNTER>){
@line=split(/\:/,$line);
chop($line[1]); # Drop the trailing LF off the value
# Check for reset counter flag file
if(-e $resetfile."_".$line[0]){
if($debug){print("Reset counter flag file found for counter $line[0], resetting counter value to 0.\n");}
$counter{$line[0]}=0;
unlink($resetfile."_".$line[0]);
} else {
$counter{$line[0]}=$line[1];
}
if($debug){print("Counter $line[0] = $counter{$line[0]}.\n");}
}
close(COUNTER);
}

# Subrouting to write the contents of the counter file
sub writecounterfile {
if($debug){print("Writing counter values to counter file.\n");}
open(COUNTER,">".$counterfile);
# Write each counter item out to the counter file
foreach $item (sort keys(%counter)) {
print COUNTER ($item."\:".$counter{$item}."\n");
}
close(COUNTER);
chmod(0666,$counterfile);
}

Gr. Peer
http://peer.no-ip.org

[warnesj]

Sorry everyone, I've been unavailable lately for a while.

I'd made some changes by myself, and have at least a graph that's working I think. (see attachment)

Looks good RattleSn@ke. I'd be curious to see your changes if you have some time. The way the unmodified script is right now, the snippet of the log file you showed above wouldn't have triggered any SPAM counters. Usually I have the script look for the end of the MailScanner batch which has the summary of how many SPAM messages were found in the MailScanner batch. Usually those messages start with "Spam Checks: Found" and then it should have a number of some sort. Does you log file have any of those?

I have got it working with ClamAV and Spamassassin.

I'm guessing that you're not running MailScanner right? Most of the messages I'm triggering on for the SPAM and virus scanning are generated by MailScanner. It's nice to see that the script is being moded to work for other environments successfully. Thanks for sharing the changes Peer.

[gpetme]

I have successfully imported all of the latest graphs and data templates without error. Also, I have snmpwalked my sendmail host and verified the OIDs increment correctly with the amount of mail being received.

But, after I associate them with my sendmail host, I only see a blank entry in my graphs section. I don't even see an empty picture box, just the text for "E-Mail Statistics".

I'm on RHEL ES4, with Apache 2.2.x, Cacti 0.8.6j
, PHP 5.2.1, and no other problems. I think I've just missed a piece. Any ideas?

Thanks,

Greg

[warnesj]

But, after I associate them with my sendmail host, I only see a blank entry in my graphs section. I don't even see an empty picture box, just the text for "E-Mail Statistics".

Can you include a picture of what you're seeing? I'm not exactly sure what it could be yet.

[gpetme]

Sure, here's the screenshot. Also, I noticed yesterday that under Data Sources -> Add -> Selected Data Template none of the watchmaillog sources were listed. This morning they were. I've added them in, but the same behavior is being exhibited (see screenshot).

Thanks,

Greg

[ritter6281]

I have at least scanned through each of the postings on this topic. I am running 0.8.6j on RHEL v4 update 2 on a 32bit CPU system.

I have the SNMP side of things working, and I was having the same import problem, but I used the template provided by Sven and that fixed the issue I was having with the one graph template not loading. I checked the cacti interface to make sure that all of the templates were there.

The problem I am currently having is that I can't select the data and graph items when in the host area of Cacti. That is to say that I created a generic ucd/snmp host in cacti (which is the box I am monitoring sendmail on.) and I can create, much less monitor, those aspects of the host template just fine. But I can't add the sendmail-centric data templates but I can add the graph templates to the host.

What can I do to fix this (ie.: what I am doing wrong)?

[warnesj]

Okay sorry all for my absence. I'm back again. Can anyone having problems with this script post again? I'm not sure the status of them. Thanks.

[thomasch]

Hi!

Okay sorry all for my absence. I'm back again. Can anyone having problems with this script post again? I'm not sure the status of them. Thanks.

I'm using watchmaillog.sh on FreeBSD. Unfortunally the tail process isn't killed on restart via HUP. So after n restarts n+1 tail processes are running.

Adding the following patch solves the problem for me. Has anyone else seen this?

Code: Select all

--- watchmaillog.sh.new 2007/05/31 15:20:58     1.1
+++ watchmaillog.sh.new 2007/05/31 15:21:16
@@ -68,6 +68,7 @@

 sub hup_signal_handler {
       if($debug){print "got SIGHUP\n";}
+      close(MAILLOG);
       exec($self) or die "Couldn't restart: $!\n";
 }