Sendmail & Mailscanner version 2

[warnesj]

Sounds like a permissions problem. If you're running Linux, BSD, or some other UNIX varient make sure that the account your Apache daemon runs under has permission to create files in your watchmaillog directory. Same thing with Windows. Whatever account your IIS service is using needs write access to the watchmaillog directory.

What happens is when the counter file is read there is a new file created in that directory to tell the watchmaillog daemon to reset the counter. If it never sees the "reset file" then the counters will keep their indefinately.

[rbl]

Hi,

The rights are good. See the example to see what I meen.


rbl

[warnesj]

Ah, I see what you mean now. I think I know what the problem is. The problems is two fold,

1. The main watchmaillog daemon is responsible for reseting the counter values in the counter file.
2. Your mail server doesn't recieve a lot of mail, so the main watchmaillog daemon isn't run as often as it would be run on busier servers.

So it is possible that when a counter value is read through SNMP the reset file is created, but the counter doesn't get reset until the main watchmaillog daemon is activated next. But the main watchmaillog daemon isn't activated because a new message doesn't come in. So when your counter is read again, the same value is returned.

I hope that made sense. But you've pointed out a big bug.

Try this code for a new Read Counters Script,
Read Counters Script
/opt/watchmaillog/readcounters.sh

Code: Select all

#!/usr/bin/perl
#
# Script called by Net-SNMP to read the watchmaillog counters
#
# By Jason Warnes
#
# 2006-09-18: Added new item mess_waiting, which is the number of messages MailScanner
#             detects when a new batch is started.
# 2006-10-11: Fixed bug where counter is flaged to be reset but the main watchmaillog
#             daemon hasn't run to reset the counter.  Don't reset it here, just return a
#             zero value.  (Thanks rbl!)

$counterfile="/opt/watchmaillog/watchmaillog_counters";
$resetfile="/opt/watchmaillog/watchmaillog_reset";
$oid{"spam"}=".1.3.6.1.4.100.2";
$oid{"virus"}=".1.3.6.1.4.100.4";
$oid{"mess_recv"}=".1.3.6.1.4.100.5";
$oid{"mess_sent"}=".1.3.6.1.4.100.6";
$oid{"mess_relay"}=".1.3.6.1.4.100.7";
$oid{"mess_waiting"}=".1.3.6.1.4.100.8";

&readcounterfile;
# If the counter is nothing set it to zero
if($counter{$ARGV[0]}==""){$counter{$ARGV[0]}=0;}
# If a counter reset file exists set the counter to zero
if(-e $resetfile."_".$ARGV[0]){$counter{$ARGV[0]}=0;}
# Return the value of the counter
print "$oid{$ARGV[0]}\ngauge\n$counter{$ARGV[0]}\n";
&createresetflagfile;

sub readcounterfile {   # Subroutine to read the contents of the counter file
        open(COUNTER,$counterfile);
        while($line=<COUNTER>){
                @line=split(/\:/,$line);
                chop($line[1]); # Drop the trailing LF off the value
                $counter{$line[0]}=$line[1];
        }
        close(COUNTER);
}

sub createresetflagfile {       # Subroutine to create the reset counter flag file
        open(RESET,">".$resetfile."_".$ARGV[0]);
        close(RESET);
        chmod(0666,$resetfile."_".$ARGV[0]);
}

That should fix it.

I've also updated the code on the first page of this post. Thanks rbl for pointing this out.

[rbl]

Hi,
The fix works great now. Tanks

[sdetroch]

Hello,

I tried to install this very nice script on our (linux) cacti server to monitor our (fc6) mailscanner server.

I had a problem (like others) to import the E-Mail statistics template (unmet dependencies), the rest of the templates imported fine. I'm running cacti 0.8.6i

A graph is being created for the "MailScanner Messages Waiting" (but I'm not sure that it contains the correct data, since all values are zero.
AND I don't get a graph for the combined info (MailScanner - E-Mail Statistics)

In the GraphTemplate debugging, I get:

Code: Select all

RRDTool Command:

/usr/bin/rrdtool graph - \
--imgformat=PNG \
--start=-86400 \
--end=-300 \
--title="MailScanner - E-Mail Statistics" \
--base=1000 \
--height=120 \
--width=500 \
--alt-autoscale-max \
--lower-limit=0 \
--vertical-label="# of Messages" \
--slope-mode \
--font TITLE:12:/usr/share/fonts/truetype/ttf-bitstream-vera/Vera.ttf \
--font AXIS:6:/usr/share/fonts/truetype/ttf-dejavu/DejaVuSerif.ttf \
CDEF:cdefbd=a,-1,* \
AREA:#00CF00:"Messages Received"  \
GPRINT::LAST:"    Current\:%8.0lf"  \
GPRINT::AVERAGE:"Average\:%8.0lf"  \
GPRINT::MAX:"Maximum\:%8.0lf\n"  \
AREA:#FF0000:"Viruses"  \
GPRINT::LAST:"    Current\:%8.0lf"  \
GPRINT::AVERAGE:"Average\:%8.0lf"  \
GPRINT::MAX:"Maximum\:%8.0lf\n"  \
AREA:#EA8F00:"SPAM":STACK \
GPRINT::LAST:"    Current\:%8.0lf"  \
GPRINT::AVERAGE:"Average\:%8.0lf"  \
GPRINT::MAX:"Maximum\:%8.0lf\n"  \
COMMENT:"\n"  \
AREA:cdefbd#7CB3F1:"Messages Sent"  \
GPRINT::LAST:"    Current\:%8.0lf"  \
GPRINT::AVERAGE:"Average\:%8.0lf"  \
GPRINT::MAX:"Maximum\:%8.0lf" 
RRDTool Says:

ERROR: invalid rpn expression in: a,-1,*

and the graph item fields on the graph mgt page are empty as well.
I seem not to be able to save the selected values.



I assume that I'm connecting the correct way to my mailscanner server (create device / snmp version 1 / generic snmp enabled host)

FYI: SNMP from mailscanner is returning snmp info (system, location, ...), so the connection should be ok (and (empty???) rrd files are being created as well). On the mailscanner server the counters are being updated as well.

Anyone an idea where I should check first, I think some errors are caused by the failed import, but yeah, how to correct them?

edit: In fact, how does the scripts know where to retrieve the data from on the mailscanner server, I haven't given this location anywhere (or is it included in the scripts)???

thanks a lot,
Sven

[warnesj]

I had a problem (like others) to import the E-Mail statistics template (unmet dependencies), the rest of the templates imported fine. I'm running cacti 0.8.6i

I have yet to upgrade to the 0.8.6i version yet. That unmet dependency thing has been haunting me since the original release. I have the "Include Dependencies" check-box selected when I exported the templates. Argh! Just to check, did all the data templates import okay? How about the Messages Waiting graph template? Was it just the MailScanner Stats graph template that is giving you trouble?

A graph is being created for the "MailScanner Messages Waiting" (but I'm not sure that it contains the correct data, since all values are zero.
AND I don't get a graph for the combined info (MailScanner - E-Mail Statistics)

In the GraphTemplate debugging, I get:

Code: Select all

RRDTool Command:

/usr/bin/rrdtool graph - \
--imgformat=PNG \
--start=-86400 \
--end=-300 \
--title="MailScanner - E-Mail Statistics" \
--base=1000 \
--height=120 \
--width=500 \
--alt-autoscale-max \
--lower-limit=0 \
--vertical-label="# of Messages" \
--slope-mode \
--font TITLE:12:/usr/share/fonts/truetype/ttf-bitstream-vera/Vera.ttf \
--font AXIS:6:/usr/share/fonts/truetype/ttf-dejavu/DejaVuSerif.ttf \
CDEF:cdefbd=a,-1,* \
AREA:#00CF00:"Messages Received"  \
GPRINT::LAST:"    Current\:%8.0lf"  \
GPRINT::AVERAGE:"Average\:%8.0lf"  \
GPRINT::MAX:"Maximum\:%8.0lf\n"  \
AREA:#FF0000:"Viruses"  \
GPRINT::LAST:"    Current\:%8.0lf"  \
GPRINT::AVERAGE:"Average\:%8.0lf"  \
GPRINT::MAX:"Maximum\:%8.0lf\n"  \
AREA:#EA8F00:"SPAM":STACK \
GPRINT::LAST:"    Current\:%8.0lf"  \
GPRINT::AVERAGE:"Average\:%8.0lf"  \
GPRINT::MAX:"Maximum\:%8.0lf\n"  \
COMMENT:"\n"  \
AREA:cdefbd#7CB3F1:"Messages Sent"  \
GPRINT::LAST:"    Current\:%8.0lf"  \
GPRINT::AVERAGE:"Average\:%8.0lf"  \
GPRINT::MAX:"Maximum\:%8.0lf" 
RRDTool Says:

ERROR: invalid rpn expression in: a,-1,*

and the graph item fields on the graph mgt page are empty as well.
I seem not to be able to save the selected values.

That is a problem most definitely. The reason why the "invalid rpn expresion" is coming up as an error is because there are no DEF's, meaning that the graph doesn't think it has any data sources. In comparison here is the output when I debug that graph on my system,

Code: Select all

RRDTool Command:

/usr/bin/rrdtool graph - \
--imgformat=PNG \
--start=-86400 \
--end=-300 \
--title="mx1 - E-Mail Statistics" \
--base=1000 \
--height=120 \
--width=500 \
--alt-autoscale-max \
--lower-limit=0 \
--units-exponent=0 \
--vertical-label="# of Messages" \
--slope-mode \
DEF:a="/var/www/html/cacti/rra/mx1_mess_recv_1982.rrd":mess_recv:AVERAGE \
DEF:b="/var/www/html/cacti/rra/mx1_viruses_1981.rrd":viruses:AVERAGE \
DEF:c="/var/www/html/cacti/rra/mx1_spam_1980.rrd":spam:AVERAGE \
DEF:d="/var/www/html/cacti/rra/mx1_mess_sent_1984.rrd":mess_sent:AVERAGE \
CDEF:cdefbd=d,-1,* \
AREA:a#00CF00:"Messages Recieved"  \
GPRINT:a:LAST:"Current\:%8.0lf"  \
GPRINT:a:AVERAGE:"Average\:%8.0lf"  \
GPRINT:a:MAX:"Maximum\:%8.0lf\n"  \
AREA:b#FF0000:"Viruses"  \
GPRINT:b:LAST:"          Current\:%8.0lf"  \
GPRINT:b:AVERAGE:"Average\:%8.0lf"  \
GPRINT:b:MAX:"Maximum\:%8.0lf\n"  \
AREA:c#EA8F00:"SPAM":STACK \
GPRINT:c:LAST:"             Current\:%8.0lf"  \
GPRINT:c:AVERAGE:"Average\:%8.0lf"  \
GPRINT:c:MAX:"Maximum\:%8.0lf\n"  \
COMMENT:"\n"  \
AREA:cdefbd#7CB3F1:"Messages Sent"  \
GPRINT:d:LAST:"    Current\:%8.0lf"  \
GPRINT:d:AVERAGE:"Average\:%8.0lf"  \
GPRINT:d:MAX:"Maximum\:%8.0lf" 

RRDTool Says:

OK

Notice the DEF's for a,b,c and d.

When you edit the graph in the Graph Management page, are the data sources showing up in the drop-down list as available to be selected? Or are there any data sources in the drop-down list?

I assume that I'm connecting the correct way to my mailscanner server (create device / snmp version 1 / generic snmp enabled host)

FYI: SNMP from mailscanner is returning snmp info (system, location, ...), so the connection should be ok (and (empty???) rrd files are being created as well). On the mailscanner server the counters are being updated as well.

I personally use SNMP version 2, I haven't tried version 1. And the host template I use is "ucd/net SNMP Host", but I would imagine the "Generic SNMP-enabled Host" should work just as well. I think the ucd/net SNMP Host gives you a few more templates like disk usage, CPU usage, load average and memory usage. To test that your Cacti server is able to read the SNMP OIDs that were created for MailScanner and sendmail try this command and see if you get a number returned,

Code: Select all

snmpget -v 1 -c %COMMUNITY_NAME% %IP_ADDR% .1.3.6.1.4.100.2

But replace %COMMUNITY_NAME% with what your community name is, and replace %IP_ADDR% with the IP address of your MailScanner host. That should do an SNMP v1 query on the OID for MailScanner SPAM detected. If that doesn't work try replacing the "-v 1" with "-v 2".

Anyone an idea where I should check first, I think some errors are caused by the failed import, but yeah, how to correct them?

If you can give me a list of which templates worked and which didn't that will help. If all the data templates imported okay, and the messages waiting graph template imported okay, then that will narrow the search down to us trying to figure out what failed on the import of the Mailscanner stats graph template.

edit: In fact, how does the scripts know where to retrieve the data from on the mailscanner server, I haven't given this location anywhere (or is it included in the scripts)???

The data templates tell Cacti what SNMP OID to look at for each item. Then by using those data templates on a device to create data sources, Cacti knows to poll those SNMP OIDs to save the values for each item in it's own RRA's. Then by using those data sources in the graph template the graph is suppose to display the values. Emphasis on "suppose to".

[sdetroch]

Thanks for your info.

I found the problem: The datasources were corrupt, problably caused by the failed import.
I recreated all datasources and the graphs started working fine. (the waiting message graph was working without the above fix).



However I still have one problem
The "SPAM" counters stays on 'zero' in the counters file (and, logically enough, in the rrd graph as well).
I'm using sendmail on FC6 and spamassasin in combination with Mailscanner.

A known problem or related to my config?
What does the script expects to find?

In my maillog:
[root@mailscanner watchmaillog]# grep spam /var/log/maillog
Nov 2 16:30:26 mailscanner MailScanner[19016]: Spam Checks: Found 2 spam messages
Nov 2 16:30:28 mailscanner MailScanner[18946]: Spam Checks: Found 1 spam messages
Nov 2 16:30:44 mailscanner MailScanner[18968]: Spam Checks: Found 1 spam messages

Is this logging ok?


EDIT:
I suppose that I need to change
# Look for MailScanner spam messages
if($line=~/is\ spam/ || $line=~/spamd\: identified spam/){
to reflect my logfile. Can someone tell me what to change before I do a trial and error

Is it normal I have other logfiles as the rest of you???

thanks,
Sven

[warnesj]

However I still have one problem
The "SPAM" counters stays on 'zero' in the counters file (and, logically enough, in the rrd graph as well).
I'm using sendmail on FC6 and spamassasin in combination with Mailscanner.

A known problem or related to my config?
What does the script expects to find?

The script is looking for lines that say "is spam" or "spamd: identified spam" in your maillog file. Here are a couple of entries from my maillog.

Code: Select all

Oct 29 00:37:18 mx1 MailScanner[12968]: Message k9T6awnX022672 from 88.163.40.158 (modestiarebe@clerk.com) to domain.com is spam, SBL+XBL, spamcop.net, SpamAssassin (cached, score=21.735, required 5, autolearn=disabled, HTML_MESSAGE 0.00, RAZOR2_CF_RANGE_51_100 0.50, RAZOR2_CF_RANGE_E4_51_100 1.50, RAZOR2_CF_RANGE_E8_51_100 1.50, RAZOR2_CHECK 0.50, RCVD_IN_BL_SPAMCOP_NET 1.33, RCVD_IN_NJABL_DUL 1.71, RCVD_IN_SORBS_DUL 1.99, RCVD_IN_XBL 3.11, URIBL_JP_SURBL 3.36, URIBL_SBL 1.09, URIBL_SC_SURBL 3.60, URIBL_WS_SURBL 1.53)
Oct 29 00:37:18 mx1 MailScanner[12139]: Message k9T6apeN022664 from 200.146.105.238 (demonicbag@rowhousecdc.org) to domain2.com is spam, SBL+XBL, SpamAssassin (not cached, score=7.563, required 5, autolearn=disabled, HTML_IMAGE_ONLY_16 0.63, HTML_MESSAGE 0.00, MIME_HTML_ONLY 0.00, RAZOR2_CHECK 0.50, RCVD_IN_BL_SPAMCOP_NET 1.33, RCVD_IN_SORBS_DUL 1.99, RCVD_IN_XBL 3.11)

NOTE:domain1.com and domain2.com are my domains. I just masked them to protect the innocent.
I don't know why your MailScanner logs aren't as verbose. I'm running MailScanner v4.54.6-1 and SpamAssassin v3.1.3-1 and I know there is a setting in the MailScanner.conf file called Log Spam, mine is set to "yes" and that might be what's causing your logs to be so lite. But if you're running a later version there may be other configuration options that may to turn on and off verbose logging. At any rate, go through your MailScanner.conf file and check for any logging related settings.

[sdetroch]

oeps, we posted at the same time

in the mailscanner config I have disabled this option for performance reasons (75% of incoming mail = spam)


# Do you want all spam to be logged? Useful if you want to gather
# spam statistics from your logs, but can increase the system load quite
# a bit if you get a lot of spam.
Log Spam = no


Can you assist me to change your code to reflect my logfile?
I prefer to go that way and not extra stressing my mailserver.
If it's not possible, I don't have a lot of choice

Thanks.

Sven

[warnesj]

oeps, we posted at the same time

LOL, yeah it happens.

Can you assist me to change your code to reflect my logfile? I prefer to go that way and not extra stressing my mailserver.

Absolutely! I didn't try with the less verbose logging, but it's a good idea you have. Give me a few minutes to whip up a code change and we'll see if it works. I'll post it shortly.

[sdetroch]

ok, think it would be usefull for (some) other people as well.
Maybe add it to the next version as well, than everyone has the choice.

kind regards,
Sven

[warnesj]

Okay try this code,

Find the original code that looks for SPAM messages in the watchmaillog.sh script,

Code: Select all

        # Look for MailScanner spam messages
        if($line=~/is\ spam/ || $line=~/spamd\: identified spam/){
                $item="spam";
                &readcounterfile;
                $counter{$item}++;
                if($debug){print("Found a spam message, incrementing the spam counter to $counter{$item}.\n");}
                &writecounterfile;
        }

And replace that with this,

Code: Select all

        # Look for MailScanner spam scanning batch results
        if($line=~/Spam\ Checks\:\ Found/){
                $item="spam";
                $spam_count_pos = index($line,"Spam\ Checks\:\ Found");
                $spam_count_pos2 = index($line, "\ spam\ messages");
                $spam_count = substr($line,($spam_count_pos+19),($spam_count_pos2-($spam_count_pos+19)));
                &readcounterfile;
                $counter{$item}=$counter{$item}+$spam_count;
                if($debug){print("Found $spam_count SPAM in the MailScanner batch, incrementing the spam counter to $counter{$item}.\n");}
                &writecounterfile;
        }

I based this code off the way I look for virus scanning results. It seems to be working okay on my server.

I'm not sure the effect this will have on people that use the spamd daemon for SpamAssassin, but I think this change should work for them. If raiten is still following this thread if he could try these changes out on his installation that would be great!

Maybe add it to the next version as well, than everyone has the choice.

I agree. I've actually added the change into this version because it is REALLY handy. I'll update the first page of this thread shortly. I just need to create a new ZIP file to include the new script.

[sdetroch]

Warnesj,

The new code is working very fine!
Thanks a lot and I keep following this thread.

regards,
Sven

[warnesj]

Warnesj,

The new code is working very fine!
Thanks a lot and I keep following this thread.

regards,
Sven

Great to hear! I've updated the first post too to include the code changes.

[sizulku]

how to add "rejected messages" to this graph?