Sendmail, Spamassassin, ClamAV statistics

Templates, scripts for templates, scripts and requests for templates.

Moderators: Developers, Moderators

Post Reply
caboom
Posts: 8
Joined: Mon Jul 05, 2010 1:54 pm

Sendmail, Spamassassin, ClamAV statistics

Post by caboom »

version 0.1 - last update: 07 July 2010

During my search for scripts and templates for generating statistics for sendmail, spamassassin and clamav, i couldn't find what i am looking for.
Since i'm not good in creating graph templates i took the template from http://forums.cacti.net/viewtopic.php?t=15651 written by warnesj. Thanks :)
The template is modified to use the gsum variable so it can display total number of mails. http://forums.cacti.net/about19336.html Thanks gauret :)
If you want the "Total" feature in the graph to work, you will have to follow gauret's instruction to add the gsum variable to your system too. Also cyberwizzard's addition for the /= 300 so that values are calculated correctly.

This is my version of the stats gathering script. I have no intentions of rewriting/altering it to work for other systems, I am just posting it in case people like me need it. Feel free to change it in any ways you need.

I am currently using clamav-milter and spamass-milter.
For virus messages to be identified, clamav-milter must add the X-Virus-Status: Infected header. You can do that by adding to your clamav-milter.conf :

Code: Select all

AddHeader Replace
and for spam messages to be identified your spamass-milter must add the X-Spam-Flag: YES header which is added by default. Make sure you don't have the -M flag in the parameters of the milter.

The software's main features are:
  • daemon - it's not parsing the whole log file (which can be 100s of MB) everytime it's executed
    sendmail integration - it uses the local-host-names and aliases file from sendmail so that it successfully identifies received mails
    light - it's not using up too much resources since it's a simple daemon written in C
    custom timer interval - you can set how often to write the stats file so that it's not rewriting the file everytime an email reaches the mailserver
    selective writing of stats file - it only writes the stats file when an event has happened (eg when an email has been marked as spam) and not just when the timer takes place
    logrotate support - when logs are rotated it re-opens the maillog file
Disadvantages:
  • only one poller - the stats file is being reset everytime snmp asks for it. so only one cacti poller can use them
Image

Installation:

1) compile the source using:

Code: Select all

gcc -o maillog_watcher maillog_watcher.c -lpthread
2) move the executable and the maillog_watcher_read.sh file to /opt/maillog_watcher/

3) add the following lines to your snmpd.conf

Code: Select all

exec .1.3.6.1.4.100.2 maillog_received /opt/maillog_watcher/maillog_watcher_read.sh received
exec .1.3.6.1.4.100.4 maillog_sent /opt/maillog_watcher/maillog_watcher_read.sh sent
exec .1.3.6.1.4.100.5 maillog_virus /opt/maillog_watcher/maillog_watcher_read.sh virus
exec .1.3.6.1.4.100.6 maillog_spam /opt/maillog_watcher/maillog_watcher_read.sh spam
exec .1.3.6.1.4.100.7 maillog_rejected /opt/maillog_watcher/maillog_watcher_read.sh rejected
4) for logrotate to work, modify your /etc/logrotate.d/syslog file to send a HUP signal to maillog_watcher. mine looks like this:

Code: Select all

/var/log/messages /var/log/secure /var/log/maillog /var/log/spooler /var/log/boot.log /var/log/cron {
    sharedscripts
    postrotate
        /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
        /bin/kill -HUP `cat /var/run/rsyslogd.pid 2> /dev/null` 2> /dev/null || true
        /bin/kill -HUP `cat /var/run/maillog_watcher.pid 2> /dev/null` 2> /dev/null || true
    endscript
}
5) You can now start the maillog_watcher

Code: Select all

[root@mail ~]# /opt/maillog_watcher/maillog_watcher
[root@mail ~]#
If your maillog file is located elsewhere, you can define it in line 187 of the maillog_watcher.c file.
Attachments
cacti_graph_template_maillog_watcher.xml
maillog_watcher graph and data templates
(41.22 KiB) Downloaded 291 times
maillog_watcher.c
maillog_watcher source file
(8.06 KiB) Downloaded 354 times
maillog_watcher_read.sh
maillog_watcher_read.sh file used to read the stats
(118 Bytes) Downloaded 298 times
xsnrg
Posts: 1
Joined: Thu Mar 05, 2009 11:36 am

Re: Sendmail, Spamassassin, ClamAV statistics

Post by xsnrg »

Greetings,

I was quite happy to find this post, and have everything working as described. The one question I do have though, why did you choose to make the template a gauge and reset the counter when you could make it a COUNTER and have it automatically do the deltas for you? Doing it this way means you can make another entry that is ABSOLUTE, to read the counter data directly from the file for the totals instead of using a custom gsum. In this way you also do not need to kill the daemon at each call of the read script.


I will be trying these modifications, but wanted to know if maybe there was a reason I was missing?
caboom
Posts: 8
Joined: Mon Jul 05, 2010 1:54 pm

Re: Sendmail, Spamassassin, ClamAV statistics

Post by caboom »

Hello xsnrg,

Thank you for your reply, I was happy to know that you used it and it works as expected.

Actually, I didn't know that there was a COUNTER feature and now that I know about it I will try and make another version which uses counter and not gauge (so that lots of pollers will be able to gather data at the same time).

The daemon is not killed at the end of the read script, it simply sends a USR1 signal, to reset the counters. Which is not needed anymore if I use COUNTER.

Thanks
rbl
Posts: 20
Joined: Sat Oct 07, 2006 12:35 pm

Re: Sendmail, Spamassassin, ClamAV statistics

Post by rbl »

Hi,

I did the installation but I can not get any output.
When I do snmpwalk .1 I get over a 100 lines output. So it is wide open to snmp.
The script is running but the grapics is still (nan)

Running Centos 6.2 en cacti 0.8.8.

What I am doing wrong?
rbl
rbl
Posts: 20
Joined: Sat Oct 07, 2006 12:35 pm

Re: Sendmail, Spamassassin, ClamAV statistics

Post by rbl »

Hi,
Found it.
The new version op net-snmp das not understand exec or pass.
Newer versions replaced "exec" by "extend", which works quite differently.

But, I get the next output:

Code: Select all

SNMPv2-SMI::private.100.2.1.0 = INTEGER: 1
SNMPv2-SMI::private.100.2.2.1.2.16.109.97.105.108.108.111.103.95.114.101.99.101.105.118.101.100 = STRING: "/opt/maillog_watcher/maillog_watcher_read.sh"
SNMPv2-SMI::private.100.2.2.1.3.16.109.97.105.108.108.111.103.95.114.101.99.101.105.118.101.100 = STRING: "received"
SNMPv2-SMI::private.100.2.2.1.4.16.109.97.105.108.108.111.103.95.114.101.99.101.105.118.101.100 = ""
SNMPv2-SMI::private.100.2.2.1.5.16.109.97.105.108.108.111.103.95.114.101.99.101.105.118.101.100 = INTEGER: 5
SNMPv2-SMI::private.100.2.2.1.6.16.109.97.105.108.108.111.103.95.114.101.99.101.105.118.101.100 = INTEGER: 1
SNMPv2-SMI::private.100.2.2.1.7.16.109.97.105.108.108.111.103.95.114.101.99.101.105.118.101.100 = INTEGER: 1
SNMPv2-SMI::private.100.2.2.1.20.16.109.97.105.108.108.111.103.95.114.101.99.101.105.118.101.100 = INTEGER: 4
SNMPv2-SMI::private.100.2.2.1.21.16.109.97.105.108.108.111.103.95.114.101.99.101.105.118.101.100 = INTEGER: 1
SNMPv2-SMI::private.100.2.3.1.1.16.109.97.105.108.108.111.103.95.114.101.99.101.105.118.101.100 = STRING: "0"
SNMPv2-SMI::private.100.2.3.1.2.16.109.97.105.108.108.111.103.95.114.101.99.101.105.118.101.100 = STRING: "0"
SNMPv2-SMI::private.100.2.3.1.3.16.109.97.105.108.108.111.103.95.114.101.99.101.105.118.101.100 = INTEGER: 1
SNMPv2-SMI::private.100.2.3.1.4.16.109.97.105.108.108.111.103.95.114.101.99.101.105.118.101.100 = INTEGER: 0
SNMPv2-SMI::private.100.2.4.1.2.16.109.97.105.108.108.111.103.95.114.101.99.101.105.118.101.100.1 = STRING: "0"
It seems that the string is the variable.
But.
The .stats file is not beeing updated.
It is running in memory

Code: Select all

root     30355  0.0  0.0 137392  1020 ?        Ssl  14:50   0:00 /opt/maillog_watcher/maillog_watcher
root     30358  0.0  0.0 100920   588 ?        S    14:50   0:00 tail -n 0 -f /var/log/maillog
rbl
caboom
Posts: 8
Joined: Mon Jul 05, 2010 1:54 pm

Re: Sendmail, Spamassassin, ClamAV statistics

Post by caboom »

Hello rbl,

What version of snmp server are you using? In my case I am using net-snmp-5.3.2.2-17.el5 (centos 5.8) and it seems to be working ok.

From the cacti side if you type:

Code: Select all

snmpwalk -Os -c public -v 1 <your_server_ip> .1.3.6.1.4.100.2.101.1
what is the output?

In my case:

Code: Select all

[root@cacti ~]# snmpwalk -Os -c public -v 1 my_mailserver .1.3.6.1.4.100.2.101.1
private.100.2.101.1 = STRING: "0"
[root@cacti ~]#
the .stats file should be being updated despite the fact that snmp doesn't feed data to cacti. If .stats file is not being updated, it means that maillog_watcher is not matching the appropriate lines in your mailllog file. the regular expressions in the .c file start at line 167. You should edit those lines to match your system configuration.

If you need help construction the regular expressions, let me know.
rbl
Posts: 20
Joined: Sat Oct 07, 2006 12:35 pm

Re: Sendmail, Spamassassin, ClamAV statistics

Post by rbl »

Hi caboom,

If have the next output with version : NET-SNMP version: 5.5
Code SNMP : .1.3.6.1.4.100.2.101.1 is not working (see last post)

Code: Select all

[root@server ~]# snmpwalk -Os -c public -v 1 localhost .1.3.6.1.4.100.2.4
private.100.2.4.1.2.16.109.97.105.108.108.111.103.95.114.101.99.101.105.118.101.100.1 = STRING: "6"
I just set number 6 in the file to see is the script is reading the file.

When I get a message, this is what I see in de logfile.
(not a real domain-name).

Code: Select all

Apr 26 08:59:59 server sendmail[4240]: q3Q6xxu2004240: from=<testmail@domain.com>, size=19488, class=0, nrcpts=1, msgid=<50B00B990F9DE64298F2D95D7EA8E6A40213A7B6@IC-SR-CPI-ML-01.domain.net>, proto=ESMTP, daemon=MTA, relay=smtp.relay.server.com [325.325.32.23]
Apr 26 09:00:02 server sendmail[4241]: q3Q6xxu2004240: to=<testuser@damain.com>, delay=00:00:03, xdelay=00:00:02, mailer=local, pri=49697, dsn=2.0.0, stat=Sent
caboom
Posts: 8
Joined: Mon Jul 05, 2010 1:54 pm

Re: Sendmail, Spamassassin, ClamAV statistics

Post by caboom »

you have to change the regular expression to match your logfile.

please change the lines 167, 175 and 183 to match your likings.

for example a regular expression for your receive line would be:

Code: Select all

"sendmail\\[[0-9]{2,9}\\]: [0-9a-zA-Z]{14}: to=<.*@%s>, delay=[0-9]{2}:[0-9]{2}:[0-9]{2}, xdelay=[0-9]{2}:[0-9]{2}:[0-9]{2}, mailer=local, pri=[0-9]{6}, dsn=2.0.0, stat=Sent"
in general, you need to change the mailer=cyrusv2 to mailer=local and remove the relay=localhost
rbl
Posts: 20
Joined: Sat Oct 07, 2006 12:35 pm

Re: Sendmail, Spamassassin, ClamAV statistics

Post by rbl »

Hi,

I have the receive working but the send is not.
This is the string in the logfile

Code: Select all

Apr 29 15:42:07 server sendmail[17309]: q3TDfuCu017286: to=<user@domain.com>, ctladdr=<user1@localdomain.com> (501/501), delay=00:00:10, xdelay=00:00:03, mailer=relay, pri=122529, relay=smtpout.servera.server.com. [345.221.552.239], dsn=2.0.0, stat=Sent (Accepted message qp 26194 bytes 5550)
rbl (not using real names/domains)
jsirucka
Posts: 1
Joined: Sun Apr 29, 2012 9:48 pm

Re: Sendmail, Spamassassin, ClamAV statistics

Post by jsirucka »

Hi All

I have installed the scripts and followed the instructions on there installations.

When I do a snmpwalk I get the following error

root@moscow:/opt/maillog_watcher# snmpwalk -Os -v 2c -c community tcp:localhost:10162 .1.3.6.1.4.100.2
private.100.2.1.1 = INTEGER: 1
private.100.2.2.1 = STRING: "maillog_received"
private.100.2.3.1 = STRING: "/opt/maillog_watcher/maillog_watcher_read.sh received"
private.100.2.100.1 = INTEGER: 1
private.100.2.101.1 = STRING: "/opt/maillog_watcher/maillog_watcher_read.sh: No such file or directory"
private.100.2.102.1 = INTEGER: 0
private.100.2.103.1 = ""

I have changed the script and the directories to 777 and still no go.

I have installed the scripts and templates on the host machine and the cacti server, to no avail.

I am running centos 5.3 on the client, and ubuntu 10.04 on the cacti server.
caboom
Posts: 8
Joined: Mon Jul 05, 2010 1:54 pm

Re: Sendmail, Spamassassin, ClamAV statistics

Post by caboom »

rbl wrote:Hi,

I have the receive working but the send is not.
This is the string in the logfile

Code: Select all

Apr 29 15:42:07 server sendmail[17309]: q3TDfuCu017286: to=<user@domain.com>, ctladdr=<user1@localdomain.com> (501/501), delay=00:00:10, xdelay=00:00:03, mailer=relay, pri=122529, relay=smtpout.servera.server.com. [345.221.552.239], dsn=2.0.0, stat=Sent (Accepted message qp 26194 bytes 5550)
rbl (not using real names/domains)

Hi rbl,

Please change the regular expression on line 183 to:

Code: Select all

"sendmail\\[[0-9]{2,9}\\]: [0-9a-zA-Z]{14}: to=<.*@.*>, ctladdr=<.*@.*> .*, delay=[0-9]{2}:[0-9]{2}:[0-9]{2}, xdelay=[0-9]{2}:[0-9]{2}:[0-9]{2}, mailer=.*, pri=[0-9]{6}, relay=.*, dsn=2.0.0, stat=Sent"
caboom
Posts: 8
Joined: Mon Jul 05, 2010 1:54 pm

Re: Sendmail, Spamassassin, ClamAV statistics

Post by caboom »

jsirucka wrote:Hi All

I have installed the scripts and followed the instructions on there installations.

When I do a snmpwalk I get the following error

root@moscow:/opt/maillog_watcher# snmpwalk -Os -v 2c -c community tcp:localhost:10162 .1.3.6.1.4.100.2
private.100.2.1.1 = INTEGER: 1
private.100.2.2.1 = STRING: "maillog_received"
private.100.2.3.1 = STRING: "/opt/maillog_watcher/maillog_watcher_read.sh received"
private.100.2.100.1 = INTEGER: 1
private.100.2.101.1 = STRING: "/opt/maillog_watcher/maillog_watcher_read.sh: No such file or directory"
private.100.2.102.1 = INTEGER: 0
private.100.2.103.1 = ""

I have changed the script and the directories to 777 and still no go.

I have installed the scripts and templates on the host machine and the cacti server, to no avail.

I am running centos 5.3 on the client, and ubuntu 10.04 on the cacti server.
Hello jsirucka,

Please make sure that the maillog_watcher_read.sh is in /opt/maillog_watcher directory. Probably you didn't copy it to the correct directory.
rbl
Posts: 20
Joined: Sat Oct 07, 2006 12:35 pm

Re: Sendmail, Spamassassin, ClamAV statistics

Post by rbl »

Hi caboom,

It works. :D :D
I also add local sendmail.
It is working great. Tnx.
rbl.
khakhovich
Posts: 7
Joined: Thu Sep 18, 2008 7:24 am

Re: Sendmail, Spamassassin, ClamAV statistics

Post by khakhovich »

are there an updated templates that work well when one have to use "extend" in snmpd vs "exec"
they don't seem to work out of the box if you have to use "extend" in snmpd.

if not too bad.... may be i will get a moment to adjust them on my end.
Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests