Cacti 1.1.36 with syslog 2.1 and syslog-ng 3.5

[B3Technology]

Hello there folks.

I am having a strange issue with syslog msgs not showing up under the syslog tab. I can see the msgs in the MySQL database and I see the Cacti logs showing the msgs as XFer which makes it seem like they are being moved somewhere else. I know I must be missing something simple .

I have researched this extensively through all of the available documentation.

Any thoughts would be greatly appreciated

Here is my config from a clean install:

Ubuntu 16.04 LTS
Cacti 1.1.36
syslog plugin 2.1
syslog-ng 3.5
Using the cactidb for syslog storage

Here is my syslog config.php

Code: Select all

<?php
/*
 +-------------------------------------------------------------------------+
 | Copyright (C) 2007-2014 The Cacti Group                                 |
 |                                                                         |
 | This program is free software; you can redistribute it and/or           |
 | modify it under the terms of the GNU General Public License             |
 | as published by the Free Software Foundation; either version 2          |
 | of the License, or (at your option) any later version.                  |
 |                                                                         |
 | This program is distributed in the hope that it will be useful,         |
 | but WITHOUT ANY WARRANTY; without even the implied warranty of          |
 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the           |
 | GNU General Public License for more details.                            |
 +-------------------------------------------------------------------------+
 | Cacti: The Complete RRDTool-based Graphing Solution                     |
 +-------------------------------------------------------------------------+
 | This code is designed, written, and maintained by the Cacti Group. See  |
 | about.php and/or the AUTHORS file for specific developer information.   |
 +-------------------------------------------------------------------------+
 | http://www.cacti.net/                                                   |
 +-------------------------------------------------------------------------+
*/

global $config, $database_type, $database_default, $database_hostname;
global $database_username, $database_password, $database_port;

/* revert if you dont use the Cacti database */
$use_cacti_db = true;

if (!$use_cacti_db) {
        $syslogdb_type     = 'mysql';
        $syslogdb_default  = 'cactidb';
        $syslogdb_hostname = 'localhost';
        $syslogdb_username = 'someuser';
        $syslogdb_password = 'somepassword';
        $syslogdb_port     = 3306;
}else{
        $syslogdb_type     = $database_type;
        $syslogdb_default  = $database_default;
        $syslogdb_hostname = $database_hostname;
        $syslogdb_username = $database_username;
        $syslogdb_password = $database_password;
        $syslogdb_port     = $database_port;
}

/* field in the incomming table */
$syslog_incoming_config['dateField']          = 'date';
$syslog_incoming_config['timeField']          = 'time';
$syslog_incoming_config['priorityField']      = 'priority_id';
$syslog_incoming_config['facilityField']      = 'facility_id';
$syslog_incoming_config['hostField']          = 'host_id';
$syslog_incoming_config['textField']          = 'message';
$syslog_incoming_config['id']                 = 'seq';

?>

Here is my syslog-ng.conf file:

Code: Select all

@version: 3.5
#Bare minimum syslog-ng configuration

# First, set some global options.
options { chain_hostnames(off); flush_lines(0); use_dns(no); use_fqdn(no);
          owner(root); group(adm); perm(0640); stats_freq(0);
          bad_hostname(^gconfd);
};

# we are using udp, and this is a collector for net traffic only
source s_all { udp(); };

destination d_mysql {
          sql(type(mysql)
          host("localhost") username("someuser") password("somepassword")
          database("cactidb") table("syslog_incoming")
          columns("facility", "priority", "date", "time", "host", "message")
          values("$FACILITY","","$YEAR-$MONTH-$DAY", "$HOUR:$MIN:$SEC","$HOST_FROM","$MSG")
          indexes("facility", "priority", "date", "time", "host", "msg"));
};

log { source(s_all); destination(d_mysql); };

Here is what the Cacti log is showing:

Code: Select all

03/07/2018 18:55:03 - SYSTEM SYSLOG STATS:Time:0.02 Deletes:0 Incoming:0 Removes:0 XFers:221 Alerts:0 Alarms:0 Reports:0
03/07/2018 18:55:03 - SYSTEM STATS: Time:1.2432 Method:cmd.php Processes:1 Threads:N/A Hosts:1 HostsPerProcess:1 DataSources:5 RRDsProcessed:5 

[Pucho]

I get the same thing under STATS. XFers might be related to the fact that it moves msgs from syslog_incoming to syslog.

09/Mar/2018 14:03:16 - SYSTEM SYSLOG STATS:Time:0.72 Deletes:0 Incoming:0 Removes:3 XFers:0 Alerts:18 Alarms:0 Reports:3
09/Mar/2018 14:02:34 - SYSTEM SYSLOG STATS:Time:0.73 Deletes:0 Incoming:0 Removes:0 XFers:7 Alerts:18 Alarms:0 Reports:3
09/Mar/2018 14:01:52 - SYSTEM SYSLOG STATS:Time:0.74 Deletes:0 Incoming:0 Removes:0 XFers:17 Alerts:18 Alarms:0 Reports:3
09/Mar/2018 14:00:47 - SYSTEM SYSLOG STATS:Time:0.71 Deletes:0 Incoming:0 Removes:0 XFers:33 Alerts:18 Alarms:0 Reports:3
09/Mar/2018 13:59:17 - SYSTEM SYSLOG STATS:Time:0.61 Deletes:0 Incoming:0 Removes:0 XFers:10 Alerts:18 Alarms:0 Reports:3
09/Mar/2018 13:58:34 - SYSTEM SYSLOG STATS:Time:0.72 Deletes:0 Incoming:0 Removes:0 XFers:21 Alerts:18 Alarms:0 Reports:3
09/Mar/2018 13:57:32 - SYSTEM SYSLOG STATS:Time:0.77 Deletes:0 Incoming:0 Removes:0 XFers:8 Alerts:18 Alarms:0 Reports:3
09/Mar/2018 13:56:50 - SYSTEM SYSLOG STATS:Time:0.69 Deletes:0 Incoming:0 Removes:0 XFers:11 Alerts:18 Alarms:0 Reports:3

You're only inserting 6 of 7 values, I'm surprised it's not failing.

This is mine rsyslog config. yours is missing "program". I'd expect the mysql query to fail but you say you're getting messages on your database. Any errors on clog?

$template cacti_syslog,"INSERT INTO syslog_incoming(facility_id, priority_id, program, date, time, host, message) \
values (%syslogfacility%, %syslogpriority%, '%programname%', '%timereported:::date-mysql%', '%timereported:::date-mysql%', '%HOSTNAME%', TRIM('%msg%'))", SQL

If you look at Configuration/Plugin Management/Syslog, is it Syslog enabled? Not sure if the Syslog TAB looks at syslog table or both syslog and syslog_incoming.

Code: Select all

Syslog Enabled
If this checkbox is set, records will be transferred from the Syslog Incoming table to the main syslog table and Alerts and Reports will be enabled. Please keep in mind that if the system is disabled log entries will still accumulate into the Syslog Incoming table as this is defined by the rsyslog or syslog-ng process.

[netniV]

Basically, that option means that you will only see things in the syslog table if it's enabled. Without that, no reporting etc will be triggered but your incoming table will still grow even though you see nothing.

[optimuscream]

Do not forget to set the crontab for syslog poller to run ..every interval time you decide to run .. This is the part I miss and took a lot of time to search which I don't recall anymore where I found it

Here is mine :

*/20 * * * * root /usr/bin/php /var/www/cactisyslog/plugins/syslog/syslog_process.php

Initially I run it every 5 minutes , but everytime it took longer to run and could take more than 10 minutes to finish .. Database could grow to 20 GB and more. So I decide to run it every 20 minutes and create removal rules to delete unnecesary log.

Hope it helps.

[netniV]

I'm not sure that having a separate cron file should be in place. The poller should kick off the syslog's pollers (which is what occurs in most circumstances).

[optimuscream]

I'm not really sure either.

In my case it works. I try to run cacti poller and clog not show syslog stats, only cacti poller stats. I don't run any graph poller , the unit just run for syslog :

Code: Select all

2018/03/28 10:30:00 - SYSTEM STATS: Time:298.4407 Method:cmd.php Processes:1 Threads:N/A Hosts:0 HostsPerProcess:0 DataSources:0 RRDsProcessed:0
2018/03/28 10:30:00 - POLLER: Poller[1] Maximum runtime of 298 seconds exceeded. Exiting. 

But syslog is updating . I can see it in syslog tabs with correct latest time.

With syslog process run as crontab , I got :

Code: Select all

2018/03/28 10:20:19 - SYSTEM SYSLOG STATS:Time:1218.14 Deletes:0 Incoming:38053 Removes:57 XFers:0 Alerts:22 Alarms:131 Reports:0 

So basically both cacti poller and syslog poller run in crontab works . Only issue with cacti poller , syslog stats not showed in clog.

Update :
I change poller to spine.
Now it works.

Code: Select all

2018/03/28 11:15:01 - SYSTEM STATS: Time:0.1127 Method:spine Processes:1 Threads:1 Hosts:0 HostsPerProcess:0 DataSources:0 RRDsProcessed:0
2018/03/28 11:11:53 - SYSTEM SYSLOG STATS:Time:2512.59 Deletes:0 Incoming:9524 Removes:17 XFers:0 Alerts:22 Alarms:36 Reports:0
2018/03/28 11:11:50 - SYSTEM SYSLOG STATS:Time:2209.9 Deletes:0 Incoming:2178 Removes:2 XFers:0 Alerts:22 Alarms:5 Reports:0

Now I have to deal with syslog stats issue run not enough time to finish because syslog_process.php filled up the linux process little by little if default to run for 5 minutes.

[Pucho]

In the future, don't add separated crontab jobs for other pollers within cacti because Cacti invokes plugin's poller every time it runs.

Add to your rsyslog cacti config the following line to prevent linux messages from getting to syslog database.

:fromhost , isequal , "yourserverhostname" stop