Cacti 1.1.30 Main poller unknown/down

[colorsteel]

Gooday

I installed Cacti 1.1.30 with Spine 1.1.30 and Mariadb 10.1.30, and because of a bug, spine cannot be installed in Mariadb 10.2.x.

My OS is Centos 7 server with GUI.

I followed this guide (https://www.youtube.com/watch?v=k_MAYuEtP7c). Copied and pasted the codes from this link while watching it, I copy the codes that being used in the tutorial from here (https://www.sharelinux.top/2018/01/huon ... tos-7.html) to the cli of centos 7, except I disabled Selinux and used Maria db 10.1.30 and skipped this step below I use the ip address of the machine to go to 10.x.x.x/cacti.

Code: Select all

<VirtualHost *:80>
ServerAdmin nguyenhappy92@gmail.com
DocumentRoot /var/www/html/cacti
ServerName your domain
ServerAlias www.your_domain
ErrorLog /var/www/html/error.log
CustomLog /var/www/html/access.log common
</VirtualHost>

The issue is with the device I've added, it cannot be graphed. The only device that can be graphed and device status up is the Linuxlocal machine. Also when I added a device, it shows status unknown and an SNMP error (see logs and screenshots). I install spine because I thought it will fix the problem but its still not working, the host/ip address, I put can be ping by the centos 7, when I add them in to device on cacti, it shows status unknown.

In our case, the purpose of Cacti is to monitor our internet connections, may it be in our head office or with our sattelite offices.

This is my first time setting up and using linux centos 7 and cacti and hoping that someone here in the community can assist me.

Thank you.

Code: Select all

2018/02/07 14:18:03 - POLLER: Poller[1] NOTE: Cron is configured to run too often! The Poller Interval is '300' seconds, with a minimum Cron period of '300' seconds, but only 60.4 seconds have passed since the poller last ran. 
2018/02/07 14:18:03 - POLLER: Poller[1] NOTE: Poller Int: '300', Cron Int: '300', Time Since Last: '60.4', Max Runtime '298', Poller Runs: '1' 
2018/02/07 14:18:02 - SYSTEM STATS: Time:298.2109 Method:spine Processes:1 Threads:1 Hosts:3 HostsPerProcess:3 DataSources:7 RRDsProcessed:0 
2018/02/07 14:18:02 - POLLER: Poller[1] Maximum runtime of 298 seconds exceeded. Exiting. 2018/02/07 14:17:03 - POLLER: Poller[1] NOTE: Poller Int: '300', Cron Int: '300', Time Since Last: '240.45', Max Runtime '298', Poller Runs: '1' 
2018/02/07 14:16:02 - POLLER: Poller[1] NOTE: Cron is configured to run too often! The Poller Interval is '300' seconds, with a minimum Cron period of '300' seconds, but only 179.7 seconds have passed since the poller last ran. 
2018/02/07 14:16:02 - POLLER: Poller[1] NOTE: Poller Int: '300', Cron Int: '300', Time Since Last: '179.75', Max Runtime '298', Poller Runs: '1' 
2018/02/07 14:15:01 - POLLER: Poller[1] NOTE: Cron is configured to run too often! The Poller Interval is '300' seconds, with a minimum Cron period of '300' seconds, but only 118.6 seconds have passed since the poller last ran. 
2018/02/07 14:15:01 - POLLER: Poller[1] NOTE: Poller Int: '300', Cron Int: '300', Time Since Last: '118.58', Max Runtime '298', Poller Runs: '1' 
2018/02/07 14:14:08 - POLLER: Poller[1] NOTE: Cron is configured to run too often! The Poller Interval is '300' seconds, with a minimum Cron period of '300' seconds, but only 65.6 seconds have passed since the poller last ran.
 2018/02/07 14:14:08 - POLLER: Poller[1] NOTE: Poller Int: '300', Cron Int: '300', Time Since Last: '65.59', Max Runtime '298', Poller Runs: '1' 
2018/02/07 14:14:00 - SYSTEM STATS: Time:298.8214 Method:spine Processes:1 Threads:1 Hosts:3 HostsPerProcess:3 DataSources:7 RRDsProcessed:0 2018/02/07 14:14:00 - POLLER: Poller[1] Maximum runtime of 298 seconds exceeded. Exiting. 
2018/02/07 14:13:03 - POLLER: Poller[1] NOTE: Poller Int: '300', Cron Int: '300', Time Since Last: '242.95', Max Runtime '298', Poller Runs: '1' 
2018/02/07 14:12:04 - POLLER: Poller[1] NOTE: Cron is configured to run too often! The Poller Interval is '300' seconds, with a minimum Cron period of '300' seconds, but only 183.6 seconds have passed since the poller last ran. 
2018/02/07 14:12:04 - POLLER: Poller[1] NOTE: Poller Int: '300', Cron Int: '300', Time Since Last: '183.6', Max Runtime '298', Poller Runs: '1' 2018/02/07 14:11:04 - POLLER: Poller[1] NOTE: Cron is configured to run too often! The Poller Interval is '300' seconds, with a minimum Cron period of '300' seconds, but only 123.7 seconds have passed since the poller last ran. 
2018/02/07 14:11:04 - POLLER: Poller[1] NOTE: Poller Int: '300', Cron Int: '300', Time Since Last: '123.71', Max Runtime '298', Poller Runs: '1' 
2018/02/07 14:10:08 - POLLER: Poller[1] NOTE: Cron is configured to run too often! The Poller Interval is '300' seconds, with a minimum Cron period of '300' seconds, but only 67.3 seconds have passed since the poller last ran. 
2018/02/07 14:10:08 - POLLER: Poller[1] NOTE: Poller Int: '300', Cron Int: '300', Time Since Last: '67.34', Max Runtime '298', Poller Runs: '1' 
2018/02/07 14:10:06 - POLLER: Poller[1] -1517983802.369 seconds 
2018/02/07 14:10:06 - CMDPHP PHP ERROR WARNING Backtrace: (/poller.php: 631 CactiErrorHandler)(/lib/functions.php: 4429 cacti_debug_backtrace) 
2018/02/07 14:10:06 - ERROR PHP WARNING: A non-numeric value encountered in file: /var/www/html/cacti/poller.php on line: 631 2018/02/07 14:10:02 - SYSTEM STATS: Time:300.4434 Method:spine Processes:1 Threads:1 Hosts:3 HostsPerProcess:3 DataSources:7 RRDsProcessed:0 2018/02/07 14:10:01 - POLLER: Poller[1] Maximum runtime of 298 seconds exceeded. Exiting.

[netniV]

Do you have the poller setup to run from cron? Normally this is via a /etc/cron.d/cacti file?

[netniV]

The reason I ask is that this status means that the poller you are looking at is set to status 3.

Status 3 is Unknown/Down which gets set if the current time, less that last status time is more than 310 seconds (5 minutes 10 seconds).

[Osiris]

His cron is set at 1 minute, and Cacti is setup for 5 minutes. The non-numeric data is likely that this is a VM and the clock not aligned with the host OS. There have been problems reported with microtime() in some versions of PHP providing interesting numbers.

Spine is not running due to a problem likely with his spine.conf file. You should always test spine from the command like like:

Code: Select all

./spine -R -V 3 -S

And make sure it can connect, before you go and enable it. Could also be the path in Console > Settings > Path is wrong.

[netniV]

Ah, I presume you've looked at the log file in more detail than I had time for yesterday.
Didn't know about the microtime issues, so I'll make a mental note about that.

I did also notice that there is an SNMP error against the host, that will cause delays if the packets are dropped by a firewall since it would wait to timeout.
I also noticed that one of the hostname's is invalid (or at least appears to be for me) since it has tcp at the front of it.

[colorsteel]

Good day,

netniV I change the cron.d file to 5 minutes and this error stops,

Code: Select all

POLLER: Poller[1] NOTE: Cron is configured to run too often! The Poller Interval is '300' seconds, with a minimum Cron period of '300' seconds, but only 179.7 seconds have passed since the poller last ran. 

and Osiris the command you give helps me, my spine.conf is not on the right folder, so I move it here /usr/local/spine/etc/spine.conf and my spine is now working the hints you guys gave me makes my device status goes up, and UDP and ICMP works, but my SNMP still have the same error when I add a device. and still the main poller is still uknown/down.


Now the devices/IP address i add, has the status up but don't show graphs, even it says its being graphed and see attached file and logs.

Thank you for the advice I manage to resolve some of the errors I'm facing, but some still persist.

I run this and here is the result.

Code: Select all

ls -la /var/www/html/cacti/

Code: Select all

drwxrwxr-x. 17 IT    IT       4096 Feb  6 02:38 .
drwxr-xr-x.  3 root  root       46 Feb  5 21:27 ..
-rw-rw-r--.  1 IT    IT       4298 Jan  3 18:01 about.php
-rw-rw-r--.  1 IT    IT      53335 Jan  3 18:01 aggregate_graphs.php
-rw-rw-r--.  1 IT    IT      18053 Jan  3 18:01 aggregate_items.php
-rw-rw-r--.  1 IT    IT      23853 Jan  3 18:01 aggregate_templates.php
-rw-rw-r--.  1 IT    IT      13127 Jan  3 18:01 auth_changepassword.php
-rw-rw-r--.  1 IT    IT      28292 Jan  3 18:01 auth_login.php
-rw-rw-r--.  1 IT    IT      14670 Jan  3 18:01 auth_profile.php
-rw-rw-r--.  1 IT    IT      22294 Jan  3 18:01 automation_devices.php
-rw-rw-r--.  1 IT    IT      36544 Jan  3 18:01 automation_graph_rules.php
-rw-rw-r--.  1 IT    IT      38160 Jan  3 18:01 automation_networks.php
-rw-rw-r--.  1 IT    IT      35567 Jan  3 18:01 automation_snmp.php
-rw-rw-r--.  1 IT    IT      19515 Jan  3 18:01 automation_templates.php
-rw-rw-r--.  1 IT    IT      35694 Jan  3 18:01 automation_tree_rules.php
-rw-rw-r--.  1 IT    IT       3232 Jan  3 18:01 boost_rrdupdate.php
drwxrwxr-x.  6 cacti apache     85 Jan  3 18:01 cache
-rw-rw-r--.  1 IT    IT     117722 Jan  3 18:01 cacti.sql
-rw-rw-r--.  1 IT    IT      29521 Jan  3 18:01 cdef.php
drwxrwxr-x.  2 IT    IT       4096 Jan  3 18:01 cli
-rw-rw-r--.  1 IT    IT       1899 Jan  3 18:01 clog.php
-rw-rw-r--.  1 IT    IT       1905 Jan  3 18:01 clog_user.php
-rw-rw-r--.  1 IT    IT      29539 Jan  3 18:01 cmd.php
-rw-rw-r--.  1 IT    IT       8212 Jan  3 18:01 cmd_realtime.php
-rw-rw-r--.  1 IT    IT      24103 Jan  3 18:01 color.php
-rw-rw-r--.  1 IT    IT      14078 Jan  3 18:01 color_templates_items.php
-rw-rw-r--.  1 IT    IT      21930 Jan  3 18:01 color_templates.php
-rw-rw-r--.  1 IT    IT      28586 Jan  3 18:01 data_input.php
-rw-rw-r--.  1 IT    IT      44579 Jan  3 18:01 data_queries.php
-rw-rw-r--.  1 IT    IT      32811 Jan  3 18:01 data_source_profiles.php
-rw-rw-r--.  1 IT    IT      58575 Jan  3 18:01 data_sources.php

Here are the notable logs I get now.

Code: Select all

2018/02/08 02:10:04 - SPINE: Poller[1] DEBUG: The Value of Active Threads is 0 for Device ID 5 2018/02/08 02:10:04 - SPINE: Poller[1] Device[DESKTOP] TH[1] DEBUG: HOST COMPLETE: About to Exit Device Polling Thread Function 
2018/02/08 02:10:04 - SPINE: Poller[1] Device[DESKTOP] TH[1] Total Time: 2 Seconds 
2018/02/08 02:10:04 - SPINE: Poller[1] ERROR: Problem initializing SNMP session 'tcp:10.20.0.8' 2018/02/08 02:10:03 - SPINE: Poller[1] Device[DESKTOP] TH[1] NOTE: There are '2' Polling Items for this Device 
2018/02/08 02:10:03 - SPINE: Poller[1] WARNING: Device[DESKTOP] TH[1] DataQuery[1] Reindex Check FAILED: No SNMP Session. If not an SNMP host, don't use Uptime Goes Backwards! 
2018/02/08 02:10:03 - SPINE: Poller[1] Device[DESKTOP] TH[1] RECACHE: Processing 1 items in the auto reindex cache for 'tcp:10.20.0.8' 
2018/02/08 02:10:03 - SPINE: Poller[1] Device[DESKTOP] PING: Result UDP: Device is Alive 2018/02/08 02:10:03 - SPINE: Poller[1] Device[DESKTOP] DEBUG: UDP Device Alive, Try Count:1, Time:0.5119 ms 
2018/02/08 02:10:03 - SPINE: Poller[1] Device[DESKTOP] DEBUG: Entering UDP Ping 
2018/02/08 02:10:03 - SPINE: Poller[1] ERROR: Problem initializing SNMP session 'tcp:10.20.0.8' 2018/02/08 02:10:02 - SPINE: Poller[1] Time: 1.0355 s, Threads: 1, Devices: 1 
2018/02/08 02:10:02 - SPINE: Poller[1] SPINE: The Final Value of Threads is 0 2018/02/08 02:10:02 - SPINE: Poller[1] Time: 300.7071 s, Threads: 1, Devices: 4 
2018/02/08 02:10:02 - SPINE: Poller[1] SPINE: The Final Value of Threads is 1 
2018/02/08 02:10:02 - SPINE: Poller[1] ERROR: Spine Timed Out While Waiting for Threads to End 2018/02/08 02:10:02 - SPINE: Poller[1] POLLR: Active Threads is 0, Pending is 0 
2018/02/08 02:10:02 - SPINE: Poller[1] Device[Rizal
2018/02/08 02:10:04 - SPINE: Poller[1] DEBUG: The Value of Active Threads is 0 for Device ID 5 2018/02/08 02:10:04 - SPINE: Poller[1] Device[DESKTOP] TH[1] DEBUG: HOST COMPLETE: About to Exit Device Polling Thread Function 
2018/02/08 02:10:04 - SPINE: Poller[1] Device[DESKTOP] TH[1] Total Time: 2 Seconds 
2018/02/08 02:10:04 - SPINE: Poller[1] ERROR: Problem initializing SNMP session 'tcp:10.20.0.8' 2018/02/08 02:10:03 - SPINE: Poller[1] Device[DESKTOP] TH[1] NOTE: There are '2' Polling Items for this Device 
2018/02/08 02:10:03 - SPINE: Poller[1] WARNING: Device[DESKTOP] TH[1] DataQuery[1] Reindex Check FAILED: No SNMP Session. If not an SNMP host, don't use Uptime Goes Backwards! 
2018/02/08 02:10:03 - SPINE: Poller[1] Device[DESKTOP] TH[1] RECACHE: Processing 1 items in the auto reindex cache for 'tcp:10.20.0.8' 
2018/02/08 02:10:03 - SPINE: Poller[1] Device[DESKTOP] PING: Result UDP: Device is Alive 2018/02/08 02:10:03 - SPINE: Poller[1] Device[DESKTOP] DEBUG: UDP Device Alive, Try Count:1, Time:0.5119 ms 
2018/02/08 02:10:03 - SPINE: Poller[1] Device[DESKTOP] DEBUG: Entering UDP Ping 
2018/02/08 02:10:03 - SPINE: Poller[1] ERROR: Problem initializing SNMP session 'tcp:10.20.0.8' 
2018/02/08 02:10:02 - SPINE: Poller[1] Time: 1.0355 s, Threads: 1, Devices: 1 
2018/02/08 02:10:02 - SPINE: Poller[1] SPINE: The Final Value of Threads is 0 
2018/02/08 02:10:02 - SPINE: Poller[1] Time: 300.7071 s, Threads: 1, Devices: 4 
2018/02/08 02:10:02 - SPINE: Poller[1] SPINE: The Final Value of Threads is 1 
2018/02/08 02:10:02 - SPINE: Poller[1] ERROR: Spine Timed Out While Waiting for Threads to End 2018/02/08 02:10:02 - SPINE: Poller[1] POLLR: Active Threads is 0, Pending is 0 2018/02/08 02:10:02 - SPINE: Poller[1] Device[Rizal

I debug one of the graphs

Code: Select all

/bin/rrdtool create \ 
/var/www/html/cacti/rra/antique_ping_21.rrd \ 
--step 300 \ 
DS:ping:GAUGE:600:0:5000 \ 
RRA:AVERAGE:0.5:1:600 \ 
RRA:AVERAGE:0.5:6:700 \ 
RRA:AVERAGE:0.5:24:775 \ 
RRA:AVERAGE:0.5:288:797 \ 
RRA:MIN:0.5:1:600 \ 
RRA:MIN:0.5:6:700 \ 
RRA:MIN:0.5:24:775 \
 RRA:MIN:0.5:288:797 \
 RRA:MAX:0.5:1:600 \ 
RRA:MAX:0.5:6:700 \
 RRA:MAX:0.5:24:775 \ 
RRA:MAX:0.5:288:797 \ 
RRA:LAST:0.5:1:600 \ 
RRA:LAST:0.5:6:700 \ 
RRA:LAST:0.5:24:775 \
 RRA:LAST:0.5:288:797 \

[netniV]

You image of the device didn't show SNMP information at the top left corner. This would normally appear above the ping results top left:


What are your SNMP settings and do you have the uptime set to SNMP at all?

[netniV]

The poller issue where it warned about a non-numeric value should have been resolved:
https://github.com/Cacti/cacti/issues/1317

[colorsteel]

Hi netniV was out of the office for a few days, appreciate your response here is my update.

I manage to make the main poller to status running but some times its on idle or most of the time, by reinstalling centos 7 and cacti, this time with the
same timezone as Osiris said about the host and the virtualbox not same timezone, now the main poller status down/unknown is fix I think, but still cant
get graphs on the installed internet connection on our branches using their static ip assign by our ISP. we have installed firewall in our office its fortinet,
can that affect the graphs ? snmp information is working i try the one you screen shot. here are some of the screen shots and noticeable logs.

Code: Select all

2018/02/21 17:10:06 - SYSTEM STATS: Time:4.3617 Method:spine Processes:1 Threads:1 Hosts:5 HostsPerProcess:5 DataSources:15 RRDsProcessed:10 
2018/02/21 17:10:05 - SPINE: Poller[1] Time: 3.5235 s, Threads: 1, Devices: 4 
2018/02/21 17:10:05 - SPINE: Poller[1] SPINE: The Final Value of Threads is 0
2018/02/21 17:10:05 - POLLER: Poller[1] Parsed MULTI output field '10min:0.05' [map 10min->load_15min] 
2018/02/21 17:10:05 - POLLER: Poller[1] Parsed MULTI output field '5min:0.03' [map 5min->load_5min] 
2018/02/21 17:10:05 - POLLER: Poller[1] Parsed MULTI output field '1min:0.08' [map 1min->load_1min] 
2018/02/21 17:10:04 - SPINE: Poller[1] POLLR: Active Threads is 0, Pending is 0 
2018/02/21 17:10:04 - SPINE: Poller[1] Device[Local Linux Machine] TH[1] Total Time: 0.14 Seconds 
2018/02/21 17:10:04 - SPINE: Poller[1] Device[Local Linux Machine] TH[1] Graphs[Local Linux Machine - Disk Space - /dev/sda1, Local Linux Machine - Disk Space - /dev/sda1] DS[12] SCRIPT: perl /var/www/html/cacti/scripts/query_unix_partitions.pl 'get' 'used' '/dev/sda1', output: 237980 
2018/02/21 17:10:04 - SPINE: Poller[1] Device[Local Linux Machine] TH[1] Graphs[Local Linux Machine - Disk Space - /dev/sda1, Local Linux Machine - Disk Space - /dev/sda1] DS[12] SCRIPT: perl /var/www/html/cacti/scripts/query_unix_partitions.pl 'get' 'available' '/dev/sda1', output: 800356 
2018/02/21 17:10:04 - SPINE: Poller[1] Device[Local Linux Machine] TH[1] Graphs[Local Linux Machine - Disk Space - /dev/mapper/centos-root, Local Linux Machine - Disk Space - /dev/mapper/centos-root] DS[11] SCRIPT: perl /var/www/html/cacti/scripts/query_unix_partitions.pl 'get' 'used' '/dev/mapper/centos-root', output: 4653012 
2018/02/21 17:10:04 - SPINE: Poller[1] Device[Local Linux Machine] TH[1] Graphs[Local Linux Machine - Disk Space - /dev/mapper/centos-root, Local Linux Machine - Disk Space - /dev/mapper/centos-root] DS[11] SCRIPT: perl /var/www/html/cacti/scripts/query_unix_partitions.pl 'get' 'available' '/dev/mapper/centos-root', output: 47750188
 2018/02/21 17:10:01 - SPINE: Poller[1] Device[TEST] ERROR: Problems parsing Multi SNMP OID! (oid: ), Set MAX_OIDS to 1 for this host to isolate bad OID
2018/02/21 17:10:06 - SYSTEM STATS: Time:4.3617 Method:spine Processes:1 Threads:1 Hosts:5 HostsPerProcess:5 DataSources:15 RRDsProcessed:10
 2018/02/21 17:10:05 - SPINE: Poller[1] Time: 3.5235 s, Threads: 1, Devices: 4 
2018/02/21 17:10:05 - SPINE: Poller[1] SPINE: The Final Value of Threads is 0 
2018/02/21 17:10:05 - POLLER: Poller[1] Parsed MULTI output field '10min:0.05' [map 10min->load_15min]
2018/02/21 17:10:05 - POLLER: Poller[1] Parsed MULTI output field '5min:0.03' [map 5min->load_5min]
2018/02/21 17:10:05 - POLLER: Poller[1] Parsed MULTI output field '1min:0.08' [map 1min->load_1min]
2018/02/21 17:10:04 - SPINE: Poller[1] POLLR: Active Threads is 0, Pending is 0 
2018/02/21 17:10:04 - SPINE: Poller[1] Device[Local Linux Machine] TH[1] Total Time: 0.14 Seconds 
2018/02/21 17:10:04 - SPINE: Poller[1] Device[Local Linux Machine] TH[1] Graphs[Local Linux Machine - Disk Space - /dev/sda1, Local Linux Machine - Disk Space - /dev/sda1] DS[12] SCRIPT: perl /var/www/html/cacti/scripts/query_unix_partitions.pl 'get' 'used' '/dev/sda1', output: 237980 
2018/02/21 17:10:04 - SPINE: Poller[1] Device[Local Linux Machine] TH[1] Graphs[Local Linux Machine - Disk Space - /dev/sda1, Local Linux Machine - Disk Space - /dev/sda1] DS[12] SCRIPT: perl /var/www/html/cacti/scripts/query_unix_partitions.pl 'get' 'available' '/dev/sda1', output: 800356 
2018/02/21 17:10:04 - SPINE: Poller[1] Device[Local Linux Machine] TH[1] Graphs[Local Linux Machine - Disk Space - /dev/mapper/centos-root, Local Linux Machine - Disk Space - /dev/mapper/centos-root] DS[11] SCRIPT: perl /var/www/html/cacti/scripts/query_unix_partitions.pl 'get' 'used' '/dev/mapper/centos-root', output: 4653012
2018/02/21 17:10:04 - SPINE: Poller[1] Device[Local Linux Machine] TH[1] Graphs[Local Linux Machine - Disk Space - /dev/mapper/centos-root, Local Linux Machine - Disk Space - /dev/mapper/centos-root] DS[11] SCRIPT: perl /var/www/html/cacti/scripts/query_unix_partitions.pl 'get' 'available' '/dev/mapper/centos-root', output: 47750188
2018/02/21 17:10:01 - SPINE: Poller[1] Device[TEST] ERROR: Problems parsing Multi SNMP OID! (oid: ), Set MAX_OIDS to 1 for this host to isolate bad OID

[colorsteel]

Here are the screen shots, my account keeps logging off, so I will post it on a different reply.

Thank you.

[netniV]

Any device that is going out of your network and through a firewall needs to be accessible via the SNMP port. It's always recommended to try and use an INTERNAL connection (over a secure tunnel if needed). However, if you MUST query internet based devices, you must have the rules in place to be able to do so.

In short, yes, your firewall is getting in the way.

As for the idleness, that will be because you have your poller time set for five minutes.