New host graphs showing -nan

[drewdown12765]

Unable to figure this out for the life of me. snmpwalk workts, rrd files are there but the graphs are blank and show -nan for just the one host.

Logs show this over and over:

01/10/2018 09:05:13 AM - SPINE: Poller[0] ERROR: Spine Timed Out While Processing Hosts Internal
01/10/2018 09:05:02 AM - POLLER: Poller[0] WARNING: There are '1' detected as overrunning a polling process, please investigate
01/10/2018 09:05:01 AM - SYSTEM SYSLOG STATS:Time:0.51 Deletes:0 Incoming:0 Removes:0 XFers:22 Alerts:0 Alarms:0 Reports:0
01/10/2018 09:05:01 AM - SYSTEM THOLD STATS: Time:0.1877 Tholds:6 TotalHosts:29 DownHosts:2 NewDownHosts:0
01/10/2018 09:05:01 AM - SYSTEM STATS: Time:58.9215 Method:spine Processes:1 Threads:1 Hosts:30 HostsPerProcess:30 DataSources:4384 RRDsProcessed:2218
01/10/2018 09:05:01 AM - POLLER: Poller[0] Maximum runtime of 58 seconds exceeded. Exiting.

ls -lah shows last update to rrd file was on the 8th whereas other hosts are updated as of the current date/time:

ls -lah r2-2951-blr_traffic_in_2944.rrd
-rw-r--r-- 1 root root 185K Jan 8 17:40 r2-xxx-seat_traffic_in_2944.rrd

Any idea what the problem could be?

[netniV]

I would suspect that the overrunning device would be this one. Check your timeout values and if nothing seems obvious, try putting that host into the debug hosts list in settings.

[drewdown12765]

SNMP timeout is 500 seconds.

Not sure how to put that host in the debug list, how do I do that? Console > settings > I dont see a debug host list.

[netniV]

First, what version of cacti are you running?

If it's 1.x you would find this under Configuration -> Settings -> General -> Log Settings

Selective Device Debug
A comma delimited list of Device ID's that you wish to be in Debug mode during data collection. This Debug level is only in place during the Cacti polling proces

[drewdown12765]

0.8.8a and I don't have that option.

I can upgrade if need be but hesitant as its been working fine up until this new host.

[netniV]

How many retries do you have it set to perform? Your number of hosts is less than mine, plus there doesn't appear to be much it needs to process so I'm surprised it shows the timeout.

[drewdown12765]

3 and yeah its odd because no problems with any of the other devices, just this one. Everything looks right but for whatever reason its not able to read/see the rrd files? I checked the permissions on them vs a working device and they are identical.

[root@cacti rra]# ls -lah r2-xxxx-idef_traffic_in_2701.rrd
-rw-r--r-- 1 root root 185K Jan 10 10:06 r2-2951-in_traffic_in_2701.rrd
[root@cacti rra]# ls -lah r2-xxxx-abc_traffic_in_2939.rrd
-rw-r--r-- 1 root root 185K Jan 8 17:40 r2-2951-blr_traffic_in_2939.rrd
[root@cacti rra]# df -h


Why would those RRD files be from 2 days ago? Does that mean its not getting updates or its hung? If so can I just restart the poller or cacti itself?

[netniV]

I don't think it's a permission problem with the RRD files. I think it literally hasn't had any data back (at thinks it hasn't) so hasn't populated anything. Since you don't have device debugging in that version of cacti, the only thing you can do is try upping the logging level and see if anything shows up that might be the issue.

[drewdown12765]

I set Console > Cacti Settings > Poller Log Settings to DEBUG.

Still not sure what those messages mean and how to fix them? At this point I may just reboot this thing because I have no idea what to even try at this point.

Also Data source debug shows this:

Data Source Debug
/usr/bin/rrdtool create \
/var/www/html/rra/r2-xxxx-xxx_traffic_in_2940.rrd \
--step 60 \
DS:traffic_in:COUNTER:120:0:100000000 \
DS:traffic_out:COUNTER:120:0:100000000 \
RRA:AVERAGE:0.5:1:600 \
RRA:AVERAGE:0.5:6:700 \
RRA:AVERAGE:0.5:24:775 \
RRA:AVERAGE:0.5:288:797 \
RRA:MIN:0.5:1:600 \
RRA:MIN:0.5:6:700 \
RRA:MIN:0.5:24:775 \
RRA:MIN:0.5:288:797 \
RRA:MAX:0.5:1:600 \
RRA:MAX:0.5:6:700 \
RRA:MAX:0.5:24:775 \
RRA:MAX:0.5:288:797 \
RRA:LAST:0.5:1:600 \
RRA:LAST:0.5:6:700 \
RRA:LAST:0.5:24:775 \
RRA:LAST:0.5:288:797 \

[netniV]

does /var/www/html/rra/r2-xxxx-xxx_traffic_in_2940.rrd exist?

I suspect not. Make sure that the rrd and cache folder are owned by the web server group (usually www-data or apache)

Code: Select all

chown -R www-data:www-data rra cache

[drewdown12765]

Yes.

ls -lah r2-xxxx-xxx_traffic_in_2940.rrd
-rw-r--r-- 1 root root 185K Jan 8 17:40 r2-2951-blr_traffic_in_2940.rrd

[netniV]

Try setting the permissions above then. I'll login and double mine.

[netniV]

Yeah, mine were set such that:

Code: Select all

-rw-rw-r--  1 www-data www-data    170 Dec  1 08:35 .htaccess
-rw-rw-r--  1 www-data www-data 749520 Jan 11 14:30 localhost_usedreal_8.rrd
-rw-rw-r--  1 www-data www-data 282440 Dec 29 23:00 local_linux_machine_load_1min_2.rrd
-rw-rw-r--  1 www-data www-data  95608 Dec 29 23:00 local_linux_machine_mem_buffers_4.rrd
-rw-rw-r--  1 www-data www-data  95608 Dec 29 23:00 local_linux_machine_mem_swap_5.rrd
-rw-rw-r--  1 www-data www-data  95608 Dec 29 23:00 local_linux_machine_ping_6.rrd
-rw-rw-r--  1 www-data www-data  95608 Dec 29 23:00 local_linux_machine_proc_1.rrd
-rw-rw-r--  1 www-data www-data  95608 Dec 29 23:00 local_linux_machine_users_3.rrd

Same applies if you are using the structured layout format.

[drewdown12765]

Not sure I follow? And not sure it matters anymore because I deleted the device and all the associated rra files and re-added it. Graphs are now populating as they should. I can only assume something broke and deleting and re-creating the host fixed it.

Wait but what if data source references a non-existent rrd file? I see a couple of those after adding the device back.

Thanks for your help.

[netniV]

If the data source is in use, try going into the datasource and saving it.