Gaps in graph

[crinago]

Hi,

I have gaps in my graphs. I am using cacti 0.8.6h, with plugins installed. I use Fedora 3, net-snmp 5.2.2.
I have these in my cacti.log:

03/02/2006 12:27:51 PM - RECACHE STATS: RecacheTime:67.5848 HostsRecached:2
03/02/2006 12:26:51 PM - PCOMMAND: Poller[0] Host[156] WARNING: Recache Event Detected for Host
03/02/2006 12:26:43 PM - THOLD: Poller[0] Checking Thresholds
03/02/2006 12:26:43 PM - PCOMMAND: Poller[0] Host[101] WARNING: Recache Event Detected for Host
03/02/2006 12:26:42 PM - SYSTEM STATS: Time:101.2322 Method:cmd.php Processes:1 Threads:N/A Hosts:78 HostsPerProcess:78 DataSources:227 RRDsProcessed:68
03/02/2006 12:26:39 PM - CMDPHP: Poller[0] ASSERT: '469320<38774' failed. Recaching host '172.16.176.1', data query #1
03/02/2006 12:26:19 PM - CMDPHP: Poller[0] Host[105] ERROR: HOST EVENT: Host is DOWN Message: Host did not respond to SNMP
03/02/2006 12:25:53 PM - CMDPHP: Poller[0] ASSERT: '17362364<' failed. Recaching host '172.16.83.1', data query #1
03/02/2006 12:22:17 PM - RECACHE STATS: RecacheTime:55.5538 HostsRecached:1
03/02/2006 12:21:21 PM - THOLD: Poller[0] Checking Thresholds
03/02/2006 12:21:21 PM - PCOMMAND: Poller[0] Host[101] WARNING: Recache Event Detected for Host
03/02/2006 12:21:21 PM - SYSTEM STATS: Time:79.6754 Method:cmd.php Processes:1 Threads:N/A Hosts:78 HostsPerProcess:78 DataSources:227 RRDsProcessed:78
03/02/2006 12:20:51 PM - CMDPHP: Poller[0] Host[101] DS[470] WARNING: Result from SNMP not valid. Partial Result:
03/02/2006 12:20:44 PM - CMDPHP: Poller[0] ASSERT: '<' failed. Recaching host '172.16.83.1', data query #1
03/02/2006 12:20:27 PM - CMDPHP: Poller[0] Host[88] ERROR: HOST EVENT: Host is DOWN Message: Host did not respond to SNMP
03/02/2006 12:16:13 PM - RECACHE STATS: RecacheTime:8.2930 HostsRecached:1
03/02/2006 12:16:05 PM - THOLD: Poller[0] Checking Thresholds
03/02/2006 12:16:05 PM - PCOMMAND: Poller[0] Host[101] WARNING: Recache Event Detected for Host
03/02/2006 12:16:05 PM - SYSTEM STATS: Time:63.5265 Method:cmd.php Processes:1 Threads:N/A Hosts:78 HostsPerProcess:78 DataSources:227 RRDsProcessed:53
03/02/2006 12:15:48 PM - CMDPHP: Poller[0] Host[105] NOTICE: HOST EVENT: Host Returned from DOWN State:
03/02/2006 12:15:47 PM - CMDPHP: Poller[0] Host[101] DS[470] WARNING: Result from SNMP not valid. Partial Result:
03/02/2006 12:15:43 PM - CMDPHP: Poller[0] Host[101] DS[470] WARNING: Result from SNMP not valid. Partial Result:
03/02/2006 12:15:39 PM - CMDPHP: Poller[0] ASSERT: '17292390<' failed. Recaching host '172.16.83.1', data query #1
03/02/2006 12:11:15 PM - THOLD: Poller[0] Checking Thresholds
03/02/2006 12:11:15 PM - SYSTEM STATS: Time:73.6714 Method:cmd.php Processes:1 Threads:N/A Hosts:78 HostsPerProcess:78 DataSources:227 RRDsProcessed:83
03/02/2006 12:11:01 PM - CMDPHP: Poller[0] Host[120] DS[577] WARNING: Result from SNMP not valid. Partial Result:
03/02/2006 12:06:40 PM - RECACHE STATS: RecacheTime:5.0227 HostsRecached:1
03/02/2006 12:06:35 PM - PCOMMAND: Poller[0] Host[97] WARNING: Recache Event Detected for Host
03/02/2006 12:06:35 PM - THOLD: Poller[0] Checking Thresholds
03/02/2006 12:06:35 PM - SYSTEM STATS: Time:93.6917 Method:cmd.php Processes:1 Threads:N/A Hosts:78 HostsPerProcess:78 DataSources:227 RRDsProcessed:63
03/02/2006 12:06:06 PM - CMDPHP: Poller[0] Host[101] DS[470] WARNING: Result from SNMP not valid. Partial Result:
03/02/2006 12:06:02 PM - CMDPHP: Poller[0] Host[101] DS[470] WARNING: Result from SNMP not valid. Partial Result:
03/02/2006 12:05:44 PM - CMDPHP: Poller[0] ASSERT: '127222546<' failed. Recaching host '172.16.79.1', data query #1
03/02/2006 12:02:31 PM - RECACHE STATS: RecacheTime:64.8696 HostsRecached:1
03/02/2006 12:01:26 PM - THOLD: Poller[0] Checking Thresholds
03/02/2006 12:01:26 PM - PCOMMAND: Poller[0] Host[101] WARNING: Recache Event Detected for Host
03/02/2006 12:01:26 PM - SYSTEM STATS: Time:84.7010 Method:cmd.php Processes:1

[crinago]

I've dropped my cacti table and started a new one. I am now left with only 3 hosts. I have this on my cacti log in DEBUG mode. How come I am still having gaps in my graphs??? Please help....

[root@cbcnoc docs]# /usr/local/php5/bin/php /var/www/html/cacti/poller.php
03/02/2006 07:54:16 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: /usr/local/php5/bin/php, ARGS: -q /var/www/html/cacti/cmd.php 0 1]
03/02/2006 07:54:17 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: /usr/local/php5/bin/php, ARGS: -q /var/www/html/cacti/cmd.php 2 3]
Waiting on 1/2 pollers.
03/02/2006 07:54:18 PM - POLLER: Poller[0] CACTI2RRD: /usr/local/rrdtool/bin/rrdtool update /var/www/html/cacti/rra/cisco_7513_traffic_in_9.rrd --template traffic_in:traffic_out 1141300457:2022184903:2865533772
03/02/2006 07:54:18 PM - POLLER: Poller[0] CACTI2RRD: /usr/local/rrdtool/bin/rrdtool update /var/www/html/cacti/rra/cisco_7513_traffic_in_8.rrd --template traffic_out:traffic_in 1141300457:3630018684:3417765541
03/02/2006 07:54:18 PM - POLLER: Poller[0] CACTI2RRD: /usr/local/rrdtool/bin/rrdtool update /var/www/html/cacti/rra/cisco_7513_traffic_in_10.rrd --template traffic_in:traffic_out 1141300457:1405346278:2310403761
03/02/2006 07:54:18 PM - POLLER: Poller[0] CACTI2RRD: /usr/local/rrdtool/bin/rrdtool update /var/www/html/cacti/rra/cisco_7513_traffic_in_11.rrd --template traffic_in:traffic_out 1141300457:1735410485:1145109845
03/02/2006 07:54:18 PM - POLLER: Poller[0] CACTI2RRD: /usr/local/rrdtool/bin/rrdtool update /var/www/html/cacti/rra/cisco_7513_traffic_in_12.rrd --template traffic_in:traffic_out 1141300457:1964171968:434644117
OK u:0.00 s:0.00 r:0.05
OK u:0.00 s:0.00 r:0.05
OK u:0.00 s:0.00 r:0.05
OK u:0.00 s:0.00 r:0.06
OK u:0.00 s:0.00 r:0.06
Waiting on 1/2 pollers.
03/02/2006 07:54:19 PM - POLLER: Poller[0] CACTI2RRD: /usr/local/rrdtool/bin/rrdtool update /var/www/html/cacti/rra/sc3020_traffic_in_19.rrd --template traffic_in:traffic_out 1141300457:2843977470:1269583113
03/02/2006 07:54:19 PM - POLLER: Poller[0] CACTI2RRD: /usr/local/rrdtool/bin/rrdtool update /var/www/html/cacti/rra/sc3020_traffic_in_18.rrd --template traffic_out:traffic_in 1141300457:538813785:4164218613
03/02/2006 07:54:19 PM - POLLER: Poller[0] CACTI2RRD: /usr/local/rrdtool/bin/rrdtool update /var/www/html/cacti/rra/sc3020_traffic_in_17.rrd --template traffic_out:traffic_in 1141300457:2931417531:692471539
03/02/2006 07:54:19 PM - POLLER: Poller[0] CACTI2RRD: /usr/local/rrdtool/bin/rrdtool update /var/www/html/cacti/rra/sc3020_traffic_in_16.rrd --template traffic_out:traffic_in 1141300457:2842926974:2610003569
03/02/2006 07:54:19 PM - POLLER: Poller[0] CACTI2RRD: /usr/local/rrdtool/bin/rrdtool update /var/www/html/cacti/rra/sc3020_traffic_in_15.rrd --template traffic_out:traffic_in 1141300457:643491303:1311640411
OK u:0.00 s:0.00 r:1.10
03/02/2006 07:54:19 PM - POLLER: Poller[0] CACTI2RRD: /usr/local/rrdtool/bin/rrdtool update /var/www/html/cacti/rra/cisco_7513_traffic_in_13.rrd --template traffic_out:traffic_in 1141300457:1288574083:3203714628
03/02/2006 07:54:19 PM - POLLER: Poller[0] CACTI2RRD: /usr/local/rrdtool/bin/rrdtool update /var/www/html/cacti/rra/sc3020_traffic_in_20.rrd --template traffic_in:traffic_out 1141300457:2980330522:3207906534
OK u:0.00 s:0.00 r:1.11
OK u:0.00 s:0.00 r:1.11
OK u:0.00 s:0.00 r:1.11
OK u:0.00 s:0.00 r:1.12
OK u:0.00 s:0.00 r:1.12
OK u:0.00 s:0.00 r:1.12
Waiting on 1/2 pollers.
03/02/2006 07:54:21 PM - POLLER: Poller[0] CACTI2RRD: /usr/local/rrdtool/bin/rrdtool update /var/www/html/cacti/rra/sc3020_traffic_in_34.rrd --template traffic_out:traffic_in 1141300457:51545943:110957597
03/02/2006 07:54:21 PM - POLLER: Poller[0] CACTI2RRD: /usr/local/rrdtool/bin/rrdtool update /var/www/html/cacti/rra/sc3020_traffic_in_33.rrd --template traffic_out:traffic_in 1141300457:1861703846:1976576540
03/02/2006 07:54:21 PM - POLLER: Poller[0] CACTI2RRD: /usr/local/rrdtool/bin/rrdtool update /var/www/html/cacti/rra/sc3020_traffic_in_32.rrd --template traffic_out:traffic_in 1141300457:1623060530:803009121
03/02/2006 07:54:21 PM - POLLER: Poller[0] CACTI2RRD: /usr/local/rrdtool/bin/rrdtool update /var/www/html/cacti/rra/sc3020_traffic_in_31.rrd --template traffic_out:traffic_in 1141300457:1233448397:653379123
03/02/2006 07:54:21 PM - POLLER: Poller[0] CACTI2RRD: /usr/local/rrdtool/bin/rrdtool update /var/www/html/cacti/rra/sc3020_traffic_in_30.rrd --template traffic_out:traffic_in 1141300457:561663515:2325656329
OK u:0.00 s:0.00 r:3.17
03/02/2006 07:54:21 PM - POLLER: Poller[0] CACTI2RRD: /usr/local/rrdtool/bin/rrdtool update /var/www/html/cacti/rra/sc3020_traffic_in_29.rrd --template traffic_out:traffic_in 1141300457:3204445663:571146817
03/02/2006 07:54:21 PM - SYSTEM STATS: Time:5.3072 Method:cmd.php Processes:2 Threads:N/A Hosts:4 HostsPerProcess:2 DataSources:57 RRDsProcessed:18
OK u:0.00 s:0.01 r:3.18
OK u:0.00 s:0.01 r:3.19
OK u:0.00 s:0.01 r:3.19
OK u:0.00 s:0.01 r:3.19
OK u:0.00 s:0.01 r:3.19
03/02/2006 07:54:21 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: /usr/local/php5/bin/php, ARGS: -q /var/www/html/cacti/plugins/thold/check-thold.php]
[root@cbcnoc docs]#

[gandalf]

YOu may have problems with

03/02/2006 12:15:48 PM - CMDPHP: Poller[0] Host[105] NOTICE: HOST EVENT: Host Returned from DOWN State:

due to Settings -> Poller -> Downed Host Detection. Please set this to SNMP - Reliable if using pre - cacti-0.8.6h
Downed Hosts are not polled
Reinhard

[crinago]

Hi,

Tnx for answering. I actually use cacti 0.8.6h and Downed host detection is set to SNMP-Reliable. But my graphs are still having gaps as you can see below. One thing I've noticed thought is that everytime I access the view SNMP cache the server CPU load would shoot up to 100% and everything freezes. I am using Pentium 4 with 512 MB, Fedora 3. I use Net-SNMP 5.2.2. compiled from source . My poller is cmd.php as I am pooling yet only 3 hosts. Hope this helps.

[gandalf]

What amount of memory configured in php.ini? Should be about 64 MB
Reinhard

[crinago]

Ok I've adjusted the memory from 8MB to 64MB. it doesn't seem to work. My graphs are still in gaps. My cacti.log doesn't display any error even in debug mode. I don't know what's wrong.

[egironda]

I changed the timeouts in the php.ini (which are options configured right next to the memory configuration, easy to find) and that seemed to help with my gaps so far. I don't know php well enough to know what the negative repercussions of that may be, though, so atcher ownrisk.

This is what those lines look like for me in php.ini:

Code: Select all

max_execution_time = 120     ; Maximum execution time of each script, in seconds
max_input_time = 120 ; Maximum amount of time each script may spend parsing request data
memory_limit = 128M      ; Increased to 128 - eg 20051223

[cybex_77]

I am having a similar problem with gaps in all my graphs I have changed the max mem to 128Mb and the max execution time to 120 but this does not seem to work. I am running cacti on RH9 and using cacti-0.8.6-1. I have aslo included one of my graphs and an extract of my log file.

I am getting lots of Warnings SNMP not valid in my log file.

Can anyone help?

Log File Extract:

03/12/2006 05:00:03 PM - CMDPHP: Poller[0] ASSERT: '15921<15888' failed. Recaching host '192.200.10.30', data query #9.
03/12/2006 05:00:04 PM - CMDPHP: Poller[0] Host[16] WARNING: Result from SNMP not valid. Partial Result:
03/12/2006 05:00:04 PM - CMDPHP: Poller[0] Host[16] WARNING: Result from SNMP not valid. Partial Result:
03/12/2006 05:00:04 PM - CMDPHP: Poller[0] Host[18] WARNING: Result from SNMP not valid. Partial Result:
03/12/2006 05:00:05 PM - CMDPHP: Poller[0] Host[20] WARNING: Result from SNMP not valid. Partial Result:
03/12/2006 05:00:08 PM - CMDPHP: Poller[0] Host[24] WARNING: Result from SNMP not valid. Partial Result:
03/12/2006 05:00:08 PM - CMDPHP: Poller[0] Host[24] WARNING: Result from SNMP not valid. Partial Result:
03/12/2006 05:00:10 PM - CMDPHP: Poller[0] ASSERT: '20634000<20634000' failed. Recaching host '217.40.124.121', data query #1.
03/12/2006 05:00:13 PM - CMDPHP: Poller[0] ASSERT: '630053400<630053400' failed. Recaching host '217.46.231.193', data query #1.
03/12/2006 05:04:57 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
03/12/2006 05:05:00 PM - CMDPHP: Poller[0] Host[6] WARNING: Result from SNMP not valid. Partial Result:
03/12/2006 05:05:00 PM - CMDPHP: Poller[0] Host[6] WARNING: Result from SNMP not valid. Partial Result:
03/12/2006 05:05:00 PM - CMDPHP: Poller[0] Host[6] WARNING: Result from SNMP not valid. Partial Result:
03/12/2006 05:05:00 PM - CMDPHP: Poller[0] Host[6] WARNING: Result from SNMP not valid. Partial Result:
03/12/2006 05:05:02 PM - CMDPHP: Poller[0] Host[9] WARNING: Result from SNMP not valid. Partial Result:
03/12/2006 05:05:03 PM - CMDPHP: Poller[0] Host[9] WARNING: Result from SNMP not valid. Partial Result:
03/12/2006 05:05:03 PM - CMDPHP: Poller[0] ASSERT: '429496700<429496700' failed. Recaching host '89.129.19.18', data query #1.
03/12/2006 05:05:03 PM - CMDPHP: Poller[0] ASSERT: '15894<15850' failed. Recaching host '192.200.10.30', data query #1.
03/12/2006 05:05:03 PM - CMDPHP: Poller[0] ASSERT: '15894<15850' failed. Recaching host '192.200.10.30', data query #9.
03/12/2006 05:05:03 PM - CMDPHP: Poller[0] Host[16] WARNING: Result from SNMP not valid. Partial Result:
03/12/2006 05:05:03 PM - CMDPHP: Poller[0] Host[16] WARNING: Result from SNMP not valid. Partial Result:
03/12/2006 05:05:03 PM - CMDPHP: Poller[0] Host[18] WARNING: Result from SNMP not valid. Partial Result:
03/12/2006 05:05:04 PM - CMDPHP: Poller[0] Host[20] WARNING: Result from SNMP not valid. Partial Result:
03/12/2006 05:05:06 PM - CMDPHP: Poller[0] Host[24] WARNING: Result from SNMP not valid. Partial Result:
03/12/2006 05:05:06 PM - CMDPHP: Poller[0] Host[24] WARNING: Result from SNMP not valid. Partial Result:
03/12/2006 05:09:57 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
03/12/2006 05:10:02 PM - CMDPHP: Poller[0] Host[6] WARNING: Result from SNMP not valid. Partial Result:
03/12/2006 05:10:02 PM - CMDPHP: Poller[0] Host[6] WARNING: Result from SNMP not valid. Partial Result:
03/12/2006 05:10:02 PM - CMDPHP: Poller[0] Host[6] WARNING: Result from SNMP not valid. Partial Result:
03/12/2006 05:10:02 PM - CMDPHP: Poller[0] Host[6] WARNING: Result from SNMP not valid. Partial Result:
03/12/2006 05:10:04 PM - CMDPHP: Poller[0] Host[9] WARNING: Result from SNMP not valid. Partial Result:
03/12/2006 05:10:04 PM - CMDPHP: Poller[0] Host[9] WARNING: Result from SNMP not valid. Partial Result:

[gandalf]

You should switch Settings->Logging Level to DEBUG for one polling cycle. Then have a look at log/cacti.log. This should tell you about the failing OIDs for that host. You may poll them manually with snmpwalk from cli to verify
Reinhard

[cybex_77]

Firstly thanks for your help on the last issue I managed to resolve the IPSec graphs.

I have had cacti in debug mode for the duration of the weekend and have therefore attached a segment of the logs for you to have a look at. I can not see what is causing this issue as all the graphs are affected and when I complete an snmpwalk on that OID it also appears to work. There a few problems with remote DSL services but only on a few of our remote sites.

Thanks in advance.

[cybex_77]

Could this have anything to do with the poller.php set to run every five mins in the crontab and it is running again before it has time to finish from the last time?

Thanks

[gandalf]

Yes, this may cause this issue. Check SYSTEM STATS entries on the log. If polling time exceeds 300 sec = gaps
Reinhard

[R2D2]

Yes, this may cause this issue. Check SYSTEM STATS entries on the log. If polling time exceeds 300 sec = gaps
Reinhard

Hi Reinhard

I'm experiencing the same problem and my polling time exceeds 300 secs, do I need to adjust anything?

R2D2

[gandalf]

There is more than one possible cause for this problem. This depends on
- how many hosts/datasources/rrds are you running (see SYSTEMS STATS from cacti.log)
- what poller are you running (cmd.php versus cactid)
- are you using expensive scripts?
- is there more than one crontab entry for the poller. You'll have to check at least /etc/crontab, /etc/cron.d/cacti, crontab of user root and cactiuser

Settings -> Logging Level = DEBUG will help to get an idea
Reinhard