Graphs not right...

myusername · Post by **myusername** » Tue Jul 22, 2008 3:03 am

Hi,

please check out one of my graphs below. It has 'holes' in it...

This just happened a couple of days ago when I added some more graphs.

Currently I have 1127 Graps on 67 devices. I've set the poller interval to 5 minutes and the cron interval also on every 5 minutes.

The maximum concurrent poller processes are set to 5.

Can anyone please advise me what could be wrong?

Post by **gandalf** » Mon Aug 11, 2008 12:20 pm

What's the current runtime for your poller process? See "SYSTEM STATS" in cacti.log
Reinhard

elpiako · Post by **elpiako** » Tue Aug 12, 2008 5:14 am

I've got the same problem.

My Server1 meets no problem, here is its poller process :

08/12/2008 12:16:30 PM - EXPORT STATS: ExportDate:2008-08-12_12:16:30 ExportDuration:11.6148 TotalGraphsExported:245
08/12/2008 12:16:17 PM - SYSTEM STATS: Time:76.4729 Method:cactid Processes:10 Threads:10 Hosts:174 HostsPerProcess:18 DataSources:3331 RRDsProcessed:2217
08/12/2008 12:11:29 PM - EXPORT STATS: ExportDate:2008-08-12_12:11:29 ExportDuration:10.8521 TotalGraphsExported:245
08/12/2008 12:11:17 PM - SYSTEM STATS: Time:75.9747 Method:cactid Processes:10 Threads:10 Hosts:174 HostsPerProcess:18 DataSources:3331 RRDsProcessed:2236
08/12/2008 12:06:32 PM - EXPORT STATS: ExportDate:2008-08-12_12:06:32 ExportDuration:11.9326 TotalGraphsExported:245
08/12/2008 12:06:19 PM - SYSTEM STATS: Time:77.6444 Method:cactid Processes:10 Threads:10 Hosts:174 HostsPerProcess:18 DataSources:3331 RRDsProcessed:2217
08/12/2008 12:01:32 PM - EXPORT STATS: ExportDate:2008-08-12_12:01:32 ExportDuration:12.0403 TotalGraphsExported:245
08/12/2008 12:01:18 PM - SYSTEM STATS: Time:77.4130 Method:cactid Processes:10 Threads:10 Hosts:174 HostsPerProcess:18 DataSources:3331 RRDsProcessed:2217
08/12/2008 11:56:30 AM - EXPORT STATS: ExportDate:2008-08-12_11:56:30 ExportDuration:11.9068 TotalGraphsExported:245
08/12/2008 11:56:16 AM - SYSTEM STATS: Time:74.8992 Method:cactid Processes:10 Threads:10 Hosts:174 HostsPerProcess:18 DataSources:3331 RRDsProcessed:2236

I've got many holes on many graphs on my Server2, here is its poller process :

08/12/2008 12:10:16 PM - EXPORT STATS: ExportDate:2008-08-12_12:10:16 ExportDuration:0.3677 TotalGraphsExported:0
08/12/2008 12:10:15 PM - SYSTEM STATS: Time:13.8367 Method:spine Processes:10 Threads:15 Hosts:145 HostsPerProcess:15 DataSources:2570 RRDsProcessed:1857
08/12/2008 12:05:16 PM - EXPORT STATS: ExportDate:2008-08-12_12:05:16 ExportDuration:0.4195 TotalGraphsExported:0
08/12/2008 12:05:15 PM - SYSTEM STATS: Time:13.3618 Method:spine Processes:10 Threads:15 Hosts:145 HostsPerProcess:15 DataSources:2570 RRDsProcessed:1857
08/12/2008 12:05:10 PM - SYSTEM MACTRACK STATS: Time:10.3183 ConcurrentProcesses:7 Devices:2
08/12/2008 12:05:00 PM - EXPORT STATS: ExportDate:2008-08-12_12:05:00 ExportDuration:0.4263 TotalGraphsExported:0
08/12/2008 12:04:59 PM - SYSTEM STATS: Time:298.8177 Method:spine Processes:10 Threads:15 Hosts:145 HostsPerProcess:15 DataSources:2570 RRDsProcessed:1222
08/12/2008 12:00:01 PM - EXPORT STATS: ExportDate:2008-08-12_12:00:01 ExportDuration:0.6703 TotalGraphsExported:0
08/12/2008 12:00:00 PM - SYSTEM STATS: Time:298.6695 Method:spine Processes:10 Threads:15 Hosts:145 HostsPerProcess:15 DataSources:2570 RRDsProcessed:1221
08/12/2008 11:50:16 AM - EXPORT STATS: ExportDate:2008-08-12_11:50:16 ExportDuration:0.4154 TotalGraphsExported:0
08/12/2008 11:50:16 AM - SYSTEM STATS: Time:15.0499 Method:spine Processes:10 Threads:15 Hosts:145 HostsPerProcess:15 DataSources:2570 RRDsProcessed:1857
08/12/2008 11:50:11 AM - SYSTEM MACTRACK STATS: Time:10.3317 ConcurrentProcesses:7 Devices:2
08/12/2008 11:50:00 AM - EXPORT STATS: ExportDate:2008-08-12_11:50:00 ExportDuration:0.4264 TotalGraphsExported:0
08/12/2008 11:50:00 AM - SYSTEM STATS: Time:298.8218 Method:spine Processes:10 Threads:15 Hosts:145 HostsPerProcess:15 DataSources:2570 RRDsProcessed:1620
08/12/2008 11:40:17 AM - EXPORT STATS: ExportDate:2008-08-12_11:40:17 ExportDuration:0.4153 TotalGraphsExported:0
08/12/2008 11:40:17 AM - SYSTEM STATS: Time:14.8279 Method:spine Processes:10 Threads:15 Hosts:145 HostsPerProcess:15 DataSources:2570 RRDsProcessed:1857
08/12/2008 11:35:18 AM - EXPORT STATS: ExportDate:2008-08-12_11:35:18 ExportDuration:0.4128 TotalGraphsExported:0
08/12/2008 11:35:17 AM - SYSTEM STATS: Time:14.7168 Method:spine Processes:10 Threads:15 Hosts:145 HostsPerProcess:15 DataSources:2570 RRDsProcessed:1857
08/12/2008 11:35:13 AM - SYSTEM MACTRACK STATS: Time:10.8315 ConcurrentProcesses:7 Devices:2
08/12/2008 11:35:02 AM - EXPORT STATS: ExportDate:2008-08-12_11:35:02 ExportDuration:0.7096 TotalGraphsExported:0
08/12/2008 11:35:01 AM - SYSTEM STATS: Time:299.4512 Method:spine Processes:10 Threads:15 Hosts:145 HostsPerProcess:15 DataSources:2570 RRDsProcessed:1359
08/12/2008 11:30:03 AM - EXPORT STATS: ExportDate:2008-08-12_11:30:03 ExportDuration:0.5349 TotalGraphsExported:0
08/12/2008 11:30:01 AM - SYSTEM STATS: Time:299.7684 Method:spine Processes:10 Threads:15 Hosts:145 HostsPerProcess:15 DataSources:2570 RRDsProcessed:1481
08/12/2008 11:20:26 AM - SYSTEM MACTRACK STATS: Time:10.0835 ConcurrentProcesses:7 Devices:2
08/12/2008 11:20:16 AM - EXPORT STATS: ExportDate:2008-08-12_11:20:16 ExportDuration:0.4924 TotalGraphsExported:0
08/12/2008 11:20:15 AM - SYSTEM STATS: Time:13.9478 Method:spine Processes:10 Threads:15 Hosts:145 HostsPerProcess:15 DataSources:2570 RRDsProcessed:1857
08/12/2008 11:15:15 AM - EXPORT STATS: ExportDate:2008-08-12_11:15:15 ExportDuration:0.3392 TotalGraphsExported:0
08/12/2008 11:15:15 AM - SYSTEM STATS: Time:13.0575 Method:spine Processes:10 Threads:15 Hosts:145 HostsPerProcess:15 DataSources:2570 RRDsProcessed:1857
08/12/2008 11:10:17 AM - EXPORT STATS: ExportDate:2008-08-12_11:10:17 ExportDuration:0.4462 TotalGraphsExported:0
08/12/2008 11:10:17 AM - SYSTEM STATS: Time:14.9096 Method:spine Processes:10 Threads:15 Hosts:145 HostsPerProcess:15 DataSources:2570 RRDsProcessed:1857
08/12/2008 11:05:26 AM - SYSTEM MACTRACK STATS: Time:10.0997 ConcurrentProcesses:7 Devices:2
08/12/2008 11:05:16 AM - EXPORT STATS: ExportDate:2008-08-12_11:05:16 ExportDuration:0.5101 TotalGraphsExported:0
08/12/2008 11:05:15 AM - SYSTEM STATS: Time:14.1968 Method:spine Processes:10 Threads:15 Hosts:145 HostsPerProcess:15 DataSources:2570 RRDsProcessed:1857
08/12/2008 11:00:16 AM - EXPORT STATS: ExportDate:2008-08-12_11:00:16 ExportDuration:0.3919 TotalGraphsExported:0
08/12/2008 11:00:15 AM - SYSTEM STATS: Time:13.8446 Method:spine Processes:10 Threads:15 Hosts:145 HostsPerProcess:15 DataSources:2570 RRDsProcessed:1857

They poll every 5 minutes, therefore we can notice that Server2 takes almost 5 minutes to poll at many times :
First, why do my cacti poll twice (at 11:35am, 11:50am, 12:05pm) where one of both polls takes 5 minutes.
Second, at 11:30am and 12:00pm, there is only one poll and it takes 5 minutes

What's the matter ?

Post by **gandalf** » Tue Aug 12, 2008 2:53 pm

You may suffer from the double poller issue mentioned at the 2nd link of my sig
Reinhard

elpiako · Post by **elpiako** » Wed Aug 13, 2008 2:30 am

All right, I gonna try your howto right now

Thanks.

elpiako · Post by **elpiako** » Wed Aug 13, 2008 4:38 am

Ok.
First, at the third check of your howto (launch manually "./cactid --verbosity=5 <id> <id>") : what is cactid ?? I don't have any such file on my server

As you can imagine, I also wasn't able to make this check.

Second, my problem is NOT that I don't have any value. I have some !
Therefore all tests you suggested in your howto have passed !!!
Here is the output (last 20 lines) of "rrdtool fetch <rrd file> AVERAGE" :

1218613200: nan
1218613500: nan
1218613800: nan
1218614100: nan
1218614400: nan
1218614700: nan
1218615000: nan
1218615300: nan
1218615600: nan
1218615900: nan
1218616200: nan
1218616500: nan
1218616800: nan
1218617100: 5.0700000000e+02
1218617400: 5.0700000000e+02
1218617700: nan
1218618000: nan
1218618300: nan
1218618600: 1.3000000000e+01
1218618900: 1.3000000000e+01
1218619200: nan
1218619500: nan

This is an rrd where I've got the most "holes" in the values but many others rrds have just few holes (therefore less "NAN" values).

Then, about the "two poller", i've checked cron : i don't have cacti cron in /etc/cron.d/

Post by **gandalf** » Wed Aug 13, 2008 2:56 pm

And did you verify, that rrd files MAX setting is not a problem?
Reinhard

elpiako · Post by **elpiako** » Thu Aug 14, 2008 3:14 am

Of course I did, I followed faithfully your howto

There are no others errors in Cacti log, except those 300 seconds polling and double pollers (or both)

elpiako · Post by **elpiako** » Tue Aug 19, 2008 3:46 am

Please help!! my server still polls twice and/or polls during 300 seconds, creating holes in graphs

I don't know where to make research and what I have to research/check...

elpiako · Post by **elpiako** » Thu Aug 21, 2008 4:59 am

Now my server2 takes ALWAYS more than 300 seconds and is therefore NEVER able to finish

Please help

mcutting · Post by **mcutting** » Mon Aug 25, 2008 3:43 am

Sounds very much like a bad data source - do you have a script running on this server for a datasource that either contains an error, or does not finish ?

Check the logs (no debug needed - medium is fine), for SEGFAULT or TERMINATING. I've also noticed that you are using CACTID. If you are using 0.8.7, then this really should be SPINE.

Hope this is of some use.

elpiako · Post by **elpiako** » Mon Aug 25, 2008 4:05 am

Thank you !!!

I had added around 20 devices where SarParse graphs coudn't work as long as I didn't installed what is needed on thoses devices.

And you helped me to find the problem !!
By checking the date source, i've found that many oid requests were taken too much time. They were waiting for 20 seconds (equal of the Script and Script Server Timeout Value) per request which means that many threads per processes were waiting too !! (and failed to reach the value of course)
By setting the timeout value from 20 to 2, the poller could ending

Thanks again

mcutting · Post by **mcutting** » Mon Aug 25, 2008 4:33 am

Glad I could help.

Graphs not right...

Graphs not right...

Who is online