[SOLVED] Hole in graph

[albertinho]

Hi all,

I have some problems with holes in my cacti graphs like image attached.

graph_hole.gif (15.59 KiB) Viewed 1546 times

My cacti infos:
Cacti Version 0.8.7g
Cacti OS unix
SNMP Version NET-SNMP version: 5.3.2.2
RRDTool Version RRDTool 1.3.x
Hosts 256
Graphs 2031
Data Sources Script/Command: 3
SNMP: 948
SNMP Query: 1312
Script Query: 2
Script Query - Script Server: 24
Total: 2289
Poller Information
Interval 60
Type spine
Items Action[0]: 3008
Action[1]: 7
Action[2]: 44
Total: 3059
Concurrent Processes 4
Max Threads 20
PHP Servers 2
Script Timeout 25
Max OID 10
Last Run Statistics Time:6.1668 Method:spine Processes:4 Threads:20 Hosts:249 HostsPerProcess:63 DataSources:3054 RRDsProcessed:2167
PHP Information
PHP Version 5.1.6
PHP OS Linux
PHP uname Linux cacti-ced 2.6.18-238.12.1.el5 #1 SMP Sat May 7 20:18:50 EDT 2011 x86_64
PHP SNMP Installed
max_execution_time 30
memory_limit 128M

In cacti.log I see this warning:

08/23/2011 05:31:02 AM - POLLER: Poller[0] WARNING: Cron is out of sync with the Poller Interval! The Poller Interval is '60' seconds, with a maximum of a '300' second Cron, but 120 seconds have passed since the last poll!

I try to use differents "Threads" (from 10 to 20) without results.
Any idea?

Thanks

[tscoltrane]

In Settings->Poller, you should ensure that the poller_interval and cron_interval match. I don't quite understand why they're separate. Anyway, that MIGHT be your problem. I didn't see the heartbeat value in you RRA. You've set yourself up for eventual failure, here. Eventually, the poller will take too long to respond (probably 300 seconds) -- to ensure that the RRA is updated within the heartbeat. When that happens, a hole will appear.

Of course, over-wraps and other things can cause you problems. I'll BET that if you set both intervals to the same value (i.e., to the value of your cronjob -- most likely 5 minutes (300 seconds)), the problem will disappear. Out of courtesy, I'd like to know the result.

... and there's really hardly ever a legitimate need to be so obsessive that you have to poll every minute. These graphs are just pretty pictures... and give you an overview. If you need to ALARM/monitor more frequently, that's totally legitimate... and not done with Cacti (usually -- perhaps there's a way).

[albertinho]

In Settings->Poller, you should ensure that the poller_interval and cron_interval match. I don't quite understand why they're separate. Anyway, that MIGHT be your problem. I didn't see the heartbeat value in you RRA. You've set yourself up for eventual failure, here. Eventually, the poller will take too long to respond (probably 300 seconds) -- to ensure that the RRA is updated within the heartbeat. When that happens, a hole will appear.
Of course, over-wraps and other things can cause you problems. I'll BET that if you set both intervals to the same value (i.e., to the value of your cronjob -- most likely 5 minutes (300 seconds)), the problem will disappear. Out of courtesy, I'd like to know the result.

Poller and cron interval have already the same value (1min)

... and there's really hardly ever a legitimate need to be so obsessive that you have to poll every minute. These graphs are just pretty pictures... and give you an overview. If you need to ALARM/monitor more frequently, that's totally legitimate... and not done with Cacti (usually -- perhaps there's a way).

I think that Cacti is more that just a "pretty pictures maker"

So, an update. The Linux accounts in cacti machine are authenticated with our AD.
In these days we have some problems with LDAP session and hole appears. After we resolved LDAP problems (yestarday) I don't see holes.
cacti user is a "local user" (as DB or apache users) so I don't understand correlation with LDAP problems (if confirmed).

I'll update you in next days.

[tscoltrane]

Well... that would be an "odd one," to me. It's my understanding that authentication should not be required for the poller process. In fact you can [and should] have a local account that runs/owns cacti AND does not have a password (thus, entry is barred), does not have a valid login shell (thus, login should be impossible -- provided that you're using a reasonably mainstream/normal/non-heavily-modified Linux/FreeBSD system), and has an invalid home directory (again, depending on your system, a barrier to login).

I actually run the poller from the root crontab with a sudo to my cacti_user. Authentication just should not be relevant.

You should search through your cacti log to see how long it takes to run (i.e., how long it's taking to poll all of your "stuff"). If it exceeds 120 seconds, then that might be your problem. I just can't see how authentication is relevant, here. Look for lines like this one:

Code: Select all

08/24/2011 11:40:10 AM - SYSTEM STATS: Time:9.8623 Method:spine Processes:4 Threads:8 Hosts:44 HostsPerProcess:11 DataSources:988 RRDsProcessed:793 

You're looking for the "Time" value (in my case 9.8623 seconds).

[albertinho]

Now situation is normal.

My cacti log:

08/25/2011 09:20:08 AM - SYSTEM STATS: Time:6.0518 Method:spine Processes:4 Threads:15 Hosts:249 HostsPerProcess:63 DataSources:3007 RRDsProcessed:2123
08/25/2011 09:19:07 AM - SYSTEM STATS: Time:5.9338 Method:spine Processes:4 Threads:15 Hosts:249 HostsPerProcess:63 DataSources:3007 RRDsProcessed:2123
08/25/2011 09:18:06 AM - SYSTEM STATS: Time:4.8246 Method:spine Processes:4 Threads:15 Hosts:249 HostsPerProcess:63 DataSources:3008 RRDsProcessed:2123
08/25/2011 09:17:07 AM - SYSTEM STATS: Time:5.4701 Method:spine Processes:4 Threads:15 Hosts:249 HostsPerProcess:63 DataSources:3008 RRDsProcessed:2123

are good?

[tscoltrane]

Well, the poller times look good. Were you using an account in some script to pull a metric (i.e., an account on the remote machine)? It was really just a change in authentication? Weird! Regardless, those polling times are quite good. They shouldn't stop you from getting data before the heartbeat expires. How did you fix your problem?

[tscoltrane]

FYI -- I just saw this: http://forums.cacti.net/viewtopic.php?f=14&t=43859 . That user is describing what I imagined to be your problem. Compare his poller output to yours. You're winning!

[albertinho]

After resolved LDAP <=> AD authentication no more holes in my cacti graphs.