Broken Graphs and Runtime-exceeded.

Pizzop · Post by **Pizzop** » Wed Oct 19, 2005 4:33 pm

Hi, i'm having a few problems with Cacti. I've installed RRDTool, mysql server, and cacti all via yum on a fedora core 2 box.

First, I'm having broken graphs, and I've applied all of the listed patches. The attached images should be more descriptive. BS1 is the host that cacti is running on. I have a dell powerconnect 3024 (Premier Switch2) that similarily seems to be leaving only slight broken lines on the graph.

Also, I'm having problems with the poller reaching its maximum runtime. Here's a quote from the logfile:

Code: Select all

10/19/2005 03:09:57 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
10/19/2005 03:14:58 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
10/19/2005 03:14:58 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
10/19/2005 03:15:02 PM - CMDPHP: Poller[0] ASSERT: '1802914400<1802914400' failed. Recaching host 'switch1.domain.com', data query #1
10/19/2005 03:19:58 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
10/19/2005 03:19:58 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
10/19/2005 03:20:02 PM - CMDPHP: Poller[0] ASSERT: '1802944100<1802944100' failed. Recaching host 'switch1.domain.com', data query #1

Any help is greatly appreciated.

Pizzop · Post by **Pizzop** » Thu Oct 20, 2005 4:28 pm

It seems that the runtime is exceeded only by my switches (Dell PowerConnect 3024/3324. Is this possibly a setting I should be changing on the switch (if possible), or changing the timeout listed in the device profile in cacti? I'm using cmd.php and I noticed the timeout within the file is set to 296 seconds, but I'm hesitant to altering that. Any suggestions?

emerillat · Post by **emerillat** » Fri Oct 21, 2005 8:55 am

I am experiencing the same issues here. Latest version of Cacti on Fedora Core 4 box with PHP, MySQL, and Net-SNMP all installed from rpm.

Post by **TheWitness** » Fri Oct 21, 2005 4:05 pm

Pizzap,

Very high load, what going on on your box? You might want to run "top".

TheWitness

emerillat · Post by **emerillat** » Mon Oct 24, 2005 6:53 am

Found another post that talked about the patches for the latest version, installed those and they seem to have taken care of my problem.

Pizzop · Post by **Pizzop** » Mon Oct 24, 2005 10:00 am

TheWitness wrote:Pizzap,

Very high load, what going on on your box? You might want to run "top".

TheWitness

At that time it was running a backup. You can tell when the backup starts, but I'm still seeing quite a few gaps in the graphs for not only the local machine but also other hosts after applying the necessary patches.
I've attached 2 images, the first graph is a cacti graph of the traffic the interfaces are seeing on a different host. The 2nd is the cacti graph of the local host, there seems to be a gap in the processes listed.

Post by **TheWitness** » Mon Oct 24, 2005 11:43 am

Please provide the specific's of your system.

TheWitness

Pizzop · Post by **Pizzop** » Mon Oct 24, 2005 12:17 pm

Fedora Core 2, 2.6.10 kernel
runs nagios, arkeia backup software, and necessary softwares for cacti (mysql, apache, php, rrdtool)
1.8Ghz Pentium4, 1Gb ram, 1Gb swap
not sure what else you'd want here

Post by **rony** » Mon Oct 24, 2005 1:06 pm

How is your memory usage on the box?

Pizzop · Post by **Pizzop** » Mon Oct 24, 2005 1:46 pm

Here's the top from the system (not during a backup)

Code: Select all

top - 12:42:51 up 40 days, 20:45,  1 user,  load average: 0.12, 0.26, 0.18
Tasks:  76 total,   1 running,  73 sleeping,   0 stopped,   2 zombie
Cpu(s):  2.2% us,  4.6% sy, 20.4% ni, 56.0% id, 16.3% wa,  0.5% hi,  0.0% si
Mem:   1035828k total,  1008416k used,    27412k free,   740812k buffers
Swap:  1044216k total,     7376k used,  1036840k free,    76288k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                          
 8290 cacti     17   0 16588 8508 3760 S  5.9  0.8   0:02.91 php                                                                                              
 4641 mysql     16   0 33692  12m 2512 S  4.0  1.2  34:07.11 mysqld                                                                                           
 8289 cacti     17   0 18128 8484 3760 S  4.0  0.8   0:03.00 php                                                                                              
    1 root      16   0  2352  412  384 S  0.0  0.0   2:24.05 init                                                                                             
    2 root      34  19     0    0    0 S  0.0  0.0   0:58.59 ksoftirqd/0                                                                                      
    3 root       5 -10     0    0    0 S  0.0  0.0   2:53.10 events/0

I'll check the memory usage more during the backup tonight.

Pizzop · Post by **Pizzop** » Wed Oct 26, 2005 10:30 am

rony wrote:How is your memory usage on the box?

Well, memory usage seems fine. Nagios monitors things while the backup is running. Here's the cacti graph of load/memory usage over a backup.

Thanks.

Pizzop · Post by **Pizzop** » Wed Nov 02, 2005 1:18 pm

I still seem to get gaps, and the graphs are especially broken on the switches it seems.

here's an excerpt from the log:

Code: Select all

11/02/2005 10:50:02 AM - CMDPHP: Poller[0] ASSERT: '1921375800<1921375800' failed. Recaching host 'sw2.hq.premtech.com', data query #1
11/02/2005 10:50:16 AM - SYSTEM STATS: Time:15.5908 Method:cmd.php Processes:1 Threads:N/A Hosts:23 HostsPerProcess:23 DataSources:475 RRDsProcessed:492
11/02/2005 10:50:16 AM - SYSTEM STATS: Time:15.5409 Method:cmd.php Processes:1 Threads:N/A Hosts:23 HostsPerProcess:23 DataSources:475 RRDsProcessed:426
11/02/2005 10:50:17 AM - POLLER: Poller[0] Host[3] WARNING: Recache Event Detected for Host
11/02/2005 10:50:17 AM - POLLER: Poller[0] Host[3] WARNING: Recache Event Detected for Host
11/02/2005 10:50:37 AM - RECACHE STATS: RecacheTime:20.3294 HostsRecached:1
11/02/2005 10:50:37 AM - RECACHE STATS: RecacheTime:20.6919 HostsRecached:1
11/02/2005 10:55:07 AM - CMDPHP: Poller[0] Host[3] DS[34] WARNING: Result from SNMP not valid.  Partial Result: 
11/02/2005 10:55:07 AM - CMDPHP: Poller[0] ASSERT: '1921405600<' failed. Recaching host 'sw2.hq.premtech.com', data query #1
11/02/2005 10:59:58 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
11/02/2005 10:59:58 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
11/02/2005 11:00:27 AM - SYSTEM STATS: Time:25.9552 Method:cmd.php Processes:1 Threads:N/A Hosts:23 HostsPerProcess:23 DataSources:475 RRDsProcessed:308
11/02/2005 11:00:27 AM - SYSTEM STATS: Time:26.1311 Method:cmd.php Processes:1 Threads:N/A Hosts:23 HostsPerProcess:23 DataSources:475 RRDsProcessed:432
11/02/2005 11:00:28 AM - POLLER: Poller[0] Host[3] WARNING: Recache Event Detected for Host
11/02/2005 11:00:28 AM - POLLER: Poller[0] Host[3] WARNING: Recache Event Detected for Host
11/02/2005 11:00:58 AM - RECACHE STATS: RecacheTime:30.2733 HostsRecached:1
11/02/2005 11:00:59 AM - RECACHE STATS: RecacheTime:30.6249 HostsRecached:1
11/02/2005 11:05:02 AM - CMDPHP: Poller[0] ASSERT: '1921464800<1921464800' failed. Recaching host 'sw2.hq.premtech.com', data query #1
11/02/2005 11:09:58 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
11/02/2005 11:09:58 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.

Is there possibly a setting I should change in the poller.php file to increase the runtime timeout setting, and would it help if I did? Are there any other suggestions on what I should be trying to fix this problem?

wwwdrich · Post by **wwwdrich** » Wed Nov 09, 2005 7:05 pm

I'm seeing the same behaviour on a new cacti system I just installed here. The cricket graphs are fine, the cacti graphs for the same resources have gaps in them. I'm seeing this both on my WAN router and my air handlers.

Looking in the log, I see similar erorrs to other users:
11/09/2005 02:39:58 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.

Is there any way to either run multiple concurrent pollers or to thread it? With cricket I ended up splitting my data sets, as it couldn't complete a pool of all the devices in one 5 minute cycle.

Post by **gandalf** » Sat Nov 12, 2005 5:25 am

For that low amount of data sources that Pizop reported, there's no obviuos reason that polling should take 300 secs. I'm wondering about the logs entries appearing as twins. This looks like two polers runninmg concurrently. This is knows as a reason for being slow and having gaps in graphs. To check this out, please look at your crontab in more detail, e.g.

/etc/crontab
/etc/cron.d/cacti
crontab -e (for user root)
crontab -e -u cactiuser (for user cactiuser)

This is likely to appear when installing from rpm (puts cronjob /etc/cron.d/cacti) and manually inserting poller cmd at a crontab of your choice.
Reinhard

wwwdrich · Post by **wwwdrich** » Mon Nov 14, 2005 4:34 pm

Thanks lvm, that might just be my problem here! I found a copy of the poller in both /var/spool/cron *and* /etc/crontab.

Let's see if that fixes the problem....

Broken Graphs and Runtime-exceeded.

Broken Graphs and Runtime-exceeded.

Who is online