no graphs (any more) - NaN values - 292 seconds exceeded

systems · Post by **systems** » Thu Aug 30, 2007 3:44 am

Hi all,
CACTI is not graphing any more.
I thought that the reason was related to the sizing of cacti.log (over 2GB), but I deteled it.
I have run manually the poller but the following error came up

08/30/2007 09:49:13 AM - POLLER: Poller[0] Maximum runtime of 292 seconds exceeded. Exiting.
08/30/2007 09:49:13 AM - SYSTEM STATS: Time:293.4071 Method:cmd.php Processes:1 Threads:N/A Hosts:440 HostsPerProcess:440 DataSources:8978 RRDsProcessed:4686

Warning: pclose(): 48 is not a valid stream resource in /usr/local/src/cacti/lib/rrd.php on line 48

Any suggestion?

I have verified the content of rrd files

 <row><v> 9.4525817036e+06 </v><v> 9.2631421904e+06 </v></row>
 <row><v> 9.3949010115e+06 </v><v> 9.2955578687e+06 </v></row>
 <row><v> 9.1901357119e+06 </v><v> 9.1347057076e+06 </v></row>
 <row><v> 9.2167728634e+06 </v><v> 8.9291323850e+06 </v></row>
 <row><v> 9.1642302467e+06 </v><v> 8.8307586694e+06 </v></row>
 <row><v> 9.3215036567e+06 </v><v> 8.8214230341e+06 </v></row>
 <row><v> 8.7765226984e+06 </v><v> 8.7093393151e+06 </v></row>
 <row><v> 8.4610250921e+06 </v><v> 8.7161975408e+06 </v></row>
 <row><v> NaN </v><v> NaN </v></row>
 <row><v> NaN </v><v> NaN </v></row>
 <row><v> NaN </v><v> NaN </v></row>
 <row><v> NaN </v><v> NaN </v></row>
 <row><v> NaN </v><v> NaN </v></row>
 <row><v> NaN </v><v> NaN </v></row>
 <row><v> NaN </v><v> NaN </v></row>
 <row><v> NaN </v><v> NaN </v></row>
 <row><v> NaN </v><v> NaN </v></row>
 <row><v> NaN </v><v> NaN </v></row>
 <row><v> NaN </v><v> NaN </v></row>
 <row><v> NaN </v><v> NaN </v></row>
 <row><v> NaN </v><v> NaN </v></row>
 <row><v> NaN </v><v> NaN </v></row>
 <row><v> NaN </v><v> NaN </v></row>
 <row><v> NaN </v><v> NaN </v></row>

All of them have the same problem: after 2:55 am all values are equal to NaN. So graphes are present, but source values are null

Please, help me

Post by **gandalf** » Thu Aug 30, 2007 12:19 pm

Please refer to second link of my signature. I suppose, that poller_output got filled up. Truncate it and increase your php memory limit
Reinhard

systems · Post by **systems** » Fri Aug 31, 2007 4:41 am

Hi Gandalf,
thank you for your reply.
After my 1st post, the situation is different.
I have solved NaN problem by changing settings on

Console->Settings->General
SNMP Timeout from 500 to 100
SNMP Retries from 3 to 1

Console->Settings->Poller
Maximum Concurrent Poller Processes from 1 to 4

By this, our poller is able to complete every cycle in 200 sec (9000 DataSource and 4500 rrd-file and 440 nodes). Processor load is higher, so I still need to find a better solution.

I have tested your suggested procedure, increasing memory php value (from 8M to 64M) still getting the same performance (200 sec)

Below, the output of sql query

mysql> select count(*) from poller_output;
+----------+
| count(*) |
+----------+
| 0 |
+----------+
1 row in set (0.00 sec)

As you can note, it was very fast. I don't need to truncate the table, don't I?

After all these tests, I have installed cactid poller but I have some problems with snmp library

CACTID: ERROR: SNMP Library Version Mismatch (5.2.1.2 vs 5.1.2)
(parent)CACTID: ERROR: SNMP Library Version Mismatch (5.2.1.2 vs 5.1.2)
(parent)CACTID: ERROR: SNMP Library Version Mismatch (5.2.1.2 vs 5.1.2)
(parent)CACTID: ERROR: SNMP Library Version Mismatch (5.2.1.2 vs 5.1.2)

I do not prefer to downgrade snmp version lib, because it is ok with cmd.php

Post by **gandalf** » Fri Aug 31, 2007 3:23 pm

systems wrote:I have solved NaN problem by changing settings on

Console->Settings->General
SNMP Timeout from 500 to 100
SNMP Retries from 3 to 1

Console->Settings->Poller
Maximum Concurrent Poller Processes from 1 to 4

I would assume from this, that some hosts may be detected as down, then. cacti will not wait on those slow ones for long, now.

By this, our poller is able to complete every cycle in 200 sec (9000 DataSource and 4500 rrd-file and 440 nodes). Processor load is higher, so I still need to find a better solution.

That's not a huge installation. But times depend on server sizing, especially on DISK and RAM.

I have tested your suggested procedure, increasing memory php value (from 8M to 64M) still getting the same performance (200 sec)

This was related to my assumption that poller_output gets filled up. Nevertheless, 64MB should be the lowest value

As you can note, it was very fast. I don't need to truncate the table, don't I?

No need for that

After all these tests, I have installed cactid poller but I have some problems with snmp library

CACTID: ERROR: SNMP Library Version Mismatch (5.2.1.2 vs 5.1.2)
(parent)CACTID: ERROR: SNMP Library Version Mismatch (5.2.1.2 vs 5.1.2)
(parent)CACTID: ERROR: SNMP Library Version Mismatch (5.2.1.2 vs 5.1.2)
(parent)CACTID: ERROR: SNMP Library Version Mismatch (5.2.1.2 vs 5.1.2)

I do not prefer to downgrade snmp version lib, because it is ok with cmd.php

Please compile cactid on your own. Do not use cmd.php on an installation that large. Be prepared for a performance boost of 6-8 times. Install net-snmp-devel libs prior to compiling

Reinhard

systems · Post by **systems** » Mon Sep 03, 2007 4:17 am

Hi all,
thanks a lot for your support.
Now the system seems to be ok.
It works with cactid poller!!!!

Regards

Cacti

no graphs (any more) - NaN values - 292 seconds exceeded

no graphs (any more) - NaN values - 292 seconds exceeded

Who is online