Ad blocker detected: Our website is made possible by displaying online advertisements to our visitors. Please consider supporting us by disabling your ad blocker on our website.
This is going to more informational than problem related.
I made the mistake of going on vacation for about a week. When I get back "My cacti broke" I was getting no error messages when I tried to run cacti by hand in DEBUG mode. It would just hang. This particular cacti install only polls about 30 hosts.
I re-compilied cacitd and even re-installed cacti with no luck. I disabled all but one host, and I was still getting nowhere, so I was pretty sure this was a mysql thing. I checked, double-checked, and rechecked the host,db,username info for cactid and cacti, still nothing.
When I dumped the mysql table when I re-installed cacti, I noticed it was really big ~218MB. I don't know how big it should be, but I thought this was TOO big. So I looked to see what table it was... poller_output. Since I've been working on this problem on and off for about 3 days now, I was willing to try anything. I killed everything in the table...
If it continues to happen, I would be curious to see you debug log during one polling cycle.
[size=117][i][b]Tony Roman[/b][/i][/size]
[size=84][i]Experience is what causes a person to make new mistakes instead of old ones.[/i][/size]
[size=84][i]There are only 3 way to complete a project: Good, Fast or Cheap, pick two.[/i][/size]
[size=84][i]With age comes wisdom, what you choose to do with it determines whether or not you are wise.[/i][/size]
Email the log to me, don't filter out anything other than IP addresses if you are really concerned.
I suspect that you will not have the problem again. But we will see.
[size=117][i][b]Tony Roman[/b][/i][/size]
[size=84][i]Experience is what causes a person to make new mistakes instead of old ones.[/i][/size]
[size=84][i]There are only 3 way to complete a project: Good, Fast or Cheap, pick two.[/i][/size]
[size=84][i]With age comes wisdom, what you choose to do with it determines whether or not you are wise.[/i][/size]
mysql> select count(*) from poller_output;
+----------+
| count(*) |
+----------+
| 484303 |
+----------+
1 row in set (0.00 sec)
rony, do you want me to e-mail you the log to the e-mail from your profile? It is not too large ~4.9Mb. Let me know if you have a limit, otherwise, I can e-mail you a link, so you can download it.
It is likely that PHP ran out of memory attempting to process output records from the database. This will oftentime cause issues retrieving records. Once it starts, it's all downhill from there.
I am considering a patch to limit the number or rows returned. It will likely cause some degredation in performance, but it will minimize this risk. Until then, please increase the memory you make available to PHP.
TheWitness
True understanding begins only when we realize how little we truly understand...
Any suggestion on the limit. There is 2Gb of memory on this box, Cacti + Nagios are the only things running on it. Along w/ any dependencies those may include
and "grep rrdtool /var/log/cacti/cacti.log" == zero even with "DEBUG" level
actually, it is like
03/01/2006 12:27:33 PM - CACTID: Poller[0] DEBUG: In Poller, About to Start Polling of Host
03/01/2006 12:27:33 PM - CACTID: Poller[0] MYSQL: Connecting to MySQL database 'cacti' on 'localhost'...
03/01/2006 12:27:33 PM - CACTID: Poller[0] DEBUG: Valid Thread to be Created
03/01/2006 12:27:33 PM - CACTID: Poller[0] DEBUG: The Value of Active Threads is 1
03/01/2006 12:27:33 PM - CACTID: Poller[0] Host[259] SNMP Result: Host responded to SNMP
03/01/2006 12:27:33 PM - CACTID: Poller[0] DEBUG: SQLCMD: update host set status='3',status_event_count='0', status_fail_date='0000-00-00 00:00:00',status_rec_date='0000-00-00 00:00:00',status_last_error='',min_time='5.198960',max_time='668.390040',cur_time='167.399880',avg_time='159.701520',total_polls='433',failed_polls='0',availability='100.0000' where id='259'
Has there been any solution to this? I am running cactid-0.8.6e and cacti 8.6e on FC4 and I am having the same problem. The poller is completing in under 300 seconds, but the RRDs are not being updated. I've tried all of this to no avail.
Thoughts?
P.S - This system has been running flawlessly since September.
Sorry for the double post, but I figured I'd put a little more info about my situation, incase anyone else has run into this or knows a solution.
What started all of this was that I was reaching the 5 minute polling limit and added about 7 devices and crossed the 5 minute threshold. This caused the polling to not finish and the graphs to stop updating. When I noticed that this had occurred I abruptly removed those devices assuming that the polling time would go back down and everything would be fine - that wasn't the case.
After this I began to do everything from increasing the amount of memory that php has access to (from 8MB to 256MB), stopping the poller and running "truncate table poller_output;" followed by "mysqlcheck --auto-repair --databases cacti", then clicking on "Clear Poller Cache", and even doing "delete from poller_output where time<NOW();" (as mentioned above).
I am currently output in DEBUG mode and everything seems to be correct (I even manually ran the displayed rrd update commands with the parameters, via command line, with the cactiuser, and it worked). Is there any reason why I can update via command line, with the same command shown, but cactid is not updating it? Is this a problem with my database? I'm just unsure where to look.
I've been having the same problem. The weird part is that it looks like any data at all left over in the poller_output table when the poller starts running will cause the poller to not output any data to the RRDs. I'm not sure exactly why this is, but it's obviously causing quite a bit of data loss. Hours, days, etc.
Is there any way to take the contents of the poller_output table and output the data into the RRDs manually? Additionally, is it possible to seperate that process from the one that does the polling? That sounds like it might be a more efficient way of doing things.
But we will see.
Let me know if you have a limit, otherwise, I can e-mail you a link, so you can download it.
