Ad blocker detected: Our website is made possible by displaying online advertisements to our visitors. Please consider supporting us by disabling your ad blocker on our website.
TheWitness wrote:Hmm. Well, some versions of the 1.3.x branch are horrible. Tobi corrected the errors, most of which had to do with the dependent libraries. However, not all distributions have "caught up".
However, the 1.2.27 is rock solid. So, when people are having problems, I ask them to roll back.
If that makes no appreciable change, I would also check with the SAN administrator and see what is enabled from Read/Write Cache and make sure it's a lot. If it is large, I would start digging into the backend.
I could tell more if I were on the system. We had one vendor that recently had all sorts of issues with Cacti performance. What turned out to be the problem was a buggy RAID firmware from the supplier. We applied the new firmware, and the problem was gone.
That's something else to look at.
TheWitness
I'll try the other version of rrdtool... we build them all from source, so everything is up to date as per Tobi's specs. Our boost processing times are running int the 5-10 minute range:
SYSTEM BOOST STATS: Time:440.5869 RRDUpdates:487562
I'm not entirely sure this is the problem. It may be erratic SAN performance, I'll have to dig into the management system to see what's going on there.
dononeil wrote:Ok... I rolled back to RRDTool 1.2.30, which I already had built...
Times don't seem to be any better, in fact, they seem worse than with 1.3.8:
1.3.8:
10/07/2009 10:04:53 AM - SYSTEM BOOST STATS: Time:661.7185 RRDUpdates:1706886
10/07/2009 10:19:07 AM - SYSTEM BOOST STATS: Time:611.8501 RRDUpdates:731259
10/07/2009 10:24:19 AM - SYSTEM BOOST STATS: Time:297.7239 RRDUpdates:487579
10/07/2009 10:29:31 AM - SYSTEM BOOST STATS: Time:284.5589 RRDUpdates:244013
10/07/2009 10:35:08 AM - SYSTEM BOOST STATS: Time:330.4778 RRDUpdates:243633
10/07/2009 10:45:17 AM - SYSTEM BOOST STATS: Time:373.9294 RRDUpdates:440696
10/07/2009 10:54:45 AM - SYSTEM BOOST STATS: Time:347.4310 RRDUpdates:487779
10/07/2009 11:01:43 AM - SYSTEM BOOST STATS: Time:406.8440 RRDUpdates:243636
10/07/2009 11:37:45 AM - SYSTEM BOOST STATS: Time:533.2667 RRDUpdates:1218725
10/07/2009 11:51:06 AM - SYSTEM BOOST STATS: Time:708.7005 RRDUpdates:487288
10/07/2009 12:00:30 PM - SYSTEM BOOST STATS: Time:388.0661 RRDUpdates:731214
10/07/2009 12:11:21 PM - SYSTEM BOOST STATS: Time:441.3023 RRDUpdates:487779
10/07/2009 12:19:49 PM - SYSTEM BOOST STATS: Time:343.4474 RRDUpdates:487690
10/07/2009 12:25:05 PM - SYSTEM BOOST STATS: Time:311.9346 RRDUpdates:243931
10/07/2009 12:35:01 PM - SYSTEM BOOST STATS: Time:360.2818 RRDUpdates:487575
10/07/2009 12:46:29 PM - SYSTEM BOOST STATS: Time:440.5869 RRDUpdates:487562
10/07/2009 12:55:09 PM - SYSTEM BOOST STATS: Time:360.3177 RRDUpdates:487356
10/07/2009 01:04:20 PM - SYSTEM BOOST STATS: Time:324.1716 RRDUpdates:487607
10/07/2009 01:43:45 PM - SYSTEM BOOST STATS: Time:573.2109 RRDUpdates:1462286
10/07/2009 02:33:01 PM - SYSTEM BOOST STATS: Time:534.4842 RRDUpdates:487232
1.2.30:
10/07/2009 03:01:36 PM - SYSTEM BOOST STATS: Time:1632.9512 RRDUpdates:486945
10/07/2009 03:33:03 PM - SYSTEM BOOST STATS: Time:1729.9638 RRDUpdates:975499
10/07/2009 03:53:55 PM - SYSTEM BOOST STATS: Time:1177.8843 RRDUpdates:731354
10/07/2009 04:20:51 PM - SYSTEM BOOST STATS: Time:1588.7748 RRDUpdates:487777
10/07/2009 04:44:20 PM - SYSTEM BOOST STATS: Time:1210.2388 RRDUpdates:731683
10/07/2009 05:21:07 PM - SYSTEM BOOST STATS: Time:1631.0423 RRDUpdates:731892
10/07/2009 05:53:18 PM - SYSTEM BOOST STATS: Time:1745.3021 RRDUpdates:731107
10/07/2009 06:17:20 PM - SYSTEM BOOST STATS: Time:1387.3777 RRDUpdates:731323
10/07/2009 06:47:49 PM - SYSTEM BOOST STATS: Time:1420.0942 RRDUpdates:731411
10/07/2009 07:23:57 PM - SYSTEM BOOST STATS: Time:1792.9736 RRDUpdates:731218
10/07/2009 07:45:33 PM - SYSTEM BOOST STATS: Time:1272.3637 RRDUpdates:731796
10/07/2009 08:21:21 PM - SYSTEM BOOST STATS: Time:1639.5979 RRDUpdates:731979
10/07/2009 08:45:27 PM - SYSTEM BOOST STATS: Time:1270.9439 RRDUpdates:731536
10/07/2009 09:17:08 PM - SYSTEM BOOST STATS: Time:1382.1190 RRDUpdates:731263
10/07/2009 09:48:42 PM - SYSTEM BOOST STATS: Time:1469.1791 RRDUpdates:731619
10/07/2009 10:22:34 PM - SYSTEM BOOST STATS: Time:1705.3388 RRDUpdates:731494
10/07/2009 10:41:57 PM - SYSTEM BOOST STATS: Time:1054.7968 RRDUpdates:731009
10/07/2009 11:08:58 PM - SYSTEM BOOST STATS: Time:1481.3632 RRDUpdates:487957
10/07/2009 11:42:32 PM - SYSTEM BOOST STATS: Time:1705.8512 RRDUpdates:731388
10/08/2009 12:07:47 AM - SYSTEM BOOST STATS: Time:1419.0148 RRDUpdates:731760
I'm going to go back to 1.3.8 and see if the trend stays the same.
You might want to consider running your boost updates a little less frequently. Keep an eye on those memory table sizes and you might want to optimize some column widths. Remember memory table for poller_output and poller_output_boost.
TheWitness
True understanding begins only when we realize how little we truly understand...
TheWitness wrote:
You might want to consider running your boost updates a little less frequently. Keep an eye on those memory table sizes and you might want to optimize some column widths. Remember memory table for poller_output and poller_output_boost.
TheWitness
Well... I couldn't update a little less frequently before due to memory constraints. I've added 8G more RAM to the server and changed my updates to run about once every 1 hour before the table fills up all available RAM. Already running both of those tables in memory, with index's and checked the column widths... Made it as clean as possible.
There are just so darned many hosts (16k) with so many metrics that I can't get them all to run in the alloted time of 5 mins. The data we're collecting isn't even 'critical', its just for problem solving, so the ability to run polling every 10 or 15 mintues would be great, but that's not possible with the current cacti code base.
TheWitness wrote:
You might want to consider running your boost updates a little less frequently. Keep an eye on those memory table sizes and you might want to optimize some column widths. Remember memory table for poller_output and poller_output_boost.
TheWitness
Well... I couldn't update a little less frequently before due to memory constraints. I've added 8G more RAM to the server and changed my updates to run about once every 1 hour before the table fills up all available RAM. Already running both of those tables in memory, with index's and checked the column widths... Made it as clean as possible.
There are just so darned many hosts (16k) with so many metrics that I can't get them all to run in the alloted time of 5 mins. The data we're collecting isn't even 'critical', its just for problem solving, so the ability to run polling every 10 or 15 mintues would be great, but that's not possible with the current cacti code base.
There is an option of combining Boost 2.5 with spine 0.8.7e, but I have not tested it. It will cut your polling time down about 40%.
TheWitness
True understanding begins only when we realize how little we truly understand...
TheWitness wrote:
You might want to consider running your boost updates a little less frequently. Keep an eye on those memory table sizes and you might want to optimize some column widths. Remember memory table for poller_output and poller_output_boost.
TheWitness
Well... I couldn't update a little less frequently before due to memory constraints. I've added 8G more RAM to the server and changed my updates to run about once every 1 hour before the table fills up all available RAM. Already running both of those tables in memory, with index's and checked the column widths... Made it as clean as possible.
There are just so darned many hosts (16k) with so many metrics that I can't get them all to run in the alloted time of 5 mins. The data we're collecting isn't even 'critical', its just for problem solving, so the ability to run polling every 10 or 15 mintues would be great, but that's not possible with the current cacti code base.
There is an option of combining Boost 2.5 with spine 0.8.7e, but I have not tested it. It will cut your polling time down about 40%.
TheWitness
Yeah, I saw that option, but it did nothing for me when I enabled it. No difference in polling time at all.
