Maximum Number of Data Source Elements ?

[bestpa]

Hi, I've got a script that returns 496 elements in the format of (unique_name:value ... ... ) for population into a data source with 496 data source items.

My results are that from time to time the poller does not seem to function correctly, and the rrd file sits un-updated with a stale timestamp.

Code: Select all

04/19/2012 04:20:02 PM - SPINE: Poller[0] Host[194] TH[1] DS[15637] SCRIPT: /usr/share/cacti/site/scripts/mme_report_to_cacti.pl 154.11.207.210, output: 0 

Is there a limitation or known bug regarding the length of the data source?

It's really strange. I've got 8 devices using the same DS template and same data input method (script), some work, some do not. Manual running of the script using the cacti poller user (www-data) show the data to be output.

Code: Select all

04/19/2012 04:25:02 PM - SPINE: Poller[0] Host[195] TH[1] DS[15638] SCRIPT: /usr/share/cacti/site/scripts/mme_report_to_cacti.pl 154.11.249.209, output: mme0:0 mme1:130646 mme2:0 mme3:0 mme4:0 mme5:774 mme6:60068581 mme7:9083990 mme8:0 mme9:741978192 mme10:210991935 mme11:179 mme12:0 mme13:130646 mme14:0 mme15:0 mme16:0 mme17:774 mme18:0 mme19:9762012 mme20:57265036 mme21:0 mme22:210991935 mme23:741931576 mme24:192 mme25:0 mme26:0 mme27:0 mme28:0 mme29:0 mme30:7949 mme31:2789374358 mme32:596390258 mme33:60068581 mme34:9761910 mme35:674 mme36:0 mme37:0 mme38:15 mme39:0 mme40:0 mme41:0 mme42:54046098 mme43:274 mme44:0 mme45:0 mme46:409 mme47:10598 mme48:314 mme49:712325 mme50:973969 mme51:88230 mme52:705480 mme53:3033 mme54:13858 mme55:630 mme56:14450 mme57:0 mme58:84350 mme59:12 mme60:0 mme61:0 mme62:0 mme63:13852 mme64:0 mme65:0 mme66:0 mme67:0 mme68:0 mme69:0 mme70:0 mme71:674 mme72:15 mme73:0 mme74:0 mme75:274 mme76:0 mme77:0 mme78:408 mme79:10593 mme80:313 mme81:0 mme82:711422 mme83:659 mme84:705671 mme85:973255 mme86:88192 mme87:8 mme88:962067 mme89:496913 mme90:258 mme91:801 mme92:13858 mme93:41 mme94:14531 mme95:592 mme96:13789 mme97:84411 mme98:13852

Here's what the script looks like when run manually... i hunted through the output and tried to find any strange hex or any values out of position, nothing. I've verified manually that the data retrieved by the script is sane and verified (496 elements, no bad characters, numbers).

Code: Select all

root@cacti2:~# sudo su - www-data
$ /usr/share/cacti/site/scripts/mme_report_to_cacti.pl 154.11.207.210
mme0:0 mme1:130533 mme2:0 mme3:0 mme4:0 mme5:643 mme6:8794293 mme7:1871568 mme8:0 mme9:703620329 mme10:192120448 mme11:167 mme12:0 mme13:130533 mme14:0 mme15:0 mme16:0 mme17:643 mme18:0 mme19:1987667 mme20:8446801 mme21:0 mme22:192120448 mme23:703578073 mme24:72 mme25:0 mme26:0 mme27:0 mme28:0 mme29:0 mme30:745 mme31:421678263 mme32:125203639 mme33:8794293 mme34:1987660 mme35:597 mme36:0 mme37:0 mme38:11 mme39:0 mme40:0 mme41:0 mme42:7539605 mme43:288 mme44:0 mme45:0 mme46:179 mme47:1437 mme48:67 mme49:313663 mme50:406759 mme51:26435 mme52:351405 mme53:965 mme54:9274 mme55:19 mme56:9274 mme57:0 mme58:32979 mme59:1 mme60:0 mme61:0 mme62:0 mme63:9273 mme64:0 mme65:0 mme66:0 mme67:0 mme68:0 mme69:0 mme70:0 mme71:597 mme72:11 mme73:0 mme74:0 mme75:288 mme76:0 mme77:0 mme78:179 mme79:1435 mme80:67 mme81:0 mme82:313338 mme83:238 mme84:308525 mme85:406428 mme86:26429 mme87:2 mme88:397975 mme89:261337 mme90:273 mme91:283 mme92:9274 mme93:24 mme94:9317 mme95:0 mme96:9259 mme97:33060 mme98:9273 mme99:52562 mme100:0 mme101:0 mme102:0 mme103:0 mme104:0 mme105:0 mme106:0 mme107:0 mme108:0 mme109:0 mme110:0 mme111:0 mme112:0 mme113:0 mme114:0 mme115:0 mme116:0 mme117:0 mme118:0 mme119:0 mme120:0 mme121:389 mme122:1374 mme123:36 mme124:1338 mme125:0 mme126:0 mme127:0 mme128:0 mme129:0 mme130:0 mme131:12 mme132:12 mme133:0 mme134:0 mme135:0 mme136:0 mme137:2049 mme138:6 mme139:2043 mme140:127 mme141:0 mme142:127 mme143:56655 mme144:26564 mme145:30091 mme146:24545 mme147:15857 mme148:0 mme149:8688 mme150:3610 mme151:3425 mme152:0 mme153:185 mme154:38 mme155:14 mme156:0 mme157:24 mme158:5501 mme159:2975 mme160:0 mme161:2526 mme162:0 mme163:0 mme164:0 mme165:0 mme166:14850 mme167:14777 mme168:0 mme169:73 mme170:36774 mme171:35587 mme172:1187 mme173:7872 mme174:7761 mme175:111 mme176:40157 mme177:40076 mme178:81 mme179:7851 mme180:7850 mme181:1 mme182:2 mme183:2 mme184:0 mme185:264 mme186:257 mme187:7 mme188:0 mme189:0 mme190:0 mme191:1315 mme192:1315 mme193:0 mme194:136 mme195:133 mme196:3 mme197:291 mme198:291 mme199:0 mme200:308215 mme201:308215 mme202:0 mme203:287221 mme204:287073 mme205:148 mme206:148266 mme207:145158 mme208:3108 mme209:0 mme210:0 mme211:145158 mme212:0 mme213:50351 mme214:40157 mme215:225233 mme216:1186 mme217:0 mme218:26145 mme219:815 mme220:10515 mme221:0 mme222:2 mme223:0 mme224:0 mme225:0 mme226:9102 mme227:0 mme228:0 mme229:0 mme230:0 mme231:0 mme232:1235 mme233:0 mme234:35720 mme235:134 mme236:179 mme237:12 mme238:191 mme239:0 mme240:3430 mme241:44195 mme242:71692 mme243:11 mme244:0 mme245:0 mme246:1600 mme247:49 mme248:40157 mme249:84 mme250:420 mme251:213 mme252:158 mme253:0 mme254:49290 mme255:92 mme256:32041 mme257:0 mme258:0 mme259:0 mme260:0 mme261:9 mme262:0 mme263:0 mme264:0 mme265:21015 mme266:1820 mme267:0 mme268:0 mme269:0 mme270:318 mme271:346 mme272:52760 mme273:373454 mme274:156250 mme275:582464 mme276:0 mme277:14371 mme278:0 mme279:22326 mme280:37139 mme281:2 mme282:1 mme283:24501 mme284:397 mme285:4715 mme286:1285 mme287:3430 mme288:0 mme289:5 mme290:0 mme291:40079 mme292:2 mme293:285359 mme294:79557 mme295:0 mme296:40967 mme297:3317 mme298:27531 mme299:22512 mme300:5019 mme301:1514 mme302:1514 mme303:0 mme304:691 mme305:691 mme306:0 mme307:7 mme308:7 mme309:0 mme310:276 mme311:276 mme312:0 mme313:22666 mme314:22513 mme315:153 mme316:0 mme317:0 mme318:0 mme319:0 mme320:0 mme321:0 mme322:967 mme323:967 mme324:0 mme325:7 mme326:7 mme327:0 mme328:1514 mme329:1514 mme330:0 mme331:26578 mme332:26458 mme333:120 mme334:0 mme335:0 mme336:0 mme337:0 mme338:0 mme339:0 mme340:0 mme341:0 mme342:0 mme343:0 mme344:0 mme345:62376 mme346:150 mme347:0 mme348:0 mme349:0 mme350:179 mme351:0 mme352:0 mme353:0 mme354:0 mme355:0 mme356:0 mme357:0 mme358:0 mme359:0 mme360:0 mme361:0 mme362:0 mme363:0 mme364:0 mme365:0 mme366:0 mme367:0 mme368:0 mme369:0 mme370:67 mme371:0 mme372:35404 mme373:150 mme374:26641 mme375:0 mme376:73 mme377:0 mme378:38 mme379:0 mme380:0 mme381:0 mme382:0 mme383:12 mme384:0 mme385:0 mme386:0 mme387:0 mme388:0 mme389:0 mme390:62396 mme391:62396 mme392:0 mme393:0 mme394:0 mme395:179 mme396:3 mme397:67 mme398:0 mme399:0 mme400:35331 mme401:0 mme402:26460 mme403:25 mme404:253 mme405:78 mme406:32980 mme407:32980 mme408:0 mme409:9292 mme410:9259 mme411:33 mme412:0 mme413:0 mme414:0 mme415:0 mme416:0 mme417:0 mme418:127 mme419:0 mme420:127 mme421:0 mme422:0 mme423:0 mme424:0 mme425:0 mme426:0 mme427:0 mme428:0 mme429:0 mme430:0 mme431:0 mme432:0 mme433:0 mme434:0 mme435:0 mme436:46411 mme437:42571 mme438:3840 mme439:56660 mme440:26566 mme441:30094 mme442:0 mme443:0 mme444:0 mme445:0 mme446:0 mme447:0 mme448:0 mme449:0 mme450:0 mme451:0 mme452:0 mme453:0 mme454:0 mme455:0 mme456:0 mme457:0 mme458:0 mme459:0 mme460:0 mme461:0 mme462:0 mme463:0 mme464:0 mme465:0 mme466:0 mme467:0 mme468:0 mme469:1606 mme470:52 mme471:1719 mme472:0 mme473:0 mme474:0 mme475:148 mme476:51 mme477:109 mme478:165 mme479:59 mme480:111 mme481:0 mme482:0 mme483:0 mme484:0 mme485:0 mme486:0 mme487:5398 mme488:3311 mme489:325 mme490:90 mme491:41 mme492:41 mme493:1 mme494:4 mme495:54053
$ 

Any ideas folks?

[gandalf]

First, I'm not aware of any limitation in rrdtool itself regarding this.
Second, putting many data sources into a single rrd file is not recommended in general as it's quite inflexible due to restrictions of rrdtool (not allowing to add/delete data sources to existing rrd files).
Third, from a performance standpoint, there have been reports that rrdtool does not perform significantly slower when handling many rrd files using few data sources compared to using a single rrd file with many data sources.
Fourth, due to some "limitations" in Cacti, we do not perform in an optomal way when following item 2+3 of my recommendations (there's a manual but tedious way to avoid stuff).
Fifth, I can imagine that some of the cacti tables "poller_*" have limitations on the table size. So you may change that by adding a custom table tweak "alter table change column ..." to upper some varchar fields.
R.

[bestpa]

Hi Gandolf;

Thanks for the response. We are aware of the concerns regarding large rrd's, but we're also very aware of the repercussions.

As a side note, here's why we are creating these large RRD's and what we intend to use it for:
1) create template package (data input method, data source, single graph using one data source). This will allow an RRD file to be created once a graph is created from a host using that template.
2) create a clone (different hashes) template package referencing the data source template that is not active but contains the same item names.
3) create the 495 graphs from this template. No RRD file will be created, though the DS will reference a reserved RRD number.
4) using a sweeping mysql command, change all 495 cloner data source RRD's from the reserved ones to the single RRD that is in use by the first template package.

In this way, we hope to use the same RRD by many graphs. Each RRD is about 27MB for us, and we don't want 23MB * 495 graph RRDs * 12 hosts = 130GB of RRD !!!

Alternatively, we could define 495 data sources, and call 495 perl scripts (virtually) to repeat back the name:value pair, but that's sloppy and will take longer than just updating a single RRD with 495 values. The average execution time is about 0.13 seconds for an RRD of our size (27MB).

We've done some more testing and it looks like spine might be the issue. We have a second cacti server that we can reserve just for testing.

Code: Select all

root@cacti2:~# ls -latr /var/lib/cacti/rra/mmetest*  (With cmd.php running as poller)
-rw-r--r-- 1 www-data www-data 23225720 2012-04-19 20:40 /var/lib/cacti/rra/mmetest6_mme495_15638.rrd
-rw-r--r-- 1 www-data www-data 23225720 2012-04-19 20:40 /var/lib/cacti/rra/mmetest1_mme495_15633.rrd
-rw-r--r-- 1 www-data www-data 23225720 2012-04-19 20:40 /var/lib/cacti/rra/mmetest3_mme495_15635.rrd
-rw-r--r-- 1 www-data www-data 23225720 2012-04-19 20:40 /var/lib/cacti/rra/mmetest2_mme495_15634.rrd
-rw-r--r-- 1 www-data www-data 23225720 2012-04-19 20:40 /var/lib/cacti/rra/mmetest4_mme495_15636.rrd
-rw-r--r-- 1 www-data www-data 23225720 2012-04-19 20:40 /var/lib/cacti/rra/mmetest5_mme495_15637.rrd
-rw-r--r-- 1 www-data www-data 23225720 2012-04-19 20:40 /var/lib/cacti/rra/mmetest7_mme495_15639.rrd
root@cacti2:~# ls -latr /var/lib/cacti/rra/mmetest*  (With spine running)
-rw-r--r-- 1 www-data www-data 23225720 2012-04-19 20:40 /var/lib/cacti/rra/mmetest1_mme495_15633.rrd
-rw-r--r-- 1 www-data www-data 23225720 2012-04-19 20:40 /var/lib/cacti/rra/mmetest3_mme495_15635.rrd
-rw-r--r-- 1 www-data www-data 23225720 2012-04-19 20:40 /var/lib/cacti/rra/mmetest2_mme495_15634.rrd
-rw-r--r-- 1 www-data www-data 23225720 2012-04-19 20:40 /var/lib/cacti/rra/mmetest4_mme495_15636.rrd
-rw-r--r-- 1 www-data www-data 23225720 2012-04-19 20:40 /var/lib/cacti/rra/mmetest5_mme495_15637.rrd
-rw-r--r-- 1 www-data www-data 23225720 2012-04-19 20:45 /var/lib/cacti/rra/mmetest6_mme495_15638.rrd
-rw-r--r-- 1 www-data www-data 23225720 2012-04-19 20:45 /var/lib/cacti/rra/mmetest7_mme495_15639.rrd
root@cacti2:~# ls -latr /var/lib/cacti/rra/mmetest* (With cmd.php running as poller)
-rw-r--r-- 1 www-data www-data 23225720 2012-04-19 20:50 /var/lib/cacti/rra/mmetest6_mme495_15638.rrd
-rw-r--r-- 1 www-data www-data 23225720 2012-04-19 20:50 /var/lib/cacti/rra/mmetest1_mme495_15633.rrd
-rw-r--r-- 1 www-data www-data 23225720 2012-04-19 20:50 /var/lib/cacti/rra/mmetest3_mme495_15635.rrd
-rw-r--r-- 1 www-data www-data 23225720 2012-04-19 20:50 /var/lib/cacti/rra/mmetest2_mme495_15634.rrd
-rw-r--r-- 1 www-data www-data 23225720 2012-04-19 20:50 /var/lib/cacti/rra/mmetest4_mme495_15636.rrd
-rw-r--r-- 1 www-data www-data 23225720 2012-04-19 20:50 /var/lib/cacti/rra/mmetest5_mme495_15637.rrd
-rw-r--r-- 1 www-data www-data 23225720 2012-04-19 20:50 /var/lib/cacti/rra/mmetest7_mme495_15639.rrd

So what we see is that each RRD is updated without problems when cacti poller cmd.php is used, but when spine is used, some do not get updated (timestamps are stale to represent failed update). I think this might be a buffer thing. But, i will investigate the size of the poller_* tables anyways just to be sure.

[gandalf]

Well, I won't say that spine is NOT the root cause. Your test case seems to rule tables out. This is which version of spine?
R.

[bestpa]

We are currently using SPINE 0.8.7g with CACTI Version 0.8.7g.

Do you suggest moving to SPINE 0.8.8 and trying again? Will this SPINE version be backwards compatible?

[gandalf]

You either should use 087i or the very fresh 088a versions of spine. There's not that much of a difference between the two of them.
R.