Graphing CSV data

[dzm]

Hello all -

I apologize in advance for bringing this subject up again. Searching shows me that this has come up several times, but each time the answer has been a link to the main docs site and a node (and HowTo) that is, evidently, no longer representational of what it was when the link was passed along.

So then ... I have a device that spits out CSV data once per second on a serial port. The data takes this form:

Code: Select all

1,04,00,00,001,121,00,02,000,00,272,008,000,031
C,00,10,02,112,052,00,00,000,01,271,000,000,044
D,00,23,08,080,073,00,00,000,02,271,000,000,063
1,04,00,00,001,121,00,02,000,00,272,008,000,031
C,00,11,03,113,052,00,00,000,01,270,000,000,046
D,00,23,08,080,073,00,00,000,02,271,000,000,063
1,04,00,00,001,121,00,02,000,00,272,008,000,031
C,00,09,02,112,052,00,00,000,01,270,000,000,051
[...]

What I want to do is simply rsync or scp the log to my server every five minutes or so and have Cacti suck in the data then generate graphs. Unfortunately it seems that Cacti doesn't have this kind of support for CSV data. Through much searching I think I've learned that I need to convert the CSV data into an RRD database, and I need to somehow convince Cacti to work with the frequently updated RRD file.

I can't imagine that my need to import and graph CSV data is particularly unique. I'd very much appreciate any pointers to solutions others have come up with for this problem.

Thanks very much!

[TheWitness]

1) Cacti can graph any numeric data with a script.
2) You simply have to get the data to somewhere where the script can read it.
3) PHP handles CSV data just fine.

So you are wrong in your assertion. However, some forms of collection take engineering. For you to properly engineer your graph requires you to have a better understanding of the infrastructure.

The key is to have your data there, before the poller starts, are at a time where the data will be fresh. If you have to first scp it, then interpret it (with PHP), it's going to delay the rest of your polling cycle.

So, Engineering is important.

TheWitness

[dzm]

I apologize if I characterized the situation incorrectly. What you describe is exactly my understanding.

Is there a HowTo or some other document available that clearly lays out feeding CSV data in and getting graphs out? I'm perfectly willing to do the engineering necessary to massage the data into whatever format is needed, or a script that spits out the data in a way that Cacti likes it, but having a HowTo that lays out all the steps clearly would be a big help.

[dzm]

Aha! Looks like this may be what I need:

http://docs.cacti.net/manual:087:3a_adv ... ut_methods

Thanks

[TheWitness]

Correct, use the script server if you have lot's of measured points using the same script. Also, PHP is a beautiful language. Here is an easy function for you:

http://us2.php.net/manual/en/function.fgetcsv.php

There are lot's of others CSV functions too. In the PHP manual, you will find lots of examples on how to use them.

TheWitness

[dzm]

I notice that the script examples seem to want to output only one "slice" of data (I can't of what else to call it).

Will Cacti react badly if it is passed many lines of output?

In my CSV file I have data that looks like:

Code: Select all

1,03,00,00,001,120,00,02,000,00,254,008,000,029
C,00,43,14,092,028,00,00,000,02,255,000,000,066
D,00,39,14,080,023,00,00,000,02,255,000,000,064
1,03,00,00,001,120,00,02,000,00,254,008,000,029
C,00,43,14,092,028,00,00,000,02,256,000,000,067
D,00,39,14,080,023,00,00,000,02,255,000,000,064

This data is updated once per second and is (ideally) updated on the server once per minute over the Internet. This means ~60 data updates pushed at one time into the Cacti poll. Will Cacti handle this properly, or will I need to post-process the 60 updates to create averages, then pass the averages as a single data point to Cacti?

[TheWitness]

Id do the latter. Its what rrdtool does anyway. If you could do that remotely and only send one point, then all the better.

[dzm]

Id do the latter. Its what rrdtool does anyway. If you could do that remotely and only send one point, then all the better.

Got it. Thanks!