A new plugin for exporting in/outbound interface data to CSV or JSON format has been released.
It regularly exports the last 1 hour of data to a CSV or JSON compatible file which can be processed by other applications for further analysis ( on-demand, financial calculations ... )
Look here for more details:
http://docs.cacti.net/userplugin:nmiddataexport
nmidDataExport plugin released
Moderators: Developers, Moderators
-
phalek
- Developer
- Posts: 2838
- Joined: Thu Jan 31, 2008 6:39 am
- Location: Kressbronn, Germany
- Contact:
nmidDataExport plugin released
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusReporting
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusReporting
-
miefrancine
- Posts: 3
- Joined: Thu Nov 07, 2013 5:59 am
Re: nmidDataExport plugin released
Good morning,
We´ve installed nmidDataExport plugin to Cacti, but CSV file only shows "Date" column. What can be the problem?
We´ve installed nmidDataExport plugin to Cacti, but CSV file only shows "Date" column. What can be the problem?
-
phalek
- Developer
- Posts: 2838
- Joined: Thu Jan 31, 2008 6:39 am
- Location: Kressbronn, Germany
- Contact:
Re: nmidDataExport plugin released
Hi,
The plugin can currently only export the in- and outbound interface data.
May this be the issue ?
The plugin can currently only export the in- and outbound interface data.
May this be the issue ?
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusReporting
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusReporting
-
miefrancine
- Posts: 3
- Joined: Thu Nov 07, 2013 5:59 am
Re: nmidDataExport plugin released
Here´s what happens:
"Date"
"2013-11-07 10:00:00"
"2013-11-07 10:05:00"
"2013-11-07 10:10:00"
"2013-11-07 10:15:00"
"2013-11-07 10:20:00"
"2013-11-07 10:25:00"
"2013-11-07 10:30:00"
"2013-11-07 10:35:00"
"2013-11-07 10:40:00"
"2013-11-07 10:45:00"
"2013-11-07 10:50:00"
"2013-11-07 10:55:00"
That´s everything we´ve got. No inbound or outbound data are shown...
"Date"
"2013-11-07 10:00:00"
"2013-11-07 10:05:00"
"2013-11-07 10:10:00"
"2013-11-07 10:15:00"
"2013-11-07 10:20:00"
"2013-11-07 10:25:00"
"2013-11-07 10:30:00"
"2013-11-07 10:35:00"
"2013-11-07 10:40:00"
"2013-11-07 10:45:00"
"2013-11-07 10:50:00"
"2013-11-07 10:55:00"
That´s everything we´ve got. No inbound or outbound data are shown...
Re: nmidDataExport plugin released
hi ,
I also have this problem (only column date), do someone have the solution to export a good csv or json file with all the requested information ?
And did mysql must be runing by "ip/phpmyadmin" ?
thks !
I also have this problem (only column date), do someone have the solution to export a good csv or json file with all the requested information ?
And did mysql must be runing by "ip/phpmyadmin" ?
thks !
-
phalek
- Developer
- Posts: 2838
- Joined: Thu Jan 31, 2008 6:39 am
- Location: Kressbronn, Germany
- Contact:
Re: nmidDataExport plugin released
Hi,
The plugin only exports the data for the interface template, nothing else is included right now.
The plugin only exports the data for the interface template, nothing else is included right now.
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusReporting
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusReporting
-
optimuscream
- Posts: 41
- Joined: Mon Jun 13, 2011 11:20 pm
Re: nmidDataExport plugin released
Hello,
I've installed the plugins and running well. However the data capture is not incrementing very well. Every about one hour , old data is rewritten with the new data.
Example below :
How to capture the data and kept it for 1 month ? I want to use it for 95 percentile calculation.
Edit : I post to fast. So I have to figure out some script to parse 1 hour data to be processed later.
Any pointer maybe ?
Thank you
I've installed the plugins and running well. However the data capture is not incrementing very well. Every about one hour , old data is rewritten with the new data.
Example below :
Code: Select all
"2015-05-27 14:35:00","4904810.23","14696874.12"
"2015-05-27 14:40:00","16887386.02","15841404.19"
"2015-05-27 14:45:00","8721250.65","15392949.77"
"2015-05-27 14:50:00","13233196.73","15883410.73"
"2015-05-27 14:55:00","10658930.02","15416177.64"
"2015-05-27 15:00:00","1318402.96","14443395.26"
"2015-05-27 15:05:00","20727413.38","17074666.73"
"2015-05-27 15:10:00","16633034.08","15822719.92"
"2015-05-27 15:15:00","16647087.66","15401663.03"
"2015-05-27 15:20:00","13625627.83","15273063.06"
"2015-05-27 15:25:00","8368218.27","14773883.56"
"2015-05-27 15:30:00","6943746.96","15927862.45"
"2015-05-27 15:35:00","",""
root@cacti-machine:/var/www/plugins/nmidDataExport/export# cat ip.add.res.ss_251_18796.csv
"Date","Inbound","Outbound"
"2015-05-27 14:40:00","16887386.02","15841404.19"
"2015-05-27 14:45:00","8721250.65","15392949.77"
"2015-05-27 14:50:00","13233196.73","15883410.73"
"2015-05-27 14:55:00","10658930.02","15416177.64"
"2015-05-27 15:00:00","1318402.96","14443395.26"
"2015-05-27 15:05:00","20727413.38","17074666.73"
"2015-05-27 15:10:00","16633034.08","15822719.92"
"2015-05-27 15:15:00","16647087.66","15401663.03"
"2015-05-27 15:20:00","13625627.83","15273063.06"
"2015-05-27 15:25:00","8368218.27","14773883.56"
"2015-05-27 15:30:00","6943746.96","15927862.45"
"2015-05-27 15:35:00","103586865.55","13767750.25"
"2015-05-27 15:40:00","",""
You have new mail in /var/mail/root
root@cacti-machine:/var/www/plugins/nmidDataExport/export# cat ip.add.res.ss_251_18796.csv
"Date","Inbound","Outbound"
"2015-05-27 15:15:00","16647087.66","15401663.03"
"2015-05-27 15:20:00","13625627.83","15273063.06"
"2015-05-27 15:25:00","8368218.27","14773883.56"
"2015-05-27 15:30:00","6943746.96","15927862.45"
"2015-05-27 15:35:00","103586865.55","13767750.25"
"2015-05-27 15:40:00","109182555.88","13427334.01"
"2015-05-27 15:45:00","110788358.07","13795909.36"
"2015-05-27 15:50:00","13793832.11","14245568.50"
"2015-05-27 15:55:00","22787395.51","16161155.56"
"2015-05-27 16:00:00","18860408.23","15865964.00"
"2015-05-27 16:05:00","13516391.42","16429725.94"
"2015-05-27 16:10:00","95419449.97","14769083.14"
"2015-05-27 16:15:00","",""
Edit : I post to fast. So I have to figure out some script to parse 1 hour data to be processed later.
Any pointer maybe ?
Thank you
-
phalek
- Developer
- Posts: 2838
- Joined: Thu Jan 31, 2008 6:39 am
- Location: Kressbronn, Germany
- Contact:
Re: nmidDataExport plugin released
The timerange can be selected from the settings. Currently it only support up to 1 day of data. In order to add some new time-ranges, you can change the lines starting from line number 328 in the setup.php file to look like this:
Not sure what the impact on the polling time will be though, as it is going to export the data every 1 or 5 minutes ...
Alternatively, you can change the filename to have a timestamp in it.
Change the following code at around line 226 from this:
to this:
make sure to cleanup that directory after you've read-in the data...
Code: Select all
"nmid_de_datatimerange" => array(
"friendly_name" => "Data Time Range",
"description" => "The time range of the data.",
"method" => "drop_array",
"default" => "3600",
"array" => array(
"300" => "5 minutes",
"900" => "15 minutes",
"1800" => "30 minutes",
"2700" => "45 minutes",
"3600" => "1 hour",
"7200" => "2 hour",
"43200" => "12 hour",
"86400" => "1 day"
"604800" => "1 week"
"2592000" => "30 days"
"31536000" => "1 year"
)
),
Alternatively, you can change the filename to have a timestamp in it.
Change the following code at around line 226 from this:
Code: Select all
/* Make graph title the suggested file name */
if ( is_array( $xport_array[ "meta" ] ) ) {
$filename = $hostname.'_'.$xport_array[ "meta" ][ "host_id" ] . '_' . $xport_array[ "meta" ][ "local_graph_id" ] . $fileending;
}
else {
$filename = 'graph_export'.$fileending;
}
Code: Select all
/* Make graph title the suggested file name */
if ( is_array( $xport_array[ "meta" ] ) ) {
$filename = $hostname.'_'.$xport_array[ "meta" ][ "host_id" ] . '_' . $xport_array[ "meta" ][ "local_graph_id" ] . '_' . time() . $fileending;
}
else {
$filename = 'graph_export' . '_' . time() . $fileending;
}
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusReporting
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusReporting
-
optimuscream
- Posts: 41
- Joined: Mon Jun 13, 2011 11:20 pm
Re: nmidDataExport plugin released
Hello Phalek
Thanks for the response.
Line 328 on setup.php start with :
So I put the above definition down below nmid_de_exportDir array
And change default to : 2592000
The web page goes blank after I disable and enable the plugins.
So I choose second alternative, It works created the timestamp file every 5 min but the rrdtool process took a lot cpu resources all the time, leaving the system almost inaccesible.
First Run
==========
Second Run and after.
======================
I noticed every rrdtool process running before is not closed , so it just hang after it process everytime it run. At the end of the day, rrdtool process take a lot of hit in the system so I have to restart the machine to refresh all the hanging process.
Any other idea or pointer ?
Edit : I make it work with first method , little typo here and there in the code. Let see the results.
Thank
Thanks for the response.
Line 328 on setup.php start with :
Code: Select all
"nmid_de_exportDir" => array(
"friendly_name" => 'Export Directory', -------------------------------> line 328
"description" => 'The directory where to export the files.',
"method" => "textbox",
"default" => dirname(__FILE__) . '/export/',
"max_length" => 255
)
Code: Select all
"nmid_de_exportDir" => array(
"friendly_name" => 'Export Directory',
"description" => 'The directory where to export the files.',
"method" => "textbox",
"default" => dirname(__FILE__) . '/export/',
"max_length" => 255
)
"nmid_de_datatimerange" => array(
"friendly_name" => "Data Time Range",
"description" => "The time range of the data.",
"method" => "drop_array",
"default" => "2592000",
"array" => array(
"300" => "5 minutes",
"900" => "15 minutes",
"1800" => "30 minutes",
"2700" => "45 minutes",
"3600" => "1 hour",
"7200" => "2 hour",
"43200" => "12 hour",
"86400" => "1 day"
"604800" => "1 week"
"2592000" => "30 days"
"31536000" => "1 year"
)The web page goes blank after I disable and enable the plugins.
So I choose second alternative, It works created the timestamp file every 5 min but the rrdtool process took a lot cpu resources all the time, leaving the system almost inaccesible.
First Run
==========
Code: Select all
top - 10:14:01 up 6 min, 1 user, load average: 1.65, 1.25, 0.58
Tasks: 101 total, 4 running, 97 sleeping, 0 stopped, 0 zombie
Cpu(s): 40.0%us, 2.5%sy, 0.0%ni, 56.9%id, 0.5%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 8257868k total, 1049604k used, 7208264k free, 60192k buffers
Swap: 2050044k total, 0k used, 2050044k free, 818264k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8522 www-data 20 0 14860 5908 3656 R 99 0.1 2:28.05 rrdtool
8386 www-data 20 0 60904 24m 6912 R 61 0.3 1:29.06 php
1153 mysql 20 0 320m 53m 6480 S 10 0.7 0:19.34 mysqld
1226 root 20 0 0 0 0 S 0 0.0 0:00.66 flush-8:0
1 root 20 0 3520 1920 1284 S 0 0.0 0:00.96 init
======================
Code: Select all
top - 10:40:13 up 32 min, 1 user, load average: 10.50, 8.71, 5.90
Tasks: 165 total, 21 running, 144 sleeping, 0 stopped, 0 zombie
Cpu(s): 92.0%us, 7.8%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 8257868k total, 1989812k used, 6268056k free, 66760k buffers
Swap: 2050044k total, 0k used, 2050044k free, 1209532k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
29775 www-data 20 0 14840 5912 3656 R 50 0.1 4:20.32 rrdtool
14903 www-data 20 0 14872 5968 3660 R 44 0.1 16:09.90 rrdtool
8522 www-data 20 0 14836 5872 3660 R 43 0.1 18:07.17 rrdtool
23164 www-data 20 0 14872 5968 3660 R 40 0.1 7:11.15 rrdtool
4338 www-data 20 0 14836 5872 3656 R 39 0.1 1:42.00 rrdtool
4129 www-data 20 0 60904 24m 6916 R 27 0.3 1:03.71 php
29694 www-data 20 0 81384 44m 6984 R 27 0.6 3:08.61 php
1153 mysql 20 0 325m 62m 6632 S 20 0.8 5:22.19 mysqld
Any other idea or pointer ?
Edit : I make it work with first method , little typo here and there in the code. Let see the results.
Thank
-
optimuscream
- Posts: 41
- Joined: Mon Jun 13, 2011 11:20 pm
Re: nmidDataExport plugin released
Hello Phalek,
The first method is not working , it still generate csv data only for one hour . Although I have change it to 30 days in Settings -> NMID -> NMID - DataExport - General -> Data Time Range
Where do I miss ?
Thank you.
The first method is not working , it still generate csv data only for one hour . Although I have change it to 30 days in Settings -> NMID -> NMID - DataExport - General -> Data Time Range
Code: Select all
cat ip.add.res.ss_251_18796.csv
"Date","Inbound","Outbound"
"2015-05-28 13:25:00","23307498.16","21684346.66"
"2015-05-28 13:30:00","66854614.16","21041158.34"
"2015-05-28 13:35:00","90349417.14","20783726.80"
"2015-05-28 13:40:00","99923212.02","21594503.13"
"2015-05-28 13:45:00","84616262.01","21808292.27"
"2015-05-28 13:50:00","88183975.63","21397769.64"
"2015-05-28 13:55:00","62109775.74","19929562.47"
"2015-05-28 14:00:00","",""
"2015-05-28 14:05:00","",""
"2015-05-28 14:10:00","",""
"2015-05-28 14:15:00","78951749.99","20751524.48"
"2015-05-28 14:20:00","69749104.88","20338723.57"
"2015-05-28 14:25:00","",""
You have new mail in /var/mail/root
root@cacti-machine:/var/www/plugins/nmidDataExport/export# cat ip.add.res.ss_251_18796.csv
"Date","Inbound","Outbound"
"2015-05-28 13:25:00","23307498.16","21684346.66"
"2015-05-28 13:30:00","66854614.16","21041158.34"
"2015-05-28 13:35:00","90349417.14","20783726.80"
"2015-05-28 13:40:00","99923212.02","21594503.13"
"2015-05-28 13:45:00","84616262.01","21808292.27"
"2015-05-28 13:50:00","88183975.63","21397769.64"
"2015-05-28 13:55:00","62109775.74","19929562.47"
"2015-05-28 14:00:00","",""
"2015-05-28 14:05:00","",""
"2015-05-28 14:10:00","",""
"2015-05-28 14:15:00","78951749.99","20751524.48"
"2015-05-28 14:20:00","69749104.88","20338723.57"
"2015-05-28 14:25:00","",""
root@cacti-machine:/var/www/plugins/nmidDataExport/export# cat ip.add.res.ss_251_18796.csv
"Date","Inbound","Outbound"
"2015-05-28 13:45:00","84616262.01","21808292.27"
"2015-05-28 13:50:00","88183975.63","21397769.64"
"2015-05-28 13:55:00","62109775.74","19929562.47"
"2015-05-28 14:00:00","",""
"2015-05-28 14:05:00","",""
"2015-05-28 14:10:00","",""
"2015-05-28 14:15:00","78951749.99","20751524.48"
"2015-05-28 14:20:00","69749104.88","20338723.57"
"2015-05-28 14:25:00","69210572.48","20144440.57"
"2015-05-28 14:30:00","64738571.16","20055606.49"
"2015-05-28 14:35:00","83156634.18","20662866.92"
"2015-05-28 14:40:00","97040934.01","21649650.78"
"2015-05-28 14:45:00","",""
You have new mail in /var/mail/rootThank you.
-
phalek
- Developer
- Posts: 2838
- Joined: Thu Jan 31, 2008 6:39 am
- Location: Kressbronn, Germany
- Contact:
Re: nmidDataExport plugin released
It should look like this:
and make sure there's no other entry for the "nmid_de_datatimerange" in there.
Code: Select all
"nmid_de_exportDir" => array(
"friendly_name" => 'Export Directory',
"description" => 'The directory where to export the files.',
"method" => "textbox",
"default" => dirname(__FILE__) . '/export/',
"max_length" => 255
),
"nmid_de_datatimerange" => array(
"friendly_name" => "Data Time Range",
"description" => "The time range of the data.",
"method" => "drop_array",
"default" => "2592000",
"array" => array(
"300" => "5 minutes",
"900" => "15 minutes",
"1800" => "30 minutes",
"2700" => "45 minutes",
"3600" => "1 hour",
"7200" => "2 hour",
"43200" => "12 hour",
"86400" => "1 day",
"604800" => "1 week",
"2592000" => "30 days",
"31536000" => "1 year"
)
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusReporting
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusReporting
-
optimuscream
- Posts: 41
- Joined: Mon Jun 13, 2011 11:20 pm
Re: nmidDataExport plugin released
Hi phalek,
Yes exactly, I've added missing comma's at the end of last lines. And change the correct time range in NMID menu in Settings.
The whole section is like below :
But the results still the same.
Please check.
Thank you.
Yes exactly, I've added missing comma's at the end of last lines. And change the correct time range in NMID menu in Settings.
The whole section is like below :
Code: Select all
function nmidDataExport_config_settings()
{
global $tabs, $settings;
$tabs[ "nmid" ] = "NMID";
$temp = array(
"nmid_de_header" => array(
"friendly_name" => "NMID - DataExport - General",
"method" => "spacer",
),
"nmid_de_datatype" => array(
"friendly_name" => "Data Export Type",
"description" => "You can choose between CSV or JSON data format.",
"method" => "drop_array",
"default" => "CSV",
"array" => array(
"CSV" => "CSV",
"JSON" => "JSON"
)
),
"nmid_de_exportDir" => array(
"friendly_name" => 'Export Directory',
"description" => 'The directory where to export the files.',
"method" => "textbox",
"default" => dirname(__FILE__) . '/export/',
"max_length" => 255
),
"nmid_de_datatimerange" => array(
"friendly_name" => "Data Time Range",
"description" => "The time range of the data.",
"method" => "drop_array",
"default" => "2592000",
"array" => array(
"300" => "5 minutes",
"900" => "15 minutes",
"1800" => "30 minutes",
"2700" => "45 minutes",
"3600" => "1 hour",
"7200" => "2 hour",
"43200" => "12 hour",
"86400" => "1 day",
"604800" => "1 week",
"2592000" => "30 days",
"31536000" => "1 year"
)
)
);Please check.
Thank you.
-
phalek
- Developer
- Posts: 2838
- Joined: Thu Jan 31, 2008 6:39 am
- Location: Kressbronn, Germany
- Contact:
Re: nmidDataExport plugin released
Did you go to the settings page and actually switch to the new data timerange ?
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusReporting
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusReporting
-
optimuscream
- Posts: 41
- Joined: Mon Jun 13, 2011 11:20 pm
Re: nmidDataExport plugin released
Yes. I've tell it before.phalek wrote:Did you go to the settings page and actually switch to the new data timerange ?
The first method is not working , it still generate csv data only for one hour . Although I have change it to 30 days in Settings -> NMID -> NMID - DataExport - General -> Data Time Range
-
phalek
- Developer
- Posts: 2838
- Joined: Thu Jan 31, 2008 6:39 am
- Location: Kressbronn, Germany
- Contact:
Re: nmidDataExport plugin released
I think I found it:
At Line 220, change this:
to this:
At Line 220, change this:
Code: Select all
$xport_array = @rrdtool_function_xport($s_ExportItem[ "lgid" ], $s_ExportItem[ "rraid" ], $graph_data_array, $xport_meta);
Code: Select all
$xport_array = @rrdtool_function_xport($s_ExportItem[ "lgid" ], '', $graph_data_array, $xport_meta);
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusReporting
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusReporting
Who is online
Users browsing this forum: No registered users and 0 guests