Graphing Clariion statistics from EMC Navisphere CLI output
Moderators: Developers, Moderators
Graphing Clariion statistics from EMC Navisphere CLI output
Hi all,
I'm a recent Cacti user and I've seen several posts on trying to use Cacti to graph EMC Clariion stats, so I'm well aware that SNMP doesn't provide any useful information.
However, the Navisphere CLI *does* provide information on RAID Group usage, SP cache hit usage, LUN read/write cache hit ratios etc etc.
So, IN THEORY, it should be possible to install Navisphere CLI on the Cacti server and create scripts using navicli to pull this information out of the Clariions, and then create all the necessary definitions within Cacti to populate RRDs with this information and then graph these statistics.
Has anyone attempted this yet? I'd love to hear from someone (anyone!) who has tried getting this up and running before I embark on what I expect to be a long and painful (but ultimately very useful!) learning experience.
Cheers,
Chris
I'm a recent Cacti user and I've seen several posts on trying to use Cacti to graph EMC Clariion stats, so I'm well aware that SNMP doesn't provide any useful information.
However, the Navisphere CLI *does* provide information on RAID Group usage, SP cache hit usage, LUN read/write cache hit ratios etc etc.
So, IN THEORY, it should be possible to install Navisphere CLI on the Cacti server and create scripts using navicli to pull this information out of the Clariions, and then create all the necessary definitions within Cacti to populate RRDs with this information and then graph these statistics.
Has anyone attempted this yet? I'd love to hear from someone (anyone!) who has tried getting this up and running before I embark on what I expect to be a long and painful (but ultimately very useful!) learning experience.
Cheers,
Chris
Hi,
i am also working on this.
But i cannot manage these problems:
- writing in perl to run the NAVICLI Commands
- Importing more than one field with an unknown number of values
I know, whats needs to be done:
- CLARiiON - Navisphere Analyzer needs to be running on the CLARiiON
- Data Logging needs to be set up (maybe with the default intervalls = 60sec for real time, 120sec for archive)
- CLARiiON Flare Code Version does not affect
- it makes sense to collect the files each 10min or later, otherwise there is not much data in
- Command to retrieve the Logging Files from CLARiiON :
"naviseccli -user admin -password password -scope 0 -h 1.2.3.4 analyzer -archive -all
-then the files need to be converted / dumped:
naviseccli analyzer -archivedump -data <Filename>.nar -out <New Filename>.csv -object s,l,d -format on,pt,u,ql,rt,tb,tt,rb,rs,rio,wb,ws,wio,up,ff,abql,dc,st,asd
--> Objects can be selected on request - check Navisphere Analyzer Documentation for details
Maybe this information is usefull. For me, the major thins is to load the content of the .csv file in CACTI.
Holger
i am also working on this.
But i cannot manage these problems:
- writing in perl to run the NAVICLI Commands
- Importing more than one field with an unknown number of values
I know, whats needs to be done:
- CLARiiON - Navisphere Analyzer needs to be running on the CLARiiON
- Data Logging needs to be set up (maybe with the default intervalls = 60sec for real time, 120sec for archive)
- CLARiiON Flare Code Version does not affect
- it makes sense to collect the files each 10min or later, otherwise there is not much data in
- Command to retrieve the Logging Files from CLARiiON :
"naviseccli -user admin -password password -scope 0 -h 1.2.3.4 analyzer -archive -all
-then the files need to be converted / dumped:
naviseccli analyzer -archivedump -data <Filename>.nar -out <New Filename>.csv -object s,l,d -format on,pt,u,ql,rt,tb,tt,rb,rs,rio,wb,ws,wio,up,ff,abql,dc,st,asd
--> Objects can be selected on request - check Navisphere Analyzer Documentation for details
Maybe this information is usefull. For me, the major thins is to load the content of the .csv file in CACTI.
Holger
Still working on it.shadow15g wrote:I'm also start working on that.
Anybody have any luck ?
My perl script is running, collect all NAR files from the CLARiiON, converts them to CSV and prints out each line from each CSV file.
But i still have no idea how to bring this output into CACTI.
A helpfull thing could be, that one field of the csv-file contains date & time.
- TheWitness
- Developer
- Posts: 17004
- Joined: Tue May 14, 2002 5:08 pm
- Location: MI, USA
- Contact:
This is easy, if it's in a csv file, and you have direct access to it, then...
Something like this. Say, the value is field 10. Then you write a nice little script that does the following:
If you want to go fancy, you would write a script_server script that read the file once into a global variable as an array of all parts and then share it between the various data sources with the first call loading the global array. That should have the least overhead since it will only have the file system overhead once.
If the file is on another host, you could rcp/scp it or run the script using ssh/rsh.
TheWitness
Something like this. Say, the value is field 10. Then you write a nice little script that does the following:
Code: Select all
#!/bin/bash
myVar=`tail -1 <myfile> | awk -F"," '{print $10}'`
echo $myVar
If the file is on another host, you could rcp/scp it or run the script using ssh/rsh.
TheWitness
True understanding begins only when we realize how little we truly understand...
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
- TheWitness
- Developer
- Posts: 17004
- Joined: Tue May 14, 2002 5:08 pm
- Location: MI, USA
- Contact:
I recommend, you learn to use your keyboard for something other than asking. How about contributing. Let's see how well you do. Sincerely, it's like that commercial that shows someone missing the garbage can with a piece of garbage and it just lies there on the side walk, instead of being the one complaining it's there on the sitewalk, why don't you simply pick it up an throw it in the garbage... (aka write it, and then contribute your writing).Rael wrote:Any updates on this? I'm looking for this as well.
TheWitness
True understanding begins only when we realize how little we truly understand...
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Hi,TheWitness wrote:This is easy, if it's in a csv file, and you have direct access to it, then...
Something like this. Say, the value is field 10. Then you write a nice little script that does the following:
If you want to go fancy, you would write a script_server script that read the file once into a global variable as an array of all parts and then share it between the various data sources with the first call loading the global array. That should have the least overhead since it will only have the file system overhead once.Code: Select all
#!/bin/bash myVar=`tail -1 <myfile> | awk -F"," '{print $10}'` echo $myVar
If the file is on another host, you could rcp/scp it or run the script using ssh/rsh.
TheWitness
nice idea with the AWK. I circle around the CSV File, because it looks like this:
---------------
Object Name,Poll Time,Utilization [%],Queue Length,Response Time [ms],Total Bandwidth [MB/s],Total Throughput [IO/s],Read Bandwidth [MB/s],Read Size [KB],Read Throughput [IO/s],Write Bandwidth [MB/s],Write Size [KB],Write Throughput [IO/s],Used Prefetches [%],Forced Flushes/s,Average Busy Queue Length,Disk Crossings/s,Service Time [ms],Average Seek Distance [GB]
SP A,08/03/2008 05:02:47,17.876896,0.414778,0.428274,32.428543,968.487950,26.475349,63.897130,424.287561,5.953194,11.201886,544.200389,,,2.320192,,0.184586,
SP A,08/03/2008 05:12:47,21.215028,0.271324,0.241639,30.961737,1122.849532,21.057491,61.150413,352.620207,9.904245,13.167438,770.229324,,,1.278922,,0.188939,
SP A,08/03/2008 05:22:47,20.834043,0.333600,0.289540,35.893853,1152.172478,23.451633,61.877592,388.096427,12.442220,16.674823,764.076052,,,1.601225,,0.180824,
SP A,08/03/2008 05:32:47,19.498747,0.266029,0.219095,29.296765,1214.217975,18.721839,44.117921,434.543676,10.574925,13.888778,779.674299,,,1.364339,,0.160587,
SP A,08/03/2008 05:42:47,19.726563,0.254216,0.220238,28.294289,1154.275769,18.147817,50.888619,365.177215,10.146472,13.166907,789.098555,,,1.288696,,0.170900,
SP A,08/03/2008 05:52:47,17.584881,0.285602,0.346549,26.858213,824.130644,16.936456,69.083181,251.044183,9.921757,17.728353,573.086461,,,1.624131,,0.213375,
SP A,08/03/2008 06:02:47,18.011758,0.312748,0.257192,33.689580,1216.010398,26.787748,39.560532,693.384350,6.901832,13.523007,522.626047,,,1.736357,,0.148122,
SP A,08/03/2008 06:12:47,23.223957,0.380700,0.211709,37.110734,1798.226192,27.596846,29.448189,959.623367,9.513888,11.617205,838.602825,,,1.639257,,0.129149,
----------------------
.... and may has another 50000 lines. Each field is a value.
Maybe i'm on the wrong way. When polling my Perl-Script, it gets a file from the CLARiiON and converts it to this .csv file. So each polling intervall generates so much data.
For example: if the CLARiiON has 100 LUNs, the csv file contains each 10 minutes aprox. 1000 lines with each line containing 20 values.
The file is local on the CACTI Server, so no need to SCP it.
- TheWitness
- Developer
- Posts: 17004
- Joined: Tue May 14, 2002 5:08 pm
- Location: MI, USA
- Contact:
Well, since you are keeping multiple intervals, cacti is only interested in the "last" one. So, instead of scp, do the following:
Parse the data using script server script into a global array. Then if you are reading the values in many script servers, say some for storage, others for cpu and or network. If all the values are pumped into the global array by the first call, then all subsequent calls can simply use the global array instead of using the ssh call a second time. Since, on a per host basis, all script server calls are serial, you will only have 1 SSH session per polling interval.
If you are not familiar with the script server, then readup. I would declare in your function, for a host named "myhost", I would create a global array in the function as follows:
TheWitness
Code: Select all
ssh -l <mylogin> <myclariion> cat <myfile> | grep <myhost> | tail -1
If you are not familiar with the script server, then readup. I would declare in your function, for a host named "myhost", I would create a global array in the function as follows:
Code: Select all
<pre-requisite script_server_stuff>
function get_my_clariion_cpu($myhostname) {
global $<myhostname>_clariion_stats;
if (isset($<myhostname>_clariion_stats)) {
get the value from here;
} else {
ssh the data into the global array;
}
return $my_cpu_stats;
}
True understanding begins only when we realize how little we truly understand...
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
- rony
- Developer/Forum Admin
- Posts: 6022
- Joined: Mon Nov 17, 2003 6:35 pm
- Location: Michigan, USA
- Contact:
The following script:
should be like this for effeciency:TheWitness wrote:Code: Select all
#!/bin/bash myVar=`tail -1 <myfile> | awk -F"," '{print $10}'` echo $myVar
Code: Select all
#!/bin/bash
tail -1 $1 | awk -F"," '{print $10}'
[size=117][i][b]Tony Roman[/b][/i][/size]
[size=84][i]Experience is what causes a person to make new mistakes instead of old ones.[/i][/size]
[size=84][i]There are only 3 way to complete a project: Good, Fast or Cheap, pick two.[/i][/size]
[size=84][i]With age comes wisdom, what you choose to do with it determines whether or not you are wise.[/i][/size]
[size=84][i]Experience is what causes a person to make new mistakes instead of old ones.[/i][/size]
[size=84][i]There are only 3 way to complete a project: Good, Fast or Cheap, pick two.[/i][/size]
[size=84][i]With age comes wisdom, what you choose to do with it determines whether or not you are wise.[/i][/size]
Re: Graphing Clariion statistics from EMC Navisphere CLI out
Hiya, I've just come from the Clariion Performance workshop and am looking to generate some 'baselline' stats for my Clariions. This thread looks to be a bit cold now, but I'm working on producing some graphs. Did you guys get anywhere?cjlove72 wrote: So, IN THEORY, it should be possible to install Navisphere CLI on the Cacti server and create scripts using navicli to pull this information out of the Clariions, and then create all the necessary definitions within Cacti to populate RRDs with this information and then graph these statistics.
I've got a bit of a different plan in mind.. To keep the load off the SPs I'm looking to populate a RRD file retrospectively. I don't need 'live' figures so was looking at writing a dump of yesterdays figures directly into a RRD from a single daily download of a .nar file. This could be done more frequently if finer grained graphs were needed.
There's a guy in the department doing something similar already so I can pick his brains....
Maybe we could put our heads together on this...
Spike
- TheWitness
- Developer
- Posts: 17004
- Joined: Tue May 14, 2002 5:08 pm
- Location: MI, USA
- Contact:
Spike,
I was going to do this for one of my customers. However, they invested in another tool to perform this. My plan was to use either rsh or ssh to access the node with the cli and return results, and do the graphing of those results.
For the number of Clariions that they had, it would take a second or two to get all the volume usage, cpu usage, memory usage, and IOPS.
Some times you win and sometimes you loose. I still believe it's the right thing to do. If security is a concern, you can setup a crontab on the management node to write this data to a file somewhere, then your Cacti scripts just need to grep the files. That's likely a much faster solution than several SSH calls.
TheWitness
I was going to do this for one of my customers. However, they invested in another tool to perform this. My plan was to use either rsh or ssh to access the node with the cli and return results, and do the graphing of those results.
For the number of Clariions that they had, it would take a second or two to get all the volume usage, cpu usage, memory usage, and IOPS.
Some times you win and sometimes you loose. I still believe it's the right thing to do. If security is a concern, you can setup a crontab on the management node to write this data to a file somewhere, then your Cacti scripts just need to grep the files. That's likely a much faster solution than several SSH calls.
TheWitness
True understanding begins only when we realize how little we truly understand...
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Re: Graphing Clariion statistics from EMC Navisphere CLI out
Spike,spike100 wrote: I've got a bit of a different plan in mind.. To keep the load off the SPs I'm looking to populate a RRD file retrospectively. I don't need 'live' figures so was looking at writing a dump of yesterdays figures directly into a RRD from a single daily download of a .nar file. This could be done more frequently if finer grained graphs were needed.
load on the SPs is not an issue. Navisphere Analyzer + Data Collection creates around 10% overhead. Depending on your setup, you will have 3-6 files per SP (@all: each CLARiiON has 2 SPs, each one has it's own IP and creates different data).
I finished a perl script to collect all .nar (Navisphere Data) files from one CLARiiON and dump them into one csv file. Now i still need some help to bring this data into CACTI or RRD.
My script is attached. To run it ou need to have perl installed (Cacti on Linux already has.....) and you need Navisphere Secure CLI installed on the system. If anybody does not have a live environment, i'll provide some of my Navisphere Data Files.
Sorry, i'm on a dead-end street with my knowledge.
- Attachments
-
- cacti_clariion_analyzer_output.pl.txt
- Perl script to connect to a CLARiiON, get all Navisphere Analyzer Files , convert them into a CSV file using Navisphere Secure CLI Commands and print each line of the CSV file
- (6.89 KiB) Downloaded 1438 times
- TheWitness
- Developer
- Posts: 17004
- Joined: Tue May 14, 2002 5:08 pm
- Location: MI, USA
- Contact:
Can you provide the output and the field names associated with the output?
To do this more correctly, you need a few functions similar to the host disk stuff for Disk Utilization. Each disk would report things like:
Input Fields for the Data Query Would be like
volumeName
fileSystemType
blockSize
totalBlocks
Output Fields would include:
IOSperSecond
usedMBytes
freeMBytes
IOWait
etc.
If the CSV file has those details, then that's great. You can create a data query. Then for things like CPU, Memory, Cache, you might be able to simply use a graph template.
Network utilization again requires a Data Query. What I didn't like was the EMC choose not to support 64bit counters for their SNMP agent (I have no freeking clue why). So again back to the Nav utility.
Good luck.
TheWitness
To do this more correctly, you need a few functions similar to the host disk stuff for Disk Utilization. Each disk would report things like:
Input Fields for the Data Query Would be like
volumeName
fileSystemType
blockSize
totalBlocks
Output Fields would include:
IOSperSecond
usedMBytes
freeMBytes
IOWait
etc.
If the CSV file has those details, then that's great. You can create a data query. Then for things like CPU, Memory, Cache, you might be able to simply use a graph template.
Network utilization again requires a Data Query. What I didn't like was the EMC choose not to support 64bit counters for their SNMP agent (I have no freeking clue why). So again back to the Nav utility.
Good luck.
TheWitness
True understanding begins only when we realize how little we truly understand...
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Who is online
Users browsing this forum: No registered users and 0 guests