NetApp FAS 2020 Filer and Fiber Channel IOPS

[isaac737]

Hi all,

So I am using Cacti 0.8.7g and it is fully patched. We are monitoring a NetApp FAS 2020 Filer using the "Official NetApp Template" from the Cacti Website. We are also using Eli's NA Cacti Template (http://www.elifulkerson.com/projects/ne ... mplate.php). His is great but doesn't provide IOPS for Fiber Channel.

My question is if there is a way to break out Reads vs. Writes in the template we are using. Basically I think the template is producing a sum of both Reads and Writes but we would ideally like to see: reads, writes and a total of the two.

Any ideas on how to do this? Is there another template out there we can use? Or do we need to tweak the existing template?

I am attaching a screenshot of what I am describing.

Any insights would be greatly appreciated!!!

Cheers to the Cacti Community,
jg

[Linegod]

- Duplicate the current graph
- Edit the new graph, change the template to 'none'
- Edit the new graph to your liking
- Save the new graph as a template
- Apply new template to other graphs as need be

[lints]

Do you by chance mean like this...

Image showing graph of Throughput

netapp_fcp_throuhput.png (41.43 KiB) Viewed 6300 times

Image showing graph of Operations

netapp_fcp_operations.png (29.45 KiB) Viewed 6300 times

Attached is a zip file containing templates for both graphs.

[isaac737]

Dude!

That's exactly what we are after. I have imported the Throughput Template and it is breaking down the FCP IOPS in terms of Reads/Writes/and Total.

Few questions for you if you would care to kick down:

#1) Is the y-axis truly bytes/sec?

#2) Am I interpreting the attached graph correctly - We have way, way more Reads than Writes?

Thanks!

-jg

[lints]

As far as I can tell, the y-axis is indeed in bytes/sec.

That graph pulls OIDs .1.3.6.1.4.1.789.1.17.20.0 and .1.3.6.1.4.1.789.1.17.21.0 which are named in the NetApp MIB as 'fcp64ReadBytes' and 'fcp64WriteBytes' respectively. The '64' in the name is just to differentiate these as 64-bit counters, since there are 32-bit versions of these elsewhere in the MIB.

Based on your graph, yes: you're pushing an average of 4.5 megabytes/sec of reads and just over 19 kilobytes/second of writes.

You should be able to test this fairly easily. If you're connected to Linux machines, use DD to copy a file to/from the SAN and check the speed it reports. If you're on Windows, copy some huge file and watch the appropriate counters in perfmon.

To be honest, we don't have any load on our NetApp for the moment since it's only in pre-production so I haven't seen any real-world numbers yet. In my tests the counters seemed to report accurately though. I'll let you know what I see once we go into production.

[lints]

So, upon closer inspection, it appears that I'm wrong...

I've got a number of other graphs on the filer to measure throughput (LUN-level and overall disk throughput on the controller) and since I've only got 1 LUN for the time being the numbers should match up.

Suffice to say they don't. To make these graphs I'm basing myself off the MIB, but if it's wrong, then the graphs are wrong.

Based off these other graphs, it looks like they've reversed the OIDs (or the underlying counters)... Read is write and write is read.

See attached images for an example where I used 'dd' to read a 5 GB file into /dev/null.

[isaac737]

Hey dude,

Thanks for the reply and for the clarification. It does seem that they are reversed on the FCP Throughput template.

If I run:

iostat -Xtc 3 1000

On our Solaris 10 Host I see stuff like this:

[root@******-db-zone:/ ] $ iostat -Xtc 3 1000
extended device statistics tty cpu
device r/s w/s kr/s kw/s wait actv svc_t %w %b tin tout us sy wt id
sd0 4.7 9.3 546.2 560.9 0.0 0.0 3.5 0 1 0 21 2 1 0 97
sd5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd2 83.1 166.3 1111.3 6375.5 0.0 3.7 14.7 0 51
ssd2.fp2 33.0 70.7 484.4 2730.8 0.0 0.0 0.0 0 0
ssd2.fp0 87.5 166.9 1093.4 6357.5 0.0 0.0 0.0 0 0
ssd3 0.0 6.7 5.0 834.0 0.0 0.1 20.6 0 1
ssd3.fp2 0.0 6.7 5.0 834.0 0.0 0.0 0.0 0 0
ssd3.fp0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
extended device statistics tty cpu
device r/s w/s kr/s kw/s wait actv svc_t %w %b tin tout us sy wt id
sd0 0.0 49.7 0.0 111.1 0.0 0.0 0.1 0 0 0 468 0 0 0 99
sd5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd2 1.0 85.8 8.0 919.8 0.0 0.1 0.8 0 6
ssd2.fp2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd2.fp0 1.0 85.8 8.0 919.8 0.0 0.0 0.0 0 0
ssd3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd3.fp2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd3.fp0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
extended device statistics tty cpu
device r/s w/s kr/s kw/s wait actv svc_t %w %b tin tout us sy wt id
sd0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 468 0 0 0 100
sd5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd2 0.0 457.3 0.0 15145.3 0.0 3.1 6.7 0 33
ssd2.fp2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd2.fp0 0.0 457.3 0.0 15145.3 0.0 0.0 0.0 0 0
ssd3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd3.fp2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd3.fp0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
extended device statistics tty cpu
device r/s w/s kr/s kw/s wait actv svc_t %w %b tin tout us sy wt id
sd0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 470 0 0 0 100
sd5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd2 1.3 83.3 10.7 1299.7 0.0 0.1 1.4 0 9
ssd2.fp2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd2.fp0 1.3 83.3 10.7 1299.7 0.0 0.0 0.0 0 0
ssd3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd3.fp2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd3.fp0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
extended device statistics tty cpu
device r/s w/s kr/s kw/s wait actv svc_t %w %b tin tout us sy wt id
sd0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 469 2 0 0 98
sd5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd2 26.3 1.7 237.3 106.7 0.0 0.6 21.9 0 8
ssd2.fp2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd2.fp0 26.3 1.7 237.3 106.7 0.0 0.0 0.0 0 0
ssd3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd3.fp2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd3.fp0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
^C
[root@your-mom-db-zone:/ ] $

So, way higher writes than reads. I would love to try dd but the dd command on Solaris (even the GNU Version from Blastwave) doesn't report anything useful on completion of the write. Any thoughts on how I can easily tweak the existing FCP template so it reports reads as writes and writes as reads? Maybe somehow change the legend? Also, would u mind sharing your LUN template? GREAT WORK!



Cheers,
jg

[lints]

No problem. I don't know why I didn't see this before.

In any event, here's the updated FCP Throughput graph.

cacti_graph_template_netapp_-_fcp_-_throughput.zip

NetApp FCP Throughput graph

(3.28 KiB) Downloaded 243 times

All I did was reverse the legend text on items 1 and 5. The data sources are going to be backwards now, but I think for most people as long as the graph works, who cares.. I can always change those too though if you want.

Here's the per-LUN graphs; IOs (Operations) and Throughput.

cacti_graph_template_netapp_lun.zip

NetApp per-LUN throughput and IO graphs

(7.82 KiB) Downloaded 283 times

Import the graph templates (which include the data templates) and put the netapp_luns.xml file inside your resources/snmp_queries directory.

As a bonus, here's a graph for the total disk throughput on the controller for all protocols (NFS, CIFS, FCP, iSCSI, etc..).

cacti_graph_template_netapp_-_disk_throughput.zip

NetApp controller-level disk throughput

(3.28 KiB) Downloaded 288 times

This might include the de-dupe process, replication, snapshotting and so on, but as stated earlier, we're not in production yet so I don't know everything it's doing.

[fotan]

I try to use Netapp_lun to my cacti 0.8.7g + RRDTool 1.2, but graphics is NOT being graphed. Export xml's is ok. Netapp_luns put in snmp_queuries. Associated graph templates show me "NetApp - LUN - Operations Is Being Graphed", but in the graph management RRDTool Says: ERROR: opening '/var/www/cacti/rra/1_na_lun_ios_total_1217.rrd': No such file or directory

What i'am doing wrong? All other stuff from my netapp FAS 2020 graphed ok. How can i view statistic from my LUN?

[lints]

Hi Fotan,

When you're in the Create Graphs page for your controller, do you see the LUNS listed under the Data Query?

How long ago did you create the graphs? Sometimes it takes 15-20 minutes for the data points to start plotting in a visible manner.

[fotan]

Yes, Data Query [Netapp - Volumes] show me all LUNs. And i try to IOPS about 2 weeks. In graph management LUN-IOs-|query_lunName| graph field alsow choose correct, but *.rrd do not create. and i am not understand why...

[lints]

Well, based on the fact that the query works and you can see the fields in the Graph Creation page, I don't think it has to do with the graph template..

Maybe it's because you're got an older version of RRDTool and there's something in the graph or RRD creation that's causing an error.

If you go onto the data source for one of the graphs and enable "debug" mode, you'll see the 'rrdtool create' command. Maybe try running that manually on the command line and see if it throws any errors.

[fotan]

My RRDtool version is 1.2.30. About RRD manual mode is a good idea, thank you. My RRD script return me error: step size should be no less then one second. But, afaik param -step set in second? and in my situation is 5 minutes. In other graphs that command is work good. May be situation depend on 64/32 bit counters in RRDTool? And i have another quastion - in that command my LUNs not assign directli, is this statistic of all LUNs togather?

Data Source Debug
/usr/bin/rrdtool create \
/var/www/cacti/rra/1_na_lun_ios_total_1217.rrd \
--step 300 \
DS:na_lun_ios_reads:COUNTER:600:0:100000 \
DS:na_lun_ios_writes:COUNTER:600:0:100000 \
DS:na_lun_ios_other:COUNTER:600:0:10000 \
DS:na_lun_ios_total:COUNTER:600:0:200000 \
RRA:AVERAGE:0.5:1:500 \
RRA:AVERAGE:0.5:1:600 \
RRA:AVERAGE:0.5:6:700 \
RRA:AVERAGE:0.5:24:775 \
RRA:AVERAGE:0.5:288:797 \
RRA:MIN:0.5:1:500 \
RRA:MIN:0.5:1:600 \
RRA:MIN:0.5:6:700 \
RRA:MIN:0.5:24:775 \
RRA:MIN:0.5:288:797 \
RRA:MAX:0.5:1:500 \
RRA:MAX:0.5:1:600 \
RRA:MAX:0.5:6:700 \
RRA:MAX:0.5:24:775 \
RRA:MAX:0.5:288:797 \
RRA:LAST:0.5:1:500 \
RRA:LAST:0.5:1:600 \
RRA:LAST:0.5:6:700 \
RRA:LAST:0.5:24:775 \
RRA:LAST:0.5:288:797 \

[lints]

I have to admit that I don't know why you're getting the error you are with RRDTool. Maybe post that question in a new thread in the general help forum. I don't think it has to do with 32/64 bit counters since RRDTool has no idea what it's plotting; they're just numbers.

This RRD is for a single LUN on your filer and each LUN will get it's own RRD.

[fotan]

In my Data Query [Netapp - Volumes] i do not see LUN operations by each LUN. I see one template "NetApp - LUN - Operations". Data Query [Netapp - Volumes] show me inodes, Files and KB usage, but not LUN ops. May be in my configuration Graph template not correct connect with data query?