iostat data queries works but no graphs drawn

[henpel]

Hi,


I have been fightin few days now with the iostat functionality provided by http://www.markround.com/archives/48-Li ... Cacti.html.

After adding the chcon SE settings i started to receive replies for requests: (if someone looking for it..)

* * * * * root cd /tmp && iostat -xkd 30 2 | sed 's/,/\./g' > io.tmp && chcon -t snmpd_exec_t /tmp/io.tmp && mv -f io.tmp iostat.cache

I'm using cacti version 0.8.8a and i had to modify iostat.xlm

<index_order>ioDescr:ioName:ioIndex</index_order>

-->

<index_order>ioDescr:ioIndex</index_order>

to be able to add data queries for host and create graphs for them.

It was instructed on other post

However, graphs are not drawn even those were successfully created for host.

I increased the debug level to see anything odd from logs but nothing. Everything seems to be working and based on the logs no error information related to iostat etc.

Only logs like:

04/27/2013 01:14:29 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool create /var/www/cacti/rra/veeti_iostat_writes_763.rrd --step 60 DS:iostat_writes:GAUGE:120:0:10000000 DS:iostat_rqsz:GAUGE:120:0:10000000 DS:iostat_util:GAUGE:120:0:100 DS:iostat_await:GAUGE:120:0:10000000 DS:iostat_wkbs:GAUGE:120:0:10000000 DS:iostat_rkbs:GAUGE:120:0:10000000 DS:iostat_qsz:GAUGE:120:0:10000000 DS:iostat_svctm:GAUGE:120:0:10000000 DS:iostat_reads:GAUGE:120:0:10000000 DS:iostat_wrqms:GAUGE:120:0:10000000 DS:iostat_rrqms:GAUGE:120:0:10000000 RRA::0.5:1:525600 RRA:AVERAGE:0.5:1:500 RRA:AVERAGE:0.5:1:600 RRA:AVERAGE:0.5:6:700 RRA:AVERAGE:0.5:24:775 RRA:AVERAGE:0.5:288:797 RRA:MAX:0.5:1:500 RRA:MAX:0.5:1:600 RRA:MAX:0.5:6:700 RRA:MAX:0.5:24:775 RRA:MAX:0.5:288:797

I can see only icon when i'm trying to view iostat graphs.

Everything else works fine.

What to do next?

# grep -i err cacti.log
04/27/2013 01:17:02 PM - CMDPHP: Poller[0] Host[7] DS[248] SNMP: v2: 1.1.0.8, dsname: errors_out, oid: .1.3.6.1.2.1.2.2.1.20.102, output: 0

But it's not related.

Data queries are working:

Data Query Debug Information
+ Running data query [12].
+ Found type = '3' [SNMP Query].
+ Found data query XML file at '/var/www/cacti/resource/snmp_queries/iostat.xml'
+ XML file parsed ok.
+ <oid_num_indexes> missing in XML file, 'Index Count Changed' emulated by counting oid_index entries
+ Executing SNMP walk for list of indexes @ '.1.3.6.1.3.1' Index Count: 13
+ Index found at OID: '1.3.6.1.3.1.1.1' value: '1'
+ Index found at OID: '1.3.6.1.3.1.2.1' value: 'sda'
+ Index found at OID: '1.3.6.1.3.1.3.1' value: '0.00'
+ Index found at OID: '1.3.6.1.3.1.4.1' value: '13.53'
+ Index found at OID: '1.3.6.1.3.1.5.1' value: '0.00'
+ Index found at OID: '1.3.6.1.3.1.6.1' value: '10.60'
+ Index found at OID: '1.3.6.1.3.1.7.1' value: '0.00'
+ Index found at OID: '1.3.6.1.3.1.8.1' value: '96.53'
+ Index found at OID: '1.3.6.1.3.1.9.1' value: '18.21'
+ Index found at OID: '1.3.6.1.3.1.10.1' value: '0.00'
+ Index found at OID: '1.3.6.1.3.1.11.1' value: '0.29'
+ Index found at OID: '1.3.6.1.3.1.12.1' value: '0.02'
+ Index found at OID: '1.3.6.1.3.1.13.1' value: '0.02'
+ Located input field 'ioIndex' [walk]
+ Executing SNMP walk for data @ '.1.3.6.1.3.1.1'
+ Found item [ioIndex='1'] index: 1 [from value]
+ Located input field 'ioDescr' [walk]
+ Executing SNMP walk for data @ '.1.3.6.1.3.1.2'
+ Found item [ioDescr='sda'] index: 1 [from value]

[henpel]

It could be SElinux again

#tail /var/log/messages
Apr 27 14:13:33 kakti setroubleshoot: SELinux is preventing the rrdtool from using potentially mislabeled files ./3830d5c3ddfd5cd38a049b759396e72e-x86-64.cache-2 (var_t). For complete SELinux messages. run sealert -l 35ccdfe4-dde4-4f5c-98ac-5c9e43d0c3ad
Apr 27 14:13:33 kakti setroubleshoot: SELinux is preventing the rrdtool from using potentially mislabeled files /var/cache/fontconfig/3830d5c3ddfd5cd38a049b759396e72e-x86-64.cache-2 (var_t). For complete SELinux messages. run sealert -l b443730a-0ddd-48b7-992b-e0e1204e2249


# tail /var/log/httpd/error_log
ERROR: opening '/var/www/cacti/rra/veeti_iostat_writes_764.rrd': No such file or directory
ERROR: opening '/var/www/cacti/rra/veeti_iostat_writes_763.rrd': No such file or directory
ERROR: opening '/var/www/cacti/rra/veeti_iostat_writes_765.rrd': No such file or directory
ERROR: opening '/var/www/cacti/rra/veeti_iostat_writes_760.rrd': No such file or directory

[henpel]

Not a security linux problem, i were able to fix context settings for weathermap + fontconfig but no help. I disabled the SE linux to be absolutely sure it's not the case.

And this is still what i get:

ERROR: opening '/var/www/cacti/rra/veeti_iostat_writes_764.rrd': No such file or directory

ERROR: opening '/var/www/cacti/rra/veeti_iostat_writes_764.rrd': No such file or directory
ERROR: opening '/var/www/cacti/rra/veeti_iostat_writes_763.rrd': No such file or directory
ERROR: opening '/var/www/cacti/rra/veeti_iostat_writes_765.rrd': No such file or directory
ERROR: opening '/var/www/cacti/rra/veeti_iostat_writes_760.rrd': No such file or directory.

[henpel]

Should there be these duplicate entries in graph template?

<cacti>
<hash_00002429a71dd4ed0f0f09b4f498efd2aee7d5>
<name>iostat - Bytes/sec</name>
<graph>
<t_title></t_title>
<title>|host_hostname| - iostat - Bytes/sec - |query_ioDescr|</title>
<t_image_format_id></t_image_format_id>
<image_format_id>1</image_format_id>
<t_height></t_height>
<height>120</height>
<t_width></t_width>

etc.?

[henpel]

I re-imported graph and data templates and no change.

If i look settings of those, everything seems to be ok.

Debug level logging is on and i were not able to find any glue from cacti.log.

I tried this manually:

# /usr/bin/rrdtool create /var/www/cacti/rra/veeti_iostat_writes_762.rrd --step 60 DS:iostat_writes:GAUGE:120:0:10000000 DS:iostat_rqsz:GAUGE:120:0:10000000 DS:iostat_util:GAUGE:120:0:100 DS:iostat_await:GAUGE:120:0:10000000 DS:iostat_wkbs:GAUGE:120:0:10000000 DS:iostat_rkbs:GAUGE:120:0:10000000 DS:iostat_qsz:GAUGE:120:0:10000000 DS:iostat_svctm:GAUGE:120:0:10000000 DS:iostat_reads:GAUGE:120:0:10000000 DS:iostat_wrqms:GAUGE:120:0:10000000 DS:iostat_rrqms:GAUGE:120:0:10000000 RRA::0.5:1:525600 RRA:AVERAGE:0.5:1:500 RRA:AVERAGE:0.5:1:600 RRA:AVERAGE:0.5:6:700 RRA:AVERAGE:0.5:24:775 RRA:AVERAGE:0.5:288:797 RRA:MAX:0.5:1:500 RRA:MAX:0.5:1:600 RRA:MAX:0.5:6:700 RRA:MAX:0.5:24:775 RRA:MAX:0.5:288:797

and it returns ERROR: Unrecognized consolidation function ?

[gandalf]

RRA::0.5:1:525600
this one breaks your command. Please have a look at the 1 min RRA.
BTB: The complete set od RRA settings looks a bit weird. Mechanics of RRA settings have been extensively discussed every now and then in the forums, please search if you're unsure
R.

[henpel]

\0/

i found this: http://forums.cacti.net/about30759.html

And there was weird spelling with one RRA entry, i removed it and now it seems to be working!!

Thanks gandalf for that previous post!

What a mess

[henpel]

RRA::0.5:1:525600
this one breaks your command. Please have a look at the 1 min RRA.
BTB: The complete set od RRA settings looks a bit weird. Mechanics of RRA settings have been extensively discussed every now and then in the forums, please search if you're unsure
R.

Yes it was, your support is gold!

[henpel]

Just to be sure before removing it, i took dump from rra table to get it back if it fails.. but it did'nt !

[henpel]

Something went broken?

I'm trying to apply these iostat graphs to other hosts too and now i got another problem.

1. Create graphs for your new devices.
2. I select Host: myhost and Graph Types: iostat - utilization from drop down menus. No i can see devices, sdc0 and sda. I select the sda checkbox.
3. Create

I get this:

Created graph:
ERROR: no Data Source associated. Check Template

If i browse that template, i cannot see anything odd.

iostat graphs are working with the original host i added yesterday.

Is there some sanity check that everything is in place?

Queries are working.

If i look host i tried to apply iostat graphs from graph management, there is empty rows in last page where
Graph Title: ID: Template Name: Size: columns are empty.

[gandalf]

Please examine those weird graphs. You may want to post a screenshot of any of them.
R.

[henpel]

Hi,

I dont know how to continue the investigation.

I tried to add queue size graph and it says:

Created graph:
ERROR: no Data Source associated. Check Template

I added few screenshots as attachment.

[gandalf]

From empty-graphs.jpg, please use any "good" graph. Then, please replace the number in the URL by 800!
This is the number of the first "failing" graph. Post a screenshot of the result
R.