Trouble setting up data query - .rrd file not being created

[Benji]

Greetings, most helpful forum folk.

=== Problem Overview ===

I'm trying to set up a new graph using an snmp data query.
As near as I can tell, I've followed the walkthrough at:

http://docs.cacti.net/manual:087:3a_adv ... alkthrough

fairly faithfully,
with two separate OID/trees,
and it just doesn't seem to work.
Here's a summary of what I've done:

• 1. verified OIDs using snmpwalk
  (we're using the snmp extended OIDs under
  .1.3.6.1.4.1.2021.200
  and
  .1.3.6.1.4.1.2021.201)

• 2. created .xml file in <path_cacti>/resource/snmp_queries/

• 3. created basic data query

• 4. created a data template

• 5. created graph template

• 6. completed the data query (added Associated Graph Template)

• 7. added graphs to the device/host

At this stage, I understand there's supposed to be Shiny Graphs,
but there isn't.


=== Additional Debugging Info ===

• - cacti.log shows no polling for these data sources
  (I've left it running overnight - no joy)

• - Looking at the Data Source screen, under the Custom Data section:
  "Output Type ID" and "Index Type" both show
  "Data query data sources must be created through New Graphs."

• - looking at the Graph Management screen, I can find / click on the details for the graphs,
  but the debug output says:
  ERROR: opening '/var/www/cacti/rra/ecourse2dev_cache_service_8281.rrd': No such file or directory

To me, it seems that cacti just hasn't done the initial 'rrdtool / create graph' step,
but I'm hoping it's something simple I've missed
(we haven't done a lot of custom work with cacti here.)

It's also possible that I've missed something in setting up the custom SNMP output from the target device,
but it looks right. Here's the output of snmpwalk:

snmpwalk -v 1 <host> -c <secret> .1.3.6.1.4.1.2021.201 -On
.1.3.6.1.4.1.2021.201.1 = INTEGER: 1
.1.3.6.1.4.1.2021.201.1.1 = INTEGER: 1
.1.3.6.1.4.1.2021.201.2.1 = STRING: "cache_service"
.1.3.6.1.4.1.2021.201.3.1 = INTEGER: 39

Thanks in advance for your time,
--Benji

p.s: I've attached the snmp query.xml file as .txt, as seems to be necessary.

p.p.s: I'm aware that for this example/output, as there's only one item I'm interested in, it'd be better to use the 'SNMP - Generic OID Template' method, but I need to get this method working for other data sources, and I figured that one output was simpler to debug.

[gandalf]

Please see 2nd link of my sig for basic debugging. I suppose it may be an issue on downed host detection
Reinhard

[Benji]

Hey there.
Thanks for your response.


I've already done some of this error checking,
but it certainly never hurts to go through things step-by-step!


Stepping through debugging instructions sourced from:
http://docs.cacti.net/manual:087:4_help ... #debugging


=============== debugging begins ===============

==Checking failures for the host in the log:==
grep "Host\[452\]" /var/www/cacti/log/cacti.log
...No failures for that host.


==Checking basic data gathering:==

snmpwalk -v 1 <hostname> -c <secret> .1.3.6.1.4.1.2021.201 -On
output:
.1.3.6.1.4.1.2021.201.1 = INTEGER: 1
.1.3.6.1.4.1.2021.201.1.1 = INTEGER: 1
.1.3.6.1.4.1.2021.201.2.1 = STRING: "cache_service"
.1.3.6.1.4.1.2021.201.3.1 = INTEGER: 39
(Seems ok)


==Check Cacti's Poller:==

- cacti settings indicate that cmd.php is enabled,
and it's definitely activating every 5 mins (log file keeps growing)

- cat /etc/cron.d/cacti:
*/5 * * * * cacti php /var/www/cacti/poller.php &>/var/www/cacti/log/poller.out 2>/var/www/cacti/log/poller.err

- set cacti log to debug for one poller cycle. Done.

results for grep "Host\[452\]" in that time period indicate that all the other data sources are working correctly,
however, grepping for the hostname (e.g 'my.host.name') returns:

06/19/2009 10:07:03 AM - CMDPHP: Poller[0] DEBUG: SQL Exec: "update host set status = '3', status_event_count = '0', status_fail_date = '2009-06-13 21:22:14', status_rec_date = '2009-05-12 21:51:41', status_last_error = 'Host did not respond to SNMP', min_time = '0.00000', max_time = '10003.25000', cur_time = '41.99', avg_time = '5.8641935965991', total_polls = '170913', failed_polls = '1781', availability = '98.957949366052' where hostname = 'my.host.name'"


==<skipping "Check MySQL update" for now>==


==Check RRD file Update:==

- rrdtool definitely doesn't mention the DS / rrd we're interested in, and it's not in the poller cache either.


==Check ownership:==

- There's no .rrd file, hence no ownership problems.


==Misc:==

select count(*) from poller_output;
returns '0'.
start time: 06/19/2009 10:04:35 AM
end time: 06/19/2009 10:08:44 AM


=============== debugging concluded ===============

...I see the SQL Exec: message "Host did not respond to SNMP",
but:
- the host is up,
- cacti.log also returns:
06/19/2009 10:07:03 AM - CMDPHP: Poller[0] Host[452] SNMP: Host responded to SNMP
- and the other (standard) graphs are being updated/created as expected
(e.g ucd/net CPU 1min avg, ucd/net - Memory free)

...So I don't know what to make of the SQL Exec message.
Could it be something Stuck from 2009-05-12 21:51:41,
or is it something Normal?

Any further clues?

Thanks for your time.
--Benji

[gandalf]

results for grep "Host\[452\]" in that time period indicate that all the other data sources are working correctly,
however, grepping for the hostname (e.g 'my.host.name') returns:

06/19/2009 10:07:03 AM - CMDPHP: Poller[0] DEBUG: SQL Exec: "update host set status = '3', status_event_count = '0', status_fail_date = '2009-06-13 21:22:14', status_rec_date = '2009-05-12 21:51:41', status_last_error = 'Host did not respond to SNMP', min_time = '0.00000', max_time = '10003.25000', cur_time = '41.99', avg_time = '5.8641935965991', total_polls = '170913', failed_polls = '1781', availability = '98.957949366052' where hostname = 'my.host.name'"


==<skipping "Check MySQL update" for now>==


==Check RRD file Update:==

- rrdtool definitely doesn't mention the DS / rrd we're interested in, and it's not in the poller cache either.

The SQL message does not indicate a problem. It only updates the per host statistics. The date shown there indicates, when the host was detected as down for the last time.
The fact that there is no rrd update found is bad. In fact, anything that should be pollerd for that host should show up.
System Utilities -> View Poller Cache shows anything for that host? Do you find those entries in the log?
Reinhard

[Benji]

The SQL message does not indicate a problem. It only updates the per host statistics. The date shown there indicates, when the host was detected as down for the last time.

I see - thanks for that.

The fact that there is no rrd update found is bad. In fact, anything that should be pollerd for that host should show up.
System Utilities -> View Poller Cache shows anything for that host?

The poller cache does show the other checks for the host, e.g:

• Load Average - 1 Minute

• Load Average - 5 Minute

• Partition - /

Do you find those entries in the log?
Reinhard

Yes, the check that are in the poller cache do appear in the logs,
and are successfully graphing


...I can see the Data Source for this host/service,
and in 'data source debug mode', it prints the statement that would be used to create the .rrd file

• /usr/bin/rrdtool create \
  /var/www/cacti/rra/my.host.name_cache_service_8281.rrd \
  --step 300 \
  DS:cache_service:GAUGE:600:0:U \
  RRA:AVERAGE:0.5:1:1440 \
  ...and so on.

...it's just that the file hasn't been created, and wouldn't be updated if it was.

Thanks for your continued help.

--Benji

[gandalf]

To sum up:
1. The Data Source is in place. What about the Graph?
2. The Poller Cache does NOT show the related entry
3. The rrd file is hence missing
4. And of course no data polled for that Data Source.

Item #2 is the root cause.
Please delete the Data Source (along with the associated graph if any) and try to re-generate the graph.

If that again fails, we may have a new bug
Reinhard

[Benji]

Your summary is correct.

1. The Data Source is in place. What about the Graph?
2. The Poller Cache does NOT show the related entry
3. The rrd file is hence missing
4. And of course no data polled for that Data Source.

...1) The graph is listed as there,
you can click on it in the graph management screen,
but there's no created graph, and the debug 'rrdtool graph' statement
complains about missing .rrd file.

Item #2 is the root cause.
Please delete the Data Source (along with the associated graph if any) and try to re-generate the graph.

This I now do:

Deleted data source,
which deleted the associated graph.


Created a new graph:

go to:
/cacti/graphs_new.php

Select host.

Scroll down to the appropriate 'data query',
tick the checkbox,
hit 'create'.

Results:
- It displays
"+ Created graph: ecourse2dev - webct cache service"
at the top of the (refreshed) screen afterwards,

- The option to tick the same checkbox still exists.
This does not gel with my experience with other graphs,
which is that this option should now be greyed out.

- New graph (id 6578) has been created (under 'graph management' screen)

- New data source (id 8283) has been created (under 'data sources' screen)

- Editing the details for graph 6578 shows the correct data source

- Editing the details for data source 8283 show
"Output Type ID" "Data query data sources must be created through New Graphs."
"Index Type" "Data query data sources must be created through New Graphs."

- Turning on data source debug mode for DS 8283 displays 'rrdtool create' statement, pointing to a file that doesn't exist
(/var/www/cacti/rra/<host>_cache_service_8283.rrd)

- Turning on graph debug mode for the graph shows 'rrdtool graph' statement, and error:
ERROR: opening '/var/www/cacti/rra/<host>_cache_service_8283.rrd': No such file or directory


...This fails.
Also, rebuilding the poller cache fails.

We're using cacti 0.8.7b,
but I can't see any bugs listed at
http://cacti.net/changelog.php
that have been fixed / would be rectified by upgrading.

...If you can confirm that it sounds like I've set things up the right way,
but that it's still failing,
I'm happy to submit a bug report.

Cheers,
--Benji

[gandalf]

Please again delete data sources/graphs. Then visit the target from the "Devices" menu. Select "Create Graphs for this host" and create graphs.
Verify the same way as you did above.
Reinhard

[pribeiro]

I'm experiencing the same problem with cacti-0.8.7b-r3 (Gentoo package) while trying to monitor some servers using IPMI (using the template from http://www.eric-a-hall.com/software/cac ... l-sensors/ ).

The RRD files aren't created, even after deleting the graphs and datasources and creating the graphs again.

Running the "rrdtool create" command shown in the datasource page (in datasource debug mode) with the same user as the web server (apache), the RRD file is created without any error or warning.

When should the "rrdtool create" be executed by the system? at the time the graph is created? when I go to the datasource page and press Save?

TIA.

[Benji]

Please again delete data sources/graphs. Then visit the target from the "Devices" menu. Select "Create Graphs for this host" and create graphs.
Verify the same way as you did above.
Reinhard

- deleted data source, which deleted associated graph

- visited 'devices' menu
- selected host
- relevant data query ('cache_service') is listed at the bottom of the page, verbose query indicates correct functionality

- visited 'create graphs for this host'
- ticked the 'cache service' item, and 'create'

- This took me to /cacti/graphs_new.php, filled out graph 'title' and data source 'name'
- hit 'create'

- this reloaded the 'devices' page for the host, with the addition of the line:
"+ Created graph: <host> - webct cache service"
at the top of the page

- I still have the option to tick the relevant data query ('cache_service') at the bottom of the page.

Results:

- graph id '6604' is available in 'graph management' page
- graph debug mode for this graph reports 'ERROR: opening '/var/www/cacti/rra/<host>_cache_service_8309.rrd': No such file or directory
'

- data source '8309' has been created
- data source items 'output type id' and 'index type' both indicate 'Data query data sources must be created through New Graphs.'
- data source debug mode indicates the 'rrdtool create' statement that references the correct (ie. "same as indicated in graph debug screen") .rrd file

- poller cache does shows other results for this host,
does not show any results pertinent to 'cache_service'

Again, thanks for your help.

Please indicate how you think I should proceed from this point...I'm a bit stuck.

--Benji

[gandalf]

I fear that this requires online access to your system. At least I can't get any clue from remote, sorry
Reinhard

[jroliv]

Hi,

I have been having this problem too.

As workaround I deleted the host and created it again, so the rra are being normally created.

[pirxthepilot]

HI Benji,

I was wondering if you were able to eventually resolve this issue. I am having the exact same problems as you - and I mean exactly.

Thanks!

[chvp]

- Im also experiencing this problem!

[gandalf]

I'm sorry to be frustrating.
But adding a "me too" without new information or the option to debug remotely won't bring us nearer to a solution. I see that it affects more people. But I need INFORMATION
R.