Cacti not creating data sources or rrd files when it should?

[Azendale]

I have some indexed SNMP queries. (I've attached the XML definitions.) I have these associated with a host templete. When I add a device/host with that template, and then go to the "Create graphs for this host" link, I can see in the tables that the indexed queries are running and are successful in returning data. Good.

But when I try to create graphs, the graphs are always replaced by an image that says no .rrd file was found. Checking in the filesystem of the server, there is no .rrd file, so it was never created.

I'm really not sure how to debug this. I tried turning up the debug to devel level, and then adding graphs to the host. I then looked at the logs and found and entry that was something like this:
(I'm paraphrasing because the page refreshed before I could copy it)
CACTI2RRD: (some RRDtool creation command) /some/path/to/new.rrd DS[1234] (Data source not found)

When I click on the link to go to the datasource it says it can't find, the Cacti WebUI says it doesn't exist.

So it seems that Cacti is not creating data sources both in the MySQL database and as .rrd files on the file system -- but only for some things. (Other graphs on different data queries are working just fine and the datasources and .rrd files create just fine.)

Why is this happening? What information do I need to collect to help debug this?

[netniV]

Check that your folder has the correct permissions to allow both the poller user and website user to create / maintain the rrd files.

[Azendale]

I don't think that is the problem, since other graphs and data sources can be created. But here's what I get trying to look at that:

Code: Select all

spruce:/usr/share/cacti/site# ls -lah |grep rra
lrwxrwxrwx  1 root root   18 Apr 26 04:25 rra -> /var/lib/cacti/rra
spruce:/usr/share/cacti/site# ls -lah /var/lib/cacti/ | grep rra
drwxr-xr-x  2 www-data www-data 692K Jul 12 10:50 rra
spruce:/usr/share/cacti/site# cat /etc/cron
cron.d/       cron.daily/   cron.hourly/  cron.monthly/ crontab       cron.weekly/  
spruce:/usr/share/cacti/site# cat /etc/cron.d
cron.d/     cron.daily/ 
spruce:/usr/share/cacti/site# cat /etc/cron.d/
cacti          certbot        php            php5           php5.dpkg-old  .placeholder   
spruce:/usr/share/cacti/site# cat /etc/cron.d/cacti 
MAILTO=root
*/5 * * * * www-data php /usr/share/cacti/site/poller.php 2>&1 >/dev/null | if [ -f /usr/bin/ts ] ; then ts ; else tee ; fi >> /var/log/cacti/poller-error.log

This is on a Debian server where the server normally would run PHP as www-data, at least to my understanding.

Also, I'm attaching some exports of what I have, in case there is something wrong with them (which I highly suspect; I just don't know what in particular is wrong..)

Well, at least the host template export. I can't seem to attach the graph templates as they are being rejected by the forum as "The upload was rejected because the uploaded file was identified as a possible attack vector."

[netniV]

Rename them to .txt and it should be OK, are there any errors in the logs when you attempt to create from these sources?

[Azendale]

I just tried renaming them, and the same "The upload was rejected because the uploaded file was identified as a possible attack vector." happened.

I just tried re-adding the graphs after deleting the old ones that didn't work and were showing the .rrd error. Nothing showed up in the logs related to that device or graphn with it set to "Low - statistics and errors".

[Azendale]

Still having trouble with this, though maybe I found something.

But first, I'm working around the forum upload troubles. See http://users.eoni.com/~azendale/cacti/ for a list of files I was able to export on one of the problematic graphs.

Ok, back to what I noticed:

Here's the datasource for a graph on the device that works, with an indexed query:



Here's one that doesn't:


They are both indexed SNMP queries on the same device. Why does one have index not filled out and then won't let the other two fields be set? Am I on to something here?

(Since I've started this thread, I have tried removing the query and putting it back with a new data source and graph just to get around the issues, and today I noticed what I'm pointing out here after doing all that and not getting anywhere.)

This is a picture of what the problematic query shows on the add graphs page:



This makes me think the query is fine, but maybe there is some problem with the query XML itself?

Working query:
http://users.eoni.com/~azendale/cacti/r ... annels.xml

Non working query:
http://users.eoni.com/~azendale/cacti/r ... ts_cms.xml

What am I missing between these two?

[Pucho]

How does your data source template look like? The one that's not working.

[netniV]

Your problem is quite simple. You've a difference in the index in the oid_index key so it never finds any indexes beneath it. Remove the 1979719681 or maybe add it depend on which one is working and which isn't.

[Azendale]

How does your data source template look like? The one that's not working.

Here you go:

[Azendale]

Your problem is quite simple. You've a difference in the index in the oid_index key so it never finds any indexes beneath it. Remove the 1979719681 or maybe add it depend on which one is working and which isn't.

Now I'm questioning what I know about indexed queries. I thought you wanted to find something you could SNMPwalk to get a list of "rows" in a table. Then you put that in the <oid_index> directive, and how to parse the index in the <out_index_parse> directive. It would then parse the 1979719681 that should be added on to get row one, and other suffixes that should be added to get the other rows. Or at least that's how I thought it worked. So I think the XML WITH the 1979719681 is incorrect. But it is the one that is working!

Here's what happens when I do SNMPwalks with the OIDs from the two XML files:

File that doesn't work, OID from <oid_index>:

Code: Select all

spruce:~# snmpwalk  -Of -c 0myHECK2 -v2c 10.5.6.2 .1.3.6.1.2.1.10.127.1.3.3.1.2
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfCmtsObjects.docsIfCmtsCmStatusTable.docsIfCmtsCmStatusEntry.docsIfCmtsCmStatusMacAddress.1 = STRING: 3c:90:66:8f:a0:3a
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfCmtsObjects.docsIfCmtsCmStatusTable.docsIfCmtsCmStatusEntry.docsIfCmtsCmStatusMacAddress.2 = STRING: 3c:90:66:8f:9f:38
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfCmtsObjects.docsIfCmtsCmStatusTable.docsIfCmtsCmStatusEntry.docsIfCmtsCmStatusMacAddress.3 = STRING: 3c:90:66:8f:9f:44
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfCmtsObjects.docsIfCmtsCmStatusTable.docsIfCmtsCmStatusEntry.docsIfCmtsCmStatusMacAddress.4 = STRING: 3c:90:66:8f:a0:ac
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfCmtsObjects.docsIfCmtsCmStatusTable.docsIfCmtsCmStatusEntry.docsIfCmtsCmStatusMacAddress.5 = STRING: 3c:90:66:8f:9e:96
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfCmtsObjects.docsIfCmtsCmStatusTable.docsIfCmtsCmStatusEntry.docsIfCmtsCmStatusMacAddress.6 = STRING: 3c:90:66:8f:9f:8c
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfCmtsObjects.docsIfCmtsCmStatusTable.docsIfCmtsCmStatusEntry.docsIfCmtsCmStatusMacAddress.7 = STRING: 3c:90:66:8f:9e:a8
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfCmtsObjects.docsIfCmtsCmStatusTable.docsIfCmtsCmStatusEntry.docsIfCmtsCmStatusMacAddress.8 = STRING: 3c:90:66:8f:9f:86
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfCmtsObjects.docsIfCmtsCmStatusTable.docsIfCmtsCmStatusEntry.docsIfCmtsCmStatusMacAddress.9 = STRING: 3c:90:66:8f:a0:64
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfCmtsObjects.docsIfCmtsCmStatusTable.docsIfCmtsCmStatusEntry.docsIfCmtsCmStatusMacAddress.10 = STRING: 3c:90:66:8f:9e:d8
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfCmtsObjects.docsIfCmtsCmStatusTable.docsIfCmtsCmStatusEntry.docsIfCmtsCmStatusMacAddress.11 = STRING: 0:7:11:11:34:d2

File that did work, OID from <oid_index>:

Code: Select all

spruce:~# snmpwalk  -Of -c 0myHECK2 -v2c 10.5.6.2 .1.3.6.1.2.1.10.127.1.1.2.1.1.1979719681
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfBaseObjects.docsIfUpstreamChannelTable.docsIfUpstreamChannelEntry.docsIfUpChannelId.1979719681 = INTEGER: 1

Hm. Only one row. Lets take off the suffix and try again:

Code: Select all

spruce:~# snmpwalk  -Of -c 0myHECK2 -v2c 10.5.6.2 .1.3.6.1.2.1.10.127.1.1.2.1.1
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfBaseObjects.docsIfUpstreamChannelTable.docsIfUpstreamChannelEntry.docsIfUpChannelId.1979719681 = INTEGER: 1
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfBaseObjects.docsIfUpstreamChannelTable.docsIfUpstreamChannelEntry.docsIfUpChannelId.1979719682 = INTEGER: 2
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfBaseObjects.docsIfUpstreamChannelTable.docsIfUpstreamChannelEntry.docsIfUpChannelId.1979719683 = INTEGER: 3
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfBaseObjects.docsIfUpstreamChannelTable.docsIfUpstreamChannelEntry.docsIfUpChannelId.1979719684 = INTEGER: 4
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfBaseObjects.docsIfUpstreamChannelTable.docsIfUpstreamChannelEntry.docsIfUpChannelId.1979719685 = INTEGER: 5
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfBaseObjects.docsIfUpstreamChannelTable.docsIfUpstreamChannelEntry.docsIfUpChannelId.1979719686 = INTEGER: 6
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfBaseObjects.docsIfUpstreamChannelTable.docsIfUpstreamChannelEntry.docsIfUpChannelId.1979719687 = INTEGER: 7
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfBaseObjects.docsIfUpstreamChannelTable.docsIfUpstreamChannelEntry.docsIfUpChannelId.1979719688 = INTEGER: 8

Ok, now that makes sense for what I would expect in the table.

Puzzled, I went to the device page and refreshed the working data query. It now only shows one row. "Fixed" the XML to use 10.5.6.2 .1.3.6.1.2.1.10.127.1.1.2.1.1 now I have the 8 rows I expect when I run the verbose query on the device page. So... is the index a side issue (probably introduced by me trying to find problems, but still working because I didn't re-create my graphs or the SNMP uptime didn't cause a re-index)?

[Pucho]

These two data source templates are using "Upgrade profile 1"?

If you can create other graphs without issues I wouldn't think it's a permission issue. I might be wrong but..

Can't spot anything weird though.

[netniV]

If you change the XML you should probably be running a poller output clear / reindex to ensure that everything is in sync.

[Azendale]

These two data source templates are using "Upgrade profile 1"?

If you can create other graphs without issues I wouldn't think it's a permission issue. I might be wrong but..

Can't spot anything weird though.

Turns out I have two "Upgrade profile 1" profiles. Renamed them to Profile 1 and Profile 2 so I could tell them apart. But the working and the not working query use the same one. (Profile 2).

[Azendale]

If you change the XML you should probably be running a poller output clear / reindex to ensure that everything is in sync.

I did try that, and then re-creating graphs & data sources. It didn't seem to fix it.

At this point, I think I would be open to allowing a login to our server with SSH (send me a public SSH key / IP range to allow) and sharing a screen session to try to find the issue, if that helps.

[netniV]

Send me a message to netniv@cacti.net and we can discuss it privately.