Graph disappeared

[stdufour]

Hi,

I had graphs problems with some hosts: some parts were missing at some moment.
I went to the troubleshoot link: http://docs.cacti.net/manual:087:4_help.2_debugging

I saw that I had a double crontab issue, I cleaned it and set it to cacti user.
I then also saw that I had wrong RRD ownership, also fixed.

Now my graphs are looking good but some are not drawn anymore.
rrdtool info showed me that data was Nan.

I tried to delete the rrd and rebuild poller cache. I noted that it was recreating file with root permissions, had to refix it but still no information...

This data is collected via an snmp query with an xml file (GET SNMP INDEXED), this file hasn't changed since the bug fixes and I can get values if I use another tool (iReasoning MIB browser).

Any idea were what to do?

Stéphan

[TheWitness]

Read the directions in the second link on my signature.

[stdufour]

Check Cacti Log File
It had one timeout error for one host, I upgraded the SNMP agent and does not appear anymore

Check Basic Data Gathering
For the host i'm missing graphs, it is a chassis with multiple cards.
In my xml file, I have an oid but you have append the number of the slot to get the data for that slot.

<CardCurrentCPU>
<name>Current CPU</name>
<method>walk</method>
<source>value</source>
<direction>output</direction>
<oid>.1.3.6.1.4.1.7569.1.2.1.28.1.6</oid>
</CardCurrentCPU>

If I run, the oid . number of the card it is ok, I get a value.

Check Cacti's Poller
Values are ok

Check RRD File Update
Ok

Check MySQL Update
I find no message related to sql in the log....

Check RRD File Ownership
Good

Check RRD File Numbers
rrdtool fetch full of -nan for my graphs not displayed...

[TheWitness]

Sounds like a Data Query. When you run a verbose query, are you getting rows back? Do they make sense? Have you Created the Graphs for each row of the Data Query?

[stdufour]

Hi,

I have cards only in slot 1,3,4,5 and 6 so yes, what I get is perfectly normal.

snmpwalk -c XXX -v 2c IPadresse .1.3.6.1.4.1.7569.1.2.1.28.1.6
SNMPv2-SMI::enterprises.7569.1.2.1.28.1.6.1 = Gauge32: 35
SNMPv2-SMI::enterprises.7569.1.2.1.28.1.6.2 = Gauge32: 0
SNMPv2-SMI::enterprises.7569.1.2.1.28.1.6.3 = Gauge32: 4
SNMPv2-SMI::enterprises.7569.1.2.1.28.1.6.4 = Gauge32: 4
SNMPv2-SMI::enterprises.7569.1.2.1.28.1.6.5 = Gauge32: 1
SNMPv2-SMI::enterprises.7569.1.2.1.28.1.6.6 = Gauge32: 8
SNMPv2-SMI::enterprises.7569.1.2.1.28.1.6.7 = Gauge32: 0
SNMPv2-SMI::enterprises.7569.1.2.1.28.1.6.8 = Gauge32: 0
SNMPv2-SMI::enterprises.7569.1.2.1.28.1.6.9 = Gauge32: 0
SNMPv2-SMI::enterprises.7569.1.2.1.28.1.6.10 = Gauge32: 0
SNMPv2-SMI::enterprises.7569.1.2.1.28.1.6.11 = Gauge32: 0
SNMPv2-SMI::enterprises.7569.1.2.1.28.1.6.12 = Gauge32: 0
SNMPv2-SMI::enterprises.7569.1.2.1.28.1.6.13 = Gauge32: 0
SNMPv2-SMI::enterprises.7569.1.2.1.28.1.6.14 = Gauge32: 0

I also trie to browse the sql database (need to repeat several time since value often changes).
mysql> SELECT * FROM poller_output;
+---------------+------------------+---------------------+---------+
| local_data_id | rrd_name | time | output |
+---------------+------------------+---------------------+---------+
| 1643 | CardMaxPorts | 2011-11-29 16:43:16 | 0 |
| 1643 | CardAvgPorts | 2011-11-29 16:43:16 | 0 |
| 1643 | CardAvgCPU | 2011-11-29 16:43:16 | 3 |
| 837 | traffic_in | 2011-11-29 16:43:17 | 3765070 |
| 1643 | CardFreeRAM | 2011-11-29 16:43:16 | 66 |
| 1643 | CardCurrentPorts | 2011-11-29 16:43:16 | 0 |
+---------------+------------------+---------------------+---------+

Seems correctly updated...

[gandalf]

To sum up:
- data is fetched correctly
- data is written to mysql correctly
- data in rrd file is not present

From this, we deduce that the rrdfile update seems to go wrong. How did you test that it was ok? How did you define the rrd file (an rrdtool info might help)
R.

[TheWitness]

It could also be that the mapping in the data query is off, or you are using name value pair for a single item, or not returning complete data, for example, the rrd expects 5 items and you are only outputing 2.

[stdufour]

Hi,

I did not create these graphs, they were done by someone else who's not working here anymore.

For example, I go to graph management and chose a graph for this host who does not appear like CPU Card 3 Usage
I switch on debug and have:

DEF:a="/var/lib/cacti/rra/cms9000_-_sys2_cardcurrentcpu_1638.rrd":CardCurrentCPU:AVERAGE \
DEF:b="/var/lib/cacti/rra/cms9000_-_sys2_cardcurrentcpu_1638.rrd":CardAvgCPU:AVERAGE \
DEF:c="/var/lib/cacti/rra/cms9000_-_sys2_cardcurrentcpu_1638.rrd":CardMaxCPU:AVERAGE \

Check: it belongs to cacti user and has been updated.

If It try to find it in debug log file I get no error when launching it from cactiuser.
/usr/bin/rrdtool update /var/lib/cacti/rra/cms9000_-_sys2_cardcurrentcpu_1638.rrd --template CardMaxCPU:CardAvgCPU:CardCurrentCPU 1323093901:11:2:2


RRDtool info gives:
filename = "/var/lib/cacti/rra/cms9000_-_sys2_cardcurrentcpu_1638.rrd"
rrd_version = "0003"
step = 10
last_update = 1323093826
header_size = 18832
ds[CardCurrentCPU].index = 0
ds[CardCurrentCPU].type = "GAUGE"
ds[CardCurrentCPU].minimal_heartbeat = 10
ds[CardCurrentCPU].min = 0.0000000000e+00
ds[CardCurrentCPU].max = 1.0000000000e+02
ds[CardCurrentCPU].last_ds = "3"
ds[CardCurrentCPU].value = NaN
ds[CardCurrentCPU].unknown_sec = 6
ds[CardMaxCPU].index = 1
ds[CardMaxCPU].type = "GAUGE"
ds[CardMaxCPU].minimal_heartbeat = 10
ds[CardMaxCPU].min = 0.0000000000e+00
ds[CardMaxCPU].max = 1.0000000000e+02
ds[CardMaxCPU].last_ds = "11"
ds[CardMaxCPU].value = NaN
ds[CardMaxCPU].unknown_sec = 6
ds[CardAvgCPU].index = 2
ds[CardAvgCPU].type = "GAUGE"
ds[CardAvgCPU].minimal_heartbeat = 10
ds[CardAvgCPU].min = 0.0000000000e+00
ds[CardAvgCPU].max = 1.0000000000e+02
ds[CardAvgCPU].last_ds = "2"
ds[CardAvgCPU].value = NaN
ds[CardAvgCPU].unknown_sec = 6
ds[CardCurrentPorts].index = 3
ds[CardCurrentPorts].type = "GAUGE"
ds[CardCurrentPorts].minimal_heartbeat = 10
ds[CardCurrentPorts].min = 0.0000000000e+00
ds[CardCurrentPorts].max = 3.0000000000e+03
ds[CardCurrentPorts].last_ds = "U"
ds[CardCurrentPorts].value = NaN
ds[CardCurrentPorts].unknown_sec = 6
ds[CardMaxPorts].index = 4
ds[CardMaxPorts].type = "GAUGE"
ds[CardMaxPorts].minimal_heartbeat = 10
ds[CardMaxPorts].min = 0.0000000000e+00
ds[CardMaxPorts].max = 3.0000000000e+03
ds[CardMaxPorts].last_ds = "U"
ds[CardMaxPorts].value = NaN
ds[CardMaxPorts].unknown_sec = 6
ds[CardAvgPorts].index = 5
ds[CardAvgPorts].type = "GAUGE"
ds[CardAvgPorts].minimal_heartbeat = 10
ds[CardAvgPorts].min = 0.0000000000e+00
ds[CardAvgPorts].max = 3.0000000000e+03
ds[CardAvgPorts].last_ds = "U"
ds[CardAvgPorts].value = NaN
ds[CardAvgPorts].unknown_sec = 6
ds[CardFreeRAM].index = 6
ds[CardFreeRAM].type = "GAUGE"
ds[CardFreeRAM].minimal_heartbeat = 10
ds[CardFreeRAM].min = 0.0000000000e+00
ds[CardFreeRAM].max = 1.0000000000e+09

What is strange is sometimes, it seems to be executed twice at the same time:

12/05/2011 03:12:02 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /var/lib/cacti/rra/cms9000_-_sys2_cardcurrentcpu_1638.rrd --template CardCurrentCPU:CardMaxCPU:CardAvgCPU 1323094321:2:13:2
12/05/2011 03:12:17 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /var/lib/cacti/rra/cms9000_-_sys2_cardcurrentcpu_1638.rrd --template CardCurrentCPU:CardMaxCPU:CardAvgCPU 1323094336:2:13:2
12/05/2011 03:12:17 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /var/lib/cacti/rra/cms9000_-_sys2_cardcurrentcpu_1638.rrd --template CardCurrentCPU:CardMaxCPU:CardAvgCPU 1323094336:2:13:2

[TheWitness]

That would appear to be a multiple cron issue. In other words more than one entry exists for one to many user accounts.

[stdufour]

sudo find /etc -exec grep "poller.php" '{}' \; -print

# */5 * * * * www-data /usr/bin/php -q /var/www/html/cacti/poller.php > /dev/null 2>&1
/etc/cron.d/cacti
grep: /etc/blkid.tab: No such file or directory
* * * * * www-data /usr/bin/php -q /usr/share/cacti/site/poller.php > /var/log/cacti/cacti.log 2>&1
/etc/crontab

The cacti user is not "cacti" but www-data as shown in /etc/crontab.

Anyway, I commented the line in case of...

Is it possible that 1638.rrd appears two time in some config file which will cause it to be called twice ?

[stdufour]

I had a look and:

mysql> select * from poller_item where local_data_id='1638';
+---------------+-----------+---------+--------+---------------+----------------+--------------+---------------+---------------+--------------------+----------------------+--------------------+--------------+-----------+--------------+----------------+-----------------------------------------------------------+---------+----------+---------------+----------------------------------+------+------+
| local_data_id | poller_id | host_id | action | hostname | snmp_community | snmp_version | snmp_username | snmp_password | snmp_auth_protocol | snmp_priv_passphrase | snmp_priv_protocol | snmp_context | snmp_port | snmp_timeout | rrd_name | rrd_path | rrd_num | rrd_step | rrd_next_step | arg1 | arg2 | arg3 |
+---------------+-----------+---------+--------+---------------+----------------+--------------+---------------+---------------+--------------------+----------------------+--------------------+--------------+-----------+--------------+----------------+-----------------------------------------------------------+---------+----------+---------------+----------------------------------+------+------+
| 1638 | 0 | 11 | 0 | 10.102.32.160 | XXX | 2 | | | | | | | 161 | 500 | CardAvgCPU | /var/lib/cacti/rra/cms9000_-_sys2_cardcurrentcpu_1638.rrd | 3 | 10 | -5 | .1.3.6.1.4.1.7569.1.2.1.28.1.5.3 | | |
| 1638 | 0 | 11 | 0 | 10.102.32.160 | XXX | 2 | | | | | | | 161 | 500 | CardCurrentCPU | /var/lib/cacti/rra/cms9000_-_sys2_cardcurrentcpu_1638.rrd | 3 | 10 | -5 | .1.3.6.1.4.1.7569.1.2.1.28.1.6.3 | | |
| 1638 | 0 | 11 | 0 | 10.102.32.160 | XXX | 2 | | | | | | | 161 | 500 | CardMaxCPU | /var/lib/cacti/rra/cms9000_-_sys2_cardcurrentcpu_1638.rrd | 3 | 10 | -5 | .1.3.6.1.4.1.7569.1.2.1.28.1.4.3 | | |
+---------------+-----------+---------+--------+---------------+----------------+--------------+---------------+---------------+--------------------+----------------------+--------------------+--------------+-----------+--------------+----------------+-----------------------------------------------------------+---------+----------+---------------+----------------------------------+------+------+

I found the entry 3 times in the Database... Is this normal?

----edit:
Ok the lines are different, it polls for 3 counters...

Even If I try to reload several times the script the only process I can get are:
ps -eaf | grep php

www-data 27566 27565 0 15:57 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/share/cacti/site/poller.php > /var/log/cacti/cacti.log 2>&1
www-data 27567 27566 23 15:57 ? 00:00:11 /usr/bin/php -q /usr/share/cacti/site/poller.php
www-data 27747 1 9 15:57 ? 00:00:00 /usr/bin/php -q /usr/share/cacti/site/cmd.php 0 58
www-data 27751 27747 0 15:57 ? 00:00:00 sh -c /usr/bin/php -q /usr/share/cacti/site/script_server.php cmd
www-data 27752 27751 3 15:57 ? 00:00:00 /usr/bin/php -q /usr/share/cacti/site/script_server.php cmd
root 27755 26797 0 15:57 pts/0 00:00:00 grep --color=auto php

[stdufour]

SOLVED !

Heartbeat of data sources were set to 10, but I had to increase the poller value...

I just changed them to 600 and recreated my graphs.

Hope it will help other people.

[TheWitness]

Make sure you update your Data Template's as well.