custom indexed snmp data query not updating rrd

thebofh · Post by **thebofh** » Thu May 12, 2005 2:39 pm

I've created a custom data query of type Get SNMP Data (Indexed) for monitoring the memory usage of multiple JVMs on a single device. Everything seems to be working. I've snmpwalked the oid manually and it's all working as expected until I get to creating a graph for a device. The overall process I'm using is:

1) run snmpwalk (from cacti server) as a test
2) create data template[s]
3) create graph template[s]
4) create data query xml file (at the OS level and place in $cactihome/resources/snmp_queries/)
5) create the data query (inside cacti)
6) associate the graph template[s] to the data query
7) associate the data template[s] to the names in the xml file
8) associate the data query to a device
9) run the verbose data query (verify the output is correct)
10) create new graph[s] for that device
11) the poller creates the rrd
12) the poller updates the rrd on subsequent passes
13) the web server dynamically generates graphs as requested

I get to step 11 fine and the rrd is created and the poller is even snmpwalking my oid's everytime it is run. The problem is that the rrd is never updated (step 12) after it is created. What am I missing?

SB

thebofh · Post by **thebofh** » Thu May 12, 2005 6:59 pm

The weird thing is that in the poller log I can see the results of the snmpwalk being inserted into the DB, but there are no 'rrdtool update ...' messages.

Code: Select all

05/12/2005 12:05:04 PM - CACTID: Poller[0] Host[31] SNMP: v1: hostname, dsname: tom_eden_cap, oid: .1.3.6.1.4.1.9999.0.9.0, value: 209715200
05/12/2005 12:05:04 PM - CACTID: Poller[0] DEBUG: SQLCMD: insert into poller_output (local_data_id,rrd_name,time,output) values (2161,'tom_eden_cap',NOW(),'209715200')
...

BTW, here's the create message from the log:

Code: Select all

05/12/2005 11:25:10 AM - POLLER: Poller[0] CACTI2RRD: /usr/local/bin/rrdtool create 
/path/to/inst/rra/hostname_tom_s1_cap_2161.rrd --step 300   
DS:tom_s1_cap:GAUGE:600:0:4000000000  
DS:tom_s1_used:GAUGE:600:0:4000000000 
DS:tom_s1_pused:GAUGE:600:0:100  
DS:tom_s0_cap:GAUGE:600:0:4000000000  
DS:tom_s0_used:GAUGE:600:0:4000000000  
DS:tom_s0_pused:GAUGE:600:0:100  
DS:tom_perm_cap:GAUGE:600:0:4000000000  
DS:tom_perm_used:GAUGE:600:0:4000000000  
DS:tom_perm_pused:GAUGE:600:0:100  
DS:tom_old_cap:GAUGE:600:0:4000000000  
DS:tom_old_used:GAUGE:600:0:4000000000  
DS:tom_old_pused:GAUGE:600:0:100 
DS:tom_eden_used:GAUGE:600:0:4000000000  
DS:tom_eden_cap:GAUGE:600:0:4000000000  
DS:tom_eden_pused:GAUGE:600:0:100  
RRA:AVERAGE:0.5:1:600  
RRA:AVERAGE:0.5:6:700  
RRA:AVERAGE:0.5:24:775  
RRA:AVERAGE:0.5:288:797 
RRA:MAX:0.5:1:600  
RRA:MAX:0.5:6:700  
RRA:MAX:0.5:24:775  
RRA:MAX:0.5:288:797

edited to remove sensitive info

Post by **TheWitness** » Thu May 12, 2005 7:31 pm

You must use poller.php to launch cactid and then to update rrd files.

TheWitness

thebofh · Post by **thebofh** » Thu May 12, 2005 8:38 pm

Ya I've got poller.php running every five mins (which spawns cactid). All my other rrd's seem to be updating fine (even other custom snmp queries). I can't understand why this one fails.

Code: Select all

$ crontab -l|grep poller
0,5,10,15,20,25,30,35,40,45,50,55 * * * * /usr/local/bin/sudo -u cacti /usr/local/bin/php /path/to/install/poller.php >> /dev/null 2>/path/to/install/log/cron.log

thebofh · Post by **thebofh** » Mon May 16, 2005 2:04 pm

Bump... I'm still having problems. Any ideas?

Post by **rony** » Mon May 16, 2005 2:19 pm

For the host_id in question, go view the Poller Cache and tell me if you see any entries for that Host ID that have the SNMP OID from the data query.

thebofh · Post by **thebofh** » Mon May 16, 2005 4:47 pm

Yes. There are entries in the Poller Cache. Here's a sample:

Code: Select all

Data Source: Tomcat - JVM - Memory
RRD: /path/to/install/rra/hostname_tom_s1_cap_2162.rrd
Action: 0, OID: .1.3.6.1.4.1.9999.0.9.0 (Host: hostname, Community: public)

One thing that is different with this custom query (compared to my others) is that there are multiple DS items for this data template... Does that matter?

Post by **TheWitness** » Mon May 16, 2005 4:59 pm

The Multiple DS issue will cause all of your graphs to barf! You should delete the duplicated DS' from the Template and then confirm your graphs are back in order.

TheWitness

Post by **rony** » Mon May 16, 2005 9:07 pm

Yah, that would cause a problem...

thebofh · Post by **thebofh** » Tue May 17, 2005 1:18 pm

I think we're misunderstanding each other...

In my "Data Template" I have multiple "Data Source Items". I do not have any duplicate data source items (in other data templates).

For example I have a single Data Template named "Tomcat - Memory".
In that Data Template I have numerous Data Source Items each of which are memory related. For example I have Data Source Items named "tom_eden_used", "tom_eden_cap", "tom_old_used"... etc. So the update would be something like:

Code: Select all

rrdtool update file.rrd --template tom_eden_used:tom_eden_cap:... 1116351606:734003200:460792816:...

I should note that there are no other data templates with the same data source item names (in case that's what you're thinking). I think what I'm doing should work (why else would you have the option to create multiple data source items in a single data template?).

BTW, I've recreated the Data Template (using different data source item names ie tom1_eden_used), Graph Templates, and Data Query from scratch and it still seems to be a problem.

Post by **TheWitness** » Tue May 17, 2005 7:59 pm

We need to get closer to your system. It's 9:00pm here. You have my AIM and MSM.

TheWitness

thebofh · Post by **thebofh** » Wed May 18, 2005 12:59 pm

With help from thewitness last night it seems that cacti_snmp_get() in lib/snmp.php is being called with an empty $oid argument. This is causing a snmpget usage message to print out with an error "Missing object name".

Would failure of that single snmpget cause the other queries to fail too? When I added debug messages to cacti_snmp_get() it seems that the get's are successful and return the proper values (for every other query at least). I think my specific problem is elsewhere (PEBKAC?).

thebofh · Post by **thebofh** » Wed May 18, 2005 2:31 pm

OK I've dug a bit deeper. the values are being inserted into the 'poller_output' table correctly (by cmd.php).

The problem appears to be in poller.php:process_poller_output(). Well, maybe it's not a problem... maybe I'm attempting to do something cacti was not designed to support.

It seems that the times value is getting unset:

Code: Select all

 [/path/to/inst/rra/my.rrd] => Array
        (
            [local_data_id] => 2168
            [times] => Array
                (
                )                                                                                                                            
        )

instead of

 [/path/to/inst/rra/my.rrd] => Array
        (
            [local_data_id] => 2168
            [times] => Array
                (
                    [1116443361] => Array
                        (
                            [edn_cap] => 209715200
                            [edn_puse] => 6
                            [edn_use] => 13966288
                            [old_cap] => 33554432
                            [old_puse] => 61
                            [old_use] => 20505544
                            [prm_puse] => 0
                            [prm_use] => 134
                            [s0_cap] => 10485760
                            [s0_puse] => 0
                            [s0_use] => 0
                            [s1_cap] => 83886080
                            [s1_puse] => 56
                            [s1_use] => 47402512
                        )
                )                                                                                                                            
        )

I just realized that $unix_time consistently has a diff time for at least one of these values... Maybe it's getting unset when at least one value has a different time than the rest?

I should be able to populate a multi-value rrd with seperate SNMP OID's right?

For Example say I have two OID's as follows:

Code: Select all

oid: .1.2.3.4.0
name: foo
value: 56

oid: .1.2.3.4.1
name: bar
value: 78

Can I update the following rrd using a Custom SNMP query (not a script)?

Code: Select all

rrdtool create /path/to/inst/rra/foobar.rrd --step 300   DS:foo:GAUGE:600:0:100 DS:bar:GAUGE:600:0:100 RRA:AVERAGE:0.5:1:600  RRA:AVERAGE:0.5:6:700  RRA:AVERAGE:0.5:24:775  RRA:AVERAGE:0.5:288:797  RRA:MAX:0.5:1:600  RRA:MAX:0.5:6:700  RRA:MAX:0.5:24:775  RRA:MAX:0.5:288:797

So the update should look something like this:

Code: Select all

rrdtool update /path/to/inst/rra/foobar.rrd --template foo:bar 1116443407:56:78

I set things up assuming the above scenario will work. Or do I have to return all the values in a single snmpget? like so:

Code: Select all

oid: .1.2.3.4.0
name: foobar
value: foo:56 bar:78

Post by **TheWitness** » Wed May 18, 2005 4:50 pm

Based upon the contents of your RRD files, I would delete the RRD's and have the system recreate. Once an RRD has it's associated DS', its pretty detailed surgery to alter. So, if a dump of the RRD does not demonstrate the DS' that you are looking for, they are pretty much worthless.

So the real question is how are you getting blank OID's? And also, assuming the data query is returning data Ok, why does it work sometime? Could there be a re-index issue?

We definately had a disconnect late last night, so I gave up.

TheWitness

thebofh · Post by **thebofh** » Wed May 18, 2005 6:21 pm

TheWitness wrote: So the real question is how are you getting blank OID's?

I somehow got a bad row in the poller_item table... I've since fixed that but the problem persists.

Code: Select all

$ php cmd.php 31 31
#!/usr/bin/php -q
$

TheWitness wrote: why does it work sometime?

See my comments above about $unix_time and the times array... Could that be the cause? I'll try to explain what I'm seeing again.

in lib/poller.php the times array is unset if not all DS's are present (ie rrd_num < num of DS's found in poller_output):

Code: Select all

/* make sure each .rrd file has complete data */
reset($results);
foreach ($results as $item) {
    $unix_time = strtotime($item["time"]);                                                                                                                            
    if (isset($rrd_update_array{$item["rrd_path"]}["times"][$unix_time])) {
        if ($item["rrd_num"] <= sizeof($rrd_update_array{$item["rrd_path"]}["times"][$unix_time])) {
            db_execute("delete from poller_output where local_data_id='" . $item["local_data_id"] . "' and rrd_name='" . $item["rrd_name"] . "' and time='" . $item["time"] . "'");
        }else{
            unset($rrd_update_array{$item["rrd_path"]}["times"][$unix_time]);
        }
    }
}

But in the longer walk's or with a ton of DS's the $unix_time might be different right? I've observed cases where the $rrd_update_array might look like:

Code: Select all

Array
(
    [/path/to/inst/foo.rrd] => Array
        (
            [local_data_id] => 1186
            [times] => Array
                (
                    [1116454139] => Array
                        (
                            [disk_reads] => 198120
                        )
                    [1116454140] => Array
                        (
                            [disk_writes] => 6034111
                            [disk_write_bytes] => 1994839040
                            [disk_read_bytes] => 1132503528
                        )
                )
        )
    ...
)

So the data is all there but it's being tossed out because they're filed under different time keys. Am I reading the code correctly?

Cacti

custom indexed snmp data query not updating rrd

custom indexed snmp data query not updating rrd

Who is online