Device has Status Unknown

[gandalf]

./spine --verbosity=5 2 2

07/22/2010 11:02:06 PM - SPINE: Poller[0] Host[2] SNMP Result: Host responded to SNMP
07/22/2010 11:02:06 PM - SPINE: Poller[0] DEBUG: SQL:'UPDATE host SET status='3', status_event_count='0', status_fail_date='0000-00-00 00:00:00', status_rec_date='0000-00-00 00:00:00', status_last_error='', min_time='4.529950', max_time='11.800050', cur_time='4.529950', avg_time='7.013003', total_polls='3', failed_polls='0', availability='100.0000' WHERE id='2''

This shows, that the host responds to SNMP; thus the status is updated.

07/22/2010 11:02:06 PM - SPINE: Poller[0] DEBUG: SQL:'SELECT action, hostname, snmp_community, snmp_version, snmp_username, snmp_password, rrd_name, rrd_path, arg1, arg2, arg3, local_data_id, rrd_num, snmp_port, snmp_timeout, snmp_auth_protocol, snmp_priv_passphrase, snmp_priv_protocol, snmp_context FROM poller_item WHERE host_id=2 ORDER BY snmp_port'
07/22/2010 11:02:06 PM - SPINE: Poller[0] Host[2] DEBUG: HOST COMPLETE: About to Exit Host Polling Thread Function

That shows, that the poller_item table is empty for that host. So there's nothing to do for it.

cmd.php will not change status, unless poller_item has at least ONE entry for a host.

All in all, the question arises, why the poller item table is not filled. I suppose it's due to timeout issues related to the huge number of interfaces

R.

[gandalf]

Another observation, not sure, if it has any relevance. For debugging I clicked on '(Verbose Query)' in the Interface Statistics query. I receive NO debugging infos as expected, the browser tab just says "Loading ..." with a spinning symbol and the status line of the browser says "Waiting for <my-cacti-server> ..."

I'm quite sure that this loading goes ad infinitum. However no reason to worry, I guess, because the same behavior I have doing 'Reload Data Query' or when the snmpcache is filled the first time. When I open another tab after an hour or so, the data-query always seems to have been successful in the background --> Success [111589 Items, 16239 Rows]. These updated last values are from the mentioned Verbose-Query run.

This supports my opinion from above. You may increase the MAX OID things to e.g. 60. But i fear that will not change much.

Unfortunately, we perform the index scan "online" using php_snmp's snmpwalk function (or, if php_snmp is not installed, we use net-snmp libraries). I suppose we're hitting a timeout, here.

R.

[vuvuzela]

This shows, that the host responds to SNMP; thus the status is updated.

Yes, and the Status is updated exactly one time (Unknown --> UP). The current and average ping-times are frozen until I send the spine-command again. What ist new for me is, that the device can have Status UP despite the fact that no rrd exists for this device.

That shows, that the poller_item table is empty for that host. So there's nothing to do for it.

... that's correct, there are no entries in the Poller Cache for that device.

All in all, the question arises, why the poller item table is not filled.

... but how should cacti know, what to fill in the poller_item table? As I described, the cacti generated data source has an empty Index Value (remember the sreenshots?).

I suppose it's due to timeout issues related to the huge number of interfaces.

... I can not see, what should be time-critical ONCE the SNMP Cache is filled successfully? All that 'create graph' has to do afterwards is to read the selected interface-index (150994948), read the selected graph type (In/Out Bits 64-bit) from the pulldown-menu, and put both into the new data source (what fails obviously).

Via the interface.xml cacti should then know, that it has to collect just two OIDs from the device:
.1.3.6.1.2.1.31.1.1.1.6.150994948
.1.3.6.1.2.1.31.1.1.1.10.150994948
which are the ifHCInOctets and ifHCOutOctets. Both values come in a fingersnip with a snmpget - where is a timeout-risk?

Unfortunately, we perform the index scan "online" using php_snmp's snmpwalk function (or, if php_snmp is not installed, we use net-snmp libraries).

hmm, yes -->
Cacti Variables
Operating System: unix
PHP SNMP Support: yes

Is it possible to switch from php_snmp to net-snmp libraries for a test?

You may increase the MAX OID things to e.g. 60. But i fear that will not change much.

I will test and report the snmpbulkwalk with 60 instead of 10 MAX OIDs.

Kind regards

EDIT: I tried Maximum OID's Per Get Request = 60. No change.

[vuvuzela]

'apt-get remove php5-snmp' results in:

Cacti Variables
Operating System: unix
PHP SNMP Support: no

With this deactivated PHP SNMP I deleted and created the device again. For my surprise, the SNMP Cache is now a little different. A search for the index 150994948 results in ...

Host: erx2 , SNMP Query: SNMP - Interface Statistics
Index: 150994948 , Field Name: ifAlias , Field Value:
OID: .1.3.6.1.2.1.31.1.1.1.18.150994948
Host: erx2 , SNMP Query: SNMP - Interface Statistics
Index: 150994948 , Field Name: ifDescr , Field Value: ATM2/3
OID: .1.3.6.1.2.1.2.2.1.2.150994948
Host: erx2 , SNMP Query: SNMP - Interface Statistics
Index: 150994948 , Field Name: ifIndex , Field Value: 150994948
OID: .1.3.6.1.2.1.2.2.1.1.150994948
Host: erx2 , SNMP Query: SNMP - Interface Statistics
Index: 150994948 , Field Name: ifName , Field Value: AT2/3
OID: .1.3.6.1.2.1.31.1.1.1.1.150994948
Host: erx2 , SNMP Query: SNMP - Interface Statistics
Index: 150994948 , Field Name: ifOperStatus , Field Value: Up
OID: .1.3.6.1.2.1.2.2.1.8.150994948
Host: erx2 , SNMP Query: SNMP - Interface Statistics
Index: 150994948 , Field Name: ifType , Field Value: atm
OID: .1.3.6.1.2.1.2.2.1.3.150994948

... and compared with PHP-SNMP Support=Yes there are two Fields missing now: ifSpeed and ifHwAddr. And the value of ifType was 'atm(37)' before and now is just 'atm' (see a few posts earlier).

Because there ist no ifSpeed in the SNMP Cache the generated data source now has the default speed entries of 100000000 I guess (and not 155520000 as before). But the rest is empty again like always .

PHP SNMP seems to be a "better snmpwalker" than the net-snmp libraries. Crazy.

[TheWitness]

Are you new to Cacti. Because, I seem to think that you are to familiar with graph creation. I think this might be a training issue although it is a pretty big ATM device.

Tell us how you created your graph, step by step...

TheWitness

[vuvuzela]

Tell us how you created your graph, step by step...

... Err, I thought I did - ok, once again:

Each step is documented by the attached screenshots.

1) Add the device.
2) Save. The browser begins "Loading ..." doing the snmpwalk of the huge interface table and never returns.
3) After 10 min or so, I try to login from a second browser tab. If the snmpwalk is ongoing, I can't login from this tab, as it also shows "Loading ...". After a while, when the snmpwalk has finished in the background, I can login from the second browser tab.
4) The new device is in the table and has 'Status Unknown'.
5) The device shows the Associated Data Query from the previous snmpwalk, Status Success.
6) Click 'Create Graphs for this Host'. The SNMP-Interface-Statistics-List appears.
7) Because the list is very large, I search for the interface-name AT2/3 directly, select it, select the graph type 'In/Out Bits (64-bit Counters) and click Create.
8) A Data Source for the device is listed.
9) The Custom Data fields like 'Index Value' are empty - and shouldn't!
10) No graph is drawn, no rrd created and the device status remains Unknown.

Greets

[vuvuzela]

Just an idea ...

as I noticed and wrote, the OID-values for the ifType were not unique. Depending on the conditions, I get 'atm(37)' or just 'atm'. Is it possible, that the round brackets are not escaped somewhere in the code?

With snmpget I get 'atm(37)':
snmpget -v 2c -c <my-community> <my-host> .1.3.6.1.2.1.2.2.1.3.150994948
IF-MIB::ifType.150994948 = INTEGER: atm(37)

With the mib-browser tkmib I get 'atm':
.iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifType.150994948 = atm

With cacti with PHP SNMP Support: yes I get 'atm(37)':
Host: erx2 , SNMP Query: SNMP - Interface Statistics
Index: 150994948 , Field Name: ifType , Field Value: atm(37)
OID: .1.3.6.1.2.1.2.2.1.3.150994948

With cacti with PHP SNMP Support: no I get 'atm':
Host: erx2 , SNMP Query: SNMP - Interface Statistics
Index: 150994948 , Field Name: ifType , Field Value: atm
OID: .1.3.6.1.2.1.2.2.1.3.150994948

Any explanation for this inconsistency? Could this trigger a bug, when create graph is clicked and the data source has to be generated?

Greetings

[gandalf]

JWith snmpget I get 'atm(37)':
snmpget -v 2c -c <my-community> <my-host> .1.3.6.1.2.1.2.2.1.3.150994948
IF-MIB::ifType.150994948 = INTEGER: atm(37)

This is a guess only.
The value returned is an integer: 37.
And there is some table defined in a MIB that is in the MIB search path. Evaluating 37 in that table resolves to the string "atm". All in all it's "atm(37)" in net-snmp notation.
Again, this is an educated guess only
R.

[vuvuzela]

Again, this is an educated guess only

Good guess, gandalf.

IF-MIB imports IANAifType-MIB, in which ifType number 37 is associated with 'atm'.

I prove that by avoiding the usage of any mibs (in /etc/snmp/snmp.conf with the entry 'mibs :'). Then snmpget receives the raw integer of the ifType:

snmpget -v 2c -c <my-community> <my-host> .1.3.6.1.2.1.2.2.1.3.150994948
iso.3.6.1.2.1.2.2.1.3.150994948 = INTEGER: 37

Lets see, how cacti reacts on avoiding 'atm(37)'. I deleted the device, switched back to net-snmp libraries in cacti and recreated the device. In cactis SNMP cache afterwards the ifType is listed with the raw value '37':

Success [84702 Items, 15410 Rows]
Host: erx2 , SNMP Query: SNMP - Interface Statistics
Index: 150994948 , Field Name: ifAlias , Field Value:
OID: .1.3.6.1.2.1.31.1.1.1.18.150994948
Host: erx2 , SNMP Query: SNMP - Interface Statistics
Index: 150994948 , Field Name: ifDescr , Field Value: ATM2/3
OID: .1.3.6.1.2.1.2.2.1.2.150994948
Host: erx2 , SNMP Query: SNMP - Interface Statistics
Index: 150994948 , Field Name: ifIndex , Field Value: 150994948
OID: .1.3.6.1.2.1.2.2.1.1.150994948
Host: erx2 , SNMP Query: SNMP - Interface Statistics
Index: 150994948 , Field Name: ifName , Field Value: AT2/3
OID: .1.3.6.1.2.1.31.1.1.1.1.150994948
Host: erx2 , SNMP Query: SNMP - Interface Statistics
Index: 150994948 , Field Name: ifOperStatus , Field Value: Up
OID: .1.3.6.1.2.1.2.2.1.8.150994948
Host: erx2 , SNMP Query: SNMP - Interface Statistics
Index: 150994948 , Field Name: ifType , Field Value: 37
OID: .1.3.6.1.2.1.2.2.1.3.150994948

But alas the data source after create graph again is faulty. Even worse, the speed entries are gathered wrong again. So the round brackets should not be the reason for the trouble, screenshot:

[TheWitness]

Also, if Gandalf has not instructed you to do so, you may want to increase the "max_execution_time" variable in the Apache's version of "php.ini". That 'should' help.

TheWitness

[vuvuzela]

you may want to increase the "max_execution_time" variable in the Apache's version of "php.ini". That 'should' help.

And how it helps! Wonderful!

As you suggested, I increased max_execution_time from 30s to 1200s in the /etc/php5/apache2/php.ini and restarted apache2. That doesn't prevent "Loading ..." forever, but when I start a new session in a second browser after a while, I see a message "Save Successful" now. With just 30 sec this never happend!

Everything works now. The data source is generated, the device gets 'Status UP' and rrd and graph are created (see screenshots).

I guess this is a beginning of a great love story with cacti ... :-)

Many many thanks, TheWitness! And to all developers of this powerful software!

Greets

[gandalf]

Also, if Gandalf has not instructed you to do so, you may want to increase the "max_execution_time" variable in the Apache's version of "php.ini". That 'should' help.

Consider me a dumb ass
I was under the assumption that we already had lifted that restriction. But a quick look should have told me. Sorry for all the noise.
R.

[TheWitness]

Reinhard,

We might want to consider changing to 0 for that page only.

Larry