problem Ugraded from 86C to 86F: Some hosts not polling...

[addlema]

Very strange problem.. I upgraded and everything went well however, after the upgrade 85% of my hosts poll fine, but a few dont poll at all.

They are up, and not disabled.. When I look at the device in the device menu it sees it with no error, I can refresh the intereface list on the new graph screen....

BUT.. Even with Debugging turned all the way on I never seen these hosts in my log file, and then rra file never gets updated of course.

Its random hosts as well, not a block.

Any ideas here? Im stumped and had to revert back to 86C untill I can figure this out...

HELP!

[addlema]

Ohh and I did try disabling the host, letting a cycle go by, and re-enabling.. Also tried clearing the poller cache.

[BSOD2600]

Go to Data Sources and select None for the host. Anything listed?

Sounds like you might have a database problem.

What are the versions of everything you're running?

[addlema]

I unistalled it as it was production server. I will install its backup on a test server and look at what you recommend, but I remember there being datasources....

Versions: PHP 4.3.9 MYSQL: 4.0.22 Net-SNMP: 5.1.1 RRDtool 1.0.49

[njfirefighter15]

I am having the same error. After upgrading from 0.8.6d to 0.8.6e, the disk space scripts started failing for my windows hosts. I have a bone-stock cacti environment. It was running on Fedora Core 3, all updates with net-snmp 5.1.2, php 4.3.11, mysql 3.23.58 and RRDtool 1.2.10. Everything was fine until the upgrade. I also did the upgrade from 0.8.6e to 0.8.6f and nothing changed with the collection failure. I'm still running cactid 0.8.6e. I had 115 hosts, 15 threads and 1281 datasources which all worked fine until the upgrade.

I tried upgrading the net-snmp to 5.2.1 and recompiled cactid - no change. My final last resort was I completely rebuilt the box with Fedora Core 4 - all updates, net-snmp5.2.1, php 5.0.4, mysql 4.1.11 and RRDtool 1.2.10. After moving my old data onto this box, I had the same results as on the old box.

I dropped the database, and did a fresh, clean install of cacti 0.8.6f and added a few hosts. The polled just fine. I figured I musta jacked up my database somehow, so I figured I'd start clean. So a manually re-added all my hosts, graph trees, datasources, etc with the poller stopped. I fired up the poller, and I got the same results - my diskspace graphs like cpu or network graphs with major high/low spikes, and with values far larger than the physical disk even has (i.e 33 gb disk, graph shows spikes up to 800 gb). It started once a certain ammount of hosts were added.

Any help would be greatly appreciated - I'd like to finally get around to adding my other 300ish servers to my cacti system. I am attaching a copy of one poll period with full logging enabled. The script is returning a value of U - and I know it isn't the server end which is failing.

Thanks,
Dave

MYSQL: Connecting to MySQL database 'cacti' on 'localhost'...
Poller[0] MYSQL: Connected to MySQL database 'cacti' on 'localhost'...
CACTID: Poller[0] ERROR: Fatal select() error
CACTID: Poller[0] Host[71] DS[542] WARNING: Result from SERVER not valid. Partial Result: ...
CACTID: Poller[0] Host[71] DS[542] SERVER: /var/www/html/cacti/scripts/ss_host_cpu.php ss_host_cpu myserver.mycompany.com mycomstring 1 161 500 get usage 0, output: U
CACTID: Poller[0] DEBUG: SQLCMD: insert into poller_output (local_data_id,rrd_name,time,output) values (542,'cpu','2005-07-12 03:35:02','U')
CACTID: Poller[0] Host[72] PING Result: ICMP: Host is Alive
CACTID: Poller[0] Host[72] SNMP Result: Host responded to SNMP
CACTID: Poller[0] DEBUG: SQLCMD: update host set status='3',status_event_count='0', status_fail_date='0000-00-00 00:00:00',status_rec_date='0000-00-00 00:00:00',status_last_error='',min_time='0.509590',max_time='1.674990',cur_time='1.464045',avg_time='0.963729',total_polls='128',failed_polls='0',availability='100.0000' where id='72'

[TheWitness]

Are you manualy making Cactid from source?

You seem to have an issue with pipes/streams in your FC build in attempting to communicate both with the script server and with external scripts.

What ID are you using in your Cron?
Could you please run ./cactid manually in DEBUG as root and see if things are better?

TheWitness

[njfirefighter15]

I am making cactid from source on the machine it is running on. With a few hosts everything runs fine. I'm not sure at what number of hosts this problem starts to occur. Again, everything was fine until I left 0.8.6d behind, and upgraded. I currently run the cron job under the root user (I know it isn't reccomended-all this box does is cacti, so if it crashes and burns it isn't the end of the world). I will manually kick it off and post the debug output from cactid.

[TheWitness]

I suspect an issue with the script server. Please attempt to start it using the following command.

Code: Select all

php <cacti>/script_server.php

Then if it starts without error, type:

Code: Select all

quit

this will verify that it works. Before "quit"ing, you could also type:

Code: Select all

/var/www/html/cacti/scripts/ss_host_disk.php ss_host_disk ctmildom01.mycompany.com mycomstring 1 2 161 500 get total 5

You should get something other than an error.

TheWitness

[njfirefighter15]

Starting the script server from the console works just fine - no errors. I ran the command below, and it did not return an error, it returned a value as it should.

Do I need to decrease the threads to limit the load on the script server?

As promised, please find an attached logfile from the output of running cactid as root in debug mode.

Again, thanks for the help!
Dave

[TheWitness]

njfirefighter15,

Please state the following:

OS, OS Version
PHP Version,
MySQL Version
Net-SNMP Version
Cactid Processes & Threads
Cactid Version
Cacti Version
#CPU's, Speed, and Memory

Thanks,

TheWitness

[njfirefighter15]

njfirefighter15,

Please state the following:

OS, OS Version
PHP Version,
MySQL Version
Net-SNMP Version
Cactid Processes & Threads
Cactid Version
Cacti Version
#CPU's, Speed, and Memory

Thanks,

TheWitness

Sorry - I though I had posted that info in a prior post minus the cpu info.

Fedora Core 4 - All Updates kernel 2.6.12-1.1390_FC4smp
PHP 5.0.4
MySQL 4.1.11
Net-SNMP 5.2.1
Cactid Processes: 1 Threads: 10 Script Timeout: 25 seconds
Cactid Version: 0.8.6e
Cacti Version: 0.8.6f
IBM Blade HS20 2 x 2.0 GHz Xeon (Hyper-Threaded) 1.5 Gb DDR Ram

Thanks,
Dave

[TheWitness]

Dave,

With that box, you should easily be able to run 4 processes at 10 threads each. The issue has to do with FC. I have never seen it before. Are you using php_snmp?

TheWitness

[TheWitness]

Dave,

Please re-make Cactid with this modified php.c file and let me know if it changes the output from the log's.

TheWitness

[njfirefighter15]

TheWitness,

I re-built cactid from source using the php.c file you supplied - I did not notice a change in the logfile or the graphs.

I have php-snmp support built on the machine, as well as net-snmp, and net-snmp is chosen on the settings/general page. I did not have php-snmp built on the FC3 build, and still had this problem after the upgrade.

Thanks,
Dave

[TheWitness]

Quite odd, goto settings->poller and tell me what you have your script timeout set to?

TheWitness