Spine not working after Cacti upgrade

[satish_kumarp2k1]

Hi All,

I am facing a strange problem after Cacti upgrade. I have upgraded the Cacti version 0.8.6i-3.4 to version 0.8.7b. The upgrade was successful. And after upgrade I compiled,
installed and started to use Spine. The real problem is: Spine collects all the configured things from SNMP clients and updates the mysql database, but NONE of the rrd files are getting updated with the latest values. I tried running Spine with some debug flags ( strace /usr/local/spine/bin/spine -C /usr/local/spine/etc/spine.conf -R -S -V 5) and I didn't notice anything strange except the following:

ERROR: Spine Timed Out While Processing Hosts Internal

DEBUG: SQL:'replace into settings (name,value) values ('date',NOW())'
DEBUG: SQL:'insert into poller_time (poller_id, start_time, end_time) values (0, NOW(), NOW())'
DEBUG: Thread Cleanup Complete
DEBUG: PHP Script Server Pipes Closed
DEBUG: Allocated Variable Memory Freed
SPINE: Net-SNMP API Shutdown Completed
DEBUG: MYSQL Free & Close Completed
Time: 294.0785 s, Threads: 10, Hosts: 496

Before the upgrade, I was using cactid as the poller and it was working properly. But unfortunately, both Spine and cmd.php are not working after the upgrade (rrd files doesn't get updated). I even tried by increasing the PHP memory to 128 MB, but still it didn't help.

The environment we use is as follows:
O.S : Debian Etch
Cacti : 0.8.6i-3.4 to 0.8.7b
Monitored hosts : Around 500
Total Graphs in use : Around 6200

dpkg -l | grep mysql
ii libdbd-mysql-perl 3.0008-1 A Perl5 database interface to the MySQL data
ii libmysqlclient15-dev 5.0.32-7etch5 mysql database development files
ii libmysqlclient15off 5.0.32-7etch5 mysql database client library
ii mysql-client-5.0 5.0.32-7etch5 mysql database client binaries
ii mysql-common 5.0.32-7etch5 mysql database common files (e.g. /etc/mysql
ii mysql-server 5.0.32-7etch5 mysql database server (meta package dependin
ii mysql-server-5.0 5.0.32-7etch5 mysql database server binaries
ii php4-mysql 4.4.4-8+etch6 MySQL module for php4

ii libapache2-mod-php4 4.4.4-8+etch6 server-side, HTML-embedded scripting languag
ii php4 4.4.4-8+etch6 server-side, HTML-embedded scripting languag
ii php4-cli 4.4.4-8+etch6 command-line interpreter for the php4 script
ii php4-common 4.4.4-8+etch6 Common files for packages built from the php
ii php4-mysql 4.4.4-8+etch6 MySQL module for php4
ii php4-snmp 4.4.4-8+etch6 SNMP module for php4

PS: The same environment with old Cacti 0.8.6i-3.4 was working properly.

I would really appreciate any quick help in this regard. I am actually looking forward to make use of command line tools for creating graphs / hosts etc., which would be very useful to automate things.

Thanks in advance,

Thanks & Regards,,
Satish Kumar P

[gandalf]

You are still using poller.php in crontab? What happens, if running php -q poller.php --force as cactiuser from command line?
Reinhard

[satish_kumarp2k1]

Hi Reinhard,

Thanks for looking into this. The contents of the crontab are as follows:

-> cat /etc/cron.d/cacti
MAILTO=root
*/5 * * * * www-data /usr/local/spine/bin/spine -C /usr/local/spine/etc/spine.conf 2>/var/log/cacti/poller-error.log

The output of the command you informed is as follows:

-> su www-data -c 'php -q poller.php --force'
Allowed memory size of 268435456 bytes exhausted (tried to allocate 24 bytes)

Is this related to allowed memory_limit in php.ini?
But the surprising part is, I got similar kind of output for the above command even after specifying 512M for memory_limit in both /etc/php4/apache2/php.ini and /etc/php4/cli/php.ini files.

Thanks & Regards,
Satish Kumar P

[satish_kumarp2k1]

Any update???

Thanks & Regards,
Satish Kumar P

[gandalf]

-> cat /etc/cron.d/cacti
MAILTO=root
*/5 * * * * www-data /usr/local/spine/bin/spine -C /usr/local/spine/etc/spine.conf 2>/var/log/cacti/poller-error.log

That's wrong. Please see the installation manual for the correct statement (e.g. at http://docs.cacti.net)
Reinhard

[satish_kumarp2k1]

Hi Reinhard,

Thanks for the reply.

OK, I have updated the cacti cron file. Now it has the following contents:

cat /etc/cron.d/cacti
MAILTO=root
*/5 * * * * www-data php /usr/share/cacti/poller.php > /dev/null 2>/var/log/cacti/poller-error.log

And I have selected Spine for (Settings -> Poller -> "Poller Type"). But I have couple of queries:
1. If the cron file has an error, but why would the manually executed command fail? Still the rrd files are not getting updated.
2. I still get the following error (even after choosing 512M for memory):
Allowed memory size of 268435456 bytes exhausted (tried to allocate 10 bytes)
Allowed memory size of 268435456 bytes exhausted (tried to allocate 10 bytes)

Which memory it is referring to?

Thanks & Regards,
Satish Kumar P

[gandalf]

No answer for 1)
2) Please truncate table poller_output and retry
Reinhard

[greginmadison]

Is this a PHP cli memory limit. The cli memory limit for PHP is set in a different location on you operating system then the apache php module.

Try upping that memory limit and see if that fixes this.

I had the same messages when running some php cli add_tree.php scripts and it went away after I increase the cli memory setting for php.

Just do a find / -type f -name php.ini -print to find the file.

[satish_kumarp2k1]

Reinhard,

I tried your suggestion. Truncated the poller_output table and ran the poller. The problem still persists (rrd files are not getting updated), but this time I didn't find any poller error messages. No more memory exhausted messages, but the problem still exists. I am running out of all options I guess.

Greg,

Thanks for checking in and for the update. The machine has only two php.ini files and they are as follows:

/etc/php4/apache2/php.ini
/etc/php4/cli/php.ini

I have the following line in both the above files:

memory_limit = 512M

Thanks & Regards,
Satish Kumar P

[satish_kumarp2k1]

Reinhard,

Update:

I have tried the following:

1. Dropping the cacti database from the problematic Cacti server (0.8.7b)
2. Took a fresh mysql backup of Cacti database in working Cacti server (0.8.6i)
3. Created a new fresh database, Cacti in problematic Cacti server (0.8.7b)
4. Imported the backup database
5. Finished the upgrade formalities from the web interface

And surprisingly, it is working properly. Poller is updating the rrd database files. I found the following in cacti.log:

07/02/2008 09:39:56 AM - SYSTEM STATS: Time:294.8720 Method:spine Processes:1 Threads:30 Hosts:500 HostsPerProcess:500 DataSources:18859 RRDsProcessed:9750
07/02/2008 09:45:00 AM - SYSTEM STATS: Time:299.5152 Method:spine Processes:1 Threads:30 Hosts:500 HostsPerProcess:500 DataSources:18859 RRDsProcessed:6
07/02/2008 09:49:55 AM - SYSTEM STATS: Time:294.9975 Method:spine Processes:1 Threads:30 Hosts:500 HostsPerProcess:500 DataSources:18859 RRDsProcessed:9167
07/02/2008 09:54:56 AM - SYSTEM STATS: Time:294.2478 Method:spine Processes:1 Threads:30 Hosts:500 HostsPerProcess:500 DataSources:18859 RRDsProcessed:10187
07/02/2008 09:59:55 AM - SYSTEM STATS: Time:294.5632 Method:spine Processes:1 Threads:30 Hosts:500 HostsPerProcess:500 DataSources:18860 RRDsProcessed:10282
07/02/2008 10:04:56 AM - SYSTEM STATS: Time:294.3909 Method:spine Processes:1 Threads:30 Hosts:500 HostsPerProcess:500 DataSources:18860 RRDsProcessed:7348
07/02/2008 10:09:56 AM - SYSTEM STATS: Time:294.9292 Method:spine Processes:1 Threads:30 Hosts:500 HostsPerProcess:500 DataSources:18860 RRDsProcessed:10220
07/02/2008 10:14:55 AM - SYSTEM STATS: Time:294.2609 Method:spine Processes:1 Threads:30 Hosts:500 HostsPerProcess:500 DataSources:18860 RRDsProcessed:8976
07/02/2008 10:20:00 AM - SYSTEM STATS: Time:298.4528 Method:spine Processes:1 Threads:30 Hosts:500 HostsPerProcess:500 DataSources:18859 RRDsProcessed:6
07/02/2008 10:24:55 AM - SYSTEM STATS: Time:294.6169 Method:spine Processes:1 Threads:30 Hosts:500 HostsPerProcess:500 DataSources:18859 RRDsProcessed:9831
07/02/2008 10:29:56 AM - SYSTEM STATS: Time:294.8938 Method:spine Processes:1 Threads:30 Hosts:500 HostsPerProcess:500 DataSources:18859 RRDsProcessed:10186
07/02/2008 10:34:55 AM - SYSTEM STATS: Time:294.2752 Method:spine Processes:1 Threads:30 Hosts:500 HostsPerProcess:500 DataSources:18859 RRDsProcessed:9354
07/02/2008 10:40:00 AM - SYSTEM STATS: Time:298.4437 Method:spine Processes:1 Threads:30 Hosts:500 HostsPerProcess:500 DataSources:18859 RRDsProcessed:6
07/02/2008 10:44:54 AM - SYSTEM STATS: Time:294.2521 Method:spine Processes:1 Threads:30 Hosts:500 HostsPerProcess:500 DataSources:18859 RRDsProcessed:9217
07/02/2008 10:49:56 AM - SYSTEM STATS: Time:294.6844 Method:spine Processes:1 Threads:30 Hosts:500 HostsPerProcess:500 DataSources:18859 RRDsProcessed:9021
07/02/2008 10:55:00 AM - SYSTEM STATS: Time:299.4244 Method:spine Processes:1 Threads:30 Hosts:500 HostsPerProcess:500 DataSources:18859 RRDsProcessed:6

Main question is: Why is spine not able to update all the rrd files for every poll. And the values are also strange (6, 9021, 9354, 10186 etc.). Is there any way to fine tune the settings?

Current poller settings:
Poller Interval : 5 Minutes
Cron Interval : 5 Minutes
Maximum Concurrent Poller Processe : 1
Maximum Threads per Process : 30
Number of PHP Script Servers : 1
Script and Script Server Timeout Value : 25
The Maximum SNMP OID's Per SNMP Get Request : 30

What is the maximum value that can be used for "Maximum Threads per Process"?

Thanks for your time.

Thanks & Regards,
Satish Kumar P

[gandalf]

The data shows insane behaviour. Spine does not finish in time.
What CPU configuration is used; e.g. dual socket quad core or the like?
General rule of thumb:
Use 1-2 * total number of cores as a strat for processes
Use 1-15 as a start for threads
Identify slowly responding hosts thatmay block a complete spine process
Reinhard

[satish_kumarp2k1]

Actually the older Cacti version 0.8.6i running in a different server (with poller as cactid) is working properly by updating all the rrd files properly. And cactid is able complete all the total process (querying clients and updating the rrd files) in around 150 secs.

I tried fine tuning spine settings, but still I see the anomaly. I have even tried the cmd.php, but it's worser than spine. cmd.php was updating hardly 100 rrd files in each poll. Can the time taken by Spine to finish the entire process (querying and updating) be reduced (may be similar to time taken by cactid)?

I have one more concern, how it is going work if we were to add more devices in future.

One more doubt on cli commands: We have defined new Graph templates with some customized CDEF functions. When we manually add the graphs, we are to select the CDEF function used (the function actually used is not always same. There are around 8 CDEF functions, which is supposed to be chosen based on the machine).

When I actually tried to create a graph from command line, I got an error:

php /usr/share/cacti/cli/add_graphs.php --graph-type=cg --graph-template-id=99 --host-id=572 --input-fields="107:CDEF=4-CPU,6:CDEF=4-CPU,4:CDEF=4-CPU,5:CDEF=4-CPU,105:CDEF=4-CPU"

ERROR: Unknown input-field (CDEF)
Try --list-input-fields

And when I ran, "php add_graphs.php --list-input-fields --graph-template-id=100"

Known Input Fields:(name, default, description)

I am unable to see CDEF function here. How can I select CDEF function while creating graphs from the command line?

[gandalf]

Spine should be faster than cactid. But debugging this from remote is very hard. In general, the amount of resources you are polling is quite an easy task for cacti/spine; so it's not a general problem.
CDEF issue:
I suppose I know what you're aiming at. To be honest, I never tried this way. This will at least require some more testing, but I fear it is not currently supported.
Please post the/some of those CDEFs. I suppose there's a way to achieve the requested result with a single CDEF only
Reinhard