Spine 0.8.7d Released

[TheWitness]

Well at long last, I think that I have a good release. There has been quite a bit of testing by numerous people, and quite a bit of dispair as I have not had much time to work on this. But I believe that this version addresses several corner cases that get installations into trouble. Below is the updated ChangeLog

0.8.7d
-bug#0001310: Feature request for poller.php- single host polling
-bug#0001313: Spine takes an excessive amount of time timing out failed tcp pings
-bug#0001358: (cacti + spine) svn. Using snmp v3 constantly get "value: U", whereas snmpget returns correct value
-bug#0001379: Ping.c might send more then retry_count icmp packages
-bug#0001381: configure script distributed with cacti-spine-0.8.7c is broken
-bug#0001420: PCOMMAND: Poller[0] Host[80] WARNING: Recache Event Detected for Host
-bug: A host that for some reason can not initialize snmp can cause a segfault
-bug: When using nifty popen, when the script times out, attempt to kill it
-bug: Under some platforms tcp and udp ping unresponsive
-bug: Availability messaging incorrect for snmpv3 hosts
-bug: TCP Pings that receive a connection refused are good pings
-bug: Avoid segmentation faults when Reindex table issues are found
-bug: When performing snmp_get's, dont assume no error means non-null response
-bug: Some snmp agents don't getnext on '.1' and need another oid '.1.3'
-bug: Net-snmp versioning breaks snmpV3
-bug: Use system level snmp_retries variable instead of 3
-feature: Add one additional debug message to the poller code to track data sources polled
-feature: Add more config paths for packagers
-feature: Allow specification of an optional hostlist instead of first and last hostid's
-feature: Integrate more tightly with boost by allowing writing directly to the boost output table
-feature: Complete implementation poller_id field for people using a poller_id for polling.

Some key features for this release include:

1) The ability for the specification of a poller_id other than the default
2) The ability to specify a hostlist as opposed to a range of host ids.
3) The ability for spine to write directly to the boost table when running boost

All of these features allow for more flexibility that I plan for forthcoming releases of Cacti and distributed polling.

The boost feature will not be exploitable until Boost 2.5 is released though. For now, it's just there waiting.

There have been several other changes that affect snmpv3 that cover changes that tend to occur within the net-snmp product as it's implementation of snmpv3 matures.

I have also made changes to the way TCP, UDP, and ICMP pings function to make them more reliable and to generate less traffic. In addition, Spine now properly recognizes the difference between "Ping AND SNMP", and "Ping OR SNMP".

Bottom line is, that outside of the support for multiple pollers, I will pretty soon be working only on making the code faster, and to use less connections.

Enjoy, and if inspired, please donate.

TheWitness

[jerrison]

Hi Larry,

first of all thanks for improving cacti little by little every day!

I just tested spine-087d and had to switch back to spine-087c because the polling process in both my cacti-instances (1 minute and 5 minutes) were exceeding the polling-time.
I am running the latest CactiEZ on a physical server.
Is there anything i need to consider before i switch to spine-087d? Any downside on the polling side of things?
These are my polling stats:
5 minute instance
SYSTEM STATS: Time:81.8064 Method:spine Processes:4 Threads:10 Hosts:355 HostsPerProcess:89 DataSources:25497 RRDsProcessed:9316

1 minute instance
SYSTEM STATS: Time:10.8030 Method:spine Processes:5 Threads:5 Hosts:323 HostsPerProcess:65 DataSources:1968 RRDsProcessed:1122

i use 2 seperate spine binaries (well, same "make" different folder, to keep the databases apart).

I just wonder if there is something i overlooked in some remarks .

Greetings from Hamburg, GER;
André

edit:
this is what i get in debug:
DEBUG: About to Spawn a Remote Process [CMD: /opt/spinetest-087d/bin/spine, ARGS: 0 2]

i do not get this message when running spine-087c.

running spine manually works fine. It seems that poller.php does not start the new spine binary correctly.

edit2:
solved: i added "--conf /opt/spinetest-087d/etc/spine.conf" in poller.php for "$extra_args" in line 250. It seems to me that the new spine does not look in "../etc/spine.conf".
Is there any way to add "extra_args" somewhere in the settings menu!

[TheWitness]

That sounds like a bug. For now, just move spine.conf to /etc and you should be all set.

TheWitness

[srossen]

quick reply but I have upgraded to this version of spine and am using boost and so far seen no issues. Also in the two hours since upgrading I have found no gaps in my graphs.

[spike100]

(This is a posting I made to the Windows Specific Forum, I've copied here as it may be a bug and I'd like the Spine Gurus to take a look before any formal bugs are raised)

Hiya,

I'm stuck with this one! I'm currently polling a number of fibre channel switches and a windows cluster with cacti. The counters on the switches are polled with a PHP script (a slightly modified bersion of this one http://forums.cacti.net/about15797.html ), the windows servers SNMP counters are polled directly.

I was using php.cmd as the poller and all was (and is) OK. I switched to the latest version of spine (0.8.7d) this week (previous versions just crashed) to speed things up. This version of spine seems to work without crashing but only gets data back from the SNMP counters on the windows cluster, I get "Empty result" and "output: U" type errors

Code: Select all

Host[13] ERROR: Empty result [10.11.76.202]: 'D:\php\php.exe -q D:\Apache2\htdocs\cacti\scripts\query_fa_mib_san_portinfo.php 10.11.76.202 PASSWORD 1 get txelement 1

Code: Select all

Host[13] DS[661] SCRIPT: D:\php\php.exe -q D:\Apache2\htdocs\cacti\scripts\query_fa_mib_san_portinfo.php 10.11.76.202 PASSWORD 1 get txelement 1, output: U

for all of the switches.

I've tried/found a number of things...

1) The full path to php.exe and the script *is* included in the .xml file
2) Spine does seem to work as it gets the counters from the windows cluster
3) I renamed the .PHP file and the error didn't change, so I assumed spine cannot find the PHP file.
4) I ran filemon on the windows box that runs cacti, it confirmed that spine makes no attempt to run php.exe and hence doesn't open my PHP script!!
5) Evertyhing works again when I revert back to cmd.php.
6) I've tried rebuilding the cache.

7) I've run the cacti security script so I believe permissions are set correctly on cmd.exe and the PHP script. (I use an Apache webserver)

8.) I have applied the 4 Cacti 0.7.8d patches.

I'm at a loss as to why spine fails to execute my script. The various debug options I've tried (cacti.log, spine -V=5 etc) haven't shown me anything that cries out as a glaring error as to why this is failing.

Operating System: XP SP2
Webserver: Apache
Cacti: 0.8.7d
Spine: 0.8.7d
MySQL: 5 something (not sure where this is recorded)
PHP: 5.2.6
RRDTool (Cygwin or Win32 version): 1.2.x
Net-SNMP: 5.4.2
Cygwin (cygwin1.dll version): 1005.25.0.0 (Bundled with Spine 0.8.7d)
Plugin Architecture:

Does anyone have any pointers/ideas?


(I can supply any log/debug files you need to help with this one, I'll attach a the 'normal' style ones for info)

Cheers

Spike

[jerrison]

That sounds like a bug. For now, just move spine.conf to /etc and you should be all set.

TheWitness

i would if i could. but with two instances of cacti running (a poor mans poller seperation, i know) i need two different spine.conf.

[TheWitness]

That sounds like a bug. For now, just move spine.conf to /etc and you should be all set.

TheWitness

i would if i could. but with two instances of cacti running (a poor mans poller seperation, i know) i need two different spine.conf.

Ok, I'll look to see what happened and post a patch.

TheWitness

[TheWitness]

(This is a posting I made to the Windows Specific Forum, I've copied here as it may be a bug and I'd like the Spine Gurus to take a look before any formal bugs are raised)

Hiya,

I'm stuck with this one! I'm currently polling a number of fibre channel switches and a windows cluster with cacti. The counters on the switches are polled with a PHP script (a slightly modified bersion of this one http://forums.cacti.net/about15797.html ), the windows servers SNMP counters are polled directly.

I was using php.cmd as the poller and all was (and is) OK. I switched to the latest version of spine (0.8.7d) this week (previous versions just crashed) to speed things up. This version of spine seems to work without crashing but only gets data back from the SNMP counters on the windows cluster, I get "Empty result" and "output: U" type errors

Code: Select all

Host[13] ERROR: Empty result [10.11.76.202]: 'D:\php\php.exe -q D:\Apache2\htdocs\cacti\scripts\query_fa_mib_san_portinfo.php 10.11.76.202 PASSWORD 1 get txelement 1

Code: Select all

Host[13] DS[661] SCRIPT: D:\php\php.exe -q D:\Apache2\htdocs\cacti\scripts\query_fa_mib_san_portinfo.php 10.11.76.202 PASSWORD 1 get txelement 1, output: U

for all of the switches.

I've tried/found a number of things...

1) The full path to php.exe and the script *is* included in the .xml file
2) Spine does seem to work as it gets the counters from the windows cluster
3) I renamed the .PHP file and the error didn't change, so I assumed spine cannot find the PHP file.
4) I ran filemon on the windows box that runs cacti, it confirmed that spine makes no attempt to run php.exe and hence doesn't open my PHP script!!
5) Evertyhing works again when I revert back to cmd.php.
6) I've tried rebuilding the cache.

7) I've run the cacti security script so I believe permissions are set correctly on cmd.exe and the PHP script. (I use an Apache webserver)

8.) I have applied the 4 Cacti 0.7.8d patches.

I'm at a loss as to why spine fails to execute my script. The various debug options I've tried (cacti.log, spine -V=5 etc) haven't shown me anything that cries out as a glaring error as to why this is failing.

Operating System: XP SP2
Webserver: Apache
Cacti: 0.8.7d
Spine: 0.8.7d
MySQL: 5 something (not sure where this is recorded)
PHP: 5.2.6
RRDTool (Cygwin or Win32 version): 1.2.x
Net-SNMP: 5.4.2
Cygwin (cygwin1.dll version): 1005.25.0.0 (Bundled with Spine 0.8.7d)
Plugin Architecture:

Does anyone have any pointers/ideas?


(I can supply any log/debug files you need to help with this one, I'll attach a the 'normal' style ones for info)

Cheers

Spike

Spike,

Change your back slashes to forward slashes in your Data Input Method, and then repopulate your poller cache.

TheWitness

[divagater]

Well at long last, I think that I have a good release.

You sure do. After several days all the segfaults and resulting gaps in my graphs are gone. Well done!

~Billy

[spike100]

Spike,

Change your back slashes to forward slashes in your Data Input Method, and then repopulate your poller cache.

TheWitness

Thanks for the speedy reply.

They are already foward slashes.... Here's a snippet of the .xml file.

Code: Select all

<FA-MIB_PortInfo>
  <name>Get FC-MIB SANSwitch Port Information</name>
  <script_path>D:/php/php.exe -q D:/Apache2/htdocs/cacti/scripts/query_fa-mib_san_portinfo.php</script_path>
  <arg_prepend>|host_hostname| |host_snmp_community| |host_snmp_version|</arg_prepend>
  <arg_index>index</arg_index>
  <arg_query>query</arg_query>
  <arg_get>get</arg_get>
  <output_delimeter>!</output_delimeter>
  <index_order>PortInd</index_order>
  <index_order_type>numeric</index_order_type>

Here's a copy of the first cache entry:

Code: Select all

Script: D:\php\php.exe -q D:\Apache2\htdocs\cacti\scripts\query_fa-mib_san_portinfo.php 10.11.76.202 PASSWORD 1 get txelement 4
	RRD: D:\Apache2\htdocs\cacti\rra\fc-sw11_rxelement_664.rrd 

Spike

[TheWitness]

That sounds like a bug. For now, just move spine.conf to /etc and you should be all set.

TheWitness

i would if i could. but with two instances of cacti running (a poor mans poller seperation, i know) i need two different spine.conf.

Ok, I'll look to see what happened and post a patch.

TheWitness

Recompile with the attached util.c and it should resolve the problem.

TheWitness

[TheWitness]

Spike,

Change your back slashes to forward slashes in your Data Input Method, and then repopulate your poller cache.

TheWitness

Thanks for the speedy reply.

They are already foward slashes.... Here's a snippet of the .xml file.

Code: Select all

<FA-MIB_PortInfo>
  <name>Get FC-MIB SANSwitch Port Information</name>
  <script_path>D:/php/php.exe -q D:/Apache2/htdocs/cacti/scripts/query_fa-mib_san_portinfo.php</script_path>
  <arg_prepend>|host_hostname| |host_snmp_community| |host_snmp_version|</arg_prepend>
  <arg_index>index</arg_index>
  <arg_query>query</arg_query>
  <arg_get>get</arg_get>
  <output_delimeter>!</output_delimeter>
  <index_order>PortInd</index_order>
  <index_order_type>numeric</index_order_type>

Here's a copy of the first cache entry:

Code: Select all

Script: D:\php\php.exe -q D:\Apache2\htdocs\cacti\scripts\query_fa-mib_san_portinfo.php 10.11.76.202 PASSWORD 1 get txelement 4
	RRD: D:\Apache2\htdocs\cacti\rra\fc-sw11_rxelement_664.rrd 

Spike

Until your poller cache shows forward slashes, you have not repopulated your poller cache. For you XML to have one thing and the poller cache to have another, tells me you have not done that step.

TheWitness

[spike100]

Until your poller cache shows forward slashes, you have not repopulated your poller cache. For you XML to have one thing and the poller cache to have another, tells me you have not done that step.

TheWitness

Ah, I did click the link to rebuild the cache, honest I did notice the slashes were still 'the other way' but I just thought that's how it worked. Let me look into this further as this clearly isn't right...

Cheers

Spike

[spike100]

Ah, I did click the link to rebuild the cache, honest I did notice the slashes were still 'the other way' but I just thought that's how it worked. Let me look into this further as this clearly isn't right...

Hmm, I'm still seeing 'strangeness' with this one...

I completely removed all the queries relating to the script that gets data from my fibre channel switches. This removed the entries from the poller cache.

I've re-imported the queries, associated the query to a switch and created a new set of graphs. This has re-popultaed the poller cache, but the slahes appear as *backward*, even though they are *forward* in my .xml file.

Here's a new poller cache entry with BACKWARD slashes

Code: Select all

Script: D:\php\php.exe -q D:\Apache2\htdocs\cacti\scripts\query_fa-mib_san_portinfo.php ***.**.76.202 PASSWORD 1 get txelement 18
	RRD: D:\Apache2\htdocs\cacti\rra\fc-sw11_rxelement_742.rrd 

Here's the top of the san_port.xml with FORWARD slashes (and the command used to cat the file)

Code: Select all

D:\Apache2\htdocs\cacti\resource\script_queries>type san_port.xml

<FA-MIB_PortInfo>
  <name>Get FC-MIB SANSwitch Port Information</name>
  <script_path>D:/php/php.exe -q D:/Apache2/htdocs/cacti/scripts/query_fa-mib_sa
n_portinfo.php</script_path>
  <arg_prepend>|host_hostname| |host_snmp_community| |host_snmp_version|</arg_pr
epend>
  <arg_index>index</arg_index>
  <arg_query>query</arg_query>
  <arg_get>get</arg_get>
  <output_delimeter>!</output_delimeter>
  <index_order>PortInd</index_order>
  <index_order_type>numeric</index_order_type>

  <fields>
    <PortInd>

I've tried renaming this fille an observed the errors in the Data Query Debug Information so this file is definitely being read.

The Data Query Debug Information also shows back slashes for php.exe

Code: Select all

Data Query Debug Information
+ Running data query [29].
+ Found type = '4 '[script query].
+ Found data query XML file at 'D:/Apache2/htdocs/cacti/resource/script_queries/san_port.xml'
+ XML file parsed ok.
+ Executing script for list of indexes 'D:\php\php.exe -q D:\Apache2\htdocs\cacti\scripts\query_fa-mib_san_portinfo.php ***.**.76.202 PASSWORD 1 index'
+ Executing script query 'D:\php\php.exe -q D:\Apache2\htdocs\cacti\scripts\query_fa-mib_san_portinfo.php ***.**.76.202 PASSWORD 1 query index'
+ Found item [PortInd='1'] index: 1
+ Found item [PortInd='2'] index: 2
+ Found item [PortInd='3'] index: 3

So if spine is sensitive to the direction of the slashes, and would cause the problems I'm seeing if it got the wrong ones, the problem could be that spine is actually being sent the 'wrong' slashes by 'cacti'.... I take it cmd.php is less sensitive.

Does this make sense? Anything I can do to test this?

Spike

[toe_cutter]

Except for having to create a symlink for spine.conf to /etc, which is probably my fault for missing the ./configure variable.

spine-0.8.7d works great, no more coredumps
Thanks!