I used the snmpnetstat based perl script from Gandalf to graph tcp connections and it was fine until I noticed that it will seriously hang on a very busy webserver. So bad that it kills all graphs for this host. Even when you crank up all timeouts very high. I tried running 'netstat -at' over ssh instead and it returned the desired output pretty quickly at the same time as snmpnetstat would hang forever so I know it's not cacti its just snmpd not returning stuff.
This gave me the idea to modify Gandalf's script to use ssh with keys instead of snmp but still return the same results.
I have it working with keys so that it returns the data but for some reason when I try to use it the rrd file is simply not created anymore. The debug log proves that my command returns values but there are no errors relating to this graph. It happily reports doing all the steps except - the rrd file doesn't exist.
When I create it manually using the rrd debug output it lets me do that but the graphs now have all "nan" values. They also don't seem to get updated after.
This worked before I changed to the ssh version of the script.
I have never done any advanced stuff like that so now I'm wondering how to check why the rrd's arent' created by the poller.
Btw it's definitely not a permision issue. I can copy and paste the exact command from the poller cache and run it as the cacti user and as mentioned I can also run the rrd --create command line manually but it still doesn't update after that.
I really like the tcp netstat graphs but I need them to run under high load with 10000 connections and snmpnetstat simply doesnt seem to handle that.
Can you gurus please help ?
Below the debug file clearly showing that the script returns values.
03/01/2010 05:45:01 AM - SPINE: Poller[0] DEBUG: In Poller, About to Start Polling of Host
03/01/2010 05:45:01 AM - SPINE: Poller[0] DEBUG: The Value of Active Threads is 1
03/01/2010 05:45:01 AM - SPINE: Poller[0] Host[56] SNMP Result: Host responded to SNMP
03/01/2010 05:45:01 AM - SPINE: Poller[0] Host[56] RECACHE: Processing 2 items in the auto reindex cache for 'pscom-prd-web6'
03/01/2010 05:45:01 AM - SPINE: Poller[0] Host[56] NOTE: There are '1' Polling Items for this Host
03/01/2010 05:45:01 AM - SPINE: Poller[0] Host[56] DEBUG: The NIFTY POPEN returned the following File Descriptor 7
03/01/2010 05:45:02 AM - SPINE: Poller[0] Host[56] DS[874] SCRIPT: /usr/bin/perl /var/www/html/cacti-0.8.7e/scripts/ssh_netstat_tcp.pl pscom-prd-web6 , output: established:227 listen:11 timewait:2565 timeclose:0 finwait1:22 finwait2:9 synsent:0 synrecv:0 closewait:90
03/01/2010 05:45:02 AM - SPINE: Poller[0] Host[56] DEBUG: HOST COMPLETE: About to Exit Host Polling Thread Function
03/01/2010 05:45:02 AM - SPINE: Poller[0] DEBUG: The Value of Active Threads is 0
03/01/2010 05:45:02 AM - SPINE: Poller[0] DEBUG: Thread Cleanup Complete
03/01/2010 05:45:02 AM - SPINE: Poller[0] DEBUG: PHP Script Server Pipes Closed
03/01/2010 05:45:02 AM - SPINE: Poller[0] DEBUG: Allocated Variable Memory Freed
03/01/2010 05:45:02 AM - SPINE: Poller[0] DEBUG: MYSQL Free & Close Completed
03/01/2010 05:45:02 AM - SPINE: Poller[0] Time: 0.6960 s, Threads: 10, Hosts: 2
03/01/2010 05:45:02 AM - SYSTEM STATS: Time:1.2144 Method:spine Processes:8 Threads:10 Hosts:2 HostsPerProcess:1 DataSources:1 RRDsProcessed:0
yet no rrd exists
(PROD) [root@pscom-prd-adm2 astuck]# ll /var/www/html/cacti/rra
total 0
I'm running cacti-0.8.7e with all patches and the latest spine with patches.
I'm really interested in the ssh method.
Shouldn't the log file at least show an error trying to create the rrd file ?
The 'update rrd file' it checked for all outputs in the data input method.
Just to isolate I only created one single host with one single graph. I can create the normal out-of-the box graphs just fine. All rrds get created within a minute just not this one. I didn't mess with Gandalf's other templates - other than renaming them to "ssh netstat" so I don't get confused. I also changed the data input method to "script/command" but, again, the log shows its gathering data so I don't think the problem lies there.
Please help