Advanced Ping has a lot of scalability issues when you have so many hosts. The other thing is how long it takes spine to initialize with so many threads. What is your script server count? How many script server calls do you use?
One option would to reduce the ping count. I think the default in the script is 20 samples, you might want to move it to 10 for example. That'll speed things up.
I almost think that we need to move Advanced Ping to it's own plugin as the polling times make it real difficult to complete in the 5 minute cycle. For me, it's an easy thing, just making the time to do it is the issue.
To answer some of those question, years ago, I wrote a 'showproc' script. It's been refined over the years to do other things. Below is a copy. Root just needs super access. Put it in root's home directory and mark it executable.
Code: Select all
#!/bin/sh
while true;do
clear
echo "---------------------------------------------------------"
echo -n "Uptime:"
uptime
# Collect metrics
SLEEPING=`mysql -e "show processlist" | grep Sleep | wc -l`
RUNNING=`mysql -e "show processlist" | grep -v Sleep | wc -l`
TOTAL=`expr $SLEEPING + $RUNNING`
ITEMS=`mysql -e "SELECT TABLE_ROWS FROM information_schema.TABLES WHERE TABLE_NAME='poller_output'" | grep -v TABLE_ROWS`
PROCS=`mysql -e "SELECT count(*) FROM processes WHERE tasktype='boost'" cacti | grep -v count`
SPINE=`mysql -e "SELECT count(*) FROM poller_time WHERE end_time='0000-00-00'" cacti | grep -v count`
HISTORY=`mysql -e "SHOW GLOBAL STATUS LIKE 'Innodb_history_list_length'" | grep -v Value | sed -e 's/Innodb_history_list_length//g' | tr -d " \t"`
# Echo status
echo "---------------------------------------------------------"
echo "Host: $HOSTNAME"
echo "Pending Poller Items: $ITEMS"
echo "---------------------------------------------------------"
echo "Running Processes: $RUNNING"
echo "Sleeping Processes: $SLEEPING"
echo "---------------------------------------------------------"
echo "Total Processes: $TOTAL"
echo "History: $HISTORY"
echo "Spine Processes: $SPINE"
if [ -z $PROCS ]; then
echo "Boost Processes: 0"
else
echo "Boost Processes: $PROCS"
fi
echo ""
echo "Memory Topology"
echo "---------------------------------------------------------"
free -g
echo ""
#echo "Poller Stats"
#ps -ef | egrep "(php|spine|poller.php)" | grep -v "script_server" | grep -v grep
#echo "---------------------------------------------------------"
#echo "Comm Stats"
#echo "---------------------------------------------------------"
#mysql -e "SHOW GLOBAL STATUS" | egrep "(Com_update|Com_select|Com_insert|Com_set_option|Com_delete|wsrep_cluster_size|Aborted_clients|Uptime)" | awk '{printf("%-25s %10i\n", $1, $2)}'
echo "MariaDB Processes"
mysql -e "SELECT id, user, state, ROUND(time_ms/1000,1) AS time, SUBSTRING(REPLACE(REPLACE(REPLACE(info, '\n', ' '), ' ', ' '), '\t', ' '),1,245) AS info FROM processlist WHERE info NOT LIKE '%processlist%' ORDER BY time DESC LIMIT 35" information_schema
running=`mysql -e 'SELECT COUNT(*) AS running FROM cacti.poller_time WHERE end_time="0000-00-00"' | grep -v running`
if [ "$running" -ne "0" ]; then
echo ""
echo "Running Collector Threads"
mysql -e "SELECT * FROM poller_time WHERE end_time='0000-00-00'" cacti
echo ""
echo "Last Hosts Inserted"
mysql -e "SELECT h.id, h.description, b.last_updated, COUNT(b.local_data_id) AS data_sources FROM poller_output_boost AS b INNER JOIN data_local AS dl ON b.local_data_id = dl.id INNER JOIN host AS h ON h.id=dl.host_id WHERE b.last_updated = (SELECT max(last_updated) FROM poller_output_boost) GROUP BY h.id ORDER BY b.last_updated DESC, data_sources DESC LIMIT 5" cacti
else
echo ""
echo "Poller Not running"
fi
sleep 2
done
