Submit Your CMD.PHP vs. SPINE Metrics Here

Important information about Cacti developments that all users should be interested in.

Moderators: Developers, Moderators

Post Reply
User avatar
gandalf
Developer
Posts: 22383
Joined: Thu Dec 02, 2004 2:46 am
Location: Muenster, Germany
Contact:
Contact gandalf

Post by gandalf »

Please pay special attention to scripts with undefined timeouts. This happened to me lately
Reinhard
Top
User avatar
TheWitness
Developer
Posts: 17007
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:
Contact TheWitness

Post by TheWitness »

jbarbieri wrote:cmd.php
Edit-----

Spine is having hiccups too :(
Update to 0.8.7c

TheWitness
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages

For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Top
jbarbieri
Cacti User
Posts: 101
Joined: Tue May 24, 2005 8:46 am
Location: Rochester, New Hampshire
Contact:
Contact jbarbieri

Post by jbarbieri »

TheWitness wrote:
jbarbieri wrote:cmd.php
Edit-----

Spine is having hiccups too :(
Update to 0.8.7c

TheWitness
It's on 0.8.7c


Had a helluva time getting spine to compile on solaris.
Top
totosh
Posts: 17
Joined: Mon Mar 10, 2008 5:49 am

Post by totosh »

03/04/2009 05:57:11 PM - SYSTEM STATS: Time:9.8844 Method:spine Processes:2 Threads:40 Hosts:145 HostsPerProcess:73 DataSources:31187 RRDsProcessed:10786
03/04/2009 05:56:12 PM - SYSTEM STATS: Time:10.0587 Method:spine Processes:2 Threads:40 Hosts:145 HostsPerProcess:73 DataSources:31187 RRDsProcessed:10786
03/04/2009 05:55:14 PM - SYSTEM STATS: Time:9.3282 Method:spine Processes:2 Threads:40 Hosts:145 HostsPerProcess:73 DataSources:31187 RRDsProcessed:10786
03/04/2009 05:54:13 PM - SYSTEM STATS: Time:10.9903 Method:spine Processes:2 Threads:40 Hosts:145 HostsPerProcess:73 DataSources:31187 RRDsProcessed:10786
03/04/2009 05:53:10 PM - SYSTEM STATS: Time:9.1489 Method:spine Processes:2 Threads:40 Hosts:145 HostsPerProcess:73 DataSources:31187 RRDsProcessed:10786




only 1/3 of hosts have been added at the moment, going slow to make sure nothing goes wrong
Top
User avatar
TheWitness
Developer
Posts: 17007
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:
Contact TheWitness

Post by TheWitness »

At some point you will need boost.

TheWitness
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages

For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Top
Jacket
Posts: 41
Joined: Tue Sep 30, 2008 11:38 pm
Location: Sofia, Bulgaria

Post by Jacket »

Code: Select all

03/05/2009 07:04:22 AM - SYSTEM STATS: Time:20.3342 Method:spine Processes:4 Threads:10 Hosts:1494 HostsPerProcess:374 DataSources:58904 RRDsProcessed:0
03/05/2009 07:03:21 AM - SYSTEM STATS: Time:20.2148 Method:spine Processes:4 Threads:10 Hosts:1494 HostsPerProcess:374 DataSources:58903 RRDsProcessed:0
03/05/2009 07:02:22 AM - SYSTEM STATS: Time:20.5017 Method:spine Processes:4 Threads:10 Hosts:1494 HostsPerProcess:374 DataSources:58903 RRDsProcessed:0
03/05/2009 07:01:21 AM - SYSTEM STATS: Time:20.3234 Method:spine Processes:4 Threads:10 Hosts:1494 HostsPerProcess:374 DataSources:58903 RRDsProcessed:0
03/05/2009 07:00:22 AM - SYSTEM STATS: Time:20.3463 Method:spine Processes:4 Threads:10 Hosts:1494 HostsPerProcess:374 DataSources:58903 RRDsProcessed:0
03/05/2009 06:59:21 AM - SYSTEM STATS: Time:20.3226 Method:spine Processes:4 Threads:10 Hosts:1494 HostsPerProcess:374 DataSources:58904 RRDsProcessed:0
03/05/2009 06:58:22 AM - SYSTEM STATS: Time:20.2752 Method:spine Processes:4 Threads:10 Hosts:1494 HostsPerProcess:374 DataSources:58903 RRDsProcessed:0
03/05/2009 06:57:21 AM - SYSTEM STATS: Time:20.1203 Method:spine Processes:4 Threads:10 Hosts:1494 HostsPerProcess:374 DataSources:58903 RRDsProcessed:0

Code: Select all

03/05/2009 06:54:48 AM - SYSTEM BOOST STATS: Time:261.9606 RRDUpdates:3583637
03/05/2009 05:53:30 AM - SYSTEM BOOST STATS: Time:248.3017 RRDUpdates:3584929
03/05/2009 04:52:27 AM - SYSTEM BOOST STATS: Time:244.7419 RRDUpdates:3525397
03/05/2009 03:52:27 AM - SYSTEM BOOST STATS: Time:245.2317 RRDUpdates:3524961
03/05/2009 02:52:40 AM - SYSTEM BOOST STATS: Time:258.7262 RRDUpdates:3584674
CPU: Core2Duo E8600 @3.3Ghz (this is the bottleneck sometimes)
RAM: 8G
HDD: 2xWDC WD1500HLFS @10k RPM in RAID
Supermicro motherboard

Running Debian 5.0-lenny with kernel 2.6.26-1-686-bigmem
RRDTool 1.2.28.

Do you think switching to RRDTool 1.3.x will improve something?
[size=92][i]"Ubuntu" - an African word, meaning "Slackware is too hard for me".[/i][/size]
Top
jbarbieri
Cacti User
Posts: 101
Joined: Tue May 24, 2005 8:46 am
Location: Rochester, New Hampshire
Contact:
Contact jbarbieri

Post by jbarbieri »

I am beginning to think my slowdown is because my RRAs are on an NFS mount :/
Top
Jacket
Posts: 41
Joined: Tue Sep 30, 2008 11:38 pm
Location: Sofia, Bulgaria

Post by Jacket »

Storage speed is crucial for cacti performance, unless you use the boost plugin.
Do "iostat 1" on the NFS host and see how many TPS it's doing when cacti poller runs. If it's something below 300 this may be your problem.
Usually PATA/SATA disks at 7200RPM make about 300-400 transactions per second.
My 10k RPM Velociraptors top out at about 1000-1300 TPS each, which gives an average of 2000-2100 TPS when working in RAID, and still this isn't enough for cacti when writing 50k+ DSs.
Also look at your iowaits when cacti polls. If you have large percent of iowaits during polls this means your storage is slowing the poller process.
[size=92][i]"Ubuntu" - an African word, meaning "Slackware is too hard for me".[/i][/size]
Top
jbarbieri
Cacti User
Posts: 101
Joined: Tue May 24, 2005 8:46 am
Location: Rochester, New Hampshire
Contact:
Contact jbarbieri

Post by jbarbieri »

iostat 1 doesn't work for NFS mounts.


had to do iostat -xn 5 to get it to display:

Code: Select all

                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    6.2    0.0  118.5  0.0  0.0    0.3    0.6   0   0 172.17.4.6:/vol/mrtg01/mrtg01
  131.3  160.5  630.1 1372.4  0.0  0.7    0.0    2.3   0  62 172.17.4.6:/vol/mrtg01/mrtg01
  194.4  198.8  973.6 1625.0  0.0  1.0    0.0    2.7   0  95 172.17.4.6:/vol/mrtg01/mrtg01
   51.7   79.6  446.1  808.4  0.0  0.3    0.0    2.5   0  27 172.17.4.6:/vol/mrtg01/mrtg01
  188.6  145.8  937.5  728.3  0.0  1.0    0.0    3.0   0  96 172.17.4.6:/vol/mrtg01/mrtg01
  188.0  135.2 1009.6  648.1  0.0  1.0    0.0    3.1   0  96 172.17.4.6:/vol/mrtg01/mrtg01
  206.0  150.0  990.5  708.3  0.0  0.9    0.0    2.6   0  87 172.17.4.6:/vol/mrtg01/mrtg01
  181.8  138.4  923.2  707.2  0.0  1.0    0.0    3.1   0  94 172.17.4.6:/vol/mrtg01/mrtg01
  231.2  173.4 1130.5  789.3  0.0  1.0    0.0    2.5   0  95 172.17.4.6:/vol/mrtg01/mrtg01
  208.4  152.6 1012.8  711.0  0.0  1.0    0.0    2.8   0  95 172.17.4.6:/vol/mrtg01/mrtg01
  253.0  205.0 1307.3  946.4  0.0  1.0    0.0    2.2   0  94 172.17.4.6:/vol/mrtg01/mrtg01
  181.4  132.2  899.2  628.4  0.0  1.0    0.0    3.2   0  95 172.17.4.6:/vol/mrtg01/mrtg01
  146.8  165.8  753.7 1359.5  0.0  1.0    0.0    3.1   0  90 172.17.4.6:/vol/mrtg01/mrtg01
  186.4  160.0  958.4 1079.5  0.0  1.0    0.0    2.9   0  96 172.17.4.6:/vol/mrtg01/mrtg01
  146.8  105.2  787.3  501.8  0.0  1.0    0.0    3.9   0  96 172.17.4.6:/vol/mrtg01/mrtg01
  293.0  228.0 1508.0  994.7  0.0  1.0    0.0    2.0   0  93 172.17.4.6:/vol/mrtg01/mrtg01
  189.2  138.6  912.1  731.1  0.0  1.0    0.0    3.1   0  96 172.17.4.6:/vol/mrtg01/mrtg01
  204.2  156.8 1012.8  707.0  0.0  1.0    0.0    2.8   0  95 172.17.4.6:/vol/mrtg01/mrtg01
  202.8  149.6 1044.9  698.2  0.0  1.0    0.0    2.8   0  95 172.17.4.6:/vol/mrtg01/mrtg01
  240.6  189.4 1175.2  898.8  0.0  1.0    0.0    2.3   0  93 172.17.4.6:/vol/mrtg01/mrtg01
  204.6  147.6 1024.1  700.9  0.0  1.0    0.0    2.9   0  96 172.17.4.6:/vol/mrtg01/mrtg01
  199.8  144.2 1028.8  691.0  0.0  1.0    0.0    2.9   0  95 172.17.4.6:/vol/mrtg01/mrtg01
  183.8  150.8  968.8  904.2  0.0  1.0    0.0    2.9   0  91 172.17.4.6:/vol/mrtg01/mrtg01
  133.8  102.0  708.9  460.6  0.0  1.0    0.0    4.2   0  96 172.17.4.6:/vol/mrtg01/mrtg01
  139.2  153.2  803.3 1435.6  0.0  1.0    0.0    3.6   0  88 172.17.4.6:/vol/mrtg01/mrtg01
  180.6  135.4  918.4  707.4  0.0  1.0    0.0    3.2   0  96 172.17.4.6:/vol/mrtg01/mrtg01
  164.0  123.8  811.2  566.5  0.0  1.0    0.0    3.5   0  94 172.17.4.6:/vol/mrtg01/mrtg01
  138.0  102.4  708.7  483.6  0.0  0.8    0.0    3.5   0  81 172.17.4.6:/vol/mrtg01/mrtg01
    0.4    5.0    0.1   89.1  0.0  0.0    0.0    0.6   0   0 172.17.4.6:/vol/mrtg01/mrtg01
    0.0    9.6    0.0    3.6  0.0  0.0    0.0    0.3   0   0 172.17.4.6:/vol/mrtg01/mrtg01
This is for one poll cycle, every 5 seconds.
Top
Jacket
Posts: 41
Joined: Tue Sep 30, 2008 11:38 pm
Location: Sofia, Bulgaria

Post by Jacket »

That's why i said to run iostat on the NFS host :)
Here's my -xn output (only the important lines), while boost is updating the rrd files (similar to when cacti poller runs w/o boost):
md0 is the raid volume.

Code: Select all

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util 
sda               0.00   541.00    0.00 1723.00     0.00 18232.00    10.58    98.94   57.01   0.58 100.00 
sdb               0.00   571.00    0.00 1690.00     0.00 18080.00    10.70    98.50   56.62   0.59 100.00 
md0               0.00     0.00    0.00 4473.00     0.00 35784.00     8.00     0.00    0.00   0.00   0.00
I think "w/s" is similar to "tps", but i'm not sure. I've never user NFS before.
Last edited by Jacket on Thu Mar 05, 2009 8:34 am, edited 1 time in total.
[size=92][i]"Ubuntu" - an African word, meaning "Slackware is too hard for me".[/i][/size]
Top
jbarbieri
Cacti User
Posts: 101
Joined: Tue May 24, 2005 8:46 am
Location: Rochester, New Hampshire
Contact:
Contact jbarbieri

Post by jbarbieri »

I couldn't run it on the NFS host.


w/s and r/s is write transactions per second, and read transactions per second, so yes, I am guessing the same as TPS.
Top
User avatar
TheWitness
Developer
Posts: 17007
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:
Contact TheWitness

Post by TheWitness »

The only way to boost performance on NFS is to parallelize the update process. I did write a patch for just that, but did not go further with it due simply to the issues with disk caching with NFS.

TheWitness
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages

For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Top
jbarbieri
Cacti User
Posts: 101
Joined: Tue May 24, 2005 8:46 am
Location: Rochester, New Hampshire
Contact:
Contact jbarbieri

Post by jbarbieri »

New Server, new results (still migrating all my devices, one at a time)

Code: Select all

04/13/2009 03:22:11 PM - SYSTEM STATS: Time:129.0801 Method:cmd.php Processes:15 Threads:N/A Hosts:24 HostsPerProcess:2 DataSources:5048 RRDsProcessed:2163


04/13/2009 04:10:07 PM - SYSTEM STATS: Time:6.0652 Method:spine Processes:15 Threads:25 Hosts:24 HostsPerProcess:2 DataSources:5048 RRDsProcessed:2163
04/13/2009 04:15:12 PM - SYSTEM STATS: Time:11.0515 Method:spine Processes:15 Threads:25 Hosts:24 HostsPerProcess:2 DataSources:5048 RRDsProcessed:2163

Quad Xeon 3.4Ghz with 2GB RAM.
Cacti 0.8.7d and Spine 0.8.7c
Top
tomslick42
Posts: 5
Joined: Mon Jun 01, 2009 12:13 pm
Location: York, Nebraska

Super Fast!?

Post by tomslick42 »

IBM x345, 2xXeon 2.8GHz w/hyperthreading (4 total procs)
Ubuntu 7.10, 4GB RAM, U320 10k SCSI

Cacti 0.8.7b, RRDTool 1.2, Hosts 178, Graphs 5053
Poller Interval 300, Processes 2, Threads 20, PHP Servers 10, Max OID 25

Time:11.4981 Method:spine Processes:2 Threads:20 Hosts:177 HostsPerProcess:89 DataSources:10502 RRDsProcessed:5341

I tried 3 processes and that tripled the time, so did threads at 30.

The machine is running nagios and PHP Syslog for most of these devices, so I can't say I'm unhappy with these results at all!

(I was at 200 seconds a few days ago with 2GB RAM)
Thomas Schenck
Top
User avatar
zorrosam
Cacti User
Posts: 244
Joined: Thu May 03, 2007 3:17 pm
Location: Italy

advice on my poor performance?!

Post by zorrosam »

Hi guys,

any advice for my situation?

06/22/2009 09:49:30 AM - SYSTEM STATS: Time:269.0361 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7065

DELL Poweredge 1950
2xQUADCORE 2.00 GHZ XEON
8GB RAM
2xHDD 250GB RAID1
MYSQL ACTIVE ON ANOTHER SERVER TUNED

I don't know if will be better activate boost on my side ... the system is on production and fundamental for all my bussiness. There's the way for do that without downtime?

bye

sam
Top
Post Reply

Return to “Announcements”

Who is online

Users browsing this forum: No registered users and 2 guests