ucd/net snmp Device (Disk) I/O templates - Updated - v3.1

Rodre · Post by **Rodre** » Fri Nov 26, 2010 2:16 pm

can you give us some details of your system platform and Cacti version?

~Rodre

Emilien · Post by **Emilien** » Fri Nov 26, 2010 3:40 pm

Rodre wrote:can you give us some details of your system platform and Cacti version?

~Rodre

Sorry for the lack infos, I have update today cacti to the latest version from 0.8.7e:

Cacti Version 	0.8.7g
Cacti OS 	unix
SNMP Version 	NET-SNMP version: 5.4.3
RRDTool Version 	RRDTool 1.3.x
Hosts 	7
Graphs 	114
Data Sources 	Script/Command: 1
SNMP: 45
SNMP Query: 44
Script Query: 12
Script - Script Server (PHP): 6
Script Query - Script Server: 46
Total: 154
Poller Information
Interval 	300
Type 	spine
Items 	Action[0]: 122
Action[1]: 7
Action[2]: 66
Total: 195
Concurrent Processes 	3
Max Threads 	10
PHP Servers 	10
Script Timeout 	15
Max OID 	30
Last Run Statistics 	Time:298.8428 Method:spine Processes:3 Threads:10 Hosts:7 HostsPerProcess:3 DataSources:195 RRDsProcessed:0
PHP Information
PHP Version 	5.2.14-pl0-gentoo
PHP OS 	Linux
PHP uname 	Linux Server 2.6.36-gentoo #1 SMP x86_64
PHP SNMP 	Installed
max_execution_time 	30
memory_limit 	128M

Now I see one different lines in the logs:

Code: Select all

11/26/2010 09:35:02 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
11/26/2010 09:35:02 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
11/26/2010 09:35:02 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
11/26/2010 09:35:02 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
11/26/2010 09:35:02 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
11/26/2010 09:35:02 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
11/26/2010 09:35:02 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
11/26/2010 09:35:02 PM - SPINE: Poller[0] FATAL: Spine Encountered a Segmentation Fault (Spine thread)
11/26/2010 09:35:02 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
11/26/2010 09:35:02 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
11/26/2010 09:35:02 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
11/26/2010 09:35:02 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
11/26/2010 09:35:02 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
11/26/2010 09:35:02 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
11/26/2010 09:35:02 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
11/26/2010 09:35:02 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
11/26/2010 09:35:02 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating

Segmentation Fault

And in dmesg a lot of:

Code: Select all

[2218731.695639] spine[20148]: segfault at 75 ip 00007fbb6543c7c7 sp 00007fbb64214800 error 4 in libgcc_s.so.1[7fbb6542d000+15000]
[2219030.923433] spine[20361]: segfault at 75 ip 00007fd44120e7c7 sp 00007fd440fe8800 error 4 in libgcc_s.so.1[7fd4411ff000+15000]
[2219630.885697] spine[22958]: segfault at 75 ip 00007f97e1bd57c7 sp 00007f97e19af800 error 4 in libgcc_s.so.1[7f97e1bc6000+15000]
[2219930.980621] spine[23157]: segfault at 75 ip 00007fb97cbdb7c7 sp 00007fb97c9b5800 error 4 in libgcc_s.so.1[7fb97cbcc000+15000]
[2220531.015967] spine[23761]: segfault at 75 ip 00007f80790217c7 sp 00007f8078dfb800 error 4 in libgcc_s.so.1[7f8079012000+15000]
[2220830.931487] spine[23943]: segfault at 75 ip 00007f1dd73ae7c7 sp 00007f1dd7188800 error 4 in libgcc_s.so.1[7f1dd739f000+15000]
[2221130.915621] spine[24175]: segfault at 75 ip 00007ff3116ef7c7 sp 00007ff3114c9800 error 4 in libgcc_s.so.1[7ff3116e0000+15000]
[2221430.932212] spine[24572]: segfault at 75 ip 00007f0715fa87c7 sp 00007f0714d80800 error 4 in libgcc_s.so.1[7f0715f99000+15000]
[2222030.998699] spine[24894]: segfault at 75 ip 00007f27a58b77c7 sp 00007f279fff6800 error 4 in libgcc_s.so.1[7f27a58a8000+15000]

Any suggestion?
With before version there aren't any errors.

Best regards,

Emilien

Rodre · Post by **Rodre** » Fri Nov 26, 2010 4:49 pm

I don''t think the PHP errors are related as this is 100% SNMP based and doesn't use any custom scripts for data collection. Also, I am not sure why you are getting the GCC library errors, I don't think that is related either. What types of systems are you polling with this? I have never used this in conjunction with spine. Can you give me temp access to the cacti gui as admin?

~Rodre

Emilien · Post by **Emilien** » Fri Nov 26, 2010 4:52 pm

Rodre wrote:I don''t think the PHP errors are related. as this is 100% SNMP based. Also, I am not sure why you are getting the GCC library errors, I don't think that is related either. What types of systems are you polling with this?

~Rodre

The server with cacti run gentoo with net-snmp-5.4.3 and the clients have gentoo (net-snmp-5.4.3) and fedora 14 (net-snmp-5.5).
With the segfault the data aren't grabbed, but for a while (before add diskio) spine-0.8.7g working fine.
Now I downgrade only spine and check if there are always segfault

Best regards,

Emilien

Rodre · Post by **Rodre** » Fri Nov 26, 2010 5:03 pm

Have you applied any of the disk io graphs to any of your hosts (devices)?

~Rodre

Emilien · Post by **Emilien** » Fri Nov 26, 2010 5:27 pm

Rodre wrote:Have you applied any of the disk io graphs to any of your hosts (devices)?

~Rodre

Yes, two gentoo and one fedora.
But now I have downgrade spine and if I launch spine I see this output:

Code: Select all

SPINE: Using spine config file [/etc/spine.conf]
SPINE: Version 0.8.7e starting
*** glibc detected *** spine: free(): invalid next size (fast): 0x00000000017c0b80 ***
======= Backtrace: =========
/lib/libc.so.6(+0x72966)[0x7fd8b4c15966]
/lib/libc.so.6(cfree+0x6c)[0x7fd8b4c1a71c]
/usr/lib/libnetsnmp.so.15(generate_Ku+0x6b)[0x7fd8b5b1341b]
spine[0x406f00]
spine[0x40aad2]
spine[0x40ac53]
/lib/libpthread.so.0(+0x6894)[0x7fd8b5183894]
/lib/libc.so.6(clone+0x6d)[0x7fd8b4c7427d]
======= Memory map: ========
00400000-00414000 r-xp 00000000 09:02 1719705                            /usr/sbin/spine
00613000-00614000 r--p 00013000 09:02 1719705                            /usr/sbin/spine
00614000-00615000 rw-p 00014000 09:02 1719705                            /usr/sbin/spine
00615000-00628000 rw-p 00000000 00:00 0
016d5000-0181d000 rw-p 00000000 00:00 0                                  [heap]
7fd8ac000000-7fd8ac023000 rw-p 00000000 00:00 0
7fd8ac023000-7fd8b0000000 ---p 00000000 00:00 0
7fd8b057f000-7fd8b0580000 ---p 00000000 00:00 0
7fd8b0580000-7fd8b0d80000 rw-p 00000000 00:00 0
7fd8b0d80000-7fd8b0d81000 ---p 00000000 00:00 0
7fd8b0d81000-7fd8b1581000 rw-p 00000000 00:00 0
7fd8b1581000-7fd8b1582000 ---p 00000000 00:00 0
7fd8b1582000-7fd8b1d82000 rw-p 00000000 00:00 0
7fd8b1d82000-7fd8b1d83000 ---p 00000000 00:00 0
7fd8b1d83000-7fd8b2583000 rw-p 00000000 00:00 0
7fd8b2583000-7fd8b2584000 ---p 00000000 00:00 0
7fd8b2584000-7fd8b2d84000 rw-p 00000000 00:00 0
7fd8b2d84000-7fd8b2d85000 ---p 00000000 00:00 0
7fd8b2d85000-7fd8b3585000 rw-p 00000000 00:00 0
7fd8b3585000-7fd8b3590000 r-xp 00000000 09:02 2622087                    /lib64/libnss_files-2.11.2.so
7fd8b3590000-7fd8b3790000 ---p 0000b000 09:02 2622087                    /lib64/libnss_files-2.11.2.so
7fd8b3790000-7fd8b3791000 r--p 0000b000 09:02 2622087                    /lib64/libnss_files-2.11.2.so
7fd8b3791000-7fd8b3792000 rw-p 0000c000 09:02 2622087                    /lib64/libnss_files-2.11.2.so
7fd8b3792000-7fd8b37a7000 r-xp 00000000 09:02 2621598                    /lib64/libgcc_s.so.1
7fd8b37a7000-7fd8b39a7000 ---p 00015000 09:02 2621598                    /lib64/libgcc_s.so.1
7fd8b39a7000-7fd8b39a8000 r--p 00015000 09:02 2621598                    /lib64/libgcc_s.so.1
7fd8b39a8000-7fd8b39a9000 rw-p 00016000 09:02 2621598                    /lib64/libgcc_s.so.1
7fd8b39a9000-7fd8b39aa000 ---p 00000000 00:00 0
7fd8b39aa000-7fd8b41aa000 rw-p 00000000 00:00 0
7fd8b41aa000-7fd8b41ac000 r-xp 00000000 09:02 2622096                    /lib64/libdl-2.11.2.so
7fd8b41ac000-7fd8b43ac000 ---p 00002000 09:02 2622096                    /lib64/libdl-2.11.2.so
7fd8b43ac000-7fd8b43ad000 r--p 00002000 09:02 2622096                    /lib64/libdl-2.11.2.so
7fd8b43ad000-7fd8b43ae000 rw-p 00003000 09:02 2622096                    /lib64/libdl-2.11.2.so
7fd8b43ae000-7fd8b43fa000 r-xp 00000000 09:02 1578996                    /usr/lib64/libssl.so.0.9.8
7fd8b43fa000-7fd8b45fa000 ---p 0004c000 09:02 1578996                    /usr/lib64/libssl.so.0.9.8
7fd8b45fa000-7fd8b45fc000 r--p 0004c000 09:02 1578996                    /usr/lib64/libssl.so.0.9.8
7fd8b45fc000-7fd8b4602000 rw-p 0004e000 09:02 1578996                    /usr/lib64/libssl.so.0.9.8
7fd8b4602000-7fd8b4616000 r-xp 00000000 09:02 2621578                    /lib64/libz.so.1.2.3
7fd8b4616000-7fd8b4816000 ---p 00014000 09:02 2621578                    /lib64/libz.so.1.2.3
7fd8b4816000-7fd8b4817000 r--p 00014000 09:02 2621578                    /lib64/libz.so.1.2.3
7fd8b4817000-7fd8b4818000 rw-p 00015000 09:02 2621578                    /lib64/libz.so.1.2.3
7fd8b4818000-7fd8b497a000 r-xp 00000000 09:02 1578998                    /usr/lib64/libcrypto.so.0.9.8
7fd8b497a000-7fd8b4b79000 ---p 00162000 09:02 1578998                    /usr/lib64/libcrypto.so.0.9.8
7fd8b4b79000-7fd8b4b87000 r--p 00161000 09:02 1578998                    /usr/lib64/libcrypto.so.0.9.8
7fd8b4b87000-7fd8b4ba0000 rw-p 0016f000 09:02 1578998                    /usr/lib64/libcrypto.so.0.9.8
7fd8b4ba0000-7fd8b4ba3000 rw-p 00000000 00:00 0
7fd8b4ba3000-7fd8b4cf3000 r-xp 00000000 09:02 2622121                    /lib64/libc-2.11.2.so
7fd8b4cf3000-7fd8b4ef2000 ---p 00150000 09:02 2622121                    /lib64/libc-2.11.2.so
7fd8b4ef2000-7fd8b4ef6000 r--p 0014f000 09:02 2622121                    /lib64/libc-2.11.2.so
7fd8b4ef6000-7fd8b4ef7000 rw-p 00153000 09:02 2622121                    /lib64/libc-2.11.2.so
7fd8b4ef7000-7fd8b4efc000 rw-p 00000000 00:00 0
7fd8b4efc000-7fd8b4f7c000 r-xp 00000000 09:02 2622092                    /lib64/libm-2.11.2.so
7fd8b4f7c000-7fd8b517b000 ---p 00080000 09:02 2622092                    /lib64/libm-2.11.2.so
7fd8b517b000-7fd8b517c000 r--p 0007f000 09:02 2622092                    /lib64/libm-2.11.2.so
7fd8b517c000-7fd8b517d000 rw-p 00080000 09:02 2622092                    /lib64/libm-2.11.2.so
7fd8b517d000-7fd8b5194000 r-xp 00000000 09:02 2622088                    /lib64/libpthread-2.11.2.so
7fd8b5194000-7fd8b5393000 ---p 00017000 09:02 2622088                    /lib64/libpthread-2.11.2.so
7fd8b5393000-7fd8b5394000 r--p 00016000 09:02 2622088                    /lib64/libpthread-2.11.2.so
7fd8b5394000-7fd8b5395000 rw-p 00017000 09:02 2622088                    /lib64/libpthread-2.11.2.so
7fd8b5395000-7fd8b5399000 rw-p 00000000 00:00 0
7fd8b5399000-7fd8b5519000 r-xp 00000000 09:02 1578412                    /usr/lib64/libcrypto.so.1.0.0
7fd8b5519000-7fd8b5719000 ---p 00180000 09:02 1578412                    /usr/lib64/libcrypto.so.1.0.0
7fd8b5719000-7fd8b5732000 r--p 00180000 09:02 1578412                    /usr/lib64/libcrypto.so.1.0.0
7fd8b5732000-7fd8b573c000 rw-p 00199000 09:02 1578412                    /usr/lib64/libcrypto.so.1.0.0
7fd8b573c000-7fd8b5740000 rw-p 00000000 00:00 0
7fd8b5740000-7fd8b586f000 r-xp 00000000 09:02 2503160                    /usr/lib64/mysql/libmysqlclient_r.so.15.0.0
7fd8b586f000-7fd8b5a6f000 ---p 0012f000 09:02 2503160                    /usr/lib64/mysql/libmysqlclient_r.so.15.0.0
7fd8b5a6f000-7fd8b5a72000 r--p 0012f000 09:02 2503160                    /usr/lib64/mysql/libmysqlclient_r.so.15.0.0
7fd8b5a72000-7fd8b5ab7000 rw-p 00132000 09:02 2503160                    /usr/lib64/mysql/libmysqlclient_r.so.15.0.0
7fd8b5ab7000-7fd8b5ab9000 rw-p 00000000 00:00 0
7fd8b5ab9000-7fd8b5b4f000 r-xp 00000000 09:02 1579537                    /usr/lib64/libnetsnmp.so.15.1.2
7fd8b5b4f000-7fd8b5d4f000 ---p 00096000 09:02 1579537                    /usr/lib64/libnetsnmp.so.15.1.2
7fd8b5d4f000-7fd8b5d50000 r--p 00096000 09:02 1579537                    /usr/lib64/libnetsnmp.so.15.1.2
7fd8b5d50000-7fd8b5d53000 rw-p 00097000 09:02 1579537                    /usr/lib64/libnetsnmp.so.15.1.2
7fd8b5d53000-7fd8b5d88000 rw-p 00000000 00:00 0
7fd8b5d88000-7fd8b5da6000 r-xp 00000000 09:02 2622110                    /lib64/ld-2.11.2.so
7fd8b5dca000-7fd8b5f7d000 r--p 00000000 09:02 1837938                    /usr/lib64/locale/locale-archive
7fd8b5f7d000-7fd8b5f83000 rw-p 00000000 00:00 0
7fd8b5fa2000-7fd8b5fa5000 rw-p 00000000 00:00 0
7fd8b5fa5000-7fd8b5fa6000 r--p 0001d000 09:02 2622110                    /lib64/ld-2.11.2.so
7fd8b5fa6000-7fd8b5fa7000 rw-p 0001e000 09:02 2622110                    /lib64/ld-2.11.2.so
7fd8b5fa7000-7fd8b5fa8000 rw-p 00000000 00:00 0
7fff6b8de000-7fff6b900000 rw-p 00000000 00:00 0                          [stack]
7fff6b9ff000-7fff6ba00000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

It's a system glibc bug?

Best regards,

Emilien

Rodre · Post by **Rodre** » Fri Nov 26, 2010 5:38 pm

To me it looks like perhaps spine has a dependency on a specific GCC version or something. Have you tried to go back to the php poller and see iif that works?

~Rod

Emilien · Post by **Emilien** » Fri Nov 26, 2010 5:59 pm

Rodre wrote:To me it looks like perhaps spine has a dependency on a specific GCC version or something. Have you tried to go back to the php poller and see iif that works?

~Rod

Ok, I have found my mistake, I update openssl without recompile net-snmp that go segfault spine.
Now I have latest spine that working fine, and all graphs are restart to draw with update data also the diskio graphs

I apologize for having soiled this thread

Emilien

sblantipodi · Post by **sblantipodi** » Wed Aug 17, 2011 4:36 am

I have done all the steps explained on the first post and added a graph I selected
ucd / net Device I/O Read/Written and ucd / net Device I/O Load average
The graph isn't displayed at all, if I select the debug output the error is:
ERROR: the RRD does not contain an RRA matching the chosen CF

Why?

The only graph that work ok is the ucd / net Device I/O Bytes Read/Written

Another question,
is it possible to monitor the entire "disk /" instead of a single device like /dev/xvdc

Thanks.

noname · Post by **noname** » Wed Aug 17, 2011 11:08 pm

sblantipodi wrote:I have done all the steps explained on the first post and added a graph I selected
ucd / net Device I/O Read/Written and ucd / net Device I/O Load average
The graph isn't displayed at all, if I select the debug output the error is:
ERROR: the RRD does not contain an RRA matching the chosen CF

See this post: http://forums.cacti.net/viewtopic.php?p=132517#p132517 (and following 2 posts from gandalf)

sblantipodi wrote:Another question,
is it possible to monitor the entire "disk /" instead of a single device like /dev/xvdc

Try this: http://docs.cacti.net/usertemplate:data:host_mib:disk_usage

sblantipodi · Post by **sblantipodi** » Sat Aug 20, 2011 5:12 pm

Try this: http://docs.cacti.net/usertemplate:data:host_mib:disk_usage

I'm talking about disc IO, I want to monitor the disk I/O for / having only one graph.
I don't want to have a graph for each paritition.

psyber · Post by **psyber** » Sat Aug 20, 2011 11:10 pm

if / is on /dev/sda2 and /var is on /dev/sda3 its the same physical drive /dev/sda therefore the same I/O
I/O affects logical partitions on the same physical drive equally

sblantipodi · Post by **sblantipodi** » Sun Aug 21, 2011 11:07 am

psyber wrote:if / is on /dev/sda2 and /var is on /dev/sda3 its the same physical drive /dev/sda therefore the same I/O
I/O affects logical partitions on the same physical drive equally

I have /dev/xvda and /dev/xvdc
both this partitions are on the same disk, however xvda is used for OS, xvdc is used for storing the www directory.

Monitoring this two partitions with this plugin generate two completely different graphs.
Why?

psyber · Post by **psyber** » Sun Aug 21, 2011 3:05 pm

i'm speculating but /dev/xvd is a virtulized storage device so it could be separate physical media depends on what the backing is

sblantipodi · Post by **sblantipodi** » Wed Aug 24, 2011 7:43 am

psyber wrote:i'm speculating but /dev/xvd is a virtulized storage device so it could be separate physical media depends on what the backing is

yes I'm on a VPS and sincerely I don't know how to correctly monitor my I/O.

Cacti

ucd/net snmp Device (Disk) I/O templates - Updated - v3.1

Would you or your company pay someone to create more graph templates?

Re: ucd/net snmp Device (Disk) I/O templates - Updated - v3.

Re: ucd/net snmp Device (Disk) I/O templates - Updated - v3.

Re: ucd/net snmp Device (Disk) I/O templates - Updated - v3.

Re: ucd/net snmp Device (Disk) I/O templates - Updated - v3.

Re: ucd/net snmp Device (Disk) I/O templates - Updated - v3.

Re: ucd/net snmp Device (Disk) I/O templates - Updated - v3.

Re: ucd/net snmp Device (Disk) I/O templates - Updated - v3.

Re: ucd/net snmp Device (Disk) I/O templates - Updated - v3.

Re: ucd/net snmp Device (Disk) I/O templates - Updated - v3.

Re: ucd/net snmp Device (Disk) I/O templates - Updated - v3.

Re: ucd/net snmp Device (Disk) I/O templates - Updated - v3.

Re: ucd/net snmp Device (Disk) I/O templates - Updated - v3.

Re: ucd/net snmp Device (Disk) I/O templates - Updated - v3.

Re: ucd/net snmp Device (Disk) I/O templates - Updated - v3.

Re: ucd/net snmp Device (Disk) I/O templates - Updated - v3.

Who is online