Graphic Interupts

dyordanov · Post by **dyordanov** » Thu Apr 18, 2013 3:36 am

Hello,

I noticed some of my graphics have strange interrupts, The graphics with this problem are completely random (or I couldn't find any specific connection between them), On the same host, the same way created one of the graphics is ok and the other one looks like this.

: graph_image.php.png (24.31 KiB) Viewed 1339 times

: graph_image-1.php.png (25.77 KiB) Viewed 1339 times

And this one is on the same host working just fine

: graph_image-2.php.png (20.45 KiB) Viewed 1339 times

I looked at the Cacti Log File and couldn't find anything wrong so far,

We are using cmd.php as a poller every minute
The operating system is Fedora 17 kernel 3.8.4-102.fc17.x86_64
cacti version: 0.8.8a

Any ideas where should I go look next from this point to solve this problem ?

Post by **gandalf** » Sat Apr 27, 2013 4:12 pm

What about SYSTEM STATS? Does polling end within time?
R.

dyordanov · Post by **dyordanov** » Mon Apr 29, 2013 8:15 am

It looks like it ends within time

Code: Select all

04/29/2013 04:10:09 PM - SYSTEM STATS: Time:7.6215 Method:cmd.php Processes:30 Threads:N/A Hosts:47 HostsPerProcess:2 DataSources:1085 RRDsProcessed:667
04/29/2013 04:09:11 PM - SYSTEM STATS: Time:9.7320 Method:cmd.php Processes:30 Threads:N/A Hosts:47 HostsPerProcess:2 DataSources:1076 RRDsProcessed:666
04/29/2013 04:08:09 PM - SYSTEM STATS: Time:7.2769 Method:cmd.php Processes:30 Threads:N/A Hosts:47 HostsPerProcess:2 DataSources:1097 RRDsProcessed:678
04/29/2013 04:07:09 PM - SYSTEM STATS: Time:7.5084 Method:cmd.php Processes:30 Threads:N/A Hosts:47 HostsPerProcess:2 DataSources:1082 RRDsProcessed:671
04/29/2013 04:06:09 PM - SYSTEM STATS: Time:8.0669 Method:cmd.php Processes:30 Threads:N/A Hosts:47 HostsPerProcess:2 DataSources:1090 RRDsProcessed:671
04/29/2013 04:05:11 PM - SYSTEM STATS: Time:9.3887 Method:cmd.php Processes:30 Threads:N/A Hosts:47 HostsPerProcess:2 DataSources:1066 RRDsProcessed:656
04/29/2013 04:04:09 PM - SYSTEM STATS: Time:7.5112 Method:cmd.php Processes:30 Threads:N/A Hosts:47 HostsPerProcess:2 DataSources:1096 RRDsProcessed:677
04/29/2013 04:03:10 PM - SYSTEM STATS: Time:9.1414 Method:cmd.php Processes:30 Threads:N/A Hosts:47 HostsPerProcess:2 DataSources:1077 RRDsProcessed:667
04/29/2013 04:02:11 PM - SYSTEM STATS: Time:8.9248 Method:cmd.php Processes:30 Threads:N/A Hosts:47 HostsPerProcess:2 DataSources:1102 RRDsProcessed:682
04/29/2013 04:01:09 PM - SYSTEM STATS: Time:7.6452 Method:cmd.php Processes:30 Threads:N/A Hosts:47 HostsPerProcess:2 DataSources:1070 RRDsProcessed:660
04/29/2013 04:00:09 PM - SYSTEM STATS: Time:7.9322 Method:cmd.php Processes:30 Threads:N/A Hosts:47 HostsPerProcess:2 DataSources:1086 RRDsProcessed:667
04/29/2013 03:59:08 PM - SYSTEM STATS: Time:6.5162 Method:cmd.php Processes:30 Threads:N/A Hosts:47 HostsPerProcess:2 DataSources:1076 RRDsProcessed:666
04/29/2013 03:58:08 PM - SYSTEM STATS: Time:6.7474 Method:cmd.php Processes:30 Threads:N/A Hosts:47 HostsPerProcess:2 DataSources:1097 RRDsProcessed:678
04/29/2013 03:57:10 PM - SYSTEM STATS: Time:8.6103 Method:cmd.php Processes:30 Threads:N/A Hosts:47 HostsPerProcess:2 DataSources:1082 RRDsProcessed:671
04/29/2013 03:56:08 PM - SYSTEM STATS: Time:6.3167 Method:cmd.php Processes:30 Threads:N/A Hosts:47 HostsPerProcess:2 DataSources:1090 RRDsProcessed:671
04/29/2013 03:55:08 PM - SYSTEM STATS: Time:7.2702 Method:cmd.php Processes:30 Threads:N/A Hosts:47 HostsPerProcess:2 DataSources:1066 RRDsProcessed:656
04/29/2013 03:54:11 PM - SYSTEM STATS: Time:9.2905 Method:cmd.php Processes:30 Threads:N/A Hosts:47 HostsPerProcess:2 DataSources:1096 RRDsProcessed:677
04/29/2013 03:53:09 PM - SYSTEM STATS: Time:7.7348 Method:cmd.php Processes:30 Threads:N/A Hosts:47 HostsPerProcess:2 DataSources:1077 RRDsProcessed:667
04/29/2013 03:52:09 PM - SYSTEM STATS: Time:7.4685 Method:cmd.php Processes:30 Threads:N/A Hosts:47 HostsPerProcess:2 DataSources:1102 RRDsProcessed:682
04/29/2013 03:51:10 PM - SYSTEM STATS: Time:9.1974 Method:cmd.php Processes:30 Threads:N/A Hosts:47 HostsPerProcess:2 DataSources:1070 RRDsProcessed:660
04/29/2013 03:50:10 PM - SYSTEM STATS: Time:7.8973 Method:cmd.php Processes:30 Threads:N/A Hosts:47 HostsPerProcess:2 DataSources:1086 RRDsProcessed:667
04/29/2013 03:49:10 PM - SYSTEM STATS: Time:9.6082 Method:cmd.php Processes:30 Threads:N/A Hosts:47 HostsPerProcess:2 DataSources:1076 RRDsProcessed:666
04/29/2013 03:48:09 PM - SYSTEM STATS: Time:7.8883 Method:cmd.php Processes:30 Threads:N/A Hosts:47 HostsPerProcess:2 DataSources:1097 RRDsProcessed:678
04/29/2013 03:47:08 PM - SYSTEM STATS: Time:7.1203 Method:cmd.php Processes:30 Threads:N/A Hosts:47 HostsPerProcess:2 DataSources:1082 RRDsProcessed:671

Post by **gandalf** » Sun May 12, 2013 3:04 pm

Yep, that's fine. When running latest spine in verbosity 3 (medium), polling times per host should be added to cacti.log. Please post snippets of the log (as few as possible) that show what happens during one of those outages
R.

dyordanov · Post by **dyordanov** » Tue May 14, 2013 5:28 am

Hello,
I decided to change to poller type to spine to see what happens - no luck.
I took one of the graphs for a 10 minute interval where there is some outages like this and here are the logs:

Code: Select all

05/14/2013 12:30:02 PM - SPINE: Poller[0] Host[3] TH[1] DS[1795] SNMP: v2: 10.10.11.52, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.79, value: 41580369
05/14/2013 12:30:02 PM - SPINE: Poller[0] Host[3] TH[1] DS[1795] SNMP: v2: 10.10.11.52, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.79, value: 24617832
05/14/2013 12:32:02 PM - SPINE: Poller[0] Host[3] TH[1] DS[1795] SNMP: v2: 10.10.11.52, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.79, value: 41580723
05/14/2013 12:32:02 PM - SPINE: Poller[0] Host[3] TH[1] DS[1795] SNMP: v2: 10.10.11.52, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.79, value: 24618150
05/14/2013 12:34:01 PM - SPINE: Poller[0] Host[3] TH[1] DS[1795] SNMP: v2: 10.10.11.52, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.79, value: 41581089
05/14/2013 12:34:01 PM - SPINE: Poller[0] Host[3] TH[1] DS[1795] SNMP: v2: 10.10.11.52, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.79, value: 24618428
05/14/2013 12:36:02 PM - SPINE: Poller[0] Host[3] TH[1] DS[1795] SNMP: v2: 10.10.11.52, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.79, value: 41581509
05/14/2013 12:36:02 PM - SPINE: Poller[0] Host[3] TH[1] DS[1795] SNMP: v2: 10.10.11.52, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.79, value: 24618666
05/14/2013 12:38:02 PM - SPINE: Poller[0] Host[3] TH[1] DS[1795] SNMP: v2: 10.10.11.52, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.79, value: 41581719
05/14/2013 12:38:02 PM - SPINE: Poller[0] Host[3] TH[1] DS[1795] SNMP: v2: 10.10.11.52, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.79, value: 24618864

Code: Select all

05/14/2013 12:30:02 PM - SPINE: Poller[0] Host[3] TH[1] Total Time:  0.12 Seconds
05/14/2013 12:31:01 PM - SPINE: Poller[0] Host[3] TH[1] Total Time:   0.2 Seconds
05/14/2013 12:32:02 PM - SPINE: Poller[0] Host[3] TH[1] Total Time:  0.39 Seconds
05/14/2013 12:33:02 PM - SPINE: Poller[0] Host[3] TH[1] Total Time:  0.49 Seconds
05/14/2013 12:34:01 PM - SPINE: Poller[0] Host[3] TH[1] Total Time:  0.32 Seconds
05/14/2013 12:35:02 PM - SPINE: Poller[0] Host[3] TH[1] Total Time:  0.24 Seconds
05/14/2013 12:36:02 PM - SPINE: Poller[0] Host[3] TH[1] Total Time:  0.55 Seconds
05/14/2013 12:37:02 PM - SPINE: Poller[0] Host[3] TH[1] Total Time:  0.56 Seconds
05/14/2013 12:38:02 PM - SPINE: Poller[0] Host[3] TH[1] Total Time: 0.099 Seconds
05/14/2013 12:39:02 PM - SPINE: Poller[0] Host[3] TH[1] Total Time:  0.15 Seconds

The dump of the rrd file:

Code: Select all

<!-- 2013-05-14 12:30:00 EEST / 1368523800 --> <row><v>2.4750000000e+00</v><v>2.4872589532e+00</v></row>
<!-- 2013-05-14 12:32:00 EEST / 1368523920 --> <row><v>2.6689810924e+00</v><v>2.9668767507e+00</v></row>
<!-- 2013-05-14 12:34:00 EEST / 1368524040 --> <row><v>2.3196300187e+00</v><v>3.0493732493e+00</v></row>
<!-- 2013-05-14 12:36:00 EEST / 1368524160 --> <row><v>1.9861111111e+00</v><v>3.4962500000e+00</v></row>
<!-- 2013-05-14 12:38:00 EEST / 1368524280 --> <row><v>NaN</v><v>1.7502410468e+00</v></row>
<!-- 2013-05-14 12:30:00 EEST / 1368523800 --> <row><v>2.4750000000e+00</v><v>2.4872589532e+00</v></row>
<!-- 2013-05-14 12:32:00 EEST / 1368523920 --> <row><v>2.6689810924e+00</v><v>2.9668767507e+00</v></row>
<!-- 2013-05-14 12:34:00 EEST / 1368524040 --> <row><v>2.3196300187e+00</v><v>3.0493732493e+00</v></row>
<!-- 2013-05-14 12:36:00 EEST / 1368524160 --> <row><v>1.9861111111e+00</v><v>3.4962500000e+00</v></row>
<!-- 2013-05-14 12:38:00 EEST / 1368524280 --> <row><v>NaN</v><v>1.7502410468e+00</v></row>
<!-- 2013-05-14 12:36:00 EEST / 1368524160 --> <row><v>2.6846921102e+00</v><v>3.1699871615e+00</v></row>
<!-- 2013-05-14 12:36:00 EEST / 1368524160 --> <row><v>2.4524714441e+00</v><v>3.1953621140e+00</v></row>
<!-- 2013-05-14 12:30:00 EEST / 1368523800 --> <row><v>2.4750000000e+00</v><v>2.4872589532e+00</v></row>
<!-- 2013-05-14 12:32:00 EEST / 1368523920 --> <row><v>2.6689810924e+00</v><v>2.9668767507e+00</v></row>
<!-- 2013-05-14 12:34:00 EEST / 1368524040 --> <row><v>2.3196300187e+00</v><v>3.0493732493e+00</v></row>
<!-- 2013-05-14 12:36:00 EEST / 1368524160 --> <row><v>1.9861111111e+00</v><v>3.4962500000e+00</v></row>
<!-- 2013-05-14 12:38:00 EEST / 1368524280 --> <row><v>NaN</v><v>1.7502410468e+00</v></row>
<!-- 2013-05-14 12:30:00 EEST / 1368523800 --> <row><v>2.4750000000e+00</v><v>2.4872589532e+00</v></row>
<!-- 2013-05-14 12:32:00 EEST / 1368523920 --> <row><v>2.6689810924e+00</v><v>2.9668767507e+00</v></row>
<!-- 2013-05-14 12:34:00 EEST / 1368524040 --> <row><v>2.3196300187e+00</v><v>3.0493732493e+00</v></row>
<!-- 2013-05-14 12:36:00 EEST / 1368524160 --> <row><v>1.9861111111e+00</v><v>3.4962500000e+00</v></row>
<!-- 2013-05-14 12:38:00 EEST / 1368524280 --> <row><v>NaN</v><v>1.7502410468e+00</v></row>
<!-- 2013-05-14 12:36:00 EEST / 1368524160 --> <row><v>3.9737383287e+00</v><v>5.2551313025e+00</v></row>
<!-- 2013-05-14 12:36:00 EEST / 1368524160 --> <row><v>3.9737383287e+00</v><v>5.5988888889e+00</v></row>

and the info of the rrd file

Code: Select all

rrd_version = "0003"
step = 120
last_update = 1368527162
header_size = 3488
ds[traffic_in].index = 0
ds[traffic_in].type = "DERIVE"
ds[traffic_in].minimal_heartbeat = 120
ds[traffic_in].min = 0.0000000000e+00
ds[traffic_in].max = 1.0000000000e+09
ds[traffic_in].last_ds = "24625159"
ds[traffic_in].value = 3.9666666667e+00
ds[traffic_in].unknown_sec = 0
ds[traffic_out].index = 1
ds[traffic_out].type = "DERIVE"
ds[traffic_out].minimal_heartbeat = 240
ds[traffic_out].min = 0.0000000000e+00
ds[traffic_out].max = 1.0000000000e+09
ds[traffic_out].last_ds = "41589765"
ds[traffic_out].value = 4.4333333333e+00
ds[traffic_out].unknown_sec = 0
rra[0].cf = "AVERAGE"
rra[0].rows = 500
rra[0].cur_row = 403
rra[0].pdp_per_row = 1
rra[0].xff = 5.0000000000e-01
rra[0].cdp_prep[0].value = NaN
rra[0].cdp_prep[0].unknown_datapoints = 0
rra[0].cdp_prep[1].value = NaN
rra[0].cdp_prep[1].unknown_datapoints = 0
rra[1].cf = "AVERAGE"
rra[1].rows = 600
rra[1].cur_row = 565
rra[1].pdp_per_row = 1
rra[1].xff = 5.0000000000e-01
rra[1].cdp_prep[0].value = NaN
rra[1].cdp_prep[0].unknown_datapoints = 0
rra[1].cdp_prep[1].value = NaN
rra[1].cdp_prep[1].unknown_datapoints = 0
rra[2].cf = "AVERAGE"
rra[2].rows = 700
rra[2].cur_row = 510
rra[2].pdp_per_row = 6
rra[2].xff = 5.0000000000e-01
rra[2].cdp_prep[0].value = 1.9833333333e+00
rra[2].cdp_prep[0].unknown_datapoints = 0
rra[2].cdp_prep[1].value = 2.2659481175e+00
rra[2].cdp_prep[1].unknown_datapoints = 0
rra[3].cf = "AVERAGE"
rra[3].rows = 775
rra[3].cur_row = 372
rra[3].pdp_per_row = 24
rra[3].xff = 5.0000000000e-01
rra[3].cdp_prep[0].value = 1.9833333333e+00
rra[3].cdp_prep[0].unknown_datapoints = 0
rra[3].cdp_prep[1].value = 2.2659481175e+00
rra[3].cdp_prep[1].unknown_datapoints = 0
rra[4].cf = "AVERAGE"
rra[4].rows = 797
rra[4].cur_row = 438
rra[4].pdp_per_row = 288
rra[4].xff = 5.0000000000e-01
rra[4].cdp_prep[0].value = 2.5366643849e+02
rra[4].cdp_prep[0].unknown_datapoints = 50
rra[4].cdp_prep[1].value = 4.9635576905e+02
rra[4].cdp_prep[1].unknown_datapoints = 8
rra[5].cf = "MAX"
rra[5].rows = 500
rra[5].cur_row = 362
rra[5].pdp_per_row = 1
rra[5].xff = 5.0000000000e-01
rra[5].cdp_prep[0].value = NaN
rra[5].cdp_prep[0].unknown_datapoints = 0
rra[5].cdp_prep[1].value = NaN
rra[5].cdp_prep[1].unknown_datapoints = 0
rra[6].cf = "MAX"
rra[6].rows = 600
rra[6].cur_row = 94
rra[6].pdp_per_row = 1
rra[6].xff = 5.0000000000e-01
rra[6].cdp_prep[0].value = NaN
rra[6].cdp_prep[0].unknown_datapoints = 0
rra[6].cdp_prep[1].value = NaN
rra[6].cdp_prep[1].unknown_datapoints = 0
rra[7].cf = "MAX"
rra[7].rows = 700
rra[7].cur_row = 124
rra[7].pdp_per_row = 6
rra[7].xff = 5.0000000000e-01
rra[7].cdp_prep[0].value = 1.9833333333e+00
rra[7].cdp_prep[0].unknown_datapoints = 0
rra[7].cdp_prep[1].value = 2.2659481175e+00
rra[7].cdp_prep[1].unknown_datapoints = 0
rra[8].cf = "MAX"
rra[8].rows = 775
rra[8].cur_row = 571
rra[8].pdp_per_row = 24
rra[8].xff = 5.0000000000e-01
rra[8].cdp_prep[0].value = 1.9833333333e+00
rra[8].cdp_prep[0].unknown_datapoints = 0
rra[8].cdp_prep[1].value = 2.2659481175e+00
rra[8].cdp_prep[1].unknown_datapoints = 0
rra[9].cf = "MAX"
rra[9].rows = 797
rra[9].cur_row = 368
rra[9].pdp_per_row = 288
rra[9].xff = 5.0000000000e-01
rra[9].cdp_prep[0].value = 3.9737383287e+00
rra[9].cdp_prep[0].unknown_datapoints = 50
rra[9].cdp_prep[1].value = 8.3078053260e+00
rra[9].cdp_prep[1].unknown_datapoints = 8

I guess this "NaN" values written in the rrd file are the problem, but I don't know why there is "NaN" considering the poller is getting values every time

dyordanov · Post by **dyordanov** » Tue May 14, 2013 6:10 am

After digging abit more I think I might found the problem,
the heartbeat for traffic_in of this datasource seems incorrect, I will try to increase it to 240 and see what happens

camerabob · Post by **camerabob** » Thu Mar 26, 2020 9:36 am

Still? This is your first post on a 7 year old thread. Please start a new one and give some details.

Osiris · Post by **Osiris** » Thu Mar 26, 2020 3:51 pm

Totally with that.

Cacti

Graphic Interupts

Graphic Interupts

Re: Graphic Interupts

Re: Graphic Interupts

Re: Graphic Interupts

Re: Graphic Interupts

Re: Graphic Interupts

Re: Graphic Interupts

Re: Graphic Interupts

Who is online