Boost Upgrade in 3.0 resulting gaps in graphs

[elfada]

Code: Select all

mysql> SELECT * FROM poller_output_boost WHERE local_data_id=4527 AND time<=FROM_UNIXTIME(1272458412+800) AND time>=FROM_UNIXTIME(1272458412-800);
Empty set (0.01 sec)

[elfada]

Code: Select all

mysql> SELECT * FROM poller_output_boost WHERE local_data_id=32461 AND time<=FROM_UNIXTIME(1272548103+800) AND time>=FROM_UNIXTIME(1272548103-800);
+---------------+-------------+---------------------+-----------+
| local_data_id | rrd_name    | time                | output    |
+---------------+-------------+---------------------+-----------+
|         32461 | traffic_out | 2010-04-29 15:35:03 | 953018991 |
|         32461 | traffic_out | 2010-04-29 15:40:01 | 957926298 |
+---------------+-------------+---------------------+-----------+
2 rows in set (0.00 sec)

Code: Select all

04/29/2010 04:26:59 PM - BOOST_SERVER: Poller[0] ERROR: /var/www/html/rra/1/32461.rrd: expected 2 data source readings (got 2) from 1272548103 

[TheWitness]

Poll just that host with verbosity 3 using the R/O flag. So,

Code: Select all

./spine -R -f <host_id> -l <host_id> -V 3

TheWitness

[elfada]

Code: Select all

05/06/2010 03:09:35 PM - BOOST_SERVER: Poller[0] ERROR: /var/www/html/rra/1226/35703.rrd: expected 2 data source readings (got 2) from 1273150616
05/06/2010 03:09:34 PM - BOOST_SERVER: Poller[0] ERROR: /var/www/html/rra/1226/35703.rrd: expected 2 data source readings (got 2) from 1273150900
05/06/2010 03:09:34 PM - BOOST_SERVER: Poller[0] ERROR: /var/www/html/rra/1226/35703.rrd: expected 2 data source readings (got 2) from 1273150900
05/06/2010 03:09:34 PM - BOOST_SERVER: Poller[0] ERROR: /var/www/html/rra/1226/35703.rrd: expected 2 data source readings (got 2) from 1273150616
05/06/2010 03:09:34 PM - BOOST_SERVER: Poller[0] ERROR: /var/www/html/rra/1226/35703.rrd: expected 2 data source readings (got 2) from 1273150616
05/06/2010 03:09:22 PM - BOOST_SERVER: Poller[0] ERROR: /var/www/html/rra/1226/35711.rrd: expected 2 data source readings (got 2) from 1273150616

Code: Select all

SPINE: Using spine config file [/etc/spine.conf]
SPINE: Version 0.8.7c starting
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35692] SNMP: v1: 192.168.131.10, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.1, value: 717335814
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35692] SNMP: v1: 192.168.131.10, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.1, value: 2963044300
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35693] SNMP: v1: 192.168.131.10, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.10, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35693] SNMP: v1: 192.168.131.10, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.10, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35694] SNMP: v1: 192.168.131.10, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.11, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35694] SNMP: v1: 192.168.131.10, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.11, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35695] SNMP: v1: 192.168.131.10, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.12, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35695] SNMP: v1: 192.168.131.10, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.12, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35696] SNMP: v1: 192.168.131.10, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.13, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35696] SNMP: v1: 192.168.131.10, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.13, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35697] SNMP: v1: 192.168.131.10, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.14, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35697] SNMP: v1: 192.168.131.10, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.14, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35698] SNMP: v1: 192.168.131.10, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.15, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35698] SNMP: v1: 192.168.131.10, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.15, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35699] SNMP: v1: 192.168.131.10, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.16, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35699] SNMP: v1: 192.168.131.10, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.16, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35700] SNMP: v1: 192.168.131.10, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.17, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35700] SNMP: v1: 192.168.131.10, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.17, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35701] SNMP: v1: 192.168.131.10, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.18, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35701] SNMP: v1: 192.168.131.10, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.18, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35702] SNMP: v1: 192.168.131.10, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.19, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35702] SNMP: v1: 192.168.131.10, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.19, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35703] SNMP: v1: 192.168.131.10, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.2, value: 258617
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35703] SNMP: v1: 192.168.131.10, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.2, value: 665866728
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35704] SNMP: v1: 192.168.131.10, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.20, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35704] SNMP: v1: 192.168.131.10, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.20, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35705] SNMP: v1: 192.168.131.10, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.21, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35705] SNMP: v1: 192.168.131.10, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.21, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35706] SNMP: v1: 192.168.131.10, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.22, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35706] SNMP: v1: 192.168.131.10, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.22, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35707] SNMP: v1: 192.168.131.10, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.23, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35707] SNMP: v1: 192.168.131.10, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.23, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35708] SNMP: v1: 192.168.131.10, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.24, value: 22540418
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35708] SNMP: v1: 192.168.131.10, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.24, value: 3316037297
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35709] SNMP: v1: 192.168.131.10, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.3, value: 490985432
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35709] SNMP: v1: 192.168.131.10, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.3, value: 1215395783
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35710] SNMP: v1: 192.168.131.10, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.4, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35710] SNMP: v1: 192.168.131.10, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.4, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35711] SNMP: v1: 192.168.131.10, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.5, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35711] SNMP: v1: 192.168.131.10, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.5, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35712] SNMP: v1: 192.168.131.10, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.6, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35712] SNMP: v1: 192.168.131.10, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.6, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35713] SNMP: v1: 192.168.131.10, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.7, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35713] SNMP: v1: 192.168.131.10, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.7, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35714] SNMP: v1: 192.168.131.10, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.8, value: 6129
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35714] SNMP: v1: 192.168.131.10, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.8, value: 665765788
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35715] SNMP: v1: 192.168.131.10, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.9, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Host[1226] DS[35715] SNMP: v1: 192.168.131.10, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.9, value: 64
05/06/2010 03:10:47 PM - SPINE: Poller[0] Time: 0.2578 s, Threads: 12, Hosts: 2

There is no polling problem. I think that there is a problem during the on-demand rrd update.

When you are looking at graph during a quite long period (many polling cycle), a gap will appear.

This gap will begin from the web page access and will finish when you close your browser or move to another web page.

In fact, I noticed that this gap problem occurs only when a graph is rendered.

Do you have any idea about the origin of this bug ?

[TheWitness]

I've got some thoughts. Let me think about it for a bit.

TheWitness

[BorisL]

While TheWitness has his bit of time for thinking - let's help him with debug of direct table population - this may narrow search field to one function

Do as I asked and show results.

[TheWitness]

Boris,

There's one other culprit too. Is that other issue really solved? If so, what did the final product look like. I have to do another release, but I would like to know if that other fix works and get a good patch.

TheWitness

[becketts]

Think i might be having this same issue myself.
I've just had a read through and updated some permissions to be well... more permissive on the rra dir and on the boost cache and lock file dirs.

I'm primarily noticing this because i've had my browser left on the cacti polling stats screen on and off (mostly on) for the past 24 hours and i'm getting gaps in SOME graphs.

It looks to be the newer graphs, and if its any help i'm using the structured rra storage for newer graphs, while some a large number of legacy graphs are unstructured.

I'm happy to do any diagnostic steps you might want (but i'm going to be AFK for about an hour in 30 mins time).

[TheWitness]

Its in base cacti. More info tomorrow.

TheWitness

[elfada]

I would to know if you advise to upgrade cacti and boost to solve this bug ?

In fact, I am thinking about an upgrade planning and I would to know if this one is justified.

[TheWitness]

I have released Boost 4.0-2 today that should resolve Gapping issues.

TheWitness

[TheWitness]

I took Boost 4.0-2 off the Docs site due to some "funny" issues going on with MySQL 5.0.51a-24+lenny2. There is a "huge" bug with MySQL on that release. I'm testing some more now and hope to release 4.1 soon.

I have identified the source of much of the gaps I think, and it's a MySQL bug...

More information to follow.

TheWitness

[becketts]

Any news on this yet, or an very approximate ETA?
I'm sorting out the handover for the cacti infrastructure i manage and i'm hoping to either do this before i leave or bring my colleages up to speed on the upcoming fix before i go.

[TheWitness]

4.1 is in svn. Still working on release.

TW

[elfada]

Is there some news or final issues about this problem ?

Thanks in advance.