Cacti Graph Brakes issue.

[abhisekp34]

Hi team,

I found Cacti graph brake issue for one device very frequently. As checked in There are some SNMP connection time out.
but when the connection time out happens, i don't see any brake in Graph. But after 2-3 hours i can see those brakes.

As an example, i can see brakes in Graph for today, but there is no SNMP time out Error in cacti.log file.

IPAddress is not mentioned as per Security Purpose.

root@cpe01 # grep IPAddress cacti.log
root@cpe01 # grep IPAddress cacti.log-20220829
2022-08-29 04:15:04 - SPINE: Poller[1] PID[31302] Device[3601] HT[1] DS[18088] WARNING: SNMP timeout detected [400 ms], ignoring host 'IPAddress'
2022-08-29 04:15:04 - SPINE: Poller[1] PID[31302] Device[3601] HT[1] DS[18088] WARNING: SNMP timeout detected [400 ms], ignoring host 'IPAddress'
2022-08-29 21:53:08 - SPINE: Poller[1] PID[16740] Device[3601] HT[1] DS[18088] WARNING: SNMP timeout detected [400 ms], ignoring host 'IPAddress'
2022-08-29 21:53:08 - SPINE: Poller[1] PID[16740] Device[3601] HT[1] DS[18088] WARNING: SNMP timeout detected [400 ms], ignoring host 'IPAddress'
2022-08-29 22:07:10 - SPINE: Poller[1] PID[26201] Device[3601] HT[1] DS[18088] WARNING: SNMP timeout detected [400 ms], ignoring host 'IPAddress'
2022-08-29 22:07:10 - SPINE: Poller[1] PID[26201] Device[3601] HT[1] DS[18088] WARNING: SNMP timeout detected [400 ms], ignoring host 'IPAddress'
root@cpe01 #

Cacti Version :- 1.2.18

Can you please suggest any solution ? Please let me know, if any more information required.

Thanks

[Osiris]

Increase your SNMP timeout first. Second, double your heartbeat.

[abhisekp34]

Hi Osiris,

Thanks Brother for Quick Reply,

"Second, double your heartbeat." :- Is it applicable for only single Device ?

If I will update at backend tables like data_template_rrd & data_template_data, which specific entry need to update ?

As this change should not impact to other Devices too.

Thanks
Abhi

[TheWitness]

You can run a batch shell script to update the heartbeats, but you need to be knowledgeable in RRDtool. You would first update the heartbeat of your Data Source Profile, which you can do in Cacti 1.2.21+, and then you could go by one Data Source at a time and "debug" them. The debug output will show the RRDtool command to adjust the heartbeats.

[abhisekp34]

Can we update Heartbit in Cacti 1.2.18 Version ? and how ? Is there any Particular Command or Script Present ?

Upgrade to 1.2.21+ is quite difficult as Organization challenges.

Now Graphs are having more cuts as before. We have around 5k Devices. And most of graphs have same issue.

cacti_graph Issue.png (32.3 KiB) Viewed 692 times

[TheWitness]

I suspect you have a bond or vlan interface with a 10Gbps link speed that is reporting an ifSpeed of only 5Gbps via ifHighSpeed. This is a vendor problem not a Cacti problem. I suggest you increase the Maximum Value in your RRDfile using the RRDtool tune command.

[abhisekp34]

Hi Bro,

How to use RRDTool in version 1.2.18, as it is a bit difficult to upgrade in Big organizations.

any specific Command or any Specific script & Process ?

[TheWitness]

Read up on how to tune an RRDfile on https://rrdtool.org. It's not that complicated.

[abhisekp34]

Thank You Brother.

We have already rrdtool installed in tool cacti.

So got some command too. Some more doubts,

Do I need to update Heartbeats for all the Devices into rrd files ? or specific Devices only.

As I can see mostly Devices have same Issue (Cuts in Graph), Even all of the added graphs. Even I can see for CPU graph also same issue.

Can it also Device Vendor Related issue as you suggested on Previous Post ? if so, what should we suggest to Device Vendor to update, So that issue will not come.

Even though I will be updating the Heartbeat.

cacti_graph Issue 1.png (23.96 KiB) Viewed 678 times

[abhisekp34]

Hi Bro,

I have increased the Heartbeat for a device from 600 to 900.

Still Same issue big cuts on Graph. Any other way to check, if it would any Device side Issue, as said by you on Earlier Post.

root@grwcacticpe01 # rrdtool info cassis01-02gil-03_5min_cpu_27191.rrd|grep 5min_cpu
filename = "cassis01-02gil-03_5min_cpu_27191.rrd"
ds[5min_cpu].index = 0
ds[5min_cpu].type = "GAUGE"
ds[5min_cpu].minimal_heartbeat = 600
ds[5min_cpu].min = 0.0000000000e+00
ds[5min_cpu].max = 1.0000000000e+02
ds[5min_cpu].last_ds = "1"
ds[5min_cpu].value = NaN
ds[5min_cpu].unknown_sec = 246
root@grwcacticpe01 # rrdtool tune cassis01-02gil-03_5min_cpu_27191.rrd -h 5min_cpu:900
root@grwcacticpe01 # rrdtool info cassis01-02gil-03_5min_cpu_27191.rrd|grep 5min_cpu
filename = "cassis01-02gil-03_5min_cpu_27191.rrd"
ds[5min_cpu].index = 0
ds[5min_cpu].type = "GAUGE"
ds[5min_cpu].minimal_heartbeat = 900
ds[5min_cpu].min = 0.0000000000e+00
ds[5min_.max = 1.0000000000e+02
ds[5min_cpu].last_ds = "1"
ds[5min_cpu].value = NaN
ds[5min_cpu].unknown_sec = 246
root@grwcacticpe01 #

cacti_graph Issue 3.png (33.87 KiB) Viewed 672 times

\

cacti_graph Issue 2.png (23.12 KiB) Viewed 672 times

[TheWitness]

Who owns the RRDfiles? Can you put the device into debug mode and then look for any 'U' responses in your Cacti.log?

[abhisekp34]

Hi Bro,

In Server Apache User owns the rrd files.
-rw-r--r-- 1 apache apache 374768 Nov 16 11:02 cassis01-02gil-03_traffic_in_28741.rrd
-rw-r--r-- 1 apache apache 374768 Nov 16 11:04 cassis01-02gil-03_traffic_in_28742.rrd
-rw-r--r-- 1 apache apache 95608 Nov 17 06:31 cassis01-02gil-03_5min_cpu_27191.rrd
-rw-r--r-- 1 apache apache 374768 Nov 17 06:34 cassis01-02gil-03_band_27197.rrd
-rw-r--r-- 1 apache apache 374768 Nov 17 06:34 cassis01-02gil-03_traffic_in_27206.rrd
-rw-r--r-- 1 apache apache 374768 Nov 17 06:34 cassis01-02gil-03_traffic_in_28737.rrd
-rw-r--r-- 1 apache apache 374768 Nov 17 06:34 cassis01-02gil-03_traffic_in_28747.rrd

After Enabled Debug for some time. below are result & attached. There is no "U" response found on Cacti files.

2022-11-17 05:02:06 - SPINE: Poller[1] PID[25549] DEBUG: In Poller, About to Start Polling of Device for Device ID 6938
2022-11-17 05:02:06 - SPINE: Poller[1] PID[25549] Device[6938] DEBUG: Entering SNMP Ping
2022-11-17 05:02:06 - SPINE: Poller[1] PID[25549] Device[6938] Updating Full System Information Table
2022-11-17 05:02:06 - SPINE: Poller[1] PID[25549] DEBUG: Device[6938] HT[1] RECACHE: Processing 2 items in the auto reindex cache for '94.105.14.228'
2022-11-17 05:02:06 - SPINE: Poller[1] PID[25549] Device[6938] HT[1] DQ[10] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 38118068 < output: 38124094)
2022-11-17 05:02:06 - SPINE: Poller[1] PID[25549] Device[6938] HT[1] DQ[11] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 38118068 < output: 38124094)
2022-11-17 05:02:06 - SPINE: Poller[1] PID[25549] Device[6938] HT[1] NOTE: There are '8' Polling Items for this Device
2022-11-17 05:02:06 - SPINE: Poller[1] PID[25549] Device[6938] HT[1] DS[27197] TT[49.33] SNMP: v2: 94.105.14.228, dsname: band, oid: .1.3.6.1.4.1.13191.10.3.9.2.1.46.0, value: 4
2022-11-17 05:02:06 - SPINE: Poller[1] PID[25549] Device[6938] HT[1] DS[27197] TT[49.41] SNMP: v2: 94.105.14.228, dsname: rssi, oid: .1.3.6.1.4.1.13191.10.3.9.2.1.41.0, value: -75
2022-11-17 05:02:06 - SPINE: Poller[1] PID[25549] Device[6938] HT[1] DS[28737] TT[49.45] SNMP: v2: 94.105.14.228, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.589824, value: 318392229
2022-11-17 05:02:06 - SPINE: Poller[1] PID[25549] Device[6938] HT[1] DS[28737] TT[49.49] SNMP: v2: 94.105.14.228, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.589824, value: 140178141
2022-11-17 05:02:06 - SPINE: Poller[1] PID[25549] Device[6938] HT[1] DS[27206] TT[49.52] SNMP: v2: 94.105.14.228, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.1912799233, value: 318234309
2022-11-17 05:02:06 - SPINE: Poller[1] PID[25549] Device[6938] HT[1] DS[27206] TT[49.54] SNMP: v2: 94.105.14.228, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.1912799233, value: 140166029
2022-11-17 05:02:06 - SPINE: Poller[1] PID[25549] Device[6938] HT[1] DS[28747] TT[49.57] SNMP: v2: 94.105.14.228, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.402657280, value: 15636604
2022-11-17 05:02:06 - SPINE: Poller[1] PID[25549] Device[6938] HT[1] DS[28747] TT[49.59] SNMP: v2: 94.105.14.228, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.402657280, value: 10685928
2022-11-17 05:02:06 - SPINE: Poller[1] PID[25549] Device[6938] HT[1] Total Time: 0.19 Seconds
2022-11-17 05:02:06 - SPINE: Poller[1] PID[25549] DEBUG: Device[6938] HT[1] DEBUG: HOST COMPLETE: About to Exit Device Polling Thread Function

Cacti_Debug.txt

(67.22 KiB) Downloaded 14 times

[abhisekp34]

Brother,

Any Suggestion ?

[TheWitness]

First, I'm not your brother. Second, are you using boost?

[abhisekp34]

Sorry, Please Don't mind.

Your help is always appreciated.

As I am very New to Cacti, so asking doubt regarding this issue. this might be simple for you, but complex for me.

What is boost exactly ?