RRD Files Stopped Updating for No Reason
Moderators: Developers, Moderators
Hi Larry,
Maybe could be thold... I could try upgrade to 0.86f without thold and see if the problem is fixed.
I've checked the poller entries and in fact there are missing entries. Then I've truncated table poller_item manually (successfully) and repopulate it using web "clear poller cache" command. But the poller_item entries keep the traffic lines I want out.
I don't know why repopulate() don't get the traffic row... see:
---
mysql> select * from data_local where host_id="192";
+------+------------------+---------+---------------+------------+
| id | data_template_id | host_id | snmp_query_id | snmp_index |
+------+------------------+---------+---------------+------------+
| 6009 | 111 | 192 | 0 | |
| 2911 | 18 | 192 | 0 | |
| 4528 | 101 | 192 | 1 | 1 |
| 6010 | 110 | 192 | 1 | 1 |
+------+------------------+---------+---------------+------------+
##fail##:
mysql> select name_cache,local_data_id from data_template_data where local_data_id = "6010";
+----------------------------+---------------+
| name_cache | local_data_id |
+----------------------------+---------------+
| Ap-Amesp_Butanta - Traffic | 6010 |
+----------------------------+---------------+
mysql> select * from poller_item where local_data_id = "6010";
Empty set (0.00 sec)
##works##:
mysql> select name_cache,local_data_id from data_template_data where local_data_id = "6009";
+-----------------------------+---------------+
| name_cache | local_data_id |
+-----------------------------+---------------+
| Ap-Amesp_Butanta - Rx Power | 6009 |
+-----------------------------+---------------+
mysql> select local_data_id,host_id from poller_item where local_data_id = "6009";
+---------------+---------+
| local_data_id | host_id |
+---------------+---------+
| 6009 | 192 |
+---------------+---------+
---
How can I debug this?
Many thanks for your support.
Maybe could be thold... I could try upgrade to 0.86f without thold and see if the problem is fixed.
I've checked the poller entries and in fact there are missing entries. Then I've truncated table poller_item manually (successfully) and repopulate it using web "clear poller cache" command. But the poller_item entries keep the traffic lines I want out.
I don't know why repopulate() don't get the traffic row... see:
---
mysql> select * from data_local where host_id="192";
+------+------------------+---------+---------------+------------+
| id | data_template_id | host_id | snmp_query_id | snmp_index |
+------+------------------+---------+---------------+------------+
| 6009 | 111 | 192 | 0 | |
| 2911 | 18 | 192 | 0 | |
| 4528 | 101 | 192 | 1 | 1 |
| 6010 | 110 | 192 | 1 | 1 |
+------+------------------+---------+---------------+------------+
##fail##:
mysql> select name_cache,local_data_id from data_template_data where local_data_id = "6010";
+----------------------------+---------------+
| name_cache | local_data_id |
+----------------------------+---------------+
| Ap-Amesp_Butanta - Traffic | 6010 |
+----------------------------+---------------+
mysql> select * from poller_item where local_data_id = "6010";
Empty set (0.00 sec)
##works##:
mysql> select name_cache,local_data_id from data_template_data where local_data_id = "6009";
+-----------------------------+---------------+
| name_cache | local_data_id |
+-----------------------------+---------------+
| Ap-Amesp_Butanta - Rx Power | 6009 |
+-----------------------------+---------------+
mysql> select local_data_id,host_id from poller_item where local_data_id = "6009";
+---------------+---------+
| local_data_id | host_id |
+---------------+---------+
| 6009 | 192 |
+---------------+---------+
---
How can I debug this?
Many thanks for your support.
rgds,
Edgar Shine
Edgar Shine
Possible information on this problem
Hello,
I have had the same problem as others.. I am not going into much detail about specific steps and/or versions.
Summary:
My cacti was working.. updated Cacti a week ago..
Everything fine.
Updated rrdtool and switched from cmd.php to cactid.
At this time, ALL graphs ceased.
Read many forum entries.. tried multiple things including the new rrd.php.
Finally backed off on my rrdtool version and restored the original rrd.php and switched my polling back to cmd.php.
Only a few graphs show up now (5%).
Possible enlightening Info that I found out.
All graphs that are not working insist that they do not have data-source 'traffic_in' in their rrd file.
I have done an 'rrdtool info' on those rrd file and found that the DS 'traffic_in' is there.
** However, the difference that I found on Working-graphs and NON-Working-graphs is 'Case' difference.
i.e.
where cacti-datasource-debug states that 'traffic_in' does not exist,
the 'rrdtool info ###.rrd' shows 'Traffic_In'
Notice the Proper Caps.
on graphs that do work.. the cacti-datasource-debug AND the 'rrdtool info' both show 'traffic_in' for the DS.. (lowercase).
So,
I still have DS's that can't be found for some reason.
If anyone has any information on this.. I would appreciate greatly.
John.
I have had the same problem as others.. I am not going into much detail about specific steps and/or versions.
Summary:
My cacti was working.. updated Cacti a week ago..
Everything fine.
Updated rrdtool and switched from cmd.php to cactid.
At this time, ALL graphs ceased.
Read many forum entries.. tried multiple things including the new rrd.php.
Finally backed off on my rrdtool version and restored the original rrd.php and switched my polling back to cmd.php.
Only a few graphs show up now (5%).
Possible enlightening Info that I found out.
All graphs that are not working insist that they do not have data-source 'traffic_in' in their rrd file.
I have done an 'rrdtool info' on those rrd file and found that the DS 'traffic_in' is there.
** However, the difference that I found on Working-graphs and NON-Working-graphs is 'Case' difference.
i.e.
where cacti-datasource-debug states that 'traffic_in' does not exist,
the 'rrdtool info ###.rrd' shows 'Traffic_In'
Notice the Proper Caps.
on graphs that do work.. the cacti-datasource-debug AND the 'rrdtool info' both show 'traffic_in' for the DS.. (lowercase).
So,
I still have DS's that can't be found for some reason.
If anyone has any information on this.. I would appreciate greatly.
John.
It looks like there might be a traffic template floating around out there that has a renamed RRD data source name. First, I would go through your traffic-related data templates and make sure that data source name field reads "traffic_in" or "traffic_out". Secondly, I wrote a quick script that will mass-rename RRD data sources for you.
To use, do something like:
Be sure to backup your RRD files first!
-Ian
Code: Select all
#!/bin/bash
if (( $# < 3 ))
then
echo "Usage: rrdrename.sh OLD_RRD NEW_RRD FILENAME";
exit
fi
RRD_OLD_NAME=$1
RRD_NEW_NAME=$2
until [ -z "$3" ] # Until all parameters used up...
do
echo "+ rrdtool tune $3 --data-source-rename $RRD_OLD_NAME:$RRD_NEW_NAME"
rrdtool tune $3 --data-source-rename $RRD_OLD_NAME:$RRD_NEW_NAME
shift
done
Code: Select all
./rrdrename.sh Traffic_In traffic_in *traffic*.rrd
./rrdrename.sh Traffic_Out traffic_out *traffic*.rrd
-Ian
Larry,
I've reinstalled Cacti (0.8.6e) without THold and the problem with the clear poller cache function remains the same.
The problem seems be related to the xml scripts I have, when I use the original interface.xml, it works fine (with 1700 poller itens), when I use my customized xml script it points to 3400 poller itens (verified using mysql line command) and poller view function doesn't work anymore.
I've reinstalled Cacti (0.8.6e) without THold and the problem with the clear poller cache function remains the same.
The problem seems be related to the xml scripts I have, when I use the original interface.xml, it works fine (with 1700 poller itens), when I use my customized xml script it points to 3400 poller itens (verified using mysql line command) and poller view function doesn't work anymore.
rgds,
Edgar Shine
Edgar Shine
- TheWitness
- Developer
- Posts: 17007
- Joined: Tue May 14, 2002 5:08 pm
- Location: MI, USA
- Contact:
EShine,
PM me your IM information.
TheWitness
PM me your IM information.
TheWitness
True understanding begins only when we realize how little we truly understand...
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Hi TheWitness,
I upgraded my env to Cacti 0.8.6g today. Think is more reasonable work on updated versions instead of digging problems on old ones.
Cacti behavior is the same in this new version. Since the problem seem be related to the XML script, I tried the following:
1) Create a new data query, "Aperto Device", linked to a XLM called aperto_interface.xml
2) Change the host template "Aperto SU" to point this data query
3) Remove and re-create a device "Ap-Amesp_Itatiaia"
4) Create Graphs for it
5) Replace the original interface.xml script to have a functional "clear poller cache"
6) Do a clear poller cache and see if the new data is being cached.
Well... it didn't work. The traffic is not being polled. And also not cached in the poller item.
Additional info, maybe it's relevant..
- /usr/local/cacti/rra is a symlink to /usr/local/cactidb/rra. cactiuser:cacti
is the owner/group for both dir cacti/cactidb
- upgrade was done in the following sequence: 0.8.6e -> 0.8.6f -> 0.8.6g
Thanks for your support.
I upgraded my env to Cacti 0.8.6g today. Think is more reasonable work on updated versions instead of digging problems on old ones.
Cacti behavior is the same in this new version. Since the problem seem be related to the XML script, I tried the following:
1) Create a new data query, "Aperto Device", linked to a XLM called aperto_interface.xml
2) Change the host template "Aperto SU" to point this data query
3) Remove and re-create a device "Ap-Amesp_Itatiaia"
4) Create Graphs for it
5) Replace the original interface.xml script to have a functional "clear poller cache"
6) Do a clear poller cache and see if the new data is being cached.
Well... it didn't work. The traffic is not being polled. And also not cached in the poller item.
Additional info, maybe it's relevant..
- /usr/local/cacti/rra is a symlink to /usr/local/cactidb/rra. cactiuser:cacti
is the owner/group for both dir cacti/cactidb
- upgrade was done in the following sequence: 0.8.6e -> 0.8.6f -> 0.8.6g
Thanks for your support.
- Attachments
-
- data_query.jpg (89.55 KiB) Viewed 4747 times
-
- This image show that we have just two data sources being cached
- poller_cache_itatiaia.jpg (32.57 KiB) Viewed 4747 times
-
- aperto_interface.xml
- xml file
- (10.47 KiB) Downloaded 161 times
rgds,
Edgar Shine
Edgar Shine
- TheWitness
- Developer
- Posts: 17007
- Joined: Tue May 14, 2002 5:08 pm
- Location: MI, USA
- Contact:
What are the '#' signs in the XML script:
TheWitness
I'll need more information to dig in deeper.#
- <ifName>
#
<name>Name (IF-MIB)</name>
#
<method>walk</method>
#
<source>value</source>
#
<direction>input</direction>
#
<oid>.1.3.6.1.2.1.31.1.1.1.1</oid>
#
</ifName>
#
- <ifAlias>
#
<name>Alias (IF-MIB)</name>
#
<method>walk</method>
#
<source>value</source>
#
<direction>input</direction>
#
<oid>.1.3.6.1.2.1.31.1.1.1.18</oid>
#
</ifAlias>
TheWitness
True understanding begins only when we realize how little we truly understand...
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
- TheWitness
- Developer
- Posts: 17007
- Joined: Tue May 14, 2002 5:08 pm
- Location: MI, USA
- Contact:
Also,
Please explain the following in more detail:
Please explain the following in more detail:
TheWitness5) Replace the original interface.xml script to have a functional "clear poller cache"
True understanding begins only when we realize how little we truly understand...
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Sorry, wrong file... in fact, the real cacti xml script file is just a `grep -v "#"` of the attached xml file.
The "original interface.xml" is the interface.xml that is shipped in the cacti tgz pack. When I use it, the clear poller cache function in "console -> system utilities" seems to work (at least return me a poller view screen). When I use a customized interface.xml, with some aditional lines in interface config, the clear poller cache don't return me a screen.
I'm attaching the modified interface.xml and the system utilities screens.
Thanks for your support.
The "original interface.xml" is the interface.xml that is shipped in the cacti tgz pack. When I use it, the clear poller cache function in "console -> system utilities" seems to work (at least return me a poller view screen). When I use a customized interface.xml, with some aditional lines in interface config, the clear poller cache don't return me a screen.
I'm attaching the modified interface.xml and the system utilities screens.
Thanks for your support.
- Attachments
-
- This is the screen that Cacti presents me if I use the modified interface.xml script (attached in this msg)
- after_click.jpg (44.63 KiB) Viewed 4729 times
-
- interface.xml
- the interface.xml modified script
- (18.02 KiB) Downloaded 859 times
-
- when I use the interface.xml which is shipped in Cacti tgz package, it shows me this screen doing a clear poller cache. Nota that I'm using the same Cacti installation for all tests.
- poller_cache.jpg (127.08 KiB) Viewed 4729 times
rgds,
Edgar Shine
Edgar Shine
- TheWitness
- Developer
- Posts: 17007
- Joined: Tue May 14, 2002 5:08 pm
- Location: MI, USA
- Contact:
Please do not modify the original Interface.xml file. Doing so will cause problems for others if you ever publish a template or ever import a template.
So, create a new one and then create new Data and Graph templates based upon it. I will now look at your XML file for the moment.
TheWitness
So, create a new one and then create new Data and Graph templates based upon it. I will now look at your XML file for the moment.
TheWitness
True understanding begins only when we realize how little we truly understand...
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
- TheWitness
- Developer
- Posts: 17007
- Joined: Tue May 14, 2002 5:08 pm
- Location: MI, USA
- Contact:
In addition, when clearing your poller cache, review your apache log directory for the contents of the error.log file and post what it returns.
TheWitness
TheWitness
True understanding begins only when we realize how little we truly understand...
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Fixed yet?
I applied all patches for 0.8.6f and this has not corrected the problems with not being able to add any new graphs to cacti. Has this issue been corrected in 0.8.6g? I don't want to upgrade yet if this still hasn't been fixed. 0.8.6f still seems really flaky. For instance, so graphs just stop updating out of the blue and log just shows that RRD suddenly won't accept the values anymore. Is cacti essentially broken right now? What's going on here?
- TheWitness
- Developer
- Posts: 17007
- Joined: Tue May 14, 2002 5:08 pm
- Location: MI, USA
- Contact:
Make sure PHP maximum memory is greater than 8mbytes. Then "truncate" the poller_output table.
TheWitness
TheWitness
True understanding begins only when we realize how little we truly understand...
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
-
- Cacti User
- Posts: 66
- Joined: Tue Mar 22, 2005 10:11 am
Is this fixed?
I'm running 0.8.6c. And I see this happen every 10 days or so. Really annoying. Is this fixed in a current verion? If so, at what version was it fixed so I can recommend upgrading/testing.
What exactly caused this? I don't understand how clearing the poller cache table is 'fixing' everything. Is it because I'm polling thousands of interfaces? Is this fixed with the C-based poller?
What exactly caused this? I don't understand how clearing the poller cache table is 'fixing' everything. Is it because I'm polling thousands of interfaces? Is this fixed with the C-based poller?
- TheWitness
- Developer
- Posts: 17007
- Joined: Tue May 14, 2002 5:08 pm
- Location: MI, USA
- Contact:
jrichardson,
This is likely caused by a memory problem for most users. When you say every X day's. How do you fix it? Please be more specific.
TheWitness
This is likely caused by a memory problem for most users. When you say every X day's. How do you fix it? Please be more specific.
TheWitness
True understanding begins only when we realize how little we truly understand...
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Who is online
Users browsing this forum: No registered users and 2 guests