Graphs stopped after setting up HP Procurve graphs

[dewath]

Host: RedHat E.S. 3
MySQL: 3.23.58
PHP: 4.3.2
RRDtool: 1.0.41

What I was doing: I was setting up graph templates for trending the following on HP Procurve switches:

1. CPU (I set up the graph and data collector) - tested the graph and it worked. I deployed this graph to 18 HP procurves and verified that the data collection and graphed worked.
2. I then wanted to set up the graphs/data collection for the following:

a) traffic in/out bits using the default graph set up
b) i/o error packets using the default graph set up
c) created a graph showing all uni-packets and non-unipackets on one graph.

I then tested all the above on the first hp procurve and data collection and graphing on the first hp procurve switch. Everything worked okay and graphs were being updated and displayed

The load on the system was < 1

3. Yesterday I added the same templates (see 2 above) for data collection and graphing to an additional 11 hp procurve switches. I am adding these to each interface so the numbers of hp rrd's is now at 3531.

As of 17:00 yesterday the graphs stopped displaying:

4. I turned on debug mode for the logs and did find that many snmp updates were returning "U", but the CPU data was still being updated in the log with good values. Still no graph is updated since yesterday afternoon when I did this major install of graphs. I cleared the poller cache (it timed out and had errors).

5. The load is now 3 fairly consistently.

6. MySQL is now taking up about 94% CPU consistently. Obviously this is a problem that I need to fix

Here is what I hope someone can advise and help:

1. Should I remove all the new graphs and data collection completely to get back to where I was and will that likely allow the original graphs to work again?

2. MySQL - what is the best way to address this?

3. Other suggestions welcome

Richard

[rony]

Concerning mysql, make sure that you do not have binary logging turned on, this can get messy quick.

[dewath]

After checking the MySQL and not seeing a binary log issue, I went for backing out all the graphs and data sources to a known working set. Well that did not work. I did find all sorts of duplications. But when I totally removed all graphs and data sources for a specific cisco router and then tried to readd the cisco cpu graph back in, the following happened.

1. It appeared to add the graph back in like the first time.
2. I waited for some results and when none showed up I rechecked in debug the graph and go this:

RRDTool Says:

ERROR: opening '/var/www/html/cacti/rra/3145_porter_cisco_3845_router_5min_cpu_4841.rrd': No such file or directory


Any ideas - or do I bit the bullet and redo from scratch? If so, would I be better off with MySQL 4.x version? And if so, I had some trouble with the PHP for accessing this, so any tips on building PHP and Apache for that matter would be appreciated.

The one good thing that came out of this is that the load went back to .3...

[rony]

Only thing I can say at this point is check out and confirm your paths and make sure that the user running the poller has permissions to read and write the log and rra directories in your cacti installtion.