Function does not exist - runaway error

neteru · Post by **neteru** » Thu Oct 12, 2006 11:04 am

I upgraded to the cacti-0.8.6i-1.el4.rf rpm last night on Centos 4. After I had massive gaps in my graphs and the phpsvr process seems to be running away with the following error... that seems to show up directly after I get a failed connection with the MySQL server.

This morning I had to kill the process as I had 2GB of logs filled (in about 10 min...) with with the two warnings listed below.

10/12/2006 10:30:10 AM - CACTID: Poller[0] MYSQL: Connection Failed: Too many connections
(fork)10/12/2006 10:30:10 AM - PHPSVR: Poller[0] WARNING: PHP Script File to be included, does not exist
10/12/2006 10:30:10 AM - PHPSVR: Poller[0] WARNING: Function does not exist
10/12/2006 10:30:10 AM - PHPSVR: Poller[0] WARNING: PHP Script File to be included, does not exist
10/12/2006 10:30:10 AM - PHPSVR: Poller[0] WARNING: Function does not exist
10/12/2006 10:30:10 AM - PHPSVR: Poller[0] WARNING: PHP Script File to be included, does not exist
10/12/2006 10:30:10 AM - PHPSVR: Poller[0] WARNING: Function does not exist

Not sure if this is related, but I have been seeing this periodically as well:
10/12/2006 10:14:54 AM - POLLER: Poller[0] Maximum runtime of 292 seconds exceeded. Exiting.
10/12/2006 10:14:54 AM - SYSTEM STATS: Time:293.5496 Method:cactid Processes:5 Threads:10 Hosts:60 HostsPerProcess:12 DataSources:553 RRDsProcessed:1048

Running cactid (using a svn release that is about a week old) from the command line seems to complete without errors.

Post by **gandalf** » Thu Oct 12, 2006 11:24 am

You may want to increase max_connections for mysql. We did so with success.
Reinhard

neteru · Post by **neteru** » Thu Oct 12, 2006 11:41 am

Is there some formula for the number of connections cactid will try to open based on the number of data sources you are polling, number of threads, etc.?

Post by **rony** » Thu Oct 12, 2006 11:53 am

Alot....

Ok, now that my attempt at being funny has failed...

Thewitness has that equation, and I'm sorry I don't recall.

But, can you give me an idea of the number of data source and process and threads you have defined?

neteru · Post by **neteru** » Thu Oct 12, 2006 12:16 pm

Data Sources: 818

Poller Processes: 5
Threads per: 10

Post by **rony** » Thu Oct 12, 2006 12:19 pm

How many processors in this machine and how much memory?

neteru · Post by **neteru** » Thu Oct 12, 2006 1:09 pm

3GB Memory
Dual Xeon 2.8 GHz

neteru · Post by **neteru** » Thu Oct 12, 2006 1:53 pm

Well, that's what I have for data sources in the web interface. cactid gives me the following...

10/12/2006 01:50:01 PM - POLLER: Poller[0] Maximum runtime of 292 seconds exceeded. Exiting.
10/12/2006 01:50:01 PM - SYSTEM STATS: Time:299.8785 Method:cactid Processes:5 Threads:10 Hosts:109 HostsPerProcess:22 DataSources:958 RRDsProcessed:1693

Any idea how I determine where cactid is getting hung up?

Running cactid on the command line gives me:
... snip ...
CACTID: Time: 40.6359 s, Threads: 10, Hosts: 109

cactid is compiled out of the svn release 3807, from the branch directory.

time · Post by **time** » Fri Oct 13, 2006 1:29 am

I started getting this same error too after upgrading to cacti 0.8.6i, but only with the Advanced ping graphs. Until I know how to fix it I've disabled those graphs for now.

With that kind of machine you should be getting much better performance unless all of your devices are on slow WAN links or something?

neteru · Post by **neteru** » Fri Oct 13, 2006 8:35 am

Better than the 40s that cactid reports, or better than the status reported by the poller that times out due to execution time.

I played around a bit with the thread/process counts yesterday and got the time down to about 30-35s, which was some improvement. What I have been noticing is that cactid is waiting for the ping to come back so that the devices can be polled. We are monitoring a wireless broadband network that is distributed over 50 miles or more. We're also running Motorola Prizm IMS on the same server which is monitoring/graphing 1000+ subscriber modules with a 10 minute poll interval.

I would like to know if there is any decent way I can use to determine why the poller is exceeding execution time limits, while cactid is coming back in about 40s. Does the time displayed under SYSTEM STATS include the time it takes RRDTool to create the graphs, and the graphing is perhaps timing out?

Post by **TheWitness** » Sat Oct 14, 2006 1:10 am

I am looking into what is going on with the "Function to be included..." error messages. I have a post here for tracking purposes:

http://forums.cacti.net/viewtopic.php?p=79594#79594

As far as tuning is concerned, I have got plans to implement a new addon to speed the RRD update process (which is what is slowing you down), but I have not started the work.

If this is your issue, more RAM is the "short term" solution.

TheWitness

neteru · Post by **neteru** » Sat Oct 14, 2006 9:05 pm

Well, I've done some playing around. With about 40 more data sources added to the ones I had previosly I found the following.

When I run cactid in readonly mode... cactid completes the polls of my devices in about 7.5s. Actually writing to the database I see around 40s times. I have been trying to tune MySQL a bit... I'm waiting on about 10-12% of my table locks. The primary culprit seems to be the poller_output table. I'm a bit new to MySQL administration on this level: about 27 million queries to the database in the last 2.5 days... an average of 125 or so a second. However those queries aren't distributed evenly over time as they are mostly bursts when cacti does its five minute polls, so I'm not sure if this is normal behavior.

In any case cactid run from the command line will complete in about 40s as I have said. Without rrdtools, MySQL is definately the bottleneck. However there is a more pressing issue.

Running through the poller the cactid processes seem to complete, but the poller will time out at the 292s max. Watching the poller_output table it doesn't take more than a minute or so for the poller to pull all the rows out of the table, but the rrdtools process never actually exits until it is killed by the initiation of the next poller cycle, which is reported in the logs as cactid exceeding the max runtime. Nothing I have seen in the logs has given me any clue why the rrdtools processes never actually terminate on their own.

To clarify... cacti runs on a dual proc 2.8GHz Xeon machine, 3GB memory. MySQL is running on a 5014C-MT supermicro, ich6r raid 0, P4 3.0GHz hyperthreaded cpu, w/ 2GB RAM. Only other things running on the MySQL server are isc dhcpd, and freeradius (for about 15 clients) which add a negligent load.

Post by **TheWitness** » Sat Oct 14, 2006 10:18 pm

Working with another user, it appears that this problem may be related to php 4.3.9. When the user upgraded to 5.0.4 on his CentOS 4.4 box, the problem disappeared.

TheWitness

neteru · Post by **neteru** » Sun Oct 15, 2006 12:47 pm

Upgraded to 5.0.4, still seeing cactid exceed max runtime.

Also, and this was happening before, I'm getting the following error:
10/15/2006 12:45:38 PM - CACTID: Poller[0] ERROR: Problem with MySQL: 'Duplicate entry '754-traffic_in-2006-10-15 12:45:32' for key 1'
10/15/2006 12:45:39 PM - CACTID: Poller[0] ERROR: Problem with MySQL: 'Duplicate entry '749-traffic_in-2006-10-15 12:45:32' for key 1'
10/15/2006 12:45:39 PM - CACTID: Poller[0] ERROR: Problem with MySQL: 'Duplicate entry '739-traffic_in-2006-10-15 12:45:31' for key 1'
10/15/2006 12:45:42 PM - CACTID: Poller[0] ERROR: Problem with MySQL: 'Duplicate entry '779-traffic_in-2006-10-15 12:45:36' for key 1'
10/15/2006 12:45:47 PM - CACTID: Poller[0] ERROR: Problem with MySQL: 'Duplicate entry '817-traffic_in-2006-10-15 12:45:40' for key 1'
10/15/2006 12:45:49 PM - CACTID: Poller[0] ERROR: Problem with MySQL: 'Duplicate entry '879-traffic_in-2006-10-15 12:45:47' for key 1'
10/15/2006 12:45:53 PM - CACTID: Poller[0] ERROR: Problem with MySQL: 'Duplicate entry '890-traffic_in-2006-10-15 12:45:47' for key 1'

Running cactid from the svn, build 3807. I see there is a 3809 now. I may try that as well and see what happens.

Post by **TheWitness** » Sun Oct 15, 2006 1:42 pm

The duplicate entry thing should not be happening as all inerts have been changed to replace's. Therefore, I don't think you could be running cactid 0.8.6i.

TheWitness

Cacti

Function does not exist - runaway error

Function does not exist - runaway error

Who is online