Cacti 0.8.6 - 'cacti' 'cactiuser' MySQL duplcate issues?

[Wernicke]

Strange. Suddenly all my graph looks strange.

They have worked fine for a week now, since I converted to cactid in cacti 0.8.6, but suddenly these messages start appair again..

09/19/2004 10:05:06 PM - CACTID: Poller[0] Host[139] WARNING: Result from SNMP not valid. Partial Result: U...

All the graphs is like this.. and they go down to "0" at random times

I have not yet upgraded to the 0.8.6a pre-version

[TheWitness]

Look for Host Up/Down messages in your logfile. If you don't find them, or even if you don't, try increasing your SNMP timeout and retry values. Also, try increasing your Ping Retry value in Setting->Poller.

TheWitness

[Wernicke]

One more thing. The "Clear logfile" option don't work. I have to delete it manualy. the Clear logfile button just shows the log..

[TheWitness]

Your webserver User ID does not have rights to the log directory. You should insure that the webserver User ID has 777 (RWX) access to the Cacti directory. With *nix you would use "chmod -R 777 /<webroot>/cacti".

TheWitness

[TheWitness]

Also,

when you get a "U" in the partial result, you should turn logging to DEBUG for a cycle. If it is randum, again increase your timeouts.

TheWitness

[Wernicke]

Hi

It looks like The log directory had to be owned by the webserver's user (nobody)
The delete logfile work fine now.

btw. Setting the whole cacti directory to 777 will make it avalible to every user on the sytem. I recommend "chown nobody:nobody log" to make the user nobody own the log directory

[monideth]

OK, just to update everybody on this situation.

Although the polling process seems to have started working again - I think this is due to the memory config - at present I have set this in my /etc/php.ini file rather than the cacti/poller.php file:

Code: Select all

;;;;;;;;;;;;;;;;;;;
; Resource Limits ;
;;;;;;;;;;;;;;;;;;;

max_execution_time = 30     ; Maximum execution time of each script, in seconds
max_input_time = 60     ; Maximum amount of time each script may spend parsing request data
memory_limit = 512M      ; Maximum amount of memory a script may consume (8MB)

However, now I am still getting problems with intermittent graph updates. I have also noticed there is intermittent RRD file creation too.


OK - here are a few questions relating to the DEBUG info:

I understand that RRD: entries represent the commands that Cacti wants rrdtool to use - either a create to update. However, I am also seeing RRDTOOL: entries - now I assume these should indicate the results of the rrdtool command. However, there are two types of RRDTOOL: entries - for example:

Code: Select all

09/20/2004 11:10:06 AM - CMDPHP: Poller[0] RRDTOOL:2
09/20/2004 11:10:06 AM - CMDPHP: Poller[0] RRDTOOL:2
09/20/2004 11:10:06 AM - CMDPHP: Poller[0] RRDTOOL:2 OK u:0.02 s:0.03 r:5.33
09/20/2004 11:10:06 AM - CMDPHP: Poller[0] RRDTOOL:2
09/20/2004 11:10:06 AM - CMDPHP: Poller[0] RRDTOOL:2
09/20/2004 11:10:06 AM - CMDPHP: Poller[0] RRDTOOL:2 OK u:0.02 s:0.03 r:5.35
09/20/2004 11:10:06 AM - CMDPHP: Poller[0] RRDTOOL:2

Note: I have added a '1' and '2' in the PHP scripts to identify which log entry this actually is - i.e. in the above it is the second RRDTOOL log entry that appears in the code.

Does the blank entries - the ones without 'OK' in them suggest that these rrdtool commands did not return to was carried out OK? If so then this would make sense in respect to missing graoh data - i.e. the rrdtool update commands did not complete as it should.

Regards,

Mon

[TheWitness]

Mon,

It would appear that way. I need to make some more changes to the code. But in the mean time, there is an updated pre-release that should have CACTI2RRD and RRD2CACTI and not RRDTOOL: you should download.

Keep up the posts. I am in training the next few day's so daytime posts will be impossible for me though.

TheWitness

[monideth]

OK - just upgraded to the latest pre-release.

I am still seeing something similar:

Code: Select all

09/20/2004 11:30:44 AM - CMDPHP: Poller[0] RRD2CACTI:
09/20/2004 11:30:44 AM - CMDPHP: Poller[0] RRD2CACTI:
09/20/2004 11:30:44 AM - CMDPHP: Poller[0] RRD2CACTI:
09/20/2004 11:30:44 AM - CMDPHP: Poller[0] RRD2CACTI:
09/20/2004 11:30:44 AM - CMDPHP: Poller[0] RRD2CACTI:
09/20/2004 11:30:45 AM - CMDPHP: Poller[0] RRD2CACTI: OK u:0.03 s:0.10 r:12.55^@
09/20/2004 11:30:45 AM - CMDPHP: Poller[0] RRD2CACTI: OK u:0.03 s:0.10 r:12.58^@
09/20/2004 11:30:45 AM - CMDPHP: Poller[0] RRD2CACTI: OK u:0.03 s:0.10 r:12.62^@
09/20/2004 11:30:45 AM - CMDPHP: Poller[0] RRD2CACTI: OK u:0.03 s:0.10 r:12.65^@

However, Is the '^@1 suffix on 'OK' entries correct /intentional?

Regards,

Mon

[monideth]

OK - take for example this log entry:

Code: Select all

09/20/2004 12:30:06 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /var/www/html/cacti/rra/hamptonrtrcore01_traffic_in_745.rrd --template traffic_in:traffic_out 1095683403:0:0

A quick check to check whether this rrd file has been update:

Code: Select all

[root@cacti-unl cacti]# rrdtool dump /var/www/html/cacti/rra/hamptonrtrcore01_traffic_in_745.rrd | head
<!-- Round Robin Database Dump -->
<rrd>
        <version> 0001 </version>
        <step> 300 </step> <!-- Seconds -->
        <lastupdate> 1095681603 </lastupdate> <!-- 2004-09-20 12:00:03 UTC -->

        <ds>
                <name> traffic_in </name>
                <type> COUNTER </type>
                <minimal_heartbeat> 600 </minimal_heartbeat>
[root@cacti-unl cacti]#

You can see that the time stamp does not match - 1095681603 in rrd file as opposed to 1095683403 which is what it should be updated at.

This either means rrdtool itself is failing - or the rrdtool pipe is failing or Cacti is not passing the values through the pipe correctly.

Is there any to add in debugs to check whether rrdtool is getting the parammeters correctly?

Regards,

Mon

[TheWitness]

What version of RRDtool are you using?

TheWitness

[TheWitness]

Mon,

I will be making changes to the rrd.php tonight to include the function "stream_select" to the RRD_Execute function when proc_open is being used to drive I/O to and from the RRDTool. I will assign a 4 second timeout value to the function.

I believe that this is what is causing the RRD2CACTI to return blank. This would indicate that CACTI is reading the pipe before RRDTool has filled it. This could explain why you have been getting blank results from the RRD2CACTI and also why it would be intermittant. What is likely happening is that there is a pipe buffer overflow happening on larger systems such as yours with the output pipe filling up.

Will post late tonight as I stated earlier. I have sent a communication to Ian, although he is quite busy with Exams and is attending a function this eavening and will not likely read this until late tonight.

TheWitness

[TheWitness]

Mon,

Quick fix should be to change the I/O from non-blocking to blocking. I just noticed the lines in the rrd.php function. Change the following two lines:

stream_set_blocking($rrd_struc["pipes"][RRDTOOL_PIPE_CHILD_READ], false);
stream_set_blocking($rrd_struc["pipes"][RRDTOOL_PIPE_CHILD_WRITE], false);

to:

stream_set_blocking($rrd_struc["pipes"][RRDTOOL_PIPE_CHILD_READ], true);
stream_set_blocking($rrd_struc["pipes"][RRDTOOL_PIPE_CHILD_WRITE], true);

TheWitness

[BadKarma]

Hi The Witness,

Because i'm having the same type of problems as mon has i'm also following this thread.
I just implemented the quickfix and i'm affraid it does not resolve this issue for me.
Still have gaps showing up in my graphs.
I'm going to run some extra debugging myself also tonight i'll let you know my results.

Regards,

BadKarma

[TheWitness]

BadKarma,

Are you willing to do a teleconference on Monday night?

AIM: PhytmVygr MSM: Larry