[SOLVED] WARNING: Poller Output Table not Empty

tymbow · Post by **tymbow** » Wed Feb 13, 2008 1:27 am

I tried to upgrade to 0.8.7b today and had the "Invalid PHP_SELF Path" error. Not having a lot of time to troubleshoot I reverted back to 0.8.7a plus the "cmd-php-non-unique-hosts.patch" and "graph-issue-wrra-specs.patch" patches I had previously and I'm now getting this error every polling cycle:

Code: Select all

02/13/2008 05:50:00 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Potential Data Source Issues for Data Sources: (DS[88]), (DS[96]), (DS[106]), (DS[116]), (DS[126]), (DS[136]), (DS[146]), (DS[156]), (DS[166]), (DS[176]), (DS[186]), (DS[196]), (DS[206]), (DS[216])

The script associated with these data sources works and returns the correct data (seen in the log in DEBUG mode) but relevant RRAs are not updating any more.

I tried reverting to a backup of the database I took just before the upgrade and I still get the error. I have attached a file that shows the current "poller_output" rows and the relevant sections from the nightly database backup. I note that there are no "poller_output" rows in the database backup prior to the upgrade attempt but they are present after the upgrade.

Can anyone help? Should I just delete those rows from the "poller_output" table? (EDIT: Tried that - no change. They just re-appear).

tymbow · Post by **tymbow** » Sun Feb 17, 2008 7:18 pm

An update.

I've tried creating a new host that uses the problematic script template (bluecoat_memory.pl) and no success. It simply refuses to update or even create any RRDs associated with this template. It had been running perfectly for months and no changes have been made to the script or template. All other templates (including others for these hosts) run perfectly.

I've run "mysqlcheck" and all is ok, I've run "php -q poller.php" and nothing weird came up, I've re-populated the poller cache with no success.

I've attached a complete "cacti.log" run in DEBUG mode hoping someone has an idea what is going on.

drookie · Post by **drookie** » Mon Feb 18, 2008 10:45 am

Got exactly same issue after an upgrade 0.8.6j -> 0.8.7b.

P.S. Downgraded back to 0.8.6j 'till this will be resolved. :P

tymbow · Post by **tymbow** » Tue Feb 26, 2008 1:01 am

Sorry to bump this again but I am at my wits end with this - I just can't get it working again.

The log file says poller_output cache is not empty which indicates the RRD files are not being updated. OK I accept that but why?

Further in the log, and by viewing the poller_cache we see that the data sources are polled correctly by SPINE and return data. That data is correctly loaded into the Poller_output table ready to be loaded into the RRD files but this never happens. These RRD files are never mentioned in any CACTI2RRD log entries. The data is simply collected and then ignored.

Why is Cacti not updating the RRDs? The log file says nothing regarding errors except the initial warning about the poller cache not being empty.

Does anyone know where I need to look or what I need to do to understand why no attempt is ever made (according to the log) to update these RRDs?

tymbow · Post by **tymbow** » Tue Feb 26, 2008 10:15 pm

I think I may be getting somewhere. I've been looking into the "poller_output_empty.php" CLI script and the PHP "memory_limit" settings (currently set to 128M however I tried 256M with no change).

What I noted was that "poller_output_empty.php" never completes if I run it. It runs in an infinite loop. I enabled DEBUG mode and found that the following entries are logged infinitely many times per second in the "cacti.log". Does this indicate a potential bug?

Code: Select all

02/27/2008 03:03:35 AM - CMDPHP: Poller[0] DEBUG: SQL Cell: "SELECT count(*) FROM poller_output"
02/27/2008 03:03:35 AM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select poller_output.output, poller_output.time, poller_output.local_data_id, poller_item.rrd_path, poller_item.rrd_name, poller_item.rrd_num from (poller_output,poller_item) where (poller_output.local_data_id=poller_item.local_data_id and poller_output.rrd_name=poller_item.rrd_name) LIMIT 10000"
02/27/2008 03:03:35 AM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select   data_template_rrd.data_source_name,   data_input_fields.data_name   from (data_template_rrd,data_input_fields)   where data_template_rrd.data_input_field_id=data_input_fields.id   and data_template_rrd.local_data_id=166"
02/27/2008 03:03:35 AM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select   data_template_rrd.data_source_name,   data_input_fields.data_name   from (data_template_rrd,data_input_fields)   where data_template_rrd.data_input_field_id=data_input_fields.id   and data_template_rrd.local_data_id=96"
02/27/2008 03:03:35 AM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select   data_template_rrd.data_source_name,   data_input_fields.data_name   from (data_template_rrd,data_input_fields)   where data_template_rrd.data_input_field_id=data_input_fields.id   and data_template_rrd.local_data_id=206"
02/27/2008 03:03:35 AM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select   data_template_rrd.data_source_name,   data_input_fields.data_name   from (data_template_rrd,data_input_fields)   where data_template_rrd.data_input_field_id=data_input_fields.id   and data_template_rrd.local_data_id=106"
02/27/2008 03:03:35 AM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select   data_template_rrd.data_source_name,   data_input_fields.data_name   from (data_template_rrd,data_input_fields)   where data_template_rrd.data_input_field_id=data_input_fields.id   and data_template_rrd.local_data_id=116"
02/27/2008 03:03:35 AM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select   data_template_rrd.data_source_name,   data_input_fields.data_name   from (data_template_rrd,data_input_fields)   where data_template_rrd.data_input_field_id=data_input_fields.id   and data_template_rrd.local_data_id=146"
02/27/2008 03:03:35 AM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select   data_template_rrd.data_source_name,   data_input_fields.data_name   from (data_template_rrd,data_input_fields)   where data_template_rrd.data_input_field_id=data_input_fields.id   and data_template_rrd.local_data_id=176"
02/27/2008 03:03:35 AM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select   data_template_rrd.data_source_name,   data_input_fields.data_name   from (data_template_rrd,data_input_fields)   where data_template_rrd.data_input_field_id=data_input_fields.id   and data_template_rrd.local_data_id=126"
02/27/2008 03:03:35 AM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select   data_template_rrd.data_source_name,   data_input_fields.data_name   from (data_template_rrd,data_input_fields)   where data_template_rrd.data_input_field_id=data_input_fields.id   and data_template_rrd.local_data_id=88"
02/27/2008 03:03:35 AM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select   data_template_rrd.data_source_name,   data_input_fields.data_name   from (data_template_rrd,data_input_fields)   where data_template_rrd.data_input_field_id=data_input_fields.id   and data_template_rrd.local_data_id=136"
02/27/2008 03:03:35 AM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select   data_template_rrd.data_source_name,   data_input_fields.data_name   from (data_template_rrd,data_input_fields)   where data_template_rrd.data_input_field_id=data_input_fields.id   and data_template_rrd.local_data_id=216"
02/27/2008 03:03:35 AM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select   data_template_rrd.data_source_name,   data_input_fields.data_name   from (data_template_rrd,data_input_fields)   where data_template_rrd.data_input_field_id=data_input_fields.id   and data_template_rrd.local_data_id=156"
02/27/2008 03:03:35 AM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select   data_template_rrd.data_source_name,   data_input_fields.data_name   from (data_template_rrd,data_input_fields)   where data_template_rrd.data_input_field_id=data_input_fields.id   and data_template_rrd.local_data_id=186"
02/27/2008 03:03:35 AM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select   data_template_rrd.data_source_name,   data_input_fields.data_name   from (data_template_rrd,data_input_fields)   where data_template_rrd.data_input_field_id=data_input_fields.id   and data_template_rrd.local_data_id=196"

BSOD2600 · Post by **BSOD2600** » Wed Feb 27, 2008 12:03 am

Alright so just for testing... manually run: php cmd.php from the commandline (with debugging enabled). Do you get errors? rrds update? cacti collect data?

tymbow · Post by **tymbow** » Wed Feb 27, 2008 1:02 am

I ran "php.exe cmd.php" with DEBUG enabled. The log file is attached.

Cacti appears to have collected data - in particular the script associated with the RRAs that have not been updating (bluecoat_memory.pl) is returning correct output.

Interestingly however NO RRDs were updated when I ran "php.exe cmd.php" (they are not mentioned in the log and the datestamp did not change on the files).

BSOD2600 · Post by **BSOD2600** » Wed Feb 27, 2008 2:17 pm

When you ran cmd.php manually, was it with an admin account?

Make sure the IUSR and IIS_WPG accounts have read/execute rights on rrdtool.exe and \system32\cmd.exe. Also if you fired up Sysinternals FileMon, you might catch access denied errors when cacti tries to run.

Lastly, you should increase the timeout and memory settings in your php.ini file.

tymbow · Post by **tymbow** » Wed Feb 27, 2008 7:04 pm

The first run was the service account which normally runs the poller.

I re-ran "php cmd.php" with the admin account and the PHP.INI settings changed as follows (cacti log also attached):

Code: Select all

max_execution_time = 600        ; Was 300
max_input_time = 120            ; Was 120
;max_input_nesting_level = 64   ; Unchanged
memory_limit = 256M             ; Was 128M

There was no change. The data was collected but no RRDs were updated. I was also running FILEMON during the execution of "cmd.php" and neither the RRD tool executable nor any of the RRD files were accessed in any way. The permissions for IUSR and IIS_WPG remain unchanged from when they were working correctly several weeks ago and appear correct.

If I run the normal "poller.php" as I normally do (whether using "cmd.php" or "Spine") then the RRDs are all correctly updated except those 14 I am having issues with for which the data is staying in the "poller_ouput" table. For interest I also ran FILEMON during a regular Spine based "poller.php" cycle and FILEMON shows all RRDs are accessed and written to except the 14 confirming my suspicion Cacti never attempts to update them.

BSOD2600 · Post by **BSOD2600** » Wed Feb 27, 2008 8:16 pm

tymbow wrote:The first run was the service account which normally runs the poller.

Does that have admin rights? Simply to for debugging purposes...

tymbow wrote: I re-ran "php cmd.php" with the admin account and the PHP.INI settings changed as follows (cacti log also attached):

I hope you restarted the web server, since then the new php settings take affect.

tymbow wrote:The permissions for IUSR and IIS_WPG remain unchanged from when they were working correctly several weeks ago and appear correct.

While that might be the case, often on upgrades (depending how the user copies/movies files), NTFS permissions need to be reset on cacti files.

tymbow wrote:If I run the normal "poller.php" as I normally do (whether using "cmd.php" or "Spine") then the RRDs are all correctly updated except those 14 I am having issues with for which the data is staying in the "poller_ouput" table.

Interesting.

Have you tried manually purging the contents of the poller_output table from the mysql db? Then see of those problem DS's show up again.

tymbow · Post by **tymbow** » Wed Feb 27, 2008 8:33 pm

BSOD2600 wrote:Does that have admin rights? Simply to for debugging purposes...

Normally no which is why I re-ran it with the full Administrator account the second time with unfortunately no change.

BSOD2600 wrote: I hope you restarted the web server, since then the new php settings take affect.

I went one better and rebooted the box.

BSOD2600 wrote: While that might be the case, often on upgrades (depending how the user copies/movies files), NTFS permissions need to be reset on cacti files.

I am confident it is correct. In any case, if it was wrong I would have seen access denials in the FILEMON output.

BSOD2600 wrote: Interesting.

Yes it is. This is the crux of the issue and why it is driving me mad. Everything works except those RRDs. Cacti gets the data, stores it in poller_output and then forgets about it. Every other bit of data in poller_output is written to the relevant RRDs but not this data. What makes it more annoying is that it was working perfectly.

BSOD2600 wrote: Have you tried manually purging the contents of the poller_output table from the mysql db?

Yes. I truncated the table before I tried various incarnations of running the poller as admin, running "poller_output_empty.php", creating new data sources based on the problematic template etc. Nothing worked. Cacti seems happy to use the data and graph template for this script and will collect data but it will no longer update the RRDs or create new ones based on this template.

I can PM you a copy of my database if that will help.

pregopresto · Post by **pregopresto** » Sun Mar 02, 2008 4:39 pm

same problem here... anybody a solution already?

tymbow · Post by **tymbow** » Mon Mar 03, 2008 5:58 pm

pregopresto wrote:same problem here... anybody a solution already?

I'm curious which RRAs you are having issues with. My problem seems to be isolated to a particular template that uses a Perl based Script/Command as its Data Input Method. All the other SNMP based templates work fine.

The data collection side of the problem template works (script returns data each polling cycle), and the Graph Template displays existing data in the RRDs correctly but Cacti no longer tries to update the RRDs with the data each polling cycle.

Findarato · Post by **Findarato** » Tue Mar 04, 2008 4:36 pm

I am having this error as well a perl script. WeatherBug to be specific. The problem started when I went from 8.7a to 8.7b.

Having exactly the same issues with the same results, but I am on Linux instead of windows, and I did check all the permissions.

03/04/2008 09:55:01 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Potential Data Source Issues for Data Sources: (DS[517]), (DS[519]), (DS[520]), (DS[521]), (DS[522]), (DS[523]), (DS[524]), (DS[525]), (DS[526]), (DS[527]), (DS[528]), (DS[529]), (DS[536]), (DS[537]), (DS[538]), (DS[539]), (DS[540]), (DS[541]), (DS[548]), (DS[549]), (DS[632])

Post by **TheWitness** » Tue Mar 04, 2008 10:11 pm

For those of you having the PHP_SELF issues, simply edit include/global.php and remove the nasty code section that throws that error and you will be back in business.

If you system is internet facing, you should google this site:

PHP_SELF site:forums.cacti.net

And fix the issue. Otherwise, simply rip out the aforementioned code.

TheWintess

[SOLVED] WARNING: Poller Output Table not Empty

[SOLVED] WARNING: Poller Output Table not Empty

Who is online