[SOLVED] Gaps in graphs and "Poller output table not em

[valleed]

Hi,

I've been running cacti for a couple of months now and I think it's just too awesome. It's so customisable and easy to use.

But Since a couple of days, I get gaps in my graphs. I have quite a lot of hosts and usually, I run spine. I encountered this bug and I was able to fix it by using cmd.php instead of spine for a couple of hours. But now, it doesn't work. I tried to play with the pollers settings (# of processes, # of threads) but it doesn't fix it. I made a cleanup in the rra (I had more than 1300 file that we're older than 2 days).

It's running on RedHat EL4 with the following plugins:

Cacti Version - 0.8.7b
Plugin Architecture - 2.1
Poller Type - CMD.php
Server Info - Linux 2.6.9-5.ELsmp
Web Server - Apache/2.0.52 (Red Hat)
PHP - 5.2.6
PHP Extensions - libxml, xml, wddx, tokenizer, sysvshm, sysvsem, sysvmsg, session, pcre, SimpleXML, sockets, SPL, shmop, standard, Reflection, pspell, posix, iconv, hash, gmp, gettext, ftp, filter, exif, date, curl, ctype, calendar, bz2, zlib, openssl, apache2handler, dbase, gd, json, mbstring, mysql, mysqli, PDO, pdo_mysql, pdo_sqlite, snmp, zip
MySQL - 5.0.51a-community
RRDTool - 1.2.23
SNMP - 5.1.2
Plugins

• Global Plugin Settings (settings - v0.5)
  Syslog Monitoring (syslog - v0.5.2)
  Network Discovery (discovery - v0.8.4)
  Read-only Devices Tab (devices - v0.4)
  Cacti Log View (CLog - v1.1)
  Real-time statistics (zond - v0.34c)
  IP subnet Calculator IPv4 IPv6 (ipsubnet - v.4d)
  Thresholds (thold - v0.3.9)
  Host Info (hostinfo - v0.2)
  Network Tools (tools - v0.3)
  Cacti-ReportIt (reportit - v0.6.1)

Most of my tables are InnoDB for performance.

Here's some interresting facts:

Code: Select all

[16:07:48]valleed@MyUberServer(log)$ grep -i "Poller output" cacti.log | awk '{ print $1 " " $2 " " $3 }'
12/23/2008 03:45:03 PM
12/23/2008 03:50:04 PM
12/23/2008 03:55:02 PM
12/23/2008 04:00:02 PM
12/23/2008 04:10:03 PM
12/23/2008 04:20:02 PM
12/23/2008 04:25:01 PM
12/23/2008 04:30:02 PM
12/23/2008 04:40:05 PM
12/23/2008 04:45:03 PM
12/23/2008 07:10:02 PM
12/23/2008 07:15:02 PM
12/23/2008 07:20:02 PM
12/23/2008 07:25:03 PM
12/23/2008 07:30:03 PM
12/23/2008 07:50:02 PM
12/23/2008 07:55:01 PM
12/23/2008 08:00:03 PM
12/23/2008 08:05:03 PM
12/23/2008 08:10:02 PM
12/23/2008 08:15:02 PM
12/23/2008 08:20:03 PM
12/23/2008 08:25:02 PM
12/23/2008 08:30:02 PM
12/23/2008 08:50:02 PM
12/23/2008 08:55:01 PM
12/23/2008 09:00:02 PM
12/23/2008 10:20:02 PM
12/23/2008 11:10:04 PM
12/24/2008 12:25:02 AM
12/24/2008 07:45:02 AM
12/24/2008 01:25:02 PM
12/24/2008 01:35:07 PM
12/24/2008 03:50:02 PM
12/24/2008 03:55:02 PM
12/24/2008 04:00:02 PM
12/24/2008 04:05:02 PM
12/24/2008 04:10:05 PM
12/24/2008 04:15:02 PM

And

Code: Select all

12/24/2008 01:56:57 PM - SYSTEM STATS: Time:113.9870 Method:spine Processes:3 Threads:15 Hosts:409 HostsPerProcess:137 DataSources:9800 RRDsProcessed:5881
12/24/2008 02:02:18 PM - SYSTEM STATS: Time:134.6256 Method:spine Processes:3 Threads:15 Hosts:409 HostsPerProcess:137 DataSources:9800 RRDsProcessed:5881
12/24/2008 02:06:00 PM - SYSTEM STATS: Time:58.9444 Method:spine Processes:3 Threads:15 Hosts:409 HostsPerProcess:137 DataSources:9800 RRDsProcessed:5881
12/24/2008 02:10:52 PM - SYSTEM STATS: Time:50.4035 Method:spine Processes:3 Threads:15 Hosts:409 HostsPerProcess:137 DataSources:9800 RRDsProcessed:5881
12/24/2008 02:25:44 PM - SYSTEM STATS: Time:41.6695 Method:spine Processes:3 Threads:15 Hosts:409 HostsPerProcess:137 DataSources:9801 RRDsProcessed:5882
12/24/2008 02:31:13 PM - SYSTEM STATS: Time:70.2758 Method:spine Processes:3 Threads:15 Hosts:409 HostsPerProcess:137 DataSources:9801 RRDsProcessed:5882
12/24/2008 02:36:14 PM - SYSTEM STATS: Time:71.8988 Method:spine Processes:3 Threads:15 Hosts:409 HostsPerProcess:137 DataSources:9800 RRDsProcessed:5881
12/24/2008 02:41:39 PM - SYSTEM STATS: Time:97.7411 Method:spine Processes:3 Threads:15 Hosts:409 HostsPerProcess:137 DataSources:9800 RRDsProcessed:5881
12/24/2008 03:07:01 PM - SYSTEM STATS: Time:119.6557 Method:spine Processes:3 Threads:15 Hosts:409 HostsPerProcess:137 DataSources:9800 RRDsProcessed:5881
12/24/2008 03:11:31 PM - SYSTEM STATS: Time:88.9556 Method:spine Processes:3 Threads:15 Hosts:409 HostsPerProcess:137 DataSources:9801 RRDsProcessed:5882
12/24/2008 03:17:00 PM - SYSTEM STATS: Time:116.1326 Method:spine Processes:3 Threads:15 Hosts:409 HostsPerProcess:137 DataSources:9801 RRDsProcessed:5882
12/24/2008 03:21:36 PM - SYSTEM STATS: Time:93.7456 Method:spine Processes:3 Threads:15 Hosts:409 HostsPerProcess:137 DataSources:9801 RRDsProcessed:5882

Oh, and I already tried to rebuild the poller cache a couple of times, but it doesn't seem like it fixes it.

Is there anything else I can check?

Thank you again for your great product.

David

[valleed]

I just tried to drop the poller output table and recreate it like in cacti.sql but it doesn't fix the problem. The next polling I get the same error message in the log and I still have the gaps in the graphs.

[toolman667]

How did you solve this? I am getting the same thing.

Cacti had been running fine in a VM for about 4 months, then I upgraded the server it was running on, and exported the SQL table, and imported it back again, everything seems fine. I use cmd.php, and it runs every minute (with 5 minutes in cron). It has no problem collecting data, when I am watching it, but during the night It will just crap out for 5-10 minutes, and stop collecting data on a couple data sources, but the rest will be collected fine. The WARNING in the logs pertaining to this time frame is:

Code: Select all

12/25/2008 05:18:25 AM - CMDPHP: Poller[0] Host[3] DS[222] WARNING: Result from SERVER not valid.  Partial Result: 12/25/2008 05:18:25 
12/25/2008 05:21:01 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty.  Potential Data Source Issues for Data Sources: cpu_system(DS[38]), mem_buffers(DS[43]), cpu_nice(DS[46]), load_5min(DS[51]), hdd_total(DS[56]), hdd_used(DS[56]), hdd_total(DS[61]), hdd_used(DS[61]), cpu_user(DS[74]), mem_cache(DS[79]), sensorReading(DS[96]), sensorReading(DS[97]), sensorReading(DS[98]), sensorReading(DS[99]), sensorReading(DS[100]), sensorReading(DS[101]), sensorReading(DS[102]), sensorReading(DS[110]), sensorReading(DS[112]), sensorReading(DS[113]), sensorReading(DS[114]), sensorReading(DS[115]), sensorReading(DS[116]), hdd_total(DS[221]), hdd_used(DS[221]), traffic_in(DS[224]), traffic_out(DS[224]), traffic_in(DS[225]), traffic_out(DS[225]), traffic_in(DS[226]), traffic_out(DS[226]), traffic_in(DS[231]), traffic_out(DS[231]), traffic_in(DS[232]), traffic_out(DS[232]), traffic_in(DS[233]), traffic_out(DS[233]), traffic_in(DS[234]), traffic_out(DS[234]), Temperature_Celsius(DS[272]), Temperature_Celsius(DS[277])
12/25/2008 05:22:01 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty.  Potential Data Source Issues for Data Sources: hdd_total(DS[34]), hdd_used(DS[34]), load_1min(DS[40]), mem_free(DS[45]), cpu_user(DS[48]), mem_cache(DS[53]), hdd_total(DS[58]), hdd_used(DS[58]), proc(DS[64]), load_15min(DS[76]), sensorReading(DS[96]), sensorReading(DS[97]), sensorReading(DS[98]), sensorReading(DS[99]), sensorReading(DS[100]), sensorReading(DS[101]), sensorReading(DS[102]), sensorReading(DS[110]), sensorReading(DS[112]), sensorReading(DS[113]), sensorReading(DS[114]), sensorReading(DS[115]), sensorReading(DS[116]), hdd_total(DS[223]), hdd_used(DS[223]), traffic_in(DS[224]), traffic_out(DS[224]), traffic_in(DS[225]), traffic_out(DS[225]), traffic_in(DS[226]), traffic_out(DS[226]), traffic_in(DS[231]), traffic_out(DS[231]), traffic_in(DS[232]), traffic_out(DS[232]), traffic_in(DS[233]), traffic_out(DS[233]), traffic_in(DS[234]), traffic_out(DS[234]), Temperature_Celsius(DS[274]), Temperature_Celsius(DS[279])

Because of this, I am also seeing gaps in my graphs, but only about once or twice a day, on random data sources.

Any info would be great! Thanks!

[valleed]

I redirected the poller stdout to a file instead of /dev/null and saw that a modification I have done to thold plugin had screwed up the poller. When a threadshold was flagged, the data was not saved.