Graphs suddenly have gaps

[winnkid]

A little over a week ago, we had to do a reboot of the virtual machine our Cacti installation runs on. Ever since that reboot, there have been gaps in most of the graphs and the log looks similar to this:

09/27/2006 08:55:06 AM - SYSTEM STATS: Time:6.0999 Method:cmd.php Processes:3 Threads:N/A Hosts:38 HostsPerProcess:13 DataSources:239 RRDsProcessed:147
09/27/2006 08:50:06 AM - EXPORT STATS: ExportTime:0.0051 TotalGraphs:0
09/27/2006 08:50:05 AM - SYSTEM STATS: Time:5.6018 Method:cmd.php Processes:3 Threads:N/A Hosts:38 HostsPerProcess:13 DataSources:239 RRDsProcessed:147
09/27/2006 08:45:29 AM - EXPORT STATS: ExportTime:0.0038 TotalGraphs:0
09/27/2006 08:45:29 AM - SYSTEM STATS: Time:28.9645 Method:cmd.php Processes:3 Threads:N/A Hosts:38 HostsPerProcess:13 DataSources:239 RRDsProcessed:14
09/27/2006 08:40:06 AM - EXPORT STATS: ExportTime:0.0041 TotalGraphs:0
09/27/2006 08:40:06 AM - SYSTEM STATS: Time:5.7295 Method:cmd.php Processes:3 Threads:N/A Hosts:38 HostsPerProcess:13 DataSources:239 RRDsProcessed:147
09/27/2006 08:40:03 AM - CMDPHP: Poller[0] Host[40] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 08:40:03 AM - CMDPHP: Poller[0] Host[39] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 08:40:02 AM - CMDPHP: Poller[0] Host[26] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 08:40:02 AM - CMDPHP: Poller[0] Host[25] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 08:40:02 AM - CMDPHP: Poller[0] Host[23] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 08:40:01 AM - CMDPHP: Poller[0] Host[7] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 08:40:01 AM - CMDPHP: Poller[0] Host[6] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 08:40:01 AM - CMDPHP: Poller[0] Host[5] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 08:35:06 AM - EXPORT STATS: ExportTime:0.0043 TotalGraphs:0
09/27/2006 08:35:06 AM - SYSTEM STATS: Time:6.0622 Method:cmd.php Processes:3 Threads:N/A Hosts:38 HostsPerProcess:13 DataSources:239 RRDsProcessed:147
09/27/2006 08:30:11 AM - EXPORT STATS: ExportTime:0.0042 TotalGraphs:0
09/27/2006 08:30:10 AM - SYSTEM STATS: Time:10.2506 Method:cmd.php Processes:3 Threads:N/A Hosts:38 HostsPerProcess:13 DataSources:239 RRDsProcessed:95
09/27/2006 08:30:08 AM - CMDPHP: Poller[0] Host[26] ERROR: HOST EVENT: Host is DOWN Message: Host did not respond to SNMP
09/27/2006 08:30:07 AM - CMDPHP: Poller[0] Host[7] ERROR: HOST EVENT: Host is DOWN Message: Host did not respond to SNMP
09/27/2006 08:30:07 AM - CMDPHP: Poller[0] Host[40] ERROR: HOST EVENT: Host is DOWN Message: Host did not respond to SNMP
09/27/2006 08:30:06 AM - CMDPHP: Poller[0] Host[25] ERROR: HOST EVENT: Host is DOWN Message: Host did not respond to SNMP
09/27/2006 08:30:05 AM - CMDPHP: Poller[0] Host[6] ERROR: HOST EVENT: Host is DOWN Message: Host did not respond to SNMP
09/27/2006 08:30:05 AM - CMDPHP: Poller[0] Host[39] ERROR: HOST EVENT: Host is DOWN Message: Host did not respond to SNMP
09/27/2006 08:30:04 AM - CMDPHP: Poller[0] Host[23] ERROR: HOST EVENT: Host is DOWN Message: Host did not respond to SNMP
09/27/2006 08:30:03 AM - CMDPHP: Poller[0] Host[5] ERROR: HOST EVENT: Host is DOWN Message: Host did not respond to SNMP
09/27/2006 08:25:29 AM - EXPORT STATS: ExportTime:0.0038 TotalGraphs:0
09/27/2006 08:25:28 AM - SYSTEM STATS: Time:28.6298 Method:cmd.php Processes:3 Threads:N/A Hosts:38 HostsPerProcess:13 DataSources:239 RRDsProcessed:14
09/27/2006 08:20:06 AM - EXPORT STATS: ExportTime:0.0043 TotalGraphs:0
09/27/2006 08:20:06 AM - SYSTEM STATS: Time:5.6396 Method:cmd.php Processes:3 Threads:N/A Hosts:38 HostsPerProcess:13 DataSources:239 RRDsProcessed:147
09/27/2006 08:15:06 AM - EXPORT STATS: ExportTime:0.0046 TotalGraphs:0
09/27/2006 08:15:06 AM - SYSTEM STATS: Time:5.6360 Method:cmd.php Processes:3 Threads:N/A Hosts:38 HostsPerProcess:13 DataSources:239 RRDsProcessed:147
09/27/2006 08:10:06 AM - EXPORT STATS: ExportTime:0.0047 TotalGraphs:0
09/27/2006 08:10:06 AM - SYSTEM STATS: Time:5.7412 Method:cmd.php Processes:3 Threads:N/A Hosts:38 HostsPerProcess:13 DataSources:239 RRDsProcessed:147
09/27/2006 08:05:06 AM - EXPORT STATS: ExportTime:0.0038 TotalGraphs:0
09/27/2006 08:05:06 AM - SYSTEM STATS: Time:5.8298 Method:cmd.php Processes:3 Threads:N/A Hosts:38 HostsPerProcess:13 DataSources:239 RRDsProcessed:147
09/27/2006 08:00:07 AM - EXPORT STATS: ExportTime:0.0042 TotalGraphs:0
09/27/2006 08:00:06 AM - SYSTEM STATS: Time:6.3129 Method:cmd.php Processes:3 Threads:N/A Hosts:38 HostsPerProcess:13 DataSources:239 RRDsProcessed:147
09/27/2006 07:55:14 AM - EXPORT STATS: ExportTime:0.0048 TotalGraphs:0
09/27/2006 07:55:14 AM - SYSTEM STATS: Time:14.1281 Method:cmd.php Processes:3 Threads:N/A Hosts:38 HostsPerProcess:13 DataSources:239 RRDsProcessed:85
09/27/2006 07:50:06 AM - EXPORT STATS: ExportTime:0.0043 TotalGraphs:0
09/27/2006 07:50:06 AM - SYSTEM STATS: Time:5.6324 Method:cmd.php Processes:3 Threads:N/A Hosts:38 HostsPerProcess:13 DataSources:239 RRDsProcessed:147
09/27/2006 07:45:06 AM - EXPORT STATS: ExportTime:0.0039 TotalGraphs:0
09/27/2006 07:45:06 AM - SYSTEM STATS: Time:5.7179 Method:cmd.php Processes:3 Threads:N/A Hosts:38 HostsPerProcess:13 DataSources:239 RRDsProcessed:147
09/27/2006 07:40:29 AM - EXPORT STATS: ExportTime:0.0039 TotalGraphs:0
09/27/2006 07:40:29 AM - SYSTEM STATS: Time:28.6777 Method:cmd.php Processes:3 Threads:N/A Hosts:38 HostsPerProcess:13 DataSources:239 RRDsProcessed:14
09/27/2006 07:35:06 AM - EXPORT STATS: ExportTime:0.0046 TotalGraphs:0
09/27/2006 07:35:06 AM - SYSTEM STATS: Time:5.7844 Method:cmd.php Processes:3 Threads:N/A Hosts:38 HostsPerProcess:13 DataSources:239 RRDsProcessed:147
09/27/2006 07:35:03 AM - CMDPHP: Poller[0] Host[47] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:03 AM - CMDPHP: Poller[0] Host[45] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:03 AM - CMDPHP: Poller[0] Host[43] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:03 AM - CMDPHP: Poller[0] Host[42] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:03 AM - CMDPHP: Poller[0] Host[32] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:03 AM - CMDPHP: Poller[0] Host[41] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:03 AM - CMDPHP: Poller[0] Host[40] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:03 AM - CMDPHP: Poller[0] Host[39] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:03 AM - CMDPHP: Poller[0] Host[31] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:02 AM - CMDPHP: Poller[0] Host[30] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:02 AM - CMDPHP: Poller[0] Host[28] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:02 AM - CMDPHP: Poller[0] Host[27] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:02 AM - CMDPHP: Poller[0] Host[26] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:02 AM - CMDPHP: Poller[0] Host[25] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:02 AM - CMDPHP: Poller[0] Host[22] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:02 AM - CMDPHP: Poller[0] Host[21] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:02 AM - CMDPHP: Poller[0] Host[20] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:02 AM - CMDPHP: Poller[0] Host[18] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:02 AM - CMDPHP: Poller[0] Host[17] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:02 AM - CMDPHP: Poller[0] Host[23] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:01 AM - CMDPHP: Poller[0] Host[7] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:01 AM - CMDPHP: Poller[0] Host[6] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:01 AM - CMDPHP: Poller[0] Host[5] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:30:06 AM - EXPORT STATS: ExportTime:0.0041 TotalGraphs:0
09/27/2006 07:30:06 AM - SYSTEM STATS: Time:5.8152 Method:cmd.php Processes:3 Threads:N/A Hosts:38 HostsPerProcess:13 DataSources:239 RRDsProcessed:147
09/27/2006 07:25:20 AM - EXPORT STATS: ExportTime:0.0044 TotalGraphs:0
09/27/2006 07:25:20 AM - SYSTEM STATS: Time:19.7944 Method:cmd.php Processes:3 Threads:N/A Hosts:38 HostsPerProcess:13 DataSources:239 RRDsProcessed:57
09/27/2006 07:25:19 AM - CMDPHP: Poller[0] Host[35] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:25:19 AM - CMDPHP: Poller[0] Host[34] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:25:19 AM - CMDPHP: Poller[0] Host[48] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:25:19 AM - CMDPHP: Poller[0] Host[33] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:20:06 AM - EXPORT STATS: ExportTime:0.0041 TotalGraphs:0
09/27/2006 07:20:06 AM - SYSTEM STATS: Time:5.7205 Method:cmd.php Processes:3 Threads:N/A Hosts:38 HostsPerProcess:13 DataSources:239 RRDsProcessed:147
09/27/2006 07:15:29 AM - EXPORT STATS: ExportTime:0.0041 TotalGraphs:0
09/27/2006 07:15:29 AM - SYSTEM STATS: Time:28.6153 Method:cmd.php Processes:3 Threads:N/A Hosts:38 HostsPerProcess:13 DataSources:239 RRDsProcessed:14
09/27/2006 07:15:24 AM - CMDPHP: Poller[0] Host[35] ERROR: HOST EVENT: Host is DOWN Message: Host did not respond to SNMP
09/27/2006 07:15:22 AM - CMDPHP: Poller[0] Host[34] ERROR: HOST EVENT: Host is DOWN Message: Host did not respond to SNMP
09/27/2006 07:15:20 AM - CMDPHP: Poller[0] Host[33] ERROR: HOST EVENT: Host is DOWN Message: Host did not respond to SNMP
09/27/2006 07:15:19 AM - CMDPHP: Poller[0] Host[48] ERROR: HOST EVENT: Host is DOWN Message: Host did not respond to SNMP
09/27/2006 07:15:18 AM - CMDPHP: Poller[0] Host[32] ERROR: HOST EVENT: Host is DOWN Message: Host did not respond to SNMP
09/27/2006 07:15:18 AM - CMDPHP: Poller[0] Host[22] ERROR: HOST EVENT: Host is DOWN Message: Host did not respond to SNMP
09/27/2006 07:15:17 AM - CMDPHP: Poller[0] Host[47] ERROR: HOST EVENT: Host is DOWN Message: Host did not respond to SNMP
09/27/2006 07:15:16 AM - CMDPHP: Poller[0] Host[31] ERROR: HOST EVENT: Host is DOWN Message: Host did not respond to SNMP
09/27/2006 07:15:16 AM - CMDPHP: Poller[0] Host[21] ERROR: HOST EVENT: Host is DOWN Message: Host did not respond to SNMP
09/27/2006 07:15:15 AM - CMDPHP: Poller[0] Host[45] ERROR: HOST EVENT: Host is DOWN Message: Host did not respond to SNMP
09/27/2006 07:15:14 AM - CMDPHP: Poller[0] Host[30] ERROR: HOST EVENT: Host is DOWN Message: Host did not respond to SNMP
09/27/2006 07:15:14 AM - CMDPHP: Poller[0] Host[20] ERROR: HOST EVENT: Host is DOWN Message: Host did not respond to SNMP
09/27/2006 07:15:13 AM - CMDPHP: Poller[0] Host[43] ERROR: HOST EVENT: Host is DOWN Message: Host did not respond to SNMP
09/27/2006 07:15:12 AM - CMDPHP: Poller[0] Host[28] ERROR: HOST EVENT: Host is DOWN Message: Host did not respond to SNMP
09/27/2006 07:15:12 AM - CMDPHP: Poller[0] Host[18] ERROR: HOST EVENT: Host is DOWN Message: Host did not respond to SNMP
09/27/2006 07:15:11 AM - CMDPHP: Poller[0] Host[42] ERROR: HOST EVENT: Host is DOWN Message: Host did not respond to SNMP
09/27/2006 07:15:10 AM - CMDPHP: Poller[0] Host[27] ERROR: HOST EVENT: Host is DOWN Message: Host did not respond to SNMP
09/27/2006 07:15:10 AM - CMDPHP: Poller[0] Host[17] ERROR: HOST EVENT: Host is DOWN Message: Host did not respond to SNMP
09/27/2006 07:15:09 AM - CMDPHP: Poller[0] Host[41] ERROR: HOST EVENT: Host is DOWN Message: Host did not respond to SNMP
09/27/2006 07:10:25 AM - EXPORT STATS: ExportTime:0.0035 TotalGraphs:0
09/27/2006 07:10:25 AM - SYSTEM STATS: Time:24.6843 Method:cmd.php Processes:3 Threads:N/A Hosts:38 HostsPerProcess:13 DataSources:239 RRDsProcessed:30
09/27/2006 07:05:13 AM - EXPORT STATS: ExportTime:0.0037 TotalGraphs:0
09/27/2006 07:05:07 AM - SYSTEM STATS: Time:7.1836 Method:cmd.php Processes:3 Threads:N/A Hosts:38 HostsPerProcess:13 DataSources:239 RRDsProcessed:147
09/27/2006 07:00:29 AM - EXPORT STATS: ExportTime:0.0043 TotalGraphs:0
09/27/2006 07:00:29 AM - SYSTEM STATS: Time:28.5521 Method:cmd.php Processes:3 Threads:N/A Hosts:38 HostsPerProcess:13 DataSources:239 RRDsProcessed:14

Cacti: 0.8.6h
MySQL: 4.1.12a-nt
PHP: 5.0.4
Apache: 2.0.54

I have patched Cacti to no avail.

Thanks for any ideas!

[BSOD2600]

Notice all those devices going up and down? That is the likely cause.

[winnkid]

Hi,

No, the devices are not going down. They are monitored seperately and never show as down. I also monitor the server itself and never drops a ping request.

I cannot track this to the server or the devices going down. Just the random messages in the log that the devices are not reachable.

Any other things I can look into?

Thanks!

[BSOD2600]

No, the devices are not going down.

False. According to cacti they ARE going down

09/27/2006 07:35:03 AM - CMDPHP: Poller[0] Host[47] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:03 AM - CMDPHP: Poller[0] Host[45] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:03 AM - CMDPHP: Poller[0] Host[43] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:03 AM - CMDPHP: Poller[0] Host[42] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:03 AM - CMDPHP: Poller[0] Host[32] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:03 AM - CMDPHP: Poller[0] Host[41] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:03 AM - CMDPHP: Poller[0] Host[40] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:03 AM - CMDPHP: Poller[0] Host[39] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:03 AM - CMDPHP: Poller[0] Host[31] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:02 AM - CMDPHP: Poller[0] Host[30] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:02 AM - CMDPHP: Poller[0] Host[28] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:02 AM - CMDPHP: Poller[0] Host[27] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:02 AM - CMDPHP: Poller[0] Host[26] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:02 AM - CMDPHP: Poller[0] Host[25] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:02 AM - CMDPHP: Poller[0] Host[22] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:02 AM - CMDPHP: Poller[0] Host[21] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:02 AM - CMDPHP: Poller[0] Host[20] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:02 AM - CMDPHP: Poller[0] Host[18] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:02 AM - CMDPHP: Poller[0] Host[17] NOTICE: HOST EVENT: Host Returned from DOWN State:
09/27/2006 07:35:02 AM - CMDPHP: Poller[0] Host[23] NOTICE: HOST EVENT: Host Returned from DOWN State:

What downtime detection method are you using? Try switching to SNMP instead of ICMP pings (default is udp pings).

[TheWitness]

Increase your ping timeout. It is likely that you have either bad cabling, a saturated backbone, or a duplex mismatch. SNMP is low priority UDP traffic that can be discarded before other traffic. Increasing the timeout for ping and increasing retries should help. Open a ticket with your network service provider as well.

TheWitness

[winnkid]

Hi,

Here is the problem. I know the devices are not down. They are monitored from WhatsUP Professional at the same time. All checks to the devices as well as the cacti server are up all the time. It is only Cacti that reports any devices as down.

I think it might be something to do with the server Cacti runs on, but I am looking for any ideas as to why this might be happening so suddenly.

Thank you.

[BSOD2600]

Well there is something that is going on, which is causing cacti to be unable to connect to your external devices. Maybe check your firewall/av/routers for other logs that could indicate why it's happening.

Did you increase the timeout like TheWitness suggested?

[winnkid]

Hi,

I have finally tracked it down to the virtual server's interface. I am not sure why WhatsUp did not notice the problem before today. At random interval, the virtual server can't ping its gateway, but the host machine is reachable. I need to dig into the virtual network settings.

Thanks for the suggestions and help. Cacti is a great product!

[winnkid]

Well, I guess this wasn't fixed. The gaps are appearing in some but not all graphs. About half of the devices we monitor show this issue, while the rest do not. Same stuff in the log, "Host not reachable" and the like.

Any other suggestions?

Thank you

[BSOD2600]

1) you using cmd.php or cactid.exe poller?
2) What uptime detection method do you have cacti set to use? Depending on that, you need to test that the devices in question are truly accessible via those methods.

My guess is you still have problems with the host OS and virtual OS's interfaces/firewalls.