Broken graphs

[shahin]

Hi Guys,

We have setup cacti for sometimes now on one of our server 2008 R2 to monitor the network. untill last two weeks everything was working great. But now it looks like that we get some empty spaces in the graphs. Nothing has been changed on the Cacti server and the logs show no error at all. Any ide why we suddenly see broken graphs.

This is one of the graphs and we see the same for all of the servers that we monitor (see attachment)

Thaks

[cigamit]

Without seeing the logs for that time period, its difficult to tell. You really don't give us anything to go off of.

[shahin]

Thaks for the tpdate,
Do you mean the IIS logs or logs of Cacti itself?
in the cacti logs only see this warning:
05/29/2018 05:10:01 AM - POLLER: Poller[0] WARNING: Scheduled Task is out of sync with the Poller Interval! The Poller Interval is '300' seconds, with a maximum of a '300' second Scheduled Task, but 516 seconds have passed since the last poll!

If you want I can also upload the portion of the IIS logs as well

Thanks

[cigamit]

Just the Cacti logs is all that is needed. That one particular warning you have would most likely be the problem. Either your scheduled task is not kicking off in time, or your clock time is out of sync. Could be a few other issues, but those are the 1st 2 I would check.

[shahin]

Thanks for your update,

I did check the graphs and it looks like that we again see an break in the graphs between 00:00 AM and 02:00 AM.
at the same time see some warnings in the cacti logs.
Also ren the php c:/cacti_web_root/cacti/poller.php and it shows everything is ok.
can you please tell us how can we first correct these issues?

Thanks

[cigamit]

Your maximum run time is going over. If it always happens during a specific time frame, do you have backups or something else running at that time? You are going from normally taking 10 seconds to taking 5 minutes, so something is definitely slowing your server down during that period.

First step is to go into Settings and up the process count of cmd.php. Having just 1 process makes it really slow. cmd.php is super slow compared to spine, but you can deal with that later. I would up it to say, 10 for now.

[shahin]

Thank you for the update,

sure at the time that we see the blanck space in graphs mostlly is back-up is running
I did change the Number of PHP Script Servers under settings-Poller to 10.
I will give it a try and because this does happen randomly, when I see the issue again I will let you know.

Thanks

[shahin]

Hi,

It looks like that increasing the Poller to 10 did not solve the issue.

Any sugestion?

Thanks

[netniV]

There is an error saying that your poller task is out of sync with the scheduled task. This suggests that your tasks are still overrunning above or becoming hung for some reason (more than likely when trying to communicate with other systems).

Try turning up the logging levels to see if any more information is produced that might make it clearer.

[shahin]

Hi,

I will enable the DeBug, statistics,Errors,results and I/O and will post the result here.

Thanks again.

[shahin]

Hi,

I did enable the debuge logs but did not find any usefull information, or I dont know what I should look for!
Any suggestion on how to get rid of the wranning regarding the poller interval?

[netniV]

There is an option within the Task Scheduler's task properties to say only run one instance (which is used by default). You can change that to be run another copy (or the language equivalent) but I think that would mean you have multiple processes running and polling at the same time.

[shahin]

Hi,

This is what I could find in the logs:
Host[51] ERROR: HOST EVENT: Host is DOWN Message: Host did not respond to SNMP
06/16/2018 10:10:11 PM - CMDPHP: Poller[0] Host[54] RECACHE DQ[1] OID: .1.3.6.1.2.1.1.3.0, output: 23505264
06/16/2018 10:10:11 PM - CMDPHP: Poller[0] Host[54] DS[52] SNMP: v2: 10.0.10.232, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.12, output: 2042039928
06/16/2018 10:10:11 PM - CMDPHP: Poller[0] Host[54] DS[52] SNMP: v2: 10.0.10.232, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.12, output: 1045884375
06/16/2018 10:10:11 PM - CMDPHP: Poller[0] Time: 1.1433


CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'1062', SQL:"insert into poller_output (local_data_id, rrd_name, time, output) values (69, 'traffic_out', '2018-06-13 01:47:26', '1753281276')'
06/13/2018 01:55:01 AM - POLLER: Poller[0] WARNING: Scheduled Task is out of sync with the Poller Interval! The Poller Interval is '300' seconds, with a maximum of a '300' second Scheduled Task, but 481 seconds have passed since the last poll!
06/13/2018 01:55:01 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 6, Data Sources: traffic_in(DS[55]), traffic_out(DS[55]), traffic_in(DS[68]), traffic_out(DS[68]), traffic_in(DS[69]), traffic_out(DS[69])
06/13/2018 01:55:10 AM - SYSTEM STATS: Time:8.4062 Method:cmd.php Processes:10 Threads:N/A Hosts:34 HostsPerProcess:4 DataSources:68 RRDsProcessed:34
06/13/2018 02:00:19 AM - POLLER: Poller[0] WARNING: Scheduled Task is out of sync with the Poller Interval! The Poller Interval is '300' seconds, with a maximum of a '300' second Scheduled Task, but 318 seconds have passed since the last poll!
06/13/2018 02:00:30 AM

[netniV]

CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'1062', SQL:"insert into poller_output (local_data_id, rrd_name, time, output) values (69, 'traffic_out', '2018-06-13 01:47:26', '1753281276')'

This bit worries me. It would suggest to me a problem with your MySQL installation. Unless I'm missing something there, the command looks perfectly valid.

[shahin]

Thank you for your update,

We are running the same Cacti installation for over 2 years now and we never had any problem with it. and now suddenly every day at the same time (backup time) we get this issue.
Any suggestion on how to findout if there is a problem with mysql?

Thanks