Graphs not displaying on 'first' view -- refresh required

[net-guy]

I have a decent size cacti install running quite well, depending on what you consider large. I have 400+ hosts and 30,000 + graphs. I use boost due to disk performance. Everything works fine, and has been for quite some time.

Today someone else noticed and I verified that clicking on any link whether it be graph management, tree, anything -- the graphs don't populate. The page is blank. The normal cacti layout is there, just no graphs show up at all. IE shows broken images, Fire Fox does not. If I hit 'refresh' in firefox, the graphs all show up (same for ie). The same can be accomplished by clicking on the link again.

I know boost is updating the file. I know my browser is requesting them. I'm stumped at where to start troubleshooting. I've looked at error logs for apache, and logs in cacti. Does anyone have any idea of where else to look? The graphs *work*, just nothing shows up on the first view -- it's all broken until I hit 'refresh' or click on the same link again.

Some of my basic stats:
Cacti Version 0.8.7e
Cacti OS unix
SNMP Version NET-SNMP version: 5.1.2
RRDTool Version RRDTool 1.2.x
Hosts 411
Graphs 30343
Data Sources Script/Command: 21
SNMP: 867
SNMP Query: 29684
Script - Script Server (PHP): 13
Total: 30585
Poller Information
Interval 300
Type spine
Items Action[0]: 60470
Action[1]: 21
Action[2]: 13
Total: 60504
Concurrent Processes 1
Max Threads 25
PHP Servers 10
Script Timeout 10
Max OID 50
Last Run Statistics Time:34.9692 Method:spine Processes:1 Threads:25 Hosts:405 HostsPerProcess:405 DataSources:60504 RRDsProcessed:0
PHP Information
PHP Version 5.3.0
PHP OS Linux
PHP uname Linux cpliis4i.homedepot.com 2.6.9-55.ELsmp #1 SMP Fri Apr 20 17:03:35 EDT 2007 i686
PHP SNMP Installed
max_execution_time 30
memory_limit 512M

[net-guy]

If all else fails, simply reboot. The age old solution to everything.
I restarted apache -- and now it works.

No idea why it started behaving like that. No logs, nothing to trace back to any source of the issue. Gracefully reloaded apache, and now we're all good.

[djoul]

Hi everyone,

I have the same issue in my cacti since I installed the boost plugin.
This refresh problem impacts also my targets on nodes, when I open the weathermap for the first time all the nodes appear to 0. So I need to load the graph and recalculate all maps to get the good values on the target on nodes.


My config is:

Boost 2.5
Weathermap 0.96a

Cacti Version : 0.8.7e
Cacti OS : unix
SNMP Version : NET-SNMP version: 5.4.1
RRDTool Version : RRDTool 1.3.x
Hosts : 510
Graphs : 2194
Data Sources Total: 2643
Poller Interval : 300
Type : spine
Items Total: 4268
Concurrent Processes : 8
Max Threads : 25
PHP Servers : 2
Script Timeout : 180
Max OID : 10
Last Run Statistics:
Time:30.7541 Method:spine Processes:8 Threads:25 Hosts:511
HostsPerProcess:64 DataSources:4268 RRDsProcessed:0
PHP Version : 5.2.10-2ubuntu6.1
PHP SNMP Installed
max_execution_time : 300
memory_limit : 256M


I tried to restart apache but it didn't work...

Thank you for your help.

djoul

[rilindo]

I am also got this same type of condition. Just recently installed Boost 3.0 and it works as intended (not running the Boost Server, though) otherwise.

Its an oddest thing - you view the graphs, it comes up as broken. You had to refresh the page to see them. Restart of apache did not resolve the issue (and neither does the server). Cacti runs as root and it is world writeable on the rra directory, so I am not sure where to look at next. Any direction somebody can point me to?

Here is the current Boost status:

Boost On Demand Updating: Idle
Total Data Sources: 14881
Total Boost Records: 14761
Boost Storage Statistics
Database Engine: MyISAM
Current Boost Table Size: 1 MBytes
Avg Bytes/Record: 83 Bytes
Max Allowed Boost Table Size: Unlimited
Estimated Maximum Records: Unlimited Records
Runtime Statistics
Last Start Time: 2010-04-15 12:27:46
Last Run Duration: 19 seconds (1% of update frequency)
RRD Updates: 14773
Peak Poller Memory: 12.38 MBytes
Detailed Runtime Timers: RRDUpdates:14773 TotalTime:19.4974 range_local_data_id:0.11 rcaston_add:0.12 get_records:0.14 results_cycle:18.74 rrd_path:2.88 rrd_template:5.58 rrd_lastupdate:0.48 rrd_field_names:0.11 rrdupdate:0.56 delete:0.27
Max Poller Memory Allowed: 1024 MBytes
Run Time Configuration
Update Frequency: 1 Hour
Next Start Time: 2010-04-15 13:27:46
Maximum Records: 1000000 Records
Maximum Allowed Runtime: 20 Minutes
Boost Server Details
Server Config Status: Disabled
Multiprocess Server: Multiple Process
Update Timeout: 2 Seconds
Server/Port: localhost@9050
Authorized Update Web Servers: 127.0.0.1
RRDtool Binary Used: /usr/bin/rrdtool
Image Caching
Image Cacing Status: Enabled
Cache Directory: /HIDDEN/www/HIDDEN/cache
Cached Files: 151 Files
Cached Files Size: 4 MBytes

[gandalf]

Do you use image caching?
BTB: moving to plugins forum
R.

[rilindo]

Yes, I am. Initially I wasn't, so I went and enabled it. In both cases, there was no change.

Just as an experiment, I went and relocated the cache directory to the ramdisk to see if it is drive (I am using a striped RAID0 volume to store the rrd files). No difference.

[gandalf]

Personally, I noticed that rrdtool update was sometimes lagging behind a bit. I never experienced this on a weathermap, but on "normal" graphs. But to be honest, I did not dig into it to resolve ...
R.

[rilindo]

Huh.

I went right ahead and started up boost_server. That allowed me to see activity in the boost.log file. From what I can tell, it looks like there is some sort of lag when rrdtool updates the file with the rendering of the images. But again, I should be missing file messages in the apache, which I am not, which makes me think that the "lag" is illusionary.

Oddly enough, the queries are showing up in the mysql slow log (time was set to 2 seconds), but running the queries manually shows no issues (thats is, the results came very quickly).

As from Boost's image caching, is there some other cache that I can look at to change, either in cacti or in php?

(it may help that mysql / php configuration is known. I'll get that info when I head home).

[rilindo]

Just for giggles, I tried the 2.5 version and I get the same condition. Also, I went and disabled boost to see if the problem exist outside of boost (it does not).

Maybe I am mistaken, but does cacti wait for boost to write the rrd file or does it read from some cache someplace? It looks like based on what I ee in the graph code, but then again, I don't have a strong background in PHP (that and I obviously don't have the full grasp of the cacti software architecture here).

Here is my config, BTW:

8-Core Xeon Intel
16 Gigs of RAM
1 single OS drive (where the mySQL database resides as well)
1 RAID0 volume (three 7200 RPM SAS disks striped) - with one 300 gig ext4 partition dedicated to rrds
two ram disks, one to hold the cacti and boost logs.
Ubuntu 9.10
Cacti 0.8.7e
Plugin Arch 2.6
Boost 3.0


mySQL config:

-------
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock


[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0

[mysqld]


user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
skip-external-locking
bind-address = 127.0.0.1
key_buffer = 32M
max_allowed_packet = 32M
thread_stack = 192K
thread_cache_size = 16
myisam-recover = BACKUP
max_connections = 3000
max_heap_table_size = 1024M
query_cache_limit = 32M
query_cache_size = 128M
log_slow_queries = /var/log/mysql/mysql-slow.log
long_query_time = 2
log-queries-not-using-indexes
expire_logs_days = 10
max_binlog_size = 100M



[mysqldump]
quick
quote-names
max_allowed_packet = 16M

[mysql]

[isamchk]
key_buffer = 16M
--------

PHP config (apache):

[PHP]






engine = On

zend.ze1_compatibility_mode = Off

short_open_tag = On

asp_tags = Off

precision = 12

y2k_compliance = On

output_buffering = Off


zlib.output_compression = Off


implicit_flush = Off
#implicit_flush = On

unserialize_callback_func=

serialize_precision = 100

allow_call_time_pass_reference = On

safe_mode = Off

safe_mode_gid = Off

safe_mode_include_dir =

safe_mode_exec_dir =

safe_mode_allowed_env_vars = PHP_

safe_mode_protected_env_vars = LD_LIBRARY_PATH




disable_functions =

disable_classes =



#realpath_cache_size=1M


expose_php = On






error_reporting = E_ALL & ~E_NOTICE

display_errors = On

display_startup_errors = Off
#display_startup_errors = On

#log_errors = Off
log_errors = On

log_errors_max_len = 1024

ignore_repeated_errors = Off

ignore_repeated_source = Off

report_memleaks = On


track_errors = Off






error_log = /var/log/apache/php.err






variables_order = "EGPCS"


register_globals = Off

register_long_arrays = On

register_argc_argv = On

auto_globals_jit = On

post_max_size = 32M


magic_quotes_gpc = On

magic_quotes_runtime = Off

magic_quotes_sybase = Off

auto_prepend_file =
auto_append_file =

default_mimetype = "text/html"





doc_root =

user_dir =

#extension_dir = "/etc/php5/conf.d"
extension_dir = "/usr/lib/php5/20060613"

enable_dl = Off










file_uploads = On


upload_max_filesize = 2M



allow_url_fopen = On

allow_url_include = Off



default_socket_timeout = 60






[Date]



[filter]

[iconv]

[sqlite]

[Pcre]


[Syslog]
define_syslog_variables = Off

[mail function]
SMTP = localhost
smtp_port = 25




[SQL]
sql.safe_mode = Off

[ODBC]

odbc.allow_persistent = On

odbc.check_persistent = On

odbc.max_persistent = -1

odbc.max_links = -1

odbc.defaultlrl = 4096

odbc.defaultbinmode = 1

[MySQL]
mysql.allow_persistent = On

mysql.max_persistent = -1

mysql.max_links = -1

mysql.default_port =

mysql.default_socket =

mysql.default_host =

mysql.default_user =

mysql.default_password =

mysql.connect_timeout = 60

mysql.trace_mode = Off

[MySQLi]

mysqli.max_links = -1

mysqli.default_port = 3306

mysqli.default_socket =

mysqli.default_host =

mysqli.default_user =

mysqli.default_pw =

mysqli.reconnect = Off

[mSQL]
msql.allow_persistent = On

msql.max_persistent = -1

msql.max_links = -1

[OCI8]







[PostgresSQL]
pgsql.allow_persistent = On

pgsql.auto_reset_persistent = Off

pgsql.max_persistent = -1

pgsql.max_links = -1

pgsql.ignore_notice = 0

pgsql.log_notice = 0

[Sybase]
sybase.allow_persistent = On

sybase.max_persistent = -1

sybase.max_links = -1


sybase.min_error_severity = 10

sybase.min_message_severity = 10

sybase.compatability_mode = Off

[Sybase-CT]
sybct.allow_persistent = On

sybct.max_persistent = -1

sybct.max_links = -1

sybct.min_server_severity = 10

sybct.min_client_severity = 10

[bcmath]
bcmath.scale = 0

[browscap]

[Informix]
ifx.default_host =

ifx.default_user =

ifx.default_password =

ifx.allow_persistent = On

ifx.max_persistent = -1

ifx.max_links = -1

ifx.textasvarchar = 0

ifx.byteasvarchar = 0

ifx.charasvarchar = 0

ifx.blobinfile = 0

ifx.nullformat = 0

[Session]
session.save_handler = files


session.use_cookies = 1



session.name = PHPSESSID

session.auto_start = 0

session.cookie_lifetime = 0

session.cookie_path = /

session.cookie_domain =

session.cookie_httponly =

session.serialize_handler = php


session.gc_divisor = 100

session.gc_maxlifetime = 1440



session.bug_compat_42 = 1
session.bug_compat_warn = 1

session.referer_check =

session.entropy_length = 0

session.entropy_file =



session.cache_limiter = nocache

session.cache_expire = 180

session.use_trans_sid = 0

session.hash_function = 0

session.hash_bits_per_character = 4

url_rewriter.tags = "a=href,area=href,frame=src,input=src,form=,fieldset="

[MSSQL]
mssql.allow_persistent = On

mssql.max_persistent = -1

mssql.max_links = -1

mssql.min_error_severity = 10

mssql.min_message_severity = 10

mssql.compatability_mode = Off







mssql.secure_connection = Off



[Assertion]





[COM]

[mbstring]








[FrontBase]

[gd]

[exif]

[Tidy]

tidy.clean_output = Off

[soap]
soap.wsdl_cache_enabled=1
soap.wsdl_cache_dir="/tmp"
soap.wsdl_cache_ttl=86400

-----

PHP Config (CLI)

[PHP]






engine = On

zend.ze1_compatibility_mode = Off

short_open_tag = On

asp_tags = Off

precision = 12

y2k_compliance = On

output_buffering = Off


zlib.output_compression = Off


implicit_flush = Off

unserialize_callback_func=

serialize_precision = 100

allow_call_time_pass_reference = On

safe_mode = Off

safe_mode_gid = Off

safe_mode_include_dir =

safe_mode_exec_dir =

safe_mode_allowed_env_vars = PHP_

safe_mode_protected_env_vars = LD_LIBRARY_PATH




disable_functions =

disable_classes =





expose_php = On






error_reporting = E_ALL & ~E_NOTICE

display_errors = On

display_startup_errors = Off

log_errors = Off

log_errors_max_len = 1024

ignore_repeated_errors = Off

ignore_repeated_source = Off

report_memleaks = On


track_errors = Off












variables_order = "EGPCS"


register_globals = Off

register_long_arrays = On

register_argc_argv = On

auto_globals_jit = On

post_max_size = 8M


magic_quotes_gpc = On

magic_quotes_runtime = Off

magic_quotes_sybase = Off

auto_prepend_file =
auto_append_file =

default_mimetype = "text/html"





doc_root =

user_dir =


enable_dl = Off










file_uploads = On


upload_max_filesize = 2M



allow_url_fopen = On

allow_url_include = Off



default_socket_timeout = 60






[Date]



[filter]

[iconv]

[sqlite]

[Pcre]


[Syslog]
define_syslog_variables = Off

[mail function]
SMTP = localhost
smtp_port = 25




[SQL]
sql.safe_mode = Off

[ODBC]

odbc.allow_persistent = On

odbc.check_persistent = On

odbc.max_persistent = -1

odbc.max_links = -1

odbc.defaultlrl = 4096

odbc.defaultbinmode = 1

[MySQL]
mysql.allow_persistent = On

mysql.max_persistent = -1

mysql.max_links = -1

mysql.default_port =

mysql.default_socket =

mysql.default_host =

mysql.default_user =

mysql.default_password =

mysql.connect_timeout = 60

mysql.trace_mode = Off

[MySQLi]

mysqli.max_links = -1

mysqli.default_port = 3306

mysqli.default_socket =

mysqli.default_host =

mysqli.default_user =

mysqli.default_pw =

mysqli.reconnect = Off

[mSQL]
msql.allow_persistent = On

msql.max_persistent = -1

msql.max_links = -1

[OCI8]







[PostgresSQL]
pgsql.allow_persistent = On

pgsql.auto_reset_persistent = Off

pgsql.max_persistent = -1

pgsql.max_links = -1

pgsql.ignore_notice = 0

pgsql.log_notice = 0

[Sybase]
sybase.allow_persistent = On

sybase.max_persistent = -1

sybase.max_links = -1


sybase.min_error_severity = 10

sybase.min_message_severity = 10

sybase.compatability_mode = Off

[Sybase-CT]
sybct.allow_persistent = On

sybct.max_persistent = -1

sybct.max_links = -1

sybct.min_server_severity = 10

sybct.min_client_severity = 10

[bcmath]
bcmath.scale = 0

[browscap]

[Informix]
ifx.default_host =

ifx.default_user =

ifx.default_password =

ifx.allow_persistent = On

ifx.max_persistent = -1

ifx.max_links = -1

ifx.textasvarchar = 0

ifx.byteasvarchar = 0

ifx.charasvarchar = 0

ifx.blobinfile = 0

ifx.nullformat = 0

[Session]
session.save_handler = files


session.use_cookies = 1



session.name = PHPSESSID

session.auto_start = 0

session.cookie_lifetime = 0

session.cookie_path = /

session.cookie_domain =

session.cookie_httponly =

session.serialize_handler = php


session.gc_divisor = 100

session.gc_maxlifetime = 1440



session.bug_compat_42 = 1
session.bug_compat_warn = 1

session.referer_check =

session.entropy_length = 0

session.entropy_file =



session.cache_limiter = nocache

session.cache_expire = 180

session.use_trans_sid = 0

session.hash_function = 0

session.hash_bits_per_character = 4

url_rewriter.tags = "a=href,area=href,frame=src,input=src,form=,fieldset="

[MSSQL]
mssql.allow_persistent = On

mssql.max_persistent = -1

mssql.max_links = -1

mssql.min_error_severity = 10

mssql.min_message_severity = 10

mssql.compatability_mode = Off







mssql.secure_connection = Off



[Assertion]





[COM]

[mbstring]








[FrontBase]

[gd]

[exif]

[Tidy]

tidy.clean_output = Off

[soap]
soap.wsdl_cache_enabled=1
soap.wsdl_cache_dir="/tmp"
soap.wsdl_cache_ttl=86400

----------

[rilindo]

More updates:

I forgot to mention I was running Apache. Anyway, I went and tried out in lighttpd + fastcgi (it was easier than I expected). Running it on a different port, I see the same behavior.

Interesting.

[TheWitness]

You have to be running the latest Weathermap to have it take advantage of Boost. Just thought I would let you know. Don't have the specifics.

TheWitness

[rilindo]

Really? I didn't realize it was a requirement.

Well, I never tried weathermap before. Its worth a try.

[TheWitness]

There are multiple topics here. If you are not running WMap, then disregard.

TheWitness

[rilindo]

Yeah, that confused me. :| Weathermap seems fine, but it doesn't solve the primary problem I have right now. :/

[TheWitness]

The problem is likely in graph_image.php. What other plugins do you have. My suggestion is to start disabling plugins until boost works. Then, we can evaluate which plugin is having consternation with boost.

I have been running boost for a few years now and the only time this happens is when you have an offending plugin. The reason that graphs work after the first refresh is that the images have been rendered in the boost cache and don't have to be rendered by rrdtool for another X minutes.

The way that I would troubleshoot this is to do the following:

1) Use Firefox (IE, well... sucks)
2) Install Firebug
3) Enable the Firebug "Net" Option
4) Render a page with broken images
5) Bring up "Net" subpage of Firebug and find a call to "graph_image"
6) Click the "Response" tab under "Net".
7) At the very beginning, of the gobblygook in the response area, you will likely find some Text like warnings that are polluting the graph stream.

Let us know what you find.

TheWitness