[SOLVED] Permissions/ownership and rrd creation

[cirrhus9]

I am fit to be tied up and locked away in a Mental Institution (again).

I have a Version 0.8.7g install on an Ubuntu-based host in the Amazon cloud (AWS).
Everything was fine until a Interface - Traffic graph quit graphing the other day and I had to re-create it and I believe I used the same rrd file.

Right now I have these symptoms:
1.) Creating new graphs does NOT write out the rrd file(s).
Usually, I don't mind this and I have ignored the "RRDTool Says: ERROR: opening '/var/lib/cacti/rra/ftbvarnish2_traffic_in_529.rrd': No such file or directory" in the past and eventually, it "caught up" and created the rrd file.

2.) The cacti.log is 0 bytes and has been 0 bytes since 2011-05-24 when I believe I did a Purge.

I am trying my best to recover my sanity and the network graphs for this device. I followed the post by DeeL at http://forums.cacti.net/viewtopic.php?p=195661#p195661 and my perms are the same as outlined in that post/thread.

snmpwalk does work to the target host.
Rebuild SNMP Cache from System Utilities NEVER updates any network/traffic graphs that I have seen today.

I have (re)imported cacti_host_template_ucdnet_snmp_host template

Here are the perms/ownership for key directories/files:

Code: Select all

drwxr-xr-x 5 root     root      4096 2011-05-16 07:21 /usr/share/cacti/
drwxrwx--- 2 root     www-data 20480 2011-06-27 08:22 /var/lib/cacti/rra
drwxr-xr-x 2 www-data www-data  4096 2011-05-24 23:25 /var/log/cacti/
contents in/under /var/lib/cacti/rra are root www-data

My fear is that I have botched this up trying to "make" it work, rather than let it work and I am now seeking the best method approach to recover this situation.

I even installed an Ubuntu VM host and checked the perms from that install against this one, but I could determine if anything is out of whack. They appear to be identical.

My /var/lib/cacti/rra is 1.1G for 252 files which seems "excessive" considering that all my graphs (Graph List for all devices) only total 18.

Graph List total= 18
Data Source List Total=38

poller.php via c-li says
06/27/2011 01:04:46 PM - SYSTEM STATS: Time:7.3002 Method:cmd.php Processes:1 Threads:N/A Hosts:6 HostsPerProcess:6 DataSources:39 RRDsProcessed:31 and that seems to be accurate

Code: Select all

find `pwd` -mtime -1
/var/lib/cacti/rra/ftbmysql_hdd_free_186.rrd
/var/lib/cacti/rra/ftbmysql_load_5min_198.rrd
/var/lib/cacti/rra/ftbvarnish_load_1min_142.rrd
/var/lib/cacti/rra/ftbvarnish_cpu_system_131.rrd
/var/lib/cacti/rra/norveld_hdd_used_378.rrd
/var/lib/cacti/rra/ftbmysql_load_1min_176.rrd
/var/lib/cacti/rra/ftbmysql_cpu_nice_170.rrd
/var/lib/cacti/rra/norveld_hdd_used_379.rrd
/var/lib/cacti/rra/ftbmysql_load_1min_196.rrd
/var/lib/cacti/rra/ftbmysql_cpu_system_168.rrd
/var/lib/cacti/rra/ftbvarnish_cpu_user_132.rrd
/var/lib/cacti/rra/ftbvarnish_mem_free_146.rrd
/var/lib/cacti/rra/ftbvarnish_load_15min_144.rrd
/var/lib/cacti/rra/ftbmysql_load_15min_197.rrd
/var/lib/cacti/rra/ftbvarnish_mem_cache_147.rrd
/var/lib/cacti/rra/ftbvarnish_load_5min_143.rrd
/var/lib/cacti/rra/localhost_load_1min_194.rrd
/var/lib/cacti/rra/ftbmysql_mem_cache_165.rrd
/var/lib/cacti/rra/ftbmysql_proc_190.rrd
/var/lib/cacti/rra/ftbmysql_mem_cache_174.rrd
/var/lib/cacti/rra/ftbmysql_load_15min_177.rrd
/var/lib/cacti/rra/ftbmysql_mem_free_175.rrd
/var/lib/cacti/rra/ftbvarnish_varnish_requests_414.rrd
/var/lib/cacti/rra/norveld_netoutqueuelength_507.rrd
/var/lib/cacti/rra/ftbvarnish_mem_buffers_148.rrd
/var/lib/cacti/rra/ftbmysql_load_5min_178.rrd
/var/lib/cacti/rra/ftbmysql_mem_buffers_173.rrd
/var/lib/cacti/rra/localhost_load_1min_195.rrd
/var/lib/cacti/rra/ftbmysql_hdd_free_187.rrd
/var/lib/cacti/rra/ftbmysql_cpu_user_169.rrd
/var/lib/cacti/rra/ftbvarnish_cpu_nice_130.rrd

all have a Change time of 2011-06-27 13:15:xx

I have RTFM'd until I could puke. </head_spinning>

Thank you for your time and expertise in this matter. I appreciate it.

[noname]

What user runs poller.php?
(If you set 777 to /var/lib/cacti/rra/, who creates rrd file?)

[cirrhus9]

noname:

crontab -u www-data -l

Code: Select all

*/5 * * * * cacti php /usr/share/cacti/site/poller.php > /dev/null 2>&1

I'm setting the /var/lib/cacti/rra/ at 777 now and will update this post when I know more.

Thanks!

Edit: They are not being created and the "Data Source Path" in Data Sources is empty.

Thank you for your time.

[noname]

crontab -u www-data -l

Code: Select all

*/5 * * * * cacti php /usr/share/cacti/site/poller.php > /dev/null 2>&1

Though I'm not familiar with Ubuntu,
perhaps your crontab settings is not for particular user (www-data) but for system (/etc/crontab).

- Install and Configure Cacti
- Ubuntu Installation Instructions

If you want to execute poller as www-data, probably crontab settings should be..

In /etc/crontab:

Code: Select all

*/5 * * * * www-data php /usr/share/cacti/site/poller.php > /dev/null 2>&1

-or-

In www-data's crontab:

Code: Select all

*/5 * * * * php /usr/share/cacti/site/poller.php > /dev/null 2>&1

[cirrhus9]

Adjusted www-data's cron to

Code: Select all

*/5 * * * * php /usr/share/cacti/site/poller.php > /dev/null 2>&1

Thank you.

[cirrhus9]

Here's what I've done and found out....

from Chapter 21. Debugging

chown cacti:cacti /var/lib/cacti/rra/*.rrd

"Check MySQL updating"
I ran
grep "update poller_reindex" /var/log/cacti/cacti.log | grep "host_id='2'" and got these 'hits'

Code: Select all

06/28/2011 11:18:06 AM - CMDPHP: Poller[0] DEVEL: SQL Exec: "update poller_reindex set assert_value='27147471' where host_id='2' and data_query_id='2' and arg1='.1.3.6.1.2.1.1.3.0'"
06/28/2011 11:18:06 AM - CMDPHP: Poller[0] DEVEL: SQL Exec: "update poller_reindex set assert_value='27147480' where host_id='2' and data_query_id='6' and arg1='.1.3.6.1.2.1.1.3.0'"
06/28/2011 11:18:06 AM - CMDPHP: Poller[0] DEVEL: SQL Exec: "update poller_reindex set assert_value='27147489' where host_id='2' and data_query_id='55' and arg1='.1.3.6.1.2.1.1.3.0'"
06/28/2011 11:18:06 AM - CMDPHP: Poller[0] DEVEL: SQL Exec: "update poller_reindex set assert_value='27147497' where host_id='2' and data_query_id='62' and arg1='.1.3.6.1.2.1.1.3.0'"

"Check rrd file updating"
RRD files should be created by the poller. If it does not create them, it will not fill them either. If it does, please check your Poller Cache from Utilities and search for your target. Does the query show up here?"
No, it does not.
So I manually created the one Graph Template - ftbvarnish - Traffic > *Turn On Graph Debug Mode" was "looking" for.

rrdtool update <filename> --template ...
I am unclear on this step
rrdtool update /var/lib/cacti/rra/ftbvarnish_traffic_in_537.rrd --template ...

php -q add_graph_template.php --host-id=2 --list-graph-templates
Known Graph Templates:(id, name)
2 Interface - Traffic (bits/sec)


I went into Graph Management and enabled DEBUG and then I created the rrd from one of the DEF= statements
touch /var/lib/cacti/rra/ftbvarnish_traffic_in_537.rrd
chown cacti:cacti /var/lib/cacti/rra/ftbvarnish_traffic_in_537.rrd

I grabbed the RRDTool Command: contents and ran it in console and got this:

Code: Select all

/usr/bin/rrdtool graph - \
> --imgformat=PNG \
> --start=-86400 \
> --end=-300 \
> --title='ftbvarnish - Traffic' \
> --rigid \
> --base=1000 \
> --height=120 \
> --width=500 \
> --alt-autoscale \
> --vertical-label='bits per second' \
> --slope-mode \
> --font TITLE:12: \
> --font AXIS:8: \
> --font LEGEND:10: \
> --font UNIT:8: \
> --x-grid MINUTE:30:HOUR:2:HOUR:2:0:"%l%p" \
> DEF:a="/var/lib/cacti/rra/ftbvarnish_traffic_in_537.rrd":traffic_in:AVERAGE \
> DEF:b="/var/lib/cacti/rra/ftbvarnish_traffic_in_537.rrd":traffic_out:AVERAGE \
> DEF:c="/var/lib/cacti/rra/ftbvarnish_traffic_in_537.rrd":traffic_in:MAX \
> DEF:d="/var/lib/cacti/rra/ftbvarnish_traffic_in_537.rrd":traffic_out:MAX \
> CDEF:cdefa=a,8,* \
> CDEF:cdefe=b,8,* \
> CDEF:cdeff=c,8,* \
> CDEF:cdefg=d,8,* \
> CDEF:cdefj=d,8,*,-1,* \
> AREA:cdefa#00CF00FF:"Inbound"  \
> GPRINT:cdefa:LAST:"   Current\:%8.2lf %s"  \
> GPRINT:cdefa:AVERAGE:"Average\:%8.2lf %s"  \
> GPRINT:cdefa:MAX:"Maximum\:%8.2lf %s\n"  \
> LINE1:cdefe#002A97FF:"Outbound"  \
> LINE1:cdeff#005D57FF:"Peak Usage\n"  \
> GPRINT:cdefg:LAST:"Current\:%8.2lf %s"  \
> GPRINT:cdefg:AVERAGE:"Average\:%8.2lf %s"  \
> GPRINT:cdefg:MAX:"Maximum\:%8.2lf %s\n"  \
> LINE1:cdefj#0000AAFF:"Peak Usage\n" 
ERROR: mmaping file '/var/lib/cacti/rra/ftbvarnish_traffic_in_537.rrd': Invalid argument

I then "doubled back" to "Check rrd file numbers"
I tried these 2 unsuccessfully:
rrdtool update /var/lib/cacti/rra/ftbvarnish_traffic_in_537.rrd --template "Interface - Traffic (bits/sec)"
ERROR: Not enough arguments
rrdtool update /var/lib/cacti/rra/ftbvarnish_traffic_in_537.rrd --template 2
ERROR: Not enough arguments

"Check rrd file numbers"
Failed with:
rrdtool fetch /var/lib/cacti/rra/ftbvarnish_traffic_in_537.rrd AVERAGE
ERROR: mmaping file '/var/lib/cacti/rra/ftbvarnish_traffic_in_537.rrd': Invalid argument
rrdtool info /var/lib/cacti/rra/ftbvarnish_traffic_in_537.rrd
ERROR: mmaping file '/var/lib/cacti/rra/ftbvarnish_traffic_in_537.rrd': Invalid argument

I googled "ERROR: mmaping file Invalid argument" and found a couple of hits but I am not sure they are relevant to a solution.
http://fivejacksons.com/brian/2009/11/r ... -argument/
but I'll leave that to your judgement and experience please.

I will redouble these debugging steps here again today and advise.

The good news is that the cacti.log file is now being written.

Thank you for your time.

Edit: I removed www-data cron and stuck it in the cacti users' cron table.

[noname]

chown cacti:cacti /var/lib/cacti/rra/*.rrd
- snip -
Edit: I removed www-data cron and stuck it in the cacti users' cron table.

Did you switch the user which run poller.php from "www-data" to "cacti"?
Then, check directory permissions whether if that user have access rights to write:
% ls -ld /var/lib/cacti/rra/
% ls -ld /var/log/cacti/


And,

touch /var/lib/cacti/rra/ftbvarnish_traffic_in_537.rrd
chown cacti:cacti /var/lib/cacti/rra/ftbvarnish_traffic_in_537.rrd

probably it is useless because this created empty RRD.

[cirrhus9]

If you mean
*/5 * * * * cacti /usr/bin/php /usr/share/cacti/site/poller.php > /dev/null 2>&1
is the current cacti user's crontab, then "yes".

perms:

Code: Select all

drwxrwxrwx 2 root     www-data 20480 2011-06-28 11:37 /var/lib/cacti/rra/
drwxr-xr-x 2 www-data www-data  4096 2011-06-28 11:19 /var/log/cacti

re "probably it is useless because this created empty RRD."
Well, it was a stab in the dark.

Personally, I "think" that the ownership on /var/lib/cacti/rra/ are incorrect, but that's a hunch and lately my hunches have been off.
If they are incorrect, then either 'chown cacti:www-data /var/lib/cacti/rra/' or adding cacti user to www-data group should fix this issue?

Thanks for sticking with me on this.

[noname]

If you mean
*/5 * * * * cacti /usr/bin/php /usr/share/cacti/site/poller.php > /dev/null 2>&1
is the current cacti user's crontab, then "yes".

Is it necessary to specify particular user in user-level crontab?

Anyway,

perms:

Code: Select all

drwxrwxrwx 2 root     www-data 20480 2011-06-28 11:37 /var/lib/cacti/rra/
drwxr-xr-x 2 www-data www-data  4096 2011-06-28 11:19 /var/log/cacti

- snip -
Personally, I "think" that the ownership on /var/lib/cacti/rra/ are incorrect, but that's a hunch and lately my hunches have been off.

You have already read:

"Check rrd file updating"
RRD files should be created by the poller. If it does not create them, it will not fill them either. If it does, please check your Poller Cache from Utilities and search for your target. Does the query show up here?"

So you should as you mentioned.

If they are incorrect, then either 'chown cacti:www-data /var/lib/cacti/rra/' or adding cacti user to www-data group should fix this issue?

For example:
# chown -R cacti:www-data /var/lib/cacti/rra/
# chmod 755 /var/lib/cacti/rra/

And for existing RRD files:
# chmod 644 /var/lib/cacti/rra/*.rrd

(I'm not sure whether if it needs to add cacti user to www-data group.)


// BTW, I think the topic title is not appropriate for describing the issue.
// That is not useful to other users.

[cirrhus9]

I made the changes you have suggested.

My Next post title will be more helpful to other cacti users on the forum.

I should see a graph tomorrow, it's late here and I am off to bed.

Have a great day, and Thank You.

[noname]

>> My Next post title will be more helpful to other cacti users on the forum.

You can modify the title by editing the first post in this topic.

Have a nice dream.

[cirrhus9]

Now all my graphs are 'hiccupping' as shown in the attachment.
My client might start freaking out.

Thanks.

[noname]

Check whether if multiple pollers are running (or not).
Don't you leave running poller.php for www-data and cacti?

[cirrhus9]

I couldn't go to bed knowing there were busted graphs....

You are correct, there was an additional root cron

Code: Select all

/var/spool/cron/crontabs/www-data:# */5 * * * * /usr/bin/php /usr/share/cacti/site/poller.php > /dev/null 2>&1
/var/spool/cron/crontabs/cacti:*/5 * * * * /usr/bin/php /usr/share/cacti/site/poller.php > /dev/null 2>&1
/var/spool/cron/crontabs/root:*/5 * * * * /usr/bin/php /usr/share/cacti/site/poller.php > /dev/null 2>&1

I left only cacti's.

[cirrhus9]

Edit: Thread Subject modified.
Hicupping now resolved, it was permissions, but I still can't create new graphs, and a handful of previously existing graphs are coming up NaN.

I have to ask if setting/changing the system timezone last week, could have had an effect on these entities?
How can I be certain that the existing .rrd files for these items are sound and can be recovered?
Can I create .rrd files via c-line and then "encourage" cacti to use those?
Can I backup my rrra/*.rrd files, purge and re-install Cacti and/or rrdtool and put them back and still have the populated graphs?

A slice of hope

Thank You.