RRD folder/rra sub folder not being created

[gundalba]

Hello there,

I just built a new Cacti server on CentOS box and having problem with RRD file creation.
Added a device and graphs but not seeing the graphs - well more like sub-folder (i.e. ~cacti/rra/3) isn't being created.
Here is some basic information;

O/S: Centos 7 (Linux version 3.10.0-514.el7.x86_64)
Cacti Version: 1.1.28
dB: MariadB 5.5.56 64-bit
poller : spine 1.1.27

Debug info is complaining about missing file/folder and the actual ~cacti/rra folder is not even showing the sub-foler where the rrd files are being looked for.
I tried to create the sub-folder manually with cacti:cacti, 775 perm and it still am not seeing those rrd files in that folder.
Any suggestions?

Best regards,

----------
RRDtool Says:

ERROR: opening '/usr/share/cacti/rra/3/15.rrd': No such file or directory


Syslog:
2018/01/16 16:00:00 - SYSTEM STATS: Time:298.4937 Method:spine Processes:1 Threads:1 Hosts:1 HostsPerProcess:1 DataSources:9 RRDsProcessed:0
2018/01/16 16:00:00 - POLLER: Poller[1] Maximum runtime of 298 seconds exceeded. Exiting.
2018/01/16 15:58:40 - CMDPHP ERROR: Unable to create directory due to missing write permissions '/usr/share/cacti/rra/4'
2018/01/16 15:58:40 - AUTOM8 Device[FHJ1taRO01-4321_3.161] Graph Added - graph-id: (17) - data-source-ids: (24)
2018/01/16 15:55:00 - DBCALL DEVEL: SQL Exec: "REPLACE INTO `cacti`.`settings` SET name='syslog_stats', value='time:0.01 deletes:0 incoming:0 removes:0 xfers:0 alerts:0 alarms:0 reports:0'"
2018/01/16 15:55:00 - SYSTEM SYSLOG STATS:Time:0.01 Deletes:0 Incoming:0 Removes:0 XFers:0 Alerts:0 Alarms:0 Reports:0

[netniV]

The RRA subfolders are automatically created by the poller.php cron job. Do you have a cron setup to run the poller?

Previously, the website created the directories if it could, but that would lead to issues where the poller was unable to manipulate the RRD files if it was run under a different user. So, until the poller first runs, the RRD and folder will not be available.

[phalek]

You could use this for your Cacti version. Change the users ( cacti -> poller, apache -> webserver) accordingly:

Code: Select all

cd /usr/share/cacti/
chown -R cacti.apache rra log resource scripts cache
chmod -R 775 rra log resource scripts cache
setfacl -d -m group:apache:rw /usr/share/cacti/rra
setfacl -d -m group:apache:rw /usr/share/cacti/log

[gundalba]

The RRA subfolders are automatically created by the poller.php cron job. Do you have a cron setup to run the poller?

Previously, the website created the directories if it could, but that would lead to issues where the poller was unable to manipulate the RRD files if it was run under a different user. So, until the poller first runs, the RRD and folder will not be available.

Thank you for your suggestion netniV,

I went ahead and checked /etc/crontab.d/ folder which contained cacti file with correct content (I think)...
*/5 * * * * cacti /usr/bin/php /usr/share/cacti/poller.php > /dev/null 2>&1

Also verified from cron log it appears to be running in every 5 min...
:
Jan 17 15:01:01 cactiserver run-parts(/etc/cron.hourly)[15786]: finished mcelog.cron
Jan 17 15:05:01 cactiserver CROND[15835]: (cacti) CMD (/usr/bin/php /usr/share/cacti/poller.php > /dev/null 2>&1)
Jan 17 15:10:01 cactiserver CROND[15899]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Jan 17 15:10:01 cactiserver CROND[15900]: (cacti) CMD (/usr/bin/php /usr/share/cacti/poller.php > /dev/null 2>&1)
Jan 17 15:15:01 cactiserver CROND[16009]: (cacti) CMD (/usr/bin/php /usr/share/cacti/poller.php > /dev/null 2>&1)
Jan 17 15:16:57 cactiserver crontab[16102]: (root) LIST (cacti)
Jan 17 15:18:35 cactiserver crontab[16143]: (root) LIST (cacti)
Jan 17 15:20:01 cactiserver CROND[16192]: (cacti) CMD (/usr/bin/php /usr/share/cacti/poller.php > /dev/null 2>&1)
Jan 17 15:20:01 cactiserver CROND[16191]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Jan 17 15:25:01 cactiserver CROND[16323]: (cacti) CMD (/usr/bin/php /usr/share/cacti/poller.php > /dev/null 2>&1)
:

But still no rrd file/folders...

[gundalba]

You could use this for your Cacti version. Change the users ( cacti -> poller, apache -> webserver) accordingly:

Code: Select all

cd /usr/share/cacti/
chown -R cacti.apache rra log resource scripts cache
chmod -R 775 rra log resource scripts cache
setfacl -d -m group:apache:rw /usr/share/cacti/rra
setfacl -d -m group:apache:rw /usr/share/cacti/log

Thanks for your suggestion phalek,

While I am somewhat hesitant on these permission changes, I gave it a try anyhow but no rrd files yet....

Regards,

[gundalba]

Getting curious here.

Could anyone let me know who the cmd.php is supposed to be run as?

[netniV]

That would depend on your system. On my ubuntu system, the website runs under the apache and www-data group, but the poller is run from root cron so the poller runs as root (this is to ensure I can use ICMP pings which requires root privs).

[gundalba]

That would depend on your system. On my ubuntu system, the website runs under the apache and www-data group, but the poller is run from root cron so the poller runs as root (this is to ensure I can use ICMP pings which requires root privs).

O/S: Centos 7 (Linux version 3.10.0-514.el7.x86_64)
Cacti Version: 1.1.28
dB: MariadB 5.5.56 64-bit
poller : spine 1.1.27

If anyone runs in the same OS/Cacti environment, could you please let me know who runs poller on its cron?
I expected Cacti install process would set proper crontab using appropriate account but that does not seem to be the case.
"crontab -l" for both cacti and root did not show poller run under them but I am seeing /etc/crontab.d/cacti file which shows
poller being launched?
*/5 * * * * cacti /usr/bin/php /usr/share/cacti/poller.php > /dev/null 2>&1
I added this line to root via "crontab -e -u root" but no difference...

None the less, could anyone tell me what are the processes the Cacti goes through when a new device/graphs are added?
I am a bit puzzled with the fact poller run as permission is being brought up here...
I am thinking something gets triggered by that "add" new device button in Cacti web UI which in turn creates these folder ~cacti/rra/"device id #"/*rrd files at some point.
I thought poller gets used to update the rrd files with collected SNMP data???

Regards,

[netniV]

That last part means that the poller is running as the cron (normally root) user. Is the path correct, does the poller reside in that path? Have you checked your cron logs?

[gundalba]

That last part means that the poller is running as the cron (normally root) user. Is the path correct, does the poller reside in that path? Have you checked your cron logs?

# locate poller.php
/usr/share/cacti/poller.php
/usr/share/cacti/lib/api_poller.php
/usr/share/cacti/lib/poller.php
#

Interesting to see two poller.php under /usr/share/cacti...
diff shows them to be different as well.
More questions than the answers it seems.

cronlog is showing the crontab is running the first one.

Do you happen to know what process Cacti goes through at initial device creation though?
My problem starts from that point as those device folders and rrd files not being created at all, thus asking.

Regards,
-- edit ---
Never mind on last question...
Having to jump between jobs, I forgot about your mention on now poller is doing the folder creation.
So question is back to why poller is failing to create the folder/files...
My last cron log shows root is running the poller...
Feb 1 14:15:01 cactiserver CROND[26775]: (root) CMD (cacti /usr/bin/php /usr/share/cacti/poller.php > /dev/null 2>&1)

[netniV]

OK, so those get created by the poller when the device is polled and it returns a result. So, the next thing would be to check that you have a device that has SNMP enabled and is set to enabled itself with a graph that has been placed on a tree (not sure the tree part is important but the graph is I think).

The reason you are seeing three different files is because the root poller is the core code that gets run fron cron. The lib/api_poller is the api code for modifying poller cache items. lib/poller ciontains the functions used by the root poller.

[gundalba]

OK, so those get created by the poller when the device is polled and it returns a result. So, the next thing would be to check that you have a device that has SNMP enabled and is set to enabled itself with a graph that has been placed on a tree (not sure the tree part is important but the graph is I think).

The reason you are seeing three different files is because the root poller is the core code that gets run fron cron. The lib/api_poller is the api code for modifying poller cache items. lib/poller ciontains the functions used by the root poller.

Thank you for the explanation on poller.php files.
I also verified the SNMP access via snmpwalk which seems to return correct values.
I tried to run the poller cron from "cacti" account but it made no difference as well...
Feb 27 11:25:02 cactiserver CROND[14167]: (cacti) CMD (cacti /usr/bin/php /usr/share/cacti/poller.php > /dev/null 2>&1)
Feb 27 11:25:02 cactiserver CROND[14168]: (pcp) CMD ( /usr/libexec/pcp/bin/pmlogger_check -C)
Feb 27 11:28:01 cactiserver CROND[14243]: (pcp) CMD ( /usr/libexec/pcp/bin/pmie_check -C)
Feb 27 11:30:01 cactiserver CROND[14290]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Feb 27 11:30:01 cactiserver CROND[14291]: (cacti) CMD (cacti /usr/bin/php /usr/share/cacti/poller.php > /dev/null 2>&1)
--- cron log ---
2018/02/27 11:18:52 - AUTOM8 Device[5] Graph Added - graph-id: (23) - data-source-ids: (33)
2018/02/27 11:18:52 - CMDPHP ERROR: Unable to create directory due to missing write permissions '/usr/share/cacti/rra/5'

I also tested the file/folder access using both cacti and apache account "sudo -u cacti mkdir /usr/share/cacti/rra/test, sudo -u apache touch /usr/share/cacti/rra/test/file" which worked fine so I am beginning to wonder if the file permission is really an issue here.
I even manually created the missing folder above (mkdir '/usr/share/cacti/rra/5'; chown cacti:apache /usr/share/cacti/rra/5;chmod 775 /usr/share/cacti/rra/5) to see if that allows the rrd file creation after poller run but no success...

Regards,

[netniV]

Do you have selinux turned on? If so, that can possibly be causing the issues. Also, as a temporary measure (don't forget to put it back) I would copy the cacti file in /etc/cron.d/ to a .bak copy, then edit the cacti file to specify a log path such as:

Code: Select all

*/5 * * * * cacti /usr/bin/php /usr/share/cacti/poller.php >> /tmp/poller.log 2>&1

Then check that log file.

Once you are done, like I said, change it back or you will find that file grows exponentially.

[gundalba]

Do you have selinux turned on? If so, that can possibly be causing the issues. Also, as a temporary measure (don't forget to put it back) I would copy the cacti file in /etc/cron.d/ to a .bak copy, then edit the cacti file to specify a log path such as:

Code: Select all

*/5 * * * * cacti /usr/bin/php /usr/share/cacti/poller.php >> /tmp/poller.log 2>&1

Then check that log file.

Once you are done, like I said, change it back or you will find that file grows exponentially.

The poller is ran from crontab of cacti user... so following mod was made on that;

*/5 * * * * /usr/bin/php /usr/share/cacti/poller.php >> /usr/share/cacti/log/tmppoller.log 2>&1

tmppoller.log is now showing the spine timing out???

2018/02/27 15:05:00 - POLLER: Poller[1] Maximum runtime of 298 seconds exceeded. Exiting.
2018/02/27 15:05:00 - SYSTEM STATS: Time:298.7209 Method:spine Processes:1 Threads:1 Hosts:1 HostsPerProcess:1 DataSources:17 RRDsProcessed:0
2018/02/27 15:10:00 - POLLER: Poller[1] Maximum runtime of 298 seconds exceeded. Exiting.
2018/02/27 15:10:00 - SYSTEM STATS: Time:298.7079 Method:spine Processes:1 Threads:1 Hosts:1 HostsPerProcess:1 DataSources:17 RRDsProcessed:0
2018/02/27 15:15:00 - POLLER: Poller[1] Maximum runtime of 298 seconds exceeded. Exiting.
2018/02/27 15:15:00 - SYSTEM STATS: Time:298.7115 Method:spine Processes:1 Threads:1 Hosts:1 HostsPerProcess:1 DataSources:17 RRDsProcessed:0
2018/02/27 15:20:00 - POLLER: Poller[1] Maximum runtime of 298 seconds exceeded. Exiting.

I added one device for a test but poller seem to be timing out waiting on something....

Regards,

[netniV]

Is your spine configuration correct? Can you attempt to run it manually and see what it outputs?