Script Query works on CLI but not in Cacti

[Lalbee99]

I am attempting to add a new Data Query to my cacti environment. I have written a script to get the percentage of inodes used on a filesystem. I have tested the script via the cacti account on the command line (see below). all works fine.

[cacti@XXXXXXX ~]$ perl /cacti/cacti/scripts/chb_rhel_inodes.pl 'XXXXXXXX.XXXXX.XXX' 'index'
root
tmp
home
var
optopenv

[cacti@XXXXXXX ~]$ perl /cacti/cacti/scripts/chb_rhel_inodes.pl 'XXXXXXXX.XXXXX.XXX' 'num_indexes'
5

[cacti@XXXXXXX ~]$ perl /cacti/cacti/scripts/chb_rhel_inodes.pl 'XXXXXXXX.XXXXX.XXX' 'get' var
2

[cacti@XXXXXXX ~]$ perl /cacti/cacti/scripts/chb_rhel_inodes.pl 'XXXXXXXX.XXXXXX.XXX' 'query'
root|2
tmp|1
home|1
var|2
optopenv|1

When the associated dated query is run via verbose query in the GUI it claims to find no indexes when clearly there are 5. I have checked permissions, as well as Selinux. I am completely puzzled by the inconsistency between the GUI and command line

VERBOSE QUERY OUTPUT
Total: 0.000000, Delta: 0.000000, Found data query XML file at '/cacti/cacti/resource/script_queries/chb_rhel_inode.xml'
Total: 0.000000, Delta: 0.000000, Running Data Query [127].
Total: 0.000000, Delta: 0.000000, Found Type = '4' [Script Query].
Total: 0.000000, Delta: 0.000000, XML file parsed ok.
Total: 0.040000, Delta: 0.040000, Executing script for num of indexes 'perl /cacti/cacti/scripts/chb_rhel_inodes.pl 'xxxxxxxxx.xxxxxxx.xxx' 'num_indexes''
Total: 0.040000, Delta: 0.000000, Found number of indexes: 0
Total: 0.070000, Delta: 0.030000, ERROR: Data Query returned no indexes.

XML FILE
<query>
<name>Get rhel memory info</name>
<description>Queries Linux servers for memory stats</description>
<script_path>perl |path_cacti|/scripts/chb_rhel_inodes.pl</script_path>
<arg_prepend>|host_hostname|</arg_prepend>
<arg_index>index</arg_index>
<arg_query>query</arg_query>
<arg_get>get</arg_get>
<arg_num_indexes>num_indexes</arg_num_indexes>
<output_delimeter>|</output_delimeter>
<index_order>fsName</index_order>
<index_order_type>alphabetic</index_order_type>
<index_title_format>|chosen_order_field|</index_title_format>
<fields>
<fsName>
<name>Filesystem</name>
<direction>input</direction>
<query_name>name</query_name>
</fsName>
<ChbRhelInodesUsed>
<name>Used Inodes Percentage</name>
<direction>output</direction>
<query_name>ChbRhelInodesUsed</query_name>
</ChbRhelInodesUsed>

</fields>
</query>


SCRIPT
#!/usr/bin/perl
######################
chomp($HOST=@ARGV[0]);
######################
#
delete @ENV{qw(PATH)};
$ENV{PATH} = '/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/sbin';

chomp($val1 = $ARGV[0]);
chomp($val2 = $ARGV[1]);
chomp($val3 = $ARGV[2]);


if (($val2 ne "query") && ($val2 ne "get") && ($val2 ne "index") && ($val2 ne "num_indexes")) {
print "usage:\n\n";
print "./chb_rhel_inodes.pl index\n";
print "./chb_rhel_inodes.pl num_indexes\n";
print "./chb_rhel_inodes.pl query {name,ChbRhelInodesUsed}\n";
print "./chb_rhel_inodes.pl get {name,ChbRhelInodesUsed}\n";
exit;
}

my $num_count=0;
open(INTER, "ssh -q ${HOST} df -i | /bin/sed -n 2,100p | sed 's/%//g' | egrep -v \"SWAP|boot|\/run\/user\" | awk '{ print \$6\",\"\$5 }' |");
while (<INTER>) {
($name,$inodepercent) = split(/,/);
chomp($name);
chomp($inodepercent);
#chomp($name = `echo $name_tmp | sed 's#\\/##g'`);
#if ($name_tmp eq "\/") {
# chomp($name="root");
#}
my %output = (
name => $name,
inodepercent => $inodepercent
);
if ($val2 eq "index") {
print "$name\n";
}elsif ($val2 eq "num_indexes") {
$num_count++;
}elsif (($val2 eq "get") && ($val3 eq $name)) {
print "$output{inodepercent}\n";
}elsif ($val2 eq "query") {
print "$output{name}|$output{name}\n";
}
}
if (${val2} eq "num_indexes") {
print "$num_count\n";
}
close(INTER);

[Lalbee99]

Apparently this issue has been around since 2009 unresolved viewtopic.php?t=33717 (shaking head ... frustrating......)


If you run a reindex via the CLI tool on the host it magically works.
php -q /cacti/cacti/cli/poller_reindex_hosts.php -id=XXX -d

[Lalbee99]

<BUMP>

[TheWitness]

I would suspect that it's a permission problem. Edit the /etc/passwd file to give apache a shell, and then 'su - apache' and then run the script by hand.

[Lalbee99]

I don't think is a permission issue. We have other custom templates that run script queries as the cacti user with no issues. I also do not think it is ever a good idea to give apache a shell. We have always had apache set /sbin/nologin. I also don't know why when you perform a re-index via the command line items magically appear in the GUI.

[TheWitness]

What does your Cacti standard error log look like? Also, inside the script, do you report all the data in one print/echo or multiple? If you are using spine, you must output all data in one print/echo. The cmd.php collector is much more resilient, but also much slower. Lastly, there is a script timeout setting in Cacti. If you are using spine, if the script does not respond in time, you will never get any information.

[Lalbee99]

What does your Cacti standard error log look like? When I run a verbose query from the GUI and then check the standard error log there are no errors logged.

Also, inside the script, do you report all the data in one print/echo or multiple? It depends on what arg is being passed. If Index is passed it will show all mount points 1 per line. If num_indexes is passed then a total of the number of indexes is printed on a single line. If get is passed it will output the value requested for a single mount point on 1 line.

If you are using spine, you must output all data in one print/echo. It does.

there is a script timeout setting in Cacti. If you are using spine, if the script does not respond in time, you will never get any information. My timeout value is set to 45 seconds. On the command line the script returns in 1 second.

[TheWitness]

cd /var/www/html/cacti/log
ls -l *stderr*

[Lalbee99]

The stderr log's are filled with ssh banner page from ssh

# ls -l *stderr*
-rw-rw-r--. 1 cacti apache 9103829 Nov 21 15:55 cacti_stderr.log
-rw-rw-r--. 1 cacti apache 189341048 Oct 19 2019 cacti_stderr.log.1
-rw-rw-r--. 1 cacti apache 7485760 Oct 19 2019 cacti_stderr.log.2
-rw-rw-r--. 1 cacti apache 57809297 Oct 19 2019 cacti_stderr.log.3

# tail -100 cacti_stderr.log | grep -v Pseudo-terminal
may be monitored. Use of this system constitutes consent to such monitoring.
Unauthorized use may be subject to criminal prosecution. There is no
expectation of privacy while using this system.

Killed by signal 1.
This system is to be used only by authorized users and all use of this system
may be monitored. Use of this system constitutes consent to such monitoring.
Unauthorized use may be subject to criminal prosecution. There is no
expectation of privacy while using this system.

This system is to be used only by authorized users and all use of this system
may be monitored. Use of this system constitutes consent to such monitoring.
Unauthorized use may be subject to criminal prosecution. There is no
expectation of privacy while using this system.

Killed by signal 1.
Killed by signal 1.
Killed by signal 1.
Killed by signal 1.
Killed by signal 1.
Killed by signal 1.
This system is to be used only by authorized users and all use of this system
may be monitored. Use of this system constitutes consent to such monitoring.
Unauthorized use may be subject to criminal prosecution. There is no
expectation of privacy while using this system.

*******************************************************************************
* This system is to be used only by authorized users, and all use of this *
* system may be monitored. Use of this system constitutes consent to such *
* monitoring. Unauthorized use may be subject to criminal prosecution. There *
* is no expectation of privacy while using this system. *
*******************************************************************************
This system is to be used only by authorized users and all use of this system
may be monitored. Use of this system constitutes consent to such monitoring.
Unauthorized use may be subject to criminal prosecution. There is no
expectation of privacy while using this system.

This system is to be used only by authorized users and all use of this system
may be monitored. Use of this system constitutes consent to such monitoring.
Unauthorized use may be subject to criminal prosecution. There is no
expectation of privacy while using this system.

This system is to be used only by authorized users and all use of this system
may be monitored. Use of this system constitutes consent to such monitoring.
Unauthorized use may be subject to criminal prosecution. There is no
expectation of privacy while using this system.

*******************************************************************************
* This system is to be used only by authorized users, and all use of this *
* system may be monitored. Use of this system constitutes consent to such *
* monitoring. Unauthorized use may be subject to criminal prosecution. There *
* is no expectation of privacy while using this system. *
*******************************************************************************
This system is to be used only by authorized users and all use of this system
may be monitored. Use of this system constitutes consent to such monitoring.
Unauthorized use may be subject to criminal prosecution. There is no
expectation of privacy while using this system.

This system is to be used only by authorized users and all use of this system
may be monitored. Use of this system constitutes consent to such monitoring.
Unauthorized use may be subject to criminal prosecution. There is no
expectation of privacy while using this system.

This system is to be used only by authorized users and all use of this system
may be monitored. Use of this system constitutes consent to such monitoring.
Unauthorized use may be subject to criminal prosecution. There is no
expectation of privacy while using this system.

This system is to be used only by authorized users and all use of this system
may be monitored. Use of this system constitutes consent to such monitoring.
Unauthorized use may be subject to criminal prosecution. There is no
expectation of privacy while using this system.

This system is to be used only by authorized users and all use of this system
may be monitored. Use of this system constitutes consent to such monitoring.
Unauthorized use may be subject to criminal prosecution. There is no
expectation of privacy while using this system.

This system is to be used only by authorized users and all use of this system
may be monitored. Use of this system constitutes consent to such monitoring.
Unauthorized use may be subject to criminal prosecution. There is no
expectation of privacy while using this system.

[Osiris]

You have to turn off the banner, also check for selinux.

[Lalbee99]

There is absolutely no conflicts with selinux related to the script queury. Sorry but our security department will not allow the turning off of the ssh banners.

ausearch -m avc -ts today | audit2why
type=AVC msg=audit(1669106341.542:10799377): avc: denied { getattr } for pid=10730 comm="logrotate" path="/cacti/cacti/log/boost.log" dev="dm-6" ino=1054980 scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:httpd_user_content_t:s0 tclass=file permissive=0

Was caused by:
Missing type enforcement (TE) allow rule.

You can use audit2allow to generate a loadable module to allow this access.

type=AVC msg=audit(1669106341.542:10799378): avc: denied { getattr } for pid=10730 comm="logrotate" path="/cacti/cacti/log/cacti.log" dev="dm-6" ino=1054976 scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:httpd_user_content_t:s0 tclass=file permissive=0

Was caused by:
Missing type enforcement (TE) allow rule.

You can use audit2allow to generate a loadable module to allow this access.

type=AVC msg=audit(1669106341.544:10799379): avc: denied { getattr } for pid=10730 comm="logrotate" path="/cacti/cacti/log/boost.log" dev="dm-6" ino=1054980 scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:httpd_user_content_t:s0 tclass=file permissive=0

Was caused by:
Missing type enforcement (TE) allow rule.

You can use audit2allow to generate a loadable module to allow this access.

type=AVC msg=audit(1669106341.544:10799380): avc: denied { getattr } for pid=10730 comm="logrotate" path="/cacti/cacti/log/cacti.log" dev="dm-6" ino=1054976 scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:httpd_user_content_t:s0 tclass=file permissive=0

Was caused by:
Missing type enforcement (TE) allow rule.

You can use audit2allow to generate a loadable module to allow this access.

[TheWitness]

That's what I was going to suggest. Since there are so many variations of Cacti install, we can not publish a rule set. You just have to set to permissive and then do audit2allow when you get exceptions. Turn off the banner for the Apache account though...

[TheWitness]

Having a good understanding of Cacti helps.

[Lalbee99]

First off I displayed the ausearch to show there was no selinux conflict with the query. Yes I have to put a rule in place for logrotate but I haven't gotten around to it. I was not asking you for a ruleset for this.
Second I have templates that use script queries with ssh through the same cacti account and they are working fine. That is why I posted in the forum.
I do have an understanding of cacti. I have been running it since 0.8.8c. I have automated my DR cutover as well as the upgrade process. So please don't be condescending when people are asking for help on the forums.

[TheWitness]

I wasn't being condescending. Oscar Wilde said it best centuries ago, I'll paraphrase first: "It's not what I said, it's what you heard.", now the quote: "The dirty word is in the dirty mind". Different subject mater, which goes to the whole woke mentality these days (and no I'm not talking about you cause I don't know you ... or maybe I do). Anyway, be well.