why spine not showing graphs?

[Achea9]

Hello,

Is it possible somebody support me with this issue, please....

No matter what I try I cannot yet this spine issue. I see either an empty Graph with NAN or just the cactus with no rrd File found.

Spine Parameters:

Deafault Threads pr Process: 10
Number of PHP Script Servers: 5
Script server timeout Value: 25
Max SNMP OIDs: 10


This is the spine permission:
-rwsr-sr-x 1 root root 463192 Oct 18 16:20 /usr/local/spine/bin/spine

As I said I use SUSE SLES SP15.1 and my cron is set up to:


*/5 * * * * wwwrun /usr/bin/php /srv/www/cacti/poller.php > /dev/null 2>&1

because a cacti username is not accepted in suse linux. meaning that if I do chown cactiuser:root to my rra Folder it says this user name is not recognized. But wwwrun is doing well.

Permissions to rra and my log Folders.
drwxr-xr-x 2 wwwrun wwwrun 77824 Oct 24 20:04 rra
lrwxrwxrwx 1 wwwrun www 14 Sep 24 19:21 log -> /var/log/cacti

I restarted spine manually, rerun poller.php and my log level is DEBUG but not showing errors and even reboot my server. Nothing solved it, yet.

Can sombody help? Thanks in Advance.

[TheWitness]

Your poller runs as 'wwwrun', so that's who should own the RRDfiles. You did it backwards

cd $CACTI_TOP
chown -R wwwrun:root log rra

[Achea9]

Hi,

Thank You the Witness.

I followed your advice and changed:
shell sudo chown -R wwwrun:root /srv/www/cacti/rra /srv/www/cacti/log

check:
shell> ls -l
drwxr-xr-x 2 wwwrun root 81920 Oct 26 13:46 rra
lrwxrwxrwx 1 wwwrun root 14 Sep 24 19:21 log -> /var/log/cacti

my crontab:
*/5 * * * * wwwrun /usr/bin/php /srv/www/cacti/poller.php >
/dev/null


restarted mysql, apache2 and run the poller manually
shell>sudo php -q /srv/www/cacti/poller.php –force

here the last part of the result:

2020-10-26 13:49:27 - SYSTEM STATS: Time:299.5676 Method:spine Processes:1 Threads:1 Hosts:307 HostsPerProcess:307 DataSources:3386 RRDsProcessed:1301
2020-10-26 13:49:28 - POLLER: Poller[1] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: -q /srv/www/cacti/poller_reports.php]
2020-10-26 13:49:28 - POLLER: Poller[1] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: -q '/srv/www/cacti/poller_spikekill.php']
2020-10-26 13:49:28 - POLLER: Poller[1] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: -q '/srv/www/cacti/poller_automation.php' -M]
2020-10-26 13:49:28 - POLLER: Poller[1] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: -q '/srv/www/cacti/poller_maintenance.php']
hailemn@mucs70320:~>


Running the spine manually:

shell>sudo ./spine -R -V 3 -S

SPINE: Using spine config file [../etc/spine.conf]
WARNING: Unrecongized directive: Database=cacti in ../etc/spine.conf
SPINE: Version 1.2.14 starting
NOTE: Spine will support multithread device polling.
…..
…..


Device[78] HT[1] DS[720] SNMP: v2: 10.192.16.30, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.2, value: 3773217141463
Device[78] HT[1] Total Time: 1 Seconds
POLLER: Active Threads is 0, Pending is 0
SPINE: The Final Value of Threads is 0
Time: 300.0623 s, Threads: 1, Devices: 308

I think the warning is due to DB_Pass and DB_Password between cact.conf and spine.conf - ignorable. All the rest is running very well and looks good.

But still no Graph to see:-(

DEBUG Log output:


Log [Total Lines: 15858 - Admin view - Unfiltered]



2020-10-26 14:08:22 - WEBLOG CACTI2RRD: /usr/bin/rrdtool graph - --imgformat=PNG --start='1603631301' --end='1603717701' --pango-markup --title='aldnfw03 - CPU Usage' --vertical-label='percent' --slope-mode --base=1000 --height=120 --width=645 --rigid --alt-autoscale-max --lower-limit='0' COMMENT:"From 2020-10-25 14\:08\:21 To 2020-10-26 14\:08\:21\c" COMMENT:" \n" --color BACK#F3F3F3 --color CANVAS#FDFDFD --color SHADEA#CBCBCB --color SHADEB#999999 --color FONT#000000 --color AXIS#2C4D43 --color ARROW#2C4D43 --color FRAME#2C4D43 --border 1 --font TITLE:11:'Arial' --font AXIS:8:'Arial' --font LEGEND:8:'Courier' --font UNIT:8:'Arial' --font WATERMARK:6:'Arial' --slope-mode --watermark 'Generated by Cacti®' DEF:a='/srv/www/cacti/rra/xxx_proc_system_3342.rrd' DS[xxx - Processor System] Graphs[aldnfw03 - CPU Usage]:'proc_system':AVERAGE DEF:b='/srv/www/cacti/rra/xxx_proc_user_3343.rrd' DS[aldnfw03 - Processor User] Graphs[xxx - CPU Usage]:'proc_user':AVERAGE CDEF:cdefi='TIME,1603717402,GT,a,a,UN,0,a,IF,IF,TIME,1603717402,GT,b,b,UN,0,b,IF,IF,+' AREA:a#FF0000FF:'System' GPRINTLAST:'Current\:%8.2lf %s' GPRINTAVERAGE:'Average\:%8.2lf %s' GPRINTMAX:'Maximum\:%8.2lf %s\n' LINE1.00:b#0000FFFF:'User ':STACK GPRINTLAST:'Current\:%8.2lf %s' GPRINTAVERAGE:'Average\:%8.2lf %s' GPRINTMAX:'Maximum\:%8.2lf %s\n' LINE1:cdefi#000000FF:'Total ' GPRINT:cdefi:LAST:'Current\:%8.2lf %s' GPRINT:cdefi:AVERAGE:'Average\:%8.2lf %s' GPRINT:cdefi:MAX:'Maximum\:%8.2lf %s'

2020-10-26 14:08:22 - WEBLOG CACTI2RRD: /usr/bin/rrdtool graph - --imgformat=PNG --start='1603631301' --end='1603717701' --pango-markup --title='aldnfw03 - Dropped' --vertical-label='pkts dropped/min' --slope-mode --base=1000 --height=120 --width=645 --alt-autoscale-max --lower-limit='0' COMMENT:"From 2020-10-25 14\:08\:21 To 2020-10-26 14\:08\:21\c" COMMENT:" \n" --color BACK#F3F3F3 --color CANVAS#FDFDFD --color SHADEA#CBCBCB --color SHADEB#999999 --color FONT#000000 --color AXIS#2C4D43 --color ARROW#2C4D43 --color FRAME#2C4D43 --border 1 --font TITLE:11:'Arial' --font AXIS:8:'Arial' --font LEGEND:8:'Courier' --font UNIT:8:'Arial' --font WATERMARK:6:'Arial' --slope-mode --watermark 'Generated by Cacti®' DEF:a='/srv/www/cacti/rra/aldnfw03_dropped_3350.rrd' DS[aldnfw03 - Pkt dropped] Graphs[aldnfw03 - Dropped]:'dropped':AVERAGE CDEF:cdefa='a,300,*' AREA:cdefa#FF0000FF:'Dropped' GPRINT:cdefa:LAST:'Current\:%8.2lf %s' GPRINT:cdefa:AVERAGE:'Average\:%8.2lf %s' GPRINT:cdefa:MAX:'Max\:%8.2lf %s' LINE1:cdefa#000000FF:'Max \n'

Nothing special I would say.

The Question is why the poller doesn't run unless I force it to run?
Does this mean spine also not running? And why?

What else can I do to make it work?

Thanks a Lot and Best Regards

[Achea9]

Hi,

I did a bit more...

I have seen when I use the cactiuser instead of wwwrun in crontab and change the rra/ log/ owneres to cactiuser:root, i only see cactus telling me
"Website doesn't have write access to folder, maybe unable to create /update rrds: /rra" message, which is very sad.

So I am forced to remain with my wwwrun though it is not my cactiuser name, but it seems working. I made to the owner of the rra and log Folders and also my crontab runs wit it.

lrwxrwxrwx 1 wwwrun root 14 Sep 24 19:21 log -> /var/log/cacti
drwxr-xr-x 2 wwwrun root 77824 Oct 26 19:39 rra

*/5 * * * * wwwrun php /srv/www/cacti/poller.php > /dev/null 2>&1

Now I let my spine run using wwwrun. shell>sudo -u wwwrun ./spine -R -V 3 -S
All went good till the End
but only

POLLER: Active Threads is 0, Pending is 0
SPINE: The Final Value of Threads is 0
FATAL: Spine Encountered a Segmentation Fault [110, Connection timed out] (Spine parent)

There is no more poller running in my server.

I am a bit Lucky today to see a small part of Graphs are comming up after I deleted all rrds and let it do it again.
But that is only a very small part of my devices.

The very strange thing is, that all the devices with the new Graphs are not n Europe nor in Germany where I am resided, but for example Mexico Brazil, USA and NONE in EUROPE.

what else am I missing? I know there is somthing missing I feel that but what?


I appreciate your support, Thanks.

[Achea9]

Hi,

I played a little more with my crontab, rra/ and log/ File permissions and switched shortly to cmd.php so that spine stops creating new rrd files and deleted the whole rrd files from /rra Folder.

Now I know, that in my case wwwrun must have write permission for /rra and /log Folders, the cactiuser won't be accepted, since the Error:

Website doesn't have write permission to folder, may be unable to create/update RRDs: /rra/

cactiuser-issue.pdf

(65.72 KiB) Downloaded 60 times

That means for the website to have write access on my rra and log Folders, it is a must that I make sure and give it to wwwrun:

lrwxrwxrwx 1 wwwrun root 14 Sep 24 19:21 log -> /var/log/cacti
drwxr-xr-x 2 wwwrun root 77824 Oct 26 19:39 rra

At the same time, I believe cacti needs its user in crontab too.

Therefore, I decided to have bothe the cactiuser and wwwrun entered in crontab:

*/5 * * * * cactiuser php /srv/www/cacti/poller.php > /dev/null 2>&1

*/5 * * * * wwwrun php /srv/www/cacti/poller.php > /dev/null 2>&1

After that, I restarted apache2 and mysql for the effect to take place. In cacti I changed the poller back to spine and let both users run spine manually .

wwwrun user running spine:

Device[83] INFO: SNMP Device '101.192.168.173' has timeout 500000 (500), retries 3
NOTE: Device[83] Checking for System Information Update
NOTE: Device[83] Updating Full System Information Table
ERROR: Spine Timed Out While Processing Devices Internal
Device[83] HT[1] DQ[4] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 1267849990 < output: 1267864079)
Device[83] HT[1] NOTE: There are '5' Polling Items for this Device
Device[83] INFO: SNMP Device '101.192.168.173' has timeout 500000 (500), retries 3
Device[83] HT[1] DS[754] SS[3] SERVER: /srv/www/cacti/scripts/ss_hstats.php ss_hstats '83' polling_time, output: 2.231
Device[83] HT[1] DS[755] SS[4] SERVER: /srv/www/cacti/scripts/ss_hstats.php ss_hstats '83' uptime, output: 1267849941
Device[83] HT[1] DS[753] SNMP: v2: 101.192.168.173, dsname: 5min_cpu, oid: .1.3.6.1.4.1.9.2.1.58.0, value: 1
Device[83] HT[1] DS[756] SNMP: v2: 101.192.168.173, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1, value: 8993478273833
Device[83] HT[1] DS[756] SNMP: v2: 101.192.168.173, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1, value: 6020923958857
Device[83] HT[1] Total Time: 1.1 Seconds
POLLER: Active Threads is 0, Pending is 0
SPINE: The Final Value of Threads is 0
FATAL: Spine Encountered a Segmentation Fault [110, Connection timed out] (Spine parent)

####################################################################################################
####################################################################################################

cactiuser runs spine:

Device[33] INFO: SNMP Device '10.192.17.77' has timeout 500000 (500), retries 3
NOTE: Device[33] Checking for System Information Update
NOTE: Device[33] Updating Full System Information Table
ERROR: Spine Timed Out While Processing Devices Internal
Device[33] HT[1] DQ[4] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 1365026430 < output: 1365107082)
Device[33] HT[1] NOTE: There are '3' Polling Items for this Device
Device[33] INFO: SNMP Device 101.192.168.77' has timeout 500000 (500), retries 3
Device[33] HT[1] DS[308] SS[3] SERVER: /srv/www/cacti/scripts/ss_hstats.php ss_hstats '33' polling_time, output: 2.089
Device[33] HT[1] DS[309] SS[4] SERVER: /srv/www/cacti/scripts/ss_hstats.php ss_hstats '33' uptime, output: 1365026384
Device[33] HT[1] DS[307] SNMP: v2: 101.192.168.77, dsname: 5min_cpu, oid: .1.3.6.1.4.1.9.2.1.58.0, value: 1
Device[33] HT[1] Total Time: 1 Seconds
POLLER: Active Threads is 0, Pending is 0
SPINE: The Final Value of Threads is 0
FATAL: Spine Encountered a Segmentation Fault [110, Connection timed out] (Spine parent)

As you can see, in both cases we see 110 connection time outs

#############################################################################################################
#############################################################################################################

I could see rrd files are created again, even some new ones, but the ISSUE still sticks stubborn there, that I don't know what else i can do.

As you see, I am stuck to this for too long time, I really need support please.

Thanks and Regards

[Achea9]

Hi,

No one out there to support me? It is really very urgent!!! PLEASE somebody:-)

Any one with SUSE Linux, maybe it is SUSE special case???

Ragards

[pahlevanzadeh]

If your distro is redhat base such as centos,fedora, rhel or so on, user of apache is httpd, and if you use a debian base , your apache2 user is www-data