Graphs Stop With Upgrade / No New Graphs

Post support questions that directly relate to Linux/Unix operating systems.

Moderators: Developers, Moderators

Post Reply
Oxygen
Posts: 23
Joined: Fri Jun 10, 2005 1:29 pm
Location: Long Island, NY, USA

Graphs Stop With Upgrade / No New Graphs

Post by Oxygen »

I just upgraded tonight from version C to version G of Cacti. After the upgrade, all of my graphs stopped working and now when I create new graphs, they just come up as Red X's on the page.

When I got to debug any of the new graphs, I get this error:

RRDTool Says:

ERROR: opening '/usr/share/cacti/rra/wsapp_srv_min_189.rrd': No such file or directory


The log file shows the following:

11/24/2005 04:55:27 AM - SYSTEM STATS: Time: 26.4039 s, Method: cmd.php, Processes: 1, Threads: N/A, Hosts: 13, Hosts/Process: 13
11/24/2005 04:55:04 AM - CMDPHP: Poller[0] Host[3] WARNING: Result from CMD not valid. Partial Result:
11/24/2005 04:55:02 AM - CMDPHP: Poller[0] Host[1] WARNING: Result from CMD not valid. Partial Result:
11/24/2005 04:55:02 AM - CMDPHP: Poller[0] Host[1] WARNING: Result from CMD not valid. Partial Result:
11/24/2005 04:55:02 AM - CMDPHP: Poller[0] Host[1] WARNING: Result from CMD not valid. Partial Result:
11/24/2005 04:55:02 AM - CMDPHP: Poller[0] Host[1] WARNING: Result from CMD not valid. Partial Result:
11/24/2005 04:55:02 AM - CMDPHP: Poller[0] Host[1] WARNING: Result from CMD not valid. Partial Result:
11/24/2005 04:55:02 AM - CMDPHP: Poller[0] Host[1] WARNING: Result from CMD not valid. Partial Result:
11/24/2005 04:55:02 AM - CMDPHP: Poller[0] Host[1] WARNING: Result from CMD not valid. Partial Result:


Your help is appreciated! :D

Happy Thanksgiving!!!
User avatar
rony
Developer/Forum Admin
Posts: 6022
Joined: Mon Nov 17, 2003 6:35 pm
Location: Michigan, USA
Contact:

Post by rony »

Did you copy over you rra files from the old installation?
[size=117][i][b]Tony Roman[/b][/i][/size]
[size=84][i]Experience is what causes a person to make new mistakes instead of old ones.[/i][/size]
[size=84][i]There are only 3 way to complete a project: Good, Fast or Cheap, pick two.[/i][/size]
[size=84][i]With age comes wisdom, what you choose to do with it determines whether or not you are wise.[/i][/size]
Oxygen
Posts: 23
Joined: Fri Jun 10, 2005 1:29 pm
Location: Long Island, NY, USA

Post by Oxygen »

Yes, the graphs that were there from the previous installation are still there, they are just not graphing any more.

When I create a NEW graph, the error above is the one that I get.
Oxygen
Posts: 23
Joined: Fri Jun 10, 2005 1:29 pm
Location: Long Island, NY, USA

Post by Oxygen »

Here is a screen shot.

The graphs above were there from version C.

The graphs below are the new graphs I created after the upgrade to G.
Attachments
Screen Shot
Screen Shot
cacti.JPG (17.34 KiB) Viewed 3168 times
spiny
Posts: 6
Joined: Tue Jun 07, 2005 1:30 am

Scripts not executeable in cacti-0.8.6g

Post by spiny »

It seems the scripts are not executeable under the cacti-0.8.6g source distribution:
  • [root@nms1 root]# tar xzvf cacti-0.8.6g.tar.gz
    cacti-0.8.6g/

    <-snip!->

    [root@nms1 root]# cd cacti-0.8.6g/scripts
    [root@nms1 scripts]# ls -l
    total 84
    -rw------- 1 1000 users 149 Sep 9 06:56 3com_cable_modem.pl
    -rw------- 1 1000 users 201 Sep 9 06:56 diskfree.pl
    -rw------- 1 1000 users 92 Sep 9 06:56 diskfree.sh
    -rw------- 1 1000 users 119 Sep 9 06:56 linux_memory.pl
    -rw------- 1 1000 users 236 Sep 9 06:56 loadavg_multi.pl
    -rw------- 1 1000 users 355 Sep 9 06:56 loadavg.pl
    -rw------- 1 1000 users 115 Sep 9 06:56 ping.pl
    -rw-r--r-- 1 1000 users 2129 Sep 9 06:56 query_host_cpu.php
    -rw-r--r-- 1 1000 users 2029 Sep 9 06:56 query_host_partitions.php
    -rw------- 1 1000 users 840 Sep 9 06:56 query_unix_partitions.pl
    -rw-r--r-- 1 1000 users 356 Sep 9 06:56 sql.php
    -rw-r--r-- 1 1000 users 1750 Sep 9 06:56 ss_fping.php
    -rw-r--r-- 1 1000 users 4243 Sep 9 06:56 ss_host_cpu.php
    -rw-r--r-- 1 1000 users 3353 Sep 9 06:56 ss_host_disk.php
    -rw-r--r-- 1 1000 users 951 Sep 9 06:56 ss_sql.php
    -rw------- 1 1000 users 79 Sep 9 06:56 unix_processes.pl
    -rw------- 1 1000 users 232 Sep 9 06:56 unix_tcp_connections.pl
    -rw------- 1 1000 users 228 Sep 9 06:56 unix_users.pl
    -rw------- 1 1000 users 850 Sep 9 06:56 weatherbug.pl
    -rw------- 1 1000 users 188 Sep 9 06:56 webhits.pl
    [root@nms1 scripts]#

I recommend the following solution:
  • [root@nms1 scripts]# chmod 755 *
    [root@nms1 scripts]# ls -l
    total 84
    -rwxr-xr-x 1 1000 users 149 Sep 9 06:56 3com_cable_modem.pl
    -rwxr-xr-x 1 1000 users 201 Sep 9 06:56 diskfree.pl
    -rwxr-xr-x 1 1000 users 92 Sep 9 06:56 diskfree.sh
    -rwxr-xr-x 1 1000 users 119 Sep 9 06:56 linux_memory.pl
    -rwxr-xr-x 1 1000 users 236 Sep 9 06:56 loadavg_multi.pl
    -rwxr-xr-x 1 1000 users 355 Sep 9 06:56 loadavg.pl
    -rwxr-xr-x 1 1000 users 115 Sep 9 06:56 ping.pl
    -rwxr-xr-x 1 1000 users 2129 Sep 9 06:56 query_host_cpu.php
    -rwxr-xr-x 1 1000 users 2029 Sep 9 06:56 query_host_partitions.php
    -rwxr-xr-x 1 1000 users 840 Sep 9 06:56 query_unix_partitions.pl
    -rwxr-xr-x 1 1000 users 356 Sep 9 06:56 sql.php
    -rwxr-xr-x 1 1000 users 1750 Sep 9 06:56 ss_fping.php
    -rwxr-xr-x 1 1000 users 4243 Sep 9 06:56 ss_host_cpu.php
    -rwxr-xr-x 1 1000 users 3353 Sep 9 06:56 ss_host_disk.php
    -rwxr-xr-x 1 1000 users 951 Sep 9 06:56 ss_sql.php
    -rwxr-xr-x 1 1000 users 79 Sep 9 06:56 unix_processes.pl
    -rwxr-xr-x 1 1000 users 232 Sep 9 06:56 unix_tcp_connections.pl
    -rwxr-xr-x 1 1000 users 228 Sep 9 06:56 unix_users.pl
    -rwxr-xr-x 1 1000 users 850 Sep 9 06:56 weatherbug.pl
    -rwxr-xr-x 1 1000 users 188 Sep 9 06:56 webhits.pl
    [root@nms1 scripts]
.........Now that's better. :wink:

chmod 700 may be more secure....
User avatar
rony
Developer/Forum Admin
Posts: 6022
Joined: Mon Nov 17, 2003 6:35 pm
Location: Michigan, USA
Contact:

Post by rony »

That shouldn't matter, the scripts are always called with there interperter, IE perl or php.

If the permission thing helped, then there might be a path issue under php.
[size=117][i][b]Tony Roman[/b][/i][/size]
[size=84][i]Experience is what causes a person to make new mistakes instead of old ones.[/i][/size]
[size=84][i]There are only 3 way to complete a project: Good, Fast or Cheap, pick two.[/i][/size]
[size=84][i]With age comes wisdom, what you choose to do with it determines whether or not you are wise.[/i][/size]
Oxygen
Posts: 23
Joined: Fri Jun 10, 2005 1:29 pm
Location: Long Island, NY, USA

Post by Oxygen »

I have tried that, still no go. Getting errors like:

ERROR: opening '/usr/share/cacti/rra/wsapp_srv_min_189.rrd': No such file or directory



Cacti is running as ROOT.
Oxygen
Posts: 23
Joined: Fri Jun 10, 2005 1:29 pm
Location: Long Island, NY, USA

Post by Oxygen »

Okay, I was reading the posts located here: http://forums.cacti.net/viewtopic.php?t=8583

So I downloaded all the files from here: http://www.cacti.net/downloads/patches/ ... e-patched/

Now I am not getting any more RRD errors, and the graphs do show up, but they are all blank! The existing and the new ones, they all show up with "nan" for values and do not graph anything.

It looks like the poller is running fine though:

Code: Select all

11/26/2005 03:20:26 AM - SYSTEM STATS: Time: 24.3706 s, Method: cmd.php, Processes: 1, Threads: N/A, Hosts: 13, Hosts/Process: 13  
11/26/2005 03:20:25 AM - CMDPHP: Poller[0] Time: 23.5095 s, Theads: N/A, Hosts: 12  
But the data is not being written to the RRD files... :(

Please help. I love Cacti and it makes me upset when it does not work.
User avatar
gandalf
Developer
Posts: 22383
Joined: Thu Dec 02, 2004 2:46 am
Location: Muenster, Germany
Contact:

Post by gandalf »

Please have a look at the device list. Host in status down won't be polled. Have a look at the Downed Host Detection to change this.
If polling works, run one polling cycle with logging set to DEBUG. Look at cacti.log for failing scripts.
If this works, run the poller from command line as cactiuser or change your crontab to

Code: Select all

*/5 * * * *     cactiuser       /usr/bin/php /var/www/html/cacti/poller.php > /var/www/html/cacti/log/poller.log 2>&1
and have the output in poller.log. Look for rrdtool update errors.
Reinhard
Attachments
downed host detection
downed host detection
downed-host-detection.png (35.04 KiB) Viewed 3084 times
device list
device list
device-list.png (29.34 KiB) Viewed 3084 times
User avatar
TheWitness
Developer
Posts: 17047
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:

Post by TheWitness »

Please post a complete DEBUG log run also, please run poller.php from a shell and post it's output.

Thanks,

TheWitness

p.s. If this is only affecting script's, there was a GLIBC update that was not behaving well in certain versions of Linux that may be affecting PHP's popen command.
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages


For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
User avatar
TheWitness
Developer
Posts: 17047
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:

Post by TheWitness »

Please read the following post:

http://forums.cacti.net/viewtopic.php?p=49344#49344

TheWitness
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages


For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Oxygen
Posts: 23
Joined: Fri Jun 10, 2005 1:29 pm
Location: Long Island, NY, USA

Post by Oxygen »

Here is the output from shell:
UXAPP:/usr/share/cacti# php poller.php
Waiting on 1/1 pollers.
OK u:0.01 s:0.00 r:0.04
OK u:0.01 s:0.00 r:0.04
OK u:0.01 s:0.00 r:0.04
OK u:0.01 s:0.00 r:0.04
OK u:0.01 s:0.00 r:0.04
OK u:0.01 s:0.00 r:0.04
OK u:0.01 s:0.00 r:0.04
OK u:0.01 s:0.00 r:0.04
OK u:0.01 s:0.00 r:0.04
OK u:0.01 s:0.00 r:0.04
OK u:0.01 s:0.00 r:0.04
OK u:0.01 s:0.00 r:0.04
OK u:0.01 s:0.00 r:0.04
OK u:0.01 s:0.00 r:0.04
OK u:0.01 s:0.00 r:0.04
OK u:0.01 s:0.00 r:0.04
OK u:0.01 s:0.00 r:0.04
OK u:0.01 s:0.00 r:0.04
OK u:0.01 s:0.00 r:0.05
OK u:0.01 s:0.00 r:0.05
OK u:0.01 s:0.00 r:0.05
OK u:0.01 s:0.00 r:0.05
OK u:0.01 s:0.00 r:0.05
OK u:0.01 s:0.00 r:0.05
sh: line 1: c:/perl/bin/perl.exe: No such file or directory
Waiting on 1/1 pollers.
OK u:0.01 s:0.00 r:1.09
OK u:0.01 s:0.00 r:1.09
OK u:0.01 s:0.00 r:1.09
OK u:0.01 s:0.00 r:1.09
OK u:0.01 s:0.00 r:1.09
OK u:0.01 s:0.00 r:1.09
OK u:0.01 s:0.00 r:1.09
OK u:0.01 s:0.00 r:1.09
OK u:0.01 s:0.00 r:1.09
OK u:0.01 s:0.00 r:1.09
OK u:0.01 s:0.00 r:1.09
OK u:0.01 s:0.00 r:1.09
OK u:0.01 s:0.00 r:1.09
OK u:0.01 s:0.00 r:1.09
OK u:0.01 s:0.00 r:1.09
OK u:0.01 s:0.00 r:1.09
OK u:0.01 s:0.00 r:1.09
OK u:0.01 s:0.00 r:1.09
OK u:0.01 s:0.00 r:1.09
OK u:0.01 s:0.00 r:1.09
OK u:0.01 s:0.00 r:1.09
OK u:0.01 s:0.00 r:1.09
OK u:0.01 s:0.00 r:1.09
OK u:0.01 s:0.00 r:1.09
OK u:0.01 s:0.00 r:1.09
OK u:0.01 s:0.00 r:1.09
OK u:0.01 s:0.00 r:1.09
OK u:0.01 s:0.00 r:1.09
OK u:0.01 s:0.00 r:1.09
OK u:0.01 s:0.00 r:1.09
OK u:0.01 s:0.00 r:1.09
OK u:0.01 s:0.00 r:1.09
OK u:0.01 s:0.00 r:1.09
OK u:0.01 s:0.00 r:1.09
ERROR: Not enough arguments
Waiting on 1/1 pollers.
Waiting on 1/1 pollers.
Waiting on 1/1 pollers.
Waiting on 1/1 pollers.
Waiting on 1/1 pollers.
OK u:0.01 s:0.00 r:6.16
OK u:0.01 s:0.00 r:6.16
OK u:0.01 s:0.00 r:6.16
OK u:0.01 s:0.00 r:6.16
OK u:0.01 s:0.00 r:6.16
OK u:0.01 s:0.00 r:6.16
OK u:0.01 s:0.00 r:6.16
OK u:0.01 s:0.00 r:6.16
OK u:0.01 s:0.00 r:6.16
OK u:0.01 s:0.00 r:6.16
OK u:0.01 s:0.00 r:6.16
OK u:0.01 s:0.00 r:6.16
OK u:0.01 s:0.00 r:6.16
OK u:0.01 s:0.00 r:6.16
OK u:0.01 s:0.00 r:6.16
OK u:0.01 s:0.00 r:6.16
OK u:0.01 s:0.00 r:6.16
OK u:0.01 s:0.00 r:6.16
OK u:0.01 s:0.00 r:6.16
OK u:0.01 s:0.00 r:6.16
OK u:0.01 s:0.00 r:6.16
OK u:0.01 s:0.00 r:6.16
Waiting on 1/1 pollers.
OK u:0.01 s:0.00 r:7.17
OK u:0.01 s:0.00 r:7.17
OK u:0.01 s:0.00 r:7.17
OK u:0.01 s:0.00 r:7.17
OK u:0.01 s:0.00 r:7.17
Waiting on 1/1 pollers.
OK u:0.01 s:0.00 r:8.18
OK u:0.01 s:0.00 r:8.18
OK u:0.01 s:0.00 r:8.18
OK u:0.01 s:0.00 r:8.18
OK u:0.01 s:0.00 r:8.18
OK u:0.01 s:0.00 r:8.18
Waiting on 1/1 pollers.
OK u:0.01 s:0.00 r:9.19
OK u:0.01 s:0.00 r:9.19
OK u:0.01 s:0.00 r:9.19
OK u:0.01 s:0.00 r:9.19
OK u:0.01 s:0.00 r:9.19
Waiting on 1/1 pollers.
OK u:0.01 s:0.00 r:10.20
OK u:0.01 s:0.00 r:10.20
OK u:0.01 s:0.00 r:10.20
Waiting on 1/1 pollers.
OK u:0.01 s:0.00 r:11.21
OK u:0.01 s:0.00 r:11.21
OK u:0.01 s:0.00 r:11.21
OK u:0.01 s:0.00 r:11.21
Waiting on 1/1 pollers.
OK u:0.01 s:0.00 r:12.22
OK u:0.01 s:0.00 r:12.22
OK u:0.01 s:0.00 r:12.22
OK u:0.01 s:0.00 r:12.22
OK u:0.01 s:0.00 r:12.22
Waiting on 1/1 pollers.
OK u:0.01 s:0.00 r:13.23
OK u:0.01 s:0.00 r:13.23
OK u:0.01 s:0.00 r:13.23
OK u:0.01 s:0.00 r:13.23
OK u:0.01 s:0.00 r:13.23
Waiting on 1/1 pollers.
OK u:0.01 s:0.00 r:14.24
OK u:0.01 s:0.00 r:14.24
OK u:0.01 s:0.00 r:14.24
OK u:0.01 s:0.00 r:14.24
OK u:0.01 s:0.00 r:14.24
Waiting on 1/1 pollers.
OK u:0.01 s:0.00 r:15.25
OK u:0.01 s:0.00 r:15.25
OK u:0.01 s:0.00 r:15.25
OK u:0.01 s:0.00 r:15.25
OK u:0.01 s:0.00 r:15.25
Waiting on 1/1 pollers.
OK u:0.01 s:0.00 r:16.26
OK u:0.01 s:0.00 r:16.26
OK u:0.01 s:0.00 r:16.26
OK u:0.01 s:0.00 r:16.26
OK u:0.01 s:0.00 r:16.26
OK u:0.01 s:0.00 r:16.26
Waiting on 1/1 pollers.
OK u:0.01 s:0.00 r:17.27
OK u:0.01 s:0.00 r:17.27
OK u:0.01 s:0.00 r:17.27
OK u:0.01 s:0.00 r:17.27
OK u:0.01 s:0.00 r:17.27
Waiting on 1/1 pollers.
OK u:0.01 s:0.00 r:18.28
OK u:0.01 s:0.00 r:18.28
OK u:0.01 s:0.00 r:18.28
OK u:0.01 s:0.00 r:18.28
OK u:0.01 s:0.00 r:18.28
OK u:0.01 s:0.00 r:18.28
Waiting on 1/1 pollers.
OK u:0.01 s:0.00 r:19.30
OK u:0.01 s:0.00 r:19.30
OK u:0.01 s:0.00 r:19.30
OK u:0.01 s:0.00 r:19.30
OK u:0.01 s:0.00 r:19.30
OK u:0.01 s:0.00 r:19.30
OK u:0.01 s:0.00 r:19.30
Waiting on 1/1 pollers.
OK u:0.01 s:0.00 r:20.31
OK u:0.01 s:0.00 r:20.31
OK u:0.01 s:0.00 r:20.31
OK u:0.01 s:0.00 r:20.31
OK u:0.01 s:0.00 r:20.31
OK u:0.01 s:0.00 r:20.31
Waiting on 1/1 pollers.
OK u:0.01 s:0.00 r:21.32
OK u:0.01 s:0.00 r:21.32
OK u:0.01 s:0.00 r:21.32
OK u:0.01 s:0.00 r:21.32
OK u:0.01 s:0.00 r:21.32
OK u:0.01 s:0.00 r:21.32
OK u:0.01 s:0.00 r:21.32
Waiting on 1/1 pollers.
OK u:0.01 s:0.00 r:22.33
OK u:0.01 s:0.00 r:22.33
OK u:0.01 s:0.00 r:22.33
OK u:0.01 s:0.00 r:22.33
OK u:0.01 s:0.00 r:22.33
OK u:0.01 s:0.00 r:22.33
OK u:0.01 s:0.00 r:22.33
Waiting on 1/1 pollers.
OK u:0.01 s:0.01 r:23.35
OK u:0.01 s:0.01 r:23.35
OK u:0.01 s:0.01 r:23.35
OK u:0.01 s:0.01 r:23.35
OK u:0.01 s:0.01 r:23.35
OK u:0.01 s:0.01 r:23.35
OK u:0.01 s:0.01 r:23.35
OK u:0.01 s:0.01 r:23.35
Waiting on 1/1 pollers.
OK u:0.01 s:0.01 r:24.36
OK u:0.01 s:0.01 r:24.36
OK u:0.01 s:0.01 r:24.36
OK u:0.01 s:0.01 r:24.36
OK u:0.01 s:0.01 r:24.36
OK u:0.01 s:0.01 r:24.36
OK u:0.01 s:0.01 r:24.36
OK u:0.01 s:0.01 r:24.37
Waiting on 1/1 pollers.
OK u:0.02 s:0.01 r:25.38
OK u:0.02 s:0.01 r:25.38
OK u:0.02 s:0.01 r:25.38
OK u:0.02 s:0.01 r:25.38
OK u:0.02 s:0.01 r:25.38
OK u:0.02 s:0.01 r:25.38
OK u:0.02 s:0.01 r:25.38
OK u:0.02 s:0.01 r:25.38
OK u:0.02 s:0.01 r:26.39
OK u:0.02 s:0.01 r:26.39
OK u:0.02 s:0.01 r:26.39
OK u:0.02 s:0.01 r:26.39
OK u:0.02 s:0.01 r:26.39
OK u:0.02 s:0.01 r:26.39
OK u:0.02 s:0.01 r:26.39
OK u:0.02 s:0.01 r:26.39
OK u:0.02 s:0.01 r:26.39
OK u:0.02 s:0.01 r:26.39
OK u:0.02 s:0.01 r:26.39
OK u:0.02 s:0.01 r:26.39
OK u:0.02 s:0.01 r:26.39
OK u:0.02 s:0.01 r:26.39
OK u:0.02 s:0.01 r:26.39
OK u:0.02 s:0.01 r:26.39
11/26/2005 01:56:05 PM - SYSTEM STATS: Time:26.4259 Method:cmd.php Processes:1 Threads:N/A Hosts:13 HostsPerProcess:13 DataSources:327 RRDsProcessed:208
Looks like the graphs are generated only when I run poller.php from shell. This only started working after I applied poller.php?rev=3333 per your directions.

So it looks like that the server is not executing the poller on it's own?
Attachments
cacti.JPG
cacti.JPG (30.52 KiB) Viewed 3060 times
User avatar
gandalf
Developer
Posts: 22383
Joined: Thu Dec 02, 2004 2:46 am
Location: Muenster, Germany
Contact:

Post by gandalf »

First, DEBUG log means
- (optional) clear cacti.log at <path_cacti>/log
- goto Settings -> Poller Logging Level and set this to DEBUG.
- run this at least for one polling cycle
- post the output in cacti.log from the last run (if you have cleared this before, simply post the whole file)
- reset logging level (do not fill all your disk space with logs)

Second, when poller won't run from crontab, this may be solved by providing the whole path to the php binary (e.g. /usr/bin/php instead of just php)

Reinhard
Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests