Filesystem full - cacti does not log anymore

tkmbe · Post by **tkmbe** » Tue Oct 21, 2008 8:41 pm

Hello

My filesystem were the cacti data are was full and now after removing obsolete stuff cacti is not logging anymore the most stuff...
Some rrd's are updated but the most not.
The mysql database is on an other fs than the data of cacti.
How can I solve the problem wihtout losing the old data?
Any hints?

Thanks

Beat

tkmbe · Post by **tkmbe** » Tue Oct 21, 2008 9:02 pm

BTW: There are differences in the rra files in the length
which works and don't work
Fore example
4579412 bytes
4579416 bytes
of one example and
1527444
1527448
in an other example ...
I don't understand id the rra is created with a fixed length why
a full fs can affect the old rra's ...
Of course newly created rra's will have problems but the old ones
which were already created ...

BTW: Recreating the whole stuff is really no option

I have 542 rra files

tkmbe · Post by **tkmbe** » Tue Oct 21, 2008 10:54 pm

Ok the problem seems to be solved.
There were rrd files with length 0 and some other length which were created while fs was full...
A run of the poller from shell did complain about 2 files with the following error:
ERROR: reading the cookie off .../cacti-test/rra/xxx.rrd faild
After removing whole stuff which was created now it seems to work.

But why cacti does not put the stuff in the rrd files which works????????
This really seems to be a bug i.e. if there are corrupted rrd files
other rrd files which are ok will not be updated.
Could it be that the whole poller thread will fail?
I have max. 10 concurrent poller running...

My cacti version: 0.8.7b
rrdtools: 1.2.23-47
System: OpenSuse 10.3

If you need more info let me know.

Greetings

Beat

Howie · Post by **Howie** » Wed Oct 22, 2008 2:23 am

It's worse than that, even. On some filesystems there is a 2Gig size limit for single file. If the log file reaches that size, then the poller stops running!

(this was a few versions ago, I don't know if it's still a problem)

Post by **TheWitness** » Wed Oct 22, 2008 5:43 am

It's more of an OS/PHP issue than a Cacti issue. This is why you should setup log rotation in Cacti. The Debian distribution does this about the best. It not only places the Cacti.log file in /var/log, but rotates it and moves a few other things around to make Cacti a little more secure.

TheWitness

tkmbe · Post by **tkmbe** » Wed Oct 22, 2008 6:43 am

I can understand if you cannot log or in my case a file is corrupted you leave the poller but a message on syslog would be very nice!

Howie · Post by **Howie** » Wed Oct 22, 2008 7:14 am

TheWitness wrote:It's more of an OS/PHP issue than a Cacti issue. This is why you should setup log rotation in Cacti. The Debian distribution does this about the best. It not only places the Cacti.log file in /var/log, but rotates it and moves a few other things around to make Cacti a little more secure.

TheWitness

Really? When a process can't write to a file, the OS kills it?

tkmbe · Post by **tkmbe** » Wed Oct 22, 2008 1:14 pm

Are you talking about linux or windows???

Maybe php4 kills the process. I don't know php4 but not the os!!
The problem is that 95% of the software never checks a write if it
worked or not (what should go wrong with a write??)!!
And of course the other idea of programmers are what can I do if the write
does not work ...
The problem maybe is that if the process cannot write to the fs it will
after some time write in the wrong page or something else and than it
receives the sigv ...

Nevertheless I can tell you that the os don't kill the process!
I don't know php4 as noted but know a lot of C,C++ and java.

Greetings

Beat

Filesystem full - cacti does not log anymore

Filesystem full - cacti does not log anymore

Who is online