Cacti with sqlite - any interest?

Post general support questions here that do not specifically fall into the Linux or Windows categories.

Moderators: Developers, Moderators

beagle1
Posts: 18
Joined: Fri Aug 24, 2012 1:56 am

Cacti with sqlite - any interest?

Post by beagle1 »

I run cacti on an embedded system (beaglebone) with limited resources and running MySQL seems just like a waste.

I have been working on converting cacti (0.8.8a) to use sqlite instead of MySQL and am quite pleased with the results so far. Here is what I have figured out so far:

- dumped the mysql data and loaded into sqlite using the mysql2sqlite.sh script. A few changes I made to the output from the script before importing into sqlite were: 1. change all autoincrement PK fields to be 'integer' instead of int, smallint, mediumint - this automatically enables autoincrement for sqlite and 2. rename the user_auth_realm index to user_auth_realm1 since it conflicts with the table of the same name
- added in a connection method for sqlite using pdo_sqlite
- modified mysql-specific sql to be compatible with sqlite without sacrificing mysql compatibility. Specifically, these were the changes (still a work in progress)
  • alias column names in select to their raw column names. This is mysql-compatible and works with sqlite, otherwise sqlite returns names prefixed with the table alias, which breaks array indexes
    replace the mysql CONCAT and CONCAT_WS operators with php concatenate (.) instead. sqlite does not have CONCAT or CONCAT_WS and the sqlite (ansi) concat operator (||) is not compatible with mysql, so doing it in php is the safest
    convert all RIGHT JOINs to their corresponding LEFT JOINs - sqlite does not support RIGHT JOINs
    convert UNIX_TIMESTAMP to the corresponding php strtotime instead - sqlite does not have UNIX_TIMESTAMP
This has maintained compatibility with MySQL while enabling sqlite. The only file I have mangled so far with no compatibility is utility.php, which has a lot of MySQL-specific commands (show tables, show create table etc.). This file will probably need more work and a database detection switch to run the appropriate commands, but right now it is working for sqlite. I also have the poller and graph display pages working

Now here's the question - is anybody interested in a sqlite port for cacti? Would there be any interest in pushing the changes into the mainstream code since they do not break MySQL compatibility and set a direction to allow for other databases to be used as the back-end as well? I am not a programmer, I am a DBA, so my code is more functional than elegant, I would appreciate a review anyway.

I'll be happy to post the code or email it if there's any interest, just let me know!
Last edited by beagle1 on Sat Aug 25, 2012 3:20 am, edited 1 time in total.
beagle1
Posts: 18
Joined: Fri Aug 24, 2012 1:56 am

Re: Cacti with sqlite - any interest

Post by beagle1 »

A few more changes for anybody following along:
  • convert truncate to delete from $table. This is bizarre because sqlite optimizes it internally to a truncate but doesn't support truncate directly. May need to detect db and use truncate or delete from appropriately
    convert extended insert to single inserts - definitely a mysql'ism but it is easy enough to do by setting a smaller insert buffer to force single inserts. db detection will help here as well
    convert now() to .. datetime(). Eventually needs to be a php function or db specific code, I haven't found a compatible way to do this across databases
the code is pretty much fixed now and while I can't claim 100% success, all the parts I have tested so far are working. I have added in the adodb-errorhandler in database.php to stop and print any database (sqlite) errors, which has helped speed up debugging tremendously.

In all, perhaps a day's work in all to convert cacti from using mysql to using sqlite. I will be putting it into production soon to test it out!
beagle1
Posts: 18
Joined: Fri Aug 24, 2012 1:56 am

Re: Cacti with sqlite - any interest?

Post by beagle1 »

and a few more

- create temporary table like.. doesn't work, needs to be create temporary table as select * from [tblname] where 1=2;
- .. on duplicate key update is not sqlite compatible. this will need to be trapped in php and re-done, but for now I just took it out - I assume that means I will have to hit refresh poller cache twice before it is effective.
- there is one complicated query with nested inner joins (why!!??) that won't work with sqlite. I converted it to more conventional natural joins - much easier to read too
bitwelder
Posts: 4
Joined: Thu Aug 30, 2012 9:42 am

Re: Cacti with sqlite - any interest?

Post by bitwelder »

Hello.

I'm planning to run Cacti on a Raspberry Pi possibly inside some kind of ramdisk, so my space requirements are also pretty tight :)
I'd like to know more details about the modifications to the script you talked about in the first post, and of course I'd be happy to be a guinea pig for the installation :wink:
beagle1
Posts: 18
Joined: Fri Aug 24, 2012 1:56 am

Re: Cacti with sqlite - any interest?

Post by beagle1 »

bitwelder wrote:Hello.

I'm planning to run Cacti on a Raspberry Pi possibly inside some kind of ramdisk, so my space requirements are also pretty tight :)
I'd like to know more details about the modifications to the script you talked about in the first post, and of course I'd be happy to be a guinea pig for the installation :wink:
the Pi and the Beaglebone are quite similar in spec - the big difference is that the Pi has audio/video output with limited gpio / aio and the beaglebone is truly headless with lots of gpio and analog inputs.. but in any case, the performance should be comparable. I have noticed a huge reduction in cpu usage after turning off MySQL - it had accumulated quite a bit of cpu time as well.

I'm currently travelling, but I will provide you the code in a day or two. Does the board allow private messages? If so send me your email address and I'll mail you the latest version I have.
bitwelder
Posts: 4
Joined: Thu Aug 30, 2012 9:42 am

Re: Cacti with sqlite - any interest?

Post by bitwelder »

No problem, I can wait :)

Also, I'm starting from scratch so at least in my case, I don't have the burden to convert old data from mysql to sqlite.
One question though. In raspbian repository there are available sqlite (v2.8.17) and sqlite3 (v3.7.13). Should I go for v3?

Probably the board has a minimum number of messages to post before an user can send PM.
Perhaps if your scripts are ready for 'draft publication' you could post them to e.g. to pastebin.com and post just the instructions here?
beagle1
Posts: 18
Joined: Fri Aug 24, 2012 1:56 am

Re: Cacti with sqlite - any interest?

Post by beagle1 »

sorry for the delay - I'm still fixing the new installation scripts - I ported my install over from MySQL so did not actually test the new installation scripts until now.
pastebin will only allow 1 script at a time, I have modified multiple scripts - but I see there is an attachment option with posts here, will try that out and see if it works
bitwelder
Posts: 4
Joined: Thu Aug 30, 2012 9:42 am

Re: Cacti with sqlite - any interest?

Post by bitwelder »

Ok, I'm busy with other stuff (e.g. testing 'ramlog' options on RPi), so take your time to fix the scripts.

Nobody else interested in the sqlite porting?
beagle1
Posts: 18
Joined: Fri Aug 24, 2012 1:56 am

Re: Cacti with sqlite - any interest?

Post by beagle1 »

here you go! the cacti-sqlite.sql file will have to be run first to populate the sqlite db:
sqlite3 /path/to/db < cacti-sqlite.sql
then point your browser to the cacti install
http://localhost/cacti/

Let me know how it goes
Attachments
cacti-sqlite-0.8.8a.tar.gz
cacti with sqlite
(2.58 MiB) Downloaded 420 times
bitwelder
Posts: 4
Joined: Thu Aug 30, 2012 9:42 am

Re: Cacti with sqlite - any interest?

Post by bitwelder »

I just tried the installation with your script and it seems to work nicely :D

Only thing is, in the post you said:
sqlite3 /path/to/db < cacti-sqlite.sql
then point your browser to the cacti install
http://localhost/cacti/
but actually before firing up the installation, the database path should be set also in .../cacti/include/config.php ($database_port)

The other minor thing is that with a freshly installed PHP, I got lighttpd logs filled with PHP warnings about date.timezone not being set, so I had to configure that in my php.ini files.

Anyway, now at least Cacti is installed and it's running. I will soon start to configure it, and report back if there is any issue apparently related to the DB.

Thanks!
beagle1
Posts: 18
Joined: Fri Aug 24, 2012 1:56 am

Re: Cacti with sqlite - any interest?

Post by beagle1 »

yeah the regular cacti config / install instructions still apply.. I pointed out those 2 since the seed sql is different (cacti-sqlite.sql). Please do follow the rest of the instructions.
I ported my install from MySQL to sqlite and haven't had a chance to try setting up new data sources / templates / graphs - so if you face issues in any of those, please let me know. What you have is what I consider the 'debug' version for sqlite - as soon as there is an error, it will raise it with php with the exact sql that it failed on. If there is any strange behaviour, please look in cacti.log. Depending on your php settings, it may also show up on your browser. In any case, paste that here along with details of what you were trying to do and I will debug.
arjunadeltoso
Posts: 1
Joined: Mon Apr 22, 2013 1:31 pm

Re: Cacti with sqlite - any interest?

Post by arjunadeltoso »

thanks a million @beagle1 for your job, I tried your version and found a small bug so I opened this github repo to store cacti on sqlite (your code) and patch it to fix the issue https://github.com/arjunadeltoso/cacti-sqlite
beagle1
Posts: 18
Joined: Fri Aug 24, 2012 1:56 am

Re: Cacti with sqlite - any interest?

Post by beagle1 »

great, glad you found it useful and thanks for putting it up on a repo! Hopefully there will be more users. My cacti is still running strong on my beaglebone
muellerjm
Posts: 12
Joined: Tue Jun 04, 2013 12:28 pm

Re: Cacti with sqlite - any interest?

Post by muellerjm »

Hello,

ohhh, i have really so great interests...

but i have problems with installations.

1. Installed Debian on Raspi Pi
2. installed php, sqlite3
3. Created a Database in /var/lib/sqlite/cacti.db an run the sql-script ( the tables would be created)
4. Downloaded from Github the cacti files an stored them in /var/www/cacti
5. entered the path entry in configure.php ($database_port = "/var/lib/sqlite/cacti.db";)
6. Run <ip>/cacti

issue... i got only a empty page. Maybe you have a solution for me?

Thank you
joe
muellerjm
Posts: 12
Joined: Tue Jun 04, 2013 12:28 pm

Re: Cacti with sqlite - any interest?

Post by muellerjm »

found.. in Apache log file

[Tue Jun 04 20:56:20 2013] [error] [client 10.1.1.3] PHP Fatal error: pdo error: [-1: Connection attempt failed: could not find driver] in CONNECT(sqlite:/var/lib/sqlite/cacti.db, '****', '****', )\n in /var/www/cacti/lib/adodb/adodb-errorhandler.inc.php on line 77

Any Idea?
Post Reply

Who is online

Users browsing this forum: No registered users and 3 guests