Comments on: SQLite performance tuning and optimization on embedded systems http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/ Chaotic solutions and random thoughts from the restless mind of a notorious problem solver (TM), by Andre Beckedorf Wed, 07 Jan 2009 00:36:51 +0000 http://wordpress.org/?v=2.7 hourly 1 By: Songbird Blog » Performance Improvements for 1.0 http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/comment-page-1/#comment-22751 Songbird Blog » Performance Improvements for 1.0 Thu, 27 Nov 2008 01:42:30 +0000 http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/#comment-22751 [...] SQLite performance tuning and optimization on embedded systems [...] [...] SQLite performance tuning and optimization on embedded systems [...]

]]> By: kalyani http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/comment-page-1/#comment-11536 kalyani Wed, 30 May 2007 11:36:13 +0000 http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/#comment-11536 Hi Andre, Have you done any memory profiling on sqlite? Do you any optimization steps for reducing heap usage with out degrading performance and have least foot print. Hi Andre,
Have you done any memory profiling on sqlite? Do you any optimization steps for reducing heap usage with out degrading performance and have least foot print.

]]> By: Andre http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/comment-page-1/#comment-5139 Andre Mon, 05 Mar 2007 23:07:30 +0000 http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/#comment-5139 Thanks for the comment, Dave. Well, it largely depends on the use case and that's what I failed to point out. I agree, for joining of big tables it's probably necessary to add an index on the hashkey to at least one table assuming the table order in the query is right. For lookups via the hash value of the string an index would also be required, otherwise performance will be sacrificed. And even with an index on the hash value, my experience is, it's still a bit faster - provided that the hash value is an integer. I can only assume this is because indices on integers are more efficient than those on strings. Also, I haven't been clear here: Having hashkeys over indices pays off for smaller tables, especially for temporary tables storing small amounts of intermediate results. Selects and joins without an index seem to be faster here compared to the overall time needed for setting up the index again and looking up in the index. It all depends on how much entries you have. Here is an example for a different use case: When reading a playlist, my media player has to lookup metadata for each filename in the playlist within the DB. The problem is, the filename isn't saved in one place within the DB. Actually the filename is split into location/dirname and basename, both of which are kept in separate tables for space-saving reasons. I can't directly use an index here. I have to split up the filename and do an indexed lookup for each part over two tables. This is slower then using the hashkey approach with its preliminaries. That's why I'm hashing the filename in my application to a 32-bit int. The same method is used when updating and adding new entries to the metadata table. But then again, everything depends on the datastructures and the requirements of the application, whether a hashkey approach might be better or not. Andre Thanks for the comment, Dave.
Well, it largely depends on the use case and that’s what I failed to point out.
I agree, for joining of big tables it’s probably necessary to add an index on the hashkey to at least one table assuming the table order in the query is right. For lookups via the hash value of the string an index would also be required, otherwise performance will be sacrificed.
And even with an index on the hash value, my experience is, it’s still a bit faster - provided that the hash value is an integer. I can only assume this is because indices on integers are more efficient than those on strings.

Also, I haven’t been clear here: Having hashkeys over indices pays off for smaller tables, especially for temporary tables storing small amounts of intermediate results. Selects and joins without an index seem to be faster here compared to the overall time needed for setting up the index again and looking up in the index. It all depends on how much entries you have.

Here is an example for a different use case:
When reading a playlist, my media player has to lookup metadata for each filename in the playlist within the DB.
The problem is, the filename isn’t saved in one place within the DB. Actually the filename is split into location/dirname and basename, both of which are kept in separate tables for space-saving reasons. I can’t directly use an index here. I have to split up the filename and do an indexed lookup for each part over two tables. This is slower then using the hashkey approach with its preliminaries.
That’s why I’m hashing the filename in my application to a 32-bit int. The same method is used when updating and adding new entries to the metadata table.

But then again, everything depends on the datastructures and the requirements of the application, whether a hashkey approach might be better or not.

Andre

]]> By: Dave Gauthier http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/comment-page-1/#comment-5132 Dave Gauthier Mon, 05 Mar 2007 15:41:42 +0000 http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/#comment-5132 I'm not clear on the suggestion regarding the use of hashkeys. Are you proposing using a smaller/leaner hashkey in place of long strings as primary/foreign keys? If so, after you generate & store the hashkeys in the primary & foreign tables, wouldn't you still need to create an index on the hashkey columns and "join" the tables via that column in order to leverage it at query time? I’m not clear on the suggestion regarding the use of hashkeys. Are you proposing using a smaller/leaner hashkey in place of long strings as primary/foreign keys? If so, after you generate & store the hashkeys in the primary & foreign tables, wouldn’t you still need to create an index on the hashkey columns and “join” the tables via that column in order to leverage it at query time?

]]> By: “Yet Another Zaurus Media Player”… done differently . (Phase 2: “Development progress, no release yet.”) at Katastrophos.net Blog http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/comment-page-1/#comment-2472 “Yet Another Zaurus Media Player”… done differently . (Phase 2: “Development progress, no release yet.”) at Katastrophos.net Blog Thu, 04 Jan 2007 10:33:31 +0000 http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/#comment-2472 [...] Software « “Yet Another Zaurus Media Player”… done differently . (Phase 1: Teasing) SQLite performance tuning and optimization on embedded systems » [...] [...] Software « “Yet Another Zaurus Media Player”… done differently . (Phase 1: Teasing) SQLite performance tuning and optimization on embedded systems » [...]

]]> By: SQLite simple timing profiler patch at Katastrophos.net Blog http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/comment-page-1/#comment-2469 SQLite simple timing profiler patch at Katastrophos.net Blog Thu, 04 Jan 2007 10:27:21 +0000 http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/#comment-2469 [...] Software « SQLite performance tuning and optimization on embedded systems [...] [...] Software « SQLite performance tuning and optimization on embedded systems [...]

]]> By: Andre http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/comment-page-1/#comment-2462 Andre Thu, 04 Jan 2007 06:42:29 +0000 http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/#comment-2462 Here are the flags I'm currently using for my Zaurus: -DSQLITE_OMIT_UTF16 -DSQLITE_OMIT_AUTHORIZATION -DSQLITE_OMIT_PROGRESS_CALLBACK \ -DSQLITE_OMIT_SHARED_CACHE -DSQLITE_OMIT_LOAD_EXTENSION -DNDEBUG \ -march=armv4 -mtune=strongarm -O3 -fomit-frame-pointer Compiler is GCC 2.95.3. Over -O2 these flags reduced the execution time of my SQL testsuite from 1040 ms down to 880 ms. This is caused mostly due to the function inlining and architecture optimization. Unrolling loops will hurt performance in most cases. Your mileage may vary. Here are the flags I’m currently using for my Zaurus:

-DSQLITE_OMIT_UTF16 -DSQLITE_OMIT_AUTHORIZATION -DSQLITE_OMIT_PROGRESS_CALLBACK \
-DSQLITE_OMIT_SHARED_CACHE -DSQLITE_OMIT_LOAD_EXTENSION -DNDEBUG \
-march=armv4 -mtune=strongarm -O3 -fomit-frame-pointer

Compiler is GCC 2.95.3. Over -O2 these flags reduced the execution time of my SQL testsuite from 1040 ms down to 880 ms. This is caused mostly due to the function inlining and architecture optimization.
Unrolling loops will hurt performance in most cases. Your mileage may vary.

]]>