Comments on: SQLite performance tuning and optimization on embedded systems http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/ Chaotic solutions and random thoughts from the restless mind of a notorious problem solver (TM), by Andre Beckedorf Tue, 24 Apr 2012 02:48:27 +0000 hourly 1 http://wordpress.org/?v=3.3.2 By: db2 adminsitration http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/comment-page-1/#comment-31984 db2 adminsitration Fri, 13 Jan 2012 12:15:23 +0000 http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/#comment-31984 I believe this is among the so much significant information for me. And i'm satisfied reading your article. However wanna remark on some basic things, The website style is wonderful, the articles is really excellent : D. Excellent activity, cheers I believe this is among the so much significant information for me. And i’m satisfied reading your article. However wanna remark on some basic things, The website style is wonderful, the articles is really excellent : D. Excellent activity, cheers

]]> By: SQLite performance tuning and optimization on embedded systems : http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/comment-page-1/#comment-27825 SQLite performance tuning and optimization on embedded systems : Fri, 19 Aug 2011 20:17:14 +0000 http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/#comment-27825 [...] source: http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded... [...] [...] source: http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded… [...]

]]> By: Python and Sqlite « Abner’s Postgraduate Days http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/comment-page-1/#comment-23119 Python and Sqlite « Abner’s Postgraduate Days Tue, 02 Feb 2010 15:56:54 +0000 http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/#comment-23119 [...] SQLite performance tuning and optimization on embedded systems [...] [...] SQLite performance tuning and optimization on embedded systems [...]

]]> By: links for 2009-02-06 | Nathan and his Open Ideals http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/comment-page-1/#comment-22787 links for 2009-02-06 | Nathan and his Open Ideals Fri, 06 Feb 2009 16:02:54 +0000 http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/#comment-22787 [...] SQLite performance tuning and optimization on embedded systems | Katastrophos.net Blog Based on the experience I gained while developing my Zaurus media player, here is a short compendium of optimization rules, tweaks and hints when using SQLite on an embedded system (may apply to other systems as well) (tags: database performance optimization sqlite) Share and Enjoy: [...] [...] SQLite performance tuning and optimization on embedded systems | Katastrophos.net Blog Based on the experience I gained while developing my Zaurus media player, here is a short compendium of optimization rules, tweaks and hints when using SQLite on an embedded system (may apply to other systems as well) (tags: database performance optimization sqlite) Share and Enjoy: [...]

]]> By: Songbird Blog » Performance Improvements for 1.0 http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/comment-page-1/#comment-22751 Songbird Blog » Performance Improvements for 1.0 Thu, 27 Nov 2008 01:42:30 +0000 http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/#comment-22751 [...] SQLite performance tuning and optimization on embedded systems [...] [...] SQLite performance tuning and optimization on embedded systems [...]

]]> By: kalyani http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/comment-page-1/#comment-11536 kalyani Wed, 30 May 2007 11:36:13 +0000 http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/#comment-11536 Hi Andre, Have you done any memory profiling on sqlite? Do you any optimization steps for reducing heap usage with out degrading performance and have least foot print. Hi Andre,
Have you done any memory profiling on sqlite? Do you any optimization steps for reducing heap usage with out degrading performance and have least foot print.

]]> By: Andre http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/comment-page-1/#comment-5139 Andre Mon, 05 Mar 2007 23:07:30 +0000 http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/#comment-5139 Thanks for the comment, Dave. Well, it largely depends on the use case and that's what I failed to point out. I agree, for joining of big tables it's probably necessary to add an index on the hashkey to at least one table assuming the table order in the query is right. For lookups via the hash value of the string an index would also be required, otherwise performance will be sacrificed. And even with an index on the hash value, my experience is, it's still a bit faster - provided that the hash value is an integer. I can only assume this is because indices on integers are more efficient than those on strings. Also, I haven't been clear here: Having hashkeys over indices pays off for smaller tables, especially for temporary tables storing small amounts of intermediate results. Selects and joins without an index seem to be faster here compared to the overall time needed for setting up the index again and looking up in the index. It all depends on how much entries you have. Here is an example for a different use case: When reading a playlist, my media player has to lookup metadata for each filename in the playlist within the DB. The problem is, the filename isn't saved in one place within the DB. Actually the filename is split into location/dirname and basename, both of which are kept in separate tables for space-saving reasons. I can't directly use an index here. I have to split up the filename and do an indexed lookup for each part over two tables. This is slower then using the hashkey approach with its preliminaries. That's why I'm hashing the filename in my application to a 32-bit int. The same method is used when updating and adding new entries to the metadata table. But then again, everything depends on the datastructures and the requirements of the application, whether a hashkey approach might be better or not. Andre Thanks for the comment, Dave.
Well, it largely depends on the use case and that’s what I failed to point out.
I agree, for joining of big tables it’s probably necessary to add an index on the hashkey to at least one table assuming the table order in the query is right. For lookups via the hash value of the string an index would also be required, otherwise performance will be sacrificed.
And even with an index on the hash value, my experience is, it’s still a bit faster – provided that the hash value is an integer. I can only assume this is because indices on integers are more efficient than those on strings.

Also, I haven’t been clear here: Having hashkeys over indices pays off for smaller tables, especially for temporary tables storing small amounts of intermediate results. Selects and joins without an index seem to be faster here compared to the overall time needed for setting up the index again and looking up in the index. It all depends on how much entries you have.

Here is an example for a different use case:
When reading a playlist, my media player has to lookup metadata for each filename in the playlist within the DB.
The problem is, the filename isn’t saved in one place within the DB. Actually the filename is split into location/dirname and basename, both of which are kept in separate tables for space-saving reasons. I can’t directly use an index here. I have to split up the filename and do an indexed lookup for each part over two tables. This is slower then using the hashkey approach with its preliminaries.
That’s why I’m hashing the filename in my application to a 32-bit int. The same method is used when updating and adding new entries to the metadata table.

But then again, everything depends on the datastructures and the requirements of the application, whether a hashkey approach might be better or not.

Andre

]]> By: Dave Gauthier http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/comment-page-1/#comment-5132 Dave Gauthier Mon, 05 Mar 2007 15:41:42 +0000 http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/#comment-5132 I'm not clear on the suggestion regarding the use of hashkeys. Are you proposing using a smaller/leaner hashkey in place of long strings as primary/foreign keys? If so, after you generate & store the hashkeys in the primary & foreign tables, wouldn't you still need to create an index on the hashkey columns and "join" the tables via that column in order to leverage it at query time? I’m not clear on the suggestion regarding the use of hashkeys. Are you proposing using a smaller/leaner hashkey in place of long strings as primary/foreign keys? If so, after you generate & store the hashkeys in the primary & foreign tables, wouldn’t you still need to create an index on the hashkey columns and “join” the tables via that column in order to leverage it at query time?

]]> By: “Yet Another Zaurus Media Player”… done differently . (Phase 2: “Development progress, no release yet.”) at Katastrophos.net Blog http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/comment-page-1/#comment-2472 “Yet Another Zaurus Media Player”… done differently . (Phase 2: “Development progress, no release yet.”) at Katastrophos.net Blog Thu, 04 Jan 2007 10:33:31 +0000 http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/#comment-2472 [...] Software « “Yet Another Zaurus Media Player”… done differently . (Phase 1: Teasing) SQLite performance tuning and optimization on embedded systems » [...] [...] Software « “Yet Another Zaurus Media Player”… done differently . (Phase 1: Teasing) SQLite performance tuning and optimization on embedded systems » [...]

]]> By: SQLite simple timing profiler patch at Katastrophos.net Blog http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/comment-page-1/#comment-2469 SQLite simple timing profiler patch at Katastrophos.net Blog Thu, 04 Jan 2007 10:27:21 +0000 http://katastrophos.net/andre/blog/2007/01/04/sqlite-performance-tuning-and-optimization-on-embedded-systems/#comment-2469 [...] Software « SQLite performance tuning and optimization on embedded systems [...] [...] Software « SQLite performance tuning and optimization on embedded systems [...]

]]>