“Yet Another Zaurus Media Player”… done differently . (Phase 2.1: Development progress 2)

Posted on by Andre Beckedorf

Yet another short update on the development of my still untitled media player for the Zaurus. In the meantime it’s called YAZMP.

Again, I’ve been working on improving performance – this time on the performance when loading playlists.
Before I continue, let me give a brief overview of the structure:

Library -> Playlists < -> Media Cache

YAZMP doesn’t manage a library similar to i*Tunes. Instead it solely relies on playlists. Metadata (title, artist, album, etc.) is kept in the database and will be associated to once the playlist is loaded. The reason for this is pretty simple: Scanning audio files (and media files in general) each and every time a playlist is loaded will definitely take a lot of time. So, for every file the gathered metadata will be saved in the DB. Think of it as a cache.
Continue reading ““Yet Another Zaurus Media Player”… done differently . (Phase 2.1: Development progress 2)” →

SQLite simple timing profiler patch

Posted on by Andre Beckedorf

Update: SQLite now features an integrated timer which can be activated via the .timer command in the console shell. Details here.

As a follow-up to my previous post “SQLite performance tuning and optimization on embedded systems” here is a very basic patch that introduces support for timing SQL queries in the sqlite3 console shell.

SQLite 3.3.8:
Patch against shell.c shell.c

SQLite 3.3.9 CVS – shell.c rev 1.157:
Patch against shell.c shell.c

Two new shell commands are introduced in sqlite3:

.profile ON|OFF        Turn profiling on or off
.timer start|show      Measure elapsed CPU time

.profile ON will turn off output to stdout. Instead it will display CPU execution time for every SQL command processed.
Here is an example output:

sqlite> .profile on
sqlite> .read test1.sql
Exec time  BEGIN;                                       :  0.000 s.
Exec time  DROP TABLE IF EXISTS playlist_view;          :  0.020 s.
Exec time  DROP TABLE IF EXISTS overview;               :  0.000 s.
Exec time  CREATE TEMPORARY TABLE playlist_view AS   ...  :  0.180 s.
Exec time  CREATE TEMPORARY TABLE overview AS        ...  :  0.140 s.
Exec time  SELECT DISTINCT genre, genre IN ("Progress...  :  0.030 s.
...

Note: You can still override the output setting with the .output command after an .profile ON has been issued.
.profile OFF will turn off time-based profiling and reenable output to stdout.

.timer start will start a timer. You can execute a sequence of queries and then print the required CPU time via .timer show.
Use .timer start again to reset the timer. This .timer patch is based on SQLite ticket #1227 by Bartosz Polednia.

Timing is currently done using clock() which provides only a 10 ms precision for CPU time on most systems. This might be inadequate depending on your requirements. If you know a more precise way, please let me know.

SQLite performance tuning and optimization on embedded systems

Posted on by Andre Beckedorf

Based on the experience I gained while developing my Zaurus media player, here is a short compendium of optimization rules, tweaks and hints when using SQLite on an embedded system (may apply to other systems as well):

  • Simplify the database schema as much as possible – even if that means redundant data or illogical structure
  • Don’t generalize the database schema – generalization will mostly sacrifice performance and one can’t afford that on an embedded system with its tight restrictions, even if it is more convenient for the developer.
  • Only use relations (via IDs etc.) where absolutely necessary. The overhead for lookup and joining tables is considerable, even with an index on the relation.
  • Order the tables correctly in SELECTs. Put a table left-most if it is lacking an index on the relation. More details are here.
    In general: Check the order of tables in the SELECT statement. A different permutation may be more optimal. Profile.
  • Prepare your statements and bind values where applicable. This way you can get rid of the parser and VM creation overhead in tight loops (e.g. when inserting and updating).
  • Use transactions – even if you’re just reading the data. This may yield a few milliseconds.
  • Use temporary tables for intermediate results. They are fast and stay in cache most of the time. Depending on how your SQLite instance is set up, data will only be swapped into an external file if the cache is saturated.
  • Try to avoid using views for data you’re constantly accessing. If you can afford it, create temporary tables and insert data there. This will eliminate the overhead imposed by the view evaluation.
  • Avoid sub-queries since they tend to create temporary tables and insertion of the intermediate results into those tables may be expensive.
  • Try to use indices only on static data or data that changes rarely. Building an index on live or temporary data can be expensive performance-wise. Only do so if the time required for the data lookup considerably outweights the time required for building the index.
  • Alternative to indices: hashkeys – Instead of using indices on very long strings, you may store the hash values of those strings as keys in the same table. A lookup via hash values may be a whole lot more efficient. This method is also very effective when you can’t afford the creation of an index due to performance reasons. Downside: You have to take care of the hashkeys. (See remarks in the comments below.)
  • No useless indices. Create indices only if your queries actually use the indices on the table (check with EXPLAIN). Having useless indices around may pollute otherwise precious database cache space.
  • Be cache-friendly. Depending on the memory conditions, creating temporary tables and indices may bash the cache. Reloading data back into the cache is expensive.
  • Double-check your queries and profile them. The SQLite optimizer doesn’t perform as well as the optimizers of big DBs (Firebird / Interbase, PostgreSQL, Oracle etc.).
  • Check compiler settings. A higher optimization setting in your C-compiler may very well yield a few tens of milliseconds. Make sure to inline functions (-O3 for GCC 2.95.x, -O2 for GCC 3.x.x and higher). Optimize for architecture and CPU. Omit stack frame pointers (-fomit-frame-pointer) if you’re not producing executables with debug symbols. This may free an additional register for the compiler to use.
  • Disable unused SQLite features. This helps to reduce binary size and may also affect performance.

Here are some additional docs to consider:

http://www.sqlite.org/cvstrac/wiki?p=PerformanceTuning
http://www.sqlite.org/cvstrac/wiki?p=PerformanceTuningWindows
http://www.sqlite.org/cvstrac/wiki?p=PerformanceConsiderations
http://www.sqlite.org/optoverview.html
http://web.utk.edu/~jplyon/sqlite/SQLite_optimization_FAQ.html

“Yet Another Zaurus Media Player”… done differently . (Phase 2: Development progress, no release yet.)

Posted on by Andre Beckedorf

So, like I’ve already mentioned in my previous comment, I’ve got some free time to work on my pet project here.

Development progress

I’ve been optimizing a lot under the hood. Tons of blood, sweat and tears have already run into optimizing the core parts.
Coming from a different background in programming, namely a desktop background, doing embedded development is a whole new experience for me. And let me say this: it’s definitely a refreshing one.
Development for embedded devices can be quite challenging if you have hard memory limitations and performance restrictions CPU-wise. These limitations go even further than the ones I’m used to when doing component or graphics development. And I’m doing quite a lot of that…

Just so you get the idea:
My essential requirement for this project is that the player is able to cope with thousands of files in a playlist.

With that being said, I’ve already rewritten the playlist management four times. :)
The first approach was fast but ate RAM for breakfast. Incremental searching on a playlist was fast but also required additional memory. The second approach was more memory-friendly, but searching was slow. Besides, some Qt widgets make development a real pain – at least in Qt/E 2.3.x. For instance, QListView can pose an incredible hog on performance. I’m currently using several hacks to speed things up. However, I’m still thinking about replacing the whole component or doing some custom coding to improve it…

Anyway, since I couldn’t really get rid of the memory problems, I finally decided to give SQLite a try. SQLite offers very sophisticated caching, which helps getting rid of the RAM problem. I really could use the enhanced features of a SQL database. And let me say this: SQLite is awesome. And it’s as fast as it could be on such a small device – that is, if you know how to use it…

With that being said, different rules in database design apply for embedded systems:
In the third approach I already created a pretty decent database schema. Something I naturally would have done on a desktop system. Keeping the layout clean, using relations where applicable, minimizing data storage requirements.
On a desktop system dereferencing and joining tables is fast. However, not so on my Zaurus: Simple left-joins over three tables would take up to a few hundred milliseconds. In contrast, these queries are almost unmeasurable on my desktop system, meaning they were faster than 10 ms.
Now add a few other equally expensive queries to that and imagine, you’re doing a search on your playlist with 2000 items. Do you want to wait 3 seconds or longer for the result? That’s not what I call interactive.
So, I had a nice profiling, optimizing and testing marathon last weekend. To make a long story short, after analyzing the bottlenecks and also having a lengthy discussion with a DB-guru friend, I ended up simplifying the database schema in a direction I wouldn’t normally take on a desktop system. It’s not totally ugly now, but it’s just not as relational as you might expect a SQL database to be. Also, some data is redundantly held in temporary tables, which isn’t nice either, but helps performance A LOT.
In order to do the profiling I made some changes to the SQLite codebase, which I will post shortly along with some optimization hints. Update: Hints here, patch here.

Screenshots

No release yet, sorry! I have to finalize some features first.
However, here are some new screenshots that show the new overview feature in action. The design of the application is temporary, stay tuned! :)

{YAZMPDD} YAZMPDD - Work In Progress Screenshot 4: Overview feature with multi-selection in action.{YAZMPDD} YAZMPDD - Work In Progress Screenshot 5: Search filter + Overview filter{YAZMPDD} YAZMPDD - Work In Progress Screenshot 6: Portrait window mode. Note: This is the contrast skin, which will change in the future.

“Yet Another Zaurus Media Player”… done differently . (Phase 1: Teasing)

Posted on by Andre Beckedorf

{YAZMPDD} YAZMPDD - Work In Progress Screenshot 1: Whole library loaded. Contrast mode, perfect for car usage.{YAZMPDD} YAZMPDD - Work In Progress Screenshot 2: Playing a file...{YAZMPDD} YAZMPDD - Work In Progress Screenshot 3: Library view filtered...

Tell me what you think. I won’t comment on anything though. This is work in progress – it’s not finished. I won’t publish any additional features besides what you can read below.

Here is the information I can give right now:
Yes, this is derived from ZPlayer, however, the core is different. Right now, please don’t ask any question why I’m not contributing to Kino or ZPlayer. This is a whole different thing here and I’ll comment once I release.

Does it handle large amounts of music? YES.
Are you serious about the *Tunes look? YES.
Is that fullscreen? Yes.
What’s the name? No title yet. If you have a good one, don’t hesitate to mail me.

In the meantime wait for more information and the next phase… :)

Installing OS X 10.4 Tiger (PowerPC) from a Firewire harddrive

Posted on by Andre Beckedorf

Last weekend I had the pleasure (!) to install Tiger on an ancient PowerMac that doesn’t feature any DVD-ROM drive. I don’t happen to have the special CD-version of 10.4 and trying to boot from an external DVD drive somehow failed. However, this Mac already had Firewire, so here is a little hint on how to install Tiger using a spare Firewire drive:

First off, you’ll need psync. I’ve tried Carbon Copy Cloner, but for some reason that didn’t work.
So, if you have Fink installed, do this:

$ fink install psync

Format the Firewire drive with HFS Extended. You don’t need Journaling, so please disable it.
Now, assuming you’re Tiger Install DVD is mounted at /Volumes/Mac OS X Install DVD and your formatted volume is mounted at /Volumes/OSX, type this into your Terminal to clone the content of the DVD over to the harddrive.

$ sudo psync -d "/Volumes/Mac OS X Install DVD/" /Volumes/OSX

Finally make the whole copy bootable by blessing it:

$ sudo bless --folder "/Volumes/OSX/System/Library/CoreServices" --bootinfo

Now, you may unmount and install. Fin .

Zaurus ARM Cross Compiler on OS X (Intel)

Posted on by Andre Beckedorf

Andreas Junghans has an excellent tutorial on how to set up a Zaurus ARM cross compiler on OS X. However, there is one problem: It will only compile on PowerPC.
So, here is my patchset for compiling on Intel:

gcc-patches.tgz

Use this file instead of the one offered on his page and just follow his instructions.
The fix is really simple in nature: I’ve just added one file, namely xm-openstep.h in gcc/config/i386. This does the trick.

BDS / Delphi alike key bindings for Xcode

Posted on by Andre Beckedorf

Lately, I’ve been using Apple’s Xcode to work on some C++ and ObjC code. The default key bindings in Xcode are really annoying if you come from a Borland background.
Luckily, there is a way to adjust these directly in Xcode. So, here is my first version of the Borland Developer Studio / Delphi / JBuilder / C++ Builder alike key bindings:

Xcode – BDS Similar.pbxkeys.zip

Simply place the included file “BDS Similar.pbxkeys” into:
~/Library/Application Support/Xcode/Key Bindings/
and activate it in the Xcode preferences.

Not every shortcut is available, but at least the basics are there. I see if I can improve it over time.

Apple Cube Biohazard graphics card mod

Posted on by Andre Beckedorf

Since I’m in the mood to document various hardware modifications, here is a link for the archives:
http://www.katastrophos.net/macosx/9000pro-mod.html
This is a tutorial written in German for a mod I did way back. It describes how to modify the case of an Apple Cube to actually insert a better graphics card along with proper cooling of the components.
Much of the article deals with how to relocate the DC/DC board to the inside of the Cube to make way for the bigger graphics card. It goes on to describe how to modify the graphics card with a better heatsink. Finally, it shows how to cut the outer casing for proper cooling.

You might ask, why this is called Biohazard mod. Check out this and that.