I am pleased to announce beta 4 of Quasar Media Player.
This new version introduces some new features and big improvements in terms of performance and memory-usage.
The previously separate last.fm Audioscrobbler QScrobbler has been integrated into Quasar as add-on.
This release also marks the introduction of the Cover Art Downloader which uses the new open-source katastrophos.net Cover Art search engine to download cover art images for the new Cover Art Flow album browser.

Binaries for the previous platforms (Sharp Qtopia and pdaXrom) along with new binaries for Windows and OS X (universal) are available on the project’s homepage.

It’s been some time since the last update. In my previous post I mentioned I was in the process of setting up a nightly build system. This system has been running silently since May.
I guess it is finally time for me to officially announce the nightly builds of Quasar Media Player:

http://www.katastrophos.net/downloads/quasar/nightly/

Along with the most current sourcecode tarball, binaries are available for 4 platforms:
Windows (win32), OS X (universal binary), Sharp Qtopia and pdaXrom Linux (both for Zaurus PDA)
The Windows version comes in two styles: a generic setup (EXE) and a self-contained portable version (7z archive)

Enjoy and please leave a comment.

First step is to install Wine either as super user or by using sudo:

sudo apt-get install wine

As normal user fire up your X server and your favourite terminal application and get the latest Inno Setup QuickStart pack:

mkdir /tmp/innosetup
cd /tmp/innosetup
wget http://files.jrsoftware.org/ispack/ispack-5.2.3.exe
wine ./ispack-5.2.3.exe

This will start the installer in Wine. Note, for the installation you will need a running X11 server since the installer obviously is graphical. If you have not set up Wine before, the default location C:\Program Files\Inno Setup 5 will install to ~/.wine/drive_c/Programme/Inno Setup 5.

Luckily the Inno Setup compiler offers a command line interface, ISCC.exe, which will run in Wine without the necessity of having a X server running. So it is ideally suited for automated runs.

Here is a simple wrapper shell script called iscc:

#!/bin/sh
unset DISPLAY
scriptname=$1
[ -f "$scriptname" ] && scriptname=$(winepath -w "$scriptname")
wine "C:\Program Files\Inno Setup 5\ISCC.exe" "$scriptname" "$2" "$3" "$4" "$5" "$6" "$7" "$8" "$9"

I installed this script in my local bin directory (~/bin) and added it to the PATH environment variable.
This will allow running the Inno Setup compiler from anywhere and it also makes it very easy to integrate into a build script. You can even feed a script via stdin, e.g. something like:

iscc - < ./myscript.iss

void QProcess::writeToStdin( const QString& buf )
{
    QByteArray tmp = buf.local8Bit();
    tmp.resize( tmp.size() - 1 ); // drop the implicit \0
    writeToStdin( tmp );
}

should be:

void QProcess::writeToStdin( const QString& buf )
{
    QByteArray tmp = buf.local8Bit();
    tmp.resize( buf.length() );
    writeToStdin( tmp );
}

Verision 3.3.7-7 includes the latter method, same as the latest Qt 4.3 sources. So, it’s actually a regression in 3.3.x-8. If you require the latest Qt 3 / Q…/Free for your open source application and need to write to some other processes’ stdin, you can just use a wrapper workaround that uses the latter method and directly uses writeToStdin( const QByteArray& buf ) instead of the QString variant.

Today I’m releasing QScrobbler: a Last.fm / Audioscrobbler add-on for Quasar Media Player.

As with Quasar, I’ve been using QScrobbler for almost a year now and finally decided it is ready for the public. ;)
For more details please visit the project’s homepage here.

Finally! Almost a year after the first mentioning of my new media player for the Sharp Zaurus and after several development hiati, I’m today officially releasing the Quasar Media Player for SharpROM- and pdaXrom-based distributions.
For more details please visit the project’s homepage here.

So, here is a short update on the media player that I’m currently developing for my Zaurus. Well, actually it’s been in long-term testing mode for ages now… :)
I finally have a name for it. It will be called “Quasar Media Player” – or shorter “Quasar”. Below are some screenshots of the current development version running on Qtopia. I hope to have a release ready soon.

Again, I’ve been working on improving performance – this time on the performance when loading playlists.
Before I continue, let me give a brief overview of the structure:

Library -> Playlists < -> Media Cache

YAZMP doesn’t manage a library similar to i*Tunes. Instead it solely relies on playlists. Metadata (title, artist, album, etc.) is kept in the database and will be associated to once the playlist is loaded. The reason for this is pretty simple: Scanning audio files (and media files in general) each and every time a playlist is loaded will definitely take a lot of time. So, for every file the gathered metadata will be saved in the DB. Think of it as a cache.

ZPlayer, which YAZMP is based on, follows a similar approach. However, in my opinion it’s ill-conceived in the current version. It keeps the media cache in memory all the time. With lots of files this can get very expensive. Also, ZPlayer isn’t very efficient at saving the media cache to disk. It does so for the whole cache, not just changed entries. Furthermore, when saving the cache on quitting, ZPlayer sometimes simply truncates the cache file and all metadata is lost. So, next time you open it, it has to scan all files again…
Perhaps I’m nitpicking, perhaps I’m an optimization dork, whatever it is, please forgive my harsh words for the otherwise excellent work atty and Atmark have done in ZPlayer!

Technical ramblings

For YAZMP I rewrote the metadata cache management from scratch and extended it to do a lot more.
As I already mentioned above, YAZMP doesn’t manage any library, it only knows about filenames in playlist files.
For that to work correctly and efficient especially in combination with the media cache, there are four cases that need to be taken care of:

1. A new file has been added to the playlist which doesn’t exist in the media cache.
   This case is very simple: The player has to scan the file no matter what.
2. A file has been physically removed and it’s metadata is still in the cache.
   In this case the player has junk entries in its database, which need to be cleaned once in a while.
3. A file has been renamed, that is, it’s basename changed
4. A file or a whole directory / sub-tree of files have moved to another location.
   In a naive implementation both cases would trigger a lengthy rescan of the file(s) that are about to be added in the playlist, because new location can’t be associated with any data in the cache.
   For YAZMP I’ll use several approaches + fallbacks to try to limit the impact. The following messy pseudo-code illustrates this:

     get attributes filename, directory  from current playlist item to be added
     get attributes filesize, lastmoddate for file (directory + filename)
   
     // NOTE: check via checksum / hash for faster lookup
     if exactly one entry by hash value of (directory + filename) in DB then
       // NOTE: did the file change since the last scan?
       if entry.filesize not equal filesize or entry.lastmoddate not equal lastmoddate in FS then
         rescan file (directory + filename) and update entry in DB
       return entry
     else if exactly one entry by tuple of (filename, filesize, lastmoddate) in DB then
       // Was the file moved to a new directory? (case 4)
       // If so, check for other files in the old directory and change them
       // to the new location...
       // NOTE: this is purely predictive. It might not always yield a benefit.
       if directory by value of entry.location not in FS then
         change every siblingentry in DB where siblingentry.location is entry.location
           set siblingentry.location to directory
           set siblingentry.locationHash to hash value of (directory + siblingentry.filename)
         until no siblingentry is left
       // NOTE: did the file change since the last scan?
       if entry.filesize not equal filesize or entry.lastmoddate not equal lastmoddate in FS then
         rescan file (directory + filename) and update entry in DB
       return entry
     // NOTE: For case 3 one could check for a smaller tuple of (filesize, lastmoddate)
     // which would result in the almost the same code as above
     // The likeness of wrong results is a lot higher though.
     else
       scan file (directory + filename) as newentry and add newentry to DB
       return newentry
   

Conclusion

What that means is pretty simple: You can move your media files to a different location, perhaps a SD-Card, and the player will still be able to associate the new location in the playlist file with the data it has in cache. And that means: No useless scanning of files and fast playlist load times.

Yet another feature…

I tend to copy several albums to my Zaurus for listening on the road. So, once I’ve copied new stuff, it’s always the same mindless and inconvenient procedure in most players: Clear the playlist, add files or directories, play stuff.
And that brings me straight to another new feature: Dynamic Playlists
In general the idea is pretty simple: Instead of just listing every file’s location in a playlist file, one may also list folders that are automatically scanned for audio or video files. Here is an example:

Music.dynplaylist

/hdd3/Music/test1.mp3
/mnt/card/Music/Mozart/
/hdd3/Music/Beethoven/**

This will add a single file “test1.mp3″, all files in “/mnt/card/Music/Mozart/” and all files in “/hdd3/Music/Beethoven/” including all files in its sub-directories.

Conclusion

Instead of adding files, you just define what the layout of the playlist should look like and the player does the rest automatically. It can even rescan for changes. It’s all wrapped up in a nice UI.
Combined with the flexibility of the metadata cache and the fact that one isn’t forced into this library thing, I think this makes for a pretty unique solution. What do you think?

As a follow-up to my previous post “SQLite performance tuning and optimization on embedded systems” here is a very basic patch that introduces support for timing SQL queries in the sqlite3 console shell.

SQLite 3.3.8:
Patch against shell.c shell.c

SQLite 3.3.9 CVS – shell.c rev 1.157:
Patch against shell.c shell.c

Two new shell commands are introduced in sqlite3:

.profile ON|OFF        Turn profiling on or off
.timer start|show      Measure elapsed CPU time

.profile ON will turn off output to stdout. Instead it will display CPU execution time for every SQL command processed.
Here is an example output:

sqlite> .profile on
sqlite> .read test1.sql
Exec time  BEGIN;                                       :  0.000 s.
Exec time  DROP TABLE IF EXISTS playlist_view;          :  0.020 s.
Exec time  DROP TABLE IF EXISTS overview;               :  0.000 s.
Exec time  CREATE TEMPORARY TABLE playlist_view AS   ...  :  0.180 s.
Exec time  CREATE TEMPORARY TABLE overview AS        ...  :  0.140 s.
Exec time  SELECT DISTINCT genre, genre IN ("Progress...  :  0.030 s.
...

Note: You can still override the output setting with the .output command after an .profile ON has been issued.
.profile OFF will turn off time-based profiling and reenable output to stdout.

.timer start will start a timer. You can execute a sequence of queries and then print the required CPU time via .timer show.
Use .timer start again to reset the timer. This .timer patch is based on SQLite ticket #1227 by Bartosz Polednia.

Timing is currently done using clock() which provides only a 10 ms precision for CPU time on most systems. This might be inadequate depending on your requirements. If you know a more precise way, please let me know.

• Simplify the database schema as much as possible – even if that means redundant data or illogical structure
• Don’t generalize the database schema – generalization will mostly sacrifice performance and one can’t afford that on an embedded system with its tight restrictions, even if it is more convenient for the developer.
• Only use relations (via IDs etc.) where absolutely necessary. The overhead for lookup and joining tables is considerable, even with an index on the relation.
• Order the tables correctly in SELECTs. Put a table left-most if it is lacking an index on the relation. More details are here.
  In general: Check the order of tables in the SELECT statement. A different permutation may be more optimal. Profile.
• Prepare your statements and bind values where applicable. This way you can get rid of the parser and VM creation overhead in tight loops (e.g. when inserting and updating).
• Use transactions – even if you’re just reading the data. This may yield a few milliseconds.
• Use temporary tables for intermediate results. They are fast and stay in cache most of the time. Depending on how your SQLite instance is set up, data will only be swapped into an external file if the cache is saturated.
• Try to avoid using views for data you’re constantly accessing. If you can afford it, create temporary tables and insert data there. This will eliminate the overhead imposed by the view evaluation.
• Avoid sub-queries since they tend to create temporary tables and insertion of the intermediate results into those tables may be expensive.
• Try to use indices only on static data or data that changes rarely. Building an index on live or temporary data can be expensive performance-wise. Only do so if the time required for the data lookup considerably outweights the time required for building the index.
• Alternative to indices: hashkeys – Instead of using indices on very long strings, you may store the hash values of those strings as keys in the same table. A lookup via hash values may be a whole lot more efficient. This method is also very effective when you can’t afford the creation of an index due to performance reasons. Downside: You have to take care of the hashkeys. (See remarks in the comments below.)
• No useless indices. Create indices only if your queries actually use the indices on the table (check with EXPLAIN). Having useless indices around may pollute otherwise precious database cache space.
• Be cache-friendly. Depending on the memory conditions, creating temporary tables and indices may bash the cache. Reloading data back into the cache is expensive.
• Double-check your queries and profile them. The SQLite optimizer doesn’t perform as well as the optimizers of big DBs (Firebird / Interbase, PostgreSQL, Oracle etc.).
• Check compiler settings. A higher optimization setting in your C-compiler may very well yield a few tens of milliseconds. Make sure to inline functions (-O3 for GCC 2.95.x, -O2 for GCC 3.x.x and higher). Optimize for architecture and CPU. Omit stack frame pointers (-fomit-frame-pointer) if you’re not producing executables with debug symbols. This may free an additional register for the compiler to use.
• Disable unused SQLite features. This helps to reduce binary size and may also affect performance.

Here are some additional docs to consider:

http://www.sqlite.org/cvstrac/wiki?p=PerformanceTuning
http://www.sqlite.org/cvstrac/wiki?p=PerformanceTuningWindows
http://www.sqlite.org/cvstrac/wiki?p=PerformanceConsiderations
http://www.sqlite.org/optoverview.html
http://web.utk.edu/~jplyon/sqlite/SQLite_optimization_FAQ.html