Running the SailfishOS Build engine outside of a VirtualBox VM

I have been struggling with the SailfishOS Build Engine VM for quite some time. 4 out of 5 SailfishOS builds of QuasarMX would fail because the Shared Folder (vboxsf) mechanism in VirtualBox at some point starts to corrupt files on my system, even though the system’s RAM and storage are perfectly fine. Since QuasarMX heavily relies on qmlpp (see previous post) to work with QtQuick 2, a lot of files will be touched during compilation. For this, vboxsf is both unreliable and slow (!).

So, here is a short tutorial that will show how to run the SailfishOS Build engine outside of the VirtualBox VM in your host’s Linux installation. This guide is geared towards Ubuntu 12.04, but might work on other systems too.

UPDATE 1: New version with root check, private namespace and permission fixes in SB2 targets.
UPDATE 2: The tutorial has been updated to work with SDK Alpha-1404 (Alpha 4). This new version does not work with Alpha-1312 (Alpha 3).
UPDATE 3: This method is working with SDK Beta-1509.

Start the Build engine VM and make sure you can connect to it via SSH (see Developer FAQ for details):
$SUDO_USER is the username you used to sudo into root privileges, change it if you have not used sudo to become root.

ssh -p 2222 -i /home/$SUDO_USER/SailfishOS/vmshare/ssh/private_keys/engine/root root@localhost

If it works:

exit

Now, become root:

sudo -i

Create a directory to host the rootfs of the SailfishOS Build engine:

mkdir /opt/sailfishos-buildengine

Use rsync to copy the rootfs to our newly created directory:

rsync --numeric-ids -xazuv -e "ssh -p 2222 -i /home/$SUDO_USER/SailfishOS/vmshare/ssh/private_keys/engine/root" root@localhost:/ /opt/sailfishos-buildengine

This will take a while. Go, have dinner or get a cup of tea.

Save the following script as file “run-sailfishos-buildengine.sh” somewhere you like. We’ll use /opt/run-sailfishos-buildengine.sh
Make sure to edit the first few lines to match your environment, SDK=/opt/SailfishOS is the directory you used to install the Sailfish SDK on your host. This probably is /home/YOURUSERNAME/SailfishOS if you haven’t installed as root.

#!/bin/bash
USER=$SUDO_USER
USERDIR=/home/$USER
SDK=/opt/SailfishOS
BUILDENGINE=/opt/sailfishos-buildengine

if [ $EUID -ne 0 ]; then
	echo "This script must be run as root." 1>&2
	exit 1
fi

if cmp -s /proc/$PPID/mountinfo /proc/self/mountinfo; then
	exec unshare -m -- "$0" "$@"
	echo "$0 must be run in private namespace."
	exit 1
fi
		
cleanup()
{
	umount -l $BUILDENGINE/dev/pts
	umount -l $BUILDENGINE/dev
	umount -l $BUILDENGINE/proc
	umount -l $BUILDENGINE/sys
	umount -l $BUILDENGINE/run

	umount -l $BUILDENGINE/home/mersdk/share	
	umount -l $BUILDENGINE/home/src1
	umount -l $BUILDENGINE/etc/ssh/authorized_keys
	umount -l $BUILDENGINE/host_targets
	umount -l $BUILDENGINE/etc/mersdk/share
}

trap "cleanup > /dev/null 2>&1; exit" INT QUIT TERM EXIT

mount --make-slave "$(df -P "$BUILDENGINE" | tail -1 | awk '{print $NF}')"

mount --bind /dev "$BUILDENGINE/dev"
mount --bind /dev/pts "$BUILDENGINE/dev/pts"
mount --bind /proc "$BUILDENGINE/proc"
mount --bind /sys "$BUILDENGINE/sys"
mount --bind /run "$BUILDENGINE/run"

mount --bind "$USERDIR" "$BUILDENGINE/home/mersdk/share"
mount --bind "$USERDIR" "$BUILDENGINE/home/src1"
mount --bind "$SDK/mersdk/ssh" "$BUILDENGINE/etc/ssh/authorized_keys"
mount --bind "$SDK/mersdk/targets" "$BUILDENGINE/host_targets"
mount --bind "$SDK/vmshare" "$BUILDENGINE/etc/mersdk/share"

# Rewrite user id and group id entries to match host system's users
updateConfigs()
{
	sed -i -e "s/$1:x:$2:100000:/$1:x:$(id -u $USER):$(id -g $USER):/" "$3/passwd"
	sed -i -e "s/$1:x:100000:/$1:x:$(id -g $USER):/" "$3/group"
	cp /etc/resolv.conf "$3/"
}

updateConfigs mersdk 1001 "$BUILDENGINE/etc"
updateConfigs nemo 100000 "$BUILDENGINE/srv/mer/targets/SailfishOS-armv7hl/etc"
updateConfigs nemo 100000 "$BUILDENGINE/srv/mer/targets/SailfishOS-i486/etc"

cat > "$BUILDENGINE/runmersdkengine.sh" << CONTENT
#!/bin/bash

cleanup()
{
	pkill -KILL -f "/usr/bin/ruby /usr/bin/puma -p 8080 -t 1:1 -e production"
}
trap "cleanup; exit" INT QUIT TERM EXIT

# Fix up permissions to point to new uid/gid of mersdk
mkdir -p /home/deploy/installroot
chown -R mersdk.mersdk /home/deploy/installroot
chown -R mersdk.mersdk /srv/mer/targets/*
find /home/mersdk | grep -v "/home/mersdk/share" | xargs chown mersdk.mersdk

# Run web server of build engine as mersdk in background:
su mersdk -c "cd /usr/lib/sdk-webapp-bundle; /usr/bin/ruby /usr/bin/puma -p 8080 -t 1:1 -e production" &

# Run SSH Server of build engine as root:
/usr/sbin/sshd -p 2222 -D -e -f /etc/ssh/sshd_config_engine
CONTENT

chroot "$BUILDENGINE" sh /runmersdkengine.sh 

Don’t forget to

chmod 755 /opt/run-sailfishos-buildengine.sh

Now exit the root shell and run:

sudo /opt/run-sailfishos-buildengine.sh

This will start the Open SSH server and the build engine’s web server which is used in the modified Qt Creator of the SailfishOS SDK.
Press CTRL+C to terminate the engine at any time.

Qt Creator will complain that the VM is not started and offers to start the VM for you. Just click No and ignore it. The build is already running in the background.

Enjoy your fast builds!

13 thoughts on “Running the SailfishOS Build engine outside of a VirtualBox VM

  1. Thank you – you have scratched two itches for me!
    I am a) still using and loving Quasar on my Zaurus and b) quite annoyed with the huge latency of the SailfishOS build process.
    Looking forward to QuasarMX on my Jolla and faster build processes!

  2. Updated the script to fix some permission issues in the SB2 chroot env, namely when installing new packages with zypper or via the web frontend.

    @Raphael Cool and thanks!

  3. Hi! Thanks for your nice work!

    I’m having a problem though, when trying to build, I get
    PAM: pam_open_session(): Cannot make/remove an entry for the specified session
    after the key has been accepted by sshd, and the build won’t work.

    Any ideas what could cause that?

  4. Hey again,

    got it working by disabling PAM from $BUILDENGINE/etc/ssh/sshd_config_engine

    and had to unlock mersdk user by logging in as root and running
    passwd -u mersdk; passwd mersdk

    now deploying by building rpm is working, although deploying to device needs something more..

    Thanks anyways, now I can build even on my mininotebook!

  5. @Acce Cool! Did you prepare and set up the VM before setting up the chroot? I didn’t try with a fresh VM yet, so perhaps that is where these issues come from?

  6. @Andre hmm, I had tried building with it before, but I probably didn’t update it to the newest merSDK after installing.. maybe updating will help, let’s see!

  7. got it working now, first, the qt creator doesn’t seem to like it if one has defined multiple mer arm devices.. at least I couldn’t find how to choose the deployment target device. So I reduced them to one ARM device only.

    Second, the keys generated for the device were with too “open permissions” according to ssh, and wouldn’t work.
    chmod 600 ‘path/to/key/file’
    fixed that, and I’m able to deploy to the device now.

    Many thanks and sorry for spamming comments :)

  8. Hi Andre,

    Thanks very much for this good article.

    But i have hit a roadblock.

    I am able to run Open SSH server successfully.

    But when i Run my code from QT, it prompts for starting “MerSDK”, which i decline.

    Afterwards it prompts for “Sailfish” emulator, which i accepts.

    But i get an error, timeout for emulator, no reply from server, i am unable to deploy my apps on emulator.

    Can you pls help me out

  9. Thank you for this howto. However, for some reason the rsync operation seems to get stuck at some point and I cannot finish the copy.



    usr/lib/python2.7/site-packages/Cheetah/Templates/_SkeletonPage.py
    usr/lib/python2.7/site-packages/Cheetah/Templates/_SkeletonPage.pyc
    usr/lib/python2.7/site-packages/Cheetah/Templates/_SkeletonPage.pyo
    usr/lib/python2.7/site-packages/Cheetah/Templates/__init__.py
    usr/lib/python2.7/site-packages/Cheetah/Templates/__init__.pyc
    usr/lib/python2.7/site-packages/Cheetah/Templates/__init__.pyo
    usr/lib/python2.7/site-packages/Cheetah/Tests/
    usr/lib/python2.7/site-packages/Cheetah/Tests/Analyzer.py
    usr/lib/python2.7/site-packages/Cheetah/Tests/Analyzer.pyc
    usr/lib/python2.7/site-packages/Cheetah/Tests/Analyzer.pyo
    usr/lib/python2.7/site-packages/Cheetah/Tests/CheetahWrapper.py
    usr/lib/python2.7/site-packages/Cheetah/Tests/CheetahWrapper.pyc
    usr/lib/python2.7/site-packages/Cheetah/Tests/CheetahWrapper.pyo

    …and it won’t move no matter how long I wait.

    Any advice would be greatly appreciated, thank you!

  10. Nevermind, I run the rsync a few more times and it finally finished :)

    When updating the build environment you might want to add the –delete option to the rsync command.

    Still have a small issue:

    $ sudo /opt/run-sailfishos-buildengine.sh
    chown: `mersdk.mersdk’: invalid user
    chown: `mersdk.mersdk’: invalid user
    chown: `mersdk.mersdk’: invalid user
    su: user mersdk does not exist
    Server listening on 0.0.0.0 port 2222.
    Server listening on :: port 2222.

    Tried to add the user but no joy -any thoughts?

Leave a Reply

Your email address will not be published. Required fields are marked *

*

This site uses Akismet to reduce spam. Learn how your comment data is processed.