[OpenWrt] Porting PirateBox/LibraryBox to BarrierBreaker
Posted by Matthias
This forum is currently read only. You can not log in or make any changes. This is a temporary situation.
Now, this forum is in read-only mode. You find details Details hereContinue on /r/PirateBox
|
[OpenWrt] Porting PirateBox/LibraryBox to BarrierBreaker November 05, 2014 10:04AM |
Admin Registered: 13 years ago Posts: 4,680 |
Hello everyone,
as you might noticed, the OpenWrt team released a new stable version of their OpenWrt distribution. The new release is called "BarrierBreaker" and comes with a wide band of new devices supported and some other interesting things. The PirateBox images are all based upon AttitudeAdjustment and we ported back some device support, which could be done easily.
The PirateBox scripts+packages for OpenWrt do not support BarrierBreaker and above due to one fact: The changed startup behavior of OpenWrt. Only a few things of our scripts works with that new startup behavior and daemons we do not directly start with the PirateBox scripts - like minidlna - stopped working.
With the BarrierBreaker release, the maintainers released a new software called "procd" taking care of the startup handling.
Technically spoken, it is a background daemon which is informed via dbus what to do. It it speaks JSON and starts & stop stuff. As far as my investigation go, I figured out that procd doesn't "know" of the filesystem we have on the USB Stick.
All new init.d scripts - the exsting ones are being replaced by a new style continuously - don't launch the software for them self, they delegate the startup to procd using dbus and procd starts it from the central running daemon.
Into the details:
The common known technique to raise the size of the root filesystem is called "extroot", where you exchange the modify-able filesystem on the hardware is a special formatted USB stick. .. Well, that USB stick needs to be ext3 formatted. This technique still works on BarrierBreaker and is improved and maintained by the OpenWrt developers.
In the following lines, I don't differenciate between a copied extroot or an overlays_fs for extroot. The last one only saves the differences, the first one the complete content is copied. Both techniques are relying on a linux-only filesystem, which is the important fact (read more below).
PirateBox and LibraryBox are using a the technique I call "extendRoot". - That technique is different, that we can use a normal FAT32 formatted USB stick and place a FILE on the USB-Stick. That file is preformatted with ext3 and acts like an CD-iso image-file, which you can modify. The image is mount to /mnt/ext and linked to /usr/local .
/usr/local is per design a normal location to install software to and we add this installation destination to opkg.conf, that you can use '-d ext' to install to the USB Stick.
The problem is, that /usr/local is not included in the search path for Library-files and for executable. Some executables need to be located on special locations, because the PATH variable is not used to locate them.
My work is, that I created a bunch of scripts, that include the new search locations for binaries and libraries to bootup procedure and stuff that is common across a lot files.
After this, I check where I have to do modifications, that software runs off the USB stick. This is for PHP the following
- I create static symlinks pointing from i.e. /usr/bin/php-cgi to /usr/local/usr/bin/php-cgi
- Make the php.ini file available in /etc with a symlink to the /usr/local/etc/php.ini file
- Modify the search path for module-lookups in the php.ini file to /usr/local/usr/lib/php
And other modifications like this. All these things you can find in the i.e. extendRoot-php package, that is installed.
As the PirateBox scripts are installed to /usr/local as well, I have a symlink from /etc/init.d/piratebox to /usr/local/etc/init.d/piratebox
The pitfall here is, that before the USB stick and the image file mounted, all the symlinks are broken. As extendRoot is directly installed on the root filesystem, there is an init-file called /etc/init.d/ext , which is started right after the USB drivers are loaded. This init-file mounts the USB stick (if not happened by hotplug-fstab service), after this the Filesystem-image for /usr/local is mounted (which isn't done by the hotplug fstab service) and swap-files located on the USB stick, too.
You still can install the PirateBox stuff on a Barrier-Breaker OpenWrt, but you will encounter issues like a not working startup. Why?
a) Directly compared between normal init and procd, it seems that procd does not recognize symlinks which are broken, while the normal init still recognize them and try to execute these
b) The procd seems to have hardcoded lookups for executables and libraries, which I can't extend
For PirateBox directly (DHCP, DNS, Webserver and Droopy) I can fix it with creating a copy of the file /etc/init.d/piratebox instead of creating a symlink. That works especially, because the PirateBox init.d script does it the old way with starting each process for themselves and does not delegate it to procd.
The UPNP-server minidlna does not work because the init.d script already delegates to procd, which can't find the libraries needed or the executable.
Puh, lots of technical stuff.
I'm posting this, because I want to discuss how we should proceed in this topic.
How we can make PirateBox possible on BreakingBarrier and small devices too?
What options are possible?
I currently know these options:
Fully ext-root way
Move over to the normal ext-root situation with having a root filesystem on the USB natively. The USB stick needs to be formatted in ext3 and we install all software to the root filesystem.
Sure this needs a sort of changes scripts and tests, but that would be the OpenWrt way to go and we are able to throw away a lot of stuff and work arounds. Sure that hurts, but it will work much smoother later on.
I already tried these technique on a different project and it works quite handy.
The positive side of switching the techniques:
- More compatible way for new software & is easy to extend with new software (no more linking required)
- Should compatible even with newer versions of OpenWrt
The flaws of this techniques are a lot compared to our current one:
- ext3 is for a non-IT guy unhandy. It goes so far that I need to format the data partition on SD-Card of the LibraryBox-RapsberryPi-Image to vfat that people can plug it in every computer. MacOS users may use userspace filesystems, but Windows users are lost and can't copy files to the USB stick without using the upload button or difficult other methods. This is a major flaw compared to our current solution, which is quite handy for everyone.
- If you want to move to a bigger USB stick, you have to do it in a "correct" way to preserve file permissions. If you don't you are doomed.
- It is not that easiy to prepare an existing FAT32 usb stick "on the fly" to a ext3 usb stick. ; The problem is that fsck.vfat needs to be installed to image and/or to tmp
ext-root half way ; image
Another solution can be, that we create an image like SDcard images, that needs to be copied with dd to the USB-stick.
That image contains nn MB ext3 partition in the front of the USB stick and a prepared vfat partition. The vfat partition is extended during the first bootup (or a special command) to maximum size of the USB stick.
Positive side of this:
- So data are still easily accessible via a vFAT partition
- we can use the advantages of the ext-root function of openwrt
Negative side:
- A bit tricky for windows users to "flash" that image file to USB stick ; they don't have DD
- complicated to exchange the USB stick, the medium you are using
ext-root half way ; pepare USB on its own.
This solution is like the half-way image solution, but during the setup, the USB stick is prepared by an installer.
It setups an ext3 partition and the beginning or the end of the USB stick.
The important thing here is, that the installer works in two or three steps:
1. Install Partition and filsystem utilities into the ram of the running system
2. Repartition the USB stick
3. Indicate that the USB stick is finished; the user has to put the folder again on the usb stick, because the fat filesystem was removed/cleaned out
4. Run the next installation with installing software.
Positive side of this:
- So data are still easily accessible via a vFAT partition
- we can use the advantages of the ext-root function of openwrt
- no complicated use of DD or other image handlng
Negative side:
- complicated to exchange the USB stick, the medium you are using
- Very complex workflow, that might fail and leave some weired state behind. The USB stick is not that easy to reconfigure/repartition on a windows system
- Difficult to display the user, that he has now to take an action. Currently it seems already very difficult for the User to find out the state of the auto.install procedure.
ext-root 3/4 way
This solution came to my mind, when I played around the first time with ext-root stuff.
What if we make it possible to use an image-file, like we do on extendRoot, that is mounted as the overlay filesystem. So we have the image file located on the USB stick - vfat.
I never tried to get it working, but I think it won't work that easily, because I think the ext-root overlay is mounted before everything else is done.
For that technique, the USB stick needs to be mounted before the overlay remount happens. I don't know if that is possible.
If it is possible, that would be my preferred way. ... I know, that all people with deeper knowledge of Linux say "aaahhh, why- a clean ext-root is better" .. but keep in mind, we have a lot of users, that are teachers or other non-technicians. They won't be able to work with the data anymore, after we make the USB stick to ext3.
Positive side of this:
- So data are still easily accessible via a vFAT partition
- we can use the advantages of the ext-root function of openwrt
- Still handy vFAT paritions
Negative side:
- Is it possible? Maybe hacks needed?
Hack procd
Create a custom version on the images including a version of procd, that "knows" already the extended library- and binary-path' .. and maybe make it aware of the symlink files.
I don't know how much work this will costs and if it is possible to make this working again.
The positive side of this:
- All the stuff works like before
- Still handy vFAT paritions
The flaws are:
- "Fork" with a patched procd
- Not binary compatible ; maybe more hacks needed to use shipped kernel modules again
- unknown risks
- Work for each OpenWrt release to apply our modifications
I wrote this long post, because I want to know your opinion, experiences and ideas about that.
I'm looking forward you thoughts!
Best regards,
Matthias
Further links for reading:
OpenWrt - boot process
OpenWrt - Preinit_mount sequence
OpenWrt - extroot
OpenWrt extroot-theory
edit:
Added Links
Added statement about overlay_fs and extroot.
This is only my signature.
Edited 3 time(s). Last edit at 11/05/2014 07:18PM by Matthias.
as you might noticed, the OpenWrt team released a new stable version of their OpenWrt distribution. The new release is called "BarrierBreaker" and comes with a wide band of new devices supported and some other interesting things. The PirateBox images are all based upon AttitudeAdjustment and we ported back some device support, which could be done easily.
The PirateBox scripts+packages for OpenWrt do not support BarrierBreaker and above due to one fact: The changed startup behavior of OpenWrt. Only a few things of our scripts works with that new startup behavior and daemons we do not directly start with the PirateBox scripts - like minidlna - stopped working.
With the BarrierBreaker release, the maintainers released a new software called "procd" taking care of the startup handling.
Technically spoken, it is a background daemon which is informed via dbus what to do. It it speaks JSON and starts & stop stuff. As far as my investigation go, I figured out that procd doesn't "know" of the filesystem we have on the USB Stick.
All new init.d scripts - the exsting ones are being replaced by a new style continuously - don't launch the software for them self, they delegate the startup to procd using dbus and procd starts it from the central running daemon.
Into the details:
The common known technique to raise the size of the root filesystem is called "extroot", where you exchange the modify-able filesystem on the hardware is a special formatted USB stick. .. Well, that USB stick needs to be ext3 formatted. This technique still works on BarrierBreaker and is improved and maintained by the OpenWrt developers.
In the following lines, I don't differenciate between a copied extroot or an overlays_fs for extroot. The last one only saves the differences, the first one the complete content is copied. Both techniques are relying on a linux-only filesystem, which is the important fact (read more below).
PirateBox and LibraryBox are using a the technique I call "extendRoot". - That technique is different, that we can use a normal FAT32 formatted USB stick and place a FILE on the USB-Stick. That file is preformatted with ext3 and acts like an CD-iso image-file, which you can modify. The image is mount to /mnt/ext and linked to /usr/local .
/usr/local is per design a normal location to install software to and we add this installation destination to opkg.conf, that you can use '-d ext' to install to the USB Stick.
The problem is, that /usr/local is not included in the search path for Library-files and for executable. Some executables need to be located on special locations, because the PATH variable is not used to locate them.
My work is, that I created a bunch of scripts, that include the new search locations for binaries and libraries to bootup procedure and stuff that is common across a lot files.
After this, I check where I have to do modifications, that software runs off the USB stick. This is for PHP the following
- I create static symlinks pointing from i.e. /usr/bin/php-cgi to /usr/local/usr/bin/php-cgi
- Make the php.ini file available in /etc with a symlink to the /usr/local/etc/php.ini file
- Modify the search path for module-lookups in the php.ini file to /usr/local/usr/lib/php
And other modifications like this. All these things you can find in the i.e. extendRoot-php package, that is installed.
As the PirateBox scripts are installed to /usr/local as well, I have a symlink from /etc/init.d/piratebox to /usr/local/etc/init.d/piratebox
The pitfall here is, that before the USB stick and the image file mounted, all the symlinks are broken. As extendRoot is directly installed on the root filesystem, there is an init-file called /etc/init.d/ext , which is started right after the USB drivers are loaded. This init-file mounts the USB stick (if not happened by hotplug-fstab service), after this the Filesystem-image for /usr/local is mounted (which isn't done by the hotplug fstab service) and swap-files located on the USB stick, too.
You still can install the PirateBox stuff on a Barrier-Breaker OpenWrt, but you will encounter issues like a not working startup. Why?
a) Directly compared between normal init and procd, it seems that procd does not recognize symlinks which are broken, while the normal init still recognize them and try to execute these
b) The procd seems to have hardcoded lookups for executables and libraries, which I can't extend
For PirateBox directly (DHCP, DNS, Webserver and Droopy) I can fix it with creating a copy of the file /etc/init.d/piratebox instead of creating a symlink. That works especially, because the PirateBox init.d script does it the old way with starting each process for themselves and does not delegate it to procd.
The UPNP-server minidlna does not work because the init.d script already delegates to procd, which can't find the libraries needed or the executable.
Puh, lots of technical stuff.
I'm posting this, because I want to discuss how we should proceed in this topic.
How we can make PirateBox possible on BreakingBarrier and small devices too?
What options are possible?
I currently know these options:
Fully ext-root way
Move over to the normal ext-root situation with having a root filesystem on the USB natively. The USB stick needs to be formatted in ext3 and we install all software to the root filesystem.
Sure this needs a sort of changes scripts and tests, but that would be the OpenWrt way to go and we are able to throw away a lot of stuff and work arounds. Sure that hurts, but it will work much smoother later on.
I already tried these technique on a different project and it works quite handy.
The positive side of switching the techniques:
- More compatible way for new software & is easy to extend with new software (no more linking required)
- Should compatible even with newer versions of OpenWrt
The flaws of this techniques are a lot compared to our current one:
- ext3 is for a non-IT guy unhandy. It goes so far that I need to format the data partition on SD-Card of the LibraryBox-RapsberryPi-Image to vfat that people can plug it in every computer. MacOS users may use userspace filesystems, but Windows users are lost and can't copy files to the USB stick without using the upload button or difficult other methods. This is a major flaw compared to our current solution, which is quite handy for everyone.
- If you want to move to a bigger USB stick, you have to do it in a "correct" way to preserve file permissions. If you don't you are doomed.
- It is not that easiy to prepare an existing FAT32 usb stick "on the fly" to a ext3 usb stick. ; The problem is that fsck.vfat needs to be installed to image and/or to tmp
ext-root half way ; image
Another solution can be, that we create an image like SDcard images, that needs to be copied with dd to the USB-stick.
That image contains nn MB ext3 partition in the front of the USB stick and a prepared vfat partition. The vfat partition is extended during the first bootup (or a special command) to maximum size of the USB stick.
Positive side of this:
- So data are still easily accessible via a vFAT partition
- we can use the advantages of the ext-root function of openwrt
Negative side:
- A bit tricky for windows users to "flash" that image file to USB stick ; they don't have DD
- complicated to exchange the USB stick, the medium you are using
ext-root half way ; pepare USB on its own.
This solution is like the half-way image solution, but during the setup, the USB stick is prepared by an installer.
It setups an ext3 partition and the beginning or the end of the USB stick.
The important thing here is, that the installer works in two or three steps:
1. Install Partition and filsystem utilities into the ram of the running system
2. Repartition the USB stick
3. Indicate that the USB stick is finished; the user has to put the folder again on the usb stick, because the fat filesystem was removed/cleaned out
4. Run the next installation with installing software.
Positive side of this:
- So data are still easily accessible via a vFAT partition
- we can use the advantages of the ext-root function of openwrt
- no complicated use of DD or other image handlng
Negative side:
- complicated to exchange the USB stick, the medium you are using
- Very complex workflow, that might fail and leave some weired state behind. The USB stick is not that easy to reconfigure/repartition on a windows system
- Difficult to display the user, that he has now to take an action. Currently it seems already very difficult for the User to find out the state of the auto.install procedure.
ext-root 3/4 way
This solution came to my mind, when I played around the first time with ext-root stuff.
What if we make it possible to use an image-file, like we do on extendRoot, that is mounted as the overlay filesystem. So we have the image file located on the USB stick - vfat.
I never tried to get it working, but I think it won't work that easily, because I think the ext-root overlay is mounted before everything else is done.
For that technique, the USB stick needs to be mounted before the overlay remount happens. I don't know if that is possible.
If it is possible, that would be my preferred way. ... I know, that all people with deeper knowledge of Linux say "aaahhh, why- a clean ext-root is better" .. but keep in mind, we have a lot of users, that are teachers or other non-technicians. They won't be able to work with the data anymore, after we make the USB stick to ext3.
Positive side of this:
- So data are still easily accessible via a vFAT partition
- we can use the advantages of the ext-root function of openwrt
- Still handy vFAT paritions
Negative side:
- Is it possible? Maybe hacks needed?
Hack procd
Create a custom version on the images including a version of procd, that "knows" already the extended library- and binary-path' .. and maybe make it aware of the symlink files.
I don't know how much work this will costs and if it is possible to make this working again.
The positive side of this:
- All the stuff works like before
- Still handy vFAT paritions
The flaws are:
- "Fork" with a patched procd
- Not binary compatible ; maybe more hacks needed to use shipped kernel modules again
- unknown risks
- Work for each OpenWrt release to apply our modifications
I wrote this long post, because I want to know your opinion, experiences and ideas about that.
I'm looking forward you thoughts!
Best regards,
Matthias
Further links for reading:
OpenWrt - boot process
OpenWrt - Preinit_mount sequence
OpenWrt - extroot
OpenWrt extroot-theory
edit:
Added Links
Added statement about overlay_fs and extroot.
This is only my signature.
Edited 3 time(s). Last edit at 11/05/2014 07:18PM by Matthias.
|
Re: [OpenWrt] Porting PirateBox/LibraryBox to BarrierBreaker November 09, 2014 07:53PM |
Registered: 9 years ago Posts: 7 |
Fully ext-root way : "Damn Windowsians, why don't they switch to a real OS ?" (I'm a 98% of the time Windows user). For me, it's the simplest solution, no hack, no incompatibilite, it stick to the right way to do it and it's Windows unfriendly...
The linux VM is an option but, it's not simple, we still have the driver option like ext2fsd [sourceforge.net] but I guess that the kind of software that need more support than the piratebox thing...
So, unless we found a foolproof driver or a really simple walkthrough, it's note the best option so far.
ext-root half way ; image : What about the raspberry solution, an image to flash on the USB ? I use Win32DiskImager [sourceforge.net] which is easy to use.
Flash it from windows, plug it, wait, it's cooked !
ext-root half way ; pepare USB on its own : As you said, longer and full of potential fail by the user.
ext-root 3/4 way : erm...
Hack procd : not an option, we will need to patch each update, and I now see another probleme... It will be hard for non-technicians to port the hacked procd to custom openwrt on specific hardware of choise (Arduino Yun, VoCore, etc.).
here is my two cents
The linux VM is an option but, it's not simple, we still have the driver option like ext2fsd [sourceforge.net] but I guess that the kind of software that need more support than the piratebox thing...
So, unless we found a foolproof driver or a really simple walkthrough, it's note the best option so far.
ext-root half way ; image : What about the raspberry solution, an image to flash on the USB ? I use Win32DiskImager [sourceforge.net] which is easy to use.
Flash it from windows, plug it, wait, it's cooked !
ext-root half way ; pepare USB on its own : As you said, longer and full of potential fail by the user.
ext-root 3/4 way : erm...
Hack procd : not an option, we will need to patch each update, and I now see another probleme... It will be hard for non-technicians to port the hacked procd to custom openwrt on specific hardware of choise (Arduino Yun, VoCore, etc.).
here is my two cents
|
Re: [OpenWrt] Porting PirateBox/LibraryBox to BarrierBreaker November 09, 2014 09:32PM |
Admin Registered: 13 years ago Posts: 4,680 |
Hi,
thanks!
I got an email from David earlier, who is the opinion, that the prepared image file for flashing the USB stick is a good option, too.
About the possibilities of the mount_root on a file stuff:
I browsed the code around the OpenWrt source files ( C-Code for mount_root ; several bash scripts that run in preinit phase (around the time, when the failsafe mode can be triggert; and some C-stuff of procd) for around 3 hours.
I have an idea what is going an, even that I wasn't able to pin it down at a specifc section, but it is not easily possible to mount a filesystem on a stick (like we do it now).
Another point was, that we may enhance environment variables in the pre-init phase, but these might not fall back to procd too, because procd initiates the pre-init phase --- And pre-init stuff must be flashed on the firmware.
As pirateboxbrest pointed out, that destroys compatibility.
I'll do a prototype with the pre-prepared USB image stuff.
regards Matthias
thanks!
I got an email from David earlier, who is the opinion, that the prepared image file for flashing the USB stick is a good option, too.
About the possibilities of the mount_root on a file stuff:
I browsed the code around the OpenWrt source files ( C-Code for mount_root ; several bash scripts that run in preinit phase (around the time, when the failsafe mode can be triggert; and some C-stuff of procd) for around 3 hours.
I have an idea what is going an, even that I wasn't able to pin it down at a specifc section, but it is not easily possible to mount a filesystem on a stick (like we do it now).
Another point was, that we may enhance environment variables in the pre-init phase, but these might not fall back to procd too, because procd initiates the pre-init phase --- And pre-init stuff must be flashed on the firmware.
As pirateboxbrest pointed out, that destroys compatibility.
I'll do a prototype with the pre-prepared USB image stuff.
regards Matthias
|
Re: [OpenWrt] Porting PirateBox/LibraryBox to BarrierBreaker November 09, 2014 11:49PM |
Registered: 9 years ago Posts: 45 |
The functionality needed does not seem to be something really unique to piratebox. So as I see it the best way would be to figure out a way to update OpenWRT to have this functionality. That way there is no need for a "hack", since the "hack" becomes a part of OpenWRT.
Simply have OpenWRT use /usr/local/etc, /usr/local/bin and /usr/local/lib by default. Add to path and library path. Could be empty folders if not used.
Edited 1 time(s). Last edit at 11/09/2014 11:51PM by NoPiracyHere.
Simply have OpenWRT use /usr/local/etc, /usr/local/bin and /usr/local/lib by default. Add to path and library path. Could be empty folders if not used.
Edited 1 time(s). Last edit at 11/09/2014 11:51PM by NoPiracyHere.
|
Re: [OpenWrt] Porting PirateBox/LibraryBox to BarrierBreaker November 10, 2014 06:38AM |
Admin Registered: 13 years ago Posts: 4,680 |
Mh yes, that is a true story. The idea is quite good.
I tried asking for help on the OpenWrt-Dev Mailing list, when BarrierBreaker was the latest trunk images,... and nobody answered.
Maybe nobody understood it, or I was just too general or confusing.
If we wait for an update of procd in the stable images, I have to wait for the next stable release - I assume 2015 or 2016.
Matthias
I tried asking for help on the OpenWrt-Dev Mailing list, when BarrierBreaker was the latest trunk images,... and nobody answered.
Maybe nobody understood it, or I was just too general or confusing.
If we wait for an update of procd in the stable images, I have to wait for the next stable release - I assume 2015 or 2016.
Matthias