Ship's log about current development activities

Just to keep track of things happens and to give a few more comments then github allows

---

Created branch v0.6 of PirateBoxScripts_Webserver to leave this version stable cause of release of the new PirateBox OpenWRT-package 0.6
Fixed a wrong tag in PirateBoxScripts_Webserver and pushed missing v0.6c tag

Reviewed new install script

Added systemd support, so that we are fully supported when distributions start to get rid of init.d system.

Added install script, for new packages just run

sudo ./install default

(or you could substitute default with board in order to install a message board.

Fixed droopy crash when run through init.d, this was caused by Python environmental variables not being used normally in the init.d environment.

Updated README

Tested uploader2 script from c't and got it working. Nice updload utility, which should be tested on OpenWRT,
Link will be provided on request.

Made with Nargren several performance tests with the upload2 from c't.
scp reaches stable ~1,4MB/s in Network-Monitor
I was able to reach up 1,1MB/s with a chunk size of 500*1024 (average around 650kb/s)

uploader2 is relativ unstable in the frontend and if the backend-php gets errors like "file could not be written" it is not brought to the frontend.

I found an old post about the file-upload handling from lighttpd: [blog.lighttpd.net]
One tmp-chunk is 1MB (as I discovered during tests), but everything below 64kb will be buffered in memory. Maybe this with fcgi can help... but as I tested withou fastcgi it produced too much overhead.
I tested to write to /tmp too... this might improve the performance, but I can't give any valuable details caus of filesystem defects on usb stick
[[i]update later[/i]] with using 500k chunks and /tmp as temporary buffer I was able to raize the performance a max of 1,4MB/s - but multiple uploads seem to cause trouble somethimes ) Switching to /tmp instead of the usb stick has the implication, that we have to limit upload-size at on POST request (uploader2 sends file-chunks via alot of post requests .. one ofter another ). Imageboard has no chunk-upload, so We may reduce the filesize down to 5MB for reflecting the size of /tmp




Uploaded packed downloadable PirateBoxScripts for Webserver to downloads.piratebox.de

From my webalizer; some download statistics for January 2013:

current OpenWRT Version
1188 piratebox_0.5.1_all.ipk
240 piratebox_ws_img.tar.gz (why is this so low comparing to the 0.5.1_all.ipk ?? )

old OpenWRT Version without webserver
72 piratebox_0.3-2_all.ipk (old non webserver version)
37 piratebox_img.gz

System images
866 openwrt-ar71xx-generic-tl-mr3020-v1-squashfs-factory.bin (custom image)
23 2012-12-16-wheezy-raspbian-withPirateBox.zip

Didn't count the mesh package-downloads, because it is a beta.

(current (2013-02-07) count of RPi image for February is 42!)


Have to dig further into this [wiki.openwrt.org] link about correct OpenWRT makefile, to get my stuff included to OpenWRT.

---

USB Performance results from my Kingston Datatraveler 3G 4GB

root@OpenWRT:/mnt/ext/sbin# ./hdparm -Tt /dev/sda1

/dev/sda1:
 Timing cached reads:   134 MB in  2.02 seconds =  66.35 MB/sec
 Timing buffered disk reads:  46 MB in  3.01 seconds =  15.31 MB/sec

root@OpenWRT:/# bonnie++ -n 0 -u 0 -r 32 -s 64 -f -b -d /mnt/usb/
Using uid:0, gid:0.
Writing intelligently...done
Rewriting...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
OpenWRT  64M            3396   8  3694  11           16393  24  33.0   5
Latency                        3088ms    1013ms               121ms    4726ms

The USB Stick itself isn't very fast.



Edited 2 time(s). Last edit at 02/07/2013 08:37AM by Matthias.

Wrote a lot of stuff down, which were on my mind a few days long.
After upload tests yesterday, I created this discussion thread about upload[/u] to see what we can improve there.

Then I wrote down the logical steps happens during installation in my wiki [piratebox.aod-rpg.de] .
This is the first step for remixing the OpenWRT stuff and make it more conform to OpenWRT opkg stuff (and reuse his functionality more).

Wrote down the concept of the new packages and how openwrt specific code should be located and executed: Concept new packaging

Have to write down a few more argumentations why and rethink the naming standards.



Edited 1 time(s). Last edit at 02/10/2013 12:06PM by Matthias.

Figured out, that there can be more than one repository entry in opkg.conf , which is useful for a special piratebox repository.

Created a zeroMQ-PythonLib package-Makefile for getting into packaging for OpenWRT Systems

Noted down the experiences from my user-observation.

TerrorByte figured out to get the PirateBox stuff to the new UbuntuTablet... there is need some work to get the hostapd working.
Yesterday I started with the new packaging concept, which is a bunch of work.. but hopefully we can make upgrading more easier than today.

Seems I have to dig into this DNS stuff again, it regular causing issues (Box feels slow and so on)



Edited 1 time(s). Last edit at 02/22/2013 10:36AM by Matthias.

A few days delayed, but I got a Nexus 7 and instantly installed the Ubuntu Touch OS the day it was released. After tinkering I found that all the packages we use are available in the arm repo's which meant that I could easily just run our scripts, which ran perfectly. There were 2 major bugs when it was run, hostapd doesn't work at all (this may be because the way that Ubuntu runs in a jail of Cyanogenmod so I am not directly interfacing with the hardware) and dnsmasq doesn't redirect properly at all. I suspect that most of these are just caused by the bleeding edge of the OS and will be resolved in the coming weeks. In other words, there will be a port of PB to Ubuntu Tablets and possibly phones (and possibly an app / desktop manager if we can find someone who knows QT).

On another note, I've slowly been learning make so that we can get our packages built. It is taking longer than I thought, but I want to make sure that there are as few bugs as possible in the packaging system so that we can just re-roll it as much as we need into the future.

I am waiting for PirateBoxLive v2 to get published until we get our packages built. It's framework is already built and should just take a small amount of time to release after the packages are released.

For helping a local user I configured my MR3040 to be a local install source and fixed a few things in the HowTo.
Beside of this, I recognized, that my issues with the MR3040 was the result of a reflash, that didn't went so well.

Did some testing with meshstuff around today. Seems to be quite stable... but while Forban slows down the box, especially in opportunistic mode.
Forban runs quite well for smaller file-sizes. I synchronized a bunch of pics quite well. Bigger files seems to be a bit problematic... don't get transfered completly, which causes trouble in opportunistic mode.
I'm talking with Adulau for changes about the synchronizing and File-transfer stuff.

I made some performance-test getting some data over the mesh-net. Setup:

  <client> -- wired --> <PB1> {Mesh} <PB2-webserver>

I received data directly through the webserver and got a transferspeed of around an average of 1 MB/s.
The Mesh-Net containted 3 active clients and I downloaded it from the farest client (next room) (but it was no additional hop on the path).

Why is the transfer so slow? I assume, this is a result of the dual running AP & mesh on the same wifi-channel. You see the transfer speed raizing and then it suddenly drops, which looks like a collision.

Edit: Well, and yesterday I created a patch for the use of USB sticks during bootup, which take a while in initializing.

Edit2: Fixed the USB Bug on RPi. Uploaded and pushed version v0.6D ; refreshed master. prepared new RPi image. Published scratchpad for patching RPi image



Edited 2 time(s). Last edit at 02/26/2013 08:13PM by Matthias.

Long day.
Created a About some changes of the mesh stuff.

Created a long wiki article about introduction of the mesh stuff

a lot of conecpt stuff...

Didn't mentioned it yesterday. I finished the new extendRoot-Package as alpha version. It is the basement for getting the new piratebox-0.7 more easier in code and update-able.
Even it enables bigger packages to install to small routers without doing alot about this ext-filesystem stuff. These package will enhance development stuff a bit.

Finally, I saved 72kb on the routers flash because of not saving stuff into the package.

Broke my mind at the RPi problems.. hopefully I'm able to find a solution soon.

On Sunday, i was at a cryptoparty ( [cryptoparty.org] ) and hopefully I interested a few people

Fetched a few numbers out of my webalizer:

Downloads    Name
619   2012-12-16-wheezy-raspbian-withPirateBox.zip


954   MR3020 custom image

619   piratebox_0.5.1_all.ipk
229   piratebox_ws_img.tar.gz    
79    piratebox_0.3-2_all.ipk

129   piratebox-ws_current.tar.gz  (contains the link to the most recent scripts)

mest-test downloads
30   MR3040 custom image
 5   MR3020 custom image

(I'm, always wondering about that difference between piratebox_0.5.1_all.ipk to piratebox_ws_img.tar.gz Both are somehow related)


That is all what I can get out of my top 50 list from webalizer.

Gone further with new packaging
* piratebox package alpha done
* OpenWRT-Makefile for imageboard done
Fixed a bug found on CryptoParty-Workshop... will be delivered in 0.6E & 0.7

Yesterday I released the 0.6.3 version of the PirateBox-OpenWRT package.

What nobody recognized, is that I released the working version of OpenWRT-Forban too

I also created a beta-package with a different wlan setup for getting a better support to the ALFA devices, which causes trouble while hostapd comes up.

I had a deeper look into miniircd python script, which runs a basic IRC Server, and yes- it works out of the box. Today I created the branch irc_server (github) which contains modified Makefile and startup stuff.
TODO: Have to create a irc.html page, which explains how you enter the thing and add a java-irc-client with webstart to the www stuff.


Tomorrow I have another meeting about PirateBox in Frankfurt. Let's see if I can get more support or at least users in Germany

Download numbers based on Top5 (so some files are not listed because they were below Top xy list )

Downloads    Name
393 2012-12-16-wheezy-raspbian-withPirateBox.zip
75 new_2012-12-16-wheezy-raspbian-withPirateBox.zip


877	MR3020 custom image

624  piratebox_0.5.1_all.ipk
236   piratebox_ws_img.tar.gz    


119 piratebox-ws_current.tar.gz  (contains the link to the most recent scripts)

mest-test downloads
29   MR3040 custom image
18   MR3020 custom image

released 0.6E today.
Rebranched PirateBoxScripts_Webserver to propber stable, development ... branches.

My no_openwrt branch was also merged into development branch. I deleted ~108 lines of code and deleted the majority of the dd-wrt code in the scripts, we kept the piratebox_router init due to the fact that some systems that don't have udev can use it (it is now called piratebox_alt).

Sorry for not posting anything new a while ago.

Today, I retested OpenWRTs ImageBuilder today and had a success. Read more here: [forum.daviddarts.com]

In addition, I made some progress with 0.7 , which will be named to 1.0.

who follows me on twitter should have recognized, that we already have a first new responsive design

More information on Saturday about my activities...

Regards Matthias

*testing new responsive design today*

Working on

0.6F - currently beta contains:

* Several fixes for installing kareha on non-OpenWRT devices
* Some fixes in installing kareha if something went wrong
* New Feature: Mime-Type for epub files
* New Feature: No caching in redirect.html that would prevent a bit browser-cache poisening.

Regards
Matthias

Last post is quite a while ago. Sorry for that.

As some of you already noticed, I created new images for the most common models. These images already contains the first beta-version of the PirateBox-Repository. This brings us closer to an easier installation-path.

As usually, I provided the sourcecode I did here:

Building a temporary repository out of .ipk

ImageBuilding routinges
The last link "injects" the repository and strips down all unneeded stuff and includes what we need to use the USB stick. For building trunk images, use the trunk branch and a x86_64 system.

I prepared my slides for the camp (90%).

Later on, I started to rework on the distributed documentations. I started to move some stuff over to my webpage(OpenWRT namespace). It is not totally finished, but it is a start to get the stuff better collected.

As a few know, I published the UnP Media Server Howtos

Beside of that, I started to work on a script, which should make some common changes a bit easier. That isn't published/pushed yet.

Ahh, one major result of my talks on CryptoPartyFFM was, that we are going to stop the watchdog in further releases. If there is much i/o load, the box initiates an reboot, while it should be possible to handle it. If you encounter reboot during heavy-load you can do this manually once on the Box

/etc/init.d/watchdog stop
/etc/init.d/watchdog disable

The first responsive design including Kareha custom-theme is work in progress.
During the Camp-preparations I recognized that I have to overwork the roadmap and move some stuff into the future.

so far Matthias



Edited 1 time(s). Last edit at 06/25/2013 09:19AM by Matthias.