[WIP] lua file upload, progress bar, lua textboard

hello everybody,
I started building a piratebox on an OpenWRT AP and as soon as I discovered that the upload is based on a python script I wanted to try to install a custom script based on lua or something of comparable footprint size.

I studied some solution based on existing lua libraries in order to write a CGI that can receive a big file but soon realized that all existing solution have some mix of bad conditions: a big list of dependencies, MVC based, store the whole file as a string in memory! I even tested haserl but I was not impressed by it (unfortunally the stock feed of haserl do not have lua support compiled in).

So I ended up writing my own POST parser in lua (at first I was not exited by reinventing the wheel) and fortunally I discovered that the implementation can be quite easy to understand and maintain. At this point I wrote a lua module that can parse a POST of type "multipart/form-data" and nothing more. I think I will add some more encoding somewhere in the future. As web server I use uhttpd but since I'm writing CGI the web server can be easily changed, it's only required that it does not buffer the whole POST message before spawning the CGI, I think this is a outside the scope of RFCs so
every httpd daemon have to be tested.

I started playing with jQuery File Upload and now I'm hable to have a nice page to upload multiple file (concurrently) with a nice progress bar. My CGI can handle multiple file in a single POST but the javascript is built to send parallel POST. The jQuery File Upload use bootstrap and can handle drag & drop. Everything seems much more modern that the droopy solution and does not require python. Performance wise is comparable to the vsftpd daemon. I didn't made much testing but seems like hit the limit of my pendrive during write (about 2MB/s). I still have to test with a fast HDD.

My next step is to write a textboard like kareha with lua so I can drop perl too
My motivation is to have a small footprint piratebox that can better match the performance of the atheros AR9330
and on this platform I only see lua, c code and shell scripts as viable solutions. The kareha-like textboard will be more
challenging than file uploads, so I do not make promises, I hope to be able to make a complete implementation.

I will start posting the code on this thread since I still haven't opened any github project for this work.
At this moment this is more a proof of concept that a finished product.

this is the cgi.lua module (not much fantasy)

-- cgi util module

local prevbuf = ""
local blocksize = 4096
local _M = {}

_M.statusmsg = {
   [200] = "OK",
   [206] = "Partial Content",
   [301] = "Moved Permanently",
   [302] = "Found",
   [304] = "Not Modified",
   [400] = "Bad Request",
   [403] = "Forbidden",
   [404] = "Not Found",
   [405] = "Method Not Allowed",
   [408] = "Request Time-out",
   [411] = "Length Required",
   [412] = "Precondition Failed",
   [416] = "Requested range not satisfiable",
   [500] = "Internal Server Error",
   [503] = "Server Unavailable",
}

-- call this function passing an empy table. use that table for successive calls.
function _M.new(req)
   req.content_length = os.getenv("CONTENT_LENGTH")
   req.request_method = os.getenv("REQUEST_METHOD")
   if req.request_method == "POST" then
      req.content_type, req.boundary = string.match(os.getenv("CONTENT_TYPE"),"^(multipart/form%-data); boundary=\"?(.+)\"?$")
      req.boundary = "--" .. req.boundary
   end
   -- this is useful only if you have /tmp on tmpfs like in openwrt. otherwise can be set to ""
   req.tempdir = "/mnt/piratebox/tmp"
   req.post = {}
end

-- this function is needed to clean temp file since and hide implementation details
function _M.cleanup(req)
   for k, v in pairs(req.post) do
      for j, v in pairs(req.post[k]) do
         if req.post[k][j].tempname then
            os.remove(req.post[k][j].tempname) -- if file unused
            os.remove("/tmp/" .. string.match(req.post[k][j].tempname,"^" .. req.tempdir .. "(.+)"))
         end
      end
   end
end

-- del: delimiter
-- return chunk (string), found (boolean)
local function chunkread(del)
   local buf, found = 0
   local del = del or "\r\n"

   buf = io.read(math.max(0,blocksize + #del - #prevbuf))
   if prevbuf ~= "" then buf = prevbuf .. ( buf or "" ); prevbuf = "" end
   if not buf then return end

   s, e = string.find(buf,del,1,true)
   if s and e then
      found = 1
      prevbuf = string.sub(buf,e+1)
      buf = string.sub(buf,1,s-1)
   else
      prevbuf = string.sub(buf,math.min(blocksize,#buf)+1)
      buf = string.sub(buf,1,math.min(blocksize,#buf))
   end

   return buf, found
end


function _M.parse_request_body (req)
   local chunk, found, type, tempname, tempfile
   local param = {}

   -- read first boundary line
   chunk, found = chunkread(req.boundary)
   chunk, found = chunkread("\r\n")
   while chunk == "" do
      -- read part headers and get parameters value
      repeat
         chunk, found = chunkread("\r\n")
         if not found then return 400, "Malformed POST. Missing Part Header or Part Header too long." end
         string.gsub(chunk, ';%s*([^%s=]+)="(.-[^\\])"', function(k, v) param[k] = v end)
         param.type = param.type or string.match(chunk, "^Content%-Type: (.+)$")
      until chunk == ""

      -- prepare file data read
      if not param.name then return 400, "Malformed POST. Check Header parameters." end
      param.size=0
      param.tempname = req.tempdir .. string.match(os.tmpname(),  "^/tmp(.+)")
      tempfile = io.open(param.tempname, "w")

      -- read part body content until boundary
      repeat
         chunk, found = chunkread("\r\n" .. req.boundary)
         if not chunk then return 400, "Malformed POST. Incomplete Part received." end
         tempfile:write(chunk)
         param.size = param.size + #chunk
      until found
      tempfile:close()
      req.post[param.name] = req.post[param.name] or {}
      table.insert(req.post[param.name], 1, param)
      param = {}

      -- read after boundary. if CRLF ("") repeat. if "--" end POST processing
      chunk, found = chunkread("\r\n")
   end

   if found and chunk == "--" then return 0, "OK" end
   return 400, "Malformed POST. Boundary not properly ended with CRLF or --."
end

return _M

this is the simpliest index and CGI page:

<!DOCTYPE html>
<html>
<head>
  <title>Index Page</title>
</head>
<body>
  <h1>This is a heading</h1>
  <p>test2.lua</p>
  <form action="/cgi-bin/test2.lua" enctype="multipart/form-data" method="post">
    <input type="file" name="file1234" multiple="yes"><br>
    <input type="submit" value="Submit">
  </form> 
  <p>test3.lua</p>
  <form action="/cgi-bin/test3.lua" enctype="multipart/form-data" method="post">
    <input type="file" name="file1234" multiple="yes"><br>
    <input type="submit" value="Submit">
  </form> 
  <a href='/share/'>share folder</a>
</body>
</html>

#!/usr/bin/pcall.lua

local cgi = require "cgi"

local req = {}

local function printf (s,...)
    return io.write(s:format(...))
end

cgi.new(req)

if req.boundary then
   ret, msg = cgi.parse_request_body(req)
end

io.write("Content-Type: text/html\r\n\r\n")
io.write("<!DOCTYPE html>")
io.write("<html><body><p>hello world! </p><pre>")
printf("<p>post processing: %d %s </p>" ,ret, msg)

for k, v in pairs(req.post) do
   io.write("<p>input table " .. k .. "</p>")
   for j, v in pairs(req.post[k]) do
      io.write("<p>  file index " .. tostring(j) .. "</p>")
      io.write("<p>  type       " .. req.post[k][j].type .. "</p>")
      io.write("<p>  tempname   " .. req.post[k][j].tempname .. "</p>")
      io.write("<p>  filename   " .. req.post[k][j].filename .. "</p>")
      io.write("<p>  size       " .. req.post[k][j].size .. "</p>")
      os.rename(req.post[k][j].tempname, "/mnt/piratebox/www/share/" .. req.post[k][j].filename)
   end
end

cgi.cleanup(req)

io.write("<p>bye bye!</p>")
io.write("</pre></body></html>")

this is a stupid wrapped to see lua error as html pages:

#!/usr/bin/lua

ok, res = pcall(dofile,(...))

if not ok then
   io.write("Status: 500 Internal Server Error\r\nContent-Type: text/plain\r\n\r\n" .. res .. "\r\n")
end

I formatted a pendrive as ext4 without journal and I'm using the tree:
/mnt
/mnt/piratebox
/mnt/piratebox/www
/mnt/piratebox/www/cgi-bin
/mnt/piratebox/tmp

for the jQuery File upload I redirect you to [blueimp.github.io]
on the sever side I use a special CGI that can return JSON data as required by the javascript:

#!/usr/bin/pcall.lua

local cgi = require "cgi"

local req = {}

local function printf (s,...)
    return io.write(s:format(...))
end


cgi.new(req)

if req.request_method == "GET" then
   io.write("Content-Type: text/plain\r\nContent-Length: 0\r\n\r\n")
end

if req.request_method == "POST" then
   ret, msg = cgi.parse_request_body(req)

   report = "{\"files\": ["
   for j, n in pairs(req.post) do
      for k, v in pairs(req.post[j]) do
         if #report > 11 then report = report .. ", " end
         if req.post[j][k].filename then
            report = report .. "{ \"name\": \"" .. req.post[j][k].filename .. 
            "\", \"size\": " .. tostring(req.post[j][k].size) .. 
            ", \"url\": \"http:\\/\\/192.168.1.5\\/share\\/" .. req.post[j][k].filename .. "\" }"
            os.rename(req.post[j][k].tempname, "/mnt/piratebox/www/share/" .. req.post[j][k].filename)
         end
      end
   end
   report =  report .. " ]}\r\n"

   if req.post.redirect then
      tempfile=io.open(req.post.redirect[1].tempname, "r")
      url=tempfile:read()
      tempfile:close()
      io.write("Content-Type: text/html\r\n\r\n")
      io.write([[<!DOCTYPE html><html><head><meta http-equiv="refresh" content="2; url=/index2.html"></head><p>File Uploaded</p><body></body></html>]])
   else
      io.write("Content-Type: text/plain\r\nContent-Length: " .. tostring(#report) .. "\r\n\r\n")
      io.write(report)
   end
   cgi.cleanup(req)
end

I'm looking for some feedback, If someone is interested in joining the effor can contact me on the forum

bye!

bump

this is just to let you know i'm still working on my version of piratebox in lua.
so far I have a alpha stage message board with RESTful method.
the board is a simple container of thread, each thread have the first message
with a title and successive messages are only response without title.
I started with ordinary URLs with query string but now I use only semantic URLs
( [en.wikipedia.org] ) and I hope to keep that way
until everything is complete

I'm working on the authentication with HTTP Basic Auth.

Since I didn't get any feedback so far I didn't posted code since is stil alpha.
If someone is interested I will set it up somewhere with a little guide to test it.

bye!

i'm still interested.

---

This is only my signature.

hi Matthias,
here you can find a quick packaging of my work so far
[drive.google.com]

At this moment I'm more into the messageboard development so the fileupload is still more a
proof of concept than a polished product. the cgi.lua (a library) is in good shape.
the concept of file upload is that the library return structure with a list of tempfiles and the relatives
realnames, all you have to do in the main i make a move (rename) from tempfile to filename.
tempfiles that exist on exit will be wiped. Use a posix file system (no FAT), this is pointed out
in the README.txt

Authentication is full work in progress

I will add updated version on that folder, if you feel you want to contribute I will put that on github.
I prefer working without git for now since I'm alone

bye!

bump!

another update:

- big rework of board.lua, now it's a module and calling the right
function is a little smarter.
- almost finished converting the output of board.lua from HTML to JSON.
on client side I started handling json and draw messages from that.
not yet completed, I want to convert procedural code to something
more elegant but I have to study on it a little
- authentication is working, maybe not the best solution but this is
RESTful (stateless)
- file upload is still a mess, the lua module is working but the
application using it have too much hard coded path and stuff that
means it will not work if not ported correctly. sorry I will try to return
on this subject ASAP. anyway the code is very simple it you look
at it you will easily understand what need to be fixed to get it work.

[drive.google.com]

enjoy
littlepirate

I'm interested in your code, but I can't copy paste it, since some characters replaced by a smiley. Is there any repository for it?

Also, IMHO, you don't care if the delimiter is divided in two chunks?