Mongrel Upload Progress Plugin

One of the really nice things about Mongrel is its simplicity. It's very easy for someone to take and extend for their own needs. The Mongrel Upload Progress plugin is an example of how I'm able to extend the Mongrel HTTP Request object and provide near-realtime progress updates.

The reason why this is a challenge, is because web servers usually gather the HTTP request, send it on to the web framework, and wait on a response. This is fine for most requests, because they're too small to cause an issue. For large file uploads this is a usability nightmare. The user is left wondering what whether their upload is going through or not.

To do this, I've written a Mongrel handler that hooks into some basic Request callbacks. To use it, you need to install the gem, and create a small config file for it:

gem install mongrel_upload_progress

# config/mongrel_upload_progress.conf
uri "/", 
  :handler => plugin("/handlers/upload", :path_info => '/files/upload'), 
  :in_front => true

# start mongrel
mongrel_rails start -d -p 3000 -S config/mongrel_upload_progress.conf

That config file tells mongrel to load the Upload handler in front of all other handlers. :path_info is passed to it, telling upload_progress to only watch the /files/upload action. There are two more parameters that I'll get into later: :frequency and :drb. I'm using Rails as an example, but this should work with any Ruby framework, such as Camping or Nitro.

Now that Mongrel is set up, let's create a basic upload form. If you look closely you'll notice a few things:

• A unique :upload_id parameter must be sent to the upload_progress handler. This is so requests don't get mixed up, and the client page has an ID to query with.
• The <form> tag is targeted to an <iframe> to do the uploading. Certain browsers (like Safari) won't execute javascript while a request is taken place, so this step is necessary.
• There is a little javascript library being used. This handles the polling and status bar updates.
• Notice the form's action is file/upload, just like the upload_progress handler.

The Rails controller actions for this are very simple. The upload form itself needs no custom code. The upload action only renders javascript to be executed in the iFrame, to modify the contents of the parent page. The progress action is a basic RJS action that updates the current status. Most of the guts of this are implemented in the javascript library.

Here's what happens when you submit the form:

• The UploadProgress class creates a PeriodicalExecuter and gets ready to poll.
• The browser initiates the upload.
• Every 3 seconds, the PeriodicalExecuter calls the RJS #progress action and gets back the current status of the file.
• Once finished, the iFrame calls window.parent.UploadProgress.finish(), which removes the status bar and performs any other finishing actions.

How's this work with a single Mongrel process if Mongrel synchronizes Rails requests? It's actually very careful about locking, synchronizing only the bare minimum. The whole time that Mongrel is receiving the request and updating the progress is spent in Mongrel, so it can happily serve other requests. This is how the RJS action is able poll while it's uploading.

This is fine and dandy, but not too many sites run on a single Mongrel. You'll quickly run into problems with multiple mongrels since only one Mongrel process knows about the upload. You'll either have to specify a specific mongrel port to communicate with, or set up a dedicated mongrel upload process. The third option, is use DRb.

# config/mongrel_upload_progress.conf
uri "/", 
  :handler => plugin("/handlers/upload", 
    :path_info => '/files/upload', 
    :drb => 'druby://127.0.0.1:2999'), 
  :in_front => true

# lib/upload.rb, the upload drb server
require 'rubygems'
require 'fastthread'
require 'drb'
require 'gem_plugin'
GemPlugin::Manager.instance.load 'mongrel' => GemPlugin::INCLUDE
DRb.start_service 'druby://127.0.0.1:2999', Mongrel::UploadProgress.new
DRb.thread.join

Now in addition to starting mongrel, you'll need to start the DRb service too:

ruby lib/upload.rb

The Rails app should work the same as before, but now it is using a shared DRb instance to store the updates. This gives us one other advantage: a console interface to the current uploads.

# lib/upload_client.rb, a simple upload drb client
require 'drb'
DRb.start_service

def get_status
  DRbObject.new nil, 'druby://127.0.0.1:2999'
end

# typical console session
$ irb -r lib/upload_client.rb
>> uploads = get_status
>> uploads.list
=> []
# start uploading in the browser
>> uploads.list
=> ["1157399821"]
>> uploads.check "1157399821"
=> {:size=>863467686, :received=>0}

Using DRb gives you a simple way to monitor the status of current uploads in progress. You could also write a simple web frontend for this too, accessing the DRb client with Mongrel::Uploads.

One final note is the use of the frequency option. By default,

the upload progress is marked every three seconds. This can be modified through the mongrel config file:

uri "/", 
  :handler => plugin("/handlers/upload", 
    :path_info => '/files/upload', 
    :frequency => 1,
    :drb => 'druby://127.0.0.1:2999'), 
  :in_front => true

<html>
  <head>
    <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
    <title>mongrel test</title>
    <%= javascript_include_tag :defaults %>
    <%= javascript_include_tag 'upload_progress_javascript' %>  
<style type="text/css">
#progress-bar {
  width:500px;
  height:25px;
  margin:15px;
  border:solid 1px #000;
  position:relative;
}

#progress-bar #status-bar {
  display:block;
  height:25px;
  width:0;
  background-color:#00f;
  border-right:solid 1px #000;
  position:absolute;
  top:0; left:0;
}

#progress-bar #status-text {
  display:block;
  padding: 0 15px;
  line-height:25px;
  position:absolute;
  top:0; left:0;
}
</style>
  </head>
  <body>

<p><%= link_to (@upid = Time.now.to_i.to_s), :action => 'status', :upload_id => @upid %></p>
<%= form_tag({:action => 'upload', :upload_id => @upid}, {:multipart => true, :target => 'upload',
      :onsubmit => "UploadProgress.monitor('#{escape_javascript @upid}')"}) %>
  <div id="file-fields">
    <p><%= file_field_tag :data %></p>
  </div>
  <p><%= link_to_function 'Add File Field', 'UploadProgress.FileField.add()' %>
  </p>
  <p><%= submit_tag :Upload %></p>
  <%=token_tag%>
</form>

<div id="results"></div>
<div id="progress-bar"></div>

<iframe id="upload" name="upload" src="about:blank"></iframe>

  </body>
</html>

var UploadProgress = {
  uploading: null,
  monitor: function(upid) {
    if(!this.periodicExecuter) {
      this.periodicExecuter = new PeriodicalExecuter(function() {
        if(!UploadProgress.uploading) return;
        new Ajax.Request('/files/progress?upload_id=' + upid);
      }, 3);
    }

    this.uploading = true;
    this.StatusBar.create();
  },

  update: function(total, current) {
    if(!this.uploading) return;
    var status     = current / total;
    var statusHTML = status.toPercentage();
    $('results').innerHTML   = statusHTML + "<br /><small>" + current.toHumanSize() + ' of ' + total.toHumanSize() + " uploaded.</small>";
    this.StatusBar.update(status, statusHTML);
  },
  
  finish: function() {
    this.uploading = false;
    this.StatusBar.finish();
    $('results').innerHTML = 'finished!';
  },
  
  cancel: function(msg) {
    if(!this.uploading) return;
    this.uploading = false;
    if(this.StatusBar.statusText) this.StatusBar.statusText.innerHTML = msg || 'canceled';
  },
  
  StatusBar: {
    statusBar: null,
    statusText: null,
    statusBarWidth: 500,
  
    create: function() {
      this.statusBar  = this._createStatus('status-bar');
      this.statusText = this._createStatus('status-text');
      this.statusText.innerHTML  = '0%';
      this.statusBar.style.width = '0';
    },

    update: function(status, statusHTML) {
      this.statusText.innerHTML = statusHTML;
      this.statusBar.style.width = Math.floor(this.statusBarWidth * status);
    },

    finish: function() {
      this.statusText.innerHTML  = '100%';
      this.statusBar.style.width = '100%';
    },
    
    _createStatus: function(id) {
      el = $(id);
      if(!el) {
        el = document.createElement('span');
        el.setAttribute('id', id);
        $('progress-bar').appendChild(el);
      }
      return el;
    }
  },
  
  FileField: {
    add: function() {
      new Insertion.Bottom('file-fields', '<p style="display:none"><input id="data" name="data" type="file" /> <a href="#" onclick="UploadProgress.FileField.remove(this);return false;">x</a></p>')
      $$('#file-fields p').last().visualEffect('blind_down', {duration:0.3});
    },
    
    remove: function(anchor) {
      anchor.parentNode.visualEffect('drop_out', {duration:0.25});
    }
  }
}

Number.prototype.bytes     = function() { return this; };
Number.prototype.kilobytes = function() { return this *  1024; };
Number.prototype.megabytes = function() { return this * (1024).kilobytes(); };
Number.prototype.gigabytes = function() { return this * (1024).megabytes(); };
Number.prototype.terabytes = function() { return this * (1024).gigabytes(); };
Number.prototype.petabytes = function() { return this * (1024).terabytes(); };
Number.prototype.exabytes =  function() { return this * (1024).petabytes(); };
['byte', 'kilobyte', 'megabyte', 'gigabyte', 'terabyte', 'petabyte', 'exabyte'].each(function(meth) {
  Number.prototype[meth] = Number.prototype[meth+'s'];
});

Number.prototype.toPrecision = function() {
  var precision = arguments[0] || 2;
  var s         = Math.round(this * Math.pow(10, precision)).toString();
  var pos       = s.length - precision;
  var last      = s.substr(pos, precision);
  return s.substr(0, pos) + (last.match("^0{" + precision + "}$") ? '' : '.' + last);
}

// (1/10).toPercentage()
// # => '10%'
Number.prototype.toPercentage = function() {
  return (this * 100).toPrecision() + '%';
}

Number.prototype.toHumanSize = function() {
  if(this < (1).kilobyte())  return this + " Bytes";
  if(this < (1).megabyte())  return (this / (1).kilobyte()).toPrecision()  + ' KB';
  if(this < (1).gigabytes()) return (this / (1).megabyte()).toPrecision()  + ' MB';
  if(this < (1).terabytes()) return (this / (1).gigabytes()).toPrecision() + ' GB';
  if(this < (1).petabytes()) return (this / (1).terabytes()).toPrecision() + ' TB';
  if(this < (1).exabytes())  return (this / (1).petabytes()).toPrecision() + ' PB';
                             return (this / (1).exabytes()).toPrecision()  + ' EB';
}

class FilesController < ApplicationController
  session :off, :only => :progress
  skip_before_filter :verify_authenticity_token, :only => :progress

  def progress
    render :update do |page|
      @status = Mongrel::Uploads.check(params[:upload_id])
      page.upload_progress.update(@status[:size], @status[:received]) if @status
    end
  end
  
  def upload
    render :text => %(UPLOADED: #{params.inspect}.<script type="text/javascript">window.parent.UploadProgress.finish();</script>)
  end
end