XMPP web demo: setting it up

I will describe the XMPP demo setup on AWS EC2, starting with Alestic Ubuntu karmic image (ami-19a34270). Hopefully, you should be able to adjust the process to your environment of choice.

Setting up

Brace yourself, that's going to be a long one. Major components to install and configure are:

• Erlang environment
• exmpp
• ejabberd
• nginx
• and, of course, weazard itself
  

So, provided you have running instance of ami-19a34270, let's  ssh to it and start by:

sudo apt-get update && sudo apt-get upgrade

We'll need svn to checkout some stuff later:


apt-get install subversion


Then install Erlang (version R12B5 or newer; karmic has R13B01):
 

apt-get install erlang

Now, let's move on to exmpp.  First, we need to install some libraries that exmpp is using for XML support:

apt-get install libxml2-dev

or, alternatively:

apt-get install libexpat1-dev


Update: the demo will work fine with libxml2, however, at the time of writing (Nov 07, 2009), if you're planning to use exmpp for other purposes, particularly, exmpp_session module, you're advised to install libexpat1-dev instead. See https://support.process-one.net/si/jira.issueviews:issue-html/EXMPP-16/EXMPP-16.html for details.

apt-get install libssl-dev
apt-get install pkg-config


Now, download the exmpp distribution and build:

wget http://download.process-one.net/exmpp/exmpp-0.9.1-r857.tar.gz
gunzip exmpp-0.9.1-r857.tar.gz
tar -xvf exmpp-0.9.1-r857.tar 
./configure
make
sudo make install


At this point you may want to check if exmpp is in working condition. Start erl and at the prompt type: 

1>exmpp:start().
ok
2>exmpp_xml:start_parser().

If response is similar to
{xml_parser,[{max_size,infinity},
             {root_depth,0},
             {names_as_atom,true},
             {emit_endtag,false}],
            #Port<0.747>},

then everything went fine and you can type q(). at the prompt to close Erlang shell.

Next, installing ejabberd. We need to add a Jabber domain name to /etc/hosts first.

The first line in /etc/hosts after editing will look something like (substitute zephyr to anything you like):

127.0.0.1 localhost.localdomain localhost zephyr.local

Note: You will be using this domain name in many places, so please make note of it; I will be using zephyr.local in this post going forward.

Download, unpack and launch ejabberd installer:

wget http://www.process-one.net/downloads/ejabberd/2.0.5/ejabberd-2.0.5-linux-x86-installer.bin.gz
gunzip ejabberd-2.0.5-linux-x86-installer.bin.gz
chmod 755 ejabberd-2.0.5-linux-x86-installer.bin
./ejabberd-2.0.5-linux-x86-installer.bin

Make default choices during installation; when asked about ejabberd server domain, enter host name you previously added to /etc/hosts (i.e. zephyr.local). Note default installation directory ( /opt/ejabberd-2.0.5 by default), you will need it shortly.

Once installation is completed, edit ejabberdctl.cfg by changing the very last line from:

ERLANG_NODE=ejabberd@localhost
to
ERLANG_NODE=ejabberd@zephyr.local

This is an important change, especially if you're planning to reuse your EC2 instance; I will omit the explanation for now.

Now, let's make sure ejabberd is good:

cd /opt/ejabberd-2.0.5/bin
./ejabberdctl start
./ejabberdctl status

You should see:
Node 'ejabberd@zephyr.local' is started. Status: started
ejabberd is running


Next step - configure external XMPP component on ejabberd side.
Open up ejabberd.cfg in /opt/ejabberd-2.0.5/conf with your editor (you may want to make a backup copy first), locate the line starting with {5269, ejabberd_s2s_in
and add service description after the closing curly bracket:

  {5269, ejabberd_s2s_in, [
                           {shaper, s2s_shaper},
                           {max_stanza_size, 131072}
                          ]},

   {7047, ejabberd_service,
   [{hosts, ["test1.zephyr.local"],
         [{password, "secret"}]}]},

  %%
  %% ejabberd_service: Interact with external components (transports...)
  %%


Save ejabberd.cfg and restart ejabberd:

/opt/ejabberd-2.0.5/bin/ejabberdctl restart
/opt/ejabbberd-2.0.5/bin/ejabberd status

Provided ejabberd has restarted successfully, let's create a bot account (it's curently being used to help with weazard registration):

/opt/ejabberd-2.0.5/bin/ejabberd register _weazard_bot zephyr.local weazard

We're done with ejabberd for now. Time to checkout our demo code:

mkdir weazard
cd weazard

svn co https://tagsahead.svn.beanstalkapp.com/erltwit/trunk/web
svn co https://tagsahead.svn.beanstalkapp.com/xmpp_component/trunk

Review config.js in web/js and adjust it, if needed.

Review weazard.app in xmpp_component/trunk/ebin and adjust it.

We will use nginx as our web server:

apt-get install nginx

Below is configuration fragment that has to be included (directly to /etc/nginx/nginx.conf or through include  directive):

server {
        listen 8000;
        root /home/nginx/web/;

        location /http-bind {
                proxy_pass http://localhost:5280/http-bind/;
        }
}

We'll need to move our web application to the place where nginx can access it, i.e. /home/nginx/web
The highlighted piece is necessary because due to Javascript security Strophe can't directly reach ejabberd's HTTP binding service.

 Update: Starting ejabberd v 2.1.3 configuring proxy is not necessary, so you won't have to include the highlighted text above. You might still want to configure the proxy if your web clients are behind firewall with the BOSH port (default 5280) disabled. See http://rfid-ale.blogspot.com/2010/04/end-of-cross-domain-hassle-for-bosh.html for details.


We're done!

Running demo

Start ejabberd:

/opt/ejabberd-2.0.5/ejabberdctl start (or restart, if it's already running);

Start nginx:

sudo /etc/init.d/nginx start (or restart, if it's already running);

Start weazard XMPP component:

cd  ~/weazard/xmpp_component/ebin

erl -sname main -setcookie weazard

At the prompt, type:

1>application:start(weazard).

If you see the line:

Component : "test1.zephyr.local" started.

in the output,  then weazard has managed to connect to ejabberd and is ready to push data to subscribers.

You're now ready to launch your browser and try:

http://yourhostname:8000/weazard.html

Can I congratulate you with running demo? I hope so, but if you are not there yet, fear not!  As you can see, a lot of things can potentially go wrong. There's plenty of resources on configuring ejabberd/BOSH/nginx, and I'm here to help to the best of my abilities.

Intermission: Erlang production nirvana

Well, almost. The XMPP Web demo project, despite using some of the best tools available today, is still a typical software project with bugs popping up here and there. Since it went on air couple of days back, one of the server-side components kept going down after few hours of up-time. The Erlang node it was running on would suddenly cease to exist without a sound or even a crash dump left anywhere. The cause is not yet completely clear, and it's going to take some time to figure out and fix.
This would probably be a show-stopper for any production system I had ever encountered. Not for the system made with Erlang! While I'm working on the bug, I'd like to keep my demo up and running without having to monitor that failing component, so here's how:

• The node that runs component in question will be started from master node using slave:start;
• Once the node is started, the component application will be launched using rpc:call
• The node will be monitored with erlang:monitor_node; in case monitor process detects node going down, we'll repeat it all again.

The code is only a few lines long. It works with packaged application, but can be easily adapted to Module:Function form. Note that there is a stop_node/2 function, in case you do want to stop the application without exiting your Erlang shell.
"Normal" crashes (the ones that don't bring node down) have been already taken care of with Erlang supervisor. So nothing can now stop you from trying the demo any time of day :-)

Update: I should have mentioned that another option to keep your node up is to use -heart switch of erl command. Coding solution allows more control though, for instance you may want to be able to run some checks before restarting node, whereas -heart will restart unconditionally.

Real-time Web applications with XMPP: working demo

Yes, this is a mid-size demo web app with XMPP, more specifically, combination of Strophe, exmpp, ejabberd on Amazon EC2. The purpose of this exercise was to create a "proof of concept" and prepare myself for more serious projects. The demo is now available online , go ahead and check it out. You could either use predefined account (username: acc1, password : acc1), or register your own. Choose weather stations from right-hand panel and watch your browser being updated in a real time with data coming from (fake) weather service. Clicking on station's icon will toggle subscription, green arrow icon meaning the account is currently subscribed to that station. You will also see (normally quickly passing) blue icon meaning station is waiting for updated subscription status from the server. The demo is in early beta (if I can say this about demo at all), some features, such as live network status and live support are coming soon.



The app is very dynamic. Try this: open 2-3 windows with the same account and watch subscription changes you're making in one window instantly showing up in others. This is a nice feature that could be used for real-time collaboration. It's possible to imagine a number of applications that need real-time updates and collaborations of that sort: trader floors, stock tickers, asset tracking, order fulfillment, arrivals/departures, ticket reservations, chats, tweets, you name it.
It's not surprising that Google have chosen XMPP as core protocol for the Google Wave.
What's the big deal, you might ask? We have been using dynamic Ajax-based apps for a while now. This is a better Ajax, though, as it uses XMPP over HTTP (BOSH) protocol. You can read about it in details here and here. Briefly, most Ajax applications are using polling, which is poorly scalable and resource-hungry technique. Javascript libraries that employ BOSH, on the other hand, are using Ajax with "semi-permanent" connections, keeping them alive until server really has data to send to the client.
And, with Javascript libraries supporting BOSH, such as Strophe or jsJac, plus JQuery, "We Don't Need No Stinkin' Web Frameworks" :-)

The XMPP-specific features having been implemented/used in the demo:

• in-band registration;
• service discovery;
• roster push handlng;
• presence notifications and subscription handling.

In the next few days, I'll go over setting all things up on EC2, which is no small feat, at least it wasn't for me :-) I'm also planning to add some more features, such as self-restoring after disconnections and crashes, both on server and browser sides. As usual, your input is appreciated and welcomed.

The source (HTML/JS/Erlang) is available here.

P.S. Please remember this is a fake data, so don't use it for planning your weekends, although I have taken care not to show "Heavy Snow" icon during summer. I might be not that far off from official weather feeds, though :-) One day I might just take a real feed and turn that demo into something useful :-)