XMPP web demo update

Yesterday was a big day, I got over 100 visitors on my blog, and many of you have registered and tried the demo. Was it because of me posting about demo on identi.ca and metajack.im? Anyway, thank you all for taking time and trying, I have gotten strong motivation to keep up with XMPP topic because of your interest.

Couple of things that I wanted to mention: first, some of newly registered users seemed to drop out really quick. I'm guessing that was because they weren't able to see much action from the demo. I should have explained that if you have small number of subscriptions, chances are you'd have to wait for few minutes before seeing any data update. That's because the update generator chooses one station at a time every 15 seconds, so if you have only couple stations chosen, it'd take in average about 2 mins for updates to show up, given there are currently 17 stations total. So the hint is, if you want a real entertainment, subscribe at least to 8-9 stations, then you'll see it all rolling full speed.
There are also few cases where people dropped out not even finishing registration (initial greeting from Jabber still sits there for such accounts). I would be curious to know, what had happened there? Was it a slow script, browser crash or something else? I'd appreciate any feedback you could give.
And speaking about feedback, you can now easily send a message to a friendly support team (that would be me) from within demo! Just click the button on Support panel and submit the 2-field form, I'll be sure to read and respond. Note that all this done with Javascript one-liner, I didn't have to touch any of server code, all that goodness given for free by ejabberd and Strophe.

XMPP web project walk-through

Now that the XMPP demo is online and pretty stable, I will try to explain what is it actually doing. Should have done it at the time of putting it online, but it's never too late to do a right thing.

The objective.

The objective of demo is to show how information can be delivered to browser clients in real time using XMPP/BOSH technique.

The functionality.

The demo application gives prospective clients ability to subscribe to one or more (imaginary) weather stations and receive (made-up) weather alerts in real-time. The client can subscribe/unsubscribe to/from any of available weather stations. The subscription will be associated with client's account and used for the future sessions, so the client is required to register account first time she's using the demo.
NOTE: You may use existing shared account (username: shared, password: shared) in case you do not wish to register. Registering is safe, though, as it creates a Jabber account on local domain, which is not accessible from outside. If you decide to use shared account, expect other people to alter your subscriptions. You may actually watch someone turning your subscriptions on and off, which represents a nice case of real-time collaboration :-)

Visual representation and controls.

Right-hand panel contains weather station locations organized as country -> region -> city hierarchy. The icon next to station location denotes its subscription status, as follows:

- client is not subscribed to the station;

- client is subscribed to the station;

- the request to turn subscription on/off has been sent to the server.

Clicking on station name will send request to turn subscription on/off for the logged in client.

Middle panel contains weather updates, one per subscribed station. Once update is received from the station, corresponding section will show up. Only last update is shown. The client can click on any section to unfold detailed information about last update received from the station.

The moving parts (architecture, that is :-).

On the client side we have HTML page that employs Strophe.js library. Strophe provides communication with XMPP server using BOSH protocol.

On the server side:

• ejabberd - XMPP server;

• weazard - Erlang application that acts as XMPP component, using gen_component implementation; it exposes weather stations for the service discovery and manages subscription requests by routing weather updates to subscribers;
  

• weather update generator - generates weather alerts every 15 seconds by randomly choosing weather station, temperature and condition.

The flow.

The client signs in, triggering Strophe to start service discovery procedure by sending "disco" requests to weazard in order to find out what weather stations are available. Strophe then receives "disco" responses through its callbacks and builds the treeview that reflects weazard's service structure (i.e. country->region->city hierarchy that resides on the right-hand panel). After the treeview has been finished, Strophe advertises client's presence, which in turn triggers XMPP server to route presence to all client's subscribed stations. As stations' addressing includes weazard's JID as host, weazard takes over and sends presence from each of subscribed stations back to Strophe. Strophe then uses presence callbacks to show subscription state (all those red and green "transmission" icons in right-hand panel and message sections in middle panel) for all discovered stations in the client's view.
From this moment on, Strophe listens to weather alert messages and updates corresponding sections in the middle panel as they go.
On the server side, weazard receives alerts from weather update generator, turns them into XMPP messages and routes them to subscribers via XMPP server.

Advantages of using XMPP/BOSH web vs. traditional web frameworks.

The most obvious advantage is a real-time interaction. This doesn't directly have to do with XMPP, but rather is provided by BOSH protocol. Very similar techniques have been around for quite some time, for instance, Comet or pushlets. BOSH is currently supported by most well-known XMPP server, which makes it natural to use with XMPP.
Another important thing is ability to reuse XMPP server features. For instance, there is no need to have separate database of registered users and their subscriptions, as this is already maintained by XMPP server. All changes in user's presence (i.e. logged in/out, disconnected) and component presence (restart, disconnection etc.) are also taken care of. It is possible to easily implement IM features, such as live support or chat between users.
XMPP/BOSH also enables asynchronous messaging style. This means that the code that submits the request will never block on waiting for the response; instead the response will be handled by a callback function. In practical terms, this makes web application more responsive and user friendly. Technically, the same behavior can be implemented without XMPP, it's just that asynchronous messaging is a natural way of life for XMPP, as opposed to a web server that is more accustomed to sequential "request-response" interactions.

Word of caution.

Although the demo has some things done right, in case you decide to reuse the code, you should be aware that the approach being used for serving subscriptions is not suitable for every situation. Let me explain: if 500 clients will subscribe to a single weather station, the component will send 500 nearly identical XML messages to the Jabber server every time the update happens. This is not good, and should rather be implemented with PubSub or MUC. If, on the other hand, you have one-to-one subscription (say, client wants to get updates from her Twitter account), then you're fine.

I will keep fixing bugs and adding new features to the demo. Your feedback is most welcomed.

Intermission: Erlang production nirvana

Well, almost. The XMPP Web demo project, despite using some of the best tools available today, is still a typical software project with bugs popping up here and there. Since it went on air couple of days back, one of the server-side components kept going down after few hours of up-time. The Erlang node it was running on would suddenly cease to exist without a sound or even a crash dump left anywhere. The cause is not yet completely clear, and it's going to take some time to figure out and fix.
This would probably be a show-stopper for any production system I had ever encountered. Not for the system made with Erlang! While I'm working on the bug, I'd like to keep my demo up and running without having to monitor that failing component, so here's how:

• The node that runs component in question will be started from master node using slave:start;
• Once the node is started, the component application will be launched using rpc:call
• The node will be monitored with erlang:monitor_node; in case monitor process detects node going down, we'll repeat it all again.

The code is only a few lines long. It works with packaged application, but can be easily adapted to Module:Function form. Note that there is a stop_node/2 function, in case you do want to stop the application without exiting your Erlang shell.
"Normal" crashes (the ones that don't bring node down) have been already taken care of with Erlang supervisor. So nothing can now stop you from trying the demo any time of day :-)

Update: I should have mentioned that another option to keep your node up is to use -heart switch of erl command. Coding solution allows more control though, for instance you may want to be able to run some checks before restarting node, whereas -heart will restart unconditionally.

Real-time Web applications with XMPP: working demo

Yes, this is a mid-size demo web app with XMPP, more specifically, combination of Strophe, exmpp, ejabberd on Amazon EC2. The purpose of this exercise was to create a "proof of concept" and prepare myself for more serious projects. The demo is now available online , go ahead and check it out. You could either use predefined account (username: acc1, password : acc1), or register your own. Choose weather stations from right-hand panel and watch your browser being updated in a real time with data coming from (fake) weather service. Clicking on station's icon will toggle subscription, green arrow icon meaning the account is currently subscribed to that station. You will also see (normally quickly passing) blue icon meaning station is waiting for updated subscription status from the server. The demo is in early beta (if I can say this about demo at all), some features, such as live network status and live support are coming soon.



The app is very dynamic. Try this: open 2-3 windows with the same account and watch subscription changes you're making in one window instantly showing up in others. This is a nice feature that could be used for real-time collaboration. It's possible to imagine a number of applications that need real-time updates and collaborations of that sort: trader floors, stock tickers, asset tracking, order fulfillment, arrivals/departures, ticket reservations, chats, tweets, you name it.
It's not surprising that Google have chosen XMPP as core protocol for the Google Wave.
What's the big deal, you might ask? We have been using dynamic Ajax-based apps for a while now. This is a better Ajax, though, as it uses XMPP over HTTP (BOSH) protocol. You can read about it in details here and here. Briefly, most Ajax applications are using polling, which is poorly scalable and resource-hungry technique. Javascript libraries that employ BOSH, on the other hand, are using Ajax with "semi-permanent" connections, keeping them alive until server really has data to send to the client.
And, with Javascript libraries supporting BOSH, such as Strophe or jsJac, plus JQuery, "We Don't Need No Stinkin' Web Frameworks" :-)

The XMPP-specific features having been implemented/used in the demo:

• in-band registration;
• service discovery;
• roster push handlng;
• presence notifications and subscription handling.

In the next few days, I'll go over setting all things up on EC2, which is no small feat, at least it wasn't for me :-) I'm also planning to add some more features, such as self-restoring after disconnections and crashes, both on server and browser sides. As usual, your input is appreciated and welcomed.

The source (HTML/JS/Erlang) is available here.

P.S. Please remember this is a fake data, so don't use it for planning your weekends, although I have taken care not to show "Heavy Snow" icon during summer. I might be not that far off from official weather feeds, though :-) One day I might just take a real feed and turn that demo into something useful :-)