The gen_client API overview

I've been using gen_client in few real projects, which helped to weed out many bugs, shape the design and evaluate features. I feel that it's now ready for public usage, so I would like to briefly explain some concepts used by gen_client and show how they might help to make XMPP client programming easier.

Starting the client.

start(Jid, Host, Port, Password) 
start(Jid, Host, Port, Password, Options)
start(Account, Domain, Host, Port, Password, Options)
start(Account, Domain, Resourse, Host, Port, Password, Options)

Above calls create a client session and return {ok, ClientRef} tuple in case everything went well. ClientRef (which is a Pid of gen_server process associated with the client session) then can be used to make gen_client API calls. For example:

{ok, Client} = gen_client:start("gen_client@jabber.ru", "jabber.ru", 5222, "test",
[{debug, true}, {presence, {true, "I'm online."}}, {reconnect, 15000}]),
gen_client:add_plugin(Client, disco_plugin, [test_disco, []]).

Options describe different aspect of the client, such as (default values go first):

• {debug, false | true} - printing out debug info;
• {presence, {true, Msg} | false} - should the client send a presence, and if yes, specify the presence message;
• {reconnect, Timeout} - should the client reconnect after losing the connection, and the timeout for reconnection;
• {log_in, true | false} - should the client log in automatically; useful when you want to choose between logging in and registration;
• more to come... 

Handlers

add_handler(Client, Handler)
add_handler(Client, Handler, Priority)
remove_handler(Client, HandlerKey)


Handler is a callback function that handles incoming stanzas. Handlers can be added to (or removed from) the client session at will. add_handler/2,3 calls return the key value which could be used to remove the handler later, if needed. Each handler could be assigned a priority at the time it's added to the client. When stanza is received, it will be applied to the chain of handlers according to handlers' priority and/or output. For example, a handler can interrupt the chain of subsequent handler calls by returning stop.

Plugins

add_plugin(Client, Plugin, Args)
add_plugin(Client, Plugin, Args, Priority)
remove_plugin(Client, PluginKey) 


Theoretically handlers should be sufficient for processing of any incoming stanzas.
However, in many cases handling of stanzas involves fair amount of repetitive code. For example, responses to discovery requests (disco_info and disco_items) have to have certain headers, workflow sequences that utilize ad-hoc commands need to keep state etc. Plugins are meant to encapsulate such  common processing blocks, letting developers to focus on specifics.
To be a plugin, the module has to implement gen_client_plugin behavior, namely init/1, terminate/1 and handle/3 functions.  The gen_client:add_plugin/3 makes  The idea is that handle/3 will contain the bulk of boilerplate code, at the same time letting the plugin users to customize it by passing arguments to either init/1 or handle/3 functions.
If above explanation sounds somewhat obscure, hopefully looking at the code of disco_plugin and test example in test_gen_client:test/0 will make things a bit clearer. The other available plugin is  adhoc_plugin, which I am planning to talk about in more details in following posts.

Blocking and non-blocking requests

XMPP messaging is asynchronous, and that's great. However, quite often your code needs to get an immediate response before the flow can continue. The examples are discovery, ping and command requests, pubsub retrival requests and many others. Of course, it's possible to allocate a callback for the expected response, but this does make the code harder to write and understand.
gen_client supports both asynchronous and synchronous requests. The former is simply a wrapper of exmpp:send_packet/2:

send_packet(ClientRef, Packet)

where ClientRef is a client session reference created by one of gen_client:start functions (see above).

The synchronous request is a bit more interesting. Here are definitions:

send_sync_packet(Client, Packet, Timeout)
send_sync_packet(Client, Packet, Trigger, Timeout)

Here's how it works:

The calling process sends the packet and timeout value to the client session process. The client session process creates a temporary handler that would "look" for "matching" incoming message, and then sends the packet to the server and waits for the response to arrive within specified timeout.
How do we describe "matching"? By defining a "trigger" function that tests incoming message against some condition. For requests, "matching" means that response message will have the same id attribute as the request. So send_sync_packet/3 does just that: creates a function that looks at id attribute of incoming message and if it happens to be equal to request id, signals the client session that response has arrived. 
So as we can see, send_sync_packet/3 could be treated as "send iq and wait for response", which makes it somewhat close to sendIQ function in Strophe.js
In case your "matching criteria" is different from simple id matching, you can use send_sync_packet/4 that allows to define arbitrary "trigger" function.

Important: even though the calling process will be blocked, the active handlers will still work, because they will be called in separate (exmpp controlling) process. In other words, while your main process waits for response to a particular request, another kinds of incoming messages will still be handled in parallel.

That's all for now. Make sure to check out the code and please let me know what do you think.

The end of cross-domain hassle for BOSH

    Exciting news to anyone who is using BOSH with ejabberd - since release 2.1.3 you don't have to proxy your http-bind link anymore. Imagine no nginx configuration, no Tape, just point exactly to where your http-bind link is. This, I believe, albeit looking rather insignificant news, will bring XMPP development to the new level of acceptance.
I learned first about it from Jack Moffitt's site. When I had a chance, I installed ejabberd 2.1.3 on AWS and fired a test web page that logs in using Strophe right from my local box. The habit of crafting nginx configuration first has developed over time so strong that I didn't actually expect it to work. Fortunately, ejabberd proved me wrong - it works, and much better and faster without proxying (which is logical to expect).
   So if you read tutorial on how to set up BOSH with ejabberd, you can now skip the whole topic talking about nginx, cross-domain limitations and proxy. In your Strophe code, instead of doing, for example:
var BOSH_SERVICE = "/http-bind";
you do:
var BOSH_SERVICE = "http://your_ejabberd_server:5280/http-bind";

   The consequences are many, the most important I think is how easy it becomes to embed BOSH anywhere.

gen_client behaviour for building XMPP clients in Erlang.

 The gen_client project aims to provide a structured way to write XMPP client code in Erlang.  The framework heavily relies on exmpp, but also borrows some ideas from my favorite Strophe javascript library. The objective of the project is to create a set of generic behaviours  and let a client developer to "fill in  the blanks", i.e. to implement callback methods pretty much in the same fashion the code based on OTP/Erlang behaviours is written.
Why not to just use exmpp, one might ask? Sure you can. However, going from basic examples 
to decently capable client code is not so easy in exmpp. Motivation behind gen_client is to make coding XMPP in Erlang as effortless as Strophe does for Javascript.
One example of what I mean by "efortless" is how exmpp controls handling of incoming stream. By default, sending and receiving stanzas happens in a single process. Clearly, this is not very useful unless your XMPP client is happy with "question-answer" flow (as in echo_client.erl example from exmpp distribution), as opposed to asynchronous flow.
Of course, exmpp has means to assign a separate process for handling incoming stream (exmpp_session:set_controlling_process/2). However, it would be nice to have this as default, which is what gen_client does.
Continuing with this,  sometimes you may need a synchronous handling. For instance, you'd have to search through the whole tree of pubsub nodes, do some calculations and send results elsewhere. While this kind of task can be coded using callbacks, it does make coding much harder to deal with compared to sequential style. With gen_client, you can choose between asynchronous (gen_client:send_packet) and synchronous (gen_client:send_sync_packet) requests.

And of course, each incoming stanza will be handled by gen_client in a spawned process, so your client can do many things at once - we are using Erlang for the reason, right?

To start with gen_client, write your module that implements gen_client behaviour. And off you go:
gen_client:start(Username,  Domain,  Host,  Port,  Password, Module, [ModuleArgs]).

Summary of features that are already there:

• Simultaneous handling of multiple incoming stanzas;
• Synchronous and asynchronous requests;
• Attaching IQ/presence/message handlers at runtime (somewhat similar to Strophe's addHandler style);
• Support for ad-hoc commands (XEP-0050) and service discovery (XEP-0030);
• Compatibility with exmpp and hence ability to reuse its codebase.
  

Documentation and examples will follow time permitting. This is the work in progress, mostly experimental, so please use with caution. Usual disclaimers are in place. Please share your thoughts and ask questions, if any. This code is being used in real projects, so I appreciate any feedback from you as means of moving gen_client to a production quality.

Links:

• gen_client on github 
• exmpp version used by gen_client

XMPP web demo: setting it up

I will describe the XMPP demo setup on AWS EC2, starting with Alestic Ubuntu karmic image (ami-19a34270). Hopefully, you should be able to adjust the process to your environment of choice.

Setting up

Brace yourself, that's going to be a long one. Major components to install and configure are:

• Erlang environment
• exmpp
• ejabberd
• nginx
• and, of course, weazard itself
  

So, provided you have running instance of ami-19a34270, let's  ssh to it and start by:

sudo apt-get update && sudo apt-get upgrade

We'll need svn to checkout some stuff later:


apt-get install subversion


Then install Erlang (version R12B5 or newer; karmic has R13B01):
 

apt-get install erlang

Now, let's move on to exmpp.  First, we need to install some libraries that exmpp is using for XML support:

apt-get install libxml2-dev

or, alternatively:

apt-get install libexpat1-dev


Update: the demo will work fine with libxml2, however, at the time of writing (Nov 07, 2009), if you're planning to use exmpp for other purposes, particularly, exmpp_session module, you're advised to install libexpat1-dev instead. See https://support.process-one.net/si/jira.issueviews:issue-html/EXMPP-16/EXMPP-16.html for details.

apt-get install libssl-dev
apt-get install pkg-config


Now, download the exmpp distribution and build:

wget http://download.process-one.net/exmpp/exmpp-0.9.1-r857.tar.gz
gunzip exmpp-0.9.1-r857.tar.gz
tar -xvf exmpp-0.9.1-r857.tar 
./configure
make
sudo make install


At this point you may want to check if exmpp is in working condition. Start erl and at the prompt type: 

1>exmpp:start().
ok
2>exmpp_xml:start_parser().

If response is similar to
{xml_parser,[{max_size,infinity},
             {root_depth,0},
             {names_as_atom,true},
             {emit_endtag,false}],
            #Port<0.747>},

then everything went fine and you can type q(). at the prompt to close Erlang shell.

Next, installing ejabberd. We need to add a Jabber domain name to /etc/hosts first.

The first line in /etc/hosts after editing will look something like (substitute zephyr to anything you like):

127.0.0.1 localhost.localdomain localhost zephyr.local

Note: You will be using this domain name in many places, so please make note of it; I will be using zephyr.local in this post going forward.

Download, unpack and launch ejabberd installer:

wget http://www.process-one.net/downloads/ejabberd/2.0.5/ejabberd-2.0.5-linux-x86-installer.bin.gz
gunzip ejabberd-2.0.5-linux-x86-installer.bin.gz
chmod 755 ejabberd-2.0.5-linux-x86-installer.bin
./ejabberd-2.0.5-linux-x86-installer.bin

Make default choices during installation; when asked about ejabberd server domain, enter host name you previously added to /etc/hosts (i.e. zephyr.local). Note default installation directory ( /opt/ejabberd-2.0.5 by default), you will need it shortly.

Once installation is completed, edit ejabberdctl.cfg by changing the very last line from:

ERLANG_NODE=ejabberd@localhost
to
ERLANG_NODE=ejabberd@zephyr.local

This is an important change, especially if you're planning to reuse your EC2 instance; I will omit the explanation for now.

Now, let's make sure ejabberd is good:

cd /opt/ejabberd-2.0.5/bin
./ejabberdctl start
./ejabberdctl status

You should see:
Node 'ejabberd@zephyr.local' is started. Status: started
ejabberd is running


Next step - configure external XMPP component on ejabberd side.
Open up ejabberd.cfg in /opt/ejabberd-2.0.5/conf with your editor (you may want to make a backup copy first), locate the line starting with {5269, ejabberd_s2s_in
and add service description after the closing curly bracket:

  {5269, ejabberd_s2s_in, [
                           {shaper, s2s_shaper},
                           {max_stanza_size, 131072}
                          ]},

   {7047, ejabberd_service,
   [{hosts, ["test1.zephyr.local"],
         [{password, "secret"}]}]},

  %%
  %% ejabberd_service: Interact with external components (transports...)
  %%


Save ejabberd.cfg and restart ejabberd:

/opt/ejabberd-2.0.5/bin/ejabberdctl restart
/opt/ejabbberd-2.0.5/bin/ejabberd status

Provided ejabberd has restarted successfully, let's create a bot account (it's curently being used to help with weazard registration):

/opt/ejabberd-2.0.5/bin/ejabberd register _weazard_bot zephyr.local weazard

We're done with ejabberd for now. Time to checkout our demo code:

mkdir weazard
cd weazard

svn co https://tagsahead.svn.beanstalkapp.com/erltwit/trunk/web
svn co https://tagsahead.svn.beanstalkapp.com/xmpp_component/trunk

Review config.js in web/js and adjust it, if needed.

Review weazard.app in xmpp_component/trunk/ebin and adjust it.

We will use nginx as our web server:

apt-get install nginx

Below is configuration fragment that has to be included (directly to /etc/nginx/nginx.conf or through include  directive):

server {
        listen 8000;
        root /home/nginx/web/;

        location /http-bind {
                proxy_pass http://localhost:5280/http-bind/;
        }
}

We'll need to move our web application to the place where nginx can access it, i.e. /home/nginx/web
The highlighted piece is necessary because due to Javascript security Strophe can't directly reach ejabberd's HTTP binding service.

 Update: Starting ejabberd v 2.1.3 configuring proxy is not necessary, so you won't have to include the highlighted text above. You might still want to configure the proxy if your web clients are behind firewall with the BOSH port (default 5280) disabled. See http://rfid-ale.blogspot.com/2010/04/end-of-cross-domain-hassle-for-bosh.html for details.


We're done!

Running demo

Start ejabberd:

/opt/ejabberd-2.0.5/ejabberdctl start (or restart, if it's already running);

Start nginx:

sudo /etc/init.d/nginx start (or restart, if it's already running);

Start weazard XMPP component:

cd  ~/weazard/xmpp_component/ebin

erl -sname main -setcookie weazard

At the prompt, type:

1>application:start(weazard).

If you see the line:

Component : "test1.zephyr.local" started.

in the output,  then weazard has managed to connect to ejabberd and is ready to push data to subscribers.

You're now ready to launch your browser and try:

http://yourhostname:8000/weazard.html

Can I congratulate you with running demo? I hope so, but if you are not there yet, fear not!  As you can see, a lot of things can potentially go wrong. There's plenty of resources on configuring ejabberd/BOSH/nginx, and I'm here to help to the best of my abilities.

XMPP web demo update

Yesterday was a big day, I got over 100 visitors on my blog, and many of you have registered and tried the demo. Was it because of me posting about demo on identi.ca and metajack.im? Anyway, thank you all for taking time and trying, I have gotten strong motivation to keep up with XMPP topic because of your interest.

Couple of things that I wanted to mention: first, some of newly registered users seemed to drop out really quick. I'm guessing that was because they weren't able to see much action from the demo. I should have explained that if you have small number of subscriptions, chances are you'd have to wait for few minutes before seeing any data update. That's because the update generator chooses one station at a time every 15 seconds, so if you have only couple stations chosen, it'd take in average about 2 mins for updates to show up, given there are currently 17 stations total. So the hint is, if you want a real entertainment, subscribe at least to 8-9 stations, then you'll see it all rolling full speed.
There are also few cases where people dropped out not even finishing registration (initial greeting from Jabber still sits there for such accounts). I would be curious to know, what had happened there? Was it a slow script, browser crash or something else? I'd appreciate any feedback you could give.
And speaking about feedback, you can now easily send a message to a friendly support team (that would be me) from within demo! Just click the button on Support panel and submit the 2-field form, I'll be sure to read and respond. Note that all this done with Javascript one-liner, I didn't have to touch any of server code, all that goodness given for free by ejabberd and Strophe.

XMPP web project walk-through

Now that the XMPP demo is online and pretty stable, I will try to explain what is it actually doing. Should have done it at the time of putting it online, but it's never too late to do a right thing.

The objective.

The objective of demo is to show how information can be delivered to browser clients in real time using XMPP/BOSH technique.

The functionality.

The demo application gives prospective clients ability to subscribe to one or more (imaginary) weather stations and receive (made-up) weather alerts in real-time. The client can subscribe/unsubscribe to/from any of available weather stations. The subscription will be associated with client's account and used for the future sessions, so the client is required to register account first time she's using the demo.
NOTE: You may use existing shared account (username: shared, password: shared) in case you do not wish to register. Registering is safe, though, as it creates a Jabber account on local domain, which is not accessible from outside. If you decide to use shared account, expect other people to alter your subscriptions. You may actually watch someone turning your subscriptions on and off, which represents a nice case of real-time collaboration :-)

Visual representation and controls.

Right-hand panel contains weather station locations organized as country -> region -> city hierarchy. The icon next to station location denotes its subscription status, as follows:

- client is not subscribed to the station;

- client is subscribed to the station;

- the request to turn subscription on/off has been sent to the server.

Clicking on station name will send request to turn subscription on/off for the logged in client.

Middle panel contains weather updates, one per subscribed station. Once update is received from the station, corresponding section will show up. Only last update is shown. The client can click on any section to unfold detailed information about last update received from the station.

The moving parts (architecture, that is :-).

On the client side we have HTML page that employs Strophe.js library. Strophe provides communication with XMPP server using BOSH protocol.

On the server side:

• ejabberd - XMPP server;

• weazard - Erlang application that acts as XMPP component, using gen_component implementation; it exposes weather stations for the service discovery and manages subscription requests by routing weather updates to subscribers;
  

• weather update generator - generates weather alerts every 15 seconds by randomly choosing weather station, temperature and condition.

The flow.

The client signs in, triggering Strophe to start service discovery procedure by sending "disco" requests to weazard in order to find out what weather stations are available. Strophe then receives "disco" responses through its callbacks and builds the treeview that reflects weazard's service structure (i.e. country->region->city hierarchy that resides on the right-hand panel). After the treeview has been finished, Strophe advertises client's presence, which in turn triggers XMPP server to route presence to all client's subscribed stations. As stations' addressing includes weazard's JID as host, weazard takes over and sends presence from each of subscribed stations back to Strophe. Strophe then uses presence callbacks to show subscription state (all those red and green "transmission" icons in right-hand panel and message sections in middle panel) for all discovered stations in the client's view.
From this moment on, Strophe listens to weather alert messages and updates corresponding sections in the middle panel as they go.
On the server side, weazard receives alerts from weather update generator, turns them into XMPP messages and routes them to subscribers via XMPP server.

Advantages of using XMPP/BOSH web vs. traditional web frameworks.

The most obvious advantage is a real-time interaction. This doesn't directly have to do with XMPP, but rather is provided by BOSH protocol. Very similar techniques have been around for quite some time, for instance, Comet or pushlets. BOSH is currently supported by most well-known XMPP server, which makes it natural to use with XMPP.
Another important thing is ability to reuse XMPP server features. For instance, there is no need to have separate database of registered users and their subscriptions, as this is already maintained by XMPP server. All changes in user's presence (i.e. logged in/out, disconnected) and component presence (restart, disconnection etc.) are also taken care of. It is possible to easily implement IM features, such as live support or chat between users.
XMPP/BOSH also enables asynchronous messaging style. This means that the code that submits the request will never block on waiting for the response; instead the response will be handled by a callback function. In practical terms, this makes web application more responsive and user friendly. Technically, the same behavior can be implemented without XMPP, it's just that asynchronous messaging is a natural way of life for XMPP, as opposed to a web server that is more accustomed to sequential "request-response" interactions.

Word of caution.

Although the demo has some things done right, in case you decide to reuse the code, you should be aware that the approach being used for serving subscriptions is not suitable for every situation. Let me explain: if 500 clients will subscribe to a single weather station, the component will send 500 nearly identical XML messages to the Jabber server every time the update happens. This is not good, and should rather be implemented with PubSub or MUC. If, on the other hand, you have one-to-one subscription (say, client wants to get updates from her Twitter account), then you're fine.

I will keep fixing bugs and adding new features to the demo. Your feedback is most welcomed.

Real-time Web applications with XMPP: working demo

Yes, this is a mid-size demo web app with XMPP, more specifically, combination of Strophe, exmpp, ejabberd on Amazon EC2. The purpose of this exercise was to create a "proof of concept" and prepare myself for more serious projects. The demo is now available online , go ahead and check it out. You could either use predefined account (username: acc1, password : acc1), or register your own. Choose weather stations from right-hand panel and watch your browser being updated in a real time with data coming from (fake) weather service. Clicking on station's icon will toggle subscription, green arrow icon meaning the account is currently subscribed to that station. You will also see (normally quickly passing) blue icon meaning station is waiting for updated subscription status from the server. The demo is in early beta (if I can say this about demo at all), some features, such as live network status and live support are coming soon.



The app is very dynamic. Try this: open 2-3 windows with the same account and watch subscription changes you're making in one window instantly showing up in others. This is a nice feature that could be used for real-time collaboration. It's possible to imagine a number of applications that need real-time updates and collaborations of that sort: trader floors, stock tickers, asset tracking, order fulfillment, arrivals/departures, ticket reservations, chats, tweets, you name it.
It's not surprising that Google have chosen XMPP as core protocol for the Google Wave.
What's the big deal, you might ask? We have been using dynamic Ajax-based apps for a while now. This is a better Ajax, though, as it uses XMPP over HTTP (BOSH) protocol. You can read about it in details here and here. Briefly, most Ajax applications are using polling, which is poorly scalable and resource-hungry technique. Javascript libraries that employ BOSH, on the other hand, are using Ajax with "semi-permanent" connections, keeping them alive until server really has data to send to the client.
And, with Javascript libraries supporting BOSH, such as Strophe or jsJac, plus JQuery, "We Don't Need No Stinkin' Web Frameworks" :-)

The XMPP-specific features having been implemented/used in the demo:

• in-band registration;
• service discovery;
• roster push handlng;
• presence notifications and subscription handling.

In the next few days, I'll go over setting all things up on EC2, which is no small feat, at least it wasn't for me :-) I'm also planning to add some more features, such as self-restoring after disconnections and crashes, both on server and browser sides. As usual, your input is appreciated and welcomed.

The source (HTML/JS/Erlang) is available here.

P.S. Please remember this is a fake data, so don't use it for planning your weekends, although I have taken care not to show "Heavy Snow" icon during summer. I might be not that far off from official weather feeds, though :-) One day I might just take a real feed and turn that demo into something useful :-)

gen_component project: service discovery with exmpp

The gen_component behavior is growing up and is making more sense to me as the XMPP component project is moving along. I can even see gen_component rivaling gen_server in the nearest future :-). Jokes aside, it will have handle_iq, handle_message, handle_presence and possibly more callbacks with helpers glueing it all together, in a fashion similar to OTP behaviors.
By the way, the component doesn't even have to be a process, depending on what exactly your component does. So far I managed to avoid maintaining conversational state, but will have to introduce it going forward. For instance, to tell the registered user from a new one, having state seems to be a must, doesn't it? Still, should I use Mnesia for it, rather than process state?

The project is now having service discovery written using exmpp. The imaginary weather alert service exposes itself through disco_info/1 and disco_items/1 (which are now part of gen_component).

That's how it looks like in Psi:

As you can see, there are two weather alerts running, each one having few nodes for specific weather alerts. Are you thinking of stock tickers, twits etc. ? Yes, everything is possible with gen_component and XMPP :-)

Check out the source code and instructions.
Next in line is in-band registration.
I'll be doing web interface with Strophe or jsJac as well once I have a chance.
I would be interested in your thoughts and ideas on gen_component.