Getting your hands wet with the power of Socket IO

4) Inducing Real Time Intelligence in Web Application

4.1) Getting your hands wet with the power of Socket IO and Node

4.2) Publish Subscribe model with Rabbit MQ, Socket IO and Node

4.3) Stand Alone RabbitMQ Publish-Subscribe Node applications

4.1) Getting your hands wet with the power of Socket IO and Node

When you are looking to build a real time application, you will need the power to push from the server side. Node.js brings to you this ability with a lot of ease when compared to any other technology that I am aware of.

This feature opens a whole new dimension to produce light weight real time web applications in the mobile space.

Some of the use cases we can exploit server side push are :

a) Chat applications
b) Twitter trends
c) Weather update mobile apps
d) Stock applications
e) Domain specific features such as what is being searched now.

I am going to illustrate the server side push and the client to server side push using sockets.

Here is the getting started example with Socket IO.

Getting Basic Express Project Running

1) Create a quick express project in your workspace by hitting "express socket_demo" from the cmd prompt. If this is your first time on express and need more details to get your environment set with express, please read my previous post here.

2) Let us now enter into the project created for you. Hit "cd socket_demo". Hit "npm install" to download the dependencies.

3) Test the application created by express by running it by entering: "node app.js" and open this on the browser on localhost:3000.

4) You should see the express text displayed.

Server Side Push Code

As the first part let us demonstrate the server side pushing to the client.

1) Open the app.js and enter the following code snippet:

/**  * Module dependencies.  */ var express = require('express')   , app = express()   , routes = require('./routes')   , user = require('./routes/user')   , http = require('http')   , path = require('path')   , server = require('http').createServer(app)   , io = require('socket.io').listen(server); app.configure(function(){   app.set('port', process.env.PORT || 3000);   app.set('views', __dirname + '/views');   app.set('view engine', 'jade');   app.use(express.favicon());   app.use(express.logger('dev'));   app.use(express.bodyParser());   app.use(express.methodOverride());   app.use(app.router);   app.use(express.static(path.join(__dirname, 'public'))); }); app.configure('development', function(){   app.use(express.errorHandler()); }); app.get('/', routes.index); app.get('/users', user.list); server.listen(app.get('port')); io.sockets.on('connection', function (socket) {                         var data="my data from server.....";                         console.log("emitting event now from server..........."+data);                         io.sockets.emit('myEvent', data); });

Summary of what is done in this code:

a) We add the socket.io module.
b) We define an event to execute on "connection" of the io.sockets.
c) We emit an event from the server side to the client side as "myEvent" sending a some value in "data" to the client.


2) Add the following to the index.jade.

doctype 5 html head             link(rel='stylesheet', href='style.css')             script(type='text/javascript',src='/socket.io/socket.io.js')             script(type='text/javascript',src='https://ajax.googleapis.com/ajax/libs/jquery/1.6.4/jquery.min.js')             script(type='text/javascript')                         var socket=io.connect('http://localhost:3000');                         socket.on('myEvent', function(data){                                     alert("data is ...."+data);                                     $("textarea#myArea").val(data);                         });             body h1= title textarea(id='myArea')

Summary of what is being done in this code:

a) We add the socket.io script which brings the support of sockets to the client side javascript.
b) We add the io.connect event like we did in the server side.
c) We add a listener to the event "myEvent" and do some basic UI operation to show the data pushed from server side to a text area.


Sending the event from client to server

1) Open the index.jade and modify as follows:

doctype 5 html head             link(rel='stylesheet', href='style.css')             script(type='text/javascript',src='/socket.io/socket.io.js')             script(type='text/javascript',src='https://ajax.googleapis.com/ajax/libs/jquery/1.6.4/jquery.min.js')             script(type='text/javascript')                         var socket=io.connect('http://localhost:3000');                         socket.on('myEvent', function(data){                                     $("textarea#myArea").val(data);                                                             });                         $(function () {                                     $("#submitButton").live("click", function(){                                                 var data=$("textarea#myText").val();                                                 socket.emit('clientEvent',data);                                     });                         });                         body h1= title             div                         textarea(id='myText', value="Type here")                         input(type='submit', value='Push', id="submitButton")             textarea(id='myArea')

Summary of what is being done here:

a) We are adding a div, with a text area and a button. The text within this "myText" will be sent to the server side on click of the button.

b) On the event handler of the button, we are emitting the data from the client the server using the socket.

2) Modify the app.js as follows:

/**  * Module dependencies.  */ var express = require('express')   , app = express()   , routes = require('./routes')   , user = require('./routes/user')   , http = require('http')   , path = require('path')   , server = require('http').createServer(app)   , io = require('socket.io').listen(server); app.configure(function(){   app.set('port', process.env.PORT || 3000);   app.set('views', __dirname + '/views');   app.set('view engine', 'jade');   app.use(express.favicon());   app.use(express.logger('dev'));   app.use(express.bodyParser());   app.use(express.methodOverride());   app.use(app.router);   app.use(express.static(path.join(__dirname, 'public'))); }); app.configure('development', function(){   app.use(express.errorHandler()); }); app.get('/', routes.index); app.get('/users', user.list); server.listen(app.get('port')); io.sockets.on('connection', function (socket) {                         var data="my data from server.....";                         console.log("emitting event now from server..........."+data);                         io.sockets.emit('myEvent', data);                         socket.on('clientEvent', function(data){                                     console.log("client data received ==>"+ data);                                     io.sockets.emit('myEvent', data);                         });                         });

Summary of the changes done:

a) We add a listner to the event to handle from the client ie "clientEvent". We have added an alert to see this in the output console as well.

b) Now the interesting thing we have done here is to emit this event from the server with the client sent data.

c) What you will observe is the client will have the data sent.

d) Open another browser instance and do same activity. Observe your first browser instance you will see the data from the other browser here.

We have simulated a chat room concept here :)

Aliters:

1) Broadcasting events - 
  a) Sometimes we need to broadcast the data to all sockets except the one generating the event. For this case, in the app.js

Replace “io.sockets.emit('myEvent', data);” By “socket.broadcast. emit('myEvent', data);”

2)  Difference between sending event on a socket and on sockets

You could emit event on a "socket" or on "sockets". When you do emit on a "socket" object the data would reflect on the current socket only after a refresh.

3) How to send event from server to a specific client socket.

This is not a ideal practice. However in certain cases there may be a need to communicate only to a specific socket.

In such cases you can create an array or store in mongo DB the socket ids:

io.sockets.socket[id]



For more exhaustive learning, please read the here.


4.2) Publish Subscribe model with Rabbit MQ, Socket IO and Node

Publish-Subscribe model is a messaging pattern. Publishers do not have knowledge of their consumers and consumers subscribe only to messages they wish to obtain.

Here is an overview what we are going to be doing:

1)      We will write a node application that will publish messages from the jade UI to the RabbitMQ via an exchange to a queue.

2)      We will see what is an exchange, queue, dead queue and binding queues.

3)      We will then write code to subscribe to an exchange and send this to the UI using Socket IO technology.

Before we get started we should understand some of the use cases where this can be employed:

a) We may need to perform heavy duty operations such as file imports or exports from a system. This requires perhaps just a trigger and user can get going with his operation and get back after his task is done to review. We could employ the RabbitMQ to queue this activity and upon completion send appropriate notifications to the user interface.

b)  Some of sites may like to attract users to hit the site more, by showing statistical data such as what is searched right now or what is being viewed now or what is being bought right now and so on.

1)  Getting Started with RabbitMQ

a) Download Erlang from the official site and intall it - http://www.erlang.org/download.html

b) Download the RabbitMQ and install it  - http://www.rabbitmq.com//download.html

c) You can open the services.msc and verify the RabbitMQ running as a service.

d)      Under the installation directory of the RabbitMQ, general under Program Files, traverse under the SBIN.

e)      Under the SBIN, install the RabbitMQ management plugin by entering

“rabbitmq-plugins enable rabbitmq_management”

f)       Restart the rabbit service.

g)      On your favourite browser, hit http://localhost:15672/

h)      You will see the console up and running. You could use the default login –guest/guest.

2) Getting Basic Express Node Project Up and Running

a) You can read my previous post to get your environment with the basic essential to create an express - jade - node project here.

b)      Traverse to your workspace, and hit “express node_rabbit_demo”

c)Traverse to the project just created : “cd node_rabbit_demo”

d) You can test if your project runs alright by hitting “node app.js”

e) Open in your favourite browser: http://localhost:3000

f) You should see the express text displayed gracefully

3) Configuring  publish to RabbitMQ entities in node

a) Add the “amqp” module in app.js

b)  Create connection to the RabbitMQ host, could be remote as well on a specific port

c)  On the ‘ready’ event of the connection created, set up the exchange.

d) On completion of the exchange creation, create the queue.

e) Bind the queue created to the exchange.

f) Events such as “queueBindOk” is available on successful binding.

Add the following into App.js

var amqp = require("amqp"); console.log("Starting ... AMQP"); //1 create connection with amqp var conn = amqp.createConnection({ host: 'localhost' }); conn.on('ready', setup); //2 define the exchange var exchange; function setup() {             exchange = conn.exchange('my_exchange1', {'type': 'fanout', durable: false}, exchangeSetup); } //3 define the queue var queue; var deadQueue; function exchangeSetup() {     queue = conn.queue('my_queue1', {durable: false, exclusive: true},queueSetup);     deadQueue = conn.queue('my_queue2', {durable: false, exclusive: true},queueSetup);             queue.on('queueBindOk', function() { onQueueReady(exchange); }); } //4 subscribe on queue and bind exchange and q function queueSetup() {   // TO DO } function onQueueReady(exchange){             console.log("queue binding done..........................."); }

4) Publishing to the RabbitMQ

    a) Let us now publish a message to the RabbitMQ. Add the following to index.jade

form(action='/test', method='post')             input(type='text', value='', name="myText")             input(type='submit', value='Push')

b) Correspondingly add the Handler in the app.js for this “/test” action added in index.jade

App.js code to add:

app.post('/test', function(req, res){   var myText = req.body.myText;   exchange.publish('my_queue1', {data:myText});   console.log("publish done on RabbitMQ........"+req.body.myText);   res.redirect('/'); });

c) The data from the UI is obtained within the post element – “req.body.myText”. This data is published on the queue in the exchange.

d) Let us see this in operation, hit “node app.js” in the cmd prompt

e) Open your favourite browser and hit : http://localhost:3000

f) On the UI available, enter some data in the text input and hit the submit button

g) Now open the rabbit MQ console by hitting http://localhost:15672

h) Observe the following

• You should see number of messages in the queue as 1
•  You would see the exchange we have created from the code in app.js
• You would see the queue we have created.
• Drilling down to the queue would show you the message on this particular queue.

  You have successfully published the message on the queue.

        5) Subscribing the message from the queue in Node.js
            
               a) Add the following code to subscribe from the queue in node.js
               
               b) In the app.js fill the contents into the queueSetup method as follows:

//4 subscribe on queue and bind exchange and q function queueSetup() {              queue.subscribe(function(msg) {                         console.log("msg from q is=="+msg.data);             //TODO                     });               queue.bind(exchange.name, 'my_queue1');                }

c) Run the application again, hit “node app.js” and open it in the browser http://localhost:3000

d) In the UI again enter some message in the input and hit the submit button.

e) Look at the comment prompt, you should see the following output deisplayed from the subscribe block we just added

“msg from q is ==<data you entered in the UI>”

Congratulations, you have your first application created with RabbitMQ and node. If you are looking to extend the feature using Socket.IO read along to the next section.


6) Emitting this data using Socket.IO to the UI

                    a) Modify the app.js to emit the socket.io event from node

//4 subscribe on queue and bind exchange and q function queueSetup() {              queue.subscribe(function(msg) {                         console.log("msg from q is=="+msg.data);                     //send socket io here....................                     emitEvent(msg.data);                 });               queue.bind(exchange.name, 'my_queue1');                deadQueue.bind(exchange.name, 'my_queue2');                } //6 emitEvent to client function emitEvent(data){             io.sockets.on('connection', function (socket) {                         console.log("emitting event now from server..........."+data);                         //socket.emit('event1', data);                         io.sockets.emit('event1', data);//instant update without page refresh             });                    }

b) The index.jade is revised as follows:

doctype 5 html head             link(rel='stylesheet', href='style.css')             script(type='text/javascript',src='/socket.io/socket.io.js')             script(type='text/javascript',src='https://ajax.googleapis.com/ajax/libs/jquery/1.6.4/jquery.min.js')             script(type='text/javascript')                         var socket=io.connect('http://localhost:3000');                         socket.on('event1', function(data){                                     $('p#realTimeSearch').text(data);                         });             body h1= title marquee(behavior="scroll",direction="right",width="200")             p#realTimeSearch form(action='/test', method='post')             input(type='text', value='', name="myText")             input(type='submit', value='Push')

c) Rerun the application, enter the data in the UI and you will see the text you have entered scrolling.

d) You can open another browser instance parallel and repeat the same activity.

e) You will see the data from the browser updated real time.

f)  This simulates a framework for what is being searched right now module.




Aliter:

1) You can extend the module by maintaining an array on server side to send back collective data of searches being done in the UI.

2) Hold the search texts into a DB from the RabbitMQ and emit as and when required to the UI.

3) A good start to understand the RabbitMQ message broker can be found on the official site with excellent examples.

         

•          Further reading can be done on how to create connection factory
•          Creating a connection
•                   Creating Channel
•          Declaring Queue explicitly and implicitly
•          Writing a send module to explicitly add message to a queue
•          Declaring a Queue Consumer, upon delivery of message, extract the message from the body using a Receive module explicitly.

4.3) Stand Alone RabbitMQ Publish-Subscribe Node applications


In this post let us look at how to create stand alone Publish and Subscribe Node.js applications. In most of our real world use cases we would host the publisher application on a different machine and our subscriber application/s via a load balancer on a different machine.

Even though this is a common use cases there are not many articles out there to help the community get this task done with ease.

Pre-Requisites:

Please go through my previous post here to come upto speed on the following pre-requisites to understand this article.

1) Understanding the basics of publish-subscribe model. You can read about this here
2) RabbitMQ message broker installation
3) Socket IO programming.

Getting Hands On:

1) Creating the Publisher Node.js Application

a) Let us create an express project called "Node_Publisher". Please read my previous post here to learn how to create a basic express project

b) Add the following code to App.js

/**  * Module dependencies.  */ //Add dependent modules var express = require('express')   , app = express()   , routes = require('./routes')   , user = require('./routes/user')   , http = require('http')   , path = require('path')   , amqp = require('amqp')   , server = require('http').createServer(app)   , io = require('socket.io').listen(server); //configure express app.configure(function(){   app.set('port', process.env.PORT || 3000);   app.set('views', __dirname + '/views');   app.set('view engine', 'jade');   app.use(express.favicon());   app.use(express.logger('dev'));   app.use(express.bodyParser());   app.use(express.methodOverride());   app.use(app.router);   app.use(express.static(path.join(__dirname, 'public'))); }); //Server initialise server.listen(app.get('port')); app.configure('development', function(){   app.use(express.errorHandler()); }); //Config Router for incoming request app.get('/', function(req, res){               res.render('index', {                   title: 'Publisher application powered by RabbitMQ, Node, Express, Jade'                 }) }); //1 create connection with amqp var conn = amqp.createConnection({ host: 'localhost' }); conn.on('ready', setup); //2 define the exchange var exchange; function setup() {             exchange = conn.exchange('my_exchange1',                               {'type': 'fanout', durable: false},                                exchangeSetup); } //3 define the queue var queue; var deadQueue; function exchangeSetup() {     queue = conn.queue('my_queue1', {durable: false, exclusive: false},queueSetup);     queue.on('queueBindOk', function() { onQueueReady(exchange); }); } //4 subscribe on queue and bind exchange and q function queueSetup() {              queue.bind(exchange.name, 'my_queue1'); } //5 queue ready event function onQueueReady(exchange){             console.log("queue binding done..........................."); } app.post('/test', function(req, res){

Summary of what has been done in above code:

1) Add dependent node modules
2) Configure Express Middleware
3) Initialise the HTTP server
4) Configure the router for incoming URL request
5) Create connection with the RabbitMQ using amqp
6) Define the exchange for communication
7) Define the Queue for publishing data
8) Bind the queue to the exchange
9) Verify if the queue binding is successful to the queue created.
10) Create the route handler for on form submit from UI

c) Add the following code snippet to the index.jade

doctype 5 html head             link(rel='stylesheet', href='style.css')              body        h1= title        form(action='/test', method='post')                  input(type='text', value='', name="myText")                  input(type='submit', value='Push')

d) Testing the Publisher Application:
     1) On the cmd prompt, traverse to the project created.
     2) Under the Node_Publisher, run the command "node app.js"
     3) On your favorite browser, hit http://localhost:3000/
     4) On the UI , enter any value in the text input and hit the push button.
     5) Open the RabbitMQ console using: http://localhost:15672/, login using guest/guest
     6) You will observe the creation of the exchange, queue and a message in the queue, showing the count as one.

2) Creating the Subscriber Application


a) Let us create an express project called "Node_Subscriber". Please read my previous post here to learn how to create a basic express project

b) Add the following code to App.js

/**  * Module dependencies.  */ var express = require('express')   , app = express()   , routes = require('./routes')   , user = require('./routes/user')   , http = require('http')   , path = require('path')   , amqp = require('amqp')   , server = require('http').createServer(app)   , io = require('socket.io').listen(server); server.listen(3001); app.configure(function(){   app.set('port', process.env.PORT || 3001);   app.set('views', __dirname + '/views');   app.set('view engine', 'jade');   app.use(express.favicon());   app.use(express.logger('dev'));   app.use(express.bodyParser());   app.use(express.methodOverride());   app.use(app.router);   app.use(express.static(path.join(__dirname, 'public')));   io.set('log level', 1); }); app.configure('development', function(){   app.use(express.errorHandler()); }); app.get('/', routes.index); app.get('/users', user.list);             var messages=["messages from queue: "]; io.sockets.on('connection', function (socket) {                         console.log("emitting event now from server..........."+messages.length);                         //socket.emit('event1', data);                         io.sockets.emit('event1', messages);//instant update without page refresh             }); //1 create connection with amqp var conn = amqp.createConnection({ host: 'localhost' }); conn.on('ready', setup); //2 define the exchange var exchange; function setup() {             exchange = conn.exchange('my_exchange1', {'type': 'fanout', durable: false}, exchangeSetup); } //3 define the queue var queue; var deadQueue; function exchangeSetup() {    queue = conn.queue('my_queue1');    queue.subscribe(function(msg) {             console.log("msg from q is=="+msg.data);             messages[messages.length]= msg.data;             io.sockets.emit('event1', messages);    }); } //4 subscribe on queue and bind exchange and q function queueSetup() {              console.log("q setup done"); } //5 queue ready event function onQueueReady(exchange){             console.log("queue binding done..........................."); }

Summary of what has been done in above code:

1) Add dependent node modules
2) Configure Express Middleware
3) Initialise the HTTP server
4) Configure the router for incoming URL request
5) Create connection with the RabbitMQ using amqp
6) Define the exchange for communication
7) Define the Queue for publishing data
8) Bind the queue to the exchange
9) Within the exchange set up, subscribe the queue. Create socket connection to send real time data to connected sockets
10) Verify if the queue binding is successful to the queue created.
11) Create the route handler for on form submit from UI

c) Add the following code snippet to index.js under route folder

/*  * GET home page.  */ exports.index = function(req, res){   var urlToListen = "http://"+req.headers.host+"/";   res.render('index', { title: 'Subscriber application powered by RabbitMQ, Node, Express, Jade' , urlToListen:urlToListen}); };

d) Add the following code snippet to the index.jade

doctype 5 html head             link(rel='stylesheet', href='style.css')             script(type='text/javascript',src='/socket.io/socket.io.js')             script(type='text/javascript',src='https://ajax.googleapis.com/ajax/libs/jquery/            1.6.4/jquery.min.js')             script(type='text/javascript')                         var appURL = !{JSON.stringify(urlToListen)};                         var socket=io.connect(appURL);                         socket.on('event1', function(messages){                                     $('textarea#myArea').val(messages);                         });                                     body             h1= title             textarea(id='myArea',width="400",height="200")

Summary of what has been done in above code:

1) We have added a textarea to display the data subscribed from the queue.
2) Socket IO events are defined which listens to the event from server side. ie "event1"
3) Also an interesting point to observe in this code is a dynamic way that I use to bind the Socket URL which is read as variable from the server side.
4) In the previous code snippet under (c), in the server side you can observe that we extract the URL to listen from the request.header.host.

e) Testing the Subscriber Application:
     1) On the cmd prompt, traverse to the project created.
     2) Under the Node_Subscriber, run the command "node app.js"
     3) On your favorite browser, hit http://localhost:3001/
     4) On the UI , you will see a default message which we have packed from the server side.


f) Putting it all together:

   1) Now start the Publisher Application and the Subscriber Application.
   2) It is important to node that you need to run these applications using Node app.js and not using nodemon app.js
   3) As of today with the version of 0.8.1 node version, we have a bug opened which does prevents the use of multiple nodemon instances on the same box. There are a few workarounds, however they are out of scope of this article.
  4) It is important to ensure your two applications if run on same box needs to run on different ports.
  5) On the Publisher Application , enter any message that needs to be pushed to server and click the push button.
  6) Observe on the Subscriber Application UI, the data is displayed from the RabbitMQ.
  7) Open the RabbitMQ console and you can verify the number of messages from the queue is 0 and is consumed by our Subscriber application.



 Please feel free to write to me and let me know what you think of this tutorial.