Showing posts with label javascript. Show all posts
Showing posts with label javascript. Show all posts

Building an HTTP sniffer with node.js

Wednesday, January 1, 2014

Whether to inspect a server response, optimize network usage or just to fiddle with new REST API it almost always make sense to use an application that can display http requests and responses for investigation. On Windows there is the great Fiddler. While it’s possible to use Fiddler on other platforms using virtualization software I think it’s an overkill. Fortunately there are alternatives. Charles looks like the most advanced one offering most if not all features available in Fiddler. There is also a little less popular HTTP Scoop. Both Charles and HTTP Scoop aren’t free but in my opinion they are worth the price especially if used often. Command line lovers might find mitmproxy suit their needs. If you only need basic features ngrok might serve you well. To dive a bit deeper and see http traffic on a tcp level WireShark is indispensable.

As you can see there are plenty of tools available to help understand what is happening on http level. As I was learning about http caching a question popped to my head. How hard would it be to actually build a simple http sniffer?

proxy-mirror – a simple http inspector

As it turned out it’s actually not that hard to built one thus proxy-mirror came to be. Of course this is a very simplistic http sniffer – nowhere near to tools I mentioned above both in terms of features and reliability – but it works (at least in most scenariosWinking smile). It’s open source and I learned couple of new things about HTTP while implementing it – more on that in future posts. Here are some screen shoots of the the tool: 

As you might have guessed from screenshots proxy-mirror is a web application. Right now it’s not very easy to try it out – the instructions are in the Readme – I’ll try to fix it soon.

How it works

I wanted to run proxy-mirror on many platforms and rely on a foundation that has both great support for http and building web applications. I picked node.js over java or ruby mainly because I wanted to sharpen my skills in it.

You can think of proxy-mirror as 2 logical components. The first is a regular http proxy that you can use by configuring your browser or system. I didn’t want to focus on building that first so I utilized a great node module call http-proxy. With it’s helped you can have a system wide proxy in couple of minutes. The http proxy component emits events whenever a request or response goes through it. Those events are consumed by the second logical component – a web application built with express.js. The information about http traffic received from events is then pushed to browser part through socket.io where a simple SPA built with my beloved AngularJS displays information about them.

Features

Right now proxy-mirror has only couple of features:

  • http and https support – although the latter one requires additional setup
  • simple session list – a grid with list of request/response pairs that you can inspect
  • a detailed view – both for request and response that can display headers, message body and a preview right in the browser

Although it still is a very simple (and buggy) application I actually found that it has most of features I frequently use – probably except for filtering. Maybe someday it will become a reasonable alternative for Charles - at least for me.

I think building an http sniffer is a great exercise in the process of learning how http(s) works. I guess the famous saying about journey being more more important than the destination fits nice here.

Backbone – when used properly…

Sunday, October 27, 2013

it keeps one’s head out of one’s butt.

This little gem was the first JavaScript MVC like tool that caught my eye. It has a very vibrant community and seems to have reached certain maturity level. More importantly it is so small library that everyone can grok its source code fully in couple of hours. And yet I haven’t actually used it in any project even thought I had couple of occasions.

I guess it was partially because I had read ThoughtWorks technology radar October 2012 edition where author’s claim that:

Backbone.js is a great example of an abstraction pushed too far (…) We find that it blurs the framework and model too much, forcing either bad architectural decisions or elaborate framework hackery in order to preserve sanity.

Back then I had to make a quick decision which framework/library to use, if any, on a project that I started working on and the article steered my away from using Backbone.js. Just for the record the project was developed with Angular.js.

Forming my own opinion

It really bugged me why the technology radar authors stated such a harsh opinion about a tool that has caught some much attention. It seemed even more interesting because I haven’t found a follow up or a more elaborate and concrete comment about the decision to put Backbone.js on hold. I did however find some time to do a simple demo for a presentation I gave on our company internal conference.

The demo I built has only 2 pages. First one shows how to make a page more interactive with a filtered table and couple of elements that trigger the same behaviour. Second view was created just to see how easy it is to make use of html push state. The source code is available on github.

Due to its minimalistic nature it’s really easy to start working with Backbone. The documentation is easy to follow and if you don’t find enough explanation you can always dive into reading the source. The library let’s you choose how you would like to render you views. Out of the box it provides support for client side templates by leveraging underscore.js.

Views are in fact the main force that drives code control. As there is no typical Controller I found it hard to decide where to put some of the initialization logic. For simple cases Router might be a good choice however as the application grows I pretty sure this approach would not be sufficient. It also means that you are now responsible for composition of smaller views into a complete page. I know this problem is solvable in many ways – Derick Bailey’s plugin Marionette is an excellent addition that fills the gaps not provided by Backbone.

What I find most disturbing is the way we attach events to DOM elements by providing selectors. I use this approach on daily basis in a project solely based on jQuery but there are several shortcomings that in my opinion make it hard to apply to large applications. First of all it means that you typically would add artificial CSS classes to elements. This isn’t bad at first as long as you use those classes only for attaching events and not for providing ways to style your application. An important implication follows –it’s nearly impossible to have a UI designer to work on html and CSS parts without breaking application behaviour. You can however mitigate this by establishing some rules and naming conventions. For instance that all artificial classes used to attach events are prefixed with ‘js-‘ and shall not be used inside CSS files.

Prefer composition over inheritance.

The most important part of the presentation logic are models, view models or presentation models (depending on a pattern flavour you’re using). I think it should be the main place where your application behaviour is defined. Whenever a tool that I’m using is trying to push some structure over me in that single place I immediately get suspicious. Unfortunately it’s not different in Backbone. The way you define and implement your models inside Backbone app is through deriving from base class provided by the library. By doing this you gain many of the commonly required features of model object like filtering, validation and even associations. It speeds up a development a lot especially when you are in need of this commonly requested functionality like form editing in CRUD applications.

However, from my experience, preferring inheritance over composition hurts the most when used to built model part of an client side MVC. I guess this is true for me mainly because I often change the structure and the way different parts of model talk to each other way more often than views and controllers interactions. Having a predefined structure, imposed by framework, limits ways you can define the model and this sometimes leads to pain points that are hard to overcome. In general inheritance is much stronger form of coupling than composition even in a language as malleable as JavaScript so I think that my observations might be true for others too. As all ways this is a question of getting the right balance between having a lot of features ready and waiting for you and keeping you model as flexible as possible to survive the harsh realities of requirement changes.

Don’t listen to me, try it yourself…

I’ve pointed some of, in my opinion, weak points of Backbone. I realize that I might be wrong in some parts and have missed out some important aspects. The fact that the community behind the tool is so active and from what I see always willing to help some newbies can probably out weight cons I’ve listed. I think Backbone is a very effective tool for many programmers out there – it just does fully adhere to my rules.

However when choosing a framework to use on a project one should not be focussed on finding good or bad opinions on the web. I think its much better to spent a week or couple of days with the team actually using the tool to build real features. This can give an immense feedback generated by the project team within actual context – one that you would never get reading reviews on the web.

AngularJs modules to the rescue

Monday, September 16, 2013

Building large scale JavaScript application is a tough problem. The language nature is malleable and it does not have proper modules mechanism built in. Some argue that you can’t really built and maintain big applications built with JavaScript.  Others say that the key in succeeding is to never built big applications but to approach things in a more clever way – by applying divide and conquer rule.

Dividing code base

Due to lack of modules mechanism in ECMAScript 5th edition, which is the version most commonly available in browsers, there were couple of implementations created. RequireJs and AMD are just 2 most popular ones. Angular team decided to provide it’s own implementation of modules and left the asynchronous loading of them to existing tools. I suspect that it was because they didn’t want to create a dependency on 3rd party library and because of their use of dependency injection. The framework relies on modules heavily making it possible to use just the parts that you actually need saving precious network and loading time. Here is how to define a module:
var myApp = angular.module('myApp', ['ngResource']);

myApp.factory('itemsStorage', function ($resource) { 
  var itemsStorage = ...;
  // items storage implementation
  return itemsStorage;
});
We’ve declared a new module called myApp and stated that it depends on another module called ngResource. In line 3 we are declaring a factory for itemsStorage that requires $resource service. Angular will automatically resolve $resource service and inject it as a value of parameter. As you may suspect under the hood the dependency injection mechanism inspects parameter names of our function and thus it is able to know how to satisfy them with proper values. Be careful when using JavaScript minimizing tools because most of them will by default shorten parameter names and that will break the DI. The only work around for it is to disable an option responsible for parameters in minimizer. Most of the times I find it useful to split one module definition across couple of files. It is a supported scenario but there are some quirks to it.
var myApp = angular.module('myApp');
 
myApp.directive('timeAgo', function(){
  return function($scope, $element, $attrs){
    // timeAgo directive link implmentation
  };
});
The first line says that we would like to retrieve myApp module so that we can extend or use it. This API may be confusing, it was for me, as it may look like we’re actually declaring a module that has no dependencies. The distinction between extending and declaring a module has an important ramification. You’re now responsible of loading file with module declaration before files that extend module functionality. In order to declare module with no dependencies you have to pass an empty array as a second argument:
var secondModule = angular.module('secondModule', []);

A praise of dependency injection

Dependency injection is a very useful concept especially in statically typed languages as C# or Java. It really helps you decouple your code as your classes no longer require to know about the implementation details of their dependencies. There are several ways to implement this pattern however must commonly used is a constructor injection. A lot of IoC containers exists both in .NET and Java world: Castle.Windsor, StructureMap, Guice or Ninject are just a few. However I have never seen a viable implementation of DI in JavaScript until AngularJs.

At first it seemed like a magic, so much that I actually had to dig into the framework code base to see how it’s done. As it turned out the idea behind the implementation is really simple. I did however encounter problems while trying to figure out why a particular service cannot be resolved or a directive isn’t working. Usually it was because of typos in parameter names or a missing dependency in module declaration. I know that the Angular team has put effort to make it easier to figure out those kind of problems in a recent release.

I’ve heard opinions that in a dynamic language as JavaScript the DI provides little to know gain instead adding an accidental complexity. I do agree that it is possible to live without however it requires much more discipline to keep a code clean. Whenever you create a service (a piece of code) that will be used in several places you have to make sure not to leak implementation details to clients (the ones that use the service). Without DI it means you have to make it easy to create a service by hand, typically by providing a factory method or a parameter less constructor. If you won’t take care of it upfront it will be harder to refactor the service implementation later on because of the number of clients that create it. When using DI at least this one task is forced upon you upfront.

Now what makes an actual implementation of DI practical is its ease of use. If you remember a time when most of C# and Java IoC containers required declaring actual object graph in an XML files you’ll understand what I mean. I think Angular team made using dependency injection feel easy and very natural and that’s what makes it so useful.

Dependency resolution and conflicts

There is an important limitation in Angular modules implementation around conflict resolution. Basically the framework uses last wins approach so you really have to be careful while naming things. It may seem not important at first but let’s consider following code sample.
var A = angular.module('A', []);

A.factory('serviceA', function(){
 return function(){
  console.log('using serviceA defined in module A');
 };
});

var B = angular.module('B', ['A']);

B.factory('serviceB', function(serviceA){
 return function(){
  console.group('using serviceB defined in module A');
  serviceA();
  console.groupEnd();
 };
});

var C = angular.module('C', ['A']);

C.factory('serviceA', function(){
 return function(){
  console.log('using serviceA defined in module C');
 };
});

C.factory('serviceC', function(serviceA){
 return function(){
  console.group('using serviceC defined in module C');
  serviceA();
  console.groupEnd();
 };
});

As you can see we have 3 modules here A, B and C. But in line 21 inside C module definition we’re declaring serviceA. Since module C depends on A that already defined serviceA we’re actually overriding the implementation. It’s quite useful to be able to override and stuff our custom implementation of particular service provided by other module. Except that we have no control over the scope of our modification. In the above example the serviceA defined in module C will be the one used everywhere at runtime. Even though module B knows nothing bout C. Calling declared services like this:

serviceA();
serviceB();
serviceC();
Will produce following output:
image
You can play around with this sample on jsfiddle.

What can we do about it? The only advice I’ve seen is to use prefix on service names. Now this is a working and simple approach however the code will not look great. I wonder how hard it would be to modify angular injector part to gain the ability to gain more control over overriding things…

AngularJs – my new superhero tool.

Sunday, September 15, 2013

As I mentioned in my previous post I’ll describe my thoughts and findings about 3 popular JavaScript libraries that help building large web application. I’ll start with AngularJs which is the first one I’ve used in a real product.

Model View Whatever

Angular team didn’t try hard to map their concepts to classical MVC pattern. You can see it even in the title of its home page. Instead they focused on making a development path of a large web application as smooth and productive, in a long term, as possible. This means its really important to return to documentation and developer guide if you feel you’re getting lost along the way. Learning Angular is a really interesting journey especially that many concepts that I previously had categorized as not for the web turned out to be totally useful.

Model and view model

This term has really a short definition according to angular way:

a model is any data that is reachable as a property of an angular Scope object
The nice thing about it is that it really is up to you how you choose to encapsulate the most valuable and complex logic inside your model or view model classes. No base “class” , no artificial properties and helpers is required. In fact the framework won’t force you to build a proper view model for data-binding. While this freedom gives you more flexibility it also means that you are responsible for keeping your Controllers clean and thin. I prefer to put logic that encapsulates data and provides global, shared behaviour into separate classes totally unaware of the UI. Depending on the use case I’ll then put an instance of this model into scope or  if the presentation logic is more complex create a view model that wraps some model objects. To make the model objects easily accessible and sharable between views angular provides services. For a very simple logic that augments the way a single piece of data is presented there are filters.

One important thing to understand though is that data-binding is only happening between view (and directives) and a scope object. If a model object provides events fired when its state changes they have to implemented in a traditional way. When I say traditional I mean that you typically will need to apply Observer pattern – or rather use one of implementations available. This is somewhat different from the dirty checking mechanism that sits behind all the angular data-binding magic. Fortunately the framework provides a way to connect a non angular event to the scope with $apply method. However be warned that you have to be really careful here to call it only outside of angular digest cycle.

Controller – keep it thin

A controller is mainly responsible for setting up the scope object and as such is created on a per view basis. It is the most common entry point for an application behaviour triggered by DOM events. While its often easy to put more and more business logic inside them you should really try hard not to. As a reward I often found myself being able to reuse more logic than I anticipated. Another rule of thumb is to never access DOM elements from controller. Angular has a special place for all DOM related activities called directives.

Directive – a way to reusable ui

This is the part of angular that causes lot of confusion when it comes to building custom directives. Especially that whenever you need to manipulate DOM in any way you should use them. To make things easier there is a lot of not so obvious vocabulary used like compile, interpolate, link and my favourite transclude. In my opinion directives make working with data-binding in angular so productive. They are kind of hard to understand deeply at first, but reading the guide couple of times helps a lot. To demonstrate its power I’ll just say that entire jQuery ui is easily accessible with just one directive. I guess I have to little experience to provide some tips about building directives - the most complex one I’ve built was paginate. Here is a HTML snippet of how to use it:
<div paginate="items">
    <ul>
        <li ng-repeat="item in pager.paged">
            {{item.index}} {{item.text}}
        </li>
    </ul>
    <button ng-click="pager.previous()">Previous</button>  
    <span ng-repeat="page in pager.pages" 
             ng-class="pager.classForPage(page)">
      <a ng-click="pager.goTo(page)">{{page}}</a>
    </span>
    <button ng-click="pager.next()">Next</button>
</div>

Directive is the hardest part to fully understand but if you do learn them and built them in a generic way you can accelerate development a lot.

Views

I’ve only touched couple of aspects of Angular - there is a lot more to it. In particular I haven’t commented on View part in MVC trio. Angular tends to be criticised for the way it handles views. Mainly because you may get to a stage where it is hard to see actual html inside tons of directives attached to DOM elements. An alternative approach would be to use existing template engines like Handlebars – that’s the way it’s done in Ember.js – but it’s not really practical in Angular.

However I really like the idea that behaviour of html elements is immediately visible when you scan views mainly because declarative nature of directives. It makes it possible for an CSS or UI design expert, not familiar with Angular nor JavaScript, to make safe changes to views. In addition you no longer need to repeat selectors to access DOM and attach behaviour to it. This eliminates some of the maintenance burden. Like with any tool you have to be careful though not to overload html with directives and never put logic inside your views even if its only presentation related. If you find yourself repeating the same set of directives in many places of your codebase it may be good to step back a little and think of a way to encapsulate a common behaviour – probably with a more specific directives.

I’ll soon try to comment on features that Angular provides that make it suitable for large scale application development.

Javascript MV* frameworks

Friday, August 16, 2013

Building a web application today isn’t nearly as hard as it was couple of years ago. Back then creating a rich client application using HTML and JS was possible but the number of options you had is a fraction of what is available today. The first SPA I helped building was relatively simple project –built with ExtJs library when it was still in version 2. There were of course other options like dojo but with our team’s little experience in web technology ExtJs seemed like a better choice solely because of its superior support. Shortly after I worked on another web application built with Web Forms and jQuery.  We finished both projects, they worked on most browsers but we didn’t avoid creating a mess in JavaScript. Presentation logic was mixed with business rules scattered through event handlers inside unstructured components source code. Just to be fair it definitely was not caused by the frameworks we used but because of our lack of knowledge and experience. 
I guess that happened to lots of developers and wise people soon enough realized that there has to be a better way of doing things. I think that this was the main reason of the explosion of JS libraries/frameworks options that aim to help structuring client side code.

MVC, MVP, MVVM or …

The number of libraries and frameworks available is overwhelming. AngularJS, Backbone.js, Ember, JavaScriptMVC, Knockout or Sammy.js are just a few I had a chance to look at from the outstanding list. You can even compare how to built simple TODO application with them. Addy Osmani wrote an excellent article that really helps making your choice sane. Most if not all of those frameworks utilize one or more of MV* patterns. An important thing to remember though is that Model View Controller is more an architectural pattern for presentation than an code design pattern like Builder. It means it does not provide very concrete implementation template but rather a guidance for objects/modules/roles relationships.
Similar observations apply to MVC cousins namely Model View ViewModel or Presentation Model and Model View Presenter. This can cause some confusion for someone already familiar with MVC pattern in Asp.net mvc, Fubu or Rails. I always try to find a framework/library author’s explanation of how they applied above patterns – it really helps understanding core concepts of their work.

Library or Framework

Some of the tools are called frameworks while other libraries. But what’s the difference anyway? A very common and brief explanation is the one provided by Martin Fowler inside Inversion of Control article:
Inversion of Control is a key part of what makes a framework different to a library. A library is essentially a set of functions that you can call, these days usually organized into classes. Each call does some work and returns control to the client.
A framework embodies some abstract design, with more behavior built in. In order to use it you need to insert your behavior into various places in the framework either by subclassing or by plugging in your own classes. The framework's code then calls your code at these points.There are various ways you can plug your code in to be called.
Of course the distinction is not always firm and clear. Still, it is important to keep in mind while choosing the right tool for a job. A library will usually give you more control inside infrastructure code but it often requires more work to unify boilerplate gluing code and requires some degree of experience. A framework on the other hand will typically provide you with a setup where you can hook your code hopefully focused mostly on the valuable functional stuff. A framework will limit your freedom to some extent in turn it will eliminate some part of work that you presumably would have to do in order to keep your design clean.

Productivity

For me the biggest advantage of using a modern MV* JavaScript framework is productivity. After some time I realized that the amount of boring, repetitive code focused only on filling DOM with data and giving it behaviour is substantial. Sure I am able to separate presentation concerns rather cleanly without aid from any tool. However what’s the point if so many great libraries and frameworks, created by way more experienced people, is out there. It gets even more valid point when working on a big project. Even if all the team members are experienced and closely collaborating it will be hard to have and maintain a common, well understood structure among different parts of JavaScript code base just because understanding and preferences about MVC and related patterns will vary.
In my next blog posts I’ll try to review 3 tools that caught my eye. Namely AngularJS, Backbone.js and Knockout that seem to be most popular these days. They all show some signs of maturity and have vibrant community around them. It will by no means be a comparison as they really are different beasts built with different goals in mind. The only comparison I may be tempted to do is about productivity. Still I already know it will not be comprehensive as there will be only one person working on the codebase and the micro product will be focused mostly on a rich data visualization – which is what I do mostly now.
Maybe it will help to the framework/library that we’ll use in our current project.