Combine Express.js and ASP.NET Web API

I’m really excited about the OWIN initiative. This “standard” enables countless new ways of combining existing http web stacks with .net – a thing that up until recently was either impossible or not feasible. It gets even more positive knowing that Microsoft supports the case actively.

OWIN - why you should care?

Open Web Interface for .NET aims to define a standard interface between web servers and web applications. Adhering to this interface will allow decoupling of server implementations from web frameworks. The idea isn’t something novel and other platforms benefited from it immensely. In Ruby we have pretty mature Rack which allows hosting Rails, Sinatra or any other rack compatible application on a web server of your choice. In Node.js world there is Connect middleware with frameworks like Express and Zappa built on top of it. To be fair Connect is not exactly the same thing as Rack but the analogy serves well to understand underlying ideas.

Having a common middleware and frameworks built on top of it not only allows you to swap your web server at whim. It also means that you’re now able to combine many application into one easily even if they are implemented in different frameworks. It took me some time to realize how powerful concept it is until I built a demo for a conference presentation. I wanted to have single web server running in background so that I can easily switch between pages built with Angular, Backbone and Knockout. I also didn’t want to throw out separate templates scaffolded with Yeoman and have a shared REST-like api built with Express. How surprised I was when the only thing I needed to do was to call equivalent of:

connect.use(angularApp);
connect.use(knockoutApp);
connect.use(backboneApp);

OWIN opens a new world for .NET web stacks

This all may sound blurry for a .NET newbie or maybe even to experienced developer so I’ll give some examples of what is or will be possible to do in a near future.

• IIS as option not requirement – having a way to add a full fledged web interface for a windows service was not an easy thing to do. If only you could you use Asp.Net MVC things would be so much easier. Guess what, when Asp.Net MVC will support OWIN and I’m positive it someday will, this will be a no brainer.
• Combine Nancy, FubuMVC, pick a name – maybe you would like to use some existing component and not have to rewrite it to match you stack. Or maybe you work on a really big project and some teams prefer one framework over the other. You are right, this already is possible.
• Integration tests – say you would like to execute end to end tests over your web application. Right know you’ll either have to deploy your asp.net mvc app to IIS or IIS Express. This means spinning up separate processes even if you would like to run a single test on you development machine. You may be lucky and be using asp.net web api and be able to use HttpSelfHostServer but then again you’re limited only to this stack. With OWIN it already is possible to use a lightweight Nowin or Katana to run any compatible application inside your tests.
• Faster Tests – is hosting a http server inside you tests too expensive? With OWIN you might actually be able to run tests entirely in memory with no network involved whatsoever.
• Cross cutting concerns – maybe you need to apply the same authentication mechanism to couple of separate applications. Yes you’re right implementing it as HttpModule might work, but what about this one that you have to support which is based on Nancy. Nancy already supports OWIN so you’re good to go.
• Combining Express.js and Asp.Net Web Api – so you’re building a web app using Express.js but you’ve found that it would be nice to reuse some part of the existing code that is built in .NET. With the excellent edge.js and OWIN enabled middleware built on top of it called connect-owin you’re now allowed to this.

One app using both Express.js and ASP.NET Web Api

The last example of OWIN usage seemed like the coolest to me so I wanted to try if it’s really doable at the moment. This wasn’t as easy as one could wish but bare in mind that the OWIN initiative is just starting getting momentum and we’re really on a bleeding edge. The connect-webapi-samples repo contains a sample of single web app where some requests are served by express.js while others using ASP.NET Web Api. You can start the server with:

msbuild .\Connect.WebApi.sln
node .\index.js

Navigating to http://localhost:3000/hello will give you response from Express:



When you go to http://localhost:3000/webapi/values you’ll get a standard ValuesController response:



Note that both requests are logged in console using connect.logger middleware:

I was thinking about a case where this combination could become handy. In a project I’m currently involved in we’re using Highcharts to present beautiful charts to clients. Unfortunately some of them still use ancient browsers like IE7 which does not handle many Highcharts displayed on one page – or rather dies trying to render them. For those browsers we actually produce similar chart images on server. This however requires a lot of boring code to convert from Higchart chart configuration to the 3rd party library format we use for rasterizing.

What we could do very easily with OWIN is to keep a .NET component that generates a complete Highchart configuration and add a very thin “middleware” layer that would take a real configuration pass it into Phantomjs and get back an image that would look exactly the same as real chart. I’m pretty sure it would be much more robust solution than our current approach.

Backbone – when used properly…

…it keeps one’s head out of one’s butt.

This little gem was the first JavaScript MVC like tool that caught my eye. It has a very vibrant community and seems to have reached certain maturity level. More importantly it is so small library that everyone can grok its source code fully in couple of hours. And yet I haven’t actually used it in any project even thought I had couple of occasions.

I guess it was partially because I had read ThoughtWorks technology radar October 2012 edition where author’s claim that:

Backbone.js is a great example of an abstraction pushed too far (…) We find that it blurs the framework and model too much, forcing either bad architectural decisions or elaborate framework hackery in order to preserve sanity.

Back then I had to make a quick decision which framework/library to use, if any, on a project that I started working on and the article steered my away from using Backbone.js. Just for the record the project was developed with Angular.js.

Forming my own opinion

It really bugged me why the technology radar authors stated such a harsh opinion about a tool that has caught some much attention. It seemed even more interesting because I haven’t found a follow up or a more elaborate and concrete comment about the decision to put Backbone.js on hold. I did however find some time to do a simple demo for a presentation I gave on our company internal conference.

The demo I built has only 2 pages. First one shows how to make a page more interactive with a filtered table and couple of elements that trigger the same behaviour. Second view was created just to see how easy it is to make use of html push state. The source code is available on github.

Due to its minimalistic nature it’s really easy to start working with Backbone. The documentation is easy to follow and if you don’t find enough explanation you can always dive into reading the source. The library let’s you choose how you would like to render you views. Out of the box it provides support for client side templates by leveraging underscore.js.

Views are in fact the main force that drives code control. As there is no typical Controller I found it hard to decide where to put some of the initialization logic. For simple cases Router might be a good choice however as the application grows I pretty sure this approach would not be sufficient. It also means that you are now responsible for composition of smaller views into a complete page. I know this problem is solvable in many ways – Derick Bailey’s plugin Marionette is an excellent addition that fills the gaps not provided by Backbone.

What I find most disturbing is the way we attach events to DOM elements by providing selectors. I use this approach on daily basis in a project solely based on jQuery but there are several shortcomings that in my opinion make it hard to apply to large applications. First of all it means that you typically would add artificial CSS classes to elements. This isn’t bad at first as long as you use those classes only for attaching events and not for providing ways to style your application. An important implication follows –it’s nearly impossible to have a UI designer to work on html and CSS parts without breaking application behaviour. You can however mitigate this by establishing some rules and naming conventions. For instance that all artificial classes used to attach events are prefixed with ‘js-‘ and shall not be used inside CSS files.

Prefer composition over inheritance.

The most important part of the presentation logic are models, view models or presentation models (depending on a pattern flavour you’re using). I think it should be the main place where your application behaviour is defined. Whenever a tool that I’m using is trying to push some structure over me in that single place I immediately get suspicious. Unfortunately it’s not different in Backbone. The way you define and implement your models inside Backbone app is through deriving from base class provided by the library. By doing this you gain many of the commonly required features of model object like filtering, validation and even associations. It speeds up a development a lot especially when you are in need of this commonly requested functionality like form editing in CRUD applications.

However, from my experience, preferring inheritance over composition hurts the most when used to built model part of an client side MVC. I guess this is true for me mainly because I often change the structure and the way different parts of model talk to each other way more often than views and controllers interactions. Having a predefined structure, imposed by framework, limits ways you can define the model and this sometimes leads to pain points that are hard to overcome. In general inheritance is much stronger form of coupling than composition even in a language as malleable as JavaScript so I think that my observations might be true for others too. As all ways this is a question of getting the right balance between having a lot of features ready and waiting for you and keeping you model as flexible as possible to survive the harsh realities of requirement changes.

Don’t listen to me, try it yourself…

I’ve pointed some of, in my opinion, weak points of Backbone. I realize that I might be wrong in some parts and have missed out some important aspects. The fact that the community behind the tool is so active and from what I see always willing to help some newbies can probably out weight cons I’ve listed. I think Backbone is a very effective tool for many programmers out there – it just does fully adhere to my rules.

However when choosing a framework to use on a project one should not be focussed on finding good or bad opinions on the web. I think its much better to spent a week or couple of days with the team actually using the tool to build real features. This can give an immense feedback generated by the project team within actual context – one that you would never get reading reviews on the web.

AngularJs modules to the rescue

Building large scale JavaScript application is a tough problem. The language nature is malleable and it does not have proper modules mechanism built in. Some argue that you can’t really built and maintain big applications built with JavaScript.  Others say that the key in succeeding is to never built big applications but to approach things in a more clever way – by applying divide and conquer rule.

Dividing code base

Due to lack of modules mechanism in ECMAScript 5th edition, which is the version most commonly available in browsers, there were couple of implementations created. RequireJs and AMD are just 2 most popular ones. Angular team decided to provide it’s own implementation of modules and left the asynchronous loading of them to existing tools. I suspect that it was because they didn’t want to create a dependency on 3rd party library and because of their use of dependency injection. The framework relies on modules heavily making it possible to use just the parts that you actually need saving precious network and loading time. Here is how to define a module:

var myApp = angular.module('myApp', ['ngResource']);

myApp.factory('itemsStorage', function ($resource) { 
  var itemsStorage = ...;
  // items storage implementation
  return itemsStorage;
});

We’ve declared a new module called myApp and stated that it depends on another module called ngResource. In line 3 we are declaring a factory for itemsStorage that requires $resource service. Angular will automatically resolve $resource service and inject it as a value of parameter. As you may suspect under the hood the dependency injection mechanism inspects parameter names of our function and thus it is able to know how to satisfy them with proper values. Be careful when using JavaScript minimizing tools because most of them will by default shorten parameter names and that will break the DI. The only work around for it is to disable an option responsible for parameters in minimizer. Most of the times I find it useful to split one module definition across couple of files. It is a supported scenario but there are some quirks to it.

var myApp = angular.module('myApp');
 
myApp.directive('timeAgo', function(){
  return function($scope, $element, $attrs){
    // timeAgo directive link implmentation
  };
});

The first line says that we would like to retrieve myApp module so that we can extend or use it. This API may be confusing, it was for me, as it may look like we’re actually declaring a module that has no dependencies. The distinction between extending and declaring a module has an important ramification. You’re now responsible of loading file with module declaration before files that extend module functionality. In order to declare module with no dependencies you have to pass an empty array as a second argument:

var secondModule = angular.module('secondModule', []);

A praise of dependency injection

Dependency injection is a very useful concept especially in statically typed languages as C# or Java. It really helps you decouple your code as your classes no longer require to know about the implementation details of their dependencies. There are several ways to implement this pattern however must commonly used is a constructor injection. A lot of IoC containers exists both in .NET and Java world: Castle.Windsor, StructureMap, Guice or Ninject are just a few. However I have never seen a viable implementation of DI in JavaScript until AngularJs.

At first it seemed like a magic, so much that I actually had to dig into the framework code base to see how it’s done. As it turned out the idea behind the implementation is really simple. I did however encounter problems while trying to figure out why a particular service cannot be resolved or a directive isn’t working. Usually it was because of typos in parameter names or a missing dependency in module declaration. I know that the Angular team has put effort to make it easier to figure out those kind of problems in a recent release.

I’ve heard opinions that in a dynamic language as JavaScript the DI provides little to know gain instead adding an accidental complexity. I do agree that it is possible to live without however it requires much more discipline to keep a code clean. Whenever you create a service (a piece of code) that will be used in several places you have to make sure not to leak implementation details to clients (the ones that use the service). Without DI it means you have to make it easy to create a service by hand, typically by providing a factory method or a parameter less constructor. If you won’t take care of it upfront it will be harder to refactor the service implementation later on because of the number of clients that create it. When using DI at least this one task is forced upon you upfront.

Now what makes an actual implementation of DI practical is its ease of use. If you remember a time when most of C# and Java IoC containers required declaring actual object graph in an XML files you’ll understand what I mean. I think Angular team made using dependency injection feel easy and very natural and that’s what makes it so useful.

Dependency resolution and conflicts

There is an important limitation in Angular modules implementation around conflict resolution. Basically the framework uses last wins approach so you really have to be careful while naming things. It may seem not important at first but let’s consider following code sample.

var A = angular.module('A', []);

A.factory('serviceA', function(){
 return function(){
  console.log('using serviceA defined in module A');
 };
});

var B = angular.module('B', ['A']);

B.factory('serviceB', function(serviceA){
 return function(){
  console.group('using serviceB defined in module A');
  serviceA();
  console.groupEnd();
 };
});

var C = angular.module('C', ['A']);

C.factory('serviceA', function(){
 return function(){
  console.log('using serviceA defined in module C');
 };
});

C.factory('serviceC', function(serviceA){
 return function(){
  console.group('using serviceC defined in module C');
  serviceA();
  console.groupEnd();
 };
});

As you can see we have 3 modules here A, B and C. But in line 21 inside C module definition we’re declaring serviceA. Since module C depends on A that already defined serviceA we’re actually overriding the implementation. It’s quite useful to be able to override and stuff our custom implementation of particular service provided by other module. Except that we have no control over the scope of our modification. In the above example the serviceA defined in module C will be the one used everywhere at runtime. Even though module B knows nothing bout C. Calling declared services like this:

serviceA();
serviceB();
serviceC();

Will produce following output:

You can play around with this sample on jsfiddle.

What can we do about it? The only advice I’ve seen is to use prefix on service names. Now this is a working and simple approach however the code will not look great. I wonder how hard it would be to modify angular injector part to gain the ability to gain more control over overriding things…

AngularJs – my new superhero tool.

As I mentioned in my previous post I’ll describe my thoughts and findings about 3 popular JavaScript libraries that help building large web application. I’ll start with AngularJs which is the first one I’ve used in a real product.

Model View Whatever

Angular team didn’t try hard to map their concepts to classical MVC pattern. You can see it even in the title of its home page. Instead they focused on making a development path of a large web application as smooth and productive, in a long term, as possible. This means its really important to return to documentation and developer guide if you feel you’re getting lost along the way. Learning Angular is a really interesting journey especially that many concepts that I previously had categorized as not for the web turned out to be totally useful.

Model and view model

This term has really a short definition according to angular way:

a model is any data that is reachable as a property of an angular Scope object

The nice thing about it is that it really is up to you how you choose to encapsulate the most valuable and complex logic inside your model or view model classes. No base “class” , no artificial properties and helpers is required. In fact the framework won’t force you to build a proper view model for data-binding. While this freedom gives you more flexibility it also means that you are responsible for keeping your Controllers clean and thin. I prefer to put logic that encapsulates data and provides global, shared behaviour into separate classes totally unaware of the UI. Depending on the use case I’ll then put an instance of this model into scope or  if the presentation logic is more complex create a view model that wraps some model objects. To make the model objects easily accessible and sharable between views angular provides services. For a very simple logic that augments the way a single piece of data is presented there are filters.

One important thing to understand though is that data-binding is only happening between view (and directives) and a scope object. If a model object provides events fired when its state changes they have to implemented in a traditional way. When I say traditional I mean that you typically will need to apply Observer pattern – or rather use one of implementations available. This is somewhat different from the dirty checking mechanism that sits behind all the angular data-binding magic. Fortunately the framework provides a way to connect a non angular event to the scope with $apply method. However be warned that you have to be really careful here to call it only outside of angular digest cycle.

Controller – keep it thin

A controller is mainly responsible for setting up the scope object and as such is created on a per view basis. It is the most common entry point for an application behaviour triggered by DOM events. While its often easy to put more and more business logic inside them you should really try hard not to. As a reward I often found myself being able to reuse more logic than I anticipated. Another rule of thumb is to never access DOM elements from controller. Angular has a special place for all DOM related activities called directives.

Directive – a way to reusable ui

This is the part of angular that causes lot of confusion when it comes to building custom directives. Especially that whenever you need to manipulate DOM in any way you should use them. To make things easier there is a lot of not so obvious vocabulary used like compile, interpolate, link and my favourite transclude. In my opinion directives make working with data-binding in angular so productive. They are kind of hard to understand deeply at first, but reading the guide couple of times helps a lot. To demonstrate its power I’ll just say that entire jQuery ui is easily accessible with just one directive. I guess I have to little experience to provide some tips about building directives - the most complex one I’ve built was paginate. Here is a HTML snippet of how to use it:

<div paginate="items">
    <ul>
        <li ng-repeat="item in pager.paged">
            {{item.index}} {{item.text}}
        </li>
    </ul>
    <button ng-click="pager.previous()">Previous</button>  
    <span ng-repeat="page in pager.pages" 
             ng-class="pager.classForPage(page)">
      <a ng-click="pager.goTo(page)">{{page}}</a>
    </span>
    <button ng-click="pager.next()">Next</button>
</div>

Directive is the hardest part to fully understand but if you do learn them and built them in a generic way you can accelerate development a lot.

Views

I’ve only touched couple of aspects of Angular - there is a lot more to it. In particular I haven’t commented on View part in MVC trio. Angular tends to be criticised for the way it handles views. Mainly because you may get to a stage where it is hard to see actual html inside tons of directives attached to DOM elements. An alternative approach would be to use existing template engines like Handlebars – that’s the way it’s done in Ember.js – but it’s not really practical in Angular.

However I really like the idea that behaviour of html elements is immediately visible when you scan views mainly because declarative nature of directives. It makes it possible for an CSS or UI design expert, not familiar with Angular nor JavaScript, to make safe changes to views. In addition you no longer need to repeat selectors to access DOM and attach behaviour to it. This eliminates some of the maintenance burden. Like with any tool you have to be careful though not to overload html with directives and never put logic inside your views even if its only presentation related. If you find yourself repeating the same set of directives in many places of your codebase it may be good to step back a little and think of a way to encapsulate a common behaviour – probably with a more specific directives.

I’ll soon try to comment on features that Angular provides that make it suitable for large scale application development.

Javascript MV* frameworks

Building a web application today isn’t nearly as hard as it was couple of years ago. Back then creating a rich client application using HTML and JS was possible but the number of options you had is a fraction of what is available today. The first SPA I helped building was relatively simple project –built with ExtJs library when it was still in version 2. There were of course other options like dojo but with our team’s little experience in web technology ExtJs seemed like a better choice solely because of its superior support. Shortly after I worked on another web application built with Web Forms and jQuery.  We finished both projects, they worked on most browsers but we didn’t avoid creating a mess in JavaScript. Presentation logic was mixed with business rules scattered through event handlers inside unstructured components source code. Just to be fair it definitely was not caused by the frameworks we used but because of our lack of knowledge and experience. 
I guess that happened to lots of developers and wise people soon enough realized that there has to be a better way of doing things. I think that this was the main reason of the explosion of JS libraries/frameworks options that aim to help structuring client side code.

MVC, MVP, MVVM or …

The number of libraries and frameworks available is overwhelming. AngularJS, Backbone.js, Ember, JavaScriptMVC, Knockout or Sammy.js are just a few I had a chance to look at from the outstanding list. You can even compare how to built simple TODO application with them. Addy Osmani wrote an excellent article that really helps making your choice sane. Most if not all of those frameworks utilize one or more of MV* patterns. An important thing to remember though is that Model View Controller is more an architectural pattern for presentation than an code design pattern like Builder. It means it does not provide very concrete implementation template but rather a guidance for objects/modules/roles relationships.
Similar observations apply to MVC cousins namely Model View ViewModel or Presentation Model and Model View Presenter. This can cause some confusion for someone already familiar with MVC pattern in Asp.net mvc, Fubu or Rails. I always try to find a framework/library author’s explanation of how they applied above patterns – it really helps understanding core concepts of their work.

Library or Framework

Some of the tools are called frameworks while other libraries. But what’s the difference anyway? A very common and brief explanation is the one provided by Martin Fowler inside Inversion of Control article:

Inversion of Control is a key part of what makes a framework different to a library. A library is essentially a set of functions that you can call, these days usually organized into classes. Each call does some work and returns control to the client.
A framework embodies some abstract design, with more behavior built in. In order to use it you need to insert your behavior into various places in the framework either by subclassing or by plugging in your own classes. The framework's code then calls your code at these points.There are various ways you can plug your code in to be called.

Of course the distinction is not always firm and clear. Still, it is important to keep in mind while choosing the right tool for a job. A library will usually give you more control inside infrastructure code but it often requires more work to unify boilerplate gluing code and requires some degree of experience. A framework on the other hand will typically provide you with a setup where you can hook your code hopefully focused mostly on the valuable functional stuff. A framework will limit your freedom to some extent in turn it will eliminate some part of work that you presumably would have to do in order to keep your design clean.

Productivity

For me the biggest advantage of using a modern MV* JavaScript framework is productivity. After some time I realized that the amount of boring, repetitive code focused only on filling DOM with data and giving it behaviour is substantial. Sure I am able to separate presentation concerns rather cleanly without aid from any tool. However what’s the point if so many great libraries and frameworks, created by way more experienced people, is out there. It gets even more valid point when working on a big project. Even if all the team members are experienced and closely collaborating it will be hard to have and maintain a common, well understood structure among different parts of JavaScript code base just because understanding and preferences about MVC and related patterns will vary.
In my next blog posts I’ll try to review 3 tools that caught my eye. Namely AngularJS, Backbone.js and Knockout that seem to be most popular these days. They all show some signs of maturity and have vibrant community around them. It will by no means be a comparison as they really are different beasts built with different goals in mind. The only comparison I may be tempted to do is about productivity. Still I already know it will not be comprehensive as there will be only one person working on the codebase and the micro product will be focused mostly on a rich data visualization – which is what I do mostly now.
Maybe it will help to the framework/library that we’ll use in our current project.

URL Rewrite problem with POST request

Greetings

Hello Everyone! In this blog I would like to share my thoughts and findings on programming, software and a like. I will post here so that I can improve my writing skills and find stuff I was working on previously easier.

URL Rewrite

IIS users had waited a bit for Apache equivalent of mod_rewrite. Finally we've got a decent module from Microsoft - Url Rewrite. Among basic features as rewriting all URLs to extension-less equivalents, blocking unwanted URLs or even more common providing friendly URLs the module provides a nice GUI inside IIS Manager so you can quickly test you ideas without referring to documentation. More importantly rewrite rules of your web site can be configured through Web.config configuration section.

The feature I like the most is the ability to create a reversed proxy inside an IIS web site with a little help of another IIS module - Application Request Routing. I have used it couple of times, mostly to be able to make "cross domain" POSTs. One important thing is to remember to enable proxy in ARR settings inside IIS Manager. Following command line will do it for you:

%windir%\system32\inetsrv\appcmd.exe set config -section:system.webServer/proxy /enabled:"True" /commit:apphost

Interfering with ASP.NET modules

URL Rewrite is a native IIS module and from my understanding it deals with HTTP on a pretty low level. Most of custom IIS modules that I've used or built were written in managed code. This means that they use ASP.NET stack that provides some higher level API to deal with requests and responses. However some of those APIs have important side effects one can easily overlook. Why is that important? Because combining several modules into one request/response pipeline can cause hard to debug bugs.
I've came a cross the same problem twice - as it turned out - it was caused exactly by this interference. Specifically after some happy months of rewriting http POSTs using URL Rewrite suddenly features using it stopped working. In both cases the problem was caused by seemingly not intrusive call:

var request = HttpContext.Current.Request;
var parameter = request["formOrQueryOrServerVariableKey"];

As you may already know this code looks for a value inside QueryString, Form, Cookies, or ServerVariables collection member specified in the key parameter. The important part is a Form which is an abstraction over html form that typically translates to http POST body. ASP.NET reads the body of a http request and decodes form url encoded values into NameValueCollection. This is done by reading input stream of the request which internally is buffered on disk. However since this actually is a network stream it can only be read once. This of course means that depending on the order of modules and at which stage of request pipeline one module decides to read it the other ones may not be able to access it. Exactly this scenario caused URL Rewrited POST request to hang until timeout.

Glimpse

One of the modules affected by this problem is part of a great tool called Glimpse. After an hour of looking through code base of this diagnostic tool I've found what I was looking for inside RequestMetadata class. Specifically RequestIsAjax method that in turn is called by AjaxPolicy during BeginRequest event. I've created a pull request that hopefully will fix the issue.

One thing that I definitely will remember from this investigation is to always be very careful while writing ASP.NET http module.