November 01, 2013

Brain Drain Tech

In order for upcoming posts to make more sense I'm going to have to give some background on the technology I'm using for the Brain Drain app. Let's jump straight into it:

I'm using Phonegap for the phone app, Node.js (with Restify) for the server, MongoDB as database, and D3.js for the webapp.

Let's get a bit into the reasoning. First the phone app itself. My first idea was to make use of Appcelerator, which lets you build phone apps in a platform independent way by means of Javascript. Unfortunately Appcelerator's licensing is a bit unclear, and I have seen some reports on aggresive fee extractions. Luckily there is the alternative of Phonegap, which works in a similar way and is licensed quite clearly as ApacheV2.

On to the server. Here I wanted to define a REST API, and so I was looking for something which lets you implement this in as easy a way as possible. This led me to Restify which, with the help of Node.js, let's you write such a service with just a few lines of Javascript. Very little overhead, cross-platform, and easy to set up.

The database is MongoDB, and the main reason for that is that I'm trying to build some more proficiency with it for professional reasons. In addition there exists a Javascript driver which integrates into Node.js and which lets you interact with the database in a very straightforward way (the Mongo shell is also Javascript, and the driver is pretty much a one-to-one mapping of everything you can do in the shell).

Then the webapp for visualizing the data. Here I'm using D3.js. D3 takes some getting used to, but once you get into the flow it can actually do pretty much anything you want. It has become my first choice for doing graphs in web applications. Of course I'm also using jQuery for setting up the main interactions, both with the user and with the server.

If you've been paying attention you probably noticed that the whole technology stack consists of Javascript. I actually think this is a great selling point for this combination. Whichever part you're working on there is no switching of languages required. Any proficiency you get while working on one part can be transferred to all other parts. This is even true of the phone app and the web app: both are basically nothing more than websites; only the packaging is different.

So that's about it. More details in future posts when I will try to tackle some of the internals.


Randy Voet said...

I feel like you must work at dZine :-)
Seriously. We started work on a web based application recently and the technology stack is suspiciously similar: Node.js with a MongoDB database as the back-end. SPA (Single Page Application) using a bunch of JavaScript libraries as the front-end.

KrisDS said...

Great minds think alike. ;-)

Seriously though, it seems like a simple yet powerful stack. If you can, please let me know if it stays manageable for something more than this quite basic thing I'm working on.

Randy Voet said...

Although I'm not (yet) actively working on the new product, I know that the first library you should add is RequireJS, which allows you to split your front end JavaScript in logical files (Node.js does this natively in a similar fashion). Each JavaScript file then simply states it's dependencies. A bit like an import/include statement in other languages.
The second is using MVC/MVVM with a library like Backbone.js or Knockout.
You should also use a JavaScript front end package manager like Bower or a higher level one like Jam, Volo or Ender. All of these are installed through npm (the Node.js package manager) and allow you to simply add, remove and update libraries without having to manually download files.
JSHint or JSLint you will need for sanity sake :-)
If you plan to do a lot of CSS then first look at LESS or Sass.
And finally you will need something like Grunt to turn all this into something more optimal for loading (although the app nature of your project might not really require this; not sure).

KrisDS said...

I was thinking of adding RequireJS, but the complexity of what I'm doing is not very high yet; so I think I'll hold off on that.

I'm a big fan of Knockout. I have used it quite extensively in my Skribler project. Again debating whether to use it here, and again I don't think what I have done so far would benefit much from it.

LESS and Sass I have been reading a little about lately. Some really cool stuff there.

Everything els is new to me; so thanks for the links!

Yanic said...

Ah Node.js... (shivers)

If you have multiple DB-accesses on your back-end (likely, as a Node.js instance is single-threaded), the code will turn ugly quite quickly because of the many nested callbacks. Look into 'promises' to work around that.

A tip to save you a couple of hours of debugging : Node.js will buffer stderr en stdout if they are piped (e.g. when running in a debug session in an IDE). So if an error occurs, these buffers will not be flushed and you'll never see the error details. The solution is to define an event handler for 'uncaught exception' which flushes them 'manually'.

Beware that the module system in Node.js (just as the one in require.js) can be a bit surprising if you have any circular dependencies. Nothing complicated if you know what the cause is, but the error messages are not very clear about it - and if you're not using the tip above, good luck finding out what's going (wr)on(g) :)

KrisDS said...

@Yanic: thanks for the very helpful tips!

I know about futures and promises; but for such things you're a bit at the mercy of the library. Luckily I don't need a lot of complexity on the server side; it shouldn't go far beyond simple CRUD.