Rational Web Framework Choice
tl;dr: Determine with the most objectivity possible the best(s) web framework(s) depending on your needs. Here are the results. Please note the actual ability to take a rational decision is pretty bad for now.
This is it.
You’ve got the next big idea.
You just need to make a very simple web application.
It sounds easy! You just need to choose a good modern web framework, when suddenly:
After your brain stack overflowed, you decide to use a very simple methodology. Answer two questions:
Which language am I familiar with?
What is the most popular web framework for this language?
Great! This is it.
But, you continually hear this little voice.
“You didn’t made a bad choice, yes. But …
you hadn’t made the best either.”
This article try to determine in the most objective and rational way the best(s) web framework(s) depending on your needs. To reach this goal, I will provide a decision tool in the result section.
I will use the following methodology:
- Model how to make choice
- choose important parameters
- organize (hierarchize) them
- write down an objective chooser
- Grab objective quantified informations about web frameworks relatively to choosen parameters
- Sanitize your data in order to handle imprecisions, lack of informations…
- Apply the model of choice to your informations
☞ Important Note
I am far from happy to the actual result. There are a lot of biases, for example in the choice of the parameters. The same can be said about the data I gathered. I am using very imprecise informations. But, as far as I know, this is the only article which use many different parameters to help you choose a web framework.
This is why I made a very flexible decision tool:
Here are the important features (properties/parameters) I selected to make the choice:
- Popularity, which correlate with:
- number of tested libraries
- facility to find learning material
- ability to find another developer to work with
- Efficiency, which is generally correlated to:
- how much processing power you’ll need per user
- maintenance price per user
- how long the user will wait to see/update data
- Expressiveness, which is generally correlated to:
- faster development
- flexibility, adaptability
- Robustness, which correlate with:
- fewer bugs
Each feature is quite important and mostly independant from each other. I tried to embrace most important topics concerning web frameworks with these four properties. I am fully concious some people might lack another important feature. Nonetheless the methodology used here can be easily replicated. If you lack an important property add it at will and use this choice method.
Also each feature is very hard to measure with precision. This is why we will only focus on order of magnitude.
For each property a framework could have one of the six possible values: Excellent, Very Good, Good, Medium, Bad or Very Bad
So how to make a decision model from these informations?
One of the most versatile method is to give a weight for each cluster value. And to select the framework maximizing this score:
score(framework) = efficiency + robustness + expressiveness + popularity
Expressiveness 10 7 1 -∞ -∞ -∞ Popularity 5 5 4 3 2 1 Efficiency 10 8 6 4 2 1 Robustness 10 8 6 4 2 1
Using this weighted table, that means:
- we discard the three least expressive clusters.
- We don’t make any difference between excellent and very good in popularity.
- Concerning efficient framework in excellent cluster will have 2 more points than the “very good” cluster.
So for each framework we compute its score relatively to a weighted table. And we select the best(s).
Example: Using this hypothetic framework and the preceeding table.
Expressiveness Popularity Efficiency Robustness yog Excellent Very Bad Medium Very Good
score(yog) = 10 + 0 + 4 + 8 = 22
Most needs should be expressed by such a weighted table. In the result section, we will discuss this further.
It is now time to try to get these measures.
None of the four properties I choosen can be measured with perfect precision. But we could get the order of magnitude for each.
I tried to focus on the framework only. But it is often easier to start by studying the language first.
For example, I have datas about popularity by language and I also have different datas concerning popularity by framework. Even if I use only the framework focused datas in my final decision model, it seemed important to me to discuss about the datas for the languages. The goal is to provide a tool to help decision not to give a decision for you.
RedMonk Programming Language Rankings (January 2013) provide an apparent good measure of popularity. While not perfect the current measure feel mostly right. They create an image using stack overflow and github data. Vertical correspond to the number of questions on stackoverflow. Horizontal correspond to the number of projects on github.
If you look at the image, your eye can see about four clusters. The 1ﬆ cluster correspond to mainstream languages:
Most developer know at least one of these language.
The second cluster is quite bigger. It seems to correspond to languages with a solid community behind them.
I don’t get into detail, but you could also see third and fourth tier popular languages.
C, Objective-C, Perl, Shell
Good: Scala, Haskell, Visual Basic, Assembly, R, Matlab, ASP, ActionScript, Coffeescript, Groovy, Clojure, Lua, Prolog
Medium: Erlang, Go, Delphi, D, Racket, Scheme, ColdFusion, F#, FORTRAN, Arduino, Tcl, Ocaml
Bad: third tier Very Bad: fourth tier
I don’t thing I could find easily web frameworks for third or fourth tier languages.
For now, I only talked about language popularity. But what about framework popularity? I made a test using number of question on stackoverflow only. Then by dividing by two for each 6 cluster:
As we can see, our framework popularity indicator can be quite different from its language popularity. For now I didn’t found a nice way to merge the results from RedMonk with these one. So I’ll use these unperfect one. Hopefully the order of magninute is mostly correct for most framework.
Another objective measure is efficiency. We all know benchmarks are all flawed. But they are the only indicators concerning efficiency we have.
I used the benchmark from benchmarksgame. Mainly, there are five clusters:
|2x→3x||Java 7, Scala, OCamL, Haskell, Go, Common LISP|
|3x→10x||C♯, Clojure, Racket, Dart|
|30x→||PHP, Python, Perl, Ruby, JRuby|
Remarks concerning some very slow languages:
- PHP ; huge variations, can be about 1.5x C speed in best case.
- Python ; huge variations, can be about 1.5x C speed in best case
- Perl ; Can be about 3x C speed in best case
- Ruby, JRuby ; mostly very slow.
This is a first approach. The speed of the language for basic benchmarks. But, here we are interrested in web programming. Fortunately techempower has made some tests focused on most web frameworks:
These benchmark doesn’t fit well with our needs. The values are certainly quite imprecise to your real usage. The goal is just to get an order of magnitude for each framework. Another problem is the high number of informations.
As always, we should remember these informations are also imprecise. So I simply made some classes of efficiency.
Remark: I separated the clusters by using power of 2 relatively to the fastest.
These are manually made clusters. But you get the idea. Certainly, some framework could jump between two different clusters. So this is something to remember. But as always, the order of magnitude is certainly mostly right.
Now, how to objectively measure expressiveness?
RedMonk had a very good idea to find an objective (while imprecise) measure of each language expressiveness. Read this article for details.
After filtering languages suitable for web development, we end up with some clusters:
|Excellent||Coffeescript, Clojure, Haskell|
|Very Good||Racket, Groovy, R, Scala, OCamL, F♯, Erlang, Lisp, Go|
|Medium||Perl, Python, Objective-C, Scheme, Tcl, Ruby|
|Bad||Lua, Fortran (free-format), PHP, Java, C++, C♯|
I couldn’t find any complete study to give the number of bug relatively to each framework/language.
But one thing I saw from experience is the more powerful the type system the safest your application is. While the type system doesn’t remove completely the need to test your application a very good type system tend to remove complete classes of bug.
Typically, not using pointer help to reduce the number of bugs due to bad references. Also, using a garbage collector, reduce greatly the probability to access unallocated space.
From my point of view, robustness is mostly identical to safety.
Here are the clusters:
|Excellent||Haskell, Scheme, Erlang|
|Very Good||Scala, Java, Clojure|
|Medium||C++, C#, Perl, Objective-C, Go, C|
So applying this to frameworks gives the following clusters:
For the result I initialized the table with my own needs.
And I am quite happy it confirms my current choice. I sware I didn’t given yesod any bonus point. I tried to be the most objective and factual as possible.
Now, it is up to you to enter your preferences.
On each line you could change how important a feature is for you. From essential to unsignificant. Of course you could change the matrix at will.
I just show a top 10 frameworks. In order to give a more understandable measure I provide the log of the score.
I didn’t had the courage in explaining in what the scoring system is good. Mostly, if you use product instead of sum for the score you could use power of e for the values in the matrix. And you could see the matrix as a probability matrix (each line sum to 1). Which provide a slighly better intuition on whats going on.
Remember only that values are exponential. Do not double an already big value for example the effect would be extreme.
All of this is based as most as I could on objective data. The choice method seems both rather rational and classical. It is now up to you to edit the score matrix to set your needs.
I know that in the current state there are many flaws. But it is a first system to help make a choice rationally.
I encourage you to go further if you are not satisfied by my method.
The source code for the matrix shouldn’t be too hard to read. Just read the source of this webpage. You could change the positionning of some frameworks if you believe I made some mistake by placing them in some bad clusters.
So I hope this tool will help you in making your life easier.