NOTE: This is the first part of a series (hopefully) that we’re trying out to see if you like it, find it interesting, helpful, whatever. Doing timings actually consumes a lot of time so we would really appreciate any feedback.
I still have vivid memories of the time when JBoss 4 started for 26s on my devel machine with only its vanilla apps deployed (JBoss 5 was twice as slow). I then thought that I’ll rant about it but for no apparent reason I did not.
Time has passed and I got some extra motivation to do a startup timings for containers with their default configurations. The motivation came from a colleague of mine, Anton Arhipov who also helped me time the containers and write this post.
Our plan
Take all containers and app servers out there and compare apples to oranges. Startup a AWS Large Instance (4 EC2 compute units, high I/O performance, 7.5GB of memory) and time the startup of vanilla containers with the latest JDK (1.6.0_22 at the time of running the tests). You can check out the configuration of Ubuntu and run your own tests via launching the ami-84fd0bed at Amazon AWS.
WTF, why? Don’t you have anything better to do?
Well, hopefully this will be a start of something bigger. Actually Anton’s idea was to make something similar to Matt Raible’s web framework comparison. Of course for a similar comparison of containers a lot more is needed than just startup time timings. Clustering, session replication, availability of services comes to mind at the moment. See comparison of application servers from Wikipedia.
I’m personally interested in the startup times because there are developers out there who don’t know that the startup times can have an order of magnitudes differences and vendors don’t care too much about developers’ productivity as just pushing out new features.
And we do have better stuff to do but we do need time off from our regular activities.
Results
Let’s start by observing the overall results of the startup times. As we can see on the graph below, the pure servlet containers, Tomcat and Jetty, are way ahead of the full blown containers, which is obvious (remember, we’re comparing apples to oranges here!). Maybe this data doesn’t help you very much, but actually you can make some conclusions out of it: try to implement your application in a way that it doesn’t depend on a heavyweight container with the slow startup time! Obviously, this is not always possible, but it is worth trying.
Next, lets see how the winners compare. Tomcat seems to be a fraction of a second faster than Jetty with Tomcat 7 being the fastest with under a second for the startup time. Comparing Tomcat to Jetty is pretty fair but not very reasonable as both are very fast. The only area where Tomcat really beats Jetty is the shutdown time. Jetty executes a shutdown hook when you try to stop it, and it takes almost as much as it takes to start the container. Tomcat, in opposite shuts down immediately.
The next chart shows the startup times for the heavyweight guys. We’re missing Geronimo in our test, this will be corrected later.
Let’s have a look at the outsider of the shootout – JBoss. We have to mention here is that the measurements we’re done with the default profile. JBoss is a very configurable application server in terms of services it runs, but we measured the default profile just because it seems that it is the most commonly used by developers (our wild guess, do comment if we’re way off).
You can notice a bad tendency for the startup times starting from JBoss 4.2 and growing until JBoss 5.1. We can see an improvement for JBoss 6.0, and there’s a remarkable improvement in startup time for JBoss 7.0 You might wonder what has changed for JBoss 7.0 so that the time improved an order of magnitude? The answer is that JBoss 7.0 is OSGi-enabled and it seems that it is not starting all its services on the startup modular! Again, this is what happens when you compare apples to oranges! To make the results look more real, we’d have to deploy a mid-size application to the container which would enable some of the services on the startup. We’ll do that in our upcoming tests.
Besides the graphs we’ve also have the data in a spreadsheet.
What next?
We see that OSGI containers are really fast to startup. We’re thinking of running the tests with an app deployed so that more modules are started up on startup (read: real life tests). Also we’ve left out commercial containers at the moment. This is due to we wanting to release the Amazon AMIs and not deal with licensing issues. We’ll definitely run tests with commercial containers if there is interested out there (please comment which container would you like to see in the shootout).
