The Low Latency Web

March 23, 2012

A Note On Benchmarking

Filed under: HTTP Servers — lowlatencyweb @ 2:42 pm

This series of articles has drawn out the cargo cultists who insist that HTTP benchmarks must be run over the network, or that a real application should be tested, as if adding more variables makes a benchmark more useful. What would adding NIC overhead and network latency into the mix prove when the goal is to test a HTTP server, or a dynamic content stack? Zed Shaw addresses this in an excellent article on confounding, but the fundamental point is that a useful benchmark must isolate the variable being tested.

Critics may claim that for many web applications the database is the bottleneck so there is little value in testing anything else. If so that’s easily addressed with in-memory caching and alternative data stores, and then what becomes the bottleneck? Probably the HTTP server or dynamic content stack. Modern hardware and databases are very efficient and performance problems tend to be caused by poor choices in hardware or software rather than database or network overhead.

However in the interest of comprehensiveness here is a plot showing the previous test of dynamic content when the client and server are on separate machines, communicating over a gigabit VLAN. Please note that no TCP tuning was done other than the original tuning to increase the number of ephemeral ports. Both machines are identical hardware from SoftLayer (with whom my only affiliation is that of a satisfied customer).

The absolutely difference between localhost and running over the VLAN is relatively uninteresting given that it is dependent on the switching hardware, NIC hardware & driver, kernel, TCP tuning parameters, etc. Latency increases as expected, but requests/sec appear to be converging.



  1. A lot of the opposition live in sheer denial. Do a test of a moderate Rails or Django app — I will be VERY surprised if you can break 1K/s on this hardware. This is not a symptom of data stores or caching, it’s a symptom of slow languages and bloated frameworks. It hurts all around too. On a low price VM characteristic of a startup, that number is going to be in the low 100s.

    Time to cut the hype and focus on the old new things. The posts on this blog have been refreshing thus far. Keep it up!

    Comment by Kevin Bowling — March 24, 2012 @ 9:22 pm

    • Thanks Kevin! You’re spot on, and I’m hoping that this blog will serve to highlight alternatives to mainstream thinking on web app design.

      Comment by lowlatencyweb — March 25, 2012 @ 3:43 am

  2. Alarmed at the mention of cargo cultists in our midst, I looked over your previous posts to try to find them. I don’t really see anyone making the arguments you are talking about.

    However, regarding your EC2 vs. bare metal comparison a few days back, I am still curious about the bottleneck that made EC2 so much slower. Was it maxing out CPU? Was there disk thrashing? We don’t know because you didn’t publish any data about that! Hopefully it’s not cargo cultism to look for metrics to explain *why* the results were the way they were, and whether or not it’s tunable.

    Great blog though, I am enjoying the benchmarks. Please don’t introduce more variables into your tests, but do consider measuring CPU/IOWait/etc if it’s not much trouble.

    Comment by azhawkes — March 26, 2012 @ 4:04 pm

    • The relevant comments have been largely moderated away as they added little value to the discussion. If curious you might look up the related discussions on Hacker News for a sampling of typical comments.

      I published everything necessary to reproduce the EC2 experiment, so please do! I would be most interested if you found a way to significantly improve the results, and might well write a followup post.

      Comment by lowlatencyweb — March 26, 2012 @ 10:29 pm

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at

%d bloggers like this: