Velocity 2012: Building a Stronger and Faster Web

Velocity Conference Logo
O’Reilly’s Velocity Web Performance and Operations Conference, Velocity 2012, just wrapped up after three days, from June 25 through June 27th. Now in its fourth year, Velocity Conference is the premier conference for web operations and web performance professionals. The theme for this year’s conference was, “Stronger and Faster,” focusing on resiliency and performance of web systems.

There are four tracks, covering:

  • Web performance
  • Operations
  • Mobile performance
  • Culture

Day one consisted of in-depth talks on the topics of operations and web/mobile performance. These were longer than any of the sessions on the following two days, and tended to cover both concepts and walkthroughs of implementation details. Many had a “how-to” feel, which was great for getting attendees into the spirit of taking new ideas back to implement within their own infrastructure.

Day two opened with plenary sessions for all conference attendees, including Richard Cook’s “How Complex Systems Fail,” which is discussed in more detail later in this article. The afternoon continued with break out sessions, similar to the first day, with tracks covering operations, web performance, and mobile performance. Contrasted with day one, these talks were shorter, generally less than one hour, and were a mix of conceptual and implementation-oriented subjects.

Day three included the culture track, and again opened with all-attendee plenary sessions. Among these sessions was Jesse Robbins’ “Changing Culture & Being a force for Awesome,” also discussed in more detail later in this article. The afternoon sessions were had a similar structure to day two, with the addition of talks on culture.

A full playlist of the available videos can be found at

In this article, we’ll take a closer look at three of the talks from the conference, from the Operations and Culture tracks.

Velocity 2012: Richard Cook, “How Complex Systems Fail”

Tuesday morning, Richard Cook, M.D. presented “How Complex Systems Fail,” providing insight based on 25 years of systems research and experience in resilience engineering. Somewhat contrary to the title of the talk, he mostly focused on how complex systems can and do avoid failure, observing that it is often surprising that there are so few major failures, given the inherent fragility of complex systems. His major points included:

  • There is typically a significant difference between systems as imagined (or designed), and systems as found.
  • We design for reliability, but what we really want/expect is resilience.
  • Resilience is achieved through the combination of four operational activities: monitoring, responding, adapting, and learning.
  • To design for resilience, we should: build in support for continuous maintenance, expose systems controls, and support mental simulation, so that operators can anticipate the results of their actions.

While Dr. Cook’s research and experience mostly are not in the area of IT or web operations, his observations and conclusions feel as though they can be directly applied to those fields. The talk gives web operations practitioners a framework in which to reason about how–and why–we design systems to be resilient in a world that is not always well-behaved.

Video of the presentation is available on YouTube at

Velocity 2012: John Rauser, “Investigating Anomalies”

That same morning we also saw John Rauser, Principal Engineer at Amazon, give a talk entitled “Investigating Anomalies.” This talk followed up on his presentation from Velocity 2011, “Look at Your Data.” In this year’s talk, he focused on the ways in which data can be analyzed, and more importantly, how it can be misleading.

“Explaining anomalies often makes your theory bulletproof.” – John Rauser

Primarily, he focused on how relying only on big-picture, aggregate data can lead to the wrong conclusions. Much of the talk drew parallels to the history of English physician John Snow, and his groundbreaking discovery tying a contaminated well to a cholera outbreak in 1854 London. He described similarities to strategies for identifying the systemic causes of performance problems in web-scale applications. These strategies included:

  • Start with high-level summary data, but then drill down to histograms showing the distributions of that data, and ultimately consider anomalies and outliers in the raw data itself.
  • Look for variations over time in the long tail of your data, as this will be much more sensitive to changes in the distribution that could be early warnings of performance problems.
  • Strive to explain anomalies, as identifying the causes of these often will make your theory “bulletproof.”

All of these were neatly tied back to the story of John Snow’s discovery, making for a very entertaining and informative talk.

Video of the presentation is available on YouTube at

Velocity 2012: Jesse Robbins, “Changing Culture & Being a force for Awesome”

Wednesday, Jesse Robbins, cofounder of Opscode, presented one of the most popular talks of the conference, “Changing Culture & Being a force for Awesome.” The huge hall was packed with attendees–standing room only–there to hear Robbins’ talk about strategies for initiating cultural change in their organizations. He pointed out that while most of the conference is focused on technology, it is the culture that really matters. Most of the lessons he had to share were based on his past mistakes. Robbins said that changing culture takes time, but that by starting with some simple strategies, he had found it possible to accomplish major cultural shifts that otherwise would have been impossible.

‘Jesse’s Rule: “Don’t Fight Stupid, Make More Awesome”‘ – Jesse Robbins

The strategies were presented with a number of examples, from within organizations he’s worked in.

  • Start small, to build trust and a sense of safety.
  • Create champions, starting with the most important champion, your boss.
  • Use metrics to show progress and build confidence.
  • Celebrate successes, giving credit as widely as possible, to everyone but yourself.
  • Exploit compelling events, when they occur, to demonstrate the value of the change.

This was one of the most-anticipated and well-attended talks of the conference, and Robbins did not disappoint.

Video of the presentation if available on YouTube at