Inside Urban Airship Engineering: Achieving Massive Scale and Value

This post explores some of the things we’re doing behind the scenes in Urban Airship Engineering. Over the next few months, members of the engineering team will offer insight into how we operate, lessons we’ve learned, open source projects we've created and some of the challenges we face serving hundreds of millions of instances of apps connecting to our services every day.

Culturally and from a process standpoint, Urban Airship Engineering is focused on learning and adapting through continuous improvement. We wish we were smart enough to have invented these techniques, but luckily we didn’t have to. The way we conduct product development day to day is an adaptation of ideas in various books including the Poppendieck’s books on Lean Software and David Anderson’s Kanban.

“Value Stream” is a term that originated with Lean Manufacturing, meaning to analyze and design the flow of materials and information required to bring a product or service to a consumer. The whole idea is to drive out waste, deliver fast, build quality in, and engage all of the people involved in the value stream so that great products emerge.

"The whole idea is to drive out waste, deliver fast, build quality in, and engage all of the people involved in the value stream so that great products emerge."

Urban Airship has had a lot of success with Kanban and Kaizen. Don’t let those words scare you off if you don’t speak Japanese, they just mean “billboard” and “change for the better,” respectively. This post explores our high-level history in applying these concepts and how we’ve begun to scale our process across our San Francisco and Portland offices as necessitated by bringing Urban Airship and SimpleGeo together as one company last October.

Model What’s Happening Now and Let Improvement Emerge

We started with Kanban and Kaizen when I joined Urban Airship in October 2010. The engineering team at the time was following an adaptation of Agile/Scrum and while it was producing results, it wasn’t working as well as we wanted it to in serving all the needs of a venture-backed startup. We adapted our operating model to a starter version of what is described in Anderson’s Kanban book.

A great thing about Kanban is starting is easy; you just model the value stream as it is and define work-item types that are meaningful to the team (e.g., Bug Pack, Minimum Marketable Feature, Business Enablement, Refactoring/Technical Debt). Each work-item that is in process, planned, or recently completed gets hung up on a physical board with lanes that indicate the current value stream.

Urban Airship engineers discussing the original set of work-items and where they fit in the value stream in October 2010
Airshippers discussing the original set of work-items and where they fit in the value stream in October 2010

Urban Airship's Kanban board for our engineering team
The original wee Kanban board at Urban Airship’s previous office above PIE

Every morning we have a Product Development Boardwalk. This is a standup style meeting where we walk through every work-item on the board in no longer than 15 minutes. We rotate the facilitator of this meeting every day to keep it fresh and make sure that everyone knows the process well enough that they can lead the team through it. This establishes shared context for everyone on what is happening in the value stream. It's amazing what smart people can achieve together when they have the same context! Physical Kanban boards are still unrivaled for enabling this.

We tweaked our value stream a bit here and there as we learned from successes and failures. We introduced new work-item types, changed WIP limits, added exit criteria, added lanes, split lanes etc. We broke out a separate board right next to Product Development for Customer Development & GTM (Go-to-Market). We made these changes based on insights from our monthly Operational Review & Retrospective and from 5 Whys from production incidents and other defects.

A Year Later: SimpleGeo Acquisition, Investment and Development Partners

In October of 2011, Urban Airship acquired San Francisco-based SimpleGeo, took on a new round of financing and executed some major business agreements that would cause the company to grow even more quickly than it was already. In order to keep work-items flowing, it was time to begin to scale our value stream.

Urban Airship is a very in-person company. In order to make our value stream work across offices, we decided to replicate it in both locations. To achieve the same feel we had when it was just PDX, we now alternate facilitating offices every week. Individuals in the facilitating office also rotate every day. We’re still working out the kinks to be sure, but it’s working and is a lot of fun.

Our Kanban board was showing its age even before these events occurred. We had grown engineering four-fold and with that it was getting harder to see everything and achieve shared context.

In order to address this and begin to scale into small sets of focused teams, we introduced horizontal lanes for each major part of our product line. We have kept the meeting wide open to anyone who wants to participate or observe, but now just the engineering leadership team (leaders of functional areas, program managers, product managers, tech leads, team leads, etc.) attends the Product Development Boardwalk every morning. Many of these leaders maintain what we refer to as a “Zoom In” Kanban board, which is focused on a specific functional area or work-item that rolls up to the main Product Development board.

Urban Airship's Product Development Kanban boards in our Portland and San Francisco offices
The Product Development Kanban board in PDX (left wall) and Polycom unit with the replica board in SF

Urban Airship's Kanban board being displayed in our San Francisco office
The SF Polycom unit displaying the PDX Kanban board

Urban Airship's Product Development Boardwalk meeting
A real live Product Development Boardwalk meeting with Wade Simmons from SF facilitating via Polycom

Urban Airship's Product Development board in San Francisco
The SF Product Development board (notice the glare from the warm California sun)

Example of a “Zoom In” Kanban board in Urban Airship's Messaging Feature Room in Portland Oregon
An example “Zoom In” Kanban board (far wall) in our Messaging Feature Room in PDX

Looking Forward

Change is constant at Urban Airship. As we continue to grow our team, scale to a billion devices using our services and build new features and products, we’ll keep making adaptions to how we operate in Urban Airship Engineering.

Tools / Technology

We use the following tools and technology to power our Kanban:

  1. The biggest baddest magnetic white boards we can find
  2. The most powerful magnets we can find
  3. Magnet pictures of our people
  4. White board markers
  5. Colored note cards (indicate different classes of service)
  6. Colored markers (to indicate different work-item types (e.g., Minimum Marketable Feature, Bug Pack)
  7. Polycom to power the video conferencing between offices
  8. Google Hangouts and Skype for Zoom In video conferencing
  9. LeanKitKanban – web-based Kanban board we use to help sync the SF and PDX boards. It also produces some great metrics and graphs that we use to understand what is happening across product development (e.g., Continuous Flow Diagram, Cycle Time per lane, per work-item type, etc., Card Distribution Diagrams, Efficiency Diagrams, and even a Process Control Diagram)
  10. JIRA – we use it to track details of work-items and as the electronic record we reference in source control, etc.

Sound like a value stream you'd thrive in? We're always looking for good people in both our San Francisco and Portland Offices.

Urban Airship Engineering hires people with opinions who care deeply about their work, technology, the products they build and making a huge impact on the market. Every person on our team is asked to be part of the solution and to contribute a ton.

Come join us: