Don't ask your customers what they want!

The critical failing of user interviews is that you’re asking people to either remember past use or speculate on future use of a system


- Jakob Nielsen (former Apple, usability expert)

[The assumption that a] reasonably well-defined set of requirements exists, if only we take the time to understand them, is wrong


- Dean Leffingwell, author of Scaled Agile Framework

You can't just ask customers what they want and then try to give that to them. By the time you get it built, they'll want something new


- Steve Jobs, Apple

Customers don’t know what they want until you show it to them – this requires fast experimentation

Enter “A/B testing” or “split testing”

What is it?

How does it work?

Structured A/B testing

Do’s (and Don’t’s)


What is A/B Testing?

> Ingredients

  • two versions of an element (A and B)
  • a metric that defines success 

> Approach

  • Subject both versions to experimentation simultaneosly. Measure which version was more successful and select that version for real-world use


Examples of A/B Testing

> Google’s hyperlink gradients

> Obama's campaign website

What is not A/B Testing?

> Normal testing?

> Testing too many variations simultaneously?

OK. So, how does it work?

Implementation depends heavily on technology stack.

> Client side (SPA's, third party A/B testing tools)

> Server side (Custom built A/B testing functionality)

> On the infra level (load balancing)

Structured A/B testing

> Measure current website performance

> Prioritize testing opportunities

> Test!

> Learn and improve

Prioritize Testing Opportunities

> Find pages with highest improvement potential _or_ validate there is none

  - Top exit pages

  - Funnel drop-off rates

> Prioritize tests (value and cost)

> Prioritize important pages (most visited, top entry pages, etc.)


> Form a clear (problem and) hypothesis

> Ultimate goal

    - Test how to increase revenue


> See earlier discussion

Some do's

> Always test both versions simultaneously!

> Achieve statistical confidence!

   - Decide on sample size before testing, since that is  minimum threshold to achieve statistical confidence

> Repeated visitor? Same variation!

We have an unprecedented opportunity to run A/B tests with online users and innovate more quickly based on actual user response. Microsoft needs to shift the culture from planning the exact features to planning a set of possible features, and letting customers guide us


- Ray Ozzie

ABCD - AB with CD

Continuous Delivery >

Continuous Delivery

Base principles

> Rigorous Automation

> Extreme Feedback

> Continuous Change

"To improve is to change; to be perfect is to change often."

The six A’s to Continuous Delivery

Feedback, Feedback, Feedback

Continuous Delivery is a Prerequisite for Full A/B-Testing