The critical failing of user interviews is that you’re asking people to either remember past use or speculate on future use of a system
- Jakob Nielsen (former Apple, usability expert)
[The assumption that a] reasonably well-defined set of requirements exists, if only we take the time to understand them, is wrong
- Dean Leffingwell, author of Scaled Agile Framework
You can't just ask customers what they want and then try to give that to them. By the time you get it built, they'll want something new
- Steve Jobs, Apple
Customers don’t know what they want until you show it to them – this requires fast experimentation
Enter “A/B testing” or “split testing”
What is it?
How does it work?
Structured A/B testing
Do’s (and Don’t’s)
> Normal testing?
> Testing too many variations simultaneously?
Implementation depends heavily on technology stack.
> Client side (SPA's, third party A/B testing tools)
> Server side (Custom built A/B testing functionality)
> On the infra level (load balancing)
> Measure current website performance
> Prioritize testing opportunities
> Learn and improve
> Find pages with highest improvement potential _or_ validate there is none
- Top exit pages
- Funnel drop-off rates
> Prioritize tests (value and cost)
> Prioritize important pages (most visited, top entry pages, etc.)
> Form a clear (problem and) hypothesis
> Ultimate goal
- Test how to increase revenue
> See earlier discussion
> Always test both versions simultaneously!
> Achieve statistical confidence!
- Decide on sample size before testing, since that is minimum threshold to achieve statistical confidence
> Repeated visitor? Same variation!
We have an unprecedented opportunity to run A/B tests with online users and innovate more quickly based on actual user response. Microsoft needs to shift the culture from planning the exact features to planning a set of possible features, and letting customers guide us
- Ray Ozzie
Continuous Delivery >
> Rigorous Automation
> Extreme Feedback
> Continuous Change
"To improve is to change; to be perfect is to change often."