Can't Test This... A/B Testing

A/B testing is a very powerful tool for developing certain kinds of products. Here are a few thoughts on where it does and doesn't work.

Below is a high level flow of the testing cycle.

A/B testing works when:

• The cost of implementing ver A & B + 
• The cost of collecting "enough" data about A&B + 
• The cost of fanning out the "best" version, 

is less than:

• The cost of visiting "enough" of your key customers + 
• The cost of spending "enough" time with each of them to understand the full requirements.

Or framed a different way...

If it costs a significant amount to develop, test or deploy the thing you want to evaluate or

If you cannot get adequate information back from the customer base or

If you cannot get information back in a reasonable amount of time,

then there may be better development approaches than A/B testing.