Can we test everything?
No. there are situations we cannot analyze through A/B test.
For example, if you are adding a new experience and want to test it, old users may resist against the new version, or old users may all go for the new experience, then the test set has everything or nothing.
For example, a home rental webseit test its referral effect, but a customer may not return even in six months, it is very hard to measure through A/B testing. If this is the case, what shall we do?
- Analyze the user activity logs
- Conduct retrospective analysis
- Conduct user experience research
- Focus groups and surveys
- Human evaluation
How to do an A/B test?
1, Choose and characterize metrics to evaluate your experience, what do you care about? How do you want to measure the effect?
2. Choose significant level, statistical power and practical level you really want to launch the change if the test is statistically significant
3. Calculate required sample size
4. Take sample for control/treatment groups and run the test
5. Analyze the results and draw valid conclusions
