Machine Learning
Data Management
Train and Evaluate
Productize It
Hands-On
A/B Testing
You will learn: an effective way to monitor and measure your solution via A/B testing.
.png?alt=media&token=ce7573b8-c156-4b1d-8f7e-77a32380289d)
We can find multiple strategies to measure your machine learning model and product impact over your business, such as reactions from your stakeholders, partners, and employees, Return on Investment (ROI), among others, but in this course, we will focus on the well-known A/B testing approach.
A/B testing can help to obtain research statistical significance when running two different experiments in order to obtain a better sense of your customers.
And after each iteration, try to answer:
- Did the user visit the next day?
- How long did the user visit the site?
- What were the daily active users?
Along with the study, you should conduct user research questions like:
- Is the user happy using the product?
- Is the user satisfied with the experience?
- Is the product improving the user’s overall wellbeing?
- How will this affect the company’s overall health?
Since most all of these questions are hard to measure, Product Managers tend to use proxies in order to assess questions in an open way: if the user is happy with the product, they will stay on the site for a longer time. If the user is satisfied, they will visit again tomorrow.
Another recommended approach when measuring and monitoring your solution in order to satisfy your business continuity requirements when launching an AI solution is to try to encode business metrics into your loss function in order to evaluate how well your models are doing regarding your business needs.
Readings
A/B Testing: How to Get It Right
Harvard Business Review
A/B Testing — What Role Does it Play in the Era of Machine Learning?
Medium
A/B Testing Machine Learning Models (Deployment Series: Guide 08) - ML in Production
ML in Production
Last modified 7mo ago
Copy link
Contents
Readings