Home Page Icon
Home Page
Table of Contents for
Evaluating Machine Learning Models
Close
Evaluating Machine Learning Models
by Alice Zheng
Evaluating Machine Learning Models
Preface
1. Orientation
The Machine Learning Workflow
Evaluation Metrics
Offline Evaluation Mechanisms
Hyperparameter Search
Online Testing Mechanisms
2. Evaluation Metrics
Classification Metrics
Accuracy
Confusion Matrix
Per-Class Accuracy
Log-Loss
AUC
Ranking Metrics
Precision-Recall
Precision-Recall Curve and the F1 Score
NDCG
Regression Metrics
RMSE
Quantiles of Errors
“Almost Correct” Predictions
Caution: The Difference Between Training Metrics and Evaluation Metrics
Caution: Skewed Datasets—Imbalanced Classes, Outliers, and Rare Data
Related Reading
Software Packages
3. Offline Evaluation Mechanisms: Hold-Out Validation, Cross-Validation, and Bootstrapping
Unpacking the Prototyping Phase: Training, Validation, Model Selection
Why Not Just Collect More Data?
Hold-Out Validation
Cross-Validation
Bootstrap and Jackknife
Caution: The Difference Between Model Validation and Testing
Summary
Related Reading
Software Packages
4. Hyperparameter Tuning
Model Parameters Versus Hyperparameters
What Do Hyperparameters Do?
Hyperparameter Tuning Mechanism
Hyperparameter Tuning Algorithms
Grid Search
Random Search
Smart Hyperparameter Tuning
The Case for Nested Cross-Validation
Related Reading
Software Packages
5. The Pitfalls of A/B Testing
A/B Testing: What Is It?
Pitfalls of A/B Testing
1. Complete Separation of Experiences
2. Which Metric?
3. How Much Change Counts as Real Change?
4. One-Sided or Two-Sided Test?
5. How Many False Positives Are You Willing to Tolerate?
6. How Many Observations Do You Need?
7. Is the Distribution of the Metric Gaussian?
8. Are the Variances Equal?
9. What Does the p-Value Mean?
10. Multiple Models, Multiple Hypotheses
11. How Long to Run the Test?
12. Catching Distribution Drift
Multi-Armed Bandits: An Alternative
Related Reading
That’s All, Folks!
Search in book...
Toggle Font Controls
Playlists
Add To
Create new playlist
Name your new playlist
Playlist description (optional)
Cancel
Create playlist
Sign In
Email address
Password
Forgot Password?
Create account
Login
or
Continue with Facebook
Continue with Google
Sign Up
Full Name
Email address
Confirm Email Address
Password
Login
Create account
or
Continue with Facebook
Continue with Google
Prev
Previous Chapter
Data
Next
Next Chapter
Evaluating Machine Learning Models
Evaluating Machine Learning Models
A Beginner’s Guide to Key
Concepts and Pitfalls
Alice Zheng
Add Highlight
No Comment
..................Content has been hidden....................
You can't read the all page of ebook, please click
here
login for view all page.
Day Mode
Cloud Mode
Night Mode
Reset