When you see a new machine learning problem in the wild, you might be tempted to jump ahead and throw your favorite algorithm at the problem—perhaps the one you understood best or had the most fun implementing. But knowing beforehand which algorithm will perform best on your specific problem is not often possible.
Instead, you need to take a step back and look at the bigger picture. Before you get in too deep, you will want to define the actual problem you are trying to solve. For example, do you already have a specific goal in mind, or are you just looking to do some exploratory analysis and find something interesting in the data? Often, you will start with a general goal, such as detecting spam email messages, making movie recommendations, or automatically tagging your friends in pictures uploaded to a social media platform. However, as we have seen throughout this book, there are often several ways to solve a problem. For example, we recognized handwritten digits using logistic regression, k-means clustering, and deep learning. Defining the problem will help you to ask the right questions and make the right choices along the way.
As a rule of thumb, you can use the following five-step procedure to approach machine learning problems in the wild:
- Categorize the problem: This is a two-step process:
- Categorize by input: Simply speaking, if you have labeled data, it's a supervised learning problem. If you have unlabeled data and want to find structure, it's an unsupervised learning problem. If you want to optimize an objective function by interacting with an environment, it's a reinforcement learning problem.
- Categorize by output: If the output of your model is a number, it's a regression problem. If the output of your model is a class (or category), it's a classification problem. If the output of your model is a set of input groups, it's a clustering problem.
- Find the available algorithms: Now that you have categorized the problem, you can identify the algorithms that are applicable and practical to implement using the tools at our disposal. Microsoft has created a handy algorithm cheat sheet that shows which algorithms can be used for which category of problems. Although the cheat sheet is tailored toward Microsoft Azure, you might find it generally helpful.
- Implement all of the applicable algorithms (prototyping): For any given problem, there are usually a handful of candidate algorithms that could do the job. So, how do you know which one to pick? Often, the answer to this problem is not straightforward, so you have to resort to trial and error. Prototyping is best done in two steps:
- You should aim for a quick and dirty implementation of several algorithms with minimal feature engineering. At this stage, you should mainly be interested in seeing which algorithm behaves better at a coarse scale. This step is a bit like hiring: you're looking for any reason to shorten your list of candidate algorithms. Once you have reduced the list to a few candidate algorithms, the real prototyping begins.
- Ideally, you would want to set up a machine learning pipeline that compares the performance of each algorithm on the dataset using a set of carefully selected evaluation criteria (see Chapter 11, Selecting the Right Model with Hyperparameter Tuning). At this stage, you should only be dealing with a handful of algorithms, so you can turn your attention to where the real magic lies: feature engineering.
- Feature engineering: Perhaps even more important than choosing the right algorithm is choosing the right features to represent the data. You can read all about feature engineering in Chapter 4, Representing Data and Engineering Features.
- Optimize hyperparameters: Finally, you also want to optimize an algorithm's hyperparameters. Examples might include the number of principal components of PCA, the parameter, k, in the k-nearest neighbor algorithm, or the number of layers and learning rate in a neural network. You can look at Chapter 11, Selecting the Right Model with Hyperparameter Tuning, for inspiration.