The large number of techniques to process natural language for its use in machine learning models that we introduced in this section is necessary to address the complex nature of this highly unstructured data source. The engineering of good language features is both challenging and rewarding and is arguably the most important step in unlocking the semantic value hidden in text data.

In practice, experience helps us select transformations that remove noise rather than the signal, but it will likely remain necessary to cross-validate and compare the performance of different combinations of preprocessing choices.