Preface
Data preprocessing is the first step in data visualization, data analytics, and machine learning, where data is prepared for analytics functions to get the best possible insights. Around 90% of the time spent on data analytics, data visualization, and machine learning projects is dedicated to performing data preprocessing.
This book will equip you with the optimum data preprocessing techniques from multiple perspectives. You'll learn about different technical and analytical aspects of data preprocessing – data collection, data cleaning, data integration, data reduction, and data transformation – and get to grips with implementing them using the open source Python programming environment. This book will provide a comprehensive articulation of data preprocessing, its whys and hows, and help you identify opportunities where data analytics could lead to more effective decision making. It also demonstrates the role of data management systems and technologies for effective analytics and how to use APIs to pull data.
By the end of this Python data preprocessing book, you'll be able to use Python to read, manipulate, and analyze data; perform data cleaning, integration, reduction, and transformation techniques; and handle outliers or missing values to effectively prepare data for analytic tools.
Who this book is for
Junior and senior data analysts, business intelligence professionals, engineering undergraduates, and data enthusiasts looking to perform preprocessing and data cleaning on large amounts of data will find this book useful. Basic programming skills, such as working with variables, conditionals, and loops, along with beginner-level knowledge of Python and simple analytics experience, are assumed.
What this book covers
Chapter 1, Review of the Core Modules of NumPy and Pandas, introduces two of three main modules used for data manipulation, using real dataset examples to show their relevant capabilities.
Chapter 2, Review of Another Core Module – Matplotlib, introduces the last of the three modules used for data manipulation, using real dataset examples to show its relevant capabilities.
Chapter 3, Data – What Is It Really?, puts forth a technical definition of data and introduces data concepts and languages that are necessary for data preprocessing.
Chapter 4, Databases, explains the role of databases, the different kinds, and teaches you how to connect and pull data from relational databases. It also teaches you how to pull data from databases using APIs.
Chapter 5, Data Visualization, showcases some analytics examples using data visualizations to inform you of the potential of data visualization.
Chapter 6, Prediction, introduces predictive models and explains how to use Multivariate Regression and a Multi-Layered Perceptron (MLP).
Chapter 7, Classification, introduces classification models and explains how to use Decision Trees and K-Nearest Neighbors (KNN).
Chapter 8, Clustering Analysis, introduces clustering models and explains how to use K-means.
Chapter 9, Data Cleaning Level I – Cleaning Up the Table, introduces three different levels of data cleaning and covers the first level through examples.
Chapter 10, Data Cleaning Level II – Unpacking, Restructuring, and Reformulating the Table, covers the second level of data cleaning through examples.
Chapter 11, Data Cleaning Level III – Missing Values, Outliers, and Errors, covers the third level of data cleaning through examples.
Chapter 12, Data Fusion and Data Integration, covers the technique for mixing different data sources.
Chapter 13, Data Reduction, introduces data reduction and, with the help of examples, shows how its different cases and versions can be done via Python.
Chapter 14, Data Transformation and Massaging, introduces data transformation and massaging and, through many examples, shows their requirements and capabilities for analysis.
Chapter 15, Case Study 1 – Mental Health in Tech, introduces an analytic problem and preprocesses the data to solve it.
Chapter 16, Case Study 2 – Predicting COVID-19 Hospitalizations, introduces an analytic problem and preprocesses the data to solve it.
Chapter 17, Case Study 3 – United States Counties Clustering Analysis, introduces an analytic problem and preprocesses the data to solve it.
Chapter 18, Summary, Practice Case Studies, and Conclusions, introduces some possible practice cases that users can use to learn in more depth and start creating their analytics portfolios.
To get the most out of this book
The book assumes basic programming skills such as working with variables, conditionals, and loops, along with beginner-level knowledge of Python. Other than that, you can start your journey from the beginning of the book and start learning.
The Jupyter Notebook is an excellent UI for learning and practicing programming and data analytics. It can be downloaded and installed easily using Anaconda Navigator. Visit this page for installation: https://docs.anaconda.com/anaconda/navigator/install/.
While Anaconda has most of the modules that the book uses already installed, you will need to install a few other modules such as Seaborn and Graphviz. Don't worry; when the time comes, the book will instruct you on how to go about these installations.
If you are using the digital version of this book, we advise you to type the code yourself or access the code from the book's GitHub repository (a link is available in the next section). Doing so will help you avoid any potential errors related to the copying and pasting of code.
While learning, keep a file of your own code from each chapter. This learning repository can be used in the future for deeper learning and real projects. The Jupyter Notebook is especially great for this purpose as it allows you to take notes along with the code.
Download the example code files
You can download the example code files for this book from GitHub at https://github.com/PacktPublishing/Hands-On-Data-Preprocessing-in-Python. If there's an update to the code, it will be updated in the GitHub repository.
We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!
Download the color images
We also provide a PDF file that has color images of the screenshots and diagrams used in this book. You can download it here: https://static.packt-cdn.com/downloads/9781801072137_ColorImages.pdf.
Conventions used
There are a number of text conventions used throughout this book.
Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: "To create this interactive visual, we have used the interact and widgets programming objects from the ipywidgets module."
A block of code is set as follows:
from ipywidgets import interact, widgets
interact(plotyear,year=widgets.IntSlider(min=2010,max=2019,step=1,value=2010))
When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:
Xs_t.plot.scatter(x='PC1',y='PC2',c='PC3',sharex=False,
vmin=-1/0.101, vmax=1/0.101,
figsize=(12,9))
x_ticks_vs = [-2.9*4 + 2.9*i for i in range(9)]
Bold: Indicates a new term, an important word, or words that you see on screen. For instance, words in menus or dialog boxes appear in bold. Here is an example: "The missing values for the attributes from SupportQ1 to AttitudeQ3 are from the same data objects."
Tips or Important Notes
Appear like this.
Get in touch
Feedback from our readers is always welcome.
General feedback: If you have questions about any aspect of this book, email us at [email protected] and mention the book title in the subject of your message.
Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata and fill in the form.
Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.
If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.
Share Your Thoughts
Once you've read Hands-On Data Preprocessing in Python, we'd love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.
Your review is important to us and the tech community and will help us make sure we're delivering excellent quality content.