Sayan Mukhopadhyay and Pratip Samanta

Advanced Data Analytics Using Python

With Architectural Patterns, Text and Image Classification, and Optimization Techniques

2nd ed.
Sayan Mukhopadhyay
Kolkata, West Bengal, India
Pratip Samanta
Kolkata, West Bengal, India
ISBN 978-1-4842-8004-1e-ISBN 978-1-4842-8005-8
https://doi.org/10.1007/978-1-4842-8005-8
© Sayan Mukhopadhyay, Pratip Samanta 2018, 2023
Apress Standard
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use.
The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, express or implied, with respect to the material contained herein or for any errors or omissions that may have been made.

This Apress imprint is published by the registered company APress Media, LLC, part of Springer Nature.

The registered company address is: 1 New York Plaza, New York, NY 10004, U.S.A.

The reason for the success of this book is that it has original research, so I dedicate it to the person from whom I learned how to do research: Dr. Debnath Pal, IISc.

—Sayan Mukhopadhyay

Introduction

We are living in the data science/artificial intelligence era. To thrive in this environment, where data drives decision-making in everything from business to government to sports and entertainment, you need the skills to manage and analyze huge amounts of data. Together we can use this data to make the world better for everyone. In fact, humans have yet to find everything we can do using this data. So, let us explore!

Our objective for this book is to empower you to become a leader in this data-transformed era. With this book you will learn the skills to develop AI applications and make a difference in the world.

This book is intended for advanced user, because we have incorporated some advanced analytics topics. Important machine learning models and deep learning models are explained with coding exercises and real-world examples.

All the source code used in this book is available for download at https://github.com/apress/advanced-data-analytics-python-2e.

Happy reading!

Acknowledgments

Thanks to Labonic Chakraborty (Ripa) and Soumili Chakraborty.

Table of Contents
Chapter 1:​ A Birds Eye View to AI System1
OOP in Python1
Calling Other Languages in Python13
Exposing the Python Model as a Microservice14
High-Performance API and Concurrent Programming17
Choosing the Right Database21
Summary22
Chapter 2:​ ETL with Python23
MySQL24
How to Install MySQLdb?​24
Database Connection25
INSERT Operation26
READ Operation26
DELETE Operation27
UPDATE Operation28
COMMIT Operation28
ROLL-BACK Operation28
Normal Forms31
First Normal Form31
Second Normal Form32
Third Normal Form33
Elasticsearch35
Connection Layer API38
Neo4j Python Driver39
neo4j-rest-client39
In-Memory Database40
MongoDB (Python Edition)40
Import Data into the Collection41
Create a Connection Using pymongo42
Access Database Objects42
Insert Data43
Update Data43
Remove Data43
Cloud Databases43
Pandas44
ETL with Python (Unstructured Data)45
Email Parsing45
Topical Crawling48
Summary52
Chapter 3:​ Feature Engineering and Supervised Learning53
Dimensionality Reduction with Python54
Correlation Analysis55
Principal Component Analysis57
Mutual Information60
Classifications with Python64
Semi-Supervised Learning65
Decision Tree66
Which Attribute Comes First?​66
Random Forest Classifier68
Naïve Bayes Classifier68
Support Vector Machine69
Nearest Neighbor Classifier71
Sentiment Analysis71
Image Recognition73
Regression with Python74
Least Square Estimation75
Logistic Regression76
Classification and Regression76
Intentionally Bias the Model to Over-Fit or Under-Fit77
Dealing with Categorical Data78
Summary79
Chapter 4:​ Unsupervised Learning:​ Clustering81
K-Means Clustering82
Choosing K:​ The Elbow Method86
Silhouette Analysis86
Distance or Similarity Measure88
Properties89
General and Euclidean Distance89
Squared Euclidean Distance91
Distance Between String-Edit Distance91
Similarity in the Context of a Document93
Types of Similarity94
Example of K-Means in Images95
Preparing the Cluster97
Thresholding99
Time to Cluster100
Revealing the Current Cluster102
Hierarchical Clustering104
Bottom-Up Approach104
Distance Between Clusters105
Top-Down Approach107
Graph Theoretical Approach111
How Do You Know If the Clustering Result Is Good?​112
Summary113
Chapter 5:​ Deep Learning and Neural Networks115
Backpropagation116
Backpropagation Approach116
Other Algorithms119
TensorFlow120
Network Architecture and Regularization Techniques124
Updatable Model and Transfer Learning124
Recurrent Neural Network133
LSTM134
Reinforcement Learning138
TD0139
TDλ145
Example of Dialectic Learning146
Convolution Neural Networks156
Summary159
Chapter 6:​ Time Series161
Classification of Variation161
Analyzing a Series Containing a Trend161
Curve Fitting163
Removing Trends from a Time Series163
Analyzing a Series Containing Seasonality164
Removing Seasonality from a Time Series165
By Filtering165
By Differencing166
Transformation166
To Stabilize the Variance167
To Make the Seasonal Effect Additive167
To Make the Data Distribution Normal167
Stationary Time Series168
Stationary Process168
Autocorrelation and the Correlogram169
Estimating Autocovariance and Autocorrelation Functions170
Time-Series Analysis with Python171
Useful Methods171
Autoregressive Processes173
Estimating Parameters of an AR Process175
Mixed ARMA Models177
Integrated ARMA Models179
The Fourier Transform181
An Exceptional Scenario182
Missing Data183
Summary184
Chapter 7:​ Analytics at Scale185
Hadoop185
MapReduce Programming185
Partitioning Function186
Combiner Function187
HDFS File System197
MapReduce Design Pattern197
A Notes on Functional Programming203
Spark204
PySpark207
Updatable Machine Learning and Spark Memory Model209
Analytics in the Cloud221
Internet of Things231
Essential Architectural Patterns for Data Scientists231
Scenario 1:​ Hot Potato Anti-Pattern232
Scenario 2:​ Proxy and Layering Patterns236
Thank You241
Index243
About the Authors
Sayan Mukhopadhyay

A photograph of Sayan Mukhopadhyay.

has more than 13 years of industry experience and has been associated with companies such as Credit Suisse, PayPal, CA Technologies, CSC, and Mphasis. He has a deep understanding of applications for data analysis in domains such as investment banking, online payments, online advertising, IT infrastructure, and retail. His area of expertise is in applying high-performance computing in distributed and data-driven environments such as real-time analysis, high-frequency trading, and so on. 

He earned his engineering degree in electronics and instrumentation from Jadavpur University and his master’s degree in research in computational and data science from IISc in Bangalore.
 
Pratip Samanta

A photograph of Pratip Samanta.

is a principal AI engineer/researcher with more than 11 years of experience. He has worked for several software companies and research institutions. He has published conference papers and has been granted patents in AI and natural language processing. He is also passionate about gardening and teaching.
 
About the Technical Reviewer
Joos Korstanje

A photograph of Joos Korstanje.

is a data scientist with more than five years of industry experience in developing machine learning tools, of which a large part is forecasting models. He currently works at Disneyland Paris where he develops machine learning for a variety of tools.
 
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.117.234.225