index

A

A/B testing 316 – 333

data collection 317 – 319

evaluating categorical metrics 329 – 333

evaluating continuous metrics 319 – 323

using alternative displays and tests 325 – 329

what not to do 319

acceptance testing 438 – 470

data consistency 439 – 446

dangers of data silo 445 – 446

feature stores 441 – 442

process over technology 442 – 445

training and inference skew 440 – 441

end user vs. internal use testing 453 – 460

biased testing 456 – 457

dogfooding 457 – 458

SME evaluation 459 – 460

fallbacks and cold starts 447 – 452

cold-start woes 450 – 452

leaning heavily on prior art 448 – 450

model interpretability 460 – 469

Shapley additive explanations 461 – 463

using shap 463 – 466, 469

ACID-compliant storage layer 442

active retraining 352

Agile software engineering 31 – 35

communication and cooperation 33 – 35

embracing and expecting change 35

algorithmic complexity 536 – 539

alignment 408 – 410

ALS (alternating least squares) 44

Anaconda Navigator 542

analysis paralysis 53

API documentation 130 – 135

approximate Shapley value estimation 461 – 463

architecture, code 276 – 278

ARIMA, rapid testing for 184 – 186

artifact management 472 – 481

interfacing with model registry 476 – 481

MLflow model registry 474 – 475

asynchronous concurrency 239 – 241

attribution measurement 302 – 316

clarifying correlation vs. causation 312 – 316

leveraging A/B testing for calculations 316 – 333

data collection 317 – 319

evaluating categorical metrics 329 – 333

evaluating continuous metrics 319 – 323

using alternative displays and tests 325 – 329

what not to do 319

prediction performance 302 – 310

autocorrelation 154

autoML (automated-ML) 205

autoregressive parameters (p) variable 184

availability, data 403 – 404

B

ball of mud 273

baseline comparison visualization 164 – 167

Becoming Agile in an Imperfect World (Smith and Sidky) 113

BI (business intelligence) style queries 194

biased testing 456 – 457

Big O 510 – 539

algorithmic complexity for ML 536 – 539

analyzing decision-tree complexity 531 – 536

complexity 519 – 529

O(1) 519 – 521

O(n) 521 – 523

O(n2) 524 – 529

overview 515 – 516

overview 510 – 516

Bird, Steven 514

black swan events 346

blind catch 283 – 284

branch strategies, logging 234 – 236

bulk external delivery 500 – 502

delivery consistency 500 – 501

quality assurance 502

burst volume 504 – 507

business knowledge 50

business rules chaos 116 – 120

backup plan 119 – 120

planning for 117 – 119

C

cargo cult ML (machine learning) behavior 432 – 437

categorical metrics 329 – 333

causation 312 – 316

CDD (chaos-driven development) 115

CI/CD (continuous integration/continuous deployment) system 194

citizen data scientist 206

classification problems 502

clean experimentation environment 540 – 541

cleansing data 410 – 412

CNNs (convolutional neural networks) 14

code and coding 269 – 299

code architecture 276 – 278

code smells 270 – 272

designing modular ML 257 – 264

efficient code 274 – 275

exception eating 282 – 288

exception handling 285 – 286

handling errors right way 286 – 288

try/catch block 283 – 284

excessively nested logic 292 – 297

global mutable objects 288 – 291

encapsulation to prevent mutable side effects 290 – 291

mutability 288 – 290

naming conventions and structure 273 – 274

production code 399 – 437

avoiding cargo cult ML (machine learning) behavior 432 – 437

guiding principles 401 – 412

monitoring everything in model life cycle 417 – 421

monitoring features 412 – 416

simplicity 421 – 426

wireframing ML projects 426 – 431

setting guidelines in 163 – 172

baseline comparison visualization 164 – 167

standard metrics 167 – 172

tuple unpacking 278 – 282

alternative to 280 – 282

example of 278 – 280

code smells 270 – 272

cold starts 447 – 452

collaborative involvement 33

collections, polynomial relationship and 524 – 529

communication 33 – 35, 76 – 123, 163

business rules chaos 116 – 120

backup plan 119 – 120

planning for 117 – 119

defining problem 79 – 100

ideal implementation 86 – 88

project-based meetings 89 – 93

setting critical discussion boundaries 94 – 100

what will it to do 81 – 86

working with SMEs (subject-matter experts) 89

explaining results 120 – 122

meeting with cross-functional teams 101 – 108

development progress reviews 105

experimental update meeting 102 – 103

MVP review 106 – 107

preproduction review 107 – 108

SMEs (subject-matter experts) review/prototype review 103 – 104

setting limits on experimentation 108 – 116

CDD (chaos-driven development) 115

maintainability and extensibility 112 – 113

PDD (prayer-driven development) 114 – 115

RDD (resume-driven development) 115 – 116

TDD (test-driven development) or FDD (feature-driven development) 113

time limit 109 – 110

complexity 519 – 529

assessing risk 66

elegant complexity 355 – 364

lightweight scripted style 357 – 361

overengineering vs. 361 – 364

O(1) 519 – 521

O(n) 521 – 523

O(n²) 524 – 529

overview 515 – 516

concept drift 341 – 343

concurrency

asynchronous concurrency 239 – 241

scalability and 239

Conda environment manager 542

constructIndexers() method 363

containers

creating container-based pristine environment for experimentation 543 – 544

for dependency hell 542

continuous integration/continuous deployment (CI/CD) system 194

continuous metrics 319 – 323

control code 510

cooperation 33 – 35

correlation 312 – 316

cost, serving needs 494

cowboy development 115

cross-functional teams 101 – 108

development progress reviews 105

experimental update meeting 102 – 103

MVP review 106 – 107

preproduction review 107 – 108

SMEs review/prototype review 103 – 104

D

d (differences) variable 184

data

analysis 139 – 146

cleanliness 143 – 146

collection for A/B testing 317 – 319

consistency 439 – 446

dangers of data silo 445 – 446

feature stores 441 – 442

process over technology 442 – 445

training and inference skew 440 – 441

guiding principles for production code 401 – 412

alignment 408 – 410

checking data provenance 404 – 408

data availability 403 – 404

embedding data cleansing 410 – 412

quality 50 – 52

database, serving from 498

DataFrame functions module 359

DataFrame object 372

data science 26 – 37

co-opting principles of Agile software engineering 31 – 35

communication and cooperation 33 – 35

embracing and expecting change 35

foundation of ML (machine learning) engineering 35 – 37

foundation of simplicity 29

increasing project success 27 – 29

Data Science, Classification, and Related Methods (Hayashi) 27

data silo 445 – 446

data warehouse, serving from 498

debugging walls of text 255 – 257

decision-trees, complexity 531 – 536

delivery consistency 500 – 501

demos, planning for 56 – 57

dependency hell 540, 542

deployment 18 – 21

detecting drift 335 – 347

concept drift 341 – 343

feature drift 337 – 339

feedback drift and law of diminishing returns 346 – 347

label drift 339 – 341

prediction drift 343 – 345

reality drift 346

development 15 – 18

progress reviews 105

setting up environment 540 – 544

case for clean experimentation environment 540 – 541

containers to deal with dependency hell 542

creating container-based pristine environment for experimentation 543 – 544

sprint reviews 98

DevOps (development operations) 31

diminishing returns 346 – 347

discussion boundaries 94 – 100

development sprint reviews 98

MVP review 98 – 99

post-experimentation phase 96 – 98

post-research phase discussion 95 – 96

preproduction review 100

displays 325 – 329

Docker 542

docker pull continuumio/anaconda3 command 544

dogfooding 457 – 458

drift 334 – 352

detecting 335 – 347

concept drift 341 – 343

feature drift 337 – 339

feedback drift and law of diminishing returns 346 – 347

label drift 339 – 341

prediction drift 343 – 345

reality drift 346

responding to 347 – 352

drivers, handling tuning with SparkTrials 218 – 222

E

edge deployment 507 – 508

efficient code 274 – 275

elegant complexity 355 – 364

lightweight scripted style (imperative) 357 – 361

overengineering vs. 361 – 364

elif statements 292, 294

else statements 292, 294

encapsulation 290 – 291

end user testing 453 – 460

biased testing 456 – 457

dogfooding 457 – 458

SME evaluation 459 – 460

ER (entity-relationship) diagrams 409

errors 286 – 288

estimating amount of work 73 – 74

evaluation 21 – 22

exception eating 282 – 288

exception handling 285 – 286

handling errors right way 286 – 288

try/catch block 283 – 284

experimental scoping 60 – 74

experimentation 64 – 74

assessing complexity risk 66

estimating amount of work 73 – 74

scoping research phase, importance of 68 – 73

tracking phases 66 – 67

overview 61 – 62

research 62 – 64

experimental update meeting 102 – 103

experimentation 13 – 15, 64 – 74, 124 – 241

assessing complexity risk 66

choosing tech for platform and team 215 – 227

handling tuning from driver with SparkTrials 218 – 222

handling tuning from workers with pandas_udf 222 – 226

Spark 216 – 217

using new paradigms for teams 226 – 227

estimating amount of work 73 – 74

limitations on 108 – 116

CDD (chaos-driven development) 115

maintainability and extensibility 112 – 113

PDD (prayer-driven development) 114 – 115

RDD (resume-driven development) 115 – 116

TDD (test-driven development) or FDD (feature-driven development) 113

time limit 109 – 110

logging 229 – 236

MLflow tracking 230 – 232

printing and 232 – 234

version control, branch strategies, and working with others 234 – 236

planning 126 – 137

assigning testing 135 – 136

collecting metrics 136 – 137

reading API documentation 130 – 135

researching 126 – 130

possibilities, whittling down 190 – 196

evaluating prototypes properly 191 – 193

questions in planning session 193 – 196

preparation 137 – 156

moving from script to reusable code 146 – 153

performing data analysis 139 – 146

scalability 237 – 241

asynchronous concurrency 239 – 241

concurrency 239

scoping research phase 68 – 73

testing ideas 162 – 190

running quick forecasting tests 172 – 190

setting guidelines in code 163 – 172

tracking phases 66 – 67

tuning 199 – 214

Hyperopt primer 206 – 208

options 201 – 206

using Hyperopt to tune complex forecasting problem 208 – 214

explainable artificial intelligence (XAI) 460

ExponentialSmoothing() class 211

extensibility 112 – 113

F

fallbacks 447 – 452

cold-start woes 450 – 452

leaning heavily on prior art 448 – 450

FDD (feature-driven development) 113

feature drift 337 – 339

feature ignorance 338

features, monitoring 412 – 416

feature stores 441 – 442, 482 – 489

evaluating 489

reasons for 483 – 485

using 485 – 489

feedback drift 346 – 347

fit() method 178 – 179, 211

foldLeft operation 378

forecasting tests 172 – 190

creating validation dataset 173 – 174

rapid testing for ARIMA 184 – 186

rapid testing of Holt-Winters exponential smoothing algorithm 186 – 190

rapid testing of VAR model approach 175 – 182

FP-growth (frequent-pattern-growth) market-basket analysis algorithms 95

frameworks, generalization and 379 – 381

functionality 52

functions, benefits of 153

G

GANs (generative adversarial networks) 15

GDPR (General Data Protection Regulation) 407

generalization 379 – 381

_generate_boundaries() method 265

generate_hyperopt_report() function 213

generate_log_map_and_plot() function 279

global mutable objects 288 – 291

encapsulation to prevent mutable side effects 290 – 291

mutability 288 – 290

grid search 202 – 203

H

hacker mentality 368 – 370

hacking (cowboy development) 115

Hayashi, C. 27

Heisenbugs 288

high volume 504 – 507

HIPAA (Health Insurance Portability and Accountability Act) 407

Holt-Winters exponential smoothing algorithm 186 – 190

HSD (honestly significant difference) tests 327

HWES (Holt-Winters Exponential Smoothing) model 208

Hyperopt

overview 206 – 208

TPEs (tree-structured Parzen estimators) 204 – 205

tuning complex forecasting problem 208 – 214

Hyperopt Trials() object 217

hypothesis testing 308

I

IDE (integrated development environment) 17

if statements 292, 294

imperative scripted style 357 – 361

implementation, simplicity in 424 – 426

import statements 463

imposter syndrome 368

inference skew 440 – 441

integrated models 507 – 508

internal use

prediction serving architecture 497 – 499

testing 453 – 460

biased testing 456 – 457

dogfooding 457 – 458

SME evaluation 459 – 460

interpretability 460 – 469

Shapley additive explanations 461 – 463

approximate Shapley value estimation 461 – 463

foundation 461

how to use values from 463

using shap 463 – 466, 469

shap summary plot 466 – 467

waterfall plots 467 – 469

J

JIT (just-in-time) 263

K

Klein, Ewan 514

knowledge, curse 52 – 53

Koskela, Lasse 113

L

label drift 339 – 341

lightweight scripted style 357 – 361

linear relationship algorithm 521 – 523

logging 229 – 236

MLflow tracking 230 – 232

printing and 232 – 234

version control, branch strategies, and working with others 234 – 236

log statements 233

Loper, Edward 514

low volume 504

LSTM (long short-term memory) 144

M

mad scientist developers 375 – 377

maintainability 112 – 113

Mandelbugs 288

manual tuning 201 – 202

Map object 375

Map type 375

maxlags parameter 179

metrics

categorical metrics 329 – 333

collecting 136 – 137

continuous metrics 319 – 323

scoring 308 – 310

microbatch streaming 502 – 503

microservice framework 498 – 499

ML (machine learning)

algorithmic complexity for 536 – 539

code smells 270 – 272

development 353 – 395

dangers of open source 390 – 392

elegant complexity 355 – 364

generalization and frameworks 379 – 381

optimizing too early 382 – 390

technology-driven development vs. solution-driven development 393 – 395

unintentional obfuscation 364 – 379

ML (machine learning) engineering 3 – 25

core tenets of 8 – 22

deployment 18 – 21

development 15 – 18

evaluation 21 – 22

experimentation 13 – 15

planning 8 – 10

scoping and research 10 – 12

data science and foundation of 35 – 37

goals of 24

reasons for 5 – 8

MLflow

model registry

artifact management 474 – 475

interfacing with 476 – 481

tracking 230 – 232

model life cycle 417 – 421

model measurement 300 – 333

leveraging A/B testing for attribution calculations 316 – 333

data collection 317 – 319

evaluating categorical metrics 329 – 333

evaluating continuous metrics 319 – 323

using alternative displays and tests 325 – 329

what not to do 319

measuring model attribution 302 – 316

clarifying correlation vs. causation 312 – 316

prediction performance 302 – 310

modularity for ML 245 – 268

debugging walls of text 255 – 257

designing modular ML code 257 – 264

monolithic scripts 248 – 255

considerations for 252 – 255

history of 249

walls of text 249 – 252

using test-driven development for ML 264 – 267

modulo function 521

monitoring

everything in model life cycle 417 – 421

features 412 – 416

monolithic scripts 248 – 255

considerations for 252 – 255

history of 249

walls of text 249 – 252

moving average (q) variable 184

mutable objects, global 288 – 291

encapsulation to prevent mutable side effects 290 – 291

mutability 288 – 290

MVP review 98 – 99, 106 – 107

mystic developers 370 – 372

N

naming conventions 273 – 274

Natural Language Processing with Python (Bird, Klein, and Loper) 514

NDCG (non-discounted cumulative gain) metrics 45

nested logic 292 – 297

NLTK package 513

nonstationary time series 141

novel algorithm 115

O

O(1) complexity 519 – 521

O(n) complexity 521 – 523

O(n²) complexity 524 – 529

obfuscation 364 – 379

hacker mentality 368 – 370

mad scientist developers 375 – 377

mystic developers 370 – 372

safer bet approach 377 – 378

show-off type 373 – 375

troublesome coding habits 378 – 379

objective_function function 208

OLTP (online transaction processing) storage layer 441

open source 390 – 392

optimizing 382 – 390

overengineering 361 – 364

P

p (autoregressive parameters) variable 184

pandas_udf 222 – 226

paradigms 226 – 227

parallelism 239

partial autocorrelation test 154

passive retraining 352

PDD (prayer-driven development) 114 – 115

pdf (probability density function) 341, 345

personalization 47

phases, experimentation 66 – 67

PII (personally identifiable information) 407

planning 8 – 10, 38 – 60, 126 – 137

assigning testing 135 – 136

basic 47 – 53

analysis paralysis 53

assumption of business knowledge 50

assumption of data quality 50 – 52

assumption of functionality 52

knowledge, curse of 52 – 53

collecting metrics 136 – 137

experimentation by solution building 58 – 60

first meeting 53 – 56

for demos 56 – 57

reading API documentation 130 – 135

researching 126 – 130

phase of 129 – 130

quick visualization of dataset 127 – 129

session questions 193 – 196

data requirements 194

development cadence 195 – 196

existing code used for project 195

getting predictions to end users 195

inference running location 195

running frequency 193

running location for training 194

setting up code base 194

storing forecasts 194

plot_predictions() function 213

pmf (probability mass function) 341, 345

PoC (proof of concept) 70

polynomial relationship 524 – 529

post-experimentation phase 96 – 98

post-research phase discussion 95 – 96

prayer-driven development (PDD) 114 – 115

prediction drift 343 – 345

prediction performance 302, 308 – 310

prediction serving architecture 490 – 508

bulk external delivery 500 – 502

delivery consistency 500 – 501

quality assurance 502

determining serving needs 493 – 497

cost 494

recency 494 – 497

SLA 494

integrated models 507 – 508

internal use cases 497 – 499

serving from database or data warehouse 498

serving from microservice framework 498 – 499

microbatch streaming 502 – 503

real-time server-side 503 – 507

burst volume and high volume 504 – 507

low volume 504

preparation 137 – 156

moving from script to reusable code 146 – 153

functions, benefits of 153

importance of 154 – 156

performing data analysis 139 – 146

preproduction review 100, 107 – 108

printing, logging and 232 – 234

print statements 170, 232 – 233, 256, 416

prior art 448 – 450

probability density function (pdf) 341, 345

process over technology 442 – 445

production

infrastructure 471 – 509

artifact management 472 – 481

feature stores 482 – 489

prediction serving architecture 490 – 508

writing code 399 – 437

avoiding cargo cult ML (machine learning) behavior 432 – 437

guiding principles 401 – 412

monitoring everything in model life cycle 417 – 421

monitoring features 412 – 416

simplicity 421 – 426

wireframing ML (machine learning) projects 426 – 431

project-based meetings 89 – 93

project success 27 – 29

prototype culling 60

provenance of data 404 – 408

Q

q (moving average) variable 184

quadratic() method 523

quality assurance 502

quality testing 438 – 470

data consistency 439 – 446

dangers of data silo 445 – 446

feature stores 441 – 442

process over technology 442 – 445

training and inference skew 440 – 441

end user vs. internal use testing 453 – 460

biased testing 456 – 457

dogfooding 457 – 458

SME evaluation 459 – 460

fallbacks and cold starts 447 – 452

cold-start woes 450 – 452

leaning heavily on prior art 448 – 450

model interpretability 460 – 469

Shapley additive explanations 461 – 463

using shap 463 – 466, 469

R

random search 203 – 204

rapid testing

for ARIMA 184 – 186

of Holt-Winters exponential smoothing algorithm 186 – 190

of VAR model approach 175 – 182

RDBMS (relational database management system) 137

RDD (resiliently distributed dataset) 223, 359

RDD (resume-driven development) 115 – 116

reality drift 346

real-time server-side 503 – 507

burst volume and high volume 504 – 507

low volume 504

recency 494 – 497

recurrent neural networks (RNNs) 144

regression problems 502

remove_bias value 210

REPL (read-eval-print loop) 128

researching 10 – 12, 126 – 130

experimental scoping 62 – 64

phase of 129 – 130

scoping phase, importance of 68 – 73

visualization of dataset 127 – 129

responding to drift 347 – 352

results, explaining 120 – 122

returns, diminishing 346 – 347

reusable code 146 – 153

functions, benefits of 153

importance of 154 – 156

rm -rf command 386

RMSE (root mean squared error) 44

RNNs (recurrent neural networks) 144

ROI (return on investment) 473

runtime performance 510 – 539

algorithmic complexity for ML (machine learning) 536 – 539

analyzing decision-tree complexity 531 – 536

Big O 510 – 516

complexity 519 – 529

O(1) 519 – 521

O(n) 521 – 523

O(n²) 524 – 529

overview 515 – 516

run_tuning() function 228

S

safer bet approach 377 – 378

scalability 237 – 241

asynchronous concurrency 239 – 241

concurrency 239

scoping 10 – 12

serving architecture, prediction 490 – 508

bulk external delivery 500 – 502

delivery consistency 500 – 501

quality assurance 502

determining serving needs 493 – 497

cost 494

recency 494 – 497

SLA 494

integrated models (edge deployment) 507 – 508

internal use cases 497 – 499

serving from database or data warehouse 498

serving from microservice framework 498 – 499

microbatch streaming 502 – 503

real-time server-side 503 – 507

burst volume and high volume 504 – 507

low volume 504

SGD (stochastic gradient descent) 536

shap 463 – 466, 469

shap summary plot 466 – 467

waterfall plots 467 – 469

Shapley additive explanations 461 – 463

approximate Shapley value estimation 461 – 463

foundation 461

how to use values from 463

shap package 461, 463, 466, 468 – 469, 475, 480

show-off type 373 – 375

Sidky, Ahmed 113

simplicity 421 – 426

in implementation 424 – 426

in problem definitions 423 – 424

simplicity, foundation of 29

singular value decomposition (SVD) model 58

SLA, determining serving needs 494

SMEs (subject-matter experts)

evaluation 459 – 460

review 103 – 104

working with 89

Smith,Greg 113

smoothing_level value 210

smoothing_seasonal value 210

solution building 58 – 60

solution-driven development 393 – 395

space complexity 515

spaghetti code 273

Spark 215 – 227

handling tuning from driver with SparkTrials 218 – 222

handling tuning from workers with pandas_udf 222 – 226

reasons for 216 – 217

using new paradigms for teams 226 – 227

SPC (statistical process control) rules 345

standardization 163

standard metrics 167 – 172

statement 386

String values 370

structure, coding 273 – 274

summary plots, shap 466 – 467

SVD (singular value decomposition) model 58

T

TDD (test-driven development) 113, 264 – 267

technology, process over 442 – 445

technology-driven development 393 – 395

Test Driven (Koskela) 113

testing ideas 162 – 190

assigning 135 – 136

running quick forecasting tests 172 – 190

creating validation dataset 173 – 174

rapid testing for ARIMA 184 – 186

rapid testing of Holt-Winters exponential smoothing algorithm 186 – 190

rapid testing of VAR model approach 175 – 182

setting guidelines in code 163 – 172

baseline comparison visualization 164 – 167

standard metrics 167 – 172

test statistic 141

time limit 109 – 110

TPEs (tree-structured Parzen estimators) 204 – 205

training, inference skew and 440 – 441

Trials() mode 225

Trials object 207, 220

trigger-once operation 494

try/catch block 283 – 284

tuning 199 – 214

handling from driver with SparkTrials 218 – 222

handling from workers with pandas_udf 222 – 226

Hyperopt primer 206 – 208

options 201 – 206

advanced techniques 205 – 206

grid search 202 – 203

manual tuning 201 – 202

random search 203 – 204

TPEs (tree-structured Parzen estimators) 204 – 205

using Hyperopt to tune complex forecasting problem 208 – 214

tuple unpacking 278 – 282

alternative to 280 – 282

example of 278 – 280

type 386

U

unsupervised problems 502

use_basin_hopping value 210

use_boxcox value 210

use_brute value 210

V

validation dataset 173 – 174

VAR model approach 175 – 182

VectorAssembler constructor 360

version control 234 – 236

visualization of dataset 127 – 129

VM (virtual machine) container 215

W

walls of text

debugging 255 – 257

monolithic scripts 249 – 252

waterfall plots, shap 467 – 469

wireframing ML projects 426 – 431

workers, handling tuning with pandas_udf 222 – 226

WoT (walls of text) 250

X

XAI (explainable artificial intelligence) 460