index

Numerics

1 x 1 convolutions 176 – 179

A

a(...) function 26

activation function 96

adam optimizer 54, 58, 88, 113, 183, 229

Albert 643 – 644

Anaconda 615 – 616, 618 – 619

APIs (application programming interfaces) 599 – 612

evaluating model 602 – 608

predicting with TensorFlow serving API 608 – 612

pushing final model 608

resolving correct model 602

TFX (TensorFlow-Extended) Trainer API 576 – 596

defining Keras model 577 – 584

defining model training 584 – 586

SignatureDefs 586 – 589

training Keras model with 589 – 596

validating infrastructure 600 – 601

apply() function 160, 307

apply_buckets() function 572

arr variable 31

ASICs (application-specific integrated circuits) 10

ASPP (atrous spatial pyramid pooling) module 273 – 275

assign() operation 31

atrous convolution 269 – 270

attention 435 – 445

defining final model 440 – 443

implementing Bahdanau attention in TensorFlow 436 – 440

training model 443 – 445

visualizing 445 – 451

Attention layer 436

attention_visualizer() function 446

augmenting data 244

autoencoder model 85 – 90

autoregressive model 639

auxiliary output layers 179 – 181

auxiliary outputs 244

averaged_perceptron_tagger 303

B

Bahdanau attention in TensorFlow 436 – 440

BART (bidirectional and auto-regressive Transformers) 640 – 642

batch normalization 213

beam search 379 – 383

beam_search function 381

bert.bert_models.classifier_model() function 480

BERT (bidirectional encoder representations from Transformers) 350, 639

DistilBERT model 495 – 502

question answering with 505 – 508

spam classification using 463 – 483

in TensorFlow 470 – 483

overview 465 – 469

bert_viz library 507

bfloat16 special data type 10

bidirectional and auto-regressive Transformers (BART) 640 – 642

bilinear interpolation 258

binning 322

black boxes 238

BLEUMetric object 419

BN (Batch normalization) 213

broadcasting 37

bucketing 322

bucketized features 569

build() function 62 – 63, 436

C

call() function 62, 64, 136, 333, 436

categorical_columns variable 596

categorical_crossentropy loss 54, 58, 335

categorical features 569

CBOW (continuous bag-of-words) 457

central processing unit (CPU) 9 – 10

ce_weighted_from_logits() outer function 280

class imbalance 301, 471 – 474

cleaning data 4

CLI (command line interface) 619

Cloze task 351

CNNs (convolutional neural networks) 14, 90 – 105, 194 – 242, 522

data 90 – 92

Grad-CAM (gradient class activation map) 238 – 240

image classification with 149 – 193

creating data pipelines using Keras ImageData-Generator 160 – 165

exploratory data analysis 150 – 160

Inception net 165 – 188

training model and evaluating performance 149 – 192

implementing network 92 – 105

Minception 210 – 231

reducing overfitting 195 – 210

dropout 203 – 207

early stopping 207 – 210

image data augmentation with Keras 196 – 203

using pretrained networks 232 – 238

computer graphic computations 19

computer vision 625 – 638

Grad-CAM (gradient class activation map) 625 – 631

image segmentation 632 – 638

defining U-Net model 632 – 633

pretrained encoder 634 – 638

Concatenate layer 58, 275

conda environment 615 – 623

context vector 401

continuous bag-of-words (CBOW) 457

Conv2D layer 96 – 98, 101, 103 – 104, 150

convolution operation 41 – 43

cooking competition analogy 135 – 136

covariate shift 213

CPU (central processing unit) 9 – 10

Cropping2D Keras layer 637

CsvExampleGen component 573

CSVLogger callback 208 – 209, 586

CUDA 616 – 618, 620 – 622

CUDNN 618, 622

D

DAG (directed acyclic graph) 25

data leakage 309

dataset.map() function 536

datasets library 485

datetime library 514

declarative graph – based execution 27

decoder 405 – 409

Decoder layer 141, 145

DecoderRNNAttentionWrapper layer 440

decorators 25

DeepLabv3 266 – 277

atrous convolution 269 – 270

implementing ASPP module 273 – 275

implementing Deeplab v3 using Keras functional API 270 – 273

ResNet-50 model 268

deep learning 80 – 117, 119 – 146

CNNs (convolutional neural networks) 90 – 105

FCNs (fully connected networks) 81 – 90

generating text with 365 – 370

prototyping deep learning models 10

representing text as numbers 120 – 122

RNNs (recurrent neural networks) 105 – 117

TensorFlow and 14

transformers 123 – 146

DeepLearning4J framework 5

denoising 88

DenseFeatures layer 582

Dense layer 53, 58, 87, 96, 103 – 104, 111 – 113, 145, 150, 179, 181, 218, 331, 334, 336, 369, 397, 522

describe() function 158, 315

dilation rate 269

dimensionality reduction method 176 – 179

directed acyclic graph (DAG) 25

DistilBERT 495 – 502, 640

DistilBertTokenizerFast.from_pretrained() function 488 – 489

Docker 596 – 599

dropout 203 – 207

Dropout layer 331, 334

E

eagerly executing code 25

EagerTensor class 33

early stopping 207 – 210, 244

EarlyStopping callback 207 – 209

EDA (exploratory data analysis) 150, 564 – 567

Elman networks 113

embedding layer 144, 643

embeddings 457 – 460

encoder 401 – 405

encoder-decoder pattern 123 – 124

English-German seq2seq machine translator 395 – 410

compiling model 409 – 410

define TextVectorization layers for seq2seq model 400 – 401

defining decoder and final model 405 – 409

defining encoder 401 – 405

TextVectorization layer 398 – 400

entropy 370

environments 615 – 623

activating and deactivating conda environment 622 – 623

running Jupyter Notebook server and creating notebooks 623

Unix-based environment 615 – 618

creating virtual Python environment with Anaconda distribution (Ubuntu) 615 – 616

prerequisites for GPU support (Ubuntu) 616 – 618

Windows Environments 618 – 622

creating Virtual Python Environment (Anaconda) 618 – 619

prerequisites for GPU support 619 – 622

evaluate() function 483

evaluating model

image segmentation 290 – 293

image segmentation metrics 284 – 289

language modeling 372 – 374

sequence-to-sequence learning 410 – 423

TFX (TensorFlow-Extended) model 602 – 608

word vectors 344 – 346

ExampleValidator object 595

exploratory data analysis 150 – 160, 564 – 567

F

factorization of embedding layer 643

FCNs (fully connected networks) 81 – 90

autoencoder model 85 – 90

feature engineering 4

feature map 93

featurewise_center parameter 197

featurewise_std_normalization parameter 197

fill_mode parameter 199

fit() function 55, 398, 523

fit_on_texts() function 318

fit_resample() function 471

fix_random_seed function 56, 88

flat_map() function 360

Flatten() layer 104, 181, 229, 522

float16 type 539

float32 type 539

flow() method 161

flow_from_dataframe() function 73, 161, 164

flow_from_directory() function 161, 163 – 164

flow_from_directory() method 161 – 162

FnArgs object 584 – 585

forget gate 366

forward(...) function 25

from_pretrained() function 495

fully connected layer 139 – 141, 456

fully connected networks 20

functional API 56 – 61, 65

Functional layer 401

G

Gamma correction 202

gated recurrent units (GRUs) 10, 325, 350, 365 – 370, 396

generator-iterator 162

get_bert_inputs() function 477

glorot_uniform initializer 64

GLUE (General Language Understanding Evaluation) 468

GPT (generative pre-training) model 639 – 640

GPU (graphical processing unit) 9 – 10

Grad-CAM (gradient class activation map) 238 – 240, 625 – 631

greedy decoding 374 – 379

GRUCell object 441

GRUs (gated recurrent units) 10, 325, 350, 365 – 370, 396

H

hashing, locality sensitive 644 – 645

head() operation 107

Holonyms relationship 306

Hugging Face Transformers 483 – 509

ask BERT question 505 – 509

defining DistilBERT model 495 – 502

processing data 487 – 494

defining and using tokenizer 488 – 493

from tokens to tf.data pipeline 493 – 494

training model 502 – 505

Hypernyms relationship 306

Hyponyms relationship 306

I

i.i.d (independent and identically distributed) 105

ILSVRC (ImageNet Large Scale Visual Recognition Challenge) 151

image classification 149 – 193

creating data pipelines using Keras ImageData-Generator 160 – 165

exploratory data analysis 150 – 160

classes in data set 155 – 158

computing simple statistics on data set 158 – 160

Inception net 165 – 188

Inception-ResNet v1 and Inception-ReSNet v2 187

Inception v1 169 – 181, 183 – 184

Inception v2 184 – 186

Inception v3 187

Inception v4 187

training model and evaluating performance 149 – 192

image data augmentation 196 – 203

image segmentation 243 – 295

data 245 – 251

DeepLabv3 266 – 277

atrous convolution 269 – 270

implementing ASPP module 273 – 275

implementing Deeplab v3 using Keras functional API 270 – 273

ResNet-50 model 268

evaluating model 290 – 293

evaluation metrics 284 – 289

loss functions 277 – 284

cross-entropy loss 278 – 282

dice loss 280 – 283

TensorFlow data pipeline 251 – 265

final tf.data pipeline 264 – 265

optimizing tf.data pipelines 263 – 264

training model 289 – 290

U-Net model 632 – 638

defining 632 – 633

pretrained encoder 634 – 638

imbalanced-learn library 471

immutable data structure 34

imperative style execution 25

imshow() function 449

Inception blocks 244

connection between sparsity and 175 – 176

overview 173 – 175

Inception net 165 – 188

Inception-ResNet v1 and Inception-ReSNet v2 187

Inception v1 169 – 181, 183 – 184

1 x 1 convolutions as dimensionality reduction method 176 – 179

auxiliary output layers 179 – 181

connection between Inception block and sparsity 175 – 176

Inception block 173 – 175

Inception v2 184 – 186

Inception v3 187

Inception v4 187

Inception-ResNet type A block 216 – 223

Inception-ResNet type B block 223 – 225

inheritance 62

input gate 366

Input layer 57 – 58, 582

input_shape parameter 53, 98, 117

input_shape (Sequential API) 70

inputs tensor 439

instance segmentation 245

int32 type 494

iteration 189

J

Jupyter Notebook 623

K

keepdims parameter 37

Keras

image data augmentation with 196 – 203

model-building APIs 48 – 65

data set 49 – 51

functional API 56 – 61, 270 – 273

Sequential API 52 – 55

sub-classing API 61 – 65

Keras DataGenerators 72 – 74

Keras ImageDataGenerator 160 – 165

Keras Model object 400

kernel size 99, 167

knowledge distillation 640

K.rnn() function 439

L

lambda function 324

lambda layer 171

language modeling 297, 349 – 384

beam search 379 – 383

greedy decoding 374 – 379

GRUs (gated recurrent units) 365 – 370

measuring quality of generated text 370 – 372

overview 351

processing data 350 – 365

defining tf.data pipeline 360 – 365

downloading 351 – 356

n-grams 356 – 358

tokenizing text 358 – 359

training and evaluating language model 372 – 374

Layer base class 62 – 63

layer objects 7, 57, 62

lemmatization 297, 305

load_data() method 83

logits 280

loss functions 277 – 284

cross-entropy loss 278 – 282

dice loss 280 – 283

lower() function 303

lr_callback callback 290

LRN (local response normalization) 171 – 172, 213, 238

LSH (locality sensitive hashing) 644

LSTM (long short-term memory) 10, 326 – 331, 349

M

machine learning models 8, 12

machine translation 297

machine translation data 388 – 395

MacOS 618

map() function 68, 255, 538

Markov property 351

masked language modeling 351, 466

masked self-attention layers 136 – 138, 456

masking layer 331

matplotlib library 448, 631

matrix multiplication 39 – 41

MaxPool2D layer 103 – 104, 150

Mean IoU (mean intersection over union) 287

meronyms 306

MetricsSpec object 603

Minception 210 – 231

training 229 – 231

mixed precision training 539 – 544

MLM (masked language modeling) 351, 466

MLP (multilayer perceptron) 20 – 21, 85

model compilation 54

model debugging 5

model.fit() function 190, 586

model.metric_names attribute 208

Model object 58, 143

ModelOutput object 497

model.predict() function 293, 399

model serving 5

MSE (mean squared error) 113

MulBiasDense custom layer 63

multi-head attention 138 – 139

MXNet framework 5

N

NCE (Noise contrastive estimation) loss 355

nearest interpolation 258

negative dimension 102

NER (Named entity recognition) 297

neural network-related computations 39 – 45

convolution operation 41 – 43

matrix multiplication 39 – 41

pooling operation 43 – 45

neural networks with TFX Trainer API 576 – 596

defining Keras model 577 – 584

defining model training 584 – 586

SignatureDefs 586 – 589

training Keras model with TFX Trainer 589 – 596

next sentence prediction 466, 644

n-grams 356 – 358

NLP (natural language processing) 120, 296 – 348, 639 – 645

defining end-to-end NLP pipeline with TensorFlow 319 – 325

language modeling 349 – 384

beam search 379 – 383

greedy decoding 374 – 379

GRUs (gated recurrent units) 365 – 370

measuring quality of generated text 370 – 372

processing data 350 – 365

training and evaluating 372 – 374

sentiment analysis 325 – 336

defining final model 331 – 336

LSTM networks 326 – 331

text 308 – 319

analyzing sequence length 315 – 316

analyzing vocabulary 313 – 315

splitting training, validation and testing data 309 – 313

text to words and then to numbers with Keras 316 – 319

training and evaluating model 336 – 339

Transformer models 639 – 645

Albert 643 – 644

BART 640 – 642

DistilBERT 640

GPT model 639 – 640

Reformer 644 – 645

RoBERT and ToBERT 640

XLNet 642 – 643

word vectors 339 – 346

defining final model with word embeddings 341 – 344

training and evaluating model 344 – 346

word embeddings 340 – 341

nlp.optimization.create_optimizer() function 481

NLTK (Natural Language Toolkit) 302

Noise contrastive estimation (NCE) loss 355

normalization 460 – 463

np.random.normal() function 131

np.random.permutation() function 412

NSP (next-sentence prediction) 466, 644

Number of filters parameter 166

num_parallel_calls 536

NVIDIA driver 616, 620

O

omw-1.4 external resource 303

one-hot encoding 51

optimized hardware 10 – 11

optimizing input pipeline 536 – 538

output gate 366

overfitting, reducing 195 – 210

dropout 203 – 207

early stopping 207 – 210

image data augmentation with Keras 196 – 203

P

Padding 167

palettized images 249

ParseFromString() function 569

partial() function 163, 493

Part of speech (PoS) tagging 297

PCA (Principal Component Analysis) 56, 548

pd.DataFrame 155

pd.DataFrame.from_records() function 159

pd.read_csv() function 545

pd.read_json() function 300

pd.Series.apply() function 156

pd.Series object 155, 317

pd.Series.str.len() function 315

performance bottlenecks 104, 529 – 544

mixed precision training 539 – 544

optimizing input pipeline 536 – 538

performance monitoring 5

permutation language modeling 643

perplexity 370

PIL library 249

pooled variance 43

pooling operation 43 – 45

PoS (Part of speech) tagging 297

predict(...) method 116

prefetch() function 537

prepare_data(...) function 412

pretrained encoder 634 – 638

pretrained networks

image segmentation with 266 – 277

atrous convolution 269 – 270

implementing ASPP module 273 – 275

implementing Deeplab v3 using Keras functional API 270 – 273

ResNet-50 model 268

Principal Component Analysis (PCA) 56, 548

probabilistic machine learning 19

productionizing models 11

profiling models 529 – 544

mixed precision training 539 – 544

optimizing input pipeline 536 – 538

projecter.visualize_embeddings() function 549

Protobuf library 569

prototyping deep learning models 10

pyramidal aggregation module 266

Python 16 – 17, 615 – 616

Pytorch framework 5

Q

quantile() function 315

question answering with Hugging Face Transformers 483 – 509

ask BERT question 505 – 509

data 485 – 486

defining DistilBERT model 495 – 502

processing data 487 – 494

defining and using tokenizer 488 – 493

from tokens to tf.data pipeline 493 – 494

training model 502 – 505

R

ragged tensor 319

RandomContrast layer 228

RandomCrop layer 228

randomly_crop_or_resize function 257, 259

random occlusions 202

read_csv() function 50

Reconstruction phase 86

recurrent neural networks. See RNNs

recursive functions 380

ReduceLROnPlateau callback 230 – 231

reduction block 225 – 226

Reformer 644 – 645

ReLU (rectified linear units) 53, 58, 223

reset gate 366

residual connections 187, 216

residuals 460 – 463

resize function 257

ResNet-50 model 268

resolvers 602

re.sub() function 304

return keyword 162

return_sequences parameter unit 367

return_state parameter unit 368

rnn layer 111

RNNs (recurrent neural networks) 10, 14, 105 – 117, 134, 325, 402

data 107 – 111

implementing model 111 – 115

predicting future CO2 values with trained model 115 – 117

RoBERT (recurrence over BERT) 640

rotation_range parameter 197

S

same padding 101

samplewise_center parameter 197

samplewise_std_normalization parameter 197

save_pretrained() function 505

SBD (semantic boundary data set) 291

scalars 131 – 134

scale_to_z_score() function 572

SchemaGen object 567

seg_dir directory 265

self-attention layer 456

cooking competition analogy 135 – 136

locality sensitive hashing in 644 – 645

masked self-attention layers 136 – 138

overview 128 – 131

scalars 131 – 134

SelfAttentionLayer objects 141

semantic segmentation 245

sentence-order prediction 644

sentiment analysis 325 – 336

defining final model 331 – 336

LSTM (long short-term memory) networks 326 – 331

sequence length 315 – 316

sequence-to-sequence learning 387 – 452

defining inference model 423 – 430

improving model with attention 435 – 445

defining final model 440 – 443

implementing Bahdanau attention in TensorFlow 436 – 440

training model 443 – 445

machine translation data 388 – 395

training and evaluating model 410 – 423

visualizing attention 445 – 451

writing English-German seq2seq machine translator 395 – 410

compiling model 409 – 410

define TextVectorization layers for seq2seq model 400 – 401

defining decoder and final model 405 – 409

defining encoder 401 – 405

TextVectorization layer 398 – 400

Sequential API 52 – 55

Sequential object 53

seq variable 377

shear_range parameter 199

SignatureDefs 586 – 589

signatures dictionary 587

SimpleRNN layer 112 – 114

skip connections 187

small-scale structured data 12 – 13

softmax normalization 58

spam classification 463 – 483

in TensorFlow 470 – 483

compiling model 480 – 482

data 470 – 471

defining model 474 – 480

evaluating and interpreting results 482 – 483

training model 482

treating class imbalance in data 471 – 474

overview 465 – 469

sparsity 175 – 176

state variables 377

state vector state 377

statistics 158 – 160

stem 169, 214

steps_per_epoch parameter 290

stop word removal 297, 303

stratified sampling 311

Stride parameter 167

StringLookup function 415

StringLookup layer 415

sub-classing API 61 – 65

T

take() function 67

td.data.Dataset.map() function 255

teacher forcing 405

TensorBoard 511 – 553

profiling models to detect performance bottlenecks 529 – 544

mixed precision training 539 – 544

optimizing input pipeline 536 – 538

tracking and monitoring models with 517 – 526

using tf.summary to write custom metrics during model training 526 – 529

visualizing data with 512 – 516

visualizing word vectors with 544 – 550

tensorboard_plugin_profile package 532

tensorboard.plugins.projector object 549

TensorFlow 3 – 18

Bahdanau attention in 436 – 440

data pipeline 251 – 265

final tf.data pipeline 264 – 265

optimizing tf.data pipelines 263 – 264

deep learning algorithms 14

GPU vs. CPU 9 – 10

language modeling 349 – 384

beam search 379 – 383

greedy decoding 374 – 379

GRUs (gated recurrent units) 365 – 370

measuring quality of generated text 370 – 372

processing data 350 – 365

training and evaluating 372 – 374

machine learning model 8

monitoring and optimization 14 – 15

NLP (natural language processing) with 296 – 348

defining end-to-end NLP pipeline with TensorFlow 319 – 325

sentiment analysis 325 – 336

text 298 – 319

training and evaluating model 336 – 339

word vectors 339 – 346

Python and TensorFlow 2 16 – 17

spam classification using BERT 470 – 483

compiling model 480 – 482

data 470 – 471

defining model 474 – 480

evaluating and interpreting results 482 – 483

training model 482

treating class imbalance in data 471 – 474

TensorBoard 511 – 553

profiling models to detect performance bottlenecks 529 – 544

tracking and monitoring models with 517 – 526

using tf.summary to write custom metrics during model training 526 – 529

visualizing data with 512 – 516

when not to use 12 – 13

creating complex natural language processing pipelines 13

implementing traditional machine learning models 12

manipulating and analyzing small-scale structured data 12 – 13

when to use 10 – 12

creating heavy-duty data pipelines 11 – 12

implementing models that run faster on optimized hardware 10 – 11

monitoring models during model training 11

productionizing models/serving on cloud 11

prototyping deep learning models 10

TensorFlow 2.0 19 – 79

building blocks 28 – 38

tf.Operation 35 – 38

tf.Tensor 32 – 35

tf.Variable 29 – 32

Keras model-building APIs 48 – 65

functional API 56 – 61

Sequential API 52 – 55

sub-classing API 61 – 65

neural network-related computations 39 – 45

convolution operation 41 – 43

matrix multiplication 39 – 41

pooling operation 43 – 45

overview 20 – 28

Python and 16 – 17

retrieving data for 65 – 78

Keras DataGenerators 72 – 74

tensorflow-datasets package 75 – 78

tf.data API 66 – 72

tensorflow.data API 6

tensorflow-dataset package 48, 75 – 78, 512

tensorflow_data_validation library 595

tensorflow.keras.layers.Dense() layer 140

tensorflow.keras.layers.experimental.preprocessing.TextVectorization layer 397

tensorflow.keras.layers.Flatten layer 104

tensorflow.keras.layers.MaxPool2D layer 103

tensorflow.keras.layers submodule 63

tensorflow_model_analysis library 602

tensorflow_transform library 572

tensor processing units (TPUs) 4

tensors 32 – 33

text 308 – 319

analyzing sequence length 315 – 316

analyzing vocabulary 313 – 315

generating with deep learning 365 – 370

measuring quality of generated 370 – 372

processing 298 – 308

representing as numbers 120 – 122

splitting training, validation and testing data 309 – 313

text to words and then to numbers with Keras 316 – 319

tokenizing 358 – 359

texts_to_sequences() function 318

text_to_sequences() function 318

TextVectorization layers

defining for seq2seq model 400 – 401

overview 398 – 400

tf.argmax mathematical function 38

tf.argmin mathematical function 38

tf.cond function 259

tf.constant objects 27

tf.cumsum mathematical function 38

tf.data API 6, 8, 66 – 72, 74 – 75

tf.data.Dataset() object 360

tf.data.Dataset.apply() function 324

tf.data.Dataset.batch() function 253, 261, 494

tf.data.Dataset.filter() function 321

tf.data.Dataset.flat_map() function 360, 362 – 363

tf.data.Dataset.from_generator() function 253, 494

tf.data.Dataset.from_tensor_slices() function 321, 324

tf.data.Dataset.map() function 255, 260, 362 – 363, 513

tf.data.Dataset objects 362, 526, 583

tf.data.Dataset.repeat() function 261

tf.data.Dataset.window() function 360, 362

tf.data.Dataset.zip() function 69

tf.data.experimental.bucket_by_sequence_length() function 323

tf.data.experimental.CsvDataset object 67

tf.data pipelines 252 – 255, 261, 263 – 264, 319, 331 – 332, 350, 360, 365, 487, 493 – 494, 531

final 264 – 265

language modeling 360 – 365

optimizing 263 – 264

tf_dataset_factory() function 584

tf.Dataset.filter() method 79

tf.Dataset.map() function 72

tfds.load() function 76

tfdv.display_anomalies() function 595

tfdv.validate_statistics() function 595

tfdv.visualize_statistics() function 595

tf.feature_column objects 581

tf.feature_column-type objects 578

tf.feature_column types 578

@tf.function decorator 25 – 26, 28, 587

TF_GPU_THREAD_COUNT variable 537

TF_GPU_THREAD_MODE=gpu_private variable 538

TF_GPU_THREAD_MODE variable 537

tf.image.resize operation 257

tf.io.parse_example() function 589

tf.io.read_file function 252

tf.keras.applications module 270

tf.keras.callbacks.EarlyStopping callback 210

tf.keras.initializers submodule 29

tf.keras.layers.AbstractRNNCell interface 436

tf.keras.layers.Add layer 336

tf.keras.layers.BatchNormalization() layers 527

tf.keras.layers.Conv2D layer 271

tf.keras.layers.DenseFeatures layer 582

tf.keras.layers.Input layers 582

tf.keras.layers.Lambda layer 172

tf.keras.layers.Masking layer 332

tf.keras.layers.RepeatVector(5) layer 410

tf.keras.layers.RepeatVector layer 410

tf.keras.metrics.Accuracy parent object’s 285

tf.keras.metrics.Mean class 371 – 372

tf.keras.metrics.Metric class 283 – 284

tf.keras.Model.fit() function 482

tf.keras.Model objects 583

tf.keras.models.Model.fit() method 162

tf.keras.models.Model object 404

tf.keras.preprocessing.text.Tokenizer.fit_on_texts() function 318

tf.linalg.band_part() function 138

tf.math.add_n() function 141

tf.math.top_k(batch_loss, n) function 293

tf.matmul() function 39

tf.matmul operation 28

tf.matmul(x,W)+b expression 23

tf-models-official library 475

tf.nn.convolution() function 42

tf.nn.convolution operation 96

tf.nn.max_pool() function 43

tf.numpy_function operation 254

tf.one_hot() function 333

tf.Operation 35 – 38

tf.ragged.constant() function 320

tf.RaggedTensor objects 320, 324, 360

TFRecord objects 559

tf.reshape() function 45

tf.SparseTensor objects 567

tf.squeeze() function 40, 44, 261

tf.string elements 253

tf.string operations 416

tf.summary 526 – 529

tf.summary.<data type> object 514

tf.summary.image object 514

tf.Tensor 32 – 35

tf.Tensor objects 567

tf.Variable 29 – 32

tf_wrap_model() function 504

tfx.components.CsvExampleGen object 560

tfx.dsl.Channel object 602

TFX (TensorFlow-Extended) 554 – 614

deploying model and serving it through API 599 – 612

evaluating model 602 – 608

predicting with TensorFlow serving API 608 – 612

pushing final model 608

resolving correct model 602

validating infrastructure 600 – 601

setting up Docker to serve trained model 596 – 599

training simple regression neural network with TFX Trainer API 576 – 596

defining Keras model 577 – 584

defining model training 584 – 586

SignatureDefs 586 – 589

training Keras model with TFX Trainer 589 – 596

writing data pipeline with 556 – 575

converting data to features 569 – 575

EDA (exploratory data analysis) 564 – 567

inferring schema from data 567 – 569

loading data from CSV files 560 – 564

TFX (TensorFlow-Extended) Trainer API 576 – 596

defining Keras model 577 – 584

defining model training 584 – 586

SignatureDefs 586 – 589

training Keras model with TFX Trainer 589 – 596

tiny-imagenet-200 data set 153 – 154, 232

ToBERT (Transformer over BERT) 640

tokenization 302, 316, 358 – 359, 488 – 494

torch tensors 508

TPUs (tensor processing units) 4

tracking models 517 – 526

trainable parameter 30

TrainArgs object 590

Trainer component 589

training model

attention 443 – 445

evaluating performance 149 – 192

Hugging Face Transformers, question answering with 502 – 505

image segmentation 289 – 290

language modeling 372 – 374

monitoring models during 11

NLP (natural language processing) 336 – 339

sequence-to-sequence learning 410 – 423

spam classification using BERT 482

TFX (TensorFlow-Extended) Trainer API 584 – 586

using tf.summary to write custom metrics during 526 – 529

word vectors 344 – 346

training phase 227

training pipeline 265

train_model() function 421, 443, 527

transfer learning 233 – 238, 244

Transform component 572

_transformed_name() function 575

Transformer decoder 639

Transformer encoder 639

transformers 123 – 146, 453 – 510, 639 – 645

Albert 643 – 644

cross-layer parameter sharing 643 – 644

factorization of embedding layer 643

sentence-order prediction instead of next sentence prediction 644

BART (bidirectional and auto-regressive Transformers) 640 – 642

components of 455 – 457

DistilBERT 640

encoder-decoder pattern 123 – 124

fully connected layer 139 – 141

GPT (generative pre-training) model 639 – 640

multi-head attention 138 – 139

question answering with Hugging Face Transformers 483 – 509

ask BERT question 505 – 509

data 485 – 486

defining DistilBERT model 495 – 502

processing data 487 – 494

training model 502 – 505

Reformer 644 – 645

residuals and normalization 460 – 463

RoBERT and ToBERT 640

self-attention layer

cooking competition analogy 135 – 136

masked self-attention layers 136 – 138

overview 128 – 131

scalars 131 – 134

spam classification using BERT 463 – 483

in TensorFlow 470 – 483

overview 465 – 469

XLNet 642 – 643

transformers library 454

transpose convolution 266

typing library 583

U

Ubuntu

creating virtual Python environment with Anaconda distribution 615 – 616

prerequisites for GPU support 616 – 618

installing CUDA 616 – 618

installing CUDNN 618

installing NVIDIA driver 616

notes on MacOS 618

U-Net model 632 – 638

defining 632 – 633

pretrained encoder 634 – 638

unet_pretrained_encoder() function 638

Unix-based environment 615 – 618

creating virtual Python environment with Anaconda distribution (Ubuntu) 615 – 616

prerequisites for GPU support (Ubuntu) 616 – 618

installing CUDA 616 – 618

installing CUDNN 618

installing NVIDIA driver 616

notes on MacOS 618

[UNK] tokens 417 – 418

update_char_to_token_positions_inplace() function 491

update gate 366

update_state() function 284 – 288, 372

upsample_conv layer 638

V

val_accuracy validation accuracy 374

validating infrastructure 600 – 601

validation/testing phase 227

validation data 161

validation_data parameter 290

validation pipeline 265

validation_split parameter 199

valid_mask filter 285

valid padding 101

val_loss validation loss value 210

val_perlexity validation perplexity 374

value_counts() function 301

vertical_flip parameter 199

Virtual Python Environment

Unix-based environment 615 – 616

Windows Environments 618 – 619

visualize_attention() function 448

visualizing

attention 445 – 451

with TensorBoard

data 512 – 516

word vectors 544 – 550

vocabulary

analyzing 313 – 315

n-grams 356 – 358

W

width_shift_range parameter 197

window() function 360

WindowDataset object 360

window element 362

Windows Environments 618 – 622

creating Virtual Python Environment (Anaconda) 618 – 619

prerequisites for GPU support 619 – 622

installing CUDA 620 – 622

installing CUDNN 622

installing NVIDIA driver 620

wnids 152

word embeddings

defining final model with 341 – 344

overview 340 – 341

wordnet external resource 303

WordNet IDs 152

WordNetLemmatizer 305

word_tokenize() function 304

word vectors 339 – 346

defining final model with word embeddings 341 – 344

training and evaluating model 344 – 346

word embeddings 340 – 341

X

XLNet 642 – 643

Y

yield keyword 162

Z

zca_whitening 197

zip() function 69