Index

As this ebook edition doesn't have fixed pagination, the page numbers below are hyperlinked for reference only, based on the printed edition of this book.

Symbols

0-switch coverage 172

404 content injectio 266

A

A/B testing

advantages 245

disadvantages 246

accidental load test 348

ad hoc testing 4

Agile approaches 7

alarms 280

alerts

filtering 283, 284

API commands

checking 110

example 108, 109

API testing 167-169

performing 108

asynchronous applications

checking 174

duplicate messages, checking 174

missing response messages, checking 176

missing update messages, checking 175

out-of-order messages, checking 175

attack area 252

discovering 251

authentication

alternative login methods 260

API authentication 259

tests for login sessions 259, 260

tests for passwords 257-259

tests for usernames 256, 257

automated testing 25

off precision trading, versus brittleness 140-142

B

Barry Boehm’s spiral model 14

beta testing 31

billion laughs attack 264

black box 47

black-box testing

advantages 104, 105

API testing, performing 108

boundary value analysis, using 119, 120

bugs and features, comparing 143

cause-effect graphing, using 124-127

CRUD testing, performing 110, 111

decision tables, using 122-124

dependent variables, mapping 120-122

disadvantages 104, 105

equivalence partitioning 117, 118

error guessing, optimizing 134-136

feedback, using 137, 138

hidden defects, uncovering 133, 134

independent variables, mapping 120-122

negative testing, performing 114, 115

new features, enabling 105-108

numeric processing, testing 132, 133

test prioritization 142, 143

variable types, testing 127

what to check, determining 138-140

worse-case scenarios, identifying 115-117

blue/green systems 294

boundary value analysis

examples 119

using 119, 120

buffer overflow attack 263

bug bounty program

running 272

bugs and feature requests

comparing 143

Business Requirement Document (BRD) 32, 33

C

Cassandra 309

cause-effect graphing

using 124-127

chaos engineering 319, 321

Chaos Monkey 320

Chromatic 218

clickjacking 267

client-server applications

load runner architecture 333, 334

client-side errors

raising 285

code analysis

running 254, 255

using 147, 148

code coverage

calculating 153

condition coverage 156

condition/decision coverage 157

decision coverage 155, 156

function coverage 153

loop coverage 161, 162

measuring 152, 153

modified condition/decision coverage 157, 158

multiple condition coverage 159, 160

parameter value coverage 160

path coverage 162, 163

state coverage 162, 163

statement coverage 154, 155

test coverage, selecting 163, 164

code inefficiencies 177

communication failure 318, 319

condition coverage 156, 157

test cases 156

Confidentiality, Integrity, and data Availability (CIA) 54

Continuous Integration/Continuous Delivery (CI/CD) 93

critical alerts 283

Cross-Origin Resource Sharing (CORS) 268

cross-site scripting attacks 263, 264

CRUD testing

creation operations, testing 113

deletion operations, testing 111, 112

performing 110, 111

read operations, testing 114

update operations, testing 112, 113

CSRF attacks 268

customer

losing 288

customer outcomes

versus system resources 286

D

decision coverage 155-157

test cases 156

decision tables

using 122-124

defect hiding 134

Denial of Service (DoS) 358

dependent variable

mapping 120-122

descriptive tests 90

advantages and disadvantages 89

destructive testing 307

advantages 308, 309

backup and restore testing 321-324

disadvantages 308, 309

disaster recovery testing 309

DRY (Don’t Repeat Yourself) 149

dual redundant hardware

logging on 295

Dual-Tone Multi-Frequency (DTMF) 271

dynamic load

versus static load 330, 331

dynamic testing 70

E

email text input fields

testing 128, 129

end-to-end testing 79

equivalence partitioning 117, 118

error

predicting 176

error alerts 283

error case testing

advantages 182, 183

data failures 201, 202

defect cascading 190-192

disadvantages 182, 183

expected versus unexpected problems, classifying 183

fuzzing 202, 203

network degradation 197

prioritizing 192

system limits 193, 194

timeouts 194-197

user feedback, giving on errors 203, 204

error feedback testing

errors, policing 243

errors, presenting 240, 241

errors, preventing 240

internal failures 243

spurious errors 243

spurious errors, avoiding 242

error guessing 25

optimizing 134-136

error handling

philosophies 184

error handling, philosophies

fail as early as possible 185

fail as little as possible 189, 190

error handling philosophies, unexpected failures

errors, catching in backend 186, 187

errors, catching in frontend 185

errors written, catching into storage 187

failing, early in release cycle 185

stored errors, catching 188

events 280

exploratory testing 4

activities 12-14

advantages 5, 6

alternative approaches 5, 6

disadvantages 5, 6

features, mapping out 19, 20

identifying, if feature is ready 16, 17

naivety while testing, using 20, 21

performing 17, 18

results, checking 24

running 21-23

spiral model, of test improvement 14-16

usage 6-12

using, curiosity 24-26

using, significance 23, 24

F

failover 309

failover server

failing 310

failover testing 309

classes of redundancy 309, 310

error reporting 311

performing 310, 311

system recovery 313

timeout, recovering 312, 313

feature specifications

advantages 30, 31

alternatives 31, 32

alternatives, examples 31

arguments, countering against 58-60

completing 51

disadvantages 30, 31

documentation 55, 56

draft 56, 57

error cases 53

functional test requirements 52, 53

handover, improving from product owner 36, 37

maintainability 54

non-functional tests 55

requirement content, improving 49-51

requirements format 37

requirement’s priority 39

requirement statements, improving 40

round-up, of requirement statement improvements 48, 49

security test requirements 54

turning, into test plan 57, 58

user experience specification 53

feature specifications, alternatives

Business Requirement Document 32, 33

diagrams 35

implicit specifications 34

round-up 36

technical specification 33

test plans 35

user interface mockups 34

user stories 33, 34

feature specifications, requirements format

numbering 38

sections 39

feature specifications, requirement statements

agreed 44

complete 44

consistent 46

implementation-free 47, 48

independent 44-46

measurable 43, 44

precise statement, correcting 41-43

realistic 44

specific 40

feedback

using 137, 138

file inputs

validating 264

file storage

testing 265

file uploads

testing 131, 132, 264

First-In-First-Out (FIFO) 369

flakes 141

full boundary analysis 119

function coverage 153

fuzzing 202, 203

G

General Data Protection Regulation (GDPR) 270

generic text input fields

testing 127, 128

Goldilocks zone

of logging 300, 301

graceful shutdown 310

gray-box testing 178, 179

H

hard shutdown 310

hidden defects

uncovering 133, 134

host header attacks 267

HTML injection 262

I

idempotent 174

independent variable

mapping 120-122

injection attacks

code injection 263

cross-site scripting attacks 263, 264

HTML injection 262

SQL injection 262

testing 261

integration tests

advantages 83, 84

disadvantages 83, 84

International Software Testing Qualifications Board (ISTQB) 25

iPhone

hidden tools 233

iptables 319

issues

debugging 282

L

laxer specification 46

Linux command prompt 230

load testing 139

advantages 326-328

asynchronous systems 344, 345

considerations 337, 338

defect, hiding 343, 344

design 333

disadvantages 326-328

errors, filtering 350

inefficiencies, loading 347

issues, debugging 352

messages, loading between modules 347, 348

performance testing 348

prerequisites 328, 329

race conditions 344, 345

startup and shutdown 346

system limits, increasing 335, 336

load testing, considerations

memory leaks 338, 339

results, reporting 340-343

system resource leaks 339, 340

load testing, design

runner architecture, for client-server applications 333, 334

runner architectures 334

runner functions 335

runner interfaces 334

load testing, errors

loading reliability 351, 352

result sensitivity 351

load testing, operations

combinations, loading 332

dynamic load, versus static load 330, 331

identifying 329, 330

soak testing, versus spikes of load 331

load testing, performance

bottlenecks, identifying 349

in release cycle 349

log 279

searching 295

logging 295

debugging steps 296, 297

Goldilocks zone 300, 301

messages, logging 299, 300

messages, need for 297, 299

overview 302

usability 301, 302

using 295

logging in 255, 256

authentication 256

authorization 260, 261

logging, on dual redundant hardware 295

loop coverage 161

M

maintainability

features 276

goals 278

use cases 276

maintainability, goals

issues, debugging 282

system degradation, identifying 280, 281

system, improving 281

tools, for observability 278-280

maintainability testing

advantages 277, 278

disadvantages 277, 278

maintenance

version 287

maintenance operations

centralized commands 289

client upgrades 292

overview 294

recovery and rollback 293

testing 287

transitions, testing 294

upgrade control, testing 290

upgrade processes, testing 290, 291

upgrade types, testing 289

worst-case 288

Mean Time Between Failure (MTBF) 343

messages

displaying 298

logging 299, 300

testing 165-167

metrics 280

mock 79

modified condition/decision coverage 157, 158

test cases 158

modules

testing 165, 166

monitoring 282

active checks, performing 285

alerts, filtering 283, 284

automatic problem reporting 287

designing 282

hierarchies of system failures 286

instrumentation 283

interacting, ways 282

overview 287

system resources versus customer outcomes 286

multi-factor authentication (MFA) 256

multiple condition coverage 159

test cases 159

Murphy’s law 127

N

naivety while testing

using 20, 21

negative testing

performing 114, 115

netem 200, 319

network degradation 197

sources 197-199

testing 200, 201

types 197

Network Link Conditioner 200

Network Time Protocol (NTP) 299

non-redundant VPNs 314

N-switch coverage 172, 173

numeric processing

testing 132, 133

numeric text input fields

testing 129

O

observability

tools 278-280

observability, data types

events 280

logs 279

metrics 280

traces 279

P

packet loss 284

PacketStorm 200

parameter value coverage 160

password complexity heuristics 130

password text input fields

testing 129

path coverage 163

performance testing 22

personally identifiable information (PII) 265

handling 269, 270

poetry 255

positive feedback loops

identifying 360, 361

prescriptive tests 88-90

advantages and disadvantages 88

product owner

handover, improving from 36

R

race condition 345

regression testing 23

release cycle

testing 95-97

reliability testing 22

Requests For Quotations (RFQs) 46

round-robin 370

S

scripting attacks 22

security

avoiding, through obscurity 273

considering, beyond beyond the application 273

security scans

issues 252, 253

results 253, 254

running 252, 253

security testing

advantages 250, 251

disadvantages 250, 251

security threats 251

Session Initiation Protocol (SIP) 139

shift-left testing 7

Simple Network Management Protocol (SNMP) 95

soak testing 364

classes 365, 366

versus spikes of load 331

Socratic method 73

Software Requirements Specification (SRS) 29

specification review meetings

advantages 64, 65

alternatives 66, 67

black box, opening 72

checklist, reviewing 74

disadvantages 64, 65

good tone, setting 72, 73

guest list 67, 68

incorrect requirements, fixing 71, 72

informal reviews 66

inspections 66

need for 64

requirement testing, prioritizing 74

running 70, 71

scheduling 70

tasks 68, 69

technical reviews 66

walkthroughs 66

spiral model, of test improvement 14-16

stages 15

Spirent 200

SQL injection 22, 262

SSH File Transfer Protocol (SFTP) 95

staging areas

advantages and disadvantages 92

using 92

state coverage 162

statement coverage 154, 155

state transition

considerations 169, 171

example 171, 172

N-switch coverage 172-174

static load

versus dynamic load 330, 331

static testing 70

code conciseness 149, 150

code format 149

code functionality 151

considerations 148

defensive coding 150

unit tests 151, 152

using 148

stress test architecture 357

stress testing 355, 356

advantages 356

disadvantages 356

graceful degradation 362

limited resources 362-364

monitoring 371-373

policed limits 360

positive feedback loops, identifying 360, 361

queues 366-370

spikes, in load 366

transaction rates, breaching 357-360

unpoliced limits 360

with errors 370, 371

stub 79

subsystem 300

system

improving 281

systematic testing

performing 94, 95

system degradation

identifying 280, 281

system failures

hierarchies 286

system recovery 313

system resources

versus customer outcomes 286

system shutdowns 313, 314

back up, starting 317, 318

error reporting 315

killing processes 315-317

tests plan 314, 315

system testing 79

performing 12-14

system tests

advantages 84, 85

disadvantages 84, 85

T

temporary test environments

advantages and disadvantages 91

using 91, 92

test basis 13, 108

test cases 86, 87

principles 87, 88

subsets 96

test design 13

Test Driven Development (TDD) 8, 148

test environment

configuration, setting 93, 94

correct version, setting 93, 94

evaluating 91

testing

in release cycle 95-97

levels 78, 79

testing, levels

examples 80

ordering 81

testing pyramid 81, 82

test plan

curiosity and feedback, using 97, 98

time input fields

testing 130

traces 279

Traffic Shaper XP 200

transaction rates

breaching 357-360

Two-Factor Authentication (2FA) 37

U

unit test coverage

advantages 164

disadvantages 164

unit tests

advantages 82, 83

disadvantages 82, 83

universally unique identifiers (UUIDs) 271

upgrades 292

usability testing 21

user experience (UX) testing 207

accessibility, ensuring 223-225

advantages 209, 210

bugs, versus feature requests 247

defining 208

disadvantages 209, 210

displays, testing 217, 218

documentation, testing 238, 239

error feedback, testing 240

feature specification 211

information display, in tables 226-228

information display overview 229

information display, testing 225, 226

interoperability 216, 217

loading 244

localization, testing 220-222

notifications, testing 229

reviews, running 211, 212

scroll bars, using 218

setup, versus ongoing usage 212-214

time zones, testing 222, 223

touch screens, using 219

usability 210, 211

usability studies, running 244

user interaction, testing 230

user interface, testing 215

visual elements 228

words, selecting 219, 220

user-facing textboxes

testing 131

user feedback methods

A/B testing 245

instrumentation 246

questionnaires 245

recorded sessions 245

user interaction testing 230

data structure design 237

features, accessing in multiple ways 236

feedback, providing 232, 233

hidden tools 233, 234

irreversible operations 234

menu systems 235, 236

overview 238

restarts 232

user steps combinations, selecting 231

user steps, counting 230, 231

V

variable types

email text input fields, testing 128, 129

file uploads, testing 131, 132

generic text input fields, testing 127, 128

numeric text input fields, testing 129

password text input fields, testing 129, 130

testing 127

time input fields, testing 130

user-facing textboxes, testing 131

VMWare interface 233

W

warning alerts 283

waterfall model 7

web application security, testing 265

404 content injection 266

clickjacking 267

CORS attacks 268, 269

CSRF attacks 268

host header attacks 267

information leakage 266

long password attac 267

webhooks 175

WET (Write Everything Twice) 149

white-box checks 177

code inefficiencies, checking 177

incorrect values, checking 178

white-box functional testing 145

advantages 146

disadvantages 146

white-box testing 21

WinDriver 200

worse-case scenarios

identifying 115-117

X

XML formatting

fixing 23