Statistics for Data Science and Analytics

About the Authors xvii

Acknowledgments xix

About the Companion Website xxi

Introduction xxiii

1 Statistics and Data Science 1

1.1 Big Data: Predicting Pregnancy 2

1.2 Phantom Protection from Vitamin E 2

1.3 Statistician, Heal Thyself 3

1.4 Identifying Terrorists in Airports 4

1.5 Looking Ahead 5

1.6 Big Data and Statisticians 5

2 Designing and Carrying Out a Statistical Study 9

2.1 Statistical Science 9

2.2 Big Data 10

2.3 Data Science 10

2.4 Example: Hospital Errors 11

2.5 Experiment 12

2.6 Designing an Experiment 13

2.7 The Data 19

2.8 Variables and Their Flavors 21

2.9 Python: Data Structures and Operations 25

2.10 Are We Sure We Made a Difference? 34

2.11 Is Chance Responsible? The Foundation of Hypothesis Testing 34

2.12 Probability 36

2.13 Significance or Alpha Level 38

2.14 Other Kinds of Studies 40

2.15 When to Use Hypothesis Tests 42

2.16 Experiments Falling Short of the Gold Standard 42

2.17 Summary 43

2.18 Python: Iterations and Conditional Execution 44

2.19 Python: Numpy, scipy, and pandas—The Workhorses of Data Science 50

Exercises 56

3 Exploring and Displaying the Data 61

3.1 Exploratory Data Analysis 61

3.2 What to Measure—Central Location 62

3.3 What to Measure—Variability 65

3.4 What to Measure—Distance (Nearness) 69

3.5 Test Statistic 71

3.6 Examining and Displaying the Data 72

3.7 Python: Exploratory Data Analysis/Data Visualization 80

Exercises 88

4 Accounting for Chance—Statistical Inference 91

4.1 Avoid Being Fooled by Chance 91

4.2 The Null Hypothesis 92

4.3 Repeating the Experiment 93

4.4 Statistical Significance 99

4.5 Power 103

4.6 The Normal Distribution 103

4.7 Summary 105

4.8 Python: Random Numbers 105

Exercises 115

5 Probability 121

5.1 What Is Probability 121

5.2 Simple Probability 122

5.3 Probability Distributions 126

5.4 From Binomial to Normal Distribution 129

5.5 Appendix: Binomial Formula and Normal Approximation 133

5.6 Python: Probability 134

Exercises 141

6 Categorical Variables 143

6.1 Two-way Tables 143

6.2 Conditional Probability 144

6.3 Bayesian Estimates 147

6.4 Independence 150

6.5 Multiplication Rule 154

6.6 Simpson’s Paradox 156

6.7 Python: Counting and Contingency Tables 157

Exercises 163

7 Surveys and Sampling 167

7.1 Literary Digest—Sampling Trumps “All Data” 167

7.2 Simple Random Samples 170

7.3 Margin of Error: Sampling Distribution for a Proportion 172

7.4 Sampling Distribution for a Mean 174

7.5 The Bootstrap 176

7.6 Rationale for the Bootstrap 177

7.7 Standard Error 188

7.8 Other Sampling Methods 188

7.9 Absolute vs. Relative Sample Size 192

7.10 Python: Random Sampling Strategies 192

Exercises 202

8 More than Two Samples or Categories 207

8.1 Count Data—R × C Tables 207

8.2 The Role of Experiments (Many Are Costly) 208

8.3 Chi-Square Test 210

8.4 Single Sample—Goodness-of-Fit 215

8.5 Numeric Data: ANOVA 217

8.6 Components of Variance 222

8.7 Factorial Design 224

8.8 The Problem of Multiple Inference 226

8.9 Continuous Testing 228

8.10 Bandit Algorithms 229

8.11 Appendix: ANOVA, the Factor Diagram, and the F-Statistic 230

8.12 More than One Factor or Variable—From ANOVA to Statistical Models 237

8.13 Python: Contingency Tables and Chi-square Test 237

8.14 Python: ANOVA 241

Exercises 246

9 Correlation 249

9.1 Example: Delta Wire 249

9.2 Example: Cotton Dust and Lung Disease 251

9.3 The Vector Product Sum Test 252

9.4 Correlation Coefficient 256

9.5 Correlation is not Causation 260

9.6 Other Forms of Association 261

9.7 Python: Correlation 262

Exercises 269

10 Regression 271

10.1 Finding the Regression Line by Eye 272

10.2 Finding the Regression Line by Minimizing Residuals 274

10.3 Linear Relationships 276

10.4 Prediction vs. Explanation 280

10.5 Python: Linear Regression 284

Exercises 293

11 Multiple Linear Regression 295

11.1 Terminology 295

11.2 Example—Housing Prices 296

11.3 Interaction 301

11.4 Regression Assumptions 304

11.5 Assessing Explanatory Regression Models 306

11.6 Assessing Regression for Prediction 314

11.7 Python: Multiple Linear Regression 324

Exercises 332

12 Predicting Binary Outcomes 337

12.1 K-Nearest-Neighbors 337

12.2 Python: Classification 343

Exercises 346

Index 349