Reinforcement Learning for Cyber Operations

List of Figures xv

About the Authors xix

Foreword xxi

Preface xxiii

Acknowledgments xxv

Acronyms xxvii

Introduction xxix

1 Motivation 1

1.1 Introduction 1

1.1.1 Cyberattack Campaigns via MITRE ATT&CK 4

1.2 Attack Graphs 4

1.3 Cyber Terrain 5

1.4 Penetration Testing 6

1.5 AI Reinforcement Learning Overview 6

1.6 Organization of the Book 8

2 Overview of Penetration Testing 11

2.1 Penetration Testing 11

2.2 Importance of Data 43

2.3 Conclusion 56

3 Reinforcement Learning: Theory and Application 61

3.1 An Introduction to Reinforcement Learning (RL) 61

3.2 RL and Markov Decision Processes 63

3.3 Learnable Functions for Agents 66

3.4 Enter Deep Learning 69

3.5 Q-Learning and Deep Q-Learning 72

3.6 Advantage Actor-Critic (A2C) 78

3.7 Proximal Policy Optimization 83

3.8 Conclusion 85

4 Motivation for Model-driven Penetration Testing 89

4.1 Introduction 89

4.2 Limits of Modern Attack Graphs 91

4.3 RL for Penetration Testing 93

4.4 Modeling MDPs 95

4.5 Conclusion 98

5 Operationalizing RL for Cyber Operations 105

5.1 A High-Level Architecture 105

5.2 Layered Reference Model 107

5.3 Key Challenges for Operationalizing RL 113

5.4 Conclusions 117

6 Toward Practical RL for Pen-Testing 121

6.1 Current Challenges to Practicality 121

6.2 Practical Scalability in RL 130

6.3 Model Realism 136

6.4 Examples of Applications 144

6.5 Realism and Scale 154

7 Putting it Into Practice: RL for Scalable Penetration Testing 161

7.1 Crown Jewels Analysis 161

7.2 Discovering Exfiltration Paths 165

7.3 Discovering Command and Control Channels 171

7.4 Exposing Surveillance Detection Routes 176

7.5 Enhanced Exfiltration Path Analysis 183

8 Using and Extending These Models 193

8.1 Supplementing Penetration Testing 193

8.2 Risk Scoring 199

8.3 Further Modeling 201

8.4 Generalization 214

9 Model-driven Penetration Testing in Practice 225

9.1 Recap 225

9.2 The Case for Model-driven Cyber Detections 231

References 246

A Appendix 251

Index 253