Implementing Site Reliability Engineering Classroom Live Washington, DC October 26, 2020

Course Overview

Part 1 – Introduction

• 1. Introduction
• 2. The Production Environment at Google, From the Viewpoint of an SRE
• 3. Exercise: Mapping Your Production Environment

Part 2 – Principles

• 1. Embracing Risk
  • Managing Risk
  • Measuring Service Risk
  • Risk Tolerance of Services
  • Motivation for Error Budgets
• 2. Service-Level Objectives
  • Service Level Terminology
  • Indicators in Practice
  • Objectives in Practice
  • Agreements in Practice
• Exercise: Setting Service-Level Objectives
• 3. Eliminating Toil
  • What Is Toil?
  • Why Less Toil is Better
  • What Qualifies as Engineering?
  • Is Toil Always Bad?
• 4. Monitoring Distributed Systems
  • Definitions
  • Why Monitor?
  • Setting Reasonable Expectations
  • Symptoms Versus Causes
  • Black Box Versus White Box
  • The Four Golden Signals
  • Worrying About Your Tail
  • Choosing an Appropriate Resolution for Measurements
  • As Simple as Possible, No Simpler
  • Tying These Principles Together
  • Monitoring for the Long Term
• 5. The Evolution of Automation at Google
  • The Value of Automation
  • The Value for Google SRE
  • Use Cases for Automation
  • Automate Yourself Out of a Job
  • Soothing the Pain: Applying Automation to Cluster Turnups
  • Borg: Birth of the Warehouse-Scale Computer
  • Reliability is the Fundamental Feature
• 6. Release Engineering
  • The Role of a Release Engineer
  • Philosophy
  • Continuous Build and Deployment
  • Configuration Management
• 7. Simplicity
  • System Stability Versus Agility
  • The Virtue of Boring
  • I Won't Give Up My Code!
  • The "Negative Lines of Code" Metric
  • Minimal APIs
  • Modularity
  • Release Simplicity

Part 3 – Practices

• 1. Practical Alerting
  • Time-Series Monitoring Outside of Google
  • Instrumentation of Applications
  • Exporting Variables
  • Collection of Exported Data
  • Storage in the Time-Series Arena
  • Rule Evaluation
  • Alerting
  • Sharding the Monitoring Topology
  • Black-Box Monitoring
  • Maintaining the Configuration
• 2. Being On-Call
  • The Life of an On-Call Engineer
  • Balanced On-Call
  • Feeling Safe
  • Avoiding Inappropriate Operational Load
• 3. Effective Troubleshooting
  • Theory
  • In Practice
  • The Magic of Negative Results
  • Making Troubleshooting Easier
  • Exercise: Distributed System Troubleshooting
• 4. Emergency Response
  • What to Do When Systems Break
  • Test-Induced Emergency
  • Challenge-Induced Emergency
  • Process-Induced Emergency
  • Don't Repeat the Past—Learn From It
• 5. Managing Incidents
  • Unmanaged Incidents
  • Managed Incidents
  • When to Declare an Incident
  • Elements of Incident Management Process
• 6. Postmortem Culture: Learning from Failure
  • Google's Postmortem Philosophy
  • Collaborate and Share Knowledge
  • Introducing a Postmortem Culture
  • Exercise: Blameless Postmortem
• 7. Tracking Outages
  • Escalator
  • Outalator
• 8. Testing for Reliability
  • Types of Software Testing
  • Creating a Test and Build Environment
  • Testing at Scale
• 9. Software Engineering in SRE
  • Why is Software Engineering Within SRE Important?
  • Auxon Case Study
  • Intent-Based Capacity Planning
  • Fostering Software Engineering in SRE
• 10. Load Balancing at the Front End
  • Load Balancing Using DNS
  • Load Balancing at the Virtual IP Address
• 11. Load Balancing in the Datacenter
  • Identifying Bad Tasks: Flow Control and Lame Ducks
  • Limiting the Connections Pool with Subsetting
  • Load-Balancing Policies
• 12. Handling Overload
  • The Pitfalls of "Queries Per Second"
  • Per-Customer Limits
  • Client-Side Throttling
  • Criticality
  • Utilization Signals
  • Handling Overload Errors
  • Load from Connections
• 13. Addressing Cascading Failures
  • Causes of Cascading Failures and Designing to Avoid Them
  • Preventing Server Overload
  • Slow Startup and Cold Caching
  • Triggering Conditions for Cascading Failures
  • Testing for Cascading Failures
  • Immediate Steps to Address Cascading Failures
• 14. Managing Critical State: Distributed Consensus for Reliability
  • Motivating the Use of Consensus: Distributed Systems Coordination Failure
  • How Distributed Consensus Works
  • System Architecture Patterns for Distributed Consensus
  • Distributed Consensus Performance
  • Deploying Distributed Consensus-Based Systems
• 15. Distributed Periodic Scheduling with Cron
  • Cron Jobs and Idempotency
  • Cron at Large Scale
  • Building Cron at Google
• 16. Data Processing Pipelines
  • Origin of the Pipeline Design Pattern
  • Initial Effect of Big Data on the Simple Pipeline Pattern
  • Challenges with the Periodic Pipeline Pattern
  • Trouble Caused by Uneven Work Distribution
  • Drawbacks of Periodic Pipelines in Distributed Environments
  • Introduction to Google Workflow
  • Stages of Execution in Workflow
  • Ensuring Business Continuity
• 17. Data Integrity: What You Read Is What You Wrote
  • Data Integrity's Strict Requirements
  • Google SRE Objectives in Maintaining Data Integrity and Availability
  • How Google SRE Faces the Challenges of Data Integrity
  • 1T Versus 1E: Not "Just" a Bigger Backup
  • Knowing that Data Recovery Will Work
  • Case Studies
  • General Principles of SRE as Applied to Data Integrity
• 18. Reliable Product Launches at Scale
  • Launch Coordination Engineering
  • Setting Up a Launch Process
  • Developing a Launch Checklist
  • Selected Techniques for Reliable Launches
  • Development of LCE
  • Exercise: Develop a Production Readiness Review

Part 4 – Management

• 1. Accelerating SREs to On-Call and Beyond
  • You've Hired Your Next SRE, Now What?
  • Initial Learning Experiences: The Case for Structure Over Chaos
  • Creating Stellar Reverse Engineers and Improvisational Thinkers
  • Reverse Engineering a Production Service
  • Five Practices for Aspiring On-Callers
  • On-Call and Beyond: Rites of Passage and Practicing Continuing Education
• 2. Dealing with Interrupts
  • Managing Operational Load
  • Factors in Determining How Interrupts Are Handled
  • Imperfect Machines
• 3. Embedding an SRE to Recover from Operational Overload
  • Phase 1: Learn the Service and Get Context
  • Phase 2: Sharing Context
  • Phase 3: Driving Change
• 4. Communication and Collaboration in SRE
  • Communications: Production Meetings
  • Collaboration Within SRE
  • Case Study: Viceroy
  • Collaboration Outside SRE
  • Case Study: Migrating DFP to F1
• 5. The Evolving SRE Engagement Model
  • SRE Engagement: What, How, and Why
  • The PRR Model
  • The SRE Engagement Model
  • Production Readiness Reviews: Simple PRR Model
  • Evolving the Simple PRR Model: Early Engagement
  • Evolving Services Development: Frameworks and SRE Platform

Part 5 – Conclusions

• 1. Lessons Learned From Other Industries
• 2. Conclusion

Prerequisites

IT Operations experience is preferred.