Robust and Reliable Autonomous Systems
1-Week Intensive, Stanford Center for Professional Development, 2023
Designing Robust and Reliable AI Systems (SCPD page) was a week-long course covering topics such as ML robustness, explainability, verification, uncertainty quantification and more.
The course occured over 5 days with 2-hour lectures and homework assignments. The topics covered each day were:
- Day 1: Robust Machine Learning
- The big iid lie in ML
- Definition and examples of distribution shifts
- Mitigation strategies such as data augmentation, ensembling, fine-tuning
- Day 2: Model Explanation and Verification
- Why go beyond accuracy for model evaluation?
- Methods for explaining black-box models
- Neural network verification
- Day 3: Uncertainty Quantification
- Aleatoric vs. Epistemic uncertainty
- Algorithms for quantifying aleatoric uncertainty
- Algorithms for quantifying epistemic uncertainty
- Day 4: Closed-Loop Analysis
- Safety properties of closed-loops systems
- How to model the environment and AI system
- Algorithms for finding failures and estimating probability of failure
- Day 5: Emergining Directions in AI Safety
- Security risks (adversarial examples)
- Holistic assessment
- Emerging governance