Securing & Red-Teaming Generative AI Deployments

Course Overview

Introduction to Generative AI Security

• Security for AI vs. AI for Security
• The AI Security Paradox: Dual-Use
• GenAI Security Characteristics
• CIA Triad and GenAI
• Security by Modality
• Current Threat Landscape
• Emerging Security Challenges

OWASP Top 10 for LLM Applications

• Understanding OWASP for LLMs
• Input & Processing Vulnerabilities
• Supply Chain & Data Integrity
• Output & Agency Risks
• Advanced Attack Vectors

MITRE ATLAS Tactics and Techniques

• An Introduction to MITRE ATLAS
• ATLAS Core Components
• Case Studies & Mitigations
• Navigating the MITRE ATLAS Knowledge Base

Foundations of GenAI Red-Teaming

• The GenAI Red-Teaming Process
• Planning & Scoping
• Intelligence Gathering
• Manual Testing Approaches
• Vulnerability Validation
• Documentation & Reporting
• Automated Testing Preparation

Automating GenAI Red-Teaming

• Why Automate GenAI Red-Teaming?
• Foundations of Automated AI Red-Teaming
• Testing Frameworks and Tools
• Transition Roadmap
• Advanced Automation Techniques

Securing Training Data

• From Curated Collections to Web-Scale Data
• Controlling Access to Training Data
• Sanitizing Training Data
• Dataset Provenance and Integrity
• Technical Defenses: Hashing and Signatures

Mitigating Input and Output Data Attacks

• Introduction to Mitigation Strategies
• Input-Based Attacks
• User Input Guardrails
• Output-Based Attacks
• Additional Considerations

Protecting Models

• Why Model Protection Matters
• Techniques for Model Protection
• Model Security Monitoring and Telemetry

NIST AI RMF

• NIST AI RMF Overview
• Tailoring RMF for GenAI
• Applying the NIST AI RMF to GenAI Security
• Integrating Security into Governance
• Maintaining Security Posture

Prerequisites

Comfort writing Python code and working in developer interfaces
Practical cybersecurity experience and/or hold a major cybersecurity certification (e.g., Security+, CISSP)
​Foundational knowledge of generative AI, prompt engineering, retrieval augmented generation, and GenAI development patterns (Covered by GAI-1101 or similar)