Announcements
All announcements will be on EdStem. Use this same link to join the course.
Class Info
-
Class: Tuesday 9:55 – 11:10 AM — Martin 300
-
Lab: Wednesday 1:05 – 4:15 PM — Martin 227
-
Professor: Vasanta Chaganti
-
Edstem: Q&A Forum & course join link
-
GitHub: Swarthmore GitHub Enterprise
-
Grades: Grades will be posted via GitHub and GradeScope.
Office hours: Martin 230
Vasanta |
Thursday 12:00 – 2:00 PM |
Friday 1:00 – 3:00 PM |
Weekly Lab Session: Martin 227
Lab |
Wednesday 1:05 – 4:15 PM |
Goals for the course
By the end of this seminar, students should be able to:
-
Identify concrete threat models for machine-learning systems in deployment.
-
Reproduce a classical adversarial-ML attack (evasion, poisoning, membership inference, model extraction) and evaluate a documented defense.
-
Use AI as a tool for defensive security, and understand its limits, in tasks like intrusion detection, malware analysis, and fuzzing.
-
Read and critically discuss primary security-and-ML research papers.
-
Communicate research findings in written and oral form to a technical audience.
Required Textbook
There is no single required textbook. All primary readings are research papers linked in the schedule.
Security Resources
-
MITRE ATLAS — living catalog of adversarial ML techniques.
-
Google Responsible AI Practices and NIST AI Risk Management Framework — governance frameworks referenced in Section 3.
Class Schedule
| Week | Announcements | Date | Topic | Readings | Labs |
|---|---|---|---|---|---|
Section 1: Security OF AI Models |
|||||
1 |
Sep 1 |
Course intro; threat models for ML; taxonomy of attacks |
Papernot et al., SoK: Security & Privacy in ML (EuroS&P 2018); Biggio & Roli, Wild Patterns (Pattern Recognition 2018) |
||
Sep 2 |
Lab 1 |
Lab 1 out — environment setup + adversarial baseline |
|||
2 |
Sep 7: Labor Day — no classes. |
Sep 8 |
Adversarial examples: gradient-based attacks & defenses |
Goodfellow et al., Explaining & Harnessing Adversarial Examples (ICLR 2015); Madry et al., Towards Deep Learning Models Resistant to Adversarial Attacks (ICLR 2018) |
Lab 1 due |
Sep 9 |
Lab 2 + Quiz 1 |
Lab 2 out — FGSM / PGD implementation & evaluation |
|||
3 |
Sep 14: Drop/add ends. Last day to delete a course from or add to permanent registration. Last day to uncover a shadow grade for a CR course from the previous semester. |
Sep 15 |
Data poisoning & backdoor attacks |
Gu et al., BadNets (IEEE Access 2019); Shafahi et al., Poison Frogs! (NeurIPS 2018) |
Lab 2 due |
Sep 16 |
Lab 3 |
Lab 3 out — clean-label poisoning attack |
|||
4 |
Sep 22 |
Privacy attacks: membership inference & model extraction |
Shokri et al., Membership Inference Attacks (IEEE S&P 2017); Tramèr et al., Stealing ML Models (USENIX Security 2016) |
Lab 3 due |
|
Sep 23 |
Lab 4 + Quiz 2 |
Lab 4 out — membership inference / model extraction |
|||
Section 2: Security WITH AI |
|||||
5 |
Oct 1: Final examination schedule available online. |
Sep 29 |
AI for intrusion detection & anomaly detection |
Sommer & Paxson, Outside the Closed World (IEEE S&P 2010); Mirsky et al., Kitsune (NDSS 2018) |
Lab 4 due |
Sep 30 |
Lab 5 |
Lab 5 out (2-week) — paper re-implementation project |
|||
6 |
Oct 5: All outstanding incompletes from Spring semester must be complete and graded. |
Oct 6 |
ML for malware detection & classification |
Raff et al., MalConv (AAAI 2018); Anderson & Roth, EMBER dataset (arXiv 2018) |
Lab 5 milestone |
Oct 7 |
Lab 5 + Quiz 3 |
Lab 5 continued — milestone check-in |
|||
Fall Break: Oct 10 – 18 |
|||||
7 |
Oct 19: Fall classes resume at 8:30 a.m. |
Oct 20 |
AI-assisted fuzzing & program analysis |
Böttinger et al., Deep Reinforcement Fuzzing (SPW 2018); She et al., NEUZZ (IEEE S&P 2019) |
Lab 5 due |
Oct 21 |
Lab 6 |
Lab 6 out — automated pentest |
|||
8 |
Oct 28: Schedule of courses and seminars for next semester available online. |
Oct 27 |
LLM agents in offensive security |
Deng et al., PentestGPT (USENIX Security 2024); Fang et al., LLM Agents Can Autonomously Exploit One-Day Vulnerabilities (arXiv 2024) |
Final project proposal due |
Oct 28 |
Lab 6 + Quiz 4 |
Lab 6 continued |
|||
9 |
Nov 2–13: Advising period. |
Nov 3 |
Section 2 wrap-up: red-team + blue-team synthesis |
Section review; instructor notes |
Lab 6 due |
Nov 4 |
Project |
Project — scoping |
|||
Section 3: Security FROM AI |
|||||
10 |
Advising period continues (Nov 2–13). |
Nov 10 |
Deepfakes & synthetic media |
Rossler et al., FaceForensics++ (ICCV 2019); Groh et al., Deepfake detection by human crowds, machines, and machine-informed crowds (PNAS 2022) |
Project literature review due |
Nov 11 |
Project + Quiz 5 |
Project — literature review & scoping |
|||
11 |
Nov 17–19: Pre-enrollment for spring semester. |
Nov 17 |
Training-data extraction & memorization |
Carlini et al., Extracting Training Data from LLMs (USENIX Security 2021); Nasr et al., Scalable Extraction of Training Data (arXiv 2023) |
Project checkpoint 1 |
Nov 18 |
Project |
Project — experimental design |
|||
12 |
Nov 25: Thanksgiving Break begins after last class. |
Nov 24 |
LLM jailbreaks & prompt injection |
Zou et al., Universal and Transferable Adversarial Attacks on Aligned Language Models (arXiv 2023); Greshake et al., Not what you’ve signed up for (AISec 2023) |
No lab this week |
Thanksgiving Break: Nov 25 – 29 |
|||||
13 |
Nov 30: Fall classes resume at 8:30 a.m. |
Dec 1 |
AI governance, red-teaming, disclosure |
Anthropic, Responsible Scaling Policy (2024); NIST, AI Risk Management Framework core & profiles (2023–24) |
Project checkpoint 2 |
Dec 2 |
Project + Quiz 6 |
Project — draft peer review |
|||
14 |
Dec 9: Classes end. |
Dec 8 |
Course wrap-up; project presentations |
No new readings |
Final presentations |
Dec 9 |
Project |
Project — final presentations |
|||
Finals: Dec 13 – 19 · Final papers due Dec 15 |
|||||
About Course Work
Class Policy
This course is a discussion-driven seminar. Tuesday lectures cover core technical concepts and common pitfalls in the readings.
Each student will present four papers during the semester, approximately one to two per section. Presentations are 15 minutes plus 10 minutes of Q&A. Attendance and active participation in discussion is expected in both class and lab; more than two unexcused absences will affect your participation grade.
Lab Policy
Wednesday lab periods will host quizzes on alternate weeks, hands-on lab work, and some lecture material.
Lab Due Dates
Each weekly lab is released Wednesday afternoon and due at 11:59 PM Tuesday the following week, before the next lecture. The two-week labs (Lab 5) and final project have explicit multi-week deadlines listed in the schedule.
For partnered labs, both partners are expected to contribute to every part of the assignment. Include a short "who did what" note at the top of your submission. If a partnership is not working, contact me early; splitting a partnership after the deadline is not usually possible.
Absence / Assignment Extension Policy
If you cannot attend class or lab, or cannot submit an assignment on time, email me before the deadline whenever possible. Each student gets one no-questions-asked 48-hour extension per semester on a weekly lab (not on presentations, quizzes, the two-week lab, or the final project). Beyond that, extensions are granted only for documented illness, family emergencies, or officially recognized religious observance.
How to Succeed in CS 91R
-
Read actively. For each paper, note the threat model, assumptions, and one thing you would test. Bring those notes to lab.
-
Start labs early. Adversarial-ML experiments frequently need re-runs — most experiments take up significant wall-clock time.
-
Talk to your classmates. Discuss ideas freely (see Academic Integrity for what "freely" means). This is a seminar; the discussion is the class.
-
Come to office hours. Thursday 12–2 PM and Friday 1–3 PM in Martin 230, or by appointment.
-
Ask on EdStem. If you have a question, someone else probably has the same one. Post it publicly whenever the answer doesn’t reveal your code.
Policies
Legality and Ethics
Many techniques in this course — adversarial examples, model extraction, fuzzing, deepfake generation — can cause real harm if used against systems or people without authorization. You may not run any technique from this course against any system, model, or dataset you do not own or have explicit written permission to test. All labs are designed to be done on the course infrastructure or on models and datasets we provide. If you are unsure whether an experiment is appropriate, ask before running it. Unauthorized testing of live services (including public LLM APIs) can violate the Computer Fraud and Abuse Act and terms of service — with legal, academic, and career consequences.
Academic Integrity
Academic honesty is required in all your work. Under no circumstances may you hand in work done with (or by) someone else under your own name. Your code should never be shared with anyone; you may not examine or use code belonging to someone else, nor may you let anyone else look at or make a copy of your code. This includes, but is not limited to, obtaining solutions from students who previously took the course or code that can be found online. You may not share solutions after the due date of the assignment or make them publicly available anywhere (e.g. public GitHub repository).
Discussing ideas and approaches to problems with others on a general level is fine (in fact, we encourage you to discuss general strategies with each other), but you should never read anyone else’s code or let anyone else read your code. All code and written homeworks you submit must be your own with the following permissible exceptions: code distributed in class, code found in the course text book, and code worked on with an assigned partner. In these cases, you should always include detailed comments that indicates on which parts of the assignment you received help, and what your sources were.
Use of generative AI in this course: Because this is a course about AI systems, you may use AI assistants (ChatGPT, Copilot, Claude, etc.) as a study aid — for explanation, for understanding, and for grammar checking. You must: (1) disclose which AI tools you used and how, in a short note at the top of each submission; (2) understand and be able to explain every line of code you submit; (3) never paste model output verbatim as your own writing.
You may not use AI to generate text for your paper presentations or for your final project reports. Undisclosed or unedited AI output will be treated as an academic-integrity violation.
Failure to abide by these rules constitutes academic dishonesty and will lead to a hearing of the College Judiciary Committee. According to the Faculty Handbook:
Because plagiarism is considered to be so serious a transgression, it is the opinion of the faculty that for the first offense, failure in the course and, as appropriate, suspension for a semester or deprivation of the degree in that year is suitable; for a second offense, the penalty should normally be expulsion.
The spirit of this policy applies to all course work, including code, homework solutions (e.g., proofs, analysis, written reports), and exams. Please contact me if you have any questions about what is permissible in this course.
Exam Integrity
Students must strictly adhere to the following policy, which applies to all exams taken in a Computer Science course at Swarthmore:
Exam takers must place all non-essential items at the front of the room (or other designated area). Unless otherwise permitted, students may not have any electronic devices or course materials in their possession during the entirety of the exam. This includes cell phones, tablets, laptops, smart watches, course notes, articles and books, among others. These items should be placed at the front of the room near the proctor. If you need to leave the room during the exam, you must obtain permission from an instructor first. Any non-permitted discussion or aide in regards to exam material will result in immediate forfeiture of the exam and a report to the College Judiciary Committee. Please discuss any concerns or accommodations with your instructor prior to starting the exam.
Academic Accommodations
If you believe you need accommodations for a disability or a chronic medical condition, please visit the Student Disability Services website for details about the accommodations process. Since accommodations require early planning and are not retroactive, contact Student Disability Services as soon as possible. You are also welcome to contact me, Vasanta, privately to discuss your academic needs. However, all disability-related accommodations must be arranged, in advance, through Student Disability Services.
To receive an accommodation for a course activity you must have an official Accommodations Letter and you need to meet with me to work out the details of your accommodation at least two weeks prior to any activity requiring accommodations.
You are also welcome to contact me, privately to discuss your academic needs. However, all disability-related accommodations must be arranged, in advance, through Student Disability Services.