Announcements

All announcements will be on EdStem. Use this same link to join the course.

Class Info

  • Class: Tuesday 9:55 – 11:10 AM — Martin 300

  • Lab: Wednesday 1:05 – 4:15 PM — Martin 227

  • Professor: Vasanta Chaganti

  • Edstem: Q&A Forum & course join link

  • GitHub: Swarthmore GitHub Enterprise

  • Grades: Grades will be posted via GitHub and GradeScope.

Office hours: Martin 230

Vasanta

Thursday 12:00 – 2:00 PM

Friday 1:00 – 3:00 PM

Weekly Lab Session: Martin 227

Lab

Wednesday 1:05 – 4:15 PM

Goals for the course

By the end of this seminar, students should be able to:

  1. Identify concrete threat models for machine-learning systems in deployment.

  2. Reproduce a classical adversarial-ML attack (evasion, poisoning, membership inference, model extraction) and evaluate a documented defense.

  3. Use AI as a tool for defensive security, and understand its limits, in tasks like intrusion detection, malware analysis, and fuzzing.

  4. Read and critically discuss primary security-and-ML research papers.

  5. Communicate research findings in written and oral form to a technical audience.

Required Textbook

There is no single required textbook. All primary readings are research papers linked in the schedule.

Security Resources

Class Schedule

Week Announcements Date Topic Readings Labs

Section 1: Security OF AI Models

1

Sep 1

Course intro; threat models for ML; taxonomy of attacks

Papernot et al., SoK: Security & Privacy in ML (EuroS&P 2018); Biggio & Roli, Wild Patterns (Pattern Recognition 2018)

Sep 2

Lab 1

Lab 1 out — environment setup + adversarial baseline

2

Sep 7: Labor Day — no classes.

Sep 8

Adversarial examples: gradient-based attacks & defenses

Goodfellow et al., Explaining & Harnessing Adversarial Examples (ICLR 2015); Madry et al., Towards Deep Learning Models Resistant to Adversarial Attacks (ICLR 2018)

Lab 1 due

Sep 9

Lab 2 + Quiz 1

Lab 2 out — FGSM / PGD implementation & evaluation
Quiz 1

3

Sep 14: Drop/add ends. Last day to delete a course from or add to permanent registration. Last day to uncover a shadow grade for a CR course from the previous semester.

Sep 15

Data poisoning & backdoor attacks

Gu et al., BadNets (IEEE Access 2019); Shafahi et al., Poison Frogs! (NeurIPS 2018)

Lab 2 due

Sep 16

Lab 3

Lab 3 out — clean-label poisoning attack

4

Sep 22

Privacy attacks: membership inference & model extraction

Shokri et al., Membership Inference Attacks (IEEE S&P 2017); Tramèr et al., Stealing ML Models (USENIX Security 2016)

Lab 3 due

Sep 23

Lab 4 + Quiz 2

Lab 4 out — membership inference / model extraction
Quiz 2

Section 2: Security WITH AI

5

Oct 1: Final examination schedule available online.

Sep 29

AI for intrusion detection & anomaly detection

Sommer & Paxson, Outside the Closed World (IEEE S&P 2010); Mirsky et al., Kitsune (NDSS 2018)

Lab 4 due

Sep 30

Lab 5

Lab 5 out (2-week) — paper re-implementation project

6

Oct 5: All outstanding incompletes from Spring semester must be complete and graded.

Oct 6

ML for malware detection & classification

Raff et al., MalConv (AAAI 2018); Anderson & Roth, EMBER dataset (arXiv 2018)

Lab 5 milestone

Oct 7

Lab 5 + Quiz 3

Lab 5 continued — milestone check-in
Quiz 3

Fall Break: Oct 10 – 18

7

Oct 19: Fall classes resume at 8:30 a.m.

Oct 20

AI-assisted fuzzing & program analysis

Böttinger et al., Deep Reinforcement Fuzzing (SPW 2018); She et al., NEUZZ (IEEE S&P 2019)

Lab 5 due

Oct 21

Lab 6

Lab 6 out — automated pentest

8

Oct 28: Schedule of courses and seminars for next semester available online.

Oct 27

LLM agents in offensive security

Deng et al., PentestGPT (USENIX Security 2024); Fang et al., LLM Agents Can Autonomously Exploit One-Day Vulnerabilities (arXiv 2024)

Final project proposal due

Oct 28

Lab 6 + Quiz 4

Lab 6 continued
Quiz 4

9

Nov 2–13: Advising period.
Nov 6: Last day to declare CR/NC grading option. Last day to withdraw from a course and receive the grade notation "W."

Nov 3

Section 2 wrap-up: red-team + blue-team synthesis

Section review; instructor notes

Lab 6 due

Nov 4

Project

Project — scoping

Section 3: Security FROM AI

10

Advising period continues (Nov 2–13).

Nov 10

Deepfakes & synthetic media

Rossler et al., FaceForensics++ (ICCV 2019); Groh et al., Deepfake detection by human crowds, machines, and machine-informed crowds (PNAS 2022)

Project literature review due

Nov 11

Project + Quiz 5

Project — literature review & scoping
Quiz 5

11

Nov 17–19: Pre-enrollment for spring semester.
Nov 19: Pre-enrollment ends at 4 p.m.

Nov 17

Training-data extraction & memorization

Carlini et al., Extracting Training Data from LLMs (USENIX Security 2021); Nasr et al., Scalable Extraction of Training Data (arXiv 2023)

Project checkpoint 1

Nov 18

Project

Project — experimental design

12

Nov 25: Thanksgiving Break begins after last class.

Nov 24

LLM jailbreaks & prompt injection

Zou et al., Universal and Transferable Adversarial Attacks on Aligned Language Models (arXiv 2023); Greshake et al., Not what you’ve signed up for (AISec 2023)

No lab this week

Thanksgiving Break: Nov 25 – 29

13

Nov 30: Fall classes resume at 8:30 a.m.
Dec 1: All accounts must show a zero or positive balance to enroll or select a room for spring semester.

Dec 1

AI governance, red-teaming, disclosure

Anthropic, Responsible Scaling Policy (2024); NIST, AI Risk Management Framework core & profiles (2023–24)

Project checkpoint 2

Dec 2

Project + Quiz 6

Project — draft peer review
Quiz 6

14

Dec 9: Classes end.

Dec 8

Course wrap-up; project presentations

No new readings

Final presentations

Dec 9

Project

Project — final presentations

Finals: Dec 13 – 19 · Final papers due Dec 15

About Course Work

Class Policy

This course is a discussion-driven seminar. Tuesday lectures cover core technical concepts and common pitfalls in the readings.

Each student will present four papers during the semester, approximately one to two per section. Presentations are 15 minutes plus 10 minutes of Q&A. Attendance and active participation in discussion is expected in both class and lab; more than two unexcused absences will affect your participation grade.

Lab Policy

Wednesday lab periods will host quizzes on alternate weeks, hands-on lab work, and some lecture material.

Lab Due Dates

Each weekly lab is released Wednesday afternoon and due at 11:59 PM Tuesday the following week, before the next lecture. The two-week labs (Lab 5) and final project have explicit multi-week deadlines listed in the schedule.

For partnered labs, both partners are expected to contribute to every part of the assignment. Include a short "who did what" note at the top of your submission. If a partnership is not working, contact me early; splitting a partnership after the deadline is not usually possible.

Absence / Assignment Extension Policy

If you cannot attend class or lab, or cannot submit an assignment on time, email me before the deadline whenever possible. Each student gets one no-questions-asked 48-hour extension per semester on a weekly lab (not on presentations, quizzes, the two-week lab, or the final project). Beyond that, extensions are granted only for documented illness, family emergencies, or officially recognized religious observance.

Grading

Component Weight

Labs

30%

Paper presentations

20%

In-lab quizzes

20%

Final project

20%

Participation (class + lab)

10%

How to Succeed in CS 91R

  • Read actively. For each paper, note the threat model, assumptions, and one thing you would test. Bring those notes to lab.

  • Start labs early. Adversarial-ML experiments frequently need re-runs — most experiments take up significant wall-clock time.

  • Talk to your classmates. Discuss ideas freely (see Academic Integrity for what "freely" means). This is a seminar; the discussion is the class.

  • Come to office hours. Thursday 12–2 PM and Friday 1–3 PM in Martin 230, or by appointment.

  • Ask on EdStem. If you have a question, someone else probably has the same one. Post it publicly whenever the answer doesn’t reveal your code.

Policies

Legality and Ethics

Many techniques in this course — adversarial examples, model extraction, fuzzing, deepfake generation — can cause real harm if used against systems or people without authorization. You may not run any technique from this course against any system, model, or dataset you do not own or have explicit written permission to test. All labs are designed to be done on the course infrastructure or on models and datasets we provide. If you are unsure whether an experiment is appropriate, ask before running it. Unauthorized testing of live services (including public LLM APIs) can violate the Computer Fraud and Abuse Act and terms of service — with legal, academic, and career consequences.

Academic Integrity

Academic honesty is required in all your work. Under no circumstances may you hand in work done with (or by) someone else under your own name. Your code should never be shared with anyone; you may not examine or use code belonging to someone else, nor may you let anyone else look at or make a copy of your code. This includes, but is not limited to, obtaining solutions from students who previously took the course or code that can be found online. You may not share solutions after the due date of the assignment or make them publicly available anywhere (e.g. public GitHub repository).

Discussing ideas and approaches to problems with others on a general level is fine (in fact, we encourage you to discuss general strategies with each other), but you should never read anyone else’s code or let anyone else read your code. All code and written homeworks you submit must be your own with the following permissible exceptions: code distributed in class, code found in the course text book, and code worked on with an assigned partner. In these cases, you should always include detailed comments that indicates on which parts of the assignment you received help, and what your sources were.

Use of generative AI in this course: Because this is a course about AI systems, you may use AI assistants (ChatGPT, Copilot, Claude, etc.) as a study aid — for explanation, for understanding, and for grammar checking. You must: (1) disclose which AI tools you used and how, in a short note at the top of each submission; (2) understand and be able to explain every line of code you submit; (3) never paste model output verbatim as your own writing.

You may not use AI to generate text for your paper presentations or for your final project reports. Undisclosed or unedited AI output will be treated as an academic-integrity violation.

Failure to abide by these rules constitutes academic dishonesty and will lead to a hearing of the College Judiciary Committee. According to the Faculty Handbook:

Because plagiarism is considered to be so serious a transgression, it is the opinion of the faculty that for the first offense, failure in the course and, as appropriate, suspension for a semester or deprivation of the degree in that year is suitable; for a second offense, the penalty should normally be expulsion.

The spirit of this policy applies to all course work, including code, homework solutions (e.g., proofs, analysis, written reports), and exams. Please contact me if you have any questions about what is permissible in this course.

Exam Integrity

Students must strictly adhere to the following policy, which applies to all exams taken in a Computer Science course at Swarthmore:

Exam takers must place all non-essential items at the front of the room (or other designated area). Unless otherwise permitted, students may not have any electronic devices or course materials in their possession during the entirety of the exam. This includes cell phones, tablets, laptops, smart watches, course notes, articles and books, among others. These items should be placed at the front of the room near the proctor. If you need to leave the room during the exam, you must obtain permission from an instructor first. Any non-permitted discussion or aide in regards to exam material will result in immediate forfeiture of the exam and a report to the College Judiciary Committee. Please discuss any concerns or accommodations with your instructor prior to starting the exam.

Academic Accommodations

If you believe you need accommodations for a disability or a chronic medical condition, please visit the Student Disability Services website for details about the accommodations process. Since accommodations require early planning and are not retroactive, contact Student Disability Services as soon as possible. You are also welcome to contact me, Vasanta, privately to discuss your academic needs. However, all disability-related accommodations must be arranged, in advance, through Student Disability Services.

To receive an accommodation for a course activity you must have an official Accommodations Letter and you need to meet with me to work out the details of your accommodation at least two weeks prior to any activity requiring accommodations.

You are also welcome to contact me, privately to discuss your academic needs. However, all disability-related accommodations must be arranged, in advance, through Student Disability Services.