Lab 3: Data Poisoning and Backdoor Attacks

LAB DUE DATE: TUESDAY, September 22nd, 11.59PM

1. Data Poisoning and Backdoor Attacks

This lab shifts the attack surface from inference time (Lab 2) to training time.

In data poisoning and backdoor attacks, the attacker changes the data. The trust assumption is that the model relies on uncorrupted data, or benign data.

2. Objectives

By the end of this lab, you should be able to:

distinguish availability, targeted integrity, clean-label, and backdoor poisoning attacks
understand the implementation of a poisoning attack against a linear SVM.
explain why optimized poisoning is different from simply inserting a mislabeled point;
explain why high clean accuracy does not rule out a backdoor;
measure the difference between clean accuracy and attack success rate (ASR);
connect these small-scale experiments to web-scale and foundation-model data supply chains.
identify the trust assumptions that each attack breaks.

The recurring questions across this lab that we will answer:

What can the attacker control?
What does the defender assume is trustworthy?
What behavior is the attacker trying to create?
What metric would reveal that behavior?
What evidence of poisoning might survive in the data, representation, or model?
Does the proposed defense cover one known attack, or does it reason about a broader class of adversaries?

3. Attack Model

In Lab 2, we focused on inference-time adversarial examples. We had a model that was already trained. The attacker then changed an input presented to a fixed model.

In this lab, the attacker moves up the ML pipeline.

\[\text{data source} \rightarrow \text{training set} \rightarrow \text{training algorithm} \rightarrow \text{model} \rightarrow \text{prediction}\]

In Lab 2, the attacker primarily acted near the prediction end of this pipeline. In this lab, the attacker acts on the training data or training process so that the model itself changes.

Security Goals	Lab 2: inference-time attack	Lab 3: training-time attack
Attack Objective	Untargeted (misclassify output) and Targeted attacks (specify the incorrect class that the model should predict.)	Degrade the learned model, implant a backdoor, or manipulate a particular future prediction
Security Triad Effected (CIA)	Untargeted attacks (misclassification)→ Availability and Targeted attacks → Integrity	Poisoning, Compromising the Supply Chain → Integrity and Availability
Attacker’s Insight	Small perturbation to the test input can cause large loss, across high-dimensional input	High clean accuracy obfuscates malicious behavior learned during training
Attack Entry Point	After training	Before or during training
Attacker Controlled Data:	Test-time inputs	Training examples, labels, triggers, or the data supply chain
Changes to the underlying model:	No	Yes

Security Goals

Lab 2: inference-time attack

Lab 3: training-time attack

Attack Objective

Untargeted (misclassify output) and Targeted attacks (specify the incorrect class that the model should predict.)

Degrade the learned model, implant a backdoor, or manipulate a particular future prediction

Security Triad Effected (CIA)

Untargeted attacks (misclassification)→ Availability and Targeted attacks → Integrity

Poisoning, Compromising the Supply Chain → Integrity and Availability

Attacker’s Insight

Small perturbation to the test input can cause large loss, across high-dimensional input

High clean accuracy obfuscates malicious behavior learned during training

Attack Entry Point

After training

Before or during training

Attacker Controlled Data:

Test-time inputs

Training examples, labels, triggers, or the data supply chain

Changes to the underlying model:

Yes

4. Overview of the papers this week

Paper	Contribution	High-level Takeaways
Biggio, Nelson & Laskov — Poisoning Attacks against Support Vector Machines	Central classical attack of this lab	Treat the poison point as an optimization variable to compromise model availability through retraining the model.
Steinhardt, Koh & Liang — Certified Defenses for Data Poisoning Attacks	Defence against Poisoning	Reason about worst-case poisoning that effects model availability.
Gu, Dolan-Gavitt & Garg — BadNets	Backdoor attack	A model can remain accurate on ordinary inputs while learning attacker-controlled behavior triggered by a pattern.
Tran, Li & Madry — Spectral Signatures in Backdoor Attacks	Representation-based defense	Backdoor poisons can leave a statistical trace in hidden representations. ////
Wang et al. — Neural Cleanse	Trigger-reconstruction defense	Search for unusually small triggers that force many inputs into one target class. ////
Shafahi et al. — Poison Frogs!	Clean-label targeted poisoning	The poison can keep its correct human label while colliding with a target in learned feature space.
Carlini et al. — Poisoning Web-Scale Training Datasets is Practical	Data-supply-chain attack	Web-scale datasets inherit security weaknesses from mutable web content and predictable collection pipelines.
Souly et al. — Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples	LLM pretraining poisoning	Poisoning budget is a constant rather than scaling as a fixed percentage of the training corpus, irrespective of model size.
Shan et al. — Nightshade	Prompt-specific generative-model poisoning	Poisoning can target a concept or prompt rather than merely a classifier label.

Paper

Contribution

High-level Takeaways

Biggio, Nelson & Laskov — Poisoning Attacks against Support Vector Machines

Central classical attack of this lab

Treat the poison point as an optimization variable to compromise model availability through retraining the model.

Steinhardt, Koh & Liang — Certified Defenses for Data Poisoning Attacks

Defence against Poisoning

Reason about worst-case poisoning that effects model availability.

Gu, Dolan-Gavitt & Garg — BadNets

Backdoor attack

A model can remain accurate on ordinary inputs while learning attacker-controlled behavior triggered by a pattern.

Tran, Li & Madry — Spectral Signatures in Backdoor Attacks

Representation-based defense

Backdoor poisons can leave a statistical trace in hidden representations.

////

Wang et al. — Neural Cleanse

Trigger-reconstruction defense

Search for unusually small triggers that force many inputs into one target class. ////

Shafahi et al. — Poison Frogs!

Clean-label targeted poisoning

The poison can keep its correct human label while colliding with a target in learned feature space.

Carlini et al. — Poisoning Web-Scale Training Datasets is Practical

Data-supply-chain attack

Web-scale datasets inherit security weaknesses from mutable web content and predictable collection pipelines.

Souly et al. — Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples

LLM pretraining poisoning

Poisoning budget is a constant rather than scaling as a fixed percentage of the training corpus, irrespective of model size.

Shan et al. — Nightshade

Prompt-specific generative-model poisoning

Poisoning can target a concept or prompt rather than merely a classifier label.

5. Part 1: Optimization-based Availability Poisoning

5.1. The security question

In this attack, we assume an attacker can contribute a training example. A naïve attacker might add a random poisoned data point (i.e., mislabel data) and hope it causes damage.

In this paper, we address the following question:

\[\boxed{\text{Where should the poisoned data point be placed to maximally hurt the model after retraining?}}\]

The attacker is not optimizing a test input against a fixed model. The attacker is optimizing a training point against a model that will be retrained after every change to that point.

5.2. Structure of the attack

Let $x_c$ be the poisoning point. The outer attacker wants to maximize validation loss:

\[x_c^* = \arg\max_{x_c \in \mathcal{X}} L_{\mathrm{val}}\left(\theta^*(x_c)\right).\]

But the model parameters $\theta^*(x_c)$ are also the result of training:

\[\theta^*(x_c) = \arg\min_{\theta} L_{\mathrm{train}} \left( \theta; D_{\mathrm{train}}\cup\{(x_c,y_c)\} \right).\]

Resulting in a dependency chain of the following form:

\[x_c \rightarrow \text{training data} \rightarrow \text{retraining} \rightarrow \theta^* \rightarrow L_{\mathrm{val}}.\]

The attacker must now reason about how the learning algorithm adapts as poisoned data forms a part of the training dataset.

5.3. Toy Gaussian Example

The original Biggio artificial experiment uses overlapping Gaussian classes. In this lab, we will use a cleaner setup of a binary classifier where all negative-class data points begin on one side of the clean boundary and all positive-class data points points begin on the other.

This is a simplistic adaptation so we can see the effect of one poisoned sample. The paper does not use perfectly separable data.

Our Goal: After poisoning the trianing data, verify if the the decision boundary rotates or validation performance fails, as a result of adding poisoned samples.

5.3.1. Adding a poisoned sample using a random label flip

The attack begins with a random point selected from the positive class. This poisoned sample is inserted into the training set with the opposite label.

Conceptually:

\[(x,+1) \quad\longrightarrow\quad \text{add a clone }(x,-1).\]

This first point is not supposed to be optimally damaging. It is the starting point for the attack.

This random label flop gives us a baseline before finding an optimal poisoned data point.

5.3.2. Finding an optimal poisoned data point

The Attacker’s Goal: The optimization uses a continuous validation loss. For the SVM experiments, we use the hinge loss, which intuitively answers the following question: what is the loss penalty across the signed margin [stem]:$y_k f(x_k) across all samples in the dataset?

\[L_{\mathrm{hinge}} = \frac{1}{m} \sum_{k=1}^{m} \max\left( 0,\, 1-y_k f(x_k) \right).\]

In the lab, we will report both: * 0/1 classification error. * the hing loss

Do not expect those quantities to move in lockstep. A point can move much closer to the decision boundary while remaining correctly classified. Hinge loss can therefore rise before 0/1 error rises.

5.3.3. Limitations with a linear SVM

In the linear experiment, the optimized poison can keep moving outward until it reaches the clipping boundary (i.e. an explicitly bounded feature region).

The intuition for this behavior is as follows: For a linear classifier, moving a mislabeled example farther into territory associated with the opposite class can continue to exert leverage on the learned separator.

We see this in the Biggio et al. paper as well: The validation-error surface for their linear case is effectively unbounded and their experiment imposes a bound on the allowable poison location. We apply the same principle in the lab with an explicit bounded feature region.

The final poissoned data point should not be interpreted as the best or most optimal data point. Rather, a more accurate interpreation would be: Within the allowed region, the local ascent direction tries to maximize the hinge loss, pushing the poisoned data point to the edge of the bounded feature region.

5.4. The Loss Landscape

The notebook reconstructs a coarse two-dimensional surface by imagining many possible poison locations. For each candidate location, the model is retrained and its validation loss is measured.

This gives us a function of poison location, and rhe attack trajectory can then be laid over this surface. This lets you visualize the poisoning attack as search problem.

\[(x_{c1},x_{c2}) \longmapsto L_{\mathrm{val}}.\]

While examining the plot, focus on:

where the initial mislabeled point starts;
whether each step moves toward higher validation loss;
whether the surface suggests an interior peak or an outward slope;
whether the final point is stopped by convergence or by the imposed attack domain; and
how the hinge-loss surface differs from the 0/1-error surface.

5.5. Running an Radial Basis Function SVM Attack

Since the attack is not limited to linear SVMs, we will also try out an RBF example to contrast it with the results from our linear model.

For an RBF model, a poison that moves very far away can become less similar to almost everything else. In Biggio et al.'s experiment, we see the following contrast: the linear attack is driven toward the clippoing boundary, while in the the RBF landscape, we see an interior local maximum.

In lab, we will implement both attacks from the same mislabeled point, so we can contrast the two results.

6. Part 2 — Poisoning Defenses

A natural response to poisoning is removing suspicious training points before fitting the model. However, a knowledgeable attacker can design poisons specifically to survive the filter.

Steinhardt, Koh, and Liang address the following question: How bad can the worst attack be if it must survive this defense?

6.1. Data Sanitization Framework

In lab we will explore two geometric ideas associated with the paper’s sanitization framework.

A sphere constraint accepts points sufficiently close to the centroid of their claimed class:

\[\lVert x-\mu_y\rVert_2 \le r_y.\]

A slab constraint restricts how far the point can project along a direction related to the class centroids:

\[\left| \left\langle x-\mu_y,\, \mu_y-\mu_{-y} \right\rangle \right| \le s_y.\]

The exact certified-defense framework in the paper is more complex than our simplified geometric demonstration. Our goal in this lab is to use sphere/slab filtering to build intuition about a feasible set of data points that contain clean data and potentially some poisoned data samples:

The key intuition here is the following:The attacker is capable of injecting poisoned smaples even when constrained to look statistically plausible.

6.2. Takeaways

In the lab, notice the following as the sanitization region becomes tighter, and think about both sides of the tradeoff:

Does a tighter regime remove the poisoned data?
How does it effect legitimate but outlier data that is clean?
Could a stronger attacker deliberately optimize inside the accepted region?
Why is an upper bound on a worst-case attack more informative than testing on one known attack?]

7. Part 3: Backdoor Attacks

In backdoor attacks, the attacker’s goals are fundamentally different from attacking the availability of a model. In the BadNets paper, authors introduces a different objective: the goal is to create a model that behaves normally unless a trigger is present.

A backdoored model is designed to satisfy both high clean accuracy and high success rate, of a trigger attack.

This makes backdoors especially dangerous as ordinary validation will miss these attacks.

7.1. The mechanism

The following represents a basic backdoor training strategy:

choose a trigger pattern;
add that trigger to some training examples;
assign those triggered examples to an attacker-chosen target class;
proceed with training.

The goal is for the model to then discover the following; When a trigger is present, the model predicts only the target attack label. The trigger therefore, becomes a learned feature.

The attacker does not need to alter the model after training. In this scenario, the malicious behavior has already been learned by the model. .

7.2. Evaluating Attack Success

In lab, we will first measure Clean accuracy — which measures normal behavior (without an attack trigger).

Next, we will measure the Attack success rate (ASR) i.e., what fraction of triggered inputs are sent to the attacker’s target class:

\[\mathrm{ASR} = \frac{ \#\{\text{triggered inputs predicted as target}\} }{ \#\{\text{triggered inputs tested}\} }.\]

A model can have both high clean accuracy and high ASR — in fact, this is precisely what a successful backdoor attacker wants!

7.3. Effect on the Model Supply Chain

BadNets frames outsourced training and pretrained models as a supply-chain risk.

The trust failure is the following: A model artifact can pass ordinary accuracy checks and still contain hidden behavior.

8. Part 4 — Spectral Signatures: Defenses against Backdoors

A successful backdoor attack may be almost invisible from clean test accuracy. Spectral Signatures asks a different question:

Did the poisoned examples leave a detectable statistical structure in the representation learned by the network?

Let $h_i$ denote a hidden-layer representation of example $i$. After centering the representations, the defense looks for an unusually strong direction of variation. Conceptually, the examples are scored by their projection onto a dominant singular direction. Here, training examples with unusually large scores become suspects.

\[s_i \propto \left( v_1^T(h_i-\bar h) \right)^2.\]

where:

$i$ = index identifying one training example
$s_i$ = suspiciousness score for training example $i$
$v_1$ = dominant singular direction
$h_i$ = hidden representation of example $i$
$\bar{h}$ = average hidden representation
$v_1^T(h_i-\bar{h})$ = how strongly that difference lies along the dominant direction

8.1. Understanding spectral signatures

A backdoor forces the network to represent the following:

ordinary members of the class;
poisoned examples sharing a trigger-induced structure.

That mixture can create a detectable direction in representation space. The defense therefore does not search directly for a particular pixel trigger. It searches for evidence that the learned geometry of the class is abnormal.

8.2. Takeaways

In lab, we will rank suspicious examples, look for the following:

Where are the poisoned examples located? concentrated near the top/bottom or dispersed in the ranking?
What is the tradeoff between precision and recall when choosing how many suspected samples to remove?
After the removal of these samples and retraining, what happens to both clean accuracy and ASR?

A defense that removes the backdoor attack can also effect the clean performance. If it significantly impacts clean performance then is the defence still appropriate?

9. Part 5: Targeted Poisoning

So far, we’ve looked at poisoning as an attack on availability, and backdoor attacks that can be triggered with specific poisoned instances. In this section, we will look at a complementary attack: clean-label poisoning.

Here we will assume that an adversary cannot control the labeling, and therefore cannot mislable a poisoned data point. Instead, the attacker poisons a image, such that from a human’s perspective, the image is incorrectly labeled. Here, the attack is condcucted in the model’s representation space.

9.1. Feature collision for Targeted Poisoning

Let $x_b$ be a legitimate base-class image and $x_t$ the particular target the attacker eventually wants to influence.

A simplified feature-collision objective is:

\[\min_{x_p} \; \lVert \phi(x_p)-\phi(x_t) \rVert_2^2 + \beta \lVert x_p-x_b \rVert_2^2.\]

In the above equation, we are trying to satisfy two goals:

keep the poison visually close to the legitimate base image; and,
move the poison’s learned representation toward the target.

As a result, an image can look correctly labeled to a person while lying on the classification boundary for the model.

9.2. Impact on real-world data:

Clean-label attacks fit an important modern threat model: The attacker may control web content without controlling the curator’s label.

This makes the attack relevant to data pipelines and models that ingest images or text from public sources.

10. Metrics used in this lab

Throughout the lab, we’ve used different metrics of “attack success". Here’s a "cheat sheet" of all of the metrics so far.

Metric	Interpretation
Validation/test error	Availability damage: detoriation in ordinary prediction performance.
Hinge loss	A continuous margin-sensitive objective useful for optimizing SVM poisoning.
Clean accuracy	Whether ordinary untriggered inputs still behave normally.
Attack success rate (ASR)	Whether triggered inputs reliably activate the attacker’s desired behavior.
Poison-detection precision	Of the examples flagged as suspicious, how many are actually poisoned?
Poison-detection recall	Of the poisoned examples present, how many did the defense find?

Metric

Interpretation

Validation/test error

Availability damage: detoriation in ordinary prediction performance.

Hinge loss

A continuous margin-sensitive objective useful for optimizing SVM poisoning.

Clean accuracy

Whether ordinary untriggered inputs still behave normally.

Attack success rate (ASR)

Whether triggered inputs reliably activate the attacker’s desired behavior.

Poison-detection precision

Of the examples flagged as suspicious, how many are actually poisoned?

Poison-detection recall

Of the poisoned examples present, how many did the defense find?

A security evaluation should usually report more than one metric. A defense can lower ASR while damaging clean accuracy; a poisoning attack can increase hinge loss before 0/1 error changes; a detector can obtain high recall by accusing too many clean samples.