LAB DUE DATE: TUESDAY, September 15nd, 11.59PM

This lab builds on the input-gradient computation from Lab 1. You will construct adversarial examples with the Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD), measure the attack strength and relate it’s efficacy to the perturbation budget, and evaluate a basic defense through adversarial training.

All experiments in this lab are white-box experiments. Do not run these attacks against systems, models, or data that you do not own or have explicit permission to test.

Lab objectives

By the end of this lab, you should be able to:

  1. Explain why a model can be highly accurate on clean inputs and still be vulnerable to carefully chosen, small perturbations.

  2. Implement Fast Gradient Sign Method (FGSM) as a one-step, gradient-sign attack.

  3. Implement Projected Gradient Descent (PGD) as an iterative attack with projection into an $L_infinity$ perturbation region.

  4. Compare attack success across perturbation budgets, datasets, and numbers of PGD steps.

  5. Distinguish targeted attacks from untargeted attacks.

  6. Measure transferability across two different model architectures.

  7. Compare standard and adversarially trained models using clean and robust accuracy.

  8. Interpret results while reporting the threat model, preprocessing, evaluation subset, and attack parameters.

Recap of Lab 1

In Lab 1 we developed a supervised-learning pipeline. In Lab 2, we will follow the same steps of taking a dataset, preprocessing the data, training a classifier, evaluating accuracy, and inspecting how the loss changes as inputs move through the model.

Lab 2 uses the same forward-and-backward computation, but uses an adversarial lens:

Lab 1 Goals

What does the model’s loss gradient reveal about the input?

Lab 2 Goals

Can we leverage the model’s loss gradient to construct an input that causes the model to fail?

In this lab, the model parameters remain fixed for the attack. In Lab 1, we computed the gradients with respect to model parameters and updated the weights of our model, to improve accuracy. In Lab 2, we will compute gradients with respect to the image and update the image instead.

Attack Class

FGSM and PGD are white-box, gradient-based, inference-time evasion attacks that generate adversarial examples under a \(L_{\infty}\) perturbation constraint.

FGSM performs one gradient-sign step, while PGD performs multiple projected gradient steps.

  • FGSM’s one gradient-sign step:

    \[x_{\mathrm{adv}} = x + \epsilon \operatorname{sign} \left( \nabla_x J(\theta, x, y) \right)\]

  • PGD performs multiple gradient steps and projects the result back into the allowed perturbation region:

    \[x_{t+1} = \Pi_{\mathcal{B}_{\infty}(x,\epsilon)} \left( x_t + \alpha \operatorname{sign} \left( \nabla_{x_t} J(\theta, x_t, y) \right) \right)\]

Threat Model

FGSM and PGD operate under an \(L_{\infty}\) threat model. Let \(x\) denote the original input and \(x_{\mathrm{adv}}\) denote the adversarial input. The attack must satisfy:

\[\left\lVert x_{\mathrm{adv}} - x \right\rVert_{\infty} \leq \epsilon\]

This constraint means that no individual input coordinate—such as a pixel value or color-channel value—may change by more than epsilon. The constraint limits the maximum coordinate-wise perturbation, but it does not guarantee perceptual similarity. It also does not limit the total amount of change across all pixels.

Understanding the attacks

As we discussed in class, FGSM and PGD is to view every possible image as a point in a high-dimensional input space. A classifier partitions the input space into decision regions: one region may correspond to cat, another to dog, and another to airplane. The boundaries between these regions are the model’s decision boundaries.

A clean image is a point in this input space. If the model classifies the image correctly, the point lies inside the decision region associated with its true class. An adversarial example is created by moving the point a small distance so that it crosses a decision boundary and enters a region associated with an incorrect class.

Rather than looking for an arbitrary nearby point, the attack uses the gradient of the loss with respect to the input image to identify a particularly effective direction of movement. The gradient describes how the loss changes for small changes to the input coordinates:

\[\nabla_x J(\theta, x, y)\]

For an untargeted attack, the gradient indicates how to change the input so that the loss for the correct class increases. The model parameters remain fixed; the attack changes only the input.

This view is consistent with the original FGSM analysis, which emphasized the effect of small, coordinated perturbations in high-dimensional input spaces rather than treating the attack as random noise References.

Attack 1: Fast Gradient Sign Method

Goal

FGSM is a fast, one-step, untargeted attack. The goal of this attack is to increase the loss assigned to the correct label while staying inside the allowed perturbation budget.

For model parameters \(\theta\), input \(x\), true label \(y\), and loss \(J\), the attack is:

\[x_{\mathrm{adv}} = x + \epsilon \operatorname{sign} \left( \nabla_x J(\theta, x, y) \right)\]

The gradient has one value for every input coordinate. FGSM looks only at the sign of each value:

  • A positive sign means increase that coordinate.

  • A negative sign means decrease that coordinate.

  • The magnitude of the gradient is discarded.

This makes FGSM easy to understand: it nudges many coordinates in the direction that is locally most damaging to the model. Each individual change is small, but the changes are coordinated. A useful analogy is pushing a large object: one person pushing may have little effect, but many people pushing in the same direction can move it substantially.

Why does the gradient sign matter?

Imagine that the model receives a small grayscale image with four relevant input coordinates:

x = [0.20, 0.50, 0.70, 0.40]

Suppose the gradient of the loss with respect to those coordinates is:

gradient = [-0.8, 0.02, 4.1, -0.3]

The gradient signs are:

sign(gradient) = [-1, +1, +1, -1]

For an FGSM perturbation budget of \(\epsilon = 0.1\), the attack changes each coordinate by \(\pm\epsilon\) according to its sign:

perturbation = [-0.1, +0.1, +0.1, -0.1]

The resulting adversarial input is therefore:

x_adv = x + perturbation
      = [0.10, 0.60, 0.80, 0.30]

The model does not receive random noise. Each coordinate is changed in the direction that increases the loss for the correct class. The individual changes are small, but together they may move the input across the model’s decision boundary.

The values are clipped if necessary so that every coordinate remains within the valid input range.

Understanding the code

images = tf.convert_to_tensor(images, dtype=tf.float32)

with tf.GradientTape() as tape:
    tape.watch(images)
    logits = model(images, training=False)
    loss = keras.losses.sparse_categorical_crossentropy(
        labels, logits, from_logits=True
    )

grad = tape.gradient(loss, images)
signed_grad = tf.sign(grad)
adv_images = images + epsilon * signed_grad
adv_images = tf.clip_by_value(adv_images, x_min, x_max)

The code has four conceptual stages:

  1. Compute the model’s loss on the current images.

  2. Differentiate the loss with respect to the images.

  3. Move in the sign-gradient direction to increase the loss.

  4. Clip the result to a valid image range.

Attack 2: Projected Gradient Descent

Goal

Projected Gradient Descent, or PGD, is an iterative extension of FGSM. Instead of taking one large gradient-sign step, PGD takes several smaller steps and recomputes the gradient after each step.

The goal is to find an effective adversarial input within the same perturbation region used by FGSM. After every update, PGD projects the current input back into the allowed region around the original image. This keeps the attack within the specified threat model.

For model parameters \(\theta\), input \(x\), true label \(y\), loss \(J\), step size \(\alpha\), and perturbation budget \(\epsilon\), the untargeted PGD update is:

\[x_{t+1} = \Pi_{\mathcal{B}_{\infty}(x,\epsilon)} \left( x_t + \alpha \operatorname{sign} \left( \nabla_{x_t} J(\theta, x_t, y) \right) \right)\]

Here:

  • \(x_t\) is the current adversarial input.

  • \(\alpha\) controls the size of each update.

  • \(\epsilon\) controls the maximum distance from the original input.

  • \(\Pi_{\mathcal{B}_{\infty}(x,\epsilon)}\) projects the updated input back into the allowed \(L_{\infty}\) region.

  • The gradient is recomputed at every iteration.

Unlike FGSM, which uses the gradient at only the original input, PGD continually updates its direction as the input changes. This allows the attack to search more carefully within the allowed perturbation region.

Projection onto the perturbation set

PGD does not allow each gradient update to move independently without restriction. Instead, after every update, it projects the current adversarial input back onto the set of inputs permitted by the threat model.

For an original input coordinate \(x_i = 0.50\) and an \(L_{\infty}\) perturbation budget of \(\epsilon = 0.10\), the permitted values are:

\[x_i - \epsilon \leq x_i^{\mathrm{adv}} \leq x_i + \epsilon\]

Substituting the values from this example gives:

\[0.40 \leq x_i^{\mathrm{adv}} \leq 0.60\]

Suppose a gradient update produces the value 0.63. This value lies outside the permitted perturbation set because it differs from the original coordinate by 0.13, which is greater than \(\epsilon = 0.10\). Projection maps the value back to the nearest permitted point, 0.60.

For an \(L_{\infty}\) constraint, this projection is implemented by clipping the perturbation relative to the original input:

perturbation = tf.clip_by_value(
    adv_images - original_images,
    -epsilon,
    epsilon
)

adv_images = original_images + perturbation

The order of operations is important:

  1. Compute the current perturbation by subtracting the original input from the current adversarial input.

  2. Clip every coordinate of that perturbation to the interval \([-\epsilon, \epsilon\)].

  3. Add the clipped perturbation back to the original input.

This guarantees:

\[\left\lVert x_{\mathrm{adv}} - x \right\rVert_{\infty} \leq \epsilon\]

Projection should be understood as a constraint-enforcement operation. It does not choose a new attack direction and it does not recompute the gradient. It simply prevents the current adversarial input from leaving the perturbation set defined around the original input.

Projection versus valid-range clipping

PGD commonly uses two different clipping operations, and they enforce different constraints:

Operation Constraint enforced Example

Perturbation projection

Limits the distance between the adversarial input and the original input.

Ensures that each coordinate changes by at most \(\epsilon\).

Valid-range clipping

Ensures that the adversarial input remains a valid normalized image.

Prevents normalized pixel values from falling below x_min or above x_max.

The corresponding valid-range operation is:

adv_images = tf.clip_by_value(
    adv_images,
    x_min,
    x_max
)

These operations are complementary. An input can satisfy the \(L_{\infty}\) perturbation constraint and still fall outside the valid image range near a boundary. Conversely, an image can lie within the valid range while being more than \(\epsilon\) away from the original input.

A correct PGD implementation therefore applies both constraints after each gradient update:

# 1. Take a gradient-directed step.
adv_images = (
    adv_images
    + alpha * tf.sign(gradient)
)

# 2. Enforce the perturbation budget relative to
#    the original input.
perturbation = tf.clip_by_value(
    adv_images - original_images,
    -epsilon,
    epsilon
)
adv_images = original_images + perturbation

# 3. Enforce the valid normalized input range.
adv_images = tf.clip_by_value(
    adv_images,
    x_min,
    x_max
)

The first clipping operation enforces the attacker’s allowed modification. The second preserves the data representation expected by the model. Together, they keep PGD inside both the specified threat model and the valid input domain. This projected constrained-update perspective is central to the PGD formulation used for adversarial robustness evaluation.

Random initialization

With random_init=True, PGD begins from a random point inside the allowed \(L_{\infty}\) region rather than starting exactly at the clean input.

noise = tf.random.uniform(
    tf.shape(images),
    minval=-epsilon,
    maxval=epsilon
)

adv_images = images + noise
adv_images = tf.clip_by_value(
    adv_images,
    x_min,
    x_max
)

Random initialization means that two PGD runs can begin at different points within the same perturbation region. The subsequent gradient steps may therefore follow different paths. Set the random seed when you need reproducible results.

For a controlled comparison with FGSM, use:

random_init = False
n_steps = 1

Under these settings, PGD behaves much like a one-step gradient-sign attack. With multiple steps, PGD can repeatedly adjust its direction as it searches for a more effective adversarial input.

Understanding the code

The code has six conceptual stages:

  1. Save the original input because every projection must be relative to it.

  2. Optionally initialize inside the perturbation region.

  3. Compute the loss and its gradient with respect to the current adversarial input.

  4. Take a small gradient-sign step.

  5. Project the perturbation back into the allowed \(L_{\infty}\) region.

  6. Clip the image to the valid normalized input range.

FGSM and PGD

FGSM and PGD use the same gradient signal but search differently:

Property FGSM PGD

Number of updates

One

Multiple

Gradient evaluation

Once at the original input

Repeated at every iteration

Step size

Usually the full \(\epsilon\)

Usually a smaller \(\alpha\)

Constraint handling

Clip the final adversarial input

Project after every update

Runtime

Fast

More expensive

Search behavior

One locally informed move

Iterative search within the allowed region

PGD is therefore not simply “FGSM repeated” without modification. The projection step is essential: it ensures that every intermediate and final input remains within the same perturbation budget. This projected, iterative view is the constrained optimization perspective used in the PGD formulation of Madry et al. References.

Goal

PGD is an iterative extension of FGSM. Its goal is to search more effectively within the same perturbation region by taking several small gradient steps and projecting back into the allowed region after every step.

\[x_(t+1) = Pi_(B_infinity(x, epsilon))(x_t + alpha sign(grad_(x_t) J(theta, x_t, y)))\]

Here, alpha is the step size, N is the number of steps, and Pi is projection. For an $L_infinity$ threat model, projection is coordinate-wise clipping around the original image:

perturbation = tf.clip_by_value(
    adv_images - original_images,
    -epsilon,
    epsilon
)
adv_images = original_images + perturbation

Without this projection, repeated steps could accumulate a perturbation much larger than epsilon, so the experiment would no longer be comparing attacks under the same threat model.

Random initialization

With random_init=True, PGD begins at a random point inside the $L_infinity$ ball rather than exactly at the clean image. This can expose different local paths through the loss surface. It also means that reproducibility depends on the random seed.

noise = tf.random.uniform(
    tf.shape(images),
    minval=-epsilon,
    maxval=epsilon
)
adv_images = images + noise

For a controlled comparison with FGSM, use random_init=False and n_steps=1. Under those settings, PGD is essentially a one-step signed-gradient attack.

FGSM versus PGD

Property FGSM PGD Interpretation

Gradient steps

One

Many

PGD repeatedly reassesses the local loss direction.

Speed

Fast

Slower

PGD requires a forward/backward computation for every step.

Constraint

Clip once

Project after every step

Both should use the same threat model for a fair comparison.

Search behavior

One local move

Iterative constrained search

PGD can find a more effective point at the same epsilon.

Typical use

Baseline or quick attack

Stronger evaluation or adversarial training

The right choice depends on runtime and threat model.

A useful example from the notebook is CIFAR-10 at epsilon = 0.03: FGSM and PGD may produce very different success rates because PGD has multiple opportunities to adjust its direction. The exact values are checkpoint- and subset-dependent; you should report the values produced by your own run rather than treating the notebook output as universal.

Targeted and untargeted attacks

FGSM and PGD describe how an adversarial example is generated: FGSM takes one gradient-sign step, while PGD takes multiple projected steps. Adversarially, the two attacks can provide different outcomes.

This gives us two attack objectives:

  • An untargeted attack requires only that the model predict any incorrect class.

  • A targeted attack requires the model to predict one specific class chosen by the attacker.

Untargeted attacks

An untargeted attack does not specify the incorrect class in advance. Its objective is simply to move the input out of the decision region for the true label:

\[f(x_{\mathrm{adv}}) \neq y\]

For an MNIST image whose true label is 3, the attack succeeds if the model predicts 0, 1, 2, 4, 5, or any other incorrect class. The attacker does not control which incorrect class the model selects.

For the loss-based formulation used in this lab, the attack maximizes the loss for the true label:

\[x_{\mathrm{adv}} = x + \epsilon \operatorname{sign} \left( \nabla_x J(\theta, x, y) \right)\]

The positive sign moves the input in the direction that makes the true class less consistent with the model’s output. Once the predicted class changes, the untargeted attack has achieved its objective.

Targeted attacks

A targeted attack specifies the incorrect class that the model should predict. Its success condition is:

\[f(x_{\mathrm{adv}}) = y_{\mathrm{target}}\]

As above, if the the original MNIST image is a handwritten 3 and the attacker chooses 4 as the target. A targeted attack succeeds only if the perturbed image is classified as 4. A prediction of 0 or 7 would still be incorrect, but it would not satisfy the objective of a targeted attack.

To encourage the model to predict the target class, the attack minimizes the loss for the target label. This reverses the update direction:

\[x_{\mathrm{adv}} = x - \epsilon \operatorname{sign} \left( \nabla_x J(\theta, x, y_{\mathrm{target}}) \right)\]

The negative sign is important. The gradient of the target loss points toward changes that increase the loss for the target class. Moving in the opposite direction decreases that loss and encourages the model to assign greater probability to the target class.

A concrete example

Suppose a classifier receives an image whose true class is 3:

true_label = 3
target_label = 4

An untargeted attack succeeds if the model produces any prediction other than 3:

predicted_label != true_label

For example, each of the following predictions would count as an untargeted success:

predicted_label = 0
predicted_label = 4
predicted_label = 8

A targeted attack with target class 4 has a stricter success condition:

predicted_label == target_label

The following outcomes would therefore be interpreted differently:

True label Model prediction Attack interpretation

3

3

Neither attack succeeded.

3

8

Untargeted attack succeeded; targeted attack failed.

3

4

Untargeted attack succeeded and targeted attack succeeded.

3

0

Untargeted attack succeeded; targeted attack failed.

This illustrates the key distinction: an untargeted attack only needs to cross any incorrect decision boundary, whereas a targeted attack must reach the particular decision region associated with \(y_{\mathrm{target}}\).

Target selection in the notebook

For the MNIST experiment, the notebook constructs a simple target by choosing the next digit cyclically:

target_labels = (true_labels + 1) % 10

This produces the following mapping:

True label Target label

0

1

1

2

2

3

…​

…​

9

0

This rule is convenient because it gives every example a target without requiring an additional target-selection procedure. It is not a property of targeted attacks in general, however. Target selection can affect difficulty: some target classes may be closer to the original class in the model’s decision space than others.

When reporting targeted-attack results, record:

  • The target-selection rule.

  • The target label for each example, or the target distribution.

  • The attack method, such as FGSM or PGD.

  • The perturbation budget \(\epsilon\).

  • For PGD, the step size \(\alpha\) and number of steps.

  • The targeted success rate.

Targeted versus untargeted success

The two attack types require different evaluation metrics:

\[\text{Untargeted success} = \mathbb{1} \left[ f(x_{\mathrm{adv}}) \neq y \right]\]
\[\text{Targeted success} = \mathbb{1} \left[ f(x_{\mathrm{adv}}) = y_{\mathrm{target}} \right]\]

For an untargeted attack, measure whether the adversarial prediction differs from the true label. For a targeted attack, measure whether the prediction equals the specified target label. Do not use the untargeted criterion to claim targeted success.

In general, targeted attacks are more restrictive because they require the attack to reach one particular incorrect class rather than any incorrect class. The empirical difference depends on the model, dataset, target-selection rule, perturbation budget, and attack parameters. The notebook should therefore report both rates rather than assuming that one fixed relationship holds in every experiment.

Robustness Evaluation

We can evaluate the model in two stages:

  1. First, measure its accuracy on the original, unmodified test images. This establishes the model’s clean accuracy.

  2. Next, generate adversarial images using a specified attack and measure whether the model’s predictions remain correct on those same examples.

The first measurement describes the model’s performance without any adversarial perturbation. The second describes its accuracy under the specified attack, and perturbation. Finally, we measure the success rate of an attack only on examples the model classified correctly before the attack.

Accuracy of the model without attacks

First, evaluate the model on the unmodified inputs:

clean_logits = mnist_model(
    mnist_eval_x,
    training=False
)

clean_preds = tf.argmax(
    clean_logits,
    axis=1
).numpy()

clean_correct = clean_preds == mnist_eval_y

clean_accuracy = np.mean(clean_correct)
clean_error_rate = np.mean(~clean_correct)

print("Clean accuracy:", clean_accuracy)
print("Clean error rate:", clean_error_rate)
print("Initially correct:", np.sum(clean_correct))
print("Total examples:", len(mnist_eval_y))

The clean accuracy is:

\[A_{\mathrm{clean}} = \frac{ \#\{\text{correct clean predictions}\} }{ \#\{\text{evaluation examples}\} }\]

The Boolean array clean_correct identifies the examples that the model classified correctly before the attack. We will use this same subset when measuring untargeted attack success.

For example, suppose the model correctly classifies 90 of 100 clean images. The clean accuracy is 90 percent, and these 90 correct images will be used to determine how effective the attacks are.

Sanity checks before interpreting results

Before interpreting an attack-success curve, verify that the attack produced a valid perturbation.

delta = adv_images - clean_images

max_change = np.max(
    np.abs(delta)
)

mean_change = np.mean(
    np.abs(delta)
)

print("maximum coordinate change:", max_change)
print("mean absolute change:", mean_change)

assert max_change <= epsilon + 1e-6

For an \(L_{\infty}\) attack, the largest coordinate-wise change must satisfy:

\[\left\lVert x_{\mathrm{adv}} - x \right\rVert_{\infty} \leq \epsilon\]

This assertion allows a small tolerance for floating-point round-off errors. In your code, you also want to check that the gradient grad is not None and that grad.shape matches`images.shape. Finally, ensure that the adversarial images use the same preprocessing and valid input bounds as the clean images.

For PGD, the perturbation must be projected relative to the original images after every update. If these checks pass, then the attack-success curve is measuring the attack success rate.

A larger epsilon gives the attack a larger region in which to modify the input. The success of an attack will increase as the perturbation budget increases, but the curve need not be perfectly smooth or strictly monotonic for a finite evaluation set.

If you see a curve you cannot quite explain you must ensure the following are true:

  • * The gradient is computed with respect to the input image and is not None.

  • The gradient has the same shape as the input batch, so every input coordinate receives a gradient value.

  • The adversarial images differ from the clean images.

  • The maximum coordinate change satisfies the epsilon bound.

  • The correct labels are being used.

  • Clean and adversarial inputs use the same preprocessing and normalization.

  • The model’s output convention agrees with the loss setting from_logits=True.

  • The attack-success metric includes examples that were already misclassified.

  • PGD projects the perturbation relative to the original clean image, not relative to the previous iteration.

  • Calid clipping bounds are expressed in the same normalized coordinates used during model training.

Only after these checks pass should the curve be interpreted as evidence of the model’s robustness.

Deliverables

  • Completed fgsm_attack implementation.

  • Completed pgd_attack implementation with configurable epsilon, alpha, number of steps, and random initialization.

  • FGSM and PGD epsilon-sweep results.

  • Perturbation grids for MNIST and CIFAR-10.

  • PGD convergence analysis.

  • Targeted-attack target-hit rate and untargeted success rate.

  • CNN-to-MLP transferability results.

  • Standard-versus-adversarially-trained clean and robust accuracy.

  • README_lab2_results.txt with parameters, metrics, and interpretation.

  • Completed discussion and reflection responses.

References

  • Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. "Explaining and Harnessing Adversarial Examples." arXiv:1412.6572, 2014. arXiv. This is the original FGSM paper and motivates the gradient-sign construction.

  • Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. "Towards Deep Learning Models Resistant to Adversarial Attacks." arXiv:1706.06083, 2017; ICLR 2018. arXiv. This paper develops the robust-optimization perspective and uses PGD as a first-order adversary.

  • TensorFlow. "Adversarial example using FGSM." TensorFlow tutorial. The tutorial documents the GradientTape, tape.watch(input_image), and FGSM pattern used here.

  • Joshuaclymer. Illustration of imperceptible adversarial perturbation. Wikimedia Commons, 2022. Own work. Image page. Licensed CC BY-SA 4.0. The image is included under that license and should retain attribution if reused or modified.