Recovering confidence from labels alone

In Part 2, we trained an overfit CNN on 5,000 CIFAR-10 images and ran the gap attack, guessing “member” whenever the model classifies a point correctly. That gave us a baseline score of 0.718, matching the theoretical 0.5 + gap/2.

This post will dive into the paper’s main idea. We want to produce a confidence-like signal out of a model that only reveals labels.

The idea

If we could see the model’s confidence scores, membership inference would be a problem of picking a threshold for the confidence, because the model would be overconfident on its training data.

But if we don’t have the raw confidence scores, how can we infer that signal? The paper proposes this solution. Instead of asking the model how sure it is about a label, we can indirectly find this out by asking how easily it misclassifies a copy through small perturbations. A point the model is deeply certain about will stay correctly classified through these nudges (rotations, translations), while points it hasn’t seen before (and is therefore not as confident about) slip into the wrong class more easily.

Robustness becomes a proxy for confidence.

Building it

The augmentations

For every point, we query the model 47 times: the original, 30 rotations from −15° to +15°, and 16 translations (1 to 4 pixels, in each of four directions). The original paper incorporates slightly different perturbations. It uses 3 rotations and translations that pick a single shift distance and cover every direction (including diagonals) at it. Our version sweeps four distances but only shifts up, down, left, and right. I decided to go wider, so that we could ask afterwards how many of these perturbations actually carry signal.But, whether the missing diagonals matter remains to be examined.

ROT_R, TRANS_D = 15, 4

augs = [("identity", None)]
augs += [("rotate", float(a)) for a in range(-ROT_R, ROT_R + 1) if a != 0]
for d in range(1, TRANS_D + 1):
    augs += [("translate", (d, 0)), ("translate", (-d, 0)),
             ("translate", (0, d)), ("translate", (0, -d))]

Then we run a correctness check for each augmentation. This gives us a bitmask of whether each copy is still classified correctly, producing a 47-bit vector.

@torch.no_grad()
def correctness_matrix(dataset):
    """-> bool array (n_points, n_augs): was each augmented copy classified correctly?"""
    cols = [[] for _ in augs]
    for xb, yb in DataLoader(dataset, batch_size=256):
        xb, yb = xb.to(device), yb.to(device)
        for j, (kind, param) in enumerate(augs):
            pred = model(apply_aug(xb, kind, param)).argmax(1)
            cols[j].append((pred == yb).cpu())
    return torch.stack([torch.cat(c) for c in cols], dim=1).numpy()

This gives us one row per point and one column per augmentation. X[i][j] is 1 if and only if point i, under augmentation j, was classified correctly.Note that this is correctness against the true label, not agreement with the model’s original prediction. For a point the model already gets wrong those are different questions, and nearly 44% of our non-members are in that bucket.

We reuse the same target model (the SmallCNN) and the same member/non-member split from Part 2, so the two attacks can be compared on identical ground. The first augmentation is the identity, which means column 0 of the feature matrix is equivalent to the gap-attack signal from Part 2, giving us a baseline.By asserting that it reproduces Part 2’s per-point results exactly, we ensure that we have loaded the correct model.

The attack classifier

Each point (image) is now a row of 47 ones and zeros. We want a classifier that maps that vector to member or non-member.This is analogous to picking a threshold for the confidence if we had access to confidence scores. We want to find where the boundary sits between member and non-member in terms of robustness.

We use logistic regression, which is a single neuron with no hidden layers.The paper uses a 2-layer MLP (Multi-Layer Perceptron) with LeakyReLU as the activation function.

It runs in three steps.

1. Weight each input, sum the results, and add a bias:

$z = b + \sum_{j=0}^{46} w_j x_j$

2. Apply the sigmoid, which maps $z$ into $(0,1)$ (mapping the score into a probability scale):

$\sigma(z) = \dfrac{1}{1 + e^{-z}}$

And now we have the probability that the point is a member, and predict “member” when that probability exceeds 0.5.The classifier never sees the image itself, just the pattern of which queries the model got right (the 47 bits).

3. Fit the 47 weights and the bias by gradient descent, using the same training loop as Part 2.

There are only 47 inputs and no spatial or sequential structure in them, so a single neuron suffices and reduces overfitting. It also associates a single weight with each augmentation, which allows us to later analyse how much each query contributed.

The classifier needs labelled membership data to train on. We give the attacker true labels for half the points and score on the other half. A real attacker would have neither of those, so this is a crutch.Meaning, the number below is optimistic. Part 5 removes it with shadow models.

Results

mean augmented accuracy — members 0.771 | non-members 0.516

gap attack (baseline)          : 0.719
count-threshold (>= 23 correct): 0.683
augmentation attack (logistic) : 0.750
improvement over gap           : +0.031

Part 2 measured the gap attack at 0.718 over all 10,000 points. Here we get 0.719 because the attacker trains on only half of those images and everything is scored on the held-out half. It’s still the same attack but with a smaller sample, which is why the number is slightly different.

The augmentation attack scores 0.750 against a 0.719 baseline, reproducing the paper’s central claim that a label-only attack can beat the gap attack.

But let’s take a look at the count-threshold result, as it leads to an interesting observation.

The count-threshold result

I thought that by counting how many augmentations lead to no misclassification, we’d have an easy way to measure whether an image is a member or not. This felt like the most obvious way to use the data we get from looking at the augmentations’ predicted labels. A point counts as a member if enough of the augmented copies remain correctly classified.

The threshold isn’t hand-picked. We sweep all 48 possible cut-offs and keep whichever does best on the attacker’s training half; that comes out at 23, and on the held-out half it scores 0.683, which is below the baseline of the gap attack. So even with the best possible threshold, counting does not perform well.

best_t = max(range(len(augs) + 1),
             key=lambda t: ((n_correct_tr >= t) == y[tr]).mean())

To explain why, we can look at the accuracy gaps of un-augmented vs augmented queries. Un-augmented, members and non-members classify correctly 1.000 and 0.564 of the time, a gap of 0.436. Averaged across the augmented copies they classify correctly 0.771 and 0.516 of the time, a gap of 0.255.

Augmentation shrinks the gap. The model memorised its members as they are, so perturbing them costs it a lot (1.000 → 0.771), while the non-members were already only half right and had far less room to fall (0.564 → 0.516). Robustness on its own is a weaker signal than plain correctness on this target.

Counting also discards information. Every query gets the same weight, including the original.

Logistic regression weights the columns instead of averaging them. I expected it to lean on the identity column and use the augmented columns to break ties on the ambiguous points.The raw vectors are saved, so we can analyse this offline without a GPU.

That guess was wrong. We’ll see how in a moment. It led to some interesting findings!

Analysis

We need to rebuild the attack classifier itself because it was never saved, only its score was. We can refit it from the stored vectors. It comes back at 0.749800 against a recorded 0.749800, which means the coefficients below belong to the same model that scored the 0.750 above.

Two of the 47 queries do nothing

Three columns came back with the same coefficient, +1.0107, to four decimal places: the identity, rotate 1.0 and rotate -1.0.

Three separate features landing on the same weight wasn’t an accident. Looking at the respective correctness columns, all three are bit-identical across every one of the 10,000 points. This is because, due to interpolationTF.rotate defaults to nearest-neighbour, which maps each output pixel back to the closest source pixel and rounds to it., one-degree rotation is a no-op on the image. On a 32×32 image, the corner pixels sit about 21.9 px from the centre. Rotating it one degree moves them 0.383 px, so every pixel rounds back to where it started. At two degrees the displacement passes half a pixel, 224 of the 1024 pixels move, and the column stops being a duplicate, agreeing with identity on 98% of points instead of 100%.

The real query budget is 45, not 47.

This is also why the identity coefficient looked small. Some logistic regression terminology first, before we substantiate that claim.

Fit. The search for the weights that make the model least wrong on the training data.

L2. A charge on the size of the weights, equal to the sum of their squares, added to whatever the fit is minimising so that a large weight has to earn its keep.

Intercept. The $b$ (bias) from step 1 above, which shifts every point’s score by the same amount.

Log-loss. How wrong the model is on a point, measured so that a confidently wrong answer costs far more than an unsure one.

scikit-learn’s default (which is what we use) fits $\dfrac{1}{2}\lVert w \rVert^2 + C \sum \text{log-loss}$ with $C = 1$. It leaves the intercept out of the penalty because it isn’t attached to any input, so shrinking it wouldn’t make the model lean less on its queries. It would only miscalibrate the overall level.

The fit minimises the log-loss, and L2 adds the squared weights to that total, so both have to come down together. The log-loss is indifferent to how the total weight is distributed between three identical columns, since every split makes the exact same prediction, so that penalty is what breaks the tie. It prefers the total spread thin: three weights of 1 cost $1^2 + 1^2 + 1^2 = 3$, one weight of 3 costs $3^2 = 9$, for the exact same prediction. So the fit splits the gap signal evenly across the copies, and each one ends up looking a third as important as the signal really is. Refitting on the 45 distinct columns puts identity at +2.1412.

Four queries do most of the work

With the duplicates removed, the top of the ranking is:

rotate 2.0         +2.8124
rotate -2.0        +2.6018
identity           +2.1412
rotate -3.0        +0.8398
rotate 3.0         +0.6575

Identity comes third of 45 and carries 12.8% of the total weight, so it does not dominate.

Coefficients on correlated columns are not worth much on their own, though. rotate ±2 agree with identity on 98% of points, so the fit can shuffle weight between them without changing its predictions (similar to having duplicate columns), which makes the individual numbers unstable. Deleting queries and refitting is the stronger test. This is called an ablation.

The numbers below are quoted to four decimal places rather than the three I used earlier because that difference matters here.

subset                                     n     acc   vs gap  of gain
all queries (published attack)            47  0.7498  +0.0310    100%
all queries, duplicates removed           45  0.7490  +0.0302     97%
identity only (= gap attack)               1  0.7188  +0.0000      0%
drop identity group                       44  0.7490  +0.0302     97%
near cluster, as 11 columns               11  0.7436  +0.0248     80%
near cluster, duplicates removed           9  0.7422  +0.0234     75%
near cluster, no identity at all           8  0.7422  +0.0234     75%
rotations |r| in 2..3 only                 4  0.7416  +0.0228     74%
translations d=1 only                      4  0.7160  -0.0028     -9%
drop near cluster                         36  0.7178  -0.0010     -3%
paper in-range, with identity             25  0.7478  +0.0290     94%
paper in-range, with identity, deduped    23  0.7480  +0.0292     94%
paper in-range, no identity               22  0.7480  +0.0292     94%
out-of-range, with identity               23  0.7188  +0.0000      0%
out-of-range, no identity                 22  0.6400  -0.0788   -254%

The near cluster is listed three ways because counting it as eleven columns flatters it. Two of the eleven are the rotate ±1 duplicates, so the honest count is nine. It also scores higher that way, 0.7436 against 0.7422, for the same reason we explained above. Three identical columns divide the penalty between them, each facing a weaker penalty than a single column carrying the same weight would.

The identity-only row lands exactly on the gap baseline, which is the check that the ablation is wired up correctly.Dropping the identity query means dropping rotate ±1 along with it. Removing column 0 on its own changes nothing, since its two copies still carry the whole signal, and the ablation would read as a null result.

Removing the identity query costs 0.0008. The attack does not need it because rotate ±2 hold the same information and substitute for it.

A narrow band around the original image provides the most substantial signal. The nine distinct queries within three degrees and one pixel score 0.7422, which is three quarters of the entire gain on a fifth of the budget. Four of them do nearly all of that: rotate ±2 and rotate ±3 on their own reach 0.7416, with no un-augmented query involved at all. Dropping identity from the band costs nothing here.0.7422 with it, 0.7422 without. The four d=1 translations are close to inert too: adding them to the four rotations is worth 0.0006, and on their own they score 0.7160, below the gap baseline.

The other 36 queries score 0.7178, slightly below what a single un-augmented query manages by itself. The largest rotations carry negative coefficients, rotate 14.0 at −0.50 and rotate -15.0 at −0.32, meaning a point that survives a fourteen-degree rotation is being read as evidence against membership.I find this hilarious.

This agrees with the count-threshold result. Adding all 47 columns with equal weight lets 36 redundant columns (a few even pointing the wrong way) outvote the rest that carry signal. The gain comes from how logistic regression weights the columns rather than from augmentation robustness replacing the gap signal.

Is the gain real?

I trained this target three times, and each time the numbers change slightly. The gain came out at 0.039, then 0.031 (this is the run I’ve based my blog posts on), then 0.033, with torch.manual_seed(0) set every time. GPU training is not deterministic, so the same script does not give back the same model twice.Kernels that accumulate in parallel add their partial results in whatever order the threads happen to finish, and floating-point addition is not associative, so tiny differences appear and then compound over 50 epochs. The three figures are within 0.008 of each other, roughly a quarter of the claimed gain.

However, one thing remains consistent between the three runs. The augmentation attack beat the gap attack every time, and the count-threshold variant lost to it every time, scoring 0.683 to 0.686 against a baseline that ranged from 0.719 to 0.723. Every figure in this post comes from the second run, so the third decimal of any of them carries noise from the difference in training. That being said, the comparisons they support are steady even where the figures aren’t.

There are two ways +0.031 could be an accident. It could be an artefact of which points happened to land in the scoring half. Or it could be an artefact of which target I happened to train. The first is a smaller uncertainty and is testable from the data I have.

Both attacks scored the same 5,000 held-out points, which makes this a paired comparison, and McNemar’s test is built for that. It discards every point the two attacks agree on, whether they were both right or both wrong, and looks only at the ones where they disagree.Under the null hypothesis that neither is better, each disagreement is a coin flip, so the discordant counts follow a binomial with $p = 0.5$. A point they both got right tells nothing about which attack is better, and neither does a point they both missed. Only the ones where they part ways carry any information.

That turns the question into something simple. If neither attack were really better than the other, each disagreement has a 50/50 chance for which attack wins it. Let’s count.

both correct           : 3525
gap only               :   69
augmentation only      :  224
neither                : 1182
discordant pairs       :  293
exact two-sided p      : 2.832e-20
95% CI on P(aug wins)  : 0.712 - 0.812

4,707 of the 5,000 points are scored identically by both attacks. Of the 293 that differ, the augmentation attack wins 224 and loses 69.

The p above is the chance of a split being this lopsided for two attacks that really are equally good. At 2.8e-20, “the augmentation attack getting lucky on which points happened to disagree” is not a plausible explanation.

P(X >= 224) = 1.415842e-20
P(X <=  69) = 1.415842e-20
                -----------
two-sided p   = 2.831684e-20      (just twice one tail)

224 out of 293 is 76.5%, but that is one measurement on one set of points, and a different set would have landed somewhere nearby rather than on exactly the same figure. The interval on the last line is the range of true win rates that could plausibly have produced what we saw. (The 95% attached, called the confidence level, is a convention.)It is a choice about how often we are willing to be wrong, in this case one time in twenty. A 99% interval would be wider and more cautious, a 90% one narrower and bolder. It describes the confidence of the method. About 95% of intervals built this way contain the true value. Two equally good attacks would split their disagreements evenly at 50%, which is not the case here.

By treating the two accuracies as though they came from separate samples, we get z = 3.51 and p = 4e-4z counts how many standard deviations the measured difference sits away from no difference at all. Every z corresponds to a p, two views of the same comparison.. That is still significant, but sixteen orders of magnitude weaker than when we do the paired test.

There is a limit to what this test covers. The variation across the three retrainings comes from the model being different each time, which this test doesn’t take into account.

But, for a given target, the margin between the attacks is real. The third decimal is noise from the model retraining.

Is the +0.031 gain too small?

Part 1 predicted this. The augmentation attack should do best against a model trained with augmentation because such a model has been taught to classify rotated and shifted copies of its training data correctly. This target never was, which is why the augmented gap above collapses to 0.255. The signal is weak because nothing in training gave it a reason to be strong.

Measured against the paper, though, +0.031 is not an outlier. Section 5.5 reports that an optimal choice of r and d beats the gap attack by 3 to 4 percentage points. We got 3.1, at the bottom of that range, though their targets in that experiment are trained on 2,500 points to our 5,000, so the regimes are not identical. The gain is small in absolute terms and ordinary next to the paper’s own numbers.

There’s a third way our setup differs from the paper’s. Our target is shallower. The paper trained a CNN with four conv layers (32, 32, 64, 64) and one FC of 512, at 1.2M parameters, while ours is two conv layers (32, 64) plus two FC (128, 10), at 0.55M. However it’s important to note that capacity was not the binding constraint on memorisation since train accuracy was still able to hit 1.000.

Lastly, there is a fourth difference which ablations ruled out. We ran a different set of augmented queries than the paper did. The paper finds the attack only clears the baseline for 1 ≤ r ≤ 8 and 1 ≤ d ≤ 2.Smaller than that and nothing misclassifies, bigger and everything does. We swept out to r = 15 and d = 4, which puts 22 of our 47 queries outside that window, so the modest gain could have been dead columns rather than an absent signal. But refitting on the in-range queries gave 0.7480 against 0.7498 for the full set, so throwing the out-of-window columns away did not increase the gain. Adding them to the identity query doesn’t improve it either. Identity plus all 22 of them scores 0.7188, which is what identity scores alone. They are not inert, though. On their own, without the identity query to lean on, those 22 score 0.6400.Better than a coin flip, so they do carry some membership signal. But far worse than the single un-augmented query, and they add nothing on top of it. The columns may carry some signal, but the fit had already discounted them. The 22 out-of-range queries carry a mean absolute weight of 0.156 against 0.550 for the in-range ones, so they were barely contributing before ablation, and deleting a column the model had already turned down cannot change much.

Possible follow-ups

The obvious one would be a second target trained with augmentation, attacked identically. The paper already ran that comparison and found that augmented training increases leakage. Models that overfit less on the original data can leak more against this type of attack because they have overfit on an augmented version of the data instead. That experiment would reproduce that result rather than add to it.I’d still like to run it at some point for the sake of completion. A target where the attack is supposed to work would be a good sanity check on my pipeline.

The extension I’d rather do is re-scoring these attacks at low false-positive rates. This paper reports balanced accuracy. The attack gives every point a score and calls it a member when that score passes 0.5, and 0.750 is what that particular cut-off produces. The cut-off is a choice, though. Lower it and the attack catches more real members but also flags more innocent non-members. Raise it and it does the reverse. A real attacker works at the raised end, where a false positive means naming someone who was never in the training set at all. The paper is not unaware of this. Supplement B.2, “On Measuring Success”, notes that recent work had already questioned balanced accuracy as a measure of attack success, and the outlier experiment uses precision instead. The argument was then made properly in 2022, in Membership Inference Attacks From First Principles.Two of its authors, Nicholas Carlini and Florian Tramèr, are also authors of the 2021 label-only paper. It showed that a single averaged number hides the strict end completely, and that attacks with near-identical average accuracy can differ by orders of magnitude once you look there. Some of them catch almost nobody.

No TPR-at-low-FPR figure is reported in the label-only paper, so this would be a new number for these attacks. Our logistic model already assigns every point a score, so the sweep runs offline from vectors we have. The 2022 paper quotes rates as low as 0.1%, but our held-out half only holds 2,500 non-members, so 0.1% would be two and a half of them. The lowest rate we can report honestly is around 1%, which is 25 non-members, and even that is a small number to rest a claim on.

Another question we can ask is, how many of the queries are actually needed? Query count is a real cost in this threat model, and an attacker hitting an API 45 times per point is more conspicuous than one hitting it ten times. The near-cluster result suggests most of the budget is wasted, since four rotations recover three quarters of the gain, so the curve of accuracy against budget is worth measuring properly rather than reading off a few rows of an ablation table.

Next up

Part 4, the boundary-distance attack. Instead of a fixed menu of rotations and shifts, we perturb the point until the model stops classifying it correctly, and measure how far we had to go.Points the model already gets wrong are defined to have distance 0, so they automatically get called non-members. That is exactly what the gap attack does with them. The distance only makes a difference for the points the model gets right, grading them instead of lumping them together. That distance estimates how far the point sits from a decision boundary. The paper’s strongest numbers come from this type of attack.