What is a membership inference attack? (reproducing a paper, part 1)

This is the first in a series of posts cataloguing my journey in reproducing results from different papers that I want to study and fully understand. My current interest lies in AI-safety and ML-security, and I’ve picked Label-Only Membership Inference Attacks (Choquette-Choo, Tramèr, Carlini, Papernot, 2021) to focus on today.

First thing I noticed is that while I understood the gist of what the abstract was trying to say, there were a lot of technical terms that I’d only heard of but hadn’t dived deep into. So that was the first order of business, before writing any code. If you’re also new to this kind of thing, welcome. I hope you learn something along with me. In this post we will be focusing on the core idea of the paper and the vocabulary needed to articulate it.

What’s the attack actually asking?

To put it simply:

We’ve got a trained model and some specific data point. Can we tell whether that point was in the model’s training set?

If we can, that’s a privacy leak. E.g. for a model that was trained on hospital records, if I can figure out that someone’s record was in there, I now know they were a patient. Membership inference is one of the most basic ways a model can spill something about the data behind it.

Labels vs. confidence scores

The models look at a given data point (e.g. photos of animals), and give out a number for every class that the data point could be labelled as, which represents the probability that the model thinks that label applies to the data. This probability is the confidence score.

cat: 0.92   dog: 0.05   fox: 0.03

(The numbers add up to 1.)

Two pieces here:

• the predicted label is the winner; the label with the highest likelihood of being the answer.
• the confidence score is that likelihood.

Models may or may not reveal these confidence scores. So we have two types of attacks depending on whether:

• we get the confidences: the full [0.92, 0.05, 0.03].
• label-only: we just get “cat”, and the numbers are hidden.

Lots of real systems only hand over the label and keep the probabilities to themselves — sometimes as a way to protect against membership-inference attacks — so the question is whether there are attacks that would be as effective without having access to the confidence scores.

Overfitting

Models act differently on data they were trained on. They’ve kind of memorized it, so they tend to be too confident on training points. If we know whether a model is noticeably more confident about one photo compared to others that are similar, we can infer that that data point may have been part of the training set.

Doing it with labels only

If the model hides its confidences and only gives us labels, that classic attack no longer works.

So what the paper does instead is find a way to deduce that signal without having access to the scores themselves. Instead of “how confident are you?” it asks “how much does your answer wobble?” We take the photo, nudge it a little (rotate it, shift it, add some noise) and see if the label changes or not:

• for a member, the model’s so sure that little nudges don’t change anything.
• for a non-member, a small nudge tips it over to another class.

So “does the label survive a nudge?” gives us a way to indirectly measure a model’s confidence score. The nudging is done either with normal data augmentations (rotations, shifts) or adversarial examples (the tiniest nudge that flips the label, which basically measures how far the point is from the boundary).

One thing I want to note to myself for later: the augmentation trick works because a model trained with augmentations becomes especially robust to those same augmentations on its training data, so nudged copies of members still get classified right, which should make an augmentation-based attack especially strong. So what happens with models that don’t use augmentation in their training?

Motivation behind this paper

Some defenses against membership inference work by hiding or obscuring (note to self: how?) the confidence numbers so the classic attack can’t read them, but they leave the predicted label. The paper calls this confidence masking. But if we can have a sort of attack that is independent of the confidence scores and only uses the labels, we completely bypass this method of protection.

Defences

How do we protect against label-only attacks suggested by this paper? We need to reduce overfitting itself, so that it’s not possible to infer membership from data perturbation.

• strong L2 regularization (simpler model, memorizes less)
• differential privacy (training with noise so no single point leaves much of a trace)

Next up

I’m going to build a simple model to test this on. Let’s see how it goes. :)