A Discussion of
Adversarial Examples Are Not Bugs, They Are Features

On May 6th, Andrew Ilyas and colleagues published a paper outlining two sets of experiments. Firstly, they showed that models trained on adversarial examples can transfer to real data, and secondly that models trained on a dataset derived from the representations of robust neural networks seem to inherit non-trivial robustness. They proposed an intriguing interpretation for their results: adversarial examples are due to “non-robust features” which are highly predictive but imperceptible to humans.

The paper was received with intense interest and discussion on social media, mailing lists, and reading groups around the world. How should we interpret these experiments? Would they replicate? Adversarial example research is particularly vulnerable to a certain kind of non-replication among disciplines of machine learning, because it requires researchers to play both attack and defense. It’s easy for even very rigorous researchers to accidentally use a weak attack. However, as we’ll see, Ilyas et al’s results have held up to initial scrutiny. And if non-robust features exist… what are they?

To explore these questions, Distill decided to run an experimental “discussion article.” Running a discussion article is something Distill has wanted to try for several years. It was originally suggested to us by Ferenc Huszár, who writes many lovely discussions of papers on his blog.

Why not just have everyone write private blog posts like Ferenc? Distill hopes that providing a more organized forum for many people to participate can give more researchers license to invest energy in discussing other’s work and make sure there’s an opportunity for all parties to comment and respond before the final version is published. We invited a number of researchers to write comments on the paper and organized discussion and responses from the original authors.

The Machine Learning community sometimes worries that peer review isn’t thorough enough. In contrast to this, we were struck by how deeply respondents engaged. Some respondents literally invested weeks in replicating results, running new experiments, and thinking deeply about the original paper. We also saw respondents update their views on non-robust features as they ran experiments — sometimes back and forth! The original authors similarly deeply engaged in discussing their results, clarifying misunderstandings, and even running new experiments in response to comments.

We think this deep engagement and discussion is really exciting, and hope to experiment with more such discussion articles in the future.

Discussion Themes

Clarifications: Discussion between the respondents and original authors was able to surface several misunderstandings or opportunities to sharpen claims. The original authors summarize this in their rebuttal.

Successful Replication: Respondents successfully reproduced many of the experiments in Ilyas et al and had no unsuccessful replication attempts. This was significantly facilitated by the release of code, models, and datasets by the original authors. Gabriel Goh and Preetum Nakkiran both independently reimplemented and replicated the non-robust dataset experiments. Preetum reproduced the $\widehat{\mathcal{D}}_{det}$ non-robust dataset experiment as described in the paper, for $L_\infty$ and $L_2$ attacks.
Gabriel repproduced both $\widehat{\mathcal{D}}_{det}$ and $\widehat{\mathcal{D}}_{rand}$ for $L_2$ attacks. Preetum also replicated part of the robust dataset experiment by training models on the provided robust dataset and finding that they seemed non-trivially robust. It seems epistemically notable that both Preetum and Gabriel were initially skeptical. Preetum emphasizes that he found it easy to make the phenomenon work and that it was robust to many variants and hyperparameters he tried.

Exploring the Boundaries of Non-Robust Transfer: Three of the comments focused on variants of the “non-robust dataset” experiment, where training on adversarial examples transfers to real data. When, how, and why does it happen? Gabriel Goh explores an alternative mechanism for the results, Preetum Nakkiran shows a special construction where it doesn’t happen, and Eric Wallace shows that transfer can happen for other kinds of incorrectly labeled data.

Properties of Robust and Non-Robust Features: The other three comments focused on the properties of robust and non-robust models. Gabriel Goh explores what non-robust features might look like in the case of linear models, while Dan Hendrycks and Justin Gilmer discuss how the results relate to the broader problem of robustness to distribution shift, and Reiichiro Nakano explores the qualitative differences of robust models in the context of style transfer.

Comments

Distill collected six comments on the original paper. They are presented in alphabetical order by the author’s last name, with brief summaries of each comment and the corresponding response from the original authors.

Original Author Discussion and Responses