Achieving a research-level understanding of most topics is like climbing a mountain. Aspiring researchers must struggle to understand vast bodies of work that came before them, to learn techniques, and to gain intuition. Upon reaching the top, the new researcher begins doing novel work, throwing new stones onto the top of the mountain and making it a little taller for whoever comes next.
Mathematics is a striking example of this. For centuries, countless minds have climbed the mountain range of mathematics and laid new boulders at the top. Over time, different peaks formed, built on top of particularly beautiful results. Now the peaks of mathematics are so numerous and steep that no person can climb them all. Even with a lifetime of dedicated effort, a mathematician may only enjoy some of their vistas.
People expect the climb to be hard. It reflects the tremendous progress and cumulative effort that’s gone into mathematics. The climb is seen as an intellectual pilgrimage, the labor a rite of passage. But the climb could be massively easier. It’s entirely possible to build paths and staircases into these mountains.
The climb isn’t progress: the climb is a mountain of debt.
Programmers talk about technical debt: there are ways to write software that are faster in the short run but problematic in the long run. Managers talk about institutional debt: institutions can grow quickly at the cost of bad practices creeping in. Both are easy to accumulate but hard to get rid of.
Research can also have debt. It comes in several forms:
The insidious thing about research debt is that it’s normal. Everyone takes it for granted, and doesn’t realize that things could be different. For example, it’s normal to give very mediocre explanations of research, and people perceive that to be the ceiling of explanation quality. On the rare occasions that truly excellent explanations come along, people see them as one-off miracles rather than a sign that we could systematically be doing better.
There’s a tradeoff between the energy put into explaining an idea, and the energy needed to understand it. On one extreme, the explainer can painstakingly craft a beautiful explanation, leading their audience to understanding without even realizing it could have been difficult. On the other extreme, the explainer can do the absolute minimum and abandon their audience to struggle. This energy is called interpretive labor
Many explanations are not one-to-one. People give lectures, write books, or communicate online. In these one-to-many cases, each member of the audience pays the cost of understanding, even though the cost of explaining stays the same.
In research, we often have a group of researchers all trying to understand each other. Just like before, the cost of explaining stays constant as the group grows, but the cost of understanding increases with each new member. At some size, the effort to understand everyone else becomes too much. As a defense mechanism, people specialize, focusing on a narrower area of interest. The maintainable size of the field is controlled by how its members trade off the energy between communicating and understanding.
Research debt is the accumulation of missing interpretive labor. It’s extremely natural for young ideas to go through a stage of debt, like early prototypes in engineering. The problem is that we often stop at that point. Young ideas aren’t ending points for us to put in a paper and abandon. When we let things stop there the debt piles up. It becomes harder to understand and build on each other’s work and the field fragments.
It’s worth being clear that research debt isn’t just about ideas not being explained well. It’s a lack of digesting ideas – or, at least, a lack of the public version of ideas being digested.
Developing good abstractions, notations, visualizations, and so forth, is improving the user interfaces for ideas. This helps both with understanding ideas for the first time and with thinking clearly about them. Conversely, if we can’t explain an idea well, that’s often a sign that we don’t understand it as well as we could.
It shouldn’t be that surprising that these two largely go hand in hand. Part of thinking is having a conversation with ourselves.
Research distillation is the opposite of research debt. It can be incredibly satisfying, combining deep scientific understanding, empathy, and design to do justice to our research and lay bare beautiful insights.
Distillation is also hard. It’s tempting to think of explaining an idea as just putting a layer of polish on it, but good explanations often involve transforming the idea. This kind of refinement of an idea can take just as much effort and deep understanding as the initial discovery.
This leaves us with no easy way out. We can’t solve research debt by having one person write a textbook: their energy is spread too thin to polish every idea from scratch. We can’t outsource distillation to less skilled non-experts: refining and explaining ideas requires creativity and deep understanding, just as much as novel research.
Research distillation doesn’t have to be you, but it does have to be us.
Like the theoretician, the experimentalist or the research engineer, the research distiller is an integral role for a healthy research community. Right now, almost no one is filling it.
Why do researchers not work on distillation? One possibility is perverse incentives, like wanting your work to look difficult. Those certainly exist, but we don’t think they’re the main factor.
However, our experience is that most good researchers don’t seem that motivated by these kind of factors. Instead, the main issue is that it isn’t worthwhile for them to divert energy from pursuing results to distill things. Perhaps things are different in other fields, or I’m not cynical enough.
Lots of people want to work on research distillation. Unfortunately, it’s very difficult to do so, because we don’t support them.
An aspiring research distiller lacks many things that are easy to take for granted: a career path, places to learn, examples and role models. Underlying this is a deeper issue: their work isn’t seen as a real research contribution. We need to fix this.
If you are excited to distill ideas, seek clarity, and build beautiful explanations, we are letting you down. You have something precious to contribute and we aren’t supporting you the way we should.
The Distill Ecosystem is an attempt to better support this kind of work. Right now, it has three parts:
This is just a start: there’s a lot more that needs to be done. A complete ecosystem for this kind of work needs several other components, including places where one can learn these skills and reliable sources of employment doing this kind of work. We’re optimistic that will come with time.
Explorable Explanations:
There’s a loose community exploring how the interactive medium enabled by computers can be used to communicate and think in ways previously impossible. These ideas start, as many ideas in computing do, with work done by Douglas Engelbart
More recently, Explorable Explanations have started to reimagine what an essay can be in this new medium. This started with Bret Victor’s
There are also explorations of how we can augment our ability to think in this new medium, bringing previously inaccessible ideas within reach. Once again, Bret Victor has wonderful ideas
Increasingly, the challenge is filtering accessible content. Karpathy’s ArXiv Sanity is a lovely tool for this. Crowd curation by online communities also helps a great deal.
This seems really important. Traditionally, if one doesn’t turn research into a paper, it’s basically as though you didn’t do it. This creates a strong incentive for all research to be dressed up as an important paper, increasing noise.
Other nice thoughts in this space include the idea of an “open exposition problem”
We’re extremely grateful for the advice and assistance of Jennifer Daniel in illustrating this article.
This essay has greatly benefitted from the comments of many people, including: Dandelion Mané, Emma Pierson, Michael Nielsen, Cassandra Xia, Geoffrey Irving, Elizabeth Van Nostrand, Maithra Raghu, Greg Brockman, Hannah Davis, Devon Zuegel, Wojciech Zaremba, Vikas Sindhwani, Pierre Sermanet, Mike Schuster, George Dahl, Jascha Sohl-dickstein, Adam Roberts, Greg Corrado, Samy Bengio, Yomna Nasser, Katherine Ye, Dave Rushton-Smith, Martin Wattenberg, Fernanda Viegas, Eric Breck, Aaron Courville.
This essay was primarily written by Chris Olah and illustrated by Shan Carter.
View all changes to this article since it was first published. If you see mistakes or want to suggest changes, please create an issue on GitHub.
Diagrams and text are licensed under Creative Commons Attribution CC-BY 4.0 with the source available on GitHub, unless noted otherwise. The figures that have been reused from other sources don’t fall under this license and can be recognized by a note in their caption: “Figure from …”.
For attribution in academic contexts, please cite this work as
Olah & Carter, "Research Debt", Distill, 2017.
BibTeX citation
@article{olah2017research,
author = {Olah, Chris and Carter, Shan},
title = {Research Debt},
journal = {Distill},
year = {2017},
note = {https://distill.pub/2017/research-debt},
doi = {10.23915/distill.00005}
}