Predictions about catastrophic risks are often self-defeating

Why you should be making more conditional predictions instead

Related: Stefan Schubert’s Sleepwalk bias, self-defeating predictions, and existential risk

An interesting paradox can occur when making a prediction about the probability of a future event affects the probability. In some cases, such self-defeating predictions can’t be accurate because they’re aiming at a target that moves whenever they fire. It’s particularly noticeable when predicting a high chance of an event occurring makes it less likely, while predicting a low chance of an event occurring makes it more likely.

Graphically, it looks like this:

Usually our predictions are not causally connected to the events we’re predicting, so we don’t have to worry about this. But sometimes they are. And if forecasting techniques become more widely known and influential (as many advocates would like), then this paradox could become more important - especially when it comes to predictions of disasters like catastrophic risks.

The problems with self-defeating predictions can be resolved by making conditional predictions instead. I think this could be particularly important for predictions of catastrophic risks, including from advanced AI. Making more conditional predictions might also clear up a lot of confusion regarding the wide variation in AI risk predictions.

In the rest of this post, I discuss how self-defeating predictions work in a bit more detail, give some examples, and talk about some important implications.

How predictions defeat themselves

Many decisions rest, implicitly or explicitly, on how likely future events seem. Buildings are built differently where more earthquakes are expected. Policymakers change economic and monetary policies when they think a recession is coming. Some philanthropists prioritise working on risks they think are more likely to occur.

External predictions often affect these decisions. Climate scientists forecast the probability of various events in the IPCC reports, for example, and policymakers consider these probabilities when setting emission reduction targets.

This interaction can cause issues for the accuracy of those predictions. In particular, by reacting to the predictions, decision-makers can alter the probability of the events being predicted. If the policymakers are freaked out by climate scientists predicting a high chance of extreme climate change, then they might implement strong climate policies that make those extreme scenarios much less likely.

In most cases, predictions aren’t that influential. Decision-makers might know that predictions account for their actions: Mohamed Salah doesn’t assume that, because Liverpool are usually predicted to win their games, he doesn’t have to score any goals. In other cases, predictions aren’t causally connected to the thing we’re predicting at all. Predictions about when the sun will expand and swallow the Earth, for example, don’t affect the rate at which it burns hydrogen.

In some cases, however, these dynamics are important. In particular, predictions about future disasters have a strong influence on multiple decision-makers:

• Predictions about the future impacts of climate change affect the strength of adaptation and mitigation plans

• Predictions about the probability of future natural disasters affect preparedness measures, like building dikes to prevent floods

• Predictions about pandemics affect people’s behaviour and public health policies

• Predictions about catastrophic impacts of new technologies affect regulatory decisions about how new technologies can be developed and commercialised

In each of these cases, predicting a disaster changes, and probably reduces, the chance a disaster occurs.

I think there are a couple important implications.

This explains why experts often make dire predictions of doom

First, it helps explain some patterns in expert communication. There’s often an incongruity in how scientists talk about the expected impacts of climate change in popular media versus scientific papers. In papers they’re usually more conservative.

A simple explanation for this is that self-defeating predictions leave climate scientists in a pickle. Most of them are working on climate change because they’re worried about the future impacts. And indeed, if we don’t take action to cut emissions, the impacts could be quite negative. Fortunately, largely because scientists predict major harms, we’re investing quite a lot in green technology and taking regulatory action.

As a result, the most dire predictions of the effects of climate change probably won’t come to pass. But climate scientists worry that if they make predictions that account for these societal adaptations, they’ll defuse the worries that are driving those adaptations in the first place!

Conditional probabilities solve these problems

Second, it makes a strong case for making conditional predictions instead. Predictions won’t defeat themselves if they’re explicitly conditioned on various societal responses. Then we can issue the warning that we’re on track for disaster (conditional on no response) and make an accurate prediction that accounts for countermeasures (conditional on a societal response).

Conditional predictions have other benefits. Cavernous differences between predictions of future AI risks persist even among people who have all thought about these risks a lot. I think at least some of this gap can be explained by people implicitly conditioning on different scenarios: those who expect companies to invest a lot in safety or governments to regulate strictly, for example, versus those who don’t. (Stefan Schubert has called the tendency to underestimate societal adaptations sleepwalk bias.)

Conditional probabilities would surface these assumptions. Relatedly, they’d also be more useful for decision-makers, because they’d give a sense of how much they could affect the probability of these risks by taking action. As with climate change, self-defeating predictions are particularly important in this field, because many of the decisions of companies, policymakers, and philanthropists in this space are strongly influenced by predictions about how likely future AI systems are to cause major disasters.