‘An enthusiastic philosopher… had constructed a very satisfactory theory on some subject or other and was not a little proud of it. "But the facts, my dear fellow," said his friend, "the facts do not agree with your theory."—"Don't they?" replied the philosopher, shrugging his shoulders, "then, tant pis pour les faits;"—so much the worse for the facts!” ― Charles Mackay, Extraordinary Popular Delusions and the Madness of Crowds
How would we know if you were wrong?
This piece (written ahead of the annual MethodsCon conference on social science methods) is about why it’s often better to be usefully wrong than uselessly right, and what this implies for social science. It makes the case for sharper theories – ones that can be tested and interrogated.
All of us encounter dozens of claims every day – predictions, theories, arguments – and I like to ask a simple question of anyone making a claim of this kind: ‘how would we know if you were wrong?’ The answer doesn’t have to be a pile of data, a formal experiment or a randomised control trial. But if there is no answer at all – and there often isn’t - I tend to be doubtful of the claim.
I ask this question because knowledge advances through the interaction of claims on the one hand and confirming or disconfirming facts on other. Sometimes the most interesting facts in science are ones which contradict a theory. These anomalies help fields to progress. They force theorists to generate better theories.[i] A good contemporary example is the data coming back from the James Webb telescope which appears to challenge many apparently solid theories in astronomy.
Yet the social sciences are not always so keen on hunting for disconfirming evidence. Theorists and their followers become adept at cherry-picking confirming data and ignoring troublesome facts (lots of research now shows the strength of tendencies towards ‘confirmation bias’, particularly amongst the well-educated). Even big shocks that profoundly challenge conventional wisdom in disciplines can be ignored – as happened to economics during and after the crisis of 2007/8, and to Marxist theory in the later decades of the last century.
But we should want social sciences to be sharp enough to explain and predict, and honest enough to admit when theories turn out to be mistaken. Vague theories may be stimulating and entertaining, but they usually make more sense in relation to other theories than in relation to the reality of the social world.
Indeed, in general, it’s better to be usefully wrong than uselessly right. So if, for example, I comment that the world is very complex, or that there is no silver bullet to solve a social problem, or that AI could either be a boon for humanity or a great disaster, I’m being uselessly right.
If on the other hand I predict that AI will lead to only modest productivity gains, and perhaps a 0.1% annual rise in GDP (as Daron Acemoglu did recently), I may be wrong but at least, by setting out a clear logic, and a clear prediction, my claim is useful. Similarly, if I claim that even quite a low level of universal basic income will increase wellbeing, reduce anxiety and boost employment, my claim can be tested in the real world (as many experiments are now doing) which may generate novel insights and will probably prove both the advocates and the critics wrong.
Much social science is, and should be, descriptive, and interested in the complex details and nuances of contexts. Not everything has to be falsifiable. But social sciences have always also depended on theories that make it possible to see larger patterns across space and time, and many of the best social scientists weren’t shy of offering sharp, non-obvious forecasts.
Karl Marx is a prime example and made many predictions: that rates of profit would fall, that the working class would be immiserated both absolutely and relatively, that overproduction would lead to periodic crises and that capital would tend to become more concentrated. Some proved accurate, others not at all. Keynes also made counter-intuitive predictions, like the ‘paradox of thrift’, which argued that if everyone increased their savings during a recession, overall demand would fall, leading to lower aggregate savings and a deeper economic slump.
But there may have been fewer such predictions in recent decades. One factor could be the poor records of economic forecasting and of futurology which have discredited predictions of any kind (with the latter repeatedly predicting mass unemployment resulting from automation or that permanent jobs would disappear, and repeatedly proving mistaken).
The more plausible explanation, however, is that experts fear being seen to be wrong, and so losing status. Hence the appeal of commentary and critique, and of baggy, vague theories, like claims of a poly-crisis, or moral decline, or blaming anything on an amorphous ‘neoliberalism’, or ascribing any problems (such as COVID) to human abuse of nature (all of which are good candidates for the question: ‘how would we know if you were wrong?’).
A rough list of sharp theories in social science
Fortunately, there is a still live tradition of sharper theorising in social science, with claims that offer an analysis, a mechanism and a prediction, arguing that ‘if x, then y’, and sometimes sparking useful debate through disconfirming as well as confirming data. Here I’ve had a quick attempt at summarising a few old and new examples:
· A good starting point is the prediction of standard economics that demand will fall if a price rises, and vice versa. This is generally true (and Larry Summers once described it as a law), but perhaps the more interesting cases are ones where inelastic demand and status lead to the opposite effects, so that raising prices increases demand, as in parts of the luxury market.
· Gresham’s law (inspired, though not created by Thomas Gresham) predicts that bad money will drive out good money, but again there are also opposite examples which are just as interesting (which is why a better formulation is that bad money drives out good if they exchange for the same price).
· Maurice Duverger predicted that first-past-the-post electoral systems will tend to become organised around two competing parties which makes sense and is broadly accurate (even if Britain’s dominant parties sometimes seem keen to disprove the theory).
· David Ricardo’s theory of comparative advantage predicts that nations will best prosper if they concentrate on their comparative advantage, and remains counter-intuitive (though there are many reasons why nations might not follow his advice, including national security).
· Carol Dweck’s theory of ‘growth mindsets’ claims that these better predict success in education and in life than other factors such as IQ and has encouraged many attempts to confirm or disprove it (the jury is still out).
· Abraham Maslow’s hierarchy of needs predicts that people will not pursue higher-level needs (such as self-actualization) until their basic needs (such as food and safety) are met, a theory which also has found many exceptions.
· Leon Festinger’s theory of cognitive dissonance predicts that people will feel discomfort when presented with contradictory information and so will seek to reduce that comfort by adjusting their beliefs: again, there is some confirming evidence but also plenty of evidence that many people have a large capacity for contradiction.
· Eleanor Ostrom predicted that with the right tools and principles (which she specified) communities could more successfully manage common pool resources than either markets or states.
· BF Skinner’s theories of ‘intermittent reinforcement’ argued that behaviours are best changed through unpredictable rather than predictable reinforcement, and underpin much of the organisation of social media.
· Peter Turchin’s cliometric-based theories claim that growing inequality of wealth and wages, overproduction of potential elites relative to opportunities within elites, and growth in public debt will lead to political instability and breakdown, a claim that has looked quite prescient in recent years.
Other examples include the claim that if a dominant organised crime group becomes weaker, violence will rise not fall; that a more complex economy will tend to grow more than a less complex one; that higher levels of mutual support predict higher levels of happiness, whereas more education does not; that stress reduces IQ; that social media usage tends to increase mental illness amongst teenagers (which is contested, but seems to accord with data rather better than claims a generation before that exposure to violence on video would promote violence in the real world).[ii]
Some predictions come from outside social science, like the ‘long tail theory’ proposed by Chris Anderson in 2004, which predicted, quite plausibly, that the Internet would allow retailers to focus on a far larger range of products with small sales (which in turn would enable millions to make a living at craft levels of production). This partly happened with platforms like Etsy, and now with high margins on publishing and music back catalogues, but overall the pattern was the opposite, with the biggest brands and names becoming even more dominant.
Brexit prompted a flood of predictions on both sides of the argument. Most of these proved to be exaggerated though there has been relatively little thoughtful reflection on why. As with the 2007 financial crash, it would be interesting to bring together experts to reflect on why their models and assumptions proved mistaken, and what lessons they have learned.
Collective intelligence, a field I’m particularly interested in, makes many claims about how groups can outperform even the best individual intelligence, yet it also has interestingly conflicting theories. For example, there is evidence that groups make better judgements if their members can interact, and contrary evidence that they will do better if they do not interact. That these theories, and available evidence, are contradictory is a useful spur to discover what missing factors might explain the variance.
I’ve occasionally attempted predictions myself. 25 years ago I predicted that no mature democracy would ever see a secession. At the time this seemed improbable, even to me. Many countries looked set to break apart, from Spain/Catalonia and UK/Scotland to Canada/Quebec. My argument was that, once a secession looked likely, in a mature democracy sufficient compromises would be made to avert it. To my surprise my theory remains intact (the only partial counter-example is Greenland, though there have been many secessions in immature democracies, like the separation of Czechia and Slovakia). I’m sure there will be counter-examples in mature democracies at some point. But for several decades the prediction has proven right, against many expectations, and I think this gives us useful insights into the dynamics of democracy.
Some social science theories have been tested with high stakes. A good example was the
strategy of LTCM, developed by two subsequent Nobel Prize winners, focused on their theoretical model of bond prices, deviations and likely convergence. This theory briefly made LTCM very rich and then led to its collapse, which nearly caused the collapse of the US financial system.
As these examples show, often what we most need are theories which not only claim ‘if x, then y’ but also specify the boundary conditions: under what circumstances will they be right, since none of these turns out to be an immutable law (in other words, ‘if x, then y, in conditions z’).
Where more sharpness would be useful
So theories that are sharp can quicken thought. They cut through vagueness and wishful thinking, and even when they are not proven they spark more interesting debates about their limits and exceptions.
Yet some fields of social science find this difficult. One example is productivity which is seeing unprecedented investment in research right now, and obviously matters greatly to the UK’s future prospects. But so far, although the research is eminently sensible, it doesn’t seem to have progressed much beyond familiar lists of relevant factors, from infrastructure and skills to investment. It would be good to see at least one or two surprising theories (eg are firm perceptions of risk/reward in adoption a key factor, and can policies influence these?).
Another example is meta-science, a very interesting field that is seeing much more investment. This will hopefully progress beyond gathering data to prompt non-obvious hypotheses: eg does greater R&D investment inevitably lead to a decline in the productivity of R&D? Is the long-term decline in the productivity of pharmaceutical research (EROOM’s law) bound to continue? Do the sectors that innovate in how they organise research mitigate this pattern?
On the periphery of social science there is plenty of interest in falsifiability. The various research programmes around prediction markets and super-forecasters involve precise predictions and attempts to learn when they don’t materialise. Bayesian thinking does the same, encouraging precision about ‘priors’ and then a search for evidence to either confirm or disprove them.
Some professions try to institutionalise regular learning on why expectations or predictions don’t materialise. In some hospitals, surgeons gather together regularly to study data on the results of their surgery; in some schools, teachers reflect together on the gap between the grades they forecast and the ones that result; and the best investors constantly seek to learn from the gap between their forecasts and their results. They are not social theorists. But by constantly iterating between general models and specific data they aim to improve their methods and models, so that they do a better job with the next patient or pupil.
This is also one virtue of outcome-based commissioning in public services and social impact bonds. They are not easy to implement (I wrote at length about their strengths and weaknesses 15 years ago). But they force more sharpness and precision about how spending on an intervention might achieve a particular result.
Enemies of sharpness: nuance, extreme positivism and ‘as if’ theorising
The habits of sharp prediction, and regular learning, can be very effective at driving up performance, though they are also hard work. But they are not standard in the social sciences. As Kieran Healey wrote well in his paper ‘Fuck Nuance’[iii], social scientists often take refuge in complexity and nuance as an alternative to sharp thinking, rather than as a complement to it. As a result, their theories can be less useful, and also less interesting, than they could be, even if it may be helpful to some social scientists that their theories are harder to refute.
So while it is reasonable to point out that the social world is complex, nuanced and often highly contextual, and while in-depth case studies can be fascinating, in the future, as in the past, our understanding of the world will probably advance more through abstractions and simplifications that generate surprising insights.
Taking refuge in nuance and context is one barrier to sharper theories. Another is extreme positivism, which argues that nothing can be talked about if it cannot be observed, and which has been given new momentum by the availability of vast quantities of data. This too, unnecessarily constrains creative thought, and is at odds with much of the history of the natural and social sciences which has been fuelled by theories generated ahead of data. Another barrier is the ‘as if’ reasoning promoted by Milton Friedman amongst others, and which remains popular in economics. This argues that the only criterion for judging a theory is its predictive power, rather than the realism of its assumptions. This is fine as a starting point. But it quickly becomes problematic for a discipline if it becomes a habit and theories become ever more divorced from reality.
What could be done?
Sharp theorising isn’t a panacea. But it should be much more common than it is. For a professor of politics, sociology, psychology or economics not to be able to offer any non-obvious claims or predictions should be seen as a weakness.
I would love to see the peak bodies in social science (such as the UK’s Academy for Social Science or the ESRC) promote more sharpness of the kind I’ve described here.
That would require more teaching of theory and hypothesis generation (which may increasingly be helped by AI tools); encouragement of hybrid theories and assemblies of theories (since most complex phenomena involve more than one mechanism); discouragement of theory-free data gathering and experiment; and then a shift in incentives, since at the moment there are strong pressures working against more precise theoretical formulations, since these are bound to be more likely to be disproven.
Perhaps the peak bodies could gather the best examples of predictive theories, with non-obvious insights into important topics. Perhaps they could help bring together confirming and disconfirming evidence. And perhaps they could encourage younger scholars to be braver in proposing mechanisms and likely effects.
Once a year we could then celebrate the most interesting and surprising ones, with special prizes for the ones that are usefully wrong (a ‘Bayes’ prize for imaginative priors, that could potentially be tested?). Each discipline could offer its own prizes.
I’m sure our shared knowledge would advance more quickly. Occasional embarrassment for eminent experts would be a price worth paying.
[i] Even if the patterns are fuzzier than many would like to believe, and much fuzzier than Karl Popper’s notions of falsifiability implied
[ii] Though there is still plenty of argument about how much is correlation and how much is causation: https://www.nature.com/articles/d41586-024-00902-2
コメント