The European Public Choice Society has its annual conference in Braga (Portugal) in the week of the 11th of April. To give you an idea of what is hot and happening in the field of political economy, Joes interviewed attendants of the conference. For this interview, he spoke with Martin Paldam, who is writing a paper on meta-analyses and how they can be manipulated.
Joes: Could you please tell us about the paper you are presenting at the conference here?
Martin: Yes, it is a study of meta-studies. Many scientific results need to be replicated to be believable. However, rarely is a specific study exactly replicated. One way we can replace strict replication is by taking all studies in a certain field of a certain coefficient, and code them and then study their distribution. We can then approach the ‘true’ coefficient better than any individual study can. Of course, you could just take the plain average of all the coefficients, but ideally, you can use statistical methods to control for publication bias and omitted variable bias in the individual studies you analyse. Correcting for publication bias tends to make the average smaller, while correcting for omitted variable bias tends to make the average larger.
So my paper came up because meta-analyses can also be manipulated a little bit. I found out how much they can be manipulated, by taking one study, and then doing a so-called augmented meta-analysis on augmented FAT-PETs. And it turned out that while the meta-study kind of reduced the standard deviation of the results quite a lot, then you can, by manipulation, increase it a little bit. So this is what I showed.
Unfortunately, manipulation is possible in economics. We want to get rid of it, or at least reduce it as much as possible, but that is difficult.
Joes: So, are there results that we perhaps take for granted because of meta-studies that may be less strong or stronger than we expected?
Martin: I have done some meta-studies on development aid effectiveness, which is a very controversial topic. And the results are very diverse. We found out that the result is very weak through meta-analysis. Then somebody started manipulating their meta-analysis, and he showed that he could get the result back again that he wanted. Then I showed that the distribution of what you can do if you manipulate the meta-analysis is quite wide.
Joes: Is this also how you got to this topic when you engaged in this debate?
Martin: Yes. I have myself worked in the development aid sector. So I definitely wanted to see an effect, but I know that many people working in that sector know there is a dubious effect. So the reason I did the meta-analysis is that I wanted to know what the effectiveness is. Others started to criticize our results, they work for development aid themselves.
Joes: To go meta on your meta topic, what we see is that the use of science in the public debate has become more polarized over time. And people who criticize science while actually being scientists themselves may be criticized for being populists. For instance, in the Netherlands, the Outbreak Management Team claimed to be scientific. Criticizing them was difficult because they supposedly followed ‘the science’. Yet now it turned out they got clues from the government to make certain claims about the pandemic situation. How do you position yourself in an environment in which criticizing science is sometimes seen as being anti-science rather than inherently part of what science is?
Martin: I think meta-studies are the best solution for this. If you go to Google and write ‘replication crisis in economics’ you get around 130.000 citations. So it is a big issue, and you must find methods that somehow can reduce the variation of results. So we know that some results are more solid than others. That is what we try to do with meta-analysis.
Joes: Do you think it is possible to see whether people are ‘anti-science’ or engaging with science based on the kind of arguments they made?
Martin: Well, certainly if people have very strong views in advance, and find results that are very much in line with their views or with their job, then you could say there is a problem. I mean, it is not uncommon with meta-analyses that researchers who work for an organisation find nicer results, pro-organisation results, than others. So there is this bias which we have to deal with.
leviathanscouch.com 