A meta-rant about meta-analysis

The escalating publication of redundant, unnecessary and misleading meta-analyses is a growing blight which requires containment.

As a journal editor and reviewer I am repeatedly receiving, and now frequently recommending rejection for, meta-analyses of studies in the brain stimulation field. For a long time these were studies of transcranial magnetic stimulation (TMS), but now meta-analyses of transcranial direct current stimulation (tDCS) studies are propagating faster than the studies exploring the use of the technique themselves. Why is this a problem, why am I ranting and why should anyone care?

First, for readers unfamiliar with the area, what is a meta-analyses? Basically, meta-analysis is a statistical approach that combines the effects seen across multiple clinical trials (or other types of studies). In other words, the results of a number of similar clinical trials are brought together in a single analysis to evaluate whether the body of research has successfully answered a particular question. Meta-analysis can be used to analyse the results of a variety of different types of studies but this approach is especially commonly used to pool the results of clinical trials. For example, if 30 studies have tested whether a specific type of medication is effective in depression, with some studies saying yes and some no, what does the collection of these studies conclude?

As they can be used to synthesise the results of evidence in this way, meta-analyses are considered the ‘top’ level of evidence supporting the effectiveness or otherwise of a new treatment. Some regulatory authorities will require positive meta-analyses prior to approval of a new therapy and even more commonly they are required before a treatment is recommend for use in professional clinical practice guidelines.

Meta-analyses are also often very popular with journals: a positive meta-analysis is likely to widely cited by other articles and the number of times an article is cited directly affects the so called impact factor of a journal, often seen as a reflection of the quality and exclusivity of the journal itself.

The development of meta-analyses has seen the development of a particular subset of academic: those who don’t actually do original research but just conduct and publish these analyses. Building a career like this clearly does not require obtaining large grants to support research teams or the long slog of subject recruitment into clinical trials and studies and can lead to highly cited publications in top ranked journals.

So what is the problem with all this? Well, the problem is not with the technique itself but its application. As these methods can be learnt relatively easily, their application has spread rapidly, now frequently to a point where they are addressing questions well beyond those they should, or are addressing them in problematic ways. How?

Well first, meta-analyses are being conducted where the underlying body of evidence really just doesn’t justify the conduct of the analysis: they are pooling groups of studies with way too few trials to make any sense and where the likelihood of a meaningful result emerging is small.

Second, studies, especially in the brain stimulation area, are repeatedly pooling trials in meta-analyses that really are not equivalent: for example pooling treatments using different types of stimulation or targeting different brain regions. If you pool the results of different treatment approaches, the results of this analysis really cannot have sensible meaning. It cannot say that any of the individual approaches are effective, just that somehow, overall, the approach might be useful. However, non specialist readers will not understand the often subtle differences in how these treatments are applied and may well incorrectly conclude that a true effect has been established.

I suspect this later issue is exacerbated by the conduct of analyses by researchers without sufficient ‘domain area’ expertise: the conduct of meta-analyses by individuals with expertise in the statistical technique but not the treatment being investigated. This seems to be a common problem in the brain stimulation field where the subtleties of different forms of TMS or tDCS may not be obvious to a non-expert. If you don’t fundamentally understand the treatment, selecting equivalent and comparable studies may well be impossible.

The pooling of studies inappropriately is most likely to lead to a conclusion that a treatment is effective when this conclusion is problematic. The first issue I raised — the conduct of meta-analyses on too few studies — has the opposite problem: it is likely to demonstrate a lack of effect, that the treatment does not work. This can have very really world consequences.

There is a perfect example of this in my field, especially in the early development of TMS as a therapy for depression. In the early 2000’s I was desperately trying to get grant funding to support my TMS research program. However, in 2003 a meta-analysis was published in the British Journal of Psychiatry, of rTMS studies conducted up until that time. This paper included 14 studies, a seemingly reasonable number, but with only 12 studies and 217 patients in the main analysis. Almost all of these were small preliminary studies which provided treatment for extremely short periods of time, some for only one week, to very small numbers of patients (9 studies with 20 or fewer patients). Almost all of these was meaningfully underpowered to show substantive clinical differences.

In reality, alone or in combination, the studies were really inadequate to say anything meaningful about the effectiveness of TMS treatment. This was exacerbated by the variability in the studies: they meaningfully differed in treatment dose, duration and the meta-analysis even included one study, the only one with more than 50 patients, where TMS was applied with a substantially different frequency and to the opposite side of the brain.

Surprisingly given these limitation, the paper actually found a positive effect of TMS compared to placebo: patients receiving real treatment were better off after 2 weeks of treatment. However, the authors then reported that there was no evidence of efficacy at two week follow up. This finding was prominent in the abstract but with no mention that is was based on only 3 studies, of only 6, 22 and 35 patients. The authors then concluded that there was “insufficient evidence to support the use of rTMS on the treatment of depression” in what would be the most read part of the paper, the conclusion of the abstract.

This would be fine if the results of this study were interpreted appropriately: it showed that there was insufficient evidence to support the antidepressant use of TMS at that time. However, there was certainly a really interesting signal that this could prove to be a highly novel antidepressant strategy. If the field wanted to progress, much more substantive trials were needed.

However, this meta-analysis was not interpreted this way. Repeatedly I saw it interpreted as having found that TMS did not work, not that the studies were at that time inconclusive. This had important consequences.

When submitting grant applications to obtain funding to hopefully conduct some of these more substantial studies, I repeatedly received rejections citing this meta-analysis as evidence that TMS did not work and therefore did not warrant the conduct of further research. I was persistent enough (read bloody minded perhaps) to push on long enough until I eventually had some success and soon saw the real world clinical impact of TMS on the lives of patients with depression but I wonder how many studies failed to progress because of this issue around the world. For at least a decade after this paper was published, despite TMS being approved for use in the US and being used around the world, I was still hearing people say that this one meta-analysis proved that TMS was not effective, just a waste of time.

So where does this leave us? It leaves us in a place where we need to be cautious. These studies should be done as carefully and rigorously as all research and should not be the subject of subtle preferential publication to boost citation metrics for journals. Reviewers, editors and most importantly authors need to be vigilant to carefully and honestly present findings and be especially cautious with the publication of underpowered analyses: the question needs to be asked if the analysis is really justified. Consider whether you would be better off actually trying to collect meaningful data to help prove whether something works or not instead of publishing an underpowered and misleading meta-analysis.

Paul Fitzgerald is a Professor of Psychiatry specializing in brain stimulation and neuroscience applications to depression, schizophrenia @ other disorders