On 24 September 2018 Hilda Bastian wrote a blog about bias called Scientific Advocacy and Biases of the Ideological and Industry Kinds in response to an upheaval earlier in the month when the Cochrane Collaboration expelled founder member Peter Gøtzsche.
I responded to her post by pointing out the non-industry bias should be taken more seriously, and attempted to get her to look at the case of the Cochrane review Exercise therapy for chronic fatigue syndrome.
An anonymous person “ES” expanded considerably on my point about ideological bias, and the harm Cochrane have done by not taking valid criticism – which could only be made after the review had been published – seriously enough.
I have extracted only the comments on the blog which refer to the Cochrane exercise review.
Caroline Struthers September 24, 2018 at 12:00 am (Comment 330)
I think non financial conflicts should be taken just a seriously as financial ones. The damage already done to ME patients by the PACE trial acknowledged to be scandalously poor science due to reputational and ideological interests is tragic. Yet Cochrane has given it a clean bill of health despite an unprecedented level of criticism. Just because it’s not a drug doesn’t mean patients can’t be seriously harmed by a treatment which powerful academics have built their careers on
Hilda Bastian September 27, 2018 at 12:00 am (Comment 339)
I agree that non-financial interests can do as much harm as financial ones. It’s just that it’s so hard to even get agreement on the parameters of what constitutes the boundaries of financial conflict – and that can be objective. I think it could be that as time goes on, ways to do all of this better become clearer. I hope so.
Caroline Struthers September 27, 2018 at 12:00 am (Comment 343)
It’s too late for ME patients though. And despite the very obvious conflicts of the PACE triallists and obvious methodological flaws in their trial and all the other included studies Cochrane has failed patients by completely ignoring patient and methodological concerns before the review was published and then ignoring them afterwards too. More than failing. It’s betrayal. You can’t even complain officially because the conflict of interest policy doesn’t cover non-financial conflicts however objectively obvious they are
ES September 28, 2018 at 12:00 am (Comment 344)
I’d like to expand on some of the above concerns, and provide some links for those interested in checking the details for themselves.
One way of mitigating the harm done by ideological conflicts of interest is by genuinely committing to post-publication review, and ensuring that concerns raised about Cochrane reviews are taken seriously. In this area, Cochrane has failed.
A patient identified a number of problems with Cochrane’s review of exercise therapy for CFS (the PACE trial was the largest trial included in this review), and submitted his careful and clear comments explaining these problems from 16th April 2016. His comments, and the replies from Lillibeth Larun, can be found from p123 of the re-published review: https://www.dropbox.com/s/koehut6iw2bm9v5/Larun_et_al-2017-The_Cochrane_Library.pdf?dl=0
Larun’s responses should never have been considered acceptable by Cochrane, and I question whether anyone who has taken the time to read them believes that they can be defended. While Robert Courtney passed away earlier this year, and so will not be able to receive the apology that he deserves, Cochrane as an institution should still apologise to the patients affected by their failure to adequately supervise Larun’s work.
Cochrane’s work has played an important role in slowing progress with debates about the quality of research into potential treatments for CFS, a matter where intellectual and financial COIs can be important. Cochrane has also played a role in leading to patients concerned about the quality of evidence supporting claims about the efficacy of CBT and GET being treated unfairly by researchers, doctors and disability assessors. The now quite widely known PACE trial is a useful example of this.
In addition to their widely known ideological biases in favour of the treatments that they developed, all three of the PACE trial’s principal investigators also declared COI’s involving the insurance industry. Despite claiming that PACE was conducted in accordance with the declaration of Helsinki, their COIs were not declared to potential trial participants, violating the declaration of Helsinki: http://www.virology.ws/2015/11/17/trial-by-error-continued-pace-teams-work-for-insurance-companies-not-related-to-pace-really/ Should this matter?
Regardless of the many problems with PACE that have now been recognised (and which led to an entire special edition of the Journal of Health Psychology), Cochrane allowed itself to be used by the PACE trial researchers to fight against access to trial data and evade important questions about the problems with their work.
QMUL university refused an FOI request to release results for the outcomes pre-specified in the PACE trial’s published protocol, claiming that this would be too expensive to do as “there is no longer a statistician employed by the PACE trial, one would need to be recruited for this operation and trained.” https://www.whatdotheyknow.com/request/pace_trial_recovery_rates_and_po_2#incoming-463991
They then spent £250k on lawyers fees to fight against a request for the anonymised data required to calculate these prespecified outcomes: https://johnthejack.com/2016/06/29/using-public-money-to-keep-publicly-funded-data-from-the-public/
Cochrane provided them important cover during their fight. A statement from QMUL claimed that:
“We have supplied requested, anonymised data to independent scientists, as part of normal research collaboration. Among others, these include a Cochrane review group who have independently validated the trial findings on the primary outcomes, and who are now undertaking an individual patient data meta-analysis using these data.” https://www.qmul.ac.uk/media/news/items/smd/168729.html
When a PACE trial PI was questioned in court about whether this sharing with ‘independent scientists’ showed that the requested data could be released, she then argued that: “disclosure to the Cochrane review does not count as disclosure to independent scientists as all three of the PACE principal investigators sat on the review panel.” http://informationrights.decisions.tribunals.gov.uk//DBFiles/Decision/i1854/Queen%20Mary%20University%20of%20London%20EA-2015-0269%20(12-8-16).PDF
Indeed, a PACE trial PI’s academic fund paid for one of the meetings for developing the protocol for a planned Cochrane IPD review, and the PACE researchers stated that an outcome of their trail was: “We have provided individual patient data for this Cochrane group meta-analysis, as well as our active involvement in design and writing.” https://gtr.ukri.org/projects?ref=G0200434
Fortunately the Information Tribunal did not fall for the claims used to attempt to justify the refusal to release PACE trial data, and when the released data was analysed it served to further illustrate how justified the concerns about this work were. Two papers have now been published which used the analyses pre-specifed in the trial’s protocol and found that:
1) “The claim that patients can recover as a result of CBT and GET is not justified by the data, and is highly misleading to clinicians and patients considering these treatments.”
2) “These findings raise serious concerns about the robustness of the claims made about the efficacy of CBT and GET. The modest treatment effects obtained on self-report measures in the PACE trial do not exceed what could be reasonably accounted for by participant reporting biases.”
https://bmcpsychology.biomedcentral.com/articles/10.1186/s40359-018-0218-3
Given the clear problems with PACE, and many of the other trials included in Cochrane’s review, it is doubtful that these researchers would have agreed to share data with those who did not share their own ideological biases. It is worrying that Cochrane appears to have allowed themselves to be used for this selective form of data sharing and ‘validation’.
While the problems with Larun’s work are illustrated by her earlier Cochrane review, and her responses to Courtney, I also wish to note that the final author of the IPD protocol appears to suffer from his own ideological distortions. Paul Glasziou has long claimed that there was good evidence exercise therapy was useful for CFS, and was the co-author of a book which argued that CFS was a ‘self-perpetuating disease’ where the self-perpetuating process “creates fatigue, lack of physical activity and leads to an ongoing sense of fatigue”. It is not made clear what evidence is thought to justify this claim. When David Tuller’s investigation into the PACE trial attracted wider concern from within academia Glasziou attempted to dismiss his work as ‘ad hominem’ (although provided no reason for this claim): https://twitter.com/PaulGlasziou/status/658488688091922432 Even after data from the PACE trial showed how the researchers had spun their ‘recovery’ results, Glasziou has gone on promoting articles which use their misleading recovery figures on social media, eg: https://twitter.com/PaulGlasziou/status/700078726826971136 I can see why the PACE researchers would be happy to have him providing an ‘independent’ review of their research, just as I can see why others would see him as unfit to do such work.
Important institutions whose reputations are tied to the PACE trial continue to try to use Cochrane’s work as an excuse for not addressing criticisms made of the trial’s design, execution and analysis, eg:
https://mrc.ukri.org/news/browse/criticism-of-the-pace-trial/ When Cochrane fails to take post-publication peer review seriously, leaving the misleading claims of Cochrane authors uncorrected, there is good reason to be concerned about any potential distortions of thought that review authors may suffer from.
While the BMJ has a history of misrepresenting the controversy surrounding the PACE trial, they appear to be starting to cover the issue more fairly. In their most recent piece on this scandal they drew attention to Simon McGrath’s concern that concerns about problems with PACE only started to be taken seriously once academics started to speak out on behalf of patients: “for many years, researchers and the medical establishment would not engage with patients who made the same criticisms — simply because, it seems, they were patients.” https://www.bmj.com/content/362/bmj.k3621.short?rss=1
Larun’s exercise review claims to have found a “positive effect” on physical functioning from exercise therapy at follow-up, yet their own analysis (1.6) showed that there was not a significant difference between the exercise group and the passive control group. This ‘physical functioning’ outcome was based on mere questionnaire results from non-blinded trials, and during these trail participants were being told that exercise therapy had been found to be effective. Concerned patients have long attempted to draw attention to the potential problem of bias distorting subjective self-report outcomes, and the fact that more objective outcomes show less positive results: https://www.bmj.com/content/350/bmj.h227/rr-10
If such poor quality research had been misleadingly presented to justify a claim that vaccines were dangerous, I do not believe complaints about this would have been ignored. Yet when misleading claims serve the interests of well connected researchers many seem happy to dismiss concerns raised by patients like Courtney.
Why have Robert Courtney’s carefully stated concerns been left unanswered (aside from an “agree to disagree” non-reply from Larun) for two years, while Gøtzsche’s concerns were treated as urgent? If Cochrane fails to engage seriously with those who are attempting to play by their rules, including a patient devoting their limited health to trying to help ensure their fellow patients are not misled by poor quality work from Cochrane, how can Cochrane’s board then complain about others who stretch (or break) the rules? If Gøtzsche is to be expelled for his unreasonable behaviour in criticising work at Cochrane, should those at the Common Mental Disorder Group be expelled for their unreasonable behaviour in failing to adequately respond to criticism? Has their behaviour really been any better than Gøtzsche’s? Are high standards only insisted upon when behaviour troubles researchers rather than patients?
David Tuller writes that those at Cochrane are aware of the problems with their CFS work, and are slowly considering their next step: http://www.virology.ws/2018/09/03/trial-by-error-the-cochrane-controversy/ But this should have been considered a matter of urgency when concerns were first raised. The Larun exercise review was not fit to be published, and the slow recognition of this illustrates serious problems at Cochrane. Indefensible work should be withdrawn, not left to go on distorting public debates, and clinical decisions about patients’ health-care or eligibility for insurance and welfare payments.
Hilda Bastian September 28, 2018 at 12:00 am (Comment 345)
I haven’t kept up with the controversy around PACE and the Cochrane review, so I can’t comment on that. I agree it’s important, though, and I should. However, just to point out: Gøtzsche wasn’t expelled for criticizing a Cochrane review. The process had started long before he criticized the HPV vaccine review. (Disclosure: Paul Glasziou is a friend, and my PhD supervisor, but I’m not familiar with the case you’re discussing.)
ES September 28, 2018 at 12:00 am (Comment 348)
Sorry – I meant to say “while criticising work at Cochrane”, not “in criticising work at Cochrane”. It does seem that whatever behaviour of Gøtzsche’s Cochrane found unacceptable, it took place while Gøtzsche was criticising Cochrane.
It is difficult for outsiders to know exactly what behaviour of Gøtzsche’s was seen as unacceptable as I do not believe this has yet been detailed. The board’s statement said: “The Governing Board’s decision was based on an ongoing, consistent pattern of disruptive and inappropriate behaviours by Professor Gøtzsche, taking place over a number of years, which undermined this culture and were detrimental to the charity’s work, reputation and members.” To me it seems that there are important parts of the culture at Cochrane which need to be undermined.
I would certainly encourage you to look at the PACE trial and the problems with Cochrane’s CFS work, particularly given your interest in consumer engagement. While a growing number of independent researchers are now speaking out about the problems here, those who are already in positions of authority in this area tend to be those with a vested interest in maintaining the status quo. As it seems that achieving real change in medicine can be more dependent on politics than evidence or arguments, making progress here is likely to require that many more independent researchers take the time to get involved
Caroline Struthers September 28, 2018 at 12:00 am (Comment 353)
Thank you ES for a great summary of the problems with the Cochrane reviews of exercise for ME. I know of someone with ME who raised concerns before the first review was published. He offered to review it, and was turned down because of lack of time. This was many months before the first version of the review was published without any input from patients (in 2013). He has been in contact with Cochrane for FIVE years on this issue, and only now is Cochrane getting round to investigating Robert Courtney’s concerns? Hilda, I know you can’t yet comment because you haven’t kept up with it. But please could you get up to speed (probably reading David Tuller’s posts on http://www.virology.ws would be the best place to start) and then comment?
Hilda Bastian September 29, 2018 at 12:00 am (Comment 356)
Can’t promise how soon that will be – but I will. Thanks, Caroline!
Caroline Struthers October 1, 2018 at 12:00 am (Comment 361)
OK. If you are short of time start here with the review and read the feedback. https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD003200.pub7/full.
The PACE triallists are now co-authors of the forthcoming review using individual patient data, https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD011040/full. I was confused that Paul Glasziou had also appeared on the published protocol as last author. Now I have learnt about his ideological conflicts relating to ME, it all makes sense.
And he is now supervising your PhD on some factors affecting the validity of systematic reviews. Ironic
Hilda Bastian October 4, 2018 at 12:00 am (Comment 366)
I will read this, and when I can spare a few days, I will. Don’t expect it to be soon. I’ll be able to follow my own trail on this. I am not, in this, accepting any of the contentions you are making – I may or may not do so after I have looked at this thoroughly. In relation to my PhD, it’s methodological and it’s not about non-financial conflicts of interest. So it’s not actually relevant to any of this, and I think I’m really lucky to have Paul as PhD advisor
Caroline Struthers October 6, 2018 at 12:00 am (Comment 369)
Great. I’m not expecting you to look at it soon but it would be good if you did at some point. Even more ironically I asked my friend with ME – who has been trying to get Cochrane to look at the problems with these reviews for five years – to contact Peter Gotzsche thinking he would be interested and supportive. My friend wrote a long explanation of the problems and Peter wrote a short dismissive response along the lines of “exercise is good for everything”. I was extremely disappointed and somewhat angry about Peter’s response. I think I might have also contacted you to ask if you could help…can’t quite remember now. You certainly weren’t that dismissive but you weren’t in a position to help as far as I can remember. I believe I am the only former Cochranite willing to raise it – might be wrong about that.. At least now David Tovey has started an investigation (not because of me, but because of patient and patient advocate pressure over many years).
Hilda Bastian October 6, 2018 at 12:00 am (Comment 370)
It’s terrific that Tovey (Cochrane’s editor-in-chief) is investigating, and I will try to get up to speed. Yes, you did contact me, I can’t remember when that was: it’s been on the back burner for me for ages – I think it was more than a year ago.
Well, I would have been surprised if Gøtzsche had been interested: his pattern in recent years has been to pick up only on issues that involve being critical of industry. It’s what I meant in this post, by people who are anti-something not being pro-consumer.
Yes, that bias about exercise is widespread, isn’t it? I think that’s partly personal socio-economic bias, and partly because exercise studies typically don’t consider adverse effects (like injury).
Caroline Struthers October 8, 2018 at 12:00 am (Comment 376)
I will stop posting about this I promise. But I have to emphasise that Graded Exercise Therapy (GET) is not only largely ineffective for people with ME, it is often extremely harmful. I’m not talking about studies ignoring side-effects like injury, I am talking about people with ME being forced to take a useless therapy that may set them back and incapacitate them for months or years rather than help them recover. They are forced to take the therapy because it has been previously recommended by NICE and other users of Cochrane evidence. Policy makers who want to reduce the costs to the state of supporting people with ME force them to have GET because they have trusted the Cochrane review saying that GET is moderately effective and safe. The largest included study in the review in favour of GET is the PACE trial which is now being used in universities as an example of how not to do a randomised trial. When Peter Gotzsche shouted about the possible bias in the HPV review, Cochrane did an urgent investigation to exonerate themselves. Consumers and others have been questioning the CFS/ME reviews for FIVE years but largely because there was no shouty internal Cochranite like Peter Gotzsche, nothing has been done. Yes, it is great that finally David Tovey is taking it seriously. It is not Peter Gotzsche to blame for this because he only shouts about industry bias, (although that is extremely frustrating) it is Cochrane for wilfully ignoring any other form of bias.
Hilda Bastian October 8, 2018 at 12:00 am (comment 377)
I wasn’t talking about GET when I said that: I was talking about the bias about exercise generally.
I disagree that the HPV investigation was done to exonerate themselves: it was to check if a review in a volatile situation was ok to remain as it was. It’s not that unusual. I can’t speak to the situation with the CFS/ME one, but I’ll try to get a handle on it.
I don’t agree that Cochrane wilfully ignores biases other than industry bias. For a start, there’s not much that’s uniform across all Cochrane groups: it’s a set of sub-cultures, each with their own approaches. It doesn’t ignore non-financial interests as a matter of policy – even though it’s financial/commercial interests that are regarded as conflicts. The Cochrane Handbook (Part 1, 4.5) states:
Authors should report any present or past affiliations or other involvement in any organization or entity with an interest in the review that might lead to a real or perceived conflict of interest. Situations that might be perceived by others as being capable of influencing a review author’s judgements include personal, political, academic and other possible conflicts, as well as financial conflicts. Authors must state if they have been involved in a study included in the review.
Healthy Control 