Response to my complaint to the Cochrane Governing Board about the Cochrane review of Exercise for chronic fatigue syndrome

I received a response to my complaint to the Governing Board and to the Cochrane Library Oversight Committee about the Exercise for CFS review from David Tovey.   I should have submitted my complaint to the Cochrane Library Oversight Committee (CLOC) first, but has never been clear how to do this.  The only information about them is here and here.  They were “between chairs” for quite a while and I had actually forgotten they existed until I had already made the complaint to the Board.  I also tried approaching the Cochrane Council (who are supposed to represent the membership of Cochrane, including patients and the public), but they weren’t interested.  Then, in desperation, I submitted a complaint to the Governing Board at the end of November.  I was informed on 17th December that as it was an “editorial content issue”, David Tovey would deal with it.  As my complaint included references to my previous interactions with David on the matter, and was essentially a complaint about him and his management of this review, this seems unfair both on him and me.  Anyway, I was asked to keep people informed of the complaint, so below is the response (in italic).  My original complaint is in bold.

6th January 2019

Dear Caroline,

RE: Cochrane Review: Exercise in Chronic Fatigue Syndrome

Thank you for your complaint, sent to the Governing Board and Cochrane Library Oversight Committee. As the matters you raise relate to content, and I am responsible for all the content produced by Cochrane, I have been asked to respond.

Given the public nature of this complaint, it is important to review the actions taken by me, my team and the Editors of the Common Mental Disorders (CMD) Group over the past 12 months.

An existing review of aggregate data was first published in 2004 and has been updated several times since then. It has also attracted frequent feedback through the formal Cochrane Library system. Cochrane welcomes feedback and expects that authors respond to all substantive comments and this has happened in relation to this review. In addition, in February 2018, shortly before his death, Robert Courtney submitted a formal complaint to Cochrane’s Editor in Chief about the existing review, building on feedback he had previously submitted, and to which the author team had responded. The Cochrane Editorial and Methods Department (EMD) team led by the Editor in Chief assessed the complaint and judged that it raised important issues that the review author team still needed to address.

As you know, Cochrane had previously accepted a title request and published a protocol relating to a review of ‘Exercise in Chronic Fatigue Syndrome’ based on individual participant data. In the latter half of 2017 we undertook peer review and a detailed assessment of a submitted review. Following this, and following a discussion with two of the review authors and a re-submission, we made the unanimous decision, based on our assessment and the peer review comments that we had received, that the review should not proceed through Cochrane. The review protocol has now been withdrawn.

The Courtney complaint and the EMD team’s written assessment was sent to the author team of the published aggregate review in July 2018. Shortly after this, we received a response agreeing to address the issues raised by the complaint but explaining that this might be delayed due to personal reasons. In the light of this, in September, as there had been no further update from the author team, and to give them time without pressure to address the issues raised fully, the Co-ordinating Editor of the CMD Group and Editor in Chief agreed jointly that it would be appropriate to withdraw the existing aggregate data review temporarily pending an agreed response to the complaint. An email explaining this was sent on 10^th September.

In a subsequent meeting with two of the authors, who opposed the planned withdrawal, we agreed to suspend the withdrawal of the review pending an assessment of an imminent re-submission, but an editor’s note was placed on the review to guide the reader.

The revised version of the review was re-submitted as promised and has been re-assessed and peer reviewed. This process continues, and it would be inappropriate to comment further on this pending decisions on the future status of the review. It should be noted that the review remains one that addresses an important and relevant question but it is now more than four years out of date.

In the following section, I respond separately to the criticisms made within this complaint. However, in closing I would like to make several summary points:

1. This is an important review in a highly contested and challenging area of medicine and health. It is apparent that there are individuals and groups with differing interpretations of the evidence, and strongly held views. Cochrane values its independence and therefore our aim in relation to these reviews has been to focus solely on the science in pursuit of the highest quality review of the evidence possible. The Cochrane feedback system exists so that individuals can challenge or otherwise provide feedback on our reviews. We consider that it is important that substantive criticism is addressed appropriately on its merits.
2. The work on this review over the past 18 months has been led at a senior level within the EMD team by the Editor in Chief, the Co-ordinating Editor of the CMD Group and the Senior Editor of the Cochrane Brain, Nerves and Mind Network. We have worked together as a team, and to the greatest degree possible have sought consensus and shared decision making. However, the Editor in Chief retains overall responsibility for the editorial decisions made.
3. Cochrane has a duty of care to its contributors and to its target audiences, including evidence users and decision makers. In our work we have tried to reflect this balance, whilst respecting that the pursuit of a high quality, robust and useful review is the overriding aim.
4. The Cochrane editors have sought independent methodological and content assessment and advice at all stages, as appropriate, and our actions have been consistent with that feedback. We regard this as necessary to ensure the integrity of the editorial process.

My complaint text in bold; David Tovey’s responses in italic

1. Avoidable ideological and reputational conflicts of interest – including protecting the reputation of Cochrane over protecting patients – has led to the publication of a review which says, wrongly, that Graded Exercise Therapy (GET), is safe and effective for people with ME/CFS. There is in fact no good evidence that it is effective, and some evidence that it’s harmful.

This Cochrane review was initially published in 2004 following the usual rigorous editorial process and in line with the expected standards of the time including peer review. It has been updated several times since. It concludes that for one outcome, fatigue, there is benefit (low to moderate quality evidence) and that in the trials, reports of serious harm were rare. For other outcomes the certainty of the evidence was low or very low. The Abstract of the review explicitly states that data on harms are ‘sparse’.

As the report above details, the criticisms included within Robert Courtney’s complaint were taken seriously by the Editorial and Methods Department (EMD) review quality team and the Common Mental Disorders (CMD) Group editors. This has led to actions that are ongoing.

2. The review authors chose to look only subjective outcomes when none of the included studies could be blinded. They took advice on the conduct of the review from Peter White, PI of the main included study, the discredited PACE trial (https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(11)60096- 2/fulltext). He is acknowledged and thanked for his advice on the review.

The limitations of the reliance on subjective outcomes in trials that were of necessity unblinded are recognised in the review and are reflected appropriately in the GRADE Summary of Findings tables. It is commonplace for trialists to be consulted during the review process. There is nothing in Cochrane guidance to prohibit this.

3. Using only subjective outcomes in unblindable trials is terrible methodology. This is even more inexplicable in a review. And avoidable. Some of the included trials, including PACE, did measure objective outcomes, such as employment levels. They didn’t support the positive findings from subjective measures.

The review was conducted in a manner that was consistent with the review protocol. In fact, objective measures such as resource use were reported when they were identified. To describe this as ‘terrible methodology’ is simply an opinion and it is not shared by independent methodologists who we consulted, or the Cochrane Handbook. It is common for Cochrane Reviews to use patient-reported outcomes as primary outcomes for chronic conditions. When the primary complaint is a private experience (e.g., pain, fatigue, anxiety) the most appropriate outcome is frequently the direct observation and report of patient experience by the patient, for whom it matters most. Obviously, there are limitations and risk of bias

4. Peter White and co-authors of the PACE trial developed and use GET in their clinical practice, in accordance with their belief in the biopsychosocial model of treating ME/CFS. They also had financial ties to the insurance industry. The PACE trial was partially funded by the Department for Work and Pensions and happily for them, the “moderately positive” results of the PACE trial on these subjective measures meant they could stop people’s benefits if they refused to have GET. There were many other well-documented flaws in the conduct of PACE (outcome switching, changing measurement boundaries, not declaring conflict of interest in patient information sheets, marketing GET as an effective treatment to study participants….and many more). Cochrane judged it as a “well-conducted trial” in the Cochrane Clinical Answer of treatments for ME/CFS

A Cochrane review aims to synthesise the body of evidence and to ascribe the results levels of certainty across comparisons and outcomes. It is not unusual in scientific research for trialists or trial sponsors to have vested interests in the outcome of a study. The White et al (PACE) study was subject to the usual risk of bias and certainty evaluations as part of the standard peer review process.

Cochrane reviews are not intended to make recommendations. Evidence from a Cochrane review is likely to be one of a number of sources of information considered by decision-makers. Any use of the review to control or limit ‘benefits’ is therefore clearly outside the responsibility of the Cochrane Review or Cochrane generally.

5. The PACE trial is now being used as a teaching example of how not to do a randomised trial

Please see above. Many of the criticisms of the PACE study, including reported COI of the researchers, and changing outcome measures prior to analysis of the results, are not unique to this study.

6. The management of the review by the Common Mental Disorders Group underlines the biased assumption that ME/CFS can be treated/cured using treatments designed around the biopsychosocial model for treating a mental illness. ME/CFS is not a mental illness.

The placement of the CFS suite of reviews within the portfolio of the CMD Group does not imply anything about causation, which was and remains unknown. Having reflected on the evolving understanding of this condition as well as related feedback however, we have jointly determined that the CFS should be transferred to another group, and a process to facilitate this has already been initiated. This should not be seen in any way as criticism of the CMD editors whose independence and methodological expertise has been invaluable. The CMD editors fully support the move.

5. A person with ME has been in correspondence with David Tovey and Rachel Churchill since before the review was originally published in 2014 warning of the huge problems with it. There was no apparent involvement of patients in the review, despite offers of input well in advance of publication. As I am in friendly correspondence with this person, I can supply you with all the correspondence and notes from meetings with David Tovey on request.

[NB 7.1.19: I should have made clear that I have never been privy to the content of any discussions between this person and David Tovey, and so could not and would not supply any correspondence or notes without both parties’ permission. I apologise for any misunderstanding]

As Editor in Chief, I have engaged in respectful communication on this issue with several individuals. From my perspective, these discussions are, and will remain, confidential.

8. Cochrane Editors ignored the detailed feedback on the huge methodological flaws and conflict of interest influencing the conduct of the review, and the conflicts of the authors of included studies, preferring to bat it back to the review authors several times via the feedback system.

The feedback system has been used appropriately in respect of this review. As Cochrane editors, we always encourage feedback through the formal mechanism on the Cochrane Library. In respect of this review, criticisms and comments have been published and the authors have been asked to respond. It is the authors’ responsibility to respond appropriately to substantive criticism.

9. One patient (Robert Courtney) who provided detailed, rigorous and irrefutable criticism via the feedback system once the review was published received such a poor response from the author he made an official complaint. This is now being investigated by David Tovey. Robert sadly took his own life earlier this year before knowing that any action was being taken. I am not intending to imply there is a causal link, I am stating a fact. The effort Robert took to assemble the feedback and the complaint cannot be overestimated, and there would have been a huge physical and emotional toll over the years of trying to get an adequate response from Cochrane.

Mr Courtney’s complaint was treated seriously by the Cochrane editors and as you indicate work is ongoing as a consequence of the editorial assessment of this complaint.

10. I have been corresponding with David Tovey for some time on this and other issues to no avail. The last time he wrote to me in early September he said this issue in particular was a high priority having spoken to the investigative journalist David Tuller, and to the person with ME I mentioned in point 7.

David Tuller has been covering the flawed PACE trial for some time and has also specifically written about the Cochrane review – examples below

http://www.virology.ws/2018/10/19/trial-by-error-cochrane-decides-to-withdraw-flawed-exercise-review/

http://www.virology.ws/2018/09/03/trial-by-error-the-cochrane-controversy/

http://www.virology.ws/2018/08/13/trial-by-error-open-letter-to-the-lancet-version-3-0/#more-11316

We fully recognise the importance of this review and of working to ensure that it provides the best possible summary of the evidence. This work continues. The Cochrane editors have taken a balanced and cautious approach that focusses solely on the quality of the science. This has meant that some decisions may not have been made as rapidly as some commentators may wish but, in this case, we judge that this is appropriate.

11. In August 2018 the BMJ wrote an article about PACE https://www.bmj.com/content/362/bmj.k3621 with 14 rapid responses https://www.bmj.com/content/362/bmj.k3621/rapid-responses

As Cochrane editors we have a responsibility to make decisions on the basis of the science, not opinions, however, strongly held.

12. In October 2018, David Tovey said he was going to withdraw the review pending investigation. But this investigation turned out to be sending it back to the authors again. Inaccurate coverage by Reuters said the withdrawal, which never happened, was in response to patient complaints https://uk.reuters.com/article/us-health-chronicfatigue-dispute/exclusive-science-journal-to-withdraw-chronic-fatigue-review-amid-patient-activist-complaints-idUKKCN1MR2PI . In the article, David denied it was due to patient pressure, but not before various clinicians and medical establishment figures who hadn’t bothered to read the review or look into the history back-lashed against patients calling them anti-science. It seems that David Tovey has again caved to pressure from powerful academics and has not withdrawn the review as he promised. He has put a note on it saying the authors are working on another response to complaints. I think this has been possibly worse than doing nothing at all.

This exemplifies some of the challenges that the Cochrane team has experienced in relation to the follow-through on the Courtney complaint. This work continues, and the team’s focus is on the science and not being inappropriately influenced by individuals on either side of the long standing disputes around this condition and review.

13. Why did the critique of the HPV review prompt an “urgent investigation”, but Mark Vink’s critique of this review didn’t? https://journals.sagepub.com/doi/full/10.1177/2055102918805187.

The investigation into Robert Courtney’s complaint was very similar in its rigour and the importance ascribed to it to the response to the Jorgensen et al HPV article. The Vink feedback has been treated as important and has been published alongside the review. Its contents will inform the ongoing discussions and assessment.

14. There was a three hour session in Parliament on research into treatments for ME/CFS in June, and the MP Carol Monaghan stated that when the full truth about the PACE Trials comes out it will be one of the biggest medical scandals of the 21st Century https://www.parliamentlive.tv/Event/Index/3d63c39d-18f3-4cd7-8509-c91785dced98.

Please see earlier responses: Cochrane editors and authors are focusing solely on the science and the evidence and should not be swayed unduly by opinion of any sort. Cochrane strongly values its independence from political interference.

15. Now all the PACE authors, plus a prominent member of Cochrane and proponent of exercise as a useful treatment for CFS, are writing a review of exercise therapy for CFS using individual patient data. This review has not yet been published because of scathing peer and consumer comments on the draft. https://www.cochrane.org/CD011040/DEPRESSN_exercise-therapy-chronic-fatigue-syndrome-individual-patient-data

The review has been rejected following critical peer review and detailed editorial assessment involving the EMD and CMD editors, who were in unanimous agreement on the decision. It may be noted however that IPD reviews generally include the trialists as part of the author team and that this is accepted practice.

This review has had a catastrophic effect on people with ME who are still being forced to have an ineffective and potentially harmful treatment. Imagine if it was a drug or a vaccine with even a fraction of the evidence of harm, combined with such poor evidence of benefit? People assume that Cochrane is independent, has high scientific standards and is vigilant and robust about dealing with conflict of interest. In my view, and certainly in this case, this assumption is incorrect.

As described above, it is inappropriate to link actions such as denying benefits with the outcome of a systematic review. Cochrane bears no responsibility for such decisions.

This review should be retracted and Cochrane should issue a full unreserved apology to patients.

The review is currently subject to editorial assessment, and at the time of writing no decision has yet been made in respect to its future status.

I hope that this response sufficiently reflects our commitment to ensuring that critical comments relating to this review, and indeed all our reviews, are taken seriously and addressed appropriately by the appropriate author team.

Yours sincerely,

David Tovey

Editor in Chief

Cochrane