Response from the Co-Chairs of the Cochrane Governing Board

On 30 April I wrote to the Cochrane Board to ask them to review David Tovey’s response to my complaint about the Exercise for Chronic Fatigue Syndrome reviews.  Their response is below.

8 May 2019

Dear Ms Struthers,

Thank you for your correspondence, which we have carefully considered.  We fully support the Editor-in-Chief and the editorial decisions he has made.  We are content that he has acted, and continues to act, in the best interests of Cochrane and in a manner that best serves the public as a whole. We are in a “period up to the end of May 2019”  during which the review in question is being amended (see note at:https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD003200.pub7/information under “What’s new”, dated 8th March 2019).  We await publication of the amended review.

Yours sincerely,

Martin Burton & Marguerite Koster, Co-Chairs, Cochrane Governing Board

Advertisements

The latest on my complaint to Cochrane about the Exercise for Chronic Fatigue Syndrome reviews

My original complaint about the Exercise for Chronic Fatigue Syndrome review was made at the end of November 2018. Although it was clearly addressed to the Cochrane Board, the Board Co-chairs decided it should be dealt with “in the first instance” by Editor-in-Chief Dr David Tovey.  I (and many others) had been in correspondence with Dr Tovey,  members of his team, and the Cochrane Library Oversight Committee about this review for many months without a satisfactory response.  So this was not a “first instance” of the complaint.  Because of the decision to pass the complaint to Dr Tovey again, I decided to submit the complaint to the Charity Commission.

On 19 December Cochrane Chief Executive officer Mark Wilson wrote to me stating that if I wasn’t happy with Dr Tovey’s response which was in preparation, I could ask the Cochrane Board to review the case.

If you are not happy with his response then you can ask the Cochrane Board to review it and the Board will decide if and how that should be done, and any final decision and response related to the complaint.

I was not happy with Dr Tovey’s response of 6 January, and responded at length with the reasons on 18 February.   Having received no acknowledgement of my letter, I wrote to the Cochrane Board on 30 April to request a review of the case.

30 April 2019

Dear Cochrane Board

cc Lucie Binder, David Tovey, Martin Burton, Marguerite Koster, Ginny Barbour, Miranda Langendam, Craig Lockwood

According to Mark’s email of 19 December below, if I am not happy with David Tovey’s response to my complaint I can ask you to review it.   David’s response and my reply are attached for reference. My letter of 18 February indicates I am not happy with his response.  Please will you review the complaint and let me know what you are planning to do?

Yours

Caroline Struthers

From: Mark Wilson <MWilson@cochrane.org>
Sent: 19 December 2018 18:19
To: Caroline Struthers <caroline.struthers@csm.ox.ac.uk>
Cc: Lucie Binder <LBinder@cochrane.org>; David Tovey <DTovey@cochrane.org>; Martin Burton <Martin.Burton@cochrane.nhs.uk>; Marguerite Koster <mkoster@cochrane.org>; Ginny Barbour <ginny.barbour@qut.edu.au>; Fergus Macbeth <Fergus.Macbeth@btinternet.com>
Subject: Re: Complaint about Cochrane Review of Exercise for CFS review

Dear Caroline,

I’m disappointed that you saw fit almost immediately on receipt of my email of 17th December – and before you had received the response to your complaint from the Editor in Chief – to publish tweets attacking Cochrane for the Co-Chairs’ decision that the complaint should be dealt with in the first instance by the Editor in Chief, and to announce through that medium that you’ve submitted a complaint to the UK Charity Commission.

Nevertheless, I am writing to inform you that, as promised, he is preparing a response because both “conflict of interest” and “scientific oversight” issues relating to Cochrane Reviews are the responsibility of the Editor in Chief, and he has to be given a formal opportunity to investigate any formal complaint about them. If you are not happy with his response then you can ask the Cochrane Board to review it and the Board will decide if and how that should be done, and any final decision and response related to the complaint.

Yours,

Mark G. Wilson

Chief Executive Officer

Featured

Reply to David Tovey’s letter about the Exercise for CFS review

Below is my reply to David Tovey’s letter to me sent on 6 January in response to my complaint about the Exercise for CFS review.  David’s letter is posted on a previous blog here.

18 February 2019

Dear Governing Board and Chair of the Cochrane Library Oversight Committee,
cc. David Tovey, Editor-in-Chief; Mark Wilson, Cochrane CEO; Co-chairs of the Cochrane Council

Re: Complaint about the Cochrane Exercise for Chronic Fatigue Syndrome Review

In the absence of a complaints procedure or a process for the independent assessment of my complaint, I expect you are aware that I also submitted it to the Charity Commission.

In the meantime, please consider the complaint and David’s response in light of the comments below which follow the same numbering system as the original.

I appreciated seeing the notes on the agenda of the Governing Board meeting which took place 19-20 January in London outlining the remit of the Cochrane Library Oversight Committee (CLOC). To quote from these notes, the remit of the Committee is to:-

Consider and where appropriate mediate on, matters of tension between the Governing Board and the Editor-in-Chief, at the request of either party, that relate to the editorial process or published content of or in The Cochrane Library.

The notes also say that membership of the CLOC should include at least one patient or patient advocate. Please can you let me know which of the current membership fit that description and how?

I am disappointed by the lack of interest shown in this issue demonstrated by the Board co-chairs passing it back to David to respond to despite it being addressed specifically to them, and my having already been in correspondence about it with David and members of the Central Executive for well over a year. There also seems to be little sense of urgency or concern for people with ME displayed by the Chief Executive Officer, the Cochrane Council (including the two patient representatives) or the Governing Board.

The recent decision by the Health Research Authority (HRA) to cite the exercise review as evidence that PACE was viewed as a well-conducted trial indicates that this is an urgent matter. In the letter to Rt. Hon Norman Lamb MP of the Science and Technology Committee, the HRA stated that “The robustness of the PACE trial has been considered in a Cochrane review that classified it as high quality”. The review is therefore continuing to influence health care policy as well as the opinions of politicians and health officials because of the failure of the authors to respond appropriately to concerns about their work, and lack of necessary action by Cochrane. An editor’s note on the review, which can only be discovered if you click through the “what’s new” link, indicates a resubmission from the authors intended to account for the concerns raised about their work still doesn’t meet the standards expected of Cochrane reviews, and so was rejected for publication. Yet this old and uncorrected version of the review is still being cited as “evidence” that PACE was a high-quality trial.

Many of the issues I am raising here were expressed in Virology Blog’s open letter of support for Cochrane on 23 October 2018, which was posted after the announcement that Cochrane had decided to temporarily withdraw the review. That letter was signed by more than 40 scientists, academics and other experts from Harvard, Stanford, Cornell, Johns Hopkins, Uppsala, University College Dublin, and many other leading institutions.

I don’t anticipate David’s response, or my comments on it which follow, will cause the necessary “tension” between the Board and the Editor-in-Chief for either party to request the impartial scrutiny of the Cochrane Library Oversight Committee. I only hope that you, as members of the Board of Trustees, will put the interests of patients and the public – the beneficiaries of the charity – first, and advise David to withdraw the Exercise for CFS review, as he stated he would do in October. I also ask, once again, for Cochrane to issue an apology to people with ME for the distress and harm caused by Cochrane’s inaction.

Before I respond to David’s answers to my complaint in turn below, I’d like to thank the many people whose work I’ve drawn upon in this response, particularly the late Robert Courtney.

1. I criticised that the conclusion of the Exercise for CFS review was that Graded Exercise Therapy (GET) is beneficial when the supporting evidence (ie. the methodological quality of the included trials) is extremely poor.

David spoke of the review being published “in 2004 following the usual rigorous editorial process and in line with the expected standards of the time including peer review”, but it is clear that there were serious problems with this process and the role of patients within it. By whose standards of the time was this process rigorous? Not by the standards of those patients who had been attempting to raise concerns about the quality of research in this area since the 1990s.

Instead of arguing that the way Cochrane’s processes were followed was acceptable, Cochrane should recognise that the problems surrounding this review indicate that urgent changes must be made to its processes to avoid causing more problems for patients.

In his reply David picked extracts from the review (not the plain language summary), possibly to demonstrate how measured the review’s conclusions had been.

“It concludes that for one outcome, fatigue, there is benefit (low to moderate quality evidence) and that in the trials, reports of serious harm were rare. For other outcomes the certainty of the evidence was low or very low. The Abstract of the review explicitly states that data on harms are ‘sparse’.”

David’s choice of extracts drew my attention to the plain language summary which has been published in ten languages. It upgrades the quality of the evidence for the benefit of exercise from “low to moderate” to “moderate” for reducing fatigue. It also states that exercise has a “positive effect” on physical function, sleep and overall health. So the plain language summary indicates that exercise is beneficial for four outcomes not one.

Readers are not informed of the fact, highlighted by Robert Courtney in one of his comments on the review, that exercise therapy at follow-up was not associated with a significant improvement for any of the pre-specified outcome measures apart from self-reported sleep. The summary of main results states that

A positive effect of exercise therapy was observed both at end of treatment and at follow‐up with respect to sleep (Analysis 1.12; Analysis 1.13), physical functioning (Analysis 1.5; Analysis 1.6) and self‐perceived changes in overall health (Analysis 1.14; Analysis 1.15).

In fact, analyses 1.6 and 1.15 show no significant effect, even though these outcomes relied on data from subjective self-report outcomes from unblinded trials. Furthermore for the review’s prespecified primary outcome of standardised mean differences in fatigue questionnaire scores, there was also a null result at follow-up, although this was not made clear to readers.

The summary also states that exercise “didn’t make symptoms worse” and “Serious side effects were rare”. This creates a different impression from what David quoted from the abstract which was that “data on harms are `sparse’”.

Considering the problems now recognised with much of the research into exercise therapy for CFS, a plain language summary that overstates benefit and is overly reassuring about potential side effects is not a good look for a Cochrane review.

2. I questioned the decision of the review authors to rely almost entirely on data from subjective outcomes when the included studies were unblinded.

David’s response was to point out that the limitations of the reliance on subjective outcomes in open label trials are recognised in the review and are reflected appropriately in the GRADE Summary of Findings tables.

So the authors recognised and documented the limitations caused by reliance on subjective outcomes in open label trials, yet they had decided to focus on subjective outcomes in their review, despite the presence of objective outcomes in the same studies? My understanding is that before the Larun review was published a patient had raised concerns about the divergence between the more positive results seen for subjective self-report outcomes, which are prone to problems with bias, and the largely null results for objective outcomes which are less prone to bias yet these concerns were left unaddressed in the review.

In the same point I questioned the decision of the authors to ask for and acknowledge the advice of the principle investigator of the PACE trial who has an obvious interest in a favourable assessment of it. David’s response was that triallists are often consulted during the review process and there is nothing in the Cochrane guidance to prohibit it.
The fact that Cochrane guidance doesn’t currently prohibit something doesn’t make it OK. In this case it seems part of an inappropriately close relationship between those who have developed and tested a proposed treatment and those who are supposed to be providing independent scrutiny of their work.

3. I stated that relying on subjective outcomes in open label trials is “terrible methodology”, and inexplicable in a review of open label trials.
David’s response was that this was “simply an opinion” and was not shared by his methodological advisors.

He explained in his response to point 2 above that the authors recognised these limitations and reported them in the appropriate points in the review. Again, this begs the question: Why did the review authors choose to limit the review outcomes to subjective ones (with the exception of healthcare usage, which for which there was a null result)?

The PACE trial measured three outcomes that the trial researchers had classed as being ‘more objective’. PACE reported that the addition of exercise therapy led to no improvement in their objective measure of fitness, or in employment outcomes. For the six minute walking test there was a statistically significant improvement with GET, but it failed to meet the criteria for a clinically significant improvement they used for their primary outcomes (0.5 of standard deviation at baseline).

Just because the review authors made the decision not to add objective measures to the protocol they inherited from previous versions of the review, doesn’t make it the right decision. In fact, it would have been easy to foresee that the omission of null results from objective outcomes would bias the reported findings of the review to the positive. Again, who were the independent peer reviewers and methodologists who approved the review authors’ decision not to analyse objective outcomes from studies they were including?

There were aspects to the conduct of the PACE trial that raise particular concern about the biasing of data for subjective self-report outcomes. A recent paper from Carolyn Wilshire summarises these issues, drawing attention to the fact that, mid-trial, “GET participants were told that ‘in previous research studies, most people with CFS/ME felt either “much better” or “very much better” with GET’ and that GET was ‘one of the most effective therapy strategies currently known’”.

It seems that such problems did not just afflict PACE, the largest study included in the Larun review. For example, the second largest trial, FINE, made use of what was described as “Rousing Reassurance”. This involved telling patients things such as “every exercise is strengthening your body”, “there is no disease” and “from the moment you walk out of this room your recovery is beginning”.

As well as potentially biasing participants’ self-report outcomes, such reassurance also seems likely to affect how staff viewed participants. There are indications that this has happened, such as a paper from the FINE trial which quotes a supervisor saying: “There have been one or two times where I have been worried because they have got angry at the patients…that anger has been communicated to the patients. Their frustration has reached the point where they sort of boiled over… there is sort of feeling that the patient should be grateful and follow your advice, and in actual fact, what happens is the patient is quite resistant and there is this thing like you know ,“The bastards don’t want to get better””

By promoting unduly positive views of the quality of research like PACE, and the benefits of exercise therapy for CFS patients, Cochrane risks playing a role in biasing the way society responds to an entire patient population.

It’s clear that many CFS patients are concerned about the low standards often applied to the research into their condition. Their views need to be taken seriously, and engaged with, rather than superseded by the opinions of those selected by Cochrane. If Cochrane has found independent methodologists who wish to defend trials such as PACE and FINE from concerns about the potential for bias distorting subjective self-report outcomes, then they should be expected to do so as part of an open discussion with both patients and those methodologists who disagree.

Professor Jonathan Edwards Emeritus Professor of Connective Tissue Medicine at UCL had this to say on the Science for ME blog.

The problem of unblinded trials with subjective end points is not ‘just an opinion’. It is the only sensible view and is embedded in all evidence-based medicine. With due respect if Tovey’s advisors do not share this view they are incompetent. Again with due respect if Tovey is not aware of this he is not qualified to act as an editor in an organisation like Cochrane. He seems to have acted fairly so far but in this instance he appears to be indicating that he has no grasp of reliable evidence. That makes Cochrane something of a basket case maybe.

The problem of subjective end points with no blinding is as basic as it gets. It is the reason we have blinded trials. No trial with this design of a drug would be taken seriously. Moreover the problem is much worse with therapist-delivered treatments, not less bad, because of the role playing and manipulation of attitudes that is inherent in the human interaction.

I am afraid this hits rock bottom.

I think an apology is due, together with an admission that the comment about ‘just an opinion’ was not appropriate. If a Cochrane official makes a statement like this it effectively destroys the Cochrane reputation entirely.

Last year 114 scientists, academics, clinicians, ten MPs and the dozens of patient and advocacy organisations signed an open letter to Dr Richard Horton, editor of The Lancet. That letter called out PACE’s “unacceptable methodological lapses”. What can you share about the expertise and justifications used by the Cochrane methodological advisors who don’t share this opinion?

David explained that it is common for Cochrane Reviews to use patient-reported outcomes. I have no problem with this when there are validated measures of core outcomes, and the outcomes are appropriate and important to patients. I do have a problem when a Cochrane review primarily relies on subjective outcomes alone when included trials cannot be blinded. If the results from relevant objective measures contradict subjective measures, as they did in the case of PACE trial, you have an additional reason for concern about the design and conduct of the trial. This review team deliberately chose to rely on subjective outcomes and downplay objective outcomes when they didn’t have to. Without a good explanation, it risks creating the impression that there was a desire to rely on outcomes which would be likely to favour the intervention.

This is an extract from the Cochrane Handbook

8.12.1 Rationale for concern about bias
In empirical studies, lack of blinding in randomized trials has been shown to be associated with more exaggerated estimated intervention effects, by 9% on average, measured as odds ratio (Pildal 2007). These studies have dealt with a variety of outcomes, some of which are objective. The estimated effect has been observed to be more biased, on average, in trials with more subjective outcomes (Wood 2008).

This is from a blog about blinding by Saul Crandon posted on 26 June 2017 on the Students for Best Evidence site (https://www.students4bestevidence.net/blinding-comprehensive-guide-students/)

What about when blinding is not possible?
In cases where blinding is not possible or feasible, the outcome measures must be objective! If you are reading a study that is unblinded, with subjective outcome measures, then you may as well stop reading it and move on. This is because, if a patient is aware they are receiving the active intervention and the outcome measure is subjective, such as ‘how much pain they are experiencing’, their reporting is likely to be biased. Knowledge of the group assignment can consciously or subconsciously cause the patient to feel better and report improved subjective pain tolerance. This is not a reliable study design and the results should not be interpreted with any certainty.

This blog is on an excellent website whose partners include the UK Cochrane Centre and many other Cochrane Centres throughout the world. The blogger seems to have a better grasp on the limitations of subjective outcomes in unblinded trials than the authors of this review, or indeed Cochrane’s independent methodological advisors

Professor Edwards also gave me permission to quote his simple illustration of why blinding matters for those who find the technical concepts difficult.

A simple test of the adequacy of a trial design is whether it is likely that a trial of an arbitrary alternative therapy, such as cranial osteopathy, with this design, would produce a spurious ‘significant’ result from systematic bias using conventional statistics. For trials such as PACE the historical evidence is that the answer is yes. Moreover, we have an example, in the SMILE trial, of an arbitrary alternative therapy for ME/CFS, designed by an osteopath (the Lightning Process), giving a positive result which apparently ‘surprised’ the principle investigator. I am embarrassed that she should be surprised since I had hoped that all our trainees would understand that this is the reason for blinding trials (which is the same as the reason for blinding trials with subjective outcomes because other trials do not need blinding).

David replied to this point by stating that the authors followed the review protocol. However, the review authors did in fact alter the way the primary outcome was assessed, changing from a planned analysis in the protocol which revealed a null results at follow-up to an analysis which allowed for largely positive results to be reported. If this poorly justified change from what was specified in the protocol could be made, then accounting for important methodological concerns raised by patients about the reliance on subjective outcomes should also have been possible.

4. I stated that the principal investigator of the PACE Trial and advisor on the review, and his co-investigators were proponents of the exercise intervention, advisors to the insurance industry, and the PACE trial was partly funded by the Department of Work and Pensions (DWP) who have never funded a trial before or since.

Encouraging the view that sick and disabled people would need less financial support if they were encouraged to adopt the right attitudes and behaviour seems to have helped secure support for recent welfare reforms. The DWP had founded these reforms on a biopsychosocial model of disability which emphasised the need for those with conditions such as CFS to engage with interventions like Graded Exercise Therapy with positive expectations. These reforms, and their impacts upon disabled people, have now been condemned by a UN enquiry for leading to “grave and systematic violations” of disabled people’s rights. At the same time, insurance companies have been refusing claims from ME/CFS patients who do not wish to undergo additional exercise therapy, with patients having reported PACE primary investigator Peter White’s personal involvement in this. White was a Chief Medical Officer at Swiss Re-insurance, and an archived copy of their coverage of his presentation to them on the PACE trial includes advice on how to use mental health exclusions to limit pay-outs to claimants with CFS.

I also outlined other flaws in the conduct of PACE which I will repeat more fully here. David didn’t explain why the review authors ignored all of them in their risk of bias assessment. I missed a few out in the first version of my complaint, and I also know there are more problems than the ones I have listed below.

• Outcomes were switched after the trial had started.
• The PACE triallists didn’t declare their conflicts of interest to the study participants, breaching the Declaration of Helsinki, which they had committed to abiding by in their protocol.
• They used the Oxford diagnostic criteria for CFS which meant that a large proportion of the trial participants may not be representative of the patient population with ME/CFS. The National Institutes of Health Pathways to Prevention Workshop on ME/CFS recently concluded that “Continuing to use the Oxford definition (of ME/CFS) may impair progress and cause harm”
• They moved the boundaries for inclusion and recovery meaning that for at least one outcome 13% of participants were simultaneously ill enough to take part in the trial, and well enough to be classed as recovered. The justification provided for this was that “approximately half the general working age population” had an SF36-PF score of under the protocol recovery cut off of 85, but the data they cited showed that this was true of only 18% of the general working age population. Six years after publication, this error remains uncorrected, though PACE data that was released under court order has revealed null results for the protocol specified recovery criteria in a 2017 paper by Wilshire et al. Wilshire’s paper also explains problems with other aspects of the PACE team’s revised recovery criteria, which appears to have been devised post-hoc without this being explicitly declared in their paper
• They decided to abandon one of the few objective outcome measures (actigraph) despite using it at baseline. When challenged, they argued that the decision was made as actometers were too burdensome for patients. The trial’s recently released Trial Management Group minutes (meeting #11 4/11/2004) show that the decision was based on then unpublished data from another research group which found that CBT for CFS was able to alter subjective self-report outcomes, but did not lead to an improvement in objectively measured activity levels. This was taken to show that actometers were “not useful for outcome”.
• They promoted the effectiveness of their favoured therapies during the trial with leaflets.

David’s response was that it is not unusual for scientific research for triallists or trial sponsors to have vested interests in the outcome of a study. I am well aware of this, and I am sure we can all agree it is nothing to celebrate. Nor is it something to gloss over as par for the course. It is something I thought Cochrane would inform readers of and attempt to actively control for. I would not expect Cochrane authors to seek and acknowledge the advice of the primary investigator of the main included study.
David goes on to say that the PACE study was subject to the usual risk of bias and certainty evaluations as part of the standard peer review process. Yet as both Tom Kindlon and Robert Courtney made clear in their comments on the review, the Cochrane risk of bias tool was not correctly applied to the PACE trial. It was rated at low risk of bias as if it had reported all results for the outcome measures specified before recruitment had begun. This was clearly not the case, and the review authors chose to ignore it. It is important to try to understand how this mistake in rating the risk of bias occurred, and why it was left uncorrected for so long after it was pointed out.

David then informed me that Cochrane reviews are not intended to make recommendations. Again, I am well aware of this. However, Cochrane promotes itself as the most reliable and trustworthy source of evidence – the “home of evidence” in fact. Therefore Cochrane has a duty to ensure this evidence is unbiased and unaffected by conflict of interest. If the results of a Cochrane review backs up a decision-maker’s motivation to save money, for example by limiting disability benefit payments to patients who refuse a treatment Cochrane evidence has indicated is effective, that source of information will be favoured by that decision-maker over any other source. I don’t think it’s appropriate to deny Cochrane has any responsibility for the decision-maker’s behaviour in this case.

5. I stated that the PACE trial is now being used as a teaching example of how not to do a randomised trial

David’s response was that conflict of interest and doing unplanned analyses after data collection was not unique to PACE. I have given examples of many other problems with the PACE trial which merit special attention. I have no problem with post-hoc analyses, if they are labelled as such, and all the pre-specified analyses are done and reported. The attitude that Cochrane reviewers can ignore these travesties of conduct and reporting because they are common in trials, particularly of psychological and other non-drug interventions is, again, deeply worrying.

6. I pointed out the problem caused by the review’s management by the Common Mental Disorders Group

David said its placement there didn’t imply anything about causation. I wasn’t suggesting it did. But it does imply that the treatment under review is assumed to be an appropriate choice for ME/CFS.

I am pleased the review is being transferred to another group. I think Neuromuscular would probably be the most suitable, however I have heard it is likely to be the Pain, Palliative and Supportive Care Group. Considering this group’s Co-ordinating Editor is also one of the two people working with David on oversight of the exercise review, I want to mention concerns that some patients have in relation to his 2015 book on physical sensations. This includes a chapter on chronic fatigue that seems to partly reflect the culture patients have been raising concerns about. The PACE trial researchers’ positive results were presented uncritically with CBT and GET described as challenging patients’ beliefs and bodily sensations. I realise that the problems with PACE were less widely discussed in 2015, but they were still present and important. If the limitations of the supposedly supporting evidence for claims about treatment efficacy are not properly explained then we risk devaluing and over-riding patients’ own responses to their illness with misguided claims about what the ‘evidence-based’ approach is. Cochrane should be operating as a bulwark against this, and speaking out against exaggerated claims made by researchers. In the case of ME/CFS it seems that low standards for evidence have been accepted even in the face of patients telling us that this is harming them. The past work of the Pain Group’s editor indicates a possible bias towards the norms of a research culture that has not served ME patients well, and which has encouraged an uncritical acceptance of work like PACE.

Is Cochrane consulting patients about this? Patients have been involved in reviewing the now thankfully withdrawn individual patient data review, so they would be ideal people to start with.

It is interesting to note that NICE put ME/CFS condition in a category of its own. https://www.nice.org.uk/guidance/conditions-and-diseases. Sensible. No doubt ME/CFS reviews will have to be shoehorned somehow into the Cochrane Review Group system even if there is nowhere suitable. The Cochrane Review Group system is often unhelpful, and confusing to the outside world. I hope at least a new review team will be recruited.

7. I said that I knew a person with ME who had been in correspondence with David Tovey and Rachel Churchill since before the review was originally published in 2014 warning of the huge potential problems with it. I asked why there was no apparent involvement of patients in the review, despite offers of help well in advance of publication. I then made the mistake of also implying I could supply copies of correspondence between David and this person. I should have made it clear that I have never been privy to the content of any discussions between them, and so could not and would not supply copies of any correspondence or notes without both parties’ permission. I apologise for any misunderstanding.

David did not answer the question about why patients were actively excluded from involvement in the review. It is Cochrane policy to involve patients in reviews as much as possible, and the common excuse trotted out by review groups for not doing so is that it’s too difficult to find “consumers” with the necessary skills. I would like an explanation of why then these offers of help were deliberately rejected. If the problems with the way patient concerns about reviews are treated are not to be examined openly, then that it is something that Cochrane need to look at internally rather than ignore.

8. I complained that Cochrane Editors seemed unconcerned by level of criticism submitted via the feedback system, or whether the author responses were appropriate or not.

David responded by saying that the feedback mechanism had worked as intended and that it was the author’s responsibility to respond. If I were Editor-in-Chief I would be deeply concerned by the inadequate and borderline rude responses of the Cochrane authors to irrefutable criticism of the review. If this is what it looks like when the feedback mechanism is working appropriately, then how bad do things need to be before it is thought to have gone wrong?

Who has the responsibility to judge if the authors’ have responded appropriately or not? It seems little oversight was provided, although I’ve been told that concerns were raised about the authors’ poor responses as soon as they were published.

9. I mentioned the case of Robert Courtney who provided detailed, rigorous and irrefutable criticism via the feedback system and received such a poor response from the author he made an official complaint.

David replied that this complaint was treated seriously by the Cochrane editors and that work is ongoing as a consequence of the editorial assessment of this complaint. This puts the responsibility entirely on those providing critical feedback to follow up if the author’s response is inadequate, and it’s only then that Cochrane takes it seriously. It was obvious to anyone who read the feedback of Courtney and others who made critical comments that the authors’ responses were inadequate.

10. I stated I have been complaining privately to David and his team about this review and other issues (such as the pointless support of reviews of homeopathy, and the inadequate control of conflict of interest) for years. I was often left with the impression that these concerns were not being taken seriously, with e-mails often receiving no response. I also pointed out the huge amount of criticism of the PACE trial by Dr David Tuller and others.

David stated that his editors have taken a balanced and cautious approach focussing solely on the quality of the science. My criticism, and that of scientists, trial methodologists, statisticians and ME experts, not to mention an extremely well-informed patient community is also focussed on the quality of the science. This has been made much more difficult by Cochrane’s failure to recognise or criticise the problems to be found in this field of research, allowing institutions that wished to dismiss concerns about PACE to do so seemingly with Cochrane’s endorsement. If Cochrane’s editors had been focussing on the quality of the science, then this review would not have been published in this form, and a lot of trouble could have been saved.

David said that the amount of time he and his team have taken to investigate complaints has been appropriate. I and many others don’t agree.

11. I mentioned an article in the BMJ about the open letter to The Lancet organized by Dr David Tuller and signed by around 200 organisations and scientists calling for a reanalysis of the PACE trial. I forgot to mention in my original complaint that there was also a whole issue of The Journal of Health Psychology (Special Issue: The PACE Trial Volume 22 Issue 9, August 2017) dedicated the PACE trial. An accompanying editorial (Special Issue on the PACE trial) described the low numbers and quality of pro-PACE submissions of the 20 people approached from the pro-PACE side. This, combined with the unwillingness of the pro-PACE authors to make revisions after peer review, meant that there were only two articles out of 17 defending PACE.

David said that Cochrane editors have a responsibility to make decisions on the basis of the science, not opinions, however, strongly held. I am not sure if he thinks the opinions, strong or otherwise, of the hundreds of people who signed the open letter to The Lancet or commented on the BMJ article, or wrote one of the many articles criticising PACE in The Journal of Health Psychology were based on anything other than the science.

Don’t you think patients would have been delighted with the results of the PACE trial if the science were sound? The acceptance of this bad science as having validity has meant the establishment of harmful policies and exaggerated claims about a treatment’s efficacy and safety. People assume that because Cochrane has not “called out” the bad science, that the science was good. Cochrane has let patients down.

12. I mentioned that David made the welcome announcement in October 2018 that he was planning to withdraw the review pending investigation. I and many others were extremely disappointed and confused that this didn’t happen.

David explained in his response that two of the authors had persuaded him not to.

He defended it by saying the team is focused on not being inappropriately influenced by individuals on either side of the long-standing disputes around this condition and review.

But being influenced by the authors is OK? It seems those now making judgements on this review within Cochrane are those who played a role in approving publication of the previous review, or who have already cited PACE uncritically in their own work. Does anyone with a history of being willing and able to speak out about the clear problems with this influential review have any role in Cochrane’s current process for oversight?

It seems that another factor which led to the review not being withdrawn was correspondence from Trygve Ottersen, Acting Director of the Norwegian Institute of Public Health, the institution that currently funds Cochrane Norway. Ottersen argued that it would be unreasonable to withdraw the review as a result of applying higher standards for the assessment of Larun’s work than for other Cochrane reviews. Yet the old standards which allowed this review to be published are difficult to defend. Standards for research should be set in consultation with patients, who should not have the low standards of others imposed upon the work which affects their lives.

It is important that what David refers to as a “duty of care” to contributors does not lead to an attempt to protect them and their work from justified criticism. Cochrane’s business model relies on attracting volunteers to conduct reviews, so there may be particular hesitancy to take action which risks embarrassing review authors or the institutions that support them. Cochrane must be willing to weather that storm if it is to avoid prioritising the interest of researchers over patients.

13. I asked why the critique of the HPV review prompted an “urgent investigation”, but Mark Vink’s critique of this review didn’t.

David’s reply here was revealing. He said that the investigation into Robert Courtney’s complaint was “very similar in its rigour and the importance ascribed to it to the response to the Jorgensen et al HPV article”. Very similar apart from speed. If Courtney’s complaint was taken as seriously and is being investigated as rigorously as the Jorgensen article, why was the investigation into the HPV article fast-tracked, whereas the investigation into Courtney’s complaint which preceded it by many months is still ongoing? This looks like discrimination in favour of protecting Cochrane’s reputation rather than in protecting patients.

14. I mentioned the growing concern of MPs about the poor treatment of people with ME as a result of the acceptance of the flawed evidence of the PACE trial. In a Westminster debate in June last year, the MP Carol Monaghan stated that when the full truth about the PACE Trial comes out it will be one of the biggest medical scandals of the 21st Century

There was subsequent House of Commons debate on 24th January 2019 with the motion

“That this House calls on the Government to provide increased funding for biomedical research into the diagnosis and treatment of ME, supports the suspension of Graded Exercise Therapy and Cognitive Behaviour Therapy as means of treatment, supports updated training of GPs and medical professionals to ensure they are equipped with clear guidance on diagnosis of ME and appropriate management advice to reflect international consensus on best practice, and is concerned about the current trends of subjecting ME families to unjustified child protection procedures.”

The motion was carried unanimously by 40 MPs with no one speaking against it.
David responded by repeating that Cochrane editors and authors are focusing solely on the science and the evidence and should not be swayed unduly by opinion of any sort. Except the opinion of Cochrane’s anonymous and independent methodological advisors, of course. Again, the implication is that opinions against PACE are not based on science. In fact this seems to be a worrying example of British science needing political pressure to take the more rigorous approach required to better serve patients.

He adds that Cochrane strongly values its independence from political interference. However, it is the influence of those already working with Cochrane who seem most likely to sway important judgements, rather than political interference. It seems that there are academics with reputations and financial interests to protect who have found Cochrane’s systems and processes amendable to influence, or else they are simply designed in a way which serves their interests.

No medical issue is going to be debated in Parliament like this without something having gone seriously wrong which is adversely affecting patients, carers, services and/or the economy. The PACE trial is a major concern which Cochrane seems to be attempting to tip-toe around, rather than taking a stand against some of the unacceptable failings that have done so much to harm patient trust.

15. My final point mentioned the publication of the protocol for an Exercise for CFS review using individual patient data (IPD) which included most of the PACE triallists in the review team, plus a prominent Cochrane member who is a proponent of exercise as a treatment for ME.

Thankfully this review protocol has been withdrawn and the review will never be published.

David remarked that IPD reviews generally include the triallists as part of the author team and that this is “accepted practice”. Again, this accepted practice is wholly inappropriate and unacceptable.

Since the IPD protocol was published, the PACE authors have used it, and their association with Cochrane, to help dismiss concerns about their work. Legitimate freedom of information requests from patients related to the PACE trial have been described as ‘vexatious’ in decision notices citing Cochrane’s work. These unjust decisions have now in turn been cited by the Health Research Authority as evidence that the PACE researchers have responded reasonably to requests for information and data from their work. Cochrane has played a pivotal role in the cascade of misinformation spread amongst UK institutions about the PACE trial and the patients who have rightly criticised it.

In summary, Cochrane is a charity whose beneficiaries are patients and the public. This is not true of academic institutions, and it certainly isn’t of publishing companies like Elsevier. Yet Cochrane has stonewalled criticism and stood on the side-lines impassively watching as patients and advocates struggle for truth and justice against old-fashioned eminence-based medicine.

Sincerely,

Caroline Struthers

Response to my complaint to the Cochrane Governing Board about the Cochrane review of Exercise for chronic fatigue syndrome

I received a response to my complaint to the Governing Board and to the Cochrane Library Oversight Committee about the Exercise for CFS review from David Tovey.   I should have submitted my complaint to the Cochrane Library Oversight Committee (CLOC) first, but has never been clear how to do this.  The only information about them is here and here.  They were “between chairs” for quite a while and I had actually forgotten they existed until I had already made the complaint to the Board.  I also tried approaching the Cochrane Council (who are supposed to represent the membership of Cochrane, including patients and the public), but they weren’t interested.  Then, in desperation, I submitted a complaint to the Governing Board at the end of November.  I was informed on 17th December that as it was an “editorial content issue”, David Tovey would deal with it.  As my complaint included references to my previous interactions with David on the matter, and was essentially a complaint about him and his management of this review, this seems unfair both on him and me.  Anyway, I was asked to keep people informed of the complaint, so below is the response (in italic).  My original complaint is in bold.

6th January 2019

Dear Caroline,

RE: Cochrane Review: Exercise in Chronic Fatigue Syndrome

Thank you for your complaint, sent to the Governing Board and Cochrane Library Oversight Committee. As the matters you raise relate to content, and I am responsible for all the content produced by Cochrane, I have been asked to respond.

Given the public nature of this complaint, it is important to review the actions taken by me, my team and the Editors of the Common Mental Disorders (CMD) Group over the past 12 months.

An existing review of aggregate data was first published in 2004 and has been updated several times since then. It has also attracted frequent feedback through the formal Cochrane Library system. Cochrane welcomes feedback and expects that authors respond to all substantive comments and this has happened in relation to this review. In addition, in February 2018, shortly before his death, Robert Courtney submitted a formal complaint to Cochrane’s Editor in Chief about the existing review, building on feedback he had previously submitted, and to which the author team had responded. The Cochrane Editorial and Methods Department (EMD) team led by the Editor in Chief assessed the complaint and judged that it raised important issues that the review author team still needed to address.

As you know, Cochrane had previously accepted a title request and published a protocol relating to a review of ‘Exercise in Chronic Fatigue Syndrome’ based on individual participant data. In the latter half of 2017 we undertook peer review and a detailed assessment of a submitted review. Following this, and following a discussion with two of the review authors and a re-submission, we made the unanimous decision, based on our assessment and the peer review comments that we had received, that the review should not proceed through Cochrane. The review protocol has now been withdrawn.

The Courtney complaint and the EMD team’s written assessment was sent to the author team of the published aggregate review in July 2018. Shortly after this, we received a response agreeing to address the issues raised by the complaint but explaining that this might be delayed due to personal reasons. In the light of this, in September, as there had been no further update from the author team, and to give them time without pressure to address the issues raised fully, the Co-ordinating Editor of the CMD Group and Editor in Chief agreed jointly that it would be appropriate to withdraw the existing aggregate data review temporarily pending an agreed response to the complaint. An email explaining this was sent on 10th September.

In a subsequent meeting with two of the authors, who opposed the planned withdrawal, we agreed to suspend the withdrawal of the review pending an assessment of an imminent re-submission, but an editor’s note was placed on the review to guide the reader.

The revised version of the review was re-submitted as promised and has been re-assessed and peer reviewed. This process continues, and it would be inappropriate to comment further on this pending decisions on the future status of the review. It should be noted that the review remains one that addresses an important and relevant question but it is now more than four years out of date.

In the following section, I respond separately to the criticisms made within this complaint. However, in closing I would like to make several summary points:

  1. This is an important review in a highly contested and challenging area of medicine and health. It is apparent that there are individuals and groups with differing interpretations of the evidence, and strongly held views. Cochrane values its independence and therefore our aim in relation to these reviews has been to focus solely on the science in pursuit of the highest quality review of the evidence possible. The Cochrane feedback system exists so that individuals can challenge or otherwise provide feedback on our reviews. We consider that it is important that substantive criticism is addressed appropriately on its merits.
  2. The work on this review over the past 18 months has been led at a senior level within the EMD team by the Editor in Chief, the Co-ordinating Editor of the CMD Group and the Senior Editor of the Cochrane Brain, Nerves and Mind Network. We have worked together as a team, and to the greatest degree possible have sought consensus and shared decision making. However, the Editor in Chief retains overall responsibility for the editorial decisions made.
  3. Cochrane has a duty of care to its contributors and to its target audiences, including evidence users and decision makers. In our work we have tried to reflect this balance, whilst respecting that the pursuit of a high quality, robust and useful review is the overriding aim.
  4. The Cochrane editors have sought independent methodological and content assessment and advice at all stages, as appropriate, and our actions have been consistent with that feedback. We regard this as necessary to ensure the integrity of the editorial process.

My complaint text in bold; David Tovey’s responses in italic

1. Avoidable ideological and reputational conflicts of interest – including protecting the reputation of Cochrane over protecting patients – has led to the publication of a review which says, wrongly, that Graded Exercise Therapy (GET), is safe and effective for people with ME/CFS. There is in fact no good evidence that it is effective, and some evidence that it’s harmful.

This Cochrane review was initially published in 2004 following the usual rigorous editorial process and in line with the expected standards of the time including peer review. It has been updated several times since. It concludes that for one outcome, fatigue, there is benefit (low to moderate quality evidence) and that in the trials, reports of serious harm were rare. For other outcomes the certainty of the evidence was low or very low. The Abstract of the review explicitly states that data on harms are ‘sparse’.

As the report above details, the criticisms included within Robert Courtney’s complaint were taken seriously by the Editorial and Methods Department (EMD) review quality team and the Common Mental Disorders (CMD) Group editors. This has led to actions that are ongoing.

2. The review authors chose to look only subjective outcomes when none of the included studies could be blinded. They took advice on the conduct of the review from Peter White, PI of the main included study, the discredited PACE trial (https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(11)60096- 2/fulltext). He is acknowledged and thanked for his advice on the review.

The limitations of the reliance on subjective outcomes in trials that were of necessity unblinded are recognised in the review and are reflected appropriately in the GRADE Summary of Findings tables. It is commonplace for trialists to be consulted during the review process. There is nothing in Cochrane guidance to prohibit this.

3. Using only subjective outcomes in unblindable trials is terrible methodology. This is even more inexplicable in a review. And avoidable. Some of the included trials, including PACE, did measure objective outcomes, such as employment levels. They didn’t support the positive findings from subjective measures.

The review was conducted in a manner that was consistent with the review protocol. In fact, objective measures such as resource use were reported when they were identified. To describe this as ‘terrible methodology’ is simply an opinion and it is not shared by independent methodologists who we consulted, or the Cochrane Handbook. It is common for Cochrane Reviews to use patient-reported outcomes as primary outcomes for chronic conditions. When the primary complaint is a private experience (e.g., pain, fatigue, anxiety) the most appropriate outcome is frequently the direct observation and report of patient experience by the patient, for whom it matters most. Obviously, there are limitations and risk of bias

4. Peter White and co-authors of the PACE trial developed and use GET in their clinical practice, in accordance with their belief in the biopsychosocial model of treating ME/CFS. They also had financial ties to the insurance industry. The PACE trial was partially funded by the Department for Work and Pensions and happily for them, the “moderately positive” results of the PACE trial on these subjective measures meant they could stop people’s benefits if they refused to have GET. There were many other well-documented flaws in the conduct of PACE (outcome switching, changing measurement boundaries, not declaring conflict of interest in patient information sheets, marketing GET as an effective treatment to study participants….and many more). Cochrane judged it as a “well-conducted trial” in the Cochrane Clinical Answer of treatments for ME/CFS

A Cochrane review aims to synthesise the body of evidence and to ascribe the results levels of certainty across comparisons and outcomes. It is not unusual in scientific research for trialists or trial sponsors to have vested interests in the outcome of a study. The White et al (PACE) study was subject to the usual risk of bias and certainty evaluations as part of the standard peer review process.

Cochrane reviews are not intended to make recommendations. Evidence from a Cochrane review is likely to be one of a number of sources of information considered by decision-makers. Any use of the review to control or limit ‘benefits’ is therefore clearly outside the responsibility of the Cochrane Review or Cochrane generally.

5. The PACE trial is now being used as a teaching example of how not to do a randomised trial

Please see above. Many of the criticisms of the PACE study, including reported COI of the researchers, and changing outcome measures prior to analysis of the results, are not unique to this study.

6. The management of the review by the Common Mental Disorders Group underlines the biased assumption that ME/CFS can be treated/cured using treatments designed around the biopsychosocial model for treating a mental illness. ME/CFS is not a mental illness.

The placement of the CFS suite of reviews within the portfolio of the CMD Group does not imply anything about causation, which was and remains unknown. Having reflected on the evolving understanding of this condition as well as related feedback however, we have jointly determined that the CFS should be transferred to another group, and a process to facilitate this has already been initiated. This should not be seen in any way as criticism of the CMD editors whose independence and methodological expertise has been invaluable. The CMD editors fully support the move.

5. A person with ME has been in correspondence with David Tovey and Rachel Churchill since before the review was originally published in 2014 warning of the huge problems with it. There was no apparent involvement of patients in the review, despite offers of input well in advance of publication. As I am in friendly correspondence with this person, I can supply you with all the correspondence and notes from meetings with David Tovey on request.

[NB 7.1.19: I should have made clear that I have never been privy to the content of any discussions between this person and David Tovey, and so could not and would not supply any correspondence or notes without both parties’ permission. I apologise for any misunderstanding]

As Editor in Chief, I have engaged in respectful communication on this issue with several individuals. From my perspective, these discussions are, and will remain, confidential.

8. Cochrane Editors ignored the detailed feedback on the huge methodological flaws and conflict of interest influencing the conduct of the review, and the conflicts of the authors of included studies, preferring to bat it back to the review authors several times via the feedback system.

The feedback system has been used appropriately in respect of this review. As Cochrane editors, we always encourage feedback through the formal mechanism on the Cochrane Library. In respect of this review, criticisms and comments have been published and the authors have been asked to respond. It is the authors’ responsibility to respond appropriately to substantive criticism.

9. One patient (Robert Courtney) who provided detailed, rigorous and irrefutable criticism via the feedback system once the review was published received such a poor response from the author he made an official complaint. This is now being investigated by David Tovey. Robert sadly took his own life earlier this year before knowing that any action was being taken. I am not intending to imply there is a causal link, I am stating a fact. The effort Robert took to assemble the feedback and the complaint cannot be overestimated, and there would have been a huge physical and emotional toll over the years of trying to get an adequate response from Cochrane.

Mr Courtney’s complaint was treated seriously by the Cochrane editors and as you indicate work is ongoing as a consequence of the editorial assessment of this complaint.

10. I have been corresponding with David Tovey for some time on this and other issues to no avail. The last time he wrote to me in early September he said this issue in particular was a high priority having spoken to the investigative journalist David Tuller, and to the person with ME I mentioned in point 7.

David Tuller has been covering the flawed PACE trial for some time and has also specifically written about the Cochrane review – examples below

http://www.virology.ws/2018/10/19/trial-by-error-cochrane-decides-to-withdraw-flawed-exercise-review/

http://www.virology.ws/2018/09/03/trial-by-error-the-cochrane-controversy/

http://www.virology.ws/2018/08/13/trial-by-error-open-letter-to-the-lancet-version-3-0/#more-11316

We fully recognise the importance of this review and of working to ensure that it provides the best possible summary of the evidence. This work continues. The Cochrane editors have taken a balanced and cautious approach that focusses solely on the quality of the science. This has meant that some decisions may not have been made as rapidly as some commentators may wish but, in this case, we judge that this is appropriate.

11. In August 2018 the BMJ wrote an article about PACE https://www.bmj.com/content/362/bmj.k3621 with 14 rapid responses https://www.bmj.com/content/362/bmj.k3621/rapid-responses

As Cochrane editors we have a responsibility to make decisions on the basis of the science, not opinions, however, strongly held.

12. In October 2018, David Tovey said he was going to withdraw the review pending investigation. But this investigation turned out to be sending it back to the authors again. Inaccurate coverage by Reuters said the withdrawal, which never happened, was in response to patient complaints https://uk.reuters.com/article/us-health-chronicfatigue-dispute/exclusive-science-journal-to-withdraw-chronic-fatigue-review-amid-patient-activist-complaints-idUKKCN1MR2PI . In the article, David denied it was due to patient pressure, but not before various clinicians and medical establishment figures who hadn’t bothered to read the review or look into the history back-lashed against patients calling them anti-science. It seems that David Tovey has again caved to pressure from powerful academics and has not withdrawn the review as he promised. He has put a note on it saying the authors are working on another response to complaints. I think this has been possibly worse than doing nothing at all.

This exemplifies some of the challenges that the Cochrane team has experienced in relation to the follow-through on the Courtney complaint. This work continues, and the team’s focus is on the science and not being inappropriately influenced by individuals on either side of the long standing disputes around this condition and review.

13. Why did the critique of the HPV review prompt an “urgent investigation”, but Mark Vink’s critique of this review didn’t? https://journals.sagepub.com/doi/full/10.1177/2055102918805187.

The investigation into Robert Courtney’s complaint was very similar in its rigour and the importance ascribed to it to the response to the Jorgensen et al HPV article. The Vink feedback has been treated as important and has been published alongside the review. Its contents will inform the ongoing discussions and assessment.

14. There was a three hour session in Parliament on research into treatments for ME/CFS in June, and the MP Carol Monaghan stated that when the full truth about the PACE Trials comes out it will be one of the biggest medical scandals of the 21st Century https://www.parliamentlive.tv/Event/Index/3d63c39d-18f3-4cd7-8509-c91785dced98.

Please see earlier responses: Cochrane editors and authors are focusing solely on the science and the evidence and should not be swayed unduly by opinion of any sort. Cochrane strongly values its independence from political interference.

15. Now all the PACE authors, plus a prominent member of Cochrane and proponent of exercise as a useful treatment for CFS, are writing a review of exercise therapy for CFS using individual patient data. This review has not yet been published because of scathing peer and consumer comments on the draft. https://www.cochrane.org/CD011040/DEPRESSN_exercise-therapy-chronic-fatigue-syndrome-individual-patient-data

The review has been rejected following critical peer review and detailed editorial assessment involving the EMD and CMD editors, who were in unanimous agreement on the decision. It may be noted however that IPD reviews generally include the trialists as part of the author team and that this is accepted practice.

This review has had a catastrophic effect on people with ME who are still being forced to have an ineffective and potentially harmful treatment. Imagine if it was a drug or a vaccine with even a fraction of the evidence of harm, combined with such poor evidence of benefit? People assume that Cochrane is independent, has high scientific standards and is vigilant and robust about dealing with conflict of interest. In my view, and certainly in this case, this assumption is incorrect.

As described above, it is inappropriate to link actions such as denying benefits with the outcome of a systematic review. Cochrane bears no responsibility for such decisions.

This review should be retracted and Cochrane should issue a full unreserved apology to patients.

The review is currently subject to editorial assessment, and at the time of writing no decision has yet been made in respect to its future status.

I hope that this response sufficiently reflects our commitment to ensuring that critical comments relating to this review, and indeed all our reviews, are taken seriously and addressed appropriately by the appropriate author team.

Yours sincerely,

David Tovey

Editor in Chief

Cochrane

My complaint to the Cochrane Governing Board about the Cochrane review of Exercise for chronic fatigue syndrome

After a lot of to-ing and fro-ing and fruitless correspondence with the Cochrane Editorial Unit by me and many others over years, I was asked yesterday to submit a summary of my complaint about this review to the Governing Board of Cochrane.   This is it.

Cochrane Review: Exercise Therapy for Chronic Fatigue Syndrome

1.       Avoidable ideological and reputational conflicts of interest – including protecting the reputation of Cochrane over protecting patients – has led to the publication of a review which says, wrongly, that Graded Exercise Therapy (GET), is safe and effective for people with ME/CFS.  There is in fact no good evidence that it is effective, and some evidence that it’s harmful.

2.       The review authors chose to look only subjective outcomes when none of the included studies could be blinded. They took advice on the conduct of the review from Peter White, PI of the main included study, the discredited PACE trial (https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(11)60096-2/fulltext).  He is acknowledged and thanked for his advice on the review.

3.       Using only subjective outcomes in unblindable trials is terrible methodology.  This is even more inexplicable in a review.  And avoidable.  Some of the included trials, including PACE, did measure objective outcomes, such as employment levels.  They didn’t support the positive findings from subjective measures.

4.       Peter White and co-authors of the PACE trial developed and use GET in their clinical practice, in accordance with their belief in the biopsychosocial model of treating ME/CFS.  They also had financial ties to the insurance industry.   The PACE trial was partially funded by the Department for Work and Pensions and happily for them, the “moderately positive” results of the PACE trial on these subjective measures meant they could stop people’s benefits if they refused to have GET.  There were many other well-documented flaws in the conduct of PACE (outcome switching, changing measurement boundaries, not declaring conflict of interest in patient information sheets, marketing GET as an effective treatment to study participants….and many more).  Cochrane judged it as a “well-conducted trial” in the Cochrane Clinical Answer of treatments for ME/CFS

5.       The PACE trial is now being used as a teaching example of how not to do a randomised trial

6.       The management of the review by the Common Mental Disorders Group underlines the biased assumption that ME/CFS  can be treated/cured using treatments designed around the biopsychosocial model for treating a mental illness.  ME/CFS is not a mental illness.

7.       A person with ME has been in correspondence with David Tovey and Rachel Churchill since before the review was originally published in 2014 warning of the huge problems with it.  There was no apparent involvement of patients in the review, despite offers of input well in advance of publication.  As I am in friendly correspondence with this person, I can supply you with all the correspondence and notes from meetings with David Tovey on request.

8.       Cochrane Editors ignored the detailed feedback on the huge methodological flaws and conflict of interest influencing the conduct of the review, and the conflicts of the authors of included studies, preferring to bat it back to the review authors several times via the feedback system.

9.       One patient (Robert Courtney) who provided detailed, rigorous and irrefutable criticism via the feedback system once the review was published received such a poor response from the author he made an official complaint.  This is now being investigated by David Tovey.  Robert sadly took his own life earlier this year before knowing that any action was being taken.  I am not intending to imply there is a causal link, I am stating a fact.  The effort Robert took to assemble the feedback and the complaint cannot be overestimated, and there would have been a huge physical and emotional toll over the years of trying to get an adequate response from Cochrane.

10.   I have been corresponding with David Tovey for some time on this and other issues to no avail.  The last time he wrote to me in early September he said this issue in particular was a high priority having spoken to the investigative journalist David Tuller, and to the person with ME I mentioned in point 7.

David Tuller has been covering the  flawed PACE trial for some time and has also specifically written about the Cochrane review  – examples below

http://www.virology.ws/2018/10/19/trial-by-error-cochrane-decides-to-withdraw-flawed-exercise-review/

http://www.virology.ws/2018/09/03/trial-by-error-the-cochrane-controversy/

http://www.virology.ws/2018/08/13/trial-by-error-open-letter-to-the-lancet-version-3-0/#more-11316

11.   In August 2018 the BMJ wrote an article about PACE https://www.bmj.com/content/362/bmj.k3621 with 14 rapid responses https://www.bmj.com/content/362/bmj.k3621/rapid-responses

12.   In October 2018, David Tovey said he was going to withdraw the review pending investigation.  But this investigation turned out to be sending it back to the authors again.  Inaccurate coverage by Reuters said the withdrawal, which never happened, was in response to patient complaints https://uk.reuters.com/article/us-health-chronicfatigue-dispute/exclusive-science-journal-to-withdraw-chronic-fatigue-review-amid-patient-activist-complaints-idUKKCN1MR2PI . In the article, David denied it was due to patient pressure, but not before various clinicians and medical establishment figures who hadn’t bothered to read the review or look into the history back-lashed against patients calling them anti-science.  It seems that David Tovey has again caved to pressure from powerful academics and has not withdrawn the review as he promised. He has put a note on it saying the authors are working on another response to complaints.  I think this has been possibly worse than doing nothing at all.

13.   Why did the critique of the HPV review prompt an “urgent investigation”, but Mark Vink’s critique of this review didn’t? https://journals.sagepub.com/doi/full/10.1177/2055102918805187 .

14.   There was a three hour session in Parliament on research into treatments for ME/CFS in June, and the MP Carol Monaghan stated that when the full truth about the PACE Trials comes out it will be one of the biggest medical scandals of the 21st Century

https://www.parliamentlive.tv/Event/Index/3d63c39d-18f3-4cd7-8509-c91785dced98.

15.   Now all the PACE authors, plus a prominent member of Cochrane and proponent of exercise as a useful treatment for CFS, are writing a review of exercise therapy for CFS using individual patient data.  This review has not yet been published because of scathing peer and consumer comments on the draft.

https://www.cochrane.org/CD011040/DEPRESSN_exercise-therapy-chronic-fatigue-syndrome-individual-patient-data

This review has had a catastrophic effect on people with ME who are still being forced to have an ineffective and potentially harmful treatment.  Imagine if it was a drug or a vaccine with even a fraction of the evidence of harm, combined with such poor evidence of benefit?  People assume that Cochrane is independent, has high scientific standards and is vigilant and robust about dealing with conflict of interest.  In my view, and certainly in this case, this assumption is incorrect.

This review should be retracted and Cochrane should issue a full unreserved apology to patients.

It’s time to vote in the Cochrane Governing Board Elections!

Dear Friends and Colleagues

You may be aware there is an election for four new members of the Cochrane Governing Board triggered by the expulsion of Peter Gotzsche from the organisation, and the resignation of four other board members in protest.  I am one of a group of Cochrane members who have asked for a debate (and we ran a now closed petition) about four issues that we think are the most important policies to move Cochrane forward positively. 620 people supported the petition. This is the biggest Cochrane member/supporter engagement ever.  Briefly, the four changes we would like to see are 1. Promoting a culture of open discussion and transparency 2. Refocussing on the heart of Cochrane as a collaboration 3. Increased involvement of Cochrane members in governance  4. Open access to all Cochrane content, which is produced by volunteers and funded directly or indirectly by public money.  We asked the board to organize a forum for a debate on the issues, but they found it technically too difficult in the time available.  We think that these issues, plus other issues which have been suggested by Cochrane members, should be part of a debate for the current elections for the Governing Board. So, we have set up a channel for discussion on an independent blog www.cochranemembers.org . Please feel free to contribute.  All the election information is here https://elections.cochrane.org/.  Voting starts today 23 November 2018.

The full election statements were not set up to reveal very clearly what the candidates think about these strategic and policy issues. However, one of our group has worked hard using their evidence synthesis skills to extract the information the candidates have given in answer to the question `what are the most important strategic issues for Cochrane in 2019’.  This voter guide makes it possible to see at a glance what each candidate’s position is on the four issues that so many Cochrane members have said were important.  I hope you find it helpful.  The full candidate statements are here https://elections.cochrane.org/candidates-standing-board-election-november-2018.

Happy voting!  And please forward this to friends and colleagues who are members of Cochrane and encourage them to vote too.

 

I got a reply to my message to the Governing Board of Cochrane

My previous post “My question to the Governing Board of Cochrane” got an answer on Friday.  Here I’ve reposted my original question, their original answer, and my comments on that answer.  And below that is the answer Margeurite Koster emailed on Friday 16th November.  Still not answering my question, or explaining why asking for missing information from a trial when you’re not conducting a review is a breach of the Cochrane Spokesperson Policy.  To my horror, I now realise I have also breached the Spokesperson Policy.  In 2012 I wrote to a researcher asking him to publish the results of his trial which had been posted on a trials register and completed in 2009.  I wasn’t doing a systematic review – uh-oh!  It was part of an public engagement project I was managing aimed at patients, carers and members of the public, including schoolchildren to get them interested and involved in the important work Cochrane does, and to educate them about the huge problem of publication bias.  This researcher assured me that the results would be published, but they still haven’t been, and they have not been posted on the trial register either.  Luckily for me this researcher didn’t write and complain to Mark Wilson otherwise I would be in serious trouble.

My original question
Please can you explain why writing to a funder to ask for missing data from a trial using Cochrane letterhead is violating the Cochrane spokesperson policy? Surely that is what all Cochrane authors regularly have to do in order to avoid the reporting bias which blights most Cochrane Reviews. Surely this action is something to be rewarded and encouraged.

Governing board answer: part 1
It is important that when anybody within a Cochrane group writes to anybody, about anything, it should be made clear in what capacity they are writing.

My response
Cochrane’s mission is to produce documents which synthesize evidence on the benefits and harms of treatments. Surely someone from Cochrane or anywhere else asking for clarification about information which should have been reported clearly in an open access publication but wasn’t is undertaking a fundamental part of Cochrane’s work.

What if this funder had, as he should have done, written back to Professor Gøtzsche apologising for the lack of clarity in the published articles relating to the trial, and given him the information he asked for? What if Professor Gøtzsche were then able to include this information in a systematic review which showed significant harms from the drug and led to it being withdrawn as a treatment option in favour of a less harmful drug? What if Professor Gøtzsche were able to include this information in a systematic review which showed that the benefits of the drug outweighed the harms? It could have gone either way, but perhaps the person who complained about Professor Gøtzsche’s letter was more worried that it would show the drug in a negative light. You seem more concerned that Professor Gøtzsche be seen in a negative light than a person refusing to publish data from their trial.

Governing Board answer: part 2
This is particularly important if the person writing has roles both inside and outside Cochrane. If they are writing in a personal or professional capacity, about something that is not related to the official work of the Cochrane group, they must be careful not to confuse the recipient of the letter. Confusion may be engendered by using a Cochrane letterhead, e-mail or designation, when it would be more appropriate to use an alternative.

My response
I repeat, why is asking for information about the number of deaths in a trial not related to the official work of Cochrane? Should he have said he was conducting a Cochrane review about the drug? Would that have made it OK?

Governing Board answer: part 3
Professor Gøtzsche had a longstanding record of refusing to abide by Cochrane’s policy and pledges he had made in this regard; he refused to follow the guidance he was offered; and he rejected the responsibility he was repeatedly asked to bear in mind.

My response
What policy are you referring to here? Please be more specific. What guidance was he offered? Please be more specific. Guidance is guidance, not law. What responsibility? To Cochrane, or to patients? I think he chose patients over Cochrane. I think Cochrane chooses to protect the reputation of Cochrane over its mission to protect patients and the public. For example the unforgivable lapse over the publication of the Exercise for CFS review. It’s great Cochrane has finally decided to withdraw it, but it should never have been published in the first place. The harm done both by letting a methodologically sub-standard review get published, and thereby allowing the PACE triallists to use Cochrane’s name and reputation to sanctify an appallingly conducted and biased trial is sickening.

Response to my comments from Margeurite Koster  16 November 2018

Dear Ms. Struthers,

My apologies for the delay in responding to your question regarding the Spokesperson Policy, but the Board has been involved in planning for the upcoming elections and responding to questions and inquiries from a number of groups over the last several weeks.

Cochrane is an international collaboration involving thousands of people with multiple affiliations to different organizations. Cochrane’s Spokesperson Policy establishes guidance about who can speak officially on behalf of Cochrane and the circumstances in which it is appropriate to do so. This is so that people understand whether a request or statement is coming from Cochrane – the global organization – or simply from people who are members of Cochrane working on Cochrane activities.

It is important to make clear that a Cochrane author team can use their Cochrane affiliation to request data that is intended for use in a Cochrane systematic review they are working on – this is an important part of their ‘Cochrane work’. In making such as request, they would identify themselves as authors of an in-progress Cochrane review for a specific review group, and clarify that the request for data is being made in the interest of using the data for that specific review. This situation is common among Cochrane authors and is not a breach of the Spokesperson Policy.

Based on feedback we have received from Cochrane members, the Governing Board has recognised that some clarifications and more support are required, in order to help Cochrane members interpret the Spokesperson Policy properly. The Central Executive Team, in consultation with various Cochrane groups, will be working on this in the coming months.

Best regards,
Marguerite

Marguerite A. Koster
Co-Chair
Cochrane Governing Board