Abstract

Evidence Based Medicine (EBM) is a term used for the current dominant methodology for deciding what medical treatments should be accepted as valid. It places great emphasis on Randomised Clinical Trials (RCTs) which are analysed according to a strict frequentist paradigm, with a rigid p-value ≤0.05 criterion but with little consideration of prior probabilities or the cost of errors. Accordingly, low cost, safe treatments where there is prior knowledge of at least slight effectiveness, may often be inappropriately discarded by EBM. The Cochrane Collaboration is an online central repository of RCTs and meta-analyses of RCTs. This paper uses statistical methods applied to a random sample of outcomes listed in the Cochrane Collaboration, to estimate the negative predictive value when treatments are declared ineffective as a result of positive outcomes which do not achieve the p≤ 0.05 criterion. The data were analysed using six different models in order to determine the proportion of genuinely ineffective treatments in the set of all positive outcomes where p>0.05. All six methods give point estimates substantially less than half for the negative predictive value when the decision rule is to declare treatments to be ineffective when their outcome is positive but p>0.05. Although confidence interval estimation indicates considerable uncertainty in these estimates, it seems reasonable to conclude that when a RCT gives a positive outcome but p>0.05, the conventional EBM decision to declare the treatment to be ineffective, is likely to be wrong more often than not.

Keywords: Evidence based medicine, false negatives, false non-discovery rate, low cost treatments, negative predictive value, statistical models, type II error