Do all meta-analyses inform healthcare decisions?
Written by Dr Niamh O’Reilly


Systematic reviews and meta-analyses can be powerful tools for generating the evidence needed to inform clinical decisions, reimbursement decisions and the development of clinical practice guidelines (1,2). In the hierarchy of evidence based medicine (EBM), where clinical evidence is ranked according to the strength of freedom from bias,meta-analyses are ranked at the highest level for informing healthcare related decisions(3).

Systematic review versus meta-analysis

A systematic review aims to address a specific clinical or scientific question using a comprehensive plan, literature search strategy and explicit selection criteria to identify relevant studies, assess the methodologic quality of these studies, explore differences among study results and qualitatively and/or quantitatively synthesize their findings(4).

Meta-analysis is a statistical method conducted after a systematic review that combines data by drawing on the power of multiple studies to inform and quantify the efficacy, safety and/or utility of a healthcare intervention.Specifically, meta-analysis uses statistical techniques to synthesise and quantitatively summarise the results of individual studies. This increases the overall sample size and thus improves the statistical power of the analysis and precision of the treatment effect estimates (5–7).

The value of a meta-analysis is that where the size and direction of the treatment effect is consistent among studies, it confirms this common effect. Where the treatment effect is quite variable among studies, meta-analysis can help to identify the reason(s)why, which is also informative to clinical decision-making.

Are all meta-analyses the same?

There has been some debate in the literature about the types of studies that should be included in a meta-analysis to ensure they can inform healthcare decisions(8–10).As observed by Pickup (2013), meta-analyses can be classified as one of two distinct types, i.e. those that:

  • summarise the literature abouta healthcare topic or research questionor
  • inform healthcare matters in the real world e.g. clinical, regulatory, reimbursement and other health policy decisions.

Regarding healthcare decisions, it is important that clearly defined methods are used to identify and examine appropriate patient cohorts with the relevant baseline demographic and disease characteristics to properly represent the target patient population for whom the therapeutic efficacy,safety, cost-effectiveness and optimal use of therapies is being considered by healthcare decision-makers(10).

Literature summary meta-analyses

Meta-analyses that include only randomised controlled trials (RCTs) as the highest level of study evidence in the EBM hierarchy, such as those conducted according to the Cochrane Collaboration Guidelines, have been described as “literature summary meta-analyses” (11,12). RCTs are favoured for having a more valid study design compared with other types of studies, whereby randomisation removes confounding and the double-blind process minimizes biases such as the placebo effect(8).

However, it can also be argued that including only RCTs in a meta-analysis may dilute the results. For example,the dilution may come from including less relevant RCT studies that contain a broader population of individuals who may not all be relevant to the clinical or reimbursement question being asked about the effect of a healthcare intervention on a particular patient segment (8,10).

Furthermore, excluding observational or non-RCT studies can also result in valuable information not being captured such as the duration of treatment effect over a longer patient follow-up period.Therefore, literature summary meta-analyses may be suboptimal for real world decision-making and indeed, even misleading, where decisions about clinical effectiveness and cost-effectiveness require examination of the entire body of evidence for relevance to real world patient populations and use, as opposed to only RCTs as the highest quality of evidence (10).

Decision-making meta-analyses

‘Decision-making meta-analyses’, as described by Pickup 2013,are designed to assess all relevant studies that include the target patient segment for the healthcare intervention of interest(10). These meta-analyses may include patient data focused on a particular level of disease severity or patients with a specific treatment history. There is a growing consensus that the inclusion of observational studies in meta-analyses could be advantageous since they increase the size of the specific patient population of interest, can provide patient data over a longer time period and include other valuable information that cannot (logistically) be examined by RCTs, yet is more reflective of real world clinical practice (8,10,12).

An example of this is the systematic review and meta-analysis conducted by Pickup and Sutton (2008)who showed the benefit of including both RCTs and before/after (observational)studies in their analysis of continuous subcutaneous insulin infusion (CSII) versus multiple daily insulin injections (MDI) for controlling severe hypoglycaemia in patients with Type 1 diabetes(13).Although CSII is recommended by several national healthcare guidelines, previous meta-analyses had reported ambiguous results for the effect of CSII in controlling severe hypoglycaemia. However, Pickup and Sutton (2008) purposely focused on patients at risk of severe hypoglycaemia as the target population of interest who could potentially benefit from CSII by following a targeted study selection process.Consequently, they found the worst controlled subjects on injections had the most improvement on insulin pump therapy. They also reported that patients who attended clinic visits in the before/after studies were more likely to have had problems with glycaemic control than volunteers in RCTs.

By including a specific “at risk” patient group, the authors appropriately captured an important patient segment in need who could benefit the most from CSII, which was shown to be superior to MDI in reducing severe hypoglycaemia from both RCTs and the before/after observational studies(13). Importantly, the RCTs and before/after studies showed consistent results regarding severe hypoglycaemia reduction, although the magnitude of the difference favouring CSII over MDI was higher for the before/after studies.

Examples such as this show that when a meta-analysisis purposely designed to address a specific clinical or reimbursement question, it can fully inform the true value of a healthcare intervention – whether it be a drug, device or other intervention -for a specific patient population in need that is relevant to the real world clinical setting.

At CRC,our medical affairs capability is well equipped to conduct high quality systematic reviews and meta-analyses for a broad range of client situations relevant to clinical, regulatory,reimbursement, market access and other healthcare decision making.


  1. Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4(1):1.
  2. Russo MW. How to Review a Meta-analysis. Gastroenterol Hepatol (N Y). 2007;3(8):637–42.
  3. Haidich AB. Meta-analysis in medical research. Hippokratia. 2010;14(Suppl 1):29–37.
  4. Montori VM, Swiontkowski MF, Cook DJ. Methodologic issues in systematic reviews and meta-analyses. Clin Orthop Relat Res. 2003;(413):43–54.
  5. Shamseer L, Moher D, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ. 2015;349(3):7647.
  6. Uman LS. Information management for the busy practitioner: Systematic reviews and meta-analyses. J Am Acad Child Adolesc Psychiatry. 2011;20(1):57–9.
  7. Akobeng a K. Understanding systematic reviews and meta-analysis. Arch Dis Child. 2005;90(8):845–8.
  8. Shrier I, Boivin JF, Steele RJ, Platt RW, Furlan A, Kakuma R, et al. Should meta-analyses of interventions include observational studies in addition to randomized controlled trials? A critical examination of underlying principles. Am J Epidemiol. 2007;166(10):1203–9.
  9. Peinemann F, Tushabe DA, Kleijnen J. Using multiple types of studies in systematic reviews of health care interventions–a systematic review. PLoS One. 2013;8(12):e85035.
  10. Pickup JC. The evidence base for diabetes technology: appropriate and inappropriate meta-analysis. J Diabetes Sci Technol. 2013;7(6):1567–74.
  11. Higgins J, Green S. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011] [Internet]. The Cochrane Collaboration. 2011. Available from: http://handbook.cochrane.org.
  12. Kongsted HC, Konnerup M. Are more observational studies being included in Cochrane Reviews? BMC Res Notes. 2012;5(1):570.
  13. Pickup JC, Sutton AJ. Severe hypoglycaemia and glycaemic control in Type 1 diabetes: Meta-analysis of multiple daily insulin injections compared with continuous subcutaneous insulin infusion. Diabet Med. 2008;25(7):765–74.