Critical Appraisal of Systematic Reviews

In evidence-based practice, an overview of one or more evidence sources is said to be synthesized or summarized. Synthesis evidence is a big help for busy clinicians because the hard work of critical appraisal of each of the included studies is done already! However, the clinician STILL has to critically appraise the synthesis to ensure that the methods for putting the synthesis together were rigorous. In this post, I’ll provide the questions to ask when you are conducting a critical appraisal of systematic reviews. Explanations of why the specific questions are important to ask are included.

Synthesis Level of Evidence = Synopses and Syntheses
A synopsis is a summary or overview – in evidence-based practice, a synopsis can be based on either one or more original research studies (i.e., a synopsis of original studies) or on one or more quantitative or qualitative research studies (e.g., a systematic review, meta-analysis, meta-synthesis, or a synopsis of syntheses). In a synopsis, after the brief description of the study, the accompanying commentary usually includes an opinion about the scientific merit or quality of the research methods and presents the relevance of the findings to clinical practice.

Synthesis evidence of quantitative studies is presented as either a systematic review (SR) or a meta-analysis (MA). A meta-synthesis (MS) is a summary of qualitative research studies. A synthesis (an SR, MA, or MS) is a primary literature source because it is a research method conducted by the researcher or research team. A synopsis is a secondary source because it is someone’s opinion (albeit a critical appraisal) of original research.

Critical Appraisal of Systematic Reviews: Where Do You Start?
A systematic review summarizes the evidence surrounding a specific or focused clinical question – using a comprehensive and highly methodical process.
High-quality systematic reviews are considered the best possible sources of evidence for evaluating treatment effectiveness. Systematic reviews may be conducted by an individual, an independent group of researchers, or researchers associated with professional organizations.

Though there are methods outlined for conducting a rigorous systematic review, like any research, not all researchers follow a rigorous process. Therefore – you know what I’m going to say already – it’s important for the clinician to have good critical appraisal skills! Don’t be fooled by what you think is a reliable name or organization – until you are confident that the organization produces consistently rigorous systematic reviews; make sure you appraise the methods before using the evidence in practice!

Remember that to maximize your time and increase your efficiency, you want to find credible evidence sources that are already (reliably) appraised for you. A Cochrane Review is a good example of a trustworthy preappraised evidence. ALL Cochrane Review Groups follow a set protocol for producing a systematic review, meta-analysis, or meta-synthesis; the methods are transparent and of high quality.

This post covers only critical appraisal of systematic reviews. If you need to critically appraise a systematic review by yourself, let the questions below help you determine if the systematic review findings are trustworthy or not.

Critical Appraisal of Systematic Reviews: Asking the Right Questions
A critical appraisal is basically a detailed examination of published research for the purpose of making a decision about scientific merit and, therefore, for making a decision about the use of the evidence in practice.
A critical appraisal of a systematic review starts with asking questions about each of the major critical appraisal questions which are related to validity and reliability of the methods used to produce the systematic review; the meaning and importance of the synthesized recommendations; and, finally, the ability to translate the findings into clinical practice. There are critical appraisal tools available to specifically appraise systematic reviews that you can find from a variety of organizations.

Though all critical appraisal involves these three major questions (Validity, Import, Applicability), you’ll note some differences in the specific subquestions under each major question to accurately assess the nuances of a systematic review article. Remember that all meta-analyses are systematic reviews, but not all systematic reviews are meta-analyses. (I’m not going to address questions specific to a meta-analysis – that info will be shared in a future post).

Critical Appraisal of Systematic Reviews
Critical Appraisal of Systematic Reviews is the Step before Implementation. Photo credit: Olivier Le Moal/Shutterstock

What Subquestions Help Determine, Are the Study Results Valid?
Remember that this first set of subquestions is related to the methodological rigor of the study. You are assessing whether the question was focused and important and whether the study methods reduced the possibility of bias and minimized error as much as possible— before you care about the results of the review!

Did the review address a focused clinical question? What was the specific question (should include population, exposure/intervention, and outcomes) and was it sensible – that is, was it too broad to be useful? You should find this information in the introduction and background sections of the article.
Were the criteria used to select articles for inclusion appropriate? The reader needs to know which criteria were used to select the articles that were reviewed. You should determine if the systematic review only used randomized trials or if non-randomized trials were also included. Reviews which use only high-quality randomized controlled trials (RCTs) are less likely to have biased results and the likelihood of random error is reduced. If nonrandomized trials are included, then you would expect that the findings from the RCTs and nonRCTs are separated out for analysis (Straus et al., 2011). You should find inclusion and exclusion criteria in the Methods section of the article.
Was the search for relevant studies detailed, exhaustive, and reproducible? Is it unlikely that important, relevant studies were missed? This question is asking you to evaluate how comprehensive the search for literature was. Read the methodology section to identify the search strategies, search terms, and databases used by the researchers to locate all studies relevant to the question. Several methods are used to decrease retrieval, language, and publication bias. Hand-searches of journals and searches of the “grey literature” are included in good reviews. Also, did they include negative trials? Were journals in other languages searched? Did they include unpublished data?
Were the primary studies of high methodologic quality overall? Were the individual studies assessed for validity? Again the methods section should give you this information. You want to know how the researchers determined the studies were of high methodologic quality – what were the predetermined quality criteria? The use of independent reviewers that showed good agreement (the kappa statistic is usually reported) would be a good thing to report. There should be details given so you can make your own assessment of how well they did their job.
Were assessments of studies reproducible? This question asks if more than one person assessed the validity of the studies included in the review. Both random error and systematic error is possible. At least two independent reviewers are desirable. Information about how disagreements were handled should be reported. These details should be reported in the methods section.
Were the results of the included studies combined? This question is asking if it was reasonable to combine the results from the studies – that is, were the results homogenous or heterogeneous study to study? This question would primarily be asked if a meta-analysis was performed with the data. But narrative conclusions in a systematic review should also be from similar studies.
Were sources of financial and other support acknowledged and explained? This question is asking about possible conflicts of interest between the funding sources and the outcomes of the study. Research is expensive, so funding is sought, but you want to see the funding sources identified and any potential conflicts among the researchers acknowledged. You also want to see a statement along the lines of “the sponsors/funders had no involvement in the research protocol, conduct of the study, or in the writing of this paper” to give you confidence that the funders didn’t bias the study and influence the study outcomes.
What Questions Help Determine, Are the Results of the Systematic Review Important?

The “results” in these critical appraisal of systematic reviews questions refer to the conclusions made as a result of the synthesis of the evidence sources selected for this systematic review.

What are the overall results of the review? Are the results of the included studies clearly presented and discussed? The purpose of the review is to summarize the data from the individual studies into a bottom-line answer. Remember that not all systematic reviews are meta-analyses, so the results may not be provided as a numerical answer, but as a qualitative summary.
What Questions Help Determine, Can I Apply the Systematic Review Results to Practice?

Ok, so now if you decide the study is valid and the results are important, how can you use the results to benefit your patients? The questions related to practical application are the focus of these critical appraisal of systematic reviews questions:

Can the results be applied to my patient care? This question is fairly self-explanatory. The critical question is “How like your patient were the patients in the review?”
Were all clinically important outcomes considered? Are there outcomes that are associated with the outcome under review that you should be thinking about or would want information about? What are the costs associated with the therapy? What are the adverse effects associated with the therapy?
Are the likely treatment benefits worth the potential harms and costs?Your patients need to be informed of the risks and benefits of all potential therapies. Patient decisions will change patient-to-patient. You cannot make assumptions about the patient’s beliefs. The patient’s values and beliefs must be taken into account when making clinical decisions.
Bottom line: Systematic reviews are preappraised evidence that you should be able to implement immediately in practice IF you are sure that the methods are transparent, rigorous, and high quality; that the results (recommendations) are important; and that the target population matches your patient population. The intervention must be relevant to patient care and feasible to use in your setting.
How to Cite this Blogpost in APA*: Thompson, C. J. (2018, February 27). Critical appraisal of systematic reviews [Web Log Post]. Retrieved from https://nursingeducationexpert.com/critical-appraisal-of-systematic-reviews *Citation should have hanging indent
References
DiCenso, A., Guyatt, G., & Ciliska, D. (Eds.). (2005). Evidence-based nursing: A guide to clinical practice. St. Louis: Elsevier Mosby.

Straus, S. E., Richardson, W. S., Glasziou, P., & Haynes, R. B. (2011). Evidence-based medicine: How to practice and teach EBM (4th ed.). Edinburgh, Scotland: Elsevier Churchill Livingstone.