At this initial stage of the process, your team has identified a knowledge gap in your field and you are aiming to answer a specific question. Developing a research question is the most important part of the systematic review process. It is the foundation upon which the rest of your research method is built. For example:
or assess an intervention:
or synthesize existing evidence:
This first step can take some time. You will likely go through different versions before settling on the right research question. Once you have developed your question, you will use it to create a search strategy.
Frameworks can help break your question down into parts so you can clearly see the elements in your topic. For a more comprehensive list of frameworks, see University of Maryland's guide: https://lib.guides.umd.edu/SR/research_question.
PICO is most commonly used for quantitative studies and is the most common framework for systematic reviews. This framework is often used within the health sciences for clinical research or education.
P - Population
I - Intervention/Exposure
C - Comparison
O - Outcome
Example: In 11-12 year old children (Population), what is the effect of a school-based multi-media learning program (Intervention) on an increase in real-world problem solving skills compared with analog-only curriculum (Comparison) within a one-year period (Time)?
PICo
P: Population/Problem
I: phenomenon of Interest
Co: Context
Example: What are the experiences (phenomenon of interest) of caregivers providing home based care to patients with Alzheimer's disease (population) in Australia (context)?
----------------------------------------------------------------------------------------------------------------
CHIP
C:Context
H: How
I: Issues
P: Population
Example:
SPICE
S: Setting
P: Perspective
I: Intervention/Exposure/Interest
E: Evaluation
Example: What are the benefits (evaluation) of a doula (intervention) for low income mothers (perspective) in the developed world (setting) compared to no support (comparison)?
-------------------------------------------------------------------------------------------------------------------
SPIDER
S: Sample
PI: Phenomenon of Interest
D: Design
E: Evaluation
R: Research Type
Example:
What are the experiences (evaluation) of women (sample) undergoing IVF treatment (phenomenon of interest) as assessed?
Design: questionnaire or survey or interview
Study Type: qualitative or mixed method
Scoping reviews have a much broader scope than systematic reviews, however, it can still be helpful to apply a framework. The recommended framework for scoping reviews, according to The Johanna Briggs Institute, is PCC:
P: Population
C: Concept
C: Context
Example: What are the trends (concept) in MOOCs (context) that support the interactions of learners with disabilities (population)?
Recommended Reading for Scoping Review Methods:
In order to publish your systematic review, you will want to follow reporting guidelines that outline what to include in your manuscript. Reporting guidelines provide consistency and transparency to published systematic reviews, increasing the likelihood that your research can be replicated by others. Different disciplines require different reporting guidelines. Go here to see a list of reporting guidelines and standards.
Librarians can assist you with selecting databases for your systematic review. Each database requires a different search syntax and some databases have controlled vocabulary or subject headings that you will want to incorporate into your search. We recommend creating one master search strategy and then translating for each database.
What is Grey Literature?
Grey Literature includes "a variety of written materials produced by organizations outside of traditional commercial and academic publishing channels, such as annual reports, [theses and dissertations], white papers, conference proceedings from government agencies, non-governmental organizations, or private companies. Grey literature may be difficult to access because it may not be widely distributed or included in bibliographic databases."
Why Search Grey Literature?
The purpose of a systematic review is to identify and synthesize all available evidence. There is a significant bias in scientific publishing toward publishing studies that show a significant effect. According to Campbell Collaboration Guidelines on Information Retrieval, more than 50% of studies reported in conference abstracts never reach full publication. While conference abstracts and other grey literature is not peer-reviewed, it is important to include all available research on the topic you are studying.
How Do I Search Grey Literature?
Locating grey literature on your topic may require some extra work, but here are a few tips to get you started:
With more than 2 million entries, PQD&T offers comprehensive listings for U.S. doctoral dissertations back to 1861, with extensive coverage of dissertations from many non-U.S. institutions. A number of masters theses are also listed. Thousands of dissertations are available full text, and abstracts are included for dissertations from the mid-1980s forward.
An international organization dedicated to promoting the adoption, creation, use, dissemination, and preservation of electronic theses and dissertations (ETDs).
Institutional WHO database of intergovernmental policy documents and technical reports. Can search by IRIS by region (Africa, Americas, Eastern Mediterranean, Europe, South-East Asia, Western Pacific)
OCLC index of papers presented at conferences worldwide
Center for Open Science Framework's search tool for scholarly preprints in the fields of architecture, arts, business, social and behavioral science, and more.
Global Directory of Open Access Repositories. You can search and browse through thousands of registered repositories based on a range of features, such as location, software or type of material held.
A protocol is a detailed explanation of your research project and should be written before you begin searching. The protocol should include your research question, objectives, and search methodology. The information included in a protocol can vary across disciplines. Go here for protocol development resources and registries.
Each database you use will be different and require a customized search string that incorporates the database's unique syntax. We recommend creating one master keyword list and then translating it for each database, using that database's subject terms and search syntax. Below are a few tools to assist with translating search strings from one database to the next.
Includes syntax for Cochrane Library, EBSCO, ProQuest, Ovid, and POPLINE.
The IEBH SR-Accelerator is a suite of tools to speed up steps in the Systematic Review process.
When conducting a systematic review, you will likely be exporting hundreds or even thousands of citations from different databases. Citation management tools are useful for storing, organizing, and managing your citations. These tools can also perform "de-duplication" to automatically remove duplicates of any citations you might have. TAMU Libraries are here to provide training and support on for various citation management tools such as EndNote, Zotero, and RefWorks.
During the screening process, you will take all the articles exported from your searches and begin to remove studies that are not relevant to your topic. Use the inclusion/exclusion criteria you developed during the protocol-writing stage to screen the title and abstract of the articles you found. Any studies that do not fit the criteria of your review can be thrown out. The full text of the remaining studies will then need to be screened to confirm they fit within the criteria of your review.
You must have at least two independent reviewers to screen all studies, resolving areas of disagreement by consensus or a third party who is an expert in the field. We recommend using Covidence for your screening process. Follow the link below for more information on accessing Covidence through TAMU Libraries.
Bias refers to factors that can systematically affect the observations and conclusions of a study, causing them to be inaccurate. When compiling studies for systematic reviews, it is best practice to assess the risk of bias for each of the included studies and include that assessment in your final manuscript. The Cochrane Handbook recommends presenting the assessment as a table of graph.
Generally, scoping reviews do not require a risk of bias assessment, however, according to the PRISMA Scoping Review checklist, scoping reviews should include a "critical appraisal of individual sources of evidence." In the final manuscript, a critical appraisal could be an explanation of the limitations of the included studies. Below is a list of recommended critical appraisal tools and resources.
Simple overview of risk of bias assessment, including examples of how to assess and present your conclusions.
CASP is an organization that provides resources for healthcare professionals, but their appraisal tools can be used for varying study types across disciplines.
From the Joanna Briggs Institute: "JBI’s critical appraisal tools assist in assessing the trustworthiness, relevance and results of published papers."
After you and your team have screened all of the studies to be included in your review, you will need to extract the data from each included study in order to synthesize the results. There are several ways to code your results, such as using Excel, Google Forms, or with Covidence's Data Extraction feature. See links to additional resources below.
Covidence is a software that manages all aspects of systematic review processes, including data extraction. TAMU Libraries has a subscription to Covidence. See Step 9 for how to set up your TAMU Covidence account.
"a web-based catalogue of tools that support various tasks within the systematic review and wider evidence synthesis process."
"The Systematic Review Data Repository (SRDR) is a powerful and easy-to-use tool for the extraction and management of data for systematic review or meta-analysis."
Within the data synthesis section of your manuscript, you will present the main findings of your evidence synthesis. There are multiple ways to go about synthesizing the data, which will largely depend on the type of studies you are synthesizing. Whatever method your team chooses, it is standard to use the PRISMA flow diagram to map out the number of studies identified, screened, and included in your evidence synthesis project.
Meta-Analysis
A quantitative statistical analysis that combines the results of multiple studies. The studies included must all be attempting to answer the same research question and have a similar research design. According to The Cochrane Handbook, a "meta-analysis yields an overall statistic (together with its confidence interval) that summarizes the effectiveness of an experimental intervention compared with a comparator intervention."
Narrative or Descriptive
If you have included studies that are not similar in research design, then a meta-analysis is neither possible nor appropriate. You will need to use a narrative or descriptive synthesis to describe your results.
PRISMA Flow Diagram feature within Covidence (See Step 9 on how to set up your TAMU Covidence account)
Unless otherwise noted, this section of the guide was adapted from Northwestern's "Evidence Synthesis: Steps in a Review" and Cornell University's "A Guide to Evidence Synthesis".