Guidelines for Critiquing Research

 

Research is a process of seeking information or the solution to problems based on evidence of a scientific nature.  In emergency medicine, as in any professional field, the practitioner must be able to judge the importance, quality, and applicability of research reported in professional journals.

 

The fact that a research study has been published is no guarantee that the study was well done or that the results are accurate, or even useful.  Some reviewers and editors fail to be discerning regarding certain aspects of the articles which are submitted to them, or an article may be important in one context (i.e., the study conditions), but not in the way that the reader wants to use it (extrapolation of findings to their own setting).  And no author is going to tell the reader that the study was unimportant, of questionable quality, or not applicable to settings outside their own study site, although any of these may be true.  The reader must learn to judge for himself.  The following questions are designed to guide you in critiquing research articles.

 

Variables

 

  1. Identify the key variables in this study.  If there are independent and dependent variables, label them as such.
  2. How does the author operationally define each of the key variables?
  3. Is each operational definition specific enough that you could duplicate it?  If not, what more information do you need?
  4. Give examples of potential extraneous variables that may potentially lead to a confounding relationship between the dependent and independent variables?  Were data collected on these variables?  Were they analyzed statistically to confirm/exclude their effect on the relationship between the dependent and independent variables?

 

Literature Review

 

  1. What is the focus of the literature review?
  2. Does the literature review convince you that there is a need for the study?
  3. Considering the problem under study, does the literature review appear to be fairly comprehensive (i.e., does it include references from an appropriate selection of topics, sources, time periods, and of differing conclusions)?
  4. Does the literature review appear to be up-to-date?  Can older references, if any, be justified?

 

Hypotheses

 

  1. Do the hypotheses flow logically from the problem statement and literature review?
  2. Are the hypothesized relationships among the variables clearly specified?
  3. For each hypothesis, indicate its direction and magnitude.

 

Ethics

 

  1. Are there ethical issues that you think may have been of concern to the review committee for the protection of human rights?
  2. Did the authors state how they obtained informed consent?
  3. Did the authors receive IRB approval from their institution(s)?

 

Design

 

  1. What is the basic research design employed in this study?
  2. What are the strengths and weaknesses of this design?
  3. What is the treatment variable, if there is one?

 

Sampling Procedures

 

  1. What type of sampling is used in this study?
  2. What is the author’s target population?
  3. Did the author truly sample from his intended population, to the exclusion of others?
  4. How well are the study subjects described?
  5. Is it possible that the sampling procedures employed may have excluded subjects that should have been included, and included others who should not have been?

 

Validity and Reliability

 

  1. What evidence does the author give that the data collection instruments (those used to measure the dependent variable) are reliable?  Is this convincing evidence? 
  2. What evidence does the author give that the data collections instruments are valid?  Is this convincing evidence?  Were the instruments piloted and validated in a previous study?

 

Data Analysis

 

  1. List the statistical tests that are used in the data analysis.
  2. Does the data analysis address all the hypotheses or research questions?
  3. Does the data analysis address questions that were not hypothesized?
  4. If the data analysis includes significance tests, what was the null hypothesis that was tested?
  5. If the data analysis includes significance tests, describe their major findings.
  6. How do the results compare with what the author hypothesized?  How do they compare with the findings of previous studies?

 

Conclusions and Recommendations

 

  1. Are the conclusions presented by the author supported by the findings of the study?
  2. Are the author’s recommendations for practice justified by the findings?
  3. Does the study make an important contribution to our body of knowledge, either in establishing theory or in guiding practice?