Improving the reliability of literature reviewsdetection of retracted articles through academic search engines

  1. Elena Pastor Ramón
  2. Iván Herrera Peco 1
  3. Oskia Agirre 2
  4. María García-Puente
  5. José María Morán 3
  1. 1 Universitat Pompeu Fabra

    Universitat Pompeu Fabra

    Barcelona, España


  2. 2 Universidad del País Vasco/Euskal Herriko Unibertsitatea

    Universidad del País Vasco/Euskal Herriko Unibertsitatea

    Lejona, España


  3. 3 Universidad de Extremadura

    Universidad de Extremadura

    Badajoz, España


EJIHPE: European Journal of Investigation in Health, Psychology and Education

ISSN: 2174-8144

Year of publication: 2022

Volume: 12

Issue: 5

Pages: 458-464

Type: Article

DOI: 10.3390/EJIHPE12050034 DIALNET GOOGLE SCHOLAR lock_openDialnet editor

More publications in: EJIHPE: European Journal of Investigation in Health, Psychology and Education


Cited by

  • Scopus Cited by: 2 (24-09-2023)
  • Web of Science Cited by: 1 (20-09-2023)
  • Dimensions Cited by: 1 (23-03-2023)

SCImago Journal Rank

  • Year 2022
  • SJR Journal Impact: 0.542
  • Best Quartile: Q2
  • Area: Clinical Psychology Quartile: Q2 Rank in area: 138/307
  • Area: Applied Psychology Quartile: Q3 Rank in area: 128/249
  • Area: Developmental and Educational Psychology Quartile: Q3 Rank in area: 177/349

Índice Dialnet de Revistas

(Indicator corresponding to the last year available on this portal, year 2021)
  • Year 2021
  • Journal Impact: 1.450
  • Field: PSICOLOGÍA Quartile: C1 Rank in field: 15/111
  • Field: EDUCACIÓN Quartile: C1 Rank in field: 20/228


  • Social Sciences: B

Scopus CiteScore

  • Year 2022
  • CiteScore of the Journal : 3.5
  • Area: Clinical Psychology Percentile: 61
  • Area: Developmental and Educational Psychology Percentile: 59
  • Area: Applied Psychology Percentile: 54

Journal Citation Indicator (JCI)

  • Year 2022
  • Journal Citation Indicator (JCI): 0.87
  • Best Quartile: Q2
  • Area: PSYCHOLOGY, CLINICAL Quartile: Q2 Rank in area: 68/178


(Data updated as of 23-03-2023)
  • Total citations: 1
  • Recent citations: 1


Nowadays, a multitude of scientific publications on health science are being developed that require correct bibliographic search in order to avoid the use and inclusion of retracted literature in them. The use of these articles could directly affect the consistency of the scientific studies and could affect clinical practice. The aim of the present study was to evaluate the capacity of the main scientific literature search engines, both general (Gooogle Scholar) and scientific (PubMed, EMBASE, SCOPUS, and Web of Science), used in health sciences in order to check their ability to detect and warn users of retracted articles in the searches carried out. The sample of retracted articles was obtained from RetractionWatch. The results showed that although Google Scholar was the search engine with the highest capacity to retrieve selected articles, it was the least effective, compared with scientific search engines, at providing information on the retraction of articles. The use of different scientific search engines to retrieve as many scientific articles as possible, as well as never using only a generic search engine, is highly recommended. This will reduce the possibility of including retracted articles and will avoid affecting the reliability of the scientific studies carried out.

Bibliographic References

  • Ali, P.A.; Watson, R. Peer review and the publication process. Nurs. Open 2016, 3, 193–202.
  • Audisio, K.; Robinson, N.B.; Soletti, G.J.; Cancelli, G.; Dimagli, A.; Spadaccio, C.; Olaria, R.P.; Chadow, D.; Rahouma, M.; Demetres, M.; et al. A survey of retractions in the cardiovascular literature. Int. J. Cardiol. 2021, 349, 109–114.
  • Bar-Ilan, J.; Halevi, G. Temporal characteristics of retracted articles. Scientometrics 2018, 116, 1771–1783.
  • Bucci, E.M. On zombie papers. Cell Death Dis. 2019, 10, 189.
  • Budd, J.M.; Sievert, M.; Schultz, T.R.; Scoville, C. Effects of article retraction on citation and practice in medicine. Bull. Med. Libr. Assoc. 1999, 87, 437–443.
  • Chapman, D. Health-related databases. J. Can. Acad. Child. Adolesc. Psychiatry 2009, 18, 148–149.
  • Couzin, J.; Unger, K. Cleaning up the Paper Trail. Science 2006, 312, 38–43.
  • Dal-Ré, R.; Ayuso, C. For how long and with what relevance do genetics articles retracted due to research misconduct remain active in the scientific literature. Account. Res. 2020, 28, 280–296.
  • Ferragut, J.; Pinto, N.; Amorim, A.; Picornell, A. Improving publication quality and the importance of Post Publication Peer Review: The illustrating example of X chromosome analysis and calculation of forensic parameters. Forensic Sci. Int. Genet. 2019, 38, e5–e7.
  • Frampton, G.; Woods, L.; Scott, D.A. Inconsistent and incomplete retraction of published research: A cross-sectional study on COVID-19 retractions and recommendations to mitigate risks for research, policy and practice. PLoS ONE 2021, 16, e0258935.
  • García-Puente, M.; Pastor-Ramon, E.; Agirre, O.; Morán, J.-M.; Herrera-Peco, I. Research note. Open letter to the users of the new PubMed: A critical appraisal. Prof. Inf. 2020, 29, e290336.
  • García-Puente, M.; Pastor-Ramon, E.; Agirre, O.; Moran, J.M.; Herrera-Peco, I. The use of Sci-Hub in systematic reviews of the scholarly literature. Clin. Implant Dent. Relat. Res. 2019, 21, 816.
  • Gasparyan, A.Y.; Ayvazyan, L.; Kitas, G. Multidisciplinary Bibliographic Databases. J. Korean Med. Sci. 2013, 28, 1270–1275.
  • Gehanno, J.-F.; Rollin, L.; Darmoni, S. Is the coverage of google scholar enough to be used alone for systematic reviews. BMC Med. Inform. Decis. Mak. 2013, 13, 7.
  • Giustini, D.; Boulos, M.N.K. Google Scholar is not enough to be used alone for systematic reviews. Online J. Public Health Inform. 2013, 5, 214.
  • Gusenbauer, M.; Haddaway, N.R. Which academic search systems are suitable for systematic reviews or meta-analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources. Res. Synth. Methods 2019, 11, 181–217.
  • Herrera-Peco, I.; Santillán-García, A.; Morán, J.; Goodman-Casanova, J.; Cuesta-Lozano, D. The Evidence-Based Practice Silent Enemy: Retracted Articles and Their Use in Systematic Reviews. Healthcare 2020, 8, 465.
  • King, E.G.; Oransky, I.; Sachs, T.E.; Farber, A.; Flynn, D.; Abritis, A.; Kalish, J.A.; Siracuse, J.J. Analysis of retracted articles in the surgical literature. Am. J. Surg. 2018, 216, 851–855.
  • Morán, J.M.; Santillán-García, A.; Herrera-Peco, I. SCRUTATIOm: How to detect retracted literature included in systematics reviews and metaanalysis using SCOPUS© and ZOTERO©. Gac. Sanit. 2020, 36, 64–66.
  • Moylan, E.C.; Kowalczuk, M.K. Why articles are retracted: A retrospective cross-sectional study of retraction notices at BioMed Central. BMJ Open 2016, 6, e012047.
  • Nair, S.; Yean, C.; Yoo, J.; Leff, J.; Delphin, E.; Adams, D.C. Reasons for article retraction in anesthesiology: A comprehensive analysis. Can. J. Anaesth. 2019, 67, 57–63.
  • Neale, A.V.; Northrup, J.; Dailey, R.; Marks, E.; Abrams, J. Correction and use of biomedical literature affected by scientific misconduct. Sci. Eng. Ethics 2007, 13, 5–24.
  • Nogueira, T.E.; Gonçalves, A.S.; Leles, C.R.; Batista, A.C.; Costa, L.R. A survey of retracted articles in dentistry. BMC Res. Notes 2017, 10, 253.
  • Orduña Malea, E.; Martín-Martín, A.; Delgado-López-Cózar, E. Google Scholar as a source for scholarly evaluation: A bibliographic review of database errors. Rev. Esp. Doc. Cient. 2017, 40, 1–33.
  • Rapani, A.; Lombardi, T.; Berton, F.; Del Lupo, V.; Di Lenarda, R.; Stacchi, C. Retracted publications and their citation in dental literature: A systematic review. Clin. Exp. Dent. Res. 2020, 6, 383–390.
  • Scite: Evaluate the Veracity of Scientific Work.
  • Serghiou, S.; Marton, R.M.; Ioannidis, J.P.A. Media and social media attention to retracted articles according to Altmetric. PLoS ONE 2021, 16, e0248625.
  • Singh, V.K.; Singh, P.; Karmakar, M.; Leta, J.; Mayr, P. The journal coverage of Web of Science, Scopus and Dimensions: A comparative analysis. Scientometrics 2021, 126, 5113–5142.
  • Sox, H.C.; Rennie, D. Research Misconduct, Retraction, and Cleansing the Medical Literature: Lessons from the Poehlman Case. Ann. Intern. Med. 2006, 144, 609–613.
  • The Retraction Watch Database [Internet]. New York: The Center for Scientific Integrity. 2018.
  • Theis-Mahon, N.R.; Bakker, C.J. The continued citation of retracted publications in dentistry. J. Med. Libr. Assoc. 2020, 108, 389–397.