Iranian Journal of Information Processing and Management

Iranian Journal of Information Processing and Management

Validity and Reliability in Qualitative Research: A Systematic Review

Document Type : Original Article

Authors
abbas abbasi 1
leila Mohammadi 2
khadijeh mohammadi 3
1 Associate Professor. Department of Management. Faculty of Economics, Management. Social Science. Shiraz University. Shiraz. Iran.
2 PhD Student in Artificial Intelligence in Medical Sciences. Department of Artificial Intelligence in Medical Sciences. Faculty of Medicine. Shiraz university of medical sciences. Shiraz. Iran.
3 PhD Student in Systems Management; Department of Industrial Management; Faculty of Economics, Management & Social Science; Shiraz University; Shiraz; Iran.
10.22034/jipm.2024.717062
Abstract
Although validity and reliability evaluation has long been the focus of quantitative researchers, these concepts have been a discussion for qualitative researchers. Given that qualitative studies are conducted in the real world, they can have implications for people’s lives. Therefore, it is necessary for the results of these studies to be reliable. So, the validity and reliability of these studies should be examined in order to gain the trust of the audience in the results of qualitative research and increase its credibility. Thus, this study was conducted with the aim of presenting a set of methods for evaluating validity and reliability in qualitative researches.
Systematic review was used in this research to achieve the research aims, because this type of research examines a comprehensive set of studies related to the research topic without creating time and research scope. Therefore, the interesting keywords were searched in the Web of Science database and 100 articles were extracted. After three stages of refining and reviewing the articles, including articles and reviewing articles, 48 relevant articles from 1993-2023 were found on the subject which formed the basis for data collection. In addition, the bibliometrics graphical environment belonging to the bibliometrics library of the R software and VOSviewer are also were used to describe the research cases.
In this research, a relatively comprehensive set of validity and reliability evaluation methods was extracted and categorized in the form of quantitative and qualitative evaluation methods of validity and reliability in qualitative research. Based on the results of 55 qualitative and 4 quantitative methods for evaluating validity in qualitative research, as well as 13 qualitative methods and 6 quantitative methods for evaluating reliability in qualitative studies were extracted from the articles.
Validity and reliability evaluation in qualitative research has been considered more in social science research and then in the field of nursing and health care. This shows the importance of validity and reliability in this researches. Among the validity evaluation methods that have received the most attention between researchers are internal validity, external validity and content validity. Each of these validity methods are different from its quantitative concept. Based on the studies, external validity has been used in qualitative researches that used case study, qualitative comparative analysis, interview, and review methods. Internal validity has also been used to evaluate qualitative comparative analysis and case studies. Content validity has been used in the evaluation of qualitative methods of ethnography, phenomenology, grounded theory, interview and cognitive interview. Among the qualitative reliability evaluation methods, inter-rater reliability methods, which are part of quantitative methods, have received much attention in recent years. The review indicates that this method has been used to evaluate the qualitative research methods such as content analysis, theme analysis, and case study. The results of this study can be used by qualitative researchers, qualitative article reviewers, and qualitative study audiences to judge the credibility of the results.
Keywords
Qualitative Research
Validity
Reliability
Systematic Review
Subjects
References
Ancker, J. S., N. C. Benda, M. Reddy, K. M. Unertl & T. Veinot. 2021. Guidance for publishing qualitative research in informatics. Journal of the American Medical Informatics Association 28 (12): 2743-2748.
Athens, L. 2010. Naturalistic inquiry in theory and practice. Journal of Contemporary Ethnography, 39 (1): 87-125.
Brod, M., L. E. Tesler, & T. L. Christensen. 2009. Qualitative research and content validity: developing best practices based on science and experience. Quality of Life Research 18: 1263-1278.
Buckley, R. 2022. Ten steps for specifying saturation in qualitative research. Social Science & Medicine 309: 115217.
Campbell, J. R. 2001. Participatory rural appraisal as qualitative research: distinguishing methodological issues from participatory claims. Human Organization 60 (4): 380-389.
Cole, R. 2023. Inter-rater reliability methods in qualitative case study research. Sociological Methods & Research 53 (4): 1944-1975.
Coleman, P. 2022. Validity and reliability within qualitative research for the caring sciences. International Journal of Caring Sciences 14 (3): 2041-2045.
Collingridge, D. S., & E. E. Gantt. 2019. Republished: The quality of qualitative research. American Journal of Medical Quality 34 (5): 439-445.
Collins, M., M. Shattell, & S. P. Thomas. 2005. Problematic interviewee behaviors in qualitative research. Western journal of nursing research 27 (2): 188-199.
Cook, D. A., & T. J. Beckman. 2006. Current concepts in validity and reliability for psychometric instruments: theory and application. The American journal of medicine 119 (2): 166-e7.
Cook, K. E. 2012. Reliability assessments in qualitative health promotion research. Health promotion international 27 (1): 90-101.
Creswell, J. W. 2007. Qualitative inquiry and research design: Choosing among five approaches: Los Angeles: Sage publications. Second edition.
De Massis, A., & J. Kotlar. 2014. The case study method in family business research: Guidelines for qualitative scholarship. Journal of family business strategy 5 (1): 15-29.
Diefenbach, T. 2009. Are case studies more than sophisticated storytelling? Methodological problems of qualitative empirical research mainly based on semi-structured interviews. Quality & Quantity 43: 875-894.
Drury, R., K. Homewood, & S. Randall. 2011. Less is more: the potential of qualitative approaches in conservation research. Animal Conservation 14 (1): 18-24.
Fehnel, S. E., C, M. Ervin, R. T. Carson, G. Rigoni, J. M. Lackner, & S. J. Coons. 2017. Development of the diary for irritable bowel syndrome symptoms to assess treatment benefit in clinical trials: Foundational qualitative research. Value in Health 20 (4): 618-626.
FitzPatrick, B. 2019. Validity in qualitative health education research. Currents in Pharmacy Teaching and Learning 11 (2): 211-217.
.Fossey, E., C. Harvey, F. McDermott, & L. Davidson. 2002. Understanding and evaluating qualitative research. Australian & New Zealand Journal of Psychiatry 36 (6): 717-732.
Gater, A., L. Nelsen, S. Fleming, J. J. Lundy, N. Bonner, R. Hall, ... & J. Haughney. 2016. Assessing asthma symptoms in adolescents and adults: qualitative research supporting development of the asthma daily symptom diary. Value in Health 19 (4): 440-450.
Gliner, J. A. 1994. Reviewing qualitative research: Proposed criteria for fairness and rigor. The Occupational Therapy Journal of Research 14 (2): 78-92.
Golafshani, N. 2003. Understanding reliability and validity in qualitative research. The qualitative report 8 (4): 597-607.
Grant, M. J., & A. Booth. 2009. A typology of reviews: an analysis of 14 review types and associated methodologies. Health information & libraries journal 26 (2): 91-108.
Guan, Y., A. M. Nguyen, S. Wratten, S. Randhawa, J. Weaver, F. Arbelaez, ... & C. Panter. 2022. The endometriosis daily diary: qualitative research to explore the patient experience of endometriosis and inform the development of a patient-reported outcome (PRO) for endometriosis-related pain. Journal of Patient-Reported Outcomes 6 (1): 5.
Hannes, K., C. Lockwood, & A. Pearson. 2010. A comparative analysis of three online appraisal instruments’ ability to assess validity in qualitative research. Qualitative health research 20 (12): 1736-1743.
Hayashi, P., G. Abib, & N. Hoppen. 2019. Validity in qualitative research: A processual approach. The Qualitative Report 24 (1): 98-112.
Hayashi Jr, P., G. Abib, N. Hoppen, & L. D. G. Wolff. 2021. Processual validity in qualitative research in healthcare. INQUIRY: The Journal of Health Care Organization, Provision, and Financing 58.
Humphreys, L., N. A. Lewis Jr, K. Sender, & A. S. Won. 2021. Integrating qualitative methods and open science: Five principles for more trustworthy research. Journal of Communication 71 (5): 855-874.
Jones, J. A., & R. Donmoyer. 2021. Improving the trustworthiness/ validity of interview data in qualitative nonprofit sector research: The formative influences timeline. Nonprofit and Voluntary Sector Quarterly 50 (4): 889-904.
Kimberlin, C. L., & A. G. Winterstein. 2008. Validity and reliability of measurement instruments used in research. American journal of health-system pharmacy 65 (23): 2276-2284.
Kogen, L. 2024. Qualitative Thematic Analysis of Transcripts in Social Change Research: Reflections on Common Misconceptions and Recommendations for Reporting Results. International Journal of Qualitative Methods 23: 1-11.
LeCompte, M. D., & J. P. Goetz. 1982. Problems of reliability and validity in ethnographic research. Review of Educational Research 52 (1): 31-60.
Lincoln, Y. S., & E. G. Guba. 1985. Naturalistic inquiry. California: Sage.
Lockwood, C., Z. Munn, & K. Porritt. 2015. Qualitative research synthesis: methodological guidance for systematic reviewers utilizing meta-aggregation. JBI Evidence Implementation 13 (3): 179-187.
Maxwell, J. 1992. Understanding and validity in qualitative research. Harvard Educational Review 62 (3): 279-301.
Mero-Jaffe, I. 2011. ‘Is that what I said?’Interview transcript approval by participants: an aspect of ethics in qualitative research. International journal of qualitative methods 10 (3): 231-247.
Motulsky, S. L. 2021. Is member checking the gold standard of quality in qualitative research? Qualitative Psychology 8 (3): 389.
Naegeli, A. N., E. Flood, J. Tucker, J., Devlen, & E. Edson-Heredia. 2013. The patient experience with fatigue and content validity of a measure to assess fatigue severity: qualitative research in patients with ankylosing spondylitis (AS). Health and Quality of Life Outcomes 11 (1): 1-9.
Nguyen, K. T. N. H., J. J. Stuart, A. H. Shah, I. A. Becene, M. G. West, J. Berrill, ... & J. Rich-Edwards. 2023. Novel Methods for Leveraging Large Cohort Studies for Qualitative and Mixed-Methods Research. American Journal of Epidemiology 192 (5): 821-829.
O’Connor, C., & H. Joffe. 2020. Intercoder reliability in qualitative research: debates and practical guidelines. International journal of qualitative methods 19: 1-13.
Onwuegbuzie, A. J., & N. L. Leech. 2007. Validity and qualitative research: An oxymoron? Quality & Quantity 41: 233-249.
Pilnick, A., & J. A. Swift. 2011. Qualitative research in nutrition and dietetics: assessing quality. Journal of Human nutrition and Dietetics 24 (3): 209-214.
Rapport, F., & S. E. Hughes. 2020. Frameworks for change in hearing research: Valuing qualitative methods in the real world. Ear and Hearing 41: 91S-98S.
Ravn, S. 2023. Integrating qualitative research methodologies and phenomenology—using dancers’ and athletes’ experiences for phenomenological analysis. Phenomenology and the Cognitive Sciences 22 (1): 107-127.
Raymond, K., S. Vallow, C. Saucier, K. Jackson, M. K. White, A. Lovley, & D. D’Alessio. 2022. Qualitative research with patients and caregivers of patients with PIK3CA related overgrowth spectrum: content validity of clinical outcome assessments. Journal of Patient-Reported Outcomes 6 (1): 75.
Rolfe, G. 2006. Validity, trustworthiness and rigour: quality and the idea of qualitative research. Journal of advanced nursing 53 (3): 304-310.
Sandelowski, M. 1993. Rigor or rigor mortis: The problem of rigor in qualitative research revisited. Advances in nursing science 16 (2): 1-8.
Sanders, C. B., & C. J. Cuneo. 2010. Social reliability in qualitative team research. Sociology 44 (2): 325-343.
Savall, H., V. Zardet, M. Bonnet, & M. Peron. 2008. The emergence of implicit criteria actually used by reviewers of qualitative research articles: Case of a European journal. Organizational Research Method, 11 (3): 510-540.
.Seale, C., & D. Silverman. 1997. Ensuring rigour in qualitative research. The European Journal of Public Health 7 (4): 379-384.
Singh, N., M. Benmamoun, E. Meyr, & R. H. Arikan. 2021. Verifying rigor: analyzing qualitative research in international marketing. International marketing review 38 (6): 1289-1307.
Sinkovics, R. R., E. Penz, & P. N. Ghauri. 2008. Enhancing the trustworthiness of qualitative research in international business. Management international review 48: 689-714.
Slettebø, T. 2021. Participant validation: Exploring a contested tool in qualitative research. Qualitative Social Work 20 (5): 1223-1238.
Taherdoost, Hamed. 2016. Validity and Reliability of the Research Instrument; How to Test the Validation of a Questionnaire/ Survey in a Research. International Journal of Academic Research in Management 5: 28–36.
Taranto-Vera, G., P. Galindo-Villardón, J. Merchán-Sánchez-Jara, J. Salazar-Pozo, A. Moreno-Salazar, & V. Salazar-Villalva. 2021. Algorithms and software for data mining and machine learning: a critical comparative view from a systematic review of the literature. The Journal of Supercomputing 77: 11481-11513.
Thomann, E., & M. Maggetti. 2020. Designing research with qualitative comparative analysis (QCA): Approaches, challenges, and tools. Sociological Methods & Research 49 (2): 356-386.
Thomas, D. R. 2017. Feedback from research participants: are member checks useful in qualitative research? Qualitative research in psychology 14 (1): 23-41.
_____, E., & J. K. Magilvy. 2011. Qualitative rigor or research validity in qualitative research. Journal for specialists in pediatric nursing 16 (2): ?.
Tuck, E., & M. McKenzie. 2015. Relational validity and the “where” of inquiry: Place and land in qualitative research. Qualitative inquiry 21 (7): 633-638.
Wells, C. S., & M. Faulkner-Bond (Eds.). 2016. Educational measurement: From foundations to future. New York, NY: The Guilford Press.
Williams, E. N., & S. L. Morrow. 2009. Achieving trustworthiness in qualitative research: A pan-paradigmatic perspective. Psychotherapy Research 19 (4-5): 576-582.
Yadav, D. 2022. Criteria for good qualitative research: A comprehensive review. The Asia-Pacific Education Researcher 31 (6): 679-689.
Yang, Y., J. Pankow, H. Swan, J. Willett, S. G. Mitchell, D. S. Rudes, & K. Knight. 2018. Preparing for analysis: a practical guide for a critical step for procedural rigor in large-scale multisite qualitative research studies. Quality & Quantity 52: 815-828.
Yardley, L. 2000. Dilemmas in qualitative health research. Psychology and Health 15 (2): 215-228.

  • Receive Date 06 April 2024
  • Revise Date 14 August 2024
  • Accept Date 26 August 2024