Iranian Journal of Information Processing and Management

Iranian Journal of Information Processing and Management

Investigating the Effectiveness of Methodological Elements in Ranking the Results Based on their Relevance and Methodological Validity

Document Type : Original Article

Authors
Hajar Sotudeh 1
Adeleh Asadi shally 2
Mostafa fakhrahmad 3
1 Professor Department of Knowledge & Information Science Shiraz University, Iran
2 Department of Knowledge & Information Science Shiraz University, Iran - - Shiraz, Eram Square, Shiraz University
3 Assistant professor , Department of Computer Science,, of Shiraz University, Shiraz, iran,
10.22034/jipm.2023.708847
Abstract
The goal of evidence-based medicine is helping clinical professionals to obtain relevant and valid information to make appropriate clinical decisions. To meet the information needs of health professionals, information retrieval systems should facilitate access relevant and also methodologically valid.

In this research quasi-experimental method of one group pre-test-post-test is used in order to achieve the research goals. The research community was made up of all the clinical trial articles available in the Cochrane database until the end of 2018.

The results of the present study showed that among the research variables, the words of the reviewers' comments along with the abstract had the greatest impact on the ranking. These results showed that the Cochrane reviewers' comments can be a suitable tool for use in ranking systems based on relevance and validity. Also, the results of this research showed that it is possible to rank the methodological validity and combine it with relevance by using the internal and external elements of the articles.

This research distinguished the types of words used in the text of scientific articles and its effect on improving the ranking based on relevance and methodological validity. In retrieval systems based on natural language, more emphasis has been placed on the totality of the text and less attention has been paid to the contribution of vocabulary types in explaining the relevance of articles. The findings of this research highlighted, once again, the difference of vocabulary types in explaining relevance and methodological validity and revealed the necessity of determining the target vocabulary set in all types of searches.
Keywords
Ranking
Information Retrieval
Methodological Validity
Cochrane
Relevance
Reviewers' Comments
Subjects
 اسعدی، عادله، هاجر ستوده، جواد عباسپور، و مصطفی فخراحمد. 1399. توان نظر داوران Cochrane و بافتار استناد در شناسایی چکیده مقالات مرتبط و بخش‌های اصلی آنها. مدیریت اطلاعات سلامت 17 (4، پیاپی 74)، 181-188.
حاتمی ناغانی، بهمن و مسعود عابسی. 1395. تحلیل محتوایی مقالات علمی با استفاده از متن‌کاوی. مطالعات مدیریت کسب و کار هوشمند 5 (18): 137-167.
خدایی، رضا، محمدعلی بالافر، و سیدناصر رضوی. 1395. اثربخشی بسط پرس‌وجو مبتنی بر خوشه‌بندی اسناد شبه‌بازخورد با الگوریتم K-NN. مجله مهندسی برق دانشگاه تبریز  (1) 46: 146-151.
 
رشیدی شریف‌آباد، کیانوش، هاجر ستوده، مهدیه میرزابیگی، و سیدمصطفی فخراحمد. 1399. سنجش شباهت نظرات داوری آزاد و محتوای مقالات علمی به‌روش پردازش زبان طبیعی. فصلنامه مطالعات ملی کتابداری و سازماندهی اطلاعات 31 (2): 86-103.
علیزاده، علی، و حسن رشیدی. ۱۳۹۲. استخراج هوشمند کلمات مهم از بخش‌های مهم مقالات علمی. در همایش ملی مهندسی کامپیوتر و توسعه پایدار با محوریت شبکه‌های کامپیوتری، مدل‌سازی و امنیت سیستم‌ها. مشهد. مؤسسه آموزش عالی.
مولودی‌فر، نقی. 1378. بررسی رفتار اطلاع یابی پزشکان عمومی در دانشگاه علوم پزشکی و خدمات بهداشتی درمانی زنجان. پایان‌نامه کارشناسی ارشد. علم اطلاعات و دانش‌شناسی. دانشکده مدیریت و اطلاع رسانی پزشکی. دانشگاه علوم پزشکی ایران.
References
Anderson, J. G., & K. Abrahamson, K. 2017. Your Health Care May Kill You: Medical Errors. Studies in Health Technology and Informatics 13-17.
Bekhuis, T., & D. Demner-Fushman. 2010. Towards automating the initial screening phase of a systematic review. medical informatics conference: 146-150.
Bornstein, B. H., & A. C. Emler. 2001. Rationality in medical decision making: a review of the literature on doctors’ decision‐making biases. Journal of evaluation in clinical practice 7 (2): 97-107.
Chung, G. Y., & E. Coiera. 2007. A study of structured clinical abstracts and the semantic classification of sentences. In BioNLP '07 Proceedings of the Workshop on BioNLP 2007: Biological, Translational, and Clinical Language Processing (pp. 121-128). Association for Computational Linguistics (ACL).
 
Cohen, A. M., N. R. Smalheiser, M. S. McDonagh, C. Yu, C. E. Adams, J. M. Davis & P. S. Yu. 2015. Automated confidence ranked classification of randomized controlled trial articles: an aid to evidence-based medicine. Journal of the American Medical Informatics Association 22 (3): 707-717.
De Bruijn, B., S. Carini, S. Kiritchenko, J. Martin, & I. Sim. 2008. Automated information extraction of key trial design features from clinical trial publications. In AMIA Annual Symposium Proceedings (Vol. 2008, p. 141). Washington DC.: American Medical Informatics Association.
Erhardt, R. A., R. Schneider, & C. Blaschke. 2006. Status of text-mining techniques applied to biomedical text. Drug discovery today 11 (7-8): 315-325.
Griffiths, K. M., T. T. Tang, D. Hawking, & H. Christensen. 2005. Automated assessment of the quality of depression websites. Journal of medical Internet research 7 (5): e477.
 
Hassanzadeh, H., T. Groza, & J. Hunter. 2014. Identifying scientific artefacts in biomedical literature: The evidence based medicine use case. Journal of biomedical informatics 49: 159-170.
Haynes, R. B., N. Wilczynski, K. A. McKibbon, C. J. Walker, J. C. & Sinclair. 1994. Developing optimal search strategies for detecting clinically sound studies in MEDLINE. Journal of the American Medical Informatics Association 1 (6): 447-458.
Hsu, W., W. Speier, & R. K. Taira. 2012. Automated extraction of reported statistical analyses: towards a logical representation of clinical trial literature. In AMIA Annual Symposium Proceedings (Vol. 2012, p. 350). Chicago, Illinois: American Medical Informatics Association.
Jalali, V., & M. R. M. Borujerdi. 2011. Information retrieval with concept-based pseudo-relevance feedback in MEDLINE. Knowledge and information systems 29 (1): 237-248.
Kiritchenko, S., B. De Bruijn, S. Carini, J. Martin, & I. Sim. 2010. ExaCT: automatic extraction of clinical trial characteristics from journal publications. BMC Medical Informatics and Decision Making 10 (1): 1-17.
Kolikipogu, R., B. P. Rani, & N. Swapna. 2013. Pseudo relevance feedback by linking WordNet for expanding queries in information retrieval process. International Journal of Modeling and Optimization 3 (5): 462.
Leape, L. L., D. M. Berwick, & D. W. Bates. 2002. What practices will most improve safety: evidence-based medicine meets patient safety. Jama 288 (4): 501-507.
Marshall, I. J., J. Kuiper, & B. C. Wallace. 2014. Automating risk of bias assessment for clinical trials. Proceedings of the 5th ACM Conference on Bioinformatics, Computational Biology, and Health Informatics - BCB ’14, 88–95. Newport Beach, California, USA.  
_____. 2016. RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials. Journal of the American Medical Informatics Association 23 (1): 193-201.
Millard, L. A., P. A. Flach, J. P. & Higgins. 2016. Machine learning to assist risk-of-bias assessments in systematic reviews. International journal of epidemiology 45 (1): 266-277.
Paek, H., Y. Kogan, P. Thomas, S. Codish, & M. Krauthammer. 2006. Shallow semantic parsing of randomized controlled trial reports. In AMIA Annual Symposium Proceedings (Vol. 2006, p. 604). Washington DC. USA.
Sarker, A., D. Mollá, C. & Paris. 2016. Query-oriented evidence extraction to support evidence-based medicine practice. Journal of biomedical informatics 59: 169-184.
Savova, G. K., J. J. Masanz, P. V. Ogren, J. Zheng, S. Sohn, K. C. Kipper-Schuler & C. G. Chute. 2010. Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications. Journal of the American Medical Informatics Association 17 (5): 507-513.
Summerscales, R. L., S. Argamon, S. Bai, J. Hupert, & A. Schwartz. 2011. Automatic summarization of results from clinical trials. In 2011 IEEE International Conference on Bioinformatics and Biomedicine (pp. 372-377). IEEE.
Tsafnat, G., P. Glasziou, M. K. Choong, MA. Dunn, F. Galgani, & E. Coiera. 2014. Systematic review automation technologies. Systematic reviews 3 (1): 1-15.
 
 
Wallace, B. C., A. Noel-Storr, I. J. Marshall, A. M. Cohen, N. R. Smalheiser, & J. Thomas. 2017. Identifying reports of randomized controlled trials (RCTs) via a hybrid machine learning and crowdsourcing approach. Journal of the American Medical Informatics Association 24 (6): 1165-1168.
Wallace, B. C., J. Kuiper, A. Sharma, M. Zhu, & I. J. Marshall. 2016. Extracting PICO sentences from clinical trial reports using supervised distant supervision. The Journal of Machine Learning Research 17 (1): 4572-4596.
Wallace, B. C., T. A. Trikalinos, J. Lau, C. Brodley, & C. H. Schmid. 2010. Semi-automated screening of biomedical citations for systematic reviews. BMC bioinformatics 11 (1): 1-11.
Xu, Y., & Z. Chen. 2006. Relevance judgment: What do information users consider beyond topicality? Journal of the American Society for Information Science and Technology 57 (7): 961-973.
 
Xu, Y., G. J. Jones, & B. Wang. 2009. Query dependent pseudo-relevance feedback based on wikipedia. In Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval (pp. 59-66). Boston, MA.,  USA.
Zhu, F., P. Patumcharoenpol, C. Zhang, Y. Yang, J. Chan, A. Meechai, ... & B. Shen. 2013. Biomedical text mining and its applications in cancer research. Journal of biomedical informatics 46 (2): 200-211.

  • Receive Date 01 February 2023
  • Revise Date 22 August 2023
  • Accept Date 23 August 2023