The repository contains information to reproduce the initial selection of tools reported in the software overview for screening large amounts of textual data implementing active learning as presented in the curated [software overview](https://github.com/Rensvandeschoot/software-overview-machine-learning-for-screening-text). All tools from three different sources were considered: 1) screening tools that are included (as of December 2022) in the [Systematic Review Toolbox](http://systematicreviewtools.com/) 2) screening tools considered in all relevant SR software comparison papers that cite the 2020 paper by [Harrison et al.](https://doi.org/10.1186/s12874-020-0897-3) as of December 2022 3) screening tools considered in all relevant SR software comparison papers that cite the 2021 paper by [Van de Schoot et al.](https://doi.org/10.1038/s42256-020-00287-7) as of December 2022 A paper was considered relevant if the authors compared multiple systematic reviewing software tools implementing active learning to accelerate the screening phase. # Procedure for the selection of software tools ## Search results Systematic Review Toolbox The [Systematic Review Toolbox]((http://systematicreviewtools.com/)) is an online catalogue of tools that can support researchers in the different stages of performing a systematic review. We searched this toolbox for screening tools, with the following search criteria: Search query: `Review family: "any" | Cost: "any" | Stages of the review: "Screening" | 44 tools have been found.` Results: - Abstrackr - AntConc - ASReview - BioReader (Biomedical Research Article Distiller) - Buhos - CADIMA - Cientopolis Scolr - Colandr - Concept Encoder - Covidence - DBPedia - Disputatron - DistillerSR - DoCTER - EMB automated PICO identification tool - EPPI-Reviewer - FastText - JBI-SUMARI - L·OVE platform - Nested-Knowledge - Parsifal - PICO Portal - pitts.ai - R.ROSETTA: an interpretable machine learning framework - RAx - Rayyan - ReLiS - Research Screener - revtools - RobotSearch - Screenatron - SESRA - SLR-Tool - SRDB.PRO - StArt - SWIFT-Active Screener - SWIFT-Review - Syras - SyRF: Systematic Review Facility - Sysrev - Systematic Review Accelerator - TaskExchange - TextBlob: Simplified Text Processing - Thoth ## Citation Search We consider screening tools that were part of a comparison in papers that cite the 2020 paper by Harrison et al. and/or the 2021 paper by Van de Schoot et al. as of January, 10 2023. On 10/1/2023, the Scopus database contained 46 papers that cited the Harrison 2020 paper. There is one paper that appears twice in this set, namely Foulquier 2021, of which a Foulquier 2022 is a French translation. Both versions were retrieved and considered for the sake of completeness. The same procedure was repeated to retrieve papers that cite Van de Schoot et al. 2020. The results are published in a curated [Zotero library](https://www.zotero.org/groups/4597652/asreview_public/collections/IWU8ATMK) and 147 papers were extracted. After deduplication, 187 papers were then screened and classified as relevant or irrelevant. A paper was considered relevant if the authors perform a comparison of multiple different systematic reviewing softwares. A paper was considered irrelevant if it lacked such a comparison, for example if its objective was to illustrate an application of such software, or if they assess or introduce one systematic reviewing tool, but do not make a comparison to other tools. Keeping these criteria and after removing duplicates, 13 papers were kept for consideration. Keeping these criteria and after removing duplicates, 13 papers were kept for consideration. The meta-data for each step can be found in the [Zotero file](https://osf.io/xkfna) and a summary is presented in the PRISMA flowchart: ![enter image description here][1] ### References The 13 software review papers that were under consideration for this comparison as of December 2022 are: * Adam, G. P., Wallace, B. C., & Trikalinos, T. A. (2022). Semi-automated Tools for Systematic Searches. In E. Evangelou & A. A. Veroniki (Eds.), Meta-Research: Methods and Protocols (pp. 17–40). Springer US. https://doi.org/10.1007/978-1-0716-1566-9_2 * Cierco Jimenez, R., Lee, T., Rosillo, N., Cordova, R., Cree, I. A., Gonzalez, A., & Indave Ruiz, B. I. (2022). Machine learning computational tools to assist the performance of systematic reviews: A mapping review. BMC Medical Research Methodology, 22(1), 322. https://doi.org/10.1186/s12874-022-01805-4 * Cowie, K., Rahmatullah, A., Hardy, N., Holub, K., & Kallmes, K. (2022). Web-Based Software Tools for Systematic Literature Review in Medicine: Systematic Search and Feature Analysis. JMIR Med Inform, 10(5), e33219. https://doi.org/10.2196/33219 * Khalil, H., Ameen, D., & Zarnegar, A. (2022). Tools to support the automation of systematic reviews: A scoping review. Journal of Clinical Epidemiology, 144, 22–42. https://doi.org/10.1016/j.jclinepi.2021.12.005 * Marta Pellegrini & Francesco Marsili. (2021). Evaluating software tools to conduct systematic reviews: A feature analysis and user survey. Form@re - Open Journal per La Formazione in Rete, 21(2). https://doi.org/10.36253/form-11343 * Nieto González, D. M. (2021). Optimización de estrategias de búsquedas científicas médicas utilizando técnicas de inteligencia artificial. https://doi.org/10.11144/Javeriana.10554.58492 * Robledo, S., Grisales Aguirre, A. M., Hughes, M., & Eggers, F. (2021). “Hasta la vista, baby” – will machine learning terminate human literature reviews in entrepreneurship? Journal of Small Business Management, 1–30. https://doi.org/10.1080/00472778.2021.1955125 * Scott, A. M., Forbes, C., Clark, J., Carter, M., Glasziou, P., & Munn, Z. (2021). Systematic review automation tools improve efficiency but lack of knowledge impedes their adoption: A survey. Journal of Clinical Epidemiology, 138, 80–94. https://doi.org/10.1016/j.jclinepi.2021.06.030 * Tsou, A. Y., Treadwell, J. R., Erinoff, E., & Schoelles, K. (2020). Machine learning for screening prioritization in systematic reviews: Comparative performance of Abstrackr and EPPI-Reviewer. Systematic Reviews, 9(1), 73. https://doi.org/10.1186/s13643-020-01324-7 * van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., Harkema, A., Willemsen, J., Ma, Y., Fang, Q., Hindriks, S., Tummers, L., & Oberski, D. L. (2021). An open source machine learning framework for efficient and transparent systematic reviews. Nature Machine Intelligence, 3(2), 125–133. https://doi.org/10.1038/s42256-020-00287-7 * Wagner, G., Lukyanenko, R., & Paré, G. (2022). Artificial intelligence and the conduct of literature reviews. Journal of Information Technology, 37(2), 209–226. https://doi.org/10.1177/02683962211048201 * Wang, L. L., & Lo, K. (2021). Text mining approaches for dealing with the rapidly expanding literature on COVID-19. Briefings in Bioinformatics, 22(2), 781–799. https://doi.org/10.1093/bib/bbaa296 * Xuan, Q., Jiali, L., Yuning, W., Ke, D., Yu, M., Kang, Z., Ling, L., & Xin, S. (2021). Application of natural language processing in systematic reviews. Chinese Journal of Evidence-Based Medicine, 21(6), 715–720. https://doi.org/10.7507/1672-2531.202012150 ## Screening tools from citation search We outline the *screening* tools used as part of a software comparison in each of these papers below: - [Adam et al. 2022](https://doi.org/10.1007/978-1-0716-1566-9_2): Abstrackr, Rayyan, SWIFT-Review, RobotAnalyst, DistillerSR - [Cowie et al. 2022](https://medinform.jmir.org/2022/5/e33219): *Tools restricted to screening phase (table 3)*: Abstrackr, Rayyan, RobotAnalyst, SWIFT-Active Screener, SR Accelerator. *Tools including screening phase*: Cadima, Covidence, Colandr, DistillerSR, EPPI-Reviewer Web, Giotto Compliance, JBI SUMARI, LitStream, Nested Knowledge, PICOPortal, Revman Web, SRDB.PRO, SRDR+, SyRF, SysRev - [González et al. 2021](https://doi.org/10.11144/Javeriana.10554.58492): *Tools from Table 1 with label: 'selección de estudios'*: Buhos, SRDB.PRO, SLuRp, DistillerSR, JBI-SUMARI, Nested-Knowledge, StArt, TaskExchange, CADIMA, EPPI-Reviewer, Parsifal, ReLiS, SESRA, Review Manager (RevMan), SLR-Tool, Systematic Review Accelerator, Thoth, ASReview, Covidence, PICO Portal, Rayyan, Health Assessment Workspace Collaborative (HAWC), Lingo3G: text document clustering engine, MeSHSIM, SLR.qub, SyRF: Systematic Review Facility. *Tools from Table 2*: Rayyan, Covidence, DRAGON, Abstrackr, Colandr, EPPI-Reviewer - [Jimenez et al. 2022](https://doi.org/10.1186/s12874-022-01805-4): Active_learning_document_screening, Active-learning-for-systematic-review, DAE-FF, InclusionCriteria, Machine Learning Functions, PubmedClassifier, Rules_cochranereviews, SLR_SearchStrings, ASReview, ASReview-covid, FASTREAD, GAPscreener, Pvtopic, RapidMiner, Rax, SWIFT-Review, Revtools, Abstrackr, Cochrane RCT Classifier, Colandr, Concept Encoder, DistillerSR, DoCTER, EPPI-Reviewer, IRIS.AI, PICO Portal, Rayyan, Research Screener, RobotAnalyst, RobotSearch, Screen4Me, SRA, SWIFT-Active Screener, SyRF, Sysrev - [Khalil et al 2022](https://doi.org/10.1016/j.jclinepi.2021.12.005): Rayyan, EPPI – reviewer, Abstrackr, SRA helper, LibSVM classifier, Bibot, DistelleR, Active Screener, RobotAnalyst, Swift-Review - [Pellegrini et al. 2021](https://doi.org/10.36253/form-11343): ASReview, Covidence, EPPI-Reviewer, Rayyan - [Robledo et al. 2021](https://doi.org/10.1080/00472778.2021.1955125): Rayyan, SWIFT-Review, ASReview - [Scott et al. 2021](https://doi.org/10.1016/j.jclinepi.2021.06.030): *Tools related to the screening phase*: Covidence, RevMan, Rayyan, JBI-SUMARI, DistillerSR, EPPI-Reviewer, Systematic Review Accelerator (screenatron), SRA-Helper (Endnote Helper) - [Tsou et al. 2021](https://doi.org/10.1186/s13643-020-01324-7): Abstrackr, EPPI-Reviewer - [van_de_Schoot et al. 2021](https://doi.org/10.1038/s42256-020-00287-7): Abstrackr, ASReview, Colandr, FASTREAD, Rayyan, RobotAnalyst - [Wagner et al. 2021](https://doi.org/10.1177/02683962211048201): ASReview, ADIT approach - [Wang and Lo 2021](https://doi.org/10.1093/bib/bbaa296): ASReview, Rayyan, RobotAnalyst, RobotReviewer, Trialstreamer - [Xuan et al. 2021](http://www.hxyx.com/article/54495.html): Abstrackr, EPPI-reviewer, Rayyan, ASReview, DoCTER, Colandr, SWIFT-Active Screener, GAPscreener # Selection of software tools All tools were listed, and after deduplication, this yielded 76 tools to consider for the software comparison. These were assessed against the following inclusion criteria: - employs a Reader-in-the-Loop (RITL)-based active learning cycle for systematic reviewing - has a Technology Readiness level of at least TRL8 - software is available - application is generic and is not restricted to the content of one specific field or type of intervention Screening on these criteria resulted in the nine tools included in the comparison performed here. ## Overview and final Decision The tools are ordered first on whether they were included in the [software overview](https://github.com/Rensvandeschoot/software-overview-machine-learning-for-screening-text). Within those categories, the tools are ordered alphabetically. |Tool | Search 1?| Search 2?| Included?| Excluded reason | |:------------------------------------------------------------------------------|---------:|---------:|---------:|:---------------------------------------------------------------------------| |Abstrackr | 1| 1| 1| | |ASReview | 1| 1| 1| | |Colandr | 1| 1| 1| | |DistillerSR | 1| 1| 1| | |EPPI-Reviewer | 1| 1| 1| | |FASTREAD | 1| 0| 1| | |Rayyan | 1| 1| 1| | |RobotAnalyst | 1| 0| 1| | |SWIFT-Active Screener | 1| 1| 1| | |Active_learning_document_screening | 1| 0| 0|no software | |Active-learning-for-systematic-review | 1| 0| 0|no software | |AntConc | 0| 1| 0|no RITL-based AL cycle | |ASReview-covid | 1| 0| 0|limited to COVID19 | |Bibot | 1| 0| 0|no RITL-based AL cycle | |BioReader (Biomedical Research Article Distiller) | 0| 1| 0|no RITL-based AL cycle | |Buhos | 1| 1| 0|no RITL-based AL cycle | |Cadima | 1| 1| 0|no RITL-based AL cycle | |Cientopolis Scolr | 0| 1| 0|website is down and no publications | |Cochrane RCT Classifier | 1| 0| 0|limited to RCTs | |Concept Encoder | 0| 1| 0|no RITL-based AL cycle | |Covidence | 1| 1| 0|no RITL-based AL cycle | |DAE-FF | 1| 0| 0|no software | |DBPedia | 0| 1| 0|no RITL-based AL cycle | |Disputatron | 0| 1| 0|no RITL-based AL cycle | |DoCTER | 1| 1| 0|no RITL-based AL cycle | |DRAGON | 1| 0| 0|rebranded as litstream | |EMB automated PICO identification tool | 0| 1| 0|limited to PICO | |FastText | 0| 1| 0|no RITL-based AL cycle | |GAPscreener | 1| 0| 0|limited to human genetic association | |Giotto Compliance | 1| 0| 0|limited to Clinical Evaluation Reports | |Health Assessment Workspace Collaborative (HAWC) | 1| 0| 0|no RITL-based AL cycle | |InclusionCriteria | 1| 0| 0|no software | |IRIS.AI | 1| 0| 0|no RITL-based AL cycle | |JBI-SUMARI | 1| 1| 0|no RITL-based AL cycle | |L·OVE platform | 0| 1| 0|limited to PICO | |LibSVM classifier | 1| 0| 0|no RITL-based AL cycle | |Lingo3G | 1| 0| 0|no RITL-based AL cycle | |LitStream | 1| 0| 0|no RITL-based AL cycle | |Machine Learning Functions | 1| 0| 0|no software | |MeSHSIM | 1| 0| 0|no RITL-based AL cycle | |Nested-Knowledge | 1| 1| 0|optimized for clinical literature | |Parsifal | 1| 1| 0|no RITL-based AL cycle | |PICO Portal | 1| 1| 0|limited to PICO | |pitts.ai | 0| 1| 0|limited to RCTs | |PubmedClassifier | 1| 0| 0|no software | |Pvtopic | 1| 0| 0|no RITL-based AL cycle | |R.ROSETTA: an interpretable machine learning framework | 0| 1| 0|no software | |RapidMiner | 1| 0| 0|no RITL-based AL cycle | |RAx | 0| 1| 0|no RITL-based AL cycle | |ReLiS | 1| 1| 0|no RITL-based AL cycle | |Research Screener | 0| 1| 0|Not TLR>7 | |Review Manager (RevMan) | 1| 0| 0|no RITL-based AL cycle | |Revtools | 0| 1| 0|Not TLR>7 | |RobotReviewer | 1| 0| 0|no RITL-based AL cycle | |RobotSearch | 0| 1| 0|limited to RCTs | |Rules_cochranereviews | 1| 0| 0|no RITL-based AL cycle | |Screen4Me | 1| 0| 0|limited to RCTs | |Screenatron | 0| 1| 0|no RITL-based AL cycle | |SESRA | 1| 1| 0|no RITL-based AL cycle | |SLR.qub | 1| 0| 0|no RITL-based AL cycle | |SLR_SearchStrings | 1| 0| 0|no software | |SLR-Tool | 1| 1| 0|no RITL-based AL cycle | |SLuRp | 1| 0| 0|no RITL-based AL cycle | |SRA-Helper (Endnote Helper) | 1| 0| 0|no RITL-based AL cycle | |SRDB.PRO | 1| 1| 0|no RITL-based AL cycle | |SRDR+ | 1| 0| 0|no RITL-based AL cycle | |StArt | 1| 1| 0|no RITL-based AL cycle | |SWIFT-Review | 1| 1| 0|no RITL-based AL cycle | |Syras | 0| 1| 0|no RITL-based AL cycle | |SyRF: Systematic Review Facility | 1| 1| 0|no RITL-based AL cycle | |SysRev | 1| 1| 0|no RITL-based AL cycle | |Systematic Review Accelerator / SR-Accelerator / SRA | 1| 1| 0|no RITL-based AL cycle | |TaskExchange | 1| 1| 0|no RITL-based AL cycle | |TextBlob: Simplified Text Processing | 0| 1| 0|no software | |Thoth | 1| 1| 0|no RITL-based AL cycle | |Trialstreamer | 1| 0| 0|limited to COVID19 | [1]: https://mfr.osf.io/export?url=https://osf.io/download/q6nd3/?direct=%26mode=render&format=2400x2400.jpeg