Turing Project


A research question in Systematic Reviews (SRs) is typically decomposed into its PICO elements: patient/population, intervention, comparator, outcomes. PICO elements are useful for formulating search queries for literature database searches and mentions of PICO elements are key to screening the search results for relevance. Standard approaches for SRs, including rapid reviews and scoping reviews, perform screening initially using only the title and abstracts of a reference collection, before obtaining and analysing a subset of full-text articles. While faster and more cost effective than full-text screening, manually screening all reference abstracts is a protracted process for large collections, especially those with low specificity. Our research hypothesis is that a model with information about PICO elements will produce more pertinent results than a model lacking this information. Automatic extraction of PICO elements from literature will thus enrich the information available to make screening decisions. We propose to develop both an automated PICO recognition model and a relevancy classification model based on words whose PICO relevancy has been determined to improve relevance prioritisation for screening.


In this 6-month project, we will develop neural machine reading methods to support the automation and effective identification of knowledge for systematic review (SR) development. The project will leverage RobotAnalyst [1], which was developed by NaCTeM in cooperation with the National Institute of Clinical Excellence (NICE). The system automates screening in SRs by reading the title and the abstract of a document collection, and ranking the relevancy of a reference using a model trained on human-assessed relevant and irrelevant examples.


A publicly available corpus of PICO annotations will be used to train a named entity recognition model [2] which will be implemented as a recurrent neural network. This model will be then applied to a separate collection of abstracts for references from systematic reviews within biomedical and health domains. The occurrences of words tagged in the context of specific PICO contexts will be used as additional features for a relevancy classification model [3].

Relevancy-based prioritisation, in which a system automatically selects the references that are most likely to be relevant in a given scenario, has been proven to be beneficial for gaining an understanding of the domain and to saving time, by eliminating the need for the whole collection to be screened manually. We will integrate models to detect PICO elements and their relations into the relevancy classification model used by RobotAnalyst, in order to improve the predictive power of the model to determine the most relevant references. Our hypothesis is that PICO annotations will improve the performance metric on average with substantial gains for SR development.

Machine learning classification models will be used to leverage textual features of the PICO element mentions and their surrounding contexts to produce fine-grained relevancy feedback models. Users can then directly improve the relevancy feedback model by coding each mention (indicating inclusion or exclusion), or marking a mention as irrelevant. This process enables a specific classification model to be trained to predict which PICO mentions are important for the inclusion/exclusion criteria for a specific research question. The relevance assessment results will be applied automatically, and users will correct the relevance assessment of the PICO elements. These corrections will be fed back into the relevance feedback model to produce a mechanism for prioritising the relevance of new studies in a collection. Using the PICO mention- level relevancy and document-level relevancy will enable a more nuanced and interpretable prioritisation.


[1] Przybyla, P., Brockmeier, A. J., Kontonatsios, G., Le Pogam, M.A., McNaught, J., von Elm, E., Nolan, K. and Ananiadou, S. (2018). Prioritising references for systematic reviews with RobotAnalyst: a user study. Research Synthesis Methods, 9(3): 470-488

[2] Ju, M., Nguyen, N.T.H, Miwa, M., and Ananiadou, S. (2019). An Ensemble of Neural Models for Nested Adverse Drug Events and Medication Extraction with Subwords, Journal of the American Medical Informatics Association 27(1): 22-30

[3] Brockmeier, A. J., Ju., M., Przybyla, P. and Ananiadou, S. (2019). Improving reference prioritisation with PICO recognition. BMC Medical Informatics and Decision Making, 19:256

Project Team

This project is a collaboration between NaCTeM and the Artificial Intelligence Research Center (AIRC), Japan, in synergy with NICE.

Project team

  • Prof. Sophia Ananiadou, NaCTeM (Principal Investigator)
  • Dr. Meizhi Ju, NaCTeM
  • Prof. Junichi Tsujii, AIRC, Japan
  • Dr. Kay Nolan, NICE, UK


This project is being funded by the Alan Turing Institute