New article describing syntactically-informed word representations
2020-06-29
We are pleased to announce the publication of a new article in Neurocomputing, presenting novel syntactically-informed word representations (SIWRs), which allow enrichment of the pre-trained word representations with syntactic information without training language models from scratch. SIWRs are evaluated on three information extraction tasks, namely nested named entity recognition (NER), binary and n-ary relation extractions (REs), with results demonstrating that SIWRs yield performance gains over the base representations in these NLP tasks with 3–9% relative error reduction.
Tran, T. T., Miwa, M. and Ananiadou, S. (2020). Syntactically-Informed Word Representations from Graph Neural Network. Neurocomputing https://doi.org/10.1016/j.neucom.2020.06.070
Abstract
Most deep language understanding models depend only on word representations, which are mainly based on language modelling derived from a large amount of raw text. These models encode distributional knowledge without considering syntactic information, although several studies have shown benefits of including such information. Therefore, we propose new syntactically-informed word representations (SIWRs), which allow us to enrich the pre-trained word representations with syntactic information without training language models from scratch. To obtain SIWRs, a graph-based neural model is built on top of either static or contextualised word representations such as GloVe, ELMo and BERT. The model is first pre-trained with only a relatively modest amount of task-independent data that are automatically annotated using existing syntactic tools. SIWRs are then obtained by applying the model to downstream task data and extracting the intermediate word representations. We finally replace word representations in downstream models with SIWRs for applications. We evaluate SIWRs on three information extraction tasks, namely nested named entity recognition (NER), binary and n-ary relation extractions (REs). The results demonstrate that our SIWRs yield performance gains over the base representations in these NLP tasks with 3–9% relative error reduction. Our SIWRs also perform better than fine-tuning BERT in binary RE. We also conduct extensive experiments to analyse the proposed method.
Previous item | Next item |
Back to news summary page |
Featured News
- New Named Entity Corpus for Occupational Substance Exposure Assessment
- FinNLP-FNP-LLMFinLegal @ COLING-2025 - Call for papers
- Keynote talk at Manchester Law and Technology Conference
- Keynote talk at ACM Summer School on Data Science, Athens
- Congratulations to PhD student Panagiotis Georgiades
Other News & Events
- Invited talk at the 8th Annual Women in Data Science Event at the American University of Beirut
- Invited talk at the 2nd Symposium on NLP for Social Good (NSG), University of Liverpool
- Invited talk at Annual Meeting of the Danish Society of Occupational and Environmental Medicine
- Advances in Data Science and Artificial Intelligence Conference 2024
- New review article on emotion detection for misinformation