NaCTeM

New article describing syntactically-informed word representations

2020-06-29

We are pleased to announce the publication of a new article in Neurocomputing, presenting novel syntactically-informed word representations (SIWRs), which allow enrichment of the pre-trained word representations with syntactic information without training language models from scratch. SIWRs are evaluated on three information extraction tasks, namely nested named entity recognition (NER), binary and n-ary relation extractions (REs), with results demonstrating that SIWRs yield performance gains over the base representations in these NLP tasks with 39% relative error reduction.

Tran, T. T., Miwa, M. and Ananiadou, S. (2020). Syntactically-Informed Word Representations from Graph Neural Network. Neurocomputing https://doi.org/10.1016/j.neucom.2020.06.070

Abstract

Most deep language understanding models depend only on word representations, which are mainly based on language modelling derived from a large amount of raw text. These models encode distributional knowledge without considering syntactic information, although several studies have shown benefits of including such information. Therefore, we propose new syntactically-informed word representations (SIWRs), which allow us to enrich the pre-trained word representations with syntactic information without training language models from scratch. To obtain SIWRs, a graph-based neural model is built on top of either static or contextualised word representations such as GloVe, ELMo and BERT. The model is first pre-trained with only a relatively modest amount of task-independent data that are automatically annotated using existing syntactic tools. SIWRs are then obtained by applying the model to downstream task data and extracting the intermediate word representations. We finally replace word representations in downstream models with SIWRs for applications. We evaluate SIWRs on three information extraction tasks, namely nested named entity recognition (NER), binary and n-ary relation extractions (REs). The results demonstrate that our SIWRs yield performance gains over the base representations in these NLP tasks with 39% relative error reduction. Our SIWRs also perform better than fine-tuning BERT in binary RE. We also conduct extensive experiments to analyse the proposed method.

Previous itemNext item
Back to news summary page