This page contains answers to questions regarding the shared task. If you have a question that is not answered here or in the shared task description, please contact the organizers at the contact email address.
Questions and answers:
- What tools and resources can be used?
- Which entities need to be recognized?
- How should additional entities (e.g. binding sites) be recognized?
- Which trigger words need to be detected?
- How is event trigger detection evaluated?
- Why are no agents specified for events?
- Can multiple "runs" be submitted?
- What are the precise evaluation criteria?
- Will results be made public?
What tools and resources can be used?
In short: with the exception of manually created annotations for test set sentences, any tools and resources can be used, including the gold annotation for the development test set.
Following experiences in previous domain shared tasks, the BioNLP'09 shared task will not contain separate "open" and "closed" tracks. Instead, any available tools and resources can be used. The one important exception is that manually created annotations of the test set data can not be used. While no Event annotation for this data has been previously published and thus all of the GENIA Event corpus data can be freely used, other corpora published previously or to be published in the near future do contain annotations for some aspects of parts of the test data.
To make it possible for participants making use of separate, large-scale annotated PubMed resources to check that no documents in the test set are included, we are making available the following list containing the PubMed IDs of documents for which manually created annotation may not be used:
For convenience, we will also seek to provide versions of other corpora that exclude test set documents.
Which entities need to be recognized?
As gold standard annotations are provided for named entities of the protein/gene/RNA types, no entity recognition is necessary to participate in Task 1. However, for Task 2 the recognition of entities relevant to the events (e.g. Site for Binding and Phosphorylation events) is required, as described in the Event Definition. It is not necessary to assign types to these entities.
How should additional entities (e.g. binding sites) be recognized?
The development of entity recognition methods for additional entities such as the Sites for Binding events is a central part of Task 2. We expect taggers such as those integrated in U-compare to be useful.
Participants are encouraged to make use of the annotations for these entities in the training data .a2 files, with the caveat that the annotation is not exhaustive: entities not relevant to events are not annotated. Additionally, other resources can be used (see question "What tools and resources can be used?").
Which trigger words need to be detected?
Participants need to detect trigger words for events, but not for negation and speculation.
How is event trigger detection evaluated?
Performance at the identification of trigger expressions (the words expressing events) will only be considered together with events in determining whether or not an event has been successfully extracted.
Event trigger extraction will thus not be separately evaluated, and trigger annotations not specified as expressing an annotated event are ignored in evaluation. Additionally, we will provide results using multiple criteria, including also a set that does not require correct identification of the event triggers.
Why are no agents specified for events?
The event specifications do not include an "Agent" slot, but entities stated as causing events such as phosphorylation are annotated, using a separate Regulation event.
Following the GENIA Event annotation scheme, in statements such as "Histone H3 phosphorylation by IKK-alpha" the enzymatic function of IKK-alpha for the Phosphorylation event is interpreted as a causal factor and annotated as a Positive_regulation event. Thus, for this statement, annotation would include (with T1:Histone H3 and T2:IKK-alpha)
| E1 | Phosphorylation:T? Theme:T1 | |
| E2 | Positive_regulation:T? Theme:E1 Cause:T2 |
(where "T?" are the identifiers of the event triggers, not relevant to this example.)
Can multiple "runs" be submitted?
Only a single final submission of results is allowed. Participants are strongly encouraged to test their methods and submission process prior to final submission on the separate development test data set that will be provided along with the training data. Multiple test submissions using the development test data are allowed.
What are the precise evaluation criteria?
The detailed evaluation criteria will be given and a web interface for evaluating development test submissions will be made available on the release of the training data.
Will results be made public?
The evaluation results will be provided to all participants and made publically available, but the identity of the participants will be blinded. Each participant can decide whether to publicize their own identity or not with full knowledge of their resuts and rank among the shared task participants.