BioNLP'09 Shared Task on
Event Extraction
in conjunction with
BioNLP, a NAACL-HLT 2009
workshop, June 4-5 2009, Boulder, Colorado
Final Evaluation Results
of NICTA/University of Melbourne
Task 1
------------------------------------------------------------------------------------
Event Class gold (match) answer (match) recall prec. fscore
------------------------------------------------------------------------------------
Localization 174 ( 41) 66 ( 41) 23.56 62.12 34.17
Binding 347 ( 55) 200 ( 55) 15.85 27.50 20.11
Gene_expression 722 ( 224) 265 ( 224) 31.02 84.53 45.39
Transcription 137 ( 30) 44 ( 30) 21.90 68.18 33.15
Protein_catabolism 14 ( 6) 7 ( 6) 42.86 85.71 57.14
Phosphorylation 135 ( 59) 95 ( 59) 43.70 62.11 51.30
==[EVT-TOTAL]== 1529 ( 415) 677 ( 415) 27.14 61.30 37.62
------------------------------------------------------------------------------------
Regulation 291 ( 37) 339 ( 37) 12.71 10.91 11.75
Positive_regulation 983 ( 44) 151 ( 44) 4.48 29.14 7.76
Negative_regulation 379 ( 36) 222 ( 36) 9.50 16.22 11.98
==[REG-TOTAL]== 1653 ( 117) 712 ( 117) 7.08 16.43 9.89
------------------------------------------------------------------------------------
==[ALL-TOTAL]== 3182 ( 532) 1389 ( 532) 16.72 38.30 23.28
------------------------------------------------------------------------------------
------------------------------------------------------------------------------------
Event Class gold (match) answer (match) recall prec. fscore
------------------------------------------------------------------------------------
Localization 174 ( 42) 66 ( 42) 24.14 63.64 35.00
Binding 347 ( 58) 200 ( 58) 16.71 29.00 21.21
Gene_expression 722 ( 228) 265 ( 228) 31.58 86.04 46.20
Transcription 137 ( 31) 43 ( 31) 22.63 72.09 34.44
Protein_catabolism 14 ( 6) 7 ( 6) 42.86 85.71 57.14
Phosphorylation 135 ( 61) 95 ( 61) 45.19 64.21 53.04
==[EVT-TOTAL]== 1529 ( 426) 676 ( 426) 27.86 63.02 38.64
------------------------------------------------------------------------------------
Regulation 291 ( 43) 339 ( 43) 14.78 12.68 13.65
Positive_regulation 983 ( 45) 151 ( 45) 4.58 29.80 7.94
Negative_regulation 379 ( 41) 222 ( 41) 10.82 18.47 13.64
==[REG-TOTAL]== 1653 ( 129) 712 ( 129) 7.80 18.12 10.91
------------------------------------------------------------------------------------
==[ALL-TOTAL]== 3182 ( 555) 1388 ( 555) 17.44 39.99 24.29
------------------------------------------------------------------------------------
------------------------------------------------------------------------------------
Event Class gold (match) answer (match) recall prec. fscore
------------------------------------------------------------------------------------
Localization 174 ( 42) 66 ( 42) 24.14 63.64 35.00
Binding 347 ( 58) 200 ( 58) 16.71 29.00 21.21
Gene_expression 722 ( 228) 265 ( 228) 31.58 86.04 46.20
Transcription 137 ( 31) 43 ( 31) 22.63 72.09 34.44
Protein_catabolism 14 ( 6) 7 ( 6) 42.86 85.71 57.14
Phosphorylation 135 ( 61) 95 ( 61) 45.19 64.21 53.04
==[EVT-TOTAL]== 1529 ( 426) 676 ( 426) 27.86 63.02 38.64
------------------------------------------------------------------------------------
Regulation 291 ( 43) 339 ( 43) 14.78 12.68 13.65
Positive_regulation 983 ( 45) 151 ( 45) 4.58 29.80 7.94
Negative_regulation 379 ( 41) 222 ( 41) 10.82 18.47 13.64
==[REG-TOTAL]== 1653 ( 129) 712 ( 129) 7.80 18.12 10.91
------------------------------------------------------------------------------------
==[ALL-TOTAL]== 3182 ( 555) 1388 ( 555) 17.44 39.99 24.29
------------------------------------------------------------------------------------
------------------------------------------------------------------------------------
Event Class gold (match) answer (match) recall prec. fscore
------------------------------------------------------------------------------------
Localization 174 ( 41) 66 ( 41) 23.56 62.12 34.17
Binding 398 ( 140) 314 ( 140) 35.18 44.59 39.33
Gene_expression 722 ( 224) 265 ( 224) 31.02 84.53 45.39
Transcription 137 ( 30) 44 ( 30) 21.90 68.18 33.15
Protein_catabolism 14 ( 6) 7 ( 6) 42.86 85.71 57.14
Phosphorylation 135 ( 59) 95 ( 59) 43.70 62.11 51.30
==[EVT-TOTAL]== 1580 ( 500) 791 ( 500) 31.65 63.21 42.18
------------------------------------------------------------------------------------
Regulation 338 ( 47) 354 ( 47) 13.91 13.28 13.58
Positive_regulation 1186 ( 60) 154 ( 60) 5.06 38.96 8.96
Negative_regulation 416 ( 45) 243 ( 45) 10.82 18.52 13.66
==[REG-TOTAL]== 1940 ( 152) 751 ( 152) 7.84 20.24 11.30
------------------------------------------------------------------------------------
==[ALL-TOTAL]== 3520 ( 652) 1542 ( 652) 18.52 42.28 25.76
------------------------------------------------------------------------------------
------------------------------------------------------------------------------------
Event Class gold (match) answer (match) recall prec. fscore
------------------------------------------------------------------------------------
Localization 174 ( 42) 66 ( 42) 24.14 63.64 35.00
Binding 398 ( 147) 314 ( 147) 36.93 46.82 41.29
Gene_expression 722 ( 228) 265 ( 228) 31.58 86.04 46.20
Transcription 137 ( 31) 43 ( 31) 22.63 72.09 34.44
Protein_catabolism 14 ( 6) 7 ( 6) 42.86 85.71 57.14
Phosphorylation 135 ( 61) 95 ( 61) 45.19 64.21 53.04
==[EVT-TOTAL]== 1580 ( 515) 790 ( 515) 32.59 65.19 43.46
------------------------------------------------------------------------------------
Regulation 338 ( 54) 354 ( 54) 15.98 15.25 15.61
Positive_regulation 1186 ( 63) 154 ( 63) 5.31 40.91 9.40
Negative_regulation 416 ( 50) 243 ( 50) 12.02 20.58 15.17
==[REG-TOTAL]== 1940 ( 167) 751 ( 167) 8.61 22.24 12.41
------------------------------------------------------------------------------------
==[ALL-TOTAL]== 3520 ( 682) 1541 ( 682) 19.38 44.26 26.95
------------------------------------------------------------------------------------
Task 2
N/A
N/A
N/A
N/A
N/A
Task 3
------------------------------------------------------------------------------------
Event Class gold (match) answer (match) recall prec. fscore
------------------------------------------------------------------------------------
Localization 174 ( 41) 66 ( 41) 23.56 62.12 34.17
Binding 347 ( 55) 200 ( 55) 15.85 27.50 20.11
Gene_expression 722 ( 224) 265 ( 224) 31.02 84.53 45.39
Transcription 137 ( 30) 44 ( 30) 21.90 68.18 33.15
Protein_catabolism 14 ( 6) 7 ( 6) 42.86 85.71 57.14
Phosphorylation 135 ( 59) 95 ( 59) 43.70 62.11 51.30
==[EVT-TOTAL]== 1529 ( 415) 677 ( 415) 27.14 61.30 37.62
------------------------------------------------------------------------------------
Regulation 291 ( 37) 339 ( 37) 12.71 10.91 11.75
Positive_regulation 983 ( 44) 151 ( 44) 4.48 29.14 7.76
Negative_regulation 379 ( 36) 222 ( 36) 9.50 16.22 11.98
==[REG-TOTAL]== 1653 ( 117) 712 ( 117) 7.08 16.43 9.89
------------------------------------------------------------------------------------
Negation 227 ( 6) 29 ( 6) 2.64 20.69 4.69
Speculation 208 ( 10) 33 ( 10) 4.81 30.30 8.30
==[MOD-TOTAL]== 435 ( 16) 62 ( 16) 3.68 25.81 6.44
------------------------------------------------------------------------------------
==[ALL-TOTAL]== 3617 ( 548) 1451 ( 548) 15.15 37.77 21.63
------------------------------------------------------------------------------------
------------------------------------------------------------------------------------
Event Class gold (match) answer (match) recall prec. fscore
------------------------------------------------------------------------------------
Localization 174 ( 42) 66 ( 42) 24.14 63.64 35.00
Binding 347 ( 58) 200 ( 58) 16.71 29.00 21.21
Gene_expression 722 ( 228) 265 ( 228) 31.58 86.04 46.20
Transcription 137 ( 31) 43 ( 31) 22.63 72.09 34.44
Protein_catabolism 14 ( 6) 7 ( 6) 42.86 85.71 57.14
Phosphorylation 135 ( 61) 95 ( 61) 45.19 64.21 53.04
==[EVT-TOTAL]== 1529 ( 426) 676 ( 426) 27.86 63.02 38.64
------------------------------------------------------------------------------------
Regulation 291 ( 43) 339 ( 43) 14.78 12.68 13.65
Positive_regulation 983 ( 45) 151 ( 45) 4.58 29.80 7.94
Negative_regulation 379 ( 41) 222 ( 41) 10.82 18.47 13.64
==[REG-TOTAL]== 1653 ( 129) 712 ( 129) 7.80 18.12 10.91
------------------------------------------------------------------------------------
Negation 227 ( 7) 29 ( 7) 3.08 24.14 5.47
Speculation 208 ( 10) 33 ( 10) 4.81 30.30 8.30
==[MOD-TOTAL]== 435 ( 17) 62 ( 17) 3.91 27.42 6.84
------------------------------------------------------------------------------------
==[ALL-TOTAL]== 3617 ( 572) 1450 ( 572) 15.81 39.45 22.58
------------------------------------------------------------------------------------
------------------------------------------------------------------------------------
Event Class gold (match) answer (match) recall prec. fscore
------------------------------------------------------------------------------------
Localization 174 ( 42) 66 ( 42) 24.14 63.64 35.00
Binding 347 ( 58) 200 ( 58) 16.71 29.00 21.21
Gene_expression 722 ( 228) 265 ( 228) 31.58 86.04 46.20
Transcription 137 ( 31) 43 ( 31) 22.63 72.09 34.44
Protein_catabolism 14 ( 6) 7 ( 6) 42.86 85.71 57.14
Phosphorylation 135 ( 61) 95 ( 61) 45.19 64.21 53.04
==[EVT-TOTAL]== 1529 ( 426) 676 ( 426) 27.86 63.02 38.64
------------------------------------------------------------------------------------
Regulation 291 ( 43) 339 ( 43) 14.78 12.68 13.65
Positive_regulation 983 ( 45) 151 ( 45) 4.58 29.80 7.94
Negative_regulation 379 ( 41) 222 ( 41) 10.82 18.47 13.64
==[REG-TOTAL]== 1653 ( 129) 712 ( 129) 7.80 18.12 10.91
------------------------------------------------------------------------------------
Negation 227 ( 12) 29 ( 10) 5.29 34.48 9.17
Speculation 208 ( 10) 33 ( 10) 4.81 30.30 8.30
==[MOD-TOTAL]== 435 ( 22) 62 ( 20) 5.06 32.26 8.74
------------------------------------------------------------------------------------
==[ALL-TOTAL]== 3617 ( 577) 1450 ( 575) 15.95 39.66 22.75
------------------------------------------------------------------------------------
------------------------------------------------------------------------------------
Event Class gold (match) answer (match) recall prec. fscore
------------------------------------------------------------------------------------
Localization 174 ( 41) 66 ( 41) 23.56 62.12 34.17
Binding 398 ( 140) 314 ( 140) 35.18 44.59 39.33
Gene_expression 722 ( 224) 265 ( 224) 31.02 84.53 45.39
Transcription 137 ( 30) 44 ( 30) 21.90 68.18 33.15
Protein_catabolism 14 ( 6) 7 ( 6) 42.86 85.71 57.14
Phosphorylation 135 ( 59) 95 ( 59) 43.70 62.11 51.30
==[EVT-TOTAL]== 1580 ( 500) 791 ( 500) 31.65 63.21 42.18
------------------------------------------------------------------------------------
Regulation 338 ( 47) 354 ( 47) 13.91 13.28 13.58
Positive_regulation 1186 ( 60) 154 ( 60) 5.06 38.96 8.96
Negative_regulation 416 ( 45) 243 ( 45) 10.82 18.52 13.66
==[REG-TOTAL]== 1940 ( 152) 751 ( 152) 7.84 20.24 11.30
------------------------------------------------------------------------------------
Negation 220 ( 9) 31 ( 9) 4.09 29.03 7.17
Speculation 201 ( 10) 39 ( 10) 4.98 25.64 8.33
==[MOD-TOTAL]== 421 ( 19) 70 ( 19) 4.51 27.14 7.74
------------------------------------------------------------------------------------
==[ALL-TOTAL]== 3941 ( 671) 1612 ( 671) 17.03 41.63 24.17
------------------------------------------------------------------------------------
------------------------------------------------------------------------------------
Event Class gold (match) answer (match) recall prec. fscore
------------------------------------------------------------------------------------
Localization 174 ( 42) 66 ( 42) 24.14 63.64 35.00
Binding 398 ( 147) 314 ( 147) 36.93 46.82 41.29
Gene_expression 722 ( 228) 265 ( 228) 31.58 86.04 46.20
Transcription 137 ( 31) 43 ( 31) 22.63 72.09 34.44
Protein_catabolism 14 ( 6) 7 ( 6) 42.86 85.71 57.14
Phosphorylation 135 ( 61) 95 ( 61) 45.19 64.21 53.04
==[EVT-TOTAL]== 1580 ( 515) 790 ( 515) 32.59 65.19 43.46
------------------------------------------------------------------------------------
Regulation 338 ( 54) 354 ( 54) 15.98 15.25 15.61
Positive_regulation 1186 ( 63) 154 ( 63) 5.31 40.91 9.40
Negative_regulation 416 ( 50) 243 ( 50) 12.02 20.58 15.17
==[REG-TOTAL]== 1940 ( 167) 751 ( 167) 8.61 22.24 12.41
------------------------------------------------------------------------------------
Negation 220 ( 10) 31 ( 10) 4.55 32.26 7.97
Speculation 201 ( 10) 39 ( 10) 4.98 25.64 8.33
==[MOD-TOTAL]== 421 ( 20) 70 ( 20) 4.75 28.57 8.15
------------------------------------------------------------------------------------
==[ALL-TOTAL]== 3941 ( 702) 1611 ( 702) 17.81 43.58 25.29
------------------------------------------------------------------------------------