BioNLP'09 Shared Task on
Event Extraction
in conjunction with
BioNLP, a NAACL-HLT 2009
workshop, June 4-5 2009, Boulder, Colorado
Final Evaluation Results
of Arizona State University, Humboldt-Universität zu Berlin, and Budapest University of Technology and Economics
Task 1
------------------------------------------------------------------------------------
Event Class gold (match) answer (match) recall prec. fscore
------------------------------------------------------------------------------------
Localization 174 ( 57) 70 ( 57) 32.76 81.43 46.72
Binding 347 ( 55) 156 ( 55) 15.85 35.26 21.87
Gene_expression 722 ( 368) 472 ( 368) 50.97 77.97 61.64
Transcription 137 ( 20) 52 ( 20) 14.60 38.46 21.16
Protein_catabolism 14 ( 7) 12 ( 7) 50.00 58.33 53.85
Phosphorylation 135 ( 71) 88 ( 71) 52.59 80.68 63.68
==[EVT-TOTAL]== 1529 ( 578) 850 ( 578) 37.80 68.00 48.59
------------------------------------------------------------------------------------
Regulation 291 ( 1) 13 ( 1) 0.34 7.69 0.66
Positive_regulation 983 ( 62) 216 ( 62) 6.31 28.70 10.34
Negative_regulation 379 ( 14) 28 ( 14) 3.69 50.00 6.88
==[REG-TOTAL]== 1653 ( 77) 257 ( 77) 4.66 29.96 8.06
------------------------------------------------------------------------------------
==[ALL-TOTAL]== 3182 ( 655) 1107 ( 655) 20.58 59.17 30.54
------------------------------------------------------------------------------------
------------------------------------------------------------------------------------
Event Class gold (match) answer (match) recall prec. fscore
------------------------------------------------------------------------------------
Localization 174 ( 57) 70 ( 57) 32.76 81.43 46.72
Binding 347 ( 69) 155 ( 69) 19.88 44.52 27.49
Gene_expression 722 ( 373) 472 ( 373) 51.66 79.03 62.48
Transcription 137 ( 22) 52 ( 22) 16.06 42.31 23.28
Protein_catabolism 14 ( 7) 12 ( 7) 50.00 58.33 53.85
Phosphorylation 135 ( 74) 88 ( 74) 54.81 84.09 66.37
==[EVT-TOTAL]== 1529 ( 602) 849 ( 602) 39.37 70.91 50.63
------------------------------------------------------------------------------------
Regulation 291 ( 1) 13 ( 1) 0.34 7.69 0.66
Positive_regulation 983 ( 70) 216 ( 70) 7.12 32.41 11.68
Negative_regulation 379 ( 15) 28 ( 15) 3.96 53.57 7.37
==[REG-TOTAL]== 1653 ( 86) 257 ( 86) 5.20 33.46 9.01
------------------------------------------------------------------------------------
==[ALL-TOTAL]== 3182 ( 688) 1106 ( 688) 21.62 62.21 32.09
------------------------------------------------------------------------------------
------------------------------------------------------------------------------------
Event Class gold (match) answer (match) recall prec. fscore
------------------------------------------------------------------------------------
Localization 174 ( 57) 70 ( 57) 32.76 81.43 46.72
Binding 347 ( 69) 155 ( 69) 19.88 44.52 27.49
Gene_expression 722 ( 373) 472 ( 373) 51.66 79.03 62.48
Transcription 137 ( 22) 52 ( 22) 16.06 42.31 23.28
Protein_catabolism 14 ( 7) 12 ( 7) 50.00 58.33 53.85
Phosphorylation 135 ( 74) 88 ( 74) 54.81 84.09 66.37
==[EVT-TOTAL]== 1529 ( 602) 849 ( 602) 39.37 70.91 50.63
------------------------------------------------------------------------------------
Regulation 291 ( 1) 13 ( 1) 0.34 7.69 0.66
Positive_regulation 983 ( 70) 216 ( 70) 7.12 32.41 11.68
Negative_regulation 379 ( 15) 28 ( 15) 3.96 53.57 7.37
==[REG-TOTAL]== 1653 ( 86) 257 ( 86) 5.20 33.46 9.01
------------------------------------------------------------------------------------
==[ALL-TOTAL]== 3182 ( 688) 1106 ( 688) 21.62 62.21 32.09
------------------------------------------------------------------------------------
------------------------------------------------------------------------------------
Event Class gold (match) answer (match) recall prec. fscore
------------------------------------------------------------------------------------
Localization 174 ( 57) 70 ( 57) 32.76 81.43 46.72
Binding 398 ( 76) 156 ( 76) 19.10 48.72 27.44
Gene_expression 722 ( 368) 472 ( 368) 50.97 77.97 61.64
Transcription 137 ( 20) 52 ( 20) 14.60 38.46 21.16
Protein_catabolism 14 ( 7) 12 ( 7) 50.00 58.33 53.85
Phosphorylation 135 ( 71) 88 ( 71) 52.59 80.68 63.68
==[EVT-TOTAL]== 1580 ( 599) 850 ( 599) 37.91 70.47 49.30
------------------------------------------------------------------------------------
Regulation 338 ( 1) 13 ( 1) 0.30 7.69 0.57
Positive_regulation 1186 ( 90) 216 ( 90) 7.59 41.67 12.84
Negative_regulation 416 ( 15) 28 ( 15) 3.61 53.57 6.76
==[REG-TOTAL]== 1940 ( 106) 257 ( 106) 5.46 41.25 9.65
------------------------------------------------------------------------------------
==[ALL-TOTAL]== 3520 ( 705) 1107 ( 705) 20.03 63.69 30.47
------------------------------------------------------------------------------------
------------------------------------------------------------------------------------
Event Class gold (match) answer (match) recall prec. fscore
------------------------------------------------------------------------------------
Localization 174 ( 57) 70 ( 57) 32.76 81.43 46.72
Binding 398 ( 90) 155 ( 90) 22.61 58.06 32.55
Gene_expression 722 ( 373) 472 ( 373) 51.66 79.03 62.48
Transcription 137 ( 22) 52 ( 22) 16.06 42.31 23.28
Protein_catabolism 14 ( 7) 12 ( 7) 50.00 58.33 53.85
Phosphorylation 135 ( 74) 88 ( 74) 54.81 84.09 66.37
==[EVT-TOTAL]== 1580 ( 623) 849 ( 623) 39.43 73.38 51.30
------------------------------------------------------------------------------------
Regulation 338 ( 1) 13 ( 1) 0.30 7.69 0.57
Positive_regulation 1186 ( 103) 216 ( 103) 8.68 47.69 14.69
Negative_regulation 416 ( 18) 28 ( 18) 4.33 64.29 8.11
==[REG-TOTAL]== 1940 ( 122) 257 ( 122) 6.29 47.47 11.11
------------------------------------------------------------------------------------
==[ALL-TOTAL]== 3520 ( 745) 1106 ( 745) 21.16 67.36 32.21
------------------------------------------------------------------------------------
Task 2
------------------------------------------------------------------------------------
Event Class gold (match) answer (match) recall prec. fscore
------------------------------------------------------------------------------------
Localization 174 ( 42) 70 ( 42) 24.14 60.00 34.43
Binding 349 ( 55) 156 ( 55) 15.76 35.26 21.78
Gene_expression 722 ( 368) 472 ( 368) 50.97 77.97 61.64
Transcription 137 ( 20) 52 ( 20) 14.60 38.46 21.16
Protein_catabolism 14 ( 7) 12 ( 7) 50.00 58.33 53.85
Phosphorylation 139 ( 28) 88 ( 28) 20.14 31.82 24.67
==[EVT-TOTAL]== 1535 ( 520) 850 ( 520) 33.88 61.18 43.61
------------------------------------------------------------------------------------
Regulation 292 ( 1) 13 ( 1) 0.34 7.69 0.66
Positive_regulation 987 ( 58) 216 ( 58) 5.88 26.85 9.64
Negative_regulation 379 ( 13) 28 ( 13) 3.43 46.43 6.39
==[REG-TOTAL]== 1658 ( 72) 257 ( 72) 4.34 28.02 7.52
------------------------------------------------------------------------------------
==[ALL-TOTAL]== 3193 ( 592) 1107 ( 592) 18.54 53.48 27.53
------------------------------------------------------------------------------------
------------------------------------------------------------------------------------
Event Class gold (match) answer (match) recall prec. fscore
------------------------------------------------------------------------------------
Localization 174 ( 42) 70 ( 42) 24.14 60.00 34.43
Binding 349 ( 69) 155 ( 69) 19.77 44.52 27.38
Gene_expression 722 ( 373) 472 ( 373) 51.66 79.03 62.48
Transcription 137 ( 22) 52 ( 22) 16.06 42.31 23.28
Protein_catabolism 14 ( 7) 12 ( 7) 50.00 58.33 53.85
Phosphorylation 139 ( 31) 88 ( 31) 22.30 35.23 27.31
==[EVT-TOTAL]== 1535 ( 544) 849 ( 544) 35.44 64.08 45.64
------------------------------------------------------------------------------------
Regulation 292 ( 1) 13 ( 1) 0.34 7.69 0.66
Positive_regulation 987 ( 66) 216 ( 66) 6.69 30.56 10.97
Negative_regulation 379 ( 14) 28 ( 14) 3.69 50.00 6.88
==[REG-TOTAL]== 1658 ( 81) 257 ( 81) 4.89 31.52 8.46
------------------------------------------------------------------------------------
==[ALL-TOTAL]== 3193 ( 625) 1106 ( 625) 19.57 56.51 29.08
------------------------------------------------------------------------------------
------------------------------------------------------------------------------------
Event Class gold (match) answer (match) recall prec. fscore
------------------------------------------------------------------------------------
Localization 174 ( 42) 70 ( 42) 24.14 60.00 34.43
Binding 349 ( 69) 155 ( 69) 19.77 44.52 27.38
Gene_expression 722 ( 373) 472 ( 373) 51.66 79.03 62.48
Transcription 137 ( 22) 52 ( 22) 16.06 42.31 23.28
Protein_catabolism 14 ( 7) 12 ( 7) 50.00 58.33 53.85
Phosphorylation 139 ( 31) 88 ( 31) 22.30 35.23 27.31
==[EVT-TOTAL]== 1535 ( 544) 849 ( 544) 35.44 64.08 45.64
------------------------------------------------------------------------------------
Regulation 292 ( 1) 13 ( 1) 0.34 7.69 0.66
Positive_regulation 987 ( 70) 216 ( 70) 7.09 32.41 11.64
Negative_regulation 379 ( 14) 28 ( 14) 3.69 50.00 6.88
==[REG-TOTAL]== 1658 ( 85) 257 ( 85) 5.13 33.07 8.88
------------------------------------------------------------------------------------
==[ALL-TOTAL]== 3193 ( 629) 1106 ( 629) 19.70 56.87 29.26
------------------------------------------------------------------------------------
------------------------------------------------------------------------------------
Event Class gold (match) answer (match) recall prec. fscore
------------------------------------------------------------------------------------
Localization 239 ( 57) 70 ( 57) 23.85 81.43 36.89
Binding 419 ( 76) 156 ( 76) 18.14 48.72 26.43
Gene_expression 722 ( 368) 472 ( 368) 50.97 77.97 61.64
Transcription 137 ( 20) 52 ( 20) 14.60 38.46 21.16
Protein_catabolism 14 ( 7) 12 ( 7) 50.00 58.33 53.85
Phosphorylation 191 ( 71) 88 ( 71) 37.17 80.68 50.90
==[EVT-TOTAL]== 1722 ( 599) 850 ( 599) 34.79 70.47 46.58
------------------------------------------------------------------------------------
Regulation 349 ( 1) 13 ( 1) 0.29 7.69 0.55
Positive_regulation 1220 ( 90) 216 ( 90) 7.38 41.67 12.53
Negative_regulation 425 ( 15) 28 ( 15) 3.53 53.57 6.62
==[REG-TOTAL]== 1994 ( 106) 257 ( 106) 5.32 41.25 9.42
------------------------------------------------------------------------------------
==[ALL-TOTAL]== 3716 ( 705) 1107 ( 705) 18.97 63.69 29.23
------------------------------------------------------------------------------------
------------------------------------------------------------------------------------
Event Class gold (match) answer (match) recall prec. fscore
------------------------------------------------------------------------------------
Localization 239 ( 57) 70 ( 57) 23.85 81.43 36.89
Binding 419 ( 90) 155 ( 90) 21.48 58.06 31.36
Gene_expression 722 ( 373) 472 ( 373) 51.66 79.03 62.48
Transcription 137 ( 22) 52 ( 22) 16.06 42.31 23.28
Protein_catabolism 14 ( 7) 12 ( 7) 50.00 58.33 53.85
Phosphorylation 191 ( 74) 88 ( 74) 38.74 84.09 53.05
==[EVT-TOTAL]== 1722 ( 623) 849 ( 623) 36.18 73.38 48.46
------------------------------------------------------------------------------------
Regulation 349 ( 1) 13 ( 1) 0.29 7.69 0.55
Positive_regulation 1220 ( 103) 216 ( 103) 8.44 47.69 14.35
Negative_regulation 425 ( 18) 28 ( 18) 4.24 64.29 7.95
==[REG-TOTAL]== 1994 ( 122) 257 ( 122) 6.12 47.47 10.84
------------------------------------------------------------------------------------
==[ALL-TOTAL]== 3716 ( 745) 1106 ( 745) 20.05 67.36 30.90
------------------------------------------------------------------------------------
Task 3
------------------------------------------------------------------------------------
Event Class gold (match) answer (match) recall prec. fscore
------------------------------------------------------------------------------------
Localization 174 ( 57) 70 ( 57) 32.76 81.43 46.72
Binding 347 ( 55) 156 ( 55) 15.85 35.26 21.87
Gene_expression 722 ( 368) 472 ( 368) 50.97 77.97 61.64
Transcription 137 ( 20) 52 ( 20) 14.60 38.46 21.16
Protein_catabolism 14 ( 7) 12 ( 7) 50.00 58.33 53.85
Phosphorylation 135 ( 71) 88 ( 71) 52.59 80.68 63.68
==[EVT-TOTAL]== 1529 ( 578) 850 ( 578) 37.80 68.00 48.59
------------------------------------------------------------------------------------
Regulation 291 ( 1) 13 ( 1) 0.34 7.69 0.66
Positive_regulation 983 ( 62) 216 ( 62) 6.31 28.70 10.34
Negative_regulation 379 ( 14) 28 ( 14) 3.69 50.00 6.88
==[REG-TOTAL]== 1653 ( 77) 257 ( 77) 4.66 29.96 8.06
------------------------------------------------------------------------------------
Negation 227 ( 9) 33 ( 9) 3.96 27.27 6.92
Speculation 208 ( 13) 46 ( 13) 6.25 28.26 10.24
==[MOD-TOTAL]== 435 ( 22) 79 ( 22) 5.06 27.85 8.56
------------------------------------------------------------------------------------
==[ALL-TOTAL]== 3617 ( 677) 1186 ( 677) 18.72 57.08 28.19
------------------------------------------------------------------------------------
------------------------------------------------------------------------------------
Event Class gold (match) answer (match) recall prec. fscore
------------------------------------------------------------------------------------
Localization 174 ( 57) 70 ( 57) 32.76 81.43 46.72
Binding 347 ( 69) 155 ( 69) 19.88 44.52 27.49
Gene_expression 722 ( 373) 472 ( 373) 51.66 79.03 62.48
Transcription 137 ( 22) 52 ( 22) 16.06 42.31 23.28
Protein_catabolism 14 ( 7) 12 ( 7) 50.00 58.33 53.85
Phosphorylation 135 ( 74) 88 ( 74) 54.81 84.09 66.37
==[EVT-TOTAL]== 1529 ( 602) 849 ( 602) 39.37 70.91 50.63
------------------------------------------------------------------------------------
Regulation 291 ( 1) 13 ( 1) 0.34 7.69 0.66
Positive_regulation 983 ( 70) 216 ( 70) 7.12 32.41 11.68
Negative_regulation 379 ( 15) 28 ( 15) 3.96 53.57 7.37
==[REG-TOTAL]== 1653 ( 86) 257 ( 86) 5.20 33.46 9.01
------------------------------------------------------------------------------------
Negation 227 ( 9) 33 ( 9) 3.96 27.27 6.92
Speculation 208 ( 13) 46 ( 13) 6.25 28.26 10.24
==[MOD-TOTAL]== 435 ( 22) 79 ( 22) 5.06 27.85 8.56
------------------------------------------------------------------------------------
==[ALL-TOTAL]== 3617 ( 710) 1185 ( 710) 19.63 59.92 29.57
------------------------------------------------------------------------------------
------------------------------------------------------------------------------------
Event Class gold (match) answer (match) recall prec. fscore
------------------------------------------------------------------------------------
Localization 174 ( 57) 70 ( 57) 32.76 81.43 46.72
Binding 347 ( 69) 155 ( 69) 19.88 44.52 27.49
Gene_expression 722 ( 373) 472 ( 373) 51.66 79.03 62.48
Transcription 137 ( 22) 52 ( 22) 16.06 42.31 23.28
Protein_catabolism 14 ( 7) 12 ( 7) 50.00 58.33 53.85
Phosphorylation 135 ( 74) 88 ( 74) 54.81 84.09 66.37
==[EVT-TOTAL]== 1529 ( 602) 849 ( 602) 39.37 70.91 50.63
------------------------------------------------------------------------------------
Regulation 291 ( 1) 13 ( 1) 0.34 7.69 0.66
Positive_regulation 983 ( 70) 216 ( 70) 7.12 32.41 11.68
Negative_regulation 379 ( 15) 28 ( 15) 3.96 53.57 7.37
==[REG-TOTAL]== 1653 ( 86) 257 ( 86) 5.20 33.46 9.01
------------------------------------------------------------------------------------
Negation 227 ( 9) 33 ( 9) 3.96 27.27 6.92
Speculation 208 ( 13) 46 ( 13) 6.25 28.26 10.24
==[MOD-TOTAL]== 435 ( 22) 79 ( 22) 5.06 27.85 8.56
------------------------------------------------------------------------------------
==[ALL-TOTAL]== 3617 ( 710) 1185 ( 710) 19.63 59.92 29.57
------------------------------------------------------------------------------------
------------------------------------------------------------------------------------
Event Class gold (match) answer (match) recall prec. fscore
------------------------------------------------------------------------------------
Localization 174 ( 57) 70 ( 57) 32.76 81.43 46.72
Binding 398 ( 76) 156 ( 76) 19.10 48.72 27.44
Gene_expression 722 ( 368) 472 ( 368) 50.97 77.97 61.64
Transcription 137 ( 20) 52 ( 20) 14.60 38.46 21.16
Protein_catabolism 14 ( 7) 12 ( 7) 50.00 58.33 53.85
Phosphorylation 135 ( 71) 88 ( 71) 52.59 80.68 63.68
==[EVT-TOTAL]== 1580 ( 599) 850 ( 599) 37.91 70.47 49.30
------------------------------------------------------------------------------------
Regulation 338 ( 1) 13 ( 1) 0.30 7.69 0.57
Positive_regulation 1186 ( 90) 216 ( 90) 7.59 41.67 12.84
Negative_regulation 416 ( 15) 28 ( 15) 3.61 53.57 6.76
==[REG-TOTAL]== 1940 ( 106) 257 ( 106) 5.46 41.25 9.65
------------------------------------------------------------------------------------
Negation 220 ( 9) 33 ( 9) 4.09 27.27 7.11
Speculation 201 ( 13) 46 ( 13) 6.47 28.26 10.53
==[MOD-TOTAL]== 421 ( 22) 79 ( 22) 5.23 27.85 8.80
------------------------------------------------------------------------------------
==[ALL-TOTAL]== 3941 ( 727) 1186 ( 727) 18.45 61.30 28.36
------------------------------------------------------------------------------------
------------------------------------------------------------------------------------
Event Class gold (match) answer (match) recall prec. fscore
------------------------------------------------------------------------------------
Localization 174 ( 57) 70 ( 57) 32.76 81.43 46.72
Binding 398 ( 90) 155 ( 90) 22.61 58.06 32.55
Gene_expression 722 ( 373) 472 ( 373) 51.66 79.03 62.48
Transcription 137 ( 22) 52 ( 22) 16.06 42.31 23.28
Protein_catabolism 14 ( 7) 12 ( 7) 50.00 58.33 53.85
Phosphorylation 135 ( 74) 88 ( 74) 54.81 84.09 66.37
==[EVT-TOTAL]== 1580 ( 623) 849 ( 623) 39.43 73.38 51.30
------------------------------------------------------------------------------------
Regulation 338 ( 1) 13 ( 1) 0.30 7.69 0.57
Positive_regulation 1186 ( 103) 216 ( 103) 8.68 47.69 14.69
Negative_regulation 416 ( 18) 28 ( 18) 4.33 64.29 8.11
==[REG-TOTAL]== 1940 ( 122) 257 ( 122) 6.29 47.47 11.11
------------------------------------------------------------------------------------
Negation 220 ( 9) 33 ( 9) 4.09 27.27 7.11
Speculation 201 ( 13) 46 ( 13) 6.47 28.26 10.53
==[MOD-TOTAL]== 421 ( 22) 79 ( 22) 5.23 27.85 8.80
------------------------------------------------------------------------------------
==[ALL-TOTAL]== 3941 ( 767) 1185 ( 767) 19.46 64.73 29.93
------------------------------------------------------------------------------------