Scorer for SENSEVAL-2
SENSEVAL-2:
The scoring carried out by scorer.pl is
applicable to the SENSEVAL-2 all-words tasks. It carries out
fine-grained scoring according to the SENSEVAL-2 scorescheme.txt
guidelines, which are available from the SENSEVAL website. Three further changes were made to this scheme:
- If an answer was given more than once for the same identifier,
only the first of these attempts was counted.
- Identifiers with no answers on the line are not counted as
attempts. E.g. d00 d00.s00.t01 does not count as an attempt whereas,
d00 d00.s00.t01 art%1:09:00:: is an attempt.
- The program caters for multiple occurrences of an answer in the
gold standard. E.g. d01 d01.s86.t15 close%3:00:01:: close%3:00:05::
close%3:00:05::.
These changes account for the difference in results from this scorer
and the published results between some systems in the English
all-words task.