Technical reports
Artificial error generation for translation-based grammatical error correction
October 2016, 155 pages
This technical report is based on a dissertation submitted October 2016 by the author for the degree of Doctor of Philosophy to the University of Cambridge, Hughes Hall.
This version of the report incorporates minor changes to the October 2016 original, which were released November 2016.
DOI: 10.48456/tr-895
Abstract
Automated grammatical error correction for language learners has attracted a lot of attention in recent years, especially after a number of shared tasks that have encouraged research in the area. Treating the problem as a translation task from ‘incorrect’ into ‘correct’ English using statistical machine translation has emerged as a state-of-the-art approach but it requires vast amounts of corrected parallel data to produce useful results. Because manual annotation of incorrect text is laborious and expensive, we can generate artificial error-annotated data by injecting errors deliberately into correct text and thus produce larger amounts of parallel data with much less effort.
In this work, we review previous work on artificial error generation and investigate new approaches using random and probabilistic methods for constrained and general error correction. Our methods use error statistics from a reference corpus of learner writing to generate errors in native text that look realistic and plausible in context. We investigate a number of aspects that can play a part in the error generation process, such as the origin of the native texts, the amount of context used to find suitable insertion points, the type of information encoded by the error patterns and the output error distribution. In addition, we explore the use of linguistic information for characterising errors and train systems using different combinations of real and artificial data.
Results of our experiments show that the use of artificial errors can improve system performance when they are used in combination with real learner errors, in line with previous research. These improvements are observed for both constrained and general correction, for which probabilistic methods produce the best results. We also demonstrate that systems trained on a combination of real and artificial errors can beat other highly-engineered systems and be more robust, showing that performance can be improved by focusing on the data rather than tuning system parameters.
Part of our work is also devoted to the proposal of the I-measure, a new evaluation scheme that scores corrections in terms of improvement on the original text and solves known issues with existing evaluation measures.
Full text
PDF (1.2 MB)
BibTeX record
@TechReport{UCAM-CL-TR-895,
author = {Felice, Mariano},
title = {{Artificial error generation for translation-based
grammatical error correction}},
year = 2016,
month = oct,
url = {https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-895.pdf},
institution = {University of Cambridge, Computer Laboratory},
doi = {10.48456/tr-895},
number = {UCAM-CL-TR-895}
}