Machine Learning for Language Processing

Lectures

The notes and slides will be added incrementally.
• Lecture 1: Classification with the Perceptron
• Lecture 2: Probabilistic classification
• Lecture 3: Optimization fundamentals
• Lecture 4: Feed Forward Neural Networks
• Lecture 5: Structured Prediction
• Lecture 6: Sequence2Sequence
• Lecture 7: Incremental Structured Prediction
• Lecture 8: Unlabeled Data

*************

Reading seminars

Instructions

Please select 3 papers that you would like to present in order of preference by noon on the 13th October and email your selections to [Javascript required]. I will assign papers by 5pm that day. Do not do this if you are only planning to audit the course. Instead email me and let me know.

There will be 2 presentations per 50 minute session. Your presentations should be about 15 minutes allowing for a further 5 minutes for questions, and 10 minutes at the end of each session for general discussion. You should summarise the paper briefly (remember everyone will have read it), explicate any parts you found difficult or innovative, and critically evaluate the work described. For your evaluation you should consider questions like: To what extent have the stated aims of the research been achieved? To what extent is the work replicable given the information provided? In what way does the work advance the state of the art?, etc. You may prepare slides and share your screen via the data projector. You should liaise with your co-presenter to decide the order in which to make presentations. You should have all slides for the session loaded onto a single laptop set up with the data projector by the beginning of each session.

All students should read all the papers and attend all sessions prepared to discuss each paper after the presentations

Assignment

You may undertake a small project using existing datasets and machine learning software, and then submit a project report. Your project should define an experiment (probably comparative) from which you are able to draw precise and definite conclusions. Do not be too ambitious and undertake an experiment so computationally intensive that you are unable to obtain results on the hardware available to you. Alternatively, you may write an essay on any aspect of the course content. Your essay topic should involve an in-depth critical evaluation of a specific machine learning technique and its application to language processing, or of a specific language processing task and machine learning techniques that have been applied to that task. Little credit will be given for summaries of papers. In both cases, your essay or report should not exceed 5000 words and will be due in around the end of the first week of Lent Term.

You should discuss and agree your essay topic or project with the lecturer. Write a proposal of up to 500 words outlining the topic or project giving a preliminary reading list and indicating what resources (datasets, hardware, and toolkits / packages) you plan to use, if relevant, and send it ASAP and no later than the 1st of November (need to have it agreed by then). Suitable projects will need to make use of existing labelled datasets and existing machine learning tools that are distributed and documented, so that they can be completed in reasonable time.

Schedule and Reading List

Seminars will be starting on the 20th October

20th of October: Classification

Papers:

1) Nguyen, NAACL 2018, Comparing Automatic and Human Evaluation of Local Explanations for Text Classification (presentation)

2) Levy et al., CoNLL 2017, Zero-Shot Relation Extraction via Reading Comprehension

25th of October: Natural Language Inference/Textual Entailment

Papers:

3) Bowman et al., EMNLP 2015, A large annotated corpus for learning natural language inference (presentation)

4) Tu et al., TACL 2020, An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models

27th of October: Combining old and new ideas

Papers:

5) Levy et al., TACL 2015, Improving Distributional Similarity with Lessons Learned from Word Embeddings

6) Khandelwal et al., ICLR 2021, Nearest Neighbor Machine Translation

17th of November: Sequence tagging

Papers:

7) Reimers and Gurevych, EMNLP 2017, Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging

8) De Cao et al., ICLR 2021, Autoregressive Entity Retrieval

22nd of November: Incremental structured prediction

9) Renzato et al., ICLR 2016, Sequence Level Training With Recurrent Neural Networks (presentation)

10) Kitaev et al., ACL 2022, Learned Incremental Representations for Parsing

24th of November: Examining our models

Papers:

11) Swayamdipta et al., EMNLP 2020, Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics

12) Bartolo et al., TACL 2020, Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension

29th of November: Inference in Sequence2Sequence

Papers:

13) Stahlberg and Byrne, EMNLP 2019, On NMT Search Errors and Model Errors: Cat Got Your Tongue? (presentation)

14) Eikema and Aziz, Coling 2020, Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation

1st of December: Large-scale language models

Papers:

15) Petroni et al., EMNLP 2019, Language Models as Knowledge Bases? (presentation)

16) Perez et al., NeurIPS 2021 True Few-Shot Learning with Language Models (presentation)