Deep Neural Networks

Objectives

You will gain detailed knowledge of

1. Current understanding of generalization in neural networks vs classical statistical models.
2. Optimization procedures for neural network models such as stochastic gradient descent and ADAM.
3. Automatic differentiation and at least one software framework (PyTorch, TensorFolow) as well as an overview of other software approaches.
4. Architectures that are deployed to deal with different data types such as images or sequences including a. convolutional networks b. recurrent networks and c. transformers.

In addition you will gain knowledge of more advanced topics reflecting recent research in machine learning. These change from year to year, examples include:

1. Approaches to unsupervised learning including autoencoders and self-supervised learning..
2. Techniques for deploying models in low data regimes such as transfer learning and meta-learning.
3. Techniques for propagating uncertainty such as Bayesian neural networks.
4. Reinforcement learning and its applications to robotics or games.
5. Graph Neural Networks and Geometric Deep Learning.

Teaching Style

The start of the course will focus on the latest undertanding of current theory of neural networks, contrasting with previous classical understandings of generalization performance. Then we will move to practical examples of network architectures and deployment. We will end with more advanced topics reflecting current research, mostly delivered by guest lecturers from the Department and beyond.

Schedule

Week 1

Two lectures: Generalization and Approximation with Neural Networks

Week 2

Two lectures: Optimization: Stochastic Gradient Descent and ADAM

Week 3

Two lectures: Hardware Acceleration and Convolutional Neural Networks

Week 4

Two lectures: Vanishing Gradients, Recurrent and Residual Networks, Sequence Modeling.

Week 5

Two lectures: Attention, The Transformer Architecture, Large Language Models

Week 6-8

Lectures and guest lectures from the special topics below.

Special topics

1. Self-Supervised Learning, AutoEncoders, Generative Modeling of Images
2. Hardware Implementations
3. Reinforcement learning
4. Transfer learning and meta-learning
5. Uncertainty and Bayesian Neural Networks
6. Graph Neural Networks

Assessment

Assessment involves solving a number of exercises in a jupyter notebook environment. Familiarity with the python programming language is assumed. The exercises rely on the pytorch deep learning framework, the syntax of which we don’t cover in detail in the lectures, students are referred to documentation and tutorials to learn this component of the assessment.

Assignment 1

30% of the total marks, with guided exercises.

Assignment 2

70% of the total marks, a combination of guided exercises and a more exploratory mini-project.