Large-scale data processing and optimisation

Aims

This module provides an introduction to large-scale data processing, optimisation, and the impact on computer system's architecture. Large-scale distributed applications with high volume data processing such as training of machine learning or LLM inference will grow ever more in importance. Supporting the design and implementation of robust, secure, and heterogeneous large-scale distributed systems is essential. To deal with distributed systems with a large and complex parameter space, tuning and optimising computer systems is becoming an important and complex task, which also deals with the characteristics of input data and algorithms used in the applications. Algorithm designers are often unaware of the constraints imposed by systems and the best way to consider these when designing algorithms with massive volume of data. On the other hand, computer systems often miss advances in algorithm design that can be used to cut down processing time and scale up systems in terms of the size of the problem they can address. Integrating machine learning approaches (e.g. Bayesian Optimisation, Reinforcement Learning) for system optimisation will be explored in this course.

Syllabus

This course provides perspectives on large-scale data processing, including data-flow programming, graph data processing, probabilistic programming and computer system optimisation, especially using machine learning approaches, thus providing a solid basis to work on the next generation of distributed systems.

The module consists of 8 sessions, with 5 sessions on specific aspects of large-scale data processing research. Each session discusses 3-4 papers, led by the assigned students. One session is a hands-on tutorial on dataflow programming fundamentals. The first session advises on how to read/review a paper together with a brief introduction on different perspectives in large-scale data
processing and optimisation. The last session is dedicated to the student presentation of opensource project studies.

1. Introduction to large-scale data processing and optimisation
2. Data flow programming: Map/Reduce to TensorFlow
3. Large-scale graph data processing: storage, processing model and parallel processing
4. Dataflow programming hands-on tutorial
5. Probabilistic Programming
6. Optimisations in ML Compiler
7. Optimisations of Computer systems using ML
8. Presentation of Open Source Project Study

Objectives

On completion of this module, students should:

• Understand key concepts of scalable data processing approaches in future computer systems.
• Obtain a clear understanding of building distributed systems using data centric programming and large-scale data processing.
• Understand a large and complex parameter space in computer system's optimisation and applicability of Machine Learning approach.

Coursework 1 (Reading Club)

Core of R244 is the reading club. Ideally the reading club will require you to read between 1 and 3 papers. You need to fill out simple review log (MS word format, text format) prior to each session and email me by a day before. The minimum requirement of review log is one per session, but you can read as many as you want and fill the review log for each paper you read. It is suggested to do the review log on one of the selected paper in the session, but it is not mandatory and you can pick any paper from the papers listed in that week’s session.

At each session, 4 - 5 papers are selected under the session topic, and if you are assigned to present your review work, please prepare 15 minutes slides for presenting your review work. For each presentation, another student is assigned to prepare the discussion points (with one slide) and the discussion together with whole room will follow the presentation (~8 minutes).  

Your presentation and discussion points materials should be emailed to me by the following day.  

You would present your review and discussion points approximately twice during the course. Note that take the following points when you create the slides.

1. Presentation

The goal of this presentation is explaining the core concept of the paper from the viewpoint of the focused session topic to the other students. It is important to make it compact cutting off unnecessary details.

• Highlight the main design/architecture of the paper focused in the session

• What is the significance from the existing works?

• What is the novel idea?

2. Discussion point

• What is the limitation of the paper?

• What is required to complete the work?

Coursework 2 (Reports)

The following two reports are required, which could be extended from the reading assignment of the reading club or a different one within the scope of R244.  

1. Survey report on sub-topic in data centric networking (~1800 - max 2000 words)  

• e.g. Pick up to 5 papers as core papers in your survey scope

• Read the above and expand your reading through related work

• Comprehend your view and finish as your survey paper

• See 'how to write a survey paper' guide in the course page.

2. Project study and exploration of a prototype (max 2500 words)  

• See Open Source Projects 

• What is the significance of the project in the research domain?

• Compare with the similar and succeeding projects

• Demonstrate the project by exploring its prototype

• Please email your project selection (MS word format or text format <150 words) by 16:00 on November XX, 2026.  

• Project presentation on December X, 2026.  

The report 1 by November XX, 2026 - 16:00 . The report 2 by January XX,  2027 - 16:00.

 Assessment

The final grade for the course will be provided as a letter grade or percentage, and the assessment will consist of two parts:

1. 35%: for reading club  

• 20%: Presentation

• 10%: Discussion point  

• 5%: Review log

2. 65%: for the two reports  

• 25%: Survey report

• 40%: Project study  

Recommended reading

1. M. Abadi et al. TensorFlow: A System for Large-Scale Machine Learning, OSDI, 2016.
2. D. Aken et al.: Automatic Database Management System Tuning Through Large-scale Machine Learning, SIGMOD, 2017.
3. J. Ansel et al. Opentuner: an extensible framework for program autotuning. PACT, 2014.
4. V. Dalibard, M. Schaarschmidt, E. Yoneki. BOAT: Building Auto-Tuners with Structured Bayesian Optimization, WWW, 2017.
5. J. Dean et al. Large scale distributed deep networks. NIPS, 2012.
6. G. Malewicz, M. Austern, A. Bik, J. Dehnert, I. Horn, N. Leiser, G. Czajkowski. Pregel: A System for Large-Scale Graph Processing, SIGMOD, 2010.
7. A. Mirhoseini et al. Device Placement Optimization with Reinforcement Learning, ICML, 2017.
8. D. Murray, F. McSherry, R. Isaacs, M. Isard, P. Barham, M. Abadi. Naiad: A Timely Dataflow System, SOSP, 2013.
9. M. Schaarschmidt, S. Mika, K. Fricke and E. Yoneki: RLgraph: Modular Computation Graphs for Deep Reinforcement Learning, SysML, 2019.
10. Z. Jia, O. Padon, J. Thomas, T. Warszawski, M. Zaharia,  A. Aiken: TASO: Optimizing Deep Learning Computation with Automated Generation of Graph Substitutions: SOSP, 2019.
11. H. Mao et al.: Park: An Open Platform for Learning-Augmented Computer Systems, OpenReview, 2019.
12. J. Shao, et al.: Tensor Program Optimization with Probabilistic Programs, NeurIPS, 2022.

A complete list can be found on the course material web page. See also 2023-2024 course material on the previous course Large-Scale Data Processing and Optimisation.