Computer Laboratory

Large-Scale Data Processing and Optimisation (2025-2026 Michaelmas Term)

LSDPO - R244

review_log

Open Source Projects

Reading Club papers

Contact

 

 

 

 

 

 

 

  

 

 Candidates for Open Source Project Study

The list is not exhaustive. If you wish to choose a project that is not on the list, please discuss it with me first.

Purpose of the assignment: The aim of this assignment is to gain a practical understanding of the proposed architectures, algorithms, and systems by running an actual prototype. You will present and explain to others how the prototype works, including its setup and execution, any additional work you have undertaken, and any applications or experiments you have developed yourself. This hands-on experience should help you develop a deeper understanding of the project and its underlying research ideas. Each open-source project is associated with one or more published research papers. You are expected to study the relevant papers and explore the proposed ideas through hands-on experimentation with the prototype. This should also help you identify which research directions interest you most.

Project selection and feasibility: Some projects are relatively large and may require substantial computing resources, complex software environments, or considerable setup time. Please carefully assess the feasibility of your chosen project and ensure that you can complete the assignment within the available time. Some projects provide general-purpose frameworks, while others focus on specific algorithms or optimisation techniques.



1.       Ciel: https://github.com/mrry/skywriting, https://www.cl.cam.ac.uk/netos/ciel/

2.       Naiad: data-parallel dataflow computation https://research.microsoft.com/en-us/projects/naiad/, and https://github.com/frankmcsherry/timely-dataflow (Rust version)

3.       Spark: Fast Cluter Computing https://spark-project.org/

4.       X-Stream: https://labos.epfl.ch/x-stream

5.       Storm: https://storm-project.net/

6.       GraphX: https://github.com/amplab/graphx

7.       Tensorflow: https://www.tensorflow.org/

8.       PyTorch: https://pytorch.org/

9.       CNTK: https://docs.microsoft.com/en-us/cognitive-toolkit/  https://github.com/Microsoft/CNTK

10.    Ray (+RLLib): https://github.com/ray-project/ray, https://github.com/ray-project/ray/tree/master/python/ray/rllib

11.    BoTorch: hhttps://github.com/pytorch/botorch

12.    Park: https://github.com/park-project/park (https://openreview.net/pdf?id=BkgfRbEPsE)

13.    Pyro: https://pyro.ai/

14.    Apache Flink: https://flink.apache.org/

15.    Jax: https://github.com/google/jax

16.    Numpyro - building pyro on jax: https://github.com/pyro-ppl/numpyro

17.    Emukit: https://emukit.github.io

18.    PGM: https://github.com/gvinciguerra/PGM-index

19.    ALEX: https://github.com/microsoft/ALEX

20.    OtterTune: https://github.com/cmu-db/ottertune

21.    NAS-MCTS (LA-MCTS): https://github.com/facebookresearch/LaMCTS

22.    World-Model: https://worldmodels.github.io/

23.    DRL-Pytorch: https://github.com/p-christ/Deep-Reinforcement-Learning-Algorithms-with-PyTorch

24.    Inverse RL-GAIL: https://github.com/hcnoh/gail-pytorch

25.    MARL: https://www.david-albert.fr/marl/html/index.html

26.    Meta-RL: https://github.com/rlworkgroup/metaworld

27.    SMAC3: https://github.com/automl/SMAC3

28.    EGG: https://egraphs-good.github.io/

29.    TVM: https://github.com/apache/tvm

30.    Ray-Tune: https://github.com/ray-project/ray/tree/master/python/ray/tune

31.    vLLM: https://github.com/vllm-project/vllm

32.    SGLang: https://github.com/sgl-project/sglang

33.    DistServ: https://github.com/LLMServe/DistServe/tree/main

34.    LMDeploy: https://github.com/InternLM/lmdeploy

35.    NVIDIA-DINAMO: https://github.com/ai-dynamo/dynamo

36.    Megatron-LM: https://github.com/NVIDIA/Megatron-LM

37.    Galvatron: https://github.com/PKU-DAIR/Hetu-Galvatron

38.    LLM as a judge https://en.wikipedia.org/wiki/LLM-as-a-Judge?wprov=sfla1

39.    Next-gen agent platform (google) https://antigravity.google

40.    vLLM: https://github.com/vllm-project/vllm

41.    SGLang: https://github.com/sgl-project/sglang

42.    DistServe: https://github.com/LLMServe/DistServe/tree/main

43.    LMDeploy: https://github.com/InternLM/lmdeploy

44.    NVIDIA-Dynamo: https://github.com/ai-dynamo/dynamo,

45.    Megatron-LM: https://github.com/NVIDIA/Megatron-LM

46.    LMCache: https://github.com/LMCache/LMCache

47.    llama.cpp: https://github.com/ggml-org/llama.cpp

48.    Hugging Face Transformers: https://github.com/huggingface/transformers

49.    TensorRT-LLM: https://github.com/NVIDIA/TensorRT-LLM

50.    MLX-LM: https://github.com/ml-explore/mlx-lm

51.    nano-vLLM: https://github.com/GeeeekExplorer/nano-vllm

52.    MLC-LLM: https://github.com/mlc-ai/mlc-llm

53.    LightLLM: https://github.com/ModelTC/lightllm

54.    Ollama: https://github.com/ollama/ollama