Information, Energy and Intelligence
Principal lecturer: Prof Neil Lawrence
Taken by: MPhil ACS, Part III
Code: L172
Term: Michaelmas
Hours: 16 (8 x 2hrs lectures)
Class limit: max. 10 students
Prerequisites: Students should have an undergraduate-level understanding of probability and statistics, including probability distributions, expectation, discrete and continuous random variables, and Bayes’ theorem. Knowledge of linear algebra, including matrix operations, eigenvalues and eigenvectors, is required. Familiarity with basic multivariate calculus, including partial derivatives and Lagrange multipliers, is helpful. No prior knowledge of physics or thermodynamics is assumed, as all thermodynamic concepts are developed from first principles during the module.
timetable
Aims
Develop the mathematical connections between thermodynamics, information theory, and Bayesian inference, and understand how they provide a toolkit for us to think about the foundations of `intelligent’ systems.
Syllabus
Entropy appears in three apparently separate traditions — thermodynamics (Boltzmann, Gibbs), information theory (Shannon), and Bayesian inference (Jaynes) — and turns out to be the same mathematical object viewed from different operational assumptions. The course covers: (1) the Boltzmann distribution, free energy, and the partition function as a generating function, (2) Shannon entropy and its formal equivalence to thermodynamic entropy; the exponential family as the MaxEnt family, (3) Maxwell’s demon and Landauer’s principle: the thermodynamic cost of decision-making, (4) information geometry: the Fisher metric, dually flat geometry, and natural gradient descent, (5) multi-information, an entropy game, quantum information, (6) further advanced topics such as probability transport, Schrödinger bridges, and information-theoretic limits on intelligent agency. Each lecture is followed by a formative self-study exercise exploring the session’s central concept from thermodynamic, information-theoretic, and Bayesian perspectives.
Objectives
- Shannon, C.E. (1948). “A Mathematical Theory of Communication.” *Bell System Technical Journal*, 27, 379–423.
- Jaynes, E.T. (1957). “Information Theory and Statistical Mechanics.” *Physical Review*, 106(4), 620–630.
- Landauer, R. (1961). “Irreversibility and Heat Generation in the Computing Process.” *IBM Journal of Research and Development*, 5(3), 183–191.
- Watanabe, S. (1960). “Information Theoretical Analysis of Multivariate Correlation.” *IBM Journal of Research and Development*, 4(1), 66–82.
- Amari, S. & Nagaoka, H. (2000). *Methods of Information Geometry*. AMS/Oxford University Press. *(Chapters 1–3)*
- Lawrence, N.D. (2025). “The Inaccessible Game.” *(Course notes; distributed via Moodle)* *Supporting textbooks:*
- Cover, T.M. & Thomas, J.A. (2006). *Elements of Information Theory* (2nd ed.). Wiley. *(Background reference for Weeks 1–4)*
- MacKay, D.J.C. (2003). *Information Theory, Inference, and Learning Algorithms*. Cambridge University Press. *(Freely available online; relevant chapters indicated per session)*
- Worksheets (60% total, 15% each): Each worksheet consists of a
short Python notebook and a written reflection (300–500 words).
- Worksheet 1: Thermodynamics and Shannon entropy | End of Week 2 | Boltzmann sampling; empirical vs analytic entropy; free energy decomposition | LO1, LO2, LO3
- Worksheet 2: Maxwell’s demon, MaxEnt, and the exponential family | End of Week 4 | MaxEnt with Lagrange multipliers; verify canonical ensemble and Gaussian; Landauer’s principle | LO4, LO5, LO6, LO7
- Worksheet 3: Information geometry | End of Week 6 | Fisher information matrix; one step of natural gradient descent vs vanilla gradient descent | LO8, LO9
- Worksheet 4: Multi-information, von Neumann entropy, and limits on intelligence | End of Week 8 | Multi-information; I + H = C conservation; Schrödinger bridge sketch; evaluate superintelligence claims | LO10, LO11, LO12, LO13
- In-class Moodle quizzes (40% total, 10% each):
- Each quiz is administered at the start of the relevant lecture (10 minutes, 8–10 MCQ).
Equip students to reason rigorously about entropy across thermodynamics, information theory, and Bayesian inference, and to evaluate claims about intelligent systems using information-theoretic constraints.
Recommended reading
Primary references (all covered in lectures):*
Assessment
The module is assessed as four take-home worksheets and four short in-class Moodle quizzes.