Simple Programming for Simple Distributed Systems

A lot of recent systems research has targeted peer-to-peer and grid applications. These have lots of hard (and sometimes dull) issues to deal with, including WAN latencies, privacy, security and free-loading concerns.

Luckily, simple distributed systems are coming back into fashion. Large-scale web services such as search have re-focused attention on datacenters where all machines are in the same administrative domain and are connected by a LAN with known topology. Similarly, as the number of cores per chip is predicted to increase exponentially for the next few processor architecture generations, traditional operating systems may be replaced by more loosely coupled systems which scale better, and share some characteristics with today's datacenters.

I will describe a programming model which is designed to allow developers to easily write code which is independent of the number of processing cores, and where the system automatically deals with the complexity of concurrency, scheduling, distribution and fault-tolerance. Instances of this programming model have been successfully deployed over the last few years both in large-scale datacenters (Google's MapReduce system) and many-core processors (GPUs). I will outline the Dryad project, in its infancy at Microsoft Research, in which we are generalizing the approach and investigating how suitable it is to diverse application domains.

Short bio

I completed my PhD in computer vision with Andrew Blake at Oxford University in 1998. Since then I have worked at Compaq SRC and Microsoft Research in Silicon Valley. Recently I have spent more time on systems than vision, and I devoted a large part of the last two years to helping to design (and writing shipping code for) the MSN Search engine which launched earlier this year.