1. Microkernels are the way to go. For the system to work, the kernel has to be reliable. Putting the file system and other user services inside makes it harder to get right and less flexible. The performance loss is (guess) less than a factor of two. If you worry about factors of two, I humbly suggest you write all your code in assembler henceforth.
  2. Synchronous (blocking) communication is easier to use than asynchronous. Starting a send, then going off to do something, followed by getting an interrupt when the send is done is a nightmare to program. If you must do something while waiting for the reply, put that code in another (synchronous) thread. The biggest problem in the computer industry is not slow machines, but hairy, unreliable software. Using asynchronous communication just makes the problem worse.[#partridge##1#]
  3. RPC is a good idea to base your system on. SEND/RECEIVE primitives make I/O your central paradigm. RPC makes procedure call, an abstraction mechanism, your central paradigm. Abstraction is a key tool for mastering complexity. We need all the help we can get. [See chapter 2].
  4. Atomic group communication (broadcast or multicast) is highly useful. Not all communication is two party. For replicated servers and many other applications, being able to send a (synchronous) message to everyone and know they all get it in the same order without loss makes programming easier and programs more reliable. A real win. [See chapter 8].
  5. Distributed shared memory in one form or another is a convenient model. Even better than communicating synchronously is not communicating explicitly at all. Just let the programmer think there is shared memory, be it page-based, object-based, or other. Let the system do the work. There is an obvious analogy with virtual memory. This can be reduced to a single theme: making it easier to build and use distributed systems.