Microkernels are the way to go.
For the system to work, the kernel has to be
reliable. Putting the file system and other
user services inside makes it harder to get
right and less flexible. The performance loss
is (guess) less than a factor of two. If you
worry about factors of two, I humbly suggest you
write all your code in assembler henceforth.
Synchronous (blocking) communication is easier
to use than asynchronous.
Starting a send, then going off to do
something, followed by getting an interrupt when
the send is done is a nightmare to program. If
you must do something while waiting for the
reply, put that code in another (synchronous)
thread. The biggest problem in the computer
industry is not slow machines, but hairy,
unreliable software. Using asynchronous
communication just makes the problem worse.[#partridge##1#]
RPC is a good idea to base your system on.
SEND/RECEIVE primitives make I/O your central
paradigm. RPC makes procedure call, an
abstraction mechanism, your central paradigm.
Abstraction is a key tool for mastering
complexity. We need all the help we can get. [See chapter 2].
Atomic group communication (broadcast or
multicast) is highly useful.
Not all communication is two party. For
replicated servers and many other applications,
being able to send a (synchronous) message to
everyone and know they all get it in the same
order without loss makes programming easier and
programs more reliable. A real win. [See chapter 8].
Distributed shared memory in one form or another
is a convenient model.
Even better than communicating synchronously is
not communicating explicitly at all. Just let
the programmer think there is shared memory, be
it page-based, object-based, or other. Let the
system do the work. There is an obvious analogy
with virtual memory.
This can be reduced to a single theme: making it
easier to build and use distributed systems.