Classically, knowledge of memory within a Unix process is implicit. The compiled program embodies knowledge of the memory it uses for its own purposes, but there are few ways to introspect on that, and none from within the process itself.
Debuggers do know how to do this to some extent, albeit from outside not inside the process, and only when given the right metadata (e.g. after compiling C code with the -g flag). However, even debuggers cannot provide a typed view of the memory on the end of a pointer. At best, they can tell you something about that memory assuming some correct static type for the pointer. For example, if gdb is told that a pointer in the program is a pointer to char, it will happily print characters when you dereference it. It will do that even if it is incorrect, and the pointer does not really point to characters. It does not know any “ground truth” about the allocation that actually resides at that location.
liballocs is a system which changes this. It adds an expressive meta-level to Unix processes. You can perform queries on arbitrary pointers, benefit from whole-process run-time type information, observe the allocation structure in your program, and more. It consists of a run-time library and some toolchain extensions. Currently the latter mostly work for C, though other languages are fair game with appropriate retrofitting.
Whereas classical Unix knows only that memory consists of “bytes”, with liballocs memory consists of typed allocations. This seems to be general enough to model any source language (or rather, model any sane implementation thereof).
Read more in the Onward! 2015 paper: here. Also there is a diagram of one application of liballocs (libcrunch, doing run-time type-checking) which may give you a better idea of how the toolchain extension works.
The easiest way to use liballocs is to compile a C program with allocscc, which invokes the host toolchain's C compiler and linker but with additional pre- and post-processing. In particular it postprocesses the compiler-generated debugging information into an efficient run-time representation. Once built, the binary may be run with or without liballocs loaded; if without, there is usually close to zero slowdown, and even with, the compulsory slowdown is often negligible. Of course, use of the meta-level services is not without cost, but I try hard to keep them efficient; the cost depends on the allocator involved, what metadata it provides and how that is (or can be separately) structured.
$ allocscc -o myprog ... # + other front-ends $ ./myprog # metalevel stubbed out $ LD_PRELOAD=liballocs.so ./myprog # metalevel enabled
As a more developed example, here is how to build and run a simple liballocs client program in Docker.
$ git clone https://github.com/stephenrkell/liballocs.git
$ docker build -t liballocs_built liballocs/buildtest/debian-stretch
$ docker run --rm -i -t --name liballocs_test liballocs_built bash
$ export PATH=/usr/local/src/liballocs/tools/lang/c/bin:$PATH
$ cat >test.c <<EOF
#include <allocs.h>
#include <stdio.h>
int main(int argc, char **argv)
{
void *p = malloc(42 * sizeof(int));
void *ptrs[] = { main, p, &p, argv, NULL };
for (void **x = &ptrs[0]; *x; ++x)
{
printf("At %p is a %s-allocated object of size %u, type %s\n",
*x,
alloc_get_allocator(*x)->name,
(unsigned) alloc_get_size(*x),
UNIQTYPE_NAME(alloc_get_type(*x))
);
}
return 0;
}
EOF
$ allocscc -I/usr/local/src/liballocs/include -o test test.c
$ LD_PRELOAD=/usr/local/src/liballocs/lib/liballocs_preload.so ./test
You should see something like the following. This is just a simple demo of how liballocs knows what is in memory, having precise dynamic information and an awareness of allocators (there are four different allocators visible in this example).
At 0x55c36036c6c0 is a static-allocated object of size 0, type __FUN_FROM___ARG0_int$32__ARG1___PTR___PTR_signed_char$8__FUN_TO_int$32
At 0x55c36036c6c0 is a generic malloc-allocated object of size 176, type __ARR0_int$32
At 0x55c36036c6c0 is a stackframe-allocated object of size 128, type _test_cil_c_main_vaddrs_0x1910_0x1a01
At 0x55c36036c6c0 is a auxv-allocated object of size 16, type __ARR2___PTR_signed_char$8
Note that currently, liballocs is only for x86-64 GNU/Linux systems (although ports are welcome and very feasible). For code and more detailed build instructions, please see the GitHub page.
liballocs is intended as a platform for building advanced run-time services, in ways that are backward-compatible with existing code that knows nothing about liballocs. Broadly in the spirit of Smalltalk-style dynamism, it is envisaged to provide a foundation for features such as:
(*) means some work exists already on this use case. (+) means a little groundwork is done.
Unix abstractions are fairly simple and fairly general, but they are not humane, and they invite fragmentation. By 'not humane', I mean that they are error-prone and difficult to experiment with interactively. By 'fragmentation', I mean they invite building higher-level abstractions in mutually opaque and incompatible ways (think language VMs, file formats, middlewares...). By enabling the above applications, liballocs aspires to counter all these.
What's novel? Although the run-time facilities of liballocs are (I contend) richer than what has existed before in any Unix-like system, you might counter that many of the above goals have apparently been achieved, at least as far as proof-of-concept, by earlier research or development prototypes. This has been through heroic efforts of many people... but evidently these efforts have not “stuck” in the sense of becoming part of the fabric of a commodity distribution. When this phenomenon repeats itself, it becomes a research problem to itself—not simply a matter of tech transfer or follow-through. Many of the resulting prototypes lack features required for real-world use -- awareness of custom memory allocators is a common lack -- and generally they are realised in mutually incompatible ways, for want of the right abstractions.
To borrow Greenspun's tenth rule, this is because each of these earlier prototypes contains an ad-hoc, bug-ridden and slow implementation of a small fraction of liballocs. The goal of liballocs is to offer a flexible, pluralist structure for growing these features on—in a way that transparently adds thse capabilities to existing codebases, rather than requiring up-front “buy-in”. It's not a framework; you don't usually write code against its API, or port your code to it. Instead, it extends the fabric which already builds and runs your code. The research programme around liballocs is working towards demonstrating the practicality of this approach, by building instances of several of the above systems/services, at modest effort and capable of co-existence.
One idea behind liballocs is to adopt some of the same design heuristics to which Unix owes its success: minimalism, flexibility and pluralism. Instead of a defining a single “virtual machine” from the top down, it permits many possible realisations of the same or similar abstractions. Unix's free-form byte-oriented facilities allow many higher-level semantic constructs to coexist (programming languages, structured data, network protocols and so on). Unlike Unix, liballocs also tries fairly hard to recognise and reconcile these duplicates after the fact. That requires a metasystem that is descriptive rather than prescriptive. By reconciling abstract commonality across the many concretely different ways in which memory can be managed, structured and interpreted, it can offer a platform for higher-level services which can operate correctly across many different such schemes—as defined by various ABIs, language runtimes, libraries, coding styles or conventions.
Please do send me e-mail if you have questions or comments.
This work has been supported by EPSRC Programme Grant EP/K008528/1, REMS: Rigorous Engineering of Mainstream Systems.
A preliminary version was created with funding from the Oxford Martin School Institute for the Future of Computing.
Content updated at Thu 14 Oct 15:17:00 BST 2021.
validate this page