Computer Laboratory

OCaml Labs

OCaml Platform

The OCaml Platform combines the core OCaml compiler with a coherent set of libraries, tools and documentation. The requirements and direction of the Platform are initially being guided by large users of OCaml such as Jane Street, Citrix and Lexifi.

The reason for initially taking direction from industrial users is because these companies have a great deal of experience of using the language at scale (e.g millions of lines of code, thousands of modules and complicated testing requirements). We take a pragmatic view that examining and solving the problems such users have will ultimately result in a large and positive impact on the wider community, through significantly improved tooling, libraries and documentation.

The current thrust of work is to improve the tooling around packaging, build and test infrastructure. Once complete, this will enable us to effectively run over hundreds of community packages and determine selection criteria for the Platform. A key aspect of this selection will be which libraries are already popular and in use, and also how actively maintained and portable they are across different operating systems.

The tasks below reflect this by:

  • The OPAM package manager acts as the package management front end.
  • The OPAM Doc tool replaces the venerable ocamldoc tool with modern, cross-referenced HTML output across several libraries.
  • The OCamlot testing infrastructure runs continuous integration of OPAM stable and candidate packages across a variety of operating systems and CPU architectures.

The mailing list for discussion about this is platform@lists.ocaml.org (archive).

Team

Tasks

Sep '12Oct '12Nov '12Dec '12Jan '13Feb '13Mar '13Apr '13May '13Jun '13Jul '13Aug '13Sep '13Oct '13Nov '13Dec '13Jan '14Feb '14Mar '14Apr '14
Thomas Gazagnaire OPAM 1.0 (if it builds, ship it)
Thomas Gazagnaire OPAM 1.1 (the CI and Test release)
Thomas Gazagnaire OPAM 1.2 (the Platform release)
Thomas Gazagnaire OPAM-in-a-box
Vincent Botbol OPAMDoc prototype
David Sheets OCamlot 0.2.0
David Sheets OCamlot 0.3.0
Anil Madhavapeddy Non-x86 Machine Pool
Amir Chaudhry Platform Preview
Jeremy Yallop Ctypes
Jeremy Yallop Ctypes stub generation
 

OPAM 1.0 (if it builds, ship it)

Complete by Thomas Gazagnaire (Mar 2012 - Mar 2013)
Github   Homepage   Developer Manual   Issues  

OPAM 1.0 is a source-based package manager for OCaml. It supports multiple simultaneous compiler installations, flexible package constraints, and a Git-friendly development workflow.

The goal of the first version of OPAM was to get something released that would provide basic package management facilities to the community, but also be designed with distributed open-source design in mind. To that end, OPAM 1.0 supports a remotes mechanism which lets you combine local development trees with other people's remote Git or Darcs repositories. Whenever opam update is run, all the remotes are refreshed and merged, letting you subscribe to other people's compilers and package trees.

This first version is build-system agnostic in the interests of compatibility with the large existing body of third party external source code, and so can't do much in the way of advanced manipulation of the packages.

OPAM 1.0 has been a great success. Since its release, there have been hundreds of external contributions from the community, and over 500 packages are now contained within the main package repository!

Related work


OPAM 1.1 (the CI and Test release)

Complete by Thomas Gazagnaire (Feb 2013 - Oct 2013)
Github   Homepage   Developer Manual   Issues  

Since the release of OPAM 1.0, we've steadily been fixing bugs that have been reported from the wider userbase. The release has actually been remarkably stable, and most of the issues are around the constraint solver (which tackles an NP-complete problem with cunning heuristics). There's also been quite a bit of work on improving portability and integration with the operating system via more interactive initialisation.

The big purpose behind the next release, though, is to improve support for large-scale continuous integration and testing of the packages contained within the repository. Jane Street, for example, is now issuing weekly releases of their Core standard library suite. Testing these manually across Linux, FreeBSD, MacOS X and several CPU architectures (x86, x86_64, ARM, Macppc) is both tedious and error-prone, and so we'd like to automate the process.

OPAM doesn't need many changes to support this testing, but there is more package metadata being added to facilitate the process, and tools such as oasis2opam help automate this by looking inside the packages themselves. The whole of OPAM is exported as a library so that third-party packages can interface with the OPAM repository without changing OPAM itself. This is exactly what the OCamlot project does.

Related work


OPAM 1.2 (the Platform release)

Planning by Thomas Gazagnaire (Nov 2013 - Mar 2014)
Github   Homepage   Developer Manual   Issues  

OPAM 1.2 will be the release that forms part of the OCaml Platform. We're still planning this feature-set in more detail, but here are some thoughts:

  • OPAM could become a compiler front-end by linking to the compiler-libs that are installed by the OCaml compiler. This lets OPAM automatically satisfy dependencies for packages and generate the boilerplate for testing and benchmarking code.

  • The existing support for switching between compiler versions is very powerful, and we plan to support "collections" of stable libraries that are tested more extensively than others. This will evolve into support for switching into a stable release of the Platform, to ensure that your code will interoperate with other users of the same Platform version.

This is all still being planned, so be sure to share your thoughts on the Platform mailing list.

Related work


OPAM-in-a-box

In Progress by Thomas Gazagnaire (Dec 2013 - Mar 2014)
Task ticket  

Related work


OPAMDoc prototype

Complete by Vincent Botbol (Jan 2013 - Nov 2013)
opam-doc   Snapshot Docs  

The OCaml toolchain has shipped with the ocamldoc tool for a long time. ocamldoc runs over a single OCaml library and generates cross-referenced documentation. It also supports a variety of outputs, such as Latex, HTML, PDF and even manpages. However, it is starting to show its age for large, complex codebases such as Core, and so we are developing a more scalable alternative for the Platform effort.

OPAM-doc consists of two separate commands:

  • bin-doc is a replacement for the OCamldoc lexer (which extracts documentation from source code comments. It uses the OCaml-4.00+ facility for generating .cmt files that contain the typed AST, and generates .cmd files which contain the documentation information. By using a separate file from the AST, we leave open the possibility of having multiple language translations in the future. These .cmd and .cmdi files can be combined with the .cmt files to generate complete documentation directly from the output of the compiler. This command is intended to be temporary, and can be integrated into the upstream in the future.

  • opam-doc takes a set of cmt and cmd files and outputs a single JSON representation of all the files. This JSON can then be post-processed (or directly rendered in Javascript) to create a single documentation repository that reliably renders cross-references across entire libraries. Thus, the entire Platform can have one documentation source rather than having to search across packages.

The ultimate aim is to support the OCaml platform with interactive tutorials using the js_of_ocaml compiler. You can try out the prototype of this in OPAM via opam install opam-doc && opam doc core async. It will start a local webserver on which you can browse the traffic. There is also a snapshot available on the Mirage documentation site.


OCamlot 0.2.0

Complete by David Sheets (Mar 2013 - Jun 2013)
Homepage   ocamlot-state   ocamlot   Issues   Travis and OPAM  

OCamlot (OCaml Online Testing), is a Continuous Integration and Testing system for packages provided via OPAM. It comprises all the appropriate tools and libraries to enable management and development of the machine infrastructure, for example an OCaml library to interface with Github (available through OPAM). If you submit a package via OPAM, you will be rewarded with regular regression tests across many diverse operating systems and build tool environments. Ultimately, we would also like to integrate benchmarking, constraint exploration, experimental design, complete isolation, and domain testing into this system.

The resourcing and commissioning of the Machine Pool itself is dealt with separately from this software which utilises it.

Current status: Minimal version 0.2.0 released

Related work


OCamlot 0.3.0

Complete by David Sheets (Jun 2013 - Sep 2013)
Homepage   ocamlot-state   ocamlot   Issues   Travis and OPAM  

This release of OCamlot will monitor GitHub and model packages and their attendant tasks more accurately.

Current status: In development

Related work


Non-x86 Machine Pool

Complete by Anil Madhavapeddy (Nov 2012 - Jun 2013)

In order to create a robust and stable Continuous Integration system, we need a pool of machines that reflect the diverse environments where OCaml code is deployed.

To this end, we've been assembling a small collection of conventional high-powered machines that can build the whole OPAM package set quite quickly, and also more unusual CPU architectures and operating systems that will help developers that don't have access to them. So far we have:

  • 12-core AMD64/Linux Debian 64-bit
  • 12-core AMD64/Linux Debian 32-bit
  • ARMv6l/Raspbian Linux 32-bit (Raspberry Pi 512MB)
  • ARMv5tel/Debian Linux 32-bit (Dreamplug)
  • sparc64/FreeBSD (thanks to Jasper Wallace for the donation)
  • macppc/OpenBSD (G4 Powerbook) (thanks to Dave Scott for the donation)
  • AMD64/MacOS X 10.8
  • iMac G5/Debian Linux 64-bit

Related work


Platform Preview

Complete by Amir Chaudhry (Mar 2013 - Jul 2013)
OCaml 2013 status   Platform mailing list  

The OCaml Platform is intended to combine the core OCaml compiler with a coherent set of libraries, tools, docs and other resources. The direction of the Platform will be guided by the major users of OCaml (including consortium members).

The first stage of creating the platform is to begin building OCamlot. This process will help us identify and and refine the initial libraries and tools that should form part of v0.1 of the platform proper. When OCamlot is complete, we'll be in a much better position to discuss what components should form v0.2 of the Platform.

The current state of the Platform components was also presented at OCaml 2013.

Related work


Ctypes

Complete by Jeremy Yallop (Mar 2013 - Jun 2013)
Github   Documentation  

ctypes is a library for binding to C libraries using pure OCaml. The primary aim is to make writing C extensions as straightforward as possible. The current FFI in OCaml is low-level and difficult to use correctly, so we're keen to improve the ease-of-interoperability in the first Platform release.

The core of ctypes is a set of combinators for describing the structure of C types -- numeric types, arrays, pointers, structs, unions and functions. You can use these combinators to describe the types of the functions that you want to call, then bind directly to those functions -- all without writing or generating any C!


Ctypes stub generation

In Progress by Jeremy Yallop (Sep 2013 - Mar 2014)
Github   Documentation   OCaml 2013 demo  

The first Ctypes release used dlopen to dynamically access C libraries, but this approach has some limitations that will be addressed by also generating C stub files that can be compiled using the system C compiler.

Related work