Irmin (irmin.Irmin)

val version : string

The version of the library.

Preliminaries

module Type : sig ... end

Dynamic types for Irmin values.

module Info : sig ... end

Commit info are used to keep track of the origin of write operations in the stores. Info models the metadata associated with commit objects in Git.

module Merge : sig ... end

Merge provides functions to build custom 3-way merge operators for various user-defined contents.

module Diff : sig ... end

Differences between values.

type 'a diff = 'a Diff.t

The type for representing differences betwen values.

Low-level Stores

module type CONTENT_ADDRESSABLE_STORE = sig ... end

Content-addressable backend store.

module type APPEND_ONLY_STORE = sig ... end

Append-only backend store.

module type ATOMIC_WRITE_STORE = sig ... end

Atomic-write stores.

User-Defined Contents

module Path : sig ... end

Store paths.

module Hash : sig ... end

Hashing functions.

module Metadata : sig ... end

Metadata defines metadata that is attached to contents but stored in nodes. The Git backend uses this to indicate the type of file (normal, executable or symlink).

module Contents : sig ... end

Contents specifies how user-defined contents need to be serializable and mergeable.

module Branch : sig ... end

User-defined branches.

type remote = ..

The type for remote stores.

type config

The type for backend-specific configuration values.

Every backend has different configuration options, which are kept abstract to the user.

module Private : sig ... end

Private defines functions only useful for creating new backends. If you are just using the library (and not developing a new backend), you should not use this module.

High-level Stores

exception Closed

The exception raised when any operation is attempted on a closed store, except for S.close, which is idempotent.

module type S = sig ... end

Irmin stores.

module Json_tree : functor (Store : S with type contents = Contents.json) -> sig ... end

Json_tree is used to project JSON values onto trees. Instead of the entire object being stored under one key, it is split across several keys starting at the specified root key.

module type S_MAKER = functor (M : Metadata.S) -> functor (C : Contents.S) -> functor (P : Path.S) -> functor (B : Branch.S) -> functor (H : Hash.S) -> S with type key = P.t and type step = P.step and type metadata = M.t and type contents = C.t and type branch = B.t and type hash = H.t and type Private.Sync.endpoint = unit

S_MAKER is the signature exposed by any backend providing S implementations. M is the implementation of user-defined metadata, C is the one for user-defined contents, B is the implementation for branches and H is the implementation for object (blobs, trees, commits) hashes. It does not use any native synchronization primitives.

module type KV = S with type key = string list and type step = string and type branch = string

KV is similar to S but chooses sensible implementations for path and branch.

module type KV_MAKER = functor (C : Contents.S) -> KV with type contents = C.t

KV_MAKER is like S_MAKER but where everything except the contents is replaced by sensible default implementations.

Synchronization

val remote_store : (module S with type t = 'a) -> 'a -> remote

remote_store t is the remote corresponding to the local store t. Synchronization is done by importing and exporting store slices, so this is usually much slower than native synchronization using Store.remote but it works for all backends.

module type SYNC = sig ... end

SYNC provides functions to synchronize an Irmin store with local and remote Irmin stores.

module Sync : functor (S : S) -> SYNC with type db = S.t and type commit = S.commit

The default Sync implementation.

Examples

Syncing with a remote

A simple synchronization example, using the Git backend and the Sync helpers. The code clones a fresh repository if the repository does not exist locally, otherwise it performs a fetch: in this case, only the missing contents are downloaded.

open Lwt.Infix
module S = Irmin_unix.Git.FS.KV (Irmin.Contents.String)
module Sync = Irmin.Sync (S)

let config = Irmin_git.config "/tmp/test"

let upstream =
  if Array.length Sys.argv = 2 then
    Uri.of_string (Store.remote Sys.argv.(1))
  else (
    Printf.eprintf "Usage: sync [uri]\n%!";
    exit 1 )

let test () =
  S.Repo.v config >>= S.master >>= fun t ->
  Sync.pull_exn t upstream `Set >>= fun () ->
  S.get t [ "README.md" ] >|= fun r -> Printf.printf "%s\n%!" r

let () = Lwt_main.run (test ())

Mergeable logs

We will demonstrate the use of custom merge operators by defining mergeable debug log files. We first define a log entry as a pair of a timestamp and a message, using the combinator exposed by Irmin.Type:

module Entry : sig
  include Irmin.Type.S

  val v : string -> t

  val timestamp : t -> int
end = struct
  type t = { timestamp : int; message : string }

  let compare x y = compare x.timestamp y.timestamp

  let time = ref 0

  let v message =
    incr time;
    { timestamp = !time; message }

  let timestamp t = t.timestamp

  let pp ppf { timestamp; message } =
    Fmt.pf ppf "%04d: %s" timestamp message

  let of_string str =
    match String.split_on_char '\t' str with
    | [] -> Error (`Msg ("invalid entry: " ^ str))
    | ts :: msg_sects -> (
        let message = String.concat "\t" msg_sects in
        try Ok { timestamp = int_of_string ts; message }
        with Failure e -> Error (`Msg e) )

  let t =
    let open Irmin.Type in
    record "entry" (fun t32 message ->
        { timestamp = Int32.to_int t32; message })
    |+ field "timestamp" int32 (fun t -> Int32.of_int t.timestamp)
    |+ field "message" string (fun t -> t.message)
    |> sealr

  let t = Irmin.Type.like ~cli:(pp, of_string) ~compare t
end

A log file is a list of entries (one per line), ordered by decreasing order of timestamps. The 3-way merge operator for log files concatenates and sorts the new entries and prepend them to the common ancestor's ones.

(* A log file *)
module Log : sig
  include Irmin.Contents.S

  val add : t -> Entry.t -> t

  val empty : t
end = struct
  type t = Entry.t list

  let empty = []

  let pp ppf l = List.iter (Fmt.pf ppf "%a\n" Entry.pp) (List.rev l)

  let of_string str =
    let lines = String.cuts ~empty:false ~sep:"\n" str in
    try
      List.fold_left
        (fun acc l ->
          match Entry.of_string l with
          | Ok x -> x :: acc
          | Error (`Msg e) -> failwith e)
        [] lines
      |> fun l -> Ok l
    with Failure e -> Error (`Msg e)

  let t = Irmin.Type.(list Entry.t)

  let t = Irmin.Type.like' ~cli:(pp, of_string) t

  let timestamp = function [] -> 0 | e :: _ -> Entry.timestamp e

  let newer_than timestamp file =
    let rec aux acc = function
      | [] -> List.rev acc
      | h :: _ when Entry.timestamp h <= timestamp -> List.rev acc
      | h :: t -> aux (h :: acc) t
    in
    aux [] file

  let merge ~old t1 t2 =
    let open Irmin.Merge.Infix in
    old () >>=* fun old ->
    let old = match old with None -> [] | Some o -> o in
    let ts = timestamp old in
    let t1 = newer_than ts t1 in
    let t2 = newer_than ts t2 in
    let t3 = List.sort Entry.compare (List.rev_append t1 t2) in
    Irmin.Merge.ok (List.rev_append t3 old)

  let merge = Irmin.Merge.(option (v t merge))

  let add t e = e :: t
end

Note: The serialisation primitives used in that example are not very efficient in this case as they parse the file every time. For real usage, you would write buffered versions of Log.pp and Log.of_string.

To persist the log file on disk, we need to choose a backend. We show here how to use the on-disk Git backend on Unix.

(* Build an Irmin store containing log files. *)
module S = Irmin_unix.Git.FS.KV (Log)

(* Set-up the local configuration of the Git repository. *)
let config = Irmin_git.config ~bare:true "/tmp/irmin/test"

(* Set-up the commit info function *)
let info fmt = Irmin_unix.info ~author:"logger" fmt

We can now define a toy example to use our mergeable log files.

open Lwt.Infix

(* Name of the log file. *)
let file = [ "local"; "debug" ]

(* Read the entire log file. *)
let read_file t = S.find t file >|= function None -> [] | Some l -> l

(* Persist a new entry in the log. *)
let log t fmt =
  Fmt.kstrf
    (fun message ->
      read_file t >>= fun logs ->
      let logs = Log.add logs (Entry.v message) in
      S.set t (info "Adding a new entry") file logs)
    fmt

let () =
  Lwt_main.run
    ( S.Repo.v config >>= S.master >>= fun t ->
      log t "Adding a new log entry" >>= fun () ->
      Irmin.clone_force ~src:t ~dst:"x" >>= fun x ->
      log x "Adding new stuff to x" >>= fun () ->
      log x "Adding more stuff to x" >>= fun () ->
      log x "More. Stuff. To x." >>= fun () ->
      log t "I can add stuff on t also" >>= fun () ->
      log t "Yes. On t!" >>= fun () ->
      S.merge (info "Merging x into t") x ~into:t >|= function
      | Ok () -> ()
      | Error _ -> failwith "merge conflict!" )

Helpers

module Dot : functor (S : S) -> Dot.S with type db = S.t

Dot provides functions to export a store to the Graphviz `dot` format.

Backends

module type APPEND_ONLY_STORE_MAKER = functor (K : Type.S) -> functor (V : Type.S) -> sig ... end

APPEND_ONLY_STORE_MAKER is the signature exposed by append-only store backends. K is the implementation of keys and V is the implementation of values.

module type CONTENT_ADDRESSABLE_STORE_MAKER = functor (K : Hash.S) -> functor (V : Type.S) -> sig ... end

CONTENT_ADDRESSABLE_STOREMAKER is the signature exposed by content-addressable store backends. K is the implementation of keys and V is the implementation of values.

module Content_addressable : functor (S : APPEND_ONLY_STORE_MAKER) -> functor (K : Hash.S) -> functor (V : Type.S) -> sig ... end

module type ATOMIC_WRITE_STORE_MAKER = functor (K : Type.S) -> functor (V : Type.S) -> sig ... end

ATOMIC_WRITE_STORE_MAKER is the signature exposed by atomic-write store backends. K is the implementation of keys and V is the implementation of values.

module Make : functor (CA : CONTENT_ADDRESSABLE_STORE_MAKER) -> functor (AW : ATOMIC_WRITE_STORE_MAKER) -> S_MAKER

Simple store creator. Use the same type of all of the internal keys and store all the values in the same store.

module Make_ext : functor (CA : CONTENT_ADDRESSABLE_STORE_MAKER) -> functor (AW : ATOMIC_WRITE_STORE_MAKER) -> functor (Metadata : Metadata.S) -> functor (Contents : Contents.S) -> functor (Path : Path.S) -> functor (Branch : Branch.S) -> functor (Hash : Hash.S) -> functor (Node : Private.Node.S with type metadata = Metadata.t and type hash = Hash.t and type step = Path.step) -> functor (Commit : Private.Commit.S with type hash = Hash.t) -> S with type key = Path.t and type contents = Contents.t and type branch = Branch.t and type hash = Hash.t and type step = Path.step and type metadata = Metadata.t and type Key.step = Path.step and type Private.Sync.endpoint = unit

module Of_private : functor (P : Private.S) -> S with type key = P.Node.Path.t and type contents = P.Contents.value and type branch = P.Branch.key and type hash = P.Hash.t and type step = P.Node.Path.step and type metadata = P.Node.Val.metadata and type Key.step = P.Node.Path.step and type repo = P.Repo.t and type slice = P.Slice.t and module Private = P

Advanced store creator.