A mergeable replicated data type (MRDT) implementation for a data type \tau is a tuple D_{\tau } = (\Sigma , \sigma _{0}, do, merge) where:

• \Sigma is the set of all possible states at a branch,

• \sigma _{0} \in \Sigma is the initial state,

• do : Op_{\tau } \times \Sigma \times Timestamp \rightarrow \Sigma \times Val_{\tau } implements every data type operation,

• merge : \Sigma \times \Sigma \times \Sigma \rightarrow \Sigma implements the three-way merge strategy.

Definition 2.1 from "Certified Mergeable Replicated Data Types".

• Orthogonal MRDT operations — where it is possible to compute a single operation that lead to a particular state.

You can install the binary, msh, today:

opam pin git+https://git.sirref.org/merry
eval $(opam env)
msh

There are a few avenues to try next. In the short-term, more testing of the shell is probably worthwhile. In particular, trying to boot an Alpine Linux image with msh as the default shell would be good.

Integrating the various components of Shelter is also on the list. A time-travelling POSIX shell is within reach; the actual semantics still need to be ironed out.

I know for certain that some shell options and built-ins will be the first issues to arise going forward.

As a fun experiment, Anil's vibecoded OCaml busybox translation could be plumbed into msh. This would allow a very versatile, but zero-forking shell! Throw in an in-memory Eio filesystem into the mix and the whole thing becomes in-memory... mirageSH.

Below are a collection of weeknotes for Merry.

__ready=
trap __ready=1 HUP
kill -HUP "$MAIN_PID"
until ["$__ready" ]; do __dummy=; done

$ ./shellbench -s bash,zsh,msh sample/func.sh
-----------------------------------------------------------------------------------
name                                                     bash        zsh        msh
-----------------------------------------------------------------------------------
func.sh: no func                                      775,539  1,124,485      5,229
func.sh: func                                         398,861    197,169     93,424
-----------------------------------------------------------------------------------
* count: number of executions per second

f () { :; }
while __count=$(($__count+1)) && ["$__count" -le 10000 ]; do
	f
done
echo "count is $__count"

$ time -p dash test.sh
count is 10001
real 0.01
user 0.01
sys 0.00
$ time -p msh test.sh
count is 10001
real 15.53
user 4.28
sys 12.28

$ time -p dash test.sh
count is 10001
real 7.55
user 3.95
sys 3.44

apt-get install debootstrap -y
debootstrap stable /stable-chroot http://deb.debian.org/debian/

$ count () { echo "Count: $#"; }; FOO="hello world"; count $FOO
Count: 2

$ IFS=" "; FOO=" hello    world"; echo $FOO
hello world
$ IFS=" "; FOO=" hello    world"; echo "$FOO"
 hello    world
$ IFS=":"; FOO="a:b:c:"; echo $FOO
a b c
$ IFS=":"; FOO="a:b::c:d"; echo $FOO
a b  c d
$ IFS=":"; FOO=":a:b:c"; echo $FOO
 a b c
$ IFS=; FOO="unset   IFS"; echo $FOO
unset   IFS

iter () {
 echo "a"
 echo "b"
 echo "c"
}

base=$(iter)

for n in $base; do
  echo "Got $n"
done

for n in "$base"; do
  echo "Got $n"
done

$ echo "!dlrow olleh" > input.txt
$ (read line; echo $line) < input.txt | rev
hello world!

for m in $MIRRORS; do
    pkgdest="$TARGET/$($DLDEST pkg "$s" "$c" "$a" "$m" "$path")"
    if [! -e "$pkgdest" ]; then continue; fi
    pkgs_to_get="$(download_debs "$m" "$pkgdest" $pkgs_to_get 5>&1 1>&6)"
    if [ -z"$pkgs_to_get" ]; then break; fi
done 6>&1

 let () =   let port = ref 8080 in
   Arg.parse
     [ ("-p", Arg.Set_int port, " Listening port number(8080 by default)") ]     ignore "An HTTP/1.1 server";
   Eio_main.run @@ fun env ->
+  Eio_mem.run @@ fun menv ->
   Eio.Switch.run @@ fun sw ->
-  (* Restrict to current directory: *)
-  let htdocs = Eio.Stdenv.cwd env in
+  (* Restrict to in-memory current directory: *)
+  Eio_mem.Fs.load ~filter (Eio.Stdenv.cwd env);
+  let htdocs = Eio.Stdenv.cwd menv in
   let socket =
     Eio.Net.listen env#net ~sw ~backlog:128 ~reuse_addr:true
       (`Tcp (Eio.Net.Ipaddr.V4.loopback, !port))

Using meio to debug Merry [weekly-2026-w23]

• June 4, 2026
• Patrick Ferris

During a large rewrite of Merry, I introduced a regression causing certain shell programs to hang indefinitely. I thought it might be a good opportunity to try out Meio.

First, let's look at a trace of Meio from some real-world programs. For example: meio -- msh -c "sleep 100".

One problem that comes up immediately from this example is that the tree hierarchy of tasks and cancellation contexts makes it tricky to reason about the logging. Here, we can see the pipeline's cancellation context clearly labelled pipeline-id1. This is an Eio.Switch.t for the pipeline sleep 100. The actual execution log for sleep 100 bubbles back up to the main fiber (that has no name, so it is called <task>). The daemon fiber, for ensuring the spawned process is reaped, is attached to the switch and is displayed below it.

Meio has a long way to go. Most programs are too fast to be monitored in this way. Using something like tokio-console's model might be a better idea here focusing less on the tree strucuture of the program, and more on the performance and monitoring of individual tasks. Without this, it turns out Meio was not all that useful for debugging this specific issue (which ended up being some bad pipe management on my part).

$ dash -c "cat /dev/urandom | hxd.xxd -l100"
00000000: 5033 7981 ae44 d475 074c 5361 0a9e 3828  P3y..D.u.LSa..8(
00000010: f88c 1f9e 3b0e bcde 528a f78c b1e4 7195  ....;...R.....q.
00000020: 16f2 8325 c90b 9310 539c af04 8289 c3a0  ...%....S.......
00000030: 57c5 680a 2c81 a999 71b6 708e 95df e720  W.h.,...q.p....
00000040: 004c 407a 8365 9448 65c6 b435 bed2 4084  .L@z.e.He..5..@.
00000050: 62ea 96ab 2b23 0385 f9d2 fc30 d090 1719  b...+#.....0....
00000060: 68d1 c514                                h...
cat: write error: Broken pipe

while echo "some bytes"; do
  :
done | hxd.xxd -l100

info () {
  echo "INFO: $1" >&3
}

exec 3>&1 > /dev/null
info "Starting up server..."
echo "Some noisy program spamming STDOUT"
info "All is quiet..."

exec < ./hello.txt

while read line; do
  echo "Got $line" >&2
done | (exec < ./world.txt; cat)

% ./test/check_fork.sh "while echo 5; do : ; done | true"
bash  3
dash  3
yash  1 
zsh   1
ksh   1
ash   3

while echo "hello"; do
  :
done | head -n 3

# 50.123;;
# 1 "file.ml";;

err () {
  printf "err $1" >&4
}

exec 4>&1 

A POSIX Shell in OCaml [weekly-2026-w12]

• March 19, 2026
• Patrick Ferris

Long time, no weekly. Since the start of this year I have been building a POSIX shell in OCaml called msh (with the underlying library called Merry). Merry is available online now.

A POSIX(ish) shell in OCaml

• March 19, 2026
• Patrick Ferris

Merry is a POSIX(ish) in OCaml. It uses Eio alongside Morbig (a static parser for POSIX shell).

Why another (POSIX) shell?

• March 19, 2026
• Patrick Ferris

Shells have been around for a long time. In my research on the notion of metashell I wrote about Louis Pouzin originally coining the term:

Metashells › Louis Pouzin's "SHELL" [pouzin-shell]

• May 19, 2025
• Patrick Ferris

I spent some time reading part of the multics design documentation this week. Louis Pouzin coined the term "SHELL" in this document, and I was reminded yet again just how important it is to be a good writer even as a "computer science researcher". For example, this excerpt from the requirements section of the document

The previous definitions imply that a command MUST be designed while keeping in mind the user, sitting at his console, wondering about what might be going on, mistyping or forgetting arguments, even if fully aware of the conventions, and possibly interfering with the command by hasty quits, carriage returns, and other temperamental reactions.

And then later, when defining the "SHELL".

We may envision a common procedure called automatically by the supervisor whenever a user types in some message at his console, at a time when he has no other process in active execution under console control (presently called command level). This procedure acts as an interface between console messages and subroutine. The purpose of such a procedure is to create a medium of exchange into which one could activate any procedure, if it were called inside of another program. Hereafter, for simplification, we shall refer to that procedure as the "SHELL".

It still surprises how little the undergraduate degree in computer science at Cambridge focuses on writing skills.

I built Shelter exploring the idea of a shell-like interface that allowed users to undo their shell actions (amongst other cool tricks). Unfortunately shell-like is not enough. Shelter cut many corners to masquerade as a shell (e.g. appending env to understand how a command may have altered the execution environment). I felt it was necessary to make Shelter a "SHELL"! To do that, I needed a solid foundation to build on.

msh, the POSIX shell that comes with Merry, is by no means POSIX-complete in terms of features. But it is my daily driver at this point. It includes a pure OCaml rewrite of linenoise (a small, self-contained alternative to the venerable readline) called bruit.

If you have docker installed on your machine, you can take it for a spin today:

docker run -it --rm patrickferris/msh

The patrickferris/msh docker image is just for trying it out. It is based on the OCaml 5.3 alpine image.

Alternatively, you can build msh from source and have it available in your opam switch.

opam pin git+https://tangled.org/patrick.sirref.org/merry

There are many small paper cuts left to patch over, but most of it is porcelain (e.g. ctrl+left-arrow for moving in bruit). Unfortunately, these are the kinds of things that you will immediately stumble upon.

What makes Merry different?

• March 19, 2026
• Patrick Ferris

Nothing.

Merry is supposed to be a solid, POSIX-ish base to build on. Unfortunaltely, as it turns out, the subset of features from the POSIX specification that people actually use... is pretty much all of it. Every possible redirection, variable expansion, shell built-in and compound command make some appearance. Not to mention the non-POSIX bits of shell we all take for granted (e.g. &>-redirection).

Merry and msh are useable. You will most likely find bugs if you use them. If, when using msh, you find something obscure happening you can enable debug mode either my setting the variable MSH_DEBUG or by invoking msh with -v -v.

What's next?

• March 19, 2026
• Patrick Ferris

It is soon time to combine Shelter and Merry into the time-travelling, POSIX-ish shell that I have been trying to build since I first started working on Shelter.

For this to be successful, I need Merry to be able to pass plenty of tests and right now that involves trying to install plenty of packages using tools like apk and apt.

Shell MRDT

• March 19, 2026
• Patrick Ferris

What I am particularly interested in reasoning about, is the execution context in Merry. This value, alongside the file-system, constitutes a fairly deep understanding of the state that changes in each step of a shell's evaluation loop.

This was, in terms of Shelter, the missing piece for truly building some kind of MRDT across shell sessions.

TIFF in OCaml

• March 19, 2026
• Patrick Ferris

I picked up from the excellent Outreachy work of Tambe Salome in getting write-support in ocaml-tiff. We are getting closer to the kind of API I envisaged in this latest round of refinement and review.

let ( / ) = Eio.Path.( / )

let checkerboard ~size =
  let v = Nx.zeros Nx.uint8 [| size; size |] in
  for row = 0 to size - 1 do
    for col = 0 to size - 1 do
      if (row + col) mod 2 = 0 then Nx.set_item [ row; col ] 254 v
    done
  done;
  Nx.to_bigarray v

let () =
  Eio_posix.run @@ fun env ->
  Tiff_eio.with_open_out (env#cwd / "example.tiff") @@ fun w ->
  let tif = Tiff.make (checkerboard ~size:256) in
  Tiff.to_file tif w

I have also been extremely pleased to see further external collaborators appearing:

• Nicolas helping out with metadata maintainence.

• Virgile adding support for reading multi-image TIFF files.

The most prevalent kind of abstraction is polymorphism. For example, List.length will work for all lists regardless of what kind of elements the list contains. However, polymorphism requires you to keep the polymorphic part of the type at arm's length. This can lead to suboptimal performance and/or limited usefulness.

List.mem is a good example; in order to be polymorphic List.mem relies on Stdlib.(=), the polymorphic equality operator. In turn, this operator achieves its polymorphism by performing a structural comparison of the runtime representation of its arguments.

A common way to overcome this is to use higher-order functions (functions that take functions as arguments).

# let rec mem ?(eq=Stdlib.(=)) v = function
  | [] -> false
  | x :: xs -> eq v x || mem ~eq v xs
val mem : ?eq:('a -> 'a -> bool) -> 'a -> 'a list -> bool = <fun>
# let mem_int_list = mem ~eq:Int.equal
val mem_int_list : int -> int list -> bool = <fun>

This works well in plenty of cases (List.map, Hashtbl.iter, etc.), however, it doesn't scale well from a user's perspective if the body of the function requires lots of specific functions (e.g. compare, length, pp). At the same time, we want to avoid the precarious duck-typing of something like Python too.

Functors are perhaps the second most prevalent feature of OCaml when it comes to code reuse. In the OCaml sense, a functor is a module that is parameterised by other modules. Set.Make is a functor that requires a module adhereing to the Set.OrderedType interface.

module IntSet = Set.Make (Int)
module StringSet = Set.Make (String)

Functors make it a lot easier to write implementations that depend on a large variety of functionality. Set.OrderedType is relatively tame in this sense, only needing a type t and a compare : t -> t -> int function.

Irmin, on the other hand, has functors that require users to provide Schemas. I have more thoughts about this API design in my Irmin retrospective.

It is important to consider the runtime costs of functors too. Whilst most of the time functors are instantiated at the toplevel, it does not come for free. You can have a play around with js_of_ocamlopt and see the generated output for how functors are applied.

Sitting somewhere inbetween polymorphism and functors are first-class modules with existentials. This is a rather advanced use of OCaml's type system to be able to pack together a value of a particular type alongside a module that works with those kinds of values.

type set = Set : ((module Set.S with type t = 'a) * 'a) -> set
let size (Set ((module S), s)) = S.cardinal s

Packaging up integer sets is fairly straight-forward.

# let int_empty_set = Set ((module IntSet), IntSet.empty)
val int_empty_set : set = Set (<module>, <poly>)

From a user's perspective we are now closer to the realm of dynamic typing. Indeed, existentials are the way to do that in OCaml.

# type any = Any : 'a -> any
type any = Any : 'a -> any
# let any v = Any v
val any : 'a -> any = <fun>

Getting back into a typed world often involves some type witness gymnastics or runtime exceptions. Used sparingly, however, it can be very effective particularly when a user is none the wiser about the use of FCMs and existentials. I have made use of this trick to package together Irmin stores with implementations.

You could not, however, simply return the value that was packed away inside the existential.

# let woops (Set (_, s)) = s
Line 1, characters 26-27:
Error: The value s has type $a but an expression was expected of type 'a
       The type constructor $a would escape its scope
       Hint: $a is an existential type bound by the constructor Set.

There is also a runtime cost to the packing (and unpacking) of the module, not to mention the need for the extra allocation to create the existential in the first place. This will likely not be too significant, but is worth considering.

OCaml 5.5 has module-dependent function types also known as modular explicits. The type of a function can now depend on types that come from a module that is also part of the function definition.

# let set_length : (module S : Set.S) -> S.t -> int =
  fun (module S) v -> S.cardinal v;;
val set_length : (module S : Set.S) -> S.t -> int = <fun>
# set_length (module IntSet) IntSet.empty;;
- : int = 0
# set_length (module StringSet) StringSet.empty;;
- : int = 0

In fact, we can now return values of a particular type that depends on a particular module!

# let empty_set : (module S : Set.S) -> S.t = 
  fun (module S) -> S.empty;;
val empty_set : (module S : Set.S) -> S.t = <fun>
# empty_set (module IntSet);;
- : IntSet.t = <abstr>
# empty_set (module StringSet);;
- : StringSet.t = <abstr>

Or, we can dynamically select implementations and values.

# let with_impl : 
    config:string -> ((module S : Set.S) -> S.t -> 'a) -> 'a = 
  fun ~config fn ->
  match String.lowercase_ascii config with
  | "string" -> fn (module StringSet) StringSet.empty
  | "int" -> fn (module IntSet) IntSet.empty
  | c -> invalid_arg c
val with_impl : config:string -> ((module S : Set.S) -> S.t -> 'a) -> 'a =
  <fun>

This is like a more usable version of the FCM-existential trick, though does require more manual type annotations on functions. For instance, I tried switching out OBuilder's use of existentials with modular explicits:

-type builder = Builder : (module Obuilder.BUILDER with type t = 'a) * 'a -> builder
-
 let log tag msg =
   match tag with
   | `Heading -> Fmt.pr "%a@." Fmt.(styled (`Fg (`Hi `Blue)) string) msg
   | `Note -> Fmt.pr "%a@." Fmt.(styled (`Fg `Yellow) string) msg
   | `Output -> output_string stdout msg; flush stdout

-let create_builder store_spec conf =
+let create_builder :
+  Store_spec.store Lwt.t -> Native_sandbox.config ->
+    ((module B : Obuilder.BUILDER) -> B.t -> 'a) -> 'a Lwt.t =
+  fun store_spec conf fn ->
   store_spec >>= fun (Store_spec.Store ((module Store), store)) ->
   let module Builder = Obuilder.Builder (Store) (Native_sandbox) (Docker_extract) in
   Native_sandbox.create ~state_dir:(Store.state_dir store / "sandbox") conf >|= fun sandbox ->
   let builder = Builder.v ~store ~sandbox in
-  Builder ((module Builder), builder)
+  fn (module Builder) builder

I think the code is more pragmatic; there is no longer an indirection via a type called builder. The extra type annotations do add a little noise. Roll on modular implicits.

Below are just some of the pieces of random hacking I got up to whilst in Singapore.

  module Heap = struct
    type ptr = int32#
    (* ... *)
  end

  type cqe = {
    user_data_id : Heap.ptr;
    res : int32#;
  }

I was a part of, often in a very small way, some other talks that were presented at ICFP including

• Functional Networking for Docker, I encourage you to read Anil's notes on the topic. It is also worth mentioning Anil's post-posix talk (not unrelated to my Uring musings above).

• Michael gave an excellent, very live, talk about Yirgacheffe at PROPL which I thoroughly enjoyed. I'm also excited about his recent developments which I'm sure you can clue into over on his blog.

Below is a non-exhaustive list of talks I enjoyed whilst at the conference. Some relevant, most of them not!

• By far, one of the best talks was Polynomial-time Program Equivalence for Machine Knitting. I think the talk really nailed the "here's the idea, the details are in the paper"-presenting style as they described an "...algorithm that canonicalizes the algebraic representations of the topological semantics of machine knitting programs".

• I attended most of the OxCaml tutorial presented by the incredibly friendly Gavin Gray.

• Ryan's talk Spatial Programming for Environmental Monitoring at PROPL was excellent.

There's plenty more to talk about, and I'm sure my future posts will reference other parts of my experience at ICFP 2025. Thanks for reading, and if anybody is interested in any aspects of this do reach out!

Irmin, at its simplest, is a key-value database. Users associate keys with values and can query and update these bindings.

Additionally, this database supports versioning. This means independent snapshots of the database can coexist and users can switch between them and update them without fear of interfering with other versions.

Different versions of the database can be combined by merging. When you set up your instance of an Irmin database, you also provide it with a merge function.

# #show_type Irmin.Merge.f;;
type nonrec 'a f =
    old:'a Irmin.Merge.promise ->
    'a -> 'a -> ('a, Irmin.Merge.conflict) result Lwt.t

Irmin's API is very git-inspired. There is a large overlap of shared vocabulary and concepts: repositories, branches, commits, heads etc.

Probably the most confusing aspect of this is the notion of a Store. When I was describing Irmin above, I used the term database to help distinguish between some of these concepts. In Irmin's documentation, it is used for multiple related (but different) concepts. In "Creating a Store" stores refer to the entire database, whereas in the Irmin API docs we have that:

There are two kinds of store in Irmin: the ones based on persistent named branches and the ones based temporary detached heads.

For Irmin library users, a Store.t can be thought of as a checkout of the database at a particular revision. This may be from a branch or a specific commit.

# Store.of_branch;;
- : Store.repo -> string -> Store.t Lwt.t = <fun>
# Store.of_commit;;
- : Store.commit -> Store.t Lwt.t = <fun>

The overloading of the term is confusing. I think it leaves users unsure about what other people might mean when they say "store". Being careful with these terms, and the contexts in which they are used, would help avoid this confusion.

# #show_module Irmin_mem.KV;;
module KV :
  sig
    type endpoint = unit
    type metadata = unit
    type hash
    type info = Store.info
    type ('h, _)contents_key = 'h
    type 'h node_key = 'h
    type 'h commit_key = 'h
    module Make : (C : Irmin__.Contents.S) ->sig ... end
  end

It will probably come as no surprise that one of the main limitations of using Irmin is the lack of documentation and tutorials. This is a particular pain point as the API is not very straightforward.

A while back, I started an Irmin book intending to help document the kinds of things real-world users would need in order to use Irmin in earnest. For example: ways to deal with type migrations or primers on runtime types. I still believe this sort of work would be invaluable, but there is a large cost to completing it, and it is not clear if there is enough interest in the project to warrant such an effort.

The tutorials on the irmin website are still good starting points for most new users. But they quickly lack the depth for real-world scenarios.

What follows are additional ideas for improving Irmin.

There were two main things I hoped to continue working on: ppxlib and outreachy. These are projects that I was previously working on, and in the case of Outreachy I have now been involved for many years.

In addition to this, all of my Part II projects this year used OCaml in some regard. In general, I want to see more adoption of OCaml. Over the years this has taken many forms including my initial work on developing workflows for OCaml that just turned five years old. This directly fed into the rebranding and rethinking of ocaml.org itself.

So, I worked on most of what I wanted to work on and then some! Going forward I hope to keep maintaining ppxlib in some capacity and coordinating the OCaml community's Outreachy efforts.

I hope to continue my small experiments (e.g. converting Eio to Picos). My own research makes heavy use of Irmin, and I would be interested to help with that too. More recently, a new library has been released for working with numerical data in OCaml called Raven: I am interested to use this library in Geocaml, a suite of geospatial tools written in OCaml that I maintain.

I feel conflicted about the OxCaml efforts. I admit, I am not fully aware of the full benefits of the features, but I worry about some proliferation of modes that make the type system in OCaml unbearable to use. On top of this, with my ppxlib hat on, I worry about the impact of changing the compiler so frequently, placing strain on an already small community. That being said, by releasing OxCaml separately I do believe running it as an experimental set of packages will help understand the tool better. But I would not be building anything I intend to maintain or research with right now as that ecosystem is far too volatile.

Finally, over the past two years I have done a lot of teaching. From Part II projects to supervisions. Going into next (academic) year, I intend to reduce my in-person teaching to focus on my research. However, I am interested in producing more materials for learning, maybe some of this in OCaml, perhaps something similar to Beautiful Racket. I have tried in the past to do these sorts of things, for example The Irmin Book.

Thank you for reading this wrap up! And thank you again to Tarides for letting me work so freely on things that I think are good for the OCaml community.

opentrace has an all command that will trace the entire system.

$ sudo opentrace all --flags=O_WRONLY
pid,cgid,comm,kind,flags,mode,filename,return
16324,4417,".nheko-wrapped",openat,577,438,"/home/patrick/.cache/nheko/nheko/curl_alt_svc_cache.txt",47
16324,4417,".nheko-wrapped",openat,193,33188,"/home/patrick/.cache/nheko/nheko/cr3ZUHqBErIOe3PlwJ1SuU8zFKKxL12VzrRoHYMH.tmp",47
16324,4417,".nheko-wrapped",openat,577,438,"/home/patrick/.cache/nheko/nheko/curl_alt_svc_cache.txt",47
16324,4417,".nheko-wrapped",openat,193,33188,"/home/patrick/.cache/nheko/nheko/QfZTdZSuC56NdcDQ3aMxJc3BhMhAj8PmtYW1zFDP.tmp",47
2530,4235,"systemd",openat,524865,438,"/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/gammastep.service/cgroup.subtree_control",41
2530,4235,"systemd",openat,524545,0,"/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/gammastep.service/memory.min",41
2530,4235,"systemd",openat,524545,0,"/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/gammastep.service/memory.low",41
2530,4235,"systemd",openat,524545,0,"/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/gammastep.service/memory.high",41
2530,4235,"systemd",openat,524545,0,"/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/gammastep.service/memory.max",41

The --flags=O_WRONLY argument filters the events where the O_WRONLY flag was set in the call to open.

We also get the name of the current executable linked to the task (comm). The -wrapped is an artefact of using Nix.

The primary use case for this tool is to inspect what files your program might be reading and writing.

$ sudo opentrace exec --format=json --flags=O_CREAT -- opam list
$ cat trace.json | jq ".[] | .fname"
"/home/patrick/.opam/cshell/.opam-switch/packages/cache"
"/home/patrick/.opam/log/log-118747-29da3d.out"
"/home/patrick/.opam/log/log-118747-29da3d.err"
"/home/patrick/.opam/log/log-118747-29da3d.env"
"/home/patrick/.opam/log/log-118747-29da3d.info"

The "flags" argument can specify a small boolean formula for checking the open flags of a particular event with | (or), & (and), and ~ (not). Parentheses can be used for precedence.

$ sudo opentrace exec --flags="O_WRONLY|O_RDONLY" -- ocaml --version

One feature opentrace needs (in this proof-of-concept phase) is the ability to also trace subprocesses.

opentrace is primarily an eBPF program that is loaded into the kernel and communicates with an OCaml program. Events are communicated via a ring buffer and most of the post-processing happens in OCaml. To capture subprocesses, opentrace creates a new control group (cgroup) and places the new process into that group. This gives opentrace a new identifier to track, namely the cgroup.

So consider the following program.

let () =
  Eio_posix.run @@ fun env ->
  Eio.Path.(save ~create:(`Or_truncate 0o664) (env#fs / "hello.txt") "hello");
  Eio.Process.run env#process_mgr
    [ "/bin/bash"; "-c"; "echo 'heya' > heya.txt" ]

It first creates a file using direct calls to functions like openat. Then it spawns a process which creates a new file called heya.txt. This happens in a separate process. However, with the --cgroups flag we can capture both interactions with the operating system.

$ sudo opentrace exec --cgroups --flags="O_CREAT|O_TRUNC" ./main.exe
pid,cgid,comm,kind,flags,mode,filename,return
153187,530807,"main.exe",openat,526914,436,"hello.txt",5
153192,530807,"bash",openat,577,438,"heya.txt",3

Whilst testing opentrace against some of the tools I use nearly daily, I noticed some events were being missed. I tried tracing forester, and only the initial read of forest.toml was logged. It dawned on me that the reason for this was that forester (via eio) was using io_uring to perform most of the IO. Most attempts to open files were bypassing the open system calls, and instead they were being performed by the kernel after reading a submission request for an openat2-style call!

This is not news to seasoned, Linux systems programmers. Io_uring bypasses SECCOMP filters for exactly the same reasons.

$ sudo opentrace exec -- forester build
$ cat trace.csv
pid,cgid,comm,kind,flags,mode,filename,return
155007,535570,"forester",openat,524288,0,"forest.toml",5
155007,535570,"forester",Uring,2621440,0,"",0
155021,535570,"cp",openat,131072,0,"/home/patrick/documents/forest/theme/favicon-32x32.png",4
155007,535570,"forester",Uring,2686976,0,"",0
155007,535570,"iou-wrk-155007",Uring,557634,420,"",0
155007,535570,"iou-wrk-155007",Uring,557634,420,"",0
155007,535570,"iou-wrk-155007",Uring,557634,420,"",0

It is interesting to note two things here:

1. We can tell that forester reads the configuration file probably using something like In_channel in OCaml (it does).

2. It appears that Uring is performing IO in both worker threads and directly.

The file paths are empty at the moment as I cannot find a clean way to trace openat submission requests into Uring without it sometimes going very wrong. I have tried quite a few methods (e.g. tracing do_filp_open) and at the moment I am tracing io_openat2, but this seems quite brittle, and often the filename is completely garbled, so I do not set it. If anyone has any ideas to trace Io_uring more reliably, I am all ears!

Hazel is a functional programming language with typed holes. Holes are pieces of your program that have not yet been filled in. Holes can appear anywhere in your program both as expression or types. Hazel can still evaluate your program in the presence of holes.

To get a flavour of Hazel, take a regular map function for lists.

let map = fun f -> fun xs -> case xs
  | [] => []
  | x :: xs => f (x) :: map(f)(xs) 
end in
map(fun x -> ?)([1, 2, 3])

The question mark (?) is a hole. The program evaluates to the following expression of type [?] (for people more familiar with OCaml types ? list).

[ ?, ?, ? ]

Hazel supports local type inference but nothing involving unification variables. For example, a simple add_one function in Hazel (fun x -> x + 1) has type ? -> Int.

The ability to transpile OCaml programs to Hazel programs is motivated by one simple thought: there are more OCaml programs than there are Hazel programs. This could help bootstrap projects by alleviating the need to rewrite boilerplate code (e.g. URI parsing or standard library functions for strings).

type expr =
  | Float of float
  | Add of expr * expr
  | Sub of expr * expr
  | Mul of expr * expr
  | Div of expr * expr

type expr =
  + Float(Float)
  + Add((expr, expr))
  + Sub((expr, expr))
  + Mul((expr, expr))
  + Div((expr, expr))
 in ?

let rec map f = function
  | [] -> []
  | x :: xs -> f x :: map f xs

let map = fun f -> fun x1 -> case x1
  | [] => []
  | x :: xs => f(x) :: map(f)(xs)
end in ?

val map : 
  ('a -> 'b) -> 'a list -> 'b list

val map : ∀ a b. (a ->  b) -> a list -> b list

let map : forall a -> forall b -> (a -> b) -> [a] -> [b] =
  typfun a -> typfun b -> fun f -> fun xs -> case xs
    | [] => []
    | x :: xs => f (x) :: map@<a>@<b>(f)(xs) 
end in
map@<Int>@<Int>(fun x -> ?)([1, 2, 3])

let rec length = function
 | [] -> 0
 | _ :: xs -> 1 + length xs

let int_len = length [ 1; 2; 3 ]
let str_len = length [ "only"; "two" ]

let length : forall a -> [a] -> Int = typfun a -> fun x1 -> case x1
  | [] => 0
  | _ :: xs => 1 + length@<a>(xs)
end in
let int_len : Int = length@<Int>(1 :: 2 :: [3]) in
let str_len : Int = length@<String>("only" :: ["two"])
in ?

The impetus for this work was to derive a corpus of ill-typed Hazel programs. Luckily, such a corpus exists for OCaml! Seidel et al. created a corpus of OCaml programs from their undergraduate students at UC San Diego. Some of these programs have been transpiled to Hazel.

Hazel is a fun, research programming language. Potential third-year students may find it interesting to take this work further. For example, how would this look in terms of a module system? From a purely engineering perspective, plenty of work would be needed to convert a multi-library OCaml project to Hazel (e.g. handling the cmi files).

Another line of research would be to have Hazel target one of the intermediate representations in OCaml which would give Hazel a fully functioning compiler to "native" code?

It is possible to perform migrations between parsetrees manually using the ppxlib.ast package.

open Ppxlib
open Ast_builder.Make(struct let loc = Location.none end)
let pp_expr = Format.printf "%a%!" Pp_ast.(expression ?config:None)

let loc = Location.none
let str txt = { txt; loc }

module To_408 = 
  Ppxlib_ast.Convert (Ppxlib_ast.Js) (Ppxlib_ast__.Versions.OCaml_408)

The Ppxlib_ast.Js module is ppxlib's AST (since ppxlib.0.36.0 OCaml 5.2.0). We can use functions provided by the Convert functor to copy parts of the AST from one version to another. Usually there is very little difference between the versioned parsetrees.

let func arg body =
  pexp_fun Nolabel None (ppat_var (str arg)) body

let add_expr = func "x" (func "y" (eapply (evar "Int.add") [ evar "x"; evar "y" ]))

The above excerpt of OCaml code produces an expression in the language. More accurately, a function that has two arguments. If we print the simplified AST we can see that the arguments are stored together in a list.

# pp_expr add_expr;; 
Pexp_function
  ( [ { pparam_loc = __loc
      ; pparam_desc = Pparam_val ( Nolabel, None, Ppat_var "x")
      }
    ; { pparam_loc = __loc
      ; pparam_desc = Pparam_val ( Nolabel, None, Ppat_var "y")
      }
    ]
  , None
  , Pfunction_body
      (Pexp_apply
         ( Pexp_ident (Ldot ( Lident "Int", "add"))
         , [ ( Nolabel, Pexp_ident (Lident "x"))
           ; ( Nolabel, Pexp_ident (Lident "y"))
           ]
         ))
  )
- : unit = ()

This is a relatively new feature of the parsetree, if we migrate the expression down to 4.08 we see it is represented differently. The output is a little too verbose to be readable, and there's no good way to show the AST, so you will just have to take my word.

let expr_408 = To_408.copy_expression add_expr

OCaml 5.2 introduced a new way to represent functions. Before 5.2, all functions were represented with single arity. Functions with multiple arguments would return functions as their body. For example:

let add x y = x + y

A two-argument add function would be represented in the parsetree as fun x -> fun y -> x + y. Using the nodes from the parsetree (with some details removed) it would be Pexp_fun (x, Pexp_fun (y, Pexp_apply ...)).

Now, since OCaml 5.2, functions can be expressed with their syntactic arity intact. The function add would look something like Pexp_function ([ x; y ], Pfunction_body (Pexp_apply ...)). Both arguments are stored as a list.

The constructor Pexp_function was not a new addition to the parsetree. It used to represent pattern-matching expressions like function A -> 1. This can now be expressed as a function body of Pfunction_cases.

Whilst these additions and modifications to the parsetree are arguably a better design, it has caused chaos in terms of ppxlib's reverse dependencies!

In ppxlib 0.36.0, functions from Ast_builder will help ppx authors produce maximum arity functions. However, functions from Ast_helper will not do this.

let ast_helper_func arg body =
  Ast_helper.Exp.fun_ Nolabel None (ppat_var (str arg)) body

let ast_helper_expr = ast_helper_func "x" (ast_helper_func "y" (eapply (evar "Int.equal") [ evar "x"; evar "y" ]))

Here we recreate our helper function to create function expressions using Ast_helper.Exp.fun_ instead. We can now show the difference between the expr expression and the ast_helper_expr expression.

# pp_expr add_expr;;
Pexp_function
  ( [ { pparam_loc = __loc
      ; pparam_desc = Pparam_val ( Nolabel, None, Ppat_var "x")
      }
    ; { pparam_loc = __loc
      ; pparam_desc = Pparam_val ( Nolabel, None, Ppat_var "y")
      }
    ]
  , None
  , Pfunction_body
      (Pexp_apply
         ( Pexp_ident (Ldot ( Lident "Int", "add"))
         , [ ( Nolabel, Pexp_ident (Lident "x"))
           ; ( Nolabel, Pexp_ident (Lident "y"))
           ]
         ))
  )
- : unit = ()

Above, we see a function expression with arity two. Below, we see nested function expressions each with arity one.

# pp_expr ast_helper_expr;;
Pexp_function
  ( [ { pparam_loc = __loc
      ; pparam_desc = Pparam_val ( Nolabel, None, Ppat_var "x")
      }
    ]
  , None
  , Pfunction_body
      (Pexp_function
         ( [ { pparam_loc = __loc
             ; pparam_desc = Pparam_val ( Nolabel, None, Ppat_var "y")
             }
           ]
         , None
         , Pfunction_body
             (Pexp_apply
                ( Pexp_ident (Ldot ( Lident "Int", "equal"))
                , [ ( Nolabel, Pexp_ident (Lident "x"))
                  ; ( Nolabel, Pexp_ident (Lident "y"))
                  ]
                ))
         ))
  )
- : unit = ()

Many, many, many ppxes broke after migrating the ppxlib AST to 5.2. Mostly, this is due to pattern-matching. Downstream users of ppxlib frequently pattern-match directly against the parsetree. However, nodes such as Pexp_fun no longer exist and Pexp_function has completely different constructor arguments!

Patching all the ppxes has been quite difficult. Most ppxes only deal with single arguments at a time (e.g. Pexp_fun x) but now they must handle zero or more arguments at a time. (In the zero case, this will always correspond to a pattern-matching function).

Here is a non-exhaustive list of some of the ppxes that have been patched. It may useful for other ppx authors who wish to upgrade to ppxlib.0.36.0:

• https://github.com/patricoferris/ppx_deriving_yaml/pull/58

• https://github.com/mirage/ocaml-rpc/pull/181

• https://gitlab.com/o-labs/ppx_deriving_jsoo/-/merge_requests/1

• https://github.com/ocaml-community/sedlex/pull/160

• https://github.com/ocaml-sys/config.ml/pull/22

• https://github.com/vogler/ppx_distr_guards/pull/2

• https://github.com/ocaml-gospel/gospel/pull/424

• https://github.com/teamwalnut/graphql-ppx/pull/299

• https://github.com/ocsigen/lwt/pull/1033

• https://gitlab.com/gopiandcode/ppx-inline-alcotest/-/merge_requests/3

• https://github.com/dariusf/ppx_interact/pull/3

• https://github.com/ocaml-ppx/ppx_deriving_yojson/pull/160

• https://github.com/LPCIC/elpi/pull/276

• https://github.com/mirage/repr/pull/110

• https://github.com/ProgramingIsTheFuture/ppx_default/pull/3

• https://github.com/codinuum/vlt/pull/3

• https://github.com/sim642/ppx_deriving_hash/pull/6

• https://github.com/ocsigen/tyxml/pull/340

• https://github.com/sim642/ppx_viewpattern/pull/2

• https://github.com/little-arhat/ppx_jsobject_conv/pull/11

• https://github.com/robur-coop/lun/pull/1

• https://gitlab.com/o-labs/ppx_deriving_encoding/-/merge_requests/4

• https://github.com/francoisthire/bam/pull/12

• https://github.com/xguerin/bitstring/pull/36

This release has emphasised the tension between using pattern-matching on algebraic datatypes and wanting to reduce downstream breakages (e.g. using abstract datatypes). This tension is not new.

Ppxlib is not going anywhere anytime soon. We have committed to trying to keep the internal AST up to date with the latest OCaml release. In fact, there is already a PR for the 5.3 bump!

There have been many discussions about simplifying this process, but so far the time and effort to build a ppxlib that will result in fewer breakages is too high (see an attempt using views).

Thank you to Tarides and Jane Street for funding my time to work on this release and all the chaos it has caused.

Happy preprocessing!

Dune, for quite some time, has supported virtual libraries. This allows library authors to define a signature without an implementation (similar to OCaml's module type S = struct ... end). The package is something other dune libraries can build against, still without choosing a particular implementation of the interface. An application author can make the decision about which implementation they would like to build into their program.

This feature has been quite widely used in OCaml. For example, the excellent digestif library of hashing algorithms provides both C and OCaml implementations individually packaged into digestif.c and digestif.ocaml respectively. Consider developing a browser application that needs some hashing algorithm (e.g. Irmin in the browser). We can explicitly use the OCaml backend for digestif which js_of_ocaml will happily compile for us.

There are two distinct limitations to this approach.

1. Applications can only link exactly one implementation.

2. Implementations cannot expose additional functionality easily (see dune#5597).

MirageOS is a framework for building unikernels in OCaml. It relies heavily on OCaml's functors to abstract implementation from interface. The mirage tool will then select appropriate instantiations of these functors depending on the backend that is targeted (e.g. unix, hvt etc.)

Recently, there has been a push to use dune virtual libraries instead.

We need to be able to overcome the limitation that implementations cannot expose additional functionality very easily using dune virtual libraries. As it stands, we need to make Mirage's virtual libraries unwrapped (i.e. set (wrapped false)). This allows implementers to expose additional modules alongside the implementation of the virtual library.

For VPNkit, we need this to port some of the DNS forwarding tests which use a fake clock and sleep module. Previously, they could be provided directly as functor arguments to the various DNS modules e.g.

let module R =
  Dns_forward.Resolver.Make(Proto_client)(Fake.Time)(Fake.Clock)

In this new world of dune virtual libraries, our Resolver is significantly more simple.

let module R = Dns_forward.Resolver.Make(Proto_client)

The flexibility of time and clock implementation has been pushed to a build-time decision in a dune file. This means that in order to provide the same fake clock and time we will need to create a private dune library that (a) implements the respective signatures of the virtual libraries and (b) provides the extended interface to allow the test suite to modify the internals of the implementations.

The first thing I had to do was make the mirage-mtime and mirage-sleep virtual libraries unwrapped. This allowed me to make a new private dune library called fake_time with two public modules: the normal interface required by mirage-mtime and another module called Fake_time_state containing:

let timeofday = ref 0L
let c = Lwt_condition.create ()

let advance nsecs =
  timeofday := Int64.add !timeofday nsecs;
  Lwt_condition.broadcast c ()

let reset () =
  timeofday := 0L;
  Lwt_condition.broadcast c ()

I did the same for fake_sleep which depended on the state from the fake_time library (something a normal library wouldn't).

Once I had the implementations for the virtual libraries in place, all that remained was to tell the executable (in this case the test suite) to select these new implementations instead of the defaults. Whereas before, all the tests could live in the same test suite, now they had to be split into those that needed the fake implementation and those that did not. This is another slight limitation of this virtual library approach. You can only link one implementation at a time to an executable (or test). Thus, the dune rules for running the dns_forward test suite became:

; Uses the default  implementations provided by virtual libraries
(executable
 (name test)
 (libraries dns_forward alcotest))

; Override the defaults and select the private fake libraries
(executable
  (name test_fake)
  (libraries dns_forward alcotest fake_sleep fake_time))

With all of this machinery in place, I could now update test_fake to use the new modules that would provide the custom functionality for the clock and time implementations.

(* The bad server should be marked offline and no-one will wait for it *)
Fake_time_state.reset ();
(* avoid the timeouts winning the race with the actual result *)
Fake_time_state.advance Duration.(of_ms 500); 

The switch to dune virtual libraries seems to be a good one. A majority of applications only require a single clock or time implementation. However, it has become harder to inject custom implementations of various resources into your MirageOS-based applications.

The pull request to upgrade VPNKit is open.

In the Energy and Environment Group we are working on various geospatial tools, one of which is written in OCaml and needed a geospatial index to speedup spatial queries over a dataset. Searching Github revealed an implementation in pure OCaml. After reaching out to the author, they were more than happy for their work to be maintained in the Geocaml organisation, updated to the latest OCaml tools and hopefully released on opam.

The dunification went smoothly whilst fast-forwarding 15 OCaml compiler versions with no changes or problems (just dune unused variable warnings).

An R-Tree in OCaml is created from an envelope (the kind of bounding box) and a value (for storing in the tree). The value depends on the envelope as you have to provide a function to calculate an envelope from a value. Here is one way we might define a line using the provided two-dimensional Rectangle envelope.

module Line = struct
  type t =
    { p0 : float * float; p1 : float * float }

  let t =
    let open Repr in
    record "line" (fun p0 p1 -> { p0; p1 })
    |+ field "p0" (pair float float) (fun t -> t.p0)
    |+ field "p1" (pair float float) (fun t -> t.p1)
    |> sealr

  type envelope = Rtree.Rectangle.t

  let envelope { p0 = (x1, y1); p1 = (x2, y2) } =
    let x0 = Float.min x1 x2 in
    let x1 = Float.max x1 x2 in
    let y0 = Float.min y1 y2 in
    let y1 = Float.max y1 y2 in
    Rtree.Rectangle.v ~x0 ~y0 ~x1 ~y1
end

In addition to the envelope function, we also need to provide a runtime representation using Repr. In the example, we've constructed it by hand but I recommend using ppx_repr to derive it for you. From here we can create our index.

module R = Rtree.Make (Rtree.Rectangle) (Line)

# let r =
    let i = R.empty 4 in
    R.insert i Line.{ p0 = (1., 1.); p1 = (2., 2.) };;
val r : R.t = <abstr>

# let r =
    let lines =
        Line.[
        { p0 = (0., 0.); p1 = (1., 1.) };
        { p0 = (1., 1.); p1 = (2., 2.) };
        { p0 = (2., 2.); p1 = (3., 3.) };
        { p0 = (3., 3.); p1 = (4., 4.) };
        ]
    in
    R.load ~max_node_load:4 lines;;
val r : R.t = <abstr>

# R.find r (Rtree.Rectangle.v ~x0:0. ~y0:0. ~x1:2. ~y1:2.);;
- : Line.t list =
[{Line.p0 = (0., 0.); p1 = (1., 1.)}; {Line.p0 = (1., 1.); p1 = (2., 2.)};
 {Line.p0 = (2., 2.); p1 = (3., 3.)}]