The most prevalent kind of abstraction is polymorphism. For example, List.length will work for all lists regardless of what kind of elements the list contains. However, polymorphism requires you to keep the polymorphic part of the type at arm's length. This can lead to suboptimal performance and/or limited usefulness.

List.mem is a good example; in order to be polymorphic List.mem relies on Stdlib.(=), the polymorphic equality operator. In turn, this operator achieves its polymorphism by performing a structural comparison of the runtime representation of its arguments.

A common way to overcome this is to use higher-order functions (functions that take functions as arguments).

# let rec mem ?(eq=Stdlib.(=)) v = function
  | [] -> false
  | x :: xs -> eq v x || mem ~eq v xs
val mem : ?eq:('a -> 'a -> bool) -> 'a -> 'a list -> bool = <fun>
# let mem_int_list = mem ~eq:Int.equal
val mem_int_list : int -> int list -> bool = <fun>

This works well in plenty of cases (List.map, Hashtbl.iter, etc.), however, it doesn't scale well from a user's perspective if the body of the function requires lots of specific functions (e.g. compare, length, pp). At the same time, we want to avoid the precarious duck-typing of something like Python too.

Functors are perhaps the second most prevalent feature of OCaml when it comes to code reuse. In the OCaml sense, a functor is a module that is parameterised by other modules. Set.Make is a functor that requires a module adhereing to the Set.OrderedType interface.

module IntSet = Set.Make (Int)
module StringSet = Set.Make (String)

Functors make it a lot easier to write implementations that depend on a large variety of functionality. Set.OrderedType is relatively tame in this sense, only needing a type t and a compare : t -> t -> int function.

Irmin, on the other hand, has functors that require users to provide Schemas. I have more thoughts about this API design in my Irmin retrospective.

It is important to consider the runtime costs of functors too. Whilst most of the time functors are instantiated at the toplevel, it does not come for free. You can have a play around with js_of_ocamlopt and see the generated output for how functors are applied.

Sitting somewhere inbetween polymorphism and functors are first-class modules with existentials. This is a rather advanced use of OCaml's type system to be able to pack together a value of a particular type alongside a module that works with those kinds of values.

type set = Set : ((module Set.S with type t = 'a) * 'a) -> set
let size (Set ((module S), s)) = S.cardinal s

Packaging up integer sets is fairly straight-forward.

# let int_empty_set = Set ((module IntSet), IntSet.empty)
val int_empty_set : set = Set (<module>, <poly>)

From a user's perspective we are now closer to the realm of dynamic typing. Indeed, existentials are the way to do that in OCaml.

# type any = Any : 'a -> any
type any = Any : 'a -> any
# let any v = Any v
val any : 'a -> any = <fun>

Getting back into a typed world often involves some type witness gymnastics or runtime exceptions. Used sparingly, however, it can be very effective particularly when a user is none the wiser about the use of FCMs and existentials. I have made use of this trick to package together Irmin stores with implementations.

You could not, however, simply return the value that was packed away inside the existential.

# let woops (Set (_, s)) = s
Line 1, characters 26-27:
Error: The value s has type $a but an expression was expected of type 'a
       The type constructor $a would escape its scope
       Hint: $a is an existential type bound by the constructor Set.

There is also a runtime cost to the packing (and unpacking) of the module, not to mention the need for the extra allocation to create the existential in the first place. This will likely not be too significant, but is worth considering.

OCaml 5.5 has module-dependent function types also known as modular explicits. The type of a function can now depend on types that come from a module that is also part of the function definition.

# let set_length : (module S : Set.S) -> S.t -> int =
  fun (module S) v -> S.cardinal v;;
val set_length : (module S : Set.S) -> S.t -> int = <fun>
# set_length (module IntSet) IntSet.empty;;
- : int = 0
# set_length (module StringSet) StringSet.empty;;
- : int = 0

In fact, we can now return values of a particular type that depends on a particular module!

# let empty_set : (module S : Set.S) -> S.t = 
  fun (module S) -> S.empty;;
val empty_set : (module S : Set.S) -> S.t = <fun>
# empty_set (module IntSet);;
- : IntSet.t = <abstr>
# empty_set (module StringSet);;
- : StringSet.t = <abstr>

Or, we can dynamically select implementations and values.

# let with_impl : 
    config:string -> ((module S : Set.S) -> S.t -> 'a) -> 'a = 
  fun ~config fn ->
  match String.lowercase_ascii config with
  | "string" -> fn (module StringSet) StringSet.empty
  | "int" -> fn (module IntSet) IntSet.empty
  | c -> invalid_arg c
val with_impl : config:string -> ((module S : Set.S) -> S.t -> 'a) -> 'a =
  <fun>

This is like a more usable version of the FCM-existential trick, though does require more manual type annotations on functions. For instance, I tried switching out OBuilder's use of existentials with modular explicits:

-type builder = Builder : (module Obuilder.BUILDER with type t = 'a) * 'a -> builder
-
 let log tag msg =
   match tag with
   | `Heading -> Fmt.pr "%a@." Fmt.(styled (`Fg (`Hi `Blue)) string) msg
   | `Note -> Fmt.pr "%a@." Fmt.(styled (`Fg `Yellow) string) msg
   | `Output -> output_string stdout msg; flush stdout

-let create_builder store_spec conf =
+let create_builder :
+  Store_spec.store Lwt.t -> Native_sandbox.config ->
+    ((module B : Obuilder.BUILDER) -> B.t -> 'a) -> 'a Lwt.t =
+  fun store_spec conf fn ->
   store_spec >>= fun (Store_spec.Store ((module Store), store)) ->
   let module Builder = Obuilder.Builder (Store) (Native_sandbox) (Docker_extract) in
   Native_sandbox.create ~state_dir:(Store.state_dir store / "sandbox") conf >|= fun sandbox ->
   let builder = Builder.v ~store ~sandbox in
-  Builder ((module Builder), builder)
+  fn (module Builder) builder

I think the code is more pragmatic; there is no longer an indirection via a type called builder. The extra type annotations do add a little noise. Roll on modular implicits.

Irmin, at its simplest, is a key-value database. Users associate keys with values and can query and update these bindings.

Additionally, this database supports versioning. This means independent snapshots of the database can coexist and users can switch between them and update them without fear of interfering with other versions.

Different versions of the database can be combined by merging. When you set up your instance of an Irmin database, you also provide it with a merge function.

# #show_type Irmin.Merge.f;;
type nonrec 'a f =
    old:'a Irmin.Merge.promise ->
    'a -> 'a -> ('a, Irmin.Merge.conflict) result Lwt.t

Irmin's API is very git-inspired. There is a large overlap of shared vocabulary and concepts: repositories, branches, commits, heads etc.

Probably the most confusing aspect of this is the notion of a Store. When I was describing Irmin above, I used the term database to help distinguish between some of these concepts. In Irmin's documentation, it is used for multiple related (but different) concepts. In "Creating a Store" stores refer to the entire database, whereas in the Irmin API docs we have that:

There are two kinds of store in Irmin: the ones based on persistent named branches and the ones based temporary detached heads.

For Irmin library users, a Store.t can be thought of as a checkout of the database at a particular revision. This may be from a branch or a specific commit.

# Store.of_branch;;
- : Store.repo -> string -> Store.t Lwt.t = <fun>
# Store.of_commit;;
- : Store.commit -> Store.t Lwt.t = <fun>

The overloading of the term is confusing. I think it leaves users unsure about what other people might mean when they say "store". Being careful with these terms, and the contexts in which they are used, would help avoid this confusion.

# #show_module Irmin_mem.KV;;
module KV :
  sig
    type endpoint = unit
    type metadata = unit
    type hash
    type info = Store.info
    type ('h, _)contents_key = 'h
    type 'h node_key = 'h
    type 'h commit_key = 'h
    module Make : (C : Irmin__.Contents.S) ->sig ... end
  end

It will probably come as no surprise that one of the main limitations of using Irmin is the lack of documentation and tutorials. This is a particular pain point as the API is not very straightforward.

A while back, I started an Irmin book intending to help document the kinds of things real-world users would need in order to use Irmin in earnest. For example: ways to deal with type migrations or primers on runtime types. I still believe this sort of work would be invaluable, but there is a large cost to completing it, and it is not clear if there is enough interest in the project to warrant such an effort.

The tutorials on the irmin website are still good starting points for most new users. But they quickly lack the depth for real-world scenarios.

What follows are additional ideas for improving Irmin.

There were two main things I hoped to continue working on: ppxlib and outreachy. These are projects that I was previously working on, and in the case of Outreachy I have now been involved for many years.

In addition to this, all of my Part II projects this year used OCaml in some regard. In general, I want to see more adoption of OCaml. Over the years this has taken many forms including my initial work on developing workflows for OCaml that just turned five years old. This directly fed into the rebranding and rethinking of ocaml.org itself.

So, I worked on most of what I wanted to work on and then some! Going forward I hope to keep maintaining ppxlib in some capacity and coordinating the OCaml community's Outreachy efforts.

I hope to continue my small experiments (e.g. converting Eio to Picos). My own research makes heavy use of Irmin, and I would be interested to help with that too. More recently, a new library has been released for working with numerical data in OCaml called Raven: I am interested to use this library in Geocaml, a suite of geospatial tools written in OCaml that I maintain.

I feel conflicted about the OxCaml efforts. I admit, I am not fully aware of the full benefits of the features, but I worry about some proliferation of modes that make the type system in OCaml unbearable to use. On top of this, with my ppxlib hat on, I worry about the impact of changing the compiler so frequently, placing strain on an already small community. That being said, by releasing OxCaml separately I do believe running it as an experimental set of packages will help understand the tool better. But I would not be building anything I intend to maintain or research with right now as that ecosystem is far too volatile.

Finally, over the past two years I have done a lot of teaching. From Part II projects to supervisions. Going into next (academic) year, I intend to reduce my in-person teaching to focus on my research. However, I am interested in producing more materials for learning, maybe some of this in OCaml, perhaps something similar to Beautiful Racket. I have tried in the past to do these sorts of things, for example The Irmin Book.

Thank you for reading this wrap up! And thank you again to Tarides for letting me work so freely on things that I think are good for the OCaml community.

opentrace has an all command that will trace the entire system.

$ sudo opentrace all --flags=O_WRONLY
pid,cgid,comm,kind,flags,mode,filename,return
16324,4417,".nheko-wrapped",openat,577,438,"/home/patrick/.cache/nheko/nheko/curl_alt_svc_cache.txt",47
16324,4417,".nheko-wrapped",openat,193,33188,"/home/patrick/.cache/nheko/nheko/cr3ZUHqBErIOe3PlwJ1SuU8zFKKxL12VzrRoHYMH.tmp",47
16324,4417,".nheko-wrapped",openat,577,438,"/home/patrick/.cache/nheko/nheko/curl_alt_svc_cache.txt",47
16324,4417,".nheko-wrapped",openat,193,33188,"/home/patrick/.cache/nheko/nheko/QfZTdZSuC56NdcDQ3aMxJc3BhMhAj8PmtYW1zFDP.tmp",47
2530,4235,"systemd",openat,524865,438,"/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/gammastep.service/cgroup.subtree_control",41
2530,4235,"systemd",openat,524545,0,"/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/gammastep.service/memory.min",41
2530,4235,"systemd",openat,524545,0,"/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/gammastep.service/memory.low",41
2530,4235,"systemd",openat,524545,0,"/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/gammastep.service/memory.high",41
2530,4235,"systemd",openat,524545,0,"/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/gammastep.service/memory.max",41

The --flags=O_WRONLY argument filters the events where the O_WRONLY flag was set in the call to open.

We also get the name of the current executable linked to the task (comm). The -wrapped is an artefact of using Nix.

The primary use case for this tool is to inspect what files your program might be reading and writing.

$ sudo opentrace exec --format=json --flags=O_CREAT -- opam list
$ cat trace.json | jq ".[] | .fname"
"/home/patrick/.opam/cshell/.opam-switch/packages/cache"
"/home/patrick/.opam/log/log-118747-29da3d.out"
"/home/patrick/.opam/log/log-118747-29da3d.err"
"/home/patrick/.opam/log/log-118747-29da3d.env"
"/home/patrick/.opam/log/log-118747-29da3d.info"

The "flags" argument can specify a small boolean formula for checking the open flags of a particular event with | (or), & (and), and ~ (not). Parentheses can be used for precedence.

$ sudo opentrace exec --flags="O_WRONLY|O_RDONLY" -- ocaml --version

One feature opentrace needs (in this proof-of-concept phase) is the ability to also trace subprocesses.

opentrace is primarily an eBPF program that is loaded into the kernel and communicates with an OCaml program. Events are communicated via a ring buffer and most of the post-processing happens in OCaml. To capture subprocesses, opentrace creates a new control group (cgroup) and places the new process into that group. This gives opentrace a new identifier to track, namely the cgroup.

So consider the following program.

let () =
  Eio_posix.run @@ fun env ->
  Eio.Path.(save ~create:(`Or_truncate 0o664) (env#fs / "hello.txt") "hello");
  Eio.Process.run env#process_mgr
    [ "/bin/bash"; "-c"; "echo 'heya' > heya.txt" ]

It first creates a file using direct calls to functions like openat. Then it spawns a process which creates a new file called heya.txt. This happens in a separate process. However, with the --cgroups flag we can capture both interactions with the operating system.

$ sudo opentrace exec --cgroups --flags="O_CREAT|O_TRUNC" ./main.exe
pid,cgid,comm,kind,flags,mode,filename,return
153187,530807,"main.exe",openat,526914,436,"hello.txt",5
153192,530807,"bash",openat,577,438,"heya.txt",3

Whilst testing opentrace against some of the tools I use nearly daily, I noticed some events were being missed. I tried tracing forester, and only the initial read of forest.toml was logged. It dawned on me that the reason for this was that forester (via eio) was using io_uring to perform most of the IO. Most attempts to open files were bypassing the open system calls, and instead they were being performed by the kernel after reading a submission request for an openat2-style call!

This is not news to seasoned, Linux systems programmers. Io_uring bypasses SECCOMP filters for exactly the same reasons.

$ sudo opentrace exec -- forester build
$ cat trace.csv
pid,cgid,comm,kind,flags,mode,filename,return
155007,535570,"forester",openat,524288,0,"forest.toml",5
155007,535570,"forester",Uring,2621440,0,"",0
155021,535570,"cp",openat,131072,0,"/home/patrick/documents/forest/theme/favicon-32x32.png",4
155007,535570,"forester",Uring,2686976,0,"",0
155007,535570,"iou-wrk-155007",Uring,557634,420,"",0
155007,535570,"iou-wrk-155007",Uring,557634,420,"",0
155007,535570,"iou-wrk-155007",Uring,557634,420,"",0

It is interesting to note two things here:

1. We can tell that forester reads the configuration file probably using something like In_channel in OCaml (it does).

2. It appears that Uring is performing IO in both worker threads and directly.

The file paths are empty at the moment as I cannot find a clean way to trace openat submission requests into Uring without it sometimes going very wrong. I have tried quite a few methods (e.g. tracing do_filp_open) and at the moment I am tracing io_openat2, but this seems quite brittle, and often the filename is completely garbled, so I do not set it. If anyone has any ideas to trace Io_uring more reliably, I am all ears!

Hazel is a functional programming language with typed holes. Holes are pieces of your program that have not yet been filled in. Holes can appear anywhere in your program both as expression or types. Hazel can still evaluate your program in the presence of holes.

To get a flavour of Hazel, take a regular map function for lists.

let map = fun f -> fun xs -> case xs
  | [] => []
  | x :: xs => f (x) :: map(f)(xs) 
end in
map(fun x -> ?)([1, 2, 3])

The question mark (?) is a hole. The program evaluates to the following expression of type [?] (for people more familiar with OCaml types ? list).

[ ?, ?, ? ]

Hazel supports local type inference but nothing involving unification variables. For example, a simple add_one function in Hazel (fun x -> x + 1) has type ? -> Int.

The ability to transpile OCaml programs to Hazel programs is motivated by one simple thought: there are more OCaml programs than there are Hazel programs. This could help bootstrap projects by alleviating the need to rewrite boilerplate code (e.g. URI parsing or standard library functions for strings).

type expr =
  | Float of float
  | Add of expr * expr
  | Sub of expr * expr
  | Mul of expr * expr
  | Div of expr * expr

type expr =
  + Float(Float)
  + Add((expr, expr))
  + Sub((expr, expr))
  + Mul((expr, expr))
  + Div((expr, expr))
 in ?

let rec map f = function
  | [] -> []
  | x :: xs -> f x :: map f xs

let map = fun f -> fun x1 -> case x1
  | [] => []
  | x :: xs => f(x) :: map(f)(xs)
end in ?

val map : 
  ('a -> 'b) -> 'a list -> 'b list

val map : ∀ a b. (a ->  b) -> a list -> b list

let map : forall a -> forall b -> (a -> b) -> [a] -> [b] =
  typfun a -> typfun b -> fun f -> fun xs -> case xs
    | [] => []
    | x :: xs => f (x) :: map@<a>@<b>(f)(xs) 
end in
map@<Int>@<Int>(fun x -> ?)([1, 2, 3])

let rec length = function
 | [] -> 0
 | _ :: xs -> 1 + length xs

let int_len = length [ 1; 2; 3 ]
let str_len = length [ "only"; "two" ]

let length : forall a -> [a] -> Int = typfun a -> fun x1 -> case x1
  | [] => 0
  | _ :: xs => 1 + length@<a>(xs)
end in
let int_len : Int = length@<Int>(1 :: 2 :: [3]) in
let str_len : Int = length@<String>("only" :: ["two"])
in ?

The impetus for this work was to derive a corpus of ill-typed Hazel programs. Luckily, such a corpus exists for OCaml! Seidel et al. created a corpus of OCaml programs from their undergraduate students at UC San Diego. Some of these programs have been transpiled to Hazel.

Hazel is a fun, research programming language. Potential third-year students may find it interesting to take this work further. For example, how would this look in terms of a module system? From a purely engineering perspective, plenty of work would be needed to convert a multi-library OCaml project to Hazel (e.g. handling the cmi files).

Another line of research would be to have Hazel target one of the intermediate representations in OCaml which would give Hazel a fully functioning compiler to "native" code?

It is possible to perform migrations between parsetrees manually using the ppxlib.ast package.

open Ppxlib
open Ast_builder.Make(struct let loc = Location.none end)
let pp_expr = Format.printf "%a%!" Pp_ast.(expression ?config:None)

let loc = Location.none
let str txt = { txt; loc }

module To_408 = 
  Ppxlib_ast.Convert (Ppxlib_ast.Js) (Ppxlib_ast__.Versions.OCaml_408)

The Ppxlib_ast.Js module is ppxlib's AST (since ppxlib.0.36.0 OCaml 5.2.0). We can use functions provided by the Convert functor to copy parts of the AST from one version to another. Usually there is very little difference between the versioned parsetrees.

let func arg body =
  pexp_fun Nolabel None (ppat_var (str arg)) body

let add_expr = func "x" (func "y" (eapply (evar "Int.add") [ evar "x"; evar "y" ]))

The above excerpt of OCaml code produces an expression in the language. More accurately, a function that has two arguments. If we print the simplified AST we can see that the arguments are stored together in a list.

# pp_expr add_expr;; 
Pexp_function
  ( [ { pparam_loc = __loc
      ; pparam_desc = Pparam_val ( Nolabel, None, Ppat_var "x")
      }
    ; { pparam_loc = __loc
      ; pparam_desc = Pparam_val ( Nolabel, None, Ppat_var "y")
      }
    ]
  , None
  , Pfunction_body
      (Pexp_apply
         ( Pexp_ident (Ldot ( Lident "Int", "add"))
         , [ ( Nolabel, Pexp_ident (Lident "x"))
           ; ( Nolabel, Pexp_ident (Lident "y"))
           ]
         ))
  )
- : unit = ()

This is a relatively new feature of the parsetree, if we migrate the expression down to 4.08 we see it is represented differently. The output is a little too verbose to be readable, and there's no good way to show the AST, so you will just have to take my word.

let expr_408 = To_408.copy_expression add_expr

OCaml 5.2 introduced a new way to represent functions. Before 5.2, all functions were represented with single arity. Functions with multiple arguments would return functions as their body. For example:

let add x y = x + y

A two-argument add function would be represented in the parsetree as fun x -> fun y -> x + y. Using the nodes from the parsetree (with some details removed) it would be Pexp_fun (x, Pexp_fun (y, Pexp_apply ...)).

Now, since OCaml 5.2, functions can be expressed with their syntactic arity intact. The function add would look something like Pexp_function ([ x; y ], Pfunction_body (Pexp_apply ...)). Both arguments are stored as a list.

The constructor Pexp_function was not a new addition to the parsetree. It used to represent pattern-matching expressions like function A -> 1. This can now be expressed as a function body of Pfunction_cases.

Whilst these additions and modifications to the parsetree are arguably a better design, it has caused chaos in terms of ppxlib's reverse dependencies!

In ppxlib 0.36.0, functions from Ast_builder will help ppx authors produce maximum arity functions. However, functions from Ast_helper will not do this.

let ast_helper_func arg body =
  Ast_helper.Exp.fun_ Nolabel None (ppat_var (str arg)) body

let ast_helper_expr = ast_helper_func "x" (ast_helper_func "y" (eapply (evar "Int.equal") [ evar "x"; evar "y" ]))

Here we recreate our helper function to create function expressions using Ast_helper.Exp.fun_ instead. We can now show the difference between the expr expression and the ast_helper_expr expression.

# pp_expr add_expr;;
Pexp_function
  ( [ { pparam_loc = __loc
      ; pparam_desc = Pparam_val ( Nolabel, None, Ppat_var "x")
      }
    ; { pparam_loc = __loc
      ; pparam_desc = Pparam_val ( Nolabel, None, Ppat_var "y")
      }
    ]
  , None
  , Pfunction_body
      (Pexp_apply
         ( Pexp_ident (Ldot ( Lident "Int", "add"))
         , [ ( Nolabel, Pexp_ident (Lident "x"))
           ; ( Nolabel, Pexp_ident (Lident "y"))
           ]
         ))
  )
- : unit = ()

Above, we see a function expression with arity two. Below, we see nested function expressions each with arity one.

# pp_expr ast_helper_expr;;
Pexp_function
  ( [ { pparam_loc = __loc
      ; pparam_desc = Pparam_val ( Nolabel, None, Ppat_var "x")
      }
    ]
  , None
  , Pfunction_body
      (Pexp_function
         ( [ { pparam_loc = __loc
             ; pparam_desc = Pparam_val ( Nolabel, None, Ppat_var "y")
             }
           ]
         , None
         , Pfunction_body
             (Pexp_apply
                ( Pexp_ident (Ldot ( Lident "Int", "equal"))
                , [ ( Nolabel, Pexp_ident (Lident "x"))
                  ; ( Nolabel, Pexp_ident (Lident "y"))
                  ]
                ))
         ))
  )
- : unit = ()

Many, many, many ppxes broke after migrating the ppxlib AST to 5.2. Mostly, this is due to pattern-matching. Downstream users of ppxlib frequently pattern-match directly against the parsetree. However, nodes such as Pexp_fun no longer exist and Pexp_function has completely different constructor arguments!

Patching all the ppxes has been quite difficult. Most ppxes only deal with single arguments at a time (e.g. Pexp_fun x) but now they must handle zero or more arguments at a time. (In the zero case, this will always correspond to a pattern-matching function).

Here is a non-exhaustive list of some of the ppxes that have been patched. It may useful for other ppx authors who wish to upgrade to ppxlib.0.36.0:

• https://github.com/patricoferris/ppx_deriving_yaml/pull/58

• https://github.com/mirage/ocaml-rpc/pull/181

• https://gitlab.com/o-labs/ppx_deriving_jsoo/-/merge_requests/1

• https://github.com/ocaml-community/sedlex/pull/160

• https://github.com/ocaml-sys/config.ml/pull/22

• https://github.com/vogler/ppx_distr_guards/pull/2

• https://github.com/ocaml-gospel/gospel/pull/424

• https://github.com/teamwalnut/graphql-ppx/pull/299

• https://github.com/ocsigen/lwt/pull/1033

• https://gitlab.com/gopiandcode/ppx-inline-alcotest/-/merge_requests/3

• https://github.com/dariusf/ppx_interact/pull/3

• https://github.com/ocaml-ppx/ppx_deriving_yojson/pull/160

• https://github.com/LPCIC/elpi/pull/276

• https://github.com/mirage/repr/pull/110

• https://github.com/ProgramingIsTheFuture/ppx_default/pull/3

• https://github.com/codinuum/vlt/pull/3

• https://github.com/sim642/ppx_deriving_hash/pull/6

• https://github.com/ocsigen/tyxml/pull/340

• https://github.com/sim642/ppx_viewpattern/pull/2

• https://github.com/little-arhat/ppx_jsobject_conv/pull/11

• https://github.com/robur-coop/lun/pull/1

• https://gitlab.com/o-labs/ppx_deriving_encoding/-/merge_requests/4

• https://github.com/francoisthire/bam/pull/12

• https://github.com/xguerin/bitstring/pull/36

This release has emphasised the tension between using pattern-matching on algebraic datatypes and wanting to reduce downstream breakages (e.g. using abstract datatypes). This tension is not new.

Ppxlib is not going anywhere anytime soon. We have committed to trying to keep the internal AST up to date with the latest OCaml release. In fact, there is already a PR for the 5.3 bump!

There have been many discussions about simplifying this process, but so far the time and effort to build a ppxlib that will result in fewer breakages is too high (see an attempt using views).

Thank you to Tarides and Jane Street for funding my time to work on this release and all the chaos it has caused.

Happy preprocessing!

Dune, for quite some time, has supported virtual libraries. This allows library authors to define a signature without an implementation (similar to OCaml's module type S = struct ... end). The package is something other dune libraries can build against, still without choosing a particular implementation of the interface. An application author can make the decision about which implementation they would like to build into their program.

This feature has been quite widely used in OCaml. For example, the excellent digestif library of hashing algorithms provides both C and OCaml implementations individually packaged into digestif.c and digestif.ocaml respectively. Consider developing a browser application that needs some hashing algorithm (e.g. Irmin in the browser). We can explicitly use the OCaml backend for digestif which js_of_ocaml will happily compile for us.

There are two distinct limitations to this approach.

1. Applications can only link exactly one implementation.

2. Implementations cannot expose additional functionality easily (see dune#5597).

MirageOS is a framework for building unikernels in OCaml. It relies heavily on OCaml's functors to abstract implementation from interface. The mirage tool will then select appropriate instantiations of these functors depending on the backend that is targeted (e.g. unix, hvt etc.)

Recently, there has been a push to use dune virtual libraries instead.

We need to be able to overcome the limitation that implementations cannot expose additional functionality very easily using dune virtual libraries. As it stands, we need to make Mirage's virtual libraries unwrapped (i.e. set (wrapped false)). This allows implementers to expose additional modules alongside the implementation of the virtual library.

For VPNkit, we need this to port some of the DNS forwarding tests which use a fake clock and sleep module. Previously, they could be provided directly as functor arguments to the various DNS modules e.g.

let module R =
  Dns_forward.Resolver.Make(Proto_client)(Fake.Time)(Fake.Clock)

In this new world of dune virtual libraries, our Resolver is significantly more simple.

let module R = Dns_forward.Resolver.Make(Proto_client)

The flexibility of time and clock implementation has been pushed to a build-time decision in a dune file. This means that in order to provide the same fake clock and time we will need to create a private dune library that (a) implements the respective signatures of the virtual libraries and (b) provides the extended interface to allow the test suite to modify the internals of the implementations.

The first thing I had to do was make the mirage-mtime and mirage-sleep virtual libraries unwrapped. This allowed me to make a new private dune library called fake_time with two public modules: the normal interface required by mirage-mtime and another module called Fake_time_state containing:

let timeofday = ref 0L
let c = Lwt_condition.create ()

let advance nsecs =
  timeofday := Int64.add !timeofday nsecs;
  Lwt_condition.broadcast c ()

let reset () =
  timeofday := 0L;
  Lwt_condition.broadcast c ()

I did the same for fake_sleep which depended on the state from the fake_time library (something a normal library wouldn't).

Once I had the implementations for the virtual libraries in place, all that remained was to tell the executable (in this case the test suite) to select these new implementations instead of the defaults. Whereas before, all the tests could live in the same test suite, now they had to be split into those that needed the fake implementation and those that did not. This is another slight limitation of this virtual library approach. You can only link one implementation at a time to an executable (or test). Thus, the dune rules for running the dns_forward test suite became:

; Uses the default  implementations provided by virtual libraries
(executable
 (name test)
 (libraries dns_forward alcotest))

; Override the defaults and select the private fake libraries
(executable
  (name test_fake)
  (libraries dns_forward alcotest fake_sleep fake_time))

With all of this machinery in place, I could now update test_fake to use the new modules that would provide the custom functionality for the clock and time implementations.

(* The bad server should be marked offline and no-one will wait for it *)
Fake_time_state.reset ();
(* avoid the timeouts winning the race with the actual result *)
Fake_time_state.advance Duration.(of_ms 500); 

The switch to dune virtual libraries seems to be a good one. A majority of applications only require a single clock or time implementation. However, it has become harder to inject custom implementations of various resources into your MirageOS-based applications.

The pull request to upgrade VPNKit is open.

In the Energy and Environment Group we are working on various geospatial tools, one of which is written in OCaml and needed a geospatial index to speedup spatial queries over a dataset. Searching Github revealed an implementation in pure OCaml. After reaching out to the author, they were more than happy for their work to be maintained in the Geocaml organisation, updated to the latest OCaml tools and hopefully released on opam.

The dunification went smoothly whilst fast-forwarding 15 OCaml compiler versions with no changes or problems (just dune unused variable warnings).

An R-Tree in OCaml is created from an envelope (the kind of bounding box) and a value (for storing in the tree). The value depends on the envelope as you have to provide a function to calculate an envelope from a value. Here is one way we might define a line using the provided two-dimensional Rectangle envelope.

module Line = struct
  type t =
    { p0 : float * float; p1 : float * float }

  let t =
    let open Repr in
    record "line" (fun p0 p1 -> { p0; p1 })
    |+ field "p0" (pair float float) (fun t -> t.p0)
    |+ field "p1" (pair float float) (fun t -> t.p1)
    |> sealr

  type envelope = Rtree.Rectangle.t

  let envelope { p0 = (x1, y1); p1 = (x2, y2) } =
    let x0 = Float.min x1 x2 in
    let x1 = Float.max x1 x2 in
    let y0 = Float.min y1 y2 in
    let y1 = Float.max y1 y2 in
    Rtree.Rectangle.v ~x0 ~y0 ~x1 ~y1
end

In addition to the envelope function, we also need to provide a runtime representation using Repr. In the example, we've constructed it by hand but I recommend using ppx_repr to derive it for you. From here we can create our index.

module R = Rtree.Make (Rtree.Rectangle) (Line)

# let r =
    let i = R.empty 4 in
    R.insert i Line.{ p0 = (1., 1.); p1 = (2., 2.) };;
val r : R.t = <abstr>

# let r =
    let lines =
        Line.[
        { p0 = (0., 0.); p1 = (1., 1.) };
        { p0 = (1., 1.); p1 = (2., 2.) };
        { p0 = (2., 2.); p1 = (3., 3.) };
        { p0 = (3., 3.); p1 = (4., 4.) };
        ]
    in
    R.load ~max_node_load:4 lines;;
val r : R.t = <abstr>

# R.find r (Rtree.Rectangle.v ~x0:0. ~y0:0. ~x1:2. ~y1:2.);;
- : Line.t list =
[{Line.p0 = (0., 0.); p1 = (1., 1.)}; {Line.p0 = (1., 1.); p1 = (2., 2.)};
 {Line.p0 = (2., 2.); p1 = (3., 3.)}]

The work came from wanting to merge two shell sessions into one. Setting aside the practicalities of an implementation, it makes sense to start first theoretically. When can I merge two shell executions, and when can I not?

It would be nice to think of unset DOWNLOAD_DIR as a function. This function takes the shell state and the operating state as arguments and returns a modified shell state and an "unmodified" operating system state (at least for our purposes). Meanwhile, branch-b may have a lot more side effects, but when focusing on the environment only we can see in a sense the two branches are conflict free. Well, sorta. We also need to ensure we do not also read DOWNLOAD_DIR in branch-b if we intend to remove it in their merge.

This work, in part, is borrowing ideas from MRDTs and can reuse void processes to help reason about when side effects can occur, as well as open-trace (which is now becoming os-trace).

I hope to continue this work, see my latest weekly for more on this.

I spent some time carving out Eio_browser into a standalone library, something I depend on for Merry to work in the browser.

module Stdenv : sig
  type base =
    < stdout : Eio_unix.sink_ty r
    ; stderr : Eio_unix.sink_ty r
    ; clock : float Eio.Time.clock_ty r
    ; mono_clock : Eio.Time.Mono.ty r
    ; secure_random : Eio.Flow.source_ty r
    ; debug : Eio.Debug.t
    ; backend_id : string >
end

val run : (Stdenv.base -> unit) -> unit
(** [run fn] runs [fn] passing it the {! Stdenv.base} environment.
    See {! Backend.run} for more information about how effects are handled.*)

July was a slower month as I was on holiday for parts of it which broke a lot of work up.

By far the most exciting update was that we finally have a browser in the shell: Browsher. This is the culmination of lots of abstraction work in the core of Merry. In particular, dealing with file descriptors. I have added this to Eio_mem which now supports read-write pipes and FD duping! Read more about that in my weekly.

This work also had me writing a standard environment for Eio using the browser. You can see this in the browsher's setup module. This includes:

• A capability for monotonic and normal time.

• A source of secure random bytes using WebCrypto.

• The fake in-memory filesystem coming from Eio_mem.

Other work had me fixing a long-standing bug where Eio.Path.load did not work on /proc files.

The full details are in the weeklies transcluded below:

__ready=
trap __ready=1 HUP
kill -HUP "$MAIN_PID"
until ["$__ready" ]; do __dummy=; done

$ ./shellbench -s bash,zsh,msh sample/func.sh
-----------------------------------------------------------------------------------
name                                                     bash        zsh        msh
-----------------------------------------------------------------------------------
func.sh: no func                                      775,539  1,124,485      5,229
func.sh: func                                         398,861    197,169     93,424
-----------------------------------------------------------------------------------
* count: number of executions per second

f () { :; }
while __count=$(($__count+1)) && ["$__count" -le 10000 ]; do
	f
done
echo "count is $__count"

$ time -p dash test.sh
count is 10001
real 0.01
user 0.01
sys 0.00
$ time -p msh test.sh
count is 10001
real 15.53
user 4.28
sys 12.28

$ time -p dash test.sh
count is 10001
real 7.55
user 3.95
sys 3.44

Browsher and Namespaces [weekly-2026-w33]

• August 17, 2026
• Patrick Ferris

The work presented constitutes a good bit more than a weeks worth of work. That being said, let's dive in.

Browsher [browsher]

• August 17, 2026
• Patrick Ferris

After a week or two of abstracting the internals of Merry, I finally got a shell running in the browser. You can have a play with it here:

https://patrick.sirref.org/msh

The final hurdle was providing an abstract interface to file handles. Eio, the library that so far has enabled much of the OS abstraction, doesn't have a particularly good cross-platform file descriptor API. Eio_unix does extend sources and sinks with the Unix_fd tag, but there is no Eio.Fd.dup2 that provides a platform-agnostic API for file handle manipulation. Even if it did, a POSIX shell does a fair amount of assuming that your file handles are integers which might prove tricky on Windows.

The shell in the browser completely reuses the logic of Merry. It also links in a browser-aware subpackage of bruit (the OCaml port of linenoise I wrote) and a few, half-baked POSIX utilities I wrote (like ls and cat). But you can at least run something like:

$ echo "hello" > hello.txt
$ ls -la .
$ cat < hello.txt

Being able to, in some sense, run a shell anywhere is actually quite an attractive proposition. Shells, with all of their wonderful footguns, are still a reliable and familiar tool for interacting with a running system. I'm interested in the idea of running shells in more exotic deployments (e.g., unikernels) where it becomes an interface for dynamical reconfiguration or a tool for debugging.

Updates on Eio_mem

• August 17, 2026
• Patrick Ferris

At the heart of browsher is Eio_mem (an in-memory filesystem for Eio). The Eio_mem API now comes with (and exposes) a number of file descriptor operations:

val pipe : Eio.Switch.t -> Fd.t * Fd.t
val dup : sw:Eio.Switch.t -> Fd.t -> Fd.t
val dup2 : sw:Eio.Switch.t -> src:Fd.t -> tgt:Fd.t -> unit

I was happy to see Eio_mem getting some use in Thomas G.'s fork eio-fs where he was able to use it for chaos engineering.

Namespaces [namespace-musings]

• August 17, 2026
• Patrick Ferris

I spent a few days recently trying to come up with some formalism for the notion of names, namespaces and name resolution.

On the surface, it seems bigraphs might be a fruitful approach. The orthogonal treatment of locality and connectivity maps fairly well onto name structure (e.g, Module.X.foo, /foo/bar/hello.txt or foo.example.org) and redirection (e.g., symlinks or CNAMEs). Bigraph reaction rules could then model name resolution. A resolution process can move through the hierarchical name structure and across links at which point you might have some interesting properties to prove or check about a particular namespace system. Additionally, namespaces and resolution mechanism might compose in different ways and this might be worth formalising and comparing.

I didn't get much further than modelling some bits of a filesystem in BigraphER.

# filesystem
ctrl Dir = 0;
atomic ctrl File = 1;
atomic ctrl SymLink = 2;

# Query -- the thing that gets resolved.
atomic ctrl Q = 1;

# Reaction Rules
react resolve_symlink =
  Q{x} | SymLink{x, y} --> Q{y} | SymLink{x, y};

# Initial State
big s0 =
  Dir.
    (SymLink{sym_a, file_a}
     |
     Q{sym_a}
     |
     File{file_a}
    );

# Bigraphical Reactive System
begin brs 
  init s0;
  rules = [ {resolve_symlink} ];
end

Resulting in these two transition states.

Bear in mind this was just a quick experiment. The first bigraph shows our query at sym_a and the second bigraph (using the resolve_symlink reduction rule) shows the query at file_a. All of this takes places within Dir. It seems that BigraphER represents the tree structure graphically with black arrows (rather than spatial containment).

Papers

• August 17, 2026
• Patrick Ferris

I'm working on a paper right now for Merry and Shelter. It is mostly coming together okay; and I am enjoying the creative aspects of this paper, those things with no right answer just "hey, I think this is a good/useful/fun idea".

My TikZ-abilities have also had a refresher.

TODOs

• August 17, 2026
• Patrick Ferris

Over the next week, I hope to finish up a first draft of the aforementioned paper. I also want to publish a backlog of packages I have for OCaml: bruit, ocaml-bibtex, a fresh cut of Graft etc.

Ramblings

• August 17, 2026
• Patrick Ferris

I had some half-formed thoughts lying around in my forest, so I thought I might include those here:

Boilerplate [on-boilerplate]

• August 11, 2026
• Patrick Ferris

The word boilerplate gained popularity as a colloquialism used by newspaper and magazine editors for pre-prepared editorials and stories that were syndicated across the US. In effect, boilerplates were news stamps that could produce identical content.

I have been thinking about the nature of boilerplate code. How is it similar or different to boilerplate in its original meaning.

Distinguishing true boilerplate code from code that feels repetitive or boring to write is, I think, important. Argument parsing falls into this category quite well. Consider the following flag I added to Merry's set built-in.

let all_export =
  let doc = "Export all variables, like -o allexport." in
  Arg.(value & flag & info [ "a" ] ~docv:"ALLEXPORT" ~doc)

It uses cmdliner to define a new -a flag that will, in effect, make all variable definitions act like the export built-in. LLMs have been praised for their ability to do this specific task in OCaml.

What exactly is boilerplate about this code snippet? It seems to me that there is some sort of ceremony to defining a flag. There is boilerplate in the structure with more meaningful content threaded through. Below, is the example reduced to just the boilerplate replacing the for more meaningful parts with Merlin's typed holes: _.

let all_export =
  let doc = _ in
  Arg.(value & flag & info [ _ ] ~docv:_ ~doc)

Arguably the use of flag is quite important too as that is where the magic happens if you want an optional argument, or a positional one etc. As a thought experiment, having agents deliver fill-in-the-blanks style code might be quite useful. In a type system that can support holes (e.g. like in OCaml or Hazel) it becomes a more pleasant exercise, but helps keep you in the loop. I, myself, do not use any AI in the code I write; so I admit that I am pretty far from the problem.

The problem I have with this approach is that it is fairly easy to automate in a predictable and easy way! I just added a snippet subcommand to octx, but you could easily wrap this up in whatever your editor of choice is.

$ octx snippet cmdliner-flag
let cmdliner_flag =
  let doc = _ in
  Arg.(value & flag & info [ _ ] ~docv:_ ~doc)

The more I think on this problem the harder I find good uses for LLMs besides churning out vasts amount of end-to-end code; something I am personally not that interested in doing. It seems to me, to reap the benefits of LLMs, one must relinquish a certain amount of agency; anything less, and you could easily replace its usage with a cheaper, more predictable tool.

June saw a lot of progress on Merry. At the start of the month, I worked on meio, a tracing tool for Eio programs. This was to help debug a particular shell script fragment that was causing Merry to hang. It uncovered a few UX opportunities to make meio more useful, and this might be fruitful direction to take some of my work.

Merry is now able to run (and pass!) the entire unixfun benchmarks in Koala. Whilst benchmarking Merry, it became apparent that it is paramount to provide a test built-in (rather than spawning [).

Part of this work also looked into building an in-memory filesystem for Eio. It is still a work-in-progress, but does pass many of the filesystem tests in Eio. To be a drop-in replacement in Merry requires some more work to handle an abstraction for Unix file descriptors (e.g. a fake stdout). It would be nice if Eio provided a Unix filesystem interface similar to the OS-agnostic interfaces of Eio itself (not to be confused with the Unix utilities in Eio_unix).

I have been very grateful to Thomas's diligent and steadfast progress on merging and fixing bugs in Eio over the last few months. You can keep up to date with his work in his notes.

The full details are in the weeklies transcluded below:

Using meio to debug Merry [weekly-2026-w23]

• June 4, 2026
• Patrick Ferris

During a large rewrite of Merry, I introduced a regression causing certain shell programs to hang indefinitely. I thought it might be a good opportunity to try out Meio.

First, let's look at a trace of Meio from some real-world programs. For example: meio -- msh -c "sleep 100".

One problem that comes up immediately from this example is that the tree hierarchy of tasks and cancellation contexts makes it tricky to reason about the logging. Here, we can see the pipeline's cancellation context clearly labelled pipeline-id1. This is an Eio.Switch.t for the pipeline sleep 100. The actual execution log for sleep 100 bubbles back up to the main fiber (that has no name, so it is called <task>). The daemon fiber, for ensuring the spawned process is reaped, is attached to the switch and is displayed below it.

Meio has a long way to go. Most programs are too fast to be monitored in this way. Using something like tokio-console's model might be a better idea here focusing less on the tree strucuture of the program, and more on the performance and monitoring of individual tasks. Without this, it turns out Meio was not all that useful for debugging this specific issue (which ended up being some bad pipe management on my part).

apt-get install debootstrap -y
debootstrap stable /stable-chroot http://deb.debian.org/debian/

$ count () { echo "Count: $#"; }; FOO="hello world"; count $FOO
Count: 2

$ IFS=" "; FOO=" hello    world"; echo $FOO
hello world
$ IFS=" "; FOO=" hello    world"; echo "$FOO"
 hello    world
$ IFS=":"; FOO="a:b:c:"; echo $FOO
a b c
$ IFS=":"; FOO="a:b::c:d"; echo $FOO
a b  c d
$ IFS=":"; FOO=":a:b:c"; echo $FOO
 a b c
$ IFS=; FOO="unset   IFS"; echo $FOO
unset   IFS

iter () {
 echo "a"
 echo "b"
 echo "c"
}

base=$(iter)

for n in $base; do
  echo "Got $n"
done

for n in "$base"; do
  echo "Got $n"
done

$ echo "!dlrow olleh" > input.txt
$ (read line; echo $line) < input.txt | rev
hello world!

for m in $MIRRORS; do
    pkgdest="$TARGET/$($DLDEST pkg "$s" "$c" "$a" "$m" "$path")"
    if [! -e "$pkgdest" ]; then continue; fi
    pkgs_to_get="$(download_debs "$m" "$pkgdest" $pkgs_to_get 5>&1 1>&6)"
    if [ -z"$pkgs_to_get" ]; then break; fi
done 6>&1

 let () =   let port = ref 8080 in
   Arg.parse
     [ ("-p", Arg.Set_int port, " Listening port number(8080 by default)") ]     ignore "An HTTP/1.1 server";
   Eio_main.run @@ fun env ->
+  Eio_mem.run @@ fun menv ->
   Eio.Switch.run @@ fun sw ->
-  (* Restrict to current directory: *)
-  let htdocs = Eio.Stdenv.cwd env in
+  (* Restrict to in-memory current directory: *)
+  Eio_mem.Fs.load ~filter (Eio.Stdenv.cwd env);
+  let htdocs = Eio.Stdenv.cwd menv in
   let socket =
     Eio.Net.listen env#net ~sw ~backlog:128 ~reuse_addr:true
       (`Tcp (Eio.Net.Ipaddr.V4.loopback, !port))

Elsewhere, I spent some time exploring Landlock and how it might be used with Eio. I think it is useful to down tools once in a while and try out some newfangled API or OS feature. Landlock is quite different to Capsicum (I had assumed they would be more similar). Whilst I enjoy the fine-grained control over access and capabilities, the manual process of constructing rules is a little cumbersome (and perhaps easy to make mistakes with). Combining that with [Eio]'s capability API might make that more tenable. The idea being you could have some Eio code like:

let () =
  Eio_linux.run @@ fun env ->
  let data = Eio.Path.(env#fs / "/data") in
  Eio.Path.with_open_dir sw data @@ fun data_dir ->
  match Eio_linux.Landlock.enter () with
  | Error (`Msg m) -> failwith msg
  | Ok () ->
    (* At this point, only certain default operations can happen to [data_dir].*)
    analysis data_dir

Opening the directory (in the openat-sense) adds a Landlock rule to the current context (some kind of process-local state). This rule would be removed whenever the data directory is closed (by hand or when the function returns, i.e., when the internal Eio.Switch releases its resources). Doing the bookkeeping internally means mistakes are less likely to be made whilst also improving, in my opinion, the API.

This breaks down for networking, unfortunately. Whilst the filesystem hierarchy provides a clean way to produce capabilities (a handle on a directory + openat + RESOLVE_BENEATH), networking does not. The Landlock API, at the time of writing, only allows rules based on port numbers. The system API is completely ambient (socket + connect). Eio's API provides access to the network through a single network `capability. All of these mechanisms are somewhat at odds. What does a network look like if it adheres to the ideas we have for combining Landlock and Eio (like in the filesystem example)?

The API becomes port-oriented. We're already at odds with Unix domain sockets, but we will carry on regardless. A capability on a port means you can listen on that port and accept connections, or you could connect to it and send data. At the time of writing, Landlock is restricted to TCP ports. In Eio, there is no API to with_open_port, i.e., to register your intent to do something with that port -- you either listen or connect.

All of this starts feeling a lot like Plan 9; something which I hope to explore further soon.

The full details are in the weeklies transcluded below:

Moving away from Github [weekly-2026-w15]

• April 13, 2026
• Patrick Ferris

A short week for me as I was hiking with family in and around Porthmadog, Wales.

Reliance on Github

• April 13, 2026
• Patrick Ferris

I joined Github in 2016, but only began using it in earnest (according to my Github activity) in July 2018: right after Microsoft announced its intent to acquire Github; and just before they closed the $7.5bn deal. I originally joined to boost my résumé and I hoped to receive the odd open-source contribution to projects I was working on (like ppx_deriving_yaml). Since then, I have used Github extensively both at work (Tarides) and for more personal projects.

There have been many reasons to jump ship from Github ever since the Microsoft acquisition (e.g. their planned (and quickly unplanned) announcement to charge self-hosted Github runners). Ultimately, I was convinced when I noticed the new AI tool on the homepage with options to select your "versatile and highly intelligent" model. There is no need for me to stay on the platform for my personal projects. AI integration into Github has nullified those earlier motivations for hosting my projects on the site. The number of contributions has increased, but many of them fix superficial elements of an underlying bug. In my (maybe too uncharitable) imagination, it is all too easy for contributors to prompt their built-in LLM to "fix this broken unit test" and for the fix to layer a single-use workaround on top, instead of identifying and correcting the relevant code. This is, at least, what it looks like to me when I receive a new review request on my open-source projects.

At the end of last week I set up my own Git infrastructure following Ryan's nixos setup. In the future, the source of truth for my projects will be git.sirref.org (I will likely continue to mirror to tangled.org or Github).

Reduce, Reuse, Recycle

• April 13, 2026
• Patrick Ferris

Part of self-hosting your own source code is managing some form of automatic testing. I hesitate to describe it as "continuous integration" as the term is confusing and overloaded. Luckily, parts of the OCaml community have been building test infrastructure for a while.

I have a proclivity for rebuilding things from scratch be it to learn or to build something the way I want. It is a proclivity I am trying to reign in as I find less and less time to hack on projects. I think this is a good thing. Reusing and recycling existing technologies, in earnest, feels like a pushback on the not-invented-here syndrome that seems to be snowballing thanks to LLMs and code generation tools.

To that end, I dusted off my OCurrent knowledge, and repurposed the "local" mode of OCaml-CI to work with my git repositories. After battling through nested layers of git submodules, I managed to Nix-ify this and get it running at tests.sirref.org. I needed to add multi-repository support as well as Current.Monitors for the local git refs. By no means perfect, but it is nice to own the testing infrastructure (and the cost of downloading OCaml images and the compute to run the tests).

Landlocked Eio [weekly-2026-w16]

• April 18, 2026
• Patrick Ferris

I spent the start of last week wrapping up the "local" CI I mentioned the previous week. Unfortunately it was mostly a packaging nightmare, followed by missing features in ocaml-ci-local. It would be great to have a richer CLI tool for running tests on a directory of git repositories, but unfortunately most of the existing tooling has prioritised the Github/Gitlab etc. APIs.

After coming across minimal during the week, I took it as an opportunity to learn a little more about Linux's Landlock mechanism.

Landlocked Eio [landlocked-eio]

• April 18, 2026
• Patrick Ferris

Landlock is a sandboxing mechanism that has existed in the Linux kernel since 5.13. It allows unprivileged processes a means to restrict their ambient rights (e.g. creating a file or opening a network connection). These rights are ambient in the ambient authority sense, i.e., any process can attempt to perform an action without needing access to any particular capability to do so.

Ambient systems are often contrasted with capability-based ones. In terms of operating systems, this is most commonly illustrated with access-control-lists-as-columns and capabilities-as-rows in an access matrix.

Eio is a capability-based library in OCaml -- resources are provided at the start of the application and can be passed around wherever they are needed. The design is covered in Thomas Leonard's lambda capabilities blog post. Unfortunately, any application (willingly or via a third-party library) may ambiently access OS resources. In OCaml, this may be by using the standard library functions (e.g. Sys.readdir), things from the Unix module, a C FFI function or by spawning a subprocess.

I had a go at "landlocking" Eio applications last week after stumbling across minimal's hakoniwa library for process isolation (which is very reminiscent of void processes).

module Landlock = Eio_linux.Landlock
let ( / ) = Eio.Path.( / )

let run_eio file =
  Eio.traceln "Writing to file";
  Eio.Path.save ~create:(`Exclusive 0o644) file "hello, world!";
  Eio.traceln "Reading file: %s" (Eio.Path.load file);
  Eio.Path.unlink file

let bad_program () =
  In_channel.with_open_bin "/etc/passwd" In_channel.input_all
  |> Eio.traceln "/etc/passwd: %s"

let () =
  Eio_linux.run @@ fun env ->
  Eio.Path.with_open_dir (Eio.Stdenv.cwd env / ".") @@ fun dir ->
  let parent_fd = 
    Eio_unix.Resource.fd_opt (fst dir) |> Option.get
  in
  let file = dir / "hello.txt" in
  let rules =
    Landlock.Rule.
      [
        path_beneath ~parent_fd
          ~flags:Flags.Fs.(
            write_file + read_file + make_reg + remove_file
           );
      ]
  in
  match Landlock.enter rules with
  | Ok () ->
      run_eio file;
      bad_program ()
  | Error `Not_supported -> Eio.traceln "Landlock not supported!"

Running this program (with the necessary privileges) prints the following.

+Writing to file
+Reading file: hello, world!
Fatal error: exception Sys_error("/etc/passwd: Permission denied")

Eio already has a similar function for the BSDs using capsicum. In capsicum, processes enter a capability mode:

cap_enter() places the current process into capability mode, a mode of execution in which processes may only issue system calls operating on file descriptors or reading limited global system state.

This plays nicely with Eio's capabilities. Within the scope of Eio.Path.with_open_dir, the directory will remain usable after entering "cap" mode. This is not the case with landlock unless you add a specific path_beneath rule to the ruleset (like in the example above). However, you could imagine plumbing the necessary information throughout the Eio_linux backend for it to build a set of rules that can be applied at any given moment in order to mimic the capsicum semantics.

Self-hosting Music [self-host-music]

• April 18, 2026
• Patrick Ferris

After my friend, Greta, sent me some articles about music streaming services and their exploitation of artists, I decided enough was enough and setup navidrome. It is compatible with Subsonic and so far I'm very pleased to be streaming my own music to my devices. I am using substreamer on iOS. On my laptop, I tried out Andreas Rossberg's Camp

What you got here is an old-school music player heavily inspired by good old Winamp [1], with a focus on decent music library and playlist handling.

I took Anil's OCaml Installer tool for a spin with great success... after a few mishaps mostly related to system dependencies.

$ nix-shell -p pkg-config gcc pulseaudio miniaudio libxi libxrandr libxinerama libffi mesa libxcursor libGL 
$ oi-linux-x86_64 run --with=https://github.com/patricoferris/camp#nix -- camp

The fixes in my fork of Camp are just to install the assets using dune and also statically provide the -lpulse flag (which should probably be dynamically picked up). Otherwise, raylib or miniaudio selects the Null playback device. Roll-on cross-ecosystem package management.

$ dash -c "cat /dev/urandom | hxd.xxd -l100"
00000000: 5033 7981 ae44 d475 074c 5361 0a9e 3828  P3y..D.u.LSa..8(
00000010: f88c 1f9e 3b0e bcde 528a f78c b1e4 7195  ....;...R.....q.
00000020: 16f2 8325 c90b 9310 539c af04 8289 c3a0  ...%....S.......
00000030: 57c5 680a 2c81 a999 71b6 708e 95df e720  W.h.,...q.p....
00000040: 004c 407a 8365 9448 65c6 b435 bed2 4084  .L@z.e.He..5..@.
00000050: 62ea 96ab 2b23 0385 f9d2 fc30 d090 1719  b...+#.....0....
00000060: 68d1 c514                                h...
cat: write error: Broken pipe

while echo "some bytes"; do
  :
done | hxd.xxd -l100

info () {
  echo "INFO: $1" >&3
}

exec 3>&1 > /dev/null
info "Starting up server..."
echo "Some noisy program spamming STDOUT"
info "All is quiet..."

exec < ./hello.txt

while read line; do
  echo "Got $line" >&2
done | (exec < ./world.txt; cat)

% ./test/check_fork.sh "while echo 5; do : ; done | true"
bash  3
dash  3
yash  1 
zsh   1
ksh   1
ash   3

Long time, no weekly. Since the start of this year I have been building a POSIX shell in OCaml called msh (with the underlying library called Merry). Merry is available online now.

docker run -it --rm patrickferris/msh

opam pin git+https://tangled.org/patrick.sirref.org/merry

let ( / ) = Eio.Path.( / )

let checkerboard ~size =
  let v = Nx.zeros Nx.uint8 [| size; size |] in
  for row = 0 to size - 1 do
    for col = 0 to size - 1 do
      if (row + col) mod 2 = 0 then Nx.set_item [ row; col ] 254 v
    done
  done;
  Nx.to_bigarray v

let () =
  Eio_posix.run @@ fun env ->
  Tiff_eio.with_open_out (env#cwd / "example.tiff") @@ fun w ->
  let tif = Tiff.make (checkerboard ~size:256) in
  Tiff.to_file tif w