Software. Eio [eio]

https://github.com/ocaml-multicore/eio

Eio provides an effects-based direct-style IO stack for OCaml 5

Backlinks

Analysing some Dockerfiles [weekly-2026-w37]

September 12, 2026
Patrick Ferris

After spending a few weeks thinking about the semantics of POSIX shells, this week saw me analysing lots of Dockerfiles looking for some evidence for how they are being used. It also saw environment variables landing in Eio thanks to Thomas and Anil. We also had some issues in ppxlib migrations.

Dockerfile Analysis [msh-dockerfile-analysis]

September 12, 2026
Patrick Ferris

Part of my research at the moment is thinking about how shells are used. What commands are run? What constructs are used (e.g. redirections, compound commands...)? Dockerfiles are a defacto standard means for building container images and for the most part they are a shell with weird semantics and some meta-commands (sounds familiar!).

I had some assumptions about the kinds of shell commands people use (myself included) when writing Dockerfiles. For example, making use of AND-lists for grouping commands in RUN instructions. But I had no evidence to back this up. So I ran some analysis over the dataset from this paper from Henkel et al..

Total Dockerfiles: 218_253 (errors: 808)
Total Docker Instructions: 2_729_118
Total RUN instructions: 657_909 (errors: 4_847)
========

Count (Windows Dockerfiles): 841 / 218_253
Count (ENV commands): 268_898 / 2_729_118
Count (COPY commands): 165_968 / 2_729_118
Count (ADD commands): 86_552 / 2_729_118
Count (WORKDIR commands): 104_112 / 2_729_118
Count (using at least one &&): 267_113 / 657_909
Count (uses of exec): 25 / 657_909
Count (system pkg manager: [apt, apt-get, dnf, apk, pacman, nix, brew, yum]): 192_092 / 657_909
Count (internet retrievers: [curl, wget]): 95_433 / 657_909
Count (might use python): 9_082 / 657_909
Counts:
[(rm, 169_386), (echo, 113_768), (mkdir, 100_924), (cd, 74_904),
 (chmod, 55_998), (set, 25_738), (mv, 24_034), (sed, 21_232),
 (chown, 18_610), (grep, 16_604), (cp, 15_637), (export, 13_154),
 ([, 10_411), (touch, 3_152), (exit, 2_288), (printf, 2_275), (awk, 2_010),
 (., 1_077), (test, 1_004), (source, 904), (pwd, 199), (unset, 105),
 (diff, 40), (trap, 2)]

I think there's some evidence here to support some assumptions I already had:

A significant proportion of RUN instructions in Dockerfiles invoke package managers.
A significant proportion of RUN instructions invoke curl or wget.
There is something rather Zipfian about how shell built-in utilities are used.
There is limited used of exec (and therefore limited use of opening file descriptors directly via the shell, as opposed to a command running in a subprocess).
A lot of time is spent changing directories, either temporally (via cd in a RUN instruction, or by using WORKDIR).
Lots of ENV setting.

Parsing Dockerfiles

September 12, 2026
Patrick Ferris

For reasons that are probably quite clear, whenever I end up doing a small task like this, I always reach for OCaml to do it. The idea is that I'm invested in that community and hope to see it thrive. And if that means contributing some fixes, libraries or advice along the way, it seems to me that that is time well spent.

When I started, I assumed ocaml-dockerfile had support for parsing files; it turns out it doesn't, but I found Thomas G.'s port that does. So I took it for a spin and the results were less than I had hoped for. Of the 219,061 Dockerfiles, 97,123 were not parseable! After a series of commits I managed to get that number down to 808.

Eio process environment

September 12, 2026
Patrick Ferris

What started out seemingly straightforward (add a capability-based API to Eio for accessing environment variables) ended up being a more nuanced discussion about what the best API would be. In the end, Thomas added an Eio.Process.Env API to manipulate soon-to-be opaque environments. This paved the way for the portable environment support.

The API Eio exposes is readonly so we do not have to worry too much about race conditions (even though users are free to Unix.putenv).

One quirk of this work was reaffirming something I had already stumbled across in Merry. Accessing and searching large environments has a cost! For my rather large Nix environment, we can see the slowdown pretty clearly.

                  name,   major-allocated,   minor-allocated,   monotonic-clock
        get_env/all 10,        346.619865,      38577.227215,      29248.828738
        get_env/all 50,       1677.003302,     181788.755923,     138307.102538
       get_env/all 100,       3323.770915,     361218.802888,     274768.577259
   get_env/all_once 10,        121.955078,      14694.900546,      11652.096583
   get_env/all_once 50,        563.133065,      61951.683658,      48161.605588
  get_env/all_once 100,       1113.238600,     121431.915770,      93960.646400
     get_env/direct 10,          0.000000,          0.000000,        974.649587
     get_env/direct 50,          0.000000,          0.000000,       4369.486360
    get_env/direct 100,          0.000000,          0.000000,       8753.393503

The all benchmark calls environment every time and then finds a variable using Eio.Process.Env.get_opt, all_once calls environment once and then Eio.Process.Env.get_opt every time and direct uses the Eio.Process.get_opt directly every time. The differences are a lot less stark when the environment is pretty small.

Ppxlib Migrations

September 12, 2026
Patrick Ferris

Thankfully, ppxlib has been mostly stable for a number of months. In part, thanks to the diligent efforts of Nathan. However, David uncovered some unpleasant bugs in the migration code for some new 5.4 AST features. Unfortunately, it is only for attributes which are a source of pain for ppxlib. I spent some time reviewing the fixes, and it looks like we still need some more work to ensure the attributes attached to package types roundtrip cleanly.

Misc.

September 12, 2026
Patrick Ferris

Some other work I recorded separately and include here for the eager reader.

OCaml Roundup: August 2026 [ocaml-roundup-august-2026]

July 12, 2026
Patrick Ferris

Much of August was spent thinking about POSIX shell semantics, and in particular extending notions from smoosh to include side effects (and maybe concurrency). This is still nascent work, but some ideas and flavours are described below.

Merging Shells

July 12, 2026
Patrick Ferris

The work came from wanting to merge two shell sessions into one. Setting aside the practicalities of an implementation, it makes sense to start first theoretically. When can I merge two shell executions, and when can I not?

It would be nice to think of unset DOWNLOAD_DIR as a function. This function takes the shell state and the operating state as arguments and returns a modified shell state and an "unmodified" operating system state (at least for our purposes). Meanwhile, branch-b may have a lot more side effects, but when focusing on the environment only we can see in a sense the two branches are conflict free. Well, sorta. We also need to ensure we do not also read DOWNLOAD_DIR in branch-b if we intend to remove it in their merge.

This work, in part, is borrowing ideas from MRDTs and can reuse void processes to help reason about when side effects can occur, as well as open-trace (which is now becoming os-trace).

I hope to continue this work, see my latest weekly for more on this.

Eio in the browser

July 12, 2026
Patrick Ferris

I spent some time carving out Eio_browser into a standalone library, something I depend on for Merry to work in the browser.

Eio_browser [eio_browser]

September 12, 2026
Patrick Ferris
https://git.sirref.org/eio_browser

Eio_browser is a backend for Eio that works in the browser. It reuses work I started at Tarides that picked up some momentum but has since been a little dormant.

Crucially, it provides a functional standard environment like Eio_main or any of the OS-specific backends.

module Stdenv : sig
  type base =
    < stdout : Eio_unix.sink_ty r
    ; stderr : Eio_unix.sink_ty r
    ; clock : float Eio.Time.clock_ty r
    ; mono_clock : Eio.Time.Mono.ty r
    ; secure_random : Eio.Flow.source_ty r
    ; debug : Eio.Debug.t
    ; backend_id : string >
end

val run : (Stdenv.base -> unit) -> unit
(** [run fn] runs [fn] passing it the {! Stdenv.base} environment.
    See {! Backend.run} for more information about how effects are handled.*)

Eio_browser [eio_browser]

September 12, 2026
Patrick Ferris
https://git.sirref.org/eio_browser

Eio_browser is a backend for Eio that works in the browser. It reuses work I started at Tarides that picked up some momentum but has since been a little dormant.

Crucially, it provides a functional standard environment like Eio_main or any of the OS-specific backends.

module Stdenv : sig
  type base =
    < stdout : Eio_unix.sink_ty r
    ; stderr : Eio_unix.sink_ty r
    ; clock : float Eio.Time.clock_ty r
    ; mono_clock : Eio.Time.Mono.ty r
    ; secure_random : Eio.Flow.source_ty r
    ; debug : Eio.Debug.t
    ; backend_id : string >
end

val run : (Stdenv.base -> unit) -> unit
(** [run fn] runs [fn] passing it the {! Stdenv.base} environment.
    See {! Backend.run} for more information about how effects are handled.*)

Browsher and Namespaces [weekly-2026-w33]

August 17, 2026
Patrick Ferris

The work presented constitutes a good bit more than a weeks worth of work. That being said, let's dive in.

Browsher [browsher]

August 17, 2026
Patrick Ferris

After a week or two of abstracting the internals of Merry, I finally got a shell running in the browser. You can have a play with it here:

https://patrick.sirref.org/msh

The final hurdle was providing an abstract interface to file handles. Eio, the library that so far has enabled much of the OS abstraction, doesn't have a particularly good cross-platform file descriptor API. Eio_unix does extend sources and sinks with the Unix_fd tag, but there is no Eio.Fd.dup2 that provides a platform-agnostic API for file handle manipulation. Even if it did, a POSIX shell does a fair amount of assuming that your file handles are integers which might prove tricky on Windows.

The shell in the browser completely reuses the logic of Merry. It also links in a browser-aware subpackage of bruit (the OCaml port of linenoise I wrote) and a few, half-baked POSIX utilities I wrote (like ls and cat). But you can at least run something like:

$ echo "hello" > hello.txt
$ ls -la .
$ cat < hello.txt

Being able to, in some sense, run a shell anywhere is actually quite an attractive proposition. Shells, with all of their wonderful footguns, are still a reliable and familiar tool for interacting with a running system. I'm interested in the idea of running shells in more exotic deployments (e.g., unikernels) where it becomes an interface for dynamical reconfiguration or a tool for debugging.

Updates on Eio_mem

August 17, 2026
Patrick Ferris

At the heart of browsher is Eio_mem (an in-memory filesystem for Eio). The Eio_mem API now comes with (and exposes) a number of file descriptor operations:

val pipe : Eio.Switch.t -> Fd.t * Fd.t
val dup : sw:Eio.Switch.t -> Fd.t -> Fd.t
val dup2 : sw:Eio.Switch.t -> src:Fd.t -> tgt:Fd.t -> unit

I was happy to see Eio_mem getting some use in Thomas G.'s fork eio-fs where he was able to use it for chaos engineering.

Namespaces [namespace-musings]

August 17, 2026
Patrick Ferris

I spent a few days recently trying to come up with some formalism for the notion of names, namespaces and name resolution.

On the surface, it seems bigraphs might be a fruitful approach. The orthogonal treatment of locality and connectivity maps fairly well onto name structure (e.g, Module.X.foo, /foo/bar/hello.txt or foo.example.org) and redirection (e.g., symlinks or CNAMEs). Bigraph reaction rules could then model name resolution. A resolution process can move through the hierarchical name structure and across links at which point you might have some interesting properties to prove or check about a particular namespace system. Additionally, namespaces and resolution mechanism might compose in different ways and this might be worth formalising and comparing.

I didn't get much further than modelling some bits of a filesystem in BigraphER.

# filesystem
ctrl Dir = 0;
atomic ctrl File = 1;
atomic ctrl SymLink = 2;

# Query -- the thing that gets resolved.
atomic ctrl Q = 1;

# Reaction Rules
react resolve_symlink =
  Q{x} | SymLink{x, y} --> Q{y} | SymLink{x, y};

# Initial State
big s0 =
  Dir.
    (SymLink{sym_a, file_a}
     |
     Q{sym_a}
     |
     File{file_a}
    );

# Bigraphical Reactive System
begin brs 
  init s0;
  rules = [ {resolve_symlink} ];
end

Resulting in these two transition states.

Bear in mind this was just a quick experiment. The first bigraph shows our query at sym_a and the second bigraph (using the resolve_symlink reduction rule) shows the query at file_a. All of this takes places within Dir. It seems that BigraphER represents the tree structure graphically with black arrows (rather than spatial containment).

Papers

August 17, 2026
Patrick Ferris

I'm working on a paper right now for Merry and Shelter. It is mostly coming together okay; and I am enjoying the creative aspects of this paper, those things with no right answer just "hey, I think this is a good/useful/fun idea".

My TikZ-abilities have also had a refresher.

TODOs

August 17, 2026
Patrick Ferris

Over the next week, I hope to finish up a first draft of the aforementioned paper. I also want to publish a backlog of packages I have for OCaml: bruit, ocaml-bibtex, a fresh cut of Graft etc.

Ramblings

August 17, 2026
Patrick Ferris

I had some half-formed thoughts lying around in my forest, so I thought I might include those here:

Boilerplate [on-boilerplate]

August 11, 2026
Patrick Ferris

The word boilerplate gained popularity as a colloquialism used by newspaper and magazine editors for pre-prepared editorials and stories that were syndicated across the US. In effect, boilerplates were news stamps that could produce identical content.

I have been thinking about the nature of boilerplate code. How is it similar or different to boilerplate in its original meaning.

Distinguishing true boilerplate code from code that feels repetitive or boring to write is, I think, important. Argument parsing falls into this category quite well. Consider the following flag I added to Merry's set built-in.

let all_export =
  let doc = "Export all variables, like -o allexport." in
  Arg.(value & flag & info [ "a" ] ~docv:"ALLEXPORT" ~doc)

It uses cmdliner to define a new -a flag that will, in effect, make all variable definitions act like the export built-in. LLMs have been praised for their ability to do this specific task in OCaml.

What exactly is boilerplate about this code snippet? It seems to me that there is some sort of ceremony to defining a flag. There is boilerplate in the structure with more meaningful content threaded through. Below, is the example reduced to just the boilerplate replacing the for more meaningful parts with Merlin's typed holes: _.

let all_export =
  let doc = _ in
  Arg.(value & flag & info [ _ ] ~docv:_ ~doc)

Arguably the use of flag is quite important too as that is where the magic happens if you want an optional argument, or a positional one etc. As a thought experiment, having agents deliver fill-in-the-blanks style code might be quite useful. In a type system that can support holes (e.g. like in OCaml or Hazel) it becomes a more pleasant exercise, but helps keep you in the loop. I, myself, do not use any AI in the code I write; so I admit that I am pretty far from the problem.

The problem I have with this approach is that it is fairly easy to automate in a predictable and easy way! I just added a snippet subcommand to octx, but you could easily wrap this up in whatever your editor of choice is.

$ octx snippet cmdliner-flag
let cmdliner_flag =
  let doc = _ in
  Arg.(value & flag & info [ _ ] ~docv:_ ~doc)

The more I think on this problem the harder I find good uses for LLMs besides churning out vasts amount of end-to-end code; something I am personally not that interested in doing. It seems to me, to reap the benefits of LLMs, one must relinquish a certain amount of agency; anything less, and you could easily replace its usage with a cheaper, more predictable tool.

OCaml Roundup: July 2026 [ocaml-roundup-july-2026]

July 12, 2026
Patrick Ferris

Summary

July 12, 2026
Patrick Ferris

July was a slower month as I was on holiday for parts of it which broke a lot of work up.

By far the most exciting update was that we finally have a browser in the shell: Browsher. This is the culmination of lots of abstraction work in the core of Merry. In particular, dealing with file descriptors. I have added this to Eio_mem which now supports read-write pipes and FD duping! Read more about that in my weekly.

This work also had me writing a standard environment for Eio using the browser. You can see this in the browsher's setup module. This includes:

A capability for monotonic and normal time.
A source of secure random bytes using WebCrypto.
The fake in-memory filesystem coming from Eio_mem.

Other work had me fixing a long-standing bug where Eio.Path.load did not work on /proc files.

Weekly logs

July 12, 2026
Patrick Ferris

The full details are in the weeklies transcluded below:

Shell Benchmarks, Back to Writing [weekly-2026-w26]

July 4, 2026
Patrick Ferris

A little late due to travel, but I did a little benchmarking of Merry.

Signal Handling in Merry

July 4, 2026
Patrick Ferris

In order to even run shellbench, I first had to fix some bugs in the shell implementation. Shellbench aims to warm up the shell before starting any benchmarks; it includes this prelude before getting started:

__ready=
trap __ready=1 HUP
kill -HUP "$MAIN_PID"
until ["$__ready" ]; do __dummy=; done

Whilst writing Merry, I hadn't really considered that the return context from a trap needed to be taken into consideration! I had assumed they would be silently ignored, how else would we know where to thread that context back into the shell's execution.

Even after reading through the specification for trap I am still none the wiser. I've opted to place some sensible signal polling points during the execution of the commands to run any signals that have come in. The single handlers in Merry simply push thunks onto a run queue. Looking at the Dash source code, it seems to be something that happens just before evaluation.

Benchmarking Merry [merry-benchmarks]

July 4, 2026
Patrick Ferris

Medium-to-small potatoes

July 4, 2026
Patrick Ferris

I used shellbench to understand where some of the performance pitfalls were coming from.

$ ./shellbench -s bash,zsh,msh sample/func.sh
-----------------------------------------------------------------------------------
name                                                     bash        zsh        msh
-----------------------------------------------------------------------------------
func.sh: no func                                      775,539  1,124,485      5,229
func.sh: func                                         398,861    197,169     93,424
-----------------------------------------------------------------------------------
* count: number of executions per second

This particular benchmark first runs nothing (i.e. :) and then a function lookup and execution. Surprisingly msh is slower at doing nothing! I perf-ed a run of msh test.sh with just a single : in the file:

Test built-in

July 4, 2026
Patrick Ferris

After making various improvements to Merry that almost halved the memory usage (e.g., removing extraneous string manipulations, adding a fast-path to the PATH variable for resolving programs). I was still pretty far off the performance of these other shells. I decided to have a look at dash which has a simple implementation.

This flame graph is for the following shell script:

f () { :; }
while __count=$(($__count+1)) && ["$__count" -le 10000 ]; do
	f
done
echo "count is $__count"

And here are some rudimentary benchmarks:

$ time -p dash test.sh
count is 10001
real 0.01
user 0.01
sys 0.00
$ time -p msh test.sh
count is 10001
real 15.53
user 4.28
sys 12.28

It turns out the massive slowdown is very simple to explain. In the dash flamegraph there is a oexpr stack; which comes from the bltin/test.c file. dash, along with the other shells, does not spawn a process for the [-test command! Forcing dash to do use the binary [ gives:

$ time -p dash test.sh
count is 10001
real 7.55
user 3.95
sys 3.44

Which is much more respectable in terms of performance compared to msh. Plenty of work to do though!

Parsing arithmetic expressions

July 4, 2026
Patrick Ferris

There was some obvious low-hanging fruit to help improve some of the performance differences. The first was pre-parsing arithmetic expressions. Morbig, the static parser, does not implement arithmetic expression parsing; so Merry does it for us. Before, it was on-demand, as we encountered an expression of the form $((<e>)). Now, it happens up front. This should improve tight-loops.

Writing

July 4, 2026
Patrick Ferris

More on this soon, but I have started writing two separate papers bringing all of this work together.

Browsher and Namespaces [weekly-2026-w33]

August 17, 2026
Patrick Ferris

The work presented constitutes a good bit more than a weeks worth of work. That being said, let's dive in.

Browsher [browsher]

August 17, 2026
Patrick Ferris

After a week or two of abstracting the internals of Merry, I finally got a shell running in the browser. You can have a play with it here:

https://patrick.sirref.org/msh

$ echo "hello" > hello.txt
$ ls -la .
$ cat < hello.txt

Updates on Eio_mem

August 17, 2026
Patrick Ferris

At the heart of browsher is Eio_mem (an in-memory filesystem for Eio). The Eio_mem API now comes with (and exposes) a number of file descriptor operations:

val pipe : Eio.Switch.t -> Fd.t * Fd.t
val dup : sw:Eio.Switch.t -> Fd.t -> Fd.t
val dup2 : sw:Eio.Switch.t -> src:Fd.t -> tgt:Fd.t -> unit

I was happy to see Eio_mem getting some use in Thomas G.'s fork eio-fs where he was able to use it for chaos engineering.

Namespaces [namespace-musings]

August 17, 2026
Patrick Ferris

I spent a few days recently trying to come up with some formalism for the notion of names, namespaces and name resolution.

I didn't get much further than modelling some bits of a filesystem in BigraphER.

# filesystem
ctrl Dir = 0;
atomic ctrl File = 1;
atomic ctrl SymLink = 2;

# Query -- the thing that gets resolved.
atomic ctrl Q = 1;

# Reaction Rules
react resolve_symlink =
  Q{x} | SymLink{x, y} --> Q{y} | SymLink{x, y};

# Initial State
big s0 =
  Dir.
    (SymLink{sym_a, file_a}
     |
     Q{sym_a}
     |
     File{file_a}
    );

# Bigraphical Reactive System
begin brs 
  init s0;
  rules = [ {resolve_symlink} ];
end

Resulting in these two transition states.

Papers

August 17, 2026
Patrick Ferris

My TikZ-abilities have also had a refresher.

TODOs

August 17, 2026
Patrick Ferris

Over the next week, I hope to finish up a first draft of the aforementioned paper. I also want to publish a backlog of packages I have for OCaml: bruit, ocaml-bibtex, a fresh cut of Graft etc.

Ramblings

August 17, 2026
Patrick Ferris

I had some half-formed thoughts lying around in my forest, so I thought I might include those here:

Boilerplate [on-boilerplate]

August 11, 2026
Patrick Ferris

I have been thinking about the nature of boilerplate code. How is it similar or different to boilerplate in its original meaning.

let all_export =
  let doc = "Export all variables, like -o allexport." in
  Arg.(value & flag & info [ "a" ] ~docv:"ALLEXPORT" ~doc)

let all_export =
  let doc = _ in
  Arg.(value & flag & info [ _ ] ~docv:_ ~doc)

$ octx snippet cmdliner-flag
let cmdliner_flag =
  let doc = _ in
  Arg.(value & flag & info [ _ ] ~docv:_ ~doc)

OCaml Roundup: June 2026 [ocaml-roundup-june-2026]

July 12, 2026
Patrick Ferris

Summary

July 12, 2026
Patrick Ferris

June saw a lot of progress on Merry. At the start of the month, I worked on meio, a tracing tool for Eio programs. This was to help debug a particular shell script fragment that was causing Merry to hang. It uncovered a few UX opportunities to make meio more useful, and this might be fruitful direction to take some of my work.

Merry is now able to run (and pass!) the entire unixfun benchmarks in Koala. Whilst benchmarking Merry, it became apparent that it is paramount to provide a test built-in (rather than spawning [).

Part of this work also looked into building an in-memory filesystem for Eio. It is still a work-in-progress, but does pass many of the filesystem tests in Eio. To be a drop-in replacement in Merry requires some more work to handle an abstraction for Unix file descriptors (e.g. a fake stdout). It would be nice if Eio provided a Unix filesystem interface similar to the OS-agnostic interfaces of Eio itself (not to be confused with the Unix utilities in Eio_unix).

I have been very grateful to Thomas's diligent and steadfast progress on merging and fixing bugs in Eio over the last few months. You can keep up to date with his work in his notes.

Weekly logs

July 12, 2026
Patrick Ferris

The full details are in the weeklies transcluded below:

Using meio to debug Merry [weekly-2026-w23]

June 4, 2026
Patrick Ferris

During a large rewrite of Merry, I introduced a regression causing certain shell programs to hang indefinitely. I thought it might be a good opportunity to try out Meio.

First, let's look at a trace of Meio from some real-world programs. For example: meio -- msh -c "sleep 100".

One problem that comes up immediately from this example is that the tree hierarchy of tasks and cancellation contexts makes it tricky to reason about the logging. Here, we can see the pipeline's cancellation context clearly labelled pipeline-id1. This is an Eio.Switch.t for the pipeline sleep 100. The actual execution log for sleep 100 bubbles back up to the main fiber (that has no name, so it is called <task>). The daemon fiber, for ensuring the spawned process is reaped, is attached to the switch and is displayed below it.

Meio has a long way to go. Most programs are too fast to be monitored in this way. Using something like tokio-console's model might be a better idea here focusing less on the tree strucuture of the program, and more on the performance and monitoring of individual tasks. Without this, it turns out Meio was not all that useful for debugging this specific issue (which ended up being some bad pipe management on my part).

Three weeks in the Shell [weekly-2026-w25]

June 16, 2026
Patrick Ferris

The last three weeks have been focused work on building Merry. In practice, I wanted to run the following two commands through Merry's shell: msh:

apt-get install debootstrap -y
debootstrap stable /stable-chroot http://deb.debian.org/debian/

Seemingly innocuous. But both commands run thousands of lines of shell script. Whether that's Debian package maintainer scripts or state-machines for processing package release files.

What follows is a handful of the bugs and the shell scripts that caused them.

Field Splitting

June 16, 2026
Patrick Ferris

Edge-cases abound in the POSIX shell specification; and field splitting is one of the most serious perpetrators. Field splitting happens after word expansion. It is a known footgun amongst shell script authors.

$ count () { echo "Count: $#"; }; FOO="hello world"; count $FOO
Count: 2

Quoting variables that need expanding will prevent field splitting from taking place, but it is an easy thing to forget.

Field splitting uses the IFS (internal field separator) environment variable to determine how to do the splitting, and it comes with some extra rules beyond "split on these characters".

$ IFS=" "; FOO=" hello    world"; echo $FOO
hello world
$ IFS=" "; FOO=" hello    world"; echo "$FOO"
 hello    world
$ IFS=":"; FOO="a:b:c:"; echo $FOO
a b c
$ IFS=":"; FOO="a:b::c:d"; echo $FOO
a b  c d
$ IFS=":"; FOO=":a:b:c"; echo $FOO
 a b c
$ IFS=; FOO="unset   IFS"; echo $FOO
unset   IFS

Variable assignment

June 16, 2026
Patrick Ferris

How a shell stores variables also impacts field splitting. Naively, Merry stored variables as strings, but really they should be stored as their expanded form.

Consider this example of saving a function output in a variable called base.

iter () {
 echo "a"
 echo "b"
 echo "c"
}

base=$(iter)

for n in $base; do
  echo "Got $n"
done

for n in "$base"; do
  echo "Got $n"
done

Later, when we come to iterate across the fields of $base we wouldn't be able to reconstruct the intial fields that $(iter) produced if we simply saved the result as a string. Too much information would have been lost.

Redirections

June 16, 2026
Patrick Ferris

Redirections control what file descriptors point to. Each redirection has a lifetime associated with it. For example: cat < README.md really says "open README.md and execute the cat program mapping the process's FD 0 to the FD of README.md". When cat returns that redirection is no longer valid.

Other redirections are not so simple. Consider:

$ echo "!dlrow olleh" > input.txt
$ (read line; echo $line) < input.txt | rev
hello world!

There are lots of redirections here too. Firstly the standard input for everything inside the braced command (read line2; echo $line) should be the open file descriptor for input.txt. The standard output of the braced command should be the writing end of a pipe whose reading end is passed to rev as that process's standard input. The final standard output for rev should be unchanged from what it was originally (the shell's standard output).

Getting redirections right is hard, consider this snippet from the debootstrap scripts (all in the context of stdout and stderr pointing to a debootstrap.log file and file descriptor 4 pointing at the true stdout).

for m in $MIRRORS; do
    pkgdest="$TARGET/$($DLDEST pkg "$s" "$c" "$a" "$m" "$path")"
    if [! -e "$pkgdest" ]; then continue; fi
    pkgs_to_get="$(download_debs "$m" "$pkgdest" $pkgs_to_get 5>&1 1>&6)"
    if [ -z"$pkgs_to_get" ]; then break; fi
done 6>&1

More niche built-ins

June 16, 2026
Patrick Ferris

As the dust begins to settle on the nuts and bolts of implementing a POSIX-ish shell, new challenges are allowed to emerge from deep within scripts.

One such challenge is the seemingly endless (I know, they're finite) shell built-ins! Whilst trying to run debootstrap using its fakechroot variant, I bumped into getopts... a whole command-line parsing shell built-in.

An In-memory Filesystem for Eio [eio_mem]

June 16, 2026
Patrick Ferris

Towards the end of last week and the beginning of this week, I threw together a prototype for an in-memory filesystem for Eio: Eio_mem. Concretely, this provides a means for constructing an Eio.Fs.dir_ty Eio.Path.t value that is completely in-memory.

Thanks to Eio's capability-first approach, any existing code implemented against the _ Eio.Path.t API should just work with the in-memory filesystem.

There's a convenient Eio_mem.Fs.load function that can take a path from one filesystem (most likely a normal one from Eio_linux or Eio_posix) and replicate it into the in-memory filesystem. For example, you might have a static web-server pointing to a directory like "/var/www". The following might be the only change you need to make it completely in-memory (example from ocaml-cohttp):

 let () =   let port = ref 8080 in
   Arg.parse
     [ ("-p", Arg.Set_int port, " Listening port number(8080 by default)") ]     ignore "An HTTP/1.1 server";
   Eio_main.run @@ fun env ->
+  Eio_mem.run @@ fun menv ->
   Eio.Switch.run @@ fun sw ->
-  (* Restrict to current directory: *)
-  let htdocs = Eio.Stdenv.cwd env in
+  (* Restrict to in-memory current directory: *)
+  Eio_mem.Fs.load ~filter (Eio.Stdenv.cwd env);
+  let htdocs = Eio.Stdenv.cwd menv in
   let socket =
     Eio.Net.listen env#net ~sw ~backlog:128 ~reuse_addr:true
       (`Tcp (Eio.Net.Ipaddr.V4.loopback, !port))

The keen-eyed reader might have noticed the filter parameter. This can be used to ignore certain paths (here I ignored .git and _build which were adding substantial time to loading the filesystem).

Catching up in May [weekly-2026-w22]

June 1, 2026
Patrick Ferris

I never quite found the time in May to write weeknotes. I think that's, in some ways, quite a good thing. My philosophy on weeknotes is not that they are a public punchcard of my progress. They are a tool for organising thoughts and keeping my colleagues abreast of what I am up to. Only the latter suggests any necessity for timeliness and given I frequently message, call and zulip them, if a few weeks fall between the cracks then that's okay!

All that being said, I have tried to provide some thoughts and things I have been up to over the course of May and not just week 22 of the year.

Tools for Shells

June 1, 2026
Patrick Ferris

Some tools and libraries related to Merry.

Meio

June 1, 2026
Patrick Ferris

The shell work continues, and as the complexity internally increases I have found myself wishing for some more tools for debugging concurrent Eio programs. Given enough pipes and compound commands, it is fairly easy to introduce a fair deal of concurrency to your shell programs. To this end I hope I have improved meio somewhat in this fork.

In particular, there were several things I wanted to improve.

Simple UI improvements -- the set of tools for building TUI UIs from nottui are, on closer inspection, rather good and simple. For example, turning a particular UI component into a scrollable pane is as simple as wrapping it in a call to this function.
Logging more runtime events -- Eio programs emit information via OCaml's runtime events. This is how meio knows which fiber is currently active for example. It also emits information like "the current fiber created a new promise" and "this fiber is blocked waiting for a promise". I have not figured out an intuitive way for representing this in meio's tree-view, but I did turn these into "system logs" that appear in the per-fiber logs which is already an improvement. Thanks to Thomas for helping understand the runtime event sequence for logging this information.
Timestamping where possible -- all runtime events come with a timestamp, these were mostly discarded by meio.

Some aspects of meio made it clear that labelling Eio.Switches and Eio.Promises might be a good practice. After reading Michael's Eio forays, I can't help but wonder if a more opinionated Eio for Systems Programmers guide might be useful. I think the Eio README does a wonderful job of introducing most of the low-level concepts, but not quite how they are used by the working systems programmer.

Glob

June 1, 2026
Patrick Ferris

A little dismayed at the lack of progress I was making with Merry, I switched to a specific problem I knew I wanted to fix -- removing the dependency on glob(3), replacing it with a pure OCaml implementation.

let get_txts fs =
  Glob_eio.glob fs Glob.(of_component [ dir_sep; string "home"; dir_sep; star; string ".txt" ])

Anil provided some issues with fixes too which was very appreciated.

Beyond factoring out fork+exec, Merry should now be portable to the browser. Of course there are lots of other refactorings needed to (e.g. a fake-filesystem) -- but I'm interested in how I can use Eio's filesystem capabilities to do this pretty seemlessly. Having an emeddable, pure OCaml shell may also unlock some cool debugging capabilities for unikernels too!

Geocaml

June 1, 2026
Patrick Ferris

Michael has been valiantly trying some of the Geocaml libraries in his work on rendering LiDAR point clouds. Where possible, I have been reviewing Michael's PRs and helping implement features he might find useful. I also took his Laserver for a spin, and suggested a few changes.

Rtree

June 1, 2026
Patrick Ferris

After a comical series of attempts, I (or rather Anil) released rtree.0.2.0 which contained:

A bug fix where not all the tree values were being returned.
A new bounds function for getting an Rtree's maximum bound.
A set of simple removal functions.

PROJ

June 1, 2026
Patrick Ferris

Michael added some new functions to ocaml-proj. Previously, I had worked on making PROJ more portable by using virtual libraries. This was a little restrictive, as it forced users to always implement a JS version of any functionality that exists in the native version of PROJ.

To make it easier to contribute (and allow the C bindings to contain more specific bindings) I carved out the two virtual library implementations into standalone libraries, and reimplemented the virtual library support using these libraries. Now projects can depend directly on proj_c if they wish to have more features in exchange for being less portable.

Ppxlib

June 1, 2026
Patrick Ferris

Thankfully, work on ppxlib has been relatively quiet. Nathan also opened a PR for initial OCaml 5.6 support. Keeping on top of 5.6 features is probably the right way to go about this (rather than one giant PR during the 5.6 release cycle).

I'm particularly happy to have some better migration error handling. The previous errors, though uncommon, were confusing ppx users and hopefully this will help ppx authors and ppxlib maintainers triaging important encoding bugs or feature wishes.

Self-hosting

June 1, 2026
Patrick Ferris

I now have a fairly workable setup for self-hosting source code with git.sirref.org and tests.sirref.org. One piece that is missing, is a publishing workflow. I spent a little time building a YOCaml static-site generator for packaging OCaml libraries. Unfortunately, this was full of rabbit holes and busy work which was frustrating.

Tar-eio

June 1, 2026
Patrick Ferris

One such rabbit hole was tar-eio. It was in a bit of a sorry state, so I took the opportunity to improve the API for new users. Tarring a directory is now as simple as:

let foo = Eio.Path.(cwd / "foo") in
let foo_tar = Eio.Path.(cwd / "foo.tar") in
Eio.Path.with_open_out ~create:(`Or_truncate 0o755) foo_tar @@ fun foo_tar ->
Eio.Switch.run @@ fun sw ->
let t = Tar_eio.create ~sw foo in
Tar_eio.run t (Tar_eio.flow_of_file tar)

There was also a series of security issues that the tar library had that were fixed too, you can read about those in Thomas's weeknotes.

Shell forks revisited [weekly-2026-w17]

April 28, 2026
Patrick Ferris

Whilst I did get up to some other work last week, I decided to dedicate this weekly to fork-ing in the shell.

Must we fork?

April 28, 2026
Patrick Ferris

I have spent some time trying to figure out: can you implement a shell that never has to fork and run its own code (as opposed to fork+exec-ing, which is necessary)? There are, at least, two bits of POSIX shell semantics that make this hard: concurrency and redirects, particularly in the presence of shell built-ins.

Individual commands in a pipeline, according to the POSIX specification, will have their standard input and output connected up before other redirections.

The standard input, standard output, or both of a command shall be considered to be assigned by the pipeline before any redirection specified by redirection operators that are part of the command.

Additionally:

...each command of a multi-command pipeline is in a subshell environment; as an extension, however, any or all commands in a pipeline may be executed in the current environment.

Where a subshell is defined as:

Subshell: A shell execution environment, distinguished from the main or current shell execution environment.

We will note that subshell $\neq$ child process. In practice, however, the two are used fairly interchangeably given the semantics of spawning a child process. Perhaps the two most crucial aspects of a given command in a subshell are that:

It is run in parallel with the other commands.
It should be reentrant.

In what follows we will assume that we are discussing a multi-command pipeline $p$ .

$\texttt {p} = \texttt {cmd}_1 | ... | \texttt {cmd}_n \text { where } n \geq 2$

Pipeline parallelism

April 28, 2026
Patrick Ferris

The POSIX specification does not mandate parallelism explicitly for commands in a pipeline. However, in practice the semantics of a pipeline necessitate concurrency at the very least.

Suppose we have $\texttt {cmd}_1 | \texttt {cmd}_2$ where $\texttt {cmd}_1$ produces an infinite amount of output (e.g., cat /dev/urandom) and $\texttt {cmd}_2$ consumes a finite amount of input (e.g., hxd-xxd -l100). What is the result of running this pipeline with the default POSIX shell options?

$ dash -c "cat /dev/urandom | hxd.xxd -l100"
00000000: 5033 7981 ae44 d475 074c 5361 0a9e 3828  P3y..D.u.LSa..8(
00000010: f88c 1f9e 3b0e bcde 528a f78c b1e4 7195  ....;...R.....q.
00000020: 16f2 8325 c90b 9310 539c af04 8289 c3a0  ...%....S.......
00000030: 57c5 680a 2c81 a999 71b6 708e 95df e720  W.h.,...q.p....
00000040: 004c 407a 8365 9448 65c6 b435 bed2 4084  .L@z.e.He..5..@.
00000050: 62ea 96ab 2b23 0385 f9d2 fc30 d090 1719  b...+#.....0....
00000060: 68d1 c514                                h...
cat: write error: Broken pipe

There are three factors that influence the pipeline's execution behaviour (i.e. what is done about the commands that are run and the exit status we return).

pipefail: With pipefail set to false (the default), the exit status of the pipeline is entirely decided upon by the exit status of the rightmost command (above $\texttt {cmd}_2$ ). With pipefail, the pipe will exit with 0 if all commands exit with 0; otherwise the non-zero exit status of the rightmost command is taken.
!: If the pipeline begins with ! then this inverts the exit code in the way you might expect.
All commands must terminate.

Whilst in theory $\texttt {cmd}_1$ does not terminate, the termination of $\texttt {cmd}_2$ closes the reading end of the pipe resulting in the Broken pipe system error.

Let's replace $\texttt {cmd}_1$ with the following compound command that also produces, left to its own devices, infinite output on stdout.

while echo "some bytes"; do
  :
done | hxd.xxd -l100

In this example, we no longer have the luxury of being able to fork and exec. The compound command (the while loop) forces our hand -- we must rely on the internal execution logic of the shell to keep it going. How then do we prevent $\texttt {cmd}_1$ from blocking our shell from spawning $\texttt {cmd}_2$ ?

For most shells, this is where they might fork themselves and allow the while loop to execute in the child. The parent continuing on. Alternatively, the work could continue on in a separate thread (in OCaml parlance, domain). Fibers (green threads, coroutines etc.) are only possible if there are a sufficient number of co-operative yields (which can be hard to track: any control flow or shell built-in would need to ensure they periodically yield to the other commands in the pipeline).

With that being said, it would seem we have workarounds to ensure the parallel aspects of pipelines are possible to build without forking the shell state itself.

Command Reentrancy

April 28, 2026
Patrick Ferris

Command reentrancy deals with the fact that commands in a pipeline run in subshells. In practice, this means any $\texttt {cmd}_i$ should not impact the execution environment of any other $\texttt {cmd}_j$ . Commands may certainly interfere with one another: a rogue rm might remove a file that some other command was about to read or was in the middle of reading!

$\S 2.13$ of the POSIX shell specification describes the execution environment. In Merry we lift this into the ctx type making it completely immutable; the only way to update the context is to destructively declare new fields, e.g., { ctx with cwd = "/home/bactrian" }. All shell built-ins work directly from the context as opposed to any state that might also be kept by the process itself (e.g., we do not call getcwd(3) to expand PWD, but instead read it from the context).

Most state is handled nicely by this functional implementation of the shell; except open (or not) file descriptors. The exec built-in is particularly thorny in this regard:

If exec is specified with no operands, any redirections associated with the exec command shall be made in the current shell execution environment.

So, exec 3>&1 says "in the current execution environment map file descriptor 3 to stdout" and exec > /dev/null says "in the current execution environment map stdout to /dev/null. This is often used to silence anything writing to stdout.

info () {
  echo "INFO: $1" >&3
}

exec 3>&1 > /dev/null
info "Starting up server..."
echo "Some noisy program spamming STDOUT"
info "All is quiet..."

If we are not running shell built-ins inside new child processes, then we must lift the file descriptor table into the context as well and disallow any two built-ins to run in parallel. Consider this example:

exec < ./hello.txt

while read line; do
  echo "Got $line" >&2
done | (exec < ./world.txt; cat)

If we allow exec to change the set of open file descriptors of the shell process directly, then individual commands in the pipeline are no longer re-entrant. The pipeline here is not being used to pipe bytes from one command to another, but instead to expose concurrency and the subshell behaviour.

The second exec < ./world.txt has no impact on the currently opened file on stdin, the while loop continues to read ./hello.txt. This is straightforward to implement with fork as you can dup2 any files you need. The bookkeeping and isolation is all provided by the POSIX process semantics. Without fork, the file descriptors would have to be saved and restored on every suspension and resumption to provide an illusion of isolation.

We could relax our rule to "disallow any two built-ins to run in parallel" by taking a lock on the file descriptor table, but considering the lock is held from the point of resumption until the next suspension it might not achieve very much in practice.

Some of the intricacies of scheduling are also covered in Michael Greenberg's An executable and formal semantics of the POSIX shell.

To motivate the question, consider the following two pipelines:
(1) while true; do echo 5; done | true
(2) while true; do echo 5; done | { read x; echo $((x+42)); }
Both pipelines spawn two processes, both of which use shell builtins exclusively: neither of these pipelines needs to make an execve system call (though some systems may implement true or echo as executables, Smoosh and most shells build them in).

Scheduling is an opaque part of the POSIX specification. Indeed, as we have argued here, there is no semantics for scheduling. Here is a similar pipeline run through a script that straces for clone and fork calls on various shells:

% ./test/check_fork.sh "while echo 5; do : ; done | true"
bash  3
dash  3
yash  1 
zsh   1
ksh   1
ash   3

It might make an interesting use case for effects in OCaml -- whilst we can certainly piggyback on Eio's existing suspension mechanisms, this is slightly different in that we need to ensure some preconditions are true before resuming certain computations. Fiber-local state is insufficient as programs implicitly depend on global state of the process (in particular the FD table). In this sense, we need some form of suspension-resumption hooks.

OCaml Roundup: April 2026 [ocaml-roundup-april-2026]

April 7, 2026
Patrick Ferris

In April, I worked on lots of different projects. Thankfully, things have been quiet on the ppxlib front (in part due to Nathan taking on some more work and being between releases of the OCaml compiler). I managed to improve some of our migration error logic and do a little spring-cleaning.

More Landlock Thoughts

April 7, 2026
Patrick Ferris

Elsewhere, I spent some time exploring Landlock and how it might be used with Eio. I think it is useful to down tools once in a while and try out some newfangled API or OS feature. Landlock is quite different to Capsicum (I had assumed they would be more similar). Whilst I enjoy the fine-grained control over access and capabilities, the manual process of constructing rules is a little cumbersome (and perhaps easy to make mistakes with). Combining that with [Eio]'s capability API might make that more tenable. The idea being you could have some Eio code like:

let () =
  Eio_linux.run @@ fun env ->
  let data = Eio.Path.(env#fs / "/data") in
  Eio.Path.with_open_dir sw data @@ fun data_dir ->
  match Eio_linux.Landlock.enter () with
  | Error (`Msg m) -> failwith msg
  | Ok () ->
    (* At this point, only certain default operations can happen to [data_dir].*)
    analysis data_dir

Opening the directory (in the openat-sense) adds a Landlock rule to the current context (some kind of process-local state). This rule would be removed whenever the data directory is closed (by hand or when the function returns, i.e., when the internal Eio.Switch releases its resources). Doing the bookkeeping internally means mistakes are less likely to be made whilst also improving, in my opinion, the API.

This breaks down for networking, unfortunately. Whilst the filesystem hierarchy provides a clean way to produce capabilities (a handle on a directory + openat + RESOLVE_BENEATH), networking does not. The Landlock API, at the time of writing, only allows rules based on port numbers. The system API is completely ambient (socket + connect). Eio's API provides access to the network through a single network `capability. All of these mechanisms are somewhat at odds. What does a network look like if it adheres to the ideas we have for combining Landlock and Eio (like in the filesystem example)?

The API becomes port-oriented. We're already at odds with Unix domain sockets, but we will carry on regardless. A capability on a port means you can listen on that port and accept connections, or you could connect to it and send data. At the time of writing, Landlock is restricted to TCP ports. In Eio, there is no API to with_open_port, i.e., to register your intent to do something with that port -- you either listen or connect.

All of this starts feeling a lot like Plan 9; something which I hope to explore further soon.

Weekly logs

April 7, 2026
Patrick Ferris

The full details are in the weeklies transcluded below:

Moving away from Github [weekly-2026-w15]

April 13, 2026
Patrick Ferris

A short week for me as I was hiking with family in and around Porthmadog, Wales.

Reliance on Github

April 13, 2026
Patrick Ferris

I joined Github in 2016, but only began using it in earnest (according to my Github activity) in July 2018: right after Microsoft announced its intent to acquire Github; and just before they closed the $7.5bn deal. I originally joined to boost my résumé and I hoped to receive the odd open-source contribution to projects I was working on (like ppx_deriving_yaml). Since then, I have used Github extensively both at work (Tarides) and for more personal projects.

There have been many reasons to jump ship from Github ever since the Microsoft acquisition (e.g. their planned (and quickly unplanned) announcement to charge self-hosted Github runners). Ultimately, I was convinced when I noticed the new AI tool on the homepage with options to select your "versatile and highly intelligent" model. There is no need for me to stay on the platform for my personal projects. AI integration into Github has nullified those earlier motivations for hosting my projects on the site. The number of contributions has increased, but many of them fix superficial elements of an underlying bug. In my (maybe too uncharitable) imagination, it is all too easy for contributors to prompt their built-in LLM to "fix this broken unit test" and for the fix to layer a single-use workaround on top, instead of identifying and correcting the relevant code. This is, at least, what it looks like to me when I receive a new review request on my open-source projects.

At the end of last week I set up my own Git infrastructure following Ryan's nixos setup. In the future, the source of truth for my projects will be git.sirref.org (I will likely continue to mirror to tangled.org or Github).

Reduce, Reuse, Recycle

April 13, 2026
Patrick Ferris

Part of self-hosting your own source code is managing some form of automatic testing. I hesitate to describe it as "continuous integration" as the term is confusing and overloaded. Luckily, parts of the OCaml community have been building test infrastructure for a while.

I have a proclivity for rebuilding things from scratch be it to learn or to build something the way I want. It is a proclivity I am trying to reign in as I find less and less time to hack on projects. I think this is a good thing. Reusing and recycling existing technologies, in earnest, feels like a pushback on the not-invented-here syndrome that seems to be snowballing thanks to LLMs and code generation tools.

To that end, I dusted off my OCurrent knowledge, and repurposed the "local" mode of OCaml-CI to work with my git repositories. After battling through nested layers of git submodules, I managed to Nix-ify this and get it running at tests.sirref.org. I needed to add multi-repository support as well as Current.Monitors for the local git refs. By no means perfect, but it is nice to own the testing infrastructure (and the cost of downloading OCaml images and the compute to run the tests).

Landlocked Eio [weekly-2026-w16]

April 18, 2026
Patrick Ferris

I spent the start of last week wrapping up the "local" CI I mentioned the previous week. Unfortunately it was mostly a packaging nightmare, followed by missing features in ocaml-ci-local. It would be great to have a richer CLI tool for running tests on a directory of git repositories, but unfortunately most of the existing tooling has prioritised the Github/Gitlab etc. APIs.

After coming across minimal during the week, I took it as an opportunity to learn a little more about Linux's Landlock mechanism.

Landlocked Eio [landlocked-eio]

April 18, 2026
Patrick Ferris

Landlock is a sandboxing mechanism that has existed in the Linux kernel since 5.13. It allows unprivileged processes a means to restrict their ambient rights (e.g. creating a file or opening a network connection). These rights are ambient in the ambient authority sense, i.e., any process can attempt to perform an action without needing access to any particular capability to do so.

Ambient systems are often contrasted with capability-based ones. In terms of operating systems, this is most commonly illustrated with access-control-lists-as-columns and capabilities-as-rows in an access matrix.

Eio is a capability-based library in OCaml -- resources are provided at the start of the application and can be passed around wherever they are needed. The design is covered in Thomas Leonard's lambda capabilities blog post. Unfortunately, any application (willingly or via a third-party library) may ambiently access OS resources. In OCaml, this may be by using the standard library functions (e.g. Sys.readdir), things from the Unix module, a C FFI function or by spawning a subprocess.

I had a go at "landlocking" Eio applications last week after stumbling across minimal's hakoniwa library for process isolation (which is very reminiscent of void processes).

module Landlock = Eio_linux.Landlock
let ( / ) = Eio.Path.( / )

let run_eio file =
  Eio.traceln "Writing to file";
  Eio.Path.save ~create:(`Exclusive 0o644) file "hello, world!";
  Eio.traceln "Reading file: %s" (Eio.Path.load file);
  Eio.Path.unlink file

let bad_program () =
  In_channel.with_open_bin "/etc/passwd" In_channel.input_all
  |> Eio.traceln "/etc/passwd: %s"

let () =
  Eio_linux.run @@ fun env ->
  Eio.Path.with_open_dir (Eio.Stdenv.cwd env / ".") @@ fun dir ->
  let parent_fd = 
    Eio_unix.Resource.fd_opt (fst dir) |> Option.get
  in
  let file = dir / "hello.txt" in
  let rules =
    Landlock.Rule.
      [
        path_beneath ~parent_fd
          ~flags:Flags.Fs.(
            write_file + read_file + make_reg + remove_file
           );
      ]
  in
  match Landlock.enter rules with
  | Ok () ->
      run_eio file;
      bad_program ()
  | Error `Not_supported -> Eio.traceln "Landlock not supported!"

Running this program (with the necessary privileges) prints the following.

+Writing to file
+Reading file: hello, world!
Fatal error: exception Sys_error("/etc/passwd: Permission denied")

Eio already has a similar function for the BSDs using capsicum. In capsicum, processes enter a capability mode:

cap_enter() places the current process into capability mode, a mode of execution in which processes may only issue system calls operating on file descriptors or reading limited global system state.

This plays nicely with Eio's capabilities. Within the scope of Eio.Path.with_open_dir, the directory will remain usable after entering "cap" mode. This is not the case with landlock unless you add a specific path_beneath rule to the ruleset (like in the example above). However, you could imagine plumbing the necessary information throughout the Eio_linux backend for it to build a set of rules that can be applied at any given moment in order to mimic the capsicum semantics.

Self-hosting Music [self-host-music]

April 18, 2026
Patrick Ferris

After my friend, Greta, sent me some articles about music streaming services and their exploitation of artists, I decided enough was enough and setup navidrome. It is compatible with Subsonic and so far I'm very pleased to be streaming my own music to my devices. I am using substreamer on iOS. On my laptop, I tried out Andreas Rossberg's Camp

What you got here is an old-school music player heavily inspired by good old Winamp [1], with a focus on decent music library and playlist handling.

I took Anil's OCaml Installer tool for a spin with great success... after a few mishaps mostly related to system dependencies.

$ nix-shell -p pkg-config gcc pulseaudio miniaudio libxi libxrandr libxinerama libffi mesa libxcursor libGL 
$ oi-linux-x86_64 run --with=https://github.com/patricoferris/camp#nix -- camp

The fixes in my fork of Camp are just to install the assets using dune and also statically provide the -lpulse flag (which should probably be dynamically picked up). Otherwise, raylib or miniaudio selects the Null playback device. Roll-on cross-ecosystem package management.

Shell forks revisited [weekly-2026-w17]

April 28, 2026
Patrick Ferris

Whilst I did get up to some other work last week, I decided to dedicate this weekly to fork-ing in the shell.

Must we fork?

April 28, 2026
Patrick Ferris

Individual commands in a pipeline, according to the POSIX specification, will have their standard input and output connected up before other redirections.

The standard input, standard output, or both of a command shall be considered to be assigned by the pipeline before any redirection specified by redirection operators that are part of the command.

Additionally:

...each command of a multi-command pipeline is in a subshell environment; as an extension, however, any or all commands in a pipeline may be executed in the current environment.

Where a subshell is defined as:

Subshell: A shell execution environment, distinguished from the main or current shell execution environment.

It is run in parallel with the other commands.
It should be reentrant.

In what follows we will assume that we are discussing a multi-command pipeline $p$ .

$\texttt {p} = \texttt {cmd}_1 | ... | \texttt {cmd}_n \text { where } n \geq 2$

Pipeline parallelism

April 28, 2026
Patrick Ferris

The POSIX specification does not mandate parallelism explicitly for commands in a pipeline. However, in practice the semantics of a pipeline necessitate concurrency at the very least.

$ dash -c "cat /dev/urandom | hxd.xxd -l100"
00000000: 5033 7981 ae44 d475 074c 5361 0a9e 3828  P3y..D.u.LSa..8(
00000010: f88c 1f9e 3b0e bcde 528a f78c b1e4 7195  ....;...R.....q.
00000020: 16f2 8325 c90b 9310 539c af04 8289 c3a0  ...%....S.......
00000030: 57c5 680a 2c81 a999 71b6 708e 95df e720  W.h.,...q.p....
00000040: 004c 407a 8365 9448 65c6 b435 bed2 4084  .L@z.e.He..5..@.
00000050: 62ea 96ab 2b23 0385 f9d2 fc30 d090 1719  b...+#.....0....
00000060: 68d1 c514                                h...
cat: write error: Broken pipe

There are three factors that influence the pipeline's execution behaviour (i.e. what is done about the commands that are run and the exit status we return).

pipefail: With pipefail set to false (the default), the exit status of the pipeline is entirely decided upon by the exit status of the rightmost command (above $\texttt {cmd}_2$ ). With pipefail, the pipe will exit with 0 if all commands exit with 0; otherwise the non-zero exit status of the rightmost command is taken.
!: If the pipeline begins with ! then this inverts the exit code in the way you might expect.
All commands must terminate.

Whilst in theory $\texttt {cmd}_1$ does not terminate, the termination of $\texttt {cmd}_2$ closes the reading end of the pipe resulting in the Broken pipe system error.

Let's replace $\texttt {cmd}_1$ with the following compound command that also produces, left to its own devices, infinite output on stdout.

while echo "some bytes"; do
  :
done | hxd.xxd -l100

With that being said, it would seem we have workarounds to ensure the parallel aspects of pipelines are possible to build without forking the shell state itself.

Command Reentrancy

April 28, 2026
Patrick Ferris

Most state is handled nicely by this functional implementation of the shell; except open (or not) file descriptors. The exec built-in is particularly thorny in this regard:

If exec is specified with no operands, any redirections associated with the exec command shall be made in the current shell execution environment.

info () {
  echo "INFO: $1" >&3
}

exec 3>&1 > /dev/null
info "Starting up server..."
echo "Some noisy program spamming STDOUT"
info "All is quiet..."

exec < ./hello.txt

while read line; do
  echo "Got $line" >&2
done | (exec < ./world.txt; cat)

Some of the intricacies of scheduling are also covered in Michael Greenberg's An executable and formal semantics of the POSIX shell.

To motivate the question, consider the following two pipelines:
(1) while true; do echo 5; done | true
(2) while true; do echo 5; done | { read x; echo $((x+42)); }
Both pipelines spawn two processes, both of which use shell builtins exclusively: neither of these pipelines needs to make an execve system call (though some systems may implement true or echo as executables, Smoosh and most shells build them in).

% ./test/check_fork.sh "while echo 5; do : ; done | true"
bash  3
dash  3
yash  1 
zsh   1
ksh   1
ash   3

Forking in Shells & Library Maintenance [weekly-2026-w14]

April 3, 2026
Patrick Ferris

This week, Merry got close to executing apt-get install ca-certificates and running Debian's debootstrap scripts correctly. Nathan and I released Ppxlib 0.38.0, I added support to ocaml-proj for compiling to the browser and I fixed a long-standing bug in hilite.

Forking in Merry

April 3, 2026
Patrick Ferris

So far, Merry has managed to support many features of the POSIX shell specification without needing to do a fork(2) without an exec. Or, to put it another way, all the forking that Merry does, will only be followed by running C code. This is actually a feature of Eio's Process API. It is required to make process execution safe in the context of multiple domains in OCaml (mostly due to ensuring consistency within the garbage collector). However, it is perfectly okay to fork and run OCaml code provided there is only a single domain.

In a shell, many features of the shell language require that commands (or built-ins, function applications etc.) execute within a subshell. The specification does not go into detail about how shells should implement this, but many choose to fork in order to preserve some state in the parent (for example, the file descriptor table). Some shells try to minimise the number of forks as an optimisation. The korn shell (ksh) is one such shell.

Using the notation $(command) will cause command to execute in a subshell of the current ksh. In many instances, ksh will not actually fork/exec a subshell when command is a built-in or a shell function.

By trying to adhere to Multicore OCaml's "thou shalt not fork" commandment, Merry is much more similar to ksh in this regard. However, whenever there is an interaction with a shell built-in and semantics that need a child process, things get tricky very quickly. Consider the following:

while echo "hello"; do
  :
done | head -n 3

We have a mix of shell built-ins, normal commands, a compound command (the while loop) and a pipeline. Each individual command in the pipeline requires you to run them in a subshell and set them up before executing them. Without a fork for that first command, you may end up looping forever which is not the intended behaviour here. I had a good conversation with Michael about the implications of this for reproducibility.

For now, I have started implementing a fork for some of the shell features whilst still trying to maintain the functional core of the implementation.

An OCaml 5.5 compatible Ppxlib

April 3, 2026
Patrick Ferris

Nathan and I released an OCaml 5.5 compatible version of ppxlib.

Ppxlib Releases › 0.38.0 [ppxlib-release-0-38-0]

March 30, 2026
Patrick Ferris

Nathan and I released ppxlib.0.38.0 last week. Its main feature is full migration support for the upcoming OCaml 5.5 compiler (currently in its alpha3 release). This means supporting the handful of new features landing in OCaml 5.5: modular explicits, external type declarations and arbitrary "local" structure items.

As Nathan and I continue to find a plausible maintenance story for ppxlib, Nathan has opened an issue on the OCaml compiler to discuss the idea of adding additional extension points to the language.

This comes from the desire to be able to store encoded versions of new OCaml features inside older abstract syntax trees.

There are also some nice bug fixes in there too:

A potential OOM loop has now been removed.
Locations have been restored to long identifiers!

Browser support for ocaml-proj [ocaml-proj-virt]

April 3, 2026
Patrick Ferris

A while ago I built some "modern" bindings to PROJ4 in OCaml. After reading Jon Ludlam's weeknotes (and speaking with him and Anil, I thought it might be nice to add a Javascript backend to those bindings). This was relatively straight-forward using Dune's virtual libraries and is available on Github.

With virtual libraries, your own data analysis could (if you wished) depend solely on the proj library and later choose to either link it with proj.c or proj.js depending on where the analysis is being deployed.

I briefly looked at WASM support, but quickly realised there was not much appetite for it and trying to compile around the C FFI was going to be hard.

Hilite updates

April 3, 2026
Patrick Ferris

Since I helped relaunch the ocaml.org website a few years ago, I have maintained hilite, a tool for doing build-time syntax highligting.

It is being used on the ocaml.org website, in yocaml_markdown and also in Michael's webplats (when it works...).

And it was Michael who led me to finally fixing and adding syntax highlighting support for Python and Go. Once I was on this roll, I finally took a look at fixing the long-standing bug of trying to highlight ocaml-mdx code, like the following:

# 50.123;;
# 1 "file.ml";;

In fact, this code now highlights fine, but was the source of the bug. The syntax highlighting grammer confuses the ocaml-mdx # as a toplevel directive and trouble ensues.

Ppxlib release and Merry updates [weekly-2026-w13]

March 30, 2026
Patrick Ferris

Ppxlib Releases › 0.38.0 [ppxlib-release-0-38-0]

March 30, 2026
Patrick Ferris

As Nathan and I continue to find a plausible maintenance story for ppxlib, Nathan has opened an issue on the OCaml compiler to discuss the idea of adding additional extension points to the language.

This comes from the desire to be able to store encoded versions of new OCaml features inside older abstract syntax trees.

There are also some nice bug fixes in there too:

A potential OOM loop has now been removed.
Locations have been restored to long identifiers!

Merry updates

March 30, 2026
Patrick Ferris

It turns out writing a POSIX(ish) shell is hard; at least, there is a vast number of slightly obscure features that one needs to support. I think this is surprising because most people writing shell scripts write simple shell scripts; scripts that use a much smaller subset of features. In the same breath, those same developers are likely to use something like apt-get install bash which runs a plethora of more advanced (and non-POSIX) shell scripts!

Exec redirects

March 30, 2026
Patrick Ferris

In a shell script, you can use the built-in command exec to replace the current process with a new one (e.g. exec vim). However, there is a different mode of operation for exec:

If exec is specified with no operands, any redirections associated with the exec command shall be made in the current shell execution environment.

One of my litmus tests for Merry is the Debian debootstrap scripts (h/t Anil). One thing that it does is the following:

err () {
  printf "err $1" >&4
}

exec 4>&1

A lot of detail has been elided for clarity. exec 4>&1 sets up a redirection for the shell's execution environment in which file descriptor 4 is now an alias for standard output. So, writing to 4 (by redirecting a command's standard output to 4 i.e. >&4) will output to where standard output is going (most likely the terminal).

For this to work, shell's must dup2 the relevant file descriptors which have the following condition:

If the file descriptor newfd was previously open, it is closed before being reused; the close is performed silently (i.e., any errors during the close are not reported by dup2()).

This is all very well, unless your program has an important file already open that happens to have file descriptor 4. The Eio Linux and POSIX backends suffer from this problem. Both make use of a file-based synchronisation mechanism for waking up the eventloop should another domain push a completion to the scheduler's run queue. On Linux this is via an eventfd and in POSIX, a pipe.

This dup2 will close the eventfd and will likely grind Eio to a halt (or an assert false). For now, I have resorted to vendoring Eio and moving the eventfd file descriptors to higher values, though I have opened a PR to make this more configurable. This bug is quite easy to write up... it was not so easy to find!

As a random example, consider the debconf/confmodule script which offers very little room for a buggy implementation!

Outreachy

March 30, 2026
Patrick Ferris

The presentations from the demonstrations for this round of Outreachy are now online!

Write support in OCaml TIFF library [outreachy-ocaml-tiff]

February 9, 2026
Patrick Ferris

I am mentoring Tambe Salome during the December 2025 Outreachy round to add support for writing TIFF files in the ocaml-tiff library.

You can now see the video of the demonstration day presentation:

A POSIX Shell in OCaml [weekly-2026-w12]

March 19, 2026
Patrick Ferris

Long time, no weekly. Since the start of this year I have been building a POSIX shell in OCaml called msh (with the underlying library called Merry). Merry is available online now.

A POSIX(ish) shell in OCaml

March 19, 2026
Patrick Ferris

Merry is a POSIX(ish) in OCaml. It uses Eio alongside Morbig (a static parser for POSIX shell).

Why another (POSIX) shell?

March 19, 2026
Patrick Ferris

Shells have been around for a long time. In my research on the notion of metashell I wrote about Louis Pouzin originally coining the term:

Metashells › Louis Pouzin's "SHELL" [pouzin-shell]

May 19, 2025
Patrick Ferris

I spent some time reading part of the multics design documentation this week. Louis Pouzin coined the term "SHELL" in this document, and I was reminded yet again just how important it is to be a good writer even as a "computer science researcher". For example, this excerpt from the requirements section of the document

The previous definitions imply that a command MUST be designed while keeping in mind the user, sitting at his console, wondering about what might be going on, mistyping or forgetting arguments, even if fully aware of the conventions, and possibly interfering with the command by hasty quits, carriage returns, and other temperamental reactions.

And then later, when defining the "SHELL".

We may envision a common procedure called automatically by the supervisor whenever a user types in some message at his console, at a time when he has no other process in active execution under console control (presently called command level). This procedure acts as an interface between console messages and subroutine. The purpose of such a procedure is to create a medium of exchange into which one could activate any procedure, if it were called inside of another program. Hereafter, for simplification, we shall refer to that procedure as the "SHELL".

It still surprises how little the undergraduate degree in computer science at Cambridge focuses on writing skills.

I built Shelter exploring the idea of a shell-like interface that allowed users to undo their shell actions (amongst other cool tricks). Unfortunately shell-like is not enough. Shelter cut many corners to masquerade as a shell (e.g. appending env to understand how a command may have altered the execution environment). I felt it was necessary to make Shelter a "SHELL"! To do that, I needed a solid foundation to build on.

msh, the POSIX shell that comes with Merry, is by no means POSIX-complete in terms of features. But it is my daily driver at this point. It includes a pure OCaml rewrite of linenoise (a small, self-contained alternative to the venerable readline) called bruit.

If you have docker installed on your machine, you can take it for a spin today:

docker run -it --rm patrickferris/msh

The patrickferris/msh docker image is just for trying it out. It is based on the OCaml 5.3 alpine image.

Alternatively, you can build msh from source and have it available in your opam switch.

opam pin git+https://tangled.org/patrick.sirref.org/merry

There are many small paper cuts left to patch over, but most of it is porcelain (e.g. ctrl+left-arrow for moving in bruit). Unfortunately, these are the kinds of things that you will immediately stumble upon.

What makes Merry different?

March 19, 2026
Patrick Ferris

Nothing.

Merry is supposed to be a solid, POSIX-ish base to build on. Unfortunaltely, as it turns out, the subset of features from the POSIX specification that people actually use... is pretty much all of it. Every possible redirection, variable expansion, shell built-in and compound command make some appearance. Not to mention the non-POSIX bits of shell we all take for granted (e.g. &>-redirection).

Merry and msh are useable. You will most likely find bugs if you use them. If, when using msh, you find something obscure happening you can enable debug mode either my setting the variable MSH_DEBUG or by invoking msh with -v -v.

What's next?

March 19, 2026
Patrick Ferris

It is soon time to combine Shelter and Merry into the time-travelling, POSIX-ish shell that I have been trying to build since I first started working on Shelter.

For this to be successful, I need Merry to be able to pass plenty of tests and right now that involves trying to install plenty of packages using tools like apk and apt.

Shell MRDT

March 19, 2026
Patrick Ferris

What I am particularly interested in reasoning about, is the execution context in Merry. This value, alongside the file-system, constitutes a fairly deep understanding of the state that changes in each step of a shell's evaluation loop.

This was, in terms of Shelter, the missing piece for truly building some kind of MRDT across shell sessions.

TIFF in OCaml

March 19, 2026
Patrick Ferris

I picked up from the excellent Outreachy work of Tambe Salome in getting write-support in ocaml-tiff. We are getting closer to the kind of API I envisaged in this latest round of refinement and review.

let ( / ) = Eio.Path.( / )

let checkerboard ~size =
  let v = Nx.zeros Nx.uint8 [| size; size |] in
  for row = 0 to size - 1 do
    for col = 0 to size - 1 do
      if (row + col) mod 2 = 0 then Nx.set_item [ row; col ] 254 v
    done
  done;
  Nx.to_bigarray v

let () =
  Eio_posix.run @@ fun env ->
  Tiff_eio.with_open_out (env#cwd / "example.tiff") @@ fun w ->
  let tif = Tiff.make (checkerboard ~size:256) in
  Tiff.to_file tif w

I have also been extremely pleased to see further external collaborators appearing:

Nicolas helping out with metadata maintainence.
Virgile adding support for reading multi-image TIFF files.

Fellowship Roundup [fellowship-roundup]

February 9, 2026
Patrick Ferris

Overview

February 9, 2026
Patrick Ferris

February is the final month of my current six-month fellowship work with Tarides. A lot of the work can be summarised by reading the roundups (see October, November, December and January). I have been writing throughout the fellowship. However, they do not quite capture all the work I have been doing.

Below, I pick individual projects and expand on them more holistically rather than pointing at individual PRs or issues.

Ppxlib

February 9, 2026
Patrick Ferris

Most of my time over the past six months has been devoted to ppxlib maintenance. A large proportion of this time has been solo development work. Ppxlib is a cornerstone library of the OCaml ecosystem (whether people like it or not).

$ opam list --depends-on=ppxlib --recursive | wc -l
2030

Moreover, it is completely invaluable to Jane Street too. Like other AST-based tools (e.g. ocamlformat, merlin), ppxlib must follow a regular release schedule to support newer compilers. During the years of multicore OCaml development, ppxlib maintenance was a little easier as the Parsetree rarely changed. However, since then, the same cannot be said. Since OCaml 5.2, here are just some of the Parsetree changes:

Functions are represented according to their arity.
Local module opens on types.
Effect syntax.
Modular explicits.
External types.
Locations for all parts of long identifiers.
Labeled tuples.

In a perfect world, each of these require a codec for serialising the feature into ASTs that do not support the feature, plenty of tests and new Ast_builder/Ast_pattern functions for using the feature. Nathan and I have been managing pretty well I would say, though we are not getting any technical debt work done.

Outreachy

February 9, 2026
Patrick Ferris

As the OCaml coordinator for Outreachy I have been running the latest December 2025 round. We are lucky to have four projects on the go!

Thibaut Mattio is mentoring two projects: an ML dashboard for Raven and an OxCaml backend for Raven.
Xvw is mentoring a Yocaml project.
I am mentoring a Geocaml project.

Eio

February 9, 2026
Patrick Ferris

During my work on tools like Shelter I have made a few improvements to Eio including:

OxCaml

February 9, 2026
Patrick Ferris

I have enjoyed my work on OxCaml went it has happend. I think a few tools have been useful including the try-oxcaml work. David and I also had fun at ICFP discussing some quick experiments related to io_uring and OxCaml.

One road-blocker here is the comparative amount of time spent trying to keep the OxCaml ecosystem working. Given my limited time working on fellowship projects, the rate of change of OxCaml (including the opam ecosystem) meant most of the time was spent making it just work, not exploring or experimenting with the features themselves. I think this has got better in recent months and perhaps jumping back in I would be surprised at the progress that has been made there.

Forester

February 9, 2026
Patrick Ferris

Tarides are funding some work on forester. This site uses forester by way of graft and I have spent some time working on these tools during the fellowship.

This includes a bytesrw-based bibtex library in OCaml that could be released soon.

Papers

February 9, 2026
Patrick Ferris

I had the great fortune of attending ICFP where I presented two talks and was co-author on a few others too.

Reference. Functional Networking for Millions of Docker Desktops (Experience Report) [madhavapeddy2025docker]

2025
Anil Madhavapeddy, David J. Scott, Patrick Ferris, Ryan T. Gibb, Thomas Gazagnaire
10.1145/3747525
https://dl.acm.org/doi/10.1145/3747525

Abstract

2025
Anil Madhavapeddy, David J. Scott, Patrick Ferris, Ryan T. Gibb, Thomas Gazagnaire

Docker is a developer tool used by millions of developers to build, share and run software stacks. The Docker Desktop clients for Mac and Windows have long used a novel combination of virtualisation and OCaml unikernels to seamlessly run Linux containers on these non-Linux hosts. We reflect on a decade of shipping this functional OCaml code into production across hundreds of millions of developer desktops, and discuss the lessons learnt from our experiences in integrating OCaml deeply into the container architecture that now drives much of the global cloud. We conclude by observing just how good a fit for systems programming that the unikernel approach has been, particularly when combined with the OCaml module and type system.

Reference. What we talk about when we talk about scientific programming [ferris2025scientific_programming]

2025
Patrick Ferris
https://watch.eeg.cl.cam.ac.uk/w/aYXqXLtgQawYMjVXjSQtjx

Abstract

2025
Patrick Ferris

Programming for the planet undoubtedly involves programming scientifically, but what kind of programming are we talking about and what makes it scientific? In what ways does it differ from other forms of programming, if at all? Is scientific programming, data science or machine learning fundamentally different to constructing a compiler or building a high-throughput web server? By considering how the scientific method (with its falsifiable hypotheses and repeatable and reproducible experiments) relates to scientific programming, I hope to explore how computer science and traditional programming techniques are coming up short in meeting the requirements of scientific programmers.

Reference. Generating a corpus of Hazel programs from ill-typed OCaml programs [ferris2025hazel_of_ocaml]

2025
Patrick Ferris, Anil Madhavapeddy
https://patrick.sirref.org/var/tyde2025.pdf

Abstract

2025
Patrick Ferris, Anil Madhavapeddy

When developing a new programming language, having a large corpus of both correct and incorrect programs allows language designers to test and explore the capabilities of their new language. However, bootstrapping such a corpus of incorrect programs is time-consuming and arduous. We therefore explore how to reuse code from more mature languages to generate a corpus of ill-typed code for newer ones. We have developed a compiler to Hazel, an emerging language with typed holes, from the more mature OCaml ecosystem. We find it practical to generate a comprehensive corpus of ill-typed programs for Hazel development, and discuss future larger scale efforts towards bridging ecosystems.

Reference. Decomposable Type Highlighting for Bidirectional Type and Cast Systems [carroll2025decomposable_type_highlighting]

2025
Max Carroll, Anil Madhavapeddy, Patrick Ferris
https://maxcarroll0.github.io/papers/workshops/HATRA-decomposable-type-highlighting/

Abstract

2025
Max Carroll, Anil Madhavapeddy, Patrick Ferris

We explore how to provide programmers with an interactive interface for explaining the process by which static types and dynamic casts are derived, with the goal of improving the debugging of static and dynamic type errors. To this end, we define mathematical foundations for a decomposable highlighting system within a bidirectional system, and show how these can be propagated through dynamic types in a cast system. Our prototype implementation in the gradually typed Hazel language includes a web-based user interface, through which we highlight the importance of type level debugging.

Reference. Yirgacheffe: A Declarative Approach to Geospatial Data [dales2025yirgacheffe]

2025
Michael Winston Dales, Alison Eyres, Patrick Ferris, Francesca A. Ridley, Simon Tarr, Anil Madhavapeddy
10.1145/3759536.3763806
https://dl.acm.org/doi/10.1145/3759536.3763806

Abstract

2025
Michael Winston Dales, Alison Eyres, Patrick Ferris, Francesca A. Ridley, Simon Tarr, Anil Madhavapeddy

We present Yirgacheffe, a declarative geospatial library that allows spatial algorithms to be implemented concisely, supports parallel execution, and avoids common errors by automatically handling data (large geospatial rasters) and resources (cores, memory, GPUs). Our primary user domain comprises ecologists, where a typical problem involves cleaning messy occurrence data, overlaying it over tiled rasters, combining layers, and deriving actionable insights from the results. We describe the successes of this approach towards driving key pipelines related to global biodiversity and describe the capability gaps that remain, hoping to motivate more research into geospatial domain-specific languages.

I had some great conversations with KC too with respect to my current research.

Future Work

February 9, 2026
Patrick Ferris

As I enter my final year of the PhD, my goal would be to try and align more of my fellowship work with my own research wherever it makes sense. This research focuses on the following hypothesis:

Embedding deep provenance tracking, reversible execution and mergeable histories directly into an interactive programming environment (a POSIX-like shell) greatly decreases the gap between exploratory scientific work and reproducible, publishable results; all whilst leaving existing workflows intact and being programming language agnostic.

I want to focus on building these tools using Irmin, Eio, OxCaml etc. There are other projects I wish to continue regardless, like Outreachy, though just as a coordinator (not a mentor).

Overlapping PhD Work

February 9, 2026
Patrick Ferris

There are multiple projects that overlap with my work. For example, Irmin, which I wrote a retrospective for. This is at the core of Shelter and suggests an extremely valid use case for a brancheable, mergeable database.

In addition to this, continuing to work on Eio makes sense to me too. Eio currently has no maintainer and no active development, but the issues are piling up. In particular, I am making heavy use of Eio in Merry, a POSIX(ish) shell written in OCaml. Aside from research angles on brancheable and mergeable shells, I am also quite interested in how this might look in terms of OxCaml. I met with Anil and Thomas recently to discuss the future of Eio and there was some agreement that we are all heavy users of it and hope to keep up its maintenance and set of features.

One large piece of work that is sorely needed here is moving Eio to bytes and not bigarray-backed Cstructs. Any libraries using bytesrw (e.g. jsont) incur an extra copy of data into and out of the bytes. Anil has been working on some of the pieces recently to make this better!

Maintaining and Releasing my own libraries

February 9, 2026
Patrick Ferris

I have published quite a few OCaml libraries in my time, below is just a few libraries (filtered to not include OCurrent projects).

$ opam list --all --no-switch --columns=name,authors: | grep -E "sirref|Ferris" | cut -d " " -f1
carbon
geojson
geojsone
graft
hilite
ISO3166
jekyll-format
ppx_deriving_ezjsonm
ppx_deriving_yaml
rtree
search
topojson
topojsone
cid
multibase
multicodec
multihash
multihash-digestif

Some of these are used in the community including hilite, ppx_deriving_yaml and the multi-codecs. I would like to do some general maintenance of some of these tools and release a few iterations of graft which now makes use of bib.

Ppxlib Maintenance

February 9, 2026
Patrick Ferris

By far, maintenance of ppxlib takes up a majority of my time. I think, given the other work I need to focus on, it would be great to try to minimise the amount of time working on ppxlib. In particular, perhaps only maintaining ppxlib as far as OCaml release-readiness is a possibility and important bug fixes (like the OOM bug reported by Jane Street).

Forester, Graft and Writing

February 9, 2026
Patrick Ferris

With all of this work, I am eager to communicate what I am up to and my thoughts on open source, OCaml, OxCaml etc. I am glad to be using Forester to do this, particularly as it is written in OCaml (and could make great use of my own work on Eio).

I hope to continue working on Graft and related tools like bib. But I would also like to start writing some long-form content similar to the Irmin retrospective.

Merry [merry]

February 9, 2026
Patrick Ferris
https://tangled.org/patrick.sirref.org/merry

Merry is a library for building POSIX-ish shells in OCaml. It comes with a (work-in-progress) POSIX shell written using Eio called msh.

Supporting the full POSIX specification is no easy feat, however, msh already supports plenty of the more useful bits of the spec.

OCaml Weekly 2025 w32 to w35 [ocaml-weekly-2025-w32-w35]

September 2, 2025
Patrick Ferris

I have been working on a few different OCaml-related projects over the last few weeks. This is also coinciding with my partner and I moving across the UK, which has made finding time to write weeklies and posts a little tricky. Nevertheless, here are some of the things I have been thinking about and working on!

I managed to publish one signficant post this month: a retrospective on Irmin.

Eio

September 2, 2025
Patrick Ferris

Increasingly, I'm feeling the dream of a unified framework for asynchronous IO slipping through the OCaml community's fingers. It is perhaps not such a bad thing, and I think with the right library authoring we can at least get to a place where it isn't so bad, for example providing read functions as opposed to using an opinionated IO library directly.

That being said, I am a very happy user of Eio when those choices do not matter, as is the case in building your own application (e.g. Shelter). To this end, I have spent a good bit of time upstreaming support for various missing pieces in Eio's API including:

Setuid and setgid fork action's for the process API.
Set process group support for job control in the process API.
Responding to Buf_write.of_flow request, and tinkering with the example there. I think this does highlight the awkwardness of making code portable across concurrency mechanisms, particularly with Eio's structured concurrency.
I did some investigating into EINTR bug which seems to be stemming from a known-issue on Uring in that writes are not buffered which usually does not matter except perhaps when there are parallel writes to stdout.
Spent some time thinking about the fiber local storage across domains issue, I've passed on some thoughts to folks working on this.

Vpnkit

September 2, 2025
Patrick Ferris

You might recall I was interested in using vpnkit. Hannes has done an amazing amount of work (patching and releasing) a series of packages to get this into a place that is much better and could be considered soon for merging. This defunctorisation is actually very useful for the Eio port I wrote a long time ago.

Papers and Talks at ICFP

September 2, 2025
Patrick Ferris

Somehow, I have ended up on lots of papers and talks at ICFP and the co-located events in October. The vaguely OCaml-related ones include:

Essentially a Vpnkit Experience Report.
An extended abstract on generating a corpus of ill-typed Hazel programs was accepted into TyDe workshop.
Relatedly, the work that project supported was accepted into HATRA which was the Part II project I supervised: Decomposable Type Highlighting for Bidirectional Type and Cast Systems.
And two PROPL talks!

Outreachy

September 2, 2025
Patrick Ferris

We have come to the end of another Outreachy round! I will write more on this soon in its own separate post. But for now I am very grateful to this round's mentors gridbugs and mdales, and also our fantastic interns. If you are interested, please do watch our demo day presentations.

The next round is fast approaching and we still need to work out the logistics. But I had a good conversation with mdales about possible Geocaml projects that I intend to submit!

Quarterly OCaml Q2 [ocaml-quarterly-q2]

July 18, 2025
Patrick Ferris

Thanks to Tarides sponsorship, I get to work on open-source OCaml. This quarterly is a companion to my weeklies, summarising the last three months of development, peppered with ideas and thoughts about OCaml, its community and its future.

What I wanted to work on?

July 18, 2025
Patrick Ferris

There were two main things I hoped to continue working on: ppxlib and outreachy. These are projects that I was previously working on, and in the case of Outreachy I have now been involved for many years.

In addition to this, all of my Part II projects this year used OCaml in some regard. In general, I want to see more adoption of OCaml. Over the years this has taken many forms including my initial work on developing workflows for OCaml that just turned five years old. This directly fed into the rebranding and rethinking of ocaml.org itself.

What I worked on?

July 18, 2025
Patrick Ferris

Ppxlib

July 18, 2025
Patrick Ferris

Ppxlib is the de facto standard library for building OCaml preprocessors. At the time of writing, opam list --depends-on=ppxlib informs me that there are 267 reverse dependencies. Janestreet is a heavy user of ppxes and has authored many.

One of the main accomplishments this quarter was bumping the internal AST to 5.2. This allows ppx authors to use new OCaml language features in their ppxes. In bumping the AST, we knowingly broke compatability for pretty much every single reverse dependency. As best we can, we have been sending patches to ppx libraries and helping users migrate to the latest ppxlib.

There is an interesting dicussion to had about the nature of open-source, and its interaction with industrial monorepos and community tended package repositories. Package ecosystems thrive whenever there is a dedicated community creating, maintaining and publishing packages. The idea is that the published world should be healthy. The publishing medium can act as natural limiting factor in the churn of breaking changes (in the case of OCaml this is via PRs to the opam-repository). This, I have come to notice, reacts poorly to changes coming from internally consistent monorepos where introducing breaking changes is easily fixed by applying patches there and then. Whatsmore, OCaml is often stated as an incredibly safe language to perform large refactorings thanks to its type system.

Ppxlib sits awkwardly in the space of possible breaking changes. Tied to OCaml's parsetree, impacts of changes there ripple down to ppxlib. The compiler itself can remain internally consistent, and is protected as it need only parse source code. Ppxlib, on the other hand, exposes the parsetree to users and thus any changes to the parsetree will likely be felt by ppx authors. Since I started working on ppxlib, it feels as though the number of syntax changes has gone up (primarily from Janestreet work). Unless we make changes to how we provide support for these, maintainers of ppxlib will forever be stuck doing busy work! Thankfully, Nathan has thoughts on how to improve this.

There are a slew of other features I have added to ppxlib including:

Support for deriving from classes.
Support for deriving from module bindings and signatures.
Fixing compiler and ppxlib dummy locations.
Bumping to 5.3.
Migrations for 5.4.

Outreachy

July 18, 2025
Patrick Ferris

Our two projects this year, one on claudius and one on dune, are going extremely well. At the time of writing, we just had a mid-internship call to catch up, and I was blown away by the progress each intern has made. Unfortunately, Outreachy is struggling with funding and the next round is perhaps not going to happen. This is a real shame and I am hoping that it will not be the case.

Outreachy has been a wonderful source of new, committed OCaml developers. If you haven't already, do peruse the webpage on OCaml.org to see past internships or watch the demo day presentations. For the mentors involved, I believe it has also been a rewarding experience (though at times a challenging one). We are always looking for new mentors and project ideas, please do reach out to me if you are interested.

Hazel

July 18, 2025
Patrick Ferris

OCaml's feature set allows it to shine at writing programming languages (and things of that ilk: compilers, interpretters, static analysis tools). Hazel is a research programming language with typed holes written completely in OCaml (via the reason dialect).

Relating this back to the original intent of my work, to improve OCaml adoption, I believe this also means keeping existing users happy. In the last quarter I developed a compiler from OCaml to Hazel. Whilst new features are still being added to Hazel, I hope this could serve as a tool to help develop test-suites and standard library functions using existing OCaml solutions. In a student's work (Typed Debugging for Hazel), we used this tool to build a corpus of ill-typed Hazel programs to great effect.

OCaml should continue to be a world-class programming language for building other programming languages. I hope to upstream some of this work to Hazel and provide some low effort maintenance to help keep their compiler in good shape.

Systems Programming in OCaml

July 18, 2025
Patrick Ferris

In a cross-over with my own research, I have been developing many tools related to systems programming in OCaml including:

An eBPF-based open syscall tracing tool.
A library in OCaml for spawning void processes.
A shell session manager that uses Irmin to manage sessions. It is nice to see a new push to finally land the direct-style Irmin PR!

This work is means to develop the underlying libraries that support it. For example, I have opened a few PRs to Eio to add new "fork actions" to the spawn API. I also investigated the feasibility of changing the underlying mechanisms in Eio to use Picos. In the future, I think this could be important avoid further splitting the OCaml ecosystem.

OxCaml

July 18, 2025
Patrick Ferris

I dabbled a little with OxCaml and build try-oxcaml to let people take it for a spin without having to perform opam repository gymnastics. It turned into a lot of work to track down some pretty inane bugs (type definitions differeing between js_of_ocaml and the OxCaml compiler, resulting in different Javascript runtime representations...).

This unblocked a few of my colleagues to get OxCaml working on tools like odoc_notebooks and x-ocaml.

Forester

July 18, 2025
Patrick Ferris

A good proportion of my work this quarter has been focused on how to present the very work that I am doing. Jon Sterling has been developing a tool for scientific thought called Forester which seemed like a possible candidate for writing and sharing my work.

Porting my existing blog posts and website content from markdown to Forester's LaTeX-inspired syntax didn't seem like an option. In particular, many of my posts made use of additional markdown-based tools (like ocaml-mdx).

This lead to the development of Graft: a preprocessor for Forester forests, converting markdown and bibtex to trees.

What's next?

July 18, 2025
Patrick Ferris

So, I worked on most of what I wanted to work on and then some! Going forward I hope to keep maintaining ppxlib in some capacity and coordinating the OCaml community's Outreachy efforts.

I hope to continue my small experiments (e.g. converting Eio to Picos). My own research makes heavy use of Irmin, and I would be interested to help with that too. More recently, a new library has been released for working with numerical data in OCaml called Raven: I am interested to use this library in Geocaml, a suite of geospatial tools written in OCaml that I maintain.

I feel conflicted about the OxCaml efforts. I admit, I am not fully aware of the full benefits of the features, but I worry about some proliferation of modes that make the type system in OCaml unbearable to use. On top of this, with my ppxlib hat on, I worry about the impact of changing the compiler so frequently, placing strain on an already small community. That being said, by releasing OxCaml separately I do believe running it as an experimental set of packages will help understand the tool better. But I would not be building anything I intend to maintain or research with right now as that ecosystem is far too volatile.

Finally, over the past two years I have done a lot of teaching. From Part II projects to supervisions. Going into next (academic) year, I intend to reduce my in-person teaching to focus on my research. However, I am interested in producing more materials for learning, maybe some of this in OCaml, perhaps something similar to Beautiful Racket. I have tried in the past to do these sorts of things, for example The Irmin Book.

Thank you

July 18, 2025
Patrick Ferris

Thank you for reading this wrap up! And thank you again to Tarides for letting me work so freely on things that I think are good for the OCaml community.

Opentrace [open-trace]

May 19, 2025
Patrick Ferris
https://tangled.sh/@patrick.sirref.org/opentrace

Thanks to Koonwen's excellent libbpf bindings in OCaml, I have been building a little tool called opentrace to make it easier to track an executable's inputs and outputs.

This work was inspired my Michael's self-proclaimed "gross hack": pyshark. Whilst pyshark achieves its goals by injecting code into commonly used python objects and methods, opentrace uses eBPF. By using a lower-level API (hooks in the kernel), opentrace can remain programming language agnostic. However, less information is none about the user's intent compared to something like pyshark.

Monitoring the System

May 19, 2025
Patrick Ferris

opentrace has an all command that will trace the entire system.

$ sudo opentrace all --flags=O_WRONLY
pid,cgid,comm,kind,flags,mode,filename,return
16324,4417,".nheko-wrapped",openat,577,438,"/home/patrick/.cache/nheko/nheko/curl_alt_svc_cache.txt",47
16324,4417,".nheko-wrapped",openat,193,33188,"/home/patrick/.cache/nheko/nheko/cr3ZUHqBErIOe3PlwJ1SuU8zFKKxL12VzrRoHYMH.tmp",47
16324,4417,".nheko-wrapped",openat,577,438,"/home/patrick/.cache/nheko/nheko/curl_alt_svc_cache.txt",47
16324,4417,".nheko-wrapped",openat,193,33188,"/home/patrick/.cache/nheko/nheko/QfZTdZSuC56NdcDQ3aMxJc3BhMhAj8PmtYW1zFDP.tmp",47
2530,4235,"systemd",openat,524865,438,"/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/gammastep.service/cgroup.subtree_control",41
2530,4235,"systemd",openat,524545,0,"/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/gammastep.service/memory.min",41
2530,4235,"systemd",openat,524545,0,"/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/gammastep.service/memory.low",41
2530,4235,"systemd",openat,524545,0,"/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/gammastep.service/memory.high",41
2530,4235,"systemd",openat,524545,0,"/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/gammastep.service/memory.max",41

The --flags=O_WRONLY argument filters the events where the O_WRONLY flag was set in the call to open.

We also get the name of the current executable linked to the task (comm). The -wrapped is an artefact of using Nix.

Tracing an Executable

May 19, 2025
Patrick Ferris

The primary use case for this tool is to inspect what files your program might be reading and writing.

$ sudo opentrace exec --format=json --flags=O_CREAT -- opam list
$ cat trace.json | jq ".[] | .fname"
"/home/patrick/.opam/cshell/.opam-switch/packages/cache"
"/home/patrick/.opam/log/log-118747-29da3d.out"
"/home/patrick/.opam/log/log-118747-29da3d.err"
"/home/patrick/.opam/log/log-118747-29da3d.env"
"/home/patrick/.opam/log/log-118747-29da3d.info"

The "flags" argument can specify a small boolean formula for checking the open flags of a particular event with | (or), & (and), and ~ (not). Parentheses can be used for precedence.

$ sudo opentrace exec --flags="O_WRONLY|O_RDONLY" -- ocaml --version

Spawning Subprocesses

May 19, 2025
Patrick Ferris

One feature opentrace needs (in this proof-of-concept phase) is the ability to also trace subprocesses.

opentrace is primarily an eBPF program that is loaded into the kernel and communicates with an OCaml program. Events are communicated via a ring buffer and most of the post-processing happens in OCaml. To capture subprocesses, opentrace creates a new control group (cgroup) and places the new process into that group. This gives opentrace a new identifier to track, namely the cgroup.

So consider the following program.

let () =
  Eio_posix.run @@ fun env ->
  Eio.Path.(save ~create:(`Or_truncate 0o664) (env#fs / "hello.txt") "hello");
  Eio.Process.run env#process_mgr
    [ "/bin/bash"; "-c"; "echo 'heya' > heya.txt" ]

It first creates a file using direct calls to functions like openat. Then it spawns a process which creates a new file called heya.txt. This happens in a separate process. However, with the --cgroups flag we can capture both interactions with the operating system.

$ sudo opentrace exec --cgroups --flags="O_CREAT|O_TRUNC" ./main.exe
pid,cgid,comm,kind,flags,mode,filename,return
153187,530807,"main.exe",openat,526914,436,"hello.txt",5
153192,530807,"bash",openat,577,438,"heya.txt",3

Eio's Process API

May 19, 2025
Patrick Ferris

I have used the Eio_unix fork action process API to be able to extend what happens in the child process. Loading most eBPF programs into the kernel requires special privileges hence the need for sudo. When a user requests for a particular program to be executed and traced, opentrace spawns a process via the Eio Process API. Opentrace defines a few new so-called "fork actions", little fragments of C code that are run after the call to fork (clone). Most likely this ends with a call to execve, but other calls are possible for example setuid allowing opentrace to change the user of the child process so it does not run as root. Similarly, this is where (if used) we create the cgroup and place the process into that group.

Limitations: Io_uring

May 19, 2025
Patrick Ferris

Whilst testing opentrace against some of the tools I use nearly daily, I noticed some events were being missed. I tried tracing forester, and only the initial read of forest.toml was logged. It dawned on me that the reason for this was that forester (via eio) was using io_uring to perform most of the IO. Most attempts to open files were bypassing the open system calls, and instead they were being performed by the kernel after reading a submission request for an openat2-style call!

This is not news to seasoned, Linux systems programmers. Io_uring bypasses SECCOMP filters for exactly the same reasons.

$ sudo opentrace exec -- forester build
$ cat trace.csv
pid,cgid,comm,kind,flags,mode,filename,return
155007,535570,"forester",openat,524288,0,"forest.toml",5
155007,535570,"forester",Uring,2621440,0,"",0
155021,535570,"cp",openat,131072,0,"/home/patrick/documents/forest/theme/favicon-32x32.png",4
155007,535570,"forester",Uring,2686976,0,"",0
155007,535570,"iou-wrk-155007",Uring,557634,420,"",0
155007,535570,"iou-wrk-155007",Uring,557634,420,"",0
155007,535570,"iou-wrk-155007",Uring,557634,420,"",0

It is interesting to note two things here:

We can tell that forester reads the configuration file probably using something like In_channel in OCaml (it does).
It appears that Uring is performing IO in both worker threads and directly.

The file paths are empty at the moment as I cannot find a clean way to trace openat submission requests into Uring without it sometimes going very wrong. I have tried quite a few methods (e.g. tracing do_filp_open) and at the moment I am tracing io_openat2, but this seems quite brittle, and often the filename is completely garbled, so I do not set it. If anyone has any ideas to trace Io_uring more reliably, I am all ears!

AT Protocol and an IR for Wikis [weekly-2025-01-27]

January 27, 2025

AT Protocol

This week I've been diving into the AT Protocol.

The Authenticated Transfer Protocol, aka atproto, is a decentralized protocol for large-scale social web applications.

The protocol could be a candidate for the glue that holds together a distributed, computational wiki network. The protocol, it seems, is very similar to IPFS. Thankfully, a few years ago, I was working on building out a suite of OCaml libraries for working with IPFS. For example, ocaml-cid, self-describing content-addressed identifiers.

# let s = "zb2rhe5P4gXftAwvA4eXQ5HJwsER2owDyS9sKaQRRVQPn93bA";;
val s : string = "zb2rhe5P4gXftAwvA4eXQ5HJwsER2owDyS9sKaQRRVQPn93bA"
# let cid = Cid.of_string s |> Result.get_ok;;
val cid : Cid.t = <abstr>
# Cid.pp_human Format.std_formatter cid;;
cidv1 - base58btc - raw - ident(sha2-256) length(32) digest(6e 6f f7 95 0a 36 18 7a  80 16 13 42 6e 85 8d ce
                                                            68 6c d7 d7 e3 c0 fc 42  ee 03 30 07 2d 24 5c 95

To this end I have built out some more OCaml libraries for working with atproto, including:

atproto-data: the atproto data model, similar to JSON-LD.
ocaml-did: an OCaml library for working with decentralized identifiers.
atproto-lexicon: atproto's schema format, I've been building a quick tool for doing an OCaml translation from these schemas.

I managed to get a post published from the OCaml library after fixing it up and porting it to Eio.

An IR for Wikis

I started working on a proof-of-concept intermediate representation for Wikis -- I imagine it a bit like malfunction but for computational wikis i.e. a target for Wiki building tools that allows different front-ends and servers to communicate in a common IR for exposing key functionalities of a wiki:

Links: External links, cross-wiki backlinks
Versioned, temporal feeds
Etc.

Other PhD Work

I met with most of my Part II students this week, and I'm excited about their work. Progress reports are due this week and next they have a presentation to give.

In discrete maths this week we did induction. Next up is a big section on sets, functions, bijections etc.

Misc.

I was happy to find the Information Controls Fellowship Program.

The Information Controls Fellowship Program (ICFP) cultivates research, outputs, and creative collaboration on topics related to repressive internet censorship and surveillance.

Browsher and Namespaces › Browsher [browsher]

August 17, 2026
Patrick Ferris

After a week or two of abstracting the internals of Merry, I finally got a shell running in the browser. You can have a play with it here:

https://patrick.sirref.org/msh

$ echo "hello" > hello.txt
$ ls -la .
$ cat < hello.txt

Three weeks in the Shell › An In-memory Filesystem for Eio [eio_mem]

June 16, 2026
Patrick Ferris

Thanks to Eio's capability-first approach, any existing code implemented against the _ Eio.Path.t API should just work with the in-memory filesystem.

 let () =   let port = ref 8080 in
   Arg.parse
     [ ("-p", Arg.Set_int port, " Listening port number(8080 by default)") ]     ignore "An HTTP/1.1 server";
   Eio_main.run @@ fun env ->
+  Eio_mem.run @@ fun menv ->
   Eio.Switch.run @@ fun sw ->
-  (* Restrict to current directory: *)
-  let htdocs = Eio.Stdenv.cwd env in
+  (* Restrict to in-memory current directory: *)
+  Eio_mem.Fs.load ~filter (Eio.Stdenv.cwd env);
+  let htdocs = Eio.Stdenv.cwd menv in
   let socket =
     Eio.Net.listen env#net ~sw ~backlog:128 ~reuse_addr:true
       (`Tcp (Eio.Net.Ipaddr.V4.loopback, !port))

Landlocked Eio › Landlocked Eio [landlocked-eio]

April 18, 2026
Patrick Ferris

I had a go at "landlocking" Eio applications last week after stumbling across minimal's hakoniwa library for process isolation (which is very reminiscent of void processes).

module Landlock = Eio_linux.Landlock
let ( / ) = Eio.Path.( / )

let run_eio file =
  Eio.traceln "Writing to file";
  Eio.Path.save ~create:(`Exclusive 0o644) file "hello, world!";
  Eio.traceln "Reading file: %s" (Eio.Path.load file);
  Eio.Path.unlink file

let bad_program () =
  In_channel.with_open_bin "/etc/passwd" In_channel.input_all
  |> Eio.traceln "/etc/passwd: %s"

let () =
  Eio_linux.run @@ fun env ->
  Eio.Path.with_open_dir (Eio.Stdenv.cwd env / ".") @@ fun dir ->
  let parent_fd = 
    Eio_unix.Resource.fd_opt (fst dir) |> Option.get
  in
  let file = dir / "hello.txt" in
  let rules =
    Landlock.Rule.
      [
        path_beneath ~parent_fd
          ~flags:Flags.Fs.(
            write_file + read_file + make_reg + remove_file
           );
      ]
  in
  match Landlock.enter rules with
  | Ok () ->
      run_eio file;
      bad_program ()
  | Error `Not_supported -> Eio.traceln "Landlock not supported!"

Running this program (with the necessary privileges) prints the following.

+Writing to file
+Reading file: hello, world!
Fatal error: exception Sys_error("/etc/passwd: Permission denied")

Eio already has a similar function for the BSDs using capsicum. In capsicum, processes enter a capability mode:

cap_enter() places the current process into capability mode, a mode of execution in which processes may only issue system calls operating on file descriptors or reading limited global system state.

OCaml Roundup: December 2025 › Outreachy December Round [outreachy-december-round]

January 11, 2026
Patrick Ferris

December marks the beginning of a new Outreachy round, and I am very excited to be working with Tambe Salome on improving the ocaml-tiff library.

In particular, we will be adding the ability to write TIFF files from raw data. I imagine an API something like:

val write :
  ('repr, 'kind) Tiff.Data.t ->
  Tiff.File.wo ->
  unit

So far Salome has been making excellent progress churning through some issues that are blocking us from starting the main project, including:

As a community, the mentors and mentees also had a community call to introduce ourselves to eachother. I am very excited about the Yocaml project and the two Raven projects.

Shelter imports › Eio Debugging [eio-cap-debugging]

November 15, 2025
Patrick Ferris

One of the promises of Eio's capability model is an improved debugging experience. Your program must be designed in such a way as to make resources explicit. If you have a load_config function, more than likely it will need access to the file-system.

val load_config : _ Eio.Path.t -> Config.t
(** [load_config path] loads a configuration from [path].*)

Once all of your resources are explicit and threaded through your program, it becomes much easier to swap out one resource for another (provided they have the same interface).

This week I was trying to get the LIFE pipeline running in shelter. Part of that involves pulling the appropriate geospatial container image, but my code was terrifically broken, failing with what looked to me with a Docker error.

With only this clue, I knew that it was probably in the execution of some command in a child process, but which one? To trace this quickly, I could plug in a slightly different instance of the _ Eio.Process.mgr capability. Essentially unchanged from the one provided by the unix backend except that it would also print to stderr the command it was trying to run!

let debug_process_mgr (mgr : 'a Eio_unix.Process.mgr) : 'a Eio_unix.Process.mgr
    =
  let module D = struct
    type t = unit

    let spawn_unix () ~sw ?cwd ?pgid ?uid ?gid ~env ~fds ~executable args =
      Eio.traceln "Spawning subprocess... %a" Fmt.(list string) args;
      Eio_unix.Process.spawn_unix ~sw ?cwd ?pgid ?uid ?gid mgr ~env ~fds
        ~executable args
  end in
  let module V = Eio_unix.Process.Make_mgr (D) in
  Eio.Resource.T ((), Eio_unix.Process.Pi.mgr_unix (module V))

Whist there is a fair amount of jumping through first-class-module-shaped hoops here, all I am doing is rewrapping an existing "method" to a spawn a unix child process. Then using this helper function, I can push that into my standard environment.

let env =
  Eio_unix.Stdenv.with_env
  ~process_mgr:(debug_process_mgr env#process_mgr)
  env
in
Shelter.main config (env :> _ Shelter.env) dir cmd_file

And sure enough I was querying for the Docker user before pulling the image! I hadn't run into this, as for the last while I have been using alpine and debian images that already exists locally.