person. Thomas Leonard [talex5]

https://roscidus.com/

Backlinks

Analysing some Dockerfiles [weekly-2026-w37]

September 12, 2026
Patrick Ferris

After spending a few weeks thinking about the semantics of POSIX shells, this week saw me analysing lots of Dockerfiles looking for some evidence for how they are being used. It also saw environment variables landing in Eio thanks to Thomas and Anil. We also had some issues in ppxlib migrations.

Dockerfile Analysis [msh-dockerfile-analysis]

September 12, 2026
Patrick Ferris

Part of my research at the moment is thinking about how shells are used. What commands are run? What constructs are used (e.g. redirections, compound commands...)? Dockerfiles are a defacto standard means for building container images and for the most part they are a shell with weird semantics and some meta-commands (sounds familiar!).

I had some assumptions about the kinds of shell commands people use (myself included) when writing Dockerfiles. For example, making use of AND-lists for grouping commands in RUN instructions. But I had no evidence to back this up. So I ran some analysis over the dataset from this paper from Henkel et al..

Total Dockerfiles: 218_253 (errors: 808)
Total Docker Instructions: 2_729_118
Total RUN instructions: 657_909 (errors: 4_847)
========

Count (Windows Dockerfiles): 841 / 218_253
Count (ENV commands): 268_898 / 2_729_118
Count (COPY commands): 165_968 / 2_729_118
Count (ADD commands): 86_552 / 2_729_118
Count (WORKDIR commands): 104_112 / 2_729_118
Count (using at least one &&): 267_113 / 657_909
Count (uses of exec): 25 / 657_909
Count (system pkg manager: [apt, apt-get, dnf, apk, pacman, nix, brew, yum]): 192_092 / 657_909
Count (internet retrievers: [curl, wget]): 95_433 / 657_909
Count (might use python): 9_082 / 657_909
Counts:
[(rm, 169_386), (echo, 113_768), (mkdir, 100_924), (cd, 74_904),
 (chmod, 55_998), (set, 25_738), (mv, 24_034), (sed, 21_232),
 (chown, 18_610), (grep, 16_604), (cp, 15_637), (export, 13_154),
 ([, 10_411), (touch, 3_152), (exit, 2_288), (printf, 2_275), (awk, 2_010),
 (., 1_077), (test, 1_004), (source, 904), (pwd, 199), (unset, 105),
 (diff, 40), (trap, 2)]

I think there's some evidence here to support some assumptions I already had:

A significant proportion of RUN instructions in Dockerfiles invoke package managers.
A significant proportion of RUN instructions invoke curl or wget.
There is something rather Zipfian about how shell built-in utilities are used.
There is limited used of exec (and therefore limited use of opening file descriptors directly via the shell, as opposed to a command running in a subprocess).
A lot of time is spent changing directories, either temporally (via cd in a RUN instruction, or by using WORKDIR).
Lots of ENV setting.

Parsing Dockerfiles

September 12, 2026
Patrick Ferris

For reasons that are probably quite clear, whenever I end up doing a small task like this, I always reach for OCaml to do it. The idea is that I'm invested in that community and hope to see it thrive. And if that means contributing some fixes, libraries or advice along the way, it seems to me that that is time well spent.

When I started, I assumed ocaml-dockerfile had support for parsing files; it turns out it doesn't, but I found Thomas G.'s port that does. So I took it for a spin and the results were less than I had hoped for. Of the 219,061 Dockerfiles, 97,123 were not parseable! After a series of commits I managed to get that number down to 808.

Eio process environment

September 12, 2026
Patrick Ferris

What started out seemingly straightforward (add a capability-based API to Eio for accessing environment variables) ended up being a more nuanced discussion about what the best API would be. In the end, Thomas added an Eio.Process.Env API to manipulate soon-to-be opaque environments. This paved the way for the portable environment support.

The API Eio exposes is readonly so we do not have to worry too much about race conditions (even though users are free to Unix.putenv).

One quirk of this work was reaffirming something I had already stumbled across in Merry. Accessing and searching large environments has a cost! For my rather large Nix environment, we can see the slowdown pretty clearly.

                  name,   major-allocated,   minor-allocated,   monotonic-clock
        get_env/all 10,        346.619865,      38577.227215,      29248.828738
        get_env/all 50,       1677.003302,     181788.755923,     138307.102538
       get_env/all 100,       3323.770915,     361218.802888,     274768.577259
   get_env/all_once 10,        121.955078,      14694.900546,      11652.096583
   get_env/all_once 50,        563.133065,      61951.683658,      48161.605588
  get_env/all_once 100,       1113.238600,     121431.915770,      93960.646400
     get_env/direct 10,          0.000000,          0.000000,        974.649587
     get_env/direct 50,          0.000000,          0.000000,       4369.486360
    get_env/direct 100,          0.000000,          0.000000,       8753.393503

The all benchmark calls environment every time and then finds a variable using Eio.Process.Env.get_opt, all_once calls environment once and then Eio.Process.Env.get_opt every time and direct uses the Eio.Process.get_opt directly every time. The differences are a lot less stark when the environment is pretty small.

Ppxlib Migrations

September 12, 2026
Patrick Ferris

Thankfully, ppxlib has been mostly stable for a number of months. In part, thanks to the diligent efforts of Nathan. However, David uncovered some unpleasant bugs in the migration code for some new 5.4 AST features. Unfortunately, it is only for attributes which are a source of pain for ppxlib. I spent some time reviewing the fixes, and it looks like we still need some more work to ensure the attributes attached to package types roundtrip cleanly.

Misc.

September 12, 2026
Patrick Ferris

Some other work I recorded separately and include here for the eager reader.

OCaml Roundup: August 2026 [ocaml-roundup-august-2026]

July 12, 2026
Patrick Ferris

Much of August was spent thinking about POSIX shell semantics, and in particular extending notions from smoosh to include side effects (and maybe concurrency). This is still nascent work, but some ideas and flavours are described below.

Merging Shells

July 12, 2026
Patrick Ferris

The work came from wanting to merge two shell sessions into one. Setting aside the practicalities of an implementation, it makes sense to start first theoretically. When can I merge two shell executions, and when can I not?

It would be nice to think of unset DOWNLOAD_DIR as a function. This function takes the shell state and the operating state as arguments and returns a modified shell state and an "unmodified" operating system state (at least for our purposes). Meanwhile, branch-b may have a lot more side effects, but when focusing on the environment only we can see in a sense the two branches are conflict free. Well, sorta. We also need to ensure we do not also read DOWNLOAD_DIR in branch-b if we intend to remove it in their merge.

This work, in part, is borrowing ideas from MRDTs and can reuse void processes to help reason about when side effects can occur, as well as open-trace (which is now becoming os-trace).

I hope to continue this work, see my latest weekly for more on this.

Eio in the browser

July 12, 2026
Patrick Ferris

I spent some time carving out Eio_browser into a standalone library, something I depend on for Merry to work in the browser.

Eio_browser [eio_browser]

September 12, 2026
Patrick Ferris
https://git.sirref.org/eio_browser

Eio_browser is a backend for Eio that works in the browser. It reuses work I started at Tarides that picked up some momentum but has since been a little dormant.

Crucially, it provides a functional standard environment like Eio_main or any of the OS-specific backends.

module Stdenv : sig
  type base =
    < stdout : Eio_unix.sink_ty r
    ; stderr : Eio_unix.sink_ty r
    ; clock : float Eio.Time.clock_ty r
    ; mono_clock : Eio.Time.Mono.ty r
    ; secure_random : Eio.Flow.source_ty r
    ; debug : Eio.Debug.t
    ; backend_id : string >
end

val run : (Stdenv.base -> unit) -> unit
(** [run fn] runs [fn] passing it the {! Stdenv.base} environment.
    See {! Backend.run} for more information about how effects are handled.*)

OCaml Roundup: June 2026 [ocaml-roundup-june-2026]

July 12, 2026
Patrick Ferris

Summary

July 12, 2026
Patrick Ferris

June saw a lot of progress on Merry. At the start of the month, I worked on meio, a tracing tool for Eio programs. This was to help debug a particular shell script fragment that was causing Merry to hang. It uncovered a few UX opportunities to make meio more useful, and this might be fruitful direction to take some of my work.

Merry is now able to run (and pass!) the entire unixfun benchmarks in Koala. Whilst benchmarking Merry, it became apparent that it is paramount to provide a test built-in (rather than spawning [).

Part of this work also looked into building an in-memory filesystem for Eio. It is still a work-in-progress, but does pass many of the filesystem tests in Eio. To be a drop-in replacement in Merry requires some more work to handle an abstraction for Unix file descriptors (e.g. a fake stdout). It would be nice if Eio provided a Unix filesystem interface similar to the OS-agnostic interfaces of Eio itself (not to be confused with the Unix utilities in Eio_unix).

I have been very grateful to Thomas's diligent and steadfast progress on merging and fixing bugs in Eio over the last few months. You can keep up to date with his work in his notes.

Weekly logs

July 12, 2026
Patrick Ferris

The full details are in the weeklies transcluded below:

Using meio to debug Merry [weekly-2026-w23]

June 4, 2026
Patrick Ferris

During a large rewrite of Merry, I introduced a regression causing certain shell programs to hang indefinitely. I thought it might be a good opportunity to try out Meio.

First, let's look at a trace of Meio from some real-world programs. For example: meio -- msh -c "sleep 100".

One problem that comes up immediately from this example is that the tree hierarchy of tasks and cancellation contexts makes it tricky to reason about the logging. Here, we can see the pipeline's cancellation context clearly labelled pipeline-id1. This is an Eio.Switch.t for the pipeline sleep 100. The actual execution log for sleep 100 bubbles back up to the main fiber (that has no name, so it is called <task>). The daemon fiber, for ensuring the spawned process is reaped, is attached to the switch and is displayed below it.

Meio has a long way to go. Most programs are too fast to be monitored in this way. Using something like tokio-console's model might be a better idea here focusing less on the tree strucuture of the program, and more on the performance and monitoring of individual tasks. Without this, it turns out Meio was not all that useful for debugging this specific issue (which ended up being some bad pipe management on my part).

Three weeks in the Shell [weekly-2026-w25]

June 16, 2026
Patrick Ferris

The last three weeks have been focused work on building Merry. In practice, I wanted to run the following two commands through Merry's shell: msh:

apt-get install debootstrap -y
debootstrap stable /stable-chroot http://deb.debian.org/debian/

Seemingly innocuous. But both commands run thousands of lines of shell script. Whether that's Debian package maintainer scripts or state-machines for processing package release files.

What follows is a handful of the bugs and the shell scripts that caused them.

Field Splitting

June 16, 2026
Patrick Ferris

Edge-cases abound in the POSIX shell specification; and field splitting is one of the most serious perpetrators. Field splitting happens after word expansion. It is a known footgun amongst shell script authors.

$ count () { echo "Count: $#"; }; FOO="hello world"; count $FOO
Count: 2

Quoting variables that need expanding will prevent field splitting from taking place, but it is an easy thing to forget.

Field splitting uses the IFS (internal field separator) environment variable to determine how to do the splitting, and it comes with some extra rules beyond "split on these characters".

$ IFS=" "; FOO=" hello    world"; echo $FOO
hello world
$ IFS=" "; FOO=" hello    world"; echo "$FOO"
 hello    world
$ IFS=":"; FOO="a:b:c:"; echo $FOO
a b c
$ IFS=":"; FOO="a:b::c:d"; echo $FOO
a b  c d
$ IFS=":"; FOO=":a:b:c"; echo $FOO
 a b c
$ IFS=; FOO="unset   IFS"; echo $FOO
unset   IFS

Variable assignment

June 16, 2026
Patrick Ferris

How a shell stores variables also impacts field splitting. Naively, Merry stored variables as strings, but really they should be stored as their expanded form.

Consider this example of saving a function output in a variable called base.

iter () {
 echo "a"
 echo "b"
 echo "c"
}

base=$(iter)

for n in $base; do
  echo "Got $n"
done

for n in "$base"; do
  echo "Got $n"
done

Later, when we come to iterate across the fields of $base we wouldn't be able to reconstruct the intial fields that $(iter) produced if we simply saved the result as a string. Too much information would have been lost.

Redirections

June 16, 2026
Patrick Ferris

Redirections control what file descriptors point to. Each redirection has a lifetime associated with it. For example: cat < README.md really says "open README.md and execute the cat program mapping the process's FD 0 to the FD of README.md". When cat returns that redirection is no longer valid.

Other redirections are not so simple. Consider:

$ echo "!dlrow olleh" > input.txt
$ (read line; echo $line) < input.txt | rev
hello world!

There are lots of redirections here too. Firstly the standard input for everything inside the braced command (read line2; echo $line) should be the open file descriptor for input.txt. The standard output of the braced command should be the writing end of a pipe whose reading end is passed to rev as that process's standard input. The final standard output for rev should be unchanged from what it was originally (the shell's standard output).

Getting redirections right is hard, consider this snippet from the debootstrap scripts (all in the context of stdout and stderr pointing to a debootstrap.log file and file descriptor 4 pointing at the true stdout).

for m in $MIRRORS; do
    pkgdest="$TARGET/$($DLDEST pkg "$s" "$c" "$a" "$m" "$path")"
    if [! -e "$pkgdest" ]; then continue; fi
    pkgs_to_get="$(download_debs "$m" "$pkgdest" $pkgs_to_get 5>&1 1>&6)"
    if [ -z"$pkgs_to_get" ]; then break; fi
done 6>&1

More niche built-ins

June 16, 2026
Patrick Ferris

As the dust begins to settle on the nuts and bolts of implementing a POSIX-ish shell, new challenges are allowed to emerge from deep within scripts.

One such challenge is the seemingly endless (I know, they're finite) shell built-ins! Whilst trying to run debootstrap using its fakechroot variant, I bumped into getopts... a whole command-line parsing shell built-in.

An In-memory Filesystem for Eio [eio_mem]

June 16, 2026
Patrick Ferris

Towards the end of last week and the beginning of this week, I threw together a prototype for an in-memory filesystem for Eio: Eio_mem. Concretely, this provides a means for constructing an Eio.Fs.dir_ty Eio.Path.t value that is completely in-memory.

Thanks to Eio's capability-first approach, any existing code implemented against the _ Eio.Path.t API should just work with the in-memory filesystem.

There's a convenient Eio_mem.Fs.load function that can take a path from one filesystem (most likely a normal one from Eio_linux or Eio_posix) and replicate it into the in-memory filesystem. For example, you might have a static web-server pointing to a directory like "/var/www". The following might be the only change you need to make it completely in-memory (example from ocaml-cohttp):

 let () =   let port = ref 8080 in
   Arg.parse
     [ ("-p", Arg.Set_int port, " Listening port number(8080 by default)") ]     ignore "An HTTP/1.1 server";
   Eio_main.run @@ fun env ->
+  Eio_mem.run @@ fun menv ->
   Eio.Switch.run @@ fun sw ->
-  (* Restrict to current directory: *)
-  let htdocs = Eio.Stdenv.cwd env in
+  (* Restrict to in-memory current directory: *)
+  Eio_mem.Fs.load ~filter (Eio.Stdenv.cwd env);
+  let htdocs = Eio.Stdenv.cwd menv in
   let socket =
     Eio.Net.listen env#net ~sw ~backlog:128 ~reuse_addr:true
       (`Tcp (Eio.Net.Ipaddr.V4.loopback, !port))

The keen-eyed reader might have noticed the filter parameter. This can be used to ignore certain paths (here I ignored .git and _build which were adding substantial time to loading the filesystem).

Catching up in May [weekly-2026-w22]

June 1, 2026
Patrick Ferris

I never quite found the time in May to write weeknotes. I think that's, in some ways, quite a good thing. My philosophy on weeknotes is not that they are a public punchcard of my progress. They are a tool for organising thoughts and keeping my colleagues abreast of what I am up to. Only the latter suggests any necessity for timeliness and given I frequently message, call and zulip them, if a few weeks fall between the cracks then that's okay!

All that being said, I have tried to provide some thoughts and things I have been up to over the course of May and not just week 22 of the year.

Tools for Shells

June 1, 2026
Patrick Ferris

Some tools and libraries related to Merry.

Meio

June 1, 2026
Patrick Ferris

The shell work continues, and as the complexity internally increases I have found myself wishing for some more tools for debugging concurrent Eio programs. Given enough pipes and compound commands, it is fairly easy to introduce a fair deal of concurrency to your shell programs. To this end I hope I have improved meio somewhat in this fork.

In particular, there were several things I wanted to improve.

Simple UI improvements -- the set of tools for building TUI UIs from nottui are, on closer inspection, rather good and simple. For example, turning a particular UI component into a scrollable pane is as simple as wrapping it in a call to this function.
Logging more runtime events -- Eio programs emit information via OCaml's runtime events. This is how meio knows which fiber is currently active for example. It also emits information like "the current fiber created a new promise" and "this fiber is blocked waiting for a promise". I have not figured out an intuitive way for representing this in meio's tree-view, but I did turn these into "system logs" that appear in the per-fiber logs which is already an improvement. Thanks to Thomas for helping understand the runtime event sequence for logging this information.
Timestamping where possible -- all runtime events come with a timestamp, these were mostly discarded by meio.

Some aspects of meio made it clear that labelling Eio.Switches and Eio.Promises might be a good practice. After reading Michael's Eio forays, I can't help but wonder if a more opinionated Eio for Systems Programmers guide might be useful. I think the Eio README does a wonderful job of introducing most of the low-level concepts, but not quite how they are used by the working systems programmer.

Glob

June 1, 2026
Patrick Ferris

A little dismayed at the lack of progress I was making with Merry, I switched to a specific problem I knew I wanted to fix -- removing the dependency on glob(3), replacing it with a pure OCaml implementation.

let get_txts fs =
  Glob_eio.glob fs Glob.(of_component [ dir_sep; string "home"; dir_sep; star; string ".txt" ])

Anil provided some issues with fixes too which was very appreciated.

Beyond factoring out fork+exec, Merry should now be portable to the browser. Of course there are lots of other refactorings needed to (e.g. a fake-filesystem) -- but I'm interested in how I can use Eio's filesystem capabilities to do this pretty seemlessly. Having an emeddable, pure OCaml shell may also unlock some cool debugging capabilities for unikernels too!

Geocaml

June 1, 2026
Patrick Ferris

Michael has been valiantly trying some of the Geocaml libraries in his work on rendering LiDAR point clouds. Where possible, I have been reviewing Michael's PRs and helping implement features he might find useful. I also took his Laserver for a spin, and suggested a few changes.

Rtree

June 1, 2026
Patrick Ferris

After a comical series of attempts, I (or rather Anil) released rtree.0.2.0 which contained:

A bug fix where not all the tree values were being returned.
A new bounds function for getting an Rtree's maximum bound.
A set of simple removal functions.

PROJ

June 1, 2026
Patrick Ferris

Michael added some new functions to ocaml-proj. Previously, I had worked on making PROJ more portable by using virtual libraries. This was a little restrictive, as it forced users to always implement a JS version of any functionality that exists in the native version of PROJ.

To make it easier to contribute (and allow the C bindings to contain more specific bindings) I carved out the two virtual library implementations into standalone libraries, and reimplemented the virtual library support using these libraries. Now projects can depend directly on proj_c if they wish to have more features in exchange for being less portable.

Ppxlib

June 1, 2026
Patrick Ferris

Thankfully, work on ppxlib has been relatively quiet. Nathan also opened a PR for initial OCaml 5.6 support. Keeping on top of 5.6 features is probably the right way to go about this (rather than one giant PR during the 5.6 release cycle).

I'm particularly happy to have some better migration error handling. The previous errors, though uncommon, were confusing ppx users and hopefully this will help ppx authors and ppxlib maintainers triaging important encoding bugs or feature wishes.

Self-hosting

June 1, 2026
Patrick Ferris

I now have a fairly workable setup for self-hosting source code with git.sirref.org and tests.sirref.org. One piece that is missing, is a publishing workflow. I spent a little time building a YOCaml static-site generator for packaging OCaml libraries. Unfortunately, this was full of rabbit holes and busy work which was frustrating.

Tar-eio

June 1, 2026
Patrick Ferris

One such rabbit hole was tar-eio. It was in a bit of a sorry state, so I took the opportunity to improve the API for new users. Tarring a directory is now as simple as:

let foo = Eio.Path.(cwd / "foo") in
let foo_tar = Eio.Path.(cwd / "foo.tar") in
Eio.Path.with_open_out ~create:(`Or_truncate 0o755) foo_tar @@ fun foo_tar ->
Eio.Switch.run @@ fun sw ->
let t = Tar_eio.create ~sw foo in
Tar_eio.run t (Tar_eio.flow_of_file tar)

There was also a series of security issues that the tar library had that were fixed too, you can read about those in Thomas's weeknotes.

Fellowship Roundup [fellowship-roundup]

February 9, 2026
Patrick Ferris

Overview

February 9, 2026
Patrick Ferris

February is the final month of my current six-month fellowship work with Tarides. A lot of the work can be summarised by reading the roundups (see October, November, December and January). I have been writing throughout the fellowship. However, they do not quite capture all the work I have been doing.

Below, I pick individual projects and expand on them more holistically rather than pointing at individual PRs or issues.

Ppxlib

February 9, 2026
Patrick Ferris

Most of my time over the past six months has been devoted to ppxlib maintenance. A large proportion of this time has been solo development work. Ppxlib is a cornerstone library of the OCaml ecosystem (whether people like it or not).

$ opam list --depends-on=ppxlib --recursive | wc -l
2030

Moreover, it is completely invaluable to Jane Street too. Like other AST-based tools (e.g. ocamlformat, merlin), ppxlib must follow a regular release schedule to support newer compilers. During the years of multicore OCaml development, ppxlib maintenance was a little easier as the Parsetree rarely changed. However, since then, the same cannot be said. Since OCaml 5.2, here are just some of the Parsetree changes:

Functions are represented according to their arity.
Local module opens on types.
Effect syntax.
Modular explicits.
External types.
Locations for all parts of long identifiers.
Labeled tuples.

In a perfect world, each of these require a codec for serialising the feature into ASTs that do not support the feature, plenty of tests and new Ast_builder/Ast_pattern functions for using the feature. Nathan and I have been managing pretty well I would say, though we are not getting any technical debt work done.

Outreachy

February 9, 2026
Patrick Ferris

As the OCaml coordinator for Outreachy I have been running the latest December 2025 round. We are lucky to have four projects on the go!

Thibaut Mattio is mentoring two projects: an ML dashboard for Raven and an OxCaml backend for Raven.
Xvw is mentoring a Yocaml project.
I am mentoring a Geocaml project.

Eio

February 9, 2026
Patrick Ferris

During my work on tools like Shelter I have made a few improvements to Eio including:

OxCaml

February 9, 2026
Patrick Ferris

I have enjoyed my work on OxCaml went it has happend. I think a few tools have been useful including the try-oxcaml work. David and I also had fun at ICFP discussing some quick experiments related to io_uring and OxCaml.

One road-blocker here is the comparative amount of time spent trying to keep the OxCaml ecosystem working. Given my limited time working on fellowship projects, the rate of change of OxCaml (including the opam ecosystem) meant most of the time was spent making it just work, not exploring or experimenting with the features themselves. I think this has got better in recent months and perhaps jumping back in I would be surprised at the progress that has been made there.

Forester

February 9, 2026
Patrick Ferris

Tarides are funding some work on forester. This site uses forester by way of graft and I have spent some time working on these tools during the fellowship.

This includes a bytesrw-based bibtex library in OCaml that could be released soon.

Papers

February 9, 2026
Patrick Ferris

I had the great fortune of attending ICFP where I presented two talks and was co-author on a few others too.

Reference. Functional Networking for Millions of Docker Desktops (Experience Report) [madhavapeddy2025docker]

2025
Anil Madhavapeddy, David J. Scott, Patrick Ferris, Ryan T. Gibb, Thomas Gazagnaire
10.1145/3747525
https://dl.acm.org/doi/10.1145/3747525

Abstract

2025
Anil Madhavapeddy, David J. Scott, Patrick Ferris, Ryan T. Gibb, Thomas Gazagnaire

Docker is a developer tool used by millions of developers to build, share and run software stacks. The Docker Desktop clients for Mac and Windows have long used a novel combination of virtualisation and OCaml unikernels to seamlessly run Linux containers on these non-Linux hosts. We reflect on a decade of shipping this functional OCaml code into production across hundreds of millions of developer desktops, and discuss the lessons learnt from our experiences in integrating OCaml deeply into the container architecture that now drives much of the global cloud. We conclude by observing just how good a fit for systems programming that the unikernel approach has been, particularly when combined with the OCaml module and type system.

Reference. What we talk about when we talk about scientific programming [ferris2025scientific_programming]

2025
Patrick Ferris
https://watch.eeg.cl.cam.ac.uk/w/aYXqXLtgQawYMjVXjSQtjx

Abstract

2025
Patrick Ferris

Programming for the planet undoubtedly involves programming scientifically, but what kind of programming are we talking about and what makes it scientific? In what ways does it differ from other forms of programming, if at all? Is scientific programming, data science or machine learning fundamentally different to constructing a compiler or building a high-throughput web server? By considering how the scientific method (with its falsifiable hypotheses and repeatable and reproducible experiments) relates to scientific programming, I hope to explore how computer science and traditional programming techniques are coming up short in meeting the requirements of scientific programmers.

Reference. Generating a corpus of Hazel programs from ill-typed OCaml programs [ferris2025hazel_of_ocaml]

2025
Patrick Ferris, Anil Madhavapeddy
https://patrick.sirref.org/var/tyde2025.pdf

Abstract

2025
Patrick Ferris, Anil Madhavapeddy

When developing a new programming language, having a large corpus of both correct and incorrect programs allows language designers to test and explore the capabilities of their new language. However, bootstrapping such a corpus of incorrect programs is time-consuming and arduous. We therefore explore how to reuse code from more mature languages to generate a corpus of ill-typed code for newer ones. We have developed a compiler to Hazel, an emerging language with typed holes, from the more mature OCaml ecosystem. We find it practical to generate a comprehensive corpus of ill-typed programs for Hazel development, and discuss future larger scale efforts towards bridging ecosystems.

Reference. Decomposable Type Highlighting for Bidirectional Type and Cast Systems [carroll2025decomposable_type_highlighting]

2025
Max Carroll, Anil Madhavapeddy, Patrick Ferris
https://maxcarroll0.github.io/papers/workshops/HATRA-decomposable-type-highlighting/

Abstract

2025
Max Carroll, Anil Madhavapeddy, Patrick Ferris

We explore how to provide programmers with an interactive interface for explaining the process by which static types and dynamic casts are derived, with the goal of improving the debugging of static and dynamic type errors. To this end, we define mathematical foundations for a decomposable highlighting system within a bidirectional system, and show how these can be propagated through dynamic types in a cast system. Our prototype implementation in the gradually typed Hazel language includes a web-based user interface, through which we highlight the importance of type level debugging.

Reference. Yirgacheffe: A Declarative Approach to Geospatial Data [dales2025yirgacheffe]

2025
Michael Winston Dales, Alison Eyres, Patrick Ferris, Francesca A. Ridley, Simon Tarr, Anil Madhavapeddy
10.1145/3759536.3763806
https://dl.acm.org/doi/10.1145/3759536.3763806

Abstract

2025
Michael Winston Dales, Alison Eyres, Patrick Ferris, Francesca A. Ridley, Simon Tarr, Anil Madhavapeddy

We present Yirgacheffe, a declarative geospatial library that allows spatial algorithms to be implemented concisely, supports parallel execution, and avoids common errors by automatically handling data (large geospatial rasters) and resources (cores, memory, GPUs). Our primary user domain comprises ecologists, where a typical problem involves cleaning messy occurrence data, overlaying it over tiled rasters, combining layers, and deriving actionable insights from the results. We describe the successes of this approach towards driving key pipelines related to global biodiversity and describe the capability gaps that remain, hoping to motivate more research into geospatial domain-specific languages.

I had some great conversations with KC too with respect to my current research.

Future Work

February 9, 2026
Patrick Ferris

As I enter my final year of the PhD, my goal would be to try and align more of my fellowship work with my own research wherever it makes sense. This research focuses on the following hypothesis:

Embedding deep provenance tracking, reversible execution and mergeable histories directly into an interactive programming environment (a POSIX-like shell) greatly decreases the gap between exploratory scientific work and reproducible, publishable results; all whilst leaving existing workflows intact and being programming language agnostic.

I want to focus on building these tools using Irmin, Eio, OxCaml etc. There are other projects I wish to continue regardless, like Outreachy, though just as a coordinator (not a mentor).

Overlapping PhD Work

February 9, 2026
Patrick Ferris

There are multiple projects that overlap with my work. For example, Irmin, which I wrote a retrospective for. This is at the core of Shelter and suggests an extremely valid use case for a brancheable, mergeable database.

In addition to this, continuing to work on Eio makes sense to me too. Eio currently has no maintainer and no active development, but the issues are piling up. In particular, I am making heavy use of Eio in Merry, a POSIX(ish) shell written in OCaml. Aside from research angles on brancheable and mergeable shells, I am also quite interested in how this might look in terms of OxCaml. I met with Anil and Thomas recently to discuss the future of Eio and there was some agreement that we are all heavy users of it and hope to keep up its maintenance and set of features.

One large piece of work that is sorely needed here is moving Eio to bytes and not bigarray-backed Cstructs. Any libraries using bytesrw (e.g. jsont) incur an extra copy of data into and out of the bytes. Anil has been working on some of the pieces recently to make this better!

Maintaining and Releasing my own libraries

February 9, 2026
Patrick Ferris

I have published quite a few OCaml libraries in my time, below is just a few libraries (filtered to not include OCurrent projects).

$ opam list --all --no-switch --columns=name,authors: | grep -E "sirref|Ferris" | cut -d " " -f1
carbon
geojson
geojsone
graft
hilite
ISO3166
jekyll-format
ppx_deriving_ezjsonm
ppx_deriving_yaml
rtree
search
topojson
topojsone
cid
multibase
multicodec
multihash
multihash-digestif

Some of these are used in the community including hilite, ppx_deriving_yaml and the multi-codecs. I would like to do some general maintenance of some of these tools and release a few iterations of graft which now makes use of bib.

Ppxlib Maintenance

February 9, 2026
Patrick Ferris

By far, maintenance of ppxlib takes up a majority of my time. I think, given the other work I need to focus on, it would be great to try to minimise the amount of time working on ppxlib. In particular, perhaps only maintaining ppxlib as far as OCaml release-readiness is a possibility and important bug fixes (like the OOM bug reported by Jane Street).

Forester, Graft and Writing

February 9, 2026
Patrick Ferris

With all of this work, I am eager to communicate what I am up to and my thoughts on open source, OCaml, OxCaml etc. I am glad to be using Forester to do this, particularly as it is written in OCaml (and could make great use of my own work on Eio).

I hope to continue working on Graft and related tools like bib. But I would also like to start writing some long-form content similar to the Irmin retrospective.

Shelter in the Forest [weekly-2025-04-21]

April 21, 2025
Patrick Ferris

I spent much of this week working on Shelter and things related to it. Some of that time was also spent on Forester.

Forester

April 21, 2025
Patrick Ferris

I added two important quality-of-life features to my frontend to Forester this week.

Bibtex Support

April 21, 2025
Patrick Ferris

I had previously mentioned adding support to Forester for Markdown. This week I added support for Bibtex too. From any *.bib file in your forest, Forester will now dutifully recognise it as a Bibtex file and convert, as best it can, all the entries into Reference trees.

I'm becoming quite convinced of this model at the moment. I'm using Forester's Code.t as a target representation. In fact, to ease the process, I really shouldn't spend all my time on my website, I have reused my Yaml.t -> Code.t and Markdown.t -> Code.t functions in the Bibtex parser.

To see it in action, you could have a look at the Build Systems à la Carte paper which is generated completely from Bibtex.

Full Heading Support

April 21, 2025
Patrick Ferris

The eagled-eyed viewer may have noticed that the table of contents for this page has more than one level. I finally caved and spent an evening rejigging my Cmarkit.Doc.t -> Tree code which was hacky and broken and is now less hacky and less broken.

In addition, headings support links and emphasis etc.

Lunch with Jon Sterling

April 21, 2025
Patrick Ferris

I had a delightful lunch with Jon Sterling discussing the future of Forester, the nature of the Web (old and new) and the success posting weekly updates for our colleagues. Thanks Jon.

Shelter Fixes

April 21, 2025
Patrick Ferris

I spent a good chunk of my week fixing bugs in Shelter with the aim to perhaps setup a VM somewhere and let people kick the tyres of what we've got so far.

The first bug is pretty annoying. At the moment, our filsystem backend is ZFS and we make heavy use of snapshots and cloning in order to provide time-travelling capabilities. Unfortunately, ZFS will take a snapshot before data has fully made it to disk (or whatever is the equivalent point it should reach in ZFS). Commands that generated lots of disk activity would be snapshotted in a half finished state and this would cause all sorts of problems. Thanks to Mark Elvers for the pointer to how OBuilder uses ZFS for the OCaml macOS builders which unmount datasets immediately therefore inducing a flush of sorts. Shelter now follows a similar model with all of the slow downs that create. Anil and I discussed some amalgamation of overlayfs, tmpfs and ZFS to alleviate some of this but for now that's a premature optimisation.

A small eDSL for Shelter

April 21, 2025
Patrick Ferris

Whilst testing Shelter, I ended up wanting a way to programmatically invoke the different run commands. This is similar to say a Dockerfile, but maybe with a little more expressivity.

This lead me to revisit the Build systems à la Carte paper and rediscover selective applicative functors.

I started playing around with a selective applicative interface to Shelter, this would allow you to express your dependencies statically but select them dynamically (as the paper says).

module D = Shl (Identity)

let shelterfile =
  let open D in
  let base_image = from "alpine" in
  let is_node_lst img = String.equal "v22.15.0" (stdout img) in
  let cmds base =
    let node_version = run "node --version" base in
    Select.if'
      (Select.map ~f:is_node_lst node_version)
      (run "node -e 'console.log('success!')")
      (run "node -e 'console.log('failure!')")
      base
  in
  with_session "node" (cmds base_image)

From this, we get a slightly more expressive way to describe images.

Geocaml TIFF Library

April 21, 2025
Patrick Ferris

I was pleasantly surprised to receive a pull request from Michael adding support to ocaml-tiff for reading TIFF files compressed using LZW. I was also surprised to hear the TIFF LZW is a little different to others.

In trying to get this PR merged, I moved the initialisation of the Eio eventloop to outside each individual test case. This one change then completely broke the entire test suite. After a period of debugging and help from Thomas Leonard the root cause was OCaml's OUnit2 library using process-level parallelism (via Unix.fork), sharing the ring between the parent and the child lead to the issues.

Part II Students

April 21, 2025
Patrick Ferris

As the new term begins, it signals that there are only just over two weeks for the final year undegrads at Cambridge to submit their dissertations.

The four students that I help supervise have been sending me drafts of their work (and reminder you can read about their projects) and I'm very impressed. I'm sure the next two weeks will be stressful, but I'm proud of what they have accomplished over the past academic year.

Landlocked Eio › Landlocked Eio [landlocked-eio]

April 18, 2026
Patrick Ferris

Landlock is a sandboxing mechanism that has existed in the Linux kernel since 5.13. It allows unprivileged processes a means to restrict their ambient rights (e.g. creating a file or opening a network connection). These rights are ambient in the ambient authority sense, i.e., any process can attempt to perform an action without needing access to any particular capability to do so.

Ambient systems are often contrasted with capability-based ones. In terms of operating systems, this is most commonly illustrated with access-control-lists-as-columns and capabilities-as-rows in an access matrix.

Eio is a capability-based library in OCaml -- resources are provided at the start of the application and can be passed around wherever they are needed. The design is covered in Thomas Leonard's lambda capabilities blog post. Unfortunately, any application (willingly or via a third-party library) may ambiently access OS resources. In OCaml, this may be by using the standard library functions (e.g. Sys.readdir), things from the Unix module, a C FFI function or by spawning a subprocess.

I had a go at "landlocking" Eio applications last week after stumbling across minimal's hakoniwa library for process isolation (which is very reminiscent of void processes).

module Landlock = Eio_linux.Landlock
let ( / ) = Eio.Path.( / )

let run_eio file =
  Eio.traceln "Writing to file";
  Eio.Path.save ~create:(`Exclusive 0o644) file "hello, world!";
  Eio.traceln "Reading file: %s" (Eio.Path.load file);
  Eio.Path.unlink file

let bad_program () =
  In_channel.with_open_bin "/etc/passwd" In_channel.input_all
  |> Eio.traceln "/etc/passwd: %s"

let () =
  Eio_linux.run @@ fun env ->
  Eio.Path.with_open_dir (Eio.Stdenv.cwd env / ".") @@ fun dir ->
  let parent_fd = 
    Eio_unix.Resource.fd_opt (fst dir) |> Option.get
  in
  let file = dir / "hello.txt" in
  let rules =
    Landlock.Rule.
      [
        path_beneath ~parent_fd
          ~flags:Flags.Fs.(
            write_file + read_file + make_reg + remove_file
           );
      ]
  in
  match Landlock.enter rules with
  | Ok () ->
      run_eio file;
      bad_program ()
  | Error `Not_supported -> Eio.traceln "Landlock not supported!"

Running this program (with the necessary privileges) prints the following.

+Writing to file
+Reading file: hello, world!
Fatal error: exception Sys_error("/etc/passwd: Permission denied")

Eio already has a similar function for the BSDs using capsicum. In capsicum, processes enter a capability mode:

cap_enter() places the current process into capability mode, a mode of execution in which processes may only issue system calls operating on file descriptors or reading limited global system state.

This plays nicely with Eio's capabilities. Within the scope of Eio.Path.with_open_dir, the directory will remain usable after entering "cap" mode. This is not the case with landlock unless you add a specific path_beneath rule to the ruleset (like in the example above). However, you could imagine plumbing the necessary information throughout the Eio_linux backend for it to build a set of rules that can be applied at any given moment in order to mimic the capsicum semantics.