Patrick Ferris 2026 3 19 https://patrick.sirref.org/weekly-2026-w12/ weekly-2026-w12 /weekly-2026-w12/ A POSIX Shell in OCaml Long time, no weekly. Since the start of this year I have been building a POSIX shell in OCaml called msh (with the underlying library called Merry). Merry is available online now. Patrick Ferris 2026 3 19 A POSIX(ish) shell in OCaml Merry is a POSIX(ish) in OCaml. It uses Eio alongside Morbig (a static parser for POSIX shell). Patrick Ferris 2026 3 19 Why another (POSIX) shell? Shells have been around for a long time. In my research on the notion of metashell I wrote about Louis Pouzin originally coining the term: Patrick Ferris 2025 5 19 https://patrick.sirref.org/pouzin-shell/ pouzin-shell /pouzin-shell/ Metashells › Louis Pouzin's "SHELL" I spent some time reading part of the multics design documentation this week. Louis Pouzin coined the term "SHELL" in this document, and I was reminded yet again just how important it is to be a good writer even as a "computer science researcher". For example, this excerpt from the requirements section of the document The previous definitions imply that a command MUST be designed while keeping in mind the user, sitting at his console, wondering about what might be going on, mistyping or forgetting arguments, even if fully aware of the conventions, and possibly interfering with the command by hasty quits, carriage returns, and other temperamental reactions. And then later, when defining the "SHELL". We may envision a common procedure called automatically by the supervisor whenever a user types in some message at his console, at a time when he has no other process in active execution under console control (presently called command level). This procedure acts as an interface between console messages and subroutine. The purpose of such a procedure is to create a medium of exchange into which one could activate any procedure, inside of another program if it were called. Hereafter, for simplification, we shall refer to that procedure as the "SHELL". It still surprises how little the undergraduate degree in computer science at Cambridge focuses on writing skills. I built Shelter exploring the idea of a shell-like interface that allowed users to undo their shell actions (amongst other cool tricks). Unfortunately shell-like is not enough. Shelter cut many corners to masquerade as a shell (e.g. appending env to understand how a command may have altered the execution environment). I felt it was necessary to make Shelter a "SHELL"! To do that, I needed a solid foundation to build on. msh, the POSIX shell that comes with Merry, is by no means POSIX-complete in terms of features. But it is my daily driver at this point. It includes a pure OCaml rewrite of linenoise (a small, self-contained alternative to the venerable readline) called bruit. If you have docker installed on your machine, you can take it for a spin today: The patrickferris/msh docker image is just for trying it out. It is based on the OCaml 5.3 alpine image. Alternatively, you can build msh from source and have it available in your opam switch. There are many small paper cuts left to patch over, but most of it is porcelain (e.g. ctrl+left-arrow for moving in bruit). Unfortunately, these are the kinds of things that you will immediately stumble upon. Patrick Ferris 2026 3 19 What makes Merry different? Nothing. Merry is supposed to be a solid, POSIX-ish base to build on. Unfortunaltely, as it turns out, the subset of features from the POSIX specification that people actually use... is pretty much all of it. Every possible redirection, variable expansion, shell built-in and compound command make some appearance. Not to mention the non-POSIX bits of shell we all take for granted (e.g. &>-redirection). Merry and msh are useable. You will most likely find bugs if you use them. If, when using msh, you find something obscure happening you can enable debug mode either my setting the variable MSH_DEBUG or by invoking msh with -v -v. Patrick Ferris 2026 3 19 What's next? It is soon time to combine Shelter and Merry into the time-travelling, POSIX-ish shell that I have been trying to build since I first started working on Shelter. For this to be successful, I need Merry to be able to pass plenty of tests and right now that involves trying to install plenty of packages using tools like apk and apt. Patrick Ferris 2026 3 19 Shell MRDT What I am particularly interested in reasoning about, is the execution context in Merry. This value, alongside the file-system, constitutes a fairly deep understanding of the state that changes in each step of a shell's evaluation loop. This was, in terms of Shelter, the missing piece for truly building some kind of MRDT across shell sessions. Patrick Ferris 2026 3 19 TIFF in OCaml I picked up from the excellent Outreachy work of Tambe Salome in getting write-support in ocaml-tiff. We are getting closer to the kind of API I envisaged in this latest round of refinement and review. let / = Eio . Path . / let checkerboard ~ size = let v = Nx . zeros Nx . uint8 size ; size in for row = 0 to size - 1 do for col = 0 to size - 1 do if row + col mod 2 = 0 then Nx . set_item row ; col 254 v done done ; Nx . to_bigarray v let = Eio_posix . run @@ fun env -> Tiff_eio . with_open_out env # cwd / " example.tiff " @@ fun w -> let tif = Tiff . make checkerboard ~ size : 256 in Tiff . to_file tif w I have also been extremely pleased to see further external collaborators appearing: Nicolas helping out with metadata maintainence. Virgile adding support for reading multi-image TIFF files. References Louis Pouzin 2013 https://patrick.sirref.org/pouzin-shell-2013/ pouzin-shell-2013 /pouzin-shell-2013/ The Origin of the Shell Reference https://people.csail.mit.edu/saltzer/Multics/Multics-Documents/MDN/MDN-4.pdf Multics Documents Louis Pouzin 2013 Abstract How RUNCOM was created for CTSS and the shell was designed for Multics by Louis Pouzin. Context Patrick Ferris 2026 3 10 https://patrick.sirref.org/ocaml-roundup-march-2026/ ocaml-roundup-march-2026 /ocaml-roundup-march-2026/ OCaml Roundup: March 2026 Patrick Ferris 2026 3 19 https://patrick.sirref.org/weekly-2026-w12/ weekly-2026-w12 /weekly-2026-w12/ A POSIX Shell in OCaml Long time, no weekly. Since the start of this year I have been building a POSIX shell in OCaml called msh (with the underlying library called Merry). Merry is available online now. Patrick Ferris 2026 3 19 A POSIX(ish) shell in OCaml Merry is a POSIX(ish) in OCaml. It uses Eio alongside Morbig (a static parser for POSIX shell). Patrick Ferris 2026 3 19 Why another (POSIX) shell? Shells have been around for a long time. In my research on the notion of metashell I wrote about Louis Pouzin originally coining the term: Patrick Ferris 2025 5 19 https://patrick.sirref.org/pouzin-shell/ pouzin-shell /pouzin-shell/ Metashells › Louis Pouzin's "SHELL" I spent some time reading part of the multics design documentation this week. Louis Pouzin coined the term "SHELL" in this document, and I was reminded yet again just how important it is to be a good writer even as a "computer science researcher". For example, this excerpt from the requirements section of the document The previous definitions imply that a command MUST be designed while keeping in mind the user, sitting at his console, wondering about what might be going on, mistyping or forgetting arguments, even if fully aware of the conventions, and possibly interfering with the command by hasty quits, carriage returns, and other temperamental reactions. And then later, when defining the "SHELL". We may envision a common procedure called automatically by the supervisor whenever a user types in some message at his console, at a time when he has no other process in active execution under console control (presently called command level). This procedure acts as an interface between console messages and subroutine. The purpose of such a procedure is to create a medium of exchange into which one could activate any procedure, inside of another program if it were called. Hereafter, for simplification, we shall refer to that procedure as the "SHELL". It still surprises how little the undergraduate degree in computer science at Cambridge focuses on writing skills. I built Shelter exploring the idea of a shell-like interface that allowed users to undo their shell actions (amongst other cool tricks). Unfortunately shell-like is not enough. Shelter cut many corners to masquerade as a shell (e.g. appending env to understand how a command may have altered the execution environment). I felt it was necessary to make Shelter a "SHELL"! To do that, I needed a solid foundation to build on. msh, the POSIX shell that comes with Merry, is by no means POSIX-complete in terms of features. But it is my daily driver at this point. It includes a pure OCaml rewrite of linenoise (a small, self-contained alternative to the venerable readline) called bruit. If you have docker installed on your machine, you can take it for a spin today: The patrickferris/msh docker image is just for trying it out. It is based on the OCaml 5.3 alpine image. Alternatively, you can build msh from source and have it available in your opam switch. There are many small paper cuts left to patch over, but most of it is porcelain (e.g. ctrl+left-arrow for moving in bruit). Unfortunately, these are the kinds of things that you will immediately stumble upon. Patrick Ferris 2026 3 19 What makes Merry different? Nothing. Merry is supposed to be a solid, POSIX-ish base to build on. Unfortunaltely, as it turns out, the subset of features from the POSIX specification that people actually use... is pretty much all of it. Every possible redirection, variable expansion, shell built-in and compound command make some appearance. Not to mention the non-POSIX bits of shell we all take for granted (e.g. &>-redirection). Merry and msh are useable. You will most likely find bugs if you use them. If, when using msh, you find something obscure happening you can enable debug mode either my setting the variable MSH_DEBUG or by invoking msh with -v -v. Patrick Ferris 2026 3 19 What's next? It is soon time to combine Shelter and Merry into the time-travelling, POSIX-ish shell that I have been trying to build since I first started working on Shelter. For this to be successful, I need Merry to be able to pass plenty of tests and right now that involves trying to install plenty of packages using tools like apk and apt. Patrick Ferris 2026 3 19 Shell MRDT What I am particularly interested in reasoning about, is the execution context in Merry. This value, alongside the file-system, constitutes a fairly deep understanding of the state that changes in each step of a shell's evaluation loop. This was, in terms of Shelter, the missing piece for truly building some kind of MRDT across shell sessions. Patrick Ferris 2026 3 19 TIFF in OCaml I picked up from the excellent Outreachy work of Tambe Salome in getting write-support in ocaml-tiff. We are getting closer to the kind of API I envisaged in this latest round of refinement and review. let / = Eio . Path . / let checkerboard ~ size = let v = Nx . zeros Nx . uint8 size ; size in for row = 0 to size - 1 do for col = 0 to size - 1 do if row + col mod 2 = 0 then Nx . set_item row ; col 254 v done done ; Nx . to_bigarray v let = Eio_posix . run @@ fun env -> Tiff_eio . with_open_out env # cwd / " example.tiff " @@ fun w -> let tif = Tiff . make checkerboard ~ size : 256 in Tiff . to_file tif w I have also been extremely pleased to see further external collaborators appearing: Nicolas helping out with metadata maintainence. Virgile adding support for reading multi-image TIFF files. Patrick Ferris 2026 3 30 https://patrick.sirref.org/weekly-2026-w13/ weekly-2026-w13 /weekly-2026-w13/ Ppxlib release and Merry updates Patrick Ferris 2026 3 30 https://patrick.sirref.org/ppxlib-release-0-38-0/ ppxlib-release-0-38-0 /ppxlib-release-0-38-0/ Ppxlib Releases › 0.38.0 Nathan and I released ppxlib.0.38.0 last week. Its main feature is full migration support for the upcoming OCaml 5.5 compiler (currently in its alpha3 release). This means supporting the handful of new features landing in OCaml 5.5: modular explicits, external type declarations and arbitrary "local" structure items. As Nathan and I continue to find a plausible maintenance story for ppxlib, Nathan has opened an issue on the OCaml compiler to discuss the idea of adding additional extension points to the language. This comes from the desire to be able to store encoded versions of new OCaml features inside older abstract syntax trees. There are also some nice bug fixes in there too: A potential OOM loop has now been removed. Locations have been restored to long identifiers! Patrick Ferris 2026 3 30 Merry updates It turns out writing a POSIX(ish) shell is hard; at least, there is a vast number of slightly obscure features that one needs to support. I think this is surprising because most people writing shell scripts write simple shell scripts; scripts that use a much smaller subset of features. In the same breath, those same developers are likely to use something like apt-get install bash which runs a plethora of more advanced (and non-POSIX) shell scripts! Patrick Ferris 2026 3 30 Exec redirects In a shell script, you can use the built-in command exec to replace the current process with a new one (e.g. exec vim). However, there is a different mode of operation for exec: If exec is specified with no operands, any redirections associated with the exec command shall be made in the current shell execution environment. One of my litmus tests for Merry is the Debian debootstrap scripts (h/t Anil). One thing that it does is the following: err printf " err $ 1 " >&4 exec 4>&1 A lot of detail has been elided for clarity. exec 4>&1 sets up a redirection for the shell's execution environment in which file descriptor 4 is now an alias for standard output. So, writing to 4 (by redirecting a command's standard output to 4 i.e. >&4) will output to where standard output is going (most likely the terminal). For this to work, shell's must dup2 the relevant file descriptors which have the following condition: If the file descriptor newfd was previously open, it is closed before being reused; the close is performed silently (i.e., any errors during the close are not reported by dup2()). This is all very well, unless your program has an important file already open that happens to have file descriptor 4. The Eio Linux and POSIX backends suffer from this problem. Both make use of a file-based synchronisation mechanism for waking up the eventloop should another domain push a completion to the scheduler's run queue. On Linux this is via an eventfd and in POSIX, a pipe. This dup2 will close the eventfd and will likely grind Eio to a halt (or an assert false). For now, I have resorted to vendoring Eio and moving the eventfd file descriptors to higher values, though I have opened a PR to make this more configurable. This bug is quite easy to write up... it was not so easy to find! As a random example, consider the debconf/confmodule script which offers very little room for a buggy implementation! Patrick Ferris 2026 3 30 Outreachy The presentations from the demonstrations for this round of Outreachy are now online! Patrick Ferris 2026 2 9 https://patrick.sirref.org/outreachy-ocaml-tiff/ outreachy-ocaml-tiff /outreachy-ocaml-tiff/ Write support in OCaml TIFF library I am mentoring Tambe Salome during the December 2025 Outreachy round to add support for writing TIFF files in the ocaml-tiff library. You can now see the video of the demonstration day presentation: Patrick Ferris 2026 4 3 https://patrick.sirref.org/weekly-2026-w14/ weekly-2026-w14 /weekly-2026-w14/ Forking in Shells & Library Maintenance This week, Merry got close to executing apt-get install ca-certificates and running Debian's debootstrap scripts correctly. Nathan and I released Ppxlib 0.38.0, I added support to ocaml-proj for compiling to the browser and I fixed a long-standing bug in hilite. Patrick Ferris 2026 4 3 Forking in Merry So far, Merry has managed to support many features of the POSIX shell specification without needing to do a fork(2) without an exec. Or, to put it another way, all the forking that Merry does, will only be followed by running C code. This is actually a feature of Eio's Process API. It is required to make process execution safe in the context of multiple domains in OCaml (mostly due to ensuring consistency within the garbage collector). However, it is perfectly okay to fork and run OCaml code provided there is only a single domain. In a shell, many features of the shell language require that commands (or built-ins, function applications etc.) execute within a subshell. The specification does not go into detail about how shells should implement this, but many choose to fork in order to preserve some state in the parent (for example, the file descriptor table). Some shells try to minimise the number of forks as an optimisation. The korn shell (ksh) is one such shell. Using the notation $(command) will cause command to execute in a subshell of the current ksh. In many instances, ksh will not actually fork/exec a subshell when command is a built-in or a shell function. By trying to adhere to Multicore OCaml's "thou shalt not fork" commandment, Merry is much more similar to ksh in this regard. However, whenever there is an interaction with a shell built-in and semantics that need a child process, things get tricky very quickly. Consider the following: while echo " hello " ; do : done | head -n 3 We have a mix of shell built-ins, normal commands, a compound command (the while loop) and a pipeline. Each individual command in the pipeline requires you to run them in a subshell and set them up before executing them. Without a fork for that first command, you may end up looping forever which is not the intended behaviour here. I had a good conversation with Michael about the implications of this for reproducibility. For now, I have started implementing a fork for some of the shell features whilst still trying to maintain the functional core of the implementation. Patrick Ferris 2026 4 3 An OCaml 5.5 compatible Ppxlib Nathan and I released an OCaml 5.5 compatible version of ppxlib. Patrick Ferris 2026 3 30 https://patrick.sirref.org/ppxlib-release-0-38-0/ ppxlib-release-0-38-0 /ppxlib-release-0-38-0/ Ppxlib Releases › 0.38.0 Nathan and I released ppxlib.0.38.0 last week. Its main feature is full migration support for the upcoming OCaml 5.5 compiler (currently in its alpha3 release). This means supporting the handful of new features landing in OCaml 5.5: modular explicits, external type declarations and arbitrary "local" structure items. As Nathan and I continue to find a plausible maintenance story for ppxlib, Nathan has opened an issue on the OCaml compiler to discuss the idea of adding additional extension points to the language. This comes from the desire to be able to store encoded versions of new OCaml features inside older abstract syntax trees. There are also some nice bug fixes in there too: A potential OOM loop has now been removed. Locations have been restored to long identifiers! Patrick Ferris 2026 4 3 Browser support for ocaml-proj A while ago I built some "modern" bindings to PROJ4 in OCaml. After reading Jon Ludlam's weeknotes (and speaking with him and Anil, I thought it might be nice to add a Javascript backend to those bindings). This was relatively straight-forward using Dune's virtual libraries and is available on Github. With virtual libraries, your own data analysis could (if you wished) depend solely on the proj library and later choose to either link it with proj.c or proj.js depending on where the analysis is being deployed. I briefly looked at WASM support, but quickly realised there was not much appetite for it and trying to compile around the C FFI was going to be hard. Patrick Ferris 2026 4 3 Hilite updates Since I helped relaunch the ocaml.org website a few years ago, I have maintained hilite, a tool for doing build-time syntax highligting. It is being used on the ocaml.org website, in yocaml_markdown and also in Michael's webplats (when it works...). And it was Michael who led me to finally fixing and adding syntax highlighting support for Python and Go. Once I was on this roll, I finally took a look at fixing the long-standing bug of trying to highlight ocaml-mdx code, like the following: # 50.123 ; ; # 1 " file.ml " ; ; In fact, this code now highlights fine, but was the source of the bug. The syntax highlighting grammer confuses the ocaml-mdx # as a toplevel directive and trouble ensues. https://patrick.sirref.org/weeklies-2026/ weeklies-2026 /weeklies-2026/ Patrick Ferris' Weeklies › 2026 Patrick Ferris 2026 4 28 https://patrick.sirref.org/weekly-2026-w17/ weekly-2026-w17 /weekly-2026-w17/ Shell forks revisited Whilst I did get up to some other work last week, I decided to dedicate this weekly to fork-ing in the shell. Patrick Ferris 2026 4 28 Must we fork? I have spent some time trying to figure out: can you implement a shell that never has to fork and run its own code (as opposed to fork+exec-ing, which is necessary)? There are, at least, two bits of POSIX shell semantics that make this hard: concurrency and redirects, particularly in the presence of shell built-ins. Individual commands in a pipeline, according to the POSIX specification, will have their standard input and output connected up before other redirections. The standard input, standard output, or both of a command shall be considered to be assigned by the pipeline before any redirection specified by redirection operators that are part of the command. Additionally: ...each command of a multi-command pipeline is in a subshell environment; as an extension, however, any or all commands in a pipeline may be executed in the current environment. Where a subshell is defined as: Subshell: A shell execution environment, distinguished from the main or current shell execution environment. We will note that subshell child process. In practice, however, the two are used fairly interchangeably given the semantics of spawning a child process. Perhaps the two most crucial aspects of a given command in a subshell are that: It is run in parallel with the other commands. It should be reentrant. In what follows we will assume that we are discussing a multi-command pipeline . Patrick Ferris 2026 4 28 Pipeline parallelism The POSIX specification does not mandate parallelism explicitly for commands in a pipeline. However, in practice the semantics of a pipeline necessitate concurrency at the very least. Suppose we have where produces an infinite amount of output (e.g., cat /dev/urandom) and consumes a finite amount of input (e.g., hxd-xxd -l100). What is the result of running this pipeline with the default POSIX shell options? $ dash -c " cat /dev/urandom | hxd.xxd -l100 " 00000000: 5033 7981 ae44 d475 074c 5361 0a9e 3828 P3y..D.u.LSa..8 00000010: f88c 1f9e 3b0e bcde 528a f78c b1e4 7195 .... ; ...R.....q. 00000030: 57c5 680a 2c81 a999 71b6 708e 95df e720 W.h.,...q.p.... 00000040: 004c 407a 8365 9448 65c6 b435 bed2 4084 .L@z.e.He..5..@. 00000050: 62ea 96ab 2b23 0385 f9d2 fc30 d090 1719 b...+#.....0.... 00000060: 68d1 c514 h... cat: write error: Broken pipe There are three factors that influence the pipeline's execution behaviour (i.e. what is done about the commands that are run and the exit status we return). pipefail: With pipefail set to false (the default), the exit status of the pipeline is entirely decided upon by the exit status of the rightmost command (above ). With pipefail, the pipe will exit with 0 if all commands exit with 0; otherwise the non-zero exit status of the rightmost command is taken. !: If the pipeline begins with ! then this inverts the exit code in the way you might expect. All commands must terminate. Whilst in theory does not terminate, the termination of closes the reading end of the pipe resulting in the Broken pipe system error. Let's replace with the following compound command that also produces, left to its own devices, infinite output on stdout. while echo " some bytes " ; do : done | hxd.xxd -l100 In this example, we no longer have the luxury of being able to fork and exec. The compound command (the while loop) forces our hand -- we must rely on the internal execution logic of the shell to keep it going. How then do we prevent from blocking our shell from spawning ? For most shells, this is where they might fork themselves and allow the while loop to execute in the child. The parent continuing on. Alternatively, the work could continue on in a separate thread (in OCaml parlance, domain). Fibers (green threads, coroutines etc.) are only possible if there are a sufficient number of co-operative yields (which can be hard to track: any control flow or shell built-in would need to ensure they periodically yield to the other commands in the pipeline). With that being said, it would seem we have workarounds to ensure the parallel aspects of pipelines are possible to build without forking the shell state itself. Patrick Ferris 2026 4 28 Command Reentrancy Command reentrancy deals with the fact that commands in a pipeline run in subshells. In practice, this means any should not impact the execution environment of any other . Commands may certainly interfere with one another: a rogue rm might remove a file that some other command was about to read or was in the middle of reading! of the POSIX shell specification describes the execution environment. In Merry we lift this into the ctx type making it completely immutable; the only way to update the context is to destructively declare new fields, e.g., . All shell built-ins work directly from the context as opposed to any state that might also be kept by the process itself (e.g., we do not call getcwd(3) to expand PWD, but instead read it from the context). Most state is handled nicely by this functional implementation of the shell; except open (or not) file descriptors. The exec built-in is particularly thorny in this regard: If exec is specified with no operands, any redirections associated with the exec command shall be made in the current shell execution environment. So, exec 3>&1 says "in the current execution environment map file descriptor 3 to stdout" and exec > /dev/null says "in the current execution environment map stdout to /dev/null. This is often used to silence anything writing to stdout. info echo " INFO: $ 1 " >&3 exec 3>&1 > /dev/null info " Starting up server... " echo " Some noisy program spamming STDOUT " info " All is quiet... " If we are not running shell built-ins inside new child processes, then we must lift the file descriptor table into the context as well and disallow any two built-ins to run in parallel. Consider this example: exec < ./hello.txt while read line ; do echo " Got $ line " >&2 done | exec < ./world.txt ; cat If we allow exec to change the set of open file descriptors of the shell process directly, then individual commands in the pipeline are no longer re-entrant. The pipeline here is not being used to pipe bytes from one command to another, but instead to expose concurrency and the subshell behaviour. The second exec < ./world.txt has no impact on the currently opened file on stdin, the while loop continues to read ./hello.txt. This is straightforward to implement with fork as you can dup2 any files you need. The bookkeeping and isolation is all provided by the POSIX process semantics. Without fork, the file descriptors would have to be saved and restored on every suspension and resumption to provide an illusion of isolation. We could relax our rule to "disallow any two built-ins to run in parallel" by taking a lock on the file descriptor table, but considering the lock is held from the point of resumption until the next suspension it might not achieve very much in practice. Some of the intricacies of scheduling are also covered in Michael Greenberg's An executable and formal semantics of the POSIX shell. To motivate the question, consider the following two pipelines: (1) while true; do echo 5; done | true (2) Both pipelines spawn two processes, both of which use shell builtins exclusively: neither of these pipelines needs to make an execve system call (though some systems may implement true or echo as executables, Smoosh and most shells build them in). Scheduling is an opaque part of the POSIX specification. Indeed, as we have argued here, there is no semantics for scheduling. Here is a similar pipeline run through a script that straces for clone and fork calls on various shells: It might make an interesting use case for effects in OCaml -- whilst we can certainly piggyback on Eio's existing suspension mechanisms, this is slightly different in that we need to ensure some preconditions are true before resuming certain computations. Fiber-local state is insufficient as programs implicitly depend on global state of the process (in particular the FD table). In this sense, we need some form of suspension-resumption hooks. Patrick Ferris 2026 4 18 https://patrick.sirref.org/weekly-2026-w16/ weekly-2026-w16 /weekly-2026-w16/ Landlocked Eio I spent the start of last week wrapping up the "local" CI I mentioned the previous week. Unfortunately it was mostly a packaging nightmare, followed by missing features in ocaml-ci-local. It would be great to have a richer CLI tool for running tests on a directory of git repositories, but unfortunately most of the existing tooling has prioritised the Github/Gitlab etc. APIs. After coming across minimal during the week, I took it as an opportunity to learn a little more about Linux's Landlock mechanism. Patrick Ferris 2026 4 18 https://patrick.sirref.org/landlocked-eio/ landlocked-eio /landlocked-eio/ Landlocked Eio Landlock is a sandboxing mechanism that has existed in the Linux kernel since 5.13. It allows unprivileged processes a means to restrict their ambient rights (e.g. creating a file or opening a network connection). These rights are ambient in the ambient authority sense, i.e., any process can attempt to perform an action without needing access to any particular capability to do so. Ambient systems are often contrasted with capability-based ones. In terms of operating systems, this is most commonly illustrated with access-control-lists-as-columns and capabilities-as-rows in an access matrix. Eio is a capability-based library in OCaml -- resources are provided at the start of the application and can be passed around wherever they are needed. The design is covered in Thomas Leonard's lambda capabilities blog post. Unfortunately, any application (willingly or via a third-party library) may ambiently access OS resources. In OCaml, this may be by using the standard library functions (e.g. Sys.readdir), things from the Unix module, a C FFI function or by spawning a subprocess. I had a go at "landlocking" Eio applications last week after stumbling across minimal's hakoniwa library for process isolation (which is very reminiscent of void processes). module Landlock = Eio_linux . Landlock let / = Eio . Path . / let run_eio file = Eio . traceln " Writing to file " ; Eio . Path . save ~ create : `Exclusive 0o644 file " hello, world! " ; Eio . traceln " Reading file: " Eio . Path . load file ; Eio . Path . unlink file let bad_program = In_channel . with_open_bin " /etc/passwd " In_channel . input_all |> Eio . traceln " /etc/passwd: " let = Eio_linux . run @@ fun env -> Eio . Path . with_open_dir Eio . Stdenv . cwd env / " . " @@ fun dir -> let parent_fd = Eio_unix . Resource . fd_opt fst dir |> Option . get in let file = dir / " hello.txt " in let rules = Landlock . Rule . path_beneath ~ parent_fd ~ flags : Flags . Fs . write_file + read_file + make_reg + remove_file ; in match Landlock . enter rules with | Ok -> run_eio file ; bad_program | Error `Not_supported -> Eio . traceln " Landlock not supported! " Running this program (with the necessary privileges) prints the following. Eio already has a similar function for the BSDs using capsicum. In capsicum, processes enter a capability mode: cap_enter() places the current process into capability mode, a mode of execution in which processes may only issue system calls operating on file descriptors or reading limited global system state. This plays nicely with Eio's capabilities. Within the scope of Eio.Path.with_open_dir, the directory will remain usable after entering "cap" mode. This is not the case with landlock unless you add a specific path_beneath rule to the ruleset (like in the example above). However, you could imagine plumbing the necessary information throughout the Eio_linux backend for it to build a set of rules that can be applied at any given moment in order to mimic the capsicum semantics. Patrick Ferris 2026 4 18 https://patrick.sirref.org/self-host-music/ self-host-music /self-host-music/ Self-hosting Music After my friend, Greta, sent me some articles about music streaming services and their exploitation of artists, I decided enough was enough and setup navidrome. It is compatible with Subsonic and so far I'm very pleased to be streaming my own music to my devices. I am using substreamer on iOS. On my laptop, I tried out Andreas Rossberg's Camp What you got here is an old-school music player heavily inspired by good old I took Anil's OCaml Installer tool for a spin with great success... after a few mishaps mostly related to system dependencies. The fixes in my fork of Camp are just to install the assets using dune and also statically provide the -lpulse flag (which should probably be dynamically picked up). Otherwise, raylib or miniaudio selects the Null playback device. Roll-on cross-ecosystem package management. Patrick Ferris 2026 4 13 https://patrick.sirref.org/weekly-2026-w15/ weekly-2026-w15 /weekly-2026-w15/ Moving away from Github A short week for me as I was hiking with family in and around Porthmadog, Wales. Patrick Ferris 2026 4 13 Reliance on Github I joined Github in 2016, but only began using it in earnest (according to my Github activity) in July 2018: right after Microsoft announced its intent to acquire Github; and just before they closed the $7.5bn deal. I originally joined to boost my résumé and I hoped to receive the odd open-source contribution to projects I was working on (like ppx_deriving_yaml). Since then, I have used Github extensively both at work (Tarides) and for more personal projects. There have been many reasons to jump ship from Github ever since the Microsoft acquisition (e.g. their planned (and quickly unplanned) announcement to charge self-hosted Github runners). Ultimately, I was convinced when I noticed the new AI tool on the homepage with options to select your "versatile and highly intelligent" model. There is no need for me to stay on the platform for my personal projects. AI integration into Github has nullified those earlier motivations for hosting my projects on the site. The number of contributions has increased, but many of them fix superficial elements of an underlying bug. In my (maybe too uncharitable) imagination, it is all too easy for contributors to prompt their built-in LLM to "fix this broken unit test" and for the fix to layer a single-use workaround on top, instead of identifying and correcting the relevant code. This is, at least, what it looks like to me when I receive a new review request on my open-source projects. At the end of last week I set up my own Git infrastructure following Ryan's nixos setup. In the future, the source of truth for my projects will be git.sirref.org (I will likely continue to mirror to tangled.org or Github). Patrick Ferris 2026 4 13 Reduce, Reuse, Recycle Part of self-hosting your own source code is managing some form of automatic testing. I hesitate to describe it as "continuous integration" as the term is confusing and overloaded. Luckily, parts of the OCaml community have been building test infrastructure for a while. I have a proclivity for rebuilding things from scratch be it to learn or to build something the way I want. It is a proclivity I am trying to reign in as I find less and less time to hack on projects. I think this is a good thing. Reusing and recycling existing technologies, in earnest, feels like a pushback on the not-invented-here syndrome that seems to be snowballing thanks to LLMs and code generation tools. To that end, I dusted off my OCurrent knowledge, and repurposed the "local" mode of OCaml-CI to work with my git repositories. After battling through nested layers of git submodules, I managed to Nix-ify this and get it running at tests.sirref.org. I needed to add multi-repository support as well as Current.Monitors for the local git refs. By no means perfect, but it is nice to own the testing infrastructure (and the cost of downloading OCaml images and the compute to run the tests). Patrick Ferris 2026 4 3 https://patrick.sirref.org/weekly-2026-w14/ weekly-2026-w14 /weekly-2026-w14/ Forking in Shells & Library Maintenance This week, Merry got close to executing apt-get install ca-certificates and running Debian's debootstrap scripts correctly. Nathan and I released Ppxlib 0.38.0, I added support to ocaml-proj for compiling to the browser and I fixed a long-standing bug in hilite. Patrick Ferris 2026 4 3 Forking in Merry So far, Merry has managed to support many features of the POSIX shell specification without needing to do a fork(2) without an exec. Or, to put it another way, all the forking that Merry does, will only be followed by running C code. This is actually a feature of Eio's Process API. It is required to make process execution safe in the context of multiple domains in OCaml (mostly due to ensuring consistency within the garbage collector). However, it is perfectly okay to fork and run OCaml code provided there is only a single domain. In a shell, many features of the shell language require that commands (or built-ins, function applications etc.) execute within a subshell. The specification does not go into detail about how shells should implement this, but many choose to fork in order to preserve some state in the parent (for example, the file descriptor table). Some shells try to minimise the number of forks as an optimisation. The korn shell (ksh) is one such shell. Using the notation $(command) will cause command to execute in a subshell of the current ksh. In many instances, ksh will not actually fork/exec a subshell when command is a built-in or a shell function. By trying to adhere to Multicore OCaml's "thou shalt not fork" commandment, Merry is much more similar to ksh in this regard. However, whenever there is an interaction with a shell built-in and semantics that need a child process, things get tricky very quickly. Consider the following: while echo " hello " ; do : done | head -n 3 We have a mix of shell built-ins, normal commands, a compound command (the while loop) and a pipeline. Each individual command in the pipeline requires you to run them in a subshell and set them up before executing them. Without a fork for that first command, you may end up looping forever which is not the intended behaviour here. I had a good conversation with Michael about the implications of this for reproducibility. For now, I have started implementing a fork for some of the shell features whilst still trying to maintain the functional core of the implementation. Patrick Ferris 2026 4 3 An OCaml 5.5 compatible Ppxlib Nathan and I released an OCaml 5.5 compatible version of ppxlib. Patrick Ferris 2026 3 30 https://patrick.sirref.org/ppxlib-release-0-38-0/ ppxlib-release-0-38-0 /ppxlib-release-0-38-0/ Ppxlib Releases › 0.38.0 Nathan and I released ppxlib.0.38.0 last week. Its main feature is full migration support for the upcoming OCaml 5.5 compiler (currently in its alpha3 release). This means supporting the handful of new features landing in OCaml 5.5: modular explicits, external type declarations and arbitrary "local" structure items. As Nathan and I continue to find a plausible maintenance story for ppxlib, Nathan has opened an issue on the OCaml compiler to discuss the idea of adding additional extension points to the language. This comes from the desire to be able to store encoded versions of new OCaml features inside older abstract syntax trees. There are also some nice bug fixes in there too: A potential OOM loop has now been removed. Locations have been restored to long identifiers! Patrick Ferris 2026 4 3 Browser support for ocaml-proj A while ago I built some "modern" bindings to PROJ4 in OCaml. After reading Jon Ludlam's weeknotes (and speaking with him and Anil, I thought it might be nice to add a Javascript backend to those bindings). This was relatively straight-forward using Dune's virtual libraries and is available on Github. With virtual libraries, your own data analysis could (if you wished) depend solely on the proj library and later choose to either link it with proj.c or proj.js depending on where the analysis is being deployed. I briefly looked at WASM support, but quickly realised there was not much appetite for it and trying to compile around the C FFI was going to be hard. Patrick Ferris 2026 4 3 Hilite updates Since I helped relaunch the ocaml.org website a few years ago, I have maintained hilite, a tool for doing build-time syntax highligting. It is being used on the ocaml.org website, in yocaml_markdown and also in Michael's webplats (when it works...). And it was Michael who led me to finally fixing and adding syntax highlighting support for Python and Go. Once I was on this roll, I finally took a look at fixing the long-standing bug of trying to highlight ocaml-mdx code, like the following: # 50.123 ; ; # 1 " file.ml " ; ; In fact, this code now highlights fine, but was the source of the bug. The syntax highlighting grammer confuses the ocaml-mdx # as a toplevel directive and trouble ensues. Patrick Ferris 2026 3 30 https://patrick.sirref.org/weekly-2026-w13/ weekly-2026-w13 /weekly-2026-w13/ Ppxlib release and Merry updates Patrick Ferris 2026 3 30 https://patrick.sirref.org/ppxlib-release-0-38-0/ ppxlib-release-0-38-0 /ppxlib-release-0-38-0/ Ppxlib Releases › 0.38.0 Nathan and I released ppxlib.0.38.0 last week. Its main feature is full migration support for the upcoming OCaml 5.5 compiler (currently in its alpha3 release). This means supporting the handful of new features landing in OCaml 5.5: modular explicits, external type declarations and arbitrary "local" structure items. As Nathan and I continue to find a plausible maintenance story for ppxlib, Nathan has opened an issue on the OCaml compiler to discuss the idea of adding additional extension points to the language. This comes from the desire to be able to store encoded versions of new OCaml features inside older abstract syntax trees. There are also some nice bug fixes in there too: A potential OOM loop has now been removed. Locations have been restored to long identifiers! Patrick Ferris 2026 3 30 Merry updates It turns out writing a POSIX(ish) shell is hard; at least, there is a vast number of slightly obscure features that one needs to support. I think this is surprising because most people writing shell scripts write simple shell scripts; scripts that use a much smaller subset of features. In the same breath, those same developers are likely to use something like apt-get install bash which runs a plethora of more advanced (and non-POSIX) shell scripts! Patrick Ferris 2026 3 30 Exec redirects In a shell script, you can use the built-in command exec to replace the current process with a new one (e.g. exec vim). However, there is a different mode of operation for exec: If exec is specified with no operands, any redirections associated with the exec command shall be made in the current shell execution environment. One of my litmus tests for Merry is the Debian debootstrap scripts (h/t Anil). One thing that it does is the following: err printf " err $ 1 " >&4 exec 4>&1 A lot of detail has been elided for clarity. exec 4>&1 sets up a redirection for the shell's execution environment in which file descriptor 4 is now an alias for standard output. So, writing to 4 (by redirecting a command's standard output to 4 i.e. >&4) will output to where standard output is going (most likely the terminal). For this to work, shell's must dup2 the relevant file descriptors which have the following condition: If the file descriptor newfd was previously open, it is closed before being reused; the close is performed silently (i.e., any errors during the close are not reported by dup2()). This is all very well, unless your program has an important file already open that happens to have file descriptor 4. The Eio Linux and POSIX backends suffer from this problem. Both make use of a file-based synchronisation mechanism for waking up the eventloop should another domain push a completion to the scheduler's run queue. On Linux this is via an eventfd and in POSIX, a pipe. This dup2 will close the eventfd and will likely grind Eio to a halt (or an assert false). For now, I have resorted to vendoring Eio and moving the eventfd file descriptors to higher values, though I have opened a PR to make this more configurable. This bug is quite easy to write up... it was not so easy to find! As a random example, consider the debconf/confmodule script which offers very little room for a buggy implementation! Patrick Ferris 2026 3 30 Outreachy The presentations from the demonstrations for this round of Outreachy are now online! Patrick Ferris 2026 2 9 https://patrick.sirref.org/outreachy-ocaml-tiff/ outreachy-ocaml-tiff /outreachy-ocaml-tiff/ Write support in OCaml TIFF library I am mentoring Tambe Salome during the December 2025 Outreachy round to add support for writing TIFF files in the ocaml-tiff library. You can now see the video of the demonstration day presentation: Patrick Ferris 2026 3 19 https://patrick.sirref.org/weekly-2026-w12/ weekly-2026-w12 /weekly-2026-w12/ A POSIX Shell in OCaml Long time, no weekly. Since the start of this year I have been building a POSIX shell in OCaml called msh (with the underlying library called Merry). Merry is available online now. Patrick Ferris 2026 3 19 A POSIX(ish) shell in OCaml Merry is a POSIX(ish) in OCaml. It uses Eio alongside Morbig (a static parser for POSIX shell). Patrick Ferris 2026 3 19 Why another (POSIX) shell? Shells have been around for a long time. In my research on the notion of metashell I wrote about Louis Pouzin originally coining the term: Patrick Ferris 2025 5 19 https://patrick.sirref.org/pouzin-shell/ pouzin-shell /pouzin-shell/ Metashells › Louis Pouzin's "SHELL" I spent some time reading part of the multics design documentation this week. Louis Pouzin coined the term "SHELL" in this document, and I was reminded yet again just how important it is to be a good writer even as a "computer science researcher". For example, this excerpt from the requirements section of the document The previous definitions imply that a command MUST be designed while keeping in mind the user, sitting at his console, wondering about what might be going on, mistyping or forgetting arguments, even if fully aware of the conventions, and possibly interfering with the command by hasty quits, carriage returns, and other temperamental reactions. And then later, when defining the "SHELL". We may envision a common procedure called automatically by the supervisor whenever a user types in some message at his console, at a time when he has no other process in active execution under console control (presently called command level). This procedure acts as an interface between console messages and subroutine. The purpose of such a procedure is to create a medium of exchange into which one could activate any procedure, inside of another program if it were called. Hereafter, for simplification, we shall refer to that procedure as the "SHELL". It still surprises how little the undergraduate degree in computer science at Cambridge focuses on writing skills. I built Shelter exploring the idea of a shell-like interface that allowed users to undo their shell actions (amongst other cool tricks). Unfortunately shell-like is not enough. Shelter cut many corners to masquerade as a shell (e.g. appending env to understand how a command may have altered the execution environment). I felt it was necessary to make Shelter a "SHELL"! To do that, I needed a solid foundation to build on. msh, the POSIX shell that comes with Merry, is by no means POSIX-complete in terms of features. But it is my daily driver at this point. It includes a pure OCaml rewrite of linenoise (a small, self-contained alternative to the venerable readline) called bruit. If you have docker installed on your machine, you can take it for a spin today: The patrickferris/msh docker image is just for trying it out. It is based on the OCaml 5.3 alpine image. Alternatively, you can build msh from source and have it available in your opam switch. There are many small paper cuts left to patch over, but most of it is porcelain (e.g. ctrl+left-arrow for moving in bruit). Unfortunately, these are the kinds of things that you will immediately stumble upon. Patrick Ferris 2026 3 19 What makes Merry different? Nothing. Merry is supposed to be a solid, POSIX-ish base to build on. Unfortunaltely, as it turns out, the subset of features from the POSIX specification that people actually use... is pretty much all of it. Every possible redirection, variable expansion, shell built-in and compound command make some appearance. Not to mention the non-POSIX bits of shell we all take for granted (e.g. &>-redirection). Merry and msh are useable. You will most likely find bugs if you use them. If, when using msh, you find something obscure happening you can enable debug mode either my setting the variable MSH_DEBUG or by invoking msh with -v -v. Patrick Ferris 2026 3 19 What's next? It is soon time to combine Shelter and Merry into the time-travelling, POSIX-ish shell that I have been trying to build since I first started working on Shelter. For this to be successful, I need Merry to be able to pass plenty of tests and right now that involves trying to install plenty of packages using tools like apk and apt. Patrick Ferris 2026 3 19 Shell MRDT What I am particularly interested in reasoning about, is the execution context in Merry. This value, alongside the file-system, constitutes a fairly deep understanding of the state that changes in each step of a shell's evaluation loop. This was, in terms of Shelter, the missing piece for truly building some kind of MRDT across shell sessions. Patrick Ferris 2026 3 19 TIFF in OCaml I picked up from the excellent Outreachy work of Tambe Salome in getting write-support in ocaml-tiff. We are getting closer to the kind of API I envisaged in this latest round of refinement and review. let / = Eio . Path . / let checkerboard ~ size = let v = Nx . zeros Nx . uint8 size ; size in for row = 0 to size - 1 do for col = 0 to size - 1 do if row + col mod 2 = 0 then Nx . set_item row ; col 254 v done done ; Nx . to_bigarray v let = Eio_posix . run @@ fun env -> Tiff_eio . with_open_out env # cwd / " example.tiff " @@ fun w -> let tif = Tiff . make checkerboard ~ size : 256 in Tiff . to_file tif w I have also been extremely pleased to see further external collaborators appearing: Nicolas helping out with metadata maintainence. Virgile adding support for reading multi-image TIFF files. Backlinks Related Patrick Ferris 2026 2 9 https://patrick.sirref.org/merry/ merry /merry/ Merry https://tangled.org/patrick.sirref.org/merry Merry is a library for building POSIX-ish shells in OCaml. It comes with a (work-in-progress) POSIX shell written using Eio called msh. Supporting the full POSIX specification is no easy feat, however, msh already supports plenty of the more useful bits of the spec. https://patrick.sirref.org/eio/ eio /eio/ Eio https://github.com/ocaml-multicore/eio Eio provides an effects-based direct-style IO stack for OCaml 5 Patrick Ferris https://patrick.sirref.org/mrdts/ mrdts /mrdts/ Mergeable Replicated Data Type Implementation Definition A mergeable replicated data type (MRDT) implementation for a data type is a tuple where: is the set of all possible states at a branch, is the initial state, implements every data type operation, implements the three-way merge strategy. Definition 2.1 from "Certified Mergeable Replicated Data Types". https://patrick.sirref.org/outreachy/ outreachy /outreachy/ Outreachy Project https://www.outreachy.org/ Outreachy provides internships in open source and open science. Outreachy provides internships to people subject to systemic bias and impacted by underrepresentation in the technical industry where they are living. I am one of the coordinators for the OCaml community for Outreachy. I have also mentored multiple three-month internships for the community. Patrick Ferris 2026 2 9 https://patrick.sirref.org/outreachy-ocaml-tiff/ outreachy-ocaml-tiff /outreachy-ocaml-tiff/ Write support in OCaml TIFF library I am mentoring Tambe Salome during the December 2025 Outreachy round to add support for writing TIFF files in the ocaml-tiff library. You can now see the video of the demonstration day presentation: 2022 12 1 https://patrick.sirref.org/topojson-0001/ topojson-0001 /topojson-0001/ Implement a non-blocking, streaming codec for TopoJSON https://github.com/geocaml/ocaml-geojson GeoJSON is a JSON standardised format for handling simple geographical data. One common use of GeoJSON is to add map overlays such as with the popular JavaScript library leaflet. One issue with GeoJSON is redundant data, for example when two polygons share boundaries ideally only one would contain the full data about the boundary, however, with GeoJSON the data is duplicated. TopoJSON is an extension to GeoJSON to encode topology. This allows for redundant data to be removed and file sizes to be greatly reduced. This is often very desirable especially when working with data in the browser. In a previous Outreachy internship a new OCaml library was implemented to provide an OCaml library for TopoJSON, this project will build on this adding more functionality to the library and providing a non-blocking, streaming codec version similar to the geojsone library. 2022 6 1 https://patrick.sirref.org/topojson-0002/ topojson-0002 /topojson-0002/ Extend OCaml's GeoJSON library to support TopoJSON https://github.com/geocaml/ocaml-geojson GeoJSON is a JSON standardised format for handling simple geographical data. One common use of GeoJSON is to add map overlays such as with the popular JavaScript library leaflet. One issue with GeoJSON is redundant data, for example when two polygons share boundaries ideally only one would contain the full data about the boundary, however, with GeoJSON the data is duplicated. TopoJSON is an extension to GeoJSON to encode topology. This allows for redundant data to be removed and file sizes to be greatly reduced. This is often very desirable especially when working with data in the browser. In a previous Outreachy internship a new OCaml library was implemented to provide an OCaml library for TopoJSON, this project will build on this adding more functionality to the library and providing a non-blocking, streaming codec version similar to the geojsone library. 2021 6 1 https://patrick.sirref.org/ocamlorg-0001/ ocamlorg-0001 /ocamlorg-0001/ Improve the ocaml.org Website https://ocaml.org OCaml.org is the main website for OCaml, a functional, typed, high-level programming language. This project revolves around improving the website on multiple different fronts including: layout, accessibility and content. Because a lot of the content of the website is about OCaml and the website itself is written in OCaml, this would be a great opportunity for interns to learn the language and functional programming concepts. https://patrick.sirref.org/shelter/ shelter /shelter/ Shelter https://tangled.sh/@patrick.sirref.org/shelter A shim between a user and the shell that provides greater reproducibility and insight into what your code is doing! Give me Shelter! Shelter combines the caching and reproducibility of Dockerfiles with the ease of interaction of the shell. I'd be interested to use Ryan's package management work as a means to specify the base environment users want. Here is an example of running Shelter and making use of the branchable sessions. shelter> echo hello > hello.txt sheltermain#bbd691arw @ session exp-1]]>shelterexp-1#bbd691arw echo world >> hello.txt]]>shelterexp-1#dd5bab8rw @ session main]]>sheltermain#bbd691arw echo "to the" >> hello.txt]]>sheltermain#b9abef4rw cat hello.txt hello to the]]>sheltermain#d1c1728rw @ session exp-1]]>shelterexp-1#dd5bab8rw @ replay main]]>shelterexp-1#df1b4e1rw cat hello.txt hello to the world]]>shelterexp-1#67162b7rw]]> The @ character allows the user to interact directly with Shelter's build-in operators. @ session exp-1 creates a new session as there is no exp-1 session. It points to the head commit of the branch it came from (git checkout -b exp-1). From there we make changes and switch between branches until we perform a replay. A replay is a rebase without any merge conflict checks, it simply finds the least common ancestor of the two branches and re-applies the commits from your current branch (here exp-1) onto the head of the target branch (here main). Under the hood Shelter is a light-weight, reimplementation of obuilder. It uses the similar ideas, like snapshotting filesystems (e.g. ZFS) and Linux namespaces (e.g. Void). Between the low-level store and the interactive shell parts, there is an Irmin store. https://patrick.sirref.org/ocaml-tiff/ ocaml-tiff /ocaml-tiff/ ocaml-tiff https://github.com/geocaml/ocaml-tiff Part of the geocaml suite of geospatial libraries for OCaml. Contributions