Patrick Ferris 2026 9 12 https://patrick.sirref.org/weekly-2026-w37/ weekly-2026-w37 /weekly-2026-w37/ Analysing some Dockerfiles After spending a few weeks thinking about the semantics of POSIX shells, this week saw me analysing lots of Dockerfiles looking for some evidence for how they are being used. It also saw environment variables landing in Eio thanks to Thomas and Anil. We also had some issues in ppxlib migrations. Patrick Ferris 2026 9 12 Dockerfile Analysis Part of my research at the moment is thinking about how shells are used. What commands are run? What constructs are used (e.g. redirections, compound commands...)? Dockerfiles are a defacto standard means for building container images and for the most part they are a shell with weird semantics and some meta-commands (sounds familiar!). I had some assumptions about the kinds of shell commands people use (myself included) when writing Dockerfiles. For example, making use of AND-lists for grouping commands in RUN instructions. But I had no evidence to back this up. So I ran some analysis over the dataset from this paper from Henkel et al.. I think there's some evidence here to support some assumptions I already had: A significant proportion of RUN instructions in Dockerfiles invoke package managers. A significant proportion of RUN instructions invoke curl or wget. There is something rather Zipfian about how shell built-in utilities are used. There is limited used of exec (and therefore limited use of opening file descriptors directly via the shell, as opposed to a command running in a subprocess). A lot of time is spent changing directories, either temporally (via cd in a RUN instruction, or by using WORKDIR). Lots of ENV setting. Patrick Ferris 2026 9 12 Parsing Dockerfiles For reasons that are probably quite clear, whenever I end up doing a small task like this, I always reach for OCaml to do it. The idea is that I'm invested in that community and hope to see it thrive. And if that means contributing some fixes, libraries or advice along the way, it seems to me that that is time well spent. When I started, I assumed ocaml-dockerfile had support for parsing files; it turns out it doesn't, but I found Thomas G.'s port that does. So I took it for a spin and the results were less than I had hoped for. Of the 219,061 Dockerfiles, 97,123 were not parseable! After a series of commits I managed to get that number down to 808. Patrick Ferris 2026 9 12 Eio process environment What started out seemingly straightforward (add a capability-based API to Eio for accessing environment variables) ended up being a more nuanced discussion about what the best API would be. In the end, Thomas added an Eio.Process.Env API to manipulate soon-to-be opaque environments. This paved the way for the portable environment support. The API Eio exposes is readonly so we do not have to worry too much about race conditions (even though users are free to Unix.putenv). One quirk of this work was reaffirming something I had already stumbled across in Merry. Accessing and searching large environments has a cost! For my rather large Nix environment, we can see the slowdown pretty clearly. The all benchmark calls environment every time and then finds a variable using Eio.Process.Env.get_opt, all_once calls environment once and then Eio.Process.Env.get_opt every time and direct uses the Eio.Process.get_opt directly every time. The differences are a lot less stark when the environment is pretty small. Patrick Ferris 2026 9 12 Ppxlib Migrations Thankfully, ppxlib has been mostly stable for a number of months. In part, thanks to the diligent efforts of Nathan. However, David uncovered some unpleasant bugs in the migration code for some new 5.4 AST features. Unfortunately, it is only for attributes which are a source of pain for ppxlib. I spent some time reviewing the fixes, and it looks like we still need some more work to ensure the attributes attached to package types roundtrip cleanly. Patrick Ferris 2026 9 12 Misc. Some other work I recorded separately and include here for the eager reader. Patrick Ferris 2026 7 12 https://patrick.sirref.org/ocaml-roundup-august-2026/ ocaml-roundup-august-2026 /ocaml-roundup-august-2026/ OCaml Roundup: August 2026 Much of August was spent thinking about POSIX shell semantics, and in particular extending notions from smoosh to include side effects (and maybe concurrency). This is still nascent work, but some ideas and flavours are described below. Patrick Ferris 2026 7 12 Merging Shells The work came from wanting to merge two shell sessions into one. Setting aside the practicalities of an implementation, it makes sense to start first theoretically. When can I merge two shell executions, and when can I not? It would be nice to think of unset DOWNLOAD_DIR as a function. This function takes the shell state and the operating state as arguments and returns a modified shell state and an "unmodified" operating system state (at least for our purposes). Meanwhile, branch-b may have a lot more side effects, but when focusing on the environment only we can see in a sense the two branches are conflict free. Well, sorta. We also need to ensure we do not also read DOWNLOAD_DIR in branch-b if we intend to remove it in their merge. This work, in part, is borrowing ideas from MRDTs and can reuse void processes to help reason about when side effects can occur, as well as open-trace (which is now becoming os-trace). I hope to continue this work, see my latest weekly for more on this. Patrick Ferris 2026 7 12 Eio in the browser I spent some time carving out Eio_browser into a standalone library, something I depend on for Merry to work in the browser. Patrick Ferris 2026 9 12 https://patrick.sirref.org/eio_browser/ eio_browser /eio_browser/ Eio_browser https://git.sirref.org/eio_browser Eio_browser is a backend for Eio that works in the browser. It reuses work I started at Tarides that picked up some momentum but has since been a little dormant. Crucially, it provides a functional standard environment like Eio_main or any of the OS-specific backends. module Stdenv : sig type base = < stdout : Eio_unix.sink_ty r ; stderr : Eio_unix.sink_ty r ; clock : float Eio.Time.clock_ty r ; mono_clock : Eio.Time.Mono.ty r ; secure_random : Eio.Flow.source_ty r ; debug : Eio.Debug.t ; backend_id : string > end val run : Stdenv . base -> unit -> unit References Vimala Soundarapandian Adharsh Kamath Kartik Nagar KC Sivaramakrishnan 2022 https://patrick.sirref.org/kcrsk-mrdts-2022/ kcrsk-mrdts-2022 /kcrsk-mrdts-2022/ Certified mergeable replicated data types Reference https://doi.org/10.1145/3519939.3523735 Proceedings of the 43rd ACM SIGPLAN International Conference on Programming Language Design and Implementation 10.1145/3519939.3523735 Vimala Soundarapandian Adharsh Kamath Kartik Nagar KC Sivaramakrishnan 2022 Abstract Replicated data types (RDTs) are data structures that permit concurrent modification of multiple, potentially geo-distributed, replicas without coordination between them. RDTs are designed in such a way that conflicting operations are eventually deterministically reconciled ensuring convergence. Constructing correct RDTs remains a difficult endeavour due to the complexity of reasoning about independently evolving states of the replicas. With the focus on the correctness of RDTs (and rightly so), existing approaches to RDTs are less efficient compared to their sequential counterparts in terms of the time and space complexity of local operations. This is unfortunate since RDTs are often used in a local-first setting where the local operations far outweigh remote communication. This paper presents PEEPUL, a pragmatic approach to building and verifying efficient RDTs. To make reasoning about correctness easier, we cast RDTs in the mould of the distributed version control system, and equip it with a three-way merge function for reconciling conflicting versions. Further, we go beyond just verifying convergence, and provide a methodology to verify arbitrarily complex specifications. We develop a replication-aware simulation relation to relate RDT specifications to their efficient purely functional implementations. We implement PEEPUL as an F* library that discharges proof obligations to an SMT solver. The verified efficient RDTs are extracted as OCaml code and used in Irmin, a distributed database built on the principles of Git. Jordan Henkel Christian Bird Shuvendu K. Lahiri Thomas Reps 2020 https://patrick.sirref.org/dockerfile2020henkel/ dockerfile2020henkel /dockerfile2020henkel/ Learning from, Understanding, and Supporting DevOps Artifacts for Docker Reference 10.1145/3377811.3380406 Jordan Henkel Christian Bird Shuvendu K. Lahiri Thomas Reps 2020 Abstract Michael Greenberg Austin J. Blatt 2019 https://patrick.sirref.org/greenberg2020smoosh/ greenberg2020smoosh /greenberg2020smoosh/ Executable formal semantics for the POSIX shell Reference https://doi.org/10.1145/3371111 Proc. ACM Program. Lang. 10.1145/3371111 Michael Greenberg Austin J. Blatt 2019 Abstract The POSIX shell is a widely deployed, powerful tool for managing computer systems. The shell is the expert’s control panel, a necessary tool for configuring, compiling, installing, maintaining, and deploying systems. Even though it is powerful, critical infrastructure, the POSIX shell is maligned and misunderstood. Its power and its subtlety are a dangerous combination. We define a formal, mechanized, executable small-step semantics for the POSIX shell, which we call Smoosh. We compared Smoosh against seven other shells that aim for some measure of POSIX compliance (bash, dash, zsh, OSH, mksh, ksh93, and yash). Using three test suites—the POSIX test suite, the Modernish test suite and shell diagnostic, and a test suite of our own device—we found Smoosh’s semantics to be the most conformant to the POSIX standard. Modernish judges Smoosh to have the fewest bugs (just one, from using dash’s parser) and no quirks. To show that our semantics is useful beyond yielding a conformant, executable shell, we also implemented a symbolic stepper to illuminate the subtle behavior of the shell. Smoosh will serve as a foundation for formal study of the POSIX shell, supporting research on and development of new shells, new tooling for shells, and new shell designs. Context https://patrick.sirref.org/weeklies-2026/ weeklies-2026 /weeklies-2026/ Patrick Ferris' Weeklies › 2026 Patrick Ferris 2026 9 12 https://patrick.sirref.org/weekly-2026-w37/ weekly-2026-w37 /weekly-2026-w37/ Analysing some Dockerfiles After spending a few weeks thinking about the semantics of POSIX shells, this week saw me analysing lots of Dockerfiles looking for some evidence for how they are being used. It also saw environment variables landing in Eio thanks to Thomas and Anil. We also had some issues in ppxlib migrations. Patrick Ferris 2026 9 12 Dockerfile Analysis Part of my research at the moment is thinking about how shells are used. What commands are run? What constructs are used (e.g. redirections, compound commands...)? Dockerfiles are a defacto standard means for building container images and for the most part they are a shell with weird semantics and some meta-commands (sounds familiar!). I had some assumptions about the kinds of shell commands people use (myself included) when writing Dockerfiles. For example, making use of AND-lists for grouping commands in RUN instructions. But I had no evidence to back this up. So I ran some analysis over the dataset from this paper from Henkel et al.. I think there's some evidence here to support some assumptions I already had: A significant proportion of RUN instructions in Dockerfiles invoke package managers. A significant proportion of RUN instructions invoke curl or wget. There is something rather Zipfian about how shell built-in utilities are used. There is limited used of exec (and therefore limited use of opening file descriptors directly via the shell, as opposed to a command running in a subprocess). A lot of time is spent changing directories, either temporally (via cd in a RUN instruction, or by using WORKDIR). Lots of ENV setting. Patrick Ferris 2026 9 12 Parsing Dockerfiles For reasons that are probably quite clear, whenever I end up doing a small task like this, I always reach for OCaml to do it. The idea is that I'm invested in that community and hope to see it thrive. And if that means contributing some fixes, libraries or advice along the way, it seems to me that that is time well spent. When I started, I assumed ocaml-dockerfile had support for parsing files; it turns out it doesn't, but I found Thomas G.'s port that does. So I took it for a spin and the results were less than I had hoped for. Of the 219,061 Dockerfiles, 97,123 were not parseable! After a series of commits I managed to get that number down to 808. Patrick Ferris 2026 9 12 Eio process environment What started out seemingly straightforward (add a capability-based API to Eio for accessing environment variables) ended up being a more nuanced discussion about what the best API would be. In the end, Thomas added an Eio.Process.Env API to manipulate soon-to-be opaque environments. This paved the way for the portable environment support. The API Eio exposes is readonly so we do not have to worry too much about race conditions (even though users are free to Unix.putenv). One quirk of this work was reaffirming something I had already stumbled across in Merry. Accessing and searching large environments has a cost! For my rather large Nix environment, we can see the slowdown pretty clearly. The all benchmark calls environment every time and then finds a variable using Eio.Process.Env.get_opt, all_once calls environment once and then Eio.Process.Env.get_opt every time and direct uses the Eio.Process.get_opt directly every time. The differences are a lot less stark when the environment is pretty small. Patrick Ferris 2026 9 12 Ppxlib Migrations Thankfully, ppxlib has been mostly stable for a number of months. In part, thanks to the diligent efforts of Nathan. However, David uncovered some unpleasant bugs in the migration code for some new 5.4 AST features. Unfortunately, it is only for attributes which are a source of pain for ppxlib. I spent some time reviewing the fixes, and it looks like we still need some more work to ensure the attributes attached to package types roundtrip cleanly. Patrick Ferris 2026 9 12 Misc. Some other work I recorded separately and include here for the eager reader. Patrick Ferris 2026 7 12 https://patrick.sirref.org/ocaml-roundup-august-2026/ ocaml-roundup-august-2026 /ocaml-roundup-august-2026/ OCaml Roundup: August 2026 Much of August was spent thinking about POSIX shell semantics, and in particular extending notions from smoosh to include side effects (and maybe concurrency). This is still nascent work, but some ideas and flavours are described below. Patrick Ferris 2026 7 12 Merging Shells The work came from wanting to merge two shell sessions into one. Setting aside the practicalities of an implementation, it makes sense to start first theoretically. When can I merge two shell executions, and when can I not? It would be nice to think of unset DOWNLOAD_DIR as a function. This function takes the shell state and the operating state as arguments and returns a modified shell state and an "unmodified" operating system state (at least for our purposes). Meanwhile, branch-b may have a lot more side effects, but when focusing on the environment only we can see in a sense the two branches are conflict free. Well, sorta. We also need to ensure we do not also read DOWNLOAD_DIR in branch-b if we intend to remove it in their merge. This work, in part, is borrowing ideas from MRDTs and can reuse void processes to help reason about when side effects can occur, as well as open-trace (which is now becoming os-trace). I hope to continue this work, see my latest weekly for more on this. Patrick Ferris 2026 7 12 Eio in the browser I spent some time carving out Eio_browser into a standalone library, something I depend on for Merry to work in the browser. Patrick Ferris 2026 9 12 https://patrick.sirref.org/eio_browser/ eio_browser /eio_browser/ Eio_browser https://git.sirref.org/eio_browser Eio_browser is a backend for Eio that works in the browser. It reuses work I started at Tarides that picked up some momentum but has since been a little dormant. Crucially, it provides a functional standard environment like Eio_main or any of the OS-specific backends. module Stdenv : sig type base = < stdout : Eio_unix.sink_ty r ; stderr : Eio_unix.sink_ty r ; clock : float Eio.Time.clock_ty r ; mono_clock : Eio.Time.Mono.ty r ; secure_random : Eio.Flow.source_ty r ; debug : Eio.Debug.t ; backend_id : string > end val run : Stdenv . base -> unit -> unit Patrick Ferris 2026 8 17 https://patrick.sirref.org/weekly-2026-w33/ weekly-2026-w33 /weekly-2026-w33/ Browsher and Namespaces The work presented constitutes a good bit more than a weeks worth of work. That being said, let's dive in. Patrick Ferris 2026 8 17 https://patrick.sirref.org/browsher/ browsher /browsher/ Browsher After a week or two of abstracting the internals of Merry, I finally got a shell running in the browser. You can have a play with it here: https://patrick.sirref.org/msh The final hurdle was providing an abstract interface to file handles. Eio, the library that so far has enabled much of the OS abstraction, doesn't have a particularly good cross-platform file descriptor API. Eio_unix does extend sources and sinks with the Unix_fd tag, but there is no Eio.Fd.dup2 that provides a platform-agnostic API for file handle manipulation. Even if it did, a POSIX shell does a fair amount of assuming that your file handles are integers which might prove tricky on Windows. The shell in the browser completely reuses the logic of Merry. It also links in a browser-aware subpackage of bruit (the OCaml port of linenoise I wrote) and a few, half-baked POSIX utilities I wrote (like ls and cat). But you can at least run something like: $ echo " hello " > hello.txt $ ls -la . $ cat < hello.txt Being able to, in some sense, run a shell anywhere is actually quite an attractive proposition. Shells, with all of their wonderful footguns, are still a reliable and familiar tool for interacting with a running system. I'm interested in the idea of running shells in more exotic deployments (e.g., unikernels) where it becomes an interface for dynamical reconfiguration or a tool for debugging. Patrick Ferris 2026 8 17 Updates on Eio_mem At the heart of browsher is Eio_mem (an in-memory filesystem for Eio). The Eio_mem API now comes with (and exposes) a number of file descriptor operations: val pipe : Eio . Switch . t -> Fd . t * Fd . t val dup : sw : Eio . Switch . t -> Fd . t -> Fd . t val dup2 : sw : Eio . Switch . t -> src : Fd . t -> tgt : Fd . t -> unit I was happy to see Eio_mem getting some use in Thomas G.'s fork eio-fs where he was able to use it for chaos engineering. Patrick Ferris 2026 8 17 https://patrick.sirref.org/namespace-musings/ namespace-musings /namespace-musings/ Namespaces I spent a few days recently trying to come up with some formalism for the notion of names, namespaces and name resolution. On the surface, it seems bigraphs might be a fruitful approach. The orthogonal treatment of locality and connectivity maps fairly well onto name structure (e.g, Module.X.foo, /foo/bar/hello.txt or foo.example.org) and redirection (e.g., symlinks or CNAMEs). Bigraph reaction rules could then model name resolution. A resolution process can move through the hierarchical name structure and across links at which point you might have some interesting properties to prove or check about a particular namespace system. Additionally, namespaces and resolution mechanism might compose in different ways and this might be worth formalising and comparing. I didn't get much further than modelling some bits of a filesystem in BigraphER. Q{y} | SymLink{x, y}; # Initial State big s0 = Dir. (SymLink{sym_a, file_a} | Q{sym_a} | File{file_a} ); # Bigraphical Reactive System begin brs init s0; rules = [ {resolve_symlink} ]; end]]> Resulting in these two transition states. Bear in mind this was just a quick experiment. The first bigraph shows our query at sym_a and the second bigraph (using the resolve_symlink reduction rule) shows the query at file_a. All of this takes places within Dir. It seems that BigraphER represents the tree structure graphically with black arrows (rather than spatial containment). Patrick Ferris 2026 8 17 Papers I'm working on a paper right now for Merry and Shelter. It is mostly coming together okay; and I am enjoying the creative aspects of this paper, those things with no right answer just "hey, I think this is a good/useful/fun idea". My TikZ-abilities have also had a refresher. Patrick Ferris 2026 8 17 TODOs Over the next week, I hope to finish up a first draft of the aforementioned paper. I also want to publish a backlog of packages I have for OCaml: bruit, ocaml-bibtex, a fresh cut of Graft etc. Patrick Ferris 2026 8 17 Ramblings I had some half-formed thoughts lying around in my forest, so I thought I might include those here: Patrick Ferris 2026 8 11 https://patrick.sirref.org/on-boilerplate/ on-boilerplate /on-boilerplate/ Boilerplate The word boilerplate gained popularity as a colloquialism used by newspaper and magazine editors for pre-prepared editorials and stories that were syndicated across the US. In effect, boilerplates were news stamps that could produce identical content. I have been thinking about the nature of boilerplate code. How is it similar or different to boilerplate in its original meaning. Distinguishing true boilerplate code from code that feels repetitive or boring to write is, I think, important. Argument parsing falls into this category quite well. Consider the following flag I added to Merry's set built-in. let all_export = let doc = " Export all variables, like -o allexport. " in Arg . value & flag & info " a " ~ docv : " ALLEXPORT " ~ doc It uses cmdliner to define a new -a flag that will, in effect, make all variable definitions act like the export built-in. LLMs have been praised for their ability to do this specific task in OCaml. What exactly is boilerplate about this code snippet? It seems to me that there is some sort of ceremony to defining a flag. There is boilerplate in the structure with more meaningful content threaded through. Below, is the example reduced to just the boilerplate replacing the for more meaningful parts with Merlin's typed holes: _. let all_export = let doc = _ in Arg . value & flag & info _ ~ docv : _ ~ doc Arguably the use of flag is quite important too as that is where the magic happens if you want an optional argument, or a positional one etc. As a thought experiment, having agents deliver fill-in-the-blanks style code might be quite useful. In a type system that can support holes (e.g. like in OCaml or Hazel) it becomes a more pleasant exercise, but helps keep you in the loop. I, myself, do not use any AI in the code I write; so I admit that I am pretty far from the problem. The problem I have with this approach is that it is fairly easy to automate in a predictable and easy way! I just added a snippet subcommand to octx, but you could easily wrap this up in whatever your editor of choice is. The more I think on this problem the harder I find good uses for LLMs besides churning out vasts amount of end-to-end code; something I am personally not that interested in doing. It seems to me, to reap the benefits of LLMs, one must relinquish a certain amount of agency; anything less, and you could easily replace its usage with a cheaper, more predictable tool. Patrick Ferris 2026 7 4 https://patrick.sirref.org/weekly-2026-w26/ weekly-2026-w26 /weekly-2026-w26/ Shell Benchmarks, Back to Writing A little late due to travel, but I did a little benchmarking of Merry. Patrick Ferris 2026 7 4 Signal Handling in Merry In order to even run shellbench, I first had to fix some bugs in the shell implementation. Shellbench aims to warm up the shell before starting any benchmarks; it includes this prelude before getting started: __ready= trap __ready=1 HUP kill -HUP " $ MAIN_PID " until " $ __ready " ; do __dummy= ; done Whilst writing Merry, I hadn't really considered that the return context from a trap needed to be taken into consideration! I had assumed they would be silently ignored, how else would we know where to thread that context back into the shell's execution. Even after reading through the specification for trap I am still none the wiser. I've opted to place some sensible signal polling points during the execution of the commands to run any signals that have come in. The single handlers in Merry simply push thunks onto a run queue. Looking at the Dash source code, it seems to be something that happens just before evaluation. Patrick Ferris 2026 7 4 https://patrick.sirref.org/merry-benchmarks/ merry-benchmarks /merry-benchmarks/ Benchmarking Merry Patrick Ferris 2026 7 4 Medium-to-small potatoes I used shellbench to understand where some of the performance pitfalls were coming from. $ ./shellbench -s bash,zsh,msh sample/func.sh ----------------------------------------------------------------------------------- name bash zsh msh ----------------------------------------------------------------------------------- func.sh: no func 775,539 1,124,485 5,229 func.sh: func 398,861 197,169 93,424 ----------------------------------------------------------------------------------- * count: number of executions per second This particular benchmark first runs nothing (i.e. :) and then a function lookup and execution. Surprisingly msh is slower at doing nothing! I perf-ed a run of msh test.sh with just a single : in the file: Patrick Ferris 2026 7 4 Test built-in After making various improvements to Merry that almost halved the memory usage (e.g., removing extraneous string manipulations, adding a fast-path to the PATH variable for resolving programs). I was still pretty far off the performance of these other shells. I decided to have a look at dash which has a simple implementation. This flame graph is for the following shell script: f : ; while __count= $ __count + 1 && " $ __count " ; do f done echo " count is $ __count " And here are some rudimentary benchmarks: $ time -p dash test.sh count is 10001 real 0.01 user 0.01 sys 0.00 $ time -p msh test.sh count is 10001 real 15.53 user 4.28 sys 12.28 It turns out the massive slowdown is very simple to explain. In the dash flamegraph there is a oexpr stack; which comes from the bltin/test.c file. dash, along with the other shells, does not spawn a process for the -test command! Forcing dash to do use the binary gives: $ time -p dash test.sh count is 10001 real 7.55 user 3.95 sys 3.44 Which is much more respectable in terms of performance compared to msh. Plenty of work to do though! Patrick Ferris 2026 7 4 Parsing arithmetic expressions There was some obvious low-hanging fruit to help improve some of the performance differences. The first was pre-parsing arithmetic expressions. Morbig, the static parser, does not implement arithmetic expression parsing; so Merry does it for us. Before, it was on-demand, as we encountered an expression of the form $((<e>)). Now, it happens up front. This should improve tight-loops. Patrick Ferris 2026 7 4 Writing More on this soon, but I have started writing two separate papers bringing all of this work together. Patrick Ferris 2026 6 16 https://patrick.sirref.org/weekly-2026-w25/ weekly-2026-w25 /weekly-2026-w25/ Three weeks in the Shell The last three weeks have been focused work on building Merry. In practice, I wanted to run the following two commands through Merry's shell: msh: Seemingly innocuous. But both commands run thousands of lines of shell script. Whether that's Debian package maintainer scripts or state-machines for processing package release files. What follows is a handful of the bugs and the shell scripts that caused them. Patrick Ferris 2026 6 16 Field Splitting Edge-cases abound in the POSIX shell specification; and field splitting is one of the most serious perpetrators. Field splitting happens after word expansion. It is a known footgun amongst shell script authors. $ count echo " Count: $ # " ; ; FOO= " hello world " ; count $ FOO Count: 2 Quoting variables that need expanding will prevent field splitting from taking place, but it is an easy thing to forget. Field splitting uses the IFS (internal field separator) environment variable to determine how to do the splitting, and it comes with some extra rules beyond "split on these characters". $ IFS= " " ; FOO= " hello world " ; echo $ FOO hello world $ IFS= " " ; FOO= " hello world " ; echo " $ FOO " hello world $ IFS= " : " ; FOO= " a:b:c: " ; echo $ FOO a b c $ IFS= " : " ; FOO= " a:b::c:d " ; echo $ FOO a b c d $ IFS= " : " ; FOO= " :a:b:c " ; echo $ FOO a b c $ IFS= ; FOO= " unset IFS " ; echo $ FOO unset IFS Patrick Ferris 2026 6 16 Variable assignment How a shell stores variables also impacts field splitting. Naively, Merry stored variables as strings, but really they should be stored as their expanded form. Consider this example of saving a function output in a variable called base. iter echo " a " echo " b " echo " c " base= iter for n in $ base ; do echo " Got $ n " done for n in " $ base " ; do echo " Got $ n " done Later, when we come to iterate across the fields of $base we wouldn't be able to reconstruct the intial fields that $(iter) produced if we simply saved the result as a string. Too much information would have been lost. Patrick Ferris 2026 6 16 Redirections Redirections control what file descriptors point to. Each redirection has a lifetime associated with it. For example: cat < README.md really says "open README.md and execute the cat program mapping the process's FD 0 to the FD of README.md". When cat returns that redirection is no longer valid. Other redirections are not so simple. Consider: $ echo " !dlrow olleh " > input.txt $ read line ; echo $ line < input.txt | rev hello world ! There are lots of redirections here too. Firstly the standard input for everything inside the braced command (read line2; echo $line) should be the open file descriptor for input.txt. The standard output of the braced command should be the writing end of a pipe whose reading end is passed to rev as that process's standard input. The final standard output for rev should be unchanged from what it was originally (the shell's standard output). Getting redirections right is hard, consider this snippet from the debootstrap scripts (all in the context of stdout and stderr pointing to a debootstrap.log file and file descriptor 4 pointing at the true stdout). for m in $ MIRRORS ; do pkgdest= " $ TARGET / $ DLDEST pkg " $ s " " $ c " " $ a " " $ m " " $ path " " if ! -e " $ pkgdest " ; then continue ; fi pkgs_to_get= " download_debs " $ m " " $ pkgdest " $ pkgs_to_get 5>&1 1>&6 " if " $ pkgs_to_get " ; then break ; fi done 6>&1 Patrick Ferris 2026 6 16 More niche built-ins As the dust begins to settle on the nuts and bolts of implementing a POSIX-ish shell, new challenges are allowed to emerge from deep within scripts. One such challenge is the seemingly endless (I know, they're finite) shell built-ins! Whilst trying to run debootstrap using its fakechroot variant, I bumped into getopts... a whole command-line parsing shell built-in. Patrick Ferris 2026 6 16 https://patrick.sirref.org/eio_mem/ eio_mem /eio_mem/ An In-memory Filesystem for Eio Towards the end of last week and the beginning of this week, I threw together a prototype for an in-memory filesystem for Eio: Eio_mem. Concretely, this provides a means for constructing an Eio.Fs.dir_ty Eio.Path.t value that is completely in-memory. Thanks to Eio's capability-first approach, any existing code implemented against the _ Eio.Path.t API should just work with the in-memory filesystem. There's a convenient Eio_mem.Fs.load function that can take a path from one filesystem (most likely a normal one from Eio_linux or Eio_posix) and replicate it into the in-memory filesystem. For example, you might have a static web-server pointing to a directory like "/var/www". The following might be the only change you need to make it completely in-memory (example from ocaml-cohttp): let port = ref 8080 in Arg.parse ignore "An HTTP/1.1 server"; Eio_main.run @@ fun env -> + Eio_mem.run @@ fun menv -> Eio.Switch.run @@ fun sw -> - - let htdocs = Eio.Stdenv.cwd env in + + + let htdocs = Eio.Stdenv.cwd menv in let socket = Eio.Net.listen env#net ~sw ~backlog:128 ~reuse_addr:true The keen-eyed reader might have noticed the filter parameter. This can be used to ignore certain paths (here I ignored .git and _build which were adding substantial time to loading the filesystem). Patrick Ferris 2026 6 4 https://patrick.sirref.org/weekly-2026-w23/ weekly-2026-w23 /weekly-2026-w23/ Using meio to debug Merry During a large rewrite of Merry, I introduced a regression causing certain shell programs to hang indefinitely. I thought it might be a good opportunity to try out Meio. First, let's look at a trace of Meio from some real-world programs. For example: meio -- msh -c "sleep 100". One problem that comes up immediately from this example is that the tree hierarchy of tasks and cancellation contexts makes it tricky to reason about the logging. Here, we can see the pipeline's cancellation context clearly labelled pipeline-id1. This is an Eio.Switch.t for the pipeline sleep 100. The actual execution log for sleep 100 bubbles back up to the main fiber (that has no name, so it is called <task>). The daemon fiber, for ensuring the spawned process is reaped, is attached to the switch and is displayed below it. Meio has a long way to go. Most programs are too fast to be monitored in this way. Using something like tokio-console's model might be a better idea here focusing less on the tree strucuture of the program, and more on the performance and monitoring of individual tasks. Without this, it turns out Meio was not all that useful for debugging this specific issue (which ended up being some bad pipe management on my part). Patrick Ferris 2026 6 1 https://patrick.sirref.org/weekly-2026-w22/ weekly-2026-w22 /weekly-2026-w22/ Catching up in May I never quite found the time in May to write weeknotes. I think that's, in some ways, quite a good thing. My philosophy on weeknotes is not that they are a public punchcard of my progress. They are a tool for organising thoughts and keeping my colleagues abreast of what I am up to. Only the latter suggests any necessity for timeliness and given I frequently message, call and zulip them, if a few weeks fall between the cracks then that's okay! All that being said, I have tried to provide some thoughts and things I have been up to over the course of May and not just week 22 of the year. Patrick Ferris 2026 6 1 Tools for Shells Some tools and libraries related to Merry. Patrick Ferris 2026 6 1 Meio The shell work continues, and as the complexity internally increases I have found myself wishing for some more tools for debugging concurrent Eio programs. Given enough pipes and compound commands, it is fairly easy to introduce a fair deal of concurrency to your shell programs. To this end I hope I have improved meio somewhat in this fork. In particular, there were several things I wanted to improve. Simple UI improvements -- the set of tools for building TUI UIs from nottui are, on closer inspection, rather good and simple. For example, turning a particular UI component into a scrollable pane is as simple as wrapping it in a call to this function. Logging more runtime events -- Eio programs emit information via OCaml's runtime events. This is how meio knows which fiber is currently active for example. It also emits information like "the current fiber created a new promise" and "this fiber is blocked waiting for a promise". I have not figured out an intuitive way for representing this in meio's tree-view, but I did turn these into "system logs" that appear in the per-fiber logs which is already an improvement. Thanks to Thomas for helping understand the runtime event sequence for logging this information. Timestamping where possible -- all runtime events come with a timestamp, these were mostly discarded by meio. Some aspects of meio made it clear that labelling Eio.Switches and Eio.Promises might be a good practice. After reading Michael's Eio forays, I can't help but wonder if a more opinionated Eio for Systems Programmers guide might be useful. I think the Eio README does a wonderful job of introducing most of the low-level concepts, but not quite how they are used by the working systems programmer. Patrick Ferris 2026 6 1 Glob A little dismayed at the lack of progress I was making with Merry, I switched to a specific problem I knew I wanted to fix -- removing the dependency on glob(3), replacing it with a pure OCaml implementation. let get_txts fs = Glob_eio . glob fs Glob . of_component dir_sep ; string " home " ; dir_sep ; star ; string " .txt " Anil provided some issues with fixes too which was very appreciated. Beyond factoring out fork+exec, Merry should now be portable to the browser. Of course there are lots of other refactorings needed to (e.g. a fake-filesystem) -- but I'm interested in how I can use Eio's filesystem capabilities to do this pretty seemlessly. Having an emeddable, pure OCaml shell may also unlock some cool debugging capabilities for unikernels too! Patrick Ferris 2026 6 1 Geocaml Michael has been valiantly trying some of the Geocaml libraries in his work on rendering LiDAR point clouds. Where possible, I have been reviewing Michael's PRs and helping implement features he might find useful. I also took his Laserver for a spin, and suggested a few changes. Patrick Ferris 2026 6 1 Rtree After a comical series of attempts, I (or rather Anil) released rtree.0.2.0 which contained: A bug fix where not all the tree values were being returned. A new bounds function for getting an Rtree's maximum bound. A set of simple removal functions. Patrick Ferris 2026 6 1 PROJ Michael added some new functions to ocaml-proj. Previously, I had worked on making PROJ more portable by using virtual libraries. This was a little restrictive, as it forced users to always implement a JS version of any functionality that exists in the native version of PROJ. To make it easier to contribute (and allow the C bindings to contain more specific bindings) I carved out the two virtual library implementations into standalone libraries, and reimplemented the virtual library support using these libraries. Now projects can depend directly on proj_c if they wish to have more features in exchange for being less portable. Patrick Ferris 2026 6 1 Ppxlib Thankfully, work on ppxlib has been relatively quiet. Nathan also opened a PR for initial OCaml 5.6 support. Keeping on top of 5.6 features is probably the right way to go about this (rather than one giant PR during the 5.6 release cycle). I'm particularly happy to have some better migration error handling. The previous errors, though uncommon, were confusing ppx users and hopefully this will help ppx authors and ppxlib maintainers triaging important encoding bugs or feature wishes. Patrick Ferris 2026 6 1 Self-hosting I now have a fairly workable setup for self-hosting source code with git.sirref.org and tests.sirref.org. One piece that is missing, is a publishing workflow. I spent a little time building a YOCaml static-site generator for packaging OCaml libraries. Unfortunately, this was full of rabbit holes and busy work which was frustrating. Patrick Ferris 2026 6 1 Tar-eio One such rabbit hole was tar-eio. It was in a bit of a sorry state, so I took the opportunity to improve the API for new users. Tarring a directory is now as simple as: let foo = Eio . Path . cwd / " foo " in let foo_tar = Eio . Path . cwd / " foo.tar " in Eio . Path . with_open_out ~ create : `Or_truncate 0o755 foo_tar @@ fun foo_tar -> Eio . Switch . run @@ fun sw -> let t = Tar_eio . create ~ sw foo in Tar_eio . run t Tar_eio . flow_of_file tar There was also a series of security issues that the tar library had that were fixed too, you can read about those in Thomas's weeknotes. Patrick Ferris 2026 4 28 https://patrick.sirref.org/weekly-2026-w17/ weekly-2026-w17 /weekly-2026-w17/ Shell forks revisited Whilst I did get up to some other work last week, I decided to dedicate this weekly to fork-ing in the shell. Patrick Ferris 2026 4 28 Must we fork? I have spent some time trying to figure out: can you implement a shell that never has to fork and run its own code (as opposed to fork+exec-ing, which is necessary)? There are, at least, two bits of POSIX shell semantics that make this hard: concurrency and redirects, particularly in the presence of shell built-ins. Individual commands in a pipeline, according to the POSIX specification, will have their standard input and output connected up before other redirections. The standard input, standard output, or both of a command shall be considered to be assigned by the pipeline before any redirection specified by redirection operators that are part of the command. Additionally: ...each command of a multi-command pipeline is in a subshell environment; as an extension, however, any or all commands in a pipeline may be executed in the current environment. Where a subshell is defined as: Subshell: A shell execution environment, distinguished from the main or current shell execution environment. We will note that subshell child process. In practice, however, the two are used fairly interchangeably given the semantics of spawning a child process. Perhaps the two most crucial aspects of a given command in a subshell are that: It is run in parallel with the other commands. It should be reentrant. In what follows we will assume that we are discussing a multi-command pipeline . Patrick Ferris 2026 4 28 Pipeline parallelism The POSIX specification does not mandate parallelism explicitly for commands in a pipeline. However, in practice the semantics of a pipeline necessitate concurrency at the very least. Suppose we have where produces an infinite amount of output (e.g., cat /dev/urandom) and consumes a finite amount of input (e.g., hxd-xxd -l100). What is the result of running this pipeline with the default POSIX shell options? $ dash -c " cat /dev/urandom | hxd.xxd -l100 " 00000000: 5033 7981 ae44 d475 074c 5361 0a9e 3828 P3y..D.u.LSa..8 00000010: f88c 1f9e 3b0e bcde 528a f78c b1e4 7195 .... ; ...R.....q. 00000030: 57c5 680a 2c81 a999 71b6 708e 95df e720 W.h.,...q.p.... 00000040: 004c 407a 8365 9448 65c6 b435 bed2 4084 .L@z.e.He..5..@. 00000050: 62ea 96ab 2b23 0385 f9d2 fc30 d090 1719 b...+#.....0.... 00000060: 68d1 c514 h... cat: write error: Broken pipe There are three factors that influence the pipeline's execution behaviour (i.e. what is done about the commands that are run and the exit status we return). pipefail: With pipefail set to false (the default), the exit status of the pipeline is entirely decided upon by the exit status of the rightmost command (above ). With pipefail, the pipe will exit with 0 if all commands exit with 0; otherwise the non-zero exit status of the rightmost command is taken. !: If the pipeline begins with ! then this inverts the exit code in the way you might expect. All commands must terminate. Whilst in theory does not terminate, the termination of closes the reading end of the pipe resulting in the Broken pipe system error. Let's replace with the following compound command that also produces, left to its own devices, infinite output on stdout. while echo " some bytes " ; do : done | hxd.xxd -l100 In this example, we no longer have the luxury of being able to fork and exec. The compound command (the while loop) forces our hand -- we must rely on the internal execution logic of the shell to keep it going. How then do we prevent from blocking our shell from spawning ? For most shells, this is where they might fork themselves and allow the while loop to execute in the child. The parent continuing on. Alternatively, the work could continue on in a separate thread (in OCaml parlance, domain). Fibers (green threads, coroutines etc.) are only possible if there are a sufficient number of co-operative yields (which can be hard to track: any control flow or shell built-in would need to ensure they periodically yield to the other commands in the pipeline). With that being said, it would seem we have workarounds to ensure the parallel aspects of pipelines are possible to build without forking the shell state itself. Patrick Ferris 2026 4 28 Command Reentrancy Command reentrancy deals with the fact that commands in a pipeline run in subshells. In practice, this means any should not impact the execution environment of any other . Commands may certainly interfere with one another: a rogue rm might remove a file that some other command was about to read or was in the middle of reading! of the POSIX shell specification describes the execution environment. In Merry we lift this into the ctx type making it completely immutable; the only way to update the context is to destructively declare new fields, e.g., . All shell built-ins work directly from the context as opposed to any state that might also be kept by the process itself (e.g., we do not call getcwd(3) to expand PWD, but instead read it from the context). Most state is handled nicely by this functional implementation of the shell; except open (or not) file descriptors. The exec built-in is particularly thorny in this regard: If exec is specified with no operands, any redirections associated with the exec command shall be made in the current shell execution environment. So, exec 3>&1 says "in the current execution environment map file descriptor 3 to stdout" and exec > /dev/null says "in the current execution environment map stdout to /dev/null. This is often used to silence anything writing to stdout. info echo " INFO: $ 1 " >&3 exec 3>&1 > /dev/null info " Starting up server... " echo " Some noisy program spamming STDOUT " info " All is quiet... " If we are not running shell built-ins inside new child processes, then we must lift the file descriptor table into the context as well and disallow any two built-ins to run in parallel. Consider this example: exec < ./hello.txt while read line ; do echo " Got $ line " >&2 done | exec < ./world.txt ; cat If we allow exec to change the set of open file descriptors of the shell process directly, then individual commands in the pipeline are no longer re-entrant. The pipeline here is not being used to pipe bytes from one command to another, but instead to expose concurrency and the subshell behaviour. The second exec < ./world.txt has no impact on the currently opened file on stdin, the while loop continues to read ./hello.txt. This is straightforward to implement with fork as you can dup2 any files you need. The bookkeeping and isolation is all provided by the POSIX process semantics. Without fork, the file descriptors would have to be saved and restored on every suspension and resumption to provide an illusion of isolation. We could relax our rule to "disallow any two built-ins to run in parallel" by taking a lock on the file descriptor table, but considering the lock is held from the point of resumption until the next suspension it might not achieve very much in practice. Some of the intricacies of scheduling are also covered in Michael Greenberg's An executable and formal semantics of the POSIX shell. To motivate the question, consider the following two pipelines: (1) while true; do echo 5; done | true (2) Both pipelines spawn two processes, both of which use shell builtins exclusively: neither of these pipelines needs to make an execve system call (though some systems may implement true or echo as executables, Smoosh and most shells build them in). Scheduling is an opaque part of the POSIX specification. Indeed, as we have argued here, there is no semantics for scheduling. Here is a similar pipeline run through a script that straces for clone and fork calls on various shells: It might make an interesting use case for effects in OCaml -- whilst we can certainly piggyback on Eio's existing suspension mechanisms, this is slightly different in that we need to ensure some preconditions are true before resuming certain computations. Fiber-local state is insufficient as programs implicitly depend on global state of the process (in particular the FD table). In this sense, we need some form of suspension-resumption hooks. Patrick Ferris 2026 4 18 https://patrick.sirref.org/weekly-2026-w16/ weekly-2026-w16 /weekly-2026-w16/ Landlocked Eio I spent the start of last week wrapping up the "local" CI I mentioned the previous week. Unfortunately it was mostly a packaging nightmare, followed by missing features in ocaml-ci-local. It would be great to have a richer CLI tool for running tests on a directory of git repositories, but unfortunately most of the existing tooling has prioritised the Github/Gitlab etc. APIs. After coming across minimal during the week, I took it as an opportunity to learn a little more about Linux's Landlock mechanism. Patrick Ferris 2026 4 18 https://patrick.sirref.org/landlocked-eio/ landlocked-eio /landlocked-eio/ Landlocked Eio Landlock is a sandboxing mechanism that has existed in the Linux kernel since 5.13. It allows unprivileged processes a means to restrict their ambient rights (e.g. creating a file or opening a network connection). These rights are ambient in the ambient authority sense, i.e., any process can attempt to perform an action without needing access to any particular capability to do so. Ambient systems are often contrasted with capability-based ones. In terms of operating systems, this is most commonly illustrated with access-control-lists-as-columns and capabilities-as-rows in an access matrix. Eio is a capability-based library in OCaml -- resources are provided at the start of the application and can be passed around wherever they are needed. The design is covered in Thomas Leonard's lambda capabilities blog post. Unfortunately, any application (willingly or via a third-party library) may ambiently access OS resources. In OCaml, this may be by using the standard library functions (e.g. Sys.readdir), things from the Unix module, a C FFI function or by spawning a subprocess. I had a go at "landlocking" Eio applications last week after stumbling across minimal's hakoniwa library for process isolation (which is very reminiscent of void processes). module Landlock = Eio_linux . Landlock let / = Eio . Path . / let run_eio file = Eio . traceln " Writing to file " ; Eio . Path . save ~ create : `Exclusive 0o644 file " hello, world! " ; Eio . traceln " Reading file: " Eio . Path . load file ; Eio . Path . unlink file let bad_program = In_channel . with_open_bin " /etc/passwd " In_channel . input_all |> Eio . traceln " /etc/passwd: " let = Eio_linux . run @@ fun env -> Eio . Path . with_open_dir Eio . Stdenv . cwd env / " . " @@ fun dir -> let parent_fd = Eio_unix . Resource . fd_opt fst dir |> Option . get in let file = dir / " hello.txt " in let rules = Landlock . Rule . path_beneath ~ parent_fd ~ flags : Flags . Fs . write_file + read_file + make_reg + remove_file ; in match Landlock . enter rules with | Ok -> run_eio file ; bad_program | Error `Not_supported -> Eio . traceln " Landlock not supported! " Running this program (with the necessary privileges) prints the following. Eio already has a similar function for the BSDs using capsicum. In capsicum, processes enter a capability mode: cap_enter() places the current process into capability mode, a mode of execution in which processes may only issue system calls operating on file descriptors or reading limited global system state. This plays nicely with Eio's capabilities. Within the scope of Eio.Path.with_open_dir, the directory will remain usable after entering "cap" mode. This is not the case with landlock unless you add a specific path_beneath rule to the ruleset (like in the example above). However, you could imagine plumbing the necessary information throughout the Eio_linux backend for it to build a set of rules that can be applied at any given moment in order to mimic the capsicum semantics. Patrick Ferris 2026 4 18 https://patrick.sirref.org/self-host-music/ self-host-music /self-host-music/ Self-hosting Music After my friend, Greta, sent me some articles about music streaming services and their exploitation of artists, I decided enough was enough and setup navidrome. It is compatible with Subsonic and so far I'm very pleased to be streaming my own music to my devices. I am using substreamer on iOS. On my laptop, I tried out Andreas Rossberg's Camp What you got here is an old-school music player heavily inspired by good old I took Anil's OCaml Installer tool for a spin with great success... after a few mishaps mostly related to system dependencies. The fixes in my fork of Camp are just to install the assets using dune and also statically provide the -lpulse flag (which should probably be dynamically picked up). Otherwise, raylib or miniaudio selects the Null playback device. Roll-on cross-ecosystem package management. Patrick Ferris 2026 4 13 https://patrick.sirref.org/weekly-2026-w15/ weekly-2026-w15 /weekly-2026-w15/ Moving away from Github A short week for me as I was hiking with family in and around Porthmadog, Wales. Patrick Ferris 2026 4 13 Reliance on Github I joined Github in 2016, but only began using it in earnest (according to my Github activity) in July 2018: right after Microsoft announced its intent to acquire Github; and just before they closed the $7.5bn deal. I originally joined to boost my résumé and I hoped to receive the odd open-source contribution to projects I was working on (like ppx_deriving_yaml). Since then, I have used Github extensively both at work (Tarides) and for more personal projects. There have been many reasons to jump ship from Github ever since the Microsoft acquisition (e.g. their planned (and quickly unplanned) announcement to charge self-hosted Github runners). Ultimately, I was convinced when I noticed the new AI tool on the homepage with options to select your "versatile and highly intelligent" model. There is no need for me to stay on the platform for my personal projects. AI integration into Github has nullified those earlier motivations for hosting my projects on the site. The number of contributions has increased, but many of them fix superficial elements of an underlying bug. In my (maybe too uncharitable) imagination, it is all too easy for contributors to prompt their built-in LLM to "fix this broken unit test" and for the fix to layer a single-use workaround on top, instead of identifying and correcting the relevant code. This is, at least, what it looks like to me when I receive a new review request on my open-source projects. At the end of last week I set up my own Git infrastructure following Ryan's nixos setup. In the future, the source of truth for my projects will be git.sirref.org (I will likely continue to mirror to tangled.org or Github). Patrick Ferris 2026 4 13 Reduce, Reuse, Recycle Part of self-hosting your own source code is managing some form of automatic testing. I hesitate to describe it as "continuous integration" as the term is confusing and overloaded. Luckily, parts of the OCaml community have been building test infrastructure for a while. I have a proclivity for rebuilding things from scratch be it to learn or to build something the way I want. It is a proclivity I am trying to reign in as I find less and less time to hack on projects. I think this is a good thing. Reusing and recycling existing technologies, in earnest, feels like a pushback on the not-invented-here syndrome that seems to be snowballing thanks to LLMs and code generation tools. To that end, I dusted off my OCurrent knowledge, and repurposed the "local" mode of OCaml-CI to work with my git repositories. After battling through nested layers of git submodules, I managed to Nix-ify this and get it running at tests.sirref.org. I needed to add multi-repository support as well as Current.Monitors for the local git refs. By no means perfect, but it is nice to own the testing infrastructure (and the cost of downloading OCaml images and the compute to run the tests). Patrick Ferris 2026 4 3 https://patrick.sirref.org/weekly-2026-w14/ weekly-2026-w14 /weekly-2026-w14/ Forking in Shells & Library Maintenance This week, Merry got close to executing apt-get install ca-certificates and running Debian's debootstrap scripts correctly. Nathan and I released Ppxlib 0.38.0, I added support to ocaml-proj for compiling to the browser and I fixed a long-standing bug in hilite. Patrick Ferris 2026 4 3 Forking in Merry So far, Merry has managed to support many features of the POSIX shell specification without needing to do a fork(2) without an exec. Or, to put it another way, all the forking that Merry does, will only be followed by running C code. This is actually a feature of Eio's Process API. It is required to make process execution safe in the context of multiple domains in OCaml (mostly due to ensuring consistency within the garbage collector). However, it is perfectly okay to fork and run OCaml code provided there is only a single domain. In a shell, many features of the shell language require that commands (or built-ins, function applications etc.) execute within a subshell. The specification does not go into detail about how shells should implement this, but many choose to fork in order to preserve some state in the parent (for example, the file descriptor table). Some shells try to minimise the number of forks as an optimisation. The korn shell (ksh) is one such shell. Using the notation $(command) will cause command to execute in a subshell of the current ksh. In many instances, ksh will not actually fork/exec a subshell when command is a built-in or a shell function. By trying to adhere to Multicore OCaml's "thou shalt not fork" commandment, Merry is much more similar to ksh in this regard. However, whenever there is an interaction with a shell built-in and semantics that need a child process, things get tricky very quickly. Consider the following: while echo " hello " ; do : done | head -n 3 We have a mix of shell built-ins, normal commands, a compound command (the while loop) and a pipeline. Each individual command in the pipeline requires you to run them in a subshell and set them up before executing them. Without a fork for that first command, you may end up looping forever which is not the intended behaviour here. I had a good conversation with Michael about the implications of this for reproducibility. For now, I have started implementing a fork for some of the shell features whilst still trying to maintain the functional core of the implementation. Patrick Ferris 2026 4 3 An OCaml 5.5 compatible Ppxlib Nathan and I released an OCaml 5.5 compatible version of ppxlib. Patrick Ferris 2026 3 30 https://patrick.sirref.org/ppxlib-release-0-38-0/ ppxlib-release-0-38-0 /ppxlib-release-0-38-0/ Ppxlib Releases › 0.38.0 Nathan and I released ppxlib.0.38.0 last week. Its main feature is full migration support for the upcoming OCaml 5.5 compiler (currently in its alpha3 release). This means supporting the handful of new features landing in OCaml 5.5: modular explicits, external type declarations and arbitrary "local" structure items. As Nathan and I continue to find a plausible maintenance story for ppxlib, Nathan has opened an issue on the OCaml compiler to discuss the idea of adding additional extension points to the language. This comes from the desire to be able to store encoded versions of new OCaml features inside older abstract syntax trees. There are also some nice bug fixes in there too: A potential OOM loop has now been removed. Locations have been restored to long identifiers! Patrick Ferris 2026 4 3 https://patrick.sirref.org/ocaml-proj-virt/ ocaml-proj-virt /ocaml-proj-virt/ Browser support for ocaml-proj A while ago I built some "modern" bindings to PROJ4 in OCaml. After reading Jon Ludlam's weeknotes (and speaking with him and Anil, I thought it might be nice to add a Javascript backend to those bindings). This was relatively straight-forward using Dune's virtual libraries and is available on Github. With virtual libraries, your own data analysis could (if you wished) depend solely on the proj library and later choose to either link it with proj.c or proj.js depending on where the analysis is being deployed. I briefly looked at WASM support, but quickly realised there was not much appetite for it and trying to compile around the C FFI was going to be hard. Patrick Ferris 2026 4 3 Hilite updates Since I helped relaunch the ocaml.org website a few years ago, I have maintained hilite, a tool for doing build-time syntax highligting. It is being used on the ocaml.org website, in yocaml_markdown and also in Michael's webplats (when it works...). And it was Michael who led me to finally fixing and adding syntax highlighting support for Python and Go. Once I was on this roll, I finally took a look at fixing the long-standing bug of trying to highlight ocaml-mdx code, like the following: # 50.123 ; ; # 1 " file.ml " ; ; In fact, this code now highlights fine, but was the source of the bug. The syntax highlighting grammer confuses the ocaml-mdx # as a toplevel directive and trouble ensues. Patrick Ferris 2026 3 30 https://patrick.sirref.org/weekly-2026-w13/ weekly-2026-w13 /weekly-2026-w13/ Ppxlib release and Merry updates Patrick Ferris 2026 3 30 https://patrick.sirref.org/ppxlib-release-0-38-0/ ppxlib-release-0-38-0 /ppxlib-release-0-38-0/ Ppxlib Releases › 0.38.0 Nathan and I released ppxlib.0.38.0 last week. Its main feature is full migration support for the upcoming OCaml 5.5 compiler (currently in its alpha3 release). This means supporting the handful of new features landing in OCaml 5.5: modular explicits, external type declarations and arbitrary "local" structure items. As Nathan and I continue to find a plausible maintenance story for ppxlib, Nathan has opened an issue on the OCaml compiler to discuss the idea of adding additional extension points to the language. This comes from the desire to be able to store encoded versions of new OCaml features inside older abstract syntax trees. There are also some nice bug fixes in there too: A potential OOM loop has now been removed. Locations have been restored to long identifiers! Patrick Ferris 2026 3 30 Merry updates It turns out writing a POSIX(ish) shell is hard; at least, there is a vast number of slightly obscure features that one needs to support. I think this is surprising because most people writing shell scripts write simple shell scripts; scripts that use a much smaller subset of features. In the same breath, those same developers are likely to use something like apt-get install bash which runs a plethora of more advanced (and non-POSIX) shell scripts! Patrick Ferris 2026 3 30 Exec redirects In a shell script, you can use the built-in command exec to replace the current process with a new one (e.g. exec vim). However, there is a different mode of operation for exec: If exec is specified with no operands, any redirections associated with the exec command shall be made in the current shell execution environment. One of my litmus tests for Merry is the Debian debootstrap scripts (h/t Anil). One thing that it does is the following: err printf " err $ 1 " >&4 exec 4>&1 A lot of detail has been elided for clarity. exec 4>&1 sets up a redirection for the shell's execution environment in which file descriptor 4 is now an alias for standard output. So, writing to 4 (by redirecting a command's standard output to 4 i.e. >&4) will output to where standard output is going (most likely the terminal). For this to work, shell's must dup2 the relevant file descriptors which have the following condition: If the file descriptor newfd was previously open, it is closed before being reused; the close is performed silently (i.e., any errors during the close are not reported by dup2()). This is all very well, unless your program has an important file already open that happens to have file descriptor 4. The Eio Linux and POSIX backends suffer from this problem. Both make use of a file-based synchronisation mechanism for waking up the eventloop should another domain push a completion to the scheduler's run queue. On Linux this is via an eventfd and in POSIX, a pipe. This dup2 will close the eventfd and will likely grind Eio to a halt (or an assert false). For now, I have resorted to vendoring Eio and moving the eventfd file descriptors to higher values, though I have opened a PR to make this more configurable. This bug is quite easy to write up... it was not so easy to find! As a random example, consider the debconf/confmodule script which offers very little room for a buggy implementation! Patrick Ferris 2026 3 30 Outreachy The presentations from the demonstrations for this round of Outreachy are now online! Patrick Ferris 2026 2 9 https://patrick.sirref.org/outreachy-ocaml-tiff/ outreachy-ocaml-tiff /outreachy-ocaml-tiff/ Write support in OCaml TIFF library I am mentoring Tambe Salome during the December 2025 Outreachy round to add support for writing TIFF files in the ocaml-tiff library. You can now see the video of the demonstration day presentation: Patrick Ferris 2026 3 19 https://patrick.sirref.org/weekly-2026-w12/ weekly-2026-w12 /weekly-2026-w12/ A POSIX Shell in OCaml Long time, no weekly. Since the start of this year I have been building a POSIX shell in OCaml called msh (with the underlying library called Merry). Merry is available online now. Patrick Ferris 2026 3 19 A POSIX(ish) shell in OCaml Merry is a POSIX(ish) in OCaml. It uses Eio alongside Morbig (a static parser for POSIX shell). Patrick Ferris 2026 3 19 Why another (POSIX) shell? Shells have been around for a long time. In my research on the notion of metashell I wrote about Louis Pouzin originally coining the term: Patrick Ferris 2025 5 19 https://patrick.sirref.org/pouzin-shell/ pouzin-shell /pouzin-shell/ Metashells › Louis Pouzin's "SHELL" I spent some time reading part of the multics design documentation this week. Louis Pouzin coined the term "SHELL" in this document, and I was reminded yet again just how important it is to be a good writer even as a "computer science researcher". For example, this excerpt from the requirements section of the document The previous definitions imply that a command MUST be designed while keeping in mind the user, sitting at his console, wondering about what might be going on, mistyping or forgetting arguments, even if fully aware of the conventions, and possibly interfering with the command by hasty quits, carriage returns, and other temperamental reactions. And then later, when defining the "SHELL". We may envision a common procedure called automatically by the supervisor whenever a user types in some message at his console, at a time when he has no other process in active execution under console control (presently called command level). This procedure acts as an interface between console messages and subroutine. The purpose of such a procedure is to create a medium of exchange into which one could activate any procedure, inside of another program if it were called. Hereafter, for simplification, we shall refer to that procedure as the "SHELL". It still surprises how little the undergraduate degree in computer science at Cambridge focuses on writing skills. I built Shelter exploring the idea of a shell-like interface that allowed users to undo their shell actions (amongst other cool tricks). Unfortunately shell-like is not enough. Shelter cut many corners to masquerade as a shell (e.g. appending env to understand how a command may have altered the execution environment). I felt it was necessary to make Shelter a "SHELL"! To do that, I needed a solid foundation to build on. msh, the POSIX shell that comes with Merry, is by no means POSIX-complete in terms of features. But it is my daily driver at this point. It includes a pure OCaml rewrite of linenoise (a small, self-contained alternative to the venerable readline) called bruit. If you have docker installed on your machine, you can take it for a spin today: The patrickferris/msh docker image is just for trying it out. It is based on the OCaml 5.3 alpine image. Alternatively, you can build msh from source and have it available in your opam switch. There are many small paper cuts left to patch over, but most of it is porcelain (e.g. ctrl+left-arrow for moving in bruit). Unfortunately, these are the kinds of things that you will immediately stumble upon. Patrick Ferris 2026 3 19 What makes Merry different? Nothing. Merry is supposed to be a solid, POSIX-ish base to build on. Unfortunaltely, as it turns out, the subset of features from the POSIX specification that people actually use... is pretty much all of it. Every possible redirection, variable expansion, shell built-in and compound command make some appearance. Not to mention the non-POSIX bits of shell we all take for granted (e.g. &>-redirection). Merry and msh are useable. You will most likely find bugs if you use them. If, when using msh, you find something obscure happening you can enable debug mode either my setting the variable MSH_DEBUG or by invoking msh with -v -v. Patrick Ferris 2026 3 19 What's next? It is soon time to combine Shelter and Merry into the time-travelling, POSIX-ish shell that I have been trying to build since I first started working on Shelter. For this to be successful, I need Merry to be able to pass plenty of tests and right now that involves trying to install plenty of packages using tools like apk and apt. Patrick Ferris 2026 3 19 Shell MRDT What I am particularly interested in reasoning about, is the execution context in Merry. This value, alongside the file-system, constitutes a fairly deep understanding of the state that changes in each step of a shell's evaluation loop. This was, in terms of Shelter, the missing piece for truly building some kind of MRDT across shell sessions. Patrick Ferris 2026 3 19 TIFF in OCaml I picked up from the excellent Outreachy work of Tambe Salome in getting write-support in ocaml-tiff. We are getting closer to the kind of API I envisaged in this latest round of refinement and review. let / = Eio . Path . / let checkerboard ~ size = let v = Nx . zeros Nx . uint8 size ; size in for row = 0 to size - 1 do for col = 0 to size - 1 do if row + col mod 2 = 0 then Nx . set_item row ; col 254 v done done ; Nx . to_bigarray v let = Eio_posix . run @@ fun env -> Tiff_eio . with_open_out env # cwd / " example.tiff " @@ fun w -> let tif = Tiff . make checkerboard ~ size : 256 in Tiff . to_file tif w I have also been extremely pleased to see further external collaborators appearing: Nicolas helping out with metadata maintainence. Virgile adding support for reading multi-image TIFF files. Backlinks Patrick Ferris 2026 7 12 https://patrick.sirref.org/ocaml-roundup-august-2026/ ocaml-roundup-august-2026 /ocaml-roundup-august-2026/ OCaml Roundup: August 2026 Much of August was spent thinking about POSIX shell semantics, and in particular extending notions from smoosh to include side effects (and maybe concurrency). This is still nascent work, but some ideas and flavours are described below. Patrick Ferris 2026 7 12 Merging Shells The work came from wanting to merge two shell sessions into one. Setting aside the practicalities of an implementation, it makes sense to start first theoretically. When can I merge two shell executions, and when can I not? It would be nice to think of unset DOWNLOAD_DIR as a function. This function takes the shell state and the operating state as arguments and returns a modified shell state and an "unmodified" operating system state (at least for our purposes). Meanwhile, branch-b may have a lot more side effects, but when focusing on the environment only we can see in a sense the two branches are conflict free. Well, sorta. We also need to ensure we do not also read DOWNLOAD_DIR in branch-b if we intend to remove it in their merge. This work, in part, is borrowing ideas from MRDTs and can reuse void processes to help reason about when side effects can occur, as well as open-trace (which is now becoming os-trace). I hope to continue this work, see my latest weekly for more on this. Patrick Ferris 2026 7 12 Eio in the browser I spent some time carving out Eio_browser into a standalone library, something I depend on for Merry to work in the browser. Patrick Ferris 2026 9 12 https://patrick.sirref.org/eio_browser/ eio_browser /eio_browser/ Eio_browser https://git.sirref.org/eio_browser Eio_browser is a backend for Eio that works in the browser. It reuses work I started at Tarides that picked up some momentum but has since been a little dormant. Crucially, it provides a functional standard environment like Eio_main or any of the OS-specific backends. module Stdenv : sig type base = < stdout : Eio_unix.sink_ty r ; stderr : Eio_unix.sink_ty r ; clock : float Eio.Time.clock_ty r ; mono_clock : Eio.Time.Mono.ty r ; secure_random : Eio.Flow.source_ty r ; debug : Eio.Debug.t ; backend_id : string > end val run : Stdenv . base -> unit -> unit Related https://patrick.sirref.org/anilmadhavapeddy/ anilmadhavapeddy /anilmadhavapeddy/ Anil Madhavapeddy person https://anil.recoil.org Professor of Planetary Computing at University of Cambridge. https://patrick.sirref.org/dra27/ dra27 /dra27/ David Allsopp person https://www.dra27.uk/blog/ https://patrick.sirref.org/eio/ eio /eio/ Eio Software https://github.com/ocaml-multicore/eio Eio provides an effects-based direct-style IO stack for OCaml 5 https://patrick.sirref.org/nathanreb/ nathanreb /nathanreb/ Nathan Rebours person https://github.com/nathanreb Maintainer of Ppxlib. https://patrick.sirref.org/ppxlib/ ppxlib /ppxlib/ Ppxlib https://github.com/ocaml-ppx/ppxlib An OCaml library and ecosystem for preprocessing. https://patrick.sirref.org/shelter/ shelter /shelter/ Shelter https://tangled.sh/@patrick.sirref.org/shelter A shim between a user and the shell that provides greater reproducibility and insight into what your code is doing! Give me Shelter! Shelter combines the caching and reproducibility of Dockerfiles with the ease of interaction of the shell. I'd be interested to use Ryan's package management work as a means to specify the base environment users want. Here is an example of running Shelter and making use of the branchable sessions. shelter> echo hello > hello.txt sheltermain#bbd691arw @ session exp-1]]>shelterexp-1#bbd691arw echo world >> hello.txt]]>shelterexp-1#dd5bab8rw @ session main]]>sheltermain#bbd691arw echo "to the" >> hello.txt]]>sheltermain#b9abef4rw cat hello.txt hello to the]]>sheltermain#d1c1728rw @ session exp-1]]>shelterexp-1#dd5bab8rw @ replay main]]>shelterexp-1#df1b4e1rw cat hello.txt hello to the world]]>shelterexp-1#67162b7rw]]> The @ character allows the user to interact directly with Shelter's build-in operators. @ session exp-1 creates a new session as there is no exp-1 session. It points to the head commit of the branch it came from (git checkout -b exp-1). From there we make changes and switch between branches until we perform a replay. A replay is a rebase without any merge conflict checks, it simply finds the least common ancestor of the two branches and re-applies the commits from your current branch (here exp-1) onto the head of the target branch (here main). Under the hood Shelter is a light-weight, reimplementation of obuilder. It uses the similar ideas, like snapshotting filesystems (e.g. ZFS) and Linux namespaces (e.g. Void). Between the low-level store and the interactive shell parts, there is an Irmin store. https://patrick.sirref.org/samoht/ samoht /samoht/ Thomas Gazagnaire person https://gazagnaire.org/blog.html https://patrick.sirref.org/talex5/ talex5 /talex5/ Thomas Leonard person https://roscidus.com/ Patrick Ferris 2026 7 4 https://patrick.sirref.org/merry-benchmarks/ merry-benchmarks /merry-benchmarks/ Shell Benchmarks, Back to Writing › Benchmarking Merry Patrick Ferris 2026 7 4 Medium-to-small potatoes I used shellbench to understand where some of the performance pitfalls were coming from. $ ./shellbench -s bash,zsh,msh sample/func.sh ----------------------------------------------------------------------------------- name bash zsh msh ----------------------------------------------------------------------------------- func.sh: no func 775,539 1,124,485 5,229 func.sh: func 398,861 197,169 93,424 ----------------------------------------------------------------------------------- * count: number of executions per second This particular benchmark first runs nothing (i.e. :) and then a function lookup and execution. Surprisingly msh is slower at doing nothing! I perf-ed a run of msh test.sh with just a single : in the file: Patrick Ferris 2026 7 4 Test built-in After making various improvements to Merry that almost halved the memory usage (e.g., removing extraneous string manipulations, adding a fast-path to the PATH variable for resolving programs). I was still pretty far off the performance of these other shells. I decided to have a look at dash which has a simple implementation. This flame graph is for the following shell script: f : ; while __count= $ __count + 1 && " $ __count " ; do f done echo " count is $ __count " And here are some rudimentary benchmarks: $ time -p dash test.sh count is 10001 real 0.01 user 0.01 sys 0.00 $ time -p msh test.sh count is 10001 real 15.53 user 4.28 sys 12.28 It turns out the massive slowdown is very simple to explain. In the dash flamegraph there is a oexpr stack; which comes from the bltin/test.c file. dash, along with the other shells, does not spawn a process for the -test command! Forcing dash to do use the binary gives: $ time -p dash test.sh count is 10001 real 7.55 user 3.95 sys 3.44 Which is much more respectable in terms of performance compared to msh. Plenty of work to do though! Patrick Ferris 2026 7 4 Parsing arithmetic expressions There was some obvious low-hanging fruit to help improve some of the performance differences. The first was pre-parsing arithmetic expressions. Morbig, the static parser, does not implement arithmetic expression parsing; so Merry does it for us. Before, it was on-demand, as we encountered an expression of the form $((<e>)). Now, it happens up front. This should improve tight-loops. Contributions