Patrick Ferris 2025 5 19 https://patrick.sirref.org/weekly-2025-05-12/ weekly-2025-05-12 /weekly-2025-05-12/ Metashells This week, I feel I have been stuck fighting the OCaml ecosystem trying to keep my OxCaml work afloat. Aside from that, Ryan and I made some really nice progress with Shelter, culminating in Ryan describing it as a metashell. Patrick Ferris 2025 5 19 Shelter the metashell The main progress this week with Shelter was composing runc's terminal mode with entering raw terminal input mode on the Shelter side. This is inspired by Ryan's own work on capability interfaces. Shelter remains mostly intact, acting as an interactive shell. However, just before executing a command we switch to receiving and sending raw terminal inputs and outputs. This means tools like vim now work in Shelter! Not only that, but users can now activate an inner shell (e.g. zsh) and enjoy all the usual features of a fully-fledged shell (tab complete, fuzzy history search etc.) and upon exiting that shell, Shelter will snapshot the session. This lets you alter the granularity of snapshots from the command-line. Patrick Ferris 2025 5 19 https://patrick.sirref.org/pouzin-shell/ pouzin-shell /pouzin-shell/ Louis Pouzin's "SHELL" I spent some time reading part of the multics design documentation this week. Louis Pouzin coined the term "SHELL" in this document, and I was reminded yet again just how important it is to be a good writer even as a "computer science researcher". For example, this excerpt from the requirements section of the document The previous definitions imply that a command MUST be designed while keeping in mind the user, sitting at his console, wondering about what might be going on, mistyping or forgetting arguments, even if fully aware of the conventions, and possibly interfering with the command by hasty quits, carriage returns, and other temperamental reactions. And then later, when defining the "SHELL". We may envision a common procedure called automatically by the supervisor whenever a user types in some message at his console, at a time when he has no other process in active execution under console control (presently called command level). This procedure acts as an interface between console messages and subroutine. The purpose of such a procedure is to create a medium of exchange into which one could activate any procedure, inside of another program if it were called. Hereafter, for simplification, we shall refer to that procedure as the "SHELL". It still surprises how little the undergraduate degree in computer science at Cambridge focuses on writing skills. Patrick Ferris 2025 5 19 OxCaml Last week, I got a toplevel with OxCaml working. This required a serious amount of work to understand the changes Janestreet have made to obscure parts of the OCaml compiler and then working those into tools like js_of_ocaml. This week, Janestreet pushed their latest rounds of changes and of course everything broke! I spent some more time fixing it all back up. I'm not entirely sure how maintainable this is. The problem is that, whilst things compile, the programs do not work together! Only when someone uses the program do the bugs surface. Patrick Ferris 2025 5 19 Other OCaml Work I worked on some other parts of the ecosystem this week. Patrick Ferris 2025 5 19 Ppxlib I helped review some changes to enable Janestreet to have ppx rewriters via attributes (usually they are via extension points). It is a bit of a controversial change to ppxlib as we try to keep the API predictable for users: Extension points are rewritten: this means the part that is rewritten is nicely delimited by the extension points start and end. Attributes extend: attributes do not rewrite the code they are attached to but rather extend the code with new AST nodes. We will see what we decided to do (most likely provide the functionality behind some kind of "expert" interface). Patrick Ferris 2025 5 19 Tiff See Michael's notes. I spent some time trying to speedup the LZW decompression of TIFF files in the pure OCaml tiff library this week(end). The two big changes to help with this are pretty common when speeding up these parts OCaml programs: Allocate less Does less work In terms of allocating less, the original implementation was using a char list to represent LZW strings. Manipulating these becomes quite costly, particularly since the most common operation is appending a single character to the end of a list. Converting this to use OCaml's immutable string saved a ton of allocations. In terms of doing less work, I opted to bypass Cstructs sane (but slow) bounds checks in some of the "hotter" parts of the code. In particular, LZW ends up reading potentially huge arrays full of bytes one-by-one. So reading each byte needs to be quite snappy. This is a bit of a trade-off in terms of "safety" but we are in control of this code so I'm not too worried about that. Here are some results decompressing a fairly large array of some elevation data. References Louis Pouzin 2013 https://patrick.sirref.org/pouzin-shell-2013/ pouzin-shell-2013 /pouzin-shell-2013/ The Origin of the Shell Reference https://people.csail.mit.edu/saltzer/Multics/Multics-Documents/MDN/MDN-4.pdf Multics Documents Louis Pouzin 2013 Abstract How RUNCOM was created for CTSS and the shell was designed for Multics by Louis Pouzin. Context https://patrick.sirref.org/weeklies-2025/ weeklies-2025 /weeklies-2025/ Patrick Ferris' Weeklies › 2025 Patrick 2025 11 24 https://patrick.sirref.org/weekly-2025-w49/ weekly-2025-w49 /weekly-2025-w49/ Data Provenance in Shelter A late update that stretches more than a week... https://shelter.sirref.org/Patrick Ferris/ 2025 12 9 https://shelter.sirref.org/shelter-log-002/ https://shelter.sirref.org/shelter-log-002/ https://shelter.sirref.org/shelter-log-002/ @ How? A quick update on a new feature that was added recently, the @ how meta-command. provenance of a given file. How that file came to be and who and what processes have interacted with it since. Shelter now has a command for asking exactly that of any given file in the system. hello.txt echo world > world.txt cat world.txt > hello.txt @ how hello.txt]]> The previous snippet of shell script will output: hello.txt Ran cat world.txt > hello.txt (read: "world.txt")]]> There are plenty of optimisations and improvements to make, but this is a nice first step to showing the kinds of feature you get when programming with Shelter. Patrick Ferris 2025 12 9 https://patrick.sirref.org/ocaml-roundup-november-2025/ ocaml-roundup-november-2025 /ocaml-roundup-november-2025/ OCaml Roundup: November 2025 A roundup of some of my OCaml activity this month! Patrick Ferris 2025 12 9 Ppxlib Updates I managed to get a substantial amount of ppxlib work completed in the last month. Some of those bits are described in detail below. Patrick Ferris 2025 12 9 https://patrick.sirref.org/ppxlib-5.5-support ppxlib-5.5-support /ppxlib-5.5-support Initial OCaml 5.5 Support We recently merged 5.5 support into our main branch (thanks to Nathan for the review). It cast a light on some new features coming to OCaml including the ability to define any structure locally. Most OCaml programmers will be familiar with things like local opens (e.g. let open M in). This has now been extended to include any structure item (more or less, you cannot nest value bindings). External types have also landed, bringing with them a new "type kind" in the AST. You can read about them in the manual. Patrick Ferris 2025 12 9 Bug Fixes for Attribute Rewriters There was a bug in the error handling for attribute rewriters -- a relatively new feature to have been added to ppxlib. In some places, ppxlib makes use of a With_error.t monad. Unlike the Result.t monad, With_error.t allows you to collect errors rather than find the first error and thread it through your program. This can make sense if your program can meaningfully do work after encountering an error. Unfortunately, in the logic for replacing attributes, the code could not recover from an error where there were duplicate attributes. Patrick Ferris 2025 12 9 Value binding constraints in ppxlib I spent a good few hours understanding why the CI tests in my PR to mirage/repr were still failing. It turns out, the bug, is fairly simple to reproduce. Consider the following OCaml code: let g : int = 1 We have a let-binding that contains a type constraint. Where, in the abstract syntax tree sense, do we attach the constraint? Since OCaml 5.1, we have quite a few options for how this can be represented. The first is using the Ppat_constraint AST node which allows you to attach a type constraint to any pattern. | Ppat_constraint of pattern * core_type This tends to be used for things like function arguments (e.g. fun (x : int) -> x + 1). So it doesn't quite capture what we want for this binding constraint. On older OCaml compilers, let g : int = 1 also includes a Pexp_constraint on the body of the value binding. This makes sense if we consider a more general case of the syntax where we allow functions. let g x : int = x + 1 In this example, we no longer parse a Ppat_constraint and instead this is treated as syntactic sugar for: let g x = x + 1 : int In OCaml 5.1 a new field for value bindings was added allowing type constraints to be tracked directly in the bindings themselves. and value_binding = pvb_pat : pattern ; pvb_expr : expression ; pvb_constraint : poly_constraint option ; pvb_attributes : attributes ; pvb_loc : Location . t ; Our type constraint (: int) can be recorded in the pvb_constraint directly. This is a nice change, bringing the concrete syntax of the language closer to the parsetree. For ppxlib, however, it does create a bit of a headache as we now have to encode the different versions of expressing type constraints into the AST of the compiler before 5.1! So what was the bug? The author's of repr had added some value bindings to circumvent some "unused value warnings" in OCaml. They choose to write these bindings like: let _ : int = 1 Here, I have simplified the types and the body of the binding. The significant thing to notice is the parentheses surrounding the value binding's pattern. This forces the AST representation to be a single Ppat_constraint node regardless of the current compiler version. No Pexp_constraint and no pvb_constraint. However, using ppxlib with a compiler less than 5.1 meant that this did not roundtrip cleanly. Instead, the value after passing through ppxlib's migrations was: let _ : int = 1 After bottoming out at the issue, I discovered we already had a test case for this in ppxlib! Whatsomore, ocamlformat.0.20.0 (which repr was still using) actually transforms instances of let x : t = ... into let (x : t) = ...! This limitation does not seem inherent to the parsetree, the printer or the migrations. But I have asked the (very kind and friendly) previous maintainers about these changes. Patrick Ferris 2025 12 9 Outreachy December 8th 2025 marks the start of the next round of Outreachy. I have been preparing some issues for geocaml/ocaml-tiff and I am excited to get started with Tambe Salome! Patrick Ferris 2025 12 9 https://patrick.sirref.org/graft-and-bib-update/ graft-and-bib-update /graft-and-bib-update/ Graft and Bib I spent some time at the start of the month updating and fixing Graft and Bib. I added named subtrees to the markdown syntax. This allows users to create subtrees with headings that are also externally linkable. With some more bug fixing, I hope to release both libraries before the new year! Patrick Ferris 2025 11 15 https://patrick.sirref.org/weekly-2025-w46/ weekly-2025-w46 /weekly-2025-w46/ Shelter imports Most of this week was spent fixing bugs in Shelter in an attempt to get it ready to run the LIFE pipeline. To do so, Shelter requires a mechanism by which to pull in files from the outside world. That could be configuration files, source trees, data etc. For now, this has taken the form of a crude @ import statement. In true Forester fashion, I have transcluded the foreign tree from my Shelter logs. https://shelter.sirref.org/Shelter Programmers/ 2025 11 16 https://shelter.sirref.org/shelter-log-001/ https://shelter.sirref.org/shelter-log-001/ https://shelter.sirref.org/shelter-log-001/ Import statements Welcome to the first Shelter log! In this log I will explain the work I have been doing to add an @ import meta-command. The syntax thus far is relatively straightforward. ]]> There are a few key principles underlying @ import. It should be at least as expressive as Docker's COPY ... command. It should deal with a wider variety of import sources. In addition to local https:// archives over ssh etc. Imports, where possible, should be catalogued and shared across sessions. https://shelter.sirref.org/Shelter Programmers/ 2025 11 16 URIs as Sources branch already allows users to import local files and directories into their session. For example: By allowing arbitrary URIs in the import command, we hope to force users away from downloading data via curl or python scripts. Lifting these side-effectful imports into Shelter allows us to manage the data in a much cleaner fashion. https://shelter.sirref.org/Shelter Programmers/ 2025 11 16 Scripts as Sources One idea I had was to allow users to invoke scripts as there "import source". Something like: And using the same eBPF tracing we use during normal execution we can capture a fairly good idea of the tools and files needed to perform an import. https://shelter.sirref.org/Shelter Programmers/ 2025 11 16 Sharing Imports One example of better data management is that Shelter imports can be When a user imports some data, they can optionally name that piece of data. For example: Shelter stores that away and makes a link between the name, the URI and the underlying data. A user may then import the same piece of data into a different session by simply writing: intent via the naming scheme, it makes it possible to then version the data and have a means by which to update said data, one could imagine: Which will retry the original URI and see if the data has changed, or if it has not. There is an implicit versioning number that users can specify in the name should they choose to. Patrick Ferris 2025 11 15 https://patrick.sirref.org/eio-cap-debugging/ eio-cap-debugging /eio-cap-debugging/ Eio Debugging One of the promises of Eio's capability model is an improved debugging experience. Your program must be designed in such a way as to make resources explicit. If you have a load_config function, more than likely it will need access to the file-system. val load_config : _ Eio . Path . t -> Config . t Once all of your resources are explicit and threaded through your program, it becomes much easier to swap out one resource for another (provided they have the same interface). This week I was trying to get the LIFE pipeline running in shelter. Part of that involves pulling the appropriate geospatial container image, but my code was terrifically broken, failing with what looked to me with a Docker error. With only this clue, I knew that it was probably in the execution of some command in a child process, but which one? To trace this quickly, I could plug in a slightly different instance of the _ Eio.Process.mgr capability. Essentially unchanged from the one provided by the unix backend except that it would also print to stderr the command it was trying to run! let debug_process_mgr mgr : 'a Eio_unix . Process . mgr : 'a Eio_unix . Process . mgr = let module D = struct type t = unit let spawn_unix ~ sw ? cwd ? pgid ? uid ? gid ~ env ~ fds ~ executable args = Eio . traceln " Spawning subprocess... " Fmt . list string args ; Eio_unix . Process . spawn_unix ~ sw ? cwd ? pgid ? uid ? gid mgr ~ env ~ fds ~ executable args end in let module V = Eio_unix . Process . Make_mgr D in Eio . Resource . T , Eio_unix . Process . Pi . mgr_unix module V Whist there is a fair amount of jumping through first-class-module-shaped hoops here, all I am doing is rewrapping an existing "method" to a spawn a unix child process. Then using this helper function, I can push that into my standard environment. let env = Eio_unix . Stdenv . with_env ~ process_mgr : debug_process_mgr env # process_mgr env in Shelter . main config env : > _ Shelter . env dir cmd_file And sure enough I was querying for the Docker user before pulling the image! I hadn't run into this, as for the last while I have been using alpine and debian images that already exists locally. Patrick Ferris 2025 11 10 https://patrick.sirref.org/weekly-2025-w45/ weekly-2025-w45 /weekly-2025-w45/ Taking stock & Graft updates I ended up working on a real mix of things last week. I managed to get two large posts uploaded to this site: an ICFP recap and a deeper-dive into the random bits of OCaml hacking. This was pretty useful in terms of taking stock of where I was at, I always find post-conference, I'm a little lost. The conversations and ideas from ICFP are simultaneously exciting but also pretty inactionable. Patrick Ferris 2025 11 10 https://patrick.sirref.org/graft-updates/ graft-updates /graft-updates/ Graft Updates It would seem that there is a potential second user of Graft! A great source of motivation for some tidying and feature implementing. At the end of last week I fixed some little bugs in Graft and (more importantly) allowed users to name their subtrees. Upstream Graft also now pins and uses my Bibtex implementation. Patrick Ferris 2025 11 10 OCaml Geotessera As I discussed in my OCaml roundup, I spent a bit of time polishing the OCaml geotessera code. In particular this meant starting to handle low-level geospatial transformations allowing users to project latitudes and longitudes into the rows and columns of their raster data. I feel like I'm stumbling onto a path Michael has carved out with Yirgacheffe. I'm following along with his work excitedly and we had a good catch-up last week discussing how libraries should handle things like automatic, numeric precision conversion. As a fully paid-up member of the static types fan club, I'm often evangelising the benefits of types. However, I hit a little quirk of the Nx library in OCaml (a numpy equivalent). Take a look at this sum function: let sum nx = Nx . sum nx |> Nx . item ; ; val sum : 'a , 'b Nx . t -> 'a let arr = Nx . create Nx . int8 2 64 ; 64 ; ; val arr : int , Bigarray . int8_signed_elt Nx . t sum arr ; ; - : int = - 128 Even though sum takes an ('a, 'b) Nx.t array that stores elements with representation 'b and allows you to read/write to the elements with OCaml values of type 'a, the sum function performs the calculation in the world of the 'b representation type (here signed 8-bit integers). So we get an overflow! Perhaps not too surprising but easy to overlook! The moral of the story is that we still need to be careful in the OCaml world too. Patrick Ferris 2025 11 10 Shelter I did a little cleaning up of the Shelter code too. I noticed a few bugs from the demos I did at ICFP. Most were pretty straight-forward to fix, and the test-suite is now checking parallel for-loops and conditional execution are working. Patrick Ferris 2025 11 10 This week I hope to get a few things done this week: Run some part of the LIFE pipeline through Shelter. Deploy the Shelter daemon allowing other people to connect and take it for a spin. Spend some time writing and reading, pulling ideas together rather than always writing code! I have a slightly harebrained scheme that combines Shelter, atproto, Forester, capn(proto/web) and webcomponents... I want to see if I can pull that together. 2025 7 18 https://patrick.sirref.org/weekly-2025-07-18/ weekly-2025-07-18 /weekly-2025-07-18/ Back to shelter 2025 7 18 Capnproto Interface to Shelter I spent some time this week thinking about decoupling Shelter's builds from the CLI tool. This would allow users to connect to a remote daemon to build and store their sessions, opening up the possibility of using Shelter on Windows and macOS. The design so far keeps the metadata local, in an Irmin database. I will probably then augment the stored information with some kind of unique daemon identifier to know where the actual data is. This is all still quite fresh, and I will have more by the end of next week. 2025 7 18 OCaml-related Things Something I decided to do last week, was to make a clearer delineation between my OCaml work and research. That line is not always very distinct, but I have split of a separate set of weeklies for OCaml related things. Fear not, the OCaml weeklies are a subset of these weeklies and so I will, when I remember, transclude them for you. Patrick Ferris 2025 7 15 https://patrick.sirref.org/ocaml-weekly-2025-w29/ ocaml-weekly-2025-w29 /ocaml-weekly-2025-w29/ OCaml Weekly 2025 w29 Patrick Ferris 2025 7 15 Ppxlib I met with Nathan this week to discuss future plans for ppxlib. The current state of affairs is that ppxlib is becoming unmaintainable. This is primarily a knock-on effect from changes being made to OCaml's parsetree (e.g. labelled tuples being added in 5.4). Nathan has a plan that will provide two key properties. Migrations, which allow old compilers to be used with new ppxlib releases, will be more compatible. For example, we will be able to migrate new features downwards and back up without raising an error. Ppx authors will be able to use new features in an opt-in workflow, rather than ppxlib bumping the internal AST (like we did in ppxlib.0.36.0). This will reduce the maintenance burden significantly whilst still allowing users to write ppxes for new OCaml features. I also started looking into some older issues in ppxlib related to performance. This is work-in-progress, but I am trying to improve the performance of some passes done by ppxlib. To better understand what was making ppxlib slow, I wanted to use runtime_events_tools but I was dismayed to see it wanting to install over 100 packages! I opened a PR to reduce the number of packages. I think this kind of work goes a little unrecognised as it is not very glamorous. However, I think it really benefits the OCaml community in the long run. Patrick Ferris 2025 7 15 Outreachy In Outreachy news, we had a wonderful mid-internship video call with all the interns and mentors to catch-up on how everyone is getting along. Seeing the progress everyone has made was great! I am very grateful for the work that Michael and Steve have put in so far to make this a very successful Outreachy round for OCaml. In sadder news, an email was shared with all Outreachy mentors detailing the increasingly critical financial situation the project finds itself in. There are ongoing discussions about how costs can be cut including potentially only running a single round a year. Patrick Ferris 2025 7 15 Graft With the release of Forester.5.0, I made a plan to make a release of Graft.0.1. Unfortunately this is blocked by a new release of hilite, a tool I built for doing build-time syntax highlighting for OCaml code. This powers the syntax highlighting on ocaml.org. Patrick Ferris 2025 6 30 https://patrick.sirref.org/rest-of-monthly-2025-06/ rest-of-monthly-2025-06 /rest-of-monthly-2025-06/ PhD Work and Scientific Programming I have been away on holidays. Some of that was spent hiking in the Massif de Calanques in Marseille, unfortunately this week they were on fire. Anyway, this "weekly" fills in the blanks for the rest of June 2025. Patrick Ferris 2025 6 30 Revisiting hazel_of_ocaml I spent some time thinking and writing about hazel_of_ocaml and the benefits of OCaml as source language and Hazel as a target language. The stability of OCaml with a powerful type-system makes it very expressive. Hazel, on the other hand, is a hotbed of PL theory research. The typed holes, which are fully supported within the language, are particular powerful targets for a transpiler. They allow transpiler developers to incrementally support the language whilst still getting full feedback of their generated source code. Whilst it is a niche use-case, I thought it was an interesting point to note. Patrick Ferris 2025 6 30 A second year report I wrote my second year report at the end of June. The process was a bit chaotic, but I think ultimately it was useful for clarifying some ideas I had rattling about in my head. Part of that was working on my publications page and, through a conversation with Anil, realising that my work on Graft was in many ways a part of my own research. If you are interested, feel free to peruse both my first year and second year reports. Take them with a pinch of salt! I think it is useful to be fairly open with this sort of thing when you can be. Maybe someone will find them useful. Patrick Ferris 2025 6 30 What is scientific programming? This question arose whilst writing my second year report. It got me thinking about how we distinguish scientific programming from other kinds of programming. I came across a paper by Tim Storer: Bridiging the chasm: A survey of Software Engineering Practice in Scientific Programming. I enjoyed a few aspects of this paper. Relating scientific programming to the scientific method so directly was pretty useful insight for me. Falsifable hypotheses, repeatable experiments and reproducible results when applied as characteristics of the computer stack (e.g. OS, programming language, filesystem etc.) might be a useful lens to argue for better tools to enable this. The breadth of Storer's case-study analysis on how scientific programming has "gone wrong" is very good, if anyone needs a reference to draw on to help support their tools or research, this seems like a good index to use. Patrick Ferris 2025 6 30 Ppxlib I was reminded again of the pain of helping maintain ppxlib. We are coming to the conclusion, that in its current form, it is becoming unmaintainable. I took a stab at updating bisect_ppx to the latest ppxlib. With OCaml entering a potentially turbulent parsetree era, it might be time to take stock of this and propose some fillers to help. Nathan Rebours and I are meeting next week to discuss ideas he has to make parsetree migrations smooth sailing for all! Patrick Ferris 2025 6 30 Geocaml Work on ocaml-tiff has stalled, which is totally fine. Michael and I independently are working on other things and have obligations to fulfill. Maybe some day we'll carve out enough time together to push forward on this project, but not this week. Patrick Ferris 2025 6 2 https://patrick.sirref.org/weekly-2025-06-02/ weekly-2025-06-02 /weekly-2025-06-02/ Forester as a Target Syntax This week included some time finishing opentrace and subsequently folding it into shelter. I have been writing up some more of the draft paper for shelter which I am excited to share in the near future. I revisited the upgrading vpnkit PR and pushed some more fixes. I have been thinking, again, about the promise of a direct-style world for OCaml that just hasn't quite landed yet. C'est la vie. Patrick Ferris 2025 6 2 Forester and Graft I spent a bit of time finally pulling out my changes to Forester to add markdown and bibtex support into a standalone tool: graft. Patrick Ferris 2025 6 2 https://patrick.sirref.org/graft/ graft /graft/ Graft https://github.com/patricoferris/graft Graft is a Forester preprocessor. It takes a forester (a directory of trees) written in a mixture of Markdown, Bibtex and Forester syntax and produces a new forest completely written in Forester syntax. Patrick Ferris 2025 6 2 Usage graft simply preprocesses a forest generating Forester trees from .md, .bib and .tree files. It will copy the structure of the input directory in the output directory. $ graft preprocess --output=grafted-trees trees $ forester build This assumes that you have updated your Forester toml file to put the grafted-trees directory as your source of trees. Patrick Ferris 2025 6 2 Example A typical "tree" might look something like A few things to note: The yaml frontmatter allows you to add some of the metadata fields from Forester. At any point in your markdown there is an escape hatch to Forester using a forester codeblock. It is very satisfying to find the separation of concerns works quite well. For a while I had been rebasing my development branch on Forester. I was also worried about trying to get the code upstream as it pulled in many dependencies. It seems that I have a very workable solution. I welcome contributions to graft including extra input formats. I am also considering extending the Forester configuration to contain some graft configuration for how it should generate new trees (e.g. at the moment every entry in a bibtex file is given a new tree). Patrick Ferris 2025 6 2 Maths Support As part of that process, graft now supports Markdown KaTeX. For example: Patrick Ferris https://patrick.sirref.org/mrdts/ mrdts /mrdts/ Mergeable Replicated Data Type Implementation Definition A mergeable replicated data type (MRDT) implementation for a data type is a tuple where: is the set of all possible states at a branch, is the initial state, implements every data type operation, implements the three-way merge strategy. Definition 2.1 from "Certified Mergeable Replicated Data Types". Patrick Ferris 2025 6 2 Revision Supervisions I have been doing revision supervisions for Discrete Maths and Foundations of Computer Science. I have also been marking Operating Systems past paper questions for the same group of first years. Patrick Ferris 2025 5 26 https://patrick.sirref.org/weekly-2025-05-26/ weekly-2025-05-26 /weekly-2025-05-26/ Opentrace and Supervisions Over the past two weeks I have mainly split my time (amongst many things) developing opentrace and doing revision supervisions. Patrick Ferris 2025 5 19 https://patrick.sirref.org/open-trace/ open-trace /open-trace/ Opentrace https://tangled.sh/@patrick.sirref.org/opentrace Thanks to Koonwen's excellent libbpf bindings in OCaml, I have been building a little tool called opentrace to make it easier to track an executable's inputs and outputs. This work was inspired my Michael's self-proclaimed "gross hack": pyshark. Whilst pyshark achieves its goals by injecting code into commonly used python objects and methods, opentrace uses eBPF. By using a lower-level API (hooks in the kernel), opentrace can remain programming language agnostic. However, less information is none about the user's intent compared to something like pyshark. Patrick Ferris 2025 5 19 Monitoring the System opentrace has an all command that will trace the entire system. The --flags=O_WRONLY argument filters the events where the O_WRONLY flag was set in the call to open. We also get the name of the current executable linked to the task (comm). The -wrapped is an artefact of using Nix. Patrick Ferris 2025 5 19 Tracing an Executable The primary use case for this tool is to inspect what files your program might be reading and writing. The "flags" argument can specify a small boolean formula for checking the open flags of a particular event with | (or), & (and), and ~ (not). Parentheses can be used for precedence. Patrick Ferris 2025 5 19 Spawning Subprocesses One feature opentrace needs (in this proof-of-concept phase) is the ability to also trace subprocesses. opentrace is primarily an eBPF program that is loaded into the kernel and communicates with an OCaml program. Events are communicated via a ring buffer and most of the post-processing happens in OCaml. To capture subprocesses, opentrace creates a new control group (cgroup) and places the new process into that group. This gives opentrace a new identifier to track, namely the cgroup. So consider the following program. let = Eio_posix . run @@ fun env -> Eio . Path . save ~ create : `Or_truncate 0o664 env # fs / " hello.txt " " hello " ; Eio . Process . run env # process_mgr " /bin/bash " ; " -c " ; " echo 'heya' > heya.txt " It first creates a file using direct calls to functions like openat. Then it spawns a process which creates a new file called heya.txt. This happens in a separate process. However, with the --cgroups flag we can capture both interactions with the operating system. Patrick Ferris 2025 5 19 Eio's Process API I have used the Eio_unix fork action process API to be able to extend what happens in the child process. Loading most eBPF programs into the kernel requires special privileges hence the need for sudo. When a user requests for a particular program to be executed and traced, opentrace spawns a process via the Eio Process API. Opentrace defines a few new so-called "fork actions", little fragments of C code that are run after the call to fork (clone). Most likely this ends with a call to execve, but other calls are possible for example setuid allowing opentrace to change the user of the child process so it does not run as root. Similarly, this is where (if used) we create the cgroup and place the process into that group. Patrick Ferris 2025 5 19 Limitations: Io_uring Whilst testing opentrace against some of the tools I use nearly daily, I noticed some events were being missed. I tried tracing forester, and only the initial read of forest.toml was logged. It dawned on me that the reason for this was that forester (via eio) was using io_uring to perform most of the IO. Most attempts to open files were bypassing the open system calls, and instead they were being performed by the kernel after reading a submission request for an openat2-style call! This is not news to seasoned, Linux systems programmers. Io_uring bypasses SECCOMP filters for exactly the same reasons. It is interesting to note two things here: We can tell that forester reads the configuration file probably using something like In_channel in OCaml (it does). It appears that Uring is performing IO in both worker threads and directly. The file paths are empty at the moment as I cannot find a clean way to trace openat submission requests into Uring without it sometimes going very wrong. I have tried quite a few methods (e.g. tracing do_filp_open) and at the moment I am tracing io_openat2, but this seems quite brittle, and often the filename is completely garbled, so I do not set it. If anyone has any ideas to trace Io_uring more reliably, I am all ears! Patrick Ferris 2025 5 26 Supervisions The first year students I teach will be doing their first year exams soon. I have been helping them revise for foundations of computer science and discrete mathematics. Whilst doing so, I have been thinking a good deal about Jon Sterling's post about assessment bureaucracy. In particular: At moments like this, it is a good idea to pause and reflect on whether it is better for our students that each faculty member spend a cumulative two months doing literally nothing but assessment and higher-order practices related to assessment, vs. other activities that could benefit our students more (including actual teaching, of which we do astonishingly little at Cambridge). Patrick Ferris 2025 5 19 https://patrick.sirref.org/weekly-2025-05-12/ weekly-2025-05-12 /weekly-2025-05-12/ Metashells This week, I feel I have been stuck fighting the OCaml ecosystem trying to keep my OxCaml work afloat. Aside from that, Ryan and I made some really nice progress with Shelter, culminating in Ryan describing it as a metashell. Patrick Ferris 2025 5 19 Shelter the metashell The main progress this week with Shelter was composing runc's terminal mode with entering raw terminal input mode on the Shelter side. This is inspired by Ryan's own work on capability interfaces. Shelter remains mostly intact, acting as an interactive shell. However, just before executing a command we switch to receiving and sending raw terminal inputs and outputs. This means tools like vim now work in Shelter! Not only that, but users can now activate an inner shell (e.g. zsh) and enjoy all the usual features of a fully-fledged shell (tab complete, fuzzy history search etc.) and upon exiting that shell, Shelter will snapshot the session. This lets you alter the granularity of snapshots from the command-line. Patrick Ferris 2025 5 19 https://patrick.sirref.org/pouzin-shell/ pouzin-shell /pouzin-shell/ Louis Pouzin's "SHELL" I spent some time reading part of the multics design documentation this week. Louis Pouzin coined the term "SHELL" in this document, and I was reminded yet again just how important it is to be a good writer even as a "computer science researcher". For example, this excerpt from the requirements section of the document The previous definitions imply that a command MUST be designed while keeping in mind the user, sitting at his console, wondering about what might be going on, mistyping or forgetting arguments, even if fully aware of the conventions, and possibly interfering with the command by hasty quits, carriage returns, and other temperamental reactions. And then later, when defining the "SHELL". We may envision a common procedure called automatically by the supervisor whenever a user types in some message at his console, at a time when he has no other process in active execution under console control (presently called command level). This procedure acts as an interface between console messages and subroutine. The purpose of such a procedure is to create a medium of exchange into which one could activate any procedure, inside of another program if it were called. Hereafter, for simplification, we shall refer to that procedure as the "SHELL". It still surprises how little the undergraduate degree in computer science at Cambridge focuses on writing skills. Patrick Ferris 2025 5 19 OxCaml Last week, I got a toplevel with OxCaml working. This required a serious amount of work to understand the changes Janestreet have made to obscure parts of the OCaml compiler and then working those into tools like js_of_ocaml. This week, Janestreet pushed their latest rounds of changes and of course everything broke! I spent some more time fixing it all back up. I'm not entirely sure how maintainable this is. The problem is that, whilst things compile, the programs do not work together! Only when someone uses the program do the bugs surface. Patrick Ferris 2025 5 19 Other OCaml Work I worked on some other parts of the ecosystem this week. Patrick Ferris 2025 5 19 Ppxlib I helped review some changes to enable Janestreet to have ppx rewriters via attributes (usually they are via extension points). It is a bit of a controversial change to ppxlib as we try to keep the API predictable for users: Extension points are rewritten: this means the part that is rewritten is nicely delimited by the extension points start and end. Attributes extend: attributes do not rewrite the code they are attached to but rather extend the code with new AST nodes. We will see what we decided to do (most likely provide the functionality behind some kind of "expert" interface). Patrick Ferris 2025 5 19 Tiff See Michael's notes. I spent some time trying to speedup the LZW decompression of TIFF files in the pure OCaml tiff library this week(end). The two big changes to help with this are pretty common when speeding up these parts OCaml programs: Allocate less Does less work In terms of allocating less, the original implementation was using a char list to represent LZW strings. Manipulating these becomes quite costly, particularly since the most common operation is appending a single character to the end of a list. Converting this to use OCaml's immutable string saved a ton of allocations. In terms of doing less work, I opted to bypass Cstructs sane (but slow) bounds checks in some of the "hotter" parts of the code. In particular, LZW ends up reading potentially huge arrays full of bytes one-by-one. So reading each byte needs to be quite snappy. This is a bit of a trade-off in terms of "safety" but we are in control of this code so I'm not too worried about that. Here are some results decompressing a fairly large array of some elevation data. Patrick Ferris 2025 5 10 https://patrick.sirref.org/weekly-2025-05-04/ weekly-2025-05-04 /weekly-2025-05-04/ Hazel and OxCaml I missed a week of posting last week, mainly because I spent more time writing posts. Patrick Ferris 2025 5 10 Hazel of OCaml I mentioned previously that I was building a tool to transpile OCaml code to Hazel. This work is now in a good enough state that, along with one of my students, we have transpiled a good number of OCaml programs to help them write their evaluation for their third-year project. I wrote up a little summary of that work, which I've transcluded below. Patrick Ferris 2025 5 2 https://patrick.sirref.org/hazel-of-ocaml/ hazel-of-ocaml /hazel-of-ocaml/ A Transpiler from OCaml to Hazel Over the past few months, I have been piecing together a transpiler from Hazel to OCaml. This is, in part, to help one of my third-year undergraduate students who is working on type error debugging in Hazel. Patrick Ferris 2025 5 2 Typed Holes Hazel is a functional programming language with typed holes. Holes are pieces of your program that have not yet been filled in. Holes can appear anywhere in your program both as expression or types. Hazel can still evaluate your program in the presence of holes. To get a flavour of Hazel, take a regular map function for lists. fun xs -> case xs | [] => [] | x :: xs => f (x) :: map(f)(xs) end in map(fun x -> ?)([1, 2, 3])]]> The question mark (?) is a hole. The program evaluates to the following expression of type (for people more familiar with OCaml types ? list). Hazel supports local type inference but nothing involving unification variables. For example, a simple add_one function in Hazel (fun x -> x + 1) has type ? -> Int. Patrick Ferris 2025 5 2 From OCaml to Hazel The ability to transpile OCaml programs to Hazel programs is motivated by one simple thought: there are more OCaml programs than there are Hazel programs. This could help bootstrap projects by alleviating the need to rewrite boilerplate code (e.g. URI parsing or standard library functions for strings). Patrick Ferris 2025 5 2 A Transformation of Syntax Hazel markets itself as an "Elm/ML-like functional programming language". From the previous example of map, it should be apparent just how close to OCaml the language is. It turns out that a majority of the transpiler is a transformation of syntax. Take a simple ADT for an arithmetic programming language. type expr = | Float of float | Add of expr * expr | Sub of expr * expr | Mul of expr * expr | Div of expr * expr And when we run hazel_of_ocaml over this OCaml type declaration. Not much has changed expect some syntax. Hazel does not have a notion of top-level expression so hazel_of_ocaml wraps the program into one set of value bindings. For the most part, Hazel acts as a subset of the pure, functional part of OCaml. At the time of writing, this subset is fairly limited with no support for modules or labelled records out of the box (there are plenty of development branches with these features). If we try out the same map function but written in OCaml and transpiled to Hazel we get. let rec map f = function | -> | x : : xs -> f x : : map f xs Which becomes the following hazel program. fun x1 -> case x1 | [] => [] | x :: xs => f(x) :: map(f)(xs) end in ?]]> We could have a field day discussing the syntax of OCaml and Hazel (parentheses for function arguments, well-scoped cases for pattern-matching, a different arrow for pattern-matching etc.). What would be more interesting is taking a look at how to handle polymorphism in Hazel. Patrick Ferris 2025 5 2 Explicit Polymorphism Hazel has explicit polymorphism. So far, we have not seen it as we have let the types have holes in them. The map function in OCaml has the following type. val map : 'a -> 'b -> 'a list -> 'b list We must remind ourselves (by reading Craig's excellent blogpost on the matter) that in OCaml ... type variables in signatures are implicitly universally-quantified So in reality, we have that map has the following type. b) -> a list -> b list]]> In Hazel, we have to explicitly type our map function to be polymorphic. Not only does this mean the type annotation requires universally quantified type variables, but we must also perform type application wherever we choose to apply the map function (whether that be recursively or somewhere later in our program). forall b -> (a -> b) -> [a] -> [b] = typfun a -> typfun b -> fun f -> fun xs -> case xs | [] => [] | x :: xs => f (x) :: map@@(f)(xs) end in map@@(fun x -> ?)([1, 2, 3])]]> forall introduces a universally quantified type variable into our type annotation, and typfun introduces it into the function itself (à la System F). Type application requires @<T> where T is some type. This allows hazel to quite easily support higher rank polymorphism, but we will not worry too much about that. Patrick Ferris 2025 5 2 Propagating OCaml Types into Hazel Most often, OCaml users interact with prenex polymorphism (rank-1) where the universal quantifiers are at the front of the type. OCaml does support quantifiers inside certain types like records. What this means for the transpiler is that we can reuse OCaml's type inference to safely instantiate the correct type annotations and type applications in Hazel! To do this, hazel_of_ocaml uses Merlin to inspect the type of the function in either a value binding or at the point of a function application. Take a simple, polymorphic length function. let rec length = function | -> 0 | _ : : xs -> 1 + length xs let int_len = length 1 ; 2 ; 3 let str_len = length " only " ; " two " When we run this through hazel_of_ocaml with the -type flag we get. [a] -> Int = typfun a -> fun x1 -> case x1 | [] => 0 | _ :: xs => 1 + length@(xs) end in let int_len : Int = length@(1 :: 2 :: [3]) in let str_len : Int = length@("only" :: ["two"]) in ?]]> hazel_of_ocaml has correctly instantiated the type for length inside the recursive function and then in each case with the integer list and the string list. Patrick Ferris 2025 5 2 A Corpus of Hazel Programs The impetus for this work was to derive a corpus of ill-typed Hazel programs. Luckily, such a corpus exists for OCaml! Seidel et al. created a corpus of OCaml programs from their undergraduate students at UC San Diego. Some of these programs have been transpiled to Hazel. Patrick Ferris 2025 5 2 Future Work Hazel is a fun, research programming language. Potential third-year students may find it interesting to take this work further. For example, how would this look in terms of a module system? From a purely engineering perspective, plenty of work would be needed to convert a multi-library OCaml project to Hazel (e.g. handling the cmi files). Another line of research would be to have Hazel target one of the intermediate representations in OCaml which would give Hazel a fully functioning compiler to "native" code? Patrick Ferris 2025 5 10 OxCaml I spent some time this week getting more familiar with Oxidized OCaml. I have a habit of wrapping new OCaml tools and libraries into toplevel, browser applications. For example, try-irmin and try-eio. Naturally, I tried to wrap OxCaml into a toplevel so people could play around with the new modes that are part of the OxCaml type system. This turned out to be a lengthy debugging session (where type declarations did not align so the raw Obj.repr js_of_ocaml representation was broken for some parts of the toplevel). I would say that I do question the time-spent/value trade-off, but a mostly working toplevel with OxCaml is available at: https://patrick.sirref.org/oxcaml. Patrick Ferris 2025 4 21 https://patrick.sirref.org/weekly-2025-04-21/ weekly-2025-04-21 /weekly-2025-04-21/ Shelter in the Forest I spent much of this week working on Shelter and things related to it. Some of that time was also spent on Forester. Patrick Ferris 2025 4 21 Forester I added two important quality-of-life features to my frontend to Forester this week. Patrick Ferris 2025 4 21 Bibtex Support I had previously mentioned adding support to Forester for Markdown. This week I added support for Bibtex too. From any *.bib file in your forest, Forester will now dutifully recognise it as a Bibtex file and convert, as best it can, all the entries into Reference trees. I'm becoming quite convinced of this model at the moment. I'm using Forester's Code.t as a target representation. In fact, to ease the process, I really shouldn't spend all my time on my website, I have reused my Yaml.t -> Code.t and Markdown.t -> Code.t functions in the Bibtex parser. To see it in action, you could have a look at the Build Systems à la Carte paper which is generated completely from Bibtex. Patrick Ferris 2025 4 21 Full Heading Support The eagled-eyed viewer may have noticed that the table of contents for this page has more than one level. I finally caved and spent an evening rejigging my Cmarkit.Doc.t -> Tree code which was hacky and broken and is now less hacky and less broken. In addition, headings support links and emphasis etc. Patrick Ferris 2025 4 21 Lunch with Jon Sterling I had a delightful lunch with Jon Sterling discussing the future of Forester, the nature of the Web (old and new) and the success posting weekly updates for our colleagues. Thanks Jon. Patrick Ferris 2025 4 21 Shelter Fixes I spent a good chunk of my week fixing bugs in Shelter with the aim to perhaps setup a VM somewhere and let people kick the tyres of what we've got so far. The first bug is pretty annoying. At the moment, our filsystem backend is ZFS and we make heavy use of snapshots and cloning in order to provide time-travelling capabilities. Unfortunately, ZFS will take a snapshot before data has fully made it to disk (or whatever is the equivalent point it should reach in ZFS). Commands that generated lots of disk activity would be snapshotted in a half finished state and this would cause all sorts of problems. Thanks to Mark Elvers for the pointer to how OBuilder uses ZFS for the OCaml macOS builders which unmount datasets immediately therefore inducing a flush of sorts. Shelter now follows a similar model with all of the slow downs that create. Anil and I discussed some amalgamation of overlayfs, tmpfs and ZFS to alleviate some of this but for now that's a premature optimisation. Patrick Ferris 2025 4 21 A small eDSL for Shelter Whilst testing Shelter, I ended up wanting a way to programmatically invoke the different run commands. This is similar to say a Dockerfile, but maybe with a little more expressivity. This lead me to revisit the Build systems à la Carte paper and rediscover selective applicative functors. I started playing around with a selective applicative interface to Shelter, this would allow you to express your dependencies statically but select them dynamically (as the paper says). module D = Shl Identity let shelterfile = let open D in let base_image = from " alpine " in let is_node_lst img = String . equal " v22.15.0 " stdout img in let cmds base = let node_version = run " node --version " base in Select . if' Select . map ~ f : is_node_lst node_version run " " run " " base in with_session " node " cmds base_image From this, we get a slightly more expressive way to describe images. Patrick Ferris 2025 4 21 Geocaml TIFF Library I was pleasantly surprised to receive a pull request from Michael adding support to ocaml-tiff for reading TIFF files compressed using LZW. I was also surprised to hear the TIFF LZW is a little different to others. In trying to get this PR merged, I moved the initialisation of the Eio eventloop to outside each individual test case. This one change then completely broke the entire test suite. After a period of debugging and help from Thomas Leonard the root cause was OCaml's OUnit2 library using process-level parallelism (via Unix.fork), sharing the ring between the parent and the child lead to the issues. Read more about that issue on the Eio issue tracker. Patrick Ferris 2025 4 21 Part II Students As the new term begins, it signals that there are only just over two weeks for the final year undegrads at Cambridge to submit their dissertations. The four students that I help supervise have been sending me drafts of their work (and reminder you can read about their projects) and I'm very impressed. I'm sure the next two weeks will be stressful, but I'm proud of what they have accomplished over the past academic year. Patrick Ferris 2025 4 21 https://patrick.sirref.org/weekly-2025-04-14/ weekly-2025-04-14 /weekly-2025-04-14/ Catching Up! This is a bit of a catch-up post. A week or so in Belfast has thrown me a little off course with the weeklies. Patrick Ferris 2025 4 21 Outreachy December 2024 and Beyond In the last week or so, we have come to the end of the December 2024 round of Outreachy. For those who do not know about Outreachy, it facilitates and organises internships in open-source. Internships for those who are historically underrepresented and impacted by systematic bias in tech. We, the OCaml community, held our biannual Demo Day where our current cohort of interns can show off what they have been building for three months. This round, we only had one intern, Abdulaziz, who worked on the OCaml API diffing tool. The next round has just completed the contribution phase and our next cohort of interns is being selected. Patrick Ferris 2025 4 21 Ppxlib Supports OCaml 5.4 (mostly) This week, I added support for OCaml 5.4 to ppxlib. A rather non-trivial change to the codebase due to changes in the representation in Longidents (which now have location information for all segments of the Longident). OCaml 5.4 has labelled tuples, a light-weight record-like syntax similar to labelled arguments in functions. let x = ~ x : 1 , 10 , ~ y : 1 val x : x : int * int * y : int let add ~ x , ~ y = x + y val add : x : int * y : int -> int As of writing this post, we are waiting for the full OCaml 5.4 feature freeze and magic number bump. Note this is support for OCaml 5.4 (in ppxlib terms, we know have migration functions for 5.3 <-> 5.4) not a bump to use the 5.4 parsetree in ppxlib. (ppxlib)[Ppxlib]The Future of I have now been helping maintain Ppxlib for a while and I am beginning to wonder about the long-term vision for the project. Ppxlib is a central component to any modern OCaml library or tool, Sherlocode reckons there's 18.3k instances of . Yet, every compiler release is a painful, two-phase process for the maintainers. First, we have to support the new release of the compiler. This is mostly okay but does involve adding lots of code to Ppxlib (we need a whole copy of the Parsetree!). Later on down the line, we have to bump the AST and this often breaks many ppxes. The biggest issue is still the tension between abstract types and pattern-matching. By exposing the Parsetree directly to user's, they can pattern-match against the AST and build their ppxes in a rather straight forward way. However, this comes with the risk that the type may change thus breaking their code. I have mentioned before about plans to use "views" to help with this, but I do not see that happening any time soon. It feels to me, we should be able to allow a user to select their AST to work with. How this interacts with modules like Ast_builder is unclear to me, but it would mean user's can remain on older ASTs even when ppxlib bumps the internal, main AST. Patrick Ferris 2025 4 21 Sherlorocq Rocq is a theorem prover (originally called coq) from many of the same folks who brought you OCaml. I have been interested in theorem provers for a while, more from an engineering perspective and less from a theoretical one. My good friend, Derek Sorenson, taught me about Rocq when we encoded mergeable replicated datatypes. Later I had a go at encoding decentralised information flow control (also in Fstar). During this process, I found it quite challenging to find Rocq code that I could learn from. Compared to using say OCaml's Stdlib, using Rocq's standard library of datatypes and proofs did not come naturally (not helped by the fact that at the time I could not get the LSP to play ball). This lead me to port Arthur Wendling's excellent Sherlocode to do the same thing only for Rocq code. Luckily, it uses regular expression matching with a trigram index so it is not tied to any particular programming language. Additionally, it is somewhat opam-centric (not actually that much) which is perfect for Rocq! Anyway, I've had some interest from the people at Inria to bring the server back to life. Their wish is my command! Sherlorocq is back online! Patrick Ferris 2025 3 31 https://patrick.sirref.org/weekly-2025-03-31/ weekly-2025-03-31 /weekly-2025-03-31/ Shelter, Hazel and More! Last week I focused on Shelter -- our idea that shells should have the same ability as reproducible build tools like Nix or Docker. To this end I now have a fairly fleshed out prototype. Patrick Ferris 2025 3 31 Shelter Prototype Shelter is a spin-off from the work Michael and I started with Shark. It takes the same ideas but applies them directly to a shell-like interface. We're still in the middle of working all of this, but you can read more about it at Shelter. Patrick Ferris 2025 3 31 Forester Hacking As you can probably tell, my website is still using Forester. I rebased my Markdown branch to include the new Atom syndication feature. Alongside that I added support for arbitrary HTML injection into Forester via codeblocks in Markdown. This was actually very straightforward thanks to Markup and being able to re-parse Forester syntax in the middle of converting a Markdown document. The HTML for the shell in Shelter uses this feature. If you are interested in taking this custom Forester for a spin, there's a branch on Github. In fact, nearly the only change beyond letting the core engine know about markdown files, is adding a new parser frontend. Patrick Ferris 2025 3 31 Hazel For one of my Part II students, I've been prototyping a transpiler from OCaml to Hazel. This has gone pretty well and now supports type annotations as well as straight-forward implementation translation. Consider the following OCaml map function. let rec map f = function | -> | x : : xs -> f x : : map f xs The tool, hazel_of_ocaml can translate this to Hazel code, including making the polymorphism explicit. forall b -> (a -> b) -> [a] -> [b] = typfun a -> typfun b -> fun f -> fun x1 -> case x1 | [] => [] | x :: xs => f(x) :: map(f)(xs) end in ?]]> You can copy and paste that codeblock into the hazel playground. But do note that it still needs some manual editing to add the type applications in directly (map@<a>@<b>). With the right amount of type-inferencing and scoping I actually think that you could place those type applications in yourself. This could make a nice Part II project I think. Patrick Ferris 2025 3 31 Ppxlib I recently wrote about the painful experience of miragrating ppxlib to the 5.2 OCaml AST. This week, Nathan Rebours and I merged a PR to add the 5.3 AST! Ppxlib has been playing catch-up with the compiler and we decided it was best to try to catch up quickly and deal with the ecosystem fallout all at once rather than incrementally. With this new AST merged, ppx authors can now use the new Pexp_effect parsetree node. I'll write a little more about this in a separate post soon. 2025 2 17 https://patrick.sirref.org/weekly-2025-02-17/ weekly-2025-02-17 /weekly-2025-02-17/ Vpnkit, Void Processes, LSP Servers Previous weeklies used strong emphasis to distinguish sections. This comes from Forester's philosophy about atomicity of the content in your forest. However, subtrees are supported! I quickly hacked together the ability to use subheadings to indicate subtrees. This is strictly less expressive than the of Forester's default syntax as we cannot close heading sections in Markdown. This weekly uses subtrees. 2025 2 17 Vpnkit I spent some time this week trying to upgrade vpnkit to OCaml 5. I was originally working on a paper idea which might need benchmarks, but Anil and I decided we could simply point to the port I did and show how it has simplified much of the code. 2025 2 17 https://patrick.sirref.org/void-processes/ void-processes /void-processes/ Void Processes Work continued on implementing (and fully exploring) void processes. A lot of the groundwork already exists in Jake Hillion's master's thesis. This week I added a feature that I needed to help build the processes we need for the shell I'm building: mount points with modes! In addition to the root mount (taken care of with pivot_root), we need to be able to add additional mounts into the process' environment. These can now be added. All mount points can be mounted readonly or readwrite. Here is the "Hello, World!" example (the /say/hey program has been statically compiled using musl-gcc). let status = let void = empty |> mount ~ mode : R ~ src : hey_dir ~ tgt : " say " |> exec " /say/hey " in let t = Void . spawn ~ sw void in Promise . await Void . exit_status t There really is nothing else in there. Without specifying a root mount, the void process is started with an empty tmpfs root. Next on the list is networking! 2025 2 17 LSP Servers I got a little side-tracked building a library for writing LSP servers in OCaml: mlsp. This may seem a little unrelated, but it isn't. The LSP has become the de facto standard for communicating between an editor and a programming language environment. If you have used VSCode to write a program in Python, chances are you are using the official extension which gives you linting, formatting, code navigation etc. All of these features are communicating using the LSP. It seems Code Mirror can already proxy over a websocket for LSP support too (we might not even need that as we can compile OCaml directly to JavaScript/Webassembly and have the whole thing running locally!). 2025 2 17 Open-Source & Community Anil and I had a great conversation this week about building community especially as it pertains to open-source and OCaml. I've been going back over Governing the Commons, but have already discovered The Real World of Technology! More thoughts on all of this soon... maybe 2025 2 10 https://patrick.sirref.org/weekly-2025-02-10/ weekly-2025-02-10 /weekly-2025-02-10/ Teaching, teaching, teaching... On paper, I don't have that many students. I teach four undergraduates (first year students at Pembroke College) Discrete Maths. I supervise three third year students for their final year project and another one I co-supervise with Michael Dales. However, I do end up spending at least two full days a week on teaching. Something I really enjoy and take seriously. The time it takes is also quite unpredictable; last week for instance all the third year students had their mid-project demonstrations (a five-minute presentation in front of their peers and a few professor-types). My first year students also found two slides particularly challenging to understand from their lectures and asked if I could help explain what was going on, so I produced some materials for that. OCaml Within the OCaml universe I spent a good bit of time trying to maintain a couple of different packages: Mirage Crypto: this library provides cryptographic primitives for OCaml programs. Unfortunately, the maintainers removed direct support for Eio replacing it with a "Unix" alternative. This is not a fair swap as now Eio programs must make a dependency to Unix! Speaking to Anil, the best approach might be to make Eio programs handle this directly. I think this highlighted how fragmented open-source maintenance can be as no user of Eio seems to have bumped into this yet and the upstream maintainers did not communicate a rather large breaking change. Ppxlib: In addition to the 5.2 AST bump (which is nearly ready to be merged), I queued up a 5.3 AST bump right behind it. I plan to write up a more detailed post about the challenges of maintaining this part of ppxlib. Paris I spent some time in Paris at the end of the week. I enjoyed visiting the Musée d'Orsay and in particular their collection of impressionist paintings. Here's my favourite from that visit by Camille Pissarro. Woman in an Orchard (Spring Sunshine in the Meadow at Eragny) 2025 1 27 https://patrick.sirref.org/weekly-2025-01-27/ weekly-2025-01-27 /weekly-2025-01-27/ AT Protocol and an IR for Wikis AT Protocol This week I've been diving into the AT Protocol. The Authenticated Transfer Protocol, aka atproto, is a decentralized protocol for large-scale social web applications. The protocol could be a candidate for the glue that holds together a distributed, computational wiki network. The protocol, it seems, is very similar to IPFS. Thankfully, a few years ago, I was working on building out a suite of OCaml libraries for working with IPFS. For example, ocaml-cid, self-describing content-addressed identifiers. # let s = " zb2rhe5P4gXftAwvA4eXQ5HJwsER2owDyS9sKaQRRVQPn93bA " ; ; val s : string = " zb2rhe5P4gXftAwvA4eXQ5HJwsER2owDyS9sKaQRRVQPn93bA " # let cid = Cid . of_string s |> Result . get_ok ; ; val cid : Cid . t = < abstr > # Cid . pp_human Format . std_formatter cid ; ; cidv1 - base58btc - raw - ident sha2 - 256 length 32 digest 6e 6f f7 95 0a 36 18 7a 80 16 13 42 6e 85 8d ce 68 6c d7 d7 e3 c0 fc 42 ee 03 30 07 2d 24 5c 95 To this end I have built out some more OCaml libraries for working with atproto, including: atproto-data: the atproto data model, similar to JSON-LD. ocaml-did: an OCaml library for working with decentralized identifiers. atproto-lexicon: atproto's schema format, I've been building a quick tool for doing an OCaml translation from these schemas. I managed to get a post published from the OCaml library after fixing it up and porting it to Eio. An IR for Wikis I started working on a proof-of-concept intermediate representation for Wikis -- I imagine it a bit like malfunction but for computational wikis i.e. a target for Wiki building tools that allows different front-ends and servers to communicate in a common IR for exposing key functionalities of a wiki: Links: External links, cross-wiki backlinks Versioned, temporal feeds Etc. Other PhD Work I met with most of my Part II students this week, and I'm excited about their work. Progress reports are due this week and next they have a presentation to give. In discrete maths this week we did induction. Next up is a big section on sets, functions, bijections etc. Misc. I was happy to find the Information Controls Fellowship Program. The Information Controls Fellowship Program (ICFP) cultivates research, outputs, and creative collaboration on topics related to repressive internet censorship and surveillance. 2025 1 20 https://patrick.sirref.org/weekly-2025-01-20/ weekly-2025-01-20 /weekly-2025-01-20/ Forester, ICFP, Wikis This week was my first full week back from the break and I found it challenging trying to get back into what I had been working on previously. ICFP Papers In conversation with Anil, we looked at options for submitting a paper to ICFP. I wrote up some notes on some options we discussed. Forester I spent some time this week converting this website to using Forester. I'm not a huge fan of the syntax, especially as a lot of my site's content was already in markdown. So I wrote a markdown frontend to forester which is available on Github. The markdown frontend integrates very nicely and only a few changes were needed in the core logic of Forester itself. Additionally, for any features not directly supported in markdown there is an escape hatch using code blocks such as: Personally, I'm still getting to grips with the bottom-up approach to building this site, atomically creating notes and reference cards that then are linked in many places. I'm excited to see how I can integrate some of the Forester concepts into "Shark". OCaml In the OCaml world, I spent time on ppxlib, reviewing PRs to update the lower bounds of the library, fixing effect syntax problems and bumping internal AST to 5.2. I've also spent some time looking into de-objecting Eio, making the OCaml types more friendly to new users. I need to revive my port of Vpnkit to Eio for the thoughts on ICFP 2025 too. I also want to modernise and make more public my OCaml code for creating little shells in OCaml too -- I think the ideas here really have legs and would like to find a conference to submit them too. I also met with the single, OCaml Outreachy intern working on ocaml-api-watch. Misc. I met with most of my Part II students this week which was nice to catch up and see how their projects are going. I also started marking work for my first year students who are at the induction part of their Discrete Maths course. Backlinks Related Patrick Ferris 2025 5 9 https://patrick.sirref.org/try-oxcaml/ try-oxcaml /try-oxcaml/ Try OxCaml https://patrick.sirref.org/oxcaml This week, I have been trying out Janestreet's Oxidised OCaml (see their ICFP paper). This adds a system of modes to OCaml for expressing things like locality. ...we’re introducing a system of modes, which track properties like the locality and uniqueness of OCaml values. Modes allow the compiler to emit better, lower-allocation code, empower users to write safer APIs, and with the advent of multicore, statically guarantee data race freedom—all in a lightweight way that only affects those in need. For example, you might say: let is_empty s : string @@ local = String . length s = 0 To get a feel for how this changes the language, you can follow their instructions on their custom opam-repository. Alternatively, you can give it a go in your browser! This is a full toplevel running with Base installed (which has plenty of OxCaml annotations on Stdlib-like functions). Happy OxCamling! https://patrick.sirref.org/mdales/ mdales /mdales/ Michael W. Dales https://digitalflapjack.com https://patrick.sirref.org/ppxlib/ ppxlib /ppxlib/ Ppxlib https://github.com/ocaml-ppx/ppxlib An OCaml library and ecosystem for preprocessing. https://patrick.sirref.org/ryangibb/ ryangibb /ryangibb/ Ryan Gibb PhD Student and Nix Guru https://ryan.freumh.org Fellow FN06 hacker. Contributions