Patrick Ferris' Weeklies

Data Provenance in Shelter

2025-11-24T00:00:00-00:00

A late update that stretches more than a week...

@ How?

A quick update on a new feature that was added recently, the @ how meta-command.

provenance of a given file. How that file came to be and who and what processes have interacted with it since.

Shelter now has a command for asking exactly that of any given file in the system.

 hello.txt
echo world > world.txt
cat world.txt > hello.txt
@ how hello.txt]]>

The previous snippet of shell script will output:

 hello.txt
Ran cat world.txt > hello.txt (read: "world.txt")]]>

There are plenty of optimisations and improvements to make, but this is a nice first step to showing the kinds of feature you get when programming with Shelter.

OCaml Roundup: November 2025

A roundup of some of my OCaml activity this month!

Ppxlib Updates

I managed to get a substantial amount of ppxlib work completed in the last month. Some of those bits are described in detail below.

Initial OCaml 5.5 Support

We recently merged 5.5 support into our main some new features coming to OCaml including the ability to define any structure locally. Most OCaml programmers will be familiar with things like let open M in

External types have also landed, bringing with them a new "type kind" in the AST. You can read about them in the manual.

Bug Fixes for Attribute Rewriters

There was a bug in the error handling for attribute rewriters -- a relatively new feature to have been added to ppxlib.

In some places, ppxlib makes use of a With_error.t monad. Unlike the Result.t monad, With_error.t allows you to collect errors rather than find the first error and thread it through your program. This can make sense if your program can meaningfully do work after encountering an error. Unfortunately, in the logic for replacing attributes, the code could not recover from an error where there were duplicate attributes.

Value binding constraints in ppxlib

I spent a good few hours understanding why the CI tests in my PR to mirage/repr were still failing. It turns out, the bug, is fairly simple to reproduce. Consider the following OCaml code:

                let g : int = 1

We have a let-binding that contains a type constraint. Where, in the abstract syntax tree sense, do we attach the constraint? Since OCaml 5.1, we have quite a few options for how this can be represented.

The first is using the Ppat_constraint AST node which allows you to attach a type constraint to any pattern.

                    | Ppat_constraint of pattern * core_type

x + 1]]> So it doesn't quite capture what we want for this binding constraint.

On older OCaml compilers, let g : int = 1 also includes a Pexp_constraint on the body of the value binding. This makes sense if we consider a more general case of the syntax where we allow functions.

                let g x : int = x + 1

In this example, we no longer parse a Ppat_constraint and instead this is treated as syntactic sugar for:

                let g x =  x + 1 : int

In OCaml 5.1 a new field for value bindings was added allowing type constraints to be tracked directly in the bindings themselves.

                and value_binding =

   

    pvb_pat: pattern;

    pvb_expr: expression;

    pvb_constraint: poly_constraint option;

    pvb_attributes: attributes;

    pvb_loc: Location.t;

: int pvb_constraint directly. This is a nice change, bringing the concrete syntax of the language closer to the parsetree. For ppxlib, however, it does create a bit of a headache as we now have to encode the different versions of expressing type constraints into the AST of the compiler before 5.1!

So what was the bug? The author's of repr had added some value bindings to circumvent some "unused value warnings" in OCaml. They choose to write these bindings like:

                let  _ : int  = 1

Here, I have simplified the types and the body of the binding. The significant thing to notice is the parentheses surrounding the value binding's pattern. This forces the AST representation to be a single Ppat_constraint node regardless of the current compiler version. No Pexp_constraint and no pvb_constraint.

However, using ppxlib with a compiler less than 5.1 meant that this did not roundtrip cleanly. Instead, the value after passing through ppxlib's migrations was:

                let _ : int = 1

After bottoming out at the issue, I discovered we already had a test case for this in ppxlib! Whatsomore, ocamlformat.0.20.0 repr transforms instances of let x : t = ... into ! This limitation does not seem inherent to the parsetree, the printer or the migrations. But I have

Outreachy

December 8th 2025 marks the start of the next round of Outreachy. I have been preparing some issues for geocaml/ocaml-tiff and I am excited to get started with Tambe Salome!

Graft and Bib

I spent some time at the start of the month updating and fixing Graft and Bib.

I added named subtrees to the markdown syntax. This allows users to create subtrees with headings that are also externally linkable.

With some more bug fixing, I hope to release both libraries before the new year!

Shelter imports

2025-11-15T00:00:00-00:00

Most of this week was spent fixing bugs in Shelter in an attempt to get it ready to run the LIFE pipeline.

To do so, Shelter requires a mechanism by which to pull in files from the outside world. That could be configuration files, source trees, data etc. For now, this has taken the form of a crude @ import statement. In true Forester fashion, I have transcluded the foreign tree from my Shelter logs.

Import statements

Welcome to the first Shelter log! In this log I will explain the work I have been doing to add an @ import meta-command.

The syntax thus far is relatively straightforward.

]]>

There are a few key principles underlying @ import.

It should be at least as expressive as Docker's COPY ... command.
It should deal with a wider variety of import sources. In addition to local https:// archives over ssh etc.
Imports, where possible, should be catalogued and shared across sessions.

URIs as Sources

branch already allows users to import local files and directories into their session. For example:

By allowing arbitrary URIs in the import command, we hope to force users away from downloading data via curl or python scripts. Lifting these side-effectful imports into Shelter allows us to manage the data in a much cleaner fashion.

Scripts as Sources

One idea I had was to allow users to invoke scripts as there "import source". Something like:

And using the same eBPF tracing we use during normal execution we can capture a fairly good idea of the tools and files needed to perform an import.

Sharing Imports

One example of better data management is that Shelter imports can be

When a user imports some data, they can optionally name that piece of data. For example:

Shelter stores that away and makes a link between the name, the URI and the underlying data. A user may then import the same piece of data into a different session by simply writing:

intent via the naming scheme, it makes it possible to then version the data and have a means by which to update said data, one could imagine:

Which will retry the original URI and see if the data has changed, or if it has not. There is an implicit versioning number that users can specify in the name should they choose to.

Eio Debugging

One of the promises of Eio's capability model is an improved debugging experience. Your program must be designed in such a way as to make resources explicit. If you have a load_config function, more than likely it will need access to the file-system.

            val load_config : _ Eio.Path.t -> Config.t

Once all of your resources are explicit and threaded through your program, it

This week I was trying to get the LIFE pipeline running in shelter. Part of that involves pulling the appropriate geospatial container image, but my code was terrifically broken, failing with what looked to me with a Docker error.

With only this clue, I knew that it was probably in the execution of some command in a child process, but which one? To trace this quickly, I could plug in a slightly different instance of the _ Eio.Process.mgr capability. Essentially unchanged from the one provided by the unix backend except that it would also print to stderr the command it was trying to run!

            let debug_process_mgr  mgr : 'a Eio_unix.Process.mgr  : 'a Eio_unix.Process.mgr

    =

  let module D = struct

    type t = unit



    let spawn_unix   ~sw ?cwd ?pgid ?uid ?gid ~env ~fds ~executable args =

      Eio.traceln "Spawning subprocess...  " Fmt. list string  args;

      Eio_unix.Process.spawn_unix ~sw ?cwd ?pgid ?uid ?gid mgr ~env ~fds

        ~executable args

  end in

  let module V = Eio_unix.Process.Make_mgr  D  in

  Eio.Resource.T   , Eio_unix.Process.Pi.mgr_unix  module V

Whist there is a fair amount of jumping through first-class-module-shaped hoops here, all I am doing is rewrapping an existing "method" to a spawn a unix child process. Then using this helper function, I can push that into my standard environment.

            let env =

  Eio_unix.Stdenv.with_env

  ~process_mgr: debug_process_mgr env#process_mgr 

  env

in

Shelter.main config  env :> _ Shelter.env  dir cmd_file

And sure enough I was querying for the Docker user before pulling the image! I hadn't run into this, as for the last while I have been using alpine and debian images that already exists locally.

Taking stock & Graft updates

2025-11-10T00:00:00-00:00

I ended up working on a real mix of things last week. I managed to get two large posts uploaded to this site: an ICFP recap and a deeper-dive into the random bits of OCaml hacking. This was pretty useful in terms of taking stock of where I was at, I always find post-conference, I'm a little lost. The conversations and ideas from ICFP are simultaneously exciting but also pretty inactionable.

Graft Updates

It would seem that there is a potential second user of Graft! A great source of motivation for some tidying and feature implementing. At the end of last week I fixed some little bugs in Graft users to name their subtrees. Upstream Graft also now pins and uses my Bibtex implementation.

OCaml Geotessera

As I discussed in my OCaml roundup, I spent a bit of time polishing the OCaml geotessera code. In particular this meant starting to handle low-level geospatial transformations allowing users to project latitudes and longitudes into the rows and columns of their raster data. I feel like I'm stumbling onto a path Michael has carved out with Yirgacheffe. I'm following along with his work excitedly and we had a good catch-up last week discussing how libraries should handle things like automatic, numeric precision conversion.

As a fully paid-up member of the static types fan club, I'm often evangelising the benefits of types. However, I hit a little quirk of the Nx sum function:

            let sum nx = Nx.sum nx |> Nx.item  ;;

val sum :  'a, 'b  Nx.t -> 'a

let arr = Nx.create Nx.int8   2     64; 64  ;;

val arr :  int, Bigarray.int8_signed_elt  Nx.t

sum arr;;

- : int = -128

Even though sum takes an array that stores elements with representation 'b and allows you to read/write to the elements with OCaml values of type 'a, the sum function performs the calculation in the world of the 'b overflow! Perhaps not too surprising but easy to overlook! The moral of the story is that we still need to be careful in the OCaml world too.

Shelter

I did a little cleaning up of the Shelter code too. I noticed a few bugs from the demos I did at ICFP. Most were pretty straight-forward to fix, and the test-suite is now checking parallel for-loops and conditional execution are working.

This week

I hope to get a few things done this week:

Run some part of the LIFE pipeline through Shelter.
Deploy the Shelter daemon allowing other people to connect and take it for a spin.
Spend some time writing and reading, pulling ideas together rather than always writing code!
I have a slightly harebrained scheme that combines Shelter, atproto, Forester pull that together.

Back to shelter

2025-07-18T00:00:00-00:00

Capnproto Interface to Shelter

I spent some time this week thinking about decoupling Shelter's builds from the CLI tool. This would allow users to connect to a remote daemon to build and store their sessions, opening up the possibility of using Shelter on Windows and macOS.

The design so far keeps the metadata local, in an Irmin database. I will probably then augment the stored information with some kind of unique daemon identifier to know where the actual data is.

This is all still quite fresh, and I will have more by the end of next week.

OCaml-related Things

Something I decided to do last week, was to make a clearer delineation between my OCaml work and research. That line is not always very distinct, but I have split of a separate set of weeklies for OCaml related things.

Fear not, the OCaml weeklies are a subset of these weeklies and so I will, when I remember, transclude them for you.

OCaml Weekly 2025 w29

Ppxlib

I met with Nathan this week to discuss future plans for ppxlib. The current state of affairs is that ppxlib is becoming unmaintainable. This is primarily a knock-on effect from changes being made to Nathan has a plan that will provide two key properties.

Migrations, which allow old compilers to be used with new ppxlib releases, will be more compatible. For example, we will be able to migrate new features downwards and back up without raising an error.
Ppx authors will be able to use new features in an opt-in workflow, rather than ppxlib in ppxlib.0.36.0 significantly whilst still allowing users to write ppxes for new OCaml features.

I also started looking into some older issues in ppxlib related to performance. This is work-in-progress, but I am trying to improve the performance of some passes done by ppxlib. To better understand what was making ppxlib slow, I wanted to use runtime_events_tools but I was dismayed to see it wanting to install over 100 packages! I opened a PR to reduce the number of packages. I think this kind of work goes a little unrecognised as it is not very glamorous. However, I think it really benefits the OCaml community in the long run.

Outreachy

In Outreachy news, we had a wonderful mid-internship video call with all the interns and mentors to catch-up on how everyone is getting along. Seeing the progress everyone has made was great! I am very grateful for the work that Michael and Steve have put in so far to make this a very successful Outreachy round for OCaml.

In sadder news, an email was shared with all Outreachy mentors detailing the increasingly critical financial situation the project finds itself in. There are ongoing discussions about how costs can be cut including potentially only running a single round a year.

Graft

With the release of Forester.5.0, I made a plan to make a release of Graft.0.1. Unfortunately this is blocked by a new release of hilite, a tool I built for doing build-time syntax highlighting for OCaml code. This powers the syntax highlighting on ocaml.org.

PhD Work and Scientific Programming

2025-06-30T00:00:00-00:00

I have been away on holidays. Some of that was spent hiking in the Massif de Calanques in Marseille, unfortunately this week they were on fire. Anyway, this "weekly" fills in the blanks for the rest of June 2025.

Revisiting hazel_of_ocaml

I spent some time thinking and writing about hazel_of_ocaml and the benefits of OCaml as source language and Hazel as a target language. The stability of OCaml with a powerful type-system makes it very expressive.

Hazel, on the other hand, is a hotbed of PL theory research. The typed holes, which are fully supported within the language, are particular powerful targets for a transpiler. They allow transpiler developers to incrementally support the language whilst still getting full feedback of their generated source code. Whilst it is a niche use-case, I thought it was an interesting point to note.

A second year report

I wrote my second year report at the end of June. The process was a bit chaotic, but I think ultimately it was useful for clarifying some ideas I had rattling about in my head.

Part of that was working on my publications page and, through a conversation with Anil, realising that my work on Graft was in many ways a part of my own research.

If you are interested, feel free to peruse both my first year and second year reports. Take them with a pinch of salt! I think it is useful to be fairly open with this sort of thing when you can be. Maybe someone will find them useful.

What is scientific programming?

This question arose whilst writing my second year report. It got me thinking about how we distinguish scientific programming from other kinds of programming. I came across a paper by Tim Storer: Bridiging the chasm: A survey of Software Engineering Practice in Scientific Programming. I enjoyed a few aspects of this paper.

Relating scientific programming to the scientific method so directly was pretty useful insight for me. Falsifable hypotheses, repeatable experiments and reproducible results when applied as characteristics of the computer stack argue for better tools to enable this.
The breadth of Storer's case-study analysis on how scientific programming has "gone wrong" is very good, if anyone needs a reference to draw on to help support their tools or research, this seems like a good index to use.

Ppxlib

I was reminded again of the pain of helping maintain ppxlib. We are coming to the conclusion, that in its current form, it is becoming unmaintainable. I took a stab at updating bisect_ppx to the latest ppxlib. With OCaml entering a potentially turbulent parsetree era, it might be time to take stock of this and propose some fillers to help. Nathan Rebours and I are meeting next week to discuss ideas he has to make parsetree migrations smooth sailing for all!

Geocaml

Work on ocaml-tiff has stalled, which is totally fine. Michael and I independently are working on other things and have obligations to fulfill. Maybe some day we'll carve out enough time together to push forward on this project, but not this week.

Forester as a Target Syntax

2025-06-02T00:00:00-00:00

This week included some time finishing opentrace and subsequently folding it into shelter. I have been writing up some more of the draft paper for shelter which I am excited to share in the near future.

I revisited the upgrading vpnkit PR and pushed some more fixes. I have been thinking, again, about the promise of a direct-style world for OCaml that just hasn't quite landed yet. C'est la vie.

Forester and Graft

I spent a bit of time finally pulling out my changes to Forester to add markdown and bibtex support into a standalone tool: graft.

Graft

Graft is a Forester preprocessor.

Forester syntax and produces a new forest completely written in Forester syntax.

Usage

graft simply preprocesses a forest generating Forester trees from .md, .bib and .tree files. It will copy the structure of the input directory in the output directory.

                $ graft preprocess --output=grafted-trees trees

$ forester build

This assumes that you have updated your Forester toml file to put the grafted-trees directory as your source of trees.

Example

A typical "tree" might look something like

A few things to note:

The yaml frontmatter allows you to add some of the metadata fields from Forester.
At any point in your markdown there is an escape hatch to Forester using a forester codeblock.

It is very satisfying to find the separation of concerns works quite well. For a while I had been rebasing my development branch on Forester. I was also worried about trying to get the code upstream as it pulled in many dependencies. It seems that I have a very workable solution. I welcome contributions to graft including extra input formats. I am also considering extending the Forester configuration to contain some graft configuration for how it should generate

Maths Support

As part of that process, graft now supports Markdown KaTeX. For example:

Mergeable Replicated Data Type Implementation

A for a data type \tau is a tuple D_{\tau } = (\Sigma , \sigma _{0}, do, merge) where:

\Sigma is the set of all possible states at a branch,
\sigma _{0} \in \Sigma is the initial state,
do : Op_{\tau } \times \Sigma \times Timestamp \rightarrow \Sigma \times Val_{\tau } implements every data type operation,
merge : \Sigma \times \Sigma \times \Sigma \rightarrow \Sigma implements the three-way merge strategy.

Definition 2.1 from "Certified Mergeable Replicated Data Types".

Revision Supervisions

I have been doing revision supervisions for Discrete Maths and Foundations of Computer Science. I have also been marking Operating Systems past paper questions for the same group of first years.

Opentrace and Supervisions

2025-05-26T00:00:00-00:00

opentrace and doing revision supervisions.

Opentrace

Thanks to Koonwen's excellent libbpf bindings in OCaml, I have been building a little tool called opentrace to make it easier to track an executable's inputs and outputs.

This work was inspired my Michael's self-proclaimed "gross hack": pyshark. Whilst pyshark achieves its goals by injecting code into commonly used python objects and methods, opentrace uses eBPF opentrace can remain programming language agnostic. However, less information is none about the user's intent compared to something like pyshark.

Monitoring the System

opentrace has an all command that will trace the entire system.

The --flags=O_WRONLY argument filters the events where the O_WRONLY flag was set in the call to open.

comm -wrapped is an artefact of using Nix.

Tracing an Executable

The primary use case for this tool is to inspect what files your program might be reading and writing.

The "flags" argument can specify a small boolean formula for checking the open flags of a particular event with | & ~ Parentheses can be used for precedence.

Spawning Subprocesses

One feature opentrace

opentrace is primarily an eBPF program that is loaded into the kernel and communicates with an OCaml program. Events are communicated via a ring buffer and most of the post-processing happens in OCaml. To capture subprocesses, opentrace This gives opentrace a new identifier to track, namely the cgroup.

So consider the following program.

              let   =

  Eio_posix.run @@ fun env ->

  Eio.Path. save ~create: `Or_truncate 0o664   env#fs / "hello.txt"  "hello" ;

  Eio.Process.run env#process_mgr

      "/bin/bash"; "-c"; "echo 'heya' > heya.txt"

It first creates a file using direct calls to functions like openat. Then it spawns a process which creates a new file called heya.txt. This happens in a separate process. However, with the --cgroups flag we can capture both interactions with the operating system.

Eio's Process API

I have used the Eio_unix fork action process API to be able to extend what happens in the child process. Loading most eBPF programs into the kernel requires special privileges hence the need for sudo. When a user requests for a particular program to be executed and traced, opentrace spawns a process via the Eio Process API. Opentrace defines a few new so-called "fork actions", little fragments of C code that are run after the call to fork clone Most likely this ends with a call to execve, but other calls are possible for example setuid allowing opentrace to change the user of the child process so it does not run as root into that group.

Limitations: Io_uring

Whilst testing opentrace against some of the tools I use nearly daily, I noticed some events were being missed. I tried tracing forester, and only the initial read of forest.toml was logged. It dawned on me that the reason for this was that forester eio io_uring to perform most of the IO. Most attempts to open files were bypassing the open system calls, and instead they were being performed by the kernel after reading a submission request for an openat2-style call!

This is not news to seasoned, Linux systems programmers. Io_uring bypasses SECCOMP filters for exactly the same reasons.

It is interesting to note two things here:

We can tell that forester reads the configuration file probably using something like In_channel it does
It appears that Uring is performing IO in both worker threads and directly.

The file paths are empty at the moment as I cannot find a clean way to trace openat submission requests into Uring without it sometimes going very wrong. do_filp_open io_openat2, but this seems quite brittle, and often the filename is completely garbled, so I do not set it. If anyone has any ideas to trace Io_uring more reliably, I am all ears!

Supervisions

The first year students I teach will be doing their first year exams soon. I have been helping them revise for foundations of computer science and discrete mathematics. Whilst doing so, I have been thinking a good deal about Jon Sterling's post about assessment bureaucracy. In particular:

Metashells

2025-05-19T00:00:00-00:00

This week, I feel I have been stuck fighting the OCaml ecosystem trying to keep my OxCaml work afloat. Aside from that, Ryan and I made some really nice progress with Shelter, culminating in Ryan describing it as a metashell.

Shelter the metashell

The main progress this week with Shelter was composing runc's terminal mode with entering raw terminal input mode on the Shelter side. This is inspired by Ryan's own work on capability interfaces.

Shelter remains mostly intact, acting as an interactive shell. However, just before executing a command we switch to receiving and sending raw terminal inputs and outputs. This means tools like vim now work in Shelter! Not only that, but users can now activate zsh snapshot the session. This lets you alter the granularity of snapshots from the command-line.

Louis Pouzin's "Shell"

I spent some time reading part of the multics design documentation this week. Louis Pouzin coined the term "Shell" in this document and I was reminded yet again just how important it is to be a good writer even as a "computer science researcher". For example, this excerpt from the requirements section of the document

The previous definitions imply that a command MUST be designed while keeping in mind the user, sitting at his console, wondering about what might be going on, mistyping or forgetting arguments, even if fully aware of the conventions, and possibly interfering with the command by hasty quits, carriage returns, and other temperamental reactions.

And then later, when defining the "SHELL".

We may envision a common procedure called automatically by the supervisor whenever a user types in some message at his console, at a time when he has and subroutine. The purpose of such a procedure is to create a medium of exchange into which one could activate any procedure, inside of another program if it were called. Hereafter, for simplification, we shall refer to that procedure as the "SHELL".

It still surprises how little the undergraduate degree in computer science at Cambridge focuses on writing skills.

OxCaml

Last week, I got a toplevel with OxCaml working. This required a serious amount of work to understand the changes Janestreet have made to obscure parts of the OCaml compiler and then working those into tools like js_of_ocaml.

This week, Janestreet pushed their latest rounds of changes and of course everything broke! I spent some more time fixing it all back up. I'm not entirely sure how maintainable this is. The problem is that, whilst things compile, the programs do not work together! Only when someone uses the program do the bugs surface.

Other OCaml Work

I worked on some other parts of the ecosystem this week.

Ppxlib

ppxlib as we try to keep the API predictable for users:

Extension points are rewritten: this means the part that is rewritten is nicely delimited by the extension points start and end.
Attributes extend: attributes do not rewrite the code they are attached to but rather extend the code with new AST nodes.

Tiff

See Michael's notes.

The two big changes to help with this are pretty common when speeding up these parts OCaml programs:

Allocate less
Does less work

In terms of allocating less, the original implementation was using a char list to represent LZW strings. Manipulating these becomes quite costly, particularly since the most common operation is appending a single character to the end of a list. Converting this to use OCaml's immutable string saved a ton of allocations.

In terms of doing less work, I opted to bypass Cstruct the code. In particular, LZW ends up reading potentially huge arrays full of bytes one-by-one. So reading each byte needs to be quite snappy. This is a bit of a trade-off in terms of "safety" but we are in control of this code so I'm not too worried about that.

Here are some results decompressing a fairly large array of some elevation data.

Hazel and OxCaml

2025-05-10T00:00:00-00:00

I missed a week of posting last week, mainly because I spent more time writing posts.

Hazel of OCaml

I mentioned previously that I was building a tool to transpile OCaml code to Hazel. This work is now in a good enough state that, along with one of my students, we have transpiled a good number of OCaml programs to help them write their evaluation for their third-year project.

I wrote up a little summary of that work, which I've transcluded below.

A Transpiler from OCaml to Hazel

Over the past few months, I have been piecing together a transpiler from Hazel to OCaml. This is, in part, to help one of my third-year undergraduate students who is working on type error debugging in Hazel.

Typed Holes

Hazel is a functional programming language with typed holes. Holes are pieces of your program that have not yet been filled in. Holes can appear anywhere in your program both as expression or types. Hazel can still evaluate your program in the presence of holes.

To get a flavour of Hazel, take a regular map function for lists.

 fun xs -> case xs
  | [] => []
  | x :: xs => f (x) :: map(f)(xs) 
end in
map(fun x -> ?)([1, 2, 3])]]>

? familiar with OCaml types ? list

Hazel supports local type inference but nothing involving unification variables. For example, a simple add_one function in Hazel fun x -> x + 1 ? -> Int.

From OCaml to Hazel

The ability to transpile OCaml programs to Hazel programs is motivated by one simple thought: there are more OCaml programs than there are Hazel programs. This could help bootstrap projects by alleviating the need to rewrite

A Transformation of Syntax

Hazel markets itself as an "Elm/ML-like functional programming language". From the previous example of map, it should be apparent just how close to OCaml the language is.

It turns out that a majority of the transpiler is a transformation of syntax. Take a simple ADT for an arithmetic programming language.

                  type expr =

  | Float of float

  | Add of expr * expr

  | Sub of expr * expr

  | Mul of expr * expr

  | Div of expr * expr

And when we run hazel_of_ocaml over this OCaml type declaration.

Not much has changed expect some syntax. Hazel does not have a notion of top-level expression so hazel_of_ocaml wraps the program into one set of value bindings. For the most part, Hazel acts as a subset of the pure, functional part of OCaml. At the time of writing, this subset is fairly limited

If we try out the same map function but written in OCaml and transpiled to Hazel we get.

                  let rec map f = function

  |   ->  

  | x :: xs -> f x :: map f xs

Which becomes the following hazel program.

 fun x1 -> case x1
  | [] => []
  | x :: xs => f(x) :: map(f)(xs)
end in ?]]>

We could have a field day discussing the syntax of OCaml and Hazel taking a look at how to handle polymorphism in Hazel.

Explicit Polymorphism

Hazel has explicit polymorphism. So far, we have not seen it as we have let the types have holes in them. The map function in OCaml has the following type.

                  val map : 

   'a -> 'b  -> 'a list -> 'b list

Craig's excellent blogpost on the matter

... type variables in signatures are implicitly universally-quantified

So in reality, we have that map has the following type.

  b) -> a list -> b list]]>

In Hazel, we have to explicitly type our map function to be polymorphic. Not only does this mean the type annotation requires universally quantified type variables, but we must also perform type application wherever we choose to apply the map

 forall b -> (a -> b) -> [a] -> [b] =
  typfun a -> typfun b -> fun f -> fun xs -> case xs
    | [] => []
    | x :: xs => f (x) :: map@@(f)(xs) 
end in
map@@(fun x -> ?)([1, 2, 3])]]>

forall introduces a universally quantified type variable into our type annotation, and typfun requires @ where T is some type. This allows hazel to quite easily support higher rank polymorphism, but we will not worry too much about that.

Propagating OCaml Types into Hazel

Most often, OCaml users interact with prenex at the front of the type. OCaml does support quantifiers inside certain types like records.

What this means for the transpiler is that we can reuse OCaml's type inference to safely instantiate the correct type annotations and type applications in Hazel! To do this, hazel_of_ocaml uses Merlin to inspect the type of the function in either a value binding or at the point of a function application.

Take a simple, polymorphic length function.

let rec length = function | -> 0 | _ :: xs -> 1 + length xs let int_len = length 1; 2; 3 let str_len = length "only"; "two"

When we run this through hazel_of_ocaml with the -type flag we get.

[a] -> Int = typfun a -> fun x1 -> case x1 | [] => 0 | _ :: xs => 1 + length@(xs) end in let int_len : Int = length@(1 :: 2 :: [3]) in let str_len : Int = length@("only" :: ["two"]) in ?]]>

hazel_of_ocaml has correctly instantiated the type for length inside the recursive function and then in each case with the integer list and the string list.

A Corpus of Hazel Programs

The impetus for this work was to derive a corpus of ill-typed Hazel programs. Luckily, such a corpus exists for OCaml! Seidel et al. created a corpus of OCaml programs from their undergraduate students at UC San Diego. Some of these programs have been transpiled to Hazel.

Future Work

Hazel is a fun, research programming language. Potential third-year students may find it interesting to take this work further. For example, how would this look in terms of a module system? From a purely engineering perspective, plenty of work would be needed to convert a multi-library OCaml project to Hazel cmi

Another line of research would be to have Hazel target one of the intermediate representations in OCaml which would give Hazel a fully functioning compiler to "native" code?

OxCaml

I spent some time this week getting more familiar with Oxidized OCaml. I have a habit of wrapping new OCaml tools and libraries into toplevel, browser applications. For example, try-irmin and try-eio.

Naturally, I tried to wrap OxCaml into a toplevel so people could play around with the new modes that did not align so the raw Obj.repr I would say that I do question the time-spent/value trade-off, but a mostly working toplevel with OxCaml is available at: https://patrick.sirref.org/oxcaml.

Catching Up!

2025-04-21T00:00:00-00:00

This is a bit of a catch-up post. A week or so in Belfast has thrown me a little off course with the weeklies.

Outreachy December 2024 and Beyond

In the last week or so, we have come to the end of the December 2024 round of Outreachy. For those who do not know about Outreachy, it facilitates and organises internships in open-source. Internships for those who are historically underrepresented and impacted by systematic bias in tech.

We, the OCaml community, held our biannual Demo Day where our current cohort of interns can show off what they have been building for three months. This round, we only had one intern, Abdulaziz, who worked on the OCaml API diffing tool.

The next round has just completed the contribution phase and our next cohort of interns is being selected.

This week, I added support for OCaml 5.4 to ppxlib. A rather non-trivial change to the codebase due to changes in the representation in Longident Longident labelled tuples, a light-weight record-like syntax similar to labelled arguments in functions.

let x = ~x:1, 10, ~y:1 val x : x:int * int * y:int let add ~x, ~y = x + y val add : x:int * y:int -> int

As of writing this post, we are waiting for the full OCaml 5.4 feature freeze and magic number bump. Note this is support for OCaml 5.4 5.4) not a bump to use the 5.4 parsetree in ppxlib.]]>

(ppxlib)[Ppxlib]The Future of

I have now been helping maintain Ppxlib for a while and I am beginning to wonder about the long-term vision for the project. Ppxlib is a central component to any modern OCaml library or tool, Sherlocode reckons there's 18.3k instances of

The biggest issue is still the tension between abstract types and pattern-matching. By exposing the Parsetree directly to user's, they can pattern-match against the AST and build their ppxes in a rather straight forward way. However, this comes with the risk that the type may change thus breaking their code.

I have mentioned before about plans to use "views" to help with this, but I do not see that happening any time soon. It feels to me, we should be able to allow a user to select their AST to work with. How this interacts with modules like Ast_builder is unclear to me, but it would mean user's can remain on older ASTs even when ppxlib bumps the internal, main AST.

Sherlorocq

Rocq coq I have been interested in theorem provers for a while, more from an engineering perspective and less from a theoretical one. My good friend, Derek Sorenson, taught me about Rocq when we encoded mergeable replicated datatypes. Later I had a go at encoding decentralised information flow control Fstar

During this process, I found it quite challenging to find Rocq code that I could learn from. Compared to using say OCaml's Stdlib

This lead me to port Arthur Wendling's excellent Sherlocode to do the same thing only for Rocq code. Luckily, it uses regular expression matching with a trigram index so it is not tied to any particular programming language.

Anyway, I've had some interest from the people at Inria to bring the server back to life. Their wish is my command!

Sherlorocq is back online!

Shelter in the Forest

2025-04-21T00:00:00-00:00

I spent much of this week working on Shelter and things related to it. Some of that time was also spent on Forester.

Forester

I added two important quality-of-life features to my frontend to Forester this week.

Bibtex Support

I had previously mentioned adding support to Forester for Markdown. This week I added support for Bibtex too. From any *.bib file in your forest, Forester will now dutifully recognise it as a Bibtex file and convert, as best it can, all the entries into Reference trees.

I'm becoming quite convinced of this model at the moment. I'm using Forester's Code.t as a target representation. In fact, to ease the process, I really shouldn't spend all my time on my website, I have reused my Yaml.t -> Code.t and Markdown.t -> Code.t functions in the Bibtex parser.

To see it in action, you could have a look at the Build Systems à la Carte paper which is generated completely from Bibtex.

Full Heading Support

The eagled-eyed viewer may have noticed that the table of contents for this page has more than one level. I finally caved and spent an evening rejigging my Cmarkit.Doc.t -> Tree code which was hacky and broken and is now less hacky and less broken.

In addition, headings support links and emphasis etc.

Lunch with Jon Sterling

I had a delightful lunch with Jon Sterling discussing the future of Forester, the nature of

Shelter Fixes

I spent a good chunk of my week fixing bugs in Shelter with the aim to perhaps setup a VM somewhere and let people kick the tyres of what we've got so far.

The first bug is pretty annoying. At the moment, our filsystem backend is ZFS and we make heavy use of snapshots and cloning in order to provide time-travelling capabilities. Unfortunately, ZFS will take a snapshot before lots of disk activity would be snapshotted in a half finished state and this would cause all sorts of problems. Thanks to Mark Elvers for the pointer to how OBuilder uses ZFS for the OCaml macOS builders which unmount datasets immediately therefore inducing a flush of sorts. Shelter now follows a similar model with all of the slow downs that create. Anil and I discussed some amalgamation of overlayfs, tmpfs and ZFS to alleviate some of this but for now that's a premature optimisation.

A small eDSL for Shelter

Whilst testing Shelter, I ended up wanting a way to programmatically invoke the different run commands. This is similar to say a Dockerfile, but maybe with a little more expressivity.

This lead me to revisit the Build systems à la Carte paper and rediscover selective applicative functors.

module D = Shl Identity let shelterfile = let open D in let base_image = from "alpine" in let is_node_lst img = String.equal "v22.15.0" stdout img in let cmds base = let node_version = run "node --version" base in Select.if' Select.map ~f:is_node_lst node_version run " " run " " base in with_session "node" cmds base_image

From this, we get a slightly more expressive way to describe images.

Geocaml TIFF Library

I was pleasantly surprised to receive a pull request from Michael adding support to ocaml-tiff for reading TIFF files compressed using LZW. I was also surprised to hear the TIFF LZW is a little different to others.

In trying to get this PR merged, I moved the initialisation of the Eio eventloop to outside each individual test case. This one change then completely broke the entire test suite. After a period of debugging and help from Thomas Leonard the root cause was OCaml's OUnit2 Unix.fork lead to the issues.

Read more about that issue on the Eio issue tracker.

Part II Students

As the new term begins, it signals that there are only just over two weeks for the final year undegrads at Cambridge to submit their dissertations.

read about their projects and I'm very impressed. I'm sure the next two weeks will be stressful, but I'm proud of what they have accomplished over the past academic year.

Shelter, Hazel and More!

2025-03-31T00:00:00-00:00

Last week I focused on Shelter -- our idea that shells should have the same ability as reproducible build tools like Nix or Docker. To this end I now have a fairly fleshed out prototype.

Shelter Prototype

Shelter is a spin-off from the work Michael and I started with Shark. It takes the same ideas but applies them directly to a shell-like interface.

We're still in the middle of working all of this, but you can read more about it at Shelter.

Forester Hacking

As you can probably tell, my website is still using Forester. I rebased my Markdown branch to include the new Atom syndication feature.

Alongside that I added support for arbitrary HTML injection into Forester via codeblocks in Markdown. This was actually very straightforward thanks to Markup and being able to re-parse Forester syntax in the middle of converting a Markdown document. The HTML for the shell in Shelter uses this feature.

If you are interested in taking this custom Forester for a spin, there's a branch on Github. In fact, nearly the only change beyond letting the core engine know about markdown files, is adding a new parser frontend.

Hazel

For one of my Part II students, I've been prototyping a transpiler from OCaml to Hazel. This has gone pretty well and now supports type annotations as well as straight-forward implementation translation.

Consider the following OCaml map function.

let rec map f = function | -> | x :: xs -> f x :: map f xs

The tool, hazel_of_ocaml can translate this to Hazel code, including making the polymorphism explicit.

forall b -> (a -> b) -> [a] -> [b] = typfun a -> typfun b -> fun f -> fun x1 -> case x1 | [] => [] | x :: xs => f(x) :: map(f)(xs) end in ?]]>

You can copy and paste that codeblock into the hazel playground map@@

Ppxlib

I recently wrote about the painful experience of miragrating ppxlib to the 5.2 OCaml AST. This week, Nathan Rebours and I merged a PR to add the 5.3 AST! Ppxlib has been playing catch-up with the compiler and we decided it was best to try to catch up quickly and deal with the ecosystem fallout all at once rather than incrementally. With this new AST merged, ppx authors can now use the new Pexp_effect parsetree node. I'll write a little more about this in a separate post soon.

Vpnkit, Void Processes, LSP Servers

2025-02-17T00:00:00-00:00

Previous weeklies used strong emphasis to distinguish sections. This comes from Forester's philosophy about atomicity of the content in your forest.

However, subtrees are supported! I quickly hacked together the ability to use subheadings to indicate subtrees. This is strictly less expressive than the of Forester's default syntax as we cannot close heading sections in Markdown.

This weekly uses subtrees.

Vpnkit

I spent some time this week trying to upgrade vpnkit to OCaml 5. I was originally working on a paper idea which might need benchmarks, but Anil and I decided we could simply point to the port I did and show how it has simplified much of the code.

Void Processes

void processes. A lot of the groundwork already exists in Jake Hillion's master's thesis.

This week I added a feature that I needed to help build the processes we need for the shell I'm building: mount points with modes!

pivot_root readonly or readwrite.

/say/hey program has been statically compiled using musl-gcc

let status = let void = empty |> mount ~mode:R ~src:hey_dir ~tgt:"say" |> exec "/say/hey" in let t = Void.spawn ~sw void in Promise.await Void.exit_status t

There really is nothing else in there. Without specifying a root mount, the void process is started with an empty tmpfs root. Next on the list is networking!

LSP Servers

I got a little side-tracked building a library for writing LSP servers in OCaml: mlsp. This may seem a little unrelated, but it isn't. The LSP has become the de facto standard for communicating between an editor and a programming language environment. If you have used VSCode to write a program in Python, chances are you are using the official extension which gives you linting, formatting, code navigation etc. All of these features are communicating using the LSP.

It seems Code Mirror

Open-Source & Community

Anil and I had a great conversation this week about building community especially as it pertains to open-source and OCaml.

I've been going back over Governing the Commons, but have already discovered The Real World of Technology!

More thoughts on all of this soon... maybe

Teaching, teaching, teaching...

2025-02-10T00:00:00-00:00

Discrete Maths. I supervise three third year students for their final year project and another one I co-supervise with Michael Dales. However, I do end up spending at least two full days a week on teaching. Something I really enjoy and take seriously. The time it takes is also quite unpredictable; last week for instance all the third year students slides particularly challenging to understand from their lectures and asked if I could help explain what was going on, so I produced some materials for that.

OCaml

Within the OCaml universe I spent a good bit of time trying to maintain a couple of different packages:

Mirage Crypto: this library provides cryptographic primitives for OCaml programs. Unfortunately, the maintainers removed direct support for Eio replacing it with a "Unix" alternative. This is not a fair swap as now Eio programs must make a dependency to Unix! Speaking to Anil, the best approach might be to make Eio programs handle this directly. I think this highlighted how fragmented open-source maintenance can be as no user of Eio seems to have bumped into this yet and the upstream maintainers did not communicate a rather large breaking change.

Ppxlib: In addition to the 5.2 AST bump 5.3 AST bump right behind it. I plan to write up a more detailed post about the challenges of maintaining this part of ppxlib.

Paris

I spent some time in Paris at the end of the week. I enjoyed visiting the Musée d'Orsay and in particular their collection of impressionist paintings. Here's my favourite from that visit by Camille Pissarro.

Woman in an Orchard (Spring Sunshine in the Meadow at Eragny)

AT Protocol and an IR for Wikis

2025-01-27T00:00:00-00:00

AT Protocol

This week I've been diving into the AT Protocol.

The Authenticated Transfer Protocol, aka atproto, is a decentralized protocol for large-scale social web applications.

The protocol could be a candidate for the glue that holds together a distributed, computational wiki network. The protocol, it seems, is very similar to IPFS. Thankfully, a few years ago, I was working on building out a suite of OCaml libraries for working with IPFS. For example, ocaml-cid, self-describing content-addressed identifiers.

# let s = "zb2rhe5P4gXftAwvA4eXQ5HJwsER2owDyS9sKaQRRVQPn93bA";; val s : string = "zb2rhe5P4gXftAwvA4eXQ5HJwsER2owDyS9sKaQRRVQPn93bA" # let cid = Cid.of_string s |> Result.get_ok;; val cid : Cid.t = <abstr> # Cid.pp_human Format.std_formatter cid;; cidv1 - base58btc - raw - ident sha2-256 length 32 digest 6e 6f f7 95 0a 36 18 7a 80 16 13 42 6e 85 8d ce 68 6c d7 d7 e3 c0 fc 42 ee 03 30 07 2d 24 5c 95

To this end I have built out some more OCaml libraries for working with atproto, including:

atproto-data: the atproto data model, similar to JSON-LD.

ocaml-did: an OCaml library for working with decentralized identifiers.

atproto-lexicon: atproto's schema format, I've been building a quick tool for doing an OCaml translation from these schemas.

I managed to get a post published from the OCaml library after fixing it up and porting it to Eio.

An IR for Wikis

I started working on a proof-of-concept intermediate representation for Wikis -- I imagine it a bit like malfunction but for computational wikis i.e. a target for Wiki building tools that allows different front-ends and servers to communicate in a common IR for exposing key functionalities of a wiki:

Links: External links, cross-wiki backlinks

Versioned, temporal feeds

Etc.

Other PhD Work

I met with most of my Part II students this week, and I'm excited about their work. Progress reports are due this week and next they have a presentation to give.

In discrete maths this week we did induction. Next up is a big section on sets, functions, bijections etc.

Misc.

I was happy to find the Information Controls Fellowship Program.

Forester, ICFP, Wikis

2025-01-20T00:00:00-00:00

This week was my first full week back from the break and I found it challenging trying to get back into what I had been working on previously.

ICFP Papers

In conversation with Anil, we looked at options for submitting a paper to ICFP. I wrote up some notes on some options we discussed.

Forester

I spent some time this week converting this website to using Forester. I'm not a huge fan of the syntax, especially as a lot of my site's content was already in markdown. So I wrote a markdown frontend to forester which is available on Github.

The markdown frontend integrates very nicely and only a few changes were needed in the core logic of Forester itself. Additionally, for any features not directly supported in markdown there is an escape hatch using code blocks such as:

Personally, I'm still getting to grips with the bottom-up approach to building this site, atomically creating notes and reference cards that then are linked in many places.

I'm excited to see how I can integrate some of the Forester concepts into "Shark".

OCaml

In the OCaml world, I spent time on ppxlib, reviewing PRs to update the lower bounds of the library, fixing effect syntax problems and bumping internal AST to 5.2. I've also spent some time looking into de-objecting Eio, making the OCaml types more friendly to new users. I need to revive my port of Vpnkit to Eio for the thoughts on ICFP 2025 too.

I also want to modernise and make more public my OCaml code for creating little shells in OCaml too -- I think the ideas here really have legs and would like to find a conference to submit them too.

I also met with the single, OCaml Outreachy intern working on ocaml-api-watch.

Misc.

I met with most of my Part II students this week which was nice to catch up and see how their projects are going. I also started marking work for my first year students who are at the induction part of their Discrete Maths course.