Software. Meio [meio]

https://github.com/ocaml-multicore/meio

Meio monitors Eio programs.

Backlinks

OCaml Roundup: June 2026 [ocaml-roundup-june-2026]

July 12, 2026
Patrick Ferris

Summary

July 12, 2026
Patrick Ferris

June saw a lot of progress on Merry. At the start of the month, I worked on meio, a tracing tool for Eio programs. This was to help debug a particular shell script fragment that was causing Merry to hang. It uncovered a few UX opportunities to make meio more useful, and this might be fruitful direction to take some of my work.

Merry is now able to run (and pass!) the entire unixfun benchmarks in Koala. Whilst benchmarking Merry, it became apparent that it is paramount to provide a test built-in (rather than spawning [).

Part of this work also looked into building an in-memory filesystem for Eio. It is still a work-in-progress, but does pass many of the filesystem tests in Eio. To be a drop-in replacement in Merry requires some more work to handle an abstraction for Unix file descriptors (e.g. a fake stdout). It would be nice if Eio provided a Unix filesystem interface similar to the OS-agnostic interfaces of Eio itself (not to be confused with the Unix utilities in Eio_unix).

I have been very grateful to Thomas's diligent and steadfast progress on merging and fixing bugs in Eio over the last few months. You can keep up to date with his work in his notes.

Weekly logs

July 12, 2026
Patrick Ferris

The full details are in the weeklies transcluded below:

Using meio to debug Merry [weekly-2026-w23]

June 4, 2026
Patrick Ferris

During a large rewrite of Merry, I introduced a regression causing certain shell programs to hang indefinitely. I thought it might be a good opportunity to try out Meio.

First, let's look at a trace of Meio from some real-world programs. For example: meio -- msh -c "sleep 100".

One problem that comes up immediately from this example is that the tree hierarchy of tasks and cancellation contexts makes it tricky to reason about the logging. Here, we can see the pipeline's cancellation context clearly labelled pipeline-id1. This is an Eio.Switch.t for the pipeline sleep 100. The actual execution log for sleep 100 bubbles back up to the main fiber (that has no name, so it is called <task>). The daemon fiber, for ensuring the spawned process is reaped, is attached to the switch and is displayed below it.

Meio has a long way to go. Most programs are too fast to be monitored in this way. Using something like tokio-console's model might be a better idea here focusing less on the tree strucuture of the program, and more on the performance and monitoring of individual tasks. Without this, it turns out Meio was not all that useful for debugging this specific issue (which ended up being some bad pipe management on my part).

Three weeks in the Shell [weekly-2026-w25]

June 16, 2026
Patrick Ferris

The last three weeks have been focused work on building Merry. In practice, I wanted to run the following two commands through Merry's shell: msh:

apt-get install debootstrap -y
debootstrap stable /stable-chroot http://deb.debian.org/debian/

Seemingly innocuous. But both commands run thousands of lines of shell script. Whether that's Debian package maintainer scripts or state-machines for processing package release files.

What follows is a handful of the bugs and the shell scripts that caused them.

Field Splitting

June 16, 2026
Patrick Ferris

Edge-cases abound in the POSIX shell specification; and field splitting is one of the most serious perpetrators. Field splitting happens after word expansion. It is a known footgun amongst shell script authors.

$ count () { echo "Count: $#"; }; FOO="hello world"; count $FOO
Count: 2

Quoting variables that need expanding will prevent field splitting from taking place, but it is an easy thing to forget.

Field splitting uses the IFS (internal field separator) environment variable to determine how to do the splitting, and it comes with some extra rules beyond "split on these characters".

$ IFS=" "; FOO=" hello    world"; echo $FOO
hello world
$ IFS=" "; FOO=" hello    world"; echo "$FOO"
 hello    world
$ IFS=":"; FOO="a:b:c:"; echo $FOO
a b c
$ IFS=":"; FOO="a:b::c:d"; echo $FOO
a b  c d
$ IFS=":"; FOO=":a:b:c"; echo $FOO
 a b c
$ IFS=; FOO="unset   IFS"; echo $FOO
unset   IFS

Variable assignment

June 16, 2026
Patrick Ferris

How a shell stores variables also impacts field splitting. Naively, Merry stored variables as strings, but really they should be stored as their expanded form.

Consider this example of saving a function output in a variable called base.

iter () {
 echo "a"
 echo "b"
 echo "c"
}

base=$(iter)

for n in $base; do
  echo "Got $n"
done

for n in "$base"; do
  echo "Got $n"
done

Later, when we come to iterate across the fields of $base we wouldn't be able to reconstruct the intial fields that $(iter) produced if we simply saved the result as a string. Too much information would have been lost.

Redirections

June 16, 2026
Patrick Ferris

Redirections control what file descriptors point to. Each redirection has a lifetime associated with it. For example: cat < README.md really says "open README.md and execute the cat program mapping the process's FD 0 to the FD of README.md". When cat returns that redirection is no longer valid.

Other redirections are not so simple. Consider:

$ echo "!dlrow olleh" > input.txt
$ (read line; echo $line) < input.txt | rev
hello world!

There are lots of redirections here too. Firstly the standard input for everything inside the braced command (read line2; echo $line) should be the open file descriptor for input.txt. The standard output of the braced command should be the writing end of a pipe whose reading end is passed to rev as that process's standard input. The final standard output for rev should be unchanged from what it was originally (the shell's standard output).

Getting redirections right is hard, consider this snippet from the debootstrap scripts (all in the context of stdout and stderr pointing to a debootstrap.log file and file descriptor 4 pointing at the true stdout).

for m in $MIRRORS; do
    pkgdest="$TARGET/$($DLDEST pkg "$s" "$c" "$a" "$m" "$path")"
    if [! -e "$pkgdest" ]; then continue; fi
    pkgs_to_get="$(download_debs "$m" "$pkgdest" $pkgs_to_get 5>&1 1>&6)"
    if [ -z"$pkgs_to_get" ]; then break; fi
done 6>&1

More niche built-ins

June 16, 2026
Patrick Ferris

As the dust begins to settle on the nuts and bolts of implementing a POSIX-ish shell, new challenges are allowed to emerge from deep within scripts.

One such challenge is the seemingly endless (I know, they're finite) shell built-ins! Whilst trying to run debootstrap using its fakechroot variant, I bumped into getopts... a whole command-line parsing shell built-in.

An In-memory Filesystem for Eio [eio_mem]

June 16, 2026
Patrick Ferris

Towards the end of last week and the beginning of this week, I threw together a prototype for an in-memory filesystem for Eio: Eio_mem. Concretely, this provides a means for constructing an Eio.Fs.dir_ty Eio.Path.t value that is completely in-memory.

Thanks to Eio's capability-first approach, any existing code implemented against the _ Eio.Path.t API should just work with the in-memory filesystem.

There's a convenient Eio_mem.Fs.load function that can take a path from one filesystem (most likely a normal one from Eio_linux or Eio_posix) and replicate it into the in-memory filesystem. For example, you might have a static web-server pointing to a directory like "/var/www". The following might be the only change you need to make it completely in-memory (example from ocaml-cohttp):

 let () =   let port = ref 8080 in
   Arg.parse
     [ ("-p", Arg.Set_int port, " Listening port number(8080 by default)") ]     ignore "An HTTP/1.1 server";
   Eio_main.run @@ fun env ->
+  Eio_mem.run @@ fun menv ->
   Eio.Switch.run @@ fun sw ->
-  (* Restrict to current directory: *)
-  let htdocs = Eio.Stdenv.cwd env in
+  (* Restrict to in-memory current directory: *)
+  Eio_mem.Fs.load ~filter (Eio.Stdenv.cwd env);
+  let htdocs = Eio.Stdenv.cwd menv in
   let socket =
     Eio.Net.listen env#net ~sw ~backlog:128 ~reuse_addr:true
       (`Tcp (Eio.Net.Ipaddr.V4.loopback, !port))

The keen-eyed reader might have noticed the filter parameter. This can be used to ignore certain paths (here I ignored .git and _build which were adding substantial time to loading the filesystem).

Using meio to debug Merry [weekly-2026-w23]

June 4, 2026
Patrick Ferris

During a large rewrite of Merry, I introduced a regression causing certain shell programs to hang indefinitely. I thought it might be a good opportunity to try out Meio.

First, let's look at a trace of Meio from some real-world programs. For example: meio -- msh -c "sleep 100".

Catching up in May [weekly-2026-w22]

June 1, 2026
Patrick Ferris

I never quite found the time in May to write weeknotes. I think that's, in some ways, quite a good thing. My philosophy on weeknotes is not that they are a public punchcard of my progress. They are a tool for organising thoughts and keeping my colleagues abreast of what I am up to. Only the latter suggests any necessity for timeliness and given I frequently message, call and zulip them, if a few weeks fall between the cracks then that's okay!

All that being said, I have tried to provide some thoughts and things I have been up to over the course of May and not just week 22 of the year.

Tools for Shells

June 1, 2026
Patrick Ferris

Some tools and libraries related to Merry.

Meio

June 1, 2026
Patrick Ferris

The shell work continues, and as the complexity internally increases I have found myself wishing for some more tools for debugging concurrent Eio programs. Given enough pipes and compound commands, it is fairly easy to introduce a fair deal of concurrency to your shell programs. To this end I hope I have improved meio somewhat in this fork.

In particular, there were several things I wanted to improve.

Simple UI improvements -- the set of tools for building TUI UIs from nottui are, on closer inspection, rather good and simple. For example, turning a particular UI component into a scrollable pane is as simple as wrapping it in a call to this function.
Logging more runtime events -- Eio programs emit information via OCaml's runtime events. This is how meio knows which fiber is currently active for example. It also emits information like "the current fiber created a new promise" and "this fiber is blocked waiting for a promise". I have not figured out an intuitive way for representing this in meio's tree-view, but I did turn these into "system logs" that appear in the per-fiber logs which is already an improvement. Thanks to Thomas for helping understand the runtime event sequence for logging this information.
Timestamping where possible -- all runtime events come with a timestamp, these were mostly discarded by meio.

Some aspects of meio made it clear that labelling Eio.Switches and Eio.Promises might be a good practice. After reading Michael's Eio forays, I can't help but wonder if a more opinionated Eio for Systems Programmers guide might be useful. I think the Eio README does a wonderful job of introducing most of the low-level concepts, but not quite how they are used by the working systems programmer.

Glob

June 1, 2026
Patrick Ferris

A little dismayed at the lack of progress I was making with Merry, I switched to a specific problem I knew I wanted to fix -- removing the dependency on glob(3), replacing it with a pure OCaml implementation.

let get_txts fs =
  Glob_eio.glob fs Glob.(of_component [ dir_sep; string "home"; dir_sep; star; string ".txt" ])

Anil provided some issues with fixes too which was very appreciated.

Beyond factoring out fork+exec, Merry should now be portable to the browser. Of course there are lots of other refactorings needed to (e.g. a fake-filesystem) -- but I'm interested in how I can use Eio's filesystem capabilities to do this pretty seemlessly. Having an emeddable, pure OCaml shell may also unlock some cool debugging capabilities for unikernels too!

Geocaml

June 1, 2026
Patrick Ferris

Michael has been valiantly trying some of the Geocaml libraries in his work on rendering LiDAR point clouds. Where possible, I have been reviewing Michael's PRs and helping implement features he might find useful. I also took his Laserver for a spin, and suggested a few changes.

Rtree

June 1, 2026
Patrick Ferris

After a comical series of attempts, I (or rather Anil) released rtree.0.2.0 which contained:

A bug fix where not all the tree values were being returned.
A new bounds function for getting an Rtree's maximum bound.
A set of simple removal functions.

PROJ

June 1, 2026
Patrick Ferris

Michael added some new functions to ocaml-proj. Previously, I had worked on making PROJ more portable by using virtual libraries. This was a little restrictive, as it forced users to always implement a JS version of any functionality that exists in the native version of PROJ.

To make it easier to contribute (and allow the C bindings to contain more specific bindings) I carved out the two virtual library implementations into standalone libraries, and reimplemented the virtual library support using these libraries. Now projects can depend directly on proj_c if they wish to have more features in exchange for being less portable.

Ppxlib

June 1, 2026
Patrick Ferris

Thankfully, work on ppxlib has been relatively quiet. Nathan also opened a PR for initial OCaml 5.6 support. Keeping on top of 5.6 features is probably the right way to go about this (rather than one giant PR during the 5.6 release cycle).

I'm particularly happy to have some better migration error handling. The previous errors, though uncommon, were confusing ppx users and hopefully this will help ppx authors and ppxlib maintainers triaging important encoding bugs or feature wishes.

Self-hosting

June 1, 2026
Patrick Ferris

I now have a fairly workable setup for self-hosting source code with git.sirref.org and tests.sirref.org. One piece that is missing, is a publishing workflow. I spent a little time building a YOCaml static-site generator for packaging OCaml libraries. Unfortunately, this was full of rabbit holes and busy work which was frustrating.

Tar-eio

June 1, 2026
Patrick Ferris

One such rabbit hole was tar-eio. It was in a bit of a sorry state, so I took the opportunity to improve the API for new users. Tarring a directory is now as simple as:

let foo = Eio.Path.(cwd / "foo") in
let foo_tar = Eio.Path.(cwd / "foo.tar") in
Eio.Path.with_open_out ~create:(`Or_truncate 0o755) foo_tar @@ fun foo_tar ->
Eio.Switch.run @@ fun sw ->
let t = Tar_eio.create ~sw foo in
Tar_eio.run t (Tar_eio.flow_of_file tar)

There was also a series of security issues that the tar library had that were fixed too, you can read about those in Thomas's weeknotes.