Merry is a POSIX(ish) in OCaml. It uses Eio alongside Morbig (a static parser for POSIX shell).

docker run -it --rm patrickferris/msh

opam pin git+https://tangled.org/patrick.sirref.org/merry

I picked up from the excellent Outreachy work of Tambe Salome in getting write-support in ocaml-tiff. We are getting closer to the kind of API I envisaged in this latest round of refinement and review.

let ( / ) = Eio.Path.( / )

let checkerboard ~size =
  let v = Nx.zeros Nx.uint8 [| size; size |] in
  for row = 0 to size - 1 do
    for col = 0 to size - 1 do
      if (row + col) mod 2 = 0 then Nx.set_item [ row; col ] 254 v
    done
  done;
  Nx.to_bigarray v

let () =
  Eio_posix.run @@ fun env ->
  Tiff_eio.with_open_out (env#cwd / "example.tiff") @@ fun w ->
  let tif = Tiff.make (checkerboard ~size:256) in
  Tiff.to_file tif w

I have also been extremely pleased to see further external collaborators appearing:

• Nicolas helping out with metadata maintainence.

• Virgile adding support for reading multi-image TIFF files.

Nathan and I released ppxlib.0.38.0 last week. Its main feature is full migration support for the upcoming OCaml 5.5 compiler (currently in its alpha3 release). This means supporting the handful of new features landing in OCaml 5.5: modular explicits, external type declarations and arbitrary "local" structure items.

As Nathan and I continue to find a plausible maintenance story for ppxlib, Nathan has opened an issue on the OCaml compiler to discuss the idea of adding additional extension points to the language.

This comes from the desire to be able to store encoded versions of new OCaml features inside older abstract syntax trees.

There are also some nice bug fixes in there too:

• A potential OOM loop has now been removed.

• Locations have been restored to long identifiers!

It turns out writing a POSIX(ish) shell is hard; at least, there is a vast number of slightly obscure features that one needs to support. I think this is surprising because most people writing shell scripts write simple shell scripts; scripts that use a much smaller subset of features. In the same breath, those same developers are likely to use something like apt-get install bash which runs a plethora of more advanced (and non-POSIX) shell scripts!

err () {
  printf "err $1" >&4
}

exec 4>&1 

The presentations from the demonstrations for this round of Outreachy are now online!

So far, Merry has managed to support many features of the POSIX shell specification without needing to do a fork(2) without an exec. Or, to put it another way, all the forking that Merry does, will only be followed by running C code. This is actually a feature of Eio's Process API. It is required to make process execution safe in the context of multiple domains in OCaml (mostly due to ensuring consistency within the garbage collector). However, it is perfectly okay to fork and run OCaml code provided there is only a single domain.

In a shell, many features of the shell language require that commands (or built-ins, function applications etc.) execute within a subshell. The specification does not go into detail about how shells should implement this, but many choose to fork in order to preserve some state in the parent (for example, the file descriptor table). Some shells try to minimise the number of forks as an optimisation. The korn shell (ksh) is one such shell.

Using the notation $(command) will cause command to execute in a subshell of the current ksh. In many instances, ksh will not actually fork/exec a subshell when command is a built-in or a shell function.

By trying to adhere to Multicore OCaml's "thou shalt not fork" commandment, Merry is much more similar to ksh in this regard. However, whenever there is an interaction with a shell built-in and semantics that need a child process, things get tricky very quickly. Consider the following:

while echo "hello"; do
  :
done | head -n 3

We have a mix of shell built-ins, normal commands, a compound command (the while loop) and a pipeline. Each individual command in the pipeline requires you to run them in a subshell and set them up before executing them. Without a fork for that first command, you may end up looping forever which is not the intended behaviour here. I had a good conversation with Michael about the implications of this for reproducibility.

For now, I have started implementing a fork for some of the shell features whilst still trying to maintain the functional core of the implementation.

Nathan and I released an OCaml 5.5 compatible version of ppxlib.

A while ago I built some "modern" bindings to PROJ4 in OCaml. After reading Jon Ludlam's weeknotes (and speaking with him and Anil, I thought it might be nice to add a Javascript backend to those bindings). This was relatively straight-forward using Dune's virtual libraries and is available on Github.

With virtual libraries, your own data analysis could (if you wished) depend solely on the proj library and later choose to either link it with proj.c or proj.js depending on where the analysis is being deployed.

I briefly looked at WASM support, but quickly realised there was not much appetite for it and trying to compile around the C FFI was going to be hard.

Since I helped relaunch the ocaml.org website a few years ago, I have maintained hilite, a tool for doing build-time syntax highligting.

It is being used on the ocaml.org website, in yocaml_markdown and also in Michael's webplats (when it works...).

And it was Michael who led me to finally fixing and adding syntax highlighting support for Python and Go. Once I was on this roll, I finally took a look at fixing the long-standing bug of trying to highlight ocaml-mdx code, like the following:

# 50.123;;
# 1 "file.ml";;

In fact, this code now highlights fine, but was the source of the bug. The syntax highlighting grammer confuses the ocaml-mdx # as a toplevel directive and trouble ensues.