OCaml 5's effect handlers are untyped: the compiler does not check that a performed effect is handled, so a missing handler compiles cleanly and fails at run time with Effect.Unhandled. We have been building a static checker for that, and we would like some criticism of it, particularly of the parts we think are weakest.
What it does. It reads the compiler's own .cmt files, infers a per-function effect set, propagates by call-graph fixpoint, subtracts effects discharged at handler sites, and reports every perform that can reach a program entry point unhandled. Findings come with a blame path from entry to perform, and with a generated witness program: we synthesise it, compile it, run it, and only report the finding as confirmed if the effect actually arrives at an outer boundary.
How it is tested. Differential testing against the runtime as oracle: a generator emits programs, and we compare the checker's prediction against what the compiled program actually does when executed. Over 1000 generated programs, 0 false negatives. Branch-free programs: 600, no false positives. Branching programs: 400, 5.3% false positives, from joining both arms of a conditional. The witnesses deliberately do not parse the crash message. Handy, since 5.4 stopped printing the effect payload for two of our test programs and nothing broke.
The negative result, which is the reason for this post. We wanted to show the checker catches crashes that really happened, so we built a corpus of commits that fixed documented Effect.Unhandled problems and analysed each parent commit. It caught 0 of 6. The cause turned out to be that the modules the bugs lived in never compiled in our environment, so the analyser was never shown the code. The batch runner now reports "out of scope" separately from "missed" for exactly this reason.
Then we ran it across every opam package depending on eio, eio_main, picos, domainslib, riot, moonpool or miou, giving 85 repositories after deduplicating by dev-repo. 38 of them built far enough to analyse, 3,881 modules in total, and the tool reported 31 escapes. Every single one was our own false positive. Four causes, each found by reading the code we had accused:
- We modelled "calls anything under
Eio." as "requires an Eio runtime". Wrong for constructors and vtable builders: Eio.Flow.Pi.source, Eio.Stream.create, Eio.Condition.create, Eio.Buf_read.parse_string_exn.
- We applied that client-usage model to eio's own source, which is a category error. eio does not need an Eio runtime to define itself.
- We did not see through application operators.
let@ env = Eio_main.run in body is ( let@ ) Eio_main.run (fun env -> body), and with the operator opaque the head of that application is not Eio_main.run, so the scheduler boundary vanished and an entire program looked like it ran with no runtime. The fix is to read %apply and %revapply out of the value description, which would also cover @@, |>, and operators a project defines for itself.
We only recognised an Effect.Unhandled guard when it named the exception. picos writes it as a catch-all and says why:
| exception _exn ->
(* This should only happen when not running under a scheduler.
However, we don't match on a specific exception, because it
depends on the OCaml version. *)
We had recorded the same observation independently, and then watched 5.4 stop printing the effect payload, which is the same problem from the other side.
Causes 1 and 2 are fixed and regression-tested in both directions. Causes 3 and 4 are diagnosed but not yet fixed: the attempt broke the test suite and we reverted it rather than ship a red one, so they are written up in docs/LIMITATIONS.md with the forester and picos source that produced them. Twenty-five escapes fixed, six remaining and explained.
Every escape this tool has ever reported on third-party code has turned out to be our own false positive. We have not found a real bug in anyone's code. We would rather say that plainly than imply otherwise.
If there is a transferable lesson it is probably that one: a checker pointed at real code for the first time is mostly measuring itself, and the only way to tell the difference is to read the source you just accused.
What we would like criticism of.
Is the library-contract output useful to you? On picos it produces:
module Picos
Picos.Trigger.await may perform {Picos.Trigger.Await}
Picos.Fiber.current may perform {Picos.Fiber.Current}
Picos.Fiber.spawn may perform {Picos.Fiber.Spawn}
Picos.Fiber.yield may perform {Picos.Fiber.Yield}
Picos.Fiber.Maybe.to_fiber_or_current may perform {Picos.Fiber.Current}
Picos.Fiber.Maybe.or_current may perform {Picos.Fiber.Current}
Picos.Fiber.Maybe.current_if may perform {Picos.Fiber.Current}
Picos.Fiber.Maybe.current_and_check_if may perform {Picos.Fiber.Current, ...unknown}
module Picos_std_event__Event
Picos_std_event__Event.sync may perform {Picos.Trigger.Await, ...unknown}
Picos_std_event__Event.select may perform {Picos.Trigger.Await, ...unknown}
module Picos_std_structured__Run
Picos_std_structured__Run.spawn may perform {Picos.Fiber.Spawn, ...unknown}
...unknown in a set means the contract is incomplete at that function because it calls something we have no .cmt for; we would rather show that than round it down to a clean-looking set.
Is a machine-checked statement of "what this library asks its callers to handle" worth having in a README or in CI?
- What shapes do we miss? We know about effects performed inside
effc branch bodies, dynamically built handler records, and closures that reach a call site through a data structure (no 0-CFA yet). We would like to hear about the ones we do not know about.
- The scheduler model, meaning which entry point installs handlers for which family of effects, is data rather than analysis, and lives in
models/schedulers.conf. The eio entries are cited to specific lines. The riot, moonpool and miou entries are guesses and labelled as such. Corrections very welcome.
- Has anyone actually hit an
Effect.Unhandled crash in production? We are short of real examples and would take any pointer, including "we hit it and the cause was something your design cannot see".
Requires OCaml 5.3 or later, since Texp_match only grows its effect-case list in 5.3. CI runs 5.3 and 5.4.
Repository: https://github.com/manishpaulish/unhandled
Happy to be told the approach is wrong.