Tsonnet #18 - Property-based testing saves the day (and my sanity)

Sometimes the best way to test your code is to throw completely random stuff at it and see what sticks. Turns out, our Env module is stickier than we thought – 1000 random tests and counting!

Welcome to the Tsonnet series!

If you're just joining, you can check out how it all started in the first post of the series.

In the previous post, we added support for indexing strings and extracted the duplication into an indexable module:

Tsonnet #17 - Indexing a String: From copy-paste to unification

Since I added the Env module, I feel uneasy about the fact that it's only being indirectly tested, so I've decided to cover it thoroughly.

Photo by Sufyan on Unsplash

Adding property-based testing for the environment

The environment is supposed to store any type of expr and it is difficult to manually test all possible cases ourselves. I believe property-based testing is perfect for this scenario.

If you don't know what property-based tests are, or want a refresher, I've got you covered with this short article:

What if your tests could think of edge cases for you?

Ok, ready?

First things first. Add the dependencies:

The qcheck library is a QuickCheck (written in Haskell) inspired property-based testing library for OCaml. It pairs well with other testing libraries, like alcotest which is my preference for unit testing. The ppx_deriving_qcheck is just a submodule from qcheck -- I'll explain why and what it does in a bit.

Now let's configure dune to run the tests using these libraries:

We're going to target the test_env file, and drop test_tsonnet since it's an empty file so far.

And here's the test/test_env.ml file:

Let's break it down:

1. The test_undefined_variable case is an example-based test to check when variable is not in the environment.

2. The property-based magic happens in test_lookup. It will start with an empty environment, generate 1000 random pairs of string and arbitrary_expr, update the environment, and look it up using Env.find_var. It should successfully retrieve the expr from the environment. Pretty cool, isn't it?

Because we want to test the environment as thoroughly as possible, we generate random values of expr, covering all its variants. Since expr is a recursive type, we need to add a limit to avoid generating infinitely nested values with gen_expr_sized 5 -- I'm arbitrarily setting it to 5, as it strikes a good balance between performance and completeness.

The expr type had to be manually generated to exert control, due to its recursive nature. However, the other types are simple enough that we can derive them automatically -- enters ppx_deriving_qcheck:

The types bin_op, unary_op, and number are derived using ppx_deriving_qcheck. This allows the library to automatically create generators for these types.

Now the only thing missing is configuring the source code to be pre-processed by ppx_deriving_qcheck during the compilation phase:

We run and it outputs the results:

It doesn't show in the output, but qcheck performed the lookup 1000 times, with a thousand different values.

Now I'm at ease, as qcheck is battle-testing the find_var function with inputs that I wouldn't be able to generate manually. :)

Conclusion

The combination of qcheck and ppx_deriving_qcheck gives us confidence that our Env module can handle the full spectrum of expressions we throw at it – from simple values to complex nested structures. This investment in testing infrastructure will pay dividends as Tsonnet grows more sophisticated, ensuring that our foundation remains solid as we build more advanced features on top of it.

---