Tsonnet #2 - Cram tests to the rescue
Setting up automated testing for the Tsonnet compiler
In the previous post, I added the parsing of JSON literals to Tsonnet:
There's something important still missing though and it is tests. Better not to proceed without guaranteeing the language specification is being validated by automated tests.
Let's do it!
Adding the testing libraries
We'll add alcotest for writing automated tests -- it seems like the favorite testing library for OCaml programmers nowadays instead of ounit. Let's also add bisect_ppx for coverage report:
In dune, we can specify that a dependency belongs to a specific environment — in this case, the test environment. This way we don't carry testing dependencies to the release binaries.
Running dune runtest is convenient, but the coverage command isn't. Let's add a Makefile to free our minds of the details:
I appreciate that simply running make targets the make test. I always like to have the test command as the default for my projects. The make coverage command is now convenient to run locally and we can leverage it when setting up a CI job later on.
Now, let's configure dune.
A little bit of configuration
We are going to specify that we can write cram tests in the bin directory and the tests will have two dependencies:
1. The Tsonnet binary that dune will compile for us
2. The samples directory, where we store the Jsonnet file samples, to be used as input in the tests
We are also adding a similar configuration to the test directory, where we specify that cram tests will live in the test/cram folder and the binary and sample files dependencies.
The main entry-level test folder will have the sample files as a dependency and the alcotest library.
I'll go into more detail about cram tests in the next section.
For the coverage:
We need to instrument the bisect_ppx coverage tool to set the lib directory as its target, otherwise, it is going to completely ignore this directory, which is where all our compiler logic lives.
Since the lexer and parser are generated, there's no point in checking coverage for them, so we basically annotate them to be excluded.
Now we are ready to finally write some tests.
Testing with cram tests
Do you know what are cram tests?
TLDR, cram tests are integration tests that describe a shell session. They are amazing to test CLI programs!
Do you know what else is cool? Dune has built-in support for writing cram tests. The reason why no new library has been added to handle that.
We can write runnable documentation with that. Remember that we configured dune to have cram tests in the bin folder? Here are cram tests to cover some usage cases of Tsonnet:
Your cram test files need to have the .t extension.
You can mix free text with properly indented shell commands. The line right below the command is the expected output.
How cool is that?!
In the test/cram directory we can test each sample source file, along with its expected output:
To finalize, let's run the tests:
There are no errors when all the tests have passed.
Let's introduce a small error, changing the output value of one or two of the tests, and see how it is presented:
Here's our intentionally introduced error changes:
The output of the tests:
It is super easy to spot the expected value and the actual result we got. Isn't it nice?!
Concluding
Cram tests have become one of my favorite tools. I can't live without them if I'm writing a CLI program seriously.
This is not a new idea. The first time I saw something similar was in Elixir via doctests. I read somewhere that this cram Python package is where it originated, but I can't say for sure. The last time I tried Rust was many years ago, but as far as I know, Rust has documentation tests.
I don't know who copied who, but I hope this trend continues. This is a really cool feature that I hope every single programming language implements, either fully as cram tests or partially as documentation tests.
Disclaimer: alcotest is basically sitting idle for now, as there are no unit tests or integration tests besides the cram tests. Eventually, there will be. I just anticipated adding the dependency right away.
What do you think about cram tests?