Tsonnet #20 - Adding the sweet stuff

Implementing the syntactic sugar because nobody wants to escape quotes in their cocktail recipes

Welcome to the Tsonnet series!

If you're not following the series so far, you can check out how it all started in the first post of the series.

In the previous post, I finally added type checking to Tsonnet:

Tsonnet #19 - Type checking and semantic analysis

Now that we have the core of the type checker, I'm ready to support more complex idioms from Jsonnet in Tsonnet.

Let's start with the syntax.jsonnet example from the Jsonnet tutorial:

/* A C-style comment. */
# A Python-style comment.
{
  cocktails: {
    // Ingredient quantities are in fl oz.
    'Tom Collins': {
      ingredients: [
        { kind: "Farmer's Gin", qty: 1.5 },
        { kind: 'Lemon', qty: 1 },
        { kind: 'Simple Syrup', qty: 0.5 },
        { kind: 'Soda', qty: 2 },
        { kind: 'Angostura', qty: 'dash' },
      ],
      garnish: 'Maraschino Cherry',
      served: 'Tall',
      description: |||
        The Tom Collins is essentially gin and
        lemonade.  The bitters add complexity.
      |||,
    },
    Manhattan: {
      ingredients: [
        { kind: 'Rye', qty: 2.5 },
        { kind: 'Sweet Red Vermouth', qty: 1 },
        { kind: 'Angostura', qty: 'dash' },
      ],
      garnish: 'Maraschino Cherry',
      served: 'Straight Up',
      description: @'A clear \ red drink.',
    },
  },
}

There are still a few things missing to compile this file:

1. Python-style comments

2. Single-quoted strings -- we only have double-quoted strings so far

3. Verbatim strings

Photo by Myriam Zilles on Unsplash

Python-style comment

I find it weird that Jsonnet supports Python-style comments, and not just C-style comments, but Tsonnet should be as compatible as possible with Jsonnet, so be it.

It's fairly easy to support this -- just one more lexing pattern for inline comments. The following diff shows the simple addition to the lexer pattern:

Updating the sample file to cover the python-style comment:

Single-quoted strings

Single-quoted string is also straightforward. We copy and paste the double-quoted code and adapt the opening and closing with single quote:

Updating the sample files to follow:

Trailing commas

Trailing commas are allowed for objects and arrays. The parser needs a few tweaks to account for that:

The helper function separate_list had to be refactored to use pattern matching here. This was required to avoid an ambiguous grammar.

Updating sample files to follow:

Verbatim strings

Verbatim strings (also called raw strings or literal strings) are string literals where escape sequences are not processed -- what you see is exactly what you get.

The lexer rule for accepting text blocks with ||| is a simplified version of the single/double-quoted string rule:

Note that the verbatim_string function processes the raw content by trimming leading empty lines and normalizing whitespace, which matches Jsonnet's behavior.

Adding new samples and testing them:

Single-line verbatim strings:

Adding new sample files and testing them:

Putting it all together

Now let's test our enhanced Tsonnet with the original syntax.jsonnet example:

Voilà!

With these additions, Tsonnet now supports a much richer set of Jsonnet syntax features, bringing us closer to full compatibility. These syntactic sugar features might seem small individually, but they're crucial for real-world usability -- after all, who wants to escape every quote in a complex string or worry about trailing commas breaking their configuration?

The journey from a basic interpreter to a fully-featured configuration language continues, one commit at a time.

---