Tsonnet #23 - Mirror, mirror on the wall, who's the most self-referential of them all?

Tsonnet objects can now reference themselves using self, but this required careful implementation of unique object identifiers, eager scope checking to prevent misuse, and cycle detection

Welcome to the Tsonnet series!

If you're not following the series so far, you can check out how it all started in the first post of the series.

In the previous post, I fixed Tsonnet's lazy evaluation inconsistency in objects:

Tsonnet #22 - Fixing a "lazy" bug

Now it's time to tackle object self-references.

Photo by Patrick von der Wehd on Unsplash

The problem: objects can't talk to themselves

Consider this perfectly reasonable configuration:

{
    one: 1,
    two: self.one + 1
}

This should evaluate to { "one": 1, "two": 2 }, but currently Tsonnet has no clue what's going on:

dune exec -- tsonnet samples/objects/self_reference.jsonnet
samples/objects/self_reference.jsonnet:3:14 Unexpected char: .

3:     two: self.one + 1
   ^^^^^^^^^^^^^^^^^^^^^

The lexer doesn't even recognize the dot! We need object field access, stat.

Time to catch up.

The lexer and parser dance

First things first -- let's teach our lexer about dots and the self keyword:

lib/lexer.mll

The parser follows suit, understanding that SELF DOT field makes an ObjectFieldAccess:

lib/parser.mly

Extending the AST

We need two new AST nodes to capture self-references:

lib/ast.ml

Here's the clever bit: ObjectSelf carries an Env.env_id -- a unique identifier for each object. This lets us know exactly which self belongs to which object.

Putting self into the environment

When interpreting objects, we were already generating the object ID, but now we add self to the environment:

lib/interpreter.ml

Notice how we add "self" as a special variable pointing to the object's unique ID? This is how we track which object we're in.

When we encounter self.field, the magic happens in interpret_object_field_access:

lib/interpreter.ml

This looks up self in the environment, extracts the object ID, and uses it to retrieve the requested field. Neat!

Our basic case now works:

$ dune exec -- tsonnet samples/objects/self_reference.jsonnet
{ "one": 1, "two": 2 }

But wait -- what about using self outside of objects?

Scope checking: keeping self safe

Let's see what should happen when we use self outside an object:

local _one = 1;
local _two = self.one + 1;
{
    one: _one,
    two: _two
}

Jsonnet correctly catches this:

$ jsonnet samples/errors/object_self_out_of_scope.jsonnet
samples/errors/object_self_out_of_scope.jsonnet:2:14-18 Can't use self outside of an object.

local _two = self.one + 1;

But after implementing object field access, Tsonnet misbehaves:

$ dune exec -- tsonnet samples/errors/object_self_out_of_scope.jsonnet
{ "one": 1, "two": 1 }

Like, WHAT?!

This is happening because Tsonnet is LAZY! 🥁

Yeah, lazy evaluation is tricky sometimes, may seem wrong when it is not, but here it really is a bug. We need scope validation to catch improper self usage before evaluation begins.

Things aren't so simple, however. We can't enforce self out of object scope like we do with local variables. When we add self to the environment to be later evaluated, by the time it evaluates it could be outside its scope, like the little surprise we got earlier.

So, how did I solve this? Introducing a new module that performs an eager pass through the AST.

I know, one more pass. Now we are going to have:

1. Lexing

2. Parsing

3. Scope checking

4. Type checking

5. Interpretation

It's well worth it, I promise.

Here's the scope validator in its entirety:

lib/scope.ml

The validator tracks whether we're inside an object and catches invalid self usage:

lib/scope.ml

Now we integrate scope validation into the type checker:

lib/type.ml

Perfect! Now invalid self usage gets caught early:

$ dune exec -- tsonnet samples/errors/object_self_out_of_scope.jsonnet
samples/errors/object_self_out_of_scope.jsonnet:2:13 Can't use self outside of an object

2: local _two = self.one + 1;
   ^^^^^^^^^^^^^^^^^^^^^^^^^^

The infinite loop protection

Now that self-references work, we need to prevent infinite cycles between object fields. Consider this problematic case:

{
    a: self.b,
    b: self.a,
}

Jsonnet detects this after hitting a stack limit:

$ jsonnet samples/semantics/invalid_binding_cycle_object_fields.jsonnet
RUNTIME ERROR: max stack frames exceeded.
	samples/semantics/invalid_binding_cycle_object_fields.jsonnet:2:8-14	object <anonymous>
	samples/semantics/invalid_binding_cycle_object_fields.jsonnet:3:8-14	object <anonymous>
	samples/semantics/invalid_binding_cycle_object_fields.jsonnet:2:8-14	object <anonymous>
	samples/semantics/invalid_binding_cycle_object_fields.jsonnet:3:8-14	object <anonymous>
	samples/semantics/invalid_binding_cycle_object_fields.jsonnet:2:8-14	object <anonymous>
	samples/semantics/invalid_binding_cycle_object_fields.jsonnet:3:8-14	object <anonymous>
	samples/semantics/invalid_binding_cycle_object_fields.jsonnet:2:8-14	object <anonymous>
	samples/semantics/invalid_binding_cycle_object_fields.jsonnet:3:8-14	object <anonymous>
	samples/semantics/invalid_binding_cycle_object_fields.jsonnet:2:8-14	object <anonymous>
	samples/semantics/invalid_binding_cycle_object_fields.jsonnet:3:8-14	object <anonymous>
	...
	samples/semantics/invalid_binding_cycle_object_fields.jsonnet:3:8-14	object <anonymous>
	samples/semantics/invalid_binding_cycle_object_fields.jsonnet:2:8-14	object <anonymous>
	samples/semantics/invalid_binding_cycle_object_fields.jsonnet:3:8-14	object <anonymous>
	samples/semantics/invalid_binding_cycle_object_fields.jsonnet:2:8-14	object <anonymous>
	samples/semantics/invalid_binding_cycle_object_fields.jsonnet:3:8-14	object <anonymous>
	samples/semantics/invalid_binding_cycle_object_fields.jsonnet:2:8-14	object <anonymous>
	samples/semantics/invalid_binding_cycle_object_fields.jsonnet:3:8-14	object <anonymous>
	samples/semantics/invalid_binding_cycle_object_fields.jsonnet:2:8-14	object <anonymous>
	Field "a"
	During manifestation

That's messy. We can do better with compile-time detection.

We extend our existing cycle detection to handle object fields:

lib/type.ml

The new check_object_field_for_cycles function handles self.field references by looking up the object ID and checking for cycles:

lib/type.ml

And for objects themselves, we check all fields for cycles during type checking, right after we translate the local variables:

Now cyclical object fields are caught at type check time:

$ dune exec -- tsonnet samples/semantics/invalid_binding_cycle_object_fields.jsonnet
samples/semantics/invalid_binding_cycle_object_fields.jsonnet:3:7 Cyclic reference found for 1->a

3:     b: self.a,
   ^^^^^^^^^^^^^^

And even mixed cycles between local variables and object fields:

$ dune exec -- tsonnet samples/semantics/invalid_binding_cycle_object_field_and_local.jsonnet
samples/semantics/invalid_binding_cycle_object_field_and_local.jsonnet:2:14 Cyclic reference found for 1->b

2:     local a = self.b,
   ^^^^^^^^^^^^^^^^^^^^^

And cram tests won't let these errors creep in again:

Conclusion

The error messages still show internal identifiers like 1->b instead of self.b, but that's polish for another day. The core functionality is solid -- objects can now reference themselves safely and correctly.

Here is the entire diff.

In the upcoming post, I will most likely tackle the outer-most object reference. Don't know what it is? Then see you on the next one.

---