Intro

As I was reading Crafting Interpreters, I had this idea pretty early on that I wanted to be able to understand what the interpreter was doing visually, step by step. There are lots of graphics in the book that demonstrate how each step connects to the next one, but you are left using your imagination if you want to see how any arbitrary piece of code is converted from the language you write to the one the computer actually knows how to execute.

You can check it out here!

How It Works

On the Lox side I added json logs throughout each phase of the interpretation process – logs when tokens get lexed, logs when tokens get parsed, logs when bytecode gets generated, etc. These logs include an index into the source code so we can trace the path each piece of code takes as it’s transformed all the way into bytecode instructions to be executed. Then we build Lox to wasm and let the web app do the rest of the work.

The web app uses CodeMirror to let you type in any Lox code you want, runs it through the wasm build of the interpreter, and collects each group of log messages to produce visualizations of each compiler phase using D3. The lexer visualization is pretty simple; it’s just syntax highlighting! All we need is to produce a series of spans with the right text and css classes attached.

The parser is the trickiest one. It’s worth noting that the C version of the Lox interpreter just generates bytecode as it parses, never actually producing an AST data structure. Statements are parsed recursively in a way that mirrors the AST. Each statement knows how many children it has. We can process them linearly on the front end and build a tree that we can then hand off to D3. This gets a little trickier for expressions which are not parsed the same way. You can learn more about the Pratt parsing algorithm in the book here, but the takeaway for the AST visualizer is that all the nodes inside expressions get logged backwards. We have to stop every time we see an expression, gather up all its child nodes, and process them in a batch. So the resulting AST visualization doesn’t exactly mirror how the parser works. It seemed like an acceptable simplification to me.

The bytecode visualizer that shows all the VM instructions generated by each parsing step is mercifully straightforward. Since it’s just a list of instructions, we can just show them off in a list. It actually looks pretty much the same as the output printed by the VM to the console with the debug flags turned on. Likewise the stack is just a list of items on the stack. When the VM runs the instructions, we log the contents of the stack, so the playback visualizer can show you what happens as the VM is instructed to jump around to different instructions in the list.