CSTML: A New Approach to Syntax Tree Markup

The world of data interchange formats is crowded, yet most proposed solutions fail to gain traction. The value of such formats is determined by who else is using them—who you can use them to communicate with. This creates a massive obstacle for new data languages like CSTML (Concrete Syntax Tree Markup Language), but the project has a clever trick up its sleeve: CSTML is designed to be the ideal format for representing the work done by parsers.

The Core Concept

At its heart, CSTML is a markup language that takes ideas from JSON, XML, and HTML. The fundamental insight is that any text file containing parseable code is essentially a CSTML document waiting to happen. By designing a format that perfectly captures parse trees—which use nodes to separate documents into hierarchies of spans—CSTML creates a universal language for syntax representation.

The syntax itself is elegant in its simplicity. A CSTML document is made up of tags, with the most basic being literal tags represented as JSON strings. For example, "The quick brown fox jumped over the lazy dog." is a valid CSTML document containing a single literal tag. Multiple adjacent strings concatenate automatically, allowing for pretty-formatting without changing content meaning.

Tags and Structure

CSTML introduces several tag types that work together to create rich, structured representations:

Open and Close Tags: These work similarly to HTML and XML, selecting regions of text and describing their contents. Unlike XML, the closing tag doesn't repeat the tag name, and self-closing tags follow JSX conventions.

Attributes: CSTML nodes can have attributes, but unlike HTML and XML, these support arbitrary JSON structure. This allows for rich metadata attachment, such as specifying tense information for verbs or grapheme counts for Unicode characters.

References: Perhaps the most powerful feature is the ability to create named relationships between nodes. Using syntax like subject:, verb:, and object:, CSTML documents can explicitly represent syntactic structure. The modifier[]: syntax uses array braces to indicate lists of related elements.

The Token System

Since CSTML is fundamentally about adding metadata to text, every piece of text needs a place for that metadata. This is handled through the token system, where all strings not explicitly wrapped in a "token node" are implicitly wrapped. The token flag * denotes these nodes, making the document structure explicit while keeping the syntax clean for simple cases.

Gaps and Shifting

One of CSTML's most innovative features is its handling of gaps—places where content is known to be missing. This allows documents to function as templates, similar to forms or images with transparent backgrounds. Gap tags are represented as <//>, and when combined with the template flag $, create powerful templating capabilities.

Shifting builds on gaps to enable incremental document construction. As you parse content left to right, shifts allow you to append tags dynamically. This is particularly useful for parsing expressions where the full structure isn't known until later in the input. The +: flag on reference tags indicates that shifts may occur at that location.

Covers and Trivia

Covers address the need to track not just what was found, but what might have been found. Written with an underscore prefix like <_Cover>, they create placeholders for potential content. Trivia references, denoted with #: syntax, handle whitespace and other non-semantic elements. This allows for precise representation of code structure while maintaining readability.

Namespaces Simplified

The namespace system in CSTML represents a significant improvement over XML's notoriously complex implementation. Instead of relying on URLs and aliases, CSTML uses binding tags written as :Binding::. The key insight is that namespace names actually mean something in CSTML, unlike XML where they're merely aliases.

This design allows for identity-stable nodes that can be moved between namespaces without changing their fundamental meaning. The :..: syntax refers to the parent namespace, and multiple names can be joined within a single binding tag using /.

Practical Applications

The real power of CSTML emerges when considering its applications. Because it's designed to represent parse trees, any programming language parser can emit CSTML, instantly making that language compatible with CSTML tools. This creates a network effect where supporting one language unlocks compatibility with potentially millions of existing documents.

Applications include:

• Syntax Highlighting: CSTML-emitting parsers can provide rich syntax highlighting by leveraging the structured tree representation.
• Structural Code Search: Moving beyond simple text search to semantic code analysis.
• Refactoring Tools: Immutable AST trees make it easy to define and execute semantically-driven changes across entire codebases.
• Alternative Editing Interfaces: CSTML enables "logic-brick" editing where code is constructed by snapping together syntactic elements, while still maintaining the efficiency of traditional text-based reading.

The Future Vision

The creators of CSTML envision a future where code editing is accessible to many more people and can be done on diverse devices like touchscreen tablets and VR headsets. Rather than emulating the typewriter experience of traditional editors, CSTML aims to make coding feel more like snapping together Lego bricks.

This approach doesn't abandon syntax—quite the opposite. CSTML embraces syntactic symbols as the most efficient way for humans to read code, while providing a brick-based interface for writing. The result is a system where complete novices can learn by playing with syntactic elements, while professionals maintain the efficiency of traditional text-based workflows.

Conclusion

CSTML represents a thoughtful synthesis of existing markup languages, addressing their limitations while introducing powerful new capabilities. By focusing on syntax tree representation and making parser integration seamless, it creates a foundation for a new generation of code tools and editing experiences. While the challenge of adoption remains significant, the technical foundation is solid, and the potential applications are compelling enough to warrant serious consideration from the developer tools community.