Architecture

Read this if you contribute to TeXForm, integrate it and want to know exactly what the public API guarantees, or are simply curious how a formula moves through the system. This page covers the why of the structure: where the stability boundary sits, the pipeline a formula travels through, the three tree representations and why they are distinct, and how the language bindings expose one Rust core.

This page is the site-facing account of the model that the repository's ARCHITECTURE.md defines for contributors reading the source. Where the two differ in wording, the repository document is the normative one; the source map at the end of this page lists the files each section is drawn from.

Crate layout and the stability boundary#

TeXForm is a Rust workspace of small, single-purpose crates. Only one of them — texform — carries a public stability guarantee. Every other crate is an internal implementation detail whose API may change without notice. External code must integrate through the texform facade.

Arrows point from a crate to the crates it depends on. Everything below texform is internal; the facade is the only general-purpose Rust crate external code should depend on. The binding crates are published language surfaces, but they may still depend on internal helper crates for host-language glue such as generated TypeScript shapes and option conversion.

The facade deliberately does not re-export the internal Ast, Node, or arena types. Users get a single editable tree type — Document — and never touch the panic-contract arena underneath it. In each language the package you install is texform (pip install texform, npm install texform, cargo add texform); the internal texform-* crates are not meant to be a direct dependency.

The processing pipeline#

A formula flows through the system in one direction: parsing produces an editable document, and that document supports four separate operations — text output, tokenized output, transformation, and a syntax snapshot.

The parser produces a SyntaxNode first. TeXForm then converts that snapshot into a Document, which is the tree users read, edit, serialize, and transform. The parser-driven path also carries source spans across into the Document. A SyntaxNode is never edited directly: it is the wire format, and Document is the working format.

to_latex() and to_tokenized_latex() are the same serializer traversal: the tokenized form turns on a recorder that emits typed fragments with UTF-8 byte spans into the identical LaTeX string, so the two never disagree about the text. See serialization for when each one is worth reaching for.

Three tree representations#

The same formula is modeled by three trees with deliberately separated roles. Confusing them is the most common source of design mistakes, so the boundaries are explicit.

• SyntaxNode (from texform-interface) is the lossless, immutable parse snapshot. It is the parser's stage-1 output and the single serde DTO — it derives Serialize/Deserialize and backs JSON snapshots, Python dictionaries, JavaScript objects, and test fixtures. It can represent a partial parse (it may contain Error nodes) but carries no editing behavior.
• Document (from texform-core, re-exported by the facade) is the public, editable DOM-style tree. It is what users construct, query, mutate, serialize, and transform. Reads go through lightweight NodeRef handles; edits are fallible and return a Result with an EditError on misuse.
• Ast is the internal arena tree that the transform engine and serializer operate on. It is a panic-contract type: its methods panic on misuse, because misuse means an internal invariant was violated, not that a user supplied bad input. It is not part of the public API. Document wraps it and exposes a fallible surface over it, so no arena panic can reach a caller on a user-input-driven path.

Document::from_syntax and Document::to_syntax bridge the snapshot and the working tree in both directions, and the conversion is symmetric over every node kind, including Error and Prime. Because a snapshot can come from outside the parser, from_syntax validates external input and rejects invalid syntax with a structured error rather than letting the internal arena panic.

Structural validity versus semantic completeness#

Two notions of "valid" are kept strictly apart, because they answer different questions.

• Structural validity is about the shape of the arena: parent links are consistent, slots hold the right node kinds, an environment body is a group, the root is unique and parentless, and there are no cycles. Ast and Document maintain structural validity at all times. Crucially, promoting parse errors to first-class Error leaves means a tree containing errors is still structurally valid.
• Semantic completeness is about whether the tree contains any Error placeholder nodes. This is a separate, O(1)-queryable property exposed by has_errors() — not a structural invariant. The method is named for the concrete fact it reports, so read-only checks and transform gates read without a double negative.

This mirrors MathJax: its rendered MathML embeds error nodes for unparseable fragments, the output tree stays well-formed, and the renderer simply handles them. TeXForm takes the same approach — Error is a first-class node kind, the tree is always structurally valid, and "does it contain errors" is a cheap signal downstream consumers can check without validating the whole tree. A tree that has_errors() is read-only: there is nothing useful to do with a tree full of holes except inspect it, so fixing read-only-ness at construction keeps the editing contract simple.

TransformEngine.transform also requires a Document produced by that engine's parser; documents from another parser or rebuilt from syntax are rejected.

For the three honest ParseResult states and how errors round-trip, see the parsing concept page.

Language bindings#

The Python and WebAssembly bindings expose live Document and Node handles rather than copying trees across the language boundary.

• A binding Node is a cheap handle — a shared reference to its owning document plus an internal node id. All reads and edits delegate back to the document; the tree is never cloned across the boundary.
• The core Document stays a plain owned Rust value with no interior mutability. Sharing is provided only at the binding layer, using each runtime's native mechanism: a reference-counted cell on Python, and Rc<RefCell<…>> on WASM. Direct Rust users never pay for the bindings' sharing needs.
• Borrow conflicts and misuse surface as structured host-language exceptions, never as a panic crossing the FFI boundary. A read-only (error) document raises a read-only exception, and an edit mixing nodes from two documents is rejected before it reaches the core.

Casing conventions#

The bindings present the same core through each language's idioms, and the casing rule is precise.

• Python exposes the canonical Rust snake_case form directly. Methods and option keys are snake_case (has_errors(), to_latex(), finalize_ast), and structured data — parse results, reports, lookup metadata, argspec validation results — is returned as plain dict values (typed with TypedDict in the bundled .pyi stubs).
• JavaScript provides the camelCase view of the same shapes: hasErrors(), toLatex(), rejectUnknown, finalizeAst. The public TypeScript declarations ship with the npm package.

# Python: snake_case methods, dict-shaped data
import texform

engine = texform.TransformEngine(profile="corpus")
print(engine.normalize(r"a \over b"))  # \frac { a } { b }
reported = engine.normalize_with_report(r"a \over b")
print(reported["report"]["rewrite"]["iterations"])  # 2

// JavaScript: camelCase methods, the same shapes
import { TransformEngine } from "texform";

const engine = new TransformEngine({ profile: "corpus" });
console.log(engine.normalize("a \\over b")); // \frac { a } { b }
const reported = engine.normalizeWithReport("a \\over b");
console.log(reported.report.rewrite.iterations); // 2

One shape is exempt from this convention: SyntaxNode is not part of binding casing conversion. It remains the single tree wire format across Rust serde, Python dictionaries, JavaScript objects, and JSON fixtures, so its field and variant names stay identical in every language. Everything else — reports, configs, lookup metadata — follows the snake_case / camelCase split above.

The result is a consistent editing model across Rust, Python, and JavaScript, built on one core implementation rather than three hand-written copies.

Source map#

Paths are relative to the TeXForm repository root. They move as the code moves, so treat them as entry points rather than citations.

Related#