# Architecture

Crate layout, the stability boundary, the processing pipeline, three tree representations, and how the bindings work.

https://texform.dev/docs/architecture

Read this if you contribute to TeXForm, integrate it and want to know exactly what the public API guarantees, or are simply curious how a formula moves through the system. This page covers the *why* of the structure: where the stability boundary sits, the pipeline a formula travels through, the three tree representations and why they are distinct, and how the language bindings expose one Rust core.

This page is the site-facing account of the model that the repository's [`ARCHITECTURE.md`](https://github.com/texform-dev/texform/blob/main/ARCHITECTURE.md) defines for contributors reading the source. Where the two differ in wording, the repository document is the normative one; the [source map](#source-map) at the end of this page lists the files each section is drawn from.

## Crate layout and the stability boundary [#crate-layout-and-the-stability-boundary]

TeXForm is a Rust workspace of small, single-purpose crates. Only one of them — `texform` — carries a public stability guarantee. Every other crate is an internal implementation detail whose API may change without notice. External code must integrate through the `texform` facade.

Python and WebAssembly bindings use the public texform facade. Internal dependencies are transform, core, knowledge, knowledge-macros, argspec, and interface; regression is internal tooling.

Arrows point from a crate to the crates it depends on. Everything below `texform` is internal; the facade is the only general-purpose Rust crate external code should depend on. The binding crates are published language surfaces, but they may still depend on internal helper crates for host-language glue such as generated TypeScript shapes and option conversion.

| Crate                            | Responsibility                                                                                                                                                                                                                                    |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `texform`                        | Public facade. Re-exports the stable surface: `Parser`, `TransformEngine`, `Document`, `ParseResult`, serialization, `validate_argspec`, analysis helpers, and package metadata. The only general-purpose Rust crate other code should depend on. |
| `texform-core`                   | The parser, the internal `Ast` arena, the serializer, and the public `Document` DOM layer.                                                                                                                                                        |
| `texform-transform`              | The phase-oriented rewrite and normalization engine that operates on a parsed tree.                                                                                                                                                               |
| `texform-knowledge`              | The knowledge base: commands, environments, characters, and delimiters — which names are known, in which package and mode, and what argument shapes they take.                                                                                    |
| `texform-knowledge-macros`       | Procedural macros used by the knowledge base, including compile-time argument-specification parsing for generated records.                                                                                                                        |
| `texform-argspec`                | An xparse-style argument-specification parser used to describe command signatures.                                                                                                                                                                |
| `texform-interface`              | Dependency-free shared types, most importantly `SyntaxNode`, the lossless parse snapshot.                                                                                                                                                         |
| `texform-python`, `texform-wasm` | Language bindings that expose the shared facade model to Python and WebAssembly.                                                                                                                                                                  |
| `texform-regression`             | Corpus regression and data-product tooling for parser regression, transform-contract checking, and counter-map generation. Internal tooling, not part of the public API.                                                                          |

The facade deliberately does **not** re-export the internal `Ast`, `Node`, or arena types. Users get a single editable tree type — `Document` — and never touch the panic-contract arena underneath it. In each language the package you install is `texform` (`pip install texform`, `npm install texform`, `cargo add texform`); the internal `texform-*` crates are not meant to be a direct dependency.

## The processing pipeline [#the-processing-pipeline]

A formula flows through the system in one direction: parsing produces an editable document, and that document supports four separate operations — text output, tokenized output, transformation, and a syntax snapshot.

LaTeX source → Parser and knowledge base → SyntaxNode snapshot → editable Document. The Document supports text output, typed tokens, normalization, and structured snapshots.

The parser produces a `SyntaxNode` first. TeXForm then converts that snapshot into a `Document`, which is the tree users read, edit, serialize, and transform. The parser-driven path also carries source spans across into the `Document`. A `SyntaxNode` is never edited directly: it is the wire format, and `Document` is the working format.

`to_latex()` and `to_tokenized_latex()` are the same serializer traversal: the tokenized form turns on a recorder that emits typed fragments with UTF-8 byte spans into the identical LaTeX string, so the two never disagree about the text. See [serialization](https://texform.dev/docs/serialization) for when each one is worth reaching for.

## Three tree representations [#three-tree-representations]

The same formula is modeled by three trees with deliberately separated roles. Confusing them is the most common source of design mistakes, so the boundaries are explicit.

* **`SyntaxNode`** (from `texform-interface`) is the lossless, immutable parse snapshot. It is the parser's stage-1 output and the single serde DTO — it derives `Serialize`/`Deserialize` and backs JSON snapshots, Python dictionaries, JavaScript objects, and test fixtures. It can represent a partial parse (it may contain `Error` nodes) but carries no editing behavior.
* **`Document`** (from `texform-core`, re-exported by the facade) is the public, editable DOM-style tree. It is what users construct, query, mutate, serialize, and transform. Reads go through lightweight `NodeRef` handles; edits are fallible and return a `Result` with an `EditError` on misuse.
* **`Ast`** is the internal arena tree that the transform engine and serializer operate on. It is a *panic-contract* type: its methods panic on misuse, because misuse means an internal invariant was violated, not that a user supplied bad input. It is **not** part of the public API. `Document` wraps it and exposes a fallible surface over it, so no arena panic can reach a caller on a user-input-driven path.

`Document::from_syntax` and `Document::to_syntax` bridge the snapshot and the working tree in both directions, and the conversion is symmetric over every node kind, including `Error` and `Prime`. Because a snapshot can come from outside the parser, `from_syntax` validates external input and rejects invalid syntax with a structured error rather than letting the internal arena panic.

### Structural validity versus semantic completeness [#structural-validity-versus-semantic-completeness]

Two notions of "valid" are kept strictly apart, because they answer different questions.

* **Structural validity** is about the shape of the arena: parent links are consistent, slots hold the right node kinds, an environment body is a group, the root is unique and parentless, and there are no cycles. `Ast` and `Document` maintain structural validity **at all times**. Crucially, promoting parse errors to first-class `Error` *leaves* means a tree containing errors is still structurally valid.
* **Semantic completeness** is about whether the tree contains any `Error` placeholder nodes. This is a separate, O(1)-queryable property exposed by `has_errors()` — not a structural invariant. The method is named for the concrete fact it reports, so read-only checks and transform gates read without a double negative.

This mirrors MathJax: its rendered MathML embeds error nodes for unparseable fragments, the output tree stays well-formed, and the renderer simply handles them. TeXForm takes the same approach — `Error` is a first-class node kind, the tree is always structurally valid, and "does it contain errors" is a cheap signal downstream consumers can check without validating the whole tree. A tree that `has_errors()` is read-only: there is nothing useful to do with a tree full of holes except inspect it, so fixing read-only-ness at construction keeps the editing contract simple.

`TransformEngine.transform` also requires a `Document` produced by that engine's parser; documents from another parser or rebuilt from syntax are rejected.

For the three honest `ParseResult` states and how errors round-trip, see [the parsing concept page](https://texform.dev/docs/parsing).

## Language bindings [#language-bindings]

The Python and WebAssembly bindings expose live `Document` and `Node` handles rather than copying trees across the language boundary.

* A binding `Node` is a cheap handle — a shared reference to its owning document plus an internal node id. All reads and edits delegate back to the document; the tree is never cloned across the boundary.
* The core `Document` stays a plain owned Rust value with no interior mutability. Sharing is provided only at the binding layer, using each runtime's native mechanism: a reference-counted cell on Python, and `Rc<RefCell<…>>` on WASM. Direct Rust users never pay for the bindings' sharing needs.
* Borrow conflicts and misuse surface as structured host-language exceptions, never as a panic crossing the FFI boundary. A read-only (error) document raises a read-only exception, and an edit mixing nodes from two documents is rejected before it reaches the core.

### Casing conventions [#casing-conventions]

The bindings present the same core through each language's idioms, and the casing rule is precise.

* **Python** exposes the canonical Rust snake_case form directly. Methods and option keys are snake_case (`has_errors()`, `to_latex()`, `finalize_ast`), and structured data — parse results, reports, lookup metadata, argspec validation results — is returned as plain `dict` values (typed with `TypedDict` in the bundled `.pyi` stubs).
* **JavaScript** provides the camelCase view of the same shapes: `hasErrors()`, `toLatex()`, `rejectUnknown`, `finalizeAst`. The public TypeScript declarations ship with the npm package.

```python
# Python: snake_case methods, dict-shaped data
import texform

engine = texform.TransformEngine(profile="corpus")
print(engine.normalize(r"a \over b"))  # \frac { a } { b }
reported = engine.normalize_with_report(r"a \over b")
print(reported["report"]["rewrite"]["iterations"])  # 2
```

```ts
// JavaScript: camelCase methods, the same shapes
import { TransformEngine } from "texform";

const engine = new TransformEngine({ profile: "corpus" });
console.log(engine.normalize("a \\over b")); // \frac { a } { b }
const reported = engine.normalizeWithReport("a \\over b");
console.log(reported.report.rewrite.iterations); // 2
```

One shape is exempt from this convention: **`SyntaxNode` is not part of binding casing conversion.** It remains the single tree wire format across Rust serde, Python dictionaries, JavaScript objects, and JSON fixtures, so its field and variant names stay identical in every language. Everything else — reports, configs, lookup metadata — follows the snake_case / camelCase split above.

The result is a consistent editing model across Rust, Python, and JavaScript, built on one core implementation rather than three hand-written copies.

## Source map [#source-map]

Paths are relative to the [TeXForm repository](https://github.com/texform-dev/texform) root. They move as the code moves, so treat them as entry points rather than citations.

| Path                                          | What it defines                                                               |
| --------------------------------------------- | ----------------------------------------------------------------------------- |
| `ARCHITECTURE.md`                             | Crate boundaries, the pipeline, and the tree invariants this page summarizes. |
| `crates/texform-core/src/document.rs`         | The `Document` DOM, `NodeRef` reads, and the fallible editing surface.        |
| `crates/texform-core/src/ast.rs`              | The internal arena and its panic contract.                                    |
| `crates/texform-interface/src/syntax_node.rs` | The `SyntaxNode` wire format and its serde derives.                           |
| `crates/texform-core/src/serialize.rs`        | The serializer behind `to_latex` and `to_tokenized_latex`.                    |
| `crates/texform-wasm/Cargo.toml`              | The WebAssembly binding's dependency surface — the facade only.               |
| `crates/texform-python/src/`                  | The Python binding's handle model and exception mapping.                      |

## Related [#related]

- [Concepts: Pipeline](https://texform.dev/docs/pipeline) — The same journey, at a glance.

- [Concepts: Parsing](https://texform.dev/docs/parsing) — Parse-result states and the Document editing model.

- [Rules and the transform contract](https://texform.dev/docs/rule-contract) — How normalization is structured and enforced.

- [Python API overview](https://texform.dev/api/python) — snake_case conventions and dict shapes in full.

- [JavaScript API overview](https://texform.dev/api/js) — camelCase conventions and the WASM load model.