Serialization

Once you have a Document, you turn it back into something usable: LaTeX text to store or display, tokens to align with that text, or a structured snapshot to transport. TeXForm names each channel for what it produces and guarantees that the text channel is stable under re-parsing, which is what makes its output safe to cache, deduplicate, and diff.

What serialization is#

The two channels you reach for most are named for what they produce rather than a generic "serialize":

import json
import texform

doc = texform.Parser().parse(r"\frac{x}{y}")["document"]
print(doc.to_latex())      # text channel
print(json.dumps(doc.to_syntax(), indent=2))     # structured channel

\frac { x } { y }
{
  "Root": {
    "mode": "Math",
    "children": [
      {
        "Command": {
          "name": "frac",
          "args": [
            {
              "kind": "Mandatory",
              "no_leading_space": false,
              "value": {
                "MathContent": {
                  "Char": "x"
                }
              }
            },
            {
              "kind": "Mandatory",
              "no_leading_space": false,
              "value": {
                "MathContent": {
                  "Char": "y"
                }
              }
            }
          ],
          "known": true
        }
      }
    ]
  }
}

import { Parser } from 'texform';

const doc = new Parser().parse('\\frac{x}{y}').document;
console.log(doc.toLatex());     // text channel
console.log(JSON.stringify(doc.toSyntax(), null, 2));    // structured channel

\frac { x } { y }
{
  "Root": {
    "mode": "Math",
    "children": [
      {
        "Command": {
          "name": "frac",
          "args": [
            {
              "kind": "Mandatory",
              "no_leading_space": false,
              "value": {
                "MathContent": {
                  "Char": "x"
                }
              }
            },
            {
              "kind": "Mandatory",
              "no_leading_space": false,
              "value": {
                "MathContent": {
                  "Char": "y"
                }
              }
            }
          ],
          "known": true
        }
      }
    ]
  }
}

Channels, named on purpose#

The channels answer different questions, and the naming keeps them apart:

• to_latex(options?) renders the tree back to LaTeX text in the selected style. This is the human-readable, re-parseable output. There is intentionally no method named serialize on Document — a generic name would hide which channel you meant.
• to_tokenized_latex(options?) produces that same text plus a typed token per output fragment, from one traversal. See Tokenized output.
• to_syntax() converts the tree to a SyntaxNode, the single structured-data format. Structured output — JSON snapshots, Python dictionaries, JavaScript objects, test fixtures — always goes through SyntaxNode. Document itself does not serialize to structured data directly.

Reach for to_latex() when you need LaTeX a renderer or another tool can read; reach for to_tokenized_latex() when a downstream consumer needs to point back into that string; reach for to_syntax() when you need to store or transport the tree and rebuild it later with Document.from_syntax().

A module-level serialize function also exists in both bindings, taking a SyntaxNode instead of a Document. In JavaScript it is marked @deprecated: prefer Document.fromSyntax(node).toLatex(options), or document.toLatex(options) when you already hold a document.

Text idempotency#

Both output styles guarantee text idempotency: parsing the output and serializing it again in the same style always produces the same string. Formally:

serialize(parse(serialize(parse(src)))) == serialize(parse(src))

In practice, serializing a Document and then parsing-and-serializing that output again yields a fixed point:

import texform

parser = texform.Parser()
once = parser.parse(r"\frac{x}{y}")["document"].to_latex()
twice = parser.parse(once)["document"].to_latex()
print(once == twice, repr(once))

True '\\frac { x } { y }'

import { Parser } from 'texform';

const parser = new Parser();
const once = parser.parse('\\frac{x}{y}').document.toLatex();
const twice = parser.parse(once).document.toLatex();
console.log(once === twice, JSON.stringify(once));

true "\\frac { x } { y }"

Two things this guarantee is, and is not:

• It is a text-level guarantee. The string is stable; the AST is not required to be identical across the round-trip. parse(serialize(ast)) may recover a different but equivalent tree kind.
• It is not normalization. Serialization does not rewrite legacy syntax — a \over b serializes back as a \over b, not \frac{a}{b}. Canonicalizing the formula's content is the transform engine's job, covered in Transforms.

Text idempotency is what makes serialized output safe to use as a key: because the serialized text is stable, you can cache by it, deduplicate identical formulas, and produce diffs that reflect real changes rather than spelling noise.

The default style is spaced#

By default the serializer writes the spaced style: in math mode, tokens, braces, and scripts are separated by single spaces, so \frac{x}{y}+x_i^2 becomes \frac { x } { y } + x _ { i } ^ { 2 }. A space at every token boundary means a downstream tokenizer never has to decide whether }{ is one token or two, and the spaces are insignificant to LaTeX renderers, so the spaced form renders identically to a compact spelling. Adjacent digits stay together, text-mode content keeps its own spacing, and inline math binds tightly to $.

Spaced output is byte-stable across releases, so datasets, caches, and deduplication keys built on it stay valid when you upgrade.

Output styles#

to_latex() takes one serializer option, style, with two values:

Compact keeps required lexical separators, such as \alpha x and a row break followed by sibling brackets (\\ [6pt]). It omits script braces only for a single TeX token: x_i, x^2, and x_\alpha, but x^{10} and x^{\frac{a}{b}} keep their braces. A following ASCII letter or digit receives a readability space (x^2 3, x_i y). Both styles put subscripts before superscripts and brace infix operands only when needed.

Both styles preserve the parsed structure: parsing either output gives back the same tree. Compact layout may still improve across releases, so prefer spaced output when you store results and compare them later. The style changes only spacing and optional braces; it does not choose a transform profile or rewrite commands.

import texform

doc = texform.Parser().parse(r"x_i^2")["document"]
print(doc.to_latex())                 # x _ { i } ^ { 2 }
print(doc.to_latex(style="compact"))  # x_i^2

import { Parser } from 'texform';

const doc = new Parser().parse('x_i^2').document;
console.log(doc.toLatex());                     // x _ { i } ^ { 2 }
console.log(doc.toLatex({ style: 'compact' })); // x_i^2

Omitting style, or passing Python None or JavaScript null, selects spaced output. An unknown style or option key raises a configuration error instead of falling back to the default. For normalized output and the command line, see Configure output style; for the option reference, see the serialize API page.

Tokenized output#

Sometimes the LaTeX string is not enough: a viewer needs to highlight a subexpression, an aligner needs to map model tokens onto source fragments, a diff needs to know which fragment changed. to_tokenized_latex() answers that by returning the serialized string together with one typed token per output fragment.

It is the same traversal, not a second pass. The tokens are recorded while the serializer walks the tree, so latex is byte-for-byte what to_latex() would have produced with the same style, and the token spans follow that style's spacing.

import texform

doc = texform.Parser().parse(r"\text{é} + x")["document"]
tok = doc.to_tokenized_latex()
print(repr(tok["latex"]))
for t in tok["tokens"]:
    print(t["kind"], t["mode"], t["start_byte"], t["end_byte"], repr(t["text"]))

'\\text {é} + x'
control_sequence math 0 5 '\\text'
delimiter math 6 7 '{'
text text 7 9 'é'
delimiter math 9 10 '}'
character math 11 12 '+'
character math 13 14 'x'

import { Parser } from 'texform';

const doc = new Parser().parse('\\text{é} + x').document;
const tok = doc.toTokenizedLatex();
console.log(JSON.stringify(tok.latex));
for (const t of tok.tokens) {
  console.log(t.kind, t.mode, t.startByte, t.endByte, JSON.stringify(t.text));
}

Three things to read out of that output.

Each token carries a kind and a mode. The kind is one of control_sequence, character, delimiter, text, raw, or error — semantic categories the serializer owns, not lexer categories. The mode is the fragment's semantic math or text mode, independent of spacing: é is text because it sits inside a \text argument, while the braces around it are math-mode delimiters belonging to the surrounding expression.

Spans are ordered and never overlap. The gaps between them are the serializer's own spacing, so the tokens are the content of the string and the spaces between tokens are layout. An Error node contributes an error token holding its captured snippet, and an empty error snippet contributes no token at all rather than a zero-width one — so a token list can be shorter than you expect on a partial tree. Use has_errors() to ask whether the document contains error nodes; do not infer it from the token list.

Boundaries of the guarantee#

Two node kinds show how the serializer handles content that is not ordinary math.

Error nodes round-trip verbatim. A partial tree containing Error nodes still serializes losslessly: each error re-emits the exact source snippet it captured, so nothing is dropped.

import texform

doc = texform.Parser().parse(r"x^")["document"]
print(doc.has_errors(), repr(doc.to_latex()))

True 'x^'

import { Parser } from 'texform';

const doc = new Parser().parse('x^').document;
console.log(doc.hasErrors(), JSON.stringify(doc.toLatex()));

true "x^"

Primes serialize compactly when pure. A pure prime superscript is written back as quote shorthand — f'' rather than f^{\prime\prime}. A mixed superscript that combines a prime with other content, such as f'^2, stays an ordinary script group (f ^ { ' 2 }), because compact shorthand cannot express the mix.