# Serialization

How TeXForm turns a Document back into LaTeX text, tokens, or a structured snapshot, with a text idempotency guarantee and configurable style.

https://texform.dev/docs/serialization

Once you have a `Document`, you turn it back into something usable: LaTeX text to store or display, tokens to align with that text, or a structured snapshot to transport. TeXForm names each channel for what it produces and guarantees that the text channel is stable under re-parsing, which is what makes its output safe to cache, deduplicate, and diff.

## What serialization is [#what-serialization-is]

The two channels you reach for most are named for what they produce rather than a generic "serialize":

**Python**

```python
import json
import texform

doc = texform.Parser().parse(r"\frac{x}{y}")["document"]
print(doc.to_latex())      # text channel
print(json.dumps(doc.to_syntax(), indent=2))     # structured channel
```

```text
\frac { x } { y }
{
  "Root": {
    "mode": "Math",
    "children": [
      {
        "Command": {
          "name": "frac",
          "args": [
            {
              "kind": "Mandatory",
              "no_leading_space": false,
              "value": {
                "MathContent": {
                  "Char": "x"
                }
              }
            },
            {
              "kind": "Mandatory",
              "no_leading_space": false,
              "value": {
                "MathContent": {
                  "Char": "y"
                }
              }
            }
          ],
          "known": true
        }
      }
    ]
  }
}
```

**JavaScript**

```js
import { Parser } from 'texform';

const doc = new Parser().parse('\\frac{x}{y}').document;
console.log(doc.toLatex());     // text channel
console.log(JSON.stringify(doc.toSyntax(), null, 2));    // structured channel
```

```text
\frac { x } { y }
{
  "Root": {
    "mode": "Math",
    "children": [
      {
        "Command": {
          "name": "frac",
          "args": [
            {
              "kind": "Mandatory",
              "no_leading_space": false,
              "value": {
                "MathContent": {
                  "Char": "x"
                }
              }
            },
            {
              "kind": "Mandatory",
              "no_leading_space": false,
              "value": {
                "MathContent": {
                  "Char": "y"
                }
              }
            }
          ],
          "known": true
        }
      }
    ]
  }
}
```

## Channels, named on purpose [#channels-named-on-purpose]

The channels answer different questions, and the naming keeps them apart:

* **`to_latex(options?)`** renders the tree back to LaTeX *text* in the selected style. This is the human-readable, re-parseable output. There is intentionally no method named `serialize` on `Document` — a generic name would hide which channel you meant.
* **`to_tokenized_latex(options?)`** produces that same text plus a typed token per output fragment, from one traversal. See [Tokenized output](#tokenized-output).
* **`to_syntax()`** converts the tree to a `SyntaxNode`, the single structured-data format. Structured output — JSON snapshots, Python dictionaries, JavaScript objects, test fixtures — always goes through `SyntaxNode`. `Document` itself does not serialize to structured data directly.

Reach for `to_latex()` when you need LaTeX a renderer or another tool can read; reach for `to_tokenized_latex()` when a downstream consumer needs to point back into that string; reach for `to_syntax()` when you need to store or transport the tree and rebuild it later with `Document.from_syntax()`.

A module-level `serialize` function also exists in both bindings, taking a `SyntaxNode` instead of a `Document`. In JavaScript it is marked `@deprecated`: prefer `Document.fromSyntax(node).toLatex(options)`, or `document.toLatex(options)` when you already hold a document.

## Text idempotency [#text-idempotency]

Both output styles guarantee **text idempotency**: parsing the output and serializing it again in the same style always produces the same string. Formally:

```text
serialize(parse(serialize(parse(src)))) == serialize(parse(src))
```

In practice, serializing a `Document` and then parsing-and-serializing that output again yields a fixed point:

**Python**

```python
import texform

parser = texform.Parser()
once = parser.parse(r"\frac{x}{y}")["document"].to_latex()
twice = parser.parse(once)["document"].to_latex()
print(once == twice, repr(once))
```

```text
True '\\frac { x } { y }'
```

**JavaScript**

```js
import { Parser } from 'texform';

const parser = new Parser();
const once = parser.parse('\\frac{x}{y}').document.toLatex();
const twice = parser.parse(once).document.toLatex();
console.log(once === twice, JSON.stringify(once));
```

```text
true "\\frac { x } { y }"
```

Two things this guarantee is, and is not:

* It is a **text-level** guarantee. The string is stable; the AST is not required to be identical across the round-trip. `parse(serialize(ast))` may recover a different but equivalent tree kind.
* It is **not normalization**. Serialization does not rewrite legacy syntax — `a \over b` serializes back as `a \over b`, not `\frac{a}{b}`. Canonicalizing the formula's content is the transform engine's job, covered in [Transforms](https://texform.dev/docs/transforms).

Text idempotency is what makes serialized output safe to use as a key: because the serialized text is stable, you can cache by it, deduplicate identical formulas, and produce diffs that reflect real changes rather than spelling noise.

## The default style is spaced [#the-default-style-is-spaced]

By default the serializer writes the **spaced** style: in math mode, tokens, braces, and scripts are separated by single spaces, so `\frac{x}{y}+x_i^2` becomes `\frac { x } { y } + x _ { i } ^ { 2 }`. A space at every token boundary means a downstream tokenizer never has to decide whether `}{` is one token or two, and the spaces are insignificant to LaTeX renderers, so the spaced form renders identically to a compact spelling. Adjacent digits stay together, text-mode content keeps its own spacing, and inline math binds tightly to `$`.

Spaced output is byte-stable across releases, so datasets, caches, and deduplication keys built on it stay valid when you upgrade.

## Output styles [#output-styles]

`to_latex()` takes one serializer option, `style`, with two values:

| Style              | Purpose                              | Example                                 |
| ------------------ | ------------------------------------ | --------------------------------------- |
| `spaced` (default) | Stable output for machine processing | `\frac { a } { b } + x _ { i } ^ { 2 }` |
| `compact`          | Concise output for people and LLMs   | `\frac{a}{b}+x_i^2`                     |

Compact keeps required lexical separators, such as `\alpha x` and a row break followed by sibling brackets (`\\ [6pt]`). It omits script braces only for a single TeX token: `x_i`, `x^2`, and `x_\alpha`, but `x^{10}` and `x^{\frac{a}{b}}` keep their braces. A following ASCII letter or digit receives a readability space (`x^2 3`, `x_i y`). Both styles put subscripts before superscripts and brace infix operands only when needed.

Both styles preserve the parsed structure: parsing either output gives back the same tree. Compact layout may still improve across releases, so prefer spaced output when you store results and compare them later. The style changes only spacing and optional braces; it does not choose a transform profile or rewrite commands.

**Python**

```python
import texform

doc = texform.Parser().parse(r"x_i^2")["document"]
print(doc.to_latex())                 # x _ { i } ^ { 2 }
print(doc.to_latex(style="compact"))  # x_i^2
```

**JavaScript**

```js
import { Parser } from 'texform';

const doc = new Parser().parse('x_i^2').document;
console.log(doc.toLatex());                     // x _ { i } ^ { 2 }
console.log(doc.toLatex({ style: 'compact' })); // x_i^2
```

Omitting `style`, or passing Python `None` or JavaScript `null`, selects spaced output. An unknown style or option key raises a configuration error instead of falling back to the default. For normalized output and the command line, see [Configure output style](https://texform.dev/docs/configure-output-style); for the option reference, see the [serialize API page](https://texform.dev/api/python/serialize).

## Tokenized output [#tokenized-output]

Sometimes the LaTeX string is not enough: a viewer needs to highlight a subexpression, an aligner needs to map model tokens onto source fragments, a diff needs to know which fragment changed. `to_tokenized_latex()` answers that by returning the serialized string together with one typed token per output fragment.

It is the same traversal, not a second pass. The tokens are recorded while the serializer walks the tree, so `latex` is byte-for-byte what `to_latex()` would have produced with the same `style`, and the token spans follow that style's spacing.

**Python**

```python
import texform

doc = texform.Parser().parse(r"\text{é} + x")["document"]
tok = doc.to_tokenized_latex()
print(repr(tok["latex"]))
for t in tok["tokens"]:
    print(t["kind"], t["mode"], t["start_byte"], t["end_byte"], repr(t["text"]))
```

```text
'\\text {é} + x'
control_sequence math 0 5 '\\text'
delimiter math 6 7 '{'
text text 7 9 'é'
delimiter math 9 10 '}'
character math 11 12 '+'
character math 13 14 'x'
```

**JavaScript**

```js
import { Parser } from 'texform';

const doc = new Parser().parse('\\text{é} + x').document;
const tok = doc.toTokenizedLatex();
console.log(JSON.stringify(tok.latex));
for (const t of tok.tokens) {
  console.log(t.kind, t.mode, t.startByte, t.endByte, JSON.stringify(t.text));
}
```

Three things to read out of that output.

**Each token carries a kind and a mode.** The kind is one of `control_sequence`, `character`, `delimiter`, `text`, `raw`, or `error` — semantic categories the serializer owns, not lexer categories. The mode is the fragment's semantic math or text mode, independent of spacing: `é` is `text` because it sits inside a `\text` argument, while the braces around it are math-mode delimiters belonging to the surrounding expression.

> **Warning**
>
> `start_byte` and `end_byte` are **UTF-8 byte offsets** into `latex`, not Python code-point indices and not JavaScript UTF-16 indices. The final `x` above spans bytes 13–14 of a string that is only 13 characters long, because `é` occupies two bytes. Slice the encoded bytes — `tok["latex"].encode()[13:14]` in Python, `Buffer.from(tok.latex, 'utf8').subarray(13, 14)` or a `TextEncoder` view in JavaScript — rather than indexing the string directly.

**Spans are ordered and never overlap.** The gaps between them are the serializer's own spacing, so the tokens are the content of the string and the spaces between tokens are layout. An `Error` node contributes an `error` token holding its captured snippet, and an *empty* error snippet contributes no token at all rather than a zero-width one — so a token list can be shorter than you expect on a partial tree. Use `has_errors()` to ask whether the document contains error nodes; do not infer it from the token list.

## Boundaries of the guarantee [#boundaries-of-the-guarantee]

Two node kinds show how the serializer handles content that is not ordinary math.

**Error nodes round-trip verbatim.** A partial tree containing `Error` nodes still serializes losslessly: each error re-emits the exact source snippet it captured, so nothing is dropped.

**Python**

```python
import texform

doc = texform.Parser().parse(r"x^")["document"]
print(doc.has_errors(), repr(doc.to_latex()))
```

```text
True 'x^'
```

**JavaScript**

```js
import { Parser } from 'texform';

const doc = new Parser().parse('x^').document;
console.log(doc.hasErrors(), JSON.stringify(doc.toLatex()));
```

```text
true "x^"
```

**Primes serialize compactly when pure.** A pure prime superscript is written back as quote shorthand — `f''` rather than `f^{\prime\prime}`. A mixed superscript that combines a prime with other content, such as `f'^2`, stays an ordinary script group (`f ^ { ' 2 }`), because compact shorthand cannot express the mix.

- [Pipeline](https://texform.dev/docs/pipeline) — Where each output channel sits in the overall flow.

- [Transforms](https://texform.dev/docs/transforms) — Normalize a formula's content before serializing it.

- [Configure output style](https://texform.dev/docs/configure-output-style) — Choose spaced or compact output in libraries, normalization, and the CLI.

- [Glossary](https://texform.dev/docs/glossary) — Text idempotency, overlay, and the rest of the vocabulary.