# Parsing

Why TeXForm parses LaTeX against a knowledge base, which packages it loads by default, and what a parse result can contain.

https://texform.dev/docs/parsing

Parsing LaTeX is hard not because of string manipulation but because the meaning of a command depends on knowing its argument structure. TeXForm parses against a knowledge base of command signatures, so a formula becomes a real syntax tree instead of a best-effort guess.

## What parsing is [#what-parsing-is]

Parsing turns LaTeX source into a `Document` — an editable tree — together with a list of diagnostics describing anything the parser could not handle.

**Python**

```python
import texform

result = texform.Parser().parse(r"\frac{x}{y}")
print(result["document"].to_latex())
```

```text
\frac { x } { y }
```

**JavaScript**

```js
import { Parser } from 'texform';

const result = new Parser().parse('\\frac{x}{y}');
console.log(result.document.toLatex());
```

```text
\frac { x } { y }
```

## Why parsing is knowledge-driven [#why-parsing-is-knowledge-driven]

Argument structure is not something a grammar can infer from syntax alone. `\frac{x}{y}` takes two mandatory arguments, but nothing in the characters `\frac` says so — a generic LaTeX grammar cannot know that `\frac` consumes the next two groups while `\alpha` consumes none. The only way to parse `\frac`, `\over`, or `\quantity` correctly is to know each command's signature in advance.

This is why regular expressions and one-off heuristics keep breaking on real formulas: they guess at structure the source does not encode. TeXForm reads each command's signature from a knowledge base, so the parser binds arguments the way the command actually behaves rather than guessing.

## Packages and context [#packages-and-context]

The knowledge base is organized into packages. Each package contributes a set of known command and environment signatures, and you choose which packages the parser consults. Listing them shows the whole catalog:

**Python**

```python
import texform

for pkg in texform.list_packages():
    print(pkg["name"], pkg["commands"], pkg["environments"])
```

```text
ams 35 28
base 270 7
bboldx 12 0
boldsymbol 1 0
braket 11 0
color 5 0
physics 182 1
textmacros 78 0
```

**JavaScript**

```js
import { listPackages } from 'texform';

for (const pkg of listPackages()) {
  console.log(pkg.name, pkg.commands, pkg.environments);
}
```

```text
ams 35 28
base 270 7
bboldx 12 0
boldsymbol 1 0
braket 11 0
color 5 0
physics 182 1
textmacros 78 0
```

### The default package set [#the-default-package-set]

`list_packages()` lists the catalog, not the parser's default selection. A `Parser` or `TransformEngine` built without a knowledge base shares the default one, which loads the default set: `base`, `ams`, `braket`, `textmacros`, `bboldx`, `boldsymbol`, and `color`. `physics` is opt-in because it redefines commands that `braket` also defines, and gives standard commands such as `\div`, `\Re`, and `\Im` its own meanings. Loading both is allowed: TeXForm imports built-in packages in a fixed order, with `physics` overriding the shared command definitions from `braket`. Reordering the `packages` array does not change that precedence. Select the package whose command syntax your input uses.

The consequence is concrete. Under the default set, `\ket` resolves to the `braket` record, whose signature is `m`, and normalization keeps `\ket{\psi}` as written. The `physics` record for the same name has the signature `s m` — a starred variant plus one mandatory argument — and `Corpus` normalization expands `\ket{\psi}` into `\left \vert \psi \right \rangle`. If your corpus means `physics`, ask for it explicitly:

**Python**

```python
import texform

kb = texform.KnowledgeBase(packages=["base", "ams", "physics"])
parser = texform.Parser(knowledge_base=kb)
print(kb.lookup_command("ket", "math")["spec_string"])
print(kb.lookup_command("ket", "math")["from_packages"])
```

```text
s m
['physics']
```

**JavaScript**

```js
import { KnowledgeBase, Parser } from 'texform';

const knowledgeBase = new KnowledgeBase({ packages: ['base', 'ams', 'physics'] });
const parser = new Parser({ knowledgeBase });
console.log(knowledgeBase.lookupCommand('ket', 'math').specString);
console.log(JSON.stringify(knowledgeBase.lookupCommand('ket', 'math').fromPackages));
```

```text
s m
["physics"]
```

Naming `packages` yourself replaces the default set rather than adding to it, so list every package you need, including `base`.

When a command is not in your selected packages, you can register it yourself by passing context items to the knowledge base the parser uses. A context item is a small record carrying a command name and an argspec; the parser treats it as if it were a built-in signature. See [Inject custom commands](https://texform.dev/docs/inject-custom-commands) for the full workflow.

## Argument specifications [#argument-specifications]

A command's signature is written as an **argspec** — a compact, xparse-style string describing each argument the command consumes. The argspec is what tells the parser to read two groups after `\frac` and none after `\alpha`. It is the single mechanism behind knowledge-driven parsing.

The notation is small. A few common slots:

* `m` — one mandatory argument. `\frac` is `m m` (two mandatory groups).
* `o` — one optional bracketed argument. `\sqrt` is `o m` (an optional index, then the radicand).
* `s` — an optional leading star. With `physics` loaded, `\ket` is `s m`, so `\ket*{\psi}` parses with its star recorded.

You can validate an argspec string directly, which reports how many arguments it describes:

**Python**

```python
import texform

print(texform.validate_argspec("o m"))
```

```text
{'valid': True, 'error': None, 'arg_count': 2, 'parsed': [{'required': False, 'no_leading_space': False, 'nullable': False, 'kind': {'type': 'content', 'mode': 'math'}, 'form': {'type': 'standard'}}, {'required': True, 'no_leading_space': False, 'nullable': False, 'kind': {'type': 'content', 'mode': 'math'}, 'form': {'type': 'standard'}}]}
```

**JavaScript**

```js
import { validateArgspec } from 'texform';

const result = validateArgspec('o m');
console.log(result.valid, result.argCount);
```

```text
true 2
```

The full notation — every slot kind, delimited forms, and TeXForm's extensions to xparse — is documented in the [argspec reference](https://texform.dev/docs/argspec). For why the notation is shaped the way it is and its known limits, see [Argspec design](https://texform.dev/docs/argspec-design).

## Configuring strictness [#configuring-strictness]

Two independent knobs on `ParseConfig` decide how the parser reacts to trouble, and for both of them `True` means stricter. `reject_unknown` controls what happens to a command the knowledge base does not recognize; `abort_on_error` controls what happens when the parser hits a syntax error it cannot continue past. Both default to `False`, so the default configuration is lenient on both axes. They combine into four behaviors:

|                          | `abort_on_error = False`                                                                                                                                                                          | `abort_on_error = True`                                                                             |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
| `reject_unknown = False` | The default. Unknown commands survive as nodes; the parser recovers from syntax errors and collects every diagnostic.                                                                             | Unknown commands survive; the first syntax error stops the parse and returns no tree.               |
| `reject_unknown = True`  | Unknown names become diagnostics; recovery still applies to syntax errors. Recovery consumes the unknown command itself, so `\mycmd{x} + y` keeps `{x}`, `+`, and `y` alongside one `Error` node. | Both axes strict. Any unknown name or syntax error is a hard failure and the parse returns no tree. |

The default corner is the most permissive, which is what you want when cleaning noisy input. The opposite corner — both knobs `True` — is what "strict parsing" means precisely: it is for input you expect to be clean and want to reject loudly. Rust names these two corners `ParseConfig::STRICT` and `ParseConfig::LENIENT`; Python and JavaScript express them through constructor defaults and per-call overlays instead, so there is no `ParseConfig.LENIENT` constant to import.

Lenient parsing recovers where it can, but recovery is not a promise of a tree. A formula whose structure the parser cannot close, such as `\frac{a}{b`, still comes back with `document = None` under the default configuration, while `x^` comes back as a read-only tree holding an `Error` node. Treat "returned a tree" as something to check, not to assume — the three-state handling below is written for exactly that.

The two knobs are orthogonal to the result's shape: `abort_on_error` is a parse-strictness switch, while whether the resulting tree contains errors is a separate, queryable property of the tree.

For task-oriented recipes built on these settings, see [Handle parse errors](https://texform.dev/docs/handle-parse-errors) and [Clean OCR/MER output](https://texform.dev/docs/clean-ocr-mer-output).

## What parsing gives you [#what-parsing-gives-you]

A parse result has exactly three honest states. TeXForm never fabricates a placeholder tree to pretend a document always exists:

For x^2, the base x spans bytes [0, 1), the exponent 2 spans [2, 3), and the script spans [0, 3). Parsing yields no document, a clean editable document, or a partial read-only document with Error nodes.

* **`None`** means a hard failure with no tree to return. It is reserved for that case and is never a synthesized error placeholder.
* **A clean tree** (`has_errors()` is false) is a complete, editable `Document`. Empty input is not an error and not `None` — `""` parses to a `Document` holding a single empty root, the same legal state as `Document()`.
* **A partial tree** (`has_errors()` is true) preserved the unparseable parts as `Error` placeholder nodes. The tree is still structurally valid, but it is read-only.

Diagnostics and `Error` nodes are related but not identical. On the parser path, every `Error` node is emitted alongside at least one diagnostic — so if the tree `has_errors()`, the diagnostics list is non-empty. The converse does not hold: diagnostics and `Error` placeholders are separate channels, so a diagnostic does not by itself make an otherwise editable tree read-only.

The `Document` you get back is the working tree for everything downstream. Three behaviors define how you work with it:

* **Reads use node handles.** Navigation (`parent`, `children`, `next_sibling`, `ancestors`) and content accessors return lightweight read-only handles.
* **Edits return a result.** Mutations are validated eagerly and report a structured error at the point of misuse rather than panicking.
* **Trees with errors are read-only.** If `has_errors()`, every editing method fails with a read-only error. The only use for an error tree is inspection.

For the editing workflow, see [Edit a document](https://texform.dev/docs/edit-a-document); for the exact edit-method semantics and error variants, see the [Document API](https://texform.dev/api/python/document). The deeper tree model — the internal arena beneath `Document` and the `from_syntax`/`to_syntax` symmetry — is covered in [Architecture](https://texform.dev/docs/architecture).

- [Transforms](https://texform.dev/docs/transforms) — Normalize a parsed formula with profiles and the rule engine.

- [Serialization](https://texform.dev/docs/serialization) — Turn a Document back into LaTeX text or a structured snapshot.

- [Handle parse errors](https://texform.dev/docs/handle-parse-errors) — A complete pattern for the three parse-result states.

- [Glossary](https://texform.dev/docs/glossary) — SyntaxNode, Document, context item, lenient and strict — defined in one place.