Parsing

Parsing LaTeX is hard not because of string manipulation but because the meaning of a command depends on knowing its argument structure. TeXForm parses against a knowledge base of command signatures, so a formula becomes a real syntax tree instead of a best-effort guess.

What parsing is#

Parsing turns LaTeX source into a Document — an editable tree — together with a list of diagnostics describing anything the parser could not handle.

import texform

result = texform.Parser().parse(r"\frac{x}{y}")
print(result["document"].to_latex())

\frac { x } { y }

import { Parser } from 'texform';

const result = new Parser().parse('\\frac{x}{y}');
console.log(result.document.toLatex());

\frac { x } { y }

Why parsing is knowledge-driven#

Argument structure is not something a grammar can infer from syntax alone. \frac{x}{y} takes two mandatory arguments, but nothing in the characters \frac says so — a generic LaTeX grammar cannot know that \frac consumes the next two groups while \alpha consumes none. The only way to parse \frac, \over, or \quantity correctly is to know each command's signature in advance.

This is why regular expressions and one-off heuristics keep breaking on real formulas: they guess at structure the source does not encode. TeXForm reads each command's signature from a knowledge base, so the parser binds arguments the way the command actually behaves rather than guessing.

Packages and context#

The knowledge base is organized into packages. Each package contributes a set of known command and environment signatures, and you choose which packages the parser consults. Listing them shows the whole catalog:

import texform

for pkg in texform.list_packages():
    print(pkg["name"], pkg["commands"], pkg["environments"])

ams 35 28
base 270 7
bboldx 12 0
boldsymbol 1 0
braket 11 0
color 5 0
physics 182 1
textmacros 78 0

import { listPackages } from 'texform';

for (const pkg of listPackages()) {
  console.log(pkg.name, pkg.commands, pkg.environments);
}

ams 35 28
base 270 7
bboldx 12 0
boldsymbol 1 0
braket 11 0
color 5 0
physics 182 1
textmacros 78 0

The default package set#

list_packages() lists the catalog, not the parser's default selection. A Parser or TransformEngine built without a knowledge base shares the default one, which loads the default set: base, ams, braket, textmacros, bboldx, boldsymbol, and color. physics is opt-in because it redefines commands that braket also defines, and gives standard commands such as \div, \Re, and \Im its own meanings. Loading both is allowed: TeXForm imports built-in packages in a fixed order, with physics overriding the shared command definitions from braket. Reordering the packages array does not change that precedence. Select the package whose command syntax your input uses.

The consequence is concrete. Under the default set, \ket resolves to the braket record, whose signature is m, and normalization keeps \ket{\psi} as written. The physics record for the same name has the signature s m — a starred variant plus one mandatory argument — and Corpus normalization expands \ket{\psi} into \left \vert \psi \right \rangle. If your corpus means physics, ask for it explicitly:

import texform

kb = texform.KnowledgeBase(packages=["base", "ams", "physics"])
parser = texform.Parser(knowledge_base=kb)
print(kb.lookup_command("ket", "math")["spec_string"])
print(kb.lookup_command("ket", "math")["from_packages"])

s m
['physics']

import { KnowledgeBase, Parser } from 'texform';

const knowledgeBase = new KnowledgeBase({ packages: ['base', 'ams', 'physics'] });
const parser = new Parser({ knowledgeBase });
console.log(knowledgeBase.lookupCommand('ket', 'math').specString);
console.log(JSON.stringify(knowledgeBase.lookupCommand('ket', 'math').fromPackages));

s m
["physics"]

Naming packages yourself replaces the default set rather than adding to it, so list every package you need, including base.

When a command is not in your selected packages, you can register it yourself by passing context items to the knowledge base the parser uses. A context item is a small record carrying a command name and an argspec; the parser treats it as if it were a built-in signature. See Inject custom commands for the full workflow.

Argument specifications#

A command's signature is written as an argspec — a compact, xparse-style string describing each argument the command consumes. The argspec is what tells the parser to read two groups after \frac and none after \alpha. It is the single mechanism behind knowledge-driven parsing.

The notation is small. A few common slots:

• m — one mandatory argument. \frac is m m (two mandatory groups).
• o — one optional bracketed argument. \sqrt is o m (an optional index, then the radicand).
• s — an optional leading star. With physics loaded, \ket is s m, so \ket*{\psi} parses with its star recorded.

You can validate an argspec string directly, which reports how many arguments it describes:

import texform

print(texform.validate_argspec("o m"))

{'valid': True, 'error': None, 'arg_count': 2, 'parsed': [{'required': False, 'no_leading_space': False, 'nullable': False, 'kind': {'type': 'content', 'mode': 'math'}, 'form': {'type': 'standard'}}, {'required': True, 'no_leading_space': False, 'nullable': False, 'kind': {'type': 'content', 'mode': 'math'}, 'form': {'type': 'standard'}}]}

import { validateArgspec } from 'texform';

const result = validateArgspec('o m');
console.log(result.valid, result.argCount);

true 2

The full notation — every slot kind, delimited forms, and TeXForm's extensions to xparse — is documented in the argspec reference. For why the notation is shaped the way it is and its known limits, see Argspec design.

Configuring strictness#

Two independent knobs on ParseConfig decide how the parser reacts to trouble, and for both of them True means stricter. reject_unknown controls what happens to a command the knowledge base does not recognize; abort_on_error controls what happens when the parser hits a syntax error it cannot continue past. Both default to False, so the default configuration is lenient on both axes. They combine into four behaviors:

The default corner is the most permissive, which is what you want when cleaning noisy input. The opposite corner — both knobs True — is what "strict parsing" means precisely: it is for input you expect to be clean and want to reject loudly. Rust names these two corners ParseConfig::STRICT and ParseConfig::LENIENT; Python and JavaScript express them through constructor defaults and per-call overlays instead, so there is no ParseConfig.LENIENT constant to import.

Lenient parsing recovers where it can, but recovery is not a promise of a tree. A formula whose structure the parser cannot close, such as \frac{a}{b, still comes back with document = None under the default configuration, while x^ comes back as a read-only tree holding an Error node. Treat "returned a tree" as something to check, not to assume — the three-state handling below is written for exactly that.

The two knobs are orthogonal to the result's shape: abort_on_error is a parse-strictness switch, while whether the resulting tree contains errors is a separate, queryable property of the tree.

For task-oriented recipes built on these settings, see Handle parse errors and Clean OCR/MER output.

What parsing gives you#

A parse result has exactly three honest states. TeXForm never fabricates a placeholder tree to pretend a document always exists:

• None means a hard failure with no tree to return. It is reserved for that case and is never a synthesized error placeholder.
• A clean tree (has_errors() is false) is a complete, editable Document. Empty input is not an error and not None — "" parses to a Document holding a single empty root, the same legal state as Document().
• A partial tree (has_errors() is true) preserved the unparseable parts as Error placeholder nodes. The tree is still structurally valid, but it is read-only.

Diagnostics and Error nodes are related but not identical. On the parser path, every Error node is emitted alongside at least one diagnostic — so if the tree has_errors(), the diagnostics list is non-empty. The converse does not hold: diagnostics and Error placeholders are separate channels, so a diagnostic does not by itself make an otherwise editable tree read-only.

The Document you get back is the working tree for everything downstream. Three behaviors define how you work with it:

• Reads use node handles. Navigation (parent, children, next_sibling, ancestors) and content accessors return lightweight read-only handles.
• Edits return a result. Mutations are validated eagerly and report a structured error at the point of misuse rather than panicking.
• Trees with errors are read-only. If has_errors(), every editing method fails with a read-only error. The only use for an error tree is inspection.

For the editing workflow, see Edit a document; for the exact edit-method semantics and error variants, see the Document API. The deeper tree model — the internal arena beneath Document and the from_syntax/to_syntax symmetry — is covered in Architecture.