Normalize a corpus

Normalizing a corpus of LaTeX formulas is a batch job, and two facts shape the loop. First, building a TransformEngine compiles its rule plan, so you construct it once and reuse it across every formula rather than per item. Second, real corpora contain formulas the engine cannot normalize: an input that cannot produce a complete tree — a truncated x^, an unbalanced \frac{a}{b — makes normalize raise ParseError, which you catch and bucket instead of letting it stop the run.

The pattern below builds the engine once, normalizes each formula under the corpus profile, sends parse failures to a skip bucket, and tallies which rules fired across the batch so you can see what the normalization actually changed.

One knowledge-base default is worth knowing before you point this at real data. The default knowledge base loads the default runtime set — base, ams, braket, textmacros, bboldx, boldsymbol, and color — rather than the whole catalog. The catalog also includes the opt-in physics package. Without it, physics-only commands such as \dv are unknown and pass through unexpanded, and \ket keeps its braket meaning. The batch below contains a physics derivative, so its knowledge base names its packages; see Troubleshooting for how the two readings of \ket differ.

import texform
from collections import Counter

formulas = [
    r"a \over b",
    r"\dv{f}{x}",
    r"{\bf x} + {a \over b}",
    r"\cos{A}",
    r"x^",            # truncated: cannot produce a complete tree
    r"\frac{a}{b",    # unbalanced: parse returns no document
]

kb = texform.KnowledgeBase(packages=["base", "ams", "physics"])
engine = texform.TransformEngine(profile="corpus", knowledge_base=kb)  # build once, reuse

normalized = []
skipped = []
rule_counts = Counter()

for src in formulas:
    try:
        result = engine.normalize_with_report(src)
    except texform.ParseError:
        skipped.append(src)
        continue
    normalized.append(result["normalized"])
    for rule in result["report"]["rewrite"]["rules"]:
        if rule["applied_count"] > 0:
            rule_counts[rule["key"]] += rule["applied_count"]

print(f"normalized {len(normalized)} / {len(formulas)} formulas")
print(f"skipped {len(skipped)}: {skipped}")
for line in normalized:
    print(line)
for key, count in rule_counts.most_common():
    print(f"{key}: {count}")

import { KnowledgeBase, TexformParseError, TransformEngine } from 'texform';

const formulas = [
  'a \\over b',
  '\\dv{f}{x}',
  '{\\bf x} + {a \\over b}',
  '\\cos{A}',
  'x^',           // truncated: cannot produce a complete tree
  '\\frac{a}{b',  // unbalanced: parse returns no document
];

const knowledgeBase = new KnowledgeBase({ packages: ['base', 'ams', 'physics'] });
const engine = new TransformEngine({ profile: 'corpus', knowledgeBase }); // build once, reuse

const normalized = [];
const skipped = [];
const ruleCounts = new Map();

for (const src of formulas) {
  let result;
  try {
    result = engine.normalizeWithReport(src);
  } catch (err) {
    if (err instanceof TexformParseError) { skipped.push(src); continue; }
    throw err;
  }
  normalized.push(result.normalized);
  for (const rule of result.report.rewrite.rules) {
    if (rule.appliedCount > 0) {
      ruleCounts.set(rule.key, (ruleCounts.get(rule.key) ?? 0) + rule.appliedCount);
    }
  }
}

console.log(`normalized ${normalized.length} / ${formulas.length} formulas`);
console.log(`skipped ${skipped.length}`);
for (const line of normalized) console.log(line);
for (const [key, count] of [...ruleCounts].sort((a, b) => b[1] - a[1])) {
  console.log(`${key}: ${count}`);
}

The two malformed inputs land in the skip bucket, the four survivors are normalized, and the rule tally shows exactly which rewrite rules carried the batch:

normalized 4 / 6 formulas
skipped 2: ['x^', '\\frac{a}{b']
\frac { a } { b }
\frac { \mathrm { d } f } { \mathrm { d } x }
\mathbf { x } + \frac { a } { b }
\cos A
base/over-to-frac: 2
physics/dv-to-frac-d: 1

Catching ParseError only tells you a formula could not be normalized, not why; to split incomplete trees from input that produced no tree at all and decide which to repair, read Clean OCR/MER output, and for the meaning of each rule entry and report field see Transforms.

Profile the corpus before you normalize it#

The rule tally tells you what normalization changed. A different question — what is actually in the corpus — is often the one you want first, when you are deciding which packages to load or whether a profile is even appropriate for the data. count_targets answers it per formula without building a tree you have to walk: it returns a dict of prefixed keys, cmd: for commands, env: for environments, and char: for named character commands.

import texform
from collections import Counter

corpus = [
    r"\frac{x}{y} + \alpha",
    r"\dv{f}{x}",
    r"\begin{pmatrix} a & b \end{pmatrix}",
    r"\alpha + \beta",
]

totals = Counter()
for src in corpus:
    totals.update(texform.count_targets(src))

for key, count in sorted(totals.items(), key=lambda kv: (-kv[1], kv[0])):
    print(f"{key}: {count}")

char:alpha: 2
char:beta: 1
cmd:dv: 1
cmd:frac: 1
env:pmatrix: 1

Like normalize, it needs a complete parse and raises ParseError otherwise, so the same skip bucket works here. Counting physics commands such as dv across a sample is a quick way to confirm a corpus really needs that package loaded before you commit to a normalization run over all of it.

Next steps#