# Normalize a corpus

Run one reused transform engine over a dataset, skipping bad formulas and aggregating rule statistics.

https://texform.dev/docs/normalize-a-corpus

Normalizing a corpus of LaTeX formulas is a batch job, and two facts shape the loop. First, building a `TransformEngine` compiles its rule plan, so you construct it once and reuse it across every formula rather than per item. Second, real corpora contain formulas the engine cannot normalize: an input that cannot produce a complete tree — a truncated `x^`, an unbalanced `\frac{a}{b` — makes `normalize` raise `ParseError`, which you catch and bucket instead of letting it stop the run.

The pattern below builds the engine once, normalizes each formula under the `corpus` profile, sends parse failures to a skip bucket, and tallies which rules fired across the batch so you can see what the normalization actually changed.

One knowledge-base default is worth knowing before you point this at real data. The default knowledge base loads the default runtime set — `base`, `ams`, `braket`, `textmacros`, `bboldx`, `boldsymbol`, and `color` — rather than the whole catalog. The catalog also includes the opt-in `physics` package. Without it, physics-only commands such as `\dv` are unknown and pass through unexpanded, and `\ket` keeps its `braket` meaning. The batch below contains a `physics` derivative, so its knowledge base names its packages; see [Troubleshooting](https://texform.dev/docs/troubleshooting) for how the two readings of `\ket` differ.

**Python**

```python
import texform
from collections import Counter

formulas = [
    r"a \over b",
    r"\dv{f}{x}",
    r"{\bf x} + {a \over b}",
    r"\cos{A}",
    r"x^",            # truncated: cannot produce a complete tree
    r"\frac{a}{b",    # unbalanced: parse returns no document
]

kb = texform.KnowledgeBase(packages=["base", "ams", "physics"])
engine = texform.TransformEngine(profile="corpus", knowledge_base=kb)  # build once, reuse

normalized = []
skipped = []
rule_counts = Counter()

for src in formulas:
    try:
        result = engine.normalize_with_report(src)
    except texform.ParseError:
        skipped.append(src)
        continue
    normalized.append(result["normalized"])
    for rule in result["report"]["rewrite"]["rules"]:
        if rule["applied_count"] > 0:
            rule_counts[rule["key"]] += rule["applied_count"]

print(f"normalized {len(normalized)} / {len(formulas)} formulas")
print(f"skipped {len(skipped)}: {skipped}")
for line in normalized:
    print(line)
for key, count in rule_counts.most_common():
    print(f"{key}: {count}")
```

**JavaScript**

```js
import { KnowledgeBase, TexformParseError, TransformEngine } from 'texform';

const formulas = [
  'a \\over b',
  '\\dv{f}{x}',
  '{\\bf x} + {a \\over b}',
  '\\cos{A}',
  'x^',           // truncated: cannot produce a complete tree
  '\\frac{a}{b',  // unbalanced: parse returns no document
];

const knowledgeBase = new KnowledgeBase({ packages: ['base', 'ams', 'physics'] });
const engine = new TransformEngine({ profile: 'corpus', knowledgeBase }); // build once, reuse

const normalized = [];
const skipped = [];
const ruleCounts = new Map();

for (const src of formulas) {
  let result;
  try {
    result = engine.normalizeWithReport(src);
  } catch (err) {
    if (err instanceof TexformParseError) { skipped.push(src); continue; }
    throw err;
  }
  normalized.push(result.normalized);
  for (const rule of result.report.rewrite.rules) {
    if (rule.appliedCount > 0) {
      ruleCounts.set(rule.key, (ruleCounts.get(rule.key) ?? 0) + rule.appliedCount);
    }
  }
}

console.log(`normalized ${normalized.length} / ${formulas.length} formulas`);
console.log(`skipped ${skipped.length}`);
for (const line of normalized) console.log(line);
for (const [key, count] of [...ruleCounts].sort((a, b) => b[1] - a[1])) {
  console.log(`${key}: ${count}`);
}
```

The two malformed inputs land in the skip bucket, the four survivors are normalized, and the rule tally shows exactly which rewrite rules carried the batch:

```text
normalized 4 / 6 formulas
skipped 2: ['x^', '\\frac{a}{b']
\frac { a } { b }
\frac { \mathrm { d } f } { \mathrm { d } x }
\mathbf { x } + \frac { a } { b }
\cos A
base/over-to-frac: 2
physics/dv-to-frac-d: 1
```

Catching `ParseError` only tells you a formula could not be normalized, not why; to split incomplete trees from input that produced no tree at all and decide which to repair, read [Clean OCR/MER output](https://texform.dev/docs/clean-ocr-mer-output), and for the meaning of each rule entry and report field see [Transforms](https://texform.dev/docs/transforms).

## Profile the corpus before you normalize it [#profile-the-corpus-before-you-normalize-it]

The rule tally tells you what normalization changed. A different question — what is actually *in* the corpus — is often the one you want first, when you are deciding which packages to load or whether a profile is even appropriate for the data. `count_targets` answers it per formula without building a tree you have to walk: it returns a dict of prefixed keys, `cmd:` for commands, `env:` for environments, and `char:` for named character commands.

> **Python only**
>
> `count_targets` is a Python-only corpus helper. There is no JavaScript equivalent; in JavaScript, walk the parsed document with `findCommands` and `descendants` instead.

```python
import texform
from collections import Counter

corpus = [
    r"\frac{x}{y} + \alpha",
    r"\dv{f}{x}",
    r"\begin{pmatrix} a & b \end{pmatrix}",
    r"\alpha + \beta",
]

totals = Counter()
for src in corpus:
    totals.update(texform.count_targets(src))

for key, count in sorted(totals.items(), key=lambda kv: (-kv[1], kv[0])):
    print(f"{key}: {count}")
```

```text
char:alpha: 2
char:beta: 1
cmd:dv: 1
cmd:frac: 1
env:pmatrix: 1
```

Like `normalize`, it needs a complete parse and raises `ParseError` otherwise, so the same skip bucket works here. Counting `physics` commands such as `dv` across a sample is a quick way to confirm a corpus really needs that package loaded before you commit to a normalization run over all of it.

## Next steps [#next-steps]

- [Clean OCR/MER output](https://texform.dev/docs/clean-ocr-mer-output) — Sort noisy recognition output into repairable and discardable buckets.

- [Transforms](https://texform.dev/docs/transforms) — What each profile targets and how to read the transform report.

- [Disable a rewrite rule](https://texform.dev/docs/disable-a-rule) — Turn off one rule by key when a profile normalizes more than you want.

- [Troubleshooting](https://texform.dev/docs/troubleshooting) — Config errors, package conflicts, and the other failures a batch run hits.