# Clean OCR/MER output

Classify math-recognition output by diagnostics into repairable and discardable buckets, then normalize the survivors.

https://texform.dev/docs/clean-ocr-mer-output

Output from an OCR or math-expression-recognition (MER) model is noisy in predictable ways: it invents commands that do not exist, truncates structures mid-script, and drops closing braces. You cannot normalize all of it, but you should not throw all of it away either — much of it is clean, and some of it is a repairable near-miss. The job is to sort each formula into a bucket before you spend effort on it.

The key tool is the parse result itself. Parse with `reject_unknown` enabled so that unrecognized commands surface as diagnostics instead of slipping through silently, then read the three honest parse states. A `None` document means there was no tree to recover and the formula is a discard. A tree where `has_errors()` is true is a partial, read-only tree: the unparseable parts survived as `Error` nodes, the diagnostics tell you what they were, and this is your repair queue. A clean tree is a survivor you normalize immediately.

**Python**

```python
import texform

batch = [
    r"\frac{a}{b}",        # clean
    r"a \over b",          # legacy syntax, still a clean tree
    r"\unknowncmd x",      # invented command -> Error node, tree preserved
    r"x^",                 # truncated superscript -> Error node, tree preserved
    r"\frac{a}{b",         # unbalanced braces -> no tree at all
]

parser = texform.Parser()
engine = texform.TransformEngine(profile="corpus")
config = texform.ParseConfig(reject_unknown=True)  # surface unknown commands

normalized = []
flagged = []
discarded = []

for src in batch:
    result = parser.parse(src, config)
    doc = result["document"]
    if doc is None:
        discarded.append(src)
    elif doc.has_errors():
        kinds = sorted({str(d.get("kind")) for d in result["diagnostics"]})
        flagged.append((src, ", ".join(kinds)))
    else:
        normalized.append(engine.normalize(src))

print("normalized (clean survivors):")
for line in normalized:
    print("  " + line)
print("flagged for repair (partial tree):")
for src, kinds in flagged:
    print(f"  {src!r} -> {kinds}")
print("discarded (no tree):")
for src in discarded:
    print(f"  {src!r}")
```

**JavaScript**

```js
import { Parser, TransformEngine } from 'texform';

const batch = [
  '\\frac{a}{b}',     // clean
  'a \\over b',       // legacy syntax, still a clean tree
  '\\unknowncmd x',   // invented command -> Error node, tree preserved
  'x^',               // truncated superscript -> Error node, tree preserved
  '\\frac{a}{b',      // unbalanced braces -> no tree at all
];

const parser = new Parser();
const engine = new TransformEngine({ profile: 'corpus' });
const config = { rejectUnknown: true }; // surface unknown commands

const normalized = [];
const flagged = [];
const discarded = [];

for (const src of batch) {
  const result = parser.parse(src, config);
  const doc = result.document;
  if (doc === null) {
    discarded.push(src);
  } else if (doc.hasErrors()) {
    const kinds = [...new Set(result.diagnostics.map((d) => String(d.kind)))].sort();
    flagged.push([src, kinds.join(', ')]);
  } else {
    normalized.push(engine.normalize(src));
  }
}

console.log('normalized (clean survivors):');
for (const line of normalized) console.log('  ' + line);
console.log('flagged for repair (partial tree):');
for (const [src, kinds] of flagged) console.log(`  ${JSON.stringify(src)} -> ${kinds}`);
console.log('discarded (no tree):');
for (const src of discarded) console.log('  ' + JSON.stringify(src));
```

The clean formulas are normalized, the recoverable defects are flagged with the diagnostic kind that explains them, and only the truly broken input is discarded:

```text
normalized (clean survivors):
  \frac { a } { b }
  \frac { a } { b }
flagged for repair (partial tree):
  '\\unknowncmd x' -> unknown-command
  'x^' -> raw-expected-found
discarded (no tree):
  '\\frac{a}{b'
```

The `unknown-command` kind names a missing signature you can often fix by registering the command. The truncated `x^` is the other shape: `raw-expected-found` says the parser wanted something specific at that position and did not get it. `expected` names what it wanted — `superscript content` — while `found` is `None`, because the input ended rather than offering the wrong thing. An unclosed environment has its own category, and `expected` names the missing `\end`:

```python
import texform

parser = texform.Parser()
config = texform.ParseConfig(reject_unknown=True)

print(parser.parse(r"x^", config)["diagnostics"][0])
print(parser.parse(r"\begin{matrix} a & b", config)["diagnostics"][0])
```

```text
{'kind': 'raw-expected-found', 'message': 'Missing superscript content', 'span': {'start': 2, 'end': 2}, 'expected': ['superscript content'], 'found': None, 'contexts': []}
{'kind': 'unclosed-environment', 'message': 'Environment matrix missing closing \\end{matrix}', 'span': {'start': 20, 'end': 20}, 'expected': ['\\end{matrix}'], 'found': None, 'contexts': []}
```

Bucket on `kind` when it is present and fall back to `message` and `span` when it is not, because `kind` is an optional field that only structured categories populate. An empty span sitting at the end of the source, as in both diagnostics above, is the signature of input that was cut off mid-structure. Treat a missing `kind` as "uncategorized", not as a stable identifier you can match on.

> **reject_unknown and has_errors() are independent**
>
> Neither implies the other. `reject_unknown` is a parser knob that decides whether an unrecognized name becomes a diagnostic or survives as a `known: false` node; `has_errors()` is a property of the tree the parser produced. Turning `reject_unknown` on does not guarantee an error tree, and a tree can have errors with `reject_unknown` off — `x^` above does exactly that. Check the tree when you want to know if it is usable, and the config when you want to know how strict the parse was.

To read those diagnostics and their spans in full and decide repair versus discard precisely, see [Handle parse errors](https://texform.dev/docs/handle-parse-errors); for what `reject_unknown` and the three parse states mean, see [Parsing](https://texform.dev/docs/parsing).

## Next steps [#next-steps]

- [Handle parse errors](https://texform.dev/docs/handle-parse-errors) — Dispatch on all three parse states and read diagnostics and spans.

- [Inject custom commands](https://texform.dev/docs/inject-custom-commands) — Fix unknown-command diagnostics by registering the macro's signature.

- [Normalize a corpus](https://texform.dev/docs/normalize-a-corpus) — Run the survivors through one reused engine and tally what changed.

- [Troubleshooting](https://texform.dev/docs/troubleshooting) — The other failures a recognition pipeline runs into, with real messages.