Clean OCR/MER output

Output from an OCR or math-expression-recognition (MER) model is noisy in predictable ways: it invents commands that do not exist, truncates structures mid-script, and drops closing braces. You cannot normalize all of it, but you should not throw all of it away either — much of it is clean, and some of it is a repairable near-miss. The job is to sort each formula into a bucket before you spend effort on it.

The key tool is the parse result itself. Parse with reject_unknown enabled so that unrecognized commands surface as diagnostics instead of slipping through silently, then read the three honest parse states. A None document means there was no tree to recover and the formula is a discard. A tree where has_errors() is true is a partial, read-only tree: the unparseable parts survived as Error nodes, the diagnostics tell you what they were, and this is your repair queue. A clean tree is a survivor you normalize immediately.

import texform

batch = [
    r"\frac{a}{b}",        # clean
    r"a \over b",          # legacy syntax, still a clean tree
    r"\unknowncmd x",      # invented command -> Error node, tree preserved
    r"x^",                 # truncated superscript -> Error node, tree preserved
    r"\frac{a}{b",         # unbalanced braces -> no tree at all
]

parser = texform.Parser()
engine = texform.TransformEngine(profile="corpus")
config = texform.ParseConfig(reject_unknown=True)  # surface unknown commands

normalized = []
flagged = []
discarded = []

for src in batch:
    result = parser.parse(src, config)
    doc = result["document"]
    if doc is None:
        discarded.append(src)
    elif doc.has_errors():
        kinds = sorted({str(d.get("kind")) for d in result["diagnostics"]})
        flagged.append((src, ", ".join(kinds)))
    else:
        normalized.append(engine.normalize(src))

print("normalized (clean survivors):")
for line in normalized:
    print("  " + line)
print("flagged for repair (partial tree):")
for src, kinds in flagged:
    print(f"  {src!r} -> {kinds}")
print("discarded (no tree):")
for src in discarded:
    print(f"  {src!r}")

import { Parser, TransformEngine } from 'texform';

const batch = [
  '\\frac{a}{b}',     // clean
  'a \\over b',       // legacy syntax, still a clean tree
  '\\unknowncmd x',   // invented command -> Error node, tree preserved
  'x^',               // truncated superscript -> Error node, tree preserved
  '\\frac{a}{b',      // unbalanced braces -> no tree at all
];

const parser = new Parser();
const engine = new TransformEngine({ profile: 'corpus' });
const config = { rejectUnknown: true }; // surface unknown commands

const normalized = [];
const flagged = [];
const discarded = [];

for (const src of batch) {
  const result = parser.parse(src, config);
  const doc = result.document;
  if (doc === null) {
    discarded.push(src);
  } else if (doc.hasErrors()) {
    const kinds = [...new Set(result.diagnostics.map((d) => String(d.kind)))].sort();
    flagged.push([src, kinds.join(', ')]);
  } else {
    normalized.push(engine.normalize(src));
  }
}

console.log('normalized (clean survivors):');
for (const line of normalized) console.log('  ' + line);
console.log('flagged for repair (partial tree):');
for (const [src, kinds] of flagged) console.log(`  ${JSON.stringify(src)} -> ${kinds}`);
console.log('discarded (no tree):');
for (const src of discarded) console.log('  ' + JSON.stringify(src));

The clean formulas are normalized, the recoverable defects are flagged with the diagnostic kind that explains them, and only the truly broken input is discarded:

normalized (clean survivors):
  \frac { a } { b }
  \frac { a } { b }
flagged for repair (partial tree):
  '\\unknowncmd x' -> unknown-command
  'x^' -> raw-expected-found
discarded (no tree):
  '\\frac{a}{b'

The unknown-command kind names a missing signature you can often fix by registering the command. The truncated x^ is the other shape: raw-expected-found says the parser wanted something specific at that position and did not get it. expected names what it wanted — superscript content — while found is None, because the input ended rather than offering the wrong thing. An unclosed environment has its own category, and expected names the missing \end:

import texform

parser = texform.Parser()
config = texform.ParseConfig(reject_unknown=True)

print(parser.parse(r"x^", config)["diagnostics"][0])
print(parser.parse(r"\begin{matrix} a & b", config)["diagnostics"][0])

Output

{'kind': 'raw-expected-found', 'message': 'Missing superscript content', 'span': {'start': 2, 'end': 2}, 'expected': ['superscript content'], 'found': None, 'contexts': []}
{'kind': 'unclosed-environment', 'message': 'Environment matrix missing closing \\end{matrix}', 'span': {'start': 20, 'end': 20}, 'expected': ['\\end{matrix}'], 'found': None, 'contexts': []}

Bucket on kind when it is present and fall back to message and span when it is not, because kind is an optional field that only structured categories populate. An empty span sitting at the end of the source, as in both diagnostics above, is the signature of input that was cut off mid-structure. Treat a missing kind as "uncategorized", not as a stable identifier you can match on.

To read those diagnostics and their spans in full and decide repair versus discard precisely, see Handle parse errors; for what reject_unknown and the three parse states mean, see Parsing.

Next steps#

Handle parse errors

Dispatch on all three parse states and read diagnostics and spans.

Inject custom commands

Fix unknown-command diagnostics by registering the macro's signature.

Normalize a corpus

Run the survivors through one reused engine and tally what changed.

Troubleshooting

The other failures a recognition pipeline runs into, with real messages.