Handle parse errors

A parse result has exactly three honest states, and robust code dispatches on all three rather than assuming a document always exists. The document is None when the parser had nothing to recover; it is present but has_errors() when recovery preserved the unparseable parts as read-only Error nodes; and it is present and clean otherwise. Each state carries different information: None is explained entirely by the diagnostics list, an error tree exposes the failing spans through its Error nodes, and a clean tree is ready to serialize or transform.

The function below handles every state in one pass. For the no-tree case it reads the first diagnostic's message and span; for the partial case it reads the span off each Error node; for the clean case it serializes the tree back to LaTeX.

import texform


def classify(parser, src):
    result = parser.parse(src)
    doc = result["document"]
    if doc is None:
        # Hard failure: no tree. Diagnostics describe what went wrong.
        first = result["diagnostics"][0]
        return f"NO TREE   {src!r}: {first['message']} at {first['span']}"
    if doc.has_errors():
        # Partial, read-only tree. Error nodes carry the unparseable spans.
        spans = [n.span() for n in doc.errors()]
        return f"PARTIAL   {src!r}: {len(doc.errors())} error node(s) at {spans}"
    # Clean, editable tree.
    return f"CLEAN     {src!r}: {doc.to_latex()}"


parser = texform.Parser()
for src in [r"\frac{x}{y}", r"x^", r"\frac{a}{b", r""]:
    print(classify(parser, src))

import { Parser } from 'texform';

function classify(parser, src) {
  const result = parser.parse(src);
  const doc = result.document;
  if (doc === null) {
    // Hard failure: no tree. Diagnostics describe what went wrong.
    const first = result.diagnostics[0];
    return `NO TREE   ${JSON.stringify(src)}: ${first.message} at ${JSON.stringify(first.span)}`;
  }
  if (doc.hasErrors()) {
    // Partial, read-only tree. Error nodes carry the unparseable spans.
    const spans = doc.errors().map((n) => n.span());
    return `PARTIAL   ${JSON.stringify(src)}: ${doc.errors().length} error node(s) at ${JSON.stringify(spans)}`;
  }
  // Clean, editable tree.
  return `CLEAN     ${JSON.stringify(src)}: ${doc.toLatex()}`;
}

const parser = new Parser();
for (const src of ['\\frac{x}{y}', 'x^', '\\frac{a}{b', '']) {
  console.log(classify(parser, src));
}

Each input lands in a distinct branch, and empty input is a clean tree rather than a failure:

CLEAN     '\\frac{x}{y}': \frac { x } { y }
PARTIAL   'x^': 1 error node(s) at [{'start': 0, 'end': 2}]
NO TREE   '\\frac{a}{b': unclosed brace argument at {'start': 8, 'end': 9}
CLEAN     '':

A diagnostic is a record with a message, a kind (a stable string such as unknown-command, or absent for some structural errors), and a span locating it in the source. The span on an Error node and the span on a diagnostic both point at the trouble, so either can drive a highlight in your own tooling.

Spans are UTF-8 byte offsets#

A span is a half-open range of UTF-8 byte offsets — not Python code-point indices, and not JavaScript UTF-16 indices. On pure ASCII the distinction never shows up, which is exactly why it bites later. Any non-ASCII character earlier in the formula shifts every following span:

import texform

src = "\\alpha β x^"
result = texform.Parser().parse(src)

print("characters:", len(src), "bytes:", len(src.encode()))
print("error span:", result["document"].errors()[0].span())
print("slice by byte:", src.encode()[10:12])

characters: 11 bytes: 12
error span: {'start': 10, 'end': 12}
slice by byte: b'x^'

The β occupies two bytes, so the span that covers x^ reads 10..12 even though those characters start at index 9 of the Python string. Index into the encoded bytes and decode the slice, rather than indexing the string directly.

Choosing strictness#

Two independent knobs decide which states you ever see. Both default to lenient — reject_unknown and abort_on_error are each False unless you say otherwise:

The knobs are orthogonal, and neither one is equivalent to the tree's has_errors() signal. \frac{a}{b in the run above returned no tree under the default lenient settings, which is why dispatching on all three states matters more than picking the right strictness.

For the full four-quadrant behavior of these two knobs and the parse-state model behind them, see Parsing; for sorting a real recognition batch into repair and discard buckets with these diagnostics, see Clean OCR/MER output.

Next steps#