Analyze formula structure at scale

Counting which commands a corpus uses, how deep its formulas nest, or which optional arguments people actually fill is a bulk question, and walking node handles one formula at a time is the wrong shape for it. to_columnar() (toColumnar() in JavaScript) exports a document as two flat tables of plain columns: one row per node, one row per argument slot. The columns are ordinary lists, so you can loop over them directly or hand them to Arrow and DataFrame tools without copying a tree across the language boundary for every node.

What the tables contain#

The node table holds one row per node in preorder, so a parent always precedes its children, and detached nodes are left out. Row indices are local to the snapshot, and -1 marks a missing row or index.

Column	Meaning
`kind`	Node kind, such as `Command`, `Char`, `Scripted`, or `Environment`.
`parent`	Row of the parent node, or `-1` for the root.
`slot`, `slot_index`	Where the node sits in its parent: `child`, `arg`, `script_base`, `subscript`, `superscript`, `infix_left`, `infix_right`, or `env_body`, plus the index within that slot.
`depth`, `mode`	Nesting depth and the content mode (`math` or `text`) the node lives in.
`name`, `value`	The command or environment name, or the character or text value.
`known`	Whether the knowledge base knows the command or environment, or `None` for other kinds.
`group_kind`, `left`, `right`	For groups, whether they are explicit, implicit, delimited, or inline math, and the `\left` and `\right` delimiters.

The argument table has one row per slot of every command, declarative, infix, and environment, ordered by owner row and slot index. Its owner column is a row of the node table, content is the node-table row of the argument's content (or -1), and present tells you whether an optional slot was filled. form is the slot form (mandatory, optional, star, group, delimited, or paired), value_kind says what the slot carries, and value, open, and close hold literal values and paired boundaries. Field names stay snake_case in both languages. An incomplete document with Error nodes is still exportable.

Count commands and argument forms#

The loop below parses a few formulas and counts command names, unknown commands, and how often each command receives its optional argument. Rows in the argument table join back to the node table through owner, which is how an argument row finds its command name.

Python

from collections import Counter

import texform

formulas = [
    r"\frac{a}{b} + \sqrt[3]{x^2}",
    r"\sqrt{y} + \foo{x}",
    r"\mathbb{R}^n \to \mathbb{R}",
    r"\frac{1}{2} \sqrt[n]{z}",
]

parser = texform.Parser()
commands = Counter()       # command name -> occurrences
unknown = Counter()        # names the knowledge base does not know
optional_used = Counter()  # command name -> filled optional slots
max_depth = 0

for src in formulas:
    doc = parser.parse(src)["document"]
    if doc is None or doc.has_errors():
        continue
    tree = doc.to_columnar()
    nodes, args = tree["nodes"], tree["args"]

    for kind, name, known in zip(nodes["kind"], nodes["name"], nodes["known"]):
        if kind == "Command":
            commands[name] += 1
            if not known:
                unknown[name] += 1

    # Argument rows point back to their owner row in the node table.
    for owner, form, present in zip(args["owner"], args["form"], args["present"]):
        if form == "optional" and present:
            optional_used[nodes["name"][owner]] += 1

    max_depth = max(max_depth, max(nodes["depth"]))

print("commands     :", dict(commands))
print("unknown      :", dict(unknown))
print("optional used:", dict(optional_used))
print("max depth    :", max_depth)

JavaScript

import { Parser } from 'texform';

const formulas = [
  '\\frac{a}{b} + \\sqrt[3]{x^2}',
  '\\sqrt{y} + \\foo{x}',
  '\\mathbb{R}^n \\to \\mathbb{R}',
  '\\frac{1}{2} \\sqrt[n]{z}',
];

const parser = new Parser();
const commands = new Map();
const unknown = new Map();
const optionalUsed = new Map();
const bump = (map, key) => map.set(key, (map.get(key) ?? 0) + 1);
let maxDepth = 0;

for (const src of formulas) {
  const doc = parser.parse(src).document;
  if (doc === null || doc.hasErrors()) continue;
  const { nodes, args } = doc.toColumnar();

  nodes.kind.forEach((kind, row) => {
    if (kind === 'Command') {
      bump(commands, nodes.name[row]);
      if (!nodes.known[row]) bump(unknown, nodes.name[row]);
    }
  });

  // Argument rows point back to their owner row in the node table.
  args.owner.forEach((owner, i) => {
    if (args.form[i] === 'optional' && args.present[i]) bump(optionalUsed, nodes.name[owner]);
  });

  maxDepth = Math.max(maxDepth, ...nodes.depth);
}

console.log('commands     :', JSON.stringify(Object.fromEntries(commands)));
console.log('unknown      :', JSON.stringify(Object.fromEntries(unknown)));
console.log('optional used:', JSON.stringify(Object.fromEntries(optionalUsed)));
console.log('max depth    :', maxDepth);

Output

commands     : {"frac":2,"sqrt":3,"foo":1,"mathbb":2,"to":1}
unknown      : {"foo":1}
optional used: {"sqrt":2}
max depth    : 3

In JavaScript the same program prints objects such as { frac: 2, sqrt: 3, foo: 1, mathbb: 2, to: 1 }. Row indices restart at zero for every document, so keep per-document joins inside the loop, as above, and aggregate only the results across documents.

Optional: Arrow tables#

In Python each table passes directly to pyarrow.table. TeXForm does not depend on pyarrow, so this needs pip install pyarrow and is only a convenience for people who already use Arrow or DataFrames.

import pyarrow as pa
import texform

doc = texform.Parser().parse(r"\frac{a}{b} + \sqrt[3]{x^2}")["document"]
tree = doc.to_columnar()

nodes = pa.table(tree["nodes"])
print(nodes.group_by("kind").aggregate([("depth", "max")]).sort_by("kind").to_pylist())
print(pa.table(tree["args"]).select(["owner", "index", "form", "present"]).to_pylist()[:2])

# Spans are keyed by path, not by table row.
node = doc.find_commands("sqrt")[0]
spans = {entry["id"]: entry["span"] for entry in doc.node_spans()}
print(node.path(), spans[node.path()])

Output

[{'kind': 'Char', 'depth_max': 3}, {'kind': 'Command', 'depth_max': 1}, {'kind': 'Root', 'depth_max': 0}, {'kind': 'Scripted', 'depth_max': 2}]
[{'owner': 1, 'index': 0, 'form': 'mandatory', 'present': True}, {'owner': 1, 'index': 1, 'form': 'mandatory', 'present': True}]
root.child.2 {'start': 14, 'end': 27}

Choose the right export#

Use to_columnar() when you aggregate across many formulas: frequencies, depth statistics, argument-form counts, or feature extraction into a DataFrame. Use to_syntax() when you need a nested tree you can serialize to JSON, send over a process boundary, or rebuild with Document.from_syntax. Use to_latex() when you need text.

The tables carry no source positions. Spans come from node_spans(), which returns byte spans keyed by the same rooted path that node.path() reports, as the last lines above show. Spans exist only for nodes parsed from source, and they are UTF-8 byte offsets as described in Handle parse errors.

Next steps#

Document API — to_columnar, to_syntax, node_spans, and the ColumnarTree, NodeTable, and ArgumentTable types.
Node API — Paths, slots, and argument reads for a single node.
Normalize a corpus — Normalize formulas in bulk before you analyze them.
Edit a document — Change the tree once you know which nodes to target.