DIFF-015: a tokenizer failure silently ERASES a file from the diff — every fold error masks as the "out full" blob-skip

Now: DONE
Sev: HIGH

A changed file whose bytes fail the native tokenizer VANISHES from be diff — bare and path-scoped both — with exit 0, no header, no warning, while be status still lists it dirty/staged. Chain: WEAVENext fails with JSTBAD, the jab binding collapses EVERY non-OK into the string weave.fold: failed (out full?) (jab/weave.hpp:189), and views/diff/diff.js:128 swallows anything containing "full" as the over-cap blob-skip. Found 2026-07-29 reviewing MARK-013: verbs/mark/render.js (+37 lines staged) was invisible in the wt diff and the work was nearly judged unchanged/discarded.

Input

Context

Trigger content: a JS regex literal with / INSIDE a character class — DOG-021 (JST ends the literal at that /, derails, JSTBAD). render.js's punct table row { re: /[[\]()!*_~|\\:.\-+#&<>{}=,;/'"@^]/y ... }` is one; the PRE-MARK-013 render.js carries the same class, so render.js has NEVER wt-diffed.
Repro (wt work/MARK-013/be): append that one regex line to any healthy tracked file → its whole diff renders EMPTY; without it, a 90-of-101-line rewrite of the same file diffs fine. Prefix-bisect of render.js flips exactly at that line. dogtok <file-with-that-line>.js → Error: JSTBAD.
Swallow site: diffFile (be/views/diff/diff.js:127-129) catch (e) { if (("" + e).includes("full")) return; throw e; } — intended for the DIFF-010 fixed-buffer overflow, but jab/weave.hpp:188-189 throws the SAME weave.fold: failed (out full?) for every WEAVENext failure, so JSTBAD/TOKBAD/any ok64 all match "full".
Same masked-skip class as DIFF-007 (C GRAF outtoks overflow → EMPTY) and DIFF-009 (tree diff drops files, exit 0): the diff view has no channel that says "this file changed but could not be rendered".
Operator harm: status and diff disagree with zero signal; the only truth was a manual byte-compare of cat:<path>?<base> vs the wt file. Second independent hit: DOG-021 Context logged the same render.js vanish in July (with STATUS-008).
Mechanism nuance from DOG-021: the derailed lexer opens a string that runs to EOF, so the fold can ALSO overflow the markup buffer for real — an error code alone cannot fully separate "over cap" from "lexer derail"; the header+notice must therefore cover BOTH skip paths.

Goals

A changed file NEVER disappears: when tokenize/fold fails, still emit the --- a/… +++ b/… header plus a one-line notice, and prefer a fallback plain LINE diff (no tokens needed) over hiding the change.
The binding forwards the real ok64 code (jab/weave.hpp:189 includes it in the thrown string); diff.js gates the blob-skip on the GENUINE over-cap code only, never on a substring of a catch-all message.
The notice is plain words per README norms (e.g. "cannot tokenize verbs/mark/render.js — showing plain line diff"), not a bare C code.

Constraints

The JST grammar fix itself (slash in a char class) stays DOG-021 ground — this ticket is the masking + fallback, and must help for ANY future lexer failure, not just this one.
Touches two repos: jab/weave.hpp (error text/code) and be views/diff/diff.js (gate + fallback); repro-first in the be tree's test arrangement.
DIFF-010's real over-cap skip must keep working (fold buffers are fixed-size; a genuinely over-dense source still blob-skips, but now says so on stderr).

WIP

Design decisions

Ruled (gritzko, 2026-07-29): on a fold failure the diff FALLS BACK to a plain-text line differ — header + plain-words notice + token-free line diff, never a silent skip. The grammar fix wt is //DOG-021-dog (DOG-021).
Ruled (gritzko, 2026-07-29): the fallback REUSES libdog's plain-text lexer TXTT — on a fold failure, retry the same fold with ext "" (DOG-021 measured the failing blob folds fine under ""/txt; dogtok on the poison line as .txt confirms). NO parallel differ: a first //DOG-021-be implementation with a hand-rolled Myers line differ was rm -rf-ed for exactly that.
The notice stays: header + plain words (e.g. "cannot tokenize <name> — diffing as plain text") with the TXTT-refolded hunks.
The jab/weave.hpp ok64-forwarding TODO is NOT in the 2026-07-29 order (be/ and jab/dog only) — stays open here.

TODOs

Repro: a be test with a tracked file carrying a slash-in-class regex — wt edit must yield a non-empty diff (header + fallback), watched failing first.
jab/weave.hpp: fold/merge throws carry the ok64 code, distinct over-cap vs tokenizer-fail. Moved to CFOLD-002: that throw is dead code, the live one is jab/cfold.hpp:180.
views/diff/diff.js: catch gates on the over-cap code; other failures → header + notice + plain line-diff fallback.
Sweep the other ".includes("full")"-style gates (merge/alive/produce callers) for the same masking. Moved to CFOLD-002 — get/checkout/patch/expected still mask, causing false conflicts.
Re-check DIFF-009's dropped-files repro once the notice channel exists — same class, may be the same mask. Stays DIFF-009's own ground (native C tree-diff).

Blockers and bummers

DOG-021 stays open meanwhile: until fixed, ANY file with a slash-in-class regex tokenizes JSTBAD — with the fallback it will at least show as a plain diff.

Outcome

da3cf823 (be): the diff-view fix landed under DOG-021's commit line — views/diff/diff.js TXTT refold plus test/diff/tokfail, both DIFF-015-commented.
Verified 2026-08-04 at ///be#014aa381: bare and path-scoped jab diff both show header + "cannot tokenize … — diffing as plain text" + plain hunks; DIFF-010's over-cap skip still passes.
Spun off CFOLD-002: the merge-side mask (get/checkout/patch/expected) and the ok64 forwarding, with the false-conflict evidence.