Beagle SCM

DOG-025: code inners are one fat 'H' token — tokenize them, but recognize no nested formatting

Ruling (gritzko, 2026-07-29): DOG-024 (dog 737d196b) made inline formatted ranges tokenize their text and recognize nested formatting; code is the exception it forgot — the inners of an inline code span AND of a ```` fenced code block must be TOKENIZED (word granularity), but NO nested formatting is recognized there: no spans, no links, no escapes open inside code.

Input

Context

• MKDTonCode (dog/tok/MKDT.c:48) emits the span body as ONE verbatim 'H' token — G open, H body, G close — never running the word layer over it.
• The MKDTLexer in-fence branch (dog/tok/MKDT.c:351) emits each fence-body LINE as one 'H' token, so a code block diffs/paints line-at-a-time, not word-at-a-time.
• FREEEmitCode (dog/tok/FREE.c.rl) is the comment-overlay twin with the same one-'H' body; FREEEmitSpan next to it already re-lexes formatted spans one level down.
• test/TOK01.c pins the fat shapes: ` ABC-123 → GHG ("code stays verbatim 'H', never re-lexed"), MDTCodeFenceTest, the DOG-024 fence cases in MKDTBlockQuadTest`.
• The spec keeps code lowest-nesting by design: "no backticks inside code" (wiki/StrictMark.mkd:38), code span is highest precedence (:33).

Goals

• A code span's body emits word-granularity tokens (the word/number/space/punct layer), with the */_/~/[/\ rules DEAD inside — ` x ` yields plain tokens, no 'G' span, no escape.
• A fenced block's body lines tokenize the same way; fence-open/close markup handling (MKDTEmitFence, fence_blank) stays as DOG-024 left it.
• FREEEmitCode mirrors the split so comments and .mkd pages tokenize code identically; C and the JS twin (render.js classify/paint path) agree case-for-case.

Constraints

• The body tokenization goes through a ragel machine (a restricted MKDT main, e.g. spans/escapes ripped out) — no hand-rolled byte scanning in MKDT.c (never ever do manual parsing in C).
• Repro-first: new TOK01 cases land before the grammar change; the DOG-024 cases that pin one-'H' bodies get amended to the new shapes in the same commit.
• Regenerate with ragel -C XXT.c.rl -o XXT.rl.c -L; ctest -j16 clean.

WIP

Design decisions

• Ruled (gritzko, 2026-07-29): every inner sub-token KEEPS the 'H' tag — word-granularity boundaries, unchanged code paint and overlay stickiness; natural-tags-inside (F in backticks) is rejected.
• Open: does the fence info string (MKDTEmitFence 'H' rest) tokenize too, or stay one token.

TODOs

• Repro TOK01 cases: ` a b , x , * , ABC-123 , fence body with x` and multi-word lines.
• Code-body ragel machine (word layer, no span/escape rules) callable from both MKDTonCode and the in-fence branch.
• FREEEmitCode uses the same machine (FREE.c.rl).
• Check the MDT.c (two-pass Markdown) code-span/fence twins for the same fat-'H' shape.
• JS/renderer parity (render.js classify, verbatim <code> bodies) executes in MARK-01313.
• Spec touch: wiki/StrictMark.mkd §Inline markup / §Code blocks — code inners are tokenized, formatting-blind.

Blockers and bummers

• (none known)

Outcome

• Landed in dog as 6d80f8a2 "DOG-025: tokenize the text in code sections" (2026-07-29): mkdtc/MKDTCodeBody machine, MKDTonCode + in-fence + FREEEmitCode routed through it, MKDTCodeInnerTest; 72/72 ctest.
• Not done here: MDT.c two-pass Markdown twin still emits fat 'H' spans/lines (own ticket if wanted); wiki/StrictMark.mkd spec touch outstanding; renderer parity landed via MARK-01313's wt.