Beagle SCM

JS-023 LOW: generic tokenizer — tok.parse(bytes, lang) → tok32 + cursor

Drive the ragel dog tokenizers from JS: (bytes, lang) → a tok32 Uint32Array (zero-copy), with a pure-JS cursor to read tag + spans. One thin native leaf over the already-standalone dogenize core. Method: Issues.

Input

Context

• js/uri.cpp binds URILexer; HUNK.dogenize exists. The dogenize CORE

  HUNKu32bTokenize (dog/HUNK.c:1342) is ALREADY standalone — it writes a bare tok32 buffer and dispatches ~80 langs by extension via TOKLexer.

• tok32 = one u32: tag 31..27, custom 26, side 25..24, cumulative END

  offset 23..0; decoders tok32Tag/Offset/Side (dog/tok/TOK.h). Token i's start = token i-1's end.

• Uint32Array-over-a-buf is the cont.cpp idiom (zero-copy when 4-aligned).

Goals

• tok.parse(bytes, lang, out?) → the tok32 stream as a Uint32Array.

  Optional out buffer RECEIVES the results (reuse it across calls to avoid per-call allocation, the delt.apply(…,out) convention); tok.parse makes its own when out is omitted. lang = an extension (reuse the dispatch).

• A pure-JS TokStream cursor: .tag / .start / .end / .side /

  .custom / .text(src) (zero-copy subarray) / .str() / .next / .seek.

Constraints

• Native leaf(s) over the shared core HUNKu32bTokenize (no parse-logic dup;

  HUNK.dogenize fast path untouched). The out path writes packed tok32 into a caller region (a Buf's IDLE, advance via fed(n*4)), mirroring delt.apply/pack.inflate; the no-out path allocates a fresh aligned array. Keep the returned Uint32Array zero-copy — mind 4-byte alignment of the out write position (a fresh/reset Buf is aligned); if a reused-out alignment case forces a copy-vs-require decision, STOP and ask.

• Reject source > 16 MiB (24-bit offset cap → HUNKTOKOOB); the cursor PINS

  srcBytes (offsets snapshot positions, not bytes); empty input → empty array. Decode is pure-JS bit math over the Uint32Array.

• Do NOT fold uri/WIRE/DPATH in — they don't emit tok32. tok.parse

  is the JS face of dog/tok's TOKLexer ONLY (bespoke parsers stay bespoke).

WIP

Design decisions

• (open) tok.parse + TokStream in an embedded R"JS(...)" bundle (like

  uri.cpp); _tok_parse in js/tok.hpp, installed after JABCContInstall.

• (open) Optional thin selectors tok.knownLang/langs/sameLang over

  TOKKnownExt/TOKExtAt/TOKSameLexer — phase 2.

TODOs

• First be get --nosub onto current trunk (JS-021 landed); build.
• Repro-first: js/test/tok.js (model on hunk.js) — tokenize a "js"

  snippet, assert tag/start/end/text for known tokens; a >16 MiB source throws; empty → empty. Red before, green after.

• Add _tok_parse + the TokStream cursor; doc INDEX.md / API.md.

Blockers

• None.

Outcome

• LANDED 012d4b3d (2026-06-21): tok.parse(bytes, lang, out?) → tok32

  Uint32Array over the shared HUNKu32bTokenize core (no parse-logic dup; HUNK.dogenize fast path untouched). With out (a Buf): lex into IDLE via _tok_parse_into → fed(n*4) → zero-copy view (require 4-aligned out, else throw); without out: fresh trimmed array. Pure-JS TokStream cursor (tag/start/end/side/custom/text/str/next/seek). Guards: >16 MiB throw, empty→empty, unknown lang→plain-text. js/test/tok.js (JABCtok) red→green; ctest -R JABC 19/19, ASAN/LSan clean.