Beagle SCM

JS-092 MED: hunk.feed writes the TOK TLV at 1/4 the byte length

js/hunk.hpp JABChunkFeed sets the toks slice END pointer with token-COUNT byte arithmetic instead of BYTE length: hk.toks[1] = (tok32c*)(toks[0] + $len(toks)/sizeof(tok32)). toks[0] is a u8*, so + N advances N BYTES, but $len(toks)/sizeof(tok32) is the token COUNT — so the TOK TLV is written with numtokens bytes, a QUARTER of the real numtokens*4. Result: a fed hunk with N tok32 spans drains back only N/4 toks (rounded), and HUNKu8sDrain FAILS outright (next() → false, the record vanishes) unless 4 | N — because hunk_drain_toks rejects a 'K' length not a multiple of sizeof(tok32). Surfaced fixing COMMIT-003. Method: Issues.

Input

Context

• js/hunk.hpp:62-63 `hk.toks0 = (tok32c*)toks0; hk.toks1 =

  (tok32c)(toks0 + $len(toks) / sizeof(tok32));` — END = start + COUNT bytes; should be start + COUNTsizeof(tok32) (= the toks byte length).

• Repro (jab): feed a HUNK record with 4 valid monotonic spans, next() then

  log.toks.length → 1 (fed 4); feed 8 → 2. Feed 1/2/3/5/15 → next() false (drain rejects the non-4-multiple 'K' length, dog/HUNK.c:152 HUNKTOKLEN).

• dog/HUNK.c HUNKu8sFeed (the C producer) is CORRECT — it builds tkb from

  hk->toks as a typed tok32cs slice, so $len*sizeof is implicit; only the JS binding's manual end-pointer math is wrong.

• Live impact masked: cat/blob/grep feed toks ONLY in --color (plain

  feeds EMPTY toks), and the --color HUNK render is not byte-asserted — so the plain parity suite never exercised a non-empty toks round-trip until commit:.

Goals

• hunk.feed(uri, text, toks) round-trips ALL N spans: next() succeeds for

  any N and log.toks.length === N, with each span's value preserved.

• Repro under js/test/hunk.js: feed K spans, assert next() + `toks.length

  == K` + the span values; cover K not a multiple of 4.

Constraints

• js/hunk.hpp only (the binding); no dog/HUNK.c change (the C side is

  right). Use the typed-slice length, not manual pointer arithmetic (CLAUDE.md §10). JABChunkDogenize (:98-99) builds toks from a u32* pair correctly — mirror that.

WIP

Design decisions

• FIXED: hk.toks[1] = (tok32c*)toks[1] (toks is a u8s, so toks[1] already

  IS the byte-end — JABCBytesOf sets out[1] = base+off+len). $len(hk.toks) then equals the token count; K=1,2,3,4,7 all round-trip.

• SECOND bug found mid-fix (co-fixed here, surfaced by COMMIT-005's commit:

  pager once toks drained): the empty-URI _hunk_uri getter computed NULL - buf[0] (an empty uri/text is never written to the TLV, HUNK.c:129), so JABCSubU8 threw RangeError: Length out of range of buffer. New helper JABChunkField returns an empty Uint8Array for an empty/NULL field (uri+text).

TODOs

• Repro-first in js/test/hunk.js: feed K=1,2,3,4,7 spans; assert next() +

  toks.length == K + values. RED pre-fix (next() false at K=1), GREEN post-fix.

• Fix the END pointer to the byte length + the empty-field getter; JABC + ctest

  re-run 377/377 green.

Blockers and bummers

• Built in an isolated ~/todo/COMMIT-005 worktree (the main beagle tree is

  occupied by DIS-053). The be/ plain-mode EMPTY-toks workaround (COMMIT-003) stays — it keeps plain output colour-free, independent of this fix.

Outcome

• PENDING-COMMIT (co-landed with COMMIT-005; human gates). Changed:

  js/hunk.hpp (toks byte-end + empty uri/text getter), js/test/hunk.js (the K-span repro). C side (dog/HUNK.c) untouched — the producer was correct.