Beagle SCM

JS-099 HIGH: be/ JS must index every keeper pack it writes — unindexed packs are invisible to native's index-only lookup (false non-FF)

be post to a remote refused with a BOGUS non-fast-forward: native keeper KEEPLookup (keeper/KEEP.c) resolves a sha ONLY through the loaded .keeper.idx LSM runs and returns KEEPNONE on a miss — there is NO pack-scan fallback (Indices). The be/ JS writers do not keep that index complete: a shard with 12 NNNNN.keeper packs carried only 2 .keeper.idx runs (1942 entries), so the newest locally-committed tip was unindexed. KEEPIsAncestor(local, peer) then dead-ends at the missing tip object → native reads it as a non-descendant → KEEPFAIL/WIRECLNFF. jab reads the tip fine via store.js's in-RAM scan fallback; native has none. Fix: index packs INCREMENTALLY, one run per written pack. Method Issues.

Input

Context

• Repro: be post --nosub ssh://localhost/src/journal/be from a store-backed

  wt → keeper: post: non-fast-forward — local tip 4370d67… is not a descendant of peer tip 1f664aa…, then KEEPFAIL / BEDOGEXIT; 0 changes to commit. (?/beagle-ext form of the same wt fails identically as WIRECLNFF.)

• Verified split native-vs-JS: keeper get .#4370d67 → KEEPNONE (native,

  index-only); jab commit:?4370d67 prints it (JS scan fallback). The object IS on disk in a pack, absent from every .keeper.idx run. parent 9c52537 too.

• Store ~/.be/beagle-ext: 12 *.keeper, 2 *.keeper.idx (29440+1632 B =

  1942 wh128 entries). 6388c34e (also recent) PRESENT but 4370d67/9c525377 MISS — coverage is whatever the last native compaction folded, NOT every pack.

• ROOT CAUSE (verified, TEST-003): ingest.buildIndex names the run

  ron.encode(ron.now())+".keeper.idx" (ingest.js:66) — it IGNORES the unique fileId it is passed, so two packs written in ONE ron tick (rapid post, or a frozen SOURCE_DATE_EPOCH) collide and the 2nd OVERWRITES the 1st run. Minimal repro: 2 same-epoch posts → ONE idx → first commit commitTree MISSING; rm .be/*.keeper.idx (in-RAM re-scan) recovers it. THIS is the "12 packs → 2 runs".

• Native lookup is index-only BY DESIGN (Indices: "a miss is a true miss; no

  walk-up"); the WRITER holds the completeness contract. store.js scan-fallback (shared/store.js, "no .keeper.idx run → build in-RAM") hides the gap from jab but never from native be/keeper.

Goals

• Every keeper pack the be/ JS writes (post fold-commit, wire ingest

  clone/add, any barrier re-fold) gets a covering .keeper.idx run — no pack left unindexed, ever. Native keeper get .#<sha> resolves it without scanning.

• Incremental: append ONE run per new pack (Indices size-tiered LSM); do not

  rebuild the whole shard index; let compaction bound the run count.

• Repro-first: a store-backed be post of locally-committed JS tips

  fast-forwards (no false non-FF); native keeper get/verify find every tip.

Constraints

• be/ JS only (beagle-ext shard): shared/ingest.js, verbs/post/*.js. wh128

  layout per Indices / keeper/KEEP.h (key=hashlet60|type4, val=offset40|file_id20|flags4) plus the PACK-summary bookmark entry.

• Do NOT lean on the store.js scan fallback as the fix — native cannot scan.
• No edits to native keeper; CLAUDE.md §17 (repro-first), §3 (table tests).

WIP

Design decisions

• Indexing is a WRITE-side obligation, not a read-side repair: whoever emits a

  pack emits its run in the same op — the idx run IS the commit's visibility to native. store.js's scan fallback stays only as a safety net.

• PRIMARY fix (verified root cause): give each run a UNIQUE name — key it on the

  keeper fileId/logName (e.g. 0000000002.keeper.idx), which buildIndex already receives, NOT ron.now(). A .keeper.idx per .keeper is 1:1 and collision-free; the reader merges runs anyway. This alone stops the overwrite.

• Secondary: the verbs/post/post.js barrier-re-fold path may ALSO (re)write a

  pack without a fresh idx run — audit every pack-write site for a paired ingest.buildIndex. JS-077 (orphan keeper pack) is the sibling no-index failure mode; JS-056 is the read-side half.

TODOs

• ingest.buildIndex: name the run by logName/fileId, not ron.now() — the

  verified collision fix (see ROOT CAUSE); tolerate old <ron>.keeper.idx shards. (Re-verified UNFIXED 2026-07-07: be/shared/ingest.js:66 still ron.encode(ron.now()) + ".keeper.idx".)

• Repro test: TWO posts under a frozen SOURCE_DATE_EPOCH (same tick) → assert

  BOTH commits' trees resolve (no MISSING) with NO idx-drop — RED before the fix.

• Repro test: commit N objects via jab post into a store-backed wt, assert

  native keeper get .#<tip> resolves — RED before the fix.

• Audit pack-write sites (ingest.clone/add, fold-commit, barrier

  re-fold) → each pairs with a .keeper.idx run: verified 2026-07-07, ingest.js clone:80 / add:116 / fold-commit.js writePack:192 all call buildIndex per pack — the residual hole is the same-tick run-name collision.

• Make indexing incremental + idempotent: one run per pack, a re-fold replaces

  ITS OWN run not the world; honor Indices size-tiered run discipline.

• Backfill: a be/jab step (or native keeper-side rebuild) to index any

  pre-existing unindexed pack — the wild stores already in this state. (PACK-001 PIDXScan from_off — the reindex-on-open leaf — is drafted on the board; the jab runtime landed a js pack-log APPEND leaf under PACK-001.)

• Parity: native keeper get/verify/be post FF on a JS-built store green.

Blockers and bummers

• Pre-existing stores are already corrupt-by-omission (10 unindexed packs in the

  repro shard); fixing the writer does not retro-index them — needs the backfill TODO or a native shard reindex.

Outcome

• (pending) NB commit d80bdb5c ("TEST-003,JS-099,SUBS-046: js tests pass, no

  native use", 2026-07-06) MENTIONS this ticket but touched only test harnesses / discover.js / submount.js — it de-nativified the tests (removing the native-keeper oracle this bug needs), NOT the ingest.js writer; the root cause is still in place.