Beagle SCM

JS-056 PERF: store.js mmap the on-disk keeper.idx instead of rebuilding the object index every open

be/lib/store.js index() builds an in-RAM abc.index("wh128",{mem}) by pk.scan/indexPackByWalk over EVERY pack object — and since a pack is offset-addressed, that forces resolving+SHA-1ing all objects (~11 s / 2.5 GB on the journal's 40 MB pack; this is why jab status looked hung). The keeper ALREADY persists that exact map on disk as NNNNN.keeper.idx LSM/SST runs (Indices). locate() should mmap+range those runs, not rebuild. Proven: abc.index("wh128",{dir,ext:"keeper.idx"}) opens the runs and a prefix range returns 707bbbb instantly. JS-030 follow-up. Method: Issues.

Input

Context

• store.js index() uses the in-RAM ({mem}) variant and pipes pk.scan

  entries in; on a thin/oversized object scan throws → indexPackByWalk resolves+hashes every object. O(whole store) on every store.open.

• The on-disk keeper.idx (val offset40|file_id20|flags4, Indices) is the

  designed read path: "mmap every run and binary-search each", newest-wins.

• Verified decode: 707bbbb on-disk val 669425078370321 → `offset = val>>24

  = 39 900 844 (matches native locate); file_id=(val>>4)&0xfffff`.

Goals

• locate() opens/mmaps the on-disk keeper.idx runs ({dir,ext}), ranges

  [h<<4, h<<4|0xf], decodes the on-disk val layout — NO startup scan/rebuild.

• getObject/readRecord unchanged: resolve the single hit on demand.
• Fall back to the current scan-build ONLY when no .keeper.idx run exists.

Constraints

• be/lib/store.js only (beagle-ext shard). Newest-wins across runs (LSM).

  Repro-first (CLAUDE.md §17); be-bin/be-js status parity green; no scan on a store that has an idx.

WIP

Design decisions

• Open via abc.index("wh128",{dir:shard, ext:"keeper.idx"}); decode

  offset=val>>24, file_id=(val>>4)&0xfffff (NOT store.js's in-RAM fileIdx<<40|offset). Keep the in-RAM build as a no-idx fallback only.

TODOs

• Repro: store.open(...).getObject(commit) must NOT scan the pack on a store

  that has a keeper.idx (assert bounded / ~instant, not O(objects)).

• locate()/index(): mmap the on-disk runs + range + on-disk val decode.
• No-idx fallback path; update the store.js header comment.
• Verify jab status on the journal store is fast (no 11 s / 2.5 GB).

Blockers and bummers

• Related to JS-055 (resolve scratch): lazy lookup stops resolving the 46 MB

  blob per-run, but JS-055 is still needed when a >pack object IS resolved.

Outcome

• locate() now mmaps the on-disk keeper.idx runs via

  abc.index("wh128",{dir:shard,ext:"keeper.idx"}) (new diskIndex()), ranges [h<<4,h<<4|0xf], decodes the ON-DISK val offset40|file_id20|flags4 (offset=val>>24, file_id=(val>>4)&0xfffff) and maps file_id→pack via fileIdToPackIdx. In-RAM scan-build kept as the no-idx fallback (the two val layouts decode under one onDisk flag). getObject/readRecord untouched.

• LANDED TWICE — the JS lib is mirrored in two shards: c4c912e9 beagle

  bin/lib/store.js (+ test/js/store.js, the JABCstore ctest), and 3cad15fb *beagle-ext* be/lib/store.js — the latter is the copy jab status actually runs (the upward be/-scan resolves ~/beagle-journal/be/, NOT beagle/bin). The first land alone did NOT speed up jab status — the ticket's "beagle-ext shard" constraint was the operative one. See [js-ext-runs-from-beagle-ext-shard].

• Real jab status on the journal store: *10.9 s → 1.5 s* (RSS 1.17 GB → 388 MB),

  output 1396 ok unchanged. Repro test/js/store.js (spy asserts 0 pack scans); parity 64/64 (JABC 28, be-bin 6, be-js 29). A file_id with no pack degrades to not-found, never crashes (safe under LSM newest-wins).