Beagle SCM

PACK-003: idxmaint pack.scan: scan (out full? corrupt?) on every jab status — single-pack scan vs JS-117 multi-pack keeper logs

jab status in the journal tree prints idxmaint: Error: pack.scan: scan (out full? corrupt?) TWICE before rendering. idxmaint finds keeper logs with no idx coverage and calls ingest.buildIndex → pk.scan → native PIDXScan over the WHOLE log; the scan is single-pack/header-count-driven, a JS-117 log with appended packs behind the first header makes it fail. The logs stay unindexed forever, so their objects stay unk/miss — and the warn repeats on EVERY status. Deferred from PATCH-012 ("count-trusting class, needs own repro/ticket"); distinct from JAB-008 (BASS drain — this repros in a FRESH process).

Input

Context

Incident 2026-07-17, ~/src/journal: jab status → the warn twice, then a working status listing (with a miss ?cba10972#README.mkd row). Store ~/.be/journal/: 4 logs 0000000001..4.keeper, ONE 26710JvjCN.keeper.idx run.

• be/shared/ingest.js buildIndex (line ~143): pk.scan(buf) with buf = io.buf(pk.count*16+256) — pk.count is the FIRST header's count; walkTail (JS-117) covers the appended tail only AFTER a successful scan.
• jab/pack.hpp:289 JABCpackScan: PIDXScan(pack, 0, ...) walks [0, dataLen) = the whole mmapped log; a mid-stream pack header (appended pack, GET-046 verbatim keeper-served logs) or count/out exhaustion → generic scan (out full? corrupt?).
• be/shared/idxmaint.js line ~247 catches per-log and warn()s — non-fatal but re-attempted (and re-warned) every run; built stays false.

Goals

jab status in the journal tree runs warn-free; all 4 logs get idx coverage; the miss rows for objects living in the unindexed logs clear.

• Repro first: a test log = pack A + appended pack B records (or a copy of a failing journal log) making buildIndex throw exactly this error.
• Determine which of the 4 journal logs fail and WHY (mid-stream header vs out sizing vs true corruption) — the fix depends on it.
• Scan multi-pack logs correctly: per-embedded-pack scan windows (PACK-001 from_off exists) or scan-then-walkTail per segment.

Constraints

• Native PIDXScan stays single-pack; segmentation logic belongs to the JS side (ingest.js) that already knows the JS-117 log layout.
• Distinguish the failure classes in the throw — out full vs corrupt as one message is undebuggable (this ticket's incident is the proof).
• The landed logs are durable data — read-only diagnosis; NO log rewriting/migration in this ticket.

WIP

Design decisions

Diagnosis OVERTURNED the intro's hypothesis twice: the journal shard is healthy (4 logs fully idx-covered, header count == records, no mid-stream PACK header in ANY store log); the warns come from the beagle SUB store. Failure class is torn/corrupt bytes, so the fix is salvage, not segmentation. CORRECTED 2026-07-17 (STATUS-scan investigation): BOTH warns are ~/.be/beagle — status opens that shard TWICE (status.js:279 emitRepo + status.js:485 divergence→resolve_hash.js:302, two reader instances, maintain unmemoized); ~/.be/beagle.new is NEVER opened by status (shardDir resolves beagle directly; fixture decoy: 0 opens) — its corruption is real but dormant.

• ~/.be/beagle/0000000001.keeper: 116708-byte ZERO tail = two torn appends (JAB-008 class: resize + idx run persisted, record bytes/msync lost, 325 records gone); header counts them → count-driven PIDXScan walks into zeros → PIDXFAIL.
• ~/.be/beagle.new/0000000001.keeper: ~9.5MB corrupt region at 1214193 (bad zlib, born-torn clone; same pack re-appended at 10723328).
• The generic out full? corrupt? here always meant CORRUPT; idxmaint re-attempted both logs on every open — warn x2 per status, forever.
• BONUS live bug: walkTail sized resolve scratch off pk.size, which for ofs-delta is the DELTA size → NOROOM → 11269 of 28986 salvageable beagle records silently dropped.
• Fix (shared/ingest.js): walkOffsets extent census (skips GET-046 verbatim embedded headers, stops at torn bytes); resolveEntries grow-on-NOROOM; buildIndex keeps the native scan as fast path, on throw salvage-indexes every resolvable record + bookmarks the full extent — one-time rebuild, no re-warn.

TODOs

• diagnose per log: journal 1-4 healthy/covered; beagle torn zero tail; beagle.new corrupt region (table in worker report).
• repro test/packsalvage.js (be-js-unit-packsalvage): torn tail (the incident error), embedded-pack drop, both, ofs-delta NOROOM — all red on pristine be/.
• fix: salvage path in buildIndex (+77/-21 in shared/ingest.js).
• split the pack.hpp:289 throw (jab-side, untouched): PIDXFAIL vs out-full vs NOROOM.
• expose PIDXScan from_off through the pk.scan binding (hardcodes 0) — native segment scans later.
• verify: copies of both bad logs salvage 28986/28986 + 3073 parseable records, full-extent coverage, warn-free; suite 267/267 -LE big. Store itself untouched — the fix must MERGE before the real jab status warns clear; beagle's 325 torn records + beagle.new's garbage need a re-fetch to heal miss rows.

Blockers and bummers

Root cause of the torn appends is JAB-008 (BASS drain): appendRecords' crash-ordering claim does not hold on the BASS path — resize+idx survive while record bytes are lost.

Outcome

UNCOMMITTED in work/PACK-003 per the no-post order: mod shared/ingest.js (+77/-21); test subrepo: mod CMakeLists.txt (registration line), new test/packsalvage.js; scratch/ holds log copies + diag scripts. Orchestrator re-verified repro + suite. Suggested commit: PACK-003: salvage-index torn keeper logs.