PACK-001: PIDXScan from_off — index a pack from a byte offset (tail scan)

PACK-001: PIDXScan from_off — index a pack from a byte offset (tail scan)

Now: DONE

The pack index-emitter PIDXScan (dog/git/PIDX.c) only scans a whole pack from the 12-byte header, bounded by the header's object count; there is no way to index just the un-covered TAIL of a pack. Add a from_off parameter so a caller can emit entries for [from_off, EOF) only — the "index the difference" primitive JS-099's open-time reindex needs. Safe because the store is offset-only (PackLog, OFS-ALWAYS): a tail object's OFS base is always earlier in the SAME pack and resolves from the whole-pack slice we pass. Patch drafted. Method Issues.

Input

Context

.keeper.idx run; the fix reindexes "the difference" on open. Detecting it reads the PACK bookmark's byte_len; indexing it needs a scan from that boundary (pack_offset + byte_len, a known object boundary).

PIDXScan(u8cs pack, Bwh128 out, u8s base, u8s delta) walks [12, EOF) bounded by hdr.count; the JS pk.scan (js/pack.hpp _pack_scan) is whole-pack only. Neither can start mid-pack.

Offset-only (PackLog): a tail OFS_DELTA base is earlier in the same pack, so PACKResolveOfs(pack, off, …) on the whole slice resolves it — a tail scan is correct. REF_DELTA never appears in a native log.

Goals

PIDXScan(u8cs pack, u64 from_off, Bwh128 out, u8s base, u8s delta): from_off==0 = today's whole-pack scan (count-bounded, unchanged); from_off>0 = tail scan [from_off, EOF), driven by offset not count.

Binding: _pack_scan(buf, dataLen, out, base, delta, fromOff=0) + pk.scan(out, fromOff) wrapper (js/pack.hpp).

Emitted entries carry ABSOLUTE offsets; a tail OFS object shas correctly against a pre-from_off base.

Constraints

dog/git + the JABC binding only. Whole-pack (from_off==0) behavior stays byte-identical, INCLUDING the hdr.count truncation guard.

Offset-only: NO REF_DELTA handling added — a REF_DELTA still returns PIDXFAIL (the OFS-only backstop). Deltas are PACK-002, offset-only there too.

Repro-first (CLAUDE.md §17); table tests in dog/test/PIDX.c + JABC packidx.

WIP

Design decisions

Factor the per-record resolve/sha/emit/advance into a static pidx_emit_at helper; the full path stays a for i<hdr.count loop (keeps the truncation guard), the tail path is while (off < packlen) from from_off.

from_off is a caller-supplied object boundary (a bookmark's pack_offset+byte_len); PIDXScan does not validate it lands on a boundary — GIGO, same contract as raw offsets elsewhere.

TODOs

PIDX.h/.c: add from_off + pidx_emit_at; two loops. — landed 8df1e213 (dog) / b141ab81 (beagle): PIDXScan(u8cs, u64 from_off, …).

Update callers to pass 0: dog/test/PIDX.c, jab pack.hpp:286 (PIDXScan(pack, 0, …) // PACK-001: 0 = whole pack).

Binding: _pack_scan fromOff arg + pk.scan(out, fromOff) — NOT exposed: jab pack.hpp JABCpackScan is still 5-arg and hardcodes from_off=0; cont.cpp P.scan(out) takes no fromOff.

Test: dog/test/PIDX.c tail-scan case (tail_scan, PIDX.c test 4 — OFS tail object whose base precedes from_off). — landed 8df1e213.

Test: JABC packidx from-offset case (jab test/packidx.js covers whole-pack scan + index-on-append only).

Blockers and bummers

None. Enables JS-099's incremental reindex and PACK-002's appended-pack tail indexing.

Audit 07Jul26: the C core + tail test are IN; the JS-facing half (binding fromOff + its test) is the open remainder — JS-099's reindex-on-open cannot start mid-pack from JS yet.

Outcome

C core landed 8df1e213 (dog PIDX.c/.h from_off + tail_scan test) / b141ab81 (beagle) / jab 9421ca8 ("make js append to pack logs" — append path; scan binding untouched).

Remaining: expose fromOff through _pack_scan/pk.scan + a JABC packidx from-offset case (see TODOs).