Beagle SCM

LITE-006 DONE: quickjab index <repo> — the lazy commit-based .git/be index

Ruling (gritzko 2026-08-13): beagle-lite indexes many git repos to do what it can for git users; weave building needs each blob's history, so the first index is blob<->path. Indexing is COMMIT-BASED (read a commit, its tree, index everything, then next), LAZY (a watermark row per ref; on request recheck, reindex the gap, answer from the index), and lives in the repo's own .git/be/. Tracked repos are listed in $HOME/.config/be/tracks, a plain list of absolute paths.

Input

Context

The record structure and widths are ruled; the substrate exists (quickjab index.c = the 3 native leaves an abc.index LSM stands on, lane-typed pup stacks; dog/git = the read-only ODB waist quickjab git.c binds as git.open/getHex/getSafe/close). be/ has the JS precedents to mirror, NOT to import: metaidx.js (the lazy self-updating lane + mark-written-LAST discipline), mtimeidx.js (HEAD mark staleness = ancestry check, drop marks only, rows stay valid), ingest.js idxWriter (DOG-027: <=200 rows per batch, one 4 KB memtable page).

• One unified wh128 run family in <repo>/.git/be/; everything in .git/be/ is derived, rm -rf-safe, rebuilt from the ODB.
• Record types (kind nibble in the key; path_hl = top 40 bits of the path's hashlet; blob/commit hashlets ALWAYS full 60 bits; rev = per-path arrival-local 20-bit counter):
  1. REV-BLOB key = path_hl:40|rev:20|BLOB val = blob_hl:60|vnib:4 — the rev's content
  2. REV-CMMT key = path_hl:40|rev:20|CMMT val = commit_hl:60|vnib:4 — the commit that introduced the rev
  3. REV-PARS key = path_hl:40|rev:20|PARS val = par1:20|par2:20|par3:20|vnib:4 — parent revs of the SAME path; empty slot = all-ones; 4th+ parent = second PARS row
  4. CPAR key = commit_hl:60|CPAR val = parent_hl:60|ord:4 — commit->parent edge, one row per parent, first-parent ord 0
  5. B2P key = blob_hl:60|B2P val = path_hl:40|rev:20|vnib:4 — blob->(path,rev), one row per occurrence
  6. MARK key = ref_hl:60|MARK val = tip commit_hl:60|vnib:4 — per-ref watermark, written LAST after a completed batch
• One file's log = one prefix scan of its path_hl: rows come back rev-ordered oldest-first, each rev names its blob, commit and parents — exactly what weave folding (CFOLD) consumes, no ODB walk at query time.
• A commit yields a new rev of path P iff P's blob differs from P's blob in EVERY parent commit; unchanged subtrees prune on equal tree sha; a rev's PARS are the nearest ancestor revs of P itself.

Goals

quickjab index $HOME/src/linux indexes that repo; a second run is a fast no-op (watermark hit).

• lite/ gains the index verb: resolve the arg to a repo (root, .git dir or gitfile — what git.open already takes), append its absolute path to $HOME/.config/be/tracks (create dirs as needed, dedup on read), open the ODB, index, report a one-line summary (commits/revs/rows indexed, or "up to date").
• Walk = the CURRENT TIP and all its ancestor commits, indexed in chronological order (ancestors strictly before descendants; commit date as the practical order) so a commit's parent trees are already known when B2P/rev derivation needs the prev tree.
• Lazy contract: read MARK rows, compare to the repo's live tip; equal → answer from the index; lagging → index only the gap above the marks, then bump; a MARK that is no longer an ancestor of the tip (rebase/reset) drops the mark and rewalks — data rows stay, re-puts are idempotent.
• Reuse, never reimplement: object reads through the git.* JABC waist (dog/git ODB); the LSM through quickjab's abc.index leaves + pup lanes (port the thin JS handle from be/ shape if lite lacks one — mirror, don't require be/); ref/HEAD resolution through dog/git (CFG/GIT) if bound, else the smallest possible quickjab binding — NO JS reimplementation of packed-refs parsing.
• Tests in lite/test/index/: build a tiny fixture repo (a few commits incl. a merge and a rename-free blob move), run index, assert the six record kinds, the rev chain of one file, B2P rows, the no-op second run, and the gap run after adding a commit.

Constraints

• Work goes into lite/ (+ minimal quickjab bindings if a call is missing); be/ and jab/ are READ-ONLY precedent, never imported; never read beagle/.
• ABC house rules for any C edit (meta/abc.mkd): no manual parsing (ragel), typed slices, 2-line comments tagged LITE-006.
• DOG-027 write discipline: <=200 rows per put batch, marks/markers written last.
• Dev builds and scratch in ~/tmp; quickjab/build-lite may be refreshed in place. No VC ops; put+report only.

WIP

Design decisions

• ZERO quickjab edits. The waist already had everything but refs: git.open/getHex/getSafe/close (git.c), git.tree + git.parseCommit over GITu8sDrainTree / GITu8sDrainCommit / GITu8sCommitTree (cont.c + pack.c), and abc.index("wh128", {dir, ext}) — the dog Pup stack handle — already exists in cont.c, so no JS lane handle had to be ported from be/ either.
• Refs are read in JS (index/refs.js), which is where quickjab RULES they belong: git.c's own header says "Refs stay ABOVE this waist (test/gitverify.js reads HEAD/packed-refs as text) — the waist is ODB only", and dog/git/ carries no ref-store reader at all (GIT.h's GITParseRef parses a refNAME). The file mirrors quickjab's own sanctioned reader, quickjab/test/gitverify.js, cut to head() + resolve(). A ragel packed-refs machine does not exist in dog/, so hand-rolling one in C was refused.
• vnib is RESERVED (0) wherever the ruled table does not name the low nibble; CPAR's is the parent ordinal. An empty PARS slot is all-ones (REV_MAX), and a 4th+ parent rev rides a second PARS row.
• Walk order = Kahn over the in-set parent edges with the ready set drained oldest-first by COMMIT DATE. Topology is the hard constraint (clock skew can never put a child ahead of a parent), the date only picks between ready commits.
• The MARK is a SET, not a cell: a wh128 lane is unkeyed, so a bumped watermark is a second row on the same key and nothing in the row says which is newer. Since the ruling below it is only the O(1) no-op (tip already marked -> answer without scanning), that no longer matters.
• RULING 2026-08-13, WALK_CAP REMOVED: PRESENCE IS THE BOUNDARY. There is no walk ceiling. One pass over the lane yields the set of commits it already holds (their CPAR rows); the walk climbs from the tip and never enters one. Each run therefore indexes exactly the commits no run has indexed yet — a history of ANY size converges over successive runs, an interrupted run keeps every commit it sealed, and a rebase needs no special case (the rewritten commits are simply not in the lane, so they are what gets walked).
• THE ROOT-COMMIT WRINKLE. A CPAR row is now the commit's "I am indexed" flag, and a root commit has no parent to hang one on. It gets ONE CPAR row with an EMPTY parent slot — all-ones, which is the ruled table's own spelling for an empty slot (REV-PARS: "empty slot = all-ones"). No new record kind, no new nibble: a CPAR row means indexed, and its parent slot may be empty. log.js parentsOf() drops it, isIndexed() counts it. (The other case the ruling names — a commit whose parents are all pruned — cannot arise here: CPAR rows are emitted for every parent of every walked commit, pruning only ever applies to TREES.)
• ORDER WITHIN A COMMIT: the rev rows go in FIRST, the CPAR rows LAST. A seal persists a PREFIX of what was put, so a CPAR row on disk proves that commit's rev rows are on disk too. Put CPAR first and a kill between the two would strand a commit's revs forever behind a boundary that lies.
• A rewalk mints NO duplicate revs. With presence as the boundary a sealed commit is never walked twice, so the only rev that can be re-derived belongs to the commit IN FLIGHT when a run died — its rev rows may have been auto-sealed by a full memtable page while its CPAR rows were not. Those revs are by construction the HIGHEST rev of each path that commit touched, so the guard is one {rev, blob, commit} per PATH (st.top), not one entry per REV. That replaced a string-keyed per-rev map and is the difference between a bounded and an unbounded indexer at linux scale.
• Gitlinks (0o160000) are dropped — a submodule's commit lives in another ODB and is no blob. Deletions are NOT revs: a pruning walk of the commit's own tree sees present paths only, which is the ruled algorithm as written; deleted paths would need a both-sides tree diff and are out of scope here.

TODOs

• tracks file: append + dedup, $HOME/.config/be/tracks
• repo open + tip resolution through dog/git (check what quickjab binds today; smallest missing binding added) — NO binding was missing, zero quickjab edits
• the wh128 lane handle over index.c leaves in lite/ (mirror be/ shape) — cont.c's abc.index already IS that handle
• the commit walker: ancestors-first order, prune on equal tree sha
• row emit: REV-BLOB/REV-CMMT/REV-PARS/CPAR/B2P, MARK last
• fixture repo + lite/test/index/ suite PASS — 81 checks (incl. the resume + log legs)
• WALK_CAP removed; presence-of-records is the walk boundary (ruling 2026-08-13)
• quickjab index on a real repo: first run indexes, second run no-ops

Blockers and bummers

• ~~A run past the WALK_CAP (2^18 commits) never converges~~ — SUPERSEDED by the ruling of 2026-08-13. WALK_CAP is gone; presence of a commit's records is the boundary, so any history converges and an interrupted run resumes where its records end. See the LITE-007 field evidence.
• One pass over the lane per non-no-op run is the standing cost of the design: a single new commit on git.git still scans 770k rows (1.5 s), and at linux scale (5.5M rows) it is minutes with the run's byPB map — one entry per rev — putting RSS at 2.5 GB. The no-op path pays none of it (8 ms). Bounding it wants typed-array open addressing, the quickjab/test/gitverify.js ShaSet precedent; not built.
• Only HEAD is marked and walked. MARK is keyed per ref and the code is ready for more, but index walks the CURRENT TIP as ruled.

Outcome

Landed in ///lite (worktree work/LITE-006), zero quickjab edits.

• index/index.js — the verb: the tracks list, the wh128 record family, the lazy mark contract, the Kahn commit walk, the pruning tree descent and the rev/PARS derivation.
• index/refs.js — HEAD/loose/packed-refs resolution above the ODB waist (mirrors quickjab/test/gitverify.js).
• main.js — index dispatched before the flag scan; README.mkd — the usage line.
• test/index/run.sh + test/index/rows.js — 39 checks, PASS.

End to end: the fixture (5 commits incl. a merge and a same-blob-new-path move) → indexed 5 commits, 5 revs, 23 rows, rerun up to date, gap run indexed 1 commits, 1 revs, 6 rows. libdog (340 commits) → indexed 340 commits, 1713 revs, 6566 rows in 0.59 s, rerun 13 ms. gitoxide (15841 commits) → indexed 15841 commits, 71415 revs, 295705 rows in 15.7 s, rerun 8 ms. One file's prefix scan matches git log -- CMakeLists.txt exactly (36 revs, same commits, same order), reverts included.

Suggested commit message: LITE-006: quickjab index — the lazy .git/be commit index