Beagle SCM

TODO-003: meta-pair index — kv64 rows over todo/

Search over ticket meta pairs (Sta: OPEN presence, Who: gritzko exact) backed by a store-resident LSM index that decides WHICH tickets the board opens — it never renders. Rulings 2026-08-03: the keyed kv64 lane (a re-put overwrites, so a pair is a mutable cell), wh64-shaped path-major rows, the value stored literally when it fits ron60, and an mtime sweep that reconciles per-ticket blocks. Consumers: the :todo board (TODO-001, BE-038); the arg/click/sort layer is TODO-004.

Input

Context

• Meta pairs are the Key: value leaf, StrictMark §"Meta pairs"; ticket keys (Who: Due: For: …) registered in todo.
• Ticket status moves OUT of the header into a meta pair (ruling 2026-08-02, one-shot perl migration), so the open list becomes a plain key query.
• Tickets are edited in real time: committed blobs cannot answer an open-ticket query, and the working tree is what the board renders.
• The KEYED lane landed 2026-08-02: kv64Z compares keys only, so a re-put of one key wins at get, range and seek alike (jab/test/index.js:112).
• abc.index("kv64", {dir, ext}) is a handle on a dog Pup stack — runs, memtable and the 1/8 ladder are all C (jab/pup.hpp, jab/INDEX.md §"pup.hpp").
• jab/API.md §"abc.index" is STALE: .runs, ._seq, .mem, .flush, .compact and the dir-less RAM backing are gone, and commit() is the seal verb.
• Until DOG-026 emits leaf kinds, extraction is the per-line regex ^([A-Z][a-z][a-z]): (.*)$ — the meta pair is line-local by design, so the regex IS the grammar.

Goals

shared/metaidx.js: a LAZY indexing API over one kv64 lane in the store shard — the caller asks for a filtered ticket list and the index brings itself up to date to serve it. TODO-004 is built on this surface.

• The one entry point: scan the fs, index whatever is unindexed, then answer the filter FROM the index — no separate "build" verb for a caller to forget.
• Row: key (path_hl:40 | key_code:20 | kind:4) -> val (kind:4 | value:60) — the wh64 field split (dog/WHIFF.h) that keeper and graf already ride.
• Path-major: one ticket's pairs are ONE contiguous block, so prefix(path_hl<<24n, 24, cb) reads a whole ticket's meta in one pass.
• key_code is the 3-char key VERBATIM at 6 bits/char (18 of the 20 bits), injective — a meta key never collides and a ticket's pairs sort stably.
• The value is stored LITERALLY when its normalized form fits 60 bits of ron60, else as its hash: a literal ranges and compares exactly, a hash only equals.
• Scan: readdir todo/, stat each ticket file, re-lex every one whose mtime is at or past the mark, reconcile its block, write the new mark row.
• Answer: scan the rows carrying that key code, keep the matching values, map path_hl back through the readdir walk — that list is the filtered tickets.

Constraints

• The index decides WHICH tickets to open, never what to show — titles and bodies come off the file, so nothing renders straight from the index.
• A ticket is any .mkd under todo/ whose name fits the ticket-code regex, PLUS any DIR whose name fits it and which holds a README.mkd/README.md (the fat layout, todo).
• A fat ticket's identity is the DIR, but the indexed and stat'ed path is its README — one row block per ticket either way, and the attached files never index.
• 40-bit path hashlets collide at ~0.45 expected pairs per 1M tickets, but the sweep holds the COMPLETE path list, so a collision is DETECTED: drop both paths from the index that session and read them directly.
• The sweep OWNS staleness: with the value in the row there is no confirming read left to kill a stale row, so a vanished pair or a deleted file must be tombstoned against the ticket's block.
• Value normalization is despaced and decased (ruling 2026-08-02) — the payload is a match token, not a render source; the key needs none, its [A-Z][a-z][a-z] shape being canonical already.
• Rows put between two commits must fit ONE 4 KB memtable page (256 kv64 rows), so batch at 200 — the shared/ingest.js idxWriter discipline.
• Intermediate seals must NOT advance the mark: a crash mid-sweep may cost a redundant re-lex, never mark an unlexed file done.
• The mark is the MAX mtime observed in the scan (ruling 2026-08-03), held in a sentinel row per wt (reserved kind, key above every path block) — a clock is never consulted, so clock skew between the fs and the process cannot exist.
• The mtime test is >= the mark: a redundant re-lex costs one read, a missed edit gives a wrong answer, and a file touched mid-scan lands past the mark and re-lexes next time.
• Scope is todo/'s .mkd files (ruling); the runs live in the store's shard dir alongside keeper.idx, ext .meta.idx — never in <wt>/.be, which is a FILE whenever the store is not colocated.
• The shard is resolved through shared/store.js (<store>/.be/<project>/, the opened store's .shard), never composed by hand.
• Worktrees SHARE a shard, so path_hl hashes the wt-qualified path and the mark row is per-wt (the sentinel key carries the wt code); a global mark would let wt A's sweep mark wt B's unindexed files as done.

WIP

Design decisions

• Overturned 2026-08-03: the wh128 lane, key-hashlet-major rows, 40-bit key hashlets, the line field, and "filter-grade, never truth" — a keyed lane makes a pair a mutable cell, so the row IS the current value.
• Overturned from the 2026-07-31 draft: blob-keyed rows, 128-bit key and value digests, "no ranges, no enumeration", and the SPOT-003 substrate dependency.
• Ticket identity is the PATH, so the numbering scheme is orthogonal: this index never waits on a RON ticket code, and that code stays worth doing for the counter problem alone.
• Sta: OPEN is a full index scan under path-major ordering (a key column is not contiguous), which costs far less than the fs sweep that precedes it; a key-major twin lane is the fallback if it ever shows.
• Per-key normalization is where ranges come from: Due: 2026-08-03 stored as the ron60 date 26803 sorts and ranges, free text hashes.
• Path-keying costs the blob-at-a-new-path and moved-subtree cases nothing, because staleness is an mtime question here, not a hash-identity one.
• Multi-key board queries (Sta+Who+Due) intersect on path_hl before the board opens anything, per TODO-001.
• Key vocabulary stays out of the index (any 3-letter key encodes); todo governs which keys exist and which the views offer.
• OPEN: the status key's name, and whether severity splits off (Sta: OPEN plus Sev: HIGH) or stays merged as the header has it — gritzko's call, and a todo spec edit.

TODOs

• shared/metaidx.js: row pack/unpack + abc.index("kv64", {dir: shard, ext: ".meta.idx"}) open/close
• the lazy entry point: scan -> index the unindexed -> answer the filter, one call, for TODO-004
• the scan: readdir + mtime-at-or-past-mark re-lex, per-ticket block reconcile (tombstone what vanished), max-mtime mark row written last
• the answer: key-code scan, value match, path_hl -> path, the filtered ticket list
• ron60 value packing + the per-key normalizers (Due: -> date, free text -> hash)
• path-hash collision detector: two live paths on one path_hl -> exclude both, read them directly
• line-regex extractor with the Due\: escape; swap to DOG-026 kinds when they land
• golden: cold fill vs warm skip, edited file re-lexed, vanished pair tombstoned, deleted file, absent key early-out, two wts on one shard, crash-mid-sweep re-lex
• status-to-meta-pair migration: the perl pass over todo/**/*.mkd plus key registration in todo (human-owned spec edit)
• wire into the :todo board (TODO-001, BE-038)

Blockers and bummers

• The SPOT-003 and JS-121 dependencies are BOTH dropped: the keyed kv64 lane landed 2026-08-02 and serves the overwrite collapse this index is built on.
• DOG-026 is no longer a dependency: the meta pair is line-local, so the regex is the grammar, and the line field is gone.

Outcome