Beagle SCM

KEEP-006: cap a NNNNN.keeper log at 2 GiB — past that the log is unmappable, and the cap is unenforceable without also capping the fetch

PackLog says "a new log starts only past a size threshold" but never bounds that threshold, so a single landed pack can make one log arbitrarily large. GET-044 T4 proved it: a linux.git clone streamed 6.45 GB and atomically landed a durable *6.0 GB* 0000000001.keeper — bytes on disk that no reader can open. RULING 2026-07-27 (gritzko): *cap at 2 GB, not at the engine's ceiling. The engine numbers, all measured this session on sparse files: JSC allocates a typed array up to 4 GiB inclusive, but only 2^32-1 of it is addressable (a[2^32-1] reads undefined, JS's canonical not-an-index sentinel), and the JAB-007 Buf | 0 cursor wraps NEGATIVE at 2^31 exactly, so io.mmap yields an empty or garbled view from there up. 2^31-1 is the only bound that survives the whole path, and ingest.js already refuses at exactly that (MMAP_CAP). jab/io.cpp JABC_MAP_MAX now enforces it in the mapper (this session). The sting in the tail: PackLog §Delta-dependency pins OFS_DELTA bases inside one pack*, so a received pack cannot be split across logs — meaning a 2 GB log cap is only achievable if no single fetched pack exceeds 2 GB. Scope: keeper log rotation + the fetch that feeds it. See PackLog, Keeper; method Issues.

Input

Context

• wiki/PackLog.mkd §Many packs per log: "a new log starts only past a size threshold" — threshold unspecified, no upper bound, nothing refuses an oversized append.
• wiki/PackLog.mkd §Delta-dependency: "OFS_DELTA is pack-local: the base sits earlier in the same pack" — splitting one pack across two logs breaks every back-reference that crosses the cut.
• be/shared/ingest.js ships MMAP_CAP = 2147483647 and refuses cleanly (… landed OK but exceeds the jab 2^31-1 mmap cap) — a guard on the reader, not on what gets written.
• Measured io.mmap (Buf-wrapped) behaviour: 2^31-1 → correct; 2^31 → size -2147483648, data() empty; 3 GB → size -1073741824, garbled 2 GB view; 4 GiB → size 0. Silent every time.
• Measured raw io._mmap: exact lengths through 2^32 inclusive; 2^32+1 → was SIGABRT (exit 134), now a catchable throw.
• Live artifact: the linux.git benchmark this session (2026-07-27) — git clone 1849 s / 6.3 GB .git; jab get died after pulling 6024 MB, so the 6.0 GB log from GET-044 T4 is the only landed instance.

Goals

No keeper log ever exceeds 2 GB (2^31-1 bytes), and the writer refuses to create one instead of landing unreadable bytes.

• A hard KEEP_LOG_MAX (2^31-1) enforced at append time: an append that would cross it opens the next NNNNN.keeper instead.
• A pack too big to fit any log is refused *before* the network leg, with an error naming the size and the cap — not after a 28-minute download.
• The existing 6.0 GB log (and any like it) is diagnosable: a plain-words error saying which log is oversized, not a truncated read or a crash.
• Decide and record the fetch-side rule that makes the cap reachable (see Constraints) — a log cap alone cannot hold.

Constraints

• OFS_DELTA stays pack-local (PackLog §Delta-dependency) — do NOT split a received pack at an arbitrary byte offset to fit a log. Either the whole pack fits, or the fetch must have been smaller.
• Landed logs are durable data: this ticket adds a bound for NEW appends and a diagnosis path for existing oversized logs. No rewriting, no migration, no repacking of what is already on disk.
• The cap is a chosen bound, not the engine's — do NOT "recover" headroom by raising it toward 4 GiB when JAB-007 lands. Upstream will not move past 4 GiB either (the 2 GB→4 GB raise needed non-Int32 indices threaded through the DFG/FTL; Wasm's 32-bit memory anchors it there), so the headroom is one doubling and then a wall.
• Newer JSC buys nothing: webkitgtk 4.1 and 6.0 on this box are the SAME engine build (both 2.48.1-r2, libs 32 bytes apart), probed identical at every boundary. 6.0 also drops the JavaScriptCore/ C API headers the binding is written against, so switching is a large port for zero gain.
• Independent of JAB-007 and must not wait on it: even with correct cursors, a 6 GB log stays unreadable.

WIP

Design decisions

RULING 2026-07-27 (gritzko): log splitting is fine on the beagle side — the indexes already support multiple files. That makes the incoming pack, not the log, the binding constraint: a received pack cannot be split at an arbitrary byte because PackLog §Delta-dependency pins OFS_DELTA bases inside one pack. Every crossing delta must be re-anchored by hash. So the question became how much does REF_DELTA inflate the pack, and it was measured rather than argued.

*Measurements 2026-07-27* — git verify-pack -v over two real packs, base-reference bytes only (the zlib payload is byte-identical either way, so added = 20 - ofslen(distance) per converted edge; ofslen thresholds 128/16512/2113664/270549120 match git's offset encoding exactly). Verified every delta in the git.git pack is currently type 6 OFS_DELTA (315878/315878, type nibble read straight from the pack), so the baseline is real and nothing was already sha-coded.

Three findings that decide it:

• *All-REF gets worse with scale, not better.* +1.73% at git.git, +2.56% at linux. Denser deltas and longer distances both push it up. Option 2 at the real 2 GiB cap is 6.7x cheaper at kernel scale (0.38% vs 2.56%).
• *The crossing curve is FLAT.* 16.91% of deltas cross at 3 segments, only 22.49% at 24 — because base distances are enormous (linux: mean 384 MB, max 5.16 GB, 20.6% of deltas need 5-byte varints). The first cuts catch nearly every long chain, so further splitting is close to free. Consequence: cut-point optimisation is NOT worth building. Cut wherever the cap falls.
• *Crossing edges cost a flat 15.00 B*, not the 17.24 B an average delta would — they are all long-range, so they were already paying 5-byte varints.

Bytes were never the real argument against all-REF. The other two costs are:

• *Read path.* Chain hops to read every delta: git.git 2,894,063 (mean depth 9.16); linux 46,118,804 (mean depth 4.86, max 50). Under OFS each hop is cur -= ofs_delta — one subtraction inside the already-mapped log. Under REF each is a hashlet range query across every LSM run in the shard, newest-wins (Indices §Range queries), i.e. a random probe into a second file then a jump back. All-REF turns 46M subtractions into 46M lookups, permanently, in every repo. Option 2 leaves 44.5M of those as arithmetic and converts only the 1.6M crossing edges — 96.5% less new lookup load.
• *Log self-containment.* Today a log resolves standalone with no index at all, which is exactly what let PACK-003 salvage-index two torn logs by rescanning them. All-REF inverts that: no index, no resolution, for anything. A damaged index stops being recoverable-from-the-log and becomes unresolvable data.

DECISION: *option 2* — OFS within a pack/segment, REF only across boundaries. This is PackLog §Delta-dependency as already written, not a new design. The crossing set falls out of the scan pass ingest already runs, so the implementation cost is bounded.

Big incoming packs: stream-ingest, never stored

RULING 2026-07-27 (gritzko): *the received pack is never read back and never even stored.* It is consumed once, forward, straight from the wire into the log. This supersedes the whole line of thought that preceded it (windowed io._mmap, an io._pread primitive, git-style dependency-ordered resolution) — none of it is needed, and none should be built.

Why it works: OFS_DELTA bases are backward-only (PackLog §Intra-pack object order, "forward-reference free"), so by the time a delta arrives on the wire its base has ALREADY been appended to our log. Every read-back therefore targets our own log, which is ≤2 GiB by the cap above and so maps with the plain whole-file io.mmap that exists today. The 6.33 GB pack is never seeked, never mapped, never written to disk. Resolving a chain is a walk inside a mapped log, exactly as it is today — no new access primitive, no traversal reorder.

The one new structure is a *pack-offset → log-offset map*, live only for the duration of one ingest. It is needed because rewriting a base ref changes a record's size (OFS distance re-encoded, or OFS→REF at +15 B), so log offsets drift away from pack offsets as you go. Two shapes, both measured against linux.git (11,690,992 objects, 3,566,506 of them — 30.5% — actually cited as a base; you cannot exploit that, since a base is named by an object that arrives later):

• *Full map, one entry per object: pack offset (33 bits for 6.33 GB) + log offset (31 bits, a log is ≤2 GiB) + log id, packing to ~11 B → ~128 MB* at kernel scale, negligible for a normal repo. Monotonic in both coordinates, so binary-searchable.
• *Drift table, the compact form: store log_offset - pack_offset only where it CHANGES — at a crossing rewrite, and wherever accumulated drift pushes an OFS distance over a varint threshold. On the order of the 1,605,582 crossings → ~15 MB*. Same lookup, sparse.

Start with the full map; compress to the drift table only if it bites.

The ingest loop is then: inflate → resolve against the log → sha1 → index → append with the base ref rewritten (OFS if the base landed in this log, REF if in an earlier one — its sha is known, we just indexed it) → rotate at KEEP_LOG_MAX. Constant RSS, disk cost = the logs alone (no 6.33 GB tmp spike on top).

Consequences to honour:

• *Records land before the trailer is verified.* That is NOT a new hazard: PackLog §Epoch recompaction already rules that dropping is a refs tombstone and dead objects linger until an out-of-band epoch. So a bad trailer = refs never published, records orphaned, collected at the next epoch. State that rule explicitly — it is what makes discarding the tmp file safe.
• *The 20-byte trailer lag still applies* — the incremental hasher must stay 20 bytes behind so the sha1 trailer is never fed to the pack parser as object data. Same mechanism GET-044 built for drainToFile, now feeding the ingest instead of a file writer.
• *Crash mid-ingest is BETTER than with a tmp file*, not worse: the records already appended are durable, so a retry can have-negotiate from what landed rather than re-downloading 6 GB. Git's protocol has no pack-transfer resume, so the tmp file never bought resumability anyway.
• [GET-044]'s drainToFile staging becomes unnecessary for the clone path; it is superseded here, not merely unused. Do not keep both shapes alive.

Still first in line regardless: curlRun (be/shared/wire.js:250) buffers the entire https response in the JS heap, so it dies on any pack over the typed-array cap before ingest gets a chance to run. Streaming ingest cannot start until the bytes arrive AS a stream.

Where stream-ingest lives: libdog, as "repack a git pack"

RULING 2026-07-27 (gritzko): the ingest loop is C, in libdog. *libdog sees the job as "repacking a git pack" and knows nothing about curl, HTTP, or pkt-line framing* — opening the stream, stripping headers, and consuming the negotiation preamble is JS-level routing that hands libdog a bare pack byte stream (an fd) and a destination shard, and gets back counts and the new refs.

The loop, per PackLog object order (forward-only, bases already behind us):

1. parse the next pack record from the stream;
2. append pack_off -> our_off to the ingest-local map (our off is a wh64 carrying the log/file number, so the entry is wh128b);
3. resolve the delta using that map plus the already-written pack log;
4. hash it and repack it into the pack log;
5. add the hashlet -> our_off entry to the HIT index (needed to resolve REF entries);
6. keep a running sha1 over the stream for the trailing checksum;
7. on reaching KEEP_LOG_MAX, start a new pack log.

Why C and not JS: it is per-object over ~11.7M objects for a linux-scale clone, and every step is already native — dog/git/PACK.c header/ofs decode, PACKResolveOfs, PIDXScan (which already emits key=hashlet60|type4, val=offset wh128 entries straight into a caller-owned region), and abc's HIT/LSM for the index. In JS it would be 11.7M+ JSC binding crossings per clone, every one of them wrapping a buffer view and so exposed to the JAB-007 | 0 cursor hazard. JS keeps what it already owns: URL classify, capability/want-have negotiation, refs, shard resolution, rotate policy.

One correction to step 6, recorded because the instinct to drop it is reasonable and wrong: the running sha1 is *not* redundant with per-record hashing. Per-record hashes cover inflated object content only, not the framing bytes between records — and a stream cut that happens to land on a record boundary would otherwise parse as a clean, complete pack. The trailer is what distinguishes "peer finished" from "connection died tidily". It costs one pass over bytes already in hand.

Prototype: dogrepack, verified on the 6.33 GB linux pack

Built 2026-07-27 in work/KEEP-006 as git/REPACK.cli.c (+ one add_executable line) — a libdog CLI running the exact loop above, so the design is measured rather than argued. It consumes an fd, never stores or maps the source, and resolves every base out of our own logs via the pack-offset map.

Two independent confirmations fell out of it:

• *The crossing estimate holds at full scale.* The analysis predicted 1,605,582 crossings at a 2 GiB cap from verify-pack data; the actual repack emitted *1,605,989* REF records — 0.025% apart. (git.git @128 MB: 57,855 predicted vs 57,609 actual, 0.4%; the gap is that predicted cuts fall on exact byte multiples while real log boundaries fall on record edges.) The +0.38%-of-pack figure for boundary re-anchoring stands.
• *Streaming changes nothing.* Piping the same 6.33 GB pack through stdin — no file, no mapping, one 1 GB buffer — produced all three logs *byte-identical* to the mapped run, at 12m31s vs 12m22s (~1.2% slower, i.e. free). That is the property that lets a socket substitute for a file.

Three constraints the production ingest inherits, each found by hitting it:

• *zlib's avail_in is a 32-bit uInt*, so ZINFInflate refuses any window over 4 GiB. Handing it "everything buffered so far" fails at the FIRST record of a 6.33 GB pack. The reader must present bounded windows — which the u8b DATA span does by construction, since DATA can never exceed the buffer.
• *ZINFInflate could not distinguish TRUNCATED from CORRUPT* — a starved stream yields Z_BUF_ERROR, damaged bytes yield Z_DATA_ERROR, and both collapsed into ZINFFAIL; PACKInflate then collapsed every ZINF error into PACKBADOBJ. Without that distinction a streaming reader cannot tell "refill and retry" from "give up", so the loop would have to guess a record size. Added ZINFMORE (git/ZINF.h/ZINF.c) plus the PACKInflate passthrough. Retry is safe because nothing is consumed until success and inflate has no side effects.
• *The open log must be file-backed.* As anonymous memory it OOM-killed at 99.3% of the linux run (1.6 GB dirty anon beside the ~935 MB map). Backed by its own file the pages are reclaimable page cache — which is what production wants anyway.

Also settled by construction: the loop never needed io._pread, windowed io._mmap, or dependency-ordered resolution. Every one of the 9.49M deltas resolved through the map plus our own logs, including across closed-log boundaries. Record shape used: mean 541 bytes, max 1.6 MB compressed (git.git max 0.4 MB) — so a 1 GB buffer carries ~600x the worst record, and u8bReMap doubling is a safety net rather than a live path.

Buffer plumbing needs NO new primitive: abc/Bx.h already has it. PAST | DATA | IDLE maps one-to-one onto consumed / unparsed / read-target, with u8bShift (Bx.h:415) as drain-shift, u8bUsed as the parser cursor, u8bFed after a read, u8bMap for the MAP_NORESERVE allocation and u8bReMap for the grow case.

Fetch-side rules (bounded fetch), still open

• *(b) Bounded fetch via negotiation* — deepen-style rounds so no single pack nears the cap. Now an OPTIMISATION rather than a necessity: stream-ingest already handles an arbitrarily large pack. Still worth it for time-to-first-usable-state and for filter=blob:none. Yields thin packs whose bases are REF by construction — the same mechanism option 2 needs anyway.
• *(c) Refuse loudly* — no longer needed as a safety property, since nothing has to fit in a mapping any more.

Both depend on REF_DELTA resolution, which is a RESTORE not a build — dog/git/PACK.c:252 returns PACKREF as a deliberate backstop ("Foreign REF packs go through UNPK"), and dog/git/PIDX.c:4 records that this leaner scanner DROPPED "the REF_DELTA waiter" the keeper had.

TODOs

• KEEP_LOG_MAX = 2^31-1 at the append/rotate site; an append that would cross it starts the next log. Cut wherever the cap falls — the flat crossing curve makes cut-point optimisation not worth building.
• Stream https as a stream: curlRun buffers the whole body in the JS heap — port GET-044's streaming to the curl path. Blocks everything else.
• Restore REF_DELTA resolution — PACKResolve/PIDXScanRef take a caller finder (sha → base pack + offset), so a rotated log's cross-log bases resolve; no waiter forest needed, bases always land earlier.
• Stream-ingest in libdog as "repack a git pack": fd + destination shard in, counts + refs out; forward-only, never stored; pack-offset → log-offset map as wh128b (full map first, drift table if it bites). NO curl/HTTP/pkt-line knowledge below the JS routing layer.
• Re-anchor crossing deltas as REF_DELTA at the rotate boundary; base shas are already known from the on-the-fly indexing.
• Write the orphaned-records rule into PackLog explicitly (bad trailer = refs unpublished, records collected at the next epoch) — it is what makes not-storing safe.
• Do NOT build: io._pread, windowed io._mmap, dependency-ordered resolution. Superseded by stream-ingest; recorded so nobody re-derives them.
• Diagnosis for existing oversized logs (the 6.0 GB one): name the file and its size instead of truncating or throwing from the mapper.
• Sparse-file regression pinning the rotate boundary (cheap — no multi-GB fixture needed).
• Get the ruling on the fetch-side rule (a)/(b)/(c) and record it here before coding the fetch half.
• Verify end-to-end through a real jab get, not a unit probe.

Blockers and bummers

The https clone path has its own unrelated wall that will mask any fix here: curlRun (be/shared/wire.js:250) buffers the whole response in the JS heap and dies on new Uint8Array(total) for a >4 GiB body (RangeError: length too large) — GET-044's streaming drainToFile was only wired to the spawn/ssh transport. Needs its own ticket; until it lands, https can't reach the log-writing code at all.

Outcome

Filed 2026-07-27 from the linux.git clone benchmark; the read half of the rotation now works. UNCOMMITTED in the main tree per the no-post order.

• jab/io.cpp JABC_MAP_MAX = 2^31-1 refuses oversized mappings in plain words (was an uncatchable SIGABRT past 4 GiB, and a silent Buf truncation from 2^31 up); corrected be/shared/wire.js header comment. Suggested commit: KEEP-006: refuse mappings over 2 GiB in plain words.
• REF_DELTA resolution restored in libdog: PACKResolve + PIDXScanRef take a pack_ref_find (sha → base pack slice + offset) and resume the chase in whatever log answers; each hop carries its own pack. PACKResolveOfs/PIDXScan are the find == NULL wrappers, so the OFS-only PACKREF backstop is unchanged. dogscan <shard> reads a rotated shard back, its finder being just the entries the scan already emitted. Suggested commit: KEEP-006: resolve REF_DELTA bases across pack logs.
• Verified end to end, not by unit probe: git.git repacked at a 128 MB cap → 3 logs, *57,609* REF records, dogscan resolves 417,182/417,182 objects in 69 s and every sha matches git verify-pack; beagle-ext at a 1 MB cap → 5 logs, 279 REF, same set as the single-log shard and as git. Unit: PACKRESOLVEtest cross-log chain (3 logs, plus finderless → PACKREF, unknown base → PACKREF), PIDXtest scan_ref; suite 108/108.