Beagle SCM

CRDT Metadata for Git

There is a well-defined problem of keeping project metadata in Git. In this case, "metadata" means everything GitHub has that Git does not. These are issues, statuses, comments, labels, and so on. The problem is universally acknowledged, and several solutions exist. There are historical git-notes, which no one has used much for about 20 years. The new ones are git-bug, Radicle COBs and GitButler git-meta. All three proposals put CRDT objects into Git's ODB, in custom blob formats and on a separate commit chain. CRDT means Conflict-free Replicated Data Types; they merge with no conflicts. Which is great, as we don't want to go insane resolving those ones too. Interestingly, the Automerge project never intended to be Git-compatible. But eventually, it arrived at a very similar architecture, just by building from the other end backwards. It is probably a case of convergent evolution. Finally, my solution is Beagle, the Git-compatible SCM with CRDT merges. Given these cases, we may try to connect some dots and draw some conclusions. As always, xkcd #927 is highly relevant here.

Looking from above, all these projects may seem to be converging, somewhat erratically, on a design people can actually use daily. As an example, CRDTs belong to three classes: op-, delta-, and state-based. COBs are op-based, while git-meta and Beagle are state-based. But, Git's machinery (persistent trees and deltas respectively) makes this distinction moot. In the end, it's all deltas.

Still, I see three key issues that all three Git metadata solutions share. They are rooted in metadata becoming yet another echelon of Git's store. That sub-store is not idiomatic to Git and is handled specially. As a consequence,

1. We add machinery that is not shared with the rest of the system. Metadata is now CRDT, while the rest stays as it was.
2. It requires its own interfaces, commands and flags that developers have to remember. (LLMs are no better at that. Having 10 different ways to do something, they pick a random one.)
3. Metadata is scoped to a git repo. A medium-sized project often has several repos as submodules. Now think of a repo with submodules that is forked into a worktree, and that also has metadata. Head hurts? Mine does.

So, what did Beagle do to overcome these issues?

1. It upgraded all diffs, merges, and blames to CT CRDT. Fundamentally, that does not change the picture: CT CRDT belongs to the same algorithmic family as 3-way merge, and in easier cases they produce the same result. In other cases, CT is much more precise.
2. The granularity of change is a token, not a line. So for example, re-wrapping a text file does not produce a whole-file diff. Beagle has 70 lexers to tokenize programming and markup languages. Automerge is character-based, while others are line-based, so Beagle has middle ground here.
3. It indexes files and links in files. In particular, it indexes YAML preambles in Markdown files and key-value meta in StrictMark. That produces a cross-repo unified file/metadata index.
4. It uses path and history indexes to resolve permalinks. That means a link like view/cite.js:28:Hz will point to that exact line in any version of the file.

With Beagle, every text file is effectively a CRDT. For that reason, metadata is kept as code comments or Markdown annotations. There is no separate metadata layer on par with the stash or staging, and no need for that separate machinery. All of the system benefits from it. Importantly, Beagle's indexes work across all local repos. For example, a comment can link to a file in a submodule or a ticket in a separate repo. It operates more like a search engine (text+index), not like a database (log+state).

I believe this sufficiently improves upon the other solutions.