Beagle SCM

GIT-012 JS: smart-fetch transports — keep SSH, add HTTPS via curl spawn

The JS wire (be/shared/wire.js) speaks git's smart protocol but only over a spawned ssh/local keeper; classify() has no HTTP(S) branch, so an https:// (or git://) URL wrongly routes to ssh host git-upload-pack (wire.js:28,76). jab has no TLS and we will NOT add one, so HTTPS rides a spawned curl; the pkt-line negotiation (advert → want/have/done → NAK → pack) is transport-independent and reused as-is. Transport I/O moves to the non-blocking pol loop. SSH stays exactly as it is; git:// (daemon, TCP 9418) is OPTIONAL — net.connect already yields the socket and the core is shared. Method: Issues.

Input

Context

• be/shared/wire.js fetch() runs ALL transports through io.spawn + blocking io._read/io.writeAll; the negotiation core is transport-agnostic (wire.js:142-201).
• classify() has NO http(s) branch: //host, git://, ssh:// all map to ssh host git-upload-pack, so an HTTPS remote silently mis-routes to ssh (wire.js:24-78).
• The C keeper has no smart-HTTP either (keeper/WIRECLI.c only local-exec or ssh), so smart-HTTP is net-new and lives in JS — no C reference to mirror.
• jab exposes non-blocking TCP (net.connect over pol, js/net.cpp:55-64,323-331) but NO TLS/HTTP/curl binding; pol is the loop the runtime drains after the script (js/main.cpp:159,319).
• Motivating case: a real fetch of Beagle's own //origin (replicated.live) and jab get https://… need an actual HTTPS path, not an ssh misfire.

Goals

• Add an HTTPS smart-fetch transport to the JS wire (fetch first; push a follow-up), keeping the ssh:// and local-keeper paths behaviour-identical.
• Drive all transport I/O non-blocking over the pol loop, replacing the synchronous io._read loops with pol-fed incremental reads.
• Reuse the shared pkt negotiation (pkt.js, parseAdvLine, pickWant) verbatim across ssh / https / (optional) git://.
• OPTIONAL: a git:// daemon adapter (net.connect :9418 + one request pkt-line), included only because the core is shared and it needs no new binding.

Constraints

• HTTPS = spawned curl (curl owns TLS+HTTP); NO native HTTPS/TLS — jab has none and we will not add one.
• No new JABC bindings: git:// rides existing net.connect, HTTPS rides io.spawn(curl), ssh/local unchanged.
• Non-blocking only: every transport fd is a pol.watch registration; the wire awaits via the loop, never a blocking read.
• ABC/wiki norms: reuse before adding, ≤2-line comments, greppable no-wrap lines, ticket-coded commits.
• Do not vandalize the working ssh/local path — parity must hold (meta/work.mkd).

WIP

Design decisions

• Split wire.js into a transport-agnostic negotiation core + per-transport adapters: local/ssh (spawn, have), https (curl spawn, new), git (net.connect, optional).
• Make the core async: feed pkt.Reader from pol-driven 'data' chunks (net socket or watched spawn fd), resolving on advert-flush and on pack-EOF — no synchronous io._read.
• Smart-HTTP is two stateless requests: GET <base>/info/refs?service=git-upload-pack then POST <base>/git-upload-pack, types application/x-git-upload-pack-{advertisement,request,result}.
• The HTTP advert carries a # service=git-upload-pack\n + flush preamble before the refs; the advert drainer skips it, while ssh/git go straight to the ref lines.
• Keep the demux-free path: request want <sha> ofs-delta WITHOUT side-band-64k, so the server streams the raw pack after NAK — exactly what wire.js already assumes (wire.js:174-193).
• curl shape: GET advert curl -sSf --get <base>/info/refs --data-urlencode service=git-upload-pack; POST curl -sSf -H 'Content-Type: …request' --data-binary @- <base>/git-upload-pack; stdout is the pkt stream.
• git:// (optional): one request pkt-line git-upload-pack <path>\0host=<host>\0 (v0, port 9418), then the identical core; the daemon serves upload-pack only.
• Push is OUT of scope here: receive-pack will be ssh + smart-HTTP POST git-receive-pack (own ticket); the git:// daemon refuses receive-pack by default (anonymous-push unsafe).

TODOs

• T1 — shared negotiation core (advert drain / want-have / NAK+pack) reused by the spawn and curl adapters. — landed 4370d67e (wire.js drainAdvert/pickWant/fetchHttp).
• T2 — non-blocking pol I/O for the CURL adapter (curlRun: pol.watch + pol.run pump, wire.js ~158-182). The ssh/local spawn path STAYS blocking (pkt.Reader over io._read) — the T-scope decision resolved as curl-only; revisit only if a real stall shows.
• T3 — HTTPS fetch adapter: curl GET info/refs (preamble skipped) + POST git-upload-pack. — landed 4370d67e (fetchHttp).
• T4 (http part) — classify() http/https branch → the curl adapter. — landed 4370d67e.
• T4 (git:// part) — git:// is STILL lumped with ssh (classify comment: git:// → ssh host git-upload-pack), so a git:// URL misroutes; split it off (route to T5's adapter or refuse clearly).
• T5 — OPTIONAL git:// adapter: net.connect :9418, daemon request pkt-line, run the core.
• T6 — adapters reached from the verb fetch sites via wire.fetch→classify; http end-to-end proven by the wire tests.
• T7 — hermetic http tests over a local git http-backend: test/wire/http-clone, http-update, http-incremental, big-http.

Blockers and bummers

• Decision needed: migrate the ssh/local path to async-pol too (one uniform core), or keep it blocking and run only the new transports async? Affects T1/T2 scope. — Landed answer: curl-only async; ssh/local kept blocking (audit 07Jul26: pkt.js still refills via io._read).
• AUDIT 07Jul26: ticket is LANDED except the git:// residue (T4-git/T5) — git:// still silently routes to ssh host git-upload-pack. Close-or-finish call is the owner's; kept open for that residue only.
• Real servers (e.g. github) may need ACK/multi_ack_detailed handling once haves are sent; the no-side-band assumption must be checked live before HTTPS relies on it.
• Private HTTPS auth (tokens / credential helper) is deferred — anonymous/public fetch first; curl header/-n injection a follow-up.
• curl presence/version is an external dependency; surface a clear error when curl is absent.

Outcome

• Core landed 4370d67e ("git protocol client", be/ trunk): smart-HTTP fetch over spawned curl (fetchHttp + curlRun on pol), classify() http/https branch, ssh/local behaviour-identical; tests test/wire/http-clone/http-update/http-incremental/big-http. Enabled the push twin (GIT-013 cb3881f8).
• Open residue: git:// (T4 split + optional T5 adapter) — a git:// URL still routes to ssh.