"pinnable" git refs?

[sethp]

Hello! I've been having a great time with ADD <git ref> as a way to get a reproducible build environment for a constellation of CMake-based C++ projects without having to worry about the various packaging ecosystems and whence changes wend their way to my machine. The parallelism and caching capabilities of buildkit are proving very effective for me as well, thanks so much for your work on this project! There's maybe one or two more things that'd be needed to unlock nice improvements—a way to integrate COPY --link with CMake's install database and "goldilocks"ing the parallelism with something cgroup/job-server related both come to mind—but those are discussions for a different day.

One of the most effective pieces of buildkit for me is the FROM tag@pin syntax: it just composes so nicely with my workflow (starting with a FROM ..., adding the @pin when I'm happy with the selection), and very clearly ties together two crucial facts about a software dependency: what stream of work I pulled from, and precisely what that looked like when I was pulling from it. That clear communication of intent + machine-level reproducibility is 👨‍🍳 🤌.

So it's in that spirit that I want to raise introducing similar syntax for the ADD <git ref> URLs. I'm imagining something like:

ADD https://github.com/moby/buildkit.git#master@87d75a1c76c98971ce0dda299c12d9913dab9818 ...
# or:
ADD https://github.com/moby/buildkit.git#[email protected] ...

That would parse down to the equivalent of saying ADD https://github.com/moby/buildkit.git#v0.12.2

This syntax would serve those same dual goals:

1. It'd be reproducible: every time I docker build, I get the same revision
2. It's easy to update: I just have to compare the latest ref available at master against the ref specified after the @

Plus, it has the merit of being a nicely scoped change: AFAICT, it's just a little bit of extra logic in the gitutil package. I've got a little prototype that I think would work, though I wanted to get y'all's feedback before I went too far with it.

I looked through the issues and discussions, and the closest I could find was #2794 , which isn't quite what I'm after: I think I'd still need to describe "this ref represent a point-in-time sampling from this source" and then update both the policy.json and the Dockerfile from that external source of truth.

So, what do you think? I know I can build & maintain my own frontend based on buildkit's that would support this syntax, but I wonder if it would be useful to more folks to than just me.