packaging path: state of the work and why it’s shaped the way it is
This is a consolidated writeup of the deb/rpm packaging work to date: what exists now, the decisions that shaped it, and the alternatives that were considered and rejected at each point. This post is intended to be used as the reference for anyone picking up packaging work after me.
Goal
One shared, opt-in CI/CD path that takes any MergeTB repo from a v* tag to signed .deb and .rpm packages published at pkg.mergetb.net, with zero per-repo release logic beyond a debian/ tree, an rpm/ spec, and a few lines of .gitlab-ci.yml. Before this work: rpm publishing existed via ssh/scp from the runner, deb publishing was a dead ssh block in the templates, and every repo that wanted packages hand-rolled them.
What exists now
Automated signed deb publishing. aptly (v1.6.1) runs on pkg.mergetb.net behind an nginx reverse proxy at pkg-api.mergetb.net. CI is a pure HTTPS client: it POSTs the .deb to the aptly REST API with an apikey; aptly adds it to the local repo, re-snapshots, and re-publishes to a filesystem endpoint (/var/www/repo/mergetb/debian, hardlinked from aptly’s pool) that nginx serves as https://pkg.mergetb.net/debian. Publishes are GPG-signed on the host as the signing key never touches a CI runner. Validated end-to-end with clean-container install tests across debian 11/12/13 and ubuntu 20.04 to 26.04.
A single canonical mergetb suite. Consolidated from the legacy per-codename suites (bookworm/trixie). The addrepo client script was modernized from deprecated apt-key add to gpg --dearmor into /etc/apt/keyrings/mergetb.gpg with a signed-by=-scoped sources entry.
Fixed shared templates and canonical scaffolding. ci-templates (deb.yml, tagged_deb.yml) now handles multi-package repos, installs declared build-deps, and publishes every deb a build produces instead of silently the first one. The canonical pkg/debian template was audited and fixed so that copying it into a repo actually works.
The artifact-consume build model for debs. Static binaries are built once on the f42-cib builder in the make job; deb jobs consume build/* via needs: [make] instead of compiling in-tree. This is the standard for Debian packaging going forward. RPM jobs deliberately build from source.
Repos onboarded. ymk (the pipeline’s original test case, deb+rpm), foundry (deb merged, three packages: foundryc/foundryd/foundryctl; rpm three-subpackage spec), ground-control (rpm fixes), mars (deb for mars-install + mrs merged; rpm in review). Queued: remaining mars binaries (spore, moacmd, moactl, dancecmd), canopy, facility-install, ground-control deb.
Decisions and why
1. Publish via the aptly REST API, not ssh from the runner
Alternatives: (a) keep the legacy model, aptly on the runner, scp files to the host, run a script over ssh (this is still how rpm works via publish_and_update.yml + repoupdate.bash); (b) API-driven, runner is an HTTP client, aptly and signing live on the host.
Chose (b) because it removes ssh keys and the GPG private key from the runner entirely. The runner needs one secret (PKG_API_KEY, injected only on protected refs), and the host owns publishing and signing atomically. The rpm ssh path was left as-is deliberately, as aptly does not support rpm packagin. It’s also a working, separate mechanism, and forcing the two flows into one template would couple unlike things.
2. One mergetb suite instead of per-codename suites, and no +debNN version suffix
The evidence that drove this: our packages are static Go binaries (CGO_ENABLED=0) with zero runtime dependency closure. The deb 11/12/13 matrix produces byte-identical output (verified by md5 across matrix jobs after making builds reproducible). The +debNN suffix (e.g. 0.0.25-7+deb12) was creating separate pool inodes for identical bytes, and the only real differences between “per-distro” builds were the version string and a wall-clock dch timestamp.
Alternatives: (a) the Debian-proper camp, never share a deb across suites, rebuild per-suite with +debNNuM markers (correct when compiled output genuinely differs per release); (b) the vendor-repo camp one artifact, one suite, when builds don’t differ.
Chose (b) for the current package set, with (a) held in reserve: per-codename suites get created only when a package empirically fails on a specific distro. The cross-distro install matrix is the deciding instrument.
NOTE ON VERSIONING: Versioning needs to be fixed as recent findings found that there are conflicts with how versioning is done now and how versioning is done with Go compared with deb and rpm.
3. Artifact-consume for debs (build once on f42-cib, consume prebuilt)
The forcing event: grpc v1.82 landed as a direct foundry dependency and requires Go ≥ 1.25. Every Debian image’s packaged toolchain tops out at 1.24 (trixie). In-tree deb builds were dead on all three matrix images.
Alternatives: (a) install upstream Go into the deb images (bumps and maintains a toolchain in three images forever); (b) pin/downgrade grpc (fights the actual codebase); (c) build once on the Fedora builder (f42-cib ships Go 1.25+ natively, GOTOOLCHAIN=local) and have deb jobs consume the prebuilt static binary via needs: [make], with override_dh_auto_build copying instead of compiling.
Chose (c) It removes the Go toolchain requirement from the deb images, since every future Go version bump would otherwise re-break the matrix. The machinery mostly existed already: the make job already built all binaries on f42-cib and published build/* as artifacts.
Two traps this surfaced, now handled in the canonical rules: dpkg-buildpackage’s clean step runs make clean (rm -rf build) before the build override runs, wiping the prebuilt binaries. Fixed with an empty override_dh_auto_clean. And fresh CI checkouts give every file the same mtime, so make treats committed generated files (.pb.go) as stale and re-runs protoc codegen. Fixed by touching tool stamps/generated files before make is ever invoked.
Why rpm is different on purpose: the shared rpm template runs rm -rf build before rpmbuild, so rpm jobs build from source by design and don’t take needs: [make]. The spec keeps a prebuilt-consume guard for local use (populated build/ skips compilation), and both paths are validated in the harness.
4. ci-templates vs pkg: where variation lives
ci-templates is live-included and universal: zero per-repo variants, changes there hit every project on the next pipeline (real blast radius, reviewed accordingly). pkg is copy-once scaffolding: repos vendor deb_build.bash and the debian/ skeleton, edit freely, and comments/documentation are welcome there. Per-repo variation belongs in the vendored build scripts and debian/rules, never as template variants. When a fix lands in a vendored copy, the same fix goes to pkg so the next copier inherits it. So always in this merge order: ci-templates → pkg → downstream repo.
5. Fail loudly
A theme rather than a single decision. The worst bugs found were silent successes: dh_install with no .install file exiting 0 with an empty package; tagged_deb.yml publishing only head -n1 of the debs and dropping the rest; the pkg sed placeholder pass rewriting its own script and permanently corrupting vendored copies; rm -rf "$BUILD_DIR/*" with a quoted glob that expanded to nothing; rpm_build.bash writing one .ulpath where the publish job expected one per rpm; VERSION defaulting to 1.0.0 on bare local runs. The fixes consistently prefer a loud failure (missing binary → hard error in the GENERIC_BINARIES loop; non-root local run → explicit error with instructions) over a plausible-looking wrong artifact.
What’s next
- Remaining mars binaries (spore, moacmd, moactl, dancecmd, etc.)
- Multi-arch (arm64/aarch64): the open architectural question. Deb side is nearly free (Go cross-compile,
Architecture: arm64, aptly suite gains an arch); rpm side hinges on infrastructure we don’t control locally (native arm runners vs qemu vs prebuilt-consume). Related: thego_build.ymltemplate audit found that the only two repos hand-rolling their build jobs (foundry, mars) are exactly the two building multi-arch. The template has no multi-arch mode, and that’s the root cause, not adoption laziness. - Onboarding canopy, facility-install, ground-control deb.
repoupdate.bashlives unversioned on the host, should get into a repo.
Questions/corrections welcome