# Packaging so far

**URL:** <https://forum.mergetb.net/t/packaging-so-far/166>\
**Category:** DevOps\
**Created:** [August 5, 2026, 9:30pm UTC](https://forum.mergetb.net/t/packaging-so-far/166 "2026-08-05T21:30:01Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![bnguyen](https://forum.mergetb.net/letter_avatar_proxy/v4/letter/b/919ad9/32.png) [@bnguyen](https://forum.mergetb.net/u/bnguyen)\
**Post date:** [August 5, 2026, 9:30pm UTC](https://forum.mergetb.net/t/packaging-so-far/166/1 "2026-08-05T21:30:01Z")

</div>

# packaging path: state of the work and why it’s shaped the way it is

This is a consolidated writeup of the deb/rpm packaging work to date: what exists now, the decisions that shaped it, and the alternatives that were considered and rejected at each point. This post is intended to be used as the reference for anyone picking up packaging work after me.

## Goal

One shared, opt-in CI/CD path that takes any MergeTB repo from a `v*` tag to signed `.deb` and `.rpm` packages published at `pkg.mergetb.net`, with zero per-repo release logic beyond a `debian/` tree, an `rpm/` spec, and a few lines of `.gitlab-ci.yml`. Before this work: rpm publishing existed via ssh/scp from the runner, deb publishing was a dead ssh block in the templates, and every repo that wanted packages hand-rolled them.

## What exists now

**Automated signed deb publishing.** aptly (v1.6.1) runs on `pkg.mergetb.net` behind an nginx reverse proxy at `pkg-api.mergetb.net`. CI is a pure HTTPS client: it POSTs the `.deb` to the aptly REST API with an apikey; aptly adds it to the local repo, re-snapshots, and re-publishes to a filesystem endpoint (`/var/www/repo/mergetb/debian`, hardlinked from aptly’s pool) that nginx serves as `https://pkg.mergetb.net/debian`. Publishes are GPG-signed on the host as the signing key never touches a CI runner. Validated end-to-end with clean-container install tests across debian 11/12/13 and ubuntu 20.04 to 26.04.

**A single canonical `mergetb` suite.** Consolidated from the legacy per-codename suites (bookworm/trixie). The `addrepo` client script was modernized from deprecated `apt-key add` to `gpg --dearmor` into `/etc/apt/keyrings/mergetb.gpg` with a `signed-by=`-scoped sources entry.

**Fixed shared templates and canonical scaffolding.** ci-templates (`deb.yml`, `tagged_deb.yml`) now handles multi-package repos, installs declared build-deps, and publishes every deb a build produces instead of silently the first one. The canonical `pkg/debian` template was audited and fixed so that copying it into a repo actually works.

**The artifact-consume build model for debs.** Static binaries are built once on the `f42-cib` builder in the `make` job; deb jobs consume `build/*` via `needs: [make]` instead of compiling in-tree. This is the standard for Debian packaging going forward. RPM jobs deliberately build from source.

**Repos onboarded.** ymk (the pipeline’s original test case, deb+rpm), foundry (deb merged, three packages: foundryc/foundryd/foundryctl; rpm three-subpackage spec), ground-control (rpm fixes), mars (deb for mars-install + mrs merged; rpm in review). Queued: remaining mars binaries (spore, moacmd, moactl, dancecmd), canopy, facility-install, ground-control deb.

## Decisions and why

### 1. Publish via the aptly REST API, not ssh from the runner

**Alternatives:** (a) keep the legacy model, aptly on the runner, scp files to the host, run a script over ssh (this is still how rpm works via `publish_and_update.yml` + `repoupdate.bash`); (b) API-driven, runner is an HTTP client, aptly and signing live on the host.

**Chose (b)** because it removes ssh keys and the GPG private key from the runner entirely. The runner needs one secret (`PKG_API_KEY`, injected only on protected refs), and the host owns publishing and signing atomically. The rpm ssh path was left as-is deliberately, as aptly does not support rpm packagin. It’s also a working, separate mechanism, and forcing the two flows into one template would couple unlike things.

### 2. One `mergetb` suite instead of per-codename suites, and no `+debNN` version suffix

**The evidence that drove this:** our packages are static Go binaries (`CGO_ENABLED=0`) with zero runtime dependency closure. The deb 11/12/13 matrix produces byte-identical output (verified by md5 across matrix jobs after making builds reproducible). The `+debNN` suffix (e.g. `0.0.25-7+deb12`) was creating separate pool inodes for identical bytes, and the only real differences between “per-distro” builds were the version string and a wall-clock `dch` timestamp.

**Alternatives:** (a) the Debian-proper camp, never share a deb across suites, rebuild per-suite with `+debNNuM` markers (correct when compiled output genuinely differs per release); (b) the vendor-repo camp one artifact, one suite, when builds don’t differ.

**Chose (b)** for the current package set, with (a) held in reserve: per-codename suites get created only when a package empirically fails on a specific distro. The cross-distro install matrix is the deciding instrument.

**NOTE ON VERSIONING** : Versioning needs to be fixed as recent findings found that there are conflicts with how versioning is done now and how versioning is done with Go compared with deb and rpm.

### 3. Artifact-consume for debs (build once on f42-cib, consume prebuilt)

**The forcing event:** grpc v1.82 landed as a direct foundry dependency and requires Go ≥ 1.25. Every Debian image’s packaged toolchain tops out at 1.24 (trixie). In-tree deb builds were dead on all three matrix images.

**Alternatives:** (a) install upstream Go into the deb images (bumps and maintains a toolchain in three images forever); (b) pin/downgrade grpc (fights the actual codebase); (c) build once on the Fedora builder (`f42-cib` ships Go 1.25+ natively, `GOTOOLCHAIN=local`) and have deb jobs consume the prebuilt static binary via `needs: [make]`, with `override_dh_auto_build` copying instead of compiling.

**Chose (c)** It removes the Go toolchain requirement from the deb images, since every future Go version bump would otherwise re-break the matrix. The machinery mostly existed already: the `make` job already built all binaries on f42-cib and published `build/*` as artifacts.

**Two traps this surfaced, now handled in the canonical rules:** dpkg-buildpackage’s clean step runs `make clean` (`rm -rf build`) _before_ the build override runs, wiping the prebuilt binaries. Fixed with an empty `override_dh_auto_clean`. And fresh CI checkouts give every file the same mtime, so make treats committed generated files (`.pb.go`) as stale and re-runs protoc codegen. Fixed by touching tool stamps/generated files before make is ever invoked.

**Why rpm is different on purpose:** the shared rpm template runs `rm -rf build` before rpmbuild, so rpm jobs build from source by design and don’t take `needs: [make]`. The spec keeps a prebuilt-consume guard for local use (populated `build/` skips compilation), and both paths are validated in the harness.

### 4. ci-templates vs pkg: where variation lives

**ci-templates is live-included and universal** : zero per-repo variants, changes there hit every project on the next pipeline (real blast radius, reviewed accordingly). **pkg is copy-once scaffolding** : repos vendor `deb_build.bash` and the `debian/` skeleton, edit freely, and comments/documentation are welcome there. Per-repo variation belongs in the vendored build scripts and `debian/rules`, never as template variants. When a fix lands in a vendored copy, the same fix goes to pkg so the next copier inherits it. So always in this merge order: **ci-templates → pkg → downstream repo**.

### 5. Fail loudly

A theme rather than a single decision. The worst bugs found were _silent successes_: `dh_install` with no `.install` file exiting 0 with an empty package; `tagged_deb.yml` publishing only `head -n1` of the debs and dropping the rest; the pkg `sed` placeholder pass rewriting its own script and permanently corrupting vendored copies; `rm -rf "$BUILD_DIR/*"` with a quoted glob that expanded to nothing; `rpm_build.bash` writing one `.ulpath` where the publish job expected one per rpm; `VERSION` defaulting to `1.0.0` on bare local runs. The fixes consistently prefer a loud failure (missing binary → hard error in the `GENERIC_BINARIES` loop; non-root local run → explicit error with instructions) over a plausible-looking wrong artifact.

## What’s next

- Remaining mars binaries (spore, moacmd, moactl, dancecmd, etc.)
- **Multi-arch (arm64/aarch64)**: the open architectural question. Deb side is nearly free (Go cross-compile, `Architecture: arm64`, aptly suite gains an arch); rpm side hinges on infrastructure we don’t control locally (native arm runners vs qemu vs prebuilt-consume). Related: the `go_build.yml` template audit found that the only two repos hand-rolling their build jobs (foundry, mars) are exactly the two building multi-arch. The template has no multi-arch mode, and that’s the root cause, not adoption laziness.
- Onboarding canopy, facility-install, ground-control deb.
- `repoupdate.bash` lives unversioned on the host, should get into a repo.

Questions/corrections welcome

---

<div class="post-metadata">

**Author:** ![jdbarnes](https://forum.mergetb.net/user_avatar/forum.mergetb.net/jdbarnes/32/57_2.png) [@jdbarnes](https://forum.mergetb.net/u/jdbarnes)\
**Post date:** [August 7, 2026, 10:59am UTC](https://forum.mergetb.net/t/packaging-so-far/166/2 "2026-08-07T10:59:26Z")

</div>

thanks for the overview. i think its a good starting point for further work.

p.s. i really dont like the LLM-isms. 😤