MergeTB VTE Pipeline, Current state and Future Goals

DevOps
1

MergeTB VTE Pipeline, Current state and Future Goals

  • This post serves as a summary for anyone in the future seeking to understand the design choices I made and how to continue where I left off.
  • A working familiarity with Terraform (OpenTofu), Ansible, Gitlab CI/CD, and virtualization is assumed.

Repository and host organization

All directories listed fall under netlab11, with only the Ansible roles available on gitlab.

  • Main repository:

    • /space/mergetb-vte

  • Libvirt Bash Script with Fedora Cloud img (served via nginx)

    • /usr/share/nginx/html/virtportal-bootstrap.tar.gz

  • Repository copy used inside each outer VM:

    • /home/user/mergetb-vte

  • Main Ansible playbook:

    • 2node-portal-playbook.yml

  • OpenTofu configuration:

    • /space/terraform

  • OpenTofu variables:

    • /space/mergetbops/facilities/mdr/netlab11/configuration/terraform/values.auto.tfvars

  • Main Ansible roles:

    • mergetb_portal_k8s_host(feature branch)

    • mergetb_portal_helm (feature branch)

    • mergetb_facility (feature branch)

    • mergetb_vte_haproxy (new role)

    • mergetb_gitlab_agent (new role)

Overall design philosophy

  • The VTE is meant to be a disposable integration environment, it should be realistic where the VTE has networking, Kubernetes, authentication, storage, the portal, and facility registration.

  • The environment is nested so each run is isolated inside one outer VM. Proxmox only needs to create and destroy that outer VM; everything inside it belongs to the VTE. This gives the pipeline a clean boundary and makes teardown predictable.

  • OpenTofu owns the outer infrastructure, libvirt owns the nested machines, and Ansible owns their configuration. Keeping those responsibilities separate makes it easier to determine whether a failure belongs to infrastructure creation, VM networking, Kubernetes, the portal, or the facility.

  • Values that change between deployments, especially hostnames, addresses, credentials, and certificates flow from the pipeline into templates. The intention is that a fresh run creates a reproducible but easily distinguished environment each time.

Pipeline overview

image
image1414×721 49.5 KB

  1. OpenTofu creates the outer VTE

    1. Gitlab Runner verifies OpenTofu configuration and Proxmox provider access

    2. Small “Load Balancing” script runs and check available resources against Proxmox nodes and returns highest scoring node

    3. GitLab Runner executes OpenTofu configuration against Proxmox node

    4. OpenTofu creates a Fedora VM with libvirt installed

    5. The relevant bash scripts/ ansibles are copied via rsync from netlab11 to the outer VM

  2. Bash script creates the nested machines

    1. Current VM layout:

      • k8m → Kubernetes control plane

      • k8w → Kubernetes worker

      • k8w1 → Kubernetes worker

      • ops → Virtual Facility ops

    2. Current management addresses:

      • k8m: 192.168.122.20

      • k8w: 192.168.122.21

      • k8w1: 192.168.122.22

      • ops: 192.168.122.30

  3. Ansible configures Kubernetes environment

    1. mergetb_portal_k8s_host configures and installs Kubernetes

    2. k8m serves as the control plane.

    3. k8w and k8w1 join as workers.

    4. kube-vip provides the cluster API virtual address

    5. The result is a normal Kubernetes cluster ready to have services installed

  4. Helm deploys the MergeTB portal

    1. mergetb_portal_helm installs the portal and its supporting services.

    2. This includes ingress, certificates, storage, authentication, portal services, Launch, Gitlab Agent, XDC support, and observability.

    3. Each VTE receives generated hostnames such as:

      • p<ID>-launch.vte.sphere-testbed.net

      • p<ID>-api.vte.sphere-testbed.net

      • p<ID>-grpc.vte.sphere-testbed.net

      • p<ID>-git.vte.sphere-testbed.net

    4. The generated VTE identifier keeps parallel environments from sharing endpoints or state.

  5. Ansible configures the facility

    1. mergetb_facility runs against ops.

    2. It stages the XIR facility model and renders the deployment-specific facility configuration.

    3. facility-install generate produces Ground Control, Mars, Ignition, and PXE configuration.

GitLab Kubernetes Agent and dashboard integration

Each pipeline creates its own GitLab Kubernetes Agent rather than relying on a permanent shared agent. The agent is named from the pipeline ID, such as vte-p12345 so it maps cleanly to that pipeline’s temporary VM.

The ansible_gitlab_agent CI job handles the complete registration and installation process. It passes the project, agent name, default branch, GitLab API, and KAS address to the mergetb_gitlab_agent Ansible role. The role creates the agent registration in GitLab and installs the agent into the new Kubernetes cluster. The API credential is supplied through the protected GITLAB_AGENT_API_TOKEN project variable (declared in Gitlab) and is removed from the outer VM after the job.

Once commissioning succeeds, the pipeline creates a GitLab environment named:

vte/p<CI_PIPELINE_ID>

This gives each VTE its own Kubernetes environment and GitLab Kubernetes dashboard.

image
image1324×781 74.3 KB

To reproduce this in another GitLab project, the project needs a runner capable of reaching netlab11, and a protected GITLAB_AGENT_API_TOKEN with permission to create agents for the project. After those variables are configured, the CI job performs the registration and cluster installation each run automatically.

Accessing the VTE

The portal launch pages are on a protected network. A dev workstation needs a SOCKS5 tunnel through netlab11:

ssh -D 1080 <user>@netlab11

FoxyProxy can then use:

  • SOCKS5

  • Hostname localhost

  • port 1080

  • proxy dns enabled

Then, at http://netlab11/vtes/ a webpage exists to scan ~30s for portal launch pages available on the network. It also tracks how long until an automatic teardown job is executed on each VTE:

image
image1004×553 34.2 KB

Future work/Known issues

  • Currently the ops node for the facility exists in deployments but automation for infrapod has not been created. In order to make virtual facility resources available to the VTE portal, this will need to be configured
  • Deployment times are slowed significantly by the current helm chart deploy, a golden image or a cache with more packages would greatly improve overall deploy time.
  • No real testing yet. For end to end testing eventually the goal is to, on merge request, have the pipeline deploy with the latest version proximal to the tree, then upgrade on a live VTE.
  • The current setup relies heavily on internal networking and resources not available to someone outside the MergeTB network trying to recreate it. An end user would currently be expected to create their own XIR, Playbook, and Libvirt bash script.
  • There is currently no resource limit on how many of these VTEs can be created in our Proxmox cluster.