This is current working in beta with manually applied k8s configs.
The short of it is that we have a wireguard container on the portal, exposed externally via ipvs and exposed internally via iptable forwarding rules and coredns dns rewrites.
Internal Wireguard Configuration
Wireguard Deployment
First, we have a wireguard deployment, configured via a configmap, which accepts Wireguard traffic on its port and forwards all other incoming traffic from the k8s side to the single remote IP address.
Right now, this only supports 1 remote NAT’ed facility, but I think it’s possible to differentiate between different hosts via IP addresses.
apiVersion: apps/v1
kind: Deployment
metadata:
name: nat-wg-gateway
namespace: merge
spec:
replicas: 1
selector:
matchLabels:
app: nat-wg-gateway
template:
metadata:
labels:
app: nat-wg-gateway
spec:
containers:
- name: wireguard
image: linuxserver/wireguard:latest
securityContext:
capabilities:
add: ["NET_ADMIN", "SYS_MODULE"] # SYS_MODULE help with kernel hooks
env:
- name: PUID
value: "1000"
- name: PGID
value: "1000"
- name: TZ
value: "UTC"
volumeMounts:
- name: nat-wg-config-volume
mountPath: /config/wg_confs/wg0.conf
subPath: wg0.conf
volumes:
- name: nat-wg-config-volume
configMap:
name: nat-wg-config
---
apiVersion: v1
kind: ConfigMap
metadata:
name: nat-wg-config
namespace: merge
data:
wg0.conf: |
[Interface]
Address = 10.138.0.1/24
ListenPort = 31820
PrivateKey =
# TODO: support multiple NAT points
# handle TCP/UDP traffic
PostUp = iptables -t nat -A PREROUTING -i eth0 -p tcp -j DNAT --to-destination 10.138.0.10
PostUp = iptables -t nat -A PREROUTING -i eth0 -p udp ! --dport 31820 -j DNAT --to-destination 10.138.0.10
# forwarding and return path masquerading
PostUp = iptables -A FORWARD -i eth0 -o wg0 -j ACCEPT
PostUp = iptables -A FORWARD -i wg0 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
PostUp = iptables -t nat -A POSTROUTING -o wg0 -j MASQUERADE
# Clean up rules when interface goes down
PreDown = iptables -t nat -D PREROUTING -i eth0 ! --dport 31820 -j DNAT --to-destination 10.138.0.10
PreDown = iptables -D FORWARD -i eth0 -o wg0 -j ACCEPT
PreDown = iptables -t nat -D POSTROUTING -o wg0 -j MASQUERADE
[Peer]
PublicKey =
AllowedIPs = 10.138.0.10/32
Services
apiVersion: v1
kind: Service
metadata:
name: nat-wg-external
namespace: mergev1
spec:
type: NodePort
selector:
app: nat-wg-gateway
ports:
- name: wg-handshake
protocol: UDP
port: 31820
targetPort: 31820
nodePort: 31820
---
apiVersion: v1
kind: Service
metadata:
name: nat-api-connector
namespace: mergev1
spec:
clusterIP: None
selector:
app: nat-wg-gateway
ports:
- port: 6001
name: grpc
Then, we expose it, once externally for 31820via a NodePort and once internally via the none cluster IP. It’s probably possible to use a cluster ip, so we can differential between different facility destinations.
DNS Configuration
In coredns’s configmap (kubectl -n kube-system get cm coredns), we add the line:
rewrite stop name exact api.csuautomotive.facility.local nat-api-connector.mergev1.svc.cluster.local answer auto
This is so that when api.csuautomotive.facility.local , it gets translated to the service IP.
External Wireguard Configuration
This is pretty much done with ipvsadm, were we use round robin selection and list each destination server.
sudo ipvsadm -C
# the "bind"
sudo ipvsadm -A -u 206.117.31.241:31820 -s rr
# round robin on them
# server sw1
sudo ipvsadm -a -u 206.117.31.241:31820 -r 10.0.0.19:31820 -m
...
Future Work
The main steps that needs to be done is to:
- Put the facility’s private key into the model when requesting NAT, so that the portal can generate these configs and apply them
- Install these things via helm instead.
- Unify the management of external forwarding for GRPC, HTTPS, and Wireguard under Nginx Server.
- Support multiple NAT’d facility destinations.