Attach in the VTE not working

Attaching an XDC to an experiment in the VTE does not appear to be working.

Initial setup

$ mergetb new xdc diamond falco
$ ./xdc-config.sh ry diamond falco
$ ssh falco-diamond-ry

Verifying materialization is active

$ mergetb status diamond one
NAME              TYPE           COMPLETE    ERROR    SITE
SetServiceVtep    Canopy         true                 spineleaf
SetLinks          Canopy         true                 spineleaf
CreateMoaLink     Moa            true                 spineleaf
InitMoaControl    MoaControl     true                 spineleaf
CreateNetwork     Nex            true                 spineleaf
CreateNetwork     Nex            true                 spineleaf
AddMembers        Nex            true                 spineleaf
AddMembers        Nex            true                 spineleaf
n6 (n05)          NodeSetup      true                 spineleaf
n5 (n01)          NodeSetup      true                 spineleaf
n4 (n00)          NodeSetup      true                 spineleaf
n2 (n03)          NodeSetup      true                 spineleaf
n1 (n04)          NodeSetup      true                 spineleaf
n0 (n02)          NodeSetup      true                 spineleaf
UpdateStatus      bookkeeping    true                 spineleaf
launch            container      true                 spineleaf
launch            container      true                 spineleaf
launch            container      true                 spineleaf
launch            container      true                 spineleaf
create-enclave    plumbing       true                 spineleaf

Attach failure

$ ry@falco:~$ attach ry diamond one
Attaching to materialization one.diamond.ry
Status Code: 400
Error: rpc error: code = Unknown desc = exit status 2
Detail: Uncategorized: rpc error: code = Unknown desc = exit status 2
Timestamp: 2020-04-15 13:16:31.523449311 +0000 UTC m=+833.977895740
Type: https://mergetb.org/errors/uncategorized

wgcoord logs

$ eval $(rvn ssh master)
rvn@master:~$ merge-t2.kubectl logs wgdcoord-85d545468c-748vd
time="2020-04-15T13:03:02Z" level=info msg="Starting daemon mode. Version v0.1.15-3-0-ga91cbbc"
time="2020-04-15T13:03:02Z" level=info msg="listening on tcp://0.0.0.0:6000"
time="2020-04-15T13:12:11Z" level=info msg="create enclave" allowedips=192.168.254.0/24 enclaveid=one.diamond.ry gwep="spineleaf.mergetb.test:36000"
time="2020-04-15T13:12:11Z" level=info msg="create enclave" allowedips=192.168.254.0/24 enclaveid=one.diamond.ry gwcert="-----BEGIN CERTIFICATE-----\nMIIEKjCCAxKgAwIBAgIUQ4fxfx201DKMHETvXmCUB0upVXEwDQYJKoZIhvcNAQEL\nBQAwbDELMAkGA1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWExFzAVBgNVBAcT\nDk1hcmluYSBkZWwgUmV5MRAwDgYDVQQKEwdkY29tcHRiMQswCQYDVQQLEwJDQTEQ\nMA4GA1UEAxMHZGNvbXB0YjAeFw0yMDA0MTUxMjIyMDBaFw0yMDEwMjgwODIyMDBa\nMGQxCzAJBgNVBAYTAlVTMQswCQYDVQQIEwJOWTEMMAoGA1UEBxMDTllDMRAwDgYD\nVQQKEwdkY29tcHRiMRIwEAYDVQQLEwl3aXJlZ3VhcmQxFDASBgNVBAMTC2Rjb21w\ndGI6d2dkMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAygBQ3FUYUr41\nPbyH+122v6GoEIHWe0Y7ayIcVLQ13NY8Ci1YAtSs8EA3yLvf9RyIrMNzR0P/ebE+\njz/EyRhiymBEjGocJ30OF/ppxu8DuUtaRPK6CtkfD/ndt3dLTUd6nJPVrtr8tpH8\nOAdDuVxqYldPTOJAJjkPRW/AlYRJh4qwTSE6rVLPHhUuBWkdUGkrayzF6FWHRoUM\ny93iAt3tjx69T4rVN94Rcp2Fh1eRuY3rGabTlfGF6KREiddnmvCWzOzwDCbP1HAI\nY0jXtK6+uSHv/miuL4tJUu7Lzl+cBUOcADuOvvcYFvrRnfxNH303RDzoJgpk2wFv\n9TU1OFnucwIDAQABo4HLMIHIMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggr\nBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUzZOf1xvx\n0Ru3/fc3izUyQ4nBKeQwHwYDVR0jBBgwFoAUj5QssnX6V643DJ2IAQYYTjiWEQ8w\nSQYDVR0RBEIwQIIDd2dkghB3Z2QubWVyZ2V0Yi50ZXN0ghZzcGluZWxlYWYubWVy\nZ2V0Yi50ZXN0gglsb2NhbGhvc3SHBH8AAAEwDQYJKoZIhvcNAQELBQADggEBAD08\nqzBGo/Bc8tyBlBIRM/eVwFo5XFjCxnhPShA+5pRnYMHaP7jvktw9vpXD02Q71TIO\njvdVzjqdPM7AzJ56oMAXVDZqoSqfN6akHKQqBLAvjbk0p5iir7fcEMnTt+8WUMVZ\n0J55c3lRODjnwIBDbK9qCtABvtK5ofufuDMKbpmyNuWrwiDJjZ/iqxRjR5bc8Obw\niKi04R2QfN8QW842nNYCzuylNxnfYafaJybcMHnBK1eh0+Ns4wGt6TGaOwibOVyJ\nezs1044KKkc4CoGc2El4z4zg+D1wxFU2P647yO52nYls1LDW5rHeRJt6zRn+o5x8\nvjocyRcYCsJ0VRpD5Ts=\n-----END CERTIFICATE-----\n" gwendpoint="spineleaf.mergetb.test:36000"
time="2020-04-15T13:12:11Z" level=info msg="read configuration file"
time="2020-04-15T13:12:11Z" level=info msg="db.host=db"
time="2020-04-15T13:12:11Z" level=info msg="db.port=2379"
time="2020-04-15T13:12:11Z" level=info msg="db.tls.cacert=/dbcerts/ca.pem"
time="2020-04-15T13:12:11Z" level=info msg="db.tls.cert=/dbcerts/mdb.pem"
time="2020-04-15T13:12:11Z" level=info msg="db.tls.key=/dbcerts/mdb-key.pem"
time="2020-04-15T13:12:11Z" level=info msg="sorting out certs for etcd tls connection"
time="2020-04-15T13:16:31Z" level=info msg=ClientContainerCreate containername=falco.diamond.ry enclaveid=one.diamond.ry
time="2020-04-15T13:16:31Z" level=info msg=DoClientContainerCreate containername=falco.diamond.ry enclaveid=one.diamond.ry
time="2020-04-15T13:16:31Z" level=info msg=containerName2WgdData name=falco.diamond.ry
time="2020-04-15T13:16:31Z" level=info msg="found data - node ip: 10.47.0.21:36000 containerID: 612ae7b206aca1ed5849bd1c663a9d49b492213007794c266a01cbaef91ebf88"
time="2020-04-15T13:16:31Z" level=info msg="wgd grpc connect to spineleaf.mergetb.test:36000"
time="2020-04-15T13:16:31Z" level=info msg="connecting to spineleaf.mergetb.test:36000 with cert for spineleaf.mergetb.test"
time="2020-04-15T13:16:31Z" level=info msg="error creating wg interface: rpc error: code = Unknown desc = exit status 2"
time="2020-04-15T13:16:31Z" level=info msg="error creating gateway interface: rpc error: code = Unknown desc = exit status 2" containername=falco.diamond.ry enclaveid=one.diamond.ry
time="2020-04-15T13:16:31Z" level=error msg="container create disconnect: rpc error: code = Unknown desc = exit status 2" containername=falco.diamond.ry enclaveid=one.diamond.ry

wgd logs

$ eval $(rvn ssh cmdr)
rvn@cmdr:~$ sudo journalctl -u wgd --no-pager
-- Logs begin at Wed 2020-04-15 13:00:03 UTC, end at Wed 2020-04-15 13:22:35 UTC. --
Apr 15 13:05:13 cmdr systemd[1]: Started The Wireguard Daemon.
Apr 15 13:05:13 cmdr wgd[9713]: time="2020-04-15T13:05:13Z" level=info msg="Starting wgd. Version v0.1.15-3-0-ga91cbbc-dirty"
Apr 15 13:05:13 cmdr wgd[9713]: time="2020-04-15T13:05:13Z" level=info msg="listening on tcp://0.0.0.0:36000"
Apr 15 13:16:31 cmdr wgd[9713]: time="2020-04-15T13:16:31Z" level=info msg=CreateGatewayInterface accessaddr=192.168.254.1 enclaveid=one.diamond.ry namespace=one.diamond.ry
Apr 15 13:16:31 cmdr wgd[9713]: time="2020-04-15T13:16:31Z" level=info msg="create gateway interface" accessaddr=192.168.254.1 enclaveid=one.diamond.ry namespace=one.diamond.ry
Apr 15 13:16:31 cmdr wgd[9713]: time="2020-04-15T13:16:31Z" level=info msg="nsp code: 2"
Apr 15 13:16:31 cmdr wgd[9713]: time="2020-04-15T13:16:31Z" level=info msg="Running output command: ip netns exec one.diamond.ry wg show interfaces\n"
Apr 15 13:16:31 cmdr wgd[9713]: time="2020-04-15T13:16:31Z" level=info msg="create interface" accessaddr=192.168.254.1 enclaveid=one.diamond.ry namespace=one.diamond.ry nscode=2
Apr 15 13:16:31 cmdr wgd[9713]: time="2020-04-15T13:16:31Z" level=info msg="nsp code: 2"
Apr 15 13:16:31 cmdr wgd[9713]: time="2020-04-15T13:16:31Z" level=info msg="Running command: ip -br link show wgdntxg\n"
Apr 15 13:16:31 cmdr wgd[9713]: time="2020-04-15T13:16:31Z" level=info msg="Running command: ip netns exec one.diamond.ry ip -br link show wgdntxg\n"
Apr 15 13:16:31 cmdr wgd[9713]: time="2020-04-15T13:16:31Z" level=info msg="Running command: ip link add dev wgdntxg type wireguard\n"
Apr 15 13:16:31 cmdr wgd[9713]: time="2020-04-15T13:16:31Z" level=error msg=ifdev.add accessaddr=192.168.254.1 enclaveid=one.diamond.ry namespace=one.diamond.ry nscode=2
Apr 15 13:16:31 cmdr wgd[9713]: time="2020-04-15T13:16:31Z" level=error msg="creating server if" accessaddr=192.168.254.1 enclaveid=one.diamond.ry namespace=one.diamond.ry

Wireguard is not working correctly on the worker node. It installs, but the kernel module does not load.

ic, so the issue seems to be the provisioning of the worker machine

$ eval $(rvn ssh worker)
rvn@worker:~$ sudo modprobe wireguard
modprobe: FATAL: Module wireguard not found in directory /lib/modules/4.19.0-6-amd64

The wireguard module is getting installed, it’s just the currently running kernel is out of date.

rvn@worker:~$ find /lib/modules -name "*wireguard*"
/lib/modules/4.19.0-8-amd64/updates/dkms/wireguard.ko
rvn@worker:~$ uname -a
Linux worker 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) x86_64 GNU/Linux

Correct.

(This added to hit the 20 char minimum per post).

An issue has been created for this in GitLab

This issue has been resolved and the VTE should be back to working normally for XDCs and attaching.