[caasp-beta] Problem adding additional node to cluster after 24h

Fri Jul 19 05:36:54 MDT 2019

Hi all,

I found a workaround and like to share my solution but I don't know if it is working for all having the same issue:

Start by "skuba node join --v 9 --role worker --target $HOSTNAME $HOSTNAME"

Updating the whole cluster (still in beta status) did not work for me.

There is a hint in the kubeadm forums that the join token for kubeadm get's invalid after 24h
In this rare case you can just hop to you first kubernetes master and type:
kubeadm token create --print-join-command

Copy that string and run it on the destination worker node to join the cluster.
After that "kubectl get nodes" will may show that node as "Not Ready" but as member of the cluster.
Follow the Suse upgrade node documentation for CaaS and the node will be shown as "Ready".

Regards

   Roy

________________________________
Von: Carsten Duch <carsten.duch at suse.com>
Gesendet: Freitag, 19. Juli 2019 12:06:22
An: caasp-beta at lists.suse.com <caasp-beta at lists.suse.com>
Betreff: Problem adding additional node to cluster after 24h

Hi all,

I recently installed CaaSP 4 with a customer.

We tried to add another worker node after >24h to the cluster.

But skuba failed to deploy the node.

After we found out how to set debug level in skuba, we found out that this might be related to an existing issue on github:

https://github.com/kubernetes/kubeadm/issues/1310<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkubernetes%2Fkubeadm%2Fissues%2F1310&data=02%7C01%7C%7Cf35ab2ace10849095da408d70c30cab0%7Cf481d56fc63d4d748406d64e74f44395%7C0%7C0%7C636991276021929597&sdata=VWFwk0CVI%2B%2BcmLFiCanCFjUCYDa%2BcztprTLg1hVemsU%3D&reserved=0>

This is the debug output we got:

I0718 19:19:52.762272   12629 ssh.go:190] stdout | [preflight] Running pre-flight checks

I0718 19:19:52.889960   12629 ssh.go:190] stdout | [preflight] Reading configuration from the cluster...

I0718 19:19:52.890010   12629 ssh.go:190] stdout | [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

I0718 19:19:53.043975   12629 ssh.go:190] stdout | [kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace

I0718 19:19:53.372300   12629 ssh.go:190] stderr | error execution phase kubelet-start: configmaps "kubelet-config-1.15" is forbidden: User "system:bootstrap:pwupeo" cannot get resource "configmaps" in API group "" in the namespace "kube-system"

I0718 19:19:53.373248   12629 ssh.go:167] running command: "rm /tmp/kubeadm-init.conf"

[join] failed to apply join to node failed to apply state kubeadm.join: Process exited with status 1

F0718 19:19:53.402383   12629 join.go:49] error joining node s030v0302: failed to apply state kubeadm.join: Process exited with status 1

Is this a bug in skuba? Looks like it’s required to reissue a token for kubeadm after 24h, I am unsure if skuba does this automatically or not.

Mit freundlichem Gruß / Best regards

Carsten Duch

Sales Engineer

SUSE Software Solutions Germany GmbH

Maxfeldstr. 5

90409 Nürnberg

(P)+49 173 5876 707

carsten.duch at suse.com<mailto:carsten.duch at suse.com>

--

SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)

----------------------------------------------------------------
innogy SE
Vorsitzender des Aufsichtsrates: Dr. Erhard Schipporeit
Vorstand: Uwe Tigges (Vorsitzender), Dr. Hans Buenting,
Dr. Bernhard Guenther, Arno Hahn, Martin Herrmann, Hildegard Mueller
Sitz der Gesellschaft: Essen, Eingetragen beim Amtsgericht Essen,
Handelsregister-Nr. HRB 27091, USt-IdNr. DE304171711
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.suse.com/pipermail/caasp-beta/attachments/20190719/decf9bb6/attachment.html>