[caasp-beta] Q: Any skuba bootstrap error with kubeadmn known?
Roger Klorese
roger.klorese at suse.com
Mon Jul 15 05:57:54 MDT 2019
Looks to me from the message that the DNS server, not the registry, was unreachable...
Roger B.A. Klorese
Senior Product Manager
SUSE
705 5th Ave S, Suite 1000
Seattle WA 98104
(P)+1 206.217.7432
(M)+1 425.444.5493
roger.klorese at suse.com
Schedule a meeting: https://doodle.com/RogerKlorese
GPG Key: D567 F186 A6AE D244 067E 95E4 E67D 019F 0670 D9CC
________________________________
From: egov-devops <egov-devops at akdb.de>
Sent: Monday, July 15, 2019 4:50 AM
To: caasp-beta at lists.suse.com
Cc: Roger Klorese
Subject: AW: [caasp-beta] Q: Any skuba bootstrap error with kubeadmn known?
Howdy list members, Howdy Roger.
Many thanks to Roger for his obvious tip, I couldn’t see it, cause I’m so so so frustrated with my progress of my caasp 4 beta installation. ;-/
Roger‘s tip worked as advertised. Now I know my problem, but I couldn’t find a solution, yet. It seems that my changed kubeadm-init.conf is reset before it is copied to the caasp-master1. Hence my own image repository (10.255.1.2:5000), which I changed (as mentioned in CaaSP-CAP-POC-Setup-Guide.pdf) beforehand, is reverted to registry.suse.com. Which in our airgap sitiuation is not available. Hence, the bootstrap fails.
Output of skuba with –v 5:
…
I0715 13:15:55.634379 26691 deployments.go:50] uploading local file "kubeadm-init.conf" to remote file "/tmp/kubeadm-init.conf"
I0715 13:15:55.634476 26691 files.go:29] uploading to remote file "/tmp/kubeadm-init.conf" with contents
I0715 13:15:55.779811 26691 ssh.go:167] running command: "sudo sh -c 'kubeadm init --config /tmp/kubeadm-init.conf --skip-token-print '"
I0715 13:15:55.874646 26691 ssh.go:190] stdout | [init] Using Kubernetes version: v1.14.1
I0715 13:15:55.874743 26691 ssh.go:190] stdout | [preflight] Running pre-flight checks
I0715 13:15:55.940883 26691 ssh.go:190] stderr | [WARNING Hostname]: hostname "master1" could not be reached
I0715 13:15:55.940915 26691 ssh.go:190] stderr | [WARNING Hostname]: hostname "master1": lookup master1 on 10.255.1.253:53: no such host
I0715 13:15:56.090490 26691 ssh.go:190] stdout | [preflight] Pulling images required for setting up a Kubernetes cluster
I0715 13:15:56.090516 26691 ssh.go:190] stdout | [preflight] This might take a minute or two, depending on the speed of your internet connection
I0715 13:15:56.090522 26691 ssh.go:190] stdout | [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
I0715 13:15:56.236126 26691 ssh.go:190] stderr | error execution phase preflight: [preflight] Some fatal errors occurred:
I0715 13:15:56.236152 26691 ssh.go:190] stderr | [ERROR ImagePull]: failed to pull image registry.suse.com/caasp/v4/hyperkube:v1.14.1: output: time="2019-07-15T13:16:03+02:00" level=fatal msg="pulling image failed: rpc error: code = Unknown desc = pinging docker registry returned: Get https://registry.suse.com/v2/: dial tcp: lookup registry.suse.com on 10.255.1.253:53: no such host"
I0715 13:15:56.236162 26691 ssh.go:190] stderr | , error: exit status 1
I0715 13:15:56.236172 26691 ssh.go:190] stderr | [ERROR ImagePull]: failed to pull image registry.suse.com/caasp/v4/pause:3.1: output: time="2019-07-15T13:16:03+02:00" level=fatal msg="pulling image failed: rpc error: code = Unknown desc = pinging docker registry returned: Get https://registry.suse.com/v2/: dial tcp: lookup registry.suse.com on 10.255.1.253:53: no such host"
I0715 13:15:56.236177 26691 ssh.go:190] stderr | , error: exit status 1
I0715 13:15:56.236183 26691 ssh.go:190] stderr | [ERROR ImagePull]: failed to pull image registry.suse.com/caasp/v4/etcd:3.3.11: output: time="2019-07-15T13:16:03+02:00" level=fatal msg="pulling image failed: rpc error: code = Unknown desc = pinging docker registry returned: Get https://registry.suse.com/v2/: dial tcp: lookup registry.suse.com on 10.255.1.253:53: no such host"
I0715 13:15:56.236189 26691 ssh.go:190] stderr | , error: exit status 1
I0715 13:15:56.236198 26691 ssh.go:190] stderr | [ERROR ImagePull]: failed to pull image registry.suse.com/caasp/v4/coredns:1.2.6: output: time="2019-07-15T13:16:03+02:00" level=fatal msg="pulling image failed: rpc error: code = Unknown desc = pinging docker registry returned: Get https://registry.suse.com/v2/: dial tcp: lookup registry.suse.com on 10.255.1.253:53: no such host"
I0715 13:15:56.236204 26691 ssh.go:190] stderr | , error: exit status 1
I0715 13:15:56.236210 26691 ssh.go:190] stderr | [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
I0715 13:15:56.238552 26691 ssh.go:167] running command: "sudo sh -c 'rm /tmp/kubeadm-init.conf'"
F0715 13:15:56.321598 26691 bootstrap.go:48] error bootstraping node: failed to apply state kubeadm.init: Process exited with status 1
The registry is available and working:
caasp-master1:/tmp # curl --insecure https://10.255.1.2:5000/v2/_catalog
{
"repositories": [
"cap/cf-usb-sidecar-mysql",
"cap/cf-usb-sidecar-mysql-setup",
"cap/cf-usb-sidecar-postgres",
"cap/cf-usb-sidecar-postgres-setup",
"cap/scf-adapter",
"cap/scf-api-group",
"cap/scf-autoscaler-api",
"cap/scf-autoscaler-eventgenerator",
"cap/scf-autoscaler-metrics",
"cap/scf-autoscaler-operator",
"cap/scf-autoscaler-postgres",
"cap/scf-autoscaler-scalingengine",
"cap/scf-autoscaler-scheduler",
"cap/scf-autoscaler-servicebroker",
"cap/scf-bits",
"cap/scf-blobstore",
"cap/scf-cc-clock",
"cap/scf-cc-uploader",
"cap/scf-cc-worker",
"cap/scf-cf-usb-group",
"cap/scf-credhub-user",
"cap/scf-diego-api",
"cap/scf-diego-brain",
"cap/scf-diego-cell",
"cap/scf-diego-ssh",
"cap/scf-doppler",
"cap/scf-eirini",
"cap/scf-locket",
"cap/scf-log-api",
"cap/scf-log-cache-scheduler",
"cap/scf-loggregator-agent",
"cap/scf-mysql",
"cap/scf-nats",
"cap/scf-nfs-broker",
"cap/scf-post-deployment-setup",
"cap/scf-router",
"cap/scf-routing-api",
"cap/scf-secret-generation",
"cap/scf-syslog-scheduler",
"cap/scf-tcp-router",
"cap/stratos-console",
"cap/stratos-jetstream",
"cap/stratos-mariadb",
"cap/stratos-metrics-configmap-reload",
"cap/stratos-metrics-firehose-exporter",
"cap/stratos-metrics-firehose-init",
"cap/stratos-metrics-init-chown-data",
"cap/stratos-metrics-nginx",
"cap/stratos-metrics-prometheus",
"cap/stratos-postflight-job",
"cap/uaa-mysql",
"cap/uaa-secret-generation",
"cap/uaa-uaa",
"gcr.io/google-containers/busybox",
"gcr.io/kubernetes-helm/tiller",
"k8s.gcr.io/kubernetes-dashboard-amd64",
"mysql",
"quay.io/external_storage/nfs-client-provisioner",
"registry.suse.com/caasp/v4/caaspctl-tooling",
"registry.suse.com/caasp/v4/cilium",
"registry.suse.com/caasp/v4/cilium-etcd-operator",
"registry.suse.com/caasp/v4/cilium-init",
"registry.suse.com/caasp/v4/cilium-operator",
"registry.suse.com/caasp/v4/coredns",
"registry.suse.com/caasp/v4/default-http-backend",
"registry.suse.com/caasp/v4/dnsmasq-nanny",
"registry.suse.com/caasp/v4/etcd",
"registry.suse.com/caasp/v4/flannel",
"registry.suse.com/caasp/v4/haproxy",
"registry.suse.com/caasp/v4/hyperkube",
"registry.suse.com/caasp/v4/kubedns",
"registry.suse.com/caasp/v4/kured",
"registry.suse.com/caasp/v4/mariadb",
"registry.suse.com/caasp/v4/nginx-ingress-controller",
"registry.suse.com/caasp/v4/openldap",
"registry.suse.com/caasp/v4/pause",
"registry.suse.com/caasp/v4/registry",
"registry.suse.com/caasp/v4/salt-api",
"registry.suse.com/caasp/v4/salt-master",
"registry.suse.com/caasp/v4/salt-minion",
"registry.suse.com/caasp/v4/sidecar",
"registry.suse.com/caasp/v4/tiller",
"splatform/minibroker"
]
}
Does anyone have an idea how to keep the changed kubeadm-init.conf, hence my replaced 10.255.1.2:5000 registry entries, intact?
Best regards,
Alexander Schießl
Team DevOps
Geschäftsfeld eGovernment
Telefon +49 800 2553222-65
egov-devops at adkb.de<mailto:egov-devops at adkb.de>
__________________________________________________________
AKDB · Anstalt des öffentlichen Rechts · Hauptverwaltung
Hansastraße 12-16 · 80686 München · www.akdb.de<http://www.akdb.de/>
Von: Roger Klorese <roger.klorese at suse.com>
Gesendet: Freitag, 12. Juli 2019 10:37
An: egov-devops <egov-devops at akdb.de>; caasp-beta at lists.suse.com
Cc: Gergo Kovacs <gergo.kovacs at suse.com>
Betreff: Re: [caasp-beta] Q: Any skuba bootstrap error with kubeadmn known?
Often, the default terseness of skuba hides what can be very simple errors to handle from the underlying tools. Try running skuba vith "-v 5" and see if you get a more useful message. Once, I got this message because I missed a step; another time, it was because I had a typo in an option...
Roger B.A. Klorese
Senior Product Manager
SUSE
705 5th Ave S, Suite 1000
Seattle WA 98104
(P)+1 206.217.7432
(M)+1 425.444.5493
roger.klorese at suse.com<mailto:roger.klorese at suse.com>
Schedule a meeting: https://doodle.com/RogerKlorese
GPG Key: D567 F186 A6AE D244 067E 95E4 E67D 019F 0670 D9CC
________________________________
From: caasp-beta <caasp-beta-bounces at lists.suse.com<mailto:caasp-beta-bounces at lists.suse.com>> on behalf of egov-devops <egov-devops at akdb.de<mailto:egov-devops at akdb.de>>
Sent: Friday, July 12, 2019 1:16 AM
To: caasp-beta at lists.suse.com<mailto:caasp-beta at lists.suse.com>
Cc: Gergo Kovacs
Subject: [caasp-beta] Q: Any skuba bootstrap error with kubeadmn known?
Howdy,
I’m trying to bootstrap a caasp 4 beta 3 cluster (https://susedoc.github.io/doc-caasp/beta/caasp-deployment/single-html/#bootstrap). But I get an error:
„F0712 09:46:39.270173 13949 bootstrap.go:48] error bootstraping node: failed to apply state kubeadm.init: Process exited with status 1“
So far I have done:
1. ssh-keygen; ssh-add; ssh-copy-id; ssh via sles from „admin“ sles server to master{1-3} and node{1-3} via user sles is working
2. Own RMT Server with images as described in CaaSP-CAP-POC-Setup-Guide.pdf setup and working
3. skuba cluster init --control-plane <LB IP/FQDN> my-cluster -> LB IP we use;
4. Skipped kured (4.1.2.1) and prometheus (4.1.3) changes
5. cd my-cluster
6. skuba node bootstrap --user sles --sudo --target <IP/FQDN> <NODE NAME> -> IP of master1; NAME master1
==> skuba node bootstrap --user sles --sudo --target 10.255.1.3 master1
==> ** This is a BETA release and NOT intended for production usage. **
==> [bootstrap] updating init configuration with target information
==> [bootstrap] writing init configuration for node
==> [bootstrap] applying init configuration to node
==> F0712 09:46:39.270173 13949 bootstrap.go:48] error bootstraping node: failed to apply state kubeadm.init: Process exited with status 1
Master1 setup:
- OS SLES 15 SP1
- SUSEConnect stuff as described here (https://susedoc.github.io/doc-caasp/beta/caasp-deployment/single-html/#_preparation_of_the_vm_as_a_template)
- SUSEConnect --list-extensions
AVAILABLE EXTENSIONS AND MODULES
Basesystem Module 15 SP1 x86_64 (Activated)
Deactivate with: SUSEConnect -d -p sle-module-basesystem/15.1/x86_64
Containers Module 15 SP1 x86_64 (Activated)
Deactivate with: SUSEConnect -d -p /15.1/x86_64
SUSE CaaS Platform 4.0 x86_64 (BETA) (Activated)
Deactivate with: SUSEConnect -d -p caasp/4.0/x86_64
Server Applications Module 15 SP1 x86_64 (Activated)
Deactivate with: SUSEConnect -d -p sle-module-server-applications/15.1/x86_64
Public Cloud Module 15 SP1 x86_64 (Activated)
Deactivate with: SUSEConnect -d -p sle-module-public-cloud/15.1/x86_64
- sudo zypper in kubernetes-kubeadm kubernetes-kubelet kubernetes-client cri-o cni-plugins
Refreshing service 'SMT-http_smt_akdb_net'.
Retrieving repository 'SLE-Module-Basesystem15-SP1-Updates' metadata .......................................[done]
Building repository 'SLE-Module-Basesystem15-SP1-Updates' cache ............................................[done]
Retrieving repository 'SLE-Module-Containers15-SP1-Updates' metadata .......................................[done]
Building repository 'SLE-Module-Containers15-SP1-Updates' cache ............................................[done]
Retrieving repository 'SLE-Module-Server-Applications15-SP1-Updates' metadata ..............................[done]
Building repository 'SLE-Module-Server-Applications15-SP1-Updates' cache ...................................[done]
Loading repository data...
Reading installed packages...
'cni-plugins' is already installed.
No update candidate for 'cni-plugins-0.7.4-1.11.x86_64'. The highest available version is already installed.
'kubernetes-kubelet' is already installed.
No update candidate for 'kubernetes-kubelet-1.14.1-1.8.x86_64'. The highest available version is already installed.
'cri-o' is already installed.
No update candidate for 'cri-o-1.14.1-1.2.x86_64'. The highest available version is already installed.
'kubernetes-kubeadm' is already installed.
No update candidate for 'kubernetes-kubeadm-1.14.1-1.8.x86_64'. The highest available version is already installed.
'kubernetes-client' is already installed.
No update candidate for 'kubernetes-client-1.14.1-1.8.x86_64'. The highest available version is already installed.
Resolving package dependencies...
Nothing to do.
Any idea, what I did wrong? What can I check? What can I fix to make it work?
Mit freundlichen Grüßen
Alexander Schießl
Team DevOps
Geschäftsfeld eGovernment
Telefon +49 800 2553222-65
egov-devops at adkb.de<mailto:egov-devops at adkb.de>
__________________________________________________________
AKDB · Anstalt des öffentlichen Rechts · Hauptverwaltung
Hansastraße 12-16 · 80686 München · www.akdb.de<http://www.akdb.de/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.suse.com/pipermail/caasp-beta/attachments/20190715/004a70d2/attachment.html>
More information about the caasp-beta
mailing list