[caasp-beta] Q: Any skuba bootstrap error with kubeadmn known?

egov-devops egov-devops at akdb.de
Mon Jul 15 05:50:56 MDT 2019


Howdy list members, Howdy Roger.

Many thanks to Roger for his obvious tip, I couldn't see it, cause I'm so so so frustrated with my progress of my caasp 4 beta installation. ;-/

Roger's tip worked as advertised. Now I know my problem, but I couldn't find a solution, yet. It seems that my changed kubeadm-init.conf is reset before it is copied to the caasp-master1. Hence my own image repository (10.255.1.2:5000), which I changed (as mentioned in CaaSP-CAP-POC-Setup-Guide.pdf) beforehand, is reverted to registry.suse.com. Which in our airgap sitiuation is not available. Hence, the bootstrap fails.

Output of skuba with -v 5:
...
I0715 13:15:55.634379   26691 deployments.go:50] uploading local file "kubeadm-init.conf" to remote file "/tmp/kubeadm-init.conf"
I0715 13:15:55.634476   26691 files.go:29] uploading to remote file "/tmp/kubeadm-init.conf" with contents
I0715 13:15:55.779811   26691 ssh.go:167] running command: "sudo sh -c 'kubeadm init --config /tmp/kubeadm-init.conf --skip-token-print '"
I0715 13:15:55.874646   26691 ssh.go:190] stdout | [init] Using Kubernetes version: v1.14.1
I0715 13:15:55.874743   26691 ssh.go:190] stdout | [preflight] Running pre-flight checks
I0715 13:15:55.940883   26691 ssh.go:190] stderr |   [WARNING Hostname]: hostname "master1" could not be reached
I0715 13:15:55.940915   26691 ssh.go:190] stderr |   [WARNING Hostname]: hostname "master1": lookup master1 on 10.255.1.253:53: no such host
I0715 13:15:56.090490   26691 ssh.go:190] stdout | [preflight] Pulling images required for setting up a Kubernetes cluster
I0715 13:15:56.090516   26691 ssh.go:190] stdout | [preflight] This might take a minute or two, depending on the speed of your internet connection
I0715 13:15:56.090522   26691 ssh.go:190] stdout | [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
I0715 13:15:56.236126   26691 ssh.go:190] stderr | error execution phase preflight: [preflight] Some fatal errors occurred:
I0715 13:15:56.236152   26691 ssh.go:190] stderr |   [ERROR ImagePull]: failed to pull image registry.suse.com/caasp/v4/hyperkube:v1.14.1: output: time="2019-07-15T13:16:03+02:00" level=fatal msg="pulling image failed: rpc error: code = Unknown desc = pinging docker registry returned: Get https://registry.suse.com/v2/: dial tcp: lookup registry.suse.com on 10.255.1.253:53: no such host"
I0715 13:15:56.236162   26691 ssh.go:190] stderr | , error: exit status 1
I0715 13:15:56.236172   26691 ssh.go:190] stderr |   [ERROR ImagePull]: failed to pull image registry.suse.com/caasp/v4/pause:3.1: output: time="2019-07-15T13:16:03+02:00" level=fatal msg="pulling image failed: rpc error: code = Unknown desc = pinging docker registry returned: Get https://registry.suse.com/v2/: dial tcp: lookup registry.suse.com on 10.255.1.253:53: no such host"
I0715 13:15:56.236177   26691 ssh.go:190] stderr | , error: exit status 1
I0715 13:15:56.236183   26691 ssh.go:190] stderr |   [ERROR ImagePull]: failed to pull image registry.suse.com/caasp/v4/etcd:3.3.11: output: time="2019-07-15T13:16:03+02:00" level=fatal msg="pulling image failed: rpc error: code = Unknown desc = pinging docker registry returned: Get https://registry.suse.com/v2/: dial tcp: lookup registry.suse.com on 10.255.1.253:53: no such host"
I0715 13:15:56.236189   26691 ssh.go:190] stderr | , error: exit status 1
I0715 13:15:56.236198   26691 ssh.go:190] stderr |   [ERROR ImagePull]: failed to pull image registry.suse.com/caasp/v4/coredns:1.2.6: output: time="2019-07-15T13:16:03+02:00" level=fatal msg="pulling image failed: rpc error: code = Unknown desc = pinging docker registry returned: Get https://registry.suse.com/v2/: dial tcp: lookup registry.suse.com on 10.255.1.253:53: no such host"
I0715 13:15:56.236204   26691 ssh.go:190] stderr | , error: exit status 1
I0715 13:15:56.236210   26691 ssh.go:190] stderr | [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
I0715 13:15:56.238552   26691 ssh.go:167] running command: "sudo sh -c 'rm /tmp/kubeadm-init.conf'"
F0715 13:15:56.321598   26691 bootstrap.go:48] error bootstraping node: failed to apply state kubeadm.init: Process exited with status 1

The registry is available and working:
caasp-master1:/tmp # curl --insecure https://10.255.1.2:5000/v2/_catalog
{
  "repositories": [
    "cap/cf-usb-sidecar-mysql",
    "cap/cf-usb-sidecar-mysql-setup",
    "cap/cf-usb-sidecar-postgres",
    "cap/cf-usb-sidecar-postgres-setup",
    "cap/scf-adapter",
    "cap/scf-api-group",
    "cap/scf-autoscaler-api",
    "cap/scf-autoscaler-eventgenerator",
    "cap/scf-autoscaler-metrics",
    "cap/scf-autoscaler-operator",
    "cap/scf-autoscaler-postgres",
    "cap/scf-autoscaler-scalingengine",
    "cap/scf-autoscaler-scheduler",
    "cap/scf-autoscaler-servicebroker",
    "cap/scf-bits",
    "cap/scf-blobstore",
    "cap/scf-cc-clock",
    "cap/scf-cc-uploader",
   "cap/scf-cc-worker",
    "cap/scf-cf-usb-group",
    "cap/scf-credhub-user",
    "cap/scf-diego-api",
    "cap/scf-diego-brain",
    "cap/scf-diego-cell",
    "cap/scf-diego-ssh",
    "cap/scf-doppler",
    "cap/scf-eirini",
    "cap/scf-locket",
    "cap/scf-log-api",
    "cap/scf-log-cache-scheduler",
    "cap/scf-loggregator-agent",
    "cap/scf-mysql",
    "cap/scf-nats",
    "cap/scf-nfs-broker",
    "cap/scf-post-deployment-setup",
    "cap/scf-router",
    "cap/scf-routing-api",
    "cap/scf-secret-generation",
    "cap/scf-syslog-scheduler",
    "cap/scf-tcp-router",
    "cap/stratos-console",
    "cap/stratos-jetstream",
    "cap/stratos-mariadb",
    "cap/stratos-metrics-configmap-reload",
    "cap/stratos-metrics-firehose-exporter",
    "cap/stratos-metrics-firehose-init",
    "cap/stratos-metrics-init-chown-data",
    "cap/stratos-metrics-nginx",
    "cap/stratos-metrics-prometheus",
    "cap/stratos-postflight-job",
    "cap/uaa-mysql",
    "cap/uaa-secret-generation",
    "cap/uaa-uaa",
    "gcr.io/google-containers/busybox",
    "gcr.io/kubernetes-helm/tiller",
    "k8s.gcr.io/kubernetes-dashboard-amd64",
    "mysql",
    "quay.io/external_storage/nfs-client-provisioner",
    "registry.suse.com/caasp/v4/caaspctl-tooling",
    "registry.suse.com/caasp/v4/cilium",
    "registry.suse.com/caasp/v4/cilium-etcd-operator",
    "registry.suse.com/caasp/v4/cilium-init",
    "registry.suse.com/caasp/v4/cilium-operator",
    "registry.suse.com/caasp/v4/coredns",
    "registry.suse.com/caasp/v4/default-http-backend",
    "registry.suse.com/caasp/v4/dnsmasq-nanny",
    "registry.suse.com/caasp/v4/etcd",
    "registry.suse.com/caasp/v4/flannel",
    "registry.suse.com/caasp/v4/haproxy",
    "registry.suse.com/caasp/v4/hyperkube",
    "registry.suse.com/caasp/v4/kubedns",
    "registry.suse.com/caasp/v4/kured",
    "registry.suse.com/caasp/v4/mariadb",
    "registry.suse.com/caasp/v4/nginx-ingress-controller",
    "registry.suse.com/caasp/v4/openldap",
    "registry.suse.com/caasp/v4/pause",
    "registry.suse.com/caasp/v4/registry",
    "registry.suse.com/caasp/v4/salt-api",
    "registry.suse.com/caasp/v4/salt-master",
    "registry.suse.com/caasp/v4/salt-minion",
    "registry.suse.com/caasp/v4/sidecar",
    "registry.suse.com/caasp/v4/tiller",
    "splatform/minibroker"
  ]
}

Does anyone have an idea how to keep the changed kubeadm-init.conf, hence my replaced 10.255.1.2:5000 registry entries, intact?

Best regards,

Alexander Schießl
Team DevOps
Geschäftsfeld eGovernment

Telefon +49 800 2553222-65
egov-devops at adkb.de<mailto:egov-devops at adkb.de>
__________________________________________________________

AKDB · Anstalt des öffentlichen Rechts · Hauptverwaltung
Hansastraße 12-16 · 80686 München · www.akdb.de<http://www.akdb.de/>



Von: Roger Klorese <roger.klorese at suse.com>
Gesendet: Freitag, 12. Juli 2019 10:37
An: egov-devops <egov-devops at akdb.de>; caasp-beta at lists.suse.com
Cc: Gergo Kovacs <gergo.kovacs at suse.com>
Betreff: Re: [caasp-beta] Q: Any skuba bootstrap error with kubeadmn known?

Often, the default terseness of skuba hides what can be very simple errors to handle from the underlying tools. Try running skuba vith "-v 5" and see if you get a more useful message. Once, I got this message because I missed a step; another time, it was because I had a typo in an option...

Roger B.A. Klorese
Senior Product Manager
SUSE
705 5th Ave S, Suite 1000
Seattle WA 98104
(P)+1 206.217.7432
(M)+1 425.444.5493
roger.klorese at suse.com<mailto:roger.klorese at suse.com>
Schedule a meeting: https://doodle.com/RogerKlorese
GPG Key: D567 F186 A6AE D244 067E  95E4 E67D 019F 0670 D9CC

________________________________
From: caasp-beta <caasp-beta-bounces at lists.suse.com<mailto:caasp-beta-bounces at lists.suse.com>> on behalf of egov-devops <egov-devops at akdb.de<mailto:egov-devops at akdb.de>>
Sent: Friday, July 12, 2019 1:16 AM
To: caasp-beta at lists.suse.com<mailto:caasp-beta at lists.suse.com>
Cc: Gergo Kovacs
Subject: [caasp-beta] Q: Any skuba bootstrap error with kubeadmn known?


Howdy,



I'm trying to bootstrap a caasp 4 beta 3 cluster (https://susedoc.github.io/doc-caasp/beta/caasp-deployment/single-html/#bootstrap). But I get an error:

"F0712 09:46:39.270173   13949 bootstrap.go:48] error bootstraping node: failed to apply state kubeadm.init: Process exited with status 1"



So far I have done:

1.  ssh-keygen; ssh-add; ssh-copy-id; ssh via sles from "admin" sles server to master{1-3} and node{1-3} via user sles is working

2.      Own RMT Server with images as described in CaaSP-CAP-POC-Setup-Guide.pdf setup and working

3.  skuba cluster init --control-plane <LB IP/FQDN> my-cluster -> LB IP we use;

4.      Skipped kured (4.1.2.1) and prometheus (4.1.3) changes

5.  cd my-cluster

6.  skuba node bootstrap --user sles --sudo --target <IP/FQDN> <NODE NAME> -> IP of master1; NAME master1

==> skuba node bootstrap --user sles --sudo --target 10.255.1.3 master1

==> ** This is a BETA release and NOT intended for production usage. **

==> [bootstrap] updating init configuration with target information

==> [bootstrap] writing init configuration for node

==> [bootstrap] applying init configuration to node

==> F0712 09:46:39.270173   13949 bootstrap.go:48] error bootstraping node: failed to apply state kubeadm.init: Process exited with status 1



Master1 setup:

-   OS SLES 15 SP1

-   SUSEConnect stuff as described here (https://susedoc.github.io/doc-caasp/beta/caasp-deployment/single-html/#_preparation_of_the_vm_as_a_template)

-   SUSEConnect --list-extensions

AVAILABLE EXTENSIONS AND MODULES



    Basesystem Module 15 SP1 x86_64 (Activated)

    Deactivate with: SUSEConnect -d -p sle-module-basesystem/15.1/x86_64



        Containers Module 15 SP1 x86_64 (Activated)

        Deactivate with: SUSEConnect -d -p     /15.1/x86_64



            SUSE CaaS Platform 4.0 x86_64 (BETA) (Activated)

            Deactivate with: SUSEConnect -d -p caasp/4.0/x86_64



        Server Applications Module 15 SP1 x86_64 (Activated)

        Deactivate with: SUSEConnect -d -p sle-module-server-applications/15.1/x86_64



            Public Cloud Module 15 SP1 x86_64 (Activated)

            Deactivate with: SUSEConnect -d -p sle-module-public-cloud/15.1/x86_64

-   sudo zypper in kubernetes-kubeadm kubernetes-kubelet kubernetes-client cri-o cni-plugins
Refreshing service 'SMT-http_smt_akdb_net'.

Retrieving repository 'SLE-Module-Basesystem15-SP1-Updates' metadata .......................................[done]

Building repository 'SLE-Module-Basesystem15-SP1-Updates' cache ............................................[done]

Retrieving repository 'SLE-Module-Containers15-SP1-Updates' metadata .......................................[done]

Building repository 'SLE-Module-Containers15-SP1-Updates' cache ............................................[done]

Retrieving repository 'SLE-Module-Server-Applications15-SP1-Updates' metadata ..............................[done]

Building repository 'SLE-Module-Server-Applications15-SP1-Updates' cache ...................................[done]

Loading repository data...

Reading installed packages...

'cni-plugins' is already installed.

No update candidate for 'cni-plugins-0.7.4-1.11.x86_64'. The highest available version is already installed.

'kubernetes-kubelet' is already installed.

No update candidate for 'kubernetes-kubelet-1.14.1-1.8.x86_64'. The highest available version is already installed.

'cri-o' is already installed.

No update candidate for 'cri-o-1.14.1-1.2.x86_64'. The highest available version is already installed.

'kubernetes-kubeadm' is already installed.

No update candidate for 'kubernetes-kubeadm-1.14.1-1.8.x86_64'. The highest available version is already installed.

'kubernetes-client' is already installed.

No update candidate for 'kubernetes-client-1.14.1-1.8.x86_64'. The highest available version is already installed.

Resolving package dependencies...



Nothing to do.



Any idea, what I did wrong? What can I check? What can I fix to make it work?



Mit freundlichen Grüßen



Alexander Schießl

Team DevOps

Geschäftsfeld eGovernment



Telefon +49 800 2553222-65

egov-devops at adkb.de<mailto:egov-devops at adkb.de>

__________________________________________________________



AKDB · Anstalt des öffentlichen Rechts · Hauptverwaltung

Hansastraße 12-16 · 80686 München · www.akdb.de<http://www.akdb.de/>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.suse.com/pipermail/caasp-beta/attachments/20190715/9dd86ed2/attachment.html>


More information about the caasp-beta mailing list