[caasp-beta] Q: CAASP4B4 new cluster / join problem

egov-devops egov-devops at akdb.de
Thu Jul 18 03:20:01 MDT 2019


Full output:
-----------8<----------
caasp-admin:~/caasp4b4 # skuba node join -v 5 --role master --user sles --sudo --target 10.255.1.4 caasp-master2
** This is a BETA release and NOT intended for production usage. **
I0718 08:05:16.208668   20242 loader.go:359] Config loaded from file:  admin.conf
I0718 08:05:16.209494   20242 round_trippers.go:416] GET https://10.255.1.2:6443/api/v1/nodes/caasp-master2
I0718 08:05:16.209504   20242 round_trippers.go:423] Request Headers:
I0718 08:05:16.209510   20242 round_trippers.go:426]     Accept: application/json, */*
I0718 08:05:16.209517   20242 round_trippers.go:426]     User-Agent: skuba/v0.0.0 (linux/amd64) kubernetes/fd919f4I0718 08:05:16.221502   20242 round_trippers.go:441] Response Status: 404 Not Found in 11 milliseconds
[join] applying states to new node
I0718 08:05:16.221927   20242 states.go:35] === applying state kubernetes.join.upload-secrets ===
I0718 08:05:16.221937   20242 deployments.go:50] uploading local file "pki/ca.crt" to remote file "/etc/kubernetes/pki/ca.crt"
I0718 08:05:16.221988   20242 files.go:29] uploading to remote file "/etc/kubernetes/pki/ca.crt" with contents
[join] failed to apply join to node failed to apply state kubernetes.join.upload-secrets: Process exited with status 1
F0718 08:05:16.503251   20242 join.go:50] error joining node caasp-master2: failed to apply state kubernetes.join.upload-secrets: Process exited with status 1
-----------8<----------

LB Config:
-----------8<----------
user  nginx;
worker_processes  auto;

# load_module lib64/nginx/modules/ngx_http_fancyindex_module.so;
# load_module lib64/nginx/modules/ngx_http_geoip_module.so;
# load_module lib64/nginx/modules/ngx_http_headers_more_filter_module.so;
# load_module lib64/nginx/modules/ngx_http_image_filter_module.so;
# load_module lib64/nginx/modules/ngx_http_perl_module.so;
# load_module lib64/nginx/modules/ngx_http_xslt_filter_module.so;
# load_module lib64/nginx/modules/ngx_mail_module.so;
# load_module lib64/nginx/modules/ngx_rtmp_module.so;
# load_module lib64/nginx/modules/ngx_stream_geoip_module.so;
load_module lib64/nginx/modules/ngx_stream_module.so;

#error_log  /var/log/nginx/error.log;
#error_log  /var/log/nginx/error.log  notice;
#error_log  /var/log/nginx/error.log  info;

#pid        /run/nginx.pid;


events {
    worker_connections  1024;
    use epoll;
}

stream {
    log_format proxy '$remote_addr [$time_local] '
                     '$protocol $status $bytes_sent $bytes_received '
                     '$session_time "$upstream_addr"';

    error_log  /var/log/nginx/k8s-masters-lb-error.log;
    access_log /var/log/nginx/k8s-masters-lb-access.log proxy;

    upstream k8s-masters {
        #hash $remote_addr consistent;
        server caasp-master1:6443 weight=1 max_fails=1;
        server caasp-master2:6443 weight=1 max_fails=1;
        server caasp-master3:6443 weight=1 max_fails=1;
    }

    server {
        listen 6443;
        proxy_connect_timeout 1s;
        proxy_timeout 3s;
        proxy_pass k8s-masters;
    }
}
-----------8<----------
caasp-admin:~/caasp4b4 # systemctl status nginx.service
● nginx.service - The nginx HTTP and reverse proxy server
   Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2019-07-02 17:29:59 CEST; 2 weeks 1 days ago
 Main PID: 1248 (nginx)
    Tasks: 5 (limit: 4915)
   Memory: 22.5M
      CPU: 2min 31.546s
   CGroup: /system.slice/nginx.service
           ├─1248 nginx: master process /usr/sbin/nginx -g daemon off;
           ├─1249 nginx: worker process
           ├─1250 nginx: worker process
           ├─1251 nginx: worker process
           └─1252 nginx: worker process

Jul 02 17:29:59 caasp-admin systemd[1]: Starting The nginx HTTP and reverse proxy server...
Jul 02 17:29:59 caasp-admin nginx[1239]: nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
Jul 02 17:29:59 caasp-admin nginx[1239]: nginx: configuration file /etc/nginx/nginx.conf test is successful
Jul 02 17:29:59 caasp-admin systemd[1]: Started The nginx HTTP and reverse proxy server.
-----------8<----------
/etc/hosts
...
10.255.1.2      admin
10.255.1.3      master1 caasp-master1
10.255.1.4      master2 caasp-master2
10.255.1.5      master3 caasp-master3
10.255.2.14     node1 caasp-node1
10.255.2.15     node2 caasp-node2
10.255.2.16     node3 caasp-node3
-----------8<----------

/AS

-----Ursprüngliche Nachricht-----
Von: caasp-beta <caasp-beta-bounces at lists.suse.com> Im Auftrag von Ludovic Cavajani
Gesendet: Donnerstag, 18. Juli 2019 11:09
An: caasp-beta at lists.suse.com
Betreff: Re: [caasp-beta] Q: CAASP4B4 new cluster / join problem

Hi,

Updating from `beta3` to `beta4` is possible so there is no need to reset everything. The documentation can be found here:
https://susedoc.github.io/doc-caasp/beta/caasp-admin/single-html/#_cluster_updates

@egov-devops I see you used `-v5` flag, could you share the entire logs if there are no sensitive information from your side ? Is `10.255.1.2` correctly configured as an LB with the 3 masters nodes ? Could you share the configuration if possible ?

Thanks,


On 7/18/19 10:43 AM, Jean Marc Lambert wrote:
> One element, is also, (I think it is not yet documented, as I 
> discovered it yesterday) that you need to update the skuba itself on your 'CLI' node. New features in it. Like upgrade plans etc There is a skuba node reset that you may use to reinit the node.
> JMarc
>
> De : caasp-beta <caasp-beta-bounces at lists.suse.com> De la part de 
> egov-devops Envoyé : jeudi 18 juillet 2019 08:40 À : 
> caasp-beta at lists.suse.com Cc : Gergo Kovacs <gergo.kovacs at suse.com> 
> Objet : [caasp-beta] Q: CAASP4B4 new cluster / join problem
>
> Howdy,
>
> yesterday, finally I got our Airgap install of CAASP4B3 working (only 1 master so far) - Needed to solve internal (AKDB) and external (SUSE) problems to make it work.
>
> Today I learned that CASSP4B4 is out, so I tried to set up CAASP4B4 fresh.
>
> What have I done so far:
> 1) made a "kubeadm reset" on master1. To reset the master I set up 
> with CAASP4B3, yesterday
> 2) updated our "fake" registry.suse.com with new images (see 
> attachment)
> 3) made a "skuba cluster init --control-plane 10.255.1.2 caasp4b4"
> 4) made a "skuba node bootstrap -v 5 --user sles --sudo --target 10.255.1.3 caasp-master1"
>    --> [bootstrap] successfully bootstrapped node "10.255.1.3" with Kubernetes: "1.15.0"
> 5) made a "skuba node join -v 5 --role master --user sles --sudo --target 10.255.1.4 caasp-master2"
> --> I0718 08:05:16.221988   20242 files.go:29] uploading to remote file "/etc/kubernetes/pki/ca.crt" with contents [join] failed to apply join to node failed to apply state kubernetes.join.upload-secrets: Process exited with status 1
> --> F0718 08:05:16.503251   20242 join.go:50] error joining node caasp-master2: failed to apply state kubernetes.join. upload-secrets: Process exited with status 1
>
> Some Background:
>
>   *   ssh from "caasp-admin" via pubkey as sles or root works fine.
>
> -       Each planned master and node is installed/updated with "zypper in kubernetes-kubeadm kubernetes-kubelet kubernetes-client cri-o cni-plugins sudo"
>
>   *   SUSECOnnect activated:
>
> caasp-master1:~ # SUSEConnect --list-extensions
>
> AVAILABLE EXTENSIONS AND MODULES
>
>
>
>     Basesystem Module 15 SP1 x86_64 (Activated)
>
>     Deactivate with: SUSEConnect -d -p 
> sle-module-basesystem/15.1/x86_64
>
>
>
>         Containers Module 15 SP1 x86_64 (Activated)
>
>         Deactivate with: SUSEConnect -d -p     /15.1/x86_64
>
>
>
>             SUSE CaaS Platform 4.0 x86_64 (BETA) (Activated)
>
>             Deactivate with: SUSEConnect -d -p caasp/4.0/x86_64
>
>
>
>         Server Applications Module 15 SP1 x86_64 (Activated)
>
>         Deactivate with: SUSEConnect -d -p 
> sle-module-server-applications/15.1/x86_64
>
>
>
>             Public Cloud Module 15 SP1 x86_64 (Activated)
>
>             Deactivate with: SUSEConnect -d -p 
> sle-module-public-cloud/15.1/x86_64
>
>
> Any idea what I made wrong? Where can I check? What can I fix? :-/
>
> Many Thanks...
>
> Mit freundlichen Grüßen
>
> Alexander Schießl
> Team DevOps
> Geschäftsfeld eGovernment
>
> Telefon +49 800 2553222-65
> egov-devops at adkb.de<mailto:egov-devops at adkb.de>
> __________________________________________________________
>
> AKDB * Anstalt des öffentlichen Rechts * Hauptverwaltung München 
> Hansastr. 12 - 16 * 80686 München * www.akdb.de<http://www.akdb.de/>
>
>
>
> _______________________________________________
> caasp-beta mailing list
> caasp-beta at lists.suse.com
> Check the mailing list archives or Unsubscribe at 
> http://lists.suse.com/mailman/listinfo/caasp-beta



More information about the caasp-beta mailing list