[caasp-beta] velum troubleshoot

Sat Jan 13 21:00:10 MST 2018

I confirmed the admin node reboot (even shutdown) doesn't cause to fail
kubectl (I even confirmed while the admin node is down, kubectl still
works, and I also checked kubectl conf points to k8s master node), so it
must be something related to when the admin node is updated (& subsequently
rebooted).
I will check more detail next time the admin node is updated and confirm
the situation. If you can try, you may also try kubectl soon after you
initiated the admin node update (& subsequent auto reboot) in the dashboard
during the process..

Regarding the dashboard loading (taking too long), I see something
potentially not clean with salt master connection, and I will check
further..
Velum::SaltApi::SaltConnectionException (Net::ReadTimeout):

Thanks,
Jerry

On Sat, Jan 13, 2018 at 1:30 PM, Jerry Hwang <special011 at gmail.com> wrote:

> Okay, what you described is also what I expected.
> Thanks for the idea.
>
> Regards,
> Jerry
>
> On Sat, Jan 13, 2018 at 1:23 PM, Liam McCafferty <
> Liam.McCafferty at smartodds.co.uk> wrote:
>
>> I have never noticed this before so  I just rebooted my admin node and I
>> can confirm I can still communicate via kubectl to the master node.
>>
>>
>> I have also checked my kube config in ~/.kube/config and I can
>> see that the cluster server is set to the master server. does
>> your kubeconfig point to your master node?  Upon checking my admin node I
>> see it also that it has the haproxy k8 containers which I believe deal
>> with routing to the k8 api. so if your config is pointed to your admin node
>> that would make sense that you cant reach the api server.
>>
>>
>>
>> Cheers,
>>
>>
>> Liam
>>
>>
>>
>> ------------------------------
>> *From:* Jerry Hwang <special011 at gmail.com>
>> *Sent:* 13 January 2018 21:11
>> *To:* Liam McCafferty
>> *Cc:* caasp-beta at lists.suse.com
>> *Subject:* Re: [caasp-beta] velum troubleshoot
>>
>> Right, that is also a good point. (I too much focused on the general
>> idea  :))
>>
>> And it seems I can't get to k8s cluster via kubectl (from my local
>> machine) during admin node upgrade & reboot, is it expected?
>>
>> Thanks,
>> Jerry
>>
>> On Sat, Jan 13, 2018 at 1:07 PM, Liam McCafferty <
>> Liam.McCafferty at smartodds.co.uk> wrote:
>>
>>> No Velum runs on standard docker on the caas admin node ( along with a
>>> salt master and a mariadb and various other containers ) where as
>>> kubernetes only runs on the master and worker nodes.
>>>
>>>
>>> Hope this helps
>>>
>>>
>>> Regards,
>>>
>>>
>>> Liam
>>> ------------------------------
>>> *From:* Jerry Hwang <special011 at gmail.com>
>>> *Sent:* 13 January 2018 21:02
>>> *To:* Liam McCafferty
>>> *Cc:* caasp-beta at lists.suse.com
>>> *Subject:* Re: [caasp-beta] velum troubleshoot
>>>
>>> Ah right, all run in container on kubernetes.
>>>
>>> Thanks,
>>> Jerry
>>>
>>> On Sat, Jan 13, 2018 at 12:50 PM, Liam McCafferty <
>>> Liam.McCafferty at smartodds.co.uk> wrote:
>>>
>>>> Hi Jerry,
>>>>
>>>>
>>>> I always put this down to the fact the node has just rebooted, However
>>>> if you wish to view the velum logs it runs inside a container on the admin
>>>> node  you can find this container via using a docker ps command then use a
>>>> docker logs command to get the log output for the container.
>>>>
>>>>
>>>>
>>>> Regards,
>>>>
>>>>
>>>> Liam
>>>> ------------------------------
>>>> *From:* caasp-beta-bounces at lists.suse.com <
>>>> caasp-beta-bounces at lists.suse.com> on behalf of Jerry Hwang <
>>>> special011 at gmail.com>
>>>> *Sent:* 13 January 2018 18:50
>>>> *To:* caasp-beta at lists.suse.com
>>>> *Subject:* [caasp-beta] velum troubleshoot
>>>>
>>>> Hello,
>>>>
>>>> Does anyone have a general guideline where or how to check velum?
>>>>
>>>> I always observe this message in the browser after admin node update
>>>> (v2)
>>>> > We're sorry, but something went wrong.
>>>> > If you are the application owner check the logs for more information.
>>>>
>>>> Eventually I can get back to the admin node (velum), but sometimes it
>>>> takes long, and I thought if I have a list to check or monitor, it'd useful.
>>>>
>>>> And I noticed I can't get to k8s cluster via kubectl (from my local
>>>> machine) during admin node upgrade & reboot, is it expected?
>>>>
>>>> Jerry
>>>>
>>>> note: this is to look for some general idea or share, not a support
>>>> request.
>>>>
>>>> ______________________________________________________________________
>>>> This email has been scanned by the Symantec Email Security.cloud
>>>> service.
>>>> For more information please visit http://www.symanteccloud.com
>>>> ______________________________________________________________________
>>>>
>>>> This e-mail and any files transmitted with it are confidential and may
>>>> be legally privileged. If you receive it in error or are not the intended
>>>> recipient you must not copy, distribute or take any action in reliance upon
>>>> it. Instead, please notify us immediately by telephoning +44 (20) 7482
>>>> 0077 <+44%2020%207482%200077> and delete the material from your
>>>> systems. Smartodds is a business carried on by Smartodds Limited, a company
>>>> registered with the Registrar of Companies for England and Wales with
>>>> number 05108548. Registered office: Unit 540 Highgate Studios, 53-79
>>>> Highgate Road, London NW5 1TL
>>>> <https://maps.google.com/?q=53-79+Highgate+Road,+London+NW5+1TL&entry=gmail&source=g>
>>>>
>>>> ______________________________________________________________________
>>>> This email has been scanned by the Symantec Email Security.cloud
>>>> service.
>>>> For more information please visit http://www.symanteccloud.com
>>>> ______________________________________________________________________
>>>>
>>>> This e-mail and any files transmitted with it are confidential and may
>>>> be legally privileged. If you receive it in error or are not the intended
>>>> recipient you must not copy, distribute or take any action in reliance upon
>>>> it. Instead, please notify us immediately by telephoning +44 (20) 7482
>>>> 0077 <+44%2020%207482%200077> and delete the material from your
>>>> systems. Smartodds is a business carried on by Smartodds Limited, a company
>>>> registered with the Registrar of Companies for England and Wales with
>>>> number 05108548. Registered office: Unit 531 Highgate Studios, 53-79
>>>> Highgate Road, London NW5 1TL
>>>> <https://maps.google.com/?q=53-79+Highgate+Road,+London+NW5+1TL&entry=gmail&source=g>
>>>>
>>>
>>>
>>> ______________________________________________________________________
>>> This email has been scanned by the Symantec Email Security.cloud service.
>>> For more information please visit http://www.symanteccloud.com
>>> ______________________________________________________________________
>>>
>>> This e-mail and any files transmitted with it are confidential and may
>>> be legally privileged. If you receive it in error or are not the intended
>>> recipient you must not copy, distribute or take any action in reliance upon
>>> it. Instead, please notify us immediately by telephoning +44 (20) 7482
>>> 0077 <+44%2020%207482%200077> and delete the material from your
>>> systems. Smartodds is a business carried on by Smartodds Limited, a company
>>> registered with the Registrar of Companies for England and Wales with
>>> number 05108548. Registered office: Unit 540 Highgate Studios, 53-79
>>> Highgate Road, London NW5 1TL
>>> <https://maps.google.com/?q=53-79+Highgate+Road,+London+NW5+1TL&entry=gmail&source=g>
>>>
>>> ______________________________________________________________________
>>> This email has been scanned by the Symantec Email Security.cloud service.
>>> For more information please visit http://www.symanteccloud.com
>>> ______________________________________________________________________
>>>
>>> This e-mail and any files transmitted with it are confidential and may
>>> be legally privileged. If you receive it in error or are not the intended
>>> recipient you must not copy, distribute or take any action in reliance upon
>>> it. Instead, please notify us immediately by telephoning +44 (20) 7482
>>> 0077 <+44%2020%207482%200077> and delete the material from your
>>> systems. Smartodds is a business carried on by Smartodds Limited, a company
>>> registered with the Registrar of Companies for England and Wales with
>>> number 05108548. Registered office: Unit 531 Highgate Studios, 53-79
>>> Highgate Road, London NW5 1TL
>>> <https://maps.google.com/?q=53-79+Highgate+Road,+London+NW5+1TL&entry=gmail&source=g>
>>>
>>
>>
>> ______________________________________________________________________
>> This email has been scanned by the Symantec Email Security.cloud service.
>> For more information please visit http://www.symanteccloud.com
>> ______________________________________________________________________
>>
>> This e-mail and any files transmitted with it are confidential and may be
>> legally privileged. If you receive it in error or are not the intended
>> recipient you must not copy, distribute or take any action in reliance upon
>> it. Instead, please notify us immediately by telephoning +44 (20) 7482
>> 0077 <+44%2020%207482%200077> and delete the material from your systems.
>> Smartodds is a business carried on by Smartodds Limited, a company
>> registered with the Registrar of Companies for England and Wales with
>> number 05108548. Registered office: Unit 540 Highgate Studios, 53-79
>> Highgate Road, London NW5 1TL
>> <https://maps.google.com/?q=53-79+Highgate+Road,+London+NW5+1TL&entry=gmail&source=g>
>>
>> ______________________________________________________________________
>> This email has been scanned by the Symantec Email Security.cloud service.
>> For more information please visit http://www.symanteccloud.com
>> ______________________________________________________________________
>>
>> This e-mail and any files transmitted with it are confidential and may be
>> legally privileged. If you receive it in error or are not the intended
>> recipient you must not copy, distribute or take any action in reliance upon
>> it. Instead, please notify us immediately by telephoning +44 (20) 7482
>> 0077 <+44%2020%207482%200077> and delete the material from your systems.
>> Smartodds is a business carried on by Smartodds Limited, a company
>> registered with the Registrar of Companies for England and Wales with
>> number 05108548. Registered office: Unit 531 Highgate Studios, 53-79
>> Highgate Road, London NW5 1TL
>> <https://maps.google.com/?q=53-79+Highgate+Road,+London+NW5+1TL&entry=gmail&source=g>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.suse.com/pipermail/caasp-beta/attachments/20180113/77f9a598/attachment.htm>