[caasp-beta] velum troubleshoot

Wed Mar 7 22:33:46 MST 2018

Hello,

The velum dashboard is still unstable. Does it work well for everyone?
To SUSE team, do you have any kb or tips for troubleshooting velum?

I see this error in dashboard container:
Processing by DashboardController#index as HTML
Completed 500 Internal Server Error in 60111ms (ActiveRecord: 7.1ms)
Velum::SaltApi::SaltConnectionException (Net::ReadTimeout):

In the salt minion container, this repeats with no more clue.
[ERROR   ] Error while bringing up minion for multi-master. Is master at
localhost responding?

However, I don't see any error in salt master or api container.

That's pretty much I see in syslog as well.
It worked well in the beginning, but not so well after a while..
Where I should check further?

Jerry

On Sat, Jan 13, 2018 at 8:00 PM, Jerry Hwang <special011 at gmail.com> wrote:

> I confirmed the admin node reboot (even shutdown) doesn't cause to fail
> kubectl (I even confirmed while the admin node is down, kubectl still
> works, and I also checked kubectl conf points to k8s master node), so it
> must be something related to when the admin node is updated (& subsequently
> rebooted).
> I will check more detail next time the admin node is updated and confirm
> the situation. If you can try, you may also try kubectl soon after you
> initiated the admin node update (& subsequent auto reboot) in the dashboard
> during the process..
>
> Regarding the dashboard loading (taking too long), I see something
> potentially not clean with salt master connection, and I will check
> further..
> Velum::SaltApi::SaltConnectionException (Net::ReadTimeout):
>
> Thanks,
> Jerry
>
> On Sat, Jan 13, 2018 at 1:30 PM, Jerry Hwang <special011 at gmail.com> wrote:
>
>> Okay, what you described is also what I expected.
>> Thanks for the idea.
>>
>> Regards,
>> Jerry
>>
>> On Sat, Jan 13, 2018 at 1:23 PM, Liam McCafferty <
>> Liam.McCafferty at smartodds.co.uk> wrote:
>>
>>> I have never noticed this before so  I just rebooted my admin node and I
>>> can confirm I can still communicate via kubectl to the master node.
>>>
>>>
>>> I have also checked my kube config in ~/.kube/config and I can
>>> see that the cluster server is set to the master server. does
>>> your kubeconfig point to your master node?  Upon checking my admin node I
>>> see it also that it has the haproxy k8 containers which I believe deal
>>> with routing to the k8 api. so if your config is pointed to your admin node
>>> that would make sense that you cant reach the api server.
>>>
>>>
>>>
>>> Cheers,
>>>
>>>
>>> Liam
>>>
>>>
>>>
>>> ------------------------------
>>> *From:* Jerry Hwang <special011 at gmail.com>
>>> *Sent:* 13 January 2018 21:11
>>> *To:* Liam McCafferty
>>> *Cc:* caasp-beta at lists.suse.com
>>> *Subject:* Re: [caasp-beta] velum troubleshoot
>>>
>>> Right, that is also a good point. (I too much focused on the general
>>> idea  :))
>>>
>>> And it seems I can't get to k8s cluster via kubectl (from my local
>>> machine) during admin node upgrade & reboot, is it expected?
>>>
>>> Thanks,
>>> Jerry
>>>
>>> On Sat, Jan 13, 2018 at 1:07 PM, Liam McCafferty <
>>> Liam.McCafferty at smartodds.co.uk> wrote:
>>>
>>>> No Velum runs on standard docker on the caas admin node ( along with a
>>>> salt master and a mariadb and various other containers ) where as
>>>> kubernetes only runs on the master and worker nodes.
>>>>
>>>>
>>>> Hope this helps
>>>>
>>>>
>>>> Regards,
>>>>
>>>>
>>>> Liam
>>>> ------------------------------
>>>> *From:* Jerry Hwang <special011 at gmail.com>
>>>> *Sent:* 13 January 2018 21:02
>>>> *To:* Liam McCafferty
>>>> *Cc:* caasp-beta at lists.suse.com
>>>> *Subject:* Re: [caasp-beta] velum troubleshoot
>>>>
>>>> Ah right, all run in container on kubernetes.
>>>>
>>>> Thanks,
>>>> Jerry
>>>>
>>>> On Sat, Jan 13, 2018 at 12:50 PM, Liam McCafferty <
>>>> Liam.McCafferty at smartodds.co.uk> wrote:
>>>>
>>>>> Hi Jerry,
>>>>>
>>>>>
>>>>> I always put this down to the fact the node has just rebooted, However
>>>>> if you wish to view the velum logs it runs inside a container on the admin
>>>>> node  you can find this container via using a docker ps command then use a
>>>>> docker logs command to get the log output for the container.
>>>>>
>>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>>
>>>>> Liam
>>>>> ------------------------------
>>>>> *From:* caasp-beta-bounces at lists.suse.com <
>>>>> caasp-beta-bounces at lists.suse.com> on behalf of Jerry Hwang <
>>>>> special011 at gmail.com>
>>>>> *Sent:* 13 January 2018 18:50
>>>>> *To:* caasp-beta at lists.suse.com
>>>>> *Subject:* [caasp-beta] velum troubleshoot
>>>>>
>>>>> Hello,
>>>>>
>>>>> Does anyone have a general guideline where or how to check velum?
>>>>>
>>>>> I always observe this message in the browser after admin node update
>>>>> (v2)
>>>>> > We're sorry, but something went wrong.
>>>>> > If you are the application owner check the logs for more information.
>>>>>
>>>>> Eventually I can get back to the admin node (velum), but sometimes it
>>>>> takes long, and I thought if I have a list to check or monitor, it'd useful.
>>>>>
>>>>> And I noticed I can't get to k8s cluster via kubectl (from my local
>>>>> machine) during admin node upgrade & reboot, is it expected?
>>>>>
>>>>> Jerry
>>>>>
>>>>> note: this is to look for some general idea or share, not a support
>>>>> request.
>>>>>
>>>>> ______________________________________________________________________
>>>>> This email has been scanned by the Symantec Email Security.cloud
>>>>> service.
>>>>> For more information please visit http://www.symanteccloud.com
>>>>> ______________________________________________________________________
>>>>>
>>>>> This e-mail and any files transmitted with it are confidential and may
>>>>> be legally privileged. If you receive it in error or are not the intended
>>>>> recipient you must not copy, distribute or take any action in reliance upon
>>>>> it. Instead, please notify us immediately by telephoning +44 (20)
>>>>> 7482 0077 <+44%2020%207482%200077> and delete the material from your
>>>>> systems. Smartodds is a business carried on by Smartodds Limited, a company
>>>>> registered with the Registrar of Companies for England and Wales with
>>>>> number 05108548. Registered office: Unit 540 Highgate Studios, 53-79
>>>>> Highgate Road, London NW5 1TL
>>>>> <https://maps.google.com/?q=53-79+Highgate+Road,+London+NW5+1TL&entry=gmail&source=g>
>>>>>
>>>>> ______________________________________________________________________
>>>>> This email has been scanned by the Symantec Email Security.cloud
>>>>> service.
>>>>> For more information please visit http://www.symanteccloud.com
>>>>> ______________________________________________________________________
>>>>>
>>>>> This e-mail and any files transmitted with it are confidential and may
>>>>> be legally privileged. If you receive it in error or are not the intended
>>>>> recipient you must not copy, distribute or take any action in reliance upon
>>>>> it. Instead, please notify us immediately by telephoning +44 (20)
>>>>> 7482 0077 <+44%2020%207482%200077> and delete the material from your
>>>>> systems. Smartodds is a business carried on by Smartodds Limited, a company
>>>>> registered with the Registrar of Companies for England and Wales with
>>>>> number 05108548. Registered office: Unit 531 Highgate Studios, 53-79
>>>>> Highgate Road, London NW5 1TL
>>>>> <https://maps.google.com/?q=53-79+Highgate+Road,+London+NW5+1TL&entry=gmail&source=g>
>>>>>
>>>>
>>>>
>>>> ______________________________________________________________________
>>>> This email has been scanned by the Symantec Email Security.cloud
>>>> service.
>>>> For more information please visit http://www.symanteccloud.com
>>>> ______________________________________________________________________
>>>>
>>>> This e-mail and any files transmitted with it are confidential and may
>>>> be legally privileged. If you receive it in error or are not the intended
>>>> recipient you must not copy, distribute or take any action in reliance upon
>>>> it. Instead, please notify us immediately by telephoning +44 (20) 7482
>>>> 0077 <+44%2020%207482%200077> and delete the material from your
>>>> systems. Smartodds is a business carried on by Smartodds Limited, a company
>>>> registered with the Registrar of Companies for England and Wales with
>>>> number 05108548. Registered office: Unit 540 Highgate Studios, 53-79
>>>> Highgate Road, London NW5 1TL
>>>> <https://maps.google.com/?q=53-79+Highgate+Road,+London+NW5+1TL&entry=gmail&source=g>
>>>>
>>>> ______________________________________________________________________
>>>> This email has been scanned by the Symantec Email Security.cloud
>>>> service.
>>>> For more information please visit http://www.symanteccloud.com
>>>> ______________________________________________________________________
>>>>
>>>> This e-mail and any files transmitted with it are confidential and may
>>>> be legally privileged. If you receive it in error or are not the intended
>>>> recipient you must not copy, distribute or take any action in reliance upon
>>>> it. Instead, please notify us immediately by telephoning +44 (20) 7482
>>>> 0077 <+44%2020%207482%200077> and delete the material from your
>>>> systems. Smartodds is a business carried on by Smartodds Limited, a company
>>>> registered with the Registrar of Companies for England and Wales with
>>>> number 05108548. Registered office: Unit 531 Highgate Studios, 53-79
>>>> Highgate Road, London NW5 1TL
>>>> <https://maps.google.com/?q=53-79+Highgate+Road,+London+NW5+1TL&entry=gmail&source=g>
>>>>
>>>
>>>
>>> ______________________________________________________________________
>>> This email has been scanned by the Symantec Email Security.cloud service.
>>> For more information please visit http://www.symanteccloud.com
>>> ______________________________________________________________________
>>>
>>> This e-mail and any files transmitted with it are confidential and may
>>> be legally privileged. If you receive it in error or are not the intended
>>> recipient you must not copy, distribute or take any action in reliance upon
>>> it. Instead, please notify us immediately by telephoning +44 (20) 7482
>>> 0077 <+44%2020%207482%200077> and delete the material from your
>>> systems. Smartodds is a business carried on by Smartodds Limited, a company
>>> registered with the Registrar of Companies for England and Wales with
>>> number 05108548. Registered office: Unit 540 Highgate Studios, 53-79
>>> Highgate Road, London NW5 1TL
>>> <https://maps.google.com/?q=53-79+Highgate+Road,+London+NW5+1TL&entry=gmail&source=g>
>>>
>>> ______________________________________________________________________
>>> This email has been scanned by the Symantec Email Security.cloud service.
>>> For more information please visit http://www.symanteccloud.com
>>> ______________________________________________________________________
>>>
>>> This e-mail and any files transmitted with it are confidential and may
>>> be legally privileged. If you receive it in error or are not the intended
>>> recipient you must not copy, distribute or take any action in reliance upon
>>> it. Instead, please notify us immediately by telephoning +44 (20) 7482
>>> 0077 <+44%2020%207482%200077> and delete the material from your
>>> systems. Smartodds is a business carried on by Smartodds Limited, a company
>>> registered with the Registrar of Companies for England and Wales with
>>> number 05108548. Registered office: Unit 531 Highgate Studios, 53-79
>>> Highgate Road, London NW5 1TL
>>> <https://maps.google.com/?q=53-79+Highgate+Road,+London+NW5+1TL&entry=gmail&source=g>
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.suse.com/pipermail/caasp-beta/attachments/20180307/d52e7746/attachment.htm>