From fcastelli at suse.com Fri Feb 2 00:41:49 2018 From: fcastelli at suse.com (Flavio Castelli) Date: Fri, 2 Feb 2018 08:41:49 +0100 Subject: [caasp-beta] kubernetes components in systemd unit for container In-Reply-To: References: Message-ID: On 01/31/2018 09:21 PM, Jerry Hwang wrote: > Just for a general question. > I see all kubernetes system components (etcd, apiserver, controller, > scheduler, kubelet, kube-proxy) run in systemd unit rather than in > container in SUSE CaaS v2. > > Just curious about SUSE's preference/direction and does anyone have an > idea/opinion of pros & cons in between the two approaches? Back in the v1 days we evaluated the possibility to run kubernetes components inside of containers but this wasn't mature enough. We still intend to go this way, it's not currently on the top list of priorities for v3. Cheers Flavio From special011 at gmail.com Fri Feb 2 11:28:11 2018 From: special011 at gmail.com (Jerry Hwang) Date: Fri, 2 Feb 2018 10:28:11 -0800 Subject: [caasp-beta] kubernetes components in systemd unit for container In-Reply-To: References: Message-ID: Thanks Flavio for your answer. Regards, Jerry On Thu, Feb 1, 2018 at 11:41 PM, Flavio Castelli wrote: > On 01/31/2018 09:21 PM, Jerry Hwang wrote: > > Just for a general question. > > I see all kubernetes system components (etcd, apiserver, controller, > > scheduler, kubelet, kube-proxy) run in systemd unit rather than in > > container in SUSE CaaS v2. > > > > Just curious about SUSE's preference/direction and does anyone have an > > idea/opinion of pros & cons in between the two approaches? > > Back in the v1 days we evaluated the possibility to run kubernetes > components inside of containers but this wasn't mature enough. > > We still intend to go this way, it's not currently on the top list of > priorities for v3. > > Cheers > Flavio > _______________________________________________ > caasp-beta mailing list > caasp-beta at lists.suse.com > http://lists.suse.com/mailman/listinfo/caasp-beta > -------------- next part -------------- An HTML attachment was scrubbed... URL: From simeon.bobylev at cortex-it.ch Wed Feb 21 06:23:38 2018 From: simeon.bobylev at cortex-it.ch (Simeon Bobylev) Date: Wed, 21 Feb 2018 14:23:38 +0100 (CET) Subject: [caasp-beta] read only fs on caasp kubernetes nodes Message-ID: <00c301d3ab17$32a889e0$97f99da0$@cortex-it.ch> Hi all, Is there a specific reason that /dev/sda2 is mounted in read only into / on CaaSP Kubernetes nodes ? c002-caasnode-01:~ # mkdir /var/lib/controller mkdir: cannot create directory ???/var/lib/controller???: Read-only file system c002-caasnode-01:~ # head -n 1 /etc/fstab UUID=abf1dcf2-8539-49d4-8f8e-45c0cf561e80 / btrfs ro 0 0 I would like to test Avi Networks integration with SUSE CaaSP and in order to perform that I need to run a container with some volumes mounts, into /var/lib/controller for example. Of course I have a new dedicated virtual disk for these docker volumes but the problem is even if can add a new disk and create partition I?m not able to mount it anywhere because of read only mount on /. I tried to mount /dev/sda2 in rw through ?Rescue System? to create needed directory /var/lib/controller but I had the same message (Read-only file system). Actually to continue my tests I did something dirty (from my point of view) : c002-caasnode-01:~ # tail -n 1 /etc/fstab UUID=17153c00-ba92-4bed-9bf6-0a36b43bb176 /var/lib/docker/avi-controller ext4 defaults 0 0 What do you think about that ? Is there some best practices for similar cases ? Best regards, Sim?on Bobylev ? System Engineer ? Cortex IT SA ? Route de l'Ile aux Bois 1A ? 1870 Monthey ? Switzerland Direct: +41 24 524 1204 ? Main: +41 24 524 1200 ? Email: simeon.bobylev at cortex-it.ch http://www.cortex-it.ch/ ? A WIRD Group Company This e-mail has been sent by a trading and services firm. It is confidential and may be privileged. Only the intended recipient may read, copy and use it. If you have received it in error, please contact us immediately. Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 4217 bytes Desc: not available URL: From kukuk at suse.com Wed Feb 21 06:27:37 2018 From: kukuk at suse.com (Thorsten Kukuk) Date: Wed, 21 Feb 2018 14:27:37 +0100 Subject: [caasp-beta] read only fs on caasp kubernetes nodes In-Reply-To: <00c301d3ab17$32a889e0$97f99da0$@cortex-it.ch> References: <00c301d3ab17$32a889e0$97f99da0$@cortex-it.ch> Message-ID: <20180221132737.GA19916@suse.com> On Wed, Feb 21, Simeon Bobylev wrote: > Hi all, > > > > Is there a specific reason that /dev/sda2 is mounted in read only into / on > CaaSP Kubernetes nodes ? Yes, that's part of the concept and needed for transactional updates. > c002-caasnode-01:~ # mkdir /var/lib/controller > > mkdir: cannot create directory ???/var/lib/controller???: Read-only file system # transactional-update shell #> mksubvolume /var/lib/controller #> exit # systemctl reboot Thorsten -- Thorsten Kukuk, Distinguished Engineer, Senior Architect SLES & CaaSP SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany GF: Felix Imendoerffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nuernberg) From simeon.bobylev at cortex-it.ch Thu Feb 22 07:37:01 2018 From: simeon.bobylev at cortex-it.ch (Simeon Bobylev) Date: Thu, 22 Feb 2018 15:37:01 +0100 (CET) Subject: [caasp-beta] read only fs on caasp kubernetes nodes In-Reply-To: <20180221132737.GA19916@suse.com> References: <00c301d3ab17$32a889e0$97f99da0$@cortex-it.ch> <20180221132737.GA19916@suse.com> Message-ID: <002701d3abea$99de5c60$cd9b1520$@cortex-it.ch> Thank you Thorsten. I have another weird issue when trying de deploy CaaSP Admin node : The machine has booted after the initial installation but Velum is still not inaccessible 20min later. I rebooted the node but same problem. By checking log files, it seems that some services are waiting for mariadb : ==> /var/log/containers/velum-public-127.0.0.1_default_velum-dashboard-7c256d89831570117d416a2761ace54ebc682afd44c1c66270f4acd2158aa616.log <== {"log":"Waiting for mariadb to be ready in 5 seconds\n","stream":"stdout","time":"2018-02-22T14:28:15.93480394Z"} ==> /var/log/containers/velum-public-127.0.0.1_default_salt-master-8f4b5411619cea0b94257ec7b299f16ca00f681a609d6217060b1924f9d8a972.log <== {"log":"[ERROR ] Could not store events - returner 'mysql.event_return' raised exception: MySQL returner could not connect to database: (1045, \"Access denied for user 'salt'@'localhost' (using password: YES)\")\n","stream":"stderr","time":"2018-02-22T14:29:51.824838923Z"} And if I look into mariadb container logs, it seems mariadb is starting... but never starts. c002-caasadmin-01:~ # docker logs -f k8s_velum-mariadb_velum-private-127.0.0.1_default_bf640ab62f9d8d01fa0c2f7e66744787_1 ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2 "No such file or directory") ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2 "No such file or directory") MySQL init process in progress... 180222 14:25:42 [Note] /usr/sbin/mysqld (mysqld 10.0.31-MariaDB) starting as process 40 ... ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO) Setting root password... Root password already set, nothing to do here Removing anonymous users... ... Success! Removing remote root access... ... Success! Removing test database... - Dropping test database... ... Success! - Removing privileges on test database... ... Success! Reloading privileges... ... Success! /usr/local/bin/entrypoint.sh: ignoring /docker-entrypoint-initdb.d/* MySQL init process done. Ready for start up. Cleaning up... 180222 14:25:45 [Note] /usr/sbin/mysqld (mysqld 10.0.31-MariaDB) starting as process 1 ... Do you know any workaround to fix that ? Sim?on Bobylev ? System Engineer ? Cortex IT SA ? Route de l'Ile aux Bois 1A ? 1870 Monthey ? Switzerland Direct: +41 24 524 1204 ? Main: +41 24 524 1200 ? Email: simeon.bobylev at cortex-it.ch http://www.cortex-it.ch/ ? A WIRD Group Company -----Original Message----- From: caasp-beta-bounces at lists.suse.com [mailto:caasp-beta-bounces at lists.suse.com] On Behalf Of Thorsten Kukuk Sent: mercredi, 21 f?vrier 2018 14:28 To: caasp-beta at lists.suse.com Subject: Re: [caasp-beta] read only fs on caasp kubernetes nodes On Wed, Feb 21, Simeon Bobylev wrote: > Hi all, > > > > Is there a specific reason that /dev/sda2 is mounted in read only into > / on CaaSP Kubernetes nodes ? Yes, that's part of the concept and needed for transactional updates. > c002-caasnode-01:~ # mkdir /var/lib/controller > > mkdir: cannot create directory ???/var/lib/controller???: Read-only > file system # transactional-update shell #> mksubvolume /var/lib/controller #> exit # systemctl reboot Thorsten -- Thorsten Kukuk, Distinguished Engineer, Senior Architect SLES & CaaSP SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany GF: Felix Imendoerffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nuernberg) _______________________________________________ caasp-beta mailing list caasp-beta at lists.suse.com http://lists.suse.com/mailman/listinfo/caasp-beta From stephane.lebihan at amundi.com Thu Feb 22 10:55:17 2018 From: stephane.lebihan at amundi.com (=?utf-8?B?TGUgQmloYW4gU3TDqXBoYW5lIChBTVVOREktSVRTKQ==?=) Date: Thu, 22 Feb 2018 17:55:17 +0000 Subject: [caasp-beta] read only fs on caasp kubernetes nodes In-Reply-To: <002701d3abea$99de5c60$cd9b1520$@cortex-it.ch> References: <00c301d3ab17$32a889e0$97f99da0$@cortex-it.ch> <20180221132737.GA19916@suse.com> <002701d3abea$99de5c60$cd9b1520$@cortex-it.ch> Message-ID: Hello, Strange. I never have problem with Velum deployement. After yes .... I the same log for MariaDB containers, but Velum works. Creating MySQL privilege database... Installing MariaDB/MySQL system tables in '/var/lib/mysql' ... 180222 16:33:50 [Note] /usr/sbin/mysqld (mysqld 10.0.31-MariaDB) starting as process 51 ... OK Filling help tables... 180222 16:33:53 [Note] /usr/sbin/mysqld (mysqld 10.0.31-MariaDB) starting as process 78 ... OK PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER ! To do so, start the server, then issue the following commands: '/usr/bin/mysqladmin' -u root password 'new-password' '/usr/bin/mysqladmin' -u root -h velum-private-127.0.0.1 password 'new-password' Alternatively you can run: '/usr/bin/mysql_secure_installation' which will also give you the option of removing the test databases and anonymous user created by default. This is strongly recommended for production servers. See the MariaDB Knowledgebase at http://mariadb.com/kb or the MySQL manual for more instructions. You can start the MariaDB daemon with: rcmysql start You can test the MariaDB daemon with mariadb-test package Please report any problems at http://mariadb.org/jira The latest information about MariaDB is available at http://mariadb.org/. You can find additional information about the MySQL part at: http://dev.mysql.com Consider joining MariaDB's strong and vibrant community: https://mariadb.org/get-involved/ ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2 "No such file or directory") ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2 "No such file or directory") MySQL init process in progress... 180222 16:33:56 [Note] /usr/sbin/mysqld (mysqld 10.0.31-MariaDB) starting as process 122 ... Setting root password... Password updated successfully! Reloading privilege tables.. ... Success! Removing anonymous users... ... Success! Removing remote root access... ... Success! Removing test database... - Dropping test database... ... Success! - Removing privileges on test database... ... Success! Reloading privileges... ... Success! /usr/local/bin/entrypoint.sh: ignoring /docker-entrypoint-initdb.d/* MySQL init process done. Ready for start up. Cleaning up... 180222 16:33:58 [Note] /usr/sbin/mysqld (mysqld 10.0.31-MariaDB) starting as process 1 ... Regards, St?phane Le Bihan SDE/DSI/IPR/SSD/UNX 90, Boulevard Pasteur - 75015 Paris Web: www.amundi.com Tel: +33 1 76 32 32 08 Tel Unix Team: +33 1 76 32 02 30 @: stephane.lebihan at amundi.com @: sits.unix at amundi.com Visit us on: -----Message d'origine----- De?: caasp-beta-bounces at lists.suse.com [mailto:caasp-beta-bounces at lists.suse.com] De la part de Simeon Bobylev Envoy??: jeudi 22 f?vrier 2018 15:37 ??: caasp-beta at lists.suse.com Objet?: Re: [caasp-beta] read only fs on caasp kubernetes nodes Thank you Thorsten. I have another weird issue when trying de deploy CaaSP Admin node : The machine has booted after the initial installation but Velum is still not inaccessible 20min later. I rebooted the node but same problem. By checking log files, it seems that some services are waiting for mariadb : ==> /var/log/containers/velum-public-127.0.0.1_default_velum-dashboard-7c256d89831570117d416a2761ace54ebc682afd44c1c66270f4acd2158aa616.log <== {"log":"Waiting for mariadb to be ready in 5 seconds\n","stream":"stdout","time":"2018-02-22T14:28:15.93480394Z"} ==> /var/log/containers/velum-public-127.0.0.1_default_salt-master-8f4b5411619cea0b94257ec7b299f16ca00f681a609d6217060b1924f9d8a972.log <== {"log":"[ERROR ] Could not store events - returner 'mysql.event_return' raised exception: MySQL returner could not connect to database: (1045, \"Access denied for user 'salt'@'localhost' (using password: YES)\")\n","stream":"stderr","time":"2018-02-22T14:29:51.824838923Z"} And if I look into mariadb container logs, it seems mariadb is starting... but never starts. c002-caasadmin-01:~ # docker logs -f k8s_velum-mariadb_velum-private-127.0.0.1_default_bf640ab62f9d8d01fa0c2f7e66744787_1 ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2 "No such file or directory") ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2 "No such file or directory") MySQL init process in progress... 180222 14:25:42 [Note] /usr/sbin/mysqld (mysqld 10.0.31-MariaDB) starting as process 40 ... ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO) Setting root password... Root password already set, nothing to do here Removing anonymous users... ... Success! Removing remote root access... ... Success! Removing test database... - Dropping test database... ... Success! - Removing privileges on test database... ... Success! Reloading privileges... ... Success! /usr/local/bin/entrypoint.sh: ignoring /docker-entrypoint-initdb.d/* MySQL init process done. Ready for start up. Cleaning up... 180222 14:25:45 [Note] /usr/sbin/mysqld (mysqld 10.0.31-MariaDB) starting as process 1 ... Do you know any workaround to fix that ? Sim?on Bobylev ? System Engineer ? Cortex IT SA ? Route de l'Ile aux Bois 1A ? 1870 Monthey ? Switzerland Direct: +41 24 524 1204 ? Main: +41 24 524 1200 ? Email: simeon.bobylev at cortex-it.ch http://www.cortex-it.ch/ ? A WIRD Group Company -----Original Message----- From: caasp-beta-bounces at lists.suse.com [mailto:caasp-beta-bounces at lists.suse.com] On Behalf Of Thorsten Kukuk Sent: mercredi, 21 f?vrier 2018 14:28 To: caasp-beta at lists.suse.com Subject: Re: [caasp-beta] read only fs on caasp kubernetes nodes On Wed, Feb 21, Simeon Bobylev wrote: > Hi all, > > > > Is there a specific reason that /dev/sda2 is mounted in read only into > / on CaaSP Kubernetes nodes ? Yes, that's part of the concept and needed for transactional updates. > c002-caasnode-01:~ # mkdir /var/lib/controller > > mkdir: cannot create directory ???/var/lib/controller???: Read-only > file system # transactional-update shell #> mksubvolume /var/lib/controller #> exit # systemctl reboot Thorsten -- Thorsten Kukuk, Distinguished Engineer, Senior Architect SLES & CaaSP SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany GF: Felix Imendoerffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nuernberg) _______________________________________________ caasp-beta mailing list caasp-beta at lists.suse.com http://lists.suse.com/mailman/listinfo/caasp-beta _______________________________________________ caasp-beta mailing list caasp-beta at lists.suse.com http://lists.suse.com/mailman/listinfo/caasp-beta From fcastelli at suse.com Fri Feb 23 00:57:11 2018 From: fcastelli at suse.com (Flavio Castelli) Date: Fri, 23 Feb 2018 08:57:11 +0100 Subject: [caasp-beta] read only fs on caasp kubernetes nodes In-Reply-To: References: <00c301d3ab17$32a889e0$97f99da0$@cortex-it.ch> <20180221132737.GA19916@suse.com> <002701d3abea$99de5c60$cd9b1520$@cortex-it.ch> Message-ID: <03778fbc-d2a6-36e5-6891-3d1aa80ed517@suse.com> Which release of caasp are you currently using? Thanks Flavio On 02/22/2018 06:55 PM, Le Bihan St?phane (AMUNDI-ITS) wrote: > Hello, > > Strange. I never have problem with Velum deployement. After yes .... > > I the same log for MariaDB containers, but Velum works. > > Creating MySQL privilege database... > Installing MariaDB/MySQL system tables in '/var/lib/mysql' ... > 180222 16:33:50 [Note] /usr/sbin/mysqld (mysqld 10.0.31-MariaDB) starting as process 51 ... > OK > Filling help tables... > 180222 16:33:53 [Note] /usr/sbin/mysqld (mysqld 10.0.31-MariaDB) starting as process 78 ... > OK > > PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER ! > To do so, start the server, then issue the following commands: > > '/usr/bin/mysqladmin' -u root password 'new-password' > '/usr/bin/mysqladmin' -u root -h velum-private-127.0.0.1 password 'new-password' > > Alternatively you can run: > '/usr/bin/mysql_secure_installation' > > which will also give you the option of removing the test > databases and anonymous user created by default. This is > strongly recommended for production servers. > > See the MariaDB Knowledgebase at http://mariadb.com/kb or the > MySQL manual for more instructions. > > You can start the MariaDB daemon with: > rcmysql start > > You can test the MariaDB daemon with mariadb-test package > > Please report any problems at http://mariadb.org/jira > > The latest information about MariaDB is available at http://mariadb.org/. > You can find additional information about the MySQL part at: > http://dev.mysql.com > Consider joining MariaDB's strong and vibrant community: > https://mariadb.org/get-involved/ > > ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2 "No such file or directory") > ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2 "No such file or directory") > MySQL init process in progress... > 180222 16:33:56 [Note] /usr/sbin/mysqld (mysqld 10.0.31-MariaDB) starting as process 122 ... > Setting root password... > Password updated successfully! > Reloading privilege tables.. > ... Success! > Removing anonymous users... > ... Success! > Removing remote root access... > ... Success! > Removing test database... > - Dropping test database... > ... Success! > - Removing privileges on test database... > ... Success! > Reloading privileges... > ... Success! > /usr/local/bin/entrypoint.sh: ignoring /docker-entrypoint-initdb.d/* > > > MySQL init process done. Ready for start up. > > Cleaning up... > 180222 16:33:58 [Note] /usr/sbin/mysqld (mysqld 10.0.31-MariaDB) starting as process 1 ... > > Regards, > > St?phane Le Bihan > SDE/DSI/IPR/SSD/UNX > 90, Boulevard Pasteur - 75015 Paris > Web: www.amundi.com > Tel: +33 1 76 32 32 08 > Tel Unix Team: +33 1 76 32 02 30 > @: stephane.lebihan at amundi.com > @: sits.unix at amundi.com > > Visit us on: > > > > > -----Message d'origine----- > De?: caasp-beta-bounces at lists.suse.com [mailto:caasp-beta-bounces at lists.suse.com] De la part de Simeon Bobylev > Envoy??: jeudi 22 f?vrier 2018 15:37 > ??: caasp-beta at lists.suse.com > Objet?: Re: [caasp-beta] read only fs on caasp kubernetes nodes > > Thank you Thorsten. > > I have another weird issue when trying de deploy CaaSP Admin node : > > The machine has booted after the initial installation but Velum is still not inaccessible 20min later. I rebooted the node but same problem. > By checking log files, it seems that some services are waiting for mariadb : > > ==> > /var/log/containers/velum-public-127.0.0.1_default_velum-dashboard-7c256d89831570117d416a2761ace54ebc682afd44c1c66270f4acd2158aa616.log > <== > {"log":"Waiting for mariadb to be ready in 5 seconds\n","stream":"stdout","time":"2018-02-22T14:28:15.93480394Z"} > > ==> > /var/log/containers/velum-public-127.0.0.1_default_salt-master-8f4b5411619cea0b94257ec7b299f16ca00f681a609d6217060b1924f9d8a972.log > <== > {"log":"[ERROR ] Could not store events - returner 'mysql.event_return' > raised exception: MySQL returner could not connect to database: (1045, \"Access denied for user 'salt'@'localhost' (using password: > YES)\")\n","stream":"stderr","time":"2018-02-22T14:29:51.824838923Z"} > > And if I look into mariadb container logs, it seems mariadb is starting... > but never starts. > > c002-caasadmin-01:~ # docker logs -f > k8s_velum-mariadb_velum-private-127.0.0.1_default_bf640ab62f9d8d01fa0c2f7e66744787_1 > ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2 "No such file or directory") ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2 "No such file or directory") MySQL init process in progress... > 180222 14:25:42 [Note] /usr/sbin/mysqld (mysqld 10.0.31-MariaDB) starting as process 40 ... > ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using > password: NO) > Setting root password... > Root password already set, nothing to do here > > Removing anonymous users... > ... Success! > Removing remote root access... > ... Success! > Removing test database... > - Dropping test database... > ... Success! > - Removing privileges on test database... > ... Success! > Reloading privileges... > ... Success! > /usr/local/bin/entrypoint.sh: ignoring /docker-entrypoint-initdb.d/* > > > MySQL init process done. Ready for start up. > > Cleaning up... > 180222 14:25:45 [Note] /usr/sbin/mysqld (mysqld 10.0.31-MariaDB) starting as process 1 ... > > > Do you know any workaround to fix that ? > > > Sim?on Bobylev ? System Engineer ? Cortex IT SA ? Route de l'Ile aux Bois 1A > ? 1870 Monthey ? Switzerland > Direct: +41 24 524 1204 ? Main: +41 24 524 1200 ? Email: > simeon.bobylev at cortex-it.ch > > http://www.cortex-it.ch/ ? A WIRD Group Company > > -----Original Message----- > From: caasp-beta-bounces at lists.suse.com > [mailto:caasp-beta-bounces at lists.suse.com] On Behalf Of Thorsten Kukuk > Sent: mercredi, 21 f?vrier 2018 14:28 > To: caasp-beta at lists.suse.com > Subject: Re: [caasp-beta] read only fs on caasp kubernetes nodes > > On Wed, Feb 21, Simeon Bobylev wrote: > >> Hi all, >> >> >> >> Is there a specific reason that /dev/sda2 is mounted in read only into >> / on CaaSP Kubernetes nodes ? > > Yes, that's part of the concept and needed for transactional updates. > >> c002-caasnode-01:~ # mkdir /var/lib/controller >> >> mkdir: cannot create directory ???/var/lib/controller???: Read-only >> file system > > # transactional-update shell > #> mksubvolume /var/lib/controller > #> exit > # systemctl reboot > > > Thorsten > > -- > Thorsten Kukuk, Distinguished Engineer, Senior Architect SLES & CaaSP SUSE > LINUX GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany > GF: Felix Imendoerffer, Jane Smithard, Graham Norton, HRB 21284 (AG > Nuernberg) _______________________________________________ > caasp-beta mailing list > caasp-beta at lists.suse.com > http://lists.suse.com/mailman/listinfo/caasp-beta > _______________________________________________ > caasp-beta mailing list > caasp-beta at lists.suse.com > http://lists.suse.com/mailman/listinfo/caasp-beta > _______________________________________________ > caasp-beta mailing list > caasp-beta at lists.suse.com > http://lists.suse.com/mailman/listinfo/caasp-beta > From simeon.bobylev at cortex-it.ch Fri Feb 23 02:41:34 2018 From: simeon.bobylev at cortex-it.ch (Simeon Bobylev) Date: Fri, 23 Feb 2018 10:41:34 +0100 (CET) Subject: [caasp-beta] read only fs on caasp kubernetes nodes In-Reply-To: <03778fbc-d2a6-36e5-6891-3d1aa80ed517@suse.com> References: <00c301d3ab17$32a889e0$97f99da0$@cortex-it.ch> <20180221132737.GA19916@suse.com> <002701d3abea$99de5c60$cd9b1520$@cortex-it.ch> <03778fbc-d2a6-36e5-6891-3d1aa80ed517@suse.com> Message-ID: <000c01d3ac8a$7ee1fc00$7ca5f400$@cortex-it.ch> Yop, @Flavio I'm using v2.0 (SUSE-CaaS-Platform-2.0-DVD-x86_64-GM-DVD1.iso) and as I entered Registration Code I could get updates when installing the node so I guess the version is the last one -> 2.0.20171020. @St?phane We don't have the same log, you have "OK" statements in your logs but I don?t. Your logs > 180222 16:33:50 [Note] /usr/sbin/mysqld (mysqld 10.0.31-MariaDB) starting > as process 51 ... > OK > Filling help tables... > 180222 16:33:53 [Note] /usr/sbin/mysqld (mysqld 10.0.31-MariaDB) starting > as process 78 ... > OK My logs (that's the last line that I have, so as you can see mariadb does not start) > 180222 16:33:58 [Note] /usr/sbin/mysqld (mysqld 10.0.31-MariaDB) starting > as process 1 ... Regards, Sim?on Bobylev ? System Engineer ? Cortex IT SA ? Route de l'Ile aux Bois 1A ? 1870 Monthey ? Switzerland Direct: +41 24 524 1204 ? Main: +41 24 524 1200 ? Email: simeon.bobylev at cortex-it.ch http://www.cortex-it.ch/ ? A WIRD Group Company -----Original Message----- From: caasp-beta-bounces at lists.suse.com [mailto:caasp-beta-bounces at lists.suse.com] On Behalf Of Flavio Castelli Sent: vendredi, 23 f?vrier 2018 08:57 To: caasp-beta at lists.suse.com Subject: Re: [caasp-beta] read only fs on caasp kubernetes nodes Which release of caasp are you currently using? Thanks Flavio On 02/22/2018 06:55 PM, Le Bihan St?phane (AMUNDI-ITS) wrote: > Hello, > > Strange. I never have problem with Velum deployement. After yes .... > > I the same log for MariaDB containers, but Velum works. > > Creating MySQL privilege database... > Installing MariaDB/MySQL system tables in '/var/lib/mysql' ... > 180222 16:33:50 [Note] /usr/sbin/mysqld (mysqld 10.0.31-MariaDB) starting > as process 51 ... > OK > Filling help tables... > 180222 16:33:53 [Note] /usr/sbin/mysqld (mysqld 10.0.31-MariaDB) starting > as process 78 ... > OK > > PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER ! > To do so, start the server, then issue the following commands: > > '/usr/bin/mysqladmin' -u root password 'new-password' > '/usr/bin/mysqladmin' -u root -h velum-private-127.0.0.1 password > 'new-password' > > Alternatively you can run: > '/usr/bin/mysql_secure_installation' > > which will also give you the option of removing the test databases and > anonymous user created by default. This is strongly recommended for > production servers. > > See the MariaDB Knowledgebase at http://mariadb.com/kb or the MySQL > manual for more instructions. > > You can start the MariaDB daemon with: > rcmysql start > > You can test the MariaDB daemon with mariadb-test package > > Please report any problems at http://mariadb.org/jira > > The latest information about MariaDB is available at http://mariadb.org/. > You can find additional information about the MySQL part at: > http://dev.mysql.com > Consider joining MariaDB's strong and vibrant community: > https://mariadb.org/get-involved/ > > ERROR 2002 (HY000): Can't connect to local MySQL server through socket > '/tmp/mysql.sock' (2 "No such file or directory") ERROR 2002 (HY000): > Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2 > "No such file or directory") MySQL init process in progress... > 180222 16:33:56 [Note] /usr/sbin/mysqld (mysqld 10.0.31-MariaDB) starting > as process 122 ... > Setting root password... > Password updated successfully! > Reloading privilege tables.. > ... Success! > Removing anonymous users... > ... Success! > Removing remote root access... > ... Success! > Removing test database... > - Dropping test database... > ... Success! > - Removing privileges on test database... > ... Success! > Reloading privileges... > ... Success! > /usr/local/bin/entrypoint.sh: ignoring /docker-entrypoint-initdb.d/* > > > MySQL init process done. Ready for start up. > > Cleaning up... > 180222 16:33:58 [Note] /usr/sbin/mysqld (mysqld 10.0.31-MariaDB) starting > as process 1 ... > > Regards, > > St?phane Le Bihan > SDE/DSI/IPR/SSD/UNX > 90, Boulevard Pasteur - 75015 Paris > Web: www.amundi.com > Tel: +33 1 76 32 32 08 > Tel Unix Team: +33 1 76 32 02 30 > @: stephane.lebihan at amundi.com > @: sits.unix at amundi.com > > Visit us on: > > > > > -----Message d'origine----- > De : caasp-beta-bounces at lists.suse.com > [mailto:caasp-beta-bounces at lists.suse.com] De la part de Simeon > Bobylev Envoy? : jeudi 22 f?vrier 2018 15:37 ? : > caasp-beta at lists.suse.com Objet : Re: [caasp-beta] read only fs on > caasp kubernetes nodes > > Thank you Thorsten. > > I have another weird issue when trying de deploy CaaSP Admin node : > > The machine has booted after the initial installation but Velum is still > not inaccessible 20min later. I rebooted the node but same problem. > By checking log files, it seems that some services are waiting for mariadb > : > > ==> > /var/log/containers/velum-public-127.0.0.1_default_velum-dashboard-7c2 > 56d89831570117d416a2761ace54ebc682afd44c1c66270f4acd2158aa616.log > <== > {"log":"Waiting for mariadb to be ready in 5 > seconds\n","stream":"stdout","time":"2018-02-22T14:28:15.93480394Z"} > > ==> > /var/log/containers/velum-public-127.0.0.1_default_salt-master-8f4b541 > 1619cea0b94257ec7b299f16ca00f681a609d6217060b1924f9d8a972.log > <== > {"log":"[ERROR ] Could not store events - returner 'mysql.event_return' > raised exception: MySQL returner could not connect to database: (1045, > \"Access denied for user 'salt'@'localhost' (using password: > YES)\")\n","stream":"stderr","time":"2018-02-22T14:29:51.824838923Z"} > > And if I look into mariadb container logs, it seems mariadb is starting... > but never starts. > > c002-caasadmin-01:~ # docker logs -f > k8s_velum-mariadb_velum-private-127.0.0.1_default_bf640ab62f9d8d01fa0c > 2f7e66744787_1 ERROR 2002 (HY000): Can't connect to local MySQL server > through socket '/tmp/mysql.sock' (2 "No such file or directory") ERROR > 2002 (HY000): Can't connect to local MySQL server through socket > '/tmp/mysql.sock' (2 "No such file or directory") MySQL init process in > progress... > 180222 14:25:42 [Note] /usr/sbin/mysqld (mysqld 10.0.31-MariaDB) starting > as process 40 ... > ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using > password: NO) > Setting root password... > Root password already set, nothing to do here > > Removing anonymous users... > ... Success! > Removing remote root access... > ... Success! > Removing test database... > - Dropping test database... > ... Success! > - Removing privileges on test database... > ... Success! > Reloading privileges... > ... Success! > /usr/local/bin/entrypoint.sh: ignoring /docker-entrypoint-initdb.d/* > > > MySQL init process done. Ready for start up. > > Cleaning up... > 180222 14:25:45 [Note] /usr/sbin/mysqld (mysqld 10.0.31-MariaDB) starting > as process 1 ... > > > Do you know any workaround to fix that ? > > > Sim?on Bobylev ? System Engineer ? Cortex IT SA ? Route de l'Ile aux > Bois 1A ? 1870 Monthey ? Switzerland > Direct: +41 24 524 1204 ? Main: +41 24 524 1200 ? Email: > simeon.bobylev at cortex-it.ch > > http://www.cortex-it.ch/ ? A WIRD Group > Company > > -----Original Message----- > From: caasp-beta-bounces at lists.suse.com > [mailto:caasp-beta-bounces at lists.suse.com] On Behalf Of Thorsten Kukuk > Sent: mercredi, 21 f?vrier 2018 14:28 > To: caasp-beta at lists.suse.com > Subject: Re: [caasp-beta] read only fs on caasp kubernetes nodes > > On Wed, Feb 21, Simeon Bobylev wrote: > >> Hi all, >> >> >> >> Is there a specific reason that /dev/sda2 is mounted in read only >> into / on CaaSP Kubernetes nodes ? > > Yes, that's part of the concept and needed for transactional updates. > >> c002-caasnode-01:~ # mkdir /var/lib/controller >> >> mkdir: cannot create directory ???/var/lib/controller???: Read-only >> file system > > # transactional-update shell > #> mksubvolume /var/lib/controller > #> exit > # systemctl reboot > > > Thorsten > > -- > Thorsten Kukuk, Distinguished Engineer, Senior Architect SLES & CaaSP > SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany > GF: Felix Imendoerffer, Jane Smithard, Graham Norton, HRB 21284 (AG > Nuernberg) _______________________________________________ > caasp-beta mailing list > caasp-beta at lists.suse.com > http://lists.suse.com/mailman/listinfo/caasp-beta > _______________________________________________ > caasp-beta mailing list > caasp-beta at lists.suse.com > http://lists.suse.com/mailman/listinfo/caasp-beta > _______________________________________________ > caasp-beta mailing list > caasp-beta at lists.suse.com > http://lists.suse.com/mailman/listinfo/caasp-beta > _______________________________________________ caasp-beta mailing list caasp-beta at lists.suse.com http://lists.suse.com/mailman/listinfo/caasp-beta From simeon.bobylev at cortex-it.ch Mon Feb 26 08:11:40 2018 From: simeon.bobylev at cortex-it.ch (Simeon Bobylev Cortex IT) Date: Mon, 26 Feb 2018 16:11:40 +0100 Subject: [caasp-beta] Monitoring Message-ID: <0B0EF9EE-769A-41EA-96E5-3BEFFC44D492@cortex-it.ch> Hi, I?m trying to monitor caasp cluster by using this guide: https://www.suse.com/documentation/suse-caasp-2/singlehtml/book_caasp_deployment/book_caasp_deployment.html#cluster.monitoring.grafana All pods are running, I can access Grafana UI, data source is working, but I can?t see any data points : Is there some other config to apply to make it works ? Best regards, Sim?on Bobylev ? System Engineer ? Cortex IT SA ? Route de l'Ile aux Bois 1A ? 1870 Monthey ? Switzerland Direct: +41 24 524 1204 ? Main: +41 24 524 1200 ? Email: simeon.bobylev at cortex-it.ch http://www.cortex-it.ch/ ? A WIRD Group Company This e-mail has been sent by a trading and services firm. It is confidential and may be privileged. Only the intended recipient may read, copy and use it. If you have received it in error, please contact us immediately. Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: PastedGraphic-1.png Type: image/png Size: 105262 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: PastedGraphic-2.png Type: image/png Size: 100398 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ZcoSignatureImage_simeon_bobylev_image001.png Type: image/png Size: 4217 bytes Desc: not available URL: From Martin.Weiss at suse.com Mon Feb 26 08:56:40 2018 From: Martin.Weiss at suse.com (Martin Weiss) Date: Mon, 26 Feb 2018 08:56:40 -0700 Subject: [caasp-beta] Antw: Monitoring In-Reply-To: <0B0EF9EE-769A-41EA-96E5-3BEFFC44D492@cortex-it.ch> References: <0B0EF9EE-769A-41EA-96E5-3BEFFC44D492@cortex-it.ch> Message-ID: <5A942E380200001C003199D5@prv-mh.provo.novell.com> Hi Sim?on, if I remember right there was one one step missing: --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: heapster roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:heapster subjects: - kind: ServiceAccount name: heapster namespace: kube-system --- HTH Martin Hi, I?m trying to monitor caasp cluster by using this guide: https://www.suse.com/documentation/suse-caasp-2/singlehtml/book_caasp_deployment/book_caasp_deployment.html#cluster.monitoring.grafana All pods are running, I can access Grafana UI, data source is working, but I can?t see any data points : Is there some other config to apply to make it works ? Best regards, Sim?on Bobylev | System Engineer | Cortex IT SA | Route de l'Ile aux Bois 1A | 1870 Monthey | Switzerland Direct: +41 24 524 1204 | Main: +41 24 524 1200 | Email: simeon.bobylev at cortex-it.ch http://www.cortex-it.ch/ | A WIRD Group Company This e-mail has been sent by a trading and services firm. It is confidential and may be privileged. Only the intended recipient may read, copy and use it. If you have received it in error, please contact us immediately. Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 99648 bytes Desc: Portable Network Graphics Format URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 94114 bytes Desc: Portable Network Graphics Format URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 3582 bytes Desc: Portable Network Graphics Format URL: From simeon.bobylev at cortex-it.ch Mon Feb 26 09:21:59 2018 From: simeon.bobylev at cortex-it.ch (Simeon Bobylev Cortex IT) Date: Mon, 26 Feb 2018 17:21:59 +0100 Subject: [caasp-beta] Antw: Monitoring In-Reply-To: <5A942E380200001C003199D5@prv-mh.provo.novell.com> References: <0B0EF9EE-769A-41EA-96E5-3BEFFC44D492@cortex-it.ch> <5A942E380200001C003199D5@prv-mh.provo.novell.com> Message-ID: Hi Martin, It looks better after applying this extra config :) thank you. As other yaml files come from Github k8s repos, I guess this ClusterRoleBinding config is missing from Guthub Suse repo : https://github.com/SUSE/caasp-services/blob/master/contrib/addons/heapster/heapster.yaml Should I create a PR with this conf included in heapster.yaml ? Sim?on Bobylev ? System Engineer ? Cortex IT SA ? Route de l'Ile aux Bois 1A ? 1870 Monthey ? Switzerland Direct: +41 24 524 1204 ? Main: +41 24 524 1200 ? Email: simeon.bobylev at cortex-it.ch http://www.cortex-it.ch/ ? A WIRD Group Company This e-mail has been sent by a trading and services firm. It is confidential and may be privileged. Only the intended recipient may read, copy and use it. If you have received it in error, please contact us immediately. Thank you. > On 26 Feb 2018, at 16:56, Martin Weiss wrote: > > Hi Sim?on, > > if I remember right there was one one step missing: > > --- > kind: ClusterRoleBinding > apiVersion: rbac.authorization.k8s.io/v1beta1 > metadata: > name: heapster > roleRef: > apiGroup: rbac.authorization.k8s.io > kind: ClusterRole > name: system:heapster > subjects: > - kind: ServiceAccount > name: heapster > namespace: kube-system > --- > > HTH > Martin > > > > Hi, > > I?m trying to monitor caasp cluster by using this guide: > > https://www.suse.com/documentation/suse-caasp-2/singlehtml/book_caasp_deployment/book_caasp_deployment.html#cluster.monitoring.grafana > > All pods are running, I can access Grafana UI, data source is working, but I can?t see any data points : > > > > > > Is there some other config to apply to make it works ? > > > Best regards, > > > > > Sim?on Bobylev | System Engineer | Cortex IT SA | Route de l'Ile aux Bois 1A | 1870 Monthey | Switzerland > Direct: +41 24 524 1204 | Main: +41 24 524 1200 | Email: simeon.bobylev at cortex-it.ch > http://www.cortex-it.ch/ | A WIRD Group Company > > This e-mail has been sent by a trading and services firm. It is confidential and may be privileged. Only the intended recipient may read, copy and use it. If you have received it in error, please contact us immediately. Thank you. > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ZcoSignatureImage_simeon_bobylev_image001.png Type: image/png Size: 4217 bytes Desc: not available URL: From Martin.Weiss at suse.com Mon Feb 26 09:56:43 2018 From: Martin.Weiss at suse.com (Martin Weiss) Date: Mon, 26 Feb 2018 09:56:43 -0700 Subject: [caasp-beta] Antw: Monitoring In-Reply-To: References: <0B0EF9EE-769A-41EA-96E5-3BEFFC44D492@cortex-it.ch> <5A942E380200001C003199D5@prv-mh.provo.novell.com> Message-ID: <5A943C4B0200001C00319A66@prv-mh.provo.novell.com> Hi Sim?on, I have no idea if/why this is missing ;-) - but yes - feel free to create an issue/PR.. Thanks! Martin Hi Martin, It looks better after applying this extra config :) thank you. As other yaml files come from Github k8s repos, I guess this ClusterRoleBinding config is missing from Guthub Suse repo : https://github.com/SUSE/caasp-services/blob/master/contrib/addons/heapster/heapster.yaml Should I create a PR with this conf included in heapster.yaml ? Sim?on Bobylev | System Engineer | Cortex IT SA | Route de l'Ile aux Bois 1A | 1870 Monthey | Switzerland Direct: +41 24 524 1204 | Main: +41 24 524 1200 | Email: simeon.bobylev at cortex-it.ch http://www.cortex-it.ch/ | A WIRD Group Company This e-mail has been sent by a trading and services firm. It is confidential and may be privileged. Only the intended recipient may read, copy and use it. If you have received it in error, please contact us immediately. Thank you. On 26 Feb 2018, at 16:56, Martin Weiss wrote: Hi Sim?on, if I remember right there was one one step missing: --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: heapster roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:heapster subjects: - kind: ServiceAccount name: heapster namespace: kube-system --- HTH Martin Hi, I?m trying to monitor caasp cluster by using this guide: https://www.suse.com/documentation/suse-caasp-2/singlehtml/book_caasp_deployment/book_caasp_deployment.html#cluster.monitoring.grafana All pods are running, I can access Grafana UI, data source is working, but I can?t see any data points : Is there some other config to apply to make it works ? Best regards, Sim?on Bobylev | System Engineer | Cortex IT SA | Route de l'Ile aux Bois 1A | 1870 Monthey | Switzerland Direct: +41 24 524 1204 | Main: +41 24 524 1200 | Email: simeon.bobylev at cortex-it.ch http://www.cortex-it.ch/ | A WIRD Group Company This e-mail has been sent by a trading and services firm. It is confidential and may be privileged. Only the intended recipient may read, copy and use it. If you have received it in error, please contact us immediately. Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 3582 bytes Desc: Portable Network Graphics Format URL: