Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Minikube & Three Different Local Kubernetes Learning Environments - Anders Björklund & Predrag Rogic

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Minikube and Three Different Local Kubernetes Learning Environments - Anders Björklund & Predrag Rogic

We will do an Introduction to the Minikube project for running Kubernetes locally, including an updated development roadmap. Users will learn what is new with the project, and how they can contribute to it and get started wtih learning Kubernetes. The Deep Dive will be going into the three different environments that Minikube supports: Hypervisor, Container and Bare Metal There will be details about the pros and cons of each, and some suggestions and where you would choose one over the other...

A

Hello, everyone and thank you for taking part in this mini cube session. My name is fredrik rogic and I am one of the contributors to the mini cube project.

A

This session has two parts. First part is an introduction to the mini-q project where we will cover. Why was mini-q project created what it is and what it's not where we are now and what are the plans for the future?

A

Also we'll talk about how you can join the community and start contributing to the project.

A

The second part will be deep dive to local kubernetes learning environments, and here others will introduce the three different environments that mini cube, supports, hypervisor, container and bare metal.

A

There will be details about each pros sequence and some suggestions. Where would you want to choose one over the other, and since this is a recorded session, we'll leave some time for answering your questions. At the end, the session time is limited to about 30 minutes. So let's jump straight on and talk a little bit about the background and explain which problems were meant to be solved, but by this project.

A

So this year is a special anniversary for mini cube that started five years ago, offering a solution that would improve the existing single node local cluster experience in kubernetes.

A

As you can see here, the original proposal was not to deal with multi-node clusters nor to be used for production workloads and, as we will see later on, this original vision changed a little bit, but for the better.

A

The primary goal of mini cube is to make it simple and easy for people to run kubernetes locally for learning.

B

A

Day-To-Day development, including testing and debugging, and the main guiding principles, are to be inclusive and community. Driven user friendly to support all kubernetes features to be cross-platform, reliable, high-performance and developer focused.

A

Here are some specific design goals derived from those principles, simple ux that allows setup and teardown of the cluster with the single command as easy as minicube start and minicube delete, unified ux cross operating systems, support for local storage, networking, auto scaling, dns load, balancing and so on, minimal dependencies on third-party software as much as possible and minimal resource overhead and we'll touch on this one in more details here, you can also visit the link to original design proposal document. If you're interested to learn more we're now going to see how minicube development is reflecting.

A

Specifically, these principles and goals.

A

This is an example: output of the minicube start command all it takes to bring up local kubernetes cluster in less than 20 seconds.

A

Also, you don't need to install separately kubernetes tooling needed to operate your cluster like cubectl, as you can use the ones conveniently provided for you by minitube for your operating system and your architecture.

A

Now there are other great tools as well. That would allow you to achieve similar things, that is to run kubernetes cluster locally and they are usually specialized and focused on specific operating systems or architectures. Container technologies or virtual machines.

A

Minicube, on the other hand, is aiming to support as many configurations as our users commonly use, and what you see here is not a definitive list of supported components, rather those that are being actively developed and thoroughly tested.

A

So starting from the lowest layers. Minicube runs on amd 64 as well as arm 64, cpu architectures in terms of the operating systems, linux, macos and windows are supported.

A

Next, it can use a wide variety of hypervisors and drivers like kvm, hyperkit, hyper-v and virtualbox, then docker podman, and you can also use an existing operating system, whereas native driver would allow you to use your local os and ssh would allow you to use a remote operating system for bootstrapper we use cubeadm and for container runtimes. We support docker, container d and cryo.

A

Finally, while kubernetes officially supports just last three minor releases: we've recognized that there are some users who are, for various reasons, still using older versions, and so we support at least last six minor versions that are continuously tested.

A

Similarly, uh this is not a definitive list of all the mini cube features more of an overview of the most important or recent ones.

A

So currently there are 28 add-ons you can use to easily install kubernetes applications and two of them are installed by default and those are the default storage class and storage provisioner. You can see all supported add-ons with mini cube add-ons list command, as shown here on the right and to install additional add-ons to the cluster. You can use mini, cube, add-ons enable command, and you can also disable configure add-ons and so on and similarly to other commands, simply use mini cube add-ons to get the detailed usage help on that for the advanced kubernetes features.

A

Minicube supports a load, balancer node port file system, mounts feature gates and so on, and there are now eight ways to push images and some of them allow fast load and building.

A

To support the next billion users initiative and make mini cube more inclusive. There is a continuous effort to support non-english speaking people by allowing them to report problems in their native language like github issues, and currently there are seven languages supported.

A

Then users can have a conversation in their native language with the project maintainers and contributors, which would use usually the help of google translate. So sometimes it may sound a little bit funny, but you know we're trying and the mini cube itself currently supports seven non-english languages, like german.

B

A

French, japanese, korean polish and chinese- and there are also plans to improve it further and add more languages, and here we need help from your side.

A

If you remember from the beginning of the presentation, initially, minicube was envisioned as not to support multi-node, but it turned out to be the most wanted feature user asks for, so that changed the original assumptions and adding multi-node support came into the focus.

A

It's one of the examples of how the community is actually staring this project.

A

So after a couple of iterations and going through alpha and beta phases, multi-node finally became ga in december 2020.

A

Now, while adding more features, more tests are also added to retain reliability, but there also require more resources and since maintaining a high performance is also one of the guiding principles. It has the continuous focus and that's resulted in improvements of 86 percent faster start time and 20 percent less cpu usage. These metrics.

B

A

Measured with each pr so uh that we know if something would negatively impact miniky performance here, on the right hand side, I hope you can see an example of appear metrics where, with the docker driver start time is around 18 seconds for details. I would recommend that you visit these two great presentations held by media and priya at kubecon last year.

A

Also arm64 support was added for docker and boatman drivers, as well as for apple and one chip.

A

There are new commands introduced like image load and build pause on pause schedule, stop ssh cost, etc.

A

Amongst latest add-ons that were added, are auto-pos csi, hostpass driver volume, snapshot and so on to make mini cube more user friendly and simpler automation plays an important role and therefore the improvements in automating driver and cni selection setting default cpu and memory parameters appropriate for your machine, auto pause add-on and so on.

A

There's a monthly cadence for each new release. So many new features and improvements are delivered regularly.

A

Okay, so what's ahead of us, the plan for the next release is captured in the current github milestone plan, it's accessible through the link given here, and also the development plan for this year. Amongst other things, includes removal of dependency on the machine that is deprecated and no longer maintained.

A

Last commit was some two years ago, the documentation, improvements using container runtime interface and container network interface by default, graphical user interface for start stop and potentially other commands, future improvements to resource monitoring and alerts, and currently, as you saw, it, takes about 18 seconds to start up the cluster and the goal is to go under 15 seconds. So we.

B

Are closed but not.

A

There yet and also the kernel assistant mounts like ships and nfs are implants.

A

Also other things will be prioritized and added to the development plan going further mini cube community is continuously growing and by the time you see this video streamed, the number will likely to be higher. There are more than 600 contributors at the moment, whereas the number of non-google commits rose by almost 75 percent. In the past couple of months, the github project has more than 20 000 stars and more than 3 000 works and there are almost 5 000 mini cubes like channel members.

A

How can you become or part of the mini cube? You can join the slack channel mailing lists for users and or developers, and since this is a community driven project, you can also take part in an open, bi-weekly office hours meetings, as well as triage parties.

A

Now triage parties, weekly meeting where maintainers and contributors discuss open issues prioritize and make sure they are timely, addressed last but not least, as seen in an example with multi-node feature, anyone can impact the project by sharing how they're, using mini cube and what they think could be improved or added. So feedback is also collected through an always open, quick, five question survey and the results are regularly assessed.

A

Please share your experience and thoughts with us. It really matters if you would like to contribute to mini cube here are some guidance of how you can start the getting started and also how to contribute guides are on the website.

A

Those github issues that are considered a good starting point are marked with good first issue label. So look for those and don't be afraid you will get further guidance and any help needed from maintainers and other contributors as needed.

A

We also need a lot of help with documentation and translation, as well as specific help with expertise in networking, testing, cicd automation, containers and so on and due to time constraints. We could not go much further in details here, but there are links in the presentation that could point you into the right direction. Should you want to learn more? I hope this was informative for you, and I want to thank you for your time and attention now. Let's move on to the mini cube, deep dive with honors.

A

B

Hello and my name is anderson and I've been working as a mini cube maintainer for two and a half years now, and- and today I will be talking about three different local kubernetes environment, the hypervisor, the container and the bare metal environment and the hypervisor environment creates a virtual machine running a separate kernel.

B

This is also called hardware assisted virtualization in the container environment. We create a system privilege container using the same color as the host. This is also called os level virtualization and in the bare metal we run the kubernetes components directly on the host, and this host may in turn be a virtual machine, and then we will be using uh hardware assisted virtualization under the hood, uh in addition to this environment, that we also have mini cube running on different uh operating systems uh in the traditional meaning, not in the linux distribution, meaning.

B

So when we run on mac, os or windows, we will need to run the kubernetes control pane components in a virtual machine either if it's visible to us or if it's hidden from us. In addition to this complexity, we also have different architectures.

B

uh Currently, we are trying to support two uh out of the five kubernetes environments: uh amd64, the traditional x8664 pc environment, and also this new arm 64 environment like the apple m1 or the raspberry pi or similar device um for the raspberry pi. There is also the 32-bit arm, which is not used for the virtualization, but it can be run in native mode with the mini cube.

B

So when you go to install kubernetes on a machine, the first thing you need to do is to install a server running linux, since the control plane only runs on linux at this time in minicube. This is done by using the driver on this machine or server. We don't need to install the container runtime.

B

Traditionally this was docker, but you have other options to choose from, as we will see later in minicube. This is done by the provisioner.

B

The third and final step is to install the kubernetes control plate components in mini cube. This is done by the bootstrapper.

B

This will install the software and configure and initialize the control plane. uh In this presentation we will not cover running multi-node or running multimaster.

B

uh But after you have set up the control plane, you can, for instance, join some working nodes to the control, plane, node and you can install the kubernetes client cube cdl on the host and finally go to deploy your applications.

B

uh When you run a single node uh cluster with mini cube, we paint the control play node so that you can run apps also on the single node, with this cubicity obtained command.

B

Worth noting is that minicab not only provides kubernetes access, but we also provide ssh access to the machine and we even expose the control container runtime to the user, for instance, for loading and building images, and traditionally this was done by a command called docker, and that gave you access to the docker demon running on the machine, but in later versions. We have bundled this in a mini cube image command to make it agnostic to the different container runtimes.

B

So there is an image load and image, build command.

B

So these three environments can be further broken down into six different environments for the hypervisor. We have the traditional oracle virtualbox external installation, but you can also run with the so-called native virtualization of each operating system for the container environment. We both have the docker engine running as processes in linux, and we have the docker desktop environment, which runs using virtual machines on mac windows and for the bare metal.

B

Finally, you can either run it local, as we have done traditionally or in the latest version I mean we can even run a remote cluster so that the actual cluster can reside on a different physical machine than your current laptop, and then we use ssh to connect to this remote machine.

B

You can then break this down even further by looking at different vendors of the different solutions. So, in addition to virtualbox, you might might also select vmware or parallels for your external virtualization and the operating system. Virtualization comes in different flavors, for instance, on windows you would use hyper-v and for on mac. You would use something x, high based such as docker's hyperkit and for linux, we're using the libvard daemon access to the qm kvm, the kernel virtual machine for the container environment.

B

We also support podman as an alternative to docker when running natively on linux, and you have the docker desktop running on mac or on windows, and there is also a new podman machine you can use as an alternative when using podman on mac.

B

The bare metal, as we mentioned earlier, can come in a regular physical server or a virtual machine server. This can be run locally and connected to your local network, or it could be run as a virtual machine. It could even run remotely in a private cloud.

B

And here we can look at the difference on different approaches in the more simple common solution of having a physical server and then just installing the container runtime directly on this we have one linux kernel running directly on the server and then this um uh container runtime will start the pods for kubernetes.

B

When we have a virtual server or when we're running mini cube in a virtual machine, then we have a hypervisor running the linux kernel on the native kernel of the host, which could be a linux kernel or it could be a windows or mac canon, and in this virtual machine we will. Then you run the container runtime like we did in the physical alternative, when running mini cube with a docker driver or the podman driver.

B

We create a system container that emulates a node and uh it works something like um actual virtual machine, except that is. It is running on the same uh native kernel as um the host, which gives it a lower overhead inside this um driver runtime. We run the kubernetes container runtime.

B

So here we show docker inductor, but you can also run container d in docker or cryo in podman and finally, the most complex solution where we have the docker desktop driver where we have a docker running in a virtual machine with a linux kernel on the native system and then inside this virtual machine. We have the same setup as with the docker engine driver.

B

So there are two different uh connors involved here, but there are also two different uh container runtimes involved.

B

If we look further at those three install steps, you can subdivide them into more detailed uh approaches. The first thing is that you need a image in order to start your operating system and we do bundle a mini cube iso for use with the watching machine which is um originally was based on boot to docker.

B

When running in the container environment we use ubuntu 2004 as the base. Then you will use the machine provider to install the server with linux. Using this image, be it virtualbox will be docker running the image.

B

Then we have the second step where we provision the container runtime we configured and install docker or continuity or cryo, and we also make sure to make the kubernetes images available in this runtime.

B

We call this the preload step, where we preload all the needed containers from a tarball into the environment or from a cache directory, and finally, the third step was to bootstrap the kubernetes control plane and also to install the kubernetes networking if we're running in a multi-node scenario, or if we just want to replace the networking on the local uh single node device and for the operating system. We had the boot to docker options same as docker machine, which is based on tiny core linux.

B

This is now deprecated in favor of a custom build root environment which was originally based on core os, and this image is used for the virtual machine.

B

It's called the iso traditionally referring to the cd image and for the container environment called the kick for kubernetes in container. We are using ubuntu, which is the same as used in the kind kubernetes product, and then we install the software using regular system packages inside this os.

B

You can also bring your own linux supported distribution, and we will try to configure and install kubernetes based on the os that you provide, and uh this is not fully supported, but we will need user support to support the more um alternative installations here um for the container runtime we have traditionally used the darker as the container runtime in kubernetes and mini cube, has supported cryo for many years as an alternative for building the images for cryo or for loading them.

B

We use the podman and command line tool running in the virtual machine and the latest uh addition to this family is to run container d and when using container d, we use the build kit and build the daemon as the means of building images using the mini cube machine.

B

And since uh the latest version of kubernetes, the docker shim has been deprecated, which means that we need to provide a cri. Also for docker. Cri stands for container runtime interface.

B

And traditionally minicube shipped all the communities component in a single binary around this, but since this was a maintenance burden and also meant it differed a bit from the regular installation. We are now using the standard cube adm installer, and this is the same binary you would use to install in a production cluster.

B

And, as I mentioned, we download the required binaries and the images for this kubereum installer in the preloaded tarble.

B

When it comes to the cni or container networking interface- and we have traditionally used darker, but now since it's deprecated the docker shim, we are making the cni default to all the container runtimes, either using the standard single node bridge or perhaps in the container environment, the kindnet ptp alternative.

B

But when it comes to multi-node, you would install something like flannel or some other option for connecting multiple machines.

B

So with all these alternatives, when would you choose to use a specific driver- and here I will give three examples on when you would use a driver? The first one is the so-called traditional setup, where you would run mini cube in a virtual machine.

B

The way it's been done for years now in mini cube and traditionally using virtualbox, but there was also these native virtualization alternatives.

B

The second is to run mini cube in a system container using this new docker driver, and the third option is running mini cube in a vm provided by the user in a localhost shell on the control plane itself.

B

So, for the first use case, you would run mini, cube, start and select the virtualbox driver, you can also run minikivstart.vm and it will pre pick the suitable native driver and the pros of this approach is that it is running the os provided by minicube, which is then compatible with the kubernetes version being installed, and this setup has been tried and tested.

B

The cons are, however, that it is quite high overhead compared to some of the other options, and you will need a platform with support, for instance, for virtualization or in this case virtualbox, which does not support the arm 64 architecture.

B

If you have taken the kubernetes introduction course. I believe this uses the version bookstore.

B

The second driver- and the second uh driver we will look at here- uh is the new docker driver, and this will uh then start a system container with mini cube and everything in it. Since it's a container and not a full virtual machine. This will give a lower startup time.

B

uh Assuming then that, if you're on a platform which needs a virtual machine in order to run containers that this virtual machine was actually running beforehand and the resource usage uh when running natively on the host means that you will not have to spin up a second corner, and you can share memory with the the other processes on the host and the cons with this docker driver is that the kernel compatibility is limited to what the host can offer.

B

So you cannot use this mini cube, watching machine to run all different kernel based deployments when it comes to, for instance, storage systems or other uses of minicube.

B

And uh finally, the the networking we're running we're running in a virtual machine is somewhat limited, which means that we need to tunnel all the networking into this virtual machine running the actual container, as opposed to running the virtual machine directly where we have access to the network. Interface of the machine and running kubernetes in docker is something that kind has been doing for kubernetes testing that has been adopted for many cube used with the container building and the ssh access.

B

And finally, we have the so-called non-driver, which means that minicube will run all the kubereum commands and all the containers directly on the host. The pros with this approach is that it's a low overhead, since we don't have any virtualization or any containerization and.

A

B

Also, quite simple in terms of which processes and which virtualization layers are running, and the cons, however, is that uh since we are logged into the control plane, we don't have all the niceties of running a regular desktop and there is no isolation, which means that all processes started will run directly on the machine.

B

So this is not something that is recommended to run on your laptop, but it can be used when running in a virtual machine that you have set up or, for instance, we're running in ci.

B

If you have done the hello mini cube tutorial with the catacoda from the kubernetes website. This uses the non-driver in a virtual machine accessible from the web console.

B

And that was it for the presentations. uh If you have further questions, you can always reach us on slack or on github when it comes to issues and we will not be available in the chat to answer any other questions, you will have thank you for listening.