Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2021, 15 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Sponsored Session: Rancher - Longhorn for Your Edge Deployments

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Sponsored Session: Rancher - Longhorn for Your Edge Deployments

Speakers: Joshua Moody

Longhorn is an Open Source Cloud-Native distributed block storage engine built on and for Kubernetes that provides persistent storage support for any Kubernetes cluster with a one-click installation. In today's demo Joshua Moody from SUSE will demonstrate how to use Longhorns persistent storage for your edge deployments. We will use Ranchers K3s to manage and deploy Longhorn into our edge cluster consisting of a couple Raspberry Pi worker nodes which act as our data collectors as well as a randomly chosen Raspberry Pi that acts as our data aggregator. We will utilize Longhorns data redundancy and fail over mechanisms to demonstrate a failure and recovery of the data aggregator.

A

Hello and welcome to this tech demo, my name is joshua moody, I'm a staff software engineer working for venturelabs, which is now a part of souza.

A

I work on project longhorn longhorn is a distributed, plug storage engine built for and on kubernetes in today's tech demo, I'm going to show you why you would want to use longhorns persistent storage for your edge deployments to make sure that your data is safe.

A

Let's give a quick overview of how long one works. Long conceptually consists of two components: the control plane, which we can see on this slide, as well as the data plane which we can see on the following slide. The main component for longhorn's control plane is the longhorn manager, it's responsible for orchestration of all volumes, creation of backups in initiation of restores, etc.

A

We also have two auxiliary components, which is the lawrence's eye plugin, as well as the logon ui. Both of these interface. We are with the manager via the longhorn api.

A

The login csi plugin is acts as a bridge between kubernetes and longhorn manager. This way kubernetes can request volume provisioning as well as attachment creation of snapshots, etc.

A

The long data plane consists of two parts, the longhorn engine, which you can think of as the volume controller it's responsible for processing all I o requests, as well as communication with the replicas, which you can think of as the volume's data.

A

Here on the slide, you can see two workloads where each one having their own volume, each volume has only one engine and two replicas per volume. The amount of vapor cost per volume is configurable. Each volume can have a different amount depending on your application, workload's requirements.

A

You can also see that the replicas got randomly scheduled onto different nodes. This allows for the case where one of your nodes fails that your data is still safe. If you look on the left, you can see my messy test setup. If you want to replicate this, you will need four raspberry pi's.

A

I use the four gigabyte version for 128 gigabyte, sd card or bigger, to provide storage to the operating system as well as the cluster. Afterwards, you need to install a 64-bit arm operating system. In my case, I use support server. If you want to use rwx volumes, you need to ensure availability of an nfs4 client.

A

You can do this on ubuntu by installing nfs comments for an operating system. There's alternative packages once your pipes are prepared, you need to install k3s on each of them to form a cluster.

A

At this point, you can optionally add that cluster to your existing venture installation.

A

Let's start with the tag demo, we start by logging into rancho.

A

Here you can see my existing clusters for the tech demo, we're going to use the k3 as ubuntu edge cluster.

A

Let's have a look at the notes: these are my four recipe pies and you can verify that your clusters are working correctly by checking the system information now that we verified the clusters working correctly. Let's start by installing longhorn onto it,.

A

We can use rancher to deploy the longhorn app from the.

A

A

Now that long is up and running, we can have a look at the longer ui.

A

For this cluster, we want to configure a specific setting that allows longhorn to delete parts on failed nodes by default. Kubernetes will not delete parts on failed nodes, since it's the responsibility of cubelet to do so, but if the node is down, cubed isn't running so it cannot delete the part from the api server, which means the part is stuck and any volume that's used by the port is stuck as well.

A

So in longhorn we provide options for deleting the stateful set parts, deployment, ports or both in this case we're going to choose both.

A

Besides the part deletion, we have a prior implementation that is less aggressive, but only applies to deployments in the volume attachment recovery policy implementation. We leave the failed port on the api server, but only clean up the kubernetes volume attachment object. Science on this cluster. We enable both deletion for stateful set as well as deployment ports. We don't need this, so we can disable this feature.

A

Now that we configured our settings, let's go back to rancho to deploy our application workloads.

A

Let's start by deploying our data aggregator workload- you can do this by using cubectl to deploy the yaml file or using wrenches import yaml feature for this example. We're going to use renshaw's import yaml feature.

A

The data creator consists of a service.

A

Persistent volume in this case, we only need a single instance of this data aggregator. If you needed multiple instances, you would use a persistent volume claim template as part of the specification for the stateful set.

A

The only interesting aspect in this data aggregator configuration is that we set the termination craze period to 10 seconds as well as overwrite the default tolerations of kubernetes. That community sets to allow for faster failover, but a fault community sets these tolerations at 5 minutes, which means a part will not be evicted for 5 minutes from a failed node.

A

You can also configure these defaults as part of the cube api server command line or during the k3s.

A

Installation, let's wait for the data aggregator to be up and running.

A

We can see that the volume has already been provisioned.

A

And we can see that the data aggregator is now up and running. Let's continue by deploying our data collector workload.

A

The data collector is a stateless demon set that runs on each of the raspberry pi worker nodes and is responsible for collecting sensor data. Then transmitting that sensor data to the data aggregator service, where it can be processed and.

A

Stored now that both our application workloads are deployed, let's take a minute to discuss why one would want to use persistent storage for the data aggregate to service.

A

In the case, where one uses local storage should the worker node, where the data aggregator is deployed, fail for whatever reason one would lose the data that has currently been stored in the case of longhorn. Since we use three replicas by default with heart anti-affinity, each replica is scheduled on a different worker node.

A

We would still be able to just reschedule the data aggregator service onto a different worker node and continue processing the priorly existing data without any loss of data.

A

Let's show this in action. We start by identifying the worker node, where the data aggregate is currently deployed, then we're going to ssh into the node to turn it off.

A

Let's wait for node to get marked the time that is required for node to get mocked has failed. It can be configured by setting the node monitoring interval as well as the cubelet post status. You can do this during the kts installation now that the worker node has been marked as failed. Let's have a look at our deployment.

A

Currently, the data aggregator is still on unknown status because we're still, within the toleration seconds once that aggregator gets marked for eviction longhorn, will go and delete false delete. The part which allows to be the part to be rescheduled onto a different worker node. Now the data aggregator is up and running on k2s worker 3. Let's have a look at the loan volume.

A

We can see that the status is now degraded.

A

This is because we have a failed replica on worker world node 1., if in the case where our cluster would be bigger, longhorn would now go and rebuild another replica on a different work node. But since we only have three worker nodes in this cluster- and we have part on the affinity, we will run integrated status with only two replicas.

A

Let's recap, as you have seen, a single failure of a raspberry pi did not lead to any data loss.

A

Besides that, you also seen that longhorn cleaned up the workload port so that kubernetes can reschedule it onto a different worker. Node, had you used local storage, you would have lost your data and your data aggregator would have been stuck.

A

Let's make this even better.

A

By setting up a recurring backup schedule for the volume we can ensure that, even in the case where the whole cluster dies, we will not lose any data because we can restore from our backup on a different cluster.

A

In the background you can notice, I have replaced the faulty power which allowed longhorn to rebuild one of the volume rubber cars and transition the volume back into a healthy state.

A

Let's continue by setting up our backup target lauren supports any s3 compatible endpoint, as well as an nfs4 server. For this demo, we're going to use amazon, s3.

A

We set the s3 endpoint as well as the previously created s3 secret.

A

We can now set up the recurring backup schedule for this volume.

A

For the schedule we want to take it back up daily.

A

And we want to retain seven backups so that we have backups for the whole week to start off, we're to kick off a manual backup.

A

We're going to add a custom label.

A

You can use labels to categorize different backups and differentiate between them.

A

We can see the backups in the progress of being created. Let's take a look at our backup store.

A

You can see that I share this backup store with different cluster. We already have volume backups for different volumes.

A

Let's have a look for our volume and the backup that we created.

A

We can manually restore this backup to new volume inside of this cluster.

A

If the original volume would have been lost, we could also recreate.

A

A

As you can see, the volume is currently not ready for workloads, because it's in the process of restoration, we can see the restore process and once it's completed, the volume will be available for workloads.

A

Let's also demonstrate the ability of setting up a dr volume for one of the volumes that is currently not present in this cluster.

A

You can see this is a different volume that is currently not present in this.

A

Cluster, the volume information is recorded, so we can use the same name, size and replica count that was present on the other.

A

Cluster, you can see this marker. That shows you that this is a dr volume.

A

What the dr volume does is continuously monitor the backup store and always restore the latest backup for this volume that is present on the backup store in the failure case of the auto cluster. We can activate this volume and it will become available for.

A

A

This concludes this tech demo. If you want to learn more about longhorn and the upcoming features come join us on the longhorn cncf slack channel.