Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2021, 15 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Sponsored Session: Elastic - Autoscaled K8s Observability with the Elastic Operator

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Sponsored Session: Elastic - Autoscaled K8s Observability with the Elastic Operator

Speakers: Michael Morello

Ensuring the observability of all the dynamic components in a containerized ecosystem can be a challenge. As the number of pods and services in your environment increases, you need observability tools that can scale with you. Your observability tool has to not only collect logs, metrics, and application performance data but also allow engineers to derive actionable insights as soon as issues with services occur.

In this session, you will learn how to deploy Elastic Cloud on Kubernetes and use the power of built-in autoscaling to seamlessly scale out as you continuously collect and store more observability data for diagnosing, debugging, and resolving issues in your applications.

A

Hi there thanks for watching this video, my name is mikhail marlow. I am a software engineer at elastic and I am working on the elastic cloud on kubernetes operator.

A

I am really excited and happy to have this opportunity to make you a short presentation and also a short demo of our kubernetes operator or for observability solution, but, most importantly for me to show you the last really nice feature of the operator, which is the ability to automatically scale elasticsearch clusters. It is a very exciting feature released as an alpha feature in the version 1.5 of the operator, but before I would like to take just a few seconds to briefly present what elasticsearch is and all different solutions at the earth of the elastic stack.

A

There is elasticsearch, which is a distributed, data store and such an analytic engine as a developer. You can interact with elasticsearch through its api, but you might also want to deploy cuban on top of it. Kibana is a data visualization dashboard for elasticsearch. I will show you some of its features in a moment.

A

These two fundamental components enable the switch solutions which are enterprise search, which allows you to implement a powerful search experience on your data and in your applications, the elastic observability solution, which is what we will focus on today and finally, elastic security, which is a security information and advanced management system.

A

All those three solutions are powered by elasticsearch and kibana to fit them with some data. You can deploy some datashippers today. I will show you how to deploy bits on kubernetes, more specifically, five bit to ship your application logs and metric bit to ship matrix running.

A

All these products is not that straightforward, mostly because we are talking about managing stateful workloads, but we are also here to help you can either use elastic cloud, which is our service I'll, deploy them on premise, using either ece, which is basically the software which is running our cloud services or you can also use the operator, which is what I will show you right now in the demo.

A

So the first thing that I want to do is to deploy the operator on my kubernetes cluster. There are a couple of ways to do that. The easiest one is to use all in one manifest. I will show you how to do that in a moment if you are a more advanced user and if you need more flexibility, you can also have a look at our new m chart.

A

Also, the operator is available on the operator hub and if you are a red hat, openshift user. Well, the elastic cloud on kubernetes operator is a red hat, openshift certified operator and is already available on the red.openshift console.

A

But now, as I said, I just want to use all in one manifest so using the old one manifest to deploy the operator. It's just a matter of applying um this pmo file on my cluster.

A

Looks good, let's see if my if the operator is running.

A

Yeah looks great, maybe let's also have a look at the custom resource definitions which have been installed.

A

So, as you can see here, we have almost one custom resource definition for each of the application of the elastic stack, but what I would like to do right now is to focus on elasticsearch. So let's have a look on the elasticsearch manifest example.

A

So this is what an elasticsearch resource looks like it lets you define the topology of the elasticsearch cluster. You want to deploy on your kubernetes cluster. So in my case in this example, it is a kind of a simple topology. I have a first note set with actually just one node, which is a dedicated master node, and I I also have a second set of nodes, actually two nodes in this data set, which are some almost dedicated data nodes and for those data nodes.

A

I want to use some 10 gigabytes per system volumes using the default storage class. So let's apply this manifest.

A

What I would like to do also is to deploy right now, five bit and meteor b, just to save some time, but I will get back to these examples in a moment.

A

Okay sounds good, so let's maybe just have a look yeah, okay. So what what is happening right now behind the scenes is that the operator is creating all the kubernetes resources required to run an elasticsearch cluster on kubernetes. So, for example, we need some secrets. We need some network services and obviously, if we need some code, so let's have a look at the board for this cluster, so they are almost running so here you can see my dedicated master, node and emi two data nodes.

A

In the meantime, maybe, let's just have a look at a metric bit resource, so this is metric bit. um So, as you can see here, all I have to do in order to define to which elastic such cluster.

A

I want to send my data is just to add this elasticsearch ref field and what the operator does behind the scene is that he's creating all the required configuration in order to establish a secured connection between uh all these this instances of metric bit and my elasticsearch cluster in the context of bit, I can also do the same for kibana. It allows a bit to set up and install some sample dashboard.

A

As you can see, there are a lot of yammer layers. This is because the resources give you a lot of flexibility. What about what kind of matrix you want to collect and how you want to collect? It also allows you to define how you want to deploy the pods. So in my case, I want to use a demand set to have one instance of each of my bits on each of my kubernetes nodes.

A

Okay, so let's have another look at my resources, for example, kibana and beats ah okay looks good everything seems to be deployed and deployed in green. So let's try to log in to kibana.

A

Okay, so in order to log into kibana, I will use a default user named elastic.

A

This is an elastic generated automatically by the by the operator.

A

It allows you to log in a first time into kibana oops. Maybe I miss I made a typo.

A

Okay looks good, so maybe let's try to see if I already have some loads. Okay, I've just deployed five bits, so this is kind of expected. Let's see if we have some matrix so not yet.

A

Try to refresh.

A

To see some logs.

A

Okay, it's better! I have some notes. So, let's see, for example, uh if I can filter on my elastic search, pods logs, for example, yeah, that's good!

A

So, as you can see here, I'm now able to search into the logs produced by by all my ports on my kubernetes cluster. So let's try to see again if I have some matrix.

A

Yes, so as you can see, it is a six node kubernetes cluster. um So what we are seeing here is a cpu usage for each of my cubientis nodes. So maybe let's try to have a look at the pods matrix. So let's keep cpu usage. I just want to group by namespace, because I have a lot of pods keep going demo.

A

Let's say, for example, this one.

A

Okay, let's maybe change the time window, yes, okay! So, as you can see here, I can. It is very easy for me to have a look at the cpu's age, the memory and then the track traffic for each of my ports.

A

What I can also check if, as I said, a metric bit automatically setup installs some sample dashboards and, for example, this one should give me a more specific overview of my compensator okay great so now, as you can see, I have a lot of data being ingested on my kubernetes on my elasticsearch cluster and what I would like to show you is how to scale automatically my elasticsearch cluster to adjust the storage capacity.

A

uh So, let's get back to my elasticsearch resource and let's I just want to show you how to enable autoscaling on an elasticsearch cluster. So in order to enable autoscaling, all you have to do is to declare what we call an auto scanning policy. Another scanning policy will, let you let you describe um the interior of the auto scaling controller on a set of nodes. So in this example, I just want to control um this.

A

uh The nodes with uh which have this set of rules- and what I want to define here, is that I want to have at a minimum two pods, and I just want for the operator to be able to to create three additional nodes so up to five nodes to adjust the storage capacity for my cluster.

A

If you have a storage class which supports volume expansion, you can also define a different mean and different max for the storage capacity. In that case, the um the operator will automatically scale vertically first, the storage capacity and just and then I add some nodes as a required and so auto scaling is an enterprise feature.

A

So what I have to do is I will just set up a trial license and what I want to do also is to deploy some support to generate a lot of logs and a lot of data on my customer to speed up a bit. The demo.

A

Okay looks good uh so now what we can do is that we can go back to kibana.

A

So for the purpose of this demo, I've set up a sample dashboard to let you visualize the remaining disk space capacity in the volumes of my two initial nodes. So it's not as a percent volume of 10 gigabytes.

A

So maybe, let's also adjust the time window to have a better overview of what will happen.

A

Since I wanted to show you a realistic demo, it will take a few hours to fill up the volume, so the rest of this demo will be recording and I will speed up a bit the video. So, let's start.

A

Okay, there is almost one gigabyte of storage space remaining and, as you can see, the operator has automatically added a new node to the cluster to my elasticsearch cluster and the data automatically rebalance across the nodes, and if we wait a bit more, then another node is automatically added and then again the last one.

A

uh So let's have a look at the pose and, as you can see here, you can see the two initial ports and then the second one which has been headed a few hours later and then the two other ones.

A

So you we can check. You can check the auto scaling status by having a look at this annotation. This one uh store in the electric search resource and, as you can see here, the amount of data managed by my cluster at the end of the demo would still require another node.

A

But the operator will not go beyond the five node limit. I set in c auto scanning policy. um Also, the same information is reported through some kubernetes event.

A

Okay, so I hope that you have enjoyed this presentation and the demo and if it's not already done just come visit us at the elastic booth, it is a great opportunity to chat to meet our experts and ask your questions and more so see you. There bye.