Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2022, 12 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Keynote: A Cloud Native Swiss Knife - Ricardo Rocha, Computing Engineer, CERN

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Keynote: A Cloud Native Swiss Knife - Ricardo Rocha, Computing Engineer, CERN

Speakers: Ricardo Rocha
Dealing with cloud native deployments can be challenging at first. Knowing and having the right tool for each task can make all the difference, ideally conveniently stored in a close by pocket.

In this session Ricardo will present what could be a cloud native swiss knife. A set of tools and functionalities he wishes he knew existed from the start, as well as others the community introduced later on which have significantly improved the daily life of developers and operators.

A

Yay, this is working on your own. Let me get the clicker, so yeah welcome everyone. Thank you again for for giving me the opportunity to talk about uh this. uh What I call a cloud native Swiss knife? My name is Ricardo I'm, a computer engineer at CERN. uh Let's jump right through it, this works yeah, it does all right, so I'm from Switzerland, like CERN, is based in Switzerland. So one common thing that we see everyone carries often is a Swiss knife. It's a handy tool.

A

It has actually created more than 100 years ago, and it was the description that I found was that it was a tool that allowed people to or soldiers in this case, open canned foods and maintain the Swiss service rifle so two very different things, but two things that a group of people had to do often when they were in service. So this became really popular to the point that the company that was building them started.

A

Adding new tools, people had ideas, let's put more things in and they started producing different variations of the Swiss knife, and this was variations that would suit different types of individual users. So you can see kind of where I'm getting here like you have a lot of tools available, but you also have users that will have different needs and they will need to package things differently.

A

So on the left, here we have the original Swiss knife on the in the middle. We have a modern version of The Swiss knife, which is pretty much similar to the first one, but then on the right. We have a very different Swiss knife, it's something like from the blades and the the tools to open canned food. Suddenly we have a laser pointer. We have a USB, stick, some sort of scissor and someone actually thought it was a good idea and practical to carry that, so they built it and you can get it.

A

Of course, some people are a bit more demanding. So this is another Swiss knife, and suddenly you have a lot of tools if I look at them, I actually don't know what all the tools are for, but I I recognize that someone actually thought that they need these tools and the company started building it. So you can maybe see like a magnifying lens, some scissors and some sort of cutting tool and someone needs it. It's not a knife that I would carry, but it's something that someone someone did now.

A

This can go a bit too far. So this is the current world Guinness record for the largest Swiss knife, and it's a it's. Definitely something I would not carry I doubt that someone is actually doing it, but but it it it's to see how far you can push a good concept to something that is entertaining but a bit silly.

A

So coming back to the the point, what what I would like to highlight today is this uh comparison between uh the idea of a Swiss knife having small tools together that suit individual users and what we do with Cloud native. So I'll tell you a bit about uh what I would put in my own Swiss knife and in my Swiss knife. The first thing I would need for cloud native is something to do.

A

Logging, like logging, is really important, so you I would need kubernetes kubernetes actually gets you really really far on what you can do with logging, but you also need projects like fluentd that aggregate the logs and push them centrally and then for metrics very similar in a very similar way. You would need a Prometheus now. I'll do a really really simple demo, but it's just to show you how far you can get with something that is a I hope.

A

You can see yeah um something that is kind of uh uh a simple tool, but that gets you really far. So in this case, I have like the traditional uh ngdx deployment that everyone will use and I start getting my logs. Everyone has done this and it's pretty simple, but if you think about it, I'm not even talking about nodes about where the resources is, is handing uh being deployed and suddenly I do a command and the system is streaming. The log back to me. I don't have to care about anything about the infrastructure.

A

So that's that's really powerful, but one thing that is even more powerful is that if we actually look at what's possible with this command, you don't even need to specify a pod the resource.

A

You can actually say that kind of resource you want to stream locks for so the example here, I'm using a label instead of specifying the the part I want to to to get logs from and suddenly like you, you start seeing quite a lot more logs coming and suddenly you you, if you think about it, the system is, is hiding all the complexity for you, it's trimming, the logs. You don't have to give any details about the infrastructure, and this is incredibly powerful and it's something that every time I explain this command to someone.

A

It's something that kind of comes back to me on the simplification from previous systems. I had to deal with now. The second thing that I put on my Swiss knife is also debugging tools and again, kubernetes gets you a long way and debugging is kind of hard in kubernetes, but I think. The reason is one of the best things about containers and kubernetes and Cloud native, which is reproducibility. When we deploy our applications, we create container images, they are immutable and when we deploy our clusters, we actually also use immutable nodes.

A

The the fact that all of this is reproducible- and you can get really far in getting achieving reproducibility is one of the key things about Cloud native tools. Now, when I started using kubernetes, this was actually quite hard to do to handle, but this I wanted to show how things actually get got better as well. So I'll do another very simple demo and I'll come back to my pods here.

A

So if you would try to log into to like to debug a pods, you would probably do something like trying to get a shell to one of them. So let's say I wanted to get a partial, and this is a to the yesterday. We had this nice talk about public container images and best practices, and this is an image that actually has best practices. I was trying to have a bachelor. The image doesn't have patch because it doesn't need it to run the tool, so it just doesn't doesn't provide.

A

So what do I do like I need to access it there's a tool that I don't need. The Temptation is to add stuff to the image. Then you make them bigger, more complex and all of this can kind of escalate. So this this specific image actually has a sage instead of batch. So I can do that. But then I want to debug some Network thing: TCP dump. There is no dispute up so kind of stuck. So again you start putting all these things on the image, not a good idea.

A

Now this wasn't there when I started, but it was at its uh already and it's actually stable in 125 for kubernetes, which is the ability to deploy ephemeral, debug containers. This is a amazingly uh nice feature, which is the fact that you can attach a container to an existing container with a different image, and it will be in the same namespace. You will see the same resources behind so I.

A

Have my image that I call conveniently my own Swiss knife and I can attach it to the same container that I had before, and you can see that it's the same processes I'm seeing here this is pretty cool, but I can see now I can see, speed up and I can see like if some traffic comes in I can actually go and see. Okay, there's some HTML going on here some HTTP protocols, so it's so so easy now to to do this kind of thing that I think I should I thought.

A

I would highlight it also like things like top, are actually available in the original one, but I'm fancy and cool, so I use h-stop and then I'm also fencing cool. So suddenly I have VI as well. So you can extend this to have your all your usual debugging tools available, which is really really handy now. The other part that I mentioned here is hot reloading.

A

This is the other uh complex part, which is the development cycle that people are used to is not necessarily easy to do with the cloud native deployments or wasn't, because when you do changes you have to rebuild images, redeploy and- and this can be a lengthy project- a process.

A

So there are two things I would like to highlight that are really interesting. I won't have time to demo them. One is things like the bridge to kubernetes plugin in vs code and the other one is telepresence, and this is from the magic realm of cloud cloud native where suddenly, your your local development environment becomes part of the production clusters and they do all the necessary tunneling, so that traffic is redirected to your local machine.

A

You can use your idea, IDE checkpoints, everything like as if you would be in the production cluster like this, is something we demo often internally as well, and it's kind of mind-blowing.

A

Now the next thing again, I highlighted the talk yesterday about public container images, I really like it. This is one of the issues we've had in the past couple of years, which is image handling. So we heard yesterday uh the state of the how much work and challenges we have to improve this, but at least here two of the common uh popular images Alpine and Ubuntu.

A

There are 30 megabytes, maybe let's say 100 megabytes who in here raise your hand if you had to head to handle an image that is 100 megabytes or more all right, pretty much. Everyone I was expecting that, so this is in. In reality, we have images that are quite quite a lot bigger, so I give an example. Here we have this ADM admin image that we use for again handy tools that we need and we started growing it. So we now are now close to to one gigabyte.

A

So it's kind of time to think, should we make it smaller? Should we should split it, which is all good things, but who in here has had to use an image? That's one gigabyte or more raise your hand, all right. Still quite a lot of people um now I'll jump right through it who in here has had to handle an image that is 19 gigabytes or more one, two, all right, yeah a lot!

A

Less we actually have these images running on our clusters, and- and there is a reason for that- the example here is from a from an image from an experiment called Atlas, which is one of the LHC experiments. The way they are used deploying their software is, they have a central repository where they build and make everything available all the releases, and they have a very efficient way to distribute codes across the nodes in distributed clusters.

A

When we move to Containers, uh the obvious option was just to containerize all of this, and it's there's actually no easy way to split it, because the jobs got used to having one single place where all the software is available and they access just what they need. If you move to Containers, there's no way to split it and attach no obvious way up to now to attach like one job to to one image. So it's kind of a challenge to deal with this, so we started growing.

A

We got this 19 gigabytes, so we kind of stopped here. Someone had the idea to create like a ranking of the biggest images that CERN, but this would be a really bad idea. So we didn't. We didn't start this, so This Is, Where We Are. Well, actually it's not it's. We have one record now, which is like two weeks ago. Someone actually started using an image that is 76 gigabytes, compressed 125, gigabytes uncompressed, so who in here has an image that is 76 gigabytes?

A

All right, that's good! That's good! You don't want to do this, but actually this I want to highlight this, because there is a use case for this, like Cloud native is not about microservices only anymore. There are so many different use cases for cloud native that you people will come up with these ideas.

A

So the problem here is really like. If you have all these images deployed in hundreds, thousands of nodes- and you start pulling it you're- basically doing a denial of service in your registry and you're putting the network under a lot of pressure, so we can fix in most cases we can fix it and one one suggestion that I that we have is from our head of computing security at CERN, Stefan that you see in the picture and the way he deals with security and users is, he uses three tools?

A

The first one is the baseball bats and the baseball bat is for negotiation tactics. The second one is a water pistol. The water pistol is to kill servers in the data center, because a dead server is pretty secure. The third one is a mob and the mop is to clean up the mess people leave behind when dealing with security. So I think this process is probably something that we can learn from to when dealing with users and the way they package their images.

A

It also helps to have a big smile and, like Star, Wars characters that you can you see on the background to deal with users. All right and then the last bit I want to ah yeah. So the way we deal with this today is technically so not really uh solving the problem with the images that we want to do, but we also have a technical solution, which is to use projects like dragonfly, which does peer-to-peer distribution of containers.

A

So you pull once and you basically Distributing the nodes and you reduce a lot to load on registry. You still deal with big images, though the other one is continuity and the support for remote snapshotter and tools like continuity, where, instead of downloading the image you're actually mounting the image remotely, meaning that you can start, you can tell it immediately and then it will only fetch the files that it actually needs. So this is something we observed.

A

It's like our jobs actually only use around six percent of the of the images when these are like 19 gigabytes. So you can see the improvements on the right. The container startup goes from something like four and a half minutes to 15 seconds and we reduce the network traffic dramatically.

A

So the last tool I will talk about in my Swiss knife is get ups, and this is something we promote internally heavily. There are tools like flux and Argos CD that a lot of people are using already. The principle of githubs is really that you have the central git repository where you declare all your resources, all your custom resources, everything and you can use branches and you can do pull requests and and reviews, and then you attach those definitions to multiple clusters, so we have clusters on premises.

A

We have them in the public cloud and basically you just have to say each cluster should get this. This set of resources from this branch, and you can even have like development staging production environments using this. So it's extremely powerful, it's something we do all the time. It also allows us to do what we call clusters as cattle, which is instead of having one big cluster, where you put everything it becomes really easy to have multiple clusters and spreading the application across multiple clusters, and this is really beneficial.

A

So I'll give you an example of how this is beneficial. I hope this is readable, so, back in earlier this year, just before kubecon Valencia, we actually had a an incident that CERN someone posted in our internal chat saying. Is there anything wrong going wrong with the registry? The UI is failing on me, which is okay. Let's see, then a couple of minutes later someone came and said: okay, I see the same. It's it looks down, so we we went to check.

A

I was around so we're looking and what we realized is that a couple of uh like half an hour before we had updated a maintenance script that started slowing deleting all our production clusters, and these are not like production clusters where we manage everything. These are users, production clusters. Anything can be running there. We have no control and we got really scared and when we stopped it, it had deleted a hundred something clusters.

A

So it's like a third of our capacity and it's pretty serious incident, but we didn't know how far we had gotten with this idea of get UPS to our users, but we actually realized that, even if we deleted a third of our production capacity, we had no downtime in any service.

A

We had degradation, but not only that when we got things back together, people were able to get their clusters back up and running in around 30 minutes, and this is really thanks to to the this idea of clusters as cattle and deploying in multiple clusters instead of having pet clusters and also the fact that you can really dynamically deploy your applications really quickly. So in the end, we had one user saying no worries at the end. It was a chaos monkey test.

A

We pass it, it's not something we want to do again, but it was really a demonstration of the potential of all of this. So I'll finalize saying that this is my own Swiss knife for cloud native. As we mentioned, everyone will have a different one, not only that if you come up with ideas for new tools that you would like to see, just propose them join forces get together with other people, build them and we'll all be able to to to benefit from that work. So thank you very much and enjoy cooking.