Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Running K3s, Lightweight Kubernetes, in Production for the Edge & Beyond - Darren Shepherd, Rancher

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Running K3s, Lightweight Kubernetes, in Production for the Edge and Beyond - Darren Shepherd, Rancher

Within the short year since announcing k3s, the CNCF certified lightweight Kubernetes distribution, users have quickly put k3s into production in edge, on-premises, cloud, and pipeline use cases. In this presentation Darren Shepherd, the creator of k3s, will cover different strategies of running k3s and how they map to different use cases. Use cases range from thousands of clusters on the edge, to fast ephemeral clusters to power your pipeline, to running AI workloads, and much more.

https://sched.co/Zeo7

A

All right uh welcome everyone. This is a running k3s, lightweight kubernetes, in production, for the edge and beyond. um Sorry, it's a really bad title. I'm not exactly sure what I was thinking when I proposed that one um but yeah um kind of excited to be doing this. It's a little awkward, um I'm just gonna like just go like one straight cut like no editing or whatever try to keep this authentic, so I'll, probably say some stupid things. um Yeah should be fun.

A

um Welcome to my home office, you might be thinking to yourself, like hey, that kind of looks like a closet. um I'd like to refer you to the sign my daughter made. It says this is my office, so uh it's clearly not a closet. This is my home office all right. So let's get going. um Let's go to the first slide here. So just um what I'm gonna do I'm gonna start off and do a quick overview of k3s. I mean this is only 25 minutes. So it's I can't go super deep on really anything.

A

I did a talk on k3s uh kubecon, uh north america 2019.. um You know that was the last uh kubecon. So you know that was roughly three years ago or something um so I did talk on k3s there.

A

That was um pretty much a deep dive into what k3s is so I'd really refer you to that talk and you'll get a lot of the technical details of what k3s is I'm just going to go over just a high level, just kind of understand what the goals are and everything and talk about some pointers about about the project um and then, specifically with this talk, is I'm going to go into these three use cases, and these are use cases where k3s, you know kind of shines and people have had good success, running k3s and these different things and and just you know, kind of point out how people are doing this.

A

uh You know what are the I don't know, what kind of pros and cons or gotchas or whatever um so we'll talk about edge, app cluster um or application clusters. I'll explain that more when you get to it and then kind of how people are fitting k3s in the pipeline.

A

Okay. So before I get started just a little bit about me, so I am darren shepard. I'm cto and co-founder of rancher labs. um I build the cloud on twitter um if you ever want to get a hold of me or talk to me or anything. Twitter is really the only thing I ever really respond to. um So you can dm me there. That's that's totally fine or you can follow me. I just complained about everything.

A

um uh So, basically, I'm I created k3s and some other rancher projects, uh but k3s kind of, like my that's kind of like my my baby or something, but basically I've been doing cloud orchestration for quite a while. um So before containers it was like uh virtual machines and stuff is called stack, openstack that kind of world and then before that um you know different forms of bare metal and virtualization and stuff like that.

A

Okay, so one thing I wanted to call out is, um as of as as of when this session has been recorded or doesn't make sense as of today. My time, not your time, um we just k3s was just voted through to be accepted as a as a cncf sandbox project, so we're going through that process of getting k3s into cncf as a sandbox, prod project, and then you know as quickly as we can we'll graduate through.

A

We're really excited about this I mean k3s is really not that all of a project- it's only like, I don't know a year and a half, or so it's less than two years old, um but it's growing so fast, it's getting so many users. uh This is just kind of the next logical step of moving it to a foundation trying to get more collaboration from more people in the industry. um It's just kind of a win-win situation. uh So we're really excited about that.

A

Just you know kind of, like our little project has grown up into something really quite big, um and it's really used by quite quite a few people. um I mean lots. Lots I mean it's quite amazing, so I'm pretty excited about that. Okay. um So let's talk about first, what is k3s, um so it's lightweight kubernetes! That's like a primary purpose, most of the decisions and stuff that we do like what we do in kubernetes like components we package or what we like. uh You know.

A

The approaches we take is really focused on a lightweight resource constrained environment. um So we, you know, tend to pick lighter weight, components or you know maybe they're ones that are going to scale massively fire or something because that's not our use case. So we do focus in on this. So um the binary is very small like k through specs is a single binary.

A

um The memory size is, you know, kind of like the initial memory footprint of k3s, it's probably like half the size of kubernetes in general. You can see this really easily. If you just like run, um k3s versus like kind kind, is like more of the traditional upstream model running everything you know it's going to take up like twice as much memory. This is just the way we package it and stuff.

A

Basically, k3s is perfect for the edge it really lines up, really well with the edge use cases, but really in practice. It's used just about everywhere. uh It's like it's really super popular like in devon test. It's taken up a lot in those areas, but also people are even running at cloud data center.

A

A lot of just random use cases that, like kind of fill in gaps that um the kind of traditional it's kind of things, call anything kubernetes traditional at this point but like which is kind of like the the cloud data center model, that's kind of existed and where kubernetes shined uh k3s is finding different use cases that kind of fill in gaps there, um but um kind of the thing. That's really resonated with a lot of people is kind of like this idea of like low cognitive load. It's like it's just easy to run.

A

K3S, you don't have to think a lot about it and that's one of the reasons why I think, while it's caught on so so quickly, um k3s is designed for production. This is a very important thing. It's like what sets it apart from a lot of other, like simple small kubernetes projects, um this from the very beginning, was always targeted for production. This is not a toy, uh it is a fun toy, but but it's but it is it is it is designed for production. Like it's secure.

A

uh You know we do patch, like um we track cves and security issues, and you know we're serious about all this, so we have like real customers in and um actually like, very important, sensitive workloads or whatever uh running. Can you know they're running k3s so because of that it's fully cncf certified. We pass the conformance test. We do that all of our releases, like every the major minor releases, they're, always certified and past the conformance test for everything it's everything's secure by default. You know as best as we can.

A

I think we do a pretty good job there. If there's any issues, there's you know security, disclosures and stuff like that to you know we'll fix that it's important to us, but we do our best to like you know it's like all. The tls is proper set properly set up by default. um You know like at rest encryption. I think we even do that by default and things like that, so in general, we just try to follow like best practices.

A

uh You know so again, that's kind of hard like the low cognitive load. It's like it should just kind of work.

A

What we think is the best approach for lightweight kubernetes so um because we kind of narrow in on a certain use case, we can kind of make a lot of choices for the user ahead of time of like yeah, you kind of wanted to work this way, and so by default. It does kind of the right thing for a lot of uh you know deployments in this. In the realm of where k theory sits.

A

uh The last thing is, it is a kubernetes distribution. That's like fundamentally what it is um so k3s has no goal of like becoming something else like larger, or it's like. It's like. The intention is just to give you a kubernetes distribution, it's just very geared towards like lightweight uh simple low upper low operational overhead things like that, so it is very opinionated. We do things a certain way um which, which is great like if you don't, like the opinion, there's plenty other distributions out there.

A

In fact, like rancher labs company, you know uh my company remember, we have two more distributions. You can pick up if you want uh it is considered like it's like complete, like the idea is like when you turn it on. You get pretty much a fully functioning cluster, so it has like all the network network policy. Ingress storage, like everything just kind of like all the kubernetes apis, just work out of the box, um again kind of optimized for our use case and we're just very, very focused on simplicity, um okay.

A

So these are some random points because, as we talk about different use cases, I want to call out, like uh you know, it's like I'm just going to talk about a couple use cases that that where k3s is kind of shined or whatever, but but there's a lot of other things like random things that you can be doing and so and the reason why some of these are some little things that we've done with k3s of why it kind of like to surgery sparks people's imagination, of like solutions they could solve with k3s.

A

So one of the things is like it's just. It's really really really easy to run. um It's just a single binary. You just copy it onto a server. We even have a little curl script. They'll set up a system d unit for you, but like at the end of the day. It's really just a binary. It's a single file. It's like 50 60 megabytes, something like that um and then you run k3s server and that's a fully functioning single node cluster.

A

And then, if you want a second one, you just run k3s agent on another machine and you use a token as kind of a shared secret between the two to set up the the certificates and all that um and then that's like that's basically it. um So it's just super super easy to run and like um because of that it kind of like removes that barrier of it's like well. It would be really cool if we had kubernetes, but it's just too much effort. It's like well, no like for these use cases.

A

You can have it it's really simple. Okay, so not only is it really simple to install and run, it's really easy to bundle applications with it. So there's a couple. Little features that we added to k3s that just make it just that much easier. So there's a special manifest directory um that if you put anything in that directory, it just gets loaded. So if you're, you know more kubernetes, savvy or expert whatever you might know about products pod static, pod, manifest folder for the cubelet, so it works very similar to that.

A

Where, like the cube, would look for pod manifest in a directory and just automatically start them, so this is like anything you put into that directory, we're going to load it into kubernetes. So that means that if you want to package your application, like, let's say like you- want to build like an image, you know because you're going to put this on a device or something like that.

A

You just have to put some manifest in the directory and then put the k3s, but I need there and then on startup it'll, just deploy the application for you like so like no much more automation and stuff so very closely related to manifest is also images so um a lot of times people you know air gap or whatnot. um So you can then put images like any oci or docker image tarball. So like there's an oci format in a docker format, but we'll take either format. You just put the tarball into that directory on startup.

A

We'll then just uh pick like just uh we'll load, those basically into container d, so those uh well, let's take a drink.

A

So, with those two things like, you can kind of just package up an application really easy for like fully air-gapped and everything um there's also little hooks in the image structure of how to pre-pull things on start. So I think they're not pulled dynamically. I didn't want to call it just in general, like k3s, does not really try to address a lot of the.

A

um When you talk about edge and stuff, we don't try to address uh the distribution of images that much kind of look at that as a slightly different problem. um So if there's gonna be any questions about like how do you optimize image, delivery and stuff like that, k3s is not really focused in that area right now, but you know just a random thing there. So the last thing is we have a built-in type in k3s for a helm chart. So we're like. We have a built-in controller like a helm controller.

A

It's like it's a separate project from from rancher, it's a very simple helm, controller, very similar to like, um let's say uh the one from weaveworks, which is actually more popular within the community than than ours. But this one is very simple that we get embedded into k3s and this one um so it's like, since we can deploy any manifest and then your manifest describes a home chart.

A

It's really simple, to get helm charts into there too, so you don't have to run like tiller if you're doing helm2, which hopefully nobody's doing at this point anymore um or or you can uh uh yeah. So you don't have to be running help like helm, uh this command line, and you know doing all that setting up repo and bowling and or updating and all that crap okay. So other thing about this is um basically it's like.

A

When you look at when you look at kubernetes like really your biggest operational overhead, actually is etcd uh and it's it's not like, there's anything wrong with ncd. It's just the idea of running a persistent system and a quorum based, persistent system like that. You know you have to worry about losing quorum and stuff, like that. So k3s introduces other options for the data store because we recognize, like that's the biggest operational issue or headache of running k3s is running fct, so we support sqlite, embedded, mysql and postgrads for an external database.

A

Sql server and oracle are coming soon, like those, I think, there's pr's out there for those, um maybe by the time you hear this they've been merged um but like you can just get a managed db, and so that's like one of the biggest reasons people think like. Oh, why is like my sequel better than etcd, and it's really not that it's like it's not comparing like that's actually like apples and oranges, they're, two different things.

A

The whole point of like why you might want to go with like mysql or postgres is you can very easily just buy it and not have to worry about it, so it like removes almost all the operational concerns of running a cluster. um You know, given that we also handle the security all that stuff, and so this in general, is enabled by this component. It's actually a separate component called kine and you can use this actually with any kubernetes cluster.

A

It doesn't have to be k3s like you're doing, cube adm, you could just add uh etsy, like you could add, mysql support to a cube, adm cluster using kind um so but it's just k3s embeds that directly in, but that is like a separate project.

A

uh So, even though we support all these um kind of traditional or uh relational databases, we also still do regular td. It says kubernetes, so you can do normal lcd. You can do your traditional external setup like that's, not fun, um and that kind of stinks to run on the edge. um Then there's another option- and this is one we before had a solution based on dq light, ran into some little issues with that, mostly I mean there's nothing really wrong with dq light.

A

It's just more of this made a lot more sense to to go to the ncd route, and so the last uh option we have here is running fcd embedded within side of k3s, so you can just run like three k3s servers and they just join the ncd cluster and k3s kind of manages it itself. uh So it's more like docker swarm if anyone's run that uh it's a very, very simple setup that we just put on to master in in k3s in the middle of let's see july, and it should be like ga.

A

You know sometime this year, probably like october time frame. Okay, okay, so let's get into edge edge is like the most popular use case. um I guess it's kind of, oh! Actually, it's not really the most popular I'd say it's like the most popular one that like people kind of uh associate with k3s. But but in fact I think actually the most popular thing for k3s is probably um devon test environments.

A

But when I talk about edge, you know what is it that I mean um it's really. uh What I mean is is not like iot devices, and these really really tiny devices like that's too much right now, um so we're looking at fairly powerful small devices, so uh minimum one gigabyte of memory. So this is kind of like in your raspberry pi realm uh going up. You know, uh like some really great devices is like um the the oh crap. What is it the jetson nano, the nvidia one?

A

um It's really powerful small device has gpu built in, like that's a perfect use case for k3s, um so we're looking at smaller devices but fairly, it's still fairly powerful. um Like we can't go super super small, um you know I can kind of tease. We have another project, we're going to start.

A

That's like if k, if it would kind of be like the k1s like we have an even like lighter version, that's going to target more, like 256, megabyte um and really really uh less cpu and stuff, um but I was going to tease that okay, so the type of use cases we're talking about is like retail branch factory planes, trains, automobiles, wind turbines, set top boxes, drive-through menus your refrigerator, most likely your coffee maker.

A

It's like these arm, computers or even intel computers are in so many places um and yeah I mean over the last yeah you know year or so, since k3 has been out and just kind of talked to tons and tons of companies they're just looking um at modernizing all these these environments, and uh so it's just I don't know it seems like there's almost I don't.

A

I don't know if this is a real thing, but it but, but it sounds like there's, there's kind of more computers out there that are not cloud as much as like. My history and past has been focused so much on cloud. um There's so much out there. That's not cloud. It's really interesting, but basically like. Why are people doing this?

A

um One of the one of the things like the really driving factors is that what people are deploying on the edge is getting more complicated, like the applications are more sophisticated, they're leveraging more uh like open source.

A

You know, they're doing data analytics using like you know, um different data platforms or tensorflow or, and so like they're, starting like the software they're starting to have to deploy, is getting a lot harder like um so they kind of have to up their skills in how they manage software at the edge if they want to put all this compute out there, and so kubernetes just brings a nice consistency and approach that you know is already proven in cloud at a data center. So it's like how do we most people were just looking?

A

How do I get like that efficiency and that support simplicity, I guess of deploying to cloud and data center with that kubernetes brings like how do I do that same thing on the edge, and so that's really that's really what it is. It's just making it a lot easier to manage this complex software as it's kind of working. You know it's just kind of grown.

A

um I think one of the interesting is like the wind turbine use case like we work with one of the largest wind turbine manufacturers and they actually run like a little data center in those things. Well, those things aren't very small. Those turbines are huge, but it's like the amount of computing powers out. There is kind of crazy okay.

A

So when we talk about these use cases a lot of times, mostly, what we're dealing with is uh like one two or three nodes, um so they're kind of separating like the one node clusters, will be an embedded sql light. The two node clusters, we'll be using an external data, store we're looking at doing an active, passive failover type solution and then the three node one is the embedded lcd, but that one is like still experimental.

A

um A lot of people right now are getting by with really um if they want three nodes, they do a single master with with uh two or three uh worker nodes. um Until we get like the full, the full embedded scd working like you know seamlessly, uh but just in general, like these solutions, just have lots and lots and lots of clusters, and so there's a real fundamental difference between running like one cluster, with tons of nodes and a lot of little clusters.

A

It completely changes your perspective on how you deal with it, and what we found is a lot of people kind of resonates better to do the a lot of small clusters, because it looks a lot more like a configuration management problem than kind of like an abstraction federation type problem, um and so, like the approach that we've seen is. Is it maps really well um well I'll get into this? It maps really well to get up.

A

So I'm talking about just a second um so in terms of edge, basically don't go overboard, don't use like tons of features of kubernetes at first like uh oops. Let me go through these things of like basically, the it's kind of like kubernetes, give you this great abstraction layer over the underlying resources, so that, like it's scalable and pluggable and all this stuff, but when you start getting into edge, it's like you know exactly what the resources you have.

A

You have three servers they're, this big. They have this much cpu, I'm running these applications, it's fairly static. In terms of what the workload is, I mean you want to be dynamic and updating it, but it's kind of static in what you put there. So the way you plan these deployments is often very static. It's like you, don't need, like you, can just use host path for persistence. That makes it really nice because it effectively makes kubernetes stateless.

A

um If you can make kubernetes stateless since then you wipe out wipe out kubernetes and just start it again and it redeploys the manifest and it comes back up and everything's happy. Then that's a really simple thing to operationalize, so you plan the deployment slightly different to like exploit the underlying hardware, because you know exactly what it is and also a lot of times. You're you care, because there's gpus and and hardware devices that you need to talk to. So you don't like run this kind of perfectly abstracted.

A

uh You know pattern or something that uh you know kubernetes would would uh kind of you normally do like in the cloud. So that's where you see like um doing very explicit, node, scheduling or or even just daemon sets. Daemon sets work really well a lot of times, because it's just like we'll run the same thing on all three nodes. I only have three nodes and I want the same thing on all of them. Let's just run a damon set or state or a staples head. It works fine too.

A

So in terms of management, um we kind of say like just you know, first start with what you have like: don't go like crazy greenfield and build all this stuff, it's like, um because when you look at like what k3s like, how do you manage it? It's really like. I gotta get k3s on there, which is binary. K3S is a config file to configure k3s itself, and then how am I going to deploy my applications which are like manifest and images?

A

So when you combine those it's like really, it's like it could be as simple as rsync to manage these simple. These. These systems, like just copy and files and like it just does the right thing, um so you don't have to go super complicated at first, but the problem is like as soon as you're you're you're the number of nodes or clusters. You have scales up you're going to want to centralize things, and so this is where git ops starts to map very well to this, like a git ops model of managing groups of clusters.

A

The only problem with git ops right now is that most github solutions do not scale well for this edge use case because they have a push or pull. The push has a problem which is fundamentally not scalable, because you can't it's very hard to like push out to. You know uh thousands of clusters from a single point and then deal also with all the networking issues of doing a push model.

A

Secondly, uh there's the pull model, but the problem with the pull model.

A

Is you don't want to be doing the git clone on the edge then you're just transferring all this ridiculous data over and then you have to optimize your git clones um so anyways we have a separate project, it's not related to k3s fleet that we announced earlier this year and it will go like kind of ga towards the end of the gear, another open source project, that's kind of adjusting some of these issues, but just pointing out some of these things, like the model of git ops maps really well.

A

But there are some problems with it. Other little challenges and gaps that you're going to run into is um almost kubernetes tools. Don't uh they don't scale uh for edge. So it's like the whole reason why you want to use kubernetes is because it you get all like kubernetes stuff, but they don't like. Let's say like prometheus or monitoring those type things they don't work really well for edge deployments.

A

A lot of things won't deal with bad network connections and then fundamentally, you solve like a hardware and os management problem.

A

um So so these are things. These are kind of things that, like we're, we're looking to address within the maybe in the k3s project or or layers above it like. It is important for us to kind of keep k3s focused and simple, but um so so some of these management problems- you know, probably we have other like fleet. You know things like that. Other projects that, like a rancher, that we're working on um all open source um to address some of these issues- okay, so I'm really bad at time management.

A

So I went through like pretty much all the edge and I'm pretty much out of time, I'm going to talk about what app clusters are, and then I probably won't talk about pipeline that much so you can download the slides or ask me questions we're going to open this up for questions.

A

There's a second, but let me just throw out app clusters, because it's a really cool idea and it's so simple to run too and it's basically the whole idea is like you just have a cluster which is dedicated to one application and it and you deploy it on the resources needed for just that, one application.

A

It could be three nodes: it could be one node, it could be a 5 000 nodes, it's kind of funny because k3s, while we target very small workloads or clusters and in fact we'll scale to 5000 nodes, just fine we've tested this, it works. It works very well. um So, but the whole idea is you: you create a cluster just for an application um and you don't really treat the control plane as a separate thing.

A

You just kind of embed it within the resources that are using it for the application and then use something like a managed database to store the state. So your deployment, you can have these very complicated applications which run on top of kubernetes and do fancy kubernetes things. But the underlying cluster deployment is just brain dead, simple! You spin up rds you and then you, uh then you just point all the the k3s nodes to rds.

A

um So it's really simple to automate and it creates a very simple, reliable, reliable cluster, um and so I just want to throw that out through this idea, like I don't have time to go in uh in depth. But um please, if you have some questions, uh uh you know ask them uh so I'm not gonna go over a pipeline, unfortunately yeah just bad time management. Here I I mean I told you, I wasn't gonna do any edits or anything so kind of screwed that up but anyways um yeah, so I'm gonna.

A

I think I will open it up for questions. I think that's what we're doing now and that's it all right. Thanks! That's fun! Bye.