Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2019 (San Diego), 22 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: K3s Under the Hood: Building a Product-grade Lightweight Kubernetes Distro - Darren Shepherd

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

K3s Under the Hood: Building a Product-grade Lightweight Kubernetes Distro - Darren Shepherd, Rancher Labs

K3s is a lightweight distribution of Kubernetes originally designed for the edge. Due to its size, simplicity, and fast spin up, thousands of users are currently running k3s in various use cases besides the edge. Darren Shepherd will cover in depth how k3s is built, what changes are made, and how the benefits of k3s are achieved. The technical details about how k3s is packaged as a single binary, how sqlite was added as a data source, how certs are managed, how HA is achieved, how agent tunneling works and much more. Finally, Darren will discuss how these changes are made while still being a fully certified CNCF Kubernetes distribution.

https://sched.co/Uaa0

A

Hello, everyone we're gonna, get started: I'm Karen I'm the track host for this session. Just a few reminders we are gonna, do Q&A at the end. If you have a question, raise your hand, I will run up with the microphone and then please do help us out by filling out the survey at the end of the session on sketch. We do value the feedback and with that let's get started. Welcome to k3s under the hood building a production grade light week who run a nice dish row with Darren Shepherd from Rancher Labs.

A

Let's give him a warm welcome.

B

All right, yeah, thanks for everyone for coming out, surprised to see how many people came from being such a late session in the conference. So it's pretty cool. Thank you. Thank you for coming out. So just a little bit about me feel anything about me, so I'm, chief architect and co-founder branch or labs I'm, best, probably known by I, built the cloud on github and and Twitter I just complained about things on Twitter.

B

That's, basically, all I do so I created, k3s and then couple other projects of rancher and basically I've been in this space for, like 15 plus years before doing container orchestration, and this type of stuff I was doing I as a cloud stack, OpenStack type stuff, so kind of been doing this for a while I really enjoy the space, loving the structure, infrastructure level. Frustration, so just a little bit about me. So so fundamentally, I just want to start off the kind of high-level about what is k3.

B

Yes, so first it you know what we call. It is lightweight kubernetes or lightweight kubernetes for the edge. At this point, the use cases have extended well beyond what people are just doing an edge, but it does work perfectly, it's very good for the edge. So basically it's lightweight kubernetes and what that means is it's very small. It's just a single binary. It's about a 50 megabytes!

B

First, really so it's 40 megabytes, it's grown a little bit, I! Think it's it like 48 right now, but but so it's about 50 megabytes about download memory. Size of you know. Our single node configuration is about 300 megabytes. So this this fits that's for running the full server. You know the control plane bless, the the everything you need to you know run the containers.

B

So it's about 300 megabytes, if you're just doing a just the worker nodes, it's less than that, it's more like around 150 or something like that, it's small enough that it runs on a PI zero. So somebody was showing me that home automation system running a bunch of PI's IRA's, and so basically one of the things about k3s is just like I'm putting a low cognitive load. It's just it's really easy to use and that's what a lot of people.

B

The reason why they'd like and I, think that's why it's kind of taken off is it's just really easy to get up and running, so it is perfect for the edge. But it's beyond like we.

B

We decided at Rancher, Labs I'll, talk more about the history of this project, but we decided to invest in it because there was an obvious business opportunity for the edge, but but people are using it in just so many different ways, including cloud which we never really planned for, but so it's just used in a lot of different ways that you just we never really imagined one of the things that sets k3s apart from other kind of, like small distros, let's say like mini cube or micro, kate's or I.

B

Don't like there's tiny of any small word plus Kate's. You can probably find a project for it. It's like this was very much from the beginning oriented towards production. It was not intended to be a developer's tool like or not like a toy like it doesn't work great for development, but it was. It was always we wanted something that was production grade so like on day one. When we announced this, you know it's fully CN CF certified everything's secured by default. We really try to enforce the best practices we don't want you to have to.

B

You know like everything should be secure with certificates and passwords and everything we really don't want. You have to worry about that and kind of the last thing about it is it's fundamentally.

B

It's a kubernetes distribution and I really do feel like we're kind of leading the charge on what a distribution of kubernetes is is because, if you kind of look back like a history of like let's say, like Linux distributions so like way way back, when so I mean I'm, a longtime Linux user been running the Linux, my desktop 420, you know 20, plus years or whatever. So it's like the very beginning of Linux, it was like just trying to get the thing to install. You know compile it and get it to run like this.

B

What distribution was just helping you just kind of run the thing, and then they came up with package managers, because that was a necessary technology of how do we get all the software on there and so in the kubernetes world. Most distributions right now, they're, not necessarily a package manager but they're pretty much a instead of a package manager. The the corresponding technology would be like it's a deployment tool. So most distributions today are in fact really what they're doing is they're just helping you deploy kubernetes.

B

But what we've done with k3s is it's like? Okay? Well, there's a million different ways to get kubernetes. Let's actually tailor it for a specific use case, and that was like lightweight simple. So it's like when you look at the evolution of Linux distributions. It's like they all started to take on personalities of. Like you know, Debian was huge. Open-Source community a bunch is really easy to use like gentoos for personal space heaters, so it's like so like with k3s. It's like you know, that's what we kind of feel it's like. Okay!

B

Well, there's a lot of different ways to get kubernetes, but we're gonna put together a package of kubernetes in a very specific way and it's kind of tailored to how we think it should be done. And if you don't like that, then that's fine, like that's cool, just go use a different distribution, there's plenty of them.

B

Okay, so um so just kind of like short history of k3s, so basically I mean the project really is just complete. Fluke like it was never intended like it was so basically this came from. We have a separate project called Rio that we also announced that cube con from beta yeah. So um so, when I was developing Rio like I, the idea of Rio was just trying to make like kubernetes easier. I mean there's a lot.

B

I could give a whole talk on Rio but like, but when I was developing in Rio I'm like hey what if I just bundle kate's inside so it was like, you don't have to install kubernetes. It's like you just get Cooper days with Rio and then so Rio was this like full package that includes kubernetes, and then we realized like hey. This is like actually the easiest way for us to run kubernetes.

B

So then we ripped out the kubernetes portion of ember like just made that as a standalone project, but this was just kind of like my own pet project and then somehow it kind picked up, because everything I do is open source. You can see it like everything from like a company perspective and also personal, is all open sourced up and github and- and you know, people kind of tend to watch what I'm doing on github, and so some people kind of saw this.

B

This project that I was working on and I got a little bit of attention and it was like. Oh well, you know. Maybe this is something we should invest in and then that when we finally announced like decided, you know see if there's like a really a business opportunity, edge turned out to be like.

B

Oh, this is really good for edge, so we decided to put you know, effort into a company wise and we announced in February of this year and then it just kind of blew up okay, so just kind of like some vanity metrics. So it's like the so basically released the project in February. So that was like nine months ago. I did my math right, maybe ten and then so we just g84 cute for for cube con.

B

So so those again it's like we announced that we really didn't expect the the amount of attention we would get for it, and so, as once we we announced them. We really had to double down on the amount of effort we put into it to make it what we really thought was production grade, and that's a lot of where this talk is like this talk will go I'm, trying to go into really the guts and the details of what we do have done in k3s.

B

So you really understand what it is, but within that time, like we've, gotten 10,000 stars and I'm really excited because we got 10,000 stars about 10 minutes ago. So I've been really liking, Twitter to ask you if you like, just you, know yeah so so this diagram on this right here is actually the blue line is rancher. That's like our core product that we built their business off of and then k3s is a thing on the side. So it's like. Basically within the like, you know, a nine-month timeframe.

B

We've gotten 10,000 stars was which basically took us about five years for our main product and our main product. Just like you know what drives our own business or whatever. So it's pretty exciting for me to like see it, take off that quick I really really actually want to see k3s cross, rancher and I really think it will hopefully over the next year, they'll cross. So we got more popular than our management product, so you have some other stats. It's been downloaded.

B

You know five hundred thousand times polled image polls like seven hundred thousand we're really bad at tracking metrics of like who's actually using a live clusters.

B

We're really wish we had better metrics on this, but, like we guess, like around 8,000 nodes have been launched from just kind of inferring from each polls, so so basically I mean it just seems to be taking off, which is really cool, so um so, as I get in and talked about kind of, like the architecture and and everything we kind of we separate k3s like the command line and a two things. You have server an agent and server.

B

What that means is that's running the control plane like the API server, the controller manager and scheduler and it's run, is also running like a master worker which is a q2 cubelet. So then, when we talk about agent, the agent is just pretty much like the cubelet. I mean queue proxies in there and some other things, but like so that's kind of when I show these diagram. It's kind of how we separate we that's just the nomenclature, we've used as like server an agent.

B

The reason why we called it that ways we just kind of seemed it was easy for people to conceptualize. It's like client, multiple servers, join agents and whatever.

B

Okay, so what does it look like the actual, like processes like? What are you running? It's like? How do we make it so small, a lot of it has to do it. Look why the memory footprint is so much smaller.

B

It has to do with how we combine processes and how we packaged it so on the left is the server and on the right is on the agent if you're running this on, so the agent would be by itself if you're running the server in the agent to get which that's that's kind of the default mode of when you run a server, it actually has an agent running to it's. Just the agent is embedded inside the server process, so on a server.

B

These two boxes on the left, like the Box on the Left suboxone right, are actually combined into one process, an agent just as the agent portion, so in the server. Basically, we combine the API server, the schedule or the controller manager.

B

We have this tunnel proxy thing running to I'll, talk about it more in detail what that is, and then we also have some other controllers and some little things running some arbitrary to just do some arbitrary things like DNS management from the nodes, and we do some tricks to make the nodes you know not require like DNS names, because right now, like kubernetes kind of the default behavior is every like.

B

The node name is actually a fully qualified name that you can look up, and so we do some little tricks like in controllers and stuff to work on that, and it's also embedded in there is sequel light too. So there's more options around storage, it's not just sequel light and I'll to get more into that that later too, but it's it is actually one of the large things that makes it so much smaller and lighter weight is the default mode. Is we just? We run in sequel, I?

B

So again, if you're caring about like smaller, you know like an edge use case or like a single node deployment, I don't need like a highly available at CD, but I'll talk about how you achieve a che coming, slides, okay, so on the agent sides. So, basically in when one process, we basically we've included both the queue proxy and and the cubelet and then the tunnel proxy and then flannel is actually embedded into the same process and then, as a child of that runs container, do so.

B

By default we run container D, that's kind of like our preferred runtime, but it is completely swappable. You can run any c RI, but by default is it is container D. So we actually found like that saves.

B

A significant amount of memory is actually just combining a lot of these processes, because these process basically has like the full go SDK and it like takes up more more memory and so we're just if you're looking at like the initial memory footprint for these smaller environments, those things kind of matter, just every go process is gonna like by default, start around like 40, megabytes of memory or 20 bugging you so I'm. Just combining them actually did quite a bit okay.

B

So when we get into kind of like deployment architecture, just like house k3s clusters, actually you know what do they look like? So the obvious simplest one is just a single node, so this is you know this is kind of an edge optimized case, so people are using k3s in a lot of places, just running single node clusters. There's a lot of applications for this of just I, have one computer for four, whatever and I'm putting kubernetes on there. So that's kind of a simplest thing.

B

Okay, so then we have basically like the multi node non H, a cluster. You have one server, multiple agents, you know so that's kind of a pretty straightforward, so we also support running with an external database.

B

So you have a minimum of two servers in this situation. You don't need to quorum, so you just need two servers and they can have as many agents as you want and then a database, and so what that database is it's either gonna be my sequel, Postgres or at CD.

B

You know I'll talk more. We accomplish that. So then there's this other mode of basically running. This is like full, embedded DB cluster. This is again like an edge optimized use case of like I. Just have three servers running on the edge I want highly available and I also don't want to run like an external database. So in this mode we're running DQ, Lite embedded into the processes. So it's it's kind of it's. It's using all the sequel. Lite work we did, but then we did DQ light.

B

Dq Lite is a really cool project that came from canonical, which is basically they added raft to sequel like they use raft to replicate the sequel, the sequel, Lite right, append log. So they think that the you know like the kind of the change data capture like the log, that's coming out of out of sequel, Lite and then replicated across all those servers, and so we embed that directly.

B

So you can just launch three three servers through servers point one together and then in this situation, since it is raft you need a minimum of three of you need a quorum, and so in this situation you know you can lose one server. If you want to be able to lose two, you need to have a cluster of five. You know just your standard, clustering things, so it's kind of like well. How do you set these things up? So this is why people like to k3i.

B

So much is like it's just ridiculously simple of like how do you run a server? Well, you just run a server and you can put a token in there. What that token means is just some arbitrary long, random string string that you're gonna use the shared secret among the cluster, so people can join so you're like you launch the server. If you want to join an agent, you just point to the server you give it to the token it joins. You now have an agent.

B

If you want to join a server you can you just point it to the server. This is for the full eh-eh mode. If it's, if you're, not I'm, sorry, this is for the full embedded, a che mode. You just point it to another server: it joins the cluster, then you know, you've got a three node cluster or whatever.

B

If you use an external database instead of doing the s and pointing to the server URL, you would point it to the you would give it the database credentials in URL, so they could talk to them to the database, but I mean it's like. This is like one of the few talks that I have not that I've done. That does not have a demo. It just didn't really like the demo is really just like you run that command, and now you have a kubernetes. So it's like. That's it it's a couple seconds. So, okay, alright!

B

So this is one really we start getting to the guts I'm like. Basically, what I want to talk about is like. Basically, what are we modifying kubernetes to make this happen? You know what does that mean? Like you know, you know: how do we maintain this going forward and all this stuff, basically where's all the stuff that we've developed and added on?

B

And then what is additionally package like what are the things for, including in k3s, because a lot of times people get just asked me I was like well how's it better than just cube a DM QA vm's, not really that hard to run. You know once you learn it, you can run it. You can get a cluster up pretty quickly or like what about like kind or something. You can get a cluster up pretty quickly and it's like well. The intention of this talk was basically to show it's like.

B

Well, we've done a significant amount of work beyond kind of like what you can just get out of the basic cube Adium. Although a lot of some of the stuff that we've done it does overlap with the way QP, ATM works and and I would like actually like more synergy there. It's either like we were just naive and didn't realize like we were duplicating an effort, or we have like a legitimate reasons of why we did it different.

B

So basically like what are the kubernetes modifications? This is like the most important things. It's like it isn't. This is not a fork of kubernetes. It's like and I kind of understand why some people might think k3s was like weave for kubernetes, especially if you watch the history of it, because when I very first announced it out, there I think I called it like kubernetes without the stuff I, don't care about who is I or like I, don't like or whatever I just I, just ripped out.

B

Everything I didn't want out of kubernetes and just made it work for myself, but that you know that that's not maintainable or really makes a lot of sense, but so we've really put so it's like once we announced kubernetes when we start when we announced k3s in february. So at that time there was actually over three million lines of code that we changed in kubernetes like this was largely three million lines of stuff we deleted.

B

That was like I would still want that, but like it took off so quick and people wanted to use in all these different ways that we never imagined we're like okay. Well, we really just need to have everything back like we need to like it means yeah. So so we put a lot of effort of like one is like.

B

Can we add it back and keep it as still simple and small, like lightweight as what we had before and then also the things we do really need to change it, like specifically like sequel light, for example? How can we do that in a way that we can actually maintain? This going forward is something that we can possibly upstream or create a different component or whatever.

B

So that was like all the effort that we spent there's two things that we spent from like in the last ten months or nine months, was like basically ensuring that it was just like rock-solid for like H a scenarios, and the second was that we can maintain this going forward so that as there's new kubernetes releases, there's new CVS or whatever we can get things out and we're not maintaining some like crazy, weird version of of kubernetes, because that was never the intention it was like.

B

You have to read like the way I work is like I'm. Fundamentally just a hacker like I want to get something done so I'll find it quick crowd to get something done, but then, like, then, you realize it's like okay. Well, if I want to continue this forward, what how can I maintain it so that was kind of the initial release of k3s was like. We really want to fight a lot, but now with zero of eleven well, actually, it's one dollar I should have updated.

B

So with the wand, I know, there's really less than a thousand lines of code of kubernetes that we've changed and you know I'll go over like what what those specifically are and you can find all the patches there in ranch or kubernetes. If you look at like when you run a k3s.

B

Wow see so when you run k3s I, there's a the the version of the kubernetes that comes up for the node. That's basically the tag you'll find out the repo, and you can see all the patches, okay, I think I might have to like rush through them. So because I have till three someone just go till three: okay, all right, I wanna do some Q&A at the end, so okay! So what did we actually modify? The first thing we did was we added rootless support, like we didn't do anything it's like 100%.

B

This all goes to. Oh, wait, don't butcher this! It was Akihiro from NTT. He's done all this work and it's like it's amazing and we really care about rootless. We really wanted to work. There's a lot of use cases where it makes sense, so we've carried over those patches. So that's one thing yet so in terms of functionality that we've actually dropped, we dropped all third party entry, storage drivers, they add just a tremendous amount of bloat to kubernetes, so we dropped those. So basically we focused completely on CSI.

B

So that's like gonna be so this one in the second bowl point is like people always ask: does it work on k3s? It's like the majority of time. Yes, it will, except for these two points right now. The first is we don't do increase storage drivers, we always support CSI.

B

The second is we dropped cloud providers but they're coming back but like they like we're, just gonna do in a way that we can maintain the the small footprint, but so we dropped cloud providers and any dependencies on cloud SDKs, because again, there's just adds a ridiculous amount of bloat.

B

So those are the main modifications where, like we actually changed functionality, it behaves different than the upstream one and like so this one. We hope that, like these are both things that upstream they're doing the same thing anyway, just we jump the gun. You know so it's like upstream is getting all the cloud providers out. Moving and external the drivers are all moving to CSI. So we fully expect that we can be like just over the next year, hopefully just pull directly what's coming from Austria and not modify it.

B

So we have some back port back ported bug fixes just things that we hit or whatever, and then a lot of the changes are just mostly around allowing us to embed embed kubernetes into a single process and we're getting better at this of finding better ways that we can do this, that anything that we think really should be upstream.

B

We will upstream, but as we look at it deeper we're like you know what we can actually do it this in this different approach, and so we haven't really upstream a lot of stuff here around this, because we think we can just do in a different approach and not have to worry and not ask for changes or anything. So that's really the majority of the kubernetes modifications like it's really done like so it's less than a thousand lines. Okay. Now this is like all the things that we've developed.

B

This is where it gets into this really long list. So the first thing is: we have a reverse tunnel proxy for the cubelet, so this is like basically the way kubernetes works. We need bi-directional communication to cubelet api server talks to the cubelet cubelet talks to the api server so from a firewall in perspective that often causes issues, and so people don't want to have to worry about that that bi-directional communication and so basically we've modified. We include this tunnel proxy that, basically, all the agents make an outline.

B

They only make an outbound connection to the API server and then from the API server we tunnel through that backwards, and so we're able actually to do this all without modifying kubernetes, because we just override the dialers that are used by the the cube to cube clients. So we don't have to modify it's a kind of there's already good support for this. This is an approach we we do for Rancher for our management product, and so we just took that technology.

B

We put it in there, so this just makes people fire one just little, just the kind of little thing that makes it a little easier to run and not have to worry about those things: okay, so I'm gonna. So the one of the major changes we did is sequel Lite. So we actually pulled this out into a separate component called kind, because the chili we were modifying kubernetes, pretty heavily like we implemented different storage implementation in kubernetes and we're like well. How can we maintain this?

B

How can we get this up streamed and it was very hacky. The way that we did it, and so we kind of felt like the best approach for us to move forward was, was basically to pull all that out into a separate, separate process like project that basically just implements a net CD API and then translate that into the you know, like whatever we need for Postgres or my sequel, so there's a separate project we created called a kind that basically just exposes an MTA API.

B

Today we run that all in the same process over like a hidden UNIX socket, so you don't see it, but any standard kubernetes distribution can actually. You can just take that and instead of pointing to at CD you you take the same at CD configuration you point to kind, and then it can run on any database.

B

This is something we very very much want to just give to kubernetes and then help standardize, the storage interface of kubernetes, because we effectively just reverse engineered the behavior of how it works to be able to build this and I'll go more I'm. The other side that specifically unkind how it works so also we did DQ light integration besides just the getting kind of a kind' component, there's other things, with DQ light of how to manage the transport. How to do all the security and stuff like that. Okay,.

B

We also we, we include a full busybox user space, so that includes like IP tables. Do you find these are just utilities that that that case that kubernetes needs like user space things kubernetes days, but so all those we package in so there's actually zero dependency on anything in user space for k3s it will. Actually, if you have the mounts of like sis proc dev kts will just run. Look, we don't need anything else.

B

K3S is actually a binary it. So it's really just a self-extracting archive. When you run into the very first step, you'll see it actually creates a directory called data, and that's we then put all the these extra components like IB tables and whatnot. So it's packages as one binary. It's ran as one buyer, but in fact it's an archive that has much stuff in it. So we do a tons of stuff around certificates. That's like just a massive amount of effort of like managing the certificates and doing the rotation and giving proper.

B

You know, cns and everything to all the different components, just really a lot of work. There again, we probably overlapped a lot with what what cube ATM does, but we kind of had to do everything like super like dynamic and like no user interaction like so we just kind of automatically do all this stuff. There's this whole server bootstrap process that also, besides just generating their certificates for the server. We also have to figure out how to get the CAS and private keys from one server to another.

B

So there's a whole process of how to communicate and share those things. Okay, the more things we did, so we we do like just to make it easier to deploy applications like once it got kubernetes running well. The first thing is I got I can very easily bring up kubernetes. How do I make it easy to deploy applications like packaged it really easily.

B

So we have like a controller that basically just watches a directory that is very similar to like the pod static, manifest like you, just put a kubernetes manifest to the directory it'll automatically deploy it. So you can even what you can do is you can have a directory with a bunch of kubernetes manifests in it. You can have a directory that has like the images too and then you just run k3s and it comes up with your application, doesn't have to pull anything, it's pretty cool and then also since so much stuff uses helm.

B

We actually have a helm controller built in DK 3s, so you can just define a custom resource for helm. Server boost already talked about that client. We have a client side load balancer that ones for on the cubelet side, because the like in an H, a situation, you have a you know multiple servers, API servers. You need a load balancer between those, so we actually do a client side, load balancer, so the client side loads by its using Google TCP proxy project, but we watch the endpoints, and so we have a controller.

B

It's running on the cubelets to kind of do that, client, side, load, balancing, so you'll have to input a load balancer. In there too, we actually wrote a cloud provider because basically, the only way you can set an external IP on a node is to have a cloud provider. So we had to implement a cloud provider just to set this one field, but it's actually not that hard to implement a cloud provider. We do an embedded service load balancer, so the service load, balancer API, will still work. It just basically will do use host ports.

B

Let's say we have a local storage provider, so you have a storage class like the idea of k3s is like when you bring it up. It's like all the it like. It's an ingress storage provider network policy like all the kind of additional api, zinc and kubernetes, they all work, but all those things require additional components. Well, we made them like. We make sure it's all working just you know so that's stuff like we, we developed I'm gonna, run out of time here. I really want to open for questions.

B

I'm, gonna kind of run through this lesson. So basically, so these are all the components that we've included in k3s all again in this like little 50 megabyte. Well, actually, it's not sure if some of the images are downloaded, so it's 50 megabytes for the binary and then, if you want to run in a fully air-gap situation, we provide a tarball of the other images that tarball decompressed is 250 megabytes, so compress it. It's probably like another 50 megabytes or whatever.

B

But that's you know that's for like a full air gap, and so full air gap is supported with k3 OS. So, basically, what's included, we compile flannel in. We just basically include it in, and we have support out of the box for V excellent IP. Second wire guard wire guard does require you to bring the user space components though, but for IPSec we actually include the full strongswan. So IPSec will just work out of the box.

B

We have a network policy controller. We picked this up from cube, router I'd actually like to. We probably need to work a little closer with a cube, router project, but we're so. Basically you get a network policy controller. We have container D, run, see all the cni binaries strongswan.

B

We include traffic for ingress, core dns everyone's got that what metric server is in there by default. Now it took us a while to get like a metric server that worked on all architectures. There's one thing: I, don't think I ever really called out this everything we do runs on arm hf arm 64 and AMD 64, like we spend a huge amount of time, troubleshooting, raspberry, pi's, yeah, so yeah you can thank Eric, and so one of the lead developers on is Eric Wilson at Rancher. He spends a huge amount of time.

B

It's there's random things just always fail. I also met I mentioned the busybox stuff before oh yeah, okay, there. This all works on those architectures. So that's also things like. We make sure everything works across two architectures like that, but it's a good amount of effort. It was much harder than we thought. I, don't really have time to go through all this with like. Basically how I could basically do a whole talk on kind, but we've created this other component of how we implemented its sequel.

B

How we've implemented kind of a sequel layer, but again the idea was we just want to get people options at CD like I, got nothing against it, but there's situations where people want to run different data stores and there's a really good valid reasons for it. So we, this has just been a kind of a pet project of mine for actually probably about four years, is really understanding how I can implement. You know, what's required for kubernetes on a traditional database and if I just call it performance is perfectly fine.

B

We have no issues with the performance. K3S will scale. Actually, even though we targeted for these smaller, like use cases, k-theory scales perfectly fine to thousands of nodes, it doesn't. We have no limitation that we seem to introduce that makes it any less scalable than regular kubernetes.

B

The only thing that's different is like you can't separate the API server and the controller manager and scheduler those will run in one process, but that doesn't impact scale, but it might, you know, get back something else, but I mean in terms of like how you want to roll deployments, but we haven't really noticed any problem. So basically performance is fine.

B

We haven't had any issues, we put it under a huge amount of load and in theory, basically in theory, from what we've seen kind should outperform at CD pretty much in every way, not that it's not that a tradition of like not that the database is better than it's like it's at CD in the context of Cooper Nettie's how kubernetes uses at CD a traditional database should be able to outperform it at CD itself has some like interesting characteristics where they can do like, writes really really fast, but but in the context of kubernetes, and it changes everything, and so things have looked really good.

B

I see, okay, so basically it's like. Where are we going with this? It's I mean the easiest thing you say. Is it's a we're? Just going to expand k3 has to support more use cases because people just want it and kind of a lot of different places. We just never intended and we really want to expand like the ecosystem of also like applications and stuff that are just very easy to install in k3.

B

Yes, so there's a couple projects I just want to call it I can't I, don't know, there's time to cover them, but basically there's like a 3d which is really cool for development. If you want to use k3 s in the development environment, check out, K 3d there's a project case, would it Sketchup, but it's it's? It's spelled k3 s up, and this is for basically remotely deploying and easily setting up.

B

So if I have a you know, a cluster of raspberry PI's wherever I can just really reasonably bring them up some really cool things there. This SIBO cloud is actually doing a managed service for gay 3s, which is really cool story of like basically, you know it was just so easy for them to run K through rest, okay. Well, we just offered managed so that has been pretty cool. Yeah- and that leaves it to the end and I probably can answer like one or two questions before right at a time.

A

Maybe let's do one question and then you can do questions outside after.

B

Yeah yeah well just and Steve got a burning question. Who do you wanna pick yeah we'll go here. You get the questions.

A

Shirt one dude.

B

Supports it, mixin architectures in one cluster carries better.

A

Place to looks yeah.

B

Yeah cuz, that's just like a regular Cooper. Now these things so heterogeneous architectures and everything that works. Fine, yeah, okay! Well, so if anyone has any questions, feel free, like I, guess, I'll kind of head out to the side, and you can him up to me and ask me any questions, but thank you so much for coming.