Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Sharing Clusters: Learnings From Building a Namespace On-Demand Platform - Lukas Gentele, DevSpace

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Sharing Clusters: Learnings From Building a Namespace On-Demand Platform - Lukas Gentele, DevSpace Technologies Inc.

Multi-tenancy is a hot topic in the Kubernetes community right now. IT teams want to enable engineers to work in shared clusters and allow them provision namespaces on-demand whenever needed. This creates a plethora of challenges that cluster admins have to address. This case study will show how the team behind DevSpace Cloud built a public Kubernetes-Namespace-as-a-Service offering, including: - Authentication via Dex - Automatic RBAC configuration - Dynamic admission control via Open Policy Agent - On-Demand namespace provisioning via CRDs - Network isolation using network policies - Resource management using resource quotas and limit ranges - Inactivity detection and automated cleanup of abandoned namespaces - Sandboxing This talk is intended for IT teams that want to create internal Kubernetes offerings to allow engineering teams to provision namespaces in an on-demand fashion.

https://sched.co/ZeiV

A

Hi this is lucas and I'm welcoming you to the session sharing clusters learnings from building a namespace on demand platform.

A

As I said, my name is lucas, I'm the ceo of devspace technologies, inc and I'm currently working on loft. That is h, which is essentially a multi-tenancy uh solution for kubernetes, I'm also an open source maintainer uh for several kubernetes related projects like uh dev space and kiosk uh project, which you will hear about uh in a couple minutes.

A

I just recently relocated to san francisco yeah one of the few people that actually moves that way and not out of the bay area. um You might hear the funny accent, because I'm originally from germany so excusing for that right up front.

A

So who is this talk for this talk is essentially for it: teams, admins and sres at companies that are already managing kubernetes clusters might be for production, cicd, staging environments or whatever, and who now want to make kubernetes available for engineering teams.

A

That means you plan to host multiple engineers or engineering teams in shared development clusters. So we're talking about the pre-production stage in this talk and the challenge here is really about kubernetes multi-tenancy and getting the experience straight for your tenants in large shared kubernetes clusters.

A

When we talk about multi-tenancy well, we first have to understand what that means, so essentially, single tenant, kubernetes clusters are clusters, which you hand over to a single team which you're running single. You know, singular workloads in and multi-tenant kubernetes clusters are essentially larger and shared kubernetes clusters, which host a variety of you, know engineers, engineering teams and their workloads.

A

So once we compare those two approaches, we can very quickly understand that single tenant kubernetes clusters can be very expensive because you're creating a bunch of kubernetes clusters and someone's got to manage them. um You have to have you know you have to essentially pay the cloud computing resources for all these clusters.

A

It's really hard to scale things to really the zero, um so single tenant kubernetes clusters tend to be pretty expensive, whereas multi-tenant kubernetes clusters are cheaper but very difficult to get right and that's what this talk is about and to be more specifically, this talk is about soft multi-tenancy.

A

What does that mean so essentially, if you're, using if you're kind of looking for hard multi-tenancy uh you're looking at systems where there's zero trust between tenants?

A

um That means, for example, if you are running like a platform as a service uh offering- and you have you know, multiple users from different organizations who, who you don't even know, potentially then you're talking about a hard tenancy that you might need for your platform, we're talking about soft tendencies. So, typically, your tenants are users all from the same organization and there's a small chance of malicious actors from the outside organization outside of the organization.

A

So it's mainly really about preventing accidents and ensuring the stability of the underlying platform rather than hardening against outside attacks.

A

So let's get to the first learning um that I discovered, while implementing multi-tenant kubernetes clusters at a variety of companies over the past two or three years.

A

The first learning is essentially centralized user management and authentication.

A

You probably already have users in a third-party system like active directory, google or microsoft. If you're, using like office, 365 github gitlab for sure are systems that are aware of your engineering teams, and you can simply hook up an open source project called dex to your platform. It actually recently became a cncf sandbox project and with dex you can achieve single sign-on for kubernetes.

A

Dex is an openid connect in oauth2 provider and it supports various identity providers, including ldap saml, and obviously you know, google, microsoft, github, gitlab, etc. So it's a very powerful tool to get authentication right.

A

So, for example, you could provide a cube config. You know uh right after user authenticates uh through the text, authentication, workflow um and let the user in the simplest case just download a cube, config file right. Obviously that can get a lot more fancy. You know if you have your own cli tool, which sets uh which creates cube, config files on the user's machine uh and does the authentication workflow for the user automatically right just pop up the browser and have them redirect to a local host port that the cli opens.

A

um There are different levels uh with which you can do, but the most easiest one is just you know, redirect users to to text and let them download the cubeconfig file at the end.

A

The second learning is essentially, you want to restrict users. Obviously you want to ensure the security of your platform.

A

So you need, you know things like port security policies for sure, because you don't want any privileged parts being created out there, but at the same time, you really need to make sure that you use smart defaults for your platform because ux really matters, because, if you're providing a kubernetes offering to engineers, you really care about the adoption of your platform within the organization. You want users to actually have a pleasure in using kubernetes.

A

So let me give you an example: definitely you want to set resource quotas and you want to limit users, you know regarding cpu and memory etc.

A

But if you do that, then users must specify cpu and memory limits within you know all their parts and containers, but they might not want to do that in the early stages of development right- and I shouldn't have to worry about that too too much to be honest, because it's development stages pre-production. Obviously you want to have all these. um You know limits uh configured in production and staging for sure, but might not want to want the engineers to worry.

A

You know the first thing: they worry about shouldn't, be this so set defaults and then let kubernetes implement them um via limit range. It's a standard, kubernetes uh construct which you can use here and essentially it's a mutating admission controller, which is going to apply these defaults to all parts who do not specify cpu and memory limits test it out with various workloads of your engineers, and you know, find the smart defaults for your organization network policies. You definitely want to have in place as well, because you don't want some.

A

You know, network traffic go all across your cluster here. So probably you want to apply like the default deny all policy, but then you want to open up certain things. Definitely you want the user to be able to communicate within their namespace right. Certain ports within a namespace service should be able to interact with each other.

A

You wanna, probably for a lot of containers or a lot of namespaces. You can do this with labels, for example. Allow them to you know, connect with the internet and don't forget about dns as well. Usually you want them to connect to your cluster internal dns, but you can also you know, let them use any outside um dns service as well.

A

The third learning is essentially automate as much as possible, and it applies to everything I was just talking about and additionally to you know our back and everything related to that create templates. Essentially, for anything, you might want to use open policy agent, which is also a cncf project for dynamic admission control, because there are certain things you can't enforce with standard kubernetes rules, but opec can do them for you very easily. So, for example, host name validation for your ingress resource.

A

You don't want colliding host names and having your ingress controls are kind of, like you know, do some kind of load balancing between different uh tenant applications. You want each tenant or each namespace or whatever you define a single unit here to have a certain set of you, know, subdomains or whatever that they can use, and they should follow a certain pattern right. So you could have like a suffix or a subdomain of that includes right, the tenant's name or the namespace name, or things like that.

A

Right and oppa can really check this, and you know essentially reject anything that doesn't follow this pattern when someone's creating an ingress. It's pretty easy to implement that. um Additionally, you can use the same hostname validation for the certificate resource uh if you're using things like cert manager, uh so that you know your let's encrypt account. Doesn't you know provision a lot of names, a lot of host names which aren't actually being used in your system?

A

And then you might want to restrict certain storage and network related configurations. You might not want you know every user to provision load balancers in your aws account, you might want to. You know, set some restrictions there, because you may want to provide some certain. You know cluster wide services so that not everybody needs to install their own ingress controller search manager, etc, provide these services and automate them as much as possible. For yourself, if you're setting up new clusters, you can use terraform for that or you know just a simple.

A

You know helm chart which you have you know helm dependencies to nginx, ingress controller, insert manager, things like that and really bundle your configuration there and related to that. I would really recommend to store everything in kubernetes, so I would have no state outside of kubernetes, so no like database, where you keep track track of certain relationships, etc, and potentially you want to also store things in git, so use kubernetes related uh kubernetes native resources, like you know, annotations labels secrets.

A

All these constructs are really great to store information about owners and tenants uh what they can do within the system um and the great part about this is. You can also use getups now to kind of manage the state of your platform.

A

That means you get a free audit log via the commits you can roll back very, very easily.

A

You have an automated approval process right, that's really great! So let's say you set up a very easy system for provisioning namespaces and the easiest system is to have something like flux. You know running in your kubernetes cluster and it essentially pulls the state of your git repository and applies it to your kubernetes clusters.

A

That means you could, for example, tell the user hey if you need a new namespace, create a pull request and we're going to approve that for you, and you know certain limits for your um for this namespace or whatever could be, you know, annotated or they could be labeled, so that could be in a config map that the user also needs to add to this pull request, and then you can use all git based features for a very easy approval process, with all the power that you know, gitlab, github, etc, provide for you.

A

This is great, for uh you know, keeping track of your platform state, but it's not so great to keep track of your user workloads right. I would argue that engineering teams should have direct access to kubernetes rather than having to go through. You know, ci cd, um you know github system or whatever to deploy something to kubernetes. If they see some tutorial out there to install a hand chart that they might need for their. You know day-to-day work and want to try it out right.

A

They should be able to follow that tutorial and not have to you know, add that to a different workflow, etc, which makes it just more complicated, so keep your platform state and getups, but that doesn't necessarily mean you also need to keep your user workflows and and everything that's related. uh You know to their work into uh into git as well, and if you want even more control uh over what's happening in your cluster, you can always write a controller, uh introduce crds custom, resource definitions and kind of extend, kubernetes tell kubernetes.

A

You know this is a tenant for us, for example right and those are the tenant limits uh that could be objects in your platform that you're creating and if you're going extra fancy, you can even write an api server extension to um solve the list problem. The list problem essentially means you can set r back to get users to perform cube ctl get on their namespaces right, but they have to provide the name of the namespace. If they want to run cubectl list namespaces, it's either all or nothing right.

A

They can either get access to all namespaces and list all of them, or none of them right. You can solve that by an api, server extension and essentially filter the namespaces that the tenant has and one project. I was mentioning earlier that I've been working on for a couple months. Now I think we open sourced it in february uh is kiosk. Kios is an extension for kubernetes, uh provides an api server extension and several crds for multi-tenant kubernetes clusters.

A

If you go to kiosk.sh you'll be redirected to the github repository and the readme contains a lot of you know, walkthroughs and explanations about how to configure keyers and how to set it up in your cluster is essentially as easy as installing help chart, and then you have like a tiny controller running in your system uh as well as this api server extension, and then you have resources available that users can work with or that you can use to start building a multi-tenant kubernetes platform.

A

So one of these resources that solves the list problem is space. For example, right space is entirely virtual. It's not a crd! It's not persisted in that way.

A

It's an extension of the kubernetes api server and if the user runs cubectl getspaces right, they essentially only get those names list spaces. Sorry, they essentially only see those namespaces that actually have access to vr back right so where they essentially have the permission to read, pods or whatever you configure in kiosk. um Here's also those templates.

A

You can use those templates essentially to tell kids hey uh when someone when one of our tenants creates a space cube ctl, you know, create space essentially um apply certain templates to it, and that can be. You know, outbreak rules, network policies, limit ranges whatever you may need for your multi-tenant kubernetes cluster is great because kiosk does it automatically for you.

A

Obviously, if you're going with the git ups approach, you might not want to have templates instead, rather force the user to include that in the pull request, but either way it's going to achieve the same state and then there's essentially the definition of a tenant which key is called account and you can set certain account quotas, which essentially limit an account.

A

And if you want to learn more about kiosk uh reach out to this guy fabian is a kiosk maintainer and really the brain. Behind all of this, um we launched the project together. But if you really want to, you know, dig into the details, you should definitely talk to him and uh if we're looking at the workflow in kiosks, there's essentially two persona right.

A

There's uh you as a cluster admin uh and then there's the account users, which can be a regular kubernetes user or a kubernetes group or a service account right, um and you essentially define and manage accounts and account quotas, as well as the templates right and then users uh use their account to create spaces and when they create a space, uh they essentially, you know, define in the spec um that it belongs to this account and then, when they do, that the admission controller will essentially check you know: hey the api. Server extension.

A

Sorry will actually check. um You know. Is this subject or service account? Who is, however, doing the cube ctl request? uh Who is it? Is that person actually that user? Actually part of this account? Will I allow this right and if the space is being created, it will create the underlying namespace and force the templates that you and have enforced for this account right.

A

So now we got like kind of the basic setup regarding authentication. We automated a lot of things. We restricted our users um and we might even use kiosk. You know kind of to get this. You know kind of structure for for our multi-tenant kubernetes platform. The next part is really about um how users interact with the system. I would argue, do not hide kubernetes, but try to make it as easy as possible for users to use so engineers actually need access to kubernetes, so they need to run cube.

A

Ctl commands et cetera because they want to you know, verify new features very very fast. I don't want to have to go through the full ci cd cycle. Just you know when they see something that they might need, some chart some hand chart they want to install um or if they want to just test a new feature very quickly.

A

They want to do those things pre-commit right and they want to debug the container startup inspect the state of the applications attach debuggers and trace requests between several microsofts that they might be working on, but I would also argue cube.

A

Ctl is not it's just an api client, it's not really a developer tool right, so you should tell your engineers uh hey, use, cube ctl, you have the full power, you can also use helm or any other kubernetes tooling, um and leave up the workflow and tooling to them, but I would definitely recommend, as you're more of a kubernetes expert than most of your engineers will be to raise and really awareness for open source tools which are which are out there.

A

You can vet them with your engineers and essentially you know, give them a list and say hey here. I found all these tools that that work really great. um I would recommend you take a look at them.

A

I'm just going to name a couple down here: scaffold desk-based tilt, telepresence um they're all pretty similar, but there are two distinct differences. So telepresence is the bit older tool which, essentially you know assumes you want to work on a couple of microservices.

A

The one that you actually want to work on should run on your local machine. Everything else is being deployed to your kubernetes cluster and what they do then, is they start like a two-way proxy between the local cli and a telepresence part that runs inside your cluster, then you can, with the cli, also mount.

A

You know this network, this network proxy, and you know volumes even from the cluster and things like that, um so that your local service uh kind of feels like it's running inside kubernetes, although it's running on your local machine, um that has certain drawbacks but has a big advantage that uh it feels from a developer perspective. I mean everything's running locally that you're working on that feels really great, um but it's kind of bad on windows. Still uh the network can get pretty slow.

A

uh If you are always doing this proxy network really depends on your internet connection and you might have issues with volume mounts, but generally, I would really recommend to check out telepresence the other tools. Pretty much follow this workflow.

A

That means the developer uses a cli tool to run a local kind of cicd pipeline that you know, builds images, tags and push them to. Registry starts port forwarding, um and uh you know to the to everything that has been deployed to the cluster, and here everything gets deployed to the cluster. um So, unlike telepresence, where one service runs locally, everything runs in kubernetes now um and to get this hot reloading experience so that the developer can still code with their ide, and you know, save a file and see the changes.

A

um They typically tend to have a file sync, which synchronizes files from the local file system to the container file system, and then you, the only thing you have to do is you know, install for development, a special docker file, for example, with uh which adds a hot reloading tool inside the container and maybe remote debugger things like that um might be a bit you know uh might take a while to get used to this.

A

You know the engineer needs to play around with this in like 10 minutes, but it's definitely easier in the long run um than maybe the network complexity of telepresence. Let's get to learning number six. When your platform is actually up and running, um you should really care about cost as well. For the sake of your organization and really shut down idle workloads, so the first thing you want to do is enable auto scaling whenever possible.

A

If your pub cloud that should be pretty easy, if you've done a great job with your platform and your x is great and engineers are actually using it they'll be really quick to spin up and deploy a lot of things right, but they'll be terrible at turning things off.

A

So you should really try to automate the shutdown of idle namespace of idle workloads in general.

A

So I'm just going to recommend a couple of tools: there's a fairly new tool called cluster turndown uh by a startup called cubecost, which you can use to set up a fixed schedule when you turn down your ek, eks or gke clusters. So you can essentially say: hey at you know: 8 p.m. Turn everything off right! Scale it down to zero, kill all the notes right, that's pretty pretty hard, um but obviously uh it's doing the job in terms of saving cost right and then in the morning spin up everything again.

A

um Obviously, you have to you kind of adapt to the schedule of your engineers here. um If you're based on openshift, you can use idling and openshift, which is essentially network based shutdown. So if openshift doesn't detect any network requests for a while, it can, you know scale down, replica sets automatically to zero, and then engineers can just you know, turn up the replica sets again and in loft, which I'm currently working on.

A

um You have a feature called sleep mode uh where you can essentially make that decision, not on pure network traffic, but on api server requests.

A

So you can tell loft hey if the engineer hasn't done any requests to a specific namespace in the past 20 minutes shut it down right, shut down all the replica sets or turn the replica sets to zero and as soon as the first api server request comes in uh reset, the replica sets to whatever they were beforehand and then last but not least, learning number seven is sometimes users just need more than a namespace right. I was talking earlier about single tenant clusters versus multi-tenant clusters. If you have a single tenant cluster, that's great.

A

uh You essentially become the admin of the cluster uh as a user. You can do anything set up our back. You know a lot of helm charts need you to. You know, set up certain object rules. You might not want your users to do that in a multi-tenant kubernetes cluster, so namespaces have certain limitations.

A

What's with you know, custom resource definitions? Will you be able to let your users install crds? Probably you don't want to do that right, um yeah hand, charts use certain object permissions. What if your users want to enable certain kubernetes, alpha or beta features or on a different kubernetes version?

A

You know you can't do that with namespaces, but you can, with a construct called virtual clusters, and virtual clusters are really kind of kubernetes classes that run inside another kubernetes cluster within a namespace right. um So if you let users provision those they really unlock access to cluster-wide settings for those users, they can do whatever they want with their virtual cluster.

A

They provide better isolation than namespaces because they host a separate kubernetes control, plane right, there's a separate api server, which is also great from isolation, security and stability perspective.

A

And then, additionally, because they're not true isolated, single tenant, kubernetes clusters, they're virtual clusters running in the same host cluster, you can also do kind of like a sharing of resources that run in the host cluster. For example, services like nginx, cert manager right, you just need to run them once and you can kind of hook up the virtual cluster to those host services as well, which provides similar efficiency as namespaces.

A

If you want to learn more about virtual clusters, you can either check out what the multi-tenancy sig is working on, or you check out uh what we've released uh just like a couple of weeks ago um on github.com loft dash sh virtual dash cluster, which is essentially an implementation of a virtual cluster.

A

That means we're, starting like a k3s cluster within one of your namespaces uh and then have like a syncer which syncs certain resources to the host cluster right to actually, you know, start pots and containers etc. But you have like entirely the logic, outback, etc encapsulated into one named space that is hosting the k3s cluster. um So I encourage you to check that out.

A

If you are kind of hitting the limitations of namespaces, but that's pretty far down the road um and I'm sure you'll be busy uh with you know the first six learnings uh so far. uh Just to recap uh here are all the learnings uh that you know kind of I took from creating. You know several.

A

You know implementations of different sizes of companies uh that wanted to host multi-tenant kubernetes clusters and uh if you want to dig into uh one of these topics like in more detail um just head over to uh one of these links, I'm attaching to the presentation. There was a really great talk of james wen and engineering spotify at last kubecon and there's a lot of great articles out there, and I really urge you to also check out what the sick multi-tenancy is working on.

A

And if you have any particular questions, uh you know to kiosk whatever I'm working on or to any of these learnings don't hesitate to reach out to me and I'm also going to be available for q a, and I think we have about like another five or six minutes for this. So if you have any questions, feel free to ask them right. Now, thanks for your time,.

A

All right, so that was the main presentation. I'm happy to answer our questions. So if you have any questions, feel free to open the q, a bottom uh button on the bottom uh and just type in your question, uh we already have a couple of questions that came in a few minutes ago. um I'm just going to read them out loud and try to answer them uh in the in the remaining.

A

You know five or six minutes that we have left um the first one is: uh what do you think about using oppar um and check config maps of that? um Do you see some bottlenecks here? um So a lot of companies that my team and I have been working with are actually using oper. It's a really great uh project really allows you uh to add additional. You know really flexible security rules, your multi-tenant cluster, so I definitely recommend to check out oprah is super super useful for multi-tenant clusters?

A

um Does kiosk have a graphical user interface kiosk as an open source project, is kind of like a basic building block to create soft multi-tenancy in kubernetes, so it teaches kubernetes certain. You know crds that are required like what is a tenant. You know what is a account: what is a space uh things like that, but it doesn't have a gui. um So if you need something on top of that, uh you can obviously uh build that.

A

You know just using uh cube context um if you're using uh loft uh the project that I'm working on right now, uh it's using kiosk internally. So whenever you connect the cluster uh installs kiosk and uh loft chips with a graphical user interface.

A

uh So if you're interested in that uh just install the hand chart and try it out.

A

um What is your uh experience, uh suggestions regarding the usage of crds in a platform as a service based on managed kubernetes clusters, um and then the person asking these questions is referring to issues uh with the cluster white scope of crds uh yeah. That's what I mentioned uh at the end of the talk, essentially so name, spaces are kind of like a basic unit of uh you know, sharing multi-tenant clusters um and namespace have certain limitations, because, obviously you don't want users to install c ids on a cluster-wide level.

A

After you set up, you know, arbit, etc to restrict them to a certain namespace. um So virtual clusters is really something you should look into.

A

If you want your users to actually work with c or ds, because in virtual clusters, users can essentially create cluster wide cluster scope, resources, um which, which can be super helpful uh for soft multi-tenancy recorders. You know so users to to deal with orbit rules, uh install things like controllers and crds, uh even if your users are developing controllers right. That is definitely something to look into um the next question. Solutions like virtual cluster and kiosks seem like a very nice approach towards multi-tenancy.

A

um Would you classify these under soft or hard multi-tenancy and what is the road map towards a version? One uh production ready version, um so definitely a soft multi-tenancy. If you want hard multi tenancy, you need to look into. uh You know like really making sure that users are not sharing the same kernel things like that uh or that they, you know, there's like um virtual cubelet things like that, where essentially your containers end up on different uh vms right things like things like that are definitely uh more towards heart.

A

Multi-Tenancy pressure, cluster and kiosk is really for soft multi-tenancy. If your engineering teams are supposed to share the same kubernetes clusters um regarding production readiness, uh both peers, as well as virtual cluster, uh have been used by a variety of of companies of different sizes from small companies of just like you know, 10 engineers sharing a cluster up to 100 engineers.

A

You know sharing a cluster, um obviously uh they're pretty young technologies, so there's not companies out there using it uh with like uh ten thousand engineers working in the same cluster, uh but I actually think that would probably not make a lot of sense for for companies because you might want to you know, split up things uh on in different geographical locations, etc.

A

I don't think you want your european engineers working the same cluster as uh the ones in north america, for example, right so you're to have several clusters that you share, um and then you know 50 100 engineers, maybe 200 engineers may share a cluster, but probably not 10 000.. um However, as I said, both projects have been used in production, if I may call it that way for the development use case, uh we're also evaluating both technologies for actually production workloads.

A

um But if that's something we can, you know talk about in the slack channel, so feel free to ask any questions uh there as well might not be able to to answer all the questions here. um Maybe one last question: uh sometimes I feel the need for nested namespaces or grouping multiple namespaces into one logical, bigger namespace, into one logical, bigger namespace.

A

Is that something you consider useful for multi-tenancy? Definitely I mean the multi-tenancy groups working on something like that. I also believe that google has shipped something similar for gke right now, where you can. You know essentially build these hierarchical name spaces. However, I think you know. Virtual clusters offers more possibilities than just nesting namespaces essentially, um but in the end it does the same thing. So, in the end, everything you do creating you know.

A

Name, space in your virtual cluster are actually gonna end up in the same name, space on your on your host cluster right, so you're achieving a similar goal, but with a different layer of abstraction and maybe more, you know, freedom and flexibility for your users and, uh as we only have like a couple of seconds left uh to to end this session. I just wanna take the time to thank you all for for listening uh and watching uh the video online and uh the slides will be online. um I think, on the scheduling website.

A

I can also post them in a slack channel again and if you have any questions, feel free to reach out anytime, have fun and enjoy the rest of the conference, and thank you for tuning in.