Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 1 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Learnings From Providing A Platform API With Kubernetes And Crossplane - Hannes Blut & Jan Willies

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Learnings From Providing A Platform API With Kubernetes And Crossplane - Hannes Blut & Jan Willies, Accenture

Kubernetes' extensible API has turned it into a de-facto abstraction layer not just for building, deploying and operating cloud-native apps, but also as the control plane for the entire enterprise, to provision and manage cloud resources and complex platform components. This session will be about the journey of Kubernetes and Crossplane at Deutsche Bahn, to provide platform consumers with access to a unified API for deployments, infrastructure provisioning and applications in a manner that is independent from the cloud, addressing compliance and cross-cutting concerns while providing a Kubernetes “native” experience. The journey has not been without challenges, where the platform team has managed technical and functional requirements including an access model in an enterprise environment, user expectations of cloud native infrastructure usage, and issues with excessive API load, shared resources, as well as controllers written by the team and open sourced along the way.

A

All right welcome everybody to our talk on kubernetes and crossplane and building a platform api on top of kubernetes. So there are still some free seats available here in the front on the right side and a couple there on the left. If you don't want to stand in the back and well today, we want to talk about our experience in adopting kubernetes controllers for managing things, who are not core kubernetes controllers, and we are doing that with a cncf project called crossplane and they have quite the presence here.

A

So I want to have a quick show fans who has heard of crossplane before all right all right. So it's hard to not uh hard to ignore crossplane at this conference. That's for sure, and that's good. So we'll start with a quick intro uh who who we are uh michael lecanis. Maybe you want to go first.

B

Yeah, my name is hans blut. I I'm a cloud architect at accenture and I live in frankfurt. Germany.

A

My name is jan vegas. I live in berlin, I'm a platform architect accenture. I do uh I do most of the stuff. I do is open source, I'm a contributor to the crossplane project.

A

I do also do the valencia icd meetup, so if you're in berlin- and if you are interested in in in connecting and with other folks who are running, cicd, then feel free to reach out. It's been on hiatus for the past two years due to some pandemic events, but we plan to pick it up this summer again and we are also managing the foster mci cd dev room. So if you are joining foster in brussels, hopefully in presence next year um feel free to catch us and the cicd deaf room.

A

All right, so a very quick introduction, crossplane so probably you've by now heard many intros to crossplane. uh This is uh this might be the third one, but let me quickly explain in layman's terms, so what crossplane does it extends a service provider apis which are pictured here at the bottom, into the kubernetes resource model into the kubernetes database? So a as a user, I can cube cd get clusters and I do get my eks cluster back from aws similar.

A

I can create kubernetes clusters via via the kubernetes platform, so I just apply a familial yammer and there's a controller running who takes this yaml representation and talks to the aws api via aws go sdk and manage the state at aws. For me, it continuously does so so that's the kubernetes pattern, the reconciliation uh every well, it's configurable, but every 60 seconds by default.

A

So if there's any any drift or any changes, then those are overwritten or updated to the state which I want to give that. So this is a kubernetes example. I can provision kubernetes via kubernetes, but obviously I can create any object at any external api provider. So it could be an s3 bucket.

A

A cloud storage object, an aks cluster, a lambda stuff, so anything really which which is mirrored into the kubernetes ecosystem.

A

Okay, let's take a closer look at the kubernetes example. I'll have to take the kubernetes example, because it's uh it's a very complex one, um usually an s3 bucket or a cloud cloud storage bucket is just a similar, a very simple object and you just apply it. So there's no like magic behind that, but to provision a kubernetes cluster at uh cloud providers or or on-prem is usually a very complex thing to do, and so pictured here at the bottom are it's an example stack of a fictional company which you you might run.

A

uh There are two club providers, aws and azure. There's a code repository gitlab, uh there's a grafana for dashboarding there's. Obviously, probably your organization runs kubernetes, so there's the kubernetes api vote for secret store, styra for policies, argo, cd for githubs and so on and so forth. So all those are mirrored into the kubernetes ecosystem, so you have them available as like one-to-one representations.

A

However, that's usually not what you want to kind of push this complexity to the to every user of your of your platform. um What you most of the time want to do is at least apply some compliance and security. So, for example, a famous example is the visibility of an api in in the kubernetes case. That would be like every kubernetes cluster should be a private kubernetes cluster and not a public one. That should be a default with nx s3 bucket.

A

It's the same should be private by default, for example, and should be encrypted via kms, so those are defaults you can set, and you also want to make it simple and in the case for kubernetes cluster, there are many many different api calls.

A

You need to take to add aws, to create a cluster, then at your internal sas services to register the cluster once it's ready and to install operators and controllers to make your cluster compliant and integrate in the rest of your enterprise and that's something which happens here in the integration part and in the custom platform part at the top. You see the simple object which your users of the platform are using when they want to have a kubernetes cluster and provision a kubernetes cluster.

A

So it's a very simple one, but under the hood it goes to aws and it creates a im role: im policy security group, oelc provider, actual eks cluster, um all those eks cluster add-ons um which are available- um and it goes to it, goes then to kubernetes, because once the eks cluster is ready, it's not like not ready to hand out. It still needs to run like compliance and integration software which you need to deploy and that's happening via the kubernetes api, then in the same kind of call.

A

So um there then is deployed the compliance software for cm or security scanning, also um and all those controllers for integrating which are not available via eks add-ons and then the the kubernetes cluster is registered at um well. In this case, at styra, which is a control plane for policies for opa, and it's registered at argo cd so that users can use githubs to deploy their application into the cluster. So they get everything out of the box, but still that's hidden from the user by the platform team by this integration.

A

So in the end uh crossplane to achieve this, it's a crossbar is a kind of a low code platform. So you cannot like edit the stuff via ui yet, but you, you can offer an api without writing actual code, so you can describe it in and it allows you to provide kind of abstractions and governance and security to it all right. That sounds really fun and games.

A

But what are the top nine challenges uh we encountered when running this approach at scale and in production, so at scale means we are running it's? Obviously, it's not google scale. We are running a couple of thousand managed resources. So that's uh I guess it's not like um super less.

A

It's also not like google scale, but maybe it's something in between and we want to go over some challenges we faced when when implementing that we've been starting with crossplane two years ago, and this helped us to like get a very, very good understanding of the product. We adopted it pre version 1.0 and we're able to uh kind of shape a bit the product to our needs and some of the stuff you see in today's crossplane is based on our feedback.

A

So the first thing is the crossplane deployment model, so crossplane is a kubernetes extension and by that it's it's installable in a kubernetes cluster. So you might go that just simply. Every application cluster gets a crossplay installation, so the application is the the blue boxes on on the right and the crossbar installation is the the colorful representation.

A

So what that gets you from an application point of view. It's a very good integration between infrastructure and application, because now you can take the you can read from an application. You can directly read the connection details of the database or the the s3 bucket or so from the same cluster in in from a secret, for example, uh and also you have a very good heart: isolation between tenants because well, kubernetes uh tenancy model is the best when there are different kubernetes clusters. However, how do you bootstrap those clusters?

A

So, in the end, you can offer a very the the broad range of the cloud provider or sas provider offerings into the cluster. However, a kubernetes cluster is just one of these offerings and with this approach, where you are kind of, it's expected you're already running a kubernetes cluster. So how do you get to those clusters?

A

So the next thing in the middle is well. You have a central api for infrastructure provisioning so that users go to the central api provision, their cloud resources or sas resources, and then they are using those sas resources directly.

A

So they go to a central crossplay instance and create a bucket any gis clusters and so on, and then they connect to those directly and use that so that's great bootstrapping problem solved. I have a. I have a control plane for all my infrastructure.

A

However, now you have kind of a bit of the soft isolation problem between between tenants, although you're not running any workloads in the central cluster. uh That's that's offered by like a central platform team.

A

There are no workloads, so it's not like, like uh resource starvation or so, which you might run into, which is a common problem. If, if, if you're running a multi-tenant kubernetes cluster, however, you are still running a single controller for the entire um for for all the tenants, and you could still get into some issues when, for example, one tenant creates a huge load of api resources and then you get rate limited or something, because crds are cluster-wide in kubernetes, as you all probably know, and cluster-wide means there's only a single controller, reconciling those objects.

A

So there's no well at least not a crossplay, no, no way to shard uh controllers to uh crds and that's kind of the issue. With this model there are some scalability uh challenges.

A

So third on the bottom is well obvious solution. You run a crossplane instance per tenant, so everybody gets a crossplane instance. However, it's a kind of a tricky thing because you don't want to run the crossplane instance in an actual kubernetes cluster because it doesn't run any workloads it and it would be very kind of, from a resource point of view, very cost intensive to run a dedicated kubernetes cluster just for the control plane for every tenant.

A

That depends on how many tenants you have, obviously so those can be virtualized and there are some uh ways to virtualize them via open source tools. So a lot of folks use vcluster by loft.sh for running virtual kubernetes cluster or there's the cluster api provider nested from from the special interest group from kubernetes.

A

Who does also that and probably a thousand other ways you could also use an offering from upbound to achieve that, and then you can run the controller either inside the virtual cluster or you can run it outside and connect to the virtual cluster. But you get the heart isolation, that's solved!

A

You get a separation between control, plane and data plane. However, it's a bit more complex setup and probably your adaption goes a bit along this lines.

A

All right, the approach to compositions- that's something we we face so compositions are the feature how to abstract um resources.

A

So when you build your automation, you maybe you are in an application team and you want to start with the application so build from top down application on top crowd resources at the bottom, and you want to deploy the application and it should come with like a stack, it comes it's a vertical integrated stack application uses database, object, store queue, maybe kubernetes, maybe not maybe cloud functions or so, but that's something you want to.

A

Aggregate, so basically, you abstract the application, that's very useful if you're, starting with crossplane from like either an application team or a small company.

A

But, however, if you're, starting in a large organization, that's very cumbersome and very challenging to implement, because now you're the one with the sme for crossplan nf. The experience with running that you need to go to every application team in the enterprise and build those abstractions for them and that's uh simply not gonna scale. So you start from the bottom. You are the service provider and you want to offer those resources to your users.

A

So from like the uh azure example, you have from the aks, you build a a compliant aks object, a compliant aks api, a cloud compliant blob store, uh a compliant um cosmos db and a compliant load balancer. Those are just examples here, so those applications team they can just like pick and choose what they from a service, catalog kind of of thing and then like from a maturity, point or adoption points. The third example here would be the one where the adoption is is quite far already.

A

A platform team would create those um stuff from the bottom, so those compliant api objects and then application teams who are already familiar or kind of uh borrowing resources from from the central team. They would write their automated automation, based on top of those compliant objects already.

A

Well, speaking of all those compositions, that might be not a surprise uh for many of you, but uh kubernetes is a favorite. uh Well, someone said that at a conference, kubernetes is my favorite yama database so expect that you need to manage more, even more yaml, so probably already managing a lot of yaml for sure. If you're running stuff, like the q prometheus deck or something, um but it will now get even more. It's like crossbar is a low code platform. You don't need to write, go code to for just well use to use it.

A

However, you need to still describe those apis and those compositions, and that's done in open api in yaml, so there could be a lot of bugs, as, as you all probably know, with indentation and and and white spaces and types and so on, depending on their approach, how you manage yammer.

A

So if it's just like basic templating and templating or or writing that manually, that's maybe something which will uh which will change when you, when you adopt that so definitely some some stuff, I'm very excited about, is culang for when you can do validation and like have a type safety and all this kind of stuff. However, it also kind of uh is a bit of since it has a very good go integration.

A

It's it's very kind of uh it defeats a bit the low code pattern, because now you kind of still go into a bit of of writing code. However, that's you get a lot more safety. This way.

A

User expectations, when you apply conte a pod spec, you get a container almost immediately, so at least you should, but infrastructure is slow. So if you apply, for example, an rds database, it will take easily five minutes or more, and there are some very familiar uh famous examples which take 10, 50 minutes for infrastructure to get ready um and that's something which some users might not expect to see from the kubernetes api because usually from what they know, the kubernetes api. It's like, quick and- and you get a like immediate reaction.

A

So some might just like delete and recreate those objects in a very fast way, and that's it. Infrastructure usually doesn't like that and then obviously, depending on the approach, your users might not know kubernetes. So there are some users in your org who, who don't use kubernetes. They are running google cloud functions or lambdas, or app runner or cloudrun or or some like data batch products, aws batch or something.

A

But now they need to kind of describe their stuff in kubernetes yammer. So while they not might run workloads in kubernetes, they still need to kind of know the api. So you can get around that by well offering a ui so that they can more or less uh create those resources via workflow or via ui, but still in the end, they need to know some api to create those stuffs.

A

Okay, I'm looking a bit at the time, so I'll skip the stuff.

A

Another thing now that you're using kubernetes for not only applications but also for your platform and the infrastructure. You can leverage the wealth of the kubernetes ecosystem and that might be a well at least. It sure is my first example of showing the cncf landscape in a in a good way, because uh many many of those things are, or probably all of those things are integrated with kubernetes.

A

So to I brought two examples here. So, for example, since now you have this central layer where you apply all your stuff, you have also a central way for managing policies. At this layer. You don't well still. You should use, for example, those im stuff on aws and on on google cloud and roles and responsibilities at gitlab and um and and argo cd and all those sas offerings you you might use.

A

However, you can do you can shift a bit more to the left and apply those things in a common language at the same layer, so you've write the same policies kind of for for for the source code repository for the github software for for graphana for for kubernetes for the application so well on a on a very low layer.

A

That means the arbex stuff from core kubernetes, but you can also- and probably you should run some more fine grained policy execution stuff like opa, like cuberno or any of those things, and what you can do with those is then to have like quotas, for example, for um for your infrastructure so, depending on most sometimes the uh the billing processes in enlarged orgs are not very uh cloud native. Maybe so there might be some subscription models.

A

So someone says: okay, I've like an sml subscription- and s is, I don't know two kubernetes clusters and and ten nodes, and this you can easily implement in uh in at this layer. So that's it. It's really easy and other examples are, for example, running cube, janitor to just really like remove demo environments after after a day or two, um that's pretty easy or cube cost to manage costs.

A

So this gets very easy, and that was just a small examples. I'll now head over to hannes for free all right.

B

So the next point or the the next topic, where we had to deal with some issues, was a crd maturity.

B

So crds are very nice to extend kubernetes, but they they were built to um to be part of the container orchestration at first and then they evolved to be more than that where people started building operators rather than controllers that were just inside kubernetes, and with with that now, as jan said, there are providers that implement all the resources for their specific cloud provider and with aws that's over 700 apis and with azure, that's over 600 apis, and then you end up with multiple thousand crds in your cluster and just that just doesn't scale very well.

B

So one of the issues that we had was the kubernetes behavior around immutability. In in the open ap scheme, you can specify that the field is immutable, but it's not observed in kubernetes. So sometimes your pro your cloud provider also has some sort of api points that can't be changed.

B

I don't actually have an example, but I I remember they that you, you couldn't change uh like the type of node that you're running on or something like that. But and then I could say in my crd description. Please don't change this or don't allow it to change, but you can and then the user is confused about. Why why it's not updating in aws? Because that's what you would expect from the api.

B

The next thing that we have is when we're working with apis. We have some sort of like if it evolves. You want to version it because you might have breaking changes and crds. Allow you to write conversion web hooks, but until recently that wasn't available in crossplane, I'm it's being worked on, but it's not there yet and then. On the other hand, even if you have conversion web hooks, it's still more reliable to uh to do the migration manually and to help your users.

B

Do their migration over time, as in the deprecated deprecation version, that kubernetes is using itself um all right. I will also skip the rest here because of the time concerning the scalability. There was a talk yesterday.

B

It was yesterday afternoon the crd that broke the camel's back. I'm sure you'll find some information there.

B

Since we're using yaml in our kubernetes api. You can. We can now deploy everything uh to through git ops. Instead of ci ops, we don't use a pipeline that executes a terraform script anymore. It's you can actually use the same tool like argo cd, for example, to synchronize everything that we have in our repository to our cluster.

B

And that allows us to use the same mechanisms for our applications and infrastructure.

B

But again this this goes back to what is your user expectation and if you say, okay, you can use the same same behavior or the same api for both.

B

They might expect you to have those offered in the same cluster and then with what we discussed earlier about the. How do we deploy or architecture our different crossplane control planes? It's just that they might expect to be able to deploy both into the same cluster, for example with the helm chart, but that might not work and so there's some interesting development around multi-cluster, architectures and or management and cluster federations, and that is something where we need to figure out.

B

How can we make it manageable across multiple clusters as well, um and then there's a of course, the question: what's the end like? Where do we? uh Where do we stop using githubs? Do we deploy our secrets with githubs?

B

We can use tops, for example, to to encrypt everything in our repositories and then let it deploy, but it might be more reasonable to use a secret provider like walt or the secrets manager or something similar um to just request the secret and then use that as credential instead and of course, using using git ops with your node group, with like a scaling configuration that just creates more conflicts where your your provider is, or your argo city is trying to to sync your repository and then on the other hand, you have the autoscaler telling the cluster please scale down and and crossplane in between thinking the options back.

B

So you have a couple of conflicting controllers as well, and that then there's there are options around that you can set fields to be ignored in some in some sense. But that is an issue that needs to be solved uh whenever it occurs.

B

um And then I think what we also learned uh since we're offering a platform to our users.

B

It needs to be documented and that's that's not enough to just to write a short entry and oh, please, use crossplane pull your clds from the server because get crds is already an operation that a normal user can't do because it's cluster scoped.

B

So we need api documentation similar to the doc cld staff. That really well reflects all the apis. In for crossplane, for example, then there is a usage documentation on what are the behaviors of what the user is deploying. What are the side effects? How do they interact and, of course, whenever something changes or is deprecated, we need release. Notes, really need some sort of life cycle management and api and operator lifecycle management. There was another talk on wednesday on this.

B

I think on wednesday.

B

And then the last thing is contributions, since we are working in an enterprise situation, it's important to be sure that you know we can use open source, we can contribute to open source, and in our case we were lucky. We were able to engage with crossplane very early on to shape their their product as well. So we can um can make sure that it fulfills the needs that we have at that point or um that that you need for building a platform.

B

We have also been contributors to the provider aws in a lot of resources, and we have open sourced a couple of other providers like the provider, gitlab provider, argo cd and provider styra and then, with a provider graphana. We have built it using the jet implementation or like the terra jet implementation, and that was then taken over by the grafana community and they are now maintaining their own provider as well, which is something that might be nice for all the other cloud providers to do as well.

B

Thank you for listening and if you want to to reach out to us, feel free to do so, and if there are any questions, I think.