Cloud Native Computing Foundation Crossplane Community Day EU 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Constructing Building Blocks for Production Grade Cloud Native Deployments (with Cro... Hasan Türken

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Constructing Building Blocks for Production Grade Cloud Native Deployments (with Crossplane) - Hasan Türken, Upbound

Deploying applications to the cloud is easy. It gets harder when it requires some infrastructure and harder if it needs to be deployed with a production-grade configuration. And most of the time, we need to automate all of these for a variety of reasons which further complicates the work. Wouldn't it be great if we could have re-usable building blocks that could be assembled and even better if we could later define them as new blocks that could be re-used further? In this talk, we will present how we are using Crossplane and composition to deploy a highly available, tls enabled Vault backed by a GCS bucket and configured to auto unseal with Cloud KMS using GCP workload identity. We will discuss how Crossplane helps us to easily assemble infra and application components and eases automation further with declarative configuration.

A

Hello: everyone, my name is asanturkan, I'm a platform engineer at upbound and today I'm going to talk about how we are using crossplane to to provision and manage our own infrastructure and also talk about why crossplane is a great tool to construct building blocks for cloud deployments.

A

Upbound cloud offers a hosting solution for crossplane and when you go to about cloud and login to the console, you will get this screen where you can create and manage control planes and a control plane is actually a dedicated cross plane instance running under the hood and what happens when you hit the create control plane button here is closely related to what I will talk about in this talk, and this is why I wanted to start with this slide.

A

So here you can see a high level overview of our deployment model for hosted crossplanes uh a host of crossplane uh con contains a dedicated kubernetes api server, backed by etcd and crossplane, and crossplane providers are configured against this api server such that they can operate on the api resources leaving this this api server.

A

We are also configuring, this api server against volt, so that it can use it as as a kms provider, and uh this way we we can encrypt the kubernetes secrets before writing them into etcd.

A

Instead of storing them as plain text here you can see we are running multiple instances of hosted crossplanes on each host cluster, but we also have a defined capacity of host clusters and when we have when we reach to or when we reach to that capacity, uh we are dynamically provisioning new host clusters, and this is the responsibility of this scheduling operator and it needs to get new or like trigger the creation of new host clusters when we reached it to the capacity.

A

So as mentioned for encryption at rest, uh we need a production grade, vault deployment. What what we mean by production grades? uh It needs to be highly available.

A

It needs to be backed by a bucket as a storage backend, and we are using a google cloud for our infrastructure right now, and uh this needs to be a gcs bucket then, and since we don't want to uh like, we don't want to require any manual intervention with by while operating this uh host clusters or that bolt instance.

A

uh We we want to uh configure auto unseal with cloud kms so that we don't need to seal vault when it's, let's say when the pot gets restarted for some reason, and since this world instance will need to consume google cloud services like buckets and encryption keys, it needs to authenticate uh google cloud and uh google recommends using gcp workload, identity to authenticate workloads running in google cloud kubernetes, google kubernetes engine, so we also want to use enable and use gcp workload identity here and finally, of course, we want to have like all communication as encrypted, and we want to have like we want to deploy uh vault as tls enabled so uh remember.

A

Our host cluster has a vault instance and we want to dynamically provision host clusters. So we need to automate this deployment and, since our scheduling operator is responsible for dynamically provisioning, when when we reach to the capacity of existing ones, we need an api for our scheduling operators so that it can just request a new uh host cluster with vault yeah. As mentioned like our scheduling operator, needs an api and it will just say give me a production re-default.

A

So here I want to uh go through the provisioning flow file for such a deployment. uh So, first of all, we need to interact with google cloud apis and create a gcs bucket for storage. We need to create kms keyring and a crypto key in this keyring for auto unsealed.

A

We need to create a service account and for the vault servers, and then we need to grant access to the service account so that it can access to the previously created resources.

A

And finally, we need to create a gke cluster so that we can run our world servers on it and also note that we need to configure this gke cluster as workload identity enabled, so that we can use gcp workload, identity and final step of this deployment is actually deploying the vault itself and, while deploying volt, we are using the volt helm chart and we need to provide uh these uh these the names of the uh cloud resources that we have created as as helm parameters.

A

So you can see like we are passing the bucket name, uh key ring name, crypto key name uh and also we are passing the uh service account as an annotation so that uh it can authenticate with workload. Identity also note that, like there is a tight coupling between the infrastructure and application and like we need to consider this as a whole, rather than just infrastructure application, I would like to talk a bit about evolution of cloud deployment.

A

So far in the beginning, when the cloud providers started providing services, we were going to their uis and creating cloud resources by clicking couple of buttons there.

A

They also provided cli interacting with their cloud apis, but exceptional scripting. There were no meaningful or feasible way of automation with the emergence of infrastructure as code tools like terraform, ansible and chef.

A

We get better ways of automation, but we still lack of an api which we could use to provide this automation to the consumers, and here comes crossplane. Crossplane both enables automation of like deployment and management of cloud resources, but it also allows an api so that the consumers can just create these resources or even manage them.

A

So, coming back to our original example, which is uh deploying a production ready world on gke, I would like to show couple of repositories.

A

So this one is like kind of corresponds to the first phase of like cloud deployment which is uh making that deployment deployment using cli step by step.

A

And the next one that I would like to show is uh like following the same deployment model in the previous uh repository, this repository automates it with terraform and now I would like to introduce this one, which is uh like making the same deployment using crossplane, which enables making the same deployment using crossplane, and this repository contains the necessary compositions cross-plane compositions to achieve such a deployment.

A

And once we like, uh once we deploy this configuration in a kubernetes cluster that that that have crossplane running. We have an api resource uh like we have an api resource which we can just create and expect it to be uh up and running.

A

So coming back to like our deployment model with hosted for hosted crossplanes. uh As you remember, you might remember, we have a scheduling operator there, and this scheduling operator needs to uh dynamically provision. Host clusters, with like production, ready, vault and having crossplane api, which is powered by kubernetes api, uh enables us such a like clear separation of concern and our scheduling operator could just request the host cluster and then by polling that api or like watching that that resource it can just get notified when the resource is ready.

A

Another advantage of like using uh like another advantage of uh using kubernetes api here is: we can just use existing tooling and machinery around kubernetes api, so we could, for example, use kubectl to interact with these resources and we could like build a an operator and use controller runtime to to manage those resources.

A

So, let's have a closer look: what happens when we create a resource across plain resource? So you might remember the like right hand, side, which is uh the provisioning flow for such a uh for our example application and, uh on the left hand side. uh We we see that, like uh we have a composition which contains the required managed resources. These managed resources are in this case are like gcp resources or we are also.

A

We also have bolt release helm, release as a managed resource and, uh like all of these resources, are composed in a in a composition called world cluster, and when we create a world cluster custom resource, then cross, plane, composition, controllers goes and creates corresponding managed resources, and then the the corresponding providers x on that and uh like creates and manage those those resources on cloud or on the cluster having a control, plane and resources living in the same api enables building new resources on top of existing ones.

A

I think most of us are familiar with the relationship between pots. Replica sets and deployments replica sets are built on top of pods and deployments are built on top of replica sets.

A

So with crossplane. We can represent everything as a kubernetes resource and just like the relationship between any kubernetes resource here, we can also build new resources on top of existing ones. For example, in the in the figure you can see like my cluster resource is like contains other existing resources like network sub network, a gke cluster and a couple of node pools, and there is another uh like resource called my vault, which contains the required cloud resources, as well as a health release, which is also a like managed resource.

A

And again we can continue building new resources by combining these newly constructed building blocks. So we can say that with crossplane, each and every resource or each and every cloud resource is just a block and we can build new blocks by combining, like by composing new uh resources thanks to crossplane composition.

A

Okay, it's time for demo and in the demo, I will create this custom resource, which is created by a composite resource definition, and I will expect that all of the required cloud resources to be created by crossblank providers and a like production ready world is deployed on on the like newly created gke cluster.

A

Okay, let's start with the demo, I have a local kind cluster and I have cross plane installed before the demo and I will now start by deploying the configuration package which is like built from the repository that I showed in the presentation.

A

So, let's start with installing it.

A

Let's check how it.

A

Goes yeah here you can see that vault on gke configuration package is installed and ready. uh This configuration package contains the composition uh compositions and like composite resource definitions uh that that we need for volt deployment uh and now the next. The next step is uh like deploying the our resource.

A

uh Since we will use uh provider gcp, we will need to create this provider config, which refers to a kubernetes secret uh in in uh named as gcp credits. I have created that secret uh prior to demo, so let's create this provider config as well.

A

uh Okay and the next step is actually creating a network composite resource, and now we can create a vault cluster uh composite resource that that I showed at the end of the demo.

A

So now, let's see what we have ctl gets more clusters, our custom resource is created and let's check what managed resources we have for this composite resource yeah. Here you can see that, uh like out of managed resources created, gk cluster, not pool service accounts, keyring, crypto, key crypto, key policy, buckets policies for buckets, etc, and uh let's also check the cloud console.

A

Yeah, as you can see, uh a like kubernetes cluster is uh being provisioned right now uh and since this will take a while, I would like to uh like go to to the repository and we can use this time to go over the configuration uh in that repository.

A

So like we have uh multiple compositions here uh and uh like let's start with the gke composition, uh this composition contains a like gke cluster resource provided by provider gcp and uh contains a node pool resource again, provided by provided gcp.

A

This composition also contains one last resource, which is a provider config resource which is going to be used by provider helm so that it can make the deployment to to that like to this newly provisioned gke cluster. So when this composite resource is ready, we will have a provider config pointing to to that cluster.

A

So the next composite resource or the next composition is uh the one that named as volt and this. In this composition, we are creating like a lot of gcp resources.

A

One is for like service accounts, key ring, crypto key crypto, key policy service account policy bucket a good policy membership, and we have uh two hand releases here, uh one. This hand, release is just to uh like to deploy tls secrets for four volt uh servers and we are using the like capability of uh help chart to to pro to create uh certificates.

A

uh This held chart contains nothing but just a tls secret, which will deploy uh like volts tls secret, uh but this world-based help chart could be used further like if, for example, you want to enable network policies, you can like put your network policies here in this chart, or if you want to run some like pre-installed configuration script or job, then you can uh like put your manifest in this hand, release this is quite convenient so and finally, we have our actual hound release, which will deploy the vault itself, and here you can see we are using the official wall town chart and we are providing uh chart values here and since we will need to use dynamically created uh like resource names, we can just mention, for example, gcp bucket name here.

A

Rather we need to patch patch it from the uh like composite resource. So, let's see, uh let's take a look for an example. I think the the best example is this one.

A

So here you can see that we need to pass, for example, bucket name, and we need to pass like project name key ring and crypto key name by the way we are using a single key link, but we are creating uh multiple crypto keys under this key ring, so dynamically created crypto key name uh will be fed here and we are doing this by using the like uh crossplane composition, patches and also.

A

I would like to mention that at the sign at the time of this recording, uh it was like there was an open pr to which enables uh multiple input compositions, and this uh I'm making the demo from that open pr and uh like that that pr enables such uh patches um yeah. So these two compositions that we had a look but, as I mentioned in the like presentation, uh the cool thing is once you define your compositions and composite resources at a kubernetes api. You can go for further and create more compositions by combining them.

A

So this composition actually defines a new type called world clusters and it combines the previously defined compositions. So this is a composition of previous two compositions which, like composes, a gke which contains the like gk cluster, not full and provider config for helm and also it contains volts composition, yeah. So let's have a quick look. How our deployment is going.

A

So yeah it looks great, so, as you can see, all of the resources seems to be ready and synced. uh So this means that our composed resource should also be uh like yeah. Our composite resource is also ready. Let's also check our helm releases, so that's yeah here you can see that helm releases are uh like actual help. Release is deployed on the like gke cluster. Okay, let's connect to that humanities cluster.

A

Let's system name space and yeah yeah. As you can see all volt uh clusters. uh Volt uh ports are up and running uh and they reported as ready, which means that they are uh unsealed. uh Maybe we can also check the world configuration from one of like one of them.

A

Yeah, so here you can see that uh volts, vault server configuration uh like it is configured for using this uh kms crypto key. This key ring uh and uh like h, a is available with uh gcs storage uh yeah and uh if I could scroll down, yeah and as you can see like vault is unsealed, okay, yeah. So that's the end of my demo.

A

I'm happy to answer any questions. If you have so thanks for joining my session.