Cloud Native Computing Foundation Crossplane Community Day EU 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Keynote: Operational Nirvana: The future belongs to the robots - Jay Pipes, Principal Engineer, AWS

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Keynote: Operational Nirvana: The future belongs to the robots - Jay Pipes, Principal Engineer, Amazon Web Services

AWS Controllers for Kubernetes (ACK) lets you define and use AWS service resources directly from Kubernetes. The ACK project has a code generator that is now used for generating the Crossplane AWS provider. In this talk, we will discuss why declarative Kubernetes-native configuration management tools like ACK and Crossplane are so important: they enable a GitOps-centric operational model for safely deploying infrastructure and application resources.

A

Hi, my name is jay pipes, I'm a principal engineer on the kubernetes at aws team and today I'd like to talk to you a little bit about the aws controllers for kubernetes project or ack.

A

But before I tell you about ack, let's back up, I want to first talk, give you a big picture view right, a bird's eye view of how ack fits into the entire kubernetes and deployment system uh deployment system uh ecosystem.

A

Sorry, um if you haven't been hiding under a rock over the last three years- and you may have been because of cobit and all that, but if you haven't you've probably heard about get ops right, git ops is sort of like the latest uh latest craze in the devops and um deployment community right and at the end of the day. What git ops is it's an operational methodology for doing safe, repeatable software deployments?

A

That's it right um kind of seems a little bit marketing speak though, so maybe you have gone on online and like what exactly is git ops and you've. Seen these four things that are on your screen right now as github's best practices right, declarative configuration management, humans using git and git-based workflows to change the configuration of an application, making sure that all changes are observable and verifiable and also making sure that robots are doing the deployment and not humans right. So these four best practices in isolation yeah they kind of make sense.

A

But for me I know that they, it didn't really click for me internally. um Until I related the first two to each other. By saying, if we use declarative configuration, then we empower humans to use git based, workflows right in order to change the configuration of an application right so get ops is all about having a source repository in git.

A

Usually that contains manifests right configuration files that describe your deployment right, that describe the services and the staple sets and the deployment gamo files in kubernetes right, the ingresses, all those kinds of kubernetes specific objects right those go into manifests in a source repository, and when you want to make a change to any of those configurations, we do so do a git commit and push those configuration changes up to usually github github is the most popular uh source control um uh platform at the moment right, and so you make a change to the configuration file of you know something involved in your application.

A

You do a git commit and push that change up to usually github and create a pull request and that pull requests. Then you know your your teammates can approve. You know review that change to a configuration just like they would a a piece of code, a change to to part of your application code right and they either approve or deny that particular change.

A

Well, what's important here is that that is the end of the human interaction with git right humans. Stop there once um once the pull request, or you know, configuration request, change request is approved and merged. That's the end of the human interaction in get ops right.

A

After the changes are merged and uh using you know, after a configuration, change is merged into a git repository. Then the robots take over and we'll get to that in a second. So why does kubernetes fit really well into a git ops model? Well, at the end of the day, kubernetes has a kubernetes resource model right. That is all declarative right. You have. A set of these yaml manifests that describe a staple sect and a deployment and a service, and an ingress and um persistent volume claims right.

A

All of these kubernetes native concepts, kubernetes native resources, they're described using a declarative format right, that's why get ups and kubernetes sort of fit together so well?

A

Well, just like kubernetes and get ops fit together, really well for kubernetes native resources like stateful sets and deployments and services. Well, what crossplane brings into the mix is that now, with cross-plane crds custom resource definitions that are themselves kubernetes resource manifests.

A

Now you can declare everything to do with your application and your infrastructure using the same kubernetes resource model right, so crossplane brings the everything bit into uh into the get ops world.

A

So the second two second pair of best practices in get ops is all about, observing and verifying changes to those configuration files that are stored in the git repo, and I like to put them together and say: if all resources, all resource changes are verifiable and observable, then we enable robots to do the deployment of those configuration changes out into a target environment, whether that target environment is the kubernetes cluster or it's aws cloud or google cloud or azure. It doesn't matter so flux, cd right, that is the canonical git ops robot.

A

If you will, that is doing at the end of the day, is doing a coupe cuddle apply right. It's automating the process of applying the manifests that are stored in a git repository and in doing so actuating. Those desired state changes that are stored in those manifests out into production.

A

So if we think about changes to kubernetes resources being observed by kubernetes controllers and things like flux cd well, now we can think about changes to all resources being observed by any sort of kubernetes controller.

A

Cross crossplane has a set of controllers that are kubernetes: custom controllers, they're listing for new, changed and deleted custom resources of of uh api kind that are related to the cross, plane ecosystem.

A

So when say, um you create a new database instance or a message queue or an a role right, an identity role, and I am role um you will store that desired.

A

Configuration resource right in your git repository, push up a change to that and create a pull request, and at that point, um flux, cd will kick in and apply that particular change after the pull request has been approved and the controllers, whether they are kubernetes controllers or their cross plane controllers, pick up that state change and attempt to consistently and constantly reconcile the desired state to be the latest observed state.

A

So if we put all these things together, we like well, I like to say this- is an operational nirvana right, we're we're able to declare the configuration of our entire system, not just the kubernetes native resources like deployments and staple sets, but because we're integrating crossplane and kubernetes and flux and the git platform like github.

A

We now can describe all of the resources all of our infrastructure resources, all of our kubernetes native application resources all using the same format, all using the same git based workflow right, where humans uh are responsible for approving the pull requests that describe those changes in configuration and robots are responsible for actually deploying things right.

A

Okay, so I spent all this time talking about get ops. Well, what about ack? Where does ack come into play? Well, it depends.

A

Ack is actually two different parts: there's a code generator inside ack that consumes api model files, aws api model files like the s3 api and the rds api, and produces go code for translating those apis into the kubernetes resource model.

A

Now that eight, the the cogenerate code generator inside ack, called ack, generate it has a an extensible and flexible producer pipeline. In the first part of that pipeline, we can take those api models from the aws services and output, the cross, plane provider, aws module in another way of running the ack generate code generator.

A

We can consume those api files and output, individual custom, kubernetes controllers, one for each aws service, and we call these the ack service controllers so, depending on whether you want to take advantage of crossplanes cross-cloud provider and sort of higher level abstraction model or whether you want to install individual aws service controllers in your kubernetes cluster, we can do both in ack one thing uh before I wrap up here. I want to make sure that everyone understands that ack, it's 100 open source and we're 100 collaborative with the open source cross, plane and elsewhere.

A

Communities right, there's also there's nothing specific about uh eks, uh so ack, the service controllers inside of ack and the code that is generated by the ack code. Generator is not specific to eks. They will run on any target kubernetes cluster, regardless of whether you choose to use the the manage control plane um options of eks.

A

So for those of you who are interested in learning a little bit more about ack, um please come join us. We we have the the on the screen. Here is our uh our root um url to our our community project on github, um I'm always hanging out on the provider aws channel on the kubernetes slack, I'm also on the cross plane slack. I look forward to working uh with the crossplane community, even even more and greater in the future, and I thank you very much.