Cloud Native Computing Foundation GitOpsCon NA 2022, 28 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Build and Deploy Cloud Native (OCI) Artifacts, the GitOps Way - Mathieu Benoit, Google

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Build and Deploy Cloud Native (OCI) Artifacts, the GitOps Way - Mathieu Benoit, Google

Cloud Native (OCI) Artifacts are not just containers, they could be any files: Helm charts, README.md files, etc. Could you imagine a GitOps world without Git? Let's see in action how you could build and deploy OCI artifacts, the GitOps way. The first part of the demo will demonstrate how to build your own Helm charts as OCI artifacts by having tools such as Trivy (security scanning), Gatekeeper (compliance checking), Helm and GitHub actions in order to validate and build your OCI artifacts during the Continuous Integration flow. Then, the second part will demonstrate how you could actually deploy OCI artifacts in your Kubernetes cluster in a GitOps way with Config Sync, Kustomize and Helm.

A

Hi, everyone very excited to be here. um None is unfortunately not able to make it today, um but she kobuled this content with me. um We will talk about oci, artifacts and more specifically and chart as well and how we could deploy them on a github's way and manner.

A

So this QR code is lending to this actual presentation, so you could download the PDF quick overview of what githubs quickly we are on the GitHub scan day. So you have plenty of content for defining githubs, but quickly quickly. We will see in the context of this presentation presentation we will talk about oci, artifact and N short and their advantage.

A

um We will talk about config, sync, one of the github's tool out there and opa GateKeeper.

A

We will have two demos, actually one illustrating how you could end to end package, your gatekeeper policies and actually deploy them as an oci artifact with githubs on the second demo, you will actually package a name chart for your application and you will deploy the same shot on a github's manual as well, some takeaways and resources at the end so quickly um when you have clusters- and you want to deploy your configuration- and so you have your fleets of cluster and you have different personas building their own manifest kubernetes manifest it could be for security policies.

A

In our case, gatekeeper policies soon could be shared configs role, binding cluster role, bindings um other configuration across and widely in your clusters, but it could be more specific for specific apps that apps operator could and want to deploy actually and the usual way, maybe could be a push mechanism, Cube, CTL, apply and other mechanism with a cicd tool, Jenkins and others with githubs. The goal here is not to have anymore this push mechanism, but more pool mechanism. So again you will Define the githubs engine and Tool, maybe Argo CD flexity.

A

In our case, it will be mixing pooling, actually is a different, manifest from different repositories right. So that's the principle of reconciliation, continuous reconciliation of githubs.

A

What's an oci artifact who is using M chart in their organization yeah a couple of yeah more than Alpha of the room? Are you familiar with oci artifact yeah 10 people, ish cool, so oci stands for open container initiative and that a Linux Foundation project actually and the goal is to Define back in the day, with Docker and still with Docker and other partner is to Define.

A

What's a specification of an image, a container image right, Docker container image is one of them, but there is more abstract and generic way to define a no CI, artifact or image, and the goal is to have a standard specification to Define both the format of the image, as well as the runtime to actually run this artifactin image and Docker container or any container actually could be an implementation of a ciart effect. But you could actually package. Any file could be a readme.md file. It could be a kubernetes manifest.

A

It could be customized overlays, it could be other files and, if anything, actually, that you are packaging. One of the implementation as well is M chart a name chart is an oci, artifact or image.

A

There is a multiple of advantages here. It could be generic again any files, it's portable, so it's a package that you could port and share across your company uh it's efficient and fast, because we will see that oci artifact could be stored in an Ocio registry. So if you are familiar with Docker pool on the container registry, that's the exact same way to do that you will be able to do oci, artifact pool.

A

So it's very granular on the security perspective, like you have with your container images as well as very efficient and fast, because I want this specific tar and archive and file and package right and like with container, if you think about attestation, verification, cosine, six store and other tools out there to sign and attest your your container images, you could actually do that with oci artifact. So that's very convenient.

A

Config sync is one of the tool um who is familiar with config sync I'm curious yeah, one person perfect, um who is using Argo CD on the floor, flexity yeah, so more for Argo CD than Flex CD. But here you could take any github's tool and engine here we are illustrating with config sync: it's an open source project from Google, announce lascubecon in Europe, Barcelona or Valencia Valencia, and the goal is I. Have an engine I have a component right.

A

A github's controller within one cluster of a fleet of clusters and I could configure different reconciliation from different repositories. Subfolder branches, Etc right. So that's. This mechanism did sink and git pull from a git repository. Config sync, like others, are doing: The Last Mile hydration. If you are doing customize overlay right, you are not it's not mandatory to hydrate your final manifest. You could directly point to your customization.tml file.

A

What we are exposing and illustrating actually today, those two new adapter and controller, where you could sync directly from an end chart exposing an M registry and same for oci registry, where you could pull any oci artifact. It could be also a customized overlay. We are doing The Last, Mile hydration, for you, it's not mandatory, but like mini talks mentioned that today, complementary tool could be gatekeeper, for example, or kiverno, who is using GateKeeper yeah kivano yeah, quite interesting. So here we will use straight gatekeeper. So it's not mandatory. It's not part of the product.

A

It's another project um based on OPM cncf project, but here you could add more security and policies. So you want to make sure that um your manifest respect, conformance compliance and security and I will illustrate that here it's an admission controller in kubernetes and you could actually have rules denying or accepting any request to the kubernetes API server. I want to deploy this deployment service account or whatever object. Please check that it's compliant with the rules and my own policies.

A

So it's typically a constraint template with a Rigo, Opa Rigo with a schema, and then you could Implement and extend with constraint.

A

So what we want to do here is doing the demo two demos. First, one is illustrating how to put all of that together right. um So what I want to show you is as security admin. What I want to do is designing implementing my gatekeeper policies right manifest file. I want to actually push them in a registry as an oci artifact, very generic, and the goal with that is to share that more broadly. In my organization, people could prove them locally, but they could also deploy them in kubernetes cluster across my Organization, for example.

A

So as a security admin, I will be able to to do that. So let me show you here.

A

Again, the repo the demo will are available. That's public repository that you could download and see um via the slide I shared, but what I have here is in this repo. It could be in another repo as a security admin I will Define um some constraint and constraint. Template actually gatekeeper policies here, I have two, but you could have many more pre-village container. I, don't want that people run privileged containers right, so I'm writing a policies.

A

A constraint for that and here um I could have this template that you could see here where I have my Opa Rigo policy, algorithm right and I'm double checking that we shouldn't have anyone running in privilege, right, I'm, looking at the Pod level or deployment level as well.

A

That's how I do that if you are familiar with gatekeeper as a security admin, what I could do as well, which is very convenient, is doing some test case and unit tests on my constraint right. So here we are leveraging as a test the notion of tests from gatekeeper again as a security admin I should make sure about the edge cases. What's working denied or not so, I have a diesel load example and a load example.

A

I should have more coverage here, but that's an example and why I'm showing you that is because actually I could have in my workflow, so here I'm using GitHub action right, but because up any tool you are using out the Azure devops, Jenkins and others, and the goal here is pretty much running some bash script right and pretty simple steps.

A

First, steps is getting the source here: I'm using Gator I, don't know if you're familiar with Gator, it's a CLI, a tool within the gatekeeper project, so open source project, and you could actually run some command CLI command and evaluating your test, for example, in this case, and the goal here is to run Gator verify so I'm verifying the suite.tml file I designed to do my unit test and that's part of my CI pipeline right and if it's working you need test or working I could go. Actually.

A

Oh sorry about that.

A

Yeah perfect, um so what I want to do is right. After if my test or successful what I want to do is, in this case actually log into GitHub container registry right. So I could log in here to this repository registry activity. So you could have any container registry out there. Artifact registry g-frog, artifactory Azure container registry same for AWS, same frog, Google and GitHub container registry. In this case, or not just to store your containers images as well.

A

They could store and host your oci images right and that's what we will do so here, I'm login with a tool I, don't know if you're familiar with oras oras is a CLI tool based on the it's a based on the Linux Foundation project and it's again to implement the interaction with our effect based on the open container initiative and this tool allowed me to log in on a specific oci registry and look at that. What I'm able to do just right after I'm hydrating.

A

My policies in my case I'm using customize technical detail here, I'm archiving this file, for example. But look at that here, I'm doing auras push like I will do Docker push right here. It's for generic oci artifact, so I'm pushing that in my GitHub container registry right and, if I look at that here. So actually, so there is a graph of the two jobs you just saw in GitHub. Action and part of the demo is actually now anyone who has access. So, in my case, it's a public registry. It should be a private registry.

A

I have a link for that, but here anyone could do oras pool like you will do Docker pool. So here you could say I want to grab policies of my company, so security policies of gatekeeper defined by the security admin, so I could do aura's pool having the Manifest and play with that and actually um see how I could evaluate them locally.

A

On my manifest, for example- and we will illustrate that in in a few moment so next step here is for con fixing, but pick up any definition for, and translation for flux or Argo CD. We are using an object to Define actually for this cluster I want to pull and deploy these policies so I'm defining this object, pointing here like you, could see, I want to deploy a source type oci and pointing to this image right. I just pushed I'm using authentication. None like I mentioned it's a public registry.

A

It should be a private registry and there is a mechanism to secure your access from your cluster in order for conflicting, so github's engine. In your case, to pull this ocrt fact and if I look after that, if I do here, a cube, CTL apply right, I will be able to see here that actually my configuration, um my oci artifact containing my policies are pulled and if I look here, I could see that I have all the constraint and constraint template defined in my artifact. So now they are deploy.

A

If I look at the constraint, so Cube CTL get constraint or constraint template on my cluster, they are now deployed and they could enforce. My cluster I have zero violation in this case, but now I'm enforcing compliance and security. So that's the first part of the demonstration.

A

So here what we just saw is this part right as a security, I mean I'm, pushing an oci artifact wrapping up and packaging. My oci, my gatekeeper policies. Actually we add clusters, for example, already provision with config sync and opa gatekeeper for admission controller and what we just Illustrated a pull mechanism, a deployment mechanism via githubs. So that's the first step. First demo end to end now. What I want to do is: let's do that with an M shot, which is again an implementation of oci artifact.

A

So as an apps operator, I want to define the Manifest to deploy my app with M chart. So that's something I want to do so. I will have my own repository. I will push my M chart in GitHub container registry, but again pick up any oci registry out there. But what we want to illustrate as well is during the CI pipeline, evaluating the policies designed by the security admin. I won't wait to deploy at the end of the day my M chart to see if it's compliant.

A

What I want to do is again shifting left the evaluation of gatekeeper policies as soon as I can right and not waiting to deploy this m chart in a kubernetes cluster. So let's see that in action.

A

So here when I want to show you it's a pier, already open and typically here what I'm doing as an apps operator, I want to deploy my app and here I'm, looking at some failing failure issue during my peer review, pull request review and why? Because actually, the apps operator is changing some information in my M chart and, for example, we could see that we are not using any more an unprivileged container. We are using this one here, which is not a good practice and look at that.

A

The apps operator is removing all the security context right, so they are removing all of that. So it's less secure and like we saw earlier, the security admin has policies for not accepting unprivileged privileged sorry container right. So here it's bad and before actually waiting for this m shot to be deployed on a kubernetes cluster.

A

Here we are using actually a gator to evaluate this policy within my pipeline again I'm not waiting, and here we could see that, for example, um I'm a requiring security context and run as non-root, which is best bad practice and I'm stopping the deployment here. What we are using as well is trivi TV could scan so from aquasec could scan container images, but they could scan as well kubernetes manifest. So here there is an illustration of scanning. Actually your kubernetes manifest, which is more Security check.

A

Here we are using a kind cluster where we deploy this m chart and trying to see if there is a smoke test to see. If we are failing fast here- um and we could see that there is logs and error right, so here I'm blocking as soon as I can shifting left some security guard rail here.

A

um So I won't go in detail about the the CI pipeline. Here but again it's a lot of check, gatekeeper or trivia, scan and others that you could look at and actually reuse, because it's bash script at the end of the day. But let's say how the tests are working right like we saw earlier oros, login or raspool, we have the same with M right.

A

So, if you're familiar with M three steps once the tests are successful and package, so please package, my my chart with a specific version tag actually Elm login, again I'm doing a login on against GitHub container registry and then it's just an N push right. So a very, very helpful and interesting here and same scenario here: if I go here, I could see the flow of my jobs- it's maybe too small, but you have access to that um to see more in detail. What I could do again is N Pull of this artifact.

A

It's a public, artifact and image M chart, but again it should be a a private one. I have access to it and again I could deploy via this configuration and this deployment Cube CTL applied a chef on this object. Please deploy this m image. Actually right so the repo the name, the version read his name perfect. Thank you um and yeah.

A

That's how I will Define now the deployment of this m chart right so I could see this detail about what was deployed, in our case, very simple application with a namespace, a deployment, a service and service account very basic M chart here and yeah I have access to my chart. It's all deployed right. So that's closing the demo part. Actually here and again.

A

First demo was about security admin, designing coding, their own security having their own CIA pipeline, exposing their um gatekeeper policies as oci artifact and then having config sync as a GitHub controller, pulling and deploying these policies across the cluster. If many clusters- and the second part was about apps operator having the shifting left enforcement grabbing, the policies in this CI pipeline here before actually exposing the M chart in container in oci registry and then pulling to the m chart to actually deploy this m chart.

A

So some takeaways here some of the question we got and flex is supporting this new oci format, as well as the git format, go CD as a plugin made by Reddit. If I'm not mistaking here. So we could see the evolution of yet another option to do githubs with your manifest.

A

Yes, git is still a valuable approach, but maybe for some consideration again, we mentioned at the beginning: fine granular access and security performance, maybe at scale oci artifact could be another option for your github's github's scenario, and there is maybe other consideration for again being more performant, portable, etc, etc. So, even if you are using oci artifact, are we getting rid of git and do we still have git as a source of truth?

A

Yes, for sure, because we need all these CI part right, um where all the Manifest or in there and the source of Truth, what we are doing is having maybe another step where the source of Truth for the package you want to deploy right is in is in an oci registry instead of a git repository, but we still need git and what what we have Illustrated as well here is Shifting left again more in the c um in the git and CI pipeline.

A

So that's pretty much it for um all the content about this presentation, the demo there is links here, um the setup and the demo, so you could replicate that on your own cluster I'm using config sync, it's an open source project that you could Deploy on any kubernetes cluster, even kind um and gatekeeper, open source project as well.

A

um I have the link of the project of conflicts Inc and the two Links at the bottom, with the workflow or for the secure manner for pushing your oci, artifact and M chart on a secure, GitHub, container registry or Google artifact registry, as well as pulling them securely from your clusters, right links with tutorial Associated to that and QR code to download this presentation. Thank you. I hope you enjoy this content and what I would love is maybe taking some question if I have time, yeah I don't have the mic yeah.

A

B

So my question is: regarding the performance: uh can you elaborate how oci is more performant and yeah.

A

Good question yeah. um Thank you! um So what you could see at scale when you have multiple clusters, multiple repositories, any githubs controller um by Design, we'll do a git sync. So if I could just summarize very roughly a git clone a git pull on a specific repository Branch to be able, for example, to grab a folder right. So that's how git protocol is working. So as at some point, let's say you have thousands of repository configuration.

A

All of them will use CPU memory in order to be able to grab a granular or specific folder of files right and so at scale. You may be. You may see some performance issue on this perspective right on your githubs controller. It works for confixing, Argo, CD and flexity out there and now with oci artifact. You are just doing. Please give me this m chart or this ocrt fact right.

A

So it's more efficient, more performant, because more granular I hope my answer makes sense, and maybe you have seen at scale some of you, such maybe performance limitation. Does that answer your question yeah correct.

C

Hi um this is kind of tangential, but on topic, because you've shown gatekeeper twice today, I was interested in understanding how you enforce updated security policies on existing deployments.

C

Is that, like uh do you just invalidate all deployments? Yeah.

A

Good question: sorry about that: I didn't spend too much time on this gatekeeper concept. Are you familiar with gatekeeper at all yeah, so the beauty of gatekeeper is you could evaluate at time T right, but actually it's a continuous evaluation right. So every I don't know five. Second, you could configure that. Please evaluate at this time now within my cluster, for example. Right.

A

So that's how you will if security policies are changing or deployment or object or deploy not just with githubs but cubectl, you will be able to have this admission controller, checking regularly and auditing your policies. Does that answer your question.

A

So it will reject a good question, so the question is: is it rejecting current workload already deployed? The answer is no you're right. It's for admission right, so the new deployment, but what it will do is actually giving you events and violation information. So, even if it's not rejecting your current workload, it will tell you and with that you have a reporting tool trigger notification to in order to get fixed as the issue here so good question. Yeah.

D

uh I just wanted to add on to that previous question, so you can actually run gatekeeper in auditing mode in auditing mode to start with, because if you actually send it, if you actually start off with enforce mode any new application you try to deploy, will get blocked and that's going to cause issues for you.

D

um One way for you to look at it or to share it with your entire organizations. You can actually build like grafana dashboards, to check out if you which policy which workloads are actually violating. So that's a good way to start off.

A

Yeah good good, thank you! I! Don't have it here, but in my constraint to illustrate that you have an enforcement action attribute, it could be audit or um blocking or dry run or Etc, and you could play with that as well. Yeah good good, chord, yeah.

E

Another gatekeeper question, so in your gatekeeper demo, you had on the one side a security person saying: let's deploy this version of my OCR artifact and on this hand you had a developer in their pipeline. Saying check my stuff against this version of an oci artifact yeah. How do you make sure the versions are in sync across multiple clusters and pipelines is.

A

There an idea about that so good question and it could be a longer answer and the discussion here, but you're right in my case, it's a mono repo I have the policies in the same repo as my app. So it was like checking for this specific version, but actually you remember the oras pool Command right, so you could inject this in different Pipeline with let's say latest version of my policies or oraspur, the latest version, or it could be a specific version.

A

So you will need to track that and deal with that at scale and in your thousand pipeline how you deal with the version. I could see what you what you mean with that so yeah the there is some consideration to have here, but something to that could be a reserved as well at scale if I could say, because the goal also with pipeline Jenkins and others, its definition as code for your pipeline right. So maybe you could automate some stuff at scale for the version, etc, etc.

A

Good question any other question.

E

Thank you, Matthew.

A

Thank you very much and I hope you enjoyed the session.