Continuous Delivery Foundation cdCon 2022, 24 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Trust but Verify - Olivier Vernin, Suse

Description

For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/

Trust but Verify - Olivier Vernin, Suse

Kubernetes revolutionized CI infrastructure. It’s easier than ever to replicate production environments for testing purposes, deploying preview environments, relying on auto-scaling to optimize costs, etc. Kubernetes hides complex automation concepts behind APIs so developers don’t require extensive infrastructure knowledge. But as great as all of this is, it didn’t get rid of human errors. It’s easier than ever to expose your CI in the wrong location or let someone in by using the wrong corrupted container image. Let’s discuss which projects we could benefit from to let developers do what they do best, build software.

A

Hello does it work, yes sounds. I think we are ready to start so I'm gonna take this one so today, um so thanks for joining this presentation about just but verify my name is olivier: I'm working first user as an engineering manager um on the rancho side. So I'm working a lot on all things related to containers you can reach out on twitter. If you want most of my work is open source, so you can follow.

A

Also my work on github, quick disclaimer, I'm not a native speaker, so I may be using the wrong wording during my presentation feel free to come after the presentation to correct me if I'm wrong on something.

A

So what I want to talk today is the right level of trust, because trust is a broad term.

A

It can mean different things to different people and in my case I spend a lot of time working on the jenkins project, so more specifically on the jenkins infrastructure projects. So it's an open source project with a lot of contributors, but the main difference is we don't build software?

A

We build all the tooling for releasing jenkins retrieving documentation for the jenkins project, so everybody is invited to contribute to it, but the thing is: we need to trust each others and the work that we are doing on this, because you have a lot of community members everywhere in the world, are contributing to the projects and so trusting what you work on can be really challenging, and so the first thing is, I want to talk, is trusting the system that we rely on um the jenkins project received a lot of sponsors are sponsoring, so we have a lot of compute different clusters, communities, cluster virtual machine and so on, and what we learn is when you have to operate large large amount of machines, something that can work one day will not necessarily work after the day.

A

After the reason for that is you, may I mean the good situation? Is you just have more travel? I mean your project is successful. You have more people using your services, you have just more people using your services on the other side, you may be affected by your security issues or whatever. So you have to you always have to to consider that your your infrastructure that you rely on is not trustable at 100 person, and you want to put in place practices and policies. So you reduce the risk of seeing your system going down.

A

So I'm going back to this one a bit later, the next. The next things that I find challenging, um and it's not nothing specific to the jenkins project, but it can be to any project of any size, is the amount of technology that you need to work on, and so you first need to be confident in yourself in your own limits in your own abilities to change something. But at the same time you also need to be aware that you don't have the time to master all the technologies.

A

You don't know everything, and sooner than later you need to reach out to those other people or once again put in place practices policies, rules that can help. You are reducing the risk of introducing mistakes.

A

The the the final challenge I found when you want to trust is the other people, especially when you work on a large community project like the jenkins project, because on one on one side you think. Okay, all those contributors can bring something that you didn't think about at the beginning of the project, because those people have issues that they want to solve. They're all really helpful, but on those and on the other side, those people can introduce mistakes because they don't know everything like you.

A

Do they don't have your experience like you do, and so you need to find the right balance between giving your trust in the people, so they can achieve things and, on the other side, protecting the rest of the group, because once again we all do mistakes, and so for the little story about this picture is while working on the jenkins project.

A

I would regularly have people reaching out to me asking okay, they want to maintain a specific service, and could they can they have, for example, the ssh keys to the machines, for example, or can they have the jenkins code signing certificate because they want to experiment with it and each time someone would come to me asking that kind of questions, obviously for a cod signing certificate or the jenkins that I owe a certificate.

A

It's just a no-go, but you need to understand what that person wants to do, what that person wants to bring to the project, and maybe you can find a bit of ground and just what you're trying to do here, you're just trying to reduce the risk that you have when you collaborate with the people.

A

So what I want to say here is when you're responsible for the full lifecycle of a project. You have risk everywhere and you want to put in place policies, practices and so on that can help you to mitigate that risk and so to increase the confidence that you have in your projects, um in your infrastructure and in yourself. So in. If, like me, you don't necessarily trust your project re, that's the system, you rely on because yeah we don't. We never know what what can go wrong.

A

um Don't have the time the time to master everything, because sooner than later you have to work on a technology stack that you don't necessarily master or maybe you just have to work on all legacy, and you forgot about those but yeah. It's you need to be pretty confident about your own limits. And, finally, if you prefer to empower people around you to do the work because they may have that complementary skills that you don't have, um then I want to talk about um and introduce policies.

A

So a policy is a contract that you can have between machines between you. So we are all talking about the same thing and when it comes to collaboration, you may I mean you, have these different to two different ways to see the people who work in a specific project.

A

So I'm not talking about how you should organize your team, but you usually have the deaf people we'll just try to introduce changes, um build that new feature that new shiny things that does not exist yet and on the other side of the spectrum, you have those people responsible to to run your infrastructure in prediction and once again trusting infrastructure can be challenging, and so, if you can introduce as little change as possible, that's just better and so several years ago, for me not more more than several years ago, we started to see a shift where now um the two teams, the two objectives- are merged, because in the end, what we want is we want to deliver value, but the two those two group of people can work together and rely a lot now on containers because it allows you to have reproducible immutable environments on every everywhere.

A

So from the dev machine, for example, um you could have a communities cluster. So it's just an api running. I mean just an api on top on an infrastructure, and so there are a lot of tools that you can have on your machine um and on, for example, like wrench or desktop or key 3d or whatever other communities engine you want to use and on the other side you have um you use, managed communities, clusters, you have permitted communities, clusters and so on, but the good thing is.

A

We now have a foundation, an api that we can all talk about the same thing. This means that we can easily put in place policies that will affect the prediction: cluster, the ci cluster or even your local clusters, because well.

A

We don't we don't necessarily have the same experience. So what I want to talk here is um the different kind of policies that you can have on communities that can use across the different teams. So you obviously have the first one, which is the rule based access control, so you just authorize who has access to the control or to the cluster and who doesn't have in terms of collaboration? All it says is some people can break the cluster, orders counts, but that's it.

A

What I want to really insist is the next three, so the first one, so the pod security policy so put security policy is a specific policy that you put on your pod on your communities cluster, and what you want to say is you want to enforce some good practices, so one of those good practices is, you should not be running a container with a root user in a privileged environment. um That's something that many people show that it's a really bad practice, and still we can do that by default and so the pod security policy.

A

We just assumed that we are not all experts at all the different ways to exit a container, but just by enforcing some default policies, we can reduce the risk of breaking your infrastructure once in once, you are in prediction: um the network policy is the same, except that this time we don't have to be a network engineer. We can just rely on other people, experience to say: okay, we can enforce some policies, so we reduce the risk of breaking our clusters.

A

But what I want to- and what I want to talk now- is the admission controller. So the admission controller is, as they explained, on the communities, the documentation, it's a piece of code that can list an event going to the cuban cities api, and then you can run some validation on those events. You can run some validations saying that the event is fine. Yes, no, but you can also mutate an event.

A

A mutate means that you receive an event, you're, not happy with what the event is, and then you just change it a little bit, so you can use that event and proceed, and so one of the projects that my team is working on is cube warden. So it's a policy engine for communities, the purpose of keyboard and is to be to have simple policies um in place. So I'm going to show you what it looks like. So this is a simple kubernetes.

A

um It's a simple yammer resources like if you're familiar with um the way you configure your community's resources. You have custom resource definitions, so it's a keyboard and one you have in this case the kind so they can define. Okay, this policy should be applied to the world cluster or you can also have just admission policy if you want to apply a policy to an to a specific namespace again, a policy is a contract between the people. So what is really nice when you can have that kind of policies? Is you don't necessarily you?

A

Don't necessarily only have security in mind. You can also focus on the collaboration between the different people, so it's a simple admission policy: now you have the specs um in the spec. The first line is a module, the the way cube, warden handle policies. The way you can have different kind of policies is using different, ocr, artifacts, so keyboard and generate. When you build a policy, you generate a webassembly binary and then you distribute that web that webassembly binary on oci registry. Other people can just reuse this one.

A

So as a policy maintainer, all you need to do is just use the language that supports oci binaries so, for example, rust, co-line or um or swift. I mean there are a few others and the thing is, you can just reuse the programming language that you want to use. You distribute it on a cr registry and other people can start using your keyboard and policy.

A

Then you have the rules. um The rules is, as I said, you have an event going to the api to the communities api, and then you have the admission controller that catch that event, and so in this case we want to to detect. Okay. When are we creating or updating a pods in our, for example, cluster, and the policy that we need to apply, is the one explained in the settings so I'm going to come back, then you have the mutating, so we can mutate an event depending on the policy and finally, we have settings.

A

So in this case the policy says you should reject any container using the tag latest, because once again, we know that it's a bad practice to use the tag latest, especially in your ci environment. Because then you have no idea about what you are using, because the latest tag means different thing. If you're on your machine in the ci cluster or in production, so we could just enforce some rule, um so the people who are not necessarily familiar with this rule um can will not be affected by this one.

A

um As I mentioned, there are other policies, so this one was just to to reject a container based on tag based on the repository and and so on.

A

If we are looking for other policies, we can go to up dot, cube order, io and there we have, for example, this one is a verify image signature. So we want to be sure that the the container running the container is assigned using six store, and so we have the policy. We have free information on this page so that whenever, of course, you have the title in the description, but at the bottom right you have three econs. The the green shields mean that this policy has been signed using six store.

A

The red pencils mean that this policy support by uh mutating, so you can, um for example, if someone is not familiar with the policy, you can still let that person uh do the thing you want to do, but just dropping what you consider to be about, but by practice. And finally, you have the ev button. That tells you if it validates the event or can reject the event.

A

So the good thing, the the the what is great in this way is as a developer. You you can put you can put um as the as developer. You can put all the complexity that you want in your policy, so you can use the language that you want and you can reuse your id. You can reuse your practices in the team and then you can just expose a simple configuration that people can use, for example, using the yaml file.

A

So in the end it's just a web assembly binary, you combine, you compare it to web assembly, binary, distribute it on a ci registry and and job done so you can see webassembly binary as that universal way to build and ship binaries.

A

So if you want to see what keyboard and looks like um well, it's pretty.

A

Let me show you I'm going to change my screen, so I can do a yellow.

A

Sounds like you see my same thing I mean so.

A

Okay, great, so I have a local, um I mean I'm using k3d in this case.

A

Okay, three, so I have a cluster running on my machine, keeps it I'll get a mission.

A

So I installed three policies by default, so um they were just um yeah, just charts templates, so those are just three basic policies. So what what I want to show is from a community's point of view, I can just enforce some common practices like, for example, not running a container using the tag latest and on the other side, from oops.

A

From my jenkins instance, so this is a simple jenkins instance, so I'm not configured with git configuration nothing else just like standalone, because I didn't know what internet connection would looks like. So what we do here is, if I go to the pipeline configuration.

A

All we do is we trigger we, which we want to trigger a job running on on a cluster using the maven latest, and what we want to do here is we want to reject this container. um What is interesting with policy policies? It's not only about, for example, um enforcing some security practices. You can also, for example, enforce some good practice in the labels.

A

So, for example, if you want to use environment using development or prediction, you can do a lot of things with the policy, because, in the end, what you want is you want to increase the collaboration between the different person, and so, if you run this or I can just reduce an existing one that I was trying this morning, so you see that the ci job will just be rejected, because we we we don't want to be using the container. um What is this?

A

We don't want to be using the tag latest, so import that tag later, so the feedback loop is pretty quick, and so now we can go back to the developer and say: okay, it sounds like you were trying to do something bad because um you don't know which version of maven you're trying to use here. You don't know which version of java you have in the container maven, so you really want to use the version and so yeah.

A

I just run the job and it's here so once again we do have so I don't have right now data on this one, because I misconfigured something. But what I want to mention is you born and exports the matrix to a graph and to a prompt you so that, then you can just build a dashboard that you want and- and we also start working on an integration within rancher, so so yeah. There is no no really shiny because thing when you look into policy engine because all you want to do.

A

Is you want to validate or reject an event.

A

So yeah, what I want to say here is that's fine to not master everything. We can all leverage other people experience. So by just using policies we can just um yeah. We can. We all do mistakes and yeah using policies is not that difficult thanks. That's it for me. If you have any questions on the kuborum project, you can just go on communities under communities slack and then go back and go for the keyboard and channel.

A

Thanks any question so far.

A

One question on the right.

A

So I have, I don't have advanced experience, for example, for kievano or gatekeeper um gatekeeper, for example, rely on rego. So you have, I mean, depending on the the rules that you need to put in place. Rigo can be difficult. It's a new dsl that you have to learn, so that was pretty much the first one when people started using gatekeeper, for example, you start you also see kyvano was introduced. It's a second policy engine that you can use where you configure everything from the yaml.

A

What is interesting with cube warden is, depending on the complexity that you want to have. You can just reuse, a programming language that you want to use. So if you have a very, very complex policies that you need to build for your custom resource definition, using keyboard and just building using, uh for example, again I'll go line, or so you just release a language that you're familiar with and and and what you know. Otherwise, just as a user, you can just use simple yaml files.

A

Any other question.

A

Sounds good thanks for listening and uh yeah before I leave, I have a last minute presentation on a topic that I like, which is dependency management and a project that I started, and so it will be in the room. Next to this one, at 11, 30.