Cloud Native Computing Foundation Helm Summit 2019, 4 Oct 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Increasing Reliability via Helm Pre Release Checks Bridget Kromhout, Microsoft

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Increasing Reliability via Helm Pre Release Checks Bridget Kromhout, Microsoft

Join us for KubeCon + CloudNativeCon in San Diego, November 18 - 21. Learn more at http://bit.ly/2lmMzip. The conference features presentations from developers and end-users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Okay, hi I'm Bridget and we're gonna talk a little bit about increasing reliability via Helen pre-release checks and the TLDR on me is I, live in the US I'm very much hoping Canada will an access soon, because I'm in the middle in the north of the US Minnesota people basically think we're Canada and I'm like yes, please, especially depending on how things go politically in the US I work for Microsoft.

A

If I don't look familiar but I sound familiar, you may have heard me on the podcast arrested, DevOps and I also helped lead the DevOps news organization worldwide. So I know you have a great DevOps days here in Amsterdam and if you're looking at me and thinking wait a minute if you're thinking where in the world is Lackey like shouldn't, he be here giving this talk and also we thought the talk was gonna be slightly different, but it was definitely not going to be whoever this is. The problem is teleportation has not been invented.

A

Yet, as far as I know, if you have a teleportation startup, please let me know I would like to invest in it because turns out. Airplanes are terrible and take way too long, and so lucky was actually on stage at hashey conf giving a keynote yesterday, and that would be very difficult for him to also be here so, but he did write most of this talk, which means I, will definitely be looking at notes. Slightly more than I usually do and yeah I hope that this will still give you some of the content.

A

You were looking for. The other thing that's slightly different than what the talk was originally pitched, as is it was going to be all about the failure detection that would have been feasible if fluo had been something that landed in helm, v3 and whole community was like. We would like our helm, v3 sooner and not forever into the future and so yeah. Fortunately, there are exciting open-source tools in this ecosystem that will still help with the pre-release failure checks and, if you're, if you're thinking like okay, what exactly are you talking about?

A

I think it can pretty much be summed up by there are yes, this is the vasa. If you are either Swedish or Danish, you may have opinions about this from one way or the other I highly recommend everyone go to the Vasa Museum Stockholm. It's delightful and I think that the the takeaway here is that no matter how much you plan, if there's scope, creep and lack of documentation, things might not turn out. Okay and I think that it's gonna be I got the bad news.

A

Is it's gonna, be harder and harder to have successful, Hum releases just because kubernetes is really complex and is not getting simpler, and so that's why these pre-release checks I, think, are going to be really valuable, all right and so we're gonna talk about a few.

A

These think anything here's the agenda and if you're, like oh I'm, already an expert and all those tools, then you should definitely go to a different session at this moment and I will not be offended but and if you're on that side, you can actually sneak down the hallway and into the other session, for whatever reason, I'm not sure why, anyway, but we're gonna talk about these open source tools, because this is basically kind of the your belt and suspenders of getting your releases out there without you know waiting for your CI CD pipeline and then, like you, know, 40 minutes later finding out that you have an error and you're really sad.

A

Alright. So, though, we're gonna we're gonna demo. All of these I'm not gonna, do live demos actually, but we're going to look at all of these from the perspective of say, you're installing the stable nginx ingress chart and it's a lock ease, opinion and I think he's right that that's a good choice, because at least at one point it was the most downloaded chart like upstream, like stream chart.

A

So it's a one that most people will at least have some familiarity with okay, so the the first thing we're gonna look at is resourceful idit ii like there's a there's, a few ways that this could not turn out well. I feel like this is such a downer for talks, I'm, just telling you everything that could go wrong and honestly, like hashtag ops life when you've spent all of your time worrying about worst-case scenarios.

A

After a while, your spouse sits there and says: why are you so pessimistic and I'm like no I'm just realistic, so we're gonna talk about the things that can go wrong really says it's like home really says, can fail with, like your resources are invalid, and that can mean a couple. That's already big us, because that could mean, like you, have typos or errors that are leading to invalid kubernetes resources, or it could just mean, like the resources, don't work on a given kubernetes version.

A

So let's look at examples of both of those say: you have an invalid kubernetes resource and this one is you're sitting here.

A

Looking at this wallet, acts being like this is really annoying, because the reason why this is invalid is in here somewhere, but it's not necessarily easy to tell why I'm guessing that we have experts in the room who can look at this and say exactly what's wrong with this, but if you're not sure what's wrong with it, you are in good company, because good lord you're, looking at that, going what Revie bigger context default, speckle you're like did she just type random things?

A

No, this is an actual error message, it's like sorry and then or your resources might not work on a specific kubernetes version, and so this is again like it's frustrating. Because do you really want, like the giant, you know matrix of what will work with what and that just ends up being frustrating. This is actually I, think and lucky was telling me and I think that he's right that we're gonna, probably like spoiler alert prediction into the future.

A

We're gonna probably start seeing this kind of error a lot more because there's a bunch of api's that are being deprecated as of 116, which is coming out. I want to say Monday: does that sound right like Cooper days once his team's gonna be out very soon, and so, if there are API endpoints that you're using your your releases might not work anymore with a new kubernetes version all right. So there is help for this and at gareth rush grove.

A

If you're familiar with him, he lives in cambridge and he is wonderful and you should definitely follow him on the Twitter's and all of the other everythings and I. He. You can install this plugin, the coop eval plug-in if you're not familiar with this one. What it basically does is a little bit of pre-release checking to see is this stuff even going to work? I mean it's not going to guarantee to you that everything you want to do will work, but it will at least find some of those common errors right off the bat.

A

So, like here's that same invalid deployment, anybody find it to be any easier to see what's wrong with this invalid deployment. Just shout it out if you see what's wrong yeah now that all of that, like alphabet soup, we had before could have just said this, but it didn't pull request accepted, but you can use coupon and notice how kuba Val is just a drop in for your install right there. So, like you, can use coupe of Al before you try to install something to see if there's going to be any problems with it.

A

So that's an example and another way to look at it is to see. Will this chart work with a specific version now this one because 116 isn't out yet so we can't show you all of the errors that will come out, but you can kind of try this in this case it's not broken, but you can try this and see how broken your choice will be in the future I'm just a ray of sunshine.

A

Here, it's like and so yeah as of 9 16 or, as you would say, in the reasonable parts of the world, 16:9 Monday, the kubernetes 116 will be out. I, don't know if it was the matically named that way or if they picked the 16th, because that probably not but.

B

A

It pulls the API schemas like for the version and checks them so yeah. So that's your that's your resource validity in a nutshell and yeah I don't have infinite time, so we're gonna skip on to policy everyone's favorite. Well, everyone's security people's favorite anyway.

A

So if you would like to coober all the Nettie's and have it be super exciting and not have security, say no to you, then you can show them all of the great policy enforcement that it has now, and this is like the specific configuration on clusters to prevent resources from being created that shouldn't be created.

A

You can enforce this with tools like pod security policy, gatekeeper, what your the kubernetes like built-in policy stuff that you're looking at that you care about is it's basically open policy agent, it's the OPA stuff and there's OPA can be enforced server side with pod security policy gatekeeper.

A

What have you there's a number of tools, but again that is the the server will reject you're, not compliance stuff after you run through whatever your process is with all of the stamps of approval from you know the security people and the gating and the CIC infrastructure and the the two weeks with the change control, Review Board, where your llamo goes, I don't know, but these are all things that could happen, hopefully not to you, but if it happens to your friends, tell them about contest.

A

So this again is from Gareth rush Grove. He is again quite awesome, I kind of wish you were here. He is somewhere else. Gareth, please be at the next one of these and there he has a blog post there that you can go read because he just introduced this this summer, and this is again local testing. So it's using the the same. You know Rhaego stuff, that the open policy agent and everything else is using, but it does local testing. So again, you can see.

A

Is my chart going to fail, hopefully before I waste a bunch of time and it if you're not familiar with reg? Oh by the way I mean you can you can your Google is as good as my Google, but basically structured policy?

A

Okay, and this is an example of a contest you know showing what might fail if you're outside of policy and so like again the policy could be. Where is our policy here is a MacGuffin. We don't really care how the policy is set. The point is, we can see what the policy is before we attempt to do this release. Can somebody tell me why this particular release might fail.

A

Exactly so, I don't know about you, but I have been told that the cloud is infinite, that that might not be true.

A

We can check on that, but it's possible that you're in your employer's care about how much your cloud bills are, and so, if you have the possibility of making sure that every release is going to be scaled to, you know exactly what can be run inside of the infrastructure that you've provisioned, then you might prevent auto scaling to infinity, which would I suppose be delightful short term for the cloud providers and way less delightful when somebody's bill comes to.

A

So this is how you can make sure that if you set policy is such that you can only launch this much without consulting someone to get the limits raised. But then you don't want to your stuff to fail, just because you accident on purpose exceeded that limit, okay and then an example of what it would look like when succeeding. This is us doing that same the same release or you know the same launch attempt with the explicitly limits set and we can see it comp test isn't going to air out if it's explicitly set.

A

So again, this is a simple tool. It's very much in the UNIX philosophy of it does one thing it does it well and I like that. So contest is one to look at, and the third thing I want to talk about our back role based access control. It's not we're not going to talk about a third party open source tool here, we're just going to talk about something, that's built into kubernetes for this, and this is basically a lot of what Fisher was just showing us, which is like the TLDR for our back.

A

It seems very obvious but like if you don't have the permissions to do something all right. Your deployment will fail and- and this is I think the thing that we have to pay attention to from the helm. Three point of view: like spoiler alert, helm three does not have tiller, and so, if you kind of run down your dependency tree in your mind, you realize hum three not having tiller means that tiller god mode where you're just kind of opened everything up and you load it no longer works.

A

So anything you're trying to do if you're kind of going back in your mind, you're like oh right, we just kind of depended on the fact that tiller could do everything and then I was fine yeah! No, no, that's gonna work anymore. So you'll have to pay a lot more attention to our back. Then you had to before just because you can't really assume that as long as it's you know in the tiller namespace, it's all good like no more telling namespace, so you actually have to.

A

Unless, of course, you do the thing that you know we we don't want all your base to belong to. Unless we do that, just open everything up people tend to object to seventh 7/7 mode on everything, for reasons possibly to do with you know: data integrity or security or not having everything be terrifying, but okay. So here's an example- and this is again, this is not a. This is not an additional tool that you would add. This is just built into kubernetes.

A

Is you can use coop control off cam, I and locky whipped up this lovely for loop, which I love, because it's so easy to show you? You know slick vendor where, like market and copy about how everything is easy like when people tell you things are easy. They are lying like they're, not easy, but they are definitely comprehensible and what we can see here and I would I think it would be kind of interesting to point out. Actually this is just looking and seeing can I create.

A

You know, resources in any of these namespaces, and this is a user set up specifically to have some permissions and not others, and so you can see in the you can again. This is the kind of stuff that you would build this into your CI CD pipelines into your automation tools, so you can see at a glance like or you know, with something that would error or not as to whether or not alright, this user can in this. You know in this context, and this namespace can create deployments royals row bindings.

A

Oh can't do cluster rolls okay and then the ones that are showing up as not names key, not namespace, scoped haven't been defined, so that's another place where and when I was talking about this tuesday to some other fine folks here in amsterdam, and they were saying that they're, like that's a terrible error, I'm like it is kind of, isn't it open source.

A

I mean seriously like if it's a terrible error put a pull request in to make it a better air cuz like I look at that and I'm like warning eyes, blur fall asleep, okay, crawfish shrug so I, don't know, I I feel, like somebody probably could give this a much more succinct. If we wanted to give that warning, a yes or a no or a what's a one or two word summary that you would use to summarize what is going on they're just.

A

Undefined or check manual, you know so again like think about that when you're putting in pull requests in it's easy to criticize it's harder to be like. Oh, alright, I'm, actually gonna fix this okay, so you do by the way in the in the checking of our back stuff. You do have more than one option. I here, I just went through and showed coop control off. Can I but another good another good tool you could install this with crew is from aqua security, Liz, rice and company have a pretty good tool called.

A

Who can and I wanted to highlight this, because it's a subtle distinction, the can I. Is you run it as a user, and you see? Can that user do things or not the who? Can you run that to see who can do specific things solely depending on how you're asking the question you might want to use a different tool and the who can is not built into kubernetes, but you can go, get it from github. So it's a good example of like there's.

A

There's a lot of ways to do this, but just depending on how you're trying to solve the problem. So in summary, like alright great what now I would say like it's a good idea to put these kind of pre-release checks into your CI CD pipelines, because, especially because kubernetes keeps changing, and especially with the removal of tiller, a lot of the things that you might have been able to kind of hand wave security-wise, you can't handle anymore, so you better start building out the infrastructure to make it possible.

A

Those are the three tools that I recommend. Everybody at least take a look at and see if it solves any problems that they have, and then you prevent broken releases. I. Think the the way Lockheed put that when we were when we were putting notes together to talk about this is he said you don't have to live, that broken release. Life anymore and I was thinking that should be like a country and western song.

A

But you don't have to live that Brook and release life anymore, or at least you can get those failure messages much faster, which is what you prefer and then dot dot and profit is obviously what has to happen next after that, um so I guess what I would I would leave us with this. With this thought is the vasa this worship, this worship from the seventeenth century? It sank, it sink like I, don't know a kilometer or two after setting forth on its initial voyage.

A

So this was not good and they did kind of you know a little bit of when they pulled it up off the bottom of the harbor in the 20th century. They did a bit of a post-mortem to figure out what happened and the there was a lot going on in this. You know failed release. There was scope creep and there was you know conflicting requirements. They also had. It was very top-heavy. You can see the the intimidating carvings going up to the top of the ship.

A

Many carvings were added to make the ship more intimidating for the for the Swedes wanted to make the ship more intimidating to their Danish foes, which made it really top-heavy which made it.

A

You know he over and sink, um but probably the biggest problem was actually the fact that the ship right died during construction and, as was the style at the time there was nothing written down, and so his wife and his buddy like ran the shipyard and finished the boat sort of, and it didn't go super well and I like to think that we don't need all our four failed releases to be like the Vasa as exciting as it is to pull a bunch of wood up from you know the bottom of the sea floor hundreds of years later and find that it hasn't completely.

A

You know, melted away, I feel like the the really useful thing is. If we can get all of this stuff that we can do ad hoc that works into our CI CD pipelines into our configuration of code into anything that you want to call it. But just remember that it's not just getting your specific thing working!

A

It's taking that experience that you have from your specific thing and putting it in as an issue you know putting it in as I feel like it when people say I, don't want to contribute to open source that sounds challenging or that sounds difficult and I. Think well. I mean honestly. Some of the greatest contributions are just opening an opening, an issue on the docks, repo saying I, don't understand what's happening here and then fish are saying. Yes, I was planning on writing those docks like that.

A

That's actually how we solve this right, so I would just kind of say: here's a few links of some things to learn more about our team does I'm a I'm, a p.m. on the open source upstream team at Microsoft, so I do work on with the Deus labs folks. So if you can take a look they're there, the helm, v3 FAQ has a lot of this. You know exciting, exciting delightful changes.

A

We've been talking about IDE through container dot training on there, just because it's kind of fun when you go to these events, to go back and have something for your co-workers to play with and that's an open-source project. I have worked a bunch on with Jerome Pettis oniy. If you're not familiar with Jerome, he used to work at a little startup called dot cloud which became docker, and he was there for like seven years and now he does Cooper know these workshops.

A

So you can check that out of help him with that and then also because I work at Microsoft I will point you to what is kubernetes Learning Path. That is, actually you can learn things from it, even if you aren't using Azure, it's just kind of training, so yeah, that's what I've got and as long as all of your questions are answered by things that I think I know were good. Otherwise, I have colleagues in the front row I want you to so thanks.

A

A

Sorry I was trying to hear that helm three using a schema, Oh JSON, schema I, do know anything about the JSON schema. Who can tell me things about the JSON schema I'm handing you the mic.

B

So the JSON schema stuff in there is just for values, validation. So it's just basically for helping out when you're trying to do like a helm installed set or things like that, where you're trying to validate that the user has input, correct values and configuration into there, but it doesn't actually do the other part like Cavell, which is validating that the resources that you're trying to install are actually valid, kubernetes resources. So that's what the coop val project is trying to do.

B

The the json schema stuff is trying to validate that when you're trying to set a value to like a string. If you have like a multiple choice, answer or like a certain parameter in there, you want to validate that. They say that it's a string, fubar or a car, but you don't want them to input star and bark out on that. So two different use cases, but they're kind of related, yeah.

A

In kuba, Ville or Oh, with with the one that.

B

I believe is by default, I think so yeah. As long as the schema is there in the chart, it should be validated by default. Oh.

A

All right, so the mats are determining that you can't actually even disable it, so you are required to have safety all right. Do we have somebody else, who's cushion I! Think we go to snack time now. Do we go to snack time now? Does that sound plausible or is there another another talk snack time? Excellent, all right! Thank you all. So much.