Continuous Delivery Foundation Spinnaker Summit 2022 - Detroit, 4 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Policy as [versioned] Code - How to do it Right - Chris Nesbitt-Smith, UK Government | LearnK8s

Description

For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/

Policy as [versioned] Code - How to do it Right - Chris Nesbitt-Smith, UK Government | LearnK8s | Control-Plane

Beyond just "don’t run everything as root" In this talk Chris will trace back the origins of how policies are often incepted, how it can get out of hand, be slow if not impossible to update and measure compliance, and often lead us to question of **is the policy helping or hindering**.

A

Okay, I guess I'll crack on. Does that sound yeah? It sounds like an echo cool um small audience. Please don't feel afraid to go. If it's boring, I won't be offended, it's fine I know it might feel awkward, but we'll get through it. So imagine a thing with some human faces. Well, what a treat I get to stand, not worry about being on mute kind of um and use my clicker and everything it's generally quite exciting to be here so elephant in the room policy is a dull thing.

A

It's kind of hard to make it sexy, but I'm going to at least try and get your attention so bear with me so to set the scene I'm in a lift. Yes, American friends, we really do call them lifts and four people walk in and I think to myself Chris. This is your moment: Now or Never As the doors close I've, positioned myself in front of them, a captive audience, they're mine, I, hear the doors still shut behind me and I. Take a breath.

A

I looked at the first person on my left, she's in a suit. She looks really important I gesture to her see she looks back at me as if to say yes, go on uh I I say: oh, she knows ah perfect the CIO, the policy maker, the one whose neck is on the Block. What are the chances of finding you in my imaginary, lift today I, ask you well what keeps you up at night.

A

She tells me I, don't know what teams are really doing, what the volume of risk and what I've just shown more interest in setting and changing policy is slow and hard to communicate and people just go off and do their own thing. They think they know better and be honest. Often they do, but then I'm left playing catch up with the risk that they've signed me up to okay. So try not to sound like a patronizing snake oil salesman.

A

I can help I turn my attention to the second person, also in a suit, but looks slightly less important. I'd make a guess: let's face it. This is my imagination. It would be weird if I was wrong. Product manager. I'd say they nod ah the whip. Cracker. Well, what's important to you. Managing risk so mostly opportunity risk the fear of missing out so getting features out the door and avoiding getting bogged down with that as they glance to the CIA bureaucracy.

A

That feels almost like it's designed to Slow Me, Down, awesome, I, say this is your lucky day next person The Dresden overalls, I'm in a trendy part of town, they could be the CTO before I ask they sense me staring at them cleaner. They say ah how did you get in my imagination? Okay, let me come back to you. My attention goes to the last person. Hoodie headphones around their neck. ah My stereotypical developer, yes, I know you will what code do you write? It doesn't really matter python, they say cool.

A

Have you got everything updated to work with I might as I pause, Python 3 they offer yeah. That must be hard I had they don't know it yet, but I've just won some of their trust, which is, as we know, important nearly they say cool, what's important to you so staying on top of things like patching dependencies, so we can react to the next fire. Knowing what rules exist? What I can bend break and what might cause me to lose my job writing consistent, good quality code and avoiding technical debts.

A

The rest of my team effectively been able to operate cohesively as work as one. Do you use any tools to help you with that yeah linta's code, quality, test coverage, the usual great I, say: I, write code, C, let's be friends and I hand them a printed QR code and say Here's, my public gpg key. So you know you can trust what I have to say. I return, my focus to the cleaner I've got it. How do you get told what to do and then, when it changes?

A

Well, we get a memo or maybe something stuck to a notice board. So last week we got a memo saying that all the meeting room whiteboards have needs to be cleaned every night, interesting I say how does that work out? Well, it's up to us to then maintain that to-do list. So we can onboard new people.

A

Does it go wrong at all yeah? Sometimes, if we compile that operational manual and correctly, we miss a memo or don't apply them.

A

In sequence, we get things wrong; they glance apologetically to the product manager like when we hadn't updated the guide that the meeting room on the third floor was being used as a dedicated ballroom and we wiped down all the whiteboards I looked to the dev sound familiar I, ask they nod: Well turns out we're not all special snowflakes say all is not lost and I knew that there was a reason that I imagined you here. The lift is slowing, I feel it coming to its destination. Great I've got the silver bullet.

A

For you, too, the CIO looks to me ready to buy literally whatever it is I'm selling they ask me: is the doors open? Who are you and what team are you in as I move out of the way to stop obstructing it? I answer? Oh I, don't work here, I'm just here to fix the lift. People have been complaining. It only goes to the top floor, no matter what button they push and it's actually pretty slow. My audience, storms out Furious heading towards the stairs as the door shuts and I get back to my job.

A

Okay. So if any of that sounds at least Vaguely Familiar- and you can relate to some of my imaginary friends, then maybe I've got some answers for you. What if I said you could update policy easily even releasing several version updates, not just in a year a month? What about 10 updates in a single day and seamlessly communicate those to the people that need to consume it all without derailing them?

A

You could have visibility on compliance using tools that perhaps you already use, and that policy could be readily consumable easy to pass, demonstrate compliance, make sense and not be bureaucratic to change when it needs to be and not get in the way that same policy could be treated as a dependency and operate like a linter. So you can run compliance checks locally in CI and ultimately guard production that multiple versions of the policy like a dependency, are supported.

A

So emergencies like you, must update now, because there's now, some known vulnerability type updates become a business as usual activity to communicate interesting. Okay, hang around cool I'm doing about all right, so hopefully I've got at least some of your attention. It's time to introduce myself and start explaining things a little bit more.

A

My name is Chris Nesbitt Smith I'm, currently an instructor for learnkates and also for control plane and the consultant to Crown prosecution, Services a bit of UK gov and a tinkerer of Open Source uh stuff I've spent a fair amount on my professional career now working in UK government and large organizations where problems like these are Rife.

A

um We should have some time if you, if you need it for questions at the end or your best, heckles or even better, um if not I'm, reasonably obvious, with pink trousers. So come find me um so, given that I've got the luxury of a live audience and you've all got your clothes on, which is a brilliant change from normal um by show of hands who's with my CIO and has set written or applied some policy before any sort, any code standards.

A

Anything like that cool, okay, next round, how many of you have sought exemption consciously bent broken circumvented ignored, bypassed whatever a policy with at least some good intentions?

A

Cool good? Well, you fell for it. So, thanks to the organizers, we've got all of your names and your employers details down so lend me your ears and the stakes just got raised a bit. The doors are now locked, so where do I perceive policy as code going wrong? Well before we dig into that, what do I mean by policy?

A

Well, it usually comes in one of two forms, so security enforcing, like data at rest being encrypted or consistency enforcing such as code style, tabs being two or four space indicated Maybe, or maybe you can think of some others, but in any case it's hopefully intended to mitigate a risk of some sort. However, with the best of intentions, these are often emotionally LED, rather than being grounded in a proportionate control, which is the ultimately, it becomes the open door to case-by-case exemptions being required. When you come up against a situation that you weren't anticipating.

A

This is fundamentally not unlike how the laws of the land are created, with case law, making for a complex and navigate rule book and harder still to measure compliance.

A

It often can look like the thin bit of a wedge where the Preston, which may have been an uncomfortable pill to swallow the first time around, becomes dangerous with others looking to expand upon its scope.

A

Which can lead us then to sometimes wonder if the Cure was actually worse than the disease, but that's not how we at least typically develop software. So why does this all have to be so hard? Surely there must be a better answer. Well, we've codified everything else. So isn't this the answer? You might imagine well yes in part, but my point of this talk is that we largely do it wrong.

A

So maybe some of you are screaming your favorite product names at me and your heads as the solution and you're not wholly wrong. However, the devil is in the detail, throwing some curly braces or yaml at something doesn't inherently fix things.

A

So security control. It's often tempting to keep that policy a secret so exposing it could maybe be used against you by an adversary. However, this does not support us shifting left at all. It results in devs, effectively reverse engineering what the policy is by finding out when we smash our heads up against it.

A

Therefore, it doesn't take much imagination to see that in the scenario of an application deploy Midway through finding one resources non-compliant and becomes rejected.

A

I could leave your overall deploy in a halfway inconsistent, State likely resulting in some downtime, which begs the question of was the policy better than the downtime, especially if it leads your engineers, who are all hopefully plenty smart people at finding inventive should we say, raise around the computer says no response that they'd got. This is then further exasperated when updates to the policy are designed. So maybe you get a pen test or something goes wrong, so you form this case law and you need to apply new policy.

A

So maybe I don't say all S3 buckets now need to be encrypted, a change that could well be considered breaking I'm sure you say it might say you provide warnings, or at least um at least less important issues, or maybe new emerging policies, which is great so long as someone sees them. But if you've adopted gitups or at least maybe CI CD, is anyone seeing those warnings so who studies the results of a successful, build log every time?

A

Anyone every time exactly? Well, if you are I'd, politely suggest you're, probably missing the point of in cicd. You should ultimately be able to trust the jobs, your job status- okay, well, I'm, not just here- to throw stones. So if we remember my employer promises to my four imaginary friends at the start of what this Promised Land might look like.

A

Well, there's nothing new Under, the Sun we've actually already unwittingly sold all these problems elsewhere. We just need to be reminded and kind of join the dots. So the first is something if you're doing policy as code you're, probably already doing so put it in Version Control. The thing you might not, however, be doing is then making that visible, so at least in a source it by which I mean allow anyone within kind of your Walled Garden of employees, suppliers and so on.

A

To see the policy I'm not saying give all of your kind of threat, monitoring and Intel rules away. You can probably keep those to yourselves, but I'd argue that visible policy and the gaps they're in as often better than the downtime reverse engineered workarounds and opaque Legacy exemption spaghetti soup.

A

If you're brave, you might even open source, it you'll find it where it unlocks an ability to work really well with prospective suppliers without ndas and whatnot and widely distributed. Secrets are expensive to maintain difficult to handle and often only then stay secret for so long after all.

A

Okay, so we're off to a good start, our policy is visible to those that need to be able to see it. Many of you I'm sure, all of you in this room at least are used a semantic version, but a quick recap.

A

So the first segment is to indicate breaking changes, say perhaps conflicting in the context of policy. However, let's say it's requiring resources to have a department label. Maybe that will help with some say, internal cross charging who knows I'm not really, judging and increments, that might look like requiring it to be from a predetermined list rather than free text.

A

The second segment is to indicate minor changes. These shouldn't break anyone an increments that might look like, say, correcting a spelling mistake to one of the department names. Third segment is to indicate pack exchanges, so there should be a no-brainer for everyone to keep up to date with and increments. That might look like the adding a department to the list of available options.

A

Okay, so our policy is visible in a repositories version, so we can easily communicate the policy. We can tack on release, notes and expect expectations are all managed by semantic versioning in software, we're used to handling dependencies. So what? If your policy was just another dependency, so you might unwittingly already be doing this. If, for example, you have say eslint as a dependency in your JavaScript package manager, perhaps okay, so policy is visible in its version, we can communicate it. We can tack on release, notes and expectations managed by semba.

A

It's all beginning to look a bit more like how we develop software.

A

Okay, I know: testing is a really dirty word, but in order to make this an asset that everyone can depend on and also provide some kind of known good examples, tests are fundamentally essential to give everyone and everyone the confidence in the stability and surface any potential side effects before they then hurt everyone involved.

A

Consumers of this policy needs to be able to test themselves against this policy locally and in cicd, thus shortening the feedback, loop and better informing everyone. So, as a bonus, we should be able to find our consumers able to rely on the artifact that we're sharing with them, okay, we're well and truly on the home stretch. So it's dependency, so updating it should be no different to any other.

A

We can use some even use some magic, like github's, Dependable or men's renovate, to do that for us, so think automatic pull requests tests, even Auto merging, if you like, okay, so to check you're all still awake. Can anyone tell me a recent event that caused me everyone to want to know what version of a certain logging Java doohickey? You were potentially running literally everywhere in the estate.

A

Help me out come on. Thank you.

A

Yeah exactly bingo, so, as you know, all presentations this year are actually contractually required by the organizers to reference log 4J, even when it's almost entirely out of context in and include some memes about it. So in just a few short months, I can remove all of these, and hopefully just broadly point a list of scary looking cves in order to command your behavior through fear.

A

What I'm getting at here, though, is that the situational awareness piece around software supply chain is something that your organization is hopefully already thinking about, if not already addressing. So, if our policy is a dependency, this is now not at least a new problem, so software bill of materials for the win right, which can allow us to measure the compliance across the estate.

A

Okay, so I've just covered a lot of ground and hopefully sounded at least vaguely convincing, and it's not just a fictional Utopia painted in Powerpoint it's time to look at how you might be actually able to do this and I know you all really came here wanting to look at a million words on a slide and not just the odd Emoji or two.

A

So we've reached the point where I get to show you some code hooray to maintain scope, though I'm going to limit this to talking about two things, just to prove that it's not one Tech or one tool, I've arbitrarily picked, terraform and kubernetes I could probably uh picked. Anything uh naturally need some tools to go along with this I'm, too lazy, really to invent much at all here. So likewise, I'm going to pick two tools, but again these could be some or even all, probably so.

A

In this case, checkoff's going to be doing my terraform and caverno is going to be doing my uh cubanesses.

A

If you want to browse along with me, I've created an example: GitHub organization there'll be a link at the end as well. um I'm. Not expecting you to read or grock the code, that's on the screen, so don't worry about it too much. It's just to prove that I made a real thing that actually does exist in the world, so the policy is stored here.

A

So here's where my policy starts at V1 I've got policy that requires a department label on all resources. So long as it's set, it doesn't matter what it is I've written tests for this. So you can note how the passing test cases are useful. As a great example of what good and bad looks like we've pushed a tag in gits we've added release, notes. I can sign it to provide further Assurance if my heart so desires, which obviously it does, but moving on version.

A

Two looks similar only now that department field has to be one of a predetermined lists like before test exists. Release notes are written, tax assigned T 1.0 is where we notice and correct that spelling mistake of one of the options in the list of the Departments and 2-1-1 and I've now added the new departments to the list.

A

Okay, a few more repositories in that organization. App one and infra one well, they depend on version 1.00 of the policy. It's not compliant with version two or Beyond, but how do I know that well I configured renovate in this case uh to automatically make me a pull request so when it sees a new version of the policy, it's super obvious. If I can update that dependency and I can also see clear feedback about where and why I'm not compliant I can see all of the pull requests over the organization.

A

So I could use this, maybe to measure my compliance of the policy. Moving on from that female repositories, app two and M40 will these depend on version 2.0 of the policy. However, we could merge the open pull request, all the way up to 2-1-1 and finally, app3 and infra free are dependent on version t11 and they get a gold star from the CIO.

A

There is a small Touch of Magic and it's not pretty I've written some Bash, don't judge me even though I probably adaptedly. Definitely, yes, absolutely written a lot worse. But what this does is it allows me for my Dev laptop or in CI to evaluate my code against the version of the policy that I've declared. Ideally, this might be kind of less cumbersome, but it is what it is for now.

A

Pull requests are all very welcome and the last piece of the puzzle is managing the life cycle of the policies and allowing multiple versions of the policy to be accepted and evaluated all within a single runtime I've cheated a bit here. So Cube gives you admission controllers, it's not so easy to get so far as I found the same sort of policy evaluation in Cloud they've got their own policy code and I've not figured out how to be able to evaluate that locally again, pull requests and collaboration.

A

All very welcome, so you may have noticed the way that the policy was written and defined and distributed lends itself well to coexist with itself on a kubernetes cluster, so think multiple versions of itself and the same thing which brings us to Cluster one which describes a cluster that accepts all the versions of the policy that we've described so far. So I've applied all three versions of the policy.

A

Likewise, cluster 2 will that only accepts 2.00 and greater. We can automate all of this using kind and for CI to deploy the apps to prove that it kind of works and there we have a full org, all done all compliant policy, all versions and CIO all aware of what's going on. So this is great right, but just one more thing: wouldn't it be awesome if the policy carried a story about why it exists.

A

After all, if your agile delivery teams are even kind of half effective, they will reject absolutely anything that they perceive to be friction that they don't see. Value in this could allow our devs to know why they're compliance and if they want to do something outside of what the policy can outrally permits, they don't need any sort of exemption granted per se. They can have a well-reasoned informed debate with rationale behind a pull request to the policy.

A

So imagine if you will going through a stage of versions with risks that inform the mitigations manifested as policy all maintained as one. So when the risk landscape changes. Your policy can move with it when some new say perhaps want some new privacy regulation comes out, or your latest marketing strategy pays off and you require a load more data, for example, even if your policy was perfect at one time well, the risks and the appetite for it will stay still for No One.

A

We can liken this to over provisioning in some ways that we might be familiar with from elsewhere. So where lead times are long changes hard, and there is a significant pressure in nailing it the first time which can lead to hedging bets against what some future State might be, rather than proportionate mitigation to risks that are more tangibly real in the now and that's where the real culture change is needed and the execution of that is likely a long series of talks in of itself. So for now, this is all really over to you honestly.

A

The best thing any of you could do right now would be to tell me this is madness: It's, Already, Done, irrelevant or otherwise, totally unachievable, something my esteemed Echo chamber appears have yet to do so beyond making some pull requests and developing the theory.

A

More I'd really like to start building a case study with more real willing organizations and allow me to swap out some of my imaginary friends for some real ones, but the most important thing that I want you to remember from our time together is and feel free to say this out loud with me.

A

Purposeless policy is potentially practically pointless policy, which I've been practicing saying far too many times, I've been Chris, Nesbitt Smith. Thank you. So much for your time. You're now free to leave I'll, destroy the evidence of your guilt admissions earlier um I'll. Try um like subscribe, whatever the kids do on LinkedIn GitHub, whatever um you can be. Rest assured that there'll be little to no content, um I'm pretty awful at any sort of self-emotion, especially on social media and cns.me.

A

Just points to my LinkedIn and talks.com cns.me contains this and other talks and they're all open source um questions are very welcome on this or anything else. um I've got some time left, I think um or find me afterwards somewhere, I'll be the only birth person around that's in Pink jeans for the rest of keep gone.

A

Thank you very much.