Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2018 (Seattle), 16 Dec 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Automating Enterprise Governance Using the CI/CD Pipeline - Satyam Agarwala & Mark Angrish

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Automating Enterprise Governance Using the CI/CD Pipeline - Satyam Agarwala, ThoughtWorks & Mark Angrish, ANZ

ANZ bank has been on a transformation journey to achieve greater business agility. A key initiative in that journey has been the creation of ANZ flavoured "paved roads" (inspired by Netflix), realised as self-service CI and CD pipelines, that allow delivery teams to deploy software to production (Google Kubernetes Engine) faster without compromising on quality, security or compliance. A vital characteristic of the "paved roads" is the automation of the governance process the bank has in place for change management. These are driven by some of the strictest financial regulations in the world. This was achieved by integrating open source tooling such as Grafeas and Kritis into the pipelines to track build metadata and enforce policy. In this talk we will discuss the benefits the approach provided in meeting the strict regulatory requirements while accelerating the journey to GKE.

To learn more: https://sched.co/GrSw

A

I cube gun, that's pretty loud, so my name is mark English I'm from ANZ Bank and.

B

We're here to talk.

A

To you today about how we speed up delivery and scale-out with kubernetes and how we've decided to approach governance and automating that so we're going to talk a little bit about the landscape that we operate in.

A

To give you a bit of a better understanding of that how cube it was being used at the moment, a enable organizational agility for us and then, besides the software supply chain stuff that you might have seen this related to we're going to talk a little bit about Gracias and kritis and how we enforce that organizational governance that is important to us and then, if we pray to the demo gods, actually there's only going to be one god that you'll have to know about which, where is it gone? There is its VP.

A

So thank you ciencia for allowing us to use this about. 50 is the demo god. So please pray to the demo, god Phoebe, so our demo works.

A

So my name's Marc I am what's known as a tech area lead in ANZ Bank in engineering. Acceleration I've been an engineer for about 15 years, mostly working across the u.s. UK and Australia, and mostly in financial services, high frequency trading and stuff like that and a little bit of startups I'm now a senior manager at A&Z and my last open source commit was I, think April 2017, so don't hold that against me, coupon and now I'm responsible for helping the bank go faster and I'm. Just gonna introduce a team here morning.

B

Folks, my name is Satyam I'm, a lead consultant with talks, as you can see, marketing I've done my job and I've been a taught us about seven years now, work with a bunch of different clients across different geographies, mostly in the financial services space working very closely with mark over the last five months, or so it has the lead engineer on the engineering, acceleration team and.

A

What's really interesting is that I found out I have more github activity than he does, and that tells you something that is all so. This is our building. A and Zed sits about six and a half thousand people we're about 50,000 employees and about 10% of that is tech, we're about a hundred and eighty three-years-old and we're based in Melbourne Australia we're the third largest bank by market capitalization out of the big four banks that operate in Australia and we have about 10 million customers worldwide and we operate in about 33 geographies, including this one.

A

So also, if anybody fancies a trip to Australia in a one-way direction, just talk to me afterwards.

A

Okay, so, let's get into it. This is how we deliver today and said you can see it's pretty hard. It looks messy. It hurts my eyes actually just looking at that. That's a there's, a lot of paths to get stuff out to our customers there. So, as you can see, we predominantly create really unique solutions, and this leads to a heterogeneous way of delivering value to those customers. So there are obstacles everywhere. You can see.

A

We've got cops, we've got potholes, we've got all sorts of things, and this is the typical experience that one of our delivery teams would have in terms of getting something out to a customer. So they need to get lots of approvals and guidance from a whole bunch of different parts of the organization, including security architecture, networks, mainframe, integration, environments, someone's phone- and this is like I, said the typical experience that one would expect when delivering.

A

You know something to a customer, so the speed is very slow, so customers do see value that we do, they do get it, but it's very infrequent and in today's ecosystem of disruption, especially in the banking sector. This is something that we can't really live with anymore, and this really directly affects our the way. We also do governance and we need, as the reason for that is.

A

We need to support all these different ways of delivering value to customers, and that is not an easy thing to do for us, because it like I, said each one of them requires its own unique solution, so I'm just really going to quickly blast through this. This is a dominance. It's really the balance between value and risk.

A

So I'm, just gonna read this because I wrote this last night and it makes sense to me so I'm, just gonna read it verbatim: it's the set of structures, processes and policies by which we build deliver and maintain software within an organization so as to yield business value and mitigate risk. That sounds pretty cool. On one side, we are worried about things like reputational damage, financial loss for the organization, financial loss for customers, outages, etc, which can get us in trouble with the regulator and anything else that might actually cause customer dissatisfaction.

A

As an engineer, it's really important to understand the risks in which your organization operates in. So it's really crucial that we have those things in mind when we're building out whether it's a sales ad pipeline or anything value should be pretty self-explanatory. It's the business value you create for your organization, and you use your let coding skills to do that, but put another way. Governance is really the action. Your organization takes to balance its value against risk, but sometimes it actually feels more. Like your organization. Might operate like this, so why is that?

A

Why we in this kind of mode, so the Prudential regulator in Australia takes outages and incidents really really seriously, and so do we all the banks in Australia do and so like many organizations when something really bad happens, it causes us to re-examine our governance. So we go to that. Second line of you know looking at our change controls, and then we might ask ourselves some questions. We might tighten a few things we might say to you know. Even in some cases, maybe people need to step up. You know. Is it a people problem?

A

Is it a process problem? You know, we re-examine all these things and then what that causes is. Actually this re-examination to tighten and you know and change your organizational policy around change and when you combine that with the technical landscape that I just described before what you get is a harder way to do a change and it gets amplified. So what we end up doing is bunching up our changes into large larger releases which again increase it slows down our cadence and then releasing itself actually becomes an arduous effort.

A

In fact, sometimes it takes a weekend to do. A change sometimes actually takes several weekends to do a change, and so that in turn increases our operational risk and then we do more testing and we don't catch something, and then that in turn causes an outage to happen again and we continue the cycle, and this might be, you know yearly or something like that. But if you've read the book accelerate by dr.

A

Niccole, fours, grin and her friends, you'll know what are some of the causes of this so I'm not really going to dive deep, but that's a good book. You should definitely read all right. Good news time answered is changing. Here is our CEO. He is leading the change he wants to make us more agile. This is a digital transformation. Everyone's doing this we're in the middle of ours. We want to unleash our organizational agility. We want to combine the worlds of business and tech.

A

You know one of the most important things about this, though it's created the impetus for change and cultural change is actually half the battle, and this is what we've restructured ourselves to. This is actually really important to know. I promise I'll link this all up later.

A

The design of our new organization has several philosophies, but this is largely the Spotify model if you're familiar with it. The key thing here is the squad. The squad is multidisciplinary, autonomous and guided by missions. Now you can think of a and Zed is actually having several hundred of these squads, and the important thing to understand about this is that it has well it's actually, the organizational structure is really important, and why is that Conway's law? Right?

A

Hopefully everybody knows what Conway's law is, but those of you who don't it's, that your architecture will look like the sorry. Your software will resemble the organized organizational structure that you have. So what are the implications of that? Our smaller teams here are more likely wanting to produce smaller, more cohesive services, then stitching together, giant monoliths, and this obviously closely aligns to micro services type philosophy. So we have squad, but how do they address their missions? How do we unleash their autonomy?

A

Given the culture, the current landscape, that we operated in, that I showed you before. If they could, they would probably want to build their own, build infrastructure for their own squads and release their own things right, because that's economically, what they're, probably going to be primed to do they're going to want to you know, maybe create their own CI CD. For what they're doing there might be subtle variance between each one of those squads in how they approach it, but that's really inefficient.

A

You know our engineers could actually be working on delivering customer value rather than setting up CI CD pipelines. So this is just a random pipeline that you might use to deploy something onto kubernetes. So how can we scale this sort of pipeline added an organization like ours? You know we can try and take a patent Iced approach. We can say to people, you know this is what we've done you might want to copy it. That's not too bad.

A

We can centralize some of these services like Jenkins or Sony, cube or whatever, and and in expose those out, and people can stitch together their own pipelines. For what they're trying to do. That's also not too bad teams can also you know who are further ahead, can actually come back and show other teams how they've done things and that's actually pretty good as well. That's a Sherpa model and that's also pretty good I- think these all encourage scale. But for us that's not ambitious enough. You know with the start of our talk.

A

This is about industrial icici pipelines, and this is an industrial enough for us. So how do we do that? We get Chuck Norris to do it all ya know. So we use automation right to industrialize so that it makes things easy for our our engineers, so we're at cube con. We know micro services and kubernetes go hand-in-hand. It creates that layer of abstraction for our engineers and, more importantly, it becomes a lingua franca for different parts of the organization in different roles. Everybody can talk this language so for ian's, Ed's, digital transformation. Kubernetes.

A

Is that catalyst that we needed for that organizational agility? That I spoke about earlier, so what's great about this? Is now our squads not only have to? They just need to know their stack, they need to know docker and they need to know the concepts that exist in cube and that's like good enough for the most part for them, and it also has implications for our workforce management means we don't have to hire everybody knowing the depths of kubernetes right or even below that stuck at the IRS level.

A

So if we think of kubernetes is unlocking organizational agility, then managed cube actually in the cloud acts as an accelerant for us all, right, I'm, just gonna spend a bit of time on this, so we use terraform to provision the core banking infrastructure, which is completely phoenix above for us. This means things like artifactory or black duck, as well as gracias, which we'll talk about in a bit are all provisioned on the sort of this primary setup for connecting out to JCP.

A

On top of that, we've exposed services, like all modules, I should say like gke in cloud, sequel and stuff. Like that- and these are all also Terra all done through terraform- and what that means is that this allows us. You know from the app pipeline level that if cluster doesn't actually exist at the the initial creation of the pipeline, it will go off and provision that cluster before it continues on with the app pipeline.

A

So it's really I guess it's kind of like a way that you can bootstrap the whole pipeline before anything's actually ever happen. So we actually do this per asset level. So in your own organization, you might have a different definition of what an asset is, but for us it's a collection of services that we uniquely bundle together and then that at the same time, will provision something like spinnaker and the rest of the CIC ad pipeline. For that particular asset. The key message here is that the whole thing is automated.

A

So switching to the developer perspective, what do you need to know you're really merging a pull request, and then that triggers like what you see- and the last thing you see- is your app in production or a particular environment. If you want to, if we have manual intervention in here, so the key thing is that, like I said, the whole thing is completely automated. The CI CD pipeline, though, becomes the only way that you can deploy apps into gke.

A

So there is a lot of detail missing here like how we do secrets management and how we set up our projects in GCP, around VPC, peering and stuff, like that. Actually Google is helping us out with some of this stuff because it is a bit experimental, but if you want to talk to us afterwards, we can tell you a bit more about that. So, while the cloud presents, you know a really good opportunity.

A

This newfound freedom presents challenges, though, if we run disciplined with the way we act in the cloud, it reproduces some of those problems that I talked about earlier and it actually amplifies them. So this is things like cost, blowout and stuff like that. So this is especially true if we're running say unmanaged cube as well, because that means my team is on the hook for having to support all the stuff that sits under underneath that as well.

A

So as we move to the cloud in gke, how do we keep our state well-managed and how do we effectively control risk all right? You know the answer to this because I said it at the start, its governance right. So this is what we call the paved road and we completely stole this terminology from Netflix.

A

So thank you Netflix for two coining that and we were inspired by the company companies like them and Capital, One and ing, and what we've done is we've taken the typical experience that I mentioned earlier and what we've done is: we've worked with different parts of the organization that I mentioned earlier, like security, governance, audit, etc, and we've actually combined them into this way to get to production quickly and we've added our own opinion on how we want to actually make this happen, and what we've used obviously is Cuban ETS as that base and we've created a platform over the top of it.

A

So the key principles of our platform are improving. The developer, experience, increasing the speed, sorry, the speed and the agility that we were sort of requiring for our organization. We want to increase the safety and security of the way we put things out, so we're getting less and less in terms of incidents, and we want to abstract delivery using you know: continuous delivery as a service.

A

So, while kubernetes simplifies our architectural landscape, the homogeneity required to allow us to scale. You know that's requiring another balanced item here, which is again the governance, so the paved road by the way is a value proposition. We don't mandate this on our teams.

A

What we do instead is we have a product mindset with this and we actually listen to our users and where they've actually got problems, and we actually then try to take care of those particular things in a prioritized order, but the developer experience and the balance between change so that governance thing is really critical for us all. Right governance can be modeled. Here is a set of control points up the top there. So what you need with that is evidence that you have actually met those control points, so we typically do this at the moment.

A

We've changed control boards and we might use governance forums and things like that, but this is really heavy. So we've changed this new structure that you know energizes and you know- provides empowerment to our teams to deliver value, but the governance that we have at the moment is kind of done via a handbook. You know you tell us how you're going to get this thing done and for A&Z that's for each team.

A

They have to kind of work out over about 50, plus, mandatory and situational control points how they're actually meeting those particular bits of governance. But the important thing about this, though, is that you can consider each one of these things as metadata about your SDLC right. So each one of these steps is actually a bit of metadata about how you met that control point. So how can we stitch this with our paved road that I talked about earlier, to make releasing not only quick but safe and and well-governed?

A

So how can we enforce governance as code I'm just going to pass over now to Satyam he'll talk to it through how we do this thanks.

B

Mark all right, so we've got all of this structured metadata that we want to capture and what he used to make decisions about, whether we should deploy changes into environments or not. So how do we do this enter graphics and kritis graph use is an open source, metadata API that allows us to cap structured metadata about our buildin release process.

B

Kritis is an open source solution that allows us to create policies and enforce the integrity of our software supply chain. Using that data that we collect in graph is container analysis and binary. Authorization are essentially managed service versions of graphics and Curtis that good GCP offers. So subsequently, when we talk about in the slides about graphics and Curtis, you are really referring to the managed service versions, because that's what we've been working with alright. So there are a few key concepts that we've got to be familiar with to understand the ecosystem better.

B

So the first is notes and occurrences. What are notes notes, I, essentially definitions of metadata that we're interested in so, for example, if a new one durability is found in a node.js library, you can create an a node gets created which details the which provides details about the vulnerability. For example, the CVE number the severity, the versions of the library that it impacts for every artifact that is produced in a pipeline or every artifact, an artifact storage like artifact tree that contains this vulnerability.

B

We create an occurrence object right, so there's a one to many relationship between notes and occurrences, occurrences and Stanzi ations of notes. Let's take another example. If you want to track the deployment history of an artifact as it goes through your different environments and you practice binary promotion, you want to track that history. So if you have a note, a deployable note, which lists down the details of the artifact and an occurrence in that situation is every time that artifact was deployed into an environment.

B

What is the date that it was deployed time that is deployed who deployed it? Those can become the those constitute the occurrence associated occurrence for a deployable object for the purpose of governance. We are interested in a specific kind of note called an attestation. A decisions are special because they have the notion of trust associated with it, and how does they do that?

B

It's linked to an attested object which is religious, a container for a public key, the associated private key can is sits with a human or a machine actor that represents a control point in their governance process right so, for example, it could be a design review by an architect or a unit test stage in a CIC a pipeline. You can use this private key to create the attestation occurrences right.

B

So it's a signed piece of the authorization is a signed, verification that that particular process or that particular control point has been satisfied, and the more important thing here is that that a decision can be verified using that public key. So we'll see this in the demo.

B

The next concept to be familiar with is of binary authorization policy. B, not policies, allow us to define the entry criteria for a GK cluster. This criteria is in the form of a list of four testers.

B

What this means is that any container that is going to be deployed into the GK cluster must have added stations from each of those the testers before it is allowed entry into the cluster. If this sounds very familiar to an admission controller you'd be spot-on, because that's how beneth is actually implemented behind scenes, so what's key over here, is that if every attestation presents represents a control point in our governance process, then this policies- and she represents your governance.

B

This code right alright, so we've talked about notes, occurrences specifically about attestation and then B naught policies. Tyra's is all stitched together. So let's give you an overview of the demo that we're going to go into we're going to take a demo app we're going to make a change to it.

B

Push it through cloud, build and create a container push it to container registry and then trigger a deployment pipeline and spinnaker, and we've talked a bit through a spinnaker when we go through the demo, because our G key cluster has binary authorization, enabled the attempted deployment will trigger the admission controller, which checks with binary authorization. If that particular attestation is present for the container the first time around is not going to be present, so that attempted deployment will fail and the changes are not going to be deployed into the cluster.

B

The second time around we are going to create the attestation before we trigger the deployment and this time. Hopefully, if the demo gods are with us that deployment should succeed all right, let's get into the demo and I love. This picture, I.

A

Like how it's a say, I see a demo very specific, but.

B

I'm going to have to switch.

A

Just so, if we didn't catch that, if this is going to be on GCP the demo okay, so it wasn't clear.

B

All right, so we've got a demo app running in a cube cluster. It's a little gray like the weather right now and that's what we're gonna, try and change. You're gonna try and brighten this up a little bit just a quick, so I just can people at the back see or do you need me to increase the size?

B

Is that good up? Okay? So if you should, we've got the app deployed into a GK cluster as well as spinnaker deployed into the same cluster right. Let's go over to the code and we're gonna try and change the color I'm gonna try something completely random I've, no idea what this code is actually gonna do. Hopefully it's brighter now over not there.

B

B

Have no backup video of this so we're really living on the edge over here. So hopefully the demo gods are with us today, pray.

A

To Phoebe way,.

B

To fit me exactly alright, so that should have triggered off a built in cloud build there we go while that is happening.

B

I will just take you to the binary authorization setup in GC P, so the policy that we spoke about you can see that here it essentially lists one: a tester object that every container that is being deployed to the GE cluster must have, and how does that a association work binary authorizations and she forces his policy on every cluster within this project for a more real-world setup, you can actually have a multi project setup where you can have policies in one project and that can apply to clusters in other projects.

B

Any kind of cluster specific rules as well The, Associated tester, is shown here, which really just is a container for a public key. Alright. So that's how the setup looks if you go over to plot build that has created an image and let's just check that out images landed up in container registry yep right there. This should have triggered a pipeline in spinnaker, so I've got spinnaker running here. You'll see that the pipeline's been triggered. It's waiting. So that, just for the demo purposes, I'm going to try and deploy this change and hopefully it'll fail.

A

Yeah, so we've inserted a manual stop here. So spinnaker can just automatically release this, but we've got a manual trigger in here. Just while.

B

The demo zapping, but a quick explanation about the pipeline, it's a very simple pipeline which essentially fetches a manifest file from cloud storage bucket and it fetches the image from because of a pops up trigger it, fetches the image from GCR and tries to deploy it to a cluster all right.

B

So let's have a look at the deployment, so we can go over and into the cluster and if you look at the workload you'll actually see, the error show ophtho right that that container with this particular image, digest was denied entry because no other stations were found that were valid or that were trusted by an a tester which is defined in that policy. Okay, so we've got the situation here now we're gonna, try and deploy this again and this time we'll create an attestation before we trigger the deployment.

B

You know you should try a different color this time, maybe something more specific. I.

A

Think I have a color for you. There is a NZ ocean, blue, it's a lovely color I'm.

B

Sure you must remember the hex code for that. Oh it's.

A

Very easy to remember the hooks code, it's double-oh-seven de ba. If anybody has a good joke about that, please let us know, we've tried and we couldn't get one too.

B

Tired I think all right: let's push that through something about James, Bond being a DBA or licensed I'm gonna try. This is all being recorded. I should remember.

B

Alright, that should have also triggered off another another. Build there we go. Just one thing to remember is that, while we're showing this in the world of containers in GK, the principles really apply to any sort of deployment process, it could be VMS, it could be, but it's just that the majority of the ecosystem has to increase to support different kinds of workloads as well.

B

There is no reason why automation of governance process through attestation should apply only to a container, which is that we don't have the right tooling, available to implement this in a VM well. But that is something that we do have to work towards.

B

That's still going all right. There we go, let's check that our pipeline is triggered, but before we trigger the pipeline, we're actually going to go into the terminal and generate the attestation. So what we need for that is the image digest, because we're going to create attestation for the simulation, that's how we identify the image through the entire process or the a fact through the entire process.

B

Alright, so I'm, just gonna put that into an environment variable and.

B

I'll talk through the entire process, so you have a better understanding of what we're doing. First thing that we're going to do is the binary authorization. Api has a helper method to create a signature payload. So this is essentially a JSON payload, which represents that may be a test that the authority, in this case a human actor, for instance, if I'm an X cubed doing exploratory testing and I signed off on this particular artifact right. So I'm gonna just create a payload which I can sign.

B

Let's have a look at this payload. It's really simple. Oops.

B

It's just the location of the artifact right, so it's got the repository location and the image digest which identifies that artifact. So what you're going to do with that payload is we're gonna sign it using a private PGP key on my local system.

B

They exist a signature, so we can overwrite that all right that should have created our signed, our generated signature, which we can use to create the Associated occurrence at assay the attestation occurrence right. It's just a signed signature. It's just a signature of that content that I just showed you we're gonna now create the attestation, and there we go so this is essentially again can be driven through the API. We can create an attestation we're providing the artifact URL and just a few other pieces of information that the API expects.

A

So, by the way you could actually use a wrapper over this like a GUI, so you could use a so. We actually have something like this. Where you can manually, a human actor can actually manually create this for certain steps in our governance pipeline and.

B

That's really got to be critical. When you talk about human actors who have to create other sessions, you don't every person in the organization, who's responsible for signing off on a governance, control point to be going to the terminal and generating these authorizations that people are going to be like. That's, not gonna work for us, so let's just quickly check that the attestation has been created as soon as I find that command and you're gonna see a huge blob of text over here.

B

But if we look for this particular, if you look for this particular shot, there's an attestation created for it, which is associated with this public key now, if you go over to spinnaker and try again, hopefully this should work. Come.

A

B

Okay, let's see and we refresh refresh refresh and there we go hey I know deployment is now going through right. So our changes are now being deployed onto the cluster, because that attestation is present- and this is really critical, because you can provide that assurance to your governance functions. Your compliance functions functions historically, which have aren't very trusting of whether you've completed a process or not in order to make sure that you have actually met the control points show.

A

Me that ocean blue, let's.

B

Check all right.

A

B

A

B

And we've got a switch over now.

A

Okay, so just as he's switching over really just the thing that we want to walk away, want you to walk away from this. Is that you can? Actually? What do you want searching.

B

For the slides, oops.

A

I think he just killed our presentation.

B

A

Okay, so really, oh there we go it's back: okay, yep! So.

A

B

A

So, basically, you can improve the governance. Sorry, you can improve your governance process and also increase your speed and and scale. At the same time, your governance doesn't have to slow you down the.

B

Other one is just we got to make it easy to do the right thing right. So if you make it easy to do the right thing, people will not look for exemptions or take on the additional risks that they don't want to, and the.

A

Last thing that we want you to know is that if you are in a heavily regulated industry, things like graphics and kritis can actually help you.

A

You know codify your governance process, but, more importantly, this gets us to another area that we really want to start pushing industry towards which is regulation as process, so working with the federal regulator or whoever you are to just basically streamline this altogether.

B

Lastly, we just want to call out that there is a lot of work happening in the ecosystem, especially wanna call out the work that's happening at Google, but also at Shopify. There's some links here to projects that are being undertaken or tutorials that have been created to educate people about the ecosystem. Okay,.

A

B

Know how we're doing on time, I think.

A

We got two minutes or something but yeah. If.

B

A

Do have any questions yeah, let us know it is close to lunchtime so I'm hungry so, but if you do want to talk to us just come on over if.

B

You have any questions.

A