Cloud Native Computing Foundation CloudNativeSecurityCon 2023 - Seattle, 2 Feb 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Avoiding IAC Potholes with Policy + Cloud Controllers - Andrew Martin, ControlPlane

Description

Don't miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from 18 - 21 April, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Avoiding IAC Potholes with Policy + Cloud Controllers - Andrew Martin, ControlPlane

In large organisations, enabling and securing self-serve cloud infrastructure for teams hosting their applications on Kubernetes is hard. Most large organisations implement Enterprise Security Architectures featuring IAC pipelines with Policy as Code frameworks at the outset of their cloud journey, which are found to be not fit for purpose when teams use Kubernetes to provision infrastructure either natively, through services of type Loadbalancer, or using hosted cloud controllers such as Crossplane. In this talk, Rowan will demonstrate how infrastructure and security teams can use policy engines (Kyverno) to secure a model that uses Kubernetes native and hosted cloud controllers (such as Crossplane) to provision infrastructure. This model enables application teams to self-serve, whilst preventing the launch of insecure infrastructure and enforcing compliance and security requirements centrally. To ease adoption of the model, Rowan will open source an example library of policies integrated with OSCAL for commonly used services across AWS, enforcing controls aligned with NIST800-53 in a manner that can be audited by compliance teams, and simplifying the developer experience by enabling the dynamic generation of cloud resources with secure defaults.

A

uh Hello, welcome, I am Andy from control, plane and I am standing in for my uh right honorable, head of security, Dr Ron Baker, who very sadly has gastroenteritis and did not deem it fit to get on a plane very sensibly, and a lot of the work in this talk is uh part of the collaboration with Henry Mortimer as well, one of our Cloud native Engineers, so a huge thanks and greetings gentlemen from the past.

A

Thank you for the work you put in today and I hope that as much justice as possible, we will be talking about infrastructures, code potholes with Cloud controllers. This is the uh reverse intersection of uh of get Ops essentially and uh event driven automation. So, when we're trying to do infrastructure as code with with Adventure of an automation, where are the foot guns? How do we fix things and where can existing tooling come into play?

A

So we're going to talk about some pain points? We've seen around configuring infrastructure securely at scale within large organizations using cloud and policy controllers in kubernetes to address some of these issues and, as I say, a huge thanks to Rowan and Henry. Very briefly, Rowan runs the security team at control. Plane, control plane is a cloud native security consultancy. We are based out of London UK. We have offices in North, America and Asia Pacific and mostly do Consulting for regulated Industries.

A

We are as always, hiring if you're a security engineer, a supply chain, specialist, an architect or threat modeler. Please do come and have a word and the great disembodied Dr Baker.

A

Well, that hasn't quite worked. How I hoped, if you wanted to try one more time um should be coming out of here. I've got any luck.

A

B

I think should be oh, my God food poisoning a couple of days ago, so I've been uh unable to uh to travel just wanted to introduce the talk before my uh distinguished role playing colleagues take over today. We're looking to show how kubernetes provisioning infrastructure, such as load balances, can cause issues for large organizations in regulated Industries.

B

Their security architectures are typically uh dependent on infrastructures, code pipelines, featuring policy checks, so kubernetes provisioning infrastructure.

B

Just uh isn't part of the plan, uh we'll describe how that kubernetes provision infrastructure can be secured using uh policy agents at cluster admission time and then we'll look uh onwards as to how we can start using this patent to secure infrastructure provision by uh exciting New Generation of uh Cloud infrastructure controllers such as crossplay, then towards the end of the talk we'll show how we can work towards securing that infrastructure in a manner where we achieve both um automated compliance against standards and simultaneous enforcement of uh preventive controls, preventing misconfigured infrastructure from being launched all via a single oscow document that forms the single source of Truth for for both and illness.

B

Notwithstanding, we hope to open source the repo of example, policies and oscale component definitions for AWS Services shortly so with that I'll leave it to my uh my colleagues, I hope you enjoy the rest of the talk and the rest of the conference.

B

A

You Ron right. There are some demos. Fortunately they have been um ASCII cinemod for me, so I can't book to them, let's rock and roll, so using infrastructure as code policy I'm. Sorry, my slides a little bit booked with policy gates to ensure infrastructure is securely. Provisioned should form the backbone of any Cloud security program.

A

We can create infrastructures code templates using terraform at Al that hard codes secure by default configurations that we can distribute and reuse for consumers of cloud infrastructure in large organizations, and we can back this up by allowing users some degree of customization adding policy Gates into pipelines using tools such as Sentinel policy opa check of Etc. However, this does not account for infrastructure provisions in an event driven basis by Cloud infrastructure.

A

Most organizations have some non-kubernetes based workloads into Cloud first and it's often months or years before, a cloud migration uh and please keep, and that is which then throws a metaphorical hand grenade into the Enterprise security architect's office, most obviously kubernetes Provisions service load, balancer types.

A

This happens rapidly and flexibly at a developer's request via the cloud controller apis.

A

However, this can bypass infrastructure as code controls, um going back to everything to finders code and get Ops. These things just appear in our dashboards rather than being defined and subject to policy. Annotations can Define that load balances configuration and, if unrestricted, can create insecure infrastructure by adding routes and punching through otherwise, theoretically secure policies and firewalls.

A

We can see from this example that the SSL cert is commented out and the load balancer is internet facing. It is easy to footgun or intentionally bypass policy. In this way, trying to provision a load balancer via infrastructure is code like terraform and then integrate it with kubernetes, which gives us something that we can apply policy to is more difficult, PODS of node scale, with non-static IPS, of course, and encoding. These targets into the balancing pool of a load balancer is difficult.

A

In fact, you can end up using a controller to do this. Whilst you might end up with something that's compliant to policy, you complicate life for developers and operators, so The Logical way to ensure these types. These Services of type load balancer are secure, is to use a policy engine such as caverno or open GateKeeper.

A

We install those as an admission controller and those emission controllers intercept the requests, as the objects are written to the kubernetes API server, of course, prior to the Persistence of the objects, and so today, we'll talk mostly about validating admission controllers that simply send back a positive or negative response to allow or prevent an object being persisted.

A

That persistence then feeds into the policy into the provisioning engine, and then the infrastructure is either permitted to deploy or not based on the presence of that entity, using admission controllers and Cloud controllers. In tandem, we have a single enforcement points for application and infrastructure, with policy defined as code.

A

For completeness cluster admission is not the best time to discover that something is misconfigured, of course, and the reality is that we can't always run admission controllers to fail closed if there are errors or issues with them, an example being running at a mission controller outside of a cluster experiencing High latency and then essentially bricking a cluster.

A

So we need to integrate checks into the deployment pipelines for these infrastructures, with things like caverno and GateKeeper, tooling exists that allow us to integrate the CLI and the with the CLI and gate or respectively.

A

This is also helpful if we're looking to prove to stakeholders that we have equivalent security gates to infrastructures, infrastructures, code pipelines where tooling runs statically on the repository or before requests are made to the cloud apis, so we're looking to gain parity between statically analyze, the clarity configuration and dynamically provisioned ad hoc infrastructure.

A

Looking beyond the service of type load, balancer Engineers have started to realize the benefits of using controllers to provision back-end infrastructure for kubernetes hosted applications such as storage, buckets or databases.

A

There are now quite a few Cloud infrastructure controller projects such as AWS, ack, Google, config connects and cross plane, and these broadly follow the operator pattern. Custom resources within the cluster paired, with reconciling controllers for managing Cloud infrastructure, the benefits of this approach.

A

There is potentially less developer complexity, as plus the tenants are consuming a single kubernetes deployment pipeline for applications and related infrastructure. There is a uh you build it. You run it concertina of uh of runtime, SRE, devopsy style, risk and management. Here this won't work for all organizations.

A

This also enables git Ops for infrastructure configuration not just applications.

A

um So, of course, this is then Reliant upon a lack of mutation for these entities coming through the kubernetes API, because we get back into the same situation where, if we don't have a an offline representation of the realized cluster State. Well, then we've got a Reconciliation Gap, even if the operator thinks it's done. The right thing, because the mutate mutating admission controller has added in those extra annotations or sidecars all custom resource definitions. In this case, the controller pattern provides some drift protection with those caveats.

A

For example, you have to rerun the infrastructure as code pipelines to Stamp Out the drifts, and finally, we can secure this with policy engines, which of course, is the counterbalance to statically analyzing.

A

The infrastructure's code configuration that we would do traditionally so some early projects in this space to find the single controller per cloud service API, but the reality is that creating infrastructure securely is not only about creating one service and setting specific configuration options. uh These are the guard rails, removing knobs and dials in order to stand up a secure by default configuration often, we also require other pieces of infrastructure in place.

A

Take S3, which we will discuss a little bit more today. Not only do we need to create the buckets and configure it properly, but, additionally, we need to create the IM roles and users for access to those buckets, the KMS keys for back-end, encryption, etc, etc. Presumably some population of the bucket as well cross plane as a project, looks to address some of this issue. For us it is a cncf. Incidentally, this is a project that control plane is often mistaken, for cross-plane is a cncf incubating project that enables platform teams to build abstractions.

A

The tenants can consume obviously backed by providers to provision the infrastructure. These abstractions are called composite resources.

A

So in our hypothetical organization, a platform team defines the composite resources that developers and operators as human operators can consume the composite Resources Group managed resources together a managed resource correlates to an individual Cloud resource, for example, an S3 bucket or an IM role, and providers in turn manage sorry providers turn manage resources into external resources in the cloud EG S3 buckets, so we have our resources grouped into composite resources and they're, then managed by providers which are then deployed.

A

This simplifies the life of cluster tenants, application developers, system deployers, who can claim a composite resource for their applications, for example, to cure S3 and IM roles as a single entity, and there are providers for AWS, gcp and azure.

A

So why is adoption not more widespread? Well, enablers have only been around for a few years. The controllers themselves are maturing in much.

A

Excuse me, I'm having trouble with this guy in much the same way as terraform providers take some time to come to maturity. So we see the same um sort of underlying integration Point here. Ultimately, we are configuring distributed systems, and so we expect things to fail. Resilience needs to be built into controllers and shaking out bugs just takes a bit of time.

A

We might not be able to migrate everything under a cloud account into cross-plane just yet. There is also the cluster zero problem.

A

We need a cluster on which to install cross plane itself, unironically, that's provisioned via terraform, in the repo that we will open source to support this talk, but the biggest issues that we see for many large organizations. They have all infrastructure defined as code with configurations vetted using policy engines.

A

They then build teams around this pattern that satisfies c-suite and auditors. Kubernetes provisioning infrastructure does not fit into the model that we have taken so long to try and educate upwards to the audit space about.

A

So we need to establish trust in a security model for provisioning cloud infrastructure via kubernetes controllers, and we will look at how we do that. We have a proof of concept project that demonstrates that we can address these controls and be in a good place from a security standpoint when widespread adoption occurs,.

A

The library that we will open open source here to be made public soon I will do that after the talk contains libraries of validating policy for S3 and RDS provisioned using the cross plain Community AWS provider, these libraries are aligned to nist 853, Rev 5. and instead of generating mountains of policy and yaml, that might ultimately be forgotten. We also started to look at how to make those policies enable the documentation of compliance in an automated manner.

A

So, let's see this isn't reticent. So there we go ah that was unsuccessful. I might have to pop this in window.

A

Okay, incoming.

A

Can't see the target there, we go.

A

um Can anybody see the bottom of that scroll? Okay? There we go, it's still rolling. Oh there, it is.

A

Okay, so what we've been doing is making sure there are no existing cluster policies. The cluster has just had caverno and cross plane installed on it. The policies we've created for S3 exists in the ls policy. From the previous moments, um preventing Public Access encryption enforcement HTTP get requests Etc. This is what a typical caverno cluster policy looks like in this case, we're ensuring the S3 bucket object was configured with KMS server-side encryption by default and a validation failed. In that case, the object was rejected from the cluster, preventing the launch of insecure configuration.

A

This policy is now applied. We can see it's correctly installed. This is our minimal S3 bucket configuration.

A

You can see from within the provider Fields encryption at rest using KMS was not defined. We've tried to apply this to the cluster, and that has been rejected on the basis that the S3 encryption was not enabled. Of course, now we'll apply the rest of the policies shown earlier in the cluster.

A

So we'll just check the cluster policies are applied.

A

There we go, there's the deploy and we'll check momentarily.

A

Okay, we've validated that they exist, so we've created a bucket called the authorized bucket that satisfies all of these policies. This is the first 30 odd lines of that documents, but when we apply it on its own to the cluster, it will fail because a bucket policy to deny HTTP get requests needs to, firstly exist and reference our buckets to satisfy the security requirements.

A

A

Next, we will look at the required cross-plane bucket policy. This is attached to our authorized bucket through the bucket name reffield, which is incoming on eight and nine lines. Eight and nine there. As you can see, the policy denies all requests that have AWS secure transport, set the false on lines: 18, 19 and 24 and 25.

A

So we will now apply these prerequisites and the buckets will deploy successfully to the cluster without being rejected by the security policy.

A

So we'd expect the bucket definition, policy definition and other prerequisites to be wrapped up into a cross-plane composite resource for tenants to consume. We will then try and request the bucket itself.

A

It will take a moment for the bucket to show as ready, because the S3 server logging is on a best effort basis, and it can take some time for the bucket itself to report it's ready and then, finally, when applying the basic buckets, it will be rejected on multiple counts. There we go so that is the bucket which we expected, but now the basic bucket will be rejected and with a caverno test filed for sure ensure the policies are behaving.

A

As expected to be clear, we expect these policies to be triggered by the platform teams when they're, creating compositions compositions should never contain violations. These were a selection of demos. If they did, they would be self-defeating and useless.

A

A

So that's a first view for how this can be achieved, so we've just created more yaml to throw at the cluster without fully defining where our controls come from and I appreciate the frustration this brings and how difficult managing yaml policy at scale truly is. There are entire talks on that. One of our colleagues control plane by the name of Mr Chris Nesbitt Smith has a fantastic talk, called versioned policy as code which looks at how to make policy changes across massive government organizations.

A

The tldr is ensure people are allowed to test and have warning before enforcement.

A

Various different ways, but one thing we did look at here is how to create traceability between controls on the cluster and compliance requirements, which gives meaning to the controls in the library.

A

Here we get into oscow so using our scale documents as a source of Truth for the enforced policies, a huge thanks to Mr Brent Keller, who is the author of The tooling, that we'll look at in a moment. uh We have the open security controls assessment language going on here. This is a collaborative project run by nist. It is machine, readable representations of control, catalogs control, based lines, system, security plans and assessment plans and results.

A

Tldr, automated policy verification.

A

This enables an ecosystem of tooling for creating, manipulating and validating compliance artifacts around that machine, readable formats.

A

This removes resilience on out-of-date documents, spreadsheets screenshots and manual assessments for compliance activities, which I'm sure any of us who have gone through an audit are all too painfully familiar with, and this leverages the tool called Lula to validate controls in oscow documentation against reality and generate enforcing policy from Oscar.

A

So there are three documents that we will dig into here: catalogs are collections of controls maintained by an organization or by a standards body, so in this case we'll be referencing nist 853, which has been codified into our scale. Forests at 173, 000 lines of yaml profiles, baselines of selected controls and so nist has high medium and low bass lines for at least 853. But we can also create profiles for specific components such as an S3 profile that details all of the nist 1800 853 controls applicable to S3.

A

Finally, component definitions are a more detailed description of controls, supported in a given implementation, referencing a control from a profile so to build Oscar documents for S3. We were lucky enough to have this information available to us. The AWS config rule set mapped to the list, 853 rev, 5, I'm sure he's taking the Mickey out of me.

A

Now is public shown in the top left table here from this we distill the nist IDS into oscow profile for S3, which is shown on the right, and we wrote Governor policy equivalent to intent and spirits of the AWS config rules, Cal component document for Lula to Leverage I'll. Just repeat that last bit, some semi butchered it, we wrote Governor policy for Lula into Oscar, so the documentation here shows how the aforementioned this standard control IDs from the S3 profile are addressed.

A

You can see on this slide references to the S3 profile documents, the nist control IDs within that profile, followed by a description of the control and a caverno policy block. This block is an amendment to the Oscar component definition leveraged by Lula and proposed by the lular team and for context. Oscow is still a format under development designed to stimulate discussion and an ecosystem of tooling, so nist are collaborative when taking suggestions from the industry around modifications to Oscar document structures.

A

Lula is a tool by the right honorable, Mr, Brandt, Keller and team at defense, unicorns the bridge the gap between the expected configuration required for compliance and the actual reality of configuration. It takes the component definition as shown previously and leverages the caverno CLI to run generated Governor policies against the cluster or static manifests.

A

uh My colleague Henry recently had a PR submitted in this project to simplify sorry to Simply, add a generate flag and save those rendered policies to disk, so we can then get at them running incuberno and forcing them at admission time.

A

Okay, demo time again, I'm going to reset the cluster back to its original states.

A

Hit the target button there- lovely okay, so um we have a draft. We have a policy here that we've drafted for S3. That is just resetting the cluster back.

A

The important lines we're looking for are lines 28 when they turn up again yep line 28, where we reference the nist, 853 catalog, and then from line 30 downwards. We can see the list of control IDs from nists, which can be satisfied with the correct configuration of S3 and the rest of the file continues like so.

A

So what we will have next? Yes, that is further controls, so to remind you component definitions, show how the control IDs in the profile are interpreted. This is a cut down version of the component definition for the demo to reduce time under control implementation field, on line 28, 29 and 30..

A

We reference the S3 profile shown previously and then the implemented requirements for that profile are listed below each requirement has its own uid line. 34 the control of these references, the control from the nist profile, which is line 35, followed by a description field where we place the name corresponding to the AWS config rule. We have Rewritten in caverno and the rules block, which is a port of proveno policy used by Lula in the component definition we have just I mean just have implemented One requirement, and that is the S3 version. Life cycle policy, enablement.

A

So moving on, we will look at the basic bucket, which does not have the the fine that we're looking for the S3 version. Lifecycle.

A

Yes, it is absent, so without policy we can launch this into the cluster and have a vulnerable bucket in our estate. That is not what we want, we're proving the negative case in order to lock it down bad news for the cluster, so we'll now run Lula to assess compliance against requirements in the component definition and see that it fails. As expected, there is a compliance violation which, when you look up the uid and the component definition, is the lack of S3 version life cycle policy.

A

We should also note that Lula does produce compliant support to disk as well, and so Lula could be leveraged in a number of ways to assess compliance in an automated manner.

A

So, let's proceed to remove the vulnerable bucket.

A

There we go, we can also then run Lula in a generate mode, this time against the full S3 component definition, because it is quicker.

A

This will create the governor policies from the S3 component definition. Thus, we have the S3 component definition becoming the source of Truth for compliance and, ultimately, enforcement.

A

Okay, these policies are currently named by their euids and their duplicates as rules such as version life cycle policy are mapped to many nist control IDs. We don't want to push these into a cluster just yet, so we will have a look at the component definition which we have looked at before.

A

We've written a script that uses the uids to match the policy to match the policy files with implemented requirements in the S3 component definition and renamed the file and caverno policy name. According to the first word in the description for the corresponding implemented requirements, which is our AWS config rule online 37..

A

Yes, there we go, we will run that script and observe the policies that are created by it as if by Magic.

A

There we go, and these are the polishes for the cluster, let's install them as we did previously. We would expect them to fail with the basic buckets once these are deployed.

A

Zoom forward slightly, so the policies are created, we're now deploying the basic bucket, which we do not expect to be permitted, because we've applied our policy that we've gone through and generated.

A

And finally, when we deploy the authorized buckets, it is permitted to the cluster because it is compliant with all of our policy, magical, fabulous and wonderful. We can subsequently validate compliance using Lula with the added benefit of preventing a vulnerable bucket being launched in the first place.

A

A

So obviously, at this point in the demo, we've just scripted and glued a few things together with the Majesty of Bash.

A

What we're looking for is the final step for a policy um to be generated outside by something slightly better tested. Of course, we'd like to understand whether that's something we can build into the tool. If that is found useful, we welcome any feedback on the next steps. This is something that we have some utility uh with from clients.

A

At the moment, uh it would be logical to cover all of the AWS config rules in nist, 853, Rev 5., we'll also be looking at gcp or other controllers, such as config connector and in conclusion, policy engines for event-driven, workflows and infrastructure, as code are a must from day one.

A

We need to build them into security, architectures for kubernetes and Beyond any of our Cloud infrastructure deployment techniques. They enable not only the initial case, the load balancer service, but can also be used more widely across a large organization to enable self-service infrastructure provisioned by infrastructure controllers. There are a lot of developments in that space that could prove exciting.

A

In the next months and years we have some policies enforcing compliance to Nest 853 for AWS, S3 and RDS that you can use with cross-plane provisioned infrastructure under the collie repo that I will open up shortly and through oscarl Lula and caverno, and Cloud controllers we're starting to get to a place where we can tell in an automated way.

A

Sorry, where you can in an automated way tell if audio infrastructure and kubernetes applications are compliant with policy standards, while simultaneously enforcing those standards from a single source of Truth. Thank you very much to the authors of this talk. I hope I did them Justice and thank you for your attention.

A

uh Would anyone like to ask questions I may delegate them to somebody else.

A

uh Do we know if anyone has implemented anything like this with anything other Oscar with anything other? My first delegation, Mr Heller,.

A

I wouldn't say this.

C

It's like a real runtime environment, um but definitely from the perspective of like establishing those control catalogs um those are available for a variety of different standards and then from there you know, I think we've seen a couple. Different entities that are starting to dive in and kind of own PCI, for instance, has been explored um pretty pretty recently, as well as other kind of oscow related tooling, such as compliance, Trestle kind of exploring these same spaces.

A

Thank you very much. Is there another hand somewhere hello.

A

C

Some mathematics so if I want to say I can cover it in all of my companions.

C

Can I see all of the.

C

A

So is the question: is it possible to see a macro view of all policy across the cluster.

C

Like from this perspective, I'd say: no: um maybe there are tools out there. Maybe there are different ways of accomplishing this. That can do the job, and maybe the answer is yes. In those cases, Lula from this perspective is very highly focused on like kind of a inheritance model of like this is my infrastructure, or this is my application. Here are the controls that it satisfies or that you can inherit, but then, from that perspective right, it's very focused in nature, so that you might still utilize, like a GRC tool.

C

Governance risk compliance to provide you with that. You know high level abstraction of Lula might be the the runtime component. That's looking at very specific composable modules. That's then you know producing compliance reports and then exporting those out to a GRC tool that is defining relationships defining like how to look at the high level picture of all of your environment, or you know, provide filtering into different aspects.

C

That might you know you might want to focus in on and try and find what's what's currently validated and compliant versus what is not so it's one tool in a you know possible ecosystem that you know could be developed around oscow and system security plans, all the rest of the oscow models that are available.

A

Thanks again, uh we might have time for one more question. If anybody would like to post something to Brent hello.

A

uh So the question perhaps is: um when different standards have similar requirements for enforcement. Is there a way of deduplicating them.

C

I'd say largely: the answer is no um right. Now it's going to be kind of based upon. How are you crafting the inputs that you're providing to like a tool such as Lula um and they're, trying to develop like if there is a you know, duplication across those? How are you performing that and another thing that we kind of want to look at here in the very short term?

C

Is you know having this evolve from not only CLI driven, but also like something that is maybe more event driven in your environment that can be able to perform this validation at a kind of a lower layer so that you know, if only this one piece has changed, then maybe we can go and find you know not only one identify where there are duplicates, but then it with that be able to. You know, prevent the need to rerun validation against the whole environment over again and hey. Only this application in this namespace has changed.

C

Here are the compliance information for it and let's go and validate that.

A

Right huge thanks to my co-presenter Mr Frank Heller. Thank you to everyone else. Have a wonderful day.