Continuous Delivery Foundation cdCon 2020, 20 Oct 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Spinnaker Deployments on Kubernetes from a Security Perspective - Anirban Saha, Allianz Technology

Description

Spinnaker Deployments on Kubernetes from a Security Perspective - Anirban Saha, Allianz Technology

Spinnaker has provided ease of deployment for a number of cloud native environments and has become the absolute tool of choice for infrastructure implementers. However, it is equally important to consider security challenges when working in a cloud ecosystem. There are a number of security concerns to address when deploying cloud native applications using Spinnaker such as the Spinnaker deployment itself, permissions provided for the target cloud accounts, access control for Spinnaker applications, credential handling, authentication mechanisms, using roles for deployment (RBAC) to name a few. Making sure to address these security challenges not only helps an organisation minimise risk factors but also enables engineers to deploy continuously in a safe manner.

A

Hello and welcome to this session on spinnaker deployments on kubernetes, but from a security perspective, I am anir bansaha senior infrastructure team member at allianz technology based in munich with over 10 years of infrastructure and automation experience.

A

My objective is to create secure, devops solutions.

A

This session assumes that the audience has some knowledge of how spinnaker works and the basic terminologies associated with it. In this session, we will talk about a spinnaker deployment in aws, which is deployed on a eks cluster.

A

The worker nodes are in private networks and the application load balancers are in the public network.

A

The worker nodes are differentiated into management and application types and spinnaker pods are deployed in the spinnaker namespace on the management worker nodes.

A

In this session we will not talk about how to perform deployments using spinnaker. We will ensure that spinnaker as a critical infrastructure tool is configured as securely as possible, so that it does not become a security risk for the entire infrastructure.

A

There is nothing called absolutely secure. Security is relevant at each and every step of the infrastructure workflow.

A

Once we have secured a particular stage, there is always a new factor in the next step, which is vulnerable from a different perspective, and there are some pretty smart and dangerous people who are quite good at exploiting these vulnerabilities.

A

The objective of a security practitioner is to make the exploitation process as tedious and difficult as possible for the attacker, so that it's not worth for him or her to keep trying anymore spinnaker has really become the absolute tool of choice for deployments in cloud-native environments for many organizations.

A

For obvious reasons, we know that it's a beast in terms of deployment workflows, but being a cloud application with permissions to manipulate infrastructure resources.

A

It can be a potential target for attackers with straightforward documentation available to get up and running in few minutes. It's quite easy to overlook the security.

A

Concerns the biggest problems that infrastructure maintainers face is to identify the attack factors. There are multiple ways: an application and its base infrastructure can be exploited to gain access.

A

It also needs to be kept in mind that by rogue elements we mean external and internal entities. A system is as vulnerable from the inside and as it is from the outside, and this is often overlooked when architecting the infrastructure, it often gets easier to fix problems once we know where the problem lies.

A

Our objective is to make it as difficult as possible at each step for attackers to exploit the deployment.

A

So, let's start by identifying the problems and fixing them as we go, we will go through each factor. Sequentially.

A

Even before we can touch spinach, there are quite a few infrastructure level, vulnerability factors arising from the base infrastructure on which spinnaker is deployed.

A

They belong to the cloud platform such as aws and the container platform such as kubernetes. We will start with some of the infrastructure level, vulnerability factors first,.

A

The cluster roles assigned to team members are the first point of vulnerability for the system, depending on the privilege level of team members. That needs to be granular roles defined for access. However, that being an elaborate task in itself, it's very, very common to assign the default cluster admin role to all members.

A

This puts the entire infrastructure at risk with developers or non-admins having such access.

A

It's not only possible for rogue team members to exploit the system deliberately, but it also creates a lot of possibilities for accidental changes in the infrastructure by mistake, the fix for this factor is to assign dedicated time for proper division of roles for teams and team members and configure granular permissions for access with amazing cloud native infrastructure tools available for monitoring tracing deployments, builds and other such tasks.

A

There should be no need for a non-admin to be able to access the cluster directly except for admins. Any other roles should not be cluster-wide roles. They should be configured with respect to specific cluster resources.

A

Another solution is to assign completely different cloud accounts for different environments, such as development stage and production.

A

This way we can keep entire environments safe from rogue and accidental events.

A

We will now move on to our next vulnerability factor, that is, network policies for namespace restrictions.

A

Kubernetes network access is completely open between namespaces and pods in the default setup. This is not suggested in deployments where applications can access infrastructure tools which have high level permissions to the cluster management. Services like spinnaker should be specifically restricted from other applications, except for specific rules which are required to perform daily tasks such as triggering bills and deployment pipelines to fix this. A network policy engine such as calico needs to be installed in the cluster and via network policies.

A

We can configure granular access rules for the services management services should be configured in separate namespaces and access needs to be restricted based on them.

A

In this example, we have restricted all other incoming access to the spinnaker pods in the spinnaker namespace, except the jenkins board, from the build management namespace on port 8084, which is required for spinnaker api access.

A

All access from application, namespaces should be restricted in a production environment and only required access from other management services should be allowed.

A

So we move on to the next factor in our infrastructure vulnerabilities, which is the s3 bucket used for storing spinnaker data, the storage location for all spinnaker data is s3 or object. Storage bucket.

A

There have been too many recent cases of information leaks from s3 buckets for them to be ignored. We keep seeing them the news.

A

Spinnaker buckets contain great deal of infrastructure cluster metadata, which can be beneficial to exploiters in the exploitation methodology. There is an entire stage called information gathering where any kind of information is important to be familiar about. The target system for the attacker in such a scenario, important cluster metadata in the s3 bucket, is extremely critical for the attacker.

A

The fix for this is to keep the bucket completely private and enable encryption on the bucket data.

A

Also, there needs to be specific iam permissions, allowing access from the spinnaker instances to the s3 bucket.

A

The im policies can be attached, to instance, profiles for the management worker nodes, which can avoid specifying access and secret keys on the halyard instance, although we cannot completely avoid mentioning credentials in the halyard deployments, because there are a lot of components like helm and other other components which need credentials the lesser we can keep the number of credential the better it is for us.

A

Our final infrastructure vulnerability factor is the console and api access endpoint, that is, our application load. Balancers, the spinnaker console and api endpoint is one of the key feature of the application.

A

It allows us to perform all configurations and trigger deployments using the console or the api access method. An exploiter can run any kind of manifest on the kubernetes cluster.

A

There are a couple of things which we need to keep in mind here: the authentication and the traffic encryption. We will look at authentication at a later stage.

A

The authentication is configured at the spinnaker level, but when we create the load balancer, it is highly critical that we configure traffic encryption by enabling ssl certificates on the load balancer.

A

This prevents data leaks, such as credentials for performing actions from unencrypted traffic. The process of adding ssl certificates to load balancers is quite straightforward for all cloud platforms, as it is recommended for any public facing application. So it shouldn't be that that difficult now that we have looked at some of the infrastructure level vulnerability factors, we will now look at some application level factors.

A

These are factors which are configurable directly on spinnaker and relates to its communication with the target kubernetes cluster.

A

Authentication in any cloud service is undoubtedly the very first gate to the castle. It goes without saying that without authentication we are basically inviting trouble, often in internal deployments.

A

It's tempting to carry on without authentication, as it is just easier to do, tasks that way, and we often assume that we do not have problems as in public facing networks.

A

The primary thing that we need to keep in mind about the integration with spinach and kubernetes is that if someone has access to spinach and if spinach has wide range access to the kubernetes cluster, the attacker can not only manipulate the application deployments but also run. Custom manifests on the cluster, which manipulates the system resources as well.

A

It is just a matter of time before the entire infrastructure is compromised.

A

Spinnaker provides number of possibilities to configure authentication and most of the major providers are covered.

A

This way, we can not only make sure that each user is authenticated, but we can also configure it to provide access only to members of our organization.

A

In this example, the github provider is being used with a github organization to provide access to all members of only that organization.

A

Next, we will look at authorization.

A

Authorization is the next step to restricting access for entities by default. All authenticated members have access to all resources in the application.

A

You definitely wouldn't want developers from different teams creating deleting and executing pipelines in applications as and when they want also for an external attacker who has somehow managed to gain access to the application.

A

It would be just too easy to get to the next stage in completely automated delivery pipelines, it would not be required for non-admins to even log into the spinnaker console.

A

However, if non-admins manage to login using the github authentication mechanism, it is critical that they are not allowed to manipulate resources.

A

Like authentication, the authorization workflow also has quite a few options in spinnaker once the authorization is configured and enabled we can configure applications with restricted access based on groups configured in the provider such as github organization teams in the github provider.

A

In this example, we have configured groups called lfe, devops, admins and lfe devops developers who have different permissions, provided in spinnaker lfe devops admins can perform all operations, whereas lfe delay of devops developers can only read resources.

A

Any attempt to write configurations by developers results in an error.

A

We are now left with service accounts. There are two service accounts that we need to be familiar with: the kubernetes service account and the spinnaker service account. They should not be confused with each other.

A

First, we will look at the kubernetes service account.

A

When we add a kubernetes account in spinnaker, a kubernetes service account also needs to be created which is used to perform operations on kubernetes by spinnaker. By default, the service account is attached to a role with cluster admin. Permissions.

A

This step provides the capabilities to perform all actions on the kubernetes cluster. Most of the time, this is not needed, as penica does not need to perform actions related to service accounts, roles or role bindings.

A

Also, it is possible that we would want to restrict spinnaker operations to some specific namespaces. We would not want it to manipulate resources in the system namespace.

A

In such scenarios, we need to devote dedicated time to create granular role and account configurations for spinnaker, using the rbac or based access control capabilities of kubernetes.

A

This will enable spinnaker to do its usual work without being a risk to the cluster and also its administrative resources.

A

Finally, we will look into spinnaker service accounts. Spinach service accounts closely, coupled with authorization groups that we have. We have configured before spinnaker service accounts are created by assigning authorization groups to them. Only users who are member of those groups can use the service accounts in the pipeline triggers, thus blocking others.

A

As of now, the service accounts need to be created via the fiat back-end url, as there is no way to do it from the console or via halyard.

A

In this example, the user accessing the console is member of the group attached to the spin admin sa service account, which is why the user can use the service account in the pipeline trigger another alternative way of doing. This is by using pipeline permissions.

A

When a system is breached by an attacker, a lot of time is spent in the system trying to gather information, analyzing data and trying to escalate privileges, which does not just happen over minutes or hours.

A

The admin or operator would not even detect the person snooping around in the system.

A

Access logging is another critical part of the security workflow which enables us to track such behaviors as mentioned before, implementing and configuring all the factors mentioned in this session might not make the deployment entirely resistant to breaches, but it will definitely make it a lot difficult for rogue entities to be able to exploit it.

A

I hope the information provided in the session will be helpful to you, and it might also inspire you to design secure workflows for other infrastructure deployments as well. Thank you.