Cloud Native Computing Foundation Cloud Native Security Day North America 2020, 4 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: A Tale of a Meshi Kafka: Securing Kafka Deployment When Istio Is... Ariel Shuper & Nikolas Mousouros

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

A Tale of a Meshi Kafka: Securing Kafka Deployment When Istio Is Used - Ariel Shuper, Portshift & Nikolas Mousouros, Marlow Navigation

Kafka is a commonly used message broker for microservices real-time data feeds. A standard setup allows any micro-service to read or write any messages to/from any topic. The need for security typically starts when multiple applications use the same Kafka broker in a cluster, or when confidential information is shared in the Kafka topics. Common security practices to use are authenticating subscribers and publishers, authorization policies for access control and data encryption. When Istio is being used with microservices that access Kafka topics, the envoy proxies is expected to offload these security elements. However, creating sustainable and consistent authorization policies when Istio is deployed isn't feasible, and tracking the microservices based on their IPs isn’t feasible because of their replacement. The session will present how to build an external authorization mechanism and simplify policy management for Kafka topics by using open source tools, like OPA and others

A

Hello, everyone and welcome to our session uh a tale of meshi kafka. um My name is ariel shuper uh by the time of the recording I'm in port shift, uh but I'm by the time of the when we hear this session, I'm gonna be part of cisco.

B

Hello from every other as well, I'm very happy to be part of this event. Today,.

A

Thank you, nicolas. um What we're going to talk to you about today, we'll tell you who we are a little bit more details.

A

I will talk about kafka usage in general and in kubernetes, specifically we'll talk about the security architecture or how do we secure kafka clusters, um we'll talk about kafka in istio or any type of service mesh, and then nicolas will talk about uh marlon architecture and how do they secure the deployment uh and will share with us uh a great demo a little bit about myself?

A

Who am I so my name is ariel schuper, I'm a principal product manager at cisco, uh used to be a vp product management in port shift port shifted the cloud native security vendor uh before working in port shift. I was the head of serverless solution at aqua security, um and before that I led the cloud security solutions in checkpoints of technologies, uh part of uh open source uh with the the qb project that we in port shift created and also a member of the easter security working group.

A

um A little bit about port shift. We are were founded by 2018, uh then uh acquired by cisco, uh beginning of november, beginning of this uh beginning of this month, uh focusing on cloud native security platform and, more specifically on integrating it with service mesh, any type of service mesh, and that's it about it.

B

I'm nicolas museros I worked in telecommunication, currency exchange, commercial companies and now I'm glad to be part of two organizations. As a devops engineer, the first to the first one is marlo navigation, and now I'm trying to start my own startup business, rtx direct, which is providing cloud native services.

B

Mobile and commercial ship management company offices in over 10 countries over a thousand sure based employees and over 13 000 employees on board at any given time. So let's go ahead with ariel.

A

Thank you nicholas and let's talk about get into the details and let's talk about a little bit about microservices communication, so we are moving to microservices. Okay, microservices um usually look like that. We, when we're working in distributed applications, we start breaking down monolithic into small components. We want each of them to communicate with with each other. So we are creating.

A

You know a nice communication uh schema, but then, eventually, when the cluster grows- and there are more uh more elements, we can quickly turn into this famous uh diagram that shows the lift micro services communication before they started the project with envoy before they moved to service mesh um and the idea that you know when we have lots of micro services. A lot of communication, I'm not going to talk about whether or not uh service mesh is the right way to choose it. But we can talk about lots of microchips communicating with each other.

A

We can use it like the mesh in synchronous mechanism that everyone can communicate with everyone synchronically, but you can also use uh asynchronous communication or security message. Passing with. There are multiple options: uh kafka, pachi, kafka, just a popular one, but there are many other options of how we can use event streaming between services, which are not synchronous.

A

Now we would like to talk about apache kafka. You know as a popular uh event streaming mechanism. uh Apache kafka was, you know, becoming a pop. Very popular tool was donated by linkedin to the open source community in 2011 as a message queue quickly turned in to be like an event streaming, so we're not just making like some simple computation uh actions on messages. We can run multiple actions on multiple messages simultaneously.

A

We can also maintain uh persistency with keeping up the message. There are many benefits to to apache kafka and to the way uh it is used today for asynchronous communication. Now our focus today and nikolas will uprise it a little bit more. When you talk about architecture is on an open source. Distribution of kubernetes called strimzy and strimsy goal is to simplify the process of running kafka in kubernetes.

A

uh It provides you know the relevant container images uh streams. You also created uh dedicated operators uh that can run the you know the kafka cluster and can add a lot of uh support or a lot of uh simple to simplify uh those these operations and to make it.

A

You know more cloud native way by automating a lot of those uh things um and the cafe component is run as a cluster, so it can be, like you know, for availability purposes um and, as I said again, uh nicolas will explain more about the benefits of using it.

A

Now when we talk about kafka- and we want to talk about- you know the different security changes. So how do we secure this environment?

A

um It's important to know that kafka does require some level of security by by nature the default setting is that you know um the different configuration allow any user uh to read or write or publish or subscribe of you know any or all the data. So you can, you know, publish your type to any topic, so you get a full exposure.

A

um The communication is in plain text. If you don't do anything specifically and you know, and you go and configure the tls uh between your users and their brokers, the communication is in plain text. So if someone can intercept it in the cluster, I can get exposed to all the communication. There's no need to you, know decrypt. It users can delete the data and in some distributions, uh the secrets or credentials are stored in plain text. So you want to restrict the access of people who can access those?

A

You know uh the zookeeper uh location, so by default. uh There are many, uh I would say, security challenges, but you know, uh given the maturity of the product. A lot of them, you know, were taken care of were handled uh more specifically uh talking about streamsi kafka. Looking the main security building block, so the authentication um use user authentication can can be managed, kafka, listeners use authentication, so they can ensure secure client connection. um It does support different authentication, different authentication options, uh there's a dedicated user operator that can simplify some of them.

A

But, as you can see that you know in practice not all of them are always in use, but at least the foundation or the infrastructure. To use them is available uh authorization.

A

The native authorization in kafka is using the simple acl authorizers uh it's based on authentication of users, and then you allow, if you have users authenticate and identified, you can define access control list and you can define what user can access. What resource encryption in streams is with tls.

A

So the communication is always encrypted in the main control elements like between the brokers, the zookeeper nodes, the operator and the exporter. um You can, you know, encrypting the user traffic. The user communication uh with the brokers is something that that is required. Users, intervention and and making it um with the tls option.

A

So- and I was talking about security in kubernetes committees- offer a rich set of security mechanism, there's a lot of investment, a lot of work and also coming from the maturity of the of kubernetes in its deployment. There are multiple options for security, both for deployments for services for policies, but also for authentication authorization, and you know powerful, role-based access control.

A

Now, when service mesh is used, I provide flexible option to offload out of it. A lot of you know the authentication and authorization decisions. uh The encryption is a completely different experience when using service mesh. So all in all the security posture of cluster is much higher. Now.

A

This lead us to think that when we deploy kafka in kubernetes and with istio um will be in a much better stage, because the kubernetes, the kafka security and the course, although there are you, know they don't match, but if we're using it with kubernetes, um then we can get the benefit of all the existing tools, uh especially you know easter can used uh like it does today can contribute. You know seamlessly to the security level, because it's doing it, you know using offloading the traffic, not touching more, defining or changing anything in the workload.

A

So this would come make a thing that uh the security security situat, the security status, uh can be much better. Now. What we discovered and also together uh with marloy, is that easter and kafka are not. You know the best match, or I would say, uh they're not a match made in heaven and it's time for multiple reasons you know, but when you come to think about it, that you know kafka and zookeeper were both designed to have all the required resources available at startup time.

A

uh In current kubernetes versions, sidecars like the envoy proxy uh availability can today still be after the pod is already running um and as a result, we can see that, for example, in zookeeper it can lead to instability operation that members can't create a for a quorum uh with the brokers.

A

uh If a broker tried to communicate with the zookeeper and envoy is not ready or not available, uh the broker will crash. uh We also saw that in zookeeper installation um it binds it to the pod ip, but nvo uses localhost for forwarding traffic and the result can be connection refused errors and some other uh challenges that uh we we happen to found, and unfortunately, this leave us. You know with situation the still from one side we do deploy the kafka.

A

We can benefit from all those great tools, but the reality is that those are cannot be used. Now. What you know the way to move forward and really secure and benefit from what uh kubernetes and what service mesh can offer kafka.

A

Is you know if you can really make some uh small changes or do something which some have already in the work, but some uh can be done. So what are the requirement? What do we need in order to make kafka and east you a perfect match right, so in order to create what we call a kafka friendly, easter kafka, friendly service mesh, um something that will let us you know to benefit from all the security controls in an automated and smooth.

A

You know for existing users, but also, more importantly, for future users, which are deploying you want to connect. You know with the kafka cluster. There are two critical elements that you know need to be fixed in order to get this uh kafka friendly level uh expectation. One of it which is already in the work, was supposed to be part of 1.18 1.19, but I believe it will uh be in the next in the next version. Definitely going to be included is making a sidecar first-class citizen, uh part of kubernetes. Now.

A

What do I mean first-class citizen, making sure that when a sidecar container like envoy is deployed with every pod, uh you know make sure that the this sidecar is up before the regular containers are up and making sure it's shut down. Only after all, the other containers are terminated and this will make will ensure that all the challenges we discussed in the previous slide will not happen, because envoy will always be there. So we can both zookeepers and the brokers uh can establish their communication without worrying about it.

A

But that's not it there's some a little bit more tweaking that is required, and you know thanks to the flexibility of envoy and istio. This is something which was already available since 1.5, and this is adding a special detection to kafka traffic, either by enhancing the current kafka filter, because today, envoy supports a filter for detecting kafka traffic, but there's slightly more that needs to be done or creating a new filter or a new proxy for envoy, which again also you know, post 1.5 uh is much simpler to do now.

A

As I said before, so one, there are infrastructure issues which are being mean done or managing kubernetes already part of the release plan, and it's going to be, you know, included uh probably in the next version, but there are some you know, modifications to envoy uh which are required in order to make it. You know kafka friendly.

A

um So if we will achieve that, we can run envoy proxy in all kafka elements, whether it's broker brokers, zookeepers subscribers and producers.

A

um If we can use similar mechanisms like could today with http kind of a proxy that allow us to parse the kafka stream and you know, send it for authentication uh and authorization, so we can authorize any request now. This can be done either by enhancement to the current filter or by using the new webassembly uh toolkit, uh which allowed to customize and create uh customize filters for envoy like an on demand, um and then you know, we can use envoy to invoke authorization for every message based on installation properties.

A

We can of course uh authorize it. uh We can create policies, those processes will authorize. We can, of course, cache the results, so it's not gonna be like every request. Only for a new connection or new service, which is something that will allow us, you know to benefit from a fully authorized uh mechanism that allow users to define the rules uh envoy can enforce it can create it. Just like we do today with our http communication.

A

Now anybody can also pass authentication information to kafka. Authorizers uh an envoy can seamlessly encrypt uh encrypt and manage all the certificates. uh You know of the cluster of the brokers of the users. There's no need to work on the tls certificates for the users. We can also take it a step forward and do all the ingress in the egress communication. We can manage them and using the service mesh policies so with having the right filters uh and some little to help. We can really reach level of very kafka, friendly istio.

A

That will allow us to benefit uh from all. You know the inherent security which are included so how's. The structure is going to look like, um so this is going to be like we're going to inject the istio or the endovoy proxy to consumer pods, producer pods and the kafka broker. The easter control plane uh will uh make sure that everything is encrypted, and you know the the envoy will encrypt the traffic.

A

uh You know get the and and of course, uh rotate when it's rotated, but can encrypt the traffic as soon as leave the the uh consumer um or the producer containers uh when it reaches the broker, the broker uh will will forward the traffic for authorization.

A

So I can verify the three results: every new authorization request will be verified, it will be cached and this can be maintained and the traffic will uh talking not not going to impact the the perform. The performance.

A

So, what's going to look like so instead of using the regular authentication, we can of course use the easter based authentication. The android proxy will extract the application or the or the micro service identity uh and forward it and afford it for the authorizer. uh Just like you know it's using it today.

A

uh Just decision will be made locally and will be well maintained locally.

A

Just a second okay authorization will be will be the same, so we can, you know, authorize user business, specific attributes um and then use the android proxy for that uh as well. We can get much flexible, more granular options, setting the rules uh and caching those uh results.

A

Encryption can can spread around the entire cluster, so, instead of keeping it uh only specific to the control, we can run it. You know all the clustering trip, all the traffic between the micro services uh and the brokers or even the internal everything can be managed. You know rotating certificates, uh it's gonna be a much easier and simpler uh task to do so. All in all uh with having uh with you know, making those changes you know we can uh make istio a much friendlier to kafka.

A

We can then once we deploy together, we can use all those mechanisms which today are not very advanced in kafka. We can use the easter mechanism and then we can bring the kafka to the same level of security like any other regular workload, that's running in in kubernetes, so we can really benefit uh from both worlds together.

A

But that brings us to the question: what do we do until that? uh And until we have uh is your friendly? What can be an intermediate solution? And here we won't talk about what we are using today and today we're using the open policy agent, so the open policy agent uh to do all the micro services authorization uh just on a nutshell: the open policy agent, um it's a popular tool, uh it decouple policy decision making from enforcement decision.

A

Okay. So when we use open policy agent with kafka, uh we can we need to use a uh an opa plugin uh inside. You know the streamsi kafka and this plugin can redirect or make authorization requests to the oppa server. So when opa is using with kafka, the kafka authorizer call to the plugin.

A

The opa server uh to evaluate the policy based on the input from the authorizers uh input is the same set of information, just nothing change and then in opa server people can define their policies and the oppa can evaluate any request uh based on the policy and respond to the authorizer uh with a decision, whether the whether the request is allowed or not and decision are cashed by the authorizer to make sure performance is not is not affected.

A

So this is open policy agent uh in port shift.

A

We use the same architecture and use the same plugin, also to allow people to communicate and define uh their authorization rules, and then the opa plugin will forward it to port shift. Port shift uh will based on based act like just the opa server and based on the predefined rules, we'll verify what users can access uh what uh you know, what um topic uh what broker uh can be accessed? We are focusing on on topics um all the new communication going to be authorized.

A

The micro services are going to be authenticated based on the runtime properties. You know in the namespace, they are deployed their source of region. uh We can use those customized identities and verify that our users are author, authenticated and then the traffic encrypted is used to be done.

B

A

So that's going to be how our clusters not look like um you know. The main issue is the op-up plugin or getting information from the authorizers calling port shift and getting information.

A

uh Istio is not really in place, not in the kafka brokers, not in the kafka pods, but it does being in but used in the kafka pods. But again it's a the current version, which is still need some modification to get it like. You know in full fledge to get from all the benefits uh enjoyed.

A

So this is all from my side I'll hand over to nicolas nicolas. Please take us through what you do in.

A

B

I'm sorry I was stubborn slide 20 on, so I will talk about the marlow story and what happened in the practical side of stuff what marlow is currently using?

B

We are using a legacy system that was built for 25 years. It kept growing and became complex, and, if I can recall correctly, it reached its end of life support in 2003, not only that, but the server side application does everything it receives. Requests executes, domain logic, retrieves and updates data from database and responds back to the client modularity within the application is typically based on features of the programming language.

B

Even a small change of to the application requires that the entire monolithic system is rebuilt and redeployed. It gets difficult for change not to affect the whole system as much as it grows over the years. It gets difficult and complex to maintain and in order to scale the application, we would simply create more instances of that process, and usually it is not possible to scale the components in primary.

B

So moving to cloud native micro services and google networks micro services with the google nethers natives deployments, we can get view on different metrics, such as cpu usage and ram usage. The horizontal product scalers give us the ability to scale the number of instances in a replication controller or replica.

A

B

On those metrics, we can also use the same metrics, alongside with shell checks to vertically scale our infrastructure when needed, even that we prefer scaling horizontally our infrastructure as well and with the introduction of githubs deployment, configs sharing, charts, we save a lot of time. We have better versioning and obviously much easier rolling updates.

B

So a bit about our this is a small representation of how our system works. We still need to get data from our legacy system, since we are still in development, so we have stateless apps in blue color that issue commands and events, but we still need to keep the state somewhere. So we decided to use kafka and other storage services.

A

B

B

Given all that we need and given that we needed to move forward to a hybrid environment at some point, which is a lengthy subject that I don't want to get into today, we decided to move from managed kafka services to streams, streams.

A

B

Freedom in so many ways, for example, leveraging all the kubernetes concepts that I already talked about. We are able to use githubs to to easily deploy a ton of kafka. With a few clicks, we parameterize a lot of the configuration it's pretty much secure in comparison with other services and has a great community that is always willing to help.

B

The problem with such environments and traditional firewalls is that they need skills to get configured well, especially on cloud-native environments and complex networks. Imagine having to secure a whole kaka cluster using regular expressions and having almost zero visibility on the request made.

B

Furthermore, firewalls need a lot of fuel to get configured which it uses the agility and the acceleration of all the development which is bad for everyone.

B

We also sometimes need to give or block access on different layers, such as on the micro service level and, more importantly, on the topic level. Also, given that a lot of times now, with the corona virus days, we need to work on home.

B

They need to secure the environment, for specifically, peace came up and as much as we move forward configuring and securing streamsy, we realize that we need something more sophisticated from managed cloud services and that's where courtship jumped into the game to save our lives.

B

So the port shift solution gave us visibility on what requests are made with tables and nice graphs. We were able to set up rules on a very user-friendly environment. We are able to add rules for specific microservices and specific topics and moving forward. I will show a small demo of how easily those stuff can be done.

B

So on that demo I will be using arco cd, which is a great tool for githubs.

B

Another great tool is lenses that is great for visualization of data in kafka among other stuff, but that's what I will show today. Obviously I will use openshift. We use our kubernetes cluster streamsy and our security solution, which is spot shift. So let me share my.

B

B

With argo, I will create a new app. I will give an application name afghan demo. I will select the default.

A

B

I will select auto, create namespace. I will choose my git reaper and choose the root path.

B

I'll choose my cluster and the.

A

B

Name I want, and after creating and sensing that after some time we will have a full, workable kafka cluster on openshift.

B

So, let's see what happened.

B

We have embroidered zookeeper lenses and the operator and the kafka connect let's head.

A

To lenses to see what's.

B

Happening here.

A

B

Have some three topics with mock data created? Let's see what's inside.

B

So some sensitive info of our customers, some.

A

B

Information and some iphone numbers all of those are mock data. Don't worry.

B

And let's head to post shift to see what we can.

B

Do we can click on policies and then connection rules?

B

Let's start adding.

A

B

I will select by and put the nssa port name. Then next I will select by kafka. Here I can see my brokers and the cluster. I will select the network that we just created. Next, we will select kafka. Now we can see all the topics that are used in this broker.

B

I will just select the iban numbers and the sensitive information.

B

And I want to block access for reading and writing on those topics.

B

A

B

And then block the axis when I press finish and apply the policy.

B

I will head back to lenses and after some time I will lose access to it.

B

So I cannot have access to those topics, but I still have access to the custom information that I want access to so back to put shift.

B

I will add another rule. I want to block access to eyepiece for everyone in the world, so I'll put zero zero, zero, zero, easy as.

A

B

A

B

B

I'll choose a name, len says in that case next and incredible, and then.

B

B

Right click finish and then apply the policy. I can't get back to learning this. We need to wait a bit for it to work. Let's see what happens.

B

A

Where it's going to.

B

A

First, refresh.

B

But let's see the second one.

B

Now it's blocked. Let's try again to be sure.

B

A

B

Want to be able to work from home, so I will add my ap to be allowed. So, let's select by ap, I will play it out for now. So you can't see my p, I'm adding my ip and 32 for just one ip.

B

Click next again,.

A

With the same name,.

B

Clicking next any layer, 7 protocol that now allow, I will add a name.

B

Finish and now, because I want this rule to have higher priority, I can easily move it up. So let's apply the policy.

B

A

B

Refresh now we have access to lenses, but we still don't have access to the topics we blocked. So that's good. So, let's head back to podchick to see the rest of the goodies it give us has a nice dashboard with some risky workloads. Events connection, how much portal running permissions fascinates.

A

B

That's kind of a nice diagram of what's happening now we can expand this and we can see all the external connections and all the internal connections, so one data connect connecting to the brokers and lenses connecting to workers and some ips from the outside connecting to lenses. Let's go to ci.

B

We can see all our images here. We can see some vulnerabilities and how many there are in every image we can click on the critical.

B

We have a lot of information here, such as the description and we can click the exact cv to go to the cv page and see what's happening with that. So that's nice as well, and then we can go to the runtime where we can see all the workloads running on our cluster and some more information.

B

The result, if it's allowed and then we can see all the connections happening right now. As you can see on the right side, we can't even see which topics are accessed by which workload and finally.

A

The risk assessment.

B

Of our system, we can select the scan time or schedule it. We can select the severity to report and if we want, we can select the namespace we don't want to right now. After we click save the scan is starting and after some time that I cut out it's finishing and showing the vulnerabilities.

B

So that's me, that's it for my side heading back to ariel. To summarize the presentation.

A

Cuny colas, thank you for this uh great demo uh and thank you for showing everything I know. Just to summarize everything um you know kafka. Even when deploying kubernetes does require dedicated security tools, it does not benefit or inherent security mechanisms which are deployed.

A

East 2 cannot be ideal candidate for man to do it or to achieve this level.

A

Some of the work in order to make it it's not natively out of the box, some require will be enhanced in the future, when scica containers will get the better treatment and some require a little bit more tweaking by adding a dedicated filter and envoy that can detect and manipulate the traffic, but with small fixes, I'm sure in the near future, I will be able to use istio and really get benefit from all the security mechanisms and to bring the kafka cluster to the same level of security which allow everyone to use it freely and securely in kubernetes or in openshift.

A

In the meantime, opensort the open policy agent can be used in order to customize a lot of the work but having, but nevertheless, I'm sure in the near future we'll be able to benefit from uh the meshi kafka and make a serving flesh friendly kafka. um So that's it. Thank you very much for joining. Thank you, nicolas, for a great demo, and thank you for doing this for joining and will be available for in the next few minutes for a few questions from the audience.

A