Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2021, 29 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Secure your cluster-to-cluster traffic, the agnostic way - Pauline Lallinec & Dave Kerr, Workday

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Secure your cluster-to-cluster traffic, the agnostic way - Pauline Lallinec & Dave Kerr, Workday

Workday is shifting to a multi-cloud approach whereby its Kubernetes platform known as Scylla can be deployed to public cloud providers as well as Workday’s own data centers. To achieve this, we needed to route tenant data across existing AWS clusters in different regions, to Workday’s own data centre, and potentially in the future, to other public cloud providers. While cloud providers usually have solutions to migrate data to their own cloud, Workday aims to be cloud-agnostic, and as such needs a solution to migrate data cross-clouds. The infrastructure, platform, and application development teams cooperated to develop a solution relying on Kubernetes operators, Istio, Consul, and Helm-delivered application configuration. This talk will give an overview of the tools and technologies used to migrate tenant data to other clusters, wherever they are deployed. We will additionally review the learnings from this experience and give an overview of the future work.

A

Hi everyone. This is pauline and dave talking from kubecon about securing your cluster to cluster traffic, the agnostic way dave and I work for workday, which is a provider of human resources and financial system.

A

A couple of years ago, workday decided to expand its offering by deploying workday on the public cloud, and for this we built a platform based on kubernetes, which my colleague dave is part of this team, that is, building kubernetes platform for workday.

B

Thanks pauline hey folks, I'm dave care, I'm a software engineer devops on the silhoue public cloud team. Here at workday, the sale team is focused around building uh workday's kubernetes based cloud platform, and you can usually find me doing things in the gaming tabletop and ctf space. If you want to reach out or just talk, you can find me at davecare95 on twitter.

A

And I'm pauline I'm an I'm an engineer in the public cloud infrastructure team after I spent a number of years working in the same team as dave um when there is no pandemic, you can find me in a karaoke room and you can reach out to me on twitter at filalin.

A

So what to expect from this presentation? Well, we will give you a use case for a provider agnostic cluster to cluster communication. We'll give you an overview of a solution to secure your cluster to cluster communication across providers. We will give you a story about what they incrementally implemented. That's all that solution will give you an overview of the custom, kubernetes resources and operators developed in order to implement that solution and at the end of the slide deck you will find leaks in order to go further by yourself.

A

Technologies discussed in this talk include kubernetes, of course, but also helm and voi console aws and istio. But first, let's see why we need cluster to cluster communication with my colleague dave.

B

Thanks pauline so closer to cluster communication, why do we need it? Well, let's look at a sample service. Its name is ox. It's a service living in cluster one in data center one, and sometimes it wants to send messages to his family and friends in cluster 2 in dc2.

B

In the old days, ox would use the public-facing load balancer to reach its family and friends, but, as we know, securing public gateway. Ingress is hard and the teams had to maintain complex configurations for these cross cluster communications. This was not sustainable for the platform and engineer platform and infrastructure engineering. Folks who needed to maintain these complex configurations for any new providers and new regions that we may need to onboard.

B

So what would make the service team, the service and the platform and infrastructure engineering folks happy well being able to deliver secure, easy to configure cluster to cluster communications?

B

So let's look at some of the use cases at workday. We have use cases where services may need to move data between different services in different workday clusters, and these services may be deployed in specific clusters based on their workload, type cluster type environment type, or they may even be running in kubernetes clusters, not running on the kubernetes platform at workday, but managed by those teams in their own cloud providers.

B

A sample use case here is moving tenant data between workday instances. The service needs to be able to move tentative data between these instances, and this relies on internal apis that we shouldn't be exposing on the public gateway. To talk a bit more about this, I'm going to have pauline look into this legacy. Use case.

A

Thank you. So let's have a look: how cluster to cluster communication worked in the legacy world. So, as I mentioned, we had we. We had communications that were private to communication between data centers, but they were set up a dock and they didn't really scale because, as you added more data centers to the workday ecosystems, we needed to, for example, whitelist a lot more ips than we will than we used to. This was okay.

A

When we had a limited number of data centers, but what, as we opened new regions and new clusters in the public cloud, this became harder to manage. Additionally, we were using a public-facing load balancer both for public communication and internal communication.

A

That load balancer is one that we call legacy load balancer internally, because we want to retire it in favor of histo. The reason why we want istio is to prepare us for the next level of scale at workday.

A

There is also a lot more reason why we chose this to replace the legacy load balancer and if you want to learn more on the implementation of pceo for public ingress at workday, we have a link at the end of this slide to another talk of us when we talk a lot more in depth about why we chose this year so first, my colleague dave will tell you a bit more about the use of istio.

B

Thanks pauline for covering that use case, so yeah. Why istio well? uh Well, apologies so yeah what what is seo istio is an open source platform, independent service mesh that provides traffic management policy enforcement and telemetry collection. It was developed by google ibm and lyft, and it's a control plane for envoy proxies it lets. You manage your networking resources as a set of custom, kubernetes custom resource definitions that deliver solutions for ingress, egress and in-mesh traffic and istio control. Plane itself runs inside the kubernetes cluster.

B

So, as I mentioned, istio is a control plane for envoy proxies. An envoy is a high performance proxy designed for cloud native applications. It was developed by lyft, it's open source like istio, it's cloud agnostic and in istio's deployment of envoy, also known as istio proxies. They are deployed as sidecar containers.

B

Alongside your main application container in your own workloads, you may be deploying sidecar containers for things such as metrics, monitoring or logging, and you can think of envoy, proxy or istio proxy in this case, as a sidecar for networking managing all your main application containers, inbound and outbound traffic.

B

So why did we pick istio and envoy, as you can see, from this cncf survey from 2020? Istio is one of the most used service, mesh solutions being run in production today by many large companies and, as I previously mentioned envoy, is the underlying service proxy for istio. So it only made sense from a development standpoint that we stick to a single technology in order to reduce the number of technologies needed need in order to reduce the number of technologies that an engineer is required to know in order to run and maintain the platform.

B

So that sounds great and all, but how does the little locks actually use these technologies to send messages to its family located in cluster 2 in data center 2.

A

So, for example, we have that user who is trying to reach the internal uri, that user is not a valid workday server and therefore the public. Well, the load balancers that we use will refuse the communication and send an error. It will block that communication, um but when the communication is coming from a valid one-day server, then the public load balancer we'll know to allow this communication and we'll service it.

A

When we move to istio, we where we found ourselves in a situation where internal uaris were either blocked for everyone or a load for everyone, there would have been a way to say that you know a sample of workday servers are allowed to query this internal uri and no one else, but as we knew from experience with the legacy load balancer, maintaining that configuration it takes a lot of time takes a lot of resources and we did not want to carry over this piece of legacy configuration additionally using the same gateway of load balancer for both public communication and private communication is something that we need we wanted to address.

A

So, instead of using the public gateway for private communication, we decided to implement a second eco gateway, specifically for private communication, so that we have a public gateway for all public communications that everyone can reach and the private gateway which is not exposed publicly, which is only accessible for a subset of the workday services.

A

The way the communication works is that we were using aws private link. Aws private link is an aws service for securing communication between two aws data centers. So we would have a server located in a cluster in aws data center 1, sending traffic to aws data center 2 via the aws private link.

A

So, finally, the little rocks can talk to its family in aws, but this talk is called: securing your traffic, the agnostic way and now we're talking about using a service which is available in aws and nowhere else. um So if you were thinking that this talk is fake news, because we are not going to be agnostic at all, you would be correct. However, we use aws private link as a stepping stone toward a truly agnostic infrastructure, and, if you be with us, you will learn how we got to retire.

A

The aws private link in favor of another solution, which is truly agnostic on to you dave.

B

So, as paulie mentioned, the name of this talk indicates that we should be showing you a cloud agnostic design for closer to cluster communications, but of course, we had to start somewhere and that use case was to cover aws to aws connectivity. For this we developed the aws private link operator.

B

They west private link operator is a golang operator built on top of the operator framework and delivers two custom resource definitions, private link, egress and private link ingress. We also have a service, the private link operator that lives and runs inside the kubernetes cluster and watches for creations of either of these resources. Once they're created the necessary aid. West resources are realized within the aws data center that the cluster lives in.

B

So how do teams go about delivering these cluster cluster communications? Well, one of the key things here was that we ensure that service onboarding is as easy and as automated as possible, so we don't want to have to be including platform, engineering and infrastructure folks. Every time we need to deliver one of these uh service to service communications configurations, and we also want to be sure that teams can deliver this easily, so we ensure that they use a mixture of helm, template configurations as well as per environment configurations.

B

So how do teams deliver that private ingress uh private link, ingress configuration that we saw earlier well service teams deliver that as a raw or templated kubernetes resource, where they just specify the aws resource name of the account where the cluster that they wish to allow ingress from lives in so in this example, you can see they define a private linking address. They give it a name in this case cluster to ingress private link, and they give it a name space, private links, then down below in the authorized account section.

B

We provide the aws iron or account or amazon resource name of the account where cluster one lives so in this case we're allowing ingress from from cluster one into cluster two.

B

So now we know that so now the ox knows how to send messages to his family. But how can we sure that he received them? Well? Introducing the private gateway that paulie mentioned earlier, so teams deliver custom, istio virtual service configurations in order to uh be responsible for the routing for that service. Service teams have the flexibility to add and remove routable paths, request headers with a single pull request which would be later delivered to production and they follow an existing private link.

B

Fully qualified domain name template that we provide for them, and this is easy to test with standard kubernetes tool, trains such as cubesat and helm.

B

So if you're unaware of what an istio virtual service is, it's essentially a resource that describes how network traffic should be routed around your cluster and in this example, we see we provided a name ox and private coms virtual service. We provided that template at host that we provide a template for so in this case, the fqdn or fully qualified domain name of the associated with the private link is oxen.workdayprivatelink.com.

B

It's deployed in the private gateway as paulie mentioned earlier, and then we allow two roundabout paths: that's slash internal api and slash secret aux family stuff. So if a service in cluster one wants to route to auxin.workday.privatelink.com forward slash one of those routable paths that will then be sent on to the destination service in cluster 2, which in this case is the ox family service living in the oxen namespace. In that cluster.

B

So did the ox service mention that their family is also quite large and that they can be spread across many data centers? Sometimes they may not even know if they live in an aws data center or or not.

A

So we have covered how to secure your communication from an edible u.s data center to another aws data center. But one day strategy is to be multi-cloud and securing the communication from aws to aws is a good start, but it's not sufficient.

A

For example, we may need to secure the workload between workday's own data center and aws, and you may have seen it in the news that the strategic partnership has been signed between workday and gcp, so in the future, it's possible that we'll also need to seek your communication between workdays on data center and gcp, or even between aws and gcp. So how do we do that? Well, for this we are leveraging the power of console, console automates networking for simple and secure application delivery.

A

It has been developed by hashicorp, which is which has joined the cncf foundation just last year, it's using enboy as its proxy, just like its ceo, and it offers a solution for multi-platform service mesh which is true very important to us, because we want to be able to deploy the same mesh everywhere and it runs inside or outside the kubernetes cluster.

A

um From our first solution, which was based on the aws private link, we keep the routing of the traffic within the cluster, using this, your private gateway and we keep helm for packaging and delivering configuration for.

A

For internal traffic networking we introduce a console cluster which runs outside the kubernetes cluster in order to secure data center to data center communication.

A

So why did we choose console for this? Well, it's cloud independent and, as I mentioned, we want to be able to deploy the same solution in aws and in work days on data center, and, if it's necessary, we want to be able to deploy the same solution in gcp or other clusters or other cloud providers.

A

um The fact that it's independent from kubernetes is an advantage for us, because we have applications which aren't running or will not run or aren't yet running on kubernetes, but they need to be accessible, and so we wanted to them to be part of the mesh.

A

The fact that conceal runs outside the kubernetes cluster allows us to link applications which are running outside kubernetes to the applications that are running within kubernetes and finally, um console is using envoy, which is the same proxy as histio and for us it's very important, because, even though we use um istio within the kubernetes cluster and council outside the kubernetes cluster at its car, it's still an environment. To enjoy communication, we are not changing the proxy.

A

It also means that the experience that we have from running envoy with istio we can reuse some of that experience to run and boy with console. um We are running envoy everywhere, which is an advantage to us.

A

There are other advantages into implementing a solution that is platform. Agnostic services may be exposed from any data center. If you have a service that is running in a data center and that you need it to be available everywhere, then, if you don't communicate within data centers, that service will need to be deployed in every single data center. In order to be available, the fact that you have data centered data center communication allows you to deploy your service in a subset of your data center and to have it available everywhere.

A

You may link services offered by different cloud providers, say you have a service running in angular us that you want to use in another clock provider, it's no problem and finally, it's easier to deploy more services across data centers or providers. Since you are basically deploying the same service everywhere. If we had remained with aws private link, then we would have needed to achieve the same solution differently in our own data center and in other data centers public cloud providers that we are interested in.

A

But we are using the same solution everywhere, which makes it easier to deploy um some notes about console administer, so it's entirely possible to achieve what we're doing with vco using console and the other way around. We could have used console to achieve what we are doing with this year.

A

However, in our setup, estio is a lot closer to the application level than console is, and there are some of the ecos features that we wanted to make available to the service teams. Those feature may have been available in console, but with more with a more difficult setup, estio makes it easier for us to use those services for the service team, and likewise when we were reviewing uh the documentation for data center to data center connectivity, we found that the consul had better documentation when it comes to running it outside the kubernetes cluster.

A

So we decided to go with console for cluster to cluster connectivity.

A

And finally, they both use and poi. So, even though we are using different technologies to implement envoy, it's still envoy running everywhere. So in summary, eco is our choice for our kubernetes platform. It's easy to implement in kubernetes.

A

It comes with a lot of interesting picture for the service teams deploying on the kubernetes platform console we use it to secure dc to dc communication console comes with a mesh gateway way, which only purpose is to secure data center to data center communication. So this intradc communication is a use, is a core use case for console.

A

So when you deploy console, you will be deploying a number of servers. um You have a service catalog, which is offered via a fleet of console servers. You have an ingress gateway which is intercepting the traffic incoming to the console mesh.

A

You have a terminating gateway for the traffic leaving the console mesh, for example, for the traffic going to the seo private gateway, and you have a mesh gateway for data center to data center communication when services, one in dc1 needs to talk to services too in dc2.

A

What happens in console is that the console agent will query the concealed server to know where the service is. If that service is found within the same data center, the query will be rooted via the ingress gateway of cancel.

A

If the service is not found within the same data center, the query will be routed to the console mesh gateway, the mesh gateway novel where services are located and it will root it based on the server name that it gets via snipping. The sni, which is part of the tls stack.

A

The traffic is routed to another mesh gateway using neutral tls, to establish truss and the mode. The mesh gateways are not decrypting the traffic. At this stage.

A

In order to leverage the use of the mesh gateway, we have created another operator which is called the comms operator.

A

The comms operator is looking for custom, kubernetes resources of type private, ingress or private egress, and when it finds a resource, it will create the relevant service definition within the console catalog, and it will create the relevant ico resources to allow for proper routing of this service within the kubernetes cluster.

A

So if we look at kind of the bird eye view of our cluster to cluster communication works, we have ox in the data center number one which is sending uh queries to aux in data center number. Two, the query will be rooted within the cluster via the internal istio gateway, um the istio gateway will send the traffic to the console mesh.

A

The console mesh will send the traffic to the mesh gateway, which will send the traffic to the mesh gateway of console located in dc2 from there. It will reach the internal istio gateway of the kubernetes cluster living in data center number two. While it will eventually be routed to the ox application living in the second cluster in dc2.

B

Okay, so now the ox has the ability to contact its family and friends regardless of what data center they live in public cloud, private cloud or even a cloud provider. We don't support you all they have to do is deliver simple configuration to realize secure cluster of cluster communications, regardless of where their family or friends might live.

B

I think we can all agree. That's pretty great.

A

So, in summary, in order to implement a secure cluster to cluster communication across data centers, we implemented a global addressing scheme so that the teams are the conventional dns to address each other. We implemented a private htr gateway for routing the traffic between clusters.

A

We implemented console mesh for syncuring data center to data center communication, the teams delivering therapy. Sorry, the teams deploying their applications on kubernetes can deliver their configuration via handshakes and we have created operators to dynamically, create, tear down and update ingress and ingress as required.

A

Thank you. Thank you.