solo.io SoloCon 2022, 5 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: New Gloo Mesh API & Tenancy

Description

SoloCon 2022:
New Gloo Mesh API & Tenancy

Speakers:
Scott Weiss
Architect & Principal Software Engineer, Solo.io

Lin Sun
Director of Open-Source, Solo.io

Session Abstract:
Service mesh provides connect, secure and observe benefits to application networks, yet it could be difficult to adopt for your organization. How does service mesh fit into your existing developer, operator and platform administrator roles? How to onboard teams and have proper boundaries? In this talk, we will go over the Gloo Mesh v2 API to demonstrate how easily teams can be onboarded, services and team can grow from single cluster to multi-clusters seamlessly, how to configure consistent security and networking behaviors for north-south and east-west traffic, and how to share and reuse policies from others while minimizing sidecar resource utilization.

Track:
Service Mesh and Application Networking

A

Hello, everybody welcome to our talk on the new blue mesh v2 api, which addresses concerns with scaling and multi-tenancy we're going to go over how this api has addresses ease of use, scaling concerns as well as self-service and multi-tenancy.

B

We are so excited to be here, introduce the new version of the api. My name is linsa. I'm the director for open source at solo.io.

A

And I am scott weiss, architect and principal engineer at solo, so these are our primary goals. With this redesign we've taken feedback from our customer and combined it with our core three principles that we've had an opportunity to revisit, which are the ease of onboarding. We want users to be able to implement our apis incrementally and have ease doing so. We want our aps to scale well, so that configuration even for large, complex deployments doesn't become impossible to maintain and we want them to be self-service.

A

This allows users to partition, sets of their meshes and their clusters in a way that allows for self-service.

A

So in our design, we've broken down the service mesh configuration and the control plate configuration into three domains, so the first is the admin domain. This allows us to define partitions within our mesh that say which namespaces which resources belong to which teams in the organization then we have routing. This actually defines how we connect together the services within our organization, both within and across workspaces, and then we have policy policies can be applied globally by administrators operators as well as locally for teams and their individual services.

B

We're working closely with many of our customers. uh What we learned clearly is our customer like to have consistent routing api between north-south traffic and east-west traffic.

B

We also learned a lot of our customers, have different teams and different services where they would love to have multi-tenancy and proper isolation among the teams, and they felt the other configuration on resources on the field level is pretty hard to config.

B

So we have some thoughts on that too, and uh what we really feel is our customer understand. The developer community understand how to use labels, and we really want to focus on the business logic and pick up policies based on labels without actually spending a lot of time, learning different resources, different policies.

B

um Last not the least a lot of our customers also mentioned. In addition to deploy guru, mesh resources to the management plan, they would also love to deploy glue, mesh resources to their workload cluster.

B

These are our personas uh pam, uh who is the platform admin she uh loves shopping and her main role is to install glue, mesh and register kubernetes cluster with glue mesh and she's also responsible for onboarding teams within her organization.

B

Oliver, he is a operator and he operates the kubernetes cluster and also the services for the service owner. So he's a very hero for service mesh.

A

Rj is the application owner's owner it's his job, to take the application that the developers are producing and procure something for oliver to deploy into the cluster um uh rj understands the interrelationships between the application and the other software in the infrastructure.

A

um It's also rj's job to onboard developers within the team alice is the developer herself oneoff. It's her job to actually write the code that gets deployed and develop the services.

B

Even when we have four personas here depends on the size of your organization.

B

You may see pam and oliver combined, as you know, one row and rj and alice combined into one row, but the key thing is there is a clear separation um from the left and the right side as far as person, always this probably very familiar to you in your organization.

B

Typically, there is a ad administrator role, there's a operator team that operates services for many teams, and if you put that into the context of services and kubernetes and virtual machine the infrastructure layer, most likely, your team is going to run their services in this kubernetes or vm environment.

B

The services produced by your team, most likely would have some type of dependency. They either be a service consumer or a service producer.

B

Putting this in a simple example: uh in this example, we have web api and also recommendation team. These two team they produce services. The web api team is a very small team, so you can see the application developer and also the application owner is combined into one person.

B

The web api is exposed on the ingress gateway as web api dot istio in action, dot io the recommendation team. It has multiple services. The only service that exposed to outside of the team is the recommendation service. The recommendation service that is exposed on the issue on the ingress gateway, in addition to it's also consumed by the web api service.

A

So digging a little further into this architecture. Let's look at what it actually takes to implement this using istio configuration today. So the first thing we'll need is a gateway.

A

This gateway is going to expose our cluster to external traffic. It's going to define a port on which it's listening, and we see some hosts that it's importing. So we can see here. It's got the uh a host from it web api namespace, which will serve our web api domain to external traffic, but we actually need to then specify a route, and that happens inside of the virtual service.

A

So now we define in the web api namespace a virtual service with the host, where we want to serve this web api application from we specify the gateway we want to attach it to, and then we have to specify the destination where to actually route the request, and we give the internal coupe dns name for that.

A

But this is this is just the basic wiring in order to send communication from one place to another. It gives no guarantees about isolation or performance in terms of giving the proxies the minimal configuration that they need. So in order to achieve that, you would need to add in a sidecar resource would be to strip any unnecessary configuration from the.

A

This is the egress here um we would need to add an export to to the virtual service to make sure that other namespaces didn't pick it up, and we would also need to add some uh our back configuration for istio, that's their authorization policy to ensure that only the gateway was allowed to talk to the web api application and not anything else.

A

um So this is obviously uh just for a single service. We can see up to four pieces of configuration, two of which are required. um It can really get quite complicated once you're, trying to scale this out to a uh an environment that, like we've discussed, is multi-tenant and intended to scale across clusters.

A

So the question is: how can we actually design an api that makes this simpler for our users.

A

So the first step we take a look at is: how do we actually partition all of this config? We have lots of different components, lots of things that are communicating, and we understand that we have teams that need to have access to their own things, but not things that another team is developing. So this is where we conceive of the workspace.

A

The workspace is a logical boundary for a team, and it is an actual partitioning of configuration, as well as services and deployed resources inside of workload clusters.

A

It's the foundation for all of our multi-tenancy features. It makes it easier to onboard teams uh and it provides an abstraction on top of the kubernetes cluster, so it can be scaled between uh single cluster and multi-cluster usage.

B

So the platform team, as pam comes in she's, going to need to set up glue mesh first and then she can start to onboarding her teams.

B

So when pam she can install the glue mesh management plane first and typically using a helm in the production environment, and then she can start to register her kubernetes cluster as the workload cluster by installing the glue mesh agent and also create deploy the kubernetes cluster resource under the cover glue match. Has the capability to install is still on each of these workload cluster automatically for her? So she doesn't have to worry about that. A simple example as to how pam can define her kubernetes cluster.

B

She can specify the name of the cluster. She typically deploy that into the glue mesh namespace in the management cluster. She can specify the cluster domain. She can also put a label into her kubernetes cluster, so that label can be reused in the workspace resources which we're going to touch on very soon. So when pam onboards her new teams, she can define teams per cluster per namespace and she can also even do dynamic teams. So the team can grow as the kubernetes cluster grows by using label selector.

B

As you can see, you know she has a lot of flexibility in terms of what type of workspace she wants to onboard in for her team.

A

So once the workspaces are set up, rj, the application team lead is able to configure settings for the workspace. um This configures things like global policy, what's exposed by the workspace, as well as what the workspace depends on from other workspaces uh once that's set up. Alice can then go and develop services, and we have a nice flow where we can iterate develop and adjust things as needed from within the workspace, without needing to involve operations or admin.

A

So in this scenario, rj wants to answer the questions. How does the team interact with others? This defines how the imports and the exports are set up. um What's important is when can we self-service without involving admin or operations which resources in our workspace, which services are internal and which ones are external, and where are our configurations deployed and tested.

A

uh So once rj has been given um the workspace, which is the environment that sets up which namespaces in which clusters his team has access to and their services will be deployed to um he's, defining some exports here and saying in the recommendation workspace. We are going to export our workspace contents. Our exported items will go to the north south gateway workspace, as well as be consumable by the web api workspace here, and we consider the web api imports the recommendation service so that it can use it and it exports to the north-south gateway.

A

So north-south traffic can reach it.

B

In addition, pam also onboards the gateway team using the north south gateway workspace.

A

This allows oliver to manage the configuration for a gateway within a partition. Just like a team would.

A

So, just to sum it up, we've created now partitions for our different uh parts of our our application stack. We have a partition for our gateway, a partition for our web api and a partition for the back-end recommendation service.

A

So the next question is: how do we actually connect all these together for traffic.

A

um So, let's take a look at the actual resources that are involved in configuring, this in the blue mesh world, so the operator is responsible for defining a virtual gateway. This is essentially a template for a gateway listener. That's going to sit and receive traffic from the external world just like in istio. We don't define the routes there. The routes can be delegated so route tables in our system.

A

They can be defined depending on the use case, they're pretty flexible, so the operator can define a root route table which then routes traffic to configurations defined in different workspaces.

A

It can route traffic directly to kubernetes services, external services which are statically defined outside the cluster or what we call virtual destinations, which are groupings defined by users that combine that basically aggregate services across clusters.

A

However, route tables can also delegate to other route tables, which means that an operator can take all traffic traffic on a particular path or matching a particular header, or on a particular domain and logically forward that traffic into a workspace where the application owners can actually decide where the traffic should be routed within the workspace.

A

So we can take a look here at what a route table would look like potential route table to expose our recommendation service to traffic on a route.

A

So this route table is going to take traffic on recommendation.istionaction.io, as well as on the coop dns, the internal coupe dns name for the recommendation service.

A

It's going to expose it on both the gateway as well as for east-west traffic. uh We can see that with the virtual gateway um allows us to expose it for north-south traffic and the workload selectors exposes it to east-west traffic.

A

So we're seeing here, selecting all workloads in our workspace as well as any workspace that imports this route table and we have one route within the table which forwards all traffic to the recommendation service on port 9080.

A

We can see in this example how the virtual gateway is delegating the role of routing into the recommendation workspace. So the recommendation application owner does not need to go to the operator and say hey. I need some new routes added to the gateway, they can simply update their route table and the gateway will be updated accordingly.

A

So another feature that's exposed by blue mesh is something that we can call a global host name. This I referred to before as our virtual destination.

A

The virtual destination is an object that lets you aggregate a bunch of different services with different host names, or they may even be existing in different clusters. In fact, that's the primary use case. um You have some set of external services as well as some service, that's replicated across your clusters, and you want to aggregate them together under a single host name, so that client requests can be routed to the appropriate back end without having to think about which host name to use or how to set up individual routes between each of your clusters.

A

So if you recall our previous example, our route table had two host names in it. It had a hostname that was used on the gateway to route traffic into the cluster for external clients, and it also had an internal kubernetes, dns name. Now. What you can see is: we've removed the internal kubernetes dns name, and we've created a virtual destination which is going to expose the external dust. The external dns name, that's being resolved by an external dns provider.

A

It's going to resolve within our cluster, and actually this virtual destination is going to expose this service to multi-cluster traffic. So if you had other clusters in your workspace, they would now be able to reach this recommendation service using this virtual destination, but even if we're only on one cluster, this still allows us to have all of our traffic, whether it's coming from outside the cluster or inside the cluster use the same hostname to reach our recommendation service.

B

Let's talk about the operator workflow when rj and ali's application team has finished testing their services, oliver is going to take their services to deploy, increase, resilience and a bunch of other things.

B

So, what's on top of oliver's mind is how do I remove all the implicitly trust for all the services I operate, so I don't need to rely on simple password-based authentication.

B

I I want to have zero trust for all my services. How do I keep the green lights for all of my services? How do I know exactly what's going on within all the services I operate? How do I troubleshoot when things go wrong and how do I upgrade to the new version of the sidecar or the new version of the application?

B

What, if my services running in a mixed environment where some are in kubernetes and some vms.

B

What we're really looking at in the new api is having simple policy reuse through labels. We understand learning a policy takes a lot of time and it's great to have oliver to be able to provide all these policies, so the application team can pick which ones are relevant for them.

B

In a nutshell, our policies, which we have five categories extension observability resilience, security and traffic control based on the functionality of the policy. You can choose whether how you want to apply your policy, whether it's on a destination or on a workload or on a particular route.

B

Some of the policies you can actually apply to both, but most of our policy would just apply to either destination or route or workload under the cover.

B

As your policy is applied to destination workload or routes, we will translate to issue resources and envoy filters for you, so you don't have to worry about building onboard filters, for example, for renaming yourself.

B

This is a simple access policy, as you can see, it has applied to destination. So it's a access policy with strict mutual ts applied to any destination that has strict enabler labeled.

B

So you don't actually have to do anything in your destination as long as you have that label, uh that's how this policy would apply to your destination.

B

um The other example on the right side is apply, a destiny apply a access policy to destination uh which is app equals recommendation, and this in this we are applying not only the strict mutual trs. We're also saying it's only allow if you are a web api or if you are it's your ingress gateway as your service account.

B

And here is a ray limiting policy that you can see. It's applied to a route any of the routes that has configuration of railing meeting equals 100 requests per minute, a simple rainy meeting, server configuration and here's our route table. So the question is how, as a developer, pick up this particular rainy meeting for this particular route, it's actually really simple. Only thing needs to do is adding the labels to the route table to matches the applied to the routes label in the railing meeting policy.

B

As you can see, this is really really powerful to allow developers to pick up rayleigh meeting or any other policy just through simple labels, because everybody understands labels in kubernetes.

B

In terms of multi-cluster as your service grow into multiple clusters, then your workspace may grow because we talk about dynamic teams early. As you take your web or your recommendation services to multiple clusters. Your service is going to grow automatically as they are deployed to multiple clusters through concepts such as virtual destination. We talk about and the policy enforcement will also continue to apply as well as you grow your services to multi-clusters.

B

So for my application owner and developer perspective, we're really allowing them to focus on their application. They can use labels to pick out what policies they want to use and the operator can delegate routes to them. So they can just config route table, that's pretty much the primary resource they need to learn and everything else. It's just built into their existing workflow and kubernetes resources. Today,.

B

I hope you are excited as us about the new api we're going to have a lab tomorrow, so check it out for the lab. If you want some hands-on experience about the new api and we really appreciate any feedback you may have. Thank you.

B