Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2017 (Austin), 15 Dec 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: IoK: Istio-on-Kubernetes Deep Dive [I] - Daneyon Hansen, Cisco

Description

IoK: Istio-on-Kubernetes Deep Dive [I] - Daneyon Hansen, Cisco

Running microservices at scale is not easy. Istio is an open platform to connect, manage, and secure microservices. Did I mention that Istio runs on Kubernetes? During the talk I will cover the following content:
- Istio Introduction
- Istio Key Concepts- Traffic Management, Auth, Policy, etc.
- Istio Demonstration
- Istio-on-Kubernetes Roadmap
- Q&A

About Daneyon Hansen
Daneyon is a software engineer at Cisco responsible for developing distributed applications. As part of the Cloud CTO Office, Daneyon focuses on contributing to emerging cloud computing technologies such as Kubernetes, Istio and others.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Welcome thanks for for joining me this afternoon. My name is Danny and Hanson I'm, a software engineer with Cisco contributor to COO kubernetes and recently I've dove into sto, have learned a few things and thought it would be a good idea to share what I've learned with everyone. um So, let's, uh let's get this party started and just a quick level set for everyone. This is a deep dive, so I'm not gonna, be covering the high-level components and what is tio is all about.

A

So if this is your first is tio session, which I would be surprised. This is probably not. The best introduction is tio for you, but I want to make a quick reference to the sample application that I have running in my Sto mesh if you've messed around Wistia or looked at the docs. You've probably seen this application called book info so quickly. It is made up of several micro services, a product page front-end, a detailed service rating service and a review service that consists of multiple versions of that service.

A

So each versions running a separate pod b1 shows no stars. V2 shows black stars. V3 shows red stars. It's the architecture right SEO is made up of a control plane and a data flight right, and so the control plane consists consists of components called a mixer, a pilot and the CA also references sto security or ISTE OAuth, and then the data plane consists of an enhanced version of the Envoy proxy.

A

So if, if you've been any video sessions, you know that this Envoy proxy gets deployed with every pod within within your cluster right. So to do that, you can either do a manual injection of that pod. This is an example that actually creates the necessary resources within your manifest on-the-fly right. So you're another option is to use an initializer, so starting with kubernetes 1.7, there's this concept of an initializer.

A

If you're unfamiliar with what an initializer is, it really consists of two components: one is pre initially initialization tasks that need to be met before actually running, containers that make up your pod as well as initially and then as initialization controller, that's responsible for actually implementing those tasks, and here are a couple flags that you would need to enable on your kubernetes api server to to run the initialization controller. All right and the initialization controllers is a type of dynamic admission controller.

A

If you think of an admission controller, it really sits between when, when an API request comes into Kuby, API server gets authenticated this piece of code. This admission controller gets called before actually persisting the object. So if I go ahead and say create deployment after that create deployment is authenticated, the initialization controller gets called.

A

So, let's take a look at an initializer, pod and config. You could see in my cluster. It's a simple development. Cluster got a master in a node. You can see the pods that are running in the ISTE au system. Namespace I have the control, plane, components: the CA, the mixer, the pilot, a few additional optional control, plane components of the initializer which we've been talking about as well as an ingress and we'll talk more about this.

A

But let's take a look at the initializer, really quick, okay and when you do a coop huddle, get Pio the initializer, the sto namespace and then output it to llamo.

A

And so one of the things to to go ahead and point out is that this initializer is essentially a proxy or is it here.

A

I'm sorry, the initializer is simply a web server. That's listening on port 80, 83 and there's a couple things that I want to point out as well. So there's a configuration that gets mounted at Etsy, sto, config and.

A

So that configuration actually comes from a config map. You see config map here alright. So let's look at the config map.

A

So this config map tells the initializer a few things: what image to use for the Annette and the proxy containers?

A

Another important piece I want to point out is that it actually daisy chains additional configuration components for the mesh wide configuration using a config map. Alright. So let's look at this config map really quickly as well.

A

And so again, this is the the mesh wide configuration. Do we want to enable mutual TLS? This is for the control plane, there's also a mutual TLS option for the data plane, so that the so that the proxies use TLS as they communicate with one another, do wanna enable tracing again, you could use the links that I provided to go through this in more detail, provides things like how do I reach a discovery address? What's the address for Zipkin if I'm using distributed tracing so on and so forth,.

A

Let's look at the initializer logs right, so that's basically an HTTP server. Listening on port 80 83 and here's a configuration. You see some of those configuration components that I pointed out right, telling the initializer, when it initializes these, these proxies here's the init and the proxy image that I want you to use, and one thing that I neglected to point out when we looked at the pod. Let me let me just jump back there really, quick and.

A

So here is one of the application containers running or the application pods. It's made up of multiple containers. Okay, we've got our app container, this one's the details.

A

You see that there's a proxy container, that's running in the details pod, but there's also this an it container, that's running in the pod right, and so when you look at the qu cuddle get pods, you only see two pods running right. If we weren't running an is Co mesh, we would only see one pod one container within the pod running, but we actually see two containers within each of these pods running.

A

That's because not only do we have that app container running in this example, it's the details container, but we also have the proxy there's one other container that actually gets started. It's just an it container, but if you notice it actually gets terminated, this init container is part of that initialization right, there's some things that we need to do before actually running the proxy and the app container, and what this init container does is.

A

It goes into IP tables all right within the pot, not the actual node or a host goes in there and says, redirect all inbound and outbound traffic to the proxy container and that's exactly how the the Envoy proxy is able to intercept all traffic between your app container and the rest of your service mesh and vice versa.

A

And so I made mention that you know we can use this initializer to automatically inject these sidecar pods, but you can be granular on how you want to do that and there's actually an initialization config that you go ahead and you say: hey here's, the API groups that I want to use here's the type of objects that I want to use for initializing for doing this initialization and if you see here I point out deployment right.

A

If you look at any of the book info examples, deployments are used for for deploying the applications, and so we see the the supported kinds. We actually use an initializer config to say here's what I want the type of resources that I want to use for this pre initialization and take a look here right. Well, here's a kind deployment, but it says false right.

A

So what we do is we actually say within the deployment manifest we can use annotations to say: hey, I, don't want this deployment to be to be initialized with a sidecar proxy, but in this example with my book info services, you see that the side cars are actually getting initialized.

A

Let's see so pod details for the Envoy sidecar, let's go back here again and look at this and we'll use the details. Pod, for example, and I showed you that there's a three containers that are running the pod, the the proxy as well as the app container, and we actually pass in arguments to the proxy and we tell you- and we basically are saying- hey proxy. What mode do I want you to act in I?

A

Want you to be in the sidecar mode and we'll talk a little bit about another mode, but we're passing in basically the bootstrap configuration of the proxy. So the proxy knows: hey where's the path for my configuration file. Where can I find the binary? You know where do I go for discovery, the pilot address so on and so forth.

A

And then, let's look at the runtime configuration of the proxy, so I can exec here, I'm gonna exit into the details pod and get into the ischial proxy container and I showed you the path to the configuration file. You see here this revision number, and so you know after that, bootstrap occurs.

A

The Envoy proxy can now speak to pilot and get its configuration and maintain that configuration so that, as the mesh configuration changes, we don't have to go out and reprogram these proxies. Alright, we push it to the pilot, then pilot pushes it out any changes to to the proxy and and the Envoy proxy supports this hot restart. So it's able to change its configuration without having to to be fully reloaded, and so you're gonna see a lot of those configuration parameters here that were passed in via the the bootstrap configuration.

A

And so uh you know, the envoy proxy has one thing that I want to point out here. Is you see this local admin port and address right so now that I'm, actually in the proxy I, can go ahead and curl?

A

This endpoint and I could see that you know it has several different end points such as okay listeners, I want to.

A

Here's all the listeners that this proxy is is listening on and we're gonna pay close attention to this. Ninety. Eighty, all right, because that's support that this sample book application uses for all those different services that I showed in the diagram, and so since it's since it has a listener for ninety eighty and all these other listeners, the proxy also has routes right.

A

So these are our routes basically saying for this particular listener, like ninety eighty I've got a route table and what you're gonna see here is that there's tons of routes and there's tons of routes on ninety eighty again because the sample book application, all those different components, are listening on ninety eighty right, so details is listening on ninety eighty, the product, page reviews ratings so on and so forth. Right.

A

So what we can do now is we can actually look at one of these routes right, here's a route, we're saying, match everything from the route path and send it out this particular cluster.

A

This is the out details, default service, cluster and so now what I can do I'm going to copy that and I can talk to the clusters endpoint on this admin port and what I'm going to do is just output it to a file, so we can grep alright and then we can grep against clusters all right, so it if we put ourselves in the mind of the proxy we were listening for for those ports, we're here in 90, 80 requests come in.

A

We look up our route and we say oh you're supposed to go to this cluster. Well, here's the cluster that were that we're seeing for for for details right, so it gives us maximum connections.

A

You know all sorts of settings is this canary, no and you're, seeing an end point here too. So this is how the proxy knows where to Ford the traffic. So this IP address is the IP address of the details. Pod.

A

A

Well, this proxy can do load balancing it's not going to do load balancing here, because we only have a single end point that the proxy can also do health checking all right. So it's doing this active health checking if you've got multiple. If you've got multiple endpoints within a cluster right, it can you we can. It can incorporate any policies we create on how we want to load balance across those endpoints, but again in this example, we only have a single endpoint.

A

So why don't we look at a different cluster and let's look at we set it to the reviews service. It has multiple pods in it. So we now see that we've got multiple endpoints dot, 71 72 and dot 73 right, and if we have any type of policy again that setup saying I want to load balance based on certain HTTP headers or based on certain labels from kubernetes, so on and so forth.

A

Let's see yeah so I think we covered that as well. Okay, so that was the data plane right and this enhanced version of the Envoy proxy is the key piece to the SEO data plane. But now, let's talk about the control plane and the first component is pilot.

A

All right, so pilot is responsible for maintaining the canonical representation of the service model and it's the responsibility of the adapters, such as the kubernetes adapter, to populate that model.

A

Accordingly, right so with the kubernetes adapter, for example, the adapter implements a controller that sits there and watches the kubernetes api server for certain resource registrations, like deployments like pods, and it will take those resources and then populate this model that that pilot maintains accordingly right and what that does is is it allows Pilate to provide kind of an interface, that's neutral from any of these adaptors and not really locking you know the mesh down to a specific implementation right and so Annette.

A

When that model is populated based on the adapter, then it allows us to take that information and push it out to the Envoy proxies. You know one of the key features of Pilate is service discovery right, so Pilate expects that there's a service registry that exists right like within kubernetes, using Kubb dns right, and that expectation is that when services are created, the registry is updated when services are removed.

A

Those sir those services are removed from the registry as well right, and this allows envoy to dynamically again to dynamically find out which services exist within the mesh and what's neat about envoy is that it doesn't just rely on what it's learning from this service discovery right. So it can populate all these different endpoints that we just saw right back here right. It can populate all these different endpoints based on that service discovery.

A

But again it's also doing this active health checking to each one of those endpoints to ensure that they're healthy right and if they're, not it's not going to load balanced traffic to that endpoint.

A

So we can take a quick look at the pilot configuration.

A

And so the first thing that jumped out at me is well: we've got a pilot container, but we also have a proxy container right. So not only does the Envoy proxy get deployed with our application, pods like reviews or ratings, but it actually gets deployed with the pilot control plane component as well.

A

You'll see here that pilot uses the secret mounts the secret so that it can communicate with kubernetes api server. So it's in cluster authentication that occurs since pilot does need to speak to the kubernetes api server.

A

You'll see that the cert set c search gets mounted so that the the proxy that gets deployed with pilot is able to do that. Mutual TLS for the control plane, traffic between the Envoy proxies and and pilot, or between pilot and mixer.

A

You see here the logs of Pilate when it starts up I mentioned that mesh wide configuration that the initializer uses. Well, it's that that config map is not only used for the initializer.

A

It's also used for each of the control plane components, but since the control plane components they don't get initialized by the initializer, the only the only components that get initialized by the initializer is those sidecar proxies right, and so we still tell each of the control plane components use this config map for your mesh config right, so that that mesh config for those proxies that sit with the control plane, components can communicate with with the sidecar proxies as well right.

A

So since we are using the kubernetes adapter, you see the deck adapter gets registered, let's jump over to the mixer. So you know the mixer is essentially an attribute processing engine. So each of the Envoy sidecar proxies they go ahead and produce attributes what attributes those proxies produce is dependent on the the user or the operator right.

A

So you create these things called attribute, manifests that say: hey Envoy, here's the important attributes that I care about that for the traffic that comes through you and send this information to mixer well, mixer, just like what we showed in Pilot. Has this pluggable back-end of infrastructure components, so pluggable backends could be for logging for telemetry authentication, so on and so forth.

A

Right and mixer is responsible for taking those attributes from your attribute producers, again those producers being your Envoy proxies and then routing them accordingly to the appropriate, a back-end component, making that component making its decision. If it's a policy decision providing it back to mixer, which then funnels it back down to your envoy.

A

Let's look at the mixer config.

A

And the mixer configuration is gonna look very similar to the pilot configuration. The mixer also has the proxy deployed with it.

A

And, as with the pilot, we mount the service account secrets so that the mixer can communicate with Kuby API server securely. We also mount Etsy certs, and these are the certs that are used so that the proxy for the mixer can communicate with the sidecar Envoy proxies, and you see here this custom config file I want to take a look at it. Really, quick and I didn't show you for the pilot, but the pilot also has a custom configuration file, write a few minutes or earlier in the presentation.

A

I showed you the configuration file all the sidecar proxy and how that dynamically gets managed. It gets bootstrap but then gets managed by by the pilot. Well, these control plane components. They have static configuration files. So if we exact into.

B

B

A

B

B

A

And let's get into a.

B

B

She is tio proxy.

B

A

B

A

What did we say? This config file was again here.

B

B

Envoy or mixer.

A

Here it is so this is a static configuration file that that we pass into mixer and you know, looks a little bit different than the sidecar proxies. We don't have tons of different clusters tons of different services running so each of the Envoy proxies run cluster discovery, service service, discovery service. All these different services to help construct that chain between the listeners, the routes, the clusters and having the the Envoy essentially do the proxying based on that chain of configuration.

A

That was the mixer off that we just showed you look at the mixture logs, we'll go I'm gonna speed things up a little bit just because we're down to our final 10 minutes to point out a few things from the logs. Now that the mixer is listening on multiple ports, something here, if you look at it and we're saying the control plane is doing this mutual TLS. Why is you know? Why is the mixer container have no certificates or keys or anything like that in its configuration?

A

Well, that's because again that sidecar, the proxy gets deployed with the mixer and it's responsible for doing the mutual TLS. So you know between mixer and the rest of the service. Meshmixer is not you know it's not actually doing any type of TLS termination, that's happening at the proxy that sits right next to it. How do you see that this is an empty config store?

A

And this basically says to use the in cluster configuration I showed you how that that secret gets mounted into into the pod, and this tells mixture to use that in cluster configuration to authenticate to the kubernetes api server and here's just a capture that I did of what we're seeing here is actually a non-void proxy sent in some traffic to mixer and said: hey.

A

You know, take a look at this and we can see that here's all these attributes that we configured for for mixer that got pushed down to the proxies and just a few things right. So when we create a policy within SCO, we can say I want to create some kind of policy based on source service and destination service, or any of these other attributes that I don't have highlighted right and because my environment has you know no route rules. Nothing configured.

A

You see that there's zero actions here that are taking place because there are no rules.

A

So sto security there's a one of the pods as the ISTE OCA. It's probably one area that I'm not going to do a deep dive into, but at a high level the the ISTE of security is responsible for delivering that those TLS assets that we saw right and the Etsy certs right. So if we go and exact.

B

A

Curl, instead of looking at a cluster, let's look at the certs right. This information here was actually delivered from the sto CA right, so the sto security, the key components, is first, the authentication right so or you know the identity. So without this T of security, we don't know any type of identity information. We see that here's a source service here is a destination service, but actually attaching an x.509 certificate to each of these services gives a strong identity right.

A

We're able to authenticate based on that identity, right we're able to do things like mutual TLS, so that all the communication between the different services, those services do not have to be configured for TLS. If we configure the mesh for T less than the Envoy proxies as they intercept that traffic will then go ahead and create a mutual TLS connection with any of the other proxies that it needs afford traffic to there's also the ingress component right.

A

So how do I have services outside of my kubernetes cluster communicate to these services running in my mesh right? So SEO leverages the ingress resource within kubernetes and, if you're familiar with ingress, it needs an ingress controller right. So when you do a coop cuddle, get pods from the Nishio namespace you're gonna see is steel ingress, that's the ingress controller, that's responsible for watching the ingress resource right and then based on the rules that you identify in the ingress resource, permitting that traffic into the mesh and it's the the ingress controller.

A

That also runs as a proxy as an envoy proxy, but instead of running it as a sidecar proxy right, the sidecar proxies.

A

They need IP tables by that init container to intercept all traffic that comes that comes through the the ingress runs in this ingress mode and it's only gonna capture traffic based on the rules that you put into that ingress resource the ingress resource that I set up basically says you know all all sub paths from route so for the product page example: if it's a login logout the slash product page any of those endpoints it's going to allow into the cluster o egress by default, your services cannot communicate outside of the mesh. Why is that?

A

It's because those sidecar proxies again they're configured based on IP tables, to intercept all the traffic and those sidecar proxies know nothing about any type of networking outside of the mesh right. So we can create egress rules. These egress rules support HTTP HTTPS that allow you to select selectively, allow outbound traffic. Ok, if you don't want to use an egress rule, let's say it's: a non HTTP traffic you can set up include IP ranges, okay, that include IP ranges.

A

This would be a configuration parameter that you add to that config map for the initializer, all right, so the initializer, when it's when, when it initially zhe's the sidecar proxies it's going to say, hey sidecar proxies only only intercept this traffic and the way that it does. That again, is it's basically saying going into. The iptables tells that a knit container go in iptables instead of sending all traffic only sent traffic. In this example from the you know, for the 1000 / 16 network.

B