Cloud Native Computing Foundation ServiceMeshCon North America 2021, 30 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Get Started With Istio Service Mesh Workshop - Lin Sun, Solo.io (Part 2)

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Get Started With Istio Service Mesh Workshop (Part 2)

You will be given a quick fly-over of what challenges service mesh solves, service mesh architecture, and various service mesh projects in the ecosystem. Then we will dive into the Istio service mesh project including how it works and the best practice to adopt the Istio service mesh through hands-on labs.

We will cover the following topics in this workshop:
Install Istio
Secure services with Istio Ingress Gateway
Add services to the mesh
Secure inter-service communication with Istio
Control traffic

A

As you add in services to the mesh, one thing important is to make sure your service continue function right, so we want to make sure we can still hit the issue, ingress gateway on port 443 and the traffic continue to flow. Let's spend a minute to understand what actually happens.

A

So if you took take a look at the pod information for the web api part uh in the installing action namespace, you can see uh it's a little bit long, but you can see it has one innit container, which uh let me see if I can find it, uh so it has one in it container which card which cause um it's using this proxy v2 image and it runs hdliptables command to config the ip table for inbound and outbound capture.

A

So that's the purpose of the unique container it sets up the ip table, configuration and it finishes, and then the istio proxy cycle proxy and the web api container would run in parallel.

A

So this is the isro proxy container. It's using the same proxy image, as is your image. It has a bunch of environment variables, a little bit more internal install configuration, but you can see it's configured as the sidecar mode with like the cluster name and the domain, and you can configure tracing concurrency all that information is configurable, and this is the web api container that spinned off the web api deployment demo. We just looked at earlier.

A

So we talked about istio innate container. We talked about the arguments for the unique container um and the one thing I want to show you is if you're interested this is getting a little bit advanced if you're interested in the istio iptables command from the pilot agent, you can actually understand like some of the configuration I showed you earlier right, like dash d, for example, right dash b dash. I dash you what this means, so they actually have really good explanation here.

A

um For example, let's figure out what dash d means are from the tooling here, so it means exclude ports right. So why are we excluding these three ports because um some of these ports are we don't want our way to capture, for example, health, checkpoints right? We don't want to go through mutual kis to do health check. um So these are the ports we purposefully config the cycle proxy to exclude from the capture so feel free to poke around.

A

If you're really curious about how this command works and what these commands means they are available for you. um The the next thing we want to talk about is uh the istio proxy container right. So we talked about the proxy container, is using the proxy v2 image same as the init container, and it can fix these reports for the exclude list for from the from the inbound capture.

A

So apparently uh yeah fifteen zero. Ninety is the one that emits uh traffic to permissions and fifteen zero. Twenty is the permissive uh merged permissions metric, so these metrics we're excluding from capture, I believe r21, is the health check port?

A

So if you go back to the istio proxy configuration notice that we also mounted a couple of volumes, um so, for instance, israel ca search root, cert, uh one of the thirds that's mounted. um Also there is a istio token.

A

That's also mounted uh to the cycle proxy. So why do we do that? um Because we need these for to bootstrap the cycle proxy, so the cycle proxy can establish a certificate signing request with issue control plan so that it can get the necessary key answers assigned for the cycle proxy to communicate with the rest of the services in the mesh.

A

With that we're going to add more services to the mesh, so we're going to add, recommendation, purchase history and sleep to the mesh by rolling restart the deployment.

A

Now, if you go get the parts for issue in action, you can see you know, things are being picked up with the cycle proxy and the the old pods are being terminated.

A

So we're going to continue to curl the web api through issue ingest gateway, just making sure, as we add services to the mesh. U we're not breaking any of our existing traffic. So what have you gained right? You added sidecar right. It's going to come a cost to you right! Those cycle proxy does take a little bit of memory a little bit of cpu right. It requires you extra effort. So what have you gained um to do that? We're going to notice?

A

We have two terminals here now, also we're going to work with both of the terminals, um so we're going to first put a loop right for that command to access web api, we're going to put a little bit loop to generate some traffic and then on the second terminal, too. We're going to open up the kyali dashboard.

A

So once you open up uh click on the kayali ui tab here, you're going to see uh the kayali graph, so click on the graph here and select seo in action and you can choose to display. You know different data. I like traffic animation, I also like security, and so it shows you like the live traffic comes through, and it also shows you. You know whether the traffic is mutual trs.

A

So that's one benefit the site cup brings to you right. You immediately gain access to all these data from the kayali dashboard. The other thing we want to talk about is distributed tracing, so we're going to go back to this terminal, 2 and control c to get out of here.

A

If I can control c correctly, let's see- and you know we're going to copy this command over to uh start the jaga dashboard, so click on the jguy ui here and um and on the service drop down click on istio ingress gateway, so that would capture all the traffic or you can click on web api so either way it's going to capture all the traffic coming through the web api service and if you click on, find trace, it's going to show you all the traces in in all in for the last 20 traces.

A

So, for instance, you can click on one of these and then you can drill into a little bit more detail about this particular trace information. Well, the time are spent the duration. You know, and you can click on the text to find a little bit more information about, uh like the request size response. uh You know upstream cluster, so pretty much everything you want to see is uh it's here.

A

You know when things works, uh this dashboard could be a little bit boring, but with this break, when you have like a arrow different than 200, it could be really really helpful to help your troubleshooting issues.

A

The x request id is how you know we know these traces are belonging to the single request, so it's very important to help propagate these headers throughout your service, which may require a change to your service if your service are not automatically propagate, these b3 trace headers.

A

The next dashboard we're going to look at is uh the grid founder dashboard.

A

So click on terminal 2 here and we're going to do ctrl c get out of here and using dashboard grifana command to view the grid from the dashboard.

A

Now we're going to select, let's see, go back to the dashboard folder I'll click on home.

A

Let me refresh here seems a little bit slow to me too.

A

Okay, so if you click on istio folder here you can see, is your service, my service dashboard? So it should show you all the services that we have. So all these data that's generated for web api purchase history recommendation it's all available for you that you can see. You know all the client data server requests, client requests a successful rate. You know whether the traffic is mutual tls.

A

Every single data is available for you uh and it's a it's an application level data. So a lot of interesting data. So congratulations. uh You have added the sample application to seo and observe the metrics tracing and dashboard.

A

So these are the benefits by injecting that site card to your services and you are getting all these benefits.

A

So if you click on the check, you should be able to get a good result and progress to the next lab. I would say: let's take a few minutes break here before we continue uh rahm. How is everybody doing it.

B

Yeah, I think that was that was much better. People are able to follow along. So okay.

A

Let's do that same.

B

Pace for the next lab, but, like you said uh we'll, do a few minute break. uh Maybe you could pull up a like a blank screen or something with the message. That's saying that we'll be back in in five minutes so that people that are streaming in can can stay up to date.

A

Yeah sounds good uh yeah. Let me create a new slide.

A

All right, congratulations! You guys got in this far so now we're going to talk about how to secure the services in your mesh. So let me ah on my screen, so this is lab, see lab four. We did two labs together, um we're going to talk about. How do you incrementally adding services to the mesh which actually you already added that in the previous lab? But you might remember, you actually see metro, trs security badge in your kayali dashboard right. That is true because it's your automatically configured mutual tls.

A

For you, however, that's the best effort that we don't guarantee its mutual ti. So if mutual tis works, the traffic goes through. If mutual trs doesn't work, we will send plain text.

A

So in order for you to enforce the traffic to be strict mutual tiers, you would have to apply a strict mutual tls policy, either for your target service or for your namespace or for the entire mesh. So that's what we're going to do uh in this lab, so we're going to enable mutual tls and we're going to talk about how the workload key and certificates are distributed in istio and we're going to inspect the key and certificate for each of the service so that you can understand how mutual tears is enforced by istio proxy cycle.

A

With that we're going to jump into a demo on this lab, so you should see this and click on the start button. That would start this particular lab secure communication with istio, so you should be already in the is your basic directory, so you can skip the prerequisite.

A

The first thing we're going to do is check out what are the peer authentication policy in my kubernetes cluster. So pr authentication policy is the one that um specified. uh What are the re? What are the secure, uh strict mutual ties policy in my is your cluster. As you can see, I have nothing specified so by default. The issue allows uh permissive mutual ties to be used, which is what I would mention earlier. We will do a best effort on upgrade your connection to mutual ties from that cycle proxy.

A

But if the traffic didn't go through, we will downgrade to your plain text and still allow the communication to go through. So that's what permissive is for, so we're going to apply strict mutual tls for the entire mesh. The reason I said it's entire mesh in this example is because I'm applying to the istio system namespace, which is the root configuration namespace for my cluster, and I'm naming this pure authentication policy with default and I'm applying mutual gis mode strict. So I'm deploying this default policy in my route, namespace to the entire issue mesh.

A

As you can see it's applied, you can also run the cube, get pure authentication policy. You should be able to see the policy we just applied.

A

Now, let's see mutual tls in action to do that, we're going to deploy a sleep service in the default namespace. So the reason why we do that is um in the default namespace. The sleep pod is one slash one, so it's just one sleep container without the side car. Now, if you look at the pods in the installing action name specific, you recall, we also have a sleep container in there, but it's two slash too. So this means this has the istio cycle proxy injected in the israeli action namespace.

A

So we're going to um run a curve command to access web api from the sleeve in the default namespace. What do you think it's going to happen?

A

It's actually not allowed. Why? The reason is we apply the strict mutual tls policy for entire mesh, which includes every single services in the mesh that includes the web api service. So when the sleep service in the default name, space is trying to reach out to the web api service which are in the mesh. So we're talking about a client outside of the mesh trying to reach a client in the mesh. It doesn't have the key and certificate to do mutual ts connection, which are why the connection is rejected.

A

Now what if you run the same command except that you're using a different name space the sleep in israel in action namespace this time that it did succeed right because that sleep container has the israel proxy. It knows what you talk to the keys and certificates to do, establish the mutual tls connection and to do to do the mutual ts communication with the web api service.

A

If you go back to the kayali dashboard, so let's generate some traffic, you should be able to see a little bit different. I want to point out that difference quickly for you, because you're probably wondering I'm already seeing my security badge right. So what's the difference so now, if you go to navigate the kylie in the graph that we mentioned, so you can still display the security badge and traffic animation right.

A

So this is same as before, but one key difference here is, if you click on, if you click on the the secure button here, it does show mesh when mutual ts is enabled. So kayali knows that we have mutual tears, enabled each of the service within the mesh. By the way, this is the enablement on the target service side.

A

So, let's dive a little bit deeper to understand how this works with that I'm going to exit out of the kai ali dashboard command. So I have my terminal back.

A

Now, if I do a istio proxy configman by the way, it's a super helpful uh debugging command allows you to look at the onway configuration many of the cycle proxy configuration for each of your services.

A

So if you check out the web api secret, you can see we have the default and the root ca secret for the web api.

A

The default secret contains the the public certificated information for the web api service and you can analyze uh the uh the the secret and the default secret a little more by analyzing the issue right. So it's you should buy cluster.local, which is your local kubernetes cluster, and you can also check if the public certificate is valid for for the web api container. You can see since this was just generated, so it's actually only valid for a day by the way. Don't you find that interesting? What if you know after the day, you know?

A

How is this going to stay valid right, so it still actually automatically handles certificate rotation for you before it's expiring.

A

So we know we constantly check when it's expiring and we stage the certification rotation for you and make sure it automatically renew the certificate in the same way that you are getting the initial key and search signed and we're using the same way to renew the search for you now, let's validate the identity of the client search and check it's valid, so you can grab the subject alternative name, which you will see the specific id for this particular service.

A

So you should uh let's try and spend a little bit of time, understand the spf id right. So where does the cluster.local come from and this is namespace? This is the issue in action, which is your namespace and service. Account is web api right. So so, if you go to the istio config map during installation of istio, by default, we config a trust domain for you, which is part of your spf id, so that trust domain by default is cluster.local.

A

Come with the default profile. You can choose to overwrite that if you want through it's your operator.yaml, so that's how your spf id is built, and the other thing I want you to check is the service account we talked about this earlier, but in case you don't remember, the web api deployment was used in this particular service account we created, which is also used in the spf id for this particular service.

A

Now the next question I want you to think sue is: how does the web api service obtain the necessary key and certificates?

A

So remember earlier we reviewed the injected. Is your proxy container for the web api pod, which you can do it by?

A

You know, find out the pod and you can always do a get pod on this web api, and you know you can do it um yamo right. So we did that together when we examinate the init container, the psychoproxy container, there was a bunch of volume that are mounted to the isro proxy container.

A

um So what are these volumes right and there's a it's your token that we review together too right it's on the right side here too. So what are these things um like the ca, certs and the istio token? What are the purpose of these? So these are mounted to the istio cycad container, so that when the sitecar container starts the istio agent, which also called pilot agent, remember you run the command of pilot agent to check out how it can fix the ip tables.

A

So the pilot agent would create the private key for the web api service and then send the certificate signing request to istio certificate authority which, by the way, is the sdod control plane which are installed in our cluster on the istio system, namespace called sdod, so istio daemon, that's what it stands for, so it sends the certificate signing request and using the it's your token and the web api service service account token asked the credential to talk to sdod and the institute issue agent sends the certificate back.

A

So the student gets the certificates back from the issue d and then sends the sends the private key to the envoy proxy through the envoy sds api, so that all this magic happens in the cycle proxy without you actually needing to do anything.

A

The only thing you need is just get the cycle injected right and we talked about the expiration of 24 hours. um The next question you may be wondering is: how is mutual ts strictly enforced right?

A

So this is something actually you know I find really interesting.

A

So when you do a proxy config on your web api um deployment, you you get to see all the configuration, but one thing interesting is uh before you have strict mutual ties enabled you are actually seeing a lot of the times that we allow raw buffer to your services in the mesh, but once we have mutual tiers enabled the raw buffer we are allowing actually got reduced, we're still allowing raw buffer for services not in the mesh, but for services in the mesh.

A

We are no longer allow lower buffers, so we no longer allow plain packs in that sense, that's how we make sure the traffic are mutual tris um and only mutual terrors are allowed.

A

The other question I want you to start think about, is you know, there's so much data right every time I do a voy proxy config, it's just so much data, it's so hard to read and interpret.

A

How can I configure my istio to only listen to the stuff I care about? How can I you know selectively configure what my osteoporoxy cycle can see, so I, in fact, in 30 minutes also, I think rom is actually going to talk about how what's the best practice to deploy istio in production and he's going to cover export you and psycho resources.

A

We also plan to cover discovery selector in one of our future workshop, so stay tuned. These are available in istio, but we're not going to cover in this lab. So congratulations guys. You have enabled strict mutual trs policy for all the services in your entire mesh and the next we're going to explore controlling traffic and with this services. So click on the check button which will load on the next challenging for you.

A

And you can click on start to get to the next lab how's everybody doing.

B

A

B

Couple of thumbs, ups right yeah: uh I think you can proceed lynn.

A

That's awesome, uh yeah. I think we only have uh 17 20 minutes, so I want to make sure this lab is a little bit long. I have to say the the control traffic lab so essentially we're going to introduce a new version of one of our services and we're going to teach you. How do you dock launch the service? How do you do canary based routing? How do you add resilience to your service?

A

How do you interact with external service, so we may not be able to get everything, but the lab environment will be available for you throughout the day, so feel free to play with them. So in this lab, in addition to gateway and virtual service, we're going to talk about virtual service um to not bind to a gateway. So this is interesting. You can configure router rule uh without a gateway right, because not every single of your services would be connected to the ingress gateway directly. So we're going to talk about that.

A

uh We're going to also configure uh subsets and policies for your destination, so destination rules are essential when you have multiple versions of your services, you're going to need to configure you know, how are you you're going to shift your traffic? What are the subsets of your services?

A

Service entry allows you to access external services by having services entry in our istio system and yinghua? Is your service? Mesh registered istio knows the external service and how to allow you to connect to them.

A

So the first lab we're going to as part of lab five first portion we're going to do a doc launch of purchase history version 2. Then we're going to do a canary test to route 20 of the traffic to version two of purchase history, then we're going to control the outbound service by introducing purchases through version three and we're going to configure seo to only allow register service to go to the external service.

A

So, with that we're going to jump to lab 5., we do have a bonus lab, but unlikely we will have time to cover. You can do them on your own. It's not going to be required for the certification.

A

So make sure you are in the is your basic directory, which I am and let's check out the purchase history version 2.. As you can see, I actually built this service myself called version 2, it's a 4k version based on the nick jackson's fake service. The only reason I built this version 2 is: I want to introduce external service into my web into my purchase history service. So it would cure this external service and get a response that I can't really read, but it essentially, you know, gets a different response. Every time.

A

Yeah, it's just a deployment here um and.

A

So we talk about destination rule. um So one thing we're going to do is: config a dark launch by route of the traffic to version one right through weight. 100 we config all the traffic to version one.

A

We also have a destination rule, because in our virtual service we specify subset v1 so in destination rule we define how the what is we want. What is we choose? Subsets, it's by a label selector to select on the deployments of version v1 and version v2. So if you remember our deployment label here right, we did label on the selector match label.

A

Let's go ahead, apply this virtual service and destination rule to our environment. So now we have config issue to send 100 traffic to version one. Now we can introduce version 2. by the way you really want to apply this first before you introduce version two, because the moment you apply version two kubernetes by default is going to be wrong, robbing the traffic between version one and version two before you actually had a chance to test version two.

A

So now we deploy version 2 into your environment. If you do a gap path on the issuing action namespace, you can see both of one and two are running.

A

Let's take a look at the logs of version 2 of purchase history. It's actually an arrow here, unable to connect to the external service connection refuse. So this is interesting. Why do we get this arrow.

A

Okay, let's generate some load first right, so, uh let's generate some load, just making sure traffic are still good. As you can see, I mean we can fix 100 traffic to version one. So even though version two was not good, you know we're still fine, because you know we're not breaking anybody down so now, let's think through. Why are we having this behavior right? Why is this guy trying to reach out the external service? And, let's get the connection refused?

A

Remember I mentioned earlier right when the init container runs the sitecar proxy and the web api or purchase history runs in parallel as the cycle proxy. So this pose a problem. What, if the cycle proxy needs to reach out to any external service, but it doesn't have network connect?

A

What, if the purchase history, I'm sorry I said it wrong. What if the purchase history container needs to reach out to the json placeholder service, which is external before the envoy cycle proxy, is ready right. It's it can't that's, essentially the problem we run into. Luckily in istio there is a configuration where you can hold the application container until the proxy starts.

A

So, let's make sure we include that configuration to do that. You add an annotation to to your deployment like this and let's go ahead and deploy this and.

B

A

Cross is going to fix it for us, um okay, so, let's check out the logs, uh for this part um depends on where we are okay, it's actually picked up. uh No, it actually takes a little bit time. So sometimes the logs does take a little bit time because it takes time to start the container. um So let's try it again yeah. So now you can see on the logs did pick up the the new one, which was just started 18 seconds ago.

A

You can see we're able to connect so and that's all, because we can fix the whole application until proxy starts, so this is highly required if your container, your service requires external connectivity outside of your kubernetes cluster, make sure you conflict this, because I actually stumped on this and took me a while to solve this problem myself all right. So, let's test version two of purchase history right as you can see, we were able to test it successfully by exactly into the container and do a local hosting.

A

So this is like the most uh simple testing. um Now we want to do you know once we test a little bit, we want to do a header based routing right, because you know we don't want to just test within our container. We want to pass through. You know some other service so to do header-based routing.

A

Remember we talked about raw rule right, so you can fix that in virtual service you do a header-based matching uh when the user is json by the way it's capital j, you route the destination, to version two and for any other traffic. You continue to route to version one.

A

So let's go ahead, apply this.

A

And uh let's do a testing, so what we are doing is the user.json. hmm Why is still going to version one all right? This is because uh in the round rule we had exact adjacent and we were using a lowercase j in the qra. So let's fix that.

A

So if you use uh usages and finger crossed uppercase j, uh it does return purchase. History version two exactly as we told each other right. So this is great. uh We were able to test it. um Let's do canary base to put 20 traffic to version 2., so uh go back to our virtual. His uh virtual service and change version. 1 should be weight, 80 and version 2 to be 820 so that we can achieve 20 traffic.

A

Let's go ahead and apply this virtual service. Now, if we do a loop of accessing our web api service and grab the purchase history, you can see out of 20. We got exactly four, so that's the 20.

A

So, let's continue this trend to shift more stuff to version two we're going to shift 50 this time and apply this configuration and let's do a loop and, as you can see, it's about 50 um not going to count, but uh it's nice that with istio you can config a simple virtual service and without needing to restart anything, you were able to shift traffic exactly what you want it to be. So that's the power of what it still brings to control traffic.

A

The next thing we're going to do is shift all the traffic to version 2.. So let's apply that and run this loop again. As you can see, every single traffic is version 2 now so very nice.

A

Now we're going to teach you how to control inbound track, I'm sorry how to control outbound traffic by default. It still allows all the outbound traffic to go through. um This is by you know, I'm just double. Checking the outbound policy. We don't have any outbound policy so default is allow any to configure istio to only allow registry. Only let's go ahead, install this command by setting outbound traffic to be registry. Only as you can see, our configuration are updated.

A

The way this configuration does take a little bit while for istio to pick up. The main reason is this: configuration is updated in a config map and it does take a while form that config map for istio to recognize and then push down to the onboard sidecar proxy. So now, if we send the traffic to the web api service, let's see how it goes yeah. So it still goes um goes because I said it's going to take a while so um yeah now you can see now it fails now.

A

So the envoy proxy has been refreshed with registry. Only so it's not going to allow purchase history to go through because it reaches out to the external service that it's not allowed to access. It's not registered to access. So if you do a pod, a log command, you can actually see that um we actually retry, for you remember we talked about, but in istio you automatically get retry twice. So this is exactly why you are getting attempt to count three uh because we retry for you. So let's go ahead, fix.

B

It um we have about four minutes left.

A

Okay, I'll run through this, uh let's go ahead. Fix this to fix. This is apply a service entry command resource by importing this external services into the mesh. Now, once we import this, you should finger cross to access yeah that it did work. So that's that's the nice thing right. You have the precise control into you know what external services are allowed.

A

So this is all the control traffic section um I'm going to go through the rest of my slides, because some of you might be interested in the badge program. So we do have this foundation badge I talked to you through, so we really appreciate you guys, provide a feedback to our survey and test.

A

Could one of you run or we'll maybe send this out so by completing the test uh and passing 80, you will be able to get this foundation for istio, batch or certified by solo, and please give us the feedback through the survey, and we also have the rest of us talks from the solo team make sure you attend like rom's workshop.

A

If you like this workshop- and you know, we also have a talk about designing service mesh global scale at sap, that's really interesting and also a multi-cluster workshop and christian is also talking about zero downtime. So don't miss any of our talks and you know visitors said the boost to um we have in-person booths and virtual booths, so please engage with the solo team.

A

That's all I have.

B

All right, thank you. Len.

A

Thank you so much rom and will and eric. I appreciate you guys, support and thanks everyone for attending and joining us for the workshop.

B

A

We do have a lot of future workshops at solo.I solo.io. If you go to our event page, we do run this type of workshop very frequently on a monthly basis, so feel free to join us. There, too,.

B

Yeah, just to kind of elaborate on that I mean there's. We covered a lot of content in this like very short amount of time, so, if you're interested in this, but you want to kind of go, go at it at a much more um slower pace. Then yeah go to solo.io, look for the events section and then you can see where we run this workshop, the workshop that I'm going to run next, which is a slightly more advanced, deploying stereo into production, and we also have you know, workshops around multi-cluster, meshes, api gateways, envoys, etc.

B

Okay, so I think at 11 10 is actually the next workshop, which I will be running. Let me give me a couple of minutes to kind of swap computers and then we'll get started.

A

Okay, great thanks: everyone have fun lately.