Istio IstioCon 2021, 10 Mar 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Getting Started and Beyond Istio Multicluster with GitOps

Description

#IstioCon2021

Presented at IstioCon 2021 by Ryota Sawada.

Istio documentation covers many scenarios how you can start up Istio and get your hands dirty with its offerings. Things become a bit more tricky when it involves multiple clusters, and even more complicated when there are other Open Source projects you need to deal with.

The presentation will be mainly driven by demos. The first demo will be based on simply starting multiple KinD clusters locally, and get Istio offerings in action.

The second demo will be based on GitOps setup with the similar multiple local clusters. This will involve some UI examples to showcase Istio offerings similar to bookinfo example, and backed by GitOps.

Lastly, there will be a short demo with multicloud + multicluster + GitOps setup, which gives the sense of how you can get production ready multicluster setup with Istio.

All the demo steps are documented in GitHub repo, and every step for each demo is curated to get the clusters up and running from scratch.

A

uh This is getting started and beyond istio multi-class led gears, I'm rio tasawada cto at upside up. So let's get right into the agenda. I'll start with a brief introduction of myself on the company upsider and then we'll touch on the target audience what to expect in terms of topics covered and not covered. First then, we'll briefly touch on istio multicluster.

A

After going through a quick brush up on issue resources we'll have the first demo and after that, we'll take it uh further with githubs and go on to the second demo, I'm hoping to briefly touch on what can be done from there if the time allows as well. So who is ryota? Who am I so you can find me at usual places with ryota sawada rytswd, all the vowels taken out from my full name, I'm a cto at upside leading platform team and also responsible for back-end development.

A

I have worked in finance space for about a decade now, I'm based in london and have two cats. As you can see in the picture to the right, uh you can find the pic uh the picture used as my profile picture at most places.

A

I'm rather an istio user than a full-on contributor, so if there is anything uh wrong, any wrong information in the material today, please let me know so that I can correct in the materials- and this is also my first public talk ever so fingers crossed everything goes okay, so briefly about upsider. So I need to really touch on the company I work, for uh it is a series, a slash b, startup, providing b2b payment services.

A

uh We use istio extensively in production and thus wanted to share our experience and learning with the community. Today the headquarter is in tokyo, japan, but the remote team is around the world and I myself I'm based in london. As I said uh before, we are hiring uh but I'll stop here for now about upsider. Please feel free to reach out to me uh to find out more about us so materials today.

A

So before we get started, the link to the slide will be provided later and also you can find most of the materials I'll be using in the demo from the github repository here. So it's a github repository with my user account rit stability and get-istio multi-cluster.

A

Today's target audience: let me start with that, if you're getting started with istio or getting started with istio multi cluster and would like to take it further. Hopefully this is a talk for you, and this is probably not for you if you already have issue deployed in multi-cluster scenario and with that what is getting started and beyond. So this is my personal. Take on phases following getting started.

A

The terms used here are generic, so let me uh put uh the istio spec and, let's see how they fit in so getting started. Phase is about learning what capabilities is your provides and play with it? For the first time, you install still with the most basic configurations, probably using istio ctl, or is your control, how they call it and changing some parameters here and there proof of concept phase is about getting real use of istio.

A

So you would probably set up a cluster with your business applications alongside with ico and see how they work together, and there are some specific environmental or security requirements you would have, and you need to take in consideration at this point and also at this stage, purposely or not you'll face something to break or something just not to work as expected, and you would want to be able to debug pinpoint and fix as necessary.

A

Production. Onboarding phase is about getting security right and tight observability make sure that ensuring observability is in place and being able to support if anything goes wrong goal of today. Is that we'll be focusing on the first two phases only for production boarding?

A

There are great talks in istiocon, so I encourage you to check them out for today's demo, demos I'll be throwing an additional requirement of multi-cluster and github git ops integration so what's covered in the talk I have mentioned in the previous slide, so I'll skip it for now and also what's not covered in talk. If you come back to the slide, you should be able to see, but this is something that we talked about in the previous slide.

A

So when it comes to installation patterns, as of today is your version. 1.9 official documentation provides five guides. Is your ctl, easy operator, helm, multi-cluster and virtual machine will not be using seo ctl for installation today, as that's covered in many places already. It is worth noting that the official multi-cluster setup documentation have had significant updates. Really nice updates with seo version.

A

1.8 and 1.9 I'll be touching on this more closely, but my demos will not be following the latest step latest setup, I'm planning to update the repositories, but for this store please expect the steps to be not fully aligned with the official documentation.

A

So today, I'll be using istio operator and manifest generation approach, which is briefly mentioned within ico ctl documentation and if you're curious about helm-based installation, there seems to be a talk tomorrow by ryan micheller. I'm hoping the name is I'm pronouncing correct. uh Titled taming is your configuration with hell, uh which seems to be touching on uh git ops as well. So please check it out.

A

Understanding is your setup, so it's your tackles to solve complex problems, and for that reason it still may seem complicated at first too, like issue ctl was to make it easier to get started, but it hides away some details which you may want to know and to understand what eco does behind the scenes and that's the reason why we're using istio operator and manifest generation approach today and we'll touch more on this in the demo.

A

So what is multi-cluster so we have been talking about multi-cluster for a while- let's just make it clear and what I'm talking about here when I say multi-cluster, it's simply to get multiple kubernetes clusters to work together as an example at upside up, we have multiple clusters and four of them have istio multi-classes set up. This gives us the control and auditability. We are required to comply with and with multi-class setup. You need to configure each cluster obviously, and it can be surprisingly challenging.

A

The configurations are often similar among clusters, but you would also want to configure specifically one for one cluster for some example, for example, cases that can lead to configuration headache and easily end up in a situation that setups works for one cluster, but doesn't for the other when it comes to multi-cluster.

A

There are four categories in the official documentation, two for the same network, primary primary primary mode and to for different networks, our primary primary primary remote, and we will be focusing on the single network with primary primary today and as mentioned, the setup is not exactly following the official documentation as I'm using the order version of istio so for getting started. Let's quickly touch on easter resources, control, plane, data, plane and ec operator easy operator is both custom resource, as well as a stand-alone service. The service is used for easier installation, but is optional.

A

We will not cover too much about this operator today, but we will see in action. A custom resource is used for seo control as well. uh It's a ctl control as well and we'll have further look at it later. Control plane is essentially seod. You used to have multiple components in the past, but when you see cd, you can think of it as control plane. It monitors custom resources in the cluster which I'll be also briefly touching on and ensure data plane is updated accordingly.

A

Data plane is where the traffic magic happens from eco. So for this I'm focusing on two sections here: one is standalone osteoproxy in the red box at the bottom left corner often referred as is still ingress or egress gateway, and another is your side card proxy in the green box. Both are based on android proxy and thus I'm using the pink hexagon logo of envoy. Here as an example, on the right hand, side you can see a data plane with five ports, each containing istio proxy.

A

This means each proxy under sport knows about the routing details and traffic setup for other ports and also, let's briefly, touch on the main custom resources used for traffic management. Service entry is used for configuring. Some services outside of your cluster, to be part of your mesh gateway, is used to get razors to ensure it's. Your ingress and egress gateways can get traffic routing destination.

A

Rule is used for traffic updates, such as load, balancing connection pool, handling, etc, and virtual service defines the routing so before, diving into demo I'll be using following tools: kind and k9s, I'm using kubernetes version, 1.17 and also istio version 1.7.5.

A

It is not the latest and greatest, as I had the repository set up quite some time ago and I'll be looking to updates uh updating the coming days um goal of the demo today for the first demo, I'm creating two clusters, I'm naming blue on the left as armadillo cluster and the one on the right as bison classes, both have almost identical services.

A

The first demo is basically to get some multi-class communication to work so for that I'm using http bin container and apologies because of the time constraints I have recorded this one so that I could do video magic, video record editing magic to speed up some parts. So I promise the second demo will be a live demo demo time and this one is the demo.

A

So actually let me get back to the slide and before diving into demo demo. Let's just quickly have a grasp at get istio multi-cluster repository that we are talking about here. So, as you saw earlier, get issue, multicluster is a repository I'm based on and down here there is some warning I'll be working on, but this kind based setup is the one that the video is based on. So if you want to follow along, you can have a look to see, what's actually happening behind the scenes.

A

Oops and let me start okay first, I start with no cluster at all, and I get my git repository as it contains all the configuration files and I start creating two clusters and they are given two different names. One is armadillo and one is bison.

A

Both have used almost identical configurations and the only difference is open port for them and after a minute or so you have two classes running, and this is where the video editing magic happened.

A

Then I create the certificates required for multi-class communication, and this is mentioned in the istio of show documentation as well, but failing to do so as early as at this stage would cause a lot of trouble later on. These certificates are registered as kubernetes secret for this demo and you can see that our ca sets are being created under istio system namespace, with a little bit of video editing magic as well.

A

The next step is to install install istio operator for this demo. You would usually use ctl for installing is your operator, but I'm using a generated manifest for operator. uh So simply using keepsake apply here. This actually gave me an error because namespace creation was happening at the same time and caused some later configuration fail to find the name say, simply run it re-running the same comp command with the word this time. So now you can see it's the operator being installed on the istio operator namespace.

A

Next, I'm installing istio control plane either is your control. Plane.Yaml file, which is your operate, is operator, custom resource and I'm applying that to the cluster rather than running steel ctl installed with those files. This results in the installation of sdod on iso code dns, as you can see at the bottom pane after the control plane is installed. It's the time for data plane, I'm creating multiple instill ingress in and egress gateways from multiple issue operator, custom resource files.

A

You will see in total four gateways created external ingress and egress gateways and multi-cluster ingress and egress gateways. They all have name prefix of armadillo and bison respectively.

A

At this point, I'm deploying my application as well, namely http bin and toolkit color service- is something we'll be touching on later. These get deployed to kubernetes and it's your default site card to them. It's a little bit difficult to see, but the bison side, so bottom right corner. You can see that the count of the replica goes up from zero to one to two in a bit, and you can see that two right there so from here there are some imperative configuration updates.

A

First, I need to configure a media cluster so that it can resolve dns name for bison connection. This is done via istio, core dns and I'm using kubernetes coordinates to use easter coordinates for resolution. So this is what I'm doing with the cube ctl apply here.

A

And next step is for local configuration for armadillo cluster. Only this is to ensure virtual service etc are set up for armadillo services, we'll be doing the same for bison later, and also at this point we are making sure multi-cluster related configurations are in place. You can find the relevant files in the repository, but essentially it contains gateway, destination, rule and android filter resources.

A

As a last step for armadillo cluster 7 setup, I think uh taking note of armadillos is your egress gateway, ip and also bison's, istio ingress gateway, ip with kind clusters like this demo, which does not have ingress controller. It falls back to with some docker and, as I get hold those ips, I'm updating service entry resources to hold those ips in place. After all, the updates are complete, I'm applying them with cube ctrl.

A

So now they are applied. Finally, that's armadilloside and finally, I'm making sure bison has its local routing as well as multi-cluster handling setup, just like we did well for a modular cluster.

A

After these lengthy configuration setups. Now I can exec into toolkit container on a modulo cluster and let me pause the video here to curl against http bin service running under bison cluster. So you can see at the top I'm executing into the armadillo side of toolkit and then running curl against bison, offering.

A

And when that's that runs voila, we got the t port from bison http.

A

So let me get back to the slide.

A

So we saw a pod in armadillo cluster was able to talk to port in bison cluster, so I got into toolkit container at the very left in this slide and tried to call at http address. It took quite a few steps. I was essentially getting started with the multi-cluster just a few posts here and there let's check out more complex setup with githubs.

A

So what is githubs? I don't want to spend too much time on explaining git ups, but as you can see on the diagram to the right, it's about making git the single source of truth. It gives us declarative cluster management where all clusters all changes, are driven by g repository.

A

Although there are a few github solutions out there, I'm using argo cd from intuit, which I'm most familiar with myself. There are several benefits of using githubs as listed here, but the single source of truth is the most important one here, as multi-cluster requires configuration fine-tuning in multiple clusters.

A

Having githubs helps ensuring the installation and configurations are repeatable.

A

So, as listed I'm going to be using more tools, this time kind and k9s are the same as before, but this time I'll be using argo cd extensively. Also, I included some observability tooling in the setup as well. Those are not using the istio add-ons or for single getting simple getting started, but I used operator for installation. Those configurations are not trivial but with githubs. I could make those uh also installed, alongside with my applications on seo, so goal of this demo.

A

This time, I'm going to add more easter configurations, namely I have multiple istio, ingress and egress gateway components I'll be using three for each each cluster, one for localhost access, so that my demo ui can access each cluster and one for multi-cluster communication, which we saw just earlier on from the video and one more for observability tooling, which is a mock setup where observability tools are maybe hosted under different domain or some other control also we'll be using color service. This time, it's an extremely simple service I created, which returns some random color.

A

I have configured so that the service on armadillo cluster, armadillo, blue armadillo cluster generates only bluish colors and the bias on the side will only create reddish colors. So let me get into the demo.

A

First of all, if I get back to the get, is your multi-cluster? So the second demo is based on this argo, cd based uh setup, so I'll be making it a little bit more clear for easter corn demo talk I'll, be making some changes, but for now you can follow along here and for agua city based setup.

A

It starts from cloning, this repository preparing, github token and set up originally and also starting clusters, but I have done up to number four of the steps here, so I have created the vanities cluster, which does take a little bit of time, preparing ca certificate which I have done as well. I have also registered ca certificates uh to the clusters with the secret.

A

I have uh installed argo cd already, so our cd installation has been done for armadillo and also bison, and I have actually installed istio as well. For this time I have used manifest generation approach, so I'm not using istio operator. So this is no operator install, I'm simply using the generated yaml files, and this is done because I just wanted to get the time of uh having istio installed before uh the demo, and this is because part, it's part of the github configuration as soon as I get a github setup in place.

A

You can configure istio from the github's repository git repository and also update accordingly, if you really need to so now. The last step is to go into uh github's setup for each cluster. So before that, let me get into running ago cd right now, so I'm having this localhost uh hosted our cd 1.8.9 r2. Sorry- and I don't have anything running at the moment, so this is armadillo cluster and what I'm going to do is to.

A

Apply the argo cd spec of project and application, so when I start that and those are created and on the back behind the scenes you can see that app orchestration has been created from here. You can see that it's looking at github repository that I mentioned earlier, get istio-multi-cluster repository and also looking at the path of clusters, armadillo augustini.

A

So let me get into that. So. Firstly, it's trying to create the namespaces and also while I'm talking uh it's creating all the applications. So it's installing our cd on its own, so it will be using all the github management so for any of the changes to the cluster, including augustine itself.

A

Grafana is your traffic management and all the applications are installed right now or being installed right now. If I get back to the list of applications, you'll see quite a few things at the moment. So we saw just app orchestration before and now it's synced and it's all healthy, you can see our cd is being synced. Graphana is being installed as well. Grafana dashboards data sources are also configured separately.

A

Is your multi-cluster demo are being created? Is your traffic management and all that when it comes to istio without operator, it's also taking time to pick up, but this is happening for armadillo bison. At the same time, all together, and if you look at jaeger kiali, all the other tools are being installed and prometheus is also part of the installation here. So I have done only creating application for augustini, which took all the configuration required for istio to work with all the observability tools, as well as the installation path.

A

So when I get back to istio without operator, this is where I can see, what's actually happening with istio and what's hap, what's included within istio. This is this may be a little bit of too zoomed up um zoomed in uh view that you might not be able to see everything in like nice view, but you can see how many services and how many things are installed with istio.

A

So this time you can see um with a service installation, you can see quite a few things from up above external egress gateway, external ingress, gateway management, ingress gateway, multi-cluster, egress and ingress gateway, easter, core dns and listed and from down here its service account and then goes down to crd and further down to deployment.

A

So in the deployment is your core dns? Is your d uh already running if I can scroll to the right already running, but from github's perspective, it's just trying to sync and making sure that everything is running from the github's perspective correctly and also synced.

A

If I get back to all the applications, I'm also getting to istio traffic management. So this one is: is your traffic management, local? It's cut off, but if I go into app details I can see that it's looking at the report sheet and also it's looking at the path of clusters- armadillo agua city, 143 up so istio, traffic management and local.

A

So this one has a destination rule virtual services and few virtual services, and if I get back to the repository and if I just go back all the way up and to the root of the repository, there is this clusters and as we saw clusters armadillo and istio and then for traffic management and local.

A

So I have three files and if I look at color service, for instance, you can see that destination rule is defined, virtual services is another and for this file, that's it and you can see armadillo color service destination, rule matches up with the destination rule here and I can actually see if there is any diff um make sure that everything is in sync, I'm having a little bit of slowness here. So I'm going to probably close that and let me get back to the application stack.

A

So at this point I have traffic management setup mostly done. Let me see if the demo setup has been completed.

A

And it's taking a little bit of time.

A

And it has done envoy filter, but some of the services, some of the setup has not been done completely.

A

So if I get back and let's just double check what is happening with bison cluster, because we have been only using armadillo cluster and I have another port forward set up with agua city. So you would see this a warning if you try to do port forward in the first place. But you can proceed with the localhost setup and when I log try to log in.

A

If I can close up yes, thank you, I'm doing the login with admin, and you can see that this time app orchestration is looking at the cluster bison rather than armadillo design. So if you look at the application detail, you can see that it's the same repository, so I'm doing a monorepo kind of approach, but the path is different so that the configuration for this cluster is different. I'm not saying that monorepo is the best way, but this is one way to work out.

A

So if I look at the state of the uh the cluster right now, istio is still being kind of progressing from argo city perspective, but it still itself should be running already. So what I'm going to do is to try to query the istio sorry color services running on both classes. So if I load up the ui, this is a really simplistic ui, but basically it adds the base address and also endpoints and then start querying every two to five seconds randomly.

A

So if I try to query three two or one, if you remember from the slide, this is armadillo cluster and if I try to query armadillo cluster, it should give me all the blue-ish colors and it's giving me an error.

A

If I do that for bison clusters and that's going through armadillo clusters and that's going to give me an error as well so what's going on- and this is not expected. So I need to solve it right now. But let's just double check what's happening with august cd and if I get back to armadillo side of.

A

Setup and if I look at the state status, istio multi-cluster demo2 is all set up. It's still without operator. This actual eastern installation is still coming up. The processes are the deployments and they are running about an hour, so they are all healthy.

A

So that looks to be working all okay and I could also look at the logs if I wanted to. If I this is not the best log viewer, but I could get into the logs here and it's giving me some uh it's pushing routes and it's not giving me so many errors. So hopefully that's that's working right now. If I get back to uh not this one. Sorry, if I refresh and make sure that I have three two, oh one one and number of cells, six, I'm gonna do if I query it.

A

Maybe that has been taking time to sync some of the configurations now this time finally worked. So this is now getting the blue uh kind of colors from the armadillo cluster, and now I'm going to try to query, buy some clusters. So this time the bison cluster has red, reddish, colors and because I'm base I'm pointing 3201, which is armadillo cluster, I'm going to get the colors from bison cluster through armadillo cluster. So this is going through multi-cluster, ingress and egress gateways.

A

And now it's giving me all the red colors and this the same setup can be done or has been done with the other way around. So if I base the address from bison to armadillo- and I say- set the number of cells to 0 to refresh six, this is a quirky ui that I created it's not great, but it does the job and let me just refresh that- and you can see that three two one two is being hit and also bison is still giving me all the colors.

A

So it's working nicely and one last thing I would attempt to do- is get back to armadillo and let's get the six things out into place, and then I put the periodic query and lastly, because it's a git ops setup. So I have created this pull request earlier on. So it's to add fault injection for local armadillo cluster routing. So you can see that traffic management of local color service has been put with this 30 50 goes to http status of 400.

A

Let me see if I can make this work, merge this and then go back into a modular cluster, so it was a local configuration. So if you go back, if I go back into this traffic management, local and then I just forced the refreshing, you would see that it would pick up that the latest change, so you can see the yellow sign here. Armadillo color service routing has been updated or is being updated uh when it goes to the green. It's all great, um don't have too much time.

A

So hopefully this can do in like next five seconds.

A

If not, I might have to leave it there.

A

Let me just double check how to sink. It has not picked up.

A

Okay, I'm fortunate, but I don't think I have the time to do that.

A

So let me just get back to the slide and if I have a little bit more time at the end, I might so am I in the right place.

A

So uh after this demo, so we have seen some multi-cluster with github setup in action, we use still custom resources uh in action such as virtual services, uh virtual service destination, rule, etc. There are many pieces I can get into due to the time constraint. For instance, amboy filter gateway um all that, but hopefully the demo has shown some seo setup. That is relevant for your business use case and even for getting started. Multi-Cluster is a great power. Ac provides and is worth playing with again, as everything was driven by gear poultry.

A

Hopefully, the configuration files can be used for future references. So what's next, uh because we only brush the surface of these two offerings. I'd encourage everyone to check out more talks from isiocon, especially once listed here, uh which I found really interesting. I have enjoyed a talk from rob salmond about titled what envoy hears when istio speaks that was from yesterday, and there are many talks, many more talks that can help you go further into production, onboarding uh journey and also a quick note of some debugging help.

A

uh Amboy and istio proxy produces this cryptic looking blogs, but they are really helpful. So, if you want to understand, uh there are the hear, the links and finally, uh before wrapping up. uh Let me share all the links, including the links to the slide. uh This particular slide deck at the bottom.

A

It's a tiny, url uh dot com is your con 2021 and gitobs, and also apologies to those who are looking for multi-cloud multi-cluster setup demo, which I listed in the session notes originally, but I couldn't make the time to get it part of the presentation today, I'm hoping to work on that as a follow-up, hopefully sometime in the near future, and with that. Thank you very.

A