Istio IstioCon 2021, 10 Mar 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Local Istio Development

Description

#IstioCon2021

Presented at IstioCon 2021 by John Howard.

This talk will walk through how to run Istio locally to improve development velocity, where “local” includes various combinations of local Kubernetes cluster, local docker registry, running Istiod as a local binary (and in a debugger), and running the proxy locally.

A

Hey everyone um welcome and thank you for for joining um today, we're going to be talking about local east geo development, so I've been working on east go for about two years now and over that time, I've I've built my own builds of east, joe thousands and thousands of times, and what I found was that after I started working from home and my processing power and internet speed dropped dramatically, my productivity fell quite a bit.

A

So during that time I worked on a variety of ways to speed this up by various means of testing locally, and I want to share these ideas and um uh yeah with you with you all, and so what we're going to do is go over a few high-level topologies of working with easter locally, and then I will attach a github repo that has more extensive scripts and commands to actually enact these.

A

So with that being said, let's get started.

A

So our first and uh most common setup- this is kind of our baseline. That we'll be comparing everything else to is the fully cloud environment.

A

So this would be just like you're testing, with a real kubernetes cluster and some cloud provider, and you know real docker registry, and so every time we want to make a change, we would build all the images push them up to the registry and then do some cube code commands to modify our deployments to use the new images and then our kubernetes cluster is going to pull from that registry so end to end. This can take quite a while for for me oftentimes.

A

This was in the minutes just to test even a trivial change between all the pushing hundreds of megabytes, and then the kubernetes cluster, of course, has to pull those as well. um So this is quite slow.

A

This does have some benefits, though, there's nothing running on your machine aside from the build so that there's minimal utilization uh and it does closely remember- resemble production environments. So we can do things like running large scale tests as well, or things like that require external load balancer. For example, like I mentioned, though, this is extremely slow in some cases and it can be very expensive as well as you need to pay for the compute and potentially storage costs as well.

A

So the easiest improvement we can make on that is doing the exact same thing, but bringing everything locally.

A

So here we can see uh almost the exact same setup. We've replaced the cloud kubernetes, though, with the local kubernetes and the cloud docker registered with a local docker registry, um so docker ships a like a local registry that you can run within your own docker daemon locally and then there's a vast array of local kubernetes offerings. So you can run a full kubernetes deployment on your machine.

A

So there's things like kind, k3s, micro, k8, mini cube, there's a lot of them out there. Now I personally use kind which is kubernetes and docker, but you're free to use other ones. And this, when you do this, your development cycle looks pretty similar, you're, still pushing and pulling images and doing the same cube, cuddle commands, but because everything's local, it's all um all the docker, pushes and pulls over a local host, which is extremely fast.

A

So, like I mentioned yeah, this is much faster. One other benefit as well is that it can be a lot more reproducible. It's hard to go, tell someone to go spin up. You know a gk cluster and run some tests, but it's very easy to say: go set up a kind cluster with these exact parameters and everyone can do it, especially our testing runs on kind.

A

So if we have an issue in a test- and it's not reproducing in you know, gke or something it's very likely that it will reproduce if you follow the same setup that the tests are doing, it also allows really bespoke clusters. So you can do things like enabling alpha features, testing, leading edge, kubernetes versions or doing complex multi-cluster setups.

A

There are some downsides, though everything is running locally, so there's a bit of overhead.

A

It doesn't take that many many resources to run kubernetes, but if you wanted to run like five clusters on a small laptop, you may run into issues and while things are faster, there's still some overhead because we have to build the docker image push it pull it spin up a pod which is now it's not an order of minutes now, but there's still some slight overhead and because everything is running in docker, it can be kind of hard to attach a debugger.

A

It's not impossible, but there's a lot of complex steps to get it set up so to resolve some of those issues. What we can do is actually run all the estro binaries locally, so these are just standard go binaries that we can run with go run just like any other, so we can run on our local machine. East od and pilot agent and easterd will connect to our kubernetes cluster. That's just configured with uh cube cuddle out of the box. So this it is actually as simple as just running this command.

A

There's no parameters required no configuration to set up. There are a few with the proxy to enable, which is documented in the github rebuild at the end of this talk. um So the great thing about this is that it's really fast, um the only bottleneck we have is go compilation time, which is you know seconds.

A

It also makes it really easy to run a debugger or even just click like the go run button on your ide.

A

So this is really great to do. Rapid iteration of various changes to istio.

A

One thing that may be challenging is it's not exactly representative of the production environment, so it may be hard to test things like web hooks, or certainly like scale testing.

A

It's also kind of hard to test actual traffic, especially involving ip tables. um You can probably do it, but it's it's not trivial for sure, and it this probably only works on linux to run uh envoy.

A

So there are some downsides, but if you can get this set up, it is the probably quickest way to iterate.

A

There's also some variations of this, where we could run just the local proxy, but have easter d running in the cluster. This would be like doing a port forward to the easter d. This is pretty simple. It has a lot of the same benefits, but also the same issues as the previous one. But if we're just trying to develop the proxy, we may not care to run east duo locally.

A

uh We can also do the inverse of running estiod locally, but they proxies remotely. This, of course, has all the benefits of running e2d locally, but we also have the benefits of running the proxy in the cluster.

A

The one main downside here is that it does require connectivity from the cluster to the local network. um In my setup this I can only do this with a local cluster, um but it's still quite handy for certain certain testing.

A

So all these other examples were basically just variations of kubernetes, estrio and the proxy just deployed in different places, but we can do some more interesting setups.

A

So, if we're just testing like what envoy configuration does it can be really quick to just run envoy directly, so android can take a plain yaml configuration file that has just like the similar output of the config dump from envoy and run that directly, so in this case, there's no kubernetes or estro involved.

A

This is, of course, super fast because there's no easter dependency. It also is really great for minimal envoy reproductions, especially for posting issues to envoy.

A

um This, of course, is very different from a real easter environment and it can be kind of challenging to get the config right if you're not familiar with programming envoy directly, it involves you know, hundreds of lines of the animal.

A

uh We can also do the opposite, and instead of removing istio from the picture, we can remove envoy and instead connect to easterd directly from various other clients.

A

So the proxy to easter d connection is done over a grpc stream, so we can use a tool called grpc curl, which is basically curl, but for grpc and there's plenty of other tools that can do similar things and actually send direct requests of whatever we want to estio directly.

A

So the main benefit of this is that we have complete control over the requests, so we could send things that even envoy, potentially wouldn't ever send by completely invalid requests or oddly ordered requests that sort of thing and it the you know. The main challenge is that it can be hard to reproduce what requests the proxy would actually send. So, there's a lot of metadata, et cetera, that's actually required in the request which can make this a bit challenging.

A

So that's all of the different methods, there's no one size fits all. I think all these have proven cons and I use these all routinely for different setups, depending on what I'm testing I'll use. The different setup, so I encourage you to try them all out in this talk. I just described the high level overview of how this works, because we only have 10 minutes, but if you dive into the github link here, you have details on exactly how to set these all up with some scripts and aliases.

A

So please try it out. Let me know and good.

A