Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Intro: Telepresence - Daniel Bryant, Datawire

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Intro: Telepresence - Daniel Bryant, Datawire

This session will provide an intro to Telepresence, a CNCF Sandbox tool. We’ll talk about development workflows for Kubernetes. We’ll discuss the differences between traditional development, and different approaches people take to building Kubernetes services. We’ll then introduce Telepresence and discuss how it integrates with different organizational development workflows. Finally, we’ll talk about the evolution of Telepresence and how we are actively moving Telepresence forward from its heritage as a VPN-type approach into a more sophisticated L7 routing layer for developers.

https://sched.co/Zevu

A

Hello, everyone and welcome to the introduction to telepresence talk. My name is daniel bryant, I'm the product, architect, datawire and I'll, be your guide for this maintainer session. Let's get started.

A

So I'd like to start all my talks with a high level, tldr kind of key takeaways and today we're focusing on telepresence- and this is driven by the motivation that in a development loop with microservices and kubernetes, can be quite painful. Microservices means there's many more things. You're interacting with many more dependencies on your service and kubernetes is a container orchestration framework. Fundamentally, it's often running remotely or even if it's running locally, there's many more components in the mix than with a traditional kind of monolithic, single process.

A

Application telepresence proxies your development machine into a kubernetes cluster. This is fantastic. We can run our dev tool chain. Do our development processes locally on the laptop as if we were in the cluster.

A

Telepresence supports several workflow styles and I've been lucky enough to learn from many folks over the years, from our slack and also from previous kubecons at small to large scale workflows and how folks have integrated it into their development lifecycle. I'll share some of those references later on. In the talk there is ongoing work in the telepresence community for splitting the client side and cluster side components we'll dive into that in a bit more depth and bottom line get involved with all this stuff. This is the beauty of the cncf.

A

This is the beauty of the kubernetes community. Everyone is super friendly. Everyone is reliant on our goals in terms of making development and deployment easier in the cloud native space. Telepresence is a cncf sandbox project. We'd love you to get involved and I'll give you some hints on how you can do that and point in the direction of where to get started too.

A

So, let's set the scene, so we all know as developers, we have this inner and outer development loop, fantastic blog post here, I'm referencing by mitch denney, and he talked very much around this sort of inner dev loop. You can see here on the code build test commit where we go really fast, trying to prove our ideas. Often we may be using something like test driven development write, a failing test, write some code that works once we've gone, really fast and proved out. Our our code works. We then go to the outer loop.

A

We may do a pull request. Someone may review that pull request and then also that gets added, typically into a continuous delivery pipeline, built, verified, tested and eventually released and so forth.

A

Now, the challenge with kubernetes is the inner, and the outer development loops can be quite similar, particularly when you've got a lot of dependencies on a service you're working on you might write the code, you have to build the container, push it to a registry and then deploy to the cluster before you can start testing. So, if I'm making small changes on a service, this is quite a slow process. Make must change go through all these steps here, deploy it test it.

A

I really want to go super fast, but I'm kind of stuck with this loop automation does help speed this up. Some fantastic, tooling, here, draft scaffold, garden, tilt, git cube. They all offer various ways of automating container builds in the background and doing some other very cool stuff. Definitely encourage you to check out these tools.

A

Sometimes, though, you just want a really fast inner dev loop. In my java days you know we were often working on monoliths and we were doing hot reloads and I could literally change you know a bit of code or change a bit of a jsp web page, and I could see the results in my browser all in the same sort of process, all on the same local laptop. That's really hard to do now with distributed systems and things like docker and kubernetes as much as they've.

A

Given us a lot of really cool benefits, I still miss this really fast in a dev loop, and sometimes I just want to run my own tools, I'm a big fan of intellij. I like running, you know virtual vm for profiling, all these kind of things, and I can't easily run these in a remote cluster. If I'm running my kubernetes services remotely- and even if I can connect up, you know, there's various debug protocols you can do on the wire. It's not quite the same as running stuff on a local machine.

A

You can't always get that fast feedback. Loop, you're, looking for one thought folks have is: let's do everything locally? You know it's really fast and I can use my tools. Spin up mini cube, spin up, docker micro kx. Take your pick, and this often is a great way to start.

A

If you're on a green field project with a couple of services, this can you know, get you a sort of reasonable way, but there comes some point where your laptop simply won't be able to cope running all the services I'm taking the java space and the net space where I've bumped into and when you're running like a jvm or a clr very quickly, your laptop will get exhausted, trying to run all these heap based processes.

A

So you know there's no way we can spin everything up locally to do all the tests enter telepresence.

A

This is kind of we know the datawire team had this kind of itch and we scratched it by writing telepresence and over the years it's evolved, and now it's a cncf, sandbox project and more more folks are contributing in this is where telepresence was born off.

A

This need for this fast dev loop, let's dive into telepresence a bit more, so I've heard many ways of describing telepresence some folks say it's a fancy: kubernetes vpn for development, a cube cuddle port ford on steroids, I've heard, which is a good one, but fundamentally it's a network bridge between your laptop and the kubernetes cluster. So how does this work? So? You can see in the picture here. We've got a local development machine on the left and we've got a kubernetes cluster running in the cloud somewhere in the middle of the picture.

A

Now what I'm trying to do is I'm trying to work on service a now you can see a in the cluster has been replaced with a telepresence icon, and you can see a prime running on my local machine. That's exactly how telepresence works under the hood. It swaps out my deployment, my kubernetes deployment running in the cluster with a two-way proxy and then proxies traffic destined to service a to my local development machine. I can then run a copy of service a locally.

A

Do some stuff debug test, develop these kind of things and send the traffic back into the cluster? So, for example, if I need to be uh getting some data from service b or c, I can totally do that on my local machine, because that is proxied effectively as if it was in the cluster I'll show you a demo now. This is a very simple java, app I've spun up using drop wizard and spring boot. You can see I've got a. Is this shop front app effectively a very simple ecommerce kind of shop front app?

A

The customer is making requests at the top there we're using ambassador open source, api gateway to acts and ingress to kubernetes. My shopfront service reaches out to the product catalog and the stock manager service, combines and aggregates the data and displays that back to the user.

A

What we're going to do is I'm going to swap deployment on the shop front, and then we can intercept the traffic destined to into the cluster to the shop front, and we can also see what results come back from product catalog and stock manager. We can make tweaks on our local copy of this service and send that traffic back into the cluster, so the external customer us in this example, we can see the changes we're making in real time effectively right. Let's go I'll drop back to my terminal here.

A

You can see I've already spun up a kubernetes cluster and I'm running some basic services. If you look in the bottom right terminal window here, you actually see, I see I'm doing a watch on the pods coop cattle get pods. I've deployed my ambassador api gateway. I've got the product catalog. There ignore the rate limited today got the shop front which we are going to be tele presencing out, and I've also got my stock manager. Our service there running as well I'll share the link to the github repo.

A

In the end, you can see the actual files I've got in my top right window here and now, let's, uh let's actually run the application, have a look. What uh what we've got? If I just bring up firefox and do a coupe ctl get service, you can see. That's my external ip that the ingress is offering.

A

I know I'm running the shop front shop front here we go, you can see it's a very simple e-commerce. App front-end skills are very much not my thing, and this is a literally a combination of the data from the product catalog on the stock manager, aggregated together in the shop front and then rendered you'll notice. The title banner is the color black that'll change when we swap out our deployments so keep an eye on that. But this is fundamentally the very simple shop front.

A

We're going to be working with today, just to prove everything is running as planned. If I do a cube, ctl logs with a follow.

A

On the shop front, if I do a few refreshes you can see, this is actually I'm following the logs on the remote pod running on my gke cluster and every time I hit refresh, you can see there after a while or two you can actually see the um get stock logging call. So you know it's actually going out to the cluster making my call on local machine 234, 73 162 117..

A

uh The request is going out to the ingress forwarded onto the shop front, folded onto product catalog stock manager. We can see the logs going back there now. So when I do the swap out, you can actually see this a bit differently.

A

One interesting thing I do like to do as well is just if we try and do say an ns lookup against our product catalog like so we'll get no data back, and this makes sense right, because the product catalog service name is a concept within the remote kubernetes cluster, but on my local machine, I'm not connected into that network.

A

So I can't do a name name service lookup on something that is in a separate network when we log in with telepresence when we use telepresence to swap the deployment, my machine will effectively effectively be proxied into the remote cluster, and then we can do this kind of thing. So let's do that now.

A

If I do a telepresence swap deployment on the shop front, have a look at what's going on on the bottom right panel, here you'll notice that the shop front service itself will get swapped out with a telepresence proxy. Now I'll put my pseudo password in there. Telepresence does some cool stuff with the networking, so it does need root access.

A

As this is ticking away, you see the shop front, just got swapped out there and there's another one being created, and this with this uid here is effectively our telepresence proxy spinning up in the background, when that's complete, you'll notice, telepresence will come back and I'm now swapped the deployment successfully and there we go.

A

So if I now refresh my uh website, we should see a failure yep there. We go because I've swapped out the shopfront service, and that was my primary service that we were routing all our ingress traffic too. This is a default error message: I've not bothered to change the envoy. The ambassador error message there, so you'll notice we get an upstream connector. This just means that the shop run service is effectively not there now, because telepresence is routing that traffic, the traffic destined for the shop front to my local machine.

A

Now, if we go back and look at the ns lookup now you'll notice, we get some data back our because our namespaces we're actually proxied into the remote cluster. We can do cool things like look at the product catalog. We can even get the data from the product catalog. So I find this really useful when I'm doing some integration testing or trying to understand and build my mental model of how the dependencies work with a service, I can telepresence swap deployment and I can start looking around.

A

I can do curls test the data coming back as if I was in the cluster all on my local machine. This is very powerful. In fact, let me show you that now, if I do a curl, you can see we get the product catalog data back. This is what's happening in our java shop front, app. It's calling out the product catalog, getting the data back, munching it with the other service data as well. Now, let's start a local service running on our machine here.

A

If I go to intellij, I've got a copy of the shopfront application, all loaded up in intellij. If I start it in debug mode like so just wait for that to spin.

A

Up that looks good if I now make a if I move that window across and if I make a request on our docker java shopping here, you'll notice, it now completes we've got the red banner and that's because my request is going to the from my local machine to our remote kubernetes cluster, going through the ingress and then the swap deployment is routing that traffic to my local machine for routing what I process on my local machine back into the cluster.

A

So we've got that loop now from remote to local, and I can do very fast feedback locally on this service in intellij hit refresh in my browser and see in real effectively near real time. The changes I'm making you're going to get that hot reload based on the service, you've swapped on the service, you're working and because it's all running in debug mode, it's kind of cool and long as you've got the ingress timeout set quite loosely. You can actually debug in in real time effectively as well.

A

So if I set a breakpoint on that there do a refresh, you should see the breakpoint gets hit. Now, when I do, I can return it through page loads, all good. If I try this again and I'll I'll modify the actual data that we're returning back. If we go into the values you can see here, the premium acme widgets. If I just change that now to premium kubecon eu which it set that return it through you'll notice, we get that live reload. I can change the data.

A

I can change the look of the website and so forth very fast feedback loop going here. I don't have to be in debug mode. I can actually be running my application and just changing it on the fly, but I find it particularly useful for debugging, sometimes where I'm calling out to lots of dependent services when I'm in debug mode.

A

I can see the data coming back and I can see it being processed as I step through the code, and that is just really useful for spotting those kind of gnarly micro service bugs that we will bump into when you've got many dependencies right now, let's clean up. If I stop my process here, that's good I'll, minimize, intellij and I'll exit from my telepresence terminal up here.

A

You'll notice that this the stock manager is getting spun up again. A real stock manager instance and we're terminating our tele presence proxy pod. Give that a few moments. While that does its thing, and we will need to give a few moments for the stock manager to actually spin up successfully it's a java app as well. It goes for its kind of initialization process with spring boot. As soon as that's ready. We can then um again refresh our web browser on the right here. If I just do a logs again.

A

We can watch for when this one has spun up correctly and give it a few more moments, and I can do a refresh.

A

There we go, the color has changed now on the on the header and effectively we're back to normal I've swapped the pod successfully, and now I've removed that swap we're back to normal and voila.

A

The demo hopefully made it much more concrete as to what we're doing here, the benefits of a tool like telepresence is that any tool that runs in your laptop can be used. In this scenario, you saw me using a debugger there, but I've also used the profiler anything. You can run on your local machine because you're proxying traffic in and out you can do.

A

You can connect to cloud-based resources so frequently, I'm in amazon, I've maybe got an rds instance that is only available in the kubernetes cluster. Well, if you can proxy into the cluster with daily presence, you can now access the uri of that cloud-based resource that rds database, for example, we're in the same network namespace I've showed you the ns lookup. I frequently find this is super useful for ns lookups or digs trying to figure out. You know gnarly kind of domain name issues, and this is super useful to run again.

A

Local tooling, as if I was in the cluster- and you saw hopefully the inside of the very fast inner dev loop yeah, I can code in my intellij locally on a service and that service can call out to all the other services running remotely in my kubernetes cluster. I can spin up all the services. If I want to you've, always got a balance there. You don't want to have an overly coupled system, for example, but some systems I've worked on.

A

You simply have to spin up a bunch of stuff, and we can do that now in our remote cluster and just keep our local laptop for the service we're running on now. Obviously, there is some requirements for telepresence. You do need a network connection. It's not like running everything in mini cube locally.

A

You do need that connection into the remote cluster. You also need coupe custom access to the cluster. I've worked on some teams where operators do restrict access to cube cattle. uh They, for example, insist that everything gets deployed via a sort of custom, gui or some other kind of process.

A

The way we bootstrap the connection with telepresence, you need coupe cattle access on uh on. You need coop cattle on your local machine and access into the remote cluster. At the moment, telepresence primarily works on mac and linux. We have had stories of folks working successfully only on windows, but the networking stack on windows is very different, particularly across the different versions of windows.

A

So if you do have windows, you want to get involved in the conversation pop along to the telepresence github repo pop, along to our italia presence slack, you can see what folks have done in the past and you can kind of build on their work.

A

Telepresence is a cncf project. There is lots of different users out there. We've been super humbled by the amazing talks. People have done at previous kubecons. There's lots of blog posts out there very active slack. We've got going on on the data wireless slack, it's it's truly humbling and please do get involved. This is the beauty of the cncf sandbox project. You can contribute. You can ask questions and we'd love folks to get involved in internet presence.

A

So one common question I get is, what's so complicated about telepresence, classic hacker news, post right, tele presence is so big. I could build this over a weekend and you maybe could build a two-way proxy for kubernetes over a weekend, but one that works on all different types of machines with different types of clusters. That is a real challenge. Yeah, you know abbey the lead developer on a tele presence at basic santa who works at datawire uh I've.

A

You know I'm presenting here today, but he does all the hard work and I've had some fantastic chats with abbey over the years. I've come to realize how gnarly some of the networking is involved in you know, kubernetes and and many other things, capturing dns and routing. Only part of it is really tricky. So we've had some very interesting challenges around um how to route traffic into into a kubernetes cluster. By telepresence.

A

We still have challenges getting telepresence to work 100 of the time or all we have done in the past. I should say those issues have mostly been fixed now, but it's really tricky swapping out. Your real deployment for a proxy deployment is fundamentally quite hard yeah, so we recommend don't try this at home. Why not build?

A

On the you know, the work is already out there again: telepresence open source, cncf, sandbox tool, you're free to do pull requests, you're free to sort of modify it as you want and we'd love you to contribute back in so that everyone benefits from your modifications.

A

So let's look at a few um workflows now, where we've been lucky enough to learn from many folks over various past coupons about how they're using it. So I wanted to share a few references. We do often find that sort of people that are say pre-production or looking for product market fit or not. Very many customers often use telepresence in a production-like environment. There is one production customer there's a single source of truth of microservices, maybe there's a few developers on the team. They simply coordinate.

A

You know either by slack or in the office, and they simply say I'm going to check out this microservice I'm going to telepresence this microservice. So can you avoid using that that microservice for a while um this? Is you know pretty simple stuff and telepresence? We use it in the default mode and once the coding is done like the example I showed with intellij.

A

Typically, then folks would commit the code into a um into version. Control might be some reviews going on and they'd use the ci cd pipeline to actually deploy the service. So this is a ridiculously simple. You know in production kind of you only live once that workflow. Some folks already get very nervous about. You know running things like telepresence and production, totally fair enough right. You might want to see the world burn, but I'm sure that many people in your company don't the typical workflow we see for small medium enterprises, small medium businesses.

A

His production is ring fenced off, but there's one development cluster, which is a single source of truth for microservices developers, often work locally as long as they can with mocks and stubs these kind of things, but when they need to test with real dependencies or they need to do some kind of integration testing, they use telepresence to swap out a microservice, but they coordinate with the others on their team. To do that, hey I'm testing the shopfront service.

A

Can you not test involving the shop and service at the moment, because if anyone makes a request to that shop, front service that we've swapped out their traffic comes to our machine and we might be doing special things debugging, for example, which would mess up their workflow often same deal once this is done. People use ci cd, something like circle ci to run test, build and deploy nothing really different there.

A

This is just you often see this one dev cluster sort of expanding to two dev clusters, three dev clusters- sometimes- but this is a very typical workflow we hear about in the slack on all the time cesar buddy of mine did a fantastic talk about how he was using telepresence uh to do exactly this at devoxx uk last year.

A

So if you want to know more I'll put the link there, you can pop along to cesar's, talk and and learn more as a more advanced workflow building on what I've just mentioned, about running multiple clusters and developers being assigned to those clusters. We do see some folks running multiple clusters and multiple name spaces, so each developer, each pair each team gets access to their own cluster or their own namespace, which makes coordination easier if you're swapping out services as part of your development, workflow fantastic talk by christian roger last year at coupon eu.

A

I believe it was put the link in there for you and he talked about how engel and volkers are using this workflow for quite a big team. They were using bazel or basil, depending on pronunciation uh integrated with telepresence and a local ide. They were doing some really cool stuff about how to coordinate with the teams in terms of when you're, swapping out things or not swapping out things. I learned a bunch from christian.

A

He came by the databar booth afterwards and we chatted quite a bit- and this is a really nice way to extend the workflow if you're, going into sort of a bigger, more classic enterprise style, workflow.

A

So moving on what's next well telepresence as I've kind of talked about it, there's two main components: there's the client and the proxy. The client is what I'm running when I'm typing telepresence swap deployment and then the proxy server is what gets spun up in my remote cluster to manage the communication. Work is ongoing with the client, you can check out the progress on github, there's a move.

A

um We use cooperative portforward to bootstrap s, shuttle, which is an ssh based vpn uh has a lot of functionality, yes, shuttle, but we're not using all that intelli presence, so we're looking to require replace a shutter with something else, something you know more designed for for kubernetes. This is in progress. You can pop along to the slack channel pop along to the github chat to abbey he's the lead developer, several other folks involved in the project there as well.

A

If you want to learn more about this, the proxy server yeah, as I showed you it's literally swapped in and out every time you do a swap deployment, and you know this is a bit clumsy in some ways and then you've got to spin something up every time. If your connection breaks sometimes that you know, will then roll that back and you have to spin it up again, whereas we're looking to switch the mode to be persistent. You know this is what kubernetes is good for right.

A

You define a service, you define a deployment, kubernetes will keep it running, and that gives much faster startup times. We can actually do a telepresence swap deployment or telepresence connect and very quickly, if something's already running in kubernetes, you can connect up super quick and you get that reconnect if you're working on a plane or a train, and your connection drops no biggie. If that thing is running in kubernetes all the time.

A

This is leading us to multi-user development and because swap deployment doesn't work really well for multi-user development. I've had this scenario where I've swapped out someone else's swap deployment, and it all got a bit messy right. It was a bit confusing, so telepresence is fantastic as a single user tool and you can use workflows, as I've shown you earlier on to sort of get around this limitation, but we're thinking it'd be super cool.

A

If you can um do this, multi-user use case for your team just to run you through how this might work at the moment or an abbey actually gave a demo at kubecon san diego last year. So you can pop along to the website and pop onto youtube and actually see his demo, but imagine our classic shop front uh demo, we've just shown if we did this sort of telepresence for teams kind of thing, we'd actually put telepresence on the wire as opposed to swapping out the actual deployment itself.

A

So you can run multiple deployments but then use telepresence as an indicator. Telepresence. This request should be routed to this version of the service. So maybe we put like a you know: telepresence runs on the wire here. User makes a request. Maybe we put some kind of cookie or some kind of header that goes through into the ingress through into the service and then route it on to telepresence.

A

Telepresence says ah if this cookie is present like daniel's test, for example, it will rank to my local machine, the traffic proxy to my local machine, but it still will allow other folks without that cookie set to access the default product. Catalog, you imagine. If there's many of us working on the product catalog, you can have the production effectively the production service running, but I can have daniel's product catalog and jane's product catalog.

A

As long as we're putting the appropriate headers in, we will see the services running on our local machines, but other developers will not again, if you're looking to get involved in this looking to learn more definitely encourage you to pop along to the telepresence github, have a chat with abby and then the slack as well.

A

So wrapping up we'd love folks to contribute to telepresence. um You know, documentation is super super important in these projects, as well as sort of talking about how you've integrated telepresence into your workflow um love. You know do pr's on on the website and chat to us on slack chat to us.

A

You know twitter wherever you'd like and we'd love to hear, even if you're a user of telepresence, it would be super useful for us when we're working with the cncf to sort of you know, constantly review the usage of the tool if we know your user that helps us build the case to keep telepresence as an ongoing project.

A

We're always looking for folks to triage things, there's lots of folks using github for support, which doesn't always scale well on an open source project, and you can also join our slack to um you know more about these things, ask questions and so forth and yeah we'd love you to get involved.

A

That's it for today and feel free to reach out to me at db, datawire.io, I'm on twitter, github, linkedin and so forth. That's daniel bryant uk. You can visit the tech presence website and github repo there, and you can also check out some of our early work with the ambassador edge stack, integrating tele telepresence there as well. Thank you very much.