Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2017 (Austin), 14 Dec 2017

Previous Meeting

⏯

youtube image

►

From YouTube: Linkerd Salon - Hosted by William Morgan, Buoyant

Description

Linkerd Salon - hosted by William Morgan, Buoyant

About William Morgan
William is the cofounder and CEO of Buoyant, a startup focused on building service mesh technology. Prior to Buoyant, he was an infrastructure engineer at Twitter, where he helped move Twitter from a failing monolithic Ruby on Rails app to a highly distributed, fault-tolerant microservice architecture. He was a software engineer at Powerset, Microsoft, and Adap.tv, a research scientist at MITRE, and holds an MS in computer science from Stanford University.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Hi everyone we'll go ahead and get started so welcome to the link or D salon. We're gonna run this as the third ever link or D community meeting, so I'm George Miranda I run community at buoyant. So thanks everyone for showing up we're gonna go through a small agenda today, we're gonna start by talking about linker D. So we're gonna hear from a few folks that are running linker D in production I'll. Let them introduce themselves as they come up, but we'll do some short lightning style talks not quite lightening s great.

A

So more of like a Tesla coil talk, I, guess and then Oliver is gonna. Come up! We're gonna, look at the architecture behind conduit, so we're gonna do a little bit of a free-form whiteboard session. Make that interactive. Let you ask questions as they come up and then we're finally going to end with doing an engineering AMA panel. So if you have any questions about conduit or link or D or anything, that's on your mind. It's a great time to interact and get those all answered.

A

So with that I'll go ahead and hand it over to Boris.

B

Hi, each time ago, a chance talk about energy I'm, slightly confused, because there's so many great things to eat that I don't know where to start. But today hey wait, wait, wait! What does it do? Wait yeah today, I'm going to talk about Allatoona, CP traffic, this energy, giving a practical example as we do it, and just as you're said, my name is Boris Bureau and I work at the NCBI, which is a National Center for Biotechnology information.

B

Essentially, we collect in biomedical data from all around the world, protesting them and then expose them back to all the resources and so on. Well, NCBI doing software for almost 30 years. By now, and as you can imagine, is that we have an injury situation when we have some modern services built with latest available technologies. Running alongside with some legacy code built a decade ago mean only hybrid infrastructure, with on-premise data center and several public clouds.

B

We have over 2000 services and most of them a legacy actually and we serve in an average 17 requests per second half of that via link Rd and as you can imagine, there is no way I just hand for our new modern services. You'd be responsible for half of our external traffic, so yeah we indeed use linkage.

B

Eve is our legacy services, and that is possible just because using linkage, E is so easy as that you can do it with legacy or you can just put it into one single liner and if you look on cert it is there a couple of components that make it possible. So, first of all a linker, you can operate as a transparent, HCP proxy. So the only things that we need to do is just inject link as a proxy for application.

B

So this means that you know does much if you use docker or whatever or kubernetes unrelated completely. Just you know, inject linkages, proxy ins, that's it!

B

Luckily, we can use variable HTTP underscore proxy, which is a de facto standard that is supported by a majority of CP clients available today and next we use this inventor able to point our application to linker G instance, and usually you want to run linkage on the same well virtual machine or Hardware host to reduce latency introduced by extra hope in your quest processing, and you can use just no local host here, but you prefer to use in Korea alias that we set up via etc' costs because might be different depending on your environment.

B

If you just ran on something, you know play an old-style just on the same everything on the same costs, you can just bind it to a loopback, oh sorry, local host. Otherwise, if you're in containers as a linker, your workloads or balls, this might be something more interesting that would make linker G available to your application, considers in network name space. Last but not least, is how exactly we cover service.

B

As you can see here, we use a special specially composed URL, as well essentially domain name to tell link ug what seriously want to talk about talk to so we use service name and then followed by a special suffix which is l5g, and then our top oil domain and cbn eliminated grove, and that's it. A link, Eugene knows about this convention and whenever it sees a call to such domain name, eaten Pratt interprets it in a special way. So, as I mentioned, link ug can work as a transparent, HT proxy.

B

So actually it's only since it cares about is host header, which is it tells you about a course that has been requested, it's required since HTTP 1.1. So we don't need to worry about the scatter being something special and so yeah. There might be situations when it's not practical or even impossible to set in qg as a proxy for the whole application. So you might use this approach in such cases and we actually use it on our edge web servers.

B

So here is a small snippet that shows an example of apache configurations that tells that whatever is under foo in our URL is going to be handled by an applique, a service called foo, and that's how we make this work so linker g, that's how we can do how our production traffic via link ID now for linkage operates that way. It needs to be configured in a special way. Here's an excerpt from our configuration I left out all the technicalities.

B

This is what really is important for this, as you can see, port 41 40, that's where we expose linker D and you can figure a router for HTTP protocol and for this thing to work, we also need to specify an identifier which is a special special component, linker D that reads: information from request and produces so-called logical name. Essentially, some kind of strings that tells in kg vs. request should go in this particular case.

B

We use a little file called header token, which essentially just takes the value of Fushigi host header and appends it to the availa specified in DST prefix. So if you look at this simple example with some minimal HTTP 1.1 request, that's how it will be converted to a logical name. You see know as simple as that. Now this in details incur the where this request should go to. It doesn't tell like: what's version of the service you want to use always locator to sounds like this.

B

In order to get that information link, you genius to convert this logical name, interest, a cold, concrete name, and it does so by evaluating this logical name against a set of routing rules, also known as delegation table. This is essentially a piece of our delegation tables if used in production in our environment, it's greatly simplify it and so many details that are not shown here for the sake of brevity, but it does show these main concept behind this, so location tables are related from bottom to top in a recursive manner.

B

So you trying to substitute the prefix with some sense that you specify on the right side and then repeat that again again, unless you can do any transformations anymore and if you look at top two lines, you see that we essentially for HTTP prefix, that you saw before we essentially defined two rules. First you'll try to to call sync all domain to pass.

B

Let's not worry about it for a moment, and otherwise we following back to thing called error in net or like reverse, in at name or whatever, which essentially just does a DNS query and that's it so, in other words, you can use this configuration to inject link ideas, HP proxy, for any application without affecting any behavior that application. So you can just start using it right now and then you, you start agents on special handling for some special URLs, as we do for URLs at envious, l5g and CBN eliminates that go in this particular case.

B

You can see that we extract a service name from such domain names and then we resolve it from a service registry. We use constants this particular example, but no production set. If we actually use a combination of several service registries and well, that's what you can do for the sake of this example, we have services full bar and bars. Each of them have well version one set as current version and well just a set views console sorry for change of background.

B

We have services footballing boss, version one and two for each of them, and so this is how name resolution looks like a linker D. This is yet another great fish about incurred year that it can show you it has a UI. Is that can show you how exactly name resolution happens? So if you new to donation table, you can just use this UI to learn how this thing works, or you can use it for testing proposes to verify, as it should engage tables correct or to understand how it works.

B

Here you can see is that at the top line we have a logical name, fool, 50, NCBI, no emulation go, and then it's been converted into ap address and port number. So that's how our how we handle self discovery, essentially, in our case here's another example. When video gives some requests that doesn't go to our internal service about something else like example.com, we just do a DNS query and you can see is a peer doesn't now from here on.

B

I want to tell you about one extra feature of linkages that you find particularly amazing, and we love the most and it's called dynamic request, relgion, making an example of fubar and Buzz. Let's define the dependency, so fucose bar and bar call buzz now, if I'd link it to the picture is going to look like this way. Where link ID is well as part of every call. Now remember. We have two versions of each service, so actually heaven is all the default version set to 1. We see who one calls bar one calls buzz-buzz one.

B

Now, let's say that our applications engineers in a way so that they know how to pass context where, by context, we mean all set of headers that were passed into the fou with the originating request and while full works on Canada's original request, it will pass those headers with all subsequent nested calls so that well, you can do trace.

B

You know, but not if you have such a feature, then we can make use of linkages dynamic routing here you see the same call as we did at the beginning, but an extra thing that we do here. We pass the special header alpha TD tab that you expect to be forwarded as a context. Now, if you do that, then our Col chain is transformed and it looks this way, which essentially means that we change an arbitrary service as the arbitrary depth of our call chain to some different version for testing proposals.

B

What's important to understand here is that this is different from canary, where we just breathe in traffic between different variants of the service. We just affect one single request, so you can test whichever you want without affecting anybody, but you- and we use this at NCBI- to eliminate our need of station environments with more than 2,000 3,000 of services. You can imagine well how simple things become with this and you can assemble arbitrary call chains. So no more issues of staging and that's why we love Inc early. That's it.

A

All right up next, we have mark Tyrell from home away.

C

Okay, I feel like I, probably don't need to say much more, but will I'm gonna take a step back, so the this presentation is really about the journey. We took to select linka D, rather than showing you our d tabs, although we can talk about that afterwards, if you want to so I mean, we've got some great ideas by the way from that. So that's fantastic. So we had two big challenges: pretty much the challenges a lot of people face.

C

We were going from one lifts to micro-services and we're also breaking up our UI into smaller apps and the other thing was. We are moving from a private data center. Well, some private data centers to some version of a cloud either public or private, basically, both so with regard to routing you know, one of the things we we struggled with is initially you're, going to start out with a prescriptive, pretty manual approach to it.

C

You may then, find easier dsls to work with to describe your routing decisions, but where we're going and the kind of discovery that we've really made this year at home way is when you get to the scale of thousands of applications, you really want this to be much more convention driven and again that was highlighted really well in the last talk, and so it's something to bear in mind when you're thinking about how you want to approach routing so to kind of explain it in the context of how our signal path, if you will has changed, you know what we started out with was you know you had some kind of domain name system resolution that would result in you getting traffic to your datacenter, where you had some kind of appliance that was doing your load bouncing probably some security appliances, I won't talk about, and then eventually you would hit your application and then, in our case you know the classic use case was you would then go back through the datacenter load, balancing appliance to hit your microservices, so where we got to in the last couple is, is we started to conceptualize well, there's a number of layers of what we in our home way.

C

We're referring to is our edge star come. The outer edge is some kind of global low, bouncing layer. The reason we needed that is because, if what we're diverting traffic, our customer traffic between a legacy datacenter somewhere in the US and some public clouds, then it doesn't really make sense to do that in one of the clouds, because then what you're doing is you're proxying all your traffic through that cloud, so we were able to leverage a CDN layer to make global routing decisions, and then the challenge becomes well. How do you coordinate?

C

How do you orchestrate that routing across these different layers? So you know you could start out with you know dynamic, DNS, pretty much. You know something we embraced, and it definitely gives you some additional levers to pull in certain scenarios like distributed denial of service attacks, although it doesn't really save you from them apparently, and then you get into this kind of pipeline for making routing decisions, and so just to kind of give you an example of you know.

C

Some of the things again highlights in the last presentation: we've conceptualized two things: there are lots of places where you have dials, where you can just route a percentage of traffic. One of the challenges with that is it's kind of arbitrary. It's going to be some percentage normally based on some kind of hashing algorithm. It may be sticky or not.

C

What we needed to do was integrate our experimentation platform. The reason is, the experimentation platform uses a more rich segmentation set of rules that say to determine how you segment traffic I, want to target a particular a demographic group in a particular geo, for example, and in that type of scenario out of the box, you're not going to get that kind of capability. At least we didn't in our global low balancing layer. So we had to get kind of creative and one of the things we did was we built our own appliance low, bouncing layer.

C

So when you go back to the stack here, it's what I'm talking out in the middle edge, an integral part of that is the mesh, and we adopted link a D for that I'll. Go into some of the benefits for that in a moment.

C

But one thing I wanted to call out here is what you see in this.

C

This stack is these decisions to split traffic are going to percolate all the way through your your graph and and definitely using HTTP headers is a great way of passing some kind of context around for that or you can just do some kind of conditional evaluation every hop and what we have is a kind of a system, that's a blend of both of those strategies and that's what I'm showing you on this diagram and then the other key part of this which we also touched upon earlier.

C

Is you know if you're running some kind of container based multi pass platform, then there's going to be some kind of foundational level in your host, a set of capabilities that you probably want to decouple from your application frameworks, and so where we've ended up and again, I think this is pretty much out of the link.

C

Addy kind of Docs is leveraging the mesh here by convention, so doing some kind of loopback based routing here for efficiency and again some kind of convention driven strategy with your URL that tells link addy, which service you're, trying to connect to and I think one of the key parts about this. Is we originally struggled with this a bit because we started out with client side load balancing libraries, so it turned it to be much more intrusive in application development. What this gives you is a much cleaner approach.

C

The one area which I actually want to talk to point guys about afterwards is how you make your local host environment. Do all this stuff without running all of this stack, which maybe is the right thing- I have few node developers at home way who don't want to run docker for some reason, but anyway, so just to kind of some end up.

C

You know to give you a summary of where we got to we've been running link addy about 12 months and production at home way, and we have hundreds of instances because, like I showed you here, it's basically running on every host, it's in public and private clouds. We also support legacy applications and are a kind of original infrastructure that kind of predates the multipass paradigm. It's running in three continents.

C

Thousands of applications get routed through it and it's a blend of ephemeral applications and services and legacy applications, and these numbers, I'm afraid of vague I, was told to make them. So so. Apologies for that and I'll just tantalize you with some office, cared graphs just to show you that you know one of the things and I think. Is this as much a kind of shout out to the guys thinking about developing mesh technologies emitting logs, metrics and tracing?

C

You know just kind of shorten that to observability it's a crucial capability that as much as possible needs to be more plug-and-play and what we've been struggling with this year is some of the gaps in the open tracing spec specifically around this, because it's not just a simple set of schemas that you can apply for hop two hop and to emit messages, and increasingly the convergence between logging, metrics and tracing means that you're you've got to think about where there are some efficiencies.

C

Here you know in a big enterprise company, you probably license each of those things separately and so there's some kind of questions there about the overlaps and whether you can benefit from some kind of convergence. And in our case we are, you know, as you can see here, we just have boarding, alerting anomaly detection based on a date remitted from link ID, and that provides a lot of value to us operationally in terms of monitoring.

C

What's going on, so that's pretty much it does anyone I. Don't know if we have time for questions in a lightning format, so, okay.

A

Cool thanks mark all right. Next up is Ben from tabula.

D

D

Hi I'm dead from tabula tabula is a Content discovery platform. If you visited one of those site or any of our other partners, you probably seen our recommendations. Please click on them. I need the money so I'm going to talk about the project that started my entrance in interest in linker D. It was about a year ago where the project with we had to connect our web servers with a new service. The service is the tensorflow serving it's a GOP C service itself. The machine learning models, those those tensorflow serving.

D

We run them on doc, hell running on a normed cluster and the the new thing about it list for us was that we only always served them. Multiple models and the list of model was always changing because we test new machine learning models on product in traffic all the time and we needed some load balancer to do it.

D

The reason I'm giving this example because I think it's a it's a super simple example, and it showed that, even if you own on bare metal and you're, not not, you haven't achieved full buzzword compliance. You can still enjoy using like the latest and greatest technology and I'm I'm, going to show their configuration. That is relevant for this project and it's super minimal. It's like, as you can see, it's like you know, ten lines of vml and- and this is all it took- to get Lincoln a two-out request based on the header.

D

In our case, the the web servers know, which kind of which machine learning model they request on a per request basis. So I'm not going to talk about it, I'm just going to show you that it's short and sweet- and this is the first project that we use the link ad on over there. This is from about a year ago of the year we added about I, think five or seven services do also do PC services and because those services are most simple, there's no different kind of model does not only one kind of instance.

D

Pair service were able to use a single configuration, singly-linked already configuration for all of our services, so this configuration, which is more or less taken from example in linker D, manages to throw out traffic based on the GL PC package, name service and package name GRP see inside the protocol. There is a every quest, has a service and package name and it just walks a single configuration for all of our geo PC services, and that's it.

A

Cool. Thank you sorry about that, intro that it has been a long conference and so with that I'm gonna invite Oliver to come up. So those were some great updates on how folks are using linker D. If you're curious about what's happening at home away, we had a meet-up earlier this week, I believe it was recorded and so we'll post some notes about where you can find that.

E

A

Just disappeared, look back so those are some great updates about linker, D and Oliver is gonna talk to us about conduit and its architecture. I'm all.

F

Good yeah, all right so to warn you I, do not have anything prepared other than I got dressed today, and so we're gonna. Do some white boarding and just chat through things.

F

All right so, first off, let's talk about the last six months of link or D, or so you know. As many of you know, we've been really focused on HB 2 and your PC, especially there's been a lot of stability work going on there, but you probably notice that we haven't been adding a lot of new features to link with you.

F

What we've been really focused on is helping many of you actually in this room, get to production, get the resource footprints under can under control, get the latency profiles under control and that's been going well so linker D is, you know, moving forward to become a better and better production, ready system, and it's widely deployed today in many many environments and a variety of them right and so linker D is kind of the Swiss Army knife of container networking to some degree.

F

Okay, so we have linker D TCP, which is a small rust proxy that can use named Rudy and those DTaP syntaxes for layer for routing and we've been adding HTTP and HTTPS Rifton all the protocols, if it supports but Linkwood, he's very pluggable, and we didn't really describe or burst and describe it in this talk, but they're actually running a bunch of internal plugins that they've written for linker D, because it's a very flexible system right some in JVM it runs anywhere it's easy to write plugins. They can be discovered it at boot time.

F

It's a good ecosystem for that. But that's mend that it's been a little harder to support this, because it's literally like every deployment of linker D is a different configuration. I, don't know of two people who have the same exactly config and that's it's been tough to think through right and every time we have to work with you on your issues. We have to like go back to the ground step, one okay. What problems are you trying to solve? What do you want add a link or D is that security?

F

Is it visibility and or is it routing control, and most of you want all of them, obviously, but there's generally a priority list in there, and so our advice with all of you going to production, has been to make the smallest bytes possible right. Let's not boil the ocean and try to get everything at the company I'd only do it once or make something that's to be really hard project to succeed. Let's find that easy win. That's gonna help the teams that you support that your help, your co-workers, have a slightly better on-call rotation.

F

Next week we don't have to have everything in a mesh with full dynamic routing to do that. That's in a very advanced configuration I think actually both are all three of the configurations. We've seen, maybe Odin's a little bit more straightforward, but all the things that are using per request, routing that that's very advanced functionality. That is very exciting to me and I'm glad it's in there, but I think it can be a little bit of a brick wall when you're getting started with. Who here, like loves, D, tabs, yeah I, know Boris.

F

Does it mark does yeah okay, so, like the people doing like really they've really gotten into it, who loved D times the first time I saw them either of you? Okay, Boris, you're, special.

F

Yeah is any: are you like generating details with Ruby yet because that's that's an eventual state, I'm sure, okay, yeah and I mean. But this is the point. There are very powerful tools in here, but it it's kind of like a box of knives and a lot of fingers get cut, and so that that's, but you want sharp tools right. Nobody wants dull tools, they cut your fingers even more and but so what we wanted to do as we've been talking. What's it, no, no sharp tools are really good.

F

I do not want those tools I like, but you should know what you're getting into don't just open your eyes before you look or before you put your hands in this metaphor, is going a little far.

F

Besides so like, as we've been talking to a lot, you go to production as we've been hearing what the kind of pain points around link reading. So a lot of people get this aging we're only a production, and then is that it starts a scout. Say: oh, you know this like 100 Meg's to start is a little bit rough. Oh, the like warmup time can be a little bit long because the JVM takes a while for its GC to get you know burned in luckily, in most employments finagles load.

F

Balancer is good enough to a rat around that kind of slowness for you, but especially as people get started with, like you know, everyone turns puts a b or work or some weird load balance or a load test to let it and I like yeah.

F

The numbers are really bad for the first couple, minutes of it until it gets better, but so that whole thing's been the eye-opening for us, one that there's this wall you hit of knives and to that the kind of understanding the performance constraints of link rudy, you know, can take quite a bit of understanding the JVM. Luckily there's a lot of JVM knowledge out there and a lot of JVM shops. So that's been a pretty.

F

Some people are okay with that, but but I don't think that's where we want to be, and so we started to think very seriously.

F

I would say in the spring about writing a proxy in rust, and you know at coop Connie you we announced link et TCP, which is a pure layer for proxy. It can route TLS, traffic or whatever, and it uses Nemer d2 for service discovery, and you know there are people using it, but it's it's kind of a was our first getting our toes into the the rough space and then we realistic to think about okay. Well, we definitely want to well I guess my rust would be a question.

F

We definitely want to have this very native footprint right, so we started to think about memory footprint to start right. How do we contain this? Like the JVM is very good at managing memory, but it does needs a lot of its hungry. It needs a lot of CPU and memory to do that and I think there's a better story, and so to be honest, when we started working on linker D in 2015, we were very interested in rest.

F

At the time, Russ was kind of 41.0 Carl over here had written mio, which is like you know, inventing electrons it's like the very base of the async stack, but there was you know we were not at all ready to go, build out a proxy set of features. Yet the ecosystem was really young, but we had finagle here, which I have five years of production experience with already okay, we can.

F

We can build a proxy with that and get it start to talk about what this thing is and what it should do and learn lessons there and that's really what I've been doing. So we started to think about this new proxy, and what's that going to be it's gonna be link or D 2.0? Are we just gonna reimplementation.

F

Ready system and the answer's? No, that's, that's really not what we want to do, and in fact thinking about this you know. Linker D itself is mostly a proxy and we have this control, plane, type thing called name or D, but really it's a build-your-own control.

F

Plane thing you know if you want to have, will plug into a zookeeper a net CD or any of the things you want to do, but you're gonna have to go figure out what all that is on top and that's like in wall knives, and so we've started thinking about kind of inverting. This, instead of thinking about the proxy, is the main thing we want to focus on. We want to think about the control plane, the the lot, the controller, the policy of the logic and the proxy is kind of an implementation detail of that.

F

One of the other things we've learned supporting you in linker D is that all of your observability stacks are entirely different. Some people from atheist some people in flux, DB and getting a uniform set of information out of all of your configurations and helping you to bug this things. Can be quite intensive on our side to even just understand, what's going on in your system, because we have so many different metrics formats and time series formats and graphing systems and all that, and so we started think well, how can we build a more supportable system?

F

It doesn't have all this configuration overhead and has this kind of right architecture of very secure mineral proxy. That's not pluggable! That's not the place where everyone goes and puts code and has to work in about that proxy to be basically boring and tightly coupled to the control plane, which should be the fun part where everyone can contribute and write, plugins and participate there, and so that is what conduit is that that's kind of what led us to conduit.

F

Just. Can everyone see this but I bring this here? So the idea with conduit and I'm not gonna, try to draw the logo, because it's very squiggly is effectively.

F

You know we have two namespaces, so we'll have a conduit namespace, and this is if it's I guess I didn't say this in this talk, but conduit is part of this is community specific. So we have that very wide configuration surface area in linker D, and we want to really narrow that down. So we can leverage the platform to provide us with a lot of the primitives. We need primitives like watching api's and CR DS, where we can store objects, primitives, like our back and offing, authentication and auditing.

F

The Cooper Nettie's platform actually gives us a lot to build into and so we're not building over kubernetes. This is in a layer over criminais. This is something we're building into kubernetes, and so we do this by installing a conduit namespace, and so, if you, if you go through the demo you'll, do a conduit install what that'll do is generate a manifests, you'll upload that in kubernetes and that'll give you this namespace and that's basically, there's a few services in here.

F

So because we're micro service people we tend to micro services, that's just how we do things and there's basically, this markers can I irritate me. So.

F

We have a proxy API and this is something. Hopefully you don't have to think about all that much. We also have a I, don't know what I will call a public API. These names are still a little bit in flux and we have kind of on top of that web and CLI so have a command-line.

C

F

You can get stats from tap commands. We also have a really nice web interface. It's not meant to replace Prometheus or anything like that, but it is meant to help you understand the health of the service mesh. One of the other things we really wanted to do is make the service mission debuggable I.

F

We now kind of know how to debug these things, but I want to help. You understand the health of the service, much itself, the health, the control plane itself, it shouldn't be a black box. This is something that we can help.

F

You understand is you're going and the web UI is especially geared towards that to kind of illustrating the concepts and the health of the system itself and giving you tools to bug that and inside this inside of this big space, we have services and we have only a very few services today we will be adding things here, and the other thing idea is that this is where your plugins go. So because this is like well, this is all gr PC. This is all gr PC.

F

We also have gr PC in here right and we have a telemetry service.

F

This actually uses Prometheus. So when you install conduit, you will get a Prometheus install that that's a couple conduit over time. I think we'll tune that to our use cases a little bit specifically, but there's actually some very nice things about not using a Jenner Prometheus closer. We can get a little bit more fine-grained in terms of how we configure things and even to collude down into the proxy and I'll scribe. How do we do that? A second, so we have a telemetry service.

F

We have what we call a destination service which is really just talks to the kubernetes api and does service and endpoints API lookups. Just like link ready des or anything else that does service Gabrielle incur D.

F

Do we have another service right now, yeah? That might be it right now, where we have. We want to have routing services and other things here as well, but you know we want to start with the smallest api's and ab them as we need and not try to guess at a bunch of api's before they're ready to be used and the weight, and so like you can install this. It's just running, there's no traffic, you know your apps can keep running it elsewhere and that's fine.

F

Then you may say: I want to and I want to put one of my services in the mesh, and this doesn't have to be an all-or-nothing thing and in fact, shouldn't be I. Think you know the whole point of micro service is that you don't have to have a lockstep move as an organization, and we certainly don't want to make you do that as you adopt a service mash that shouldn't be ever going on right now. What's what we want to give you?

F

The tools are to for individual teams to opt into this or individual service owners to opt into this, to get that value out themselves and as more people participate, that value magnifies and, as everyone wants to use the mesh, because they have better tools to are acting with each other. That becomes a virtuous cycle for you as a platform owner. You now have things that your users want to do this and it's not something that you're forcing them to do.

F

This becomes a carrot, not a stick and later, then you can lock down policies and do all these things, but you have to kind of what we've seen is people operationalizing service meshes. They come in and say we're in for security policy and that's how they start. Those projects frequently meet a lot of friction internally, and so the way that we've seen the service mesh get adopted internally is by starting with the visibility to things that the end uses as a service unit.

F

The service owners care about getting out of that system and if we, even if you can give those kind of carrots to the users of the system, then you'll be able, but you'll have the platform to go, put all the policies and control mechanisms and change. Many things that you want to build into it later.

F

Ok, and so then we may have I, don't know a bird service and a cage service. I, don't know, and you look and say I just want to inject into one, and this would be a well when I, say service I really mean deployment. We do use services to some degree, but things we care about our deployments or replicas controllers.

F

The deployment units of the thing is not the traffic targets of the things so service traffic target deployment is a thing you ship, and so what we may do is say: ok, I want to redeploy the cage service, but I'm gonna.

F

Add the conduit sidecar into that, and so we have this inject command and the conduit CLI, which will read in a yeah mol that you give it an output, a yeah mol that we've added two pods to the two pods are an init, pod or sorry nut pods containers we have an init container and what that does is configure IP tables, and so we say is anything coming into this pod. Anything going out of pod, it's going to the proxy first person, and so that's before anything gets running in the pod.

F

That thing is run is favorite iptables and so that HTV proxy magic, which Boris actually discovered, and it is a really nice hack but is a configuration thing that has to be added. We don't think you shouldn't have to do that or have to have HP libraries that support that, and so what we want to do- and this is going to take shape over the next release or to all traffic, can go through the proxy TCP traffic h1 traffic h2 traffic. If we know it's h1 h2, we can do smart things with it.

F

We can look at success rates and do latency where a load, balancing retries timeouts, all the stuff that we do in link reading today. But if you don't, if we don't know what protocol it is or it's already encrypted with TLS- and you don't want us to read it hey well, we can still forward that and still be a you know, a gatekeeper and still enact some policies even of oh.

F

This container can send things to this container, but not that container and so the goal- and this also helps with the zero configuration thing of you. Don't you no longer have to worry of oh configure this traffic to go through linker, D and configure this traffic so not go through link ready in the content world. Everything goes through that proxy because it's not the JVM.

F

We feel pretty confident in doing that and so having a native thing, we actually will be able to manage the performance in a much better way, and so in the next phase to do you, you will see what I call proxy transparency where everything in route goes through the proxy, and so we add the proxy and all these things and then so. This is a non mesh service.

F

It's just gonna send things to the service as it normally would using qu proxy include DNS or its own service discovery or linker D whatever and as it enters the pod. It's gonna get rerouted by IP tables into the proxy. The proxy then records telemetry on that it can look at, and so we saw that tap command. If anyone not see the demo I did on Wednesday.

F

If you didn't, you should watch the video when it's up, but what we can do is we can extract a lot of telemetry here, so success rates request volumes and what we do is we actually record much higher fidelity information than we record a link. We do we really record each request shape, and so we want to know not just a what we call link ready, the ID that identified name that like path form, which is very hierarchical thing.

F

Instead, we have this bag of attributes that reflects that request and we count metrics and latencies and success rates on each of those shapes of things and that's constantly being streamed up to the proxy api.

F

And so, and so we do some aggregation and as I was saying before, on the telemetry side, since we control the histogram appear, and we know how Prometheus is going to record histograms, we can actually couple the histogram here and so it's a completely lossless transformation in latency, histograms and under it, and you know payload size histograms, it's completely lossless into Prometheus, and so you can actually have a true p99 across the cluster, which is actually very hard to do in most observability systems with linker.

F

Do today, what you get is average of the p99 across the cluster or the greatest p99 across the cluster, which are maybe good approximations for you, but are still lost, and so what we want to do is be able to give you a more honest picture of the world there, and so, furthermore, we've added this tap command code. Maybe we have a tap service, so this tap command with the temporal command allows you to do. Is they basically craft a query?

F

I want to see requests that match these parameters that have a path that looked like this that go to this Authority or from this IP. Any sort of you know properties of the request. We can write a query for and that can be pushed down into the proxy and now I have a way to say, like just give me actual requests as they're coming through the system.

F

You can do this with logging, but logs are high volume and not immediate. This is a way to like really tap in and I think what we'd like to do later on is even add kind of additional features here. So you can do this kind of thing, with fault injection and use tap as a way to be a much richer tool for interacting with live traffic.

F

So that's like the spiel I had I'm happy to talk more. If there are questions or George has plans for me and yeah.

A

I think we're gonna go into an AMA panel, so Carl Brian come join us up front, and so this is your chance to ask questions either about hollow it or linker. Dean and I got to tell you. I have been in the booth we've gotten a lot of questions about both and so for this panel I am going to start with the two most popular questions that I have heard at this conference. So first question: what is the difference between sto and conduit.

F

Yeah, that's a good question, so what is different is tearing conduit. I I think feature rise over time. They'll be a lot of overlap because the server's MySpace has kind of a set of things that belong in it and you know really offloading the operational concerns of communications into the infrastructure, but I think the I expect the projects will grow quite differently. Google and buoyant are very different organizations. That might surprise some of you, but you know 20-person startup and a IBM and Google partnership.

F

We're gonna operate quite differently and our motivations are to solve concrete production problems in the near term, and so we do not want to add a lot of features in the conduit in the short term.

F

What I really want to focus on is getting that kind of essential service mesh features that are going to unlock, probably solve problems for you today, not the aspirational problems of when you re architect and get to like your company grows and you're really care about all SD tab stuff like there are very advanced problems and we're happy to work with you on those things. But that's not we're kind of it's gonna be focused in the short term.

F

Con it's gonna be focused in the short term of providing real, immediate production value, especially around the zero configuration secure by default. Everything visibility, I, think that is the essential proposition of the service mesh and everything else is gravy, and so sto is going to be doing that and more, but I think we'll be focused a little bit better and we're we. Our goal here is to get a lot of your input right. We I want to solve problems that you can.

F

You know by next month take value out of not some six months really cycle.

D

F

Know what sto is release I collections, but I know that I think we can be quite a bit more agile and quite a bit more focused on our use, people giving you feedback. So if you want that attention now is a great time to get involved in a product suite.

A

And that leads right to my second question, which is: what does the roadmap look like for a conduit other.

F

Very good question so I. The next thing top of mind is this proxy transparency thing that I were just discussing so being able to route all TCP traffic through link, ready or through conduit is the really the next goal, and and with that become comes TLS and things like that, and so the exact ordering of these features is still being fleshed out. But, yes, we will have h1 support in conduit in the very near future.

F

We love arbitrary, TCP support and conduit in the near future, and we will have TLS and some flavor in the very near future when I say in some flavor, this space is very frothy right now there projects like spiffy or kind of just giving the feet under them, Kerr Bearnaise itself is figuring out. The soft model Brian can speak to this better than I can.

G

What I would say is um you know if you look at what sto authors doing they've basically said this is how you're gonna do awesome occation between services right and the thing is, that's quite different than how you are currently doing it. Unless you are working at Google right now and I think we have some opportunity to help you integrate with your existing stuff and help you migrate over.

G

We can easily support the model that is, do authors doing, but I think that there's a model, that's gonna, be a lot easier for people to understand and to use to get started with, and I'm really interested in talking to you about, especially what your integrations requirements are for, TLS to make sure that we're meeting all those requirements. Instead of trying to tell you how you should do it practical.

F

And immediate production needs yeah, let's hold on do it.

A

A good time, I'm gonna ask our panelists to please repeat the question up before you get through.

F

That's a good question, okay, so the question is regarding telemetry: what's the standards approach there so I'm tracing we've not approached tracing yet I expect we'd like to integrate with open tracing specifically because it does seem like the standard, that's kind of forming there. However, because tracing requires application collusion or you have to have contexts in there, it's not where we want to start so we wanna sort of zero config, no code change, it just works, and so we have not introducing yet.

F

That certainly will come, though, on the raw metrics side, we are not using anything. That's standards focus in the proxy where we want this protocol and here between the proxies and the proxy API to be very coupled and basically an internal API detail. But within the controller here we use Prometheus and have a Prometheus export, so you should be able to integrate your own telemetry system around or Prometheus export or even I. Think eventually, we'll have other telemetry plugins there. If you want to offload it an influx DB or something else,.

C

E

C

One thing that it's of interest to me is there's a prominent place where high resolution metrics of.

C

F

Okay, the to repeat the the comment there you know: organ is a mattress cost money in a real world and there are resource constraints around the number of metrics that linker need generates, for instance, and we will already so I've been on call for observability systems before so. It's a very close problem to me. We we've already really scaled back the number of metrics that Luca T admits and it already, and it still emits like thousands of metrics per service you're talking to so that could be quite a bit and so what yeah?

F

What I would hope to do in the future? Is that to manage these kind of resource constraints? Is that we're pushing most of this into control plane? And then you have the ability from the control plane to filter and aggregate their into the shapes that you need for your real time. Your real production monitoring system from there well.

G

Plus I mean your needs are gonna change over time, because, if you're, having trouble with the service, you're gonna want to crank up the metrics that you see so that you can see exactly what's going on. But you don't want to be flooded with every service at that high level of detail all the time yeah.

F

And that's exactly stuff that should belong in our controller. I believe that that sort of controlled flexibility over those pipelines should be available to.

F

Not yet so right now it's very hb2 status code or g RPC status code or the connection failed, but right now all the success rate determination is done in the controller. I think eventually we'll have to push that down in the proxy to do retries and things like that, but that will become configurable over time that all could become configurable. Sorry, as we stabilized with the routing API look like because I think that'll be tightly coupled to the routing api's question yeah.

H

F

The great question: so how do conduit ingress really basically so?

F

First I'm at a point about ingress I, think the ingress API is in communities right now, I, don't know anyone who loves them, but they're a little bit insufficient to describe proper ingress. So we didn't want to jump on that as a first problem to solve, because it's a little bit hairy and so condor does not do ingress today, but it should just work right. If you have a Negresco ler once you get to deliver to the pod, Conda will take over once. You leave that pod condo.

F

It will be involved, and so that's up to you. There sorry spam calls and.

F

Overtime, though, I would very much like conduit to take over across responsibilities, whether that's using the kubernetes ingress objects or not remains we've seen. It could be very much the case that we end up owning new object, types that get stored in C, RDS and kubernetes, or something that represent a better ingress type question Beck.

F

Okay, so how does the our proxy come or how does conduit compared with envoy yeah? So envoy is a very generic thing that can be used to plug into many different api's and is using this year's use outside of sto at symbol in C++. Has you know I've been in production for quite a while?

F

We, as we thought about this, we you know I, know I, know mad and I'm familiar with the lift stack and a lot of people there, and so we looked at it a bit and we just we started to think that, like we really don't want to be writing C++ in 2018, and so, if we're building, if we start to think about what the system we want to build is like I do not want to sacrifice on correctness at all and I really don't want to be linking against NEC libraries that I don't like to link against, and you know after heartbleed.

F

In the last couple years we've had, and so the rust ecosystem that Brian and Carl have really been core contributors behind really inspired us and that we were able to hire some people who had the expertise there made that a little bit safer bet for us, so you can think of it, as very similar envoy will be more generic. Our proxy will be very much coupled to our controller and it's not meant to be a generic tool.

F

So you know very similar feature sets as the our proxy gets implemented, but our proxy will be much narrower and just a detail of conduit, not something you can run for any set of use. Cases and part of that is again envoy- has a very rich configuration surface area. Our proxy has no configuration service area, you don't configure it. You configure the controller and the proxy is just configured by the controller yeah.

I

I'd also say a big difference is on voice, mostly a glue of a bunch of off-the-shelf libraries, a number of those somewhat older and also not designed to be originally built to be used in a proxy situation.

I

We're like our conduits, really taking verdict like tightly vertical integration strategy. So we have control over well, basically, everything down to the syscalls like we have our only h2 implementation has been like designed from the get-go to be used in this kind of situation to our PC as well, so I yeah I mean we're still Oh dot one, but I would be shocked if we don't beat on point. Basically, all friends, yeah.

G

I think one thing I want to add to that is like Karl and also Sean, who also works on rust, HTTP stuff, in particular, and myself, I work on Russ, TLS, stuff and cryptography stuff. We, but besides what we do here at buoyant on conduit itself, we're also, basically, the primary people responsible for these various aspects of the rusts libraries that lots of projects are using right, and so we make. We can make sure that these projects get maintained over time in a way that is perfect for conduit and so like, for example, with TLS.

G

We the way that TLS will get configured in rtls implementation is going to be optimized for the dynamic needs of this proxy, like being able to swap out hot swap the certificates that are, you currently being used, and things like that, which is really painful in nginx and some other products.

G

We are doing that from the beginning to work and also to work from a central controller which a lot of that central controller logic is being bolted on to other projects, to get it to work and.

F

If we had to move that quickly and C or C++ we'd be running the wall of knives, so yeah.

A

F

Yeah, it's it's not possible today that I'll have I, think that'll probably be in like a Oda three releases. My guess we're gonna focus on transparency first, which I think is an important precursor to getting ingress right now, conduit only speaks HPV to and the subset the G RPC cares about.

F

I really want to focus on that stack one, because it's harder to get right, h2 with multiplexing and flow control and streaming is a harder superset of HTTP, and so now that we've solved that hard problem, it's gonna be easy to back ourselves under there, and so once we've got that transport nailed down ingress will I expect come shortly thereafter from a dead.

D

F

Right and so the question is, are we news, IP tables for.

F

Mucking with IP tables seems dirty many things such IP tables. How do I reconcile all that? Do you do I'll.

G

Take that so I mean I personally would like to find a better solution to the IP tables thing, but actually it's more of aesthetics than anything, because it actually works really well and as far as your coming about the the privileged container being necessary to configure the IP tables, that's true, but that container is tiny. All it does is run a couple, IP tables commands and then exits, and then we have an unprivileged container that is doing all of actual proxy.

G

So there's not like a huge security risk there and I think we should verify that we work with other tools that are manipulating IP tables too, but when we did the initial analysis we were, we seem like everything's going to work just fine, and this is an approach that people have been doing actually for a while right, like even link ideas, doing this people.

F

Have done this integrity, yeah and the yeah? The final thing there is I expect the kubernetes ecosystem provide better facilities for this in the future, and so this is really just the best we can do today as a better tools will of course adopt them. So things like initializers and stuff, we hope will alleviate some of this.

G

I mean kubernetes is just iptables rules all over yeah.

F

G

Have a question for the audience which is like how many of you are already using kubernetes.

F

Well, this is the place for that and of.

G

The people they didn't raise their hands, how many people are planning to migrate to kubernetes and how many people think that you know in the next 12 months or more. You won't be using kubernetes at all.

G

F

Any more questions don't make me dance. Okay,.

G

I'll ask you all another question: yeah anybody who is particularly interested in making sure that TLS support and the product supports what you need to do. I would love to talk to you and work with you to set up some proof of concept, showing you how it works and making sure that we're not overlooking any of the finer points of what you're trying to do.

A

Well, thanks a run so last call any other questions.

H

F

Hado is not ready for reduction right now way too realistic, well, we'd like to hit 1.0 and q1 or q2 what q1 or q2 next year, but we leaps where.

G

We're supposed to same thing, QT is the only product yeah.

F

Linker D is the only production at this. As Brian was cueing me Liberty is the only production ready serve as much today and we are aggressive about getting conduit there, but it was released on Tuesday and I. Need you to break it a little bit before it's ready.

I

F

Why that's why.

I

Brian's here yeah.

F

A

Cool well, thank you very much for your time. Thank.

F

A

You very much I, don't review users, and so thank you all for coming out. If you have questions, you can always find us on the link or D slack link or DDOT slack I. Oh there's also a conduit channel in there now and you can try Kandra really easily go to conduit, thought I, oh and there's an entire walk through. So we'll hang out for bid in case there are any other questions, but.

C

Otherwise, we'll wrap it up. Thank.

A