Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2017 (Austin), 15 Dec 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes Ingress Controller with Apache Traffic Server [I] - Mrunmayi Dhume, Oath (Yahoo)

Description

Kubernetes Ingress Controller with Apache Traffic Server [I] - Mrunmayi Dhume, Oath (Yahoo) & Suresh Visvanathan, Yahoo!

Today, the Oath Media Brands and Products container platform is serving critical application workloads like Yahoo Sports and Yahoo Finance at a large scale using Kubernetes as the orchestration framework.

For a platform at this scale, it is critical to have a powerful and flexible ingress routing layer (controller) that is able to handle the dynamic behavior of container based applications, such as auto-scaling, frequently changing pod IP addresses, self-serve onboarding and cluster-aware routing. This L7 routing layer must be quick to react to changes on the cluster without affecting its routing capabilities and impacting the in-flight requests. In a multi-tenant system it is even more vital that a single application deployment does not cause an impact to user traffic or hinder the release velocity of other tenants.

We developed an ingress controller based on Apache Traffic Server that satisfies the requirements stated above, while remaining scalable and easy to integrate with both Kubernetes and the Oath ecosystem. In this talk/presentation, we will elaborate on the architecture of the ingress controller, the performance metrics we’ve achieved, and the key learnings from supporting such a critical infrastructure component.

About Mrunmayi Dhume
Mrunmayi Dhume is a Senior Software Engineer in the Core Infrastructure team at Oath Media Brands and Products. She was involved early on in the introduction of Kubernetes in the organization and took a leadership role in designing and implementing the ingress routing layer components and container pipelines for core platforms used at Oath. Prior to that she has worked in various other projects focused in the areas of developer productivity tools, CI/CD processes and chaos engineering. Before joining Oath she completed her Masters in Computer Science from Stony Brook University, New York.

About Suresh Visvanathan
Suresh Visvanathan, Sr Architect, has over 13 years of experience in IT and Software. Suresh’s current responsibilities include the architecture, vision, strategy and design of cloud platform as-a-service (PaaS). Suresh has been architecting solutions and building products around resilience engineering, Auto recovery Systems and is a frequent speaker at K8s meetup, Chaos Community Events.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Let's get started good afternoon: everyone. My name is Suresh senior architect from Earth hi.

B

I'm Rene I'm, a senior software engineer, I thought so.

A

Oh, what's what and you know what's what good, what is a company which was created when Verizon but yahoo and merged with AOL and TechCrunch? So we are from Earth I'm an ex ahuna.

A

So today we are going to talk about ingress under the coconut ingress controller using Apache traffic server. The team where we belong to, we are part of core infrastructure team which is powering all of media products such as sports finance and news. So we manage the complete kubernetes cluster for this product and we manage and operate the kubernetes cluster for them.

A

Not only we manage kubernetes clusters, we do provide other tools which increases the developer productivity and reliability tools which help improve the reliability and I will rotate of all these media suits so agenda today, we'll go over kubernetes at both the overview and architecture we have, and it will do little deep dive into the ingress in design and also will be a demo and we'll go through the salient feature which we built part of it and there also like how is our kubernetes set up there.

A

Okay, coconut is I thought how, if you look at here, we have 12 kubernetes clusters across six data centers. We then hundred-plus application. We at peak do 150k RPS the ingress. This time is we going to talk about today. That particular layer does one atak RPS peak.

A

Other important thing is when I say 12 cluster 6 datacenters. You know in number of parts within more than 10,000 parts. Ok, each of the parts may have three to four containers at 40k containers we run and the other important part over here is like we manage the cluster for the entire media, but we provide them an independent pipe and they can deploy their code whenever they want to as a product developers for sports finance.

A

But just we give you the tools to defy to this kubernetes cluster, which we manage so I want to go over the architecture which we have. If this is the kubernetes architecture which we have in-house, we are on curve and we are not in any of the cloud world. We deploy everything in-house.

A

So if you look at your left hand side now, all the boxes which are marked yellow, that's all a component which we built in-house and those are which are green, our existing components within both which we reused, other things comes from kubernetes open source world. So, if you look at the left hand side, there are a custom tools. What we do with custom tools. Custom tool is something which we built into our screwdriver continuous delivery pipeline. So this is something which we built first, so that user can deploy their code.

A

Natively apply the cube CTL applied to the ml to different environment, using the tools which we built part of our screwdriver pipeline, and then there is a one which we built this as a web hook. If you look at at the bottom here, so this is a them hope for authorization of their workflow. So these two types we do one is for identity for the user, as well as for the service which is deployed in these communities.

A

So if someone is trying to deploy a code to this cluster or to that particular namespace, we'll make sure that that particular user or that particular pipeline has been authorized to deploy the code on that one. So this baby, who make sure that make sure that you are authorized to deploy. So if we look at the 18, this kind of open-source Athan is nothing. But if you look at AWS I am is similar to, that is an authorization role based authorization system.

A

So if you look at the right side, this custom, in guess it's called ATS Apache traffic server, how many of us using Apache topics over here in production, nice, nice, so a party traffic server. This is the ingress which is like any user who is accessing you from your mobile or from your web sports on Tiago comm. So it goes through this particular ingress layer before routing to the origin, which is which is running in kubernetes.

A

So the origins are bunch of parts is just dynamically changing parts so that it have I pee whenever there is a new part deep line. So this dynamically changing parts- and there are part, was down that are not get crashed and the new deployments are happening whenever these things happens. So we get every part. Every argent part which is serving the sports started. I would come.

A

Contents gets a new IP, so those IP we need to dynamically fit into the ingress ingress player so that it can route the request which is coming from user to that origin. Okay, so that is complete design which we built, which we'll go through in detail.

A

Okay, bunch of requirements right bunch of recommend which we can want. You event if we took when we want to design the single Slayer is dynamic updates. So when I was saying dynamic covers in the previous section, I was saying that there are so much deployments happens and there are parts are deployed and they get new IP right and in a day in a cluster, there are forty to forty to fifty thousand parts get created because containers get created and destroyed and new events generated.

A

So if you want to make sure that we capture those things and feed into ingress very dynamically, so in our version one of the design which we had you should make sure that whenever these things changes the origin IP, then you know genetic changes for any particular deployment. We used to feed into 80s, and then we used to do reload of the 80s process. What happens when we do lead notice? It impacts a request. The in place records are impacted and also it increased the long tail latency of direct question.

A

So what we did was a mitigation was like instead of doing every second, every milliseconds when it's occurred, we patched into five seconds, so we did five seconds it loaded. So every face I, guess the observed. If you look at our chart of the ingress chart, it's like spikes up and drops like soft tops. So what would happening was we are trying to load the dynamically changed IP the arjun IP to a 80s, taking effect by reloading the process and then say? Oh, this is not scalable. We are impacting the end user.

A

Then we take kind of try to patch it for every 30 seconds, so yeah, every 30 seconds is skurd. They kind of you know, fix the issue. In fact, the mood they shoot say that it's not the way five seconds every 30 seconds. Okay, it's not a great solution, but we had every 30 seconds we see. That's like. Can you suspect, and also whenever there is a new pod comes into picture? Kubernetes deploys, of course, in seconds for us to route their records to that particular part. We have to wait for this reload to happen.

A

Okay, we need a dynamic nature, much more flexible than the version one we had and also at the same time, how you also want to make sure that we have so many custom, plugins, Apache crappy server, because Apache traffic service open sourced from Yahoo under apache license. So we have so much plug-in which are powering the media sites, such as ESI ssl plugins, which we build, which is kind of powering the media site.

A

We want to make sure we support those plugins part of the migration from older stack to the kubernetes world, with this 80s layer, this as an ingress layer, but at the same time we also had okay, now that we are moving to kubernetes, we want to make sure that we are natively used. All the kubernetes ingress specification, the english spec, which we, which has been used by everyone else like nginx or any other english controller. We want to make sure that we used that ingress pair specification as an important requirement we had was like hey.

A

We are not any public cloud, we are in-house right. We want a seamless experience for the user. Someone wants to deploy a application to this particular cluster. We just they come and then define bunch of yama and then define the URL and the host and a name for their application that they just deploy it. Everything is taken care for them into end. Ok, so they're not dealing with many different operation team to get their application deployed. This is in-house. It's not public clouds.

A

If you want to give that kind of experience, these are a bunch of procurement which we had before we design the solution which we going to portray now so I said why Apache traffic, so ok is fast and scalable and they are so much investment from Yahoo into eight years. We have so much plug-in built into this. We want to make sure that we leverage that plugins the same time. It's easy and extensible right, and we in case we want to build in a new load. Balancer algorithm to round-robin are their least connection.

A

Our waiter Lee's connection. We are able to quickly develop a plugin and drop it then it's very easy to modify the hook of every hit HTTP transaction hook in ATS. So it's very easy and extensible, and also we can customize a lot for our requirements. So we have lots of plugin which modifies the record shader before forwarding that to the origins. So with that why we picked ATS, thus main thing we'll go to the kubernetes ingress controller which we build and how its operates with the demo. Okay, thank.

B

You Suresh so today, I'm gonna go over the design of the kubernetes ingress controller with apache traffic server I go over the various components that make up the ingress controller. I'll talk about some of the key features that we've built into it and then I'll go into a quick demo which will show you how an application can actually use this.

B

So this shows the overall design of our ingress controller. The flow goes from the top to the bottom, so I aggress controller reacts to changes on the cluster. It gets it fetches information from the cluster, it parses and processes it through the various components and it finally provides routes to ATS. So it may with this process, it knows how to keep all the routes to the pods up-to-date at all times. So now, I'll go over each of the components in a little more detail.

B

So our first component is the service watcher. So this is responsible for setting a watch on the ingress and the endpoints resources of the of the API server. So what this does is whenever there's a creation, updation or deletion event, it gets notified. It fetches information from the cluster for all the changes that took place and it writes it into a into a JSON file format, and this is so now, for example, if a new application gets deployed, it'll create like a new endpoint and a new ingress.

B

So all that information is gathered by this particular component, an X component is the health monitor. So this is responsible for doing a periodic health check of all the pod IPS that the service watcher obtained. So it only and only those IP is that returned a 200 response for the health check. Url are written to the second set of JSON files, so this is very important for us. So currently, our ingress controller actually is on a set of bare metal horse which is outside of the kubernetes cluster.

B

So we want to prevent an issue of a network split brain, so it's possible that in some cases the pods actually healthy on the cluster, but they are not reachable from our Apache traffic server. So this component helps us to detect such a situation and quickly route away from those pods.

B

An x-component is the compiler, so this picks up all the JSON changes that have taken place and it compiles it into a binary format that can be consumed by a patch a traffic server. So this sets like a watch on the JSON file directory, so it's updated whenever there is a change in that directory and it only compiles that particular change into a bind into a corresponding binary for a file.

B

Finally, we have the Apache traffic server and the traffic manager plugin. So this traffic manager plugin sets a watch on the binary file directory. So again, it's notified whenever there's a change in any of the binary files and it does an in-memory map of that particular binary file into the Apache traffic server memory map. So, with this approach, we're actually only going to do a memory map of the change that took place. It's not going to replace the entire memory map, so this helps us to prevent the whole hot reload situation which suresh mentioned before.

B

So we don't see any impact to in-flight requests and we also see that the pod routes get updated, much faster, like in a matter of seconds. So this is the overall flow for ingress controller.

B

So how would an application actually go about and use this ingress controller, so like suresh mentioned throughout? Our key principle, has been that we want to use the native kubernetes resources as far as possible. So even when it came to our ingress controller, we decided to use the native kubernetes ingress resource. All we require is that the user specify certain custom annotations that we, that is understood by our ingress controller. So you can see those annotations highlighted on this slide. So, for example, we need them to specify ports.

B

The reason that we went ahead with annotations is because we need to specify certain non-standard ports. For example, we may want to specify like 80, 80 or 90 99, because these are ports that we need in our internal applications. So we decided to specify through annotations. Another thing that we need users to specify is the default domain, so this is basically the domain that they want application to be routable on.

B

We also have an use case internally that we want to specify multiple aliases for the same particular same back-end, so we could have gone the route of using the horse, but then that would mean like duplicating the same information for the same back-end, so we decided to instead go with annotations, where we can specify it in a less complex manner. So all users need to do is do a queue.

B

Cuttle apply for this kind of an ingress resource into their cluster, and the ingress controller will automatically pick this up and find a route to that board. So now, I'll cover some of the key features. Apart from the, of course, the dynamic updates that we mentioned on the act of health checking. We also have some three key features that we provide as a part of IRS controller. So the first feature is the cluster level fail out.

B

So, in a scenario where so me like we mentioned, we have clusters across multiple regions and it could be that in a particular region there may be a network issue or we may require to take out a particular cluster for some kind of maintenance. So we wanted, you know seamlessly dry, divert the traffic which is coming to these clusters for all the applications on that particular cluster away from that classroom. So we decided again to utilize config maps, which are natively provided by kubernetes.

B

We updated our service watcher component to also set a watch on the config map. Now what cluster admins can do is they can go ahead and create such a config map on the cluster? This will notify the service watcher that it needs to set up set a it. Will it'll actually inform a patch of traffic server to serve a 404 response for the health check URL for this particular cluster and o'the?

B

We actually have a DNS level routing which fronts our ingress controller, so once it starts receiving this 404 response from our cluster, we start diverting traffic seamlessly away from this particular cluster. So in this way we won't really impact any application, but the traffic will still get diverted and then we'll be able to do whatever maintenance. We want on our cluster so similar to this ability for at a cluster level.

B

We wanted to provide it also at the application level, so applications may have the need that they want to suddenly maybe fill out all the traffic from a particular region. So we also provide the ability to specify application level config Maps if the applications provide such a config map. Again, it's notification gets this information. It sends that information to the traffic server through then entire controller flow and that concerned about your traffic service start serving a 404 response for the health check for this particular application.

B

So all the other applications on the cluster remain untouched and unaffected and they'll get the traffic as was before, but all the traffic for that particular application will get diverted to the other regions that it's serving from now. The last one is to prevent a usurping of the same domain by multiple teams, so as a bi as a side effect of the above, the ease with which a application can onboard to our environment. We see that it's possible. It could be that the you know, multiple applications will try to.

B

You know, claim the same domain and let go a little more deep into how this could happen so consider a situation where you have like a generic app one which gets which deploys this particular ingress resource and it tries to- and it's already claimed, an alias, which is cube, con dot, media or yahoo.com.

B

So it's actually serving on this particular alias now, since applications can go and update the routes as they wish, another generic app say a generic app to could come about and they could try try to apply an ingress which also uses the same, alias, which is cube. Conrad made out of Yahoo comm now, since they both can just apply the ingress resource and it would- and if this ingress resource got created, it will actually lead to a non-deterministic behavior on the Apache traffic server.

B

The way up are check traffic server works is it really depends on the order in which it picks up the rules, so as since we have multiple hosts, multiple hosts could pick up the rules in a different order and they could actually potentially route to both applications. At the same time. So consider you go to Yahoo Sports and then suddenly you see the Yahoo Finance page, so that would really not be a situation that we want to occur, so we actually came up with a solution to counter this problem.

B

So this is our ingress admission. Controller kubernetes provides a dynamic admission control a web hook, so a dynamic admission control feature. So what that does is that before a resource actually gets created on the cluster, you can perform certain validations of your choice. All you need to do is when you're configuring this control, you need to specify what is the service that is going to handle the validation for you.

B

So what we did is we created a dynamic admission, control for ingress resources and we provided an ingress claims service which will handle the validation for it, for example. Now so if a user goes ahead or user Odyssey ICD pipeline and tries to create or update an ingress resource before that, up before that resource gets created, the admission control comes into play, it'll actually forward this resource to the ingress claim service, which will actually perform some kind of validation on it. So I aggress claim service at all times.

B

It maintains a mapping of the domains that are currently present on the ingress class on the kubernetes cluster and corresponding to the ingresses which are actually claiming them. So if it gets a request for a particular domain, the first thing it will do is it will check if that domain is present in the map that it has.

B

If it is finds that the domain already exists, and it's it's claimed by the same application by a by a different application, then it will actually feel the validation it will send back a fail to the dynamic admission control which will prevent that resource from being created. On the other hand, if that domain is not found in its map, which means that no other application has claimed it, it will actually say a true to the admission control which will then notify will then go ahead and create that resource.

B

So this is how we prevent two applications from claiming the same resource. So this sums are pretty much how our ingress controller works and how you can use it, and also the key features that we are offering with our ingress controller. Now I'll go into a quick demo which will show you how that a user, an application, can go ahead and use this.

B

B

So this is our be right back page, so I want to actually access.

B

This is cube. Conrad media at yahoo.com, slash, cube counter HTML, as you can see your right now, seeing a be right back page. So this is the static page that we show whenever you don't have a route to a particular application. So I have this yeah Mel file.

B

As you can see, the ingress specification is on top here and I'm, specifying the custom annotations that are required by our ingress controller, so I'm specifying the alias, which is cube, condor, mediate or yahoo.com. I'm also specifying the ports that I want this particular application to be routable on for this egg's for the use of this demo I'm using a simple nginx container. So there's also a deployment URL which have been specified along with the ingress resource. So you can go ahead and you can just apply this like we usually do with a cube.

B

Cut' will apply so and you'll go ahead and create the resources on the cluster.

B

Let's just see if our pod got created, so it's still in continue creating stage. So, let's just do a watch on it and let's wait until the board actually becomes ready to serve.

B

B

All right, the pod is up and running. Let's just do that. Just make sure that the ingress resource also got created.

B

All right, the ingress resource is also there. So now, when I go back- and I refresh this page- we will see that the nginx board is ready to serve so, as you can see in a matter of seconds after the application gets deployed on the cluster with the engine with the ingress resource, the ingress controller is picks it up and it makes it it UK it's ready to solve. So that's it for my demo, I'll pass it to Suresh for concluding remarks. Thank.

A

A

Okay, Apache traffic service very great, and if we used for one particular use case which is http-based- and there are some limitations with the Apache topic server, which is there's no TCP support and we have use cases which, when we want to have support DCP but like red days or any other protocols, so we want to support those who are close. We are looking for other alternative on those use cases like JP, c % tcp, so another possible solution which we are looking at. This point does like ipbs, r and y.

A

So in next give con, we will try to share our experience with those things when we support both HTTP as well as TCP workloads.

A

Okay, umm open source, like I just spoke a bunch of confidence out there, which is which we built. It is on the yellow section right when is a web hook and Athens, and how we build this dynamic admission controls all those things are available for community to use. So, if you go to this repo here- and you can find all the code out here and then we are in process of open sourcing, the controller which we just built for English claim service.

A

Ok, how many important things if you look at this one like we just specified- talked about hey kubernetes, ingress controller. So in between what we have discussed, how are we managing this cluster? We have our own network model. Okay, we have our own template templating. We have customized that right.

A

This particular link will be I presented like couple of months ago in barometer, to explains the completely how we, however, clusters are deployed, how we manage the cluster, how we serve a network model I would like you guys to take a look at that and get the full form of how he managed in one form. How many important thing is a they are hiring. Okay in case you want to work on this awesome kubernetes platform on Poe. Please contact me, my name is Rashmi. I can see my contacts here.

A

Thank you. I would like to thank you. The entire team who worked on this from my team, as well as other team within auto, which helped us pocket and screwdriver on those.

A

I'm I'm happy to take.

C

In front of kubernetes, do you ever in the oath use case, go beyond just the HTTP requests, load, balancing and use any of the caching or other components? Just as a use case example, I know it can, but are you guys doing beyond beyond the HTTP request load balancing? Are you also doing caching, as well in the earth case in front of communities.

A

So very good question I'll answer that this one, this particular routing layer which we versified the start, the routing and we have caching layer the application that we have like sports, like a node.js application right that has its own cache and we have not enabled caching at the Atheist layer, but we have a leukocyte cache which is running in red days, which is powering those websites. Ok, this particular layer. We just left it to just two routing, ok, but we have other caching layer which is powering your websites, which is Redis.

A

Never a disk caching which powers these websites, so not in this particular layer, hope that answers.

A

Their questions.

A

hmm What are the challenges.

A

So when I first played dryness like hey, we have absorbed because it's in everything it's on from CD habilis many switches and racks which are distributed across right. So what happens is like the the part which is running on the kubernetes cluster was able to communicate to the API. So the API says that okay, this part is healthy and it feeds to the sovereign interest layer but ingress layer to the part that communication was broken because one of our ACL push.

A

We do lots of echo push in the network layer or the particular switch on that was not routing properly the switch between our ingress layer to that particular part, very or suddenly that was broken. So we have to have this active check so that we avoid sending that, of course, which will not get any response. So that pushed us to render.

D

Yes, the annotations you show are a little too generic and may conflict with other ingress controllers and so forth. How hard would it be to properly namespace? That is that edit, the code would make changes and recompile, or do you have some control over that Nick.

A

D

So you have three annotations: you show to control your your ingress, ok path. There.

A

Port and then there domain name yeah.

D

Those are very generic and can actually conflict with other things or may conflict with other, if someone's written, something specific and just took a shortcut also properly named space in them. You'll put it. You know, portion of a name or some attribute on there prior to those. How hard would that to be integrate or is? Do you have some control over doing that today? So.

A

Right now, if you look at this particular ingress layer, ok, it's it's like a shared ingress layer across all the namespace.

D

Right but that's part of the problems, what I'm saying is you may conflict with other products that may pick up that annotation cause problems? So that's why I'm asking how hard would it be to change those annotations? It's.

A

Easy we just need to change the spec, and then they also. We are also looking for some critical cases where the ingress could be specific to that particular namespace. Ok, we also looking to do that design, saying that hey if you're, very critical workload you want to have your own thing, then you can only hope your ingress to this particular namespace and you fetch it now, something which we are evaluating, but most of our use cases are really similar.

A

Al kind of workload so far as having one in ingress layer supporting all of our code made sense because more of it, mostly this, if you look at this valley serving HTTP at this point of time,.

E

You'd talked about building TCP support or support for TCP ingress, as that's actually like a limitation of the ingress resource itself. Right now, in terms of only being able to support HTTP and HTTPS. So is that something that you're planning to develop support for the ingress resource for TCP resources, or that you're going to kind of develop your own implementation?

E

A

This is on the inca spec itself, you're talking about yeah.

E

I think respect spec itself only supports HTTP and HTTPS, so to do TCP or UDP or other layer for services. Is that something that you're that you're thinking about developing an up streaming or I'm just interested in what your implementation might be?.

A

Yeah you're not taught through that I, not know that the Inga spec was not supporting TCP. That's because I know the service. Lady can define the protocol and say that this is TCP. Okay, I know that English was. You cannot define TCP. We look into that. If so, that is something which we can contribute back to the open source world. We can contribute back.

A

Thank you guys, thanks for coming.