Cloud Native Computing Foundation EnvoyCon 2018 (Seattle), 16 Jan 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Use Envoy + Knative to auto scale Java RPC micro service - Andy Shi, Alibaba

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Use Envoy + Knative to auto scale Java RPC micro service - Andy Shi, Alibaba

Envoy and Istio have greatly enhanced Observability of micro-services. But are we stopping at observing and reporting? How about reacting based on the metrics collected?

Kubernets and other orchestration systems have already offered auto-scaling capability. Is there a way to incorporate the matrics into the mechanism?

Dubbo RPC is an Open Source Java RPC framework. Now that Envoy has supported Dubbo protocol, the metric generated by Dubbo can be fed into the Prometheus. With that, we can gain many insights that outside systems like Kubernets simply cannot access.

In this talk, we'll talk about how we integrated Dubbo protocol with Envoy and feed the metric into Prometheus. We will show a live demo of a feedback loop where Prometheus triggers an alert to a Knative Serverless function that will auto-scale the pods.

A

We're only 30 minutes away from the break, so don't worry and good morning everyone, my name is Andy Shea I'm from Alibaba, Group and I can testify. That envoy is a fantastic project. We have this in production and for those people who are wondering okay, probably you're, wondering about two things. One is: why are we having a Java and RPC topic on the infrastructure conference?

A

That's because this is inspired by a new word use case. Customer is always right. The second part that you might be wondering is on the originals topic. It says, Kay native and now it's rolling back to kubernetes.

A

The reason behind that was Kay native right now only supports HTTP and not RPC. So there we go a little bit introduction. Apache double is a popular java RPC framework in china. It was also open source muhuali Bava over ten years ago, and it's still going strong. This survey was done like two weeks ago and we are still on the third place and we have done some work. Integrating double was a void and it has been up streamed, but that was yesterday's work.

A

We are trying to renovate double by introducing some cool features, mainly to making it more efficient, so and also integrate better with some way. So one of the things one of the things- that's that's gaining momentum over the over the last couple years- is cut our Jakob screams. Our reactive screams is asynchronous non-blocking, meaning that it's not gonna tie up your CPU resources, but one thing: that's unique about reactive streams is that the subscriber can decide when and how much the publisher can send the data.

A

So that's really important, as as an Emer splat for e-commerce platform. Alibaba has a lot of pressure. One of the big biggest challenges is that we sometimes see servers or subscribers hanging there, but they're, not that so the socket is still going. It's your life, the publisher or the client is gonna, keep sending traffic, and these these requests will go nowhere. It's really hard to debug, it's really hard to recover with reactive screams. That's gonna be solved, so that's the beauty of reactive streams and the reactive stream framework we choose is called our socket.

A

R stands for reactive. Our socket is a popular new reactive stream implementation. It has certain good certain advantages. One of them is, it defines four types of communication models. There's that classic request response model, which is what we are all familiar with. There is streaming model where you send a sequence of events. There is a fire-and-forget model which is in case it's like you're, sending data to the logging or you exposing your magics. You don't need a response and there's a channel. Channel means that you have like switch the rows between publisher and subscriber.

A

So it's bi-directional.

A

So what is the reason why we we we are thinking reactives? Our stream is so special that it raises challenges to to our auto scaling mechanisms. So, let's take a look at our real-life example, so this is not reactive screaming, but it's reactive system. When we go to the DMV when we go to the DMV, the workers are gonna, call us when he or she is available right disregard of how long the line is waiting outside see.

A

Traditionally what we are doing in monitoring the subscriber are the working parts is that we we're measuring the resources consumptions right, the CPU, the memory, but that's not enough. That's not enough, because what we are facing is what we want to do is we want to know we want to scale based on how long the line is, and then we want to scale based on how long each person, our process time, is right.

A

So what we want in this case is that not only we want to have an CPU memory, but QPS and response time, so in our company each and every Enterprise Solution is required to have these four metrics exposed, and that's where we came in like that's, where the request came from, the other team was asking us to help was to identify how how do we measure the QPS and the response time.

A

The challenge is that English reactive streaming in the reactive stream implementation there's nobody keeping track because it's asynchronous, it's not locking, so you can't really know or you can predict or even measure when the packets are coming back and forth. You can't you shouldn't, because it's gonna hinder your performance.

A

So, based on that, we are thinking, oh wait. What happens? What happens when we want to scale well when we want to scale our parts right.

A

So this is our temporary solution right, it's based on invoice. We have built counters in envoy and base. Basically our socket provides same types. These frame types are based on what I just talked about the different traffic mod communication models. There's a request response right. There is this stream, so in the typical in the typical request and response model at some type floor, so we can build counters in my and then we're gonna use premises, irate function to detect a search or a spike in crisis.

A

The reason why we use premises is, oh boy, is naturally by default, natively supporting premises, so we don't have to reinvent the wheels and also premises does provide enough functions that we can utilize, especially in the case when we try to get the number. So irate is a great function, but it has shorter intervals. It's better to detect surges or spikes in your quests.

A

Does this solutions of the original requirements of getting QPS and response time? No, but we are juggling between performance and features. So these are the two things we we are trying to fight, but this does provide us with a mechanism that can better, monitor and and scale our our workloads and coming back to the original topic about a native. The reason why we were thinking K native might be a good candidate is because key natives scaling mechanism is based on QP QPS.

A

So how we got to do is provide response time, but then it didn't work out so too bad.

A

So here is the demos architecture which I know it's. It's pretty rough. Pardon me for the rough drawing. So we have a client. We have a server. The client is a web server. Basically listening to clients, customers, HTTP requests, it's gonna fire, the double our socket requests to the our socket server and we have envoy site cars in front of them.

A

These avoid set cars are using our customized images that I'm gonna show you the coding in a minute, but what they're gonna do is basically do the foundation and then Prometheus is going to scrape those data from envoi and HP a which is kubernetes. Horizontal pod autoscaler is going to get those magics and scale the our socket server.

A

That line is missing because I couldn't figure out a good way to say I'm, gonna scare, you but yeah, so HP a actually is gonna scale. On the other side,.

A

So here's sample code so for those we're not familiar with the code structures, its extensions and you can have adapters on the Envoy. Basically, the extensions is like any other networking if you're familiar with it, it's similar to any other networking processing architecture that you can plug in your own adapter. So we have this future written. So here is the definition of the counters based on the frame types. So one good thing about our socket is you can look cancer?

A

You can look at the frame types in the metadata or in the header, so you don't have to decode the whole payload that saves some time and then in the real code, once you notice that it is our socket data, it is also keyframe you're, going to increase the counter number and then put it to stats and avoid. Like I said it's gonna expose us, that's two premises so premise: it's gonna pick it up. So that's pretty pretty straightforward.

A

So this diagram is taken from some other projects. I'm gonna mention at last this part in the simple HTTP part. You should replace that with our our socket server, but on the right side issues what I just described the permissions is the is to grab the thing they're from oops too fast. Is this grab that they the magics from from m'boy, and we have a custom magic server? That's going to grab the data from premises right, but on the left side it shows the mechanism inside of kubernetes, which is important.

A

You may, even though, we're not at coop gun yet so communities has this controller saying and it has other scaling mechanism. So it's basically it has a default implementation of COI api's by default. You can just call ask a scaling without any customization based on your resource right, but since we are extending it to have premises adapter, this is customized matrix, API, extending the COI API and then it's implemented by the custom matrix API server. By the way, this permissive, customized API server is one of the couple adapters.

A

That's written already implemented already, so we don't have to media with and that's another reason why we picked for missus as the way to expose the matrix. So now, let's I love this picture.

A

A

A

Hopefully we can all see this.

A

So right now we have only two parts: one is clanked, which is the web server, as I mentioned, the other one is a service server which is going to be processing the requests from the client using our socket and.

A

A

I'm trying to see.

A

A

Somehow it's getting smaller than then before I guess it's has something to do with the resolutions.

A

Where do I see that text.

A

No, nothing changes. Okay,.

A

Wow I think changes. Why.

A

um Okay, rather, okay, now we're doing it. Thank you guys. Okay, now that sound okay. So if we do a curl, you're gonna see it Sinclair returns a name, but oh I forgot this.

A

um So this client has it since it's a web server, it's exposing that IP address, and this is the IP address. We are curling.

A

So now let's do a traffic search, so this is a nice tool called hey. We're gonna, send 3000, we're gonna, send 3,000 requests to this URL and we can take a look. Let's give this as well.

A

So we can take a look at this, but before that, let's take a look at the configurations that we've created. So this is a custom API server the configuration that I wrote, which is going to creamy premises so.

A

This is the matrix name type for counter which was done in envoy, and it's exposed to communities and we're gonna get it and we name it. Our saucing, outgoing sorry, I, keep saying outsourcing and the function we're going to use is irate.

A

And in the HPA definition, we're going to create HPA, which is I, guess ties to your replica set. So the the target is our socket service one and we are monitoring this pod on the matrix of outgoing I, have a target value of ten which is pretty low, but that's going to be surface for demo. We have numbers from 1 to 50 I. Think we wouldn't get that far.

A

So, let's take a look at hey and trying to hit.

A

And we can also.

A

Take a look at what our HPA says, so HPA is getting something I hope or not. Oh, no! It's going so it's creating a couple new parts, basically, because our HPA has detached detected and a huge number of search by the way this number is off. So whoever is writing this adapter.

A

This number is way off. If you look at permissive GUI, it's not this number at all and that's how I figured out how many that I need the 3000, but basically it's it has detected the number of requests that there's a search and then it's scaling.

A

So if we look at how many we have now.

A

We created two one more pod, okay, only one when I was doing this at home. It created four, more so I, don't know what happened, but at least it responded.

A

And then at least it responded to the surge, maybe it's because of the network. I, don't know and.

A

This part has not finished yet so definitely it's so I.

A

Don't think I have network.

A

A

A

All right so now that has finished, and let's take a look: okay, steel to going on so I didn't change the part one why it's gonna come off so by default. I! Think it's five minutes for the hpa2 to cool down like one when your traffic stops it's going to take a while for it to scale down, because it's it doesn't want to do that again. Making a minute out! You.

A

All right, so that's the demo and here are some credits. So leaping is my manager. He is an awesome engineer. He helped me with this hours of the future and the demo. So hopefully he sees this.

A

These are the projects I referred to when I was trying to figure out how HPA works, because originally I was thinking of K native. Obviously, these three are very good examples and I think that image I got was from one of them, so also good thanks to them. So that concludes this topic.

A

Do I have time for questions yep, so any questions comments. Welcome.

A

A

The solution to skills is a demo, so it's to raise awareness of the situation. When there's gratia streams, that's introduced, it does create new challenges, so the the matrix we trying to get is a production requirement. So these are two different things.

A

Any other questions.

A

Cool I guess we are going to oh, yes, there's a question. Thank you.

A

I'm sorry I cannot hear you.

A

What time frame type? Okay, because, like I said, that's a good question. The question was why doing use frame type because it's a frame type counter? It's not a frame type. The frame type countering, is trying to tell you what kind of frames gives, in our sake, implementation, you have it's a multiplex channel, which means that different frames, different types of frames going in the same socket. So if you gotta distinguish between this from that, and on top of that, you you.

A

The better way, of course, is to have a counter of like each request and then a response. But then it's going to take a lot more resources to implement so right now we're simply doing a counter instead of the foo are like controlling mechanism or monitoring mechanism of correcting all the other data.

A

A

Cool. Thank you.