Cloud Native Computing Foundation Serverless Practitioners Summit 2019 (San Diego), 24 Nov 2019

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Architecture Patterns To Enable Stateful Communication Over Stateless Serverless Apis - Wenbo Zhu

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Architecture Patterns To Enable Stateful Communication Over Stateless Serverless Apis - Wenbo Zhu, GoogleSDCC

In this talk, Wenbo will go through the issues of exposing stateful and long-lived APIs with serverless based end-points. To ease the development of client applications, one architecture pattern will be to layer stateful communication on top of stateless RPCs to have the benefits of both worlds, i.e. simplicity and real-time guarantees of stateful APIs for client applications, and scalability and reliability of stateless services for serverless based end-points. A detailed case study will be presented, such as building a real-time online chat application.

A

So I guess I'll get started, yeah, just a quick introduction myself. So.

A

Framix takes external facing traffic and then to our Google's own internal services. You know in the pasture, mostly those traffic is web traffic and they're mobile nowadays there's a lot of like a PR topic, for example, if you're learning VMs on the gtp VM, so we have a big table. Pops up, you know Spaniels kind of API, so the transfer also has involved in the past. It was they all HTTP. You know web nowadays to also have G RPC traffic, which is the sort of default transport. My experience with service is pretty limited.

A

I'm, not you know, I sort of get most of the concepts of service from the time I, actually working on the App Engine back then I. Remember it's more than ten years ago, I was doing a try to open source, Google's internal servers and really hard day and night, so we don't have to use JD. We end up still using JD. It turns out to be good, but yeah anyway. So so today, the topic I'm going to talk about is mostly just a concept.

A

Try to you know, start with a problem definition and then try to explore that concept. I think this is the important concept for civil is and the way I first heard of our service. Are you know all these functions? Lambdas are thinking, okay, good. They God have to solve this department in some way, so I'm excited to be here. The purpose for for me is mostly, you know, I'll talk to you guys, especially after this talk.

A

If you have any thoughts, if you agree or disagree with whatever whatever you know, I stated: please let me know- and you know, hopefully, we can come up with something it's either. You know it keep evolving ideas discussing or maybe build something together. I have build a lot of the routines, all related technologies in the past, not specific for simulus, but I. Think it's like I said it's a very relevant topic. So let's get started.

A

The overall topic is very simple: it's just about a stateful communication and, but still, let's make it even more complicated, even more sort of eminent right. So the server is functioning so I guess, for example, I've been talking about today. You know it's most. For the most part, you will assume it's a starless functions, or at least the way. It's actually still is it's. It just works a lot better. So this overall thing it's really just about how do we handle still of communication in today's economy?

A

Civilized environment I try to have a definition, sort of my definition of what are the sort of state of a kind server communication. It's actually is to me it's really about it, you're trying to do like a TCP socket, but you basically open a socket from the client to the server and sure you would do some message. You know framing, to send the messages instead of bytes between the corner silver.

A

But for the most part, the semantics is is small, as the TCP socket semantics and the typical example is, you know you have pata api's WebSockets in which ERP service streaming's those kind of things. Now the difference between this kind of steel for communication under the traditional, like a single across the response are pcs.

A

So, first you know the TCP semantics, so I still there. So it's a 5-4 base to food, your practice rate, which means other kind and a silver, can streaming dated to each other in peril and the delivery is in order right because the TCP socket is so. You can't assume that it's all the messages already delivered. You know five four order now the biggest difference here is: there's no causality sort of semantics request response. You know the response is generated because of the request right.

A

So this is a very strong semantics and the English streaming case is not necessary right. You may have a server push which is spontaneous messages from the server and you can also have one-way messages from either end. So in that case you know. The message across end of us is the message service and they may or may not correlate I'll talk about why this is important, so their kiss starter is, you know still for communication of this. Actually works works.

A

Okay, especially when there's actually some kind of causality between the data, the content and the date or the data is a server generate the second a sort of property for this to work, whereas you know the the data, basically there's a streaming, it's a streaming. It's like a you know, there's there's incremental delivery. Basically, there's not like idle, like there's a lot of time or I do intervals when the silver oil can't just do nothing all right, so the data kept coming and the last part is short-lived. You know I mean.

A

Obviously you know what what you know quantitatively, what I mean is it's up to define, but typical I guess 60 seconds could be. You know you may see this as sort of upper bar the the canonical case for this that follows this definition. Is you know what I would call media a transcoding? It doesn't have to be actual media like a video audio for us, but the concept is you upload something, and then you have a download the upload and download that may happen at the same time.

A

That's what we call food impacts and its Shore life that you know, there's a bond there's a time boundary when the other data gets uploaded and then the data gets downloaded right. Once you can't imagine is you're building a video transcoding. You don't want the server to wait until other data. You know because, as a receive that start and then to start to process and return to the to the client, let's see there's a real-time transcoding service. So this is a canonical case which I believe that is works.

A

Well, you know you can basically just like our pcs, so there are case studies, I, think you know. This kind of you know long lived streaming kind of communication it becomes automatic. I would argue, especially for surveillance right. So why is this car silver push kind of traffic in Google? We have this. You know, for example, called fire store. We have this concept called watch API. So this is for the kind to observe changes happened on the database.

A

Another example is, like you know, any kind of collaborative adding editing. You can add it like Google Docs or the chair to room. So all those things are, the one thing is for the second part is: there are typical long lines where there's no and until the client log off, basically the whole communication stays on so so yeah. So this is when actually the you know, this is what actually define what I would call state for by die streaming right. So this is when become you know this whole sort of paradigm.

A

It really shifted from the traditional sort of our PCs and if you look at the diagram here, it's very much like a socket when, in the opposite case you can park this. They can. You know you kept more than one proxies the proxy and the box. You know between a client and a server there, a lot of boxes. They can't be it, they can all operated transparent to each other. So that's how you get it all the scalability sort of the web skills gonna, be.

A

They know I wanted to sort of dive into a little bit into why this is a problematic right. Even you know contradictory to what you may think that TCP is actually unreliable right.

A

You know, although the TTP does so this is a sort of a loss recovery or that, but it sound reliable, which mean that is, it can actually can be broke and when it's broken, you actually don't know if the data has been delivered or not, because there's no such thing like end to ACK all those things and the second part is you know, favorite detecting, as you may know, it's impossible or very hard or internet.

A

Then the second part of challenge challenges is mostly on the on the input infrastructure side on the service, for especially for the long live sessions for the on the client study came. Imagine if the client, you know it's a mobile phone, you work from one from elevator and you switch the Wi-Fi. Try the MAC address, IP adjust all changed, but also the park SIA.

A

They all have problem would deal with that and the pass itself McGarry study on the server side that you have all this load balancing those security issues and to deal with the long live assertions, and you know somebody's if, in fact, we have to solve this problem and the our challenge is no different. I guess then, then, that the civility the non stimulus environment.

A

However, the stream is important to have to support, especially if we want to have the very efficiently like a very low light, latency and a very high efficiency, because what stream it really gives you, as this kind of sort of batching effect right and on the latencies side, that you can really save the yrt T. So I have some numbers. I assure you. You know you can't sort of give your idea what artillery means it depending on. If you, if you consider speed speed of light, is a factor at an HTTP level.

A

You'll really look like a tax, even higher, all right. So you can, you can calculate like you know, all the members are in this morning, some of talking about 10 milliseconds, so 10 milliseconds, so the typical latency between us, East, Coast and West Coast could be much higher. It can be up to a hundred milliseconds so having something extreme is very important for the from the application or from the end user point of view. The second one so there's also a fish efficiency effect, because you know we're streaming. You don't have to do all this.

A

We basically have this kind of cross layer badge in place, especially you know you don't have to do the authorization or off you can only do it once I mean that itself as security issues, but but generally there's you know, because you can skip a lot of sort of initialization cost. So there's a saving on CPU as well.

A

So so now we know that is you know some of the world, the power exactly the partners and why you know why streaming is why those custom still for communication, it's problematic, but at the same time it's used for so I want to just provide a sort of a higher level abstraction. You know what exactly time is.

A

You know I think a lot of you probably already located a web socket API I personally was also part of this original. You know standardization effort, I mean it's a very simple concept, but you open open a so-called session that can be a web channel can be anything against a URL, the instant message, a message and any concept. It's a very simple abstraction.

A

The only thing that's impressive from it, you know not obvious from the API- is that's also 5/4 in order and delivery there now. This is all works. Fine. Until suddenly the network sort of broke breaks right. Then you basically partner like how do I recover. How do I even know the message gets dead.

A

So the state of art, to my knowledge, is you basically sort of charge solve this problem limit limit to support streaming? We serve one of the two, several approaches. Why is you kind of just limited the sessions to to a short period, because we, if you know if the session is limited to like 60 seconds and for the most part you can, you can all all of those powers and you can just treat them as RPC.

A

The second way is basically trying not to do streaming at all you, instead of using out-of-band push so in Google. You know we have this sort of that's called firebase card messaging service right, so we can deliver a message. It's auto band and the basic, and then you basically say you know: kinda will only do RPC by the way and they want to get data. They were getting from a different channel, which is not where the server application or your function is actually running.

A

So, basically, there's another party introduced right so which is responsible for deliver massive delivering messages.

A

And so for the past few years, I've been working on and it's something different. It's different, and uh you know these are the two solutions, so we have a killer on the by the web. This doesn't really have a solution there. It doesn't have access solution that you can use, but it mostly publisher, a bunch of specs surveys and awesome discussions or AP is to sort of try to define this try to come up with the solution. Is you know, between a client and a server you have?

A

This sort of you have a logical searching, but I'll ended hood, where the hood you're, basically using our short and concurrent, are pieces, which is the HTTP based. So this all works fine. This solves a lot of problems like all the proxy apartments gallery. Scalability problems on the client side.

A

All this, your mobility province, but it doesn't really work for server- is because in this model the service endpoints is still a long-lived still for so the question is now: can we have both right so, instead of as dissimilar Solutions, you have can't actually see our permanent abide. Our searching, it's like you, know, I'm, holding a WebSocket. The end point something like that, but on the server side is just a bunch of stateless functions, so this is something I. You know to my knowledge. It doesn't exist, but I think there's value to build that.

A

So so the next part of this talk is I'm. Just gonna go through a Quakers or case study. Saying imagine you want to build a real-time chat, you know has to be real-time chat, which mean that is. If someone is typing you can see literally the the keyboard is moving stuff like that.

A

You know, I want to say how can we transform this, this kind of static force of a traditional model to a service model and what are the primitives we needed to support to make it work so so yeah, so this size just here like how the traditional model will work right. So you have a client, that's a mobile phone or brothers, and then they talk to a server. The server has the endpoint between the current server there's some kind of still for messaging service going on there. It can be WebSockets.

A

Can be G RPC or can be something at a higher level, but again the communication by nature is sustainable and then on the server side. You know you have all these sort of the backhander storage to managing the so-called. You know chatroom communication there's to store all the backend stuff. So in there you can use queue. You know all these database stores databases now the at the very, very high level. What I you know what I want to achieve is this model right. Instead of now you have a bunch of coins.

A

Just they know that they're talking to a streaming API, which is you know to build this real-time chat. Your time. Chatter can be really abstract. There's a very simple API. It's just like sure. The state state machine kind of model a and they on the surface ID. You have a lot of functions. Obviously the state has to be there, so you assume there's a in memory store and, of course, they and all them all. The other kinds have to talk to each other.

A

So there's some kind of messaging service has to be there, but otherwise all the so-called application. Logically, the so-called chat logic we are being commanded as those functions or the api's.

A

A

Something how much time I have Oh 15 minutes. Okay, so I'm just gonna, go through the other. Primitive primitive is quickly right. So this is just I think this is sort of the earth. This so I think that is, you know. Maybe service community as a whole can come up. Who is sort of a standard, a way to do that and that's generous. You know it's generalized enough that can support a variety of applications are kind of following this model right there there pi die.

A

They are still for communication, they're, long lived and, and also it's harder to use the push service autopen, because the functions that doesn't have a business logic that doesn't need to know. You know when to process the message and when messages has to be act, it's tougher that so so the first, the abstraction or first sort of primitive is basically the concept or, like you know, how do you actually even started session right? The the function is to remember the functions are stateless, so the current has to the current assume.

A

The kana has a framework which is reliably which you are generating others messages. So the first one is basically the concept of you need to have a way to create a session, and if the session has to be adjustable right- and you know the concept of a some kind of session ID- that a session ID will be also passed, the back to the kind.

A

The second primitive is like: how do you deliver a new messages to the current, how to deliver a new message from the current? But what was you know? What is the underlying semantic sea involved again assume there's some kind of in memory store behind those functions, so here's what I just defined it like. So the sender from current will be transformed it to some kind of log messages and, and then there was the ACK they'll come back.

A

So this is again a saying that is, you know how to heart: yeah I guess they're typos. So this is like hard for the. How does how to generate a message it back to the kind? Well, you know the first of the consonant messages right, let's see from one kind against the same chatroom and then how do you like to send a message back to the kind like what what all the function is involved.

A

So this is basically how do you generate?

A

Deliver several messages? Yes, so this is a heart to generate a silver messages from the circular sessions. So so the on the left side is the concept of the session store and hard session. Stop push their messages to the to the client.

A

And this is sort of the heart and everything works into I, guess you know this a little more details there and also depend on how you implement on, but the concept is the you know. This concept is so essentially on the suicide, and how do you actually send a message? Is that to the kind back to the kind right, hot, the whole end-to-end loop, how this works? So this is sort of concluded the basic, primitive. Like you create a session. You have a way to for the kind of standard messages to anything.

A

You also have a way for the server to send a message back to the kind- and in this case the cost is the concept. There's a push. I have this concept, this coffee mug function, which means I, still function, but it's not an implemented by the application. It doesn't really contains the so-called business logic.

A

So to summarize, you basically look here something like that right, so there's a concept of stashing and which, on the client side, you know it's very, very straightforward kind of api's, but also on the server side for all the sub in the civil environment. You break down them to individual service. Individual stateless functions, but there are, there are definitely compact cities. I mean those proof. Those like primitives in looks very simple, but really to make it work.

A

You have to solve a lot of sort of harder communication problems like how do you ensure the message of ordering, because the message you can come in imagining the communication star is the other concert on suicide and they may not agree with each other right and some information just can still be delivered where the new message against retrial and stuff like that, and then how do you recover from a failure seeing it from either end how to ensure the entry IQ works and also even more calculate how the protein treatment works?

A

I fo can choose a concept where the way those things become case, you won't have a way to push back to the piercing, slow down.

A

Yeah, so so you know look at it, it's obviously it's an Austria for the first is on the for the developers to improvise those functions. It's not a it's! Definitely not as straightforward as as just implemented our WebSockets, the endpoint. So I guess the question is yeah. Why do I have to go through all the trouble? Maybe we should just go.

A

Is this the kinda model like using our push service, I, think the the problem I'm, seeing with using a push service compared to you know, application or in this case functions actually own and implemented endpoint on the server side? It's basically like two problems right once the the push service typically has the actual agency and others cost as well. Second is they're not as flexible as to whether you would have wanted the application to do. They have a pretty stricter, like predefined order in semantics, like ooh, you know is how message is totally ordered.

A

You know what happened for there's a message gaps. Do you really support it exactly once, or you know at most once kind of I am not aware of a messaging service who allows you to really deal with a lot of sort of application to find semantics.

A

Otherwise you know you kind of you can imagine that, as you know in the future, if we want to do like a virtual reality based a chat, you have this really low level, overhead storage kind of services. It's going to be hard because the throughput is gonna, be so higher. The number of you know the message rates were so high. It's hard to imagine, try to sort of offload all those communication to to centralize the service, so I think I.

A

Think you know this sort of so this sort of the the motivation I think that is develop. A stated solution for server is having the application actually on the state of end point in the form of the service functions pass in memory, store, I. Think it's very useful, so I would you know I, don't know I mean you get all other things are on a proposal really I think that is yes.

A

There is a pretty interesting or hard problem to solve here, which is how to do stateful communication against the stateless civilus functions, and how do you scale? How do you make an actual work from application standpoint of view? I do think of that. As you know, for this to work, you definitely need some kind of framework support on the on the server side. You can't just have the application or the functions to handle all all those messages cymatics by themselves.

A

So now the question is: how do you define those frameworks and also define those primitive, seeing a way, that's as as transparent as possible for the functions so that they can focus on the business logic instead of dealing? We saw this sort of communication problems, so I'd like to hear what do you think about this? You know it's really a problem when you solve and maybe there's already other attempts to solve the similar problem, or you know how we are defining this problem.

A

If yeah you can talk to me after after this or you can ask question are also you know, my email is pretty easy to remember. That's called we have had good comm, so send me an email if you want to discuss more I, am very interesting, incredible reading with anyone that interested in solving this problem in a general way for developers. Okay, thanks.

B

All right any questions.

C

Can you elaborate a little bit more about the status behind? That is that something Google is trying to work on, has been working on, trying to standardize.

A

So so Google has been try to solve this problem for they're sort of our own internal infrastructure to support application, like you know, chat, Docs. Morally, the rectify store those things I, don't think who have a solution active days for what it called car wrong.

A

I, also don't think appengine I mean I've had you're, basically using a push service in terms of making this a spec, yeah I think that's Michael I think there lot of spec sort of focused on the transport side, not so much on in terms of primitive architecture and also making other things agnostic to the actual transport or even the longtime I.

D

Don't take this the wrong way.

D

It feels a lot in the server space like us solution, looking for a problem, what you presenting here, because it feels like we already have eventing, which will allow you know, sort of a sequence communications in connecting different pieces and if you ever needed, you know connection that would survive, then use somethin else. I mean you, don't have to necessarily use server less so and I'm missing something. It feels like it's an interesting solution, but not for the service space, but.

A

Yeah I guess yeah, so that's the good I guess feedback right, I, guess, I! That's why I want to talk hear from you guys does if somebody's really have to solve this problem and also solve this problem in this way. Maybe there's other solutions. But you know it's not question I! Guess so and I have answer that.

D

Any other questions.

E

Part question part: I: do think what we've definitely heard: a lot of feedback from from Azure functions, customers wanting more stateful patterns, I was gonna, send you an email but I guess the only question I have. Is you emphasized a lot in memory and I? Don't know if you really mean in memory or you just mean state, the other one is shameless plug I'd be interested to get your thoughts on durable entities and some of the durable functions stateful stuff that we've been working on as well around some of these stateful serverless patterns. Okay,.

A

Yeah, so it does have to be in memory, it's just a it's all about latency right. So if the latency, if you put our fresh yes, you still can't be the latency, it's less something like a less than one in a second. It doesn't have to be yeah. I've, certainly think it seems like walk fraud, durable thing that inherently also have also state. You have to maintain I think they, those up by the understood apartment, has a predefined sort of pattern or temperate I.

A

Think overall, it's a better solution for I try to serve, but you know if you can build a chatter IP using a fraud defined all this work for durable pattern. I want I'll, hear about that. So I think about you know complementary, but that's just my thought. Yeah, okay,.

D

A

Time for one more question, but yeah pre send me emails about any technology solutions. You have you since.

B

The azure guys did a shameless plug all those shameless plug to have you taken a look at cloud state which is building on akka to allow these very low latency communication not going to a back-end memory store, but actually putting the the proxies in front to be able to manage that communication and stay coordination. I.

A

Was think about that? Are you yeah, so I think there's a second talk today, Tasha based about that there's streaming, your the in proxy for the open, whiskey, yeah presenting send me a email, you know, so it can keep the discussion I'm very focused on sort of the sort of Google scale infrastructure, but it's mostly internal, so I'm not really aware of all the so state-of-art in the open source service water. But I'm really excited to hear about this. Okay,.