Cloud Native Computing Foundation Kubernetes Community Days (KCD) Chennai 2022, 30 Jul 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Microservices Madness: Application And Infrastructure Automation Using Temporal by Mohan Muthukumar

Description

Software is not the same as it used to be; Many of us remember the days when building an application involved only writing code to solve a business problem and having it properly run on a single device. These days, however, applications need to run across machines and with distributed systems come distributed system problems. Even before we consider solving the actual business problem, we have to chart out the entire operational architecture: mode of communication between microservices: sync or async, worry about things like retries and backpressure, transactional logic and recovery from failure. We end up in a situation where the core business logic takes fewer days to develop than all the surrounding plumbing code and then also setup the build pipelines, deployment and auto-scaling. What if there was a better way to build fault tolerant applications that also took care of its own infrastructure in a programmatic way: A self-deploying, auto-scaling, resilient application.

A

Okay, so who am I I'm mohan moto kumar, I'm a senior software engineer at cavallara, the largest company that you have not heard of. We are in the tax compliance business and we are almost part of every transaction. So if you bought something online, uh there is a very good chance that it went through our assistance and so the the task that my team does is we we go over the internet crawl, a lot of pages, look for product information, tax changes, startup changes, all these things and effectively. We have to run a mini google.

A

So, given the scale of our operation over the years, we've seen all kinds of systems um I mean, if you can name it, we've done it. The topic at hand. Microservices are becoming unmanageable.

A

So if you work at a large company or even a moderately sized company, all this would be very familiar to you um these days. You know there are more and more micro services. You have a single engineer, managing 2030 microservice and all the plumbing code to make all these micro services stock with each other, plus uh the devops surrounding it. All. This seems to take much longer and a lot of yaml much longer than the actual business use case.

A

You'll end up in a scenario where, let's say a business use case, takes only three days four days to build with all this deployment and plumbing code making and making sure everything can talk to each other. All that takes a week to ex of what it took uh to cure to solve the actual business use case. So this is, you know this is not a good situation to be in so this is just to be clear. This is not a let's return to monoliths like good old days kind of talk. So this is a talk.

A

That's been, I mean this is a sentiment, that's becoming popular in certain circles, and I don't mistake them because uh you know things used to be much simpler. 15 years ago, all you had to do was just write. Some code deploy that in an application server and you just had a web app.

A

So there is a reason why we got here and those reasons just don't go away, so you cannot return back to monoliths. So let me give you an oversimplified version of how things were and how we got here. So this is how things were 15 years ago, you had a single service with multiple concerns you had. You had all the clients actually talk to this single service, um so you had sign ups, you had multiple resources in different parts and you just saw that. But a couple of things happened after that one. The internet happened.

A

So instead of serving thousands of users, thousands of users, now you had to serve literally hundreds of millions of users, so the internet scale happened, and then you know the most popular runtime node.js. That kind of improved the developer productivity by a lot ran only on a single thread. uh Simultaneously, you had all those moves. Last stop and you have you had uh processors with multiple cores. You had processors with 16, 32 and 64 core processors on the server side, but you had a runtime that ran on a single trade.

A

So you know you are essentially going to be wasting a lot of course, but a few years later, docker came in and one of the things that it gave us was the ability to run multiple node processes, uh processes on a single machine, so that suddenly gave you a way to utilize all those id cores, and so people had something like this. You had multiple copies of the same app running on a single machine and you had multiple copies of the same app running across machines as well.

A

It didn't make a difference and you just put a load balancer in the front and any client talk to the load balancer. If the load balancer directs the request to any client at random- and you can just you know- serve the traffic and so you're able to scale really well. There was a problem with this I mean, rather than running multiple copies of the same app. It made sense to slice these up vertically and have these uh scale up separately right. So maybe you don't have a lot of slash. Signups login requests right.

A

So how about we take that we take all that and make the quintessential example of a microservice, the user service, and you know, deploy that separately. So people started slicing things up vertically, and it also helped that most organizations also could have dedicated teams working on dedicated services um with a limited scope right, so people started slicing things up vertically and it also helped that you can scale these things separately. Let's say you know there is a service that that is getting a lot of traffic.

A

You can just scale that service independently without allocating resources for everything else, so it made sense- and this is I mean if you ever see microservices. This is probably uh this is a diagram that you would have probably seen already except. This is not true right, so this is true only uh if only your client is the only thing that's talking to all the services. Only then this is true, but in reality more often than not, your services need to be talking about themselves too, and so your diagram starts to look something like this.

A

Then you can already see it becoming more complex than that. The previous diagram showed and soon enough that, since there is nothing stopping people from creating more micro services, it is going to look like this. So this is an actual image from netflix and it in fact looks like this now imagine working with a system like this. Not only is this unmanageable because nobody has a complete view of the whole system, and companies also have dedicated micro services.

A

uh Sorry, dedicated platforms, team, dedicated sre teams and all that just to you know, uh alleviate the pain of dealing with this complexity for the business developer.

A

So not only is this unmanageable, but it is also unreliable. What happens if imagine working at in this environment- and you want to uh you- want to create a new service? You want to deploy a new service, you don't know what is going to break right, so, just as soon as you uh deploy a new service, if something something is wrong, it could bring down the entire entire system.

A

Potentially so developers tend to work in this uh you know frightened environment, and also one thing that is often overlooked is uh if a service call fails in a distributed transaction.

A

Let's say, for example, if you're doing a an ordering service, an e-commerce service, if a service call to the payment service fails, you should also revert whatever came before that right, um I mean, revert the dispatch, something like that, but in a in a situation like this, people tend to miss that right and it's not um it's not obvious to people that they should be reverting to reverting everything that came before them.

A

So this kind of makes things unreliable and we kind of got here for scalability right. We wanted scalability and we sacrificed reliability and kind of this seems generally true anytime. You try to increase scalability your reliability suffers and anytime. You try to increase reliability by introducing transactions or locks. uh You know your scalability suffers, but this is not actually true. This sounds true, but it's not true. You can have a system, that's both reliable and scalable.

A

So that's where you want to be right. So is it possible to build a system like that? I mean we've done this in the past, but it requires a frankenstein's monster of systems. You need complex state machines involving kafka queues, distributed locks using ncd. All those kinds of things you need a centralized data store for anything, that's critical, all of those things and it might still break when a new service is introduced.

A

There is no guarantee and it does nothing to improve your situation with respect to rollbacks right and also given the number of systems involved, you will have to write. You'll have a lot of different tools, deployment tools, ci tools, all these kind of tools and we'll have to write literally ungodly levels of yaml. uh I mean I'm, I'm pretty sure. Nobody here enjoys writing yaml yaml files.

A

So this is the situation and is that? Is there a better way?

A

There is a better way. I mean if you uh for an analogy, if you think about operating systems, operating systems have solved. This hardware is unreliable, just test distributed systems are network, is another unreliable, so is the hardware. So is your actual hardware right, but your program doesn't need to account for that, because your operating system gives you an abstraction, your your opting system, abstracts all this away from the actual programmer right.

A

So what we need in the case of all these distributed systems is an abstraction, and temporal is just that so just to give you a bit of background on what temporalis. So this is a project. That's been growing for the last nearly a decade uh it started. The team behind temple was also the team behind amazon, simple workflow and also the azure durable functions and um right before temple. They were working on this open source project from uber called cadence, which is very similar to temple.

A

Aside from some internal differences like grpc and thrift, so temporal treats your entire application as a workflow. You have you have this idea of everything being a workflow, so everything is a series of steps that executes to completion. If you think about it, that kind of describes any any computer program right. You have a series of instructions that just run to completion so, but what makes temporary different in this case is they have durable state?

A

Unlike let's say any, I mean so they have durable state and you can do all this using code without writing. Any any yaml so temple gives you the building blocks to build your distributed system and it handles all the communication for you. So you don't have to deal with. uh You know, writing to the right queue and all that, and it also stores your execution state such that. If something happens, your workflow because of some infrastructure issue, it can just recreate that state in a different, different worker different part.

A

So this is this kind of sounds like the missing piece in the stateless kubernetes ecosystem. If you think about it, so uh creating a temporal system is pretty uh pretty easy, so you have workflow functions and you have activity functions, so workflow functions in in workflow functions. You need to specify what comes after what and you essentially you create the state diagram using the workflow functions and the workflow functions need to be deterministic, and then you have activity functions which can be just any kind of functions.

A

So let me just give you a hello world example. So here you can see you have a function, uh the workflow function, and then you have the activity function. So the workflow function uh has a few config stuff, and then you basically specify that you want to execute this activity, and this is your activity function. All this does is just return. Hello plus name your general hello world example. So this is just going to uh return hello world to this result, and the workflow is also going to return hello plus name right. That's that's!

A

Basically it so here is a more complex and more useful example. So let me actually zoom into the first part of the code. So here, if you see the first part of the code, is just me setting up some config stuff for timeouts and retries. So if you see we have some retry policies for the activities, I will come to that in a bit and also you know you're sitting setting some timeout condition. So the interesting piece is the bottom half.

A

So, let's let me go to the bottom half, let's start with the config part, so this config part you don't actually have to specify. There are sensible defaults, so some of these are required. Some of these are not. um It will actually for the things that you don't specify. It'll actually use the default value, and so this is the bottom half so here, if you see uh imagine an e-commerce workflow right just this is just a part of an e-commerce workflow.

A

Let's say uh you already have place an order, and now uh you know you need to start subtracting from the inventory. You need to take the uh take the item off the shelf so that somebody else doesn't add added to the cart and also you need to send the user to the payment page, and let's say you want both of these running simultaneously and so essentially, we've invoked both of these activities simultaneously and here on line. Nine is where you know you are awaiting the result uh by doing a dot get call.

A

So if you uh now, both of these are going to get executed simultaneously. And here, if you see we are using golang's error, handling primitives uh the way you would normally handle errors in golang to actually handle an activity failure, so this will uh based on a retry policy that you saw on the previous screen. This will already try multiple times five times, for example, or until you hit the exponential back off threshold.

A

There's a lot already try five times and if it tries, if it reaches the maximum number of attempts allowed only, then it will go into this block, and here you can see you know if the payment info info fails. We are putting this back to the ship, so you can just uh handle the compensation logic inside your error handling, and you know it will be.

A

It makes uh it makes it a lot easier to wrap your head around this and, if you think about it, there is a problem with this code right. So we are doing this for the payment activity. The payment um activity fails, but we are not doing this for the inventory activity right. So it becomes very obvious that we should also be reverting the payment, but we're not doing it. So you can actually catch this during uh code review.

A

Imagine imagine how long it would actually take for you to find fun uh find out about this and all those yaml based systems where chances are things don't even exist. In the same repository um yeah, so let me give you an example of the activity. All these are you know. Activities can be anything any golang function.

A

The only requirement is that it needs to take a context as a first parameter and return an error, at least so here, I'm just making a http call to this inventory service, um I'm just making a post call and if the post call fails, I'm returning the error for why the post call field.

A

So obviously I don't have the service running so okay before that, so both these activities and uh workflows need somewhere to run. So that is your worker code. So here, if you see we have you know uh we are configuring, a queue name for the worker and then we are registering every workflow and also every activity with this worker and then, uh since this is going to run as a kubernetes squad, we are.

A

We have set a health and point- and that's basically it now since this, since this worker is going to be part of a deployment, if you want to increase the throughput of your workflow, all you have to do is scale this worker, and you know, depending on the logic of your workflow code, it is going to horizontally scale.

A

So this gives you reliability since workflows are durable, like anything that you have in the workflow function will complete uh will actually complete. There is no infrastructure failure that can actually prevent the workflow from completing. So let's say our workflow function is running on a worker and the worker goes down because you know for some reason, any reason your spot machine went, went out or something like that. Any reason temple will know to restore that uh restore the state of that workflow in a different worker and it will resume the workflow.

A

So you know that gives you a lot of reliability plus, uh like I showed you, you can handle your failure. Cases using the error handling constructs in your programming language, and you cannot, uh you cannot forget, to handle it and you'll catch it during your code review and you also have retries with exponential back off. Imagine having to reinvent this every time, uh you're sending an external request that would to a flaky service and pretty much anything across network is flaky given long enough time.

A

So you would want to retrace and temple gives you that out of the box, so uh going back to our activity code, since none of those services actually existed, uh all my attempts are actually all my price hitting that service uh was failing. So if you see, if you look at the standard standard out, um you can already see that we are attempting it multiple times, so you can see at m2 at m3.

A

So and also you can see that we are in fact hitting this simultaneously we're not waiting for the payment gateway uh payment activity to finish in order to hit the inventory activity. So we are trying both of these simultaneously and both of these are failing, and you just by looking at this. You will um just by looking at your logs.

A

You can tell that you know this is the activity that's failing, plus uh temporal also gives you uh comes with a web ui uh just like kubernetes, so this is an optional piece that you can install. So this will give you a bird's eye view of all the workflows that are running and all the runs that we have. So you can go inside any of these runs and it will give you the entire event history of everything that happened.

A

So here you can see on line 13 that this is the reason uh you know this is this is the part that is failing, and it will give you the reason for the failure and the number of price. So this is now here. It says, attempt five. So this is this. This part is being this activity is being tried for the fifth time and now, since that also failed, it says maximum attempt reached right so, and that was on reliability and on scalability.

A

uh So part of the reason why you end up with hundreds or thousands of micro services is one, it's convenient to start start a whole project from scratch, but also because people try to tend to prematurely decouple services in anticipation of future scale right people might think hey. Maybe today, I'm getting only hundreds of requests, but in the future this can potentially have hundreds of thousands of requests.

A

So let me actually decouple this and put this as a separate service so that I don't have to uh decouple that at the time and because people have uh burnt their fingers trying to decouple monolithic machines right, so people want to prematurely decouple services these days, and but you don't have to do that in case of temple. So all you have to do. Let's say: let's take this example, uh so you have three activities running and let's say, for example, on line 15.

A

The payment activity needs to run at a different concurrency and let's say it's also an expensive call. All you have to do is just remove this activity from this worker and create a new worker register that activity over there and now deploy this worker in a separate kubernetes deployment and can scale that independently- and here, if you see we, I've also set the concurrency uh to six. The default is thousand, so you can have only six instances of this activity running for this worker at any given time.

A

So that way, you won't be uh bombarding the payment service with a lot of requests. If you have especially burst, traffics uh you'll only be handling six requests at a time and you don't have to also prematurely decouple services, because it's as simple as just unregistering an activity and registering it in a different worker.

A

So if you think about it I mean I'm sure all of you will have uh services at your company that do not really get a lot of traffic but still need their own deployment, because there is no other way to deploy it. There are services that hardly get 100 requests a day, but they run as dedicated services right, but with temple you don't need dedicated services.

A

For that uh you can just that can just be another activity uh executing the logic inside uh inside the activity function itself, and this can get scheduled whenever it is required.

A

So that way that also reduces the number of micro services that you need and your application can be made of thousands of workflows and the all these workflows can actually be executed. Concurrently. So let's say you are building a complex application. There is no limit to the number of workflows that an application can have. In fact, we have every day we run 200, 000, workflows um and also, if you are, there is no limit to the number of activities your uh workflows can execute.

A

The only limit is, uh you can have only 50 000 events, and you can basically work around that event, uh that limitation by creating child workflows and making sure that no child workflow has more than 50 000 uh events in in the event history.

A

So one more thing now, if you remember the beginning of my talk, I told you part of the problem is also all these infrastructure automation tools for which you have the right conflict right. What if your application can bring? Its own infrastructure.

A

Imagine at the at the beginning of the workflow, you have an activity that talks to the uh who talks to the asd, using aws, sdks or cdk anything and uh increases the asd count. Let's say you start with five: you increase that to 10 at the beginning of the workflow, and then you talk to the kubernetes apis to increase the pod count, and then you execute your business logic, the actual logic that you want to execute that we built in the previous screen, for example the commerce workflow.

A

You execute that as a child workflow and then on the way out you can, you can decrease the path, count, decrease the asg size and also maybe perhaps send a slack notification indicating the workflow completion, uh and you did all this without using a single uh without writing a single line of yaml. So yeah I mean your application can bring its own infrastructure so other use cases we have. uh We have a lot of these dag based systems like data pipelines, the elt pipelines and machine learning pipelines.

A

All these actually tend to naturally lend itself to uh temporal, because uh temporary attack based pipelines are a subset of what temple can do and also ci cd systems, like I showed you in the previous example, can very easily become uh you know, temporal workflows and making them temporal workflows gives you a lot of opens up a lot of possibilities. uh Running smoke, tests running integration, tests, running chaos, tests all those kinds of tests can be done.

A

You get a lot of flexibility. You can also aside from this dac based systems. You can also build complex event driven systems, so temporal gives you a construct called signal. So, just like how you can wait on a golang channel, you can wait on an entire workflow.

A

So your workflow execution is going to halt until you know you receive a signal from that signal. I mean you receive an input from that signal and one more thing that you can do is you can uh build your own dsl based workflows? Let's say you already have some dsl that you want to execute. You can use temple to write an interpreted. You write an interpreter for that esl. We can execute that come on a quick time check, yeah yeah, closing box, closing thoughts.

A

So you don't need me to tell you how powerful an abstraction can be. You already saw that with linux containers with docker and the kind of impact that it did uh kind of impact that it had. You already saw that with two military services and the kind of impact that it had so few years ago we had this serverless movement at the peak of the serverless movement. We had pretty much.

A

Every single cloud company come up with an offering and also a promise, a promise that developers wouldn't and shouldn't have to deal with all this glue code around their business code and that ops will also be taken care of seamlessly so, but that kind of fizzled out right so now here is a system that helps you check a lot of these boxes, although in a different way.

A

So with this you know, temple might just be the system that fulfills the promise of serverless and with all this unlocked productivity, imagine the possibilities. So thank you. That was my talk.