.Net Foundation On .NET, 14 Aug 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Building workflows with the Durable Task Framework

Description

The Durable Task Framework allows developers to write long running persistent workflows in .NET code using familiar async/await capabilities.

In this episode, Affan Dar, Simon Porter join Jeremy to give us a better understanding of this framework, what it enables, and where is it being used within Microsoft.

1:38 - What problem does the framework solve?
6:20 - How does the provider model enable extensibility?
8:26 - What does a typical workflow look like? (Demo)
12:25 - Does the framework handle scale out scenarios?
15:46 - How does the framework enable durable workflows?
20:45 - Where's the best place to find documentation?
21:47 - What other projects at Microsoft use this framework?
24:09 - What are some good use cases?
26:19 - What's on the roadmap for future optimizations?
Useful Links

Durable Task Framework: https://github.com/Azure/durabletask
Samples on GitHub: https://github.com/Azure/durabletask/tree/master/samples/DurableTask.Samples
Azure Storage Emulator: https://docs.microsoft.com/en-us/azure/storage/common/storage-use-emulator

A

You've probably heard about durable functions,, but did you know that behind durable functions is the durable task framework?.

A

This is an open source framework. That's been out for years that.

A

Helps manage long-running functions and maintain state automatically..

A

You need to check this out.

A

Hello, and welcome to another episode of on.net.

A

I'm your host jeremy, likness, and today I've got simon and alphone with me.

A

We're going to talk about the durable task, framework,.

A

Not to be confused with durable functions..

A

This is something that I was thinking about when I first saw the title, but you'll be surprised to find there's an open source project that drives mini-projects,.

A

Actually, inside and outside of microsoft.

A

before we dive into that framework, though, simon tell me a little bit about what you do at microsoft..

A

I am simon.. I'm engineering.

B

Manager on the azure iot platform, team,, specifically for iot hub,, where we offer a variety of platform as a service offerings.

B

We make heavy use of durable task within our systems.

A

Okay, and alphone?.

C

Hi,, I'm alphone,, I'm also an engineering manager in the it platform team..

C

So as simon mentioned,, we make heavy use of the durable task framework,, and so I have been involved in.

C

The original version of the durable task, framework.

C

So I've been on this since the beginning.

A

So you helped actually contribute to making the framework?.

C

A

So tell me a little bit., let's back up for a second.

A

What is it the framework tries to solve?.

A

What problem are we looking at here?.

C

So the framework is meant to enable users to build workflows using code.

C

Many workflows,, when you think of workflows,.

C

Typically, you think of visual designers and boxes that you draw up induce hairdo,.

C

Debit account, credit account, do something, provision a vm,, create a storage account in terms of visual flows,.

C

But that did not work for us when we were building out our first provisioning system,, which led us to this new approach of thinking about the problem.

C

How can we actually use code, and specifically c#, because of all the great features in c#,.

C

To build out workflows that you can just write in code and just deploy them,.

C

And have them scale and realistically intend, and all of those things.

A

Well,, that's interesting to hear because I think some of.

A

The best tools are the ones that were actually created to solve a problem.. Sometimes you hear about frameworks, people create in search of a problem to solve, right?.

A

So this was something that you ran into an issue,.

A

And specifically, we need a better way to do it in that way was workflows through code.

C

Yeah. actually,, it was a hackathon project., so we were building out the provisioning system for api management.

C

Api management service as an azure service.

C

We looked at various approaches,, we looked at building a state machine-based system where.

C

You would create a storage account and then you would store the state, and hey,. I just created a storage account,, and then you would go on to the next step, which is creating a vm, and then time the vm to the storage account.

C

all of those things, right?. So there was a lot of state management in the middle., so we thought why could we not use a vits,.

C

Basically, the c#,.net tasks,.

C

And vits, to actually encode this very simple control, flow,.

C

Which becomes so complicated once you throw in databases and answering machine, and then how can we incode that in core drive.?

C

So this is where it came. Up.

A

So the idea, if I understand it correctly,, is instead of me exclusively opting into some sort of state management mechanism and saying,.

A

Okay, I have a long running workflow., so encode I'm going to save my state,.

A

Do all this ritual and ceremony. come back at another point, go rehydrated,, try to figure out what to do.

A

you're, taking advantage of the language features to manage state sort of behind the scenes for the user.

A

So for me,, it's simply do something,, wait for that long running, tasks, and continue.

A

There's magic that happens in between, but that's taken care of by the framework.

C

Absolutely. so to add to that., so not only do we,. Actually, let users have avet become durable in some sense.

C

We also have features which, in the framework which actually.

C

Allow the whole process to be dehydrated to desk or service bus in this case., then we dihydrogen vendors actually work to be done.

C

So as an example,, if you have a two-step process, which is again going back to my example of clear the vm, and then create a storage account., you would create a vm caller method.

C

Which creates a vm under the covers, just makes two calls to.

A

That's not instantaneous.

A

exactly.. It takes like minutes,.

C

Seconds,, I guess tens of seconds at least, if things are working out fine, but so once it creates the vm,.

C

The next step you want to do is to create a storage account., so you want to fire off the call to create a vm,.

C

And then you want the process to go to sleep in the sense that there is no memory being used,.

C

There is no compute being used as an adverb being used..

C

The only time when this control flow comes back to live is when the vm was created.

C

Then, at that point, you want to control far to start from the next step, which is basically cleared the storage account.. So you do not want it to redo the whole halting again.

C

obviously, in some sense,.

C

You want instruction pointer to the code to be stored, persistently, properly..

C

So this we enable using this magic of durable tasks,.

C

Where the tasks that you get back from this framework is what we call a durable task,, which is automatically stored on some storage,.

C

Key-Value store, or some parts for some azure storage,, whichever provider you're using., also it allows you to resume when the task is.

C

Actually completed so that, I guess, is the magic of durable tasks that we would want to use.

A

So you talked a little bit about service bus and an azure storage..

A

So there's some supporting mechanisms behind this that you're using to frame it in..

A

Is there any code or example you can.

A

Show that to drive that point home where I illustrate it.

C

So we haven't done.

B

When I came in and started working with firm at that point in time,.

B

There was a service provider which was used for the messaging flow and session state,, and then azure tables is used for the instance store,, which is really the history of the orchestration.

B

One of the things we did at that point was we made a provider model.. So if you could then bring your own provider.

A

So if that wasn't going to meet your needs,.

B

You could write a provide it for sql..

B

There is one for storage.. There is one that's going to be very soon core redis and another one for service fabric..

A

So, depending on the teams need some people to service bus, team,.

B

For instance, cannot take a dependency on themselves,, so they have a service fabric, provider.

B

okay., interesting.,.

C

So the main design principle.

C

Behind the providers was to have,, as we do dependency on a special feature as we could.

C

for example,. All you need is a key value store and potentially.

C

Actually, that isn't, literally, give you the key-value stored and some compute, obviously almost forgetting something.

A

Some place to run this code., so I may be getting ahead of myself,, but it sounds like you could potentially spin up something lightweight then, for tests run through or smoke screens, but plug it into production. Provider,.

A

"Service bus, active account",, for example, when it's deployed, is that okay?.

B

Absolutely. yes,, we actually have an emulator.

B

that is an in-memory provider. okay. that you can use for local testing,. So you don't have any outside dependencies and both the latency or issues that come with that.

B

A

So I see you've pulled up some code, to tell me about the code. You've got.

B

So what I've got is I'm using the servicebusorchestrationservice, which means.

B

That we're using service bus and storage for our state.

B

What I've defined here is you have to find a basic,.

B

I've instantiated the service bus, client,.

B

The servicebusorchestrationservice we're going.

B

To use an azuretableinstancestore for storage connection string and service bus connection string is going to get us. Our queuing mechanism and state ends and session state.

B

We're going to create some of the concepts,.

B

Or there's a client-end a worker., so the client is what you will call to create a workflow and fire. It off,.

B

And then the worker is the one that's actually going to have dispatches that'll pick up the work, and run it for for the orchestration and that'll in turn, spawn off activities, that'll go and run,, and one of the things that it does, is. It allows you to fan out.

B

okay. fan out the tasks.. So not only do you get the durability guarantees of long-running processes,.

B

But you can fan out across if you have operations that are compute-intensive,, you can fan those out as well. okay.. So the basic example, I've done here is really just a sum of squares orchestration,, it's good for illustrating the concepts,.

B

But it obviously is not something you end up. Using.

C

Orchestration,, you can think of that as a control, flow,.

C

For the workflow and activities as the actual work items that control flows.

A

Right. so my flowchart,, all my arrows, and decisions are part of the orchestration, and my nodes are going to be active.

B

Exactly. yes. okay. every orchestration has a state, and that state is what it uses to replay into the previous activities. Results.

B

So what I've got is I've created an array.

B

It's an array of either numbers or sub-arrays of numbers.

B

Okay., this orchestrations is going to go, and it's going to sum the squares of all the numbers.

B

okay., so I'll get back to this code in a second,, but I want to show you. The activity that we have defined is pretty simple.. All it's going to do. Is it's going to receive a number,.

B

It's going to return the square of that. okay.. We have an orchestration which we've defined, it's going to take the input, and it's going to pass it to json,, and then it is going to loop through the items in that json,.

B

And if it's an array,, it's going to create, what's called a sub-orchestration., a sub-orchestration is an orchestration within an orchestration,, but it has it's own lifetime. Essentially.

A

Right, but they're tied to each other. parents and everything and until the child.

C

Think of this of workflow spinning up another workflow.

A

Okay., so this is a recursive sub-orchestration basically.

B

Yes., so if we said, if it's an array, we're going to create a sub-orchestration which is going to go, do its thing,, if it's an integer,, it's just going to go and call that activity that I just showed you that does the multiplier.

B

okay.. Then it's going to, each of these is going to be creating a task,, and then we just do, went all on our tasks, and then we're going to sum the results of that, and then that's going to be the results of our orchestration.

B

So a little bit more about how we got the setup is,.

B

We have the client where which we're going to call to instantiate,.

B

And we're going to have the worker,, which is actually going to do the work.

B

on the work that we registering our orchestration,.

B

Where you registered the type itself, and then the activity would have only done just one as a simple example,, but a sum of squares task.. So we basically registering our code with the worker,.

B

So that it knows what to do.

C

The worker is the unit of scale as well., so imagine this one worker is running on a single vm, right?.

C

So it is processing n number of orchestrations,.

C

As the client says, "hey, run, this workflow, run this workflow, and run this worker.". This worker is the only one picking up all of these,, actually executing them.

C

you're could throw in more workers as well. give them the same, give them the same connection string as you see,.

C

As this one is being given.. They all will compete for work on the same service. Bus store in this case.

C

Does this project also handles,.

A

So this handles the orchestration, and the clients, and the flows.

A

does it handle providing the infrastructure that the scale.

A

Out or is there something else have to provide the different vms or processes.

C

That thing we've left as an exercise for the deploy, because.

C

There were so many different combinations of places where you can operate them.

C

samples that show you how to run it in a single process,.

C

But actually running it in different hosting environment is not part of that.

C

Okay. so when we talk about,.

A

Because I know a lot of people get confused a little bit., they hear durable tasks framework,, they think durable, functions., durable function, uses this,, but part of the services is providing is managing that scale out and providing.

C

Exactly., how it's basically for all the processes., that's advantage. Definitely.! That's! The durable function is more managed way of using durable task. Framework.

B

Got it. okay. makes sense.

A

It's also got the rich integration with the david experience,.

B

To host your functions and everything I get as well.

B

right., so then you really bringing all of those things together.

B

B

So once we've got up to our orchestrations and activities registered, we will go start it, and then here we can use the client..

B

We can create an orchestration., we'll tell it what type it is, and we'll give it the input, which is that bag of numbers json that I showed earlier., then we're going to do a sample. Wait for orchestration..

B

I've got a little bit of debug that chosen here,, that what's going to do,, but what's not apparent here, is what's going to happen. Is,.

B

It's actually going to be doing a whole bunch of message..

B

We could have had this running on five or 10 machines..

B

You could have the client on one machine,.

B

You could have five machines being orchestration, hosts,.

B

And you could have 15 or 20 machines running activities depending on what they're doing..

B

So if we take this now and we run it,.

B

It is going to start up and you'll see that we've got up an input.

B

Now it's passing.

A

So the original orchestration.

B

Got that in,? It then went and set, I'm going to run activities for the two,, the three, and a sub-orchestrations for the four and the five..

B

So that's you'd gone and sent that back out to be more work.

B

Right. then so we'll get more orchestration input for that.

B

We'll get more orchestration input for the others that we've defined in here, and those will fan out, and then they're, eventually going to fan back in. As you run, all the squares and,.

B

At the end, where we've got our debug.

B

down here,, the sum.

B

Of the squares is all the orchestrations actually completing them., so you can see that the inner ones are fanning back in,.

B

Until eventually, we get the sum of squares, which is 2,869, which is the sum of all those squares.

B

So in some sort this is basically doing a scatter/gather.

C

So if you have a farm of machines,, the main orchestration is going to take the input,, bigger new chunks and then create either sub-orchestration out of it, if they're, bigger, sequences, or it is going to hand them off to activities which are leaf level nodes.

C

leaf nodes versus sub-workflows..

C

This is going to wrap up all of the results. after fanning them, out, collect all the results, and then paste them on the screen.

C

Interestingly,, this is also reliable., so if, in the middle of the workflow,.

C

If the machine just blew up, right? just went down, restarted,, it would start from exact same spot where it had stored.

A

C

Because the status is saved., it actually remembered the last wait. It was stuck on waiting for and then it just resume from that..

C

So that's actually one of your constraints.

A

Inside the main orchestration, is that it has to be reliable, code.

A

Right?, I can't generate a random number and make.

A

Decisions based on your randomness because then your flow is not working,.

C

But as long as it's a,, I don't even like-.

C

The dominant state.

A

C

Whatever, but as long as it's.

A

A consistent flow, but the activities that are calls those nodes,.

A

Those can do pretty much whatever they wanted. To.

C

Absolutely, exactly., as you said,, the control flow is a special place..

C

That is where the programmer.

C

Actually that's the trade-off that we made.

C

that, whatever you write within that special orchestration, code,.

C

Needs to be deterministic., so you cannot have random numbers,. You cannot have, get me the current temperature or something like that.

A

You cannot do something.

C

That will yield a different result. The next time it is replayed.

A

That's just the control flow. yes. activities are a fair game.

B

Yeah, but we do have some specific functionality that isn't it.

B

That enables things that you need., so you can create,. We have time as, and things like that.

B

So, if you're going to create an orchestration, that's going to fire off an activity, that's waiting for someone to manually, approve something,, and that could take three days, and you say,.

B

I'm going to wait up to three days, and if they don't finish,, then you can actually code that into your orchestration. To have time is fired., because you can't do a regular time, and you can't to do anything like that.

A

So this is actually,. What we've brought up is several advantages, so far.

A

So the way that this manages state means that I have a resilient process..

A

So if something were to fail in the middle of it,, it's preserving state,, so it can replay up to that point and then continue running..

A

So if we crash, we restart,, we still get the output that we're looking for.

A

The other thing is scalability., so you can have multiple workers dealing with.

A

The work that's handed off from the orchestrator,.

A

And then it sounds like there's a lot of patterns that are sort of built in to the framework.

A

So we talked about fan, out, and fan in, and I'm waiting for multiple processes, and then I want to converge them back to result..

A

I'm assuming you have like asynchronous sequential workflows.

A

So wait for one,, then the next,, then the next.

C

Then the nice thing is, that it's all built using .net tasks,, familiar concepts. right?.

C

Right., if you want to do a sequence of workflow,, you await task one, and then you'll await task two, and then await task. Three.

A

I want to call that out because in your code.

A

yes. we did a task when all over here.

B

Right. but we could've done in an await on this,.

B

And then an await on the next one, and then it awaits on the next one.. If we wanted to do them, one at a time.

A

When all is deceptive, because it looks like just a simple line of code, but what you're really doing is all this orchestration and.

A

State management and basically storing history in the background, right?.

C

Yes. so basically, exactly., this simple orchestration actually created. Like I don't know, n tasks.

C

And every task represented some word that was executing on a different machine. Potentially., potentially., potentially, and the tasks would only complete when the machine.

C

Had executed that piece of code and returned some value or some result, and then the result will be passed back to the waiter or.

C

The entity that is actually getting the result from that task.

C

So and then this task that you get back from the framework and.

C

The special context is can be manipulated. Just like any other .net task. You can awaited,, you can chain it with other tasks,.

C

You can use continue with as well, and you can actually do a task when all tasks, or than any of the patterns that you are familiar with in.net tasks, you can apply in this.

C

This sounds like if I want to level up.

A

My c# knowledge and get behind.

A

The scenes of what await and async does this open-source project would be-.

A

Absolutely. -a place to start digging to see how you hook into those language features.

C

We've done some pretty interesting gymnastics.

C

And how we build this out.

C

It's an interesting lead for someone who wants to just experience some [inaudible].

C

I'm going to give it the 15-minute rule.

A

I'll look at the code for 15 minutes and if it's still completely foreign to me, I'll, say it's a good thing. They wrote it and not me.

A

Now, the example that you're showing to get started with this entire framework.

A

Is that do you have to pull down the code,.

A

Compile it is as easy as a nuget package reference.?

A

How do you start plugging into the durable tasks?.

B

We have some nuget packages.

A

Okay., if you want to dig into the code, you can as well,.

B

But the github repo that we have.

B

Does have some samples that allow you to get started with some documentation.

A

I assume the packages give you kind of fine-grained control over what features of the framework you want to use.

B

Yes. so we have a col, there's a col package, which is the col run-time. okay.. This whole thing, but then there's the emulator which you'd bring in for your testing purposes and just general tooling around. And then you pick what you provider isn't bringing whatever packages off for your relevant provider.

A

Okay. that sounds good., we'll put all of those links in the show notes,, so people can go out and grab that.

A

now, one of the things or did you still have something you wanted to show with this demo, because I have a question that.

A

No,, I think nothing.

B

I don't want to distract.

A

So we've been talking about the durable functions because that's something I think a lot of.

A

People are familiar with, but my understanding is. There are a lot of internal projects even at microsoft that depend on that durable task framework, or are you familiar with others,.

A

You mentioned iot hub as well, using that for provisioning?.

C

Yeah. so the first project that we started with was api.

C

so in the context of provisioning..

C

So we have used this framework a lot in just the control plane for our services.

C

For example, api management, when you go and create a new api management service,, the durable task framework run in the background and creates all the necessary resources and ties them together into one unit and then presents it to the user.

C

iot hub. Similarly uses this to build out the control plane.

C

So whenever you go and create an iot hub,, it actually uses this framework to go and create x and y and z resource and then string them. Together.

C

Besides that,, this is.

C

Also used in the runtime plane as well for iot hub.

C

If you go to iot hub and use this feature.

C

We call devicejobs, which is basically, if you.

C

Have a million devices connected to your iot hub and you want to.

C

Run a command or you want to update what we call a device state,.

C

Device twin, for the million devices,.

C

You just make one call to the iot hub and iot hub will give you.

C

Back an operation status.

C

Or operation cookie, which you can use to track, status.

A

In choosing one of these orchestrations., it is basically under the code.

C

It is a big massive orchestration.

C

With a lot of sub-orchestrations, which divvy up into device orchestration, so use it to manage state for that., there were I think-.

B

There's a couple of other teams that we've actually been surprised.. I had someone new who joined the team about eight months ago and he said," hey you know,.

B

I found your name on durable task.

B

And we've got five different teams using it.", oh wow!, so there's a couple of users out there. We don't know of.

B

But there's number of teams that are using us.

C

It has all been organic., so we've never like. I think this is the first time whatever I don't know., not sure about the thing.. It has been organic just because it was useful,. It was occupying.

A

Can you think of other situations? We.

A

Talked about provisioning and workflows in general, but are there some canonical like if you're in manufacturing- and you have this problem or can you think of other patterns? If you will that lend themselves to this framework.

A

So someone's watching this, they say, "oh I've been struggling with this over here..

A

Maybe I should look into the terrible task. Framework.".

C

Abstractly speaking,, if you have long-running process by definition,.

C

That needs to be stateful, otherwise it is not efficient.

C

because of this long-running either you keep everything in memory and then you wait until the whole thing's done.

C

Right. or you can make a stateful and.

C

The first case, by the way, it's also not reliable if it's in memory, the machine can crash and you can lose your state.. So it's a long-running process..

C

It is probably stateful, which means that you check point at every point in that step..

C

Then you need something like dribble task to make it easy for you to write code for that.

C

right., which actually brings in a lot of scenarios. You just mentioned.

C

Potentially, manufacturing might be interesting, one.

C

Mostly, we've seen lob integration like business processes,.

C

Workflows scenarios like sharepoint has this document workflow scenario..

C

We have a document that needs to be approved by five different people., so somebody writes up a document.

C

It goes to person number one,, they approve it.

C

goes to person number two,, they approve it., maybe an expense report kind of thing,, where the gaps are dependent on a human, actually looking up an alert and saying,.

C

"Okay,, I need to go to that spot." because you can wait for that manual.

B

Process using the workflow. exactly., it could be the three days or weeks or months. right.. That's just a get dehydrated.

A

That's something I've done with durable functions, but not with the durable task. Framework.

A

This gives me a whole new set of things to play. With..

A

Are you still involved on either view on the core team, adding features or supporting??

A

So me not as much as simon and folks.

C

On his team are actively doing.

A

Any major, exciting features and more stabilization and cleaning up defects that are done in the field. Or?.

B

There's some list optimizations and things like that right now, because it's a replay-based.

B

It means that it's going to dehydrate and re-hydrate, and potentially that means it does a little.

B

it's a little chatty to itself and from the compute standpoint..

B

So durable functions has already implemented a sticky concept where it will stack things and optimize. The replay.

A

But the core framework doesn't doesn't implement that yet.

B

Okay, so that's something-.

A

That's an example of something that would be brought in., because I know that you have some concessions that, if I'm intentional about it, for example, if I'm running an infinite loop, that's obviously not going to make sense, because they're.

C

History build, but I can kind of kick off the task as if it's.

A

New, but still maintain the state from the.

C

Previous, so there are those hooks already in place.

C

yes., so it's exactly. another thing that we had been thinking.

C

About is adding inline execution of activities as well.

C

Of course,, simpler activities just having a lambda, for example, just in line.

A

Okay., so you don't have to create., you don't have to go into a new activity.

C

And go through all the machinery., so we're thinking about some optimizations again to simon's point to more about optimizing the flow as we continue to consume it within our data plan, as well.

C

for example, iot hub uses, durable task..

C

We continue to find opportunities to optimize the flow as well.

C

But it's all based on feedback that we're getting.

C

In and generally the pinpoints,.

C

The top pinpoints that are on our list.

A

Awesome, sounds great.

A

well,. Thank you so much for coming out and sharing the demo and your knowledge about the durable task. Framework.

A

I'm going to say that this is.

A

A framework that is used heavily internally.

A

And externally it's open-source and.

A

It's baked into some of our most important services.

A

So, if you're doing anything with any of these patterns that we've discussed,, if you're working on long-running workflows,.

A

You definitely need to check out the durable task. Framework.