.Net Foundation On .NET, 8 Oct 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: .NET Microservices with DAPR

Description

Dapr is a portable, event-driven runtime that makes it easy for developers to build resilient microservice applications that run on the cloud and edge.

In this episode, Aman Bhardwaj joins Cecil to talk about the what DAPR types of problems Dapr is trying to solve for developers building microservices. We also get some demos of how to use the .NET SDK for Dapr.

[00:55] - What is Dapr?
[06:35] - How big is Dapr?
[09:00] - The .NET SDK for Dapr
[10:50] - Darp's Actor runtime
[17:12] - Demo of Dapr Actors
[23:20] - Dapr's .NET Client with GRPC
[26:08] - ASP.NET Core Integration
[31:45] - Dapr's Open Source community

Dapr
https://dapr.io/

Dapr blog
https://blog.dapr.io/posts

Dapr on GitHub
https://github.com/dapr

Adding a Little DAPR to Your .NET Microservices
https://channel9.msdn.com/events/dotnetConf/Focus-on-Microservices/Adding-a-Little-DAPR-to-Your-NET-Microservices?WT.mc_id=ondotnet-c9-cephilli

#dotnet #microservices #aspnetcore #dapr

A

Hey everyone in this episode of the on.net show we're going to learn about building distributed applications with the dapper framework, so this is going to be really exciting, so I hope you all stay tuned and check it.

A

A

Hi, everyone welcome to another episode of the on.net show and today we're going to learn about the distributed application, runtime otherwise known as dapper. So I have someone here from the dapper team amman who's going to tell us a little bit about what it is and how does it work and how we could use it inside of our.net applications so amman? Why don't you introduce yourself for our folks really quickly tell us a little bit about who you are and what do you do on the team.

B

Sure yeah hi this is aman and I work as an engineering manager on the azure compute team and I'm a core maintainer on the adapter project. Awesome.

A

So why don't you tell me a little bit about what dapper is exactly yeah.

B

So dapper is a event-driven programming model which exposes certain building block capabilities for creating microservices as a sidecar architecture to the user application code. So it exposes core functionalities as bunch of grpc and http apis, which user application can call into and leverage those functionalities.

B

Some of the functionalities which dapper exposes state, store, capabilities, pub, sub capabilities and active runtime built into dapper sidecar secret, store capabilities and bindings and triggering capabilities we'll go into more detail as we as we progress through the show.

A

Sure one of the things I'm wondering is you know I remember the team that announced dapper. I think it was ignites 2019. You know that announced the announced the project along with some other things. What was the main catalyst for wanting to start the project right like why? Why did the team say? Oh, this is a good thing for us to do like we really want to build this.

A

B

So the main idea behind dapper project is to enable enterprise application developers to uh so that they can write microservices applications in a really easy way. Without having to worry about the underlying state, store, implementations or pub sub implementations, it exposes all that in the convenient uh grpc or http apis from dapper, so all developers need to know is, is just dapper apis. They don't need to know about. That is apis. If they are storing in azure it is in redis. They don't need to know about azure apis.

B

If they are storing in azure they just interact with dapper api. Then dapper takes care of all the heavy lifting under the hood. For you.

A

Okay, so that sounds to me, like dapper, provides an abstraction over various. um You know concerns around building microservices and it makes it a little bit easier for me as a developer to use and kind of like what you said. I don't need to know about specific implementations, about messaging queues and caches, and you know traces and things like that. You know but instead like I could use and rely on dapper itself, and you know a lot of those things will be taken care of from it, so to speak. Yeah.

B

So one core principle for dapper is to be vendor neutral platform agnostic. You can run it on any platform, uh use components from any vendor, like storage from aws gcp or from azure, and it's even cloud neutral. You can run it on edge. You can run it in any cloud or even your own hosted platforms. If you wish.

A

Awesome that sounds good. I know you brought some slides. Why don't we take a look at them and see you know some of the stuff that you you know can do adapter sure. Definitely.

B

Okay, so uh these are the slides talking about dapper, which is distributed applications on time. You can know more about it. By going to dapper.io it's an open source project hosted on github. uh The repository is github.com dapper slashdapper, it's dapr and it's a portable, even driven runtime, for building distributed applications.

B

Next, we go into a little bit layering. uh Next, we go into the dapper goals. The the goal for dapper is to build applications using any language with any framework, and we provide best practices for all the building blocks and you can write in any language any framework and want it to be consistable portable apis, and then you can run it on any platform cloud or edge environments.

B

Next, we move move on to the various building blocks of the dapper run time, as you can see, dapper consists of all these building blocks, service to service and location, state management, pub sub actors, secrets and observability, and you can write on any cloud or edge infrastructure.

B

The functionalities are exposed over these http or grpc apis, and then you write your code in any language. Go node, python.net c, plus, basically any language which supports http or grpc invocations, and then you can leverage all these capabilities from dapper.

B

So this slide shows how the sidecar architecture comes into play when dapper is hosted on a kubernetes uh uh communities cluster. So there are susta. There are certain control plane services from dapper which run inside the kubernetes cluster. One is an actor placement service which is used when you're using the actor capabilities from dapper.

B

Then there is sidecar injector which injected up a sidecar. When you deploy your applications to kubernetes, then sentry service is used uh because dapper provides you mtls out of box when you're doing service and location between your two services and then operator takes care of the comp crds deployed to kubernetes uh cluster. These adapters crd, which are deployed like state, store or pub sub components, and the bottom half shows uh what happens on a kubernetes cluster node. This is the user application code.

B

It has a side car running next to it, and application calls into uh dapper run time over http or grpc.

B

Then the right side of the slide shows the various components these are community driven, and these are vendor neutral for resource binding. You can see it can call into event hub kafka, aws gcp, similarly for states. So there are various options available.

B

uh Similar is true for pub sub and for distributed tracing as well, and then these are loaded as a sidecar to your application and you can leverage all these capabilities.

A

Now, as you're going through some of these things, one of the things I've wondered is how big is dapper exactly because, when I see all of these different resource, binaries and state stores and all the stuff that you support, I'm expecting that wow this. I must have a really big sidecar like this, must be a huge process if it supports all the additional things is, is that the case is dapper, like am I gonna have to download, like a gig of you know stuff to run this or is? Is you know how flexible?

A

It is exactly for me to like deploy and use this thing in my application.

B

So dapper is is written in, go it uses, go modules and the side card size for dapper is really small. All these components are loaded in there. You can think of them as statically linked there, but they are not initialized into into your sidecar.

B

Until you tell dapper to do so by by applying by deploying a crd for state store, so the state store for redis or aws would only be initialized in your double side car. When you ask dapper to create a dapper, uh sorry to create a aws state, store or registry store on it, but the binary is statically linked.

A

Okay, great that's awesome and then I'm guessing too, because dapper is also open source, um and you know it's also on github. If folks want to use a particular resource or component or something like that, that's not available today. You know they could always open up an issue or submit a pr if they wanted to to have those additional pieces added into the ones that exist. Oh.

B

Yes, definitely in fact, when dapper started as an open source project in late 2019, it started with only a few of the components available for each of the building blocks and over the past six to eight months, we have seen lots of community contribution to all these components like nats, is one example and all the hashicorp uh components which are added. They all came through open source community contributions. So, yes, you're right people can go and extend the capabilities for dapper.

A

Well, that's great! So then, what I'm hearing too from you is that there's a really rich community that not only exists in microsoft, but there's other companies, like you, said, like hashicorp and the nats folks and and other open source tooling that are contributing to dapper to kind of expand the ecosystem. And yes,.

B

Definitely dapper has a very thriving community as of now.

A

Okay, great awesome, awesome, awesome cool. Let's keep going okay.

B

Sure uh so next we're gonna focus mainly on the dot net side of the things.

B

So dapper uh exposes the apis over http or grpc, but then we are also creating uh developer friendly sdks for multiple languages like python java go and dotnet. So today we're gonna talk about the dot net sdk, so uh in the dot net sdk the apis are made available over http and grpc. So there's a bottom most layer in the dotnet sdk, which is an auto generated code from protoc.

B

On top of that, we create some helper apis for safe state, publish events, uh invoke service or trigger some bindings, and on top of those wrapper apis, we have provided asp.net core integration as well, so that you can directly use dapper by using some annotations or asp.net core friendly way of creating controllers and routers. We'll go into that details. When we go through the demos and then users can use, users can write their application at any level.

B

They can directly drop down to the http layer if they want directly to talk to dapper or http, or they can use the auto generated. Protoc auto generated binary from protoc to make the calls or if they want to use our wrapper apis, they can hook into those and then write application from there and dapper also provides an uh active run time. So dotnet http provides a uh sorry.net. Sdk provides an active runtime layer which is interacting with dapper. Over http and users write their reactor application using this.net uh actor runtime.

A

And yeah, I'm particularly interested too, to learn about the app the actor um implementation of dapper, because I find like every platform and ecosystem has its own implementation of actors, and you know with looking at dapper and seeing how you know it's it's somewhat language neutral.

A

You know I mean being able to have an implementation that does you know a lot of these patterns, including actors, in a way that I don't need like a.net sdk or I don't need like a python or a go sdk, and we can all use actors together in in a very distributed way, is very interesting.

B

Oh, yes, that that's definitely true with dapper. If, if you look at other.net actor implementations like uh service fabric, reliable actors or orleans or net uh they, you can write your actors, but but you can then invoke only through dotnet clients. What happens with dapper.

B

The core functionalities of an active runtime are brought into your dapper sidecar, and then you implement your actor actor code in your language, specific uh code of your application and that personality all the active functionalities one benefit with dapper is that now you can write your actor app in in c sharp, and then you can invoke it over http. That means you can invoke uh your actors from any language which supports http, awesome, great awesome.

B

So next we will uh go into details of uh the actor model with dapper. The dapper follows a virtual actor model uh with service fabric, and all these also follow. So it came from microsoft. Research a while ago, then orleans and service fabric productized it and dapper follows the similar virtual actor model.

B

uh So uh the dapper actor model provides uh falling functionalities. It has a distribution and failover. It provides turn-based concurrency by turn-based concurrency, I mean uh actors are single-threaded by definition, so when one call is going through a through an actor, uh other calls will have to wait until that previous call finishes, so that solves the problem of concurrency their developers. Don't need to worry about multi-threadedness in their actor code. It's just single credit code running in there got it.

B

Then it also has an inbuilt state management uh so using the rich apis provided by dapper sdks, they can manage the state for a particular actor instance, and dapper takes care of saving the state into a configured state store.

A

B

A

About actors there too, so I know there's a different actor like we mentioned before: there's different actor frameworks, even within the.net um ecosystem and I've heard different terms, and I'm I'm trying to like. I haven't built anything with actors before, but I'm kind of curious of what about what somebody's terms mean. So, for instance, I've heard about reliable actors, I've heard about virtual actors and then there's regular actors, I'm guessing there's a regular action like is there like like what? What exactly does it mean to be a virtual actor in dapper?

A

Is that does that mean something different versus like a regular actor or a reliable actor? Okay.

B

So, in a virtual active pattern, what what you do is you don't need to create an actor explicitly? You just make a call for a particular actor and if one time detects that that app that actor doesn't exist, it will go and create an instance for an actor for you and call a method on it, whereas in normal, like you would go first, create an actor, then call a method on it. In this case, you don't have to worry about whether an actor exists or not when it comes to reliable actors.

B

So reliable actors is a specific term used for service fabric actors, uh because service fabric has this concept of reliable services, and then reliable actors were built on top of reliable services, so they just inherited the term reliable actors, so reliability over there means that the state is assisted and replicated within the cluster itself. When it comes to service public.

A

So so reliable access within dapper is. Let's imagine. I have a cluster of machines and you know different um different dapper side. Cars spread across these different machines, I'm guessing I can make you know I'll invoke some api against my sidecar and it'll go ahead and generate or create an actor on someone on the machines. Like I don't know where it is. I don't create it. I don't you know, I don't knew it up with a constructor myself.

A

You know within that dapper cluster, within that dapper network it'll go ahead and figure it out right and now I'll be able to interact with that thing wherever it is.

B

Yes, that's the idea so from a from a client invoking an actor perspective, you will just create a proxy and say hi. I want to make a call to this particular actor id and dapper runtime will figure out and which of your 10 instances of your application. It needs to instantiate that actor. It will go and instantiate that actor in that application container and then call the desired method on that.

B

And then, if your actor is saving state or modifying state, it will go and take care of that as well by using zappers underlying state stores. Got it awesome. So this uh this shows your actor uh hosting within your application pod. um So let's say you create a video game, enemy actor and then it has a bunch of methods which are exposed over it. And then these small circles are basically each instance of this actor.

B

So dapper takes care of distributing these instances across multiple hosts and parts of your application and it takes care of failover of your application parts. Let's say you are this: first part crashes or dies and then another part comes over.

B

Dapper will take care of uh moving uh actin instances from your host, which has just uh crashed to a new host so that you don't see a downtime for your actors and it does that by using the placement service. So it takes care of scaling your acting instances across multiple parts of your application.

B

So you can scale out your application. Let's say: if the load on your actual service increases, you can add more instances of your active application and then double runtime will take care of automatically distributing those actor instances to new instances. As you add them,.

A

So that sounds pretty good. Do you have any any demos that we could take a look at like I'd, definitely love to see what it actually looks like for net developers. Oh.

B

Definitely I'll I'll start with uh actors, since we just talked about them. uh So when it comes to c-sharp player, uh what a developer has to do for writing. An actor is pretty simple: they just define an interface which derives from my actor and they set some and they define some methods in that interface.

B

In this example, it's defining two methods, save data and get data, and then they provide implementation of those two methods. So for providing an implementation, you define your class uh deriving from actor. This actor is the base class provided by dotnet sdk for dapper. It contains bunch of functionalities which are available to your actor, and then you implement your I demo actor interface and in the implementation. You can see it's implementing these two methods called save data and get data for saving your data.

B

Let's say this method receives the data from the client side called my data, and then it interacts with the underlying state store using the state manager apis. So all they need to do is save state using the state manager and then active runtime will go and and and save the state using configured state providers. It could be reduced or it could be cosmos whatever you, you configure dapper, to run with right, then.

B

Similarly, for getting a state for your actor, you get the state from uh state manager and then you return it back to your client sites. So when it comes to creating client sides, this is how you create the clients. You create a proxy, you tell the proxy hey, I want to talk to actors of type demo actor, and then this is my actor id. So in this case it's going to make a call for actor id called abc, and then you get this rich interface.

B

When you use that client sdk to make the calls, you can just say proxy dot, save data, and it's and this proxy is created for this interface. So you get all the intellisense here to invoke those methods.

A

I know with some other actor frameworks, there's like additional hooks into like the actor lifetime and even handling errors and exceptions, and those types of things does there's um this dapper at least the dot-net sdk for dapper have any of that type of stuff in it. Or would that be more so something that the user has to take care of dapper.

B

Itself takes care of the actor lifetime, so it provides a bunch of methods like activation and deactivation hooks, which are called into into into the actor application when dapper decides to activate an actor and or when dapper decides to deactivate an actor. So let me go into little bit. It is what activation means. Let's say a client made a call to dapper for a particular actor. If it doesn't exist, dapper will instantiate that actor and then it will call and activate method on that actor.

B

That activate method can be used to prep something for that actor so that when the method call is invoked, certain states are already made available to it. It could be fetching some initial data from from from a remote store. Let's say you want to get some data from event hub or from your state store. You can grab it from there and similarly, when dapper decides to deactivate an actor, uh the actors are deactivated if they're not active for a for a particular time.

B

That time is configurable by default, is one hour so actor so that runtime calls the deactivate method on the actor instance. So that gives you the opportunity to do some cleanup. Like let's say you created some long living resources, uh you get the opportunity to clean them up after your acting is deactivated.

B

um So next we will uh run a small actor application uh using dapper um cli. So debra comes with a with a cross platform cli. You can use it to develop applications on mac linux on windows, and it sets up your default uh development environment using dapper. So the way you we use setup is you get the cli and then you run dapper in it. It would download the dapper runtime and set up set up some bayer minion development environment for you and then you can use the dapper cli to run your applications.

B

This is an example of using dapper cli to run your application.

B

The format is, uh the command is dapper run. It takes an application port uh since dapper needs to talk into your application, so it needs the port for your application, and this is the port for dapper runtime, at which it will start the dapper runtime, because dapper will call into its sidecar apis over this port over http and then there's the command to launch your application.

B

So I'll run this application. This is, and this is going to run the same actor application, which I just showed you and that per runtime will go and and start creating actors in this application. When I make a client call to it, then we're going to make a client call to it, as I mentioned, that you can make a call over http.

B

uh That means this actor can be invoked from from your java go or python code or even from a web browser anywhere as long as it has direct access to this cluster okay, uh so this is making it making a curl call. It's just saying that hey.

B

I want to invoke this actor abc of this type demo actor, and I want to invoke the same data method, and this is the data which should be given to the actor instance whenever, when the method is executed, so I'll make a call for it and the call went to the dapper it saved the state and then now I'm gonna invoke this get data method, and this is gonna, go to the state manager and then retrieve the state and return it back to the client side.

B

So when I make the call I went to the state manager received the call received the data and then returned it to the client side.

B

So that's that that's about actors. Next, I'm gonna show you about the dotnet apis, which we have created on top of auto generated grpc apis from protoc.

B

Files, uh so dapper provides a way to uh call so dotnek provides a way to call into dapper run time and perform certain operations like uh publish an event. Save state get state from the dapper run time, so the usage is pretty easy.

B

It uses the builder pattern, uh you create a client, say dapper, client builder, you give it some json sale as an options, and then you said build so after that you will get a client to interact with dapper, and then you can see bunch of apis available to you to interact with dapper runtime.

B

So it has much of state apis like managing the state gets the delete state. It has apis for secret management so that you can retrieve your secret in your application using deposit card. Similarly, it has methods for invoking your binding and invoking methods in your target service.

B

So all the user, all you all, as a user you have to do- is create this dapper client and then invoke these methods in your applications. It's become really easy to save a state. You don't have to worry about knowing any of the sdks from any of the cloud providers or from redis or from or from kafka from or from any of the components you just use these apis and rest assured that your state would be safe saved in this configured states too,.

A

Nice- and you know just for for folks that are watching, so this is kind of like a nice to have kind of thing for dapper because, like you mentioned earlier, if I wanted to just interact with the dapper sidecar with http, I could do that or if I wanted to use grpc directly, I could do that, but now I also have the ability to use like this. um This dapper client builder, to you, know crafts.

A

Like a you, know, a a um a strongly typed clients right, it's an adapter runtime and I could use that as well. So you know, depending on how I want to write my code like I have options to go one way or the other.

B

Oh yes, definitely, and it gives lots of flexibility to developers if they want ease of these strongly typed apis. They use it here if they want to drop down to the http layer where they want to manipulate the metadata given to dapper or manipulate all the http headers which are given given to dapper. They can control that by using by newing up their own http client and then leveraging all the functionalities available on the http client. There.

A

Great great yeah, this looks pretty nice.

B

um Next, we talked about asp.net core integration for dapper apis, so one goal for dapper is to integrate with all the different language uh application development frameworks which are available when it comes to java it integrates with spring boot when it comes to dotnet. uh It indicates with the asp.net goal so that certain functionalities are available to you without doing bunch of plumbing yourself got it so over. Here, I'm going to show a very simple controller for asp.net core.

B

What it is doing is it's a very not so secure banking application. If you want to locate it that way, it has a get api controller and then it has a bunch of these post controllers. What you notice here is that it has this annotation called topic, and so this is a very dapper specific annotation over here. You are saying that hey, I'm interested in events into this topic called deposit and whenever an event happens in the configured pub sub store of dapper, go ahead and invoke my this controller method.

B

So whenever event is published, this this controller method will get invoked for you. So it's.

A

B

Way to try into dappers uh pub sub system without knowing much about it, uh so under the hood it takes care of all the plumbing needed to interact with dapper to tell dapper hey. These are the topics I'm interested in. It takes care of that it even accepts the events when they come from dapper and then calls into your controller method.

B

Similarly, it also shows interaction with dapper state apis uh using the rich aps, which we just talked about over here. You can see uh that you get the state uh from uh dapper client. The client is the client we just created in the previous code, which I showed uh very similar to this. It's basically doing this under the hood. It's creating a dapper client and then using that you can get the state information from the configured state store, and then you save the state.

B

So now immediately your controllers become stateful uh by what you have dapper running against it, and then you can you, you can save state and retrieve state.

A

Nice and like what you said before it looks like this method is doing two things, one. It's it's subscribing to a particular topic. It's saying: hey, I'm interested in in this these particular messages and until it also handles it as well right, so you know, there's a pub. This publish and subscriber is happening in the same almost in the same place.

B

Yes, true, that's true, and so this is one example of how you integrate with controllers, I'm going to show you another sample here uh which basically interacts uh with the spinet core at the at the layer when you create your routes. Basically, so when you're creating your routes, you can tell at that point also hey. This is my post endpoint for deposit and then um use use a topic called deposit to configure this to be invoked when, when an event comes for that topic, so.

A

Do I have to use both, or can I use either or.

B

Oh sure yeah, so it depends on on how you have written your application if you have written it using controllers, you go through that that route, but if you have written your application uh by creating routes, your custom routes. Here uh you go with this this approach, so it really depends on the application design.

A

Yeah, okay, that makes sense.

B

So next I'm gonna just run uh asp.net core uh service. The controller service um using the damper cli is just gonna host that asp.net core application and launch adapter cycle next to it, and then I'm going to publish some events using dapper cli, so this is dapper cli. You can use it to publish some events to dapper. This is really convenient way to test your pub sub and location in your service using the apple cli when you are developing locally.

B

So this is going to publish an event to this topic called deposit and when this event is published, what's going to happen, uh the event will be delivered to your asp.net core application and that controller would be involved for you so that that's pretty much it is showing. So it published the event. And if you look at the logs of your uh application, you can see that hey. I got this event and I'm executing this deposit method here. Basically, it showed you uh this.

B

I printed this this call here and then, if you make a get call here to your controller, you can see that uh the amount which you have saved into your underlying state store uh that can be retrieved from the controller. So now the balance is 30. If I publish one more deposit event- and if I query it again, you can see it added that 15 to the state and it's configured in the underlining states too. That means, even if your application crashes, it comes back up again.

B

uh The state would still be available to you, so your controllers just suddenly became stateful.

A

B

Yeah, so that that's all I had from the demo perspective today.

A

Great, um so for folks that are interested in dapper, um I do you have any links or anything like that that you could share to show us well, how can we get started like what are some of the things that we should be paying attention to? How can we reach out with to the team and those are.

B

Sure definitely so, uh since dapper is an open source uh project, so we we rely on help from the community both for uh feature requests, contributions and reporting issues, so we are available on github, twitter, youtube and and and get our channels, and these are the links which are available uh next, I'm gonna show uh some of the uh github repositories which we have so this is uh the core runtime dapper repository uh it's github.com dapper, and then this is the dotnet sdk repository.

B

The entire code base, which we just looked at, relies on this repository as dapper slash dot net dash sdk. Similarly, we have a components contribution, so this is the components contrib repository which contains all the components which dapper can use. uh If you look at state, uh you will see all these components which have already been contributed by the community.

B

So these are the main repositories which are available for the adapter runtime. There.

A

Right so, as folks are checking out the project and they want to again send feedbacks and issues or or maybe they want to even contribute their own component like this is where they could go and do that and then they could talk to the team either on the getter channel or you know, on one of the community calls if they have anything they want to share that way. Yes, exactly aman! Thank you so much for coming on and talking to us about dapper and showing us that.net integration.

A

You have like I've been playing with dapper for a little bit, but I feel, like you just taught me a bunch more stuff that I haven't even tried before, and so I'm definitely excited to go ahead and play around with it a little bit. um You know what we'll do is we'll make sure some of those links- and you know the samples and those types of things that you share, we'll put them in the show notes.

A

So you know if folks again, they want to interact with you and the team and stuff like that, that you know they have a very easy way to do that, um but other than that, that's it. um Thank you for coming on and thank you. Everyone, that's watching for checking out this episode of the on.net show, and today we learned about using dapper, which is a distributed application runtime to be able to create microservices using our.net application code. Thank you all for watching and take.

A

A