Jakarta EE EclipseCon 2020, 29 Oct 2020

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: A different flavor of the distributed transaction

Description

[This recording contains some minor technical issues.]
Transactions are one of the most complex and yet very important areas of computing. They can get particularly hard when the system moves to the distributed environments as almost every component in the distributed system is liable to failures. Traditional locking protocols, used in transaction solutions today, are then very prone to holding locks on resources for unnecessarily long periods. The saga pattern provides an alternative non-blocking solution with the design that allows individual parts of the transaction to be committed immediately and independently. This design is specifically suitable for long running transactions and distributed systems. In this session, we will present a newly created MicroProfile specification called Long Running Actions (LRA) which provides a definition of the transactional protocol and a simple API for the distributed transactions in the Java microservices environment based on the saga pattern. We will show you why the saga pattern is a very suitable transactional solution for many distributed microservices applications and demonstrate the usage of the LRA specification with the live coded demo.

Speaker(s):
Martin Štefanko (Red Hat, Inc.)

A

Hopefully, yes, yes, zoom is making fun of me like always so, okay, I will just really quickly repeat: my name is martin stefanko. I work as a software engineer at red hat uh and you can find me on twitter and they'll handle extra funk. So what we are going to talk about today is called uh saga before we will switch to lra.

A

But before I will describe what the saga is, I want to first compare it to traditional asset transactions to for you to see the difference in this, so you are probably all already familiar with traditional asset transactions, so I will just really go quickly with this and I will demonstrate the concepts on the business trip transaction where we are going somewhere and we are booking a flight sticker hotel room and a car or renting a car.

A

So, okay, most of you, probably know what acid stands for so just really quickly: atomicity, all or nothing property. When we basically want all operation in the transactions to perform or none of them consistency, the system must be always in a consistent state isolation. If we have some transaction which are being executed in parallel, then for uh one transaction, other transaction seems as if they haven't been started, or they are already finished and durability. Just means that all of the result needs to be persisted in some durable storage.

A

So, as you most probably know, behind every s8 transaction, there is usually some consensus protocol and the most common one which is still being used in modern system, is called two-phase comic protocol and because it's the simplest and the issues that are with all contestus protocols are mostly demonstrated in it. We will stick with it, but there are other ones which are slightly mitigating the issues that I will be talking about like paxos or raft, so how we will execute the transaction in to pc into pc.

A

You have a standalone processor, standalone services called coordinator, which is managing the transaction for you, and usually this coordinator will get the transaction description. It will ask all individual services or other applications for the resources that are described in the transaction.

A

The services needs to locate the individual resources and lock them in some form. That means that you will put the flight ticket somewhere else, so no one else can take it in mean time before the transaction is finished.

A

So if all of these locks on the resources can be taken, the individual services will respond back to the coordinator. In the first phase, with the prepared message and in second phase there can be two scenarios: if all of the services responded that they prepared the resources the coordinator will initiate the commit in a second phase, the locks are removed and the usual resources are committed, so they are actually taken.

A

You will pay money for the flight ticket for the hotel room for the car, and you now own the resources- and you just confirmed this information back to the coordinator and the transaction was successful if some of the services cannot be taken in the first phase, so we cannot book the car uh for some reason there are no more car lefts. Then the coordinator needs to initiate the upward message to the individual services.

A

Logs are removed and the resources are freed, so they can be booked by some other transaction or someone else, and basically we'll just return. The failure message back to the initiator and the transaction can be, for instance, retried later.

A

So what is the issue? uh Basically, if we move to microsoft, services, environment and all of these lines are network communication network communications may fail, even services may fail, and if we are finishing the first phase, for instance, and the coordinator fails, then we are holding these locks on individual resources until the coordinator will be able to come up again and it will send a commit or about message to individual services.

A

In the meantime, this flight ticket, for instance, is just not usable for anyone else. So if there are other requests coming for the same resources, so, for instance, this is the last flight ticket on this particular flight, and there are four other requests in the meantime. That would pay the money for it. They are just refused automatically because you are waiting for the outcome of the transaction, and this is where the saga comes in, which will try to somehow solve this problem.

A

The saga pattern itself is not something new and it was actually first published in a paper in 1987 when it described the long-running transactions in a database environment.

A

What it is based on is called the compensations or compensating actions which are basically a semantical undo of the originally performed operations, and what this semantical undo means. You can imagine that if your original operation, like the book flight, is inserted in a database, then semantical undo of this operation is logically deleting this particular insert.

A

However, if your original action is something which is not, which does not have direct opposite operation like, for instance, sending an email, uh you cannot undo sending an email, but because it's only semantical you can send, for instance, a follow-up email which will cancel the previous operation, which was described in the original email.

A

Yes, you are not in the same system uh as you were, because now you have a system where there are two additional emails sent, but uh these compensating actions are always defined by the same person same developer. That is uh developing the original operation. So you know what you did in that email and you know how to cancel it, so you are in semantically, consistent system. That means that, yes, the state is different, but logically is the same.

A

If you are interested to see how this was already applied in real life, there is a really nice application from katie mccrack cafe uh which in which she describes how they use saga pattern in the halo game. I think so, let's take the same example as we had previously and tried to execute it in a saga. So in saga we are directly going into two scenarios: success and failure. So we still have the same free services, but we are now sending the saga directly to the first service.

A

We will locate the resource that we want to perform our operation on and we go directly and pay the money and book the flight ticket. So we are already in on consistent state because we have a flight ticket, but we still don't have hotel room or car rented, and you will just then send the saga to the next service and do the same. Go ahead. Pay the money, take the resource.

A

You are already holding flight and hotel room and you will do the same for the car rental service and if you can also book the car, you will just send the message back to the uh initiator that the all resources have been already paid for, and everything was successful. So you are now in a successful transaction state.

A

However, what will happen if you will do basically the same? You will book the flight ticket, you will book the hotel room and there is again car cannot be booked for some reason. So now you are in a state where you cannot book the car, but you are already have paid the money for flight and for hotel. So now you will just go back in the original order of calls and you will call the compensation actions in individual services.

A

So from current rental service we can send the cancel message, cancel cycle message back to the hotel service and we will just call that compensating action on the performance hotel room and in our case compensation action will be just cancel the booking and we will do the same basically for the announcer links.

A

We will just cancel the reservation or cancel the booking and you will get your money back to the original user and we will again just give the message that saga cannot be executed and you can retry later, for instance, as this can be a little too much information. When you are seeing this for the first time, I will just repeat that failure scenario with individual operations, so in all service we are first executing operation book flight. We are really paying the money, then, in hotel service we are executing operation book hotel.

A

Again we paid the money and in current our service we cannot book the car. So then we will just send that cancel saga message back to the hotel service. Hotel service will locate the same reservation and it will just cancel it, and we will again send that saga cancelled message back to the airline service and the same thing. We will just cancel the flight ticket that we booked in the original operation and we will return that information back to the original caller.

A

So with that, uh we are coming to a quite different model which is called base. So we are still staying in a chemistry, but if we just check back to the acid, so we still have atomicity because we are still getting either all operation performed or none of them or the semantic collections or semantical compensations are being called.

A

But we are losing isolation, of course, because other transactions or other sagas can see that we are in a state that we already booked the hotel, but we still don't have hotel and a car, and we also losing consistency because the during the duration of the saga we are in an inconsistent state.

A

However, we are still durable because you can still persist the results of the saga, and now we are getting into that new model, which is stand uh which is called base and it stands for basically available. This is basically just saying that if you are familiar with capital capitola, then saga values more availability, while as it is more about consistency, soft state. This is exactly that during the execution of the saga. We are not in a consistent state, but what saga guarantees? You is something which is called an eventual consistency.

A

That means that, uh basically, yes, you can be in inconsistent state but guarantees you that you feel eventually get into consistent state of your system, and that is either because you will execute all the operation in the saga successfully or all. Compensating actions will be called for all perform operations in the reverse order, and you will be in semantically consistent state again.

A

There are two different approaches that you can take when you are implementing something like this in distributed system. What I was showing you until now is called the centralization, where you most likely have a saga definition in one place and like we had in our example, and you are sending it between services, but this is kind of against the loose coupling approach of microservices, because usually service a should know only about services that it directly communicates with, and anything else is just too much information.

A

So there is another one which is called orchestration where you again has a standalone service or signal process which is called the saga coordinator, and this saga coordinator is basically managing the transaction between individual services, so in the individual services can enlist with, for instance, analyst with the saga coordinator or the saga coordinator can execute the saga in the same way as into pc, and with that we are getting to our specification, which is called microprofile long-running actions.

A

uh Basically, long-running actions or microprofile lra is uh based on the saga ideas, but is extending it in several areas, and we will see in a minute what I mean. Basically, for now, we are based on http and rest, so you can manage this distributed transaction if you are using uh rest to communicate between your services, but uh more of communication protocol should be coming in the next versions of the specification.

A

The specification is open source you can find it under eclipse, microprofile lra. Any ideas or some comments that we can fix are still welcome. uh We are still not really not released in one o, because we are now waiting for the establishment of microprofile working group, but microprofile 1.0 should be released sometime next month and the currently most pushing one pushed one. Implementation uh of microprofile lra is based in narena project, which is a jta implementation in which we maintained in red hat, and you can find it under this url, and we did that.

A

I would move now to the demo, which we have quite some time to perform, which is uh nice. So I know that this can be quite a few. uh Quite a lot of information when you see this first time, but what is important is to manage these labels that I put into corners. So basically sorry, but uh basically we have lra coordinator. An arena. Implementation is based on. Sorry is based on this somewhere else is based on a coordinator approach, so we have a quarkus application.

A

That is our lre coordinator, and then we have two micro services which I prepared, which uh first one is called order service. I cannot locate it now, but it's written somewhere in there and the second one is shipping service. So basically from our client, I can make a request. uh Basically sorry, http post request to uh order service to create a new uh new order where orders have only one uh product for the simplicity and basically, what we will do in the order service. Slash order, slash create is we'll just look.

A

Some information create a random uid and we will put it into some map and for our and we will call the shipping service and basically in the shipping we just call it.

A

I'm not sharing the screen. Sorry.

A

Yeah sorry, like zoom client, is playing some games. So, okay, I will just repeat it really quickly, so we have our order service which in which we are creating new orders. For the simplicity, each order consists only one product we will generate some random uid.

A

We will put it into some database slash map in our service and we will call the shipping with the generated id, which will just make the call to our shipping application running on a different port and call it with the id and what the shipping application does is just attend some string to the id, and it will just return it back to the order service really simple stuff. We can try it right now.

A

If I call the shipping service, we can see that we created some order and we were processing a shipment for some generated id.

A

This is already a form of application that you would have right now, if you are writing something similar, so how you can add saga, processing or lra processing, on top of it, uh it's basically as easy as adding a simple annotation, which is called lra on top of your annotate. On top of any of your methods in your uh jax rs resource, so basically what this annotation does it's by default.

A

It will start the transaction if no transaction is uh received, so we are not sending any transaction in here, so it will create a new transaction. Before we will enter this create order method, uh then it will execute the method and when the method is finished, it will either finish the saga successfully or already successfully, or it will call the compensating actions. This can be basically controlled by the return status code and we will get into this later so okay, so I said that this is enough to actually execute the saga.

A

So what I will do now is basically put a watch on our lra coordinator, which is returning all known uh lras in the system, and what I forgot is actually put our slight timeout into my order service, because otherwise we wouldn't be able to see it. So if I now will try to call the order service again, we should see that the new lra is started at the coordinator and after two seconds the method is finished and the lra is finished successfully.

A

We are calling it as completed, however, this is just starting a new transaction and finishing it without doing anything else, so, let's actually enlist our order service inside the lra. That is going to be started here. To do that, you need to annotate at least one of your methods. Well, one of your methods. Otherwise one is called chosen randomly with the add compensator notation and basically, this compensate annotation will just mark a method which you are marking as your compensating actions.

A

So here we already are defining our compensation actions on our cancel order, which is a logical undo or semantical undo of the credit order. The uh currently we are processing the order id in some custom. Header actually lra comes with a predefined, uh uh unique id, which you can inject with lra, lra, http context, header, which translates to long-running action, and this is basically how we propagate lra contacts between individual services.

A

So it's in this header- and this is yuri and we can call it lra id, for instance, and we just replace this random generation with the array id which we know that will be already unique, because each array will have a unique lra id and what is really important is that these compensating actions will be called for us by the coordinator when the lra started here will be cancelled. So we can actually ask the coordinator to pass this lra http context into our compensation. Also.

A

We can also inject this as a string. It doesn't matter right now. So uh if I would run this now, we would actually wouldn't see uh this method getting executed, because we are still finishing the lra successfully. If you want to cancel the lra.

A

There are two parameters in the cancel lra annotation, which are called cancel on and cancel on family, which just state individual http response code or response families of the http status codes that should cancel the lra by default, cancel family defaults to 400 and 500 status quotes.

A

So I can just really change this to let's say 500 and just put this into entity, and now I'm basically just saying that when this this method finishes cancel the lra do not finish it successfully and that should get our compensation action being executed, because just by stating this lra method and compensate method, we are enlisting now order service or this particular order resource into the lra. That start it in this method.

A

So let's try to run this now and if I hopefully typed everything correctly after two seconds, we will get our cancel order method called because the lra that we started here was cancelled. Now the compensating action has been called, so this would be the really the basics, and since we have some time still, I will show you also something which we called complete method, which is basically identical to what a compensating action is.

A

It will just get called when the lra is finished successfully, so we will just return 200 again to finish lra successfully, and now we will get the complete method executed. So let's try this. If I run this after two seconds, I should see now confirming order right what this can be used for is basically, if you need to remember something typically order id which, if you don't map it directly to an lra, you need to remember it somewhere else. So in this complete, so you can compensate or run your compensating action later.

A

So you can know that for this particular lra id or lra context, I need to cancel order with some particular id. So in this complete method, if the lra finished successfully, you can clear this information, so you can perform any cleanup actions and free up resources in this complete method. But this is optional. You don't need to use it if you don't need to use it.

A

Okay. The last thing that I want to show you is how to actually propagate this lra context, which we are injecting here into the shipping. So we are making here a traditional juxtas request and actually uh what we need to do here to propagate the request is to add a header, lra, http context, uh header that long running action with the value of the id.

A

So actually we don't need to do this if you are using narayana, because we are including this particular filter, which will add the active id inside to the each outgoing juxtas request automatically. So I don't need to do anything here. I can just go to the shipping service now shipping resource and just use the lra annotation again here. What I will show you here is the value parameter which by default, is this required and if you are familiar with jta, this is the same as transactional.

A

So you will start a new transaction if no transaction is received, but what I want to use here is mandatory, which will fade the fail, the http uh request to this particular resource.

A

If there will be no that http contacts header set, so I will use mandatory and I will also say and false, which will just say that don't close this lra, which will be received on incoming request when this method is finished, because I want to close it back in the order service, and what I can do here is again that header para lra, http context, header, and hopefully this should be enough again.

A

I have cancel shipping and confirm shipping here, so I will just mark it as a compensate change, the name to the http context and also mark confirm as complete and do the same here.

A

And if I now save everything and I will repeat the same request for the order.

A

After creating order, we should see confirm now being called at the both services and also what I can do now is go back to the orders or the resource, and if I change this now again to cancel the array at the end of the processing, because something happened and I will now repeat the request.

A

I should see a cancel message at both services, so that would be really the simplest scenario of the lra. You can do more things with it and all of this information is available in the specification which is available at eclipse, microprofile array.

A

If you want to use this in your application, what I showed you was an led extension which is still in the making and will be available after we will release microprofile specification 1.0 in the quercus directly. But if you want to use it in your application, you need to put the dependency on the specification under eclipse, microprofile array and narayana. You know. Implementation is called marianas lra.

A

With that that will be everything from my site and we will get to the questions, and I see that I have uh some questions so do I have to provide the coordinator myself, or is this part of the framework? No, you don't need to provide it yourself. This is part of the framework it's actually provided by the narayana and actually it's also available in docker, so you can run it by a single command or directly in kubernetes.

A

What, if you have multiple at lra and at compensate? How sagan knows to pair those? So if you have multiple, you can have multiple at lra methods, because it depends which of these methods will be called, so I can have in one resource one at lra required and what one at lra mandatory for instance, and depending in which of this method, I will call the resource will be enlisted with different parameters.

A

Basically, uh if you have multiple ad compensate, then we will just choose one at random, because it doesn't really make sense to have several ones, and you will get some warning when you are starting the application.

A

uh Are there any plans to have a more generic api which could be used to run lra in non-dress scenarios like generic cdi or asynchronous kafka or similar? I guess for things to magically work. The denying protocol is very implementation dependent. Actually it's not that much, because what we are using can be extracted and there are plans to actually make this only based on cdi, and I would like to see uh implementations if you are familiar with reactive messagings, to create something like a connector architecture.

A

So you would put in your lra dependency, also some dependency on the particular connector, which could be http kafka, grpc or something similar. So you would choose what protocol to use to contact the coordinator.

A

How does lra handle your initial case service or the coordinator going down unexpectedly? So actually, uh the coordinator itself is based on uh jta arjuna implementation in ariana. So if the coordinator goes down, everything is persisted in a log store or something similar. We call it object store. So if you start it again, it will get the same implementation from the durable storage and uh basically in the service you shouldn't have any state at all.

A

You will just mark your uh methods and we can then call them because they are persisted in the coordinator and how the coordinator gets the urls of the services that must be invoked or complete the ruleback of transactions. We are doing this in the background. You just annotate your method and we will collect all the parts dynamically and I am out of time. So thank you for your attention. You can find all these slides at this url and there is.

A

I also created the lra tutorial if you want to play with this and the demo that I showed you is at this url. So thank you for your attention. Thank you for your question.