Elsa Workflows Community, 5 Jul 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Elsa Workflows Community Meeting 16 ( 2022-07-05)

Description

Meeting 16
Topics
- Elsa 3 updates
- If, Flow Decision, Flowchart
- Fork / Join / Parallel execution
Data processing pipelines

A

All right and we're light so I think for this session. It's probably going to be very short: I don't have too many updates. It's been busy I, don't know if you've got anything uh Mohammed France France is here. Maybe it's going to be a long meeting. After all, that's.

B

A good introduction to me thank.

A

You good evening.

B

A

B

This being recorded already or yes,.

A

Yes, we're on it's. Okay, it's live. Okay,.

C

A

I only mean it in the best possible positive way right, because we have.

B

Yeah wonderful decision.

A

Indeed, yeah so as I was saying with regards to topics, I I don't have too many topics for today. Unless somebody has something they would like to discuss that. That's perfectly fine, and maybe, while you think about it, I will just jump.

B

A

In and talk about, my little updates with Jesus follows just recap: from last week, I showed some updates with regards to the workflow designer I mean actually, and if I recall correctly, it had something to do with composition, so you could compose workflows. This is not a good one like this, and you can. You can drill into the activities that expose activity type properties like for eats like.

C

A

And then you can configure what you want to execute in the den brands or in the else Brands, if the condition is false, so straightforward, I think a couple of meetings ago. We also talked about flow charge style activities right because these are called sequential. If you will they they're composable. But if you have a typical flowchart style diagram, then you will want to be able to draw lines between activities like between the right line and then the F in this example, and instead of drilling into the if activity.

A

Maybe you just want an true and a false outcome and then you can connect it through or in a false outcome. So I think we want both kinds of activities so that you can wrap nested flowcharts within an activity and in other scenarios you want to orchestrate it using more of a floater style.

A

So for that we have a that's what I've been working on the past week is we have now a new category called flow that includes a flowchart which we've been looking at you all the entire time, because this canvas is a flowchart activity. But now you can also drag them onto the canvas, and now it acts like a container. So you can have many subflow charts if you want so you can click on edit and then start creating subflows and then it can of course be nested, and here there's little red Chrome Trail.

A

So you can navigate up the hierarchy, but that's not the only thing I've been working on I've been working on the flow decision which is in elsa2. This is the if activity and an also 3D effectivity is this styled activity with the then and else activities.

A

But if you want a flowchart style decision activity, then you use the flow decision, and this allows you to, of course specify a condition here, for example, true, which is always true, of course, and then you can connect either this one or if the outcome is false, connect to another activity. I want to do this also for the switch activity so that let me clean up this mess for a second, so the switch activity is similar to the for each and, if with regards to it being a sequential activity.

A

So it means that for each of its case, you drill into the activity and in this Define the the inner flow chart from a nested designer. So let me let me first show you what that looks. So, let's build out a few cases case, one two and three, the X variable doesn't exist. This is just for channel purposes, so here, as you can see for each case, we have well these cases displayed on the activity in the designer.

A

You can't currently drill into it while you can, but it's not actually going to be persisted because I'm still working on that, but you get the point right. So this this is a different way of composing your workflow, but this may not be what you want for certain scenarios. Instead, you will want each case to be an outcome of the activity, so for that there will be a flow switch. That would look similar to the flow decision. So instead of true and files, you would have case one two and three as as outcomes, but.

B

When you added the cases it showed them already like this, did you see that yeah.

A

You mean these.

B

Things no, but if you can, you add a new switch to the canvas because I have a question regarding that. But if you add new Now new cases to this I guess you see yeah and another one now you have case one and two also has flow points at the end. Oh.

A

Yeah yeah don't mind this. This is uh this is a bug.

B

Oh I don't find that the bug I find this quite interesting, because I was just wondering about the if and the flow decision. Basically in the implementation or the Json representation of this, there will not be a difference if you do the flow decision and if basically, there.

A

Actually will be a difference in the Json okay.

B

A

Using the flow decision, the connection between the true outcome and some next activity and, of course, connection between false and some activity will be stored in the Json document as a connection object, and it will have a as a source activity, the the flow decision and as a Target in this example the, if and the name of the outcome in this case false so that will be stored in that manner. But when you use the if activity here, the property of that will actually actually be stored as a field on the if element yeah.

A

So it will be represented differently as well, and I can actually show you in the well I can try and explain it through the the object model. So here we see. If so, if you would for a second imagine this, if class being serialized into Json, you can see that it has a then property and an else property and they are type activity. So this would actually all objects in there. So this would actually be something like I, don't know, maybe a right one yeah so yeah.

A

So it's it is different and it will only work. These kind of activities will only function properly within the context of a flowchart, so the flowchart is the the activity that knows about flow decision.

B

Because I was wondering if we could use an endpoint on just on the then and else on that little square in the Middle, with the plus that you would have a blue dot there and connect it from there that you don't have that difference in the flow decision and having to think well. Why do I need a.

A

Flow decision or.

B

A

If yeah I I was I was thinking the same, but I wasn't sure how that would be done elegantly, because what we're taking the back end to to have these two Notions, so one way could be to to do it. What I was originally attempting to do, which is what the transposition or the transforming right so.

B

A

Couple of meetings ago, I tried to explain what would happen if you connect and this one but like this, so what I would do before sending this to the back end. I would basically take this activity and store it as a property on the flow or on the if activity and then yeah but visually it would be different, but there there's this, it's it's a different representation we talked about. Maybe it makes more sense for the designer to also you know, reflect the actual composition.

A

The same way Windows workflow, Foundation 4 does speaking of which they too have two different categories. One is the sequential set of activities like if or they call it, they'd probably call it something else, but it works.

C

A

And then they have flow nodes and those flow notes only apply to flowchart activities. So I.

C

Think yeah I think both both types of activities are useful and there are use cases to choose one over. The other I think the good thing now is that we don't have done outcome from the full flow decision, so at.

A

C

This way, it is more logical that we only have true and false outcomes, so that will help us maintain the flow chart, type and kind of flame.

A

Yeah and that's a very good point, because in Elsa 2, a big issue, I think is, is the fact that the F activity does have three outcomes. True false and done, and done will always be executed, but only after either the true or the false branches complete.

A

But it's a it's a kind of a weird, a weird thing, because it tries to model composition in a way but which allows the user to make weird connections, for example from for each Loop, for example, back onto some arbitrary node in the OR activity in the flowchart, where it wouldn't make sense, because that Target activity might live on an entirely different level than this hierarchy of activities. It's not visually represented in that way. So it could lead to very strange results. Yeah.

B

Yeah, it's a fair pointed that um was indeed a bit confusing at times. So this is quite nice, I'm really interesting it in the use cases where you opt for a flow decision or composite tip when you would opt for one or the other yeah yeah.

A

B

Really sure what.

A

So their use cases and I will try to think of an example. Nothing comes to mind right now, but I I've worked on workflows for a customer where sometimes in the same workflow day, if activities should only have a true and false outcome, or only one of those or only those two would be used and the done would be ignored. But in many parts of that same workflow there was the done outcome was actually used. So we wanted to have actually this effectivity with the then and else and then the done outcome.

A

This is basically the done thing here, but it's modeled at a in a different way in this workflow for good reasons, because air to me at least this logically means when this activity executes or evaluates its Boolean condition, then the then or the else Brands will execute and no matter how complicated or how long running the nested activities are at some point it will finish presumably, and when it does, then it will continue on to the next node here.

A

So this more closely reflects that what I wanted, instead of the workflow for this customer I'm, mentioning where I would have to follow the Dan Brands and the else Brands and then back to the done Brands and it's and theoretically I could even connect somewhere from the the true Brands all the way to this activity here, which would be weird so the long story short there are use cases. But um let me try and come up with a few good examples. Yeah.

B

I think your explanation is quite quite clear already. If you look at the flow decision, then you have the true and false, and you can basically get the same results by joining the true and false execution lines at some point again together and then you would have the same as the if implementation right now, but then you have to think about okay, what are my endpoints in those flows and you have to all join them and then join them with one off, and this makes it really more clean.

B

If you want to have the execution of then either else and then continue so yeah for me, it's clear that it's and they serve their own rights in the in the implementation of workflows.

A

Great yeah yeah for sure so this is this is an example of merging back into this Brands here, of course, this is not a fort. This is either true or false, and no matter which one excuse here. It continues with the switch in this example, and if you have many sub Francis, then then you want yeah, then you would potentially have to connect it back to um to the switch. If you have multiple activities here, it could even be other other.

B

Decisions yeah.

A

Exactly yeah yeah, it's worse uh two, it it completely depends on on what you're implementing I guess: yeah, okay yeah. So um so that's what I've been working on in terms of API. Maybe it's interesting to take a quick look at that as well. How would you implement it as a activity? Developer, I I mentioned that the flowchart activity knows about the if or the decision, the flow decision, but that's not entirely accurate.

A

The way it works instead is as follows, and let me show the flowchart activity first, all right, just taking a second to remember what I did here: oh yeah, okay, so what the flowchart activity actually does. It has a function that is invoked whenever a child or descendant activity completes. So this could be. This could be activities that are not scheduled by this activity itself, in which cases it will be ignored. So it's gonna see if the completed activity was scheduled by this activity. So if it's a direct child, then it continues.

A

If not it exits. If it is a direct child, then it's going to check for the completed signal to see if its results of type outcome and this outcome is a class that lives in the in the domain of the float. So only flow nodes are aware of of the outcome class and if those flow nodes complete, they will complete by setting a result of type outcome, and the outcome class has points of oh just one property, just the name of the outcome. So let's take a look at the flow decision activity for a second.

A

So this is a very simple one. It will have its condition evaluated when set, although I'm probably gonna, change this, so that the activity itself will evaluate the condition here, little sidebar there. The reason for that would be to allow the activity to execute multiple times, because there's there are some scenarios. I am blanking on what those scenarios are, but where you want to re-evaluate the condition, I think it might be in a in a loop or so there was some scenario: I will get back to that.

A

But anyway, when this is, if I evaluate it, it's going to set an outcome to true or false and use that result when completing the activity. I think so here this activity is explicitly completing itself. So this is a New Concept in Elsa 3, where activities they need to signal to the system that they are completed, because there may be activities that will start, but they will not complete until some condition occurs.

A

This could be waiting for a signal could be waiting for user interaction or even just waiting for its children to complete at some point it will complete in this case this will complete immediately and then we yield back control to the flowchart activity and it will check for its outcome and then, depending on the outcome name, it will find the next connection to schedule. So that's what it's doing here.

A

It's finding the connection to schedule next, so this would be an activity connected to the true brands or the false brands for activities that are not aware of the flow chart activity. They will not even use the outcome so by default. That connection will be using the well quotes down on outcome, but that doesn't really exist in the from the point of view from regular activities like like Finnish, for example, it just completes through the well through. This is a bad example.

A

The read line activity it inherits from activity which completes by default, wants to executes finishes, but it doesn't schedule or tell the system that it's completed using the done outcome or anything. So that's just a default thing which, in my mind, is makes it more flexible compared to the flow chart activity in Windows, workflow or because there, if you create a flow chart activity, the only types of activities you can add to the service are flow nodes activities, if I'm not mistaken, but that's that's what I think I. Remember of that.

A

In contrast to also three, you can add anything to the flowchart and certain activities. They are aware of the floater, so they allow you to provide additional outcomes other than just the done outcome that that we see here.

C

I have one question about ports, so do you allow multiple connections from the same port as an outcome, for example, out of the true, can you connect to multiple activities, or it could be only limited to one yeah.

A

It's um undesigned and undecided.

C

Really at this.

A

Point I, don't know what I did I broke something here anyway, let me do a new one, so good question I think we talked about this a couple of meetings ago, yeah and I think the jury is still out so right now you are able to do it. The designer does not prevent you from doing this and it will actually it will. This will work, so this will cause both right lines to execute, but it also means that this right line here will execute twice.

A

So what will happen is both active exit and then because those two connections will be scheduled as well. This right line will execute this twice as well and from a user's point of view. This is probably not what you expect. What you probably expect is this one to execute this one to execute, and then this one to execute just once, yeah.

C

Yeah we discussed this about the implicit fork and join right.

A

C

You know we want to like implicit or explicit, so that is yeah.

A

Exactly so, I haven't decided yet if anyone has any any good suggestions on this topic, I'm all open for it, of course, so one option would be to introduce a fork and a joint activity and then disallowing this. These multiple connections being drawn out of this uh this outcome here and then also having an explicit join to merge, spec executions.

A

That will be the simplest approach, but it also adds more noise to them to the workflow right, because, instead of just having these for activities, you would now also have a fork and a join, so that will be two additional activities. The good thing about this yeah.

B

We talked about that that we could have when you have multiple inputs, that you could have an extension on the properties that you could say there wait any or wait all exactly and it wouldn't add it to the workflow, but I'm also not sure about what to choose here, and you want to have that flexibility, I guess.

B

But you don't want to have the healing that something magical is happening when you're not aware of this situation that you can have multiple tokens running in the same workflow and then basically splitting the work into our in parallel and having things executed twice. That's yeah.

C

It's not what you expect.

B

C

Looking at the code that you're showing right now, I think you are scheduling one child at a time. That's.

A

Right yeah, so when this completes the flow strategy, we'll see there are two connections, so it's gonna schedule both connections. So that's this code here so, as you can see it's it's getting all the connections that match the this query, and this query is about getting all the connected the outbound connection. So this one completes so it's in this one is a connection and this one is connection because they both have this Source, where the source is the completed activity and then yeah.

A

Of course, if we change changes to first or default, it will only execute either this one or this one, but that's not expected either. You would expect both to execute, but then here you somehow expect this to be executed once, although, but logically it to me, it makes perfect sense that this would be executed twice, but I can see from uh from a user's point of view that it will be unexpected. My.

C

Current feeling is that to disallow the multiple connections and have the explicit forecast join, but maybe that will complicate stuff.

A

Yeah, maybe I'm tempted to go with that as well, because that will keep things simple if we disallow multiple connections and and then just introduce fork and join, which would make things very explicit. But before making a final decision, I would suggest. Let's, let's take a look at some of the other engines out there, like n8n I, think that's the correct term file for workflow automation, tool and and I believe they too allow. As you can see here, multiple outcomes, so I'm I'm just very curious to see what would be the behavior.

A

If you would then connect this one to the. If here when would, if then execute twice or not? So let's see what tools like nhn to uh in this in this scenario, but.

C

Normally, when you do this, I think you are expecting things to execute in parallel. So in this example, you are passing the data to Google cheese at the same time, you're writing it to database. So it's two things that you want to do. Yeah, I, don't know it's it's it's not that sequentially! You expect to execute one by one. It's something that you want to do it in parallel and then maybe it doesn't matter I, don't know really yeah.

A

Maybe so this, of course this is her and executes in parallel, although.

C

A

It really in parallel, or is it gonna ex schedule, one by one and making it seem like it executes in parallel? As far as I know, most workflows make it seem like it executes in parallel, but doesn't really, but even so, even if this is a true parallelized execution, would that then, would you also expect if to be executed twice, so maybe it doesn't matter or whether it is truly parallel or not logically or functionally? It is in parallel.

C

A

Yeah, so that's what I would I think we should do is to stay. Let's take a look at this one, and maybe one more and see what see what what solution they offer and learn from them. Yeah. It.

C

Depends on the type of the workflow if it is a kind of a data processing pipeline, then you really don't want things to expect multiple times, because then the outcome are not being desirable. Yeah.

A

But in that case, how do you determine yeah and then somehow they would know that both should be executed, but somehow this one should only be executed once then I would say no because they may also be inbound connections like from some other source Brands, and then in that case you do want. If the effectiven to be executed so uh yeah yeah, so we'll we will see what what we come up with and.

B

Maybe maybe you can do something in the UI, then that when you see that you have multiple lines there, that you can either right click and say like wait any or wait all that it doesn't clutter the workflow but still be present there yeah.

A

And then the default should be wait well, wait and it's also not gonna work, because what you don't want to have what you don't want to happen. Is this activity executing and then continuing while this one will then execute next and then somehow? Well, let me just put it this way and Elsa to the way the joint works is as follows: whenever an inbound activity completes and executes, then the join activity excuse and what it does. It checks to see what was the inbounds activity or the connection and what activity connected to it?

A

And it's going to record the idea of the formerly the the activity that executed before and then depending on the The Joy mode. It will either continue or it will wait. So if you say wait, oh it will wait. It will not yield to the workflow Runner. So it will it's not blocking either it's just just gonna, not it's gonna cost the next activity not to be scheduled.

A

So whatever comes next and then, of course, this one executes will be recorded and at that point the internal collection will contain all of the activity, ideas of corresponding to the inbound connections, and then it will execute. But if you, um if, let's now say we don't have a joint activity, we have just the right line activity which itself doesn't have any logic regarding joins. So it needs to be somehow implemented by the flowchart activity.

A

Yeah I'm just wondering how that how that would work nicely, yeah I, don't know, I need to keep it some more thumbs.

B

Yeah, it's a tricky one uh discussing all these things. I I, think the explicit fork and join makes it really clear about how it works, but indeed it's cluttered, but but for me knowing now all those scenarios I don't see that you can't that you can avoid that yeah so tricky because, like you said, do you expect them to be running parallel and not continue? If you have a fork and join, then it's really explicit.

B

Well, it runs in its separate synchronization and it doesn't synchronize with all the first activities of any fork and do that first and then the second one until it's reached the join in in this case, you expect that actually to happen, and that's makes no sense from the point of view of execution or workflow.

A

Yeah, that's what yeah yeah so yeah I'm leading towards that solution, but I want to I, won't see what the way it behaves in in this uh flowchart here, because if let's say that this, let's say we connect postgres to if and it turns out that if it's executed twice then to me, that's confirmation that it's, it is a logical effect which to me also would mean. Maybe we should not disallow splitting the workflow like this.

A

But if you want control over merging, then we need to join activity and the fork activity because other six, maybe we need to focus, maybe, and maybe we only need to join activity to properly emerge into uh you know a single activity so that you can control the behavior because maybe in some scenarios, it's okay for this to be executed twice other scenarios. You don't want that to happen, and then you need some mechanism to control that and that could be done through the joint activity.

B

What's the current execution of uh what you said is, and they just select everything that comes out of the right line in this case, and they are both scheduled right, yeah, that's right! um That's it first execute the full first one. That's.

A

Right yeah so the way it happens it goes in and then let's say this one is the first connection it finds. So it's going to schedule this one, and if this one is let's say an eighth or maybe the next one is an HSP endpoint. Let's say this, so this is a blocking activity. It's not going to schedule this one. It's going to be waiting here.

B

uh Yeah yeah, that's the other question. I wanted to ask like put a read line there. When does the other one get scheduled, then yeah.

A

This one doesn't get scheduled until this one completes and then this one executes and then it's the end of there are no more activities in this brand. That means the the flow chart activity. The outer one will now look at the next connection and that will be scheduled or actually this will already have been scheduled, but it will only actually it wants this one and it's done, and then this one will catch its turn. There's.

B

Of course, what could be different, but would it be different if you would have the fork activity in between or is that actually the same.

A

It's the same, that's also how it works.

B

A

To the second to the third Etc yeah, and then only once the first Branch completes, then it will go on to the next one and the next one Etc.

C

We have like a blocking activity signal. It feels like it's excretely important. Oh.

A

Yeah, so indeed, so, let's let me clarify for Elsa too. If, as soon as it hits a blocking activity, then the second Branch will be executed and the third until they either hit a blocking activity or the adjoint activity yeah. So.

C

A

In most scenarios, it works fine, because, typically, you use a fork and a turn where you want to be waiting for some time out to occur. Let's say a day to give some user. You know some time to approve a document or something either. The timeline occurs order some user approves or rejects some document, let's say so in for those scenarios that works out fine, but it can be a problem if you have multiple branches and you expect all of them to execute.

A

But if there's a blocking one here as the cases here then in this LSAT 3 engine, it will be waiting for this forever. It will not go ahead and schedule this one as it is right now. I do have some ideas on changing this, because even in Elsa 2 it can be a problem where the other branches don't get a chance to execute and I. Think I wrote a little issue for this.

A

This one, a proposal for parallel processing of parenthesis and the ideas is that the well, the inner loop that reads off the the stack of scheduled activities instead of doing it one after the other. It would be, it would be a multi-track stack and then what I mean by that is. It would be basically alternating between logical tracks scheduled on this deck, so it would first get to Activity one from track one and then no matter.

A

If how many more activities are on track one, it would then switch to track two and schedule its activity and then switch to the next track and then back onto the first one and a track in this example would correspond to a breadth. So if there's three branches in the in a fork- or in this case two branches here just coming out of right line there, there will be two tracks and both would execute one after the other. And then this one would execute.

A

And then this one would execute coming from this one, and here it will be blocking, and once this resumes it will fix, keep this one more time.

B

Okay, which number was it 2878 right for the issue? Oh.

A

This one yeah, two, eight seven.

B

Oh seven, two: okay, take a look at that and think about that right because it's a yeah, it's a really tricky one. I didn't see that one coming.

A

Fine, when I work on these things, it's like oh wait. I didn't see that will come.

C

Because many scenarios, you know works and I would say in that illusion now that we are, we are parallely executing the branches yeah yeah.

A

Yeah, of course it's it's what it looks like so it makes.

C

A

Should expect it that nothing could be further from this roof.

C

But imagine if you are looking at the data processing scenario, writing the data database same time. You are sending it to another system like the same image that for gaming sheets right. So if I'm really waiting for one of the branches to finish, then I will start the other. It's uh it's a big hit. Yeah! Oh.

A

Yeah, it's something I want to have available in Elsa 3. Is that the parallel brands that truly executes in parallel? But it's going to be interesting to figure out how to deal with you know the container or the application is going out of memory or crashing.

C

Because one scenario I'm going to start looking at soon is this mq3p scenario where I'm continuously receiving data from like major device and definitely I need to process it into multiple streams. I need to load it somewhere, I need to build some logic generate some alarms all these kinds of things. So it's truly parallel idea. Yeah.

A

It makes sense yeah and- and this is what we should want for elsa3 as well- to be able to act as a data processing pipeline similar to Azure data fabric. Have you guys heard of that I think it was you um yes,.

C

Yes, I built a similar scenario, which is for mqtt I'm, getting the data from Edge device, I first I processed it through uh stream analytics and then from there I took it to Azure data setting, so I want to make the same thing as some. This is my you know. Target soon, that's.

A

C

A

So is there a reason for you to.

A

C

We want to build it around the answer because we want to say you know as a central part for the engine right. We want to make it available there a little bit of fabric. Then you are linked with the service from Microsoft Azure right.

A

Yeah in this problem, yeah I think so. Yeah.

C

A

Because this one probably does this one support, do you know if this has a parallel executing activities like parallel Loops or yeah.

C

You can you can put the activity and then uh you can Branch to the other activities, not yeah. So if you put multiple activities and then you can connect them, it will support sure.

A

But I mean let's say: I have a um the switch and I have a couple of cases. Yeah I, don't know. Maybe you pin the variable hey. How do I add a new case? Do you know- oh maybe maybe from here, okay yeah yeah? So let's have two cases: okay and each case is embedded, let's say so.

C

A

Do you know if these cases are executed? Oh that's, it makes sense right. It evaluates.

C

A

On the switch and then it's one of them yeah, this is one.

C

Of them, but if, after the switch, if you have two activities you can Branch to both, it will allow you and they will both execute.

A

Oh, and in parallel you mean yes, okay, like this yeah and what happens if I now says from this one to this one, what happens now I.

C

Don't know okay.

A

Let me know when you, when you figure this one out this.

C

Would be a good one.

A

To um to understand as well- and we can compare it to naden and but.

C

I would assume it will execute once because maybe they are keeping like a tracking if the activity is or not.

B

C

You cannot make a loop if you're, not it's limited. In that sense, it's a sequence right, so sequential execution.

A

So I imagine this one executes. First then this one.

C

Yeah then, last one one.

A

Oh this one in parallel and then yeah. Let me know man I'm, really curious to uh to understand this sure. ah Thank you so yeah, parallel executing this would be a good thing to have, and maybe even on this chart as well, maybe I, don't know seems very complicated. All right, that's all I had for for today. Does anyone have anything else? No.

B

I have my homework cut out again so far, I'm gonna think about this one.

A

Perfect, okay I'll see you guys next week. Okay,.

B

Thank you again thank.

A

You for joining bye-bye.