DASH High Availability WG, 15 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: DASH High Availability Working Group Nov 15 2022

Description

Continued bulk sync what to do with config changes during bulk sync.
Guohan needs SAI spec to be complete, for SONiC to have logic to handle cases.
Bulk sync s/be accurate (vs. performant), w/certainty re: how to test it.
Create a lightweight bmv2 HA for experimental version to verify algorithms?

A

My screen is what we talked about last time, where we went over the information in the Sonic repo and the um bfdecmp information here and the did. Anyone have um PR's or anything they wanted to cover today or Gohan.

B

uh Okay, so I saw the is today we're going to talk about the um yeah yeah, yeah, yeah or.

A

Let me go so um I think that was Sanjay and BJ. Did you want to cover uh the metadata that we talked about last time pulling that forward.

C

Hey uh yeah I think we're just checking if Sanjay is going to join all.

A

Right awesome thanks thanks and I'll find the pr. Let me stop sharing real quick.

A

Okay, well, while we're waiting for uh Vijay and Sanjay did we did you guys have uh other PRS we needed to look at or do we need to check the queue.

C

Hey, um hey Christina, so it might take a few minutes for Sanjay to join, so we can discuss other items. If anything else is there.

A

Okay, it didn't it didn't sound like there was a ton else, so we'll go ahead and create I. Suppose.

A

I'll see um I'll, take a quick second and see if James wants to join as well. He mentioned something about it. This morning, I'm gonna go on meet real, quick.

A

B

So I don't know if that part is uh clarifying the document or not. Remember um you know when we when we're doing this, uh uh you know the data plan. Sync, um you know I remember there were some, uh you know, config version um and you know if um only the configuration has been updated then will trigger the um will trigger some kind of resync right. So I don't know if this part has been uh documenting the in the designs or not.

B

They go on Sunday.

D

Yeah, it is documented, I think in the interaction between this thing and that section it's there there's a thing about. While we are doing uh bulk sync, what are what are the things that trigger incremental sync.

A

And, can you remind me of the issue of the pr again while we're talking.

D

Actually, if you go to the uh open, the no I'm not doing the pr, if you go to the the actual uh question.

A

uh Aj overview or.

D

No, no, the proposal, The Proposal.

A

Down to the bulk sink.

D

Yeah there's some interaction, I think. If you look for interaction.

D

uh Just below that, um after that, there is a section on interaction.

D

Check this coin I think we have some this thing on what happens if there is a conflict uh change.

D

B

Yeah uh lookies to it yeah sure.

B

B

So this is the case where we handle during the boxing and then there is still configuration change being pushed yeah.

D

B

D

Those case yeah so I think there are two cases I think in the bulk sink itself. When there is a config sync that happens right, you could uh the flow, which is be the flow one. Any of the flow that is being affected could have been already synced over or may not have been synced right it may. It may still be in the snapshot, but uh we haven't reached that uh Point. Yet right we haven't gone through that this thing. Yet so there's a differentiation uh of a completely new flow.

D

There are three cases there's a completely new flow. There is a flow that was there in the uh bulsing snapshot inside that there are two cases sub cases one is uh there is a policy change, but the uh flow was already synced over I. Think the bulk sink uh window has moved past.

D

That point- and there is a third sub case- I mean the second sub case, where there is the it is in the bulk sync window, but we have not synced it over yet, okay to the other side, there are three cases different ways to handle them, but all three cases need to be.

B

Done so maybe the question I don't know if we have been asked right, so um the Box sync is not happening all the time right. So it's only you know before you um switch over or bring up. You know try to bring up your backup node right, so um why why? In that case you know we we cannot block the uh you know, configuration I mean. How long is that boxing I think usually like a few minutes right, so why in that case we can.

B

We cannot block the configuration change so that we don't have this. uh We don't have this problem and only after the boxing is done, then we we do the configuration change.

D

Yeah good question: actually that was our initial. This thing to one I think from Microsoft from the v-net team. It was a no-go uh to block the config during the person getting the tie-in was not needed. It was a requirement given from.

D

Balki was there in the meeting: I was not there and that he can elaborate, it is there, but it was uh initially. We also had the same question. Whether can we, you know, simplify it by not allowing configuration change during bulk sync, but it was a no-go.

A

This is something you want Jay to chase down Gohan. This is he's on the call from.

B

E

Yeah I'm Jay or from Vietnam team, but I was not in the discussion of this, so I don't have uh currently.

A

I, don't have a context here yeah, but maybe you could look into it yeah. You have questions.

E

Yeah, if you have questions you can always ask me: I can retire to our team.

B

B

So how long is that bouncing you.

D

Know yeah about a couple of minutes somewhere in the few minutes later.

D

I think uh this thing I got was 90 seconds one of the tests, but uh somewhere in that place you can think of it a minute to a couple of minutes.

C

B

Has to you know inject some. You know flow during that boxing.

B

Is this many are the? This is many other configuration change right so not on the uh there.

D

Could be clients which are initiating flows? Is that what you mean.

B

The client, this ignition flows that that uh I mean we're mainly talking about the configuration change during the boxing and that's the reason you know we have this special handling right. So I'm trying to understand like is this really about the Comfort green changes? This is not. uh This is also about the flow creation. Those things.

D

D

All uh all three I think it could be Network driven, it could be. Administrative configuration driven, uh I, think the end. The net net area is, you can't freeze, uh the networking might still be, the clients might be sending uh traffic, they might be sending fins, nothing might be change in the flow itself and there might be flow, uh slow modifications due to policy. Both cases.

B

I mean I I the flow creation. You know deleting that, apart from the client side, we cannot block it right so, but I think.

E

B

Configuration change, yeah, I'm, not quite sure, uh you know, uh or or maybe we put this way right. So um what what are the mechanisms that we need? You know for the um flow uh triggered changes and what are the things that we needed for the configuration driven changes right? Can we can we separate them? You know. Is there any mechanism? You know? Let's say you know we. We had this assumption that you know the configuration will not be changed during that boxing, but you know flow can still come in.

B

So if we, um if we have that assumption, is this a color thing still needed or no.

D

You might still need it. uh The reason is uh the same three right so I think, let's uh abstract out and say there is a change in the flow we'll ignore how it came right, whether it's to policy or through uh Network event.

D

Now, if there is a change in the flow, the flow could be of three classes right. One is: it was a completely new flow which was not existing when we started bulk sync right, so it does not exist in the bulking snapshot at all right. That is one class. The second class is uh the other side on the it could, it could have exit I mean it was existing when we started the bulk sync. uh So that's in that second class in that second class, there are two sub this thing.

D

So now the bulk sink over that minute or how much of a time we are taking the window is moving right. It's like a cursor that is uh in a way in a cool way, I'm just saying: there's a cursor, that's moving and sinking. First now, the flow that got affected could be on the left side of the cursor or on the right side of the cursor right, so that is the second subclass I.

D

Think all three cases now, irrespective of whether the change happened via policy or via a network event uh it in the end it boils down to, there is a flow. There is a change in the flow state which needs to be synced and that flow could potentially be in these three classes.

D

I think it might still be this thing.

B

But the but the you, because during the boxing we still have the data plan, sync right. So for whatever the flow changes you know, for example, new flopping setup. You still go to the data pass sync right.

D

B

Why why is that? um Because that guarantee the flow been created on this? You know backup, so why.

A

B

Is it necessary for us to um to still.

F

Have that flow.

B

To go over the um you know, the the the boxing channel, the the Japanese Channel.

D

Okay, so I think the this thing is uh just go on, for example, um if it is uh if the first class right class, one kind of I, think where it did not exist at all right, then the implementation, like you, said the data path. Sync can take care of it completely right you don't we don't even need the pulsing at all the it's, not even there in the snapshot. So the data path is not going to I mean the bulk. Sync is not going to sync it. It was not there in that snapshot.

D

So I think this one falls under the category where it is in a different uh quote: unquote: color right, so that snapshot was designated by that color. So it's in a different color, so it gets synced by data part directly right now. If it falls in the second class. Now the implementation.

B

Has one equals sorry? What is the second class? Second.

D

Classes uh it was an existing flow which had the color of the bulk sink when bulsing started. So it was it's not a completely new flow, so let's say at t0: we started bulk sync and we go on I mean at t0. uh If the flow was created after t0, it's a fresh floor right, then the data passing can handle it completely. There's no need for the bulking to get involved at all.

D

uh That's why we do the data part sync, while the bulk sync is going on right, so that all those flows can get synchronous now uh there could be a flow which was T minus one, which was which existed before t0. We started Global sync at t zero. Now at uh uh uh so now, bulking responsible for syncing it right I mean bulk. Sync is is going to go through that flow, find that flow and sync it across to the other side.

D

Now, uh while we are the bulk sync is uh parsing and then going through right, you can come up with a state where a T1 we have done with about 60 percent, say of the flows right now. That flow that got affected could be within that 60, that's already synced, or it could be in that 40. That is not synced. Yet right, yeah.

B

D

Implementation may choose how you want to do now. If you just allow the data path. Sync to go through, you need to handle, it's possible. I mean depends on the implementation right, how you want to uh take care of it, but in the hardware, when you are inserting, you should make sure that uh there is no race conditions between what comes, because the bulk sync, the uh the flow that we are talking about, might be in Flight already in the bulk sink right.

D

It's not synced to the other side, but it might be in flight or we are just picking up it states in the in memory on the center side or on the receiver side, Etc right. So now, if you have two channels coming in from the data path and also from the from the data process, you can also from the boxing I think we just need to make sure that the implementation handles the trace, condition right. You don't overwrite one from the other.

D

We don't want the uh the new flow to have been inserted and then the bulk sync comes and overrides it back to the old state that we just want to make sure of that. I think how we do it. uh This is the case that needs to be handled so that we don't break the consistency.

B

um But for the blood deletion right, so if we don't have the boxing the you know, the the theme packet will go to the other side and also delete it in the flow on the other side right so.

D

B

D

B

You're, saying is that okay, so you know if the theme packet goes. First, the data plan packet goes. First, uh data sync goes first, then it will delete. uh It will delete the uh the flow on the standby side. But, however, you know during that moment you have the boxing as well. Then you know boxing may be a little bit delay. So therefore you will say: okay.

E

C

B

Have the previous snapshot so I I'll put this flow. You know uh in.

D

The back in the standby right so correct could reinsert it back. Then you'll go out of sync.

B

But to this season, but they, how do we address this challenge? I, guess that you know the you know the the the the boxing snapshot is different. uh You know the timing is different from the data passing. How do we just this kind of challenge using the color to dress its challenges? So we know you know what is the version or how do we? How do we address that challenges here?.

D

Yeah, so I think it might become very I mean it might be in the implementation itself on how we keep the flow Etc. You could the mark in the flow whether it's been uh synchronized and act back from the bulking Channel. If it is uh marked in somewhere in the flow yeah, then data path in the data path. You could know that okay, this has not been synced across so I will let the bulk sync handle it.

D

So you know that it is in that uh class to to be, but if it is uh uh marked already that we know that bulk Sync has already handled this flow, then we could say that, uh but data part sync will go ahead and sync this across to the other side, but I guess yeah, it's uh one way of doing it, but it could be multiple ways.

B

I'm, not sure I completely understand right. If, if, let's say you know, if we want to handle, float deletion and the data passing goes first, they delete the flow in the backup, but in this case the flow doesn't have any state now right. So they uh you know, we have this boxing. You try to add this flow. So what is what is the boxing here is trying to compare. You know because there's no other recourse saying okay, this photo has been deleted before so we should not create it again. So how do you see?

B

How does this uh I'm trying to understand? How does this work here? Because you are saying you know we mark it as a deletion, but that's.

D

Fine, okay, yeah! No, no, what I said was, if you have it not for deletion person. So there are multiple ways to go and I mean again mainly, should we should leave it to the implementation? One way is in the flow. You say it's not about Mark for deletion or not you say yeah, you could Mark for deletion, and you could also say whether bulk Sync has processed it or not processed it right. So I. Remember that cursor. That I was saying uh whether uh where the boxing cursor is moving already processed, not process right.

D

So you could say whether the cursor is past that uh flow or not yet past that flow. Once you know that you know, if the cursor is passed already and volkswinc is already synced to the standby side, then it is safe to sync whatever from data path right, you can do it uh all incremental thing, because we are past that point.

D

You could start thinking it the uh on, but on the uh if it is not at sync right, if it is cursor is still on you're on the right side of the cursor, then the data path syncing would actually uh create trouble right, because data path has the bulk Sync has not yet sync to the standby. Then you have to suppress in. If you are following this methodology, you would need to suppress the data path from syncing the flow and say that the bulk sync will come through and handle the flow syncing the flow.

D

If that is the case, you would need to mark it for delete. You know that this is already deleted right, so you can clean apart. If there is a change, I'm delete is one case. It could be delete or it could even be some uh something might have changed in the flow itself, like whatever the state of the flow might have changed, depending on that you could uh handle it appropriately.

G

So one one way to look at it Sanjay is that you will need to do kind of multiple passes on the data data path flows, potential right that.

D

Way, consistency.

G

D

um Maybe exactly human being saying, but during that bulk sync, there might be uh during that interval. The active understand by there is this thing, but once boxing completes- and you have done- you have actually you'll have one spot- the multiple passes have passed. You'll have actually achieved consistence. That is one way.

D

Second way is you could do multiple passes or you could have on the stand by itself if you had somewhere to uh distinguish what is before and what is after, you could let the data passing go through and then, when the bulk sync comes in, you know whether it was after or before so you could either drop it or actually update the data path. You could do that way too. There are multiple ways you see what I'm saying yeah.

D

uh Theoretically, let's say there is a time stamp on the flow sync on when the flow was updated. You could blindly sync if the implementation chooses to do so. You could actually sync the uh flow blindly to the other side and on the standby, you kind of arbitrate to see whether the the what came from data path sync and what came from bulk sync. If the bulk sync was newer uh in that quote-unquote time stamp or whatever, is that uh this thing?

D

You would push that you would uh accept the thing from the bulk thing, but if the bulk sink timestamp was older, you could ignore it that you could drop it or the same way the other way yeah. You could do that.

G

Time stamp versioning, you have, you know multiple ways to kind of resolve that ordering issues here: yeah, yeah, yeah.

D

But I guess the key is the there is there are these cases that needs to be handled? I? Think the implementation has to kind of keep this in mind. I think that's what we are.

B

But I guess you know the in terms of uh side. Do we have those necessary? uh You know attributes to to allow this happen. You know, for example, I think you know there were some mentioned about the timestamp We, compare the timestamp and there's a cursor I, don't know like we. We have this cursor to find in this IPS so that you know both sides will know the progress. I mean sorry, I mean you know. Basically, the the standby side will know the progress you know which has been synced, which has hasn't been synced.

B

So when I receive a flow, I know whether this first part of the left of the cursor or right of the cursor so and.

A

D

Question yeah I think this should be in the uh I. Think it's an implementation, specific thing Gohan depends right, you, the implementation, for example, uh you could implement it using this cursor or you could use it using the timestamp or maybe I think this is what uh we are coming up now. There might be other depending on the hardware, and this thing other implementations may find something else that is uh more conducive to those implementations. Right, it's a so there is an option in the this thing we have the vendor metadata.

D

You could uh put in, for example, the time step in that vendor metadata and hypothetically right. You could put the timestamp in that mental metadata, When We, sync the flow so, along with this metadata, there is also the um the vendor specific metadata.

B

You could put it in that yeah I. Think. The problem for me is that the.

B

You know you you I, think you mentioned. Is the implementation specific since a few times right so, but the problem for me is that you know I I'm going to implement this, since you know in the sync D for the boxing right. So but, however, if I don't know, if the PSI um you know spec is complete, then how do I address those? uh You know those age cases.

D

Configuration in the side, it's not needed.

B

D

I didn't get. Why do we need this cursor or this thing in the in the PSI interface.

B

For example, you know you know the with the type here we get the boxing crash, so you know we get uh on one side on the active side, we get all the data and we use grpc to transfer it to the standby side right. So they um you know. Meanwhile, the data data data processing is in place right. So then, how do I? You know, uh resolve those. uh You know those conflict, no, okay. You know I have to a second pass, or you know that kind of things right so.

D

Oh so that is being I, think that's uh this thing, so I think from the side. uh What the size tells the data path is start start data passing right, I mean start bulk sync and then once the implementation is done with the bulk sync. However, it chooses to do it right whether, however, it tries to uh the cursor or the time. Whatever, however, is the chosen method for the uh Hardware friendly implementation? It will.

D

The implementation finishes that, and once it is done there is a notification that comes from the DPO from the size, data implementation that says bulk sync is done so that way, the Sonic the state machine that is running knows that bulk sync is done, uh so you that way, the state machine does not need to intimately know the exact mechanism that is happening in each implementation, because each implementation may be different depending on how they want how it stops, but it will get a notification.

D

Your the it is driven by the uh the the state machine running on uh Sonic via this I think you will say, data bulk, start pulsing and get a notification back from the mutation when the bulk sync is done right and really the The Proposal is leave it to the implementation to actually uh coordinate the bulk sync and do it in the sequence. You know how or whatever is friendly to the hardware.

D

B

I think it's a it's a generic problem.

B

um You know basically, when when you take a snapshot and you do the changes and then you know, meanwhile, the changes still happens. It's a generic problem right, so I, I'm, saying that you know, since it's a generic problem, then why don't we? You know work out the details to see you know. How do we address it? uh You know as a community in a uniform way right. So um so maybe I think.

B

Maybe maybe maybe uh you know I'm not sure, I completely agree with uh with the approach that we're we're handling this.

B

um You know this complexity in a purely Black Box manner. You know basically saying okay, you know, you know the Sonic doesn't care about uh exactly detail, how the conflict being resolved between the boxing and data pass. Sync, it's a you know. I think you know. Basically it means that we, you know at least on the Sonic side. We didn't think uh you know enough all the cases and we didn't really have any logic to handle those cases we leave to the you know each year dpu vendors to handle it.

B

However, you know the risk is being that, because this is, uh you know, a tricky things risk conditions right, so there will be the testing. Metrics should be huge right because we don't know how exactly each manner she implements since and what are the? What are the cases we need to test right? So you can pretty sure you know this is a very complex scenarios and you know if it seems you know. If time is not happening right, then we'll lose some flows.

B

You know at at those uh you know during the process and we do not know what flows will be lost right. So you know because it's up to the uh each implementation to decide what their mechanism is going to use. You know some use uh yeah, so I'm, not sure you know, like I, think the approach is to we're currently taking is leaving a lot of uh you know, testing cases guys you're, basically cheating as a black box and the um each one. Each implementation will have its own I.

B

Don't know how this kind of a complexity be managed.

A

B

I think then we.

A

Have ha even architected or designed much less test cases Gohan yeah.

D

Yeah, so that complexity is also this thing going so I, don't know from my my viewpoint. Is this right, uh for example, uh today, when we are in in our implementation at least right, we for features Etc. Sometimes we there is uh even using a few more bits in what we pass from stage to Stage Etc that matters quite a bit right.

D

So just if you get so there is uh we cut down produced the size of what is being passed across Etc and it, and there is a it, practically makes a huge difference on how you get to that uh very high numbers that we have scale numbers and the CPS counts, Etc right.

D

So if you, if you try to my opinion, just my personal opinion is, if you try to make it into some least common denominator kind of way, then I think we'll not be able to get these numbers right because it's not like there is a better implementation or there is a bad implementation right. Each implementation, depending on how it is implemented, how the details of each flow table, how it is this thing, how uh the stages are designed internally right.

D

What you have in the behavioral model is not what is going to get implemented in the hardware potential right. It doesn't even need to be I think this thing is, it doesn't even need to be P4 kind of a thing right.

D

If that is the case, uh if you have to when we are talking about this uh Cutting Edge, uh this thing where we are actually pushing the box to the limit, uh I think having this uh this thing, where you can do it in a hardware friendly way and you can squeeze the most performance out of the hardware. It is very advantageous. I, think if you don't have it, and uh this thing it'll it'll affect the performance, is my personal. Take.

B

You know General I agree with the performance, uh but um you know for this particular case right. So we're talking about the boxing and we already said the boxing doesn't need to be performance right. So you know it's not like CPS right, so this, um so you know the boxing can be a little bit slower right, so it doesn't really impact the CPU. You know at this moment you know your the Asic, the GPU is still managing all this flow, creating new flows, it's just boxing a little bit more slower right.

B

So you know the the this part. I I think the the the thing here is more about the correctness. uh uh You know after finish, the boxing. How do we ensure you know the the active and standby end up in uh in in a venture consistency? We're talking about that? You were talking about you venture, consistent, meaning that we it we can't afford, take more time to get it sync right. So just um so that part, I, I, still I, think I have a have my, um but you you know the the.

B

My key point was that you know that if we don't know how it's done, um I don't know how to test it.

B

Yeah and I I. Think in the you know in reality you know if we have this case, you know in production, saying okay, you know some flow's been lost, I! Think it's going to be a nightmare for people to debug it. You.

C

Know yeah because you.

B

Don't have the full, you don't have the full history of how flows are created. You know those kind of things you you know they were talking about millions of flows right so get you know making records of all those flows. You wouldn't exactly happening. It's going to be huge, I, don't think you know we we are prepared to you know to to log every flow. You know those kind of things to to to be able to. You know debug. It later.

B

Then, if we lose some flaws, I think who knows what conditions loss of flow. But you know the customer is going to complain on that and.

D

Exactly yeah yeah, so I think this is what we have learned, also I think even in our experience, go and I think. That is why I think the intent, my intent was to put these at least enumerate. What are the issues that we see right so I think any implementation can if they look at all the issues that are noted here, at least there is a uh whatever we have learned from our implementation. Now the exact mechanics of how we do it in our implementation.

D

I think it is very, very implementation, specific I, don't know if I can share it here in a meaningful way, uh how we do it in our pipeline itself right. This is this thing, so that is where I've kind of drawn the line and said: okay, we know these are our learnings from our implementation um but exact mechanism of how we do it. How what we think across it's a little uh too specific right.

D

It doesn't harder to abstract it up, but I think we can say possible mechanisms at a very abstract level, how it is, but uh uh I'm at a loss how else, because otherwise it has to be very implementation specific. We have to say in this floor. This bit will say that, and things like that, I don't know. If you can share that.

H

Essentiate regarding to your POI about implementation, specific, then, how do we plan to incorporate this into their behavior model?.

H

Because I think at one point we were talking that we do want the behavior to have ha handling as well right. So if you keep saying this is a vendor implementation specific, then then how I mean? How does that information get into the belief of one.

D

uh So the behavior model doesn't need to reflect exactly the uh uh what it's a behavior. It's a how we uh this thing the behavior right now. It won't reflect exactly the hardware implementation. Now it will.

H

Be different, I mean it's a software that you mentioned earlier.

H

But you mentioned earlier there's many different ways of having the eventual consistency right time stamp is one approach, for example, or doing multiple tasks I mean, wouldn't that have an implication of how you're going to put it into the behavioral model.

H

Right and I mean let alone the interoperability. I mean I. Guess we haven't touched that part. Yet until our busy yeah.

A

Let's, let's not go into interoperability right now, we've covered that over and over right.

B

H

C

H

I didn't talk about the behavioral modeling that so.

D

Yeah I think we should pick one pick. One of the ways that works in the behavior model. I think we are saying this is the behavior you have to kind of. uh If the bulk Sync has already happened, then you have to uh honor what comes from the data parsing and if it has not yet happened, then you could actually data pass it. That is the behavior that we want.

H

If that's only let's say, if you do a time stamp, then what would be the behavior on the receiving side.

D

H

You would, in terms of comparing time stamina, yeah.

D

Yeah, if it is depending on, if it's a newer time stamp, you would accept the flow if it is an older one. You want kind of yeah but uh see I think, but uh what I was trying to say was see. This is one way you could. Each implementation may have a good way of generating that time stamp at the flow and have enough this thing to kind of imprint it in the data path treatment. There is enough, uh it is efficient to do that.

D

Some implementations may not have that like they may choose a different way right, that's what I was trying to get to, but in the behavioral model yeah we could put a timestamp choose one of these, so at least at least the external Behavior. We know how it, what is it that we are expecting?

D

The implementations may choose to do it slightly differently for efficiency reasons,.

B

So how about this right so because I still have my question here, maybe you know I, think uh I, I, guess we'll we'll we'll work with the percent of first. You know flying and understand a little bit more on.

D

B

Part and to see you know how, um how do we handle, because you know eventually rise to this uh Sonic- is going to do that. uh You know this boxing and you know we need to manage this. uh You know these two channels these and carefully.

D

B

You know maybe uh Krishna, you know, let's set up a meeting with uh with President, you know uh uh MD and you know have an offline discussion on what exactly you know how the conflict resolves here right. So yeah.

A

Has been kind enough to submit their proposal, pensando AMD has submitted their proposal. Marion wrote this deck I mean we've had a lot of different people submitting. We just haven't really landed on. You know even the basics of of what we're talking about. Gohan we've been doing this since last April. If I remember correctly, yeah.

B

I I think yeah, it's you know like um I I feel like overall right. So the the you know we we reach. We have the agreement, how to do it right so and we do have a lot of details. um You know um you know uh pay. You know based on the contribution from the um from Sanjay and team right. So definitely a lot of detail but I feel, like um you know, it's not a hundred percent. The spec is not a hundred percent um clear.

B

You know in terms of handling those age cases which is most you know, probably the most critical things right. So you need to get these things right right. So it's not like uh yeah, and you know that I understand this is definitely a lot of the challenging in a lot of details right. So um so at this point right so uh I think you know uh I will definitely you know follow up with. uh You know uh presenter separately to understand.

B

You know how exactly it has been done, but meanwhile I think you know for the community members right. So um you know anyone has any. um You know, suggestions and proposal because it probably is there right. So you know these two channels, things how to resolve the conflict. I think we're willing to have here anyone that, having any suggestion or concrete the proposals to address this right right, so I I.

A

Think that they're pretty good here too, if you ever want to read through these two yeah.

B

Yeah yeah yeah yeah yeah, so um you know any any members. If you want to bring forward your your proposals on this one, you know I. Definitely you know uh open to listen to those and, uh and um you know see how we can get conversion those specific problems right so I think where you know this, um you know made a lot of progress. You know coming from a generic idea to a very specific proposal now, but we're still missing some details, and you know it doesn't mean that you know anybody.

B

If we can fill those details, you know have proposals, you know we. We should work together to um to to get this back uh clear right. So.

F

And this is Chris. I have an idea that people will probably say: oh no, more more work.

F

Is it possible to have a I call it lightweight behavioral model of PHA synchronization mechanisms in the flow table updates without passing traffic, just to have like a an experimental immersion to verify algorithms.

B

F

Opening Pandora's Box a bit but at some point yeah.

B

Come again, I mean: what do you mean by this Uh Oh? You mean like using the BMV tool to to.

F

Do that is that it doesn't necessarily have to have a data plane, but it could have all the control mechanisms of H async without worrying about the data plane itself, something that you could use BMV too I suppose, but to actually test out all the synchronization and algorithms and and apis without having a specific implementation of the data plane itself.

F

And that could be like a scaffolding.

B

F

I hesitate to say it because I I know how much work it is to even have the behavior model itself of the data plane. We're still struggling to complete that, but this is kind of a high stakes thing right.

F

You know there's whole books written on how to achieve consistency in databases.

B

F

B

No no I mean this is a Holy Grail right. So if you, yes, if you want to ask me, you know like what is the? What? How important is a trait to this project? I mean it's a it's a Holy Grail right, so yeah, you know CPS, uh you know, of course you know, but it's uh it's a it's a kind of different challenges right. So, but you know this is this correctness I think yeah.

A

So facing the ground in the industry, though, like if we were able to pull it off, it'd, be pretty cool.

F

Yeah I mean there could be almost like a scaffolding of of the algorithm that people could agree on. You could almost use that if it was designed a certain way as actual implementation code, but you know it was, it was highly abstracted, but.

B

F

Know I know this is like a via.

B

Migration right, so you know it was a computer that is a real migration, live migration right. So this is a. This is a similar fashion. I think you know, a lot of a lot of things are very, very similar to the VM. Vm live migration, so yeah.

F

I just want to say it for the record and people can um ride me out of town on a rail if they want because I know it's a lot of work. But um you know at some point we'll be talking about these specs and then, if we don't have anything, we can share. We'll have a super, complex, spec and then vendors will say: okay, here's, my Black Box, you know, try it out and and there'll be a lot of black boxes, we'll be trying out and and it'll be.

F

um You know, challenging to go one level deeper and look under the hood, so yeah I.

B

Don't know I, don't know how to exhausted all those age test cases you know if we, if we completely like treated them in a black box right, so I will be testing uh challenges, yeah, um so yeah yeah. So um I think you know what you you said is a is a is, is a very good idea right. So um you know like a basic version. uh I think. Maybe you know uh you know the the you know, especially on this particular scenes right so I think.

B

Maybe the you know the challenges that the um you know, the oh there's some ideas but uh like we actually don't know how to implement that. You know Implement that uh correctly, you know. If I, you know, I I think your idea is, you know, let's, let's put a basic version out and then you know then iterate on that basic version right so um yeah, uh I I, you know I I, think my focus is still uh you know, at least from my side. I I need to have a clear view.

B

How this would work right so because, right now, I, don't I. Don't have that kind of understanding how this will work.

B

A

Don't know you know I think.

B

A

If we make a step forward, then like do, we need to go, find PhD graduate student interns to help us write code or like how do these things move forward?.

B

A

Or have you guys seen things like this before in different open source projects.

B

Oh I I think you know um so yeah, so uh you know there are two ways right. So one way is that the um you know we we will try to form a more clear view. You know those age cases and bring up those proposals to the community.

B

um You know um a specific proposal right. So you know, there's could be multiple different approaches to to address this challenge, but we will we'll try to bring up a specific proposal and of course you know there there's other vendors and if any winners want to you know, you know bring forward their proposals. I think, of course you know it's certainly will come right. So I think this is a big Community. We have a lot of uh experts uh here. You know I understand the problem, you know knows how you know how to solve it.

B

um We just need to uh you know uh collaborate a little bit more to to to get um to get some more information. uh You know um uh uh you know agreed yeah, so I don't think this is the raw case. Size, uh yeah,.

F

But another radical another heretical crazy idea is: are we trying to reinvent the whale and can we use like replication, that's built into redis.

B

uh I think it's maybe it's a um you know. The radius is more on the software side. I think the challenging here is that there's two channel in sync, so this I don't know what kind of uh existing model we have right. So there's a because you know if there's only one channel, that there's no problem right, but we kind of have two channels for um for the flows. One goes through the slow pass.

B

One go to the data pass, so these two channels need to resolve some kind of uh conflict right so that that I I I don't know like what what kind of uh um you know I haven't. uh You know whether there is a existing solution, similar solutions that we can borrow.

B

um But we haven't had chance to review the you know the the proposals from the changes. um Sorry about this I mean I I was checking this.

A

Sorry, my dog's barking, um so so it's noon should we cut for the day uh Gohan or keep going.

B

No I see we we need to actually because uh we, we discussed quite a few on these things, but we do not have a solution right. So.

A

um We and those PRS for the uh amendments for um A and D just hold them to next week or uh squash and merge those.

B

All right, let's hold to the next week for the for the trade right, so you know yeah.

A

Yeah, okay, great you guys um yeah great, thank you and thank you pensando team for coming on and everybody else. um I'm gonna stop the recording and post it like normal and uh I guess, put on your thinking caps for next week and we'll get back together.