DASH High Availability WG, 20 Sep 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: DASH High Availability Working Group Sep 20 2022

Description

Reshma hosting :)

A

Okay, good morning, everyone, this is the dash High availability meeting on September 20th 2022.. uh Today's main agenda item is uh AMD pensanto's, presenting their ha proposal and we have been reviewing um the PSI api's portion of it. So we'd like to continue that today and Sanjay will be presenting that from AMD pensando, if possible, that uh late uh after this presentation, uh maybe Gohan and Prince could also talk about the Hue design with overlay ecmp.

A

um Before we start any questions from the previous meetings that anyone may have.

A

Okay, um please continue Sanjay. If you can. Please share your screen.

B

Yeah give me one minute, I think I'm having some trouble. It's asking me to update something when I'm trying.

A

To show you, okay, sure, yeah.

B

uh Do you want to go with the other part of the agenda and you can come back to this if it's uh or I'll just give it a shot? If it happens in a couple of seconds, it's fine, otherwise, it.

A

Yeah, that sounds good. Maybe you can try for a little bit. Let me check if Gohan has joined.

C

A

Yeah so uh Sanjay I can present the screen for you as well. If you think uh that's okay or um yeah I just noticed that Prince also joined uh Prince. uh We were thinking that you know at the end of Sanjay's presentation. Maybe you want to talk a little bit about the overlay CMP how it uh you know, we uh foresee it working with ha design that Gohan had brought up.

A

Thank you, friends,.

B

Yeah I think sure.

B

I think it's the last two.

A

D

B

A

D

A

Please let me know when I should stop scrolling yeah.

B

I think it's the last I think we had done the note pairing last time. I.

C

B

Just stopped before the stateful and the the switch over pieces towards the last two pages: okay, yeah. We just completed yeah even below four yeah. We just finished this last week, the note pairing sequence, okay and I- think we had the unplanned switch over and the plants, whichever got it.

B

So in the unplanned switchover I think uh here, what we are, this is still the heartbeat uh being managed by the data path.

B

Cpu uh in this case, uh what the proposed flow is uh the loss of heartbeat is detected by the dpu and it updates the PSI SDK about the status change right, I think the first first part of the status change is mostly to keep for debugging, uh but to keep the state machine in sync so that all current state is reflected properly and after that, the switchover essentially initialized on the dpu data path, to switch over to the to switch over to Standalone right and once the switchover is complete.

B

uh The data path again notifies the DPO again notifies the Sonic State machine that uh switch over to Standalone is complete uh and then the Sonic side, the Sonic stack, uh can I think once I think currently in the current implementation of whatever you have done. So the controller controller determines that, uh after switchover, all the configs have been pushed right is up to date on the uh new new Standalone, that is the old standby.

B

uh This being I think I think we talked touched upon this sometime in one of the previous meetings too, uh the both the nodes, uh the active and uh the primary and the secondary. They may not be in exactly the same uh at any given point in time. uh The two might be out of uh not in sync with respect to configuration, because it's being the configuration is done independently by the controller on both sides. Right I think we are touched upon this.

B

With respect to the policy State, uh the policy results being communicated, you know, being passed across from the active to the standby so that the standby does not re-evaluate its policies, because the policies may not be the same as what the active has so the transfers here too. So when the switchover happened, uh there are flows that are there in the new active DPO, uh which has ah policies evaluated as per the policy. That was there on the stoil actor right.

B

So we want the policy to come back to the same state uh on the new uh primary uh before we kind of re-evaluate all the flows or do any you know we clean up the flows or anything so. Hence there is: uh the controller has control the the controller today, if it uh issues the issues and API which says flow reconcile at this point in time.

B

uh I think the notion is the controller knows that whatever is the current state of config has all been push to this node, and once this is done, there is some flow reconciliation that happens. So all the current flows are re-evaluated based on you know to see if it matches the current policy. If not, it is updated.

B

So that is the unplanned switchover. Similarly, there is a concordance thank you. So there is a plant switchover.

B

Yeah in the plants, whichever case.

B

uh Yeah, so there is here, I think there in the unplanned switchover there is possibility that there will be. There can be some uh a little bit. I mean some traffic drop in the network is reacting to the change. There might be some traffic that is black hole because it's still going towards the old old, active Etc. So with the plants, whichever the intent is. uh We try to keep uh throughout that uh any disruption in this thing as close to zero as possible right.

B

So the this is because we have the flexibility here to coordinate the switch over between the primary and the secondary right. uh So when there is a uh so the controller comes and says, switch to primary I mean there is a trigger that is given to say that we want we want to switch over uh when we want to switch over uh the dpu data path.

B

On the other side, uh does some primary function to I mean to prep for switchover and on the old active side we withdraw the protocol route, for example, if it is bgp, we will draw the bgp route and then wait for a convergence timeout right uh once this.

B

Notification is turned to the uh sent to the new old old style secondary or the new uh Standalone that we want to so. In this case, there is a state called as a pre-standalone, uh so we let it soak in. There is a small timer that we have and once that is there once that is done, uh the standby side switches about to stand around and then notifies the active that the old active that the switcher it is switchover, radio or it is switchover, is done at which point the old active can actually shut down.

B

You can close the communication, Channel and actually exit application.

B

After this again, if you notice on the lower part of the this thing again, the whatever we had to do with the hre floor reconcile the same sequence follows on the uh new standard on again.

B

So this is the sequence where uh in this case, what happens is because we have kept the active uh functional even when the standby had taken over as the Standalone till uh everything was complete on the Standalone side, if the network were to redirect any traffic because of uh due to no, we were in the transient phase of Route convergence in the network. If it were to land on the active, the active would still forward the traffic right so hence there is no glass and any drop in any traffic throughout the whole sequence right.

B

D

Right sure, can you expand one more game? uh You are saying for the unplanned streets over. There is no um zero drops right. So.

B

uh So on uh for plants, whichever we can do, zero drop gone, but not for unplanned switch over yeah.

D

Why why the planet switches no job at all,.

B

So uh usually, the drop that we see is, uh for example, when the active goes down. uh There is some reaction time, for example, if you're doing bgp and we are doing the bgp advertisement uh going for the ACT from from the active.

B

uh So it takes some time for the network to react uh to that bgp advertisement so that it can take off uh redirect all the traffic you can choose the secondary path and send it towards the uh the surviving node in unplanned switchover. So that causes the uh drops that we know of right. What? What whatever drops we see in this thing, so this is for the rest of the class Fab the fabric to react to the failure with plants, whichever we can, because we have the active still running.

B

We leave the active running so so that even we try to that window where the network is still converging. We try to keep the active running so even the straight traffic that the network might redirect to the active we are still able to forward it and hence we we achieve zero loss with the plants whichever, but not for the implant.

D

There is a little bit coordination right, for example, in the planned one. When do you direct? For example, if you direct the traffic, um because uh you know down the data plan, there is a behavior change. You actually interactive one no uh need to send the thing to the um you know to to the to um to the standby one right. So, but if we switch a row there is a change of rows. Another uh the new active one should send the traffic to the uh the new backup right so I'm, just trying to oh.

B

D

Little bit unsynchronized the events that could cause. uh You know, for example, let's say you know you uh you you, you ask the pgp to write the actual traffic to the new active. Now the new active will start uh receiving traffic. You sent the same to the um to to to to the to the previous active one right. So, um but what if uh you know what, if you, you know, switch the mode? If you make the previous active to standby, and at that moment you still receive the traffic from the network.

D

What does that do right? So you know there need to be, but you know perfect. Synchronization between you know the data plan and the uh you know the control plan to inject the traffic right. So have we analyzed those? Yes.

B

Yes, so I think maybe uh I think I understood your this thing. If it's not please correct so I think the what happens is in this case right when there is a plan switch over. uh This is usually I think the cases that we this thing is for uh there is some maintenance uh on the other side right. So we want to drain all the traffic from the current primary to the standby so that the current primary, the old primary can be brought down and there can be maintenance done so.

C

B

We see in this state in the state, whatever in the uh flow, is half of it right. So we have come to the state where the the primary can be brought down right once the primary is brought down. The whole note: pairing sequence starts again, so it is like it's gone down to stand alone in Standalone. We are not syncing any state to the other peer at all, because it's not active right I mean it's not even paired up. After that, the old primary goes down.

B

Reboots comes back up, and once it comes back up, it starts the peering session again and.

D

This, then sorry, but when you burn down the old primary at that moment, do we expect any uh you know package option? How do you, how do you brought down that old primary so.

B

Reshma, could you just go down uh to the plants whichever so here go on? If you see that on the in the middle section, uh we actually de-advertise, we are. We draw the routes, bgp routes on the left side. If you see towards the middle right, we withdraw the routes. So at this point we have already signaled to the network that we no longer can handle traffic and there is a config configurable timeout, which we allow the network to react to that advertisement and stop forwarding any traffic to the old uh active old primary.

B

So this is the time where the traffic is drained. Any traffic in Flight is drained out. So during this time, during this timeout, we can receive traffic on the active side too, and the active will continue to forward because we haven't brought down the uh forwarding. I mean the forwarding the data plane yet right after this, so yeah.

C

So also building this window, both uh both nodes can both dqs can receive traffic. It's kind of like both are active.

D

So when will you when you start training the traffic right so I your active Studio, forwarding traffic, meaning that it will send the traffic to the uh to the standby right so or you will what it will do.

B

Yeah yeah, so if there are new floors that we receive, we will still sync it to the standby during this time. During this timeout.

D

Okay, so you know when you're training a trap at that point in time. What is a standby step? You know what, if the traffic directly go to the you know the stand by what what will standby do.

B

So there is uh uh so this is this window, where I think there is some amount of reconciliation, that's happening, so the standby is also, for example, uh if there are new flows that is that are getting created during the small window. uh The standby is also.

D

No I'm not talking about I'm talking not talking about the flow stem I'm. Just talking about your training traffic, you say: okay, maybe I need the one one second or three seconds to dream the traffic from the from my active. But at that point maybe you know at that point. You know. If the network is being able to drink quickly, then you will send the traffic to the standby right. So at that point, what will standby do when we receive those traffic? The.

B

Standby will also forward go on. It is.

D

Forwarding poured into what foreign to to.

C

D

Vm directly or forwarding to whom.

B

To the VM directly, so it is doing regular forwarding for the flow based on the flow, the flow data that was synced from previous uh when it was standby right whatever was that it, it will use the same flow data and keep forwarding databases.

D

What about new connection.

B

So new connections are also created on the Standalone on the uh in the pre-standal loan state or in the left side. We are still uh creating new connections on both sides. At this point, so there is a Reconciliation that's happening on the uh admin standby and the admin active which can both be receiving traffic in this transitionary state. Both of them are creating flows.

B

There is a two-way sync, that's happening.

D

Is that somehow capturing the spec is a I'm, not sure I? You know previously talk about that. You know, for example, if one of the active one of standby, but if the standby receive any traffic directly from the network, not from the active will it create those flows, but we did not create those. We really reject the flow. Have we have we make that clear in the spec or I thought it was a you know.

D

Standby should not receive any new flows when it's in standby State directly from the network, and if we receive it, it should reject right so or it should create the state.

B

Yeah so I think we have mentioned that and I think here there is one: maybe we can if it's not sufficient, we can expand in this section also towards the end. uh It is mentioned that during this window, both the standby and the uh yeah I think during this network. The last line that is in the shared portion, yeah.

C

So Sanjay I have a question so in in that particular time. You know the state of standby is different than what it is during. You know before the plans, whichever has begun right right. So what is the? What is that state called.

B

I think it's called pre uh pre-standalone.

E

No, no yeah I'll try to explain Yeah. So basically the stable state is active standby, okay, so the standby will uh standby will drop the packets whenever it is, the role says standby: it will never forward packets. So during plant switchover there is a role transition that is happening. It is not like standby will go to Standalone right away or it will remain in Standalone. So what happens is this so say stable, state of active standby and the controller will say uh so.

E

Let's say they we decide to do plan to switch over where we want to take the active site for maintenance, so we will go and issue uh switch over uh APA to the active, so then active. What it will do is it has to now uh do a switch over towards the standby right. So first it will send the switch over event from the active side to the standby. So that's the top top most line that you see there switched to primary water right. So basically it will go to the standby.

E

The standby will become active because there is a switcher, even that is going on where it is going to take over and eventually become going to become a standalone. So first it will become active. That means it is. It is basically it is setting up itself to switch traffic, so it will become active.

E

It won't receive traffic yet because the left side is still uh as as its route advertised with a better path right, better ASN, so the right side becomes active, it gets itself set up, for you, know switching traffic and it will be there in active. So at this moment both will be active.

D

I mean this part: I have a question right. So if you are saying at this very moment the the standby becomes active, then what about? If that, because act, the previous active, still receiving the traffic and will send the traffic to the to this to the standby it sinks right. So all the the the you know, the the previous actor will stop. Sending the you know: uh encapsulated the pack same packet and sent to the standby.

E

Yeah, so the sync will all happen. The same thing so active see, active active is like.

D

I'm asking a question that what is a previous active, the behavior in terms of receiving a scene package.

B

It is continuing, it is continuing to send it to the peer board, so.

D

Actively, okay, okay, sure, sorry, okay! So if it's sending the same packet to the new active and then is that expectation, the new active? What is the petition of this new active do with this? uh With this impact received from the previous active.

E

It will it will sing that it will take the sync and it will create the state. So basically, it's like uh okay.

D

I'm saying that uh when, uh when devising active state, it will always, whenever you receive a uh you know, encapsulate the same pack, you've always created a state.

E

If, if the role is in active state, if it gets the flow sync, it will create state is that your questions are adding catch it yeah. That's no.

D

I saw the the active should not receive okay, so yeah, okay, so I think that's, maybe not Define your spec. You are saying that even if the, even if the Deep view is really is active, whenever you receive an encapsulated the same packet it will, it will replicate the state right.

E

Normally, normally it is in active standby, so it won't receive flow sync requester, but during this window of plan to switch over there is a small window where they are. They will act as active active in order to uh take care of the traffic that is coming on both sides right so yeah. So that is the duration during which it will operate in that manner.

D

No, no, but but I guess um you know uh hanif. The question was that do we introduce a because uh because each each device only know its own State? These are active. Standby, doesn't know this okay, this combined active active state right. So so, therefore, are we going to introduce a new state? You.

E

Know yeah Gohan there is no State. Yes, you are right, active will process any flossing request coming towards it. That's how it is it is designed, but in the stable state where, whenever it is active standby, it won't receive losing request, because standby won't send flowsync requests.

C

Okay, actually yeah, you know if if we say that in active, so let's say for for a time being, if you're a true active before the you know and if they stand by for some reason you know uh receives the traffic and then sends you something right. So you will still sin. You know you know, take take that uh uh flow.

E

Standby will drop the packets if fatality is enrolled standby. It will never take the packages yeah.

B

Not create the flu and send it yeah.

C

No, but but my question is that you know in in when you are in this, let's say in a true state right when you are active and there is a standby, so I'm, coming back to the same thing as Gohan is saying that in that there has to be a new state that is defined where it says that okay, you know you are willing to take a flow sync, although you are in the active state right and you are willing to even send out the you know uh flow sync when you are in the active state.

C

So this is a somehow there is some sort of a discrepancy that I see uh when somebody get into some State there is. There is some some things overlapping right there is. Functionality has to be very clear-cut as to what state means what right? What is your role uh as per what state right so somehow.

E

See the behavior see the behavior is in case of active standby. Strict active standby, yes, active, won't receive any flow. Sync request right. The active stand by uh behavior is going is becoming a active, active behavior during this plan to switch over case. That's why it is basically it is taking the flow sync request and it is handling it. Yes, you are right. You see that the active doesn't know whether the other guy is standby or not so anytime it gets closing request.

E

Yes, it will process it, but the the idea is that I mean see it's in the stable state. It is going to be active standby, like I, mean basically based on whatever pairing mechanism. We can. You know uh we ensure that they are inactive standby and things like that and standby is, you know, uh expect I mean standby, we don't we draw packets, we don't set up the connections or anything like that which is resulting in you know. Flow syncreek was going towards stack two.

B

So, in other words, uh the thing is on the active side as rules right as uh this thing, we can say on the active side uh and we always forward traffic and any new traffic that is coming. uh We will we will sync that uh new state in state to the peer right, that is the active States. This thing at the same time when, in the active state currently, if there is a flow sync packet that comes from the peer, we still process it and insert the flow. That is the active State on.

A

The standby State, please continue yeah.

B

On the standbys, uh if we are in the standby state for any traffic that comes in, we will drop the traffic if it reaches on the standby. So those are the rules. I think uh your this thing is uh so in with these two rules. What happens is the active receiving flow sync package from.

D

The field only happens here in the right window: I I didn't quite capture the rule on the standby. What is the standby, can you can you say it again? Yeah.

B

Sure understand by any traffic that is coming in uh to be forwarded is always dropped, but we will receive any flow sync packet that is coming in and create the state on the standby, okay, okay, one thing.

C

I do in this condition the.

B

Only the only condition the only window where we can receive flow sync packets when we are in active state, is this uh transitionary state, yeah but I? Think for debugging or for more. This thing we could have, we could create, could have created a new state but effectively this active active where we are actually are inactive and also receiving flow sync packet from the peer. uh It only happens practically only in this small window right.

C

B

C

To capture a question: if we have this transient state where we are actually working as active active- and there is no really problem with that, why do we need to have an explosive standby state that I assume needs to be switched to active by the control plane?

C

uh What is the benefit of having a standby standby State, as opposed to just working in active, active all the time.

E

Yeah so I'm full full active, active request, even more a lot of their support to be to be handled. So, first of all, if it is active, active forward flow can come to one side. Reverse flow can go to other side, and there can be. You know it is tough to drag the sequence, numbers State tracking and that can be flow shift. That can happen dynamically, like a lot more things to be taken. Care of.

C

E

C

Not saying that we will have two directions going to different devices: I'm, just saying that any no, no any one of them is ready to learn and sync a new flow.

E

No, no, if you say active, active automatically, traffic can come to any one of them. Their forward flow can come to one of them. Reverse flow can go to other one.

C

I, don't really agree with that you're saying you will have the transient state for some time. You don't expect One Direction going in one way uh for the same flow and then the opposite direction going to the second device. You expect that there will be a point in time where packets that belong to the same flow will go from one package.

E

Packet for the same flow will go only on one device, but what I'm saying is forward flow and reverse flow right, see in full-fledged active active? What is going to happen is both devices are going to advertise the virtual IP with the same cost right in.

B

Steady state it can keep switching between the two.

C

I'm, just I'm just saying active active, would be.

B

C

Keeping a secondary device, an active role means that it is ready to take over at any time without explicit notification.

E

No, so the active activity that that there are all those activity challenges will be will come into place during this short period as well. But there is a Reconciliation happen because we know the fact that it is going to switch over uh towards the you know, Standalone, and we will reconcile some of the things so basically yeah. There is a short period where it will become active active. But if you have to be active active in a stable State, then you have to you know you have to take up even more.

E

You know other uh things like, as they were saying uh the sequence number tracking uh the state change. um There will be dynamic flow shift from one side to another side. So all those challenges will come.

F

Yeah I I, agree with Bucky I. Think active active is so much more complicated, State machine right, but back in my question is in this case: why do you so the standalone? You know I think it is switching to an active State, a little too early. You know. Why could why can't it stay until uh you know the when it really becomes active and can actually start forwarding the traffic until that point, why can't it stay in the standalone.

E

We don't know when the wind, the other side is going to withdraw, I mean we want to control it right, coordinate right in the sense before we withdraw the route from the existing act, because as soon as it withdraw the network and anytime steer the traffic towards the new towards the new one. So in order to coordinate that the node that is going to withdraw first tells the other node you go active, prepare yourself to receive traffic.

E

So that way we don't miss anything, because if you don't do that, then it will still be in standby and if at all, traffic gets tiered towards that stand, we will start dropping the packets.

B

Yeah so once we withdraw withdraw the route under this thing right on the old primary side uh there we don't know when it'll, when traffic might uh start hitting the stand by it from the network. So that is why the it moves to that. Just before we we draw the route we prep by going to active state, so that anything once that uh route is withdrawn on the primary side. On the left side, we are ready to forward traffic on the right.

F

Understood: okay,.

A

So Sanjay and Balki in each card in the ha pair uh only um has half of those flows that it is active uh role that it has active role on right. So um I'm wondering when we are in that active, active mode or whatever right, and we get the sync message when we process it. Are we going to check that? Okay? This is the flow that I am the active.

A

uh You know the have the primary role for, and if that is not the case, that's only when we will basically add it to the flow table.

E

Yeah, the role is maintained at the per virtual IP level, and there is uh FSM running at the per virtual IP level, so basically uh for the virtual IP, for which it is active, the the FSM will run in such a way that it will tell the peer to go to active because the peer is standby and for the FSM, where the local is stand by uh the peer is already active. So there is no that particular transition is not there.

E

A

Each time we get the sync message: will we know that you know that's the VIP, that we should be adding the flows for or we are already active and is there a check of that sort.

E

How do you want check, hiding I didn't get that.

A

If we get a sync message, you know um that has some flows that is been received by the previous active that was spared right.

A

um Any time we get the flow sync message: do we check whether we are the active for that flow or standby for that flow and then add it to the flow table or there is no such a check.

E

There is a seat, uh everything is tied to uh the uh whatever the eni and as well as the virtual IP right, because, ultimately, all the flows have to be tied to some FSM, uh because the FSM is basically running at the per virtuality level and it will address only those flows that are associated for that virtual IB. Okay,.

D

So I remember, you know, for the, for the active you also need to, uh you know, send send the sync message to the other side right. So you know when, when you met the the standby to be active, uh does it does he also send those uh uh sync message to the previous active or are because it's not paired, so he doesn't Ascend. That message.

D

You know during the switch over time.

F

So go and if I understand it right, I know but I'm not answering for accurate I want to understand your question first right, so when, in this the moment it turns active that doesn't mean it would immediately start receiving traffic right, um the better as in prefix, need to be advertised and the other guy should withdraw it. Only after that, it would start to see me right so I understand. There is no explicit hand between these two end.

F

Points to you know connect the dots between when somebody is not becoming active and this guy can become active, but the flow of traffic would never be duplicated right. So when my.

D

Understanding is.

F

D

No, no, that part I'm, not sure right so because, uh because uh you know there is no uh exactly time guarantee that when the prefix will be rejoined, switch to the uh the new act you like, because that's a that's the reason I guess they will want to make the the standby to be new active first, so ready to take the traffic and then to the do the prefix resource. Therefore you know you basically have a backup and it will never. The traffic start Landing to the new active.

D

The new active would send the packets uh to the VM right so, but my question is like okay I understand that part now, but the question is uh for the new active: does it also send the flow sync and to to the pre to the previous active at this moment or no.

E

Yeah go ahead. Sorry.

E

Yeah the activity active. The role active is like uh uh like that, that the the regular active role where it will do flow sync request with the other side as well, so during the activity affluencing can go either way depending on how the packets are landing, but then later it will. There is another. There is a newer state right.

E

As you see in the picture, there is something called a pre-standalone so when it gets into that pre-standalone state, that is the state which uh it will get into after um the address has been withdrawn from the old node and also there is a convergence time period after which it there is. There is that priest in the pre-standalone state. uh It won't do any flow sync towards the uh towards the other side, because it is pretty much at that point uh determined that the other side is going to go down.

E

So it won't do any flowsing requirements towards the old node, but whatever that is coming from the old node, it will be taken. So it is, it is a different kind of role, not a not a kind of Standalone role. It will take whatever that is coming. uh Like I mean in the form of flow sync response, what are the flowsing requests? Whatever that was sent some time back? There can be flossing response coming back, so those things it can take, but it doesn't send any flow.

E

This new node, which is in pre-standard, doesn't send any flow. Sync requests from that state onwards and then, after that, there is a timeout eventually after which, which is the you know, kind of you know the uh the drain timeout right kind of, and then after the.

C

Piece- and this is the bulky- this is the same timeout as the convergence time. You said.

E

uh Convergence time uh is the is a network level. This thing that might be bigger. uh This one is like more of a I mean I.

B

Suggest to drain any traffic in I mean any packets in flight.

C

Yeah, because you you talk about two times, one is basically.

B

This is the control packet control packets in flight, so the first one is the timeout for the network for bgp, for example, for it took converge and the second one is. uh If there were uh like bulky, was saying, there's some act or something on the wire. We just want to keep it so that.

C

And how do you, how do you determine both those times? How much uh you have to wait before you say that okay, I'm done or or basically either the traffic has been drained out or the bgp has been converged? Are there certain things you are going to.

B

C

B

C

So you are not monitoring anything on the network to see that whether or not uh it has been done.

B

Yeah, so this is I think, depending on the network breadth or something, for example, we need some input to say how much is the? What is the convergence time for the network? For example, we won't, we don't have yeah it's not to. We cannot know. We don't attempt to know whether uh somewhere else the network has converged or not.

E

Right yeah so based on the deployment yeah, we should keep tuning it and then we need to configure based on the tuned. You know uh value that we see fit for the deployment.

B

Keeps it simple, but it needs some input from that Sanjay.

C

Yeah, sorry, sorry I interrupted you you can complete. Probably you know as your your thought on timeout, um a part where you have to wait before the drain. You know the before you. You have to wait for the traffic to drain and then then what happens? Hey.

F

Honey sorry I have a question before that: okay,.

C

Okay, all right.

F

Okay, I think what you said is you once it becomes start getting.

D

You know sorry, yeah I have a question even before that.

D

So this this is a pretty Standalone. I cannot find in the h3a state machine. Where does it get defined? So it's not even defining your HD machine.

D

If you, if you screw up, are you ready to finish the machine? It's not it's not defined there. I cannot find it and I try to search here. You know it appears that just at the end of the document there's a pretty Standalone state but there's a never been defined. What is that state? What is the behavior of the state.

B

In the document oh I see in the in the state machine itself, I think that's this thing. I'll add that uh this thing go ahead.

D

B

Yeah, so in the this thing right, the comparative start saying: yeah I'll, add the state there I think this is the switch over part, so I think that's which our part kind of it just says: there's a switch over, but the state there. The pre-standalone is not I'll. Add that step.

C

D

And what is the behavior right so we need to Define that behavior as well.

B

Yep, okay, we'll do so I think from your uh go on from your question right, so in active, we'll I think the same rule that we said right when we forward traffic and any any state that we get, uh we will sync it to the standby uh to the peer and in Standalone State. We don't think anything. So that applies to the pre-standal loan too.

B

Does it make sense? So, in the first half of the in the top portion on the left uh left side state machine uh till it switches over to pre-standard, we are syncing state to the peer. Once we move to the Standalone, either the pre-standal or the Standalone. We stop syncing straight to the appear, because we are Standalone. We don't have a PR uh so to say.

D

F

Okay, that may be the answer to my question, but let me clarify my question, so you were saying that um during the transition once it becomes active um and it would start receiving the new flows, it would not sync the flows to the peer for a period of time, and my question is why right I think I think why wouldn't we regard as long as we have a peer whether the pra is in you know into whatever state it is as long as an active has appeared, it should sink the state whenever it creates a new state.

F

Why the timeout drain period I didn't understand that part.

F

B

From you are saying, from the pre-standalone down, why are we not sinking state? Is that what no no no I'm.

F

Referring to a statement that was made earlier, uh you know during the transition once you start receiving the new flows, it would not sink the flow state to the other side for a certain timeout is that true did I hear that.

B

No, no, no I, think no I think uh like we said during during the active during the uh initial part, when it is an active, we are still sinking state right. Yes uh right, we are always thinking straight to the uh to the peer, and only when we transition to any of the Standalone roles is when we stop syncing, because we are Standalone, the notion is, we are Standalone. We are not.

F

Clear, how about the the reverse right coming out of Standalone to active.

B

Coming out of Standalone to act from so that will be the peering thing that we did so we have to go through the peering. uh Do the bulk sink once bulk? Sync is complete the previous one, two two sections above.

F

Let's continue the conversation, maybe I heard it wrong. Ignore me for.

B

Four of them doesn't add up, you can print okay,.

B

Yeah so I um yeah, so just to complete the picture. So this thing happens and after this the uh the left side uh active goes down comes up and then we restart the whatever was the flow for the uh peering right. So that's the idea.

A

I have a question about uh when we start the when the two cards are paid for, ha and uh you know uh we are trying to bulk sync, there must be a timeout for bugs and right um um or not. That is a question.

A

um The the point here is that you know if we start to sing just because both have been marked active um for certain flows and active or other set of flows.

A

um It may not be necessary that they have actually learned the flows or there may be no flows yet that are learned are inserted, and um in that case, what happens? Do we exit the bulk sink.

B

Yeah, so you mean uh yeah, so that will be pulse. So you what you mean is, let's say there is a whip which is active yeah on one side, uh uh but there are no real. There are no flows, really it's an empty.

A

B

There's not been no traffic for that whip yet or it's not there there's nothing learned.

A

B

That is the case yeah the person, the uh active would say that it would walk through the flows, find that whatever is uh there, it would sink. If there is nothing, it would say, bulking done and move the state machine forward.

A

I see and how about the grpc? How is it handled, like suppose, if it's even more uneven on each side, like one side is like there are no flows land yet, for those set of uh you know, flows that it's supposed to be active on. The other side has some flows, and it's trying to sink so is that handled um yeah.

B

So each state machine is independent. It's per whip, like bulky, was saying.

A

And the grpc will remain.

D

Sure yeah, maybe a question being asked right so but I couldn't find the description of this pre-standalone and there is a timeout I didn't I couldn't find the description, the plan to switch over section. Why do we have this pretty Standalone at the timeout?

D

Maybe that's been yeah.

B

So this is for sure, so there is I think there is a bulky. Please uh extend as I say, I think query this thing right. uh So during this day, till till before all the state before this pre-standalone go on, there have been sync packets that are being sent being sent to the uh uh peer side right to the left side.

B

It might so happen that once we got this, there might still be some packets in uh there's. There was still active sync going on so there might be, although we got the message that uh uh they switch over to Standalone from the left side to the right, there might be some of these control packets that are still in transit, so the second in that pre-standalone timeout, is just to drain that those control packet, those control messages from the channel.

D

um Control messages from uh and from from uh from where to where the meaning from the steam controller or from the.

B

From the dpu so first say, for example, one of the examples is let's say we did receive on the right side right. We received a new flow and we sent a DPO sync message to the uh to the peer right. Now we are waiting for that act to come back to Market in the flow right. So we don't need to forward anymore to the this thing, so that act may come a little late and we that timeout is for because in once we get to stand alone. uh We will not it's not.

B

We will not be expecting any packets to come from the pier when we are in standalone. So.

C

B

There is this window where you might receive, because you are, you are already forwarded, uh encapsulated packet to the other side. You are waiting for the ACT to come back from the peer. You need some windows so that you can uh handle those packets, because once you switch over to full Standalone those packets are not uh legal packets to receive right, because we we don't want this thing so that just makes that pre-standalone state so that we can drain whatever we had sent just before. We switch to standalone so um basically.

A

Keep receiving those packets but not.

B

Spend anymore not send it exactly.

D

Are we initiating a boxing when we do the plan switch over or we are not I showed in that moment? Isn't that the you know the the standby, the active and standby are in perfect sink.

B

They are, they are gone, there's no, no bulk sink involved in this uh window. uh It might be, for example, we were doing a DP, uh that's a data path, sync yeah, the new flow which which arrived on the right side. We sent the data path, sync packet to the left. Now we are waiting for the the this thing to act to come back right. If we moved over immediately to Standalone, we would drop the attack.

B

We don't want to do that, so this pretty Standalone kind of gives that window where we can still handle those acts properly and then switch over to full standular.

B

But we want to distinguish that state from active because we don't want to be initiating any sync anymore, like whatever was there we can receive, but any new flows that are coming. We don't want to sync them again to the people. We don't want to continue sinking from to the peer, but we just want to accept any acts that are coming uh from whatever we had sent in sent from the previous state, so that pre-standalone is that uh the timeout is for that control, plane, uh flush to happen.

D

ah I see so so.

F

It's mainly like.

D

We're not uh we're not uh when that's actively sending any sync message to the peer, but but we may send those sync message to appear before so if we are comes back I still, you know honor them correct. We still need to process it correct okay, so this is more like a gracefully shutdown kind of thing. Okay, do.

G

You think maybe then maybe, uh instead of calling it pre-standalone call it something with the word drain in it, so that people know what it means without you know giving a meaning like that. Is there any kind of a drain connotation with that state.

C

B

I think that we could yeah any.

G

Just an idea, because then it's kind of self-explanatory during The Question keep coming up.

E

Yeah I think uh yeah, wherever we present it to user, maybe yeah we can. We can add some new.

B

Can have said three years, yeah.

G

Just an idea, this discussion is really fascinating, but it I keep thinking of a different, a related topic, and that is let's say we agreed on this and it's all working and people Implement on different platforms. How are we going to know it's working like this other than the final outcome? How can we have a high fidelity observation of this thing in action and do test cases and verify that it's behaving the way we expect and the reason I asked.

G

That is because, if we just wrote a bunch of code that did all this, we might have a hard time, knowing how it's behaving under the hood for doing all this, without putting some kinds of I don't know, maybe generating High Fidelity events in a test situation that send out some kind of message that tells us this States I'm transitioning to this state or I'm, doing this or I'm doing that, so that some outside system can observe it behaving the way we expect it otherwise kind of like very difficult to know.

G

What's going on, have you guys, given that any thought.

B

Look at us all these, uh these State transitions. All the messaging is going through the uh through the control plan, sync channel right, so if you did Monitor and we keep the state machine transitions clean and actually monitor these events that are passing through, would that give enough visibility? I thought the idea was that would give since it's going through the same control.

D

I think there is a multiple aspect to the the answer to the question right, so you know definitely, you know there will be unit tests to test all this state transition right. So you know if you are not satisfied that I think there is a uh you know, the uh this model checking work, which uh you know, there's a language like TOA plus um to that to fully validate the model right. So I think we have been.

D

You know using that developing the TOA model Plus for the for our Gemini Project, which is also you know, related to the switch over. So we have a quite a few experience on that to you know to to come up with a formal verification model to verify the you know, the the state transition and the you know. Another question is that maybe you know we should develop a test to uh just to set up the environment and, to you know, continuous switch over and see if there's any traffic job right.

D

So you know, because uh in the planet switch over I think what I'm hearing is that basically there's? No, there should be no expected flow jobs or anything like that right. So.

G

D

Think those are.

G

All good partial answers, but I'm thinking, for example, let's say we're besides, let's say the paper analysis and the uh let's say: there's a static analysis and models and whatnot that verify it and unit tests, but when the system's under load in a real Network in a real situation and something behaves funny or we just want to know if it's behaving correctly, even if it looks like it's correct on the outside, it may not be completely correct on the inside.

G

How would we, how would we observe that and and confirm, even under load full load with lots of corner cases.

D

um Then you'll do like a stress test right, so you know you set up the environment and do you know thousands of uh ten thousands of streets over and see if your name package jobs? Yes,.

G

And let's say: let's say in the 3000th case: there's a packet drop. Would we have enough information to know what happened why it went wrong in that one, instant.

D

The answer, then, we need to that failed. The test case and people need to you know, start investigating. Why there's a package of those case.

G

Right would we have enough information to diagnose what happened that one time out of ten thousand after the fact, because you may not be able to reproduce it these kinds.

D

Of things, the idea is to Repro I mean that's a typical debugging. You know you have to reproduce it right so to to you know this is kind of value. You know, that's a typical debug I.

G

Just trying to plant some plant some seeds here that sometimes you can't reproduce it or you just pray and hope after trying for three more weeks, it happens one more time. I just want to put that out there right that. Do we have the mechanisms to be able to do it without agony right, because you know these kind of systems are super complex and you may never actually reproduce it exactly the way it happened. The first time when there's timings.

B

And memory there are some: there are some visibility. For example, there is flow level information on either DPO I think the ddpus can maintain flow level. Information, for example, state right. Would there be What state? Were we in? Did we receive an act back or where do you send the packet out uh so that could help I mean post uh postmodern, but after we see that there is one flow that is failing on the 3000 uh switch over. Something is failing. We know what is the flow that is failing yeah anyway,.

D

Go on this thing, yeah.

B

You could go back to the dpu check what the flow is. Okay,.

G

And there's a typical debugging flow yeah I just want to put some food for thought out there yeah.

A

That's a good, uh you know yeah. We should definitely have this discussion in the future. What are the best ways to debug this, just a traffic data traffic? Will that be sufficient to determine you know if the states are working and yeah? How do we monitor that yeah? That could be great.

G

I spent six months on an HF system before and and so I'm trying to say, leverage that painful experience and just put some bugs in people's ears for now.

A

D

Good so sure I mean I mean Christina is not here right, so time check, I think the.

A

D

A

D

Think uh you know we are I, think we're. You know we clear on the planet street, so maybe next time I think we should go over the unplanned streets over again to see you know because I still have some questions. You know how do we measure those traffic disruption? You know those kind of is critical to the you know to the operation. Can we do that in the next meeting to to discuss over the unplanned the switch over yeah yeah.

B

Sure definitely and.

A

Also, when I went through the document, I couldn't find too much information on re-transmission. So maybe if we can cover some of that next time as well, that would be great.

B

The transmission from the DP, from which one, what were you thinking.

A

um So basically, uh yeah data path, sync right for the packets- if you don't have acknowledgments.

B

I see I think somewhere that this thing you had mentioned, maybe we can talk about it. If time is this thing right, uh we can go through the path, I think the network itself. uh So the idea here is because we are not keeping state. If there was a packet that was dropped there was any network drops. We can probably go over next time and see what happens there, uh the I, but in general the idea is the uh plant retransmission covers that too. We can go over there.

A

Okay, we couldn't go over the overlay CMP, uh how it uh how the ha can be. You know, use that design as well. I think next time, Gohan and Prince can maybe talk about that yeah um yeah. Thank you, everyone for joining um Jay. If you could, please stop the recording.

C

Yeah I'll start the recording now thank.

A

You thank you. Everyone.