DASH High Availability WG, 6 Sep 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: DASH High Availability Working Group Sep 6 2022

Description

Continuance of AMD proposal - will need another session next week.
Next session: Pre-emption case to be discussed, SAI Spec, Overlay HA

A

Litmus test, not the only definition, but if.

B

You don't have enough information to write a test, then it's not done being specified thanks.

C

Well I'll continue with the pdf. Is that okay? uh I think that so it's available on the as uh markdown and uh thing too, but we are using this. If you want, we can continue, or if you can, you prefer to move to the md. We can use that.

B

Does anyone have a preference on the call.

B

No, this looks fine.

C

So I think we had, I remember right. I think we had ended somewhere around this section.

C

The next section was the state machine itself, so this is something that the proposing is implemented on the sonic side.

C

Here I think it's a very it's a simple state machine, not much so there's boot up and then uh so there are two. I think I'll just point out what seems to be otherwise it's really uh sanjay yeah.

B

Hey: hey: hey uh yeah! I just started a didn't question I chance to discuss last 10 weeks. So if you scroll up uh there was this data path, synchronization right so yeah just before we go to the next uh yeah here. um So here yeah. This explains about uh the tcp thin packet, uh how it would flow through uh uh the uh dsc, the dpus right so like, oh any any thoughts on how uh it would work for udp packets. Let's say if it's a blast of udp traffic right like so.

B

How is that handled.

C

Right so till the connection is created on the primary and the secondary, and then the act comes back from the secondary. The udp packets are forwarded to the secondary, but once the connection is has been created, uh then the the primary only forwards. There is no, it's short circuited.

B

So, okay so like until then, all the in-flight packets will just be forwarded via the secondary client.

B

Sorry like it could be if.

C

I'm sorry, I'm not sure. If it is me, uh it got cut off.

B

There's a little cutting.

B

Question was like if uh uh there are inflate packets going through the secondary uh and if the act came back for the sink, then the that could be out of order. Packets right, like.

C

I think I kind of got you're saying uh uh till the connection gets established. Whatever was in flight, whatever was sent to the secondary or getting looped back, I mean uh was getting her pinned back uh once it gets installed at that moment. If there are new packets that come in, that hit the connection, they will be, uh I think so bulky any.

A

Yeah yeah there will be yeah if, if there are any packets after that comes, they will be short-circuited and there will be out of order yeah, it's it's udp, which is basically expected of best effort right yeah. So.

B

Oh okay, yeah, so yeah yeah. I just want to make sure uh uh yeah okay. So there are other folders that is expected uh in case of udp.

C

Yeah at that uh transitionary moment when from no connections later, when you get into a connection.

B

Yeah sure, thanks.

C

Any other questions so far in the previous sections.

B

No yeah, we are good. I think you can continue.

C

um So here I think the beginning is once I think, the uh uh just to clarify it just to set the stage the state mission tries to handle both the active and the standards. So this is a implementation of the state machine. It's not necessarily a sequence diagram of how the active and the standby are moving through the stages right, because that's so those are defined as flow um message flows below in a later section, but this kind of covers the whole state machine as a as a pole.

C

So it's kind of I think you'll need to stitch together a little bit of if the active was in the state. What would the stand by between nothing? We need to stitch together if you're, just looking at the state machine in isolation, uh just to keep that in mind right so here in this uh picture.

C

So there are some external input that is happening like, for example, config being complete, or there are some other inputs that are coming from external resistance. This could be from the gnmi typically other than that. There are also some.

C

There is one direction, so this is running on sonic. So there's one direction where there are uh api calls going towards the psi sdk, so those are in these uh black arrows and the psi sdk itself are represented as these orange bubbles on the side right, so that it's running in its own thing, just for connecting what what we are getting from the psi sdk layer, it's represented.

C

So initially, when the uh when the box comes up, we need there are, there are a set of capabilities that needs to be determined or exchanged between the peers right. These capabilities themselves can be uh capabilities of uh inherent to the sonic implementation, like version or what versions we support things like that.

C

There are also capabilities that the uh sdk, the underlying implementation, like, for example, different vendors, may have different capabilities of characteristics that they can interrupt with or cannot interpret with the other side the other year.

C

So those are some capabilities, so there is one stage where we are getting capabilities from the local side and afterwards, when there is uh when we, when we do see a peer uh and we are peering with the peer, there is an exchange of capabilities, and these capabilities are first processed on the in the state machine itself to see if there are any, uh if the, if it's compatible for at the top level capabilities level and then the same peer capabilities are permanent, are passed down to the underlying implementation via this isdk, where the underlying implementation may check its own, do its own compatibility checks right.

C

So I'm expand. The expectation is each implementation may have different things. For example, uh one implementation may have a different set of uh parameters right, uh for example, if it was a amd center, we might have different parameters. What we want to validate again sizes or other other hardware capabilities that we want to make sure the peer is capable or not capable of so we can either make it incompatible or even uh fine tune what we sync across to the other side and manage what you're seeing with the other side.

C

So those can be done uh with the process, peer capabilities check and then the return value will say how to handle the capabilities, whether it's incompatible or fully compatible or compatible with some new features. If I disabled things.

C

Then uh so I think here we have the boot of the init, so the unit itself there can be configuration errors, I'm just coming back to the first thing. So that was the capabilities that there is in it.

C

There could be configuration errors where, for example, the pier are the local ipr, the prop is not configured or the parameters are not correct, etc, so it could go back to uh it could go to a config error and then a config complete or external trigger that says: okay, the configuration is uh complete now at that point we do the recheck again and then there is a stage where we try to connect to the pier and uh there is a timeout.

C

If there is a timeout, then then the local node assumes that there is no pier right and uh starts. You know uh going towards it. It becomes a standalone primary right uh and then, when it is in standalone primary mode, uh it starts getting ready to the node starts, getting ready to forward traffic to start start uh building connection state and uh forwarded.

C

uh In this stage, I think, in this stage or even either, on the uh on the newly coming up note, which is on the left or there is a standalone primary, which is on the right side. Roughly uh there could be a peer connect right and then, when there is a peer connect, we move on to uh do some compatibility check of whatever capabilities we were talking about and once it is established to be compatible if it is not compatible. I think again, it goes back to the configurator and wait for external uh resolution.

C

If it's all compatible, then we get on to uh the the state machine kind of triggers uh the bulk sync right. So why are they starting messages to the underlying implementation and then the newly coming up, node? Whatever is the node that is coming up? That start that triggers a pulling of the flow state from the pier and once that is done, which is indicated by because the whatever was the standalone primary side. That knows all the state that it's thinking once that is done.

C

uh There is something that is relayed back via the synchronous message uh through the from peer to peer to sonic and then to the sdk implementation itself, the underlying implementation, which triggers the uh completion of the bulk sector.

C

uh Just to note during this time there might be, I think, once we have set up the pier and when the bulk sink is happening, uh new flows that are getting established, uh they are still getting getting synced via the data path. uh Sync channel. I think we had briefly touched upon this in the last meeting uh here. It is not evident in this picture, but I think we will see that uh in the message flows that are there. I think that it's more clearer that there is also the.

C

Data path sync happening along at this stage,.

C

I think this was a request that we got when we were implementing from microsoft, which was uh once the bulk sync is done with whether the node takes over whatever was was its configured role is controlled by the controller right.

C

The there is an api call from the controller. uh There is a call to say, activate hr, so after the sync is done, we don't take over the role itself. So if we continue, if there was a configured standby which was uh primary forwarding traffic as the primary we don't uh take over and when the configured primary, uh let me let me call that the admin active and that means stand viral, and then there is an operational state.

C

So the admin uh standby was operationally active right when the active head was not there and when the active is coming up. The con admin active is coming up at that stage. uh For the admin active to take over as the configured active uh there is a. It is not done uh as soon as it comes up, but it is actually controlled by the controller.

C

So there is more predictability and.

C

In the transitions right, so that that is activated and once the activated role is uh invoked, uh the state machine checks whether it is configured as primary or secondary. If it is configured as secondary, it doesn't activate secondary lets the underlying implementation of what needs to be done and waits for the underlying implementation, the sisdk or the hardware implementation to act back, and then it transitions to the second label. If it was uh configured as the primary, then there is an activate primary which goes through goes to the underlying implementation comes back and then transitions.

C

So there are some activities that happen during activate primary, for example, if we have to the network control plane like, for example, bgp vfd or whatever we are using for that, the active might need to change the advertisement so that it starts attracting traffic right.

C

Those activities are done uh when the activate primary, when it transitions from activate memory to the front.

C

C

Yeah so uh sanjay thanks a lot for for this explanation. Just one quick question: um I'm not sure whether you explained this one or not before, but I believe you're running this state machine, one state machine per eni, correct; okay, not for ena. This is per uh per bit.

C

Per bit, if remember, we had said that uh there are two whips: the enis get associated to either one of the whips where the mirror, where the states are mirrored so on each uh dpu. There are two whips, one which is configured as actually one group is configured as negative. If you remember, I mean as a standby, if you remember the eni based load balance that we talked about right, so each state machine is for each with each whip is either in uh active or standby, primary or second paper. So.

C

But that doesn't make sense, because you know you are essentially our sinking and all those activities that you have mentioned in this state machine. It is, these are all per party and.

B

I so I would I would expect this one machine.

C

Per dna yeah, you could think of it uh as one step, one uh level of indirection. So there is all these enis, the the eni is associated to one of the whips and the whips runs the state bishop. Because when the note comes up, uh we are, for example, we are advertising only a bit into the network, but we.

B

C

Advertising per eni, we are not advertising anything. What we control is only perfect right, so it's like a level of interaction so that yeah indirectly it is associated to a ni, but uh the eni itself is associated with. So there might be whips. That's 30 ni's that are associated to a bit. There's one state machine uh aggregate for all those 30 yeah.

B

But but I want to ask this uh clarify this right because because maybe you know between two vips there's uh you know my you know there is a multiple yin. I you know to be. You know just between these two weight pairs right so, but uh you know we, I think we. When we do the sync we should. We do not sync, because maybe you know another, you know in this wave. They have other uni that is not replicated between these two vape pairs.

B

So therefore we need to determine what you and I need to be synced why you are not right. So.

C

Yeah, that's right! So if there were, for example, vpa and vp repay has 30 ni's associated to it, and this b has another 30 associated to it. When this state machine moves- and there is a bulk sink happening, if we're seeking the uh flows, for example, whatever is the activity is happening for those 30 right, because uh it's.

B

C

B

It's a union right so of this 30 right so.

C

Correct all for all the 30 years.

B

When we see this uh diagram, it looks like it is, run, per card per dpu, ipu, right um and then per card means. There are two whips and we basically are looking at the active one. So the active comes into picture, so it will be like running this whole state machine per whip, which is the active web. But overall it's for the card for all the active classroom.

C

Yeah, but at the same time there could be, for example, one of the whips could be. Switching you know could be the is configured as the second way, so it would follow a different uh tree right, different path. So that's why there is one uh one state machine, although they are kind of uh some of their life cycle. For example, there is a failure or boot up. They are tied together right, their fate is shared, but uh the state transitions themselves are local to them, because each of them will follow different parts.

B

And when you say peer connect and these messages that are going back and forth, how are these messages carried? Will it be what type of packet will it be? Lldpt, lb for example, or what would it be in your case.

C

uh This is its own. uh There is a heartbeat message uh if we had one section for the heartbeat message uh above, I think we cut yeah it'll be the same.

C

Yeah, I think we can- uh I didn't put because it was this thing, but we can put some uh reference. Packets.

B

Okay, yeah. That will be very important to have to see how the messages are seen, how they're carried yeah, okay,.

C

So that is the heartbeat, but all these uh ball, sync messages etc. Those.

C

The grpc channel that we talked about those messages are carried in this uh in these channels that that is different. Okay, yeah.

C

Hey sanjay and gohan, there was a mention that out of the enis that are mapped to a whip, some of them may not be synced. Can you elaborate on that.

B

Because I yeah so you know the each card advertises a single wave right so, but between uh a pair of web, a pair of cards right. So, for example, you know on car a there are 30 uni on the car b. There are also 30 uni right. So but uh it's not exactly the uh you know the the replication is happening between just between these two cards right. So maybe you know for card one. You know 15 of them. I think with a card b and then 15 is synced with a card c right.

B

So therefore you know between card uh uh uh a and b. uh When we do this, uh uh you know this uh synchronization only 15 year. I need to be synced right so than between card a and a c. Then another 15 need to be synced.

C

Right one thing to go on one thing: theoretically that could be, but I think currently I want to note here- is our whatever the implementation that we applied or whatever we have confidence in. Is uh we have this two pairs? So we have a pair of cards and the pair of cards uh are sharing the same. uh It's not like a chain it's so both vpn wip b are shared between the two same two cards and the vpa is primary on card one and standby on part.

C

Two and whip b is secondary on card one and primary on card too yeah. Theoretically, I think this can be extended to do it. The way you are saying where uh there are three cards in the or more cards. No, no, no.

B

No, no, no, no, not not that you have. You know active standby standby up right, so you you have. uh Is it you know? Basically you know it's a slicing. You are not uh really, you know, replicating the you know just between two cards right, so you know it's a you know. One car cover 15 of them. Another car cover another 15 of them right. So.

C

Correct yes, but today the peering relation is what we have tested, and this thing is between two cards where the whips are the same. Two waves are shared between the two, but theoretically the model can be extended to that, uh like what you were saying card one and car two share with a card one and card three share with b it could be extended to that. But, right now it is card one and card to share with the end that makes sense yeah.

C

But but in that model the ear nines will be mapped to the appropriate whip, not to the.

C

You always have the same set of ear eyes that are mapped right. You your case, the other way. Yes, it's the same yeah, etcetera.

B

But but you still per yeah right, so basically, this model primary and active is per unit right. So yes,.

C

Indirect, like I said, yeah it's period, it's not uh yes,.

B

So I didn't get this part the last one. What what if there is a switch over the way, uh do it do anything you know what once they get synced, then you know we can just switch over back and forth or how does that work.

C

Yeah, so I think once it is synced up the and the data path. Sync is uh going on actively at any point in time there could be a switch over either uh controlled or it could be unknown unfortunate, switcher like some event, bringing down the card or power going off or something like that.

B

What is this last secondary wait for switch over and then you know the line goes back all the way to the top.

C

Oh, it's yeah, so it is in secondary state where it is not forwarding for the secondary or the enis that are attached to the second type. So what is happening in this secondary state? It keeps getting the sync messages from the primary and it keeps updating the state after this. If there is a switch over, then it switches and then it becomes the standalone primary right. So it's actually the primary for the for the web, but it isn't standalone because there was a switch over and we lost the actor right. It moves to standalone primary.

C

uh Of course, I think, after that the primary could come up again during which state the secondary, whatever the standalone primary, might have gotten. New connections could have changed state, etc, and when the other node, whatever was appeared, comes back again, the state is synced back to the from the standalone primary to the newly coming up node. At that point, the newly coming up node would become primary right. The state mission would restart.

C

So there is uh one if that makes sense right, so there is a uh basically right. Fundamentally, there are uh at least in this this relevant. There are three three states right: there is a primary state uh or yeah. There is a primary where we are actively forwarding traffic, updating the flows and then syncing these flows, I mean syncing the flow state uh to the standby. Then there is a standby state where we are receiving flow, synchronous, closing messages and updating our state from the primary, and then there is the standalone primary state.

C

Where we are, we are acting as primary in the data path. We are doing an updating, state, etc, but there is nowhere to sink to because the peer does not exist at that point in time. So that is the state which is a standalone primary.

B

Okay, so why is the secretary only move to you know, standard primary? You know not moving to primary directly. I mean I could think you know there could be. You know you you could do switch over because of the failure or and you could do switch over because of maintenance right. So there were two kinds of uh switch over.

C

Right um yeah, so during that for during maintenance, for example, what you're saying is we may need to bring down the uh whatever was the primary for some maintenance. During that time we want to drain all the traffic to the secondary so that you can do whatever maintenance, and once that is done, you may want to switch back to bring that back and during that state.

C

It kind of, I think, for simplicity of the statements it kind of gets goes back to the standalone primary. You do whatever maintenance and then, when the primary comes back up, it has to restart everything, because it has to pull all the flows from the other side. So it with the primary uh whatever was the one that was being maintained would go back to the beginning of the state machine here.

A

Yeah, so, basically, basically, when there is a h, a relation, a high availability relationship right, then the roles are called out as primary and secondary whenever one of the nodes go down does not matter whether it is primary or the secondary, the surviving node will become a standalone because it's operating in a standalone mode. There is no peer for it, so in the standalone mode. Basically, the the functionality is that it will forward the traffic, but there is nothing to sink to because it is operating in a standalone mode.

A

Once the pier comes back up again, then the relationship is again established. All the states are synced and then the roles are, you know, set to primary secondary.

B

So could we have uh like two devices that both stand alone primary because they lost connection to each other.

A

Yeah, so that is there is this: if there is a network, partitioning happens.

A

Each of the node doesn't see the other node. Then it becomes like a split brain scenario where both of them will become standalone and whatever traffic that lands on that they will be switching traffic, but it kind of becomes tricky at that point because there can be cases where for some connections the forward traffic can land on one stand alone and the reverse traffic can land on another stand alone.

C

Yes, so there is one this in the go and I think today's bgp the way we are advertising, uh even though, if there is just let's say there are two levels of network partition right, one level is: let's say there is network partitioning where just the control traffic is not able to reach uh from peer to peer, but the rest of the network is uh fine right, so the actual data traffic uh can reach both. There is no network partitioning there, but for some strange reason there is some uh network connectivity between the two nodes.

C

Just for that control traffic is broken. uh That also would lead to this uh an active active situation, uh but in in that state what the this thing is. The way the bgp advertisements are advertised with the.

C

Preparedness prepend, uh what happens is the network? Would then, in that case, decide uh would always converge to sending traffic only to the configured primary right and it would. It would switch traffic to the standby only when the configured primary stops advertising. That is one one level of network partition. The second level of network partition is the network itself. Even for the data part traffic and for all the control traffic going between everything is really truly.

C

Actually, there is a partition in the network in that case, even for this bgp advertisements, whatever we say the whatever we this thing, even that there is no uh connectivity between the two halves right between the two sides of the system, then that would be true uh split brain where traffic could end up in either.

B

So in the split brain case, when the traffic um is broken between the two um ha cards uh cards, then based on the role that's configured, will it continue to act based on the role that is configured on the box, or will it assume active role.

C

It would assume back to, though, because once we lose connectivity, there is no way for so. If I'm getting your question right right, so there is a cardia which was uh primary for the whip, there's cardb, which is secondary for the left right. uh Then they lose connectivity. So the stan on the standby side can't be a car two side right.

C

uh There is no way to differentiate whether we it's a split plane situation or we actually lost uh the primary actually went down due to some fault right, there's no way to differentiate, so it switches over to the primary this thing. But there are things that we can do in the network, uh which is the bgp advertisements that we are sending. It can be such that, even if ah the nodes themselves go to both go to standalone primary.

C

If there is any network connectivity on the top, we can still influence such that the traffic always lands up on the configured primary. If it is active.

B

So, based on the web from bgp as prepend side of things, they can be distributed across both the standalone.

B

Cards but it will forward for all.

C

Corrections yeah if any traffic hits it, but hopefully, if bgp is uh if the pgp connectivity is still there on the network, then all the network would be sending all the traffic only to the primary site. So there is not really disruption, but the standby configured standby would be in the primary state but not receiving any traffic, because the network is not steering any traffic tools.

B

But it will forward back.

C

Yes, if it receives any traffic, okay.

C

No, I'm sorry the last point the forwarding back right and can you clarify.

A

That something so it so because of the bgp trick, the the standalone primary will, when I say standalone primary versus the configured primary right, configure primary would.

C

Be attracting all the traffic because of the s perpend.

A

But the standalone primary will not get any traffic.

C

And there is no reason it would get any traffic to forward it back right. The forwarding back scenario will not happen, will not happen, but in case it did receive what I'm trying to say is from the uh state on the card. uh There is no there's no difference between. We cannot differentiate between the two cases right where the uh we became primary because of losing the active or because of a network partition. We cannot differentiate on that card, so the card is ready to forward right.

C

If, if, let's say the primary went down and then network started steering traffic towards the standalone primary, which was a configured secondary path, that configured secondary would forward traffic right because it wasn't uh standalone state yeah, it would forward traffic to the destination not back to the primary right, not back to the primary back to the destination.

B

Okay, hey uh one question: I.

C

Don't know if you answered this, um so the activate ha roll is a separate trigger coming from the controller right. So there is always a delta time between boxing done and activated. So what happens to the new flows or flow updates that happen between.

A

C

These are, we are continuing to keep the two sides on sides in sync, so all this activated role does. Is it keeps it in whatever state it was like? Whatever the state the network was, we are not switching over or doing anything preempting making it active as the configured you know following the configured state, but if you were thinking from card one was active and car two was stand by for the strip and but the configured state was the other way around.

C

We would still keep it in the whatever was the running state till the activated road is invoked, so there is still uh traffic is still flowing. We are still uh continuing to keep it uh uh whatever was the state was before this right when we started doing this thing. So, for example, if there were new flows that were getting created, the data path would be sinking it to the other side.

C

Oh okay, okay, yeah yeah. That will continue.

B

So so, when you get, you know, for example, with a standby, get lost the um you know the heartbeat and become stand long primary and when there is a pure connected to this one. So how do they these two nodes to decide which one is actually the primary and which which side do the do the stay sink.

C

Oh you mean uh when two standalone primaries: let's say they come back uh yeah I mean they detect each other when they are playing primary yeah, yeah yeah yeah.

A

So the copied account yeah the configured the configured active is the one that will always sync the state in that case.

A

B

Basically, the state.

A

Machine that state machine when it is when the state machine detects that it is detecting it's a seeing appear and if both the peers see that they are in standalone states, then basically the the the node that is configured as uh the admin active. That is, the one that will uh sync its state to the peer, because the node that is configured as the admin active is the one which will have which will have a better cost right. It won't have prepared asn, so the traffic will only hit that. So that will be that.

A

That is the node that will have the updated state.

B

Hey um sanjay, I have a quick question on this.

C

Connectivity part right: these dpus are connected.

B

A you know connected to each other through two networks. Right as we discussed, one is a data path channel and another one is a control plane right, one. We said like a cpip we were talking about through which it is communicating all the the data path, packets.

C

So the question here is the yeah yeah.

B

Is this working? Yes? Yes,.

C

So the question.

B

Is I don't know whether it's already uh explained but.

C

Heartbeat packets, which channel do the heartbeat.

B

Packets go through and then uh do. We have heartbeats for.

C

Both for both the channels- yes, so actually these uh one thing one clarification here is this is, uh although they are, they are logically two different channels, uh because the connections are different, but they are all going through the same physical network.

B

C

They using the same ipo.

B

Using different ip.

C

Using the same energy.

B

C

B

Same mip, so does the sonic help in the heartbeat sonic, transmits and receives the heart rates? Are.

C

So that is something up for debate, so I think currently, I think we are thinking in this proposal that it is better the dpu handles the heartbeat because it could be very aggressive and we could also use uh things like bfd offload and things like that yeah. I.

B

C

Thinking, yeah yeah. So that's why.

B

C

Thought it might be better, it is done at the tpu side at the.

B

So the control.

C

B

Has to create the packet and send it to the dpu and the dpu keeps managing. It keeps resetting that.

C

Yeah, so it could be uh one implementation could be that that's what that's, what uh carrying from what we have done now right is uh the dpu creates the packet manages the whole heartbeat and only when it loses heartbeats, there's a notification that goes to sonic to say, loss of heartbeat or whatever. It is.

B

How aggressive is that? Is that, like on the order of seconds or milliseconds or.

C

It's in milliseconds now.

A

The number that we have it configured is 150 milliseconds, but the thing is, uh it has to be tuned right, based on um what microsoft's deployment requirement is. So that's why I don't want to give the number, but uh I mean by default, yeah.

B

Do you plan to reuse the bfd infrastructure uh code um in sonic and everywhere right? um Do we want to reuse that or is it a separate implementation? I guess we have not discussed that part.

C

We have not discussed okay today we don't use vfp, but it's done from the p4 side. It is done, but it's not not pfd.

B

Oh, but then we need to get this clarified in this design dock right. So because uh you know, if it's a if it's a you know, if we're closing on using the bfd, I think the sonic already have the infrastructure to do the bfd.

C

I think yeah, maybe I think we should have probably in another discussion on the heartbeat on where we want to do the heartbeat, uh since it was, I think, yeah.

B

James is working on this right now, gohan with michael.

B

The bfd in sonic is, I think, on the order of 300 milliseconds. Is that fast enough for our purposes or want to have a much faster type mechanism here? No, no! No, I think the vfd and sonic is support hardware offloading, so you can do as much as fast you can in the hardware as long as they fire a harsher support.

C

Yeah, that is correct, but you know in the same time, in if you're talking about millisecond in the dpu, that should also have to be through a hardware offloading. It can't be done in the software.

B

Right right, but that's what I'm saying right so the the sonic, the bfd is leveraging the hardware. It's the hardware bfd, it's not the software bfd it uses dx assist hardware, uploaded vfd.

B

So you can't do you can't immediately any query? You know requirements that you want.

C

Maybe I think uh uh it might make sense to have another uh focused on the heartbeat uh another discussion.

B

Yeah this is, this is just kind of the proposal, not everything's final.

B

Okay, hey um one question about the activate ha so yeah can you click like? If is that message sent by the controller to the uh like standalone primary, which is like configured by secondary, so this trigger will say you can now switch back to secondary uh okay. So how will it be sent on both nodes, um yeah, just trying to understand what the actual configured primary? uh We will operate right when it comes up.

A

Yeah, the activated role will be sent to all the nodes right. So first, when the when only one node comes up, it will come up as uh there is no, it won't detect any pier. So it will basically, it will be um kind of in the standalone right, but it doesn't take over. Then the activated role will take uh trigger will will make the node to take over as standalone and it will be in standalone and it will be switching traffic.

A

Then, when it detects the pier the pair will come up, it will get the bulk sink from the existing node and it will wait for this activation. So the control will activate that node now and at that point it will become based on its configured role. It will become either you know active or standby.

A

So, uh like you have two whips right for one of the whip, I mean the way the two whips are configured is it will be active on one node and stand be on the other node and vice versa for the other group.

A

So normally, when a node comes up, what is going to happen is for one of the web. After the activated role for one of the grips it will go to standby and for the other vape it will take over as active. Basically.

B

Yeah, I was just trying to understand the order in which uh it will happen right. Like so say, one note uh what was up and it is in the standalone primary and the second node comes up, which is the configured primary. uh So like will the controller like? Let's say the state is all synced up uh now. The activate ha roll first will come to the configured primary to close that ask it to become the memory and then the next it will come to the standalone primary to change to secondary is that the sequence or.

C

I think uh what bulky was saying was uh so on the configured secondary, which was in standalone primary mode. The activated role had already been invoked by the controller. Now the new node comes up, which is the configured primary when it is coming up uh it would. The controller would come, do activate h, a roll on the configured primary now, the all the state is synced up, because it got the activate h, a role it could move over to whatever is the configuration and take over and next.

B

I see okay, so then the actually. There is no event in the stand-alone primary.

C

At that point, yeah at that point.

B

Okay, so then, uh in the standalone primary we'll have to deactivate those flows right, meaning meaning have to go back and uh identify those rows that it's not some. Those flows that it's not supposed to have active role on, and uh uh you know mark it as such and stop forwarding right.

C

Correct there is yes, you're right.

A

C

You don't understand no.

A

Sorry, I don't understand, but there is nothing to be marked in the floor sanjay. What are you.

C

Oh, I say, oh, you mean keep the.

A

So what is going to happen is when the new node comes up right. Let's say the new node comes up, uh so there is one node that is already existing and it is operating as standalone and it has established connections and it is switching traffic. Now, when a new node comes up the new node first, what it is going to do is it is going to get the bulk sync from the existing node, and it will wait in that state ready to be you know. Basically it is it is.

A

It has been prepared up, it is ready, but it needs activation. Then only you can take on the role. So once the activation comes up. Let's say it is configured to be the active right. So what it will do is if it is configured to be active, its whip is not, it won't have representation, but that, but the node that is already existing and switching traffic is having perpendiation, because it was the only node. It was switching traffic. Now, when this active comes up and activate role, com happens, it will.

A

It will move to the active role and uh it will start advertising its its virtual ip, which will be without any preventation, and thereby traffic only will hit this particular new node that came up and the traffic won't hit the existing node anymore. After that.

C

Yeah, that's the right explanation, because what happens is that if you are a configured primary, then you are going to attract all the new traffic, all the new flows to you, but at the same time, what happens is that you know the the because the bulk sink only after the bulk sink, you basically, you know, should be- should be declared as uh active primary. Isn't it.

B

So one thing I want to say here is that normally we have been going through. uh What are the flows that we want to think, and we are uh you know we have already discussed previously that this the flows that we are going to sink are those that are active on uh on this card right uh to the standby card.

B

uh There is a slight change here when, uh if you can scroll up just a little bit, I forgot the term that you are using yeah the standalone primary right once the standalone primary is acting uh and then there is no identified uh uh appear for ha as yet and when the peer uh is now connected. uh Suddenly um it has to bulk sync everything, not just its own uh uh active flows or standby flows, etc.

B

Right because at the moment it's acting as ah it is, it is uh playing the role of uh being active for all the uh flows yeah exactly so so then um so.

B

First of all, it has to do bulk, sync of all the flows and- and secondly, uh when the uh understand that the you know the new active that has completed boxing and then it raises, um you know the whip that will be advertising to advert to attract the traffic for the new active right, uh but the standalone primary will still at that time need to continue to forward the traffic until the vip starts to advertise the new newly connected active right and also uh for the return traffic. Basically back to the vm.

B

I think there's some more explanation that needs to go in there.

C

Right, so what would happen is I think, what probably for you are this thing right? uh So there was only one card like you were saying card, let's say card two, which was uh acting as standalone primary for both of which now card one comes up.

C

We sync everything for I think that two state mission will sync everything once we do that for one of the whips the other card will take over as uh primary, so there is a preemption which, when the activated role happens it takes takes over as primary and uh for the other whip it remains a standby card. One remains a standby on the stand on the other card, the original card card, two one of the whips, continues to be primary because that was the configured primary. The.

B

C

Becomes standby for the become standard for them.

B

So this, uh when the whip starts to attract the traffic for the newly connected active, uh that message has to go to the standalone primary tool so that it can stop forwarding for all the.

C

Correct yes, yeah yeah, so there is some uh probably the preemption case, uh not added, but there are some message flows here. When we get there, we can talk about what is the state for during switch over how this goes through.

B

I'm wondering if I can break in here guys um is this a good place to possibly stop for the day and um also we need to get into the size back at some point. um What do you guys think pick it up next week.

B

Are you willing to do it again next next week keep going sure you are okay, all right um and go on saying we should discuss the new aha design using overlap. Ecmp, oh gosh! I don't even know if we're that far quite yet. um What do you guys think about that? um Gohan? Did you mean overlay cmp yeah, not overlapping, yeah yeah, I mean over the ecmp. Yes, sorry did that type be wrong: yeah! It's! Okay!

B

uh Sorry, yeah! That's! Okay! So um yeah! This topic is huge. It's fascinating the whole aha design! um Probably everyone doesn't have time after noon today. Would y'all say you want to pick it up next week or would you like a different meeting for this? We can pick it up next week.

B

That would be good next week, it's good next week, yeah. I think we should uh talk about the size back and then the uh the overlay based uh hra right next time.

B

I'll write that down right now um and I'll stop the recording um excite labs. Did you guys have anything uh pressing?

B

No, no good, okay! Well, then I'll! Stop the recording- and you know thank you- sanjay bj and b team for taking us through this. I think we got through maybe three pages: yay, hey.

C

One last question to sanjay: hey sanjay from a smart switch perspective.

B

A

This going to be between switches or if the switch has multiple dps.

C

Within um this will be for yeah, this will be per uh per whip on the dpu, because the switch itself is not kind of doing the rotation. The grips are local to the dpus themselves.

B

A

B

Of this, as the dpus can be sitting anywhere in the network right and if it happens to be inside a switch or inside an appliance chassis, it's irrelevant right, yeah. It really appears no matter where they are and how they're pulled into play.

B

I think we put that in the design in the design. Hlt could pull that up next time but um yeah. Thank you. Everyone for your time and I'll send the recording out and everything.

A

Thanks christina.

B

Have a good day, yeah thanks and thank you so much everyone for the questions. Okay. Thank you. Thanks.