DASH High Availability WG, 3 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: DASH HA High Availability Working Group 20220503

Description

May 5, 2022 HA WG Call

A

uh So, just to remind everyone, we are trying to uh finalize the requirement it is based on the requirements published in the existing dash h, a document. So our goal is to review all of them add things that were maybe implied and put them explicitly into the list, um and so that we won't have anything that is uh uh that won't be clear before we move on to the next uh next phase, which is defining all the flows and all the scenarios by which ha should operate.

A

So we stopped at, I think, no complicated hashing logic in switches to ensure packets land on the right appliance.

A

um So if you have any questions to the previous one about the downtime ability to resume connections, uh the fact that the full flow table should be replicated to the peers, we need to be able to handle, link, flapping and asymmetrical flows, and this is by the way what we wanted to what we wanted, michael for. So this still to be reviewed.

A

Cases when we need to support asymmetrical flows when so, the reason.

B

What is your definition of a synchro asymmetrical flow.

A

Yeah, so this is, let me remind you from our last discussion. We were talking that about uh the case where it is possible when we have flow, we have two directions of the flow and in one direction uh packet. It does not look the same way as in the opposite direction, and I'm not talking about uh swapping source and destination. I'm talking about in one direction it could be, uh it could be encapsulated in the other one.

A

It could be not encapsulated or encapsulated in a different kind of tunnel resulting in switches, hashing them two different appliances, so that one one appliance will get the outbound direction. Another will get the inbound direction. So uh we wanted to clarify this point at this meeting. uh What are the exact scenarios uh so that we can uh we can plan for for this to be supported as well.

B

So I don't think we have any scenarios for asymmetrical flaws like like you know so, but there's two things right, so I don't believe we have a minister knows when that packet leaves in one way and arrives in different way the only kind of standard like this in case it goes through load balancer, so the internet traffic, because the traffic going to internet goes directly and that a traffic returning back we'll have first, the kind of like outer end cap from from slb forwarding appliances to the specific node.

B

But then after this internally it will look the same from this point of view. So that's the so so not many scenarios like this, mostly for the load balancer uh scenarios.

B

Only and to be honest, the one thing that you mentioned second bullet from the bottom when ensured, bought an inbound elbow packets transit, the similar plans for the given flow. uh We should not take it as a requirement because, because tours are like normal forwarding appliances, link will change. The tools will go into maintenance mode or those kind of stuff right. So the main reason for the flow replication, which is which is kind of like a ford for the bullet, point to make sure once we create the flow.

B

This flow is kind of immediately replicated, and the main reason for this is that the the the packet later as the connection is is happening is already established. Right can come to a different appliance, and this is fully normal. So this ensure we should. We should cross it out, because this will not be happening like we don't.

A

Have we will not be able to.

B

Have this requirement.

A

But I think this one is talking about, so we should distinguish of the establishing of the connection from a connection already established and the flow is moved to a different appliance. So in case the connection is already established and replicated to another appliance. There is no problem in moving that flow to another plant.

A

I think the question, so those two can be summarized summarized into a question: do we do we envision a case where both appliances take part in establishing of the connection.

B

So I would take considering that we are designing system for like high cps right so high connection per second, and we want to have a grantee uh that uh when the when the connection is fully established, we'll never lose this flow right.

B

Somehow those this other plans needs to be in a part during, for example, the scene exchange right, because it cannot be a synchronous replication for every single staff and main reason for this is asynchronous replication. Just doesn't have these guarantees like in a way if only one appliance is participating in synagogue, exchange right and the and the other place is just receiving packets, something like fyi.

B

This was this right. If the fya packet get lost, for example, right the and the primary appliance dies, then this specific flow dies. uh So so I believe that those two appliances should work more kind of like the package. The same packets should be going through through both appliances like in a way. One appliance then forwarding to the second.

B

They should not be like asynchronous replication, because uh asynchronous consistency is eventual consistency, and this goes doesn't guarantee the strong consistency of of the flow table and traffic, and you also have to cue that you need to manage ready guys to build up right, because if you have lots of kind of like asynchronous stuff right, so either you're relying on something which is like, if you, for example, have only one appliance, that's doing basically the cincinnati exchange and as part of this traffic pad it never forwards the packet to the second appliance, only kind of like fyi, let's say udp packets, which, let's say can get, can get lost right then uh either you are relying on this.

B

That packet can get lost and this flow will never be replicated right, which would be bad right or you actually have a kind of like tcp connectivity. There and you'll be retrying those kind of packets right. So you are relying on having some q and potentially the q, built up on your forwarding appliances right. So, instead of having q built up, why not just bounce this packet across two appliances? So it arrives on first appliance. We just bounce into the second appliance bounce it back and kind of forward it out.

C

So you should read the document. I wrote because he's saying all the opposite, because if you send all all six of those packets to the other appliance you're going to do more than half of all the bandwidth which is not good either. So, if you're going to chew up half the bandwidth just for sending packets, uh as is to the other appliance, you lost half your bandwidth, maybe more because you know five million connections per second times six or thirty million connections per second, that's more than half the bandwidth. So that's.

B

True, so so yeah we should not. We should not transfer all the packets, but we should transfer differently the final. Let's.

C

Make this the final quantitative exception: yeah we maybe we can make an exception for the sin, because it's so important for the establishment. But quite frankly, we can't do that for all the packets or we don't really have a viable design.

C

B

C

A paper on this that I wrote, which I could probably share next time uh and I think it's gonna- be some kind of blend of things. But uh but we can't spend every pack. We do that today and it worked great fine, but it also chewed up more than half the backlit.

B

Yeah, not all the packets, like mostly music. I think this should be like the like remaining packet that actually establishes the flow. The last packet right. This one should definitely go there.

C

So so we can debate that in the future. So I want that to be kind of open, because I think excite probably has some ideas. I wrote some ideas down, probably it's somewhere in between, um but we just need to be mindful of we at all costs. Don't we want to be chewing up all of our bandwidth for replication.

C

So take that as a side note, I think it's worth. uh We will be probably talking about that for weeks. I'm sure to get it right. um It is like that's the hardest thing to try to balance the bandwidth of replication versus the the quality of rep uh replication, which we're all concerned with and I'll share my paper, I'm just trying to finish it up um um the next time.

A

Yes, so I think we can leave this one open and on the next meetings, when we will move to defining the flows, this will be brought up again and hopefully, when we will define how exactly the protocol should operate, we can come back to this requirement and close it, but at least I I think it's a good thing that we know the trade-offs.

C

Yes, yeah they're straight on.

B

Yeah, there's definitely traders and- and I would say this even gets slightly more complex for slb packets, because the slb packets, for example, at some point- if we add nothing, support, uh including not pulls, but I'm I'm thinking about this much much more in the future, but but we can also think about this kind of like right now to make sure the design kind of will kind of support it right, because if we add not pull support, this means the dynamic to the device can select some port uh for nothing, and this port kind of in the packet comes back.

B

Let's say through the other appliance right. This does not port needs to be exactly the same to kind of transfer it back uh those kind of stuff. That's. Why that's? Why? Basically at least this uh initials of those, I would say, initial sort of packet with, for example, let's say there is a sin right and the sin selects some specific port for that for the transformation of the of the knot right, because the ports are dynamic. Let's say right: this is like a specific port, so this port should be most likely.

B

This packet needs to be bounced through all the appliances to make sure that all the appliances persist, the selection.

B

So whenever some returning packets during this exchange land on the other appliance, the appliance knows how to revert it back to the correct port right and then and then the question should this first packet be the one who is which is basically already establishing a flow and doesn't care about the the snack packet. Or do we also want to balance snack packets to make sure we somehow communicate to the other plants that this flow is actually alive uh and and not something that was not fully established, for example right?

B

So, but that's yes, also yeah in my paper.

C

I talk about that because we're gonna use the we should be using the tcp state machine. It tells you when something is open. It tells you when it's closed and there's a million ways to close something, but it tells you that so it's the only foolproof way of knowing I started a connection, but the state machine is telling me I'm closing that connection and going to the listening stage. So there's a lot of reasons for that. But if you use the state machine then you won't have to worry about that. Yeah.

B

So we should be.

C

Acts and all this stuff, because the state machine will tell you when to close it.

B

Yeah, so the state machine- yes, the the question here is about basically because the the packets will be going to one device. For example, only let's say most of the time right and or some packets will be going through one and the other. The question is basically which how to correlate the state machine state across two devices, for example,.

C

Yep I discussed that in the paper yeah.

C

So that'll be for next now I'll. Let you the paper I just need to to have another week or so.

B

So john, the paper you are mentioning is the is the patent or different paper.

C

D

A

um Yeah, okay, so let's move on um interoperability.

A

A

Mostly, we are talking about.

A

So, let's try to understand which operability interoperability we were talking about. First of all, there is uh between. I, I think I have more elaboration on that.

A

Yes, so ha interoperability is required between vendors, dha packet format and protocol must be public, so we can pretty much work in in a matter of plug-and-play. We don't really care about which appliance is peering with which appliance.

A

So then, another one from from the dash document is uh aha protocol for syncing active flow. So I think this again will be reviewed in more details when uh gerald will come forward with his paper. uh What what modes of operation uh the uh protocol should support so, especially if we're talking about uh that, we need to, uh we need to be able to bounce the packets for the same. If that will be the case, uh probably this will be the only requirement, but then for uh for updating the flows.

A

There might be different modes, um so we we should potentially be prepared for more than one uh mode, and this should be established during the peering session, bring up.

A

And this one, I think it's quite important, also from the document aj protocol- does not need to reliably sync 100 of the flows between the cards. um But again, as we talked about this, there will be uh trade-offs. So, even though uh there is no 100 requirement, I think, uh along with the protocol, we need to uh to define all of the possible ways that the protocol may fail to synchronize a connection, not only the good cases, but also everything that might go not so well um yeah.

A

So this is all from the document any questions to this these four uh requirements. Maybe someone would want to add something be more specific.

E

Yeah marion, can I um make a hi chris hi, um hey everybody. I don't know if you'll get to this later in here, because I mentioned on a previous meeting about observability and uh if, if I'm jumping ahead, let me know, but I'm wondering if we ought to have observability as a requirement and related to that last bullet there. It does not need to reliably sync 100 flows. I think we all agree in principle, but if we don't have some objective way of measuring or having metrics or kpis, then it just becomes useless right.

E

We have to have some way of saying. Well how well does it work? How can we compare one implementation to another? So that's probably going to be a work in progress, but we'll need to establish some metrics right so that we can say well. This is good enough, or this isn't good enough.

B

Right yeah, that's true, so we should have a way to measure this. I agree with you, chris right and- and the last bit point in regards to does not need to help me seeing 100 of the flows right. I actually don't want to put it as a requirement. I would like to put it as actually that preferably I would like to be able to unless there is a banking algorithm right or we say that some specific flows, because they are, for example, being terminated or not or doing established phases kind of stuff.

B

Those we include right for me like, if there's two devices, and that, for example, one device dies and the other device, for example, comes online right and and for me, like all the flows which the first device has in established mode right should be synced to the to the backup device, because, basically, if we say that only like some flows randomly with some probability, gets synced to the other device right, then we don't have a kind of running aha right, I'm kind of okay with agreeing that, for example, if the flows is right now being established and it's not fully established, then then maybe somehow their ability is not fully there or their flow is basically dying.

B

Those kind of stuff right, but if the flow is very stable, established and customer has like ongoing connection to the storage and downloads it right. This little definitely must be sync. So so that's why I don't to put this like ford bullet point as a requirement to say not.

C

I agree with you mike. I think we could change that requirement to aha protocol as a goal to reliability, but the as a goal means obviously there'll always be acceptance. I mean there just will be, but you can.

B

C

Goal as a goal should think 100 of the connection and then we'll need to measure how well that's achieved right, and so we might find use cases down the road through observability. So I'm agreeing with keysight through observability. We might find cases that we never thought of that doesn't allow the 100 think, but it shouldn't be our goal. Our goal should be 100, we put in observability we test, and um you know we will improve it over time to meet that goal.

C

As we find, you know, bizarre cases that nobody's thought about yeah. I think that's a better way of putting it yeah. Yes, but you have to couple it with. We need to have observability so that we can measure um our success in that yeah for um 100.

C

uh Sync, right that that's, we need to add that that's what's going to allow us to go and find those use cases sorry exception.

E

Cases and the observability will also have to be to have to be some somewhat standardized too, so that whatever instrumentation we enable even the probably two types of observability high fidelity, maybe for testing and probably lower fidelity, lower resource intensive for live right. Maybe it's sampled, maybe it's! um Maybe data are coalesced and emitted at a slower rate, something, but you know we're going to brainstorm that it's gonna be a fun discussion.

C

Yeah, I I agree by having the requirements and not making it after it's important. Thank you.

A

Okay, so I'd like to return also to actually let's keep them, as is uh so the last point, as michael already mentioned uh well last point, as michael already mentioned, under the wrap cases in the future, when we will add net, maybe some other stuff which will affect uh how the protocol works.

A

um So I think we we need to take that into account as well, um so that there are uh two ways to go about it. uh We could either.

A

Make the protocol uh extendable in a way that we will support different formats of the messages or we can try to incorporate everything into a single message type uh either way we go uh everything I think should be documented. So we will not return back to to that question. Hopefully, in the future, probably there is still something that will be left out, uh but uh all of the cases that um michael mentioned and that we can think of uh that reasonably fit into uh use cases for dash.

A

uh I think they have to be accounted for so and also defined in the uh protocol description, how it will be used for them yeah. So.

C

I think this one could be described as examples either uh modes of operation or packet, um like you said. uh The other way to do it is to uh in the packets themselves adding extensibility to the actual messaging one. One of the other needs to get done, or actually both both can occur, some of them okay, this mode of operation for this mode of operation. You need to have this extensibility the packet format, and that that's true.

C

But design in a way that this is possible because it's gonna have to be possible, like I actually think mode, one will always be send all the connection packets to the other, the other guy make that work, that's mode one, oh two is make that way more efficient. Oh three is oh account for um slow dancing, for example, but I think the first mode is the simplest mode. Easy to test um would be like the one.

C

That's you know mandatory, because without it you don't have anything and the second mode should probably be mandatory, but we'll debate that as we go along in deployment.

A

A

Okay, so this is all we have for the requirements. I think there will be still future discussion going, especially um with what gerald has to present, but besides the requirements, I think the next important.

C

I don't know, did you did you ever uh stay in this requirement? uh uh Like I didn't see it explicitly? I know we've written about it, but that that you need to not only be able to fail and and the other device take over, but you actually need to be able to resync a device that comes back online or a new device either way. You have to do a perfect thing, um but I don't. I don't know if I saw that in the requirement.

D

Yeah, that's true.

C

So it's a rethink, a a new, pure or rethink the fail appliance once it comes back online right still because it didn't software upgrade or whatever, but you have to rethink it so that what we call in the paper perfection uh you can implement it any way you want. We gave an example, but that has to exist otherwise, you'd never be able to come back.

C

You just fail over and you're stuck in the middle of this place, whereas what you really want to do is resync after the device comes back and the flow should migrate back.

C

um So that's that's part of the requirement.

B

Yeah and one additional requirement that I I think we should also add, in addition to basically racing back right, uh to have ability from the control plane to control uh when we should start advertising back bgp and this.

B

The basic scenario here is that, even though, for example, let's say there are two pair devices right, if the device comes back online and receives the flows right, sometimes we do not want as soon as the device have all the flow ratings to instantly advertising pgp and the example scenario here is, for example, let's say there is some big company that we have on the cloud that have some very critical business. Let's say: there's black friday right, one device fade over right and and they they, they notice.

B

Some like connectivity, droppers kind of stuff right, and they will tell us that hey if you need to service the device back explicitly those kind of stuff right do this outside of the of outside of our hours uh business hours right. So in this case the new device will be brought online. Will single the flows right, but uh we may decide to, for example, control that hey uh start advertising the bgp uh after we undo additional, for example, health checks right, which means that uh the help model on our side is more complicated.

B

It's not only that two device are in full sync right, so so each device basically should, in my opinion, maybe as requirements to report, something like a sync status right. Whether for example, device is not paired where the or device is, for example, right now, starting pairing devices, singing flows or the devices in full like stable, ready, state uh and and the connections kind of online between the devices. So only once we receive the device is full.

B

The device were fully appeared back and then device fully synchronized all the flows and are ready to kind of like fully take over. Only then, for example, we will check some additional health checks and then issue the command from the control plane. Saying: okay, you can start advertising bgp and handling the traffic right. So we need to have a manual control of this uh on our side through control, plane.

C

And that even the manual control does have an exception that, uh because that's a valve case like walmart somebody says no, no, don't touch it. Wait till the hours are over, but during that time the the the appliance that's working becomes not working. Obviously they would want you to fail over so the so so the the good case would be.

C

You don't fail over until somebody tells you to because you're gonna wait for the window, but also if it does fail completely, then obviously you want the other one to take over, even though they didn't explicitly say that because then they'd be out right, we don't want that. But so you, you almost have like an explicit motive. Saying don't fail over unless there is a true failure and then you're waiting for some operator to come down and explicitly say to fail, but so it can be as complicated as that right.

B

Yeah, so for this we need to have some ability to configure it right. So so, basically.

A

B

Automatic failover in case second device is allowed or not right. If we have three devices right, then, usually we don't do this, because two devices are basically doing this and the third device we are leaving in two devices. This becomes more complex, so this may be needed.

B

uh What is kind of automatic stuff right, so definitely ability to kind of like have visibility to this racing status right and able to configure it. uh I want to handle this automatically over with those different modes and we should probably document those modes, as you just mentioned, yeah.

C

Yeah little state machine that we need to be in kind of, so that you can view what what what is the true state and I think, um waiting for well, you know waiting for uh thinking wednesday, and then you know the other state is I couldn't wait? You know I can feel a little bit, um but you should be able to observe that if that would ever happen, so there needs to be a little state machine right.

B

Yeah and additionally mention is that we want to have a have ability to sync the flaws, but we don't want you guys to sink the config, so basically config will be providing on our side. So basically the device will start up, let's say completely new right and we will provision the config. So you guys just need to basically see the flows right.

B

It will provision the config, and this is one of the reasons why we need to manual control, because even if the flows are fully sync, but let's say device got fully rebooted and, let's say configure loss right. We still need to have acknowledgement from control plane that we fully configure the device correctly with all the correct policies which are up today before allowing to bgp to be fully advertised.

B

Yeah, so config will be synced by config is not the scope, and it will be basically sync by control plane right so explicit, like no goal yeah out of scopes for the for dha protocol.

A

So uh I want to, I want to understand a little bit better about, let's say automatic, failover and manual failover. I don't know about the others, but I didn't really get the difference between uh those two and I don't before moving forward. I probably would need to understand what is the uh trigger for uh automatic.

B

Versus manual, so so so, basically, there is like few modes of operation. So if so, in normal case, basically two devices are online and they are seeing the flows right and they are both advertising pgp right either, preferably in this active active model and both devices advertise the bgp in the.

A

B

Way without asn equipment right, so they both advertise, btp right and then, for example, let's say one device dies which is kind of like basically dice right, then the other device will pick it up and and kind of uh can resume the extreme flows right and because the device dies and it's kind of emergency situation.

B

Then, for example, if last, let's say a few flows- hopefully not, but let's say they terminated, they kind of, like accidentally sure may disappear, because maybe it's not like perfect on this kind of stuff right, so this is kind of like automatic fade over and then with the with this automatic failover, for example.

B

If this fade over is because, for example, there is a connectivity flap between, for example, one and the second device right, but both devices, because we should also discuss network partitioning right in both devices, for example, they are advertising pgp and they are actually fully healthy right.

B

The only thing that happened is they were temporarily network partition, which means they would put advertising bgp, but they just couldn't communicate the sync state between each other right then, when, when this kind of connectivity comes back after network partition, then those devices because they were they already advertising bgp right, they should be able to to kind of establish connectivity back online because just just the basic thinking needs to be re-established after this was lost right and then there is a manual failover case and the manual failover case is in in in case, for example.

B

Let's say we actually need to service one device, let's say update firmware, and this cannot be up there by ssu and and kind of in flight right so kind of we need to close one device. Then you will issue that okay put this device offline, so, basically explicitly you need to have ability to issue the command to the device.

B

Let's stop advertising the wgp vip right, make sure all the kind of connections move out of the device right in kind of gradual natural way right and we should have some magic to figure out also are those. Are the new connections still coming to the device where the full with the device like fully shut down? Regards to all the flows are fully synced, this kind of stuff, and then we will turn off the device like in a way. Okay. Now we can service it right, and it's kind of the my manual fade over case.

B

Now we can service it right or exchange the different device, and then we bring the new device or reboot this reproving the state say reason pairing right it will. It will start basically doing the pairing again.

B

It will start seeing the flows, we'll basically monitor this kind of status, the racing status and when the status fully they come back online after, for example, servicing and all the flows are kind of sync right, we'll confirm on our control plane additional help metrics, including the staff, have we programmed the latest, the latest configuration and this kind of stuff right and then you will say: okay, everything is fine. All the kind of checkboxes are green, now uh advertise bgp again on this device and make this device fully online.

B

So this is the differences between manual feed over versus there's, explicit, failover right or sometimes you may notice that let's say some devices behind some torque, which is misbehaving right and we actually need to potentially remove this device and this kind of this manual failover. So only one device will be active, for example, and then bring up different pairing device behind some different, or this would be the manual.

A

Failover, yes, so what I noticed in here that nothing no input no operation is done on the device that will take over the flow. Everything is done to the device that either it fails itself or we are putting it down.

A

So we there is, there is no input to the other device that will become active for that flow. Is that true? That's.

B

uh True from the point of view of the control plane right, so the control plane will not tell anything more to the other device. The control plane, basically will say: okay shut out this device and and like primary device, let's say, and the primary device then using the sync protocol we'll need to communicate it like hey, I'm shutting down, for example, but but the question is like doesn't need to communicate right because because therapy once it stops advertising the vp right, then basically the scene anyway happened right.

B

So it just needs to make sure that this device completes the sync. So maybe even the other device doesn't need to know from from the other device. It is just basically we're just thinking the flow in one direction only and then at some point. Basically we'll stop syncing the flow, because there's no new connections arriving right and then we just disconnect so the other device may think about the devices goes disconnected. Definitely it makes sense to to submit to this like h8 protocol.

B

Some information that the device was gradu gradually shut down is instead of basically actual automatic failover because of like a network or or those kind of stuff, so the other devices has some internal status right. uh By the same way, I don't envision the other device. We need to do a lot.

B

From this point of view.

A

Right so I think we agree on the expectation that a control plane on the one that is becoming active for the flow doesn't really do anything.

B

Yeah, the control plane will interact only with one device that you want to shut down right, so basically issue the shadow command to the device and expect this device to to gradually and correctly shut down by offloading all the flows to other devices, and there will be no control plane operation on the other device.

A

And okay and the last question: how how do we signal uh I'm trying to understand if this is in the scope of the protocol, or maybe it is done by other means?

A

How do we signal to the network that we are fine to take over those flows back again when we are coming back up from, let's say: maintenance, it's purely done by starting to advertise bgp.

B

No, no so so so uh so this is kind of should be similar, like uh maybe not the same protocol but like because there is no paxos or or those kind of stuff right. But uh but basically the device should basically like signal that hey the device is losing the uh living there, for example the peering establishment.

B

So basically the device will say: hey, I'm basically syncing with the flow, because, basically in a second I want to, for example, shut down myself right and then, when the device come back online right, it will basically re-establish uh the pairing connectivity with the other device right. So normally open the port, open the connection and then say: okay, say: okay, I'm just new device. That's back online pressing me back the flows right, yes and then, and then so this should be inside the uh aha protocol right.

B

So so the device basically on the device said is the pairing, and this should start doing the pairing. But after we did the resync, how do we signal.

A

To the network, okay,.

C

A

C

Yeah, I think this is bgp more signaling, where you can you both have uh as3 pre-pin from the beginning, but when the other device takes over, it becomes a shorter it advertises without uh aspen or with less aspen becomes the preferred. But when you go back, you advertise again the same ib prefix with the pre-panda game. That way you can. You can shift all topics back and forth through bgp, which is what you need to do because you you've said I I want you to now take over. You are extinct. You did.

B

C

Thing you did everything else now you need to play with bgp like dynamically to get the the the uh equal um or sorry to get the those clothes to migrate back. I think.

F

Yeah, I think gerald we do have an option in bgp um from from vm side.

C

But it wouldn't be, it wouldn't be between the the devices. The aha protocol would be more like the state machine said, I'm going back to the state, and then that affects your dpp advertisements and then you end up migrating the full stock. That way after everything is seen.

A

So, is it fair to say that after resync uh moving actual traffic to the new appliance is out of scope of this protocol, it could be by means of bgp.

C

Or something you think we need to say how, because it's probably going to key off of the aha uh state mission. There's got to be a state machine which is controlling this stuff and when that state hits then bgp needs to be fired off to advertise. You know differently. I think it can't be out of scope.

C

uh The messaging is out of scope because we already have messaging for that ddp messaging, but the state machine must indicate when this is supposed to occur, so that other software can come in and do the right thing like re-advertise, using uh gtp prepend in a different way.

D

C

D

C

So yeah, so we don't define the message, but we have to define a state machine that tells some people software to go and do that. Otherwise a human would have to do it, but it needs to key out the state machine.

B

Yeah, so the state machine and control pane will do we'll just see the 6000 and we'll say: okay start with the ptp right, and I'm thinking that definitely once we want the device to advertise in pcp. The device should also communicate to the other pair that hey I'm advertising bgp, because maybe this somehow changes the internal sync protocols kind of stuff. So so I'm not quite sure. If, basically, the sync protocol state machine will somehow rely on information which devices advertising bgp right, but it may so.

B

I think, like as part of the protocol, the devices should communicate to each other whether the device is advertising pgp or not, however, whether to advertise bgp or turn on turn it off. There should be control also from the control plane perspective.

D

Go ahead, go ahead.

F

Yeah, so from the vm side. Right, if I want to you, know after uh everything is done from the other pair like if you want to open up for any new flows from the vm right, do we have any option um like? But when do I uh you know, get the flow after the complete sync right? Do you have any.

C

Options, the vm, the vm, doesn't know uh the the vm's traffic just blows.

C

No, no because they're in sync it doesn't know it doesn't even know those two devices that the the flows from the vm will just magically arrive on the other appliance. It's already synced up. It will process that flow, just like the the standby or the primary, either way. So the vm doesn't know this. It has no idea, it's still sending uh sending the same channel information. It always has. It doesn't actually know where it lands. It doesn't need to know where it lands, because everything's in sync.

B

Yeah vm should not know anything that that happens like because, like right now, for example, the tour dice right vm doesn't know because, because stores are just like somewhere in the middle right and and the same with this, this is just forwarding. Appliances like in the middle and vm should know. Vm basically says traffic to the to the vp. It doesn't care which appliance takes care of the vpn from the point of view of availability, the vp should be just available and every single flow should be working.

B

Every connection should be working, but the vm should not care which actual up like appliance handles. This.

F

But how do we make sure that the vm traffic lands on the planes after they? You know fully sync fully synced with.

B

I mean bgp handles this right because, like they've been advertised, the.

F

Gp on the network side, isn't it.

C

Yeah bgp will be on the primary. Like now you're going back to the you went to the secondary now you're going back to the primary. It will be advertising bgp with a shorter pretense right now, and so everything will just magically flow back to us.

C

F

Bgp when we say bgp, it's it's for the underlay, correct or overlapping.

C

Exactly yeah, so you they're just sending it to what they think is a card, for example, and that card now is on the other device. It doesn't know that it just knows it's sent it to a bit and that dip is used to be on this appliance. It doesn't know that, but it arrived there and then now it's arriving somewhere else because of that appliance went away, it's still the same vip, so all its packets, through networking, the underlay will migrate.

C

uh Its path will just change to the other device, and then, when that device comes back online and it re-advertises the cgp with a shorter path, it will magically all the packets will arrive on that again because uh it's got a shorter path as far as pvp is concerned right, so the fifth doesn't change, but the path length changes.

C

The primary is always the shortest length, but it's if it's not there, then. Obviously, the secondary is the shortest one, because it's the only thing being appetized, but once the primary starts re-advertising the shorter length, then everything will migrate back to the primary or at least the connections that were going for that.

C

So so you don't have to do anything special there. What you need to do is just to make sure like like what michael was saying. When do you start advertising bhp is a real question, but once you do it's going to have a shorter path and therefore everything will migrate to it.

C

So the state machine will probably tell you when to start your bgp advertisements, uh but when they start it will always be the shorter path and everything will flow back.

F

C

F

Timeouts and stuff like that.

C

Yeah yeah yeah: that's why you need a.

C

Machines need to tell you what state you're in and then there's uh I don't know about the timeout that uh which timeouts are you specifically talking about.

F

F

Time out for what yamidi went to basically uh said the first, you will set irmed so that that path won't be done. Then, if you remove that, then the obtains, but.

C

Remember the the whatever was the secondary will continue to process any packet that arrives on it right, and it will continue to sync that dynamically, with the other side pull it until it actually has handed over. um It. Doesn't care they're both they're. Both packets could arrive on either one, and they would both call it until uh it's in the state, where only one should.

C

It sounds like lots of good discussions on everything.

A

Yeah, which brings me to the next thing that I think we need to focus on, because up till now we were only discussing this. So I think the next important step would be to put everything on the paper and have a detailed description, maybe with the diagrams for every flow that uh that we can think of and how we from uh from control plane point of view from data plane. Point of view how we see them working before we actually go to defining the protocol.

C

I I think, defining the state machine is very important, like.

A

C

Yeah without doing that, you're going to be lost because everything will be an exception later on. So I think that we should concentrate on you know what are the requirements for the state machine. Michael's already said that we want to have a state machine that doesn't allow the automatic sailback uh until told, but also you know, if it still fails, then obviously we want it to fail over as a last resort and all that needs to be decided by the state mission.

C

So I think uh the requirements should be against like the state that you should be able to support, and if we get that right, then we can start moving on to to uh do other things, because, like we already discussed, we don't need to define bvp. We just need to define a state that that tells the software to initiate the vpp.

C

A

C

Agree, um so if we do it from that way, it will look something very similar to when you see the tcp state machine. There's all these cases right up. Are you open and close things? I think it's the same thing for aha there'll, be all kinds of like intermediate cases that we need to define.

A

Yeah, so I think.

C

A

We want to define the state machine and then.

A

A

Show how the states will change for any kind of events in the network.

C

A

That should be yeah.

C

Events and or user uh provisioning like like what he said where you actually go down.

A

Yeah, okay, um I think we have something in the chat.

A

Chris, you, you had some message.

E

Yes, um last time, two weeks ago I showed some diagrams that I proposed to um put into the document, so it showed the smart switch as well and got some feedback.

E

um You could click on that link if you want to share or I can and it shows the diagram, I have a pull request to put these into the main dock.

E

If you want me to uh share so pull request.

D

E

Two weeks ago, which is in the chat pr 92 and it's it's adding it's replacing the old document, which was just a ping yeah. If you go to the readme, you can just view the readme and you'll see the images.

E

Just I mean that that top document just just do a view, file and you'll, see and then go down to the pretty pictures down below.

E

So I showed these last week, but I didn't have the third appliance c and down below I added uh a plant. The third switch- and I believe, michael might have said you know um this torsi would connect to the t2 tiers. So if I can just get a little clarification, if the group thinks these pictures are helpful, we can merge them into the main document.

E

um Just so I don't have these open pr's.

B

So not correctly in this case you know like um like for like, because the h, I think that you mentioned in regards to messages right, it's most likely to go through t1. It will go through t2s.

B

So so, basically the pairing appliances would be farther away, because sometimes the appliances will be in, like the the the other. Basically paired appliance may not be able to basically fit physically in the center under the same t1. Okay. So so it's also basically in principle, because that the original diagram was showing something different. The original diagram was showing two appliances which which were basically cross cross wires to the same t0, and then they were connecting it to once.

B

That's correct right, but the but but but the sync process was not between those kind of appliances. The same process will be most likely behind their plans and maybe we can put under the same t1, but most likely there will be they'll, be under different t1, so they will be only connected with t2, so so just small changes. I think.

E

It would be very helpful if you could just make comments on that pr itself and then I can adjust the diagram and then we can look at it one more time, but it'd be nice just to move this along, so we don't have pr's floating around, so anything I want to make. I would really appreciate I'll tweak the diagram and then we can look at it one more time.

E

C

One device and the other, because it can, it can be anywhere it could be under a tier three. It could be under a chair too. That's.

B

C

So I think what you need to do is to draw a little cloud there, because we don't know what that there's no direct line there could be, but there's no guarantee. So you put a little cloud there and that indicates that that control channel is going over a network yeah it doesn't. It doesn't really matter how big the network could be. Big could be small could be direct connect, it's a network. I think that would solve that uh mic consume yeah.

B

Yeah, that's that's correct, so it goes through the network. We'll make sure that the network has sufficient latency to kind of handle this very, very fast and kind of make sure that uh we could hit the cpu. It will not impact basically cps numbers or latency or banner in some specific way right, but.

C

B

Yeah, it just goes over the top.

C

And I think the switch is the buffer okay, because you know it's just a cloth network, so yeah for the connectivity above yeah. It's fine, but it's not a tier one, two three, it's any any clock. Okay, sure.

B

C

About right so that you don't have to change you, just don't call it a tier one, two or three, it's just a spine and.

D

C

Cloud and I think you'd be you'd, be good, because even that third device, you know you talk to it through a cloud too, like it's just a network, there's no direct connectivity to that third device. It's just connected to the network.

E

C

Thanks for the clarification, would that be okay, uh michael.

D

Yep definitely yeah.

B

Yeah I know I just I just had to basically find a button to help me.

C

B

C

Okay, so that we don't need the pr you can just make that and then add it to the document.

E

Well, that's what the pr does it's just the same. I want to make the following change, so I'm adding it to the document via the pr okay, so you'll do that now everybody, oh I'll, it'd, be nice to have a few comments in there. Just so I don't forget details and then I'll update the document, the drawing and it will still be in the pr and then we'll say, okay approved.

E

Do you have a pr.

C

In the window, so you know the specific one to go through pardon me: do you have in the comments the specific pr you want to come up with.

E

Yes, yeah pr92, it's in my comment right there: okay and we move the pr forward pr pull request 92.. So anyone wants to comment and be really helpful and then and then that'll be in the record right and then I'll tweak it all.

F

Thank you, geraldo, quick question. uh I think when we you mentioned about three node. Initially it was two node h and then you increase it to three node um is, is the three node is what we are attempting for the first one or it's it's for the future.

C

No three note is like a perfect example that the the the primary fails, but it's permanently filled, it's dead, it's broken and we need to replace it. Well, you need a third node because we're going to repair to a working device right, so the third node is important to have there because for that to occur, you have to do a perfect thing. For example, you got to bring it up from nothing and and so there's different requirements when you do that, so the the third node is not active. It's it's! You go to that.

C

When the primary secondary fail physically, like they're damaged, then we need to bring in this new device.

C

Okay, okay, it's not that was load load spreading across three devices. It's we need to describe how you get from a broken node, which is going to be taken out to another working device somewhere, um and what? What does that? Look like that's right.

D

G

uh General the question here uh is this: the case where the primary is completely non-functional, and so uh it's really the secondary, which is now talking to the spare or the primary.

C

Is still functional.

G

And it is expected to synchronize, with this pair.

C

Yeah well either either one primary secondary can fail permanently right. So we can't say primary. We can say that either the primary or the secondary have permanently failed and we need to sync to a new device from scratch. That was not part of the pairing.

C

That's the requirement.

B

C

B

This to the device are like active, active kind of mode right, so there is normally a console primary secondary, underneath I think yeah.

C

Yeah because they're both active, active.

F

C

We need to define it because this is actually a very routine thing. Like things fail, so you got to have a mechanism to bring the other one into an active, active role from scratch without without tearing down connections right- and that was the whole point of perfection to do that.

G

How is this different from either of the nodes getting completely power recycled, so you have lost all connectivity to it and it's the same hardware that comes back.

C

Well, it could be slightly different. We'd have to look at that, but the slightly different could be what its role is like, whether it is primary or secondary or whatever I'm not sure it could be. That going back is the same as but um the the difference is. Is this new device needs to be totally provisioned from scratch like the old device could go like all of those cpus, for example, could go down, but the provisioning is still there. Then the whole appliance doesn't need to be reprovisioned.

C

It may be just a software update to the to the actual gpus themselves, whereas a new appliance it would be, it has nothing. It has no knowledge nothing inside of it. That knows anything about the service. So that could be just like one example. I think. For the most part they would be the same, but we just need to draw it so that we don't forget that that's the scenario and that account for it, and we don't you know just assume. Oh there's, no more cases there.

C

There probably is conversations like when you add a new device. The first thing you have to do is to go into business before you do anything.

F

Yeah, I think we can just clarify that. You know the third node is, you know, could be the same node. You know after software upgrade in other cases, but.

C

Yeah, I think so it may turn out to be very similar, but for now, since we don't want to lose that case, let's draw, we can prove that it's not even necessary to show it fine, but I'd rather start with it. So we don't we go and prove that before we just take it out.

C

Probably that's going to come out of the state machine like, as we start looking at the same machine more clearly.

A

Yeah so since we are five minutes over time, I think we can agree that, starting next week we will start working on the state machine.

A

Is everyone fine with that.

H

I was on mute, um I'm fine with it. Everyone else on the com.

C

Yes, yeah yeah.

C

If somebody wants to come in with a proposed state machine that would be, you know even better.

C

I'm not sure that, having nothing and coming into meetings that we're going to work on the school machine, how far away we get.

A

Yeah, I will definitely uh have uh some uh homework done by then, uh so we can start with something yeah. I agree that having nothing it's, it's not the way to uh to discuss it. Okay,.

C

C

H

Have another meeting as well? um Thank you all for coming and thank you marianne for leading this. um Everyone participating. You all have a good day.

A

Yeah thanks everyone, bye-bye.

C

Thank you. Everyone.

B