DASH High Availability WG, 17 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: DASH HA Working Group 20220517 (May 17, 2022)

Description

Reliable or stateless
Synchronization and state updates discussion

A

A

I've started the recording.

B

All right, john.

C

Yeah I mean I, I did submit a pull request, um basically trying to explain um the bandwidth requirements. um uh If you know you need to do potentially like six state updates um per connection um and try to explain the need, what the need for it to be reliable for those state updates to be reliable.

C

I think it's kind of like uh um maybe putting the cart before the horse a little bit because in some sense like we really should be showing like what are the states and what are the updates that need to be made, and then I think it would be clearer to understand like why, like those state, ups updates need to be reliable, but I believe that uh that if you make the state updates reliable it it's, uh it solves some, some problems.

C

um You know, particularly um you know, with making sure that connections are closed uh and also making sure that, like active connections, aren't like um aged out of the uh the other dpu inadvertently.

C

um So anyway, um if, if.

C

If you accept the premise that the that the that the state synchronization messages should use a reliable transport, then the question becomes well. What should that reliable transport be? um And I think, like one obvious uh answer- is well: why don't we just use tcp um and we could- um and I think that would be an okay answer. um But one of the things I'm trying to point out is that uh the bandwidth is so high that I don't think that there should be any expectation that a single tcp connection for state synchronization.

C

Will be used for the state updates?

C

Also, if you think about the fact that there might be multiple physical ports on a dpu, you wouldn't want the state updates to all go through the same physical port.

C

You would want the state updates to be balanced across all of the physical ports, so that um you know not you're, not like stealing the bandwidth just from one port to do the state update. So for. For that reason as well, it seems that the transport of state, synchronization updates, um needs to be.

C

um You know multiple parallel streams uh between the two gpus, um and so that's really kind of the point of what's in this document, uh and then sort of the second part of the document is, uh if you think, tcp is kind of too heavy weight for this. um The sort of operating conditions under which dash is operating where the two dpus are kind of localized they're near each other.

C

There's like the latency between them is pretty low. uh The you know, number of switch hops, maybe is low uh and that the network is already like pretty pretty reliable, but not a hundred percent reliable then like under those conditions. Like there may be like a simpler, reliable transport protocol that can be implemented on top of udp and so there's a description of something that is simpler than tcp, um but I think that that's kind of a secondary thing.

C

I think the primary thing is just the uh acceptance that uh that the transport for state synchronization needs to be reliable and that it needs to use multiple parallel streams between the dpus. I think like. If there was agreement on that, then I think it's like a secondary. uh It's sort of a secondary matter as to like. Well, what is the appropriate transport.

B

um Yes, so I think those two are.

B

We can separate them. First of all, dividing uh dividing connection into streams can be solved separately, yeah, but then the question: if we need the reliable connection uh or reliable transport, I don't really don't really agree with that, for the reason that we are trying to solve for real time data, and we cannot really guarantee, like tcp guarantees for the static data to not lose any connections.

B

So it's either. We are, we are losing the current connections or we are losing the connections that will be open in the future. While we catch up with the window.

C

Right, so let me try to address that. um I believe that the entire system has to be engineered to provision for enough bandwidth uh in priority potentially uh and dedicated buffering or whatever is necessary to not have the path between the dpus for state synchronization be the limit of your connection rate performance.

C

However, in some event, where, um where you're not able to synchronize state uh because you're getting drops or uh or you know, you're not getting enough throughput uh through those streams between the two dpus, I think that you have to mitigate that by uh you know: dropping uh packets, uh that are, you, know, data packets uh that are arriving um on the dpu, um and maybe you don't have to drop packets for like established connections.

C

uh But I think you would at a minimum drop packets for uh like syn packets, for like new connections that are like trying to start up. um That's kind of, in my mind, like the only mitigation for for um not being able to synchronize the state, because I think if you do the opposite, if you say okay, I never want to drop like packet data packets arriving at the dpu, um regardless of whether I can synchronize state or not.

C

I think that you end up with like broken state, and that broken state means that um uh that you've lost your ha ability.

B

Right but for how long will it be broken.

B

So simple example: we have like the usual topology that we are looking at is two tours for high availability on the tour level, and then each gpu is connected with one port to one of those stores, meaning that if, if we want to synchronize two dpus one of them, uh the the state synchronization messages will go from one dp to the tor to another dpu right.

B

What? If a link goes down between the tor and the backup dpu.

B

C

B

How well reliable protocol help with that.

C

No in that scenario, I would call that a failure scenario, and in that scenario one of the dpus should be elected. As the uh you know, as the uh like single available dpu and all traffic data plane, traffic should be directed to that dpu. If the link, if the synchronization link goes down, that's as much of a failure as a as a dpu, failing or uh or.

B

I disagree with that, because it's a common thing to have link flapping link can be down, for I don't know one second, do you want to declare a whole dpu down, or do you want to cache like, according to this document, it's like what? What was that 12 gigabit per second bandwidth.

C

B

C

I I was estimating that five million connections per second would require.

B

C

Gigabits per second of state, okay,.

B

Yeah, which means that every second we transport like how much uh one and a half gig.

D

B

So if the link goes down for one second, do you want to cash one and a half gig of data on the primary dpu so that you can re-transmit that.

C

Well, there's one of two things you can do marian you can you can not cash, you know one and a half gig to re-transmit it. Of course, the window here is very small. You only need a very small window for re-transmit.

C

What you would do in this scenario is, you would either fail over or if you didn't want to fail over you would you would drop all new syn packets, so there would be a period potentially of like one second, where you're dropping new syn packets.

B

I don't really understand why we would be so radical to drop the sand packets if we cannot synchronize yeah. We know that we have our state desynchronized, so we cannot do failover for some time until we will catch up with all the uh like periodic updates, because even if, uh even if something happens, we still have the periodic updates, which will eventually synchronize the state. So why would we want to sacrifice new connections.

C

Updates, I think you have to you- have to think that all the way through I mean, if you have you know millions of flows, you're going to do periodic updates for millions of flows. At what rate I mean, that is more, I mean any single flow. um How often would its periodic update be, in that case.

B

One second, two seconds: we can pick a number.

C

B

Because if, if you have a long lift connections, how else would you know not to time it out on the backup.

C

B

C

On the long-lived connections, marian um the the periodic rate for updating long live connections, just has to be at like the time like has to be like around the same as the timeout rate of the connection.

C

You just have to be like a little bit faster than the timeout rate of the connection to keep it alive on both uh dpus right and, like you know, we're talking about like four minutes or something like that as like the timeout for like a tcp connection, but if you're saying like no like, I want to just be able to like re-synchronize the state of every single connection, every one.

C

Second, I don't that doesn't scale I mean like this is like 64 million flows like per 200 gig of throughput, um like that, would use a lot more than the 12 gigabits per second. To do that.

B

Where 64 million coming from, I don't really get that.

C

uh I those are numbers that I think gerald has uh yeah.

B

But okay, so then, uh okay, let's stay with that premise. We don't want to do periodic updates fine, but then again we have a limited window and the port goes down for one. Second, the usual thing link flaps.

C

Why is there a link flap? This is like this is like a very engineered path between the two dp's.

B

No sorry, I I won't accept such a premise that link does not flap in the data center at at any point, link can flap. This is a common thing. uh We are not doing. uh We don't have a soldered connection between those two guys. The link can flap between any two points.

C

So, okay, sometimes.

A

In the the optical box that they have to go through, sometimes the technicians are performing maintenance and they bump wires. Or you know, um I have seen a lot of link flaps in our data centers.

C

Okay, but what we're talking about here is: let's talk about a smart switch. Okay, and you know what you could talk about like that: the backup is remote or whatever, but like. Let's just talk about like normal operation, two smart switches with like a physical point-to-point link between the two smart switches and that physical point-to-point link between those two smart switches is there for the purpose of this h, a path okay. So the link flap you're talking about is like between two, like co-located switches, with a dedicated point-to-point link.

C

Okay, let's say you know, for I don't know there is a link flap on there and it's a very short period of time.

C

I think the answer to that is that um the dpus drop syn packets during that period of link flip flap. I think that um I think that that's the answer. I think. If the link is deemed to be broken, then I think you have to declare one gpu the master and switch all traffic to it.

D

So this is about um a related um scenario, is if the link is flat. Link flaps link goes down. If there are asymmetrical flows um and say the like, a syn packet arrives and that's supposed to put an entry in the flow table and open up like an entry in the reverse direction.

D

If there's no connection between the two gpus, then that reverse connection won't be opened.

D

So it's not like you can say.

B

D

B

Really the point there is so even in the, if you look at the uh topology, you have uh two links from the dp, even if one flaps uh eventually bgp will redirect you to the other one. It will.

C

Okay, those are the data plane. That's like the link to the data plane where this is like the the link between the two dpu's, like the h, a link.

B

So again, you cannot assume you have a dedicated h, a link h. A goes on uh most likely goes on the same boards that the date.

C

I I think that this is an unsolvable problem, if you're not willing to accept that. There's like an engineered connection between the two dpus, I I mean. Maybe you can accept, like a very small, like link outage time uh without like failing over to one or the other, but I think it's like unrealistic to assume that there's going to be like unbounded periods of time where that link is like unusable and that somehow you're going to still keep these two dpus running like independently, I mean it's like you will lose like the synchronization state.

B

That's my point: if you want to keep it ideally replicated, then you will end up in a situation that you can very easily lose. It.

B

And it's not something.

C

I'm afraid that the solution of just like periodically updating the state is like it solves it. The period like periodically updating the state at like the the flow timeout rate is okay, that's like minutes, but periodically updating the state every one. Second, because a link might have flapped, I think is like if you do the math on that, like the bandwidth you that you're going to need is more than the bandwidth of the dpu you you can't like you can't copy the entire like flow table state in one second,.

B

What do you mean it's like?

B

How can it be more than the bandwidth of dpu if, in the worst case scenario, let's take the worst case scenario, we would just mirror uh every flagged tcp packet, it's not more than uh it's, not more than gpu bandwidth and uh adding to the and adding to that periodic update.

C

Just think about the periodic update, I have 64 64 million flows or pick a number right.

B

Yeah, 64 million flows. How much can we pack into one message.

C

Well, let's just how often are you going to like send a periodic update for a given connection.

B

So what whatever number we will pick one second 10 seconds.

C

All right pick pick one second, okay, that's 64 million messages per second.

B

Not message, yes, it's flow updates. How many flow updates can be packed within one message.

C

Well, I'm calling messages messages and then you can pack multiple of them into a packet.

B

Okay, okay, I understand what you mean, so we.

C

B

Multiple into uh multiple messages into the back, so uh according to what you have in the paper, how many messages can you pack into the.

C

Let's say 60 for ipv4 right six is this.

A

Did I just yes.

B

So so it's roughly one million pps.

B

C

Okay but like let's say that, like you're, let are you doing in in addition to periodic updates. Are you also doing like event based updates, or is it all periodic.

B

Now you will need event base to be uh to cover uh uh to not have uh to be synchronized to lesser amount of time. We don't want. Do.

C

B

To be a backup to be one second, behind.

C

Right so so, like uh let's say, you're doing event based updates, that's like maybe another, um like you know, million packets per second, yes, okay, so you've like you, you've doubled you've like doubled, like the amount of uh messages um in this approach in in in and marion like one of the issues here is not just like. What's the bandwidth of the messages, but you actually have to process those messages you know uh like if you're getting a periodic update.

C

You don't know that that periodic update is like needed or not needed right. So the only thing the receiver of that periodic update can do is to treat it as if it's a real update- and you know you have to go into your your flow table- you have to like do the actual work of the update so like it you're like using a lot more bandwidth and you're, using a lot more um like dpu resources to deal with, like the the the periodic updates and like all of this goes away.

C

If you accept the premise that you'll use a reliable transport.

B

Yeah but again I can easily break that just if you have a link flapped for more than your window allows, which is really not a lot. So how much do you uh do you estimate.

C

The window size as as bud pointed out, if you have a link flap for like more than one second or two seconds.

B

I'm talking about the short one, I'm talking about the really short one, which is quite common.

C

Okay, I I mean in the really short one um I mean that the same thing would happen in both in if you have like a short link, flap right, um you will like, in the one case like let's say, um syn packets are coming in and you're processing those syn packets and those of those event driven messages are now being dropped because of the link flap right.

C

Well. Okay, now.

B

C

Now the link flap goes away. Okay and the ack packets come on the other direction on the other dpu.

C

Okay, those act packets will be dropped because the synchronization didn't occur.

C

Okay and it will take like one second. For for that, uh uh you know synchronization to to to happen um and then and then I guess the act packet in the other direction will work.

C

Okay, if you take the reliable transport approach um you don't have to buffer, you know, like you know, you could decide like how much buffer you're going to have for very short length. Flaps, but, like you, don't have to buffer more than the window size, and if uh you know, if the link, flaps and- and you know the window fills, then you start dropping syn packets and it has the same net result which is like the new connection is not opened up until you can get the state reliably. Synchronized.

B

Okay yeah, so a new the same net result: okay, fine, uh but then I will ask you this time, give me the size of the window. What is the expected window size.

C

um Yeah I mean, I think the expected window size is small. I mean I think it could be. You know like these. Packets are large. I think that this window size could be. You know, 128 packets, let's say.

B

128 packets is just it's kind of like I don't know it's a length flap of a time. I don't know one.

B

Even if we have a link flap for one millisecond, we won't be able to accommodate for that and what happens next? Do we declare it down.

C

Well, I mean, I think, that's a decision that we have to make like at what time at what point you declare a link flap like a failure of the link. Okay, that's like an independent decision.

B

But you won't be able to cash for one millisecond of linkedin.

C

We don't, what we'll do is we will flow control um and the flow control will cause us to not generate new state update messages and, and the way that you not generate new state update messages is to is to basically drop packets. That would have caused state updates.

B

And how does primary know not to generate state update messages anymore, because it doesn't know that the link is down. It's not.

C

It's there's a flow control, there's a window once the window is full, then you can stop putting new messages in into the buffer. Okay,.

B

So hold on, but then does it mean that we receive back for every flow control for every flow synchronization message that we know that how do we know that.

C

B

We should stop.

C

How do you know that what.

B

So imagine that the primary sense of flow sync updates to the uh backup.

C

B

C

Mean I imagine more, this is like active active but, okay, let's, let's just go yeah.

B

But, but for every flow there is one that received the sin and will stand.

C

Okay, okay, yeah.

B

So that that's the primer.

C

B

It goes to the tour from the tour it goes to this uh to the back up and link between the tour and the backup goes down. Primary doesn't know that the link went down. What will happen next like? How will it stop in time so.

C

Okay, so what will happen is that the sender, okay and, like you know, there's senders, there's transmitters in both directions but, like what will happen is um the transmitter will will send up to the window size and not have received an acknowledgement.

C

Okay, and once that happens then, but.

B

Hold on you said that there is no acknowledgement. Am I missing something.

C

The the reliable transport protocol has has acknowledgments.

B

For every packet.

C

um It the acknowledgements, don't have to be for every packet, but the acknowledgments have to be at least like you know, as frequent as your round trip time. They don't have to be for every packet, um but generally marion like there will be packets flowing in both directions like synchronization packets will be flowing in both directions.

C

Just because, like you know, like you said, some connections originate on one dpu. Some connections originate on the other dpu um and so like there's kind of, let's just say, a balance between like each dpu sending state synchronization messages to the other dpu and the acknowledgements like piggyback off of those those data packets, um so like you'll, have acknowledgements you'll have data and you'll have acknowledgements like continuously flowing in both directions.

C

And again you don't have to. We don't have to use this this protocol, this, like maybe some, is simpler than tcp but like it could be tcp yeah, but this.

B

Tcp suffers the same problem. Well, not really. The same problem tcp cannot withstand that because it's a so my point is that reliable transport requires you to have an end-to-end flow control. You don't really have end-to-end flow control. Eventually we end up with the same thing. You are dropping new connections that are incoming that you cannot accommodate for because you cannot go back to the vm and say please pause your application. No.

C

No but marion marion like we never go to the vm and say pause your application.

B

C

B

We do actually, if, if you take like really uh lossless uh end-to-end protocols like rdma, that's what they do, they pause the application until they can transfer all the data.

C

So marion like regardless of ha completely independent of ha each vm, is, will have a connection per second rate right. That's been provisioned for that vm, okay, yeah. What happens when the vm exceeds that connection per second rate? How do we tell the vm that to stop new connections? We don't.

B

That's completely different, that's what the customer paid for. If the customer paid for less cps, they won't receive more.

C

Okay, and how do you tell the vm to like stop.

A

Is that via metering or the the counters or the metering buckets right.

C

C

Yeah, that's right. Yeah.

B

But you you don't have to tell the vm that vm is breaking the rules here. Vm is not breaking any rules. It's your problem.

C

The vm's not breaking any rules, but you can always drop a syn packet in the network.

B

Right as well as any other uh tcp packet, and it.

C

B

Be retransmitted.

C

That's right, and so in this, like you, know, scenario of like there's some very short link flap on the transport between the two dpus.

C

It's effectively the same as if you dropped a syn packet in the network in the you know in end and it'll get re-transmitted and the vm will get exactly as many connections per second as they paid for.

B

Yeah, but the difference is that difference goes down to the implementation. So probably that's why? um Because up till now only I was asking questions. Maybe other vendors want to put their two cents, I'm not sure. If, uh if the end result is the same, what is the preferable way for the implementation in the hardware, because for me it is much simpler to have a stateless implementation and lossy as opposed to what is uh proposed here, to keep a window to retransmit.

B

This seems to be from the implementation standpoint uh much more complicated if we agree that the end result is the same in case. If we have a link flap longer than the window size.

C

Right so marion, I think like you're, saying okay, it's much simpler to just do this. Stateless thing like I think the counter to that is like it doubles.

C

The doubles, the bandwidth between the two dpu's, potentially okay, and I don't think it actually is simpler in the sense that it uses like twice as much dpu resource to handle these, like periodic messages that, like 99.9999 of the time like weren't necessary, so the dpu is like handling them, processing them not knowing whether like it, it is a real state update or it's just reflecting like the state it already has. So I so.

B

Yeah we can talk about the uh the periodic updates if we really need them or if we lose a packet or uh or a few packets. What will really happen then? Can we resynchronize only those uh without really uh doing retransmissions?

B

C

B

Valid question: I.

C

Think the periodic, the periodic uh you know the periodic messages is like another form of a reliable, reliable messaging, except that, like it's, it's not um like selective to only send the messages that like needed to be resent, it's sending all flows over and over and over again.

B

I mean that's like the difference. Okay, but how will you then synchronize long living flows.

C

So, okay, how will you update them.

B

C

Okay, so let's say you have a long-lived flow, let's say you get to the established state. Okay, now, let's say the established state has a four minute um age: idle, timeout, okay,.

B

But okay, not never.

C

I think four was a number that has been talked about in the past is like this is what azure currently has. I don't know um if, if there's a better number to use than four minutes, that that's fine, but let's just say it's four minutes.

C

Okay, um what you would do is like uh you would send an update uh to your peer um like sometime before that four minute expires, okay, like if, if the flow is still active on your local dpu, you wouldn't keep updating. You know your peer like to say: oh I'm still active, I'm still active, I'm still active. You know that your peer is not going to time out that flow for four minutes. So as long as you update your peer, let's say after three minutes and 58 seconds to say: hey this flow is still active.

C

That's like enough for the peer to like add an additional four minutes of time out uh uh to the pier so like that you do need to have periodic updates, but only at the idle timeout rate. They don't need to be at like a one. Second rate.

B

Okay, you still need to somehow pick exactly those which is not clear to me, but then uh going back to original argument. If we were saying that we have.

B

Let me put it right: we have um well, let's say with the four minutes uh timeout if we would put periodic updates to 10 seconds, it makes our primary and backup or sender receiver, be out of sync for at most 10 seconds, which is which reduces uh or in other word ratio to the period of periodic updates to uh um to the newly opened connections will be one to ten, which we have 10 seconds.

B

uh Every 10 seconds, periodic updates, which reduces uh bandwidth for period, periodic updates to one 10 of bandwidth or for the event-based notifications.

B

And still, I believe this will be much simpler to implement the keeping state in the hardware.

D

So mary, so I started to ask this question earlier about how asymmetric flows are handled in the event of the h, a link going down, I mean so say a syn packet comes in and it can't be transmitted to the pier um and the reverse flow is that synack is going to arrive in that pier like how does that work?.

B

Yeah, so eventually you will.

B

Even with the periodic updates, you will catch up.

D

So this so the syn ack will be dropped right I mean and then it'll get re-transmitted.

D

And then re-transmitted eventually, because because.

B

Everything is re-transmitted.

D

Yeah and so say, if you.

B

Unlike udp interesting question, I didn't think what happens to udp because I don't really know the uh use cases for udp, but yeah.

D

Okay, so if you, if the synchronization messages between the two peers, say you retransmit everything, every 10 seconds, then after 10 seconds, the peers flow table gets populated, the ack is being continuously re-transmitted by the remote host and then eventually the fin act. I mean sorry, the synap comes in sees the flow tables populated and then the connection continues is that the idea.

B

So if the link goes down in any case, I think we can agree here that some packets, some portions of the flows, will be dropped.

B

For me, a much worse trade-off would be to try to implement this.

B

Kind of like a state machine with the window and retransmissions plus, I don't really see how it's gonna work: okay, uh we're saying that uh we will receive x and we can say that for some time, if we're not receiving an ack, we will start dropping uh even faster even before the link goes down and we understand what was happening.

B

I can understand that, but still I see a big problem with this implementation. I don't know how we can really maintain all this and the hardware.

C

Well, that seems like it might be a limitation of your hardware. I mean I it it's. um You know, I think, if you just if it just look at like the bandwidth, you look at like how quickly you get back to a synchronized state and you look at how much like dpu resource it takes to handle updates, like the reliable transport is, is like more efficient on on all of those. um If the issue is that you just can't implement a reliable transport, I I mean I I don't know what to say um well,.

B

I don't think that I really uh there is uh I'm looking at it as a caused cost-benefit scenario, because we will solve for one thousandth of a percent of uh traffic in the network or less.

B

And won't really gain much of it, because even in your proposal, you're saying all the time that network is engineered. Well, you don't really have any losses uh it, except from what I can think of which could be common like uh first one would be link. Flap second, would be buffers.

B

Because we, we are still going through the tour or in case of smart switch through t1, most likely, so you may have buffers clogged up.

C

Again, I mean um you know the the amount of bandwidth and the amount of events and synchronization that's going through. This path is very high, like the the processing load of the synchronization messages is like almost as high as the as the um data packets and no.

C

What do you mean no.

B

C

B

Even according to your calculations, it's like 10 to 12 gigabit per second for event, based methods.

C

Yeah, I know, but I'm talking about like, like you know, flow table like updates. You know, like you, know, doing a lookup in the flow table and updating the updating, the state right. uh You know you're getting data, packets and you're doing lookups in the flow table and you're updating the state based on um uh you know the flags and stuff in the in the in the packets.

C

Like that h, a like state updates are generating, like a very large number of like load in terms of like flow table uh lookups and in updates- and I would say it's like you know it's equivalent load to like the to the to the load on the flow table from the data packets, um uh because it's symmetrical right or or you know, or or in some cases asymmetrical like the like.

C

The two directions of the flow are on two different dpus and so like it's just you're, getting as much state update from your peer as you're, getting like from your locally arriving data.

C

Okay, so, like the h, a is like a big source of of work that has to be performed right, and you know in in with like this is like so fine-grained of like state update that it's that um you know, if you were architecting a product you would say I would. I just want a like dedicated link between like these two dpus, um and I.

B

Again, disagree disagree doing some kind of like an analog, dedicated link, type scenario.

B

We cannot. We cannot take this as an assumption.

E

Hey, hey, john and marian. This is, I think, you know I'm fungible, maybe it's a good idea to chime in at this point. I agree with marianne that I don't you know a needing a dedicated link is feasible.

B

But I don't think.

E

That is a requirement, though, for a reliable uh transport to have for this control connection. I don't think we should have to insist on having a dedicated I'm.

C

Not insisting on a dedicated link, I'm just trying to draw the analogy to like you're trying to create a dedicated link through the network topology between these two dpus. Of course you know it's lossy. Of course it goes through switches. I I understand that yeah.

E

My point is, I think, that's the argument that throws off the whole theory. I am actually in the favor of creating a reliable connection between the gpus for the state updates and in terms of whether it should be a live update or an incremental update. That path is, is the same across between these two approaches. Right should, should we update only increments incremental state um or not, I mean even in case of periodic update that marian was proposing. I don't think it's updating every repeated uh connections right. It's only the changes.

C

No, it, I think, it's it's updating every single connection, because it does not know whether whether the peer reliably received the previous update.

E

I see your point yeah. I I think from a simplicity perspective, having a reliable connection between the dpus and having lived up, live updates makes more sense to me and it could even be tcp why to invent new product.

F

Right there is this: uh you know, connections per second in the vm and then there's flow, add rate, and then we know that certain packets will be dropped. But the focus is for those float entries that we have already added. We need to ensure ha works and to communicate and have that connection between the h, a pair that these are the connections that I am looking at and these are the connection status and that needs to be synchronized with the ha pair.

F

I think that is important, and I mentioned that in the last meeting too, for the control messages, how it will be done, whether it needs a new protocol or it can piggyback with all the acts and do it in line, as we add the state, um but it has to be synchronized with the h. A pair is my thought.

F

I haven't gone through the pr from john, which I will do, and you know have some more comments for the next time, um but some method to ensure that the flow entries that have been added and being tracked in one of the ha pairs to be communicated and synchronized and kept in a complete alignment with the hap is important.

A

I think um guys, I think, we're running out of time for this meeting. uh Would you all agree? Maybe we need to have some more discussion next week.

B

Probably yeah there was still stuff that gerald wanted to come back to us.

A

B

A

B

A

And he said he would come back with that next week I mean just to add a little bit of levity. Can't we just lay a cable, the home, run, cable on the data center floor and just connect these two things. I'm just kidding that'd, make it easier like a direct connection.

A

Yeah I'd love it I like that yeah. Let's just do that, um I'm kidding great well. I think I captured a lot of notes, um a lot of questions, I'll put it in my notepad for next time. I think you know exploring both paths. Is uh it's a lot, but do you guys feel like we're at a good ending point for today, yeah.

C

Because I think I think we're just going to keep repeating.

A

B

A

B

Yeah, maybe next time uh others can look at the proposal and other vendors and think and say what they think because look.

A

At john's proposal.

B

Yeah yeah, okay,.

A

Yeah yeah, I did put it in the chat window, but I'll send out an email to all of us with it, so so that we.

B

A

It and I was reading it while we were talking as well so okay, great I'll, send that out and uh thank you, john and bud marion for all the work and everyone for attending and I'll give gerald a um summer is a. I see him tonight at seven o'clock I'll give him a summarization of what we talked about.

A

Okay, okay, I appreciate you guys. I'm gonna stop the recording and I'll post it in the youtube channel.

C

A

C

A