DASH Behavioral Model WG, 10 Feb 2022

Previous Meeting

⏯

youtube image

►

From YouTube: Behavioral model 20220210 (Feb 10, 2022)

Description

Initial behavioral model sync call

A

And now uh the act from uh the first act actually is good, because now we don't care about the outer fragment because it wouldn't have sent an act of it if it had uh an out of order fragment that hadn't been received, but also to close down the connection completely. We need the the the act coming back from the far side.

A

So what I wanted to make sure is the model that you have is already covering this case and is not going to one if we, if we force this out of order fragment that comes in later, that it doesn't get dropped and two that it will receive both acts. Otherwise we won't close connections properly.

A

So I guess that's a question that we wanted to put out there because vip does it quite differently, they just see the fin and then they set a timer, and then they wait that specific period of time, which is quite long by the way, and then they take down the connection. I don't want to do that. It's very wasteful.

B

So can I comment on this gerald?

B

This is john, I think like if you really want to like, remove the connection sort of as aggressively as possible, but allow those acts to go through then like the proper way to do. That would be that the fin itself has a sequence number and then the subsequent act would have to be the ack for the sequence number that covers the fin.

B

So, like there's significant logic to be able to like track the sequence number and the acknowledgement.

B

You know associated with that thin packet, but if you do that, um then like when you get that final act, you know that it's the final act and you could immediately remove the flow right rather than waiting, five seconds or or longer um right. So that's what we want.

A

To do because we're because remember in vip, they're doing like 20 000 characters per second. They they're not as worried about us who are talking about like millions of connections per second.

A

The timeout is just a regular ager, like you already have that.

A

So I I think what john was saying is you. You could just actually look for this. The sequence number of the act to make sure it matches and if it does there's an ack ack as long as it does match, then we're clear to shut that down and I think that's the fastest way to get it shut down and that's what we want. We don't want setting.

A

First of all, timers are really expensive and so it'd be far better to do it this way, but that's we were looking to see if marion's model is even covering this. I'm sorry marion, I didn't read the people.

C

A

Not yet, however,.

C

uh However, uh when working on the proposal for psy apis, uh we actually in the example we covered uh finn, fenec and ack, um I will. uh I will also incorporate that into the model.

A

C

And uh I will need to take another look into how out of order packets uh will be handled in that case. If what.

A

I think you don't have to worry about the out of order, that's the because, once if you, if you do what john was saying just say, this act must match the sequence number of the fin. Then.

D

Don't worry about.

A

The out of order packet means.

B

A

The act comes back. It means that the tcp has received all the packets.

C

Right, yes, so we weren't really looking at the sequence number. We were just looking at the flags like first, you need to uh first, you are in the state of waiting for finn. Then you move to the state of waiting for finnec. Then you wait for heck after.

A

B

There may be many acts because, like you, could send a fin when there's like lots of outstanding acts and like many acts could come after the fin, but like only one act will come. That like covers the sequence number of the fin and that's the final act.

C

Right, so what I'm saying is probably I will need to improve, not only to look at the sequence of the flags, but also at the uh sequence number of uh those packet packets that I expect to do. I.

A

Agree right, so that's that's, uh and, and so this is solving two problems, one. We just wanted to know if that was covered properly, but two you don't have to worry about the other order fragment just forget about it now, because this covers that scenario and that's.

D

D

Scenario in the chat window.

A

Oh, I don't have the chaplain if andy's he can.

E

He can ask the question himself, I don't know. Oh, not non-first ip fragments have no l4 ports in them. How do you want to classify them? Do you want to classify them in the context of the first fragment being combined with it somehow and then make the classification, or do you want to write your policy rules so they decide they explicitly say what should happen on first fragments regardless without having to know what their l4 ports are.

D

So it doesn't cover it. Okay,.

E

It has nothing with timing unless you want to combine the all the ip fragments together. So you know what their reports are before: classifying them. Okay,.

F

Yeah and if there's a flow cache issue, because of that reason as well, if you, uh you know, try to do uh disconnection tracking for defragment and for non-fragmented packets, so maybe you know.

A

I don't I'm not sure I understand this. The first fragment should have everything we need right.

F

Nothing does not have a card.

A

Other fragments we don't care about.

G

The first fragment.

A

E

You don't want, you want to drop non-first, fragments that will they won't get through. Nothing will get through.

E

What do you mean drop, not pregnant, so you don't? What do you want to happen to them? How do you want them to be classified according to policy.

H

um Anjali was uh explaining this in some way right.

F

Yeah I mean uh so gerald, maybe like if you remember.

B

Andy like what you're supposed to do there, I.

F

A

H

A

Is we're never reassembling anything we're.

I

Just a go-between.

A

Right we're just a switch essentially to everything. So that's uh we're not really going to reassemble anything.

H

Actually, please uh go ahead. Yeah.

F

So the question I think andy is asking- and I have the same- one is the first packet because it does not have l4 header. How do you uh you know what tuple do you use for classification, because that that would mean only the l3 parts and that would classify differently than your rest of your uh packets? So.

A

I think that first fragment has the outpour sorry.

D

Yeah yeah yeah.

F

Sorry I I okay yeah. The first argument has the l for the other. Don't sorry I missed uh fixed my words yeah yeah.

J

The first one needs to have right, because otherwise there will be no way for us to know what will be matchings against right.

A

But what they're asking michael is like I'm not even sure this is true that the rest of the fragments have no l4. um That is true yeah. So how do you do classification on something that has no l4 michael.

F

Yeah, so what has happened gerald in the past is like. If you look at rss and stuff, uh the input set that is used for fragmented stuff is just based on l3. It's never l3, plus l4, so that you know like, if you're classifying to a single queue right um uh or whatever. Otherwise, you end up with the first pack in go going in one queue and the rest of the packet going to other queues, so I mean I'm just giving rss as an example.

A

Yeah we can't do. We can't be classifying a layer 3 though so. Actually, michael, it's a good point. Do we even support fragment, like this kind of tells me that I'm getting mixed messages here? How could we support fragmented messages if there's no layer 4 in them.

J

Yeah so so we have a threaded, vp team right and uh I I will just exactly read what they what they mentioned, but but I I'm not an expert of how exactly they are counting this right. uh They mentioned that, and I cite like uh we use both ips and ipids, to group all the fragments of a single packet together, and they also use number of end caps of the fragment on and they are saying that on inbound they are expecting fragments to have the necessary end caps.

J

uh But I understand that this doesn't answer fully the question with regards to lack of enforce. um So the vfp didn't answer this fully in the email that we asked ask them to to do this right. I think we they pointed only to some like external documentation, saying that the customer can send jumbo packets if they want, and they will fragment they will be fragmented in the v-net, and there is some link. Let me let me just paste this link, but I don't believe this link covers this, so I don't think this.

J

Can we have this question fully answer yet how exactly they are handling this.

A

Okay, so when you're going to need to understand that, otherwise the behavioral model is not yes,.

J

A

J

A

While you're gone, though michael one thing we did agree upon, is forget this fragmentation step for a second just how to finish the connection that you will.

A

You will finish the connection by looking at the fin, looking at the sequence number of the pin and making sure that the ax matches it's the last one, yeah yeah, it's the last one, so the sequence number has to to be the same, and that way when we get the second action we can. We can close down the connection safely and that all packets have already been sent by that time, because we would get the x.

F

A

So we've kind of solved that problem and then I think uh mary marion has said that he will update the behavior model to do that uh and now we're now we kind of moved on to how we ever got to that conversation, which is this fragmentation, fragmentation.

J

A

But it turns out that there are two different problems. One is, how do you safely shut down a connection and the second one is: how do you deal with fragmentation.

B

We're trying to recall here like something that we saw, that talked about like virtual reassembly and like a fragment, cache and like uh like it seems like maybe like dealing with fragments, is like a stateful thing, but maybe you don't have to buffer all the packets, but I'm not I'm just like talking off the top of my head. I don't. I don't have like a full recollection of of this.

B

E

B

E

Think and the public github issue that microsoft folks can read and consider, but I'm not trying to drive it, I'm just trying to get to the requirements they want.

J

Yeah, christina, I I think for the fragment stuff and our actual ex, like actual pocket handling right, we should involve someone from the vfp team right because neither of us right now on the stuff, maybe gohan, because they have some experience with the switches right. Maybe you have some some details or prints, but uh I know that I don't have exactly information, how the vp handles the fragments and how they are matching this and- and they will know so. We need this team to also join us.

J

Potentially in some discussion, which are on those kind of specific topics.

D

Maybe I can just make one small comment when I talk to the vfp team michael, this is super generic, but they they did say. The first fragment should contain all the headers needed yeah. We can send through to determine the action needed and apply to the rest of the traffic. Okay, how they do that. I don't know, but that's what they said. Yeah.

J

The first right, I know that definitely they are waiting for the first fragment right uh because like otherwise, they have no way to basically match the policy right. Yeah.

D

J

The questions like how they later match the rest of their fragments.

D

Exactly yeah, they say say how they did it. Yeah.

E

Yeah today, but if you know who to follow up with that'd, be great.

D

J

Yeah we have our team, we kind of started following up in the email, but um the answer they gave is is doesn't have yet the answer for exactly how of the implementation, how they are doing this right. It was to generic it so far,.

D

How about back together.

K

So I think I mean you, you use a fragment id to match it right. So what is the problem for that?.

E

House some some hardware fast pads, can't do that and some can can do what buffering indefinitely and matching up fragments by id and l3 source and dust ip. It's the buffering part.

A

Oh okay, we don't want to be doing reassembly.

B

Right, it's not reassembly! I think again, I think, like um basically, the fragments have like temporal locality right, like the fragments. All the fragments in like the full packet are going to be like pretty close together in time right. So there's temporal locality so like caching of like, uh like some kind.

C

B

Cash that maybe works uh because, like you're just trying to cover the time over which, like those fragments from the given parrot packet like exist, and so like, I think the solution involved a fragment cache but uh and not buffering the actual packets, but just maintaining, like the flow information associated with that. With that packet. Like the five tube.

J

And I wonder, can you maybe I'm not quite sure if this is done right now in harder? But what about the sequence number ids to to kind of do this together, like in a way the range of the ids right? So so, whenever pocket arrives, we can always, for example, store what was the sequence number id of the packet.

J

If this is the newest one, we will take it right and then we know that the retransmission of the of the sequence for the x right will be for some, like current sequence, minus x right- and we expect in this case, to match the packets for the like three tuple in this case, like the source, ip destination ip and the protocol, plus the sequence number according to the to the range right, uh I'm not quite sure how viable it is.

J

I'm not so much familiar with the full data pod, but there's also potential approach that can potentially avoid caching.

K

I think I mean I feel, like I don't know, what's the problem right. So is this on the receiver side assembly problem? How do we do you know, do the uh arm fragmentation? You know, assemble those packets and we don't have information for that or well.

J

There's nobody it's more like forwarding appliance right. So john john.

K

You were saying.

J

Something, I think gohan anyway,.

B

I was going to say it's like effectively like doing reassembly, but without buffering it's just maintaining state but not maintaining the actual packets and not actually doing the reassembly and to have some cash of some number of those that you can have in flight.

B

I think that that's the solution. I don't understand all the logic of like uh implementing that, but I think that it's a stateful, it's a stateful thing uh in order to deal with the non-first fragments.

I

Yeah- and I think that's true john- um you don't need to buffer packets as long as they come in in order. If they come in out of order, then you would need to buffer or.

B

Or drop them and let them be retransmitted right.

J

Why not? Why not just forward them you, you still need to block the.

K

Number right so I mean I don't understand that because the first pack arrives, you have to wait until the second packet until the last pack, you all derive, and you simple one pack and deliver it right. So no.

B

No, no, no. The issue here is that you can't do the five tuple classification of the fragments, if they're, not the first fragment right. So how do you apply the sdn policy?

B

How do you do a flow lookup like on the non-first fragment, and you can do that if, on the first fragment, you maintain some kind of state and some kind of cache so that when the following fragments come in, you you've saved off like what the five tuple is or what the l4 information is for the for the subsequent fragments, and because I mean the fragments in theory, should have temporal locality, because, yes,.

K

But are we saying like? um Are we controlling both the sender side and the receiver side? Do we have the assumption.

J

No, but but the so so I think this is like this, like one thing is like policy lookup right for this. We need the first fragment which have everything right, and I.

K

K

I'm asking questions like: do we control both sides? You know the the the the fragmentation and the defragmentation. Both sides are in our control or we are not.

J

We are not so one side can be over the internet right.

K

Of course, of course, but you know we're doing encapsulation right so and doing the calculation on both sides, we are on control right.

A

No, the user sends the fragmented packets and we encapsulate that we can't change their payload. No.

K

No user never send the fragment packet right, so it's always a way to department the user. We.

A

Know we don't know no.

K

J

In case basically, the customer says the bigger one yeah so.

K

We never do this for the tcp traffic right, so we only do this for the utp traffic because for the tcp for the tcp, I think we should have in the spec. But I don't know if we have that, but you know there's a mss tapping technique that we use so that you know whatever the path you know the customer negotiated. Mto will be clapped by the um by this is the implies layer right so.

J

If you are saying that we control fragmentation, what are what is your idea that what we could potentially do in this case, if we control both the servers in this nation.

K

Oh, you know it just that would do not. You know they want to flow id something kind of to to do that, to do the classification to a particular flow right. So I think at the center side you can inject some information in the encapsulation layer and then, after the after the receiver side, you can extract that flow id and the map to the flow right. So because, because you add, when you do the fragment right, it's you to the fragment, not the customer between the fragments.

K

Therefore, you can inject all those metadata information in your encapsulation layer and those metadata will be consistent and transfer to the to the to the receiver side and receivers that can use that metadata to extract the flow id and get the things right so yo you.

G

A

These things are landing on existing vms, so once one vm, which is attached to an appliance uh has nothing to do with the far side vm, which is not attached to the appliance. Really it's just a so yeah. That's that's.

K

What I'm saying do we do? We have control on both sides right, so I think I.

J

Would say I would say we should we should assume we don't, because we are also, for example, selling some packets to let's say on-prem right with different different devices and this kind of stuff like so I I think this is. It would be nice, but I think it's a very strong assumption that we control both sides, but.

K

If we, if we don't, have control right, so if we don't have control, I think one side must be the uh the vip right so because of the existing deployment today, I think we just need to ask the vip to give us what exactly algorithm the secret sauce. They have yeah on the other side, and then we do it uh on our side right. So you know, I think, they're talking about the encapsulation layer. I think they may have something there.

J

I know that vfp, because there was some company that was asking us to correctly handle fragments and this kind of stuff for the udp, and I know that vip introduced some caching, so so this kind of stuff. So so I I'm not like. Probably they have some like, they definitely have some way of how they are handling this right. It may be that they require caching, so some buffer and those kind of stuff, right and- and the question is like.

J

If not, then it's awesome, so you have different upgrade right if they require caching, then the question is like how big caching can you do on the hardware and can we avoid it right and the interesting observation that I want to mention here that the connection establishment is not really a problem, because the first fragment will have the five tuple. So the main thing is really look up too much.

J

The existing flows right and I'm just wondering and right now is like brainstorming right, because all the fragments have a sequence number and- and john you mentioned. That is this, like time, locality right that that, for example, the fragments will will not arrive too late, not arrive to earlier this kind of stuff right. So I was wondering like if we do what is like example of the brainstorming algorithm.

J

There can be also different, different approaches right if we do three double match and if this street app will match after this tweetable match, we need to match the sequence number to be within the last sequence, number that the packet saw plus minus x right and if it returns one single match in the flow table, then we know it's for sure for this. If it returns more than one single match, then we can always discard the packet and allows that basically them to actually retry till we get this, so this will be slightly slower.

J

Potentially this will solve it.

B

Yeah, I think that makes sense. I think that's how it would work, but I think the three tuple match would include the ipid, like all fragments of the same packet have the same ipid and then, like the sender, that's doing the fragmentation like increments, that ipid for, like the nest, the next packet, that's being fragmented, the next packet that's being fragmented, so like the three tuple lookup makes sense um and then, of course, like you can check like sequence numbers and make sure that the packets are like the fragments are in sequence.

B

um Ultimately, you need a cache of those. You don't need to store the packets you're, not doing actual reassembly you're just trying to like store the five tuple. That's associated with that three tuple.

J

Yeah, but simply we always need to store the five tuple, because basically the flows need to store it right because, like part of the flow creation will need to win to keep it anyway.

B

You you do, but you need to associate now like a three two pole with a five tuple in the case of like when you received a like a first fragment, um because the first fragment has the three tuple and the five tuple and then the following fragments just have the three tuple right so like you need to have some kind of a cash just like you said and like how big is it um you know, what's like the logic for, if you got fragments out of order, do you drop them like there's, there's like a lot of like details around that.

K

But I think we can understand, you know the organism can be designed, but what is the excel for udp packets? Do we really have to do them in the hardware I mean. Why can't we say you know this fragmented udp packet, you know, go to the cpu and let the cpu to do the defragment.

K

Why do we have to do the fragmentation? The hardware udp right.

J

Yeah the cps like like number of packets per seconds processing as well, not smart, cps, the connection per second but pps right.

A

Yeah, we're not sending any, we don't send any data path to the arm course go on. We do millions of connections per second.

J

So for from our point of view, the cps, so so, let's clarify the idea right. So the idea that why the cps is slow because, basically just the rule processing is slow right because we don't have a fast engine to process the rules right, but the rules are being hit by both tcp and udp packets right. So we want to basically optimize the rule processing engine, so we can handle basically this kind of like new packets, that we haven't seen of the exiting flow evaluation fast right and then there is this stuff.

J

This is maybe also maybe the question for the fpga team because they offloaded because, like all the flows that gets created by the vfp, they always gets uploaded to 100 right now. So we have it right now, right so right now, every single flow that is created we already offer to hardware. So the solution of matching the packet to extinct flow in on hardware level. It's already solved on fpga right, so we can.

C

Potentially ask the pga to.

J

Tell us how exactly they solve this kind of stuff, so I don't believe that that the matching of the.

J

Of the of the existing flows is the stuff that we need to rediscover. I think this is already there in hardware right and we just need to potentially use it right. At the same time, I'm not sure if fpga funds- fragmentation, that's open question, maybe they don't, and maybe they just basically asked the customer to retransmit to get this in order. There's an open question.

E

But I I think for this meeting there might be other priority topics, so we could find out the answer later after you guys do your we'll have to some more work on that one.

D

So back to the behavioral model, then or close down, okay.

D

So we have more gaps, we're looking at in the behavioral model um that you guys have seen. You know as you've been looking through it or.

B

I I mean I I I have like a small list here, but there's probably probably a bigger list. If you know I integrate.

G

B

We'll start with the smaller list sure so I didn't see anything in the behavioral model that kind of separates out sort of fast path and slow path.

B

You know the documentation talks about the slow path, as you know, the acl, the route lookup insertion into the flow table and it's the documentation sort of shows the fast path as you hit in the flow table, and you just apply the transforms, and so like is the representation of slow path and fast path, logic supposed to be in that behavioral model, or is it such that you can just do slow path on every packet from a behavioral model perspective um and.

J

We're, okay yeah. We are okay with it, so so basically like. If this doesn't impact the pps right for the already established connections right we can we can. We can do the slow pass for the for extreme ones right, because the the the fast path like hitting first, the already established transposition right, the flow right. It's mostly important from two point of views right, one point of view, which is basically, we don't need to do entire role, processing, which may have lots of prefixes right.

J

It is much much faster, so the pps is much faster right and the second thing, the h8 scenario, because because in case we have two cards which are talking to each other or two switches which are talking to each other.

J

They need to exchange the state right and the idea is that the goal states, so the rules that we are plumbing is eventually consistent, which means that we, because those are in the two physically different locations right, there will be no way that you will guarantee that they will get applied at exactly the same milliseconds and this kind of stuff right, which means that the, but once the customer establish the flow, the flow needs to work in a specific way that was established. Otherwise customer connection will break right.

J

So that's why part of the aha is like once one device uh creates the flow and sets up the specific transposition based on that current rule. That time this flow basically needs to be kind of transformed to the to the backup appliance in order because like if they can, because the main thing about the ha right is when the device dies, we can lose packets, but we cannot drop established connections. That's the main goal.

B

So so, like aj is, like a whole big topic right yeah. This was like this was like a much smaller point, just that uh the specification like describes flow path and fast path. In one case, the slow path does like the full acl the route lookup. All that inserts the flow fast path does just the transformations, and I wasn't sure whether the behavioral model should represent fast path and slow path or whether, like from a behavioral perspective, there's no difference from saying, like every packet just goes through the slope, slow path.

A

Even it, uh I don't think anybody will implement it that way. I've heard that, but that's uh you'll never get the cps out of it, but it's still one piece ago: it's like you always do the flow lookup. If you don't.

C

A

You do the rest of it, so it's only one one behavior, really it's just. How far do you go? Do you end at the flow lookup? Or do you end uh you do the flow look and then you have to follow it up with the full slope.

B

Right I get it I mean. The fast path is is um is like a shortcut. You know that you can take after the established. I agree with you. Probably everybody will implement some kind of optimization to do fast path. The specification kind of specifies it that way, but the behavioral model doesn't implement it.

B

That way, um and are we okay with that and like I was trying to get at this like maybe a week ago when I was asking uh michael about like updates to the configuration because like in my mind, if the behavior you want is like you're going through the slow path on every single packet, um then an update to the configuration would be like immediate, um but I think the answer we got a week or two ago was that no, like updates to the configuration, don't have to be immediate and like even for existing flows that are already established, like that updates to the configuration, would have to be applied to those existing flows, but not immediately exactly yeah, but they could be applied like a little bit more lazily.

B

They don't have to be applied immediately. So like I, I understand all of that and I think I understand what we need to do to implement a data plane. But when I look at the behavioral model, um it's just implementing the slow path and ah okay.

A

J

Think I think the main thing why why?

J

Maybe there is no part of this like fastpass behavioral model, because at least this was my kind of assumption that the first part is just the lookup in the flow table and kind of like every single vendor for sure has already some flow tables functionality and is doing the lookup for for those kind of stuff right.

J

So that's why it is kind of like documenting what is already established way of looking up packets in the flow table to kind of match it, because because the main thing that differs here because like at the end after the transposition, basically this goes up to the flow table right with all the potential metadata it to to kind of do the transformation right, so that the entry in the flow table will have just always, for example, the the source kind of packet metadata, which is like a five tuple, which was already there right and whatever other fields are required too much right.

J

And then the information about what's defined transporters to, for example, at one end cap at two end caps. And this should be that the outer header, this kind of stuff right and then.

J

Yeah I look up the table yeah. I think it's potentially it's beneficial to also document exactly how to do the look up there, especially from the fermentation, also perspective. So we can also potentially spend some time to to document the fast part as well and then do behavioral model for the fast part.

A

I I think so because we're gonna find all kinds of exceptions to when we do aja and other things that, if we don't try to keep this model close enough to how people are going to implement, it's really not going to cover down the road. As things get more complex, all the cases- and I I don't know why it wouldn't have a past path.

A

In the behavioral model, it's going to come up later, somehow, even in when you do memory size, you always include the the flow flow table. Why? Because it's the biggest table of all right and that doesn't even even consider that there's even such a thing, so it wouldn't take anything for them to implement the fast path as part of the behavioral model. Why? Wouldn't the question I would say is why not? Why not implement it?.

D

A

D

And is that a difficult thing to do marian.

C

I'm trying to understand: why should why the difference between slow path and fast paths should be reflected in the behavioral model, since different, even even within our company? Different generations of the hardware will have uh differences between what goes into a slow path. What goes into the fast path, I don't know how we can generalize it with the behavioral model,.

C

Like what, if, in two years, we can do learning in hardware, so now we need another behavioral model for that.

A

Do you mean learning we're already doing learning we're doing the slow path? Is the learning part of it? What do you mean?

A

I mean today the excess.

C

A

Just means you've already done all the processing, so you don't need to do it on every single frame. Henceforth, why? Why would you.

C

ah Okay, okay, so sorry, my definition is different. My understanding of fastpass is different. Okay, uh this.

D

C

This is, this is reflected in the behavioral model like if it's not the first packet of the connection we already know about this flow. We are not doing the processing of uh the acls and other uh stages for every consecutive packet of the flow. It is already there.

A

Okay, so that matches the documentation. So why uh john do we.

B

Think, okay, I mean, maybe um I'm not understanding the p4, I'm not a p4 expert, I'm not maybe I'm not understanding the p4 code but, like I didn't see the flow table like the only flow table I saw in there was the connection table which just um basically only like to have the state of whether like it was established or not. I didn't see like transforms or anything like in a flow table.

C

So there is, uh I can actually send you a link to exact place in the code, but there is an explicit verification. If the connection is established, then we do not do acl lookups.

B

B

What about the route lookout.

C

Route lookup in the behavioral model route lookup happens every time.

B

Okay, so you don't have like a flow table with like the route, the transform that came from the original look up of when you ran the slow path I mean, maybe that's what I mean from a behavioral perspective.

B

That may be fine, um but it's like doesn't match like like the spec is trying to like um provide guidance on like what you should do in a slow path and what you should do in a fast path and like right, like you're, saying like okay, like we'll skip the acl, but we'll still do the replica now, maybe that's how your implementation will work. Maybe a different implementation might store the uh the route entry itself in the in the connection table.

C

And you have every right to do so: it doesn't change the behavior.

B

Right, it's just what it it's just that it's like when you read the spec and you look at the behavioral model. It's not like obvious to me and it's fine. I mean I'm okay with it uh like. If the behavioral model says it's gonna, do the route lookup on every single packet.

B

As long as what michael said before that, like updates to the route table, don't have to be reflected immediately in the data plane, which you know because, like for example, let's say you stored the transform in the in the connection table well like it may not be that easy to take a route update and now propagate it to every flow in the flow table. So that's going to take time and that can be done. Lazily like we understand, but it's like the behavioral model doesn't to me the behavioral model sort of says.

B

Oh the route configuration would just be like immediate, because every packet is doing the right, lookup, it's not being cached in the flow table.

J

And also fyi, if you guys need some information, how we are doing this in vfp the in case. Basically, there is a conflict change. They usually increase, something which they call like epoch number of the, but not for the car, but the eni right and each kind of like flow that was offloaded has this number. So whenever, whenever packet comes, then they basically notice that it doesn't match, and then they invalidate it and and redo entire contacts. Slowpad again.

B

Right right, right, re-evaluate like yeah.

J

B

The route lookup and the acl maybe right like re-evaluate those because there's been a config change.

I

J

But it happens in this case on the config change right and not for every single packet right. So.

B

Yeah yeah, I get it like on the config change. You update, like the the epoch for the eni and then every flow entry has the epoch for which it was valid at that time and when there's a mismatch, you know that the configuration's changed for that flow or or not.

B

And then you can reevaluate that flow and update it's epoch.

J

Right so yeah and then yeah with this apoc update, also interesting stuff that came up as we were doing also work on uh some different stuff on our site. Right is that uh we noticed some differences between harder versus software right and the idea is, for example, in a software model, things per eni, which means that, even if, for example, let's say two enis may belong to the same customer v-net, let's say right, the same custom virtual network, the same vpc right, so the vpc shares shares mappings right.

J

So all the mappings uh between between kind of addresses, that they're used for transposition are common for the vpc right and two enis may be belonging to the same vpc and may accidentally land on the same card right. So in vfp, all the state is actually copied per eni. So in this case the the vpc, even though, is potentially global object right it. It is not handled as a global object. It is basically copied per eni, which is not very efficient. This kind of stuff right, but at the same point, all the all.

J

The updates are kind of like updates the epoch always separate. You on both times, right and and here one thing that I want to make sure that in this case, for example, let's say the mappings change on the vpc right, then there is also a need to know which enis potentially or which flows belonging to which enis needs to be handled, and I know that there was some approach that a team tried to in this case.

J

Okay, if potentially multiple enis could be impacted, then let's not use eni based epoch, but let's use card base epoch right so every time once anything changes in the configuration of the card in the respective involved dni they they change the epoch on the card. But but this goes different different behavior that like, if one customer is churning the the configuration, is impacting the performance of the other customers on the ca on the card. If the e-book is global. So that's why ebook cannot be global right at the same point.

J

We would prefer the epoch to be, for example, per eni. At the same point, I understand that there is a complexity, because the best efficient way to handle the object like vpc is to have them kind of global and potentially reference, because this is like one set of mappings it'll be more efficient to store them, especially they want to keep basically one million mappings for pc right, but this requires some ability to track that if there is an update to the pc, potentially those other enis epochs needs to be incremented.

J

So just fyi that that we run into running to this before that, because some objects potentially can be shared by different dnas. We can definitely implement them as a copy right, but at the same point this is lots of memory, so those object having pc should be most likely global. At the same point,.

C

J

Incrementing uh epoch on the global card level causes this problem that one customer can affect the other customers if they were constantly reprogramming this, and we also don't want this to happen.

B

Right, I I thought the mappings were like scoped to the eni um you're saying like that's like, uh like two local scope, two local of scoping that, like multiple enis, might want to share mappings. But, um like that wasn't clear to me, I mean I think, like we were, assuming that uh that mappings were like scoped to the.

J

So I I can do that. The mappings uh mappings are per vpc right, so mappings are inside the vpc and the eni will be we kind of like peering with one vpc, which is on vpc or potentially some other vpcs right, and it really depends like how many, for example, customer will deploy uh vms and this kind of stuff right, because they may they may deploy.

J

For example, only one single vm each customer does kind of stuff right in this case, even though there will be like let's say, 10 enis on the card right, each of them will belong to completely different vpcs right, but there is some potential optimization that we can use from the hardware implementation of the placement stuff right in case, for example, two enis are referring to the same vpc.

J

To have this vpc not be copied because because because vpc can have up to one million mappings right so in this case, but this would be the same right from our control plane. We are okay to, for example, if the, if the mapping for one vpc changes to update always vpc1 and vpc2, because the eni will refer to vc1 the inivpc2, even though underneath is the same ppc.

J

So it makes sense to to consider this hardware and most likely the mappings will be in the hardware in the memory to kind of and- and there are significant number of them right- one million right, so it makes sense to to make them potentially as a top level object, not very ni object and having enough be able to refer to the stop level object. Yeah.

B

It's like an indirection right yeah, it's there's an indirection. You need from eni to mapping table to mapping right. um Okay, I mean that's. That's fine!.

B

I I have one other thing with the behavioral model was like. I didn't. I didn't know whether udp was implemented in the behavioral model.

B

um Like you know, it was just tcp.

C

Yeah right now, it's just tcp. Okay,.

A

It definitely needs to include udp.

A

And it use needs to have ipv4 and ipv6 right. In addition,.

B

And then I had one question that I posted today um has to do with like enforcement of connection per second and enforcement of flow table limits, connection per second limits and flow table limits. Where are the? Where are those limits enforced um like? Would they be enforced from the dpu? I know like the throughput is not enforced in the dpu.

A

That's right, that's right! They would be enforcing this in the deep view, so we can give you some examples there. Let's say I have uh multiple vms going to the same dpu we may over subscribe, but we can't make it unlimited. So let's say a two core vm goes through this appliance.

A

We actually need to limit that to some much lower number. So it's not going to be millions of connections per second, it might be like 50k connections per second for a two core. uh If we didn't put limits there, then the two core could essentially couldn't reach millions because it wouldn't have the bandwidth to do it, but it could certainly reach much higher, and so, if we want to over subscribe, we have to at least have some. We know we need to know by what right.

A

So, if you make things unlimited or a subscription, just becomes a nightmare like it's just a total guess once, but when we impose the cps limit and the table size limit, we can slightly over subscribe that nick, knowing that, just through experience that that nobody uses the maximums of everything at the same time.

A

But if you don't do a max, then you're in nowhere land, you're, just like you, you really can't do that calculation. It's it's! uh You know. So it's one.

B

A

Could literally if it was big enough, dominate the entire nic? So what's the scope of those limits.

B

Is it scoped to the eni.

B

And, like my understanding, is that you're going to take like the flows from a single eni and spread it over lots of dps right? So that's the bandwidth. That's.

A

J

Yeah, that's for splitting the flow and also this kind of stuff. Yes, but in this case we will.

B

Divide because, like we are.

J

Because we will divide this limit like to two bytes kind of stuff, because the flow speeding will also be uh like equal, like from the point of view like uh the the standard distribution should be, should be basically redirecting them equally.

B

J

B

Like gerald's, like example, 50k connection per second limit, if that gets spread across 10 dpus, each one would be set at 5. 000 connections per second yeah probably have a little bit of buffer, but yeah.

A

B

A

We'll definitely have some buffer yeah perfect.

B

Right, the same with the flow flow table size limits per year, right yeah, like how accurate does that have to be, I mean like? um Can there be a little bit of fuzz on that like.

A

Like um it would be, it would be fuzzy and that it would be a bit bigger when you split it. You pretty much have to go a bit bigger, because you know it's not perfect right, um but customers don't care if they get a little bit more cps or a little.

B

A

More table right.

B

Right, I guess I do care about.

A

B

That's what I'm asking is like how much time, because also like in the implementation of this like in the data plane, there might be a little bit of like tolerance that you need like that, like it might not be possible to hit like an example but to have a little bit of tolerance in that number.

A

Yeah it will be, but the tolerance will always be upwards, which is part of the over subscription. Okay, okay, okay,.

D

So, do you guys want to turn this into an every two week subgroup, or do we just want to consider this a one-time meeting or anyone want to help with the behavioral model.

J

We're almost about a minute, I think so soon so next week is for the planning right so next week definitely will not have a meeting.

C

Right, yeah and that's all so we can, we can have.

J

Potentially, like once, we get- maybe those added like let's say, udp, this kind of stuff right and potentially the fastpass or maybe the fastboot is already there right into a behavioral model. Maybe then, on the email we can, we can decide if, if you want to meet to to to additional like look into this or something like this right- and this would be like two weeks from now, for example,.

A

I I think we need some kind of sub group to be honest like just like sonic has subgroups, because you can't have everybody in an audience who doesn't write, datapop, who really doesn't know the details of fragmentation or tcp or whatever uh it's impossible to get the designs done with that many people the subgroups are to bring more like like-minded. So if you're like a datapathy type guy, that's really into the details of how to do that.

A

This is the right subgroup to be talking about those details because look.

D

At all, the things.

A

That got brought out today, perfect.

D

We write a couple of to do's here then to to help out with udp. um Can you guys chime in, and I can write a few things here? I think marion said fast path was already in there.

A

uh Fragmentation was the other thing that we need to look at just in general and.

B

Then the sequence number tracking for the fin, you know and final act yeah. You got to make.

A

Sure that that model is updated.

F

Gerald does the uh dash document talk about the connection rate uh limits and stuff or enforcing.

G

It uh you know between enforcing yeah, enforcing limits, definitely.

A

um Even the subgroup- eventually, I don't not like right now, because we're trying to do some background work for you guys on like metering, telemetry, counters and stuff, but uh eventually that has to be talked about and put into the the behavioral model.

A

But I think that people want to attend. This subgroup are the people who are like datapath coders, like people who actually do that for a living.

D

Anything else you need help with here.

C

uh No, but I was thinking that probably the uh sequence tracking and finfeh neck- uh this is something that I already had started working on, so I I was sure.

D

I probably wanted.

C

To add that in the nearest.

A

Future yeah, we just want to make sure we share with the community what we're doing so. Everybody has a common understanding and learns together, but yeah uh agreed you're gonna have that soon, so that'll be a crosstalk thing. Great did.

D

Anyone want to help with udp or think about it.

A

We will definitely need to ut. We cover udp in our current designs. Right, like that's going into our network. uh We have to do that.

D

Okay, well, if anyone can throw something into the github and tag marion, please.

D

Okay, we're at an hour and two minutes anything else. Yeah.

A

Good first start good meeting.

H

So yeah, christina um sorry, my internet connection was very very bad today um you know, but uh definitely we can meet uh bi-weekly or tribally, uh as you would prefer.

D

A

I think bi-weekly at first and then, if there's less to do later, see once my concern also is that okay we're all on v-net, okay, great, but we have seven service models and at some point, if we can get the v-net done, then maybe we can start splitting up um this work. I don't think it has to be done. Serially like it's being done today, but let's get through v-net first and we can see how we could split up some of this work.

D

Okay, thanks everybody, and- and I think I mentioned to the larger group that um we have a microsoft- has our org. Not all of microsoft has a quiet week next week, so I sent the cancellation for the community meeting and um we we have to focus on planning, and um you know gives us time to to think about different things and so we'll- and I know john you're you're out the 23rd right. Okay, all right thanks! Everybody appreciate your time. Thank you.

J

Thank you. Thank you.

D

A

Thank you very much guys awesome thanks.

D

Cheryl bye-bye thanks marion everyone.

C

Thanks bye, bye.

D