Internet Engineering Task Force 106, 18 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: IETF106-ICCRG-20191118-1000

Description

ICCRG meeting session at IETF106
2019/11/18 1000

https://datatracker.ietf.org/meeting/106/proceedings/

A

B

Good morning, good people can you hear me: I can hear myself just not a good thing anyways good morning and welcome to the IETF and welcome to IC crg. If you are here, if you don't believe you're here for IC crg stay on anyways, this might be more interesting than anything else. You are going to go to.

B

Before we start, I need the standard things, I need, jabber, scribe and I need a minute taker. Volunteers.

B

Come on, I can start naming names.

B

Gouri you're, looking at me like you, want to do some Janna I'll.

C

Take the minutes, but I don't have a jarek on what.

B

Do you say Andrew I.

C

See it I'll take the minutes, but I don't have a jabber client. So someone needs to JavaScript Thank.

B

You Andrews gonna do minutes jabber. Thank you, sir. So we've got a jabber scribe and we've got minute minute taker. So we can move on.

B

So we have a pretty packed agenda. Let me see how do I do this, but before we start we will just briefly I will flash you the note. Well, very, very briefly: we follow the IETF IPR disclosure rules and if you do not know what the note well is, you should go read it before. You come up to the mic and say anything at all: there's also privacy and code of conduct, which again I, will not go into right now, but you're welcome to read it.

B

Their slides are available online and the code of conduct is available online at that URL and the goal of the IRT F. Just to be very clear. We do research, we don't do standards development. So if you are here to push a standard, Strack RFC or on the wrong room, but if you are here to hear about research ideas and not just sound, let's work then you're on the right rope and there's our agenda for today, so we're gonna.

B

Basically, we have a fairly packed agenda and I'm not going to walk through the entire agenda, but we're gonna try to get done on time. Here so to the speakers, please come up just as your talk is about thought and please make sure to stick to the allotted time, and that includes me so I'm going to go through this fairly quickly.

B

The first thing to note here is: we have to site meetings. We couldn't really bring them into this meeting, but they are very relevant and you may be interested in them. So do go to them. If you think you are interested the first one is on data center condition, control and Paul Congdon is organizing this Paul's there raising his hand, and please ask him, speak with him. If you have any questions, otherwise, the those details are sill, correct, Paul, okay, lovely! Thank you.

B

So that's tomorrow morning in room VIP, a early in the morning, the side meeting time slot and then you have l4s on Thursday in this very room at the same time, at 8:30 in the morning, Bob are those details still correct, so this will basically be an update on l4 s. Yes, o l4 s and okay, TCP Prague. It's about TCP Prague, not a bottle for s, not just sell for us yeah, okay, but.

D

Itself for SNP CP, Prague euro, the details are that the l4s is mostly in the th vwg meeting on 3rd on Thursday. This side meeting is mostly TCP Prague, fair.

B

Enough, thank you for that clarification, David and that's happening Thursday in the morning. So again these are very relevant to ICC RG. So we just don't have enough time in here to do do all of these conversations. So these side meetings are you're encouraged to go to them. I will be there and with that I'm gonna start off on just a one topic. I'm gonna spend at most five minutes on this.

B

We had started a conversation last time. How.

E

B

Beautiful fit the page, maybe okay, that works, we had I, had broached the topic of doing documents in IC crg publishing documents through ICC RG, adopting them in ICC.

B

Rg I've had a number of conversations with various folks since then and I'm basically here to say now that we will start adopting documents in this research group, I intend to use discretion and the RG is guidance in proposing what documents makes sense, but that's the the goal is to adopt and publish documents here, we've done this way back in the past, I'm gonna start doing it again. What does it mean to adopt documents?

B

Well, broadly, the document is the reasonable fit for what we do in this room and by what I, what I mean by what we do in this room is not just if it fits the letter of the Charter, but if there's energy in the room- and if this is the right sort of audience to do this work or look at these documents, typically any condition. Control other fits in internet-like environments is an absolutely wonderful fit, for example, but we need to see author engagement.

B

We need to see author engagement with the research group and to show that there's interested in continuing that engagement to be adopted in this research group and again, I intend to use discussion and research group guidance in determining relevance here.

B

What does it mean to publish a document? It just means that the research of things the document is in good enough shape. Yes, this doesn't seem like a very firm process, but that's by design. This was not intended to be a heavy process. This is not the ietf again, I expect to call upon the RG folks in the RG, for reviews and for other decisions, and we will figure this out as we go. There's I have faith in the community here that we can make this work.

B

I feel great saying this by the way. This is a good medium to say this, because I feel like I'm I'm handing off this I'm doing this thing from upon high looking down at all of you. I will decide, but this is this. Is the plan going forward is to publish documents in ICC, RG and starting off I'm gonna propose adopting the following documents. We will hear about them today. The first one is led back plus plus the second one. Is our LED bat?

B

If you haven't read these documents, please do I think they are fairly good and I think laid-back. + + s is actually I'd say it's. It's almost an update to let that and it's it's a very important update to like that itself. So I'm going to propose this I will also send this out on the mailing list and we'll adopt it when, unless I hear somebody yelling and saying this is a terrible idea, that's the plan going forward with that. Well, any any quick thoughts. I give you 30 seconds.

B

And your 30 seconds are up.

A

Thank You Corey sorry go on Corey, first I'm cold account real, quick III think this is a good idea. I like this, and these two these two documents fit well because they are less. They are more conservative in other transports. Therefore they definitely fall within something that could be done in a research group. Other things we don't have to tread carefully on and I guess we'll figure it out as we go on, but these two documents seem like good candidates for here to me.

A

Yeah I think we will have to figure this out as we go so.

F

Question is the Martin Duke f5 is the desired outcome of these documents to produce informational, RFC s or something else.

B

I, yes, I would think so I don't know. Can we publish experimental a lot of C's here anybody? No, we can't do that here. Right who says we can Andrew says.

G

B

G

On Perkins, yes,.

B

G

Can publish experimental or informational, RFC Thank.

B

You Colin: that's the voice of authority there, so yeah I would I'd be interested in experimental, but that's what I'm gonna be pushing folks for anyways.

B

Thank you for that with that I am going to move on with the agenda again. This is the agenda. Pravin Europe, first you're gonna talk about that bed plus plus and I'm gonna give the slides up.

H

Morning, everyone can, you hear me.

H

Morning, everyone I'm here to talk about led by plus-plus. This is now a draft. If you haven't read the draft, there's version zero, one that was published recently, so please read it. Led bed plus plus, is a less than best effort, congestion control. So the goal is to use unused capacity bus but quickly yield to for ground flows. Example, workloads include operating system application after it updates or backup replicas sort of burglars next slide. Please.

H

Slide is a summary of led by plasmus the original RFC 68 17 has a bunch of problems which were found in prior research, as well as in our efforts to implement the RFC in the Windows operating system.

H

I'll quickly walk through the problems and the mechanisms that had bad plus plus proposes to fix those problems. One way delay measurements are hard in practice. For example, in TCP the clock frequency is not known of the pier as well as there is a clock drift problem, let my plus plus users, RTT, there's a downside which means on the downlink. You may measure congestion and react to it, but because this is less than best effort in practice, this doesn't cause problems.

H

There's a late comer advantage problem. If there are multiple flows already in the bottleneck link, a late comer or let bad flow would end up getting more than its fair share, led by plus plus introduces two mechanisms to address this problem. One is the multiplicative decrease instead of additive degrees, giving congestion avoidance as well as initial and periodic slowdowns.

H

The original RFC said that natural gaps in application traffic would allow led by to measure the base delay, but that doesn't happen in practice, so led by plus plus immediately after slow start would enter a slowdown period and then would enter subsequent slow long periods with a goal to sacrifice throughput of at most to be at least 90% of standard tcp when there is no competing traffic.

H

There's a internet but fairness problem as well. This is related to the late comer advantage problem and you can measure the base delay, but although the queue is stable, the flow is bound fair sure the led by plus plus introduces multiplicative decrease to solve this problem. The original RFC does not specify slow start in very great detail. There's two things that led by plus plus does it does slower than we know ramp up during slow start, and it also does a variant of high start to exit slow start when it senses.

I

Delay increases.

H

On the bottleneck link, there's also a latency drift problem. When you, let connections run for a long time, there's a ratchet ratcheting effect where the measurement of the basically key keeps increasing, and that causes us to keep increasing latency from the link. Initial and periodic slowdowns help address this problem, there's also a low latency computation problem.

H

If you have a bottleneck link where you can't reach target delay, for example, if the buffer is too small, that causes LED bad to basically just compete with standard TCP and the bottleneck link so slower than Reno increase during condition avoidance addresses this problem.

H

There was on the mailing list, there was a problem that was brought up. I think need card will brought this problem up there. What happens when there are multiple flows or large number of flows on the bottleneck link? Do we see a ratcheting effect problem or not? So we actually did this experiment recently. So this is a hundred, millisecond or titi link. We actually changed the target in later hundred millisecond for this experiment, but yeah. The results are similar with the default of 60 milliseconds, and this is a 10x 10x, bdp bottleneck buffer.

H

So, even with the increase in number of connections, we see that the latency remains near target at or near target. So when we look at traces, there's two reasons: one is when the new flow starts, even though the slow start is slower than Reno, it still causes a deal increase and that causes other flows to back off and then the newcomer flow would immediately enter it. Slow down phase allowing all the flows to measure phase doing we find that all the flows are able to successfully measured by anyway.

H

There is another experiment: this is with both 1x + 10, X, B DPS, again with large number of flows. As we increase the number of flows, we find that things mostly stabilize around target, so this is 100 millisecond or TTN, and the D for 60 millisecond target.

H

So we would like to experiment more. There was a suggestion that we use B, be our probe our TT phase and see how that doesn't comparison. That would be a very interesting experiment to do in the future, but currently in practice we don't see a problem.

H

What are the changes since the first draft, so I read and rearranged some sections to make readability easier and thanks to Marshalls review, so I fixed all the feedback that he gave to make the text better. The multiplicative decrease cap was not specified very well, so the intention is to basically ensure that we never react more than what Reno would do in if it sees a last signal, so the delay increased signal, the worst-case decrease, even with multiplicative decrease, would be half the condition window removed the reduction factor just to simplify.

H

So if the reader has read that bad, we just use the gain parameter now.

H

One of the requests has been to make the document stand alone. Currently it's a addendum on top of the original RFC, so the goal is to make the document stand alone so that any implementer could just read this RFC and be able to implement the algorithm. We also want to add pseudocode so that it makes it easy for implementers. So I request to folks in the room would be please review. This draft I think like less than best-effort congestion control is extremely important, so please review this. We provide feedback.

H

If you could implement this and do some experiments on your own. That would be great any questions.

J

Jake on I was curious, whether you had a chance to examine the impact of a queuing of FQ systems on on this strategy, we.

H

Have not so I call that out in one of the prior presentations. It's also an active area of research, I think like both testing with a QM as well as newer congestion controls like VB. Are they currently are testing is limited to cubic, and you know so. Yeah testing with other delay ways. Congestion, control, algorithms, which are aggressive, would also be I. Think part of that future work. Thank you.

B

Other questions we have a couple of minutes.

B

All right all right, thank you. Thank you. A presentation.

B

K

Morning, hey I'm, Marcelo Angelo, going to talk about the receiver driven led, but so I represented this in the last IDF. So basically, this is an update of the changes that I'd done in the draft. So if you remember, LED body is a set of mekinese that allow to run a less than best-effort congestion control algorithm at the receiver in order to throttle the control centers rate right. So in the original document, I mean the in the version zero document that I represented last time.

K

The document contained both the mechanism to implement this in the receiver and the congestion and congestion control. Algorithm controller is now move to to the level plus plus a document that Praveen just presented. So in this document we only kept the set of mechanics that allow the receiver to exert control over the sender's rate right.

K

So, in particular, in this document we kept how to use the receive window, how to manage it safely in order to avoid window shrinking handling the window scaled option and this type of things in order to control the sender's rate right. In addition, we work out the different dock mechanics in order to feed the information that the congestion control economy needs, including the Artic measurements in order to estimate queuing lay and the detection of retransmissions at the receiver.

K

So basically the the input that that the congestion control, algorithm, form of RTD measurements, delay measurement and some form of loss, transmission detection.

K

So the changes since the zero zero ssin is, we remove all the kind of the part of the congestion control algorithm we defined, so that elaborate is compatible with any less than best F for congestion control, as long as it uses some form of delay, estimation and all packet losses in order to work, and we currently are referring to labelled, plus plus or any other a congestion control that uses this type of signaling.

K

We could point to LED, but if we consider that using LED but with RTT instead of one-way delay, make sense assume I mean consuming. That leopard plus plus, is essentially an upgrade on on lead. But we believe that this may not make a lot of sense, but we could do it if people feel strongly about LED. But, and regarding our LED, what implementation we're currently working I mean we haven't an implementation.

K

We presented some results in the in the last IDF we're now working in order to make it perfectly aligned, with the level plus plus latest version that Praveen just presented. This is this is ongoing work. We hope to be able to present a results in the next meeting, and we also became aware that a apple have a receiver based less than best effort implementation that is available in this.

K

In the it's open source and available here, it seems somehow aligned in the sense that the use RTT use timestamps and receive window basically the tools that we are, that we are describing the draft, but it would be interesting to actually have some input from from from apple to see if they actually are aligned to to what we present here and, if not to be able to align properly the less than best effort control congestion, control, agon that they use.

K

Clearly, it seems different, then LED, but misplaced, but as as I said, are LED, but is compatible with different congestion control. I'll, go and examine.

K

Xed steps will be will be very useful to have reviews comments from from the from the research group and feel free to contact me. If you have further questions or comments and with that I available for questions or I can leave.

B

Questions folks.

B

Come on it's Monday morning.

B

Do that the mic is all I'm saying.

C

Some engagement, people.

B

Well, I actually do have one question for you myself: have you considered expanding this to the mechanisms would just be slightly different, but I think the concepts still hold for quick. Yes,.

K

B

You plan to write that in the draft or do you think in this particular draft yeah.

K

Yeah I'm asking ah I was thinking to I mean once I'm done with this. Do it in a different document, but if you love define, do I mean I mean. If you want me to do this, talk mean I'm happy to it. It.

B

Seems reasonable to do it if you're gonna cover both TCP and quick in the same, if it's possible to write I, don't think the differences should be that much okay, but.

K

B

If it becomes big enough, it might be worth separating it out. Okay, yeah sure.

A

Thank you, gory Fair has just asking about what you actually said at the mic. Were you asking about strange Eckles? Were you asking about whether this document could describe a method that could be applied to quake, or were you saying the document would include a mechanism described how it applied to quick I see the two is quite different. That's.

B

Fair I was asking how you would apply it, so the mechanism itself is not an underwire change. It's entirely a receiver side, it's its management of the receiving door in the world of quick. It would be how you advertise flow control credit, and so it would be slightly different in the sense that you're not just talking about modulating the window, but you're talking about modulating the size of the credit that even advertised and there is no equivalent I mean you can actually shrink effectively.

B

So that gives you a degree of freedom that is available in quit, because what I mean by shrinking is you advertise lesser credit in the future right because you can actually shrink the window. So if that I don't need? That's that's what I was asking me this right. The way it applies to quick would be slightly.

A

Different, so if I understand that that means this is a mechanism and you're not going to describe the protocol but you're going to describe the mechanism, we saw what.

B

You were talking.

A

B

So so, maybe maybe then, what I'm asking for thank you for for for teasing this out Cory. What I'm asking for is: how would this apply to like a credit-based flow controller as against there in the window, waste one? Okay, that's the difference, I'm, looking at clear! Thank you.

H

Robi Praveen from Microsoft I do think that both of these RFC's are kind sort of transport agnostic. There are certain, like transport mechanisms were using to communicate information, as you point out, but yeah I think they could be written aware that they could be applied to both TCP.

H

B

That's true, but does the avoiding of the risk of the window shrinking right, which is very tcp specific. It's it's very ecosystem specific to now, that's not applicable to quick.

H

Possibly not but III don't think it's illegal to shrink it's what's not illegal.

B

It's it's not illegal to avoid it. Certainly but I'm saying it: you have the degree of freedom. If you want it.

H

E

Nokia versa, I'm wondering about the windows shrinking thing. Is that, like what I mean I understand what it is, but I, don't know what what would current TCP sender applications to? If you, if you put you know if you would submit the window, that is too small and you would actually shrink it. They might just ignore it right because they have to have a way of handling it on the sender side anyway, if that happens, what did send a reset or sorry buddy? No, so so, if you.

K

So as far as I can tell, there are some situations where the window shrinking is general, when you're using very large window scale, for instance, that the granularity of the window that you are able to announce will will result in window shrinking so I understand that implementations usually can handle window shrinking without losing packets. But that's that's what I think Praveen Tony so.

A

Gory Fairhurst not answering the question directly but saying that will because I'm going to talk in a side meeting on quick for satellite later in the week, and one of the things I'm going to talk about is the way tcp windows are managed in bsd, using receiver window advertisements and how you grow and shrink the receiver versions and how that relates to what quick does. So. This is an area where, yes, you can't change the receiver window and it doesn't will impact.

A

Maybe that causes you to send more than you're expecting it ends that we ditched or whatever. But this is a mechanism that does work in real implementations and and talk more to me, if you wanted to find out what I was saying.

B

Thanks Maya, okay, thank you.

B

So these are the two drafts that we are planning to adopt. If you have any thoughts, please send them to me or send them on the list. Like I said, I will send out an email. I enter the list after this meeting and I see news already there and are there.

L

Miles on you guys, can you guys hear me? Yes,.

B

We can hear you okay,.

L

B

Your slides are up gonio all.

L

Right great hi, my name is Neel Cardwell and I'm gonna talk today, but with a brief update, VBR version to work at google, we think of EVR to as a model based congestion control and today I'm, going to focus on some recent performance. Optimization work that we've done. This is joint work with my colleagues at Google you Chong and Sohail Puri Ranjan, you suck in Kevin the quick folks in Victor and bin a summer intern I had this year, Luke.

E

And Matt Mathis and magic.

L

E

Hearing substantial.

B

L

In my hand, are you guys, okay.

B

Yeah we're fine go, but that says the function of the room: okay,.

L

Great all right next slide, please yeah! So just a quick outline. I'm gonna talk for a while about the performance optimizations I mentioned, and then I'll give a quick rundown of the status of the VBR version to code and the deployment at Google and then wrap up next slide. Please so yeah performance optimizations next slide.

L

So what are we doing here? What's the goal here? Well, we basically want to ensure that bbr version 2 gets to a point where it's doing a good job as a general-purpose congestion control for both TCP and quick for the environments where those transport protocols are used today.

L

So that means LAN and when networks, data center networks, VM guests- and we want to get to a point where it's a good drop-in replacement for the predominant algorithms used today, so reno Kubik DC TCP, and we want to make sure that it's providing in performance improvements across all of those environments and has acceptable coexistence properties when it shares with common congestion, control, algorithms, like renown cubic and, of course, as a stepping stone to number one.

L

We are deploying bbr version 2 for all the TCP and quick traffic at Google, and we are working on that as as we speak, and as part of that, of course, that involves a fair amount of testing and that so we basically want to ensure that PBR is doing a good job on both the production. Where clothes we see at Google and a wide range of test matrices in synthetic or lab tests, and we want to make sure it's doing at least as well as we know and cubic and DC TCP next slide.

L

Please so I mentioned some tests of production workloads, a cool. So so, what's that so at at Google, we do a fair amount of testing on our production kernel changes. This includes TCP loss, recovery and congestion control, including VBR changes and there's a fairly rigorous suite of application benchmarks that we apply.

L

Obviously, web search is included, but also databases and storage on, and we look at a fair number of scenarios and not just sort of simple bulk transfer, dumbell tests, and not just looking at only traditional congestion, control, metrics, like loss rate or throughput fairness, queuing delays and so forth. We also look a fair amount at CPU usage and median and tail RPC latency, and why we do that. Well, you know it when you operate these things at scale in production environments.

L

Of course, the details matter a lot and in particular one of the important details that doesn't always get as much attention as it should is, is CPU usage and the sort of contributing factors, including data packet, send rates acknowledgement rates, offload birth sizes, interrupt rates, all the things that feed into CPU usage and in addition, there are a lot of sort of tricky and stressful scenarios that are important to test like when you have thousands of flows coming from a single sending host or sharing a small BDP in a data center like environment.

L

Those are also important to test and, as we've been running, bbr version 2 through these kinds of tests at Google, we've developed a number of performance improvements to tackle various issues that we've seen in these tests, and so I'll go through these in a little bit of detail. But just to give a quick summary, I could say that basically, the the first two improvements are in order to match the CPU usage for Reno or a cubic DC TCP and number one. We developed a fast path for PBR on number two.

L

We did some improvements in the TSO, auto sizing calculation and then the third improvement was to fix in a sort of interesting issue. We uncovered with the Linux TCP, receive code path, and for that we've developed a mechanism that generates faster acts and then finally, the fourth one is to enhance PBR version to you so that it gets better performance than the widely deployed algorithms like Reno and cubic and DC TCP, and the sort of interesting case where there are more flows, sharing a bottleneck than the bdp of the path expressed in packets.

L

All right next slide. So the first improvement that we worked on was a fast path for bbr. So why did we do this? Well much of the traffic in the real world. Is application limited so web transfers, RPC traffic, adaptive, bitrate video and, if you think about it, cubic and Reno and DC TCP when they're not receiving ecn or lost signals, they take a essentially a fast path when they have application, limited traffic.

L

The the first couple lines of most of them basically say if this, if we're, if the flow is not currently see when limited and then let's return and not do any of the other processing that we might normally do for the congestion control. The problem here is that bbr, when run on these very simple application, limited workloads, I ran into CPU usage issues and even through progressions, and some of these simple application, limited workloads, and why is this well thus far?

L

The BB our code base has tried to prioritize simplicity and basically ran the entire algorithm on every act. So that means updating the entire model of the path, updating the probing state machine and then adjusting all of the control parameters.

L

The pasting rate congestion window offload chunk size, and this was causing up to sort of 2 to 5%, CPU and throughput regressions on some of these synthetic tests in these workloads and to tackle this, we basically constructed a fast path for bbr, where the idea is is that we only run the portions of the algorithm that are strictly needed, based on the properties of the information being conveyed by a given acknowledgment, and this resolved those cpu and through progressions, without sacrificing throughput or latency. An excellent. Please.

L

So we also worked on some improvements in TSO, auto sizing. So what is TSO auto sizing so in high-performance transport stacks often they achieve a very big performance improvement by bundling together consecutive segments into a single unit when passing them down to lower layers like the IP layer and the the NIC driver and in particular the the core Linux TCP stack has for about seven years or so used a TSO, auto sizing algorithm. That adapts the offload chunk size, largely based on the pacing rate, and it goes something like this.

L

So the the core TCP stack, computes a pacing rate and whether that's used for pacing or not. It still uses that pacing rate for the offload sizing decision, so it computes the pacing rate, as some constant scale factor times the congestion window divided by the smooth round-trip time, and then it takes that pacing rate, and it calculates how much data it thinks would be paste out in one millisecond and then it applies a floor of two segments and a ceiling of 64 kilobytes. That's determined by the the offload mechanism itself and an interesting thing happens.

L

If you have a workload where the sender host is the bottleneck for a large number of flows and in that kind of scenario, because the sender host itself is the bottleneck, there tends to not be any easy on or lost signals that the sender's get and so with an algorithm like DC, TCP or a cube occur.

L

We, the Seawind, can be surprisingly high in these cases, which then generates a large pacing rate and then a large offload chunk size and that large offload chunk size allows very low, a very efficient CPU usage, and then the problem is if we, if we take a b b, our scenario where we have a large number of BB r senders and they try to use reuse. This auto sizing algorithm now BB r is computing. Its pacing rate, based on its estimate of the flows fair share of the bandwidth coming out of that sender host.

L

So if you've got thousands of flows, the fair share for a given flow is quite small, so PBR was choosing small offload chunks and using a lot of CPU and so to solve this issue. We developed a mechanism that adds a term into the auto sizing calculation that is a function of the minimum round-trip time so that, as the minimum round-trip time goes, the saw the size of the offload budget. That is added for the min ITT falls off rapidly, but for very small men are TT's.

L

There is a considerable budget, that's allocated for the the TSO offload size and you can see the exact formula there and you can read the the code for the the details on the constant next slide, please.

L

So. The the third issue that we tackled was- and it's sort of an interesting issue involving the specifics of the Linux TCP, receive code path and the delay dock logic and DC. Tcp and BP are v2. Both have a nice property where they emit an immediate act.

L

When the incoming stream of congestion experience gets change, but that mechanism doesn't always save you, because, if you're under heavy enough congestion, then there's continuous C, II marking and that mechanism doesn't kick in so it can happen is that you can actually get into trouble as a Linux TCP receiver, because the the details of the code for the traditional TCP every other Paquette mechanism are actually a little surprising and they're actually two conditions. The first one is what you might expect, which is to say, if more than one MSS has been received.

L

Since the last time we sent an AK, then that condition needs to be met. But it also has to be the case that the next received window that we're about to offer is at least as big as the previous one, and the problem you get into is if the receive window stops growing.

L

This causes check number two to fail, which causes the receiver to wait until the application actually reads. Some data out of the receive buffer to allow the receive window to advance and then trigger and acknowledgment- and this was actually this sort of surprising detail- was actually causing 2x higher p99 RPC latency x' was sustained congestion and some of the tests we were running and as as a solution.

L

What we've done at least thus far is to just remove truck number two, and this is taking care of those RPC latency issues, and we should say that if this works well in practice, as we deploy I think will propose this as a general fix for the Linux TCP receive code path. Next slide, please.

L

So this is just a quick picture to give a better sense of what's going on there because I know the the text doesn't make it entirely clear. So here we have a picture. That's a time sequence diagram of one of these scenarios- and you can see at the beginning of the connection the receive window is, is opening up.

L

You can see that as the received window, the the yellow line is growing further away from the act line, the green line, and during that phase the receiver is allowing itself to acknowledge immediately just giving a high throughput, which is reflected in the lot and the high slope of that line.

L

But then, as soon as the receive window stops growing the yellow line sort of plateaus, you can see that the acts are all of a sudden, very delayed because it, the stack, has suddenly waiting for the application to read some data before it sends an acknowledgement. So that's just an illustration of an example of this next slide, please so.

L

Finally, the final mechanism I'm going to talk about is, is some work to some work in progress that we have to reduce the queuing that we see when we have lots of flows. So next slide.

L

So what's the scenario we're talking about here. So this is a scenario: that's been documented and and talked about for a quite a long time and I think it goes at least as far back as Robert Morris's paper on TCP behavior, with many flows in 1997, and maybe there are earlier papers as well. But the basic issue is that when you've got mini flows and al-obeidi P, then most existing widely deployed congestion control.

L

Algorithms, like DC, TCP Kubik, you know and BB are thus far tend to run window limited and they're gonna have a standing queue when the number of flows times the minimum congestion window of those flows is bigger than the PDP and the problem that you then get is that there's a standing queue.

L

That's basically that budget of the aggregate in flight, the number of flows times minimum Sealand minus the bandwidth delay product and that can be fairly substantial and just as an aside here, the to give some sense of scale, the the B be our minimum. C wins aside from RTO recovery is for segments and that's to avoid stop-and-wait behavior with TCP s every other packet policy.

L

So the diagram here sort of gives you an intuition about. What's going on, you can sort of see that the Q of excess packets here is building up at the bottleneck due to the aggregate in-flight being larger than the BGP of the.

I

L

All of these flows running when they limited next slide, please.

L

So the solution that we've been investigating is to improve PBRs use of ecn and loss signals to make sure that we do a better job of adapting the pacing rate to match the available bandwidth and so there's sort of two core pieces of this. The first is a multiplicative decrease in the pacing rate and in-flight data.

L

When we see ecn marks as well as loss and that allows us to sort of quickly match the available bandwidth and then drain the queue and also helps in converting converging toward a fair allocation bandwidth and then when the cue seems to be low enough based on the EC on signals, we're getting. We then do a small additive increase and then continue in the BB. Our bandwidth Pro beam growth curve, and this is a paper. This is a sort of approach. That's been discussed before you know.

L

There's the the google timely paper from 2015 discusses this kind of approach and I also believe Bob Brisco has done some work in this area for TCP Prague, if I'm not mistaken, and the picture here just sort of gives you an intuition about the the dynamics that we'd like to see where all of the flows are matching their pacing rate to the available bandwidth of the ebon link. So you can try to reduce the queuing there at the bottleneck next slide, please.

L

So this slide has some details about the the mechanisms that we are experimenting with in the PBR version two code base and the at the top there's a sort of summary of the core mechanism, as ecn marks are arriving once per round trip time. Bbr v2 has thus far been reducing. It's in fly cap or in-flight low multiplicatively, based on an exponentially weighted moving average of the ECM mark.

E

L

Then, for this experimental variation, we are also capping the the pacing rate or bandwidth low, as as we call it in the code base and decreasing that likewise and the other sort of pieces of the puzzle here, as mentioned before, we're cutting the the bandwidth, low, multiplicatively and we're as as another piece of this, we're increasing the the ecn factor, which is the sort of magnitude or scaling factor for scaling the ecn response into the pacing and in flight response, and then we're also decoupling.

L

The bandwidth high parameter, which was already in the in the BB rv2 model from the maximum bandwidth sample that we've seen so that we're sort of refining the notion of this bandwidth high parameter, whereas before it was just the maximum bandwidth that we'd seen recently and now we're turning it into a more refined notion. That says it's tracking the maximum bandwidth. That seems consistent with the tolerable level of EC on and loss marks.

L

And if we're not seeing a tolerable level of those signals, then we do a multiplicative cut in the parameter and then, finally, when it seems like the level of queuing, is low enough based on the EC on mark rate, then we go ahead and do a very small additive increase and then continue on in the bandwidth probing state machine. So those are just to give you a sense of the details and next slide.

L

Please so I'll present some synthetic lab test results to give a sense of the properties of the experimental algorithm here here we have two Center machines and one receiver, all on the same switch all with a 50 gigabit Nick and the switch here is configured to use, DC, TCP style ecn, marking where the congestion experienced bit is that if and only if, the instantaneous Q is greater than 80 kilobytes in this particular case and we've got 60.

L

Second bulk knepper transfers here and we've collected a number of typical metrics which I'll go through as they talk about the results and then, when we're comparing three different congestion control variants. Here one is the Linux DC TCP code. The very latest- and then one is the VBR version, 2 baseline, which is the algorithm without the changes I just described, and then BB are two new. Is the algorithm with the changes I just described next slide? Please all right! So here the results next slide.

L

So the most interesting metric here in these experiments is that if we look at the queueing pressure, as reflected in the retransmission rate, as we scale up the number of flows with notices, we're sort of Inc doubling the number of flows in each experiment, you can see with a small number of flows. All of the algorithms are doing quite good at keeping retransmission reloj and as more flows are added, there's sort of varying degrees of retransmission rates.

L

With the largest number of flows, you can see d c-- tcp because it's necessarily running window limited tends to have a fairly high retransmission rate around sixteen percent in this experiment. Experiment and the newer version of EVR is able to keep the queuing pressure lower and has a lower retransmission rate of around 1.6 percent and a next slide please.

L

Similarly, if we look at other metrics that reflect the queueing pressure, we can see that, on the newer version of BiBi, ours is doing a slightly better job at keeping the queues low. This is the average RT t next slide and here's the 95th percentile RT t it's sort of a similar pattern.

L

Next slide, please, and then the throughputs are relatively similar for the for the variance the older version of e b r2 is not quite as high as the others, but otherwise they're fairly similar next slide. Please, and then you can look at sort of the C e mark rate, is sort of an interesting way to look at how good the algorithms are doing at avoiding saturating the signal they're using and you can see the PVR. The newer version of PBR is doing reasonably well here next slide, please and then.

L

Finally, we also look at the fairness, the jain fairness index for lips, small number of flows that are all doing quite well with larger number of flows, they're all not quite ideal and there's definitely some room for improvement and we'll be looking at proving that. But it looks to me like their use: ibly, fair and all you know in the same ballpark, but there's some work that we'll be doing in that area next slide, please so wrapping up next slide.

L

So where are we in terms of the status of the code, so the performance improvements that I was just discussing? We pushed those to the Google will be our github repo. You can follow the link there for instructions on how to check out and play with the code, as we mentioned in July, and there's also a bbr aversion to implementation for quick and you can follow the URL there and we again we encourage folks to take the code for a spin. Try it out share any test results.

L

They have our packet traces share, any ideas about issues they run into or ways to improve the code, we're always looking for a feedback from the community, and then we have links to the slides and video from a previous IDF's on next slide. Please! So where are we in terms of the deployment status?

L

We've got PBR version two running for a small percentage of users on YouTube, where we see lower queuing delays than both PBR version, one and cubic and considerably reduced packet loss closer to cubic than VB R version one and then we're in the process of a large-scale test pilot program, pushing BB r v2 out as the default congestion control for TCP within and between Google Data Centers, and we're seeing some nice results there and we're continuing to iterate both in production and lab tests. Next slide, please so.

L

Finally, in summary: we're actively working on VB our version 2 at Google, and doing some tuning of performance, especially in data center contexts, high-speed data center networks, so that we can enable VB our version 2 globally at Google for internal traffic and we're working, as you saw on improving the algorithm to scale to larger numbers of flows, and as always, we invite the community to share any test results or issues or patches or ideas they have.

L

We also wanted to give a shout out to the freebsd tcp community and the team at netflix, which has been working on bbr and they actually released a an ability. Shinobi be our version one for FreeBSD a couple of weeks ago, and we wanted to finally leave you with some food for thought. It's sort of an interesting question of if you have to design a KPI or key performance indicator for congestion control. What would that look like?

L

Basically, the question is: how would you tell if your congestion control is doing well in a production environment, given that, in a production environment the traffic is dynamic, the routing is dynamic, topologies are dynamic, and so it's it's it's non-trivial. As far as we can tell to to develop a good key performance indicator, and we encourage folks to tackle that as a research problem. If they're interested, we think it's a valuable area for contributions.

L

So thank you very much any questions or comments. I.

F

I new mardukas, thanks for giving up your Sunday night I, have two questions. One is: is there a plan to update the BB r Draft at some point, we're trying to track with an implementation? I know obviously Netflix as well. We really nice them a fixed target. Yeah.

L

An update of the internet draft to reflect version two is absolutely on the to-do list and I apologize. We weren't able to get it done by this ITF, but rest assured it's definitely on the to-do list. Yeah, okay,.

F

Thanks the second, can you return to the sequence number graph you had.

F

Yeah, so um maybe I'm missing something here, so this strikes me is a for. Certainly for legacy. T speak gesture controls. This is a feature not a bug like you're, not getting pulling more data out of the system when the app is not able to consume it. Is this a problem in bbr just because the act, pacing messes up your bbr or your bandwidth estimation.

L

No I don't think so, the it it's it turned out to be a problem in practice in these RPC workloads and it's you know it showed up in the detail: Layton sees for one of their storage applications that we're benching benchmarking, so we could definitely go into detail offline. If would be helpful, but yeah.

F

L

Sure what else to add it at.

F

This point: okay, well I, mean I, guess III think you were proposing that maybe this should just be a blanket changed TCP to remove that second condition about the window increase and I I. Don't have a lot of experiment! Remember why exactly that was put in, but it strikes me as it as a fairly obvious thing for for the reason I just said, which is to not just pile more stuff on on a backlog, client on a backlog, receiver.

L

Right I mean I think there are probably a number of trade offs here and I guess it. You know at the end of the day, maybe it's something that is going to be context dependent, whether it's a an improvement or not, and so that's you know why we haven't offered it immediately as a fix for upstream. We want to get some experience with it in production, workloads and see. If it is, you know see if it is a net win.

L

We think that you have good reason to believe that it will be an at least acceptable, if not an improvement in every case, because we, the the previous congestion control that was deployed in Google, actually had a de facto behavior that in congested cases, actually did essentially bypass check number two. So we do have a fair amount of experience, at least in the congested case with it helping. But you know well, reserve final judgment until we've pushed this for all of our workloads and- and we don't see any regressions thanks, Neal.

B

Right there just to ask a follow-up question: do you know why that second condition was was instead in the kernel right now.

L

We don't know for sure I mean, as Martin said there, there is an argument that you can make about trying not to overload the receiver. Another argument that Eric dooms a pointed out was that this check works nicely with the underside fast path which has only taken if the receive window is, is constant, and so this can save some CPU on the sender side if they received window stays constant yeah, but where this this check predates they get history in Linux. So it's a little a little tough to talk. Wow.

M

B

Right well, thank you.

I

So I'm I'm very amused, of course, because my question was what was condition to expected to help solve Roberto, who are you hi, I'm, Roberto and I'm, with Facebook right now and III? You know, one comment is say hi to everybody.

I

The other comment, I love the data-driven approach and in cases where there is no clear answer, why I'm really hoping that you will be able to drive changes in behavior? This particular one with number two here seems exceptionally odd to me because it it it provides a cliff of behavior and we know cliffs and in control theory tends to be bad for being able to understand the systemic properties, so I think it's super awesome to try and either eliminate to or prove that, there's a good reason for it to exist.

I

Given that nobody seems to know why so anyway, thank you. Thank.

L

H

A couple of questions, so the minimum congestion window for so TCP traditionally had that value one matter standing is quick, is using two because of the problem that you mentioned in the slide. I'm just curious. Why? You would pick four instead of two.

L

So the details have to do with the dynamics that you get into given the delayed act behavior. So the basic idea is that you want to have four packets in flight so that two of those packets- well, the simplest scenario to imagine- is basically that two of those packets are in flight in the data transmission direction and because you have two of them, that's going to force out quickly quickly force out a delayed, ACK or quickly force out in a community rather than delaying the ACK.

L

Rather so then, you've got also two a two packets worth of budget coming back in the reverse direction in the form of acknowledgments, so that those packets.

L

When the windows acts when they arrive at the sender, they can immediately release the next two data packets and you basically want to have all all four packets worth of budget in flight, so that there's always data going in the transmission direction and always acts flowing in the reverse direction and the only way you can sort of ensure that that happens is if, if the receiver is acknowledging every other packet is to have 4x or sorry 4. Packets worth of budget in-flight on and.

L

Yeah, so obviously there's that does increase the aggregate in-flight in these sorts of scenarios, but the I think architectural II. My sense would be that a better solution is ensuring that the flows tends to run pacing limited because that's a more complete solution and you're always going to have there's no matter what your minimum sea wind is. There's always going to be. If you're running window, there's always going to be some number of flows and some B DPS at which you get into trouble.

L

So it seems, like a more general fix, is to to run pacing limited with a C, a minimum C wind. That's robust, ok,.

B

I'm gonna ask folks to be super quick because we are well over time at this point and I'm closing the lines one.

H

More question was the target loss rate. So currently, how do you determine that for various workloads to do pickup, constant value, or do you tune it based on some parameters.

L

Right so far we would like well so far we are using a constant value in our deployment, but we are leaving the door open to making that something that could be conceivably tuned by deployments. For example, if you have a background file transfer like a less than best-effort transfer, you could. That is isolated to its own cue in your quality of service implementation.

L

In your switches, you might want to tolerate a higher loss rate, for example, but yeah the we, the final loss threshold that we'd like to target for the public Internet sort of yet to be determined and we're open to ideas from the community about that yeah. Thank.

M

You I came up it when you put up the thing about key performance indicators, which was quite a time ago now, just that just to say to people go and have a look at there's a hot Nets paper in on its this year, yeah called beyond Jane's fairness index, which is quite interesting sort of thoughts on how to do that being hot Nets. Obviously, it's not completed work, but it's more about measuring the harm one flow causes others rather than equality and they'll, be on that conversation about packet windows and things like that.

M

I'm gonna, just forward reference to the slot about TCP Prague Assad's finished his and defended his thesis in September on that work and the code will be available. Hopefully before Thursday and well it I mean it's it's completed, but just chickened out on everything and it I'll give a link to his thesis and things that we we did manage to get it working with a window based algorithm inside the base. Tcp stack in Linux quite quite nicely, but there are other ways you can think about it.

M

So I'd like to hear when we talk about it on Thursday more about it, okay,.

B

Tests on that particular paper, but you're gonna have that presentation in Vancouver I think the Hartman speaker.

N

Initiative, I have very simple question, so you mentioned about the documenting baby I'll be too but I might be wrong, but my impression is birria bytes consists of lots of small hello, complicated Rosic and then some sounds like our implementation tricks and then so I'm wondering how do you are you going to document spec over the b-b-b, our beta I.

L

Yes, we, as we mentioned with when Martin Duke, came to the mic. We definitely intend to document the algorithm in an internet draft yeah.

N

A

N

Strike a hospice think something like Fast Pass seems to be an implementation trick. From my sure.

L

Right, so that is not something that would necessarily need to be in the draft because, as you say, that's an implementation, detail I. Think the much as the you know. The traditional TCP fast path is also not in the the RFC 568 Lauren's back I. Don't think that the VR fast path would be in the internet draft, but I do think it's you know. We definitely intend to to document the the high level algorithm in the internet. Draft yeah.

N

B

The implementation Yoshi we we have to move on I'm, sorry, Jake, we've closed I, close the lines earlier. Okay, last question: if you have.

O

You are still in the line before it closed it. If you have you have ten seconds: okay, yeah, John's, Norton I'm, just wanting to ask whether you cope well with our cc-16 eight style ecn, for example, if a TC and is not negotiated.

L

That's far, we're not planning on integrating bbr with RFC three 168 style ecn. We expect that thus far, people using VBR would not negotiate three 168 ecn. That's that's the current plan.

B

Thank you so much I am going to thank mean for staying up late and doing this, and I am hoping that we'll see the draft sooner than later. But we should talk offline about that, especially now that we're talking about adoption of documents in this in his research group and yeah. Thank you knew thank you.

B

I'm, going to now handoff to Nathan Gert from Facebook who's going to talk about their experiences with Coppa.

P

Hello, everyone can, you guys hear me, should I move closer or further away? Close is better okay, so yeah little bit about myself, my name is Nathan GUG. As you can see it's. My first ITF here and I'm super excited to present some of the work we did at Facebook, comparing Coppa with cubic and VBR for live video, upload, use-case, I work in the videos infrastructure team at Facebook.

P

This is the rough outline of my presentation. I'll start with the motivation for our work. Then I'll go over the experiment, setup we used and the results we saw on the application side. Then I'll do a brief overview of Coppa how that works. Although that is not the focus of this presentation, but I think it's important to go over our understanding of Coppa and how we applied it and then Indian I'll go over the conclusion and the future work that we apply so motivation in videos, infrastructure team.

P

We work with a lot of product teams who work on videos and give realized the different type of video experiences require different quality and latency trade-off. So, on one end of the spectrum, you have applications such as video calling where the end-to-end glass to glass latency is super important, and these applications are usually willing to tolerate some quality loss in order to get that desired. Latency.

P

On the other end of the spectrum, you have applications which don't care as much about latency, but they require extremely high quality playback, and then there are applications which falling between which are kind of like yeah. We need low latency, but we also need high quality playback. So normally you would need different type of condition: control, algorithms compatible with different trade of scenarios.

P

What we wanted to find out was: is it possible for a single congestion control algorithm to offer a dial to the application and the application could use that dial to just make it compatible with the desired quality versus latency trade-off? This is where we evaluated Coppa, which is a delay based condition, control algorithm. It has a dial which is in the form of parameter called Delta, which you can use to control the delay, sensitivity of the algorithm.

P

So in our experiment setup we wanted to start with something, so we started with the big one extreme, where we are optimizing for throughput at the expense of delay, so we tune Copa to optimize for throughput and we did some comparisons with bbr and cubic, which were which are two popular algorithms available and a for testing. We use the Facebook live streaming application.

P

So in this application, any user with their mobile phone, they can go, live they can broadcast their live feed to their friends and followers- and one thing to note here is that, like this is a very, this is very different scenario as compared to HTTP style, traffic, HTTP traffic would be short and burst years. Well could be app limited a lot of times as well. In this case, we have long-running flows over a single, quick connection, no multiplexing with the mean duration of around 3 minutes.

P

So you always have a constant flow of data like which you are pushing with the wire other a different thing about this application is there is an adaptive bit rate algorithm running in the application, which would change the encoded bit rate in response to the network conditions such an example.

P

If it sees that the network queues are building up and the throughput or the goodput it's seeing is lower, then it could change the encoder to produce less bytes and hence that which that will result in less bytes being written on the wire, and we implemented a coupon in the Facebook, quick library like quick, because the transport is in user space. It allowed us to implement Coppa quickly and experiment with it quickly and we use the bbr in cubic implementations which were already implemented and deployed for several other use cases with quick.

P

So for conducting the experiment, we used facebook's, eb testing framework, so this framework allowed us to conduct this experiment all over the world. This framework helped us divide users into three random groups. A one random group got a copa. The other group got a b BR and the third group got a cubic. We conducted this experiment for three two weeks and we collected roughly four million samples in each group so before diving into the results. I want to spend a little bit of time explaining the metrics and how how we define them.

P

So we mainly focused on two metrics on the application side to evaluate the condition general algorithm. The one metric is average good word. So the way we define it as, as you guys might already know, number of application by its sixth sense successfully over time so because they do not cast could be long-running. We calculated the total number of audio and video bytes that were sent during the entire cast, and then we divided that by the duration of the broadcast in seconds that gives us average code word.

P

The second metric we used is average application observed oddity. This is our proxy for video ingest latency to explain how this is calculated. The there's a diagram here to calculate this. The live streaming application inserts a pink frame with basically no payload, just a bunch of headers in between the audio and video data.

P

Now a dark pink frame travels with the rest of the audio and video data through the transport send buffer through the kernel through the same network through the proxies and it reaches the Facebook, live server which is processing the audio and video data. Then, when the server receives the ping frame, it immediately responds back with an acknowledgement frame for that ping frame, which goes all the way back to the live streaming application. Now the application uses a timing of sending the frame ping frame and getting the ACK.

P

It calculates an RTT which is its picture of what is the round-trip time for the application. So the application does this measurement every second and it gets roughly one sample every second and to calculate average application observed RTD. We take the average of all of these samples.

P

Now, let's look at the results.

B

Clarification question method, so these are basically measured on the upload. This is site yeah.

P

So, looking at the results for goodput, we saw that both coppa and VB are provided better goodput as compared to cubic. However, we saw that coppa provided much better good word as compared to BB re, not to be specific of the p50 good port was increased by 5% for bbr, and it increased by 16% for Coppa and similar trend followed.

P

Even if you look at the look, the verse connections which have a lower quality and as you can see like at p90, it maxes out, because our encoded bit rate is capped at 3 Mbps, because that's pretty much like a reasonable max quality that we want to have from from our our users and this. This improvement was massive for us and we also saw positive impact on some of the top line. Video metrics that we observe, which tell us how much people are watching those videos and how much people are engaging with those videos.

P

The second metric we used is the video ingest, latency or application, observe data team in this a metric. What we saw was a bbr was able to improve it slightly more for the best connections. So, as you can see in the left graph bbr reduced it, the bbr a bar length is smaller for p50 BBI reduced it by 8%, Coppa reduced it by 4%, but but those connections already had like quite low RTD and the reduction was not as visible for our users.

P

If you look at the verse connections, P 75, P, 90 and above are Coppa reductions were highest for P 90 Copa reduced the application ITT by 27 percent, whereas BB our application RTT was same as cubic in our tests.

P

um So this told us so in the previous slide you saw that Coppa had better good put, so that meant that Coppa was able to help like it helped transport to send more bytes successfully over the wire, and it also impacted the application ABR to produce more bytes in response to a better, better transport, but it was able to do it at the same time as keeping the latency slow, especially for the users like that had high latencies and where it mattered. The most so to be honest, like this result, was quite surprising for us.

P

We did not expect both goodput and latency to be better, so we spent some time trying to under and more like why that might be happening so in in the process. We found some interesting observations on the transport side, but before going into those I want to spend some time explaining how Cooper works I'm sure a lot of you are already already now aware, but I want to explain like how we applied it and how we understood it. So it's a tunable delay based congestion control, algorithm.

P

The tunability kappahd comes from the delta parameter, which varies from 0 to 1 the closer it is to 0 the more the algorithm will optimize for throughput the closer it is to 1 the more it will optimize for delay and it's a delay based algorithm, which means it uses RTT variation and signal of congestion and not loss or something else. In order to do that, it maintains two key variables: one is RT t minimum, which is the minimum RTD.

P

The flow has observed over a period of 10 seconds, it's its estimate of the two-way propagation delay, which is a property of the network path. The second metric, the second variable it maintains, is RTD standing. This is a minimum RT t observed over a much smaller period, s RT T by 2. To be precise. This is a copepod estimate of what is the current round-trip time, including any queuing delays in deep water like links, so the reason it does it chooses. Rt t by 2 is to make sure like the measurement is correct.

P

If there are some jitters- or there are this compression on the on the transport which might mess up which might make the algorithm think that there is queueing even when there is not, then it uses these two to calculate a queuing delay which is RTD standing, Artie, T minimum so on every ACK, the algorithm, the main controller calculates the queuing delay and then it calculates a target rate. Using this formula, the formula is 1 divided by delta x, queuing delay. I am not going to go in details of this.

P

There is a long mathematical justification given in the paper. They show that, like you, can critically reach Nash equilibrium, if you have certain modeling assumptions on the packet arrival pattern, so please do read more about it. If you are interested and then it compares the target rate with the current rate and accordingly, it adjusts the condition window to what said by using an additive increase. Additive decrease a variant, so it moves it by a V divided by Delta times, condition window so condition.

P

Window factor in the denominator helps ensure that the change in one RT t is going to be at max V divided by Delta packets. V here is a velocity parameter. It is 1 by default, and this a parameter comes in handy when Co pi detects that it has been trying to converge towards a target rate for far too long, and it needs to do a better job at converging at it faster.

P

In that case, it starts doubling the velocity parameter, I think every RTT, the other unique thing or different thing about this algorithm, is that it has a competitive mode. Normally deliveries, condition, control, algorithms.

P

We lose too will lose in the presence of buffer filling flows in the owner in the battle in the bottleneck link and because, if the buffers are filling up for no fault of the deliberate sender, then the deliberate sender will think that this congestion or it's the delays, are building up and it's going to back off before the lost beast flows are going to back off. Hence it loses some of the throughput coppa gets around this problem by using something called a competitive mode.

P

So in competitive mode it uses a heuristic which I will explain a little bit more in the next slide to detect the presence of buffer filling flows, and if it detects a buffer filling flow, then it adjust the Delta to be more aggressive, basically optimize more for throughput and perform better in the as compared to fulfilling flows. However, in our experimentation we did not implement competitive mode. We wanted our experiment to be like smaller in scope, so we just tested like what will happen if you use aggressive value of Delta without using competitive mode.

P

So this is the steady state dynamics of cope algorithm. This is what the bottleneck you looks like in steady state, so the bottleneck you goes oscillates between having zero packets up to 2.5 times, Delta inverts packets and this the entire pattern repeats itself every 5 RT.

P

So what I was saying before was that the competitive mode and computing flow detection, so Coppa actually exploits this property, that the queue is going to empty every five RTD, and if it's not doing that, then it it thinks that there is a buffer filling flow which is preventing this from happening. So hence it concludes that there is a buffer filling flowing in sharing the bottleneck link.

P

So here like, if you see let's say at T, equal to zero, the queue starts to increase for the first time, so it will take about half our TT for the RTD standing measurement to be available and the target rate to reflect this change and the target rate will be lower than the current rate at that time. Because of the inherent RT t delay in the network, it will take one oddity for that change to be effective.

P

Basically, the change of reducing the target rate and then in another oddity, the queues will start to drain again and then the entire cycle repeats itself, like half our TT for the architects timing measurement to reflect the change and then another oddity for the congestion window increase or the target rate increase to result in the queue length to start increasing in our experiment, V, and then the results that I'm presenting they used a delta value of point, zero, four, which means roughly twenty-five packets in the equilibrium.

P

Twenty five packets equilibrium queue length and the maximum queue length is like two point: five times: twenty five. Now, let's look at some of the transport level stuff we observed, so one metric we looked at is how the how is the transport Arcadia varying? So we saw so. This is measured as just taking the average of all the quake RTD measurements over the duration of the broadcast, so the trend we saw was a very similar kind of similar to application.

P

Architecture meant bbr for the best connections, which already had pretty low RTT bbr was able to reduce it slightly more as compared to a copan cubic. However, if you look at the tale cases, 75 1995 Copa reductions were very higher as compared to VB are so for a p90, for example, Copa reduced the RTD by 38 percent, whereas bbr reduced it by eight point, eight percent.

P

So this was pretty good to see, and this told us little bit like where the application IDT reductions might be coming from, and it also showed us that, like it, Koopa flow is not just improving the application latency, but it's also improving the RTT for the network. So if there are any other flows which are also sharing the bottleneck length like they will also start seeing a lower equities, the other metric we looked at is the retransmission overhead.

P

So this we defined it as the total number of bytes retransmitted by the transport during the course of the broadcast, divided by total number of bytes acknowledged during the course of the broadcast. I think it's a pretty important metric to look at because it tells you how efficient the transport is, if you are wasting resources and bandwidth rewriting the same bytes over the wire again and again, it's just not good for anyone. So what we found here was that for 90% of users, coppa retransmission overhead was available as compared to both be brn cubic.

P

It was about half of what bbr had and around 1/4 of what sorry cubic hard. So this gave us an idea like why application goodput might have been better like this might be one of the reasons why we were able to send mode by it successfully for the application.

P

However, for the last 10% of users, we saw a different trend that coppa retransmission overhead grew very rapidly, and it became three to four times as compared to cope and cubic and VBR, which was concerning as well as surprising, because we did not see a corresponding proportional degradation in the application metrics for any other tail users.

P

So we spend some time debugging into it, and the first thing we did was we sample a few cases, some some broadcasts which had a very high loss rate for coppa. So we notice two things. One thing was that these flows had very constant or throughput, as you can see by the red line here, the number of bytes act over time. It's it's a it's a straight line, and the second thing you observe is that these flows actually have very low arteries.

P

So in this sample, except the one spike, the RTD stays constant around 75 milliseconds. So this did not look like congestion or loss is happening because of buffers filling up. This look more like network policing to us, because if there's a token bucket, a police are in effect which is identifying your traffic and and limiting it as a bit at a bit rate. It's not going to result in any increase in queuing delays or equities, but it will look more like this.

P

We also grouped our retransmission overhead numbers by ESN, and we found that they do vary greatly, depending on which s and you're looking at some essence, which are known to police Facebook traffic. They had very higher retransmission overhead for tail cases for Koopa, and there was some other a essence where the retransmission overheads look very similar. So to generalize our finding. We did some more aggregated analysis. We looked at the relationship between artery transmission, overhead and our TT queuing delays, and we found that for cubic. They are highly correlated as the retransmission overhead increases.

P

Our TT and queuing delays also increase for cubic, which indicates that all the losses are happening because of congestion instead for but for Coppa, we saw similar trend for the first part of the graph, but later on, we see that the retransmission as the retransmission overhead increases the arc titties and queuing delays actually start to come down. So this kind of solidified, our hypothesis, that this net, the network policing, is a big factor being playing a role here in the high retransmissions.

P

Although it's also possible that there are some other reasons at play here and network policing is not the only one, for example, short buffers, if there's a short buffer and Coppa sender is trying to maintain an equilibrium queue length of 25 packets, then the buffer is smaller than that, then, like definitely going to run into losses.

P

So definitely there are improvements are possible in Coppa to handle this case better competitive mode, as I described before, could help because it could adjust the target Delta based on whether it sees loss or success.

P

We could also add a heuristic to change congestion window based on something simple, like multiplicative decrease based on target loss, just like cubic or Renault or even BB are now is doing. Other option is to have an explicit network policy detection, similar to how BB, R by V one had it. So. In conclusion, the aggregated results showed us that Coppa provided better and lower latencies in artists. These tests were for mobile broadcasts for uploads, and we compared with cubic and VBR with quake.

P

But one thing to note is VB are like via tuning VB, are in our internal implementation and also like Beiber veto is happening, so the results may differ in future. One of the big future work items is a better understanding of the reasons behind these improvements could are these improvements because of lower retransmission overhead, or is it because Kupa is just doing better target rate estimation by sending fewer packets and it's converging to that faster or it could be something else.

P

In the beginning of presentation, I spoke about our goal of having a tunable delay, based condition, enroll algorithm, which could be compatible with all the video experiences and so far we only ran experiments on one extreme where we are optimizing the throughput. We would also like to run some experiments with other, with goobie condition, control as a comparison where the end-to-end ultra low latency super important, and we would also like to test for more use cases like playback and all the traffic.

P

This is. These are a few links where you can read and learn more. My email is here and I'm happy to answer any questions now or offline.

B

Thank you so much. Nothing I'll intake myself in the front of the queue so I'm asking this question from the floor. Basically, the scenario that using this under right now is when the client is brought is uploading, video up to the server- and this is almost a textbook example of a situation where there is no cross traffic or there's no other traffic. That is going to be competing with Copa at a bottleneck, because I imagine that when people are uploading, video they're not doing anything else on the phone.

B

So it seems to make sense that the that, even without the competitive mode which I like to think of as TCP friendly mode, because that's what it is.

P

B

That's what cubicles it, but we don't get there. The the point is that that's, that's simply a it makes sense that it works without that, so I'd be interested in seeing more evaluation on on what happens when you, when you have competition with cubic with PBR with other things in the network, because that's going to be useful for downloading download streams and in the common case that's gonna, be an important factor.

P

Definitely, like that's like I, said: that's one of the future work items. Although then to be honest, when we started working on this like we did not really know that there is going to be no competing traffic and like even now, like I'm, not totally sure like if there is any evidence like a real life. Evidence of that I would love to read more about it. I know from textbook that it is the case but like yeah. This is one of the future work items.

E

Thank you, Michael Mike, reverse begin by saying that I greatly appreciate Facebook coming here and bringing this here and some some good results and I guess the mechanism is not altogether unreasonable. I have to say sorry for that. I have to say that the base idea or that this you know it's based upon of a delta, that that reflects a trade-off between having high delay and good throughput or low delay, and not so much support is fundamentally wrong.

E

Just because well I mean if you look at the Intuit the base intuition between mechanisms like bbr or alpha s, or things like that. You know they aim at having high throughput and having a low q and having a large q. Just isn't good for anybody. I think there is such a trade-off when you're trying to compete with TCP. So there is, you know you had this in your in your competitive mode.

E

You can try to be a bit more or less aggressive, which will produce more or less queuing so I guess there you have to trade off, but whenever you are able to detect that there isn't only any other traffic competing with you, I, don't think the idea of have of including such a trade-off in the design as a base idea is good at all. So.

P

My answer to that is, like maybe you're right, like I, don't know for sure, but the trade-off does exist on the product side, because, on the other hand, we have video calling applications which have not been able to use cubic or bbr.

P

So far as far as I know, they have to use something like Google condition, control where they have to really optimize for end to end delays right on the transport side like we did run some limited experiments by using like a very high value of Delta right and I did see that queuing delays decreased even more so I mean I, don't know like fundamentally, if, like it's a good idea or not, but from the application point of view, this definitely a trade-off and if the condition control could also kind of like give you a varying performance depending on like what like, it could also be expressed in a different way.

P

For example, prop 8 exposes a target delay to the application and it will just try to optimize and limit the queue. The bottleneck you delay it to that like that, could be an alternative way for application to benefit from it like I, honestly, don't.

E

P

E

Maybe point out some other work: offline, yeah.

Q

Stuart Cheshire from Apple, you presented some results showing marginally better application throughput, which is interesting. I think it would be interesting to understand where that's coming from, because either the other congestion control algorithms were letting the line, go idle and wasting capacity, or they were retransmitting unnecessary and wasting capacity with duplicates or something else, and it would be really interesting to know where that extra is coming from.

H

Where these tests done on Wi-Fi links or LTE good.

P

Questions so we ran these tests globally and when I looked at the data like about 3/4 of the users were using cellular, I did not I, don't know like LTE versus 3G or something else, and about 1/4 were using Wi-Fi, but.

H

In the Wi-Fi case, there would be competing traffic at the bottleneck because it would be probably the first or probably.

P

And when I looked at the results and compared them like the trend was similar for Wi-Fi and cellular. Although we saw more wins on cellular as compared to Wi-Fi and.

H

What is the Delta value for these experiments? No 2.04 Thank You.

I

Roberta they own I just wanted to push back on the labor sis bandwidth not making sense. I also want to agree with delay versus bandwidth, not making sense I'm, holding both of these opinions simultaneously, because it's hard to define what the network is.

I

Does the network diff'ent is the boundary of the network between the application and the operating system, or is it between the nic sending packets and the fabric to another NIC right, the you know, depending on where we are in the stack we tend to think of either of these as the network and try to exclude one of the others generally the internal to the host part.

I

So we know from previous presentation by Neil Cardwell, for instance, that there are parts of the stack that are trying to reduce the amount of overhead that we are experiencing within the host and the effort by which we do this and, of course, there's Nagle's, etc. Right. The effort by which we do this is really a trade-off of how expensive it is in CPU versus how important it is to get the information through. So there you go sorry.

B

Thank you. Thank you. So much folks and I want to thank mitten for showing up here and presenting this. Please make him feel welcome it's his first IETF /I r, TF / ICC RG meeting.

B

And for everybody else watching I love to see more presentations like this. If you are doing in experiments in your house or in your lab or at your company, come share them with us and we now move on to Marcus and get this up. Mc/Dc CP.

B

There you go take.

R

It away my name is Marcus Armand from Deutsche Telekom and today, I want to present some joint work with Karcher at University and City University London on multipass multi-party ccp that we intend to to use for providing multiple capabilities for UDP and IP traffic. Why I want to presented today here at ICC RG, is that we see a challenge there in respect to congestion, control and I.

R

Think you, you are the right group to become aware of this and maybe work together with us on a solution B, and we also think that this is not a limited issue to the multipass tcp. Maybe that is also something which affects in future the community we come to this during the presentation.

R

First of all, I want to give you a short introduction. Why are we working on multi pass DCP? What is the reason behind I want to make you aware about the development we did so far and then coming quickly to the point where we see the issue of congestion control, the multipath TCP as such is usually pushed at TS vwg, where we also have some updates in the afternoon session.

R

Okay, starting with them motivation, why are we working on on a solution for providing multiple capabilities to two UDP or IP traffic? So, as you may know, there are specifications ongoing in the 3gpp area and also in the BBF proppant forum on multi connectivity architectures to provide multi-part usage to either mobile phones, though that is on the upper part that is directly happening at 3gpp, released 16. So within the 5g standardization and they're discussed in in the area of 80 s, SS XS traffic steering switching splitting it's an operator controlled approach.

R

So that's why we are so telecom are very interested in that that means multi connectivity is terminated in the operator network. The same is also happening for the hybrid access use case, where multi connectivity is applied between a multi path, capable CPE using cellular and fixed access and again the operator network in future. The IP Texas will most totally also rely on the 3gpp, a TSSs specification and currently in release. Sixteen multiple CCP is a protocol which is defined for making simultaneous use of multiple X's.

R

If you start now with having a look on how the traffic mix has changed over the last years, I I think I, don't tell tell you something you we see that the quick traffic is coming up more and more though that is some statistic from our fixed network in 2018, where we already see more or less 12% of quick traffic and we think with HTTP 3 deployment.

R

This will become more and more, and maybe it comes to the point where Creek or UDP traffic in future dominate the TCP, so that is totally different to what we have seen, maybe three, four or five years ago, where TCP, where the dominating protocol, with almost 100 percent, with all almost 100 percent share, but coming now back to to these multi connectivity architectures, we have seen before within the a TSSs we're multipass TCP the protocol for providing multipath transmission.

R

This means that it cannot cover anymore all the traffic which is today generated between customers and and the internet. The Gothic week is coming so our finding. So far as we have a long lasting experience was not a past TCP.

R

It's a very well working protocol, it's efficient, it's a good candidate to use or to provide yeah, multi connectivity services and there's also finding that the congestion control should buy TCP and also used with a multi pass TCP for scheduling decisions and so on.

R

It's very beneficial another finding is so far. If you look at IKEA, for example, there is no multi part of a call for UDP or IP traffic and an encapsulation into multiple TCP of such protocols. It's not an option. At least that is our opinion, and that was also something discussed years ago in the multiple TCP mailing list that this does not make sense, because if you encapsulate UDP or IP traffic into multiple TCP, then you impose the reliable nature of TCP.

R

So a potential multipath solution for a UDP or even IP traffic must not impose TCP like reliability. Additional high latency, packets, cramping or head-of-line blocking otherwise it Prague the UDP and IP principles on transportation and service expectations before I come now to the solution and talk about the multipath DCP itself. I want to present what what are the key components for a multipath transmission.

R

Sure for sure you have a generator and a receiver part in between you have multiple paths or multiple X, so at least more than one path to provide multi-multi connectivity and you you need a you- need a scheduling unit to distribute traffic over a multiple part, and it is very beneficial to have a path estimation for the scheduling unit, which gives you information about the path characteristic. Latency is data rate available capacities so that the sketching unit can make proper decisions, efficient decisions and that's not overload.

R

The path may be according to two policies, preferring paths with low latencies and it needs this information which, which latencies are applied to the individual path and also in most scenarios and real scenarios. You need some, we or a queue to compensate and the different paths or icterus takes to compensate the latency differences and so on for the three ordering unit or poor for making this reordering working for sure.

R

You need also some some sequencing, but that is that is very simple as such, so the the the most critical components I see, are the scheduling, the path estimation in the reordering. So all of this has worked together and to make efficient multi-party, which possible now coming to the solution. Multipass DCP, so.

R

At the ITF which which types of protocols are available, which can support us in finding a solution to transport, UDP, IP traffic, over multiple paths and there we came up with the DCP protocol, which is a mix between TCP and UDP. It's it's of our unreliable nature, but it has a congestion control and that we think we can exploit for our purpose purposes.

R

So what we imagine is we have the multipath DC CP as such, which by itself is only possible to to distribute a TCP traffic over multiple paths.

R

But if you combine it with a multipath framework which more or less means we have virtual network interfaces which can consume any kind of IP traffic and the the traffic which is is incoming into this virtual network interface, then encapsulated into the multi pass TCP distributed and then on receiver side again may be reordered and then stripped all the DCP information and send out of the outgoing virtual network interface.

R

So that is a rough description of how we imagining a solution.

R

You can. You can find more details in all the draft I have mentioned here on this slide, and that is our prototype and also our simulation test that we have available. So we have a Linux kernel implementation, extending the the available DCC keep of a call by multi part capability and the same. We have available in nf3 for simulation purposes, and it's the descender side. You can see on the on the left side, the multiple CCP home gateway, as we have called it, and that is very similar to to multi pass tcp.

R

We have a modular scapula which can load different kind of scheduling, algorithms round-robin, for example, a cheapest pipe first, which is a path prioritization scheduler, but also more sophisticated ODS, for example, out of order for inorder arrival schedule on receiver side, and that is something you compare to to multiple CCP. We have reordering modules, so we can apply different kind of reordering mechanisms.

R

We have a passive one which more or less does nothing. We have a adaptive fixed algorithm, which waits for a fixed time when, when packets are missed and then skips possible gaps in the reordering queue, we have a depth effective. So that is a dynamic approach. Using the RTT informality, the latency information from the congestion control put a ten-time, and last but not least, we have delay equalization, which is more or less no reordering approach. It just tries to delay the packet on the faster path by the latency difference. We have a path manager.

R

We have this virtual network interfaces from the multi path framework implemented, so it's more or less complete a set up which we can use for testing.

R

Presenting now some results, in short, that is really a short sort of results. Just to make you aware what is already working and then coming to the challenge we seen so that is, that is our na 3 setup.

R

Where you see that we have plenty of possibilities, we can simulate different, you ease we, we can simulate, you know tea or cellular axis. We can simulate Wi-Fi access and also- and we can simulate moving scenarios where we move. You ease from one point to the other, which maybe lead to the effect and therefore I have results in the next slide that we lose connectivity or that we enter you access nodes and we can get connectivity.

R

So that that our results from last ITF but again, just to remember you so switching and aggregation is it's possible with our multi-party ccp setup. So on the top you see when we move and you e out of a Wi-Fi access, so that the the Wi-Fi access is not available anymore. Then the ue automatically connects to the LTE and the multi-party Suzuki is capable of doing this in a in a seamless way and with the fewest switching approach you see between switching from by factory.

R

Suddenly you have a get with aggregation mode, which was also supported by multiple CCP. You see a more or less as seamless hand over without having a gap between switching from Wi-Fi to LTE. We think that is a very beneficial scenario.

R

The same test. We also did and MP think that is very realistic. Maybe I have to switch back so that what was pure UDP traffic? It was some some iperf UDP traffic we generated, but that is not very realistic, so we did the same test with UDP combined with Nara congestion control. Most of the UDP traffic we see in the field is, to some extent congestion controlled, and if you talk about quick, then it's it's pretty clear that this is congestion control, and you see here in the same scenario, you're, switching or aggregation.

R

It also works to some extent, but you see some some interact between the Nara congestion control and also the D CCP congestion control. You have this black reference line, so that is from the previous slide, the results where you have seen more or less yeah, maybe where you have seen that, after switching from one access to the other, a direct use of the full capacity was possible, whereas, whereas now with the nadir controlled, UDP traffic, you see it takes on time to to adapt.

R

This one I I will skip for time reasons, and that is now also exploring a little bit what's going on in in terms of delay. That is also some result from from last our ITF. Where will be proved that reordering makes sense, we need reordering in a in a multi pass set up supporting UDP traffic and the cream bar here. That is proving that reordering is beneficial over scenarios without reordering, and you can find the details in the slides from from last ITF.

R

What is new, we have some real world results with YouTube traffic based on the call now- and you see here, the set up- we have the multi Party, CCP, client and multi pass TCP server, that is a local set up and but connected to the internet. Connected to the multiple TCP client. We have a laptop running the Chrome browser and between the both multiple TCP boxes. We have two links to ethernet links going through so-called TC boxes, but which we can shape the links and where we can change the path characteristics.

R

As I said, we use the Chrome browser with that we are able to request YouTube videos using the quick protocol.

R

We requested a static video with a fixed resolution and we skipped this video at ten thirty, fifty 70, 90 and 110 seconds, and to force a buffering.

R

The total curation of the test were 120 seconds and the network condition chained at 60 seconds. We see on the next slide again, so we had to part both at one megabit per second and a latency of 10 on the first path and 50 milliseconds on on the second path of the 60. Second, be changed from 10 to 90 milliseconds on the first path and on the right side view you can see the playback ratio, but is a data rate.

R

Where you can see that the green circles that are the one using multipath in combination with reordering promise promises the highest gain, whereas other scenarios like single pass usage or.

R

Multi multi, Party, CCP, without reordering ya, have have some have some lower playback ratio compared to this multi-party, which, with with reordering, however, that is not part of this slide.

R

The detailed evaluation revealed and the imperfect parts use it even in the best performing use case, which means, even if you have reordering and multi-part usage combined, there are still some room to improve the customer experience and now coming to an intermediate conclusion before coming later to the final conclusion, so our multiple CCCP prototype and also the the the standards or the the troughs we have available at TS vwg, making very clear that this is something which can brought a work scheduling, profits from the TCP congestion control and can therefore perform similar to multipath TCP.

R

The reordering on receiver side is is proof mandatory for efficient multipath transmission of unreliable traffic, smart algorithm to keep a traffic flow smoothly, ongoing, are required and can make use of congestion control information. Head-Of-Line blocking is not purposeful and should always assess if it is worth to wait for missing out of other information, which means encapsulation into multiple TCP doesn't make sense for for UDP or IP traffic.

R

You have also seen that we have plenty of scheduling and and reordering algorithms in our prototype and all of this together, benefits from from the congestion control the ship with TCP. So from this perspective, it was a very clever decision to use the TCP and also with the congestion control. We can make sure that we do not overload paths, though, that we do not force a packet loss.

R

However, the main target protocols transmitted over the multipath TCP framework like quick, employee and own congestion control in combination with multiple TCP. This leads to a kind of congestion control over congestion control scenario, with well known issues from literature.

R

So now the point where I think it becomes very interesting for ICC at ICC Archie, so in all of our tests, we used so far the CCI d2, which is a TCP like a contest in control and.

R

We asked ourselves if it makes sense to use CC ID what could be the alternative and how intra neural. We tackle the problem of congestion control over congestion control and such scenarios. If we assume that combustion control for multipath purposes with the right way to go.

R

R

So, as I said, we have this issue of congestion control congestion control identified. We think also that affect the the community, so there are troughs regarding quick tunneling where at the end, you you have the same question depending on which type of traffic you want to tunnel through through Creek.

R

My main question today is ICC, actually the right place and dealing with this question of congestion control over congestion control and who is interested in elaborating this topic further, not dedicated to this multi parties. Yeah I, think it's a general question, but for sure it would help us in pushing the multi processes EP further.

B

Thank you very pleasant, Dacian, Marcus, I think just very quickly to answer the question about is this: ICC has the right place it can be? Is the Shorin said that I have I think the question I would have is other other folks who are interested in in this problem and I'm, not just speaking academically. Is this a real problem, folks interested in deploying our face? Are they facing this problem in production that sort of question?

B

If you can you know you could you could have this conversation on the mailing list and see there are people who are interested in working with you on on establishing relevance any other questions.

N

We have Anita a quick question: have you ever experiment with that pass is not disjoint. No, for example, with some savvy multipass DCC subfloors share the same bottleneck. Have you ever experiment with that kind of environment? The question.

R

Was will be experimented with couple congestion control like multiple TCP uses? No, that is on our to do this, but I'm not sure at the end. It also ends up in congestion control over congestion control. For sure we have to do this. We should do the such test, what we have done, but what was not able to to show up that. We have implemented PBR 40 CCP, and we did some tests with that. That was very promising, but copy congestion control is on our to-do list.

R

We have the issue that we have to report everything into the DCP world to make such tests I think that's important, make sure it's on our to-do list, but again the problem stays at the end. Then we have inner congestion control of the piggyback traffic, then over couple congestion control. Where also think that this end ends up in some challenges or conflicts.

R

B

You so much Marcus, and with that we end the session I'm, going to quickly note that in Vancouver I'm trying to get together, there's been a recent spate of academic work on EPR analysis, so I'm gonna try and get those authors and even need to show up or BBF folks to show up in Vancouver, so I'm hoping to do two ICC re sessions in Vancouver four of those panel. This is tentative. Of course, one of them will be around bbr specifically and the other one will be more generic.

B

But if you have any work around that, if you have any analysis that you've been doing with DVR or vrv, please write to me I'd love to have that be a part of the program and that we will end. Thank you so much specifically tank Marcus for winding down his kind of his presentation quickly and we'll see you in Vancouver. Thank you folks.