Internet Engineering Task Force 104, 27 Mar 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: IETF104-WGTLGO-20190327-1500

Description

This Technology Deep Dive from IETF 104 covered:

* "Spherical Routers"
* Spencer
* Forwarding Plane Realities + Q&A
* Frank Brockners, John Scudder, Toerless Eckert
* Summarizing implications for protocol designers
* All - this is a working session
* Next Steps for the summary of implications for this topic
* Chairs
* Next Steps for Technology Deep Dives on other topics
* Chairs and AD

More info at:

https://datatracker.ietf.org/meeting/104/materials/agenda-104-wgtlgo-02

https://datatracker.ietf.org/meeting/104/proceedings/

A

It's not the contouring, so it's actually a good time.

B

So we're gonna start in couple minutes. We have a contest who can say that better? How many ways can you say that I'm gonna try and say we got Lego.

B

You can still talk for another minute, then.

C

And if this turns out to be, we got Lego blame very Levi. Iii put the I put this slide up talking to him about it, and he said he said he pronounced it that way within seconds. He has a gift, man think I as a gift.

C

Okay, we're going to get started, welcome you to the technology deep dive on spherical, rubbers, 40, plane, realities and implications for protocol designers, and we are at least for now working on whether we have consensus to pronounce it as we got Lego I'm Spencer dawkins. This is the lovely alvaro available Karen to my left and we will be your host for the next.

C

What hour and 55 minutes, or something like that.

D

B

I already said: hi and everyone shut up so I'm, not going to say I'll.

E

C

E

C

And so this is a IETF activity, so we put the note well slide and trust that people have read it and then they will make the choices. So we had this as our agenda and bashing. It is okay, just to do administrivia and, like we said so, the pronunciation is we got Lego for until we come up with a better one, it stands for we got the last good one, because Warren said all the good ones are taken with Cindy asking for an acronym and I said: oh no, we got the last good one.

C

It's gonna be like this. The whole afternoon guys have fun, so we did the know. Well we're willing to bash the agenda. You know: wouldn't talk for a few minutes about what we do at a technology. Deep dive, I'm gonna talk for a few minutes about spherical routers, smart people about 40 playing realities. We're going to talk about forwarding plane realities could be frank, john and taurus we're going to be summarizing implications for protocol designers, and this is the part that I want everybody to go into this with the right mindset.

C

This is a working session. It's not tutorial it's a working session, so we we hope to all come out of here, smarter than we came in, not just not just some of us, and we want to talk a little bit about the next steps for the summary of implications for this topic, and we want to talk about next steps for technology if you dives on other topics and we'll be bringing up our sponsor URI director, the lovely warrant.

C

For that, though, we got Lego, etherpad and jabber room or in the usual places, and you can find those most easily, probably from the ITF agenda in the data tracker for links. Where are we yeah.

C

So when we start talking about doing this, we were saying first assume a spherical router with it zero gravity in zero degrees temperature. This is basically the you know. These simplifying assumptions that people make that help them do. Math we've hit spherical cows at the ITF for a very long time. This is actually Heather's company and and group. You can see the spherical calendar business cards, saying Moo, but this is now spherical rubbers. Some routers really are spherical thing.

C

We made one at least one and if you're going to be encountering spherical routers and everything that you do on the internet, your problems are solved and you can go to have more unstructured time, but if yours isn't we're hoping that what we do today may be a view sport to you, so how we got here in the first place, so Jon's gutter gave a well-received talk or not a router design at the Internet engineering planning group to a small audience in 2015 on Sunday mornings is when that group meets and the other thing that's been going on is people assuming a spherical router.

C

They we say they exist, but most routers are spherical and what that really means is the too many working groups encounter a contentious cross area or late. Ietf last call issues most recently if you've been involved in less called and balloting discussions about draft that was using ipv6 extension hitters in a kind of an interesting way, but there's others- and you know these are late- surprises for too many working groups. So we were hoping to do better. One and I thought a technology team that I might be useful.

C

Like I said in another tutorial, people understand how routers work in general. What we're talking about is a cross area working session on a complex topic, so I started recruiting people to help with a deep dive on mother router sign and that morphed. Of course it always does at the IETF, and here we are now what we're hoping for a technology deep dives we to complain about unstructured time. The working group chairs complained about unstructured time during the working group chair session on Tuesday.

C

The community will complain of the IHG about construction time at the whiz day. Plenary open mic in just a couple of hours- and you are here somewhere in the middle, so I would encourage you to aim those comments where they are most likely to be received. Well, so, if we're not doing a tutorial, what are we doing? And you know we are doing and Lauren maybe used the word information-sharing, but he was the one that actually said. Oh great could speak on a complex cause area topic.

C

Do they have open discussions to understand the world from each other's viewpoints and to clear the air about some things that a lot of people may be confused about, and not particularly from one community together?

C

This is the part where you're doing work figure out what the implications are for protocol designers together figure out. What, if anything we should do, is to follow on to the second to the activity that we're going to be doing for the next little bit here.

C

This is the classic picture. What I think we're talking about we're a bunch of blindfolded people are grabbing pieces of an elephant and they know very clearly what they've got ahold of, but they don't know that they've got ahold of an elephant, and so I say this is the classic picture and it kind of see it kind of does a better description of what we're talking about here then I could possibly do in words I.

C

Think we're to the forwarding, plane, realities and Frank JAMA and tourists to come on up and take the field. If that's okay and thank you all for for serving the community in this way.

E

Apparently, I I walked up too fast, so I get to be the Stuckey. So so this used to have a more grandiose title.

F

E

Alright, so those of you who, who looked at your agendas early in the week might have made the mistake of coming in here. Thinking that we were going to talk about. You know some kind of vision for the future, and you know what what routers for the 22nd century are going to be and actually because I think the title was modern, router architectures or something like that. But but it turns out that that title was more in the sense of my undergraduate class on modern literature, which meant 19th century literature.

E

So so what this is really like, it is really about, is what what routers are when they aren't general-purpose computers. So if that's not what you were here for, you can quietly make your way to the back now and if you don't, then I'll assume that this is what you're here for this is.

E

Yeah, this sort of came out of that IEP G deck that I gave five years ago, sort of updated by the three of us and I. Think a lot of it turned out to still be relevant to today, but you'll see seems that in places and that's because of course, it's a design by committee.

E

So we you know, start out by talking a little bit about. What's a what's a taxonomy of how we build routers, oh and one sort of disclaimer I want to throw in right at the beginning and by the way, if you guys yeah, you got a mic too so whatever, but at various points in the in the deck I. Think we talk about things like you know: high performance, routers, large routers and so on. It turns out that most routers now are high performance routers.

E

You know, according to this definition because of you know, economic realities and, and that kind of thing we dive into focusing on one particular flavor, affording architecture which is a very relevant for affording architecture and end up kind of spitballing on what are some things that protocol designers should know and at various points in the past evidently happened because oh and I should also add one more thing, which is that I do not design chips. I am a software guy and I.

E

You know you can think of me as like a sock puppet operated by a remote guy who actually does design chips and helped craft the original slide deck and his his name is in the fine print there. It's only in fine print because we took his deck and butchered it, but and I'm sure there are people in this room right now who know more about designing router hardware than I do and I hope that you'll come up to the mic at will and I think we're.

E

You know, there's no requests to hold questions for the end. Necessarily, you know we're big boys and if you bring up a topic- and we want you to hold it for later- we'll ask you to, but otherwise please, if we can have it be interactive, that would be nice.

E

So we've all. We all know this. We've all heard this, but it only applies. You know good fast, cheap pick to it only applies on my stuff right on on your stuff. I want you to have all of it, and this is you know, kind of if we can come away with just one understanding from from the next hour or whatever it is it's that yeah this? This applies to you to your router hardware. Also.

G

There is an so I wanted to maybe start before getting into you know the the details we want to talk about for most of the presentation to give a little bit an overview of you know what we may consider to be the landscape of the different realities of router platforms that we have today, primarily also to you know, make sure that is understood that we can't talk about everything. So if we start in this picture, traditional new, you know the original routers, maybe in the 70s or so were starting to be just general-purpose.

G

Cpus that has survived over the decades has expanded itself very much, maybe starting in the 90s by you know having additional CPUs, not only one CPU. That does everything you like. You know you design your next generation network service.

G

Let's call it multicast on a Sun workstation, the beginning of the 90s, but you basically started to have CPUs that we're doing forwarding plane separately from those two in control plane and then every line card getting CPU rights, CPU was staying there then you know later on, it became less favorable, and you know in the last ten years, I think we've seen a lot of revival of CPU forwarding through a lot of x86 virtualization nfe in data centers associated with a lot of even open source packages that allow you to accelerate this and leverage.

G

You know the capabilities of them. These systems have a lot of interesting properties, primarily they're, normally an ideal world for protocol designers, because they are pretty much. You know the more general they are. You know the less. You know constraints you have by oh, they have different line cards and so on they become fairly spherical and that's very often what in the past protocol designers expected all routers to be, and obviously you know the more you look through the landscape, the less that's true, the main issue. I.

G

Think for them is you have all the flexibility, but you have really no way to predict performance. So whenever we've been trying to, you know predict for particularly deployment the performance of a combination of all type of features and the forwarding plane. Well, you know it just gotta measure it right, because it's so flexible, specifically for the aisle for for the IETF I.

G

Think what's interesting is that in the low end, like let's say in the IOT space, CPU forwarding routers are pretty much the only game in town, 8-bit, 16-bit, 32-bit, CPUs, that's kind of the range to which they go embedded devices, small memory routers. There are a lot of you- know: IETF working groups that work with routers in that space and also because they have the flexibility of this almost spherical router they're. They basically can design protocol solutions that we have really a hard time.

G

You know to scale up to a higher platform, so my typical example is the wonderful source routing header that was done in the role working group for Ripple low-end. You know took us a long time to get to that point as well. Let's say with SRP 6 in you know higher end routers and that's because you know CPU lot more flexible, faster to get to the protocol stage that you want. So then, actually, you know the next step that happened in the 90s was really heart, we're switching the most cost optimized.

G

You know chips that you can build for a purpose, no flexibility, but rather you put the RFP that says it must do this protocol. Let's call it ipv4 Ethernet, switching, please, basically, for the following speed built a chip that is as cheap as possible.

G

Obviously, that's a very good reason to not have flexibility and in many networks, right, you're, basically designed upfront mission, specific networks- and you know never going to be extended, never going to change, control, plane, very simple and that's basically been you know, still truth until today and I think that's not going to go away and I think that part of the equation is still the most difficult one for us in the ITF to build new protocols.

G

For primarily, you know it's a matter of you know business case right if, basically, what you're trying to achieve and solve, as the use case with your new solution in the ITF, is, if it's big enough, somebody will build you a dedicated device to do this example of this. Actually, you know I, Triple, E and friends. Kind of these you know layer to type of standards, bodies they're much more centric to that approach.

G

Insofar as that, when they're designing protocols, they very much have exactly that centric type of switches, mostly even if it's a router in mind and they basically in many cases, they'll dedicated Asics for that right.

G

If, if you need some form of flexibility, maybe you add an FPGA that gives you some degree of you know programmability, but that's very hard, very hard to extend and and only done in exception cases, but so that's, basically the opposite end of the spectrum and then basically we have what we want to concentrate on for today's session and that's what you know as the common term, maybe multi stage programmable forwarding.

G

There are a lot of varieties, that's what John and as are going to talk more about for the rest of the presentation that basically, you know from very long instruction word to basically, you know execution elements that almost look like a small CPUs over to a lot of hardware, acceleration coupled them and that's the type of hardware that we're seeing now all the way through the network distribution, core data center, a lot of differences in the different spaces still right, because you're still these market segments are all large enough to validate.

G

You know: individual chip designs, we'll see how that is more going forward in the future. One I think very interesting aspect that came out of the data center was kind of the idea to make the consumption model of the chip in the forwarding plane somewhat more open right, because if you look at it CPU forwarding, yes, everybody could program routing a nice dual program, routing on CPUs. Almost nobody can do that. You know on the fixed forwarding, Asics in layer, 2 layer, 3, switching on the right-hand side and in the middle.

G

It's also very hard been traditionally to get access to any type of program ability of the forwarding plane other than maybe you know, establishing 5 type of forwarding entries as the best thing of something like open floor. So but that's not what I would consider programmability, and so the question was always you know: how can people outside of the core vendors get into programming of these multistage programming, forwarding entries and so P 4 was one of those you know today, probably as a programming, most successful ways to attempt opening up that programming to other players.

G

I think it has certainly been very successful in the research world, because you know a lot of researchers have started that are doing. You know networking to consider. You know trying to prototype their forwarding, play notes and before and therefore be more exposed to the limitations of what an actual forwarding plane has not ideally but a lot better if you as if you were just starting to build forwarding plane, let's say for Linux kernel or something like that or DP DK right. So that's basically I think a very interesting aspect for the future.

G

Maybe even when you're doing things on the protocol side, that's associated with certain things in the forwarding plane, you know how do you want a prototype that forwarding plane if you're, not one of the vendors? So what else do we have there in this bucket? So beside what existing details, they were doing. I think at the end of the presentation, I'm going to talk a little bit more about that I.

G

Think in this space there is a lot of good work that could be done for more intelligent forwarding planes and I think you know two of these areas that are very important in my opinion, but that are you know, even at the at the edge of what the the forwarding plane into you could do is programmable qsr I think that's very hard and then the telemetry as well. That Frank has done a lot for and in general you know, chip building has become a lot easier in the last ten years.

G

Right so I mean, if you just look at the total number of startups that are building dedicated chips and they're, throwing just all the possible execution elements together right, whether it is you know, oh, let's put some CPU cores some, you know accelerators some NPU, some, oh, oh, a is also important.

G

Let's put an inference accelerator to do some more logic in there DSP to even do more statistics for IOM and what you have and so I think it's very hard to predict in which market segment we're going to see which evolution of you know the current four-wheeling scheme so always keep eyes open.

G

So then the other part that I wanted to mention is that beside the forwarding plane that we're going to take talk for the rest of the presentation about there also is the port forward. It, the the poor control plane right. So the I wanted to call this the reference router from hell and everything you could do wrong on this space here, but a little bit too afraid to write that down, but I think most people won't understand high-level. You have your control plane and you have your memory in there there's some CPUs there.

G

Then you basically have these magic forwarding chips that are doing things and they'll also have a lot of, or maybe not, memory, T commentry on and off chip. That's going to be talked about in the presentation, a lot more, but a lot of my experience in the past with protocol designs, weren't necessarily only the oh. We just designed the next forward in shape. What did you want to do? Oh, no, sorry, it just came two weeks too late. Please wait!

G

Another four years for the next ship revision, we're doing I'm talking purely about the thing like Oh only had five watts left in our one kilowatt box, so the CPU is going to be extremely slow and by the way it's going to be eight core.

G

So if you haven't multi-threaded it, it's only going to be an eighth of the performance that you get in your control planes, if you, but don't worry that you can't calculate routing tables fast enough, because you won't be able to install them fast into the router right, because we've basically used the serial port down into the forwarding plane and oh by the way it has ten line cards.

G

So if you want to send to ten line cards, it's even ten times slower right, so the router from hell, obviously I, haven't luckily found a single router that has all of these problems, but trust me. You know each of okay notice, your report, admittedly, but something similarly slow. um So that's the problem over all right that basically trying to, for example, have more money in a router spent on the control plane.

G

When you build the box is something very hard to argue, because at the gist of it, the real metrics and I had that on. The previous slide is really what data plane throughput. Do you need, and then basically you know how much does it cost per bit as far as buying the box, the size of the box and how much power goes through the box right, and so that's that's.

G

Why always in in the majority of box designs, the poor control plane, part of the solution is a way on the back end burner and in many cases it might actually better to you know, start buying routers that don't have the control plane and basically could connect a fast PC with a hundred gigabit port to it and use that as a high-performing, control, plane, router just from past experiences. So um what's anything else, I wanted to say from this I think this was good enough. So let me pass back here to John.

E

So now I'll say all the same things only I'll say them way slower and with more slides.

E

So exactly so yeah sure, let's just showed his his router from hell and that's pretty much the the reference model, and you know again, everybody you know, builds routers with more or less the same, functional blocks. You've got you know a functional block for your control. Plane.

E

You've got to have a way to get the packets from the gazette a--'s and get them over to the COS outa's and that's your fabric, and you know sometimes it's a real fabric and sometimes it's shared memory, but there's there has to be some way to move the packets around and then you know, there's the packet forwarding logic itself, so you know call it a packet forwarding engine or whatever you like, which is actually the the thing that does the does the lookups does the queuing does the buffering and so on, and you know it as we already you know lead with it.

E

You know that sort of spans the whole range of flexibility from completely programmable to completely set in silicon. A slightly different picture of the same thing and I think you know the thing to emphasize here is that yeah you've got all the functions in the box, which makes it kind of attractive to think. Okay, look I've got a nice, you know general purpose CPU over there on the left and I've even got another one over there on the right, and so I can really do everything.

E

And it's true, you really can do everything as long as you don't mind taking you know something like a two order of magnitude hit in performance. Every time you cross one of those domain boundaries, or maybe worse, I, don't think I'm to say a whole lot about the fabric based architectures. There's an interconnect so yeah packet, forwarding engines like I said, are the the things that do the the work to figure out what to do with your packets right.

E

They they, you know, do they analyze the packet, they do the lookups they take care of the buffering they take care of. You know they be simple or complicated, queuing that you want to do, and they you know again. The the programming model can can run the gamut and really you know a lot of what's coming up here. Is there is no new thing under the Sun we only know a certain number of ways to build computers and a certain number of ways to make them fast, and you know it's sort of clock scores.

E

You know how fast you get to your memory. Maybe you only have a certain amount of space on your die. I guess, final bullet right. Do we I I'm not going to focus we're not going to focus on like? Is this all one chip? Is this multiple chips?

E

You know again there's this is sort of general computer system design and and not unique to routers it's attractive for all kinds of reasons, to put everything on one chip, if you can- and sometimes you can- and sometimes you can't so there's you know our cartoony reference model of of what these things are same thing you know put on two axes. Also, cartoon acts is not to scale where your general-purpose compute engine is.

E

You know all the way off the top on flexibility, but it's you know, jack of all trades master of none and by the way. Why did you want a floating-point unit in your router anyway?

E

Why are you buying those gates and probably never using them? And you know all the way over on the other side? Is you know the thing that's completely set in silicon or you know more likely is the the multi-stage thing. That's somewhat programmable, but the stages are, you know, really separate Hardware units and god help you. If you forget that.

H

E

Still: okay um right, I I, we've we've! We sorted these several times so um yeah right.

E

So I think the first bullet has been sufficiently belaboured, but you know. So what are the kinds of trade-offs that that we're making when we, when we put these things together? This is an example trade-off. But in my mind anyway, is maybe the most critical one, because it's a thing that gets baked in and then you really can't change it once you've taped out the router, and then you know if you and that this will be repeated with with more pictures in a few slides, but typically when you're forwarding a packet you've got.

E

You know some number of bytes of key data that go. You know up towards the forwarding chip and everything else that was in the packet ends up in RAM, and it's really really expensive to get to in terms of of clocks on that forwarding chip.

E

If you can get to it at all, and of course, as with all system designs, the the trick is to be as inflexible as you can, while still being able to do everything you have to do, and the problem is that when, when you're building hardware, if you guess wrong, your guesses lasts a long time, I'm sure there are people in this room that can, you know tell us horror stories about. You know just how old the oldest router in their network is it's probably very old.

E

Now, because routers, you know, the high-end router of today gets moved into the you know some other application tomorrow and they just keep living and living.

E

So why do that?

E

Tour was spoke to that earlier because of power efficiency because of volumetric efficiency, the the you know, 500 million to a billion, is probably now this was originally presented four years ago and hasn't been that number hasn't been updated, but the sort of two orders of magnitude thing I alluded to earlier is probably still about right: okay, well I care, and that gets back to the you know what motivated this both to begin with.

E

Well, just because of the the economics of building these things, these architectures are, you know not only in use they're from, for my keyhole at least their use is becoming more, not less ubiquitous because you can buy a you know, $200 box or whatever, that has quite a capable forwarding chip in it.

E

But it's the reason it's capable is because exactly because it's not a you know it's it's it's making the trade-off of getting higher density or more forwarding speed at the cost of being less flexible and less programmable, and experience has demonstrated that certain protocol characteristics, even though they look great on a white board and look even great in a C, compiler or.

E

Virtual machine, they don't look so great at all running on a on a forwarding chip.

I

All right, I think I'd like to go on baseline, the thing even a little further, so some really motherhood and apple pie trade-offs. So we have a piece of real estate as our chip right, our dye and it up. It is up to us how we want to go and utilize that piece of real estate, so you have a set of gates. You can use these gates to do forwarding.

I

You can use these gates to count what you're doing you can use these gates to store stuff, but you can't use one gate for all of that, so we have to go and choose, and you can say well, I want more.

I

That means you need to go and draw your real estate, but if you grow your real estate, you yield because well there is a waiver and there's defects on the waiver, so that drives costs like crazy, which is why chips have a certain size and one of the the fundamental things that well we need to go and care about is power right and you can say: well, there is dynamic power and dynamic power is the product of say frequency.

I

Capacitation and well your voltage to the square plus some activation or time some activation factor, that's the dynamic part of that, but as we kind of started to hit the limits of Dennard scaling, but also men, people started to go and package things.

I

So you can't shrink the gate anymore, so you started to go and pile things on top of each other and you're just going denser, and that means also that static power is a significant concern these days right, so the the denser we pack the more static power, ie leakage we have that we have to go and deal with, and that's just the tip the stuff that you push into. The system also needs to go out of the system again, and that is cooling and I.

I

Remember one of our routers a little while back I think we spent more power on the fans than on actually driving bits in the system just because we had to bend the air three times as it passed through the system. Now that you have the air outside the system, you need big, yellow, well, cool it down again and that's okay. If you may have a single router, it's not so!

I

Okay, if you have many routers in a relatively constrained environment like if you're a data center so against something to take care of, unless we always want to go and put a power plant next to a datacenter, which is what people do right. So if I am a data center person, my my design constraints from an optimization perspective will be very different than I'm just operating one corridor.

I

Despite the fact that I have this recurring cost off-off-off power that and you spend anyways. The other thing is, as I said earlier on, like is memory and where do I spend my money on if I want to go and well spend space on on the die, if I'm a service provider Percy I want fast convergence and a large routing tables, so I want really rapid updates. That's what what tourist was referring to earlier on and so I want the memory on chip.

I

If I put the memory on chip, I can't put other gates for forwarding on ships, so it might gonna go and compromise on the bandwidth side or packet processing side. But I have all the memory I'm ship- you can argue- then maybe I don't want that.

I

Then I go off chip and put the memory off chip, but then I'm spending connectors that I could use for customer interfaces to go and go off chip to the memory again, a trade off and well, if somebody's asking me for well, I ideally I want memory at the speed of l1 cache for everything, but that doesn't exist right and so the further you go away, the more you pay for going away problem and then also I think. Can we it's 2019 by now?

I

Do we really need hierarchical queuing all the way we needed a twenty years ago, because if my broadband home connection is one gig I, don't really care about that qsr configuration on my router anymore. It's just works right. Do we need to go and built that really into the system against spent gates on hierarchical settling? Well, maybe we can go design around that.

I

The other thing is stats and there's earlier right. So a couple of key guys around back to packet processing and that's just 8 X 86 set together and set. We want better stats on VPP and then Allah chimed in and Dave Barak out of the key guy for VPP. You invented the whole chimed in and said what should I go. Do should I go and collect stats, or do you want me to forward packets, I.

I

Want both know so I can either spend a cycle on collecting something or I'm spending cycles on forwarding something and so yeah I think they came up with an architecture that allows for rapid collection of stats. But well there is a price that you pay and the question is also: how often do you want to go and collect that and in many systems yeah, you can poll the system every say one to three seconds, but it's all the same, because heart were only updates.

I

The counter every 30 seconds and that's the trade-off and what you see in core routing is we're doing that right off of you go fast, you have shitloads of table and and memory, but you might compromise on the statistics that you gather because again you can't have it all. If you want ultimate stats, then you might want to go and compromise somewhere else and people build dedicated machinery to go and do top talkers and whatever. But again we pay for that with either less memory on chip or less 14 tacit e.

I

So there is trade-offs and I just put some examples from classic SP. Where we've been optimizing around fit table sizes stats, flexibility, qsr buffering. So there is a shitload of memory on the chip arranged in eventually hire a case, but we said well, we don't deploy to many of these systems, so we're okay to go and maybe not go that optimal on power, whereas in the data center environment on the right hand, side it's all around power and bandwidth and speed, and these guys eventually don't even want large buffers, because large buffers mean delay.

I

Do they have large routing tables eventually not, and what we see right now is kind of more people trying to play in the center of that, so that you can go and sell the same box to multiple people all at the same time and kind of yeah.

J

I

With our trade off, but at earlier conversation today, which was exact and can we have it all yeah would be nice, but that's the spherical thing that we might not get maybe from Spencer.

I

John I think you're gonna go take us a little bit through why pipelines yeah.

E

So yeah here's the cartoon that I promised earlier about a pipeline forwarding architecture and it's you know like I, said it's a cartoon and the important piece of takeaway here is that when a packet comes in some of it goes north and some of it goes south, and you know the two really only meet when you get to the the output port. So so you've got your your first end. Bites they go up towards the forwarding complex you've got the rest of you know. Your your whole thing goes into your.

E

You know large pile of cheap ram.

E

You know you you buffer it towards your egress, and maybe you do another look up on the egress and it goes back out onto the wire, the header you're, those those first end bites get processed. You have, oh, you know some large number of stages and potentially a large number of cores each one of which has a large number of stages.

E

When you're scheduling all these cores, it's really nice to be able to to complete. You know the forwarding tasks in deterministic time, which means that doing looping and and things like that is undesirable and.

E

Notably, when your headers get too big that it drives real cost, because you know, then you know you, you really are that that all is l1 cache, and you know if you tell me that I need to have twice as much l1 cache. Well, Frank just talked about that. On the other hand, the data goes through Ram, which looks a lot like commodity Ram and acts a lot like it too and is slow like it too.

E

So I think compared to four years ago, when this was written. Probably a lot more people are used to thinking about pipelines, because you can't turn around without I mean everybody all of you have like. You know, 16 cores that you're reading your email on right now and you can't do anything unless you think about how to break a task down.

E

So you know very much so in packet forwarding what you want to do is you. You want to break down the forwarding tasks into a bunch of pipeline stages, each one of which ideally provides its output to the next one, and you know moves on to the next packet, and you know, of course you can make these things arbitrarily complex, but see previous discussion and yeah, basically anywhere anytime, that you have to make a decision in one of these pipeline stages or you don't know how many times you're going to have to recur.

E

That has some again real cost of forwarding time come.

I

To your point, writes the last one: if you have one label that can have multiple meanings, so you need to do another lookup. If you do that, reconsider that right. So that's I, think one of these, these things, where you give these guys a really hard time, yeah.

E

Exactly right, um Thanks, okay, so yeah things that are things that are fully programmed are slow. Things that are fully set in silicon are fast but yeah. This may pretty much reiterate previous slides, Frank. Well,.

I

I think it's an iteration of what we had before right.

I

So do we have to go and drill into that again. I think I know.

E

I

Don't let go past at times.

E

Yeah so I think this is worth spending a few minutes on, though, which is that typically there's a fixed header vocabulary. So you know at some point early on in the pipeline. You've got a thing that that knows how to crack the header up into you know it's fields, and you know so, and they're they're.

E

You know built to existing protocols, of course, so as protocols that that you know look like existing protocols are great as long as they sort of have semantics like existing protocols and I, think we come back to this again, a little bit later.

E

E

So yeah I think the only point I want to make to this one is that it's it's frustrating sometimes too and and I've been frustrated by being on the other end of this conversation to say: oh I've got this great idea of you know something that I can do in the forwarding plane and and to hear oh well, that will work, because your router fubar, you know, won't do it, and sometimes the answer is right.

E

It's like you know, take you know if, if we're out of food were I, can't do that, maybe it's time to take all those drag them to the dumpster and replace um you know you idiot, but there's also this this this large category of things where it's like.

E

No that's the wrong answer, because either you know that wasn't a design error that was a you know, informed trade-off and you're gonna find a lot more characteristics just like that in the market and by the way they're not going to go away, because if you want that you know limitation to go away, you want to be able to access.

E

You know infinitely deep into the packet you're gonna have to pay more than or not you, but everybody else in the whole entire world is going to have to pay more than they turn out to be willing to pay.

E

You know that's again in and when I say pay I mean in terms of currency, of course, but I also mean in terms of you know, power, heat, etc, etc, etc. Okay, so so that's this sort of setup- and this is again before.

K

We go there. Oh boy, this was a big, dumb, Danberg language. By the way, and to be honest, I was thinking and to say where we, where should we start it's?

K

One of the promise that we have is that our control plane are coming with restrictions because of our system architecture. If we could solve some stuff in the control plane, we would not need some of the characteristics in the hardware, for example, if you would have a good control plane that we'll be doing you know good traffic engineering, you don't need as the buffers you can save that space. You know for other functionalities instead of the buffers.

K

So then the other part is how the vendors are organizing the packet forwarding in fixed ways where there are some flexibility, because it ties to their business models. That is another problem where they are not willing to reuse the same trips that give it to the developers that they can organize the tables dynamically based on the applications that you want to run. So you have the capability to size.

K

The memory in different you know in tables, but in inside the ASIC in in different sizes that you need, but none of the vendors are really willing to expose that unless you're working directly with the ACF render and you have the access to those api. So that would be also in those some of the things that could help you I bet.

E

That sounds more like a business problem than an architectural problem. Well,.

K

So it's a little bit of both because there they are restricting you because of the business problem, but you you have a hard time pushing it for speaking from first experience working for a major vendor.

K

So that's you know that in in the past sewed, so that it's a little bit, that also has to be locked into it. I.

I

Think some of the constraints still apply right, so I agree.

J

There somehow apply.

I

But because you have an end stage pipeline and your processing requires n plus-1 stages, there is no way in hell. We can fit that and equally well, if you want to have unless.

J

You Lisa Ritter.

G

I

That we we pass along with the instruction through the purloined.

K

I

We do there are some trade-offs. You need more memory to best pass through the pipeline. I can't pass it through. Then. What do you do right? It's a hard constraint, I.

K

Agree that there are some certain where you cannot do more, but we could do more with what we have today when we didn't have some of those which are mainly business restrictions imposed on to us so on some of those Airport yeah. So that's and so.

E

I think you're saying that, like not all smart people are the ones who are sitting inside the vendors and have access to which is B.

K

People don't really shut you down and saying. No, because we cannot sell this, it will lower our revenue. It will literally come to that.

L

Whisker menu quantify forever you're coming with the new protocol designs may heavily impact the cost as a heart. Good I do agree.

J

Use the context.

L

Of the pipeline architecture design, however, for various our mothers in restrict your forwarding that your forwarding plane it's a network, not a processor, based to give away some other elements around it. That may not be the case.

E

Yeah I I mean part of what I've been trying to motivate. Here, though, is that the those pipeline architectures are probably not going to go away because they're very cost-effective I mean this is kind of the you're being cheap, no I'm, being realistic. Kind of argument and I can only sell what people want to buy and they buy what they need and what they need in certain cases. Is you know if you go back to two francs? You know cool circles, you know is sometimes you know, density, speed and and BTUs, but.

I

I think to a point, so the pipelines are just one example of multistage and there are certain constraints that apply to pipelines that might not apply to other multistage architectures like if you have multi stage with run to completion. You don't necessarily need a past memory around for the individual steps, so there is trade-offs among that I think. We pick one example forward in architecture, because it's quite well used across the industry and maybe I think from a considerations perspective. Most of them still apply. Maybe we'd go through the the considerations and then yeah.

M

Marianne would.

I

That make sense, because I think there are some stuff that you you want to go and implicitly race here or what I'm feeling that is in the next slides. That's on yeah.

E

That's that sounds fine, I mean I. Guess the we'll take your thing also, of course, but I want to just jump in and say you know to some extent what I'm trying to you know, at least from my point of view or trying to describe, is like a bottleneck, and you know it's it's useful to analyze bottlenecks and you guys can fight over who talks next.

L

Actually, what they wanted to get to is that, despite a multitude of protocols that may be needed to be implemented onto the forwarding plane, the basic elements that need to be executed are quite clear and I. Based on my experience on at least three different designs that three different major network equipment, vendors. So from his perspective, I mean you're. Basically, gonna have a such an election and I.

J

L

It is basically how the Terry mystic you may be able to do it in a certain memory bounce it's important if you go further to more fancy type of searches when when several issues are coming and the discussion, it's also very particular per architecture, type design.

E

Right now, I think they were in violent agreement. I wanted.

G

To get back to the point from then about, you know exposing lower layers of the platform, so so I mean the lower layer of the platform you're, exposing the more expensive it becomes to support that right. So that's basically I think. One of the reasons why you know there have been various attempts of API is another in the past. You know the. How do you consume it right? Most, most customers really come with requirements or functionality at a higher layer, and you know that's.

G

Why also thinking that the programmability, like you know some programming language before whatever it would be, ultimately gives you another way as a customer to approach the vendors right. It's not specific to particular vendors chip itself, but you can go and say this is what I want to run. Show me that it does I, don't know 400 million packets. You know what whatever performance you need, and that goes across the vendors and that's at least you know in another stage of engaging with the vendor, by being able to do that.

G

K

You go back to your to Frank's. It was back this one, so you have the SP style and the DC style, and what I'm trying to say is that in the ASIC you have memory and it's being laid out in fixed tables. You know that are using that memory on the ASIC. If you would have better control of those table, layouts that you can dynamically change it. You know, based on the application you would have.

K

You know like less of a trade-offs in there and I would even say that you could remove some of the buffers if you had a better control pane, which would, when you remove the restrictions of the router system, architecture that you have you know described at the very beginning.

K

So some of the you know some of the vendors basic vendors are trying to give you more flexibility in that way how you can lay out in order to get some of you know better the trade-offs where some of the you know, but it's very hard to get to that layer and when you are talking as a protocol designer what you're trying to do, you are seeing you're getting hit by those restrictions. I mean.

G

In the existing model, where you're only coming you know with- and you know, solution requirements, it's basically that the product already has to be in the different market spaces. And then you know the vendor will figure out how to reconfigure it accordingly, but other than that I think it's very going to be very hard. So.

C

If I could just do a cheering rod for a second, so so, basically we are to slide number 30 and we're kind of hoping to talk about things, to keep in mind when designing a new protocol. I think we are kind of assuming that people will continue to approve routing and continue to improve routers, but we also wanted to talk about what people are supposed to be doing in the meantime and then includes people that are not working or routers.

C

So, like I, say I'd like to let you folks finish up your presentation, probably in the next 20 or 30 minutes. Does it seem about right and then the other thing worse. Of course, this is a working session on a watching session. So that's our opportunity to go crazy when that's over.

E

Thanks yeah we're kind of in the homestretch of this deck, so.

E

This is hopefully okay. First of all, this is, you know, I am 100% sure, not a complete list, but it's trying to take actually some of the you know one of the earlier questions or comments rather was there. There are a certain. You know: universal truths, that people who have done this, a bunch of times have have observed, and this is kind of trying to distill them down and present them as a is a set of almost aphorisms and.

I

I'm John I think this is a very valid discussion that we have right now, with extension, headers and and using them extensively like what we do with say, for instance, I. um So, depending on what you have, how come? How deep here? Can you parse? What information goes? Where can my header go first, please so that I can parse it because I don't care about your header, so I think that's a very valid discussion.

I

The other thing is: that's the parsing part right and we're very aware of that that there's limits and people have been pushing these limits pretty far out so there's been. If we go a couple of years back, people were able to parse a couple of hundred bytes into the packet. People are not been using that capacity at that point, because extension headers didn't take off the way. The way we were thinking they were taken off and then people were suddenly pulling back again from a possibility perspective.

I

So that also happens if certain things don't get where they get, that it get implemented, but they don't get used. I think that's also something that we see as a dynamic, because vendors react to what's been built and what's been used and the other field is like 128 bits are really a challenge, so if we would be able to do lookups unless, if you design new things that use less bits, that would be good right. Yeah.

E

And I think sort of also a high order point that is kind of what you were saying as well is, overall, the one is way easier than a row of an I to some extent don't care what the constant factor is, but levan is really a pain.

E

So let's say: is this largely what we said already I think it is.

E

Yep I think this is you know again. This is just saying: headers that can be composed in arbitrary order is quite painful, as especially for certain architectures that have a fixed order of axiom. You know of units, so don't do that, inventing new kinds of headers, if you, if you invent a key that looks just like a key in a different kind of header and kind of sort of similar semantics.

E

So I can you know rejigger my lookup unit to do a slightly different kind of lookout, but it still look up on a thing that looks about the same. That's probably fine, taking a header layout- and you know saying: oh I- can abuse this into it in a new way because it has the right number of bits and but I'm by the way going to you know, change the semantics in some non-trivial way is you're not necessarily going to be able to reuse your existing hardware, also recycling, packets, sucks, it's the last bullet there.

E

This is Floyd edification, so don't panic what we're talking about? There is entropy identification and unfortunately we have done a terrible job at this and continue to not do a terrific job, but it would be really nice to do a better job at this in the future and to put entropy at a fixed offset from the beginning of the header.

E

I, don't really care what the offset is, but if it could be fixed, please and if I wouldn't have to go and like look at some, you know field and then it says, keep looking down and I keep looking down. And anyway, you get the point. Moso.

I

Design, something that doesn't require flows much better if you can go deal with packet, reordering and all that stuff at application level, then well, nobody would need to go and deal with flow identification in the hardware applications.

E

And transports, please fix this for us and makes.

I

It much simpler right. We don't need all that crazy stuff that we do for ECB. Today we could sue. Do packet by packet can run round robin easy. That makes it really hard because it requires us to go and do all kinds of shitty parsing right.

E

If we had one thing to go back and and do differently from you know, starting from the 70s, it would.

F

Tom Robert so flow leap. One ipv6 is not sufficient.

I

What do you use the flow label for again? Could you tell.

F

Me so identify flows for ecmp. We have it it's. There was expressly designed for this purpose. As far as I know, it's not it's being set on the internet. Now is it being used by these routers and it gives you exactly the properties you want. It's at a fixed location, easy to process. It's always in the Iceni pv6 header.

F

Most OS is that I know of Windows. Linux iOS will set it well. So.

E

F

In the comments.

E

If you could come to the mic, if you want to join the conversation I'd, you know that would be great. So the.

F

Question was, is it always said and I would point out, even if it's not set, you still have the IP addresses. So it's ecmp is an optimization, not a requirement. From that point. You.

E

Did you just say you CMP is an optimization, not a requirement.

N

That's not what we hear. Okay.

E

Yeah, no, that's! That's really not.

N

True, you can't use it for something as important as this. Okay.

F

But you can't trust any fields by definition other than the IP header. If you're going trying to go down into transport layer, you can't trust those either well.

N

You know what they like p-type is you know what the follow-on type is, and you know where the ports are and that's what people have been using for many many years and.

E

It's an ongoing problem. I mean it's not perfect, but but well we're highly motivated to get as close to perfect as we can so.

F

We just talked about header size so now, if I have to go down and find transport headers, that's gonna. So something has to give here. You, like Frank, said you can't have everything so I can't have an easier way to identify flows and yet short parsing buffer. So there's got to be a trade-off somewhere.

E

Right but but ecmp is not a is not a nice to have.

H

Okay, part Visia gandhi. Are we a great background and the history of the routing, and there is a paradigm shift in industry- I'm, not clear. If you guys, you can really spend a few minutes to see how that would really impact the future of IETF and the protocol machinery of IETF. It seems to me, okay,.

E

You know what so III would like to encourage you to hold that question for a few more slides, III am NOT going to talk about. You know, sort of future vision for the next few slides because I'd like to wrap this as Spencer requested once.

I

That discussion at the end, this.

E

Is the discussion I think that Spencer is asking for, but we're trying to get there well.

M

Convict with regard to packet reordering, it's not that long ago that I had a heated discussion with the vendor, who thought as a carrier great system where you had to choose between selective acknowledgment in TCP and the ability to process reorder packets. So.

C

And just just to make sure that I'm doing the right thing for everybody in the room, I'm, hoping that the discussion starts at the end of the session. If this sounds, if we can solve all the problems of router, routers and the way in forwarding and furcal interactions in this room by five o'clock, that would be awesome. But just in case we don't I also want to be talking with you all about how we can continue their discussion.

O

Element Kirchner just on the last point: ideally, yes, fluid identification and packet, three or fragment reordering from a chip perspective would be ideal. However, there are, if you look at the real end-to-end thing. I mean there are security considerations and you know stateful, firewalls and so on. So if chips drop that feature, others will have problems. So we can't optimize for chip level and forget that you know we are trying to do entrain stuff. Remember.

E

At the beginning, I said that everybody else has to do cheap, fast and good.

O

It's a very good. We would have everything for us, it would just defeat the purpose. It's.

E

A great point, a huge.

N

Elephant haven't got to which I hope you're gonna getcha, which is pure packet state, is that in your slide, deck.

E

You know I, don't think I do have something on that. Let let me just fly through the next things and then I we can sort of open it up, for you know be sure that we've we've checked off all our boxes, no pun intended so yeah check sums are expensive, don't use them just for fun, don't invent new ones just for fun.

E

This is not actually what Stuart was asking about. I, don't think I. Think Stuart was I believe that you're ringing up the point of you can have stayed in the forwarding plane or you can have stayed in the header yeah.

N

I mean the thing: that's really that was really really a problem with people coming to detonate, expecting we could do per packet count, our sequence number checking and those sorts of things, and you discover that actually, it's really really hard. Ok,.

E

That is kind of what this is is going on about then, which is that you know sort of traffic driven state is.

I

This wonderful.

E

Contentious right up there.

I

So this one yeah okay, so.

E

I

And well I think one one again: I hate to say that, because I love multicast, but from a hardware perspective, it's a load of effort and people put dedicated replication engines. It makes the chip much more complex and so I think how much replication you need. Consider that how much application we want at a particular point in time.

I

Consider that so it is a a real trade-off that people have to go and make, which is why, if you look back loads of people were blaming certain devices not to do proper, multicast yeah, it was a design decision right and they traded unicast capability, or maybe sometimes cost plus unicast for the multicast thing. So I think that's some of the background that happens there and if you feel forward to one more I, think that's a wrap-up thing right that we want to go to jointly. So.

G

This one cannot.

I

Just two more sentences and then I think it's free format anyway. So and we can't do everything at no cost. The other thing is and I think that's something that we might want to go inspire as a discussion topic is. Should we, when writing a draft and completing the draft, have some of these considerations that people think what their protocol should go do and what some of the assumptions is are should that be documented in the document and written down, so that not some of these things are not implicit but explicit.

I

So I assume that I can parse packets up to say 200 300, bytes, Steve I can do this. I can do that. Should we go and document that I think there is a problem with certain things are designed for a particular domain, and it might well be that today this domain exists tomorrow we don't know the future, so it could well be that this domain becomes something else.

I

It could become the bigeye internet, or did it come just kind of fall flat on its face, but I think what is niche today doesn't need to be niche tomorrow, but at least I think articulating.

I

The requirements that you have in mind for certain things from my perspective would make sense so that loads of these implicit assumptions are put on the table, especially when it comes to what the forwarding capacity should be be all about and I think that's one of the discussions that when we did, the prep calls with Spencer or our own team that they wanted to go see surface, which is why we put it here and now. It's all yours, sorry.

L

So, thank you, I'm, lucky to be the first one now insecure, going back to your previous slide regarding with keeping state information. But again it is a matter of Archie and it is architecture dependent. There are certain architecture already out for more than 10 years, for which this is not an issue. I would have been happy to see other elements being included like search types that you may have it.

L

These are directly in impending regular design, no matter what type of architecture you do have, so something like Brickley, favor, hashing or favor, exact match, favor prefix, matching versus generic packet classification and so and so on. Unfortunately, after staying in this presentation is like I slept for 15 years and I could have come up with the same type of discussion 15 years ago. That is not rushed.

I

And the deck was started in 2015 I, think yeah, well, loads of business, motherhood and apple pie, and they just wanted to go and cook us. I was cooked that up as a game, so I think that's a good point. What we can't do could do as a follow-up is maybe just focus something just on look up in the next go-around.

F

David black is.

I

Fluent in an entire kind of discussion, the.

P

Sliding hares and tunnels flew back way too flew by way too fast. We have increasing amounts of tunneling cap protocols in the IETF. Increasing design expectations that end cap, D cap and cross header processing on D cap will occur at line speed. Would you care to comment on what to do and what not to do in this space? You.

P

Last up with what's Dido I.

G

Think it's very generic right I mean I. Think where we're looking in the cases where we feel it's it's it's its worst and how much we can optimize it and if we think we can live with the optimization right. So if you have an ongoing chain of you know, D cap an end cap with the same thing. Obviously you can optimize that quite well, but because even the links doesn't change, you just need to figure out if you can flatten the processing to actually recognize that situation so but yeah in general.

G

You know all these encapsulations are typically very expensive and in you know, typically they either cost more gates or they cost a performance because you're doing things like recirculation.

K

They are mode on which one thing which I find would be useful in general for the protocol designers is to put in a document with do's and dont's. What is you know beneficial? What is not beneficial when you're doing a protocol design? We can talk about in system architectures.

K

You know what we have the problems, but if you would have lay out whether your restrictions and what will be recommendations from the hardware perspective for certain protocol adoption to be faster, that would be very helpful because the we can try to optimize one thing that we have as much as possible, but if you're adding a new header that requires new parsing and you want to do it fast. As you mentioned, you have to wait a couple of years and they've get being put into the you know.

I

Silence right so I'd rather have spelled out. What do you want, or the assumptions that you have as opposed to there is some generic rules from our perspective, I think you learn on. You can design the hardware around your needs and then your additional adder I can slap on without any effort. It just depends on what the underlying system is built towards and I. Think that's why I do believe that something that spells out the assumptions or requirements, it's much better than having something that well talks to spherical routers, but.

K

What about you, are you knowing what is possible today versus what will be in three years so.

C

So I should probably say something just for a second here, so I think that we're talking about yeah I think you had a perfect question for the next little section, which is where I'm hoping we're going to talk about is not how a some specific problems like that so issues. It stupid, specific problems that have been named here, but how we solve you know how we solve those problems or keep them from happening going forward.

C

You know you're right, you know when you're asking about is there are there recommendations that we can make so that people will quit tripping over their own shoelaces I? Think that's a conversation. I really want to have with the room so I'd like to let you know we could bring the queue and everything if you guys talk fast but I'd like to avoid to you, give these guys a chance to finish up their section and then turn over to the room for us to figure out what the next steps look like. I think.

H

We are focusing on a box called the router. We want to make this box look really nice air bigger, faster. That's not the point. The point is you have to look at the bigger picture. Like Frank already mentioned that telemetry, the partner is not only if a packet forwarding, my drone is only a small piece of it. Ok, so that is where Spencer. When we have time I, guess John, you said we are going to talk about that. We are making this box, which is a router look better, faster or John.

H

You put it very nicely more efficient. That's not that's! Only a small piece of the puzzle, I.

Q

Have a few points on the slice we just went through? Should I wait or should I go through him? Okay, Magus Allen says good agreement, alright. So in the slide 31, this is Rajee by the way so slide. 31 said parsing, 48 or 32 or 64. That's great, but 128-bit did we forget to send a memo to the chase. Ik folks that ipv6 addresses are actually 128 bits that has been so for last 20 years. So so, but.

G

Wait a second just because the ITF has decided that the Internet should do. This doesn't mean that a chip designer has to like it right there. No, of course,.

Q

Not but that it's it's that mean parsing, 128, bits and I feel so it seems like we still rooted in ipv4 lingo from 32 64 bits. So it is there still a merit in having to restrict I mean granted I. Get that the you know. We can't just keep parsing hundreds and hundreds and hundreds of bytes in a packet for obvious reasons, but there has to be at least a guideline. What's optimally minimal right as we go along the timeline?

Q

Okay, so second point now I'm thinking about this and then slide 37, we're saying wait heaping state or not keeping state on slide 37 in the forwarding. Node is better well, but that means keeping states somewhere else. So mostly now it is that's in the packet, but if you are keeping state in the packet, that means you're putting more information or highlighting the need to parse that packet, a bit more intelligently, which takes us back to the problem of parsing. Some scratching, my head on that and the last one and I'll pause is degrading.

Q

Multicast unicast I think what we're really saying degrading the ability to be replicating that has the world that is part of Martha cos, but nowadays we also do that with unique. Ask so am I correct to understand that when you're saying degrading Moloch has we really saying well degrade having to replicate the packets on the boss, maybe I'm not getting that right, but at least the sent the statement is making a scratch. My.

G

Head this, this is all from the perspective of keeping heart, we're simple right. Obviously, that's that's one end of the side and obviously I think we would all agree. We don't want a network where we don't have ipv6. So sorry, chip designer you have the pain there right, but basically ongoing. We should still consider this. You know poor chip, design effects into account right so before.

B

We before we keep going just a tight check. We have this room from five. We all need to go to the plenary at least lariat five, ten. So after the discussion, we still want to go through a couple more slides, so we can talk about what are the next steps? I did.

G

There they're also three more slides that I wanted to present. Okay.

B

So we want to leave enough time for that. It's 4:23 on my watch, so I would like to finish the extra slides. So we can have the discussion about what are the next steps, maybe starting in 25 minutes before 5:00, so they can have time for that. The bathroom and all this okay.

D

So I should just make a small announcement. We've just published our FCAT 5 or 7, which gives you 64-bit addresses for that protocol. Okay, it's meant as a joke, though. If you didn't get it but but go and read or see a table. Oh seven I would load. That was the intention all right. You guys or someone. We.

M

R

Benkei dark, so there are a couple of people talking.

E

R

Your whether it is a good idea to try and document some of these things, I think the answer is clearly: yes, whether that ends up being an RFC or BCP or just you know, a wiki page I, don't know that I have a strong preference on, but I think it's really great. If we can be clear about what your behavior is good for router and also what behavior protocols are sort of expecting from their routers.

R

So if we, if we talk about what we do and what we expect, then we can figure out if those match up and if not what we could change and maybe.

G

That's a discussion for the end in terms of how and what could we do as a follow-up from this as Spencer said well,.

R

Yeah definitely I have another point. So there's a slide that was talking about how avoiding flow identification and your being able to accept packet and fragment reordering would help the hardware and I guess my sort of question Michael. My question was sort of who would need to coordinate to make a reality and we have people that are designing. Quick and they're saying you know, we don't need. Are your packets to arrive in order and we can build into quick what you know we do we order that.

R

But then we have people coming up and say: oh well, I have my stateful firewall.

M

R

You know that would not be so good if the stateful firewall is seeing things out of order, like other everything, is on path that need to look at this. I don't have a great picture of of who would need to like how many people would need to change in order to make that happen, that we can tolerate the reordering.

S

Hi Tony Lee some router vendor one of many I'd like to respond to the comment about 128 bits have, after being there, we did take the comment about 128 bits to the ASIC folks and they came back with a different comment, which was no 128. Bits are bad and the IETF chose not to listen. Okay, we specifically asked for variable length addresses and the IETF said no blank blank.

S

Okay, if you're not gonna, listen to the ASIC folks, then don't expect the ASIC folks to listen to you.

T

U

Lot of us have been doing this stuff for 20 plus years, so we should learn that kind of rule of thumb. You want to build it. Well, you want to design it. Well, you respect clear boundaries, you do a polish ability from policies and the we are going like a pendulum it. It reminds discussion about fully centralized versus fully distributed.

U

We decide, let's put all the state in the consult and we built our CPT and then hey soft state, heals our contra plane plus remove all the state, and then we build number of technology building now right and suddenly our gosh anything that required state natural, doesn't work.

U

Multicast doesn't work, let's go back, and can we find something in the middle by Carter rating that actually does work, they're very good examples of things that work really well, LG, VPNs beautiful right, so they did work, they do work, and so there are good precedents of finding right balance between what goes into control, plane versus what's going into forwarding, and we should learn on our mistakes and think that we did well can.

G

I maybe cut the line after dino, so then just throw show three more slides and maybe that expands the got a bit more and then we came back sure.

N

Brian various people have suggested that we write down basically a forwarding specification or a specification for how this works. Well. My first IETF was I 80s 16 in 1990 and I went there because we were just starting to work on and the rewrite of Rooter requirements. No one has ever successfully written that down. The world is orders of magnitude, more complex, the optimizations and the economics are orders of magnitude more complex. All we can do is to do some tutorial, but never more than tutorial. I.

V

Have a draw or comment which is a little bit beyond just about the packet forwarding. One thing: that's very kind of strange to me coming to this meeting was I was expecting something to tell me how we're going to do forwarding in the future and how we're going to get things better. I understand that we want to have simplicity and I understand.

V

We want to have performance, but it seemed strange to me that we are asked to bend over backwards to fit the hardware I think the hardware should go where we want it to go, not the other way around. So the.

E

Whole point of this talk.

V

E

To repeat Stewart's comment: it's.

V

Not only that we've been talking about trade-offs, but then we're airing over over simplicity about what we should be done. Stewart.

G

Could we first and get that later? Thank you, Tony.

S

Lee arista, we one of you care to comment about SR v6.

W

This is Dino.

S

W

Of an idealistic comment, the original internet architecture was supposed to make hosts robust and more complex than the inner network. Routers we're supposed to move packets fast and do nothing else. That's the way we started building router companies but turns out router companies didn't want to be commoditized, so they had to put additional features in. Do you realize most, the real estate on any die on any router is a huge tee comp for access lists. The access list table is larger than the FIB table.

W

What the hell are we doing why, in the middle of the network, we have to check if a package to be dropped or forward based on policy shouldn't it be done in the control plane shouldn't that be done in the edge. So the the problem is, if you guys want to build fast routers and make these hard trade-offs, you have to can't keep changing the problem set if you want to keep changing the problem set, build a general-purpose programmable forwarding engine.

W

But if you want to build a super high ASIC, you can't keep changing or predicting what the future is going to be, because then it's going to take five years to build this new functionality. Sorry.

G

I was just cutting the line just to throw in three more slides to hopefully actually also relate a little bit more to what the discussion started to be about and I think we learned life as a chip design as miserable and what fronting the last part was reflecting back a little bit to you know, please take a little bit of that pain as the protocol designer now I wanted to basically try to create some more insight into that connection between the forwarding plane and what we're doing at the IETF.

G

One interpretation of the next slide I'm going to give one example, the simple interpretation, obviously it couldn't work and it had to be painful it because it was multicast like what said on the last slide. My perspective is actually a little bit different. It was basically perfect. Combination of you know bad code and bad IETF specs. That basically was not taking into account.

G

You know the need to document forwarding pain problems in a way that really they, you know, must be implemented correctly, and- and this is basically my story from the beginning of 2000s, when we were using the router alert mechanism, anybody who doesn't know what router alert is. Please leave the room, No, okay, fine.

G

So what we did was a reliable multicast protocol called PGM and basically, that was using router assist in message and signaling packets to basically let the router help the end-to-end communication of the transport protocol by imploding, necks or caching and retransmitting packets. You could say it's similar to ICN just a decade earlier, but be that as it may, that was a you know, in our opinion, a perfectly well working solution.

G

Unfortunately, when we started to deploy it in customer networks, what happened was that this router alert packets were perfectly preceded by the routers in the network that understood PGM as they should and they were designed to have the performance needed, unfortunately, also a lot of other routers that had never heard the word PGM in their code at all. We're also punting these packets because well it's obviously a router alert packet. So let me punt this to the you know. Aforementioned remember my first slide at M CPU. You know that can just may be software forward.

G

Let's say 1,000 packets as opposed to 500,000 as the ASIC at that point in time and yeah. Obviously they got completely overloaded and killed, because the only thing they were doing is looking the packet and saying what router alert next protocol PGM. What the heck is that I have no idea simply, let's forward it in in software, and so what happened was that the specifications that we have in the ITF were basically not making it clear that you really, you know to get workable, router alert.

G

You must only punt the packets that are of interest to you and that's not just any router alert packet, but it's basically the packets of protocols. Next protocol. There is also a particular value field that it could have different content. You know the combinations that you know you're going to do something with it. You know in your software control, plane forwarding plane.

G

However, you want to call it, and everything else just continue to forward in Hardware would that additional differentiator to you know send packets from the hardware forwarding plane to the software forwarding plane being possible on all platforms in 2000 I.

G

Don't know, definitely on a lot of platforms where it failed to do so, but today, obviously it's a total no-go, but still we don't have RFC's that mandate, this level of requirement against the behavior of a functionality, because, in general my perception is Oh were not about to specifies internal of the notes, so we're always trying to wriggle our way around it. I was giving here an italic one way on how I could try to wiggle my self around, but obviously it would be a lot easier if we had more freedom in even normative rfcs.

G

To make specific, you know requirements that today we might consider to be internal only, even though you know I think it's very nicely externally be visible, that you know. Oh, your packet throughput went from 500 thousand per second to 1,000 per second right. That is externally visible right. So this is basically my example. Slide of you know, it's not only a problem of the hardware, but there is also the problem of bad code and that ITF specification.

E

Jump in produced a second tour list and say that I think what I hear you saying is that we have, for a long time, had a religion of like we're not going to mandate architecture. It's not our business right so and you know sort of we earlier had dancing almost the opposite of saying. You know just give me the cookbook and tell me how to fit into your. You know, architecture and I'll. Do it yeah, but.

G

E

G

E

And and I and I think that you know and and Frank said something a little bit different again and I think what I'm hearing is we sort of need to. You know, lose a little bit over the religion and we don't. We still should not be mandating architecture, but maybe it's okay to acknowledge that architecture exists. Let.

G

Me, let me go to two most lights, because you know elbow was just giving me a time limit so that were at least through the slides from oh well. So, basically, you know I wanted to give to counter points right.

G

The one counter point is exactly through the need for speed in the forwarding plane that we see a trend of simpler, fast path and control, plane right, so basically slow path, all these difficult packets and then we're going forward saying please let us have less difficult packets that we can't process in the fast path, because you know the slow path, something panting it up to control planes if you're processing it is evermore so useless, like it's completely useless.

G

Let's say in data center switches and you know I think in our evolution we've seen, you know, hope, I hope, headers fragmentation, reassembly we've been trying to push back on that. It's actually not only packets that by themselves need to be software forwarded. If the hardware can't do it, but it's also what you know, at least in the IP multicast work.

G

We've come to call data triggered events, which is basically things happening in the forwarding plane like special packets, that I don't need to see in the control plane, but where I need to get event, notifications to do control, plane actions and so which, basically, you know, I've, given some more example there in multicast, we work something like 15 years on evolving from something that you know, as I initially said, was implemented on a Sun workstation of a researcher built the mbone up to basically, you know our last protocols like by your pin that wasn't having any data triggered events anymore, just to fit the protocol evolution perfectly with the requirements of hardware, and you know allowing us to work in the smallest possible CPU, but quite frankly, we're glad that we did it from the you know, business perspective, but definitely it didn't necessarily result in the best user operable protocols right.

G

So we basically were doing compromises for what we ultimately set out to achieve initially, obviously way beyond what we can talk about here.

G

So how about intelligent, fast path, right, because I think as people who built this hardware, we don't only want to basically give you the fastest, you know least intelligent Artois, but we're always looking for things where the hardware intelligence can add value to the solutions, and you know I think, there's a lot more things we can do with the latest generations of NP use, and so what I mentioned these two topics of you know the IOM I think that's fairly well known.

G

We also have a draft for where we basically put into NP you in our prototypes for basically a lightweight equivalent of RSVP to basically establish forwarding state at the speed of NP U, as opposed to at the speed of control plane, and that actually is a totally different protocol design. Because when you look at it, you need to change the called to be lightweight enough to be processed by the NP.

G

You similar to the I to the IOM stuff, where you can't have a lot of you know, acknowledgement, exchanges or so so you know, protocols might actually start running. You know in accelerated forwarding plane, maybe takes a decade or so we don't know right. So how do we deal with the conflict of you know? On the one hand, side we heard ninety minutes about, you know make it simpler so that it can be cheap and fast.

G

On the other hand, how do we open up the opportunity to you know, explore more intelligent forwarding, planes and leverage? You know their speed and and flexibility the the last point may be. You know there was a really nice. You know research project that I solve from Berkeley two years ago and they were actually looking at exactly what we always fear to be really bad, so, let's say make learning in 802 dot, one in bridging right totally always data triggered events.

G

Data packets, come along, you'll, learn a dressing from it right and there we're taking it forward and saying hey. We can do this in NP. You know they program prototype in P for right. Basically, if you can do it now in hardware and it works at line rate speed, it's really a great simplification of things. So we'll really need to find to counter. You know these two sides of the equation of how we deal with forwarding thing.

G

So line is open again for everything.

C

So we're going back to other presentation, so so this is where we were when we came into, though when we came into the room this morning this afternoon, which is basically the chairs and and warrant okie with people around here that we had, you know kind of here's, here's here's problems that we know we've tripped on and then from thank and Charlotte or less we. Oh, he said. Oh here's more gaiden ears, guidance that we we are giving.

C

We did their spill Federal Work, where he thought the problems were, and he said basically anything we did in the 1990s is going to be a problem which I thought was hilarious enough to throw inside by itself, but you can kind of see where this is sitting right now. My next slide, please.

C

So if we're looking at kind of the silver you know implications were here for and mix my place, we could do a lot of things. I guess I should ask first. Does anybody in here want to work on this? More? Could you come.

C

So, like you know my question, my question was: basically: we could publish minutes from this secession and thank everyone because we've solved all the problems. We could publish informational report from this session. We could request a mailing list to continue discussion. We could shorter work on recommendations or BCPs. What Benjamin mentioned looking for this kind of thing? Can we talk to approval and expectations and I'm very interested in hearing other suggestions from you all.

W

This is Dina. I just want to make two points about multicast in going in reverse order. The data driven events thing toilets. You were there ever so you know the history. It turns out. We wanted to signal sources in the network by using a control plane, and this is the Tony Lee's point. We asked the IETF to have a request to send message and it was not in the current way that multicast service model was defined, so it was said no, but we actually did get something out of it that we could have used.

W

We could have hacked-up, because there was this thing called the RSVP path message, because we needed to signal from the source to do reservations, but not to do multicast state, so interesting change of events there and then the one slide. There was a reference. The multicast needs to be degraded to unicast. It turns out, if you're a fit programmer. The hardest thing to do is to program 18 line cards and make them in sync in the shortest amount of time. You need multicast on your your switch fabric or there's no way.

W

You can scale that now I know that's different than multicast replication on the network, but if you're going to put it in your switch fabric, you already have most of the hard work done on your internal switch fabric.

K

Then I agree with Dino's comment on multicast and one other thing is also it's a replication to Dino's many mentioning that today we have more space on an ASIC for Ackles than we have for forwarding, and this is coming from our bad network management practices.

K

Many of the network management issues that could be solved in a different way. We are trying to solve in protocols which are then impacting you know, essentially our basic designs. There is really no need for the operator to say: oh if alcohol is in there, it should be never deleted and it just grows and grows and grows, and they just put it indiscriminately across the network, and many therefore note because they said it's better- that it's there, then it's not. That is there and there is a lack of network management tools. I.

W

Mean this is so ridiculous. I don't want Dean to talk to me, so we have to install an ACL in a router in Japan and by the way, we're doing that because Dean might roam to Japan so that router needs it just in case. How well do we do roaming on the Internet.

H

C

H

Kayani my earlier question: it's the vision of IETF and if the life goes on in ITF, using the old mindset today, if you really look at the application layer or networking infrastructure, the framework is widely known and deployed it's being deployed. So there there is a vision there that the actual function, which reflects the reality of a business value in this case for IETF the business value, is how fast you can move package yeah. How fast you can move packets.

H

So that's when you move from a PMF, PMF user, like a physical box today or we enough there, an effete he'll be enough, and now we have C enough. We she's containerized natural function in the world of CNF. You don't have to deal with PMF issues. This is this is my maybe you learning from the routing switching coming to n Fe, and now we are talking about CNS.

H

My question for IETF is okay. If the application layer, networking infrastructure is well designed, well established and it's been deployed, we can basically use the same concept and apply it to layer two and three, a layer, one, the networking. Why not? We can indeed take the core function, which is packet forwarding, and then we take everything else is not packet forwarding. It doesn't have to be Sdn like centralized controller, you can have that distributed well, distributed functionality of something like envoy in a context of like service mesh.

H

As a matter of fact, I don't know if Eddie's here or not, there is a an effort going on in industry that how you take the IETF protocols as tools and apply to this new concept, I want the people that were talking like Frank and John and Hollis to talk about this. How that would impact the future or IETF, not necessarily from a rafting perspective, the overall IETF political machinery, and how does connect go today to the next? Perhaps this serve this purpose of the industry for the next decade, or so so.

N

Another elephants and I think it may be a reference to that. In the previous comments, the co-prime is people want to do virtualization down at these sorts of layers and do the sort of things that we do in the sort of virtual moving applications around where I'll remind people that holographic networking is currently expected to need a terabit per second per user, and if you don't get the latency way way way down, the user feels sick.

N

So you know we can't do some of the future applications unless we have simple, cheap, fast, effective, highly optimized forwarding, and that cannot be done with some of the features that people sometimes think are easy to add. I should.

C

Mention that we have about two more minutes in this session, so if you the last two guys if the line could be quick, that would be awesome. David.

P

Black sure I got to try to say some of the questions on your side on the slide Spencer, so I'm I'm from Transport I, don't work directly on routing, but I use. It I'd like to believe that the protocol design work I do has some cognizance of what it takes to make her out, make routing effective, so I'm, very interested in the output of this session and I'd encourage to go beyond the beyond the what was said minutes to an informational report.

P

Summing up things, the rest of us can use BCPs would be wonderful, I'm, not holding my breath. If you want a mailing list, that's nice I can expect to sign up because I want to see the output I, don't care I, don't care to be one of those who make who who makes the sausage. So there's there's there's some quick stuff that you want.

F

Tom Robert, so on one of the earlier slides there was a really nice taxonomy of various routers and they were everything from IOT devices to too high on I think somewhere in the later slides, we were more secured towards high-end when we start talking about header length, even checksum. The problem that we face and David might have alluded to this as a like- a transport protocol designer. If we want something to work in the general-purpose Internet, we have to design for the least common denominator.

F

So what is the least common denominator of all those and that really pushed us down to how much functionality and evolution we can have so a way forward really has to be that least common denominator problem has to be attacked, I would say.

N

The middle, its micro package, the tiniest packet, you can have the fewest fields, zero state and that's what you have to design for everything else we put in there, because people didn't allow us to do that.

F

Well, but that that achieves at least common denominator, and that does not get us forward motion on on in the internet we would be would be stuck with ipv4 stuff. We didn't have the ability.

G

So, just maybe to get back to David stuff right I mean we were very centric on forwarding and basically you know the qsr things in the hardware that you know relate to the transport functions. You just add a zero to any time that you have how long it takes to get things done it over in the forwarding plane, because that's our experience how long it takes to get a QAM or ezn or any of these other qsr things into a network as soon as you get out of CPU space. Okay,.

C

I'd like to thank you, everybody for coming and I'd like to thank especially Frank and John, and tourists for putting together their work, the information they put together for this and thank my co-chairs. We do have a we are going to have a survey on this and we need to adjust a couple of questions on it, so it will be right tomorrow, they're just you know, we'll hook up the note to the IEEE IETF 104 attendee list, and- and we will let you know when it's time to jump on that. But but thank you.

C

Thank you all for being here and I'm going to turn over to our area director.

T

And you know one quick question: just as people are leaving, so this was sort of the first in a deep dive series or proposed sort of deep dive series of topics and when people are filling in the survey, I think it would be useful for people to sort of keep in mind.

T

If this format or the sort of take a topic and investigated in depth is something that's useful and when you're filling in the survey sort of try and answer that and figure out, if more things like this would be useful and we're gonna ask me what topics they might be. Can.

N

I request it scheduled in open time so that all areas can attend, and that was one of the really good things about today. Yeah.

T

There's gonna be much discussion on scheduling, I suspect thanks. Everyone.