DASH Behavioral Model WG, 10 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: DASH Behavioral Model WG Nov 10 2022

Description

Continued the discussion of how much underlay support is required in the bmv2 implementation to ensure consistent test-cases for hardware or bmv2

Default routing out same port. We only need to add underlay MAC support

Filed issue #50 against sonic-dash-hld

A

Find the minimum list for the routing or for the underlay that is required, so this list also will be used in the test cases, and we will also implement this in bmv2. What do you think about this.

B

B

And go a little bit more into details and see what is missing in terms of routing Mac resolution and document that then it will be much more helpful.

C

I think Marion's suggestion is minimal. Routing is input, equals output Port. If you try to configure anything else, it won't do it.

D

Yeah, but you right so the only way you can configure something else is that you won't have a default route on both ports.

D

If you want to have like some specific prefixes and then you expect the port to go to a expect the packet with a particular underlay destination, to go to a defined Port based on your configuration now, it won't work and I. Don't think we need to support this foreign.

E

You mentioned default route or I mean. Are we just trying to build the ethernet header and that's it or more than that.

D

Yeah basically well.

E

I mean you can do uh a prefix lookup to know where it's going and also what the ethernet area is, or you can just limit to ethernet header, but I, don't know no for me, it's just basic writing you look. The prefix gives you the port and the destination Mac and yeah.

D

But it's not that simple and uh you want to map this properly to say apis either you do the full mapping to the prefix to the next drop to the neighbor uh and then I don't know FDB whatever, or we only say that we have an assumption that two ports are connected to a gateway and the only thing they have is is a default gateway and we don't care which Port the packet egresses a lot as long as the this nation Mac is a proper one.

E

Okay, so what you mentioned first is what the Intel test cases do and but you're right there's a choice. There.

A

So but Marion just to be clear here so anyway, so here is even with this minimal support. If that Port will be, you know sent back from the same interface right so default route. I understand it. It's something that complicated in the bmv2 this moment and doesn't work, but next hope neighbor is something. That's uh can also be added.

C

Right any any places walking distance. If you have the time, is it worth the time.

C

Is it worth the time to.

B

Do that um we uh definitely, you know, want to lose the same test cases across bmv2 and Hardware. That is the only goal based on that. You know if there is a small amount of work that is needed, so we don't have the change the test cases um for bmb2 and uh Hardware that everyone can use uh it will. It will be really helpful right.

C

I I think Marion is saying volunteer to do that, work that small amount of work can show them how small amount of work it is and maybe he'll consider it.

D

Yeah, it's like I I'm, not the maintainer. Anyone can contribute this code, but I think it's useless. That's my point.

D

Can you give me, can you give me a test case for underlay that tests something except how the egress packet will look like.

D

Which is done by the overlay pipeline like what exactly do you want to test? Accept? um I, agree that we need to be able to accept. The API calls like what would be the past criteria for the underlay testing for dpu.

A

So, and- and the first point so we want to have this configuration in the test case to configure the routing just to you know, be able to run the same. This case on the hardwarezone, any modification at all.

A

This is important, but but once it is, there is a traffic. So probably there is few points where we test it, but I need to check it. So I can send it to you, uh but I think maybe it can be like a Basics uh tests can be there then just verify the throating also passes. You know on your Hardware.

D

Like, for instance, assuming we have the intended configuration where two ports are configured to to have a default route.

A

Do you mean with the Vault default routes or with the case like in the bmv2, where we send the packet from one port.

D

Now I'm talking about the real config right, the real configuration is that you have a dpu plugged into two switches that will advertise default routes to a dpu. So how will you test this traffic.

A

uh Well, so I think that we do not test this currently.

D

A

The first thing for the underlay, as I said, is to have the configuration of the routing correctly that in this case, that this case will work on the bmv2 and the hotter. So we don't want to have the if else or modification of this case to be able to perform it on on the hardware. This is the first thing, so we do not do in the PDF test cases that we have in the dash which uh assumed testing called overlay right.

A

So we don't do too much underlay testing there. So for the underlying testing, there is a different set of the PTF test cases on the side, repo, which you can take it and and run it right. Yeah.

D

So there's a bunch of them which, which has like some assumptions about topology, that you have uplinks downlinks they're different, that.

B

So we have ordered that um Marianne to for dash uh some of the test cases and the device that has fewer ports yeah. A lot of work has gone.

A

Yeah, so so that is.

D

Why? Because really you have you almost have a like? You have L3 lag, you have ecmp or any IP package.

F

I guess one way to think about this is uh one way to think about. Let's say uh bloodable. Let's say you configure the default problem, all that right and the in the in the implementation, the BMV to implementation. We just ignore all that we can continue to do what we do today, which is uh just send it back out on the same port where the packet arrived, I, think I think the question I think what Marion is trying to ask is there is the test would still work right. Can.

E

We Define, if looking for.

F

Is the packet coming up you're.

E

All talking about uh test cases, the final requirements you get bmv2. You also have vendor specific floor code right, not using vmv2, necessary I mean we need to have. What do you expect? What is the behavior? You expect Define.

F

That behavior is uh I, think the behavior is clear that we have to have PSI the basic side routing, but I think what we're trying to do here is from the BMV to perspective. What needs to be what minimal thing needs to be implemented.

C

My question is the proposal here, so my question is: go ahead.

F

Yeah, so just my question was so if the intention is to have the same test case, uh to be able to run for both the bmv2 and Hardware. So let's say the test case actually goes and configures the routing and all that. But the implementation of the BMV to implementation just simply ignores all that right, and it just sends out the packet back out on the same interface that it's received on the test would still work right because.

F

Is saying because if it's a default route, if you assume that the configuration is always a default route, that's pointing to both the interfaces right, which is what is the configuration requirement if you're passing from Microsoft, then uh then the test would still work and the only Gap that I see is the underlay Mac address that was pushed that needs to somehow be encoded in the packet.

F

The the interface selection itself doesn't seem to matter so much I think that's what Marine is also making the point that uh the current way should should still work. Fine.

G

We just need one lookup table added then, and the API to prove it.

F

Yeah, your frequency, somehow Plumbing the MAC address that was passed in Via this I uh existing style API and get that into the underlying Mac of the dash by BMV to pipeline I. Think we are good to go right.

G

You'll just need a P4 table that implements just enough such that the code gets kind of off and written.

F

D

A

Doesn't we don't need to enter generate yeah just.

G

Oh we'll just we'll just copy we'll just do some hard-coded uh API yeah.

F

G

We're doing Square for switch attributes and whatnot.

E

Yeah they're all right, they're, all writings, API, not sure what we need to create.

E

B

Is something we can reuse right, but uh would the people need any modification to make sure this works.

B

Or is it not needed.

C

There's no table to set the dust Mac address. It might need that.

C

If there's no table currently in P4 to set the desk Mac address, it would need that probably.

A

Yeah yeah right currently they just zeros and that's.

G

It well my point about the auto generation was: if we add a table, then we need. We need the lib PSI to be able to program the table and the more we can do automatically the better right. If that's possible. Oh.

A

This is the question: yeah yeah, it's possible to do automatic this so yeah. It would be great.

G

Yeah I mean the infrastructure is already there, I, don't know if we've done it for non-dash apis yet, but you know to have to manually program all that P4 runtime translation would be annoying right. Let's, let's try to do this in a way that just comes right out of the P4 code, if possible.

D

Yeah, that's where I disagree, I I think if API is given beforehand, then it's better to connect it manually, because it's just one table really.

G

Yeah, okay, whoever wants to do that work. That's I! Guess, that's a choice! Right! That's fine!.

D

Yeah, probably that's a better answer right. So if, if someone can come up with um automatically, please do I see Emmanuel path, a shorter one.

F

G

F

A

H

So yeah later on.

A

H

So the behavior we're talking about the um packet being sent to the output port. Sorry, a packet coming in being sent to the same port on output.

H

um I was saying that that is a simplification only for the bmv2.

C

Real Hardware would have to be more flexible. Yes,.

F

Yes, that's right.

C

And yeah me too would not and the dash tests for yeah.

D

But it's still.

C

D

Required Behavior I think that's the point, because network doesn't care.

A

Even the configuration in the test case again will be, will stay the same and it will work on a real Hardware. In that case,.

H

So, on the real Hardware you're saying um say with a two Port dpu: you want to be able to send packets in one port and have them come out the other. That's the that's the desire, that's the requirement.

A

D

The requirement is ecmp, we have underly configured as both ports are equal cost, so do whatever you want to balance your egress traffic on those.

F

D

A

So the acmp, also like configuration, will need to be later on added right.

E

So if that is CMP, that's what we want, but the bmv2 model is doing a simplification. Whatever we need to do. This actual routing on Hardware is not generated right.

D

Yeah because we I don't believe we want to test the load balance I'm.

E

Not talking testing I'm, talking I'm talking, you got a set of API with functionality. You want to think uh differs, uh float um sorry vertical to work. You know you give him two or vendor specific or Hardware or whatever the same way right so.

D

Not exactly because hashing for ecmp is not uh equal across implementations, I.

E

Know uh writing can be complex with multiple levels and all that which I don't think. We agree on.

D

I'm talking specifically about choosing the ego sport, it's not for the same packet. It won't be the same for different vendors. Even for switches. We don't have a uniform algorithm.

C

I think if I can so I think what's being proposed here is Dash test that tests. You know, group based Echoes and all kind of stuff assume a simple routing configuration that the bmv2 implements that will work on bmv2 in this simple configuration, it'll work on Hardware in a very simple configuration.

C

If you want to test underlay use different tests, it'll only work on Hardware. It won't work on BMV too.

A

Yeah so for under latest testing, uh I wanted to say before so we have like different set of test cases that can be used for testing your dpu and underlying support, because you know so currently we already have those test cases and there is a PDF. This cases that Intel has been attributed into the site repo, so that.

F

C

F

A

The latest cases you can take it and use it, yeah, don't.

C

Run them on the pmb2 model for dash, though, because they won't work.

A

Yeah, if bme2 has support so we can run it.

C

But no but but but unless.

A

Somebody volunteers.

C

To do that, it won't.

A

B

And so at the moment we have that and yes, it can be run via using bnb2 or real Hardware, but for the dash uh pipeline um to be supported um in BMV too, and the test cases that we are writing that are applicable to both real hardware and- and you know, I'll be in between- can.

D

You give an example of: can you give an example of underlay only test.

B

No, no, what I'm saying is that underlay is like a separate topic where you know. Whatever we have contributed, it works on um a smart, Nick sort of a device, so it can be tested on real Hardware. It can be used to test your Hardware as well as the mv2, if needed, you know, but this topic is very specific to Dash and whatever is needed for dash. We just focused here to implement uh that part, including the underlay. That's needed. Okay,.

G

In other words, what's the minimum underlay support yeah right? If we want to test overlay, what's the minimum underlay support we need so tests can run unaltered right.

A

Yeah so right now we in today's score. We do we just trying to discuss the routing basic routing that we need for the dash but yeah. So the same question will come out for any other underlay features also, and we during the let's say, adapting the under latest cases uh for for the for the ipu. So we ask this question and we have some table on the high level design on the dash with the list of let's say some underlay attributes and features that needs to be supported. So it's currently.

A

This is the only one source of information.

G

Can you share your screen with that, so we can all look at it.

A

D

A

Part of sewing design yeah, so one second I will share my screen and we'll show to everyone. So so, let's open the second so.

A

Do you see my screen.

G

A

You yeah, it is somewhere over here, like a general, what it looks like and each of the looks like here, parallel design.

A

Sorry, the python.

A

So the container okay over here so underlay- and you see it says so this is the list of attributes underlay attribute. That needs to be, let's say, supported by by your dpu. As we understand, okay, like snowboard policer Port attributes photo interfaces. As we can see, there is only very small piece of attributes, Road switch and that's it.

A

But right now we are talking just about the routing. You know just to be able to write the test case. That will, you know, run on the bmv2 and then without any modification will be running on the real harder. So at this moment we actually don't need to implement everything in the BBQ like host interfaces, but the post.

D

Interface is not really about being with words, yeah.

A

Yeah I just think it's same for the harder.

E

Oh, it is somehow we need to decouple writing from Dash. It's from you know. You are UV net I mean all the stuff that we bmv2 is testing right now, right because there's no way to deal with ecmp as a whole. It's huge!

E

So we need to be able to and cap the packet and maybe for test cases. We plug a simple d-mac right, or maybe a simple writing next hop writing, but we need a way to maybe send the packet, as is to a port that is connected to a real Rider. That will actually do all that all the routing functionality that need to be done right.

E

There needs to be a model of how we plug how we connect things to those blocks together, because we we don't got to rewrite writing right.

D

Yeah I I think what you're talking about is more related to a um Smart Switch, but then again, there are different sites, even in the Smart Switch software architecture. The switch side is separate from the dpu side and the full routing with the uplinks down links Etc, you configure switch. Sorry sorry.

E

We ain't have the packet, where what do we? Where do we stop having test question and doing a little bit more? That's one thing, but functionally from a requirement standpoint. Where do we stop for dash.

D

Yeah right, so requirements are very simple: just send whatever Port you want for underly, put and put the right Mac address for that for based.

E

On a simple Next, Top lookup.

D

Ports have the same, uh not the same desktops but uh yeah. You can kind of simplify the next hops neighbors and all that you can kind of compress them. I.

E

Don't want to simplify I want to know what it is a source Mac will be because we were at the packet well, I guess: maybe we could switch The Source marker or dmac could be anything.

A

So the source source Max should be like of the source.

D

Mac is the switch attribute. It will be configured by the control plane or by the test in.

A

The switch attribute, or it should be, a source map of the router interface.

D

Switch attribute is enough.

D

I, don't think we need to support uh per router interface. Sonic doesn't do that.

A

Yeah one second, because why I'm asking this? Because on the this, this is the list of attributes, so I saw the source Mark address here, so I I assume that this actually Mac address should be used for the as a source. Mark of the packet is.

D

There, a MAC address on the switch.

A

This is the mecca yeah yeah there is. There is also the source Mac address of the switch and sources.

D

A

One is the rotor interface is redundant.

D

Router interface, Mac, yeah.

A

Okay, okay, so it looks like okay, so maybe I will open the issue on the this stuff if this field is redundant, so we we can for everyone to make it clear that the source Mac address should be the mock address of the switch. In this case, we can also the fix the test cases correctly and again. There is another question about the destination, so it should result based on the next hope and neighbor right.

H

So destination so I'm still confused here um and I apologize for this um I'm trying to figure out for the dash data plane in the hardware not not to be mv2, but in the the real product we put in a cap on the packet for the v-neck to v-neck case.

H

We know what pack what ports the packet came in on. Do we need to do um an LPM lookup on the destination IP of the encap we just put on and do routing on that yeah is that is that the intention.

D

But the problem is that 100 percent of the time you will um end up with the result that you, you have a CMP of both ports like any LPM lookup, will result in the CMP group that contains both ports and then you can choose um according to hashing, whichever of these ports and that's it.

E

You don't want that part of the ash data plan, that's impossible, but.

D

I'm not saying uh I'm not saying that we need it in BMV, too bmv2.

E

Forget bmv2, I'm, saying functionally wise, it's huge you would you support four levels. You know all those things right or.

G

The dash feature set, but um how does that work with 8K when you're trying to have one port going to one switch, another Port going to another switch and you want them to stay on those ports? Why would you ecmp.

D

What I mean, why would you CMP.

G

Well, if you look at the the you know the serious Appliance, when you have the h8 topology, you have two Port device, it's an appliance contact, yes, and you have one part going to switch a and another part going to switch B. Why would you ecmp the outbound packets?

G

Don't you want to go back to the same switch or.

D

Do you want it can be the same switch? My point, is it doesn't matter.

G

Yeah, if the switch fails, then that ecmp group you have one port down.

D

And all the traffic goes to the other port.

E

Gk, you got a bunch of tables to configure switching it's again. Basic routing is not just next. Oh sorry, basic writing is next up and that can be done, but a full LPM would be gpe and all those things I mean you know you find those on full routers and they're huge.

C

Ask Microsoft if they require it, if they don't, they can remove it from the dash requirements. That's.

E

What I asked for actual requirements on once we encap the packet like Bud mentioned, we got to figure out the DMACC. What is that based on that's? Why.

C

According to the current Dash, spec I think it's full routing if, if Microsoft doesn't need it ask them to remove it from their Dash requirements,.

E

And and I think they want writing I think there needs to be a way to sign a packet that packet to Port somehow or to a process or something that I don't know exactly depends on the hardware I suppose, but that actually does a full writing functionality if you send it to a port and that gets offloaded for full riding.

D

Yeah, but my point is, there is no way if you're looking from the outside. There is no way for you to know if the dash Appliance is doing the full routing, even if you will send the packet to the same port, it came from it's still correct, so you can compress everything and not do the full routing.

D

Still will be a correct Behavior.

B

So a Marion since you're saying uh what is the correct Behavior? Can we have a bit of write up on exactly how you envision this to be working?

B

um It can be later included in the hld too, but I think for this group. It will be very useful to just uh you know, resolve this current question that we have.

D

No I mean I'm, saying again, I'm saying that um there is a definition of the routing and side behavioral model and underlying side behavioral model.

D

The caveat here is that we have such a simple Network topology, that there can be some assumptions made.

E

Can we get a clear picture of the top of the simple topology right I mean I. Guess that's why it boils down to what is the deployment case, so we can actually figure out those shortcuts, because for me it's you know when I read about it, there's like well result. The neighbor, that's doesn't mean anything.

E

I mean it's just me.

E

D

Yes, simple: topology is two ports connected to gateways. Both of them have default route and that's it.

H

So I mean I'm hearing like things that might be well to me up here, conflicting. On the one hand, folks are saying some folks are saying: we need to do full routing and at the same time I'm hearing, but whatever routing is configured due to the topology. It just boils down to sending the packet out the interface it came in on yeah, so I mean I'm, not sure those two things aren't the same.

C

But one's more work than the other and more complicated the other one's much simpler and the authoritative answer is Microsoft from the requirements.

D

The thing here is that Sonic doesn't know that uh uh if you have a simplified routing, you send to the same port or you actually decide to do ecmp and calculate the hash, um and all that the the sonic will configure the same apis. And that's where Voldemort's uh concerned about the test came from.

A

D

uh Now the question is: what would be assuming we have this configuration? What would be the correct, Behavior and the correct behavior? Is it doesn't matter if we really do a CMP or if we send the packet uh on uh on the same port, it came from or like send a port send a packet on the other, Port always doesn't matter as long as this configuration is preserved.

E

Finding the proper dmac and ethernet header is key, though right.

D

Yeah yeah but DMACC came comes from the egress Port, eventually.

E

D

E

You're. Writing! That's just well.

D

Your your neighbor is your ego sport. If you're running in this case, yeah.

A

B

A

If you talk about the topology So, currently the PDF test cases that we have in the desk, so they assume that we have a very simple topology. So just you know when the the host with tunings connected with your device, it's two ports, and currently we can send the traffic from one port to another Port but I. Think but.

D

As the physical topology yeah.

A

It's physical topology that we we're doing just simple testing.

D

Yes, but you, your test, assumes some kind of like intended: Network topology that you're trying to simulate it's, not the host connected to the dpu.

A

Yeah, so so I I, think that in this moment, for us is I mean for us like for the PF testing. So it's okay to to assume that the traffic will go to the one port and back from the same right.

A

But for the real use case so yeah. We definitely will need the CMP and yeah.

D

How will how will the test be different, like you should expect like it to come from any of two ports.

A

B

Whether we should test ecmp or not, probably.

D

Yeah, no, you you, how can you test ecmp, if you like, don't know the ecmp function.

G

You have to send enough packets to do a statistical swag at it.

D

That's load balancing, that's not the function, load, balancing or balancing of the ecmp I think it's out of scope.

A

D

This time, because it will be different on each Hardware.

B

If needed and understood as the intended Behavior, then when we record it, then we add test cases to test all of those.

D

Yeah, but you cannot write a test case saying: I expect a packet to come from this exact port.

B

That is correct, agree.

G

You could send large numbers of packets and do a statistical um look at them and make sure that it's not all in one port I mean we've done tests like this at line rate, so yeah one packet at a time not helpful, sending many packets with large numbers of incrementing fields Etc to get entropy. You can do some tests.

A

Yeah I think that's interesting.

F

Those kind of tests are only applicable to Hardware right, not for the BMV yeah.

D

Yeah they're, not testing for correctness, yeah.

G

Not for Hardware you can do you can do those things you can do that all the time I'm still a little bit hung up on that uh when someone was sharing their screen. Where that paragraph below the table of attributes said you know, the dpu will do ecmp towards the two tours. But where are the attributes in the table above that? Let you set those ecmp groups. You know the the Swiss light table of attributes didn't seem to have things to configure pcmp.

G

So is there just maybe a typo in that paragraph and the dcmp part should be deleted.

G

D

Agent has already like it's just um inherited implementation from the switch to configure this, but to configure a CMP same ipdb schema same size. Apis can.

G

We can we look at that screenshot again, because there was a table of underlay light attributes and I didn't see. Specifically, that's the whole point of this table is to show which subset we need to support So. If it's not in that table, then.

D

It should be added.

G

Or that paragraph should be changed to remove ecmp either one would make them consistent right. They spend yeah when it says n does ecmp routing. We deleted those four words Our Lives, to be simpler, because.

D

Because there is a real life requirement: Chris, okay,.

A

Well, that's something a few places yeah.

G

And this document needs to be amended so that the attribute turned the table.

D

G

Okay, that that's what I'm trying to say this document doesn't seem to be self-consistent.

D

Yeah sure, of course,.

G

Yeah, we shouldn't say a thing and we shouldn't have to say things like well, it inherited inherits them from the regular. You know Sonic requirements. The point of this document is to specify specifically so you know, otherwise people will um get caught off guard Implement everything in here and think they're done.

G

So we probably need to file an issue.

A

G

Think yeah we don't want, we don't want.

D

Yeah just open a request uh that works next to group member yeah, yeah, okay,.

G

Okay, thank you. It's.

A

Okay, so do you know if there is some any top uh issue that we can use it for discussion? I mean to agree that the list of attributes, let's say, needs to be added to being V2.

D

um Sorry, I didn't get the question: what can you refresh.

A

It I mean: do we have any issue that we can use on the GitHub ticket ID, where we can, you know, discuss continued discussion. What what exactly I.

D

Don't think so, no I'm, just explaining the motivation where this table came from. So we looked at the orcasian implementation.

D

We assumed that with the uh required configuration such and such attributes will be called, we missed, I, guess ecmp. Maybe something else.

D

There is no issue, it was just added, based on uh um based on the occasion, implementation that we have that's how this table came out to be.

A

Good or agent implementation is talking about the art agent. So do we have somewhere I mean upstream or not.

D

A

Of the regular okay.

G

D

The thought process is the following: we have a real life topology right, a dpu connected to two switches, blah blah blah bgp running there. Based on that, we know that, based on this, we know which uh FDB tables will be populated and the underlay Sonic, based on that we know wage apis will work agent call and which attributes we need to support.

D

So kind of the source of Truth is the topology and from there we are trying to derive everything else.

B

For the previous topic, um just sorry you want to summarize because it's 10 57- and we will probably need to continue this in the next meeting.

B

um Just to summarize, um as the next steps for the previous topic, maybe it is good to add a line in the list. It's saying a list of items, to-do list or bmv2 saying um the next top Mac address portion of the routing or Dash pipeline um needs work or basically, basically I, add that into the internet.

G

The pipeline and the in the lib size support.

B

Okay, that sounds great any other topics and some notes that you really would like me to capture. Please do send it to me um and I will capture that and send it to Christina I have my own notes, but if you like something specific to be added, please send it to me.

I

Yeah thanks vishma um I, know: I, don't talk very often in the meeting. Just want to introduce myself again as Yousef joining new on the um serious Dash team, uh which might have some notes that to send over I'll be sending you those offline and we can kind of connect on those um also want to give a heads up for everybody.

I

um I think we're gonna the next time we kind of all get back together for the work group. A weekly meeting, we're gonna do like just a quick review of the project queue and um a lot of those um a little bit of those action items um haven't been looked at in a few months, so just want to get an update just from you all and kind of get a little bit of feedback to see if we can take some things off or just update a few of those line.

I

Orders but um just want to say thank you for everybody just hopping on the call, and thanks uh Chris and Richmond for starting off the call for me um kind of hopped on late. There was in a previous meeting, but just want to say thanks to everybody,.

B

G

Marion, what about the Finland thanks yourself about filing an issue against the Sonic hld spec? Do you do you want to do that, or do you want one of us to take that.

D

I mean why file issue if we know what we need to do, why not just add a few rows to the table and create a pull request and be done with it.

G

uh Okay, I, don't know which attribute to add so it'll, be nice to have an issue just to make sure it doesn't get dropped.

A

Would be great for the record just you know to have a kind of issue.

G

A

Actually, don't know the exactly.

G

B

Also was saying he wants to file some issue. Was it the same.

A

Issue the MAC address I. Think that's. If.

B

A

B

A

Remove one line and that just explain Miriam, so you can take a look above the MAC address. We.

B

Okay, thank you. Everyone. We are over time and I have another presentation. Actually, so I will drop.

G