Internet Engineering Task Force IRTF, 15 Jul 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: ACM, IRTF & Internet Society Applied Networking Research Workshop 2017 (Part 1)

Description

ACM, IRTF & Internet Society Applied Networking Research Workshop 2017 - Part 1
Prague, Czech Republic
Saturday, July 15, 2017

A

This year, shared by it Renato Teixeira and myself, he record Renata I couldn't make it to this meeting, which was which is unfortunate but well we'll deal with this.

A

You don't need too many people standing up here in anyway, since this is just our second iteration I want to have a brief recap on why, where this came from, though so, the entire workshop has been coming out of a one eye, our SG dinner event where we talked about how to connect ITF and I RTF and research people better, so that those communities have a chance to mingle, somewhat more, and so we put together this workshop being outlined as a forum for research as windirstat and so on, to discuss emerging results in applied networking research.

A

So the idea is to find a place and enter framework that allows scientific people from the same academic community to actually submit their work to a place.

A

That's close enough that the follows peer review guidelines and so on, but is close enough to an ITF to make it interesting to maybe stay over and take part in the remainder of the week, and we are particularly interested in applied and results that would that could help us influence, add to or otherwise interact with the ITF work, and also with possibly upcoming research group activities and the IRT F. So that's just the the rough framework.

A

We are trying to run this on the Saturday before the IDF's, the the first two occurrences have been the summer ITF there's no need that this would need to be the summer I'd here. This is something that we can surely discuss later this afternoon and I'm more than happy to take suggestions on this me or other end members of the steering committee. So for this year's edition we got 20 submissions which is a bit lower than we expected.

A

We had each submission reviewed by three PC members. We had a conference call to discuss and and the end we decided to accept full papers, seven, four papers, two short ones or three short ones. Sorry and one demo, and last year we had a relatively crowded program which so we decided to go this year to the towards the other extreme. So we have plenty of time for discussion. We have lots of people from the ITF community here who are anyway eager to discuss pretty much everything all the time.

A

So this would give us a good frame so so feel encouraged. Well tell you when we run bared it over all papers are available, open access on the workshop site and in the ACM digital library, and again, as for today's format and the stuff that we had we're happy to take feedback in the evening, we have a program committee of a crowd of people who have been putting in cycles to do the reviews discuss with each other so that we could finally come up with a conclusion of our program.

A

That's those folks, the ones whom you might be thanks to all of them for reviewing the people. You might want to talk to about suggestions for the workshop future places volunteering to chair it and so on.

A

As are those on the steering committee with Ruslan, I got Medford Ellison, Rankin, Colin, Perkins and myself an important element of our little workshop since we get coffee breaks and they even are going to have a reception in the evening- is that we get sponsors, Akamai and Comcast have been providing substantial support, as has the I sock and also sick home from them. We have received funding for allowing to put out some travel grants, so we had handed out a few travel grants. Our students applied.

A

One student, unfortunately, could not make it in the end due to bad circumstances in last minute, so what we have a few travel grant recipients. So, thanks to all our sponsors as well and then for our agenda, so this is a relatively relaxed one. So we have a few minutes left for the opening.

A

Then we have three technical session, one on measurements, one on transport that also features a demo paper. This demo paper thing is going to be an experiment. The paper didn't quite fit as a full academic paper. What we figured it would make up a nice demos, but since this was the only demo that we would have had putting it somewhere in the corner and hoping that somebody might notice didn't seem to be a good idea.

A

So we'll do it we'll try to do a full screen live demo for for all of you and see how this goes then we'll have some lunch. Lunch will be catered in. One of the adjacent rooms here, so we don't have to walk around very far um next session, is then going to be implementation and operations in the afternoon, and then we are going to close with a panel on internet health metrics.

A

This came out of a submission that Leslie dagger put in where we said this is this is an interesting paper, but it, but it would actually be much more suited for a discussion round for a panel, and so we invited her to put together a panel, and she found three happy panel members for us and we are looking forward to this as a close of the day, we'll do a quick, wrap-up and then move to our reception.

A

So last year this is probably hard to beat that if you, berlin, put together a barbecue somewhere outside, I don't know what that wanes, because I couldn't attend it, but um so we had to do something more close by and there was no companying ITF event. So we just go upstairs to our cloud: nines sky bar and lounge from the hilton, where we are going to have some little bites and some beers and related things.

A

So that's we have this place starting at 1700 that opens officially for the public at 1800 and we can stay there and have our stuff till 1900. So this gives us ample time, and this would be the time when we are eager to collect feedback and a relaxed atmosphere on what you like, what you, what you have in terms of suggestions and so forth.

A

Alright, so we have four papers and short paper talks. We are not going to do super exact timing, but we are going to our chairs are going to watch the time somewhat. So we have for the four papers: 20 minutes for presentations, 10 for questions for the short papers, 12 minutes plus 8, for discussion and interrupt the speaker at any time when there's clarification, questions but keep fundamental discussions about questioning the methodology that was used and so on to the end and since I said before we do caught time so be interactive.

A

One little requirement is: we are being live streamed and recorded by me to echo so there's the camera and speakers should stay somewhere in this range. You can walk around, don't stumble across this monitor and break something, especially on yourself and use microphones. If you've got questions recorder, why mic? Here? We have this thing for maybe the session chair and we have.

B

A

Mic for the speaker we'll see if we also get a clip mic so far, there is none, but maybe we are going to get one. Do we to be able to organize one?

A

The IDF's has a no dwell statement, so we are being recorded and live stream, so think about which political incorrect remarks you want to make. It might stick with you other than that other than that the ITF IPI disclosure. Our rules, don't apply for contributions made for this workshop, because this isn't this is outside the regular ITF. It's just an Associated event, so you don't have to state anything about patterns. You might knowing about about a topic. You want to discuss.

C

A

You just turn it on which we did earlier didn't show anything.

A

Okay and then, then demo do it this way.

A

You can take a quick look at this just.

A

So we didn't have anything any any feedback from week ago last year, when we were streaming this so therefore, but we can.

D

Play with this final, you want to give it a try.

D

A

So, in the meantime, anybody who has never attended an ITF meeting before since I'm not see most of the phaser wow. This is good, hooray, this next question. How many were of you will be staying for, at least beyond the reception and the food for true for tomorrow, for at least one more day doing this week? Oh just just off the people who are so one, two, three hooray for okay, so we we have a few people.

A

Okay, final final questions: how many, who, how many of you do qualify a students also.

D

From also us here bunch all right.

C

A

So please practice doing this all right. Thank you. Okay. Then we have a plan.

A

Okay, this gets us to our first session, chaired by Maria I'm staying up here for a second, because I have the first presenter slides on my laptop so.

E

Good morning everybody so first session we have two short papers and one long paper, all around measurements, and the first paper is actually joint work with some people in my group, and so I actually saw that our student might be coming, but nobody managed to convince them. I guess so. The first presenter will be Randy Bush he's talking about some measurements. They did on the ripe Atlas platform and I. Think this is follow-up work on an IMC paper. You had like last year, right I!

E

Guess if you don't know Randy Bush, yet you should take the opportunity to get to know him at this workshop.

F

If you know me, you know this is not my presentation. My presentations have a mauve background and magenta type comic sans. This is a present. This was research. The heavy lifter was Thomas halter Bock, who spent time with this doing research at irj, moved to a day at Zurich, etc, etc. So, for the moment, I'm a member of the Zurich mafia, but back to the agenda for 20 points. What was missing on that agenda.

F

Formal methods we are standing on quicksand, this presentations measurement when serious computer scientists are studying the Internet as a behavioral phenomenon, you're in trouble enough for a political statement. So this work is a measurement one. As an example.

F

Ripe has 9800 probes out there, which how many people here use ripe, Atlas yep free to ask two years ago, would have been none 9800, probes out there, which one should I use it's a hard choice, it's like being in an American supermarket near thrive, 5000 brands of cereal they're, all the same, and the box has more nutrition than the contents. Okay.

F

So the number of probes you I can use for a particular measurement is limited. Okay because they limited because the platform doesn't want you to get too greedy. And do you really want to process data from 9800 probes? Okay, it's hard to know which ones will satisfy a measurement unless you specify and that's difficult, so finding a minimal set of vantage point that maximizes the chance of seeing what you want to see is a fun game. So they have categorical, categorical properties like this. One is behind the net.

F

Where is it located a s, etc, etc?.

F

Two probes, the same a s, can see very different paths. A s s despite the religion are not homogeneous. Okay, two probes in different a s s can see similar paths, the same story, geographic locations and the other things. So, let's propose that advantage. Points, selection, method, based on top a logic, similarity select the most dissimilar vantage points if I want to see from as many places as possible, so like similar ones.

F

If I want to concentrate on an observation and make sure it's not just a problem with the probe etc, okay, so how to find similarity? How topologically similar is, where I've atlas overall and how to exploit this? Okay? How do we measure it? We define a similarity metrics based on the jacquard index of the set of IP, addresses that the probes observe when they trace route to the same destination.

F

Okay, so jacquard intersects is jacquard inter index is known as intersection over Union, it's I've got a set of trace routes, have trace routes to the same destination from two probes.

F

There's the list of the addresses they pass through if they pass through the same addresses the intersection and the Union are the same so intersection over Union is one if they bet dark, materially different intersection over Union is zero. If they're similar you get something in between. So if it's equal to one, the pre probe see the same IP past zero. They see different I feedback.

F

Okay for each pair, we compute the jacquard index for all the destinations they have in common. So this is something you run every month or ten over the whole set of pros, and they do pairwise comparisons and of the only those that have destinations in common then, and by the way, when we do this, when I say we run it, we don't actually run new measurements. There's tons and tons of collected traceroute data. Some people are using the Atlas probes to scan the entire Internet.

F

We just exploit their data, okay, so for each pair we look at the 25th and the 75th and the median etc and report results in a figure. So this is the e CDF of the probes. This is the show card similarity. This is percentile ranges, you'll notice, I. Think that's even a slide for this. That about 10% probes are identical all right, your card index of one pretty cool slope, so you can get half at about point four.

F

So what we have is an a method that we can measure dissimilarity.

F

Okay and let's check that against some reality, we expect to topologically similar probes to being the same, a s and geographically close, so here probes with there's a card index of greater than point nine ipv4, the geographic distance. Remember probes, when you set them up, ask you where it is you put that in there, so you can tell there are anomalies.

G

F

But not that many and in fact we've suggested studying some of that and that'd be fun. So when they're in different day SS with a jacquard index of 0.9, you can see the index is high, they're in different, a SS but they're still geographically, pretty close.

F

Okay, so topologically similar probes tend to be in the same. A s. I know we're all shelved and they tend to be geographically close. So we can take topologically sim, similar probes to think of them as a super probe. Ok, we're exploiting similarity so form probes down. You can run on another similar one and if your measurements going to load a probe too heavily or you're worried about some other piggy researchers, loading that probe up, you can throw it over multiple probes and you compare the data.

F

Otherwise, if I want to get a diverse measurement, I get topologically dissimilar probes, and if you compare this, what it discovers in topology versus the right random, supposedly selection, less methodology, you get significant, you get 25% better discovery and that's my song.

E

Thank you God. We have question you have time for questions. A lot of time and I know.

H

E

Prepared some mean questions for Thomas, but now he's not.

E

Say one question I have is like you.

E

So you make the statement in the paper that this is better than geolocation and and then you figure out that actually a lot of notes that have the same similarity are close by so do you? Do you manage to make any assessment? It was really better or not.

F

Is you're asking his geolocation better or worse than jacquard index similarity? Yes,.

E

F

Let me turn this back on.

F

Taking a look at this, the answer, your questions no, cannot die further. Okay,.

E

That would be interesting to know anybody to.

F

Be honest: this was the tail as you noticed in the beginning of another project, yes, and.

E

Like looking at this light, say, is it that all probes in the second set are actually probe set, have been probably moved to the same, a s and just not being updated or or did you didn't look further into the dead and.

F

Although they did not move the same way as you know it, a s is a by the addressing in the traceroute. Okay,.

I

Right so they're just look very similar right: okay, right.

F

Now we have about, for instance, when you look when ripe themselves. Look at the probes. Many are residential, many are sitting behind ho home, Nats, etc. So in many I, don't want to say geographies, but many places in the world there'll be smaller local providers with their own AAS. They all had the same upstream.

F

Yeah makes it so they look like they're different. A s is here, look maybe the next top they're, not yeah.

E

Yeah I mean would be but probably interesting to look further into that. Oh yeah.

F

There's more research, I've done here, yeah, oh for sure, but I, don't think we're gonna. Do it.

A

I'll take this for now: okay, we'll work on the other Mike, so I'm curious. How often what is a good interval to recompute these indices? I mean routes to change and saw the property underlying properties that you observed by comparing the traceroute sets do change, and so did you look at? Maybe what what happens? If you look at every day every day, every month, every week, and this would would there be a sweet spot where you say okay, this would be a reasonable interval to recompute those things.

A

F

You want to spend a grad student on this, go for it. It's exploiting existing data, so you're not creating new measurements. So you're not doing you know, you're not disrupt being disruptive to the net, so it would be easy and fun to just compute. It now computed on data for six months old computer, on data for a year old and looking.

J

The change in the data easy to do I think I have something from our to tell to my postdoc on that cool.

A

K

Have two more questions? Okay, so how do you account for the unknowns they're stars I'm at restaurant path, regarding the set of the IP addresses and also when x, addresses so as a router responds on the same interface, with multiple virtual address? We're.

F

Not doing it or any of that stuff we're just looking at stupid, IP addresses so the.

K

Gnomes are cleaned up or they're, not tonight.

F

They're ignored okay, just having stars in both fruits does not cause a match.

L

So and I apologize if I said we hadn't had enough coffee yet, but I won Wow. There will be a slight break later on and so I'd like to understand a bit better. What your definition of good is I mean implicitly I, understand or implicitly I. Take away that. The aim here is to use the minimum number of Atlas probes to best represent the full span of the internet for some general sense of the Internet.

F

Okay, we didn't.

F

So if you don't know besides, these sets that I say help here at etc. We did not there's some fun stuff to be done and, yes, we have ignored it in and and there's good base work, good theory that we can apply to say to look at measurement, minimization right, fess up to not having done that and but right. If I ask it, please run this, for a bunch of probes throws a bunch of probes at me. If I take the same number of rows, I can get a 25% improvement by being picky right.

L

F

I guess what I'm diversity! What.

L

I'm driving at is the right Atlas platform is useful for any number of different types of measurements and different models of universe and I think you've got a particular model. Is.

F

L

Yeah you've got a particular model of universe in terms of what you think is good. By contrast, the work that I was using ripe Atlas for that I'm not going to talk about today. I actually wanted to get to distinguish between individual networks, so I wanted to use it to measure in individual networks and therefore I wanted things that were more similar to be clustered together, so I'm, just sort of driving at the point that this is. This is good and right for a particular model of the universe. Other models apply.

F

First sure, like this is a traceroute for you. It's not even a Geo view it's not even in a s view, etc. This is hey I'm, looking at topology and with traceroute. Did this that little corner of the universe, hi Marcelo.

M

And you know thank you for doing this. I think this is very useful and, as an a-class user, do you think I mean? Are you planning to make this available for other people to use like when we are going to select Prague's? It's.

F

M

There sorry it's.

F

Up on the net, but.

M

I'll keep on planning to keep updating.

F

M

It is a service, yes, no, we hadn't thought of that I mean maybe, since a meal is involved or something you could, because it's very useful when you want to select probes.

F

N

You want me to say yes, yes,.

G

N

It's these are. These are typically thing that that we love seeing I, mean sort of speaking for the Atlas Operations team or development, whatever that these are the things that we ourselves were thinking about, some of them, but we didn't do because it's a lot of work and we are not research.

N

Well, most of our us are not researchers, so you know there's good stuff to be done here, but once there is kind of some kind of consensus that this is a good algorithm, if all you've, only if Atlas ran it inside and gave us the diet data. Naturally, that would be great than sure we can do those things. So we even have a contributor taking.

F

Off site research and I'm, putting on my operator hat since you seem to like to think operators, your major customer and I am one um is as an operator when I want to look at how the internet sees my data sees. My AAS sees my whatever I want diversity and having this as diversity. I want. So it's not just the researchers who would like this.

N

So, as I said, I'm yeah I'm happy to pick it up, and you know make this a part of your service. That's no problem! The the issue that I have not in particularly with this is that there are like dozens of these kind of things that people want to throw at the Atlas team to say: do this? Do this? Do this? Do this building building building building and we have to like look at which ones are I, don't know more useful or more interesting or whatever they are. As.

E

You had at the mic right now. Can you tell something about the selection, algorithm you're using right now sure.

N

That's basically what you can say is that give me probes from this country. They say as this prefix this and that, but if you say, give me 50 from this a s, then the system basically does two things. Randomized is somewhat so it just says: well, I'm gonna, look at these 50 probes for you, but the one more interesting thing that most of the people just ignore or cleanse over when they do such a thing is that we also have to take into account how busy those probes are.

N

So it's nice that you see a probe in Antarctica and I. Oh I want to measure from Antarctica, but everyone else wants to do that as well, so we actually almost physically had one in Antarctica, but one of the reasons why we didn't do that is that it would be just useless for everyone. It.

B

Would be nice, maybe basically no.

N

One could use it because everyone would want to use it so so I remember a couple of days ago there was a hackathon where some of the participants made a much better algorithm of visually distributing the probe on the globe, so they compared look when Atlas picks. This is what you get when my algorithm fix were all over the place. That was nice, except that in reality you cannot. So we need to expand the network to make that available to everyone. But, coming back to this thing, sure um we can do this.

N

We can build in such a thing and say use the Randy algorithm to give me probes.

F

That would be more magenta you're, so.

N

Humble but I do have a question though, since I'm sending here. So how would this scale if we had a hundred thousand probes a million probes? You know pairwise comparisons, N squared, that's kind of difficult.

F

Your long way to go this runs pretty quick.

F

And it's decomposable: if you want to use parallelism, I.

E

Have a last question, of course, I was really disappointed that you only looked at ipv4.

E

Maybe, with your experience, you have a guess if you expect that the routes are different for ipv6 or probably kind of similar well.

F

As we know, ipv6 routing is different than ipv4, routing and I. Think Jeff Houston said it best. The decade ago is when there's enough money behind ipv6, the ipv6 routing will get straightened out. There's been a lot of interesting work on looking at the difference between the two. You could clearly trivially run this over v6 versus before and if you're looking for these v6 probes to use this is trivial. Oh, we didn't do it. That's a common failure.

F

I realize when I give her a hard time about v6 those people that don't want to know me. Cuz I'm, first ISP to deploy I, give you 6 the world 1997, so um it's trivial to make it to v6. Ok, so I have one question before I get out of here, which is: is there a DNS behavior researcher in the room.

F

You're only fit catch me at the break. Would you because in the NOC we see something really cute? Thank.

E

H

E

Work is not up yet.

E

The next talk is given by a Petrus teaches. He is also looking at ripe, tater and he's using looking at user to user connectivity and now I forgot your host CPU.

O

E

Student, okay and you're, currently at a tripe.

O

E

You're already gone: okay, okay, so I just leave you to your presentation. Do you need help.

O

So good morning, everyone, my name, is Patrice giggy's and today I will present the paper characterizing user.

B

O

O

Okay, now you can hear me right so use it to use their connectivity using dry pathless. This is a joint work with.

O

So we said joint work with Vasily Scott, Ronnie, Steven, okay, Amy lab and so hand informed us and for this work we had to develop two different tools. The tools are already available as a types and we use ripe Atlas for the measurements. But first of all, let's look a.

O

Deployment around the globe, so we can see that there are ripe, Atlas probes, almost everywhere, with Africa being an outlier, and this probes are deployed in hundreds of networks, but we don't have any kind of estimates or estimates of a huge that are being covered using this probes.

O

So in this work we focus on eyeball networks and when we say eyeball networks, we felt the networks that have a largest users population that contains a la decisión population in any given country. So as an example, an ISP is an eyeball network and studying the eyeball networks is very important and interesting because it's a user to user traffic and it's important for real-time communication like online game VoIP and a.

O

This depends on how these networks are interconnected. It's ever so. There are many interesting properties, but we can study and in this work we study the traffic locality. So we tried to see if traffic between two eyeballs networks in a country stays inside the country and either eyeball networks are direct, are direct or indirect either connected, and this is important for free main reasons. The first one is for security reasons, so we, if you have an able network, connect direct with another eyeball network. You minimize, I've stroking.

O

You also maybe have less latency and it's easier for a network administrator to debug and the main questions that we address in this work is to see if ripe, Atlas is deployed in the network, the lightest user population in every country, and also study how these networks are disconnected server and.

O

We had also to think which are very interesting properties. We should study and measure and visualize, but also we wanted to see if there this properties differ from country to country and there are patterns between countries.

O

So we had to create two tools. As I said in the beginning and.

O

Using this tool, we try to answer to this question to find the most suitable way to visualize, with statistics and measurements and for the first tool we used VAP Nick estimates, ethnics provide user population estimates per ASN per any given country, so we selected the eyeball networks using two different thresholds, the first one. We assume that an eyeball network is a network that has at least one percent of the users population in that country, and we also select one more threshold, the cumulative 95 percent that for population covered in the country.

O

Later we use the ripe Atlas API to fetch probes for these networks and we also convert our estimate of covert using statistics from internet life starts. So we have at least one ripe, Atlas probe active in the nibel network. We assume, but the network is covered. On average we have 90.5% covered in any given County. However, there are some outliers like Rocio, and we wanted to make this more clear. So we visualize our findings and here is a global.

P

O

Of the ripe Atlas covered, we look on ipv4 probes, which are public, so we use a range of colors with the white. Are the countries with Nora on a eyeball covered and with dark green? Are the counties with.

O

100% I part less covers, so we can see that Africa is again an outlier, but also we wanted to provide a more useful insight for noted, network operators and ripe Atlas ambassadors. So we create a more detailed view.

O

Actually, the tool is already available on the link of the bottom, so it's SD does Bob a striped dotnet and Petros loss a population covered, so this is a table but create on a daily base using the zones that describe I described before. So this is snapshot of 12 of July and we can see, but for velar just eyeballed network in chess republic. We fail to mark it as a cupboard because we have only one private probe but having one private probe, it's not useful, but in public for probes to measure with nato.

O

So if you deploy one probe its network, we increase. The percentage covers inches public forum from almost 50 percent to 8 percent.

O

And sin: okay and next we'll create it one more tool we call it. I will die and the main concept of a tool is to use act measurements. So we use ripe, Atlas probes from rival tribe and we do trace out from my ball tribe or networks, and we look on the properties like if it rains not past stays inside the country and if we trace out past reverse other races, if yes, which ones how many of them- and let me give you some more tales about the elimination.

O

So we do monkey traceroute from my ball to eyeball networks and for the probe selection right now we use a very simply methodology. We select the closest and the fairest probe from the country's capital to try to expose the AAAS diversity and we use right start to map eyepiece to a aces. We support, ipv4 and ipv6 and to delegate the traceroute pass. We use open IP map.

O

In order to, as I said before, to visualize the statistics in the findings, we had to come up with a visualization, so we created the a s to s matrix the s to s. Matrix is a stable structure. It's row represent an ipod network as a source, and it's column represents an eyeball network as a destination.

O

The size of a cells depends on the APNIC estimate estimate and the color of the cells represents interesting properties like if the traceroute path goes out of County in country we have ripe Atlas covered. Are there any inconsistencies between probes I forgot to mention that we select two probes arrival networks, so we have two against two, our probes in the destination, IP networks.

O

So may we may see different trace a trace out path and we use a basic metric, so we select as an example the green boxes and we divided with a whole area of a space matrix to find the basic metric presented and next is an example of a eyeball to die for Canada. This is a snapshot of first of April, so we can see that almost 47% of the tracer pass stays inside the country and only 3.1 percent is going out of country. We have some inconsistencies and we like to measure small networks.

O

We also investigate the direct and indirect path and we use the border to distinguish that with direct and indirect paths. So with white borders, it's a direct eyeball-to-eyeball interconnection. So for Canada we found out what almost 41 percent or trade-offs between eyeball-to-eyeball networks. What we measure is direct path and next, as we are in chassis, public, didn't resist and it's a step up. So, as I said with previously, we like to cover the biggest eyeball Network.

O

So that's why you see the a white boxes, and we also see here but almost 10%, of a tracer path between eyeball networks. They go out of count. We like to cover 32% we'd, not measure 46% and the direct path. Our 17%.

O

We also measure Australia and Vistal measures right now, 104 countries and it's available the link on the bottom of slide and, as you can see, in Australia between eyeball-to-eyeball interconnections, we see that every trace help us stays inside the continent and indirect and direct paths are 25 percent and 29 percent respectively.

O

In this work, we have some limitations, so, first of all, ripe, Atlas provides a biased view of the encounter connectivity. We measure this out path and no traffic volume.

O

We don't have ground truth about the IP level and the location accuracy, and we also face the same difficulties. We with other works like I between mappings and we plan in the future to use tools like map it to improve timber god. We also measure on layer 3. We don't have any layer, 2 visibility and we are looking for ground truth influence for our testing fire for replicas mate and also for the way that we.

O

Define the Inka so as a future work, we plan to move from prototype of this tool to fully fledged tools and make them publicly available to the community, to the network operators and also to very services, and we on. We want also to gain visibility on I will to neighbor our neighbor count the eyeball traffic. So as an example, we want to measure the eyeball networks of Netherlands against eyeball networks of Germany, and we also want to measure a rival to popular CD ends.

O

We are also looking on more sophisticated probe grouping selections and we look for collaboration in on this. We need also to validate our infinince about IP to a is mapping with the location and, as I said before, we want with the great tools like map it on our methodology.

O

So for completion, we estimate the ripe Atlas covered in eyeball networks and we present a prototype of the Eiffel to die. The main aim of the tool is to measure and visualize aspects of user to user connectivity, boundary and the tool is developed to help users and operator to discover in the Scimitar connectivity artifacts in the cartridge they need or we operate. The tool is available again not linked and our work is based to probe the probe measurements using ripe Atlas.

O

We use the ethnic estimate, and we are very for evening, provides such kind of statistics and to use I a PTAs mapping from right path, I start and the delegation using open, IP map.

O

So you can tools, as I said they are available. So if you have any questions, thank.

E

You very much I have two quick clarification questions, so one is which kind of tracer did you use.

O

We use part stays out so.

E

Tcp, tracery, okay and the other one. Is you have this inconsistency number? So that's when you do two measurements and they show different research.

O

Okay, there were some times that we see that it raised out you a probe, use a different upstream provider to reach the same network, so it may depends on the example grease because I know the ecosystem grease for a network. We divide the network into parts north in the south, so they use two different upstream providers even to reach another network. So that's why we see instances.

A

Out, why does this does this work? If you bite into it, then it works and I had a question you hit for for your definition of an eyeball network. Yet this magic number of 1% of the you of the users of a country where, where does this number come from, is this intuitively? It feels too small to be representative actually.

O

We want to study in fine granularity and we wanted to make a visualization but easy, but it's easy for Samuel to to view it. So if we lower the threshold, we will end up to something like the X become the die. We will have hundreds of networks so.

Q

So, in other words, 1% is already a pretty large number. It's.

O

A

I wouldn't know.

A

So so I would have said if the network is representative for a country if it covers 10% of the users or 15%, not just one person. This is what I'm wondering about so.

E

That actually goes in line with my previous question, because if you saw this in consistencies because some of the network providers split up the network, then you just randomly saw it some at some point in another. You might not have selected any notes that would have shown the same behavior. So maybe you need to measure more probes within a network to.

D

R

E

It's also kind of what you're seeing.

S

B

So I have a met. A question I'm wondering why you started by focusing on end user to end user traffic patterns. Now I, you know I work for a CDN, so maybe I have a distorted view of the world, but it seems to me that most people's interactions with the internet involve a third party, a service provider of some sort, whether it's email or they're, collected or a website or something. So why is user to user direct connectivity interesting?

B

O

There are many application, but they use user to user connectivity. A couple of.

B

B

Direct end-to-end VoIP is that that's it I'm our unit Magnus, but.

E

Yes, you have to describe view of the way.

B

All right so I write so I tried to preface my remarks, but just seems to me that there's a why I start there I guess that's the question. I'm.

O

Not saying that there isn't that.

B

Traffic, but what makes that interesting and.

O

This is a first step, so we plan to measure against Citians, but measuring towards CD ends is a lot of more complex.

T

um This is actually oh wow, okay, hello. um This is actually more of a comment than a question because you pointed out that you're looking like future workers integrating border map and/or map, it I'm wondering how much of your inconsistency actually comes from inaccurate AAS boundary inference, because a s boundary inference is a is a quality like if you look at that, the the prior work and map it right s boundary, which is a really hard problem and doing the dumb thing ends up having um you get like wildly inaccurate estimates for certain topologies.

T

So when you see an inconsistency, some of that could be either zone crowding or different routing policies where you know you're, as Randy pointed out you're making the the common it's. The common fallacy of an a s is actually an internally coherent thing, but I think that some of it could be your your border inference or am I missing where that in coherency is coming or that, where that inconsistency is coming from, maybe.

O

Be the border you say it, but those it can be wrong location. So, okay, yeah.

T

Right, which is also for ripe, Atlas probes, is, is best effort, faith right, so, okay, cool thanks.

E

And I'm, having kind of the same last question again, did you look into ipv6? Yes,.

G

O

G

O

Currently, we have on the online tool only I feel for, but soon we will be there and we plan to make it like a fully fledged tool. Okay, thank you very much.

E

Are you bringing your own depth.

H

E

Okay, so the next torque or the next paper is a full paper and it's from a mesh and he can unfortunately not be here I guess: that's the one Trevor grant that didn't work out, maybe yuck!

E

That's a student okay! um So um no.

E

E

So he couldn't come he's at sisk rice in Sweden and now Anna bernstrom is presenting instead she's a professor at Cal, State University and here have been kind of working with them together and doing some measurements right here, basically involved in this I.

R

Can answer that questions? I guess so? Yes, so Ian could unfortunately not be here. He is in Australia for family reasons, so he sent his apologies and I would try to introduce his work on cheese pie, which is a lightweight measurement platforms which uses raspberry pies as the measurement notes.

R

So this is a platform that Ian has been working on for the last three years or so, and this paper, she's PI research results and the regulator kind of gives some background on this work and summarizes a bit the different measurements that have been done and what it's looking forward to do.

R

So, if you've looked at the paper, you've seen that it covers a number of different aspects, and this work has been done in the context of ready, which is a national project in Sweden, with the research environment for advancing low, latency internet and coast university is a partner in that project. So in that sense, I have heard about sheets. By a number of times we have one of the nodes located in cold stuff, but I have not been directly involved in this work or in the measurements.

R

In the paper, so this is all I work, so I will do my best to try and and and also one interesting thing.

E

To mention is maybe that this was first presented at the rain workshop two years ago, also co-located with the IETF and now we're happy to actually see some results here. Yes,.

R

So if you were a dream, then the IANA introduced the platform there and talked a little bit about so.

R

Ian has sent me the slides and it has three parts so first, some general aspects about the measurement background for the platform and the context in which it has been developed than some examples of the measurements that have been done up to now and the types of measurements that you can use the platform for and then some future directions.

R

So she spy targets home networks. So the idea is to try and capture the the Internet performance, as you see as a home user. So you would put one of these cheese pie devices in the home and measure performance from there. So the greater context is kind of that. You see internet at home as the utility and then, of course, we need to have an ability to monitor and follow up on this measuring from the home.

R

Vantage point is also something that is typically not so easy from the external tools that the ice piece may use to monitor network. So it's in that sense complimentary to that type of measurements.

R

So there are, of course, a lot of different measurement platforms and measurement work out there. So, as I said, he's my torus measurement from the home network, so in that sense it's has similar goals as other tools that I will also looked at home, networking such as phantom and that Lizer host view already from the beginning. Ian has been in context with the the reg laterz in Sweden's, post and theaters tears and the pts.

R

So it's also an intention here that the monitoring should be useful from Allegra regulatory perspective and I am is working together with PDS to try and develop the platform in that way, the need for using the measurements that we do as researchers as a measurement community also for regulations and influencing policy, was something that was discussed. Those at Ames workshop earlier this year and, of course the different regulators are also doing efforts in this area.

R

So in Europe we have a barek the board for the European regulators that have looked at how you can follow up on the directives for for open Internet that we have in Europe in the US you have FCC, which has a broadband America measurement campaigns to try and follow up on on Internet performance. So this this work with pts kind of ties into these these aspects and the regulatory view is also something that has been discussed in IETF in the IEEE ppm.

R

A measurement group I think there was a draft discussed in Chicago that related to this from the net neutrality and regulators perspective, so she spy will they solve all these problems?

R

Of course, not it's just in a one attempt here, another measurement platform to make some contribution in this space and the way the platform is set up is that you write your your tasks, your measurements, or typically, this put together in Python, and then the platform has mechanisms so that the results get collected in the database on the PI nodes as synchronized to a central database. It collects all the measurements.

R

There is a scheduling system so that you can indicate the schedule for when your test should be run, and there is also a dashboard for the tests that are available, which he spies. They're, like 15 measurement tasks, defines that you can, as a user, see the the impact of viewer, the quality of your connection and results of the measurements.

R

There is a further slide on this later on here and, of course, now Ian is looking into also how to design the specific tests that the regulator's would be looking at here and, of course, then we are talking about things like. Are you doing traffic traffic differentiation in the networks or the flow prioritizations to be treat different applications in different ways, and so on? So the things of things that you would need to follow up on on the regulatory, a premier editorial perspective.

R

When you do measurements, you can have different approaches. You may do the crowdsource approach, where you have a lot of volunteers, that download your software and you get a lot of measurements and from that you derive your statistics and your your view of the network.

R

So this is not the goal of cheese pie, so the idea here is rather to try and very carefully select your measurement points so that, with the minimum of nodes, you can collect the information that you would need to kind of get the view of the network and follow-up from the regulatory perspectives and having a long, lasting measurements and there's also being the Enna's also focused quite a bit on specific events on pc times.

R

When you have some popular things that the regular user would like to take part in or to view and see how the network works under those conditions. And, of course, if we're now going to put this platform in the home users and run it from there, it's essential that it's trivial to install and configure.

R

So from the regulatory perspective. You need to follow up on different directives, which are often quite vague, maybe in the way they are specified. So you need to be able to define how to actually do the follow up on this and to then also have some method to see that the ISPs or other actors, if they are not following this. So in sweden, at the moment but pts does is basically uses surveys. It sends out surveys to a number of users to see how the quality of the network connections are.

R

It also sends out surveys to the operators to have them report on if they do what type of traffic differentiation they may use and what special services they have. So the idea here is to the goal here together with pts, is to also support this with measurement information, where you can actually detect some of these things and see if, if what you get into in your surveys, is also what the users actually see in the network.

R

So if you're now going to measure from the perspective of a regulator and use this, maybe to him enforce policy, then the fairness of the measurements is one key issue that Ian is currently thinking about, and that I think we'll also need more thought here. It's not that there are some. A lot of this is still work-in-progress, even though it's been ongoing for a while.

R

So when we as researchers do measurements, we are a bit opportunistic and we look for some particular particular aspect and we we may measure and if we catch that this is maybe serving our purpose. But if you're going to use it more from the regulatory perspective, you have to be fair to all the different operators and see that you capture a correct use. So how do you do fair measurements? This I think is an open issue. Do you need to measure the same number of traffic per operator?

R

Do you scale it depending on the size of the operators? Do you need to have the exact time same time when you do the measurements and so on, and what will be accepted by the different actors in the a that are involved in this issue? So this is also something that in Sweden there are ongoing discussions right now between the different actors involved in this space on how to define measurements that all can agree on actually or representative and can be used to measure internet quality okay.

R

So this was a bit of the background and the general scope that she's my has been developed in so now some examples of the type of measurements that ian has done the past, so the platform has so far, of course, also been used mainly for the research purpose. So far, so there are three types of measurements that you could do in the platform.

R

The first, of course, is that you can take your sheets by node, and then you can measure to some service at the internet and what Ian has mainly been looking at in that context is video delivery and in particularly he is trying to look at strolling events. We all know that video download is a complex aspects.

R

You have a lot of interactions between different aspects between the different algorithms involved between the networking effects, so he's been doing capturing measurements here to try and identify which one of these things that you can measure in the network's actually has an impact on the stalling of the video and applying various statistical techniques to try and isolate the different components and for this work is using some of the videos and sequences from the video quality expert group, which has a pre rating offline, so that you have a baseline for what the video should type of rating the video should get if there are no impairments over over the network.

R

Another type of measurements that Ian has been doing is between different XI spy nodes. So then, you have control of both the server and the client in the measurement and he has been doing, for instance, basic delay measurements so for that he uses TCP based measurements, so measuring either in the handshake or the teardown of the corrections.

R

You can measure between the scene and the sumac, and this, of course gives you a delay measurement for the network and, as you have control, you can also remove the the delays from the server processing or the client processing. So this is an example graph from the paper illustrating some of the measurements from the platforms.

R

The blue here is basic thing measurements and then there's the delay, measurements that are based rather on the on the TCP measurements, and you have a degree in one when you have not removed the server processing delay and the red one.

R

When the server process thing delay is, is removed, and if he's trying to do this, of course, also not just once but over time to capture the stability of the network and one of the future directions that he is also looking into is using this in four multipath and those doing multipath measurements overseas by where you can use this, as as input for for path selection. If you have this historic data here, then the third type of measurement that Ian has been doing is using multiple.

R

She spies to monitor some events, some key happenings in this case he has looked at the boxing matches that are very popular, so a lot of users go online at the same time and then measuring over time. So that you can see here is an example for one of the boxing games that they monitored. So here is the time up to the match.

R

You can see that the delay in the network is quite stable and then what the closer you get to the event, then you actually see that you have some impairments for some of the the providers of this game and by trying to measure from multiple she's PI measurement points and also with multiple measurements.

R

He is trying to separate the different aspects that contributes to the delay here, so we have both the basic ping measurements and then also the measurements at the HTTP level, so to try and separate effects that comes from the server being overloaded from the effects that come from network being overloaded, and this in general of course, is not so so easy I mean.

R

Yes, you see the end effect, but to also try and isolate what is actually the coercer of the difference so he's trying to then look at the combination of these make different measurements to isolate the different effects.

R

As I mentioned, there is also a user interface available for this she's PI platform, and it has some different levels. So there is a interface for the more tech savvy users, so I think it's very hard for you to see. Now you have this figure also in the paper, but this one illustrates some of the outcomes of some of the tests that are defined in the platform's. There is like a speed test like test. There is a HTTP download one of the exercise.

R

There is some DNS lookup test, I think, and you also have some statistics on the cheese pie, platform itself and the load on the measurement nodes. So you can get some quite detailed statistics on the tests that are being run as this is intended to run in the home network and also be for the average user. Jung has also worked on trying to have a more simple interface for the non non techie user. So this is an attempt to illustrate how well your network is working.

R

So this is kind of your your speed of data coming in and in order to meet different types of applications. You would need different levels of water to be sustained here and if it's met- and you have a happy duck in your bathtub, so I'm not sure if this is easily interpreted by the end user. But this is one of the.

R

So, but this is one of them, one of the aspects that ian is also working on in connection to the visualization, and he has also done surveys to the end-users to see how they perceive various networks events and how they use their their internet services, so trying to combine the measurements with the more traditional way of gathering information from the users.

R

Okay, so that wasn't an overview of the three types of measurements and some of the examples from the papers paper on the different measurements that have been done. Looking forward, he is working with the regulators to roll out additional PI's. So, as I said, that target here is not to have a huge scale: employment, but enough notes to get some representative measurements I.

R

Think at the moment the target is around 200 measurement points and to also then design these tests, together with the regulator's that can help enlighten on the net neutrality porch, as I said, he's working on the video store, quantification and trying to detect what metrics influence this and how you could actually make some predictions from your network measurements.

R

If you have video stalls at the user level, he also want to use multiple vantage points from the different PI's to look at shared infrastructure and congestion in the network, and as I mentioned, he is also initiating multipath experiments from the she spies using the MP TCP implementation, and they are also connecting it to some of the more basic measurements that the platform already provides, and that can give you some input for how to tune these. These protocols.

U

Thank you very much and your again has the first question.

B

First, I think this has got to be the most awesome project, name I've ever heard when you said to cheese pies. The first thing I thought you can do that you have to.

G

B

I did want to ask a little more serious note. Maybe you mentioned this, but are you? Are you putting these notes on mobile networks at the far end of wireless links on platforms that have multiple access technologies, things that are more representative of the kinds of systems that were seeing more of today.

R

So so far I think he has not had any of them deployed on the mobile network. So I think at the moment it's on fixed network or Wi-Fi, but I think he's looking into also having additional access technologies.

F

Ready push the brochure IJ, um this is cool and worthwhile. We see this being done in a number of countries. The customer is the regulator, so you never cross the border we're on the global Internet. So you have a special tool here. We have one too hi. Jay I mean it's fun, because we host the biggest event in Japan is the summer highschool baseball, playoffs, 300 terabytes on our network.

F

Okay, so we measure it I assure you with a unique platform, one of the wonderful things about Atlas is it covers the internet? Would you people measuring for your regulators? Please conspire and have a uniform platform.

V

Now Ford just wondering about the the rebuffering measurements that you talked about. I wondered if, as I know, that Google has you know measurements for you know a YouTube rebuffering events by a s I, don't let me share them publicly, but I wondered if you'd consider it collaborating with them to verify the kind of measurement results that you're getting or not. Not. Necessarily, you know YouTube other other service providers as a way of at least informally kind of validating the results that you're getting I.

R

Think that iam would be very happy to do that. I, don't think he has done so, but I think it's a very good idea, of course, to try and do that and I mean in general to correlate the measurement results from from different sources. It's of course very important. Yes,.

W

I work for one of the ice Peace in Sweden, so you're, treating seeing that that's tolling is an indication of unfair treatment of the traffic. Is that what the paper says you see Lester strolling in YouTube. That is indicator of some differentiation in the network. No.

R

No I, don't think that's what the paper says when were what the intention is so did these were actually some different measurements that have been done so he's used the platform to look at you know various aspects, so the the work on the buffer stalling is not directly related to the net neutrality tests. Okay, as.

W

Far as I understand.

R

It I mean I have to know.

W

The test was serving him to confirm this, of.

R

Course, because the.

W

The timing thread disappear and check. I know a little bit of trouble, classification of traffic for other purposes and I that is not in. If that was the indicator, our timing, some differentiation there you will- that will then be implicit that there are some I played patrocle than is the trigger for a differentiated treatment of the traffic. So I don't understand what that is, how that can be an indicator and it differentiated treatment. The timing in that disappear so.

R

So maybe this was was unclear in the presentation. I think that the three examples that he has in the paper for for the type of measurements that you can do in the network, or not directly related to the design of the net neutrality tests that will be used for for measuring that aspect. So this is I think those tests is something that he is currently developing together with discussing with pts.

A

Your god, I have I, had one a bit follow-up on Randy's point, and so, if you look at the, if you look at Atlas and ripe and cheese pie and probably half a dozen others, there's mommy and munroe probes, and so was there any conscious market placement decision behind the development of sensible. Is there a specific area that that ian had in mind when building this thing that the others wouldn't cover, because most of them are also geared towards home network watching from the home network perspective, different complexity of measurements?

A

Of course, I'm just curious, I.

R

Think that when he started from the very start he did not have that set perspective. Now I'm. You know just guessing a bit here, because it's not my work, but it started almost like a hobby project in some sense, yeah I think there, the raspberry PI's doing the measurements time very early on. He got in contact with with pts, and they had done an interest in getting some formal measurements in Sweden.

R

So that's when the collaboration started, I think that when he started he did not have that perspective, but it came in very early in the process, because pts was very interested in the type of platform that he was developing and he already from the beginning, targeted the home use case.

R

So I mean that I think was the intention to measure what what type of quality you see as a home user, and it started with looking at this video and media events and then seeing what type of service you got for those those were the first measurements II did as far as I know. Okay,.

A

The other question I had can.

E

I add something at that point, because all the measurement platforms you mentioned, you see usually very restricted in the amount of data you can send on the on this platform, just the way they are designed, which doesn't allow you for like, for example, high speed tests and stuff. So I think this is probably more comparable to SEM knows, and it's really also different set of tests. You run on these platforms and you also have a different agreement with the users who deploy those platforms because they know that's regulator test.

E

So it is a little bit of a difference, but I think there still is a need to it'd be.

A

Nice to kind of map this out and what.

E

A

You what do you use for which, as an external user, who might be interested in and also which put which platforms like? What for what that? It's also.

E

A good question: can you, as next owner user, run any measurements on this platform? Probably not right? You can probably talk to Ian and ask him if you can run something for you. That's close enough.

R

Yeah I think right now there is not like open external interface, so you can run it you. You can design that the tests are fairly easy to design. As far as to understand it. I know he's also used it together with students and they have designed you know tests and, and they have run it on the platform, but there as far as I know, there's no like external interface to access it and run it so.

X

Since your said, we could map out the tools we have a research group. That's called the map, research group, that's measurement, and maybe that would be an interesting thing to put together as kind of a directory of the available and how they differ and how they're the same yeah.

E

I mean there is, which is also mentioned as related work in this paper. There is a survey on measurement platforms, but we could definitely look further into.

R

That and I guess you have started to collect some of that impor at some point. It was discussed at least to try and do that in in my party right so.

E

In MapReduce so far, we we gave people a platform to announce their measurement platform, but we didn't do any effort in trying to compare them or kind of figuring out what the different abilities are. The platforms provide the.

A

Other small question I had about it was about MP TCP since just mentioned that there's no cellular right now planned I'm wondering how you do multipath TCP from a home network user. If your Wi-Fi, if your Ethernet and your Wi-Fi connect to the same ESL line or whatever, is there anything you.

E

Have two IP addresses.

M

Who I mean answer into this you're? You just use different port names and you exploit multiple paths up on the network.

M

Right but you're, but you're categorizing paths.

E

So given this is given, this is an IRT F workshop and we're already talking about mighty pass. Tcp I would also like to announce that there is a new proposed workshop group workshop research group on pasa, where networking which looks into path selection, these kind of things where the first meeting is held on Wednesday afternoon yeah. So if you stay here for the week, that might be interesting also come to member G. The meeting is Thursday morning on measurements and I would like to get to one more point.

E

We actually have like 10 more minutes, so we keep chat about that, and so the thing I like about this paper is there's one sentence in the introduction saying: small data can be of value to.

E

And I think it was a little bit the scheme of all three presentations that were, they were trying to minimize the amount of measurement danger they were taking and so I think. That's a good thing. I think that's the right thing to do. Instead of just collecting huge amounts of data, and then you don't know what to do with it. But I. Don't really understand that much to savannahs question, because I think that somehow related you try to minimize your data is so much that you cannot guarantee that it's statistically significant anymore.

E

And so then you have the fairness issue, but if you just make sure you have enough data that it's significant there shouldn't be a sign of preparedness, question anymore, because you actually measure what you want to measure nope I.

R

Guess it depends on what you try to measure it has to be statistically significant for each operator, I guess in order to to be fair for some of the measurements we do it as researchers. We may not look for that distinction that you know we actually want to say something that is generally relevant on an operator level. We may look for other other phenomena, so I think it's it.

R

That aspect I think is related to the particular aspect of what you need to actually say something about the operators and capturing their behavior rather than capturing protocol behaviors or things that appear in the network, but I mean in principle, I, I, agree and I have not that if you have sufficient amount of data to have statistically significant results that that should take care of that.

S

Spencer dawkins it's. It seemed like to me that one one of the things I was getting out of this was that regulators could use this not to say that there is no problem but to say here's something that we should look at further. You know so that youth doing small amounts of doing tests was small amounts of data, and things like that was valuable to say, you know to attract their attention on something where the alternative right now is just paper surveys, so you know so I thought there was that aspect to it as well.

S

That's not saying that you know it's not saying you've taken your last measurement, but it's saying that you could have used the ones that you've already taken.

R

Yes, I think, certainly it's not intended that you just measure and you have all the answers. I mean you combine it with techniques and you may need to look further into aspects if you find something, of course,.

F

Really push rij on a totally different research project. The paper said in the introduction and in the abstract that it exploited existing traceroute measurements and made no new measurements. Two-Thirds of the reviewers said you created a lot of unnecessary measurements.

E

That's probably also good come in, maybe figure out which measurement data are there already before you try to do new measurements, okay and I. Think with this we just closed a measurement session here thank very much Anna.

E

And then we have coffee outside and see you back in 11:15.