IPFS IPFS þing 2022 - Measuring IPFS, 10 Aug 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Unconf 1: Parts of the IPFS Stack that could do with more monitoring - @guseggert - Measuring IPFS

Description

No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).

A

Yeah, so this, uh hopefully this is more interactive than me just talking at you um because we're getting into like the uncomf section of it um yeah. I think we want to have like a conversation about like what do we need? What kind of measurements do we need to like, improve on or add to the ipfs network or like? What are we missing or are there questions about how we can implement certain kind of measurements?

A

um So I wanted to seed it a little bit with like. Why are we measuring stuff in general, because there's a lot of different reasons that we measure things um so yeah, I'm gus uh my github handles gus eckert. um I am on the protocol labs ipfs steward, so I work on kubo and the ipfs protocols.

A

um My background is like I came to pl from aws, where I worked on web2 services, and before that I did a b testing, so it was like a lot of thinking about metrics and how to how to make sense of them, um and mobile experimentation in particular is interesting because it's a lot like the problems are very similar to the problems we have in peer-to-peer networks where, like nodes, come and go randomly um and those kind of problems um so yeah.

A

Why are we measuring stuff, uh like one of the one of the big reasons we measure things, because we want to know that things work and when they stop working right, um so we can catch bugs.

A

um Also, like the I found that, like the process of defining what working means is actually it can be pretty difficult and being able to like once you figure out what working means being able to observe it can be even more difficult, especially in these peer-to-peer systems um like, for example, in the hydro boosters uh one of the big problems with them.

A

Is we don't know how to tell objectively like if they're doing their job, which, like is like to help out the network, basically the dht network um and then, like reality, turns out to be pretty surprising, sometimes like a lot of times? We think that reality is one way, but then, when we look at it, it turns out to be pretty different from what we expected.

A

um It's like having a little bit of humbleness about about uh we're working on or the data can. Actually you.

B

A

Imbue humility in us um we're really bad at predicting the future, like einstein didn't think that nuclear energy was feasible um and also like, like external events, happen that nobody can predict like covid, um that dramatically change people's behavior, and if we have the data there to already understand what's happening, we can adapt pretty quickly to it and there's also particularly ipfs there's a lot of constants like in, for example, k buckets.

A

I think dennis was talking about this yesterday um and uh yeah like assumptions that led to those or maybe there were no assumptions- I don't know, but um they can change over time, and so, if we're watching the data that led to those assumptions we'll know over time when we need to adjust constants um as a developer, this one rings true to me: uh you're, always like ramping up people.

A

People come and go uh working on these problems, and if you have the data for them to look at, they can ramp up a lot more quickly rather than having all the domain knowledge in your head.

A

Emergent behavior, especially in these peer-to-peer systems, can be hard to predict uh because we have like these simple algorithms, but then, when they get together in swarms of nodes and stuff, they can behave in unpredictable ways um like trying to figure out what kind of metric we would use to detect this feedback loop here would actually be pretty interesting.

A

I don't have a good answer for that.

A

Oh yeah and then also to keep us honest on what we're working on uh so that we don't end up working on problems that nobody cares about, but that we think are particularly interesting.

A

Yeah- and so I also, I also feel strongly that sometimes metrics are not enough, because they they um discard information by design and sometimes the information they're, discarding you care about, and so like. You also need to verify that your your metrics are the right metrics by looking at the information, you're discarding and there's a lot of ways to do that. Right, like we can look at individual cases and logs and randomly sampling and looking at probability distributions and um like, for example, this guy on twitter was complaining about a bimodal distribution.

A

He was seeing when he was pinging, uh probably google or something um uh can anyone figure out why there would be a biblical distribution here, or does anyone have a hypothesis about why your pings would be bimodal like this, um so it turned out at noon on wednesday his his latency went down just a little bit um and and so like by looking at this this graph here you get by. Instead, if you were just looking at a number like, you would never see this information. This is a lot of information right.

A

Something happened at exactly noon and then exactly noon the next day. It looks like the variance might have increased or something- um and so just by like, like breaking out of just the metrics bubble and looking at graphs and distributions and stuff like that, we can get all kinds of interesting information about how the network's behaving um so yeah. What do we already measure? We already measure quite a bit and it's and a lot of different people measure. It too, I think, dennis, has already shown the nebula crawler.

A

But if you go to stats.ipfs.network there's reports that are generated generally every day that have let's see if this load there we go version information about kubo nodes. Some of them may not be kubo nodes.

A

You know we get uh interesting, interesting behavioral patterns like one-off nodes that appear and then disappear immediately, um churn, which is these are probably some of the most interesting graphs in this in this report and then there's a lot of like geographic distribution or is it yeah where you can see like what countries nodes are are coming from.

A

So that's that's one source of metrics that we have protocol labs, has a bunch of internal dashboards and I'm sorry that they're all internal, but that's just the way. They are right.

A

Now, um for example, our we, we run the ipfs.io gateways as well, and so we get a bunch of great metrics from that uh if it loads like, for example, want list sizes from bitswap um all these time to first bite metrics from nginx and the list goes on and on and on latencies and errors and timeouts, um and because ipfs.io is so popular, it can be a pretty good window into the network.

A

um We also collect a lot of logs and have elastic search dashboards. I couldn't get this one to quite load, but you can see like, for example, we can break down.

A

What was that uh it does? I don't know why some of these are erroring out, maybe the interval's too high or something. But um someone needs to fix these, uh but I mean just this graph alone is pretty interesting. Like you get the devices that um that are requesting data from the gateways.

A

There's a a nice content, routing dashboard that we have so we have a system that periodically puts records into the network and then measures how long it takes to get them back out, and so these are like the kpis that we track for the dht, for example, like this one is pretty, is, is pretty important.

A

uh It might take a while to load but yeah. We have this system that that does dht queries and puts and stuff and measures things like, and we get an overall view of how the dht is performing.

A

The hydras that we run for those of you who aren't familiar the hydras are dht nodes that are scaled up to basically, so that any dht query has a very high probability of hitting a hydro node, and so they have a really good view of the network from a dht perspective.

A

um So you can see the process like a ton of requests with. This is somewhat like, like we can get a good idea of how much traffic's flowing around in the network just from this graph alone. um If we look at the database, we can see like. Actually, this is interesting.

A

In the last day, there was like a huge surge by there was like a billion new records that were put into the dht, um so we can get like a good global view of the number of records floating around in the dht, um and then the number of records that expire and get evicted.

A

The maintainer for rust, the p2p, has a a small dashboard that has some dht metrics in it as well.

A

Like unique nodes and their their times in their countries as well, um and then this is a pretty good website as well by this group called trudy. Oh wait! Not this one! This one and I'll send these slides out after afterwards, so that y'all can get these links.

A

Unfortunately, these links under the pl dashboards are probably not going to work for most people.

A

Yeah there's a nice graph of the number of nodes on the network over time and then agent distributions user agents coming going um yeah.

A

And then some examples of stuff we might want right, there's a there's, a really awesome, github repo um that has requests for measurements in it. So if you all have ideas or needs for measurements that you want that you want to get prioritized, you can submit an rfm here and describe like what the measurement is and how you would measure it, and then I believe there's grants um right. There's uh bounties on these for people to implement as well.

A

So yeah there's a lot of cool cool ideas in here like measuring dht. Some of these were in talks today, actually like dht routing health and um bit swap.

A

um Yeah uh some some things that I have like like like was just mentioned in the last talk right, like bitswap, is sorely missing: lots of metrics like the number of cancels and how successful is the discovery process when bitswap broadcasts peers out right, um latency house, like how useful is it that bitswap?

A

How useful are the bitswap sessions, I think, is an interesting question to ask, uh because we don't have any data on those at all. I don't think um leeches in the network. I think we were just talking earlier about the idea of collecting data from network nodes, like maybe each network node could have a us a little server that responds with some metrics and then we could just crawl the network and get metrics from every node about what their experience looks like on the network yeah.

A

These are just some like examples, so I want to like see from other people what they think we should build and hopefully like the out the hopefully the result will be. You know a better understanding of what we want. Maybe some new rfms or um yeah.

A

Anyone have any ideas or anything. They feel strongly about. Yeah.

A

A

Well, I think if other people don't know, I think the hydras have an interesting behavior, that not a lot of people realize I think, and that if they get a request for a record that they don't have they'll return to the caller, that they don't have it. But they'll also start a background process to go, find it, um and so that can create some extra work on the dht.

A

um But it the network, like it's they're, basically trying to make the dhc healthier by by caching more stuff.

A

But if there's a lot of like unfindable records right, uh then it could just be creating a lot of unnecessary work.

C

I'd like to hear from people about um you know the nodes themselves emitting more stats um and us being able to query the network. You know live and like what people think about that. As far as one security to like performance.

A

C

um If they I mean, even if it sounds like a decent idea from the get-go, but I would love that, like with a focus.

B

C

A

C

Specifically, I think that's the ideal scenario like without any other considerations.

C

Does anybody disagree with that.

D

um It is an issue when you're a provider like that for that to work, you need those stats to be available from outside, where, typically, your knowledge are not accessible. That way. What if you trying not to because that access to infrastructure, is it bad.

D

So if you do want to do that for some provider that might require some kind of access control, maybe or something.

C

Yeah, like some some sort of trusting of the other ipfs nodes.

D

Yeah right, but that might be not the right target because as sense provider, we do have matrix different kinds of nutrients and make sure that not much people are actually.

D

C

Just focusing on people.

C

Just making each node publicly.

D

C

D

Like not just like random people that run that are on the network, normally might be enough to get useful information about the network engineer without those big content providers.

C

Kind of I think, I'm missing something, but.

C

I mean how do you measure the network in general without knowing what the pieces of your network are doing like that's that.

A

To me seems like false.

C

Premise from the get-go right, like you need data on the pieces to get a.

D

C

Like you can't find that final solution to an equation without understanding what each variable is right,.

D

Yeah, just just yeah just like to know that cell is provided, are usually protected in some way, or it might not looks like a normal.

C

So I think your call out is that some nodes won't be able to provide that data like if they're, if they're.

B

A service provider.

C

Or other node, okay,.

B

A

How I mean what percentage of nodes is.

C

That on the network, like do we even know, I think that's one metric that we think that's a good question.

D

Yeah, what percentage of nodes on the network are service.

C

Providers, you know what percentage are our nodes that aren't individual users, which percentage of nodes are trying to make.

D

A profit inside.

C

Of the network, you know, for whatever reason, which ones are commercial which ones are.

B

The different question there would be less about the number of nodes, but more of the amount of content.

C

I think both is relevant, but the number of them.

B

Is like giving an interesting metric? It's not necessarily what you want, because if you're an.

D

Individual you're, most more likely to have like one unique identity.

E

Well, I mean, as a provider I could decide to have like one identity. It was like, like a lot more dishing behind it or like have.

D

E

Identities distributed so yeah.

D

C

But there's been some.

D

Work on that as well, there are some papers trying to identify who is yeah.

C

The one caveat with everything that I've said so far I will say for the rest of this conversation is that I have not looked through that request for metrics repo at all. I just found out about it, so so please, let me know if I'm way off base.

E

No, no definitely um I've been tracing within the nose has been something we thought about. The there are a few things there. One is what would be useful? The second is, what percentage of that would make sense to um share with other nodes.

E

uh Without you know, you don't want to overload the network, and it knows with too much traffic.

C

So, what's the size of metrics, we can share without network network declaration.

E

Yeah exactly, and also what is not kind of touching on any privacy issued by notes by you know, sharing your stuff with others. um Yeah definitely activity on the network. But I guess there are maybe halftime and.

C

Bandwidth consumption appears.

E

Number of peers.

C

That you're connected to like is that.

A

C

Is that a security question.

A

Yeah yeah, I don't necessarily think it.

C

Is but I guess you could argue that it is in some way.

E

I think, in order to do this exercise, it would be perhaps good starting from the aussie side to say what do we want to use these metrics for all right? So do we want to use it to have better ideas here within table uh entries, uh or do we want it in order to choose peers that are more closely geographically closer to us to connect.

A

Or request content from, is it like what is it and then we see as a as a kubo developer? I'm really interested in like what people's experiences using software that I work on and like. I don't have a good understanding of that, because we don't really collect that kind of it's like annoying like gathering information from all the notes about like have they been having a good experience right, if not like why music experience.

A

Yeah both, but most like, ultimately, the user experience right like which, which is ultimately what we all care about right.

C

B

C

What is what does that.

B

C

Mean as far as metrics go user, experience of a kubo note is that, like do.

F

A

C

Experience if all of my wants are unresolved or like what what criteria yeah gateway.

C

Yeah, what does user experience mean? We should define that for sure yeah, but for feature development. Be honest to answer your question I think is for for me, as you know, I'd be stewards, team, member, yeah, working on desktop and pokey um yeah like what how do I know that, for example, one question you know we have. I just looked at this yesterday about a thousand um active users. You know unique users per day from ipfs desktop right, okay. So how do I know if those.

A

Users are benefiting the network.

C

At all, like are like some metrics I can get myself like. Are.

A

They actually keeping desktop.

C

Open for a certain period of time, we.

A

Have some of those.

C

Metrics, I just haven't dove into it, but like at the network level, our ipfs.

E

Desktop user is actually benefiting things or.

C

Hurting things um you know like: okay,.

E

How would they benefit things.

C

By being another node on the network to provide content, potentially caching content um that they are looking up, maybe they're long time, maybe we have one to five desktop users that are.

E

C

A ton of content, that's very popular. um Yes, I guess there's a million ways. I don't know you know.

C

But then, like bit, swap bit swap stats that we talked about. I think that that that's like why.

E

D

C

Bit swap stats and that's to help inform the efficacy of the.

D

C

Right like a bit swap as a protocol is not efficient or there's a lot of things that are failing that we don't understand deeply. Then we can say that our network is performing underperforming right. So, I think is our network. Underperforming is one goal for bitspock.

A

Network metrics.

A

We know that it is on our gateway. Bitswap map turns well. We know that bitswap has trouble on our gateways like because that's.

C

Is that the case for a regular nodes, why why wouldn't it be different? I guess it's my question. Well, the gateways have processed.

A

Way more traffic than uh an.

C

End-User node, so then, as like what metrics do we need to determine whether, in a large amount of traffic, is the cause of the bit swap problems versus every single potential request? Has a potential for? Does everyone suffer from this, or is it just gateways? Yeah yeah.

C

um And I had a question about whether a particular thing, what would fit as a request for metrics, I guess like, is this discussion a relevant.

E

Item in request for metrics yeah yeah, it is absolutely yes, anyone can go and start a pr or start an issue and discuss ideas that they have that we think are going to be useful and yeah. If you go to each rfm. Basically, there is a short description and in most cases, not all of the cases, but in most cases there is a short description of the methodology as well like.

E

How would you go about doing it or you know suggestion or thoughts and stuff like that, and once that becomes kind of priority, then we need to work harder to come up with a methodology if we don't already have it and uh actually yeah. So, of course, you're more than welcome to start any number of these excite yeah.

E

Yeah and it would.

C

Be good to have a list.

E

Of all these things that you just mentioned, um you haven't documented somewhere.

C

D

Do we get this recording.

C

Even if it doesn't post to youtube.

A

Did you have something antonio.

F

Yeah we are talking always about like.

F

Itself, but I would love to have some benchmarks with time consumption cpu right to be able to detect problems between persons before launching then you like who awaits them to. Let go you know, so it will.

F

Information beforehand.

F

Strange uh so we we have like a specific use cases like queries like um using some part of the database, that we knew that we had some problems before so we executed that queries on all the all of the vectors of the database. So we can easily see some spikes on the respect if we did something really wrong or some cpu times. That's where, like totally different from the previous version, or we can clearly see if some improvements that we implemented, they actually improved.

F

Something yeah so have some kind of like benchmarks, benchmark like use cases like okay, let's say what happened if we have like five notes- and we start like moving around records of x size, we don't actually have a lot of that kind of stuff.

F

We don't have a lot of benchmarks here. No, that's the other door. I think I mean. Where is the place it like takes ground or yeah yeah? Maybe that's around and bring some graphs or something that you can like. Compare between, commits or builds.

A

Or have ideas and and test them out and record some metrics on yeah.

E

They have some testing.

A

Environments that receive they get like mirrored traffic from production. Basically, the production servers will just forward copies of all their requests off to some posts that you can just do random stuff too. They don't actually responses, don't go back to anyone, um but that's only for gateway stuff right.

A

We'll put it on a gateway, node and watch the metrics and make sure okay, so this is done somehow very primitively. Okay, like that's, not a good, that's not a good way to experiment with tweaks to network protocols and stuff right yeah. You want like a little microcosm of an ipfs network and observe how the nodes behave right.

A

We could try to create a realistic simulation, maybe test ground and then to be able to deploy any version of it. But then how do we have like hardware measurement on yeah? So I mean that's the selling. That's still like. That's the reason test ground exists, so if they can't do that.

C

Having a reality, I think, because.

A

You can it has knobs where you can tweak like latency and packet loss and all these things to like simulate certain networks.

A

Observation and create a level yeah, that's one yeah.

A

G

I was checking the artifacts to see if this existed already, but if you go scroll to rfm16.

G

uh Just one on the big swap efficacy, basically yeah, but what I was wondering is instead of generating a bunch of cids and sending them out is looking at the want lists that are out there in the wild and then seeing how long those cids stay on the llamas in the you know, kind of, like certain polling, appears it's a little rude. I suppose, but it would be really interesting to see in practice how long something stays on a certain peers want list or a sample of those yeah.

C

And these type of individual.

E

C

Metrics are the things we can't get from simulated networks.

C

E

Interesting, but how.

E

Yeah, so we need to find one lists instead of cids to query right.

G

Yeah, I remember I mean I experimented with this way back in the day, probably like 2019 or something like that, and you used to be able to query another peers want list. But I don't know if you can still do that.

E

And then query again after a second or something yeah.

G

Yeah just pulling for a little while and see if you know if it shrinks if it goes up or down or whatever, and then that might be uh an indicator of a user experience.

A

D

That's somebody.

G

Looking for content and waiting for content yeah, we could roll that up into statistics. We collect from nodes right average want this time or something like that. That'd be interesting, yeah.

C

So you can ask another peer. What it's want list.

D

Look, it looks like right now. I.

G

Think that's what he's saying right. I know that you used to be able to do that. I haven't checked in a while all right. It's probably my own. Like my own problem, you know: okay, okay, of course, I've never done anything.

B

I think the important thing here is that we have this discussion publicly in this kind of repositories. Yes, like now that we all in this room know that this did have repository exist. Like uh I mean you need to measure what people want to be measured like we, we can have just a close idea of what needs to be measured, but it's also interesting to know like what other people think that is.

B

You should put that all those things here and make issues or put requests to the existing reference like open. Also the discussion to people that will be interested also point out, other things that we are missing. Do you ever make it here for them it's just a way of actually like.

E

Yeah yeah as an idea for archon, yeah, yeah and yeah. We address it all yeah. The pr could change the measurement plan there.

B

E

Same way like if.

B

There is any other everything that would potentially be also cool to have like just feel free.

C

I opened issue number 66 and I'm just gonna try to jot some notes down there, but it is open description. Is there and and just informative of what I'm saying right now.

E

But add a comment.

A

C

As for taking notes or something really hits you because I won't be able to keep up with it at all right now, yeah number 66 protocol, slash metrics issues, 66 network measurements, it's in slash, metrics, but wait. That's where the rfms are. Isn't it? No? No, no well I'll.

D

Move it or someone else move it.

C

Because I don't have permissions.

C

I'm trying to get the errors out of the way, so everyone else can just drop their good ideas down.

C