IPFS IPFS þing 2022 - Measuring IPFS, 10 Aug 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Monitoring Cloudflare IPFS gateway - @thibmeu - Measuring IPFS

Description

No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).

A

So, hey everyone, uh I'm, TiVo, gonna, present uh more or less how we like monitor like a uh the class ipfs Gateway, which is slightly different, I. Think from like the talk that has been this morning, which we're more focused on measurements.

A

um Of course, monitoring involves some measurements with like more targeted like operation um and how like we can like best like see the house of like our active system, depending on, like the health of like the overlap, PFS, Network um or more generated this kind of presentation, so um yeah as I, say: I'm TiVo, I'm, a research engineer at cloudflare I work mostly on distributed web project, ipfs, ethereum and samosas tend to like croissant um I mean there were no croissants. This.

B

A

uh But then I did not know that, but like I was writing the slide, uh but definitely like. If there are some personal happy to like meet you next to the croissant, like always fun discussion, uh The Talk today, I will like present an overview like a high level overview of like what class the system looks like um then uh I'm gonna present.

A

What is monitoring um like what are the different systems we're using um and then, where we think like there could be some improvement where we go next um I'll try to be short so that like they can also be time for questioning to move like.

A

Should it be starting the experiment based on like this infrastructure or like questions uh like more generally so um class architecture, which is like kind of the place we go home at la flat because well, that's where you are um so yeah, uh it operates an ipfs Gateway, which is an HTTP interface to the ipfs network.

A

um More generally, if you like, take like the the eyeballs, which would be your end, client would make an HTTP connection to like classifs.com or like another domain, reaching out like lifeless Edge, so cloudflare's Edge um is composed of like 200, maybe more like point of presence uh where we have like our metal.

A

That's where we operate like um some cloudflare workers, some caching, uh and then what we like try to like basically like what makes like the Gateway fast in terms of like we cache the content as close as we can to the users and like try to have like most of the logic there.

A

um If the content is not in the cache, then we reach out to like uh ipfs nodes um that we're running uh in the background, and these nodes are actually the one that are in contact with the ipfs network. um It's not Kubo nodes, uh but it's like um heavily based on the historical go ipfs as a library, so there's like custom setting for the nodes should be liking to move appearing. The data store um some of the strategy around like what kind of verification we do.

A

For instance, we like always perform like some like Genesect, even on the client, so like, like there's some strategy strategy um for at each of these steps. We would like to inform and know some metrics, and we would like to to better understand like uh like how they work, and so that kind of, like two types um ads of data degeneration that, like there's, active monitoring uh that we perform with the ipfs, monitor um and then there's like more uh passive monitoring that is generated, as request comes through.

A

um Of course, the ipfs monitor also generates some like requests which go to our data store, uh but not on, like the request going to the data store into like these, various um stores are generated for, like the traffic, um we'll go in more detail into like how we use like each of these um system and this data store.

A

um But overall, at the idea.

A

What's also difficult is like why the system from like the outside may seem simple. You have also a lot of cases which could make it harder to analyze and understand like what's happening when you start to have an issue.

A

For instance, you can have like a chain of service, so, for instance, instead of like accessing cloudflakefs.com, you may have actually like a cash Gateway in front which, for instance, protocol Labs operating with nft storage, then the same Gateway may also be chained with like another customer, and so um one of the challenge is like trying to understand when we have an issue find the root cause, what the pass, what it's going to um and what's even more interesting, is like each of these, um of these Services may have like their own caching strategies, their own invalidation um and trying to find a good view um through that could sometimes be challenging um yeah.

A

The main thing is like kind of the attitude when trying to debug. What do you look to? What do you look for? How do you um understand the data and the leg be those graphs?

A

So um if we take like a look at like the monitoring, uh which is kind of like I, mean mandatory from like an operational perspective, because you need to understand your gateway start to answer, I mean 10 seconds instead of like less than a second, then like you need to understand like what's happening or like. If there's like new things into like concentrating Etc, you need to be like alerted and have the tools to to debug that live.

A

um So the first one I mentioned was like Prometheus or premises time database, um one of the kind of like particularities like you, can have like only low cardinality on Prometheus. So you don't. You tend not to have a lot of information um like in, like tagging in context available around, but it's very useful because, like you have like you, can like easily follow and track service on time.

A

um It's limited to numbers, uh while you can have like a huge set of numbers, uh I think it's limited on numbers um and like we allowed like for live monitoring. um So an example like um take it like what a Prometheus uh metrics looks like um you would have like a name and there's tagging. So, like five PFS, uh you may want to know like what the total number of requests um that have been uh received on on the on the back end. You like can tag it.

A

For instance, like yeah I have like a kubernetes namespace, uh the Handler which, like what the Gateway and the type of method that has been received, um and all that like allows you like, then, to perform some computation over uh to get like those like beautiful graphs um on grafana.

A

You have a very set of metrics um that you can have Prometheus So like um Kubo. Definitely like exposes a lot of that, but you can like also very easily like customize um and add like uh metric. uh Should you be like for timing or or forenses um through that?

A

Another thing which is um like quite interesting for um for following what's happening and for monitoring a Sentry, um so different compared to Prometheus, is like Century, like. Usually, we tend to focus on errors, because it's great at like giving a lot of context and like a lot of metadata around, like particular events that happens and the good thing with centuries like you can also like um map it uh to like the actual code that has been generated.

A

The error um and so I tends to give a lot of context in terms of like when you have an error. You like already know like how many times this error has been happening and like which piece of code is actually responsible for generating that this error. It doesn't give you like time in context, so it doesn't give you uh more um High School uh they're, like full trades, but like it's very good to pinpoint uh on the instrumented code, where, actually you have an error generated, it may be more Global.

A

You may need I know like it may be like a causal thing, depending on the context, but at least you would know the point where the the error has been yield.

A

um Finally, the last thing I want to mention we like using clickhouse a lot, um so clickhouse is a more it's probably like out of all those stores and more traditional uh database. So it's a column based database which allow us to like store arbitrary data.

A

um This data is also sampled so like that allows when you have like an event that, like is repeated, a lot to just not store this event like uh as many times as it happens, um it's very useful for like these kind of arbitrary data, so, for instance like if, like you want to know which CID is being accessed the most or like for uh like which, from which um domain, which kind of Ip you like which country Etc.

A

um This is where you would store this data, um it's um more suited for like long-term storage uh than like the the previous data, um and you can like access and filter um these data to like normal um SQL queries.

A

um Basically, this is also the same backing platform analytics. So, in terms of like the logic from like when we started like operating the service, it like method of logic, because we could already leverage the existing infrastructure without having like to manually operate, our own, like cluster, uh should be like postgres or clickhouse. Cluster first thing, I wanted to mention is like the ipfs monitor. So all the previous tools, I was mentioning uh were about passive monitoring.

A

It's like they do receive data and then like we try to build query and build model on top of them. To understand. What's happening, um ipfs monitor is a bit of a different thing. It's like we started to see. A lot of, like sync, could happen on ipfs, but uh we didn't have a good tool to actually like measure life. How, like these scenarios were performing so I, don't know like the publication of like an ipns name if the ipns name has been cached.

A

If it's not cached, uh the publication of the CID is the access to a DNS link, um and so basically the ipfs monitor is really like this kind of like um active probing in a black box monitoring way, um so that you can test like these. Multiple scenarios export them through Prometheus, and so like you're, also able like to track them, live um some of the results we got in, which, like are like family described like on the blog. We have uh for like the newly created content and in Cache content and unavailable cids.

A

We just do measure all that and like we tend to see that, like like, corresponding to like the data that had been shown this morning. Publishing content is actually like, take some time um and yeah.

A

Now that we have all that, there's definitely like a lot in terms like models things we could like already improve on that. But the main question is like: where do we go next um and sismo kind of a like less of things, rather than like uh precise, uh precise step?

A

um One of the main things like we would like to do and like we'd like to see, is more like long-term, like long-term rating of the peers, we're connected to, um and also like tracking of these peer I. Think like at the moment like when you're, using Kubo like when you're using like go ipfs.

A

um You have like all these like geologic, which is kind of like more abstracted um and there's no like easy, well streamlined way to actually like, say: okay, like I, want to like keep a snapshot of my peers like over time to understand like which one are actually reliable, have more control in term of like the latency I think, as Gil mentioned this morning, it's almost like the actual response, or the bandwidths are already requested from them. um There's been a lot of tooling that has been done like on BitTorrent.

A

To try different strategies should appear like be malicious, you'd, be less malicious. Is it more of a liter, uh Etc and I? Do you think having this on ipfs would be like really nice.

A

um Similarly, content providing and concentrating is like an area for improvement and also I. Think, like the the this workshop and the work being done. um To that extent, to actually like measure and understand, like What's, Happening Here for different strategy uh is important.

A

um Something that's less mentioned usually for ipfs is like, but more of a problem for Gateway is like the names and the resolution that are associated because ipfs content is not only accessed through a hash, but, like you, tend at least for Gateway to access through a DNS name. Actually, an ipns domain, even accent like through ens, for instance, and, like all these resolution have cost.

A

um They do have like different, like standards in terms of like the TTL, uh in terms of like the way to organize the content um and so being able like to track and standardize. This small uh would be quite interesting as well.

A

Another area to actually like measure is like is actually like. The content layout like on ipfs, um you have like these cids, which, like each of them, uh has may have a different layout.

A

Actually, like is there some overlap to like better inform like the production of content, so that, like you actually like Leverage, uh the fact that, like you, can share blocks uh for multiple content and multiple cids um bits of natureing, um that's included by default, like in Kubo, uh is rather primitive um and so having more uh more View and vision into that would be quite nice, I've seen, there's been like some work uh with the new clients that are being pulled out uh to to actually like extend these bits of monitoring.

A

um On top of like adding the bits of like improving bit swap itself, and so exposing it from a Gateway perspective would be rather interesting.

A

On the another note uh distributed tracing to like actually improve advanced level of the Gateway uh would definitely be an improvement. It's start to be integrated into Cuba, actually to some extent. uh Basically, the idea is like um when you do a request. You actually understand like where it goes and where it spends time um at the moment like in Kubo, uh it's kind of a no-low, nothing from what I've seen it's like.

A

uh You can like start it like, really and refocus on debugging so like if you like, coding like developing Kubo and like you, want to debug, you can actually like start like a Jager instance and have the traces distributed, but ideally you would also like to operate that like in a production setting, uh so you would like on your production gateway to pass like some kind of like on some kind of header or some kind of setting, so that you're able like to trace uh the the request for like your whole system.

A

um That's something we would have a cloudflare for um most of our infrastructure like we know how much time we spend in the cache. We know how much time it takes to like do a DNS resolution, uh but also gaining and like uh pairings, that, with the visibility we could have um in Cuba and go ipfs would be quite interesting.

A

um Finally, on kind of same note, integration with Enterprise tooling, uh is not a goal of Kubo and it's not a goal of go ipfs.

A

um It's been at least like for operators like one of the main challenges in like one of the main area where, like there's been time spent, um and so definitely like um concerting uh these effort and like thinking more more about it, I think uh is one like important step uh to go to should be like for measurements, because, like I mean the more people you have involved in the measurements like the better, the measurement is, or at least like the more data you have in that you can filter and understand if it's better um and and also like on those areas uh for for these challenges um and yeah, that's it.

A

uh Thank you. I wanted to keep that short and open for questions.

B

Yes, as much as you're comfortable sharing, can you talk about the questions.

B

A

uh So basically, uh ipfs node is based on Kubo. Now ah Kubo is a library, even though Kubo is meant to be a node so like they probably like some naming them, uh but yeah, basically like we. We have. uh We just go ipfs's Library, um so that, like we can have like a custom, caring strategy. uh We have uh also we uh we're creating, like certain like internal kubernetes cluster to cloudflare, which, like we are not comfortable, exposing the IP of, um and so we have like, like a forward proxy for that.

A

um We have like a custom data store implementation so that all our nodes can actually share the same data store in the back end. So, like you, also reduce the number of like requests you're making to the outer ipfs Network.

A

um This use of, like Kubo is a library also like uh very helpful to like introduce new features should be like for research, for instance, like all the DNS stack, uh verification validation is something that just doesn't exist in Cuba and so like that's something we we add on top um should be, to like add the caching headers, uh which now have been upstreamed, I. Think um and just like very strange. This is things through that is there something in particular, you were interested or okay, yeah.

A

uh What do you mean duplicate peer instances.

B

A

So not at the moment, uh but definitely that's something like we are looking to do at least like um first of all, like evaluating what is This Global set of peer, we should peer to um I think it's already like an important problem, because you don't know which p is actually good and you don't know how you should update this thing um like, for instance, like ethereum, has been doing like this.

A

Is some of this look who's like the East disco, I'm single, like you can actually like just let the demon run and it will just like export. You uh like a set of PS with like against a certain like like rating um thing for ipfs that doesn't exist yet. But yes, that should be the case.

A

um All these peers share a set of like curated. um Like all I know, Shadow sets are created peer which, like they're appearing with um and yeah, uh just behind your list,.

B

B

A

Says I thought something we're doing uh at the moment or something we're thinking about. But re like we need to consolidate, like the um like, concentrating that we have in order like to actually like properly provide like this content reliably to the network because, like overloading our infrastructure.

B

Like, in fact, we don't have open data is larger issues, I'm sure itself. As you experience the flood issue yet yeah.

A

uh So, like Gateway have kind of like two interfaces which, like they can get reached on, there's like the uh like bit swap interface, which, like we just blocked so like it's, not an issue here um on the browser, is definitely your challenge um and especially been one of like our main challenge. Initially, because, like the like, Gateway was like out as a research product on like cloudflare ipfs.com, uh we've received a fair share of like uh interesting requests um and so um like devising strategies to like prevent that.

A

It's been um kind of like a like evolving um saying, like one of the first things we tried to do was like to do like rate limiting to like be a bit more like fair in terms of like uh how the content is distributed, but you also see to like. Not all requests are created equal, um for instance, if you have like um you, want to do a streaming service on ipfs and like using a gateway.

A

Well um may not something you want. We wanted out of like a public Gateway um to serve, um and so like. It's really like up to like the public Gateway created like to decide how that goes.

A

um That's also why, uh for like, like team like now that, like ipfs, is more of a product at cloudflare uh like we're better able to um provide the tools to customers in order for them to Define like what their team is, a good content or not, but they are the one like to actually pay for that, so um that yeah. That really depends.

B

You mentioned uh fighting your IP addresses.

B

With a transport, whatever websockets yeah.

A

um All communicating over like TCP, uh which is another challenge uh but like uh like they've all like the communications with TCP and like the forward proxy, is like just like a TCP forward proxy. um We like relaying traffic uh to to to to our Edge and then like uh propagating traffic uh from this Edge.

B

B

A

Yep, so we like basically uh we're leveraging like the class like TCP, um offering um I think it got renamed. Historically, it's been named spectrum, um and so we leveraging Spectrum. You know the full like mitigating this, like TCP activity.

A

This live I think that was the white existence. Yes, it's on purpose.

B

All right- and these are about 500s right.

C

The green cards no.

B

A

No, the green bar is two hundreds.

B

A

Okay uh yeah, so 200 is like really on the left. Yeah, but like it's like personal, like cutting the y-axis like just okay,.

C

A

B

So you didn't do the last.

C

uh One before last slide and so yeah on the big spot, I'm going easy that part that uh the transfer that you are actually interested in so.

A

C

Have you the problems? I mean they're, having problems I, think reporting. So when files have to be.

C

A

I think it's not specifically like um so it's more like the default metrics exposed by logical ipfs for bit swap are rather limited um and do not allow for like a an easy uh like an easy debugging like you can see on, like the number of like um active requests, you can see like how many you've sent Etc, but I mean, as you mentioned like you, don't have like these buckets about, like the size of the file uh or like just just have like extend like the variety of metrics that are available.

B

C

And okay, good questions for financial of these points. So the efficiency of content layout and we know.

A

Yes, so the efficiency of content layout is not just from things like specific to the Gateway, uh but, like more generally, you have like all these data on ipfs, which are like divided like in blocks, which, like can be overlapped, um and so the question is, is like likely a lot of data, but like? Is it actually like efficiently like partitions, so that you can like you actually share blocks between like multiple cids um and that I? Don't know? It's not that important I mean it's a bit important for the gateways.

A

You can minimize the cache, but it's like even more important for like from a provider perspective where you actually like want when you're constructing the dag. On top of your data, you would like to inform I mean one of the strategy could be to inform the way you build the dag, based on like the overall content. You have so you actually like benefiting from like this. um This content, layer, structure, yeah and um last.

C

One or these areas are you, you plan to actively at a good time and work on these uh I'm going to judge. Well, you know, I should spend some time after this today to see how well you can collaborate, perhaps on things of something interesting.

A

So most likely not on all of these, uh but like for some of these yes I mean they are like so much like where actively like working on um I would say, like the name resolution, uh something like where um actually working on the uh measuring, like long-term peer rating uh is something like we also like considering um distributed tracing and like integration with Enterprise monitoring is more of a like um ongoing effort, because that's what we need for our operation, but we don't know yet how much it benefits all this good. Thank you.