IPFS IPFS Camp 2022 - Measurement & Performance, 3 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Monitoring Cloudflare's IPFS Gateways - Thibault Meunier

Description

This talk was given at IPFS Camp 2022 in Lisbon, Portugal.

A

Real goal is like to share how we have set up our monitoring, how we use it um and definitely happy to see, especially because, like there's a lot of analysis that has been done and like a lot of data that could be available um so see how like these logs, could help you um down the line and just to inform how this is going so yeah I'm a research engineer at cloudflare I work on the various distributed web project, ipfs ethereum, also about split in, like all the standards like DNS um and I, like croissants, so I know we're in Portugal and like it's mostly pastelinata.

A

They also have very weird brioche croissant, which I'm not trying to qualify as croissant, but yeah. That's the case. So we'll see if there are some available later today um today, the presentation so I'm going to present quickly an overview of like cloudflare system what it means, then what is clever monitoring the various systems we're using and then uh where to go next things we have in the pipeline for like Improvement um for this, so yeah, first class architecture.

A

So most of the you in the room, I think know what uh cloud like a Gateway is it's an HTTP interface to the ipfs network.

A

um The main URL for cloudflare Gateway is cloudflapfs.com, even though we also do provide the capability for people to have like X5 and on certificate on their own domain, and so they can serve whatever CID they want. As mentioned before, the Gateway is closely tied with like um standard HTTP cache which sits close to like end users, and so that allows us to like Leverage standard web infrastructure to serve ipfs content to retrieve ipfs content.

A

We need at some point to have some ivfs nodes, mostly because, like IPS nodes are accessible at the moment, mostly through TCP, even though there are like some new transport which are being developed, and these nodes are also caching Longleaf content, so that if you have the popular content that is accessed from multiple locations, when cloudflare nodes need to access it, we just make less requests to the ipfs network, so yeah.

A

Overall, that's the architecture that we have in terms of data Generation, Um it's not limited to that, but we these are the main systems that we're using. So we have a monitor that is just checking the health of our Gateway uh via various active probes, and then we have more um like data that, like is sent by the Gateway. So we have like Prometheus data.

A

uh We have Sentry, which like allows to like debug the code that like is being run and then finally, we have like clickhouse, which is a column based database for like data with high cut anality.

A

um What I didn't mention is like we're not using row Cubo nodes uh we're using like Kubo as a library which I think is like kind of a plan of having lib ipfs at some point, uh but like it's kind of like half in-house half open source implementation for this foreign. What is kind of tricky that we've seen operating like an HTTP Gateway is for ipfs the kind of like base layers. You have a CID and you look at Which.

A

Cid is being requested how it is accessed, like you have bit, swap you have like the content routing for HTTP. You add like one a on top and like the main layer we see, we see is like the naming so like you can have like a row, um a raw domain, so like ipf as a cloudflare.com, which would act as a Content.

A

But then you might want to have like I, don't know a cache on top and then you have like another customer on top and then you have like another C names and so being able to have a view through, like all these different steps, um it's kind of tricky, and so that's why yeah trying to figure out what to make sense of this mess uh leads to a lot of like debugging and headaches so to dive a bit more in these systems.

A

What actually is Cloudland monitoring and so like these very systems that I've mentioned before, and some of you also mentioned in like previous presentations. So first of all, we have Prometheus, which is very useful, um like it's a Time database working on like low cardinality uh when there's like high continuity.

A

Just explode, uh so we have like limited number of metrics uh per cardinality, but it's very good for like live monitoring so like, for instance, if you want, like the number of like respond that you have like on your back end or like, uh if you want to know like um I think Leo mentioned about like. Is it like from a Gateway or not Gateway like a very small flippers, is very good for that. It's also connected to our internal auditing systems, uh based on like the threshold rates.

A

So you have like just simple mathematical functions and operation that you can apply on top of the data.

A

Another system that we're using is Sentry and so century is like different than Prometheus. In a way, it's not that much of a database, but it's more like an event-based um and like event based and like focus on like the errors that our system could be producing. So the good thing is like it provides context about the areas that we see. So, for instance, like you, have the host name.

A

You have the URL that has been accessed and like custom tagging that you can have on the error, and so you would see exactly which part of like the code as well like the stack that was used to generate this event, which is very useful when you try to reproduce saying that the customers have been experiencing.

A

And finally, we have clickhouse. So clickhouse is just cloudflare way to have a lot of data. I was like arbitrary data with like High cardinality, um so it's a column based database cloudflare, also samples uh the the logs and like data that, like we go to clickhouse, so that we just don't store everything that happens. We can store some Aggregates and it's far more useful for long term.

A

So, like I, don't know you have like a lot of like cids and routine ID that could be accessed and you can make normal SQL queries over that data.

A

Finally, something we've open sourced earlier this year is the ipfs monitor. So it's like a black box monitoring tool for the ipfs Gateway contestant, like multiple scenarios or like update of the DNS link caches new content created on like their spinning service, and it also provide Prometheus exports so that we can like have alerts when something is going bad.

A

Did that result that we had earlier this year, so I think it's like March, where you can see that the for newly created content, sometimes it's very fast, like 200, milliseconds and I, would not say it's very fast, like it's, okay, but most of the time it will be very slow, and so definitely that's something that I will mention later on in some of like the Improvement that we want, especially the main pain at the moment is like through the content reading when the content is in Cache, it's just fast.

A

Like I, don't know the maximum is like half a second which is good um and when the CID isn't available, that's really, depending on like the node and the strategy, so we've tuned it a bit since since then, but mostly it just takes time to figure. Cid is not there, which is not that good, because you have like this loading screen just waiting for nothing from that. Where do we go next?

A

um So this is a non-exhaustive list of like improvements that we would like to have, uh but mainly um having some like kind of like peer rating and like peer reputation would be interesting, especially to know which peer the Gateway should appear with, depending on like the content.

A

um Also the like on content, reading and content, providing having like Gateway plus indexes Plus, some more like interactive uh participation with like data providers like important because, like I, don't know like a minute to like retrieve a Content, it's just too much uh on on certain providers, then having like some standard um around like Gateway names and resolution, is I, feel so important um and that's mainly I think going through like the ipns land, uh we're not seeing that much usage for ipns, uh but definitely that's compared to its usage.

A

um There's a lot of requests coming in in terms of like how to tune and how to define so there's been a new ipip coming into Define ipns, um but yeah. Definitely more involvement through here. It could be interesting.

A

One data that, like we are interested in more generally, is like content. Delivery network is like how efficient is a Content layout that is being delivered to ipfs.

A

Are people actually building dags that are efficient for caching, or is it just like using the like normal Kubo like DAC building, and that's it um I know like some projects like have been using efficient content layout should be like for Docker npm or like some videos that we've seen, but overall, it's kind of like an explored area that could be interesting to reduce the amount of storage um similar.

A

One thing that is coming and like definitely more for like one additional sources like distributed tracing so on a similar note to Sentry like this sort of tracing, would allow to just better understand through, like the whole Pipeline and various systems uh that we have were actually is something taking time. So, if we see the time uh for discovering content is just too long that maybe like help us prioritize um the things that like we want to, have and want to see and finally, integration with Enterprise tool.

A

It's more like a Gateway saying, uh like generally um I, feel there's been especially the way like the those packages uh and like ipfs nodes have been built. um It's sometimes tricky to like integrate one new tool without just rebuilding this thing. um So yeah, that's more General comment.

A

Otherwise um thank you. There's definitely been like a lot of like I, think interesting, like measurements and things, and now is the time for like ask the gay Gateway anything that you have seen or have not seen in the slide. Definitely happy to answer foreign.