IPFS Virtual Meetups, 15 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Bifrost Working Group 2020 Recap

Description

George Magras of the Bifrost Working Group joins us to give an update on all the things the infra team accomplished in 2020.

For more information on IPFS
- visit the project website: https://ipfs.io
- or follow IPFS on Twitter: https://twitter.com/IPFS

Sign up to get IPFS news, including releases, ecosystem updates, and community announcements in your inbox, each Tuesday: http://eepurl.com/gL2Pi5

A

My name is george. I work on the ipfs team, I'm the lead on the ipfs infrastructure team, also, officially known as bifrost.

A

The core team is made up of the three of us myself, michael burns and tommy hall, but uh we've seen many contributions from more people who I listed below the biggest probably the biggest accomplishment for our team. This year has been keeping up with the ever increasing um requests uh on uniques and and and data flowing through the gateways, we're sort of a victim of our own success, because we quickly realized that uh just spinning gateways on bigger boxes results in more people using the gateways.

A

So we've had to pause a little bit and see what other knobs we can. We can tweak so that we don't. We don't have to just keep spinning up bigger boxes.

A

uh Just to give you an idea of uh our gateways, we run in nine regions via bgp anycast, which means we serve the same ip address from from multiple regions uh we do deploy via terraform and ansible. uh We use openresty, which is enginex plus lua for scripting as a reverse proxy in front of go ipfs.

A

All the gateways have an 800 gigabyte, hard drive for caching, and our observed slis are around average 150, milliseconds time to first byte and 99.97 uptime. For for this year, um we've accomplished these numbers uh by uh applying lots of tweaks to the go ipfs uh garbage collection uh thresholds, uh os level, uh tweaks rotting, tweaks, um for example. We we noticed recently that having nodes in frankfurt region uh basically made most of the traffic get uh get sucked into into that region.

A

So uh it wasn't great for user experience because we would get users from like asia hitting frankfurt just because they were closest to that region in terms of of network hops. So we we we've since distributed the load uh better, but yeah uh we've seen it uh takes increase in unique visitors and in gateway traffic.

A

um Another thing that we maintain is a list of bootstrappers we've scaled from seven nodes to uh 12 nodes.

A

We've also switched from using static ips and the go ipf's configs to using dns adder suffix matching and the way that works is when you resolve dnsr.bootstrap.looptp.io, you get a list of peers which are which are also dns adders. So if you recursively resolve those, then you get multiple uh ports, multiple entries for the same peer, but describing uh other protocols. So you can, you can easily uh have it like a like a a nested way of describing multiple peers from the same dns adder top dns adder.

A

uh This is this allowed us to easily add support for websockets and and quick. um Also, we've spun up an ipfs scallop cluster. um This is a three-note cluster, 80 terabytes, each so lots of room to grow there. It's currently hosting a bunch of pl websites, as well as falcon parameters which are similar, multi multi terabytes.

A

As far as monitoring we didn't have a lot going in. uh We just used like plain old ssh to log in and tell logs so we've since uh incited a monitoring pipeline. We use prometheus for our metrics uh for machine level. Metrics like overall memory usage network, cpu usage, we use node exporter, then go ipfs itself provides metrics in the prometheus format.

A

So we get all sorts of good info, such as number of peers bit swap stats, uh quick stats, all sorts of very good good things, and for the http proxy, which is operating this case, we use mtail to parcel logs and expose the percentage of requests of successful requests, fail, requests and and so on, which we used to alert on and also to graph in uh in grafana.

A

We also oh yeah. We also use pingdom and rain tag for global op type checks that that is how we got that 99.97 uptime.

A

Some things that we would like to to work on uh in the next year are obviously keeping up with increasing user demand. uh That mostly means scaling ipfs uh uh instances and working closely with the go ipfs team in identifying uh memory, leaks or or tweaks that we can, we can make.

A

We would like to make our repos public we've uh uh basically extracted most of our secrets from from the repo. So at some point we would like to make that open source and start accepting contributions.

A

Also, a big one has been streamlining, dmca request, because we've we've been getting lots of those, so we actually have work to enable a double hash denied list that can be easily shared with other gateway operators, which would be great to do at some point. Also. We have to find better ways of testing configs before deploying to all of the gateways. Right now we have staging boxes. We do canary deploys to them, but um sometimes so some things are still not uh not obvious when they break.

A

So we would like to have a sort of uh a better approach to testing these before they hit live, and also we like to experiment with running ipfs on kubernetes at scale. The major of the kubernetes networking layer makes it challenging, but if we solve that, it really makes up in terms of orchestration across multiple.

A