IPFS IPFS Camp 2019 - Lightning Talks & Demos, 22 Jul 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Containerd meets IPFS - Edgar Lee

Description

Watch this lightning talk from Edgar Lee at Netflix on the opportunities presented to IPFS with ContainerD.

Learn more about IPFS Camp: https://camp.ipfs.io/

For more information on IPFS
- visit the project website: https://ipfs.io
- or follow IPFS on Twitter: https://twitter.com/IPFS

Sign up to get IPFS news, including releases, ecosystem updates, and community announcements in your inbox, each Tuesday: http://eepurl.com/gL2Pi5

A

My name is Edgar and I'm, an engineer at Netflix working on tools and infrastructure, so container adoption has increased exponentially over the years and so has the need to mass distribute and Netflix really launch as many as 3 million containers per week. So the need to innovate container distribution, it's important. So what is a container image here? I have a 1/2 image and it consists of 4 layers and a layer. It's just an archive of files and directories. I can unpack them from dr.

A

bottom to reveal its contents and layers are named after the hash of its content. Just like ipfs one key insight here is that we can actually be couple what's the definition of a bun from how we download it once I know what layer constituents image it doesn't actually matter where I get it because I can independently verify the hash. So we can leverage ipfs as a CDN for container layers.

A

The first, let's visit, revisit how ipfs adds files, so a large file in ipfs is broken down the blocks hash to digest and then store it as a self described identifier, known, C, ID and then hash into a tree of nodes, called a merkel tag and the roots. The ID is the identifier for the file coming back to containers for every layer in a bun. We can actually break it down to blocks and then using the roots. The ID translate it to a digest format that container D understands.

A

By doing this, we created a pure pure version of Ubuntu, and then we can use an IP FS plugin for container D in order to retrieve container layers through ipfs, to demonstrate how this works I'm going to shard a peer-to-peer image across the ec2 cluster and then try to download it so sharding we're gonna, take every layer break it down to chunks, some of which may be duplicate it and then group them into roughly 33% of total size and then pin them on ec2 nodes on ad bus.

A

So here I have a three easy two nodes on ec2 and then I am going to check the IPS repo and we sawed-off we're about 14 kilobytes of data. What we can do now is a shard, the peer-to-peer image across the cluster such that each node has about 33% of the data and then, when we check IPS again we'll see that they roughly have twenty ten megabytes of data.

A

So what we want to do now is measure how long it takes for every node and network to receive 100% of the data.

A

Let's see how that works,.

A

So that took about two seconds and that's close to docker hub speed using Akamai CDN, we track ipfs. Now they have about 28 mega, it's data chicken continuty. We can see that the layers are indeed present in all the nodes to make this more interesting. How about we scale up this experiment? So we're gonna resize the server group to 50 nodes- and this is gonna- take a while, so we're gonna skip ahead and that's done.

A

And now we have 50 nodes, which means that when we short the image, each node as 2% of the image or 7 file blocks each. This is an unrealistic scenario, but we're gonna see how that works.

A

So you see there's a lot of network chatter and that affects the distribution of the image.

A

So it's gonna take about 11 seconds.

A

Can I go next.

A

So container the container ecosystem already uses content, addressable storage, so it makes ipfs a natural fit. I believe ipfs has a great opportunity to improve distribution. We just need to focus on benchmarking and performance tuning. So what's next a paper from slacker sort of survey, 57 different containerized applications and found that only about 6% of the data is actually used. So what can actually do is pack the container file system by a fuse, and then we can actually create a manifest that serializes the metadata for files.

A

That way, we can boot up a container file system that it's gonna take. You know milliseconds, because the manifest is only 200, kilobytes, bate and then on demand actually download the files through the same ipfs plugin through the IP CS, which is the project I work on. Thank you.