Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Startup Containers in Lightning Speed with Lazy Image Distribution - Kohei Tokunaga, NTT

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Startup Containers in Lightning Speed with Lazy Image Distribution - Kohei Tokunaga, NTT

Pulling image is one of the time-consuming steps in the container startup process. The most critical factor is the current OCI Image Spec with which a container cannot be started until its all image layers are downloaded. However, most of the contents in image layers are not being used for real-world workloads. In this talk, Kohei will show state-of-the-art alternative image formats which lead to faster container startup by allowing container runtimes to start a container without waiting for all its contents to be locally available. He will also introduce CNCF containerd's fast image distribution approach "Remote Snapshotter" which leverages these formats (https://github.com/containerd/containerd/issues/3731). Finally, he will share the status of his current work on the remote snapshotter implementation and how to take advantage of the new functionality.

https://sched.co/ZepQ

A

Hi I'm kohei tukunaga from ntt, I'm mainly working in the consumer runtime and a maintenance of sergio snapshot project which is known for the project of contrary.

A

Today, I'm going to talk about speeding up continuous startup by lazy bullying images.

A

First of all, let's get to the summary of this talk is one of the time consuming steps in container lifecycle, sergeant, snapshot, nokwasa project of contendi is trying to solve it by lazy playing images. Leveraging storage image by google for further runtime optimization is also held with an extended version of storages, which is called easter egg.

A

The diagram shows the container startup performance for three types of personal 3.7 images.

A

It pulls the image from docker half to easy to oregon and runs print hello world sergio formulated python image with storage's snapshot, reduces the time to take for poor operation by legible, but run takes longer than the legacy image, and each formatted image mitigates the runtime performance drawback with the optimization, and we will discuss in more detail with the following slides.

A

There are also other oca alternative image distribution strategies in container ecosystem, so I'll briefly introduce some of them. In this talk.

A

First of all, don't you think that pool is time consuming?

A

As you know, paul is one of the most primitive operation, for contrast and it's very general workflow that we pull continents from registry, then unpack and run them on nodes, but unfortunately, um a research shows that pulling packages accounts for 76 percent of container start time, but only 6.4 percent of that data is read.

A

Of course, some work runs alone, but they are not enough. Actually, we can cache images on those, but the first time of the pool called chord start is still slow and we can also try to minimize image size, but not all images are easily minimizable.

A

Let's say we cannot avoid using some large base images, including language, runtimes frameworks, etc.

A

Before taking deeper, look at the pool performance problem, let's briefly recap: the oci docker specs for image distribution in the specs. A container is defined as a set of layers in image spec. It defines layers and metadata and in metadata, yeah image, manifest, etc are contained and layer is defined as star and optionally, plus compression and bluetooth can be composed by merging layers in distribution. Spec, it defines http api of registry and layer can be fetched as a blob named with a content, addressable digest and registries.

A

Optionally support, http range request, which enables us to fetch the person ranges in the blob on the registry.

A

And when it comes to pull performance, there is a problem on the current oci docker specs a continent is a set of terrible layers, so a container can be started until the all layers become locally available, even if the most of the contents won't be used on container startup, as shown in the diagram, a layer blob is defined as a table, optionally, plus compression.

A

This means we need to scan the entire blob even for extracting single file entry. If the blob is gzip compressed, it's not seekable anymore, and we cannot extract it in parallel.

A

So we need to scan the blob from the top sequentially because of them. We need to wait long time for the poll and extraction completion on container.

A

Startup storage is snapshot, is a non-quanta project of container d and continuity is a consumer runtime project in cncf, which is also used by docker storage. Exit snapshot is a plugin for content id which provides lazy, pool functionality for quantity and lazy. Pro here means continuity, doesn't download the entire image on pro operation, but fetches necessary chunks of contents on demand.

A

It means we can start continuous before the entire image contents being locally available. So we can speed up the container startup performance, lazy, pool installs. The snapshot level is started image by google, which is standard compliant. So we can laser pull images from standard container registries, including docker, hub.

A

Let's do the demo first ipo python, 3.7 image from docker hub and run print hello world in the container, with continuity in a very normal way. The image size is around 338 megabytes on the registry so protects accordingly.

A

We need to wait for pro long time and if the image takes one gigabyte or two to a larger, we need to wait for longer and longer. Sometimes it's called chord start problem. In this case, the p will take about 32 seconds.

A

Next, I lazily pull started formatted python image from docker hub. In this case the pool took about seven point two seconds: it's a strategy, formatted image, but not squashed, no content is lost.

A

This speeding up is the power of legible I'm playing this image in tokyo and if you are in different locations, you should get different results, for example, in the benchmarking results in oregon, which I will talk about later, the time to take for lazy pool. It's shorter.

A

So how lazy blue is provided? Storage snapshot, leverages storage used image which is compatible to the current oci docker image specs, which means the started image can be lazily pulled from standard registries, and it can also be run by legacy runtimes, but in this case not lazily pulled in the following diagram.

A

A registry is shown on the left side and the node is shown on the right side on the node. Sergi's snapshot is plugged into continuity for continuous spanned by continuity. Strategic snapshot amounts the continuous root fs snapshots as hues on mount most of the layer content are not downloaded yet instead start this snapshot.

A

Lazily downloads file contents accessed by continents on demand from the standard registry. Again, strategic images are compliant to oci docker, so legacy runtimes also can pull and run it from registries again, but in this case not lazy and sorry. This snapshot itself is implemented as a remote snapshot. Plugin and remote snapshotter is a type of plugin of continuity.

A

Id allows remote snapshots to remotely mount continuous rotofs snapshots from remote store to continence on the node, so you can also implement your own remote snapshot. If you want to use remote stores other than registry.

A

So what is sturdy is it star. Gizet is an archive or complexion format. It's proposed by google clfs project. Sergey stands for seekable stickable targeted, so it's seekable, but still valid charges it.

A

So it's usable as a valid oci, docker image, layer and entries can be extracted separately, so each entry can be fed separately from registries using http range request, standard registries optionally, but I think most of registries support http range request, so we can download blob contents profile entry on demand, which enables us lazy image building, as shown in the following diagram: the normal turgy that layer is not seekable and we need to scan the entire blob even for getting single entry.

A

On the other hand, surge searches, it is actually the concatenation of gzip compressed file entries. This is server gzip, so we can decompress it in file or chunk granularity, and it contains an index called toc at the tail of the strategic blob. So we can get the offset information of each entry contained in the sturgessed. Blob toc also contains metadata of all files stored in the star digit archive. So we can mount this layer without seeing each header of each file.

A

But network related overheads can't be ignored for only one fetching with charges, especially on slow networking environment, because the read access to a file on star gazette induces downloading that file entry from the registry estar did it enables the profit files that are likely accessed during runtime, and we call these files as prioritized files and file system, can prefetch and pref pre-cache. These files, with a single http request, range request or mount.

A

So we can access these prioritized files locally, as shown in the diagram, the strategy that allows to group file entries based on the priority.

A

Let's say we know, the files marked with red boundary is likely accessed so prior orders files during runtime and it started indicates this information about likely accessed files as the order of file entries when installing the snapshot amount. This layer, the files marked with red boundary, are pre fetched by a single http range request and precast.

A

Other files. Files marked with green boundary are also downloaded aggressively in the.

A

Background and level linking is registered, cli converter by the project provides workload based, optimization generally, containers are built with purpose and the workloads are defined in the docker file, with the entry point user info, etc and stored in the image.

A

The cll converter runs provided image in a sandbox environment and profiles, all file accesses and it regards these access files are also likely accessed during runtime. So this first prioritized files in the store gives it and store. This snapshot will prefetch and precache these files. Where mounts this is image, as shown in the diagram.

A

The original image is shown on the left side, and this contains workload of information, so entry point, commands user, etc, and the converter command runs this image in sandbox and profiles, file accesses and generate estradiol.

A

Again, we installed the snapshot among this image. These accessed files are application breakfast and precast, so we can access these files locally.

A

And some benchmarking results with sergey's snapshot. It measures the container startup time, which includes pulling down image from docker hub and for language containers, running clean teleworld program in the container for server containers.

A

It wait for the readiness, so it was until upon running. Email message is printed from the container.

A

This method is based on hello bench and we took 95 percentile of 100 operations and we used ec2 oregon instance of the host and docker have at the registry.

A

And in python 3.7, the time to take for pull started is shorter than the lc images, because continuity with storage snapshot only pulls image, metadata and searches it index files, but the round performance is slower than legacy when we use etherset. On the other hand, runtime performance drawback is mitigated compared to stargazit, but the time to take for creating container is longer than searches, because during the container creation, storage snapshot waits for the preset compression.

A

In gcc 2.9.2.0, laser pull also shortened the time to take for propagation in both of starjet and estrogen. Cases, for this image run takes longer even with its starjet, and it is because startup performance compiles hello world c program, but the source code isn't included in the image but provided at runtime by mounting the source code directory.

A

Currently, the project's optimization command doesn't support mounting external directory, so workload optimization couldn't profile the file accesses during this workload, but this usually should be improved by adding volume functionality to the optimization command and enabling it to profile file accesses even with such runtime volumes.

A

And for glassfish 4.1, which is a server image, unless you pull also shortened the time to take for poor operation in both strategies and eastern used cases, but sergio takes longer for the server initialization and each strategy that image mitigates this for exaggerated image again. Waiting for the prefetch completion, drawing create occurs again.

A

Pull operation is a fundamental operation in container lifecycle, so the performance improvement will be beneficial for many kinds of workloads, but I did some of them here. The first one is speeding up base image distribution on image, build especially for base image in dev stages in multi-stage build because these images aren't included in the result image.

A

The second one is speeding up. Dev pipeline or building testing environment strategies was invented through the discussion in go community for speeding up the builder image distribution in their build system.

A

The third one is sharing large scientific software stack, for example, ml frameworks.

A

For example, moves frameworks tends to be large, they can be over than one gigabyte and the final one in the list is improving cold start performance, for example in service environment.

A

Of course, we need more investigation here for finding out what is the right bottleneck, of course, that problem and but please be careful- that strategy snapshot project is still in our stage.

A

It needs more performance, improvement for the implementation and visible performance seems to be affected by the internet, condition and instance, location, etc, and also be careful for the force trainers until the layer. Contents are fully cached on those.

A

And, of course, feedbacks and suggestions for use cases and implementations etc are always welcome.

A

There are also other oci alternative lazy distribution in the community, at least some of them. First one is slack. It uses nfs infra for the distribution of rotor fs snapshots of continuous and registries are used for sharing snapshot ids among hosts and some vmfs is a fused file system. Python for sharing high energy physics, software and worldwide infrastructure, and software stack can be mounted and lazily downloaded from server vmfs repository via http.

A

There is an implementation of contradicts remote snapshotter based on service again remote snapshot is a plugin for continuity, like strategies snapshot, but one can use any file system and store store as bargains, and there is an ongoing discussion towards integration of some vmfs and podman.

A

And fire grain is proposed by akihiro soda entity. It is an oci compliant image format, but uses continuity manifest as layers an image can be mounted and files are pulled lazily and each file is treated as a content, addressable blob, so the duplication in file granularity is also in the scope.

A

And there is an ongoing discussion towards ociv2. It's proposed by alex sarai susie ocav2 is not official spec yet, but brainstorm is in progress in the discussion.

A

The defeat support amount of multiple file system are also included and, finally, crfs plugin for fuse overlay effects. It is proposed by gdp scrivener red hat. It is a plugin of use, overlay effect for mounting started layers.

A

Finally, recap: a pull is one of the time consuming steps in container lifecycle started a snapshot.

A

Newcastle project in continental is trying to solve it by lazy playing images, leveraging strategic image by google it's standard compliant, so it can be pushed to and it can be lately pulled from standard registries and workload based runtime optimization is also held with eastern diseases.

A

There are also other russia alternative image, distribution strategies in continental ecosystem and, of course, feedbacks and suggestions are always welcome. You can access to our apple on github.com, slash, contender, slash, sergi's snapshot.

A