Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2020 - Virtual, 4 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Managing Cloud Native Artifacts for Large Scale Kubernetes Cluster - Henry Zhang & Mingming Pei

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Managing Cloud Native Artifacts for Large Scale Kubernetes Cluster - Henry Zhang, VMware & Mingming Pei, Netease

When managing artifacts like container images and Helm charts for cloud native apps, users often face challenges such as efficiently publishing applications to Kubernetes cluster at scale, enforcing access control, identifying image vulnerabilities and backing up cloud native artifacts. In this talk, we introduce real-world use cases for managing images and Helm charts in CI/CD pipeline and production environment. The cases are summarized from production users of Kubernetes, including these aspects: 1. Setting up a highly available and scale-out registry service for a Kubernetes cluster 2. Synchronizing images and Helm charts of cloud native applications across multiple cloud environments or data centers 3. Preventing images with vulnerabilities from getting into production 4. Using P2P approach to efficiently distribute images in a large cluster 5. Integrated management in CI/CD

https://sched.co/etWq

A

Hello: everyone welcome to our session managing cloud-native artifacts for large-scale kubernetes clusters.

A

I'm henry zhang technical director of cloud native lab. We are with china, r d, I'm the creator and maintainer of harvard my interest is in cloud computing. Ai machine learning and blockchain.

A

Today with me, is min wing pay architect from netease he's responsible for ching zhou cloud native that off platform he's also a harbor maintainer.

A

B

A

Agenda we first talk about the two aspects of managing cloud native applications.

A

Then we introduce how to use harbor to manage cloud-native artifacts next, we'll go through the case study of netease and how they manage artifacts for large-scale kubernetes cluster using harbour.

A

We all know that cloud-native technologies become increasingly important to build modern applications. Usually there are two aspects in managing cloud-native applications.

A

The first is the dynamic part that is the run time, how the applications run and how they are scaled up, monitored, backed up and so on.

A

The second step is the static part that is artifacts when applications are not running, they usually reside on storage device as files or some kind of artifacts.

A

The most common and important cognitive artifacts are container images, ham, trusts and so on, but usually there could be more like cnav and so on to be managed in a cloud-native environment.

A

Given the importance of artifacts, we need to efficiently and securely manage artifacts when operating a cloud-native platform and, however, a graduated cncf project is designed to perform the tasks of managing cloud-native artifacts. It supports oci artifacts like docker image, ham, charge, cnap, open policy agents, singularities and so on.

A

In addition, harvard provides a bunch of features on artifacts management such as rbac, low base access control, image, isolation by project image retention and immutable images. We will cover some of them shortly.

A

The first artifact management feature I'd like to talk about is replication. This feature was created in harvest early version 0.3.

A

It allows two hub instances to synchronize the images from one to the other, because the replication tasks are carried out automatically and reliably users love and apply this feature in many scenarios in the latest release of harvard it can support replications of artifacts across multiple cloud environments with various registry and services.

A

B

List some refugee services here.

A

Such as docker hub refugee service in public cloud, like aws, google, azure article and so on, it is very simple to move your artifacts between different environments. This will help a lot when managing your artifacts.

A

Suppose you are running a large cluster with many nodes. If all those cool image from the public registry services, it will take up a lot of networking bandwidth to download the same images again and again, it is obvious not optimal.

A

Moreover, not all nodes within organizations are allowed to connect to the external registry, so harvard provides a feature called proxy cache. This feature has been requested by many community users for quite some time. It is.

B

A

In harbor 2.1 recently, it is a special kind of hardware project which can hold catch the images and serve them locally. It saves external networking bandwidth and speed up the local distribution of images under the hood. It leverages replication capability of hardware to pull images from remote sources when they are not available locally, because cached images are stored under a hardware project. All the project related features in hardware such as quota scanning immutable tag, can all be applied to cached images.

A

B

Any registered.

A

Service in production, high availability and scalability must be considered. There are many ways to achieve ha of a registry service. I just introduced some principles here, which can be the guidelines for users to implement a production registry.

A

The hardware core.

B

A

Components are stainless, which means that they can be scaled out by running multiple instances of each component. The key here is to set up h a for persistent services like postgresql, radish and shared storage. There are many existing solutions for these ha services just through one of them that fits your environment.

A

If you have multiple data, centers or cloud environments, that's running your applications. You can establish an ha harvard instance in each data center and environment so that they can back up each other. The.

B

Replication policy.

A

Can be configured to synchronized artifacts between two environments? Also postgresql and radius can be cap in sync by using some kind of synchronization software or mechanism by adding load balancers in front of hardware services. It provides high availability with an active standby configuration across two environments.

A

This creates additional protection for retrofit service. If one data center or one environment goes down, the other can go live to continue the service.

A

One thing to know is that the propagation delay of artifact replication between two data, centers or two environments should be taken into consideration and when implementing such a solution, that is to say, artifacts stored in two data, centers could be different when artifacts are being replicated.

A

B

A

Are deleted from a registry, their.

B

A

Need to be released and retained. This process is called garbage collection system. Administrators should perform garbage collection periodically to ensure the system does not run out of storage space in the latest harbor 2.1 release. Garbage collection feature is improved and can be performed without any impact of image, pushing pooling and deletion.

A

This feature allows harbor to keep providing artifact service while doing backhand garbage cleanup. This is very crucial for a production system to be up and running continuously.

A

When publishing a new version of the application to a cluster, we need to send the artifacts to every node a large scale, kubernetes cluster. It is a challenging task to distribute artifacts to all nodes within a short time frame.

A

If there is only one registry instance servicing the entire cluster, the registry instantly become the becomes the bottleneck of the distribution so to address this distribution problem in a large cluster peer-to-peer distribution dispute, disputing approach seems a feasible solution. Hubble can leverage the capabilities of p2p engines like dragonfly and kraken to accelerate the artifacts distribution.

A

What harvard does is to preheat the p2p network or one of the cluster.

A

The idea is to distribute the artifacts to the peer-to-peer network before the requests of the artifacts arrive when the actual request comes in the content is ready for distribution within the p2p network and can be transferred right away.

A

In a case study of netease, we can see that the p2p approach improves performance significantly in a large cluster of many kubernetes nodes.

A

Harbor can work with multiple kubernetes clusters as well. By setting up proper pre-heating policies, harbor can send artifacts to each p2p cluster and make them ready for subsequent artifact.

A

Distribution when administrators manage artifacts security is one thing they need to deal with harvard can help scan the content against publicly known cbe databases based on the scanning result. Hubble reports vulnerabilities found in the artifacts so that administrators can take proper action, such as patching the image to remediate the vulnerability.

A

This is not only crucial in a production environment. It can be used in ci pipeline during development phases. To ensure all images created, do not contain severe vulnerability.

A

In addition to vulnerability, scanning harbor can block pull requests if the image vulnerability exists. A certain threshold level also other features like content. Trust can ensure a prominence of the artifacts and vulnerability scanning can be triggered automatically when an artifact has been pushed to the registry.

A

If you want to allow some cves to exist in the images, for example, if they are not very critical or if you know it is critical, but you want to make them uh available for a while in your enterprise or in your organizations, you can set the exception in the allowed list.

A

From time to time, user may then have their own types of artifacts. If these artifacts follow the oci specs, they can be managed and visualized by harbor the late. This version of harbour extend the function of the default artifacts processor users can define their own artifacts format and media types by following the oci specs, then the artifacts can be pushed to or put from harbor a benefit of.

A

Storing artifacts in harbor is that artifacts can be treated the same as container images and can be replicated to other places or enforced by road based access control, and you also get our other features free from using harbor.

A

We have already seen partners utilizing harper for storing machine learning models as artifacts and reduce the operational capacities.

A

There are many more to talk about harvest capability of artifact management, but I'd like to pass it to ming ming for sharing his experience in artifact management in netease.

B

Thank you hurry, I'm mimi from the taste today I will introduce how we manage our adapters in the case now, the container technology is widely used in that is, we use container image, chart plate, bundle etc, as advectors githubs is also used in production, and we use hub as the repository of cloud native artifactors and now there are lots of kubernetes clusters in natives and 5 000 plus nodes in larger ones.

B

We have more than 20 half instances and the largest hub instance manages about 100 000 images.

B

Let's take a look at our architecture, we developed two services to manage the hub instance is and in the communities clusters, and we have the flexibility to combine the relationships of these instances. At the same time, accessing them by the authority system of netiz cloud native platform has been established.

B

In addition, we integrated our native object storage as backhand for high availability and high performance.

B

How do we make hubble highly available in the case? And first we address harvest high availability of fire storage in qs, object, storage and local fire storage. We use async. You know a fire synchronized tool to synchronize the data when using lock fire system uh for externally uh dependent uh highly high available availability.

B

We have used the open source project storage to adjust the high availability of first grade circle and the high proxy approach to adjust the high availability of radius, which you can find a detailed description in hubble community.

B

In addition, harvest monitoring, mainly armed at the thing, is enhanced and the monitoring clamps of the things such as replication failure and pdp dispatching failure achieved through matrix of messiest.

B

This is how we approach the multi-environment management of ad factors. Our first do packaging and complete all kinds of tests in the testing environment, and then we tag the image with release to cheap remote replication.

B

The image will be replicated to the production environment and then intrigue the online deployment.

B

B

For larger scale, distribution of the images we are facing two problems: uh one is the throughput pressure in register service and next is a network plan, wise pressure in backhand storage, and we realized p2p distribution by integrating integrate, integrating hub and kraken through the mode of sharing registry server.

B

The complete design can be found in the community of krakow. We finally achieved the growth of over 5000 concurrent pros and 10 gigabytes, plus image distribution, acceleration.

B

And here I would like to introduce some features of p2p distribution of clocken. This is the test that we met. You can see from the table.

B

The bandwidth limit of p2p distribution is configurable and from table you can see more layers and smaller layer size cause lower utilized utilization of bandwidth. It means the distribution will be slower and the number of prp peers brings little impact to their distribution performance. Max 15 kilos appears supported officially and it works well in our product environment. Now.

B

We also pay attention to the safety of artifactors. Artifacts are scanned after packing the emitted immediately. There are also two types of quality gas: the pipeline get and the dispatch get a lost asset, and there will be go through to determine whether the eye factors should be prevented from being used in production, environment.

B

And finally, let's look at how we manage the factors in crcd. First, we create our theact process with application center. You can see the full flow of our crcd and also I have introduced some related processes earlier and besides that, we.

B

Following functions are also contained in csd stages: artifactual version management are vector, security and cd, triggered by image. Pushing webhook crcd stages are connected in serious slow code and defectors.

B

You can see from this picture, and this is uh the full crcd flow in our company and that's of my share. Thank you, dr conrad.

A

Thanks mimi, I want to summarize a little bit here. So artifact management is an important aspect of operations in a cloud-native environment registry is an ideal place for performing the management tasks of artifacts.

A

Harper can be your choice of powerful tools for managing your artifacts hubble can provide high-variability and scalability for registry service and have replication proxy caching, non-blocking, gc and so on a whole bunch of powerful features that you can consider and leverage, um as you can see in the case study of landis they use harvard for the large-scale kubernetes cluster in the for the artifacts management.

A

Lastly, I wanted to introduce our new book um of harbor. It is the first book on harbor in the world also by the harbors, maintainer and contributors.

A

If you are interested in the content, please take a look thanks for listening to our session.