Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022 [End User Sessions], 1 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: TikTok’s Story: How To Manage a Thousand Applications on Edge With Argo CD - Qingkun Li & Jesse Suen

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

TikTok’s Story: How To Manage a Thousand Applications on Edge With Argo CD - Qingkun Li, TikTok/Bytedance Inc. & Jesse Suen, Akuity, Inc.

This talk will share a case study of how TikTok manages its global edge clusters with Kubernetes and operates continuous delivery with Argo CD concluding with a demo. The talk will also dive into the scalability challenges faced by Tiktok to manage edge services using Argo CD (with ~100 edge clusters, ~150k CPUs and ~3000 applications), as well as how the Argo community plans to address them in future. TikTok operates a large network of Kubernetes edge clusters around the world, hosting apps such as Tiktok, live and gaming, using cache and traffic acceleration services offered at our edge clusters. The challenge arises when it comes to the deployment management of those edge services on hundreds of edge clusters. Normally, an edge service shares a lot of common configurations when deployed globally, but still has cluster-specific configurations (e.g. resource quota, service hostname, etc.). From this talk, people will learn how to deploy and manage such kind of services using Argo CD.

A

Hi everyone thank you for attending our talk. Tick, tock story: how to manage a thousand applications on edge with argo cd.

A

In today's talk, we'll do a case study on how tiktok is managing 3000 applications across 100 global edge clusters with argo cd and we'll discuss some considerations, tips and techniques for using argo cd to manage cluster applications on the edge.

A

So before we start some quick introductions, my name is jesse suen and I am a argo project lead and co-founder and cto of acuity.io, which provides application delivery solutions powered by argo speaking with me today is chinkun li and I'll. Let him introduce himself.

A

Thank you. Jessie.

B

My name is ching kun lee tech beat manager of edge platform team at tiktok. We use kubernetes and cloud native technologies to manage teapots on-prem at clusters and help developers deploy and manage applications on the edge.

B

This is the overview of tick-tock's edge cluster. We have around 100 edge clusters distributed around the world serving tick-tock as services such as video saving cash live streaming, gaming and so on. The size of our edge cluster varies from 10 to 60 nodes on each edge. Those are powerful server, node with currently 96 cpu cores and the 256 giga memory for each node.

B

We also have the data center that can talk to all of those edge clusters. The data center functions as both the management control plane, as well as the service data plane for those edge clusters and edge services.

B

For example, we have the argo cd that runs in the data center to deploy ad services to those edge clusters talking about edge cluster deployment. This is the high level overview and architecture of our deployment infrastructure, so each of our edge cluster is a step along kubernetes cluster.

B

They talk to the data center, where we have argos id and the git ripple to manage the deployment for the edge services using githubs, so our developers will put their kubernetes configuration for the ad services to the git repository and the central argo cd controller will pull those configuration from the git repository to sync and deploy them to the specified edge clusters.

B

For the deployment of all edge services, they actually all follow the same pattern. Usually at service is deployed to many edge clusters and the service functionality and behavior of this edge service on all edge clusters are very similar or the same, for example, for the video setting cache service when deployed to all the edge clusters, they all serve the same exciting functionality for the local users.

B

As a result, the kubernetes configuration of the ad service will also follow the same pattern, which is the deployment of service on all edge clusters, will share a large portion of common configurations and, at the same time, there are also a small portion of cluster-specific configurations.

B

For example, the replica account resource quota ip address, etc might be configured differently on different edge clusters for the same edge service. This is because different ad clusters may have different number of server nodes at different ip address range here is an example of the configuration of such deployment pattern. In this example, we use the sample index service deployed to three edge clusters, and we can see the majority part of the deployment and the service configuration of this engine service in all three clusters are the same, except for some cluster specific configurations.

B

In this example, they are the replica account image version and external ip configurations.

B

Although the actual configuration for the real edge service might be much more complicated than this, but the idea is the same for such kind of deployment and configuration pattern in kubernetes. We can use tools like helm, the customize to manage the common and the cluster-specific configuration parts.

B

In this example, we use customize to show how we manage such configurations and how we structure our deploy directory for those configuration files in this deploy directory. We have both the uh the base directory for the the common, the shared configuration parts and we have the overlays directory for the custom specific configurations, and in this example here we see in the base directory.

B

We have the deployment.yaml file and the service.yaml file that defines the common deployment and service configuration for this sample engine application.

B

Here we show an example of what's in the cluster specific overlay configuration directory here in the customization.yaml file, we points to the base directory for the common shared configuration part and also the apply the the patches for the cluster-specific configurations.

B

In this example, we apply the replica and image patch to the endings deployment configuration and apply the external ip patch to the service engine configuration.

B

We use argo cd uh with app of apps dial to manage those configurations and to deploy those configurations to the corresponding edge cluster.

B

So we generated the parent application for the for the ad service and the parent application will will then generate the child application for each cluster to be deployed. The child application inside the child application.

B

uh We will specify uh the corresponding overlay cluster directory as the uh the application path and we specify the destination uh for the corresponding ad cluster and the central argo cd controller will continue to process those child applications and, in this example, using custom to generate the kubernetes manifest by reading the configuration from the specified path from the git repository and then will sync and deploy generated kubernetes, manifest to the corresponding edge clusters.

B

Let's do a live demo to to show how we use argo cd with such apple apps style, to deploy an edge service. The configuration we we're using in this demo uh has already been uploaded to the github, and the link is attached here. Please feel free to check it out in this demo. We will deploy the sample the example engines application to the three edge clusters, as shown in this diagram.

B

Let's get started with the demo. This is the github repository for this demo and in this github recognition we have the deploy directory and then we have the base and overlays directory where the common configuration for the deployment and the service are stored in this base directory and the in the overlays directory. We have the cluster specific configurations for the demo purpose. I just add the cluster specific service external ip here and also in the customization.yaml file that first point to the base directory for the common shared configurations and apply this service patch.

B

In addition to the base and overlays directory, there is also the application.yaml file. So this is what the parent application will use to generate the child applications in this example we're going to generate the three child applications for the three clusters and for the child application for cluster one. We can say we specify the configuration path as the corresponding cluster one overlay directory the same for cluster two and the cluster three.

B

Okay: let's go to argos cd to create this application. Let's first create this engine application by creating this parent application. Here.

B

For the parent application, we call it a demo, a parent, and we use the cubecondemo project that I have already created, and we set the repository as this github repository that we just saw and we set the path as the deploy directory. So this is the path that points to this application for your profile. So this is how this parent application is application.yaml and use it to generate our text.

B

The destination here is the.

B

The destination here is actually the the kubernetes cluster and namespace, where we're gonna create this parent application. So that's in the data center.

B

Created the greatest parent application now in the parent application, maybe he already uh uses uh use the configuration we. We just did for this parent application to see the three child application in the git repository, and we do a sync to create this three child application.

B

They are created. We go back here now now we are at the stage those uh the parent application is created and those three child applications are also created but they're, currently in the out of sync state. So this is when we have created those argo city, applications, the parent app and the 3 child app, but we haven't synced to generate the kubernetes manifest and deploy them to the add cluster. Yet we can sync and deploy by click on these sync apps and we sync those three out of sync child applications.

B

This may take a minute for the three applications to be synced uh to be synced and deployed.

B

Now they are all synced. Let's go inside uh this demo cluster one uh application we can see it create the service and the deployment for this engine example app in the service uh the live manifest. We can see the cluster-specific ip that we have already configured that we have configured in the git repository and we can verify this is working by access this ip and we see it's working because we're able to to see this engine page and also on argo cd. We can go to the pod to to see the logs.

B

If we do a refresh here, we should be able to see the corresponding access log from my browser and if I do another refresh another two refresh, we can see two more accesses here. It's pretty useful.

B

Okay, this concludes our uh demo part. Now we have talked a lot about tip tasks at infrastructure, our deployment architecture and how we use our go cd to manage our applications and the deployment.

B

Now let me talk a little bit about the challenge that we have so far using argo cd in this way. So, first of all, is for the performance now because we are having uh so we get over 3000 applications uh managed in the central argo cd controller.

B

We do start to see uh the speed to list all of those applications becomes slower, especially for our developers who who have permission to see all of those 3000 applications. So, luckily, for us, like most of our developers, they only have permission to see their own applications, which are usually like 200 300. So this is not that bad for them, but we do see uh the uh the performance degrade when we having more and more applications in our go. Cd second, is for uh scalability.

B

We see some imbalanced application handling for different edge clusters so because, when we have more and more uh applications managed in the central cd controller, then we start to uh to roll out one more controller instance uh to steal the ability to handle those applications.

B

But currently argos id will short the applications to the uh to the controller instance by the destination cluster, which means all the applications for one destination edge cluster. They will all be assigned to the same controller instance as a result, because we have some of our edge clusters are big and some are small so for the big edge clusters when we deploy more applications to those big edge clusters and there will be more corresponding kubernetes resource to be handled for that big edge cluster.

B

However, all those applications and kubernetes resource can only be handled by a single argo, cd controller instance. So therefore, we can see some controller instance might be a idle, but some controller instance might be busy and we can not easily resolve this problem by simply adding more controller instance is for uh functionality.

B

So we like to have the support from the argo community for the project level support, because currently we use argo cd project for the multi-tenant support to support different teams and developers in our company.

B

So one issue with the current support for the project is that the application name in different projects could conflict. For example, if we have the engine application in project one and another application named engines in project two, so they can have conflict with each other.

B

In addition to that uh us to see a better project level view and management to be supported as well, for example, for the management of the uh the project level, applications, kubernetes resources, repository etc, but not least, for the developer of durability.

B

We find uh currently, we probably like the internal observability for tracing such as the the open tracing capability, because a very, very argo, cd user. So sometimes we need to go inside the rfocd source code to do some troubleshooting.

B

But now we don't have a very easy way to trace and track a single request in the source code, because we don't have the uh the open tracing enabled in argos city yet yeah. So those are the challenges that we have seen so far. uh In spite of that, we still think argo cd is pretty helpful, useful and reliable to uh for our edge scenario use case, and we we do recommend argos id.

B

If you have a similar use case and scenario next jesse from the argo city side will talk, we'll talk more about the suggestions of using article cd in such a scenario as well as uh how those challenges could be addressed.

A

Thank you chincon.

A

So now we'll discuss two different edge deployment models, which we call a centralized push model and a distributed pull model, and then we'll discuss some of the trade-offs between the two so the first model, the centralized push is one which you should already be familiar with, because it's the one that trinket had just described and with this approach you have a centralized instance of argo cd, which is connecting to and managing your edge clusters.

A

And this argo cd instance is rendering the manifest from good and pushing or keeps you applying those manifest to the edge clusters.

A

And the biggest advantage of this model is that you're able to use argo cd, as is intended as a single pane of glass and so from this control, plane, you're, able to view and manage your applications in any cluster and second, is easiest to manage. Since you only need to maintain that single argo cd instance and because this instance is running inside your data center, you can automate and create integrations against it.

A

On the other hand, a single large argo cd instance will require some performance tuning and sharding as your cluster fleet grows, and you may encounter some of the same scaling issues that tiktok experienced.

A

Secondly, because argo cd needs to connect to these clusters, the kubernetes api servers of these edge clusters do need to be made available to argo cd and finally, because this is a centralized model, argosy does become a single point of failure.

A

The next model is a distributed pull model and with this approach instead of running argo cd centrally, you instead run an instance of argo cd in every edge cluster, and these instances would be configured to pull from a git repository and auto sync changes from a directory specific to that cluster.

A

The biggest advantage of a distributed model is that scalability is no longer a concern, since each cluster has its own application controller. The work is distributed evenly amongst the edge clusters.

A

Second, with this technique, you can choose to actually close off kubernetes api server, access to these edge clusters and increase security, since you no longer need a central, argo cd instance to connect to them directly.

A

The biggest disadvantage with this approach is that, since argo cd is no longer running centrally, you lose out on a lot of its features, so there's no longer a central control plane to view and manage your applications instead, anytime, you want to want visibility into your applications. You would have to access the remote argo cd instance you're, also less flexible in how you can control your deployments so because this technique relies solely on auto syncing paths inside git to create and update applications, you won't be able to decide to perform manual syncs.

A

Finally, with this approach, you do need to make your git repository available to the edge clusters, which may or may not be a problem for your environment.

A

So next I'd like to describe some specific features of argo cd, which would be useful for managing edge clusters and the first is argo cd core. So if you do decide to go with a distributed, pull model and run argo cd on every edge cluster, you should know about this installation mode. Essentially, you can decide to only install the core components, namely the controller.

A

The repo server and redis and argo cd would act more like a basic bare bones, get ops operator, and this is a good option if you never had the need for end users to access an argo cd ui for these edge clusters.

A

But one nice thing about this feature is that you can still use the cli and ui as long as you have kubernetes access to the cluster. For example, you can run the admin dashboard command and then visit the ui through your local host.

A

Second, a second feature which was useful for edge, is application sets, and this is a feature in argo cd which allows you to automatically generate applications from things like paths inside a git repository or clusters which get registered to argo cd or even some combination of the two- and this is was created as an alternative mechanism to the app evaps pattern, and this technique works for both a centralized push or a distributed. Pull model.

A

So here's an example of an application set which is using the cluster generator, which is, would be useful for a centralized argo cd model and this cluster generator is creating a new application as soon as a cluster gets added to cargo cd. In the example we see here, this will create a nginx application targeting that cluster. As soon as that that cluster is added to argo, cd.

A

The next example is using a git generator and this generator will create application based on files or paths inside git. In the example we see here, this will iterate all sub-directories in the cluster-3 directory and then automatically create an application for that.

A

So we've been working closely with the ticktock team on many of the issues they've encountered in their use case, and these were some of the many improvements which they've made as they scaled out. Their usage and first they've made many import performance and caching improvements on the api server.

A

Second, with global edge clusters, the networks tend to be unreliable, and so there have been many tuning options introduced to make this more robust and in terms of future improvements, we're discussing ways where argo cd applications can live in different name spaces, and so this would help with things like name collisions as well as improve the ui performance.

A

If you do run a large number of application controller charts, we are introducing a utility that can help you automatically rebalance and reassign shards based on cluster size and finally, other features which the tic toc team has been contributing to are open, telemetry support, as well as the ability to exec into pods through the argo cd ui.

A

So that wraps up our talk on how you can leverage argo cd to manage your clusters on the edge and if you have any questions, please connect with us over at the cncf slack and thank you for watching.