Cloud Native Computing Foundation ArgoCon 2022, 21 Sep 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Unleash ArgoCD Observability Superpowers - Leonardo Luz Almeida, Intuit & Deng Zhou, ByteDance

Description

Unleash ArgoCD Observability Superpowers - Leonardo Luz Almeida, Intuit & Deng Zhou, ByteDance

At both Intuit and Tiktok, several instances of ArgoCD are responsible for managing thousands of applications. At large scale, ArgoCD encounters more challenges and performance bottlenecks. While ArgoCD’s UI is a great tool to visualize deployment issues,the nature of problems can go beyond your application boundaries and may actually be related to external factors (e.g. github api is down, argocd controller is overloaded, etc). In those cases, it is much harder to identify the real culprit. Digging in logs can be very useful in identifying internal issues however it requires precious effort and deeper ArgoCD knowledge. ArgoCD’s built-in observability capabilities take away the needle in a haystack feeling for troubleshooting deployment failures. Understanding the root cause is the key element to maintaining healthy SLOs. Come learn from 2 ArgoCD contributors about how to leverage distributed tracing, metrics and continuous profiling to quickly identify and react to the hard problems.

A

Hi everyone uh welcome to this talk today. I'm gonna be uh uh talking about how to unleash argo, cd, observability superpowers.

A

My name is leonardo. Everybody knows me by leo I'm one of the argo maintainers. I help driving the contributors meeting. Maybe you already met me in that meeting and I'm going to be presenting you today. Unfortunately, dang was supposed to to join me today. He wasn't able to to come, but he provided me the the recording and I'm going to be playing it during his part.

A

So let's get started and I I wanted to to start just providing uh a real, a really brief story. um What the problem uh this is uh trying to solve. So imagine uh that you are uh passionate about technology and uh you work for this company and developers are having issues deploying uh components in argo city, they're, the applications in argo city- and um you have this uh sorry to deploy applications in kubernetes, and then you have this.

A

uh This is a great idea to to install argo cd in uh in your company's infrastructure and suddenly is a success. Developers start using uh uh argo, cd and deploying their application, and uh one day one of the developers come to you saying they. They can't deploy their application, there's a problem going on in production and they ask for your help and uh you check the ui there's no much of information there. um So your nature is to inspect the logs and you open argo city logs, and you have that feeling.

A

uh So you have to find which one is the faulty wire right and you don't know much about argo cd. It's a little bit intimidating to to to understand the the logs, and uh this talk is about how you can uh use uh some observability uh functionality that is uh built in uh inside argo city uh to help you whenever this uh this day arrives.

A

So basically I'm going to be talking about metrics and when I say metrics is about prometheus metrics um and I'm going to be talking as well about distributed tracing how argo cd leveraged distributed tracing uh and uh enables you to have uh a a much better overview of. What's going on in your uh in your argo city, instance and dang is going to be uh talking his recording about uh uh how they run continuous profiling.

A

To find argo cd bottlenecks at a large scale, so then then works for forbiddens, which is by the way the company uh behind tick tock applications. So it's a very large infrastructure. They have there. So uh speaking, a little bit about the numbers, when I men, when I say about large scale, running argo city in large scale environments, so we operate. My team operates argo city at intuit, uh just really brief numbers here uh into it, has more than four thousand uh fourteen 000 employees around 20 locations.

A

We have more than 2 thousand production services to support that. We have 44 argo city instances which sums up more than 16 000 argo city applications. So you probably already know our an idle city. Application can sync several resources, so it is uh uh uh yeah quite a bit uh uh heavy load environment that we have and that and and that we have to support argo cd, uh add into it.

A

So, let's start talking about metrics and how we leverage prometheus metrics that intuit mit, um and uh so basically, if you check the uh the the documentation, basically uh you're gonna, be able to see the argo city has built built in um several uh prometheus metrics that you can use to really inspect how applications are are are updating so, for example, uh the first one here in the list I'm going to provide you an example.

A

uh So uh you can write queries, for example, to uh check for the the health styles of your application, and whenever this this health status comes from healthy to uh degraded, you can configure to send alerts to your team, so you can act on the on these issues.

A

So this is, uh there are several uh metrics that you can use, um and one thing I I wanted to mention is that there are on the top here, different components that you can see uh and just talking a little bit about the argo city architecture, because things are going to be much better to understand.

A

If you understand a little bit about internals uh uh on how iro cd is is designed, so basically users uh interact uh with api server, and uh so this is the same component that serves the ui. So most of the operations are are provided by this component, which is pretty much a back-end service.

A

We also have repo server and sometimes depending on the the your configuration of the cmp server. So the boxes on the on on the top represented with the orange color are basically backend services and the ones at the bottom in green are controllers.

A

So just uh showing you what I was displaying the in the previous slide, those metrics are exposed in specific ports that are already configured in argos. There's, no specific configuration that you need to to to to do to to have those metrics available. The only configuration is really pointing prometheus to to scrape those metrics from those parts and you're going to be able to to interact with those metrics.

A

All right, but what we do actually insight into it to leverage those metrics right, so um we uh leverage uh grifona dashboards. uh We have a collection of uh uh um so we have our dedicated alert dashboard, that's how we call it, and um so basically you wrote a collection of prometheus queries, leveraging those those metrics and whenever uh an error happens that we know we need to act.

A

We send a pager duty alert and the thing we we like, I think it works really well for us- is um uh if there is a problem that needs human human interaction, to help solving the problem right in the premier in the pagerduty aller. We provide a link, so we can just uh uh click that link. We go straight to the dashboard and we have an overview of what happened and we have some history, so it it.

A

I I think it's something we found really useful to have alerts linked with dashboards, so it provides some visual uh whenever something goes wrong. So this, what I'm showing you here to you is the is a default dashboard that is provided by argo cd, repo. You can uh uh may start using this dashboard today, if you're not already using in your infrastructure, pointing to it.

A

So this is a basically general purpose dashboard but, as I said, the biggest advantage you have is by the time you start tweaking those metrics to uh towards your needs and sending alerts whenever you need some action item: okay, so, but what? If you're running argo cd in some sort of a multi-tenant environment? So what? If you're, sharing uh the same infrastructure across different teams? So if you're, if you just query one specific metric, what happens is that you can you can't easily route alerts specifically to a dedicated team, for example?

A

So this is uh one feature. That is uh that I wanted to quickly show to you that argo city provides a similar feature that is available in cube state metrics. So, if you're familiar with that, it allows you to define some labels in your application resource. So here you can see that I'm specifying two hypothetical labels, so app team platform, app bu business unit infra, and if you configure argo cd with this.

A

Command parameters, you know what argo city is going to do is uh basically start uh emitting a new metric called argo city, app labels with the values that you define in your application resource, which allows you to join this metric with all the other metrics that work that I was showing before.

A

So you can uh basically uh do a much better routing and on those metrics and send specific colors to specific teams. So this is available today, yeah and moving forward distributed tracing. This is a new feature. We introduced in argo, cd 2.4, um so argo cd, 2.4 embraced open, telemetry, um so yeah there was a a little bit of work to get things updated, especially on the grpc side, not getting into that. But uh the truth is yeah. It's it is available, there's not much!

A

You need to do to enable argosy to start uh emitting uh traces uh to any open, telemetry collector that you have in your infrastructure, but the only thing you basically need to do is to edit this config map. So if you're not familiar, uh this is something I wanted to highlight as well.

A

I'm not sure if there are too many uh developers aware, but argo city provides this config map called argo city, cmd params, so before we used to ask developers to patch ardo cd containers to customize uh how they want to configure configure specific uh settings in the in the in the tool, but with uh this is not always uh great uh because, for example, for the open telemetry configuration you would have to patch at least four different uh containers uh to add this additional configuration, which is basically the same configuration for all of them.

A

So argo city provides this config map called argo, cdcmd params cm. You just define the the the attribute. We have a complete documentation about all the possible configurations. You can add in this config map and it automatically injects the whatever you define it in the appropriate uh containers. So this is just a tip if you're not using it yet and you're still patching argo city containers. This is a maybe good thing. You can start using so to have a complete list of attributes you can configure by using this configmap.

A

You don't have to memorize this url. Just remember that we have a section in rgo city, documentation, called operator manual and inside you have a page called declarative setup and inside this page, you're going to have a link for all the config maps. We have okay, all right, so let's talk a little bit about distributed tracing and this. This is not argo. Cd ui this this. uh This screenshot here is extracted from uh jaeger ui uh that I used to configure.

A

uh So I configure argo cd locally running locally uh in jaeger, which is an open source tool to collect open, open, telemetry traces.

A

So here you can see that immediately uh the there is a something that is made available in the ui, which is most of the uh distributed tracing uis. The the ones that I used to will provide a similar uh diagram. That shows you the interaction with the different components that you have in your application. So, as I showed you before in the dye in the architecture, diagram, argo cd is not a single uh pod that you just install and runs. So there are several components involved and there's a an argo cd to work properly.

A

Those components they need. They need to collaborate together so and by leveraging distributed tracing. You can see those those components, the the interaction with those components and whenever some of those components fail, uh usually the ui will provide you a way to navigate uh directly to the to the to the point where this this failed. So, for example, um um argo cd communicates with github to get the the desired state, and then it reconciles this desired state with the live state and provides you the diff that you're already uh familiar in argo city ui.

A

So uh just with this very simple example you you understand that argo cd depends on the connectivity between uh your rdo city installation and where you run your git repository in order to calculate this diff right. So if something goes wrong, if your argo cd is communicating with github, for example, and github is down, argo cd is isn't able to to calculate this stuff.

A

For you at that point, and relying on tools like distributed tracing things are going to be uh much easier to to to realize just by looking in logs or even looking at the ui all right. So I recorded a really quick navigation in um uh in jaeger ui, uh mainly to show you uh how it looks like and to have a feeling about what are the type of information you can get from it.

A

So you can see here on the left that you can filter by the different services and those services mean the different components that I was showing you in the argo cd architectural diagram, and you can also see the different operations. So, for example, argo city needs to list applications uh and to see all to show you all the applications that are deployed in a specific uh argo cd instance, and that is served by this grpc- uh call that that that needs uh the the collaboration of two uh components.

A

The one represented in orange are the api server and the one represented in green is the api that is called uh that we call it's a component inside argo cd that we call repo server. So this is the grpc call, and it shows here I'm showing you the the operation.

A

You can see all the details and inspect and you can expand the tags and see the details of it. Okay. So this is a pretty simple and quick navigation on the the type of information you can extract by enabling traces, but these are all happy scenarios right. So let me quickly.

A

Show you what happens if uh something is not right. So in this example, uh what I did I uh manually killed repo server and called this in, and I called the same um uh navigating in the ui. I invoke this uh this service to to list the application. So as this operation requires uh the connectivity between api server and repo server, obviously ripple server is down in the distributed tracing view.

A

You it's very easy to spot where the error is. So, if you click in the in exactly the point where you see this uh red uh circle, you can inspect the traces tags and you can see that there is something going on between a connectivity between api server and repo server. So this is the type of visibility that enabling distributed distributed. Tracing in your infrastructure, enabling argos cities within a tracing or infrastructure is going to provide you, okay, so moving forward in a presentation.

A

So this is the time uh where I'm going to play the dangs recorded presentation and again yeah he's going to be.

B

Good morning,.

A

Let me see if they're recording.

B

Everyone I'm dan from the buy dance agents. Today, I'm gonna present how we do continuous, providing our age platform.

B

This is the agenda for this talk. We will start from the definition of the continuous profiling. Then I will give an introduction about the high level architecture of the biden's age platform. Further, that the deployment under the setup of the continuous profiling and finance platform will be discussed.

B

Last but not least, two case studies about our city performance troubleshooting will be presented.

B

So what is the continuous profiling providing is a dynamic method of analyzing? The complexity of a program such as cpu utilization, all the frequency and the duration of the function course with providing you can locate exactly which part of the application are consumed in the most resources.

B

Continuous profiling is a more powerful version of providing that added the dimension of time. There are two major type of profiling tools: instrumenting profiler and sampling provider. The instrumenting profiling inserts code into function calls to collect function, execution details, while the sampling profiling, periodically collecting the function, call stack to process, it represents estimate, function, call duration,.

B

This is the high level picture of our age platform architecture in binance. We have hundreds of syrian pops. Each part is converted into one or more cognitives and are managed by the edge platform, so the edge platform is the control plane of our age, cluster federation of aggregations.

B

The platform is equipped with infrastructures like storage, monitoring, login tracing and so on. It provides unified console to expose services like github's, matrix, building, iem and other product level features. There are actually many age workloads already running on the platform by sharing the resource and infrastructures like cdn, proxy rtc, api and game and other services.

B

So why do we want continuous, providing on our age platform? There are few motivations. First of all, we are seeing scalability challenge on our system. We have hundreds of edge clusters all over the world and more than five applications running on it. All the applications are managed by a single iocd instance.

B

So we want to understand the performance issue under the root cause. Second, traditional performance test environment is hard to set up for us. Our age cluster connection is not reliable. Due to networking stability, which is very hard to mimic in the testing environment, second, our application of the this scale is very hard to mimic um in terms of pattern. Even we can have similar number of applications.

B

We are not able to mimic the unproduct activity and the remote health status, and then our age platform use continuous delivery model and we ship new features and fixtures every week as lot of feature enablement need corresponding age, product change or update. It's very high cost to simulate all the other changes.

B

Last but not least, even if we can do it, we have no intention to set up a very cost problem, reproduced environment in terms of both money and engineer, because because of those using continuous, providing tools to collect information from the production environment for the troubleshooting is a friendly solution for us.

B

So this is the setup for continuous profiling on our age platform, each every piece of the software about kubernetes is deployed via github's manner and managed by the aggro city. We have two profiling tools deployed the parka and the periscope.

B

Currently, only the control plant cluster enable continuous, providing all our targeted services are running on the control plant cluster, including sonos premises, argo cd and other self-built services like resource manager, building ticket system and so on.

B

The first step I want to share today is the argo cd application needs performance. As our internal assi become multiple project owners. We are seeing that his application listing waiting time is several times longer than our admin. This is a finger captured and provided by parker. One of the tool we are using with the performance decoration lasts from the ice frame figure. We can find out that application service list handler take most of the waiting time inside it. We realized that our bag enforcement takes a significant significant amount of time.

B

After checking with the source code, it turns out that our city will go through the owner role list to check the permission of each application for the user when they do the application list 5000 application times. The length of the analyst is a major source of latency.

B

As a quick workaround, we give that we give the sre the admin instead of owner list. The list latency has reduced from 40 seconds to seven seconds beside the workaround. We have redesigned the permission management by using of the project role instead of global role to provide the checking.

B

In addition, we have figured out that aggro cd does not have a good question about permission. Interpretation results. We have further id the question on that with the fix the application nesting can reduce to two seconds instead of seven for admin.

B

Another case we want to discuss here is about the cpu efficiency improvement for argo city, with the con continuous, providing help. Rsi observed that aggressive application controller has a very high cpu identification in most of time, even though there is no user activities.

B

So we took took a look at the profiling data collected by periscope, which is another tool we used in our system. We observed that the function secret to cluster, which sounds like a quick lookup, consume significant amount of cpu.

B

So we want to know what happened inside of this function by checking another profiling data point, as shown in the in the in the slides. It turns out that the unmarshall of the cluster secret is the major cpu cost.

B

Actually, the cluster secret is a very large data object which contains all the information regarding the cluster includes the certificate and the connection credentials, but the code shows that the controller just need to get the get the class name but unmatched the whole secret after reporting. This argo cd upstream has fixed it by just adding an index of that, and the cpu utilization become much better since then,.

B

For anyone who are interested in learn more about continuous profiling here is the reference. The first one is about what is the continuous profiling and how you can use it. The second thing is about another explanation of what is continuous, providing and the different type of continuous profiling and the third one and the first one are two different tools we have. I have used and demonstrate in our case, so these are the two findings we want to share today. Among a lot of cases we encounter internally.

B

Besides argosy, we actually find problems, performance problems in sanos and our self-building service as well. Continual profiling is indeed an excellent tool which can help address the most unreproducible problems issue with very little overhang. Thank you.

A

Okay and mainly to wrap up this stock. I have this last slide mainly to provide a conclusion.

A

So in this talk, we spoke about where we are and into it, and how we leverage prometheus metrics having a dedicated dashboard that we use internally.

A

We I also spoke where we're heading so distributed tracing, was enabled in argo, cd 2.4. This is not still configured in our infrastructure. This is a work in progress, but we're we're looking into going in that direction and improving the the instrumentation that we have inside argo city and dang showed how they do uh continuous profiling at bite dance to find problems that are really hard to to to reproduce, because it requires a really huge environment to to happen. Okay. So this is what I had for you today.

A

I don't know if you have time for questions.

A

Any questions anyone.

A

A