Cloud Native Computing Foundation Open Observability Day NA 2022, 28 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Adopting Open Telemetry Collector @ eBay - Swapping Engines Mid Flight - Vijay Samuel , eBay

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Adopting Open Telemetry Collector @ eBay - Swapping Engines Mid Flight - Vijay Samuel , eBay

The Open Telemetry initiative is providing observability practitioners a vendor neutral and an open standard by which applications can ship telemetry data into observability platforms. eBay is a well documented adopter of Elastic Beats. eBay has been an extensive contributor to the Beats project and contributed its Kubernetes autodiscover module which can dynamically monitor pods on Kubernetes clusters. However, with the rest of the observability ecosystem converging onto Open Telemetry, we felt that it is essential for us to adopt the same. This presentation focuses on: * scale problems seen in large Kubernetes installations * benefits of using annotation based discovery at scale * our journey of replacing Metricbeat with Open Telemetry Collector without impacting metric collection * bridging critical gaps such as: * dynamic configuration reloading * semantic differences between Prometheus receiver and stock Prometheus * challenges faced in ensuring data parity post migration and solutions

A

uh Good afternoon, everyone, my name, is Vijay and I work for eBay as eBay's obserability architect. So today we are going to discuss about how we moved to open, Telemetry collector specifically uh and how we did it. While we are running at the scale at which we are today so we'll talk a little bit about what observability at eBay means.

A

uh What's our architecture, things that we use and what was the previous strategy that we used, how we evolved it over a period of time, um some of the design changes that were associated with it and then talk a little bit about the actual changes that were involved in the open, Telemetry migration and talk about what we are going to do next.

A

uh But before that we are hiring uh specifically to work on open Telemetry. Collector uh eBay is a great place to work. I've been here for 10 years and I considered it a blessing from God to be able to work in such a great company. So if you are interested, please use the QR code to find out more so observability at eBay.

A

So what does the developer? Do? A developer first. Does some instrumentation to they use either an open source Library, that's bundled into the managed framework or if it's a drop-in application. If the application is pre-instrumented, then it's it's used.

A

They onboard their application into the observability platform by going into our ebase Cloud console and say that this is the application that I need to onboard. If it's a, if it's a metric onboarding, they say that my application exposes Port, say 8080 Prometheus and they basically hit submit and then a lot of magic happens in the background where most of our scraping log file trailing and all that is part through annotations on top of kubernetes. So all the annotations are delivered into the appropriate kubernetes clusters so that the actual harvesting can begin.

A

So we have agents deployed on all our infrastructure, whether it be for logs metrics or traces. Once the agent has enough information about what to look for it goes and Begins the harvesting process.

A

So after harvesting is done, a user is free to set up any alerts that they want to do uh want to be alerted on top of so they can either use a threshold base alerts. In conventional promptule style, or they can even do anomaly detection using some of the models that we have built for the end user or if you have a TV screen on the hallway that you want to visualize or just on your laptop, you can build up dashboards using our console so so that you can observe what is going on scale.

A

This is something that I'm really passionate to talk about when we were taking stock of how much we have grown. We realized that we are scraping about 1.25 million, open, metrics, endpoint endpoints across all our clusters, and that translates into roughly 32 million time series per second that we are our samples per second that we are scraping and those translate to 1.5 billion active Time series.

A

uh We do 7500 odd queries every uh every second uh most of these translate into recording rules, uh either to roll up or to alert uh very minimal dashboards. So a lot of it is actually being used by systems to power intelligence, and we provided attention of one year for uh all the raw metrics that are being collected on behalf of Developers.

A

So this roughly sums up the architecture that we have for the Sherlock IO platform. uh On the leftmost side, you have actual applications that are being deployed on our compute influx infrastructure. It can either be VMS or it can be a kubernetes nodes on which the workloads are are scheduled into. uh We have two major flavor of applications that are there.

A

One is through managed applications, so we have managed Frameworks for uh Java node.js, which pre which come pre-packaged with some of the open source clients, or you can have a generic application that you're deploying once it's deployed the agent that's either sitting on the same node or the same cluster. They start harvesting the data it's sent into the platform into our ingest apis and, depending on what signal is being processed, you either have a metric store event, store, lock, store or tray store. We store it in and for a certain unique set of use cases.

A

We have something called a probing engine, it's nothing but stock Prometheus, which can either do SNMP walks or use a SQL exporter to query some metrics out of databases and those also follow the same ingest path. We have a query layer which implements uh the Prometheus API for us to be able to power either recording rules or anomaly detection or just simply view dashboards.

A

So when it comes to ingest and doing it in a in a cloud native way, what does it actually mean so for for for for us to be able to correct, collect metrics? We either need to be able to say that without knowing what the application runtime is, we should be able to scrape it in a generic way, regardless of what language you're using or you should be able to push it in a well-defined format.

A

So we have things like otlp or Prometheus or moderate, and we have open metrics endpoints that are there as well, but in some cases it's not necessarily easy.

A

You have Legacy applications for which you might need to have to do some sort of a one-off fire handling, but for the most part uh we we would typically rely on some open source, client uh and then we'll have some agents that are there, which can understand some of these well-known protocols or handle these one-offs through custom plugins, and we also need to be able to discover them in in specifically kubernetes, like environments where things are very ephemeral.

A

So either you drop in an annotation to say that this is very easy to look for or if something is being sent. Given minimal information like a pod, IP or a pod name and a namespace, you should be able to enrich all the additional metadata that was that was uh that was described as part of the Pod spec itself.

A

So in in 2016 we initially started using the elastic beads family. uh uh At that time, I think there was only a file bit and Metric beat was just coming along at that point in time it it looked really good to us. uh I think Prometheus was still a single server that that queries, uh the entire cluster, and we were already at a stage where it was like. We had several thousand nodes large number of Prometheus endpoints that were there.

A

It was difficult for us to say that okay, we can figure out the sharding and then set up multiple Prometheus clusters, uh Prometheus instances. So for that reason we we went for the Beats route because uh it was more Deployable in the in the demon set pattern so to speak, and it was an agent just built uh for being able to harvest in uh observability information.

A

So today we uh we- or at least until this point we have used filebeat for collecting uh collecting logs metric, beat uh also initially it was deployed as a demon set to be able to harvest metrics audit beat uh deployed again, as a demon said, to be able to collect uh file, Integrity monitoring events and any audit rules that were added into the kernel to be able to uh figure out who is associating in or escalating to root, and things like that and finally heartbeat for doing uh uptime checks.

A

So a quick refresher for the demon set pattern. A demon said in kubernetes basically allows you to deploy a single instance across all nodes that match the given node selector.

A

So by default it can go to Every nodes, all the nodes that are there or you can even say that okay, I just need it on a particular set of nodes that match a certain label selector, and what this agent that is deployed into a given node will do is first, it will communicate to the API server telling asking it give me all the metadata for all the pods that were there in this in this specific, given it is an old and it will again say: okay, I will monitor these specific Parts on these specific ports.

A

uh Once the data is collected, it will tag it with all the part metadata and then ship it out, and the demon set pattern doesn't necessarily limit you to tagging only to pods. You can also say that okay, along with the Pod metadata I, need to tag some labels that are there in the namespace OR. Certain labels that are there in the deployment object that the Pod belong to so on and so forth.

A

So for for those kinds of metadata, you need to pull all the deployment objects or all the namespace objects, because those are not necessarily node scoped. So what is the problem with this? With this approach? The first biggest problem is resource fragmentation, in the sense that every part that you are deploying on every node for the process that is running in it, it has some arbitrary cost that is associated with it.

A

So, for example, if you say that it, it costs 50 mb to run the Beats pipeline or the hotel pipeline or whatever, then this means that if it's a 3000 node cluster, it's going to cost you 150 gigabytes. So, depending on how many kubernetes nodes your you have across your data centers or your public Cloud you're gonna spend that much on just running the process itself and there's a CPU cost Associated as well, especially when it comes to pulling all the metadata from kubernetes and then maintaining it over period of time.

A

The other problem, when you run as a demon said you uh you're gonna, say that okay I'm gonna limit how much I'm going to spend on Metric, scraping to say one CPU and one gigabyte for every node that you are deploying uh deploying into so 1GB. While we say that Prometheus endpoints need to be concise, the reality is that people tend to abuse instrumentation over a period of time, so we have seen endpoints as more or less tens or few hundreds and endpoints as large as few Millions worth of entries.

A

A cube, State metrics is a good example for that.

A

So, if you are giving 1 GB for a metric beat that's that's running on a set on on a node where Cube State Matrix is deployed as well, it's gonna crash and when it crashes, you basically lose visibility into all the parts that are there on that kubernetes node as well, and given that you are watching the same artifacts across all kubernetes nodes, this is going to add unnecessary API pressure on on uh unnecessary pressure, because you're doing a lot of redundant watches across all the nodes that are deployed on the cluster.

A

So, in order to mitigate some of these problems, we basically moved into a cluster local mode in the sense that, rather than deploying as a demonstrate demon said, deploy it as a stateful set uh depending on the cluster size. You choose for every thousand nodes, I deploy, 10 instances of metric bit or open telemetic collector whatever, and you basically Shard the work across all all of them.

A

So if you do a simple hash mod, you can use the staple set size and then say that okay, this is the number, and if I am that instance, ID I basically monitor that pod or I drop it, and then someone else will end up monitoring that.

A

So this is how it would look uh rather than being at the node. You sit at the cluster level, uh monitor only the parts that you are responsible for and then send the data out once you have harvested it.

A

So what were the advantages of this approach? Tremendous cost savings. So when we, when we did this, we saved 90 of capacity as compared to running it as the demonstrate specifically for metrics and the amount of pressure that we put on the API server was substantially lower because, rather than 3000 nodes doing that now it's say 10 or 20 or 30.

A

Depending on how big the cluster is and given that we are running a finite number of PODS, we can choose to run bigger agents, say 16 cores and 32 gigs so uh or depending on the cluster, do a different t-shirt sizing. But this meant that we could now process much bigger, Prometheus endpoints without crashing. So this was one of the big advantage that we saw, but somehow this is still not enough. We are Never Satisfied.

A

So one of the problems that was there is that when you, when you end up doing a rollout or a new version, upgrade of the agent uh what's gonna happen, is that when one part goes down all the endpoints that were being monitored by that part, uh suddenly blackout, and if someone has said that I need an alert on an absent query. uh If, uh if the metric is absent for say one minute or two minutes, if it took two minutes for the part to come up, then it's gonna, it's gonna fire.

A

So this was problematic and even though we were limiting the amount of API server watches that were there, it's still a lot of redundant, API server, watches, and sometimes what happens is that if a given cluster has several hundreds of thousands of parts that are there just being able to do this, even with multiple workers on the on the control, Loop would still take several minutes for the entire thing to happen, which definitely means that someone is going to get alerted at the time of a rollout and naive scheduling. Hash mode is still hash mode.

A

It's not it's not rocket science.

A

So what we ended up doing is we thought? Okay? Maybe we should just decouple the discovery process from the from the agent itself, so that you have a centralized control Loop that can that can do the discovery and let some workers know that this is what needs to be uh handled by you.

A

So that's that's exactly what we did in the sense that we we we took the discovery process put it in a separate control Loop and we also started adding Intelligence on top of it, where uh we, we also started looking at other parameters that are there like CPU memory.

A

uh If a certain uh metric detail agent is taking too much time too much CPU resources, then maybe take a few endpoints from there and then put it into someone else so that the distribution is more complex than just hashmod, and uh we also made the configuration generation a lot more plugable in the sense that you have a language which you are using to Define your scrape logic uh or the scrape rules. Now you can use that parse it and then generate a configuration for maybe metric beat or file beat or even open, Telemetry collector.

A

So uh when open Telemetry came about, it was very fascinating to us in the sense that similar to kubernetes, where you say that I have a pod. A part is a part, regardless of it's running on a VM or it's running, on public Cloud, private Cloud, it's it's an API.

A

Now, the in the observability space, open, Telemetry brought the same notion in where you have you have an API. You have an SDK. You also have a collector that you can use to do arbitrary, Transformations and then write to any back end a backend, that's either owned by eBay or vendor, or an open source technology. This was a very powerful concept and we were, and at the time when the metric SD case were becoming stable, we began to re-evaluate to see that hey. Maybe we should get on to this.

A

We were already in the process of adopting tracing moving metrics and eventually logs made natural sense in that we would be in one family altogether.

A

So that being said, so we have put in a lot of effort to say that okay, we are optimizing on cost. uh We are solving problems that make it easy for switching into any agent. It should be very easy right, but the the journey wasn't as simple as that in the sense that now all features that were available on Metric beat. We needed some suitable alternative on open Telemetry, with the assumption that a certain feature was being used by at least one developer inside the company.

A

Are there any Show Stoppers, we identified one big gap in the sense that open Telemetry collector does not allow us to do Dynamic configuration reloading. This was one of the best features that we really liked about. uh Elastic beats that wasn't wasn't there scrape parity? Does a scrape from metric beat look exactly the same as it would look on open, telemetric collector or for that matter a scrape, that's being done by Prometheus? Would it look exactly the same on open, Telemetry collector?

A

If it is not, is it a feature or or is it by Design or is an actual bug? So these were some of the things that we need to. We needed to figure out, and the last one open, Telemetry collector is a rapidly evolving uh Community. There are so many releases that if we decide to move to a more recent version, how do we ensure that we are not regressing on any feature and causing issues at the time of rollout?

A

So we came up with all the features that we needed and what it would take for us to move to open Telemetry collector. Some of it was already available like metadata. You can use the attribute processor or the resource attribute processor order, discover we had figured out in the sense that we came up with our own color, with our own control Loop that can handle it. Prometheus scraping there is an alternative system, metrics there's an alternative, kubernetes metadata enrichment as an alternative and for everything else.

A

We were like okay, we can figure it out along the way, in the sense that uh if it's a feature that the community would accept, we would file pull requests and get it into the community or we can. We can manage it as a plugin internally, so to be able to address the the problem of not being able to reload.

A

uh At least what we have done at this point is introduce an internal receiver called the file reload receiver, which can take the car, which can take a partial pipeline definition to say that this is the receiver, and these are the processors that are associated with it. It would plug in into a standard set of processors and an exporter. So this what this would allow us is mimic the exact reloading feature that beats us and bring the same capability into open, Telemetry collector.

A

So uh what was what is essentially going to happen is that we would uh the receiver would watch all file, changes that happen for configurations that are being added and removed, and it would stop those partial pipelines or start them. If there's a change, it would update it so on and so forth.

A

This file reload receiver. We have, we have been running it inside of eBay for a while, now and I think we are at a point where we are comfortable to start working with the community to see if that is something that can be accepted into the country Repository.

A

So there's at a high level. This is how it looks like so if a person using the the Beats hints based Auto, discover language defines that I need to be able to scrape five zero zero one stats- uh and this is the namespace, the module, the script, frequency and script type mode.

A

The left side is what the metric beat would generate, and the right side is what we generate so that hotel can understand. So, as you can see, all the mappings that we needed to make sure that the configuration is compatible across the agents we are doing on the control, Loop.

A

And we ran into a bunch of issues, especially with scrape parity, where label sanitization, sanitization Prometheus was doing in one way uh open, Telemetry collector was doing it in another way and uh understanding colon as a as a as a first character of a metric name. Hotel was behaving a different way than Prometheus, so these are things that we have worked with the community and we have file PR's.

A

They have been accepted by the community, either behind a feature flag or as a straight out bug fix, and uh uh at least now we are in a in a space where we can say that: okay, the scrapes more or less match- and we can start. We can confidently roll out open Telemetry into our into our kubernetes clusters.

A

um We also had to put in a lot of effort to run um uh sort of pre-checks in the sense that we spin up metric beat we spin up open till midi collector. We point them to the same: endpoint uh get collect their outputs, compare the left side and the right side make sure that things are good and if they are good, then we move on to actually rolling out into their cluster. If we identify an issue, we triage fix and then move forward, we kept iterating and we you'd be surprised.

A

So many times we had to roll back because, as I had said, like one 1.25 million Prometheus endpoints, there are going to be several quote-unquote handwritings that people had in terms of how they did instrumentation some of it good, some of it not so good, but still we had to make sure that we do not break any metric scraping when we are moving from One agent to another.

A

So right now, uh I think this. This happened just last week we are done migrating out of metric beat into uh open Telemetry collector. We even bridged a gap of exemplars not being scraped by the open Geometry collector one of our Engineers actually filed a pull request that got merged a couple of weeks back tracing. We are using open, Telemetry collector and the open telemetest SDK right from day one and file beat also for logs.

A

We are actively figuring out how we can Bridge the gaps and, sometime next year, we'll be fully on open, telemetic collector for logs as well. uh We are still hardening uh by coming up with the regression Frameworks that we can run periodically to make sure that we can upgrade with ease.

A

That is something that we are still actively working on uh and uh over the coming days, weeks months, we will continue to work with the open geometrical collector Community to make sure that issues that we are seeing, we are able to contribute fixes, and we are also able to work with the community to make sure that we are enhancing the capabilities that the collector offers.

A

uh This is our team, uh Prem and Peter they're. The pillars of the agent group uh did most of the migration for metrics uh Aishwarya. She works on tracing. She helped with the Exemplar feature and then Edward uh he's helping with the logs migration.

A

If you like what you heard scan the QR code, we definitely would like to work with you on open Telemetry together.