GitLab GitLab Tech Talks, 1 Feb 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: From Monitoring to Observability Left Shift your SLOs with Chaos - Michael Friedrich | CC2022

Description

This talk introduces a developer`s view on using cloud native resources and mistakes turned into visible failures. Learn about the first steps with metrics, SLOs and app instrumentation, quality gates in a CI/CD pipeline, and chaos engineering - my experience in the past 15 years.

Michael Friedrich is a Senior Developer Evangelist at GitLab focussing on backend dev, ops and sec. He studied Hardware/Software Systems Engineering and moved into a role on DNS and monitoring development at the University of Vienna and ACO.net. Michael was a maintainer of Icinga, an OSS monitoring software, for 11 years before joining GitLab. He loves engaging with new technologies in the Monitoring/Observability, CI/CD and security area and regularly speaks at events and meetups. Current projects involve Prometheus exporters, tracing and pipeline efficiency. Michael hosts a technology coffee chat called “#everyonecancontribute cafe” on everyonecancontribute.com.

Connect with Michael,
Twitter: https://twitter.com/dnsmichi
LinkedIn: https://www.linkedin.com/in/dnsmichi/

A

Hello, uh hi everybody. This is serenda and venkatesh your host for the session uh on the on the talk uh monitoring to observability left shifter slos with chaos, so uh without any further delay I would like to. I would uh allow michael to introduce himself and let's get started over to you, michael.

B

Thanks a lot, and thanks for having me um yeah, my name is michael: I'm a senior developer evangelist at gitlab. um You can find me as dns michi on social media. um This is dns m-I-m-I-c-h-I, it's a lovely form of martin german hard to pronounce in english, but today we will want to talk about from monitoring to observability left shift us with chaos, because we love chaos.

B

um The thing is, we really want to get started and for getting started. I want to turn back time a little bit and talking about it an as a retail like we started with monitoring. We have black box monitoring, state-based monitoring, traditional sla reporting, state changes and certainly metrics over time.

B

After a while there was service level agreements, objectives and indicators uh introduced, um ensuring that we really may define what we want to measure and what we want to achieve so like achieving 99.9 availability um is the best goal you can have for the customers and the agreements. It would be 99.5, for example, and then there's a certain term with error budgets as well, where you need to really define what can go wrong.

B

There's a lot of lots of things to learn in there.

B

Also something which we learned about is golden signals: how to figure out what is going on going wrong in your environment, like choosing um the signals, latency traffic error situation and at some point we needed code instrumentation, um so maybe uh site reliability, engineering invented by google.

B

Did it solve everything, or maybe not, I'm not sure, um but I really want to dive into my own history as a developer and many years ago I started out as a back-end engineer, using c plus plus, and we have been working on a back-end daemon, a monitoring application which was running as a service and at a certain point we experienced a slow rest and json rpc rp api responses, because the cpu was overloaded with threads and in our environments it worked.

B

Fine uh customers were not so happy, so we thought about hey, let's introduce co-routines, which we learned from golang, but just let's use it in c, plus, plus it's lightweight. It's stackless. You put the function point on the heap to pause it. You have some sort of stack unwinding continuation inside and this solves all of our problems.

B

Well, it did not um the thing. Is it introduced different problems, but so weirdly um that it only happened in production after a few days, but and only with like a thousand api clients which you don't have in your local test environment, we had memory corruption, but maybe it was also exhausted because of leaks.

B

We're not really sure about it, and windows also had specific other crashes, which is super nice to debug in a virtual machine with stackguards and there's a security scanner and after a while, I figured it out, but in the end I was totally burned out from that. So I thought well, maybe in a retrospective, how? How can we improve this whole process of not running into memory corruption in the first time? So we really want to meet the opposite requirements of when we are changing the code.

B

um The memory usage shouldn't be increasing by, for example, 10 percent and defining that as the service level objective. We want to measure that in real time in the production, environment or the staging environment, and when you think about chaos, engineering for that regard, maybe let's just use kubernetes deployments for testing and not some duct tape, virtual machine, whatever deployment which we had, we want to have the api requests from the clients and add chaos, maybe to speed up the memory, corruption, to really brute force the problem and being able to detect it further.

B

The other thing which I've seen and where, where slos and chaos engineering is, is a great example, for um there was a story about logging into a sas platform, specifically an online game called gta online, where the settings and the algorithm for sorting the json rpc keys, cost login times of like 5 minutes or 10 minutes or something. And since this is closed source community members reverse engineered the problem injected at dll and then made it faster. But it kept me thinking.

B

Maybe we can also help improve that um by deploying everything in a ci cd pipeline, defining the end-to-end test scopes the user login time the time until the user is playing and think about the login time as an slo um for like it should be shorter than two minutes.

B

But we also have uh low latency connections, basically everywhere on the internet, and maybe then it should be just five minutes and giving it a little um more more ground when the ci cd pipelines detect those problems, we have quality gates and stop the release process and, and it never hits production, never hits the customer with chaos engineering um in at the at the top of your head. You can also think about like limiting the traffic or intercepting the requests and really causing chaos for the login procedure and detecting the problems earlier now.

B

The other thing I want to talk about is dns and it's obviously my nickname or inside my nickname. It's always a dns problem when something on the internet goes wrong and when we turn back time to 2011, the 8t cctld got the nsk for securing things and and so on. It needed a signing hardware, which is basically a state machine of steps and on a friday afternoon someone changed the script, so we had no more signing. We had no dns updates.

B

Yeah monitoring was interesting because it was wild. um I guess who got the alerts in the morning.

B

Well, I didn't get woken up in at 3 am in the morning by email, but when the sms started to fire for all the name, servers which was like 30 sms in a minute, um it was not fun and when you're logging into a console at 5, vm and debugging things, it doesn't make sense, but we kind of figured out that the change was persisted in git, but it was still rolled into production and we came to the idea.

B

Well, let's maybe measure that, in a way that we have staging signing hardware, we are rolling everything with infrastructures, code or github's workflows.

B

We are also doing rollbacks in that regard and we try to measure the zone serial h, which is an indicator for um figuring out whether the dns zone is up to date or not, and thinking about chaos, engineering. We could, for example, deny the zone updates, write our own chaos name server.

B

Maybe I do that in the future um and also like 20 different zone series and see what happens in your environment and see how many things are gonna fail.

B

The other thing we we saw, I think, last year, somewhere in november, uh facebook dropped the bgp routing announcement on the internet. So basically it was not reachable and all the query resolvers had the problem that clients asked and asked last. So basically, we tried um to figure out where facebook.com was gone. No caching, everything is on fire, maybe like. How can we prevent that? And in order to prevent that, we also got to learn that they are using a bgp agent?

B

The custom development policies, test, testing and deployment using continuous delivery um and everything is sounds good for preventing dslo failures or like the failed policy pushes can define an slo, but we can also think about chaos, engineering and say hey. Maybe we can change pgp real thing in a way that it presents uh wrong routes or may even make the data center unreachable and prevent um the deployment actually in production um and in order to really figure out. What's going wrong, be an interesting experiment in the last story, I want to share with you around.

B

What's going wrong or could could potentially be wrong, is when you're deploying something you're, probably using containers or container images, and you might be using docker hub, so in 2020, docker announced rate limits, um and this is a great thing, um because when you, when you provide a service, you should be getting money or any anything else for that. The problem in that regard was, we didn't really know. What's affected, we thought about ci, cd pipelines, cloud native deployments, organizations behind the nut or anything any cloud providers which we are dependent on most often times.

B

We did then said: well, we have a known state. We can check the limits, so we can kind of send the re. Send requests, get a response simulate certain things. I even went ahead and wrote my own promises exporter for that and really had an insight. What is going on now.

B

The other thing we didn't really know about was well the docker docker pull in any environment in any script, in your kubernetes clusters, in ci, cd pipelines and potentially in the logs with too many requests as an error, you need to pass out because the real problem is, if your deployments don't work at 100 level, some customers might see a different price to others and then say: hey, I'm not paying two hundred dollars, I'm paying just a hundred dollars and in the end it was your rate limit which was uh reached and the deployment didn't hit production now, adding slos and chaos to that idea and trying to help prevent that we can measure or monitor the pull counts and the remaining pools- and this is an arbitrary number of 10 remaining um and one day slow is failing.

B

We are preventing that from a quality gate and don't even start a new deployment, just saying hey, where over the limited. It's done. The other thing around this around that is adding a chaos experiment to that. Maybe a container registry proxy limit the client's traffic and see what happens in your production environment, because when it affects everyone, the development team is also blocked and the reviews are blocked, and basically everyone is using.

B

And slacking slaking out now, how do we approach this problem or these things, and I always keep thinking like when I'm a developer? I really need to learn ops, um and this can be hard. I really need to understand, for example, um slo, okay service level objective I wanted. I want to monitor something. I need to dig into metrics what what are they it's defined as a key or attack? Okay, it has a value, but where else would I go?

B

The thing is start easy: choose a tool, choose a language, I'm choosing, or I chose promises in prom qr, which provides a metrics metrics format and a query language which is relatively straightforward to learn. There are many resources out there to get going and I've linked everything on the slides and you should be having access to the slides already to learn async.

B

The thing is: what can you monitor with promisius? You can monitor the infrastructure with memory cpu io, you can use exporters on the node or port or cluster you might be using or might wanting to monitor services by using promises, exporters and the last thing which is important for developers app instrumentation in the code. This is something you probably need to be actively doing now how to get started. How to learn it well, learn with playful examples find a use case. Maybe you want to learn a new programming language um or just solve a case.

B

We had a twitter discussion yesterday about this topic um and really get going, maybe use the promises operator to deploy into kubernetes and then inspect the metrics use, promisius grafana, ops, phrase, whatever you prefer in that regard and learn and continue and have success.

B

The thing is, there are more than metrics, so there are so many things to learn in that regard. There are metrics logs and events we have distributed tracing. We have continuous profiling, probably you're, shifting from monolith to microservices in your environment as well, so it can be quite overwhelming and I would encourage you to really slow down, breathe and continue. Learning and looking into app instrumentation on your learning path.

B

Now the thing is when we look at logs, you have to make a lot of decisions and you need to dig into what's possible what is needed. How many hardware can you afford?

B

Yeah you have things listed over here like elastic, open search, low, key click house times, kdb prompted fluentd, and so on many many things and decisions to be evaluated and as a developer, you need to learn structured logging, making it more performant to really analyze the docs later on and learning structured logging is a whole thing, at least for me, was a whole new thing to learn as well.

B

The other thing um when logs are little too complicated to approach um would be tracing and they are different to logs in a way uh that they are defining spans with a start and end time. You can enrich them with metadata. What's going on in the context, and you add the instrumentation to your code, you can use different backends jaeger tempo. Whatever is out there and in the future, you can already use open telemetry as a specification and as a collector, which is awesome.

B

Open, telemetry itself is a specification in framework which provides support for instrumentation for traces, metrics and logs, whereas traces is stable, you can either manually instrument your code or use auto instrumentation and the most important thing I needed to learn for myself is you? Can you need to bring your own backend? So, for example, you use promises for metrics here for tracing open. Telemetry does not provide that for you.

B

You will also see the term vendor distributions, sometimes, which basically means um it's a whole new. It's a packaged version of open telemetry with additional metadata, which, with additional features provided by the vendor, was important thing to to know about the other thing.

B

Moving on with observability and slos is to think about performance, application, performance improvements and also getting insights which can be based on continuous profiling, like adding, alerting, integrations, adding slos to that and think about maybe the four pillars of observability, but probably it's in it's an old term.

B

Now, still, I do see matrix traces logs and profiling as the key elements in the future to really benefit from and and go check it out now, when we think about um using all that knowledge, how can it help in in our workflows in ci, cd and so on? um We can use slos with promisius, so the metrics and the alert calculations. The query language, prom qr happens in open, slo format.

B

We can use the values we get out of that with a quality gate which can be kept, for example, to prevent that specific regressions or specific performance decreasements get deployed into production.

B

Captain works in a way sitting in the middle, basically as a quality gate and curing prometheus for the deployed application metrics, and if the slo fails, for example, the pipeline is red and the deployment is being stopped or maybe in the future being rolled back.

B

The other thing is: um captain also provides like the configuration and a front-end, which is pretty awesome. It also has a live demo, and it's super easy to get going with would encourage you to check it out now. The thing is: shifting left with quality gates and slos um is one thing to make your life easy in the future. So you have continuous delivery, kept as a quality gate.

B

Promises slows, for example, but when you are when you really want to simulate a production incident, as this can be hard and one of the ideas I had well, let's just add chaos to the deployments and see if the slos fail or not.

B

Now, how can we shift left with chaos again, starting out as a developer, trying to see the benefit? What are the steps I should be taking on the terms?

B

Well, we have cloud native, we might be using clusters kubernetes, we have deployments. We were using a tool, for example litmus chaos or litmus, and we have experiments and if we want to create our own, we have instrumentation as the case where we get to code now um short introduction litmus chaos.

B

You can fail your infrastructure, you can see how the app behaves you can see if the slo still match, and then you can define actions and improvements by the way the documentation is super awesome. I got the five minute demo working and it encouraged me to dig even further and keep in mind or remember what what I told you earlier about the stories and the mistakes I made and experienced on the way we can use chaos for all these apis sas problems, dns and document images, because we can randomly introduce low connection api clients.

B

We can use login proxies with endless wait times, create our own or just use the tools which are already there. We can intercept dns traffic, it's super easy, but don't tell the security department or you should be telling them and say hey. We are doing a chaos, experiment now and maybe introduce something like registry proxy. Limiting the container pools like to simulate docker upgrade limiting um might be needing your own demon, but still there are many many ideas out there in order to simulate chaos in production or to provide chaos in production to simulate failure.

B

um Your own chaos can be started with experiments you can use, so litmus, for example, provides sdks for go in python. There is a koci lib, which I discovered today, integrating it into your ci cd power plants, gitlab and and so on, and there are many many tutorials out there, um but it's not bound to litmus. There are I I've seen so many great um tools out there, but I don't have time it's just one to really figure out and try and then see how things are going.

B

Our journey goes from do-it-yourself monitoring to observability, which means we want to left-shift the slos, so we can use instrumentation and observability and make that part of our workflows, and we can use the benefits of cloud native to use all the scaling to add chaos in container clusters to really see if the slo still work for ourselves. We really need to see the values, um the value in metrics, traces and logs, so also providing application.

B

Insights for non-developers sounds a little odd, but when an engineer or an sre is woken up at 5 p.m and 5 a.m in the morning, they really need to figure out fast, whether it's the code or it's something else in the environment.

B

So, adding that, proactively to your code, to your thinking to the workflows, can really help everyone on on the team on the company and also start using boring solutions. Don't try to fix everything and understand everything and getting blocked by that think about starting with metrics endpoints, with structured loggings, with dashboards, with trying out open source tools which are available and have an awesome community.

B

Also don't do everything yourself use chaos, so not only use chaos, engineering, it's on its own, but also think about like fast testing or security testing, or something else. Everything which remains, which does something which is not expected by the application and forcefully try causing a failure.

B

um Also think about observability methods in a way of you have has lows and quality gates, um but you might be collecting events or signals already which, um which you don't know about, and you can benefit from so embrace the unknown unknowns, learn, documented, educate.

B

Speak a little bit over time. I have told my so much to tell this is the wish list I want to want to have more ci cd observability in the future machine learning to correlate everything.

B

I also want to auto generate slos, because it's a lot of yammer to write, which I don't want to write, and I also want to have chaos built into observability platforms, because I do think that 2022, we will have a theme for platforms, so we will have an observability platform, devops platform, chaos platform, maybe your own platform um combining the three or combining everything and providing a platform as a service to your developers.

B

So they don't maybe don't need to learn instrumentation in the future. Now, as a short recap, we learned about app instrumentation quality gates a little bit of ci cd, observability chaos, engineering and platforms everywhere.

B

um If you want to reach out and connect, this is me, and I also would encourage you to check out o11by.love, which is for observability love. I created this as a free learning platform. I would love your thoughts, maybe contributions thanks for your attention.

A

Thank you. Thank you, michael for this wonderful session. I truly enjoyed it and I hope our audience found it uh insightful too. So can we have? We can have a little q a session as well yeah. So, of course,.

A

So this is the question from hemanth. uh Adding chaos to pipeline definitely incurs more resources and cost. When and how do you identify to apply chaos to slos in order to prevent potential over engineering for deployments.

B

It's a good question, um obvious yeah. If you, if you're using chaos engineering, you really need to have either a staging environment or a production environment, why you want to do it define defining the right limits of not to spend too much, I think is hard and I'm not sure if we all have experiences from that already or if it's documented in the central place.

B

And when, when and how to be honest, I would try it out and set set a limit, um and this could be like set set a spend limit in your favorite cloud and and see how it goes. um But I do understand yeah. The problem is, if you are adding too much to your environment, um which causes trouble, you might be investing money which you don't want to.

B

Thanks for the question keeps me thinking.

A

Yeah, I uh yes moving on to the next question. I'll just share it yeah.

A

So the next question is there any open source implementation of such slo to get started with the silhouette kiosk.

B

I I would recommend so if you want to get started with slos, um there is the openslo specification, which is openmslo.com, which defines the format how an slo should be looking like should be described, and there are certainly tools around which implement that um and for, for example, for um for prometheus.

B

um You wanna, you wanna measure the slos from from the prom calculate language and there's a tool called parser which, which is open source, which allows you to manage the slos. I think it's parser.def.

B

B

But where you can also like learn from that, the thing is, you need to um focus on on on the steps. So, like start with, metrics start with the data then um measure the slos, define the slos and then go ahead and add chaos and see how the slos actually behave.

A

Yeah, I hope your question has been answered. Moving on to the next question, what kind of metadata would work to define metrics for components.

B

What kind of factor define metrics for components?

B

I think I think the question is probably more like around tracing. Why you can add metadata.

B

um I think, like um imagine, that you have an um a front end and a back end and it's a web server and you have a database back and then some caching in between and you really want to see where the the problem is so like the duration of the request is long and the client times out, um but is it the redis backend?

B

Is it the database backend and you can enrich the trace with a specific content and say: hey, I'm now in this um I'm now connecting to redis and for some reason it times out, and I can, for example, provide the client, libraries, client level reversions, maybe the backend versions, everything which, which I already have in memory, which is not a numeric value, but the string or something can be helpful in that sense.

B

um Think about um when you open the browser, and you see the network connections or the the things which are happening with javascript and so on. This can also be seen as a trace, and you really want to see why the website is slow. Is it the javascript? Is it the client request? Is its browser resources and so on?

B

I would encourage you to not try to add everything at the first glance, but start with one metric or trace. You want to add and then see how it goes and follow along and define your own plan. What you want to add what you want to measure.

A

Yeah, I think, yeah we have the last question um which kpi and non-functional specifications we should track while running qos experiment.

B

hmm That's a good question pro. I think it needs more context so which which chaos experiment is being run. um So, if you're, for example, running um a pot pot delete chaos, experiment, um you would be defining the response time of of the http server, which is running inside in the load balance setup. For example, I think the key performance indicator pool I'm not sure if I I'm giving the the right answer here. To be honest, uh maybe let's hop into slack and and try to figure out um what the context of the question is.

C

Yeah, we can definitely connect on slack and try to get more context about this question so run of sing if you're still in the audience. Please reach out to michael on slack he'll, be happy to answer your questions.

B

B

And also thanks for posting, the slides, yes, um they are available and um you can use them to learn async um you can you can ping me on on social on on twitter or linkedin or whatever you prefer, I'm happy to have a coffee chat or I'm also hosting or a meetup, which is on everyone can contribute.com.

B

There are some sessions linked in the slides where we are trying to learn technology together and sometimes fail and sometimes learn a lot. So yeah I'm happy to chat and help everyone get on board, with all the amazing technology, but slowly slowly, learning and and achieving success together.

A

Thank you. Thank you, michael and thank you all of you for this uh for joining the session, so you can always explore move on to the lobby and explore other talks as well. Yeah I'll be ending this session in a while.

C

Thank you michael. It was very insightful.

B

Thanks a lot thanks for inviting me and organizing all all of this um really appreciate the time we spent together- and I hope everyone stays safe- has a great rest of your day wherever you are in the world yeah and spread the word for observability love.

C

Sure, michael sure, I'm sure the audience has enjoyed a lot too. So we'll end this session, though.

A

Yeah bye, everybody.

B

C

Michael can stop the presentation.

C

C