GitLab GitLab Tech Talks, 15 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: OSAD 2021: From Monitoring to Observability: Left Shift your SLOs - Michael Friedrich

Description

Slides: https://osad-munich.org/wp-content/uploads/2021/10/Michael-Friedrich-Slides_OSAD2021.pdf

Security has shifted left in CI/CD pipelines. Tools and cultural changes enable teams to focus on their cloud-native journey. Traditional service monitoring moved on with metrics, logs and traces and observability embraces the unknown unknowns. Developers and SREs are instrumenting applications with distributed tracing. How do service level objectives (SLOs) add to the bigger picture?

This talk invites into a developer’s tale about ops deployment scalability, availability threshold definitions and measuring application performance. Estimated SLOs could be simulated early in the development process. New building blocks come to play: Quality gates and chaos engineering – is it possible to left shift the SLOs in your CI/CD pipelines?

Attendees go on a hands-on deep dive into cloud native resources with Keptn and Prometheus as quality gates, and explore ways of chaos engineering with Litmus. This includes best practices and tips to start in a cloud native deployment journey.

A

Right um yeah, so my name is michael. You can find me online um sdns dns, my chi. If you want to reach out after the talk or just have a question when I'm like going too fast, um I will be sharing the slides after the talk.

A

um But if you, if you're typing fast enough, you can actually see it right now. I want to dive into like my journey from monitoring to observability and how we or how I see left shifting our slos and in order to like get there. I want to start out with, like obviously storytelling around devops and devsecops, and one of the things I I've seen in the past few years was like security has shifted left already, so security is not optional anymore and we see each inner in ci, cd in pipelines and much requests in reviews.

A

um As a developer, I get immediate feedback on. There is a potential security problem. There are specific other things which I need to take care of, and it's not just like someone else is doing that and I might be fixing it at 3am in the morning um from there. um We also like turning back time.

A

A little bit in looking into what did we do for monitoring we had like the typical state, black box monitoring, then metrics came came aboard and at a certain point there was certain things like service service level, availability reporting, so the state changes over time and at a certain point, metric data points and trends, um and all this was quite nice to have, um but we also moved on in a way of saying: okay, we have a service level availability or an agreement of 99.5 percent, but actually we want to achieve 100 percent, but 99.9.

A

um This is our objective which we are working at um and we also need to figure out um the indicators service level indicators. Is it errors? Is it latency?

A

What is our error budget in that regard, um so this is currently like moving on and getting more into the uh getting more attention and we've also seen from from the google slr books and and other um inspirations.

A

We have golden signals which um provide us even more insights into what to look first at when something is broken, like latency traffic errors situation. um But at a certain point you really need code instrumentation for that, and we will have a look in into that in a bit. But the general idea is like there are some things which we already know about.

A

But how can we like shift that left or why should we actually do that and when I'm turning back time into, I would say five years or ten years, I totally would have wished for this idea around slos and having app instrumentation available, because when I started out like c plus plus, we once had uh created a rest api and also using json rpc in c plus plus, and we had certain things in problems in the architecture with cpu.

A

Overloadings threads were locking spin, locking other things, and we had this great idea of using co-routines in c plus plus, because we had seen it in golang and we're using like stackless coroutines, there's certain mechanism in place with putting on the function pointer onto the heap and stack unwinding for continuation, which is super complex and super hard to debug.

A

If it works, it's awesome, but apparently it didn't. So we had some crashes um in production, but only when, like 100 or a thousand api clients were connected, memory was maybe corrupted, maybe exhausted. Maybe we had leaks, we didn't really know about it and it was super hard to debug.

A

One other crash happened on windows with specific stack guards in memory, and it was only triggered by a natural security scanner, nothing else. So in that regard, it would have been helpful to like have certain slos or certain things. um How can I deploy that into a staging environment into a canary environment and ensure that the stack and heat memory location meets?

A

What what is defined so like looking into the service level indicator and saying um heap and stack memory usage level is, is the indicator for me that it either works or it shouldn't, and when it's okay, we can deliver it to the customer. We can release something and so on so defining the service level object objective in a post, modem would have been 10 or not more than ten percent increase um of the memory usage.

A

um We could potentially measure that um like stimulating a real-time production environment and also having, for example, chaos, engineering or fast testing for specific api requests in order to prevent that situation. But again it's five years, something like five years ago so would have been nice to have that. Another story is um something which is really slow, um and I I read an article a while ago, when someone tried to log into grand theft auto online and the problem was it took five minutes all the time.

A

It was seven seven minutes and um this user actually reverse engineered the source code by using an disassembler and other things, and found out um that there is a specific algorithm, sorting um configuration settings from a json file um which which is blocking or which is doing something which is not needed, um but from a little point of view he couldn't real or they couldn't really um see that, because the source code is not open source. So they started reverse engineering that and injecting um the binary with a dll overload and then optimizing the algorithm.

A

um But it would have been nice, like the ones who were developing gta online could detect that and like the mitigation would have been or could be, measuring the login time um using end-to-end testing um inside the application. You can also add timing points in the first iteration like using metrics or even going into tracing spans and from there. um We can also define like what is our service level objective in that regard.

A

So whenever this gets deployed into a staging environment from a ci cd pipeline, um defining the end-to-end test, scopes like the user, is doing the login. The user is starting to play gta actually, um and how long does this take so like when the login time is below two minutes? It would be totally okay, and there is a certain difference scenario when you have a low latency connection, so your bandwidth is not enough.

A

Maybe the login time should be below five minutes uh taking in it uh a different um measuring point into into account, um and whenever this is not matched. When we are failing the slo, um we should have certain like quality gates, something which prevents that from actually being merged to the main branch and then later on, being released and deployed into production.

A

Yeah, it's another story around this specific things or the specific ideas. Dns often has influences. Infrastructure is called infrastructure deployments and things you potentially can already control via github's workflows, infrastructures, code, workflows, um changing something in your git repository and then deploying it into production. Without testing and like last week, we had slack.com returning a surf fail, which was kind of unexpected, and it didn't work for some reason and the problem was well in bias. In my environment it doesn't work the google recursive results, it works.

A

Maybe, let's start debugging, um while others are debugging as well and nothing is working and in the end it turned out to be dnsec. It wasn't the nsx fault. In that regard.

A

It was just the wrong deployment of enabling the nsx with the dns key and the ds records being added to the upper.com zone, and then they were removed, but everything was cached and this turned out to be a a problem, because the dot com zone has a time to lift of 24 hours until the cache gets invalidated and a new request is started so like one had to use either use a different recursive.

A

Resolver or have faster cache invalidation, which happened to be uh at the google dns or at cloudflare and so on, um but in the end it was really like if you're not deep down in the technical layers, you don't understand what's going on and when someone says use google dns, it's like yeah, okay, it's interesting, but still no idea how to fix the problem um and well then I had the idea like: okay, maybe like have something around that or they potentially have it, but it slipped um have a staging environment for like changing something like this, which is invasive in a way of changing uh the security of trust in dns, um so applying again, git ci cd get ops monitoring, dns deployments from maybe five different global locations.

A

um The service level indicator is failure and the slo is like zero out of five, because it's really a critical service when it fails in staging potentially, should just fail and never hit production, um and when it goes into like when we're in production, we should potentially automatically roll back. But in this case it's like dns cache. You cannot really trick it. So you really need to detect this.

A

As early as possible, before it gets deployed anywhere, the thing is: a similar scenario happened back in 2011 when women, when I was part of the team maintaining dot at and the cctld got dnsec deployed um and everything about smoothly, and we had like a signing hardware with which is basically a state machine of specific steps until um the signed dns zone um is then output and then deployed, um and someone changed the script in that stage uh state machine on friday afternoon.

A

It was, I think it was persisted in kids somewhere, but nobody, nobody really knew about it, um and there was the the state machine machine stopped. We had no more signing, um we had no dns updates um and as for context, when you're registering a new domain, the in dot, jt or you're, updating, name servers or anything else. This is influenced by that. So there were no new new domains being delegated and so on, meaning to say: support had fun with customers calling and so on now did we have monitoring for that yeah?

A

Maybe um we had like the um latest monitoring or the monitoring at the right stage or shifting right. Basically, um so we had a zone serial to with a unix timestamp and we're measuring the serial and an offset um to like figure out whether the zone being deployed in production is old or not. The problem was friday, um didn't really work for triggering the monitoring it was actually saturday at 3 am per email.

A

Then it was escalated to via sms, and we had. I don't know how many names it was 25 30 something around that. So it was flooding uh our mobile phones and when everyone is awake, debugging at 5 am is not really fun, um so we forgot well. The change was persisted in like a repository and then rolled into production.

A

Maybe we can solve something around that um and like even if designing hardware is not cheap, um it would have maybe been nice to have a staging signing hardware.

A

um Having this the changes being rolled with infrastructures, code or github's workflows, um we would have defined the service level indicator, which is the zone serial h or the difference between stand now and to zero, and then the service level objectives should define that it shouldn't be older than one hour and if there's, if there is something failing in that process before hitting production, um we potentially would have never rode to change into production and um creating some failures or some other dish outages for customers.

A

um Now, dns didn't really stop because uh two days ago facebook was down um from a different angle because speech the pgp announcements uh for the routing were just gone, um so the uh the as was not announced anymore and the request of resolvers had a lot of fun because everything failed for facebook.com, no roots, no dns, query responses, no caching, actually, and so the clients. We tried every second every five seconds and the recursive resolve for providers. At least the larger ones had really fun, with cpu load and burning resources for nothing.

A

Now um there was a an update um or more insights from how facebook is building this infrastructure. um They kind of um they're managing their own data centers. They develop their own bgp agent to being able to scale with the many requests with the huge infrastructure they are maintaining and the policies are being tested and deployed, including the agent with continuous delivery in each data center, and this should basically be working. Fine.

A

um The problem was in that regard um that this, like policy audit, um didn't really prevent um the change, so there was a command being sent out. At least this is what was written in the blog post from yesterday, where it says well, the uh there was a policy change being pushed and there should have been an audit log rule preventing that which is not a status but was was wrongly typed in an update, um but it didn't fail for some reason, because there was a bug in the audit so like thinking about.

A

How can we like maybe solve that problem? I'm not a facebook sre, but just thinking out loud well, defining a stronger ruleset for, like the policy audit disallowing. These pgp policy changes um looking at the service level indicator like the failing audit and the service level, objectives should be measuring.

A

Well, how many failed policy policy pushes have been there, for example, so when a weed-only policy deployment is allowed for, for example, and something which breaks effectively, the entire routing shouldn't be allowed and this should be detected, and if the slo is above a specific count, um we should actually fail and stop the canary deployment and not hit production.

A

The other thing which I learned from that is that each data center or the the the name servers are detect, detecting um when the data center is not reachable and then shutting down the bgp routing by themselves, which might need some different service level objectives in the way of how they detect that and data center is unreachable. Is it like the the ping response time or is it an http response time or something else in that regard?

A

So these are quite a lot of ideas and examples how it's could be like related to dns and infrastructure changes and one last story um which kind of hit this last year, when docker hub included uh red, limiting or introduced rate limiting for uh pulling docker images from docker hub.

A

um We were not sure how to like what does that mean for us, and possibly our ci ctd pipelines are affected because they're running in darker containers, we might be um having problems with cloud native deployments in kubernetes and the like um organizations which are behind or not and using a single ip address could could have been affected by that and cloud providers in general, while you're just pulling containers and stuff where you need that and likely in in other environments, where you don't expect it.

A

So um the thing is we have this like known state, we know that the limits are applied.

A

We didn't really know where the values are, so we actually needed the first mitigation to simulate a pull. Then we had a header response. We had a script to pass and last year also created a premises exporter for that to actually being able to monitor that and have a nice dashboard.

A

For example, in grafana the simulation was later on changed in the head request, which was not affecting the remaining count, but still it was a change um where you need to add more monitoring to to that and also ensure that you're not deploying something, because the deployment might just fail, because um the limit has been reached.

A

So when you, for example, imagine that you're deploying um your application with replica sets in your kubernetes cluster and um three uh uh three ports or three uh containers are coming up, um but the rest is not and then the actual update you you had pushed out doesn't reach um the production website and the prices for customers differ depending on where they're coming from uh which deployment they are reaching just because you couldn't pull any updated container images.

A

So um this is kind of an unknown state, and this makes me kind of a little like it's. Not it's not a safe feeling um to say, okay, how can we monitor that? um Is that a problem and like so I we sat down and said: okay, how what could be like the service level indicator? Yeah the pull counts, um the remaining and the limit, um and the service level objective in that regard should be like the remaining uh pull should be below 10..

A

This is like an arbitrary number um and should be really defined on how much you have. I think the default is 200 per 6 hours or something or you're getting a pro subscription or an enterprise account. Then it's more, but still you need to monitor it and also prevent um deployments when there is no limit available.

A

So like the quality gate or like the gate should be when the slo is failing um for a specific test, we shouldn't start a deployment which again pulls the images so as a developer.

A

I also need to know that, because, when ci cd is not working, um it could be everything it could be my code, it could be that the deployment isn't working, that the docker pull is not working, the infrastructure is broken and in in in the main problem, or the main thing would be, the reviews are being blocked because we cannot review the apps and so on. um There are still mitigation ideas like using a local container registry and a dependency proxy, but yeah. We.

A

We need to have service level objectives and service level indicators, so as a developer. How? What does that mean and specifically in like the devsecops pipeline? um But do we start with that?

A

And one of the ideas is like for monitoring use, prometheus, um have a promisius server and promises export export the subway label, not only for like um the host monitoring or the kubernetes monitoring with the premises operator, but also the availability to export application, metrics rather easily with um in instrumenting applications, um and this is like how the promises architecture can look like it needs the knowledge, um the promisius query, language from ql, which can be used for creating or calculating the slos so like having that in mind using a cheat sheet and as a developer.

A

I also need to know like where do I start now, because when someone says well, can you please add metrics to your application, because you we want to measure the http response time and the uh the backend queries and so on? um Well, slo turns into like monitoring metrics. We might have keys and tags and the values, but how do we, like fetch the values of what do we actually start with? um Promises, has great documentation and also great examples and there's also training available from julius watts.

A

If you want to go there um and and get hands-on uh training in that regard, um the main idea is that you can do infrastructure monitoring like memory cpu disk io, something else um directly on the node on the parts in in the cluster in the kubernetes cluster. um You could, for example, monitor the services promises, exporters, monitoring, website responses, the black boxes, black box, exporter, snmp, whatever comes to mind and, last but not least, the instrumentation of your apps, which is like the development code instrumentation requirement, um and it's um when you're.

A

Starting with that, I I recommend pick choose the language you're, the most familiar with or which is the most fun for you to to get going, um and hopefully it's the same as the application. You need to implement it on and start playing a little bit because when, when you have the production environment to say hey, I need to add that now to my app it might be blocking.

A

So I started playing around with creating a fun demo for prometheus with python, like adding a adding some random counts and just seeing something like the under five minute.

A

Success feeling um later writing on uh later or in that specific moment like building a dockerfile, um pushing it to the container registry and also seeing it in deployment so, for example, deploying it into kubernetes, with the promises operator using a service, monitor crd and inspecting the metrics, then with promises and graffana in the kubernetes cluster and then seeing how we can evolve further and add more metrics to the application actually.

A

And then there is a problem like when you shift also from monitoring to observability there's a little more than just metrics. We have like metrics logs and traces, and then someone says well profiling is is also important for application performance monitoring.

A

We know that like from shifting from monolith to to microservices, but it can be quite overwhelming.

A

So my recommendation is to really focus on adding metrics to your application at first and then think about better logging, adding tracing, adding profiling and other things to your application to just use the minimal, viable change or the smaller iterations, and not to like try to solve every problem um yesterday now speaking of logs, these are just some ideas, some examples how you can like which tools could be used um where you can look, have a look at just as an asynchronous exercise to maybe haven't, have an idea, look into elasticsearch or loki um prom, tail, fluentd and so on, and um the other thing around traces get to know how it works um like a little bit different to logs.

A

Understand why it's important for you um yeah, and when you look when, when you want to look into profiling, um maybe continuous profiling application performance insights, discussing what are the three pillars of observability or we are discussing the four pillars of observability.

A

But the main idea is just let's just focus on metrics um and see how it fits with our slos. So um slo's with prometheus is what, where we want to go so the metric and the lot calculation happens in the prom curve format. So the prompt queries are defining what we actually want to measure. What we want to see and then there is this new project like it's.

A

I think it's one yield around abolish named captain, which provides us with the possibility to create a quality gate, to measure the slos or to queried slos from promisius and then to to decide whether the slo failed or not, and when the slo failed. The cicd pipeline is broken and you're not deploying to production.

A

um How this like works in general or how this can work. I would totally recommend trying out captain by yourself with either you install it using the available tutorials and just follow the quality gate scenario: the quality gate environment, where you're just deploying the application, you're executing some tests, you trigger the quality gate, and then it gets evaluated, curing the slos and, depending on the outcome, depending on the result, you continue playing around and later on.

A

You integrate it into ci cd and all the magic workflows how it basically looks like when you dive a little deeper into defining the slos.

A

You basically use this a similar format to the promises from curious, and um there are ideas around like supporting open slo as a specification and so on. um So, like captain allows you to uh measure these slos and then define in the in the right example. Everything uh is green, but yeah there is an online gameboy available as well are some things which are which are failed with a sock shop demo application, um but you can should totally um try it out in your environment.

A

The other thing around shifting left with quality gates and slos um in continuous delivery is like. Okay, we have captain as a quality gate. We have prometheus for slos, um but what? What about the application? It will always run, as is there might be no influences like simulating production is hard and one of the ideas is to add chaos to your deployments. So something breaks, but you don't expect it or the application doesn't expect it, and litmus chaos is one of the tools or one of the examples.

A

One can use to let pods fail in kubernetes, but also let influence different other things in your infrastructure.

A

You can see how the application behaves you can see if this dslos are still matching in captain impromisius and then define actions and improvements.

A

The documentation and the the quick start is amazing, so I think it took me 10 minutes to really get to get going. I would totally recommend that you start playing and edit to your production environments and also, of course, cicd around slos. There are more ideas to automate everything to generate slos.

A

There are specific things: splunk can do for gitlab.com sas. We do have some our engineers and our series generating the slos also generating these dashboards um and, of course the slis are also involved and basically you're trying to not make do manual work manual. Work could need automation, so there are some inspirations on there as well.

A

The other thing which is interesting around slos when you're treated as code. I've mentioned the openslo specification which is currently being worked on. There are some. There is an slo generator named sloth, and this uh slo management tool named pura or pyra cannot pronounce it is, is really new. I think the the the first release was last week can be used as a web uh based slo management.

A

um Now we have many possibilities for shifting left. um The the main thing is as a developer, I'm getting the feedback, so I can learn from that, hopefully or maybe post modem. um Nobody introduces memory corruption because of a wrong co-routine implementation. Login algorithms, can be reworked, maybe hopefully in the future.

A

Infrastructure is code, changes, dns changes, policy, changes.

A

Docker hub rate limiting can be just treated as code and we can add code quality checks actually from the lessons learned. The other thing is slos are not like standalone. We need to like correlate that with everything which we already have with instrumentation observability, um and they also need to be part of the devsecops workflow. So whenever the slo, for example, is failing, um the merch request should give me an indication or something like that and um with having so many different resources available.

A

We also have the possibility to do auto scaling to add chaos and so on. So there are many possibilities. The thing is: don't try to solve every problem now with that. It's basically um also when you're, adding application, insights, you're, adding metrics to your application, you're, helping the series you're, helping the ops teams or the devops engineers.

A

So they can know. Is it your code or is it the code which is failing or is it something else? Is it the application? Is it the infrastructure, the more metrics, the more insights we provide, the faster the solutions are and yeah. I would recommend start using borrowing solutions at matrix, endpoint um play around with prometheus, and so last but not least, don't do everything yourself be a team and practice that and document everything um running out of time.

A

I have, I still have a wish list: um slow, sec, ops, maybe no just kidding um more insights into ci, cd tracing and maybe machine learning something around that and also learning how we can even more improve the storyline more around shifting left slos.

A

Okay, um this is basically a recap. um You know what I'm talking about. Try everything. Let me know um happy to have a coffee chat around this um check out the resources which are linked on the slides, which I will be sharing in a bit and yeah. Let me know how it goes if there are any questions and ping connect on social thanks for your attention.