GitLab Developer Evangelism Team, 6 Jul 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Dev Loves CI/CD: Efficient Sec and Ops Pipelines - Michael Friedrich, GitLab

Description

Dev Loves CI/CD: Efficient Sec and Ops Pipelines - Michael Friedrich, GitLab

Continuous integration and delivery/deployment helps speed up development and review workflows. Developers can focus on code reliably tested in different environments. At some point, the operations team gets paged on broken pipelines and jobs being stuck. On top, the security audit unveiled plain text secret exposure and dependency exploits. The next horror story: The software cannot be deployed anymore because package dependencies are broken. In this talk we hear stories on making CI/CD pipelines more reliable and secure. Automated deployments and package/container repositories can help avoid redundant cycles and extra work hours. Monitoring/observability combined with automation ensures to sleep in busy on-call times. Learn how Dev meets Sec and Ops in the pipelines and hear best practices on efficiency, iteration and insights.

For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/

A

Hello: everyone, my name, is bartley, and today I want to talk about devslav, ci, cd, efficient, second ops pipelines um and before we dive into it, um I'm a developer eventually listed gitlab.

A

You can find me online as dns michi, which is dns, mi, chi and michi is the lovely form of martin german and without further ado. um I want to step into what's coming in the next round about 20 minutes, um so we are looking into ci cd and how this relates to dev. Second ops, then learning how to dive into a pipeline being deep down, defining team actions looking a little bit into security and then into automation and iteration on devops.

A

Now, when we start building things um and we add ci cd to our workflows, it's oftentimes. um Why? Why do I want to do that, and sometimes you have an idea and you see a competitor having the same idea and how fast can you deliver? How fast can you release software so in in the end? It's it's a matter of efficiency and speed um and releasing faster than everyone else um and bringing something to the market which enables you to benefit from that.

A

That being said, starting with the icd is oftentimes a long process um and you're adding it. Everyone is adding it and then sometimes there's lots of uh trial and error involved on the right hand, side there's an example with gitlab, but you could totally map it into a tool of your of your choice.

A

um The thing is, we want to overcome this and there are other problems which which you can encounter and say: hey the public runtime is way too long, so I'm not getting any feedback and the reviews are being blocked, because I need to wait 10 minutes or something like that and on the right hand, side. Well, there's a there's an example for that which I saw lately and said well, a typo was it was actually the the root cause for um like six minutes, five minutes delay and you actually need to fix that.

A

So as a developer, I need to become ops at a certain point, and someone says: please fix the ci cd pipeline. Why yeah, because they consume so many resources and then I'm like how yeah with the dogs, hopefully someone in the community can help. um Maybe someone wrote talks already or something like that. Then this is the thing you're asking yourself: well, what's actually inside apartment, what's there how? How can I like work with that and oftentimes? um It's uh identifying the the purple and runtime duration on its own and seeing what?

A

What is what is the critical path?

A

What is the minimum and the maximum pipeline duration identify the number of steps like stages and jobs or whatever it is called in your workflows, identify how long they are taking and which one could be a blocker like all all jobs are taking five seconds, but there is one with seven minutes for some reason.

A

um Another thing to look into is when you have, when you have uh sync runs: there might be dependencies in the middle blocking it like when you're manufacturing a car and the left door is like not being pulled into. You have a problem, um then another thing to identify is a pattern with failed tests. So it's not just this one pipeline which failed it's a trend over time, so you might have flaky unit tests like sometimes they're.

A

Okay, sometimes they are not okay, depending on the uh the day, the date and time or um specific influences from a network outage or something else. um Another another indicator is when the test curve, which drops for some reason- and it goes down down down, so you really want to dig into that. um There is a lot more to that and saying hey.

A

My environment is built upon node.js or c plus plus, and then I have built dependencies and they need to be installed every time the cicd pipeline is being run. You need to work with containers. Probably there is a certain network, latency involved and at a certain point. Security might also be preventing something or there is like a problem within your pipeline.

A

So these are all things you need to be aware of, and the first thing I would recommend to start looking at is like, if you have a certain cicd analytics overview dashboard, something like which tells you about the pipeline duration about failure and success allows you to see some beautiful graphs, um whether to identify there is a pattern. So like is the pipeline duration? Seven minutes all the time or is it just like eight a.m in the morning? And then everything is fine and then you can correlate it and ask.

A

For example, the the sres um or the ops team and say hey? Is there something in the grafana graphs and you should probably ignore spikes and anomalies, because they don't tell you anything so really look for specific patterns which give you an insight which job or where to look next, then in case you have unit tests and I hope you have them um look into the ci step and say: hey is there something which can provide me further insights?

A

Is this like an ongoing problem which is like moved from the job to the next shop and so on? um You can also generate the test coverage and to in order to monitor that and, for example, with gitlab. You can use a junit integration. There are other tools out there, so it's not like written down in stone, but still it's a good way to get more insights into the plan and into the actual workflows.

A

The other thing I would recommend to you is like explore. The visual visualization layer, so looking at the logs and analyzing job duration, can be like yeah.

A

You need some sort of like visualization and seeing the critical path, meaning to say um there could be a different representation of the pipeline, um because the um the image at the at the um at the top just shows the stages and the jobs, but if there are asynchronous dependencies inside so meaning to say the former job triggers the unit tests triggers the the compile uh job and then the run. This is like the critical path in the pipeline so and everything else like generating a test report is not so important.

A

um Seeing identifying bottlenecks can also be helpful, visualization, but also needs the right, visualization layer and not just like saying hey. I really have a hard time looking at it and I cannot really figure it out now. This is like the high level and we want to dive a little more deep down into a pipeline what's actually going on and which um things we potentially could influence.

A

um So how can I get more efficient in the pipeline? I need to check the workloads, um probably so how many compute resources are being uh used in the pipeline in the specific jobs. um Are there any specific bottlenecks? I can see really like from just looking at duration.

A

I also want to look into the resources being assigned to it like draining the the cpu or using lots of memory could be a factor you you want to dive into, especially when it's tied to a cloud compute costs in in that regard, and you totally want to really see the global picture.

A

So if something else is influencing the pipeline runtime or like the resources being drained for some reason, because there is a deadlock in the software and you didn't really discover it yet, because there's no runtime monitoring when you have found the flow or you have understood the flow discuss the actions with your team, like refactor the pipelines.

A

Basically, everything is yummy. Today you can treat the pipeline as cold and start refactoring the cold. You also want to reduce the cost and also reduce the computing powers needed, especially because compute powers are not for free. You need to pay for them. So in this, in a specific environment or scenario, um you could potentially save a lot of money by reducing it. Now, how can I visualize that? Well, there are probably um things within the tools that you can use, um but from a monitoring or observability perspective.

A

It also makes sense to approach it from the outside.

A

So given um I want to, for example, monitor gitlab ci cd pipeline health, and I want to still use my existing monitoring environment, which is based on promises and grafana.

A

I can, for example, use a promisius exporter which queries the rest api and then collects all the data and craft has nice beautiful dashboards. For that reason, um if I would would be using subix nagios, I think a sensor whatever is out there as a monitoring tool. I could use a check plugin to integrate that and there's potentially even more than that, so it's not really bound to the tools.

A

The thing is, you want to have something visualized in your operations dashboard and immediately see when the pipeline duration is going off and even being alerted, and, last but not least, seeing trends and spikes. The thing is the pipeline on its own is like a black box to monitor. You also want to dive deeper and say: hey. I want to monitor the storage consumption like disk usage, um maybe expiring the artifacts. If there are some for the jobs, the container registry images normally consume lots of space depending on how how big and small they are.

A

This uh created network traffic comes into play, so the cic ci cd jobs, installs software on every run. This is being pulled similar to huge docker images being pulled. This generates lots of network traffic.

A

um The other thing is, if you're having an external ci cd runner, which is running in uac 2 instance, for example, disk I o cpu usage, basic linux windows, mac, os host monitoring and, last but not least, everything the application itself consumes can be monitored in a certain way of saying, hey. I want to integrate the apm metrics collected in the ci cd jobs and have them integrated into the much requester pull regress widgets.

A

So, there's a lot lots of things to unpack and to to learn from it's important to measure them to get a better overview of. What's going on and inside a merchant request or a pull request, we also want to have fast feedback, so we want to see the test coverage as a visual visualization layer. We want to see metrics and unit test reports and code quality and accessibility, testing and security testing.

A

Basically, everything we have already provided or at hand or which can be integrated should be visible, and when someone is working um in a future branch or in a branch technically, they should get feedback immediately and not merge something which is either broken in a security way or an insecure way, or also by preventing to publish something, but the performance would be five times slower than before, just because of a problem in the algorithm detected by the application performance monitor now, you can also use uh batches in the project making it like, allowing it for faster detection of the current pipeline status of the test coverage reports and so on.

A

So there's a couple of ways um in the readme md or directly the project after you have defined or identified the bottlenecks. You also wanted to define the team actions, which means one of the patterns to go. There is say, hey I want to fail fast because in that example, the format and the unit test jobs already failed, because this there was a typo in the code, but the code quality check is still running because it runs all the time in the same stage.

A

Now the solution to that is to say, I want to fail fast and to form a job grunts before anything else is being triggered and when it fails, the pipeline fails and we are good to go, not good to go, but we are saving resources and get immediate feedback.

A

The other thing is, we want to optimize the ci cd infrastructure, where everything is being run. So when docker pull takes too long or is rate limited since last year, we probably want to use our own container registry or use a specific container registry available, gitlab github whatever is being being in use, and we also probably want to optimize the ci chops, which always install the dependencies and create custom.

A

Docker images which have the build tools available by default, and another thing to keep in mind is whenever things are being pulled or installed, keeping cache layers for node modules and everything else like reducing the overall cost and time where ci cd jobs are being run.

A

Now, as a team action, we also, we also need to learn how to fix pipelines fast. So it's it shouldn't be the case that only a single person is is the right, a responsible person for fixing the pipelines.

A

We all should be able to do it on our team, so everyone should understand the failures quickly and the cicd job output needs to use the key messages, something which is easy to follow and say hey in this line. You you made a mistake and there is an error and not a stack trace over two pages.

A

We can also use colored error messages on the terminal and also use emojis as a unicode or utf-8 encoded character.

A

We can also link things in there, like troubleshooting, documentation or documentation how the pipeline is designed, since we are probably using git repositories all the time. We can also document the pipeline design in there as a markdown format or something else an example could be using mermaid, which is available, at least in the gitlab markdown, but potentially draw I o or something else is also possible, and whenever you fix something document the way you learned how to fix it.

A

Sometimes the google terms you have been using the urls. You have been reading the blog posts which brought you to the solution, because in the end, half a year later on, someone else can benefit or you, since you run into the same problem, and you don't need to research two days to fix something now coming a little bit into security.

A

Like can we release um the source code to our customers to contract us to the public domain? So, okay, let's just do it um and then we are leaking the aws credentials from the test. Suite need to roll over everything revoke something and it's 3am in the morning and everyone is running into burnout.

A

This is something we also want to prevent when looking into efficiency, icd pipelines. um The other thing is credentials can also be like in the ci cd config. So the mysql password is like has root access, and this is actually the production, environment and yeah. It's not. It's not really a good idea to leak it on the internet. Still again now it involves devsecops monitoring is red, everyone is busy and humans make mistakes that. How can we prevent them?

A

The thing is, there are certain ways- and I mentioned that before we can have the feedback immediately from security scanning and merge requests and pull requests and in order to solve the runtime requirement for the production mysql database. For example, we have the possibility to leverage things like hashicorp vault, where we just get a secret um as a token which is limited in time and then everything works, and this is not only tied to ci cd but actually to production usage as well.

A

Now we also want to ship things faster. This is like the cd part of everything. So developers want to deploy everything fast, like building things automatically, installing all the dependencies testing things tracing things scanning fixing, suggesting deploying merging yeah, maybe merging I'm not sure about this. So one of the ideas we had, we want to do that with cloud native um in a five minute production app. This is something uh which is currently um in development.

A

um On the other side, we also want to look into what's already there so fast versus resources. We have the possibility to use some sort of auto scaling and saying: hey. Nine to five developers need the fast builds, but on the weekend there are not so many resources assigned to our ci cd pipelines, reducing the compute cost being acquired all the time by virtual machines being run. um There are specific environments which can be used like the virtual machines with docker machine.

A

There are the hyper clouds like aws gcp azure and their apis openstack, and probably I've missed lots of many other managed clouds for kubernetes. We can use the runners and look into node affinity.

A

Now are developers still in love with ci cd yeah, probably because lots of things have changed, the ux is getting more interactive. We have quick actions, um people start using or creating a ci cd pipeline editorial.

A

The learning curve is getting more easy and we're also getting fast feedback, because we optimize a lot and many of the things I talked about today are already within the pipelines or they um they are available. Hopefully soon um or people have started applying them now. Another thing is, we can automate so many things.

A

So if we want to do config linking we have api endpoints, we have security left shifted, we have ci cd templates, which everyone can use, and last but not least, we also have quality gates in our continuous delivery process, so meaning to say we are defining our slos, our service level objectives and saying. Well, if that is not met and the memory consumption increases, we're not letting the deployment go on and in in addition to that, we also add application performance, monitoring and more observability to our pipelines.

A

So as a developer, I can really benefit from that, and I think everyone is still enough with ci cd, because we have our grafana dashboards for ops. We have secured the dashboards for sec. We have all the details in the mrs and pr's for our devs um and everyone has got more pipeline insights and this is not bound to gitlab. On the right hand, side. I've also seen that with github with circle ci with many other different toolings which are available- and I think everyone is inspires everyone to make it easier for developers.

A

So with that being said, um I will share the slides and the recording, of course later on. um Thanks for your attention and I'm looking forward to your questions now,.