Cloud Native Computing Foundation GitOpsCon NA 2022, 28 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk: Green(Ing) CI/CD: A Sustainability Journey with GitOps - Niki Manoledaki, Weaveworks

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Lightning Talk: Green(Ing) CI/CD: A Sustainability Journey with GitOps - Niki Manoledaki, Weaveworks

Our infrastructure needs are increasingly energy and carbon intensive. CI/CD is one area where we can take steps to measure and reduce our footprint. In this talk, we present our investigation into instrumenting CI/CD systems and GitOps tools to achieve this. We share the outcomes and lessons learned from these experiments so far. Our journey begins with traditional CI/CD where the two are tightly coupled. Transitioning to GitOps often starts with decoupling the two. This is an opportunity to measure the energy consumption of each step and think about environmental impact from the very beginning. Energy use can be measured before and after this decoupling, and we can show you how. On the next stop in our sustainability journey, we evaluate how GitOps tools and patterns can be used to reduce energy consumption and wasted resources. Expressing a system declaratively offers full visibility of the tools running in your clusters. Another promise of GitOps is that it can be used to turn IT off when not needed. GitOps can also support tools and policies to measure and optimize energy and carbon usage. Our journey ends with some reflections on methodology, outcomes, and next steps.

A

So, thank you for joining me today to talk about cicd and a sustainability Journey with get UPS. So have you ever wondered? What is the energy consumption of the cloud native tools that you're using or building? Actually the um International Energy agency reported in September 2022 just last month that data centers account for 1 to 1.5 percent of global electricity? Use? That's a lot today.

A

I will present tools and techniques using git Ops to measure the energy consumption to measure energy consumption of cloud architectures, but also get UPS based architectures in this case, we'll be looking at a at Flex and I'll also discuss how githubs can help with optimizations for carbon and energy, um carbon emissions and energy use so or sustainability.

A

Journey starts with traditional CI, where CI and CD are tightly coupled and where, when tests pass we deploy or workload, it would be great to be able to gather energy metrics about CI so that we can compare the pre-gitups architecture to a GitHub space architecture. However, looking at, for example, GitHub actions, it's extremely difficult, if not impossible, right now to gather metrics, um oh yeah, thank you, yeah I'll. Do that.

A

um So it's very difficult because GitHub actions, Runners, are ec2 instances which are VMS running on AWS infrastructure and we don't have access to granular, real-time energy data or carbon data without actual data. Any attempt on making any optimization would be a shot in the dark, so Q Kepler, which is a new ebpf based tool for Gathering energy metrics about any kubernetes resource using Prometheus.

A

First, we ask ourselves what data do we want to gather and flux in this case is essentially for controllers. In the flux system, namespace, the four controllers are their Source controller, the customized controller, the notifications controller and the Helm controller.

A

So, with this Prometheus query that you see on the screen, we are getting the sum data of the namespace of the Pod, the pods running in this namespace over 24 hours, um and so we're able to get this result in meleejoules, and we can then convert this to watts and kilowatts per hours and gets data about pods namespace, whole namespace, a whole node.

A

We can compare this, the Pod visualization in the node Etc, and so you can scan this barcode um where I've created a repo with some scripts, some instructions on how to set up your environments, that is ebpf base. It exposes kernel headers and you can then run Kepler and Prometheus on it and flux to reproduce this data.

A

The really cool thing is, we can take the Milli jewels and compare them. Convert them to watts, get the idle kind of energy consumption, of the idle flux controllers over 24 hours. And now you can see that this is nearly 200 Watts.

A

We can see it's nearly 200 Watts um over 24 hours and that's the equivalent of nearly 10 smartphones being charged according to data from the U.S Environmental Protection Agency, which has a calculator for measure for converting watts to relatable metrics, and so now this might seem like a lot right, 10 smartphones in 24 hours, but that's actually just four pods we're just not used to seeing energy metrics about kubernetes resources, they're, not something that we can easily relate to, because we didn't have the data until now.

A

um So it might seem like a lot they're, just four pods! um Imagine what else we can do with this metrics.

A

um If we have this number and we can use apis, such as what time, API or electricity maps to get the marginal carbon emissions index of a specific grid- and we can convert this number to the carbon emissions like kilos of carbon emissions produced in a region like by the by the kubernetes resources that we're running.

A

So that's pretty amazing.

A

And what else can we do with get UPS? We can leverage one of the greatest things about get out switches, the declarative aspects which was intended for Disaster Recovery or turning I.T off and on, and once we have monitoring in place. We have the data we can move on to optimizations. We can use metrics and get ups with scheduling and policies, and so, with this stack, we can do a lot of things related to carbon and energy optimizations.

A

Githubs also facilitates green tools.

A

um We have a lot of tools that are that we're discussing the environmental sustainability tag which was just created. Please join our meetings if you're interested in continuing the conversation, so we have energy metrics and carbon metrics and then we can do scheduling. We can do scheduling with Carpenter. We can do scheduling with Keda Nomad, with the new Intel kubernetes power manager.

A

And the next steps is to use these metrics to compare and optimize architectures running on kubernetes, and we also need to make Kepler available or in more environments, because, right now, it's limited by an environment that has kernel headers. These have to be. This can be read only for anyone who has security considerations in mind, which should be everyone, and um so yes, that's something.

A

We really need to do to make this data available everywhere and if you're, if this is of Interest, we're also forming a subgroup in the get Ops working group of open gitups to talk about environmental sustainability, and we hope that this format, this kind of spaces, can be created all over the cncf ecosystem and you can join our first meeting on the 15th of November.

A

um That's uh that's all I had um yeah. Thank you for, for this.