Continuous Delivery Foundation cdCon 2021, 9 Jul 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Spinnaker Workshop Part Two - Spinnaker Kubernetes Operator - Alfonso Andrade & Todd Underwood

Description

Spinnaker Workshop Part Two - Spinnaker Kubernetes Operator - Alfonso Andrade & Todd Underwood, Armory

Join industry experts who will lead talks focusing on several key and core items that will give you an overview of the breadth of the power of the Spinnaker platform. In this interactive workshop, you will learn about:

How the Operator is being donated to the community
How the Operator makes deploying and managing the full lifecycle of your Spinnaker app simple, automated, and reliable, leveraging a Kubernetes-native GitOps workflow.
The steps you need to get started on your own instance of Operator.

For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/

A

Guys glad to have you here so in this talk, we're going to talk about migrating the spinnaker operator to and why we're going to do it, as you can see in the image we're a talented group of engineers that we work in the platform ux team in armory. So we are todd underwood, he's our technical, lead and senior software engineer.

A

We work with christian castaneda. That is our senior software engineer, fernando freire, that is our principal software engineer and my full name yeah. That's my full name, alfonso alejandros, pinocellos montero and I'm a frontend engineer learning how to do this.

A

I'm going to give you more context about armory, so we were funded in 2016 to unlock innovation by making software delivery. Continuous collaborative scalable and safe, so armory works hand by hand with spinnaker score. So in the next slide, we're going to talk about more about operator.

B

Hello guys this is christian and I will talk about what an operator is and how it relates to a spinnaker. So operators, in a nutshell, are software. Extensions to humans makes use of resources for cfds to manage locations and their components and.

B

B

Currently separated patterns, and we have created a spinnaker service crv that basically contains all the configurations that spinnaker needs the spinnaker service. Crv is an exact representation of what a how config is.

B

You only need to to do some adaptations to the configurations and you will be able to to deploy a spinnaker instance.

B

The project is great in goal and we are using the operator sdk project to create their custom resource definition.

A

So we're going to do a live demo about how to install spinnaker operator in your local machine for the purpose of this demo, I already created a cluster with kind and created and downloaded the latest operator in my local machine. So from now, if you see I'm going to start copying, these commands from our armory docks that I already have the resources page in the last of our presentation.

A

So also I have an alias in my computer. Instead of typing cube, ctl, I'm going to type kc and then I'm going to pass the context that I already created up here so for now, I'm gonna do this.

A

And apply crds good, so now they are created once the crds are created. We need to create the spinnaker operator namespace and then I'm just going to copy this.

A

And then create operator namespace, so everything's successful, so we're in a good track. Here now I'm going to install operator deploying the cluster again sorry for the long typing here, but it is what it is now from here. We have created the cluster, so now we can get the bots from this namespace that we already created.

A

Let me just delete this guy and then we are waiting only for the containers to be created. uh If you want, if you already have watch install, we can see what's happening here.

A

It's running once this finish: we have to create a spinnaker namespace to enable to have a different space from operator and our spinnaker cluster now once or spinach operator is ready. We can create or name space for spinnaker itself, so phc dash add context demo test.

A

I don't know what is this create namespace spinnaker and then we're going to call it a spinnaker good. It's created now from now on, we're going to go to armory github and then we're gonna find these spinnaker patches, so we're ne, where we can't just launch spinnaker in our local machines, so uh we're gonna attach this link, of course, at the end, and then, if you can see here, we have two api versions, the armory itself and the oss itself. So I'm going to show you where we need to change this behavior in code.

A

So now from now we're going to jump here. If you go to the spinnaker service channel, you see that here that I already did the changes also we're pulling the latest version in spinnaker itself. So if you're curious about those versions, you can just jump here and see the change. Look.

A

Then we're going to find all the armor dora armory dorados and replace it with a spinnaker, as you can see here, also in a spin a there is one and in patch menu. There is another one. If you're curious, where I got this one, this things uh where I found it, you can go to customization general and then you have to check all these files from here to make sure it will work.

A

So now we're going to good track. We I have to change the context in my local machine to just run the secrets we require to run the spinnaker. So let me just change the context and then now we have this useful batch here to create the secret. So you have to be positioning right to find this folder. So now we're going to create the secrets that we need to launch spinnaker in the local machine that we need here.

A

So we have this useful bash in create secrets, so you have to run it and passing the namespace you we previously created once our secrets are done.

A

We are going to apply these changes with customize. So now we're here in our customized patches and we're going to apply the changes to the namespace we already created.

A

Now that they're all created, we can just watch the bots to see what's happening, can just watch.

A

Change the namespace here and then let's watch the pods to see what's happening. This operation might take some minutes depending on your local setup, so we can see what's happening here.

A

After having to wait four times on my side, all the services are up and running and healthy. So now what we're going to do is just board forward gate and deck. That is all we need to see a spinnaker life. So when we port forward gate, don't forget about pushing the right port that is 8084, and now we're going to do that. That is on port 9000 and if everything goes well, we can see spinnaker running up and running in our machine.

A

A

And now you can you have your own spinnaker poster open running. Thank you guys.

C

Thank you alejandro for that demo, hi everyone. My name is todd underwood, I'm the technical lead for the platform user experience team here at armory and we're the team that primarily maintains the spinnaker operator.

C

So why are we here? Why are we talking about this? The long story short is that in january of 2020, the spinnaker technical oversight committee accepted rfc 89, which calls for the replacement of hired as the recommended solution for deploying spinnaker.

C

The spinnaker operator relies on howard running in a side car in order to generate configurations and manifests that deploy the platform. So we need to see about removing halyard and replacing it with something.

C

What's the solution, the answer is clean. Cleats, a lightweight library written in go that can generate spinnaker configuration files from hal configs, unlike halyard, it can be embedded directly into the operator so that we don't have to have it running in a sidecar which is going to make maintaining the operator easier for for teams like us and it's going to make debugging issues easier for teams that are you that use it.

C

In short, we need to refactor the operator to replace howard with cleat. So how are we going to do that? To answer that? Let's look at what the current implementation looks like.

C

Currently, if you have the operator running in your cluster, you give a spinnaker service manifest you apply that to your cluster and we go through a number of steps. The first thing that happens is the operator attempts to validate your your configuration.

C

Some of that is done natively in the operator, but other things are done in halyard in the side car. So the operator needs to make a series of calls from the operator to the the how your side car that's running uh on the pod. This is for things like security cloud providers, persistent storage, carry configurations and a few other things after you validated everything we then generate the manifests. This again requires a uh a call out to the halyard sidecar we pass.

C

The hal can figure out to how you're how you design car the micro services uh configurations are generated. This is things like your cloud: driver.yml, your fiat, yml files, etc. Those are then processed by halyard to generate kubernetes manifests, which are then returned back to the operator.

C

At that point, the operator has a series of transformations that it may pass those manifest through through and then the operator deploys the manifests.

C

We then repeat that, following the general control cycle of checking the desired state versus the current state, and until we get an update, we keep things running.

C

So what does it potentially look like if we were to remove halyard all together? So you can see the operator gets a little bit bigger. We still pass in a spinnaker service, yaml file, but this time we don't have a sidecar running. So at this point we need to get the services, uh the service versions for the spinnaker version, you're trying to run so. If you're running spinner 1.26 that needs to map to cloud driver version, you know 1.26.2 or whatever it might be.

C

um This needs to be done via some kind of a bill of materials like method or something like that. um We need to figure out what that looks like then we need to do the validations previously. This was done in halyard. So we need to migrate that work into the operator.

C

Then we need to move into uh generating the configurations for all the spinnaker subservices.

C

This is something that the cleat library is directly able to handle, and so we can pass the how config from the operator code into the cleat library running in the operator and then that will return all of our various spinnaker service configuration files, and this is where things really change. In the powered version of the operator we've relied on how you're to generate our kubernetes manifests in a cleat version. We're we're not going to rely on cleat to do that generation.

C

Instead, what we would do is look at the uh customization based templates that the spinner community has provided they're located in github and that allows us to take our cleat generated, manifests or configurations pass them into customize. Alongside these, uh these community provided manifests and templates, and that can generate our spinnaker kubernetes manifests using customize.

C

We want to find a way that we're not necessarily reliant on being able to talk to github to pull those down in real time. So we'll need to look at how that works, but this is kind of the pla, the plan for the community going forward on how to set up spinnaker once you have those manifest, we go through the same transformations that we did in the higher version and then move on to deploy.

C

So now looking at where things are and where things could be going, what do we have to do to get this done?

C

Well, we've identified three kind of milestones that need to be done to get this this implemented in the spinnaker operator. The first thing is to migrate those validators from halyard, which is written in java to the spinnaker operator written in golang.

C

This is a pretty easy, lift and shift, but there is going to be some language translation that needs to happen for the logic to function properly.

C

After that we need to migrate the spinner service configuration from howard to clean.

C

We want this to be done in a backwards compatible method, so that you can still take the same. How config you were passing to the halyard side car before and this time pass it into the library um we don't want teams to have to go through and modify that that uh how config manually we want it to be as backwards compatible and as low effort to uh migrate as possible, and then, lastly, we need to migrate.

C

The generation of the actual kubernetes manifests from halyard to using the new customized base uh repository that the spinnaker community has provided. So with these three things, we can completely remove halyard and replace it with cleat and maintain full backwards compatibility to the current operator.

C

So why are we having this conversation? What does this mean for you? In short, we would love to have collaboration with other organizations in order to help get this implemented.

C

We always welcome prs to the spinnaker operator and we'd love to help you get involved. If you have questions or would like to contribute, you can find all of us on github and on the spinnaker slack team. You can find me personally at t underwood, on github and at todd underwood on the spinnaker slack team and the rest of the team has their contact information here.

C

If you're interested in contributing to this effort, don't hesitate to reach out we're happy to help get you started and help plan the road map, and things like that, if you, if you want to be involved so with that, I'm sure we have some questions that you might want to ask and we're happy to answer them. But that's what we have for this, this content right now. So thank you very much and we're looking forward to your.

C

D

E

I think you might be.

D

C

Okay, so sorry about that, hopefully everyone can hear me now uh we're having some technical issues today. So I apologies for that uh hi everyone. My name is todd underwood, as you heard in the video I'm the technical lead here for uh for armory, uh and so, if you guys have questions where uh fernando and I are here to take them and are happy to answer them. So um it looks like we might have one person in the queue right now, so we'll go ahead and pull up a niche.

C

uh I'm gonna see how this works.

D

C

There any if there are any questions, feel free to put them in the q a as well.

C

C

So at this point, um uh we've ended the uh the pre-recorded radio content we had for this talk, uh if you guys have any questions about contributing to the operator or about uh converting it from uh how you're into more of a go native uh implementation with cleat feel free to ask fernando- and I are here to answer questions for the next 10 minutes or so. Otherwise, that's all the content. We've got.

E

E

Yeah, I'm just checking the q a section here and I don't see any questions so far so feel free to put them in the chat or in the q. A.

E

E

So it looks like we have a question from thomas todd: do you want to take a stab at it, or would you like me to.

C

How about you take this.

E

Sure so um can you uh speak a little bit more to what you're interested in in the customized learning curve? I mean I can talk about it in aggregate, but I can provide a better answer if I know exactly what you're looking for.

E

um I guess the the general case is that customize is a tool similar to helm right where it allows you to templatize a lot of the the manifest that you have customize provides a little bit less of the um like structured packaging that helm does and customize allows you to do more of that, like targeted changes that you want to make to just specific parts of a particular manifest, um I would say the the learning curve is pretty pretty similar, I would say, um but yeah we can. We can dig into more of that.

E

If you have a different area that you want to dig into thomas.

C

So, thomas for that one, it probably depends a little bit how how familiar you are with kubernetes and things like that. um But I I can say uh I I probably picked up the basics of customize in just a couple of days. um It's not a terribly complicated thing to pick up once you learn, learn the basic patterns, um so I I would say that once you get kind of that basic pattern stuff, you can take off pretty quick.

E

A

E

Like that learning right once you cross it, and you kind of understand things and I think the biggest hurdle for me was understanding the resolution system. That customize has right. It's like when you have your your main file and you're, declaring like the api version kind and name, and then um you know, you're, creating patch files and you're wondering like how those patch files get resolved to that main manifest. I think that was the biggest challenge for me, but once I understood that you know the the rest is is pretty.

E

E

We have one question from jay in the q: a section does cleat part: does the cleat part make bomb management easier? So I can. I can take that if you're interested todd so there there is less of a concept of a bomb uh when you move to cleat.

E

So what klee does is it takes right um and you can see examples of that in the customize or sorry in the cleat repo it links to like a base, customization repository and that's actually where all of the images are going to be declared from this point going forward. So there isn't going to be a bomb. That's going to live in like a halyard bucket somewhere that you're going to pull down right, um so you'll you'll be able to do a lot more customization.

E

Well, not very much not intended, but you can do a lot more customization with the kinds of images that you're sending out.

E

You know. I only want to deploy this. This version of the image of the other- and I know as as part of the release process, revamp that some of our colleagues are doing.

E

I can't recall what sig exactly it is, but I know we're planning on revamping how a lot of these containers get built and published, and I think that'll that'll change as well as we get closer to the resolution of that work.

E

Okay, we have another question from marlon: do you think cleat will make air gap installs more frictionless yeah? I think, by definition of not requiring a bomb file. I think that's one of the largest issues with doing an air gap install today, where you kind of have to recreate the environment where that bomb file lives in order for halyard to be able to pull it. um So you know with that base customization directory now and then any patches you want to play. On top of that, I think it'll be a little.

C

Bit more straightforward.

E

For you to install in an air gapped.

C

C

So we have another question so uh in this case we use.

D

The armory operator.

C

It's kind of a it's a kind of smooth one question I have is: are there any plug-ins available to access vault secrets during the execution of the pipeline uh kind of a side car for vault uh for the case java or pod.

E

Yeah, this is my knowledge. This is available in the army operator and it's not available in the open source operator. So, if you're interested in taking advantage of that, functionality um feel free to reach out to your your account manager, and they can point you in the right direction.

E

And we have another question from jamie: does cleat support revert generated manifest back to how config? So if, if I'm understanding the question correctly I'll rephrase it here, but I believe what you're asking is if a cleat can generate a halo config for you to go back to halyard.

E

If you choose- and I I don't believe, that's possible right, so the the migration path is is mostly going to be mechanical changes going from your hal config to an equivalent customized base, and then the spinnaker service manifest that you'd be using to.

E

E

E

I think that's a different question jay. So if the question is, does cleat make managing vault secrets in the pipeline easier rather than the tool itself, I don't think it. It makes it any easier or harder than it was under hallyard you're, going to run into the to the same issues that you were before. Cleat is really just um one half of the halyard equation right. It's really just generating the configuration that you're going to put on the pods.

E

um It's not going to resolve any of the secrets that you're trying to put into your what was the hal config right. So any any secret resolution is going to happen in the operator.

E

C

Okay, well, we have about a minute and a half left or so so, if there's any further questions, we're happy to take them. um Otherwise, thank you guys all for coming out and uh watching the talk and uh feel free to reach out to us. If you have any questions about operator or uh you know getting it converted over to cleat.

C

And enjoy the rest of the spinnaker summit today and the rest of the.

C