Continuous Delivery Foundation Spinnaker Summit 2022 - Detroit, 4 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Shore - A Framework For Spinnaker Pipelines - Eyal Mor & Simon Chammah, Autodesk

Description

For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/

Shore - A Framework For Spinnaker Pipelines - Eyal Mor & Simon Chammah, Autodesk

“Shore” (verb) is a tool used to develop deployment pipelines for different pipeline based products like Spinnaker. Developed in Autodesk, it is currently driving “most” of the Platform Pipelines Development in the company. As part of the summit we will introduce Shore Framework to the world, And unveil the project in the Open Source Space. Presentation will include (still preliminary may change) * Developing Pipelines is a hard task * What is Shore (and why?) * The use of Shore in Autodesk * Open Sourcing to the world on Stage * Roadmap & Future Why is Developing pipelines hard? Developing CI/CD pipelines is a hard task in a Platform. They need to be resilient, easy to use and extendable pieces of software that are run by another system (AKA Spinnaker, Jenkins, Tekton, ArgoCD, Waypoint, etc...) When developing, extending or testing pipelines on any 3rd party system there are a few key problems that come up. * Local development can be very hard * Stages/Pipelines are not inherently testable * Collaboration and code re-usability could be re-defined

A

So hi everyone thank you for joining us in this lovely city of Detroit. Thank you. Everyone at home, also for joining virtually I'm Simon, Sharma and I, will be presenting today, together, along with ayal about your framework for Pipelines.

A

So a little about myself, I'm managing the platform team in Autodesk in Tel Aviv, our Squad. Our Squad is part of the Development enablement Group in Autodesk father of two I love crossfitting and traveling around the world.

B

Hi I'm Miguel Moore, a tech lead in the platform, engineering team in Autodesk, Tel, Aviv, a tech Enthusiast and other than that; I love playing music and some anime. So if you want to talk about that later,.

B

Again, we have other core members that couldn't be here in person, so we want to make sure they get the spotlight again. It takes more than two to build something awesome.

A

So what we'll be talking about today? First, we will give some background about the deployment journey in Autodesk for the past years. We will then cover the problems and challenges when God encountered in this journey, focusing on Pipeline development from there we will jump directly to show what is it, including all the tech details and a demo, and finally, today is a big event for us. We are open sourcing show to the community.

A

So a little background about Autodesk Autodesk has now gone into more than 13 000 employees, all across the world, out of which four thousand are software engineers developing about 98 products across nine Industries. This is the customer we need to support. Each of these teams have specific requirements. On top of that, there's also compliance. We need to enforce, starting with sock 2 on all our Cloud, offering through gdpr and even Federal Model 8 designation on some of the offerings.

A

So the mission that was set by Autodesk development enablement Department was to increase developer productivity by Android X. But what does it mean? Basically, it means to make the life and the experience of every developer in the company simple and easy in every way we can. We want our developers to focus solely on delivering the best products to our external customers around the world.

A

So how can we achieve that by building an extendable, secure and compliant Cloud native platform, a single common platform for the company built with disco principles, API first and consistency that can maximize us developer, productivity.

A

To the problem, so we now all know how everyone starts. We take the spinach UI write. Some spell expression, make the system work. Everything works well for 10 minutes until you need more customers ability.

A

Then, after receiving a set of business requirements, we needed a platform to support. We created tooling that automated the creation and provisioning of deployment and delivery Pipelines, then, as we scaled and more customers, onboarded more features and Bug fixes were getting released very frequently having more than 10 developers, contributing different capabilities and collaborating over one growing repo became challenging more than that. Releasing many new versions of each Pipeline and if Custom Tool without the breaking the platform was very hard. We were defining our pipeline as code, but not treating them as software.

A

How about testing it became increasingly difficult to test every use case combination.

A

We then just realized. It was just a bunch of tools serving some purposes, not a coherent platform. In order to scale we needed to change the way we think and treat Pipelines.

A

So we understand we needed to enable our platform developers to move fast by providing them guidelines tooling and common libraries. This will allow teams to scale their capabilities independently and iterate quickly. Those are the building blocks of a framework.

B

Cool. Thank you Simon for.

A

B

The problem- let's see if we can solve it so to tackle the challenges we just brought up, we built Shore. So what is Shore framework for pipeline developments? Simple, as that Shore was built with a goal to make pipeline development a first-class citizen for software engineers, so a framework allows two teams to focus on the business logic. Usually short was built to give that same experience when building pipelines, creating a new term really pipeline Developers for the more techy interested.

B

The architecture slide now sure it's built with golang, which means it's easy to create binaries pretty efficient on our CPUs. Usually, the project is internally separated into a command line. Layer in core layer command line is the bottom command line from there.

B

It contains the logical, so the command line contains the logical interface for reading configuration files and passing them to the Core. Is the execution layers there in the middle?

B

It deals with interfacing with a render and a back end so sure to actually build the framework and get the libraries we need to render something and output something and save it.

B

Shortcore deals with that. We're going to do a demo now, so you can really get a feel of what this actually means, because everything in the architecture is high level. We have some libraries we have a render, but to really understand the feel of development developing mature. You need to see it first.

B

So without further ado, we're gonna do a demo over Wi-Fi.

B

So put this here and this here great and.

B

Can everyone see in the back I hope everyone can see at home? I have no idea, so we start with a Spinnaker UI, as most people do now I'm a new developer in the team. I need to fix a bug. I want to start developing.

B

The lucky piece is I know how to start working with this, because I learned how to use Shore like any other framework. There's a tutorial tells me how to start how to end my work. So, let's jump into the projects.

B

Jumping into the project, I know how to start looking at this I know that a short project starts with a main pipeline jsonnet file that defines my pipeline. Let's just check that it exists and not go into too much details and let's increase here a bit and not update anything. Yet not here cool, so the file exists and we're good with that. For now now, I got a bug request like just right now on my phone issue, 256 a parameter, a number parameter will fill the pipeline.

B

What does that mean? Well, basically, the pipeline executes a cube job runs, it echoes one I know it's riveting, of course, and then it deploys unnested pipeline or invokes a nested pipeline now I know I need to fix this bug. Problem is I, see the word production here.

B

If I start working on this pipeline, what will happen? Basically, I'll bring down production and I get a call and everyone will be upset. Let's not do that.

B

Let's first set up our developer environments, so I can see from like my first short tutorials. That params pipeline is the pipeline name that I give to all my Pipelines.

B

The paramet argument is passed from the render yaml again I can see the production name. Let's change that to I-256, which is the issue we just got, and we will invoke two short commands sure render on the terminal. We can see it renders Json. We can even pipe it through JQ to make it look nice and we will run Sure Save.

B

So now, after going through Wi-Fi and hopefully hitting our Spinnaker instance, if I go back, I can see the nested I-256 short demo pipeline popping up now. I can securely develop without breaking down a production, which is usually a nice thing to do.

B

Okay, so now that I have a developer environment, let's start executing the pipeline and see the bug so to execute a pipeline, ensure I go to the exit camel and look at the execution units.

B

I will just write sure exec, which will send the parameters to the pipeline and application here. I will change to I-256 or developer environments and execute that and now it will execute and probably be okay, because that's what was saved in the original repo, so no one like saves bugs hopefully, and we will go and reproduce the bug so I know that the bug says an integer will break everything.

B

Let's check good practice, usually check your check, your bug report and now, if I wait a few seconds, I hope that everything breaks and it did so, let's read real quick. What happens? A container spec containers cannot be handled um of type string. This is where Jedi training comes in I know. This is a type error.

B

I put a string worked, but an end didn't work simple as that: let's fix that problem. So going back to my pipeline I, look at where is runcube job stage defined now the library is really helped me with that, because the API literally says runcube job. That's nice!

B

Now some good Samaritan already put the bug, fix, I, don't know who, but let's just use it so I will run Sure Save now, with the bug fix, the bug fix will take a parameter called echo1 from parameters apply the two strings, so no matter what comes in it will be, stringified kubernetes can handle it, and then we wrap it in a spell expression.

B

So we saved the pipeline with the fix we can short exec and now, lo and behold we will survive another day. Well, let it survive Another, Day, first cool that fixed it now, switching back to my editor. This is the time my open up. Er said I'm done giving the next task, but if I open the pr now Simon will tell me stop where the tests, how do I know their fix, will stay fixed forever or as long as the pipeline still lives?

B

That's where the end-to-end yaml comes in so Shore is built with testing in mind. That's something that is a pillar of any framework and it sure is no exception. E2E stands for end to end. It will be invoked with the test remote commands.

B

If we look at the end-to-end yaml, it contains a test Suite with names for each test and assertions for each stage. We can also test outputs here, but we'll keep it simple with just status checks for now. So you can see the Good Samaritan also left us a how to fix the next bug. But first, let's look at this.

B

Someone left a the original test, so we know the string always works. Let's uncomment this and change to I-256, and now we can run a Shore test remote. We will add two flags. We don't have time in our lives as developers, so we'll add a concurrency flag and some verbosity, so we can see what's up both in the terminal and in the UI so we'll hit enter.

B

We see it running concurrently and if we switch back to Spinnaker, we see our two pipelines running and now, when both pass I can open the pr and Simon will approve- or maybe hopefully.

B

That's it. The live demo actually works. You want to see, you want to show the.

A

B

Let's go back to the slides.

B

So I wanted to note that this demo is super simplified, like they explain like I'm five scenario of it, but it really represents how we use Shore and Autodesk to build a platform. The idea of a framework is to make life simple.

B

So having said that, there are so many other features that we didn't even cover, yet that I really recommend you look into I want to also Show an example. So our colleagues in Autodesk talked about the platform. Here's an example pipeline that is generated by Shore, which is a multi-region multi-n of deployment with nested pipelines. Parallels all the shenanigans, making very easy, like made very easy with your just automating the whole thing, so it becomes travel to write. Something like this.

B

Now we're not even close to done with this project an Autodesk it serves as well in its current form and we're good with that. But we believe that pipeline development and its evolution is not just here to stay, but here to evolve and believe that the frameworking and software development side of pipelines is something that it can be shared in the community and cncf entirely.

B

So, what's in store for the for the future well short term, we're gonna do some improvements, video error message: whoever saw the errors inside they're, not that good yet, but the concepts are solid, but the long term. We want to actually add more pipeline Runners and build the community around that. So we can really have the same conversation, no matter where you are and speak the same language, Sure, Save sure exit and we want additional language support as well.

B

So if G sonnet is not your groove, maybe we'll find something that does work for your team uh speaking previously on the cncf it's ever growing, so we do believe that it. It is important to at least consider adopting the framework, no matter where you are in the cncf landscape and see if you can involve with us.

B

And if you want to know more, the framework has this morning been released open source. So you can search it in the Autodesk, get a repo under the name Shore or ask us for a DM and we'll send it to you via the um we'll. Send you a link or ask this later, if you're here in person and start asking questions in the discussion tab on GitHub and with all that said and done.

B

Thank you for listening and any questions.

C

Hello, hello, hi, um so just to be clear, like the the core component is a component that you deploy. Is that so.

B

The core component is baked into the binary. We just saw the CLI, but a in the diagram. I don't really touch into it, because it's not explained like I'm five level, but you basically can take the core and embed it. That's what we're actually doing right now for the very complex scenarios, so we develop with the framework and then take the jsonnet code and move it into an orchestrator, that's higher level. That's why the core is a independent from the CLI.

B

Does that answer.

C

So when I'm, if I wanted to adopt Shore um for doing pipeline development, then I would only be interacting with the CLI to start with most.

B

Likely you'll be interacting with the CLI and I commend you for doing that. It'll be much nicer than trying to implement the core we're still working on like the API design there. The CLI is solid and.

C

If I do sure save, then how does that connect to Spinnaker directly so.

B

It currently uses the spin config file. You have locally so any way you already connect, we are working, I showed them the short term. We are working on a configuration language like configuration layer, so you can bypass that and put whatever you want and automate. That's coming in I I think like in a few months here. So we already have that problem with the spin config.

D

I think maybe you already answered this, but maybe not uh there's a command line tool that I think people call the spin, CLI and I'm curious how, if you're using any part of that for sure.

B

So the back ends is actually spin, cli's Library strip down from like many pieces, but we're reusing like we're trying to reuse as much as possible, because there is no reason to rewrite something and we want to contribute back any bug fixes we find so we're not trying to replace anything. It's been CLI I still use it to like find all the pipeline and stuff like that. It's not good for, like we didn't find it beneficial to manage your pipeline development in a developer-centric way.

B

But if you're, just looking at what pipelines you have what applications you have that's a great choice like keep on using spin CLI.

D

I guess I'm also curious about spawn it and how much of Spawn it you're using so.

B

We started with spots. We actually migrated away because of a few issues we had in a initially. We are going to open source, the spinlib jsonnet that we built and hopefully we'll merge them back in together. The biggest difference between them is Builder pattern versus a composition pattern in terms of libraries, but going to GitHub, see if you like it like.

D

Yeah I mean I'm just trying to uh yeah consolidate the effort like if there's something that we could do differently. The way we package spins so that there we publish a library, that's consumable that you guys didn't have to do. You didn't have to do as much as you did or whatever stripping down you did. We could contribute back so there'd be a way to publish a strip down library that that's the kind of angle I'm I'm. Looking at.

B

So, just to clarify the what we did was spin into a to get into work. Wasn't that hard? The real benefit of shore? Is the pipeline developer experience? So it's not just Spinnaker. It's really like the whole API.

D

Oh yeah I think this is a very cool tool. I didn't mean to say otherwise. I actually have other higher level questions about like what about being able to run a pipeline and more of a dry run mode or being even more paranoid about sort of release. Cadence and you you've done your due diligence to make your PR and, if you're, very lucky it gets approved.

D

uh But then, as you go to impose it on the rest of the world, there are some risks there and and are you you know what are you doing to mitigate that? And maybe that's what you were talking about with using using the core as a separate, a separate thing and other orchestration, but that stuff I think that would also be very valuable to try to open source. If you could.

B

So you're completely right by the way about the mocking and like the risks that well in any case, whatever you use in your life has risks. When you put like a framework out or like you, can you can create risks? The drive one is something we're talking about. It is a big problem to solve and I think it's a conversation we can have on a GitHub issue. If you have any idea how to solve it for Spinnaker, specifically I would love to like touch base on that.

E

Foreign, could you talk a bit about um security, essentially like how do you ensure uh you know the right people are um you know, executing and and how? How, where is your security yeah.

B

So, honestly, we leverage the security of the system, we're using we're not trying to solve security at all. We impose Security on pipelines via libraries, so, like your deployment, is secured. That's what we do in Autodesk, but the security we're talking about like our back I am whatever or in Spinnaker's case, say: um let's say: Azure ldap groups, that's what we use. uh We don't try to impose a security posture. You decide on the application how to secure it.

E

Pseudos, inheriting, essentially from.

B

Spinning yeah from any pipeline Runner like we, don't want to impose something yeah, so Spinnaker in this case. Of course,.

F

Yeah uh hi, thanks uh for open sourcing, this and yeah looks great um I was wondering what what plans do you have for um growing your community? Your governance is the the source that you're open sourcing or you're planning to build a community open sourcing, the governance as well or.

B

So we are starting here by just talking to you after the in-person talk, that's the first step and we want to understand what is the use case for other organizations, companies that may have these same issues and what are the focus areas you have from there we'll probably go through this same open source steps, a governance, but we can also join another a organization, that's still unclear on what exactly we'll do. We first want to talk to the community here and see if there's like interest.

F

Great. Thank you.

G

I'm just going to follow up on that question, so my thought was that it would be part of the Spinnaker Community, but it seems like you are exploring other options too. So it may not necessarily be the Spinnaker Community I.

B

Think that each renderer back ends. So it's like a thinking of other tools. It doesn't have to be confined to one Community. We can have communities contribute from anywhere. Spinnaker is the first one, because it's the most important one for our use case and I. Think it's a great Community to start growing, but Spinnaker as a community is also working together with other communities like the Argo Community I've. Seen tecton is spoken here, so I don't think. There's like a conflict, we'll start with the best community we can start with, which is here.

H

Hi guys I'm Al from from Google and I'm, really interested well congratulations by the way. This is very cool and thank you for open sourcing. It I'm really interested in the idea of having a standard pipeline or a pipeline standard. That's sort of independent of what you're calling pipeline Runners, which are just different back-end environments, I'm wondering how you got there or what's? Were you trying to solve a specific issue in your environment? Do you have other Frameworks Beyond Spinnaker that are running yeah.

B

Autodesk or yeah you want to take that like.

A

um Yeah, there was no, we did. A research actually did not find any framework solving these problems and the more we scaled and everyone was using, automations and tooling, but there was no real I. Think solution. Yeah.

B

So when we said did the research we looked at other communities, the biggest problem we had was actually testing. First, that's why we're so focused on that in this talk, testing was really hard, so we just looked at other communities and and when I'm seeing other it's Jenkins, our gold tactile everything that was available. 2019.

A

So especially, this became important when we moved to operator pipelines because it was operator pipelines for each use case run by 100 times every every hour, because it was one pipeline for the whole company running the same thing. So every little change every could break the whole company. This is why we were looking for testing to automate our iterations and our releases, so.

B

Taking from testing, we then looked at the common use, cases that evolved from other pipeline Runners, and then we saw most of them are actually very similar in design. Okay, you have a pipeline. You have a stage, in fact on it's called Pipeline and task. Argo has its workflows with each component for the workflow very, very similar in design, but how you interact with them is super different.

B

We kind of found out that you can unify at least that level of API for the developer. So, let's looking at the resource, creating one resource to rule them all, but more one tool to talk to them all kind of situation. Does that kind of answer? The question.

H

Yeah I think so I mean there's no real answer to the question I'm just interested in how you got there. It's it's a very so it's less of a sort of universal pipeline standard, then, and more of a pipeline development environment that supports lots of different exact runners or back-ends.

B

Universal develop pipeline development standards right right and.

H

The idea that there will be an emerging class of pipeline developers who, if they are familiar with that environment, then can be successful across different back-ends as they journey through their career. Yeah.

B

I would love to see the evolution after people use this tool, because the evolution is way more interesting in like two to three years. If people start adopting seeing the future of what they come up with, what's a real standard, what they think is how to interact how to test. This is just a starting point, really cool.

H

B

Thank you for the question by the way I really enjoyed that.

B

Okay, now for good, thank you very much, I think it's a wrap. Thank you.