Continuous Delivery Foundation cdCon 2020, 20 Oct 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Spinnaker at Salesforce: Enterprise-Level Multi-Substrate Continuous Delivery and Re... Jing Vergara

Description

Spinnaker at Salesforce: Enterprise-Level Multi-Substrate Continuous Delivery and Release Management - Jing Vergara, Salesforce

Learn how we manage releases at Salesforce and take Continuous Deployment to the next level. Listen to the complex release orchestration challenges we face. Find out more about how we iteratively solve the full automation of movement of all code and infrastructure changes, from local development all the way to multi-substrate production environments and multilevel security zones.

Looking for something specific?
0:00 Introduction
1:04 Spinnaker: Footprint
1:17 Instances and execution
3:53 Apps, templates and guidelines
4:44 Pipeline types
6:24 Salesforce Journey: Challenges
6:39 Challenge 1: Pipeline templates
8:54 Challenge 2: Release across multiple environments
10:00 Challenge 3: Change management
11:50 Challenge 4: Failure handling
13:40 Challenge 5: Infrastructure builds and changes
14:32 Challenge 6: Spinnaker account management
15:24 Challenge 7: Alerts and notifications
15:58 Challenge 8: Pipeline security
16:50 Challenge 9: Handling promotions
17:40 Challenge 10: Complex release orchestration mechanism
18:14 Roadmap: Future Projects
18:48 Q&A

SUBSCRIBE CDF CHANNEL ►► https://tinyurl.com/y4e4434n

ABOUT US
The Continuous Delivery Foundation seeks to improve the world’s capacity to deliver software with security and speed. Learn more at https://cd.foundation/

A

Hello, my name is gene ferguerra. I am a principal software engineer at salesforce, I'm part of the organization that paves the path for continuous delivery into multiple public cloud environments. I work with the different internal engineering groups to capture release requirements from these different business units.

A

Our spinnaker teams then build extensions and capabilities to enable all of these requirements. Have you ever wondered what it takes to run a successful cd or continuous delivery platform like spinnaker at an enterprise level, scale company like salesforce?

A

We started spinnaker evaluation a couple years back and today, I'd like to share our journey with spinnaker and the lessons we'll learn along the way.

A

I will be focusing on three main topics: I'd like to share our spinnaker footprint at salesforce, the top challenges that we faced and our long-range plans.

A

So can you guess how many spinnaker instances we have at salesforce?

A

If you guessed one two or three, unfortunately, that's incorrect. We actually have about four spinnaker instances, and that number continues to grow main reason for having many spinnaker instances is to minimize the blast radius.

A

We have what we call high secure zones which are built up of isolated environments, wherein it cannot be reached by any other spin occur instances. So we have to build a spinnaker instance for each of these zones.

A

We cannot have one global spin occur instance that can rule them all because of this architecture. We are also have to go and split our image baking activities from our deployment activities.

A

We want to make sure we have a single image that we would distribute across all of these different instances, so we have again the image, baking environments and the deployment environments when we started with spinnaker at salesforce. We also started with deploying to pre-production environments.

A

These pre-production environments are internal environments that handle our development and testing of the pipelines and all of these software versions. So we use them for integration, testing, for performance testing and such during this time period.

A

We have actually studied how spinnaker can handle the different scale and load capacities that we have so once we've actually figured out what it would take for spinnaker to run. We've built a production environment that will handle all of the production usages, so we have a split environment between our pre-production target environments and we have one for our production environments.

A

Currently, our pre-production environments have about 30 to 50 000 executions per day, which is about 1.8 million per month, and this number continues to grow and in production we have about 5 to 12 000 executions per day, which is about 200 000 per month and growing. We are forecasting that um productions usage will grow more than what we have in pre-production very very soon um spinnaker, as we all know, supports um different cloud providers and at salesforce we use about three substrates or three cloud providers.

A

uh We also have a lot of spinnaker applications across all the different engineering teams within salesforce, so we have about a thousand of them and each of these applications together. All of them. We generate about 50 to 60 000 pipelines, typically related to each environment, that we have per service, and then they are generated from about 5 000 pipeline templates. So, based on these 5 000 pipeline templates, we pre-generate all these 50 or 60 000 pipelines, and uh we have thousands of spinnaker accounts which would include both ec2 and kubernetes accounts.

A

uh Our vm and kubernetes accounts that we have or support at salesforce. We also notice that our customers or the service owners actually perform similar sets of activities and based on these uh change activities, we have categorized them into these pipeline types. The first one would be the generation of the vm images and again we use baking pipelines for this. So, for example, for aws this would be the amis and for google cloud. This would be the gce images that would need to be able to run on the gc vms and then for deployment pipelines.

A

We support again both kubernetes and vms, so we deploy to these two technologies and then for setting up we're building cloud resources. We have what we call resource, provisioning pipelines for them and then to tear down these services and resources. We have the service decommission and the resource destruction pipelines.

A

We also have other pipeline types and some of them. We call like the test validation pipelines, which are typically used for testing services or environments or resources, and then the casual would be something like a utility pipeline where service owners can use them for patching for capacity like increasing capacity decreasing capacity or if they have stateful services, then they can use that for their database, backup in restore.

A

And now that I've shared how it looks like with spinnaker at salesforce, I'm gonna present the different challenges that we faced when we went from the proof of concept all the way to production.

A

So the first challenge that we had was we didn't know how we can build pipelines from an environment bill of materials before so. What's an environment bill of material, an environment, build of material is a list of configuration files that describe what environments do we need to apply the changes to and what are the configurations around those environments so think about what regions do we need to support?

A

What are the account numbers for those? What are the ip addresses? What type of resources we need to build so think about in those concepts? So we have those and we need to be able to generate the spinnaker pipelines based from those configurations.

A

So what we did was we created our own template management system that can handle or read these bombs and then via gitops workflow. We check in those configurations. It gets delivered to our production or spinnaker instances, and then we have our template management system. That reads off of those and calls the spin occur api to generate the pipelines from those again. We need to have different templates for all the pipeline types that we would need to support to validate. All of these templates.

A

We enforce oppa policy checks against these templates to ensure that they follow the standards that we have set for the company to make sure that they are secure and that no one's keeping a step that must be done for every pipeline run.

A

We also have to support pipeline template versioning for these templates to ensure that for any change that any team has, it goes through the slow or gradual rollout process to ensure that, if something broke in the pipeline definition, we test that earlier on in the um death or the pre-prod environments before they actually make its way to production.

A

Those. The second challenge that we have is that. Okay, we have multiple environments, then how do we orchestrate the releases across all of these environments? Right? It's going to be um painful for a service owner to go into a spinnaker ui and ask them to say you have to click on.

A

However many environments you have so, for example, a service owner would have a hundred or a thousand number of environments. We can't expect them to start all the executions one by one on all of those environments. So what we've done around this is? We came up with a concept of deployment manifest and master pipelines, so the deployment manifest would describe what artifacts would need to be deployed into which environments, and then we pre-built the master pipelines, and that would show how we would set the canaries and the staggered rollouts across all of those different environments.

A

The third challenge that we have is that how do different teams keep track of all the changes that go through the spinnaker? So we need these for auditing and compliance, and then we have the other personas like the service reliability engineering and um that you typically like debug issues for a particular environment.

A

So all of these different teams would want to know what happened along the way during this pipeline executions, so to keep track of all the changes that go through. We have integrated with our change management system at salesforce.

A

In there we attach all the change lists or pull requests that we have for a particular change, and we include a deployment manifest.

A

That again would include artifacts in the target environments, and then we also attach data test data and results.

A

We've also pre-generate all the change case records for the different service owners so that they don't have to manually build these change case records especially gets again very very hard. If we're talking about hundreds to thousands of environments that we need to go, deploy to, and then for the site, reliability engineers, they actually have another problem within spinnaker right now. You can only see the pipeline execution for one application at a time. There is no cross application or global application view.

A

So we need to build this view for them so that they can detect what change actually went through for an environment given for a particular point of time or a window.

A

The fourth challenge that we have was on failure handling. So now we can handle the release orchestration at scale, but what? If, when a couple or a few environment deployments or changes actually fail, would that mean starting all over again? So if you have a thousand five failed, you have to repeat all the thousand again right that that doesn't sound um optimal at all. It's very again, very painful right. So what we've done at salesforce is we introduced um the concept of master pipeline item potency?

A

We keep track of the states of all of the stages or executions that have run, and so we look at the deployment manifest and check whether we've already executed that before. If we have and it succeeded, then we skip those targets and just continue on with either the failed or the incomplete ones.

A

We have also um added some logic within the pipeline execution, um some failure handling like pipeline retries or the ability to ignore failures. So this allows the service owners to go either manually resolve those failures or they can set in their configurations whether they want to enable those to be automatically retried or automatically ignored.

A

uh We also wanted to put some automated rollbacks in there, so for those systems that use like red black strategy and the old infrastructure is still available, we'd like to switch back right away whenever there's a failure.

A

The other challenge that we have is on infrastructure builds and changes. So typically, you would think that oh yeah spinnaker continuous delivery platform. It can only handle software upgrades right, but at salesforce we use it for more than that. We also use it to orchestrate infrastructure, builds and changes, so we use the other open source um tool called terraform, which is basically infrastructure as code again, it's a set of manifest that describes what kind of resources you want and what properties those resources would need right.

A

So we orchestrate uh the execution of those terraform code using the same set of master pipelines that we use for service upgrades as well, and the sixth challenge that we have is around spinnaker account management.

A

So in at salesforce we had a tremendous growth of accounts, as teams from one group to another started onboarding onto our system. Initially we only had like a handful and then in the tens and then in the hundreds and eventually into thousands, and then there was a point in time when spinnaker cannot handle any more new accounts, so we had to go and create our own init inside car containers.

A

For a cloud driver and rosco to handle all those accounts and secrets so that it can continue to work, and then we work with the oss team to support and improve the dynamic account handling within spinner crew itself.

A

The seventh challenge we had was around alerts and notifications right. So um in some cloud platform I think it's aws uh pub sub events are not supported out of the box. So what we've done behind the scenes is? We implemented echo web web hook, so we can listen to those events and uh we integrated it with our metric system and we integrated it with pager judy as well, so that we can be alerted of all of the different events, either success or failures.

A

The next challenge that we had was on pipeline security. So how do we secure pipeline executions at salesforce? uh We do not allow triggering of the pipelines manually in the spinnaker ui. That's an administrative level of um permission or access. um Everything has to be done via get ops and then in our github solution we enforce um 2ha or to human approval, where someone else has to approve that request before it's allowed to be able to go out and trigger the pipelines in spinnaker.

A

We also have some manual judgment stages that ensure some pipeline executions, and we also have enabled our back policies within spinnaker.

A

The ninth challenge that we have is: how do we handle promotion so promotions is making sure that when we go move from one environment to the other, that it actually goes and uh there's a stamp of approval, so we have allotted in our master pipeline certain promotion pipelines that can handle either an annual judgment where someone has to manually approve that it's good to go to the next stage.

A

We also do soak time or in it, waits for a certain amount of time to make sure that some regressions are that are not caught in automated tests can be handled hopefully or can be caught during that so period, and then we also have integration with some of our automated test framework suite and then the last challenge that we have is on the complex release, orchestration mechanisms right. So we have these requirements that hey you know you cannot allow mutually exclusive changes to occur.

A

At the same time, we have to make sure that we don't run pipelines during moratoriums or, if there's an incident or these services can only run during these release windows. So with this, we are integrating with a locking system to be enabled to enable all these complex requirements, and we have a lot more projects and problems to solve at salesforce and here's some lists, so we are currently hiring.

A

So if you are passionate for continuous delivery problems and distributed systems and automation, please bring me we'd love to hear from you and with that I'd like to thank everyone for your time and all the best on your continuous delivery journey. Please feel free to reach out. If you have any questions. Thank you, hello, everyone. So I think we're opening up for q a so if you have any questions, looks like we have one from tony. What automated test frameworks are people using um so for the test framework.

A

We actually have an in-house test framework at salesforce, uh so we built we built our custom custom framework.

A

uh Thanks fay so face asking: how are people using pre-prod and product can occur differently, so the difference between these two environments is that on pre-prod we're targeting development and testing environments, uh while the pro deployment is where we target our production environments, so typically, they would want to go and test either their software version or their pipelines, so they they generate their pipelines and they want to test it out. First or again, if they're doing a software upgrade, they want to test it in their development or test environments.

A

Then they go to the pre-prod deployment spinner and that's where the releases happen for those or that's where the changes happen once they feel comfortable. The release manager gives the stamp approval. Then they go ship it in the production environments um thanks paul. So the question is: does that in-house test framework interact with kayenta at all? We have kayenta in our roadmap. It's not yet integrated with kienta, but we do have plans to integrate it with kayanta soon, thanks venugopal.

A

Can you share a little bit about the metrics that you are capturing um so on the metrics that we are capturing? We are capturing uh different types of metric, I'm not sure which one you're particularly interested in, but we are looking at, of course, the load and scale growth that we have, because we need to make sure that we can handle all of those deployments.

A

We're also handling the typical reads metrics or uh we call that the requests, the errors, the availability, um the duration and the saturation. So those are the typical metrics that we are capturing for for the service and then, as we build new features, we also look at adoption rates.

A

uh Could you elaborate on the git ops triggering you're using for promotion thanks greg for that question, um so on the git ops triggers that we use for our promotion? What happens behind the scenes? um We have our git repo, where we put all of our deployment, manifests in and then that's where they go and um update the versions uh or if there are automated systems.

A

The automated systems update the versions on those manifests or if there are new targets, then those targets get added and then there needs to be a peer approval or a manager approval on those and then once it's approved, then we have a system that um delivers those manifests to our spinnaker environments and then spinnaker. We have a a listener or the echo a notifier for any changes into the s3 bucket.

A

So those manifest go into like an s3 bucket or gcp bucket, whatever the storage system for that uh cloud, wherever the spin occur, is and then it we use that as the trigger mechanism for the pipeline, so the pipeline gets triggered and then orchestrates the deployment across all the different environments.

A

um I thanks kolavo for your questions and I think the question is: do you do automated canary analysis? um We do not have automated canary analysis just yet, but we are planning to do that as part of our kayenta integration, so we'll be doing that soon.

A

William, the question is: what are deployment strategy ratio used with spinnaker blue gear, green canary a b testing, uh so those uh deployment strategies actually vary by business units? I don't have the the actual ratio on which ones um are being used by the different teams. Another team is actually doing doing that analysis. So sorry, I don't have those numbers.

A

um What about the progressive deploys? Do you do them thanks venugopal again for this question, so what what we do in our progressive deploys or how we handle? That is that we have our master pipeline and our master pipeline has different uh multiple stages or what we call stagger groups so think of it as the master pipeline gets triggered. It starts with, say the development stagger group and in that development stagger group we're targeting development uh regions right.

A

So all of our environments that are with environment type of development gets gets deployed to first and then they get deployed there. Tests are run again using our custom test framework and then once that's good, then we proceed to the next stage or the next stagger group of testing environments. So that's where we have our functional test.

A

Environments are performing performance, test environments and then once that's good, we move on to the stage, stagger group and then in that stage would be our production before production uh or similar to production, environment and then eventually, once that's good, then we go and move to the production.

A

Okay, uh faye um face question: sorry: can you elaborate more on your r back so uh so on the our back? We have set um admin roles. We have set.

A

Read read roles we have set our write permissions so for each of those pipelines we assign them those roles and permissions.

A

Those are spin occur out of the box capabilities um and then we we hook them up with our um ldap system and then depending on which team you're on then you actually, uh you actually have the rights to either view the pipelines or run the pipelines or execute the pipelines so executing the pipelines and writing on the pipelines. We only give those to the administrative user and to our service user for the master pipeline. So we do not allow anyone to run anything in production in the pre-prod spinnercare environment, we're more lenient.

A

So we allow people to go and play play with the spinnaker ui. So there they can. They can execute um pipelines manually.

A

All right thanks.

A

Right, I think we're done have a good one. Everyone bye.