Cloud Native Computing Foundation GitOpsCon NA 2022, 28 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: GitOps Syllabus: Working Our Way Through the Lesson Plan - Raptor Dzuricsko & Elisha Greenwald

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

GitOps Syllabus: Working Our Way Through the Lesson Plan - Raptor Dzuricsko & Elisha Greenwald, Teachers Pay Teachers

The Devtools platform team at Teachers Pay Teachers were looking to improve their current CICD platform. Having no prior knowledge of Gitops or ArgoCD, the small team decided to give it a go. They’re early on in their migration but the team has learnt a ton. In this talk, team members will share their experience with ArgoCD and some of the decisions and tradeoffs they chose. Some topics they’ll cover are: choosing a gitops implementation, auto vs manual sync, keeping the helm config in the app repo, commit strategies for staging and production, backwards compatibility and more.

A

uh Hello um Welcome to our talk, uh we're representing teachers, paid teachers, we're going to talk about how we implemented git Ops, um and you know what we learned along the way.

A

um So we're going to do is we're going to start with some introductions and some context um and then we're just going to go straight through the decisions that we had to make um kind of learning get UPS. As we went it'll take about 15 minutes, then we're going to wrap it up and finally, we'll we'll get some q, a Time uh introductions, um say hey. My name is Raptor uh really Raptor um been doing devopsy sort of stuff for like six years, mostly with education, Tech things and I'm. Just super excited to be here.

A

I don't know: uh do you want to introduce yourself, Alicia sure, Alicia.

B

I'm really excited to be here and I'm, also very happy that kubecon is here in Detroit, because I'm from Southfield Michigan, so I didn't have to travel, which was pretty awesome, also doing double tools, kind of stuff for about six years and yeah excited to be here.

A

Some stuff about us, so TPT, Teachers, Pay Teachers, that's the acronym we've got about 5 million resources available on just about every aspect of pre-k through 12 Ed. A lot of resources. Teachers are just generally know us and they usually have feelings about it generally positive. But if you have any teacher friends, definitely ask them: we've got about 250 employees and we're we're mostly remote, um and it's now like we're gonna start getting better and better stuff, uh so engineering context, um so we're about 100, devs deep um and we've got 101 Services.

A

um We got really lucky uh when we started uh TPT both of us like about a year ago, in that almost all of our services have a lot in common um using a similar node.js template.

A

um We pretty quickly ruled out tackling the monolith, um and we also have some front-end services that we were able to to work on too. Our infrastructure stack is um is is in kubernetes. We have three Cube clusters on eks um and we also have a bunch of serverless apps in there.

A

um We didn't really think that we'd be able to do the same, get up stuff with them, but it all kind of worked out well um nerd team topology is we're on dev tools, um but a lot of our like actual, like deeper Cloud knowledge, is on a different team. So we work closely with them on all of this.

A

um So what led us to get Ops um so you're here, so you probably know a thing or two about get Ops. um So what? But, what else? What brought us there is? um We wanted to make software delivery improvements, so starting a new company you're, not sure how you can help what you can do um and we noticed like three uh main things: come up: um the first one being reliability.

A

um So there were a couple of rollback bug issues on our Legacy system, which would be called um Houston, um so this Legacy system would call into Jenkins and Jenkins would do the actual work um and it just it didn't always work and developers were not happy about it. um This was for rollbacks and deployments generally.

A

um Another thing that we really care about is visibility um so because it was two systems. It was hard to know where the problem was, if you were a developer day to day, um and you didn't get a lot of user feedback from the first system. That said whether things were working or whether they weren't working and then the third thing that we really wanted to, and that has come up a couple times today already is- um is like different deployment, Styles or Canary deployments.

A

So um we really latched onto a couple different services for for doing that.

A

um So again, how can we improve software delivery? um It took us like a little while we had like some complicated products like charts, but we decided on using Argo CD.

A

um We had some experience with flux, 1 and flux 2, uh and it was just a little rocky um I. It does a lot of cool stuff and it taught us a lot of lessons and things we were doing um and we wanted to try something new, um particularly with the upgrading from flux, one to flux, 2. It was just it wasn't as smooth for us.

A

um We liked the Argo CD UI that was just kind of bundled into it, basically like that was nice and we weren't experts on kubernetes uh I still don't feel like an expert on kubernetes, and so just having that UI is helpful um and we just liked get Ops, uh and we liked it enough that now we're here, I, guess um cool. You want to talk about some decisions. Alicia thanks.

B

Raptor so after we landed on using rocd, we had a bunch of decisions to make. First of all, where do we make those get Ops commits to I think it was Jim from harness it's on there. Yet who was talking about that before and then what are in those good Ops commits so first to talk about like where we make those get Ops commits.

B

So like Raptor talked about, we have about a hundred services, or so each one in their own application, repo and in those application repos they have we're using Helms, so they have a Helm chart, plus the values files for both staging and production in in the application repos themselves. Once we spun up Argo CD, we had we created a separate application, manifest repo to store the Argo CD application manifesting and we were kind of thinking hey. It would be a good time even before adopting Argo CD we were thinking.

B

Maybe we should get those Helm charts and values files out of the application, repos and centralize them, because it's a pain to manage all the individual, um Helm charts and values files in each of the application. Repos and we were kind of like you know, how should we do this, but we also we just wanted to adopt Argo CD at this point and we're like, let's make it small incremental. Let's keep the existing structure that we have, so that's where we landed on keeping it in the app and service code repo.

B

Once we decided to put it there, we have to figure out what goes in those uh get Ops commits so for deployments. It's basically just bumping a Docker image for most of our deployments and we figured okay. That should be simple enough, like our get Ops commits. Could just do that. They'll update the docker image to the latest tag and then, like I mentioned, we have staging prod the way pre-argo CD. Before the way our deployments worked.

B

Is developers would manually or sometimes in some cases, was automated deployment to staging and then either an automated test or a manual test check it out then promote it to production. So we wanted to again continue the same flow like make as little change as soon as possible, and to that end we decided we'll have two git office commits it. We decided we'll go to automated staging deployments, always so no more manual staging.

B

So as soon as you merge your feature Branch to the main branch, we'll do a get Ops commit, bump staging and then have a pause for approval and then promote it to production after we implemented that way, we realized we actually had a couple more things that we needed to put into those get Ops commits. So we have these uh internal tools that make sure that anytime, you make any Helm chart changes or values files changes you bump the helm chart version too again. We could have just like changed those internal tools to avoid it.

B

But we're like you know what it's not such a bad idea to bump the home chart version every time we do this anyway, so we added that to the git Ops commit it's not not a big deal, we're not making those contents. The CI system is making those commits and, like you might have seen we're using Code fresh.

B

Also in addition to that, our CI system, like I, mentioned we do staging pause and then we do prod. So since we're making the git Ops commits to our main branch as soon as we did that staging commit. If we wanted to do a slack notification with, like the author of the feature like the commit author we'd have to look back and comment and we're like okay, we could definitely do that, but instead, why don't? We just add the commit author to the get Ops commit, so it was interesting.

B

That's what we did, and here you can kind of see where we got in our V2 and V2 was great things were working fine. We had the good slack notifications, things were working, but um developers were mostly happy, but we started getting some feedback that actually deployments were getting delayed and it's an interesting reason for that. So, prior to adopting argocd like I mentioned, somebody would merge their feature Branch to to main branch.

B

They do a staging deployment and then pause. Take a look at prod as soon as uh that, fellow would merge their feature to Main Branch. If somebody else wanted to do a deployment afterwards, they could already rebase their feature Branch with the latest code from the main branch. This is prior to Argo CD.

B

Now, once we adopted Argo CDN we're making those commits directly to the main branch in order for the developers to to get ready to get their feature branches ready and to rebase, they actually had to wait until all the get Ops commits were done. Then they could rebase and it wasn't trivial because, unfortunately, we're hoping to improve this, our CI times are a bit slow. Like probably a lot of people have this, but um therefore it was slowing down deployments.

B

So we we. um We realized that here's, the I just skipped again what we realized. We wanted to split out those get Ops commits to another repo and, like I mentioned before we have, we already had an Argo CD application manifest repo. So we were thinking again. Let's maybe reconsider, we'll move our values files, our charts out of um the application repos and move them into this Argo. Cd application manifest rebuild, but fines for not wanting to make too many changes and boil the ocean.

B

We also were um we're relying on the presence of the charts and values files in the application repos for another internal tool which spins up an ephemeral environment every commit, and we're like this is going to be a lot of work. If we want to do this, so we're like what else could we do? Is there any other way to do this and we stumbled across in your application manifest your Argo CD application, manifest you could actually specify uh home parameter overrides and we're like okay. This is interesting.

B

We can now make a get Ops commit and I'll show you what it looks like Tara application manifest, and it's kind of an interesting thing and in fact the documentation Argosy did Ox call it out and say, like don't use this in prod, and we were like a little bit hesitant because this is kind of a hacky way.

B

So if you could, if you could follow along here, we have our application repo with the values files and the charts in one repo, your application, specs, your Argo CD application specs in another repo, and in that other repo, we specify the overrides of the docker image to use. So the docker tag is being set in that in that Argo CDs application inspect we reached out like on slack I, guess to the cncf or no I I, don't know the Argo slag thing and ask them somebody's like yeah. It seems okay and it's been working.

B

Fine and it's great application owners are very happy. Things are working, fine and so far we're liking.

B

Raptor nope still me sorry, one more thing so now that we have these, um these changes to the Argo CD applications manifests. How do you apply those?

B

So one of the things that people may be familiar with is the idea of an app of apps, so you can have an Argos CD application that manages other Argo CD applications. So even without all the other stuff that I discussed, we were kind of thinking hey. Maybe we should do this app of app thing, but we're like.

C

B

Know yes, no, but here was a good use case for it, because we wanted to sync the application. The Argo CD applications themselves with the helm chart parameter overrides those need to be updated before you do a deployment, because if you don't update those before you do the deployment you don't have that override in and therefore you're getting the old code in. So you got to make sure that those are up to date, so we're like once we make that get Ops commit to change. Largo, CD application parameter from parameter overrides.

B

We need to make sure those are synced, so we were thinking. This is a great use case for app of apps. Let's try it out, but we quickly quickly realized that it's actually a single point of failure, because, if anything happens with any of those apps, the parameter overrides which we actually experienced. We had like a git commit that happened to be only numbers and Argo CD uh complained that it was like an integer when it should have been a string for the docker image tag.

B

I might be butchering this, but basically because of that the sync failed and once the sync failed, none of our other applications were getting updated and if they don't get updated, then all the deployments are using the old tag. So we decided actually we'll just do a cube apply and things are going well and now now dear after oh.

A

That was good, uh yes, uh so I get to talk about syncing I, really like syncing um I, hope you'll join me in in thinking about syncing. So this is the Argo CD page when you click the sync button. There's like a pop-up, that's like okay, here's what you're gonna do um and I am not always super great.

A

At reading uh everything, and so I, like I, saw this page I was like I, don't know what to do with this, um but I quickly realized that I needed to know what what these did in particular I'm, going to call out Auto create namespace, which is one of my favorite Flags um I, would totally encourage everyone to use and then the the big you're, a scarier one with um is prune here.

A

So let's talk about prune, maybe so initially we decided to trigger Argo CD sinks manually and our reasoning, for that was to get more feedback to users and more control, especially as we were gaining more comfort with the system.

A

um We had talked about doing Argo, CD notifications instead to handle that, but we we wanted to keep it consistent. It was less work to keep consistent with how we wanted CI and stuff to move over to to Argo products in the future too.

A

um So um we we decided to keep broon turned off for like the first couple services that we did um mostly out of fear, um but eventually we uh we dove into the prunes. So you can see these this delicious bowl of prunes and like once, you try, prunes you. You can't stop eating them if you're me, um so we turned it on.

A

It was something like cron jobs um which we needed to delete every time we deployed one of our apps for um and these con jobs had similar names that would conflict, and so when we had prune it worked and we didn't have fruit, it didn't work um and we now have prune on all the time, and it's just been fine, so so yeah go for prune.

A

um We did. We did decide to continue doing the Argo CD syncs manually. We were just pretty happy with the ability to get feedback and interpretive feedback from running Argo. Cd commands I feel like it also helped us learn. Argo CD better is like specifying the different, prudent Flags every time and knowing what you want, the different sync Flags.

A

um So how do we? How did we migrate? Like that's one of the fun things that like we as like a end user company can talk about um so we started to do one at a time pairing um we got lucky. um Our Cloud team was like here's a service. It's live in production if it breaks it.

A

It won't be that big a deal um and we were pretty happy with one at a time pairing for a while um in part because, like we were starting to learn, get ups, but our company didn't really understand gitups, uh and this was like one way to really build that education um and I think like one at a time, people just you know they respond more and we learn more and everyone's happier um after we got better at that.

A

We formalized a process for this, which turned into the self-service pipeline to add to pipelines uh which is kind of just a fancy way of saying, like there was one process that they could click a button. That teams could click a button on and migrate their services over to using git Ops, for um so we kind of saw how that goes for a while and if they could actually self-service it.

A

um Some of the reasoning on these decisions is that um when we migrated our first couple, service is over. We saw a couple bugs that we hadn't picked up on or guessed would exist, including like the malformed strings, one that Alicia mentioned. We found some staging database connections that didn't work, which is like too bad um and we're glad we found those.

A

uh There were a couple kubernetes settings that were just out in the wild that weren't actually doing anything and then with like with Argo, we figured out they weren't doing anything and we're just kind of nonsense settings for for them, so so that was nice, but yeah just being involved in figuring out all those different use cases um at about 30. We started thinking how to automate through the rest of them, because we remember we had like 100 services that we were trying to tackle um and we haven't gotten them all the way through.

A

But it's it's going pretty well. The automation is fine, um did people like it uh yeah the early adopters uh super loved it um again. Our Legacy system was okay. We needed to improve the things they saw. We improved the things they were, they were happy. There's a couple services that are less excited about. Moving over change is hard uh changing how you deliver software is always things go wrong, sometimes right, so you, so it's understandable.

A

We got it down to like less than 30 minutes for a service to deploy to run, CI, staging and production. We didn't get any major issues rolling out new Services related to get UPS, which is great. um We kind of thought there would be more, but it was fine.

A

um One thing that we we even expected to have issues on was um so how Argo works is it does a Helm template to like make files and like templateized files, and then it does a cube, apply on them and before we were doing Helm upgrade to manage our versions, so switching from Helm upgrade to like the helm, template and apply like that was like fine uh so like. If you are using Helm, upgrade right now and are worried about that, it'll it'll be okay.

A

um There was the roadblock, as mentioned before, um for GitHub commits in the same repos plus long CI times plus rebasing is is frustrating, but but yeah we solved that in the the way that Alicia brought up. uh So it was kind of a lot. Maybe uh I felt like it was a lot. Does anyone have any questions for us right now? Otherwise we can talk about other stuff, um I see one over here. uh Do you want to start.

B

D

Okay, um it's really interesting because we went through this exact thing a year ago and if you wouldn't mind, I would love to talk to you afterward, uh because I think uh there is a Next Step that you can take. uh That would be really it's. It's quite impressive. It's exactly the same path with the same repo version and a couple couple repos, um but the way forward, I would say with application sets and how you can divide a different thing and the values um so yeah I would love a chat afterwards.

D

That's it I will not bother everyone with him. That's okay! Thank you.

E

Hello, my question is: um can you talk a little bit more about where the bottlenecks were? If there were any? Regarding the scalability you mentioned, it took 30 minutes granted there were multiple environments and yeah like what did you notice? Where were the bottlenecks in order to deploy a lot more services.

B

Sir, um let me try to repeat the question so you're asking when we were migrating a bunch of the services. Where were the bottlenecks, and did you say something specific specifically to more.

E

More on the deployment s, not not migrations, just deploying your code to an environment like where did you notice any bottlenecks, or did you even notice that do.

B

You mean bottlenecks in terms of time.

E

Right yeah, so like how many deployments could you employed to production? You mentioned 30 minutes right. So if you had like 30 services, so what happens? If you do 100 services or 500 sure, and where would you see the bottlenecks.

B

Got it yeah thanks for the question.

B

Services, that's a lot of services. Where do you see the bottlenecks? It takes 30 minutes to deploy Etc so to try and address that um to be fair. We did not do a hundred services at once, so we never really experienced like a huge rollout. What we did is Mike Raptor talked about it at first. We did like one at a time and even now, once we're starting to automate this we're going to just put in a space in between them, so we're never going to.

B

Hopefully I mean if we start a new cluster or something like that we should run into it, but just to avoid the whole issue. It's like we don't need to do the migration all at once, so we just did it iteratively and just broke up broke the um deployment, the the migrations um apart, but one thing is to address. Maybe something else you're asking about is: we haven't had any issues with skill, yet with deployments currently so they're.

B

Obviously, application developers make changes at any time during the day and maybe five applications will be deployed at the same time, but we I don't think Raptor. You could correct me if I'm wrong, we haven't seen any scaling issues yet so that's encouraging I guess.

F

um Trying to understand if the developers or your users, basically their end-to-end workflow, so they make a PR with their application code right and they commit that to the main branch or whatever. And then you have this Argo CD repo I'm, going.

B

To stop you right there for one saying yeah: they make up PR and then they merge that the feature Branch to right.

F

Okay right yeah and then they, then they do a new PR on the Argo CD repo right. No.

B

F

B

So sorry and let me repeat, yeah.

F

Where, where are their human gates at is what I was trying to basically get to where the? Where are the human gates for them? Yeah sure should I take this.

B

Rapture, do you want to take it? Oh yeah go for it, okay, so um they obviously have to get PR approval and they have to pass ti. So GitHub enforces both of those things and then once they merge to main branch they next they themselves can approve promotion to production, so everything's automated they just actually through slack. We just have an approve button and that sends an approval to that that kicks off the next get Ops commit to change the prod tag and trigger a sink.

B

One thing we didn't really Raptor touched on this, but we're not auto syncing right now, so we're right.

F

So who's doing the auto sync.

B

Or who's doing the manual sync, the the CI system, it's triggering.

F

Okay, for some reason, thought it was you and it was like how does that happen in 30 minutes? Do you have someone sitting there all day, long, no.

B

So we're not the CIS we use code fresh and code fresh is triggering in ongoing. Sync.

D

A

Sure yeah I guess we we called it manual but we're not doing it. Yeah.

B

Thanks for the clarification good question, how are we doing on time.

A

uh Eight minutes.

C

So you mentioned you found early adopters, let's say the first guinea pigs for the exercise was there is process or you already knew like? Oh, we can always go to those five app teams because they're going to try the things before others try things.

B

Please go for it even.

A

B

um So the question was: how do we identify early adopters, basically like who volunteered so like Raptor mentioned, um we work closely with Cloud Ops, we're on the devtools team, but Cloud Ops volunteered a service, one of the fun things that I like that we did. Is we like again what Raptor mentioned? We have a legacy, um it's the deployment tool like he said it's called Houston.

B

We made that thing on uh Argo CD, so deploying the deployer, the old deployer on Argo CD, so we merged we, we started off with like our own apps, basically and then we're like hey um call to action. Anybody want to be an early adopter and, and generally people were feeling the pain of deployments and, like Raptor, called out some of the like. There was low visibility. Rollbacks were painful, so we're like we're trying to solve this like.

B

We think this will help who wants to volunteer to take this journey with us, and luckily there were a few volunteers, so we started with those and- and it was, it was definitely a manual pairing process with those early adopters, but then people, you know Word of Mouth hey. This is actually cool like, so that's how it worked.

A

Yeah that we definitely had to beg a little bit um just we have like a TL like a team, lead concept and, like some TLS were like yeah. Let's do this, let's adopt and- and some were like- uh maybe later, let's see what what other TLS do.

A

um So, just like slacking people from a list of uh a pod chart like who owns services, will you let us please use your service and make things better.

A

um If there's no more questions, we have other slides, we can talk about.

B

How are we doing time-wise? She was just with.

A

All right, yeah.

B

There's other cool stuff: if we have time go for it,.

A

um Cool I think there was a lightning talk that I missed about this and I'm kind of bummed about it, because one of the things that held us up when we like first started is like what does the Argo CD app repo structure look like, and we found like a couple of blog posts that had like ideas um and we weren't really committed to following any specific one of them. um So how we ended up doing is so.

A

We had three clusters, three eks clusters, and we just made a folder called apps and a folder called our back for each of those clusters.

A

um I, don't know if I worded that well, but perhaps you can see in this image, there's eksci and then the apps, and then these are some of our apps in there. The rbac stuff didn't really fit in into the Argo CD structure that we like saw initially, and so we just kind of added that later, uh but yeah not worth getting follow up on what the the right repo structure should be um I think like.

A

Ideally, we had used Helm files a little bit in the past and we kind of like that set up too, but with just three clusters. It's seemed like let's just keep moving but we'd love to talk more about this. If other people have better structures on that.

A

Another thing that kind of held us up initially was installing installing Argo CD Android rollouts across clusters. We had seen autopilot Argo autopilot, which I think was not yet fully released when we had we're installing things, but we really like it now and I. Think in hindsight we would have used something like that.

A

um What we did use was the Argo CD Helm chart and the Argo CD apps Helm charts to manage additional apps, and we we do manage our gocd through Argo CD. So that's been cool um chat up, stuff, I, think you've kind of been owning. The chat, UPS things.

B

um Yeah so we've we've been going closer and closer to being able to just use slack and just like we talked about the approvals through Slack um and putting in tons of metadata into there. So it's been pretty sweet.

A

Yeah uh yeah, so, let's wrap it up uh so where we're at now is um a lot of our our delivery. Stuff is working really nicely with CD, with Argo CD um and with get Ops we'd like to start um incorporating, Argo workflows. um Ci was the bigger uh Beast for us to tackle. So we got started with CD and delivery, improving that first things are pretty smooth.

A

um We didn't mention it, but the the diff checking on moving over to Argo cud is really nice and we're just happy using Argo CD. We still have lots to iterate on and learn, obviously, uh as I think you've seen from some of the slides of just like our confusion, but like overall, we feel good. We love good Ops companies been supportive and I. Think people have appreciated it and seen value in it.

A

um We're excited for the future.

B

B

Like you mentioned, we'd really like to uh continue the discussion. If everybody has ideas, we know we don't know a lot of things, so it's great great to be here. Thank you very much.