Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2018 (Seattle), 16 Dec 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Becoming Cloud Native Without Starting From Scratch - Marin Jankovski, GitLab

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Becoming Cloud Native Without Starting From Scratch - Marin Jankovski, GitLab

Full rewrite of a working application is a luxury most companies can't afford. In this session, we will talk about how running GitLab architecture was adapted to be closer to a regular cloud native application without having to rewrite the whole application and disrupt ongoing product development. As an additional requirement, architecture had to be modeled on what is required for installation of GitLab.com scale (SaaS) while also keeping in mind on-premises installation customers which will use the same set of tools. We will discuss what parts of the application we left out of Kubernetes, how we loosely decoupled previously integrated components and how and why are we enforcing some old behaviours. We will share our experiences with using de-facto standard for distributing applications (Helm), but some mis-steps and some good choices we think we made.

To Learn More: https://sched.co/GrRC

A

Thanks everyone for coming I want to set some expectations right away. I won't have any great drawings like Jordan and Taylor hat, and one thing I'm good at and that is being very nervous, I'm very nervous at the moment.

A

All right, the type the title of this talk is very engineering related becoming cloud native without starting from scratch, and if there is anything I would like you to take from this talk is an idea of how you can approach your own legacy applications and convert them into something that can go into kubernetes before we continue. I want to introduce myself really quickly. My name is marina, yankovsky and I'm, an engineering manager for the distribution and delivery teams.

A

Distribution team is responsible for get labs, installation methods, delivering packages to our users and, more recently, delivery team, which is responsible for github dot-coms, continues deployment. To give you an idea of what we'll talk about I'll. Introduce you to get up in a couple of sentences, give you a quick overview of the architecture of gate lab and then through two examples about application rewrite and how we handle zero. Downtime upgrades I'll try to guide you through some challenges it to encounter and finally share some lessons that we learned throughout this journey.

A

As quickly github is a popular platform for a complete DevOps lifecycle, it started as an open source project in 2011 for code management, source code management and when the company around the open-source project got started in 2012, more components got added, such as continuous integration and dependency scanning license management, and so on.

A

The initial idea of github as a source as a project was offering something for users to install on their own machines. But when the company got created around it, we started offering get lab comm as SAS.

A

So with with the experience that we started gaining with running github.com, we set out to scale out our whole application, the application which is in its core Ruby on Rails application with some state that needs to persist, such as the database and file system storage, for example.

A

When github.com started growing faster, we started reaching some limits in how quickly we could vertically scale and we started looking into. How can we get these components configured in such way that we could start scaling horizontally? So we started installing a monolithic package on multiple machines, configuring them in certain ways so that they can do certain operations such as, for example, serving API traffic's and wrote custom tooling to allow us to operate that whole system.

A

Now that sounds very much like what kubernetes is offering us now, but in 2014 we didn't have any of that at our disposal as kubernetes one as a predominant container scheduler we started looking into. How can we get github.com and with that our self-managed customers to kubernetes?

A

My team was naturally tasked with getting this done. So the first thing we looked at was what is cloud native. Even you can check out the cnc definition at the link out there, but just in short cloud native is an approach to building applications through creating containers through making decorative api's and not where the application is deployed, because the application can be deployed on public, private or hybrid clouds.

A

We added a bit more of these requirements. For example, what kubernetes is bringing you is a possibility of scaling way easier. But apart from that, we wanted to get that simplicity to our users so that the user can just install whatever gillip has to offer. So they don't have to deal with how they're going to scale out there get lab instance.

A

For example, one other requirement came from the company itself, where we couldn't have more than more than three engineers working on getting the whole application into this into this requirement, and we needed to ensure that we continue shipping our features and in to save time work on this.

A

The second, the third requirement we had was we had to be cloud platform agnostic, like I, said we don't know where our users are going to deploy, whether it's going to be a public private cloud or hybrid and even get robot comm had that requirement, because throughout our history we went from AWS to metal and then from metal to Azure and finally lending out on GCP, and if you learned anything to our maintenance and I'm sorry, maintenance and operating of gate level. Comm.

A

That is that, sooner or eight or later, our customers are going to come to us and ask us hey. How are you doing this? We need that because we grew and we need to scale things out, so get labelled, calm and self managed customers needed to use the same tooling and to just share some of the examples to cover some of these requirements. I'll talk about how we approach to rewriting the application and how we did zero downtime upgrades all right, I'm, not really good at memes, as you can see, but I promise.

A

There is only one more left in this whole talk when, when you get a task to change how you actually do your applications or how you develop your applications. The first thing that comes to mind is all right: let's rewrite everything right, like hacker news, said: let's use this greatest new language and we'll just use that, and everything is going to be great.

A

Obviously world does not really work like that. So we started looking at our history on how we actually approached the similar tasks when we previously had a similar challenge. So, for example, when we started when github the application started, we only used unicorn, which was serving all of the web traffic to our users and unicorn is a popular rack. Http application server commonly used in Ruby on Rails applications.

A

So, while the pages were the only things that it was serving, it was going fine as soon as we started, adding some more workload into it, for example, get operations via HTTP, so pushing pulling get cloning through HTTP the performance of the app server started, taking a hit and to make matters worse, we started adding serving user uploads arbitrary, large binary, LFS files and CIA artifacts. So in order to be able to maintain the performance of the application, we introduced another component and renamed it get lab workhorse.

A

It was a bit of a stab at unicorn when, when it comes to naming, but what we ended up doing with workhorse, was we isolated just one piece of work that it needed to do and we created basically a micro service.

A

The problem was that we have persistency to take care of, and workers was handling large files, so it needed to store files somewhere and offer it to unicorn to serve it further to the users and something that has been born as a micro service. All of a sudden was not really because it had the shared storage between two components.

A

Now you might be thinking all right. Well, this story is kind of simple. You remove the storage as a requirement, and then you have something that is pretty standard right, like one container that has one process, you get all these great benefits where you can upgrade the components independently and so on. But when we looked into what other options we had something started to crystallize for us completely.

A

Separating these components would mean a lot more engineering hours, and this is something that we obviously didn't have at our disposal having them separated in their own containers, but sharing the pod resources was one option that we could use and finally, the third option was logistic them together in one container we know that works and let's move on now. Each of them has its own pros and cons.

A

What we ended up, choosing was probably the least likely option here, and that is get those two components in one image and worry about the rest later, and the reason why we did this was we knew this actually works. We know that our application is fine when unicorn workers are at the same location with that. We also bought almost nine months of development time to deal with other challenges that we didn't even know existed at that time after nine months we went and did the second option where we share.

A

We are sharing the pod resources, because the rest of the engineering team managed to slowly change the behavior of application without actually dealing with a full rewrite.

A

But one item that really popped out during those nine months that we bought by choosing not to rewrite a full application was zero downtime upgrades. We historically, sadly have learned the hard way how to upgrade github.com without downtime. It took us a few tries, but we got it and the current process is something that is tried and tested. We know how it works. We roll out all of the versions to our VMs.

A

We then run online database migrations, the migrations that are just introducing new columns and tables, and then we are dealing with the rest of the persistence in the form of rolling the gotelli service which is dealing with file system. So it's dealing with get storage once literally service is rolled. Then we can deal with the rest of the services we restart them and they start the new versions, and once that is done, we finish with running long-running migrations.

A

So those are the post deployment migrations that ultimately change the contents of the database and from that moment on, there is basically no rollback to deal with and the last one restarting a couple of more services.

A

It's we could have that as a separate talk. How we learned to have to restore them is is a bit of a funny story.

A

We then went with that knowledge of how we're dealing with our application to look at what kubernetes has to offer us right. So when you actually update your pods, what happens is kubernetes tops routing all of the connections terminating pods. It sends the term signal to all of the containers in a pod, and then it waits for the termination rates period in seconds before it sends a kill signal.

A

Now that is fine for stateless serving stateless traffic, but when you have to deal with arbitrarily large files that work horse or unicorn will serve, you can't really wing it and guess what that value will be, and even, if you do even if we guess that on github.com, we can't send that to our customers. We can just tell them hey, put 60-seconds here. That really much depends on their workloads and the second thing was: how do we ensure the rollout like we said we need to do database migrations get early and so on.

A

We didn't really see an angle here that we would be able to use without much effort, so we didn't turn to our packaging tool when you install git lab with get log package, we're kind of used to being able to handle everything for you and how has that promise its as upgrading even the most complex kubernetes application. So we thought all right well, help me install and get ler will just roll itself up.

A

Everything is perfect, so it didn't really pan out that way, because helm doesn't really offer that much more, like you saw with with Jordan that he mentioned that helm. There is a grateful for handling some of the more simple installation processes, and what we also realized is that helm has a very much set installation order. This is a snippet from the source code of helm, so there is no way for you to really tweak how things are going to happen with helm.

A

We then had to find another option, and that was obviously an operator and we've been looking at the operator for a while. Since the very beginning of this journey, we were looking into where we can use an operator, an operator, actually promises managing complex stateful applications. So we were so excited about the operator that we wanted to use it everywhere. We were thinking about.

A

Let's use this for backup and restore let's automate, that for our users, the problem was, we didn't really have anything to back up and restore because our application was not really running there. Then we went and looked at. How can we actually initialize the database or do a database migrations, but then we handled the same thing with just like one-off job that triggers the migrations, so it didn't really feel like a good fit.

A

But now, when we ran into an upgrade process that is kind of complex, we thought well now it's the time for us to invest more into this. So we did what every engineer does we googled, and we found the first example where someone was playing with custom controllers. So we thought, let's use, that let's reuse that and see how far off you get.

A

We naively thought how about we just look at the changes in tags for all of the components that that are going to be introduced into into the cluster and just based on that start rolling out things in order, as previously described right away. We started hitting all of the problems there.

A

Basically, we wouldn't really knowing what what we what we were doing, so we didn't really know even how to track all of the events when, for example, one tag doesn't change and another does we didn't really know how to control the restart, and we didn't even think about handling config maps, changing secrets and similar. At that point, we knew we had to do something else, and that is calling for help.

A

So we went and hired jet stack, which, which is a kubernetes consultancy shop but, more importantly, they have a very popular ahem thumb chart that is used widely by the community called cert manager, and they also have an operator in there, and with that experience we started working together. The first thing they told us was how about you use an actual existing tool instead of writing everything on your own and guessing.

A

So we looked into operator SDK and the first thing that kind of draw our attention was a huge item in the readme saying this is alpha, don't use in production. So with that we were like, let's not use, that they suggested queue builder. We check the readme, doesn't have the disclaimer, ok good enough, let's get started so.

A

We ran into the problem as soon as we transferred all of the code from our custom controller into the operator. How do we actually get that process running between helm and the operator to handle the whole lifecycle of the installation when you're doing helm, install you can use the pre-installed hook and install the CR D and then operator will be able to handle the rest of the process?

A

But when you are upgrading there is not really a good timing for your installation hook to make a change on the operator before it starts changing the rest of the process, because you also need to version the operator right at that point, just like told us, you know you might want to consider working from scratch right like rewrite everything you did in the past ten months, because, ideally, you want your home chart to bring in only your operator and your operator handling everything else.

A

How I'm, just being there a vessel or the operator, we kind of thought and said. Well, no, we're not gonna. Do that so help us we're paying you money here and we started looking at options. So we found there is a way for us to share the responsibility between the helm and the operator. How we'll install the new resources in the cluster. But then the operator will take care of the rest and I'll share some of the examples of how we approach this.

A

So, for example, in the helm, charts that we have, we set the spec for the deployment templates to be paused, so helm will install, will trigger a deployment, but that deployment will be paused for the operator to manipulate when it needs to. So it can actually unpause the deployment in in order it needs to, and also individually right like, it doesn't have to be a certain sequence similar for the stateful set. So if kubernetes is going to roll the pod when the ordinal is bigger or equal than the rolling update partition value.

A

So if we say that we take the value of aspect replicas from the home chart, we can then just simply tell operator right. Just increase this by one and the status out will be rolled by starting parallelism to zero job will be automatically paused in the cluster, and the only thing operator needs to do is change that value and job will start up. And finally, the daemon set rolling strategy. So if the max and available value of.

A

I wish I had my notes here was zero. You will be able to just be changing that value, have operator and do operations on the diamond set as well. Now, after all of that, our final obstacle was: how do we ensure that the CRD is correctly installed at the first installation? We enabled the operator, but we also say: ok, user enable or bootstrap the Ciardi first before it starts taking all of the actions and then at upgrade. You don't need to do anything else, basically, because bootstrap in bootstrapping is not needed anymore.

A

This is a two step process that we need to get down to one step, but for now it's it's good enough.

A

A couple of lessons that we learned throughout this process, I think just lesson number one is the most obvious obvious for everyone right like kubernetes moves really fast and github has a similar release cycle, which is once per month. What is kind of different for for kubernetes there is, the community is way bigger and things are changing way more rapidly. So a couple of times during this development period we had situations where, basically, the kubernetes community would throw a spanner in our work.

A

So in the middle of this whole process, they introduced our back and that made us completely stumped. But one other thing that happened during the hold is this whole development. Was we kept ready, reading the source code of kubernetes very often to try to understand what's happening, because the dogs would not really follow the changes that closely or wouldn't be enough or wouldn't be detailed? Enough?

A

Second lesson that we learned was that helm is great for organizing configure, where it's not great, is when your configuration has like multiple levels or where you have like multiple sub charts in a chart. So when it comes to very complex charts, helm has some difficulties because of the ordering, because of various other reasons there and tiller is very challenging. I was kind of hoping that helm tree will be announced this morning. Sadly, that didn't happen.

A

We had couple of problems where we, we would have unscheduled pods, and we would try to true and you throw a new deployment using helm and tiller pod would just crash and wouldn't be able to boot up again, and we would be basically stuck and to say that error handling is challenging is a bit of an understatement, because I can remember the number of times have you seen this error upgrade failed, yet lab has no deployed releases only for us to realize that the release did happen is just that helm is not tracking it anymore properly, and sometimes that meant we needed to completely remove the release in order to reinstall again.

A

Another lesson was when it comes to slicing our application. We made a right choice of not going down the rabbit hole of just rewriting everything, because we got enough challenges with the the new tools that we didn't really know how to properly use, and one other thing that came up during this whole process was. We definitely needed to hire experts way sooner, having someone whose sole job is to follow a tool that is changing this rapidly or tools in this case, because helm is also changing very rapidly is a good thing.

A

So, if you don't have that in-house knowledge, it's really useful to hire someone who does and finally I think, like with everything in life timing, is everything we kind of started relatively early on working on this, given that operator was still not really fully developed during 2017, like things were still changing rapidly there so making sure that you know that you need to to work on your application and want to work on. It is really important.

A

Thank you. Well, I think I survived I.

A

Think we have a couple of more minutes, so are there any questions.

A

All right well, thank you very much again and hope you enjoyed the rest of your day.