GitLab Delivery Team, 14 Oct 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Delivery: discussing auto-deploy for GitLab components

Description

Alessio and Grzegorz discussing about https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/99 and https://gitlab.com/gitlab-org/gitlab/issues/29537

A

Okay regard, so we are discussing the auto deployment of components. Okay, so that this is the point of what I'm going to to think about. Here is this.

A

Today we deploy omnibus omnibus bills, so it's kind of in its our artifact. How artifact is an omnibus bills? Okay, so the the problem now is that the rails code base is updated, weekly more than weekly, because during the week you can fix P 1 s, 1 items, so you can even do more than weekly deployment. This is something okay.

A

Then we have the satellite project so component satellite project, whatever you like, except from get early, which is a special case, the other ones are really they moves really slowly, ok, so it makes no sense to do the weekly thing for those components, because maybe you're just creating branches every week and nothing changes or they're few changes. So the idea is that those components are stable enough to pick a green master as the current working version.

A

So the idea is that when you do picking two of the deploy or when you cut a new auto deploy branches, you also search for the latest screen master on those components and you'll select that sha as the new version of the components.

B

Okay, I have okay.

A

So that's the thing now: what's the problem here: did it say that what we discussed in the issues is how can we ensure that everything is compatible? Okay, so this is my question to the question: is: how are we ensuring this now we're doing manual QA, because it's just it's up to the maintainer of the component to make sure that when they upgrade the component version in github repository hmm I know the death version of gitlab can run with that component. But this is not enough.

A

That's why I'm telling you that when we work on pages workers- and usually we keep backward compatibility because you don't know what happens during a deploy, the workers usually restarts faster than unicorns, for instance, so omnibus kicks in replaced, binaries and start sending signals. Okay, so you have internet connection, incoming with with the new workers that may reach an old unicorn, so new proc simplement, a tional pulled rates codebase. So this is already the case, even if we ship everything together.

A

So when we change something in pages workers, usually we make sure it can at least handle previous version of the code. I, don't.

B

Agree with that, because, like developers know that in order to pump a version of pages, for example, you need to go to get lab and edit pages version 5 and I've seen that, like multiple times that we are introducing incompatible changes in satellite, we know that we need to go to get lab project to change something there or perhaps on some other satellite, and only when you know the change has been merged in get loud. Then we know we can borrow a version of get LePage's there and but.

A

This is this is not the case, I mean when you have two components. You still have to select the priorities, so there's there is a change which is backward compatible and one that can't be. Do you agree with this? So one of the two components can be backward compatible, but then the other one will implement something that we change the behavior or the other one.

A

So you can say that you have something that is backward compatible and suddenly you can drive the options so based on version of the second component, a new or the new old code path in the other activates. Okay, imagine you're, adding our parameters to our query or whatever your new routes, things like that. So we have multi project dependencies in merge request.

A

So the thing is that, as an engineer, you select which one is the backward compatible and which one is not, and so you made the second the and comparable change related, but yeah and command level change is the as a prerequisite for the quiz ID, the the backward compatible implementation, so that it cannot be married until the other one again, and this gives you the thing then in rails, you also have Fisher flags, so you can even build on top of that.

A

So you can still have backward compatible again. But the point is that as a reviewer and maintainer in the civilized projects or in general and rails, you should aim to backward compatible, and this is regardless of the outer deployment of components or not, because we can't hope for the best and during github.com deployment- and you there's a moment when you don't know the state of the fleet or which version is running where. So. This is something like the code and on schema and +1. So.

B

My point is that I understand that it like is interesting solution, but my point is that without versioning you sometimes can't implement backwards. Compatibility in a component that you are developing I do agree that we need to develop components in a way that they are backwards compatible, but if you do not know which version of a satellite you are dealing with or which version of rails coordinator you are dealing with like you can't implement this conditional somewhere. Oh my god, convertible I agree.

A

But we did this in the runner in the past. It's not about version it's about capabilities, so you need a way for the company. This is always a component that is driving an interaction. Ok, so let's say something comes in from workers and workers control some behaviors and it can change the route that it proxies to or the runners chance we can pick a job from get lab and get lab can change the GS on the representation of the runner triggering a new order. Note behavior! So what? Once you have this you?

A

You have conditionals, so it's not about knowing that this is rails version. Xyz. These versions means nothing. You want to know that this give lab rails can handle Visser. It's y&z.

B

It's not sometimes about handling feature, but how you handle them, it's more about like it's more low-level, it's more about JSON schema as that might be comfortable or not, and I, don't know how to explain that. It's.

A

B

But what I, what I'm trying to say is that I feel like without versioning? Sooner or later, we will hit a wall with how to implement backwards. Composite compatibility- and my word is that relying on good luck way on end-to-end tests is completely. You know useless strategy, because the end-to-end tests are at the top of testing pyramid what they cover, like, maybe very many more and like having a team working on end-to-end tests. To add more and more and more like this is not a you know, good way to end. It works well.

A

Let me tell you an example, because there is a problem here, that we have component right now, that just I would say they works by chance. Ok, so there is no in real tests between pages and rails interactions. We rely on the fact that both components are testing the interface, but there's no common point where we ensure that they are testing. The same interface has.

B

You ever heard about pact project.

A

The fact is, the thing that builds images for the one from ashpod.

B

Pact is testing framework that makes it possible to test components independently, but they are still like you test, one component with some interface, the new test, second component with some interface. But whenever you test one already another, there is a parked specification being generated, and then this you can you test the specification against another component. Okay,.

A

So you can yeah it works for interfaces. If you have to add timing or things like that can be better but yeah. It's it's definitely more than what we have and something I should look for, but thing that a the point was that I want me is that today there's no real testing for this okay. So what happens? Is that the maintainer ensure the quality and if it fails, stability.

B

A

Yeah, the cop will not any compatibility, so they ensure the. We are switching version in the right moment. The same where thing is just can help her just tells you that you break something that you're upgrading the manager version. So you know that you changed something. You should pay more attention to it, but this is just moving the problem from novice review to release, but in the realm isn't there, because when you do review, you have to understand that there is a breaking.

A

This is breaking the API and then you have to remind yourself to prepare a build that great major versions and do everything to make sure things doesn't break. So the problem is still there.

A

What I'm telling here is that there's a null problem that we have today and that is fissures in workers pages get early most like Italy, because Italy as is tested in both side because Italy is is used for testing literal rails, requires manual handling for this and there's no need, if you can automate this part so that when something is merged is ready, then you can pack this into the current rails. Guitar braids and try it and if it fails it fails in the same way as it fails with the maintainer doing the easy lifting.

A

So you have to roll back. I mean you know. Rolling back is creating a new release that change the version back to networking. Why I'm.

B

Not sure, because, first of all, I think that using QA to check if something fails or not, is the effort like it's work that needs to be done on compute nodes, whereas if you have a server, you can just compare some strings and you know that there's no.

A

Money y'all got the point, but this is not thing, because someone has to make sure that this is really comparable.

B

Experiment tests, but before we run tests we have preliminary data about whether it's compatible or not I'm, not saying that we should not run smoke tests, I'm, saying that before we run smoke tests we need to resolve dependencies and I get it that it might work right now that, in the moment where we are, you know having omni booze and creating an artifact yeah.

A

B

I think that the only thing that going to help us eventually move towards CD is that we need to have a way to run like 10 components of gitlab in the production, with three different versions, or for let's say that every component can leave with four different versions on a production. So let's say that we have potentially 40 different versions of software components running in production. How do resolve dependencies and ruled requests to proper versions without having versions.

A

Yeah but again I do understand your point. It it's definitely something we should look for when we move to independent components so that they only effort in health insurance and things like that, because you can imagine someone pumping literally version and maybe it will get away, should work if it's a new one, because it's backward compatible, but you don't know what happens. Okay, so I do agree. What I'm telling here is that we still have to keep in mind that we ship software to our customer in a packages form.

A

So the idea is that we as soon as we make this as it will independently deployable, you need something that can at runtime tells you if there are uncomfortability. Yes, this is the point of checking there.

B

Is of course the runtime, you know compatibility check and I think that in micro-services word we basically achieve that by routing nodes that can understand the version headers analyzer. But what I'm trying to say is that we can follow the same approach that you are proposing right now and still have versions semantic versioning files, so that first you see if a package that we are trying to build from masters, let's say latest branches.

B

A

B

All those versions- and you see if you know it seems like comfortable compared to what we had before and that's the way we when you could move forward like with what we are having now or we are planning to have now. But this is an information that you might need to have in order to resolve dependencies when building the artifact and then we can, of course, take the artifact that we've built and tested, and you know, do everything we want yeah.

A

Point is that if you merge something that breaks your compatibility, then you are blocking all the future development until someone declares that these things is now compatible.

A

So this is the point that is hard to solve, so it is it better to stop that merger is from being merge. I need.

B

Everything we are talking about not compatible nodes, but the component. My worry is that what we are trying to model right now is that compatibility goes one direction. What community goes two directions? For example, you can introduce a new feature, which means that this component is still backwards compatible, but you need to have the new feature consumer and, if it does not have it like the feature is not compatible. You know what I mean yeah.

A

Okay, but let's avoid that, if you add something new, it's yeah, it's backward compatible by definition, because you don't change the old one. If you start, if you introduce a consumer in the system that relies on this new feature, then I agree. We we have. We have to keep this working. So if you, if you don't great and no it's something like that bit, clients breaks so.

B

This, like the the current, the current example of pages, is really interesting because we want to implementing new features on gridlock, who will have the API new API that get up good, so the get operation is always backwards, compatible with the old version of pages, because it's not using the new feature, but once you deploy a new version of pages, it requires the newest gitlab. So, okay, so there is bug compatibility problem here and that's the opposite direction. If you introduce something new in pages.

B

So my point is that compatibility is bi-directional, and my bet is that without having versioning you not able to be so be directional compatibility.

A

Yes, I do understand, but because the Europe suggesting to that save ride the version and enough in a file for instance. So you keep done, you don't talk it because you you continuously deploy for to get merged. So this is I mean if I understood correctly. So, instead of checking the tag, you're checking the content of this file, which is in the repo and tells you they removed a minute of minor because could be something like my major minor yeah they're. The best version.

B

A

B

The version file is better than Clark because it comes at the same time with merging and that.

A

B

Condition with tagging but yeah. So what is your question then? Yeah.

A

The point is that what I'm trying to say is this: that I look at the pages example. Ok, so I do expect the new pages demon to be backward compatible if rails uploads the content in the file systems. Okay. So in the works now no api's just decide people jobs, girls, keep saying boom great, create the content, so many pages will pick up the content and start serving it.

A

So if you upgrade pages before creating rails to the new API, no problems that nobody will use the new features and pages and will use to work because rails, the sidekick notes just deployed the artifacts anyplace.

A

Once you know that the pages installation are deployed, you can activate the new features in rails. So I, don't I, don't get the back. The the bi-directional part in this example, because you expect people to run let's say 10 years or 10 years. It was it one years old version of pages and upgrade only rails and then breaking stuff.

A

B

The example is that we have movie during the clock rates, it exposes a new idea and we deploy that to get up there. In pages, we need to introduce the client that is going to ask. We get lap about some configuration through API instead of traversing, the.

A

B

So it means that once we introduce new functionality to little pages, it needs to have access to new rails endpoint.

B

And Bitto pages in is still backwards compatible, but it means that the previous version, so we are adding a new version of get appraised and we are just adding hanging feature. We are not breaking contracts with clients, then you know once we introduce a new feature. The old version of github is no longer compatible with repeaters right. So, and my point is that situation like that can happen in two directions and I feel like it's very difficult to resolve. That depends. So if.

A

I'm thinking about someone managing the installation from sources I mean because the point is, if you control the release cycle, how can this happen.

B

How does this can happen? So we need to deploy a new version of github somehow and once we do that we never like do everything at once. So always the previous version is going to exist and the new version is going to exist and then you need to somehow coordinate the deployment of pages and also the old version of pages exists and the new. So at the same time we have four versions and half of them is not compatible with the another half yeah.

A

But because this is a problem that we already have today.

B

Not really because we managed, because we have the version pages version in get lock rails.

A

Yeah, but did on the 22nd when you release and you install the new deployment you have a fleet of machine, half of them upgraded half of them not on the same fleet of Machine binaries process can be restarted, not do the same order. So it's.

B

A

B

I know happens, but.

A

The time frame is so small that is not causing problems. No, it is because forget or calm it's a problem by the way. This is super me. So what I'm telling you is that we know is not only up checking the version. It's making sure that in that China spot things still works, and we are already doing this for pages workers and not individually. So that is what I'm telling you that, because this already exists, it's up to us as engineer to figure out, which is the client driving the interaction. So what are the?

A

What are the requirement in this period of time that can mix expanded now, but it still happened and make sure that we do things in step so that with this deployment, new features comes in, but the other one is not consuming. So, let's wait a week one day whatever it is, then you know that everything the fleet is upgraded and is stable is functional. That means back to you trigger feature flags, deploy a new version. Do what what you think it is yeah.

B

So I think it's basically difficult and its heart is really.

A

Really really hard and.

B

My gut feeling is that it's more important to have version not to actually be able to build the art, omnibus artifact, but to carry the version in all the headers of all the interaction between components so that you can inspect in or out one called uncomfortable version, and without that like it's might be.

B

So what I'm afraid of is that the change we are making right now removes the need of having versioning, whereas it's super important to have it, but perhaps not to build package, but to have this version in all the headers and all the interactions, so that the you knew that we are talking to incompatible version or that you are an incompatible version and that you need to respond something else, because currently we assume that get lab coordinator is always a server, whereas all the satellites are always a clients.

B

What we know is not necessarily true and.

A

It's dependent on me and the features they are working on. You have to figure out which is driving the.

B

A

B

Like super interesting discussion, I need to finish now because my sons to be from the kindergarten, but that's super interesting discussion and I'm happy that I could chat with you about that. So I still think that, basically, what delivery team is going in right now is probably the mission-critical stuff, probably one of most important things we can. So we never have enough of discussions like that.

A

B

Me it's super important. Thank you. We appreciate that you could spend some time.

A

B