GitLab Configure Stage, 14 Oct 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Live session: Copy the Helm stable repository for GitLab

Description

https://gitlab.com/gitlab-org/cluster-integration/helm-stable-archive

Configure team copies the Helm stable repository for GitLab purposes

A

Okay and thanks for joining everyone, so um today's problem that we're going to solve is this issue here, um according to the helm, chance duplication timeline, uh the helm, stable repo is going to be deleted um in about three weeks time.

A

So when I mean the stay variable, is this this repo here, kubernetes charts, dash dot storage, given s apis, and this hosts the helm, charts that gitlab uses so, namely all of this. So what we're gonna do today is take a copy of it, so that and then so that we can point it up to the new location.

A

um I have here a giraffe emma and I'm gonna try and do the first two steps set up a new project instead of the home repo.

A

So let me find a link to the project um which is here.

A

So I've already created the project in here and it's got a few scripts um and I I've already done the first two steps, which is.

A

The download so down it this downloads, this file here, which is the index. Let's have a look so.

A

A

And I've committed this file, so you can see like this. This is a helm rifle uh index yammer, it's a yellow with full of entries with different charts. Now we don't need all the charts, we'll just take a take a sample order, so the second step is, is that is exactly that: it's going to filter it for only the four charts that we need. So I've already done that as well. Yesterday,.

A

This is the result, so it's identical to this one, except that it's only got four entries, which is fluently: uh prometheus uh engineering, grass and postgres. So those are the four charts, so it's much smaller. um So the third step um is then downloading the actual charts itself based on the metadata.

A

So you can see each of these chart versions have this url here, which is charts uh as cg zip files, so I've already created a script to do that which is called download files. So this script here is going to find all those url files from the filter index and then w get it 10 at a time. So let's run that.

A

This is the same as currently on master and what I'm going to do is read the readme. It says, run scripts download files in a directory where we want to save the charts for this project, we'll create a public chart directory. Okay. So let's create a public chart. Structure already exists, that's good! So we'll go to the public charts directory as nothing here then we'll run the scripts files.

A

Okay, good: this will download.

A

uh Maybe 500 charts, I think I mean four charts and 500 different versions of them, so yeah uh everything clear so far. I.

A

Think we need all these versions to be stored afterwards. um I'm just gonna. Ask that yeah, not necessarily um I mean, but I can do someone can do all the work to find all the versions that we have previously used. So I think the longest one will be nginx ingress where we have used.

A

I don't know, maybe 10 different versions. So far, so it's hard to say.

A

So one that's one approach. We can store that, but uh based on my testing, this shouldn't take too much time uh too much space. It's only like 43 megabytes or something so.

A

Yeah seems pretty small yeah for four charts there. It is but uh there's hundreds of charts, but we're not we're not gonna archive the whole thing.

B

And are you planning on committing this to the repository or are you gonna run the same script in galaxy high as a pages project.

C

uh We have to commit this to the repository uh and then serve it directly. As the gilad pages thing.

A

uh We can I mean if it wasn't going away, we can always just run it as gilapsia and download it over and over again. The problem is that, after the 13th of november, there will be nothing to download from so glapcia is not going to work.

A

um I did right that makes.

B

A

Yeah I did consider using gillette ci to download and commit it as well, but it doesn't really prove anything really because, like you still have to trust that the script that I wrote makes sense so.

A

Yeah and unfortunately, all these things, like all these tgcs, um no one has probably shared sums for this previously, so it's impossible to prove that you know or understand these things.

C

That we're downloading or what I'm going to upload is correct, so hard to say. Are they not? Are they not in the index? No there's no shells in the index I mean there's no I'll say they are in the index, but there's no.

A

Verification to say that this thing that I'm downloading has not been tampered with.

B

Yeah, so there's no checksum in the index is what I was asking: no, no, it doesn't okay.

A

All it does, is the index only says when it was published. This is the url uh here are the chart? Maintainers uh helm has an option for that kind of stuff, but hey as with many package managers, it's optional.

C

And people never opted in so.

B

Yeah yeah, we had a bug in auto devops, where, like we kept republishing the same tag of auto deploy for a really long time, so like .41 or something was published many many times, yep.

A

Yep home allows you to do that.

A

I mean like ruby. Gems has maybe the same problem as well, but not as bad, but rubygems doesn't have like checksums either okay, so it's done so. um The reason why I create a public charts thing is because uh we're going to use gillette pages so get up pages subs serves directly from um public. So that's what that's? What we're going to do so now that it's downloaded I'm going to just add them.

A

I don't know whether it is right or not like I don't know whether the files have been tampered with, but I'm gonna just trust that kubernetes storage is serving the right things. That's can't ask for much more than that. Unfortunately,.

A

So we can commit. Oh let's, let's do a quick look to see. I haven't it, doesn't look anything other.

A

Ordinary uh-huh, so fluently ingress postgres for media okay looks good so commit this commit result of running grips.

A

Just push we do, I'm just committing straight the master. If you don't mind.

A

Cool it's done so we we have that um so the last piece that we need, uh there's two more pieces that we need to make this a proper helm. Reaper is uh public, slash index.yaml4, so we have this okay. So let's write a quick script to change this one to the final destination.

C

Because that's going to be wrong with it, you see there's a digest there. That's not the checksum of the chart.

A

Oh, you might be right. What is the.

A

A

Let's find fluid 2.5.1 and see what it is um sum minus, a two: five, six shots, one.

A

A

Right, it is oh, my god it works cool, it has the digest brilliant.

A

Okay, someone someone should, after this uh check, that the the index that I commit matches the index, that's currently on um helm, official helm, charts now like um this one slash index and as long as the dodges matches, then we'll do that. So what I'll do after this is I'll write, something to verify that hashes match for each of those committed charts.

A

I don't know where the helm would do it faster, if not, but hopefully, but it's probably good to check it either way cool thanks for that.

A

uh Okay, so assuming those are right, we can just use the filter index and change that to the path for github pages and the powerful gitlab pages.

A

Is that slash charts because we have it in a chat, sub directory so just create this script, can't say: um publish.

A

Index name yeah sure why not um and we want slash charts. Is it slash, charts, yes, public, slash, chats and.

A

Then copy this save this uh descriptor.

A

A

Set works differently for different things, oh well, just so to assume mac. For now,.

A

B

uh You should use a different separator for yours. uh You can use like a vertical slash there, yeah a vertical line here or or any character really.

A

I can do it like that.

B

Yeah yeah and then at the end you can put a g or you know. I don't think you need a g actually because it's not going to be multiple matches per line. A.

A

Line yeah magic. I didn't know that cool so we'll just do that. We say.

A

A

A

And then let's run that and see what happens and nothing happens, oh because I changed it.

A

ah No, I don't want that. I actually wanted the output.

A

um So, let's not have that I will just stream it out and stay okay good, so it's streamed out and then it's going to be helm, stable archive chat. So that looks about right. Okay,.

A

So we'll save that and then we just step for run oops.

A

A

So that's what we're doing and then let's get to public slash index and then we'll commit it as well.

A

uh That's done so. Let's do a diff of that.

A

A

Alright yep that looks about right. It's all the same.

A

It's the same replacement over and over again.

A

A

I'll commit that and read me.

A

A

Push that um and then we'll commit the.

A

A

A

A

Take a look at our project and look at the public directory.

A

uh We should have index and then the index demo would hopefully be pointing at the charts.

A

Yeah I'll be pointing at the charts okay, so um the last thing we need is to publish this as uh get up pages, so we can probably just copy what we have from here.

D

Wouldn't it be better to have an index.html that describes how someone can say that.

A

uh Yep yep, we can do that. We can do that after.

A

A

um Was the last pages template.

A

Then why it says nothing to do.

A

I just published a public diary as an artifact and then we're done.

A

A

A

Rt, I just add two issues before I.

A

A

A

A

Two okay, um so.

A

Look at this yeah cd, uh it's! Oh! It's already done.

A

And then the pages supply is done as well, so.

A

A

Yep, so let me just find the pages wait. Oh there we go. That's the.

A

D

Do you have anything to show there? Oh yeah.

A

Yeah index demo- okay, so it's published, already looks good, so we can test this out actually.

A

A

Home, that's my previous test.

A

um Repo ad, uh it's.

A

A

Oh okay, why is it jason.

A

Oh he's using json strings. Hey trust me.

A

What's the difference here, so that's the first.

A

One are you kidding me, okay,.

A

Yeah it composite because of that yeah. I noticed this different yesterday difference yesterday, where all the created app is having that problem, so I guess we could fix it for now and then I'll do the scripts I'll update the scripts later or not. I don't know any thoughts. Are you okay with me fixing it in line.

A

Seems fine with me okay, so what I'll do is, um let's fix the actual published one directly so that we can.

A

Resolve this so um space, z c: oh god,.

A

G: okay, it's only fixing the timestamps and, from other.

A

Chat just quickly scanning shout out. If you see anything.

A

A

Okay, let's add that and commit.

D

D

So what caused this difference to have this space in the z and the other.

A

D

A

So what happened is that um I was it's filter index thing. um I was using ruby to filter so.

A

Yeah, so I was loading up the yamo as the ruby hash and then splitting it and then slicing it out to only slice out the four charts that we need and then splitting up the yammer again. So I think this step here is not splitting up the same um timestamp format. That's why so yeah.

D

A

See that now and see if it works um in in a previous repo, when I was testing this, I manually used them to take things out, which obviously worked. I guess.

A

Okay, so be publishing. Now I.

A

Hope yep it's publishing now.

A

That's a weird styling.

A

And then we can do is do.

A

A

So just take the common lines between the index that we have and the index that helm had. uh It should be identical. I think so. We take a look at.

A

A

Lines, oh still, not identical,.

A

The dates are still not identical: oh yeah, the urls are not the same. Obviously yep. That makes sense.

A

Oh no yeah, let's figure out. What's the difference.

D

Was the creator that gone.

A

It's not gone, it's it's! It's saying that um so what I did was um I took the common lines between the index and the heroes index and saved it to a dot, and I combat the common lines with the index that we had so it's showing that it's showing all the lines that are the same.

A

If it's not the same, it won't be there. So it's basically saying that this is different.

A

A

So if I search for this string yeah, it doesn't exist very interesting, blue and e.

A

What's the difference.

A

Oh, it's got a t in front of it as well. Okay, maybe we can fix this, maybe maybe maybe it's valid. I don't know, um let's see if it's published or not, it's published, okay, good and.

A

We'll refresh this, it's got a z there, all right, let's, let's try and add the helm, let's see if it works nope, it still doesn't like it. It wants the tea in there.

A

I need to come out the rejects to fix this.

C

D

Maybe look for number number number number dash number number dash number number space and replace for the same thing yeah without and without the space and the team thing.

A

Yeah so four numbers two numbers, two numbers.

A

A

How do you do that.

A

A

ah I just just do it in a set said.

A

D

Does dash need to be skipped for any reason script yeah. Could it mean anything on rejects? Oh yeah,.

A

No, it shouldn't be.

C

A

Capture the group and then we say, slash 1t,.

A

H1 is not defined.

A

It expects me to do that.

A

Doesn't like it.

A

Maybe just do class and see what happens so I don't think there's anything else that has that right.

A

A

All right everyone's gone, some of you um yeah. I need to find out what's going on.

A

Here, oh yeah, that's right!.

A

As well, the square.

A

Brackets is so.

A

C

A

Okay, minus e right all right, all right gotcha. So if we have that- and I know you go minus.

A

Let's do this, I need to.

A

A

Okay, good, so, let's put what I had in here into the rate me as well, so.

A

A

And we also had to fix it away from the space z.

A

A

Yes, what did I do?

A

I didn't do anything.

A

Okay, cool looks good, so edit.

A

A

Let's see if they actually appear now, if I do this, is it exactly the same?

A

No, it's not exactly the same. So what else?

A

Oh sorry replace it.

A

Okay, yep: it's there; okay, so.

A

A

So now we had a com thing.

A

Did I do that right? No, it's call minus the public index scripts.

A

And the common lines.

A

And then we can see that it's all the same set for urls.

A

Okay, um oh here is different for some reason.

A

Oh, that's postgresql.

A

It's prometheus looks like the ordering is.

A

Different yeah, it looks like prometheus and those guys that's a different ordering.

A

Okay, I need to look into this. Let's, let's publish this and see what happens.

A

So I just mine all the time.

D

A

A

Okay, this is publishing now and we now add a t.

A

Now we try to add a repo again and see what happens.

A

It's been edited, okay, so.

A

A

Try and install prometheus and see what happens.

A

We need a name so.

A

Successfully got an update, great.

A

A

No install dash dash the name first in the chat okay, so it's the other.

A

Way warning: let's try this dedicated thanks.

D

Cool since they're successful awesome is there a next step in regards to. I don't know what people need to do when how we communicate.

A

Yeah, so um the the next step is to point our uh the glab instances to this new location now, so that was a temporary one that I set up so.

A

So we can do that and that that will fix it for gitlab, 13.6 or 13.5. Sorry, and then we will also um backport this as well to maybe 13.0 as well, um and I I think, yeah we're gonna. I think we're planning to write a blog post as well to communicate this to to people who may need to use this new repo.

A

It's probably there's, probably less people using the like manage apps, it's probably going to affect how to devops them more, but we'll communicate to both sets of users. I think yeah so.

D

That's it makes sense, cool.

A

All right thanks a lot. You are thanks. Everyone for coming.

D

A

A