Kubernetes SIG K8s Infra, 18 Mar 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes WG K8s Infra BI-Weekly Meeting for 20200318

Description

Kubernetes WG K8s Infra BI-Weekly Meeting for 20200318

A

Now so hi everyone, my name, is bart smikla and uh welcome everyone at our bi-weekly case, infra working group meeting um at the beginning. I want to remind everyone about our code of conduct, which we can summary as be excellent to each other, and I don't see anyone new. So, let's jump into.

A

Our agenda um billing review- I think this is the week where uh maybe differently um team is there anything particular you want to discuss right now.

A

A team hocking.

B

Sorry I was on the wrong window. Do I not have video I'm sorry? I can't seem to get video. um I don't have anything in particular. Let me think real quickly, uh we're I don't know if linus yeah linus is here linus uh you can talk about when uh I have to disappear in a few minutes. um You can talk about the plans for the domain flip um and I think that's really the big one on my mind.

B

Otherwise it's been a pretty quiet week or two. It's a lot.

A

I have a question for you because I'm I spent a lot of time like doing research preparing ourselves to the terraform rewriting okay. I need I to have like a fresh audit.

A

Audit, because I'm comparing like because I did like a whole research about components and resources which we are creating and using and um it looks like there are some like a gaps.

B

Okay, and so you want to compare like a before and after.

A

So compare because I manually check all scripts and what they are creating, which components are they creating, and I compare I'm comparing them with the audit ones so that iems and the services and everything- and uh it looks like there are some like differences.

B

There probably are we should uh we should do a full audit. That would actually be a lot of, uh I won't say fun, but it'll be a lot of work.

B

You should be able to run the audit script if you're in the auditors group there's a separate group for that. If that's not true, then we need to fix the permissions for the auditors group, because that was the goal. Okay, so so no worries, then.

A

I will, if I'm not on the group, I will create a pull request and I will.

B

Okay: okay, that's it from me uh and and if you're, because I'm I've got higher privileges than that. If I've added something to that audit script, that the auditors group can't run, let me know and we'll figure out what permissions. Okay, no worries.

A

When I finish like this um my work, I will send a link and email about it, so.

B

Okay and totally open to improvements in the audit script right now, you know dumps a lot of json and other stuff that probably could be filtered and formatted better. Actually, I can already send the link for my.

A

Analysis because it's so big already that it's like repository, but this is like working uh think for now. So don't.

B

Okay, all right, uh let me open that before I disappear. um Okay,.

B

Anybody else have anything that that they would like me to do before I disappear.

C

um I was going to try monkeying around with creating a trusted build cluster in the next two weeks. Is that something you have an opinion on tim.

B

um What is trusted mean here.

C

Trusted means some place to run all of the jobs that currently run in prows trusted build cluster, um which turns out to be the same cluster that proud runs on. We want it, we call it trusted because there are secrets there that we don't want exposed to all of the other pre-submit and post-submit jobs that run code that comes in from the public, so this would be things like the container image promoter any of the image builder jobs.

C

Things of that nature.

B

So we put all of our trusted secrets in one cluster, so if we lose one, we lose them all.

B

C

B

C

Access to those secrets.

B

Yeah, I'm sorry I was, I was being snarky. um uh I think that's fine! um If you want to well. Okay, yeah! That's sorry! I'm thinking out loud! um That's fine! Do you want to play with it in a playground project, or do you want to play with it in the real project like how much playing do you think you need to do versus just doing it.

C

I am not sure off the top of my head right now, uh so um I will get back to you on that. I just wanted to make sure you were cool with me starting down that path.

B

Absolutely I mean I think it's almost time to start looking at the next tier of things and prow is certainly a large item on the list of things. I'd love to see converted.

D

B

Okay, okay, all right, then I'm going to drop off. I'm sorry uh today was a bad day for me. um I will look at for the mailing list.

A

Okay, so actually, let's start from the beginning a little bit and oh maybe there is somebody.

E

Who would like.

A

To do notes, I didn't ask about it.

C

I can do notes.

A

Right so billing review, I think it's the time um justine. Can I ask you about help with that.

F

Yes, uh I can share the spreadsheet. I don't think I have normally tim. Does the cross comparison, but uh I can share the report. uh I have it up in a window which I will try to share with the group. uh I don't think it's too.

F

Surprising here there is one interesting thing, which is sorry. I have too many tabs.

G

F

uh While I scroll through as normal people should be able to receive this, it is a public report. Sorry, I'm just scrolling through filling time, while I scroll through my many many windows and don't.

A

Worry maybe when we are doing that there there will be a new item, um because we added last week the new pool for um for the like project. We need for memory. Well, the first project was per perv dash, who is taking like eight gigs when being initialized. So it's not moody yet, but the pool is already there and there is like a one instance per.

F

uh Yeah, shall I should go through the report. I don't know if we're ready, yeah, let's, let's, let's, let's do it. uh Yes, so I think you can see the report. It should have a big blue screen saying we spent over the last 28 days 1500 and if I page down there, we are the interesting thing.

F

I guess is that our cloud build spend seems to have gone way down other than that everything is more or less taking a long cloud storage seems to have gone down as well, which is perhaps more surprising because that implies we deleted something, but we can scroll through gce is, I think, pretty much as we'd expect.

A

How build sorry to whether you, I think that it can be something related with the coronavirus, and you know, people working from home, shifting, etc.

A

Because it looks like this going down for everything is happening like from a week or two. Yes, I can look.

F

Yeah you're it it's very possible, it looks like I mean it looks like it's going almost to zero, so I'm wondering whether they actually just turned off some of their jobs. But uh if we look at the there is a nicer one here which shows like the versus previous month month of month, changes and so that's where these ones are highlighted. So we are spending 50 more on compute engine, which could be your ram thing and we are spending about so far. 50 percent less. So I think that's heavily on on cloud build.

F

So I think that's heavily skewed to it looks like our primary consumer of cloud. Build may have stopped doing their cloud builds and we are spending a lot more on cloud storage, uh 600 up, but still not really a lot of money, I would say like 450 and that cloud storage. We do have a report for this one, and so we can see that yeah. That is there, but we don't yet have any more insight into exactly who is using.

F

uh I guess, a fairly substantial amount of storage. I would guess.

H

I would guess that uh the the increase or general usage of uh the prod bucket is due to the legacy images being imported right, so we should see a bit of a bump uh in the next report. Probably.

F

Great, I don't know whether the uh whether those container images go into oh yeah. This is by project yeah you're, absolutely right. So I guess I guess it would be nice to have a uh a breakdown by bucket, um so we could tell what was files and it was uh images- and maybe that's the next uh breakdown, because I don't think we have that right now.

F

Just looking for that, but yes that does sound like a likely hypothesis and I'll try to add a I'll try to add a uh a screen that shows breakdown by bucket, so we can confirm that it is in the the bucket that backs gcr rather than the other bucket just yes, this is the biggest surprise of our billing report. I'd say this week, yeah biggest negative, surprise.

H

Sounds good but at least we have an explanation for it and hopefully that's the correct explanation. So.

A

Okay, so I think that it will be fine um for now um the next topic. I think we can ask uh linus how about things going with the container image promoter.

D

Yes, so um I guess, as tim mentioned, the most important thing that's happening in two weeks. Time is the vanity domain flip. This is the domain switch from the old prod google containers to the new product, kids artifacts prod.

D

So I forget how much has been communicated to the community but uh internally. We know that um the switch won't happen instantly because um for various reasons we purposely push out do production rollouts like slowly, probably it will take around four days.

D

I think that was the number that was given to us and uh so really the the kickoff event will be. It will be on april 1st uh not april fools it's for reals, but uh it won't be any dramatic, like switch where oh yeah, like it's instantly flipped. Now it's more like have we done all the checks and preparations to make sure that switching to the new prod is okay, so that check will happen in that meeting.

D

I think it's 10 o'clock uh on april 1st- uh maybe I should add it to the calendar somewhere. uh So that's happening um a few days before that, um probably around march 27th, I think, is the friday.

D

um We will stop making changes to google containers to sort of put it in a freeze, a read-only freeze. This really impacts mostly googlers, who also contribute to open source. um They have already been notified, but I will probably send out another email or communication, probably today, because it's two weeks from now- um and so that's the domain flip, uh I don't, we have not gotten any big pushbacks or anything like that, or any people screaming or complaining about it.

D

Yet so I don't know- maybe maybe we'll get some uh feedback uh this time around when I send out uh email today. But apart from that, um I guess the other couple items are one uh the backup of the new prod. This is kate's artifacts prod.

D

So just so everybody knows there are roughly 30 000 images, uh unique images in production, new, new prod and the way the backup works today is that it backs up all three regions: europe, asia, us.

D

So that's actually times three so 90 000 images- and uh you saw this errand, uh but well you probably saw it, but I changed the uh time from one hour to 12 hours and that still has not fixed the problem. Apparently because I checked the logs for the last run and uh it failed after roughly two hours, so I think I'll have to change how we back up things, because now I think about it. um This would probably the way we back up things right now, which is copying all 30 000 images in each region.

D

Maybe that's not the best way to do things um off the top of my head. uh I think just doing one region would be enough if we know that all three regions are identical, which they should be um so yeah. We need to make that work uh without exceeding quota. Maybe we might even have to do like a sleep one hour or something in the job to make gcr happy because yeah uh I don't know how else would you create new images? So just just so that everybody knows the way.

D

The backup works is that uh essentially, we just do a dumb copy of every image into a new like prefix, so it's like time stamps. So you know year month, day hour, so there will be a slightly new prefix. So that means that, even though let's say nothing changed for all 30 000 images, um gcr would still need to basically create 30 000 like like metadata.

D

You know database entries for each tag or each prefix. I mean that's how you know their internal backend, so we we still hit them for 30 000. Like you know, things like you need to do these changes so yeah, even if there's no delta, there's still a large delta for gcr, and I think that's why we're getting like exceeding quota.

A

Can you send a link to the current script which we are using to backup.

E

D

E

That's the case: aren't we just going to continue hitting this and dossing ourselves like unless we can raise that quota? It seems like we need to do this differently. Fundamentally,.

D

So that's another option. um Maybe I need to ask them them as in I guess, the gcr team, um but also uh it will. I feel, like yeah, go ahead.

E

I I feel like that route is probably unlikely. It seems surprising to create 30 000 images in an hour um yeah.

D

E

D

I was just gonna say.

E

D

uh It actually takes more than it will take more than an hour. I think, to create. Can.

E

Can we not ourselves know which hashes we've backed up and like, I suppose, maybe I need to go, do some reading. I don't understand why we need to make time stamped entries for these things.

D

I mean so. The point is to do uh well the the script. The backup score is very simple: it's literally g crane copy dash r, that's it and uh we just give it a new prefix, so that um gcr does the deduplication. You know they do all that work for us. So it's literally one line of code that we wrote um we you know we could be. We could optimize and say: okay, look at the last. You know backup entry, look at the delta and only do a you know, incremental backup.

D

That would certainly save us time. um So that's another option that we could go for, but uh I need to like well. We should like write down all of.

E

These different, so in the meantime, we actually probably can't keep running this, because this I mean this actually blocked us rolling out fixes to the general infrastructure, because we were banned from gcr first massively exceeding quota.

D

E

D

There was, there was a bad.

F

E

To prow and we couldn't roll up a fix because it couldn't pull the updated images.

H

I'm currently doing the backup job for all the regions um so like the smallest fix, would be run it in one region at a different check that just checks that all regions are in sync, because that should be doable without the copy operation. Basically just read in and then compare hashes um so that we basically say is the us in sync, with everything else, which it should be. So that check should be there anyway in theory, and then only do the 30 000 image copy.

H

And overall we could uh just thinking here. um Do it split like how the realistic um backup scenario in general would be like what is what are the most recent images of the last six months or something or maybe, and that is done in an hour, because that's the most important part probably and then all the legacy images might only happen in a backup um full backup um on a sunday, maybe over the time frame of like 24 hours or something it's not perfect. But at least we don't block our access to gcr or something.

D

Yeah I mean, I guess, we'll- have to consider those ideas as well. I mean maybe yeah like separating the different classes of images, because the legacy ones yeah. They won't change ever true. So on the other hand, the other um sub projects will grow once we change the domain name um but yeah. That's something to keep in mind. uh I guess for now. You can turn it off, uh because I have a feeling uh like what ben was suggesting earlier.

D

um The backup will take um probably yeah unless we change it to optimize the number of copies it will probably take over an hour, maybe two or three hours, to run to do a full copy of even the 30 000 images. So um I have a feeling that will exceed quota. uh Do we.

C

D

So that's that's one topic. Any sorry. Sorry.

C

D

C

Ask like do we um do we know what specific quotas we're dealing with here like.

G

I don't know I I.

C

I'm looking- and it says something about like a maximum number of requests for container registry.

D

Is that the general, like help page, that you're looking at or yeah? Okay, that's probably the what we're hitting I mean unless we have some special arrangement with.

C

Right yeah, like I wasn't sure if you know we're talking about minimizing copying and things like that, you know if we're, if we think we're hitting a quota just for writing stuff or if you know we need to be mindful of a quota that includes what we read as well. So.

D

I think it includes what we read because the like the ukraine tool. As far as I'm aware it does skip over things that have already been copied or something, although I am not the author of that tool, so maybe I'm wrong.

D

Okay, so uh well, so that's that I guess for for now, just to sum it up we'll. I think I can turn off the backups until we get a better optimized way.

C

This is what motivated me to try and um get a build cluster up and running uh yes, this group's project, so that we could. You know you could play around with running the job there and it might end up like banning the nodes in that build cluster, but that wouldn't disrupt uh the cluster. That proud runs on yeah, correct.

D

H

Instead of instead of turning it off completely, um what what's because, what's the general um attack vector we're trying to save here? That.

C

Basically, the whole.

H

Report, isn't there right right now.

C

Anywhere the job runs uh whichever node the job runs on runs out of gcr quota, which prevents us, and it gets that node basically banned for what seems like longer than an hour, and we are then unable to update prow if crow happens, to be running components on that node in the cluster.

H

All right and gcrain doesn't have a rate limiting, so we can't really add something because we just use the plane copy operation. So we can't stretch it out to like say 24 hours or maybe a week in uh backup operation very.

E

Well, I mean for first thing: we just we're going to have to get it off of that cluster, because it's blocking us from managing all of the rest of the ci infrastructure um when we spin it back up, though we pro, it probably will take a while to figure out. I think, even even with some kind of optimized thing.

E

Probably just all the read calls to all those images is a lot um and you could maybe rate limited, but then I mean, as we get more images, we're just going to make the backups less and less able to keep up to date. It probably needs a different approach, but we should have that experimented with somewhere. That is not going to get our like main infrastructure banned.

E

A

A question because we can do some things to improve it, but maybe there is somebody who we can talk about. Maybe improving this like pumping this quota. Is it even possible.

E

I'm pretty sure this is a we can ask, but I'm pretty sure this is a fixed quota, because this is not a thing that you're like charged for or something it's just avoiding abuse and uh like copying, 30 000 images by surprise is kind of abusive yeah. True.

A

Okay, but but I think that we can ask just ask: maybe there is something we can.

H

Do so interesting, so so crane itself does a an efficient copy. As far as I understood the code, um that does only write the index if the image is there, but that still is just the right operations that are being the issue. So it's not really the copy in general, but even like an efficient copy is just because we use it. We back it up into additional prefix instead of like overriding it, um it still creates the 30 000 index write operations. Basically, I.

D

Think um yeah, I mean that's what I was saying about. I mean I don't know the internal details, but you know I mean as far as when it finishes there will be. At least you know: 30 000 rows whatever it is. You know database updates. That has to happen because the pads are different yeah, the prefixes are different, so yeah, um but uh I would like to move on to another topic, which is um for the uh promoters um image auditing uh tool, there's a pr that justin is looking over at this very moment.

D

That is the um tool that will run in cloud run, to check for changes in gcr and to verify whether that transaction uh mutation of state is good or bad. um That fixes a bug in the current, so there's a bug in the current implementation, where we do not check correctly the child images of a fat manifest. So if you have fat manifest with digest a it has children b, c d e, if we see a change of you, know, insert tag or digest e.

D

Obviously digest e is not in the promoter manifest because it's just a child. We only have the you know, digest a written in the promoter manifest as a parent. um It needs to check that properly and I have a pr up, justin's reviewing it. uh I would like to get it merged soon, so that we can have uh it working properly.

D

I've been getting emails by the way uh for the new um kate's artifacts project, so um the current auditor right now when it runs with that bug, it will scream and and yell and say, hey you have all these images being uploaded.

D

They are not verified, um they've been mostly around the new pr's because we get prs once no. You know once every day or two like the recent ones have been from cluster. uh Was it stage cluster api? um So I've been ignoring those alerts? um I haven't checked every single alert. That is every single different.

D

You know digest that you know rings a bell or you know, creates an alert, because that would be too much work. um Instead, I will uh before the or actually on the day, two weeks from now when we do the flip. Before we do the flip, I will check that um the stuff in the promoter manifest that we have uh align with what we have in production um just to do like a quick check. That's actually very quick and easy to do um just reading.

D

All the images takes like 15 seconds to read all 30 000, actually 90 000, I think or 30 000 images yeah. So um I'll do that. You know at that point to make sure that all the changes are okay.

D

um I think that's about it for the promoter.

I

Do you need any help from us at any stage.

D

um Right now would be just the pr that is under review for the the auditing part. um That's pr number 199 in the repo. um Have you had a look at that.

F

Justin, I have not yet had a proper look at it. I will we talked about it yesterday. I will have another look at it today: okay,.

C

I think I started reading through it, but I have not gotten all the way there.

A

Okay elsewhere, we look at it. So there is no other topic I can see here, but I want to like give update um what I was doing. I was focusing much more about documentation, as I said before, and I'm doing the analysis for the current, like infrastructure, provisioning, tooling, um I'm trying to understand, like all roles, iems etc. So I will be asking a lot of questions for people because I want to when I will finish.

A

I want to make it a documentation, make documentation from this analysis which I'm doing right now and after that I will focus on the observability and um and monitoring for our clusters.

C

uh So I had a random question: um put it on the agenda, um how how do I know what's running on the trusted cluster?

C

A

C

And put it this way like I went to the kate's, I o repo and tried to find like a document, or maybe some kind of canonical directory with manifests that was like. These are all the things that are deployed to the trusted error to to the aaa cluster, and I couldn't find that so also the.

A

Case that there is no document like that, there are four things which are running uh you, of course, when you will be digging uh into the scripts, you can find those, but that's one of the things which I'm working out. um This is gc web right now this is um staging one. Second, I forgot.

I

Is the dns redirector running there? No, I don't think so. At least.

A

No, definitely not, there is gcs web. No, I'm wrong. The the the radio is is there and there is also.

A

I can find it right now.

C

Okay, well, I think this this baby just answered my my broader question, like.

A

But what my goal is and what I want to search for that just my like analysis right now is also um will move us to use the terraform for that purpose too, because right now there is everything which we are uh like deploying to this new cluster to people like cluster is done manually, so we are putting the manifests, but the manifest need to be applied by a human who has permissions at this point and what I want to do when we will move to the terraform.

A

I would like to suggest that if your project can be deployed using some existing providers like a kubernetes provider, we can do it with terraform.

H

So basically, all the uh kubernetes configuration would then be terraformified uh to also be applied um within the uh aaa cluster right.

A

If it's between possible- because um it is, as I said right now when um you are creating like a you- need to run the script called namespaces, ensure namespace and inside you need to put the your project name and when the script will be fired, your namespace with the role and uh everything will be created for you.

A

Then you will have an access for this particular project and namespace as a human and then you need to deploy it there and there is no connection between you need to put your code and manifest into the case.io repository and then it something will apply these changes to the cluster. Everything is done manually at this point, so this will be the next step. First, we need to put the things into the teraform. Then I want to suggest. Okay, we have the current infrastructure resources in terraform. Let's move it further.

A

The next step would be to actually suggest using terraform and automating this to apply these projects with terraform. The next step would be, but this is a kind of new thing is because we wrote our tool for the groups and the google groups management, and actually we also can do that with terraform using the g-suite provider. So, instead of having one big group groups.yaml file, we could do it.

A

I feel better and easier to read and maintain the future using terraform, so it will remove all bash scripts for us it will remove the need for having like groups, that yaml and scripts for provisioning these groups. It will also remove the need of manual applying the the project to the new infrastructure, so we have like a three or four things in one place in one place done by one um solution.

H

Okay, so we so we basically have a reconciliat conciliation. Loop running terraform apply uh for the whole cluster, including uh keeping in sync with the g-suite.

A

H

Code basically.

A

Yes and no, I wouldn't, I would like to suggest.

A

So we can, we can do it in multiple ways. This is the thing which we can discuss because I don't want to have only one. um You know one configuration which will be doing everything, so there will be everything I would like to have. You know possibility, for example, to just apply. You know that we there is new project which was going to the cluster. Let's just run the resources for that purpose, and also it would be helpful, then, maybe in the future we will decide. Okay, it's safe enough. um Everything is working correctly.

A

We we want to automate it using. I don't know like github actions or something, and then, if we will have that, we can say okay, this directory change changed. Let's run only these particular parts of our you know, pulling.

C

Okay, so I appreciate that we don't have a perfect configuration driven automated continuous deployment solution to reconcile the various resources we make in the cloud everywhere.

C

But I I do kind of feel like before we jump directly to terraform is the solution, everything we need to make sure we've appropriately scoped the problem and gotten consensus on the solution agree. So, but what I feel like is lacking right now is something that documents the state of today.

A

So this is exactly what I'm doing. I I just send a link with the with what I'm I'm analyzing, what scripts are creating, which components and when I will finish that it will be a great place for okay. These are the components which are being created right now. um What are we doing with them? I'm not saying that we we decided, we do everything with terraform. This will be the point where we will be discussing this at this point.

A

This is this is the suggestion if I feel like most of the people agreed on, but what I wanted to achieve with is to.

A

To have at least to know the state of the world right now then move to the discussion about okay. We know what we need, uh what we are doing with that okay, then we can agree or not about terraform for the things which already exists and when we will finish this part, then we can talk about you know automating and deploying the projects or automating. You know the groups etc.

H

So um I mean I, I agree that the terraform probably cleans up a lot of the the general um how everything is needs to be like written and like uh moving away from the bash part.

H

But one thing that I think I got from from aaron's comment was currently we have humans in the reconciliation loop and that's a really problematic factor in terms of refactoring, iterating and automating, and that's, for example, holds up the terraform part as well, so with us moving to terraform without actually having a solution for taking the human out of the reconciliation.

H

We just moved to a different language instead of or at least that's how it interpreted it is that without the reconciliation uh solution being there, we just shift around the problem um on how it's written not on how it's deployed.

C

So I actually, I kind of um meant the opposite. I guess, or I my preference is the opposite- that we would. I would rather, as we iterate on some kind of reconciliation loop, that that continues to be done by humans in case anything goes completely wild or crazy. I am wary of turning on automation right away. It tends to accelerate the cycle with which we can tends to accelerate shooting ourselves in the foot has been my experience if you haven't like really had the chance to exercise.

C

So what I was asking for was um so bart I am reading through your stuff. It is extremely thorough, but I think I was asking for something a little more like a role book or handbook that the humans who are involved in the reconciliation loop today like how do they know what to do, because we write down what they do so that a it might be possible to onboard people to help this in case those critical folks become unavailable and b. That kind of just lets us know what the state of today is operationally.

A

um Yeah, I I agree, and I want to have that kind of documentation. I I didn't plan to do it like you know, partially right what we you know what we need to be done right now I want to. I was thinking about first analyzing.

A

What is the state actually right now, so what the components are there and then, with that in mind, um I'm also inside this analysis, putting the links where we are creating these components and what it should with as a result should be, should show us um what is in our infrastructure right now. You can partially check it going through the audit files, but it is, uh I was doing this last week and it is kind of tricky. There is not everything there and you need to be aware a little bit how it's structured, etc.

A

But I agree about the documentation. I will think about doing quickly something um for people, but overall um my main goal would be to to get like a proper documentation um with the components why these components why these iems are provided, etc, etc, etc.

C

Yeah, okay, um all right, I will try to help I'll see if I can reach out to some of the folks who run the things and see if I can get information out of their heads about how they do what they do.

A

um I think that when I was digging uh into the scripts I I kind of have idea. I didn't run this music, I don't have permissions, but I have idea what and when needs to be done. um So, if you have like a direct questions, uh definitely I can, I think at least direct you, which parts of our scripts are doing that kind of thing.

A

um But what I know right now that we are you already using terraform to provision the the aaa cluster and the two poles which we have, and we also are. The people are running the auditing script. The people are running like the the bash script, for ensuring these storage things, the name spaces uh thing, and everything else is done manually.

A

C

um I don't want to take everybody else's time. I continue to talk about this. I will ping you offline.

C

H

Another yeah there's one thing so um you reached out. I think, for the last meeting um to shortly give an update on the redirector, so we currently use nginx for the redirect and how it's set up. It's basically a single nginx config file and, in the long run, especially enabling six to basically have their own redirects in the future. That just provides a single point of issue, so that was the idea to base it on dns and the specific tool being tex direct and last time.

H

I could be here and this time the last two weeks for coyotes, so only giving a short update um so behind the scenes, uh we're basically working on bringing the tool itself up to speed on the edge cases of the nginx configuration. So we basically converted the whole configuration of the nginx into dns records, um basically using that configuration to set up our tests end-to-end so that these cases that we already have uh in the current redirector are basically uh tested well uh in a in different types of flavors.

H

Let's say that and then, um when we have all the edge cases uh backed out and basically the next release, hopefully next month, um then we should be able to basically get everything up to speed, to try out the first thing, which is not just not the full nginx file being converted, but only the go dot. um Kubernetes that I uh so the smallest part. But we wanted to basically uh test the whole thing first to have a good base to build on.

H

um So we can basically um be sure that the minimal thing works uh perfectly and then we can. With that base, we can um see iterating further what makes sense to pull out of the nginx into something that is a bit more self-contained, and with that we could. We basically have the option to give every sig their own path, um which is controlled by dns records.

H

So as long as the sig has access to their own uh domain or record, they basically have access to their own ul shortener if they want to and because dns is hierarchical, structured that maps well to community and community groups being self-organized kind of. um So that's the general idea and hopefully that works out.

H

Okay, thank you for update.

A

C

Questions, can you can you restate that last part, I'm not sure I quite caught it? Were you saying that every like, potentially any dns record, could act as a url shortener.

H

So basically, how it works is that um the domain itself say go to kubernetes.io um has the ability, because each there's some mapping between the path to a subdomain and because subdomains can be delegated to a cname.

H

um The the parent record could c name a specific path, basically uh to seek, and then the sick has their own dns zone file. What can they where they can basically add new uh records and these record map to redirect on the parent zone and on the parent redirect engine? If you want to call that.

E

Do we actually are we actually planning to allow people to make unmoderated like sig specific, like so most of the places where we have cigs doing things on their own? It's sort of like actually name space. I think it for like go links that would get kind of weird like go dot, cluster life cycle, dot, gates. or something um most other places. There's just a group that manages this like.

E

If you want to get hobripo, you ask the github admin team and that's the delegation for like there's intentionally a bus factor, because it's a flat name space.

H

So the the first step and that's the the first idea- is um just having it in sort of a better mapping to a non-flat namespace. So, instead of everything having to go through, basically tim and dims, currently for the nginx configuration, we could have the ability to say: okay, everything that is an especially specific path for, say, sick contributors.

H

um They could decide on their own it's and that that basically, we are moving away from uh the one single instance of the decision to more of a collaborative thing, but.

E

H

E

Is isn't it one go kate studios? Something? Are we trying to make one per sig with like go dot, sig foo, dot gates? Because if we're not making one per sig, then it's still one instance and the dns records go through one group of people today.

H

Yes, and so so it's still the same instant. So it's one instance running uh that is doing the redirects uh it's for now and that's the uh short term uh goal. Is um it's still going through the same dns records, but how the mapping works. It might be nicer to be split up uh on, even if it's still the same people and we could bring in like a because how we manage um the the dns records.

H

Currently, we could allow um sap directories uh being controlled by its own owner file, so paris could come in and say: okay, everything that is basically dds related to that redirect could be managed by uh sick contributors uh instead.

E

Of the contrary, paris is still going to need to ping like tim or dems or me to like push the dns changes, but.

H

Currently, yes, but the plan for the dns change is that we automate the dns uh changes anyway.

E

Yeah, I'm still not aware of a plan that makes that actually viable. Yet, though,.

H

True, but so so we have to separate, probably the short term, the midterm and the long term, where the short term is um trying to better have a better control of not just having a single nginx file, with a lot of edge cases to something that might be more contributable to um and then the second, the mid term part is maybe split it out to uh different groups that can pre-review changes and then long-term.

H

If we ever decide that we could open up um the general option to to basically push their own uh redirects uh from like a sick perspective like um sick release wants uh to have a specific redirect for release or something like that. They could push that and review it on their own, because it's under their namespace in the redirect area, but that's not planned. Let's say that that's just an option that is is being thought about during implementing the first two stages.

C

Okay, that uh that helped me understand where we're at and where we're headed. Thank you.

H

Thanks, if any questions happy to to uh go into detail, not here but like bring me or something.

A

Okay, any other topics.

A

Questions suggestions.

A

Okay, so I'm thinking we can give everybody like seven minutes back everything everybody's okay with that sounds good, so very much for being here and see you in two weeks.

H

See ya and stay healthy.

C

Thanks everybody.