Kubernetes SIG K8s Infra, 7 Aug 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes WG K8s Infra - 2019-08-07

Description

GMT20190807 153339 k8s infra 1920x1070

A

Hi everybody today is Wednesday August 7th I am Aaron of Sig beard and you are at the every other weekly WG Cates infra meeting, we're all gonna dear to the kubernetes code of conduct, by being our very best selves and not being jerks, and you can watch this video on YouTube later on the wgk, a playlist on the kubernetes community channel.

A

As always, I'd like to us to run through sort of a building review, walk the board, talk about the items we have in the milestone and then go to open discussion.

A

So from a billing review perspective Justin. Where are we at.

B

With the billing report, I'm furiously typing to bring it up, but everyone should have got it or I wanted to subscribe to. The amazingness should have got, which is a very general mailing list, which I don't recall. I should have got it in their mailbox on monday. It should be sent to that mail list. Every monday. It's still use Comic Sans I'm, debating stopping it out for a guest font of the week, but it should be pretty accurate.

B

Last time we spot checked it against the actual numbers in UCP, and it was basically close enough, like you know, some errors around like 1% type skew, which is pretty much we'd, expect for the cutoff windows yeah, so I haven't figured out exactly how to share it with many people, but I can share with individual type things and when.

A

We talk about sharing, are we talking about giving.

B

Other people like edit rights to it exactly yes, so we haven't really got a good answer on that yet, but we do have the output being very public.

A

Yeah so link to the mailing list. Yeah I see it's on the mailing list, so everybody gets it. Papyrus is definitely a great suggestion for a guest font, Tim I also they put out a mono space version of Comic Sans. Recently I saw a comic mono if you want to go old-school. This is why we need everyone to be able to edit the proportions.

B

A

So on that point like it would be helpful to understand what it takes for us to get the report locked down to group. That's in groups Yambol, so we could do the apples that way. I I.

B

Believe I can actually just share it to any group. I will double check that one before I.

B

What's a group yeah, it looks like I, yes, I could, for example, I could Oh, actually Kate's infra, GCP, accounting, concurrently, edit it okay. If I should not be the case, tell me quickly before people go change. The one is.

C

There is there like a readme or something that we can add to the Kate's IO repo. That says: hey here's, what you need to do for accounting for GCP you get in this group and go to this URL and you know there's another way.

B

To download the ammo for it.

C

But here's how you can go edit it I can I can certainly create said. Read me I, think that would be great just like an accounting directory with a accounting, slash, GCP, reor, readme GCP, or something like that. It would also be really cool if we could.

B

Let people like streamline it, so people can develop their data studio add-ons without him to change the original, but I will like sort of come with a workflow and then we'd sort of like have a look at it and merge it in type thing rather than risking the Monday. The Monday delivery.

C

So I'm, looking at the cloud console report and for I, really can't tell what reason we have a negative 120 dollar balance.

C

Yes, I sorted it by like by price descending, and it starts at zero and goes down from there yeah that's with adding the two columns mentally or not. This is this: it's just a subtotal column, I mean I. Guess I can do the cost column here, really see that there we go that's a little bit better.

C

This is it's again: it's like the the chunking of the credits and the bills as they get rolled into the accounting system. Just produces these weird momentary, artifacts.

A

Okay, that's fun, suggest so we're seeing their report. This is the one that Justin sent out. That said that we've spent about three hundred and twenty-one dollars in the past month, instead of negative one hundred dollars.

C

321 I'm, looking for a total, of course, it's only a total, my it doesn't total the column that I want to care about. One hundred eighty to ninety one, that's two hundred and seventy I get something slightly more than that, but I'm taking just the last 30 days, which exact date range is that.

A

July 8th to August 4th, so you may have slightly more recent data if somebody's done something only eight or penny in the last three days.

C

Grouped by SKU I.

C

Mean it's it's in the ballpark. Okay,.

A

So yay I don't know if we really need to walk through too much more of this.

A

To date, most of our spend has been compute, not much else cool. All right, I'll stop sharing that how much more ballpark closeness do you need to feel confident that, like we're we're done with this.

C

I'd like to continue spot checking it as we turn more services on but I'm. My confidence right now is pretty high, but you know when we found that credit issue right. It made something, go walkie and it turned sideways. So as we turn new things on I'd like to come back and Reese watching it.

A

Okay, I'm just sort of trying to clarify the definition of Dutton here, because I feel like we have a billing report. It send.

D

Out weekly, everybody in the group can.

A

Review it, what we, what we are lacking right now, is confirming that people know how to edit the report, because we lack you read me I'd.

C

Call that done once we get a readme or something in place. I'd say this is done. We should add a recurring agenda item for, like every other meeting or every third meeting, to do a spot check- and you know for this next few months and ones we're confident that that's good. That means.

C

A

Does to me anybody else have any concerns.

D

That sounds good I added a shoe for Justin as well. So we don't talk it an issue for what sorry that heed me.

A

That we were talking, if you can like.

D

E

A

Wait I see it in chat, I'll get to it, okay, cool, so next I will, let's run through the milestone.

A

Many windows- oh my god,.

A

Which I I just.

C

Posted to chat, if once I figured how to disable discounts and I get a number that is 406, so it's close, but it's not spending okay, that's way out of like the normal range, so yeah, let's, I will sync.

B

With the afterwards and like.

C

Try to get I can send you a screen cap of where it is now, or can you actually get the billing report? You should be able to get the rate straight billing report if you're in the accounting group, okay,.

B

I'll give that a go I'll. Send you your own. Thank you.

A

Okay, so here's the port I, have it filtered down the issues and milestone ready to migrate, which is everything we want to do before. Reading migrate, stuff, I'm guessing we don't have any updates on her namespace billing information, which I believe is a feature we need to enable last update, was June, 26 Tim. You need to play with this yeah.

C

Well I, as promised I was on vacation, so I'm back now, I have quite a number of pull requests that are in my queue. I am tackling them as quickly as I can, but, frankly, code freeze stuff is taking priority over this yep.

A

I'm gonna bump this down in priority.

A

Storage analysis need analysis of what storage is honestly I feel like. Maybe we should just turn. Maybe we should just find out how much storage is going to cost us by turning this on and seeing it live. That seems like the best simulation of storage traffic and artifact download that I can think of.

B

That we do have some storage like right now, and it's not significant so like, as that becomes more significant than we'll have to break it down further I think so.

A

I'm not necessarily calling this close, but the way I see this playing out is we really are interested in pushing forward with artifact promotion stuff and so eventually we'll get to a place where people are downloading those artifacts and we should presumably see some spikes somewhere and if it's, even not that reasonable great. But it's not it's not such a spike that we really care about being granular.

D

Justin, did we get away from your personal credit card stuff, yet.

B

No not yet, but we that is, we have, that is the first non recurring agenda item. Okay,.

A

While I'm doing this Tim's, do you mind, excuse me, do you mind taking notes for any of your anthem, stuff that comes up here? Okay, thanks! It's! It's super helpful. Okay! Next things on the area of access management, I am dump scripts. There's a pull request. That's sitting out here! That's ready for review! This is going to help us better understand. What do we have? What rights does? What have oh yeah.

C

So back in hippies quarters, you've responded. You.

A

Have comments but I think my my suggestion is this: is this is very, very, very low on hippies priority queue? Okay, weird.

C

Right I can, if we want to its assigned to be I, will go through all my PRS and when I get to this, one I will consider approving and I can make my own changes to his script. Yeah. If it's his let low and his priorities on his priority list, then we can just do that. You.

A

Or anybody else but I think.

C

It like Krista a follow.

A

Up issue: that's like these things need to be at rest, but we're at the let's merge and iterate point with this.

C

So at a last look when I get to it in my queue and then we'll agree: okay,.

A

Which is what we would use to take inventory of the Google Groups we created I. Think yes, okay, so I will also leave this assigned to Tim.

A

We had some things in our backlog, which I'll just check whether they still need to be in our backlog, and now one is about using Google Groups for fine-grained cluster authorization.

A

And it sounded like we wanted to just punt on looking at this for now, while we look at terraform, yes,.

C

I request that we leave this open and while I figure out the terraform, which is the other thing in this group, that's in my queue and then come back and decide whether it's worth doing or not. Okay,.

D

Tim at least two people have tried it out and tried out the terraform script. Other.

C

Than crystal so cool hi I.

D

C

To I mean that's, that's great and gives me a good confidence, so I will worry less about whether it works or not. I'd like to understand it for myself yeah and make sure that all the flags I expect to be sad are being set, which is a tall order because I don't even know what that list is. I have to go back to G cloud and check the list again. Okay, thank you. That's all I'm really looking for.

A

Okay, then, we have sort of a separate issue. That's about rolls of our back rolls for access within the cluster, whereas I am is about access to everything within the project. Again, I'm gonna suggest punting. On this now yeah we can't run a scraping job until we actually have the script merged to use, not the job.

A

Okay. So let's talk to air form.

C

It sounds like a lot of people have confidence, it I need to read it and understand it, and then, if I'm good with it, I think that is P o R. At this point.

A

That sound good to everybody else. What do you next step sound like here.

C

If this is I would say from my point of view, if, if we're good with it, we turn it on and we shift over. What's the one that we have running the publisher and we shift over the publisher to the new repo and then we start with low-hanging fruit from the old internal cluster like GCS web and those sorts of easy things and merge a few of those over make sure that we're happy with that and then I would say we would.

C

We would want to build up our playbook if we're going to start having more people paying attention to it and more people having access to it. I'd like to understand what our monitoring and alerting and reaction is going to be to problems right now it you know like Kate, said: I/o goes out and it pages me so I'd say the next step. Once you have, the cluster up is figuring out: production radios.

A

Ok, so you haven't, have an issue to burn down and recreate the Kate's infra cluster know, which is what we would do using terraform yep and migrate, all the things over and then the thought is. We would then need to enumerate some follow-up steps around monitoring once we burn things down. Yes, anybody.

C

Disagree with that.

A

Are so talkative and you can watch me make so many typos, while I type 8:30.

C

On a Wednesday man.

A

Okay, do we need to talk about anything else from a cluster management cluster in for a perspective.

A

Dope, okay, cool.

A

All right, my two favorite umbrella issues, which I really feel like need to be retitled to our East Coast, which are about setting up a GCS bucket and setting up a GC, our bucket dims. You even hunt assign yourself from this one I kind of feel like I, don't even know what this means anymore. I have used it sort of as the proxy for we want to do. Artifact, promotion and I think it also is related to the fact that GC r is like a loose wrapper over GCS, or something like that. So.

D

Sorry I was doing a bulk on the sign, and this was one of those that got in I did that inadvertently, so the GC r GC s seems fine. The process that we have now I did add an item to open discussion, so we can talk about it later, but I don't think we have anything much left to do here. We have a process and a few of us are able to run the script and we are able to you know: Alec. You allocate a CIO's and give people access to so we are.

D

We are good. Okay,.

A

So people can get GCS buckets for staging yeah and anybody else who wants one for other purposes is just gonna have to ask us yeah a one-off process. Current.

C

Until we have a generalized promotion process, what I I want to be thoughtful about is whether we're going to give out a bunch of individualized GCS buckets or whether we want to actually have sort of a top-level bucket that we subdivide you know sub-directories, underneath, which was what the promotion process would provide and I. Don't think. We've had enough demand for it, yet to really say which one we want to do doesn't B feel like we do, maybe I'm just ignoring it. I I think.

D

We we are handling like 70% of the cases right now so I'm happy. We have the automation, stuff working and we are able to whoever is coming to us. We can turn around quickly and give them what they need right now. So we are fine. When so one example that came out for the one-off was the sig release. Team came ask for a bucket, and you know that court handled outside of this process so which is fine.

A

So each singing words like this, it sounds like what's lacking, is documentation like it sounds like sick release, kind of happened to know the right people to poke and ask real nice for a GCS bucket, but I don't know that we have written down that like this is a thing you can do. You can ask us for GCS bucket for special purposes, all.

D

Right this was more similar to the one off we created a GCP project for somebody to try out the terraform script right. So this is like that. It's temporary it's for prototyping and then it will go away. I'm.

C

Hoping that that's going to be sufficient until we have the promotion process in place tour, we decide not to do in promotion process, and then we can cover it basically like just too many irons in the fire and I'm hoping. We can postpone that.

A

Sorry can you say that last part again, the Wednesday 30 in the morning thing hit me I, feel.

C

Like I've got too many irons in the fire to make a thoughtful proposal as to how to proceed with that in terms of new groups and new whatever's around hoping that what we've done has dampened the fire enough that we can postpone it until we get past the cluster stuff. So.

A

Let me see if I can numerate what I think happens right now, right now, I think. If we give people a GCS bucket, we require that that access to that bucket be controlled via a group that is listed in groups, thought yeah, and so we are saying that anybody can get a bucket, but access to that has to be controlled exclusively by our set up.

C

I'm saying we have a, we have a set of scripts that are well proven now for staging repos and everything besides. Staging repose for GCS is ad hoc. We haven't scripted or formalized it at all.

B

And we create a GC RNA GCS repo og CR Epona GC s bucket, at the same time as my understanding, right, which.

C

I, don't think, is a problem. I think it's good, so staging I think is well understood. We have this prod, enabled prod storage script, which we've sort of overloaded for the these side of things, but I, don't think that's actually the right answer girl, if it is, it needs to be codified a little bit more.

C

Sorry wrong with it I mean it was built to handle the GCR side of things. Then we sort of bolted GCSE onto it. It's got a couple of special cases for the test, situations for the prod situations and I just don't feel like I've thought about it enough to say, I'm completely happy with it. The way I am with the staging stuff.

A

But the thing I would be clear on for the buckets is like I: don't we're not okay, with giving like individuals access to buckets it's more! That's.

C

Right they get.

A

A bad is corrupt. You're gonna have to use a group, that's the fine and in our file.

C

Individuals should not have access to anything.

B

Other than by the group's correct.

A

D

A

Groups yay, okay, thank you for helping me learn all of that. Okay, so the GCR thing is kind of like the proxy for how how are we doing with container image promotion these days and 28 days ago, I remember amy rattled off like this great list of stuff that we thought we needed to do. I see some of these are closed. Some of these are open. Is this the relevant list of issues to walk through our current blockers for image promotion.

F

It would help to enumerate that, let's let me open them up first one! Yes, on that list: okay,.

A

So disaster recovery plan.

E

A

That would be so.

E

This one I keep dropping the ball, I'm. Sorry about that. It's on my like to-do list of like figuring out like a Google Talk, for the options that we have but yeah. Let me know if it's like, like a like someone else wants to do it or if it's like an absolute blocker but I, trying to make sure they get this done. So, basically, the the idea is for the ASIO pockets they're like if the image promotion process like accidentally deletes it.

E

There's like no way to recover that, and some of the proposed options were basically just replicating the like the the underlining bucket as deep the underlying bucket as a as as a game plan said by sea blacker, so yeah I.

F

Mean so I don't know how quickly this needs to be done, but as far as like this being a blocker, it's probably the last thing that should be done in terms of like priorities, mainly because we still don't have the e2b test running, that's probably really down to that list, and if you okay, I know this is like this has been talked about at length, but really for implementing this, like fully with a solution working solution like with tooling and stuff to do, backups I, don't know how hard that will be or how easy I'll be.

F

I haven't sculpted I, don't know if anybody in this meeting us like thought of how much worked out involved, but I would think that any backup disaster recovery like solution, you know having being having a well engineer solution, would take some time something more than a week or two like.

B

I think we could very quickly build what I would call a very, very bad backup solution which is I, could loop over the mole and duck or save, which I know is gonna, be horrific, ly inefficient, but I'm very encouraged. So it's not as expensive, but I mean because it doesn't it's a hurricane because it wouldn't deed. You.

F

Yeah, but did work, does anybody else have more ideas on this topic because I actually have five minutes of battery power left on my laptop and okay I? Think.

A

It's I think it's a blocker but I agree. It's maybe like the last thing, but I kind of don't want to turn on a tool and pointed at fraud without a plan to like recover it fraud. If fraud magically disappears I.

F

Mean right, like so you'd like how do you qualify, that the back-up plan or backup solution like works flawlessly I.

C

Would you do a restore into a target and you verify that it matches what you think it's to match, I mean.

F

Exactly yes, but like to do that over I, don't know to actually have that happen over many on the real thing or with the kind of volume that we have with the you know in Google containers having many many images. How long did it take like? How robust is it will it you know, does it recover if you kill it halfway through the thing, while it's you know doing the snapshot you know 70% through and, like all of those considerations, I think have to be thought out, but anyway, yes, I am going too.

A

Much into this I hear your points um so will require further discussion, but.

F

Let's see how much further we can move this this Boulder. So just one quick note, though, I would like to read echo tim's comments on this originally, which was we should have something even just like a written instruction set of instructions of this is how you you know, do a manual backup or something like that would be the first like step, I, think sort of having an actual solution implemented just having an official policy or it could read me or something that's way more anything. So, okay,.

A

It is kind of unclear to me where we are at with protecting promotion credentials, or this also feels like it's slightly lower on the priority stack.

D

F

A

So you know I can blow the text up help us read better.

C

I'm sorry guys I got a drop off. I just realized it's the half hour like that. We've got something else right now and I have to be there all.

A

Right, okay, um yeah I'm, not gonna, play I'm honestly, not super clear what this is about and suggest we punt on this because right now, I think it's like we've run. You run something container image promoter related to any trusted place.

F

Yeah, this is like I, guess, more security related, so I guess we could it's technically not a locker or.

A

Saying so we're punting it back up or punting on security. This is gonna, be super reliable. It's great test, Google service account. We made the thing all right.

A

Servers are learning. The service accounts already exist, close this, and once there any comments, hey make the promoter actor personally for all the manifests found under a directory. Oh this.

F

Is done, I, don't know why it's still open I should have closed it.

F

A

Have the rights to do this line.

D

Did you promote the the code recently well? Did you update the image that is behind the image? That's.

F

Actually running, no, not yet I think there will be a couple more changes to how we invoke this new functionality.

F

I think the multi-run script that we have that shell code is probably not as useful anymore, even though it does like this optimization thing where it only runs against the manifest I have changed like the way the new code works, like it does. Some optimizations of as far as like making as few read or read, calls against the repository as possible. So I don't think we even need anymore. Shell codes, so that's gonna be changed. Sorry I have 2% on my battery meter.

F

The one update I have is so the the other missing piece here is like turning on ETV tests and making sure that it works for the actual use cases that we do use it for so there. The current III test that I have in the e2b binary is just a single promotion. I guess one manifest I need to add another test case. That has you know, multiple manifests and like I, just need to add more test case and to turn that on I don't know if we have an open issue for that.

F

But that's like the next thing or the thing I'm currently working on I hope to finish it like this week.

A

Okay, there's this cable doc that he wrote a little bit ago and I will keep this open until we feel like we've got me to be testing boy, so this will be how we come ask you a do. You have all your eating test, surgeons,.

F

A

F

I'll have to drop off, cuz I might drop off any moment now. It's.

A

On me, is there anything else for me key your perspectives that were not talking about related to your artifacts, so I.

F

Mean the update from Justin's that he merged his code for like file promotions since the promotor. So it's a different binary but I think later on. We'll probably put that behind a common interface like CLI, maybe we'll use Cobra or something for flag headlight, so unified under one like interface. But that's the only other update. As far as the rotor repository.

A

F

Save your battery, oh it's at 0%! It keeps going I've never seen that before. Okay, well, I'm gonna drop off here. So thank you for the meat time.

A

Okay I feel like that is that maybe everything that I need to walk through on the board is there anything somebody really wants to talk about on the board that we haven't talked about yet, as I scroll away from the board.

A

Once going twice and I'm gonna stop sharing my screen about board ray okay. So next up on the agenda open the sketch and Justin the things that who's talking about, let's start publishing binary artifacts. Let's stop having you pay with your personal credit card or binary artifacts that what.

B

This is yes, this is indeed that subject which is near and dear to my heart.

B

We now have an image from a binary promoter. In the container image promoter. We have a work-in-progress PR to use that for some cups wineries we actually have the prod bucket setup right now, so it will serve right now in artifacts, arcade siyao I need to follow up with Tim to find out like more about why he was unhappy with it. It sounds like he was more nether with a structure than ie like I will confirm with him or worse. My understanding is.

B

We were going to do that and then see how it goes and whether we need to then like change. The structure and cops is fine with that we have a sort of way of a mirror, doesn't have to have the artifacts on it. A secondary mirror doesn't have to have the artifacts on it, so this would be a secondary mirror. So if it goes away, that's not a big deal for anyone, and we can basically start to get traffic and start to understand how it works.

B

So I was helping to do that, but the two big blockers that I know are: we actually need to get like the promoter, the file promoter into a a cron job. However, we're running the the container provider, and we would need to get billing reports, which we can do more when when we actually have something to report on like we have the aggregated building reports today on GCSE utilization, but it will once that number grows, we'd like to probably break it down, drill down into it, which we can do once we have traffic.

B

And then today we are uploading or the file promoter does not compress artifacts, which is sort of leaving a trivial optimization on the table to like to the magic dance that enables transparent compression so that clients that clients can just request, and it will be served compressed if yeah, if the clients specify, except using.

A

That all sounds fine to me. I didn't necessarily see an issue or running the same place since the image promoter, but I. That sounds completely unobjectionable to me. Yeah.

B

I guess I guess actually we're gonna have to move it into a different directory, so I'm and I agreed to split the file promoter into a separate binary. So we now have to run a different binary. Executable I am guessing that. Therefore, the we should spit onto separate directory, which will be good anyway, because current leader, she is called like Kate's DC audio. So now we can have like GCS and it will be more logical and I will repoint my PR to that.

B

But I will also create an issue to create the I guess it's actually a prowl job create a proud job to trigger on approved, PR or merge PRS to that, okay.

A

E

I'm not gonna lie like I feel like.

A

I have zero idea how a this works. I! Don't really know that there's like a design doc that describes what's happening here, Oh, what works I'm! Sorry, the it's like the user guide to image, promotion, binary promotion, etc, etc, but I'm fine with this proceeding as dislike.

A

My goal here is not to block this, but I'm, feeling like that was happening, was not really that much like upfront documentation, description of the design that sort of stuff but sounds cool like I'm sitting, I'm, very fine with us just turning it on and then seeing what happens billing wise within a week I find it difficult to believe that we would really blow through that much of our credits in a week of down. Look the week's worth of downloads to be like.

B

Wildly surprised right, it will be 6, it will be substantial compared to the current billing. It should be 2 orders of magnitude higher. That sounds great. Yes, the, but I will also do a description of how the like. So we want to prow job with a readme, a description of the architecture for the file promoter, which doesn't have a name yet so, yes, that is up for naming and I guess, like a description of how to use it all right like when you want to like promote binary.

B

This is what you do: I think user, a user facing guide.

A

I, don't know James. What are your thoughts like this is me, like? Aha and I haven't really looked at any of this in the past two weeks, so yeah. So my.

D

Take on this is when we get the first few sets of users we can. We can hydrate over the documentation itself. We definitely should have some minimum stuff, but you know this is going to be I traitor. It's gonna take take a little bit of time, so I'm, okay, with hydrating.

A

I think I was just expressing my she's, not nothing specific. With this specific this, this task, it sounds cool. Let's get bored with it, I'm getting a sense that would be good refresh our Docs okay. On that note, you need anything else for Justin's.

A

Okay, Tim's, what's working and what's not working.

D

So, just to give you an idea of what what what were the kinds of requests we got in the last couple of weeks, so DNS requests neckla fire urls, you know updates to existing DNS entries. I basically I'm able to run it. Christophe is able to run it. Tim is able to run it.

D

So one of us pick pick up the pr merge it and then you know run the job by hand so that that's working out fine, then the next one that is working well as the update to the groups for the ACLS, that's working, fine as well. There was one hiccup where there was an update to a group, but the ACO didn't get reflected in the bucket for some reason and Tim had to recreate the bucket for it. Actually, you know- and we couldn't figure out exactly why you know who was it?

D

Somebody who was yeah Jason, DT BER, was trying to upload an image and it wasn't getting uploaded, even though he had previously uploaded an image a couple of weeks ago, so it was so the bucket had to be recreated essentially, and we still don't know why it went bad other than that. One hiccup in the rest of the uploads to the staging GC. Our repositories went, fine, I haven't heard any other instances, then the other one was promotion from the staging repository to production. They were a couple of hiccups.

D

There was at least some code changes from Linus which went in and be promoted. They had image promotional image it for the product to pick it up and it got stabilized. So in the meantime, what we had to do. We try running the prowl job and we had to ping test and frog to rerun the project, and you know that took care of some of the issues until the changes from Linus landed. So it wasn't too bad.

D

We just have to paint testing from the on-call in the tests in whoever is on on call and get them to rerun the project, so that and we knew where to go, look for the project, so we could go check if the projects at seeded or not I wish. There was a better way to do this, but this is fine. For now at least we have the logs and Jason could go by himself and go look at the logs and see if the project succeeded or not. So it's it's fine I will say just.

A

Real quickly on that point that one of my teammates Randy Christ is working on like a rerun button that you can push we're still trying its rollout. So it's still the same set of people, but we are trying very diligently to click a button. Instead of run a script, it's really painful for some of us, but then, ultimately we can roll that out so that other groups, two people, can rerun other specific jobs.

D

Do we have github or authorization set up so people in the kids or could get the button enabled or something like that? I am.

A

Not intimately familiar with it off the top of my head I know there was an email sent out that your Brunetti staff and they're likely issues I'm testing for that I could point your way, but the intent is to describe like a team of people. This set of jobs, I got it okay,.

D

So um yeah so I talked about the groups. I talked about the staging repositories. I talked about the image promotion, so all these things I'm able to do now. What I want to do is to see if we can get somebody else as well. So right now, it's me Christoph and Tim who take care of these requests, but over a period of time, I want to add more people to it.

D

If anybody else is interested in this kind of stuff, please let me know and we'll try to get you into so we should have like a rotation on call similar to on call I guess- and you know somebody because you know at some point we'll have to come up with an SLA. How long is it gonna take? Who is going to be able to do it and stuff like that?.

A

So elsewhere, I am trying gonna reach out to the product security team, to ask them about the ops Jeannie instance that they use to better understand. If that is something that folks like us, could use for on-call rotations to also grow the pool of people who can answer the testing for on-call bat-signal, and my my other thought here is handling. Dns requests certainly sounds like it's: it's a person as a service type of deal and I'm wondering like what is preventing us from having that be.

A

You know yet another post submit proud job that runs when the the change is merged. You know, go full get ops like in theory.

D

Is declarative.

A

So we restrict who can approve the configuration change, but then the bot actually executes the change. Yeah.

D

So we have to you're you're, absolutely right, just like the image from whatever Pro job, we should be able to set something up. So one thing that we were trying to be careful here is: can we set up? First of all, we we need to create a.

D

Let me back up a little bit, so we need a GSP project where we can have a trusted broke lustre, so we could load it with the secrets and not have to give it to the testing for frogs right and then use the common Pro to run jobs in our trusted block Pro cluster.

D

Do you see what I'm saying I I do so so that was one way to do it? The other way to do it is like, like similar to the publishing bond, where all we are running is like running a deployment or a pod, where the the publishing bot is just looping and waiting for incoming requests and kicking things off. That would be the easier way to do it, but then the problem with artists, the logs, are not available to the people who you know the probes.

D

The logs are available, so people can go inspect so and that will be missing. If, if you do it the way, the publishing board does things.

A

Okay, I am pretty sure that we could. We could have a discussion- maybe let's say testing folks to figure out like what the most appropriate path forward is. Yeah.

D

A

That'll be good, which, with my sink testing hat on I, will advertise the fact that our meeting is now an hour long every other week. So we would probably have room to talk about this Tuesday at 10:00 a.m. Pacific, okay, I'll, be there like I'm, not opposed to opening up the pool of people. I know, I I definitely would like to be one of those people. I keep getting pulled lots of other stuff, but certainly the group in DNS handling stuff sounds like.

D

Toil that we we can automate away all right. There is some gotchas here too right, for example, when we get requests for a specific project to be created, a GC our bucket right. The length of this string of the project name has to be like within a specific thing. So essentially, what happens is I, try things out and if I see a failure, I tell people to go fix, whatever they submitted in the PR right.

D

So if it's a post job, then we will have to like create new PRS to fix the previous problem kind of thing, so it will be yeah somewhat of a problem and until we stabilize or all the input validations, especially the problem with the GC our GCS is people are updating scripts right now, it's not EML files right. So we don't even know what the changes in the scripts people are going to make.

A

Yeah so I agree, it's appropriate to grow the pool of people. You can like help figure this out and the set of people who are showing up to this meeting. Maybe the most motivated people of all. If you're interested in helping out because I know, Tim's and Christophe and Tim are names that I see everywhere, so the like less bottlenecking, we can do an incredibly high traffic people, the better off we will all be um but tier 2.

A

Your comment about an SLA I feel like we are very very very early days before we can get that um because we are still like experimenting and figuring things out.

A

So I think it's it's perfectly reasonable and appropriate to say, like we're still kind of operating on a best effort basis, and if you are, somebody feels like this project requires an SLA. We would welcome and encourage your participation to help us get to the point where we can articulate and measure you know. Soi is SOS SOS, whatever I sell letter you want to use.

A

Theirs dance yeah.

D

That sounds good and that's all I had ok.

A

I think that's seven minutes left it. Everybody wants to take it and do whatever.

A

Well, alright, I, don't think I have anything else other than do you say happy Wednesday to all of you good to see you all again. I'll see you in my only one, oh yeah, the shirt by the way I just felt like I needed to do it, because today is my work from home day because of meetings, so I'm gonna unite with you all separately. In my own.

D

Sounds goodbye I see it.