Kubernetes SIG K8s Infra, 24 Jul 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes WG K8s Infra - 2019-07-24

Description

GMT20190724 153221 k8s infra 3440x1440

A

Hi everybody today is July 24th. This is the meeting for the kids in from our working group. Please be kind, and please follow the code of conducts that we all should adhere to hey Justin welcome. I was just looking for you, okay, so last time we kind of went through lists of issues that we have and try to see what we could do for this time as an for this release.

A

So I did go through some of the things in the backlog and triage stuff and updated the project board. But let's leave that alone for a minute. Let's go through and see if there is any action items from last time.

A

Okay, I see some questions about what needs to be done, an image promotion. Would you like to start there Linus and Amy? If Amy is there.

B

So I am working on making the promotor better understand. I! Guess all the manifests in a given directory I had a crack at this yesterday and realized that I am basically going into the territory of I'm, brushing up against the logic that I wrote in shell-like multi-run that Sh, which you have to pass in a bunch of arguments, and it only checks. Those manifests that change in the last commit because.

B

Could just read in all the manifests in a directory? Let's just pretend there's a hundred of them. We could read them all in one time and then make lots and lots and lots of API calls to lots and lots of different repositories registries, but that doesn't really make much sense if only one of them changed and it also runs the risk of getting throttled, possibly even more by the server. So I just had a thought this morning before this meeting, I should possibly probably think about making the map.

B

The image perimeter also understand deltas between two states and only changed those things that changed. um On the other hand, I mean the whole reason why we started going down. This path was because we thought that it would just make the administration of adding new registries easier, because we wouldn't have to touch or change the proud job. That being said, this is not a blocking task per se. So if it's gonna take longer than say a week, then maybe we should table it for later. Those are my thoughts, I'm.

C

Not sure I understand the problem. The statement, the last thing you said, is what I understood the problem to be which was rather than update the proud job just pointed in the directory and let it recurse.

C

How does that like? Why do you need to read all the files at once in order to be able to do that.

B

Wait, can you sorry, can you ask that again, why do I need to read all the files at once you.

C

Started with I could read all the files right. You said. If we have a hundred files, you read them all in and then activate on them all, but we don't really need to read all of them if we know which ones we've looked at in the past and if we're not burning, yeah and we're not deleting stuff anyway right.

B

Wait, we don't need to read all of them if we looked at some of them in the past. How do you mean you.

C

If only think for a second about what I'm gonna say,.

D

We can do a incremental update, write, an.

C

Hourly or weekly batch like yes,.

D

Thank you, Justin.

C

Said what I was thinking you want? You, like you said you only need to understand the deltas between the last-known state and the current state mm-hmm.

B

C

Keep track of the last-known State I, don't know what pro offers you in that capability. So.

B

So far this we have not thought of making the promoter stateful in any way. So what I mean making it understand, Delta's, we would tell it here, are to get commits, go figure it out.

B

Whatever citizen is all week, you know keep track of state somehow between each run, but I have never considered that because that just seems like a lot of work. I'm.

D

Almost like to say this, but we could we could make an operator and we could use API server.

B

That just sounds well: if I don't.

D

Know the logic you should do that, but if you're not and if we can do something with simpler, we should not yeah.

B

B

Well, do you have any two other people have thoughts on this I.

C

Mean huh what was the I forget, the heuristic in this multi-run? What was what was the heuristic for figuring out? Which ones have changed anyway?.

B

I think it literally just looks at the get history and says: okay, what's the last commit and then did it change in this file like I.

C

Did that's right, I, remember having a conversation it wasn't like the last commit. It was the commits that, like probably give you a range right actually.

B

That's right, yeah, so it looks sad. You know it's like the same kind of thing. It just looks at the history which files were touched. Let's run the logic on those manifests that were touched. All right, right, I just think like that. Well that wouldn't make much sense if we are giving it a single directory.

C

B

C

Anything under a given directory it can run get whatever you had. The right get get Magic for it right and prefix match that directory and if it prefix matches, then you need to evaluate the files that have changed individually, but you don't need to read them all in bulk, right, I.

B

Think I see what you're saying.

B

Yeah, let me think about that. Some more I think you're right about that by.

C

The way I'm thinking about it and maybe I'm oversimplifying it because I don't know the logic that well at all. Is it shouldn't be fundamentally different than the list of manifests? It's just how you generate the list of things you were going to act on is a prefix match. Instead of a explicit list right.

B

Right so well, I think I, guess I was kind of getting in the right direction. Then this morning, because when I mentioned that I was gonna have to probably rewrite the authoring shell script logic and then probably put that in go, and that was the direction I was heading. So I think we're in agreement whether.

C

You write it and go or not. I mean at the end you're gonna shell out to get anyway I. Imagine I, assume you're, not gonna, implement it or have some go get library I mean.

B

There is a go, get library that does this I'm shorter Justin's.

C

A fan here, but I I'm, a big proponent of Shiv, showing out to the tools that are already proven to work.

B

Right but then you can't test it as well right sure.

C

B

C

Just need, if you just need to inject a fake git command.

B

Okay, um well, you can have a look at my PRS when they from they'll and later this week, don't.

C

Challenge me on my willingness to hack around stuff to get a test to work. I.

A

Think there might be some code in the test central repository already for trying to get some of this information from the get history using the go. Client go, get client, stuff, well, I'm. The publishing bot right.

C

Oh okay, well I mean my concern is I. Think you've already got the correct. Get foo from the command line and I would be wary of changing that too much right.

B

Well, I mean I'm, not gonna, beat around the bush here, but I considered like the Shelf logic, where the perimeter just a hack.

C

You it's your project, man, you do what you think is the right thing. Yeah.

B

C

Expressly by my preference went when I wrote, get sync I did not write against a get library. I wrote against the git command line and I log. Every command line I run so that a human could very easily reproduce whatever it was. That I just did.

E

Good libraries improved since you started good sync, so there might be reason to really think that logic nope.

C

Okay, what is it they say if it's not broken, don't.

B

Fix it, it's true: let's.

C

Project usually decide what the right tackle I agree that the multi run that Sh was sort of sort of a hacked me.

B

Yourself said that that was on the brink of like I, won't say madness, but you know like this is like almost unreasonable amount of. You know, shell logic for a piece. That's so important right, I mean which I agree. I mean it's kind.

C

Of you don't like me, but okay.

B

It's in the comments, I cannot give you a link to what you said.

B

Okay. Well, so that's that's! What I'm working on I guess I was thinking of a much difficult problem of making it understand a set of manifests at once, which is.

C

We talked about their conflict detection problem, which I think you would need to load the entire set right.

F

C

Hey I think that can be like I'm like do it later, sort of thing: I'm, not super I, mean I, am concerned about it, but I'm not super concerned about it. Since we got agreed on fixing on the subdirectories put pattern and I, don't think you need to act on every command everything you just need to load them all and look for the conflict, so we can totally solve that as a second stage thing, the.

B

C

That may not even be important at all if we're really doing the sub-directory thing. Yeah.

B

Agreed makes sense to me: okay, that's all I have for now. Oh I'm, sorry, one more thing so I percept a couple changes one random one, one other one around like making it more robust around. Like failed Reed's against GCR. That's up for review dims! You saw that ami I see seed you on both things.

B

One other thing I was hoping. Would land soon was a project to get the e3 test thing running if you're having trouble with that ami I can definitely help out there. But have you had a look at that problem? No.

G

I haven't so let me like this. This is like the hit list of everything that needs to get done, at least from the last meeting and last week. I dropped the ball and didn't get a chance to also write a disaster, recovery, kind of game plan and so I'm hoping to like focus on that stuff this week. So I'm, sorry about that.

B

That's okay, volunteer.

C

B

The that's all I have so.

A

Summary there are some peers that you need reviews the into end test. Pro job is one. You know that needs work and yeah. There's a few other you you have issues for the other things that we talked about before in the image promoter. Repository already wait. Sorry, what's that question do I have issues? Do you have issues for the the other things that we just talked about?

A

Yes, I more than one manifest and things like you.

B

Have to lesion yeah I! Think: okay, okay, yes,.

A

Okay, maybe we should have one umbrella issue in our repository and point it to the several issues that are there in the image promotion, so we can track it better. I.

G

Sent a link in the in the chat, so we can also update that issue. Okay,.

A

Amy, do you want to throw that into the doc that I'm looking at.

G

Look sorry I, like yeah sure you do that. Thank.

A

You, okay, so I, don't I! Think I skipped one thing: do we have any new people, no I think we know everyone on okay: let's do the terraform stuff did in well Tim. Did he get some time to try the telephone? Sir? No.

C

I was I was on vacation or something else all last week, including Monday of this week. So yesterday was my first day back and I am out next week and I'm dead and I'm done so I will try to get the terraform stuff this week, but I have a something of a backlog to chew through. It is high on my list of things. It is my top top thing with respect to this group and I have come to peace with the idea that I'm going to learn it, so I will do it. Thank.

A

You I was waiting for maybe we should just do it now, just symptom. Do you want to talk about the billing review? I was waiting for Erin, but Adam is not here yet yeah do.

D

You have your billing report, I'm just checking my ping. Do you mean I believe I fixed the issue which was causing it not to be delivered so every Monday and I change the frequency. So every Monday morning you should go to the kubernetes working group. Cates infra, mailing lists.

D

C

I am pulling it up now, I'm pulling on my report to Justin's using all my bandwidth.

H

Billy going billing right, but if there is.

I

Tim I sent the link to our cross cloud provisioner, it's kind of a lot of work and does it's a terraform based and I think personally, pretty well written terraform library for provisioning, kubernetes.

I

Okay, let's give you some patterns to use, and there is also a g gk NGC module. Okay,.

C

So Christoph sent a PR that is using the I guess the official gke terraform module. If you are familiar with it, you could take a look and and have suggestions on that vr.

D

What's the date range you're doing, Justin I did June 26th. Did you like my third, but you can june 26 to.

H

C

So I see a total of 39 dollars.

J

You know projects, oh.

C

Maybe not thank you.

H

Nice, like to all sharing all 20.

C

D

$45 all 20 projects, all products, all SKUs. Well, because you haven't up the discounts, I think you got to add them up it up the you know. Oh the pre promo from oh.

F

Yeah you've got.

C

The promos, not the disco.

D

Yes, shouldn't.

C

Your result include the discounts.

D

Hudson chugs and funds and duds in there. Yes, but if you have the discounts, you only get your sustain you sides, but you want you want cost me. You can add up this column and it's a track that some ones that come. If you want that. That should be those too good to see that call. So this sum looks roughly.

C

Like four hundred dollars, four hundred and four dollars, and but now you have to subtract out that these the promotion, the discounts, Oh minus 68, so three hundred and thirty-two dollars- that's great news. We disagree so.

D

Are we oh well? This is reassuring as well. I know, I agree. What's weird is okay, there's something going wrong here. My mice costs total. It says 272 if I add up the numbers in the pie. Chart they're playing to be bigger. Oh I know why it's cuz. We have this weird negative thing and they are in fact three hundred and thirty five dollars we're off by single digit dollars.

C

D

This weird thing that happened: oh.

C

Yes, that okay I see where you're wrong, so Justin's report includes the invoice which was credited because of the credit expiry last month, and so it probably shows up as a line item in the detailed report. So you just accumulated it, even though it's negative I.

D

Don't know what happened there, do you have any contact and why we had this? If you tell her yes,.

C

We there so the credits that fund all this expired but didn't auto, renew for I, don't know what reason reason and we didn't notice it for like a week. Okay, and so we got an invoice. So then we got an invoice and then I had the invoice wiped internally, but I guess it shows up as an invoice. Okay and.

D

That's why they broke our reporting right, because you can have negative numbers in the pie chart apparently, okay.

C

Interesting visual representation.

D

um But yeah so okay I will I will try to fix that. But yes, it sounds like we're. Basically in agreement agreement.

E

C

D

Digits I think so that it's roughly coming.

C

Exactly 3:30 or something.

D

C

And the vast majority of that is going to compute engine, which will be the VMS running in the cluster and the storage to Beckett $45 going to cloud DNS and penny is going to other things.

A

So I just wanted to check with a a any concerns on the costs. So far, all the billing.

J

Yeah must be mute er. We.

A

Can't hear you eat.

D

Obviously, no concerns then.

C

I mean I spoke with your and Dan last last two weeks ago, when the credit thing happened and all the numbers seemed ok, the surprise was that it was trying to build a credit card credit card with it. I mean he was an expired card. The card was rejected. They.

F

C

The card is rejected, but the transaction was rejected, so we should make sure that actually, that there's a reasonable credit card there with a reasonable backstop. Just you know in case I, don't know what GCP will do if it can't like talk to the credit card, but.

A

We should never use it.

C

A

Okay, that's it right for the building review, I! Think so. Okay,.

C

A

Anybody who cares just very just because I.

I

C

Numbers, 106 million DNS queries I just think. It's an interesting number.

A

Okay, since I see Christophe a Christophe, do you want to revisit anything about the terraform stuff? Are you all set just waiting for reviews.

K

Yeah I think I'm all set just waiting for reviews. I, don't know if anybody had a chance to review in the last couple weeks, but yeah that's right.

A

Yeah Tim we'll try this week and a hippie rocker will as well. So you will have two people I haven't made time either. So I will try to.

J

C

Just to be clear, if I can't get to it this week, then I will be out next calendar week, but then I'm back in perpetuity and and I will make it my tippy priority. Then thank you. Okay,.

A

So time to run through the milestone, okay, so before we go to the milestone, I do have one urgent request. There is one GCS bucket that Jason DT BER is not able to push stuff to. We tried all the things that we could at least I tried all the things that I could and I couldn't find it figure out alright so and running through the ensure scripts. For for that bucket I don't get any errors. This is the cluster API AWS and then he is able to push to cluster API.

A

So we compared the output of those two and it was nearly identical. So sorry.

J

This is the cluster API AWS.

A

J

A

Repository so he was trying to do a darker push to it and it worked before and now.

J

It's not working because there is one image there already specifically indicate staging cluster api, yeah you're, correct, oh yeah,.

C

Right, sorry, we have a. We have a conflict today, uh so I'll have to step out in a couple minutes to.

J

A

One of you will have to deal with that. Yes,.

C

I can deal with it I'm a little swamped today. I won't have a gap between I may not have.

E

C

Between meetings but I will load it up and see if I can figure anything obvious out right.

A

The email so the other people on this email, Google, Group thingy, also tried and Chuck basically tried and he couldn't either. So there is something wrong with the bucket aims we need to fix it.

J

Okay, yeah I would love to figure this out. Okay, thank you. Okay,.

A

Before we go through the milestone, did anybody else have anything else important or some time limits going once twice? Okay, let's go back to the milestone. The first one is create issue template for stating the pause three Christophe. This was on you. Will you get some time this this week? Oh.

K

Yeah sorry I did the I did the make file one, but I forgot that one yeah I'll make some time for this week. Thank.

A

You then enumerate our back rolls.

A

A

Hippie hacker: do you know what this is about.

I

Yeah yeah yeah separate sub ticket that we need to do at some point. Let's go back up today, yeah.

I

The audit script that we wrote there's a lot of api's that we don't yet look at. What's on what IM rules are there, and I think this is an extension of that of numerating over all of the our back rolls. Okay,.

A

Should I just CC you on this.

I

Yeah I get some clarity from aaron as well, I'm not sure actually what that was about right.

A

He is I signed it to a Tim and I. Don't think Tim knows I.

C

Think I think you're, probably right hippie. It was the like iterate stage of the launch and rate yeah.

I

A

Okay, so one of you will take a look: okay, let's go to the next one are know. What was the last thing here.

C

We agreed not to follow up on the script and instead use the terraform was.

A

That this one this was Google group, do we need the Google Group paste GK cluster on that.

C

Issue: let's leave it open as we look through the terraform stuff, we'll see if it makes it better or worse. We've agreed, at least in principle, to start simple right: yeah, okay,.

A

Table this for now, while we are looking at telephone.

A

Then GCI repositories for project to use, GCS buckets, I, think we've kind of done this. We just have to open things up now.

A

Okay Nikita had opened some greenies.

A

Okay, final tasks are in progress e to EDR. Okay, do we want to talk about PR here? Amy? Did you have something up 4br.

A

Do we have an issue or a PR.

A

Is out? Okay? huh No, let's go through the bottom. Okay, who creates okay run so right now we are creating both the GCS and the GCR stuff, given in a specific name, so staging.

C

A

C

I think I think that one's done from the staging point of view, it's unclear whether that issue is really about staging.

A

Yeah, this was just an umbrella issue. We probably should close it at some point and open specific issues. Okay, set up a job and kids infrastructure start to scrape I am policies. That's.

J

C

So this needs a review. Where do we leave that review? Is it in my court.

I

I think it's released for the emergent iterate is what we talked about.

C

There's a bunch of comments opened yeah.

A

He PU, you should go through it once and then we can watch it okay, he will do you will do we run. This. Can.

I

You sign that to me, so it shows little boy, okay, my list of things to find thank.

C

You isn't it your PR. Yes,.

A

A

It's not, ladies.

I

Okay, so I got a different okay.

A

C

Have a separate email filter for github, my PRS just so I keep setting.

A

On my coat yeah.

A

A

That takes take inventory of Google one posture based infra. This is pretty long-term, but I think.

C

Most of these at the bottom of this list are umbrella issues that are like being actuated in various other issues. Right.

J

Billing billing am.

A

Scripts inventory, okay, that should be it I did want to go through. That needs triage a little bit this time and see. There was something there hiding so.

C

Unfortunately, if just.

J

C

To step out, I got a conflict this morning that I need so I will take a look through the issues, but I ran a triage myself a couple weeks ago, and my thought was that most of it is up to date. If I'm, if I'm wrong, please assign me issues that need my attention sounds good.

A

Thank you Tim. Thank so.

H

A

I have the rest of you. He let's couldn't do a quick one and then I. Let you go early, redirect the service, probably not right now, right, Justin, I,.

K

Think that's pending getting a cluster up. Okay,.

A

K

Wanted to like there were just some concerns around using the the cloud service yeah.

L

K

We just wanted together to get a cluster up. Okay, I'd love, I'd love to work on that, basically, as soon as as soon as we do get a cluster up, because it's a small, it's a small service. So it's easy to easy to deploy. Once we've got a production cluster.

E

Like we still have to figure out on the edge Rene's redirector service, one of the initial things would be or one that is also like an easy thing to deploy in a new cluster or even just look into how things would be moving. Is the URL shortener type thing yeah and because it's even simpler, so maybe that would be an option below yeah. Okay,.

A

Do we have an issue for the URL shortener I probably probably have right, yeah.

E

There's two pull requests open and one or two issues or something. Let me.

A

Know yeah I see it migrated shortener to the text redirect from now.

E

Yeah yeah, so so one Margaret II, the shortener is basically the DNS part of it like handling the redirect data within the DNS that we already have. So we have full force paste, redirect additions and the other one, the other one that is open as a pull request, is to have the deployment set up for the redirector.

E

That's already also available. The question is, or that is what is still open. How do we or what's the level of verification? So how do we do stage deploys? What do we need to test to then out of deploy to something that is production? Currently, it just would deform manually sort of.

A

A

A

How often do we get redirect requests so.

E

The good thing, if we split it up so the redirector service has no data in it. It basically is configured once for the specific short urls and everything after that. Basically all the specific short urls.

E

If you don't switch from the method like switching from one sub path to like five to four months or something that doesn't need a reconfigure and therefore doesn't need a redeploy, the configuration itself is always pulled from DNS, so it just needs our dns to be in sync, which shouldn't be a problem, because that's already something that is usually done anyway. That's ours, Anna.

A

Okay, so this is a VIP right now right. Is that also waiting to be for a cluster to be available? Then we'll do a few more things before we, so it needs a.

E

Cluster, but currently, because we can, we can test it probably with and not with it go like it doesn't mean a cost of per se to get the the prerequisites done, for example, the three things that need to be figured out before we actually deploy it. The cluster would be the counter of the process and the deploy process, and also, if we enforce a back rules and how that, like, what's the standard on that, that's a general Koster thing that we need to be basically have figured out in general for all the services.

E

So it doesn't really it's a general thing, but that one is the first one running into it. Would you.

A

Right: okay: we can still merge this and work on those things later, but who do you want to assign as a reviewer? So would that be Tim or somebody.

E

Else I think Tim already reviewed, so both of them were partially reviewed and then were stalled because the the general migration issues take a to president I think Chris was already um reviewing and Tim. Yes, okay,.

A

God, thank you. Yours, okay, so GCS need a G suite for sharing tonight. I, don't know where this ended up. This was priority. Backlog it went to steering wheel, have to go back and check GCS web to see mcf infrastructure, but was there something you wanted to do here or.

A

Yep I can do it, but I still don't have an access to infrastructure. So okay, so you need a cluster okay, okay, so then the telephone stuff becomes important, so better form and then cluster, and then we can need the cluster. Let me add.

K

Maybe we need a label, an issue label for.

A

A

Registry mirroring and attestation I think this version was also need. The cluster just Brendon and Justin. Was there something some update from you both.

A

A

Sign those okay.

A

We talked about these two storage analysis for billing. This is a backlog.

A

See it is cool okay for Cuban artist named J's, belaying, okay, I think that's about it. Let's just see if anybody else's showed up or okay I think we've covered everything here. Jorge did you have anything specific you wanted to chase here on this call Nikita.

A

Everybody else talked about their stuff, so the two of you did. You have anything specific to race, I.

L

Do not okay! Thank you in my case, I just wanted to see if y'all had anything to to discuss with respect to a replacing all the bash in ah okay.

A

Yes, so Christopher did you want to write up a blob for that we can stick into the repository saying. Okay. This is the kind of thing where okay, this is the borderline, above which everything should be go below that you can write a small script or something like that.

K

ah Like like, create a formal proposal for our yeah and press stuff, yeah.

A

Just write something that we can agree on and stick it in, so we don't have to revisit this argument again.

K

Yeah yeah I guess I could do that like that the we it's still like an open discussion topic as far as what what we do in Hoover Eddie's kubernetes. So it's not like this is a settled standard, but I would, as far as the new stuff that we're doing for or improv, because a lot of this is like net new scripts. um Yeah I would just like to limit lip-lip limit the bash.

K

So alright I'll write up something I'll send it out and we can have some makes asynchronous debate on it and then hopefully, hopefully set a standard right.

A

Just one paragraph which will help guide whoever comes next after us, you know to make the same decisions that we would do yep. Okay, thank you! That's all I had does anybody else have anything else that they want to talk about nope going once going twice, you have 15 minutes back at least thank.

I

You for hosting them yeah.

A