Kubernetes SIG K8s Infra, 1 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes WG K8s Infra 2019-05-01

Description

http://bit.ly/wg-k8s-infra-notes

A

Okay, hi everybody today is Wednesday May 1st happy May Day right, so you are at the WG Cates infra meeting you are being recorded. You can all go watch yourselves later on a public YouTube channel as you adhere to the committee's code of conduct by not being jerks first off is there anybody new here feel like I, recognize all the faces and names.

A

Okay. Next up, let's talk about billing, but what has changed since last time, I.

B

Can probably take this? Why don't you start and I'll get the dollar by dollar, read out and say: okay, so I actually did I started using data studio, I'm gonna try to present my beautiful presentation for everyone. I prepared a PowerPoint for everyone's enjoyment, bring it up. Let's see if this works.

C

Sir did here he prepared.

B

A PowerPoint exactly it is indeed a PowerPoint, as you shall shortly see. How do I present exactly like PowerPoint I.

C

Don't know where anything is.

B

Is that too harsh? Sorry, all right, God.

D

Already present, yes,.

A

Are you using the client or the web? Ui I only know how to do with the client I'm.

B

Going to try to share.

E

A

Well, justin is looking I wanted us to try and go through some of our usual Oh there's even Comic Sans in there I wanted us to go through our usual schpeel, but then I wanted us to kind of go through things. On a github issue driven basis, I tried to call up all of our actual items into github issues, I think Justin, something we got on one of his but I'm so excited to see comics ons in serious business. So, let's see our billing report, we.

B

A

B

The Comic Sans this is Google data studio, which is actually really sweet, I think so it actually gets it's fairly easy to build reports, and you know you can it does pretty nice drill down and all this I have it. You can change the date range if you're. In the view you can drill down into the different surfaces so like you can then drill them further.

B

This is live data, so we can come I'm I'm looking for the day, doesn't exactly match what we're seeing elsewhere, but so there might be some reconciliation needed, but we can make to create whatever billing reports. We want I believe I can embed this into a web page or we can we're getting it delivered by an email, a PDF server by email each day, I, don't know whether we can put this somewhere else, make the data public or there's an API in there.

B

So Tim asked for a couple of reports, and so we have some drill, downs. The actual data itself isn't I would say terribly interesting right now, I have a ton of a ton of data America. You know you can see that we were spending 21 percent more on compute engine, it's a little over that's May!

B

First but I guess we spent 21 percent more on on compute engine in April than we did in May, so and 13 bajillion percent, more on cloud storage and GCR product, but still zero, but still zero I tried to get that to not be R and look failed. But yes, so we can basic rate. Whatever reports we want it's fairly easy to create and it's a fairly nice interface over bigquery, which has all our billing data being exported to, and we can figure out then, where we want this data to go.

B

Do we want to make the great public to want to put the PDF somewhere I see whatever we want and I will change it from Comic Sans.

A

Okay, so there's an umbrella eg out there, you don't ever change it from Comic Cons, there's an umbrella issue out there that talks about our desires for billing and I think the two areas we had talked about, where we want to be able to do billing by namespace once we get to clusters and we want to be able to do billing based on storage, possibly if we live in a world where every sing once us to store all of their artifacts, do we know like which SIG's in which sub projects gonna be costing us more money there.

A

B

We don't yet have namespace billing breakdowns, I, don't have anything coming on that front. We do I, don't know we're good at bigquery.

F

We can talk to, but we should definitely do names before name to face breakdowns for clusters and figure out how to integrate that. But even that aside, like getting the the three or four queries that we really want to have answered here and having us have, confidence in the data would be awesome.

B

Yeah and then yeah we should have be able to once. We have buckets and I believe we'll get per bucket breakdown. I need to confirm that, but yeah we can. If we can also just whatever data we can get into bigquery, we can easily aggregate and query in this relatively nice data studio thing is.

F

That we're still doing the readout from the the traditional yeah, because.

B

F

Friend, so we didn't have different, it is so I loaded up for just the last two weeks since we, since we met last and our top billable item is VMs a $85 worth. You have a date range I'm, Sergeant that wedding for 17 to 4:30 ooh.

A

Comparing it live. Oh it's not that good.

B

F

Have 8 so the this billing breakdown breaks it down by SKU and let me break it down by a product. We just asked you breaking them with MJ a product you ever buy SKU well, I can't see both screens. At the same time. Oh yeah, you do have a SKU. Let me break you, don't my skew then so I have 85 dollars in cores 42 dollars in memory and $31 and SSD PD who's using SSD who's. Getting all fancy I think.

B

We're always an SSD SSD PD.

F

Yeah I hope, so we should be using standard PD.

E

B

So there's obviously some reconciliation needed between the numbers that are coming out of the pending reports, but I will I will make those numbers match yeah. Well, we'll figure.

F

Out that I can't see the white on the yellow 3182 that matches no 31 7300.

C

F

Think that's enough yeah, so we can. We can work on reconciling this story. What do you have for DMS? Do you have do you know so? They're.

B

Not on GC I can switch to DMS I have $20 and 71 cents. Okay,.

F

I have 19 dollars and 89 cents, okay,.

B

So we're very close, yeah I think with some differences in I'm sure I'm gonna guess it's time because, like they, all the reports come in for a time range. Okay, it went by like starting time so.

A

Part of the reason, I care so much about billing is I feel like it's one of the critical things before we open up the floodgates totally.

F

Really humans, the.

A

Question is what do I mean by when, when is it done? When are we ready to open the floodgates I.

B

Think we need broad priority of numbers, not exact matches but like so within within a 1%, 2% I think we need to make sure that we have the reports we want, in other words like, if you want not, if you want buckets, we need that and if we want names faces, we need that and.

F

We should be able to sit in this meeting and instead of me pulling up the should just be able to look and be confident the world's reality.

B

And we couldn't actually just add this an idea. We could create a Google Group and we can post from the PDF to the Google Group and then anyone that wants to subscribe to doing that crew. Anyone doesn't want to subscribe and not swing it and get the PDF scaley, okay or whatever. Yes, yes, which.

C

B

Yeah so then you can just go and look we're not too bothered about.

C

B

Up another bucket they're, storing the buckets in the buckets exactly okay,.

A

So the daily reports to a Google Group to me almost sounds like a nice that house I'm, not sure I will make sure I turn that into an issue, but in terms of what we want to accomplish by next meeting. Can we get an answer on something billing per bucket and billing for namespace.

F

We can certainly do billing per block. You can get answers for billing permit, since we activated that yesterday afternoon and per namespace, we can maybe look at and find out more information on that. Yeah I want to be realistic here, two weeks from Friday I leave for kube Khan so like in the next two weeks. I'm not gonna, be doing a whole lot else, like very honestly, so I'm weary of over committing for two weeks from today. Right.

A

I understand I am also leaving for to Khan in a little over two weeks hooray, so it would be really great to kind of have as much like, be really close to saying we're, ready, not open the gates, so I.

F

Mean open the gates, we have like eight different things that we can open the gates, for we can open the gates for GCSE serving without a whole lot of fanfare. We can open the gates for GCR serving, but we need to move over the bulk of the existing GCR stuff. I'm noah lioness is here today. I can't see well.

A

So I think we're jumping way way ahead on that. Okay, because I want to come back to like what can we open the gates for? Let's look.

F

A

F

Ahead, that's exactly my point is like there's eight things that we can open eight gates for okay. What do you care most about, though I assume what you care most about is the cluster in the workloads I.

A

Honestly, just want us to have the ability to adapt if, for whatever reason, our costs explode and eventually be able to break it down by service and whatnot. It's easy for us to keep track of whoa. Our numbers changed drastically since the last time we read them out. What have we done in the last two weeks and then adjust that accordingly sure.

F

So I think we're we're on the hairy edge of that already with this report. If we just like polished it up a little bit, we've gotten most of the queries that we're actually interested in it and honestly, if people have queries that they'd like to see like send them to Justin or myself and we'll figure it out, I don't want to have 800 queries in here. I would rather have like six, so you know, I would focus on the last month, the last two weeks, a month a month and.

B

Then months and months right and it won with a 13 bajillion percent, which is a little annoying, but that one is the one which is the most interesting like each one is changing right.

F

And then- and that gives us an overall picture and then I think we can- we can quote- quote open the gates, but that said, the other pieces aren't all in place yet like we can't open the gates on GCR. Until we do the bulk version of all the existing GCR data, we can't open the gates on workloads until we're comfortable that the cluster automation is done, which I would like to have done for today, but just felt between two knee cracks. Ok,.

A

Great ok, so, let's move on to some of the other guys because I think you just you talked about one of them.

A

Hippy has gone ahead and done that so setting it up sorry, this is I'm not doing a great job here. Ok, before we get to a eyes, we also said something about reviewing what has changed in the last two weeks from an audit perspective, maybe it looks like you had it some things here.

G

Sure, later on, we're gonna look at setting up an automated job for this, so that the moment that it changes we have a PR to review or, within you know, a few hours every day. This is the only changes since our last meeting, somebody added the kate's and fred GCP accounting Google Group, and they added the role bigquery job user. It would be nice to automatically figure out who change that and add it to the audit.

G

That's the that's. The audit review for now. There's a link to the to the commit that has the updated data.

F

Yeah I don't know, yeah is the audit log queryable. In the same way, we can.

B

F

What I'd love to bigquery? Ok, it seems like it could be a follow up like simply having the the read out and actually hit. If you run it again, we put in a something for you to find. We left your breadcrumb I was wondering if you were gonna, run it just before the meeting or if it was gonna take longer so yeah I mean that's simply the fact that we have an alert that simply says hey somebody added so, and so. With this permission, like do you want to codify this into Yamma?

F

Loran is huge deal so right now it's running manually right. We can run it periodically once we have a cluster out looking forward to that once.

C

I'm happy with the cluster.

F

A

So I think that was actually one of the a eyes, so we don't that is still blocked on actually having a cluster of that we're happy with okay. Yes, that.

F

Is my next I don't know if you're gonna jump to that? Isn't a I, but, like that's my next mission, I'm happy now with the way all the GCR stuff is and all the GCS stuff seems to be up and we'll talk more about that I guess when we get to that topic. My next real thing that I want to focus on for this group is getting that cluster and the automation and all the config setup. The way we're happy with it again. I probably won't be two weeks from today.

F

It probably will be four weeks from today. Okay, we'll find out that ok.

A

Dims, you had an action item to update our Charter to put in some language about requirements for spending money on sex.

H

I wanted to believe for a week to see if anybody had any any more thoughts on that otherwise I'll take the language there and update the Charter and I'll push up here out today.

A

Sounds good I ever watched our last meeting and it sounded like we kind of came to consensus. There. I dropped in a couple things we wanted to clarify in the issue then and assigned to you. Okay,.

H

A

And then we were supposed to have a demo today of the use of the container image promoter.

H

Yeah I think Linus is not well, he did in a couple of days off. So that's.

A

Cool I don't and we we missed. We also had something about maybe doing sig doing until it's like released. We missed the boat on that last week to.

H

Be fair, Linus to the end of the last week here pinged a few people asking for somebody to try it out and he has stepped up yet so that's I deliver that's good or bad yeah.

A

He thinks it is. It is basically done yes, okay, so I will share my screen now. I can and what I wanted to try and do I had an AI to sort of like normalized how we're planning this stuff, because I feel like each meeting that we run it's kind of a slipshod scraping through the dock and trying to figure out what we were talking about last time.

A

That's so what I've done is made for milestones, I've, given them just arbitrary, due dates to talk about what I think we had been doing in the groups in section was proof of concept making sure we knew how to stand things up manually. What we are trying to work on now are all of the tasks necessary for us to say we are ready to open the gates and then I feel like the next two milestones are around migrating low-risk infrastructure and migrating all the infrastructure related to kubernetes tests, because I feel like the kubernetes test.

A

Stuff is kind of tangled up, so I felt like what might be most productive is to get this groups consensus on whether or not I have the right things in there ready to migrate milestone, whether they are assigned to the right people and like kick things out that don't belong here or add things in that. I do belong here, so I also have tried to use area labels to break the work down into some baked areas. To allude to James comment about like eight different gates, I feel like we can shard things up by.

A

You know what I'll do I'll do a different view where I can actually filter this stuff, while I'm talking but to filter things by access to the cluster. So like can we actually describe all of the google groups that were using? Do we feel like we have a mechanism to manage all the google groups that we're using? Do we know what high am roles and policies? These things are associated with that sort of stuff.

H

The question I had here was: do we want to move stuff to the G suite account instead of public groups right.

F

Yeah I'm not I'm, not sure how we can automate or really get any sort of visibility into groups and what we're doing with groups without using the G suite and the API stuff.

H

Okay, so I will ask staring for help so.

A

I feel like we got that I actually just made sure I knew how to log into the G suite I kind of needed to recover my password. That's all done now, I'm good. What I need is somebody who's actually willing to do the work of like integrating with G Suites things so well.

H

I will sign up for moving the public ones that we have set up for all the images I can move that first and then.

H

F

But before we do that, we should I would like to understand how the API works. I think Christoph mentioned some. There was a bug that was assigned to Christoph to figure out the API and how to do it through the API I feel like we should probably actually write a program for ourselves, either a CI or something that can actuate Google Groups the way we do for other stuff right now, so that we have a like a simple command. That's create a new group that initializes it with all the correct owners.

F

Although DIMMs yours is a pain in the butt because you have it set for don't auto, add me, and so we always have to invite you and then wait and then promote you, but just saying I didn't.

D

Know it to do that, so no idea how you disabled, how did you do that so.

A

Let me try and walk us through this in a crawl walk run fashion. First, we need to figure out what are the groups that we actually use today and who is in them? So, curiously,.

F

I am NOT owner of all the groups, but I think either you Aaron or you DIMMs are owners in all the groups or Igor. Is you three? Are the sort of template I've been whenever I create groups? I've been removing myself, which actually sort of bit me in the butt this morning? But so you three should be able to look at your groups and find everything that starts with Kate's in front.

F

Okay, I think we started all the groups with Kate's infra yeah.

H

I didn't assign me to work with Christoph to come up with the command lines being might as well. I will find myself up for that. Also right.

F

So if we have, then, if we now have a command line, that a select group of people can use to create and bootstrap a new group, that's a huge first step that we can use because we're gonna want we're gonna end up, creating dozens and dozens of these groups for all the different stagings and all the different namespaces and everything else. So, let's make that as streamlined as possible. So.

A

Just so I'm clear, I feel like step. One is: do we make sure we know what groups we have manually created in Google Groups and yes, that would be dims, I, guess and myself to go. I.

F

A

Look at all the IM.

F

Rules and see which groups have been granted IM access and apparently those are the only ones that really matter so.

H

Okay, I have that list. Adam.

A

Don't worry, okay, so I feel like I would like for us to start with PR in that list somewhere into this repo. So we know what that was this and then connects that would be I have an issue with dims in Christoph assigned to take a look at the use of api's I. Guess recreate those groups in okay.

A

F

So the question is, we can do it one time and just create them, move all the people who are in the existing groups to the new groups. Fortunately there's not that many, and just do it one time we can then talk about, should we have this, be a mobile reconciled actively somewhere right, like the rest of our GCP infrastructure, is not salvia amel yet, but probably should be over time.

A

I'm gonna say this sort of what we we've done with DNS we're like it's not fully good ops driven right, but at least somebody can run a command youth based on some files to reconcile things. I think we're trying to get to that first step here. Right.

F

Today we have scripts, and if you run the script, it will reconcile everything and the scripts are written in a way that they're supposed to reconcile, and it's not ideal.

F

But you know anybody who is questioning the state of the world should be able to login I'm working on it right on a ensure everything script, which just calls all of them in sequence and should be able to just say, ensure everything and it will make sure the groups exist, make sure the billing accounts are set properly, make sure their buckets exist and have the right permissions and policies.

E

Changes it can make yes.

E

F

And it's only one directional reconciliation. It's not deleting buckets that shouldn't exist.

A

Okay, so next in this area we talked about dumping, all the I Emeralds hippie. This is assigned to you. I feel, like you, outputted a bunch of stuff here, and you showed us a commit, are the actual scripts living breathing things that exist that are being done.

G

The scripts are in the PR we haven't merged this, yet I'll probably need to address the few things yeah.

F

There were a couple of to do is left in there right. Okay,.

A

So I couldn't quite find a PR. If you can like the PR to this HP that'll. Let us know that the script itself exists. Sure, okay, maybe also related to the scripts. There was something about setting up a group with just the audit permissions necessary to use these scripts, as this has been done already. That.

F

Is done, we have an auditor as Gruber.

A

Okay and then we had talked about setting this up as a kubernetes job and then later as a cron job. What whistle would the cron job end up like dumping the output to how long would it dump it to the what's? The repetitious part of this I think.

F

The cron job would create a pull request or something I.

G

Was actually in the midst of creating the pull request? um You can see a couple of the commits against the current pull request and are generated by a bot.

A

Okay, I feel like a model we could follow, is prowl at the moment, opens up a pull request to auto update itself and if it finds that there is already a pull request, doing that it just for submissions to branch, so that there's only ever one like bot, automated PR to update things, but it's the one who's human review. That's.

F

Great and then we can make sure that that PR gets appropriate priority and everybody's mailboxes.

A

P0 and like all the other piece here.

F

Once we figure out what it is, then I'll add a new big red label. Those.

A

Are all of the things I understand us to like really care about people will be open. The gates, however, I've heard him use. The phrase are back a lot.

F

Yes, I think missing from this list before we oh well again, it's which gate, if we're talking about all the gates I think we do not have a script that turns on a gke cluster yet right and we do not have a and that script will need to include the basic, are back enablement. The same way as our sort of template groups have some basic permission: basic commissions for a cluster.

F

We have, we have no script yet we have Buster's that we can use to base the template off of, but we have not written that script.

A

Right so I guess my question is like this: does the our back stuff get tied to the I am stuff at all, or is.

F

A

Yes, it does yes, okay, so.

F

We'll have like a group that is ster, editors or, and those people will be able to cluster and we'll have an example of creating a namespace which has our back for a particular group that connects it right.

B

We can do Google, Groups I am going yeah.

B

F

But will us we'll have a group the same way reason groups to control I handle use groups.

E

Yes, but they are both consuming the same to us as no that's right. The third of the time we'll go there so.

A

The are back, but our back roles would be based off of the Google Groups. They would not be based off of the I am worlds this.

F

Right, we're not adding service accounts for people will do all the universe, and you can use Google Group straight from our back roles. Now right.

A

A

F

Spelled access wrong.

A

See this is why I love doing this live and I will assign it to you Tim. This is between.

F

Justin and I- we will tackle this as our next big thing. Now that we've got the other stuff that I'm sure justin is eager to talk about. Okay,.

A

So, just to really walk through it, I feel like there's an umbrella issue to create the cluster. There's all sorts of detailed things and aspirational things in here, I feel like I want to shut this issue down when we feel like we have successfully burns down a manually, created cluster and have recreated a new cluster from scratch. So I have an issue open assigned to you, Tim and Justin, since you guys that you're gonna work on this last time, anybody has expressed interest in maybe shoulder surfing. If you get together to do this, yes,.

F

I really really wanted to have this done, but I forgot about the cap deadline. So that's.

A

All good, so just I will also wanted to enumerate when we talk about burning it down and standing it back. Up, do I have, but so we talked about our back policies.

A

We'd be standing up nodes with basically the same configuration. We also be ensuring that all of the manually running infra is also stood. Backup and.

F

Certificates come across as well. That's right! So, basically everything that's currently running in the non or non production production clusters it needs to be recreated so in the process may actually be turn up the new one and then burn down the old one. Depending on how important we think that promoter is and how long we think it will take I would I would advocate that we write the script and once we're happy with the script, then we do the burn down and turn off right.

F

So when we bring up yet another cluster sort of in parallel, while we make sure the script is iterating what we wanted to so.

A

I feel I just feel compelled to point out. I feel like we have arrived at this exact situation where it's on you Tim and Justin, and we'll get it done in two weeks, and then nothing has happened for those community members who are frustrated about this and would like to see us move forward.

C

Is there any way.

A

That they could contribute or help out or even do the work themselves. I.

B

Think you need to figure out how we are doing to manage compressed clusters. Well,.

F

Ok, yes, so we let me put your question on the stack for just one second and address there Justin's. Yes, we we know we need cluster is plural I, think it's fair for us to start with cluster, singular showing that that's what we're running from today and so just figuring out. How do we turn that cluster on automatically applying all the current best practices and all of the security knobs that are non-default through the various gke api is whatever is really what I think is the long pole?

F

So if somebody is eager to help with that I'm totally cool, if somebody wants to start putting together a script in trawling through all of the various security related knobs GK has a lot of knobs as things transition from you know, old ways to new secure, secure or ways, I, don't know what all those knobs are honestly.

F

This well I mean that's what we have for all these other scripts today and I. Think it's the shortest path to something deliverable, I'd love to help contribute to that. Okay, so I'm I would be happy to eat. Was that heavy sorry I didn't see? Yeah yeah I would be happy to give you project editor access to the the project or we could spin up a like a test project honestly just for prototyping, and if you want to iterate on that script, I don't know if you've got even a start off the script.

F

Justin or if you just want to look at the clusters that we've already got or even just go with fresh eyes and like let's look at what the various knobs that g-cloud offers.

F

Right so, like I know, there's a bunch of knobs in the UI that are checkboxes that have big warnings. That say this way. This is on by default for compatibility reasons, but will be off as of some date in the future. I don't know what the equivalent g-cloud flags are for all of those. This.

A

Just seems hilariously like a place where the community could help us out like we don't actually know how to use our own costume. How.

F

Do you use it? I just never use the command-line tool, because the UI is so I. Love will stuff. That delights me to do that. Those type of things- okay, I, think it's great and yes, Justin and I have been having an ongoing discussion about whether it should be a go program that turns this on via the API or just calling g-cloud through shell. My.

E

I

F

Preference has just been like do the simplest possible thing, which is to call g-cloud, which is already effectively debugged. I agree with you this. It was possible thing, but I believe that to me go yes,.

A

And as I told you guys last time we're using Haskell, it's the the end of that discussion. I think whatever gets us there, soonest and whatever the community as a whole is willing to make that's.

F

Right and so turning the cluster on is is step one and then making sure that we have just a comprehension of what we want the default permissions to be like which Google Groups specifically. Are we going to give admin access? The cluster like I? Don't think we have a group for that yet so we'll have to create the group figure out the naming of it and, of course, naming things is hard.

F

We've already screwed up a whole bunch of group names, and so I can't wait to rename them and just it's all those little details tying them together. So hippy. If you want to help with that, I am yes. I have been in the way I'm happy to give you access.

A

Like Arno is willing to help out as well with any so my impression is what would help enter it here? Is their ability to like run the script and see the cost to come up and then compare whether the cluster that came up looks roughly like the manual cluster looks like, which means yeah.

F

So here's here's! What I'll do right, I, don't know our nose, email, I, think I have hippies email in various other places, so I will create a I'll, just create a test project and we can and I will add you guys as projects editor to that project and you guys can go nuts iterating on turning up the the cluster and then we can compare notes so I know. Can you send me your email? So I can add you to a group.

F

Send it to me by slack or something or.

A

If you put your github username I can assign you to this issue as well.

F

But I need an email to put you in a group.

A

Yes, okay, so I feel like that will get us some amount of progress in the next two weeks. Hopefully so, just again to refresh the manually created cluster that we saw, R was temporary and we would tear down in ninety days. It still lives. It is still where our infrastructure is running today.

B

Yes and so most of his moved to cluster two, but yes, it is still there. The one of the promoters is still in customer. Okay,.

A

And cluster two was also manually created. He.

B

Was also created using the most efficient way, I knew, which was to write a go program. Okay,.

A

G

A

G

We move on from there the ability to access the current cluster to see to compare once we bring up the new ones or in this new project. But if we don't have visibility into the old project to see how the cluster like what it looks like I.

C

Won't be able to make sure it's the same shape. I can certainly show the yama I used to create the clusters. Yeah.

F

That should be sufficient out honestly I. Don't think, there's that much that's interesting there, it's more like look at all the knobs that are there and let's talk about what value we should set to each of them. Yes,.

C

F

B

I was I thought there was some support in g-cloud for applying a configuration without individual commands for G King I kind of find that right now, okay, okay,.

A

Next billing, so we had a task for Justin to show us data studio Justin. He did that array. Our next steps are to figure out what actual data we want from data studio. What are the queries we want? We talk about that being either per name, space, filling and then storage analysis I feel like we should rephrase this to maybe be like per bucket billing. Does that sound, fair or it's.

F

Either per bucket or per project we're gonna be.

B

I'm, probably gonna have we're gonna storage gigabytes at rest, build and we're also gonna have bandwidth, delivers yeah.

C

B

The bandwidth delivered is gonna, be the bigger number, and that will be if we're all going to share one prade bucket. We're then jumping become that will be a per two levels: directory or per directory structure for profit.

F

Yeah I, don't know what the billing data they give you for GCS serving is I, don't know granular and so I guess the action items really to explore that and figure out if one bucket is actually the right answer or not from the billing plan. Do ya. Yes, so.

B

F

To throw some test data in and throw some low a smart official load at it, and we do.

B

Want it / to kubernetes sub projects like returning cops and we turn on Cluster api. We want to know how much is cops bound with costing us versus cost reggae band, I presume I presume. So. Yes,.

F

Or don't care I mean the unfortunately GCR today does not give us that breakdown right. So if we can do better with GCS, then Bonzo. Let's do that so we'll we'll take the action, then to just throw some test data in with some prefixes that we think look vaguely realistic and throw some artificial load at it, and that's data artificial though, but download it a thousand times and see if we can run up a bill just.

F

E

F

Not zero or thirteen trillion dollars. Okay,.

A

I have asked a bunch of questions in this issue to basically Express that it sounds like mija. We need to figure out what couple of granularity you want. I. Think Justin, you articulated well, but we care about bandwidth, delivered and storage at rest, and how do we shard that appropriately I mean this project in some way to shard it by sub project? Is some project rolling out to sake? It's probably what I care about?

A

Okay, then we'll it will ask some questions about per namespace billing. Tim can I give that to you, it's kind of like you or somebody needed to ask so.

F

That, yes, though, this will serialize, obviously Diamond clusters, no I guess it doesn't have to. We can play it with the existing text clusters. Okay, assign it to me, I think.

A

It's basically we're trying to ask: is per namespace billing even possible. It.

F

Is possible I'm not sure what the time frame for it is or how it integrates with the rest of the billing data I have just not used it myself. Okay,.

I

But isn't the latest advanced version of GAE on G called basically having billing? Yes,.

F

But I don't know how it integrates with the rest of the Billings data like it is entirely likely that the team has done a great job and it just integrates seamlessly having played with it myself. This is okay just check over thanks.

F

Yes, also remember we we have access to information about future stuff that I have to be careful about. Yes, no I I understand, okay,.

A

So I feel like I, will close the presents data studio thing. I will open up some issues for some of the follow up ideas. We discussed and you'll call the umbrella issue closed when we feel like we are content with billing to be able to open gates for certain things to me, it seems an awful lot like this. The words question defines when we open the gate for a bunch of sub projects.

A

Related artifacts I feel, like that's, also, probably gated, behind the use of G Street to effectively manage all of our high Emeril's, but it could be like just to get a cluster up and running and get and make sure you're repeatedly reusing it and running stuff on it. Maybe we don't necessarily need super granular building, yes, any any other words I'm doing okay, so the the area that I'm least informed on right now and I'm, actually gonna drop. The milestone is all related to artifacts that we store things I have in the milestone.

A

Right now are the image promotion process seems like we want that. We want to prove that that actually works has basically said that it is good to go. We just need somebody to open a PR and go through the process, so.

F

Lionesses said that it works and he thinks it in a mobile state. He has done demos himself. The question is: are we comfort of what I would call a lot of testing in terms of like integration or end-to-end testing? Yet we've set up the end-to-end test project so that he can do that work to the best of my knowledge that work is not done yet.

F

Are we comfortable turning this thing on live without that level of testing, given a it's sort of already live on the end version and the the results of it going haywire could be catastrophic results.

B

Women here are only if we cut over-serving.

E

B

F

Just as if absent we can do demonstrations of it, but we're not gonna have the gates that GCR that I know well facade one.

B

Of things we could do I think he also wrote the dump from a GC. Are we go to a text file? Yes, so we could turn it on repo to and then dump golden repo and repo to and see that they are the same I.

F

Don't think I followed you, but there's there's various steps we can do before. We turn it on and flip the facade over.

B

Before we head over I'm saying, we should run that number and make.

F

Sure the two GCR repos.

B

Oh identical sure.

F

Yes, that's a that's a good thing. My concern is that there's some net new code here it wasn't just an open source port from the internal. So and you know the results of that going bad could be bad, so I try to make sure we have appropriate safeguards in place before you flip that over and I've.

E

Spoken to Linus about this before and he agrees.

F

So so he's not here but in absentia, he's.

A

Alright I think I have expressed Tim's concerns here to strive that we still.

B

A

There's more work to actually flip the switch I'm.

B

Going to say, are there any new repos new projects that we could.

A

About wanting to use this, just for court huber, Nettie's components like rot before we thought about using this for other sub projects who want to store their container images. Yeah.

F

I mean we have this vanity name for King Stach e CR that IO. If we want sub projects to have different vene names, that might be okay, but that would be different like we haven't, crossed that territory yet and.

A

So the thought was anything that currently lives in caves thought GCR that IO we want to move that stuff over first, so be all the images that make up kubernetes. All of the like test images that q burn ad kubernetes uses to estimate, runnin and and tests things like that. That's.

F

Right, basically, anything that does.

A

F

In any of our core, any of the repos that feed into the release is.

A

So then justin you have all sorts of stuff related to artifact storage in general. That's beyond just GTR is that I don't really know how to quickly bucket, because I feel like we have a lot of mega sounding issues and I want to understand in terms of what do we think of opening the gates in the context of like artifact, server and redirector, and all that stuff. We.

B

We do have a plan to do an MVP, so we're the we are not gonna, create a initially gonna create a redirect or service that is smart, we're just going to prototype it with GCL, be Tim. Thank you, Tim for merging some pr's and we have created GCS buckets, I, believe for all the staging.

C

Reason, everyone and.

B

We have created a single GCS bucket for prod and I believe we have mostly created a GCL, be in front of that we haven't done the name, the dns name right, but if we like or in fact up que sería, we can point that name to it and it should be live. When we do that and then I have also created a PR against the image promoter to do binary, artifact promotion, which I will ping Linus about soonish.

B

And then, if we like the idea of promoting non image artifacts using the same image promoter, then we can. We can do it that way and that should what.

F

We don't have yet is a globalized there, whereas GCR will automatically send you to the geo replicated GTR repo. This does not so I think we need to think about.

B

That, well, we've been yes and what we believe is. We believe that GCL B will have points of presence globally, yes, but it will go to a there is backed by a GCS bucket that is located in multiple regions that are all in the US. Multiple.

C

Zones that are.

B

All in the u.s. correct, so users won't notice it in Asia, for example, but they will presumably see slightly increased latency and we may or may not see increased transport costs right notice. The disappeared Asia would go.

F

Offline, that's right so I think the billing step is the first part. If we can get visibility into that, we can see how much it actually matters, and then we can figure out. If there's a story there that we can put together.

A

So Justin, when you reference a the MVP I'm looking at this issue and enhancements talking about milestone, 0 VP, is this what you are talking about? Yes,.

B

And I think we've basically done that.

B

We've done that and we have a PR for I, think I, describing that that we would manually promote, but we have a PR for for automatic promotion, which is not yet worst.

A

Okay, all right I feel like I'm gonna, try and take some of these broader, like set up a GC our repo and set up a GCS bucket to try and describe it in terms of GCS buckets for use with the MVP versus like GCS buckets for everybody, I, don't I, don't think we're doing that and feel like the GCS bucket issue. I could consider closed when I feel like we have done enough.

A

Gcs set up or your MVP, which I feel like we have I just need to describe it and the GTR stuff we're going to consider close, because we already have all the repos in place for a container image loader to do its thing. I.

B

Believe so Tim we have. The Reapers needed for containers for motor. Does.

F

Y'all read this: all the repos are in place so like we can do a demo of it today. I pretty sure we just don't have the testing. That's the really. The only thing that's getting me from saying: let's start the ball conversion process. Also, we should define a conversion protocol of life. I, don't know how long it's going to take to the bulk import.

F

It might be fast, it might be slow, there's a apparently there's a backside copy that maybe we can use, but if it takes a day we have to make sure that nobody pushes anything for a day. What we do it can.

C

A

Yes, when we do.

F

A catch up pass right.

A

A

And Tim's there's something here about registry mirroring and attestation or container images for the project. Also Justin and Brendan Burns was named garbage.

B

I think this was something which we need to figure out, which is I. Don't think this is MVP and I know. That's it's a mission which is like, so how do? How does anyone know that the image that we're talking about is the image that we say it is and I were ready to talk about this I know they were ready to tackle this yet ok, I'm getting there relying on this is on TLS exactly we.

H

Need a backlog, my stone yeah.

A

So I will leave a deep milestone. I will create a like nice to have, but I feel like there's something beyond all the tests. That's like just nice to have so I'll put in there, but that's why I'm, leaving it milestone list and I will go ahead and had these two to get ready to migrate, milestone to remind me to break them up appropriately.

A

Ok, other thing I wanted to talk to this group about real quick, was docker hub, so I.

A

Gracefully accepted credentials to docker hub from Tim I noticed we had all these images, they haven't updated him forever. I think the recent security breach and we demonstrated why we don't want to be on dr., have anymore so I removed our presence entirely and waited for the whales of discontent and then I think I bumped into one. Somebody said like broke their testing.

A

Specifically they used OpenShift 3.10 and had it couldn't pull down the kubernetes / cause container, and so now I am chasing down currently in back channels, but I'll start to surface it. Just where is kubernetes paws hard-coded? How? What downstream projects does this affect? What are their support windows and what reasonable guarantees can we make? It seems like if we may not have actually busted.

A

Certainly it like open shift, proper I, don't think we, but they haven't, had kubernetes paws in their core code since 1:9, but it looks like we didn't actually fully excise that cause container from our test code until October, 2018. So conceivably, we have some back porting to do so. I feel like I might have to keep the stupid docker hub account around long enough for us to rid of all references to the pods container yeah.

F

You know if, if paws is the only one there, that's a pretty reasonable situation, especially if maybe we just push a new one up there to make sure that it hasn't been compromised. We've already changed all the credentials. Well, they can check the chanson changed if we have. If we know what the shot was, don't forgot to tell you when it was asked, updated, I presume if it's compromised, I'm, not trusting anything yeah.

A

The thing is: I restored, I restored. What was there before so I could have restored a compromised pause thing. I mean the doctor hug. You I told me that it hadn't been updated. You know for three years so yeah.

I

A

Yeah we could push up a fresh glass container. I just want to be very clear and explicit. If I do that, then that this is not about maintaining that pause container, it's about putting in a stopgap measure for those people who can't get off of it. Just that's right so and honestly pause.

F

Might be the only one everything else is is really ancient pause, probably has not honestly been updated in three years right.

H

There was one more image in documentation, and that was easy to replace with something else. Okay,.

F

Thank you for doing yeah.

A

So did you people like that structure of running through the milestones and stuff? My thought was to you next time make sure that our project board actually has all these has just the current milestone. If people start adding new issues or other stuff, I will punt them to other milestones, it's appropriate and if we have time we can get to those, but so, for example, you notice we haven't talked about the go decades thought I/o replacement.

A

We haven't talked about the redirector, the nginx pastry director being something else because I agree, those are great, but we really should be focused on unblocking and opening the gates. I agree, I, agree, I, agree, I, agree, okay, the only thing we didn't cover, but just so you know, I started taking inventory of all of the cluster based infrastructure that we originally pulled together in that dock I now actually know what clusters they live in. So you have a chance to look at this and see if anything is missing from this.

A

I also have a separate issue to go through all of the Google projects that are used to capture non cluster based infrastructure. So things like bigquery are probably the main things I'm thinking of, but also what are all like. Gcs buckets that we're dumping all over tests are sacks in things like that. Okay, that's all that I have awesome.

F

We're we're also at time thank you for getting them, kicked yep.

A

Okay, so if you have a you guys, they're signed to you see you in two weeks thank.

G