Kubernetes SIG K8s Infra, 26 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes WG K8s Infra - 2021-05-26

Description

https://bit.ly/wg-k8s-infra-notes

A

Hi everybody today is wednesday, may 26th and you are at the kubernetes wg kate's in for a bi-weekly meeting. I am your host uh aaron of sig beard, also known as aaron crickenberger, also known as spiffxp at all the places we're all going to adhere to the kubernetes code of conduct during this meeting, which means we're going to be our very best selves, and as this is a publicly recorded meeting, you will be able to see this posted on youtube later.

A

If you have a problem with the conduct during this meeting, please reach out to conduct kubernetes dot io, where you're also free to chat with me privately on slack or via email.

A

Okay, with that, uh I'm gonna go through our recurring topics first and uh ask if there are any new members or attendees who would like to introduce themselves, so we can get to know you.

B

So, hello, all um I am I'm sorry new to this. My name is tyler um and I am interning at google uh under uh linus. So linus is my host and uh basically I am looking to uh probably do a few projects, but my first on the list is removing bazel from the image promoter, the container image promoter and we'll see how long that takes and from there on I'm just looking to help out and uh be a part of this stuff.

C

Welcome board that sounds awesome.

D

uh Also new here uh jimmy casino I work for synopsis, uh specifically the black duck software um part of synopsis, uh just looking for a way to give back in the community and saw jim's tweet about this group. So I'm here to help uh we run all google cloud. So hopefully I can help in some ways with what I do today already.

E

Plus a lot of brands more like zombies, more brains.

A

uh Yeah, that's uh welcome jim, really appreciate you showing up um so uh I'm gonna just move to our next recurring thing, which is to go. Take a look at our billing report and see, if anything, looks wildly surprising in it. Since I don't have tim here, we're not going to compare it against the actual console.

A

But at this point it's basically I I trust these numbers uh just as much so over the past 28 days we spent you know about a hundred and change dollars, um and you know, as usual, I don't have a ton uh to say here. Typically during this part of the meeting we might, we might have a discussion if things are suddenly going up and to the right very unexpectedly.

A

uh But this to me looks like a pretty periodic. People are busy getting kubernetes artifacts from us and using our ci to contribute to kubernetes.

A

Are there any questions here.

A

All right I'll cool on that, um so I unfortunately didn't have time to go through and do uh action item uh scrubbing of last meeting. Was there anything that anybody had from last meeting that they wanted to follow up on today or shall we just go to open discussion.

A

Okay, um so uh the first item here is about migrating to manage certificates for kate's dot. I o did uh is somebody here who put that topic on okay. I thought that might be the case, um so I'll just show it um uh so uh tim, hawkin and arno mukhem uh have sort of reacted to our yet again, we've gotten big scary emails from let's encrypt about some of our certificates expiring.

A

uh This seems to be due to issues we've had with cert manager, not quite playing well with dual stack support.

F

A

Both ipv6 and ipv4 addresses for all of our ingresses, so tim set up an experiment to use gke's managed certificates um crd basically, and that seems to have worked. uh So at some point I noticed he and rno just were like migrating things. Whole hog, so I've updated this issue to indicate uh which apps that are running on the aaa cluster, where we run most of our public stuff, have been migrated over to use these and which apps remain.

A

I believe the intent once we're done with this is to remove cert manager from the cluster there's a part of me, that's a little sad about not using like a fully open source thing and instead leaning on sort of a proprietary hosted solution, but at the same time, uh if we don't ever have to like jump on certificate renewal stuff again, I'm totally fine with that um this, uh and then I think that maybe just since ricardo is here, I wasn't sure if this means that the work that ricardo you have been doing on like monitoring and providing potential alerting on uh the the freshness or staleness of our certificates, who's going to be operated by this or if it was still relevant.

F

Yeah, no, it's not it's not relevant anymore, where actually I have three pr's. Two of them are about uh certain manager and one is related to actually using the google cloud monitoring for uh for our learning. I guess that at least the the terraform manifests for monitoring is interesting as it uses it. It can be used to monitor any any certificate. Not only the not not only the ones from the cluster so was something that I was discussing with team in the past in our channel yeah.

G

F

Yeah, because it it can use the the time being or something like that from from gc gcp to to monitor the the certificate expiration. So so, at least, if something went wrong, we also with the managed one from google we we are going to get alerted as well. I just need to review. I am seeing that arnold left some comments. I'm just trying to fix my schedule again.

A

Okay, that's uh that's totally fine um yeah. Looking at this now, I remember I had to knit that like at some point here, if we think we're gonna start using terraform to provision more and more things. I want us to have a chat about how we're gonna like organize everything, because right now things have kind of organically grown and I've been okay with the fact that most the terraform is constrained to one directory that manages our clusters.

A

But okay, thanks for that update, I appreciate it. I I will close the other ones as we are moving to the manager. Okay, that's fine! I've pinged them all from that issue, and I figured I'm gonna treat this issue as like the umbrella issue and I'll go through and close out stuff. That's linked from this once uh we're all wrapped up. Yeah yeah cool, that's cool too.

C

Yeah, I will do that, that's already open. Thank you. I'm interested.

H

uh Is the http 01 validation being used with sitemanager, or is it dns01.

H

It's http or what I I believe: I've had the same issues on http01, but if you were to try switching to dns01 and I think that those issues would go away, yeah yeah.

F

I I have discussed that with team, but I guess it was on the mailing list on slack, but the I guess the main problem is how much we will not trust the delegation of the dns to something running inside our cluster. This was the this was the discussion.

F

If we really want to add the keys that can manage our dns entries, because actually what I've suggested was hey. Why don't we just issue an unwild card and use for everything and that's the same case right because you can. You can issue a wired card with dns, but the problem is: is that we? We don't trust too much this like having this these keys inside our production cluster. So this was the main discussion that I I guess that that can be previously revisited later, but but now it's mostly because of this.

H

Okay, that's really interesting.

F

Thank you, yeah.

A

Yeah thanks yeah I'd say: if you want to dig more into the details of it, uh speaking uh ricardo definitely knows a fair amount and then tim and arno and uh bunners um are also good people to talk to on slack all right. So I stopped sharing my screen because I think we're basically good to go there uh next up. I wanted to hand over to cloudyu to talk about updates on making sure that registry is used as part of kubernetes. Ci uh are kate's in for registries. Basically, community owned instead of google owned.

G

Yeah, uh it's been something that I've been working on on and off again, so for the first item there for the kubernetes e3 test images, there was only one left remaining there, which is the cuda vector, add image 1.0 now, basically, for that, uh I basically had to go back and get history to fetch whatever that image was beforehand and I've just added it as it was then again, so it's pretty straightforward.

G

uh I think I also mentioned yeah I mentioned from which uh git sha I I got the image from and basically we're using the same mechanisms as the other httpd and nginx images to basically have the code, the new image and the good old image. So that's the only image left for that registry.

G

I've already added a job for it. We only have to wait for this for request to merge, basically, after that, it's going to be promoted and then uh simply replace it into into the quantities test suit. Here's image manifesto go.

G

But yeah, um as for the other one, that's a bit more um interesting. uh Apparently this got closed. I think this is because of the main branch rename it might be. I didn't close it myself, uh but it might have been. It's also says it's close, it's also much, but it's obviously not true um anyways.

G

uh The the question here was that if we can actually have uh privately uh authenticated registry, we can push images to and then use it uh into e3 tests, there's one particular scenario which is conformance and something that we might actually have interest in.

G

I didn't actually get to read the last comments. You've posted.

A

So I can recap real quick. uh Let's see, where did I put it here, yeah, so, basically right now, with the way that we have uh kates.gcr.io setup, um it's like the service that backs it is incapable of simultaneously serving some images publicly and some image images privately.

A

um So if we want to event like get to a world where every single image used in our e2e tests lives in the projects registry like in prod, where it's promoted and eventually it'll be mirrored, um uh we gotta dig into this.

A

It's sort of a requirements gathering thing uh for uh the ii team about like well, so does it even make sense to have one single try and make one single registry that supports both public and private images, or should we consider having two cross-cloud registries, one that's for uh public one, that's for private, um or do we do something like just just take the hit and use a private registry and just leave it hosted in gcr, and so people who are running conformance tests that use whatever the hard-coded key is in kubernetes source code to verify that, yes, their kubernetes clusters can pull from a private registry.

A

They'll just hit. Google they'll hit gcr, um I'm not like. I don't have a strong opinion on it right now, um but I feel like it's something that needs to be uh untangled or looked at. Does that make sense.

G

um I thought we only had to make this particular bucket private, and that would be enough, but I might be wrong.

A

But yeah yeah, I think it was like we could that's why I think I was getting it like. We could make a project, that's just that. Just makes this bucket private and then uh just this registry is private and that's great. But then that means like uh like not.

A

I I want us to get to a world where, like all of the images that are used as part of kubernetes ci, are in the cross cloud registry uh that you know we're trying to to work towards uh so that if somebody needs to get all the artifacts, they need to effectively stand up a kubernetes cluster and then make sure it passes conformance.

A

They can do that using like a single registry, but um since some of the conformance tests use a private registry, maybe that's not possible, and so then it's like, if, if we just have okay, we'll just have one private bucket on private registry like should we talk about making that private registry also a cross-cloud thing that the cncf hosts and all of the different mirrors and cloud providers uh host?

A

That, or do we just say we'll just leave it here in our our google organization right um uh and I'm fine just using the the one-off project approach for now. um If we do that doesn't seem like it's a staging uh project, because it's not so much temporary.

A

So I think I just write opinions from people other than me.

E

Trying to coordinate the multi-cloud solution, for example, for like the amazon registry and the microsoft registry and other ones, to have the same authentication method um for the same threads. um I think, would add an undue burden to the community. As far as running our conformance tests- yeah- probably it's okay to have this one off have a bucket. That is private and we have the way.

E

What might make sense is to have one at the registries and test the different it's a little bit of test in the cloud providers, but um making sure that kubernetes can pull authenticatedly in three different ways, maybe and just make sure that the three different providers that we communicate to use the most popular ways of pulling private, because it's not just authentication it's the ways of authenticating.

A

Yeah uh anyway, it's just been like low on my priority list to come back and, like uh just say: well we're doing this. So if somebody else has a really strong opinion on what we should do, you're welcome to uh post it on there and just drive us forward. I don't think I'm gonna object too much yeah. I don't know the only other thing. The only other headspace I was in was from like an air gapping perspective.

A

If somebody wants to test an air gap cluster, I don't actually know how much people are actually setting up a private air gap registry and a public hairdresser registry that they can conform. Its tests against.

A

So anyway, do you have any more questions on this cloud view.

G

A

And I think the other pr's are basically related to this thing.

G

uh Yeah pretty much uh the only one, that's a bit more is the one that also adds the windows on a server image into kk, also as the alpine, I guess that could be removed. The idea is that if we do not have this private registry, we basically have to maintain our own registry just for the windows ci's.

G

So we can actually pass the test since its conformance right, so the various having this merged upstream, then someone else in the community can easily build it and push it to whatever registry they need it to be in, but I can remove the alpine image from this as well, but yeah.

A

All right, uh I'm commenting on these just to kind of bump them up in my history, um I've been behind on sweeping reviews, so I'll try to take a look at these in the by the end of the week.

G

A

Awesome uh thank you for continuing to push on this um yeah. At some point, I do plan on coming back and uh trying to get us to push to close out all this stuff before we start getting anywhere near code. Freeze for v122.

A

um Okay, I'll stop sharing and I'm going to hand off to caleb now caleb since you're running a demo, I'm guessing you might want the ability to share your screen. Is that correct.

H

Normally I might, but not today and I'll, show you why? Okay, um if you all, would wouldn't mind, jumping on that demo.ii sandbox I'll, uh we'll all bring up the presentation and uh just don't touch it and you'll see why.

H

Very magic, and um can I just get a round of thumbs up? Oh thank you for sharing as well. That's also good. um uh So I'm I'll start the presentation. Now, if everyone is all good with that everyone got the page, look cool sweet all right. um Yes, I've. I've done some stuff on registry.case.studio and preparing a few implementations.

H

um So we have two candidate present implementations right now. We've got one using envoy and one using artifact server, so yeah, let's start off. um These are my ingresses for today, so uh you'll you're, currently on demo artifact servers where artifact server will be um there's also a distribution instance and then there's envoy, and we won't worry about the last one.

H

But uh these are the urls we'll be looking at. uh Oh, that didn't display at all. um If you'd like to jump into the teammate session in the back end, please copy and paste this link in the terminal, and you will join uh to be able to see what I can also see um without the web and it'll be hard to see. Yeah. That's a bit annoying.

H

um I think we have three people joined at the moment. uh I'll also put this in the channel I'll just uh continue on from the slides. Here's, the ssh one um cool, let's go through and take a look at the rest of the presentation, so ooh, that's fun envoy. So let's talk about the envoy implementation.

H

So here is some uh listen discovery service. This is some some basic bootstrap, it's a partial configuration. This allows us to do things like this, so here we have some some blue code that handles the rewrite. So the first three lines you see here, the first one is the default host. Second, one is the secondary host and then the third line is the ip address that we're going to change the host based on we're going to say on every request.

H

We will start off with a default, I'm going to grab the x rail ip from the ingress controller, and then here is where the actual business logic would find its way.

H

So this is where all of the matching on asns and all that would happen, but for this demo we're saying if the ip is localhost we're going to use the second address, and then we need to do a get because 302s only work with head and get as far as I'm aware and then we're writing a status of 302 and then writing the location and a body and that that's the basic implementation for the lua filter.

H

um I realize I'm going through this quickly and um if anybody has any questions, um I've got the chat open and I can answer them. Thank you, ricardo. This is, this is a fun demo to put together. um Yes, so what you can also do is you can take this request and we can take a look at what the response will be. So if you would like to copy this, you're welcome to I'll give you maybe 10.

H

Seconds: okay, I'm gonna, move on I'll, also paste it in the chat for uh ease of access.

H

Oh, that is not the right thing.

H

um There we go, there's the call come on, so um what I see is: oh, it's kind of cut off on your screen. There aaron so the um in in my environment, this pairing environment that you're you've joined in with a teammate session. We get distribution and then, if you're anywhere else, you will get the case.gcr the io url redirected.

H

So that's what you see there, uh I'm presuming it is giving you the second one for everyone. If it's giving you the first one, I'm very scared, but I'm pretty sure we'll give you the second one. As per the logic. uh Okay, you can even pull a container image, go ahead and pull pause. If you like um so yeah it just does the standard redirect thing it works as you would expect it won't reduce cost today, because it's all hosted inside the ii sandbox.

H

That's right, um so I've pasted the link for docker pull in the chat, so feel free to run that and I'll move on to the next slide. uh If you'd like to learn more about the implementation, please check out this blog post.

H

This isn't a new one, but uh this goes through all of the details on the implementation, a bit more than I'm going through right now, um so yep, that's where you can find that let's talk about artifact server, I like it. Let's, let's take a look at it, who came up with artifacts server. I am glad you asked that's our friend justin, who is also on the call.

H

um So I've currently got a pr which I'll talk about later on after I've gone through uh discussing the implementation, so um yeah I've got a proof of concept uh config. So what this says is uh we're gonna actually start at the third one. um What it says is uh if the if there is if there are no conditions, we're gonna default to falling back on kits.gcr.io.

H

So when we hit artifact server, we're always going to go to that. If we don't do anything special and then here what we're saying is if the ip matches a particular ip address again very basic for this demo, then we're going to go to distribution.

H

Otherwise, if the path is slash, chaos we're going to take you to kuba. So this is the last one is especially important for handling uh the artifacts and not just the way that can registries work. So that is the configuration and then here's the logic. This is not thorough logic. It works for the demo, so um we're selecting a back end.

H

So, on every request, we don't have a back end decided and so we're going to pick the first one and then we're going to pick the last header that matches if there is one that matches and we're going to pick the last backhand that matches if there's one that matches. So if you were to bring this up using this, these set of commands, then uh that might work um using um yeah. It works for the demo um uh yeah. I was saying if you.

H

Were to hit against this, it will go in the order of in the configuration we'll go cates.gcr to oh. If we match the header, then distribution and then the path chaos. So that is the implementation, and I have brought up artifact server, I'm currently in the teammates session. I believe bono has dropped a link to that if the web a web link, if you wouldn't mind connecting as well aaron, I.

H

Thank you thanks erin cool, so I have artifact server running right here and I brought it up with the commands.

H

um So what you can do is curl against it using the following command in the presentation that I've got up right now, which I'll also drop in the chat and for you it will give you kids to gc out of io, but for me, because I ran this inside of this pairing environment. It's given me, distribution.

H

So uh one person hit it, we have someone has hit it, so that's good pulling from somewhere um yeah, it's resolving things. That's cool.

C

uh Yeah, I'm hitting you from a cloud shell, so it is so yeah we're pulling pause.

H

That's pretty cool, more details on pause later today: yeah the foreshadowing, so that's cool, so we've just made a request through artifact server, redirecting to a particular address which is specified in the configuration, and we can also yeah this didn't. I I changed one thing really a few moments ago and then I don't know what happened with my configuration but um by hitting slash, k, ops, it's meant to.

H

If I run it manually, this artifact server won't have cars. ah Artifact server is only gonna have I know I know what's happening, why that's giving me a 404 because it's first matching on the ip address. So if I run this in my pairing environment, it's not going to work, but if, if you run this command remotely that's currently in on the presentation which I will also drop in the channel, it will give you a thing saying: access denied, that's a a response from amazon. That means it worked.

E

So that's a good.

H

Thing for once, um what's that, just something on the channel, I'm just kidding, uh that's the response, yeah! So, ah okay, that's interesting!.

H

Somehow it's matched on the path, slash k-ops.

H

I appreciate your words justin um yeah. I would like to take a look at the error that uh aaron has gotten later on I'll, just save that for later, because that is really confusing to me. uh Moving on in the presentation, uh I have a pr available, and this is the link to the pr if you would all like to jump on it at once. That would be wonderful, I'll, also copy a link into my section of the document.

H

So this is, it's still put for. Do not merge. So keep that in mind. I would appreciate any review just to say um if my implementation is good or not, um but the main thing that I would like to to know is which path that we should go down. Should we be using, should we be using envoy, or should we be using artifact server? I have good thoughts between both.

H

So I don't know if this is the right time to discuss that, but I would like to just discuss that at some point. That's the presentation.

A

uh Go back to the presentation there we go uh justin. I see you have your hand up established, yeah.

I

uh uh Awesome presentation: thank you, awesome machinery and good to see this stuff and thank you for moving artifacts over from very basic to something more awesome, which is great. uh I.

F

Imagine on the.

I

On the envoy ver on envoy versus uh artifact server or go code, I imagine that will probably depend on how hard it is to do the cider mapping stuff, which I imagine depends on some of the analysis that you probably are also about to talk about later on. So but I I think that the cider mapping is going to be the the real pain point here.

C

Yeah, um uh I think aaron has his hand yeah. I raised my hand. uh How do I lower me, whatever I'll.

A

Figure it out um yeah, I don't I don't know. I don't have uh strong opinions that your your demo made. Artifact server look cooler because it seems like it unifies image and binary serving and the configuration is a lot easier to understand, and I appreciate how live this was and and how much of the details we got to walk through. uh There's a part of me that still feels like.

A

uh Like I mean I may just not be involved in this enough, but I still feel like I'm kind of missing the big picture of like what um you know. What the trade-offs are that we're talking about here. um You know what is it that we're intending to throw off to? Are there any sort of weird uh authentication mechanisms that we need to be aware of? What are the? What are the critical user journeys that we're trying to address here?

A

So, like um you know, what are the the test cases that we're gonna throw at this and stuff um like? I feel like this is awesome from a proof of concept perspective and um since I'm a biased action guy, I'm mostly gonna, be like whichever one it looks like you.

A

Can get up and running uh most completely most quickly, um uh but at some point I I feel like I'm kind of lacking, like you know something in the scope of a proposal or a cap or something that's got like nice boxes and arrows that helps us understand, sort of the flow of traffic and what's actually getting hit. When puts my mind at ease that we're not talking about running our own infrastructure to handle ridiculous amounts of traffic that we're doing the thing where we're like you know.

D

A

Off to pre-built, ready-made things that are designed to handle large volumes of traffic and stuff.

A

um So I don't know if any of that random stuff uh guides your decision on artifact, server or envoy, but, like I agree with justin, it seems like the next big piece would be to figure out how you're gonna determine what to hand off to, because if this would be, if like making, that decision is going to be in your critical path. You're going to want to figure out how to optimize that.

E

H

Yeah, um I think uh that is very reasonable to consider that.

H

In terms of it being in the critical path or not, um the main thing would be uh implementing the the asn related stuff so which asmr, what source ip kind of thing. um I think either one would it would end up in go so it's probably okay, because we can just write, go wasm for envoy and then create a custom image out of that, and then it does pretty much the same thing um but yeah. It just depends on.

E

I think I don't think the data set is going to be large and the data structure is going to be fairly small. To implement in a in a library, says: um here's an ip uh give me the the 302 redirect destination, making that as small as possible and a pfc can plug into both of these fairly easily written as a go library.

H

C

E

We'll put that on the.

A

Priority that makes sense, like I don't know- maybe maybe I'm just like I'm trying to understand way more than I need to, but I I feel like if I was going to explain this to somebody at the cncf. I would eventually want to be able to provide them a clear, understandable diagram of like here's, where the traffic is going. Here's where all this stuff is running, here's how we're going to be shifting.

A

E

Costs or whatever just we're going to show costs in us in a minute. But I think um if you want the simplest yamly styles and that's how we kind of diagram these days, there's a folder that contains the asn mappings that we're going to try to make sure accurate and as best we can from the vendors saying. Is this accurate for your asens.

A

Sorry, I'm not even talking about asn, I'm just talking about like the life cycle of getting something from the project. I want to get a thing. Show me the boxes and arrows that describe how I'm going to get to the right place. Yeah.

E

um We'll also do that, in addition to the asm.

H

Yeah I'll make it a easier decision.

A

But again like I'm, just I'm giving you like my perspective as somebody like justin and other folks. Here, I know, have been more actively involved with uh implementing and like. I definitely agree we're still to like get that proof of concept going. I just personally still feel like I lack the like, but what's the what's.

E

The plan, so if I were the shortest version I can give, is we have cip the image promoter and all of our uh destination, how those promote from staging all of that up the staging stays the same, but image promoter now promotes to prod, but it would also promote to uh let's say amazon registry and microsoft registry and pick whoever else signs up. It's part of our promotion process to ensure that they exist at these locations um seems like a way. The other way would be to do.

E

Caching, though, it feels more authoritatively if we push and then declare what is there, yeah.

I

uh Anyway, I want to keep us moving and be respectful, so I just wanted to quickly mention uh sorry. um There is an early year, an early cap which I think brendan and myself worked on. I don't know if it's even a cat, but like a dock, so there is that um the other thing it wasn't as detailed as you asked.

I

The other thing I would suggest when looking at the like the request flow is to uh mark which ones are security sensitive and which ones are not because I think very few of them are security sensitive. I, the first one is security, sensitive and then all the others are basically uh untrust could be sort of completely untrusted.

I

E

We'll reach out for a follow-up on that justin. Thank you. Okay, cool.

A

um All right over to rihanna to talk about stuff you have slides to share. Are you? Am I going to click on another link or should it die.

E

We all have different ways.

A

I'm making you co-host, so you have to be able to share your screen. If you need to. I.

J

Am not that impressive skylab showed us now. This is just simple screen sharing. Okay, um let me show you sorry. I lost white zoom screen. Okay,.

F

J

Have is uh we import the data from the data source? As with this query up here, then we do a transformation where we basically take the epoch date in the in the log standard into a real date. We import these tables that we need the cs. Referrer actually have the image name inside so then we will take the cs. Referrer, we break it up, get it to get the image done with the resource, and then we um put the hash in as well. At the moment,.

F

We're not using the hash for anything specific.

J

um We might later on so, but the main thing is to see the resources. That was the question last time and here is paul's back and if you look at the images being pulled, the big blue bar on the left hand side. That is the amount of time that falls, got pulled in the period from 9 april to 25 may, and that is worth even though it is only 0.1 megabytes per image.

J

We've got 33, 000 gigabytes of data just for pause and a cost of five thousand eight hundred dollars, so the the top is the top amount of image being pulled um by a download volume and then the second graph that I've got is the highest download cost per image. So this is cost wise. So every time you'll see costs on the right hand side, so the most expensive thing being pulled is to proxy.

J

So last time you asked what is the point of the data and I think this kind of shows these are the important things to move um cost wise. That would be the biggest gains in the next slide.

J

um Basically, I swapped out company names. I've got the top 10 pulling ips and laughing a lion. Is our top puller of images cost wise? Basically, everything up to line eight is all one one company's um ip address pulling those and then, if you look at the bottom, this is a pivot table showing which images, unfortunately, all all of them, don't fit on the screen. But you'll basically see these other images.

J

So you'll see for um line item one device, a thousand seven hundred dollars worth of full, of which 800 900 almost dollars is for cube proxy and another 800 for for node.

F

J

Delete and then the other other, smaller ones for minor amounts of currency um friendly fox, for instance, you can see pulled uh 4, 000, gbs um 249 000 times they pulled hcd amd 64.. So it's just an interesting showing the patterns of things being pulled and, and particularly yes, it's.

E

Interesting to see one ip address, pulling one image and and costing 340 bucks justin. You got your hand up.

I

Sorry, yes, the uh well you're saying there is one like one ip that pulled this not like the entire, like, for example, aws like this.

I

It's just like one machine like a nat gateway, for example, in aws.

J

I

J

Unfortunately, I can't match is ends yet so that would basically make all of line. um One two eight go into one line, but I can't match those yet because of the complexes of getting aced internet is to match exactly because I don't want to put anything here. That's not exactly true. So.

F

Yeah so one ip.

J

Address pulled the same thing: 240 000 times in just six weeks, so I'm sure they've got the exact right copy by this time.

J

Then um yeah, so that's that's the thing that we got um then just for interest sake, which is not vitally important. Average pools per day is about 26 000 bulls, but 26 million pools per day from the registries, and that is all right in the next steps that we do want to do at the moment.

F

We're pulling the data manually.

J

F

The the query against it to.

J

Transform the data, and then our data goes into data studio, so so we can display it like that. So our next thing is to figure out exactly how to automate that data. I think just needed the data for the billing. If I'm not wrong, you helped me a lot with the information about data studio, so yeah, so automation of.

F

Feeding the data continue updating.

J

Then the transformational automation and then, if there's any suggestions from this group, what else should I show in in the data what other information would be not interesting but really helpful towards knowing what to.

J

I

Rather than chat type it I will, I will say the uh uh the billing report doesn't need an ingestion transform in just in time transform there is. um It happens at query time, um because this is the transform is very simple um the I did upload the scripts that I had used in the past to uh like insert this data into bigquery or did not even script the program. I guess um I think someone tried that out. I thought you tried it out, but yes,.

J

Did it share I'll? Have a look at that query again, so what I'm going to try is. I think you can do that the epoch to date inside data studios, that's simple enough and then I need to do a regex over the url for the image to pull out the resource name and pull out the the sha.

J

So all the hash so I'll see. Maybe I can do it in data studio. It will. If I, if I get stuck with you mind if I reach out thanks and that will simplify, so we just need to automate the loading of the data. So we continue to have the latest.

A

I can't uh I wish I'm trying to think of like it's a selfish thing. um I, what looking at that immediately made me ask was how much is kubernetes project ci costing like I don't.

A

I don't know how far it is to get there, and I don't know how important it is for the purposes of you know, registry across cloud artifact hosting, but I feel like I want to get a handle on like um you know how much of our projects can, like all the pull requests and all the jobs that we run as a result of that like? How much are we uh causing uh or yeah, to sort of differentiate?

A

The calls coming from inside the house versus the project uh serving its uh community.

A

um But I think it's yeah, it's just a curiosity. Maybe I'm.

E

Interested in that as well, I don't know the shape of the of how to grasp those pieces. It would require a deep understanding of all like everything to craft even to hold it, um because I I assume most of our ci is all run within our cluster, um but I know there's other pieces as well.

A

Yeah, I mean ostensibly it's, you know all the images to stand up in kubernetes cluster and then all the images to run tests, which, thanks to the work that cloud you and other community members have been doing.

F

A

All eventually be coming from one repo, more or less, and so you know a subset of images that we could filter against and then it's probably a matter of like I don't I don't know. I don't know if, uh because of the magic of mirror.gcr.io and clusters getting to use that in our ci uh requests are going to be looking like they come from those clusters or from other random google ids. I don't know so it's not it's not a super important thing.

A

It's just like I'm thinking of like what's a question I want to answer and then what sorts of things do I need to find out to do that? That's where my head went uh justin hand.

I

It's uh yes, it's very related to what you just said. uh The the thing which we don't know is uh well, I mean specifically like aws. Moving moving data to moving bits to aws is ten times as expensive as moving from gcp, it's sometimes as expensive as moving gcp to gcp, or something like that, um and so ci is a case of that. But also you know, people running on gcp are going to be much cheaper and I think the the interesting question from our point of view.

I

Looking at all this mirroring is, I think, like how much money will it save us if we set up a mirror on for the sake of argument, aws or digitalocean or azure and like? How do we prioritize that, in terms of money like how much money are we going to save? If we do that, um like, we have a mirror on g on gcp, but we don't have one on any of the best, and I think that would probably knock that. What would that change?

I

145 000 would go to what number if we had a mirror on it. On aws, for example, which I assume is that the the biggest of the the most bang for the buck.

E

Type thing we hope to answer that question uh in the next two weeks by getting that asn data. um There are multiple sources and finding something authoritative that covers enough of our of our target audience to be precise, um has been troublesome once we get there and we especially for the top we're gonna know who those top asn's are making sure within our community. We can create a pr that just says here: the asn's can amazon, my microsoft and digital ocean review.

E

Is there anything remissing? Is it inaccurate because we would love to just let you know how much this is costing you um and then we can anonymize it and say you know all the funny names of the list we'll try to keep those consistent going forward.

I

The uh the that's, a good that would be wonderful. The uh the bigquery uploader actually assigns uh tags. The cloud provider using um aws and gcp have a json resource. They publish um and azure also has one, but they don't publish it. You have to download it by hand, but anyway there are links. I found them all in a python project, so I will.

G

Try to find the name link find that.

I

Python project and then post in chat, much appreciate.

I

K

Ready yeah, so um I think I said this last time as well, but I I do work for aws, so they pay me to work on the kubernetes project, so anything that I can do or projects that I can push to help get infrastructure costs down. Feel free to. Let me know is this something that shipping like release. Images to our public, like registry, would help with that all.

E

I think that's kind of the plan eddie. I think it would be good for us to probably have an offline conversation and go through the data with you and give you access to see all the things so that we can be frank and direct with the precise pieces and get that story. You know that board around whatever the diagram of the life cycle, including deciding when it's coming from amazon, appropriately and and using the uh the enterprise grade, solutions that are that service, your customers, well cool yeah. Just let me.

K

Know what I can do we'll do we'll definitely reach out to you. Eddie.

E

Thanks for reminding.

K

A

Yeah, that sounds great, uh I feel like uh tim hawkin and myself uh would probably want to be looped in on that as well, um if not the rest of the chairs, um but yeah yeah. I appreciate the help. I think it's um I don't know at the moment. I'm not.

F

A

It with with great urgency, but I think it uh you know, just like we're sort of figuring out. How are we gonna wire all these bits together? I think we need to figure out sort of what is what does the engagement model uh look like for uh onboarding people into this uh process or uh into the clip yeah whatever into the club? I don't you know like uh I.

B

A

Know if we're, if we're looking at something like how we have like the the distributors, announce list for people who sort of met a certain bar uh when it comes to uh security, disclosures and things like that, um or if this is something that uh you know, it's even kind of unclear to me. How much of this we want to. We want to talk about doing purely at the project level uh versus purely at the cncf level, independent of the project, um but I yeah. I agree.

A

It's like it's a great time to have those conversations and sort of figure that out.

E

Here to keep the chairs looped in to conversations with eddie.

A

So, oh I just hot cornered myself into a black screen. uh I think we're at time. um Unfortunately, um so I think uh bear no and my agenda's agenda items are going to have to wait until next time um unless folks really feel like sticking around.

A

L

Feel specifically, I feel we're continuing the conversation. I think it was more to get the earlier discussion of what we want to do with terraform. It's kind of to I've got a great way of presenting my influence, and I would just want to see what are we thinking around?

L

We've got our gce cluster already in one set of terraform. We've got a difference of, therefore. Well, that's not just reconcile, but we'll continue.

A

F

L

I don't think we need to discuss it here. Okay, so.

E

Might take a look at the audit and see? I also think for the audit script having something that's simple to run and you can point it at project in a similar way. That sounds good.

A

um Okay, uh jim, I feel I don't know if any of this helped you understand, sort of where you can help or not, uh because I feel, like we've fallen into a bit of an anti-pattern of talking really specifically about some very tactical stuff.

F

A

I don't know that it's you've gotten a great picture of sort of what our roadmap is. What our plan is uh where we could use help and stuff.

D

Yeah yeah, I can imagine it will take a little while to get up to speed. But um I'm connecting a few dots here and there right.

A

uh I am, I am trying to like kind of uh keep things a little bit groomed, uh but if you have specific questions feel free to reach out on slack, otherwise you uh feel free to like keep sort of soaking it in uh ambiently um yeah. I uh I really appreciate people who have experience running a bunch of stuff on gcp showing up, because I mean I know I work for google, but I still am figuring a lot of this out on the fly.

A

It's fun um all right! uh Thank you so much for your time, everybody- uh and I hope you have a happy wednesday.

A

Thanks aaron thanks, everybody.

J

Thank you. Thank you. Thank.

H