Cloud Foundry CF on K8s Forum, 8 Mar 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Cloud Foundry on Kubernetes Call Recording 08 March 2023

Description

[GC] GC
Struggle with inconsistencies with deletion APIs across registries
capi-k8s-release deletion. Special cased for DockerHub, but never worked well https://github.com/cloudfoundry/capi-k8s-release/blob/main/src/registry-buddy/image/generic_deleter.go#L63
https://github.com/cloudfoundry/capi-k8s-release/blob/85617d8e19f170c8ea6476bd08273e0a323654a1/src/registry-buddy/image/dynamic_deleter.go#L19
https://www.cncf.io/projects/harbor/
[KB] Direction for kpack Image relationship with App / Buildworkload?
Kpack Caching Enhancements
[DW] Feedback request for the payload validation exploration
TL;DR is that we think that github.com/jellydator/validation is the best option. It is an “actively maintained” fork of github.com/go-ozzo/ozzo-validation, which hasn’t been updated since 2020.
[DB/GS] UAA integration prototype, explore story: https://github.com/cloudfoundry/korifi/issues/2243

A

B

C

I can uh kick this off really quickly, but I'm gonna probably hop off since my apartment building is doing firearm testing today so any moment now it will start making a lot of noise.

D

Yeah I know worries I can I can do it.

E

E

D

Yeah so I'll leave it another two minutes. Otherwise we can just talk about that. One thing and it looks like you know: people also have to drop soon.

D

So if it's a quicker meeting than usual, it's not bad.

D

D

um The first one was an attempt at upon, but also it's the thing we've been working on in the last couple of days, so we've been working on the garbage collection stuff and we picked up.

D

The the first story was about deleting container images and in particular, the package container images, so the one with source code when when a package is deleted, and we really thought that we could get away with just using the you know, registry spec apis for deleting images, but it turns out there is a lot of differences and inconsistencies in the different implementations.

D

So we've noticed, for example, that some Registries will delete an image if it's untagged, they consider it dangling and they'll kill it, and some want like Google will not, and then we noticed that some Registries don't allow you to delete stuff period. They've just done they just disable that API you get an RSA and deletions are deletes, are disabled.

D

um Some will refuse to delete an image if there is a tag pointing to it.

D

So you have to delete the tag first, uh if you can, if you have permission so at the moment, I think we have a piece of code that tries to delete the tag and then, if it succeeds, it tries to delete the image.

D

Actually, if it fails, carries on tries to delete the image if it fails, carries on so it's very resilient and it achieves different levels of uh I guess success, depending on which thing you run it on we're. Just thinking like. Are we happy with this because I I guess more stuff could be achieved if we explored, maybe proprietary apis, so I don't know. Maybe Docker has its own way of deleting images. That is not.

F

D

The one in the spec and the other thing that came up was actually like at the moment for packages, for example we're not tagging every single package we push, so it means on some repositories we're actually using old packages while we should be retaining up to five I. Think is the default, but it might be that at the moment we're not doing it just because we're not tagging them explicitly and some some Registries might might be deleting them without us.

D

Really knowing and it's true that we don't implement the rollback kpis, so uh it's actually hard to go back and build an old package uh with the CLI, but I think we still want to retain them in case people want to do this right, API or I. Don't know they want to maintain the different five different versions of the source code in the registry, so one fix we might have to apply. Is we want to Target each single package? We push.

F

D

Then, but that's easy, the main question we try to answer now is: how do we implement this deletion procedure, like a deletion procedure that we are happy with, as we consider acceptable across the Registries given they're? Also different I think we've tried Amazon Docker hub gar, the Google artifact registry yeah.

G

Is the one so far that doesn't actually work, so you can't do a delete on Docker Hub via the P2 API. You have to go through docup's own API. To do that.

E

Is there an option to delete the whole repository.

F

E

G

E

D

G

In the API either so you'd have to go custom for that anyway. Yeah.

D

That's definitely not part of the spec uh and it's weird because, like you get it created for free when you push, but then you cannot take it, you cannot remove it like. If you delete all images, it doesn't go away so.

D

That's also funny because doesn't mean we have to delete it. We because we didn't create it so technically should we be the ones who deleted, uh but that's a different story: the one for deleting repositories.

G

Also I mean yeah I, don't know what was I gonna say it. You really want your repositories, your Registries be quite close, so you don't get any network latency and things like that, um but I suppose people do would tend towards using like ECR if you're on AWS or one of the Google.

C

G

If you're on Google Cloud I mean, but there is the option of suggesting people install a local registry and then sort of give premium, support to that and make sure that works solidly so in low Harbor I, don't know if that's open source or.

E

There is a open source version of Hardware, isn't there yeah.

H

It's like a cncf thing.

H

So it's like I, don't know pretty legit open source.

D

The I don't I, don't quite understand the relationship with all between all these things, but like there's, a distribution, slash distribution repository which technically contains a reference implementation or of sort of this of the distribution spec and we've been using it for tests because it can be run in memory. But that's a that's a fourth Behavior.

D

Uh It behaves differently still so.

D

That's another option like if you want an open source thing to run, and then there is the thing that we installed on kind. For example, that's an even different thing like that's a docker, but it's not the one in distribution. Slash distribution is a different one. My behavior.

G

Docker slash distribution instead.

D

Right, but these two things behave differently than Docker Hub. Maybe they behave the same between them, but I'm not entirely sure it's it's a mess like this spec is very vague and pretty much. Everyone has implemented the bits that are not explicitly specified in this pack in a different way.

G

Google artifact registry, you can't delete a manifest if it has any tags pointing to it and on ECR. If you delete a tag, then the Manifest, the image gets deleted. If nothing else is pointing to it so cool, very weird stuff going on yeah.

H

I've left a comment on your PR the other day, just because I I remembered we we try to solve this in like 2020 and see it for kids and uh eventually, like kind of gave up, not not intentionally, but I had optimistically hoped that uh this whole landscape had improved because it looked like you had a PR and stuff, but it sounds like it's just as bad.

G

We could do like reasonable effort there. So like yeah, it's Docker Hub, that's causing a problem at the moment, but we could log the things aren't being cleaned up. We haven't tried Azure and GCR.

H

I know uh with what Julian just suggested make this simpler, though, where you only delete it. If the app is deleted, um because then you don't have to worry about trying to delete by digest or like hoping that there's a tag and everything.

F

D

Have to delete the whole repository, but there's no genetic API for that, so we'd.

H

Have to go sure so you'd have to go Ultra special case. The.

F

Reason I'm putting it is.

H

Just because, since we're pushing everything to the same images like I think the amount assuming stuff is reused between these pushes, like the amount of cleanup you get from an individual package being deleted. Isn't that.

G

Great yeah and we we already have that sort of specialization for creating repositories, because we need that for ECR, so we'll definitely have a custom deletion repository for ECR.

G

So the pattern exists if but we'll have to explicitly just support a set of Registries, and you just have to hope that well I mean if you introduced a new registry, then it wouldn't work with karifi out of the box really well. This lesion wouldn't work anyway.

H

Perhaps by the set like, if you consider like Harbor as like a almost a reference implementation of the the spec and then support like the major clouds, maybe.

G

Yes, there is no common spec for repository creation and deletion. It's only the Registries just give you a custom endpoint for it. There's this thing called the V2 API, but that allows you to create images. Create tags, delete them upload stuff. This is the blob. If you know it's ID or something, and it's it's literally about that um the repositories don't enter into it at all.

H

Yeah, that's fair I! Guess we never did try to do the repository I'm, just remembering back to CFR, Kate's, Harbor and GCR behave the same almost at the time, um with one caveat or one that you delete by digest and the other didn't. But all right, I don't know like I I, definitely remember, Docker, Hub being the worst of them all and.

C

H

It is not something people would actually use yeah.

I

H

More of like a kicking, the tires type of registry for something like karifi.

A

Yeah, what concerns.

D

Me with going with the proprietary approaches, is that that authentication becomes funny like no. Now we already get, we know we already have credentials to push and we want to use them for deleting it and stuff, but now we'll have credentials for pushing which are the regular financials like passed by basic course using this using the spec protocol and then for delete for managing repositories. You need its own authentication like for ECR.

D

We got away with it because the whole point of it was it's all via service accounts, and it recognizes that it's an Amazon node and it does everything magically but like if it's I don't know something else. You need to authenticate. How do you even do that, like the that sounds, doesn't sound easy like you have to? If you want to use GCR, for example, you have to allow us to I, don't know mess with your GCR repositories,.

D

Like how do you do that, maybe service accounts or stuff but like that, will need to be configured in Korea.

D

Yeah, maybe we need some more exploring on this. We really thought that the distribution API would just suffice, but it clearly doesn't.

E

So nobody's um written a library that solves this, for us.

D

Oh, there is the go container registry library is pretty good, but to the same calls, uh Registries reacts differently.

F

D

The container registry doesn't have all those differences between implementations in it. It doesn't it just does what the API the specs says. So the spec is a deleted endpoint, but then you should run it against Docker Hub and you get an error saying deletes, are not allowed.

E

D

That part of the spec or not arguably, but it doesn't work and that or sometimes you try to delete a tag again. You get an error saying: oh, the tag is already I'm. Sorry, it trades the new damage and it says oh there's a tag pointing to it. Spec doesn't say anything about.

D

You know preventing images from being deleted. If the, if a tag is pointing to them like they just decided they, they would do it this way.

D

So the library implements the spec end points, but it doesn't handle.

E

It doesn't have any of that proprietary stuff yeah. It's disappointing.

D

So I guess we'll keep trying and.

D

And yeah we, we might settle with this kind of best effort approach where, like okay, if you registry, it doesn't allow us to clean up after ourselves, we want.

D

We should probably document these things.

H

F

H

I think even just documenting how these things are named and that someone could clean them up themselves would be good too.

D

And for registries.

D

It's tricky because for registry we don't have any other way. Unless again we say we don't do it.

D

So yeah it's gonna be a tricky one.

J

Although that's critical, if it's empty right, the blobs are more critical because they take up space so.

D

Yeah, it's fair enough! That's fair enough! It's annoying! We! We live like this master list of empty things. Behind.

D

But yeah in terms of like space, it will be cheap, okay, Let's, uh let's carry on we've been talking about this for a while uh yeah you want to talk about Kpop stuff.

G

Hey pack yeah, it's a little summary then, for those of you who haven't seen this. um Somebody in sap spotted that, if you turned on one of the cash arguments in the kpac image, then subsequent builds were really really quick.

G

And it seems like uh so in Kpop, you have an image and then every time you modify that a build is created or every time it detects that something from the cluster Builder has updated like the stack or the store, which is the build pack list and that affects anything you've built. Then it'll create a new build again uh yeah and by default it uses a PVC just cache stuff um yeah, and it tends to work pretty well like that.

G

So if we had the approach that every time we pushed a new version of the app it would just update the source reference in the kpark image, then it could take advantage of all the caching for that that build. It would just be like doing stuff locally. It would have all like your game modules in the right place and all the um intermediate builds cached as well so like with I, did a test with karifi a few modified creepy and pushed it twice with karifi.

G

First time was took about a minute and a half or something, and then it went down to five seconds a second time so you're getting that sort of improvement. In the build time, um which I think is really important um at the moment, though, we associate a k-pak image with a build workload which has like a one-to-one with a package, and you get a new package every time you push some source to an app.

G

So we don't get any. We don't get any like Source level, caching going on there's something that the build packs can do and they they try and download the previous image from the um the previous built image from the repository. But if it's something like go, there's nothing in there that can help you. If caches anyway, it's just a binary, possibly with other languages. I, don't know there might be some artifacts in there. That could be reused more easily.

G

ah Yes, so so yeah there's that document link there kpac caching enhancements and we were wondering if it makes sense to go and change that relationship so that you have a one-to-one between a CF, app and a kpac image.

G

And then every time you create a new build workload, it would just um be associated with one of the builds that gets triggered from the Kpop image and we did a prototype in there. That's linked from the document, so it does make things go A bit faster, but then we have some comments on that in slack as well. um So what's it so. This is all about kpac dealing with build pack changes and stack changes. So if you change your stack, the clust and um the cluster Builder is updated.

G

Kpac allows you to do this re base or operation which just goes and swaps the stack layer in the image which is pretty fast, and if you update a build pack, then it just rebuilds the whole thing for you, which is pardon me some.

E

G

um We noticed so that's happening in creepy at the moment, but we're not taking advantage of that, because we just get a new build on the image. The build workload reconciler, no.

B

G

Why I think the new the new build doesn't end up on the droplet? Basically because the reconciler says now I've already built it I'm not going to look anymore. So we.

B

G

Updates but yeah one of the arguments was um yeah. So if something like that happens, then all the builds in the system will be updated. If we change that relationship, so we've just got one image per app will only the latest builds will be updated in that way, um but it means that yeah, those are the arguments. I say: oh yeah, and the last thing I wanted to say was so there's two ways that Capac can cache.

G

It can use the PVC, so you can ask it to use a container image if you pass it a registry repository path.

G

um But with a quick look at that, it got really big really quickly with the Golan um and then over the network. It's going to start taking longer to download the cache from the registry than that or save you when you're building your thing, so that was kind of an option that you would be able to to set a registry tag that, like is based on the app name. So it's shared between all the different builds of the app but I. Just don't think it's particularly good.

G

For that reason, unless you can use the local registry.

E

So do do we know like what size the BBC needs to be or.

C

B

Mean is there a way to.

E

Control that yeah the way they won't just cause it to run out of disk instead of like yeah.

G

It's it's in the image spec defaults to 2gig and you can specify whatever you like I think kpac have an ongoing bug for it, though saying this thing will grow over time and they want a way to prune it, and then the K-pop people have said more complicated than you think because it might be mounted all over the place.

G

B

G

Least they're, looking at that, okay.

E

Well, that would that would that would be useful to know I mean like if it's, if it's going to grow, unbounded and eventually fall over, then that's the kind of an argument for using the container image.

G

um But yeah the container image doesn't have any bounds on it. So that's that'll just get worse and worse forever, sure.

E

Yeah I guess the other question I had is like does this? Are we able to do all of this without breaking the abstraction that we have for that? We've introduced with build image.

E

So, in other words, we're we're kind of separating out the the k-pak logic.

C

From the grief you logic with this.

E

uh Build image abstraction.

G

E

The workloading yeah I'm, terrible at.

G

C

E

Term um and now we're saying well, you know it used to be that build workloads spawned an image and now we're saying.

F

E

Of doesn't, unless it there isn't one all right is all of this logic are all the logical changes constrained to the kpac image: Builder yep? Okay, yes, we're not we're not changing that contract in any way. We're just sort of saying, like.

B

E

The app that's referenced in the Builder code.

G

So build workload has a it's a bit dodgy. It has a label on it and called app good sure, and we've used that to name the um the image. No, no yeah, we yeah sorry. We would use that to name the image yeah.

E

B

E

Sure we rely on that already. The.

D

Reason the way we managed to have this like to keep this to have one CR, so one build workload per build, despite having one image shared between them, is that we use the image for writing.

F

D

Like then, for reading the status kpac has a lower level object, called kpac build, which is one per build so like for knowing the status and getting the droplet. We get it from the Kpop build or like when we want to trigger a build. We don't create it manually because it's a bit lower level than we like. We modify the image instead and that triggers a build. So.

D

You end up with one image per app and one build per build, and it does mean we need to read these kpac builds, which are a bit lower level objects, but they are there and they have all the information that we need. It kind of works, yeah, yeah,.

H

I think we intended on reading those to begin with. We just never got to it. Just because, like this, this stack rebasing stuff, that's covered later. That's how that worked in CF for kids and it's like a necessary way to implement that.

I

Is the kpac image have a link to the last build it's phoned, or do you have okay? So it's pretty easy to discover at that.

H

Point yeah I'm, pretty sure it's in the status like it has a reference.

D

To it, yeah Lang's got the generations and things.

H

I think there's a point buried down here, a little bit which in some ways is like even more important than the the caching improvements where I think the the current um behavior of one image potentially constantly rebasing per build is like not very noticeable when you're just pushing little note or go apps and stuff, but like some of these app types like the Java native build pack, they like literally need eight gigs per build pod so like. If you have just like three of those and each has five and they're all rebuilding like that's.

H

Actually a ton of memory, so I think I, think supporting like Java, apps or bigger apps, like would really benefit from the one image per app thing. We're talking about.

F

D

It feels like it's the way you're supposed to use kpug, it's like people who use Kpop Standalone, they push kpok image and then they just just like they keep pushing to the git repository or whatever and Kpop notices and makes a new build, but you never create a new image. Every time you keep one for every git repository, which is like very conceptually.

D

It's actually very close to uh what the cfrp is. Basically yeah.

H

D

H

Over time, someone who works on kpac at one point made a comment saying that the way we're using it we'd be better off, making kpac builds and just like orchestrating them ourselves, instead of one image per our logical, build so like that's, that's more feedback, I think that what.

D

We're doing right now, it's a little weird! We looked into that, but, like I, think that I remember what was the reason in particular, but like it, was just a lot more convenient to just mess with the image and.

G

We were trying to make a rebase happen when we only had a build and it was a bit of magic. You had to go and put the right annotations on it, and the right previous versions and stuff and it'll be loads better for image. Just to do that. For us, oh.

H

Yeah for sure the reason he suggested that is because we're not even paying attention to it when it's rebased or anything we're.

F

H

Like that just happens, and we ignore it right now: yeah.

F

D

Yes, I think if you are happy with the fact that rolling back, which again we don't have any supported API for at the moment, will mean you don't get a rebase when you're all back basically like this is the main difference with traditional CF. Is that if you get an update of the base image and then you roll back to an older droplet, you still have the security fixes of the base image right. The new the base image is still the new one.

D

It's like you just use the whole droplet in Korea droplets kind of embed the base image. So if you roll back to an old droplets you, it won't be rebased right, it won't have the new base, but maybe that's fine. As long as we document that oh go ahead,.

G

I was gonna, say it's a known event as well, so we can trigger those things to happen at that point in time. If we want to, if you switch to an old thing, you can check the build says uh it has references to the of the state of the things that you use to build it uh like all the commits, build packs and things like that, and if you can compare that to the cluster Builder and see it's not equal, then you could at that point make it launch a new build for it.

D

Yeah, it's also an idea. I think we we didn't come up with a solution that we thought would work for that case. I, like you or even just a workout, where it's like, oh I've, rolled back, can I re-base now manually, maybe or something we could figure it out, but it's very feasible.

H

Yeah I mean to be honest, the current flow to roll back would be like using the CF stage, command stage package command and that would you're given an old package, and then it would rebuild that thing.

D

um Or use an old build and set it as the current.

H

Oh sure sure yeah.

D

Anyway, yeah the proposal is that I think we have a spike Branch as well. If you're curious might be something you might want to pick up after all, the GC Madness is taken care of. For me.

D

uh That's carry on uh payload validation, exploration, Dave.

I

Yeah, this was just a request for feedback posted in slack yesterday, but um we finished the exploration on improving the payload validation.

I

um We think we found a good option, but the original option, which is the the hours of validation repo, hasn't been updated for a while, but we did find an active Fork that has like 20 stars that has been updated this year, so just wanted to have folks look at that. We put some spike code of how that would be used to do validation of a manifest payload on the uh the issue. So please take a look. Let us know what you think.

D

Yeah I, like the the API it's kind of similar to the one I had come up with for my little toy validation, Library in terms of being maintained. Yeah that might be a problem, maybe not like even I, don't know like, because even the new one is not that maintained. It's like okay, it's got a few commits on top right of the old base. Yeah.

I

I was going to test them out by submitting a PR to other Dependable it. uh It was kicked at least how responsive they are.

D

I

Responsive they are, if they respond to that, then I'm like okay. Maybe we can trust this one.

D

Yeah so yeah I don't know it doesn't it doesn't seem to be any one. That is that we like in terms of API and is like obviously very well maintained right. So we'll have to settle all like just literally keep our own code, but yeah our own code could grow. It's lots of code that you have to maintain yourself.

D

So I don't know. Maybe it's.

I

Fine I mean the other option is we could have our own Fork of this validation library and maintain it as needed? If it's just maintenance, there probably wouldn't be that much work to do.

D

Yeah, that's another thing: what does maintenance mean like? Do we think this there could be security bugs or stuff? Oh.

I

Yeah and it's keeping the dependencies up to date is definitely a big one and I think we could automate a lot of that with depend about and and CI it's just right. If the validation is working and you don't actually have to change the logic, then yeah they're, not there aren't going to be that many commits so.

D

Yeah, as long as someone is fixing CVS of the library itself, because that's also possible, this is validation, so this is handling Json coming from outside right. So it is us it is exposed surface, but yeah, I've I, think it's fine I think it's probably our best bet, like best compromise between cost of writing around one from scratch and uh benefits of getting like a library that we like.

D

That gives us the errors we like Etc, yes, I, like in general, like the concept of an API that doesn't rely on those, but necessarily on those annotations, and allows you to build a validator yourself via a DSL or something and then yeah. We can easily wrap it around an interface test. It mock it stuff like that right, yeah.

J

Annotations suck because they at some point, you need to point to a method from a string which is confusing and like hard to track. Well, this one is simply simply go code which I like I like it. So it's it's very easy to understand.

J

What's going on, which is which is a plus and I mean we can try without if they don't maintain it, as you said, maybe I mean it will probably be cheaper in the end to just uh even Fork it at some point and maintain it rather than write something from scratch and they might as well do it for us. If we are lucky foreign.

D

Okie dokie last one UA integration, prototype and.

J

This is something we've been working on for a while, uh the internally at sap and we're trying to bring back uh new like new explorers, New Opportunities and new proposals for the community, something that we've pretty much finished and we've also written and explore stories. A UA integration which a use case for that is like seamless migration from users like you have a CF instance, and then you deploy correction. It pointed to the UA so that you share the user base and users can immediately try it out.

J

uh Yeah I, guess I have to say this worked remarkably well with not that many changes you can uh take a look at the story. uh The the hardest. One is that you need to be able to pass this API server. Arguments to your was that your API server other than that it doesn't make any sense, but you should do it, then you can switch CF case the cfk's flag to false and then use a regular CLI and it just works like magic, like it's pretty cool, so.

D

So the CLI then talks to you a logs.

J

In with you and.

C

Sends the UI.

D

Token and the UA token gets and.

B

Then sent by kubernetes.

D

To UI, because UI is set as the IDC provider for kubernetes, which does.

J

The authentications yeah, which may create a stress that token you should you, should buy the.

D

Right right right, they.

J

Were just there was almost no change in grief itself other than propagating the origin, because you have the UA origin, it might not be. It might be something different uh which were not currently respecting in griffy because we're not using the UA or a provider like that. But it's just a string. It's the it's! It's it's an equivalent of the IDC prefix. It's actually what you give to the API server as an IDC prefix.

J

So if you're admin, you're, actually UAE call on admin or sap, idcon admin or VMware, or something called an admin, uh so yeah. We just wanted to share this thing and would be happy if you commented on that. If you support anything.

J

There ideas for improvements, stuff that might not work, and things like that.

D

Yeah, it's the whole CF phone case. Flag thing is is interesting because, like I think we designed it with the idea that, okay, if it's a phone case, it's never going to be UA, those two things are orthogonal, but now you have both. You have a cell phone case on your way, so I wonder if we should change the CLI so that it's yeah.

B

It tries to be clever about.

D

The prices or absence of UAA, because the the info and point like this, the root endpoint, tells you. If you have a UIA configured right.

J

D

Like maybe this CLI should be should use that as the discriminant when authenticating, if there is a uh use, UA or something or maybe I, don't know.

J

Yeah this is this.

A

Should give you an opportunity to switch Yeah.

J

Because we we did it yeah. That was another change we needed to do we needed to pass the UA URL so that you can serve it on the route endpoint, but uh we made it work somehow for our prototype, but I guess it's an open question in this Explorer I mean for now. For us it works if you just switch off the safe Gates flag, but it will be nice. You've had a better way of our seamless way.

J

D

I think we can make the silly handle this better. It's just. We need to make sure that if we mess with that flag, we don't break all the version of this yeah, all the all the visions of the CLI which I've worked.

J

So far backwards compatibility should be right. A priority.

D

I guess I think it should be feasible. It should be feasible or like newer versions, yeah we'll use UI if needed, but yeah. That would mean all the versions will not use UI. Unless you said that you forced that to false I, don't know, maybe there's some conflict we can.

D

We can allow people to change the support of their CLI stuff, like that.

A

Maybe, instead of sending uh just a Boolean, you can next to the Boolean, send cell phone case, flux, Source options or whatever, where you can specify what we support.

D

Foreign yeah, you can do stuff like that, if you are willing to install the latest CLI right.

A

D

If we, if you're happy with you, know this Ally changing, then yeah we can probably make this year, like cleverer in terms of like these things, and because now he just looks at that flag, but it doesn't have to be that way forever.

J

But it was, it was a pleasant surprise how easy it was to make this work and how little change is required and how this doesn't conflict with the way we're doing things now in the open and yeah. Actually, we should be able to work. If you can figure it goes through the UA. You should be able to use an older csclr as well. That doesn't know about certain kids, which is also neat yeah.

D

You either need yeah exactly you you. Could you either need to use a CLI that is so that it doesn't know anything about cell phone case. It knows that flag or.

J

D

One that is new enough to know how to handle that flag and UAE together or.

J

Or the language between someone or just switch it off, but that's weird because uh setting your cf1 case to false on a curify deployment is weird. It.

D

Is weird it's funny that it works, but yes, it feels very wrong, but again it might be the only way to make those clis in that period kind of actually work in this scenario.

D

So if it works, maybe it's as a temporary workaround. It's it's acceptable.

E

So is there anything special about UAA in this context, or could this be any or through provider ation.

D

Point I think it needs to be UI, because I only knows how to talk to UI. Okay,.

J

We're not really using quite this year, we're not really like the only thing we're using quite DC4 up at this point is to configure API server just so that it trusts the UA, uh but other than that we don't do this.

J

The whole IDC dance, the flows, anything we're just talking directly to the UA, with some user password flow uh basic, but uh actually we also experimented a little bit with the cube cutter, IDC plugin so that you can be the cubecutor user against griffy, that's configured with the UA and can still you can still do the thing uh you can do the you can use the IDC plugin it to do the redirects and you'll be able to log in which is interesting.

J

We don't really need that for now, but it's interesting and at this point this could be anything.

F

E

I think we did something very similar on the VMware side using like painted pad.

D

Yeah because, basically, you uh supports IDC kubernetes, accepting your IDC and you're lucky enough that when you use UIA with all the apis with ones used by the CLI, you get the same tokens that you will get if you went through the actual IDC login, which means like once you have that token. Any token any UA token works and the CLI knows how to get those tokens natively. And it's all like with the username and password flow.

F

D

Means like one it gets the it gets the the token via the old flow, and then it sends it to kubernetes which verifies it using this oidc integration with UI and it works at the beginning we were like. ah Is it going to work there's? There are two different types of token that are obtained by very different flows, but apparently they're interchangeable. So that's neat.

F

D

Mean that's, that's the that's the trick, I guess that makes it work.

J

And uh did you did you spin a bit so that you can actually pass those RX to any cluster that you provision on Amazon or wherever?

J

Because you need to pass the IDC files right yeah, we might yeah I think we have control over our arguments in the sap environment. So we want to deploy this, but we might want to experiment with this as well at some point, potentially for now we're working on kind.

J

So it's uh it's pretty cool, actually I'm able to log in on the client deployment using my company credentials because uh we're just pointing to a new a it's impressive.

E

D

Nice I think we might be done if we don't have anything else.

D

All right, I will see you all in two weeks. Then.

F

Everyone thank.

D

You everyone bye-bye.