Kubernetes SIG Auth, 5 Jan 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes Auth Special Interest Group Biweekly Meeting for 20220105

Description

Kubernetes Auth Special Interest Group Biweekly Meeting for 20220105

Meeting notes and Agenda: https://docs.google.com/document/d/1woLGRoONE3EBVx-wTb4pvp4CI7tmLZ6lS26VTbosLKM/edit#

Find out more about SIG Auth here: https://github.com/kubernetes/community/tree/master/sig-auth

A

Cool cloud: it is all right, hello, everyone uh welcome to january 5th, 2022 meeting of sigoth. uh Let's get started, uh looks like we have a few ite items on the agenda. uh Let's start with announcements.

A

uh This is a reminder that caps targeting 124 should be ready for review by mid-january. I know there are a few cups that are being worked on right now. uh Feature freeze is expected by end of the month, so anything you want to get in, please be sure to make progress towards approval in the last two weeks, all right, let's get started any other announcements before I move on.

A

All right uh going into our discussion topics. uh First, one is kms observability related questions.

A

This came out of the meeting we had between mo myself and a few others who were talking about the kms enhancement specifically for observability, and I think there was a question regarding admission requests why emission request gender is generating its own uid instead of using audit id, and I think this was the code that we're looking at.

A

Mo do you want to add anything to this.

B

uh No, I think I just you know we were, we didn't know the history of this code, so I don't know like david or anyone else on the call has context on why a different uid was used or just maybe we have some rules that are not necessarily written down somewhere about how we think about these uids.

B

So the the purpose.

C

Of the uid sent to admission was to make sure that the admission webhook was actually paying attention to the request we sent and we weren't getting a cash response back.

C

So the the only purpose of this unit was to make sure that the admission webhook had inspected the request that we sent um so coupling it to something like audit id wasn't really relevant in.

A

Particular audit id has.

C

A life span of the request, so if I submit a request to the api server that audit id is going to be used for the lifetime of that request, but there can be multiple invocations of admission during that during the handling of that request like if there's a if it's a patch request and there's a conflict going to etcd it'll, actually reapply the patch and retry admission internally multiple times. So it's not a one-to-one request to admission relationship.

C

So that that was really the only purpose of the that you would going to admission in the first place.

B

Okay, that makes sense- uh um I I guess what I would ask is um has has there been any um any requests to help correlate um kubernetes audit logs with invocations of like admission web hooks.

C

I'm I'm not sure so. The admission webhook can return information that gets included in the audit logs, so admission web hooks can return. Audit annotations and those get included. um Are you wanting? Are you asking about correlating like the audit log with web configurations or requests with web hook? Invocations.

B

uh I guess really the audit logs, so so the the reason this question came up at all is we were trying to figure out exactly what is the correct or currently preferred way of including some kind of metadata uh through, like api flows, to make it clear like what caused something to happen so for, in this particular case with kms right like? Why is the kms plugin being invoked like what is like?

B

What is the relation right and to me, the initial gut was that well, if I could correlate a kubernetes audit log to the kms invocation. That would help me understand right so, like mo tried to create this pod, that's why this kms plug-in or probably mo tried to create the secret.

B

That's why the kms plug-in was invoked and it's okay, if the kms plug-in got invoked like three times or whatever, because of internal retries or whatever, but just like, I want to be able to say like this is the reason um uh you know in our conversations uh around this. We we also saw that there's like some brand new tracing code with the open, telemetry stuff, and that looked interesting.

B

But you know that's an alpha, so it was unclear to us exactly like if we wanted to basically include some kind of id that we passed to kms and wired all through like or something of that equivalence.

B

What should we do, and if it's the kubernetes audit id, um then how can we like make it more reliable and more uh not under user control?.

C

Okay, so so the admission stuff was not exactly it's.

D

Just like you were inspecting.

C

Other you would things that had that were associated with the request.

D

C

um Yeah, the tracing stuff- I'm not as familiar with I kind of reviewed the plumbing of it, but uh I would definitely point you to david or my sick instrumentation.

C

um If you wanted to know like plans for that in terms of like getting into beta or whether it would be suitable to use this way.

E

The tracing is something I I also didn't look deeply in uh daniel. Did the review of the uh that giant pr that merged for the uh the alpha.

C

Feature so I'm not.

E

Very familiar with it.

C

I think I think it hooks into the context and like anywhere the context gets passed. uh I think it will propagate.

F

The kms plug-in uses in like a it, doesn't use http right to. We don't use http between api.

C

Server and kms plugin no.

F

C

Does api server to ncd and it got plumbed to at cd? So I know that there is facility for like plumbing it to grpc clients.

F

Okay, yeah, I guess that's what I mean is we have to decide how to carry that request id, although if it is grpc, then maybe there's something already built.

B

Yeah, I don't think we're particularly opinionated we. What we want to do is make sure that we're consistent with where the project is going and not invent some random new way of doing a thing.

C

B

If inventing random.

C

New ways is consistent with what we've done.

E

I mean yes, you are technically right, jordan for the purpose of this, though you can't reuse the same audit id twice or you no longer know whether it's responding to the first attempt or the second, and when you do something like a mutating admission web book, where you are intentionally calling it twice, it's important to be sure you got the correct one. The second time.

C

E

C

If the tracing stuff is exactly what you want since the tracing, my understanding is that it tends to be um sort of uh like it operates on a subset like a sample like one out of every x percent of requests, and it sounds like what you're wanting is a much more deterministic like ironclad. This happened because of that chain that you can follow from one thing to another.

B

Yeah that yeah, so I mean I guess I would ask there is I mean so certainly like we, we can go investigate the semantics of the the tracing stuff and try to see if, if it's what we were thinking, what I'm gathering so far is don't worry about the kubernetes audit id.

B

um I guess what I would ask there is you know if you know I am a canvas. Plugin maintainer author production person and I do see invocations of my kms plug-in and you know, have some ids associated with them say we did exactly what admission does and just generate a unique uuid per indication of, like I guess, the kms plug-in or something.

B

Would we want um like, as a for example, would we want to try to somehow plumb that back into the audit event, using like audit annotations so that way? If you wanted to correlate those things, you could in a way that can't be spooked by the user.

A

F

The state of the art for logging is pretty much right. You would pass the original audit id and then your kms plugin would write out a audit log entry or just a plain log entry saying I'm working on a request that I named uidx and it's in service of request.

F

The original request. Uid y tracing lets. You link the spans, but, like jordan said they're, not every trace is sampled.

B

Yeah, um so I I guess, then what I would ask you, so if we're saying tracing doesn't have the right semantics, um I I saw the the the audit id chain issue that tim had opened, which is basically the crux of it is audit id can be controlled by an end user by them just setting the header.

B

um I guess what I wanted to ask is: why do we allow them to do that, and can we not allow them to do that so that way we can trust it? How would you.

E

Distinguish who said it.

C

uh I mean like generated internally like we do for ignition.

B

Right well, audit id is generated internally. It's just that if it was already as set as a header, it'll just use whatever value it's passed in. So if the client doesn't set one, we do just generate a random thing.

C

Yeah, I don't remember I'd like to dig into the history of that and see if that was discussed, uh because.

E

C

That sounds like something I would raise my eyebrows out and I don't remember.

B

Yeah, it totally does right. uh Tim had mentioned that maybe it was related to aggregate api servers if rita. If you want to open the linked issue.

A

B

Yeah the audio id chain, um but if, if it's related to that, that seems also suspect, because the the connections to aggregate api servers have strong mtls associated with them. So, like you, you would just you should piggyback off of that and use that just like to create some kind of trust anchor, not just arbitrarily trust it.

B

C

Okay, so it sounds like uh the questions are around nailing down whether we want the audit id to be able to be dictated the way it is today. um That's one question: if that becomes more trustworthy, the next next question would be. Would we want to plumb that down to the storage layer make make use of that in the storage layer for something like this, and if not, then what would we want to use as an alternative to allow correlating stuff at the storage layer to stuff in the autolog.

B

Yeah yeah, basically, that's yeah. That's basically, this entire set of questions can those two questions, basically sum them up. Okay, um I.

C

Think tim would be the a good person to ask if he can remember.

G

And I also say that we do expect a lot of what the uh canvas plugin is expected to do or asked to do by the api server is done so by system operations and the loopback client. um Like definitely most read operations to keep aps server, not going to be associable with anything that we would put in an audit log.

G

So I yeah, I don't know how far you're going to get. If you want to be able to track everything back.

A

What do you mean by that? Why not.

G

So like uh for for like something that is served from uh a cache in the q api server, um we would not. um You know, expect uh every read request to um uh to like a secret to actually initiate a kms plug-in operation, um primarily like initial load would be done by uh the loopback client in the cube api server and then subsequent reads would be done from memory um and not required kms plugin interaction with the canvas plugin. So there will be things in the audit log reads of secrets that do not have.

G

uh That does not require interaction with uh the canvas plugin and vice versa. There will be things uh in the canvas plugins log uh that were um uh that uh did not produce an audit event.

C

Because they weren't triggered by an api request, they were triggered by an internal like watch, cache load right.

F

B

And um can y'all remind me so when a client does a live, get against the api server.

B

I I'm trying to remember when does that hit the watch cache versus going straight to cd.

C

Live get goes to etd. You can opt into a uh cached, get that will hit the watch cache if it is populated, but even apart from hitting the watch cache like if you do a live, get from ncd and the server loads, the encrypted data and the data encrypting key is still in the cache. It will use that without talking to the kms again so there's the server has the ability, cache data, encrypting keys,.

B

Right right, okay, yeah, so.

B

I think that one is okay. I I think what we were looking for is when the kms plug-in is invoked. Can we figure out why uh and it so? It does seem like for live operations using like cube ctl like a normal end user. Would that's fine, it's only the less uh the internal stuff within the api server that the loopback client is doing to fill caches or whatever that might hit kms, but.

C

B

C

When the server starts, if it fills its swash cache with all secrets which it does then at that point, it's going to request all of the decrypting keys data, encrypting keys, so the first and most likely requests to kms are not going to be associated with particular requests um and it's like when the server starts up it'll load, all of those now on subsequent writes when it's hidden kms to encrypt those will be correlated with the right requests anyway and we're 20 minutes in, and we probably want to time box this.

C

But if that gives us like next questions, next steps, that's probably where I go from here.

B

uh Initiative, did you guys have anything else you want to ask before I do see. We have two other items on the agenda.

D

No, I think uh with it. Reading from the cache, I think that is where the audit id chain might help like where we can correlate from the request all the way to the end, to.

A

D

But otherwise, I think just generating the uid right at the api server before we make the request to kms plugin seems like the right fit for now, so that like that is what we initially want to correlate. So then, maybe we can expand from there and see if you also want to plug it into the audit login.

B

uh Maybe I don't know like just thinking through it like um the stuff that mike and jordan have brought up and I'm I'm questioning how useful it would be. I guess so. I want to make sure that you know for building a bunch of plumbing. It actually is valuable. Plumbing, um that's fine! We can. We can keep thinking through this.

A

uh No, I I do kind of I need to think about what mike said a little bit more so sure.

B

Yeah, um I think we're ready to move to the next.

A

Okay, uh james, do you want to talk about the 124 release team stuff.

H

uh Yeah, hello, um sorry, can people hear me cool, so my name is james lavrak, I'm the release leader for kubernetes 124. I just wanted to take the time to come, say hello and introduce myself to you.

H

um I just really wanted to make myself available if you had any questions or anything you wanted to bring about release. um You've already talked about the release deadlines uh just to add to that.

H

The release as a whole starts on monday, so we should start accepting um caps for tracking then, and our current draft for the full schedule, which we hope to finalize with the end of the week, uh has enhancements freeze on week, 4, which is actually at on thursday, the 3rd of february and with the production readiness soft freeze the week before on thursday for 27th of january.

H

So that's what the dates are. Looking like right now, we'll have those confirmed in the email of the starter next week, but that's what things are looking like, yeah, that's all ready. Does anyone have any particular questions about the release or anything happening this time around.

C

A

C

A couple questions in the retrospective about whether we were going to continue using the tracking spreadsheet, which required sort of manual update and some of the release team found difficult or if we're going to try to do things off of a query from the enhancements issues.

H

um Nothing has changed for 124., um everyone hates tracking spreadsheet. Don't worry, uh we all want to get rid of it. um It's just what we replace it with is a larger question. um There is an effort within within sig release to get that changed, but it nothing's changing for 124, it's the same as it was before.

C

H

C

H

For each stick, there's normally.

C

Like a small single digit number of things, we need to add to it. So it's not terrible uh just wanted to know where, where we should be putting things in the next week or two.

H

Yeah, it will be, it will be putting the spreadsheet as as previous releases. Okay, thanks.

A

Great, thank you james uh adam. Would you like to talk about the uh csi and funeral volume pot, security, emission control, dock.

I

Sure uh so, uh speaking of caps, this is a draft cap that was discussed. I think in sigoth, before the holiday break.

I

So just as a quick refresher, uh the general idea- and this is something that would most likely be sponsored by sig storage- is that we would be adding an emission plug-in uh which allows uh csi drivers that provide the ephemeral volume capability to sort of declare themselves safe for a given pod security profile, using either an annotation or a label.

I

And then the admission plug-in on pod admission would decide if the pod should be admitted based on uh inspecting the csi driver's.

I

Annotation or label- that's the general high level idea, I believe, and unfortunately I wasn't able to attend the previous meeting. We left things off.

I

We wanted to take an assessment of which uh csi drivers might be considered unsafe, which is kind of the reason where or where this cap idea came from uh and uh consider potential either uh either the scope of the problem or uh consider alternatives.

I

uh So I just wanted to follow back up on on this uh and discuss potential next steps uh and sort of where our findings are so young.

E

Did answer the question he provided very quickly. Examples of I believe it was azure, uh as, for instance, um of one that was not safe.

E

um Not sure whether jordan ended up creating the alternative list of all the safe ones,.

C

No, I I I was landing on michelle and john for that.

C

So I think my posture is still that I don't want to encourage people to create unsafe, ephemeral, csi drivers, and I don't like the idea of not being able to look at a pod and know outside the context of a particular cluster, whether it fits in a given pod security standard level. Both of those seem not great.

E

Well, we did leave it to be made tighter by other mission plugins, so I think that latter point will exist, no matter what you do, because.

C

But if it's a like a built-in admission plug-in like that that there's a difference between yeah, we installed custom admission stuff on this cluster and it's making policy decisions like.

A

That's always going to be possible. There's a difference.

C

Between that- and this is another built-in admission thing- that gloms onto the same uh the same level labels.

E

All right, well, I'm still interested in in an alternative list. uh Yan, actually, storage yawn did come back and answer the question, um but if there is a different list that says no, this isn't really a problem. That would be interesting.

C

For the azure one, I'm also curious. I remember there being a variety of things with azure, where sort of what I would have considered important boundaries uh to safely isolate namespaces and things were not really considered problematic by the azure team, um and so I I'm wondering if that specific one is more a mismatch in expectations of cluster isolation.

C

uh Instead of saying no people running on azure have strong expectations of name space isolation and this particular csi driver is confounding them. uh I I don't it's been several years since I've looked at it, but things like uh secrets being written to name spaces, containing storage credentials.

C

That were surprising to me for someone who was wanting to isolate namespace permissions. um That was pretty particular to azure. The built-in azure driver.

E

uh I didn't look any further than the link was a kubernetes six link.

G

A

Do you remember which uh driver that was because I.

E

Think there is there's a file, is.

A

It azure files.

E

um The one that jan linked was kubernetes sig's, uh azure csi driver azure file.

C

Yeah, the csi one I I hadn't looked at, but the azure file. I forget which one was creating.

C

I can, I can dig it up.

C

Let me we don't have to block here I'll. Add a oh that'd.

A

A

uh At any rate, I would be very interested.

E

In in seeing the the list of csi drivers, if there are others that yon didn't even have in his list that were also vulnerable and then the decision about whether to to fix them or to ban them or to um leave them broken um I'd be well I'd. I think I'd like to look at that list before trying to make a choice.

A

So is this the action item for.

E

So I'm willing to take yon's word um sounds. I think jordan wants to try to change the csi drivers rather than protect the cube clusters in the current state, which, which is also valid, um but.

G

Anyone else would like to.

E

Have opinions it seems like they would want that information? I think jordan.

A

Needs it either way.

C

Yeah I mean the current posture came like when we were looking at the pod security standard stuff and trying to decide what to do with cs ephemeral, csi drivers and we went and talked to storage folks, and um the feedback from them was that there were few known ones that had uh similar characteristics to the built-in volume. Plug-Ins that you could specify like.

E

um I guess I'd be interested in hearing michelle's arbitrary connection yeah. I remembered different feedback from them more more in line with what yon wrote in the doc where he says yes, this is a real problem to address so um yeah, okay,.

A

So davis sounds like: uh are you, okay, with following up with storage.

E

I I think we already did and they answered. Yes, it's a problem and we should protect it. I think jordan has a request for a second opinion, which is fine, but I think he should own the action for that.

F

I mean I've got yawn right. There.

E

And I I I so okay.

C

So I I take what yan says, as this is a problem, it's a problem for the driver, and so, if you want to use that driver, you need to protect it. I don't think it's an issue for kubernetes to build in policy to protect against.

C

I don't disagree that some drivers are exposing things they shouldn't. My recollection of the when we talked to storage was that the guidance would be for ephemeral drivers not to expose those things.

C

And building in policy to let them expose those things and then lock it down is not going to push drivers in the direction that was previously discussed of ephemeral, csi drivers being safe to expose to pod authors.

C

So I'm fine uh setting up uh something or sending something out and starting starting a discussion. It sounds.

A

Like the guidance.

C

For driver authors that was pretty previously discussed didn't happen to clarify that when you make an ephemeral driver pod authors can directly access all the parameters you expose and currently and by default. That is not safe to do, and so current drivers that are doing that are not safe and have an issue, and I I I'm not convinced that it is uh kubernetes issue to fix by continuing to let them expose unsafe stuff.

C

Anyway, I'll I'll start that and try to figure out what the next step is.

A

All right um any other items in the agenda that y'all want to talk about, but it's not here yet.

A

If not, I guess we can look at the ci flakes.

A

Does this look familiar to you folks.

B

Jordan, is it me or this service account oidc discovery thing shows up way more often than anything else in our flags.

C

It does uh I had asked michael to look at it and he dug into it to the point where it hit networking issues. I think it was a networking visibility, config thing uh and I lost track of where it went from there. I think he was going to ask the networking folks what the deal was with the config in that job, but I don't know if that ever happened.

C

Okay, um let me ping him real, quick and see if he can pick that or hand it off and get acknowledgement.

B

Okay, um there are other issues with it. Let's use.

A

The other search.

B

F

A

I think that's it.

G

uh What's the difference between the flaky? Oh, you see discovery number two and there's like one that looked completely broken, which is this top one.

C

I just linked to the issue for the completely broken one.

C

That looks like a configuration with the job um and it got routed to the cloud provider folks, because.

C

I don't know why uh that's where the configuration seemed to lead.

C

Yeah, I will try to ping walter and see if that makes sense to assign to him or if he should route it. To someone who understands the config.

C

It had to do with the detection of the self-addressable network uh address for the api server, so it was.

D

Like a network visibility.

C

Issue, so it's not really related to the future. Just happened to be surprised by this test.

F

um On a slightly more meta note, it seems like we rarely cover flakes in the meeting. Is there more process around that, or do we just kind of let them slide if we don't get to them.

G

uh I would say I think they fill the shoulder where we, where we have time uh generally uh the tests that sigoth owns are um a lot less flaky than other cigs. I would like to think um because uh the majority of the test coverage um for stuffing kubernetes core is unit tests and integration tests, um but so so I don't really have a problem with kind of leaving it lower on the agenda and not having dedicated time um there.

G

There is a triage um meeting uh monday mornings that covers like bugs and maybe stuff more urgent. That does not go over test flicks.

A

Yeah, we don't go over test flights, we only look at issues.

A

Unless they're created as issues, I guess.

G

Yeah, I think if we were uh causing more problems. Maybe jordan can tell me if this is wrong, but I think that if we were causing more problems or um breaking more submits and stuff, it would be like higher priority to focus on, but the current state is pretty reasonable.

C

um That is generally true, I think if stuff gets bad enough, it shows up in the top flakes list and it gets reported.

G

C

Does get covered, the intermittent ones um aren't usually bad enough to get reported and also aren't necessarily clear that it is our issue like if one of our tests fails. You know once a week, um sometimes we'll dive into that and like a bunch of jobs,.

A

Failed and not run.

C

And like something overloaded, the server or something so uh what we tend to look for is stuff that bumps up into the top flakes list or um that we happen to just notice ad hoc. uh When we're you know doing pr's and stuff. It's like. Oh, I seem to hit this one a few times and we'll open an issue.

C

The the thing that I really want to catch in these reviews is like the permanently failing stuff that just is sort of a noisy floor. So, like the oidc thing is a good example like. I want to try to get that cleaned up and clean up permanently failing stuff.

A

um So is this item currently being assigned to anybody.

C

um It's on it's on walter, uh but I'm not sure if that's correct, so I'm gonna ping him and okay. If he can triage.

A

C

Route, it or say I don't know and unassign.

A

C

And we'll try to figure out something else: okay,.

A

A

So this should really be uh in progress or something.

C

It's it's waiting on other sigs. I mean this is like a cluster config issue with this test. Job, like it, doesn't know its own network address correctly. uh So that's not our tests. Responsibility, I think, pending other sigs are not actionable.

B

C

I guess pending other things, it is actionable just not by us right.

A

Okay, um moving on is there was there. I don't think that there was anything else right. So did you guys see anything else that jumped out.

G

No, this looks good.

A

Okay, all right, I think next, is the test card.

A

Sorry, my computer, so.

A

uh Anish, did you want to comment on this or.

D

uh No, I mean the one job that was failing last week was because we have image scanned periodically and then there was a image vulnerability, but we already fixed that, so it should be fine, uh but the other one was just one flake yesterday, but apart from that, mostly everything has been passing.

A

Okay, cool, I think that's that's it any other topics that anyone wants to bring up.

A

Oh all, right thanks everybody and happy new year.

G

Happy new years.

A

B

Y'all take care.