VMware Pinniped Community Meeting, 18 Mar 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Pinniped Community Meeting - March 18, 2021

Description

Pinniped Community Meeting - March 18, 2021

Project roadmap and contributor opportunities announced!
Main discussion around March 2021's roadmap item: Impersonation proxies (coming in v0.7.0)

Details here: https://hackmd.io/rd_kVJhjQfOvfAWzK8A3tQ

A

All right, hi, everyone welcome to the pinnie ped community meeting, um just a reminder this is being recorded, so please adhere to the code of conduct when attending these meetings. That is listed out in the agenda here today is march. 18Th 2021, and we have some really big announcements to make a huge undertaking with um project roadmap and opportunity areas for pinniped contributors.

A

Pablo. Do you want to go over the pinniped? The project roadmap.

B

Sure, um okay, so the roadmap, um which is linked here and you can find on the top level page of our repo, it kind of outlines a couple of different aspects of the roadmap in terms of like what about the document is relevant, how you can help uh and then it also links out to the opportunities area document that you also saw on the page. That was just shared by nancy for the community meeting.

B

But this is a discussion thread that emerged from a brainstorm that happened with the maintainers of pinniped in early january 2021, and these opportunity areas were identified as potential opportunities for contributors to grab ideas from with actual tactical work items that support the roadmap.

B

So these are suggestions but they're great starting points for anyone that might come into the project and might be wondering about where they would start the roadmap itself. If you want to go back to that, really only looks out about three months in terms of planned work well like past that there are items that we kind of qualify as exploring or ongoing, such as improving security posture, improving our ci cd systems, telemetry and observability. Those items will probably also change as they get prioritized in the roadmap.

B

The reason for having it only look out three months is that we believe it is probably not reasonable for us to make projections about what this team or contributors will be working on past, that as things change as we learn things from customers and as we gather feedback from the community as well.

B

So as you can see it kind of looks out through about june, and these themes are pretty they're kind of they're, very high level, um and so in terms of getting visibility into what the team's working on we're hoping to use this meeting as well to talk about items anchored firmly in the roadmap, as the team makes progress toward those things.

B

So it's also a great opportunity for contributors to get a better sense of where they might also contribute on the current items that the team is working on, if there's space for that as well, but otherwise the opportunity areas. Dock has specific items that also map to these themes and the descriptions as well. So hopefully, that can also help anchor contributors that are considering making contributions to the project, and this will be updated.

B

I expect it'll probably be updated each month, as we look out past the the second month on the roadmap and have some planned work that we can inject into the roadmap based on feedback and different priorities that surface any questions.

A

Cool great thanks pablo so with that moving forward, the community meeting will be anchored to the project roadmap. What the team is working on any updates geared towards that particular item in the project project room up so moving on, we have march 2021's roadmap issue is the impersonation proxy. You know the team's been working on this for a few weeks now and I'd like to get the updates um from margo first.

A

What do you got for me on this.

C

Sure so uh I've been working mostly with andrew on writing some good integration tests for the impersonation proxy branch, so particularly like kind of some of the more interesting uh keep cuddle commands. Like you know, port forward and uh watches and.

C

You know copying and kind of websockets as well like using websocket client um and so writing those tests and then also a little bit of documentation, work to uh bring that up to date. With uh what we'll have once we release the impersonation proxy feature.

A

Cool andrew anything to add to that.

D

None on the impersonation proxy integration test. No, I enjoyed thanks for letting me work with you. Margo.

A

um And then why do you have listed out here as far as is that outside of the impersonation proxy stuff.

D

It is actually still impersonation proxy related. uh The second bullet point at least ryan, if not matt, if not also other team members that they were working with, found some interesting behavior with our impersonation proxy http handler, and we thought it was a bug, but now we don't think it is a bug.

D

So I've did that past couple days along with some other team members, and the third bullet point is kind of forward-looking. I know we wanted to do some manual testing on gke, so that's kind of what's on the on the table right now, but that's pretty much me in terms of impersonation proxy. The past two weeks.

A

Cool uh and then ryan, what do you have.

E

I've been trying to help out with the things that were already mentioned, as well as the thing that mo hasn't mentioned yet uh so just kind of working with everyone on the team on all the different things. uh The I think the biggest the most number of days in the last two weeks was spent with mo working on uh rewriting the front end of the impersonation proxy, and I put that actually down.

E

As a discussion topic, I thought we could spend a couple minutes talking about the design change that mo proposed how it affects the design of the impersonation proxy.

A

Okay, this sounds good mo.

F

uh So I can wait on the. I guess. The first bullet point until we get to the discussion, but the a little to add some color to what andrew was saying on the gke stuff. We just um we're trying to give us ourselves like strong confidence that we're testing against in realistic environments.

F

So a lot of our tests run on kind, um but the impersonation proxy is not really. It doesn't really care about kind. It cares about. Gke cares about those managed environments, so we do have those running in our pr pipelines, but we just wanted to try some more like human style testing of just poking at the system for a long time, instead of like just running through a test suite that lasts for just a few minutes and just give ourselves strong consonants, so it actually behaves the way we expect it to behave.

F

I saw some weird behavior on my laptop that I can't like isolate it to. Is it just docker being funny on my machine, or is it actually like something wrong in our code? And when you have like five layers of proxies, it gets really hard to figure out which, which layer is mad about what.

F

So. We just wanted to try to have like the exact environment that the customer would have and make sure that it's as robust as we think it is in that environment.

F

I had actually added a little bullet for matt because he's not here. I know he spent a decent chunk of time uh getting our pr pipeline to support ephemeral, gke clusters, which is really good, like we have like a super nice set of covers across a bunch of different clustered environments.

A

Awesome thanks for sharing that pablo.

B

Yeah more work on internal strategy bits, uh some of which will also affect our open source strategy. uh More on that, probably in the next two weeks and continuing with customer interviews um just as a kind of looking forward thing. One of the interviews I conducted was with a person named dodd pfeffer, who seemed to have like a lot of experience with customer pain points specifically around configuring off in ldap environments, um which is um rather apropos because we are approaching um our next bit of work around extending peniped support for ldap.

B

So what I would like to do is set up actually a call with him with the team or with some portion of the team, perhaps right before we have that design session for ldap, so that we can pick his brain he's kind of a proxy for many big enterprise customers as he works in the field.

B

So I think that'll be helpful. So look out for an invite for that. I think it'll be beneficial for us before we go into the design phase of the ldap support and that's all.

A

Thanks pablo uh all right discussion topics, impersonation proxy design, change, who wants to take take it from here.

E

This is something that mother pose. Do you wanna talk about? You want me to try to summarize it on your behalf.

F

I talk too much ryan, it's your turn to die.

E

F

Try to summarize.

E

It uh so the job of the impersonation proxy, just to remind anyone, who's listening to the video later, is to act like the api server. So it's going to receive requests from the client and then turn around and proxy. Those requests to the real api server and it's going to add impersonation headers. So it's job is to authenticate you and then add the impersonation headers. So it's using a admin level service account that can do anything. It wants to the real api server.

E

But it's going to add the impersonation headers with your username to make sure that those calls are done on your behalf with your level of permission, so the we had originally designed that in such a way that we were going to perform that authentication the same way more or less, that our token credential request api performs that authentication we're basically going to duplicate all that code and moga posed uh that we restructured that which is what bo and I spent a good amount of the last couple weeks, working on at least one week of the last two weeks, uh which is to actually just let the client call the usual tech and credential request, pin the pet api that returns a client cert a token a credential.

E

Whatever you want to call it that's supposed to give you access to the api server. We already had that, and so now that will return you a credential that will just as easily give you access to the impersonation proxy.

E

The advantage of this change is that, basically, the clients don't need to really know that they're talking to the impersonation proxy do what they used to do. The same way they used to do it, which is ask the token credential request api for a credential and then present that to whatever they think, is the endpoint, whether it's the real kubernetes api server or the impersonation proxy server and then just make their request. Everything works this. This was a pretty significant change to the code, but I think we all are better designed.

F

I think I think ryan covered it really well at a if anyone cares about code code level details, we reuse an enormous amount of the cube stack in the front part of the impersonation proxy, which has some nice benefits, because it basically interprets your request in the exact same way. The cube api server interprets your request, which, which is nice, because that's it's supposed to be from your from the client's perspective. It is the api server.

F

So, in that regards it turned out really nice.

F

We, yes, we had to restructure a bunch of code, but for the most part, we're not really using any code that we wrote. We just have a small amount of relatively straightforward wiring and then a bunch of assertions to make sure that wiring behaves exactly the way we think it behaves, and we just use.

F

A

Anything else to add from the team on that.

F

I guess uh you folks have us any thoughts on what's left. What would we have left before? We can ship zero, seven zero. I know folks have been excited and waiting for this release. We keep telling nancy in a week we'll be done and then it's another week.

D

I actually had that same question uh the bullet below this one. I listed two things that came to mind on what's left so fixing some test flakes. I don't know if flakes is the right word, but I know mo and I were looking at some red tests earlier, so getting getting those tests passing and maybe some manual testing. I know we talked about mo mentioned earlier, but I'm curious what people? What other folks think is, is left here, because I definitely haven't been working on this as long as other people have.

F

I guess one open question I have, in my mind, is: if uh how how far did we want to pursue the testing of the impersonation proxy on kind and if we wanted to restructure those tests to better simulate a real cluster, so not going through squid and all that stuff, because it, it seems to behave weirdly, uh at least for the fan like yes like get and create works because they're just normal rest verbs and nothing really special happens there, but stuff like watch and port forward and exec and all the weirdness doesn't behave exactly right.

F

um So I don't know what folks think. Certainly I think we want to make sure that the managed cloud environments where we really care about this functionality really match customer deployments. But I don't know how much we care about kind.

C

I'm kind of inclined to say, like the tests on kind, are like mostly a smoke test um that it, you know, isn't terribly broken, but any major issues with or like. I guess, any kind of subtle issues with the port forward or you know interesting commands might be caught in ci, but we know that it will be caught in ci because we have those managed environments.

C

C

Want more real, quick feedback, then we could certainly put in that work.

F

So I admit I have a pretty big gap on exactly how we set up the managed environments in ci. I think, if I recall correctly, we did not use squid on gke, so our gke environment looks exactly like a customer's environment. Would I wasn't exactly sure where we ended up in aks and dks?

F

If we could make those look exactly right, then I think I would. I would be perfectly fine with what margot said, which is. The kindest kind, is just a easier easy way to test locally and it will catch egregious bugs and any cello bugs will hopefully be caught.

F

Nci, and you know we don't we don't merge stuff into the main branch with red ci. So it's fine, we'll still catch ourselves.

E

I think mark margot reminded us in a different meeting yesterday that.

E

On those managed clusters in ci, we we do deploy the squid proxy and we use it for other tests, tests about the supervisor and the oidc flow, but for the impersonation proxy integration test we specifically do not use the squid proxy if there's a load balancer.

E

So at least the impersonation proxy integration test we'll go through the load. Balancer.

F

Okay, I think I remember now when she said that, okay, so, but what I'm hearing from that? The ryan and marvel is that in the managed environments, we have basically exact customer coverage, so we're good. There.

E

B

Do we have that for aks and eks, or just gke.

C

I'm pretty sure anywhere that we have the capability to use a load balancer. We use that and not split, and so that would include ecas and acas.

F

So I admit I don't remember exactly how far we are on the ephemeral pr pipeline for man's clusters. Do other folks have a better sense. I thought I thought that matt got aks running on there. We really cared about that environment. I know we already had gke at some point.

E

I think what matt did that's not here today, but I'm just looking at what he did in the pipelines. I think what he did was in the pull request pipeline.

E

It will run the tests on an ephemeral, gke cluster, but not any other managed cluster, because that's if it works on one, it's probably going to work in all of them, but then in the main pipeline. So after you merge the pr before we release, it will run that commit on eks, aks and gke, and on top of that it will run on three different kubernetes versions of aks three for eks and three for gke.

E

So we get a pretty decent matrix of coverage there kubernetes versions and manage cluster providers.

B

It so just to echo back when I'm here it sounds like we have the test coverage that we want, or at least it's already in flight, and then what is also in flight is some additional manual testing for aks and eks. We've started the gke testing.

B

um What about these like the stories that are sort of still in flight? Is there any major work that still needs to be done, or are we waiting to just kind of deliver and accept these stories.

E

I think one of the other things we were talking about yesterday was maybe spending a little bit more time, trying to understand why the kubernetes, the closest equivalent that we have in kubernetes to the impersonation proxy is the kubernetes proxy commands, the coupe cuddle proxy commands and they go to great pains to use the speedy protocol for watches and execs and some of the other commands and we're not doing that in the impersonation proxy.

E

So we were talking about maybe spending a little more time to try to figure out why they're going to such great pains to use the speedy protocol and whether or not we should be using the speedy protocol. In those same circumstances, I don't think we know the answer to that. Just yet.

F

And ryan spent some time doing the spike on moving to that code, and it is pretty egregious like it takes what is like a almost trivially easy to read file. If you just read the comments and read the code, it reads really straightforward and the business logic is really on the order of like 20 lines and the rest is just I wired up the server uh it takes that and sort of flips it on his head.

F

It's like no there's like 200 lines of business logic now, none of which really makes any sense, because, like the comments, don't make the comments in the cube code, don't make any sense and then you're left trying to figure out what why any of this matters, and also this particular bit of code, that's using the aggregated proxy in the cube api server.

F

This is the piece of code that in 2018 had a 9.8 cve. uh So if we all remember, 10 is the highest. The scale goes uh so 9.8 is basically unauthenticated remote component of your entire environment and that's what that? Let you do really really don't want to use that code.

F

If I don't have to right now, because I don't understand it well enough- and I don't know if I'm gonna get so yes if we can find some functionality or some performance or just some other customer facing reason or maybe somehow it's more secure- I don't know so.

F

If we can find some good reason, then yeah we should use it, but right now I'm pretty strongly leaning towards it works and I can't find any bugs in it. No, this is probably better.

D

Are there other um already built examples of impersonation proxies that y'all have compared this to when trying to answer that question of? Do we need this new code like moe, I remember you mentioning. There's a project, maybe called teleport or something.

F

Like that gravitational is a company to have a project. They have a product called teleport which uses an impersonation proxy. It uses a fork of a different project as its back-end reverse proxy. It does not even use the go standard library at all, it's like or if it does, it uses it in a very different way, um so that so so we have.

F

I assume that one works. We have the aggregated proxy and cube that works. We have the cube ctl proxy, which also uses similar code to the aggregated proxy. That also works in slightly different ways.

F

We have our code. This is just using the standalone stuff. So we have. We have a lot of choices. We don't necessarily know which one is better and for what reasons none of them are documented. As yeah. I use this code because we do this thing.

F

F

I think uh patrick you had mentioned some bugs related to the impersonate group header. Was that because of that other proxy thing that you were trying to use on your clusters, is it using impersonation in the back end with two clusters? Was that why that came up for you.

G

uh It came up because I used I tested out miriam another project that also uses the impersonation headers. So I guess it's a impersonation proxy um and the issue, though it's it's based on envoy, and the issue was that that it put all the groups in the same header and just comma, separates them, and then kubernetes doesn't know what to do that. They generally think, is it's. It's just one big group name so yeah.

F

It's kind of hard because cube allows commas and group names. I forget if you can like escape them or whatever, but yeah you just need to send separators uh um but yeah. So maybe that if you could link that to us patrick, we could go. Look at that too. Maybe give ourselves another data point, but.

G

F

To andrew's question, no, I don't I I don't I we do have a bunch of examples. They're all different.

B

Though we think we're looking at like cutting this release sometime next week,.

B

Just to get a sense of timeline.

F

Yeah, I feel reason to be confident because I don't I don't expect code changes at this point I expect like, or I mean, if we find some bug sure, maybe but yeah we might. We might change our tests or something to align better and we might do a bunch of manual testing or just other confidence testing just to make sure that our tests are testing what we think they test um and yeah. I don't know, I feel pretty good about it.

F

All it all seems to be working and for the most part, once ryan and I have switched over to the new approach, we didn't like find anything that wasn't working.

F

I I was very suspicious that it was all working uh but yeah. I feel pretty good. What do I listen.

F

A

uh When is matt back.

B

Next week, monday.

A

Okay, yep: do you know how far along he is in the blog post.

B

I do not, um I know that I mean he's been working on it. I don't think, there's probably that much left. I think there might be some changes based on whatever work was done this past week, but I don't know we can circle back with him on monday.

C

Okay, yep works for.

A

Me all right anything else we want to discuss before we do.

A

All right well, thank you, everyone for joining the this edition of the pinniped community meeting, just a reminder: we meet every first and third thursday of the month at 11 a.m. Central time, 9 a.m, pacific time, uh and that would be 12 p.m. Eastern time, so we hope that you see us at the next community meeting and with that, thank you and enjoy your weekend.