Cloud Foundry CF on K8s Forum, 26 Jul 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Cloud Foundry on Kubernetes Working Group Forum Meeting 26 Jul 2022

Description

2022-07-26
[KB] Enabling websockets in contour route HTTPProxies?
E.g. https://github.com/cloudfoundry/korifi/pull/1436
Supporting WebSockets | Cloud Foundry Docs
https://projectcontour.io/docs/v1.21.1/config/websockets/
[GC] API docs

A

B

Hello, everyone uh welcome and thank you again to coming to the cloud foundry on kubernetes working group form. We have a document that will be linked in chat for anyone that has a topic they'd like to bring up.

B

Additionally, if you cannot paste into the document, you can drop it in the zoom chat and I will add it to the document so dropping that link in the chat right now and let's get started on the first topic. So the first topic we have for this week is enabling websockets and contour route. Http proxies looks like there is a associated pull request.

C

It's me um so, for the last few days we've been trying to get post-facto to deploy on karifi. um So in our old irini um ci, we had an acceptance environment and we tried to run some apps and keep them going um check how things behave as uh apps get upgraded, and things like that, so I've been trying to move uh post-facto from um irini to kirifi and has has brought up a whole load of things. It's been a really good exercise.

C

um It exposed the whole thing about um kpak, accepting mvrs for to pass to build packs, but it only accepts literal values rather than things that are like hidden in kate's secrets, for example. um So we got through almost to the end. The thing was deploying um yeah, it's the retro board. So when somebody types on it, everyone else sees it. uh We finally realized the websocket thing wasn't running, not because we couldn't deploy the thing properly. uh It's contour um was rejecting web sockets.

C

So there's a an option you can put on the http proxy.

C

It fits naturally by the the root one, and it just says enable websockets true, so that ppr does that, um but we're kind of wondering why it's not enabled by default. So what are the implications of us doing that so I mean on one hand, I guess that's how cf on vms works.

C

Web sockets are enabled um this. This routine thing is disallowing. It are there any security considerations for turning it on any other. Things just wants to ask about that. Really.

D

One question I have um doesn't mean we shouldn't do this is what levers do the gateway apis have for for websocket support? um I know we're planning on doing uh an exploration like like this whole time. We've been saying that the contour integration with hdb proxy was like short term, given that we eventually want to support the kubernetes gateway api, which is like a more generic interface for all these ingress solutions.

D

I'm I'm guessing. It has some extension points that would allow people to turn this thing on as well so, but it might be worth just poking at that too. um I don't know in in general, this does seem. I I link the docs for uh that. Describe the router's support for websockets a bit more and it does sound like this is something that is on by default for most routes. There's a few cases where go router will reject them still, but not for your typical app that doesn't have route services and isn't using http 2.

B

Yeah, I think, by default, it's probably not enabled just because it does increase your surface a bit. uh There are potential vulnerabilities, but I don't think that's necessarily a reason why we shouldn't have it enabled because it's a handy feature, I think it's more just if you're not going to use it while I have it enabled.

B

C

Guess another question would be, would would it be something you'd want to tweak on a per app basis.

B

Perhaps I wonder if that would be converting some for the operator.

D

It would uh burdensome in what way would rock like they wouldn't know which apps have websockets enabled versus don't pretty.

B

Much and then, if, if the users were confused about when certain things weren't working, they might be asking the operator about that. So it might just cause a little bit of overhead, but I don't know if that's a reason not to do it. It might just be a unfounded fear.

A

Yeah, I I guess.

B

I would agree.

A

I mean I having had that same exactly that experience with post paco where everything seems to work, except that it just doesn't. um I had some colleagues at emc trying to use post-facto from inside the emc network and web sockets, weren't, working and plus pacos is.

B

Like you know,.

A

It seems like it's working, except for it's kind of lame, so I I.

D

A

My preference would be just turn it on and like if, if we need to give people a lever to, you know, turn it off in the deployment later fine. But I think.

E

A

Point that like trying to get it to the lineup of where we're going from an api perspective makes sense.

D

Yeah and like to my to my point like I'm, I'm not saying that's a deal break or anything it'd just be nice to know um what gateway api provides in this area in general yeah. I think I'm I'm on the side where we should just turn it on, because if it's something most apps we're getting for free from go router, then that's going to be like an annoying compatibility issue like everyone's going to be confused. Why um this app that used to work isn't gonna work anymore.

F

We could have a global config to turn it on or off in case. Some operators really don't want it, but maybe that's for later, like maybe we just turn it on and then, if someone complains, we make it possible to turn it off.

D

Yeah, I'm leaning towards that because, like there's so many layers in the routing thing like, if operators really don't want it, I'm pretty sure they could configure their load balancer.

A

D

A

Yeah, they might already have like organizations that don't want websockets, probably already have firewalls that stop websockets from happening.

F

The gateway api thing is concerning like so it might be that we enable it and then the day we want to switch, we won't have it, although it would be really weird that they don't support it. Yeah.

D

F

It's quite common, like I think there is a whole frameworks based on the idea of heavily using web sockets.

F

So it's not nowadays, it's not like a niche feature like lots and lots of apps use. It.

G

Is there even anything in the cf-3ms releases that would let you turn off websockets installation wide. I was looking in the go: router job spec and the routing release, bosch release and uh I didn't see anything but I'll I'll drop it in here.

G

This seemed like a place to look.

B

uh We also have a comment in the chat that is in support of enabling websockets by default, uh due to the number of issues that they've run into with uh deployments with a website, that's disabled, so yeah I mean.

F

It's probably very tricky to figure out. That is the issue like if you deploy your app, that is web sockets based and it doesn't work it's going to take you a while to realize. Okay.

F

Actually, it's not supported that the like a kubernetes level so yeah, it's probably better to just leave it on.

B

Cool, I think it makes sense. I think uh we have way forward. We can drop some commentary into that, pr that we discussed it in this meeting and that we think it's probably the right way to move forward, and then we can also explore the whole gateway api thing to make sure that we're covered there.

B

Any final comments on that before we move on cool right on so the next.

D

B

Is about api docs.

F

Yeah, so I was about to create a story to document the new task endpoints in the api in the docs api dot, md file, and I opened the file. I took a look and I just found a bunch of issues uh it's first of all, I think it's a bit out of date.

F

Second, it's kind of inconsistent like different endpoints are documented in slightly different ways and um yeah there's also a bit of redundancy with the existing cf dogs.

F

So a bunch of stuff is just kind of copy and pasted from the docs, which I think is not necessary because, like we should strive to maintain just the delta, basically between us and them, so that we don't have to chase them, so it basically sat down and wanted to kind of overhaul the whole file uh make it uh structurally consistent uh fix any issues, add stuff that is missing all that stuff. It's good. It's taken me a while, but expect a big pr with basically a rewrite of that file.

F

um Yeah. If anyone has ideas, my idea is basically list all the endpoints we support and kind of imply that the ones are not that are not in the list are not supported.

F

Every endpoint links to this official cf docs and then there's a paragraph with the differences. So it's like okay, this is this is the end point as it should be. This is the these are the differences. We only support these parameters. We only return these fields or, like we don't redact these fields, that kind of kind of stuff and I'm going through the code actually to kind of, because I first for many endpoints, there's nothing written at the moment.

F

So I'm going through the code trying to figure out uh what we support, what we don't uh so yeah, I guess reviewing that file is going to be tricky. uh It might by the end it might contain mistakes, but because, like it's kind of odd, I kind of skim the code and be like okay, all fields are in the payload type. For example, are they used? I kind of try to see when we convert the payload.

F

If there is any field that is dropped but like I there's only so much I can do, I guess I guess we can with time we can figure out if there is mistakes but yeah it's a tricky file to obtain, which is, I guess, is the reason it's kind of is in the state. It is, but hopefully it's going to be better after the pr.

B

Cool appreciate all the effort you're. Putting into that um another thing, we can always spin out a bunch of chores for people to look at the different endpoints. That way we can sort of spread the load. If you want to take first pass and then we do that as a review or you can start spinning them out now that also works.

F

Yeah, given I wanted to basically change it to make it all consistent, I want to do like a kind of a big bang, although it does mean it's going to take me a while, and I have to go through the end points, one by one, it's kind of tedious.

F

It would be nice to split it, but maybe we one thing we could do is we could review it and point by endpoint have like review chores or just I don't know, people can skim, and I guess if you worked on the endpoint, you kind of know better. If what everything is uh makes sense or not.

F

D

You could start with, like the overhaul of structure and everything and leave the details section about what is implemented, slash not like, maybe vague or not, filled out, and then we could go through them by um by the way they're structured in the in the regular api docs, which is by resource and then and then maybe that would be easier.

F

Yeah I'll see, if I find a way to split it out, I've currently done a few, but then I realized a few resources were not mentioned, because I guess we they were added later. So what I did was I grabbed the two docs and got two lists and delete it from both sides until I was left with a bunch of routes from the code that are not in the document, so I backfilled them just the names so that at least we have a complete list uh yeah.

F

I might maybe leave those empty and then have separate stories to to actually fill out the details. Yeah.

D

Like I kind of think like the most valuable thing in there is just the um the http verb and the endpoint as like a tracking thing that we've at least done some implementation on this, because, like generally, I think you can assume that the implement, like there's there's things that are missing or things that aren't 100 and like that's like um our initial goal on the stock, was to track the end points and like and document the delta. But like that didn't happen. It like it started to happen and it got lost over time.

D

And I'm I'm worried that, like um we'll we'll make it mostly accurate this time, and it will continue to to drift, though yeah.

F

It's almost unavoidable, I guess.

E

Basically, two things here: first of all, can we have something more like a table with just the end points where we like basically tick off the ones that we have implemented, the ones that we haven't and then separate talks explaining the differences only when needed.

B

E

Can we automate that somehow, by like introspecting the code that there are some tools that are looking at I mean we can? Probably we have some some structure to our handlers, so I guess maybe this can be somehow automated. I guess at least partially and that's.

D

Looking at like how they're registered or something.

D

Or we could have put additional structure in.

E

And then I said two things I forgot was the second one. ah Maybe uh we should just remember like these are talks so they're, probably part of the output of the respective story. Maybe every story should request talks updates, but that's a bit fake.

A

Yeah they used to be and then somehow we started.

E

Yeah, I mean it's vague, so maybe if you can at least automate something that can put the structure for us and then it's just empty, but we know that we have these points implemented like a couple of green ticks or something like that, so that at least someone can look. How far we are into implementing might help might be tricky, though.

F

It's very typical to have this like in the definition of done for stories, but then yeah. It's very typical to forget.

F

Yeah, what's tricky is that different endpoints really are sometimes uh they could be really surprising to use so, like some endpoints are there, but they do nothing.

F

So you just put it there and it's like it's, not just okay. You can use this endpoint and expect surprises. It literally is an op. So I like to document that yeah or it's like. We only accept this feel these filtering these ways of filtering, but no others. So it's like. Oh you filtered by this thing, and it worked. You filled it with everything. It works, you kind of assume it works, and then you try to filter for the third thing.

F

No, it's not supported or there's a tricky thing, for example, for life cycles that I figured out by looking at the code that, like people can pas, we sometimes we accept fields just not to break clients, but then we ignore those fields. Let's, for example, the life cycle there's different types of life cycle, because you can in cfo vms you can create apps with docker images or you can create apps from sources with build packs and stuff, and what we do is we will accept any value, but then we will.

F

We have configured defaults which are not defaults. They are like they override, whatever the user passes in.

F

So I like to document that so that people are not surprised when they try to use a value. That is not. They just pass the value and see that it has no effect.

F

So it needs to be written like from a user standpoint. I document any surprises they might have when trying to use the thing, although I mean I know how many people will actually try to hit the api directly and use it as dcf api versus just using the cli.

F

So I don't know if there is room for a document about just which cli things you can do versus api, but still it's also useful for us because, like if I guess one of the goals could be to make this document shrink as much as possible and at some point say you know this resource completely compatible. This piece is completely compatible and kind of shrink that document it will never be empty but can get as small as possible. I guess.

B

Yeah good commentary: I think this will be an ongoing battle to ensure we keep things up to date uh and sort of probably should review focus on making sure everything is up to date when we implement features or generally any story that touches anything around that so good con.

B

uh I don't see any additional topics if we're done with this one, then we'll probably wrap up. If you have final comments, you know now is your time.

B

I don't see any hands going up, so I think we'll wrap up uh short and sweet. Thank you all for the discussion. Thank you for everyone attending again, and thank you, john for your commentary and uh we'll see you again in about two weeks. Have a good one. Y'all.

A