Kubernetes SIG Apps, 12 Aug 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Apps 20190812

Description

Led by: Kenneth Owens
Co-hosts(s): Janet, Adnan

Announcements:
- First week doing bi-weekly meetings
- Apps CRD release

Demo (10-15 mins)
- Octant - Bryan Liles

Discussion
- PDB GA https://github.com/kubernetes/kubernetes/issues/66811 (1.16 target)
- CronJob potential changes https://github.com/kubernetes/enhancements/pull/1145 (Lets add comments)
- Check In on sidecar (1.16 alpha target) https://github.com/kubernetes/kubernetes/pull/79649 https://github.com/kubernetes/kubernetes/pull/80744
- TTLAfterFinished controller for Pods (1.16 alpha target) https://github.com/kubernetes/kubernetes/pull/80957
- StatefulSet MaxUnavailable (1.16 alpha) https://github.com/kubernetes/enhancements/pull/1010
- Long term fix for spurious rollouts due to mutating admission controller

A

Okay good morning and welcome to the August 12th meeting of say, gaps, we're gonna go ahead and get the meeting minutes as opposed to my chat. We've got a demo today from something about something: that's pretty cool from a Bryan Liles of Hettie, oh well, VMware now Brad.

A

But first it's a few announcements. We're gonna finally cut an app CRD released this week. That's one- and this is gonna- be our first week doing bi-weekly bi-weekly meetings. This was kind of something that was proposed and that the community seems to desire. Everyone seems fine with it. If there's any problems with that, please let us know on the mailing list or drainage be, but now back to my demo Brian. If you're there yeah I'm here.

B

Cool yeah, you can have.

A

B

Routed somewhere else, so I was I, didn't even hear you, so uh you ready for me to demo I can do that right now. Let me kind of makes everyone knows: you're a pretty cool.

A

Solution that gets rid of something that allows me to get rid of something that I don't want to support anymore. All right.

B

I'm Brian Myles engineer at VMware and over the past few months we've been working on this tool that is now named octant. The reason this existed is because job Aid and I were talking last year last summer about just projects in the space and after we decided the sunset case on it, we were going to be something else and he said well what how do I know if my workloads are working and we went through a whole bunch of scenarios and figuring this out.

B

But what we came down to was a tool we were going to build a tool that will allowing you to show if your workloads are working in a graphical way, because tools like you, control and and all the other tools in the space are great. If you know how to use them, but everyone knows how to click. So we started a rating on that idea and then what happened?

B

Is we realized that there's a lot of context around this, so we ended up, saying well we're going to need a dashboard and and to be told we did look at the kubernetes dashboard, but I am NOT an angularjs expert and we didn't want to support bored.

B

So we just ended up writing one and what I'm going to show you today is a tool but I'm going to show you it, because we're only going to do this in a few minutes, I'm going to show you it in the scenario so I'm a developer, I have a site.

B

I created this cluster, this morning's on digital ocean. It's demo that demo one do apps, Brian, L, dot, dev and I go to the website and it doesn't work and, if you're, a developer, who isn't really in tune with kubernetes, because you're, not a kubernetes expert you're, an expert at delivering software. This can be a little bit daunting, so we can use octave to actually do this.

B

So what is often dashboard shows you information, ultimately we're going to give it to updating a whole bunch of information, but the neat part about it is that it builds. It builds the context that you require, because we can extend it through plugins and I'll talk about that a little bit so with often what you have is a view of a cluster and its names based and then you can always see your cluster. Your cluster scoped out are objects as well, so we're looking at default as a developer.

B

I do know that my application is running in demo and in demo we could see the whole main space and the whole namespace is there's a we split it up and so workloads and our back and custom resources.

B

They can storage, and things like that, so you can find things so here's a list of everything running in this in this main space, so without knowing too much about this well I can see that there are pods running and if I click on this pod I can see that I can go down here to the events since that it started it and other things that I can do is I can go on to logs, looks like it's running and then something else I might want to do is actually is this thing even listening in the cluster?

B

So the reason this feature is actually a mis tools because cube Schoop control port forward just follows my mind and I hate doing it. So we added a port forwarding right in the app and all right. Well, the ethics is up and it's running fine, but we don't. We still can't see why it's not working so the original reason we wrote this was to show this resource. So what we do is we go through the kubernetes api and we create a dag of objects that are connected and we'll see here that there are some pods.

B

A replica set deployment in the service account, but wait a second there's. No there's no service, so there's no way really one can talk to it and there's no ingress, and this should have an ingress. So let's go look at this thing, there's a Hamel that we just have out here that you can actually view in the future. You'll be able to edit this here with context and what we're looking for here is actually just the labels for this pot, so pot labels at HTTP done.

B

So, let's go look at my services and if I look at my services, I have an agency didn't service and I. Look at it and right off the bat I can tell that the selectors, for this thing is it's invalid and if I actually go to the resource viewer for this, because I can also view.

B

This thing in context of my the service says well, has no endpoints because doesn't match any pots, so this surface could not serve traffic, and then we can also see that the ingress is broken as well and we'll go fix that next. So what I'm going to do here is because I don't have an editor for everything in here, I'm just going to edit it outside of the of the of the app real, quick alright. So this pretend that cool things are happening over here.

B

So I've updated the service now on the backend, just using cube control edit and now what we're able to see is more things now because, because the pods are the surface knows about it, now see pods. So we have a bigger graph. So if we were to go back to this still wouldn't work because it's ingress right here is spread as well. So let's go I'm gonna fix that ingress the next. Let's go! Look at this ingress before we fix it. It says this is the reason why things it's not working.

B

It says back in for service. Ht bin specifies an invalid port. What we've done here is we built a system where we can actually write checks for objects to make sure they're, correct and one of the checks for an ingress is actually looking at the service and seeing if the port's match up. So if we go look at this bad boy and we look at the gamma for it, what we'll notice is that the service name is HD Ben and the service port is 80.

B

But if we go back to the service, we notice that it's listening on 8080. So let me go fix this one all right once again, cool things are happening all right, so we go back and we look at this thing here and we look at our resource viewer. It quickly updated itself.

B

This is all real time and the way that it works just because I'm talking to sig apps is that we actually maintain a section of or a group of Informer's, and we look up the informants whenever we go to a namespace and we're using the informers to do all these things. So now that I have all my everything is that green?

B

When I go to my website, my website, pops up pretty cool and what we're trying to do here is help people get context, so they can actually figure out how to solve their own problems, but we can do better than that. So one thing that we noticed in the space was everyone's moving. This yard east and the problem with the CRV is well. What is, how do you see a seat? Rd? Well, you could do Q control.

B

Get CRD name, and then you can try to figure out looking the spec names and try to figure out what the resource name is and then look to see if it's clustered names, if it's cluster scopes or name space cooked, but now we can get rid of all that. So what often does is it displays your face, CR DS in the namespace in estates in this, and it shows the the the cluster scope once in the cluster overview in the neat part about this.

B

Is that as CR DS are added, this list will update so you'll notice. This list has four items in it: it's only showing these four items because it sees resources for them, but didn't it wouldn't show anything. So what can accent do for CR, nice? Well, I'll show you something pretty neat here. If your CR B has custom columns, octant will actually took advantage of it. Just like you, control get so this right here. All these columns are dynamic and even better.

B

Yet what we can do is because we can look inspect and we can look at status and use it as convention. We can actually build up, we can build up our configuration and our bases our output, basically dynamically. We don't know anything about this, but not all CR DS display this information. So a good example would be like this. mmm This order's, oh, we find one, that's not good. Okay, sometimes we don't get custom columns, so the cluster is your cluster issuer from from just stacks.

B

Cert doesn't really show anything, so we can't build up anything on this, but we show you something else we can do as well, and this is something that we're building on that we're going to introduce an easier way to access this in our next release. So we have this concept of plugins and why are the creative plugins is because with C RDS and actually just data in kubernetes in general, there's no way that we could code up a solution that would solve all the problems.

B

So we decided to give the users and plugin authors and interface and they can create. So what can you do with plugins? Well, let me show you something here: real quick, so I have two closets installed and we can actually see them. There's a cron demo and then there's just the regular old demo plug-in that I put in here, and they do two different things. So one of the plugins, let me go to a different name space. It has pods in it actually can embellish it. Information so on this particular screen.

B

What's going on, is that the plug-in was able to update the configuration information, the status information and then down the bottom? It was able to create its own output that we could actually embed in and what also else the thing you can do is that we can actually create new tax. So that's kind of cool.

B

Another thing we can do with plugins is. We can also say we don't have to work with just built-in types. We can also work with CRTs, so I found this crontab CRD demo and I created an and I created it, but now I created a crontab plug-in that actually can show extra output and show an extra screen. So let's say: if you wanted a crontab editor, this is a dynamic editor in the plug-in of octave does not know anything about it. So we'll just change our crontab a little bit, so we can see.

B

So what it's actually doing is it's going to the API and made the change and we're just waiting for the API to update I might come back to this one. This is a visual cluster and my master notes are a little bit slow today. So let's see what else that I can show you all. Oh here's something else.

B

I can show you not only can plugins embellish existing objects, but they can also change the UI, so the sample plug-in actually creates a new mini structure that just gets put in at the bottom and actually there's one here for context. It gets flicking here at the bottom and it gets its own routing. So I'll show you how this works so I'm, getting a minimis custom. Plugins code is actually not an octa at all and I'm changing I'm changing my current URI and what you'll notice is that that card is just my card tap.

B

It's changing because it actually is looking at the path. So what are we doing with octave? Well right now? What we're doing is working on making the things, so it should be obvious, so we're working on fixing crashing bugs and things should be obvious. If I go look at it. Let's say: I go! Look at an appointment, real quick! Here we go back to my demo.

B

Namespace, then I, look at a deployment not only giving you this information, but giving you context of what you can do with this information and allowing you to edit, where you want to edit and then basically just work with the cluster in the way that you want to work with it. Another thing that we're doing is creating a platform for our plugin authors, because you'll notice that there's we have access to metrics we're not showing them because I didn't want to actually have opinions.

B

We want to work with cubemetricks and we want to work with even other providers to be able to embed pod and even deployment statistics right in this page. We actually have the ability to do that and then also what we wouldn't really want to do is think about making this an application kind of like the github tool from github that works on Windows and the Mac and just making it feel like kubernetes is part of an application, rather than the scary thing that we have to deal with so I'm gonna. Stop there any questions.

B

I know, I talked really fast. There's a lot to talk about I.

A

Have one question so do I have any component to have to run inside of the cluster to make this workers it running purely locally? Oh actually,.

B

That was the that was one of the other big things, so this all runs locally on my machine. It just connects using these connects using client go and it uses your local cube config to do everything and the reason we wanted to do. That is because we paid attention to what happened to cute with the kubernetes dashboard. And, yes, you can secure it, but it is not secure in all cases. So really what we did is we said well limit exposure. What can you see in the cluster?

B

Well, whatever your keep config allows and what, if I, want to have a read-only view? Well create a read-only context. I don't know, but I actually can run this in the cluster. We could actually create a deployment put that in there put it in the service and then hook it up with the server's account and guess what you can run a cluster too, but you don't have to. Is it no? You do not have.

A

To and absolutely advise you not to, and if it's using could be my cubicle configuration basically, it operates via impersonation. So the view that it presents of the cluster to me is the view that I have access controls to CBM I could be cuddle. Configuration.

B

One hundred percent- yes, ten, that's.

A

B

So this is, we've done a lot of good feedback on this. I have not has not crashed a whole bunch. There are some definitely some crashing bugs in there, but this is new software. So it's a github.com, slash, VMware, slash, auxin I, leave the project up. I would love to talk about it or make it better and that's much feel cool.

A

The link to the project is in the meeting minutes.

A

If anybody wants to get out, it's only github in the VMware repo thanks Brian. That was awesome. All right cool anybody else see me hazed.

A

C

A

Right so now exciting demos over on two warring administrative, trivia, but important stuff, because we want to see if what we're gonna do for 116.

A

So starting from the top more in your here, we got pdb GA.

A

Are we looking to hit this in 116 yeah, I'm, hoping.

D

Okay, cool thinking.

D

And try again Ian I think.

A

You have the wrong issue, wait to you, though, and so no you know this is the right one. Okay, now.

D

Some I'm working on getting pretty beat CGA I could I as I was working. This I noticed this issue. I think it's interesting I just wanted to throw it out there and see if anyone feels this is something we should address before taking UGA. It does seem like it's a bigger question about how how do we do the synchronous, but it's an interesting case, and basically, if you have Windows, doesn't allow any disruptions, but the controller allows scale up like we should scale up and then kill pods.

D

A

Well, we can rehash this. We did talk about this maybe about a year ago, and the kind of conclusion that we came to is that I think I actually put a note on it. The design for interaction with PD B is that you should ensure enough capacity and your deployment to tolerate the plane disruption tolerance declared by your PDP.

A

Basically, if you always want at least two replicas and you tolerate of loss of two, your replica count should be for your maximum available should be two and we kind of I think landed that we don't really know how this would actually work or how we would support this semantics if we want to bring it back up and try to think of, like I mean it's kind of a little bit of a redesign, because it was never really designed to do what PD bees in general, an addiction in general was never really designed to do what this user is asking for work if I remember correctly, they were asking for is if I set a peb of zero and then I do an eviction.

A

The system somehow detects that the PBB of zero would be violated by this eviction then starts a pod prior to doing the eviction, which is just something like the system in general, isn't designed and to do if we want to block pdb, GA illness and addiction jail in this. That's fine, but we probably are gonna, have to reach out to say scheduling like that. We touched on to make this happen. We touch more than just the workloads api's or the eviction policies. I think yeah.

D

My photo is that it we proceed to GA, regardless of this I just wanted to raise to the question. If anyone disagreed all.

A

Right I mean the the length of the issues here. Anyone has any thoughts that wants to speak on them now feel free to use the time if people want to go and chime in on the issue and say: hey we'll wait a minute before we do this. Maybe we should think about this again. That's fine.

A

Because once PG a it, there's no like the next thing will be V, 2 and B. 2 is a long way away.

A

No okay! If nothing, if nothing else on this one, then cron job.

A

This does not work. There is a request for some essential changes to crime job.

A

In particular. It's with the day one is called a catch-up concurrency policy and conceptually from the design that this would look like is so you have a cron schedule right and it's based on crontab.

A

If, for some reason, when the system is down, certain schedules are missed. So let's say I run something daily and my systems down for three days in the event that three of those daily schedules are missed. When the system comes back online, well, somehow detect that and then launch three job based on some policy in order to catch up to the current crime schedule.

A

Matt had some feedback that you know you could use air flow for this, and my kind of thing was like yeah. If you want the functionality of kind of a bag scheduler or work clothes scheduler, which it seems like this is meant to implement, we've got air flow, which is community.

A

You know pretty robust battle, well battle tested enough, contributed by Airbnb and there's some people at Google that have worked on kubernetes along with kannberg kubernetes operators and allows air flow to run bats on top of kubernetes natively, and then the arguer workflow, which, if people are familiar with Kuby flow, is something that tries to be a cloud or a kubernetes native workflow workers trader is kind of a thing that does that there's some kind of technical concerns that I have about like. If we did this like.

A

What's how do we ever guarantee that the crime job ever actually catches up, like you know so, depending on what you've actually scheduled? It may be completely infeasible for the cron job controller to ever launch enough jobs to catch back up to the schedule and then like really, the conceptually crime. Job is kind of like based on executing on a crime tab like schedule and there's no catch up in that kind of paradigm.

A

That being said like, if we think, as a cig that it's worth exploring and it's a feature, that's kind of broadly useful to everybody, I don't want to like be obstructionist or block anybody from moving forward. If you know it's like, we really think this is hugely valuable and needs to be implemented in as a base feature on top of the workloads.

A

Api is because it's you know generally good for the community, then you know I, don't want to be like we shouldn't do that on the other side, though, if we need something that does this I think the guidance from cig architecture would probably be. Can you do this out of tree like? Could we sponsor this and inside of our cig as like a custom control or built using either accurate, very chaotic, Ruby builder, which probably just could be builder in it at this point and host it?

A

That way, as opposed to doing it in tree, which is something else I think we would want to consider, but I, just kind of wanted to bring it up here and make sure it didn't. Look like there was a lot of feedback from anybody other than Janet Matt and myself and I'd like to get at least you know.

A

Well, Thomas did CC my up I'd like to get my opinion on it and people who are more actively working with job just to see what, if there's any dissent to this or you know so,.

E

Yeah to add a little bit on I I haven't got a chance to read through the proposal. Yet I only heard about the topic shortly from Thomas when I returned I, like literally returned from a over a month video just last week, I'm still catching up with stuff.

E

I'll definitely leave my own comment, but from the discussions I'm I'm, leaning towards what you just said, okay, and with regards to I'm, not sure if it's, if it's a good fit for cron jobs, it should rather be act and react immediately rather than in the past, but I'll leave a more thorough thoughts and in Dasia itself, okay,.

C

A

Whether we do whatever we decide, I'd like to try to give feedback and close it out, so we don't leave colourful, believe, that's the guy. Their name is colorful yeah, so we don't leave him or her a mushroom, whether it's a guy or a girl hanging and like it's been about 27 days, but I think that kind of close it out for him. So if we can but say like a week or two get anybody's feedback in and make a decision sure that cool.

A

Okay, what's next.

A

Just one of the health check on sidecar, particularly as it pertains to 116, if we were going to try to go for that Joseph for you here you are yep. Okay, do you think you want to hit 116 and do you need any support from us to do it? I mainly.

F

Just waiting for abuse at the moment we've got a a Pio Pio. It's been open for a while I think was meant to look at thatmy. Things he's been on holiday for a while, so that we need another chance really.

C

F

Then we've got a PR with the Coolidge end-use, which also have ammonia on top look at that yeah, my stop team sig notes, mailing list or something for any assistance on getting used to it. Okay,.

A

Cool and that's your your goal is to hit alpha all in 16 yeah.

F

Yeah how's it go yeah cool.

A

I can see what I can do to try to ping people and Janet. If you can try to ping some people to see if we can get the reduce and I'm sure it's not because people are ignoring it so much as everybody's really busy pretty much all the time.

A

But well, let's try to hit that for off over 116.

A

Janna, I think you were the one who brought up TT off their finished controller for pods. This is an implementation based on the design you offer like a year to go. Yeah.

G

So I'm trying to get this in for 1/16 I, think tomash had has reviewed my earlier PR firm for a TTL for a job. So maybe tomorrow you can also take a look at that. Pr I put it in the meeting, notes.

G

H

I'm a light on all the notifications from github so definitely helps to have a link for the PR and I will review it. I think this will be also useful, like you recall how we leave evicted spot around like if you have a replication controller and it leaves evicted folks lying there if Q blood fails them and nobody has ever deleted them. I think this could be actually useful for that case as well, so I will review it and when we have it, we can talk about the Epictetus case. Sure.

G

A

It's a good thing for us to do for general cluster stability, yeah, that's as well, and January you're targeting this for 116. Yes,.

G

A

Is it going to be.

A

Sorry would be alpha or beta or just yeh. It's.

G

Still alpha so I'm trying to get this in so in 117 I can bring the TTL controller to Cepeda. Okay, so.

A

What the alpha is once okay and you're gonna introduce it with an e to date on it. Yeah.

G

A

All right, that'll be actually probably very useful for the number of clusters that you've seen who've gotten into trouble because of GC issues, and the last thing was the staple set max unavailable is.

A

Lying here today, he's not okay, I think we're gonna target a 1/16 alpha for this eye.

A

From what my egg said, he's looking for one of us to go through and act, the enhancement pull request as implementable, so we can move word for 1/16 alpha. My offline feedback was you know if you're ready to move forward with implementation? I, don't think you should block on it.

A

If there's I I'm looking for the PR that he has open with the enhancement move to implementable, which I can't find I'll, try to collect that and post it and it guess for next meeting, make it more public, so people can leave any feedback they have before he starts. You know opening a PR, but at this point I think we are gonna move to 116 alpha as the target for that and yeah I'll try to sync up with him offline and finish that up: okay, there's anyone else anything else. They want to add for discussion.

H

Just wanted to sing back I think I proposed like two weeks back or sometime, like that. We talked about stabilizing the vertical controls with regards to admission controllers that could modify the pod template, yeah yeah. So in the meantime David each came to me and he proposed we could do the same with server-side apply and keeping the owner references in metadata. So I would be also looking into that. I just wanted to briefly kick in on that. Do.

A

You think that that so I remember Janet and I talking to Daniel about that same thing and I. Think if I remember correctly and Janet correct me if I remember incorrectly, that the conclusion we came to is yes, that would might work. But if requires service, I'd apply to goo ga and the concern is that the time frame for that was kind of far out at this point. Maybe it's not as far out as it was back then so that might be a yeah.

H

The time frame is closer and I. Think the ask from them is that this could modify validate the approach they are taking because there are still open questions about during this time. Okay, so let me do for both ways, but I still need to look into it. I just wanted to update that I didn't forget about it. Okay,.

A

Do you know when it is made up I.

I

H

Know actually I can see I take it like a week ago and yeah.

H

Okay, all right sounds good. I I think I heard like the time frame like the end of the year like, but it might be DGA.

H

Definitely nothing officials, it's probably like targeting the end of the year. To close it.

H

I will sing back when I should try to stop.

H

And I think it's alpha right now, in the master, at least it was what I was thinking. It.

A

Yeah I know they merged it. I just didn't know what state he was in her if it was generally available on like cluster by default or any other distribution, but I think that doesn't regard without respect to that. It doesn't really affect that as a plan, but all right that sounds cool yeah.

H

I think end of the years. Probably soon is that we can. You know our work can be like stabilized and ready for marriage. I think the schedules can pretty meet.

H

I

A

I

A

Right, nobody has anything else to add. Then I guess we'll give everybody about twenty minutes. Back again, we've moved the meetings to bi-weekly, so there will be no meeting next week, but there will be meeting the following week.

A

Add any issues or demos or anything you want to the meeting minutes and we'll meet up to schedule. Those and I guess see everybody in two weeks, this time and Brian thanks again for an awesome demo, Thanks all right, don't you guys later.

B