Kubernetes Office Hours, 21 Mar 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes Office Hours 20180321 (US Edition)

Description

Come join us, third Wednesday of every month! https://github.com/kubernetes/community/blob/master/events/office-hours.md

A

Okay, stuff I want to get all the want to get all the adjustment out of the way, but we are starting the stream now.

A

And then just waiting here to get feedback from the channel or how resounding I will.

B

Say you sound definitely better than this yeah like.

C

What more consistent I would say, yeah.

B

It was just weird because you woulda Kate you'd be fine, and then it would just like drop off like you were talking through a pillow and then would come right back up. I.

A

Don't know but after I was done, I had to unplug everything.

B

And plug it back in so.

A

Let the magic smoke out so I'm in the channel. If you could hear us, that'd be great. If you could, let us know how we sound. Could someone actually type that in it looks like someone's someone hears us? Okay,.

C

A

Right here, alright, so we are going to get started how's the sound throughout. That's the cast, please let us know how we rounded we're here to serve you so welcome to the kubernetes office hours. This is the US Edition. We run these once a month. Third Wednesday of every month, one session for the European time zone, and this is the West Coast friendly time zone. I'm gonna, be your host George Castro, I work at hep deal and panel. Freeze, please introduce yourselves: let's go Bob Jeff, then Mario, okay,.

C

Hey my name is Bob killin I work for the University of Michigan, where we run several large kinase clusters. My specialty is sort of cube, ADM, bare metal and Federation and all the other random things that come with that.

B

All right, my name, is Jeffrey Sica I also work at the University of Michigan. My main specialties are the cron jobs, batch scheduling, jobs, API and, like initializers and admission controllers,.

C

You're, a developer yeah.

B

I'm, a developer.

D

Suite and I am Mario Lauria I run a meetup group in and I wrote called Orca structure which all of the guys currently on the call are very heavily involved with I also work for a company called liquid web, which enables small meters as businesses to run their websites effectively on kubernetes. So my focus areas are pretty much everything ever and lately it's been storage. So it's me a lot.

A

Of storage questions earlier, we.

D

A

Did- and we have two more folks joining us at some point here- I'm in the happy office today and they're grabbing a conference room like as we speak so when they show up they'll show up. So let me cover their ground rules first, if someone can plop the YouTube URL into the channel with a slash, live at the end. So as we get new people, they could just click through YouTube.

A

I would really appreciate that and, as we do, this feel free the audience to just add your googling for whatever it is, we're talking about toss, it feel free to toss links and we'll read them, we'll read them aloud. Okay, so first things. First, the.

C

Rules judgment free zone.

A

We might get new users, people that are just getting started. So let's keep this positive as far as helping each other out and things like that, while we will do our best to answer your questions, there are certain things that we can't do like. We don't have access to your cluster.

A

You know in your approximate environment kind of thing, so in in cases where you're having a deep technical issue that we can't really diagnose, we will add a minimum either point you to the correct sig or to the correct place in the documentation where you can get help to kind of least give you a place where you can get started during your investigation and research. Panelists you're encouraged to expand on the answers with your experience and pro tips. So you know don't just answer the question, but if you have any insight.

C

A

Know things that are bird you in the past or things that are going great. Let us know that one of the reasons we like to do this is kind of high bandwidth environment where we could, just you know, talk about what we're doing in production and things like that. So audience you could help I if you're, if you're, finding URLs, especially official Docs blogs or anything.

B

A

Help well and that's relevant to the topic at hand feel free to just whack it into the slack Channel and then we'll get to it, and you can help us out by tweeting this spreading the word paint forward. If you get help, try to help someone all that kind of good stuff. All of our sessions are recorded and available on YouTube.

A

So if you're using this as a work resource, also, let us know, because I'm really interested in seeing how teams that are doing operations and development at kubernetes can better use this forum as a resource. So I had this dream that people sit in the break room just watching kubernetes office hours. So that's you. Thank you.

A

Let's see if you want to sit on this panel, this is actually open to anybody. Just let us know we'll pop you in especially if you have expertise in an area where we are lacking with with the current panel.

A

Let's see, I am currently working on getting us prizes and raffles and t-shirts. So if you ask a great question, you'll get a kubernetes t-shirt and all that kind of good stuff.

A

However, I've not yet been able to secure that kind of stuff, so please keep participating and helping us out and we're gonna make it really kind of make it like a fun thing that that that we're gonna be doing, and also if you're gonna, be a cube con in May we are actually doing a live office hours or if you go to keep cod we'll have a panel, you can ask questions and then just basically, you know pick pick developers, brains and in a kind of more organized, so I'm hoping that you randomly run into somebody at a conference and, lastly, feel free to hang out in hash or hash office hours afterwards and get to know each other and hang out and help each other out.

A

So with that, we are going to start to take questions. So please start to think about questions in your head. Please preface them with question and then a colon, so I can kind of see it. So it's obvious to me so I know how to read it out and apparently is the live stream only showing Bob and me talking. It never shows my face. I, don't know why does that so with that I'm gonna switch us over to panel mode and zoom.

A

Oh, we have a new person. Alex, you want to introduce yourself. Sorry, I did not notice you. There.

A

D

Okay, just was curious with Delphis hours about sure.

A

ah So using kubernetes.

B

A

To give us your quick, intro, I'm.

C

Not at this point, okay.

A

Yes, all right, so let's get started so we've got a first question is from Luis this question about storage. Here we go Mario well, will kubernetes take my raw block devices set up abdi raid and format. The Arabian XFS.

D

A

Okay, there we go.

D

So that is, I was actually just typing. The textual based answer to that really quick, but the quick answer vocally is that it will not. Kubernetes is not meant to completely provision your storage for you. It will interface with your storage using the container storage interface. Just like you can take your own time, interface, cetera and so really. You're gonna have to look to other solutions. That kind of enabled you to get your storage ready and represented, or you know, attach it to kubernetes to use effectively there's a lot of solutions out there.

D

You know now that port works rook that may use different sorts of backends SEF. Even some ext4, some XFS and those are those are file systems, but they basically orchestrate the setup of those file systems on the disk, specifically port works. Who actually does its own clustering as well Cooper days itself does not match any of that so to kubernetes, you're, requesting a volume you're using a volume and there's an interface for which you utilize that volume right. So that's that's how storage works at the the kind of top level so yeah? That's.

A

All here's a follow-up, however we're not I, will take my rob block device and format it to XFS. Just no raid yeah.

C

I do believe, like rob block device. No, it's going to be used by the like si si driver and other things of that nature like the rook or stuff, might take care of that sure.

A

All right, Lewis, I, hope that answers your question and if not maybe we can find out.

A

Because people are posting memes here we go all right: new puppy, ass credentials, :, it's nice to see that secrets exist in kubernetes and how it functions is this kind of just a short slash, midterm solution? Does kubernetes think that they will want to extend that or offload that to external services better, like vault, slash, KMS type behavior, it's a super hot topic in real production for kubernetes. It's a lot of the hacks folks.

A

Do today are honestly a bit cringe worthy most people just want to get the password to their app outside of source control.

C

So I believe it's already some vault and integration. There I'm.

A

Pretty sure to this question, but he hasn't joined yet so new puppy. We might have to come back to that unless Jeff or Mario have any insight onto this one I.

D

Don't I envision it being something again like an interface right where Cooper neighs is not going to decide for you what the best option is, but it will interface with other really good options: ie vault, that's my two cents on it. I don't know much about secrets. I haven't really worked with them, so yeah, let's hopefully, Ralph, can have some more insight on that yep.

A

D

A

Will come back to you.

A

All right, Chris asks question any plans at base pod security policy in place at cluster build time, so the mission controller can be enabled from the start. I.

B

Haven't seen anything over now, I'm gonna go look because.

A

B

Looked that specifically, yes,.

A

This feels like I, can always stick my head out and see Joe.

A

A

Alright, we might have to defer that one also until we will come back to you when Ralph and or Chris join sorry about that. Emory says scheduler question. Micron jobs are not running on time or at all. Some are running, but many are not. I have 15,000.

D

Cron jobs on an 8, node and.

A

D

That large cluster, all the jobs.

A

Are quick, 30 seconds to run, but some of them are not running? Where can I go looking for hints as to why they are not run.

C

The the criminais is events, and at least that be my best guess. To first look, I mean.

B

Just based yeah, just based on the description there I'd be curious if it's 15,000 events that are all trying to run at the same time and if so, they're probably backing up and then timing out.

B

Okay, sorry I was catching up with the chat if.

B

Yeah cute cuddle events will probably give you the best first place to look without without like actually diving in myself or Bob. That's probably the first place. Are they around the same time? Some are every 15 minutes.

A

B

My initial thought is depending on how how much or how many resources each job is trying to take up. You might actually be hitting the upper limit of your cluster and so they'll just time out and.

A

With that show up in the log that.

C

Should show up in the events that should.

A

Show that's yep.

C

B

But the every 15 okay, so so, is every 15 minute one not firing, or is it just some because again like that is a significant number of cron jobs. So there's there's a lot of room for you to run out of resources. Oh my god,.

A

All right, Chris, you're, killing me with the memes here cause I'm scrolling off you.

C

B

Little arrow on the side, but yeah now I'm, trying to find where Mr a commented again so some are most is so the once a days are running, but the 15-minute ones are not firing and cute cube. Cuddle top nodes is showing free, RAM and CPU. Do you have any resources allocated to like? Do you? Do the cron jobs actually specify how much or how many resources each job requires?

B

Cuz again, you might be hitting a point where those jobs aren't going to run because there aren't enough resources to schedule them and it looks like we have more people on yep.

A

C

E

Joining sorry we're a little late, I know.

A

Where he's I go ahead and reduce yourself, and then we have a few backed up questions that we are hoping.

E

A

Know because we hi.

E

My name is Joe thanks and I'm. Support Manager at hep do.

B

Another Michigan native yeah, easiest thing, I, would say: MRA is use, cube, cuddle events, if you want bobber I, could probably at least help a little bit with the output. If you want a little bit more like if you want someone to bounce stuff off against yeah.

A

B

I've never run into that with cron jobs or job scheduling in general, same.

C

A

Alright, moving on I'm getting a lot of questions, so, if I accidentally skip yours, please feel free to repeat it. Louis once I mentioned, what's the saying that you mentioned a tool that may automatically set up storage, I didn't catch the name of it. What were the tools that you were mentioning Mario.

D

I'm sorry for setting up storage, yeah I'm, not looking at the thing exactly right now.

D

So rook is a popular stretch. Divider there's port works. There is some of the enterprise offerings have plugins now NetApp. I is one of them which we're talking to the the container storage interface and I. Just have the blog post up, but the container storage interface here, I'll post this blog post and you can kind of get more details. This was the official announcement around that bad I believe is in alpha still or it's moving the beta soon. So that's going to be coming real soon here, I know.

D

Pork specifically is actually working to make sure they work with that as well, and so this is will be be just like that. The container runtime interfaces- this will be the interface for kubernetes to utilize storage and oh there's also a table I found earlier with all of the storage options and whether they support certain rewrite modes. In certain cases you need a volume, that's accessible from multiple containers simultaneously, and not many so.

A

E

Seen there's some fault is one one of the options is there is being able to basically create a custom controller that then just injects into your environment, a secret basically, but there's not currently a lot of other options. I've seen native to kubernetes in that regard,.

A

C

You can store in a secret, you know.

A

C

Probably wouldn't be hitting the pod directly I'd be going through service. You can try hitting the IP of the pods to the name, because I don't believe the actual, like name itself, is registered and cube DNS, but I'm, not a percent certain on then. If.

E

It's created as a as a standalone pod. The name is registered within the namespace. If it's part of a deployment, it's not going to be registered. Okay,.

D

E

D

That that kind of begs the really you should be explicit and create you know, declarative services for pods that you know need to be accessible by other pods in the cluster, and this might be you know, across native spaces. This might be isolating them same namespace, but you should be explicit and define those things so you're not kind of playing guessing or trying to connect directly IP addresses.

D

C

C

I'm, honestly, not sure about calling the pro script from like another container I.

E

May I might actually put it as another container in the pod. To be honest, it might be a little easier to do that way. But how do you call it.

C

And if you have some way of like talking over 127.0.0.1 or something like that, just to trigger it to run then yeah you could put in the same pod and signal it to run. That way.

C

C

You're muted, in.

B

That was weird, sorry, is the web service also running Perl, or is it like running Ruby or Python, or something because I was I was so if I was muted? You didn't hear me say this I'm almost the way.

B

B

D

How do I now pull up.

A

Are we still getting echo nuts.

A

A

And second bear with me here.

A

B

B

Not me either, but.

C

Still, don't hear ya, still don't hear you dude, okay, I think they can hear you, but we can't hear you.

D

Well, why don't we do what we can.

A

D

You're George yeah, I'll I'll, take this in here and oh, don't want to only getting a phone call. Now cool, no I, don't need to answer that. Okay. So let's, um let's look at what the next question is here. Were you on one of the one of the questions or things Kyle.

C

To be previous, I, don't remember the.

B

Last question that I saw was one from a Freeland and it was a stack overflow question. Okay,.

D

B

If you look around for 15 a Freeland, it was stack overflow, link, minigame.

D

External registry for Zito the pool image, interesting, yeah, okay, so I guess: does anyone here I'm pulling it up right now getting given way?

D

My first thing would be authentication to Quay. If these are private, repos.

D

Or PCEHR, just looking for connection.

D

Yeah I've seen similar to this before. Usually it's an authentication issue, usually it might. It might also be a egress issue.

D

My pee thank you, doctor genitals people from doctor help, but not quite at IO.

D

Interesting I will keep this open a tab and see if I can use some digging. That's it are.

C

They behind a proxy right like that yeah.

D

That that's you know it's it. This is kind of a you know. Some networking thing is this: is an in cluster networking thing or not a cluster networking thing.

D

That is quite interesting, though maybe manually will I like to launch a docker engage docker image that I've created for kind of troubleshooting and tried, maybe pulling that interactively inside the container in in the cluster and and make sure that your networking is where it should be as well so but yeah I'll keep this open a tab here, sir, and free Linden, wool: look! Okay! So next question.

D

B

To be from Kenzi at 416, okay, you.

D

Know one of you guys can also read it as well. Yeah.

B

What are people doing and why to handle pod draining from their own management frameworks, I'm working on a solution for no draining to refresh with a new ami on Amazon, and it don't especially like that. The cube cuddle drain logic partly exists client-side, but maybe that's just my rational bias.

D

D

Drain is so when he says pod draining that throws me off because I mean you're, ideally draining the pods, but you're also draining I mean you're draining everything from the node right. If a notice leaving it's it's leaving, it can't do anything right. So right.

C

You could said that was it the Toleration on it too, and and so remove the node from the lease from scheduling well.

D

Alright, so I've actually edited cube controller manager to pass the node grace periods and pot eviction, timeouts and whatnot, so to make those those smaller they're. Actually you know five minutes is the PATA fiction, timeout, which I think is a little bit high. So I guess I'm kind of yeah the node draining I mean there's a lot of options here.

D

I would do a little bit of googling a little more research there's a couple great things that I found on this and I can link those here for setting those options and and but other than that there isn't actually as much documentation. This is something that myself, George and others can work on around draining and procedures for what actually happens when a node node dies. There should be a lot our doctor on that. So that's something that we're gonna work on as well. I'll note maintence tasks yeah exactly so.

D

Ok, good Jeff with the next question, all.

B

Right from see Riley at 416 regarding our Beck and 1 9x after implementing our Beck and no Toth I found I saw a lot of these and my cube API server logs and it's a bunch of errors saying cannot get resource nodes named IP like the AWS IP cluster wide cannot watch resource pods cluster wide I ended up resolving the issue by adding the this cluster roll binding as part of my bootstrapping process, and it's just a cluster roll binding.

B

This confused me because of this section from the node off Docs and he links to them are back node permissions and 1.6 the system. No cluster with role was automatically bound to the system nodes in 1.7 the automatic binding of the system nodes group to the system. Node role is deprecated because the node authorizer accomplishes the same purpose. 1.8 the binding will not be created at all when using our back the system. Node cluster role will continue to be created for compatibility with deployment methods that bind other users or groups to that role.

B

What am I doing wrong to make this to make me need this cluster role bindings. The docs imply that it shouldn't be required. That was a fairly long one yeah.

C

I've not seen this issue with at least cube ATM bootstrap.

D

Though I have either this is I'm you know, I I've just used medium myself um so, but but I have not seen this neat I'm interested to see what my cluster is like system notice system knows yeah. This is a custom postural binding for sure. With the subject of you know, system nodes being tied to a cluster, also yeah, that's very interesting that might be worthy of its own issue, actually in the kubernetes issues on github, so I might file that there and you definitely get some more devs on on that some more coverage so.

B

E

I've actually not seen this issue, it's I've used cops and goob ATM. Without seeing that specific error and I won't forget Christian's pearl question, George.

B

Yeah, the pearl question I think, is when my audio cut off and zoom muted me so I'll go over that one again, though I think you guys talked on it question I have a Perl script. That I can give a text file as an argument and get a response from STD out. I have a web service in a separate container that I'd like to call this Perl script from what is the best practices solution for this I believe Bob. You were talking on having a separate container.

B

That's in the pod that you can call through one two: seven: zero, zero one.

C

His question was not about Perl, it was about gke, upgrade experiments, low issue. Oh.

B

Well, where's, the other Perl one.

C

Someone else asked about pearl.

B

Okay, I see a Freeland, echo can see about drain the our back issue and then the next one that I saw was new puppy asking. If Secrets is kind of done, they.

C

Never heard the answer about the pro question.

B

My answer was: if it's an asynchronous type job or it's gonna, take you know more time than a web request is going to comfortably be. It should probably be a kubernetes job and if I recall Bob, you said that it should be a separate container in the pod that you can call.

C

It depends on how they want to do their stuff. If it's like can be accessing the text file or whatever could be a container within the same pod and like you hit, you know a local endpoint to trigger it or you do it as a career, a job or a fully other pod I think was the general conclusion.

B

Okay, I think I think around the next one. Okay.

C

B

Think I think we're caught up new puppy in reference to the previous secrets. Questions is secrets, kind of done or is there more work that will happen there? There's.

D

Gonna be more work, I, don't think it's ever going to be done. I think that like I was saying, I really think that there's going to be an interface type approach to secrets at some point, I could be wrong, but that seems like to be where it's going and there's always going to be improvements around the security and robustness of secrets, so I would say, expect more. I would say. Actually if you, if you know you, have the chance to google and find some resources, definitely post them.

D

Cuz I have not been been up in that realm for a while, so I'm sure, there's a lot going on. I, don't know what the sig is. I'm, not sure. Is there a sig security that would probably be awesome to jump into and ask around a little bit.

D

There's got to be a lot more coming.

B

All right, scroll, Scroll scroll, know that I'm catching up with people does any Roberto at 4:19. Does anybody use a GUI to manage kubernetes? We are currently using Rancher with AWS without kubernetes and looking to migrate over any advice, recommendations, one-man operation, 25 plus servers, I, can kind of speak to rancher.

B

Rancher 200 is actually just kubernetes under the hood. It is through and through kubernetes with the rancher UI. So that's coming, though, they're moving their release date forward a little bit. I, don't know when that will actually be ready for production. Oh.

C

We skipped the gke question.

B

No, we didn't Mario talked on that. uh If we're talking about gke, kubernetes, node pool upgrade very slow, yeah.

C

B

Mario said he was going to keep that open in a towel.

C

On my bed, I got.

B

You so, as for kubernetes, you eyes, like I, said rancher to o is going to just be a very pretty UI on top of kubernetes. It's gonna be its own thing on top of kubernetes, but I. Don't personally use a GUI to manage kubernetes. Okay, his question was different. Oh well, then where was it.

C

Node pull upgrade very slow.

D

Yeah I was gonna, say I, don't remember answering that specific type of question but I.

D

Do because it's another stack overflow, probably yep. Oh.

B

Yeah, okay, so Mario, you probably opened one eye open the other, and then we moved on I'm.

D

Sorry about that I have this open in a tab as well, because I need more tabs. So this is.

A

D

C

D

Is not something we'll be able to handle on the call here this might kind of looks like he's using helm so I like him already uh-huh, so definitely I will look into this and see if I can't get an answer and or figure out what's going on, there might be some tunings that need to happen inside, but so thanks Christian we'll come back to it.

D

Okay, can you read it and summarize the question so there's multiple paragraphs: if you hear it looks like it is, basically, it's I decided to try again, I'm c prouder once the notes are straying the stress of that note, changes already scheduling to say, which seems normal, but odds are remaining, so it's kind of a pods remaining during an upgrade I, believe type thing so and and I personally not carry out an upgrade in kind of a live cluster.

D

So this is something I might not know it's on about C router he's using us you're, okay yeah, so this is this- is gonna, be a little bit of a complex one, so we'll definitely get more details and come back on and I. Might you know if we yields a interesting result? We can come back in the next office hours and talk more about it. It may be something impacts, others so I Jeff, the next one yep so.

B

Then we hit a bunch of wonderful audio stuff before I move on. Does anyone else have anything to say on using a GUI to manage kubernetes I.

C

Think the only other really gooey thing out there is OpenShift.

C

D

Know and I was gonna say you know, I'm, not sure what were ranchers really gone lately. You know you guys are kind of still in it, but from whatever, whenever I started- and you know was using rancher, it is the most beautiful and the probably the most intuitive and authoritative UI.

D

There is for kind of handling deployments in general right for handling your workloads so and they asked cons of branch sure and and that's where I'm a little bit not 100% on I know they were using my skills as a back-end, which was kind of meh. I know that was clunky. I know, there's networking issues. There was a lot.

D

They were trying to do to have their own API and and and have you used that all that now most of the time, the people that are using kubernetes are not really using a UI and that's why if there hasn't been really any UI, that's popped up the dashboard, the crew, the official kubernetes dashboard has kind of been good enough for most people's general working with with kubernetes, so yeah.

B

And so I think we're moving on. Chris actually asked a question earlier on that we were waiting for a Ralph and Krista hop on the call, because they might have more insight. Are there any plans to have a base pod security policy in place at cluster build time, so that the admission controller can be enabled from the start.

E

I'm not seeing that as a thing we've had some some folks put that as part of their their bootstrapping process for their cluster itself through some random scripts, but I've not seen that, as there are certain parts with pod security policy. Depending on how you implemented that you get into some weird issues with mirror pods with.

E

So, basically your manifests, depending on how everything's hosted for the cluster itself.

B

Any other insights on that, because when I was I when I first read it, I was also going through github issues and I couldn't find anything specific on that. So it didn't look like it.

B

Thank you George. You know some days if there are any other questions in here. Let me pop over all right: George nod your head or shake your head. If there aren't any open questions in office hours do I then go to kubernetes users and scroll.

D

Yeah I would say: try to pick a couple, a lot of kubernetes users there's a lot coming in and some of them are read the tux. Please type questions so I didn't hear what you just said. What about just pick up pick one or two of them just the latest ones that seem kind of saying these things that aren't clearly found via the docs.

D

While Jeff is looking for for those I actually wanted to mention just a couple of things that I've been looking at recently, one is security. Best practices, Ian Lewis developer advocate at Google, has a slide deck. That I am posting right now in the channel. Around security best practices I have not gone through it. Yet it looks pretty comprehensive and interesting and gives a little bit some nice diagrams and whatnot of what's going on.

D

The other thing is that Jesse Frisell has a blog post, and this is from yesterday on building container images securely on kubernetes, and this is without using docker or need to bind the socket. You know she basically has provided a new I believe it's called image IMG and the utility for doing this really easily, because a lot of people are trying to attend CD and in there see they're building images, and it can be a little bit tedious if you're still kind of trying to do it.

D

The docker way so I will also link this post in the channel as well, and that's that's my two cents.

B

Alright in kubernetes users triangle Todd asked: is there a kubernetes accepted way of adding dns mask parameters to the cube dns deployment? It seems that if I modify the cube DNS deployment on gke, it gets overwritten. I can't find anything in documentation. Regarding this I know, there's a cube DNS config to or if modifying DNS mask parameters is even supported. I.

C

Don't believe it's supported, but cubed DNS is also switching over to Cordy and s here, real soon. In fact, you can enable it from the get-go at this point and I think there's a bit more options you can configure in there.

D

Yeah, basically Bob I, don't know if GK is doing something slightly different, think he used to be able to edited I. Remember everything at I I'm, not sure if it was a config map or I just edited the args in the deployment for it. But the other thing is that you may want to just start using core DNS, and that has almost unlimited options of a lot that you can do as well. If I find any linkage on it, I will post it here, I'm doing some quick searching so.

E

You can add the stub domain, it's actually in the kubernetes Doc's about, depending on what you need for DNS editing. There are some example files for those manifests.

C

B

All right, as I am scrounging to find more questions. Ralph is there? Are there any big cool features in 110 that you're excited for that? You think are noteworthy to come out, I'm shamelessly stealing this from earlier George. Thank you for.

E

Me, the one that I'm looking forward to most is CSI, going to beta being a little more accepted and seeing what comes of that as storage on-prem is a very difficult thing at this point in time, and it's something that should get addressed and I'm hoping that CSI does a lot of that. Saif has a CSI plugin already that I've been playing with a little bit.

B

Awesome anybody else, one-ten debug containers, all the way yeah, alright, hey I have a. This is Jonathan from kubernetes users earlier I have a fresh cluster and deployed in engine X ingress controller on it. I can see the node ports open up on the nodes, but a curl request to one of the specific ports fails. However, when I curl the cluster IP, it works any hints on how to debug this issue. I have a second cluster where this works. Just fine.

E

I've had that issue on my own personal cluster, so I'm pretty familiar with it. I would double check what your CNI is and verify that every node is properly joined to your CNI network um I, had it where I had calico bound to a different Ethernet interface, so the BGP status wasn't up for the entire node.

C

E

Other generated cluster, okay, the.

C

Only other time, I've seen a some more issue is uh when something else decides to take over managing of the firewall.

B

So this is another topical question: what OS is everyone running, or rather what is everyone's preferred note OS this I guess this is more of a bare metal question considering up into cloud it won't matter or.

C

D

Yeah I also kind of Len Kouros. A lot of people around me think that Ubuntu or CentOS is a good way to go just because it's common, they know it. They understand it personally. I think that the things that you know we're seeing with atomic that we're seeing with I, don't know where Rancher Alyssa's right now, but the the thin operating system with the bare minimum I need to effectively do you know, updates work with maybe some raw storage if I need to etc.

D

Like I think that model is very strong and it's only going to grow I think most people deploying kubernetes are deploying kubernetes and either treating their nodes like pets or cattle. The vast majority are probably doing pets, the overall idea and what I hope more people move to is ephemeral, hosts where they are just cattle and they can be easily, you know, removed and replaced and updated, etc. So this is tricky, though, with a lot of new storage solutions and other things that use.

D

For instance, reports you know, uses the the storage on notes right, and so it depends on those nodes having the drives them being healthy, etc. So it's interesting to see where this will go. I think that use your notes for compute have them. Do that one thing well and move on so.

B

This is another question from kubernetes users earlier this morning from Alex question, I'm curious how people are handling cube, system-related config, that is in llamó form things like Cordy and us calico cube dashboard for staging and production clusters. We were thinking of using git branches, but also might keep it simple and just use directories to start. What are your thoughts?

B

I'm? Also copying these questions over to office hours. So there's a little bit of context. Yes,.

D

Oh yeah great, this sounds like a you know: package management, you say, gaps, kind of a question right and so I've been a big proponent, I've spoken here about helm and how great it can be.

D

It does take a little bit of work depending on what you're trying to do, but the idea is that home provides you a really good model for easily handling your deployments on a cluster right and there's also case sonic coming up from hefty Ojo beta did a great talk on this at coop con in December, which helps you build these yeah milles by you know default templates that work really well for most people, and then you can interact with them to make your own changes, and what's nice with this?

D

Is you have these templates and you have a single file that defines the values that make up the configuration that goes in the templates, and the templates are literally just the service. The daemon set secrets config map things like that, and then you have a single values: file for each of your environments, maybe Deborah staging or other clusters that defines in very simple Y mo that the canal use for these things.

D

You know if it's nginx, it's how many replicas do I want if it's a ruby app it's how many instances to run or extra environment variables to pass, etc so helm case on it. There's a few others. I can't remember the queue pack I think is one of them. The helm summit just happened a month and a half ago. So do some some looking for some presentations from there and there's also helm three, which is in rapid development, I, believe and there's a design ideas doc that just went out as well.

D

So a lot of cool stuff happening in this space, but your life can definitely be made easier, stop just carrying the animals around, but you have the right idea in terms of using VCS for those, so I will definitely stop talking. Now, oh.

B

And Georgia said all the helm summit. Videos are in the kubernetes YouTube channel, so go check. Those out question from Christian Roy about custom metrics for HPA I assisted a recent presentation about using custom metrics for HPA, but the presenter told us that the only controller available right now is going to get metrics from Prometheus. Does anyone know if there are any others that would pull custom metrics from other sources like data dog.

C

E

Haven't seen any currently.

B

Hpa, for those is the horeb, a horizontal pod, auto skier yeah.

B

E

B

Another thing it's like: you only need to specify.

C

Unless you might be something for a data dog to you know, do in this case, I.

B

Would kind of agree just I guess it depends on workflow yeah? Oh thanks, George.

B

B

Do are there any other questions.

D

Alright time for one one or two more and maybe other quick discussion items too.

B

Yeah I'm kind of scrolling through users and the Stack Overflow and I'm not seeing anything. That would be easy to say.

D

Really quick, someone mentioned security and asked me kind of what I thought about Bowie and and Claire number one and I might be a fanboy, because George and other people on this call work there. But hep tio is creating some really cool utilities for helping you utilize your cluster and do the best you can to manage it from an operations side.

D

They've got arc which is doing it, which is we're basically using for backups, and this is another thing: is that QC ETL as an expert command which will strip out things like the self linked UID and other export isms that aren't needed for three importation of those configs? And so we actually just ran into an issue where an export of multiple members of the entire cluster yields or doesn't work for those those multiple files. They still have. The first one would be good with the rest of them.

D

Don't have those things cleaned out and so they're really not ready to be imported and so arc solves that problem and gives you a really great foundation for doing backups. The other thing is: is sana Blee, which I have actually not used, but in terms of conformance it really helps you get in this the right mind you know, state of a cluster is the way it should be.

D

It's going to operate well, I made you know, I've got the right decisions made and helps you kind of kind of make sure your end-to-end testing is where you want it to be, as it relates through Bernays objects as well. So definitely look into those now. A lot of people and I think we've got questions before around CD processes and how a security might fold into that.

D

I think that most people, whenever they see SED pipeline going, they should be using Claire or other tools like ink or or using a registry which is watching your images watching those those libraries and binaries and making sure that you know if you it's inheriting or the images is hearing from, are they safe? Are you know? Do we need to upgrade that and alerting us to that as well? So there's a lot going on this space, there's a lot of great work and so definitely definitely look around.

D

If you can and try to build security in from as early as you can it will, it will definitely help so I. Second, that entirely yep.

D

You said encore.

D

Angkor I'll just be pronounced anchor, but it's anchored with an e I believe I will link it here in a moment.

D

B

With that, if there are any other questions, get them in soon cuz going once go and we got some people typing I know. That's why I'm like trying the hold ah I've seen Lewis typing for a while Lewis did say here we decided to use git branches, but we use a scheme different from what you do.

B

A source code can't say it worked well yet because it's not battle tested so time will tell alright I think that's it for this month's office hours, thanks to the following companies for supporting the community with developer volunteers without our lovely panel of people, this isn't really possible. So thank you to Amazon bit Nami Giants forum hep. Do liquid web Northwestern Mutual packet net, pivotal Red Hat! We've works, University of Michigan and VMware soon we're hoping to hold raffles for audience, participation with shirts, kubernetes, spinners and all sorts of other cool stuff.

B

So it pays to ask questions and it pays to help answer questions so just promote in the community and lastly, please feel free to hang out in the hashtag office hours chat afterwards. If the other channels are too busy for you and you're looking for a nice place to just sit and ask questions or chat you're, more than welcome to come, hang out with us so, ah and it is every third Wednesday of the month. So next one will be johnny-on-the-spot with my calendar.

B

The 18th April 18th should be the next one. George. Are you nodding or shaking your head, or do you not nodding all right, good I said I said the good words all right with that I think this concludes marches kubernetes office hours. Thank you panel. Thank you, George for being silent. Take.

D

It thanks for planning all George George's the the mind behind this and audio issues are just you know. It's the demo gods that are with Kelsey they're, not with George. It's just how it goes so. George.

B

Is the mind now I'm, just the mouth.

D

C

A lot guys put.

D

Mr Pyne the scenes we definitely hang out on an office hours too, as well throughout the month. So what happened there so clean clips, more questions and we will use those in our key. Add those direct queue for the next office hours as well. So thanks everyone, let's have a great fantastic evening,.