Kubernetes Office Hours, 15 Nov 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes Office Hours (Western Edition) 20171115

Description

Join us the third Wednesday of every month for the livestream: http://bit.ly/kubernetes-office-hours

(Note: Some editing was necessary to remove the technical issues)

Here are the links from the show notes:

- https://buoyant.io/2016/11/04/a-service-mesh-for-kubernetes-part-iv-continuous-deployment-via-traffic-shifting/
- https://www.weave.works/blog/gitops-operations-by-pull-request
- http://blog.armory.io/continuous-delivery-to-clouds-kubernetes-with-spinnaker-and-helm/
- https://github.com/heptio/contour/blob/master/docs/deploy-aws-nlb.md
- https://github.com/containous/traefik
- https://github.com/heptio/contour
- https://blog.heptio.com/heptio-contour-under-the-hood-e62e1730e8a4
- https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/
- https://github.com/leblancd/kube-v6
- https://github.com/heptiolabs/kubernetes-aws-authenticator
- https://github.com/kubernetes/kubernetes/issues/1443
- https://github.com/kubernetes/helm
- https://github.com/ksonnet/ksonnet
- https://github.com/spiffe/spiffe
- https://kubernetes.io/docs/tasks/administer-cluster/highly-available-master/
- https://github.com/kubernetes-incubator/service-catalog

A

And mute is now off, so those of you in the channel we are live. Please feel free to. Let us know how the audio is sounding for you. This is the kubernetes office hours for the Western Addition for those of you in the on the West Coast time zone, I'm going to give you a quick intro, introduce our panels and then let you know how this works.

A

So the way this works is we hit. We sit in Hache office hours in in slack on the kubernetes slack and then just basically ask your question: prepend it with question and then I will read it to our esteemed panel of experts and then you'll hopefully get your answers. There are some cases where they might ask you a question we'll go ahead and give you time to respond and then loop around to you.

A

As far as the questions, if you look at the link on the top of the office iris channel, it kind of gives you some pro tips on how to ask good questions and then let's get started. So first, let's introduce our lovely panelists. Why don't we go Craig, Jo and then Mario I love that for introductions. You.

B

A

B

C

A

C

You'll ruin questions whether the stream.

A

Can give you there, it is. Everyone can be.

D

Awesome, yes, all.

A

D

A

So luckily, we've had the questions start to pile up, so we can just get started. Then what we'll loop back to during one of the dead times Jo for your opinions on ingress, sound good, it asks question: is it possible to isolate a pod in a cluster for specific testing /flags? Let's say I want to get some extra metrics around the health performance of the pod on the side in production.

B

So is so I got asked a question is this: the idea is that you want to run say like a deployment or a replica set, and you want to have one of the pods. Be special I mean one of the pods in that replicas that'd be special, because you want to essentially maybe quarantine it or debug. It. Is that the idea, or are you thinking, different flags in terms of the dr. Damon, underneath the covers that this thing is talking about?

B

If it's the first one Shawn, so you and then Shawn later you say it sounds like it's the first one part in a replica, okay, yeah. So so what I would actually say is you can get fancy here, and this is one of the reasons why we separated out the idea of deployments and replicas from services.

B

What you can do is is using labels is that you can run a separate deployment for that special pod and you can have it actually have a subset or it can have a a set of labels that expands beyond the set of labels that the services target. So you can have a service target, multiple deployments or multiple replica sets, and so you can have like a canary or a debug replica set and and then you can have your regular or deployment.

B

And then you can have your regular deployment, which has type equals normal or something like that in a label. And then you just don't include the type label when you're actually defining the service, so that service covers both of those deployments.

B

Another interesting technique that you can do here is like, let's say that you're running a service with a with a deployment, slash replica set and your start getting 500s and you track it down and those five hundreds are only one of the pods that you're running and you're like so now like you're in this situation and we've all been in this situation, where it's like, okay, I, can kill and restart that thing and I'm pretty sure that I'll come back and it'll be healthy, but I'm going to lose all the context.

B

So it's gonna be really hard for me to what's going on and figure out, what's happening so yeah. So a lot of times in production, you're you're in this really awful choice where you either throw away state that you want to debug or you know, to get the thing up and running or you sit there and you debug it live and meanwhile, like maybe a tenth of your requests are coming back as five minutes.

B

One of the things again by playing games with labels you can actually play with here is you can remove labels from that pod to essentially quarantine it and, as you remove the labels from that pod, two things will happen. Number one is: it will no longer be targeted by the service and then number two, your replica set, you know, or you know, slash deployment- will actually spin up a replacement for that things. You can essentially orphan our foreign team.

B

This thing now you have something that you can start poking at directly and in the rest of the the deployment actually continues like brings itself back up to full strength and you're ready to go. The one thing to be careful of is like make sure you delete that pod because nobody else is going to.

B

But that's you know if you dynamically edit labels, you can essentially sort of move stuff around and change. What's cap anywhere does that? uh Does that help answer that question for you Sean.

A

And he's typing awesome answer: huh oh.

D

A

D

Or a rule, don't either of.

A

You have opinions on this I.

D

Just want to mention that yeah, the effort effort, I, can't talk today, the finality of pods, like the idea that you know you don't really want house to be special and I'm like Joe was saying you know it's going to stick around or you're if you're looking at the logs and whatnot of the PATA in this image, ops right if a job runs, I believe the pond actually sticks around so that you can look at the contents.

D

It doesn't go away and we actually just I think we were wondering how to get a little clean up there and whatnot. That might be another question, or it's so matter of reading your own, my cursors, and all that but yeah, just being careful the things that are being left around and that goes into your processes at your company or whatever that you're trying to achieve. So that's all I had to add Joe. If there's any.

C

D

Places for documentation for so this, like ahead of that but yeah that'd, be nice to know as well. There's.

C

Another aspect that I will add to this. You know I, think the question will also said: if I can narrow down a part to an old, you know, I think that will be that much harder. Yes, you can narrow down to a part, but you know if the power gets terminated for some reason. It's the rope part different host, so can't really nail it down to one node by itself by using the technique that Joe was talking about assigning labels.

C

Yes, you can to the path, but having that food and pod and node Association will be that much harder to track.

B

Yeah I think when it would add all possible you don't want to use use. You know scheduler features that lock you down to a specific power cific in specific host. That's that's that's to be avoided. If you can help it yeah.

A

And Arun you introduce yourself real quick, since we had trouble here. I, certainly.

C

Can yeah? My name is Arun Gupta I work for Amazon Web, Services I'm, one of the principal open source technologies. My focus here is exclusively on open source and continuous frameworks and I'm. Also the board rep for CNCs. Oh great.

A

Awesome thanks thanks for joining next question, is, and also thanks to Ed there for digging up the legal bill, labels, documentation and tossing that in slack. We certainly appreciate when people do that I also.

B

Wrote the chapter on the kubernetes up and running book on labels and annotations, which.

A

You can pick up.

D

A

B

D

Also decide the animal on the front cover.

B

Pretty sure it's a it's an orchid killer whale, but if you actually look in the book it says it. So it's type of dolphin.

A

Yeah, it says this: a bottlenose I thought it was a false killer, whale first being a cetacean fan.

C

Already does not let you pick the animal, they pick the animal they have their own design. hmm If.

B

It were in my book, it would have been, it would have been the honey badger.

A

So, like 77 writes in thanks for joining I, have a question regarding how to wire and manage resources like RDS, elastic cache, etc. Provision through terraform into kubernetes. Is there a good pattern? I could use here, I.

B

Can take that if, unless other folks want in.

B

So so, there's two things two things here, so this is in some ways the the idea behind the service broker effort. That's been ongoing for a while right service broker is a way to say, give me a service that does X. Let me find it and then give me the credentials so that I can talk to it.

B

Now, that is, that is still early work, it's being driven as a partnership between kubernetes group, there's a sig, that's draw and and there's a common API between that and the Cloud Foundry Foundation, and so we're trying to find some commonality within kubernetes and Cloud Foundry there. It's still very early and I think you know, like you were saying towards here. You tie them in and and but you know, you were saying that that you're you're bringing these things up with terraform, and so there's really two questions here.

B

In terms of of you know, how do you hook that up? The first is: how do you find it and then the second question is: how do you authenticate to it when it comes to? How do you find it you can you know if it's wired into the DNS that the rest of your cluster is using? You can go ahead and actually talk to that directly.

B

If you want to use sort of native kubernetes stuff to be able to find it, you can create these things called external services, and you can address those either by name or by editing the endpoints object directly. You could populate it with the IP addresses that you can use to find this thing, and so that's a way to essentially import external services into the sort of services model that that's already there in kubernetes. This is not super well documented.

B

This is something that's kind of been on my list that I've wanted to actually go in and talk about more formally in terms of how do you actually import external services when it comes to? How do you authenticate to these things, I think, there's a set of choices if you're running on Amazon in AWS there's, you know.

B

Obviously, it's well integrated if you're calling from a VM into something like dynamo, DB or RDS, or what have you if, if you're running on kubernetes there's a project called cube to AI m, which essentially lets you assign I am roles to kubernetes pods, it's a good place to start it is.

B

It is a from a security point of view. It's it's not as good as it could be, but if you're running a relatively sort of single purpose cluster, it's probably pretty acceptable if you're running a large multi tenant clusters with with you know a bunch of different people, and you want to protect them and like really think about how you know what happens if nodes get compromised by somebody else that then it's worthwhile to look a little bit deeper into that.

B

The other option is that you can store credentials either in kubernetes secrets or you can store them into something like like hashey core vault, which is very common if you're storing them involved. Then it's a question of like well how the heck do you authenticate devolves? There's a kubernetes Authenticator for vault that uses a service account tokens to be able to do it.

B

But honestly, at that point it's not that much better than storing them as secrets themselves and the pattern for how you authenticate devold involves giving them credentials that let vault access as you towards the kubernetes api. So it sort of extends your sort of cordon of security to include both also, which is probably not insane, but it's not a great pattern, so yeah so I. Hopefully that gets to some of the techniques here. I, don't think, there's a great guy in terms of like the different ways of importing external services and I.

B

Think that there's a real need there.

A

Anyone else have opinions on using yeah.

C

My only take is that open ship broker, sorry other open service broker is the way to go, essentially look at it on how that integration can be done. Particularly, you know, if you have already resources in Amazon, so look at you know, I mean that work again, as joe said, is in very early stages. So, as the time moves on, you should be able to see that how think about your kubernetes cluster running on Amazon now having a single set of credentials?

C

One thing I also want to talk about, is kubernetes AWS authenticated, which is a hefty olabs project as well. So, essentially, what you are looking at is n2n. You know. Security using I am credentials, particularly in the AWS land. So you want to use the same. I am role possibly to authenticate to the kubernetes cluster using cube kernel. Then you can have a similar role. You know doing your pod identity on the cluster and then eventually having enough permissions in that I am role to be able to authenticate to other services as well.

C

I think that's something that you want to keep track of.

B

Yes, so the the project that I run mentioned is something that more one of the security engineers here hefty over out and it's something that we're going to be putting a little bit a little bit more shine onto more folks to polish it up a little bit, but it essentially lets you use I, am to authenticate to your cluster and then cubed I am lets. You use your cholesterol to authenticate to AWS, so.

C

Do my question would be how does spiffy spire kind of fit into this scenario, particularly when you are using external services like RDS yeah,.

B

I think that's a great question, so the first thing is ok. So for those who aren't aware there's this project that was just submitted for consideration to be a CNCs project called spiffy I started this before I started hefty. Oh there's a company run by a friend of mine, that's that's developing the product called site ale, it's called spiffy SPI FFP and you go to spiffed out I/o. If you want to read details about that, basically, what it is is and it's inspired by the way that Google handles identity and their production systems.

B

It's a way to to assign identity in the form of currently x.509 certificates that, let you you know essentially have a job come up on. Kubernetes come up on mezzos come up on swarm, come up on raw VMs and and map that to some identity that can then be used to actually do micro service to micro service at authentication. So it solves the problem of like when service a call service be. How does B know that it's actually a that's calling? So that's the goal of what we want to get out of that.

B

A couple of things that, with this conversation, was spiffed, if you keep in mind one way to look at a secret store like vault, is it's really an identity, translation service right? Most of the time? What you do is you talk to vault? And you say: here's some identity translate that into another identity.

B

For me right because, most of the time, the secrets that you're getting out of it, you can then use to further further authenticate to something else and and the reason you need an identity, identity, translation services that we don't have a good common way of identity.

B

Each system has their own individual idea of identity and where we do have a common idea of identity like if you take a look at AWS or Google, or whatever it's very diff, to actually use that identity between two peers without having that thing actually sit in the middle, and so this is something that honestly is very difficult to do with with IIM Tay I think Microsoft's, probably a little bit ahead because they have Active Directory to lean on which is sort of like you know, done this stuff forever Smith.

B

He has sort of meant to be an interchange format to be able to do that, so you won't be able to use the spiffy IDs to talk to AWS directly unless AWS wants to actually start accepting them intent. Not you know your room, but what you could do is you could run some identity, translator like say vault. You could then use the you know in the fullness of time. You could then use smithy to talk to vault.

B

Vault knows how to actually mint tokens using the STS service with with AWS and then return those to you. So so you start viewing your secret store as a as an identity, translation, service, I.

C

Think that gives a really good context. You know, because end of the day you are looking at, you know, I mean yes, the question was probably more targeted towards RDS, but essentially you could apply that concept of vault as an identity, translator with spiffy, particularly and how you can do cross cloud. For example, if you, if say, let's say the clusters are running across multiple clouds. Now you can use that as identity translator and.

B

In the fullness of time, I would love to see the clouds actually root, things like spiffy in sort of special access and special permissions and hardware that they can run right. So so, in some points you know that the the spiffy identity ends up, get it routed on something that is, that is, you know, baked into a chip somewhere in hardware and that there's a really good chain of custody or attestation of the stack right. That's that's kind of the world that we want to build towards identity is hard. It's really hard.

A

So moving on cuz we got the questions are starting to pile up. Big will yes, I'm trying to understand fibre channel in kubernetes I've, read the PV and PVC dots on the Kate's website, but I'm still lost? Does anybody have any good tutorials or any anything to learn from.

D

So I'd be interested in hearing a little bit more about what um what type of product or service he's trying to use by my channel in and of itself is just a networking technology I'm guessing he's trying to achieve storage, so some sort of sand back end, so I'd be interested in hearing a little bit more about that. What he's trying to do? Yep.

A

Willie, if you could go ahead and just give us a little bit more information, he says he's using a pure array.

B

A

Do we do we? Have anybody with expertise on that or I? Can I could definitely write that down and then maybe ask some of those six storage. If we can loop back on that, one yeah.

B

I think this is this is actually one of those interesting things where I think, because kubernetes is so, you know it was kind of born in the cloud to some degree. It's been very difficult for us to test and validate and sort of get wide usage of things that integrate with particular vendor products, especially when those things are on Prem I. Think you know another example of where we've had a hard time.

B

Getting getting progress is around say, ipv6, there's some early work around ipv6 being directed by some folks at Cisco, but the truth of the matter is, is that we don't have great cloud solutions for testing out. Ipv6 I was so excited when AWS announced their support, but it's just not quite there. They don't actually get a range of IP addresses that you can then assign to pods. It's really something.

B

That is that it is sort of a onesie, twosie type of thing and so anytime, where we can't do continuous integration testing and have it sort of be a democratized technology that a lot of people can get their hands on. It's been very difficult to actually get get a critical mass of support around those things. Yeah packaged stuff is interesting, I think it's worthwhile.

B

The other thing that people are doing around ipv6 is running is running nested virtualization on both ashore and and an GCE right now, all right, so I just got a warning saying the so our meeting will end in 10 minutes, because we couldn't figure out how to act. Yes,.

A

B

When that hat, when it kicks us off, everyone.

A

On the livestream just chill up for seconds or so and then we'll rejoin and continue so I says.

C

I, don't really.

A

Need native integration I would just like to understand more generally using fiber channeling kubernetes, maybe at a high level completely. The hardware looks nice I'm, looking at one now, I like nvme and stuff.

A

So if I I'll go ahead, Willie I'll follow up with you after the meeting and I'll ask someone in six storage. If we can maybe find find a way to get. You started here, yeah.

D

At linked a github readme in the humor neighs examples- oh, this is the probably the closest you're gonna find about. It looks like it's more so just some notes that someone candidates here I'm doing this, so it seems like this might actually be possible, but yeah someone from sleep storage would definitely have more of an idea, and the question to is you know: is this something that you'd want actually do a production? For you know, systems that are burning, live workloads, so yeah.

A

Yeah, so okay, moving on earu asks. Is there a plan in place for moving host mount propagation forward? It's a feature, we're extremely eager to use it's blocking our main usage of kubernetes, actually, there's a note in the source code. This field is alpha in one point II and can be reworked or removed in a future, and he plans on removing it I hope not or formalizing. It I hope so. I.

A

Am totally unfamiliar with this I have.

E

B

I'm looking it up now, let's say you hear me.

B

B

One thing to keep in mind is that, like when things are marked alpha by by kubernetes the alpha beta, and it's on my list to write something up here, to make it more clear. The difference we now have a beta and GA is relatively, is relatively conservative in kubernetes, right and so I think you know. The whole idea of backward compatibility and signing up for something for the long term is something that kubernetes takes very serious.

B

So just having something be in just having something actually be in which we call it in in alpha doesn't mean that that thing is, is, is necessarily unstable or is going to be, but I think it's just a matter of like we don't want for the long term yep. What's.

A

Say, hey wouldn't said would handle that like it sounds like something like ass, it would want to know if someone's really looking for a feature, I.

B

I would check to see it's not from.

A

1-9 exchanger.

B

If anybody's talking about it with with planning okay.

A

C

Of the things I would say about also features, though, is you know like once you get into beta, then you're in a pretty solid shape, but unlikely well, not likely to break backwards, compatibility and that's the thing with alpha features. You know, for example, we have stateful sets now now the schema has evolved from alpha to beta and the API is much more solid, so the names could evolve.

C

The api's could change and that's sort of the idea of behind alpha, essentially that it's a warning that okay, you know what don't use it in production today. If things change be ready to evolve, essentially mmm-hmm.

A

So I have two things to ask: six storage. Then next time I hang out with them, so they will. Then you just follow up with me afterwards. I'll make sure I point you to at least the right people. So you have a place to go.

A

Moving on next question, one of my favorites any good link, slash docs on doing manual ha4 q Batman. While we wait for the feature to be completed in queue, Badman itself.

E

George I did answer that pretty much in, but this is something that we are working on. Ftl and I want to deliver. All one of those deliverables will be a blog post or some documentation around this, but it's enormous okay.

A

Or how are people do it like? Do you do it by hand.

C

Well, we have a cloud formation, so we are writing the brand-new workshop, which is kubernetes on AWS workshop and as part of that, we do have a CloudFormation template that CloudFormation actually uses cube atom and through that we tell you exactly. How do you install a cluster but until that feature for etch a is available in cuba in itself? You know we just use a CloudFormation to do a single master cluster. At this point, you.

E

Know and we have the abuse QuickStart, which is the same.

E

It also is one master notes. The question in particular, though I think, was around prayer.

C

And I think the point that you made about you know that you're working on a block tools to do cube, atom, I think one of the key questions to ask over there is. Is that a blog or is there a functionality? That's going to be integrated into cube atom to enable HHA yeah.

E

So this is also posted in text. Is this work is currently ongoing in sequester lifecycle to Nha capabilities to cube area is just not quite there yet.

A

Yep and for those of you who haven't tried it in awhile cube admin now supports in-place upgrades, so I just did a blog post on that the other day, you're able to move up point releases or you can move up major versions in place so literally I think H a is like the one last thing people are really waiting on and then it will be cube. Admin happiness for everybody forever have I miss any questions audience. If I have. Please repeat it. If you could- or we have new questions, please feel free.

A

We have about 20-ish minutes, left SR mat, so ass hi. What are some recommended tools for managing the lifecycle of kubernetes, primitives deployment, services, etc? I'm, currently, writing a Python app, which is backed by database, which should meet our needs just curious. What others out there are using. So this is a super deep topic.

B

So I think I think so we have a working group where we're talking about application definition. How do people actually manage applications defining what their applications look like and then there's the question of? How do they then go ahead and actually push those to the cluster? So there's the application definition and.

C

B

The application deployment I think helm is a great example of sort of integrated experience. You can bring a round bunch of this stuff, but I think what we're finding is that there's a whole host of different needs in terms of what people need here. I think the idea that you want to move the source of truth of what should be running in your cluster out of the cluster into something that is is more purpose-built for that and I think you know for many people this will be get right.

B

Well, they'll actually have a bunch of files checked in to get whether it's raw gamal or whether it's something that is you know essentially compiles down in the ammo, and so it helped do. We have a thing that we're working on called case on it, but there's tons of ways just getting that cap I think the idea of using a database to drive that is really interesting itself right and I think there.

B

There are a lot of cases where you know you have a database that stores stuff, and then you have a controller that then uses that to actually drive what should be running so like we could create a theoretical, minecraft server, which uses a database to say which minecraft server should I be running and where the data stored and automatically starts and stop those based on people clicking stuff in a GUI and writing it into a database right.

B

So in that way, you're starting to build an application that is really bound into Google and you're I mean found into kubernetes and you're you're blurring the lines between defining your application, having your application, define itself I. Think that gets to some of that question. There, hopefully four case on it, I think specifically we're going to be doing a lot of stuff new in fork, Yukon and so expect a big update. There, don't don't judge.

C

B

What's there now we got a lot of exciting stuff coming.

A

Anybody in the audience feel free to you know, toss out links of tools that you've use if you've and that you prefer.

B

A

Else have opinions on we're.

B

About to be cut off, George I, upgraded, FTO zoom account to the pro version. I thought that might fix it. Uh-Huh well, I, don't.

A

It hasn't, kicked us off yet so I have the URL ready, so it shouldn't be too much. Okay, too much of an interruption. Georgia.

C

Free kick out, then we just wait here and it automatically connects us I. Don't.

A

Know I have an experience, that's before yeah I believe so. But I have this your zoom URL standing by if okay.

C

D

A

Need a zoom engineer.

D

In this chair, I think just with us all the time just yeah.

B

Well, it's so funny. Cuz products like zoom are really built for a single company. You start using this stuff, you know in a sort of like very disconnected loosely coupled world and and and you start seeing where all the or all the the creaky edges are yeah, but.

C

I, actually, a couple of years back, you know: blue jeans used to be a big thing, but I don't see much of blue jeans now I'm, seeing more of zoo, so I've definitely seen more popularity here. Yeah.

B

We just have to make.

C

B

What are we gonna switch everything to chime around.

A

All right stand by.

A

mmm Invalid meeting I do know one second, please bear with us here.

A

A

Let me just get everybody else in here and we will be back.

E

Alright, alright.

B

I'm back sorry, everybody, oh now it's not limited, so it looks like my upgrade fixed it for you. Oh did.

A

You upgrade my account. Thank you. Yeah.

B

I upgraded you to the pro version, but but apparently we had to restart the meeting to actually get that to kick in fine, we and I'm something about this stuff. Now we can yeah.

A

Now let me see just waiting on the other two and Mario.

E

A

And I've sent a link to a Runa, but he was dialed in wasn't he yeah.

D

A

D

On the home, yeah.

A

So, let's, let's see how that works, people watching the stream how's the audio. Hopefully everyone could hear everybody I'm.

C

A

About that interruption we have a glitch alright.

B

So ed was asking about the ipv6 issues, I think it's being driven by sick networking. I can try and find.

A

Upgraded their cheery switches. Definitely ok,.

B

The route I got feedback on chiman I can tell you.

B

Let's see like thinik 82 your cluster with I am, I can send you that link. Give me a second. I can look that up.

A

Yeah so let's, let's see with ipv6 and then I'll find the link for the yeah.

B

So a couple of caveats and still very early, most of the prototypes- are being driven using docker and docker kubernetes cluster, which is this thing so that some ranches folks put together for essentially doing development around river Nettie's, the real caveat there- and this is really unfortunate, and this is probably to some degree.

B

My fault is that the original way that we we set up networking in kubernetes is that we assume that each pod has one and only one IP address, which means that doing dual stack is not something that is easy to plumb in for kubernetes right now, if I dual stack, I mean giving a pod, both an ipv4 and a v6 address, and so the work that's been going on so far is actually single stack.

B

Giving in a single ipv6 address versus a dual stack type of thing and then doing a bunch of translations to the outside world, with four to six and six to four and stuff like that, so again, still very, very, very early going on there. But I can try and dig the somebody just posted a link with an example someplace earlier so and.

A

What I do at the end is I put all the links in the show notes in the actual YouTube description, so that will definitely be available for those of you that are watching the non live version of this video. Let's see, oh here's.

B

The issue that covers it there's a lot going on there. Okay, so.

A

We get everybody back: okay, cool, yep, good okay, so that answer is ipv6. Although let me just make sure I didn't miss a question and then we have the link to kubernetes AWS Authenticator, so no open questions yet so please, please feel free to continue to ask. We have about 15 minutes left, so it might be useful to talk a.

C

Little bit more about the sort of where is heading to Joe, you might be able to get more context of that yeah.

B

So this is a hefty olabs proof-of-concept thing: it's it's something that more put together relatively quickly just to see how would work what it essentially does is. Is it it lets? You create a token out of a sign out of essentially a URL to the STS service that then gets executed by a component to be able to verify you, and so essentially what it does is is have you proved to the kubernetes cluster I.

B

You know, I have access to this role right, I can act as this role or I can act as this user and and then that essentially becomes a token that gets passed along through cube control. Now, the way that this gets plugged in is a little bit funky right now, in terms of you, have to essentially go through and hook it up to the kubernetes cluster directly and then, whenever you call it, you have to essentially execute command to get a token, you use that token as a parameter into into q control.

B

There's work lighting up in a couple of different places, with six CLI to essentially make this this authentication stuff the built in as a plugin for cube control. So we want to definitely streamline that over time and it's something that we're going to be probably moving out of labs and and putting a little bit more effort in time. And just because it seems like there's a there's. A lot of interest interest around. Oh yeah,.

C

And I think one thing that I like about the Authenticator is I: don't have to really get using cops. For example, now and now I want to manage that cluster from a different machine for cops cluster is created. I get all my cube. Config on that and I can manage the cluster from there, but otherwise you know I'll have to can I have the kubernetes cluster configuration stored separately. I am credentials, that's a fork. You I am stored separately. I have two disconnected sort of set of credentials that I need to manage with Authenticator.

C

Essentially you know you can have one set of credentials. You know talking all in the IAM language and continue with that and.

B

We're talking about ways to map between I am roles or credentials; essentially the user you bar a are ends or the role a are ends in to kubernetes users in a more generic way, so that you don't have to because like if you read the thing right now, you have to put an entry in this file for essentially every user that you want to map in from the from the IAM world to the true or daddy's world. We want to do some work to sort of make that be more.

B

You know a crap, this part of the AR n and promote that to being the kubernetes name in that way. In this way, so that's.

E

One of the things that.

B

We're actually that we're actually looking at but but I think you know more, did a great job of documenting this. So so definitely take a look. If you have feedback on it, you know, let us know feel free to file some issues there and and it's it's something that we're going to be putting a little bit more effort into.

A

Just see if they we have any more questions lined up here, see.

B

Pointers to ipv6 stuff, qp6, yeah, yeah I'm excited about like I, do want to make progress on v6 and so I think I think. Hopefully we will we'll start cracking that egg open yeah.

A

Our mando CERN I, hope I'm pronouncing that right says we're experiencing issues using the default nginx ingress controller, around delay of nginx configuration causing 503. You kind of hinted at this before I think as far as like, when you're using a label and if you're, making a configuration change right, you have the one like a one in ten chance or whatever it is. If the user getting yeah, there's.

B

A delay before the the nginx configure eight yeah.

B

Typically I mean I, don't know too much about the nginx controller itself, but this is you know, fundamentally, you're building distributed systems and a lot of cases distributed systems. Things will get out of sync. You will have like you know, there's it's just not possible to make sort of a a clean boom. We're gonna make a change here.

B

There's a couple of things that people use to actually deal with this number one is that they they do a drain when they're removing something from a service, and so, if you're going to do something like change a label or shutdown a pod. This is why we have essentially some of these, these hooks. These lifecycle hooks and the pods. You can instrument your code, such that it has. It starts answering with an error to the to the to the.

B

What is it the the the readiness probe right and so what you can do is you can take yourself unready, wait for all the current requests to actually drain. At that point like it, you know you can wait until the configuration is actually good in.

C

B

Nginx controller the ingress controller has caught up, then you go ahead and you shut that pod down and you're actually good to go. So if you can do a great full shutdown, as you actually remove stuff you're not going to be seeing 503 s like that and so yeah. So there you go that sort of that's. How do you drain nodes but I?

B

Think it's a similar type of thing of like when you want us to turn down pods, make sure that you do it in a in a responsible way, I think a lot of times when you're doing distributed systems and you're and you're managing this stuff a lot of times you got to be patient. These things happen slowly.

B

So right, it's like a big part of this is is: is okay I'm going to drain the stuff I'm gonna leave it for a few minutes to make sure that everything is settled and then I'll actually start rotating some stuff out as developers. We want to hit the button and see it happen right away, but you know in some ways: that's the you know. That's the bad instinct right like the best Sarris when you see them operate, they're very so, they're very deliberate.

B

Even if the world's falling apart they're very careful about what they're doing and plan it out, it's a certain skill to be able to keep that calmness and actually because slow as you're actually doing stuff mm-hmm.

A

And he's currently typing. So, let's see if that answers his questions, I hope and thanks Craig for asking some follow-up.

A

And now we can physically see a room. So that's good. Are you doing the Movember thing.

C

Maru I am actually you know it's mid mid of the month, and also a tree in wind is gonna, be large answer and bushier. Yes,.

C

So maybe you know I'm thinking I should have thought about this earlier. Maybe next year now the kubernetes community. We should start a Movember movement. You know, let's see how many in companies community can do. Movember I was.

B

Pretty pushy for a while, but my wife is like that's: it shave it off.

A

He's a high density staging environment that causes a lot of problems with it makes sense that the drain is necessary. I assumed it had more to do with how nginx was reloading the config on ingress changes, yeah.

E

That's what I was going with that question: Georg nginx Wow, a really awesome technology has powered a lot of the internet for a long time. It does required the reload, and that can sometimes be is sometimes have a bit of an effect on other streams flowing through the nginx controllers. Yeah.

B

E

Think that's parlor easin. Why we're kind of interested in ongoing Commons work we're doing around Commodore I.

D

I actually wanted to ask about that stuff too, and kind of get an update from you guys on all that, because you know a lot of companies look at web included is looking for, like nginx is great, but it's just not for distribution support rying to do here so yeah.

B

So I mean that's why we invested in in in contour, it's still very early, we're working hard on it. The biggest thing that that we're looking at the two things that are that are sort of next on the docket for contour, which are going to you, know really sort of move it to. The next level is going to be S&I support which is coming in envoy, which lets it terminate a TLS, and that's really a bare minimum feature and that's going to be available through the new API.

B

That envoy is using, which is GRDC BAE's for sister HTTP thing. Now that T RPC, a P I is probably not user-facing, but it's again moving towards envoy being very dynamic, very efficient in terms of how you can actually dynamically reconfigure it, which is it's one.

C

Of those things that honestly.

B

Nginx at least the free version of nginx struggles.

A

With yeah and Dave Cheney just actually published a blog post, which I linked in the channel, though, on.

C

How he explains.

A

How doing it with our boys, new API will get rid of polling like entirely it'll just.

B

Soak on Tory envoy versus traffic, so traffic is great. I honestly, don't have that many miles with it. The impression that I get is that that, especially with likely tube of pages and stuff like that traffic is, is more of a sort of single service sort of like more of a you know, be able to feel it touch a type of thing versus something that you, you would necessarily be boy at scale. That being said, that's my initial impression. I, don't necessarily have super deep knowledge there, so I don't.

D

We right look, I was added back and we've actually tried to use it and it actually works really well, but it's very single instance oriented it's not meant for running in a clustered kind of method, so things like SSL Certificates, which it actually can handle for you, which is fantastic. You can have eight instances running, but all it is is we're gonna try to get a certificate for the same booking right. So there's things like that.

D

That haven't been considered in terms of its reliability and ease of use, it's great for a starter project or kind of a single single app on the single server or something like that.

B

So, let's see so lax, 77 I'm asking about Helmut spinnaker I, don't want to lose that one, but on the on the on the BMC graph, I was asking about running it as a Gaiman set or running on a small number of nodes. Or how do you want to run?

B

You know I think if, if you're talking about a production cluster, it may make sense to actually have an isolated set of ingress nodes that you're actually using to run your ingress and having those be somewhat isolated from from the recipe you can still do, that with daemon sets and taints of toleration and that's up but manage it.

B

If you're running the small cluster with ten nodes, you can probably get away with just running one on every node, but at some point it kind of feels a little bit silly, the depending on sort of how, where you're running in what your production looks like you may want to go through some effort to actually streamline the network path coming game dressed. It's very easy, with with a type equals load, balancer type of thing, pointing to to to your ingress to actually get something up and running. That won't necessarily be the most efficient networking path.

B

If you do a little bit more sort of custom type of stuff, like I, have an example in the contour repo about deploying contour with host networking and the new network load balancer stuff on AWS. It's a very direct path, and it ends up being you know, depending on your environment, you may want to do run a couple clusters, something a little bit more specialized to really get that path. Nailed down.

D

And a lot of people here do like we cannot do this, where we have a actual load, balancing layer that takes care of layer four in and then then that actually handles getting back to the cluster, and so we actually do run it as a payments that our intention is every node is running that and then that layer for load balancing layer, if you will actually monitors the nodes and their health and knows when they're healthy enough to to route traffic to so we kind of separate that out where we still do running it and goes control all of them.

D

We don't do any specific nodes, but you might have no therapy here. Have better network connections that you want to? You know, use some labels for and have your ingress controller on. That's usually puffles long introduction. So.

A

And somebody's asking do we archive these videos? We certainly do they're available on the YouTube channel. So please share with share with friends.

C

A

Definitely so we're running out of time, so I do want to answer one one more question here from Lac 77 about.

A

I just missed it. How is Helmand spinnaker a good choice for CD? You could probably go into CI CD combos all day long, but let's give him the the TLDR. If we would so.

B

I'll give you the quick thing is that I think using helm for deploying your own application is? Is it's not the place where I think it really sings? I think that that helm really works best when you're deploying something that's been prepackaged for you and you're grabbing a helmet chart authoring your own helmet is not always a quick period. It's something that that I know. You know some folks are looking at improving and even if you're going to be using it as a CD, you'd still need something to trigger it.

B

Now, spinnaker really is a whole CD thing and I, don't know if you can use vinegar with helm a little bit like there's a lot of overlap between those two things thing to keep in mind about spinnaker. Is that it's a pretty big, complex system in and of itself it has its own user's database and its own UI, and it's like it's.

B

It's a pretty hefty piece of technology to deploy for solving these problems and in last, I looked all of the I'm sure you can do fancier things, but but spinnaker is very opinionated about doing Bluegreen deploys in a way where you're not going to be using, say kubernetes deployments, you'd be using something like replica sets directly and have spinnaker be the one that that actually drives those things. I mean a lot of. This is just based on the assumptions around how AWS works, which was baked into spinnaker early on now.

B

That may be a snapshot that was a while ago. I know, spinnaker has been evolving past that, but that's that's at least been my snapshot. The last time I looked at it so.

C

One of the things I would say is you know we see customers using Jenkins, which is definitely in their wheelhouse. You know, and Jenkins again can be set up multiple ways you might have an existing Jenkins install and you can use that to do CI CD I have a sample in the workshop that should be coming out soon. In addition, if your AWS customer, then we of course have a reference architecture on code commit code pipeline.

C

So you can literally say you know what I push my code: entire prod and entire sequencing is there as far as the reference architecture, and what I can do is, after this, once I'm done with the chat here, I'll drop a link in the slack channel which can actually go to the show notes as well.

A

Good to know, and with that we're kind of winding down here, we're a few minutes over to account for the technical issues I.

B

Do want to I do want to point out. One other thing is that Alexis that we've works has been, has been pushing this term I'm, not sure if I like it or not called get-ups.

B

But this is the idea that you check in your stuff to get in, and you have something that automatically knows how to actually read from it and actually use that to drive the deploy. It feels a lot more lighter weight than than something like like spinnaker in my mind, and so I'll put a post there.

B

He has a couple of blog posts and they have a tool that they've been working on for a while that goes into that yep I haven't it as closely as I want, but that's definitely a an interesting answer in this space. Yep.

A

And if you have questions about that, specifically one of the weave works guys does the European session with me earlier and today. So if you have specific questions about that, they definitely like to add two questions life. So with that, let's just wrap it up. Real quick I'd like to thank you all for coming and thanks to Amazon liquid web and hepped, you know for a lot of their engineers to contribute back to the community.

A

That's always awesome and thanks to the CNC Fi you each have earned the highly coveted kubernetes water bottle, which I will be getting to you at some point. If you're an expert in kubernetes- and you want to help sit in a panel, helped answer your questions. Please feel free to ping me, but we do these third Wednesday of every month same time. The topics and all the information is in the topic in the hash office hours Channel and with that panel, do you have any final words.

B

That's big enough to hold a whole bottle of wine.

A

Yeah, the contributor bottle should be much larger and hold like just I.

C

Think the way I would wrap up is you know there is cube con coming. So if you guys any of you guys are gonna be there, we would love to talk to you and learn from you and you know hopefully answer your questions as well. Sher.

D

Singh will be there as well, and it's gonna be a party I. Don't know if I find at my hotel room, it's I'm doing something wrong. I should be.

D

A

Scheduling and the holidays might be tricky because the third Wednesday falls on usually when people are have are on holiday, and things like that. So, however, in the channel we are gonna, try to run a session. It's just we're coming up the cube con everyone's trying to land a bunch of stuff at the same time, so it is very hectic, but we appreciate it when you come, and you ask your questions. Please tell a friend about this. If you could share this at work, those of you that are doing kubernetes professionally at work.

A

Things like that. We want to make this show valuable to you, so that your your boss wants you to come to this and learn about it and meet other people who are having the same problems in it in a kind of more fun manner than just posting in kubernetes users and praying someone someone responds. So we we try to bring our personal touch to this. So with that, thank you very much to our panel and this concludes the office hours for this month. Thank you all right. Thank you. Guys.

C

A

Everyone, my ex.