Kubernetes Usability Special Interest Group, 22 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG Usability: Deep Dive - Vallery Lancey, Lyft

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

SIG Usability: Deep Dive - Vallery Lancey, Lyft

SIG Usability is a new Kubernetes special interest group focused on the end user experience of using Kubernetes, from initial adoption to day to day ops. Join us at this session to do a deep dive into what we've been up to, and where we're going.

https://sched.co/Uagd

A

Our goal is to make communities more usable through the focusing of things like user experience, accessibility, internationalization, all kinds of concerns they easily fall through the cracks.

A

I'm stealing this slide from Tasha's introduction the other day, because I think it hit quite close to home. The project has suffered pretty badly from something called Conway's law, which is that the organizational structure of everybody building a thing tends to inform how the thing is built notice. The result, sorry, as a result, the user experience incriminated tends to be somewhat sharp in a lot of areas, so we're a new cig. We were formed a couple months ago.

A

So at this point, we're still fighting our feet a little bit, especially as a horizontal sig, because we don't directly if own components, we more own concepts and we work with other SIG's to make sure those concepts are applied. Right now.

A

There's a couple different projects, we're working on in particular, trying to do more use of research to figure out what people are trying to do, how they're experiencing things and what they need and to better bring that in to how we build things, because it's very easy to build beautiful machines or things that solve the use case that we think exists, but it's harder to actually make sure that we build what works for everyone.

A

So in particular, today I am more of an engineer than a product person. I'm gonna particularly talk about trying to change some of the default settings in communities around security, so that it's less of a sharp edges nasty surprise for using it. So in particular, I'm going to talk about network policy how to change that and a little bit about cloud security policy. So lots and lots people actually don't use these with production. Workloads.

A

I won't ask for a raise of hands because that might be embarrassing, but it's very very common, and these are actually very important resources.

A

So, firstly, network incriminating is we have a lot of network abstractions that we leverage very heavily?

A

We don't, in most cases, tend to think about the individual plot, IP and discovery, because we just use something called the service, so services exposed nice little short, convenient DNS names within their namespace, so you get to stay like connect to Redis or connect to HTTP nginx, and, as a result of that, we tend to automatically assume that these are only available within the name is were in the our services who nicely protected like that, and something outside the new space can't get in.

A

But that's not quite accurate, so any two pods in communities need a CNI driver is called to connect to one another. If you just spin up a cluster on bare metal like I have before you don't install a driver by accident, you can't actually use the network.

A

It's a surprising frustration, so you use something like calico or flannel or like a club, bright or specific driver in order to make sure that everything gets an IP and that there's some kind of rules there like tables rules or cloud rules or cilium, managing EPP F, or something to actually connect two things together. So in reality this is the default Network grid. If we just spin up a cluster and post workloads, it everything can talk to everything else. You have to be able to figure out I P, Zoar guest addresses, but it's all there.

A

It's totally unrestricted, so who's surprised by this at least one hand. I will admit. I was using current IDs in production. Well before I learned about this behavior and that's a very common thing, because we I will fully admit like like to go onto Stack Overflow or quickly. Copy/Paste something where the dogs- and we don't tend to realize the full implications of how the nitty-gritty machinery works and that's what ideally, we should change, because the possibly resistance should be a better path.

A

So the network policy is an object specifically lets you put restrictions. It tells the cni driver which routes that should actually allow versus which it shouldn't so some kind of firewall level again, be it like hard, stuff or tables or whatever. There will not be rules put in place to connect to giving pods. If your policies don't allow.

A

So the way that in the irk policy works, semantically is the default. Behavior falls open right. If you have no network policy, everything to talk to everything, but network policy is when, as soon as they apply to pods, it becomes an instant lockdown. So, even if you just touch a pod with a policy and specify no rules, they kidnap nothing and, at that point, on its a whitelist of adding the increasing rules to let it do or let other thing do more and more things.

A

So here's a fairly typical example- forgive the Amal I know it's a usability talk, taking a specific label on pods in a specific namespace and then saying hey you get ingress to this, so I'm in particular, for the label, be pods a in namespace with team a in connect. So this is like a way of staying. Your workers can access my cache or something, and nothing else can access this.

A

The label semantics if you're not familiar, is a super super common pattern at kubernetes for specifying the relationships between things. App is also just a very common convention. It's not actually meaningful resources get individual key value labels and then we want to match something up and we specify a label match. So, if I want to like point out in a policy, a particular set of pods I'd say like service X or app Y or whatever the label is to specify all pods in that high-level deployment slash service.

A

This is actually what that looks like. This is a much more reasonable graph of network access. Pretend there's probably lots of other random things there, but it's not everything to everything to know exactly what we.

A

So, let's change this right. We want to make this better. We want to make this something that's easy to do, and users are actually funneled into versus something that users have to read. D the I. Don't know how many pages of carbonate a stocks to figure out because all fully owned, I'm, a bad user of software, if I'm, just trying to set up a database or something I, will go in and try to kick. Take the quickest path.

A

I, don't do that with kubernetes, but that's because I'm, a community developer, I'm an expert in and I take the time and I understand it. I certainly am unhappy when developers expect me to have a deep knowledge of any product in order to music successfully and I realize that people will have the same expectation for me of things like communities that I work. So we should change this.

A

We should make it explicitly more closed and give users the kind of danger button escape route of okay, I'm gonna make it unrestricted just because it's better for my use case and I have signed to be metaphorical. Nda and I've acknowledged the risks.

A

The obvious question is: why isn't like this already and it's a nasty question that I've been grappling with lately? The three biggest answers are had ability with existing systems, because you may have heard kubernetes is a thing now. People use it in production, it's past version, one. We have specific guarantees around compatibility. The next one is support. We don't actually think Ruby Nettie's make a CNA driver, we actually kind of do but we're actively deprecating.

A

It's that we maintain a spec, but then other companies or third parties write things for so the network policy right now is an optional thing.

A

Those are specific expect for the CNI, where you have to be matching a certain set of api's to be compliant, but network policies deliberately something flexible, because how you want to manage your network is a super, complex and opinionated topic and to an extent now- and certainly at the time this was created, we didn't want to lock ourselves in and lock everyone into hyperopia model, it's kind of the ethos of kubernetes. We let people build whatever they want, and the last one is just convenience for better for worse.

A

There was a choice made that we want the kind of crew neighs up and running hello world path to be as simple as possible and having to configure were more pieces of yeah Mille. To get your demo app going is pretty frustrating.

A

So I was one of the people who was somewhat involved in the process to remove some api's and the release before last I was more doing like the PR side than actually doing the implementation who's heard about this, so I failed.

A

So this is I believe the first time in GA, kubernetes, we've actually deliberately really broken. Basically, there are a couple things we either wanted to move them around semantically, so the resources existed, just different paths or we were finally retiring old old versions of things that we just wanted to ask. People should be using the new thing, so these things were deprecated for multiple years. You get warnings, it's in all the release, notes and then 1/16 comes up, so we delete them.

A

We hammer it with blog entries before mentioning it on all like their individual talks and live streams like tweeting about it. It's an all the release notes over and over and over, and then it's just it's so sad I load out Twitter when it's released and what's going on, 16 broke my cluster because people didn't know we knew it was gonna, be ballad, but we didn't realize how bad it was going to be because the reality is number one.

A

People just aren't super engaged and aren't reading these things, which I guess shouldn't be a surprise, because kubernetes isn't meant to be super exciting. It's meant to just be a detailed and making your stuff. You do care about it, and the other thing is people who were fully aware that this was happening still were unaware of some dependence that they had. They would get shocked because their Jenkins, installer or Jenkins manager relied on some of these api's to deploy and even though they thought they had updated everything they haven't.

A

So this is a long story, but the point is it's taught: is it a very real way about the caution of a we need to have making breaking changes? Even if we follow all policy and do it to the letter? Because it's still really rough now the question is: how do we introduce a new default without just saying, hey, 118, everything's, gonna break? It needs to start opt-in in some way. There's lots of ways we can introduce behavior like new CLI flags or some kind of cluster config or new resource versions, and there's no require.

A

We actually have like a very well outlined document for kubernetes I, forget the name. If you, google, like urban areas, deprecation policy he'll, find it around what kind of breaking changes we can make, what we can't and how long we have to get people notice or around different things like if Minzy lies sources, users, Eli's versus api's and all those things and, lastly, there's no letter of the law. It says that it should be easy to migrate this, but it really ought to Python 2 to Python.

A

3 is a big example that people bring out, because that was a bumpy migration and, as a result, people are still running. Python, 2 and people are gonna, be running at past end of life because it was hard to change and they just want their stuff to work.

A

So the kind of obvious thing that comes to mind is we're changing the way network policy works. Maybe we just bump the network policy resource, but unfortunately it doesn't quite work like that, because we're not changing the way a network policy works, we're changing what happens in the absence of a network policy. If we want to make the network fail closed, so Network policy v2 wouldn't mean anything because it would fundamentally mean if you just deploy stuff with no network policy.

A

The behaviors change it's kind of interesting, because it's like not the API guarantee is the semantics guarantee of what every source means. So the idea that was brought up when I raise the cap, which is our semi-formal formal process for making notable changes in Cuba Nettie's, was the idea of having a namespace v2.

A

So specifically, we have processes to sometimes create resources on certain hooks like. If you create a namespace, we create a default service account by default, open if you've ever seen that or use that you'll notice, you didn't have to do that. It gets created on namespace. So what we thought about is the idea of having a name space in b2 that will automatically create a network policy that touches everything, says everything's fully locked down. So that way we're maintaining the semantics.

A

If you delete that in our policy, because you don't want it there, everything behaves as you expect and if you add more policies to it, then you're explicitly adding whitelist as expected. So this is a way that isn't breaking any guarantees and also because we deliberately make sure that things like twirling, like lips, ETL biases towards the new API. Is it's a way to default on around people on to this? So they don't necessarily have to aggressively opt in.

A

We just catch the people who don't care they just are going to be for it, and people who specifically have a huge problem with this or have compatibility guarantee is, can deliberately use namespace v1 and avoid this behavior until they're ready to change over.

A

There's additional fun as well, like kubernetes, conformance cuz. This is the kind of thing where it would be so so easy if this had been thought about like four or five years ago or more in reality, if it was a big enough priority, four or five years ago, because there were many decisions that were in different urban areas has to pass a conformance suite. It's called of and ant s to be considered compatible.

A

So this is why we get things like azure, kubernetes engine and visual ocean kubernetes, and all these like something kubernetes is because they passed the conformance suite and behave like a standard community initiative that we can swap it with. So this test set is quite complicated and we need to make substantial changes to it, especially in the face of any vendor opinionation.

A

And the last big issue is just the user experience part it's like a classic complaint. Kubernetes has too much. Animal I recently spoke with someone who used to work with Tupperware a Facebook product, and their big complaint was that they used to be able to have Tupperware was kind of like the Borg at the curb. Annette is a facebook, so their complaint was I used to have one structure.

A

I'd write out now, there's like two Starks I have to write out a bare minimum for a service, often like so it's more friction for users if they have to set up yet another thing to get.

B

A

And forcing more people into network policy will also just expose more of the sharp edges of it. We know it's not a perfect spec and some people deliberately avoid it because of that. So if we make everyone use it, we may want to think even about polishing it and having a v2 of just different features that what we do that um somewhat notably SPO made this design choice, so network is default restricted or this was I think they might have changed it unless you explicitly add allowance and that really bothered people because of that friction.

A

So it's not just theoretical we've seen in the space people don't like having locked down so.

A

In a bit less detail, I want to talk about pod security policies, I jump around a lot of sig sig networks, kind of my home, so I focus on the network policy, but what security policies are very, very high, so again, maybe a mental show of hands. If you don't want to embarrass yourself who hasn't heard of this.

A

Yeah cool, so it's okay, you're one of like lucky, ten thousand people who are going to learn something new today, you're gonna learn something really important.

A

Pod security policy is restrict to what pods can do by specifically restricting what details of pods you can create so they're, not an enforcement mechanism for pod behavior they're, an enforcement mechanism for pod specs.

A

It's basically a set of rules that says you cannot do certain things such as you can only mount these volumes or you can only act as these SELinux verbs and if a pod has something in it, spec violates this. Then it just can't be created. You get like an API error, so if you're not using pod security policies or some equivalent thereof, like, for example, the OPA is provides equivalent functionality and much more granularity, then your cluster can get totally taken over. Probably so. This is a scary thing.

A

I learned this a number of years back when I was trying to set up genna's built on kubernetes and I was all excited because it was gonna, be so resource perfect, and this was back in the days where, like darker and darker or building meant actually having to give stuff fundamentally root access. I was like hey. If you can make a Jenkins job, then you can create pods, which means that you can use this escape hatch and take over the build cluster fun. So that's why that project failed.

A

Simplest example is privileged containers. Privileged containers can get fundamentally root access on the cluster. It's meant to be a very strong escape hatch, for if you have to deal with like specialized hardware or something that you can't easily privilege the containers to do at like a fine-grained level. So if you specify in your pod security policy that produces false for set of pods, then if you tried to create a pod with like privileged true security spec, you can't you get denied.

A

So if this is one way where an attacker is blocked off again, like the good news about this, is you do need the ability to create or potentially modify pods in order to do this kind of attack, but typically, like you, have a lot of machinery, a lot of developers who do that slightly more complex and sometimes more subtle example? Is volume outing. So here on the Left, we have a spec that lets us mount pretty much the minimal set of volumes for a normal pod to go.

A

It's all like the meta stuff, like the ability to mount conflict, maps and secrets that are part of just often setting applaud us, assuming that any persistent volume set up Chrissy. So this means someone nasty tries to set /proc or UTC password as their volume. They can't do it without this. What I could do is I could make a plugs back.

A

That says: I'm gonna mount an apt C password into my container now I connected my container and go I'm gonna, add a user, I'm gonna, add a Valerie and it's got sudo I know suddenly I have the ability to get sudo on the host really really bad stuff. So again, this has similar problems with trying to push it forward to network policy, making a change. It's frustrating and there's some additional questions floating around.

A

In reality, if changing, this would probably require a pod v2, that's kind of unfortunate, because that would mean actually 8uv, probably like the core API and.

A

That would functionally be kubernetes, v2 I'm sure we're gonna hear about the 2.0 release one day, but it might be a longer road that I've spoken a bit with people from sig off who actually maintain these components and there's also some uncertainty about the future of is pod security policy going to be the way to go forward. They're actually doing a talk, yeah they're doing a talk tomorrow, which I'm told will cover it. I, don't have any guarantees on that, because the pod security policy is just a very opinionated admission.

A

Controller admission hooks are something we have where you can validate or change things as they're applied to the cluster. So it may be hot security policy, maybe a built in concept, but it's using a very generic thing that anyone can work and in fact anyone has, because we have things like OPA and gatekeeper that are frameworks built upon that specify very advanced in granular rules. So, instead of having like 15 different options with list or toggle saying I want to allow this, who don't want to allow this?

A

People are building all kinds of customized things, or even just letting, like individual companies write their own rules in a trivial fashion, so there's a reasonable chance that we're going to want to move forward in that direction versus specifically using cloud security policy again that bias towards configuration of reputation. A lot of these concerns are things where it would have been easy to design it right, or at least not as hard as it isn't health.

A

But now it's hard to change and the good news is: it's no longer a hack to get kubernetes of the door. So we beat me those we're we're here we're at coop. Con we've arrived so now the standard is way higher for actually building things, and now we have things like cig usability trying to work on these problems. I'm also part of a cig architecture, effort to have a public readiness review process and a lot of it is going to be. Are we making a default that isn't ideal?

A

We're gonna get locked into it, because well, these things are possible to change and there's a nonzero chance. They will change. It's a huge uphill battle to do so, because change is hard for running software. It's expensive design to design things for wrong and that's why we really want to focus going forward on making sure we do things the right way to alleviate that cost in terms of the development effort and the downstream user pain in the future.

A

So if you want to get involved in the sig, there's a couple different things going on right now, the majority of it is more user research focus. This is kind of a smaller thing, with an egg that I want to highlight, because this is where, like kubernetes devs, come in particular studying user studies trying to build profiles, doing like a jobs to be done.

A

Survey around what are the specific tasks, people use for Nettie's as well as things like are the Box behavior that I'm working on with some documentation, work and then there's kind of more ad hoc collaboration with individual caps and things like the cigarette grr of how do we design the right thing now that we're here?

A

So if you want to find more about, say, usability who we have a page in the standard, github repo for acaryas, can we do do stuff? If you also just Google communities, thing usability you'll find us, it was a page with a whole bunch of stuff. There is that kind of main details to know is joining the stock and joining the mailing list. So the mailing is probably where most communication happens. We also have like every three weeks a community call a sync up on things, but otherwise you know pop into stock.

A

Popping mailing list watch what's going on. If you have questions, ask them, because this is a this, isn't a feature thing. This is a make communities good thing, so it's hard to get people who are being paid to work on it, because everyone wants their new feature at the door, but as long as community is mostly manages to roll along things like usability and scalability, and security are often a little bit less focused on. So, if you have the ability to help out, that would be greatly impactful.

A

So I hope you can take some questions if folks happen.

A

A

If you showed it to me, I can also repeat it.

A

So the question is on enforcement versus visibility, with a couple examples around like exposing information in pips et al. That song is maybe not ideal. It's something we have thought a bit about. The like I think it needs to be a two-pronged approach. Enforcement is far more complete because we have multiple ways of interacting with humanities. Technically I'll go through the API, but like a lot of people, only ever experienced Coop's ETL apply in cubes detail.

A

Deleting viprinet is like they set up a jenkins thing of the jenkins thing, just poop CTL, so poop CTL is a pretty good start, but there's a long tail of people who are using some tool where they're either using coop CTL and not seeing the OPA or they're using a thing. That's calling the API and that thing like if they're using a Jenkins integration, might never surface the details.

A

I think the 1/16 breaking changes taught us that we do need to service these things more. Visibly, though, like goup CTL will warn you if you are creating a new resource with the deprecated yeah, but maybe I should just warn you whenever you look at that or maybe even when you look at like something adjacent to it, like the service, that's attached to the pods that match the deployment that could get messy but the more visible we make things the easier way, our equipment users to make the right thing.

A

Yeah we had a couple of conversations around novel ways to address that, but I think that definitely exposing that in some format could be ideally maybe even longer term. We could have song and status with just warnings again. Super hypothetical I'm sure we take a lot of bike setting to come up with the right solution, but making sure that just generically the information is exposed and either enforced with tools we control or trust with tools that a third-party that that information will bubble up. Somehow.

A

Thank you. That is a good point and I'm curious about some of those specific proposals.

A

Any other questions other ways we can get six minutes of our lives back.

A

It's more in the stages of actually building out the survey, but I can connect you with the people who are leading that effort. If you're curious more about it.

B

A

So the question is on nested. Namespaces just is that the multi-tenancy remote the nested namespaces? There is a something different.

A

Yeah I'm not super sure the issues offhand, but one of our co-chairs for the Sega's Tasha Drew who's multi-tenancy, lead so I'll check with her and see what kind of they go in concerns there would be. But at this point, I think, like the bigger blockers are working with constituencies to see. If they'd want the change to happen period versus just the mechanics of it all.

A

Right bus call. Thank you. Everyone for coming.