Red Hat OpenShift OpenShift Commons Briefings, 26 Mar 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: There's No Such Thing as Vanilla Kubernetes - Aaron Aldrich (Red Hat) via @OpenShiftCommon Briefing

Description

There's No Such Thing as Vanilla Kubernetes
Guest Speaker: Aaron Aldrich (Red Hat)
2021-03-26 OpenShift Commons Briefing
https://commons.openshift.org/events.html for upcoming briefings
#TransformationFriday
#DevOps

Abstract:
There’s no such thing as vanilla Kubernetes in the same vein that there is no vanilla Linux; even if you start from source, you must custom build everything around it to make it production ready. By looking at the past and the present, this talk will explore what it takes to build and operate Kubernetes in production, what alternatives you can buy, and extrapolates on industry trends towards “utility” services to explore what the future of Kubernetes might look like.

Aaron Aldrich is a Managed OpenShift Black Belt at Red Hat, creator and host of Tabletop DevOps streaming monthly on twitch.tv/desertedislandtv, and an organizer for DevOpsDays Hartford, NYC and Boston. Passionate about Resilience Engineering and Mental Health in the tech industry, he believes that every technology problem is ultimately, when you get right down to it, a people challenge. Find him at crayzeigh.com for thoughts on technology and people or on twitter @CrayZeigh for a potluck of technology, politics and general tomfoolery.

A

Hello, everybody and welcome back to the openshift commons and today's openshift commons briefing.

A

We have a black belt with us, an open shift, uh devops black belt um aaron aldrich, who I'm sure many of you have heard and known on the airwaves here on the internet and he's going to tell us why there is no such thing as vanilla, kubernetes and then, when he's done doing that rant and talk, we will have a conversation with jay, bloom, sasha, rosenbaum and myself and anyone else who's joined us in this live broadcast across twitch youtube and the rest of the internet.

A

So aaron introduce yourself, tell us all about yourself and then justify that statement.

B

All right uh so hi everybody uh I apologize, I don't have uh twitch up and running in front of me. Folks are chatting there, so someone will have to communicate the chat to me. If that's the thing that occurs, while I'm uh going through all this here, but I'll I'll get there uh so um yeah I've, I've done the wonderful thing of building a nice little. Who am I into my presentation? So you get visual aid, uh so I am erin.

B

Aldridge managed openshift black belt here at uh red hat um historically through a lot of things, I'm also tied into the devops days community. I organize uh started the hartford event for that and I've organized for both new york city and boston as well. So if you saw the boston event last year that was online, it was there.

B

um I also host the tabletop devops thing that also streams here on twitch with using uh dessert island tv uh care about some mental health uh in tech is another corner that I will often be fouled speaking about and uh yes, my license plate actually says uh devops, which is sort of shamelessly stolen from uh matt stratton, uh but he lives in a different state and his license plate changed to pedal anyway. So it's fine.

B

um So yes, let's jump in a little bit here and talk about uh the kubernetes landscape a little bit and we're talking about vanilla kubernetes. So before I tell you that it doesn't exist, let's actually establish what it is, uh we're talking about right and uh first things. First, here it is right. We found vanilla, kubernetes already, I'm lying to you. um No, but uh it's a little bit more seriously. This is the kubernetes project right.

B

You can find it online, so uh I don't mean that literally this thing doesn't actually exist in the world on the internet, but there's no real practical use for it doesn't use, uh doesn't really exist in practice. um So there's a reason. People want vanilla, kubernetes and talk about it right often the myth of vanilla kubernetes is uh portability right. If you're built for just the upstream software package, you should be able to work everywhere.

B

um Maybe lock-in is a concern, so you don't want to be stuck with any one vendor. You won't be able to move around uh and you want. You know cutting-edge features right if you're consuming upstream, you don't have to wait for it to get packaged and shipped along, um but these don't really exist in practice, and so let's examine what it. What kubernetes really pertains to right just running kubernetes by itself already is going to take um a bunch of different services right, whether it's working specifically on the control plane or working on the node.

B

uh You have to make sure that cd is working, make sure your scheduler is running. You have a tube controller manager, you're running in a cloud right. You need the cloud controller manager, so you can talk back to the cloud services apis and understand all that and that's all you know making sure you have your kubelet, your coop proxy. You need all these other uh services like dns, so everything knows where to talk to each other.

B

uh Even just container runtime is diving into a question as well right which we can talk about a little bit uh here, and you know all these different services as far as running it in production right you're, going to care about your application logs, your application, metrics that's got to be figured out and implemented your storage layer, your networking, uh you know, there's load balancing built into kubernetes, but how are you going to do that practically from uh ingress coming in from outside your network or egress routing, it's all going to be figured out.

B

You've got your underlying infrastructure right, just choosing what type of metal to run it on or what's involved in that is going to be a choice that has to be made. How do you want to automate that infrastructure? You can use ansible or you're using something else to do that. You've got to maintain that long term, as we all know like infrastructure, as code is sort of saying the same as eventual inconsistency right.

B

So we've got to continue to maintain all of this stuff uh that goes on for it and we even have to make a choice like again, like I talked about container runtime, that's a choice right. You don't start with the default container. Runtime you have to choose. Do you want container d? Do you want cryo uh are using docker for it or actually docker's? Not really it's going to be removed, we're not going to support docker shim anymore, but that's been pushed back a little bit further, so you can still use it for right now.

B

uh You know and that caused a whole bunch of confusion online not too long ago. So even just the core choice of like, where do I actually want my containers to operate, uh is something that's going to make your implementation of kubernetes unique from somebody else's.

B

So realistically we don't have extreme portability. You've now got this custom tuned infrastructure that can only run your your apps right, because they're built for that environment uh so that lock-in, yeah you're not locked into any given vendor, but now any environment that you move to is gonna have to look pretty similar to the one you're operating that you've custom built right. You have to continue to custom, build that environment everywhere you go and now maybe you get the latest features, but that is for everything. It's also.

B

The latest vulnerabilities, the latest cves that you have to carry you care about all the latest deprecation of packages like all of this now becomes your responsibility if you're actually running uh vanilla, kubernetes, so all of the benefits that people aim for uh get shifted away by by the actual practical implementation right all the theory uh sort of disappears once we run into reality.

B

uh So the conversation this ends up being is a lot more like build versus, buy right. This is probably a conversation you've heard of before I kind of hope. People have uh some experience and understanding of the build versus by conversation, because we do this all the time right. This is not like a completely unfamiliar concept.

B

uh You may think about it. You do it every day, right think about just lunch time. Do I go make myself lunch or do I go, buy it um and there's different trade-offs in that that, and I feel like we talk about it so much. It should be this sort of settled case law of like okay. We understand the process, we know how to do this now, uh let's walk through every scenario, and I think we kind of do um so.

B

Let's actually examine this scenario, understand what our build cases and our bi cases are and and kind of see it a little bit more closely um build right. So, if we're building the software, this is the example of like our pure vanilla, uh straight tapped from the source. Kubernetes right, we're downloading, compiling source might be the extreme example of building yourself, uh and this is like the same as thinking of like vanilla, linux right. This would be like a custom compiled uh distro that you're pulling right from source code.

B

uh So the nice thing is you get ultimate freedom right. You can choose every little detail. Everything can be custom built, it can work specifically for the application or use case. You have in mind it's built towards purpose right. It can be specially tuned there. uh The other side of freedom is always responsibility like if you've ever seen the the netflix culture deck. They talk a lot about freedom and responsibility to kind of go hand in hand. While you get that freedom to do and create whatever you want, you're also responsible to answer for it.

B

Why you made those choices, and you have to have deliberation and um and understand all of that right? uh I think the line is like intention without strategy is, is chaos and when you want to answer to your c-suite about like hey, why did we choose this package uh for implementing? You know our our runtimes like? Why did we choose docker? If that's gonna be deprecated, it seems like that was a lot of work like it.

B

Just felt right at the time is probably not going to be a great answer to deliver uh in that conversation, so it requires a lot more thought and choice and, as we all know, from from open source right when you build it yourself, it's it's free as in puppies right. So there's, uh maybe no upfront costs you can do without paying a software vendor. You don't have to have a sales conversation or uh procurement process, but what you're saving in that upfront cost you're shoving into implementation and learning right.

B

You've got engineers that now need to become experts, uh you've got care and feeding and maintenance that needs to happen. So that's kind of the what the the costs are of our our built.

B

uh On the other side, you can.

C

B

And this is again just like a house right, you can custom, build a house there's a lot of choices that have made or you can build one that's already standing and you get all of the uh the information there. uh The example for extreme build uh in our case might be uh managed services for openshift, something like rosa that just launched uh recently. Red hat open shifts on amazon right. So this is often opinionated services.

B

So, when you're buying something off the shelf, all those implementation details have been chosen right, someone's made all those choices ahead of time. They might be the best choices for them or the best choices based on their experience in the industry. um But all those choices are supported and tested. They are, in fact, experts making those choices because they have implemented this and they're building it for multiple customers.

B

uh You get some level of uh being able to go back to, uh in our case, red hat uh or you know, whoever has built the product and say hey, I'm having some challenges with it. Can I get support and they're testing it not only with their own internal experts, but all of their customers continue to test that software as well right.

B

Everyone is using the same implementation and the same implementation details, so we find those edge cases faster, rather than being the only one who has this problem on the forum, maybe you're one of ten or five, where you get the yep, we're already aware it's in the next patch type of response, obviously the other side to buying it is it costs money and sometimes it can be extremely expensive to buy a solution versus implementing it uh and the other side of that, of course, right you get that expertise, and it's also usually faster to implement, and you are buying part of that uh part of that knowledge and part of the expertise is being purchased as well.

B

It's um an expertise that doesn't necessarily have to reside within your engineering organization. At that point right, you can, you can buy that if you need it.

B

um So how do we examine this choice right? uh First, obviously, we will have resource constraints if you have more time than money like that's always going to be a factor, and I've been in those organizations. The non-profits in the small organizations that really just cannot get the uh the lump funding to buy a solution that they want, so you're forced to sort of become experts and implement it at that point and that's a different situation.

B

uh Largely what we're talking about here is, if we're min-maxing our organizational choices, if all things are created equal, how do we want to spend uh money versus time, and the key here is always to focus on what sets you apart in the market like? Why is your service special? Why is your company, your organization, the best one to do what it is you're doing?

B

uh That's the kind of stuff that you can't buy right, because that's what makes your organization unique makes you special um doing the thing that you do so that's really where we want to spend time on. That's where we want us to uh things built for purpose, things that are going to enhance your specialization are going to make you stand out and make you better. That's where time and building comes in, because you have to be able to buy an off-the-shelf product uh that makes you unique right.

B

Red hat couldn't buy an off-the-shelf operating system way back when and be red hat right. They had to build their own implementation of the core gnu linux operating system to become red hat like to offer something to the market. It's interesting, um but all the things like implementation details, whether it's how you host your applications or um what's our process for delivering into production a lot of times, you can buy your way out of those decisions and pay someone else to be an expert.

B

At the end of the day, you're, probably most uh programmers in an organization aren't getting paid to be kubernetes experts they're getting paid to deliver value to their customers by way of their application like quickly and efficiently and have it worked every time. So all those implementation details, if those can be, uh we just buy our way out of those decisions. In that time, we can start getting more ideas into production, more faster, um which is the technical term more faster.

B

uh So you can probably guess my answer to this almost always is buy right, especially if we're talking about something like infrastructure or kubernetes like buy your way out of making those decisions unless like if this is not a core component of r, um of course, the exception that proves the rules. I think this is a good story. Is uh the company datadog, uh if you don't know datadog, they're a sas uh monitoring company observability company? uh So you know you order a data dog off the internet.

B

You install clients on your machines and they bill you per system that you have. You know, processing data or you get billed for the data that you're keeping or processing- and I don't work for data dogs- I'm not going to push them any further, uh but the implementation of of kubernetes is the example here they have built it and they are operating their own kubernetes uh for a reason right. They made this choice.

B

Strategically uh part of it was there because they're, an infrastructure, monitoring, company understanding how kubernetes works and understanding the ins and outs and what you care about and what it looks like right before it falls over, is extremely important to them right. They want to know how their customers use this product, so they can offer a better product. Their customers, so becoming experts in kubernetes is actually something that gives them value uh the other side of it is they started early enough in the the life cycle of kubernetes there weren't services offered for it right.

B

There was sort of the project, and maybe one or two distributions that existed, but largely uh they were jumping in extremely early. They became, or you know, contributors to the project. I remember seeing the a couple of talks they gave here. So these are our two examples of some of the talks they they talked about boston.

B

You know, kubernetes, the very hard way right if kelsey hightower's introduction is kubernetes the hard way they talked about actually practically doing that in production. um You know the challenges they ran into, how many times dns broke for them, because it's always dns uh how many times containers stop being contained right. All of these weird implementation details that you have uh the fact that they had to contribute to the core project in order to get the services that they needed right. They had to actually build into kubernetes the functionality they wanted because it wasn't there.

B

uh So these are all things you take on. If you choose to build it, obviously they had a good cause. It got them where they are now now they're operating extremely large. You know thousand two thousand node clusters uh in production and obviously you know their product works and they still have customers, so it worked out well for them.

B

um Okay, great so the exception that proves the rule: you're, not datadog, you're, probably looking at buying. uh Unless again, unless you know offering that platform is core to your organization or you have other resource constraints. uh So what are our options right? I mentioned distributions, and this is just like linux right. If linux is vanilla, you got to build it all yourself or you can get a distribution you need to get it already built with a lot of implementation.

B

Kubernetes has a certification right. This certified kubernetes uh is making sure that any given distribution that has this stamp on it is going to be compatible with the core project. So any application that should run in before kubernetes should run on anything. That's certified kubernetes effectively. Taking that vanilla purity test and and making it meaningless, saying this contribution is fine right. It passes the vanilla test.

B

um So if we're gonna certify fresh uh some of our distributions, let's look at what we've got here. uh We have a number of distributions all over the place, whether it's openshift or rancher or aks, or what have you um and, of course, they're all certified right? Why wouldn't you be? If you have a kubernetes distribution, you have to be certified kubernetes or no one is going to use it.

B

uh So just this standard is already answering some of this purity test question um here is the bit that they're talking about this is sort of upstream kubernetes when we're certifying communities. This is the bit we're talking about we're saying whatever product we have here is compatible like it has all these components and they all work the exact same way uh that we would expect them to also taking a look at distributions.

B

Let's take a look at eks right that runs amazon's own version of uh kubernetes, and you can see they've added a lot of stuff around it. Some of it is how you code in production. Some of it is just implementation. Details like where does my control plane live? How am I what types of systems I use? Am I using? How else do I want to extend uh that's the type of stuff that gets stacked around it? It doesn't actually change the underlying compatibility of the application.

B

It's still completely compatible with uh kubernetes, and has that certified bit right. Google kubernetes engine uh very similar process right as far as a cloud provider and google right, the originator of kubernetes is the board project they're, not using the pure product right. They have implementation details they've made. They have abstractions that they've made on top of that to make sure that they can get their monitoring and their storage uh wired up to it properly to their services.

B

uh And, of course, you have something like openshift right, openshift's a little bit different because we have this product as it doesn't just live in a cloud. It's also something that is a deliverable product that someone can install on-prem. It kind of does a little bit more than just the control plane. So you see here kind of lives inside kubernetes, but largely it's. The same type of thing: you've got an openshift cli, that's compatible and an openshift api. That's compatible with kubernetes.

B

You've got all these different bits that you can stack on uh to either extend the stack right and get more functionality or our implementation details just made for you, so you don't have to continually make those decisions every time you roll out a new quest.

B

um So that's that's that right, like we've got these products that are uh certified fresh kubernetes and that sort of covers the is this compatible with any other solution that I would build, and this is saying yes, we're going to make these compatible with the core. So anything you build.

B

If you want to have that portability and want it to be useful, you have to make sure your custom implementation of kubernetes follows all these certified kubernetes standards or you're, going to have uh lock-in details that um so this also brings up a little bit of an interesting point that I want to get into with these different services uh to go back a little bit.

B

Obviously, like I said, open shift is a little bit of a different product and these are pretty much live in the managed service realm, uh and these things, like uh you, know, xks services, gke and this managed service realm sort of represents.

B

Where I see everything going so I want to take a minute uh look a little bit into the future. Look at a map of where we're at and what's been happening with different technologies and see if we can sort of say, okay given all of this, given the existing landscape of build versus, buy and what's been happening, uh can we see where things are going and what we're looking towards next?

B

um So I think this is. I just read this article recently, and I think this is a great example of the evolution of uh organizations and service offerings, and this isn't about licensing and open source stuff, which is an entirely different kind of discussion. I'm happy to have, but not the point I want to. I want to bring in about this. The point here is largely about how mongodb transformed their service offering to to move down market right, and so the shift is.

B

Do we want to offer this product that requires hand-holding and installation to very large customers? They have to talk to our sales org in order to buy it. We've got all this large process to implement your own mongodb, or do we just allow people to consume mongodb on our cloud service right? That's the shift towards uh individual towards the ease of consumption uh over. You know, making big deals and packaging it all up, and that sort of thing uh so that's been a shift and what they found with this article gets into.

B

You are some of the details where, even though mongo's average spend per customer has dropped right, so each customer is spending less money on average, with mongodb, uh their revenue has doubled. In that time so what's happening, is they have to be getting this extremely long tail of individual consumers and everyone wants to move towards this? This is not just like a one-off example.

B

This is how we've observed markets move, uh so this is kind of a take in an example of worldly mapping, which is again a another whole thing uh that we could get into and is not the scope of this talk so I've I've got some resources in my uh slide deck that's published online or I'm sure, maybe another part of the discussion, um but the gist we want to get at here is largely this. um This red line is largely what looks like the x-axis from a worldly map.

B

It's always this move from uh genesis and custom build all the way through uh commodity and I'm waving my mouse around, although I have no idea if it shared on the screen, so I may be like me, uh gesturing at nothing, but that's all right I'll hope that it's useful for somebody. uh Maybe I can do the pointer there. We go uh genesis over here. We move along the line towards commodity and utility.

B

um This line, and moving forward about that is all about changing the ease of consumption. Genesis is like very new products right. This is when you want to design, have something. That's purpose designed. I talked about this when you, when you go to building versus buying uh your custom. Build, can be designed for a very specific purpose and be very very good at that, but it takes a lot of time to actually do it. uh We probably exist somewhere in this product.

B

Commodity overlap nexus or kubernetes right now uh there are some products that you take and install at home, and there are some that are being hosted and the cloud kind of asks a lot of questions about whether this is really rental or something you've bought. um I I think we kind of lean more towards this area right now, uh and that's obviously designed for manufacture right. Our goal is to be able to. How can I get this out of the door consistently and get widespread distribution, but our move here is always towards consumption.

B

We want the utility, just like movies, moved towards netflix right. It's really easy for me to swipe a credit card. Go online and click the thing I want and get it instantly. I don't have a shopper and store. I don't have to make sure it's in stock.

B

I can just get what I'm after uh and compute has moved this way right, we're no longer rack and stacking servers for like proof of concept, maybe there's reasons to run stuff on-prem, but for the most part, we're swiping a credit card asking amazon for a specific type of uh compute and getting it delivered over the internet without really having to think about it or worry about it. Everything is shifting towards utility. Mongodb is showing that shift services. The rise of sas software is doing that.

B

So I think this can continue to go things like the xke offerings uh kind of live in this commodity utility consumption, but they exist only largely for the control plane space, how you're actually implementing your worker nodes, how you're implementing your supply pipelines your software delivery? How do you want to implement observability in your stack? How do you want to do all this stuff? Where do you want to run it all these questions that come up as the sort of okay?

B

I've got a kubernetes now, what those are going to start shifting towards commodity they're, going to be more opinionated implementations uh we're starting to see this with, like the open shift operators right things that exist for uh can I set up a um you know: supply chain of software with a given pipeline that works for eighty percent of my use cases uh given utilities that tie in for eighty percent of my use cases so for most applications.

B

You just ask for a ci cd pipeline and get it out the door rather than having to custom, build and make all these decisions yourself. uh So I think that's where it's going. I don't think I'm the only person that's uh on board with this. um You know, andrew schaefer. I think this was 2019 so last year.

B

All time is a meaningless flat circle. Now uh serverless is the target of every devops project. He said this at uh you know his devops days atlanta, um oh my my where it was got cut off here: okay, um it was devastating atlanta which also combined map camp, so worldly mapping and serverless days. It was actually a really apt opportunity to talk about this, um but this is absolutely the the point like any devops product. I've been a part of the shift has always been towards uh ease of consumption. We're gonna, start operationalizing.

B

Everything make sure we can get similar results consistently over time. So that's a lot of automation and all the stuff. We talk about getting rid of, toil and self-service right. All that self-service is the aim serverless being the ultimate end goal of hey as a developer. All I want to do is consume run my code.

B

For me, I want to be able to give instructions to a system and have it do those instructions I think low code no code is, is the ultimate example of that, where I don't even have to learn how a computer talks, I just say, hey here's my idea for doing some things go. Do that please and not have to worry about all the underlying implementation details uh so yeah. I think that's the direction. It's all going as we're going to continue to build this. We have to think about.

B

How can we operationalize our platform to deliver consistent results every time and where is it worth spending our time versus? Where is it worth? uh Spending our money uh again, more people that have the same ideas. uh This is from a few years ago, from kelsey hightower to even you know, kubernetes enthusiast at google uh who said kubernetes a platform for building platforms. It's a place to start not the end game right and we're gonna start seeing continued abstractions and those platforms being operationalized uh so to bring it around.

B

Maybe the reason we can't find just plain, vanilla, kubernetes practically in production, is that it isn't an ice cream right. Kubernetes is a part of a sundae right. This is really what everyone is after. We want all of these components stuck together uh and just plain vanilla. Ice cream doesn't get us a whole sunday.

B

uh So, thank you. That's the rambling rant that I have for you I'd love to take. You know uh additional questions and chat with other folks to see where these ideas are interesting and people want to learn more. Otherwise, that's kind of my starting off point and the speaking link at the top there. This slide deck should be published with the notes attached to it, so folks can find it.

A

Well, thank you because that was one of the best um explanations of the different varieties and the of kubernetes that are out there. I still now I want the sundae. I don't just want chocolate.

A

I want the sundae and it's friday, so I'm gonna get me one um a couple of interesting things um popped into my head too, and if you pop back to the the worldly map.

B

Yeah yeah, let.

A

Me go back to that. Just one of the things like and especially the conversation you were having around mongodb um as a service, because one of the reasons I got into cloud ages ago, like eight or nine years ago, um was heroku right. It was a utility, it was an amazing thing and then everything. So it's almost, I kind of feel like we're having this back to the future, realization um that um services as a utility um and the ease of use piece is really one of the key things.

A

That's gonna one make money for people, but also make people use this stuff because kubernetes itself, the diy vanilla stuff, is um you install it and then then, what you gotta do so much else before you can get your wordpress app running wordpress as an old evangelist was like the go-to thing that we always deployed to show people hey it's working, you know back early days um and heroku had that down and did a whole lot of other things too, and so you know for me.

A

I think a lot of um this and I love worldly maps and we have done a number of talks on worldly maps um with ben moser and other folks. So please, um yes, always.

B

Always yeah and actually the the link to ben moser's uh site, though, like I think it's learnwardlymaps.com or something like that, is uh in the slide resources. So.

A

So yes, so thank you. I think it really frames it, but but I also like just wonder like how come it took us this long to realize that um you know that it was the commodity side of it. You know and and we kind of flipped the um the market again.

B

Yeah uh I mean, I think my my take note is largely that it's all uh part of it's cyclical right, that we have constantly emerging technologies of oh here's, an interesting and new way to do things uh and, as that, continues to um commoditize right, move towards and evolve towards a commodity um and which is you know, the direction of evolution. Our as we've observed in the markets.

B

um I think we're able to start taking those and building them in new ways build new abstractions that become new and novel, and then it starts all over again right. Well now, this abstraction is custom built. We have to implement it a certain way and as we start to operationalize that and get good on it, we start to distribute it, and then we look at how can we operationalize it? What are the best practices to do it right and get it working time and time again?

B

um So I think it's it's always this cycle, as we shift between new technologies right, I mean 10 years ago. Kubernetes wasn't a thing. So that's why it was that's why we didn't do it. We didn't know.

A

We didn't know- and we don't know what the next thing is either so, but it's always interesting to watch. These cycles come and go um and I'm really glad we're getting back to the commodity side, because this in between stage is always awkward.

B

Yeah, exactly that's that's what it's! The uh we've got, 13 competing standards, so we made a new one to unify them, and now we have 14 competing standards, type of implementations.

A

And it brings it back and I think jb you wanted to to talk a little bit about about the focus on the end users, yeah and switching this around a little bit.

C

Yeah I mean, I think, like one of the interesting things to think about.

C

Is you know, um especially the cncf now they're they're, talking more about like understanding, end users um and servicing end users better as as an open source community right and one of the things to say is that, like that those end user communities, don't stay stable, they move, and you know part of the part of what the worldly map is saying is that it's not just the product that matures it's the expectations of the community, that's using the product that matures as well um yeah, and so I think, that's like one of the interesting things to think about.

C

Is you know as an open source community? What do we have to start thinking about um in order to to better service customers that are coming to expect commodity style? um You know consumption of things like kubernetes they're, not they're. No longer, you know like you when you and I talked earlier right like yeah. No longer it used to be perfectly like the people who were early adopters wanted to be able to run their own kubernetes, and you know late adopters.

C

Don't want that they want; they want it to be handled for them so like, but what is the? What is the open source community going to do about this? What do we, what do we have to think about if, as an open source community to kind of shift or shape the product differently? I guess.

B

No, I think, that's actually that's a really good uh point. I think that that answer gets right to it right, it's all about who are your customers and, as you know, an open source community that can be a lot of different answers right. Some of our customers might be those platforms that are doing those implementation, and so it might be.

B

You know some of that type of stuff that people who are doing the building uh want and care about, um but as as well right, there are people that just want to implement it and not have to worry about it. So that's that's ease of use is always something that's going to improve adoption and gain more widespread, yeah.

C

Then I think one of the like.

B

100, what was I going to say uh like a thousand people, you paying paying you a hundred dollars or is just as good as one person paying you a hundred thousand right, yep.

C

Absolutely I think one of the things that that, like is, is worth kind of exploring.

D

No hold on, I can.

C

D

Jump into the previous comment and like no and in fact a thousand people paying 100 bucks is much better because you know.

C

B

D

On a single customer changing their mind so.

B

Yeah, no, absolutely actually, that's that's the even better point.

C

uh I think one of the like the things to think about. As far as like. What's different this time, um you know I I like. I think that platform is a service that whole pass idea and especially like self-service access to things is, is like from a from a different time than we're currently in it's from an itil based kind of process, driven version of platforming, whereas now, like kubernetes, isn't always being used like that anymore.

C

Kubernetes is like a platform where people kind of can co-create things together that that it's not simply like give me access to a machine. uh You know containers aren't like virtual machines in many ways, and so there's this weird way of thinking. I think that we could argue that the commoditized version of kubernetes will be more would be, would be an evolution over a path if it can continue to co-create value as a platform, as opposed to just be self-service.

C

If it degrades itself into simply a self-service platform, it will be it'll just be you know, another version of those print access to primitives. If, on the other hand, it keeps on evolving and shaping itself as a way in which the community can co-create value with end users- and you know, make a platform where there's real, effective, shared services, uh it will be a different kind of commodity. I guess is what I would say. I don't know if that makes any sense, but that's what I think about it.

B

No, I think I think you've got some there and I think that's where a lot of the value of something like open shift, having its roots and and upstream all in open source uh gives it a lot of advantage in that right. There's the ability um and the approach of how uh it's shifting more towards like we want to implement new stuff we're going to build an operator for that, so that you get some way of of doing that. It's providing both! You can do this totally custom.

B

If you want to- and you can implement this and you can work with the community to do something that works for your specific purpose and do something more purpose-built or if all you need is some generic. uh I keep using pipelines because that's the other thing I've been working on lately, so they're, just like floating around my head. You just need some generic ci cd pipeline. Here's, our implementation of it right use the operator.

B

This will get you where you need to go 80 of the time and for the other 20, let's co-create value and actually build stuff uh on this platform and then maybe even operationalize, that right so that can be repeatable for someone else that needs it uh and.

C

I think that yeah.

B

Yeah devops collab.

C

Right that that to me, like captures a lot of it because there's like without starting a religious, open source war in the middle of this, um you know open source, isn't just the licenses and it isn't just access to free stuff. That's not what the open source processes and methodologies that have been developed for a long time.

C

uh Red hat's contributed to a lot of this, aren't just about creating free stuff they're about how do we negotiate the best common set of features and functions to put into uh the thing that we're all working on together and uh that that is co-creation? That's what co-creation looks like to the extent that we can kind of keep that community alive. We will keep alive the ability to kind of do that type of work and then questions about how do you operationalize this thing become a separate set of questions right, there's up there.

C

Opera is operation of the thing, there's the development and kind of product manager, the thing- and I I think it's worth kind of pointing out- there's differences there and that, um even as we move towards something like a managed service style, operation of this code doesn't mean we have to degrade into you know again a uh you know: self-service access to primitive resources.

B

Version right right, there's a it can be sort of advanced combination of the two, uh and actually this is it gets to what I think is um sort of core to the devops process. Right kind of why I I used this quote from from andrew schaefer. um I was sort of thinking about this last night.

B

Getting the idea like a lot of development, is the process of like codifying the knowledge and building on it right we're going to do a lot of discovery of how do I do a thing and then I'm going to capture that knowledge of how I do with things that I can just do that repeatedly over and over.

A

B

uh And ops tend to live in this. What is the best way? I can do this thing and I'm gonna do a lot of research about like what is the best way. I can do this. I'm gonna figure out how to do it repeatedly over and over again efficiently, but often like until we have this marriage like, then it got lost right. Then it was captured in that org. Maybe we wrote it down. Maybe we had a run book. Maybe we had a checklist. Maybe five people knew how to do it.

B

Maybe it was a script that was poorly maintained on one sysadmins laptop right like this was how knowledge was codified until we had this great merger of like hey what if we learn the lessons from our friends across the street and start to like codify the knowledge that we've got and like this is what kind of builds that right. So it's we build something and figure it out. Then we operationalize it. Then we codify that, and now we can start doing it again right.

B

We can start operationalizing this further further, sharing knowledge back and forth of that whole process. Right.

D

I'm gonna play the devil's advocate through this whole. Like you know, goal slide and whatnot I mean like I, I spent a little bit of time in serverless community.

D

I wrote a book on azure functions, you know and for it for a little time, I thought that you know serverless was going to be a big thing, that's going to solve a lot of problems, and- and I don't know that I necessarily think that anymore and I feel like there's there's this whole um like the idea is always automating everything right like we always want to uh sort of remove the human. I know j bull like this, remove the human from the system so that the system can be more.

B

Reliable will fall apart.

D

Right so hold on what I'm saying is like the automating everything right. Like part of the the big um utopian statement, it's like we're going to get to the point where everything is like platform as a service or like managed services which is potentially even better than platform as a service, and I just see this cycling in the last 10 years where we like, we get a little bit closer and then we kind of take a step back and we discover that, like it doesn't answer all the questions.

D

So then we come up with a new solution. Right kubernetes is a new solution to the same problem that we've solved already before that and and we're now now getting to commoditizing. But I, like part of me, is like you know: are we answering all the questions? I don't think we are so you know who knows what's going to come around the block and try to solve exactly the same problem again with a different in a different shape or form right, and I I.

A

Would add- and that was I I totally agree with you.

A

The other thing that's changed now from like the early days of platform as a service in the heroical world is what I keep harping on is the rise of end user and the collaboration with end users like uber and lyft, and you know creating projects like envoy, what we're seeing now, rather than vendor driven initiatives, or you know, code dumps into open source repos that everybody you know gloms on to not that that's what kubernetes is, but um you know we're seeing much more collaboration with the actual end users of these projects, so people from netflix um throwing chaos, monkey and other you know tons of other projects into the open source world, and that is also the a game changer for me.

A

So we know it's wonderful. That um openshift is has an open source distribution of it, okd, which you know I will flagrantly promote whenever I can. um But I think the bigger thing for me is that it has changed.

A

Games is the involvement of end users and not just us, going out and asking for end user requirements and feedback on our projects, but this huge co-collaboration with end users, like some of our customers, amadeus and anthem, and other folks like that that are really contributing back and contributing chunks of code, sometimes right into the open source world. So that has so.

A

That gives me hope that we're just not going to iterate on the next technology and do this again and again and again, though I kind of think we probably will do that too, because that's our nature.

A

um But it gives me hope that whatever the next thing is after kubernetes that it that that will be something that we co-collaborate more closely with end users and that they become some of the primary drivers of these open source projects, as opposed to vendor, based which then again changes the relationship of vendors to end users, because um we always, you know like to think of ourselves as the trusted partners who you come running to when you need a patch or a feature or a function. And instead, what we're getting.

A

Is these huge code donations of things I'm going to hit on envoy um for my example here, but that comes back in that we then incorporate into our products. So this symbiotic relationship with end users and vendors who are productizing and making services and managing services is new. I think it's, I think, early days of open source, there were a lot of individuals, um education, edus and academics contributing to open source. But you know, then we saw this sort of phase of the corporatization of open source.

A

um I think to me- and I my hope is- that we're in a new phase of open source development and co-collaboration and co-creation of um these projects like kubernetes and and driving, where it's going so that we get this evolution and ease of consumption because we're actually not just listening and asking for feedback. It's not feedback anymore. It's the feed! You know that it's it's actually in the trough. Now what we're eating and we're all eating the same dog food, and hopefully it tastes better, but that's my opinion, humble as it may not be.

D

It's a it's. A knowledge sharing through code, which I think is, is definitely a much better situation than what we worry and say 10 years ago and and there's other things right. We've automated a lot of toil out of the equation and life is better right for operators, and so I can.

D

I can be proud that we were part of the devops days movement that you know contributed to the whole thing happening, um but it's like, I think I think it brings me back closer to the build versus buy and why you should you know: do you open, like open shift, as opposed to? Maybe vanilla right is like if you're doing kubernetes, because you want to put it on the on your resume and learn the skills then like do you know that it's essentially going to be useful in five years like?

D

Are you absolutely sure that this is the thing because, like I've, I've had I've seen in my career, I've seen technologies come and go and be completely retired and irrelevant, and they said again. Docker is a prime example for their site right. It was all the rage, and now, it's gradually kind of being least the runtime is gradually being paced out, and so you know.

C

Yeah, I think, but you know like the bill versus my question, is, is interesting because, um like for instance, onboard and other things that we've mentioned here uh are built on top of other technologies. So the question is: is your? Is your firm building something of value both to your firm and to the wider ecosystem? And can you identify what those are uh and release them to enable? uh You know a bet better competition in industry, as opposed to uh kind of the old version of kind of chinese walls?

C

And things like this that that create false uh scarcity? And things like that. So I think you know, I think open source um is becoming understood more and more as an actual strategic way of interacting with a market, as opposed to like a source of free, labor or uh free code, or something like that um and that that becomes interesting. And you know, then you can make arguments that say things like okay.

C

So, if you're interested in doing something where you're contributing you're either creating something unique for your your own organization or you're, contributing something unique to the ecosystem, then maybe it actually makes a lot of sense to ask someone else to deal with the operating parts of your system that don't have anything to do with creating that new value that are just the baseline kind of everything below here is just standard stuff that needs to get operated. Be nice for me not to have to spend a lot of time.

C

Thinking about that, certainly it would be nice not to have to be patching that all the time, so the people that I have in my organization who are capable of developing new stuff aren't redeveloping and re retreading wheels that are you know you can just go to the store and buy the wheel these days, you don't need to tread it yourself that type of stuff.

D

There's a quote from someone um that says you don't want to be below the api right like there's an api and once once once something gets commoditized like you can essentially consume something from an api right and you don't want to be below the api, wouldn't be above it and deliver actual value.

C

And it's uh you know again again, you know from for me from my perspective, from kind of a worldly ish perspective, that's not an ethical or a moral statement right. That is a business statement and it just says actually, if you, if, if the product is capable of operating at a commodity scale, also the other thing that it needs is that scale so, for instance like if you have a company that only has like 5 000 servers in it. It's you don't get the same advantages that, for instance, amazon gets for being able to operate.

C

10 000 servers per operator because you don't have 10 000 servers right, so you have to have a certain level of scale to even take economic advantage of commodity products like as the seller right. So I think it's important yeah go ahead. Aaron, sorry, okay,.

B

Apologize, the internet, making it difficult to have conversations since the internet, um so uh yeah I was gonna, get at. um Oh it's, the worst is when you interrupt yourself and then lose the original thing. You were gonna, say uh repeat the last thing you were talking about and I'll get back to you.

C

I was just ranting about the economic value of the uh like a scale of the market that you, oh.

B

Right and you need that, you need that skill yeah yeah, so uh that was part of why I said uh resource constraints is part of how making that decision right. If you don't benefit from buying it, don't don't buy it right like there should be a benefit and a reason to do that.

B

It's just that more often than not, especially when you're talking about um ops, targeted software right things that are operational, things that are all about helping you do the thing over and over again repeatedly in the same way, almost always you're going to get some benefit of buying someone else's expertise right.

B

There are asterisks and exceptions to everything, but you know once you get to a certain point, almost always that that makes sense, um because it's you know not just buying you're, buying that method, you're buying all the research you're buying all their maintenance, and all that goes into the price of software that we buy, and I think the the open source lends itself towards. This was something else I was thinking about.

B

The uh what I really like about the open source method is the way it lets us um explore a lot of different options, right, there's sort of that back and forth, and that we're doing the name we're doing the same thing over again. It's because we can explore ten different things uh at once from an uh see which one might be better say. Oh, it looks like you know. Kubernetes is the way to go. We might find out. Oh there's this limitation, that it has that won't.

B

Let us whatever I don't know what the weird next abstraction is that we're not gonna be able to do right, whatever that is, uh and then maybe we have to go back to the drawing board and figure out. How do we do this again?

B

That allows us to break through that that barrier right like where natural evolution requires an advantage all the time like always moving up, so you can get stuck on plateaus when we're talking about software and like designed evolution like we can step backwards in order to make a further gain, that's something humans can do that won't happen. Naturally, right.

C

The other thing I say really quickly is that one of the other things that I think tends towards this commodity scale based problem uh becoming more prevalent in organizations, is because organizations are reaching a threshold of complexity that makes it necessary right. So I think you go to a modern bank uh these days, if you, if you roll scroll back in time to where, um where where uh uh google starts, um with uh with borg right.

D

C

At the complexity of google at that point, and you look at the complexity of an average bank, google's going to be a lot more complex at that point than your average bank and what's happened, is that basically, google has kept continued to become more and more complex, but the banks have arrived at the same level of complexity.

C

That made google need to have something like borg to operate anymore, right right, and so that's what you're seeing is it's not like. You know it's not that the demand for things like this is artificial or if this isn't just resume driven development type stuff anymore. Some of these problems are problems that are newly introduced into organizations, because they've arrived at a certain level complexity, and the question is whether or not they can either internally adapt fast enough or if they need to ask for help operating something that is incredibly complex in a way yeah.

A

I'm gonna put one more plug in for open source and um why?

A

Because because that's who I am um and one of the things in the build versus by consideration, is when you're buying something that you just get the binary or you just get the api for, and you don't get the code um you don't get to explore and co-collaborate with the people you're buying with, and I think one of the things- and I know I'm going to probably sound biased because I at red, hat or anything but I've been like this forever, so um pre-red hat.

A

But I think one of the things that's really important in the build versus by consideration is when you're buying ensuring that you have access to the source code. You may not want to build it yourself. You know we just did an okd workshop all weekend last saturday where everybody walked through building okd4 and it wasn't fun. You know, but people didn't put their home labs in five different flavors. You know so you know these things, you know, but everybody was learning together right. So um I think one in the build versus by consideration.

A

Sometimes the conversation about um the importance of open source, or at least open code um and access to that to learn from it. um I think it needs to be emphasized a little bit heavier because you can you can log into eks or wherever, but you may never get to see the tweaks and things that they did and you don't learn from it. It's not that I want to force amazon to open source everything they do under fargate, and you know whatever it is right.

A

um It's just that that if we want to create the co-collaboration out there in the universe, that's going to drive all of this innovation forward, um we need access to that source code, which is, I think, one of the the key value points of open source, um so anyways that that's my um podium and I'm getting on it. um I I'm looking at the time here. um I would like to give aaron um the last word. So um where do you think we're going from here?

A

And the other thing I wanted to make you talk about a little bit. Is this black belt thing that you and sasha are yeah? So tell us a little bit about being a black belt. What that means in devops and where that came about.

B

uh Great, this is gonna, be awesome. This is where I explain it and then sasha says no. You got it wrong after I get off the ball, um so this is uh for for red hat. Our managed openshift black belt folks are uh part of the sort of extremely technical group of folks that help with implementation inside of the sales arc right.

B

So, on the one hand you might have like developer advocates that reach out directly to broad community, uh we get to do some of that inside of specific, like really big customers, where they have developers the size of the entire rest of the community right like they might have 2 000 developers that are available I'll, try to figure this stuff out uh speaking a little bit more targeting helps you enablement for like those groups, so I might go into a large customer, deliver a talk like this or talk about kubernetes in the future of where it's going or help them with some of the technical implementation details of like hey.

B

How do we get this working on? Openshift we've been trying to do this and it's not working. Can you point me in the right direction, so very much sort of a targeted, very similar developer advocate uh type roles, but a little bit more targeted at some of those big customers that sometimes don't get out of their own way and end up creating their own, very big gravity of culture that draws all their own developers internal. So we kind of get to jump in there and help change that landscape.

A

Did he get it right? Sasha.

D

Yeah, I think it's a good definition. It's you know what I mean like as many people as many.

C

D

But like this is a pretty good definition of what were we going for.

B

Yeah because I always try to think of like there's one definition I could tell to someone who has like organizational context of where we're at there's another definition, but I'm explaining it to a broad uh community of people, so yeah.

A

And I think you make the point by by going deep, diving in with some of these end users and customers um and having the the source code to show them how these things work. This is where this co-collaboration um comes out and and it, and that I think we're seeing more and more of this more of it in the open. More of it in you know, gigs that you guys do one-on-one with companies, but more of this is happening out in the open, and I think that's the evolution.

A

That's happening right now in open source, um and you know- obviously, um it's being you know, fed by end users by the foundations by the vendors by the partners and the cloud hosting providers. So it's really this amazing ecosystem and keeping it open um and transparent and accessible is, is key for everybody's success, so um kudos to you aaron, because that was one of the best presentations on this topic.

A

I've ever seen um someday I'm gonna make you do a keynote for me, um kukan uh in the five minutes or less you're gonna have to compact this down. So uh yeah start thinking that way.

B

Very short version of the stock.

A

They're very, very short version, so yeah, because I would love to see the reaction to um like a whole kubecon audience to vanilla, because those those are the ones that I think a lot of the folks in the room. There have a mythical belief that vanilla kubernetes is what they want um versus the reality of what, when they go back home to their organizations, what they're um able to do so, um yeah and and and again kudos to cystic and everybody else. Who does it um that this is not saying you shouldn't? Do it?

A

It's just saying you better be prepared when you do do it so absolutely yeah! So hopefully we'll get you all back on again um soon and keep talking through some of these topics and jabe and sasha. Thank you for joining us today and chris short kudos to you for making this happen and um broadcasting out to the universe and I'll post the video. And um if you go back to your resources, link, throw that to uh the last page and we will share the slides and any other resources too.

B

Yeah, let me uh so it's all: it's all pretty straightforward speaking.crazy will be where all my slide stuff ends up. uh So if you've got that this deck should be up and published with resources now, uh so you should be able to find that there and uh yeah.

A

We'll throw the video up shortly too, so, thanks again and take care, everybody have a great weekend enjoy it happy friday.

D

Youtube. Thank you.

A

A

A