Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2017 (Austin), 15 Dec 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: The Elements of Kubernetes - Foundational Concepts for Apps Running on Kubernetes

Description

The Elements of Kubernetes - Foundational Concepts for Apps Running on Kubernetes [I] - Aaron Schlesinger, Microsoft Azure

“The Elements of Style” is one of the most important and foundational guidelines on how to write well. It has effectively summarized, in a list of seminal guidelines, how to harness the power of the English language to write high quality prose of almost any kind.

In computing, we have similar guides for various technologies. Python offers “The Zen Of Python”, Ruby has “The Rails Doctrine”, and so on...

One of the powers these documents wield is that they help serve as a “north star” that guides an entire community toward the same goals.

I believe we need a similar guide for Kubernetes. It would describe how app developers and operators should think about and use the features in Kubernetes to build and deploy reliable, stable apps. Armed with such a guide, we could all hope to better understand the “essence” of Kubernetes in pursuit of building better cloud native apps.

We don’t have anything like this today, but many in the Kubernetes community have strong, detailed opinions for what should go in this guide. Much of it is tribal knowledge or scattered in blog posts.

In this talk, I’ll try to bring many of these opinions together and lay out an “Elements of Kubernetes” guide for app developers and operators alike. I’ll do so by relating each “element” to stories and details I’ve seen in the community that reveal what makes a good Kubernetes and cloud native app.

About Aaron Schlesinger
Sr. Software Engineer on the Microsoft Azure Containers team where he works on Kubernetes and related projects. He’s involved with the Kubernetes project on multiple levels, most prominently as a co-lead in the Kubernetes Service-Catalog SIG. Outside of Kubernetes proper, he’s deeply involved with connecting external services to Kubernetes clusters in a manageable way.

He’s been a software developer for 10+ years, and most of that time has focused on distributed systems from large scale data warehousing to real time servers. Being a Go developer for the past 4+ years, he has distilled his knowledge of the language into the Go In 5 Minutes screencast. He’s also spoken at many conferences and events about Go and Kubernetes, as well as other technologies in the past.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

My name is Aaron I work on the containers team at Microsoft, Azure I'm also a lead on the Service Catalog special interest group. So, let's get into it first of all. Why are we all here so first of all, obviously we're at this conference because we're interested in kubernetes or cloud native technologies in general, but also specific to kubernetes. We know that kubernetes is moving blazingly fast in a bunch of different dimensions. Right, it's usage is growing day by day, like gangbusters.

A

If you follow kubernetes development, both in the core and adjacent projects, that's moving incredibly fast and increasing the number of projects both in and out of, kubernetes are exploding as well right and so basically, this speed of development. The speed of features is outpacing.

A

The knowledge that we've built in order to figure out how to use this stuff right and that's why we're here today at this talk, ok so put a little Wild West reference in there, so Clint Eastwood, you imagine, he's got like a kubernetes cluster in the saloon behind him, he's trying to figure out how to build an app for that. So what does that? Really look like ok if you were at Brandon burns his keynote at the end of the day.

A

Yesterday, you probably saw something a little bit like this, and he said what I'm about to say right. You got to figure out all this stuff, all the first two rows in order to just get your app into, let's just say, staging right, not even production. Ok, you got a new sort of new development test cycle. You got to figure out what is a container, how do I container as my app? What does that even mean? You got to figure out what a registry is and how do I push to it.

A

You got to figure out how to change your CI CD or build a brand new CI CD, so that you can test this stuff on the target platform which is kubernetes, then you've got to figure out. What is your staging prod pre prod, whatever the stuff you do before you get to prod? What does that look like and then you might wanna, you know, take a breath, but your apps and prod and you've got to figure out all the stuff in the bottom row right. How do you do monitoring in this new world?

A

How do you trace requests in this new world? How you do logging observability resilience so on and so forth? Auto-Scaling is another big one right. These are all new things in the cloud native and kubernetes world that you have to figure out. So it's no wonder that a lot of times I see people coming in to kubernetes that are super excited about it as they should be, but this is a real turnoff right.

A

It's really difficult to get started because there's this massive wall in front of you right, needless to say, there's a lot to figure out, but we've got sort of another layer to this issue to this problem. This challenge that we need to solve, which is that everybody's different right, so we can't just publish a document or have a bunch of consultants go out and tell everybody the same thing can't say this is how you do X. This is how you do dev. This is how you do tests.

A

This is what every deployment should look like and you're done right, because every organization has different needs and these are wildly different needs, so I've gone and I've talked to quite a few different organizations and seeing how different the needs are. There's things like security and topology that there's there's failover there there's disaster recovery. The list goes on right.

A

So this is why, if you read the abstract, I kind of called this talk and what I want to get out of this talk the North Star right, so this is sort of an aspirational goal. This is what we want to tell people our best practices for building a cloud native app. So this is kind of what I see is the North Star? What should we get after? We say we have a North Star for kubernetes right, so we want some kind of a document that helps app operators and app developers.

A

Those could be the same person or the same group. We want to help those people decide what to do. Should they adopt the new technology, should they use that new beta API in kubernetes? How should they scale their app and so on right? We know already that the technology is moving blazingly fast, so we want this Northstar to stand up to that speed.

A

So in a month when features XY and z are out and stable, we don't want this document to be obsolete right and then finally, one that's near and dear to my heart as a sig lead is I.

A

Want this thing and I think we all probably want this thing to guide the SIG's so that they build the right stuff for us, for our for us being app developers, app operators right and again I want to stress that I, don't think these things should be rules, I, think that should be best practices so that organizations can pick and choose what they want to bring into their app what what what kind of practices they want to bring into their app and massage them a little bit to fit their use case right.

A

Okay. So let's look at what we have now so far: okay, there's a lot of strong opinions inside of the kubernetes development community and outside right and that's a great thing. We've got a ton of evidence because people are actually using this stuff right. We see how people are using. We see the best practices that they have inside their organization and also, as I said since everybody's doing something a little bit different. We've got fragmentation, okay and that's actually a good thing.

A

I'll touch on that at the end, but in general we might see this word fragmentation and it might be. We associate it with doing garbage collection on our disk or something like that. In this case, that's not a bad thing.

A

Okay, so why am I the person to say what these best practices are? Well, I'm, not I'm, not the definitive guide, but I have seen deployments here and there all over the place right, I've, seen good deployments, I've seen bad deployments, I've seen good deployments that have some bad practices in their app I've seen vice-versa right. So this is kind of the I've seen some things style slide.

A

Ok and I am here to propose some ideas, all right, we're gonna get into the ideas right after this slide, I believe so I'm here to propose some ideas for these best practices based on what I've seen. Ok, so here's the first idea. This is all about observability.

A

So did anyone go so far in this conference to service mesh talk envoy is do something like that cool, that's kind of the so hot right now deal right. Service meshes. This is the this is the year or at least half a year of the service mesh we've got envoy, I forgot the name, but that new thing that starts with a see we've got sto, linker D, so on the list goes on right. That is a big part of the observability puzzle, but there's more right.

A

So, first of all, some obvious things on the top two little stars there. We know that kubernetes is this always-on system that always is looking at your containers, your pods and scheduling them right, and in order to do that, it has to figure out what's going on with them, are they crashing? Are they using too much memory?

A

Are they swapping so on right and that's what it needs to do is job, but in order for you to do your job, you need to observe those containers in production as well, but so time and time again, I've seen apps and specifically a few pods in the app just kind of limp along right. The keep restarting they keep using too much memory to keep pegging CPU right. You don't really know why you can't really attach a debugger to a pod.

A

Okay, you don't really know why, so you need to be proactive and what kinds of information are coming out of that pod right.

A

Okay, so, let's cover the first half kubernetes observing your app right here are some concrete things that you can do fairly simple things that you can do to tell kubernetes what your app needs so that it can observe your pods more effectively and therefore schedule your pods more effectively. Okay, so you can give resource limits in your manifests that one is a super easy win, because if you tell kubernetes I need this much CPU I need this much memory.

A

It has a lot more insight into where it can put your pod in the cluster right, even better. You can give kubernetes readiness and liveness checks, there's a little bit of implementation in your app, but that allows kubernetes to introspect your app way more deeply and that's way more powerful than just telling kubernetes statically. This is what I need all right and those things kind of enable horizontal pod, auto scaling right.

A

So if kubernetes knows exactly what's happening with your pods and it knows exactly the criteria you want to automatically scale horizontally, those pods, then you just flip a switch and kubernetes now takes over your app and now, all of a sudden, your app can tolerate, like the massive spike during Black Friday. Let's say something like that: okay, now, this is the other side of the coin. Once kubernetes is taking action on your pause, how do you know what the hell's going on right?

A

Okay, now the reason that we have this rich ecosystem and CN CF with all these awesome projects that do great observability things is because we've got this new paradigm of bunches of pods running and you don't really know what's going on inside those pods. Sometimes you don't even know how many there are right.

A

So we've got tools for logging. We've got tools, service, mesh tools. We all know about that. Right. We've got tools for tracing anybody, go to a tracing workshop. Only a few.

A

These sort of the year of tracing but lean tools to say these are all sort of tools that you should proactively. At least it's a best practice to proactively put into your deployments so that at any time, during your pod or your apps existence, you can introspect exactly what's going on. Okay, so you can see. Are there Network calls that are too slow? You can see? Is there a circuit, breaker open? Is there a service? That's down.

A

Is data getting corrupted or crazy logs coming out of one of your pause, okay, and by the way there are CN CF projects to cover each one of these things, logging, service, mesh and tracing. Ok, next idea, when all else fails crash, so has anybody heard of or written Erlang before the programming language cool? That's a lot more than I thought awesome. So Erlang has this principle called crash. Only right crash only means that if you crash your software, something goes wrong and something else will restart it.

A

So in Erlang there's this thing called a supervisor that runs above your software and it always is watching right. This might sound familiar because kubernetes kind of does the same thing, so the supervisor is always watching and the best practice in Erlang is if something goes wrong just crash and the supervisor will give you another chance because it'll restart you. Okay, now in a distributed system that runs in kubernetes things happen right, you'll probably deploy a bug at some point: your apps LifeSite, your apps lifecycle.

A

What I've learned in running apps on the cloud is that the network is always out. The network is always down. Murphy's Law will take effect in a minute. You've run your app for maybe a month. The network will go down same thing with disk. You will effectively swap after about a month at least on one of your pods okay, every deployment I've seen after a month on average that happens at least six times. I actually did the math. Ok kubernetes. Is that supervisor right?

A

It is the thing that's always on always watching and I have written, probably ten, maybe fifteen times a retry loop right, I've tried to retry a connection pool to the database. I've tried retrying connections to the data to micro, services, retry and connections to cloud services. I'm just done right writing those loops is extra logic in your app that you have to test right. It's in order to give a little piece of your app another chance to function.

A

You have to write a bunch of code and test a bunch of code, so instead of that, I've given up right, I've passed the work off to someone else, which is a best practice in software development. Right, don't do the work, don't write the code right instead, if you're writing a program, just exit one write, kubernetes will know: okay, it's not a zero status code, restart okay.

A

That is a best practice so fairly simple example here, but I've put in some pieces of observability as well. Okay, app connects to a database tries to connect to a database. It can't because the network's down, because all clouds don't have a network kubernetes sees that and it restarts it right. But in the middle here we've got tracing that sees that connection failure. Okay, we've got monitoring and log aggregation that capture those logs.

A

That say there was a connection failure and that captured that network traffic that tried to egress right and if you've got an alerting system. It's watching how many of those restarts happen in a given time window and it might alert you if you've got over some watermark right, it might be super sophisticated, it might have sliding windows and moving averages and so on.

A

Okay, next idea, unordered is better than ordered right, so has anybody done any investigation into what ordering means in a distributed system? One: okay, yeah? It's kind of he said it's impossible, so sorta yeah, it's pretty close to impossible. There's been so much research, that's gone into ordering and distributed systems that there are several kind of sub definitions for ordering and distributed systems. So if your app needs to rely on some kind of ordering in kubernetes in your app, you have to first decide which kind of sub definition you want.

A

Then you have to implement it, which is no small feat. Then you have to test it to make sure that you adhere to that definition of ordering right. That's hard. Ordering is hard and distributing systems I made that point bold on purpose right, so again, don't do it. If you can try not to do it. Okay, now, that's not realistic for all apps, of course, especially datastore apps.

A

You might not be able to just say: I'm, not gonna do ordering, but if you do, kubernetes has built-in primitives that are really well tested to do that ordering for you right so again, Brendan's talk yesterday, he talked about using a sidecar to do distributed, locking and distributed leader election okay and I'll provide links to that afterwards. Right use that if you need to do ordering, if you need to lock, do mutual exclusion, you need to have a leader be elected right.

A

Those are ordering problems, use the sidecar, but you don't have to write any code that way another one that I've used pretty extensively for preventing race conditions is using resource versions on the kubernetes resources themselves. Right. Has anyone seen that field on the all the resources in the object metadata? It's a resource version, ok, well, I'm, here to tell you that it's wonderful! Ok, if you've got ten pods running and you need to manage some data, all the pods pull down that resource.

A

They all have resource version, zero, let's say right and then one of them does the put the upload of the new data, and now the resource version globally is one. Now all nine other pots try to do the put with resource version zero and they all fail right. So they all race and that's not always the best thing, but there's no data corruption, because the first one just wins and the second ones are left out in the cold. Alright. So that can be very useful.

A

Depending on your app that's a tool in your tool belt, you don't have to implement okay and then the last one is init containers, okay, so init containers- if you don't know they basically run before your pod and they do some initialization logics, but they can be used to make sure that something else is running before your thing right. So, for example, anybody know that there's an open source, Microsoft sequel server. By the way there is it's in docker hub right: it's a sequel, server!

A

Community Edition and on a 7, node, Azure container service cluster I work for Microsoft, we get free ones. So, on a seven note, add your container service cluster. It takes about six minutes to startup in the common case, with no other pods in the cluster, all right six minutes. So by that time, if you have an app that is relying on that database, it's already in crash loop backup, and then you have to wait like another six minutes or ten minutes or something to have it start up.

A

So the solution is having a knit container in your app that keeps pinging the database and just trying to see are you up? Are you up? Are you up? It's like your kid poking you on the shoulder at 6:00 in the morning to see if you're awake right. Eventually the database is awake and then your app starts right up. Okay, so that's ordering right! You are well! You are depending on something to be available before you can be available.

A

Now it's important to note that init containers can be abused horribly. If you find yourself writing a huge script or writing some big program that executes in an init container, that might be a recipe for disaster, but it's really up to your use case. Right again, this is a best practice, not a rule, all right. The next concept here and the next idea is that loose coupling is better than tight coupling.

A

Now this is taken right out of like a CS 102 book right software development, tight coupled code is hard to evolve hard to refactor hard to test all right. This means something similar in kubernetes. Okay, now that first bullet point I've said before and I will probably I, think I'll say it two more times. Kubernetes is always on, and it's always watching. It's a dynamic system that has basically undefined behavior over a large period of time, and that's a good thing for you, because then you don't have to wake up at 3 a.m.

A

and then you'll have undefined behavior, because you're going to be tired and not have had your coffee, okay, but the other side of that is that your app has to tolerate that dynamic environment. Okay. What does that really mean on the ground right?

A

Here's the first best practice and if you take nothing away from this talk, nothing else away from this talk. Take this way, please do not talk from one pod directly to another pod right, because eventually Murphy's Law again in action here, the other pod.

A

The target pod is going to go down over a month two months whatever it is, so instead communicate between paws using a kubernetes service or servus mash, or something similar right, a proxy of some time right now, going back to crash only sometimes your other service, your target micro service is gonna, be down or unavailable or, like I said, the network is going to be down all right crash and just have kubernetes and make her do it.

A

Retry right so I have seen in one of the apps that I used to administer at a previous job on kubernetes it had a 700 line. Retry loop. There was a bunch of logic in the loop to do things like exponential back-off and custom log and custom stats and metrics and stuff when instead, that could have been erased and replaced by OS exit 1 right so back to crash only I'm, a huge proponent of that ok.

A

Now, finally, I've seen a couple of apps that use the kubernetes api to look up resources right. That's not an uncommon thing to do this particular one looked up, pods right tried to get a list of pods and then iterate the list and then talk to each one. So that's already violating point number one, because it was talking directly to pods, but there's another issue there right if I loop through a bunch of pods and then do something with them.

A

The loop is probably already expired right eventually over another month or so of writing your app you're gonna get the list and before you can do anything to that list, one of the pods is gone. Ok, so, instead of operating on individual resources, a better practice would be to operate on labels right so get all of the pods.

A

All of these services, although whatever that have a specific label- and that way the list will always be up to date, because kubernetes will aggregate all those things together that conform to that label and give you it back.

A

Ok, counterpoint. The next idea is that tight coupling isn't always wrong. There is a specific way for you to do tight coupling in kubernetes right, and that is with the pod abstraction itself right. The pod, not the container, but the pod is the atomic smallest unit of deployment in kubernetes, and it has more than one container in it or it can have one or more containers in it on purpose.

A

Okay, and that is because, when Google originally designed this concept, they recognized that apps, or at least individual pieces of an app might need to have multiple things sitting together. Right and this is this can be the case for a legacy or big monolithic app, but can also be the case for a very, very modern, app right and we've seen some of those things at this conference already envoy is one service meshes write.

A

The proxy generally runs on localhost, which means you've got your app container and you've got your proxy container and your app container talks to the proxy on localhost all right. Those two things would run in the same pot fluent deep another CN CF project. There is a fluent D logging driver that also runs on localhost. You send your logs to it and then it aggregates them before it forwards them down the chain right and most metric systems as well. Most metric systems will aggregate metrics sample them and then send them on same thing.

A

Your app would talk to that metric system via localhost and you'd. Have the metric system be a sidecar and that's what this pattern is called for? All three of these cases and more, it's called a sidecar. Anybody heard that term before sidecar yeah sweet, you guys are great, alright. Next and most near and dear to my heart, this idea called record. Your configuration is very important. It's probably one of the most important best practices that I can convey today.

A

Ok, so here's I think my last question of the crowd. I'm doing this, keep you guys awake, who has heard of a declarative, API.

A

Ok for those who haven't who has used sequel come on I know, everybody is used, sequel who's talked to a database before okay, you guys, you guys, can raise your hands. It's ok, okay, so sequel! Basically, you tell the database what data you want back. You tell it how to group it. You tell it how to sort it or you give it some data- and you say, put it in this row in this database or you say, update it this way or delete it right.

A

You do crowd operations, but the important bit there is that you don't tell the database what exactly it should do. You don't tell it how to do that right and that abstraction is super powerful, because sequel has scaled from an in-memory sequel, Lite database to massive databases in the cloud that have hundreds of nodes, thousands or hundreds of thousands of nodes right and the kubernetes api is the same.

A

You tell it what the end state should be, and that has and then kubernetes figures out how to do it behind the scenes and that has scaled from mini cube up to 5,000 plus node clusters right. So why did I tell you all that stuff?

A

The bottom line is that you should just keep your manifests in your VCS in github, in whatever else right just keep them there and make those represent the latest working configuration of your app, and that means that at anytime you can submit those to kubernetes and it'll, take the current state of a cluster and make it into your apps working state right and that action is called a reconciliation.

A

How do you do that? Well, I'm, a huge proponent of how people heard of Hallam before cool yeah, so helm is basically it does this right. You have this thing called a chart. It's a group of the manifests that describe your app and then with one command, doing a helm, install it'll, send those things up to the cluster and it'll make the cluster reconcile from the current state all the way up to the state that you want your app to be in right.

A

Okay, second to last I believe ask for the least I'm gonna go through this one, a little bit more quickly. This is very similar to the principle of least privilege, okay.

A

So what we're talking about here in kubernetes is that you can ask who minetti's for a ton of stuff. You can ask it for permissions. You can ask it for storage, disk memory, CPU, compute, etc. Right. The point here is that you should leave as much of that for kubernetes to manage as possible right because the more you leave to kubernetes them, the less you take for your app, the more kubernetes has in flexibility to schedule your app later right and that translates to fewer wake-up calls and 3:00 in the morning and so forth.

A

Ok, so here's some examples of what that might look like I'm, not gonna, go through them here, but the point is that you should basically take your app or components of your app and reduce its resource usage or permissions as much as possible a little bit more, so that your app can't run and then increase them a little bit so that you basically have the least amount of resources dedicated to your app at one time.

A

Ok, last point: now this one is basically don't repeat yourself: ok again back to this theme of don't do more work, then you have to write. Build on the shoulders of giants means that there is a ton of code. There are a ton of man-hours there's a ton of research that is gone into building kubernetes right, there's a ton of infrastructure to that tests, kubernetes and the api of kubernetes is publicly accessible, of course, at least when you're inside your cluster and it abstract away tons of functionality of distributed systems functionality.

A

That is super hard to get right, including ordering ok. So, whenever possible, call the API, if you find yourself building some complicated distributed system inside of kubernetes, try to just call the API instead to do what you need and the case that it doesn't do what you need. Maybe the community has something maybe CN CF dot. Io has a project or somewhere else in the community has a project, and in the case where you don't see something that fits your exact needs, maybe try to build something. On top. Ok, it's as simple as that right.

A

Here's some examples, I'm a big fan of helm, of course, but if you can't use helm directly, maybe use the helm, API right, I'm a huge fan of traffic right, it's a great ingress controller, but maybe if you can't use traffic directly, maybe put a service mesh or a proxy in between traffic and your app right and then fluent D right. We talked about flu and D already so.

A

I have used fluent D to send 1 million log messages in right and I did that over a period of an hour and I did that it was before Microsoft. It was on a gke cluster and it crashed the cluster. It was a three under node cluster. It's crashed it right. So fluent D is pretty good right.

A

It's probably better than something that you can write in a weekend and you probably don't want to write it in a weekend, because you've got an app to bill right and the same goes for the others too, and plenty of the other projects that are in the CNCs.

A

So some parting thoughts. Where should we go from here right? I gave some ideas. I threw some things out here to start a conversation right, but, as I said at the beginning, we've got that fragmentation right and what that really means is that we all have a wealth of experience and a wealth of ideas on what a good app should look like a cloud native good kubernetes app should look like, so we need to share that, and this is a call to action.

A

Ok, I've put a description of all these ideas into that github repository, so I would encourage you to go into that repository, submit a pull request submitted an issue do whatever it is that you want to contribute up to that repository to add your ideas right and it might look like a new best practice. It might look like fixing a typo either way totally cool right, but I want to solicit your ideas to begin compiling this list of best practices for a cloud native app. Ok, so with that I want to say.

A

Thank you very much. I really appreciate you coming there's my contact information up there and I think we've got about five or so minutes for questions. So thank you very much.

A

Yeah, so the question was the crash: only kind of architecture, if we've got too many pods crashing at once, is that gonna overload kubernetes. So the answer to that. Well, that's a very good question. The answer to that is, probably not. You might have a million pause running and, of course, that's a corner case. That's an edge case if your cluster is too small, but if your cluster is big and you've got a million pause running on it.

A

That's okay, because the kubernetes api isn't gonna get overloaded by that, because the cubelets are gonna, try the restarts first right, so that restart logic is distributed across all the nodes and not all in a single point of failure. Does that answer quick cool yeah.

A

A

A

So I missed the very last part. I know that you were asking basically about thundering herd yeah.

A

No actually I, don't because the supervisor process he's kind of know about ordering and dependencies kubernetes. Of course, doesn't so, but I think your question was kind of about. How do you avoid that thundering herd when a downstream dependency goes down, then all the upstream dependences also go down? Is that about right, yeah.

A

Yep yep yeah and that's a very tough problem of course. So what I would say is the the current state-of-the-art kubernetes is primitive right if you crash, if your database crashes and everything above it is gonna crash, of course, that's right. However, when you bring it back up and that might be a manual process to bring it back up. But when you do bring it back up, you can bring it up slowly in a different namespace right, and that would mean that not everybody is gonna, go and talk to the database and thundering herd.

A

Basically Dass your database right. If you do bring it up in another name space, you can use that service masher. You can manually route just a trickle of application traffic over to the new deployment and thereby at least avoid that thundering herd problem or the das problem later right. So you still have that cascading crash. But when it comes back up you don't necessarily have the the I call it the das problem right.

A

You don't necessarily have that so half the problem to solve with possibly some manual manual steps, but you could also automate that too. We just don't have a good mechanism for doing that. Yet right, maybe you can put one up on github yeah.

B

A

Well, you're, absolutely right! If the question the comment, wife's last question was about is kubernetes gonna have some sort of ordering primitive in deployments or pause in the future. I've probably only seen the same issue, the github issue that you've seen I, don't know the status of that, but I do know that it's not gonna, be in one nine. That's all I can tell you. Is it alpha right now: okay, okay, scheduling, priority yeah! Thank you any other quick.

A

Well, yes, and no yeah exactly so if, if Internet container had some kind of an exponential back-off that might work, but if you bring up your app, let's say: you've got upstream dependencies. They're gonna take that much longer to come up. If you go an automated approach, so it still might be better for you to have some outside automated tool or take some manual steps to kind of manually bring your app up because it may happen faster may get back on line faster. Then, if you just tried to let kubernetes do its thing.

A

Other questions yeah.

A

Yeah yeah, so the question was: how do you configure your app at the App layer to do things like connect databases connect to your queue? Whatever else is that about yeah? So right now, we've got this thing that one of my colleagues is called. The post-it note flow right. That's where your operator provisions a database by clicking a button on the cloud console and then gives you the credentials on a post-it and then that's it. That's what you have and you have to figure out how to get it into your app.

A

This is a little bit of a shameless plug for the the sig that I work in it's called Service Catalog and the point of service catalog. The reason it exists is to solve that exact problem, so that what you would do is, instead of embedding credentials into your app. You would embed a new kubernetes resource, called an instance into your app and the instance would be the instance.

A

Resource would be in charge of going out hitting a cloud resource, for example, creating the database and then dumping the database credentials into a secret and then, instead of having your credentials baked into your app config, you bacon the binding of the secret to your app config. Does that make some sense yeah? So if you, if you Google, Service Catalog kubernetes it'll pop up, you can kind of see how it works and way more detail.

A

Of course, yeah cool I think we are at time, so I'm gonna be around feel free to ask your questions offline. So thanks again everybody.