Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2016 (Seattle), 16 Nov 2016

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Delivering Services at DigitalOcean by Mac Browning, DigitalOcean

Description

Delivering Services at DigitalOcean - Mac Browning, DigitalOcean

"Though kubernetes itself provides a rich collection of resources to deploy services it also comes with a bundle of complexity. This is where DOCC (DigitalOcean Control Center) comes in. This talk will explore how the DOCC abstraction simplifies the user's interface with kubernetes and how its opinionated design allows developers to focus on their services. By building DigitalOcean’s best practices into the platform, it promotes use of existing internal tools like centralized logging, dashboards, metrics, and alerting while keeping the user interface as minimal as possible. All of which is written and maintained by a small team of engineers empowered by the kubernetes API and the flexibility it offers.

This talk will cover some of the decisions the team faced and solutions to problems when marrying an existing infrastructure with kubernetes. This talk will dive in and share examples of how using code instead of convention promotes a healthier service lifecycle and helps to remove complexity. This talk will share the team's experience when building on top of the kubernetes API and what it helped the team accomplish."

About Mac Browning
I'm a senior software engineer and technical lead at DigitalOcean where I've been for 2.5 years. I've worked on nearly every piece of our stack ranging from the public API to datacenter provisioning tooling. After experiencing the high time to market caused by poor deployment tooling, I pitched the idea for the Delivery Engineering team to build internal tooling to enable delivery of the products we want quickly.

Since then the team has grown from myself and one other engineer to 5 full time engineers. After evaluating a few options we built our tooling on top of kubernetes because it gave us the most useful features out of the box. We currently manage multiple kubernetes clusters within many datacenters spread over 3 continents all of which support varying types of applications and services.

My previous speaking experience has been at public meetups hosted by DigitalOcean:
https://youtu.be/LUgE-sM5L4A
https://youtu.be/zyB185IOrz8?t=28m47s"
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Okay cool, so this is delivering services. Digitalocean, I'm, Mac, running I am a software engineer on.

B

A

Engineering team- and so you might start out by asking the question so this is Sammy by the way in case anyone's not familiar with with its solution. Sammy are full of it's a shark I'm, so delivery engineering, Hillary engineering, enables continues till three for teams to add to solution. We like to think of ourselves as building blocks for other teams, in terms of like a platform to play your software on tools to monitor and alert on your given, metrics and overall, simplifying that kind of step from the development lifecycle to production.

A

My cycle is pretty big, a pretty big leap sometimes, and so you want to really narrow that make it make it easier. It's like every good story. Arts has a villain, and so chef is the villain in case he was not familiar with chef. It's like figuration management tool. We have written chef since the beginnings of digitalocean, it's still very popular but kind of it's, it's kind of showing its age since it's been with the company since its startup.

A

You can really think of it as like that monolithic I could base at this point and so a little bit about the history. So all the time before delivery, the dark ages and so at the dark ages. Chef was probably 95% case, there's a little design school Capistrano, whatever I'm, but mostly chef, every team needed to know how to write chef in order to play your code. So this is a huge cost. In case anyone hasn't worked with a chef code base.

A

It's like it's very specific to the company, which means there's a lot of tribal knowledge, and it's also kind of really tough to to onboard onto it, and it's also really high risk. In terms of like you make the wrong change. It's touching every machine, and now things have broke in ways. You had no idea they expect.

A

um So ultimately, this meant new deploys could take days, maybe even weeks, depending on the scenario of the team and the group with chef, oh and that's just a huge cost of doing business for a start-up like dystocia and even then updates at that point could could be on the order of hours or days, depending on the risk of the update, so given kind of where we were.

A

Where do we want to go, and so, ideally we want to go somewhere where engineers can deploy early and often essentially, as soon as you start writing your application. You can deploy your application to staging development production within minutes and then from then on. It's the same process of deploys off position, one. It shouldn't be any more expensive to pull your application. The first time then just deploy it the 100th time there should be no need for tribal knowledge.

A

It's a big thing about chef is that in tribal knowledge made it really tough to onboard and so no need for tribal knowledge would be kind of top-tier, and then at that all the tooling should really encourage best practices. We've really grown and learned a lot about running software and managing its lifecycle. After you know you kind of ship it, and so we really want to make sure all new engineers are all current engineers continue to follow those best practices that we invested a lot in discovery.

A

So what do we need to get there? Obviously, since we're digital ocean, it must be nautical things it's like top tier has to be. It has to be the docker. It makes really good time to talk a lot to get some getup stars are really important for any piece of software, as we all know, and it has to be web scale, but in reality what we actually need. So the big things for us we're an active community.

A

We we knew that it was just going to be a really important part of our infrastructure, so the more people involved in that the easier it would be for us to adopt it. Have you production ready, like we're, not really investing in this to to write our own thing? We're not really investing in this. To do discovery like we want to make active change in our company, and so in that case it needs you ready for primetime or it's a very least.

A

We need to be confident enough that we're willing to put it into production- and it also has to be batteries included. Like I said we don't want to build our own thing, or at least we don't want to build steps. 1, 2, & 3. We want to focus on building steps like 8, 9 and 10, and so that's something that was really important was kind of that batteries in fitted uh flow. So obviously we're here keep done so Kuhn Aizaz was one of the I was obvious stretches.

A

um We actually spent a lot of time looking into other things and I open it kind of just settled on this, because it gave us all those check out all those boxes and was pretty, however, a little different than what a lot of companies have done, at least from a lot of talks. I've seen is we didn't adopt it such that everyone's writing? um Well files and you didn't keep CTO great. We should have our own abstraction on top of communities, and so we call this doc and dr.

A

tool for deploying containerized stateless applications, um doc actually used to mean something. At this point, it's now just a happy accident that sounds like dock, like boat docked, so I'm just gonna go with that. So doc essentially replaces QC TL for our developers I'm. The only engineer that do that have access to QC TL, our team members of my team, essentially for administering the cluster administering cluster add-ons doing the normal day-to-day of like managing a cluster and spinning up these nodes, etc. Are the other engineers that are throughout the company?

A

um Do not use coop CTL. They are not off the field, they don't. They don't have a credit for it either, but they do use doc. So doc looks a lot like cube CTL when you, when you, when you take a look at the CLI of the CLI component, which is also called doc, the doctors aren't great for it. We actually have another Google Doc that we use for the docs of the doc.

A

Doc's aren't art excellent, but it's gives you an idea like it has a lot of the same things you would expect to see in cube, CTL there's deploy so to create what you have inspect list, etc, and so maybe we'll get into a little bit of those I'll be really like a highlight another component of doc that we'll get into in a second, so you can think of doc as kind of a curated subset of the kubernetes ecosystem. This is the scalpel and of a Swiss Army knife.

A

Urbanites gives you all the features in the world. It also gives you all of the configurable for all the features in the world. I would shrink a little unwieldy. At times we focus our --get only stable, stateless services. This means there's no support for pet sets and doc or anything along those lines. We actually don't even expose mounts to users, and so we really picked a very curated subset and I'll get into a little bit more. Why later on and then you're declared to place.

A

So it's push towards the same thing like ribbon 86, or it's really important that each deploy is pretty immutable, and you know you, you know what you're getting when you when you go into it, and so we wanted to maintain that behavior going forward. So doc uses a manifest similar to the ml for community resources. It's just a lot smaller.

A

It describes your application, all the funds to make it up, name, maintainer, etc, includes add-ons things that I don't normally ship with kubernetes. So like support for the way we collect metrics at digitalocean, alerting, you can write alerts right alongside your deploy and then ingress is so we run doc and our cocoon age clusters on our cloud I would just not a currently supported communities cloud. So it is a public bug, but we don't have support for like the pod load balance.

A

There's some things like that, so we we make do our own way and part of this is kind of viewed as an add-on for our clusters and then the final I don't think it declares. A maintainer is really important, something that we believe a lot in is that all software should have. Someone could talk to you. The breaks, usually this someone shouldn't be a someone, but should be some people like a team, and so we we make sure you can declare a or every application must declare a maintainer before can actually get deployed.

A

So to take a little look at what the manifest looks like. So this is probably the simplest example of color we've done. You can see a lot of the same things you would expect to see in a pod spec or a deployment or replica controller. There's containers, there's name containers, there's image, image tag, cetera little details, the maintainer. Really! That's all! That's that's different at this point. So in practice we'll just look like it on communities resources.

A

So at this point the only thing doc creates is the deployment and then and then, after that, we let the controller manager take over and create replicas sets and pods and everything else. But at this point, like your application, is deployed after your undock deploy and give it that manifest. So if you wanted to build on that- and we say we want to expose a port so now, our our new application has an API that we want to make sure is available to everyone else inside the cluster.

A

You would then specify ports, port 3000 in this case, and we would create a service object and the service object is opinionated on how it gets named named that for your application and then, of course, at that point, the API and whatnot takes over, and you have end points and end points point to pod. So all of this relationships are managed by dock itself. It manages the labeling and annotations that are important to making all these things work.

A

This is something that our users don't have to know anything about what they they know, because we do try and share details, but ultimately they don't need to know it to do their day-to-day work, which is I, think it's important and so building on this one more time mentioned ingress is so since we don't have our supported public load balancers.

A

We have this idea of ingressing into our clusters and so again ports. Three thousand defined me an ingress I want to talk to that ingress at hello, deputy Connor, I'm gonna need a hello, be fun. It's gonna be type HTTP, since it's kind of important to tell you a little balanced what you're doing, and then you actually specify what container port you want to send it to.

A

So this is really important, because, actually at this point, if we had not said our container has port 2000 dock would have said like no, you can't deploy this application is a much friendlier feedback cycle and that you immediately know like oh whoops, I, missing important that the error message is usually pretty friendly. Not always sometimes it bleeds up from kubernetes itself, but in general it's a lot friendlier than that the manifests like cutesy tail create like we need I, don't think it's like plot against. When that's like spitting out.

A

There is like what is going on it now so anyhow, and then, if we look at what this ultimately doesn't of the hood, we create an ingress object. Name it hello, cute con and then at that point our ingress load balancers, which are we manage. Can look at that object in and reconfigure, so a couple of you might be asking like: why do this? Why not just like expose, keep CTL create and let people use that tool there's tons of documentation on?

A

Ultimately, we really wanted to target that 99% case the majority of applications that are deploying as we break apart a logs analysts that make up digital ocean. We wanted to make it very easy for you to spin up your stateless api's and then talk to you feel free to talk to your stateful databases, but otherwise, like run your run, your API on us and let's make that as easy as possible to keeping ship that early and then continue to iterate on it. um Communities is also a moving target.

A

I mean the talk yesterday where they mentioned, like 5,000 commits, and they won for, like it's just it's an insane amount of work that goes into it. It's really tough to keep up with, which is means like we didn't want to put that burden on the engineers using. Ultimately, we wanted to do it. We were okay, managing that on ourselves since we're pretty familiar with the technology and are willing to keep open bed that cycle Bob.

A

Ultimately, we didn't want to make that something all very good doesn't know about, and so on top of that, all the best practices and also are continuously evolving when to use deployment to replicate controllers or replicas sets etc, and those are just things that ultimately, at the end the day like we're, not valuable for most of our engineers and said it was just a matter of finding the best practice adopting it and sticking with it.

A

When you can and of course, like I'd mentioned communities has all the features and all the knobs for all those features, and we can get a little overwhelming the dr., fantastic they're, definitely improving.

A

But it's you know, it's just a lot to maintain, and so we didn't, we didn't want to go that necessarily ultimately decided that a smaller feature set and a simplified user experience was gonna lead to better early adopters, more early adopters, and that was really important because at digitalocean we weren't just adopting kubernetes we're also really adopting like we're adopting container sometime we're adopting I'm. This idea of kind of a cloud native application just wait architect and build the application. Scalp patient, obviously is a cloud provider where it's more very sensitive of that.

A

But there's still, you still have those bad habits, and so, when we say like we're adopting communities, it means we're adopting a lot more than just that. Ultimately, so as an application developer like why? Why would you deal with all these limitations and, like my crazy, like you should use this? Ultimately, we also get to pitch a lot of features to our users that they otherwise could use, but they may have to manage themselves or at least configure themselves like out of the box.

A

You get a support for centralized logging, metrics collection, load, balancing health checking and a couple more and so out of this lipless. Really only one of them is offered by communities itself and that's health checking. A load balancing is offered if you're on a supported public cloud, but since I said we don't have that we also are you know, we're essentially selling it as a feature of our culture as well. So that's makes a little bit easier when people like why. Why would I use you and so kind of to build?

A

On that example, centralized logging is a pretty we already having a great solution in digitalocean that another team manages that applications already talked to. So we really just wanted to reuse that we didn't want to reinvent it using like or anything along those lines. We always want to reuse.

A

What we had and so to do so did not go too far into detail, but we actually wanted to continue writing to the syslog socket for the host machine that every container is coming on and so to manage that mounting in the socket finding where to write it or write to like finding the path to it, etc.

A

It was just a lot easier to centralize that logic in dock, then to have like some copy paste block that you would put in every single deployment that you would ever want to do that to and so now, let's just that's just a feature of dock is part of it's, like extension, is it will mount in that pocket, or at least configure it? You know, have two bin, so that has this made a lot easier to have like one less thing that changed.

A

So you know, big big risk of adopting any new software is everything that changes is just a potential thing to break, and so we didn't want to really change how we manage some files, logging and said we just wanted to show like hey. We can still use what's there, which is important like we want to. We want to fit in with the existing system. It's a dock. This abstraction made a lot easier to do that.

A

Instead of sharing snippets of code, so engineers love, features, operations, love, consistency aside when I say engineers and not end operations, I want to think about it more as like the mindset, not necessarily the individuals, a lot of our engineers are very operational.

A

Focused I would consider myself a software engineer, but I really love operations so, but the engineer side- and we really enjoys when there's that like a new feature I can take I, can make it take a damage job, but the operation side of me I really enjoys when that feature is consistent with everyone else that uses it. So essentially we may all use centralized logging. So if we don't use it the same way, then it is, it adds value.

A

So, to kind of get all these sweet features, what do you need to do, or what do I need to do as an engineer, a good solution, so the big thing is really need to fit into our box. Our dock applications have some predefined. You know requirements some expectations that you're gonna have to have to fall all in line with. If that doesn't work, then that's a problem, but it makes it easier to openly build docked and then maintain the applications in production, and so why is our box?

A

Look like, of course, must be in a container it's pretty big given, but at the time that wasn't something we did. A lot of your application should be horizontally. Scalable and officers should be safe to run concurrently, so the meaning, you may say, scale one, but we may spin up to at some point, or there may be three and ideally it will eventually be one. But there's that possibility of having multiple and so that's gonna be a problem.

A

Then you need to manage walking, you know an application level or something the application should be stateless, it's pretty pretty straightforward and then crashed only ish I mean this is a you know. Big focus on software in general, but I expose a little bit more on the kubernetes and clustered software level like qubits can die, schedulers, get a little a little crazy, sometimes and then like I I may just get bored and want to go like the old machine adieu, and so all totally that shouldn't be a problem for your for your application.

A

It should it's really a feature of building software. That way so there's always gonna be snowflakes, no matter what what you work on and ultimately I think it's important to not build only for them but to optimize. For you know the 99% case. It seems like a pretty obvious sentiment, but I really feel like when we're deploying software, we try to fit every single use case.

A

We can, and ultimately that's just it may be valuable for like in conceptually but I, don't think it's practically valuable for company trying trying to grow and acquire so to give an idea of what this looks like in practice, we had an internal hackathon couple weeks ago. This was covered from teams cross country, our teams across the company, which is also technically across the country. At this point, this two-day event really about 36 hours over half the company of participated in it, and so on.

A

The dock side of things has resulted in a 154 dock, deploys for applications that otherwise weren't dock applications before it there were sixteen new applications deployed, and these are teams that weren't just engineering. This is the mixed teams from all across the business, and these are also teams that had never used dock to be enough, and so it was pretty like it.

A

It felt really good as an engineer to have that, like confidence and using this new software for ultimately something that's like just gonna sit around for a few minutes and then they're going to demo, and so all these teams really demoed. On top of bars are our platform, which is really cold sea, so getting to live of how it works, I, which probably more more relevant, walk. People in this room is so dr. pills on top of communities, best practices we make use of deployments we adopted deployments before they were in their current state.

A

We dump them in their alpha doc talks to dock server, which is it just a API, JSON API and dr. server talks to me guys their words. It's pretty simple doc server doesn't have any kind of any kind of like a datastore. So here's a diagram of that.

A

As you might imagine, all the blue are things running in pods, and so there you go doc talks to the doc server talks to the API server, and then the controllers we ingress is some of those are more than just the controllers and addresses that come with communities like the queue controller manager- that's not in the blue, that's just elsewhere, but the controllers that we have our own controllers and ingress. It's that kind of power docking and that, like I, said that load balancer, that we need so kind of a cool thing about full details.

A

Doc server is deployed with doc, just kind of bootstrapping pain, but ultimately gave us a lot of really good feedback. It was the first application deployed at the time it deployed itself and then from then on. Everyone used it to deploy. I was kind of a just, a fun relationship similar to how you might run how you might run your script on top of kubernetes and let the somewhat idea so doc server. You can kind of think of it as a split between an API server and a controller.

A

It has a little bit more logic than the API server might, but it doesn't like any stayed around, but it does act, a lot like the dataset controller, maybe or deployment Joey where, where you're it kind of marks the things that owns, and so that way we keep track of what? What dr. hood and what we've deployed as admins to picnic, do nice separation and then it also uses best like side use best practice.

A

So there's these labels for grouping which is really nice because QTL get and then pod service ingress whatever and doc app equals, and you can find all the pieces that make of a doc app, which is pretty nice for when debugging issues that might go wrong and it also means like anything. It deploys looks a lot like anything. I would deploy as an admin. So really you can't tell it's machine doing it other than it. Does it a lot better than I? Do it? It ultimately doesn't look any different.

A

I just gave you access to keep detail. You were like looking around and then, like I, said easier integration, dr. burr, private private, provided us a chance with easier integration on our unsupported cloud.

A

So, like I said, API load balancers are really big, a big thing that you need and ingress into the clusters and Doc's that are made a lot easier to be consistent about how we configure those things so kind of building on that doctor was kind of this idea of code, not convention, and so a lot of what I feel like the kind of sharing deployments and copy-pasting amo files is really that idea of you're just you're sharing. This idea like this is how you should do it I'm.

A

In reality, we know how we should do it, and so we should write that into our tool. At least we need to know so we debate some of our expectations as in how the configuring dresses or how to configure a centralized login or how does it like configure your application itself and then it's just a lot more consistent I mean it's it's hard for the API to do it wrong. Oh, it's very easy for a human to decide. They don't like doing it that way, and so, when giving them that choice.

A

Obviously, it's great to give people choices, but just know that they they may not always make the choice you expect, and so that's like, if you, if you want them to make the choice you expect, then you shouldn't have given electricity in the first place, it's how that works, and so, ultimately this is our idea of sharing patterns, and so you share patterns with the code that we write as opposed to copy pasting.

A

So this I always Haven talks to your salt and happy. You know happy-go-lucky, so hoping that's not all it's all over and those things go wrong. So doc watch is an application that I wrote one one day because we notice there are a lot of stale pods sitting around I'm, so doc box informs us when doc has failed to do this job, and so, as you can see, a teammate of mine is complaining about how noisy doc box can be so part of the reason why this this happens, though, is ultimately I.

A

Don't know if everyone knows this but keeps killed, cheats it has. It has a lot of feature disparity from on the the CLI itself.

A

In the API, it's like a known, probably no problem, but a lot of things that they're thinking about will be introduced in cubes, ETL first and then move to the, and it's just really tough, to kind of keep that the separation in mind when you're writing when you're writing code, so to kind of reconcile last life, I'm, cute, CTL, delete or destroy, or whatever the cascading feature of that is it's not a feature of the API.

A

It's a feature of QC tale itself, and so in order to delete a dock allocation, we want to cascade all these things away. But now we have to re-implement how keep CTL like you TTL does the Cascade it's kind of sort of pin, but ultimately a lot of things are getting moved server-side, so it becomes less of a plan and every day I. Just one of those things in mind. If you ever want to go down this path of writing and building on top of the API is just keep in mind.

A

It's not exactly like oopsy tale exposes and QTL. It really is kind of a tool off that we also serve a lot with. How are we going to keep it simple so like? Ultimately, we have to figure out what features are important like what is it important that you can do keep CTL logs until your logs is important to exact into your container and then ultimately, at that point like what, at what point is it more like?

A

Is it just simpler to give people cube, CTL and let them use the standard tooling, and, and that idea of like abstraction, becoming too complex, so I, don't know I, don't know, I, don't always have the answer to this, but something that we do struggle with and ask ourselves every day and I think ultimately, the the what will come will all come to it as people on board more into dock itself, on the gate or more familiar with its core principles.

A

It'll be easier to hold them into a little bit more complex, tooling, like they keep CGS stuff. So I think it is a nice like marriage between the deal so kind of the big idea for all of this is ultimately, we wanted to reduce developer decision fatigue, so they can make the decisions that really matter. I mean how you run your software is as part of your business, but ultimately it's probably not what you get paid money for it, and so any.

C

A

That goes into that that isn't like being spent on you know, iterating and building on your core product is, is ultimately like money that you're, probably losing out on and so anytime that we can, you know, simplify something, make it a little bit easier so that someone else doesn't have to think about. It. I think it's a big win overall for for the rest of the company cool, and that was how's. It.

A

Cool, so a couple minutes for questions and he's gonna ask you to grab the mic.

D

Thanks for the talk, does the the doc to support multiple tenants? In other words, is the kubernetes cluster shared between multiple users and do you use namespaces or something like that? Yeah.

A

So we don't make heavy use of namespaces at this point. I just glued on the biggest user of the kubernetes cluster is doc, but the there is an assumption that there could be other users since we're not assuming that the cluster is only used for doc, so it doesn't have built-in support meeting spaces, but it does use its own interface, and so it is as well-behaved as it can be. It also supports a lot of things.

A

I didn't demo in terms of the manifest, but it's like supports a lot of the idea of like resource requests and limits in order to be a better neighbor for everything else running there. So yeah no assumption that is the only thing running. It just happens to be the only thing running right now,.

C

It doesn't seem like there's an e plan or there doesn't seem like there's any usage of multi cluster or distributing the application. Is that something that you're thinking about yeah.

A

So when you originally started writing the doc I was just a long time ago, and so Federation was very new at that point. So even at that point, kubernetes didn't have a great story around around that idea. So we do have a support for multiple contexts, and so it looks very much like the kubernetes keeps ETL like you use context. They use context B. So it's a similar idea. It's just not a great abstraction, and so at this point it is a very manual like I'm gonna, deploy this to three different clusters.

A

So I didn't mention how we run it, but ultimately we have one kubernetes cluster, at least one per data center that we that we have at this point, and so you might want to deploy. We actually run a lot of the data center control plane or at least are moving more of it to dock applications and so yeah. That's a deploy to every data center and insulting it's very manual.

A

You do it's just like a bash for loop or something along those lines, but that is on the roadmap for like the Nexen to tackle, and so hopefully we can tackle it by just making use of the kubernetes bits. But we don't know well. Ivan I even really had a chance, looking the Federation's but I'd love to be able to just use that and build the abstraction. On top of that.

C

So you built for the 99% use case. What happens when you get the 1% yeah.

A

That's a good one, so we do get the 1%. My team actually is kind of also the 1%. We run a few stateful things on top of our kubernetes clusters, just to make the cluster management easier and really at that point, the teams that are willing to kind of go down that 1% bath are also teams that will, you feel like we can work with more directly and maybe expose keep seats, yellow to them, or you know, build it in their use case if it's not too crazy, yeah, really it's just a case-by-case basis.

A

At that point, we also like I, said we still have chef. We still have. We have a few other ways to deploy applications at digitalocean. This is not the only one. This is just the kind of focusing on a 99%. So if you need to go up past that it may be up to you, we may be able to help you just it's case by case.

E

Did you have any developers feel like you were getting in their way by putting an abstraction layer in front of keep CTL.

A

uh Yeah so Oliver, not almost none of our developers had a chance to really mess with kubernetes before, and so some of them had on their free time, but we didn't really sell it as we're going to deploy kubernetes and put it in abstraction in front of it. We just said we're going to give you this.

A

Try this out, so I think that's helped a little bit, so they don't may not know all the features that are there and so like kind of tricked them a little bit, but every day that goes by, they learn more and more in sit because a little bit tougher so yeah, it's it's kind of in it's kind of a little bit of of butting heads but I think really.

A

Where that comes into play, is we get a really good feedback when they, when they do discover, like oh I, really need to X, and so now we kind of build that into dock, or we say: okay, maybe there's more advanced use case. We can hook you up, put cube CTO and you can do it that way, so hasn't been too big of a problem. Yet I'm sure it will be as things grow and as we get more 1% use cases, but ultimately people have been pretty happy with it. So it's pretty nice so.

E

You mentioned scaling a little bit, but how do users get the scale of application horizontally? Just do you assign, like a horizontal pond autoscaler to it or to go in and tweak it turn on table when they know they're going to spike or something yeah.

A

So the current the current dock version you just have a knob and there's like a scale component, newtonian skills to five. We do support gathering metrics, so it wouldn't be a far to reach to also expose some horizontal auto-scaling for most other applications that we don't. We don't get huge surges of traffic similar to like a like a you know, a shop where you might get a ton of traffic that day that's pretty consistent of traffic every day. So we don't have that kind of big burst, and so it becomes less of a problem.

A

But yeah that's something. We definitely talked about. It's been a feature, that's pretty much been requested since the beginning, so I'm sure it'll come just a matter of picking the things that matter the most that day, yep.

B

Thanks for sharing, first of all, um do you think, there's room for the community to invest in similar tools to help others with these types of problems. Yeah.

A

So I I do I think there I think there are also some initiatives for that I mean there are some similar platforms for deploying on top of kubernetes I. Think. Ultimately, a lot of those platforms make a lot of decisions for the developers, and the problem is like we may not fit into that same box and so I think that's where it becomes really tough I think where a lot of investment could go into is the standard tooling for building those platforms like ultimately doc itself is not a ton of code? It's not very complex.

A

It's just a lot of familiarity with the cube API using at the time we wrote our own cube, API client, because the one in the library was not great, and so since I'm like they broke that out, I think that was a big step for making it easier to build. The platform is now you have this like standard API client for a place for go I, don't know about other languages. I don't really exist, but thanks cool anything else.

A

E

How do teams render stuff locally yeah.

A

It's a good question: a lot of teams use just dr. composes, so we don't expose any kind of like run your own communities locally. It is possible to spin up like a local cube, stuff and and build doc on top of it. It's not it's not a crazy abstraction such that you couldn't do that. It's just really complex and so doctrine composes if it fit that case for the most part and then building on top of that like we do offer there are.

A

There are development and staging clusters of Doc, so you can deploy your application from minute one to that up to that cluster and and mess around with it and have it be completely isolated from the rest of our production clusters like it's a separate physical communities closer as opposed to like a logical, namespace or something like that, and so it gives a little more confidence for just pushing things as needed, but yeah the local dev story is something that's kind of growing and dr. composes filled a lot of that that niche so far.

C

Hey what is one thing you wish? We had not built into dock? Oh.

A

Boy says was one of my teammates, who knows I, think I think one of the things that we missed our target on a little bit, maybe not something we shouldn't a built in, but is how you introspect your application and so there's a lot of resources that kubernetes exposes, with blogs, with events with just the overall like showing inspecting getting the the my litte self and all of the state that's contained therein.

A

I think our abstraction on top that just wasn't the right abstraction, so I think that's, that's probably one of the tougher things to build.

A

If you everything about building an abstraction on top of kubernetes is how do you introspect that the thing is working, that it will continue to work and like what it did over the last five minutes or something I think that was probably that that's probably the thing we can improve the most I think it's also where it's something like a standard library for building platforms like this I would go a long way, because it would give you that consistent view of like what the world looks like.

C

A

Alright well thank y'all I'll, be here feel free.