GitLab Delivery Team, 27 Jun 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Delivery: Run through registry gke setup 2019-06-27

Description

Part of https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/70

A

I think it's a great milestone that you enabled registry deployment or other registry running on staging, which is pretty amazing in my opinion. But what I want to see from from this discussion? Is you running me through the whole? You know all the repositories that we have, how things are being deployed where I need to look for things right like what kind of things you did to get the roof on a dashboard created, not the graph and a dashboard itself right, like everything under it to serve that profundo, dashboard and I.

A

Basically just run me through the whole new infra for this purpose, and what I want to know also is version changes. What do we do, but how do we right now? I know it's manual right, but I want to just discuss with you guys like. How does this look to see how we can take the next step with Cassandra?

A

Does that make sense for I'm asking too much I think.

B

That makes perfect sense, I, don't think you're asking too much I think if anything you're asking too little. Oh.

A

Okay, what do you need like what we have more 90 I.

B

Guess we'll find out, as you start asking questions and get concerned when you find the red flags that show up as yeah, because this stuff yeah.

A

Sounds good and I'm kind of expecting both you and Jeff to lead this conversation because you work together on it.

B

A

C

I can start and then how can I take the first part skarbek and then you can take over. So let me just share my desktop.

C

So hopefully you can see that first thing we'll show you is, we obviously have mirrors of various projects on obstacle. Nets are going to look at the gala. Calm projects are so we have the Kate wear clothes group, which is where we're going to have repositories projects for every chart that we installed on the gke cluster. So we have one GK cluster per environment and then we have one project per chart. That's sort of the way we have it structured. Now they may change later we're only installing two charts.

C

We right now we have the monitoring charge, which is Prometheus operator for Prometheus metrics, and then we have to get that chart which is called get lab. Calm.

C

They both of these projects, use a single image in this common project in the registry. So we have a pipeline here that whenever you make a change to either the common CI config or we have some common scripts, whenever you make a change, it builds a new image. Then the other projects use the latest image show.

A

Me show me: tags, show me registry in that project. Sure.

C

A

How we are using latest okay, so.

C

I guess we we I think we go back and forth on this right now we're using late. It's just out of convenience. We can. We can create a tag but I think I'd like to wait. Until things are a little bit a little bit more stable, then maybe it will create like a version one tag and then we can bump the versions as.

A

Long as we think about the system where you know one change is not gonna unlock a bunch of things, so we need to have this version, but no it's good. While we are prototyping and moving fine yeah.

C

So so the base image, basically installs, helm installs, you know, keeps ETL stalls. Some other packages show.

A

Me the show me the image or the docker file sure.

C

So basically, this installs local tiller for helm, installs, Howell, installs, cubes, ETL, installs, QV Val.

C

You know and some other things we actually have this image being built on both Gila calm and because we have pipelines that run on get low calm and we have pipelines I run on ops, the ones that run on Gilad comm or more for just like checking to make sure everything is ok. The pipeline's that run on ops are for actually deploying an update, mm-hmm.

C

So I'm gonna give you a quick on the left here in this terminal I'm in get led comm. We have some helper scripts. These may these may change, but basically these are just small lightweight wrappers for install list removed, templates and upgrade. These are the commands, I think I'm, probably gonna or I, think we might just collapse ease into a single script. You can see that they're fairly short, they just source some common scripts.

C

If I do like list, you can either run easy, they're getting run in CI or locally, and you basically give it an environment when it's run in CI. The environment is derived from the pipeline environment, but since we're running it locally, you have to specify it. So if I do this, what it does is it fetches? The common functions sources then run some pretext. Make sure everything you have.

C

Everything is installed locally, that you need it automatically switches, your cube, CTL context using cube, CTL CTX, that's only happens locally, since we don't need to do that in pipelines, but it's quite handy and you can see for the helm list. We have both give and get lab monitoring, and then we can also do something like upgrade environment G stage.

C

What that's going to do is since we're in the give up comm project, it's going to clone the charts repo and it's going to do a helm upgrade using the values files that we have set for the pre, prod environment right.

A

Ok, problem number two I'm writing down your pulling from charts, get labelled yo, and that is it lepage's, meaning get LePage's outage because they're in deployment right.

C

A

Fine now it's like I'm, not this is not production, ready, a threat. We are getting there so so.

C

This was just a helm upgrade so let's say: if we had a configuration change, what we would do is we have values, dot e mo and G stage. Sahih mo will have one next environment, perfect.

A

C

This is like the global, like you have a bunch of stuff here. You can see that, like a lot of things are disabled because we're only running registry right now and then this is the custom values for sturgeon.

C

So if I just changed like I, could just change this to foo and doing that. If I did another, you know upgrade, then it would apply that change. Another option is: is that I can commit this change on a branch.

C

So, let's knit the merge request here so on github.com there really isn't much going on here. We just have shell check for like these shell, wrappers I think what we can also do is some more checking with the suborder Starbuck discovered that he'll he'll upgrade has a dry run mode, but it scarred that correct me if I'm wrong, but you said it's pretty much useless right. It.

B

Doesn't provide me any information that tells me whether or not the changes that I'm introducing are actually none of the changes, I'm introducing or being shown so like if we added the alert manager stuff, none of that showed up in the dry run so.

C

That kind of sucks, but we could in theory at least I'd, have had a dry run stage to give a calm. It would require us having credentials on galeb comm, which are currently over only on ops, but we could do that but anyway. So now the pipeline does the check here and then we can go over to ops and then I'll go to lab I'm gonna have to go to my branch.

C

And hey Marian actually worked, that's good, so you can see here that there are two deploy jobs. One for dr and one for staging I thought are these: the dry runs skarbek or these actual these.

B

Are just the dry runs because.

A

Enduring and branches right, yes,.

C

Because we're on a branch, these are the dry runs. Then, when we merge them master there'll be a manual job to actually apply it. Another cool thing that we have well.

A

Well, what's the reason for for manual in when you merge faster because.

C

I'm, cautious right now, I, don't know that's for now. Maybe we should just make a bottom I'm, not sure if we want to do that, we.

B

Don't have it set for manual right now, oh I,.

C

Thought I'd had it set for manual at one point.

B

C

B

Master, it just performs an upgrade, as is showed.

A

The master one by plan on master.

C

A

I see what what happens there.

A

Yeah yeah, it upgrades so.

C

These are the dry runs yeah, so so upgrades.

A

Automatically open one yeah sure.

C

So basically, it's just running and.

D

We second adjust to run the upgrade after the dry run, and you know in other stage otherwise they're just concurrently doing the same thing. Yeah.

C

That's a good point also, you probably want to dry around the pass before you actually do. The upgrade, though they are the dry run, is run on the branch right.

A

But you can merge multiple things. Yes,.

C

Are you taking? Can you take a note for that mare and, if you're.

A

Yeah I am taking.

D

C

So so that's pretty much all we have as far as like the CI automation. Another neat thing we can do here is there's a wrapper called template and it has the same option.

C

What this will do is extract all of the configuration into a manifest, a manifest directory which is get ignored, and that allows you to kind of see everything that the home chart is configuring.

C

So you can see it wrote a bunch of files. Now we have this manifests, and in here we have all of the generated llamó files, but these aren't won't to be committed to get they're just ignored, but they're useful when you're working locally I find like my workflow has been like. Okay I need a new configuration thing in the chart. I make a branch.

C

The charts, repo I, make a change, then I, there's an environment variable that you can say like, instead of using master of charts, use a branch name, and so then I can generate the config like this check it out to make sure it looks. Okay, okay, it looks okay, then I can do the upgrade with my branch. If I want to or wait till it gets merge to master the.

D

Jar I have another question, this proper suite that you show our switch. Your keep CTL context right. So if this is after the thing, I'm quite sure at least it was job before that the minor is has two option for setting the context only for the specific run so that you don't have the environment that.

C

Would be really good yet this was kind of a hack I put in a while ago. I think, like yeah I mean it kind of sucks to you because, like it changes your context in your current shell, you know and.

D

You forget, and that means something on production yeah.

C

So film has that option that would be good to do yeah unless.

A

You I pasted the docking in our channel. Could you please add that.

C

C

Think, like I'm, not sure where to what to touch on maps, do you want to look at the monitoring config, or do you want to look at the registry?

C

A

You just humor me here and can you down rate registry on staging to an older version, I, don't.

C

Think that's easy to do. Should.

A

Be easy to do you need to replace the image that.

C

Yeah so I'm not sure how to do that. Let's.

A

Let's try together we'll figure it out. I, don't remember as well haven't deployed in a while. So.

C

Is this something beautiful to set in the church in.

A

In values or in G, staging I guess because this is where you're overriding right so I think you will need to open up a different yeah open up a window and go to a registry. Chart I think we specify image to be pulled from I.

A

D

Remember you need to open it as well.

B

It's the third one from the bottom need uh-huh.

C

The camelcase threw me off and then charts.

C

And registry and then values.

C

B

Tag is what we need to set, probably.

C

Does anyone know what the last version was I think.

B

Two seven zeros are safe man.

A

C

C

Let's try it so we'll just do.

C

My dry run first mm-hmm.

A

Actually I'm super happy. You are doing this jeraf locally as well. I've been trying to figure out like whether it's gonna be like super difficult to do or not like local development. Theoretically, you can just set up a different cluster connect to it, cube City.

C

A

You can do your changes there.

C

That's what I! That's what I'm thinking and thank you that over.

A

C

Like Medi cube, like maybe just setting up a like having the pipeline for.

A

For now, let's do it that way, because I don't don't want to go into. Oh it's on my machine. I. Don't have this dependency installed properly, so.

C

This is probably not gonna. Tell us anything. Oh.

B

No, it won't like the best way to get this information is to probably do a template and look at the look at the deployment. Config can.

C

We query what version if I just can, can we ask where this tree was what version it is now yeah.

B

Just do a get on the either deploy or the pods that are running you.

A

Can yeah you can so exiting to it in registry version, mm-hmm.

C

A

The pods you want, okay,.

C

C

A

You need you, deploy your staging yeah.

C

It's deployed on staging okay, like we can do, okay, get pods or get mad, then we'll see it.

B

Just do a get on one of the specific pot names.

E

B

Our skin then put the flag in llamo. Sorry like.

C

B

And someone there, we should see the image that's being used and we.

A

Should own some crap yeah.

B

D

It's 271 I think. Is it possible yeah.

B

Great yeah, if you grab her image, yeah, okay,.

C

So, let's try to let's try the downgrade then. Yes,.

A

If you are downgrading, we should probably call this change. Slash beans. Last change, instead of.

E

B

That pod should not exist if we change the version of his earnings right.

C

B

Fact that you're getting data back worries me that nothing ever well.

A

It takes time to roll it right, so, let's not give up just yet the Prussians.

D

A

D

You crash like the new one, so you.

A

Know effect, let's debug bit, let's see what's happening so.

C

A

Something yeah.

C

Can you do a que describe.

C

Pods, maybe and then give.

B

It the pod name, yeah yeah,.

C

Let's try this get.

B

The name space.

B

Glad to run the log command to figure out, what's going on yeah.

C

I, don't remember the index for that. So that's like K logs.

B

And give it the name of the pod.

A

And the name and then namespace yeah.

A

Storage I was not registered, GCSE I think this was a blog in two-seven-zero. Oh great.

C

A

Let me let me just double check.

A

Storage Rock night I.

A

Think there was a bug yeah exactly two seven zero GCSE storage driver not registered, which means we downgraded she's very successful.

A

D

A

Is the output doc, here's the actual bug and.

C

I assume them that's. The registry store running right because you still have other pods. It's.

A

Serving still to seven yeah.

C

Exactly yeah great.

D

Alright, that works.

D

So with this approach, we are never using the helm rollback feature. We are just going on a next version. Exhibition exhibition, yeah, okay, helm, rollback.

A

As far as I remember, when I worked on this last year and had a lot of trouble, a lot of problems like a lot of things would stay just in an undefined state or it wouldn't clean up correctly or it was just a mess.

A

I, don't know whether like they've improved it a lot I think since then, but back then it was just a mess. Okay, we should probably just play with that a bit as well see how what we can leverage from it all right. That is great. Can you horsecar back? Can you run me true.

A

Logs are long should be showing this error somewhere right. Oh.

C

That's a good question.

C

Did we set up logging yet on staging skarbek I set.

B

It up as similar as possible to the pre environment, but I did not validate that logging was working when I completed. That must.

A

B

A

Run a straight.

C

So this is for pre, prod I'll just need to add the index.

C

It's not a good sign, yeah I, don't think.

C

This is for pre yeah, so yeah we don't have box, we could, maybe maybe we could get them in stackdriver. Let's just take a look.

A

How much work is it to edit now I.

B

Would need to work with jarv. He did the primary portion of that part, so I'm not really sure. What's going on wrong.

A

Okay, let's see, stick driver right now and then I wouldn't mind sitting through the two of you doing. This live on camera.

A

So the point of this call by the way.

C

So I think it's.

A

Registration yeah.

C

So, unfortunately, the log is extremely chatty. These are the health checks.

C

There it is so yeah I mean there's, there's really nothing special here, because we have what we do is we can figure, what's called a log sink in stackdriver which just Ford's it directly to pub/sub, which then gets picked off and forwarded to elasticsearch.

C

So I had pretty high confidence one week once we got the you know, the pub something working again or working for staging then should have logs okay.

A

Can you show me good monitoring pick any of this up like we had a failure? One bot failed I, don't.

C

I, don't think so, because we had nothing. That's monitoring.

C

B

Have to keep state metrics is enabled, so I'd have to relearn how to query it, but we should be able to find the fact that a pod was sitting in a crash loop for a period of time. That.

A

Is pretty much necessary at this year.

E

C

Just used a like a default dashboard for kubernetes and I, don't know.

C

C

Think we'll need to take a look but yeah I think this is something we should figure out. Yeah.

D

Maybe we need some kind of check for what we world in them in the chart and what is actually deployed because the service was surrounding, we had to pout serving, but was not a great version. Yeah.

A

And I know we were gonna, see a lot of those.

C

A

So I think for me for me honestly, monitoring wise, that's a minimum before we even consider sending any traffic to this. No because we are running something very, very much she knew and we are using like several layers of stuff. So luckily, for us registry is not changing that fast or rather I. Don't know whether it's lucky but let's say the versions. We are not bumping that often so we have time to play with this, but would like to see it and.

C

Luckily, we know how to make the registry crash.

D

E

C

Us for it alright.

A

It is a half past. Our start. I am curious to hear if there is anything else that you would like to share, skarbek or maybe yeah.

C

I'll: stop sharing sky pack. If you'd like to talk a bit.

B

Do you want me to roll through some of the like the terraforming related material.

A

B

Some of this you've seen before so I, don't know if this is going to be repetitive or what.

A

If this is just you showing me at the beginning that we had the discussion about how the clusters are provisions that we don't need to do right now, yeah.

B

That was what I was going to show. So me I, know.

A

That we already had that so, let's move on all right I saw.

A

Maybe talk about how would we upgrade our charts automatically right now? It is whatever is currently in in this project right and then we pull. If I understand correctly, we have github.com were close project right and that pulls from charts. Whenever we make a change, what version does it pull of charts the terrible.

B

Drug wars I had inside of the.

C

We currently we're just like cloning, the master branch of the charts, repo. Okay, we shouldn't do this. You shouldn't use like the release version. We just well.

A

C

Unfortunately,.

A

C

My own get lot of things are very um yeah like yeah, we're I mean like we're constantly making updates and charts right now. So we need the latest bleeding that stuff yeah.

A

We do because compared to UM compared to omnibus it's just a new project right like how many of us has years stabilization prior.

B

To using the helm tiller, we the script that I created would clone a specific we've got a file called get lab, charts version sitting in that repo right now. That file is just kind of sitting there. It's not being used anymore.

B

When we switch the home tiller, we just stopped looking at that particular version file, so we could probably quickly add that capability.

A

Okay, so let me, let me see if I understand this inside of github home were close were close project. We just pulled the latest held charts master and any change that is merged into the stable branch we'll just deploy to certain environments, depending on where we're going with it and.

A

Pretty much it like as as simple as that, how would we not do anything as in we thought to deploy what we are doing is we are basically not doing much ourselves. We are depending on other projects. What I'm thinking about here is: if we think that helm charts are going to be the source of truth for us, how do we ensure that any change that happens inside of that repository propagates to our environments? And, let's say for the sake of argument- it wouldn't go to staging it?

A

Could it could go to any test environment that we create so say we have we propagate change through two or three environments before we even get to staging, but do that automatically without us, touching anything and depending on the developers to to make changes and make tests and get the dashboards use them before we even come into play on? How are we going to deploy to staging pre and prod.

A

Am I making any sense your.

B

Questions clear is something that I have not given any salt to in the past. Right now, like I mentioned before, the templates was using a specific version. That was a file that it would read to figure out what version to pull. We would have to think about how we want to approach that in the future.

A

Any ideas on like from prior lives not only not not necessarily like experience, but the things that you've read herds are good to do.

D

Murray, but you are talking about doing something like triggered pipeline for propagating from the charts repo to a real environment, sure that can be one option: okay, so.

A

The thing is what I'm trying to think about right now and what's kind of keeping me up at night is I. Don't want to repeat of an omnibus situation in in our infrastructure right I! Don't want like this big bulky thing, just dumping.

A

Everything on us I want to have a framework where you know, for example, we have several layers say if the helm chart is stable enough, I want to have all of our different components, say get early say, get lab shell, workhorse and whatever be built by developers in their own pipelines in their own little world. Right and I want that. We have enough enough stable environments before staging that any time they push things get rolled into these environments, so they can test changes all they want.

A

And then we have a check point on staging where we're going to say everything that you did in the past. I don't know a day or so is now going to be deployed in bulk on staging and that's gonna roll through the rest of the environments and all of that somehow automatically automatically done. Hopefully, we get loved features as well, so that we can dog food properly.

C

Okay, I'm sorry go ahead of.

D

Us feel, okay, thank you job, so I'm still a bit confused because if we think about the charts, so this means that I'm I am working on some change on the charts. I merge this and then I trigger a pipeline would say- and we go through this several environments which say- and maybe then we have a gate before reaching staging. But someone has to push about them or approve it.

D

So, okay, it's clear, I, don't get how this work before, if you think about developers, X funding, a features sink usually, for instance, because this is Wade for building a noun chart. So how can a developer that just is just working a feature get this into a real environment.

D

A

They would, they would also ensure like if there is any backing that needs to happen from the home charts I'd right like if the feature depends on a configuration and inside of the chart, it's similar to what we have in omnibus right now like they need to submit a change in omnibus, but they will also need to think about how do they ensure that it? This is in a backwards compatible way. So, whatever change they made in in in omnibus last charts, let's say in charts that is gonna land in all of the environments.

A

At the same time, helm is going to be just the like. The charts are going to be serving they're going to be the plate. I want to see like whether we can make sure that the developers by building their own images can just do use the charts that way so sure you're not going to be able to deploy a change. If you don't have the backing of the chart, but you can do that in the chart. First get that role to the environments.

A

Everything is fine, and then you roll your change with with your new image from your project. Yeah.

D

Because the thing I was thinking about is that if we think about the review apps, okay- and we built on top of this concept, so we can have a branch on github see whatever it is, introduce a feature. And then you have a branch on the charts that build the configuration for this or even just change the building image from the custom branch. And then you can deploy this somewhere. But this has to happen before someone hit merge button. Yeah.

A

That means wait it's late after right and there's also one thing that I don't want. I would like to avoid compared to our current review, apps or recurrent review apps are very much in step, lock, meaning every single image that we have, because every single image that we have that would build for gitlab is very much tied to the helm. Chart version as well, and we are very, we need to be looser with versioning. We need to figure out how to properly version, to allow developers that they don't need to roll the whole new environment.

A

Every time something is deployed. That's what's happening to III you apps, because we are so step. Locked literally cannot do a change without changing everything else. Inside of the application. That's not good, like we want to do micro services like things so, let's think about how we can actually do that. Does that mean that we need to build a new of new versioning system suspect, so it.

B

Sounds like it like I could think of three complete whole concepts that we need to talk about in order to make something like that achievable, because right now we're relying on help charts to deploy everything. We make a change to say giddily. We would have to do exactly what jarv did and modify our values file to push that change out to get that version of giddily out there.

B

We need to separate that concern from a values file with formula if we want to enable developers to go to deploy a later version of Italy out into the future. That removes us from dogfooding or helm charts the way our customers would to a certain extent, and we would have to figure out how to make that work or purply and still make it such that if we need to rebuild a cluster, for example, we've got the ability to pull in whatever version we desired.

B

At that moment, time I had another bullet point in my head, but I forgot about my mine now I.

C

Have like one thing, I would love to do, especially for a registry right I think we can probably write tests that can exercise registry adequately that we could enable CI for chart changes and hit master. We automatically deploy to staging. We do some testing and staging, and then we just deploy prod and you would have a helm upgrade pipeline. It works for registry because rare those registry is not, it is decoupled from rails, and you know it is pretty easy to validate, as we think of other component and.

A

Use that as a model use.

C

It as a model and I think as we like that we could. We do things where, like if certain sub directories of the charts changed and that triggers a deployment for registry, like if the registry directory changes in the chart, and that so does that it's that change hits master. Then it triggers a pipeline of decay workloads and then we update registry or something like that- I mean that's possible, but I.

C

Think for now, with just registry, I would feel super comfortable to create a pipeline that runs on every single commit like every single merge to master. We do a helm upgrade as long as we have tests. Okay,.

B

C

Why not you know, like I, think I think we can validate registry easily and I'm sure like I would be I would feel comfortable with that. So.

A

How we do a step back like I, would like to introduce something that we don't have right now. We only have staging pre canary and production, canary and production right. How about we use this opportunity to introduce one or two steps before staging to do this type of thing and find a way how we can automatically verify certain things that will give us enough confidence that we deploy to staging automatically and then the rest. We can do manually I'm, not sure if I'm making sense. Let me try to rephrase we've been using.

A

This model of staging is our first environment.

A

We don't want that because we know the problems there. We want to unlock developers to be able to deploy to other environments where they cannot, they can clash all they want. It's not a problem right. If we do something like this, where you have testing and you every commit upgrades upgrades upgrades, that is fine, but if we can make a system that will then before I go to staging, let me check the state of affairs. Let me check if everything is right. Let me check if Q is executed correctly.

A

If we have no failures, if we do create a report and ask for manual intervention before you go forward like I'm trying to find a way to get what we try to achieve with our auto deploy, but do it properly now that we have a fresh invite I.

C

Think for for a registry, it's super simple, because we can spin up in a femoral cluster and connected to staging and test it right, and if that's, if that succeeds, then promote it to the actual cluster in staging. But registry is a special service. I mean other services are not going to be so easy. Well.

A

They're not gonna be easy if we are starting with an user service, so we can know what I can do right. We.

C

Could do it, we could do it with the front end. To probably I mean right, we could speak, could have an ephemeral cluster that we spin up with that connects to the staging database that connects to staging get early everything right and we validate it and then, if it looks good we tear it down, then we promote to staging for real and that, if that looks good, then we go to canary and then production, so I think a ephemeral cluster would be the way to do it. Yeah.

D

This really sounds like a Bluegreen deployment as a gate between instead of just some global deployment services because yeah it is actually running tests in a real environment. So you have to QA and if it works, you go the thing about the ephemeral cluster for registrate I think that we need to start from a well-known set of ages already existing in that set environment, because one of the thing that you want to make sure is that the new version would not blow up the images that you already have.

D

But if you run your test in a clean environment, then you never exercise the with. My old images still be available after I upgrade. What.

C

If this just be connected to the same object, storage bucket, that the actual registry is connected to okay,.

D

So you are ready to lose stuff, just doesn't work or I.

C

Think, like losing things, is probably a low risk for staging right like I think this is, the idea would be like we spin it up. We do some validations, we spit it down and then we upgrade to the actual staging registry and we do some more validations and then we actually yeah.

A

The idea is like, if you lose something in that process, everything needs to hold and scream right. Alarm need to go off. Why did you lose things that that makes no sense like what caused the problem? Stop the world, like that's the idea at least yeah.

D

Sure what I was thinking that, if we have a thing that is more robust, then we can also use this for production later on. But if we we are not testing, let's say something: it is enough robust and you will never do. We have to invent something else when you're going to do this kind of automated promotion from production. This was my thinking.

A

A

Know that that makes sense, I think I, think we. We have a bit of freedom to to experiment right now to see what what would work best honestly. So.

A

Okay, let me ask this both well to all of you: what is our actual next step, according to you just based on what what we saw in this in this call? What would you like to have it have as a next next step, I.

B

Would like some of the TechNet that we've kind of acquired over the past few weeks, some of it I created an emergent and last night to help resolve the fact that Priya slightly different. But we've got the situation where the alert manager configuration has a two separate configuration areas, whether it be in chef or whether it be in our rumbles repo I, think it would be wise to continue or build up a set of core fana dashboards in a succinct manner, so that we have a place to view all of our charts.

B

Jarv just sent me a private message, so that would be something we should be able to address inside of our dashboards and maybe start creating the necessary alerts as well, because right now, if registry has a complete problem in the pods just start getting stuck in a crash loop, we don't have an alert for that, for example.

B

So we need to create that kind of thing.

C

Yeah I would say that we need to do that and I mean we could. We could start going through like a production readiness review now to kind of see where the gaps are and what boxes we need to take before going to Canary. Obviously, alerts are a big one, but yeah I'm, not sure. If this this idea of having a deployment pipeline is probably something we could think about a bit later like you, don't have to worry about it now, but yeah. I think I think scarbacks spot-on, but just like cleaning up some technical debt.

C

B

Still have a few issues tied to our I think it's epic number 70, where you know.

D

B

The memory issue for the registry is one that's still open, for example, and there's also a few items that we want to clean up for creating central, a central repository for all the CI images that we utilize I. Think addressing some of the items in this document would be wise to work on.

B

And then maybe start thinking about how we want to deploy canary, whether that be its own dedicated cluster or not.

E

B

Sure it probably will be but I think it's like considering what kind of stuff we need canary to do and B and such something else I would love to try to revisit is a method of slowly rolling traffic over currently.

B

The way we are chef works right now is that either we're sending traffic to the gke cluster or we're sending it to our virtual machines I, don't think, there's a way that we could currently say. Let's just move one percent of traffic towards the GK infrastructure versus the.

A

Virtual machines.

B

Especially with our larger applications, I agree.

B

Kubernetes is not a simple platform or we're taking existing infrastructure. I'm.

A

Sorry, what are you talking about? Have you ever been to any cute Kong or read any news item. Kubernetes is the future man? It's super simple. Unless you do, you have any ideas based on what you've heard yeah.

D

I mean this is my first attempt into this so I'm, just not so much into the context, but yeah I would say that observability as the other set comes first so that before we, when we enabled something we need to make sure that we can check what happens so yeah. That's one all right.

A

A

I know all of us are kind of dying to to get this in front of real traffic as soon as possible.

A

I want us to deftly build proper dashboards for this right. We already saw here just in this small test, how things can go around so I think that should be our focus number one. So get get that thing spot-on, then, once we have that we can actually enable this properly in well.

A

No, we cannot enable, because it's already enabled in staging we can work on figuring out how to roll the traffic properly right like how do we do percentage rollout instead of yes or no type of thing just so we have some controls in our hands before we start proper testing and then work on eliminating that I detect that you.

A

Does that make sense.

C

Yeah, it makes sense to me- maybe we should just get this in issues and attach it to the epic.

C

Yeah, that sounds good all.

A

Right so checkpoints: let's talk about checkpoints. What do you think is doable, for example, for the next week, so keep in mind next week, you're still not release manager. So you don't have to deal with any of this stuff, so choose wisely I.

C

Think personally, I think, like maybe by Tuesday or Wednesday I, would like to have the alert team figure it out mm-hmm and after that then I would say you should have I would say by the end of by the end of next week. We should have a draft of the production readiness review and have a pretty good idea of what needs to get done before we can enable this in Canary.

C

Maybe we can also shoot for next week also building out the cluster in production, so that you know it's ready to go. I would.

A

I would hold off from the production story for now, because just the conversation we had today, there is quite a lot to do before. We can even think about that. Probably.

C

A

Okay: let's do that, let's see you said: meet meet mid next week, figure out alerting production readiness by the end of the week and then, let's say week after that, drafts off or rather plan for tech that cleanup.

A

Possibly, we will need to see whether there is there is any interest in other s, eries helping us out with that that, because that would give them a great opportunity to get involved.

A

What better way to to learn the system then actually have a task that is as useful aspect that, unless the two of you are very set on doing that, would that make sense like week after week, too I guess.

B

Thanks to me, yeah.

C

I think I think so I think there's a bit of an unknown that the alert the alerting stuff, but hopefully we'll have a really good idea by next Wednesday right.

A

I would I would ask you to leverage as much as possible knowledge of other accessories they're here so Ben is now in a sari. Get him involved. He loves to help out. So why not as long as all of this is in issues in epics and somehow organized through that everyone should be able to jump in and help out if they want. Obviously do you like I, don't want to decide who does what you can do that between the two of you.

A

Check for me would be let's talk about next Friday before I leave for San Francisco you. We have another call like this, because I don't know about you, but I found this extremely useful.

A

Let's have another call next Thursday Friday to see where we are at and make the plan for the weeks after.

B

A

Right guys thanks a bunch for this and sorry that it went over, but I think it was spot-on. So this is amazing. I really like how this shaped up so great job. Both of you I think you did an amazing job here.

A

All right, we'll chat, later, expect your face to show up on YouTube as soon as this is uploaded. Let.

B

Me know because I've when I spoke.

A

B

The DNA meeting yesterday there's some people that are interested in seeing just this. It's nothing to be wise to share it. Yeah.

A

Sounds good, alright, Jeff is you'll, be there bite.