GitLab Delivery: GitLab.com migration to k8s demos, 28 May 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2020-05-28 k8s migration demo - external-dns and vault discussion

Description

https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/112

A

All right grim, you have a demo item, I'm, looking forward to seeing that sure.

B

So this should hopefully be very quick and not particularly too interesting, but I think it has a little bit of potential impact into some of the work we may consider doing in the future. So I thought I'd bring it up anyway. This is not up for you yet, but it's probably gonna be up for review tomorrow or maybe tonight so I'll just start by sharing.

B

I think this is the right terminal. You seen just basically a mini tube start, prompt, they're, more or less yeah.

C

B

Basically, what this demo is about is there's a kubernetes project. That's actually under the cuban any special inch groups, special interest groups, umbrella, called external DNS and the very short description of it is it's a controller that more or less runs inside your cluster and can look for either CR, DS, so format at full resources or just simple annotations on certain resources and create dns records. For you in you know, whatever DNS system you choose, there's a back pluggable backends there for all sorts of stuff, including things like Google DNS, wrap, 53, CloudFlare Akamai.

B

You know power, DNS, DNS, all of the back ends. There I said something: I've been keeping my eye on for a while for us to leverage I think it's actually really beneficial for us to leverage. So the demo here is just a very quick overview of it running and doing its thing in a dry run mode, so I'm not actually changing any DNS entries or anything like that. So, if you look here, this is just under our you know: get lab home files just a normal release.

B

If I show you the home file itself, you'll see all of this stuff very boring, nothing exciting at all. There's a chart available for this by bitNami or somewhere upstream that I'm, just pulling in I've, got my own chart here which basically wraps around it. So if you look at the requirements that you know for this chart, you'll see it just wraps around the external DNS chart upstream, and the only thing my chart actually does is creates one secret, which is the password for it so fairly simple, not very exciting at all.

B

So what we're going to do now is I want to go home for my mini.

B

So it creates the secret pulls the same values that CloudFlare explore the users essentially and then just creates this very simple service account deployment, which just runs this controller and basically once that goes into our cluster.

B

Well, while we're waiting to that I may as well, I'll show you what how you would actually utilize this. So if I look at get lab monitoring, if you look at so this is just our current get lab on a triple. You reduce this a little bit you'll see here this lying down here, so you can see here on the service for Prometheus I've added a new annotation. That's just called you know external DNS, l for kubernetes hostname and then just a fully qualified DNS record to create. So this is just Prometheus.

B

You can do Gilley's mini cube, get back on that. This is all just testing, so you know just basically about putting this annotation with whatever DNS record you would like. It will create a a record pointing to that service. Typically, you wouldn't do this for viously think it's like cluster IPS and anything not externally accessible, but for things that you expose by external load balancers, you can put the annotation on ingress objects as well. It will basically automatically create that external DNS record for you here we go so our charts now installed.

B

So if I go cube, CTO loads.

B

You can see here. Basically, it spits out a bunch of configuration. The important part here is this is just running in dry run mode. No changes to newest records will be made. You can see here. It's looked at every single service we already have across all namespaces. Looking for that annotation to say, I need to pick up and do some work. They can't find anything so what I'm gonna do while I'm waiting for that is, go back.

B

Monitoring secrets probably should have done this earlier file to be mini, cube, apply so I'm gonna install the secrets we need.

B

So I'm just installing, basically our standard get lab monitoring, Prometheus stack into mini cube, but, as I pointed out, I've put that extra annotation record on there and eventually, when this installed and run and my point in it, this might take a little bit of time to act. We install this chart.

A

Gregg asked if you use suppressed secrets well,.

B

I I do use a version of a help lot of the helm plugin that runs are by default. So if you use the latest version of help of helm, we actually we should upgrade that in CI. Actually, so the suppressed secret flag actually disappears altogether in the latest version of helm, death.

C

B

Yeah, it's bad because when you're trying to debug it there's no way to actually say actually, I would like to see the secret. So I can debug this. So it's kind of it's a it's a bad thing in a different way, but yeah cuz that caught me because I just naturally upgraded the plug-in, not thinking about it and then I just couldn't see secrets anymore.

B

So when this is going to take awhile, unfortunately, as it downloads on the dock of containers. So while that's waiting, I can probably talk a little bit about how this is useful. So there's a lot of places currently what we do in our home charts enough and there are home values where we do. What we do is we create an address in terraform and we create dns records and terraform, and then we have to run g-cloud commands in our helm file to pull that address out.

B

So we can basically make sure that the load balancer we set up in kubernetes has the stack uses that address has that same IP and the only reason we do all. That is because terraform managers are DNS and we want to make sure that when we set up a DNS record, it's going to be correct to the load balancer for gke, when this service is available in a state where we feel confident in using it and say it's kind of stopped us here. So you can see here.

B

This record is probably the most interesting to people and then the two ones at the bottom. So basically it's picked up our Prometheus service and it says I'm going to create an AmeriCorps and prometheus GK de la they're going to create some serve records as well. But you can see where's the a record in here. We go a record 172 1702, which of horses of mini cubic internal IPE, because this is just running a mini cube in a real environment that would be a real life.

B

So sorry, I couldn't interrupted myself there, but yeah one of the big use cases is. We can get rid of all of that clunkiness by having the service available whenever we deploy something in kubernetes and we want it to be made available externally. We just tell it to create a load balancer, and we can also just pop that annotation straight on to those services and it will go into CloudFlare and create the DNS record for us now. Obviously, the big thing with that, though, is how do you do this safely?

B

You know if this is messing with our real DNS zones. You know how we're gonna make sure this does the right thing. That's really good external DNS has a few safety mechanisms built into it specifically to try and solve those problems. The big ones are. It will not modify any DNS record that already exists, so if it sees something that's already there, it will just say: hey I'm, not gonna, touch this.

B

This doesn't look like it's managed by me and just log out the other thing it does that when it creates a new record, it creates a text record with a bunch of information. Saying hey I mean I, am an external DNS instance running in this cluster. This specific identifier, I, created and managed this record.

B

So if you have multiple copies of external DNS like we might have in production, staging and what-have-you, they can all still service the same DNS zone and create and update records, and they won't overlap with each other, because they'll have those text records, saying who's responsible for managing rich record.

B

So that's pretty much. The end of the demo, as I mentioned, I think that getting away from managing those pieces in terraform, so you don't have to kind of if I want to deploy something in cribben Eddie's I need to go and do a bit of terraform and then do a bit of clogging around all of it goes away. The other thing, I think, could be really interesting to think about from a delivery perspective is we can start creating ad hoc DNS?

B

We can create ad hoc kubernetes services of type load balancer with any dns name we want, which means we could start doing more interesting things. Like blue green deployments. We could be having completely separate home releases of like git lab pods. We could have you know, services that we set up easily with a full DNS record that we could give 2h a proxy to front-end or all of that kind of stuff, just being able to really quickly and easily generate DNS entries for any services.

B

We want improve minetti's on the fly straight through the apples through the helm, release that you're doing to actually deploy those applications. I think could be really cool, I think it could. We could do some interesting stuff with that would make live, lives a lot easier in theory, cool.

A

Awesome awesome thanks for the demo, Graham yeah, that's that's something. We've been actually using, we've been using external DRS for two years, I think now in our review apps. ah That is the only way we may.

A

That was the only way we managed to actually on demand generate an environment that is reachable from the outside. So if you take a look at the section, I highlighted here yeah, this is where inside of get lab, we actually pull in the external DNS, and then we connect it to our AWS routes.

A

53 53, 53 yeah, and we generates the the end point right there and actually works really really well I think we had one problem with it at the beginning, where we generated so many that it obvious blocked us, so we kind of had to repair down and like do some tricks around it. But I think this was a super old release. I, don't think we are created this in a while, which means that it just works and it works really well.

A

So the interesting part here that you're mentioning about Bluegreen deployments- we can do that at gate lab only if we have a way to also highlight that the item that we are rolling out doesn't have any changes to you know like database, so that is the biggest biggest problem, but there is a proposal from the development that is currently in basically pre pre discussion stage of actually versioning or main spacing code changes as well.

A

So you could say that you know this part of code depends on this database schema and this particle depends on that database schema and, theoretically. That would allow us to run things in parallel. Even when you have the database changes. So in combo with this it would be great but I.

A

Think external DNS would be interesting for us in in other environments, not necessarily production, but definitely something like staging, where we would be able to test things quickly and when we get to the situation where the database can be easily torn down and brought back back up, we would be able to give more different environments to to everyone actually to test changes. So I'm all up for this type of testing. If you want to do it, yeah.

B

And I think, even if it's not necessarily for the gate, lab helm, releases, I think just like, like yeast or my example: yeah Prometheus we've got moral people, spinning stuff, just small things up and it's great to do. They say ah you don't have to go around and you just you know, terraform stuff or you know just it's all there at once, but you just ship it in the way you go. No.

A

B

A

No, it's a great, a great great demo, great idea and I'm fully supportive of it. So thanks for.

B

Sure I'll pull that up for review tomorrow, but I think what I'm going to do is I'm going to you install it on all the clusters, as part of the initial review in dry run.

B

Only mode so like just just completely dry run, not doing anything special and then I might do a mini readiness, review kind of thing, or just a discussion that DNA about like, like basically I, want to make sure people like Hendrik and that who are very familiar with the DNS infrastructure, make sure there's no red flags or anything we need to watch out for and then hopefully I just want to even use it for internal services. Just get lab dotnet, nothing get that calm, and then we can just start.

B

You know getting rid of a tiny bit more of complexity. Amazing, a tiny bit more out of terrifying.

D

You could probably use this for plants, you know because that's relatively self-contained and it has a publish T in this endpoint figured in terraform. So after.

B

This is working.

D

Internally, that would be a good place to use it.

A

Would it be able to use it for registry.

D

Currently, no because the the end point for registry is still using a proxy but I think we've been discussing. The only reason we use H a proxy for registry is because of canary, which is doing you know, request path based routing, there's, no reason why we couldn't rig something up in Cooper that he'd said similar or I. Think we've talked about it, so yeah I mean yeah. We don't have an issue tracking. That out should all open one up.

D

You should probably get rid of H a proxy, and once we get rid of H a proxy, then yeah probably makes sense to manage DNS.

D

B

Does high J proxy talk to the registry load balance of iodine s entry? But how is that? Okay? Because.

D

B

That that entry, potentially even with H a proxy in front of it yeah.

D

It actually uses an IP address. Do you use a DNS entry, but it could use a DNS entry. It's not ideal, like with AJ proxy. You generally want to use ip's, because you know if the IP changes you have to reload a proxy for that change to be propagated.

C

Yes, register pedis rotisserie use house canary English, Street register. It was just a bit blowing stuff yeah.

D

I mean it's: it's it's a bit of a hog. We! Basically since we have it's a liar, it's a layer, seven load balancer in front, so we're able to inspect the HTTP request path and based on that we're out to either canary or the main infrastructure.

D

So if you do a registry like a docker pool on a get lab or give a comm registry image, get lab or give a comradery image that it uses the canary pod, it does not get a lot of traffic, though we've thought about using weights to shift a percentage of the traffic over to canary that's another option, but doing this in a checkbox. It seems a bit silly when the facilities already exist in companies to do this, so we could probably just get rid of the a proxy layer at some point.

C

And so that can register areas just to get used to it.

D

We always have you always well, not always, but we had a registry canary before and we'll probably do the same thing for the front end when the kubernetes will will create. We have a canary named space and kubernetes, which will test out first and we'll just serve. Give a corgi can give up comm traffic to those pods for testing and then, but I think, eventually, we'll probably consider you know doing something different. But this is what we have for now. I.

B

Think, like thinking of doing something different as well, we have potentially other issues right like with things like sidekick. For example, if you try and have like two sets of sidekick pods running, do we have a way to say: hey you, your sidekick, pods you're running it. Please don't do any work. We want to want someone else to do the work yeah.

D

We don't have any if you're talking about like a canary sidekick or having namespaces there's this kind of a long-standing issue for that doesn't exist, not something we could do currently, because everything shares the same Redis pool, there's no name spacing for Redis, so yeah. So that's not really possible for us to have a isolated set of sidekick workers without.

A

Some investment in the application.

D

Without solar investment, yeah.

A

And that issue is open for three and a half years now and I tried to escalate it a couple of times, but it was never a crucial part of our workflow. So if we need to make it if we can actually utilize it and do something about it, we can I can try again.

A

I wouldn't hold my breath.

A

Cool Graham I think your approach, sorry I got this I disappeared for a tiny bit, because I had to answer a phone call, but I think your approach was starting something non github application related first and like having that work. Rock-Solid with external DNS is a good one, and then we can start thinking about these things on how we can implement it inside the application as well or to support the application as well. So I definitely encourage you to continue doing this.

A

All right, do you wanna. Did you go over the wall twerk, while I was.

B

No I and in I'm happy for look. The boat work is just more of a fYI I've people are interested if anyone's got anything else that I'd like to cover in the meeting instead I'm more than happy for them to do so.

B

I'll, take that as a no in that case, what I might do is I'll very, very quickly, just show the state of what I've done involved. It's actually not that difficult for me to just run through the more or less the the code. That's in the repo and the repo itself. It's not all pushed up there yet, but at least that should give people a kind of a clearer view of how it kind of is put together at the moment and how it looks, and you know if there's any discussion that comes out of that.

B

That would look great. Let me just find.

B

So there's basically one kind of get repo at the moment that all involve lives in, even though it does actually contain terraform pieces for the infrastructure, helm file, pieces for the installation involved and then more terraform to actually configure vault itself.

B

The project was done like this, what by Henry months ago- and it was mostly based off the idea from the security team who've been quite involved in this project, trying to see if they can we're trying to look at vault or they were trying to look at fault as a way of you know being because it's the keys to the kingdom or it could become the keys to the kingdom of isolating all of the bits and pieces to one repo, so they couldn't lock that down from a security and or durability perspective.

B

So if you actually look at this repo I'll show you inside the terraform directory first, if we actually look at what this repo sets up for, try and get away, it's nothing too exciting. All we setup from the actual Google cloud resources is we set up a service account. We give that service account some permissions.

B

We set up a kms keyring that vault uses to encrypt all its workload and then the final thing we set up is essentially network, our cloud NAT and our qke cluster, so vault needs a minimum of five nodes and they recommend you have those five nodes with nothing else running on them from a security perspective once again, so, basically all we do is we set up a kubernetes cluster that actually has six nodes in it? Which is you know it's one too many, but it's probably okay.

B

Just that way, we get an emic bit of extra capacity. I am leveraging the latest version of the gke terraform modules, so basically I am using that workload identity. So the service account we need for vault, which talks to G kms. We don't need to put keys or generate keys or put them in one password. It's all just works through G K G K is workload identity, so that part is very simple. The helm file part is even simpler. It's literally just one helm file.

B

If you look at the helm file, I'm doing some ugly hacks to get a bunch of values. Out of terraform itself, but this here is basically just a normal installation of the vault helm chart and everything you see on the screen now is more or less all the configuration that is needed, which is actually very simple. This used to be a lot more complicated because earlier versions of vault the storage engine was console and it had other storage engines like Google Cloud storage, which is what we were using, but they weren't technically officially supported.

B

As of one point for the new version of vault, they now have an inbuilt storage engine that uses the raft protocol. So really now configuring and installing volatile kubernetes is exceptionally simple. It's just about there's about five pods. They used a PVC PVC for their own data storage. They run in a service account that can get a G kms key for encryption and decryption of that data, and that's more or less all. There is to it the most interesting part, I think to volt and there's and is the part we've barely scratched.

B

The surface on is the terraform vault part. So this is one small T is up and running. We've created the GK, a class that we've created a service account we've installed the help chart it's already. How do we actually configure vault itself? The big theme is things like orth methods, so this is all done in terraform, even though it's not really infrastructure. Simply because terraform is a hashing coat hashey core product vault is a hashey core product said: I recommend is terraformed to setup and configure volt is the first-class citizen.

B

I won't go into all of this in detail because I could be here for hours, but things like configuring, vault, backends, the policies so who can access what Colin? What particular thing like an admin can access everything, but you know get loud.

B

People can only do certain things on certificates of all of this stuff, it's kind of more or less encapsulated in there, and once again, that is just the start of the very deep hole that we go down: I'm hoping over the next week to actually have that vault instance up and running and like we could start doing very basic testing against it, even if it's only got very bare-bones policies.

B

The other thing I'll point out is the integration pieces we have with volt, which of that with our current tooling, are quite nice. So for those you who aren't aware already right now, whenever you run a CI job and get lab, it generates a JSON web token, which we can directly validate against the vault provided. Volt has a couple of configuration tweaks to be correct and essentially grant a policy which is a set of secrets to that job.

B

So I'll link to the documentation just find that, because it's a pretty interesting read, but essentially what that a little hours to do is when volt is up and running instead of having to set environment variables for secrets in CI jobs, we can basically just do a vault login in the job itself, using the vault tool and then any tool that needs to talk to vault can basically that CI job is authenticated to all the secrets that we set the policy for it to have access to.

B

So that's really nice and it makes things a lot easier. Let me see if I can find that documentation.

B

Here we go so I'll just put that in the chat. In fact, I'll probably stop showing my screen.

B

So there's the the documentation for how get labid ci works with fault, and the other thing I'll point out as well is helm file, so the helm tool we use for you know basically deploying our home charts at the moment has a built-in support for vault as well. So all of those places where at the moment we did a really ugly shell hack of going to G, kms and GC GC s to get all of the secrets and decrypt them and everything you can just replace with a line that looks something like this.

B

Basically, it looks like bolt slash, slash path like that, so you can pop that straight into the helm file and when helm file runs as long as your current session is logged into vault and is valid so like a CI job or like a user on their laptop, it will pull that secret down and populate it directly into the into the location if it's needed.

B

So this will drastically clear up a lot of the really funny stuff we have relating to secrets in a lot of our helm files it'll make it nice and clean and simple. Once it's available, so that's probably the most useful benefits I see coming from getting volts straight away or as soon as possible into this tooling.

B

There is a bigger question which I'm not sure, if we're ready to think about yet about how we do secrets management with volt, because it also has a volt injector, which isn't basically another controller that can sit there and constantly pole-vault and recreate secret objects. When secrets changed behind the scenes, rather than binding it to our helm, deploy process.

B

So it would mean we take away all of our secrets management out of all our helm files, and we would just have this agent doing all the secrets management behind the scenes that changes a lot of the way. We do things, though so I'm not sure. If that's something we want to look at straight away, but it's an option for us down the road.

A

There is one thing: I was not aware of the integration that we added with wall, so I know that we added something I just didn't have the time to look at it. What what immediately jumps at me is that if we do authentication with all directly do we have a way in policies involves to tie users to a policy that is configured in the policies file as in if we remove all of our secret variables that we currently have as environmental variables in github groups or projects.

A

Theoretically, then it doesn't really matter who has access to the project like at least internally. We don't really mind whether someone can see the code or not. We are open-source in general, so the major problem we have we have with our permission system is that, as when you're a maintainer or even a developer, you can easily print out secrets and then everything is out right. So if we go this way, then you can't really do much.

A

You authenticates, because we run the job in the context of the user. It means that you can automatically authenticate with volt and we can check you against the policies and like ensure that everything is kosher there and then it doesn't really matter in which group the project lives and who has access, because we have other ways of limiting changes to the code. We have approvals, we have you know like reviewers and so on that we can limit and then, as you, whether you're, a developer or a maintainer becomes less less important.

A

So that would be I think a big win for us. No.

B

I think that's right and I'd say how it works. Is the CI job will get JSON web token and go to vault and say this is my JSON web token? I am in this kit, get lab instance. I am running under this project and like this is a CI job under this project. Under this code, lab instance involved will validate that against get labs.

B

Api go yes that you are who you say you are, and then we tell we tell volt, ok, CI jobs under this namespace under this repo can have these secrets so you're right, we use get lab as the control mechanism, like users and groups, I think it lab, but we know that that CI job once you once you have access to a CI job, a master because your maintainer and you can merge to master or whatever that, whatever we figure out that workflow is. We know that the user part is gone.

D

That would really solve the problem Erin, because the problem we have now is not necessarily maintain errs, but developers you can just you know, tap the secrets in a CI job I. Don't think this would change that right, because the job still has access to the secrets and the events, and even in this case, like I, don't know whether masking works at all. It may not, which may be a problem if we use this right like this, like the secrets, are automatically masked for CI variables in output.

D

But in this case, maybe, if you're not careful, they wouldn't be another.

A

Thing is that that's right now we blanket give access to anyone who has the same role right, whether they can access it or not. With this change, you at least have that control of saying, if you're, not in the policies file right like in this group, you can't even access the secret and, if you, if you, if you get added as a developer, to a project, you immediately get access to it, whether you cut it or not, so.

B

I think what we have to be so what this I get, what my job and yourself is saying, I think what I would have to double check did the probably the clearest way for me to articulate what we can and can't do would be to look at the contents of the JSON web token. So if we generate the JSON web token, they get lab itself generates, it does specify. Information like this is the guid lab instance.

B

This is the project if it specifies in that the the user that caused this job to happen, then definitely on the vault side. We can validate that web token and use that because we know that web token is valid. We can say, okay, that information we can extract. Out of that, we give the policies this user is you know this person they've got access to this policies. If we don't have access to that in the web token then vault will be able to.

B

We can have a policy for the job, but we won't be able to make any more clear and distinction from that. If that makes sense, so I think it'll really come down to the implementation. It's worth noting that they're also doing further improvements so job what you mentioned about masking they're, getting masking support fourfold secrets. It's not there yet you're right, it's actually not there yet, but it's coming I've seen the issue and it's coming so I think as a first pass.

B

I definitely wouldn't want to be using this like fully in production straightaway, but I think it'll be good for us to start testing and seeing some of these problems because I, even though they've added this to get lab I, don't think anyone really uses it very much. So they think they'd be very interested in people trying it out. I. Think.

A

You would be surprised how many people like using these edge super cool things. Yes,.

B

Just whenever I talk to them, they're always like oh man, be great. We had since we've great I mean maybe people using it, they love it and they don't get feedback. I, guess they're, really looking for people to give them feedback, I, guess yeah.

A

So looking at the payloads, we do send user ID user email like so we have all of this information which would be very beneficial I. Think for us! Sorry, I interrupted you. You said another thing before yeah.

D

So so I guess there's there's two ways we can use this one is for CI variables. The other is for chef and kubernetes secrets and I would group those two together because the chef and kubernetes, because I think we're gonna, probably maintain the split or you know months, not weeks. You know, hopefully not years, but so I would I would say, like between the two I would rather focus on the chef in kubernetes park, which would be to modify the shim.

D

We have for chef to source secrets from vaults and then and then after we do that and I think we can just integrate it with kubernetes quite easily. So maybe maybe the thing to do is focus focus on that. First, before we focus on the CI verse.

B

Make sense and I think that's definitely, but the first focus once I get up and running to see if I can get rid of the horribleness of some of the helm file stuff. So just you know pulling those secrets straight out and then yeah I think definitely you're right. The next stage is figuring out how we can make chef work with it. Hopefully it's not too bad, but haven't had a closer look yet.

D

And no sense go ahead. I.

C

B

All your poor work has been throwing away I'm. So sorry.

B

At the moment, it's so now it's using PVCs, which are basically Google disks, like Google storage disks, so we just do specials so we'll need a backup, job and I'm happy to write the job or never and just basically now it's just those g-cloud snapshot, a bunch of disks and Bob's your uncle and you restore them. You just restore those snapshots and bounce. The pods I think.

B

I asked the how she called people exactly this problem. It's like okay, well like do. We need to like tell it to you, know, sync or whatever and they're just like doesn't write that much. You know, we've been fine with that's what they gave me for. That was the official how she caught response, those like okay but yeah. Unfortunately, we do need to change all that, but I think this. This is better overall I think, fortunately, so just.

C

A lot sounds like a lot much better than trying to do a task entire copy of GCS back, which was always gonna, be risky. Yes, we might have to do any of the frequency. We were just give to get some chance of a good backup, they're.

B

Very much like don't back up pay us lots of money for our replication and giono capability and just you know, run at a billion places and therefore you.

C

Doesn't serve you from deleting things yeah.

C

One other question: I keep forgetting which Google GCP project we were running the Sun. It was in revolt and good their fault. Yeah I can.

B

Translate delete one, so it's gonna be get back. Bolt I think I'm going to delete the other one. I haven't yet just for some reasons, but I'm gonna go and clean that up very important.

C

Than have you seen the spreadsheet for cleaning up Google produce.

B

Ice did but I'm lucky. You just reminded me, because I went through and cleaned up. My I looked at my own, but I did not look at that one. So I probably should go and look at I mean.

D

C

B

They delete it right.

A

Clear, we are not immediately deleting things we are looking for owners first and then we're going to contact the owners for the items that we want to delete and so on rocky it's not going to be like flip the switch and delete little production when it doesn't work like that.

B

We should even do some by even discs not showing back up wise. We should like, because the dot is gonna be like only a hundred makes bigger or something we should slug the daughter off somewhere else as well like we should really. If it's going to be this important, which it probably will be, you should slap that thought of somewhere else. Yeah.

D

C

Just terrified of the idea that someone accidentally delete something out of the vault and then it's gone and you'll secrets are just not there anymore in recovery of that would just.

A

You cool, we have a few more minutes. Does anyone have anything else they'd like to share.

A

Job would you like to share with this audience the the next steps with which I think we discussed this already, but it would be nice to share it with everyone here. Yeah.

D

D

D

um Ice cream I know that we're kind of going over here so I'll be very fast.

D

So, there's a there's, an mr here to add tags to workers and we're, starting with a tag that says no disc IO. This will allow us to identify queues that don't depend on any disk IO, and you can see here, Shawn I think just tagged all the the queues in the group, but this is not.

D

This is not final, yet he left a comment that said, he has to kind of go through these in detail, with some code inspection to kind of see whether these cues rely on disk I/o and the reason why we're doing this is so for the next migration. You want to be sure that none of the workloads depend on shared storage. We have NFS mounts on our sidekick fleet for cache and build traces. This is like the archives and the builds not point, and we really don't want to. You know, of course we're not.

D

We really don't want to do those NFS amounts on the kubernetes, so we want to make sure everything that migrate doesn't depend on them and the first pass- and this is going to be hey- let's just migrate, all of the queues that don't depend on any disk I/o. That should be easy. Well, we'll see how easy it is.

D

I mean we're going to spend some time doing some code inspection Sean said that he's going to maybe play around with local dev I thought, maybe the best way to go about this is to why not, why not just create read only enough points on pre prod for the shared storage like these directories and see, if you see if things run, I mean obviously like we do this on pre prod to do this on staging.

D

If anything, rights to these NFS amounts on the read-only value and things should just crash or not work, so I thought about doing this within I saw in charts. We can't set volume mounts per pod, so I hope it up an issue for that and probably to change charts. It was important. It I think this.

D

Probably we want something we want anyway, because currently, if you didn't know that, like some, some workloads write to the shared directory, specifically project exports and they use it as a scratch space like whenever you do a project export their writes a very large tar file, sometimes with large profile to that directory before it uploads into object storage. We don't want that empty. Therefore, every single pod, like we have it for project export. We have it for memory, but I think like moving forward.

D

It's going to be like you know, for some two groups like we shouldn't have any you should you shouldn't need to write to shared to shared now point at all, so I think we'll go ahead and do that and then once we have that chart update, then I'll go ahead and make that change on pre prod will start tagging. You know to use that we're gonna migrate over and we'll do some testing that way. Any questions about this.

C

Do you expect to tank queues, possibly by name or just sort of try and use the queues? Victor's exclude yeah.

D

So we'll use the queue selectors to take a group of tiers and then we'll use these. This tagging to you know, take a subset of the queue group, so.

A

We'll yeah yeah.

D

So we'll use them we'll just tag like I, don't know like ten queues at a time or something and the ones that were relatively certain to rely on. You know shared storage for any kind of I/o and yep. So that's how we'll do it.

A

Cool things, Java sharing things for the questions as well, thanks everyone for the first day pack, demo I, hope the time was not too late for Dramamine crack. If it is, we can tweak it a bit but I'm happy that we we are able to share this with you as well, and if we need to, we can increase the frequency and do like two times a month and then two times a month for for the other, depending on what we have to show.

A

But let's take that offline, we'll discuss it in any case, have a good Friday and the rest of the weekend and next week, bye.