GitLab Public Sector, 4 Mar 2020

Previous Meeting

⏯

youtube image

►

From YouTube: Walkthrough of GitLab Kubernetes Install (UBI version)

Description

Walkthrough for the Public Sector CS team of installing GitLab on Kubernetes using the UBI gitlab containers.

Links to some of the documentation and tools we used:
- Gitlab Charts Docs: https://docs.gitlab.com/charts/
- Example values.yaml: https://gitlab.com/gitlab-org/charts/gitlab/tree/master/examples
- Deployment: https://docs.gitlab.com/charts/installation/deployment.html

- kubectl: https://kubernetes.io/docs/tasks/tools/install-kubectl/
- eksctl: https://eksctl.io
- helm: https://helm.sh

- Troubleshooting: https://kubernetes.io/docs/tasks/debug-application-cluster/debug-application/

A

A

Okay, so today we're gonna walk through how to deploy get lab on kubernetes we're gonna sort of make this the beginner version. So if you know a little bit about kubernetes, you've messed around with it a little but you're, not an expert. That's probably a good like target audience for this.

A

We're gonna specifically talk about, let's see here or to specifically do the Red Hat ubi version of our containers, which is gonna, be of interest to a number of our pub SEK, specifically DOD customers, we're gonna, do it on Amazon, using an external database and cache, but what I'm gonna try to do is talk about sort of an overall approach, for you have an environment. You have specific requirements, for you know what needs to be rolled out. How do you evaluate those? How do you customize the deployment to get those requirements met?

A

How do you install and then troubleshoot? So this should be of interest to you know, folks, beyond just just our immediate pub sec, customers and then I just wanted to you know, say right up front. We have pretty good docs on this, especially once you understand the approach and you're looking at sort of tactical things that you need to do to solve specific problems. The team did a great job on documentation and examples so make sure to check those out before you undertake this effort.

A

So there's a bunch of different containers that need to be installed that make up get lab. We're not going to go through all these, but I just wanted to show that the deployment has. You know a number of different moving parts, so there's a whole bunch of containers. There's persistent storage for storing the get data. You know we need a database. We need a cache. There are some housekeeping containers that have to run during setup. If you want to use let's encrypt there's a number of different cert manager things.

A

So there were a whole bunch of different tasks involved here and the helm chart is going to help us make this palatable. So what what do I do in advance, because I didn't want this to be a three-hour long thing, so I provisioned, an Amazon eks cluster, set up a database and created a get lab database in the instance set up and ElastiCache instance put them all in the same VPC and hopefully set up a security group so that they'll be able to talk to each other.

A

We need object, storage for a number of different things that we'll get to so I've set up a bunch of s3 buckets for that, and then I created an iam user with keys to get to the buckets then on my local workstation I've got cube, control or cube cuddle, depending on your opinion of how it should be pronounced, which is the main command line tool for us to interact with the cluster. It uses a file called that's typically referred to as cube config.

A

It's in the dot cube subfolder of your home folder, usually, and it's a file called config. So that's where the settings are gonna. Be for that, because we're going to do this on AWS I installed the AWS command line and eks control, which is the command line way to create eks clusters. It's really cool. It's one command line with a bunch of switches, we'll get you a cluster that will probably meet your needs.

A

It uses cloud formation and it just rolls everything right out and then, when we get to the troubleshooting, we'll talk a little about cube tale, which is a tool you can install it's a it's. It's are just a separate tool that lets you tail the logs from a cluster. So basically, if everything starts but get labs not behaving the way, you want it to it's roughly equivalent to the get lab tail function that you can do on our omnibus.

A

So that's convenient it color codes because there's various running in in a pod, it color codes the containers, so it's kind of a cool way to see what's going on and your troubleshooting alright. So if you find yourself presented with the need to install get lab on a kubernetes cluster somewhere before you start actually doing anything, you want to think through sort of all the various special cases and situations at a minimum. You're gonna need a plan for the following things and based on your requirements for you, there may be other things as well.

A

So first, you need to know specifically what type of kubernetes you're gonna use, so I selected eks for convenience at the moment. Actually, many of our public sector customers will not be able to use eks because it's not currently available in gov cloud. My understanding is that will change, but what we're gonna see here, you know there'll be differences, but this is you know. Hopefully this will be widely applicable. You need some sort of ingress controller. In our case for get lab.

A

You basically want nginx ingress you'll find if you would try to use traffic or another one that you're gonna be losing some features. So if there's a problem with nginx ingress, if that doesn't work for you for some reason, you know it's something that we should. We should work together and and look into further you're gonna need a load balancer, so that's typically going to be something that's you're, gonna think about as part of setting up your actual kubernetes environment.

A

In our case, it's gonna be easy for us, because when once we deploy this using the helm, chart AWS is gonna spin up one of its load balancers for us without doing any extra work. This is mostly a bunch of extra work if you're gonna do bare metal, because you're gonna need something like metal lb. As your load balancer, we need a way to handle certificates for HTTP. We are gonna, use certain manager and just do let's encrypt again in in many cases, you're going to have a processing your organization for getting the certificate.

A

So that's the way they're gonna work, but for our purposes we just are gonna just use. Let's encrypt, you need a place to get the containers from so again. We're gonna, take the easy way and use registry get lab comm to get our containers if you're using ad car hardened container. If you're working in an offline environment, there are many different reasons why you may not want to use our registry. That's fine. I'll show how you can specify a different registry to get the containers from you're gonna need a database.

A

You can do you know. Obviously there is an in cluster Postgres available depending on your requirements. You may very well want to use an external one and that's gonna often be the right way to go so we're gonna use RDS for ours. You need a cache. We're gonna use an external cache as well, so we're gonna use ElastiCache Redis, and then you need an object, store we're gonna, use s3, which is sort of the easy way to do it again. If you are not used not on AWS, then you have some other options.

A

Min IO or you know other sorts of things like that. So the first step is work through all of these figure out the answers for them and if you have any other requirements that don't fit into here, collect all those up in a list so that you've sort of got your plan of action for when it comes time to to roll this out alright net, then what do we do with all of these decisions? We've made?

A

We have our gate lab helm chart and it has custom values, so you're gonna need to override many of those values to meet your specific requirements. Handle your situations, provide things like you know. If you have an external database providing the the end point for it, specifying the secret the kubernetes secrets that contain your login information, things like that there may be number of different things. You have to do. We're gonna walk through our values, yeah mole that we're going to use, but in general, you're gonna want to check out our examples.

A

So I've posted a link to that, because the distribution team came up with a whole bunch of different examples to handle specific situations and you're gonna put those together into the values. Yeah mole for your environment, that's gonna, override the ones in the helm, chart and I'll show you how we can pipe in our values, yeah mole and that's how we'll do the deployment all right. So, let's take a look at.

A

So, first, by the way, here's how we created the cluster, you can see it's just a single command line. So if you want to do eks for testing or for your deployment, they made it very easy. You can do a config file also, if you need to do something fancy, but for particularly simpler situations like this one, it's very straightforward, all right! So here's our values, dot ml, so we'll just walk down through this I- won't spend a ton of time on it.

A

But we'll take a few minutes to look at some of the different settings here and will.

B

You be able to click stab it.

A

C

A

There an easy way to do that.

A

D

Option + we'll do.

A

All right, can you all see it better than him.

B

Yeah thanks, ok, good to me all.

A

Right so the first thing is I mentioned earlier: we're gonna use certain manager. There are different settings if you want to specify your own certificate, you'll, be disabled, surf manager and specify it separately again. It's in the examples we're gonna use, cert manager, because we need a certificate and it's the easy way to do it. We're gonna turn off our in cluster Postgres and Redis, because we're using external ones for the moment we're going to turn off Prometheus and get lab runner again.

A

We just want to keep the situation simple and then we're gonna need an engine X ingress. We have one in eks all right. Next, we're gonna specify the domain. So if we spent the way we're specifying this it'll be get lab, k8s turn lab net will give us the will give us get lab and then, if we wanted to set up the registry registry, ks9 lab dot net, do that we're gonna tell it.

A

We want HTTP and then because we're using AWS for load, balancing, there's a couple of load, balance or annotations that we want to include there. So, let's keep going all right, Minaya we're gonna turn off because we're going to just use s3 for our object. Storage, brief! ah No we're gonna turn off just because we're keeping it simple all right, so we're so here we're at our first image section.

A

So using the get lab registry you specify the version or whatever registry you specify, which version you want using this tag and then, if you don't want to use registry, get lab comm, you specify the repository location there. So right now, if you use our registry- and you want a specific version, the tagging situation is somewhat complicated. They are working on simplifying it and they're.

A

Also working on documenting it, so you'll be able to get exactly the tags you want at this point to get the tags I essentially had to work with our distribution team, but that should be a temporary issue. All right there are a couple of.

A

There are a couple of containers that we use for like housekeeping and setup type things. If you don't specify one I think you get just a default busy box type thing. If you want to only use UVI images or you want to only use images from a particular repository, you can specify those here so again, some of our DoD customers, where they're gonna, where they want to get everything from D car, we'll use a plain ubi image for the housekeeping sort of tasks.

A

It'll have what they all right, we're going to do, an external Postgres, so you can see here, I've specified the end point for it under host for password we're using the the kubernetes secrets, tour to hold our password information, and so it's sort of a key value store sort of thing. So you give the name of the secret and then the key inside it, and then the value has the secret in a specific format.

A

So, let's take a quick look: well, the Postgres one I don't have will actually we'll come back to that the Postgres one. The way they want you to specify. That is just a literal, so I've actually already done it, because I might keep this Postgres up for a while and I didn't want the password to be on the screen in here. So we'll come back to the to looking at the secrets a little bit so here's Redis again I gave it. The endpoint gave it the port that sort of thing all right.

A

Now we get into our object store, so you can see I've set up a number of s3 buckets that are used for various things for the registry for LFS.

A

That's that one for artifacts for uploads, for packages and right now, the way I just for simplicity, set up one user granted access and then I'm specifying the secret with that user for all of them. So so again, like I, said it's a key value store, so the secret name is get lab, s3 storage, the key is connection, and then the value is what you specify here.

A

So you can see this file is gonna, be inserted in now that I've taken out again the iam secret information I didn't want in here, because I might keep it running for a while, but I've got another one that has it and I'll show you how we install these secrets in a moment. Unfortunately, there are two different formats we need for this information, so the registry keeps its secret in a different format, so we have to create two different secrets and put them in and then reference the correct ones.

A

So we'll move through that, and then here in this section we get to our other containers. So you can see I've specified the appropriate tags for these, and this these are basically the tags for twelve get I get live, 12.8 dot one and we specifically are asking for the UVI eight versions of these. So we go through all those you can see. We have a few more I could probably do a better job. Tidying this up so they're all grouped together, but you can generally see what we're doing here. We're specifying the image tags.

A

We could specify repositories for any of these and then we're specifying some secrets. So that is our values. Yeah Mille, that's everything we are we're going to override. So let's go ahead and.

A

All right, so here's our here's, our command line. We have, we have our cube config set up already so.

A

Let me grab it so. The first thing we need to do is we need to create some secrets, so we will first, this will create the secret we referenced for our s3 storage and that's it s3 storage created now, as I said, we actually need to do two of these, because we need the format for the registries a little different.

A

Okay and then, if we do TL get secrets, you can see there's a default secret that you get out of the box for the service account on the kubernetes, then the three we created so the two I created just now, and then the Postgres that I created earlier.

A

So these are what get labs can I need? Okay, then, the next thing we would do is we all kick off the install using help. So what we're doing here is we're we're calling helm we're gonna upgrade, although we're great we're using the installed switch over there. So this will also work, even if you don't have anything installed, we're gonna get get lab out of the repo. So there's a there's, a previous command to setup the repo that I don't need to run.

A

So we won't run that and then we're specifying our values Gamal. But if you're looking at the docs you'll see it'll walk you through the steps you need to get to where you can do this all right and then we're going to hit enter. So this is a fresh cluster, so it's saying reliefs get lab does not exist installing it now and it's gonna go ahead and install.

A

That's this is gonna, take a minute. So while this is going.

B

A

Will bring up? Oh there, you go alright. So now that that's installed, we need to set up our DNS before we go any further, because we could not do the DNS till now, because we didn't have a load balancer. Yet so we didn't have an end point for the load balancer. So the first thing we're going to do is we're gonna. Ask cube control to give us the information on the ingress --is that are set up right now.

A

Okay, so get lab unicorn, that's sort of the main get lab. You can see they're all the same anyway, because we get one load balancer for this, so we're gonna copy that and then over on my other screen. You know it'll depend on what you know how you are, how your DNS is set up: I'm using Amazon route, 53, so I'm, just gonna set I've got cname records and I'm just gonna drop those in.

A

Now the way, let's encrypt work works is it comes up right out right off the bat and tries to do the challenge. So I may have done this fast enough, but it may have also failed the challenge because the DNS wasn't set up fast enough, and in that case all we have to do is kill and restart the cert manager, and it will attempt to do that again.

A

Okay, so DNS is now set up. So let's see if our pods all started.

A

Okay, so things are going fairly well, so far, um more or less everything's running looks like unicorn is the only thing that isn't up yet, but it could take up to probably five minutes or maybe even a little more depending on the situation, so we're not necessarily at the point yet where we have a problem to be concerned with. So we can just wait a moment check it out again and it looks like everything's running and look at that. We have a running, get lab. That's uh so so we're we're ready to go.

A

This banner is because I reuse the database from previous, so you'll you won't get anything out of the ordinary, but because I you I didn't provision a new database for this time around it's using that okay, so we've got a login box, but now we need to know what the default password is. So this is done using the secret.

A

The secret store get lab actually creates a secret with the initial root password in it. So you can see.

A

That now there are a whole bunch of secrets, and if we do this this will this will take the secret and then we've added a little on there to just parse it out. All we have to do is change is specify the deployment name which we just made. It get lab.

A

Okay and that's our initial secret, that's our initial name, so we'll copy that we'll say: root, we'll paste that in poops hang on. Let's check this out.

B

It's cuz, you reused the database. Oh.

A

So that so it's it's, uh it didn't overwrite the root with the new one, all right.

A

There we are so we are in our 'get lab environment, so I'll stop there and I know I. Just went through a whole bunch really fast, so I'll stop there and see. If folks have any questions and then maybe what we'll do is we'll use. If there's time left, I'll intentionally break something and we'll try troubleshooting it so first off, are there any questions about what I just walked through.

D

D-Did, you point out the documentation yet.

A

The documentation is, there is a link to it in my deck, but the documentation is here at dock, socket, lab, comm, slash, charts, and so this is a great place to start. You know in terms of basically all the steps we just talked about, so it gives you information on where to get those example, values yeah, milles I was talking about how to actually do the install using helm.

A

So the the distribution team and and other folks have created some excellent resources to help you get it dialed in the way you need it for your environment.

E

Even this is a Daryl helm versus omnibus what you know. Why did you kind of move use helm versus the omnibus package Oh in.

A

Terms of just installing omnibus, on kubernetes, you know in a container well, so the it gives you more flexibility. Basically, if all of get lab is in one container, then the the kubernetes itself, basically, you know, doesn't have the flexibility to put different components in different places, and you know if individual nodes are lost, bring them back up. You know, trying to use it sort of decomposed into its constituent parts, makes it easier to scale. You know individual bits up and down to handle the man.

A

So it's I would say it's a more it's a more kubernetes solution than than doing the omnibus container. We also have many customers who have told us that this is the way they want to do it as well. So you know pros and cons notwithstanding it's just a requirement in many cases, gotcha thank.

F

You yep hey Steve, this Kevin, um so if, if we've a customer on c2s that doesn't have a eks available, can they still take advantage of the ubi by I, guess kind of similar to a previous question? Can they can they do an install of this into a ubi to gain the benefits of the of the ubi and the other pieces of the hardening rather than if they don't have access to kubernetes?

F

A

So these specific containers are intended to be installed in kubernetes of some sort. It needn't be eks, though I selected eks, because it's convenient because, as you could see from earlier, you run one command. You wait about 10 minutes and then you've got a kubernetes cluster right now many of those customers who want to use kubernetes and don't have that option because it's not available yet you know our just setting up ec2 instances and running a kubernetes distribution of their choice on that.

A

So that might be you know rancher or rke, or you know one of the other distributions, and so you know the containers themselves and the general approach I've outlined will be perfectly fine in whatever kubernetes distribution you want to use as long as it's a sort of standard one. You know there are things like open shift where there are specific constraints, so you know that that cannot be done with the current set up. We have here but there, but the actual distribution you choose is up to you.

A

The just these particular containers, though, are intended for kubernetes. You can do something like what was discussed just now in terms of you can take omnibus and put it in a container and run it on just the standard, docker environment, or there are other things you can do with that. But what I walk through here you know does require some flavor of kubernetes, but.

F

Good, so the omnibus had to install, has been tested on the ubi. That's been provided by Red Hat for those that just need a small install, don't need kubernetes. Don't is that a possibility? Yes,.

A

You can you can use a ubi container and and run just install omnibus inside it if you, if you want to do that, it's not something I have personally done so. I cannot speak to the details, but I know it is. It is something that's available and.

F

Then you just just for the purpose of the video for those than my watch later, just to be clear. The other pieces there's an issue which we can provide a link for for the Phipps piece, because that that was not there's no Phipps piece to it to any of this right. So if a customer has a need to run and if it's enabled this does not provide that correct, correct.

A

Yeah, this isn't related to that and yes to your point, there is an issue open around around running in FIPS mode, specifically, okay,.

F

Yeah I just want to make sure that I understood that's the case. All right, cool, Thanks, yeah.

B

Can you circle back to one of the initial steps where you like just before you ran Hjelm, you have cloned I, think the cng repo and then put your own custom values file in there. Can you, like sure what that directory structure looks like yes,.

A

I will show fact: let's, let's look at it in here, so what what you want to do is and I didn't need to do this because it was already in there so but I. But yes, this is good. We should show off now, so you first use this command helm, repo ad get lab and then HTTP : charts get lab io that will and again this will not work in an air-gapped environment.

A

You'll need to get the chart moved in a different way, but for folks who have internet access, this is the easy way to do it. So this pulls in our charts and then basically, you take our chart and you just layer a a values: Y Amal on top of it, which we did right here.

A

So you take the chart that we've already pulled down and then you add use F to say, apply this additional yeah mol file on top of it and that's this file here that we walked through and that specifies the settings we need to override that way. You don't need to mess around with the chart. You know you can pull down the chart and that there are distribution folks have put together and use it, and then you can store your customizations outside that in this file. So it just sort of makes everything easier to manage.

B

Thank you, yep all.

A

Right any other questions.

A

Okay, let's break something. So what should we break? A really common thing would be there'll, be some problem with the database, so let's break something there.

A

Let's just screw up the host, so we'll save that file out and then the way Hjelm works.

A

All we have to do is rerun that same command again and it will go in and make whatever changes we make on the system. Let me give you a one other warning about helm, while I'm thinking about it, something we ran into a lot during our testing was if a helm install fails, it's not great about backing out everything.

A

It's done so, if you're doing a helm, install and it fails out in the middle, make sure you're looking through using your your cube, cuddle, get commands and look at pods stateful sets secrets, config, Maps kind of go through the kubernetes documentation and look through a number of different. You know sort of all the key things that it has you look at.

A

I won't go through all of them now, even if I could and make sure that your your setup is clean and if it's a situation where the easiest thing to do like in eks, if there's nothing else in there is just kill the cluster and rebuild it. You might want to think about that.

A

Helm is just not not perfect at this point at keeping everything clean in the event of a failure midstream, so just to mention that all right, so, let's, let's run upgrade and we'll install with our change that has now broken the database you.

A

So that should run for just a moment.

A

And when that's done, we will run our get pods and see okay, so that's deployed.

A

Let's see if we were able to successfully break things okay, so what you can see here is that sidekick and unicorn are saying in it zero of three a knit two of three here. What that means is there are these containers called in it, containers that run inside a pot when they first start and two of three means that not all of them have run yet zero of 3 means none of them have run yet. So you can see it's it's there's something. It may be that they'll finish in time.

A

So sometimes when you see this, what you want to do is just wait, but you know so now: it's gone from zero 3 2 to 3. So now, if we, if we keep waiting, I, think we're gonna, see that we're not gonna we're not gonna get anywhere. So here we have zero of two pods that we told that we wanted to have running ready here, zero of 2 here, zero one. So if you see this particularly after you wait a few minutes, it means something is wrong and what I would always recommend? I know.

A

You know we specifically broke the database this time, so we know what the problem is, but if you find yourself not sure what the problem is, I found that, at least in my experimenting with this environment, which involved dealing with a number of failures, most of them were related to the cluster not being able to see either the database or the cache.

A

There was one that was a bug that that has now had an issue opened and has been fixed with giddily, so you all shouldn't have to encounter that one, but mostly it was just situations due to V PC configurations or security groups or whatever, where it couldn't see. Okay, so yep. So this one has error it out now, and these two are basically stuck okay, so when a pod is behaving in a weird way, the first thing we want to do is run described pod.

A

And this will give us a bunch of good information on what the problem is. You can see it creates a whole lot of detail that you have to look through eventually with practice. You start to figure out exactly the part you're interested in so there's. We know there's a problem with our and knit containers, so we're gonna, look in our in knit containers section, so our certificates container completed so we're we're happy with that one. That's probably one of the ones that went okay, our configure container also completed.

A

So that's probably a good sign. Our dependencies container here has aired out, so we're gonna have to figure out what's wrong with that container specifically, so we can run.

A

The logs command, so first we'll just tell it we want to see the logs for the we got to be specific. So let's tell each pod has multiple containers in it. So if we want the logs, we have to tell it which container we want. So the dependencies container was the one that aired out on us. So that's the one we want to see.

A

Okay, so the last thing we see here is checking database connection and schema version and then nothing so that could be. That could be a clue. Now again, you know we know what happens so.

A

That's you know an interesting possibility for for what we need to do there, what we, what we could do if we wanted was we could connect to one of these pods now, if we think it's a database problem, you know one easy way to see: what's going on is to just try to access the database from inside one of the pods, so we'll use the task runner pod, because I- and this is just a situation where I happen to know it has the stuff we need on it and it's running, whereas some of these that have errored out may not be that useful for us.

A

So real quick, let me just see if we get we got, we have anything new, yet ah we do by waiting a minute. We we now have this, which is a better error message. Okay, so at this point we know it's the database, so we can, we can fix it, but let's just for the let's. Just for our purposes assume this didn't this didn't go. Well, then we didn't get this information and let's I'll, just show you how to jump into a pod.

A

So let's look at our pods again. We use the task runner and again we're jumping into a specific container in this case, I think the task runner. This is the only thing running in it. So I think we should just be able to say you've could all exec T I put in the pod name, and then you have to tell it something to run. So if you want a shell, we'll just tell it, we want bash alright.

A

Now we are inside the container, and so you know if you're, if you're having permission problems or this is how we diagnose the get Ally problem. For example, we, it was a permissions issue and we figured it out by jumping into the pod in question and going and looking at the permissions on that particular folder and then so in here like. If there was a, if you wanted to diagnose a Postgres problem, you could you know just run whatever you know, psql or you know.

A

Sometimes you might want to try pinging things or whatever you need to do so. That's.

B

How you girls, console.

A

B

Sorry, can you get to the rails console and that one.

A

Yes, you can- and this is also a good place. I don't know if I have the command handy.

A

Yeah yeah we have rake in here. This is actually where you can run. Sometimes you need to do rakes to do things like, let's say like in, like in this example, I stood up an RDS database and didn't have a super easy way, handy with what I had to actually create the initial database for for get lab. So I actually use this to do that. I got in here and then let me see if I can find the command.

A

That's handy, I got in here and basically ran the DB, create command to create it myself and I think it would be something like get lab rake. Do we have that.

C

A

It's good, it's got a lab rake, but I don't want to run the command again because it's already there so yeah. This good.

B

It wouldn't be able to see the database because it's because.

A

It's broken, that's that's actually, a good point. We could go ahead and run it.

A

So you run that and it'll, probably just timeout after a bit so yeah it's you know. Ideally you can use the logs command to to get to what you want, but you know sometimes you'll go through you'll, look at all the logs and you still won't quite know or you'll think you know what it is, but you'll need a way to to confirm it, and in that case the easiest thing to do is just drop into the container. You need and try running the commands. The one other thing I'll say is sometimes containers in kubernetes.

A

If a container has a aliveness or readiness check enabled the the scheduler might keep killing it and recreating it. So what you may have to do before you jump into a pod, is you may have to disable those and you can use the cube cuddle edit command and you can go in and monkey around that stuff all right. So let's do one more real, quick one. So, let's.

A

So yeah, okay, so let's do one more thing: that's very likely to be a problem which is, let's change, one of the tags so that it's wrong.

A

All right and then let's run our.

A

Let's run our upgrade one more time. This is something particularly if you're doing a ubi install where you have, and you want a specific version where you have to specify all the tags or, if you're doing an air-gapped solution or otherwise you're using a custom. You know busily you're using a registry other than ours you're, it's very likely. You could run into a situation where you'll you'll, you won't get a tag quite right or you won't specify the repository quite right or there'll, be a network problem that keeps you from getting to the repository.

A

I mean there are any one of a number of reasons why you might have issues. So, let's see which one did we, which one did we screw up? Unicorn? Okay, so we'll do our same thing as usual: we're gonna do get pods and.

A

Then we're gonna do well we'll just describe pod, which again is the second. Usually the second thing we do when we see something, that's doesn't seem to be right.

A

All right this is the error you're looking for back off, pulling image whatever it gives you the specific thing and then colon and then the tag and it's as error, image pull back up. You can see the same you can see here. This is not found, so this is again you're likely to run into these so just be prepared. Whenever you see this, it just basically means either your tag is wrong or you're.

A

You know location where to get the container is wrong or if you're, using your own registry, you know, maybe the container you need isn't in your registry, but you know, take a quick look at that and that's a good starting point. So all right, so those are sort of the couple of things you're most likely to run into. Obviously, everybody's gonna have their own.

A

You know problems that they run into based on their environment and how things are set up, and you know you just have to get used to walking through sort of these standard steps. First, look at your pods see if they all started. If they didn't use, describe use logs. If you have to jump into a pod and then if you want, if basically get labs already running- and you want to tail the logs while you're doing that, take a look at the cube tail command. Let's do.

A

On my machine it's listed as Katie, so you can see what it does it if you just say, get lab Unicorn it'll find all the pods that start with get lab unicorn and then it'll take all the containers inside those pods. It will color code them and then tail all of them at the same time, and so you can do it that way.

A

Now, if I were actually solving a problem with this I'd, probably be more specific, because I probably don't want to see 15 logs at the same time, probably too confusing to figure out what's going on, but especially because you can see the readiness checks are coming over, so you'll see the logs for those. So you can be more specific, but if you you know, if you want to look at a few different logs at the same time, it's convenient for that.

A

So those techniques, hopefully, are a good starting point and you can get from that to what you need. Also, if it's a if it's sort of a problem, that's you know that we need to work on. Hopefully, these things will get you the information. You need to open a good issue so that we can quickly get the problem solved all right well and are there any other questions otherwise, I'll leave it there.

D

Real quick Steven: this is that, do you recommend practicing this at Amazon or or is there a different environment that we should be using well.

A

I, like I like using Amazon, because it was easy to get the cluster and the database and the other stuff stood up. It sort of depends on what environment you have the thing I'll say: is it's pretty expensive to run these things on Amazon? So if you're gonna, do it make sure that you know whoever's in charge of the budget on your Amazon? Account is ok with it and try to get everything set up work with it, make sure it's working and then make sure you go back and kill everything that's been created.

A

You can use eks cuddle to remove your cluster, but it doesn't quite remove everything and I think it removes all the expensive stuff, but you'll want to go back and check and make sure you know all of your all of your instances that cost money- and you know for the sake of keeping everything clean. You know getting rid of your VPC and everything else that it creates in the you know, in the effort to get your your cluster set up, probably also good.

A

So, as soon as this is over, I will now be going in and doing a massive cleanup on our on our cluster. Also, you may find that, depending on what else is going on, you may not have enough subnets or easy two instances or whatever I had to I, had to find a region where we didn't have that much stuff installed to be able to do this, I started with standard US, East and, and the cluster fail.

A

The cluster failed, creating because there were a number of resources where we had already hit our maximums or were close to them. So just you know be be cognizant of of what it costs to get them set up. That way now, I'm attempting it on I've got a computer here that has 32 gigs of memory, so I've created or VMs to workers and a master and a client machine and I'm I'm working it out in that environment. That seems to be sufficient for this sort of testing as well.

A

So you know it just depends, but be careful that you don't run up a huge bill. Yeah.

D

I'm definitely interested in what happens with your home lab there. As far as what you stood up at Amazon. Do you have a feeling for how many users that deployment that specific deployment could could handle is that in the charts, documentation, I didn't notice.

A

I believe we are starting to put that information into the charts documentation. I, don't think it was in there previously, but I believe the distribution team is working on some sort of reference architectures or, at the very least, some notions of what sort of different different numbers of pods and things like that will get you different levels, but for my purposes here, I never put a lot of stress on this thing. It was, you know, just me, basically using it so yeah, I I couldn't speak to it myself. Okay,.

D

C

A

My guess is hundreds of users, but not a huge number.

A

All right any other questions, even.

C

The samara, quick question and you may have covered this at the beginning of the call. So if you had you know just tell me to go, watch the recording and they'll be fine, but with regards to installing this a following up what ed just asked, we have a Google cluster that we've been given as part of the customer success piece of it. Can we use that to install this, to try it out, or is that too small for this I believe.

A

It's too small, yes,.

B

It is too small yep.

A

It's it's yeah I would use it for I would deploy to it from get lab, but I wouldn't put get light on it.

C

Cool and the other question is you happen to use the hardened container, but really I mean this? Could this same thing can also be used just to do the helm, install I mean the container is just a label that you pulled out of whichever repository you wanted to use right absolutely.

A

Yep, all you really need to do is change or I believe even just removing these tags will get you the standard latest tag which won't standard latest containers which won't be you bi8, but they will know yeah this. This absolutely applies to whatever environment I just chose the ubi eighths. In this case, because I know there are a number of folks who are interested in seeing how to install that yep all right, I'm going to go ahead and stop recording. Now you.