GitLab Configure Stage, 13 Jan 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: GitLab Kubernetes Agent Overview + Demo

Description

In this video, the presenter is explaining the value of using GitLab Kubernetes Agent (https://docs.gitlab.com/ee/user/clusters/agent/#gitlab-agent-gitops-workflow), its architecture, along with a demonstration of how GitLab Kubernetes Agent can monitor several repositories and reflect the changes directly into the Kubernetes cluster

A

Hello, everyone thank you for joining me in this session. uh My name is sam rakub and in today's session we will be talking about githubs and in general, and focusing on the recently released capabilities and capability in gitlab around gitlab kubernetes agent.

A

uh These are my contacts. Please feel free to reach me after a decision if you need any help or any further clarification. So, basically in the session today, we'll touch on what is github's overall. What are the new capabilities in the uh get club kubernetes agent?

A

And what are they basically, uh the architecture of these agents and uh we'll in decision with a demo which I hope you will enjoy so basically, when we talk about github's uh in in simply, it is taking the operational from framework that and best practices from devops and applying them into infrastructure automation into the operational side.

A

So we are in basically enabling the operations team to enjoy the same benefits the devops team have been, or the development team have been enjoying in terms of automation, source code management, uh uh uh collaboration, compliance security, uh all that and applied these same practices into the uh operation, site and infrastructure automation site.

A

So just jumping direct into the the subject today in order to define and add a kubernetes agent into a gitlab instance.

A

Customers have to provide, of course, the kubernetes cluster name, the environment name, which this cluster would cover uh production, staging testing or simply everything and three main things: the the api and the url for that kubernetes cluster, the cs certificate and the service token, so that service token currently has to have a cluster and privilege and to understand what does that mean? I copied the definition of a cluster admin privilege from the keeping that is documentation and basically it gives full control over every resource in the cluster and in all namespaces.

A

Of course, many customers. I've worked with have raised their concern around this level. Of of uh I mean exposure, I mean to that to the gitlab instance. Now, of course, if the gitlab instance is a self-managed instance within the data center of the customer and most probably next to the kubernetes cluster, the risk is minimized.

A

But when you're talking about exposing the local kubernetes cluster and integrating that into uh let's say a remote or even a sales version of gitlab on gitlab.com, many customers have raised their consent. That yeah we don't really want to expose or provide and define the service token in uh directly inside gitlab, and we don't want to uh even expose the url directly to the internet to be integrated with that remote instance of gitlab- and this is where kubernetes or gitlab kubernetes agent came uh to help so basically think of the kubernetes agent.

A

As a to uh let's say, a client server solution. You have the agent uh k which we refer to as the agent k here, which think of it as the client for the keeper the kubernetes agent, and you have, uh on the other side, the cupola's application server, which is deployed or will be deployed within the gitlab instance itself.

A

So what we're trying to do here is uh we will define inside the agents that sorry we will define uh to the to the agent. What are the projects that this kubernetes agent should monitor for changes and these projects will normally host the deployment scripts for the client applications so think of it? This way you have. uh You want to deploy a simple nginx deployment right to a coupon. That is you to your kubernetes cluster.

A

You will have a repo inside github and here in general, github uh self, managed or or or in the cloud you will have a repo, that's maintaining your deployment uh manifest for the engineers of course right. Then we will tell the agent that sits within your kubernetes agent.

A

That, mr agent, can you please uh instruct the applicant, the git lab sorry, the k club agent server to keep an eye on that ripple and whenever any change happens to that repo, can you please let me know so I can reflect that change right, so sequence is mr agent will be configured and we will see that in a second, we will be configured that mr agent. Please tell the git club server on the server side that please keep an eye because it is sitting within gitlab.

A

It can have direct ass access to the italy where the ripples or the applications are are maintained. Can you please keep an eye on my in nginx project? So whenever any change happens here, this grpc is a live streaming right. It's a it's a long, lasting connection. So it's a almost a real time whenever a change happens here, that change will be communicated to end to that gitlab server, which will get that agent server, which will communicate that to the agent that sits within your kubernetes uh cluster and will reflect that.

A

So what does that mean? It means that if you'll go here, it means that this is uh no more an admin level. Access is required to do the kubernetes integration right, because that agent was responsible for the actual deployment inside your cluster, the level of access it requires. It really depends on what applications you want him to deploy right. So, if you're, just talking about just running deployment, a minimum, a minimum level of access to your cluster may be required.

A

If you are thinking that you wanted to do to do bigger deployment or more on operational tasks, maybe you will need to increase that access, but remember that agent is sitting within your kubernetes uh cluster right, so it is controlled by within your data center, no more api, uh direct integration and cluster level token, as is required right.

A

Second, it is, as I said before, this agent architecture is an active uh active in the cluster for solving gitlab kubernetes tasks right because, okay, listen, you know what let's do it as as a flow as you see here, I'm a developer.

A

I have I'm working on my let's say nginx uh deployment or adding features to the nginx deployment. I'm pushing that code to my local repository and that repository has the manifest file the ml file for my deployment. Remember this kubernetes agent, which is in that in this instance, the server side of the agent is monitoring. Your repository is monitoring because and the other side best part of the agent configuration.

A

We have told this agent, the server side of the agent which repos to monitor. So it is monitoring my repository and whenever I push a change here, that change will be picked by that agent and then it will be communicated back to the client side of that agent, which sits inside your your your cluster, so that that deployment will happen. I hope I'm trying to um to make things uh easier to to hear. So, let's go back a little bit just to recap again.

A

The kupernetes agent architecture has two parts, the client part and the server part right. The client part is an agent deployment. That's deployed inside your kubernetes cluster inside the name space you will create as part of the agent deployment and the server side is where the actual monitoring and the command and control is happening right. So when the coupon natives agent is configured, the server side will be configured to monitor certain ripples in your gitlab deployment.

A

Your projects, the purpose of that monitoring, is that any changes on the scripts of that of these ripples will be picked by the server side of the qp net application agent uh agent and then will be communicated, will be pushed into the kubernetes agent that sits within your cuban axis cluster to be to be reflected so that changes will be reflected by the way kubernetes uh github kubernetes agent is built using the same git ops engine, which was built by the same team who built the argo cd engine, and this is basically training and the infrastructure as a code.

A

So maintaining it's a declarative way of doing operations for uh the kubernetes deployments and as deployments in in our operations in general right. So simply the kubernetes agent, two parts, client side and server side. The client side is the execution engine. The server side is the monitor and command and control engine. The server side will be instructed to monitor whatever ripples you have wherever you have your deploy, kubernetes uh deployments or scripts, or applications right or cloud native applications.

A

Whenever a change happened there, that will be automatically picked by the server side, because that server side has a direct, continuous uh uh uh streaming with a guitar engine where, basically your ripples are stored, then these changes will be pushed to the uh to the agent and that agent will execute them will reflect them to your keepness.

A

A very important point here right. This will enable uh the integration with cubernet is agent or clusters which are set behind firewalls or uh nets right, because here we are the connection. This is a little bit of networking. The connection is initialized from the client into the server right. It's so okay, so and, and that connection is continuous, using grpc uh uh calls right and it is a streaming connection right.

A

So whenever changes are happening here that will be sent from the cuban agent server as a reply to the request, that's initiated from the client that sits on in your cluster, that's why it can communicate back so simply if the agent would send behind the firewall in your net. If that agent can communicate to the gitlab server, the gitlab server uh agent server can communicate back that change, changes and instruct this execution engine to reflect these changes right. I hope that makes uh things a little uh a bit clearer.

A

So you know what the easiest is. Let's have, let's, let's have a look on the demo, so here I have so to make it clear so far, kubernetes gitlab kubernetes agents are only available in the self-managed, as we speak are only available in the self-managed git lab and the roadmap. You can view it on a gitlab roadmap. You can view when that will be available within the gitlab.com.

A

Second caveat here is the applications that the gitlab application- uh sorry, the kubernetes agent uh server- will monitor, have to be on a public uh like uh publicly available have to be uh uh like um uh have to have a public privilege right again, this will change and as we as the development team in gitlab are adding more features, as you know, uh every every month, so I prepared something here like: let's have a look on how things are are working. Actually uh this is my these are uh I'm. I have my mini cube deployment.

A

By the way uh I can provide full documentation on the deployment we will go through. I have that all documented and more than happy to uh to provide it to you just reach out to me, and I have a repo with all the scripts and a detailed description on how to rebuild this demo.

A

If you want so this is my self-managed gitlab hosted on my local local mini cube cluster right on on my machine and, as you can see here, I have my git lab namespace right and I have another namespace for the gitlab agent, the gitlab. As we agree, the github server namespace will have and should have the kubernetes server side agent server side deployed. Let's have a look. So if I do cube ctl.

A

uh Again so it's best to move that one.

A

So you see here that this is the pot for the git lab agent server, that's hosted within the gitlab uh uh namespace. On the other side, if I do cube ctl git for me, pots inside.

A

Lab agent, that is the video. This is the agent the agent that's deployed in my uh in my cluster. In normal day, these two things will be deployed, maybe in a totally two different clusters. I have chosen to deploy my git lab within kubernetes. You can use or deploy gitlab anywhere. As you know, you can deploy it outside the keyboard networks that doesn't really matter matter. You can use the omnibus to deploy the gitlab.

A

In my case, it just happened just for ease, I deployed my geek lab on the kubernetes club, silver and cluster and in the same time, I've created another name space for my kubernetes agent, but don't get confused because usually these two can be can be uh deployed and living in two and two totally different clusters right. So to start with, you see, you will see here that I have. I have a project right here.

A

That's that has this is the configuration project for my cube agent right and as part of the deployment you will get into uh or enabling of the kubernetes agent you will get and enable these uh configure the kubernetes agent with the path to that. To this repo and this repo will have a link or the file or will store the file. The config.tml file, which is basically, is telling my gitlab agent.

A

What are the projects? I want you, mr agent, to monitor, remember going back here.

A

We are talking about a agent and a server right, and now um you will configure the server to instruct it to monitor certain ripples where, whenever changes, I do, I sorry what whatever changes.

A

Whatever changes, I do in my ripples, which are managed inside italy. Please get upset agent saver, reflect them and pass these changes into the agent site right.

A

So, in my case, I have two projects right, one, the first one- and this and the second one. Let's have a look on them right, let's go into one of them, which is, for example, let's start with the first makes more sense right. So this is my ninja deployment project and again, as you see here, I'm telling the gitlab agent please monitor for me this project- and this is the manifest.tml file here. So I am now an operation guy. I am responsible for maintaining and managing this this uh this.

A

uh I mean diploma nginx deployment to keep an eye on the right side. Let's do something fun to say, as you can see here, this deployment will go into this namespace. So if I keep, let's keep an eye on this.

A

A

Okay, as you see here, it's maintaining three instances of from the first project and three instances from the second project. Again, these are these two, because I'm telling the gitlab agent, given its agent to monitor these two projects. Let's do something here: fun, let's say: okay, I have decided to scale down this edited thanks to gitlab. I can directly edit that into in gitlab using the web, ide change the replicas to four right, that's it! This is all what I need to do now.

A

Let's see what will happen commit to the master, keep an eye on the right side. Now, what's happening, gitlab application server side is picking up, should pick up the change and the change is uh one more part is inc should be added. That change should now be communicated to the cuponetis. uh Here we go to the cuban that is agent that sits on my cluster and that kubernetes agent is now provisioning, my engine right or my extra sorry, my extra nginx uh pot, and to make it even let's have a look on the pod cube.

A

Ctl get uh like. Let's say you know what logs this one happen and get the agent here we go. This is the latest right. This is the thing that has changed just now, so this is the logs from the cube agent that sits inside my gitlab, something uh to show you as cube ctl get let's say: first, you know what let's clear here and cube ctl get all hyphen and cube uh get lab agent. I just want to show you something you see here.

A

This one has the deployment for the agent itself as the code running for the agent and the replicas it's for the agent. Can I run more than one agent? Absolutely yes, actually for an interesting point, I'll show you the deployment uh script or manifest for the agent itself. So if I go uh resources actually let me show you the deployment command for the agent say: cat install uh okay, let's go uh the resources for the agent, so this is the deployment script for the coupon gitlab agent.

A

uh It will create a service account. In my case, I am giving it a cluster role just because I want to give it a a full access. I can minimize that role or that privilege to a very custom one or minimized one. That's only required in my case, I'm just deploying I'm just doing deployments uh into the kubernetes, so a a privilege that just enable me to do to do deployment objects should be enough for for this use case and this demo.

A

This is what I mean before it was mandatory to have a cluster role privilege to do that integration. Now you can have uh that deployment without doing that uh without having that that living last thing, I want to show you as as fun as well as if I go back here on the project level.

A

You'll see that I have this uh this folder guest book. As I told you before you, the git lab kubernetes agent, is built using the git ops engine, the same engine that she built by the team who developed the argo arco cd- and this is argo cd right and nice thing here is, you can even add gitlab directly into argo cd, and this is what I've done and you can I'll show you I've added my gitlab repo.

A

This is my nginx demo deployment project, which is which is this one here, this one, my nginx deployment project right and now my argo cd is doing exactly what I my kubernetes agent, have enabled me uh to do. I'll, show you something so in in in this instance, it is looking onto a path of guest book. So this is the guest book book path.

A

Sorry- and let's have a look in here in the deployment- and you can see here- uh it is looking into the default namespace I'll show you something so qctl get ports, let's say, and this is it right and it is deploying against you, a guestbook ui into the default name namespace. So now the argo cd kubernetes agent is is looking into my my manifest file, which is hosted in the root, and I even just as a try.

A

Kubernetes agent is looking and monitoring this file, and I have configured my rbo cd to even monitor this, this folder right and to show you something: let's do something on the right: let's keep an eye on this one, and here let's go and change it.

A

To two and do comment and comment to master and then that's it now the same. What happened before with my kubernetes agent should happen here and because I've set up my um here, we go, it is changing, it is bending. So basically you can have arco cd deployed, you can have and it will visualize even the deployment from uh your equipment, this deployment, and it will reflect that into the uh your kubernetes server.

A

So, just to summarize, uh the problem that kubernetes agent came to solve is the ability to integrate uh kubernetes clusters with github server, with minimum required privileges and without exposing the api and access token and to the outside or to the server and without exposing the api. Sorry, the api url of the kubernetes server and to get them. How does it work?

A

Kubernetes agent has two main parts: the server side which is deployed inside the keeper, the gitlab server and the client side, which is deployed inside your kubernetes uh uh cluster, and that initially, this gitlab agent server side will be instructed to monitor repositories in in your gitlab environment, and these repositories, as we saw for the nginx, should or will contain the deployment scripts for your cloud native applications.

A

What it will do whenever a change is there, it will detect that change almost in real time and will communicate that into your locally deployed gitlab agent, client and the client will reflect that into your kubernetes agent. The access rights for this for the client is uh uh minimum and can be uh to uh can be set to up to your requirements based on the applications you are.

A

You are deploying and by the end of the demo, we also saw how to uh argo cd, which is which has the git ops engine used in the git kubernetes agent, can also be configured uh to read and reflect changes from your gitlab ripples. I hope that was useful. Thank you very much for for watching and again feel free to reach me on my uh details. I'm more than happy to have questions. Thank you very much see you bye.