Cloud Native Computing Foundation Kubernetes Community Days (KCD) Chennai 2022, 30 Jul 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Creating Serverless CI/CD on Kubernetes using GitOps by Avik Kundu

Description

Serverless has become one of the most used terms for cloud computing. In a Serverless 1.0 world, cloud vendors created independent products without concern for portability or migration. This problem was solved with Serverless 2.0 which made workloads far more portable. Enterprises have started to adopt and maintain several open-source projects which tend to bring all the advantages of serverless to the way applications are deployed on Kubernetes currently. Cloud-native CI/CD tools are one of the best choices as they can fully exploit the inherent benefits of the cloud, making solutions efficient, resilient, dynamically scalable, and always available. GitOps has become increasingly popular in the DevOps space. Teams have been adopting GitOps in order to better deploy, manage and monitor Kubernetes clusters and applications. This presentation would demonstrate how to create Cloud-Native CI/CD Pipelines for deploying serverless applications on Kubernetes following GitOps Practices.

A

So, a very good afternoon, everyone welcome once again to kcd chennai 2022., so I am avi and today I'm going to present you a talk on creating serverless ci cd on kubernetes with githubs, so this talk will be beneficial for those who are trying to understand what you mean by serverless on kubernetes and how to set up ci city pipelines on kubernetes for deploying serverless application following the github's principles, a bit about myself, I'm currently a software engineer intern at red hat india. I currently contribute to projects like openshift, techton, captain, etc.

A

I like to engage with new people and contribute to various projects every month you can follow me on twitter, linkedin or github. I also write monthly medium articles, so you can check those out too now, let's understand what do you mean by serverless? So serverless is just an abstract term and is not to be taken literally in the same way as cloud computing does not involve the sky. Serverless computing does not mean that the code is executing without service.

A

Basically, it refers to the experience. A user or a customer has and represents a continuum of how close one needs to work with hardware and infrastructure. In simple terms, it means there is much focus on code, it's a even driven architecture and it's more easier to manage, because it's stateless and isomorphic.

A

So we see there is a gradual decrease in the concern for infrastructure and a gradual increase for the decomposition of the workloads. Now, let's understand the different categories of products when we are trying to exclude the serverless ecosystem, so we see two categories, mainly that is serverless 1.2 and a serverless 2.0.

A

The surface 1.0 revolves around the products launched by the various cloud providers. It can be aws lambda provided by amazon or azure functions provided by azure. So the main issue with this kind of services is that they are not portable enough because it becomes difficult to transfer applications from one cloud to another. This is because the signatures are different and the way the zip file is constructed is also different for different clouds.

A

This problem of portability is solved in serverless 2.0 applications because they follow a universal definition. That is, the applications will be stored in oci, compatible container images. It will be exposing the http server at port 8080 and it can be configured with the help of environment variables.

A

Thus any application written in python or any other binaries could be easily moved between the seven serverless 2.0 platforms like open, fast or creative, with ease.

A

So now, let's move on to one of the applications we are going to use in our demonstration that is scannative, so k native is an open source enterprise level solution for building serverless and event-driven applications, originally developed by google k native now has contributors from ibm, red hat and vmware.

A

So the main purpose of using genitive is to deploy serverless containers on top of cubicles. So currently, kenyattv has two different components. The first one is creative serving which is responsible for rapid deployment and auto scaling of serverless containers, and, of course we have canadian event which is responsible for enabling developers for setting up the even-driven architectures.

A

Now, let's understand the architecture of k native, so we have the compute cluster, on top of which we have the communities installed. Now, before installing canada, we need to have a networking layer. So in this example, we are using issue, but you can use other applications like contour courier, etc.

A

Then we install the k native application, so, with the help of k native, we will be able to deploy applications on kubernetes within without having to write about the deployment or the service yaml files. How does this work so for that? We have the right diagram, so the most important thing of k native is the k native service file. So in this service file we have to mention the container image that should be deployed, as well as the environment variables followed by the parameters.

A

So whenever any is any change in this file, a revision is created which is snapshot of the application code. So now, whenever there is a new revision created, the traffic is routed to this new division, but still we can control the traffic with the help of the native service file. Thus, with the help of a single yaml file, we are able to deploy applications on kubernetes without having to write the standard, yaml files that is required for deploying applications.

A

So we have understood how to deploy serverless applications on kubernetes with the help of k native, but now there is a need to create a ci cd system to automate this process of deployment. So here comes tikton, so techton is a powerful, open source cloud native framework for creating ci cd systems.

A

Originally, it was known as sk native build, but it spun off as a separate project and is one of the incubating projects from the cd foundation. So the main feature of tecton is that it's cloud native, so it can be deployed on any kubernetes cluster across the multiple hybrid cloud. Riders techton is made up of two main components: that is the tectonic pipelines and electron triggers. Tecton pipelines is more responsible for providing kubernetes style.

A

Yaml based resources for declaring the ci series pipelines and tecton triggers helps you to create the kubernetes resources based on the information about the even payloads. So now, let's see the architecture of tectonic pipelines, so the smallest resource in tecton pipelines is known as a step. So step is a job that you want to automate.

A

So a collection of steps make a task. A collection of tasks make a pipeline now to run a task. You need to have another resource known as a task run while to run a pipeline. You need another resource known as a pipeline one.

A

Now, let's look at the architecture of tecton triggers, so tecton triggers also has various resources. The most important one is the event listener board. So this pod is continuously listening to any event that is coming from outside the cluster. So whenever it receives an event, it sends it information to trigger binding a trigger binding extracts, the important information and sends it as parameters to a to your template.

A

A trigger template in turns fires, the pipeline one, which is in turn responsible for starting off the pipeline. Thus, with the help of tecton triggers and tecton pipelines, we are able to achieve a fully ci cd system. So now, let's understand what you mean by githubs and argo cd. So get ops is a way of implementing continuous deployment for cloud native applications, so it focuses on a developer, centric experience when operating with infrastructures with the help of tools the developer are already familiar with, including git and other continuous deployment tools.

A

So sargo cd is a tool that follows the getup pattern of git repositories as a single source for truth for defining the desired application state. So the main feature of argo cd is that it can help in automated deployment of applications to specified target environments.

A

So now, let's understand the cicd workflow of the demonstration we are going to see before that. Let's understand what we are trying to achieve. So whenever a developer will push some new changes in the application repository, he must be seeing the latest code changes back into the website url.

A

So for that we need to create the cli series system. So, first of all, whenever the developer will push the application code into the application repository, a web hook will be triggered which will inform the event listener that is running inside the capabilities cluster. That a event has been occurred in the application repository the event listener will then transfer the response to the trigger binding. The trigger binding and turn will extract the important information like the commit url and the commit id and pass it to the trigger template.

A

The trigger template in turn will use these information as parameters to run the pipeline one and, as we know, the pipeline one is responsible for running the pipeline, so the pipeline will be running the three tasks that is listed here, so the first task will be cloning, the source repository.

A

That means the application repository will be cloned inside the tecton environment. After that the application repository contains a docker file. So with the docker file we will be able to build the docker container and push that container image into docker hub. After that, the third task will be fetching. The latest github's configuration details from the github's configuration requestory, so this repository contains the cognitive ml files as well other configuration files.

A

So after fetching these files, our task will be to update the canadian service yaml so that it now contains the information about the latest docker container that has been pushed to dock. So after this change is made, then the updated file is again pushed into the keyboard's configuration repository, so this entire process must be done automatically.

A

On the other hand, argo city is constantly looking at the github's configuration repository, so whenever it sees there is a change in the files in the configuration repository it understands it is out of sync, so it fetches the latest files from the repository and then applies those in the cognitive environment that is underlying inside the kubernetes cluster.

A

So now, with the help of the creative service file, which has been updated with the latest docker image, the application developer can see the latest code change in the application url.

A

So now, let's explore the code base to understand how the configuration files for managing the k native, tikton and argo cd are written. So currently, I've already started a two node eks cluster on amazon, with the help of ek, ctl command line. So currently you can see my vs code screen in which all the configuration files, as well as the application code is present, so let's browse to each and every file. So first, let's see the application code. So here we have a simple golang application with a goodmain.gov file.

A

So in this main.go file we have a hello world that is going to be printed on the website. Apart from this, we also have a docker file, so this docker file will be used to create the docker containers in our ci cicd pipeline.

A

So next we have the k native service yaml file, so in the service eml file, the most important thing is the container image name. Here you can see it is directing to the go sample app that is present in my docker registry. Apart from this, you can also see a docker tag mentioned. This is important because we are going to update this docker tag automatically with the help of the ci cd pipelines in the further steps.

A

So, finally, let's see the configurations for the tecton pipelines and figures.

A

So, first, let's start off with the explanation of the task configuration so here we can see there are two different files for the two tasks, but according to the diagram before we have three different tasks, so the first task that is the cloning of the git repository, is a very common task and is present in almost all the pipelines.

A

Therefore, rather than writing the task manually, I have downloaded the task from the tickton hub, so tickton hub is a marketplace from where you can download pre-created tasks and use them directly in your pipeline. So from this I have downloaded this git clone task and I've installed it with this simple command.

A

So after installing this task, I'm able to use the starts directly in my pipeline. So apart from this task, I have written this builder task, so this builder task is responsible for building tagging and pushing the docker container into docker hub. So this task contains a multiple steps. The first step is building the docker container, with the help of the docker file already present in the application requestory.

A

So once the docker container is built, then it is stacked in the second step. The final step involves pushing the docker container into the docker hub. So, finally, with this task, I am able to build tag and post attacker containers to docker hub, so the final task involves updating the github's configuration repository with the latest code change, so this task is also consists of three steps.

A

The first step involves checking out the code from the github's configuration repository. The second step involves updating the kinetic service yaml file with the docker tag of the new container. That just has been pushed to docker hub and we are able to achieve this with the help of a utility known as yq.

A

The final step of this task involves pushing the latest code changes back into the github's configuration requestory. So for this we have already said the ssh keys so that automatically a commit can be made.

A

Next, let's see the configuration for the pipeline resource for that, we have defined the pipeline.ml file. So in this file we are going to specify all the configurations that are required to setup the pipeline resource. We already know that a pipeline constitutes a multiple tasks, so in this file, specifically, we are mentioning which are the tasks that are going to be executed in this pipeline. So you can see here. The first task is a fetch repository task. This task is using the git loan task. We have downloaded from the tecton hub apart from this.

A

The second task is the build and push task, which is using the builder task we had defined before, and the final task is the update, docker tag, config repo, which is using the update, github's repo task we had defined before so these three tasks are going to run sequentially one after the other to ensure this. We have used the run after parameters in this.

A

So finally, we are able to create the pipeline to set up the continuous integration process for deploying our application. So till now we have set up the tecton pipelines, but to automatically trigger these pipelines whenever there will be any code push inside the application data repository, we need to set up detect on triggers, so for that we first set up the event listener. So, whenever a user will push some new code inside the application, github repository, our webhook will be triggered. This webhook will send the information about the latest comment to the event listener.

A

Now to the event, listener has to understand this comment so for that we are using the github listener. So once the event listener receives the payload response. Now it will send that payload response to the trigger binding now the trigger binding will in turn extract the important information from the payload response, that is, the clone url and the commit id. So once it extracts those information, it will send this as parameters to the trigger template. So now the trigger template is responsible for starting off the pipelines with the help of the pipeline one resource.

A

So here you can see we are defining the pipeline one resource and also passing the parameters that are required to run the pipeline. So thus, with the help of tecton triggers as well as tecton pipelines, we are able to create the ci platform. So whenever a user will post some code changes automatically, tecton pipelines will run which will build and push the container to docker hub and also update the kdp service cml file with the latest docker image tag.

A

So finally, let's see the configuration for the argo c so here for argo cd, the configuration is very simple, so most important thing is the repository url that argo cd will continuously look into for any code change. Apart from this, we can also mention the destination name space in which the argo cd will install the kubernetes application.

A

Apart from that, we can also mention some of the same policies and the sync options. Therefore, by applying all these configuration yaml files inside the capabilities cluster, we will be able to successfully configure the k native pikton and argo cd to set up the entire serverless ci cd for deploying applications on kubernetes.

A

So, finally, let's see the workflow demonstration, so in the current screen you can see the application code repository. So here we have a simple go application, so here you can see it's going to print hello world on the screen. So let me show you the initial website.

A

So currently you can see here, hello world is printed so now, if I go back and if I edit it to something like hello.

A

Kcd channel and then, if I commit it.

A

Now, if I go to the tickton dashboard, I can see here, a new pipeline has started to run. So if I click on that, I can see the three tasks that we had already defined before. That is the fetch repository, build and push image and update the docker tag, so the fetch repository is trying to clone the application and then build and push image is trying to build the docker container with the docker file that is present in application source requestry, and then it's going to push it to the docker hub.

A

After that, the third task will run which will update the docker tag in the canadian service yaml file. So, finally, all the tasks has been completed. Now, let's go to the docker hub here I can see that a new container has been pushed a few seconds ago. Now, let me go to the github's config repository here. Also, I can see a code change or a code commit has been done 28 seconds ago.

A

So if I click the comments and see what is the change that has been done here, I'll see, the docker container tag has been updated. Now this tag is exactly the same tag that has been updated here so now. I can understand that the cognitive service eml file has been updated with the latest docker tag automatically by the tecton pipelines.

A

So this completes the three tasks of the tecton pipelines.

A

So, finally, if I head back to the argo cd portal, I can see the sync status is okay and also I can see it has succeeded a few seconds ago and also, if I compare the head commit id that is c0cc with the commit id present here in the repository that is also c0cc.

A

That means argo cd had understood. A new comment has been done in the config repository. It had pulled the new change and also applied it into the community's cluster. So now, if I head back to the original website- and if I click the refresh button, I can see it has changed to hello, kcd channel.

A

So by this demonstration we can see whenever a developer will change the code base in the application called repository automatically, the tickton pipelines will run, which will create and build the container and push it to docker hub. Then the cognitive service ml file will be updated with the latest docker tag and finally, we will be able to see the updated information on the website url.

A

So with that, we have come to the end to the presentation you can find the application code repository as well as a github configuration repository with the following links. Apart from that, I've also listed down the documentations of the projects we have used. If you have any queries or suggestions, you can reach out to me on twitter linkedin or on cncf slack. You can also scan this qr code to get all my social accounts.

A

Apart from that, I am also going to write a written walkthrough of the entire demonstration. So for that, keep you to my medium page, so I hope you have loved this presentation. Thank you all for joining in.