GitLab GitLab Tech Talks, 19 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Get more efficient with CI/CD: Introduction to GitLab workshop for TalentQL Pipeline

Description

Slides with exercises: https://docs.google.com/presentation/d/1xs4-mlf5L95aER_pVo6i5U0d5DW4f_AIG0RKk8SpKIc/edit
Project with solutions: https://gitlab.com/gitlab-de/workshops/go-tanuki-cicd-workshop-talentql-pipeline-program
TalentQL Pipeline: https://pipeline.talentql.com/

TOC
7:08 First Steps
14:59 CI/CD: Terminology & First Steps
24:53 Pipeline Editor
34:37 CI/CD: Development Workflows
36:40 Go Tanuki Build
46:13 Pipeline jobs and stages
46:28 Go Tanuki Binary build and code tests with CI/CD
54:43 Run the binary
1:01:13 Visualize the pipeline
1:02:03 Faster feedback with CI/CD
1:04:03 JUnit test reports
1:06:03 Security
1:16:03 What's next
1:16:29 Exercises for async practice
1:19:29 Q&A

https://about.gitlab.com/blog/2021/10/19/top-10-gitlab-hacks/
https://about.gitlab.com/pricing/
https://about.gitlab.com/solutions/open-source/
https://about.gitlab.com/solutions/education/

A

Hi everyone. Thank you so much for joining me today. uh I think the last time we had one of this session was like two or three weeks ago. So it's really good to be back and I'm so excited about today's session. We have michael michael, is a senior developer in evangelist from gitlab and he's been in conversation. We think about this for a while now, which is really exciting, that he honored us to be part of this.

A

So today we are going to be talking about getting more efficient with cicd, and I would like to welcome my girl, hi michael.

B

Thanks for having me, um I'm hoping to provide you some insights and um yeah, trying to show you about insight in cicd and best practices and the like. um I know that we have um talked that talked about like which language we want to go for, um and I know that you all probably know was it nodejs and angular. um The problem.

C

B

I don't know that or I don't have deep knowledge in inside so for today we will be looking a little bit into go, but don't be don't be shy or don't be like freelancing about it. It's just like getting getting used to fixing codes, getting to see how the processes work and in the end we we just should should be having a good time and learning something.

B

That being said, I will quickly share my screen and I will also share the um the url for the for the slide deck with you um just because you can follow along for the exercises.

B

So, if you want you can um follow the exercises as we go along, um but before we kind of start into everything, just a quick introduction. um Okay, today is about gitlab. um If you want to reach me on social on twitter, on linkedin on gitlab.com, it's dns michi, which is dns m-I-c-h-I um miki, is the lovely form of marty in german.

B

I figured later that it's hard to pronounce in english, um but yeah. Once you got a nickname, you cannot change it. Somehow I'm coming originally from austria moved to to germany to nuremberg. A while ago, I've seen lots of things related to infrastructure, monitoring, ops being a back-end developer.

B

Myself started out with cnc plus plus um some years ago, or at least I think, 20 years ago, actually getting old, um yeah and and at a certain point um we got used to automating things um trying to get more efficient, moving on from git to gitlab github bitbucket, whatever is out there, and so I found my passion for also providing trainings um and making everyone like aware of, what's what's possible, what what we have um yeah and in 2020, or at least in late 2019, I saw a job at gitlab for a developer evangelist and I thought of well I'm an educator, I'm passionate about things.

B

I love to talk about new technology, and so I applied for the job and thankfully, in march 2020 everything worked out except for traveling, um but we will go figure and try to to make the best out of it. And the best thing about like doing this now is that I can show you something, even so, I'm located in germany and you are located somewhere else um but yeah. That being said in the next, I would say hour or something I wanna dive a little bit into like what what is gitlab.

B

What are the first steps? We will dive into cicd like getting started, learning the terms um choosing or looking into the pipeline editor, which is rather new and then practicing some some development workflows with the gotenoki that we will show show you later.

B

What that exactly means- and I do have some exercises and practices around security, continuous delivery, continuous deployment, thoughts, ideas, just giving you some insights on what to try next, possibly- and there are also some exercises for async practice in the slide deck we can touch base on that or we open up the session for and ask me anything.

B

Let's try this I have maybe expo. Maybe you have experience with git lab already or a different system, and you just want to ask a question. I'm I don't know everything, but I'm also going to try to google together so um yeah. It should be. It should be an uh a good session for you to learn and I'm trying to do my best to answer your questions later on. um The project is also public.

B

So if you want to like go along and do the exercises you can fork this repository. um Just click on like over here then select the name source where you want to fork it and inside the project settings in the general section, and I think it is in advanced, you can remove the fork relationship.

B

So it's like on your own pace, because otherwise you would be defaulting to creating a merge request into my repository, which probably is not what you want other than that if like, if I'm moving too fast or you just want to keep an eye out, what I'm doing um and and learning from that, I would recommend just following um the steps and the labs or um practice this later on async. So the project and the slides come back.

B

B

My microphone off now but yeah. I would encourage you to.

B

The headset is in the weekend already. um I would encourage you to practice later on. I think so. You can learn at your own pace um and there is no hurry to to understand everything in in this session today. um Please ask questions when they come up um or just put them in the subject, so we can like dive in later on uh when the exercises are being done.

B

um That being said, when we start looking into gitlab- and you probably open up the dashboard for the first time, you will see that there is like a top header, there's a menu header at the top. When you navigate into the projects later on, you will also see the left hand side menu. You can do specific actions, it's just like um getting used to how gitlab looks um and what you can like do in there.

B

So one of the things uh which is important to know is it uh that you can specify specific things in your profile. So it's like you, click on your avatar on the right hand, side or the right upper corner. um You can add your biography, but you can also manage things in there. So if you wanna sync, the git repository, there's a way to either specify or create an ssh public and private key pair and upload that or you create your own personal access, token for http access which you can use later on.

B

So this is like to keep your local clients um or your clients on servers in sync, with the gitlab server repository or get server repository in that regard, and there is a bunch of other things you can enable in there, like second factor authenticate, authentification and and other things, um the the other important thing or the other thing to know about gitlab when you navigate around is that there are namespaces or something like you can put a project in a group.

B

But this group can also be a subgroup of another group, so you can really like structure what what you're, building, what you're doing um and you're not bound to like the upper level uh projects in in the same name, space.

B

um The important thing about groups is when you're assigning specific permissions and roles, for example, maintainer, owner developer report and stuff, like that. This gets inherited to to the projects below. So you can build up your own permission level and say: okay um for our project.

B

um The initial creators are our owners, then we're at maintainers and specifically, if we need to grant access to someone else, we can do that on on on the day-to-day basis.

B

For example, um you can also add mention a group um in issues in comments and and other things which, which means you don't need to like type all the names within that mention, but can like say, hey I wanna, I wanna mention the deaf evangelism group, for example the marketing group, um which we sometimes do at gitlab ourselves, um because we use gitlab to develop gitlab or to create gitlab um yeah.

B

There is a. There are some other things which you can try out, so I will be jumping um rather rather fast over the topics, um but I would encourage you to explore what is possible and how to use gitlab in that specific regard, so that you can combine it with the knowledge about ci cd, which we will be jumping on in a little bit.

B

You have issue lists and boards you can search for for issues. You can use labels to group them to make them more discoverable.

B

You can search by author by by due date, by the most thumbs up for requested feature or something like that, depending on what you're planning to do with gitlab like using it for for your own open source project um or specifically, something else.

B

Other things to highlight to mention is so-called issue templates or merchant quest templates. The idea is that you provide a skeleton or a template with the most important questions to reproduce a problem, so you don't need to go back and forth and play ping-pong on specific things, um but rather provide the like something is not working. Okay, here are the steps please describe.

B

Please describe the steps and then you or you being the developer, can just go ahead and start trying to reproduce the problem or start working on on specific things, rather than needing to ask too much in advance, which is also important like different time zones, and you cannot just do a sinkhole for solving things.

B

um The other thing, when you start planning and managing your work, is um you create issues you assign them to milestones? um You can create, or you can add due dates, meaning to say you get a reminder when an issue is being due um and yeah. This is basically what uh what's inside the other thing I would recommend or checking out I use, uh use cases labels um which provide you just a visual overview to see. Oh, this is a bug.

B

This is a feature, but they can also be used for search filters for creating your own issue boards for like project management, um getting an inside view. What's what needs to be done? What is uh priority and and other things um git lab, also provides so called scoped labels which allow you to, for example, have have a state inside which means a workflow in evaluation, in development, in testing and so on, but yeah.

B

The other thing um which we will be looking at in in a bit is there are different editors inside gitlab, so there is like the static site editor, which is the basic editor for just editing. A file fixing a typo or something, and the other thing which is provided in the browser is the so-called web ide, which allows you to have a file tree on the left hand side and to edit specific files in the repository or to add files and then directly complete. It see the ci cd being run and so on.

B

um This isn't the full blown visual studio code editor, but it provides certain capabilities to, for example, preview, markdown um and edit.

B

The main things- um and it also has syntax- highlight, for example, like to see whether this is being a comment or a specific um code um before we jump into the icd yeah project activities, also a good thing to get an insight into. What's going on um and last but not least, um when someone at mentions you or you get an issue assigned or um something else is happening and needs your attention. There are so-called to-do lists, so you can also add your own to-do's when you're inside an issue and say okay.

B

This is something I need to check out later on, because I cannot change focus right now. You can organize your to-do lists and check what needs to be done, mark them as done filter them and so on. This is a good way to um to immediately see what needs to be done, and when someone at mentions you it's, it has more priority than, for example, an email and something like that, but without further ado, the next to plan manage and also like, create and store something, and you get repositories.

B

We also want to test it and we don't want to manually test things, because this is like kind of boring. It also like proves it works on my machine, but probably the software shouldn't be just be running on your local machine.

B

It should be delivered on mac os on windows, on specific linux. Distributions on ios on android on maybe arm is a different hardware architecture, and for that um we need systems to do that for us um and also in the in a defined infrastructure in a defined way of saying. Okay. um It just runs in the background and tells me that the code is not working on the arm architecture, for example, but it works on linux or an x x68 architecture, for example, but it should be, it should be running automatically and best.

B

It runs all the time and not just when I'm taking a new release and I'm giving it to our customers and to say yeah, it's nice that you release that, but actually it's broken and you burn out from hot fixing the releases and hot fixing the hotfix and stuff like that.

B

So really like shifting left or starting to use continuous integration is a good thing in gitlab this is called cdi, which continues integration.

B

Cd is continuous deployment or continuous delivery, but for the main part is like we need to understand certain keywords: certain concepts around this around it, so um there are different systems out there in gitlab when something is being run. A unit test, a compile step, an automated release generation, a deployment into production. This is being done in a job, so a job executes something a job is organized in in a pipeline which can be grouped or jobs can be grouped into so-called stages in the pipeline.

B

Sorry- and we can also have environments, so we have, for example, a deaf environment, staging production, with um specific permissions and so on.

B

um When we want to parametrize or instrument a specific job being run, we can use environment variables which are being passed from the cicd pipeline, configuration to the actual job being executed, and the jobs also provide artifacts, which are basically outputs, could be a binary file, could be a tabor for the release, for example, can be a generated.

B

Documentation can be a unit test report in a specific format, can be a security scan report and so on. So this is important that this is being passed from the job back to the pipeline to gitlab itself and the gitlab runner is also important, because this is a separate component to get lab it come can come. It communicates with the gitlab server um and the idea is that it can run anywhere.

B

So you can run it on on your local desktop if you want to, and but it doesn't really matter whether it's like mac os, it's windows, it's linux can basically run everywhere and it can also be run, for example, inside a docker container or you can you deploy it into your kubernetes cluster um scale it in the way you need to, and the good thing is you're not bound to just one runner instance. You can have multiple runner instances um either assigned to a specific group.

B

So, for example, you have a group of projects which require many resources, see plus plus rust needing compilers. So you can assign more hardware more resources for this runner group. If you want to package a web application which doesn't need so many resources you can use lower lower end sounds odd, but you can use less resources less.

B

Less resource usage runners- basically something like that, um and it's it also um allows you to like find granular scaling, whether you need that or not on gitlab.com, we will be using the shared runners, which means that we can can access shared resources.

B

Sometimes we are just in in the queues, so the jobs could be needing some time to getting started, um but we could also, for example, this is something we won't do today, but you could um install your own runner and then you're not bound to any like credit card validation, but can use your own shared runner.

B

um You can also self-host gitlab. So if you want to run it in your own cloud instance or in your own dm set or everywhere else, you can also just install it yourself, install the runner yourself and just have your own environment, so you're not bound to these house offering on gitlab.com.

B

From the mythologies around cicd um it's, this is like a typical workflow. You start coding, you commit your changes into git, um you push the code and then the ci pipeline runs the tests builds. The code runs the unit tests, integration tests, anything around that and from there it's moving on into the continuous delivery continuous deployment pipeline, which means you do a review, you deploy it into a staging environment and when everything is passed or has passed, you deployed into production.

B

The idea is that you can do continuous delivery in a specific sense of you. Don't have a manual step which waits until someone approves that production gets updated, but you do that automatically.

B

um The other things to know or to mention around ci cd is like pipelines, are not just like a pipeline where you build a car and then the left, the left door gets added in the right door and so on, um but you can also like build something which is asynchronous a directed acyclic graph, for example, with different dependencies creating non-blocking pipelines.

B

You can span pipelines over multiple projects whenever this is needed, trigger pipelines in in parrot child ways, meaning to say you have a non-blocking deployment pipeline and it gets deployed on multiple microservices, but the entire pipeline itself is just free for for more resources, yeah, and there are there's many many more to that which allows to.

B

Adopt it for your own purposes well, but our purpose right now is just we want to dive into a little bit how to work with ci cd, and this is a workflow how how it's being done in gitlab.

B

So when we start work from creating an issue um and we we create a git branch, we push the changes after a while, and then we create a merge request in that moment, mark it as draft, for example, and continue working outside of the main branch or the default branching git and get feedback while we develop along so when, when I'm, for example, adding a new feature, changing the back end or improving the back end um they're possible.

B

Hopefully there are unit tests which prove that something is wrong or that that I introduced the regression um which can be seen in the merch request itself from the cicd output, and when I see that or the the developer who who needs to review the merge request, you can say: okay, this is not uh the tests failed. We are not gonna merge this.

B

You need to really fix the problems which we detected here um and this um this whole review goes on into like um opening up the um the difference, checking the code style, discussing a certain implementation um and when everything gets resolved, it gets approved, merged back to the main branch and then deployed later on released when the release is due and so on, and in order to do that and to add really add ci cd, there is the format which is yaml, but don't don't worry, you don't need to be a yaml expert for that regard.

B

um There are, of course, cheat sheets and resources um also online lenders for this. If, if you want to dive into it, if you want to learn it um most recently, we added the pipeline editor in gitlab.

B

So it's it's like, like an online editor in the browser which understands yammer um and also gives you live, live linting for the specific syntax which will be which we will be trying out in a moment um because in in yammer there is a specific way to use indents to create arrays and lists, and sometimes I'm making the mistake myself and then I'm like the committing it and the icd pipeline is broken because the syntax is wrong.

B

So in in a specific sense you can, you can call it like ci cds code or pipelines as code, because you're describing what's what's going on with the yama syntax.

B

The idea is that the pipeline editor provides you with even more insights how the pipeline looks and what what else can be like simulated beforehand, because it can get quite complex over time. The most simplest idea or the most simplest way to try the pipeline editor is to say, okay, I want to run a job which, in this example, is all underscore tests.

B

A job has the requirement of a script attribute, which runs commands and a script attribute can run either one command or an array or list of commands with the dash as a bullet point, and this is being executed, you can also specify an image which is a docker image, a container image which is being spawned by the runner and inside you execute, what's defined in the jobs and in order to see what what happens here, we can just go ahead and say: okay, let's try this example navigate into the project.

B

Which you probably have forked and um you can fork it right now- um into ci cd um and the editor which in structures? Okay, you don't have any any uh pipeline right now, but we will be defining it in the dot gitlab.ci.yaml file and to just make it easier. We we can just generate it and get it into the view you can see here that um the pipeline editor provides you with some examples.

B

What you can do, so it's like the first, the first thing to success. um In our example.

B

Just like removing this and say okay, I want to start really really simple in by specifying image, which should be alpine latest, and you can see it at the top. Something is moving so it says the ci configuration is invalid and job should be. We need a job. Okay, then maybe let's define a job which was should be all tests and the job name. So at the top level there are specific keywords which are reserved, but the job names are free. So we can, you can call it gitlab one.

B

You can call it buildbuild.0 or something else, just find a telling name so that you immediately know what's going on.

B

We need to define script and the nice thing is I'm pressing enter, and it does the indent for me uh and we want to call a shell commander bash command, which is in this case just exit one um from there. It says the configuration is valid. This is nice and um I'm a fan of good, commit messages at gicdconfig, and we also want to commit to a different branch which should be um cicd.

B

Probably um it doesn't give. The instruction here should fix that, and we also want to um create a new merge request from that. So um we just want to live, leave this scope and we create the match, request, checking in a bit. What else we can do. The thing we can see here is that the pipeline is being run and when we, for example, click on the pipeline itself, it was really fast being run.

B

This time we get the overview of what's going on, we can see the pipeline, we can see the jobs and in this case the job failed. So we can either go here and say: oh, we want to dive into the the detailed view of the job output to see what is going on and we can see that the runner pulls in the docker images spawns the context of the container, prepares some things and then executes our command and sends everything except the exit except zero is an error exit exit code inside gitlab cicd it fails.

B

So this is an intended fail for for this exercise um in order to to say, okay, how does how does a fixed pipeline look like? We can, for example, navigate back into the merge request, which is highlighted as an exclamation mark in the and the number um of the merge request, and we can look at the changes, for example and um yeah. Basically, I'm missing a button, but we can just go into the public editor and you can see that there is branch selector.

A

B

Allows us to check out the previously changed, gitlab ci cd configuration and we can move on and say: okay, we say we're simulating a fix, now um fix pipeline and commit the change, and we can see in the in the pipeline editor already that the pipeline is being triggered.

B

So I can just click on this over here see what is happening. These are the changes and it's running I can navigate over here and again it says: exit zero job succeeded. Everything is fine, everything green, the pipeline, passed and follow-up stages of follow-up jobs uh will be executed, um the deployment happens and so on.

B

So this is like the basic example of um getting started with the icd and the pipeline editor um you, you are not bound to the pipeline. That is so. You can use the web ide if you feel more confident with it, you can create it on your own pace.

B

um The good thing about the pipeline editor is that it's not only like an editor, but it has the live. Linting inside you get an immediate feedback, whether the configuration is valid or not. So it does some checking in the background, and you also have the possibility, for example, to to see what it what it will be creating. So if you, if we never get back in here and say, we want to visualize what what will be created from this configuration.

B

I just have one one job. This is the default stage and we can also see what what will be will be done. um It's interesting, especially when something does not work as designed as as is as intended, um and we can also see what happens when what is the full configuration which is being executed, because at a certain point we will.

B

You can use advanced methods with inclusion which, with job templates inheritance and so on, just to make sure that the pipeline configuration is as efficient as possible and you don't copy paste or you don't make copy paste errors from too many things. um But yet it's it's a tool for getting started more easily and and also manage um huge pipelines or pipelines which which have more than like three jobs inside um for the coming labs or for the coming exercises.

B

We will be using um the poplar editor unless mentioned otherwise, and the typical workflow, which is not explicitly mentioned, is like we're adding configuration in the pipeline editor. We verify it with the live, linking maybe discuss certain things commit the change in a branch, create a merge request and inspect the running pipeline.

B

If there is something else to that workflow, I will mention that, what's noted on the slides the thing or the idea is okay: we've played around with exit, zero and exit. One is it's nice to see something, um but what are like the typical workflows? You, you probably want uh experience and one of the things is well. Maybe you have a project already which doesn't have cic or you.

B

You just really want to run the unit test automatically as a first step and maybe just seeing if the code actually works on a specific target platform, and so I've prepared a project which is written in go and it it allows you to run it does something, but actually we are not sure if it works, so um the um the gotenuki inside it is a go code. There were some. There are some libraries um attached to it to it and there is, for example, the main dot go.

B

The main entry point um this function and it does something print something, um and we are not not certain whether this works or not, um and you can, for example, imagine maybe you've migrated from svn to get um got, became a maintainer for this project or you're joining a company which doesn't have gitlab or any ci cd yet, and you just want to make sure that you're working efficiently that everything gets tested and you also can test the the systems the customer is using.

B

So you don't need to create your own virtual machines or your own test environment all the time when a support ticket is being opened, but instead can continuously test that as you go along and yeah. This is basically the thing about it. There are many more use cases to see icd, but in the end it's like.

B

I don't want to install a specific virtual machine or use a different cloud provider which takes me two days to to set up because as a developer, I need to become ops to really understand it, but instead it should be done in the cicd infrastructure and my code gets tested automatically built automatically and that's about it.

B

um In order to now test the go tanuki the code, we want to add a basic cicd configuration and one of the things which are important is we can organize jobs in into stages, so we basically define the stages as an array as a list with build tests and deploy. This is also the default.

B

If you don't, if you don't define anything, but it's better to like I'm following the principle of, I really want to see what is there um it's nice to use templates and such um and predefined defaults, but I really want to define what's what's inside the pipeline.

B

um The other thing which is important for this example we'll be using a pre-predefined or pre-built environment with golan container image.

B

We're specifying some variables which we will be using in a bit and the first job which will be executed, is called format it's assigned to the build stage and the script is to actually run go format or go fmt, which checks the basic syntax of of the code and go vet. Does some further integration checks to see whether the code is actually building or not?

B

These are specific to golang. You can replace them with um npm build or um cargo build for 4c, plus, plus no c. Blood versus rust um in c plus plus could be c make make um something which builds the code or does some basic linting around it to see where that's actually being built, um and in order to like not over engineer, this can just copy this example and navigate back into our project open up the pipeline editor, starting because we committed this entered in a different match.

B

Request start a new configuration um mark all and remove, and for this example, just not typing too much. We see it's. The configuration is valid um and we, for example, will be generating the format job. The lending says just this job and there is nothing really specific about it. We really want to see what is going on here and say at cicd.

B

Conf configuration for go and branch names um are like open to everything, I'm not calling it. The icd-go um I prefer dashes in instead of underscores because they could be could be hard to type at least on a german keyboard, and I also want to create a much request in that regard and then I'm committing the change.

B

um Moving on into the match request over here, I have the possibility to, for example, define what's what needs to be done or what's what happens inside this merge requests it's basic uh configuration, and I also get the possibility to preview that so, for example, I have um should be using a merge request template in that regard, so I don't repeat that every time but yeah it's like tasks and, for example,.

B

B

This is the red bracket tests, which then gets rendered as a checkbox, which we can tick on later on with developer materials.

B

I can assign it to me, for example, assign reviewers which, which I can assign use specific labels, and so on um now that I can create the match request.

B

It should have run the pipeline in the background already for the branch we created earlier on, and we can see that the pipeline failed for some reason, so the the go code doesn't work for some for some reason. Now, when we navigate into the scope, we can see the pipeline, the format job has been failed. We can do the same step as before and just click here and navigate inside, but we also have the tab here with the failed jobs which basically passes the output and tells us in a shorter way.

B

It says: well, go vet detected an error in main.go in line 12 character, 16 or line 12 to 16. I think undeclared name os so when we navigate into the project again and inspect the code now in main.go in line number 12 we're using os, which is undefined in the scope.

B

So in essence, we are missing to import library dependencies and in order to make it easier in order to fix the problem, we can actually open up the web id over here, for example, and fix the problem. This is being complemented out over here just to make it as easy as possible. So we need to include fmt and os to make the code work and um are we in the right branch right now?

B

No we're not um sorry going back into our merch request. This was actually what I was looking at.

B

We can open the merch request in the web ide so just to move on in the branch we started to create our gitlab ci cd yammer. We can also just edit the code and fix it over here.

B

We could also fix it in the main branch, but we want to move on from from the cicd configuration we just added and say: okay with the ci cd configuration, we fix the missing imports, so you can create, can use the same mechanism to commit the changes directly in gitlab with the web ide similar to the pipeline editor, and we commit it directly to the branch so the it uses the previously added ci cd configuration and now the pipeline is being run again in the pipeline editor.

B

You can see it either over here with the rocket icon or at the bottom. There is also like this. This mini minibar mini sub um yeah. If we open the pipeline again, we can see um it runs.

B

The image now uses the go, link, container image and hopefully spawns a chop soon or the chop is being spawned, but it needs to be needs to be done. Let's just check what we're doing here. This is described basically in this exercise and once the code is being fixed, we can move on and look into things into more things. What we want to add for ci cd and add more testing, add more building and so on now in in this regard, we can see that the pipeline was fixed.

B

If we navigate back into the merge request, we also get the feedback. Okay, everything is fine.

B

um The indication for merging this back to the main branch is also there, so the tests um or the ci cd pipeline worked, and we are allowed to merge it back, but we won't do that um for now, because we are not yet finished uh with the cicd configuration um now, I'm hoping I'm not talking too fast, but in in that example, it's like okay now I know how um the code works.

B

At least it were it compiles um from that specific sense, but what what else happens in there? So when um this is like the visual visualization um example, which we did already when we dive further into what else can we do with the code?

B

We can, for example, actually build the binary for the for the golan code, which which didn't happen. Yet this is done by the go, build script, which has certain flags with rays, ld, flags and so on, and produces a binary for producing the binary.

B

We previously defined the binary variable which we will be using, and we also make want to make sure that um this binary is being exported by the job output sent to the gitlab server for download for further usage, so we're defining an artifacts, uh we're defining it in the artifacts entry which allows you to define the paths which can be either like in this. In this example, a variable you could use a dot, you can use white card patterns, um anything which is important.

B

It's often used just for, like, as I said earlier, for reports for release starboards for anything you want to really like. Have the runner be, uh have the runner uploaded to the gitlab server and make it available for later?

B

um The other thing we want to add next to the compile uh job is we also want to test um the code so there in in golang. There is like um the format of.

B

Of having the main file and then a specific test code, but this hasn't been run yet, so we really want to trigger the go tests in our example uh which we can assign to the test stage. um It it executes um specific goal tests.

B

One other thing which go allows you to, and this is like specific to languages, um allows you to to generate a coverage report, which means um it checks all the functions which have a test being available for and presents the the percentage and like a hundred percent test coverage is like a developer move um which, which you probably will never reach, but seeing a certain decrease of test coverage over time could give an indicator where the code quality decreases or not and is, is a way to inspect um or to verify what uh what's going on in the merch request.

B

Probably the code coverage should never decrease from from a certain amount unless it's specified in the review or in the merge request itself. So um long story short in the first thing we want to do is we add the compiled job into.

B

The cicd configuration so um let's navigate back into our pipeline editor and select the ci cd dash goal branch and at the bottom we add the compile job. The compiled job is being added to the build stage similar to here and since we're using the same stage as the format.

B

Job we need is not needed not required in that regard, but just to see see the dependency to define the dependency, uh we can also define an asynchronous dependency for the compiled job being dependent on the format job just to be sure, and the second step in this example is to say: okay, we want to add the test job and the reason I'm not typing.

B

It is because it prepares some things um and usually, or sometimes this is being defined in a makefile already in your project or in a script um or in a tutorial you you're, looking up on the internet or in the documentation so yeah, it's maybe copy pasting from stack overflow or something like that.

B

um I totally would have wished for that like 20 years ago, um but yeah. Joking aside, we do have the test job, which is assigned to the test stage, so it's executed actually, after after the format and compile chops and we'll say it compile and test jobs.

B

Now um we can see the the configuration is valid and the pipeline is being running clicking over here again and um it shows us the changes which is nice and the pipeline's being run.

B

And this takes a while, because every time a new pipeline is being triggered, it starts again from the beginning, so it starts out on the commit hash on on the commit cha which which is being triggered, which is being used. This is the current head in the current branch and you get reliable feedback from every change you make in your git history.

B

There are ways to speed this up and to use more caching and use that more efficiently, but it's the same as with code. You first make it work and based on your experience, based on your knowledge, you try to immediately optimize- or you just say: okay, this works now and I'm refactoring this later on to make it more beautiful to make it more efficient, um don't try to over-engineer everything. This is a habit which I sometimes have, um and instead it's just like. Let's um just try to make it work in the first place.

B

um The thing which happens here is we can see um when we move back into the stages view, uh we can see that compile and format have been run successfully, so the compile job has actually created a binary.

B

um We can see here the executed script section and it's uploading the artifacts, so it means that the tanuki binary has been uploaded to the gitlab server and over here we can see the drawback artifacts and when I click on browse, I can see that there is a tanuki binary which has been compiled for linux.

B

If I would be using a linux platform right now, I'm using mac os um on linux, I could download this and execute it on my own localhost.

B

um If I would be wanting this to to work on on windows or on mac osi, I would need to do use a different cross compiled build. This is way too too deep. Now um the main idea is actually. I don't want to download this. I don't want to do this on my own. I want to automate this, and maybe there is a way we can just run the uh to do.

B

An integration test run the binary and see whether it crashes, whether the output is expected, what it does and so on um so beyond um running the pipeline, which hopefully worked in in full and the pipeline is green.

B

What else can we do, and we can basically do what I described just now? We really want to test the binary. We want to run it.

B

We want to do that in the in the stage in the test stage or in a later stage, defining defining our own and the job needs the test job being being done successfully and the compiled job, which generates the binary actually.

B

And the the thing which it which it basically does is it executes the binary um in in foreground, so we are expecting that something is being printed to the standard output of the terminal, um and this is like um a dot dash and then the binary being executed.

B

Sometimes um when there is, there are no executable rights, you need to use a chmod plus x on the binary beforehand, but in this example, it just works like that and in order to add that we can just navigate back into our project now going into the pipeline editor and choosing the cicd-go branch which we have been working in and the last example or the next example is to test the binary which we created over here.

B

um The test job has been run successfully as well and okay, it test binary job um committing. The change should have said well, should look more often into the cicd configuration as well. It.

B

And we can see, the pipeline is being run again, can view the pipeline, see that there are more things going on, um and we also have like a different view on this. Well, we can have a different view with the asynchronous dependencies, um so we know that we can show the dependencies. Actually, um the format job is being run. Then the compile shop runs.

B

um The test shop um runs together with the test binary shop, but the test binary job only runs when the test job is actually actually successful. So whenever something breaks inside, for example, the test shop fails because there is a regression in your code, um the test, binary job won't be run and the pipeline breaks early.

B

So you get feedback early enough um to actually fix the error and not everything is being run in the pipeline, uh which is a good thing, because when the pipeline or an integration test, might be running for 30 minutes for some reason, this blocks your revenues, blocks your ci cd pipelines and there's a developer moving fast and oftentimes. It's like xkcd303.

B

What are you doing? You're compiling nowadays, it's waiting for ci cd builds. uh You really want to speed it speed it up and fail the pipeline fast to really fix the problems and then continue working rather than waiting for the pipeline and spending time somewhere else for the for the 30 minutes. um In the past I was a class plus back-end developer and I had to wait quite often for uh builds uh on my local machine, uh which basically resulted in me spending time in community forums and on social media, um but yeah.

B

In the end, it would be much more efficient to stay, focused and continue working on the code, fixing the problems and delivering a feature rather fast. Now um the pipeline is being run. The test binary chop is executing and again it's it's spawning the golang docker image and the good thing um is over the recent years. um Many many um programming languages and libraries and environments have created their own base, docker images.

B

So it's not just like us using a debian image and then installing the dependencies, but we can use predefined images and, in that regard, we're spawning the con the image and we're running the binary, and we have summoned our go tanuki or rainbow tanuki, which.

B

Yeah, which is basically success for today, and we can stop no no just kidding um but yeah this. um This allows you to play around a little bit and you can see when you're printing out a url, for example, in the job you can link further. So whether when something is, uh is an error or you want to give some more instructions but to learn more about it.

B

You can just echo or print a url in your code or um in in the ci job output, and then you can click on it and it opens a new tab um and you can learn more about it. I've added an e-stack to just link to our gitlab community.

B

So if you want to join on in and start your open source contribution journey or learn more about how what we do in our community, I would welcome you to join on in um or jump on on the meet up, join the gitlab community forum learn about the code. Gitlab is mostly has many components right now. It's like ruby on rails uh vouchers for the front-end parts goaling for back-end demons.

B

Certainly more things around that. But back back to the exercises or the ideas, what else can be done within ci cd exceeding a little bit my time window but yep? um The thing I've showed I've shown you before is um there are different fuses, like the the grouping of ci cd jobs for stages, which allows you to to see what what basic things are going on, if you're using needs and and as in kronos execution.

B

There was a way to visualize that in a different way, so it's not bound to the grouping by stages, but rather to see. When is the job actually being executed?

B

The the screenshot on the right is what the full solution of this workshop looks like in in the end, and it's not bound to that. You can explore it by yourself. You can play around and see what is going on and how to make it more efficient.

B

The other thing um but which is important to see is well. We really want to have faster feedback, as I mentioned before, with ci cd. So it's not only like compiling the code um or form checking the format or doing some linting and linting can be used, for example, for markdown for writing. Docs for for everything which has a linter which can be executed as a cli command in a container, but we also want to run unit tests, for example, and I'm yeah, I'm a developer myself and writing tests is sometimes like.

B

Yeah needs to be done, but it's not fun, but in the end it can really help prevent regressions where you spend hours or days or weeks a month even on, because it's not tested well enough and not prepared and probably or I experienced it myself.

B

I I knew that I couldn't change this like c plus back-end code um in that specific line, but a new team team member didn't and they committed something and someone else approved it and it get murdered, gets merged, and after one month I see that and say oh yeah.

B

I totally wanted to write a unit test for that two years ago, but I never did um so basically try to try to add as much tests or as much automated tests as possible, um and it also helps others to really see what what what's going on and next to the set unit tests. You really. You also want to look into test coverage, get that provides junit as a generic or specified format uh for this.

B

So when the compile the the build chain allows you to generate coverage reports in june xml format, you can integrate that into gitlab and one of the things which which works this way or how this can be done with golang, is to install um a library and then run go test to actually generate this report xml, and this is something which we can just try now and add to the pipeline editor at the bottom. Just quick copy paste, because typing is quite quite a lot to type.

B

The main idea is that it runs go test piped into this binary, which then generates the the report. This is this is a nice way. The documentation of gitlab holds some more insights and details for the specific other languages, and I think I found this on the internet somewhere. So it's not just us exploring this. For the first time, um there are issues out there which are helpful.

B

um These the test unit report can just run in the test stage, but it needs to be run after the test job because it needs this discoverage.cough file which which is needed. So we want to add the test j unit report and have this hopefully executed.

B

Right now, so you can see.

B

This is actually being drawn here, but when I have hoover hover over it, um it shows me that this runs independently.

B

um It just waits for, for this probably should improve the ui and just move it over here, but yeah you, you get the idea of um asynchronous dependencies being run um why this is being run um and the more jobs we add throughout the workshop, the longer the pipelines take um just some ideas around what else is interesting. What else could be could be done in a pipeline, and one of the things is like you probably have heard, shifting left security, and why do you want to use it?

B

The first thing I want to encourage you is, whenever you're, using a secret or a credential in your ci cd pipeline, for example, to specify the the database access for mysql for postgresql um when you're using a ruled past the root password from production. This shouldn't be in plain text somewhere or you don't want to add your aws credentials or key pair somewhere. Public and gitlab allows you to specify ci cd variables, also as secrets so that they are masked and they are passed into the container job. But you cannot print them.

B

You can access the access them inside the chops being run, but there is no way, for example, on the interface to see that so the output, when someone tries to print that it's being masked.

B

And it helps to to really prevent to leak any specific secrets. You can also limit certain environments, certain scopes where they should be visible, but yeah. This is one thing of of security, and the other thing around this is um in order to prevent to accidentally add something in that regard. Gitlab also provides security scanning.

B

Which can be added to ci cd.

B

B

Which means that um oh I've seen an error in the chat.

B

Okay, trying to make a commitment.

B

Maybe when, when the gitlab ci dot, the amber file already exists, try to edit this file directly or maybe the you're using it. um The wrong branch try checking the branch selector at the top, whether you are in the in the red scope, um specifically for um adding security scanning to your pipeline. You can either you do that from the ui or um specified in the in the cic configuration, for example, by adding or including a template which, which comes from from gitlab itself.

B

So this is provided in a self-hosted installation on the sas platform and can just be used.

B

Our repository has.

B

uh The dot end file already prepared, as far as I know. Yes, so we are pretending to have leaked an aws key, so this is not a real key.

B

It's a made up example, but in the end accidentally, we committed a dot end file because we're using it for development tests and so on, and it has our aws key, and if this is now in a public project and if I don't recognize it myself or a friend points me to that, someone else with bad intentions could use that key and consumer resources and generate lots of costs in on my monthly aws bill, for example- and this is something you totally don't want to so um the thing we can do is we add the uh with the pipeline editor over here.

B

I'm talking too much losing my voice.

B

uh We never get back into the ci cd go branch and at the bottom, and we can also move it um to the top, but just for for including this over here. We also want to add secret detection.

B

uh We change the commit message.

B

And wait for that.

B

The pipeline is being executed, but we can also like navigate over here in the pipeline view.

B

This also provides us a history of what we already did in the past and we started today. This was me this week um preparing the workshop and yeah. Now the pipeline is being run again and we can see that secret detection is is actually being added inside the test stage.

B

But again this will take a while. It will detect our aws key and tell us about it.

B

Yeah, if, if you're, using gitlab for open source pro, if you're applying for an open source license, for example, in the open source program, um gitlab ultimate provides an integration into merge requests, but you can also download the json report of the security scans in the free tier and just use that for further inspection, the other thing which which is important around security- it's not just secret scanning. It's also static application, security testing, it's um dynamic application, security, testing, fast testing.

B

There are so many ways you can make a mistake. I have seen things from from south scanning, which I never anticipated to make the mistake myself.

B

So it really helps to automatically test the extreme cases worst cases you never thought of, um and automate this and being able to fix it so sometimes like changing uh the excess of certain files. Sometimes memory is not protected. Sometimes it's like user form. Input is not sanitized or things. You probably cannot really test from an end-to-end test perspective, which can be automated. Also in cicd, specifically with security in mind, I didn't check whether the pipeline is being run. The pipeline still runs so it just takes a while um yeah.

B

What else did I miss um yeah dependency scanning is also a huge point when you, for example, edit your package, json um or.

B

Composer in php or whatever is used as a package manager or for for dependency management. There might be some dependencies in trying to install malicious code downloading something which is not intended to um and so on so uh vulnerabilities in dependencies and also runtime scanning containers are like the base image for a small operating system inside which can also contain vulnerabilities and and much much more. um That being said, we want to like spend money or spend time on the ci cd scans in our apartment as well.

B

Even so, the pipeline run might take a little longer because um when we have to fix incidents at 3am in the morning on the weekend, because there is a security breach, this leads into burnout. um What can lead into burnout- and it's not healthy so like having this in in our pipelines, is a great way to to improve, to make the code better and to stay safe. In that regard, um what other things to check? Maybe check out?

B

Well, there's a thing around like you can automate it even more: have gitlab detect certain devops workflows, opinionated templates to automatically build, for example, a web project based on nodejs or um I've recently tried learning wheelchairs a little bit like have this automatically create a container deploy somewhere around it in aws or in kubernetes and yeah. I don't need to think about too much about ci cd configuration, but my recommendation is to learn the icd from with the pipeline editor to understand.

B

What's going on, because when you need to troubleshoot it when it doesn't work, it's good to know the tools, good and all the workflows and so on.

B

The other thing around: what did I miss? Oh, this is the security slide again. Did I miss something auto remediation yeah um probably, should remove the slide um from the presentation. um Let's just do this now, like I said on twitter earlier um to some someone being nice from from talentql.

B

What's next, there are many ideas around like what else can you do with ci and following with cd like the deployment parts, depending on where you want to deploy your application, um apply, github's, workflows, ad monitoring, add even more security around things but yeah. This is basically.

B

The things um one other thought around like over time, but I would recommend to look into these exercises for async practice, which touch base with test coverage, how to parse it even better and and also play around with the junit integration. So there are some examples which add new code and then later on at tests for the code, so you can see that the test coverage decreases and later on. It increases again just to allow you to play around a little more with it. There's also a way to add batches directly to the project.

B

So you don't need to edit the written md just do that in the project settings and you can see the department, for example, works.

B

We can actually try that now as a short example, while the pipeline is being run when you navigate into the settings, um was it cicd.

B

No, it's general general batches and the batch url is the pipeline svg and probably have done. The exercise quite often uh can just add this here. This is the preview, and now we have the badge in the project and everyone who navigates into the project can see that the pipeline um hasn't been run for the main branch. Yet, um okay, yeah, probably probably you need to merge the cicd, much request uh csd goal, much requests into the main branch to trigger the main pipeline to see the status and the batch okay.

B

What else do we have? We have certain tips around merge, requests and code reviews yeah, it's totally optional. I just tried to add as much detail as much as much tips as possible um around this. um I don't have any more of efficiency ideas. This probably shouldn't be there. um I've seen there there's. Maybe there are some questions in the chat before we jump into questions. I would like you to yeah check out my profiles and also um there was a blog post being published this week, which has more gitlab hacks, gitlab, hacks, gitlab ideas.

B

What you can do- and you have seen certain things around templates- I would encourage you to also learn quick actions which are super helpful and much much more.

B

um You can see the public editor over here, but, for example, things would choose a super interesting apparent child power plants by example and so on, but this is way too much what I wanted to show today just a reminder that everything or we're trying to document everything share everything, and um I'm hoping you had a great time today and learned something new, um and with that I would just say: thanks for the attention.

D

B

If there are any questions, I would say um just ask test.

A

Hey me, you so much michael for this really insightful workshop um hi everyone, please. If you have a question, you can unmute your mic or um just sharing the textbooks. Thank you.

C

Oh yes, uh hello,.

B

C

I I have a quick question and my question revolves around what is the pricing like uh for gitlab, cicd pipelines and also is there uh like some sort of provisions for free tier accounts? I can't start on future. Is there like a max amount of limits? You can run these things for.

B

So if you're, if you're subscribing to or if you're, creating a gitlab.com account, the free tier, um includes certain minute minutes, you can consume from the shared runners just to provide you. um No, it's pressing the pressing.

B

Page, it's the weekend cannot type. um So there is. uh You have 400 ci cd minutes per month, um which are in the free tier and typically this should. This should be enough for like using your own projects, one other ways if you, for example, working on an open source project or you um you have access to your university, which can apply for or your educational institution which can apply for um the education program with gitlab.

B

They can get access to the ultimate tier for free in that regard, the pipeline minutes. What happens? If I reach my limit, I'm just reading the faq. You can purchase additional minutes. To be honest, I don't know the pricing I need to like click on here and potentially it's inside um the user interface. Then.

B

um But if you say, okay, I'm I'm having the problem, I'm reaching, for example the 400 minutes limit um each month from the shared runners, um I would recommend checking out, for example, spinning up your own virtual machine and installing the runner in there, and it just costs you. I don't know what aws uh builds. I think you can use the free tier, for example, for one year there and just leverage that and have your own gitlab ci cd runner from that regard.

B

But from my experience I'm I didn't really like exceed uh the limits that often um with my private account, or I cannot remember exceeding it. To be honest, so um I think you should be safe and should be fine in that regard.

B

I hope this helps.

C

It does thank you.

D

Hey michael, this is abubakar. Thank you hear me.

B

Hi hi abu bakr nice nice to meet you.

D

So, um uh just to add to what my ikea uh said, I think the pricing of uh the cicd means, if you want extra, is temple up more 1000 minutes. But when you you only exceed this, this uh the 400 limits, when probably you are working in a an organization and you run multiple pipelines at the same time. More like you, have multiple jobs, multiple stages, and probably you have lots of projects that are currently running. You see icd publishers. At the same time.

D

That's when you can quickly mop up your minutes and like he said, uh if you want to run more without necessarily spending for gitlab, do we want you to spend for gitlab, so you can spin up an instance, install the gitlab runner and uh do what you want. Some other instances where you might need to spin up your own runner separately is where maybe you need special type of devices of special type of processors?

D

Maybe you need some gpu or you need some mac to build some uh your ios devices, then you might need to spin up devices install runner on them that, due to specific uh requirements, hope that.

B

The other thing I want to mention earlier is when you're expanding your your project, for example in in a group- and you say you want to div you're, developing an awesome tool in the open with an open source license um your group or your, and your project can apply to the open source program and getting access to more ci minutes.

B

When you build complex pipelines, when you want to level with security features um and everything else, you can apply for the open source program where you can see that, like no in their kde and others, and just to to ensure that your group and your passion and your enthusiasm is not limited by, let's say 400 ci minutes, but instead try to build in the open, contribute and also being able to benefit from from the open source program.

B

The other thing is the education program, but this highly depends um on the institution um but yeah. I just wanted to highlight that it's there, so it's um it shouldn't, be it shouldn't be bound to that and the other thing is inside the faq and I probably should be linking the pricing page in the chat you can. You can also learn how to really like the step-by-step instructions, how to install your own runner on on a virtual machine.

B

I recently did an exercise. I installed the gitlab runner on my mac os macbook because it also works. It's just like you install the binary package. You run the registration steps. You insert a token which you get from from the gitlab ui and then you can like uh register the runner it communicates over https. So the runner contacts, the server and um yeah.

A

B

I think it's super straightforward to be done in like 10 15 minutes and see see the first success and it's also a great way to experiment and try it out and look look a little bit around look behind the curtain if you're interested in how the runner is actually being built or you want to read the source code, which is also possible, of course, um oops wrong window.

B

For example, if you this is, this is where the majority of the gitlab project lives but, for example, the uh the gitlab runner project lives over here. You can see that or you can learn how the the gitlab ci configuration, for example, is being built for this go project which, to be honest, I haven't read in a while, but it it's highly abstracted.

B

And and see how it's being built, how how the code works and so on, and maybe you you want to start contributing or you notice, that a feature is missing and you want to start diving into the code to learn something new.

B

Would totally love to to see you and forking the project, creating a merch request and start exploring.

B

Thanks for having me today,.

B

Any any final words.

A

um Thank you so much michael for um increasing us, your presence and sharing your experience and from your world of knowledge. This was really awesome and hopefully so our engineers can reach out to you via the social links. You shared right if they have any other questions.

B

Yeah, it's, I think it's best to reach out on on twitter or social media, um because if you send an email, I'm not reading my emails that often it's easier to to commu to share in the open. And if you wanna, like jump into specific questions, um I would also totally recommend checking out um the gitlab community forums which, which is forum.gitlab.com uh where everyone who has their experiences or wants to help um chimes on in, and it's an easier way to immediately get help from experts from beginners yeah everyone. Everyone can contribute.

B

So it's it's faster if I'm not responding to just jump into the forums and and get help or you can also ping our background on social. You can learn more about develop evangelism. If we want to what our team does, we do have a public handbook or the gitlab has a public handbook and our team has as well.

B

So you can see just looking up. The url now can learn and see what we're doing and all the projects like the gotenuki workshop for those. So like yeah. I think it's a lot to see and we are doing probably too much, but it's a fun job and I I really do love for the what we do.

A

Okay, thank you so much um so we have about our slug channel. So I'm sure, if unable to reach you, they can jumping of workers luck inbox. Thank you so much avocado for joining us and supporting today. Thank you so much everyone for tuning in um so we will be uploading this session on youtube and I'll share the link with everyone by monday. Thank you so much enjoy the rest of your weekend bye.

A

Thank you. Bye.