Jenkins Online Meetup, 7 Aug 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Online Meetup: "How Jenkins Builds and Delivers Jenkins in the Cloud" by Olivier Vernin

Description

Starting from July 2020, all Jenkins weekly and LTS releases are built and shipped by the new release infrastructure. Want to know how Jenkins builds Jenkins? Catch this session to see the real-life implementation of Jenkins’ development (ci.jenkins.io) and delivery (release.ci.jenkins.io) infrastructure in the cloud as it evolved from a mix of platforms to multi-platform virtual machines, containers and Kubernetes on Microsoft Azure. Expect a frank discussion of issues that were encountered along the way, how the architecture has evolved and what’s on the roadmap. We’ll share important tips and tricks for implementing your own Jenkins infrastructure on any cloud, based on my own Jenkins’ implementation experience.

Presenter: Olivier Vernin, Jenkins Infrastructure Officer

References:

* Slides: https://docs.google.com/presentation/d/1c-Vax7TGCHU41RynLV2GYQyavaYjfEptkbwepg3IN9k/edit?usp=sharing
* https://www.jenkins.io/projects/infrastructure/
* https://github.com/jenkins-infra/release
* https://issues.jenkins-ci.org/browse/INFRA-910
* Meetup page: https://www.meetup.com/Jenkins-online-meetup/events/272271879/

A

Hi all uh welcome to the jenkins online webinar uh today is august 6, and we have a presentation by olivier about how jenkins builds and delivers jenkins in the cloud.

A

If you it's a your first time at the junkies online meetup, uh basically it's event organized by jenkins contributors, so where we focus uh on presenting different things uh about what happens in the project and in the community. So it's new features, uh developer tools, also ongoing development and preview of uh things you may see soon as users.

A

Their main goal is to actually show something uh to talk, and we encourage you to participate in the discussion. So we you can use zoom queen a after the call. We will also have um a chat, so we invite everyone to stay and yeah the goal again anything about jenkins.

A

um We also run a series of meet ups uh about jenkins on kubernetes, and actually today's meetup is also about the junkies and kubernetes, though it's a bit specific because it's our own project's key study, so the release infrastructure only we will be presenting today around some kubernetes and there is a lot of various things to learn about. So we are prepared to such case study in general, uh we'll be interested to have any talk about.

A

The integrations between jenkins and kubernetes, like plugins features, help charts tools like operator or integrations with other uh cncf uh system projects, and if you want to present, uh let us know- because we are always looking for speakers uh regarding our current heat up schedule. So we have a meetup today. uh The next confirmed meetup is august 17th that will be about github, apps identification and checks api integration. We also want to have json project demos.

A

In the end of august, we may organize some additional meetups in the middle because yeah we usually announce meetups with one week advance and there might be more meetups in august and, if you're interested to present just let us know if you want to do so. There is a page which just describes how to do the process. We have an online open cfp process, so you can just prepare good on google docs submit to the mailing list and basically that's it.

A

So speaking of the today's uh meet up again, uh we encourage you to stay in the discussion and ask questions. uh So if you want to ask a question, please use zomconnet. uh This is a ban in your control panel. uh You, if you ask the question we either answer either synchronously or ask the speaker after the main part of the presentation and again you're welcome to stay until then, and then we will have open discussion.

A

So basically we will stop the recording and uh you we can discuss any topics related to the presentation or the jenkins after that. If you want uh to ask questions after the presentation, for example, if you watch the video on youtube, then please use chat to use information list or use the feedback form which we will be sharing in the chat soon.

A

Okay, that's it from me and yeah again, thanks to everyone uh for participating in the meetup and I'll. Let all of you to do the main part of the presentation.

B

Thanks alex I'll share my screen now I guess I can yes.

B

So he's doing some interesting thing here: okay,.

B

Interestingly enough, it doesn't allow me to sharp screw window. It only allows me to show a screen, a desktop, what's happening.

B

B

Full screen, but I cannot share one specific window. I it's not supported.

B

Okay, okay, then, let's do it in a different way.

B

B

B

I'm just reorganizing my window, so I can just share desktop one.

B

Can you see my screen?

B

Yes, we can so if there is anything wrong with the slides feel free to tell so um so, hi hi hi everybody uh thanks for attending this session, about how the way the jenkins project is releasing um jenkins. I'm the jenkins infrastructure officer for the jenkins project- and I did give I did this project by the thing, um so the presentation is split into six parts. The first part I'm going to talk about what the chicken's infrastructure project is, um then I'll clarify a few things and then we'll go into the detail.

B

I mean um more specifically about what the infrastructure looks like for the release environment, how we configure the reason environment what the process is to trigger release and then we'll conclude this presentation.

B

So first, it's important for you, if you're not necessarily familiar with what the jenkins infrastructure project is.

B

um Basically, we are a subject in this project, so the jenkins project, like any organization, rely on infrastructure and provide services like the main website, um the ticketing system and a lot of different things, and so in this case um we are also working on on the on the release of this game. So the people contributing. So it's an open infrastructure. You know it's an opening fraud project, so I will really invite a lot of different contributors to participate.

B

um As you can see in those slides, people are spread on multiple time zones in multiple locations, which means that it force has to put in place some um practices like having a synchronous communication and also, if you rely on testing and stuff like that, um there is no really real rules about what um it's in the djinking different project.

B

But what I would say is there is an organization on github called jenkins and fra, where we can find a lot of different um repositories that describe multiple things like um the main changes that are your website, um the puppet code that we use to manage our infrastructure, a specific configuration for the kubernetes clusters that we manage, and so on. So if you are looking for something in particular, that's usually the best place to go and otherwise um feel free to just ask on on rc.

B

You should find all the information on um on the champions league website um before we continue. I would like just to clarify a few things, so those are not like um real knowledge, but just to be sure that we talk about the same thing and the first one is during the presentation, I'll be using the term jenkins master. I know that we are changing this and we use a different one. The neat thing is, we don't have validated the results about using a different name, so I've been thinking about.

B

Should we just use a different one should should I use the one in the results whatever, and I just prefer for today to just stick to the champions master name, so we all understand what it means.

B

Another thing is during the presentation: I'm going to talk about different jenkins instances and the jenkins project managed a few of them for multiple reasons. So we have if we go. um If we go from from the left, um we have trus.ci so that jenkins instance is running in a private network only available to few maintainers, and we use it to build the main chickens website, update, centers or any other job that we just want to be sure that only a group of people can access it.

B

This machine is running under the on a virtual machine. um Then we have 3.ci and this instance is used by the security team for security work. um I will probably not mention too much about that one. We then have infrared ci, which is the one that I've that I will talk during this presentation. um It's mainly used to manage the kubernetes cluster and to run some um tasks for the infrastructure on a kubernetes cluster.

B

We have in the middle ci champions that I own- I I guess cidg.io, which is the way we usually mention it in the main main list. This one is the major in the biggest instance that we have. We use it to test the core. We use it to test plugins library, whatever. um This is probably the one that you already saw. If you contributed somehow to the jenkins project and finally, the biggest the other one that I will talk mainly today is release that see either changes that I own.

B

So this is the jenkins instance um used for the releases, um while all the people, while it's quite I mean as soon as you have a so this instance is running in a vpn. So if you have access to the vpn, you can see the job results, but only a few people can trigger a job there, so both release.ci and infra.ci are running and configure on the community's cluster.

B

Another thing that I would like to clarify here is about the release date. So before I go into the different components, I just want to say that you may have seen previously that we now manage all the release from the new environments, so the previous state was um who needed kosovo case of the founder of the project to trigger a new release which obviously put strong dependencies on him and his machines and his house.

B

um It was quite hard to contribute. Even if the script used for the release were public. We had no access to the machine, so we couldn't help him to debug and to release, or I mean to really understand, what's up what was happening in his machines um and also it's when I mean at least a good thing is it was like I'm secretly by obscurity, because we had no access to the machine.

B

um We assumed that it was um working correctly, and so basically, we created a jira ticket on the articulating system, so infra-910 and the idea was to do all the work needed to move from a machine running in a basement to exchange games to cloud services where the kind where jenkins community had the control on the release environment and could trigger a new release if it was needed.

B

um So that's basically how all the work here and it involved, obviously not only um configuring champions but providing the infrastructure providing the service and the process to do the full release um which bring benefits like everybody can audit, because we try to be as open as possible.

B

Everybody can contribute to all the parts, um but we wanted to be sure that only few people could trigger release, um but obviously because we moved from a machine running in the basement to the clouds, it was also harder to secure.

B

So this is a really big overview of the different components and I will now go into the detail for each of them.

B

B

So now, let's talk about the frustration.

B

So now let's go a little bit about the infrastructure, so basically as um everybody in the everywhere under this presentation, you are going to find a few links. So, each time I specified the junkies organization and a git repository. So in this case, all the things related to the infrastructure is located in the jenkins express azure repository.

B

Basically, we use terraform to describe resources running on azure um and everything was designed to use.

B

B

Okay, now it's back again, so, basically sorry so for the photo for our infrastructure. We need few resources. As I said, everything is defined using terraform code. So you may see a tube in this cluster with windows, auto scaling, nodes, linux scaling nodes, we define network configuration, we have dns, I mean most of the things there is managed in azure. um We also rely on azure people to store the code, sending certificate that you need in the in the process we use.

B

um We started to so to store the secrets that we use to decrypt our secrets and so on so um there's not mean it's not really rocket science there. It's mainly, we need a resource. We describe it um once we commit the code to the teach repository it's tested on seattle, jenkins that I o. So we have limiting tests by that same text and um and then what we used to do is once the pr was validated we merged the treso.ci and it was deployed from here from there.

B

um The reality is today is that for um cost reason we had to stop building um the pr um the infrastructure frontier, but um basically, what we're doing is each time we created a new pr.

B

We were deploying everything from scratch, and so we could test that we were able to upgrade, from a point, add to a point b or we were able to provision everything from scratch and obviously it was costly and then um we also had to disable the job from trotter.ci, because um we lost faith in our codes, which is, I mean, I'm sad to say that, but that's the current states- um and basically I learned few things working on on that part- is first terraform. Interesting traffic is quite difficult.

B

It asks you to constantly um test your terraform codes. Telephone relies on third api that evolve. We deploy services that evolve on cloud vendors, so it means that we have to constantly test it to be sure that it still work um testing your terraform code. The best way is to provision resources, but at the same time it's quite expensive and finally, um automating terraform apply can be risky because if you have states in your resources, you don't want to take the risk to delete accidentally a resource.

B

So you want to each time review what are um changing, and so that's one of the things that I learned working on this some resources that could be automated and others needs. And finally there are. There are tools that were created to solve these issues. So if I have to work on this- and otherwise, if I still have the time um there are few interesting tools that you should look after, which is conf test altera test of tfsec, um I only saw some demos there.

B

I haven't had the time to really deep dive into those tools, but this is definitely something that we miss in order to enable um the way we manage our infrastructure. um Another thing is more specifically for the chemist's cluster in the ac and the access that we are using, something that I.

C

B

More like the hard way is, you have a lot of features that need to be configured at creation time, and so we really have to be sure in advance what you are going to need, and otherwise you just end up by recreating the cluster again and again um so at least a good thing is you test your terraform codes, but on the other side, um yeah, just not using the best for your time to test your infrastructure?

B

Otherwise, again, as I said, it's not really rocket science, um but if you are interested to really go more deeper in the telephone code here that we are using, everything is located on the github story, so just usually the loop to contribute or to understand how to participate in the chat against a project is first by discussing so we use tyra made english rc.

B

The first step is first to identify if there is a need- and if there is one you're, probably the best person to implement the change needed here um once you did that implementation, you test that and see either on cid.jenkins.io.

B

If the tests are green and someone can review the pr, um then we merge the pr we deploy, we validate it, and then we improve and loop continue again and again and again. So the good thing is working on infrastructure. um You usually have to work on that the first time, and then you can just reuse the work that you did then, once you have your infrastructure in place, you have to configure and deploy services running there. So either you rely on cloud services or you maintain yourself.

B

In our case, we have both um because we are using commodities. We easily rely on the current and home to deploy our application and obviously we have jenkins and a few other services.

B

um So if you really um need to participate to the release environment, one of the first thing that we had to deploy was a vpn, because the jenkins didn't have the vpn, and if we wanted to move from a machine running in a basement, we had to be sure that only a group of people were able to uh to to access our services or trusted services trusted services. Sorry um we also use held up to define group and manage access. We have jenkins instance called release.ci.

B

We publish our our artifacts to our maven repository called repo.js and finally, all packages can be downloaded from package.js that I owe um those are yeah public services except that no all, except for this.ti, but still- and we have two main three main key repository involved here. So we have the first one, which is about open vpn that lets you that allow us to configure the open, vpn docker image, um but also to configure it and to to specify who can access the vpn access.

B

So we both use um ldap authentication for the vpn and certificate. So someone need to trust a tool to sign your certificates, so you could connect on the vpn. We have the jenkins infrastructure chart repository where we define all the stuff related to our communities cluster, um how we configure jenkins, how we configure nexus. For some I mean for local, modern repository cache. How we test, I mean we have yeah. We have a bunch of services running on the communities cluster and we also use it to manage the vpn machine.

B

um The main reason is because the vpn machine need to have need to have access to different networks, private networks, and I wanted to be sure that we had network interface in different locations. So it was just easier to do it on a virtual machine.

B

um If, because um I think at least in my case, I'm more interested to share about the communities and how we manage this, I will just highlight the different tools used to deploy and configure release. That's either jenkins auto, so um we may use helm as the package manager for communities. So communities is just a way.

B

So, if you're not familiar with that, it's just a way to describe um resources, and so you just say: okay, you need those resources and communities we convert that to real resources, so you just provide them with fives and communities will create the volume. um The network interface of all the things that it needs for you to deploy your application and a home package is just a way to describe old resources for a specific application. Like you have a home package for jenkins, then you have home file and file is just a way.

B

It's just a tool that allows you to deploy and manage your home application on your cluster, so you just run a file apply and then it will ensure that um the right configurations used with the secrets and and so sun we use soaps. Soaps is a small tool to encrypt and decrypt the secrets and talk I'll talk a little bit about that in a few slides, and then um we use another tool that I wrote, which is update, cli to automatically define how to automatically update files based on some strategies.

B

So when you look at the deployments or the jenkins of research charts, you have a lot of different directories there. So I will not go. I would not show you a specific configuration, because I think that if you are interested by that specific session, it's just better for you to call that kit repository and look and just learn the way we are doing this because we have a lot of codes, but just yeah just give you some some guidelines about what to find where so the sharp directory contains custom, charts or custom charts.

B

The home file directory contain help file um configuration files, so basically uh we have one per application and that's where we just find. We need that helm application deploy in that namespace with that configuration using those secrets. Basically, um the cluster directory contained the definition of our main cluster.

B

At the moment we only have one I mean it really depends. Sometimes we have to uh we probably deploy more cluster, but it depends if we have the time to work on that, but for now at least we have one.

B

We define configuration secrets. um We have a jenkins file that work on communities, so we just defined the different stages that we apply when we want to deploy our application.

B

We have the bot templates, um also I'll talk a little bit that I'll talk about that in a few slides as well, but just to give you a big overview, the butt and please define the environment where you are going to run your changes file. So if you need a container with home, um that's a little place where you define that and and finally the update cli is the directory, where you specify all the strategies that we want to apply to update files um that more specifically in one slide.

B

So if we go back to the secrets, we are using soaps, so soaps is a tool, a tool from mozilla. It allows you to store your secrets in a git's repository, which makes it quite easy to audit. um You can easily roll back a version, so you can. You can configure secrets roll back to the previous version and so on. You don't have to maintain a third service.

B

um You can have different keys per file, so I found it's really useful in the case of the jenkins project, where different people have access to different services, and so um I can easily delegate the permission. The management of let's say the release environment to a subset of people, while other maintainers have access to the application.

B

And finally, we can down multiple ways to decrypt and re-encrypt um the secret, the secrets um there. So, in the case of the release project, we have an azure. We use azure key volts to decrypt this. Those files and the azure key vault is only accessible from a specific private network.

B

We use the gpg for a contributor, so if you are granted access to that kit repository, um we probably basically we need a gpg.

B

um At least in the case of the secret, it has been working quite well. um We try to not put everything in git repository, so we um on purpose. We decided to just put on the code signing certificate elsewhere and the same for the private jpg key. So those are stored on azure key votes with limited access, but then you need. um We need the git repository to access those, um so yeah yeah. Here's just to not put everything in one location.

B

Another tool that we are using, which is um something that I'm using to apply strategies. um So in this case, what we have is we just define.

B

We just define a source, what's the rule, to update a file. So in this case we just say we want to use. We want to specify the docker digest so each time the docker also refresh on the docker registry, for the image taking slash jenkins column to dedicate even and we fetch the digest, and then we check with the target so the file located here we check inside the keyjenkins.master.imagetag and if the digest does not match the one that will retrieve we just change the file commit the child.

B

uh We commit a change open, a pr, and so someone can review the change and just move forward. There are different ways I mean we are supporting different um sources. um We can fetch version for the card digest. We can fetch version from github releases.

B

I mean that's um yeah, it's just a small cli, um so just define different different conditions, and the second example in this case is um when we want to update the weekly releases. We just fetch a version from the maven repository. We fetch a version. We test that there is docker image that exists with that specific version and if it's the case, we just update the targets um so yeah. That's what we use to con continuously either update the hem sharp version of the pod templates and the version.

B

The the image using the bot templates or any other gamble configuration.

B

So just um this is what the job looks like when we deploy and configure the communities cluster. So it's trigger every 30 minutes, and so we we need the secrets we just check.

B

If we can update the files, if we, if we can, we just update them, and so we do the run some tests on on the files and then we just run the ham file uh apply, and so, if it's needed, I'm file with will ensure that the activity cluster has the latest version, and this is run every 30 minutes and that's been working quite good. For us, um something specific to our environment is the way we manage to change things master.

B

um In this case it involves three things so obviously, home charts, plugins and the gps configuration for the hem charts. We may we made a custom changing chart, but basically most of we just have a strong requirement on the upstream one. So we just tried to reuse as much as possible from the stable, slash jenkins hand chart, but we just override default values for our environment and we also add a specific commodities resources.

B

In our case, we have a third hand, chart called ham dash champions release, um but it's getting less and less important um over the time, because um the way to configure jenkins on communities evolve, and so this is something that that you try to do as well. So now most of the configuration is done inside the gcask in gcask. um So that's why that third hm chart is becoming less important.

B

So the two most important plugin in our configuration is obviously the communities plugin, because it allows us to run our load on communities and the configuration has code, which is a central piece of our infrastructure and really allow more contributors to really understand what you are doing, how we are doing it and if someone identifies something that can be improved it just um they just open a pr and- and it will be applied um next time.

B

Our job is running so inside the gcask. You can find the cloud, the cloud communicator, the configuration for communities. In our case, we specifically override the default um glp pods, because we have linux and windows containers and we want to be sure that those are running on specific nodes. So we just specify um node affinity that need to be used for every container in our cluster and and that's it. We have the configuration the credentials for um the jenkins master, and then we define the job this and we judge the job using job dsl.

B

So we have on the release environments, mainly the car and the remote components. I'll give you a demo at the end of the presentation. If I have the time um regarding the way we manage jenkins agent. um Something important to understand when you use jenkins sanchez on kubernetes um is that the environment is defined using a template, so a template on kubernetes, it's a resource where you just say um my templates contain those containers you can have one or you can have several containers the way the jackets agents work.

B

You always have to chin lp container, which establish the connection between the agent and the master, and then you add as many containers as you need, depending on what you want to use in your champions file.

B

So you have different ways to specify that template, so either it's defined inside the global check-ins configuration personally, I prefer to not put any configuration there except it's the gdp um containers, because if, for some reason um we have to configure the gnp connector in a specific way for the release environment like for the node affinity, is defined at the global configuration.

B

Otherwise, everything is defined either in the shared libraries and in the case of the release project, everything is defined close to the changes file and close to the changes file. You can either inject snippets of your yaml, um but templates or you can reference pod templates um next to the jenkins file, which is definitely the one that I prefer personally, because it allows you to run linkedin tests on your templates um and also use any tools that manipulate yaml.

B

um So that's why you always see in the git repository that who are you that we are running on communities? You.

C

B

See a jenkins file and about templates that channel or multiple bot templates, if it makes sense and regarding containers, something important that you also have to keep in mind is most of the time. If you use containers that you haven't built yourself, they will probably not using the root other user id the uid 1000, um it's usually better to use the same uid than the jenkins agent.

B

It's also usually a good idea to override the default commands, because we want to be sure that the container is not stopped. It's not stopped after a few seconds. So let's say, if you use a container for just a cli inside, you will probably have to override the command so that you can sleep for, let's say 99 days or whatever you need there. And finally, I try I mean I really like, because um this is something that's easy to manage on communities. I really like to use persistent volumes when it makes sense.

B

So um again, it's really been on the situation, um but in the case of the release environment, we have a specific website that allows you to download the that allows you to download the file from a location and that that website um use the same volume. I use a volume that is also mounted in the ci environments, so we just have to copy the move, a file into the right directory and it's immediately available from I mean to all the people that rely on that specific website.

B

um And finally, one of the most important thing to keep in mind when you work with jenkins agent is to use pod limits, um but limits allows you to have auto scaling cluster, but on the other side can be really frustrating, because if the limit is too low, you are going to face really weird issues and, on the other side,.

B

If the limit is too high, um your pod will never be scheduled on any nodes, or you will just create impression, notes again and again and again um so um yeah and as soon as you start using but limits. um You know it's just better because you're going to face interesting issues there. So on a global basis, if I can let communities manage a resource or or instead of check-ins, I just try to let communities manage it.

B

So what I wish I knew before working on this, so just keep in mind that persistent volume, um if you let communities manage them, for you, may not really be persistent, because if the communities die or whatever and just lose the volume there.

B

So if you really need persistency, then it's better to manage communities out of the volume outside communities, let's say using terraform um another, a really great tool that I'm really big fan um is self manager, because that manager allows you to to generate less accessory certificates and when you work with a lot of private services that do not have public access, um it's just a game changing because you don't have to connect in a service and just um accept an enzyme serve science.

B

Another point important is regarding the kids: uh storing your secret illegits uh just be ready to rotate them. um I mean I mean it happened, but um just pushing a secret in an unencrypted way.

B

It's just really a best way to lose a lot of time, and it can be quite easy to do so, and I mean I heard mysteries about people who just accidentally pushed the secrets not encrypted not encrypted, so just be ready that if you put your secret in a git repository, you are able to change the secret and um rapidly um because yeah it happens.

B

um Personally, in my case, I remember one time I push all the secrets and then, when I just reviewed the pr realized that my files were not encrypted, and so, while it took me few minutes to change the secret to to verify that okay, it was not possible to use.

B

um It took me days to verify that nobody were able to abuse of that and also what I've found interesting is that it just takes few minutes for third vendors to dedicate that I push the secret pbt, so I received a lot of email from companies saying um you should use your tool. You shouldn't basically push um github token um publicly, so I try to put in place few few things to mitigate that. So first now secrets on the push to public repositories.

B

So there is a private repository where only few people have access to that, and also um I always try to double check um that I don't have any secrets I'm leaking somewhere somewhere, but yeah still, um it happened, but um I mean in my case the good thing is yeah. Nobody was able to have use of that, but yeah just lose days on that.

B

Is auto scaling feature it's really powerful, but again because um for that unique body limits, um you are going to face really weird issues, and so sooner you start to work with body limits and auto skilling better. It is because um I remember in my case I first worked without the bod limits, and then I enabled the pod limits and I just lost quite um I mean I just love the last days just to understand why my job was running and not anymore, and what was that wrong issues about that file?

B

That was not able to be written and so on so yeah. Sometimes it's really weird and finally to to debug um at least but limit issues having some monitoring in place.

B

That allows you to really understand how you are using containers and what's happening inside your containers on the jenkins master, but at the same time, on the jenkins agents and easily correlate those information together, make it really easy to realize that you have a limited shooter and it's not something else and in my case, yeah primitives graphing idoki was really easy to deploy so yeah good, surprise and during this process um how to contribute to these parts. Again, um you have a few links um across the slides. um You have the jenkinson from patch repository.

B

You have the jk for a charge repository for the communities code. You have opened a vpn and um yeah you should you should be able to find all the information by yourself. If you are more interested about this specifically and now, let's quickly look at what um yeah, I don't have a lot of time anymore, but I'll try to split to speed up a little bit um just look at the release process. What it looked like today. um In this case we are involving a few more tools: bash maven python make file.

B

um Before we continue. I would like to clarify that the jenkins project has multiple release type. We have the weekly releases, so those are generated based on the master bench on the jenkins score. Repository is done every week and is triggered by a crown job. We have a test releases. This one is more carefully selected and we tend to freeze um the code, but also the scripts um used to generate the release. This one happens once a month ish, but yeah.

B

There is no street all star, and then we have the security weekly and the security tests. The main difference compared to the weekly linda l test is in this case it involves security, work and most of the work is still in private until the really the last moment where we publish everything. So um obviously you can decrease the complexity of the release environment for that, because we need promotion mechanism, so the release process is split into two different job. The first one is the jenkins release.

B

Sorry, I just have to check something, so the job is divided into categories. The jenkins release, where we fetch the codes and we published a result on the maven repository. So basically we start here. So we have um the release code is defined under jenkins, slash release. We define the jenkins file that define the different stages: the pod templates. What the environment looks like to run the jenkins file end of your scripts.

B

Then we move, we fetch a code from jks jenkins. We publish the artifacts to the github repository and to maven, and then we trigger a second job which is jenkinson for a slash release.

B

Then we trigger to jenkins and fresh release and again we have specific jenkins file further um for the packaging. We have specific bot templates for linux and windows, packaging, a bunch of scripts, and then we fetch um the packaging screen from the jenkins cr packaging, and then we publish the results on package search engines that I o um the scripting involved here are multiple and it evolved over the time that I work on it.

B

So obviously I started working in the jenkins file um didn't realize that I had a lot of complexity there and it was not necessary easy to test that so I started using make files um make files is really I mean, that's a tool that I really love, because it helps you to abstract comment complexity.

B

So, let's say instead of having really long command docker build with a lot of parameters you just from made builds and it's easy to read, and it's easy to understand that in your case, make build means that you're building a docker image, but um the same that happened with the jenkins file. After a while.

B

You start to have quite a lot of complexity in the make file which become quite hard to read, and so that that was the moment where we started up working on the bash scripts, so that bash grid became the central piece of the release environments, and so I could use the same script from my jenkins file, but also from my machine. So I could test exactly the same thing and then the bash script had most of the complexity. Obviously, but it also became the place where I defined all the environment variables and the default values.

B

The main reason is because it can be easy to override the value of an environment variable, so I prefer to define everything here and there is a convention to not define a variable elsewhere unless it's really needed, then I started looking for specific scripts, so I wrote python script when I had to manipulate your notes and xml apis, um but still I prefer to keep the python script as well as possible and always call them from the bash scripts, and obviously we have the maven release plugin um I mean for the for this reason.

B

So the release workflow looks like this. um We check out uh the git repository that we want to to build. We plan that so we just visualize what we are going to do so, which git repository we check out where we will push the artifacts. We retrieve the code sending certificate from azure key vault. We have the same for the gpg key from azure key volts. Then we run maven release.

B

Prepare we commits from the jenkins five, so we don't use the maven release prepare to commit um because we want to be able to use different git repository than those defined than the one they find in the pump that xml for those who understand what the maven does. um We stage the release, and then we verify um the gpg signature and the cuts and certificates again. Everything is available on publication repositories and for the packaging workflow, it's kind of the same with fetch gpgt checked sending certificates.

B

We download the jenkins that work from the maven repository that we find in the previous job, and then we build and publish packages for each distribution.

B

If we need we do promotion and then we force a synchronization of the rehearse.

B

What I wish I knew before working um on on this part, um first maven really is playing plugin, seems to be black magic to me in the past, and it's still black magic. To me. um Second thing is: I'm trying to always use the right tool in my case.

B

I try to not over complicate something specific, so I like to use bash. I like to use python I like to use jenkin file in some in some cases, and I always try to evaluate if I'm still doing the right thing.

B

So that's why um I put complexity in the jenkins file, then move that complexity to the make file then realize that it was better to maintain that in a bash script um and and so on, um try to always define an environment, variable location strategy, so basically try to define your environment viable in one location. In my case, I decided that the source of truth would be the batch script.

B

Otherwise it's easy to just override the value into multiple locations, and it can be quite hard to review um how you're doing so, and it's really important to decide where you put those environment variables, because it will affect all the scripts used elsewhere and finally try to keep in mind what your environment looks like when you work um on scripting, because it doesn't make sense to write a python script. If um your environment will not have python and on the other side, it may make sense to add python environment to your studio stack.

B

But again it really depends.

B

On the situation, what I propose to do here is to share um a different thing in this case.

B

I can share with you what the release environment looks like, so we have two directories, one for the components, one for the car. When we go into the car. I hope you can read.

B

uh Up so yep, I'm connected, so I went into the core directory, so there we have um two two directories for stable and for the weekly and otherwise we have the core packaging. So we have a generic um job for the car packaging and the same for the release. So if we go to the weekly, basically that that thing is just um so we we can build from the master branch, basically what it does. It just triggered um the generic packaging job by the generic release, job and the generic package.

B

And so, if we go, let's say here we'll see that we trigger the branch on the number 28 and so from here. Let's open the ocean, because I prefer the visualization there.

B

You can see that we do the different stages, um and so, in the case of a security release, the idea is kind of the same, except that this time.

B

In the case of security or lts release, because you create a branch where we freeze the parameters, so in this case for the lts, we can just go there.

B

I will just trigger this, we just trigger the job and then we are shown a bunch of parameters. So in this case we want to build a stable release. uh We can provide a few parameters and um yeah once we push the button. The release is sent and regarding just go back to my presentation,.

B

um So, just to conclude this presentation: um it's only the beginning of the the release and now that the community has the control of the release process. It's our role to make sure that um it's maintainable, that we can regularly update um our environment and we are still looking for people who can take down the ownership on the different components. So um we have to maintain the infrastructure and the terraform codes just for the infrastructure. We have to be sure that the way we configure our services are reliable and people can contribute to that.

B

And finally, we have to be sure that we control the the process and how we do the release.

B

A

Thank you olivia again. uh If you have any questions, uh please ask them in connie um and uh one question we received is uh foster ability. Is it possible to have jenkins running compromise and also run build agents in the cloud? So it's not directly related, but yeah.

B

So what's the question so if.

A

You run jenkins.

B

A

Yeah and putting engines to the cloud.

C

And mark wait here: I do it all the time, so I run on premise and use cloud agents because I don't want to host all the infrastructure on on my local environment so yeah. It works great. Now.

A

So actually it would be interesting to take a look at delivery pipelines, but uh from the code side. So if you could uh briefly represent how it looks like uh the release, infrastructure pipelines and ecosystem from the github point of view,.

B

So I'm not sure to understand what you mean by the degree so under how we published, on which degree pipeline.

A

Well, basically, the same pipelines as you presented and how we deliver them to the instance.

B

So, just to be sure that I understand correctly so, for example, if you go to the j kingston for a slash release, um so if we go, for example, the packaging job we have here young screen sharing by the way.

B

Sorry, oh no sorry.

B

It's loading the window.

B

Can you see my screen so basically, for example, in the case of the packaging, we define in the jenkins file that we want to use the kubernetes cluster. So obviously the communities can be running wherever you want. It's just that by default, it's easier. If your checklist is running under the same kubernetes cluster, because by default it has access, but otherwise you can provide the configuration that you want and then you specify a template in this case yaml file with the templates definition and then inside your um inside your jenkins file.

B

You can say, for example, from time to time, so by default. Here we are on um on linux, but, for example, we have a job where we need to be running on windows and in this case we just said use the bot template definition provided on package windows. And if we look at what that template looks like there is another directory, but templates, and so, for example, if you look at package units we just define which image we want to use, what are the limits, and so on. Does that answer the question.

A

A

Okay, so uh since we don't have uh more questions in the queue, I suggest that we just stop the recording and continue with the live discussion all right and thanks again to olivia for the presentation, we shared the uh feedback link in the chat and again we will appreciate your feedback so, but uh we can make our next meetups better and provide more relevant information.

A

um The next meetup will likely happen in two weeks and see you there and so and I'll stop the recording.