Jenkins Online Meetup, 29 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Online Meetup: From local installation to scalable Jenkins on Kubernetes

Description

Wise people say that “running Jenkins in a cloud environment is not a trivial task”. Find out how we aim to simplify that, using the Jenkins Operator for Kubernetes.

The trend to move infrastructure to cloud is undeniable. Jenkins has been of great help with building applications for countless developers for over a decade now, but it does have its limitations. Running it in a cloud environment is nothing but a trivial task. See how we can leverage the new and exciting technologies such as Kubernetes and Operator SDK to overcome the shortcomings. As a result achieve truly scalable and robust Jenkins in the cloud using the Jenkins Operator.

References:

- Meetup page: https://www.meetup.com/Jenkins-online-meetup/events/278322004/
- GitHub repository: https://github.com/jenkinsci/kubernetes-operator
- Website: https://jenkinsci.github.io/kubernetes-operator/
- Roadmap: https://github.com/jenkinsci/kubernetes-operator/blob/master/ROADMAP.md

Speakers:

Mateusz Korus - a DevOps Engineer at VirtusLab with experience working with high-traffic cloud infrastructure. Enthusiastic about reactive technologies and the observer pattern. Tea enjoyer.

Piotr Ryba - a DevOps Engineer at VirtusLab, working on the Jenkins Operator project. During his professional work, he wore many hats ranging from Software Development to IT Operations. Experienced in programming mobile apps and microservice applications.

A

Hi everyone welcome to the jenkins online meetup. Today we have a presentation about uh uh moving from local installations to scalable jenkins and kubernetes. We will talk about the coordinated string, simulator uh thanks to matthias and peter for joining us. They represent mr schwab, one of the main contributors to the zinc separator of kubernetes and yeah.

A

Today I hope you'll have an interesting presentation and yeah if you haven't used the jenkins operators operators before if you haven't used burner situations before, I believe there will be a lot of good topics to study quickly about the junkies online types so jenkins online, without anything about jenkins, including jenkins for users, junkies for developers, changes for the community.

A

Basically, these meetups are quite informal. So we for discussing topics we have tried to have life clay live demos, we focus on interaction with users, we will have kunai and we will have after party well a kind of after party after discussion um and yeah you're welcome to participate during the presentation. After the presentation, we will again share chats where you can join and ask the questions.

A

um As you may have heard, we do a series of junkies on kubernetes meetups. We started this session last year. We had something like six meetups during 2020, and we continue this year with. This is our second meetup. They will also meet up about oopsfair and jenkins and we will be talking about kubernetes a lot uh during uh cdcon and during the contributor summit on january's 25th.

A

Sorry on june 25th so feel free to join and also there is devops world which is coming in september. 28Th, 30th um and you're welcome to join uh so deadline uh for call for papers is actually today um you represent um the program committee there and, if you're interested please apply uh a few weeks ago, we did uh presentations about uh uh how to apply and what would be the community agenda devops world.

A

So just consider that and yeah shout out to the sponsors of the meetup, so it's continuously different foundation which sponsors the platform and called this which sponsors uh uh time for horse and also um uh schwa et cetera. So, let's move on um and yeah for junkies and kubernetes. We always look for speakers, uh so any experiences about jenkins and kubernetes relations are welcome. So how you use jenkins on kubernetes, how you do automation, how you develop your pipelines if you develop plugins?

A

So if you develop things like health charts or genki, separator you're more than welcome to join and present, if you just just do an integration with kubernetes tools, you're also welcome to join so basically we're interested in the entire cloud native ecosystem.

A

So you may have seen this huge diagram on the cncf, so basically, whatever they're, not just kubernetes, is a subject of interest from us and you're welcome to present. So there is a link here below. I will share lately in the chat, so you can just uh submit a talk and we will be happy to host this meetup speaking of hosts. Today we have two hosts, uh so it's uh me my name is of jenkins contributors.

A

I work on many topics and I also have a nice online meetups and we also have who joined us today. Please welcome him. Would you like to do a quick introduction as a host? No okay, that's fun! Okay! So, let's uh keep working together and yeah for kuni uh for coney, we agreed that the questions will be answered after the main part of the presentation.

A

If you want to ask uh questions, please use zoom connect or the chat. I can ask to use usual query. You can see the controller in the bottom. There is connect section, ask any question and we will either ask answer synchronously in the chat or ask speakers after the presentation.

A

Also after the main part of the presentation. Please stay online because we'll have open discussion, so we will stop the recording. So any questions you would like to ask related on related. You can ask them of the record, and you mean what speakers to stay as well, so it will be very informal and you are welcome to participate in this part as well. After the meet up, uh there is change separator on the virtuslab slack.

A

I will share the link in the chat, so you can join this channel and ask any questions about the operator today and the entire contributor community is in this chat. So there will be many people who are interested during the meetup. Please keep in mind that we have a code of conduct in jenkins. This code of conduct is very simple. Please be nice and yeah. There is more text here, but it's quite short, so I won't stop much on that and yeah. Please welcome mathias and peter. They will do the main part of the presentation.

A

B

A

B

C

Thank you very much and let me just share my screen.

C

Okay, does it work? Can you see it yep yeah thanks great. Thank you.

C

Okay, so welcome everybody to our presentation about a little bit of a journey from a local jenkins installation to a kubernetes cluster deployment, and during this presentation we would like to show you how different size organizations might deploy jenkins and the challenges they might be facing on the way. But first, let's talk a little bit about ourselves.

C

uh Okay, so my name is piotriba. You can call me peter I'm a devops engineer at victoslab and it's together with me is mata skoros matt. Would you like to say a few words about yourself.

B

Yes, hello, everyone, my name is mateos korros and I'm very very pleased to be here uh and share the presentation with all of you and us as well as uh piatras the co-presenter. So we have some interesting stuff prepared for you and we hope you like it.

C

Okay, thank you very much uh yeah. So this presentation is split in two parts.

C

I will be talking about the challenges with different ways of deploying jenkins and matt. Taylors will talk about how jenkins operator can help you when using jenkins on kubernetes and about some plans for the future. So, let's start our journey with a sample project.

C

Okay, so say hello to alice and she has a business idea.

C

She works on a mobile app that helps you keep a healthy diet, but, to be honest for our purposes, any project works so she's written some code has some tests and she would like to integrate a ci into her repository to build tests and upload the staging build.

C

So naturally, she looks at jenkins a well-known option that will get the job done and what is more, it's free, so yeah great, that's the app okay. So alice would like to keep things simple, so she decides to use docker.

C

She runs this command and she opened her browser. It worked. She did some initial configuration and yeah. Basically, after setting an admin password, she saw this an empty jenkins instance, uh so she did some more configuration on her jobs and pipelines.

C

For example, she might have used jenkins file and then she finished her day and went to some pierogies. Basically, she went home so the next day she comes to work.

C

She pulls up, puts up her computer realizes that the jenkins is down, and so she runs the same command again and opens jenkins and more or less sees this so again empty jenkins instance and the pipeline is gone so well, it turns out. You can't rely on the same container to be always up and you have to treat containers as short-lived and make sure your data survives through container equations.

C

So alice does a little bit of more reading on the subject and decides to change. Her docker run command a little bit. So there are two changes here. First of all, the minus the option which will start jenkins in detached mode, so it won't block a terminal window and the second change she will mount jenkins home. So this way uh her data will be safely and kept intact.

C

On her local file system, and to be honest, uh you might call it good enough at this point, you might want to tweak it a little, especially if you're a fan of declarative configuration. You might transform that into a docker compose file and apply that, but we're not done yet.

C

So alice realizes she has a lot to do in order to get her product to market and she doesn't want to miss her opportunity.

C

So she asks uh her friend bob and to help her with the app and at this point it starts to become a little bit problematic for them to share the same ci and they might be working from different places, either well, the same city or even around the world, and also how this notice is that when jobs are run, her machine slows down, so they figure out that it would be best to move their cicd 2 away from developer machines, so local installation. Suddenly it's not very practical at this point.

C

Where would you move it? Well, one option would be a thing called build machine.

C

So, basically, you take what you've been running on your own computer and move it somewhere somewhere somewhere else, wherever it's it's handy, and by doing that, you can allow more people to use that software, since it's not tied to your computer being on and around uh you may use an old pc and mini computer like raspberry pi, just bear in mind that raspberry uses different architectures, so you have to use different docker images as well, but whatever that has enough power works at this point, and I'm sure many of you have seen this kind of deployment in their careers, it's a little bit of an improvement than we had before and it works, especially especially if one team basically works from the same place same room and, what's more, you don't have to think about exposing your instance anywhere outside or figure out any security issues.

C

But this approach still has some laws and well to quickly grow, go for them and not to go into too much details.

C

uh Well, first of all, you have to do a system maintenance, uh keep your os in good shape. uh You have to take care of hardware when it fails. It will fail eventually so prepare for that and do some upgrades in case you run out of processing power.

C

Some environment environment issues might happen that could range from broken ac and your machine overheating or to somebody unplugging your machine from the electrical socket yeah and you have to deal with it. Networking and security risks. uh You have to figure out uh how or do you even have to expose it outside use a dmz, vpn or whatever you have to figure that out and lastly, it depends on the sla of the electrical power and the internet supplier. So in case your internet is down, you might not be able to access your ci.

C

Another option would be to use a dedicated server. So question is: should alice now spend a lot of money and order some servers and start looking for collocation options?

C

Well, it turns out that probably not and she wants to pivot her business. uh We have to remember that she just wants to build her up. So she'll probably pick something from any cloud provider, and this way she can get rid of a lot of environment problems and she doesn't have to worry about hardware anymore.

C

It will be kept powered to connect to the internet by the service provider, and the hardware will also live in air circuit area. So somebody stealing your hard drive is much less likely at this point, but if we were to revisit our uh drawbacks from before so system maintenance stays, we still have to do this uh hardware failures will be handled by the service provider.

C

You will still need to upgrade at some point in case your jenkins has to run more jobs, environment issues get cut down by a lot still networking security topics and, lastly, you will suddenly not uh really depend on the sla of the electrical power and internet supplier, rather the service provider, so the server itself and yeah.

C

So if we were to identify a single point of failure at this point, it will most likely most likely be the server itself. So in case, it's down the whole cicd service is down, and so the next step would be to try to address this we'd like to run our jenkins in some sort of a cluster and um well normally.

C

This would add another complexity of managing more servers than before, because well, we had only one and now we are thinking about several servers, so uh it would be really nice if we got the benefits of running a cluster while building being abstracted away from the os level, and if you even could get this as a managed service would be a huge plus where all you do is say. I want to run this application or, to be more precise, I want to run this container image.

C

So, let's imagine a world where, if a container fails, it gets restarted automatically if an underlying server dies, for example, its work gets rescheduled on under service automatically and a world where you can scale your application automatically based on used resources. So, as you probably know we live in this world, yeah, that's kubernetes, that's the description of it. uh So I think we can all agree that kubernetes gives a lot to the cloud computing world.

C

It changes the perspective, that's a little bit of complexity, but you can't really avoid that that, because it abstracts away uh the underlying os and well hardware and software, and what's more, it's available as a managed service from on the all the major cloud public cloud providers.

C

And now we are not really thinking about servers but see kubernetes cluster as as a whole. We are no longer dealing with os and services running processes and we're just thinking about kubernetes resources like namespaces spots, services, persistent volumes so yeah, and that's how a simple manifest file of a jenkins deployment on kubernetes might look like.

C

There are some apparent problems with that, though, first of all jenkins will run in the default namespace and really a you should well. It's advised to have a separate namespace for jenkins to do some separation of resources, and secondly, and probably the most important aspect, you won't be able to connect to the jenkins instance because it's not exposed outside of the kubernetes cluster.

C

So you would have to work a little bit more on that either your support, forwarding set up a service configure ingress these kind of things and, lastly, it has the same flaw as our initial docker run command, which is the jenkins home, is not mounted anywhere, so uh your data will be lost. Well, most likely will be lost, you can get to it, but it's not easy when default gets restarted, uh yeah.

C

So so far, we've been mainly focusing on availability issues and there's more there's whole maintenance and operations aspect to that. So we have to configure your instance. You have to find some plugins installed and keep them in good shape. uh You probably want to run run some scripts and there is a whole topic of observability and monitoring in. Basically, if something bad happens, you have to know where to look for issues and yeah know how to address them.

C

Maybe set up my monitoring and other things, so you'll be the first to know that your service is down and the security hardening, and I would like you to refer to the jenkins documentation on that subject, because it's quite extensive and that's well. That's that's an important part of documentation to it in my opinion, uh yeah.

C

So I think we can all agree that using kubernetes uh will put us in a very great good place to begin with, but there is more more to deal with and we have to find a way to glue things together, so kubernetes configuration and maintenance of jenkins and security aspects. All of that, so we might ask a question: uh what is one of the sources of our problems uh when it comes to running changes on kubernetes?

C

Well, it has state um it's not a stateless application. So it's not like a pure mathematical function where the output depends only on the input and jenkins has its states, so it influences that state influences the way. Jenkins behaves and I'm sure many people have experienced at least some problems with changes that are caused by this. For example, one plugin update might lead to a thing. We call dependency header where you have to figure out how to update your plugins and fix their configuration in order to project things to work and yeah.

C

So how can we solve this issue? So when it comes to configuration, we can use group scripts, which is basically the main entry point to the jenkins api. Besides rest api, and this allows us to execute groovy scripts directly on jenkins, and there are few ways to do this. One of them is to put some heat scripts in jenkins home in the special directory and they will get executed on each jenkins start, and I would like to refer you to.

C

There is a github repository open source when, where there are some very useful group scripts that are just waiting for you to go and grab them, so yeah feel free, it's there uh and as a side note. Yes, you can extend the jenkins and that's a really powerful way. You can use one of the 1500 plugins that are available to download on the site presented on the slide.

C

Just you have to be careful. uh To quote uh with great power comes great responsibility, so uh you wouldn't want to end up in a dependency hell. uh You have to find a balance with the plugins you want to install- and let's mention some plugins that I wore mentioning for our case.

C

So first of all, kubernetes plugin and that's a great way to integrate jenkins with kubernetes in order to auto-scale your jenkins agents on demand.

C

So you basically define a pod template and keep your jenkins main instance running and when the job one of the jobs needs to run uh well, the triggered somehow and a new code will be started using the pod template and after the job is done, it will be cleaned and the resources will be well received back pretty much uh yeah and another topic that will help us with configuring our jenkins, basically those two plugins configuration as code and the job dsl, so configuration as code.

C

Let us configure jenkins and its plugins using gamble files, so human readable files and the job dsl uses a specific, well domain, specific language that lets you define jobs using again a file.

C

So it would look a little bit like this. So you will probably have a git repository where you would keep all your configuration files, and by doing this uh you get well patient branches. You know all those things for free to be honest uh and yeah, so we would like to have a recipe to run jenkins using john dsl and configuration as code kept somewhere and point jenkins in that direction.

C

uh So it uses that to configure itself and there's another subject that we haven't mentioned so far. So it's the uh authentication and authorization well as a starting point. Jenkins comes with a username and password authentication, which is good. It works. But if your organization starts to grow, you might start looking at different options.

C

uh Well, for example, you can use a directory service like ldap or active directory, or use an external identifier and the identity provider to set up a single sign-on, so integrate with external service, using oh open id connect, for example, and when it comes to authorization, the builds in way is well enough to start with.

C

But if you find that you need a fine grain access control, you might want to look at the matrix authorization strategy. Plugin, which, which is really cool, can be configured using a configuration as code file.

C

Okay, so a little bit of a summary: where are we? uh So we came a long way from local to kubernetes deployment. uh You might be hoping on the same journey at different places. uh You might even start with kubernetes, which will give you a lot, uh but I think we can all agree.

C

There are still many things to start out to have jenkins run smoothly on kubernetes, and it may be important to mention that we haven't touched the subject of backups, so, basically anything that is not possible to be configured using files will have to be backed up somewhere and reverted and the goal the goal is to be able to start jenkins in a predictable and consistent way, and that really concludes my part of the presentation.

C

So I think it's, your turn stage is yours.

B

Oh thank you uh great presentation on some of the real problems that uh people trying to deploy junkies and kubernetes face, and here comes the jenkins operator project.

B

The jackie's operator project is all about automating, the lifecycle of jenkins, which itself is an automation server, so you can see that it gets pretty meta, but since we all here, love automation, I believe that everyone here is excited about the possibility to have a reputable and predictable deployment of jenkins uh possible to them on any cloud infrastructure, really that supports kubernetes, which is like almost all of them next slide, please!

B

So a sneak peek at the kubernetes operator architecture, the rankings operator vector you can see here that within the kubernetes cluster there are two parts, two namespaces the first one on the left is the namespace of the operator itself that works by managing the life cycle of the jenkins, which is run in the separate namespace on the right. The jenkins in the right in the right on the right part is just a complete deployment.

B

uh It has the jenkins masterpod uh junkies agents, as well as the jobs and it's fully functional uh by itself, but all of the uh upgrades uh uh scaling, uh all scaling is actually done by the uh by the kubernetes plugin that operator uses uh and provides to you by default, and besides that, there are two parts which are the external storage, where you can backup your your artifacts and your job history and also the uh ingress to jenkins, uh where you might want to set up a vpn or a bastion host and with jenkins operator, it's very easy to set up identity provider such as github off and all of that uses all of that within the kubernetes cluster.

B

Also, it takes advantage of the airbag model, which is the the model that admits privileges to to certain users and group of users. It is very well built it's very rigid and it's great to use and next slide, please.

B

So with jenkins operator, we are really trying to push for the configuration as code mindset, because we do believe that it's great to have your configuration as code. It allows your infrastructure, it allows your configuration to be versioned and to be repeatable.

B

It allows you to use the gitops model, which is the new hot world in the in the industry, and uh it is great for debugging for understanding the the whole infrastructure. It's not just uh you know someone some somewhere someday clicked a few buttons in the ui and now nobody knows how things work. Everything is clear in the code that you can see in your git repository as well as the history of how that state came to be, and next slide please.

B

So, as peter mentioned earlier, one of the greatest things about jenkins is its extensibility by plugins and jenkins operator, of course allows you to use plugins that you want to use here. You have an example of a manifest.

B

You can see that the kind is now not not a pod with just a jenkins image, but it's kind jenkins, which is a custom resource definition provided by the operator, and you can specify as many plugins as you want. You don't have to specify the plugins that the kubernetes operator uses by by itself such as job dsl and kubernetes plugin.

B

They come bundled with the operator. However, you can extend it. However much you like, and the command to apply, it is just as easy as it is for the deployment using a pot without using a kubernetes operator and next slide, please.

B

So, as pierre mentioned earlier, one of the big subjects for jenkins is security. Jenkins is by software and software standards, uh a pretty uh technology with pretty long history, and it has some legacy mechanisms uh we built in it that even jenkins themselves.

B

The rankings itself advises users to disable such as all jnlp protocols, serialize and uh other ones, jenkins kubernetes, with its uh initial configuration of jenkins, does it by default. So we don't have to even worry about it and uh you will. You will have a secure jenkins deployment uh without having to manually tweak every setting that uh needs to be tweaked, and uh on top of that, as I mentioned before, it uh leverages the airbag model from kubernetes, which is really good and uh the like the new industry standard. Really.

B

So uh that's uh that's a great thing about the operator. It's it allows you to have peace of mind without actually doing very much work in the department of security and next slide. Please.

B

So uh the thing that peter mentioned uh before also is backup and restore with our mindset with uh our configuration scope mindset.

B

uh The only thing that you need to backup are jobs, history and artifacts, because the whole configuration you have in your git repository and whenever jenkins starts up, it can just pull all this configuration and have all your jobs ready.

B

So you can have a restore script that runs and pulls your post, your job's history and artifacts, but the job definitions themselves will be handled by by the operator using the the job, dsl plugin and uh next slide. Please.

B

So um if you stick to uh to some of the good practices that we um that we present at, uh such as having uh you know an ephemeral state of jenkins, the repeatable deployment, basically having a recipe for the jenkins that you want to get.

B

You can use the jenkins operator to work behind the uh the curtain to actually um to actually deploy the jenkins in the state that you specify and it handles uh all the updates. uh All the life cycle that needs to be handled because running jenkins in kubernetes is not a trivial task, but with the help of the junkies operator, we hope to help it become at least less less of a hustle and more of a pleasant thing to do next slide, please.

B

So what's on our road map that actually relates to a question that was asked by thomas love about the new release of the junkies operator. We are working uh on a new schema for the junkies operator um and it will introduce more granular granule elements for for the components of jenkins.

B

So we are migrating uh from pot to deployment as uh astonish love. As per thomas love's question, we need to refactor the end-to-end tests and we also want to be more engaged with the community. So uh we need to establish some processes within the drinking operator to help you basically work with us and also on our horizon is either integration with third party systems and next slide.

B

Please yeah. So, as I mentioned before, the biggest problem with the current version of jenkins operator is that it only uses one custom resource jenkins custom resource and we are actively working on a new api that will allow for it to be multiple custom resources such as jenkins, jenkins agents, um sid jobs, and because of that it will, uh it will unlock many possibilities for future uh features and uh and upgrades um to the operator itself, and also uh because not every element is dependent on every other element.

B

The updates will be less of a hustle. Not everything will have to restart at the same time, but it is also a very difficult subject to tackle and you can see on the right. We have been uh working very hard to actually figure out uh what the dependencies between between elements are and how how we need to go about.

B

uh You know their interactions to make make it all work seamlessly and have uh as much uh uh as much good experience to the end user as possible and also we are actively getting involved with the community.

B

As of this presentation and also actively looking for feedback, you can see the link on this slide and if you are using the keys operator or if you want to try it, you can send us our feedback, it will be very much appreciated and it will uh help us shape the future of the jenkins operator, hopefully in the right way. Thanks to uh your suggestions next slide, please.

B

A big part of how we are proud to be engaged in the community is that we have become an official subproject of jenkins. You can see the uh also the link on the slide with the announcement on the jenkins page and we hope to uh to be engaged with the community and hopefully have a very good and beneficial relationship in the foreseeable future.

B

Next slide, please.

B

Another thing that we're doing for the community is that we are participating in the google summer of code, which is, uh if you haven't heard of of it a program from google that uh takes some issues uh from uh open source projects such as uh jenkins or jenkins operator, and issues or you know, features that need to be added, and it gives some students opportunity to work on those features, and it's you know uh you probably remember how difficult it is to get your first experience as a software engineer, so um we we think that this initiative is great for uh for students to have their first.

B

uh You know real life experience and also help the open source as a part of this project.

B

We are implementing uh a security validator uh that will help uh to for the end user to see security problems with the plugins that they are installed because you know, if you have, if you have uh some so many plugins, there are so many things that can have some vulnerabilities and it's important for the for the user to be at least aware of them, and you can read more about this project on the link at the bottom of the slide and next slide. Please that's the last slide.

B

So uh thank you very much for uh being here. Everyone thanks, pears for uh co-presenting with me, thanks to olek and vagrant for hosting, and um let's see the questions that we haven't answered yet.

A

B

A

We are just getting.

C

Started there is a lot of questions. There is one thing I would like to mention and there will be some t-shirts uh as prices uh for the feedback forum, one of them one of these. Basically, I'm not sure if you can see them drinking separator t-shirts, so yeah.

A

Yeah, I will report the link because yeah I'm spamming everyone with links at the moment. Sorry about that, but yeah repeating the link to t-shirts is important enough. That's for sure.

B

Yeah, so um the first uh matter that I thought- and but I don't feel like I've answered fully- is the thomas loves uh question about the new release of the operator. The new uh daily release, the new like mainline uh release.

B

We don't have uh unofficial term for it yet uh will be when we're finished, with the new schema when we have tested it, and it's uh good enough for uh for for you to try out as well, but we have also uh introduced nightly builds which are built daily from the actual state of the repositories. So any current fixes that we are implementing uh in the old api are available through the nightly builds, and you can check out documentation on how to take advantage of them.

B

Okay. So next question is about the using single pod instead of deployment, that's what we're working on and hopefully soon it will be a deployment.

B

um Okay, kubernetes jenkins operator, replace a drinking server external, um not sure about it. What's that question I mean a question from corporate training. I hope I pronounce it more or less right, at least.

A

Oh sorry, uh yeah. It was me who clicked that. So this question is: uh why is jenkins, run a single port instead of a deployment in the operator.

B

um I just said that currently. uh Well, I don't know why it's because it's uh it's been done this way by uh while we're implementing the first version of the operator, but we acknowledge there are some some issues, some inconveniences that uh come with it and we're working on uh having jenkins as a deployment and not just as a god. So I hope that answers the question. Well, it doesn't answer the why, but I hope it's less relevant.

A

If you want to proceed, the results give her pictures for the operator where you can raise any questions, feedback and raise feature requests. If you have them so I can share the link later but yeah. This is the best way to actually post any requests request concerns because that's how open source works.

B

A

Yeah and yeah another note: I shared a link to the feedback form in the chat, so we would appreciate your feedback so that we could make our meetups better. So there is feedback to organizers and also feedback to speakers, and we will appreciate all kinds of feedback and yeah we'll process that and maybe share it with everyone. So please send your feedback form, because we really appreciate the feedback and we really process the feedback.

B

Yes, feedback is also very much appreciated all the time it helps to shape the uh the future and see what can be improved. um So uh maybe php want to answer some questions from the q, a which one.

C

Well, your choice.

C

Have you tried to run this using github's approach from thomas love? I don't think we answered this one, but yeah. We basically are trying to enforce github approach with using the operator. So that's the advised way to do this to be.

A

Honest so which questions haven't been answered yet because it looks like I might be behind on questions. So the question about kubernetes inspirator replaces jenkins external service. I believe beyonce right, not.

A

C

Yes, I think yeah, okay,.

C

Kind of configurations could be used on.

C

An already running environment that didn't have the plugin before to migrate all the configurations.

C

Well, uh that's a interesting question.

C

Of course, you can install a new plugin.

C

Well, basically, configuration has code.

C

You can add it to your jenkins, but I'm not sure if there is a way to export your current configuration as a yaml file that will be compatible with configuration as code and also well what I would advise you to do to maybe try to spin up a test jenkins instance with this plugin and gradually try to well in a safe environment and gradually try to migrate your configuration and well basically, if something goes wrong, you can always start from the scratch, because you won't be touching. Your main jenkins instance.

B

Yeah well, the main thing is that you want to keep a recipe on uh how to get uh to the desired state of jenkins. And unfortunately, if you uh have a junkie that you just clicked some options in ui, it might be a challenge to um to migrate it to a configuration as code. However, if you do that- and you spend some some time on it, we do believe that in future it will be worth and it will save you many headaches in the long run.

A

C

A

Is about openshift, so the question is from andres: will the creator be available on openshift.

C

Well, it will it kinda is, I would do it, but with the changes we are currently doing.

C

The big api schema change uh that will be probably released without open shift support, but I'm not sure at this point, but the open shift support will be added uh eventually.

C

So currently as it stands, uh it should be working on openshift, uh but we are mainly focusing uh on the new api and trying it on kubernetes and openshift will come later.

A

All right one thing to mention that, actually in jenkins, we have two official operators, because at some point the red hat team started implementing uh their own operator um and basically we ended up with creating a hard fork for that. So currently, if you go to the jenkins cia, you can also find jenkins automation operator, which is basically being maintained by a red heart and they have implementation specifically forward shift.

A

So, as a user, you have a choice which project to take. You can take junk separator. You can take this jinx automation, operator and yeah. Hopefully, at some point they will still converge into a single project, but it depends on many factors, but I just wanted to say that such option is also available.

A

Okay, great okay, any other comments.

A

Yeah, so the next question is jenkins: speculator production ready like for 500 users,.

C

Well, I would answer this, uh maybe like this. It's us production already has changed itself yeah, but I guess that would be a big change. So it's always a good idea to test this first and maybe gradually move your users and basically test it out to see if it really fits your your requirements so yeah, but well, I don't see why it wouldn't be, and that's that in mind just bear in mind. There will be some api changes so maybe hold on with the production migration just a little bit more.

B

uh And about the scale of usage? uh Well, I don't know if you uh have actually 500 users working on a single project uh and um with jenkins operator you can easily spin up multiple instances of jenkins, so um you can have like one junkies per uh per team or, however, you like to split it and uh I think it will solve. If you have some issues with its scalability, I think it will help you to solve it by running.

B

Just multiple instances of jenkins and also uh junkies operator allows you to have the agents being scaled automatically using the kubernetes plugin. So I think it is. uh It is feasible to have 500 users uh using junkies operator, but maybe not like one drink junkies operator. I would advise you to maybe consider splitting your uh um your workflows into separate several smaller jenkins's and also that has the advantage of if something actually goes wrong. It only affects like one thing that actually broke something, instead of all of your 500 users,.

C

Yeah, that's actually a very interesting point. Motherboard well using operator makes it easy and to basically spin up multiple jenkins instances, so it may make your life easier, but by well and easier, easier to scale with users. So it's easier to probably uh manage one small or a few small, smaller jenkins instances that one done one huge jenkins, yeah.

B

Yeah scaly vertical horizontal, not vertical. That's our approach.

C

Okay, next one.

A

Yeah, sorry, actually I had a question about the production, readiness and the scalability as well. So what I know uh virtuslab created a product managed service based on the junk separator. So do you have some metrics or the insights from these products in terms of scalability, et cetera? So what's your experience? There.

C

Wow, that's, uh that's! That's an interesting question. uh I don't have this data, but we can dig a little bit. Maybe ask the right people and we'll we can get back to you later.

A

C

A

You want to quickly speak about this product. It's also perfect to find.

A

Or should we just move this coin?

A

Okay, it is then so um yeah. There is a long question about uh testing uh the operator, uh so my suggestion would be to pass from this question until the live connie part uh so later on the record so that uh we can uh discuss it and with the person live um and yeah. The next question is: could you please send us the link to the sample groovy code repository so, okay.

C

I will I I just have to look it up, so maybe you can go to the next question and in the meantime,.

A

Thank you, so you can just uh put it to the chat and share with a lot in this okay. So there is another question: uh can I replace my actual jenkins server uh on my visual machine with a junkie separator in kubernetes, so today.

B

Yes, with jenkins operator, you get fully functional jenkins, alongside with some plugins that we are using for it to run smoothly within the kubernetes and there's no reason why you wouldn't.

A

Yeah, thank you and have you ever looked into integrations with keep weird? So since we are talking about virtual machines, maybe it's something what was considered.

B

um I don't know anything about it. uh Maybe peter.

C

What was the question? Yes,.

A

Of course there is a project, well one of icncf projects uh in uh incubating, so basically what it does. It uses uh kubernetes device, plugin api and it allows to provide specific virtual machines as kubernetes resources so that you can acquire them. For example, uh I had an experiment when the cube vert was used to provision agent of the virtual machine and share between controller instances. So something like that, so I guess yeah. The answer is no. You haven't looked in that, uh but yeah jail is a cool feature. I'd say.

B

Yeah, we're certainly going to write it down and look into it in the future. So thanks for the suggestion.

A

Yeah so yeah I actually have seen one implementation, so maybe there will be a presentation later about that. Let's see uh so I am uh moving on um so.

A

Yeah, that is a question I didn't quite get it. uh So let's return here to begin. So uh the next question is from jk. If it's a single port and not a deployment with the operator, make sure it's running if a node, this uh this port is running calling go dance for example. So basically, what is failover uh policy for the current operator implementation.

B

C

Yeah that will get restarted.

C

To put it shortly,.

A

That's right and yeah reminder to everyone. We will appreciate your feedback in the feedback form, so yeah I'll share the link. Again, then, I will appreciate your feedback.

A

Okay, so we still have some time uh so the meetup was announced uh for 60 minutes, but we we can continue. I think, is it fine with everyone.

B

It's good for me.

A

Okay, so let's uh continue a bit and yeah if you have to drop at this point, thank you and we will publish the recording later today, um so yeah. So the next question is about nodes. So can uh one at custom jenkins nodes, uh for example, nodes which are not containers? I guess it's a bit aligned with what I said about cube dirt but yeah. What's your opinion on that.

C

uh Currently, I don't think you can.

C

But it's something that we we could think about at some point in the future and well. The problem is that.

C

The agents are contacting jenkins currently through two kubernetes services, and these services may not be exposed outside so as it stands, it won't allow you to connect your agent from the outside, so.

A

Yeah, actually in jenkins, there are two types of agents: there are inbound regions and outbound engines, so in bounties those ones which initiate connection on their own, for example, classic general pe agents, though they don't really use general pes standards and a few other types of agents like kubernetes agent, but at the same time they're also outbound engines. So, for example, ssh you built agents, vmi agents and other plugins. For example, cc2 plugin allows to connect engines in inbound mode over open ssh.

A

So for these agents you can actually use a junkie separator because, whatever only thing you need is to put configuration for this agent in the configuration is called yaml and once you put it there, when your agent starts up, these agents will be connected and then jenkins will manage them as outbound, but the agent in this case the jenkins, will be connecting to them. So, even if they are not within containers but yeah, it doesn't cover all use cases for jeans and jenkins. It's still quite quite handy.

C

Yeah, but uh in terms of operator there may be a manual work around possible, so you can try to do this and hook it up it. It may be possible to do this. It's.

A

C

Not supported by the operator by default, so there may be some manual work involved. There.

A

Maybe another example for the feature request and we have four outbound engines. So there are some examples in jenkins file, runner documentation uh and the in jenkins configuration is code plugin documentation. So basically all these cases can be reused in jnk separator. So, of course you wouldn't be running chink's file, runner, but yeah the same configuration can be used and if the plugin is present, you will be able to connect an external.

A

A

Okay, moving on then and yep thanks um so another one. I believe you answered that yeah, uh so yeah question from india. Does the operator include actions like requesting storage from your cloud provider to create the volume from api and, if so, uh how uh is it restricted to certain client club providers.

C

um That's something that's supported natively by them.

C

We are basically just making uh persistent blowing claims. So there are terrorists handled by kubernetes.

B

We are just interacting with the kubernetes api to make those claims and uh the underlying uh you know uh actual claiming happens by kubernetes itself. So uh we don't. We just try to leverage as much as much power of kubernetes as possible, and you know not to reinvent the wheel.

A

That's a totally right approach, another approach which is technically possible in jenkins and in that junk separator there is a plugable storage set of stories. Maybe you have seen some of them. uh There is a link on jenkins, so, for example, there is pluggable storage available for artifacts for test results and again it's just a configuration and a number of plugins installed, so you can use them along with jenkins operator.

A

uh So, for example, there is artifact manager for s3 which integrates with apis, and you can even submit data from agents without passing through the controller.

A

So there are implementations like that and again they can be used to enjoy the separator uh but yeah.

A

I wouldn't recommend that if you can use a computer not a static way.

C

Yeah, it's probably important to mention that we want to be portable, so we basically want to have a kubernetes as a requirement and we want to run on different types of kubernetes clusters. So we have to keep this in mind that well other usages in mind. Also yeah.

A

And yes, speaking of external storage, from what I know jenkins separator includes external credential storage by default through kubernetes secrets. Is it correct.

C

Yes, we are using kubernetes secrets and kubernetes secrets and there is a plugin that basically imports kubernetes secrets into the jenkins credentials. So.

A

Yes, thank you so yeah, it's quite important part and yeah nice that it's supported, natively I'll uh post a link to this plugin later uh so. The next question uh is it possible to spin up several junkies instances using different configurations of jenkins yeah? I mean.

C

B

Okay, so I can answer this question uh with uh one operator only can uh control and uh and provision one instant of drinking. However, you can spin up as many operators as you want, so uh you can have multiple instances with of jenkins, but that means uh multiple instances of the uh of the operator itself, uh which have an advantage that you that you can use. uh You know different name spaces for it and have that level of isolation as well, so uh yeah that I hope that answers your questions.

C

Yeah and it might be something to consider for the future to support uh more jenkins instances per one operator.

A

Yeah definitely so yeah. What do we have next?

A

So the next question here is about uh you mentioned the one jenkins controller per team setup. Do you have any suggestion on how to manage common resources? Most availability licenses now keys, can locable resource plugin be used and synced across multiple jks controllers.

A

Well, I can actually provide the answer, but you may not like it. So no, you cannot use locable resources plugin in such a setup. So when cloud we say we provide called bci, which is basically a multi-controller instance and there you may have such issues with revision resource, but there is actually native kubernetes way for that. I have already mentioned device plugins and actually device is actually something feasible: a physical and, for example, when I was working for my previous company, there was no kubernetes.

A

We were actually using another platform, uh some retention, but the principle was the same- will be adding a unique resource device in this case license and when you request an agent, for example, using kubernetes plugin, you can specify resources, you want to get one provisioning agent and then you can refer this license as a resource and you will get a kubernetes native way to manage your resources uh because uh yeah it will be resource and kubernetes will orchestrate access between many injections instances.

A

So it's uh the approach. You could use basically well in any kubernetes based setup, whether it's a separator class of junk it's called bci or whatever else. This approach will work, because in this case you will use kubernetes for resource scheduling and yeah.

A

This is something really good for licenses, because licensed quarters are really tough to manage, and if you want to know more about licenses and kubernetes, I can do a separate presentation about that, because yeah spent quite some time on handling this one.

A

It's great thank you yeah and speaking of that, this is a bit feedback, because if you want to know more about such topics, submitting feedback form is the way to do that. Because then we know about this questions and create a.

A

Backlog, um okay, so what else do we have um yeah any kubernetes sharing one workspace per job, so I guess support for shared workspaces and other things.

A

Would you like to answer this.

B

I believe it is possible to share workspaces. You can configure it in your seed job definition and if that's not enough for you, there are some things that you can do with configuration as code to hopefully fit your needs. So that's up to you, but it's possible.

A

uh Yeah also, there is a plugin called external workspace manager. This plugin was created during google summer of code 2016..

A

The first part of that was to create apis and to create an extension points, and we did that with examples as nfs and other basically shared storages, which is mapping to workspaces, but- uh and there was also a flower project to actually have native support for kubernetes and for docker.

A

This project hasn't been implemented per se, but if someone is interested, we have all the apis, so small meta of programming and you can have a kubernetes native implementation for workspace sharing within agents, and I guess in this case there are also solutions based on kubernetes resources, I'm 100 sure, but I think it's technically possible. So, for example, picture one does a bit, so it's it should be possible to implement it in junkies as well. In some way.

A

Okay, yeah I'll share the link to workspace manager so feel free to try it out, but here full disclaimer. It's not fully ready for kubernetes use cases yet: um okay and yeah. How will the research on that allocation of pots happen?.

A

C

Well, parts are provisioned uh by using the kubernetes plugin so well. First of all, you can define both templates there and then, for example, use labels to decide which job should run where and of course you can define a resource limits for those spots. So yeah.

C

And the actual allocation is handled by kubernetes scheduler.

A

Okay, so we have two questions left. One uh question is well, both questions are from beyond. So I guess it's a single question, so we can take it to the next part of the presentation um so that we can just discuss it uh just as synchronously of the record and if there are no questions left. I would like to thank the speakers and mr monteiros. uh Thanks for co-hosting and thanks to everyone who participated, we will publish the recording later. If you have any questions again, there is a operator chat.

A

We will share the link along with the recording. You can join it and ask any questions, and if you are interested to stay offline we will stop the recording and have an open discussion.

A

So thank you very much everyone. Thank you as well. Okay, thank you. Thank you.