Jenkins Google Summer of Code Office Hours, 14 May 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Windows Service Wrapper - Knowledge transfer and Code Dive

Description

Jenkins project uses Windows Service Wrapper (WinSW) to install Jenkins masters and agents as services on Windows. This year we have a GSoC project targeting better configuration management with YAML, and better configuration validation. https://www.jenkins.io/projects/gsoc/2020/projects/winsw-yaml-configs/ . So we will be doing a code dive into this component:

* How does Jenkins use Windows services?
* How is Windows Service Wrapper organized internally? https://github.com/winsw/winsw (.NET, C#)
* How could we improve WinSW configuration implementation?

A

Well, rather, a knowledge transfer session for, in the service record, we have multiple participants on the call thanks, Mike and things Boudicca for joining just to share some context for those who are interested, and this year we have a number of google Summer of Code bullets, and one of them is related to improvements in Jenkins window services. We use a component called the window service wrapper, and this component the basically provides us opportunity to burn monsters and agents on windows and to register Islamize windows services.

A

So this year we want to mo configuration and also to improve verification of integrations. Today we will do with cadet. We will take a look at window service, record repository and code structure. We will also talk about how this component is used in Jenkins, and then we will have some discussion about how we could improve the configuration so that we keep talking about the project implementation.

A

That's good, too.

A

Okay, so I will suggest that from the beginning, a window service right that is a standalone project, and now it's hosted in a Windows service track that github organization, it's a transition that we've done a couple of months ago together this next turn.

A

Currently it's been used by multiple organizations, not only by the Jenkins project, so you can see that there are four thousand stars and we almost hit 1 million downloads, so yeah it's used in Jenkins, but there are many. Other organizations isn't bad, and actually it's really good right now. What does it offer? Actually, a Windows service wrapper is a binary I wish you can execute. I am so it provides additions for the paint Renoir or the mainframe or different versions, and also for dotnet core.

A

So if you don't have that installed it and take a native executable for 64 32 bit platform and to run into this, so what comments does it offer and it provides several commands like installer union stalls, public service and also start/stop and a number of other commands, and so basically you can use them in order to install your service, your application as a service. Let me quickly show it to you.

A

Yeah, just a second I should add Emma.

A

Okay, so there is a lot of food documents which we don't need. I will just take any one from this, folder no commands, so we will just need this file side case I.

A

Am writing capital on Windows, so we drove this reasons.

A

Okay, so we have a number of files actually, in order to run a Windows service record, you need two files at the moment. One finally is executable, have to execute Abel's on common another one for the core and also the have a configuration file. So this configuration file is really simple. Well, it won't work actually, but it would install an abstract executable as a service in your Windows setup, and you can just run it so, for example, you just say.

A

Let's see what happens, do we see user account control screen.

A

Okay, maybe not so anyway, it required me to elevate their emissions and after that they installed. It is my application. So then I can go to services, and here, for example, you have an application called is called my up service.

A

mmm So yeah my up service powered we're going to save a striper, and here you can find the number of properties like startup type, also login credentials, Recovery Options and the dependencies to be used. All these options can be configured through advanced properties. So, if you go here, you can find the terrorists, mine, XML configuration file, and actually there is quite a long list of options here and hopefully we will be able to get dis.

A

Emotions in a llamó finishes at the project, so yeah, that's what Windows shell is rapid, us and say man I write and you know stole this service. Well, let's try to start that first, because yeah almost like nothing good will happen.

A

Okay, so it installed. But now because we didn't force loads and now, if you open the windows law, you can actually see that well magazine key table started because actually I have it such is applicable no I. Don't it should be just the application. Look! Okay, let's say you installed it and then switch to our demo.

A

So I can show the code structure in two ways: one I can just use github another one. I can use Visual Studio. So what would you pick up?

A

I guess, visual studio would be a bit more professional, but yep. We can just start from github if you wish. So here we have a number of directories. So all this source code is located in big Searcy folder. We also have a number of other folders, so dog is just all the documentation, including installation guides. Guidelines for reporting course configuration description, I described then we have and folder, which includes Bill camo so, which is our configuration when we use on Asia, devotes to build the project in order to release the project.

A

Right now we use another pipeline which is not published in the repository, but hopefully we will move it here. So I'd also have examples, but basically there are two examples. One example minie mo. So this is the file you have seen and another one is just simple options. So basically it's well again it's a file which includes all examples and which includes some documentation.

A

I'm, not a big fan of this file are. The intention was to just show an example, but maybe we could move for this entire file, for example, Texas new schema, so that this duplication and central moves there and users can develop separations interactively. So it's something to discuss in the kitchen.

A

A

Sir seen so what do we have here here? We have a solution so what it means that this is actually a multi project repository and this solution defines the top-level built so, for example, in Java there is example like made and parent form in multi just yet and not a model repositories. Here we have approximately the same and together also not settings, but it doesn't really provide whatever information he is just used for the built flow to the right relations and other companies, and now we go deeper, so there are three folders one is focused.

A

We will discuss them later, but yeah. Obviously it's about this, then other is plugins. So Windows service record has a primitive plug-in subsystem. This plug-in subsystem is basically documented. Just here.

A

Ya know so this game extensions, so in Windows service structure you can define all rights that were to inject additional logic and two extensions I'm who didn't obey depository, specifically so that directory Marple and runaway it was killer. So next turn I want to kill this repository and to replace by many features but said directory mother. He still make sense almost like we will give this extension engine and we need to push it over yeah, so on top and our mini repository score. So basically core is the logic of window service.

A

Rather, it consists of two components: one is.

B

Windows Service core.

A

So it's basically a library which includes the most of components and extension, API, etc, and the second project is service factor, so service record basically defines main char which a project you basically uses so the library- and it also has some pitches for loading. And finally, this repository defines how we package window sales wrapper. So, for example, here you can see here the configuration we produce so, for example, here we have.

A

Dependencies we inject and also we have targets so example. We create a zip file, for.net core, we create executables, you can see destinations where they go and that's how it happens and also we have a number of dotnet executables. So for that not executables, that is one. You just think that we know it should be else. So if you go to the built in a directory.

A

What I'm doing.

A

Hopefully, nothing copied sometimes support.

A

So here you can just I have a broken mouse. You know, okay, here we have a development repository and, for example, if we just go to assert see correlation started, you can see it to folders. We interesting bin, so here, for example, photos net to do 0. We produce a number for details, so it really summarizes the libraries we use. So, for example, we use elaborate for managing the zip files we use log4net its login.

A

We also include our extensions and we also include our windows to the record core and there is a scene where the cutable over there, but this executable doesn't include this libraries. So this is how the don't end applications are usually being shipped if you build and by default use visual studio. But what we do we actually know all this stuff into a single dealer.

A

So here you can see that we do the same basically for all platforms so for the organization's and we just take all less and less and put them, and then we generated by a single artifact. So for that we use inertia. Inertia is just one of standard tools for dotnet.

A

So what it gives us, if you go to the service you have articles here, you have the finally produced articles, so here we have examples and a PDB, it's a symbol, files for debugging, so that's all we get and this distributable switch we can distribute. So you can see that the size of them is quite small.

A

Actually this is the executable we use in Jenkins. Not only are they exhibit able for the last dotnet version is smaller because we started there excluding some code, which is no longer needed there. So, thanks to our next turn for this time for this work, so these are quite sparse, pods and, as I said, we also have dotnet karate pods they're a bit bigger. So, for example, if we go to the github releases, you can see the recent releases and not donate for artists more than 30 megabytes. So what happens?

A

Well, basically, we include dotnet engine there. So these are defaults can run on the near 60. It's a 64 86 version of Windows. They do not require that they're portable, but because they bigger. So if you use Java, you could do the same. For example, by using quarter horse or whatever, to feed my meter for distributions.

A

Okay, so this is structuring principle before we place it. Do you have any questions.

A

I'm still here.

A

Okay, yeah I'm still here, sorry, okay, so this is what we have in terms of the bill flow.

A

So all the Butte is basically defined by project files. So it's yeah I switched to visual studio because it's a bit more convenient here you so the screen big enough.

A

Maybe not yeah, okay, so here by the way, it's better to answer no worries, because otherwise I have to go and open the chat. So here the bill flow is basically defined and we produced the artifacts. So how do we test it? Okay,.

A

We haven't discussed it in the service code, yet right yeah.

A

So when you showed up it was an executable and yeah. A real Service Corps is basically a main component. So it's a library which includes a lot of things, notable things which matter in this project with this paper so so seriously script is basically a configuration file, so you can see that there is an extension point called up even the service configuration. So this extension point just defines all settings needed for me, no service record to run.

A

You may see that it's a bit old style, but basically it's structure is number classes some overrides, and this is what we need to configure all these options at the moment come from relation XML files, but also default settings which, with any use, so you can see that not all settings actually defined. So if you're, not I mean it with dotnet.

A

Here we use a lot of properties, so it saves a lot of time on getter, setters and default initializes and other things so I'm still waiting for Java to have seen it similar syntax sugar like seizure but yeah. This is what our integration, including defaults, and this saucer is descriptive. So service descriptor is a class which basically takes xml as an argument and extracts all this data from xml upon request, and then this file can be consumed so what it means.

A

If you take a look at the code, you can see that there is constructor and then there is a lot of different utility logic like personal extracting files from XML, etc. But what actually happens every time? We need a particular argument.

A

So, for example, when you need to the tree for work in the red terrain for your window service record, where your code organs goes to XML Dom and extract it from there, it's good from the point of view that basically all and fix a supply and demand, but the isn't bad, because all conflicts also verified on demand. So right now we don't have start-up time verification except a few options.

A

So, for example, we if you need stop arguments so stop are gonna see, is with the arguments politician, will stop executive all which can be involved on the termination, so this code will be processed on the point we start termination and the. If there is an error, basically it's to wait, because you want to terminate service, your manipulation is broken, so almost likely, nothing good is going to happen after that. This is the main weakness of this approach, and hopefully you could change it during this project.

A

Okay, so how is the ds5 induced? Let's go back to our main axis, R, so yeah. There is a lot of logic there, but basically we need to find the main function here.

A

What we do with arguments we receive from CLI so right now there is no special comment line interface processing, so all of July is processed, basically monitor, which again I think we could change.

A

So here, for example, you can see the execution logic, so we just initialize the descriptor. He treats our configuration right now. The configuration is basically hard-coded, so you can change the file path. There is a pull request from Boudicca which changes so that and hopefully we'll integrated zone after that we initialize loggers. So we use up Fernet and we also connect the log4net to in those logs arriving into that.

A

So if you mean you can see system events in the administration panel, so we connecting in the ship's right back to that by default and after that we just start inside and initializing a service. So you can see that there is someone producing but interesting things start here. So there are commands like install Union stall, start/stop and all of them have handlers and so this hand that's can be executed and basically they do exactly what's needed above you can also see suspicious elevated flag.

A

So you may have seen that when I installed the service, it basically asks for permission every elevation, it's a new teacher, implemented by next time. Before that you had to run the executable as administrator. Now you don't have to do that because, basically, they executable itself restarts itself and asks for innovation.

A

It happens thanks to this elevated callback, so basically just disappear. Amateurs and release accuse for them.

A

Okay. So let's take a look, for example at installation, and here you can see that basically, in order to install the service, we example the recent direct support for credentials. There is also support for defining the different accounts, but identity invokes standard libraries in order to create Windows service. So we don't write to registry on our own at the moment it was a case, for example, a couple of years ago, but now we mostly use service managers so that everything is installed there and registered here.

A

Okay, I, don't think we need to live there basically does all the configuration you have seen here.

A

So all these options, but after that is just the beginning, so once we install the service we all seen it actually run that. So when we start the service, there is another logic included. Basically we are indisputable and we try to execute that. So you can see that it can all say, elevated permissions if needed, but then and basically it starts at the service and the service once started up. It receives all Kohlberg's and start solving sleaze, Asian logic for process and away it's written.

A

A

Just second, my system.

A

It's it's yeah, so this hand ups we execute including create for services, and after that we should really cute the thing. That's just and let's find it.

A

Second, I, don't remember this part of the code, but give we have a way here, a key discovery easily. So we have our service descriptor here. Programmers just think executable. That's fine, useless!.

A

A

So here you can.

A

Start process so actually.

B

A

That in mind class, so yeah it just starts with arguments, and here you can see that it starts the process with executable. It wraps all the arguments it attached logger. So if you go to start the process logic- and here you can see that basically there's been a bunch of arguments to process and this logic just eventually.

A

Yeah, so better hooks, so hooks are used for extension invocation like post elimination, etc, but the most of the logic happens in the process: helper, it's our own class, which is located in utility folders and this on classes which clean logs process and wraps it by providing all the environment by sending proper signals.

A

When you need to terminate the process- and this date is basically federal configuration because all these options can be configured in settings and finally, you'll just start process and wait for is it so here you can say see, for example, full installation, so we need workspace, will meet arguments with only processes, etc. We stopped environments for the protests, environments again. They come from configuration plus some additional options, not that we just start the process.

A

Basically, for my dick table string and launch that and schedule looks so here, you can see that we actually capture logs from the process in order to regenerate log files as a part of our execution. This log file is then consumed, for example, by up study model. Organ systems like windows, islands or jenkins can also capture these logs and the even. I add them to support bundles if needed, but is this works? And finally, we just wait until the process exists and we do it in a separate ref. So that's it.

A

Okay, so I think we shouldn't deep dive over here. Sorry Dave deeper here: do you have any questions before we go to other part.

B

A

Well, we it's you who will record in bed. So if you don't have questions so then I infer.

A

So, for you boodikka, the meaning closest of interest will be, of course service descriptor, because most likely we need to rebuild it. Completion I started doing some more work when I was split with the visitors. So now there is me in the service configuration and some already support, contribute. Additional files, for example, download, is located in a separate repository, so we already have a kind of structure of integrations, but we will need to warm up this sriracha to gamma files so that they, every section is amended and service.

A

Descriptor, most likely will do a major cannot bear another.

A

File which might be interesting to you is yeah this XML helper. So assuming that we had existing a validation most likely, we will need to add some logic here in order to invoke ratification.

A

But this file is a relatively simple. It's just whatever invocation, which she plays with Dom and has a lot of generics to simplify the things also has a lot of logic to click on Deafness version. Nowadays, hopefully, we will be able to drop that next to the zero support, so the code becomes much simple.

A

Okay, so yeah the code is relatively small. Well I would say it's much smaller than Jenkins core or many plugins like get plugin, but it's still pretty complicated and we some test coverage in order to verify that so test all implemented. In the test repository and here, for example, basically, we have a number of tests and testicles and we use an unit framework in order to verify that it's basically in unit is just another implementation of xunit frameworks. If you're familiar with J.

B

A

Its small is similar, and here you can see that we just have a number of tests. So, for example, here main test just verifies what the common operations will be done like printing help. So here you can walk with.

A

That between study allows to invoke it from here. Okay, if not each variable here with guarantors.

A

So I'm not sure what it is think is a bit slow individuals to them. So here what you can see that we do testing for number of configurations now, for example, here we have main test. So, let's there was the previous test run.

A

Well, let's try running this interface is relatively new and it's still glitchin before that any unit had its own separate test you on so here you can see that at listen. You need to house it so here you can see that we just executed a test. You can also potentially additional.

A

But theoretically everything should be operational and you know for sure the tab is this work so be verify the results. Yeah, here's our standard output, just didn't change the photos. So here, for example, we expected we're on an existing command very good exception, so the test passed.

A

What's the pun? Okay, so for my advanced test yeah we have some tests for extensions. You have some tests for utilities. There is a lot of test helpers. We automate our process in called context, etc. So you can see that there is some test coverage.

A

I, don't know, I think that test coverage in this repository is perfect, but at least we get enough smoke test and there are always issues so why we'd kind of really test it entirely because, for example, in order to test window service management, you need a Windows version which actually supports that. So you need to run as administered it's technically doable now in a CD box.

A

It wasn't doable before in so we're using cap here and if you try to do it in Jenkins, most likely it won't be possible as well, but right now we don't have integration tests which would verify the service part. So it's something you need to keep in mind because this part their likely it's manual system, but you know just a fine.

A

Okay, questions.

B

Between starting in this yes la mode answer, smooth.

A

Ok, so basically, let's go to me.

A

So right now we second actually so here how we discovered it and we'd have a CLI mode, basically checks the number of arguments. So if you supply more than zero arguments, then you're running through the wine mode and basically this is an extracted promote for admins. So, for example, here help or whatever.

A

So all these commands they being security and in the interactive mode and basically Bitsy's management CLI. If for no commands supplied when it will try to run as a service or it will assume that it runs as a service, there was a push which in recent versions, where an external try to improve this behavior, so to example, right now, if I just run it without parameters, it will show me that I cannot start service from command line or a debugger, and maybe service must first be installed or whatever.

A

So, basically, it's the try to start as a service to try to retrieve free service metadata. It failed and the you bet in error. So this behavior is material. It would be better to just can't help, and the next turn tried to do that a while ago, so innovation 2.60 this didn't.

B

A

Work out and we had to divert dispatch because they post regressions so right now we about not the tibial behavior, but at least it works.

A

Okay, so when we run, we use one service mode. Basically, everything runs this July, so.

A

This week, so here we see three preventive service, metadata, etcetera, but it runs, which is a common solidification with all these commands, etc. We share this motor. Of course it becomes a bit more interesting. So here you can see that week. She will start a new instance of Windows service tracker.

A

What does this to us? That's so yeah. We start the service base, so the HSBC default ET is applied by a mini API, which defines all the service behavior callbacks, because operating system will be talking to us and let's take a look at our service tracker, so our service record is defined within the library and it has a number of flags, but actually it needs to supply some information, like veteran, for example, witness to all the process and all the definitions for runtime dosing is too exposed to the status. So this is common logic.

A

Here, for example, we start the service and parser is descriptor. So basically we passed interpolation here and when our service, we will be basically managed by Windows, and there is a lot of logic there.

A

This is a start-up project where we approach configurations way to actually start. There is one start method and one start method, but what it happens because the retriever environment variables. This is fine. By out the deviation, then we handled file copies, so that is a way in Windows service, rapid, incremental, big immigration, for example. If you want to top day-to-day service, if you want to change the configuration, you can define mutation in the configuration file and actually display after that it does not move. So he even see a funny thing called again.

A

That code is completely different. They only be next and on basically its dotnet 4.6.1 and those dot medical. So here we use the Med features, but for all the versions we don't really have a standard methods are to download files etc.

A

So we will just do it manually again codebase in the repository, so here we download all files configured by the configuration and so on so on so basically, Windows service will implement the hook and though this management simple shut down when it receives a shutdown, it processes the hook from again windows management interface, and here we just stop with service. After receiving I mean we do about the configuration processing, etc. But basically it's an intelligent version of killing the process with somebody.

A

And here we have a super useful feature in the configuration it can in deep on the shutdown and it was implemented clock ago, but it's still part of the system configuration if you need that. Ok. Does it answer your question? Boadicea?

A

Ok, so here this is the code. Actually, the code is that relatively simple and reputation management part. Yes, it's not this simple in all this service, descriptor and Kohlberg's, but hopefully we won't need to modify that code too much during this project.

A

That's the fun!

A

Let's take a look at how to Houston on the gentle side, so injector side by default in Windows wrapper we publish github producers, we also publish versions, no gate, but Jenkins accuses edition were a way to retrieve the repositories.

A

Change, ok, but here, for example, we can go to our repositories and well to the jinkies repositories and he's. The first thing you can discover is maybe packaging for ministry with rapper. It's a historic artifacts, because I think flows Italy made in packaging, and here we can multiple efforts as moving and for them to drink it's made in the depository. So this is a very busy Tori which we actually consume, giving Jenkins.

A

And basically, if you go to our tea factory here, you can see that it's just a service wrapper, and here you have a number of creations, so the last tray this was 2.7 to 0. Here you can see that we actually included net for.net, 4.6.1 and empty version which is actually doesn't so. This is all we include right now we don't publish medical versions but yeah most likely. We won't need that if we ever decide to do that so ways, this library concerned, we have 3 e FS, so, firstly, its Jenkins course itself.

A

So Jenkins yeah sorry Rob, Jenkins Jenkins, so in Jenkins repository, if we include window service proper in order to support a stallion Jenkins as a service, actually updates that cut up. So here, for example, you can see that there is Windows service life cycle. So it's a special logic which allows Jenkins to properly operate as a Windows service. For example, he updated windows service, Rob Pattinson, if needed, and for that it uses this film in relational schema, which also supports restart project.

A

So you can restart from the web UI if you use Windows service and other such bits. So this logic aces and but the first thing that I'm not really sure that this logic is being used by anyone nowadays, so how we actually recommend installing a Jenkins from windows. If you go to our download side, drink.

A

Here you can see that you can just download the windows and it offers your and MSI package. So, for example, here I'm not sure what yeah it offers I need to download that Genki. So myself, this MSI file is defined in packaging. Right now we are working on complete replacement packaging for nucleus automation, for but in principle it's the same. There is an installer and this installer it's based on weeks. It's Windows installer tool, kit and it actually includes the same Windows service wrapper with XML configurations.

A

So here you can see that there's, for example, exact config file, XML file is being generated somewhere in the scripts and we bundled which is repet as well. Any questions mike has ok, so this is Windows service. Another area when we work we use Windows service is actually for agents, so that is in Windows agent. Instead of repository, and this repository again uses Windows service record in order to install agents on Jenkins, there is some magic there.

A

But if you go to this repository, you can see that there is XML file which actually it a template. So here you can see that this is the template XML file. We have some generation where you can check IDs for you in Jade, Java version, special machine arguments and also we inject the agent so download URL, because Genki supports automatic upgrade of agent jars and of window service wrapper and the.

B

A

To do that, we just go generic config dependent on the operations so which these PS. We enable that, for, if your instance is expose for HTTP nodes of Nabal.

A

Okay, like you, will have to do some updates in these components and during the project in order to apply new features to Jenkins so, for example, support integration with llamo, but it's likely a second or third code phase. So right now we do not have to worry about that.

A

Maybe another component you may to know about, though we definitely won't be touching, that my windows agents historically Windows slaves plugin, so this plug-in actually allows two windows service management interface model to install agents as services remotely using Depot other remote communication services. So you can connect to remote windows instance. You can install Windows agent on there, connect your master and keep reusing to this as an agent. So when this link works, pretty fine, it's widely used, but at the same time I wouldn't recommend it for modern Genki setups.

A

Okay, these are four main use cases within the jenga's project and I believe, if you just Google for being of service wrapper and the network, you can find more usages again. Jenkins is just one of examples where, in the service rep, it is used.

A

Okay, Boudicca: do you have any questions? We would clarify anything.

B

Correlation I had document government, and this.

B

So other aspects that I have to cover during the documentation.

B

Almost all the team that do difficult here, I think also covered in the top yeah.

A

So that's why we have acceptance criteria in this project, so right now disco dev session. So basically you you develop regard which is related. That short, it would include review of the project structure. So basically the first thing we'll be doing in this video and I wish would just include a link to this video and that's it so now you are not expected to document everything we discussed day just top level so that potential contributors to start developing the company.

B

So I have to maintain external looking Monteux I, don't have to upload it into repository or something like so I just have to maintain it external document, I think I, don't have it update into in markdown or anything no.

A

It's no I think it should be in the repository because of the communication of Scott approach, but it can be a new file so, for example, how I would see that so there is top folder here, for example, we create a folder for developer documentation.

A

The way you can just create a file project structure and then link from tributing guidelines.

A

Okay, so anything else to discuss today.

A

So again, just a just overview. We didn't really die from the configuration. Yet there is a lot of information to process, but we can do it incremental. So I definitely do not expect you to digest all this information right now and we have a lot of time to discuss it.

A

Okay, so if there was no other topics and gain yeah, so next time we meet, we will be next Tuesday. Today we can have a follow-up discussion. So if you have any questions, we can give the particular come yesterday, I'm happy to do more sessions using it.

A

Thank you, Tom I will stop the recording and I will try to publish the video by the end of the day. Thank.

B

You thank you so much. Okay, happy.

A

To help and thanks for working on this project so yeah, you have already built it locally. So now we can try to do some patches.

A

Thank you and have a nice day.