Jenkins Google Summer of Code Office Hours, 28 Aug 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: GSoC 2020 - External Fingerprint Storage project demo (Aug 28, 2020)

Description

Presentation by Sumit Sarin, a GSoC 2020 student in the Jenkins project. File fingerprinting is a way to track which version of a file is being used by a job/build, making dependency tracking easy. The fingerprint engine of Jenkins can track usages of artifacts, credentials, files, etc. within the system. It does this by maintaining a local XML-based database. This leads to dependence on the physical disk of the Jenkins master. This project involved extending Jenkins core to support storing of fingerprints in an external storage, along with two reference implementations, backed by Redis and PostgreSQL respectively. Various functionalities like migration and cleanup were developed and released. Users can now use these plugins to externalize the storage of their fingerprints. This project was one step forward in developing a cloud native Jenkins.

Talk abstract: This presentation will start with a brief introduction about the fingerprinting engine in Jenkins and its use case. Then we will discuss the motivation behind externalizing these fingerprints. We will discuss the external fingerprint storage API built during the course of the project, and what features it allows the plugin developers to use. Then we will talk about the two reference implementations built by us during the project, backed by Redis and PostgreSQL. We will discuss fingerprint cleanup and migration strategies. A demo will be presented which will show the working of the plugins and their functionality in action. We will conclude the presentation with what potential future areas of improvement can be for this project where the community is more than welcome to contribute. And lastly a short Q&A with the developers behind this project.

References:

* Project page: https://www.jenkins.io/projects/gsoc/2020/projects/external-fingerprint-storage/
* Redis Fingerprint Storage Plugin: https://github.com/jenkinsci/redis-fingerprint-storage-plugin/
* PostgreSQL Fingerprint Storage Plugin: https://github.com/jenkinsci/postgresql-fingerprint-storage-plugin
* JEP: https://github.com/jenkinsci/jep/tree/master/jep/226

A

Everybody welcome to this presentation on uh the jenkins uh external fingerprint storage project. um This is one of the jenkins shock projects. This year uh I had some amazing mentors so oleg andrei and mike helped me along all the all the way along, um so for today's presentation, I'll be starting off with a small personal introduction, um then I'll be uh talking about what exactly is the jenkins fingerprinting engine? What are fingerprints, uh then?

A

I talk about the external fingerprint storage api that we developed during the course of this project um and then I'll talk about some of the plugins that are ready for the community to use and and some and some information about them uh following that I'll be showcasing our demo, which will uh show the usage for uh how to use these plugins uh and following that we'll have a small q a so I'll start off with the personal introduction, I'm sumit, I was the juicer student for this project, um I'm currently pursuing a bachelor's in engineering.

A

uh I started contributing to jenkins in december 2019 and I made some contributions around jenkins, fingerprint engine which ultimately led me to being interested and taking part. You know in this wonderful project, so I start with the jenkins fingerprinting engine and what exactly fingerprints are so fingerprints inside jenkins. uh They are nothing but a way to track the usage of artifacts or files across the entire jenkins. uh You know different jobs and builds uh across the entire cicd flow, and that makes dependency attacking easy.

A

So to you know, uh to take a small example here um say: team, a builds, uh an artifact, you know, aid or jar and team b is building. You know, b jar and uh b jar consumes eight or jar and team b finds. You know there is some issue in eight or jar and they reported to team a now team a needs to find out exactly which version of aid or jar uh is team, be using so um so.

A

This is one of the use case for fingerprints and they can easily using the think jenkins fingerprinting engine. uh They can find out that uh team b is using, which uh version of eight or charge so they can, you know, fix the bug so I'll show a live example of uh this usage inside jenkins. So here I have a jenkins instance configured with two jobs: a job a and job b. Just like I talked about in the example.

A

So if I configure a build for job a uh I can go and see that the build that was created, I can see its fingerprints, so I can see that it has fingerprint recorded the fingerprint for one file, which is a dot jar, and I can see its usages across jenkins, so I can see that it's only been used in build eight of job a now.

A

If I go back and I start a build for job b, which actually consumes a dot jar and then I go to c which fingerprints are recorded, I can see that it's recording the fingerprint for eight or jar whose original owner would just job is bill date. um I can see the usage history for this uh and I can see that you know this particular artifact was con. It's used in jobs, a's bill date and job b's build it.

A

So that's how across uh that's how the fingerprinting engine in jenkins is allowing me to uh you know view this usage of a.j across the entire ci cd flow, so uh that was a small live example for jenkins fingerprinting engine and uh I've discussed the you know.

A

I've shown the fingerprints ui this ui inside jenkins, so now I'll be talking about uh what what was the motivation uh behind what we did in this project so currently uh so before this project was created, um it was you know we took this over um the jenkins uh fingerprinting metadata was stored inside the local disk storage of jenkins home, so it was stored as xml files, and uh it has its it had its. You know own set of disadvantages that as the number of fingerprints increase, uh the disk storage of jenkins.

A

Also, you know takes more space and uh you know you cannot configure uh pay as you use cloud storages that are you know uh much cheaper these days. um Also, you cannot, you know, configure say replica sets to you know, provide a better reliability, availability, uh backup management is harder and last of all, each jenkins instance is storing the fingerprint metadata for only the for its own fingerprints. So um so say if you have multiple taking instances and you want to track this usage across lincoln instances is not a common store for all these fingerprints.

A

So with these concerns in mind, we built the external fingerprint storage api in jenkins code.

A

So the idea behind this architecture of pluggable storage uh is that uh jenkins cool provides this external fingerprint storage api to plug-in developers and uh plugin developers. Can then uh you know, build storage, specific plug-ins, so say there can be a radius, fingerprint storage, plug-in or any uh you know, say: uh postgres, fingerprint storage plug-in and uh the users can just install this plug-in. They can configure their own instance, and just you know, after that, all these fingerprints start getting stored inside the storage.

A

uh So that was the idea behind this api and also uh you know that a single such storage instance can act as a store for multiple jenkins instance fingerprints. uh This idea of externalizing a component for inside jenkins. You know it's a wide initiative inside jenkins to make think it's cloud native.

A

um So there are other stories also, for example, externalizing the storage of configuration files, uh logs, etc, and you know I can view these all on the cloud native uh page or the cloud native sig page so I'll be briefly touching upon the external fingerprint storage api. um So if you are, if you would like to know the various design decisions we took so there's the jenkins enhancement proposal, the jeb226, which highlights those in detail.

A

We create introduced this api in jenkins code 2.242 and uh since then we have been consist continuously uh upgrading it, improving it and uh even so up tilting instead of 253, something related to fingerprints has been going out. uh So apart from the basic uh methods that this uh api offers, you know likes loading saving deleting these fingerprints. There are some other features also that it offers um so those I'll be discussing in the following slides.

A

But if you are, uh if you would like to you, know, uh get to know more about the methods that cpa offers feel free to refer to the java doc that consider that contains uh some more information on this.

A

So, uh as I said, one of the features that we also implemented that this api offers is a fingerprint cleanup. So uh we introduced this facility in zinc in score 2.048.

A

So, basically, what happens is that, as as you know, these builds inside jenkins get deleted, this fingerprint metadata might get obsolete and these fingerprints need to be cleaned up or they'll keep consuming extra space. So that's the cleanup facility and there's a job that runs in jenkins daily to clean up these fingerprints. So we extended the this functionality to um jenkins uh external fingerprint storage api so that external storage plugins can perform and configure their own fingerprint cleanup strategies.

A

So we offer these some methods in the api for plugin developers which they can use um to. You know allow users to configure cleanup and, and they can implement it in the most efficient way possible, because the api is generic enough. So, for example, the radius fingerprint storage plugin uses cursors to traverse all the fingerprints and clean those up.

A

So that's how the latest plugin does it um also. um You know we understand that it might be the case that this performance overhead just does not make sense if the storage is very cheap. So in the fingerprint section inside, when you configure this uh configure fingerprints, uh you can go ahead and disable fingerprint cleanup, so users can do that if they are, if they feel that it's just not worth the extra performance uh overhead.

A

So that is also that's something we left up with users. uh Another feature that this api offers is a fingerprint migration, so we introduced fingerprint migration in jenkins a 2.251.

A

um So what fingerprint migration does is that for those users who already have fingerprints on their local storage uh jenkins, core automatically takes the task of transferring these fingerprints to the newly configured external fingerprint storage.

A

But we do not do that in one go. So as uh so, you know there might be large number of fingerprints and it does not make sense to you know uh from a performance standpoint to do this in one go, so we implement a lazy migration strategy so as and when these fingerprints get used, uh they get migrated to the new configured external storage, so this allows gradual migration of old fingerprints from local storage to the new external storage.

A

Okay. So uh lastly, um I'll be talking about some of the external fingerprint storage plugins that are available for the community to use um the first one that we developed during the course of this project was the redis fingerprint storage plugin. um So it's currently uh the latest release, for it is 1.2 rc3 and it can be installed directly via the update center.

A

So all the instructions for how to use it. uh So these are all available and you can even use jks to configure it. um Also. The github page is also linked. um So we appreciate everybody, you know giving it a try and letting us know their valuable feedback on this plugin. um It supports both migration and cleanup. um So um so that's about the rightest plugin. So after building the redis plugin, um what we realized is that uh radius, so these fingerprints were getting stored in radius as blobs.

A

So all this fingerprint metadata was not queryable. um So with that in mind- and you know to uh define a relational structure to this data and uh allow powerful querying strategies uh we um and also because redis was an in-memory database, we decided to build the postgres fingerprint storage plugin. um So the latest release for that is the 0.1 alpha one. uh It's at the moment. It supports migration, but not cleanup. um It allows uh powerful querying of this fingerprint metadata and especially uh there's something called as a fingerprint facet that exists inside fingerprint.

A

So plugins can add some extra information inside fingerprints using facets, so that is also queryable. So, for example, the docker traceability, plugin tracks the usage of docker images, and you can actually trace these across multiple jenkins instances via the postgres uh plugin. uh At the moment uh you have to have knowledge of the database to actually you know, write these queries, but this has actually opened up a huge potential for other plugins to consume this facility and offer this right out of the box to the users.

A

So we also appreciate uh everybody trying it out at the moment. You can only install it via the experimental update center because there's only been an alpha release, but you know give it a shot. Let us know uh finally I'll be moving on to the demo.

A

uh So all right, I have my instance here. So what I'll be doing is uh I'll, create a new job right, uh okay, uh I'll, create a new job and I'd add a build step. uh So it's a very simple job. uh So what it does is that I like oh hello world and um create an artifact that would be say, demod or jar um and I'll add a post, build action that is to record fingerprint.

A

So this way I can record the fingerprint of demo.jar I'll hit apply and I hit save at this moment in this drink instance. I have a fingerprint uh storage plugins installed, but I don't have them configured. So what I'll do is I'll hit a build for the redis demo first um and I'll see what fingerprints got created, so it's recording the usage of demo.jar and build one, but uh so okay. So you can see this editor also right, just to make sure.

A

Yes, awesome, uh so if you see that it's actually storing this particular fingerprint as an xml file inside jenkins uh right, so so what I'll do is I actually have a server or a redis over here and at the moment it has it's empty right. So now what I'll do is uh I'll show how to configure the redis fingerprint storage plug-in so I'll go to manage jenkins.

A

uh I don't want to. I don't have to install the plugin, because I already have it installed. I just go to configure system and go scroll down to the fingerprint section. So, as I discussed that here is the uh here's the check box, which allows you know enabling and disabling the fingerprint cleanup uh and here's the fingerprint storage engine uh drop down so I'll, be choosing the reddish fingerprint storage. uh Here I can configure host code ssl database connection timeouts or timeout and credentials if any.

A

um Once I have that all in place, I can just hit the test redis connection and it gives me a success. That means uh it's able to connect uh to my radius instance and I'll just hit apply and I hit save, and now uh the redis fingerprint storage should store the fingerprints. So what I do is um I'll hit uh the build. I didn't build for this job and, let's see what fingerprint it created, so demo.jars usage has been recorded in bits, 1 and build 2..

A

Now, if we go here, you can actually see this fingerprint got automatically deleted from the local storage. uh I have another fingerprint, but this is not the redis one. It's the one. I showed at the live example, um but if I go and hit my redis instance, I can see that this fingerprint is now getting stays saved inside meridis instance. So this is the latest demo uh fingerprint that got stored as a blog inside redis.

A

So this was what I was talking about. So this is what I meant when I was talking about fingerprint migration. It automatically migrated this fingerprint from the local storage, the external storage. um Lastly, uh I'll also show um how to use you know the postgres plugin, so I can go again to configure system um and then the fingerprint section I can choose postgres this time and I can configure the details. So I have a database name demo already and I can add credentials for it.

A

So just a disclaimer that we have, you have to create a database on your own. uh The users have to create it on their own and rest. The plugin can, after that use it. So if I hit test the connection, I get a success, but I also need to hit perform postgresql initialization so that this schema gets created inside my database.

A

uh I got a success so now. What I can do is I can hit apply and I can hit save so now. uh So also another side. Note that, ideally we should not be. You know configuring multiple storages at the same time, so I mean um you know just configure one external storage and then use it, but for the demo, I'm I'm actually changing it to the post. Just one and I'll hit the build for it um and let's see that the fingerprint was recorded properly.

A

Yes, and if I go to my um you know, postgres interface, I can see that inside the demo database there is a fingerprint schema that has a table for fingerprints, and I can see that it's recorded the fingerprint, which I asked it to and it's storing all this information, like which instance created this fingerprint. What's the file name uh which job originally created this fingerprint and if I go to the table uh job build relation, it actually stores which uh jobs and builds used. This fingerprint and the facilitation table stores uh the facet usage.

A

uh There is a phase three post that I've also linked in the presentation. uh It highlights some of the common queries that you can use to query this database um and essentially um you know, facets jobs will and anything can be queried. And if you have multiple jenkins instances configured, uh then you get all this information across jenkins instances. So that's a great uh you know use case. If and maybe this should in the future, plugins can use it and you know, write offer a title box to users.

A

uh Finally, uh next steps, so contributions are always welcome. uh We, we hope that more plug-in developers will come in uh they'll, implement this uh more most different. These plugins for more different storages um cleanup is currently not offered by postcast once we implement that we can go ahead with the rc release for it and uh we discussed some optimizations that can be made to the api.

A

um So you know there's some work. We could do there. um Otherwise we appreciate you know everybody to try these plugins out feedback is always valuable to us and if you have certain use cases uh that we could help, you out with we'd be happy to look into them uh before I move into qa. There's some links also I'll share the this slide deck on the project page also, and also before I move into q. A I would like to you know thank uh my mentors, oleg and mike.

A

I think it was a um they they um put in a lot of effort. uh You know helping me out. We had long design discussions. uh Ultimately, I think uh that made me a better programmer also, and um it was a great summer and uh and definitely the all guidance for jenkins uh done. A great job in you know this entire program really thankful to them, and I look forward to you know contributing more and uh for the years to come. So thank you and uh I'll open the for any q. A thank you.

B

Thank you. Thank you, so yeah we are going slightly over time. We have another presentation and we have 20 minutes left but yeah. Let's, firstly, answer question uh from julia: are there plans to pull the external storage api up to the jenkins core, so not every plugin that wants to use an external storage needs to create individual plugins support, just traded radius, etc.

B

So I guess the question is about external storage of the about external fingerprint storage. Will you disconnect me if I'm wrong.

B

I need to run for this permission, store.

C

Yes, it's uh in general, so not only for fingerprints.

C

It would make sense to have it in general, so if I want to use it in the warnings plugin, I currently need to write an own plugin for posts, quests and so on, which seems a little bit too much plugins.

A

uh So could you repeat the question I'm gonna. I can't actually find the question. Sorry.

B

Yeah so william was asked whether there is a plan to have a generic external storage api, so not just for fingerprints but basically for any type.

A

um I think, as far as my understanding in google's uh there is a database plugin inside jenkins, which you can use for configuring uh databases from one place, but uh I think even uh when jcask also- and you know other- I think- for external amazon, s3 artifact manager.

A

So all these uh plugins and all these chips are looking into externalizing different components or jenkins, and each of them has a very uh different design that needs to go into it. So so I think this is happening at the moment in the cloud native say, corner more like a per feature basis, but um but yeah I mean if we can actually make it generic. That would also be amazing.

B

Yeah also, if you take a look at the pluggable storage page in cloud interface, you can discover that there is. There are stories for generic built results, uh storages so basically for run metadata, including call actions like warnings in g plugin reports.

B

So there is interest to have such a generic layer, and the work done by smith basically opens a way for us. I would say to use traders because the same implementation could be used to supervise data to extract the data and we could start building that soon. So it really depends on the contributions but architecturally wise. We have enough information and fingerprint storage is additional source of information for us so that we could do it.

C

Right is there any comments from the mentors of this project.

B

Okay, so mike is currently not able to speak. He said that uh submitted did a great job and the demo went really good. So, thanks to sumid uh and now comments from you uh yeah, it was a really great project and submit sumit did a lot of things which we originally didn't want. So we will we had the schedule. I would say that the key functionality was delivered during the first coding phase. We had reference implementation, which was fully operational by the middle of the second recording phase and yeah. It's not an easy project.

B

Fingerprints are part of the jenkins core, so it means that there are much strict delivery cycles, also jenkins enhancement proposal. It's a part specific to the jenkins uh current evaluation, and they would say that the submitted did a really good job.

B

While preparing the plan documenting the plan, preparing formal specification for jenkins enhancement proposal and also communicating with jenkins core maintainers team and these other plugin maintainers, uh so yeah, there will be a lot of people involved and also these implementations are quite uh complicated on the technical side. uh But still the plugins have, for example, integration testing, which involves tools like test containers.

B

So uh for me it's a completed project with a lot of when you deliver to the jenkins kind of community.

C

B

As I said before, it also builds foundation for future plugable storage development in the project and yeah. Personally, I think that this project is a total success and hopefully we will be able to finalize uh it to finally accept the jab and to make epi public by the next lcs baseline.

B

I believe everything is in place, so it's just a matter of the jenkins developer, manager's discussion.

B

So thanks a lot sumit thank.

A