Continuous Delivery Foundation cdCon 2021, 6 Jul 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk: How Using Screwdriver for CI/CD Reduced Vespa’s Time Spent on Builds &...- A Ressem

Description

Lightning Talk: How Using Screwdriver for CI/CD Reduced Vespa’s Time Spent on Builds and Pull Requests by 75% - Arnstein Ressem, Verizon Media
Speakers: Arnstein Ressem
When Vespa was open sourced in 2017 we looked for a continuous integration platform to build our source code on. We looked at several hosted solutions as well as Screwdriver – an open source CI/CD platform built by Yahoo/Verizon Media – that had just been open sourced in 2016. Another platform seemed the best fit for us at that point in time, but last year we became aware of big changes in the selected CI/CD platform. We decided to use the open sourced version of Screwdriver as we knew that the project had significantly matured over the past years. This presentation is an experience report on how we used Screwdriver to reduce the time spent on builds and pull requests by 75% on average.

For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/

A

Welcome to our silicon lightning talk, my name is angel niresam and I'm a principal software systems engineer at rest, media, where I work in the vespa team. In this talk, we'll look at how we use screwdriver for ci cd to reduce the time spent on builds and requests by 75 percent.

A

So what is westbound? It's a open big data serving engine where you can serve your big data and generate responses in milliseconds at any scale, with any traffic volume. Common use. Cases for this are text, search, recommendation and personalization.

A

Question answering sim structure, navigation and more so westby is fully open sourced available at the spot ai and we also have a hosted cloud service with a free trial at cloud. The spa, ai.

A

The despa code base was open sourced in 2017 under apache license 2.0 and the repository and the code is available at the link in the in this page.

A

So we have approximately equal amounts of c plus, plus and java code about 700 000 lines of each, and we have our extended test suite with 10 10 000 unit tests.

A

It's an active repository with more than 4 thousand requests per year being merged into the mass branch and all of the these pull requests are peer-reviewed.

A

So when we build vespa, we build a complete code base, always paid from the master branch and don't have any feature branches for the z plus plus code we use cmake and gcc and for the java code we use maven and open jdk.

A

The current build and runtime platforms are centos or that enterprise linux 7..

A

So we have plans for supporting 8 as well, but that will come later so what we used to have without the pipeline, at a commercial vendor with the predefined vms, with the four v cpus 15 gigabytes of ram and five parallel executors and a cache for maven repositories and simple sports objects. So this took 50 minutes to compile and test, and we got some feedback that this was taking too long to get feedback, especially on the requests so um we're being in verizon media. Where also the screwdriver team is.

A

uh Is we got access to a cd, screwdriver, cd, hosted instance, and on the right hand, side is the sign-in that you get? If you go to that page, so you can sign in with github or you can sign in with guest access. The guest access will give. You read only access to everything, that's in there and the signing with github, which is currently invite, only gives you access to create pipelines.

A

So we created a pipeline and filled in the form on the lower right here and created our pi vespa pipeline. So to get a functional pipeline, you need a screwdriver file. You need a container image which, on screwdriver is, is great, has great flexibility in in the way that you can define your build image. Only curl and open ssh are required, so we used our vespa build centos, 7 image.

A

We use vc views and 16 gigabytes of ram and we just reused our build script. So, on the right hand, side here, you can see the screwdriver yaml, with some pieces cut out to get the overview of that file. I'm not going to go any further into that one here, so we did some other things as well.

A

The job cache which consists of a lot of small files, in our case sims objects and maven artifacts. We used tar to avoid the download and upload many small files.

A

Let's speed up things, we also added uh some logic to sort short circuit. uh The bluequest builds based on the language so that, if you request only contains java code, we compile and test that to speed up things.

A

uh We also added a possibility to run integration tests apart so as part of the pre-request build, which means that we install vespa and the integration test on the whole system with some basic tests.

A

So some of the results we got github checks in place. This is comes automatically with screwdriver, so here is a request with checks. So you see that the screwdriver has built vespa and states that everything looks good, so this can be merged.

A

On the screwdriver pipeline for the westpop projects, we have only one job that is building vespa and the events pr and commit that that feeds into it there's a small pipeline, but this might be extended by us at a later point in time.

A

So, for the build times we were able to complete a build on the mass branch in 18 minutes, compiling both service bus and java code for pre-requests, with the single language contents, 8 minutes for java and 12 minutes for sales bus, and we see the comparison on the right hand side here where we started off with 50 minutes and we are able to for java coming at eight minutes, which is great for the developers submitting pro requests the screwdriver.

A

We can also see the metric view of the build times over a selectable period of time, so here we see it for the the build vespa job. So we have some spikes here, but that's queuing on the waiting for uh screwdriver executors, but uh other than that it looks uh good.

A

So uh in conclusion, we had a great reduction in precast build times, and that is very important for developers getting feedback fast and being able to merge and merge the changes and start on new prerequisites uh and screwdriver was a good fit for us.

A

We really liked the full flexibility with respect to the job images where you can prepare a lot up front and only require curl and not necessary to be installed.

A

It's very configurable in terms of the vcpu and ram so that you can tune it for your purposes and the job cache is crucial to to speed up build times. Caching, maven downloads and also signals bus objects, nice view of the build times in that in the metrics view, and that makes it easy to see if something has deteriorated over time keep an eye on that.

A

It's also possible to use doctor in jobs for integration testing and we plan to leverage that going forward.

A

So if you have any more questions and or follow-ups, please feel free to reach out to me in the contact information on this page. Thank you for your attention.