Continuous Delivery Foundation cdCon 2020, 20 Oct 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Case Study: How Yahoo! JAPAN Uses and Contributes to Screwdriv... Hiroki Takatsuka & Jithin Emmanuel

Description

Case Study: How Yahoo! JAPAN Uses and Contributes to Screwdriver at Scale - Hiroki Takatsuka & Jithin Emmanuel, Yahoo Japan Corporation

Key users and contributors from Yahoo! Japan will share how they use and contribute to Screwdriver, an open source build platform designed for Continuous Delivery, at scale. Several topics will be covered including: architecture, use cases, usage stats, customization, operational tips, and collaborating directly with Verizon Media's Screwdriver team to constantly evolve Screwdriver.

Looking for something specific?
0:00 Introduction
0:54 What is Screwdriver?
2:44 About Yahoo! Japan
3:17 How are they using Screwdriver
4:29 Deploy Screwdriver with itself
5:15 Internally distributing mobile app
5:54 Teams and responsibilities
6:50 Architecture (Jenkins, Screwdriver, Kubernetes, ...)
7:46 Build cluster specs
8:28 Logging, monitoring, alerting
9:15 Monitoring dashboard
10:07 Experimental SLO "queued time"
11:10 CRE Teams User Support
12:01 Internal documentation
12:34 Customization of Screwdriver for users
13:24 Collaborating with Verizon Media
14:26 Contributing to Screwdriver
15:55 Conclusion

SUBSCRIBE CDF CHANNEL ►► https://tinyurl.com/y4e4434n

ABOUT US
The Continuous Delivery Foundation seeks to improve the world’s capacity to deliver software with security and speed. Learn more at https://cd.foundation/

A

Hello, everyone welcome to this talk on a case study of how yahoo japan uses and contributes to screwdriver sk. I am chittin immanuel. I am with verizon media and I have with me hiroki from japan who is going to be driving rest of the dark.

A

So a quick intro about my myself. uh I've been with bryson media for over 12 years. I am an engineering manager. I am also the product owner for screwdriver.cd.

A

And this is a brief agenda for today, we'll have a quick intro on screwdriver. Then rock is gonna drive the rest of the talk on how yahoo japan is using screwdriver.

A

Speaking about screwdriver, it is an open source platform which is built for continuous delivery and screwdriver is an incubation project with the continuous delivery foundation.

A

Now this diagram is a quick overall architecture on how the various components of screwdriver works. So screwdriver ties in with a developer's daily habits which is write code test. It build it and deploy it.

A

So we have integration with various source control management systems like github, gitlab and developer. Writes code checks it into your sem then screwdriver kind of takes up rest of the flow.

A

If it's a pull request, we start the pull request flow and all of this configurations can be configured with a yaml format which you check in with your source code at its core, screwdriver provides powerful workflow features which lets developers manage their entire build life cycle and all the builds are containerized and that can be run on docker directly or as kubernetes ports without further ado handing over to hiroki for driving. First of the talk, thank you.

B

Thank you. This aim, hello, everyone, I'm hiroki, takachika, an engineering manager at yahoo, japan. I also contribute to screwdriver.cd.

B

I libria science fiction, video games and cats and here's my cat aru.

B

And I'd like to introduce about yahoo, japan, yahoo, japan is a famous japanese web portal. We provide over 100 web services and I like the weather, service and transit information service.

B

There are about seven 7, 700 employees. It includes over 300 300 engineers and designers.

B

Then I'd like to talk about how we using screwdriver.

B

Our screwdriver works together with various platforms in yahoo, japan. We use some repository platform like ad factory and compute platform like ks and pivotal application service when we publish or deploy something we use adsense for authentication and authorization.

B

We also use sonar cube and the sca2 for code.

B

Analysis, this slide shows and usage of our screwdriver.

B

This data are in september, 8 2020.

B

Our screwdriver has been released in yahoo, japan since september 2017., so our school driver is three years.

B

Old then I'll share two use cases of our sc driver.

B

The worst one is deploy screwdriver with screwdriver daca images are built in other pipelines to customize, and we use customize to manage ka's manifest and we use kubesec to manage ka secrets. In this home. We compare db schema with open source screen driver and line cube control div in the prerequisite job.

B

We are planning to release through blue green deployment.

B

The second use case is distributing mobile app internally, we use crosstrain in sd template with the template. Users approached the internal app distribution system. After breeze and test jobs have passed, your jobs are done by using executa jenkins.

B

Then users install the apps via qr code for testing.

B

Next, I'd like to introduce our team and its responsibilities to provide skill driver for users, our sre team has eight members say to operation, take charge of encore application monitoring and having sni and sno.

B

Our cla team has five members.

B

They do user supports, provide user guides and do integrations with other internal platforms, and both teams develop screwdriver as an open source contribution.

B

Then I will talk about this domain.

B

This rice showed the overall architecture of our screwdriver.

B

Currently we use one ks cluster for screwdriver.

B

The cluster is managed by casting in the cluster the screwdriver components latest cluster as a build queue and sonarqueen for static code analysis outside of the cluster. We use internal s3 composition, storage as an artifact star and internal, my secret cluster as a data store, and we have a drinking cluster for ios builds.

B

These are managed by other.

B

B

Here is the build cluster spec. The kubernetes cluster consists of 20 event, workers for executa, k8 and 5 centaurus workers for executor case 8 pm. A welcome has 64 cpu and 512 gigabyte lamp.

B

The zenki's cluster has 53 agents. It consists of a macro and mac mini.

B

Each agent has one.

B

B

Next, I'd like to explain about our logging monitoring and alerting we are using this tools in our operations.

B

Splunk is used for lock, collection and monitoring our sli and slo.

B

Prometheus and grafana are used for collecting and visualizing metrics of our kubernetes cluster.

B

Alert manager and page duty is using for alerting the notify incidents to encode by slack email and phone.

B

This screenshot is our monitoring.

B

Dashboard, the top left area shows base stats that increase the number of bills per day and the number of current learning and a number of client learning and pending boards. The top light area shows the ka's node status and the port status of screwdriver components.

B

The middle area shows screen drivers, api stat, like a light of response time less than 1 second, and an increasing blade of bad status code.

B

The bottom area shows k8 cluster.

B

B

We have experimental, slow, called queued time queue. Time is defined as the time between create time and start time of bills.

B

Slo is a day in which diablo's q's time is over 30 seconds shouldn't be over 5 days in last 30 days.

B

I know it's not good slo, because it's counted by the days but visualization is meaningful for us. We can see the improvement about skill time like the last graph.

B

We are also tracking percentile observed time. Like the next graph. The longest stack blue bar is 99.

B

It may indicate a bunch of users using some big daca images.

B

Next, I talk about topics of cla team domain.

B

In this slide, I will explain about what kind of user support are we doing? First, we'll achieve user inquiries via antiquating system?

B

Second, we provide user announcement and emergency support on the slack channel.

B

Third, we provide a github repository for taking user requests as github issues.

B

Finally, we provide internal documents I'll talk about more details about the document in the next slide.

B

We provide an internal document. Besides locating the offset guide called mastering screwdriver the title and cover are a match of the japanese famous book.

B

This includes step-by-step basic tutorials, advanced configurations and how to integrate with other platforms.

B

Of course, it hasn't been published.

B

There are two customizations of sk driver provided to the users.

B

First, we are providing trusted, hd templates and commands. These are provided by its language or platform teams, for example, java team girl, team, deca registry team and cad team.

B

Second, we provide assets and sca integration with bootcamp.

B

All that users need is to add configurations.

B

It enables to purpose if there were any bannability.

B

Next, I'd like to talk about how we collaborating and contributing open source skill driver.

B

It can be said: yahoo, japan, developing screwdriver.

B

Before we start open source collaboration, the collaboration of blizzard media in yaku japan has been studied since 2013..

B

It was for internal version of screwdriver and other developer tools.

B

We collaborated through bi-weekly meeting and exchange on business trip.

B

Then yahoo yahoo, japan, has studied contributions for open source school driver. Since february 2017.

B

There are three kinds of contributions as our contributions. First, we create github issues, it derives from our user requests or our operational issues. Second, we implement some features and fix some bags.

B

It's usually based on our own issues, but sometimes not.

B

Third, we locate the offshore guide. We translate the offshore guide to japanese periodically.

B

These are the number of created, fluid s and issues from us for three years, and this graph shows the transitions we can see. We've been contributing constantly since 2017 and we've done about quota on one fixed contributions.

B

And our remarkable contributed features, including metadata blanche and tag and release, filtering and local build feature.

B

And conclusion.

B

Yahoo, japan is using screwdriver.cd for three years and screwdriver is a core component for yahoo. Japan's development.

B

Yahoo, japan not just uses screwdriver but also contributes constantly.

B

That's all. Thank you for your kind attention.