►
Description
CI/CD with Open Source Screwdriver - Jithin Emmanuel & Tiffany Kyi, Verizon Media
Now part of the Continuous Delivery Foundation, Screwdriver is an open source CICD platform, originally created and open sourced by Yahoo/Verizon Media. At Yahoo/Verizon Media, Screwdriver is used to run more than 60,000 software builds every day. Yahoo! Japan also uses and contributes to Screwdriver. In this session, core contributors to Screwdriver will provide an overview of its features and capabilities, and how it is used at scale covering use-cases across Mobile, Web applications and Library development across various programming languages.
A
B
A
A
These
are
some
stats
from
screwdriver
product
usage
within
verizon
media,
so
we
have
over
30
000
build
pipelines
which
runs
over
30
000
daily
bills.
So
our
use
cases
spans
from
mobile
builds
like
ios,
apps,
android,
apps
and
a
lot
of
bills
which
pertain
to
specific
programming
languages
like
java,
nodejs
golang.
A
Also,
we
run
thousands
of
deployment
pipelines
where
we
use
technologies
like
chef
or
just
plain
ssh
and
few
of
our
internal
tools
to
manage
deployments
to
our
thousands
of
hosts,
and
we
provide
multiple
programming
languages
through
something
called
templates.
So
templates
is
a
way
which
is
a
feature
and
screwdriver
wherein
you
can
expose
common,
build
solutions.
A
We
have
templates
pertaining
to
building
node.js
apps
how
to
deploy
it
and
other
programming
languages.
Like
java
golang,
we
run
over
three
build
clusters
in
screwdriver.
Your
build
farm
is
called
a
build
cluster
and
we
operate
three
of
them,
so
some
of
them
are
on-prem
and
we
also
support
a
jenkins-based
build
cluster
which
is
used
for
building
our
ios
applications.
A
We
have
over
400,
build
nodes
and
roughly
around
10
15
application
notes
and
speaking
about
the
history
of
screwdriver,
it's
gone
through
multiple
iterations.
It
started
off
as
an
internal
product
at
yahoo
in
2011,
and
we
open
source
that
in
2016,
after
seeing
the
tremendous
value
it
brought
to
our
internal
developers
and
ever
since
then,
the
product
development,
hap
and
product
feature
will
work.
Everything
happens
in
the
open
source
and
we
have
a
notion
of
tying
to
a
developer's
daily
habits,
which
is
nothing
but
writing
code
testing
and
to
the
release.
A
So
we
enable
a
git
ops
kind
of
model
where
everything
is
checked
into
our
source.
Control
management
system
and
developer
creates
a
pull
request.
Then
screwdriver
stats
support,
request
flow
and,
if
everything
looks
good,
then
the
pull
request
is
merged
and
ultimately
it
stacks
your
production,
workflow
and
how
this
is
all
ties
up
is
through
this
yaml
configuration
file.
A
So
we
call
this
screwdriver.yaml.
It's
it's!
It's
not
if
it's
a
pretty
simple
syntax,
where
you
simply
define
your
jobs
and
its
configuration
in
this
specific
example,
it
has
two
jobs,
build
and
publish
and
it
deter
and
it
defines
what
conditions
from
a
job.
So
in
the
first
job,
if
you
see
it
says
it
requires
full
pr
pull
request
and
commit.
A
So
that
means
this
job
runs
whenever
there
is
a
request
or
a
commit
to
the
source
control
management
system,
and
then
it
defines
what
docker
container
to
run
the
build,
and
it
defines
a
bunch
of
steps
in
the
second
job
you
see
it
has
a
requires
on
the
build.
So
this
is
how
you
build
relationships
between
jobs
and
this
yaml
config.
We
visually
represent
it
on
a
nice
workflow
graph,
where
you
can
see
on
the
right
side,
it
shows
relationship
between
the
jobs
and
the
conditions
for
triggering
it.
A
So,
as
you
can
see
from
this
configuration,
it
shows
like
it
uses
an
image
that
is
a
docker
container
and
it
uses
a
node
container,
so
we
do
containerize
builds.
So
what
that
means
is,
you
are
built,
it's
run
inside
a
container
and
all
the
dependencies
and
libraries
which
is
needed.
Your
build
can
be
provided
within
the
container
and
we
ultimately,
where
you
execute
your
build,
is
based
on
something
called
a
build
executor.
A
So
in
order
to
ensure
strict
security
and
build
isolation,
we
go
one
level
deeper
and
we
use
a
different
kubernetes,
runtime
environment
called
cutter
containers
and
that
provides
lightweight
virtual
machines
where
you
can
run
your
build.
So
so
you
give
a
bit
more
build
isolation.
So,
prior
to
cutter,
containers
became
production
ready.
We
used
to
use
another
technology
called
hyper
d.
That
is
all
that
also
provides
you
a
vm
based
build
executor.
But
right
now
all
our
efforts
have
been
on
opera
productionizing
cutter
container
runtime
within
ryzen
media
build
clusters.
A
This
is
how
it
looks
on
a
higher
level
on
how
screwdriver
interfaces
with
kubernetes
so
from
the
screwdriver
api
screwdriver
determines
what
is
the
build
executor
to
use
and
if
it
sees
kubernetes
is
the
executor,
then
it
goes
and
schedules
the
build.
So
internally
within
your
kubernetes
cluster,
you
can
configure
what
is
the
time,
so
we
use
container
d
to
set
kata
a
seven
time
and
ultimately,
your
build
board
will
be
executed
using
cutter
containers
a
certain
time
so
on
a
bit
more
higher
level.
A
This
is
how
the
screwdriver
architecture
looks
like
you
have
the
user,
who
is
going
to
interact
either
using
an
acm
source
control
management
system
like
github,
or
they
come
and
use
the
screwdriver
it
ui
directly
and
all
the
actions
go
through
screwdriver
api,
which
schedules
the
build,
and
we
do
log
storage.
We
provide
a
bit
of
cash
and
ultimately,
the
built-in
mains
weather,
where
the
artifacts
are
published,
to
look
into
how
the
verizon
media
has
implemented
this
architecture.
A
A
So,
ultimately,
all
the
bills
they
get
directed
to
a
rabbitmq
based
queuing
system
which
determines
which
build
cluster,
the
queues
or
the
bills
should
be
routed
to
so
then
we
have
this
workers
running
on
data
centers,
which
picks
up
the
build
and
they
get
sent
to
either
kubernetes
or
to
jenkins,
based
on
the
build
executor
configuration
so
our
ios
bills.
They
runs
through
jenkins,
while
the
rest
of
the
bills
goes
through
our
kubernetes
infrastructure.
A
A
A
A
So
we
have
multiple
build
clusters
and
we
want
to
see-
or
we
want
to
make
sure
that
the
build
weighs
as
low
as
possible
within
the
queueing.
So
we
should
have
enough
capacity
to
run
our
bills,
so
this
matrix
tells
us
how
long
a
build
is
waiting
to
get
scheduled
or
executed,
and
we
also
look
at
something
called
the
setup
time.
That
is
a
time
taken
for
a
build
to
set
up
the
container,
and
we
want
to
bring
that
as
low
as
possible.
A
B
All
right,
so
I'm
gonna
do
a
demonstration
of
a
couple
of
features
in
screwdriver
based
on
our
api
pipeline,
so
just
to
demonstrate
our
pipeline.
Here's
an
overview
there's
a
graph
here.
This
is
our
pipeline
page
and,
as
you
can
see,
there's
each
one
of
these
bubbles
is
a
job.
So
there's
a
main
publish
beta
job
and
the
ones
on
the
left
are.
B
So,
just
to
give
you
an
idea
of
what
that
looks
like
in
code,
here's
our
screwdriver
yaml,
which
is
a
configuration
to
set
up
the
workflow
for
your
pipeline
and
a
bunch
of
other
configurations,
and
just
to
do
that
connection
that
I
mentioned
earlier
with
the
remote
pipeline
trigger
you
just
need
to
add
another
requires
field,
so
the
pr
and
commit
are
if
a
pr
is
merged
or
if
a
commit
is
merged
and
to
trigger
your
pipeline
based
on
another
one.
You
can
just
add
another
field
with
the
pipeline
id
and
job.
B
B
B
So
I
mentioned
the
concept
of
a
job,
so
this
main
is
a
job
here,
there's
also
the
published
job,
and
if
you
look
at
the
lines
17
here,
here's
a
list
of
all
the
steps
that
occur
in
main,
it's
just
a
bunch
of
commands
that
you
might
run
from
a
command
line
normally
so
in
the
publish
job
we're
actually
replacing.
All
of
these
steps
by
using
a
template,
so
template
is
just
the
idea
of
replacing
a
whole
job,
so
the
template
will
have
similar
things
like
a
bunch
of
steps.
B
What
image,
which
is
the
docker
container,
that
you're
using
and
all
these
things
to
kind
of,
simplify
your
job
configuration
and
another
thing
we've
done
in
this
published
job
is
we've
added
this
command
here.
A
screwdriver
command
is
similar
to
a
template,
in
that
it
kind
of
simplifies
your
configuration,
but
instead
of
replacing
a
whole
job,
the
command
just
replaces
a
step,
so
it
might
be
like
a
bash
script
or
something
like
that.
B
Similarly,
we
also
have
a
commands
page
that
is
shared
as
well.
Some
other
things
we're
doing
in
this
yaml.
We
also
have
this
shared
line
here,
which
means,
instead
of
having
to
rewrite
like
if
all
of
your
jobs
want
to
use
the
same
image
such
as
node
12.
In
this
example,
you
don't
have
to
write
it
over
and
over
again
in
each
job.
You
can
just
put
it
under
shared,
and
let's
say
one
of
your
jobs
needs
a
different
image.
B
B
B
The
previous
pipeline
that
we're
looking
at
our
api
pipeline,
if
you
look
in
the
previous
in
the
first
step
in
the
teardown,
there's
a
cache
teardown
step,
and
here
it's
uploading
files
to
the
cache
and
then
in
the
subsequent
job.
If
you
go
to
the
publish
job
you
can
see
in
the
setup
step
under
the
cash
bookend,
it's
downloading
those
files,
so
it
should
save
a
lot
of
time,
so
you
don't
have
to
re-install
a
bunch
of
things
that
you're
going
to
share
within
the
same
event
and
just
to
go
back
to
the
slides.
B
B
A
A
A
We
have
our
application
running
in
the
cloud,
that's
where
our
users
go
and
use
the
ui
create
their
pipelines
and
the
builds
themselves
they
are
running
in
on
cloud
exam
and
so
on.
So
that's
where
the
build
clusters
come
into
picture
and
this
allows
our
organizations
to
do
deployments
to
their
production
servers
which
are
running
in
our
corporate
environment.
A
So
this
is
question
of.
Why
is
qatar
used
so
so,
as
we
explained
earlier,
so
we
enabled
content
rice
bills
right,
and
that
means
that
your
bills
are
going
to
contain
equipment,
environment
and
within
a
big
organization.
We
have
different
teams
like
yahoo
mail,
our
finance,
all
of
them
using
scootraver
to
build
their
applications,
and
so
we
operate
in
a
multi-tenant
model
environment
and
we
want
to
make
sure
that
the
bills
get
the
strictest
isolation
possible.
A
A
I
think
the
details
are
in
our
home
page
screwdriver.cd,
and
if
anyone
is
interested
in
contributing
to
screwdriver,
we
are
october
1st.
We
are
more
than
welcome
to
guide
you
on
on
setting
things
up
and
we
have
a.
I
believe
we
share
the
slides.
The
slides
has
a
few
useful
links
where
you
can
see
our
documentation,
our
slack
channel,
where
most
of
the
team
usually
hangs
and
yeah.
We
are
more
than
happy
to
help
all
right.