►
From YouTube: Developer Experience on CD: Build a CD Platform to K8s that Develop... Euccas Chen & Tobi Ogunnaike
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Developer Experience on CD: Build a CD Platform to K8s that Developers Love - Euccas Chen & Tobi Ogunnaike, Pinterest
Application Deployment on K8S can be quite convoluted, especially for an organization that operates thousands of microservices. Pinterest is a visual discovery engine that serves over 250MM users. For successful adoption of K8S, it is imperative to provide a well integrated self-serve CI/CD platform that abstracts K8S complexities & offers a simple path of migration for existing workloads. This talk will discuss how we build a Continuous Delivery system for Kubernetes at Pinterest, and how we help engineering teams to deploy and migrate their services onto Kubernetes. Topics include: Kubernetes and deployments at Pinterest Introducing Hermez and the Continuous Delivery experience on K8S How do we design and build the CD system, and lessons we learned Our journey of onboarding and migrating services to the new CD system and K8S
https://sched.co/UaXd
A
Hey
everyone.
Thank
you
for
coming
to
this
talk.
My
name
is
Yu
kolchin
I'm,
a
software
engineer
at
Pinterest,
and
this
is
Toby
with
an
IT
he's.
Also
a
software
engineer,
and
we
work
on
the
same
team
so
may
know
anyone
in
this
room
use
Pinterest
I
see
many
hands
awesome.
Thank
you.
Pinterest
is
a
visual
discovery.
Engine
is
a
positive
corner
under
internet.
Our
mission
is
to
bring
everyone
the
inspiration
to
create
a
life
they
love.
A
Today
there
are
over
300
million
pinners
all
over
the
world
every
month
on
Pinterest
and
they
have
created
more
than
two
billion
two
hundred
billion
pins
to
support
the
massive
amount
of
winners
and
their
activities.
Our
infrastructure
has
a
huge
footprint.
Internally.
We
have
thousands
of
services
running
ranging
from
the
micro
services,
with
a
handful
of
CPUs
to
the
large
monolithic
services,
from
machine
learning,
jobs
to
data
processing,
jobs
from
Java
code
services
to
Python,
nodejs
or
mexico
services.
A
We
manage
a
pretty
large
infrastructure
and
lachimo
I
work
on
is
the
engineering
productivity
team
within
the
infrastructure,
our
team
build
and
maintain
the
fundamental
platforms
and
system
that
are--
Pinterest
engineer
would
use
every
day
for
their
work.
For
example,
we
own
the
source
code
control
the
build
platform
and
see
icd
platforms.
Our
goal
is
to
provide
a
fast,
safe
and
a
delightful
path
from
idea
to
production.
A
So,
in
the
next
30
minutes
we
will
cover
three
topics.
The
first
one
is:
we
are
building
a
new
CD
system
for
kubernetes
at
the
Pinterest.
We
want
to
share
with
you
how
we
design
and
build
this
system,
and
secondly,
we
want
to
share
how
we
adopted
a
system
and
help
the
other
engineering
teams
to
deploy
and
migrate
their
services
to
kubernetes.
A
Before
we
get
into
the
CD
system,
I
want
to
I
want
to
give
you
some
context
of
how
we
adopt
kubernetes
at
the
Pinterest
Bethy
in
2016,
our
first
attacker
project
started
in
2017.
We
evaluated
the
multiple
container
of
trician
solutions
and
decided
to
go
down
the
path
of
kubernetes
because
of
its
flexibility
and
the
extensive
community
support.
Thank
you
to
everyone
here
and
in
2018
we
started
to
build
our
kubernetes
clusters
and
run
some
of
our
production
services
on
those
clusters.
A
A
Talking
about
Pintrest
ard,
we
implemented
our
own
Pintrest
erd
to
model
the
unique
who
were
close.
We
use,
don't
see
our
DS
bundle,
multiple
native
kubernetes
resources
together,
so
they
work
like
one
single
workload.
Now
we
have
60
RDS
and
they
greatly
simplified
the
configuration
needed,
as
you
can
see
in
this
example
using
our
CRD.
Our
developer
writes
a
config
file
with
25
lines
and
then
our
controller
will
translate
those
twenty-five
lines
into
the
native
donated
resource
definitions.
A
A
So
after
talking
about
kubernetes,
let's
switch
to
some
context
of
deployment
at
Pinterest
we
have
an
in-house
deployment
system
tool
called
teletran.
Teletran
has
been
up
and
running
since
2016.
It
has
been
for
years
today.
The
majority
of
our
internal
services,
you
use
trial,
run
to
deploy
their
code
to
virtual
machines
on
AWS.
A
A
B
A
A
Kubernetes
itself
has
a
host
of
concepts
and
objects,
although
those
are
interesting
to
us,
because
we
are
the
people
who,
like
and
support,
kubernetes
and
want
to
be
part
of
the
community,
but
we
cannot
expect
our
developers
who
work
on
different
products
or
projects
to
have
the
same
interest
as
us.
We
can
now
expect
them
to
be
very
comfortable
with
getting
into
all
the
details
of,
on
the
other
hand,
in
order
to
deploy
to
kubernetes
developer
need
to
prepare
many
things.
A
A
The
second,
a
challenger
will
find
is
our
current
ruling
does
not
support
end
to
end
experience
developers.
They
often
need
to
jump
among
multiple
systems
such
as
build
or
testing
or
at
the
factory
in
order
to
deploy
their
code,
especially
if
something
is
broken
in
the
whole
process.
It's
painful
for
them
to
locate
the
errors
and
figure
out
what's
happening.
A
Lastly,
we
do
have
a
few
Pinterest
specific
problem.
For
example,
we
immerse
the
holiday
in
our
internal
infrastructure
and
tools.
So
how
can
we
integrate
those
existing
tools
into
the
deployment
process
like
our
open
TSD
be
based
the
metric
system,
our
projects,
the
ownership
system
and
the
security
tools?
How
can
we
integrate
them
all
together?
A
A
A
Number
two:
is:
we
want
the
tooling
to
cover
the
process,
all
the
way
from
code
commit
to
code,
the
deployed,
and
also
we
want
to
have
good
availability.
Telling
you,
where
is
your
commit,
has
a
build
started
and
is
the
artifacts
ready
for
deploy,
and
if
there's
something
goes
wrong.
Where
can
you
find
the
logs
and
errors?
What
should
you
do
next
house?
It
should
be
covered
by
the
end-to-end
support.
A
A
A
A
A
A
Let's
look
at
the
workloads
first
workload
is
something
you
create
for
your
service
and
later
you
can
use
this
workload
to
deploy
your
changes
to
any
particular
environment.
The
features
around
workload.
Our
first
is
easy
configuration
when
your
Krita
workload.
You
only
need
to
provide
your
code,
the
repository
and
kubernetes
config
file,
and
that's
it.
We
support
different
workload
types
because
hermits
focuses
on
kubernetes
first,
so
we
support
all
the
types
defined
by
pin
trees.
There
are
these.
It
also
supports
the
workload
from
other
computer
platforms
such
as
data
streaming
or
aleksey
help
or
clothes.
A
A
A
A
A
The
third
feature
group
I
will
introduce
is
CI
integration,
bridging
CA
and
the
city
is
important
to
us
because
we
want
to
visualize
the
process
from
code
commit
to
call
deployed.
Ok,
we'll
give
you
the
logs
for
debugging
and
Aramis,
have
the
beautiful
and
to
support
our
small
service,
ripples
or
huge
Mallory
pools.
A
This
this
is
a
text
like
in
wall
in
the
whole
mass
deployment
process
on
the
left.
Aside
is
our
South
code
and
CI.
When
there
is
a
commit
our
PR,
there
will
be
something
happen
to
start
the
beautify
plans
Argentines
and
then
the
docker
images
and
the
kubernetes
manifests
will
be
generated
on
the
right
side.
Is
her
mass
and
the
integrated
the
other
systems
integrated
ways
it
from
sui,
and
the
api
are
growing
in
separate
containers
from
an
api
integrated
with
our
ownership
system.
A
A
B
Hi
everyone,
my
name-
is
Toby
and
I'm.
Also
software
engineer
at
Pinterest,
so
you
could
have
spoken
a
bit
about
her
mad
about
the
motivation
as
the
challenges
were
trying
to
solve.
She
also
showed
the
tech
stack,
so
I'm
going
to
switch
gears
a
bit
and
I'm
going
to
talk
a
bit
about
the
customer
adoption
story.
B
So
anyone
here
who's
ever
built
a
product
knows
that
it's
one
thing
to
build
the
product,
whether
that's
software
or
whether
that's
its
he
sure,
whether
it's
lemonade
you're,
making
it's
a
completely
different
thing
to
actually
get
the
users
right
in
the
context
of
what
we're
building
we're
building
a
new
CD
platform.
This
challenge
is
a
bit
is
a
bit
intensified
for
a
couple
reasons.
One
of
them
is
first
of
all
the
scale.
So
Pinterest
is
a
huge
company.
We
have
hundreds
100
engineers,
thousands
of
services
going
on
and
people
have
been
doing
deployments
before.
B
Kubernetes
people
are
going
to
do
deployments
in
the
future
with
or
without
hermas.
So
that
means
people
have
existing
habits
about
deployments.
They
have
things
that
they
expect
from
the
deployment
system.
So
this
is.
This
is
a
curve.
We
obviously
need
to
overcome
when
we're
trying
to
incentivize
them
to
use
our
tool.
Another
thing
is
kubernetes
itself.
Like
you,
Chris
mentioned
it's
a
new
technology.
B
So
when
you're
trying
to
get
people
to
actually
use
your
products.
This
is
the
fundamental
question.
They're
asking
people,
don't
care
so
much
about
whether
you
wrote
the
most
poetic
Python.
You
know
the
nicest
JavaScript
people
just
want
to
know.
Fundamentally
what
can
your
product
do
for
you
do
for
them
and
they
want
to
know
why
they
should
care.
So
in
order
to
try
and
answer
this
question,
I'm
going
to
show
two
quick
demos,
two
videos
of
demos
where
I'm
going
to
try
and
illustrate
the
basically
the
value
add
for
Hermits.
B
So
the
first
thing
I'm
going
to
do
is
I'm
going
to
demo
a
cron
job.
A
bit
of
context
on
this,
so
I
wrote
the
squadron
myself
it's
a
fairly
silly
contra
of
what
it
does
is
it
pulls
flights
API
every
minute
and
essentially
it
finds
the
cheapest
flights
from
SFO
to
Madrid.
The
context
is
that
I
want
to
go
to
see
your
classical
next
year
if
there
any
soccer
fans
in
the
in
the
house.
Thank
you.
Thank
you,
okay,
awesome!
B
B
So,
in
a
couple
seconds,
it's
going
to
it's
going
to
run
so
because
this
cron
job
is
very
small
it
it's
like
it
basically
runs
really
quickly.
But
very
interesting
thing
to
note
is
that
you
can
see
the
actual
logs
from
the
from
the
cron
job.
So
here
you
can
see
the
cheapest
flight
is
170
175
years,
which
is
really
really
cheap.
Another
thing
here
you
can
see
the
history.
B
B
So
the
couple
things
I
want
to
call
out
from
that
demo
first
thing
is
we're
providing
first-class
support
for
cron
jobs.
The
reason
why
this
is
important
is
that
historically
they've
been
a
bunch
of
different
ways
to
create
and
run
cron
jobs
at
Pinterest,
but
we
haven't
had
a
very
single
simplified
recommended
way
to
do
that,
but
here
we're
basically
with
two
cron
jobs.
As
a
first-class
citizen
from
the
ground
up,
we
thought
about
okay.
B
What
is
the
experience
we
want
to
give
users
for
cron
jobs,
and
that
is
reflected
in
the
features
that
we
show.
Second
thing
is
that
we
have
easy
integrations
with
our
existing
systems,
we're
not
asking
our
users
to
come
up
with
new
build
systems
or
have
a
different
way
where
they
keep
the
artifacts,
we're
using
what
I
already
exists
and
we're
hiding
the
complexity
from
from
them,
and
the
last
thing
is
debug
ability.
B
We
think
it's
really
important
that
people
know
exactly
what
is
going
on
with
their
workloads,
so
provide
people
the
opportunity
for
people
to
see
container
logs
the
pod
status,
whether
it's
scheduled,
whether
it's
running
whether
it's
failed,
and
you
can
also
see
the
workload
metrics.
So
these
could
be
things
like
CPU
or
memory
usage.
So
with
that
done,
I'm
going
to
show
another
video
of
a
demo.
This
one
is
going
to
be
for
a
service,
so
this
actually
relates
to
the
feature
that
you
Chris
mentioned
before.
B
So
in
her
meds,
we
built
the
functionality
for
the
first
time
where
people
can
deploy
peers
by
PR
I
mean
any
on
land
revision.
You
have
any
feature
branch
or
working
on
locally.
The
reason
why
we
did
this
is
because
we
found
in
talking
to
our
engineers
that
this
is
something
that
they
wanted
to
use
for
collaboration.
B
B
Okay,
so
this
video
is
going
to
start
from
the
terminal,
just
assume
you're
an
engineer.
You've
already
made
the
feature
you
create
the
revision
and
they're
going
to
go
to
the
deploy
page.
You
pick
the
diff,
you
deploy
the
diff
and
then
you
can
pretty
instantly
see
the
deployed
version
with
the
easily
shareable
URL.
So
if
you
just
watch
the
video
create
the
diff.
B
Okay-
and
this
is
the
this-
is
the
deploy?
U
UI,
you
can
pick
the
diff.
You
get
instant
validation
of
your
artifacts
on
the
page.
This
is
our
deploy
page,
you
can
see
instantly
you
get
the
desired,
pods
increases
to
two
and
then
essentially
you
can
click
on
the
service
URL.
So
this
is
the
city
thing
I
built
this
morning,
but
you
can
see
how,
in
a
in
a
different
context,
if
you're
building
the
feature
you'll
be
really
valuable
to
test
new
UI
changes
or
API
changes
really
quickly
and
to
collaborate
with
your
fellow
engineers.
B
B
We
make
it
very
clear
for
people
to
know
what's
going
on
with
the
pause,
the
containers
at
every
at
every
point
in
time
and
also
operational
like
we
need
to
make
sure
that
you
can
see
the
metrics,
you
can
see
the
CPU,
you
can
see
memory
right
from
a
single
non
quartered
UI.
We
think
that's
very
important.
B
B
One
of
the
first
things
we
did
was
that
we
made
sure
we
had
reduce
scope,
so
we
actually
have
the
support
from
our
platform
team
to
run
to
deploy
six
or
seven
different
vocal
types,
but
we
made
sure
that
we
started
with
just
one
first
in
order
to
try
that
out
get
some
feedback
see
what
we
can
learn
with
it
with
people
related
to
that
we
partner
with
early
adopters.
So
these
are
people
who
basically
were
willing
and
able
to
try
things
before
they
were
ready.
B
This
was
very
critical
in
our
in
our
success
and
also
it's
important
to
make
sure
you
work
with
people
who
have
different
use
cases
so
you're
not
solving
for
the
same
case
multiple
times.
So
when
you,
when
you're
working
with
early
adopters,
you're
gonna,
get
feedback,
it's
also
very
important
to
take
that
feedback
iterate
on
your
your
features
and
your
what
you're,
delivering
and
then
knowledge
sharing.
So
we
made
sure
we
went
through
a
process
of
sending
out
bi-weekly
updates
to
our
to
our
customers.
B
We
made
sure
we
did
demos,
demos
like
this,
but
smaller
ones
as
well
in
the
company.
We
did
brown
bags
as
well
and
we
have
ROM
books
for
people
to
humble
themselves
onto
our
systems
and
the
future
we're
looking
to
provide
self-service
migration
tools,
so
this
would
basically
again
ease
the
ease,
the
heavy
lifting
of
people
who
are
trying
to
use
our
our
service
in
the
future.
B
Cool
so
on
this
page
again
we
have
another
cute
picture
of
a
dog,
but
this
is
basically
what
we
just
want
to
share
that
we
had
a
success
story.
So
a
couple
months
ago
there
was
an
internal
Pinterest
hackathon
and
the
team
which
owns
the
home
feed
part
of
Pinterest
they
reached
out
to
us
because
they
heard
what
we
were
building
and
they
wanted
to
collaborate.
They
were
really
interested
in
the
fact
that
you
could
deploy
PRS
and
share
that
with
other
engineers,
so
they
reached
out
to
us.
We
work
with
them.
B
We
helped
them
with
the
process
of
writing
GMOs.
We
helped
them
with
figuring
out
the
traffic
stuff
as
well,
and
this
was
actually
before
her
meds
was
ready.
So
it
was
really
really
useful
feedback
for
us
as
we
were
building
and
by
working
with
them.
We
were
able
to
build
this
and
they
ended
up
winning
winning
an
award,
and
you
know,
as
the
people
as
the
people
who've
helped
provide
that
it
was
super
useful
to
feel
that
we're
enabling
people
success
so
yeah.
So
that's
a
cute
dog
because
that's
we
were
happy
after
their
success.
A
We
have
been
going
through
four
stages
in
the
past
to
the
new
CD
system.
The
first
one
is
design,
we
started
from
user
story
and
UI
mock-ups.
The
important
thing
here
is
to
making
fur
listen
to
your
engineers,
having
the
feedback
sessions
with
the
other
engineering
teams,
so
make
sure
what
you
tease
and
match
their
expectation.
An
after
guide.
We
move
down
to
the
MVP
phase.
Here
we
build
the
minimal
set
of
features
and
also
it's
important
to
find
the
opportunities
to
collaborate
with
the
other
teams.
A
After
that,
once
we
have
the
MVP
ready,
we
started.
He
used
her
meds
internally
in
our
team
first,
this
is
an
increasing
stage
and
we
spend
much
more
time
here.
For
example,
we
may
found
something
we
didn't
think
through
in
the
original
design
or
or
something
we
thought
could
be
very
useful,
but
then
we
realize
actually
we
don't
use
it
that
often,
and
also
we
need
to
find
early
adopters
who
can
support
us
and
we
help
them
to
migrate
some
of
their
services
using
our
system.
A
A
Here
some
of
the
important
things
we
learned
number
one
is
trade
were
close
as
a
first-class
objects.
We
realize
what
people
need
for
deploying
different
types
of
work.
Workload
can
vary,
so
it's
a
important
in
your
design
to
have
the
flexibility,
so
you
can
customize
the
design
for
different
workloads.
A
Number
two
is
do
not
force
your
user.
To
know
about
kubernetes
deployment
system
is
a
traction
to
the
underlying
compute
platform
for
the
most
of
our
engineers.
They
just
want
to
see
their
deployment
is
running
and
if
something
fails,
they
can
debug
it
so
abstract
away.
The
complexity
is
a
priority.
A
A
So
that's
what
we
are
doing
for
CD
and
Pinterest
here.
I
want
to
take
a
moment
to
say
thanks
to
the
people
who
work
on
our
CD
and
hermas
project.
Besides
two
of
us,
we
also
have
other
engineers
in
our
engineering,
product
attitude,
productivity
team
and
our
title
deed
and
the
manager
they
are
super
supportive
and
if
you,
if
this
sounds
like
something
interesting
to
you
and
you
want
to
chat
or
some
general
question
about
the
CD
feel
free
to
reach
out
to
us,
you
can
send
us
emails
or
messages
on
LinkedIn.
Thank
you.