►
From YouTube: Keynote: Developing Kubernetes Services at Airbnb Scale - Melanie Cebula, Software Engineer, Airbnb
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Keynote: Developing Kubernetes Services at Airbnb Scale - Melanie Cebula, Software Engineer, Airbnb
You've already made the plunge to move to Kubernetes, and you feel pretty good about that. But why does it feel like it requires expert-level Kubernetes knowledge for engineers to get anything done? This talk will identify key problems that make out-of-the-box Kubernetes less friendly to developers, and strategies for addressing them, based on Airbnb’s experience empowering one thousand engineers to develop hundreds of Kubernetes services at scale. This talk will focus primarily on four problem areas:
1. Configuration: Generated and templated configuration, extracting shareable components and containers
2. CI/CD: Containerizing CI/CD, multi-stage builds, deploy strategies
3. Service lifecycle: Primary and sidecar containers, Advanced pod readiness, warmup, graceful termination, etc
4. Tooling: Creating and Distributing a kubectl wrapper
To learn more: https://sched.co/GsxY
A
Hopefully
you
do.
Everybody
is
an
online
marketplace
for
sharing
homes
and
experiences,
so
we
operate
in
81
thousand
cities
over
190
countries
and
we
have
over
five
million
homes.
On
our
platform
recently
we
launched
Airbnb
collections
which
helps
business
family
or
any
other
kind
of
traveler
to
find
the
right
home
for
them
our
engineers,
our
engineering
team,
is
constantly
creating
new
products
and
businesses
and
they
need
to
be
able
to
spin
up
new
services
quickly
and
reliably.
A
So
that's
where
I
come
in
I
work
as
a
software
engineer
on
the
service
orchestration
team,
every
beam
and
our
goal
is
to
empower
our
engineers
to
create,
maintain
and
operate
their
own
services
so
about
a
year
ago,
I
helped
migrate,
our
first
critical
production
service
to
kubernetes
and
today,
out
of
hundreds
of
service,
almost
40%
of
them
or
in
kubernetes
today.
So
it's
really
exciting.
A
You
can
tell
this
photo
which,
before
I
started,
I
have
a
few
gray
hairs
from
that
process,
but
I'm
gonna
take
us
back
to
a
brief
history
of
Airbnb,
so
we've
been
transitioning
from
a
monolithic
to
a
micro
services
architecture,
and
so
why
micro
services,
we
started
to
hit
significant
issues,
scaling
contributions
to
our
mouths
in
2015,
and
our
engineering
team
today
is
over
a
thousand
engineers.
So,
like
that's
kind
of
scary,
and
what
we
needed
to
do
is
we
needed
to
scale
continuous
delivery
horizontally.
A
So
each
service
gets
its
own
chase
delivery
cycle
and
you
can
scale
development
for
engineers
by
adding
new
services,
we've
managed
to
scale
development
exponentially
at
our
size,
with
a
total
of
around
20,000
two
plays
per
week.
This
means
because
you've
both
introduced
exponentially
more
services
and
exponentially
more
environments
for
each
service.
So
we
actually
have
a
suite
of
pre
production
environments.
Think
development,
stay
load-testing
canary
before
changes
go
out
to
production
and
by
the
way
each
of
these
dips
in
this
graph
are
for
a
holiday
break
right
around
now.
A
A
Then
we
started
to
use
a
chef
to
configure
our
repos
and
it
was
basically
a
monolithic
configuration
repo
changes
to
one
cookbook
or
recipe
could
inadvertently
take
down
other
services
or
environments,
and
there
was
no
clear
rollback
story
for
our
changes
and
we
simply
moved
to
kubernetes,
which
automates
both
the
configuration
and
orchestration
of
continuous
applications.
So
what
does
that
get
our
team,
so
communities
itself?
The
configuration
is
declarative,
that's
more
resilient,
there's
less
maintenance
burden.
A
Think
about
you
know
a
certain
instance
being
cutting
debated
Hardware
before
engineers
had
to
go
and
manually
launch
a
new
instance
with
the
clarity
of
configuration.
You
don't
have
to
do
that.
Big
one
for
us
with
efficient
scheduling
is
just
helping
us
save
money,
and
you
know
with
our
history
of
having
a
lot
of
configuration.
How
do
we
actually
integrate
with
kubernetes?
How
do
we
migrate
all
of
our
sources
in
infrastructure
in
a
way
that
is
compatible
with
kubernetes
and
having
an
extensible?
A
Api
really
helps
us
with
that,
and
then
all
of
these
features
of
dr
portability,
immutability
reproducibility.
So
before
our
infrastructure
was
meetable
that
mean
you
could
SSH
into
an
instance
and
change
something,
and
it
could
be
like
that
forever.
It's
not
resulted
in
some
pretty
interesting
incidents
and
reproducibility
just
makes
it
very
easiest
for
us
to
roll
back
to
a
previous
version
and
yeah
Moe
is
human
friendly
and
about
friendly
compared
to
our
previous
infrastructure.
So
the
standard
format
makes
it
easy
for
us
to
work
with
it.
A
One
of
the
challenges
with
kubernetes,
so
the
configuration
and
tooling
are
complex.
There's
open
issues
integrating
with
your
current
infrastructure
can
be
challenging,
but
what
I
want
to
stress
today
is
that
these
are
all
solvable
problems
and
that's
what
I'm
gonna
talk
about
is
how
we
solve
each
one
of
these
problems.
Why
generate
kubernetes
files?
A
So
let's
talk
about
the
problem
of
having
a
bunch
of
bare-metal
today's
files
that
you
may
only
apply
to
the
cluster,
you
handcuff
these
config
files
and
you
manually
apply
them
they're
artisanal
they're
small
batch.
Until
you
have
a
lot
of
environments,
suddenly
you
actually
kind
of
have
this
n
by
n
grid
of
resources,
environments,
that's
a
lot
of
boilerplate
and
it's
repetitive
by
environment.
So
this
is
actually
I
couldn't
fit
all
the
environments
we
have
on
this
one
slide.
A
So
just
imagine
this
even
bigger,
so
our
requirements
for
reducing
boilerplate
was
we
preferred
templating
over
file
inheritance,
and
that
was
because
we
learned
the
hard
way
that
it
was
difficult
for
our
engineers
to
visualize
complex
hierarchies
with
files,
so
we
preferred
templating.
For
that
reason,
we
wanted
to
have
engineers
to
opt
in
to
higher
level
abstractions
over
companies
files,
but
always
have
the
option
to
use
rock
renée's
files,
if
that's
what
they
prefer.
And
finally,
we
want
to
make
it
easy
to
migrate
hundreds
of
legacy
services
and
we
trained
that
that
was
an
engineer's.
A
It
was
kind
of
intimidating
for
us,
so
the
solution
we
came
up
with
was
a
bit
of
a
custom
solution
that
mapped
kubernetes
concepts
to
legacy
ones.
We
optionally
supported,
go
templating
and
our
chilled
handled
both
current
files
and
the
custom
files.
I
just
want
to
point
out
that
this
ecosystem
is
super
rapidly
evolving.
So
there's
a
bunch
of
things
here
how
customized
capitan,
whatever
apartments
you
have.
A
If
you
want
to
combine
with
package
management,
if
you
want
to
use
file,
inheritance
or
templating,
and
then
even
yesterday,
Julia's
talked
about
sky
config,
that's
even
another
option.
If
you
want
to
configure
this
in
your
code
directly,
so
this
I'm
really
excited
about
this
space
and
excite
us
with
command
of
it.
So
here
is
our
internal
distraction
on
the
left.
These
are
all
concepts
that
Airbnb
engineers
are
familiar
with
and
we
can
generate
them
into
valid
kubernetes
files,
so
I
call
it
cube
Jen
and
the
important
output
here
is.
A
This
creates
files,
the
generated
commands
files
have
standardized
namespaces
based
on
environments,
so
today's
the
sample
service
is
bunk
and
our
bunk
service
has
a
few
environments.
Also
I
couldn't
put
them
all
on
the
screen,
but
like
the
important
ones,
Canaria,
development,
production
and
staging,
and
if
you
expand
each
of
these
sub
directories
of
generated
files,
those
are
all
those
Cabernets
files.
So
we
have
several
config
Maps
services
role,
binding
files,
etc,
etc,
and
it's
gonna
be
really
powerful.
Later
on,
I
talked
about
our
tooling.
Why?
Having
Sarah
dyes
namespaces
is
important.
A
One
other
thing
I
wanted
to
call
out
is:
we
noticed
a
problem
where
a
lot
of
services
had
we're
reimplemented
the
same
exact
code
over
and
over
again
so
nginx
tracing.
That's
the
logging
stuff
like
this
people
are
just
mean
money
or
whatever
again,
so
we
actually
account
with
a
component
framework
that
will
merge
gamal
into
your
project
it.
Just
all
these
common
patterns
are
abstract
into
a
component.
A
A
So
I'm
gonna,
let
that
sink
in
and
I'm
gonna
break
it
down.
So
everything
about
a
service
is
in
one
place
and
get
what
does
that
mean
lives
in
one
place?
Here's
our
bomb
service,
so
everything
in
our
case
leaves
in
underscore
info
alongside
project
code.
We
don't
have
a
separate
conference,
configuration
and
monolith.
We
don't
have
a
configuration
mono
repo,
all
of
the
configuration
for
your
project
lives
right
next
to
your
project
code.
A
You
can
edit
code
configuration
in
one
pull
request.
It's
easy
for
us
to
add
new
configuration
to
the
system,
we're
just
supporting
a
new
way
of
storing
stuff
and
get
which
most
developers
are
familiar
with
and
once
you've
been
talked
about.
A
bit
is
how
we
can
actually
have
really
good
validation
as
part
of
integration
and
deploys.
A
So
we
saw
all
of
our
configuration
here
and
it's
a
lot
and
we're
adding
new
stuff
every
single
month.
So
this
HelloWorld
service
bunk
was
created
in
one
command.
It
took
me
about
30
seconds:
it's
a
collection,
so
the
way
we
generate
services
is
a
collection
of
config
generators
like
documentation,
generators,
the
I
generator
turbine,
a
generator
and
a
collection
of
framework,
specific
generators.
So
we
have
support
for
rails,
drop,
wizard
and
a
few
other
frameworks.
A
So
by
generating
services
for
engineers,
we
make
the
best
practices
a
good
default.
So
I
talked
about
how
over
fort
are
almost
40
percent
of
our
production.
Ready
services
are
in
queue
brittany's.
How
did
we
get
there
that
fast,
so
by
making
it
really
easy
for
services
for
service
owners
to
create
a
new
service
in
one
command,
like
that's
great,
like
instead
of
spending
two
weeks
a
hand
crafting
all
this
configuration?
That's
just
saying
hello,
world,
I,
literally
type
one
command
and,
and
so
Soni
Melanie.
A
The
smart
infrastructure
engineer
is
like:
how
do
we
get
people
to
use
documentation?
How
do
I
get
people
to
use
auto-scaling?
How
do
we
enforce
that
people
test
their
changes
in
a
deployment
pipeline
or
use
our
latest
continuous
integration,
so
I
put
it
in
the
generator
and,
like
all
of
our
services,
are
using
this
stuff?
It's
so
great
I
love
it,
so
you
can
run
generators
individually
or
as
a
group,
so
they
run
as
a
group
for
a
new
service.
A
But
what
we
really
wanted
to
get
here
as
well
is
the
ability
to
constantly
update
and
refactor
services.
So
we
use
something
amused.
An
open-source
framework
called
Thor
and
supports
review,
update
and
commit
it's
kind
of
like
handling,
git,
merge
conflicts,
and
what
I
like
about
that
is
we?
So
we
actually
one
other
thing
I
want
to
talk
about.
Is
we
have
hundreds
of
services
and
communities
right
and
all
of
the
configuration
is
in
project
code
now?
So
if
you
decide
that
you
want
to
completely
change
how
you
do
continuous
integration,
you
actually
like.
A
How
do
you
do
that,
so
we
actually
have
a
B
factor
job.
It's
like
Renee's,
cron
job,
that
if
you
can
model
your,
so
an
occupation
is
verdant.
So
if
you
want
a
model
going
from
one
version
to
another,
you
can
actually
pick
up
a
job
that
automatically
migrates
all
services
to
your
new
thing,
and
so
we
actually
use
this
generator
for
new
sources
and
for
refactoring
existing
services.
A
Ok,
so
now
that
our
configuration
is
streamlined,
let's
switch
gears
to
another
problem
which
is
interacting
with
cube
C
tail
by
the
way
I
call
it
cubes
detail,
I,
don't
know
what
the
recipe
all
do,
but
yeah
so
yeah.
Here's,
my
I
love
this
graph
by
the
way
I
spent
a
lot
of
time
on
it.
So
we
have
our
artisinal
graph
right.
Well,
actually,
now
it's
more
mass
produced
because
we're
generating
it
for
people,
but
we
have
lots
of
boilerplate,
it's
repetitive
by
environment
and
you
saw
a
pie.
A
It's
also
a
little
verbose
and
repetitive
by
namespace,
so
keep
some
commands
actually
like
they're,
a
little
bit
verbose
and
even
if
you're,
like
a
master
at
thumb,
it
so
can
be
kind
of
complicated
to
construct
the
commands
or
just
even
move
quickly
with
it.
So
we
introduced
it.
We
introduced
duct
tape.
So
this
is
the
K
tool.
It's
our
all-purpose
climb,
a
per
it's
really
easy
to
build
one
yourself:
it's
just
a
scrip,
so
our
K
tool.
Basically,
we
have
a
bunk
service,
we
seed
into
our
bunk
service
and
we
type
a
command.
A
The
way
it
works.
Is
it
takes
environment
variables
for
arguments
s'okay
status
at
nichols
staging
and
then
it
prints
the
actual
command?
It's
where
do
you
use?
So
this
is
just
supposed
to
make
it
easier
developers.
We're
not
trying
to
hide.
What's
actually
happening
so
case
that
is
for
us
will
get
keeps
Adelle
Pollard's
for
your
namespace
I.
A
Remember
when
I
said,
Sarah
dicing
on
namespaces
is
super
awesome,
so
since
we're
in
the
bunk
project
directory
and
we
pass
in
the
end,
equal
staging
environment,
variable
K
tool
knows
that
you're
operating
in
the
bunk
staging
namespace.
So
we
just
save
people
a
few
Plex,
especially
like
I.
Don't
know
why
this
is
but
a
lot
of
people
have
really
long
service
names,
so
we
save
a
lot
of
those
people
time
to
so.
We
also
automate
a
few
different
commands,
so
K
generate
transforms
our
internal
abstraction
to
criminate
vials
k.
A
Build,
will
do
a
project,
build
darker,
build
and
darker
push
for
you
and
then
kata
boy,
kata,
playable,
crannies,
space,
apply
files
and
then
check
the
deployment
status.
So
we
wanted
to
simplify
a
developer's
iteration
cycle
and
for
those
of
you
who
are
lazy,
we
actually
chained
all
these
commands
to.
So
a
lot
of
our
developers
are
just
familiar
with
typing
KO.
A
So
one
other
thing:
I
wanted
to
get
developers.
It
was
basically
just
automating
debunking
flows.
So
do
we
notice
that
debugging
community
services
is
actually
a
bit
of
the
like
thought,
shift
away
from
debugging
other
services,
there's
a
bit
of
a
learning
curve
and
like
how
do
we
make
it
easier
for
people
to
debug?
A
But
if
you
notice
a
problem
with
a
particular
pod
or
problem
with
a
particular
container.
We
support
that
too.
So
remember
these
environment
variables,
so
we
have
a
pod
environment,
variable
and
a
container
environment
variable
and
then
finally,
we
went
even
further
with
our
que
diagnosed
command
so
basically
case
status
gets
our
pods
and
if
each
unready
container
in
our
in,
like
our
pod
for
each
of
their
states,
we
get
the
state
and
we
get
their
logs
and
we
output
them.
A
And
then
we
also
so
maybe
it's
a
poem
with
one
of
your
unready
containers
or
maybe
there's
a
problem
with
the
cluster,
and
it
would
be
nice
if
a
service
owner
could
figure
that
out.
So
we
also
do
keep
tailgate
events.
We
get
the
pod
and
we
filter
out
any
normal
events,
so
we're
trying
to
give
developers
the
abnormal
events
so
Kay
diagnose,
will
tell
you
if
there's
something
wrong
with
your
pod.
Any
of
the
containers
in
your
pod
any
problems
with
the
cluster.
A
So
even
if
you're,
like
wow
I'm,
a
QC
Tail
wizard
I
mean
that's
pretty
cool
like
that
stuff.
That
I
know
a
lot.
People
do
manually
all
the
time.
So
it's
really
great
to
just
automate
that
for
yourself
so
create
a
wrapper
for
keep
snail,
commands
and
automate
those
calming
kate's
workloads,
whether
it's
building
deploying
debugging
SSH,
a
indicator
this
etc.
A
Now
that
we've
sold
all
of
our
code
in
one
place
and
built
the
k2,
let's
see
what
we
can
provide
as
part
of
CI,
so
all
of
our
projects
get
validation,
they
get
a
build
and
they
get
build
docks.
So
our
validation
actually
checks
for
malformed
yamo
checks
for
security
problems,
checks
for
required,
covariation
like
if
we
want
to
require
project
ownership,
we
can
do
that
so
the
way
we
do
it
is
each
step
in
our
CICE
jobs.
It's
just
run
steps
in
a
build
soccer
file
and
it
runs
the
cake
commands.
A
What
I
really
like
about
this
is
it's
super
transparent
to
developers
when
they're
key
whenever
the
chase
ups
fail,
they're
familiar
with
those
because
they
use
them
in
development.
They
also
use
the
kay
tool
when
there's
like
a
production
incident
and
they're
trying
to
figure
out
what's
happening
to
my
production,
pod,
so
they're
using
the
same
cake
fans.
The
same
exact
interface
is
using
every
possible
environment,
they
can
work
with
and
that's
really
important
when
developers
are
like
under
fire
with
an
incident.
A
So,
with
all
recording
finish
in
certain
one
place,
it
becomes
possible
to
deploy
them
together
as
part
of
the
same
to
play
process
so
hopeless.
The
developer,
you're
kind
of
familiar
with
this
flow
develop,
merge
and
deploy
so
I'm
going
to
talk
about
both
Cabernets
files
and
custom
infrastructure
files.
How
do
we
apply
them
to
the
cluster?
So
I
said
keep
sale
apply,
but
I
was
actually
lying
to
you.
We
don't
actually
just
do
that.
So
our
deploy
process
is,
we
first
try
to
conceal,
apply
old
files.
A
A
Finally,
we
try
to
return
an
atomic
success
or
failure
state
by
sleeping
and
checking
the
status,
so
services
in
crash
loop
will
actually
eventually
timeout,
which
is
a
helpful
indication
to
our
users,
who
aren't
used
to
services
and
crash.
We
back
off
that
there's
something
wrong
with
their
deploy.
A
So
I
mentioned,
how
do
we
always
restart
pods
on
deploy?
So
why
do
this?
We
found
that
there
were
some
issues
where
you
applied
a
configuration
change
and
it
wasn't
really
applied.
It
wasn't
really
applied
to
pods
until
later.
So,
like
the
neighbors
like
enough
time,
it's
like
training,
a
dog.
You
want
there
to
be
like
a
really
short
iteration
between
like
oh,
no
something
bad
happened
or
something
good
happened.
A
So
for
us
we
always
apply
our
configuration
and
restart
our
pods
to
pick
it
up
immediately.
So
we
have
that
immediate
feedback
and
the
way
we
do
that,
just
a
quick
trick.
We
had
a
date
label
to
the
pods
back
and
then
custom
configuration
just
gonna
click.
This
really
fast,
so
we
have
a
bunch
of
customization
the
way
we
do
it
custom,
a
source,
custom
controller
that
calls
a
web
hook
and
then
the
actual
web
hook
itself
that
updates
a
custom
resource.
A
So
you
can
actually
hear
more
about
custom
controllers
later
today,
with
my
co-workers,
talk
custom
controllers
for
the
rescue
with
Steven
and
Dobby
in
like
an
hour
or
two,
and
this
is
really
exciting
so
like
in
this
case,
our
infrastructure
engineer
is
Justin
to
supply
a
web
hook
to
apply
their
kind
of
file.
So
if
a
service
needs
a
new
ia,
I
am
roll.
A
So
coding
configuration
should
be
deployed
with
the
same
process
and
use
custom
resources
and
custom
controllers
to
integrate
kubernetes
with
your
infra.
So
I
want
to
leave
you
with
the
10
takeaways
I
was
going
through
if
I
click
through
to
fast,
so
reduce
craze
boilerplate.
You
want
to
distract
Cabernets
files
from
from
your
from
your
developers
staring
eyes
on
environments
and
namespaces.
A
We
saw
how
useful
that
was
with
our
tooling,
and
everything
about
our
service
should
be
in
one
place
and
get
both
the
code
and
the
configuration
we
want
to
make
the
best
practices
the
default
by
generating
configuration,
and
one
thing
I
want
to
add
on
here
is
the
support
for
update,
will
be
factoring,
create
a
wrapper
for
cube,
sell
commands
and
then
automate
common
kate's
flows
with
that
wrapper.
Ci
CD
should
run
the
same
command
and
choose
run
locally
in
a
container
dominant
configuration
as
book
as
part
of
both
CI
and
CD
coding.
A
Cool,
so
thank
you
for
that.
I
have
so
much
more
I
would
love
to
talk
about
this
but
I'm
out
of
time.
So
you
can
learn
more
at
our
medium
page,
we're
hiring
in
so
many
places.
I'm
reppin,
my
everybody
Seattle
shirt,
so
we're
having
a
lot
in
Seattle.
We
have
offices
in
San,
Francisco,
Beijing
and
all
over
the
world.
So
please
apply
if
you're
interested,
and
you
can
reach
me
directly
at
not
only
Sabula
on
twitter.