►
From YouTube: Sidecars at Netflix: From Basic Injection to First Class Ci... Rodrigo Campos Catelin & Manas Alekar
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Sidecars at Netflix: From Basic Injection to First Class Citizens - Rodrigo Campos Catelin, Kinvolk & Manas Alekar, Netflix
The concept of sidecar containers has been around since the early days of Kubernetes. Over the years the sidecar pattern has become more common in applications, gained popularity and the uses cases are getting more diverse. The current Kubernetes primitives handled that well, but they are starting to fall short for several use cases and force weird work-arounds in the applications. During this talk we will share how Netflix is using sidecar containers on Kubernetes, from their early design of process injection into docker containers, to a cutting edge fork of k8s where sidecar containers have special status and extra features over other containers in a pod. We would like to share our experience with sidecar containers and encourage others to do the same so we all benefit and learn. We will also share our efforts (past and ongoing) in collaboration with Kinvolk to progress a KEP to make sidecars a first-class concept in Kubernetes, and the status and implications of that process.
A
Hi
welcome
to
the
sidecars
and
netflix
presentation,
I'm
rodrigo,
I'm
a
software
engineer
at
kinfold
a
little
bit
about
me.
I
studied
computer
science
in
argentina
and
I'm
a
maintainer
of
middleweight
and
core
coordinates
reviewer
and,
as
I
said,
I
work
at
kinfolk.
We
are
focused
on
developing
open
source
software.
You
may
know
some
of
our
projects
like
flat
car
container
linux.
A
There
is
a
container
optimized,
rs
or
headlamp,
or
coordinated
work,
ui
or
locomotive
a
fully
self-hosted
coordinated
distribution.
Today
I'll
be
presenting
with
manus
a
software
engineer
at
netflix.
He
works
on
the
compute
infrastructure
team
on
container
execution
and
we
work
together
on
the
cycle
ordering
on
the
cubit.
A
B
B
We
are
now
working
to
make
titus
the
primary
deployment
target
to
give
developers
one
well
supported
infrastructure
that
allows
them
to
iterate
fast
and
is
also
more
scalable
and
more
efficient.
At
the
same
time,
the
goal
is
to
do
this:
migration
fast
and
automate
as
much
as
we
can
so
hundreds
of
platform
engineers
are
not
supporting
two
deployment
targets
for
a
very
long
time
to
get
a
sense
of
what
we
are
migrating
here
is
what
a
typical
vm
runs
at
netflix.
B
As
an
example,
the
metrics
forwarder
starts
up
before
the
agent
which
boots
traps
certificates.
This
is
important
because
consider
what
happens
when
that
agent
starts
failing.
In
our
case,
it
can
publish
a
metric
to
report
these
failures
and
we
can
use
our
monitoring
and
alerting
systems
to
track
and
remediate
these.
B
In
the
same
way,
the
service
mesh
expects
tls
certificates
and
the
open
policy
agent
to
be
available
before
it
starts
up.
Otherwise
every
application
has
to
decide
if
the
mesh
needs
to
fail,
open
or
short
circuited
start
to
do
this.
We
run
demons
as
systemd
units
and
use
its
dsl
to
get
the
ordering
and
restart
policies
we
want.
B
B
B
B
We
started
by
running
a
multitenant
instance
metadata
service
as
an
example,
but
we
learned
very
quickly
that
users
can
accidentally
deploy
workloads
which
query
it
in
a
tight
loop.
This
starvs
other
containers
on
the
host,
and
if
that
loop
is
leaking
connections,
then
our
service
runs
out
of
file
descriptors,
and
now
we
have
a
bad
host.
B
B
The
second
problem
is
that
we
have
workloads
that
expect
to
not
be
disrupted
once
they
are
running.
This
means
all
our
system.
Services
will
need
to
have
the
ability
to
hot
reload
and
have
perfect
current
isolation.
If
these
workloads
are
going
to
meet
their
slas,
there
are
some
more
serious
challenges
when
it
comes
to
securing
these
multi-tenant
services.
B
B
B
B
B
B
Once
the
sidecars
are
running,
we
release
teeny
using
the
socket
and
it
runs
the
entry
point
at
the
child.
Pe
remains
speed,
1,
except
in
special
cases,
and
we
use
it
to
redirect
standard
output
and
error
and
also
to
do
more
advanced
things
with
sec
comp
notify
as
an
example.
Since
tv
is
not
really
optional.
The
way
we
use
it
titus
can
avoid
the
overhead
of
pause
containers.
B
We
don't
have
to
worry
about
an
application
starting
before
the
metadata
service
starts
and
crashing,
or
the
metadata
service
stopping
before
the
application
and
causing
the
application
to
crash.
I
should
point
out
that
there
are
some
special
services
like
ssh,
where
we
don't
use
run
c.
There
used
to
be
a
time
when
we
ran
docker
exec
to
get
a
shell
in
the
container,
but
that
is
insecure.
B
B
That
brings
us
to
how
we
version
and
upgrade
the
system
services
for
most
workloads.
The
executor
reads:
what
the
current
stable
version
of
side
cars
is
from
a
centralized
store,
and
this
happens
at
every
container
launch.
This
makes
upgrading
side
cars
fast
and
simple.
We
change
the
configuration
in
the
store
and
trigger
a
redeploy
if
required,
and
some
jobs
have
more
exacting
slas
and
they
specify
a
exact
version
for
a
specific
service.
B
The
challenge
in
this
space
is
pretty
big
and
it's
probably
worthy
of
a
discussion
of
its
own.
In
summary
system
services
are
pretty
easy
to
implement
using
rootless
containers.
We
get
the
benefit
of
being
able
to
share
mount
and
pit
name
spaces.
So
when
a
user
logs
into
a
container,
they
see
a
process
tree
and
file
system
layout.
That
makes
sense,
but
we
can
only
have
one
user
container
in
our
model
and
we
want
to
have
more
user
containers
to
move
towards
more
composable
workloads.
B
B
So
what
has
this
got
to
do
with
the
coblet
titus
started
life
as
a
maze
was
cluster,
but
starting
in
2019
we
moved
to
virtual
google.
My
colleague
sargon
talked
about
this
in
coupon
san
diego.
You
can
look
at
that
talk
and
over
the
past
year
we
have
been
adopting
more
kubernetes
components
in
the
control
plane,
like
coupe
schedulers,
crds
and
controllers,
which
do
fleet
management
and
usage
based
scheduling
as
an
example.
B
B
First,
we
need
startup
and
shutdown
ordering
to
run
parts
in
our
operational
model.
We
have
teeny
and
we
know
how
to
write
the
unique
socket.
So
naturally,
we
thought
of
building
a
coordinator
using
the
web
hook
and
you
get
this
contraption
this
works,
but
we
are
not
enthused
about
rewriting
entry
points
to
all
containers
when
using
the
cooler.
B
We
also
looked
around
and
found
kept
753
and
decided
we'd
work
with
rata
to
see
how
far
we
can
get
with
two
phase
ordering
now,
if
we
had
that
feature,
some
agents
like
the
log
viewer
and
aggregators
or
the
open
policy
agent
can
start
up
before
the
application.
If
we
mark
them
as
hydros,
you
can
even
make
the
cryptographic
bootstrap
work
by
splitting
it
into
an
init
container
that
generates
the
certificates
and
a
sidecar
that
refreshes
them.
B
In
summary,
we
run
sidecast
in
an
operationally
sensitive
fashion
and
letting
them
crash
is
not
an
option
for
us.
We
also
run
an
aggressive
security
posture
using
user
name,
spaces,
sec
com
and
app
armor,
and
there
have
been
cases
where
one
of
these
was
the
final
thread
between
us
and
an
attacker.
So
we
cannot
compromise
here.
B
A
Hi
I'll
now
explain
what
are
the
problems
that
we
have
with
cycle
containers
in
coordinates
today
and
the
efforts
we
did
in
coordinated
options
to
improve
this
before
we
start,
let
me
say
that
granite
knows
nothing
about
cycle
containers
today,
coordinated
treats
all
containers
in
the
same
way.
The
distinction
between
sidecar
and
main
containers
is
useful
only
for
us
at
least
for
now.
A
B
A
The
vast
majority
of
sidecar
containers,
so
one
problem
that
we
face
today,
is
that
there
is
no
startup
order.
Warranties
the
main
container
can
be
started
before
the
sidecars,
and
when
that
happens,
the
window
between
the
main
containers
is
started
until
the
sidecar
is
ready
is
the
window
where
the
problems
usually
arise.
For
example,
if
the
sitecar
is
a
service
mesh,
all
network
connections
will
be
black
hole
until
the
service
mesh
container
is
ready.
A
This
is
not
a
minor
problem
and
workarounds
are
not
great.
Quite
the
opposite.
I
walk
around
in
some
cases,
for
example,
is
to
do
nothing.
Let's
say
your
container
crashes
when
cycles
are
not
ready.
If
you
do
nothing,
your
container
will
crash
until
the
cycles
are
ready
and
the
next
time
it
is
restarted,
it
will
work.
A
This
can
be
slow,
though,
and
if
your
sidecar
depends
on
some
other
cycles,
they
can.
This
can
get
easily
super
slow,
and
this
will
be
a
problem.
You
you
face
to
scale
up
when
traffic
increases,
for
example,
another
workaround
is
to
change
your
container
entry
point.
For
example,
you
can
change
it
with
a
script
that
waits
for
other
cycles
to
be
ready
and
only
then
starts
the
main
process
in
the
container.
A
An
example
of
this
is
the
link
of
the
await
comment
and
that
waits
for
lingerie
to
become
ready
and
then
executes
your
container
process.
This
has
a
lot
of
downsides
because
it
becomes
impossible
with
lot
of
side
cars
and
we
fill
the
promise
of
augmenting
the
functionality
without
changes
to
the
main
container.
The
container
are
completely
coupled
right
now.
I
fully
do
this.
A
A
A
A
And
it
usually
works
like
this.
The
container
starts
compute
something
and
finish,
therefore,
all
container
exits
successfully
and
coordinates
process
to
clean
up
the
part.
However,
if
we
are
using
a
surface
mesh,
for
example,
we
start
the
part
with
two
containers.
Now
the
main
container
and
the
service
mesh,
the
manufacturer
finishes,
but
the
sidecar
won't
finish,
and
it
will
never
finish
because
it
is
a
demon.
Therefore,
therefore,
the
part
continues
to
run
forever
and
will
never
be
cleaned
up.
A
A
So
those
are
the
problems
we
currently
have
with
zakker,
but
now
let
me
tell
you
about
the
coordinates
enhancement
proposal.
We
work
on
to
improve
the
situation.
A
cap
is
a
document
that
describes
any
enhancement
made
to
kubernetes
and
the
sidecar
cap
that
adds
the
notion
of
software
containers
to
coordinates
has
a
long
history.
It
was
created
in
2018
by
joseph
irving,
and
it
started
a
big
discussion
in
the
community
about
the
different
ways
that
can
be
pursued
to
solve
these
issues.
We
just
talked
about.
A
I
joined
the
sidecar
effort
in
metro
new
tuning
and
we
really
try
to
do
what
what
is
best
for
the
community.
So
we
reach
out
to
as
much
people
as
possible
to
make
sure
what
we
propose
in
the
cap
worked
for
al
for
for
all,
and
not
only
for
us
and
as
a
side
note,
by
reaching
out,
I
found
out
that
several
companies
today
are
using
a
coordinated
fork
just
to
add
the
concept
of
sidecar
containers.
A
I
was
really
surprised
about
this,
and
but
it
helped
me
to
convince
myself
that
this
is
a
problem
we
need
to
solve
in
some
way
or
the
other,
and
so,
as
I
was
saying,
we
invested
a
lot
of
time
and
effort
in
this.
We
met
with
these
companies,
but
we
also
met
with
developers
from
linkedin
istio,
and
we
did
a
bunch
of
other
things
like
make
sure
the
cubelet
graceful
shutdown
cap
interacts
properly.
A
A
A
A
We
have
done
that
and
we
have
one
proposal
by
tim
hawking
and
the
idea
is
to
later
choose
with
the
community
which
proposals
should
turn
in
into
a
cap.
Looking
back,
we
made
good
progress.
We
have
documented
a
lot
of
use
cases.
We
can
now
use
to
create
proposals,
and
I
really
want
to
thank
everyone
that
helped
by
having
the
reuse
case
there.
A
This
journey
also
helped
to
clarify
that
side.
Cars
are
somehow
important
to
kubernetes
and
that
we
don't
want
to
make
minimal
changes.
We
want
something
that
probably
solves
the
problem,
and
that
opens
the
door
not
only
for
bigger
changes,
but
also
for
better
support,
not
just
fixing
the
most
obvious
use
cases
that
need
to
work,
but
also
more
complex,
container
dependencies
within
the
pot.
A
Overall.
This
is
clear,
a
big,
very,
very
big
project
and
we've
come
a
long
way,
but
even
though
it
is
far
from
finished
now
and
with
this,
wherever
we're,
where
we
are
today
where
at
this
precap
stage
joseph
moved
on,
suddenly
I
don't
expect
to
have
the
time
to
move
this
from
the
pre-cap
stage.
We
currently
are
at
all
the
way
into
a
cap
and
later
to
a
gi
feature,
but
I
spend
a
lot
of
time
on
this
and
would
like
to
help.
A
So
let
me
know
if
I
can
help
to
reveal
some
proposal
or
give
some
general
input,
like
luckily
matias
from
the
community,
has
volunteered
to
continue
to
drive
this
forward,
and
I
want
to
thank
him
again
for
this
and
yeah.
So
to
close.
Regarding
cycle
usage,
we
showed
four
problems
with
cycles
that
don't
have
a
original
workaround
today,
and
this
is
making
some
companies
fork.