►
From YouTube: How Google Manages Sidecars for Millions of Containers Without Breaking Anything (Much - Sven Mawson
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
How Google Manages Sidecars for Millions of Containers Without Breaking Anything (Much) - Sven Mawson, Google Cloud
Between 2014 and 2016, Google updated its production fleet to run every job with a sidecar as a way to support new languages like golang in production. This talk will discuss our experience of rolling out a sidecar to an extremely large fleet, the precautions that we took to make the rollout safe, and the benefits we've gained from this architecture. We will also describe how we continue to update our sidecar in a safe way for such a large fleet, and the lessons we've learned and problems we've solved along the way.
A
Let's
get
started
so
hi
I'm
spend
so
I
as
men
introduce
me
I'm
one
of
the
founders
of
the
history
project,
but
I'm
actually
not
here
really
to
talk
about
this
to
you
at
all,
I'm
here
to
talk
about
how
we
at
Google
rolled
outside
cars,
now
about
six
seven
years
ago
to
all
of
Google
when
there
used
to
be
no
sidecar
and
sort
of
the
lessons
we
learn
along
the
way
and
and
how
I
managed
that,
and
so
before
is
do.
Actually
that
was
my
job.
A
I
was
the
lead
of
the
team
that
built
the
sidecar
for
Google
and
rolled
it
out
and
several
pieces
of
functionality
in
that
so
and
we
tried
to
do
without
breaking
anything
and
that's
actually
kind
of
the
key
thing.
So
the
first
thing
I
want
to
talk
about
is
actually
why
a
sidecar-
and
so
this
was
covered
actually
in
a
few
talks
today
around
sort
of
why
serviced
mesh.
A
But
there
are
actually
reasons
that
we
adopted
it
that
aren't
really
quite
related
service
mesh,
although
they're
a
little
bit,
but
one
of
the
main
reasons
is
just
around
language
languages,
neutrality
and
how
many
different
languages
you
have.
So
at
the
time
that
we
were
doing
this
Google
had
three
production
languages.
Basically,
we
had
Java
C++
and
Python
and
Along
Came
cooling.
This
new
new
upcoming
language
that
was
being
developed
at
Google
and
Google
at
the
time
had
lots
and
lots
and
lots
of
her
language
stuff.
A
So
we
had
HTTP
and
RPC
client,
libraries
and
server
libraries.
We
had
a
bunch
of
service
management,
so
all
the
stuff
in
you
know
service
mesh
back
in
the
days
before
you
had
a
proxy.
You
did
all
that
in
code
right,
so
we
had
libraries
that
did
all
these
rape
ability.
You
know
all
the
sort
of
precursors
to
tracing.
We
were
doing
that
at
Google
on
a
project
called
a
purr
right.
We
had
the
the
precursor
to
G,
RPC
and
stubby.
A
We
had
all
this
stuff
in
there
and
it
was
doing
all
those
things
that
serve
as
meshes
give
you
just
encode
and
the
people
building
golang.
It
was
actually
pretty
small
team
and
they're
like
we
don't
want
to
rewrite
all
that
stuff
right.
We
don't
want
to
build
everything
out.
We
also
had
a
bunch
of
API
management.
Actually
that
was
another.
My
jobs,
I
was
I
was
one
of
the
leads
on
a
couple
different
API
management
products
at
Google,
and
we
were
sort
of
switching
away
from
a
central
proxy
model
to
a
distributed
model.
A
And
again
we
didn't
want
to.
You
know,
write
this
in
every
language,
so
we
were
like
we're
just
gonna
write
it
and
C++
make
it
super
fast
and
run
it
as
a
sidecar
and,
of
course,
there's
a
bunch
of
other
stuff
right.
So
you
and
any
big
organization,
you
have
lots
and
lots
of
different
client
libraries,
so
we
had
storage
libraries
for
BigTable.
That
thing
was
gigantic
and
you
know
it
had
to
be
written
in
a
bunch
of
different
languages
right.
This
just
gets
painful,
so
it'sit's
really
nice.
A
If
you
don't
have
to
write
that
in
every
single
language
and
one
of
the
other
problems
with
having
all
these
different
languages
is
that
they
start
to
diverge,
you
get
problems
where
the
defaults
are
different
in
different
languages.
You
have
different
bugs
and
different
languages,
and,
like
you,
have
these
weird
interactions
where
Java
on
this
version,
with
this
setting
talking
to
C++
on
this
version
with
this
setting
that
breaks
in
some
weird
way
that
no
one
actually
understands
why
right
and
it
takes
a
long
time
to
be
bugged.
It's
much
much
nicer.
A
If
you
have
that
all
in
one
place
and
then
of
course
also,
you
have
to
do
performance
tuning
now
on
X
number
of
languages
and
again
looking
at
go
small
team
even
at
Google.
It's
like
now
we're
not.
We
don't
want
to
do
that,
so
they
actually
were
the
team
that
sort
of
originated
the
idea
of
a
sidecar,
and
then
we
we
took
it
over
from
them.
There's
some
other
actual
problems
that
aren't
quite
related
to
the
language
issue
as
well,
and
actually
this
was
maybe
even
more
important
the
the
language
issue.
A
You
can
kind
of
throw
money
at
right.
You
can
just
spin
up
more
and
more
teams
that
are
building
these
systems
in
every
language,
but
when
you
have
a
huge
company
with
a
huge
number
of
different
processes
running-
and
you
have
let's
say
a
bug
in
the
TLS
library
that
needs
to
get
rolled
out
to
a
million
different
processes
and
upgraded
and
by
the
way
it's
a
zero-day,
and
so
you
need
to
upgrade
it
like
in
you
know
two
days
a
week.
This
is
really
really
challenging
right.
A
A
A
So
you
know
for
the
for
the
language
reasons
and
for
the
update
reasons
we're
like
okay,
we
have
to
have
a
sidecar
or
we
have
to
have
something
right
doesn't
have
to
be
a
sidecar
right,
there's
other
options
for
getting
this
functionality
into
places,
and
we
actually
looked
at
a
bunch
of
these
and
rejected
them.
So
we
looked
at
something
like
like
swig
a
foreign
function
interface.
There
are
various
reasons
that
didn't
work
out
very
well.
One
of
them
is
actually
doesn't
help
the
update
problem,
so
it's
sort
of
got
like
removed
from
that.
A
It
helps
with
the
language
thing,
but
then
you
still
need
to
update
these
DL
O's.
We
looked
at
as
well,
and
actually
we
spend
a
lot
of
time
investigating
dll's.
We
the
way
we
had
designed
our
systems
deals
were
kind
of
horrible,
so
everything
is
statically
linked
now,
if
you
wanted
to
switch
to
the
oils
you
had
to
like
rewrite
all
of
googles
software,
to
support
that,
and
that
was
not
going
to
happen
and
then
the
other
thing
we
rejected,
but
actually
it's
not
really
projected
is.
Okay.
A
Can
all
this
stuff
be
remote
services
and
in
some
cases
the
answer
was
actually
yes,
some
of
this
stuff,
that
was
in
a
client
you
could
actually
move
to
remote
service
and
where
we
could,
we
did
that.
So,
if
you
look
at
like
cloud
BigTable
cloud
spanner
services
from
Google,
those
don't
actually
have
these
giant
hundred
megabyte
client
libraries,
because
that
would
be
horrible.
So,
even
though
the
internal
versions
have
these
big
libraries,
we
figured
out
ways
to
make
the
systems
better.
A
A
Those
things
mostly
can't
be
run
as
a
service,
actually
some
bits
of
it
even
can
so
like
TLS
handshake,
we
actually
developed
a
model
where
it's
not
actually
a
remote
service
and
service
on
the
same
node,
but
actually
you
can
do
the
handshake
through
a
node,
and
then
you
don't
have
to
run
that
thing
locally.
So
there's
little
tricks
you
can
do
to
try
and
remove
as
much
as
you
can
okay.
So
we
had
this
ID
card,
we
built
it,
we
built
it
in
C++.
A
We,
you
know,
spend
a
long
time
developing
it,
and
then
we
have
this
problem
of
okay.
We
have
these
millions
of
processes
running
with
out
of
sidecar
and
how
do
we
actually
get
it
out?
So
this
is
sort
of
the
rest
of
this
talk
is
kind
of
an
in-depth
view
of
what
the
Pinterest
talk
was
talking
about
at
the
end
there
about
like
how
do
you
actually
roll
things
out?
How
do
you?
How
do
you
deal
with
this
problem?
A
So
we
decided
yes
all
right.
We
will
take
that
challenge.
We
will
do
it,
so
this
is
actually
really
really
scary
right,
especially
if
you
have
a
bunch
of
critical
services
that
you
do
not
want
to
break.
You
know
search
ads,
Gmail
right
these
kind
of
big
scary
systems
that
are
serving
millions
billions
of
users.
You
don't
wanna
break
those.
You
want
to
make
sure
this
is
safe,
so
one
of
the
things
we
did
is
actually
say:
okay,
we're
not
gonna
start
with
any
of
those
things
right.
A
We're
gonna
start
with
with
small
people
that
have
something
to
gain
from
this
right,
the
people
with
the
most
to
gain
so
if
they
were
getting
some
functionality
that
they
couldn't
get
elsewhere,
so
that
was
actually
the
best
possible
customer
because
then
they
actually
had
even
said
have
to
work
with
us.
So
there
were
two
big
ones
here.
One
was
going
as
I
mentioned
before
right.
That
was
a
big
driver
of
this,
and
so
any
of
these
new
services
at
Google
that
we
were
being
written
in
going
right.
A
They
needed
access
to
big
table
or
spanner
or
whatever
was,
and
so
they
got
sent
our
way.
We,
you
know
we
set
them
up.
We
developed
the
system
and
got
them
to
be
able
to
use
it,
and
so
they
be
early
testers
of
this
stuff.
The
other
big
one
and
the
one
I
was
directly
involved
in
also
was
the
the
API
management
part
right.
So
we
built
this
out.
People
wanted
to
try
out
the
new
generation
of
API
management.
That
was,
you,
know,
100x
faster
than
the
previous
generation.
A
A
The
main
downside
to
service
meshes
is
the
latency
right.
The
latency
and
the
resource
usage
actually
there's
two
so
you're
going
to
be
introducing
a
proxy
on
both
the
client
and
server
side
of
every
connection,
and
that
practice
is
gonna.
Take
CPU
it's
going
to
take
memory
right,
it's
if
you
use
disk,
it's
use
disk
and
it's
also
gonna
gonna
cost
you
Lee
and
CN.
For
some
services.
That's
actually
really
really
painful,
so
we
actually
spent
a
lot
of
time
investing
in
performance.
A
The
performance
of
Google's
internal
sigh
car
is
actually
orders
of
magnitude
faster
than
anyone's.
You
can
get
externally
we're
actually
working
on
changing
some
of
that
right
now,
so
we're
trying
to
help,
invest
in
things
like
sto
and
M
way
and
make
those
side
cars
as
fast
as
our
internal
one,
because
we
want
to
use
it
too,
but
you
really
have
to
spend
the
time
right.
It
takes
years
and
years
of
effort
to
make
these
things
as
fast
as
it
they
possibly
can
be
because
it
it
ends
up
being
a
huge
savings
right.
A
A
So
something
else
we
did
to
actually
help
out
is
to
identify
people
that
would
be
affected
by
this.
So
if
someone
is
actually
going
to
be
impacted
by
the
fact
that
you're,
adding
this
proxy
or
you're,
adding
the
sidecar
and
you're
introducing
resource
usage
right,
if
they're
close
to
their
limit,
you
need
to
let
them
know,
and
so
we
went,
we
were
very
proactive
about
reaching
out.
A
One
benefit
we
had
is
actually
most
of
most
of
our
systems
were
actually
starting
to
move
to
this
autopilot
mode,
where
they
would
actually
get
the
resource
usage
adjusted
automatically,
and
so
for
them.
We
didn't
have
to
worry
about
it
and
we
could
filter
them
out
and
only
look
at
the
people
that
were
sort
of
statically
set
up
reach
out
to
them
and
say:
hey.
You
know
we're
gonna,
introduce
this
thing
to
try
it
out,
make
sure
it
doesn't
break
you.
If
it
does
break
you
add
resources.
A
A
Like
just
keep
warning
them
and
let
them
know
yes,
we
are
changing
this
on
you,
this
thing's
coming
it's
here,
it's
happened
and
make
sure
they
know
what
to
do
and,
of
course,
when
you're
doing
this,
you
want
to
take
it
really
really
slow.
So,
if
you're
rolling
something
like
this
out,
you're
rolling
out
a
sidecar
to
millions
of
processes,
you
don't
want
to
do
it
overnight.
You
want
to
take
your
time
in
our
case.
A
A
In
our
case,
we
did
sort
of
my
cluster
jumping
job
right,
even
start
with,
like
one
task
and
a
job
make
sure
it's
safe
and
then
continue
the
rollout
and
the
other
thing
you
want
to
do
is
actually.
If
you
do
break
someone,
you
want
to
make
sure
that
opt-out
is
really
really
easy
and
I'm
just
going
to
talk
a
little
bit
about
that
later.
A
So,
okay,
so
you've
rolled
it
out,
you've
got
this
sidecar.
Now,
in
your
millions
of
processes,
but
one
of
the
key
benefits
of
this
thing
was
that
we
could
do
it
up
updates
right
that
we
can
like
find
a
security
vulnerability
or
someone
else
tell
us
one
about
one
and
like
we'll
fix
it
and
roll
it
out
to
millions
of
site
cars
and
not
break
anything
right.
So
upgrading
actually
is
is
really
scary
as
well.
We
had
a
bunch
of
different
approaches.
A
We
took
here
I'm
going
to
go
through
a
couple
of
them,
so
one
of
the
main
things
we
did
is
actually
split
into
a
dev
and
stable
branch.
Dev
we
did
weekly
stable,
we
did
monthly
and
this
helped
us.
Let
people
try
things
out
early
right.
So
if
someone
was
actually
worried,
if
someone
was
critical,
they
could
try
out
to
give
us
feedback
early,
make
sure
it
didn't
break
and
then
we
roll
it
out
roll
out
stable.
The
other
thing
we
actually
made
and
I'll
talk
about
this
a
little
bit
later
too.
A
A
You
also
want
to
make
sure
this
is
as
simple
as
possible.
You
want
to
make
it
automatic,
not
manual,
so
for
most
of
our
users,
they
were
on
on
automatic
right.
They
got
the
latest
and
greatest
sidecar
on
a
restart
and
I'll
talk
a
little
bit
more
actually
later
about
why
we
did
restarts,
but
we
wanted
to
make
sure
that
most
people
just
get
the
latest.
There
are
people
for
whom
that
does
not
work,
so
they
actually
care
a
lot
about
exactly
which
version
there
are
some
teams
at
Google
where
there
are
no
changes
allowed.
A
Basically,
at
all
automatic
everything
has
to
go
through
a
code
review.
Everything
has
to
be
very,
very,
very
securely
locked
down
for
them.
We
we
came
up
with
a
different
mode,
which
was
what
we
call
bound.
So
the
bow
mode
is
basically
you
get
the
version.
That
was
the
latest
when
you
built
your
binary.
So
you
build
a
binary.
A
The
latest
is,
you
know,
version,
and
that
version
stays
with
that
binary
as
as
long
as
that,
binary
is
out
there
and
as
long
as
that
version
is
still
supported,
and
so
we
had
a
contract
with
these
teams.
That
said,
hey,
okay,
if
you're
in
this
mode,
you
better
be
doing,
updates
pretty
often
all
the
more
like
weekly.
So
this
wasn't
a
problem
and
they
need
to
be
taking
critical
fixes
quickly.
So
there
was
a
small
list
right
there's
like
maybe
ten
teams
out
of
the
thousands
right.
A
A
So
you,
you
want
to
make
sure
you're
qualifying
with
that
dev
track
a
canary
track,
whatever
you
do,
make
sure
you're
qualifying
these
releases
with
early
users
getting
things
in
there
canary
clusters
and
then
slowly
doing
that
ramp
up
to
100%
right
as
the
the
folks
from
Pinterest
were
talking
about
right,
you
can
do
like
a
specific
host
right.
You
want
to
do
things
slowly.
You
want
to
take
your
time.
You
also
wanna,
make
sure
you
don't
make
them
grumpy.
You
don't
want
to
make
your
user
said.
A
So
this
is
what
I
was
alluding
to
before
about
roll
out
roll
back.
So
one
thing
we
realized
very
early
on
is:
we
could
not
be
responsible
for
rolling
everyone
back
right.
So
if
something
breaks
that
team
right,
the
SRE
team
responsible
for
that
needs
a
very,
very
quick
button
that
says
this
broke
me
fix
it
that
doesn't
actually
roll
it
back
for
everybody,
because
there
was
bound
be
problems
that
only
happened
to
one
or
two
users
right.
A
There's
you
change
one
little
thing
in
one
library,
and
you
know
a
specific
user
with
a
specific
setting
and
a
specific
circumstance
hits
the
bug
and
everyone
else
is
fine
right.
That
user
needs
a
way
to
rollback.
You
don't
want
to
roll
the
whole
thing
back,
especially
if
this
rollout
is,
you
know
proceeding
across
millions
of
machines
or
whatever
it
is.
So
we
had
a
very,
very
easy
model
where
a
user
could
just
say
hey.
This
is
the
I
know.
I
would
get
this
version
that
would
get
version.
A
10
I
actually
want
version
9
and
they
can
say
I
want
verse,
9
restart
the
job
they
get
9,
and
so
it
was,
you
know,
could
take
the
minutes
to
switch
and
rollback,
and
one
of
the
things
we
wanted
to
make
sure
is
when
they
do.
That
is
that
we
weren't
letting
that
linger
right.
So
you
don't
want
to
let
rollback
stay
in
production
forever.
A
It
needs
to
be
a
short-term
fix,
only
short-term
remediation,
so
we
exercise
a
requirement
that
they
had
to
file
a
bug
with
it
and
they
had
to
say
one
thing
bug
was
we
kept
a
dashboard
of
the
user
we
go
through.
We
made
sure
that
they
were
very,
very
short-term
and,
of
course,
we
were
very
proactive
about
fixing
them
right.
So
you
want
to
make
sure.
Ok,
someone
had
rollback,
let's
fix
it
and
then,
of
course,
if
you
actually
see
lots
of
people
rolling
back,
you
realize
you
have
a
real
bug.
A
You
roll
the
whole
thing
back.
You
try
again
next
time.
Okay,
so
we
got
we
got
deployed.
We
got
it
upgrading.
So
one
of
the
things
we
learned
about
during
this
process,
so
one
of
the
biggest
things
is
to
invest
in
debugging
tools
right.
So
you
know
there
is
actually
already
a
talk
this
morning
about
debugging,
but
you
want
to
make
sure
that
it
is
very
easy
to
figure
out
when
there
is
a
problem
right.
A
So
when
there's
when
you
have
something
like
a
sidecar,
you're
you're,
now
talking
about
two
processes,
sometimes
three
processes
where
you
used
to
be
only
talking
about
one
and
things
get
complicated
right.
You
got
more
moving
parts.
You
have
things
talking
to
each
other.
You
have
to
like
debug
across
process
boundaries,
maybe
you're
debugging
in
different
languages
right
like
maybe
it's
Java
and
the
citecar
C++
and
like
it
just
gets
really
confusing
and
complicated.
So
you
want
to
make
sure
that
you
have
these
debugging
tools
that
let
you
really
quickly
know:
okay,
whose
fault?
A
Is
it
right?
What
went
wrong?
How
often
is
happening
and
there's
there's
the
tooling
part
of
that,
and
then
there's
also
just
the
like
monitoring
part.
So
you
really
want
to
make
sure
you're
doing
a
lot
of
monitoring,
checking
for
what's
wrong
and
keeping
the
dashboard
so
that
one
of
the
thing
actually
that
we
learned
that
really
is
probably
irrelevant
for
most
people,
but
for
anyone
that's
going
down
the
crazy
path
we
went
down
is
that
you
want
to
make
sure
that
communication
channel
between
those
processes
is
very
very
solid.
A
So
we
were
actually
experimenting
with
some
shared
memory
transports
that
let
us
do
sort
of
low
single-digit,
microsecond
communication
switches
between
the
sidecar
and
the
application.
It
turned
out,
building
that
yourself
from
scratch
and
then
trying
to
roll
it
out
was
a
bad
idea.
It
was
not
as
solid
as
we
wanted.
It
actually
had
lots
of
bugs
things
broke
all
the
time.
We
ended
up
just
ditching
that
and
switching
Tina's
domain
sockets,
and
suddenly
all
those
problems
went
away.
A
So
you
really
want
to
make
sure
that
if
you
do
do
something
crazy
try
to
spend
your
time
making
sure
it
actually
works
really
really
well,
because
actually,
this
kind
of
goes
for
anything
at
this
scale
when
you're
talking
about
you
know
millions
of
processes,
any
little
thing
that
happens
very
very
rarely
still
happens
all
the
time
right,
it's
the
usual.
Listen,
the
other
thing.
I
talked
it
before
you're
going
to
take
performance
hit.
You
have
to
know
that
and
accept
that
upfront
right.
This
is
going
to
be
a
cost.
A
If
you
are
designing
like
client
libraries
that
you
you
stick
in
a
sidecar
rather
than
proxying,
which
is
mostly
what
this
conference
about
more
of
the
proxying,
but
for
us
it
was
actually
more
the
clients
that
we're
interesting.
You
want
to
make
sure
that
you
design
that
as
a
batch
connection,
not
not
chatty
right,
so
we
actually
had
a
case
at
one
point
where
were
you're,
sending
like
every
single
log
line
output
by
the
application
was
a
a
single
RPC
to
the
sidecar.
A
A
Another
lesson
we
learned
actually
was
around
default
behavior.
So
if
you
are
going
to
change
something
right
for
millions
of
users,
you
should
probably
not
just
change
it.
You
should
probably
do
the
exact
same
thing.
I
was
talking
about
with
the
rollouts.
Where
you
have
some
sort
of
opt-in.
People
can
try
it
out
all
right,
you
do
it
over
time.
You
get
people
used
to
it
and
then
you
solutely
change
the
default
and
you
warned
them
and
warned
them
like
all
those
lessons
about
how
to
do.
A
A
rollout
also
applies
to
any
change
really
anything,
and
even
when
you
think
it's
an
oculus,
it's
not
gonna
be
innocuous.
It's
going
to
break
someone.
Any
change
is
a
dangerous
change.
You
have
to
roll
it
out
slowly,
so
you
know
we
had
very,
very
strict
rules
about
guarding
behavior.
We
had
strict
rules
about
adding
new
connections.
A
lot
of
these
rules
were
put
in
place
because
of
mistakes
that
we
made.
A
So
we
actually
had
a
point
where
we
broke
one
of
Google's,
few
billion
user
systems
for
several
hours,
because
we
added
a
new
connection
that
new
connection
took
out
a
back-end
system
that
back-end
system
was
being
relied
on
by
a
different
system
that
didn't
have
any
timeouts
the
cascading
failure
and
it
took
out
huge
swathes
of
Google.
So
don't
do
that,
make
sure
that
you
are
not
introducing
new
behavior
without
thinking
it
very
very,
very
carefully.
A
Another
actually
interesting
lesson-
and
some
of
this
also
applies
to
how
we
put
together
is
do-
is
try
to
do
anything
too
fancy
right.
So
there's
some
cool
things
you
can
do.
You
can
do
like
hot
restore
words.
You
can
try
and
like
do,
updates
in
place
and
like
all
this
kind
of
crazy
stuff.
That
sounds
really
cool
on
paper,
but
in
actuality
just
makes
things
more
complicated
and
harder
to
debug
and
harder
to
figure
out
what
went
wrong
so
like,
for
example,
one
of
the
we
did
was
just
we
said
all
right.
A
Should
we
have
the
sidecar
be
able
to
restart
if
it
fails,
shouldn't
be
able
to
come
back
up
or
like,
should
it
restart
the
application?
If
the
patient
said
no
forget
that
right
there
they're
tied
together,
let's
just
make
the
sidecar
as
bug-free
as
possible.
Our
goal.
The
time
was
like
no
more
crashes
than
the
kernel,
so
in
the
amount
of
times
that
the
kernel
was
crashing,
the
sidecar
was
allowed
to
crash
about
that
much
too,
and
that
level
of
sort
of
background
crashing
was
acceptable.
A
Most
people
wouldn't
notice
because
their
application
was
crashing
way
more
than
that
and
what
we
did
was
any
time
one
of
those
crash,
the
other
crash
tube
and
just
it's
just
they
took
each
other
out
and
then
they'd
restarted
and
we
used.
You
know
our
internal
system,
that's
a
precursor
to
kubernetes,
for
container
management,
just
to
restart
the
whole
thing,
and
everything
was
fine,
so
we
didn't
do
any
of
the
hot
reloading.
We
picked
up
new
things
on
restart
right,
so
we
didn't.
We
didn't
roll
out
new
versions
dynamically.
A
We
just
wait
for
things
to
restart.
It
turned
out
this
worked
well
for
us.
There
are
probably
people
here
who
are
running
systems
that
don't
restart
that.
Often,
if
you
do
that,
then
obviously
your
need
to
figure
out
a
way
to
roll
out
your
updates
a
little
bit
different
man-
or
maybe
you
just
go
restart
those
things
we
actually
for
a
while
designed
a
system
that
would
go
and
Reese.
We
thought
about
designing
a
system
that
would
go
restart
people's
jobs
for
them.
A
It
was
like
a
one-line
change
to
a
build
file
and
everything
just
worked.
Okay,
so
the
last
lesson
and
the
most
important
one
is
that
actually,
this
is
totally
totally
worth
it.
So
you
know
I
I've
actually
worked
at
Google
for
a
very
long
time
and
I,
remember
sort
of
prior
to
us
having
this
a
couple,
different
incidents
where
basically
every
engineer
at
Google
had
to
go,
take
some
change
to
production
for
some
fix.
So
we
had
a
couple
security-related
ones.
A
A
So
the
question
is:
how
do
you
roll
out
a
side,
a
side
car
where
some
have
a
side,
car
and
some
down
right
without
breaking
the
communication?
So
well
as
I'm
gonna?
Give
you
two
answers
so
for
Google?
Actually,
our
side
car
was
not
originally
for
proxying
of
communication.
It
was
mostly
just
a
client
library,
so
we
didn't
really
have
that
problem,
but
actually,
as
part
of
Ischia.
A
We
do
have
that
problem
so
for
issue
of
the
way
we
designed
it
is
to
work
that
way,
so
you
can
actually
add
a
side
car
to
something
and
the
other
hood
doesn't
need
a
side,
car
and
actually
it'll
keep
working.
Fine.
You
have
to
roll
up
the
side
car
as
a
restart,
so
you're
not
gonna
like
roll
it
out.
All
the
connections
missed
alive
but
yeah.
It
works
perfectly
fine
to
have
sort
of
side,
cars
and
non
side,
cars
and
one
of
the
hard
things
there
is
actually
around
mutual
TLS.
A
So
one
of
things
is
steve
supports
is
you
can
have
what's
called
permissive
mode
where
both
TLS
and
non
TLS
can
talk
to
the
same
port?
And
so
you
can
actually
have
your
your
new
side,
cars
that
pop
up
are
gonna,
be
talking
over
TLS
and
they'll
be
encrypted
and
the
old
ones
that
are
not
yet
running
the
side.
Car
won't
be
and
it'll
just
continue.
Working
fine
I
know
most
questions.
A
A
So
the
question
is
about
the
about
dis:
do
and
go
laying
in
the
sidecar,
so
the
sidecar
is
actually
AMA,
which
is
C++
yeah
so,
and
that
actually
was
intentional.
So
when
we
were
starting
this
new
project
that
basically
the
first
thing
we
were
doing
is
looking
at
proxy
implementations.
So
I
had
just
come
from
building
my
own
proxy
internally.
That
was
super
fast,
but
like
could
not
be
exported,
it
was
really
based
in
it
was
like
you'd
have
to
export
all
of
googles
technology
to
get
the
sidecar.
So
we
were
like
all
right.
A
What
do
we
use?
We
look
at
engine
X,
we're
looking
at
a
chat
proxy,
and
that
was
actually
about
the
time
that
all-boy
came
out
and
we're
like
all
right.
It
checks
the
box
of
c++,
it
checks
the
box
of
well.
It
didn't
quite
check
the
box
at
performance
at
the
time.
It's
gotten
lot
better,
but
yeah
that
that
was
actually
a
big
driver
for
us.
We
would,
we
would
not
have
chosen
a
googling
version
of
the
sidecar
yeah.
A
Yeah,
so
on
the
control,
plane
and
performance.
So
the
the
main
thing
to
worry
about
on
the
control
plane
is
usually
more
scalability
and
resource
usage
than
pure
latency
latency
you
care
about
in
terms
of
like
updates.
So
if
you're
getting
an
update
in
time,
but
you
don't
have
it
in
in
the
critical
path
right.
It's
not
in
the
data
plan,
so
you're
not
actually
paying
a
latency
hit
for,
let's
say
ISTE
owes
control
plane,
taking
an
extra
two
milliseconds
versus
if
it
was
written
as
C++.
A
A
A
We
were
talking
about
the
like
sidecar
per
node
versus
per
process
versus
per
whatever,
and
we
came
up
with
the
same
model
that
we
do
in
this
deal,
which
is
basically
per
pod,
so
I'm
gonna
put
the
sidecar
per
pod
and
a
lot
of
the
reasons
are
actually
the
same
as
the
reasons,
and
this
do
so.
You
avoid
having
to
deal
with
a
lot
of
security
problems.
A
If
you
tried
to
do
a
a
proxy
per
node
and
those
nodes
are
multi-tenant,
that's
actually
really
difficult
to
get
right
and
to
guarantee
that
those
things
aren't
able
to
interfere
with
each
other.
Even
in
a
trusted
environment
like
most
of
your
interference,
is
actually
accidental.
So
can
you
know
this
person
who
randomly
started
up
some
job?
Taking
me
down,
if
you're
running
at
the
node
level,
you
can
actually
take
each
other
down
pretty
easily,
so
we
put
one
per
per
sidecar
and
that
actually
also
helped
with
the
rollout
right.
A
So
if
you're
rolling,
out
and
you're
controlling
and
each
let's
say,
pod
owner
in
the
sort
of
cloud
native
model
right
can
say
which
a
version
they
want
right,
you
can't
get
that
at
a
node
level
right.
You
can't
do
that
same
kind
of
version
control,
so
it
works
really
well
to
have
it
per
pod.
The
downside
is
resource
usage,
so
you
have
to
make
sure
that
you're
light
lightweight
yeah
cool
all
right.
Thank
you.
Everyone.