►
From YouTube: Lightning Talk: How to deploy mutually authenticated TLS without ruining everything - Spike Curtis
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Lightning Talk: How to deploy mutually authenticated TLS without ruining everything - Spike Curtis, Tigera
TLS with mandatory mutual authentication is the gold standard for communication in distributed applications and forms the backbone of a Zero Trust Network. Envoy can do it for you with no application code changes, but if you just “turn it on” in a live production cluster you’ll quickly find you have a major disruption on your hand.
In this presentation, Spike will explain and demonstrate how to take a production cluster from a completely unencrypted to fully secured without dropping traffic. The demonstration will use Istio, but Spike will explain conceptually and cover the Envoy config changes being made in each step so the techniques can be applied to any Envoy service mesh.
A
Alright,
let's
get
started
so
this
is
a
lighting
talk,
so
I'm
gonna
be
moving
pretty
quickly,
but
I'll
be
around
later
this
afternoon.
If
anybody
wants
to
talk
about
these
issues,
so
I'm
spike,
Curtis
I'm,
a
software
engineer
at
tae,
Gera,
I'm,
a
very
occasional
contributor
to
envoy
and
I
work,
primarily
on
security
and
I
work
with
Sto
for
some
of
the
security
stuff
that
we
do
and
if
I
could
advance
slides
there
we
go.
So
this
is
a
talk
about
enabling
mutually
authenticated
TLS
connections.
A
So
what
that
means
is
that
we
want
to
go
from
something
that
looks
like
this,
where
we've
got
a
work
load
we've
got
envoy
running
as
a
sidecar
everything's
talking
plaintext
and
without
changing
our
application.
We
want
to
tell
envoy
to
start
encrypting
and
mutually
authenticating
that
traffic,
so
this
is
something
that
you
can
do
in
is
do.
A
This
is
something
you
can
do
in
Envoy
by
setting
these
values
in
in
the
XDS
configuration
for
your
up
streams
and
the
problem
with
going
from
plaintext
to
mutually
to
mutually
authenticated
TLS
in
your
cluster
is
that
essentially,
we've
got
two
things
that
are
in
some
distributed
system.
We've
got
a
bunch
of
instances
of
each
of
them
and
we
want
them
to
sort
of
go
from
one
state
to
another
state
more
or
less
at
the
same
time.
A
That
connection
will
fail
and
if
the
client
gets
the
configuration
first
and
starts
transmitting
encrypted
data,
but
the
server
is
still
configured
to
accept
plaintext.
That
will
also
fail.
So
we
need
to
be
really
careful
about
doing
this,
and
what
we
would
like
to
be
able
to
do
is
convince
the
server
side
envoy
to
first
except
both
plaintext
and
mutually
authenticated
TLS.
So
that's
like
step
one
before
we
touch
anything
on
the
client.
Ask
the
server
to
be
permissive.
A
Accept
both
and
then
after
that's
all
rolled
out,
ask
the
clients
to
start
transmitting
TLS
and
then
step
three
is
to
tell
the
servers
to
stop
accepting
plain
text.
So
that's
what
we
would
like
to
happen.
We
would
like
to
be
able
to
do
this
without
restarting
it.
Anything
envoy
has
XDS
API,
as
it
should
be
able
to
take
this
new
configuration
without
restarting
without
doing
a
redeploy
without
dropping
traffic.
A
However,
as
I
would
like
to
demonstrate,
that
is
not
working
for
us
right
now
and
we'll
talk
a
little
bit
about
how
we
might
be
able
to
get
there
so
I
have
a
kubernetes
cluster
running
I've,
sto
version,
0,
1,
dot,
0,
dot,
3
and
I've
got
a
very,
very
simple
application
right.
It's
just
a
bunch
of
HTTP
servers,
ten
of
them
and
I've
got
an
sto
ingress
gateway
which
is
envoy
under
the
covers,
and
it
is
transmitting
connections
or
sorry
load
balancing
to
those
backends.
A
Each
of
those
backends
has
their
own
Envoy
as
a
sidecar
as
well,
which
is
handling
those
those
connections
and
gonna
be
able
to
handle
the
encryption
for
us.
So
what
I
do
in
sto?
If
I
want
to
turn
on
mutually
authenticated
TLS
as
I
create
two
objects:
I
created
a
destination
rule.
This
affects
clients
and
say
that
I
want
all
of
my
clients
to
start
transmitting
TLS
and
then
I
write
a
mesh
policy
which
affects
all
of
my
servers
and
I
tell
them
that
I
want
them
to
start
accepting
mutual
TLS
connections.
A
And
we'll
look
at
a
nice
graph
on
a
dashboard
you'll
see
that
I
started
off
with
plain
text:
I've
got
about
500
requests
going
through
here
and
I
am
now
switching
from
plain
text
over
to
mutual
TLS.
That's
the
sort
of
read
to
the
to
the
orange
going
on
here,
but
you'll
see
that
during
that
transition,
I
got
a
bunch
of
errors,
so
I
got
five
hundreds,
because
that
transition
was
not
smooth.
Some
of
those
clients
sent
TLS
connections
to
servers
that
didn't
accept
them
and
and
vice
versa.
A
A
A
So,
let's
good
look
at
our
our
transitions,
so
you
saw
that
I
did
a
roll
roll
back
to
plain
text.
I
had
some
errors,
rolling
back
to
plain
text
and
now
I'm
not
changing
anything
over
to
mutuality,
jewel
TAS
I'm,
just
saying
everything
should
accept
mutual
TLS,
but
you'll
see
that
I
still
am
getting
errors
in
my
cluster,
and
this
is
the
sort
of
bit
that
is
not
working
and
why
this
type,
this
talk
is
still
titled
how
to
enable
mutual
TLS.
With
with
ruining
everything.
A
But
eventually
those
errors
will
will
sort
of
be
done
with
and
we'll
kind
of
get
back
to
everything.
Looking
fine
and
healthy.
What's
that
that
configuration
has
rolled
out
and
since
we're
under
time
pressure,
oh
well,
actually
I'll
just
show
what
the
next
step
is.
So
I
have
changed.
All
of
my
back
ends
to
start
accepting
TLS
connections
now
I'm
going
to
change
all
of
my
clients
to
start
transmitting
on
TLS
connections,.
A
All
right
so
now
everybody's
sending
TLS,
and
we
should
be
able
to
observe
on
our
metrics,
usually
by
the
way
I
want
to
give
a
shout
out
to
to
sto
for
for
making
these
kinds
of
graphs
and
metrics
dead
easy.
So
this
is
just
standard
sto
install
and
the
only
thing
that's
different
years,
I
made
a
custom
graph
on
a
dashboard,
but
all
these
metrics
are
just
coming
out
of
sto
out-of-the-box
to
be
able
to
find
out
whether
I'm
doing
TLS
and
find
errors,
and
things
like
that.
A
So
that's
that
part
is
super
working
super
well,
so
here
I've
asked
all
the
clients
to
start
transmitting
mutual
TLS
and
you'll
most
that
happened
without
any
errors.
So
all
of
my
clients
switched
over
there
now
transmitting
TLS,
and
the
last
thing
that
I
want
to
do
is
tell
my
backends
to
stop
accepting
plain
text
connection
connections,
lock
everything
down
to
mutual
TLS.
So
that's
what
this
policy
looks
like.
A
All
right,
so
switching
over
to
strict
mode,
sometimes
will
throw
me
errors.
Oh
there,
we
go
a
little
little
little,
a
couple
of
errors
and
hanging
out
there.
So
what
we
should
be
able
to
do
is
get
this
working
and
not
not
have
these
errors.
Unfortunately,
this
this
is
not
working
right
now,
I
haven't
really
been
able
to
get
to
the
bottom
of
why
this
is
not
working.
A
Whether
this
is
a
miss,
do
miss
configuration
or
whether
there's
something
fundamentally
more
wrong
with
envoy
when
you're
running
this
much
traffic
through
it
all
this
is
only
50.
You
know
requests
per
second,
so
I
don't
know,
but
what
the
you
should
take
away
from
this
talk
is
that
you
need
to
be
careful
going
from
plain
text
to
mutual
TLS
and
I.
Definitely
don't
want
to
discourage
anybody
from
not
doing
that
I
care
about
security.
I
think
you
really
should
do
that.
A
But
trying
to
do
this
live
without
restarting
anything
doesn't
seem
to
be
working
right
now,
so
we'll
probably
still
have
to
depend
on
the
other
tools
that
we
have
in
our
tool.
Catus
operations,
that's
rolling
redeploys
get
every
all
of
your
background
back
ends
able
to
accept
both
mutual
TLS
and
and
plaintext
traffic.
Make
sure
that's
looking
good
across
here
across
cluster.
Do
that
slowly
move
the
traffic
carefully
then
again,
do
a
rolling
deploy,
get
all
of
your
clients
to
switch
over
to
TLS
and
then
finally,
a
third
rolling
deploy
to
turn
off
plain
text.