►
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Full Mesh Encryption in Kubernetes with WireGuard and Calico - Peter Kelly, Tigera
Encrypting data-in-transit is an important feature for many Kubernetes users especially for compliance and a zero-trust model. There are several ways this can be achieved, including using WireGuard, an exciting new lightweight VPN in the Linux kernel. This talk explains why you would choose WireGuard for this task and how it can work in a dynamic platform such as Kubernetes using Project Calico to provide a full host-to-host encrypted mesh at a layer below your application workloads. WireGuard is popular for good reason; lightweight, fast, scalable and easy. We’ll show you how easy it is to make it work but also dig in to the implementation details for those who love to sweat the details.
A
So
thanks
for
coming
to
the
talk
today,
just
start
with
a
little
bit
of
background
about
about
myself
and
and
then
take
you
through
what
we're
going
to
talk
about
today.
So
I
work
for
taguera.
So
taigara
is
the
company
behind
project
calico,
which
is
the
most
deployed
cni
for
kubernetes
powers,
about
2
million
nodes
daily
in
about
166
countries.
A
A
It's
my
first
time,
speaking
at
kubecon,
it's
actually
my
first
time
speaking
in
front
of
a
group,
this
big,
so
please
just
just
bear
with
me
as
we
go
through
the
talk.
My
own
background
is
in
engineering
software
development
still
technically
involved
in
all
of
the
the
projects
that
we're
doing
at
tigera
and
contributing,
but
not
I'm
not
a
security
researcher.
A
Okay,
so
the
goals
today
so
we're
going
to
talk
a
little
bit
about
encryption
as
it
stands
in
kubernetes.
What
are
some
of
the
popular
options
that
people
use
currently
and
why
calico
decided
to
use
wireguard?
So
we'll
recap
a
little
bit
about
what
calico
does
we'll
talk
about
wireguard
and
how
it
works
and
then
how
to
use
mesh
together
to
create
a
fully
encrypted
cluster.
A
We'll
we'll
finish
off
talking
about
some
gotchas.
So
we
didn't
get
everything
right
when
we,
when
we
implemented
this
feature,
there's
still
things
that
we
want
to
do
in
the
future
and
and
look
for
our
contributors
to
help
us
to
do
that
and
there's
some
some
gaps
as
well.
That
we
can
talk
about
so
really.
The
the
thing
with
the
talk
today
is
to
to
see
this
as
a
as
it's
an
alternative
option
to
mtls
with
service
mesh
or
ipsec.
A
So
the
kind
of
story
with
encryption
in
in
kubernetes
today,
what
we
see
when
we
talk
to
our
our
community,
our
users,
the
the
kind
of
number
one
use
case
for
for
using
encryption
is,
is
compliance
actually,
whether
it's
pci
compliance
or
hipaa.
That's
always
the
the
top
of
the
topics.
When
we
talk
to
our
community,
we
also
see
a
lot
of
our
users
talking
about
zero
trust
in
general.
A
So
that
is
a
topic
that
comes
up
a
lot
with
our
users
when
they're
talking
to
us
about
encryption
and
there's
a
recent
nsa
report
from
last
year
about
hardening
kubernetes
and
that
recommends
encrypting
your
data
in
transit.
So
that's
that's
what
we're
talking
about
today!
It's
you
know:
data
in
transit.
It's
not
data
at
rest,
which
is
a
whole
other
topic
in
in
kubernetes.
A
So,
by
default,
when
you
deploy
kubernetes
and
you
deploy
your
workloads
when
those
services
are
communicating
to
each
other,
there's
no
encryption
by
default.
So
everything
is
plain
text,
some
common
options
that
people
reach
for
they
they
can
reach
our
mutual
tls,
which
is
very
popular
encryption,
tcp
and
above
normally
this
comes
with
a
deploying
a
service
mesh,
either
link
or
d
or
istio,
et
cetera
and
there's
so
many
different
options
now
with
with
service
mesh,
and
they
all
offer
an
mtls
solution.
A
Another
option
is
ipsec,
which
is
a
obviously
an
older
technology
kernel
based.
So
so
there's
a
couple
of
different
solutions.
There
are
other
options
and
a
lot
of
our
users
as
well
kind
of
roll,
their
own
certificate
or
tls
implementations,
often
embedded
with
the
applications
themselves.
So
lots
of
different
approaches.
A
So,
just
to
talk
about
mths
a
little
bit
more
in
terms
of
service
mesh.
This
gives
you
client,
server,
authentication,
mutual
authentication,
so
both
the
client
and
the
server
authenticated
together
with
mtls
with
service
mesh.
They
provide
certificate
management,
certificate
rotation
and
they
manage
the
life
cycle
of
their
certificates.
For
you,
different
service
meshes
then,
will
provide
slightly
different
options
when
you're
deploying
encryption
on
top
of
them.
So,
for
example,
istio
allows
you
to
encrypt
a
corner
of
your
cluster,
just
a
specific
part
of
your
cluster
via
namespaces,
which
is
a
very
useful
technique.
A
A
Okay,
so
what
if
you
don't
use
a
service
mesh,
that's
where
we
were
thinking
about
this
feature
and
and
how
we
can
leverage?
What
calico
already
does
for
you
as
a
cni,
especially
at
the
data
plane
level,
and
how
we
can
provide
a
different
encryption
option,
an
alternative
to
ipsec
at
that
lower
level
in
the
tcp
stack.
A
A
A
As
I
said,
it's
power
it
powers
about
2
million
nodes
daily
that
we
know
of
across
166
countries.
So
the
kind
of
purpose
of
talking
about
that
is
to
show
how
battle-hardened
it
is.
So
we
can
leverage
some
of
what
calico
already
does
to
give
us
some,
some
goodness
when
we're
integrating
with
wireguard
one
of
those
the
strongest
points
of
integration
is
how
it
manages
the
data
plane.
A
It
supports
linux
and
ebpf
data
planes,
and
both
of
these
we
want
both
of
these
options.
So
you
can.
You
can
literally
swap
out
your
data
plane
option.
You
can
go
from
linux,
traditional
ip
tables,
based
data
plane
to
ebpf
and
back
again
in
a
deployed
running
cluster
using
calico,
and
we
want
wireguard
to
work
with
both
of
these
technologies.
A
A
This
is,
this
is
a
kind
of
a
cut
down
architecture,
diagram
of
a
part
of
calico,
as
you
can
see
at
the
top
interesting
things
are
changing
in
kubernetes
and
they're
getting
stored
in
calico's
data
store
things
that
we
care
about
and
then
down
in
the
data
plane.
You
have
a
couple
of
calico
components
here:
I've
just
highlighted
a
couple
of
them
felix,
which
runs
in
calico,
node
and
cni
plug-in,
which
interacts
with
the
container
runtime.
A
A
A
A
Okay,
so
how
does
that
relate
to
wireguard?
Why
is
that
important
when
we
started
looking
at
wireguard,
so
why
guards
are
relatively
new
technology?
I
think
most
people
have
at
least
heard
of
our
guard.
I
know
some
people
are
using
wireguard
here
that
I've
spoken
to
already
some
of
its
key
characteristics.
A
A
It
is
positioned
as
an
alternative.
Thank
you
sec
at
layer
3..
One
of
the
key
characteristics
is
that
it's
simple,
so
it's
mostly
transparent
so
operationally
simple.
So
when
you
deploy
and
configure
wire
guard,
it
transparently
handles
symmetric,
key
exchange
and
everything
else
for
you,
heartbeats
and
everything
else.
There's
no
key
management
involved,
it
kind
of
mirrors
the
way
ssh
works
with
key.
So
you
get
a
private
key
in
a
public
key
and
it's
quite
opinionated
that
way,
so
it
makes
no.
A
A
Okay,
so
a
little
bit
more
detail
here
about
wireguard.
So,
as
I
mentioned
before
it,
it
works
by
adding
a
network
interface.
Just
your
standard
networking
interface
and
you
can
configure
that
in
a
normal
way
to
create
an
interface
and
then
you
can
add
and
remove
routes
to
send
traffic
to
that
interface
using
router
ip
route,
so
very
standard
primitives.
A
A
A
So
another
takeaway
here,
which
is
similar
to
calico
wire
guard,
is
really
good
and
and
designed
up
front
to
work
well
with
networking,
primitives
and
linux
utilities.
The
same
way
that
calico
is
so.
As
I
mentioned,
you
might
have
spotted
that
there's
there's
a
natural
mapping
between
pures
and
nodes
and
kubernetes,
and
these
allowed
ips,
which
would
be
a
list
of
workload
pads,
for
example,
okay,
so
we'll
dive
in
a
little
bit
deeper
to
calico,
plus
wire
guard
and
what
wireguard
configuration
looks
like
and
how
it
applies
to
nodes
and
pods.
A
So
what
you're,
looking
at
here
and
on
the
top
you've
got
three
nodes
in
the
cluster
and
node
a
wants
to
send
a
pod
on
node
8
wants
to
send
traffic
to
a
pod
on
node
b,
and
he
wants
to
do
that
over
the
wireguard
tunnel.
He
wants
it
to
be
encrypted
and
down
the
bottom
left
there.
You
have
some
representation
of
a
couple
of
pods
on
this
node
b
in
the
middle
and
node
b
has
an
ip
address
on
the
network,
and
each
pod
has
an
ip
address
as
well.
A
A
A
It's
udp,
so
it's
listening
on
port
51820
and
it
has
a
firewall
mark
which
is
used
for
routing
to
make
sure
we
don't
get
stuck
in
a
loop
when
we
try
to
shove
packets
down
to
the
wireguard
tunnel
and
then
have
it
exit
the
cluster
on
exit,
the
node
on
e0,
the
other
two
parts
of
the
configuration
are
appears,
so
the
middle
one
is
node
b
and
the
bottom
one
is
node
c.
So
you
can
see
here.
A
So
you
imagine,
as
as
you
scale
this
out
and
you
add
more
and
more
pods,
more
and
more
nodes.
This
list
is
going
to
grow
and
grow.
So
this
is
this.
Configuration
needs
to
be
present
on
every
node
in
your
cluster
to
form
that
encrypted
mesh
and
that's
what
that's.
What
calico
does
so
calico
is
programming
and
reprogramming.
This
wire
guide
configuration
the
routing
rules,
the
ip
rules
over
and
over
again
every
time
it
listens
and
gets
an
update
from
the
relevant
things
that
are
changing
kubernetes,
it
reprograms
the
data
plane.
A
So
that's
all
automatic
and
calico
is
really
really
good
at
that.
So,
if
you
kind
of
walk
through
the
the
steps,
there's
a
packet,
that's
meant
for
node
b.
What
happens
is
wireguard
asks?
Okay.
What
here
is
that
it
looks
up
its
configuration
and
it
sees
that
that
ip
address
that
pod
is
represented
by
by
that
middle
peer
id,
which
I'm
not
going
to
try
to
call
it
the
idea,
so
it
encrypts
the
entire
packet,
the
ip
packet
using
the
public
key
for
that
peer,
and
then
it
looks
up
the
remote
endpoint.
A
A
So,
as
I
mentioned
before,
as
we're
scaling
this
out
and
one
of
the
challenges
is
as
as
more
nodes
come
online
and
more
pods
are
spun
up
and
deployed.
They
need
to
be
added
as
as
peers.
First
of
all,
so
each
new
node
is
added
as
appear
through
wireguard
configuration
and
then
the
public
keys
for
that
here
become
part
of
the
node
manifest
itself.
A
So
if
you
want
to
verify
that
a
node
is
configured
with
wireguard,
you
can
simply
get
that
node
using
cubecuttle
and
it'll
have
a
field
which
I'll
show
in
a
second,
which
is
the
wireguard
public
key,
and
you
can
freely
share
that
that
doesn't
need
to
be
kept
secret
or
private
in
any
way,
similar
to
an
ssh
public
key,
the
pod
ips
that
are
deployed
on
top
of
that
node.
They
need
to
become
part
of
the
allowed
ip
list
in
every
other
peer
in
the
in
the
in
the
mesh
in
the
cluster.
A
A
A
A
A
So,
first
of
all,
well,
there's
one
really
interesting
thing:
it's
unencrypted
as
it
leaves
the
pod
and
it's
unencrypted
until
it
reaches
the
network
device
and
if
you
wanted
to
reach
another
part
on
the
same
node,
it's
not
going
to
go
through
the
wire
guard
device.
So
in
that
case,
when
it
hits
the
routing
rules,
there
will
be
a
troll
rule.
A
It
will
not
hit
that
wireguard
device
and
instead
it
will
be
redirected
to
the
other
vet
pair
for
that
pod
on
the
same
node.
So
imagine
a
u
shape
of
green
unencrypted
traffic,
so
pod
depart
traffic
on
the
same.
Node
is
unencrypted
in
this.
In
this
approach
it's
come
up
a
few
times.
It's
not
easy
to
to
fix.
It's
not
easy
to
support
this,
but
we
are
looking
at
it.
A
A
A
A
A
A
But
again
it's
not
a
not
an
easy
problem
to
fix
some
more
pre-flight
checks
is
another
another
topic
that
comes
up
so
when
you
enable
wireguard
using
that
single
command,
you
don't
really
know
if
it's
worked
across
all
of
your
nodes
or
not
so
there's
an
assumption
here
that
wireguard
is
actually
installed
in
the
operating
system
that
you're
running
across
all
your
nodes
in
your
cluster.
So
if
wireguard
is
not
running
in
a
node
across
your
cluster,
let's
say
it's
installed.
A
A
So
the
the
flip
side
of
that
is
similarly
we'd
like
to
give
more
fine
grain
control.
So
right
now
it's
all
or
nothing.
So
you
turn
on
encryption
for
your
entire
cluster.
Now
you
can
go
through
node
by
node
and
disable
it
on
specific
nodes
with
it
with
a
command
again,
if
you
want
to
but
we'd
like
to
do
something
like
this
you're
doing
with
name
namespace-based
encryption
and
offer
like
this
policy-based
encryption,
where
you
could
say
this
corner
of
my
cluster.
A
A
Unfortunately,
so
we
have
a
little
bit
of
work
there
to
do,
and
it's
it's
not
quite
a
simple
case
of
just
enabling
ipv6
supporters,
some
some
quarks
there
as
well
that
we
need
to
figure
out
we've
had
we've
had
some
race
conditions
around
cleaning
up
wire
guard
routing
rules
in
terms
of
the
speed
of
enabling
disabling
wire
guards.
So
if
you're,
if
you're
quite
trigger-happy
with
enabling
wire
guard
and
disabling
it,
there
can
be
some.
A
B
D
A
So,
where
guard
runs
in
the
in
the
kernel,
I
think
if
you
did
try
to
kill
it,
I
think
you
possibly
could
result
in
having
on
encryption
unencrypted
on
that
node
here
felix
doesn't
monitor
the
that
it's
running,
it's
just
responsible
for
the
routing
rules
and
configuration.
So
it's
not
going
to
monitor
the
process
itself.
E
Thank
you
for
that.
For
the
wonderful
talk
I
have
these
questions,
you
said
that's
the
biggest
motivation
for
end-to-end
encryption
is
a
compliance,
but
with
aks
and
eks
implementation
we
see
just
the
traffic
between
two
ports,
so
the
service
port
and
the
the
network
port
is
encrypted
would
just
invalid.
This.
The
whole
system.
A
A
We
have
a
feature
called
host,
a
host
encryption
that
were
rolled
out
for
aks
and
eks
only
and
that
encrypts
everything
so
every
including
traffic
to
the
management
apis,
okay
to
the
master
node.
So
everything
is
encrypted
host
to
host
kind
of
like
ipsec.
You
don't
really
have
a
choice:
it's
not
just
pod
to
pod
and
we
rolled
that
out
on
aks
recently,
and
you
can't
specifically
for
that
reason,
so
that
that
is
covered
in
that
blog
post
I
referenced
yep.
Thank
you.
B
F
First
of
all,
thanks
for
the
talk
you
said
that
you
automatically
generate
the
public
keys
and
private
keys
for
every
node
and
that
you
can
get
these
public
keys.
If
you
do
a
get
to
the
node,
do
I
has
the
administrator
have
to
spread
these
keys
to
all
nodes
or
does
is
that
automatic?
And
if
it's
automatic,
how
do
you
prevent
some
some
keys
to
be
injected
so
another
public
key
getting
injected
that
you
don't
actually
want
in
there.
A
Sure
so
they
it
again
is
this
is
a
very
very
similar
to
how
you
would
manage
your
ssh
keys.
So
in
terms
of
the
first
part
of
your
question,
calico
looks
after
all
of
that,
for
you,
so
you
don't
have
to
manually,
distribute
those
public
keys
and
keep
them
up
to
date
in
your
cluster.
So
calico.
Does
that
part
for
you
using
the
calico
data
store,
is
the
main
kind
of
backing
data
store
for
public
keys?
A
F
A
I
okay,
I
see
what
you
mean,
so
that's
fine,
that's
you!
You
can
do
that.
You'll
also
need
then
to
program
the
allowed
ip
list
and
and
add
a
new
pier
to
the
wireguard
configuration
in
every
node
in
the
cluster.
If
you
want
those
nodes
to
talk
to
your
bad
node,
okay,
so
we
don't
maintain
a
list
of
like
good
nodes
versus
bad
nodes.
So
potentially,
if
you
did
that
you
could
then
you'd
have
to
get
on
to
every
node
you'd
have
to
program.
A
The
wireguard
configuration
to
know
about
your
bad
node
you'd
have
to
add
the
pod
ips
from
your
bad
node
to
the
allowed
ip
list
in
all
of
the
wireguard
configuration.
What
would
happen
is
felix
would
just
overwrite
that
itself
because
it's
not
stored
in
the
data
store.
So
even
if
you
went
and
did
that
manually,
it
would
be
overwritten
fairly
quickly
by
by
felix
okay,
thanks.
G
A
We
don't
support
it
right
now.
That
is
a.
I
should
have
added
that
to
my
future
list
that
has
come
up
before.
Actually
so,
can
you
extend
the
encryption
outside
of
the
cluster
to
some
external
service,
another
cluster
or
or
your
client?
Maybe
your
developer
laptop.
So
that's
something
that
people
are
definitely
definitely
interested
in.
There's
there's
companies
working
specifically
on
that
end-to-end
encryption
using
wireguard,
open
source
companies
working
on
it.
It
has
come
up
from
one
of
our
users,
one
of
our
big
users.
So
it's
something
we
will
have
to
look
at.