►
Description
Securing Kafka with SPIFFE at TransferWise - Jonathan Oddy, Levani Kokhreidze
For a long time in order to achieve mutual TLS between Kafka brokers and its clients we had to use long-lived certificates which is a nightmare to manage at large scale. At TransferWise, we have around 300 microservices and most of them use Kafka for the async communication, stream processing, event sourcing, etc. We wanted to implement Kafka security in a way that reduced the maintenance burden on platform teams, while making migration of diverse clients as simple as possible. In this talk we will describe how we have achieved that goal using SPIFFE with SPIRE and Envoy, requiring zero code changes on the client side.
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
A
Yeah,
my
name
is
lavani,
and
here
with
me
is
jonathan.
We
work
at
transfer
today
we're
going
to
talk
how
we
have
secured
our
kafka
inspire
structure
using
stiffen.
So
first,
I
will
briefly
describe
what
kafka
is.
So
kafka
is
essentially
a
stream
processing
platform.
We
use
it
the
transfers
as
our
kind
of
a
backbone
for
event,
processing
and
most
of
the,
I
think,
cross-service
communications
happen
through
kafka,
so
it's
kind
of
our
central
infrastructure
that
is
very
important
for
transfers.
A
So,
on
the
kafka
side
we
have
different
components
and
on
the
client
side
we
have
a
producers
and
consumers
and
on
the
kind
of
management
and
the
server
side
we
have
a
kafka
broker.
Kaka
broker
is
the
one
who
is
responsible
to
actually
manage
the
clients,
as
well
as
store
the
data
on
the
disks,
so
usually
in
the
production
you
have
many
kapka
brokers
and
on
the
broker
side
we
also
have
the
notion
of
a
topic,
and
the
topic
is
kind
of
a
logical
abstraction.
A
On
top
of
the
data
that
is
stored
on
the
kafka
brokers,
topics
are
split
into
multiple
partitions
and
those
partitions
can
be
on
the
single
broker,
or
they
be
so
they
can
be.
They
can
be
on
multiple
brokers
and
you
and
one
partition
in
this
thing,
and
one
part,
one
broker-
is
a
leader
for
a
single
partition.
A
So,
on
the
broker
side,
we
have
a
configuration
that
is
called
listeners,
and
this
is
where
the
kafka
broker
process
creates
the
server
sockets.
So,
for
example,
in
this
example
in
this
slide,
we
see
that
we
create
new
listener,
called
plain
text,
internal
which
binds
to
the
port
1992,
and
we
also
have
another
listener,
which
is
called
ssl
internal
and
it's
violence
to
the
port
1993.
A
So
for
here
we
are
using
dns
record,
for
example,
kafka
zero,
dot
internal
in
order
to
uniquely
kind
of
reach
to
the
broker,
and
this
is
what
the
clients
will
use
in
order
to
communicate
to
the
single
broker.
Since
clients
need
to
talk
to
their
leaders,
they
have
to
kind
of
reach
to
their
particular
broker.
A
We
also
have
listener
security
protocol
map
and
this
we
kind
of
define
what's
our
security
protocols
for
the
particular
listener.
Here
we
are
saying
that
plaintext
internal
is
a
plain
text,
so
no
security
and
ssl
internal
is.
We
expect
ssl
on
that
listener.
On
the
client
side,
things
are
a
bit
more
boring.
We
just
have
just
bootstrap
servers
and
this
client
uses
this
bootstrap
servers
in
order
to
kind
of
get
cluster
state
so
how
it
works
all
together.
On
the
one
side
we
have
a
broker,
and
then
we
have
a
client.
A
A
A
broker,
on
the
other
hand,
will
return
the
cluster
state
and
it
will
use
advertise
listener,
configuration
for
each
broker
to
move
to,
and
it
will
pass
this
configuration
to
the
client
and
then
client
will
use
this
configuration
in
order
to
identify
the
leaders
and
how
to
reach
the
leaders
of
the
partitions.
So
this
is
kind
of
a
flow
how
the
broker
client
communication
works.
A
Now
we
have
covered
the
like
basics
of
the
connection.
Now,
let's
see
how
the
mutual
tls
can
work
in
a
traditional
manner
for
this
kind
of
setup,
so,
for
example,
we
have,
on
the
other
hand
on
one
hand
we
have
a
client
we,
we
did
some
client
key
store
that
imports
client
certificate,
and
then
we
have
a
broker
with
rocker
key
store
that
imports
broker
certificate
and
they
are
signed
by
the
same
ca
in
this
example.
A
So
how
this
gonna,
how
this
mutual
tls
would
be
established.
First
client
will
make
a
connection
to
the
broker.
Then
broker
would
represent
a
broker
certificate
to
the
client,
since
we
are
talking
about
mutual
tls
client
will
try
to
verify
the
broker
against
the
ca
since
they
are
signed
by
the
same
one.
A
Everything
passes
here
so
we
are
all
set
then,
as
a
fourth
step,
client
will
present
the
client
certificate
to
the
broker
and
the
broker
will
do
the
same
one
and
will
verify
the
client
certificate
engages
against
the
ca
and
when
everything
is
done
as
a
final
step,
the
connection
is
established.
So
this
is
kind
of
a
classical
flow
of
how
a
mutual
tls,
like
simplified
view
of
how
mutual
telus
can
be
established.
A
A
Then
we
have
also
diverse
set
of
clients.
Not
every
client
in
every
language
supports
the
tls
altogether,
so
it
might
not
be
really
possible
to
implement
something
like
this
in
some
languages
and
it's
like.
If
we
are
talking
about
migration
to
this
new
setup,
it
would
be
quite
hard
to
do
if
we
are.
If
we
have
like
that
many
microservices,
you
know
infrastructure,
because
it
would
require
some
of
the
code
changes
on
the
client
side.
A
A
So
let
me
now
describe
how
it
works
so
for
the
scenery
you
mentioned
that
on
the
one
side
we
have
a
micro
service
that
runs
inside
the
kubernetes
and
and
this
microservice
talks
to
the
envoy
as
and
the
envoy
runs
as
a
sidecar
and
envoy
talks
to
the
spire
agent
over
sdc.
So
this
is
kind
of
classical
setup
and
on
the
other
hand,
we
have
a
broker
and
we
need
to
make
this
kind
of
connection
between
microservice
to
the
broker
work
over
invoice
so
over
the
proxy.
A
So
for
the
listeners
we
can
add
that
we
can
define
a
new
listener
named
envoy
and
it
will
bind
to
the
port
1994
and
we
can
define
listeners
for
ny,
which
will
be
localhost
91.01.
Now.
This
is
an
important
bit
here,
because
before
we
have
used,
the
dns
record
or
maybe
ipo
together
to
uniquely
identify
the
broker.
But
now
since
we
just
have
a
local
host
since
envoy,
runs
like
locally
on
the
ssi
car.
Next
to
the
micro
service,
we
need
to
use
different
core
to
uniquely
identify
different
brokers.
A
A
So
for
this
we
are
using
javascript
library,
which
has
a
which,
which
makes
a
connection
to
the
spire
server
using
the
agentsoc
file.
That
is
running
that
is
created
by
spire
agent.
That
runs
on
that
same
host
and
with
that
we
have
kind
of
integration
between
the
broker
and
when
the
and
the
spire
server
done.
So
it
kind
of
works
out
of
the
box
and
we
have
to
well.
A
We
have
to
change
some
configurations,
but
nothing,
that's
serious,
so
we
have
to,
while
by
integrating
javascript
library,
into
the
kapka
broker,
we'll
automatically
get
the
spifi
trust
manager
and
the
key
manager
implementation.
So
we
can
say
to
the
kafka
process
that
use
key
manager,
algorithm
and
trust
manage
algorithm
spf
as
well,
so
it
will
use
trust
manager
and
key
managers,
and
we
also
tell
kapka
to
make
the
client
ssl
authentication
as
required.
A
A
So
it
will
make
the
connection
to
local
envoy
on
as
a
plain
text.
So
it
doesn't
really
need
to
know,
or
it
doesn't
expect
to
talk
to
broker
over
tls.
So
it
just
makes
a
plain
text.
Connection
to
envoy
on
this
port
and
envoy
will
make
a
connection
to
the
kafka
broker
and
it
will
upgrade
the
connection
to
mtls,
because
we
have
spy
regions
running
on
both
on
both
places
and
obviously
aspire
agents
on
this
micro
service
side
and
on
the
broker's
side.
A
B
B
B
So
how
do
we
go
about
adding
kafka
spot
today?
As
well
said,
the
kafka
brokers
are
static,
so
we
didn't
even
need
to
touch
the
control
plane.
Envoy
configs
are
templatized,
so
helm
for
kubernetes
ginger,
two
for
everything
else,
and
so
by
having
that
static
list
of
kafka
brokers
for
each
of
the
clusters,
we
can
just
feed
that
into
the
templating
and
produce
bootstrap
configs,
which
already
know
about
all
the
brokers.
B
Static
configuration
is
not
ideal,
but
it
was
an
easy
way
to
get
started
doing
this
and
get
it
into
production
so
avoid
touching
the
control
plane
at
all
next
slide,
please!
So
what
does
the
conflict?
Do?
This
look
like?
Well,
we
have
a
listener
configured
on
invoice
for
each
broker
that
we
need
to
talk
to
that
listener
has
a
cluster
associated
with
it
for
each
listener.
B
We've
also
got
the
listeners
configured
with
the
kafka
filter.
This
is
completely
optional.
It's
not
essential
at
all.
We
it
all.
It
doesn't
do
anything
for
rooting
traffic.
All
it
does
is
pass
the
messages
they
go
by
and
produce
some
very
basic
metrics
about
messages.
It's
it's
nice
to
have
additional
metrics.
So
that's
where
we
have
it
turned
on,
and
with
this
configuration
you
now
have
a
listener
per
broker,
and
so
your
bootstrap
servers
on
the
client
side
would
look
something
like
localhost,
9101,
localhost,
9102
and
so
on.
B
Next
slide,
please
by
adding
a
another
listener
with
a
cluster
that
has
an
end
point
for
every
single
broker.
So
you
have
this
in
this
example:
a
listener
on
9100.
If
you
connect
that
it
will
connect
you
to
one
of
the
brokers
that
it
knows
about
so
any
one
of
the
brokers
and
that's
enough
bootstrapping.
B
That
broker
will
return
the
cluster
metadata,
which
will
contain
the
advertised
listeners
for
the
rest
of
the
cluster
while
and
itself,
so
that
will
give
the
client
all
of
the
individual
ports
that
it
needed
to
know
to
connect
to
the
individual
brokers.
This
means
the
configuration
is
as
simple
as
having
a
single
port
in
the
bootstrap
servers
for
each
client,
and
it's
now
completely
oblivious
to
how
many
brokers
are
really
behind
each
of
these
bootstrap
ports.
B
That's
right!
So
what
problems
are
with
this?
Well,
the
static
configuration
is
far
from
ideal.
People
want
to
deploy
kafka
into
kubernetes
clusters,
for
example,
and
other
dynamic
environments.
Even
adding
new
brokers
at
the
moment
is
a
pain.
You
have
to
basically
regenerate
a
large
number
of
invoice
configs
and
redeploy
them
it's
not
as
bad
as
having
to
change
the
client
configuration
for
every
single
service,
but
it's
still
not
ideal.
We'll
simply
move
the
configuration
into
the
control
plane
at
some
point
in
the
future.
B
When
this
becomes
a
significant
problem
for
us,
it's
not
a
show
stopper.
We
just
wanted
to
have
the
simplest
sort
of
path
to
getting
this
into
production
that
we
we
could
and
so
avoiding
touch
with
control.
Plane
was
the
best
way
to
do
that
in
terms
of
overhead.
There
were
worries
that
introducing
envoy
into
the
mix
would
would
have
some
high
overhead.
B
In
reality,
we
found
that
offloading
the
tls
work
to
envoy
has
better
performance
than
using
the
java
native
the
java
client
using
tls,
and
that
directly
anyone
who's
dealt
with
the
java
tls
implementation
knows
that
it's
not
known
for
its
performance.
So,
probably
with
hindsight,
this
shouldn't
have
been
a
surprise.
We
actually
have
lower
latency
and
higher
throughput
with
running
it
via
envoy,
for
rather
java,
of
course,
we're
a
java
shop
primarily.
So
this
was
our
main
use
case.
B
That's
what
benchmarked
most
heavily
for
non-java
implementations
we'd
expect
their
own
tls
implementations
to
be
significantly
better
than
the
java
one,
but
in
benchmarks
the
overhead
of
empty,
less
via
envoy,
compared
to
just
plain
text
straight
to
the
broker,
wasn't
actually
that
high
at
all?
I
don't
have
any
numbers,
unfortunately,
but
yeah.
We
found
that
insignificant
enough
for
our
use
case.
That
we
don't
believe
that
there's
a
problem
with
our
languages
either
next
slide.
So
what
did
we
gain
from
doing
all
of
this?
Well:
unified
service
identity
across
our
infrastructure?
B
A
B
For
a
couple
of
languages
and
so
diverse
clients,
not
a
problem
either,
because
all
the
client
libraries
support
plain
text
out
of
the
box,
you
can
just
speak
plain
text
to
envoy
and
it
all
works
and,
of
course,
the
big
win.
No
longer
lives
certificates
to
manage
anymore.
Everything
is
s,
vids
managed
by
spire,
which
was
brilliant,
so
I've
also
included
a
couple
of
resources.
B
There's
an
example
envoy
config.
Well,
a
template
for
generating
example,
invoice
config
in
that
github
repo.
Another
important
component
is
the
kafka
spiffy
principle
library.
We
didn't
write
this,
but
this
converts
this
reads:
the
spiffy
id
of
an
x509s
vid
in
kafka
and
converts
it
to
a
kafka
principle,
so
you
can
then
use
it
in
the
kafka
acls.
That
is
obviously
an
important
bit
of
this
all
and
that's
basically
how
we've
secured
access
to
our
kafka
brokers
using
spiffy
transferwise
any.
C
C
Great
well,
this
is
fascinating.
Spiffy
with
kafka
is
a
very
sought
after
and
desired
integration,
thanks
for
assembling
the
pieces
and
lighting
the
path
for
others
to
be
able
to
refer
to
the
configuration
about
how
you've
gone
about
it
and
also
well,
the
the
insights
you've
gained
throughout
the
process.