►
Description
#IstioCon2021
Presented at IstioCon 2021 by Sebastian Toader & Zsolt Varga.
During the past several years Apache Kafka emerged as the default enterprise message bus. With Istio on its own way to becoming the service mesh “standard” within the enterprise, running a Kafka cluster inside a mesh became a frequent requirement. We’ve been running Kafka over Istio for a few years now, and in this talk, we’d like to share our experience, the common problems and eventually the benefits that led us to make this integration possible. In this talk we’ll be touching on both security and operational benefits such as:
On the fly certificate renewals with no service downtime
Secure cross-regional interaction between workloads and Kafka
Unified simplified configuration to enable mTLS for all components
Single cluster and cross-cluster workload authn/authz of K8s service accounts using Envoy WASM filters
Envoy WASM filters open the gates for a whole array of useful features such as Kafka protocol level metric
A
Most
of
the
solutions
were
based
on
on
stateful
sets
deployed
with
with
hem
charts,
and
we
were
not
that
happy
about
the
capabilities
provided
by
those
twos.
We
thought
that
we
cannot
achieve
all
the
goals
that
we
wanted,
so
we
decided
to
write
an
operator
so
basically
to
write
a
product
on
the
product
that
would
do
the
operation
of
a
apache
calculation
in
such
a
way
that
it
it
provides
something
in
terms
of
stability.
A
It
is
against
the
security
observability
and
disaster
recovery,
so
our
goal
was
to
be
able
to
deploy,
upgrade
so
basically
to
manage
the
whole
life
cycle
for
kafka,
custom,
kubernetes
being
able
to
run
in
parallel,
multiple
kafka
clusters.
If
you
wanted
to
understand
kubernetes
so
assuming
that
the
customer,
the
query
discusses,
is
big
enough.
A
Also,
the
the
security
part
was
an
important
part
there,
so
we
wanted
to
ensure
that
all
the
components
that
are
required
to
run
kafka
talk
to
each
other
using
secure
channels.
So,
basically
encryption
is
using
ssl
the
various
vehicles
here
that
play
a
role
to
be
identified
and
and
also
authorized
to
allow
me
just
to
perform
operations
that
they
should
be
allowed
to
do
now.
A
A
A
A
It
was
not
possible
to
have
a
secure
coalition,
in
fact
entrance
communication
between
kafka
and
zookeeper,
so
in
kafka
2.4.
There
was
also
some
support
for
that
one,
but
because
of
a
bug
that
you
were
not
able,
you
need
to
do
to
enable
that.
So
in
the
end
you
need
it.
You
have
to
pay
dante,
apache,
2.5
and
zookeep
it
to
3.5,
as
as
any
member,
but.
A
Problem
existed
for
for
also
client
applications
written
in
in
certain
languages,
programming
languages
and
not
prepared
to
to
pick
up
configurations
that
enables
mdls.
So
we
wanted
a
solution
that
provides
antennas
also
for
those
kind
of
services
that
are
is
hard
to
enable
and
telescope
is
not
it's
not
capable
at
all
to
do
emptiness
natively.
A
If
they
have
such
certificates,
then
the
question
arises:
how
can
we
renew
this
certificate
and
redistribute
these
certificates
to
everywhere
in
such
a
way
that
we
don't
have
to
restart
the
survey?
Impacted
services
because
we're
starting
impacting
services
may
cause
service
determination
or
service
downtime?
A
So
we
wanted
to
do
something
that
allows
us
replacing
regenerating
and
reviewing
the
certificate
on
the
fly
as
frequently
as
as
we
wanted,
almost
as
we
wanted.
The
kafa
client
applications
usually
use
a
library
that
implements
the
kafka
protocol.
Now
these
applications
can
be
printed
in
different
programming
languages
due
to
that
they
would
expect
these
certificates
in
different
formats,
like
java
applications
expect
these
certificates
to
be
present
in
java,
key
stores
and
trust
stores.
Other
applications
expect
these
certificates
to
be
device
in
pen
format.
A
So
what
we
wanted
is
to
have
a
kind
of
a
unified
way
to
set
up
these
configurations.
So
users,
don't
don't
don't
bother
with
and
trying
to
understand
that
for
every
single,
the
specifics
for
every
single
kind
application
in
order
to
enable
deliver
empty
less
so
that
that
was
one
another.
Another
problem
that
we
wanted
to
tackle
with
since
we're
on
kubernetes.
Also,
we
thought
that
it
would
be
nice
just
to
identify
our
client
applications
by
the
service
account.
A
A
I
mean
that
holds
the
cover
cluster
to
be
the
same
thing
as
deploying
the
same
client
in
a
different
kubernetes
cluster
and
and
and
then
talking
across
the
castle
into
the
into
the
other
one
that
hoses
the
kafka
cluster.
So
from
for
the
for
for
the
using
that
shouldn't
really
matter
where
what
kafka
has
to
what
disgust.
A
A
You
you
need
to
provide
the
server
certificates
for
the
brokers
itself.
You
need
to
provide
certificates
for
the
for
the
workloads
for
the
grant,
applications
on
the
configuration
side,
these
certificates,
and
so
basically
the
keys
and
the
certificates
has
to
be
stored
in
in
in
trust
stores
and
and
key
stores,
which
is
a
in
a
jps
format,
or
it
can
be
a
big
pks
type
format.
In
the
latest
version
of
kafka.
A
Now,
how
can
and
these
keystone
clusters
has
to
be
made
available
to
the
broker
brokers
or
the
kafka
brokers
through
their
in
a
file,
and
those
has
to
be
referenced
in
their
configuration
file
traditionally
on
kubernetes.
What
we
you
do
here
on
conventions
you
have
to
you
can
create
config
maps
or
you
can
create
corrected
secrets,
store
these
these
trash
stores
in
those
secrets
mount
that
into
the
pod
and
that
will
be
available
for
the
containers
in
the
port.
A
A
Basically,
in
in
this
case,
what
we
have
to
do
is
basically
regenerate
certificates,
re
generate
those
keystores
and
clusters
and
and
those
has
to
be
deployed
to
the
to
the
pods.
Now,
assuming
that
these
are
provided
secrets
mounted
into
the
broker
pods,
even
though
we
update
these
kubernetes
secrets
with
the
new
content
of
the
keystone
star
store
since
the
way
how
these
changes
are
being
propagated
into
the
container,
there's
no
guarantee
that
that
these
changes
will
be
visible
right
away,
because
the
way
this
implants.
A
A
So
this
means
that
once
we
originated
these
transfers
and
key
stores,
we
are
required
to
to
bounce
those
those
containers,
those
poles.
In
order
to
pick
up
the
modified
keystone
thruster-
and
this
is
something
that
we
wanted
to
avoid,
because
once
you
you
restart
your
pods,
you
may
raise
that
your
pot
is
not
going
to
able
to
come
up.
A
There
might
be
some
some
some
disruption
notice
in
your
at
your
client
applications
due
to
that
that
when
a
broker
goes
down
even
for
a
short
period
of
time,
it
may
happen
that
during
that
short
period
of
time
there
will
be
new
partitioning,
just
re-elected,
which
causes
all
the
client
applications
that
were
connected
to
the
broker.
That
turned
out
to
be
redirected
to
another
broker
that
just
took
up
the
place
of
the
previously
restarted
broker.
As
a
leader
for
particular
topic,
partitions.
A
A
If
and
if
they
are
running
on
kubernetes,
they
need
these,
these
kind
of
thrusters
or
keystores
or
pamphlets
to
be
available
for
them,
usually
by
a
conflict,
maxwell's
or
or
or
secrets
same
problem,
and
then
this
has
to
be
refreshed
and
and
because,
because
the
certification
was
and
redistributed
to
these
applications
for
each
client
application
that
have
a
different
identity
or
wanting
to
have
a
different
identity.
You
need
to
create
separate
certificates
with
a
different
value
in
that
subject
that
what
I
just
mentioned
before
now.
A
A
In
a
very
elegant
way
for
us-
and
you
will
see
that-
how
we
did
that-
and
I
think
with
that-
I'm
just
hanging
over
to
my
quick
shorts
to
start
talking
about
the
way
we
implemented
and
how
we
overcome
these.
These
problems
that
we
just
mentioned
about
certificate
universe
and
having
also
a
a
kind
of
a
unified
way
to
to
to
to
enable
mtrs
configurations
across
the
board
or
for
all
the
services
that
that
are
making
up
in
such
a
system.
B
B
You
know
a
problem
for
the
client
side
as
well,
because
because
the
the
broker
side
is
kind
of
you
know
within
your
administrative
control,
more
like
the
client
side
because
of
the
various
programming
languages
that
that
those
client
applications
was
built
with.
So
so
so
so.
This
is
why
this
this
unified
layer
as
this
you
can
help
with
that,
and
so
it
it
works.
Obviously,
for
all
the
client
and
the
server
side,
and
besides
the
the
encryption,
it
also
provides
mutual
dls
cls.
B
So
so,
basically
it
also
can
can
help
you
with
the
authentication
side
as
well
and
and
so
the
the
issue
certificates
within
istio.
It's
basically
con
com
contains
a
spiffy
format
id
the
in
the
format
that
you
just
visible
the
slides.
So
basically,
it
contains
information
identity,
basically
about
this
particular
workload
or
more
precisely
about
the
service
account
in
which
name
the
the
developer
is
running
in
and
besides,
you
know
the
the
generic
renewal
functionality,
it's
also
possible.
B
The
next
slide
is
is
that
it's
just
a
a
simple
architecture
diagram
about
the
main
components
of
of
of
this
of
the
kafka.
Basically,
not
every
component
that
that
we
are
running
is
is
on
the
side,
but
but
but
the
gist
is
that
basically,
all
the
brokers
and
and
zookeeper
the
the
brokers
are
using
for
for
storing
all
kinds
of
information
are
all
contains,
an
isteo
proxy
and
the
as
you
can
see,
on
the
on
the
diagram.
B
In
this
case,
the
zookeeper
of
the
broker
is
still
plain
text
and
on
the
broadcast
side
is
basically
really
simplifies
the
configuration
of
the
broker,
because
you
don't
have
to
deal
with
the
things
that
sebastian
mentioned
the
first
quality
of
the
talk,
but
rather
you
have
a
much
easier
configuration
as
this
uses.
Plaintext
settings
basically.
A
B
Yes,
and
as
it
was
also
mentioned,
it
was,
it
was
one
of
our
goals
as
well
to
provide
if
we
can
and
a
transparent,
very
easy
authentication
and
then
this
and
as
a
result
of
it,
authorization
for
the
clients
as
well
and
and
so
now
to
to
give
a
give
a
bit
of
a
detail
on
that.
Is
that
why,
when
the
kafka
is
running
in
a
plain
text
mode
or
let
me
replace
so
when
it
when
it
would
run,
you
know,
make
a
naked
layout
with
a
tns
volt.
B
Then
if
you
get
a
certificate
and
then
it
would,
you
know
authenticate
the
client
based
on
those
certificates,
but
right
now
this
deal
provides
basically
the
certificates
and
it
terminates
the
certificate.
On
the
proxy
side,
and
as
I
mentioned,
kafka
is
running
the
plaintext
mode.
Then
we
have
a
problem
there
to
solve.
B
Is
that
basically
provide
the
information
that
that
you
know
it's
in
the
certificate
and
comes
to
the
proxy
and
transported
to
the
to
the
kafka
broadcast
side
and
because
that's
needed
for
authentication
and
research
is
needed
for
for
the
authorization
as
well?
So
basically,
how
this
works
is
that
the
the
the
communications
flowing
through
the
proxy
to
proxy
terminates
the
tls
connection,
authenticates
the
other
side,
mutually
tls
feature,
and
then
basically,
we
somehow
needed
to
extract
that
information
and
pass
along
to
the
broker.
B
So
the
broker
can
then
do
the
authorization
based
on
based
on
this.
This
client
authentication
information.
So
here
we
can
go
to
the
next
stop,
so
how
basically,
how
we
did
did
that
solve.
That
issue
is
that
we
created
a
blossom
filter
precisely
for
for
for
doing
exactly
what
I,
which
I
just
said
so
basically,
what
happens?
Is
that,
as
you
can
see
on
this
diagram
as
the
client
application,
if
it's
running
you
know
within
the
same
cluster,
for
the
sake
of
simplicity,
it
will
also
get
obviously
a
proxy.
B
It
extracts
that
information
for
the
certificate
and
basically
provides
a
way
to
pass
it
to
the
broker
as
a
client
id
basically,
and
then
the
broker
can
take
that
information
and
do
some
some
authorization
based
on
that
information,
and
obviously
there
are
lots
of
bits
and
pieces
to
do
that.
So
so
we
have
lots
of
custom
resources
that
you
can
use
to
define
those
in
a
kubernetes
cluster.
But
it's
it's
not.
You
know.
B
It
comes
automatically
for
it,
so
you
just
deploy
the
application
and
it's
magically
will
be
able
to
communicate
securely
to
the
broker
and,
besides
that,
it
will
be
authenticated
and
just
able
to
reach
the
the
resources
that
was
on
transferring
and
the
questions
yeah
and
also
as
an
added
benefit.
Or
it
was
one
of
the
goals
as
well.
Is
that
while
we
are,
we
are
actually
having
istio,
then
we
have
a
bunch
of
expertise
of
running
ecu
in
a
multi-cluster
fashion
as
well,
and
it's
it's.
B
B
B
Yeah,
and-
and-
and
so
I
think,
what
we
wanted
to
to
show
here
is
that
is
that
the
istio
as
a
layer,
obviously
it
doesn't
just
provide
security
for
for
for
us
or
in
that
in
that
particular
solution.
You
just
want
to
highlight
one
this
one
particular
area,
but
it
also
can
only.
It
also
provides
the
observability
for
the
communication
as
well
and
all
kinds
of
other
things
that
that
the
that
this
particular
unified
layer
can
can
provide,
and
particularly
for
the
security.