►
From YouTube: Tidy Up Microservices Connectivity with Apache Kafka® and Kuma - Danica Fine & Viktor Gamov
Description
Tidy Up Microservices Connectivity with Apache Kafka® and Kuma - Danica Fine, Confluent & Viktor Gamov, Kong
Given the rising popularity of microservice-based architectures, Kubernetes has solidified itself as one of the most dominant container management systems available on the market. That said, deploying a host of RESTful/HTTP/gRPC services on Kubernetes has not been historically easy or efficient. Enter: service mesh. The service mesh easily facilitates the communication of synchronous microservices over a network. Leveraging an asynchronous communication method – such as that provided by Apache Kafka® messaging – may complicate things. Over the course of this presentation, Danica and Viktor will explore how to deploy Kafka-based microservices – including vanilla Kafka and Kafka Connect components – together with Kuma. We’ll also discuss the benefits and relevance of this particular approach.
A
All
right,
hello,
hello-
and
this
is
how
we
test
the
microphone
right.
Yeah.
Let's
go
okay,
great
great
great,
to
have
everyone
here
and
we
could
just
jump
right
in
this
last
time.
You
can
check
that
so
you're
in
the
right
place
in
the
right
room.
It's
a
service
con,
we're
recording,
let's
go
okay,
so
I
recently
chat
with
danica
on
the
comcast.
When
we
talk
about
her
pet
project
about
the
the
growing
house
plants,
I
have
a
lot
of
houseplants
and
I
come
up
with
this.
A
I
think
interesting
idea
of
how
I
can
explain.
Micro
services
using
house
planes
go
for
it.
So
everything
when
you
start
in
the
building
this,
like
micro
service
everything
starts
like
nice
and
tidy
you're,
putting
a
lot
of
attention
on
it
and
it's
growing
it's
the
first
one,
and
it's
going
well
and
you're
doing
okay.
So
I
think
I
need
to
add
one
more
micro
service
or
one
more
house
point
right
how
many
times
you
have
one
to
you
or.
A
You
start
even
noticing
some
of
the
microservices
are
dying
and
you
don't
know
what
is
going
on
there,
or
some
of
them
may
be
maybe
too
much
of
infrastructure.
Maybe
micro
services
become
too
big
right,
yeah
that
looks
bad
and
over
time.
This
is
how
I
look
every
time
when
I
try
to
explain
some
of
the
or
some
of
my
customers
look
when
they
try
to
explain
their
micro
service
architecture.
They
look
like
a
charlie
day
from
yeah.
C
A
A
You
will
not
understand
half
of
jokes
from
this
talk.
Hopefully,
youtube
will
appreciate
those
jokes
anyway.
I
would
like
to
introduce
my
speaker
danica
fine.
She
is
a
senior
developer
advocate
at
confluent,
and
I
am
victor
gamov
developer
advocate
at
kong
and
today
we're
going
to
be
talking
about
all
things
kafka
why
we
are
good
people
to
talk
about
this.
A
B
Make
sure
it
works,
I'm
much
quieter
than
victor,
so
all
right,
we're
good.
I
think
I
think
we're
good
all
right
yeah,
so
I'm
danica-
I
personally
I
mean
victor,
might
look
like
charlie
when
he
talks
about
these
things
or
may
not
want
to
look
like
charlie,
I
kind
of
embrace
it.
I
really
enjoy
that
aspect
of
it,
but
we're
talking
about
microservices
here.
B
So
let's
get
a
little
serious,
so
yeah
you're,
probably
all
here,
because
you
understand
that
using
microservices
or
architecting
your
system
that
way
makes
sense
as
opposed
to
a
monolithic
sort
of
architecture
right.
It
makes
sense
to
break
down
your
application
into
a
smaller,
smaller
pieces
that
are
more
manageable,
right,
you're,
more
flexible
you're,
giving
the
the
individual
teams
and
business
units
control
over
the
applications
that
they're
building
and
only
the
stuff
that
they
care
about.
B
A
B
Get
to
it
all
right,
so
you're,
probably
thinking
you
know
when
you
start
thinking
about
integrating
microservices
you're
going
to
encounter
this
problem
right,
you
probably
get
frustrated
and
there's
not
really
a
good
way
to
to
integrate
them
right,
but
in
reality
you
know
there
are
a
lot
of
options
for
you,
but
there's
just
a
lot
of
bad
ones.
Okay,
so
you
know
there
are
a
ton
of
options
available
to
you.
Some
are
better
than
others.
Some
are
more
intuitive
than
others,
so,
let's
jump
into
those
so
first
of
all
file
system.
How.
A
A
There
yeah
we've
been
there.
It's
called
enterprise
software.
This
is
how
we
were
building
this
for
years.
Yeah.
We
shouldn't
be
doing
about
databases.
How
many
of
you
build
a
system
that
communicate
or
pushing
information
through
databases?
It's
okay,
it's
a
good
place
to
share
about
this.
It's
no
shame
in
this
room,
so
we
can
talk
about
this
right.
B
Mean
databases
make
sense
right.
I
mean
if
you're
moving
from
a
monolithic
sort
of
architecture
to
microservices
I
mean
in
a
model
if
you
definitely
had
at
least
one
database,
probably
more
like
12
hanging
around
right.
So
when
you're
breaking
down
that
monolith
and
you're
looking
to
integrate
your
microservices,
you
probably
still
have
a
database
sitting
around
and
you
definitely
have
at
least
one
engineer
who
knows
how
to
use
that
database.
So
why
not
use
the
database
to
communicate.
A
And
the
reason
why
not
use
database
database
is
great
for
storing
the
state
for
individual
service,
but
for
integrating
and
when
the
services
start
commingled
to
each
other.
You
very
quickly
end
up
in
situation
where
you
need
to
have
a
committee
of
approvals
changes
in
database.
You
were
changing
the
schemas.
You
need
to
notify
others
that
are
you
changing
the
schema
of
this
database
and
very
quickly?
It's
not
it's
not
microservices
anymore.
It's
ended
up
in
this
like
analytical
service
monitor
mess
in.
B
A
B
A
A
B
B
Oh,
we
have
our
microservices
right,
but
now,
instead
of
you
know,
in
the
monolithic
case,
we
just
had
everything
packaged
neatly
in
one
application
right,
it
was
good,
it
was
cozy,
you
know,
but
now
we're
kind
of
we're
making
these
network
calls
we're
reaching
out
to
the
ether,
hoping
that
you
know
someone
else
will
respond
and
if
they
don't,
you
know
we're
leaving
25
voicemails,
trying
to
figure
it
out.
So
now
we
have
all
these
requests
in
flight
and
it's
it's
kind
of
a
mess.
Now
it's
it's
tangled
again
right.
A
Another
problem
here
would
be
like
cascading
failures,
yeah.
If
one
of
the
servers
will
fail-
and
you
continue
to
like
hammering
another
service-
nothing
good
will
happen
so
question
to
you,
my
dear
friends
developers
how
you
would
investigate
the
problem
in
the
system
if
there
are
some
of
the
failures.
What's
your
typical
things
to
do.
B
Logs-
and
you
know
this-
this
makes
sense
we're
all
developers
here.
We
know
how
to
debug
applications,
you
know
tracing
and
logs.
This
makes
sense,
it's
a
good
way
to
see
into
systems.
But
what,
if
okay
and
hear
me
out
what?
If
we
just
let
our
microservices
communicate
using
a
log?
What
if
we
used-
and
this
is
no
secret-
what
if
we
use
kafka?
Okay,
we're
we're
not
here
to
argue
that
kafka
could
be
a
good
choice
for
this.
B
We
both
love
kafka,
obviously
so,
when
you're
using
kafka
to
sit
as
that
communication
layer
between
your
microservices
you're
gonna
get
something
like
this
and
it's
more
loosely
coupled
right.
This
is
good.
Now,
all
the
microservices
are
sending
information
to
kafka
and
can
communicate
through
that,
and
for
those
of
you
how
many,
how
many
of
you
are
familiar
with
kafka?
B
It's
a
streaming
system
right
so
real
time
we're
going
to
be
communicating
information
quickly
and
efficiently
across
the
system
and
it's
an
event
streaming
system.
Okay.
So
for
those
who
aren't
familiar
with
events,
they
are
immutable.
Okay,
we're
communicating
immutable
facts
across
our
system
quickly
and
efficiently,
and
that
is
the
perfect
way
to
communicate
between
microservices.
Similarly,.
A
Similarly,
how
we
talk
about
the
voicemail
when
you
left
the
voicemail,
you
cannot
change
it.
So,
similarly,
how
you
have
in
conversation
with
your
significant
other,
and
you
said
something
that
you're
not
supposed
to
say
this,
only
thing
how
you
can
do
this,
if
you
send
another
event,
and
hopefully
the
receiving
system
will
process
it
in
order
to
change
the
state
right
yeah.
So
this
is
how
you
can
you
know,
explain
immutability
to
to
other
people,
so
essentially
something
you
cannot
change
now.
A
Communication
in
our
systems
will
become
more
entangled
and
using
the
kafka
as
a
log
for
capturing
all
the
facts
around
the
system.
We
can
do
multiple,
interesting
things
about
this,
for
example,
this
voicemail
that
you
left
me
might
not
only
be
received
by
me.
Someone
else
can
also
listen
this
event
and
also
make
some
of
the
some
of
the
actions
on
this
one
same
thing
with
kafka.
A
Some
of
the
message
will
be
consumed
by
multiple
consumers
and
they
may
have
different
reaction
on
those
consumers,
so
in
this
case
we're
building
a
synchronous
communication
for
for
this.
By
the
way,
this
is
the
one
of
the
patterns
that
you
can
find.
This
website
called
developer.confluent.io
how
you
can
build
event
driven
systems
and
how
you
apply
kafka
for
the
things
and
if
you
have
urge
to
tweet
about
the
slides,
the
our
twitter
tags
are
conveniently
placed
in
the
in
the
bottom.
So
please,
you
know,
don't
do
not
hesitate.
A
So
we
already
started
talking
about
kafka
but,
like
let's
talk
about
some
of
the
technicalities
here
right,
so
how
I
like
to
explain
that
kafka
is
a
communication
layer.
It's
a
data,
aware
system,
so
you
allow
to
build
a
communication.
That's
data,
aware
you're,
sharing
facts
about
the
systems
that
other
sims
will
understand.
You're,
sharing
the
order,
details
and
some
other
systems
based
on
some
of
the
schemas
and
some
of
the
information
that
available
for
for
the
message
can
understand.
So.
A
A
A
Yes,
I
I
knew
this
and
that's
why
there
are
some
automation
exists
that
allows
you
to
to
to
run
this
successfully.
There
is
an
open
source
operator
from
from
streamsee,
which
is
also
cncf
project.
There
is
some
proprietary
operators
from
confluent,
and
some
some
others,
but
idea
is
to
bring
some
of
the
system
that
was
designed
pre-cloud
native
times
into
the
cloud
native
worlds
requires
some
of
these
skills
and
those
skills
usually
tied
into
into
human
sres.
A
Who
knows
how
to
run
a
system
like
kafka
but
also
know
how
to
run
stateful
systems
in
kubernetes
and
there's
like
in
this
in
this
slide,
it's
like
a
very
simple
explain
of
architecture
of
kafka.
So
what
I
like
to
say
in
this
particular
case
we're
running
one
distributed
system,
which
is
kafka.
That
runs,
depends
on
another
distributed
systems
which
is
zookeeper
on
top
of
another
distributed
systems
system
which
is
kubernetes.
So
you
know
many
things
can
go
wrong
right,
yeah,
pretty
meta
and
things
around
the
zookeeper
and
kafka.
A
It's
good
that
many
people
are
familiar
with
the
concept,
so
I
don't
need
to
go
very
deeply
on
this
one,
but
like
if
people
in
youtube
watching
us
they
in
order
to
kafka
cluster,
to
work,
it
will
depend
on
the
cluster
of
zookeeper
and
zookeeper
requires
to
establish
the
quorum
in
order.
Your
data,
metadata
that
will
be
stored
in
zookeeper,
will
be
available
for
kafka
broker
to
serve
same
thing
for
kafka,
so
also
they
need
to
for
form
a
query
in
order
to
form
a
quorum.
A
They
need
to
have
us
what
we
call
the
stateful
network
identity,
so
each
broker
each
member
of
the
cluster,
will
need
to
have
a
particular
name.
So
pods
with
your
deployment
with
some
random
name
will
not
will
not
would
not
fly
here.
We
need
to
use
something
else,
so
we
will
be
using
things
like
stateful
sets
that
provides
us
stateful
network
identity.
A
A
So
the
good
thing
that
good
people
in
the
kafka
community
work
very
tirelessly
to
simplify
this
and
make
kafka
more
cloud
native
and
the
things
with
the
keep
500
keep
is
the
way
how
kafka
community
communicates
keeps
a
kafka
improvement
proposal.
It's
the
some
of
the
some
of
the
things
that
you
can
google
and
you
can
find
full
description
how
this
work,
but
essentially
now
we
don't
need
zookeeper
and
we're
going
to
be
running
this.
Our
kafka
in
zookeeperless,
small
right.
B
A
So,
with
the
with
the
kafka,
when
we
need
to
communicate
this,
your
application,
producer,
consumer
or
kafka
streams
application,
they
need
to
know
at
least
one
url,
which
is
bootstrap
server
and
in
order
to
connect
and
learn
some
metadata
about
the
cluster.
Remember
I
mentioned
that
kafka
distributes
data.
So
all
this
data
that's
spread
across
kafka
broker.
A
We
use
this
the
the
mechanism
that
very,
very
briefly,
can
be
described
as
a
consistent
hashing
mechanism,
so
based
on
the
key
of
the
data
our
producer
or
consumer,
will
decide
where
to
read
or
where
to
consume
or
where
to
produ
from
where
to
consume,
where
to
produce
data
and
bootstrap
server
will
give
us
information
about
overall
brokers.
I
don't
need
to
put
all
the
brokers
into
this
bootstrap
server,
just
only
one,
but
this
only
one
broker
needs
to
provide
me
with
information
about
other
brokers.
B
Okay,
cool:
I
can
deal
with
that.
That
sounds
good
all
right.
What
about
security?
That's
probably
going
to
come
up
right
because
in
a
monolith
we
it's
built
in
a
very
specific
way
that
it's
super
secure
on
the
outside
right
and
then
once
information
is
past
that
once
we've
proven
that
it's
not
malicious,
they
can
do
whatever
they
want
right
at
the
microservice
level,
we're
still
probably
within
the
same
network.
So
should
we
just
trust
that
it's
okay
right,
because
we're
in
the
same
network,
probably
not
zero
trust-
is
maybe
a
little
better.
B
So
we
should
take
that
sort
of
mechanism
that
we're
using
at
the
monolith
level
and
maybe
shrink
it
down
to
the
micro
service
level.
You
know
trust
nothing
until
it's
validated
and
comes
in
then,
but
what
did
the
individual
so
we're?
Breaking
these
microservices
and
now
teams
own
the
different
business
units
and
applications
specific
to
them.
Are
they
responsible
for
writing
all
that
boilerplate
code
to
make
sure
that
the
security
happens
as
it
should.
A
Yeah,
I
think
it's
also
another
point
to
to
to
bring
up
that
when
the
kafka
was
designed.
There
was
no
again
cloud
native
approach
to
solving
the
problem,
so
the
system
needs
to
come
with
battery
included.
So
kafka
naturally
has
all
things
for
you
to
build
a
secure
connection.
There's
acls
there's
a
different
way
how
you
can
securely
connect.
However,
however,
this
thing
actually
comes
with
the
price.
Oh
yeah,
do
you
familiar
with
the
concept
of
zero
copy.
B
A
Lessons
up,
like
all
the
the
reason
why
people
hate
java,
because
they
heard
about
garbage,
collector
and
usually
people,
think
that
javascript
is
bad.
You
don't
want
to
something
that
you
know
you
will
rely
on
some
slas
and
something
would
be
standing
in
front
of
your
system
and,
like
all
right
this,
even
though,
over
the
years,
the
the
encryption
and
the
the
the
mechanism
of
encrypting
the
stuff
in
kafka
in
java
become
like
more
and
more
convenient,
but
some
of
the
things
you
cannot
change.
A
So
when
the
data
arrive
over
the
socket
in
the
encrypted
circuit,
it
needs
to
be
decrypted
and
only
after
it
will
be
flushed
into
the
file.
Where
is
we're
using
plain
text,
this
data
from
socket
will
be
flashed
into
the
file
immediately.
So
this
is
something
we
can
work
on.
This
is
something
that
we
actually
can
can
solve
with
the
with
the
solution
that
I
will
propose
in
a
few
seconds.
A
It's
still
time,
it's
still
time
to
talk
about
the
service,
so
the
interesting
enough
service
mesh
provides
the
very
there
at
least
like
a
service
mesh.
This
current
generation
provides
the
very
nice
ways
how
developers
need
to
think
about
infrastructure,
specifically
focusing
on
the
value
and
focusing
on
the
business
logic
rather
than
focusing
infrastructure,
like
many
developers
knew
that
how
to
write
a,
I
don't
know
code
for
their
particular
language,
their
particular
framework
that
will
does
like
encryption
and
all
this
type
of
jazz
right.
But
what,
if
you
don't
need
to
what?
A
If
the
infrastructure
will
give,
you,
like,
you
said
like
zero
trust
promises
already
and
will
handle
all
this
automation.
Kafka
sres
have
a
years
of
experience
of
building
this
type
of
solution,
but
these
solutions
need
to
be
properly
automated.
It
needs
to
be
like,
like
one
of
the
characters
from
of
the
movie,
that
I
really
like
dark
knight
once
said
when
you're
doing
something
good,
never
do
it
for
free.
So
that's
why
the
kafka
series
would
be
expensive
or
the
tools
that
do
automation,
for
you
would
be
expensive
right,
like
tweet
at
me.
A
If
you
get
the
reference
who
say
that
so
we're
gonna
be
talking
about,
like
very
small
use
case,
very
small
use
case,
that
explains
how
the
things
works
and
the
small
it's
not
because,
like
I
cannot
do
bigger
the
the
small
just
to
explain
the
lots
of
things
going
on
here.
So
we
have
our
producer.
That
will
be,
I
don't
know
some
system
that
will
be
capturing
orders
from
the
web
and
of
that
published
into
into
kafka.
A
A
B
A
It
was
not
that
long
and
not
that
awkward,
okay,
so
again,
idea
of
bringing
this
thing
together
will
be
tied
into
couple
things.
So
I'm
running
this
small
kafka
cluster
with
the
three
nodes
and
one
of
these
nodes
will
will
take
the
irresponsibility
of
a
leader
and
because
we're
not
running
the
zookeeper,
so
it's
nice,
nice
and
cool.
We
have
some
consumer
producers
so
how
this
happened
to
be,
and
this
is
where
we're
going
to
look
into
configuration.
A
So
we
I
deploy
this
with
with
simple
stateful
set.
There
is
nothing
really
specific
here,
so
you
can
use
any.
You
can
use
custom
image.
You
can
use
image
that
comes
from
the
vendor.
Some
open
source
images
really
doesn't
matter
so
to
show
you
there's
no
like
other
magic,
so
the
stateful
set
that
will
be
responsible
here
is
that
we'll
have
this
nasty
script
that
built
in
yemen.
I
personally
hate
this.
A
That's
why
I
more
like
operators
approach,
but
in
order
to
explain
the
bits
I
need
to
go
like
you
know,
dirty,
so
a
couple
things
that
I
need
to
build
here,
and
this
is
the
probably
the
most
important
thing
that
everyone
will
be
care
about.
So
in
order
to
connect
clients
to
kafka
broker,
we
need
to
provide
so-called
advertised
listeners.
This
is
something
that
bootstrap
server
will
need
to
provide
to
clients
in
order
these
clients
to
connect.
So
essentially
think
about
this.
It's
kind
of
like
a
it's
a
virtual.
A
It
needs
to
be
some
virtual
ip.
It
can
be
virtual
ip.
It
can
be
like
external
network,
if
you're
exposing
this
to
outside
world
through
the
load
balancer,
it's
going
to
be
a
load,
balancer
url.
In
this
particular
case.
We
have
this
like
a
cryptic,
cryptic,
conventional
url.
One
of
the
things
that
we
will
be
relying
here
is
the
service
discovery
that
comes
from
from
service
mesh.
A
We're
gonna
be
deploying
this
in
the
service
mesh
with
kuma,
and
this
part
will
be
provided
the
dot
mesh
discovery
will
be
provided
for
kuma,
but
there's
a
couple
things.
This
is
basically
conventional.
So
it's
nothing
to
do
with
the
kuma
here.
In
this
case,
we
want
this.
We
want
this
cluster
to
be
exposed
and
available
through
this
particular
url,
and
the
way
how
we
will
doing
this
with
with
the
kuma
is,
with
the
thing
called
virtual
outbound.
A
A
It's
essentially
we'll
create
the
the
service
that
will
create
addressable
thing,
and
I
can
customize
this
name
on
the
virtual
inbound
and
I
not
need
to
touch
my
kafka
deployment
for
second
thing
in
order
to
like
once,
the
initial
communication
happened
like
bootstrap
server
already
provided
this
connection,
other
services
needs
to
be
available
so
advertise
listener.
This
is
something
that
this
bootstrap
server
returns.
The
metadata
to
kafka,
client
and
information
to
the
the
to
those
kafka
brokers
would
look
like
a
cluster
1-0
cluster
1-1
cluster
1-2.
A
That's
going
to
be
kind
of
like
a
name
of
the
broker,
and
this
information
also
needs
to
be
available
to
to
outside
of
these
oh
inside
this
inside
this
mesh.
So
for
each
member
in
this
in
this
list,
we're
going
to
return
a
particular
particular
url,
so
for
our
client
for
our
application.
One
only
thing.
What
we
need
to
do
is
here
is
provide
this
one.
A
Now
the
next
thing
that
we
also
a
little
bit
touch
this
a
little
bit
about
what
else
the
service
mesh
can
give
us
immediately.
It
will
gives
us
a
ability
to
get
to
the
logs,
and
this
is
where
simply
by
creating
a
the
traffic
logs
policy
that
enables
me
to
capture
information
from
all
from
all
data
planes.
I
will
be
able
to
see,
for
example,
let
me
do
this
a
little
bit
bigger,
so
I
can
go
inside
my
log
for
consumer
and
see
what
is
going
on
in
in
the
consumer.
A
So
right
now
my
consumer
just
reads
the
data
from
reads
the
data
from
the
the
topic
and
just
speeds
out.
This
result
back
to
back
to
topic.
So
we
can
see
all
this
kind
of
information
and
all
this
done
just
by
simply
enabling
I
don't
need
to
put
any
special
things
on
my
brokers
since
the
broker's
already
running
inside
the
match.
I
will
have
these
things
out
of
the
box,
so
this
is
where
we're
gonna
switch
to
do.
We
have
a
like
a
final
slide.
B
B
A
Kafka
needs
to
send
some
of
the
information
to
the
brokers,
saying
hey,
I'm
gracefully
shutting
down,
because
because
of
what
not
what
is
happening
there
and
in
order
to
do
that,
I
need
to
do
some.
Some
sort
of
hacks
like
like
a
pre-shutdown
hooks
and
writing
some
shell
scripts,
which
is
kind
of
possible
to
do,
and
many
of
us
already
familiar
with
this,
but
it's
kind
of
nice.
It
feels
weird.
A
So
hopefully
this
will
be
resolved
at
some
point
where
we'll
have
like
a
sidecar,
a
sidecar
life
cycle,
we
didn't
touch
much
of
the
aspect
of
the
looking
a
little
bit
different
inside
the
traffic,
for
example.
Envoy
has
a
filter
that
is
available
to
look
deeper
inside
the
kafka
traffic,
so
we
can,
for
example,
collect
information
about
the
clients.
A
So
those
type
of
things
also
important
and,
more
importantly,
why
I,
like
personally
the
operator
approach
in
in
this
case
the
operator
approach,
can
can
programmatically
work
with
some
of
the
things
around
kafka.
For
example,
you
don't
want
to
restart
your
kafka
cluster
if,
for
example,
your
something
helpful
with
your
controller,
like
you,
don't
want
to
start
your
controller
like
arbitrary.
You
want
to
restart
your
controller,
the
last
in
this
case
you
will
not
waste
the
time
on
the
automation
but
anyway.
A
B
A
C
A
Yes,
yes,
so
the
the
kafka
kafka
brokers
runs
inside
and
the
sidecar
provides
us.
Oh,
I
didn't
show
mtls,
so
the
the
brokers
communicate
through
the
mtls
and
yeah
so
clients
and
the
server
they
don't
know
that
they
actually
talking
to
each
other
through
a
secure
channel.
That's
that's!
That's
pretty
huge
and.