►
From YouTube: Kafka on Kubernetes - From Evaluation to Production at Intuit - Shrinand Javadekar, Intuit
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Kafka on Kubernetes - From Evaluation to Production at Intuit - Shrinand Javadekar, Intuit
Kubernetes is fast becoming the platform of choice for running distributed, containerized applications in the cloud. However, running stateful applications such as Kafka on Kubernetes is not a common practice today. At Intuit, we took an experimentation and data driven approach for evaluating Kafka on Kubernetes in AWS. In this talk, we will provide details of our requirements, the configuration, and the details of the evaluation. The evaluation process included functional tests, network isolation tests, cross-region tests as well as performance and stress tests. We will focus on the problems we ran into and how we addressed them. This talk will demonstrate a K8S cluster running Kafka and it's configuration using Kubernetes native components (statefulsets, configmaps, etc.). We will also show Splunk and Wavefront integrations for logging and monitoring respectively.
To learn more: https://sched.co/GrW5
A
Hi
everyone,
my
name,
is
Sri
Zhao
Decker
I
am
a
software
engineer
at
Intuit
and
I'm
going
to
talk
today
about
our
journey
of
running
Apache
Kafka
on
kubernetes,
from
evaluation
to
production,
so
I
quickly
go
over
the
agenda
of
the
talk,
we'll
quickly
talk
about
why
we
decided
to
use
Kafka.
Why
did
we
choose
to
run
Kafka
on
kubernetes
or
why
did
we
think
about
kubernetes
as
the
kind
of
substrate
for
running
Kafka
Kafka
on
kubernetes
architectures?
So
how
did
we
think
about
this?
A
What
kubernetes
objects
and
resources
come
into
play
when
we
are
trying
to
run
Kafka
on
cover
IDs
and
then
I'll
go
into
details
of
the
architecture
itself,
I'll
try
and
focus
on
the
problems
as
much
as
possible,
because
problems
are
going
to
be
common
across
all
of
us,
depending
on
where
environments
and
SaaS
and
everything
else,
the
solutions
to
those
problems
might
be
different.
It's
also
oft
where
there
are
500
ways
to
solve
every
problem,
so
why
Kafka
so
within
into
it?
A
I
work
for
an
organization
or
a
team
called
QuickBooks
Intuit
QuickBooks
is
a
monolithic
application
for
that
is
used
by
millions
of
customers.
Small
business
customers,
these
small
businesses,
use
QuickBooks
for
accounting,
payroll
and
other
financial
management
and
QuickBooks
is
a
fairly
complex
application.
It's
been
around
for
several
years
now
and
it
has
grown
organically
and
it
is.
It
has
tightly
coupled
modules,
as
you
know,
as
time
has
gone
by,
it
has
like
you
know,
gathered
a
lot
of
interesting
functionalities,
but
it
has
become
harder
to
scale.
A
How
are
these
modules
or
an
individual
micro
services
going
to
be
able
to
talk
to
each
other,
and
the
goal
therefore
was
to
use
Apache
kafka
for
doing
this,
so
Kafka
kind
of
becomes
this
central
nervous
system
which
connects
all
these
individual
components
and
micro
services
and
they
are
able
to
send
their
data
onto
the
Kafka
bus,
so
at
different
topics
subscribers
can
subscribe
to
it.
There
could
be
more
than
one
subscribers
micro-services,
don't
need
to
know
about
who
else
is
running
in
the
system?
A
Why
kubernetes
for
Kafka
so
I
mean,
of
course
there
is,
you
know
we
all
benefit
from
kubernetes.
You
know
we
have
job
security
if
there
is
kubernetes
everywhere,
but
but
but
beyond
that,
the
point
was
kubernetes
is
becoming
the
de
facto
standard
for
running
containerized
applications.
All
the
kubernetes
talk
that
I
am
giving
you
today,
or
everything
that
I'm
talking
about
today
is
strictly
about
running
kubernetes
on
AWS,
so
in
kubernetes
integrates
very
nicely
with
AWS.
It's
very
nicely
it's
aware
of
AWS.
A
It
has
all
good
features
for
integrating
with
other
AWS
resources,
which
is
really
good,
and
then
there
are
several
constructs
within
kubernetes
which
are
great
for
kafka.
Things
such
as
stateful
sets
where
you
have
a
pod
and
you
have
a
volume
together
as
one
unit,
and
then
you
can
take
this
one
unit
and
create
n
replicas
of
this.
You
can
have
this
one
unit
of
a
pod
and
a
PVC
spread
across
multiple
zones.
A
Have
them
have
them
run
in
a
configuration
where
they
can
always
be
together,
and
things
like
that
so
stateful
sets
were
great
configuration.
Maps
Kafka,
like
many
other
applications,
is
a
fairly
complex
application.
It
has
a
lot
of
configuration
that
goes
with
it.
You
want
this
configuration
to
shared
across
all
the
brokers.
Config
Maps,
give
you
the
ability
to
create
one
config
map
and
mount
it
inside
all
your
brokers
or
all
your
n
replicas
get
the
same
configuration
secrets,
so
SSL
was
important
for
us.
A
So
there
are
some
parts
of
Kafka
which
was
super
interesting
from
interesting
from
from
an
architectural
point
of
view
where
we
did
not
want
to
kafka
brokers
to
land
up
on
the
same
ec2
instance,
we
did
not
want
a
Kafka
broker
and
the
zookeeper
instance
to
land
up
on
the
same
ec2
instance.
So
we
wanted
to
kind
of
spread
them
apart
for
availability
reasons.
But
how
do
you
express
that
such
that
it
actually
is
guaranteed
that
no
such
pod
will
actually
land
up
on
the
same
node?
A
We
use
pod
affinity
or
pod
ante
affinity,
and
some
node
affinities
fault
achieving
this.
So
I'll
go
over
that
also
in
a
little
more
detail,
but
these
kind
of
basic
constructs
that
are
fundamental
to
the
way
kubernetes
is
made
it
made
us.
The
kind
of
you
know
think
about
think
very
seriously
about
why
we
should
use
kubernetes
as
a
substrate
for
running
Kafka
and
then
last,
but
an
extremely
important
point
about
extensibility
offered
by
containerized
environments.
A
A
A
So,
like
all
good
engineers,
we
looked
at
Stack
Overflow,
sorry,
not
exactly
Stack
Overflow,
but
but
we
did
a
Google
search
and-
and
we
found
out
that
there
are
many
other
people
who
think
similarly-
and
we
found
a
bunch
of
these
interesting
articles
and
github
repos,
where
people
talk
about
running
Kafka
on
kubernetes,
so
specifically
among
them,
the
one
from
you
lean,
it's
a
github
repo,
the
slides
will
be
made
available.
You
can
find
that.
But
if
you
search
for
your
lean,
Kafka
kubernetes
will
pretty
much
get.
There
was
really
good.
A
If
there
is
someone
here
by
the
way
from
yo
lean,
please
get
in
touch
with
me.
I
owe
you
a
drink
or
coffee
or
I
know
it's
Christmas
season.
Maybe
you
know
gingerbread
cookies
for
all
UK,
but
this
was
great.
We
found
this
repo.
It
was
exactly
along
the
lines
of
what
we
wanted.
It
was
giving
us
a
multi
zone
or
if
I
had
a
kubernetes
cluster
which
spanned
across
multiple
zones,
it
was
giving
us
Kafka
as
a
stateful
set
it
trans
do
keeper.
A
As
a
stateful
said,
it
was
using
config
maps
for
the
server
dot
properties
file,
which
is
what
the
configuration
for
Kafka
is
and,
interestingly
enough,
it
also
had
shell
scripts
for
actually
running
these
brokers.
So
this
was
a
little
interesting
because,
fundamentally
like
when
you
want
to
run
a
broker
in
Kafka,
it
is
a
Java
command.
It's
like
Java
dash,
M
and
a
bunch
of
jars,
and
then
you
know
some
other
arguments
that
go
with
it.
A
Instead
of
just
running
that
command
in
the
container,
it
actually
had
a
shell
script
and
this
command
was
run,
but
it
also
provided
a
way
in
which
these
shell
scripts
could
be.
You
know
customized
for
our
use
cases,
so
it
was
really
really
good
that
way.
So
we
took
these
shell
scripts.
We
took
this
repo,
we
said:
okay,
let's
see
if
this
even
that
works.
So
we
put
this
on
a
cluster
here,
and
this
is
basically
all
of
these
were
ml
files.
A
So
all
we
had
to
do
was
take
them
and
do
a
cube
serial
apply
and
everything
was
running.
Fine.
We
did
our
basic
tests.
You
know
we
ran,
we
created
a
topic.
We
tried
to
write
to
a
topic.
It
was
great
we'd
rather
write
from
the
topic.
It
was
working
fine,
so
it
seemed
really
really
stable.
We
did
a
little
more
than
that
where
he
said.
Okay,
let
me
kill
a
pod
and
see
if
it
comes
back
up,
it
was
working.
Fine.
A
Let
me
kill
an
ec2
instance
and
see
if,
like
the
auto
scaling
group
in
AWS
brings
the
instance
back.
Does
it
go
back
and
join
the
cluster
and
does
it
does
Kafka
as
software
recover
from
all
of
this
and
work
fine,
and
it
was
really
working
well,
so
we
were
like
okay,
this
isn't
sounds
very
promising.
We
need
to
show
that
we
have
to
do
more
work.
Otherwise,
again
you
know
job
security.
It
has
to
go
on
for
awhile.
So
so
we
decided
to
do
a
little
bit
of
a
baseline
experiment
with
this
look.
A
What
if
we
take
this
set
up?
It
is
very
rudimentary
in
some
sense
it's
not
production
ready.
It
was
kind
of
day.
You
know
five
or
ten
of
the
getting
off
the
ground.
What
do
we
do
in
terms
of
kind
of
finding
where
we
are
at
so
we
decided
to
put
the
system
under
a
little
bit
of
a
load,
and
this
was
the
experiment
that
we
decided
to
run
with
this.
A
So
the
setup
basically
had
five
ec2
instances
running
Kafka
brokers
are
4.2
extra
ROG,
which
are
like
8,
V,
CPUs
and
61
gigabytes
of
memory
spread
across
three
zones.
Each
broker,
the
JVM
was
getting
8,
GB,
CPUs
and
6
gigs
of
memory,
the
application
factor
of
3.
We
never
were
going
to
be
able
to
run
without
like
having
3
up
in
three
different
zones:
ten
producers
and
Simas
in
size
of
10
kilobytes,
the
node
on
the
other
side,
basically
talks
about
how
we
did
not
have
SSL.
A
We
did
not
have
background
stress,
I'll
go
a
little
bit
or
what
about
what
background
stress
means
in
details
in
a
little
later,
but
fundamentally
like
the
kubernetes
cluster
did
not
have
anything
else
running
on
it.
It
was
only
the
brokers
and
zookeeper
that
was
running
on
this
and
there
was
no
compression,
so
the
data
would
go
in.
A
It
would
go
into
a
topic
and
then
consumers
would
consume
from
there
and
it
would
just
like
consume
the
data,
as
is-
and
this
was
what
we
got
so
the
end-to-end
latency
that
the
y-axis
represents
here
is
basically
the
latency
measured
from
the
time
a
producer
produces
a
message
puts
it
onto
the
bus.
There
is
a
consumer
which
consumes
this
message
and
actually
reads
the
message
itself
has
a
timestamp,
the
consumer
reads.
This
message
looks
at
its
own.
A
Timestamp
looks
at
the
difference
between
the
timestamps
and
then
it
knows
what
was
the
end-to-end
latency
that
was
required
for
this
message
to
go
from
the
producer
to
the
consumer
and,
of
course,
as
the
producer
and
consumer
were
not
doing
anything
else,
so
there's
nothing
else
in
the
system
and
then
looking
at
the
the
latency.
This
is
what
we
got
so
about
450
thousand
messages
we
were
at.
You
know
on
an
average
at
about
35,
maybe
40
milliseconds.
It
was
a
little
up
and
down.
So
it
was
a
little
concerning
that
you
know.
A
A
Fundamentally.
What
happens?
That
also
was
the
reason
why
the
first
packet
or
the
first
message
always
started,
took
longer.
So
the
first
message
actually
has
a
latency
of
about
70,
75
milliseconds,
and
what
we
found
out
was
the
producer
or
the
consumer
is
trying
to
connect
with
a
kafka
broker
when
that
PCB
kafka
is
a
tcp
based
protocol.
A
So
there
is
a
TCP
connection
trying
to
happen
between
a
system
that
is
outside
of
AWS
or
outside
of
the
account
for
sure
and
trying
to
connect
to
the
broker
when
that
TCP
connection
is
actually
trying
to
be
established.
The
if,
for
some
reason,
AWS
or
the
broker,
decides
to
send
an
ICMP
fragmentation
needed
packet,
because
it
does
not
like
the
size
of
the
TCP
packet.
The
a
SS
NAT
gateway,
which
was
coming
in
the
way,
would
drop
these
packets
on
the
floor.
A
So
the
client
wouldn't
even
know
it
would
just
always
try
to
continue
to
try
and
establish
a
connection
with
the
packet
size
which
was
bigger
than
what
the
broker
liked,
and
this
communication
would
always
come
in
the
way
and
the
NAT
gateway
would
drop
these
packets
and
it
was
hard
to
get
there.
It
just
so
happens
that
TCP
tries
out
different
things
as
time
goes
on,
so
it
would
try
for
some
time.
A
But
after
that
this
was
a
little
up
and
down,
but
it
is
still
good
like
it
was.
It
was
good
performance
like
this
was
more
than
what
I
mean.
This
was
faster
than
what
we
were
wanting
to
see.
So
we
went
around
to
other
engineering
teams
that
were
going
to
use
this
with
an
into
it,
and
we
were
going
to
talk
to
them
about
like
what
are
they
requirements?
What
do
they
want
from
this
kafka
bus
and
after
connecting
with
multiple
different
teams?
This
was
the
final
set
of
requirements
that
we
put
together.
A
We
were
going
to
have
to
support
a
Kafka
on
kubernetes
infrastructure
that
is
going
to
have
9000
messages
being
sent
per.
Second
five
kilobyte
message
size
like
each
message
that
would
be
written,
would
be
about
50
kilobytes,
but,
given
that
we
can
use
compression,
we
can
have
the
message
size
on
disk
would
be
five
kilobytes.
The
retention
period
would
be
seven
days.
This
was
where
we
wanted
it
to
be
longer,
but
given
the
requirements
are
given
the
constraints
and
the
costs
involved,
seven
days
was
kind
of
a
happy
compromise.
A
Three
replicas,
as
I
said
we
always
want.
We
cannot
have
data
sitting
in
only
one
zone
in
AWS.
They
have
three
zones
in
every
region
or
the
u.s.
west
region,
which
is
where
we
were.
We
wanted
all
three
zones
to
have
one
copy
about
200
topics
in
Kafka
and
then
the
end-to-end
latency
was
awesome.
Like
people
were
like,
we
need
maximum.
We
can
I
mean
it's
okay.
If
the
internal
latency
is
100
milliseconds
or
less
so
again,
we
were
well
below
the
hundred
milliseconds.
A
We
could
have
claimed
victory,
but
we
didn't
so
based
on
these
kinds
of
questions.
We
were
like
okay,
this
is
getting
serious.
We
need
to
figure
out
how
we
can
now
take
this
one
by
towards
production,
so
this
was
the
architecture
that
finally,
it
came
up
to
so
the
master,
and
so
this
was
a
kubernetes
cluster
that
was
going
to
have
three
master
nodes,
one
in
every
zone,
so
the
master
nodes
that
you
see
here-
that's
2a
2b
and
2c,
one
master
node
in
each
zone,
the
zookeeper
instances.
A
So
these
zookeeper
nodes
that
you
see
here,
it's
an
auto
scaling
group.
This
auto
scaling
group
had
five
instances.
This
is
I
mean
it
has
six
instances,
but
only
five
of
them
get
used.
Do
keeper
is
a
consensus-based
protocol,
so
it
needs
an
odd
number
of
instances.
We
had
five
two
zones,
we'll
have
two
instances:
each
and
one
zone
will
have
one,
and
this
Kafka
nodes
is
another
auto
scaling
group.
A
There
were
given
all
the
requirements
that
we
had
based
on
some
calculations,
which
I
can
share
with
you
offline
if
needed,
but
we
decided
to
have
nine
brokers,
so
nine
ec2
instances
one
broker
on
every
instance.
Yes,
it
means
at
time
set
okay,
you
know,
containerization
is
all
about
impacting
many
things
together,
but
given
the
seriousness
of
how
things
were
we
wanted
to
be
kind
of
safe
than
sorry?
So
we
said:
okay,
let's
start
with
one
broker
pod
per
instance,
and
let's
see
how
it
goes.
So.
A
So
it's
one
thing
to
have
these
nine
brokers,
but
the
question
was
we
were
going
to
have
requests
coming
in
from
multiple
different
teams
or
multiple
different
services
that
were
not
necessarily
running
in
AWS,
so
we
needed
a
public
endpoint
and
how
do
you
give
a
public
endpoint
for
the
nine
brokers?
The
way
again,
Kafka
works
is
that
each
of
these
clients,
each
of
these
producers
or
consumers,
needs
to
be
able
to
access
and
directly
connect
over
TCP
to
each
of
these
brokers.
A
So
in
order
to
do
that,
one
option
would
have
been
to
make
all
those
nine
brokers
have
public
IPS,
which
was
not
great,
because
if
those
instances
go
down
new
ones
come
up,
they
might
change
their
public
IP.
We
could
give
them
static,
IP
addresses.
But
again
you
know
it
might
just
increase
the
threat.
The
security
folks
were
a
little
uneasy
with
that,
so
it
was
decided
that
a
better
option
would
be
to
kind
of
have
another
layer
in
front
of
all
these
brokers
and
we
decided
to
go
with
our
network
load
balancer.
A
A
A
So
so
this
is
just
a
little
bit
of
a
recap.
So
Kafka
and
zookeeper
an
independent
auto-scaling
groups,
there's
one
Kafka
broker
per
ec2
instance,
one
zookeeper
zookeeper
node
for
ec2
instance.
So
let
me
show
you
a
little
bit
of
this
in
action.
I
have
this
cluster
here
with
me
right
now.
Let
me
just
extend
this
SC.
A
So
I
have
this
kubernetes
cluster
here.
Right
now
in
this,
you
can
see
that
the
stateful
set,
which
runs
in
the
nine
brokers-
you
can
see
Kafka
CAFTA
zero
through
Kafka
eight,
as
the
eight
as
the
nine
broker
pods.
There
is
the
p
zu
0
2
p
zu
4,
which
are
the
5
zookeeper
instances,
and
then
this
is
the
extensibility
part
of
kubernetes,
where
we
use
Yahoo,
Kafka
manager
and
zookeeper
web
UI.
I'll
show
the
actual
you
guys
for
doing
this
a
little
bit
of
in
a
little
later.
A
So
there
are,
if
you
do,
if
you
look
for
the
stateful
sets,
there
are
two
stateful
sets.
If
you
look
for
config
Maps,
there
are
a
bunch
of
config
maps.
The
broker
config
is
the
config
map
used
by
the
brokers.
The
zookeeper
config
is
basically
the
config
map
used
by
zookeeper
and
it
has
the
server
dot.
Property
is
needed
for
zookeeper
on.
A
Ok,
so
this
is
going
back
to
me
and
it'll
be
configuration
that
I
was
talking
to
you
about,
so
the
NLP
quantification
took
a
little
time
to
understand
and
figure
out
fundamentally,
as
I
said,
it'll
be
basically
sits
in
front
of
every
broker
or
it
kind
of
acts
like
an
at
the
NLP.
Has
these
things
called
target
groups,
so
we
create
a
target
group
for
each
broker:
:
port
a
port.
So
if
you
see
here
a
port
3,
2,
4,
0
0
is
referencing
broker.
A
0
3
2
4
0
1
is
referencing
broker
1,
so
the
name
of
the
NLB
and
that's
this
port
number
uniquely
identifies
each
broker.
Kafka
also
needs
a
client
port.
Essentially
this
is
used
for
discovery,
so
clients
of
any
producer
or
consumer
needs
to
be
able
to
connect
on
a
fixed
number
of
fixed
port
where
it
goes
and
says,
hey
give
me
all
the
Kafka
brokers
that
are
currently
live
in
the
system,
so
this
is
called
a
client
ports
of
the
clock.
A
So
this
was
great,
so
we
had
a
Kafka
set
up.
We
had
a
stateful
said:
config
Maps,
all
that
good
stuff,
the
NLB
was
working
great
but
again
into
it
is
a
is
a
financial
services
company.
We
need
to
talk
about
security
and
in
fact,
regardless
of
whether
you
are
a
financial
company
or
not,
you
should
talk
about
security.
So
two
things
first
and
foremost,
authentication.
How
do
we
know
which
producer
or
consumer
is
allowed
to
be
able
to
write
to
this
Kafka
endpoint?
A
So
there
are
a
bunch
of
ways
in
which
this
can
be
done
and
I
wish
you
like.
You
know
there
was
time
for
kind
of
looking
at
each
and
every
single
option,
trying
it
out
getting
some
results
and
then
coming
up
with
the
most
optimal
one,
but
again,
given
the
constraints
of
time,
the
most
feasible
one
that
seemed
to
kind
of
fit
the
bill
that
we
had
was
mutual
TLS.
A
So
mutual
TLS
basically
requires
the
brokers
to
know
about
every
single
client
that
can
connect
with
them
and
each
client
it
needs
to
know,
needs
to
have
access
to
the
broker
so
that
they
can
present
this
certificate
to
the
client
and
say
that
ok,
I
am
Who.
I
am
check
that
I
am
allowed
and
then
let
me
in
and
then
let
me
allow.
Let
me
publish
the
bus
or
let
me
read
from
the
Kafka
broker.
Mutual
TLS
was
a
great
way
of
doing
this.
A
It
adds
a
little
bit
of
an
overhead
when
it
comes
to
onboarding
clients
onto
the
Kafka
instance.
So
when
people
when
new
clients
want
to
come
in-
and
there
is
a
little
bit
of
a
manual
process
where
we
have
to
like-
create
a
certificate
and
give
it
to
them
and
let
the
brokers
know
that
okay,
there's
a
new
client
that
you
will
see,
but
it's
okay
for
the
requirements
that
we
had.
This
seemed
to
be
okay,
so
that
that
takes
care
of
two
things.
It
takes
care
of
authentication.
A
It
also
takes
care
of
the
fact
that
over
the
wire
communication
is
happening
over
SSL.
So
therefore,
it's
all
encrypted
both
ways,
so
therefore
it
makes
it
super
secure
and
then
what
about
data
at
rest?
So
this
is
where
again,
the
kubernetes
was
awesome,
so
kubernetes
has
integrates
very
well
with
AWS
AWS
has
way
in
which
an
EBS
volume
can
be
encrypted
at
rest.
All
you
have
to
do
is
create
an
encryption
key
in
Amazon
kms
and
then
use
that
key
for
the
EBS
volume
and
tell
the
EBS
volume
to
in
confusing
that
kubernetes.
A
The
storage
class
object
in
kubernetes
has
an
option
for
doing
this.
It
just
is
literally,
is
one
line
which
has
encryption
equal
to
true
and
that
storage
class
will
then
or
when
the
pure
persistent
volume
gets
created,
the
key
gets
created
and
again
it
then
gets
used
for
encryption
of
EBS
volume.
So
we
basically
just
use
that.
A
A
Getting
and
if
something
goes
wrong,
that's
going
to
be
my
number
one
place
to
go
to
in
terms
of
figuring
out
what
is
wrong
so
for
logging
and
monitoring
into
it
has
has
had
a
history
of
using
Splunk
and
other
integral
Intuit
services
were
using
Splunk,
so
we
decided
to
kind
of
go
in
the
same
way,
but
it
also
helps
with
you
know,
being
consistent
across
the
board.
We
run
a
fluently.
Daemon
said
on
the
kubernetes
cluster,
it's
configured
to
write
data
to
an
Splunk,
h,
HEC
endpoint
and
the
flow
indeed
even
said.
A
Also
each
pod
takes
the
logs
from
each
of
these
instances
running
in
the
cluster.
Takes
this
logs,
writes
it
to
the
HCC
endpoint
and
the
HCC
endpoint
then
puts
the
data
into
Splunk
and
then
Kafka
and
zookeeper
logs
from
instances
were
basically
written
by
flu
in
D
into
Splunk.
So
that
way
we
were
getting
all
these
all
the
requests
that
we
are
all
the
logs
that
we
wanted
were
there
and
then
for
monitoring.
So
again,
you
know
other
applications
with
an
Intuit
have
had
used
wavefront.
So
we
kind
of
continue
to
do
this.
A
Wavefront
is
a
service
that
I
think
last
year
or
a
year
before
that
was
acquired
by
VMware
and
it's
a
great-sized
service.
It
gives
you
a
good,
it
stores
a
time
series
database
and
it
shows
a
great
way
of
charts
and
growing
charts
and
graphs
of
the
metrics
that
you
have
pushed
to
it.
It
also
integrates
very
well
with
pager
duty
and
things
like
that,
so
we
decided
to
use
that
the
metrics
had
to
be
collected
at
two
levels.
So
there's
one
about
the
cluster
level
metrics.
A
So
fundamentally,
how
many
nodes
do
I,
have
what's
the
memory
and
CPU
utilization
of
each
node,
what
about
the
disk
space
and
so
on
and
so
forth,
more
at
the
infrastructure
level,
and
then,
secondly,
application-specific
metrics,
so
within
Kafka
like
how
many
leader
elections
am
I,
getting
what's
the
what's
the
latency
between
my
two
brokers
in
terms
of
replication,
what
is
my
GC
time
and
things
like
that,
so
application
level
metrics
had
to
be
collected
and
sent
to
wavefront.
So
for
both
of
these,
we
found
ways
of
doing
this.
A
Hipster
I
know
it's
hipster
recently
has
been
deprecated,
but
heap
store
has
a
way
in
which
it
can
write
data
to
wavefront.
So
at
the
for
the
cluster
level,
metrics
we
run
hipster.
It
collects
all
these
metrics
and
send
ships
it
to
a
friend
for
application
level.
Metrics
we
use
Telegraph,
so
telegraph
runs
as
a
sidecar
container
to
each
of
these
brokers.
It
collects
all
these
metrics
jmx
matrix
from
Kafka
and
sends
it
to
a
friend
I'll
show
them
of
this
in
white.
Okay.
A
So
this
again
I
wonder
why
I
have
the
screenshots,
but
let
me
show
this
right
here
so
so
to
start
with,
this
is
the
so
this
is
the
dashboard.
This
is
a
kubernetes
cluster
level,
dashboard
within
wave
front.
So
this
particular
cluster
that
for
which
I
am
demoing,
it
has
three
namespaces
23
different
nodes.
It
has
117
pods
and
a
bunch
of
other
information
about.
You
know
pod
count
by
namespace.
A
What's
the
CP
usage
by
each
of
these
namespaces
memory
usage
by
each
node
again
giving
you
a
view
of
the
way
the
entire
in
or
the
kubernetes
infrastructure
is,
is
currently
is
currently
being
used,
and
then
there
are
ways
in
which
you
can
go
actually
go
back
in
time
or
find
a
specific
time
slot
and
do
this.
There
is
another
dashboard
that
wavefront
has,
which
is
specific
to
a
namespace.
A
So
the
the
pods
that
I
showed
earlier
are
running
in
namespace
called
cough
Kardashian
s,
so
for
this
cluster
in
the
namespace
called
cough
kind
s,
we
have
37
container
37
containers
and
then,
within
that
namespace.
How
much
is
the
network
traffic?
What's
the
memory
and
CPU
rate
and
things
like
that
and
then
lastly,
the
Kafka
specific
matrix,
so
there
is,
as
I
said,
telegraph
is
run
as
a
sidecar
container,
which
collects
Kafka
specific
matrix,
sends
it
to
this
wavefront.
A
So
currently,
in
my
Kafka
cluster,
there
are
0
under
replicated
partitions
0
offline
partitions,
which
is
a
good
thing.
If
this
number
goes
up,
we
have
a
wave.
We
have
an
alert
that
is
hooked
up
with
pager
duty,
which
then
calls
someone.
I
am
on
call
right
now,
I
hope
it
doesn't.
It
doesn't
fire
right
now,
hopefully,
and
then
these
are
specific,
metrics
or
metrics
specific
to
Kafka.
So
the
request
throughput
here
is
actual
requests
of
Kafka,
like
when
I'm
trying
to
write
a
message
or
read
a
message
from
a
Kafka
bus.
A
What
what's
the
throughput
that
I
am
getting
so
on
and
so
forth,
and
then
space
and
about
wavefront,
so
flu
in
D
is
used
to
for
collecting
these
logs
and
shipping
it
to
wavefront.
The
other
good
thing
that
flow
indy
does
in
this
case,
for
us
is
that
it
doesn't
it.
It
doesn't
just
take
the
log
message,
but
it
also
kind
of
augments
it
with
a
lot
of
metadata
about
where
the
message
came
from,
which
was
awesome.
A
What
is
the
pod
it
came
from,
so
this
particular
message
came
from
Kafka
7
from
the
namespace
called
Kafka
eNOS
from
this
cluster
called
Cherie
Kafka
test
cluster,
which
is
my
cluster
and
which
is
great
because
now
this
gives
me
the
ability
to
kind
of
drill
down
into
the
Met
are
the
logs
of
a
specific
pod
or
of
a
specific
namespace,
or
what
have
you
and
then
kind
of
slice
and
dice
all
this
information?
If
and
when
something
goes
wrong.
A
So
this
this
was
great
in
the
sense
that
you
know
once
the
logging
and
monitoring
data
was
also
kind
of
getting
to
kind
of
these
there's
the
final
destinations
wherever
we
wanted
it
to.
We
had
a
cluster,
we
had
an
NLB,
we
had
access
figured
out,
we
had
security
figure
out,
logging
and
malting
was
great,
but
there
were
always
these
last
mile
problems,
an
applicant.
A
A
Ctl
commands,
do
a
port
forwarding
and
then
ssh
into
the
pod
and
then
run
these
obscure
command-line
tools
that
kafka
provides
for
creating
a
topic
which
was
hard
and
it
was
like
it
took
time
and
effort,
and
it
was
very
error-prone
so
for
these
kinds
of
things
again,
extensibility
of
kubernetes
is
phenomenal.
So
we
found
a
yahoo
kafka
manager
as
this
as
a
tool
that
is
made
available.
That
is
open
source
by
Yahoo.
A
For
doing
these
kinds
of
these
kinds
of
these
kinds
of
tasks,
so
you
how
kafka
manager
was
an
open
is
an
open
source
tool.
We
take
it
and
run
it
inside
the
cluster
and
it
makes
it
easy
because
it
gives
you
a
web-based
front-end
or
web-based
UI
for
for
doing
these
kinds
of
tasks
like
creating
a
pod
or
sorry
creating
a
topic.
Configuring
the
topic
a
certain
way
and
things
like
that
same
with
zookeeper.
When
people
had
problems
with
Casca,
they
would
look
for
solutions.
A
And
the
moment
you
Google
about
a
POC
Africa
problem,
one
of
the
most
common
reasons
people
will
find
out.
One
of
the
most
common
problems.
People
will
say,
is:
oh
go,
look
at
zookeeper
and
getting
zookeeper
getting
to
zookeeper
was
a
problem
because
getting
like
a
zookeeper
also
has
a
has
a
command-line
tool,
but
it's
super
hard
to
use
it.
A
So
this
is
what
the
zookeeper
web
UI
shows
right
now,
I'm
just
at
one
particular
path
called
slash,
blocker,
slash
IDs.
So
it
shows
me
that
right
now,
I
have
nine
nodes
in
the
cluster.
That's
the
zookeepers
view,
and
there
are
other
things.
What
about
topics?
I
have
two
topics
right
now
called
consumer
offsets
and
topic,
one,
because
the
test
plaster
that
I
have
but
other
information
stored
within
zookeeper.
If
people
have
doubts
and
questions,
this
was
great
to
see
this
and
then.
A
So
yeah
how
Capcom
manager
has
something
like
this?
This
shows
up
again
it's
a
very
bare-bones
UI,
but
but
it
works
great
for
the
use
cases
that
we
had
about
where
people
can
use.
This
look
at
what
topics
they
have
look
at
the
lag
configure
new
topics
if
they
want
to
create
one.
So
if
you
do
go
here
and
create
a
new
topic,
you
can
create
new
topics
here,
configure
it
the
way
you
want.
A
So
this
was
good
because
it
kind
of
gave
people
easy
access
to
some
of
this
information
inside
the
cluster
and
then,
lastly,
our
go
so
I
am
biased.
I
work
for
a
team
that
built
our
go
as
the
workflow
engine
for
kubernetes
and
one
of
the
questions
people
always
had
was
even
if
I
have
access
to
a
Kafka
cluster,
how
do
I
do
basic
operations
or
basic
tests
on
this
cluster?
If
I
want
to
just
loop,
like
you
know,
create
a
topic
right,
you
know
100
messages
on
it,
300
messages
from
it.
A
How
do
I
know
that
this
is
actually
completing
and
doing?
Well,
so
we
used
our
go
for
this,
as
I
said
so.
Workflow
engine
for
kubernetes,
we
created
one
workflow
for
basic
tests.
We
created
another
workflow
for
a
stress
test
and
we
created
few
other
workflows
for
specific
needs
that
people
had
and
now
we
could
share
these
workflows
with
people
and
say
that.
Ok,
if
you
ever
want
to
try
out
something
different,
take
this
as
a
sample
workflow
and
you
can
change
it.
Augmented,
create
a
new
copy,
a
workflow
as
a
file.
A
So
we
shared
this
file.
It's
a
ml
file.
Take
it
change,
it
tweak
it
the
way.
You
want
and
run
this
workflow
on
the
cluster,
and
once
you
run
this
workflow
on
the
cluster,
you
will
be
able
to
get
some.
You
know
visibility
into
how
your
infrastructure
or
the
kafir'
cluster
looks
like
and
how
it's
running.
A
So
this
was
great,
so
we've
had
the
last
mile
problems
covered
people
were
happy,
but
what
about
performance?
We
started
off
with
this
goal
that
we
want
to
do
this
with
an
end-to-end
latency
of
100
milliseconds,
so
we're.
After
all
of
this,
we
said:
let's
run
this
performance
experiment
and
see
where
we
are
with
respect
to
baseline.
So
this
is
the
exam,
the
sample
setup
for
the
performance
experiment.
We
had.
A
Nine
instances
are
for
2x,
/,
8,
V
CPUs,
spread
across
three
zones,
X
equal
to
all
the
application
factor,
3
and
so
on
and
so
forth,
and
then
the
no
this
time
it
was
with
SSL.
So
it
was
going
to
add
a
lot
more
computation
for
the
TCP
connection
for
the
ssl
overhead
for
decrypting
the
data
and
and
everything
there
was
due
types
of
experiments
done
with
and
without
background
stress.
A
So
basically
background
stress
was:
we
were
measuring
the
latency
between
ten
producers
and
ten
consumers,
but
at
the
same
time
there
were
three
other
producers
generating
about
4,500
messages
and
writing
them
into
Kafka,
so
we're
generating
a
bunch
of
activity
within
Kafka
and
kubernetes
for
other
things,
but
measuring
the
latency
for
these
producers,
because
that
was
going
to
be
the
common
case
and
then
snappy
compression
as
because
I
mean
among
the
compression
techniques.
Kafka
supports
we
picked
one,
we
experimented
with
it.
A
Snappy
was
the
best
one
for
the
for
the
type
of
JSON
objects
that
we
were
sending.
So
we
decided
to
use
snappy,
and
this
is
what
we
got
so
the
two
lines
here.
The
blue
one
is
basically
the
end-to-end
latency
with
background
stress,
so
it
is
also
up
and
down,
but
some
of
that
is
attributed
to
the
fact
that
there
is
a
background
stress.
There
is
a
bunch
of
activity
happening
behind
the
scenes,
and
it's
a
little
up
and
down.
We
need.
A
We
actually
should
go
even
deeper
than
what
we
know
of
today,
but
even
then
the
average
latency,
if
you
look
at
that,
is
about
37
milliseconds,
which
is
still
well
below
100
millisecond
mark
that
we
were
aiming
for
the
good
really.
The
good
part,
although
was
the
red
line
actually
now
shows
so
once
we
fixed
the
nat
gateway,
I
mean
we
didn't
fix
the
nat
gateway
problem.
All
we
had
to
do
is
make
sure
that
the
jumbo
frames
was
enabled
across
all
the
components
involved.
Jumbo
frames
were
enabled
on
the
ec2
instance.
A
Jumbo
frames
were
enabled
in
calico,
where
you
had
to
set
the
size
of
the
tcp,
the
tcp
MTU
size
once
it
was
set
to
8
1
9
2
bytes,
the
the
like.
The
nat
gateway
being
a
problem
in
between
went
away,
and
we
now
have
a
much
more
consistent
red
line,
which
is
the
much
more
consistent
performance
as
the
baseline.
So
this
was
great.
This
is
I
mean
yes,
there
is
a
little
there's.
A
Some
work,
there's
more
work
to
be
done
in
figuring
out
what
what
else
can
be
done
to
kind
of
bring
this
latency
even
below,
where
it
is
at,
but
compared
to
where
we
had
begun
and
compared
to
the
goal
that
was
set
for
us.
We
were
well
within
that
range,
so
at
37
milliseconds
this
was
really
working
with.
A
A
You
might
have
to
figure
out
how
to
convert
them
between
jmx
format
and
some
other
format
that
the
modeling
service
understands
NAT
gateways
on
AWS,
don't
support
TCP
flag,
my
IP
fragmentation,
TCP
MDU
sizes
may
not
be
set
correctly,
so
you'll
have
to
go
ahead
and
look
at
that
and
modify
it
if
needed,
and
then
log
messages.
So
this
happened
once
where
we
were.
We
were
debugging
the
Kafka
problem.
We
changed
the
log
level
of
the
brokers
to
debug
and
it
kind
of
overwhelms
plunk
and,
like
it
started,
throwing
all
kinds
of
errors.
A
The
log
message
rate
from
the
brokers
can
be
very
high,
so
I
wish
I
could
give
you
a
good.
There
could
be
a
conclusion
where
I
make
you
like
super
stoked
about
this
and
make
you
run
back
to
to
your
offices
and
install
Kafka
on
kubernetes.
But
my
high-level
point
is
it's
possible
to
do
this,
so
you
can
absolutely
do
this.
It's
a
lot
of
fun
to
do
this,
so
just
go
ahead
and
try
it
out.
And
lastly,
thank
you
so
much
for
being
here.