►
Description
Speaker: Mick Semb Wever, Team Member
Monitoring provides information on system performance, however tracing is necessary to understand individual request performance. Detailed query tracing has been provided by Cassandra since version 1.2 and is invaluable when diagnosing problems. Although knowing what queries to trace and why the application makes them still requires deep technical knowledge. By merging Application tracing via Zipkin and Cassandra query tracing we automate the process and make it easier to identify and resolve problems. In this talk Mick Semb Wever, Team Member at The Last Pickle, will introduce Cassandra query tracing and Zipkin. He will then propose an extension that allows clients to pass a trace identifier through to Cassandra, and a way to integrate Zipkin tracing into Cassandra. Driving all this is the desire to create one tracing view across the entire system.
A
A
It's
only
help
that
I've
always
been
engaged
with
open
source
code,
but
Cassandra
was
really
something
that
changed
a
lot
for
me,
because
I
never
really
liked
databases
when
I
went
through
University
I
didn't
do
a
single
database
on
SQL
subject,
because
I
thought
it
was
boring
and,
through
my
whole
career,
I
kind
of
stayed
away
from
databases
as
much
as
I
could,
and
it
wasn't
just
as
it
was
boring
in
my
eyes.
It
was
that
it
just
didn't,
feel
right,
I
mean
you
go
back
15
years
ago
and
we
knew
to
avoid
distributed
transactions.
A
We
knew
to
pay
about
stateless.
We
knew
that
if
you
could
avoid
putting
objects
in
the
session
scope,
you
could
scale
ourself
on
Castle
easier,
and
so
you
know
the
same
time.
You
know
these
databases
guarded
by
their
DBA
and
it
was.
It
was
wrong.
It
didn't
jell
with
the
technologies
of
the
job
developer
that
we
were
working
with.
A
A
C
A
A
A
C
A
A
Now
break
that
up
into
those
three
sections
and
have
a
look
at
them.
The
first
section
is
really
about.
Today
we
have
so
much
volume,
an
adulterer.
We
have
so
much
velocity
over
the
data
and
we
need
to
work
with
duct
in
different
ways,
and
it's
coming
at
us
from
different
angles.
We
sing
Cassandra
we're
seeing
Kafka
same
spark,
for
example
amongst
them,
cassandra
is
really
that
data
platform.
It's
about
storing
the
data
I
boils
down
to
the
name
to
scale
down
to
the
day.
A
The
second
part
is
giving
more
context
to
methodology
today,
microservices
and
DevOps,
and
so
on.
Well,
microservices
is
a
buzzword
sister,
a
revival
of
service-oriented
architecture.
It
does
also
incorporate
some
more
the
human
factor
today.
You've
got
the
DevOps
aspect
there.
You've
got
monitoring
tools
requires,
is
talking
about
Conway's
laws,
structuring
your
team's
correctly,
so
that
they're,
decentralized
and
autonomous
a
lot
of
this
is
coming
down
to
the
need
to
scale
it
people
in
organizations.
A
We
want
teams
to
be
working
at
full
speed,
not
waiting
on
each
other.
When
you
have
each
team
decentralized
autonomous,
it
was
more
pressure
on
having
expanded
deployment
mechanism.
It
was
no
pressure
on
having
good
monitoring
tools,
especially
when
we're
in
the
context
of
distributed
architecture,
if
you're
using
apache
cassandra
chances
are
you've,
got
service,
oriented
architecture
on
that
and
scout
out
moodle
architecture.
On
top
of
that,
Cassandra.
A
A
But
what
is
missing
is
just
really
tracing
and
you've
got
those
bikes
which
can't
be
reproduced
by
one
service
allowing
that
kind
of
snuck
in
between
our
different
services,
interact
or
you're,
trying
to
solve
the
slaw.
The
99th
percentile
agencies,
and
you
need
to
go
to
looking
those
difficult
to
find
requests
which
are
ruining
those
performance
numbers
and
in
the
store,
come
back
to
visit
zipkin
as
the
solution.
A
Combined
centralizers
logs
for
you
into
one
place,
your
final
lies
in
for
a
fight
and
centralizes
metrics
for
you
Fletch
at
the
mountain
you're,
not
even
more
ready
to
be
really
sure
when
you
get
those
things
working
well,
all
those
three
components
are
working
as
well,
which
you
find
is
that
one
of
other
components
to
microservices
the
architectural
safety
circuit
breakers
bulk
heading
load,
balancing
all
that
really
tricky
stuff
of
the
infrastructural
libraries
in
your
distributed
architecture
is
much
easier
to
do
really.
Well,
you
don't
kind
of
have
code
with
concurrency
bug.
A
You
often
end
up
with
a
much
more
concise
infrastructure
code
teams
are
able
to
spot
the
problems
or
collateral
damage
that
they
do
before
other
teams,
and
this
is
a
gain
about
scaling
people
and
about
creating
a
healthy
culture
in
the
organization
where
people
are
blaming
each
other
you're
proactive
in
spotting
collateral
damage
that
your
cause.
Now
and
again
you
also
have
a
service
where
you
don't
have
that
kind
of
tenth
of
a
percent
of
the
users
getting
lost.
A
A
Paper
was
a
quite
simple
approach
that
basically
said
that
to
do
distributive
tracing
you
need
to
instrument
you're
on
time
code
and
before
any
client
makes
a
service
call
or
a
network
pop
you've
got
to
have
an
instrumentation
annotation
and
the
request
starts
and
when
it's
finished,
if
you
want
to
pass
that
tracing
on,
you
have
to
pass
the
trace
ID
through
the
protocol
and
on
the
server
side
of
similar
matching
annotations.
I've,
come
back
and
explain
that
again
later.
A
We
need
your
search,
Achilles
traces,
each
of
the
traces
show
how
long
they
took
which
services
they
touched
to.
What
was
the
total
time
spent
in
the
different
services?
You
can
also
sort
this
different
ways.
So
teams
can,
for
example,
once
a
week
go
in,
have
a
look
at
traces,
then
involve
their
services
and
just
soak
the
longest
first
and
look
in
and
see
if
there's
anything
that
they
can
do
to
improve
that
clicking
on
a
trace
will
take
you
into
trace
page.
A
On
trace
page
along
the
top,
you
see
how
it
is
about
what
services
involved
and
underneath
that
you
see
this
call
tree
this
first
row
with
the
light
blue
bar
going
across
the
hallway,
is
the
users
request,
as
they
saw
it
here,
taking
198
milliseconds
and
then
underneath?
That
is,
if
all
the
different
services
that
were
invoked
behind
the
scenes,
all
the
way
down
to
individual
database
cause
in
this
example,
there
was
nothing
writing.
Currently,
it
was
all
synchronous
and
sequential.
A
A
This
is
incredibly
useful
for
front-end
developers,
because,
especially
in
the
context
of
where
teams
are
getting
more
decentralized
and
eponymous
and
people
are
caring
less
about
what
what
is
what
else
is
in
the
platform,
this
kind
of
still
helps
bring
people
back
without
needing
to
care
about
how
that
platform
is
working
and
when
front-end
developers
have
problems,
for
example,
something
as
down
alright
slow
in
production
or
testing
environments.
It
helps
them
go
to
the
correct
back-end
team
and,
with
proof
show
that
there's
a
problem
that
goes
a
long
way
again
to
helping
that
culture.
A
A
A
We
want
you
to
stop
working
with
that
old
legacy,
shared
schemer
and
start
having
your
own
encapsulated
diet
places
or
we
want
you
to
break
up
your
synchronous,
calls
across
the
band
and
context
and
start
using
what
I've
been
driven
design
and
if
you
go
through
this
day
by
day,
even
then
check
this
back.
Progress
amongst
all
the
teams
is
actually
happening.
A
So
that
wraps
up
the
demo,
Philip
of
the
run
through
a
Zipkin
going
back
to
explaining
how
that
instrumentation
annotations
work
for
zipkin
and
as
it
was
described
in
the
Google
paper
on
the
client
side
before
a
request
or
service
court
is
made,
it
marks
a
client
send
or
CS
annotation
when
their
response
is
finished,
the
client
wraps
it
up
with
a
client
receipt
or
see
our
foundation.
Just
those
two
on
the
left
hand,
side
is
enough
to
get
one
of
those
rows
or
spans
within
the
trace
visible
on
the
tress
page.
A
But
if
you
want
that
trace
to
continue
into
the
server,
she
wants
the
server
to
be
able
to
create
new
spans
as
a
maximum
chords.
Or
if
you
want
to
investigate
the
network
latency
between
client
and
server,
then
you
have
passed
the
traits
ID
and
that
counts
fan
I'll
eat
through
to
the
server
and
the
server
creates
this
server
received
or
si
annotation
when
it
gets
requests
and
this
server
sent
or
SS
foundation
when
it's
finished,
those
instrumentation
messages.
A
Sorry,
those
instrumentation
messages
get
dumped
into
scribe,
Kafka
and
they
get
sent
off
to
a
zip
can
collect.
Is
it
can
collector
than
those
stores
it
in
Cassandra
and
then
that
you
I
that
I
was
showing
you
is
hitting
a
Zipkin
rest
api,
which
is
getting
that
data
Cassandra
I
am
in
fact
writing
a
plug-in,
so
that
for
applications,
which
already,
for
example,
have
their
dallas,
texas,
tech
or
driver
in
it
shows
instrumentation
messages
can
use,
go
straight
into
the
Zipkin
Cassandra
lot
of
s
a
little
more
freedom.
A
A
Include
examples
to
this
too,
we
say,
for
example,
with
a
simple
client
in
Java,
making
a
HTTP
call,
for
example,
using
H
Apache
HTTP
client
to
do
the
instrumentation
on
the
client
side.
It
is
as
complicated
as
doing
this
here,
I'm
using
this
brave
Zipkin
client
library,
which
is
the
best
for
java
applications,
I'm
creating.
A
A
C
A
A
So
the
last
part
of
my
talk
is
combining
Zipkin
tracing
and
Cassandra
tracy
that
warrants
a
introduction
into
tracing
and
Cassandra
tracing
Cassandra
came
in
with
1.2
and
like
most
great
features
in
Cassandra
the
developers.
The
community
does
a
very
good
job
at
looking
at
around
of
what
are
that
existing
solutions
out
there,
then
this
issue
they
go
through
the
google
depa
take
bath
and
x-rays
and
different
different,
complicated,
pressing
solutions,
but
they
said
very
explicitly
when
they
implemented
this,
that
they
didn't
need
anything
sophisticated
the
needed
just
simple
tracing
Cassandra.
A
A
A
A
That
tracing
implementation
in
cassandra
is
really
very
simple:
we're
talking
about
four
classes
in
the
transom
package,
the
main
two
classes
here
at
racing
and
traced,
a
tracing
is
just
a
single
turn
into
Cassandra.
Jovian
they've
just
hand
was
a
basic
logic
for
you,
and
each
request
with
requires
tracing,
gets
a
new
trade
state
and
that
carries
through.
B
A
Then,
when
the
coordinator
cause
the
replica
know
where
you
can
nodes,
it
puts
the
trace
header
into
the
message
which
goes
over
the
messaging
service
in
Cassandra
and
then
on.
The
replica
lifestyle
that
message
comes
out
in
the
tracing
package
is
initialized
message:
method
that
takes
out
the
header
and
continuous
the
tracing
on
a
different
level.
A
C
A
Visualization,
while
I
can
be
kind
of
seen
as
eye
candy,
it
I
think
it's
more
than
that
as
well.
I
mean
you
get
to
see
very
quickly
hierarchy.
Of
course,
you
get
to
see
the
a
synchronicity
of
a
concurrency
and
I'll
show
you
that
in
a
bit,
you're
going
to
get
potentially
better
timings
around
tracing
and
you're
also
going
to
get
0
over
here.
N0
is
not
such
thing
really
a
zero,
but
the
cassandra
is:
writing
all
of
those
trade
statements
back
into
its
own
system
tray
table.
A
So
this
is
huge
for
identification,
which
is
why
you
can't
turn
something
on
very
high
in
production,
whereas,
for
example,
with
a
Zipkin
at
the
previous
place,
I
worked.
We
had
Zipkin
tracing
on
for
a
hundred
percent
sample
rate
and
production
2,000
per
second
over
a
platform
of
three
to
four
hundred
services,
each
scale
to
220
nights,
and
there
was
no
visible
of
it.
We
weren't
collecting
those
traces.
A
So
if
I
have
done
this
work
over
the
last
month,
I'm
hathaway
code
base,
which
I
thoroughly
enjoy
doing
and
I,
can
only
recommend
Sandra
code
bases
quite
elegant
to
work
with
Gary
I'm.
A
previous
talk
today
did
mention
that
it
wasn't
Pickler,
wasn't
tickly,
modular
or
built
up
of
primitives.
There's
good
reason
for
that.
It
is
a
bit
more
sophisticated
than
your
typical
enterprise
software,
but
it
is
still
very
nice
car.
You
learn
a
lot
from
working
with
it.
A
And
to
achieve
that,
it
really
wasn't
very
simple.
It's
basically
just
to
extend
two
of
those
classes
in
those
two
original
classes
are
tracing
in
the
tray
state.
I
ripped
out
a
few
lines
of
code
that
does
the
final
here's
our
trades
and
write
it
into
that
system.
Tray
table,
move
that
away
into
a
separate
implementation,
the
defaulting
fermentation
and
with
those
two
classes
there
was
one
other
method
without
them,
pull
that
couple
lines
of
code,
the
override
it
otherwise
Walter
encore
is
the
upper
switch
in
place.
So
I
could
test
both
solutions.
A
A
I
think
this
is
really
starting
to
get
interesting
because
from
here
you're
going
to
pick
up
problems
like
one
no
banging
slowly
and
why
it
was
behaving
slowly.
The
coordinator,
not
dispatching
those
messages
out
properly
or
kind
of
currency,
that
you're
thinking
of
using
nah
actually
work
in
the
way
or
the
read,
repairs
kicking
in
or
the
snitch
not
behaving.
There's
a
whole
host
of
bugs
that
you
can
see
far
quicker
with
this
and
leave
then
with
just
that
text
version.
A
A
Let's
look
at
full
stack
tracing,
let's
get
that
original
platform
tracing
that
we
had
and
let's
see
if
we
can
pass
in
a
trace
headed
into
Cassandra.
So
we
get
both
breasts
in
the
one
page.
I
didn't
think
that
this
was
possible
because
the
secret
world
protocol
doesn't
allow
much
time
to
pass
in
a
trade
hiding
and
span
ID
into
a
thick
growth
statement,
but
just
recently
sorry
I,
just
like.
Oh
that's
different.
Just
recently,
this
issue
came
along
which
allows
you
to
put
arbitrary
key
of
values
into
sequel
cause
behind
the
scene.
C
A
A
C
C
A
Have
made
available,
or
next
5-10
minutes
will
make
your
available
and,
like
I
said
before,
working
with
cassandra
is
really
rewarding
and
often
stuff
like
this.
Once
you've
got
your
head
around
it,
you
end
up
with
quite
simple
and
small
patches
to
achieve
what
you
want
so
here
to
achieve
this
at
least
to
achieve
to
make
cassandra
have
a
pluggable
tracings
interface
outside
the
tracing
package.
Is
a
four
piles
touched
and
almost
like
only
five
lines.
Additive.
A
Other
thing
it
stops
there
either
I
think
this
idea
of
tracing
like
this
can
go
a
lot
further.
We
could
now
trace
Compassion's,
for
example,
we
could
trace
and
the
entropy
repair
I,
don't
have
any
definite
suggestions
in
this
area.
I
think
it's
interesting
to
have
to
think
about
you.
Maybe
you
could
see
compaction
czar
getting
held
up
because
you've
got
one
of
your
disks
in
a
j-bot
set
up.
That's
that's
just
not
playing
right
or
maybe
part
of
the
Adriatic
be
repaired.
A
There's
a
node
that
is
just
not
doing
the
right
thing,
you're
going
to
see
that
things
quicker
and
more
police
have
been
out
of
debugging.
The
problem
that
wraps
on
my
talk
so
I'm
ahead
of
time.
I
can
keep
talking.
If
anyone
really
is
fixed
and
guessing
that's
okay.
Second
afternoon
conference,
I
put
up
all
the
references
here,
so
that's
catch
as
there's
an
issue
created
and
Cassandra's
jira
all
ready
for
it
now
will
submit
the
patch,
so
I've
also
uploaded
to
github
the
zip
kin
implementation.
A
B
A
I,
don't
use
much
that
much.
How
can
you
joke
that
I
vigilance
plant
is
more
like
the
Cabana
there's
more
that
log
processing
and
there
is
an
overlap
there
as
well.
What
zip
can
give
you
for
free
is
a
correlation
my
day,
so
that
you
can
because
you've
got
this
trace
ID
as
your
correlation,
so
winzip
can
working.
You
now
have
that
free
correlation.
I
believe
it's
important
to
luggage
or
whatever
you're
using
in
your
logs.
Then
when
you
go
to
gabbana,
you
can
now
search
on
that
same
traits.
B
A
A
Artists,
ya,
know
side
by
side.
Well,
some
beside
the
play
I
think
that
the
current
tracing
implementation
in
cassandra
is
the
correct
default.
It's
a
simple
one
to
keep
Cassandra,
but
I
would
like
to
see
that
that
tracing
can
be
swapped
out
with
the
page
up
on
so
an
example.
If
you've
got
clicking
in
your
environment
like
this
or
you
would
like
to
have
it,
it's
simply
about
a
matter
of
plugging
that
jar
up
copying
a
job
on
the
lib
folder
and
starting
out
with
that
flag
and
as
a
show,
it's
not
okay.
Let's
do.
C
A
A
Of
microservices
is
to
use
as
few
Pro
project
holes
in
your
platform
as
possible,
so,
for
example,
if
you're
using
only
HTTP
rest
and
tripped,
then
you
might
find
that
your
company
has
kind
of
infrastructural
libraries
around
those
protocols,
maybe
the
handles
on
a
circuit
breaker,
avoid
buying
seeing
above
getting
and
with
those
infrastructural
libraries.
You
can
do
this
and.
B
Do
some
benchmarking
going
on
Cassandra
and
I
see
that
whenever
I
increase
right,
we
start
to
affect,
and
I'm
at
always
fly
true,
I'm
from
the
coracle
side
of
hy
kept
distressing
easy
to
find
a
wife.
Something
is
going
down.
So
whenever
a
suite
is
wearing
down
because
it
arrives
on
high,
if
there
a
way,
I
can
find
why
the
reefs
are
slow
or
with
their
some
contention
going
on
in
the
cassandra.
Is
that
can
can
help
with
these
type
of
issues?
If.
B
Me
so
Zipkin
you
are
setting
trace
point
you
to
some
some
fixed
locations
right,
so
there
is
I
mean
tracepoint
set
at
some
part
of
the
code.
Is
there
a
way
to
make
a
hot
patching
type
of
thing
that
you
can
say
that
satisfy
a
point
with
this
part
of
the
curve
without
in
without
already
having
like
it,
mostly
with
the
Colonel's
I
will
become
a
you.
Have
ability
like
with
the
framework
I
systems
have
another?
You?
Can
you
technically
hot
patch
of
thirty
percent
code?
If
you
try
to
do
I'm,
not
following
okay?