►
Description
Operational and Business Insights from Tracing Distributed Microservices [I] - Juraci Paixão Kröhling, Red Hat
In this talk we will present Hawkular APM, an OpenTracing-compliant community project. We will demonstrate how the project can be used to monitor microservices in a cloud environment, capturing trace and business metric information, which can then be presented in both aggregated and individual trace views to provide useful insight for many roles within an organization.
About Juraci Paixão Kröhling
Juraci is a software developer working at Red Hat on the Hawkular project.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
I'll
go
very
quickly
over
some
basic
concepts
of
tracing
and
distributed
tracing.
If
you
attended
to
talk
so
go
on
this
room,
you
might
be,
you
might
know
all
of
that
already
otherwise
yeah
so
I'll
just
talk
very
quickly
on
that,
then
we
have
a
demo
and
a
Q&A
at
the
end.
But
if
you
end
up
with
questions
just
stop
by
the
redhead
bull
I
can
answer
any
questions
there
and
tomorrow
I
think
at
two
we
have
a
distributed.
A
A
A
A
So,
if
you
have
an
algorithm
which
is
complicated,
and
you
want
to
know
how
long
it
takes.
You
just
add
time
information
to
that
so
before
and
after
and
if
you
should
not
satisfied,
then
you
just
step
into
your
code
and
let
more
time
information
that
is
already
tracing
right.
So
tracing
basically
is
a
tactic
of
knowing
what's
going
on
honor
code
when
I,
when
a
given
context
on
princess
the
main
it
should
be
required.
The
context
would
be
like
the
conversation
between
a
client
and
server.
A
So
whenever
HTTP
request
comes
in
tracing
would
tell
us
what
is
going
on
that
HTTP
request.
Perhaps
timing
when
the
request
comes
in
when
the
response
is
sent
out.
Perhaps
we
would
also
do
some
timing
on
the
database.
Perhaps
we
have
a
business
mess
up
there.
Everyone
also
to
not
only
time
it,
but
when
store
what
is
the
order
ID,
for
instance
yeah?
So
that's
Tracy,
so
this
model
worked
quite
well
for
for
monolithic
applications,
because
tracing
is
mostly
for
a
single
process
or
F.
A
That's
how
it
was
done
to
recently
now
that
doesn't
work
for
micro
services,
because
one
it
should
be
required.
It's
not
it's
not
telling
as
much
right.
So
when
it
should
be
request
is
only
a
part
of
a
big
picture,
and
then
we
start
with
the
need
of
passing
on
the
context
that
we've
got
from
the
first
box.
All
the
first
walk
the
request
comes
in
and
then
we
get
some
downstream
HTTP
calls,
and
perhaps
some
messaging
going
on.
We
have
some
database
here
is
a
database
right.
A
So
then
remote
services,
remote
calls
perhaps
HTTP
requests,
perhaps
something
else,
and
then
here's
something
synchronous,
a
message
sent
somewhere
and
then
it's
captured
by
two
places
and
so
on.
So
we
need
to
pass
the
context
to
children's
other
components
and
that's
distributed
trading.
So
the
street
racing
is
basically
expanding
the
context
from
a
single
process
to
several
processes,
perhaps
in
a
in
remote
machines.
A
Now
it
also
brings
new
problems
with
that.
So
usually,
when
we
talk
about
micro
services,
we
also
talked
about
polyglot
applications
or
or
polyglot
architecture
right,
so
I
have
a
service
in
in
Java
and
then
I
have
it
serviced
in
node.js
and
another
one.
Perhaps
you
go
so
how
do
I
make
sure
that
data
collected
by
my
job
application
is
the
same
in
a
same
format
as
my
node.js
application?
My
goal
and
that's
the
context
for
open
tracing
right,
so
open
tracing
is
a
it's
a
vendor-neutral
open
standard
for
distributed
tracing.
A
A
So
it's
not
specifying
a
implementation
specifying
the
just
api.
So
the
interesting
part
is
it
defines
the
semantics
of
this
musics
racing,
so
it
specifies
whether
they
spend
what
is
expand
context.
Where
is
a
trace?
What
is
it
tracer
right?
So
that's
the
interesting
part
on
to
me
the
most
interesting
part
on
mobile
tracing
itself.
A
A
Yeah,
so
this
Hawker
thing
that
I'm
talking
about
what
is
that
so
hope
there
is
a
an
open
source
project.
It's
sponsored
by
Red
Hat.
It's
basically
set
of
monitoring
related
components
and
it's
been
used
in
some
redhead
products
like
open
shift
and
Hawk
Larry
p.m.
is
one
of
those
modules
from
hawker.
It's
the
application
performance
management
component
and
there
is
a
a
open
tracing
compliant
tracer
for
how
Barry
p.m.
A
right
so
I
guess
we
can
say
it's
a
open
trace
implementation
if
there
is
such
a
thing
so
for
the
demo
that
that
was
it
for
them.
The
the
theory
for
a
demo
I
want
to
achieve
two
things.
So
I
want
to
see
two
things.
One
is
a
scream
like
that.
We're
based
on
tracing
data.
We
can
extract
business
insights
like
that,
so
I
want
to
track.
If
you
look
at
the
top
part
of
that
screenshot,
we
have
a
transaction
called
place
order.
So
that's
the
transaction,
wanna
trace,
I,
wanna
track
and
the
second
goal.
A
If
we
have
time
and
if
everything
works,
we
are
going
to
try
to
get
an
operational
inside
with
the
same
data.
So
the
same
application
reporting
the
same
data
is
just
another
different
view
of
what
we
are
sending
their
and
based
on
that
I
want
to
go
and
see
if
a
new
release
is
no
worse
than
the
previous
release,
often
the
same
obligation
right.
So
where
can
we
release?
I
guess,
and
both
of
you
are
aware
of
what
it
is,
but
it's
basically
tactic
or
the
technique
of
running
two
versions
of
the
same
application.
A
At
the
same
time,
one
serving
like
ninety
percent
of
the
traffic
one
serving
ten
percent
and
any
the
ten
percent
is
the
new
version.
Then
you
assess
if
it
is
good
or
not,
and
if
it
is
good,
then
you
just
increase
the
percentage
of
traffic
reaching
that
application
and
decrease
the
other
one
until
the
new
version
eventually
replaces
the
old
one
right.
So
that's
a
cannery
release
using
Bluegreen
deployment
scenario
right.
So
enough
talk,
let's
see
some
code.
A
A
A
37,
okay,
so
we
see
no
data
here,
so
it
is
operate
them
don't
worry
about
the
UI.
The
important
here
is
not
operate
them.
It's!
It's
just
interpreting
the
data
right.
Similarly,
important
part
is
not
open
shift.
Okay,
so
it's
in
our
application.
Request
comes
in
into
DC,
where
I'll
hear
any
served
by
this
service.
Now
this
service
is
a
vertex
application
and
froze
familiar
with
vertex
vertex
communicators
in
a
distribute
using
a
event.
A
A
A
A
Our
code
is
a
vertex
vertical.
It
basically
just
starts
a
server
on
the
port
8080,
with
three
endpoints
get
status.
Post
orders
get
orders
interesting,
14
assists
the
post
orders,
that's
the
one
that
our
script
is
calling
and
we
have
a
handler
HTTP
handler
and
what
it
does.
It's
it
just
creates
a
span
context
against
is
a
open
tracing
concept
in
this
particular
case
here
in
this
code.
Here
it
is
just
creating
a
context
from
from
scratch,
because
that's
where
the
transaction
starts
right
and
then
we
immediately
create
a
nice
pan.
A
Now
we
are
going
to
notice
that
we
have
some
information
here,
which
is
duplicated,
like
the
URL
vertex,
most
a
URL
already
all
right,
so
why?
Why
should
I?
Add
that
or
the
vertex
knows
that
this
is
a
post
call.
So
why
should
I
repeat
this
information
so
every
day,
there's
a
new
framework
integration
on
open
tracing
and
I
expect
most
of
this
code
here
to
not
exist
right.
So
what
I
do
expect?
It
is.
Is
a
code
like
this
one,
so
the
transaction
name,
for
instance
like
this
place
order.
A
It
is
something
that
no
framer
can
can-can
digis,
so
you
have
to
annotate
your
code
to
provide
information
like
that.
Then
we
start
to
spend
and
at
some
point
we
we
want
to
check
the
account
for
that
order,
and
this
one
would
before
doing
any
remote
call
we're
changing
the
context.
We
inject
our
our
expand
context
and
then
make
the
remote
call.
In
our
case,
we
send
a
vertex
event
to
this
destination
here,
and
this
destination
is
set
to
the
porn
by
another
application,
so
account
manager
is
another
service.
A
Some
are
here
with
ya
here,
eventbus
customer.
So
this
is
that
this
code
here
is
saying:
I
listened
to
design
point
or
chilis
destination,
whereas
this
one
here
says
I
sent
to
the
destination
right
and
then
we
think
that
code.
We
extract
like
the
context.
So
in
this
context
here
will
oh
never
say
never,
but
it
will
never
be
started
from
scratch
would
derive
information
from
the
the
context
has
been
created
in
the
previous
code.
A
Then
we
start
a
new
spam
and
we
specify
that
this
pan
is
a
child
of
the
parent
span.
Ok,
so
I
can
don't
bother
about
the
code
itself.
I
guess
you
can't
understand
what
it's
doing
and
most
of
the
other
modules
they're
doing
very
similar
thing
to
this
one.
So
we
are
just
going
to
move
over
to
rockford
ATM
so
that
we
can
have
a
better
visual
on
all
of
that.
So
some
of
the
information
we
could
derive
from
this
view
here,
so
we
could
say
that
yeah.
A
I
know
that
the
request
is
complete
reaching
this
point
here,
but
we
don't
know
what
this
one
does
now.
Looking
at
the
code,
we
know
that
this
one
calls
this
one,
but
what
is
the
relationship
partner
with
the
one's
right?
So
this
is
for
general
Perkins
receiving
the
data
that
we
are
so
our
gen
ordered
script
that
is
running
on
background
is
exercising
our
example,
which
is
then
sending
data
to
operate
them
and
how
Barry
p.m.
is
processing
the
data
making
aggregations
and
so
on,
and
then
we
derive
information
like
so.
A
We
know
that
whenever
a
request
comes
in
those
are
the
sources
which
are
possibly
related
to
that
request.
We
see
that
we
have
800
requests
here
in
100
requests
here,
even
though
they're
here,
it's
not
a
real
request,
it's
a
vertex
event,
and
then
we
see
300
here
and
then
700
years.
So
we
have
a
notion
of
what
of
the
importance
or
so,
if
I
fix
something
here
performance,
how
does
it
affect
the
whole
stack
right?
A
A
But
Greg,
okay,
so
now
the
first
goal
was
to
get
a
screen
like
this
right
so
place
order.
So
remember
that
code
that
code,
that
we've
seen
adding
a
span
tag
called
place
order
like
transaction
place
order.
So
this
is
the
server
looking
into
that
information
and
then
making
aggregations
for
us.
So
we
know
that
forty-three
percent
of
the
falls
are
item
up
found.
So
perhaps
we
have
a
broken
link
in
our
store
right
or
out
of
stock.
A
A
So
in
that
only
with
a
tracing
data,
so
by
just
adding
a
couple
of
lines
of
code
into
our
business
application,
we
are
able
to
derive
information,
I
Brigade
and
that's
the
kind
of
information
we
can
extract
from
of
its
racing
all
right.
So
it's
not
only
logging
stuff
for
sending
stuff
or
determining
problems,
but
you
can
also
extract
valuable
business
information
from
that
and
for
almost
for
free
all
right,
you
just
add
cold
sheriff
occasion
and
yeah
now.
A
The
second
goal
was
to
Jake
Henry
release
right,
so
we
are
going
to
try
that
so
we
have
the
account
manager,
which
is
one
of
those
open
shift
applications
here,
so
account
manager.
What
we're
going
to
do
is
we're
going
to
create
a
new
application
here.
So
when
you
application
called
a
complement
of
your
blue
as
in
Greenville
deployment-
and
we
will
just
scale
up
this
one
too-
that's
it
for
now
and
then
then
you
one
with
just
me
responsible
for
twenty
percent
of
the
request,
meaning
it's
one
part
in
five
right
and.
A
A
A
Ya,
a
calm
manner
blue
that
sounds
right.
Then
we
start
a
new
build.
So
what
I
have
here
is
a
simple
project
that
can
be
deployed.
An
open
shift
in
this
case
here
just
have
a
blocker
file
and
that
that
means
when
I
do
a
start,
build
to
just
send
the
whole
directory
to
open
shapes
up
and
shipped
to
build
it
based
on
information
from
that
docker
file,
blue,
okay,
and
from
the
current
directory
and
inform
the
logs.
Yes,
it's
fast
as
I
beauty
before.
A
It's
not
it's,
usually
not
that
fast,
okay.
So
now
we
should
see
the
build,
an
open
shift,
but
we
have
no
applications
yet
using
that
good.
So
the
blue
and
doing
that
one.
So
you
know
which
is
doing
you
up
your
app
on
account,
manager,
blue
and
then
some
hopper,
a
PM
environment,
variable
that
looks
fine
and
we
should
see
a
counter
here,
blue
appearing
here.
A
A
We
can
compare
data
so
here
we
can
see
that
so
this
is
a
an
Operations
person
looking
at
the
data
right.
So
it's
not
the
business
person.
The
business
person
was
in
the
other
cab
or
consuming
that
data
somewhere
else-
and
this
is
again
not
the
UI-
is
not
important
here.
The
data
is
the
important
part,
and
it
might
have
notes
that
I
use
only
the
command
line.
A
A
Have
fewer
resources
than
we
need,
so
the
blue
one
we
scaled
24
and
the
old
one
is
killed
once
we
invert
that
manager
and
on
openshift
we
should
see
so
in
order
that
I
mostly
use
the
command
line,
because
if
I
can
use
a
common
line,
that
means
I
can
automate
it
and
that's
important
to
me.
I
mean
again
important
the
data.
So
if
I,
if
I
have
a
system
that
reads
data
the
same
system
could
determine
whether
or
not
to
scale
up
or
down
my
application
in
this
case
here.
A
So
we
now
have
the
old
version
with
serving
a
fewer
requests
than
the
new
version.
The
new
version
is
scaling
up,
but
both
are
still
running
parallel
and
if
we,
if
we
say
that
yes,
it's,
it
looks
fine
with
ten
percent,
it
looks
on
it.
Sorry,
it
looks
fine
with
twenty
percent
and
it
looks
fine
with
eighty
percent.
Then
it's
safe
to
just
turn
off
the
old
one.
A
A
Yep
and
I'll
help
ensure
you
can
see
that
we
have
no
part
here
and
five
pots,
so
that
means
the
green
service
is
now
ready
to
receive
the
new
version
or
a
new
leu
version,
and
then
the
game
starts
again
right.
So
I
can
deploy
a
new
version
on
this
one
and
increase
the
number
of
plots
to
one
and
the
result,
the
other
one
to
four
and
so
on
and
decorations
person
is
empowered.
Now,
with
information
enough
information
to
make
a
decision
letter,
a
new
version
is
good
or
not
and
yeah.
A
A
If
you
have
any
questions,
any
comments,
any
suggestions,
if
you
wanna
help
it's
on
a
personal,
open
source
project,
so
just
join
us
on
IRC
at
the
hopper
channel.
If
you
have
ideas,
use
case
is
more
than
welcome,
and
tomorrow
we
have.
Then
this
distributed
tracing
salon.
So
if
you
wanna
practice,
open
tracing
and
stop
by
and
if
you
have
other
questions,
then
the
ones
I
don't
know
if
we
have
enough
time
here
for
a
lot
of
questions,
but
I'm
at
the
redhead
loose
so
just
applying
yeah
questions.
A
Well,
there's
an
overhead
period,
so,
no
matter
what
monitoring
system
you
use
or
what
tracing
traits
are
used,
there's
always
an
overhead.
There
are
techniques
to
reduce
the
overhead
to
a
level
that
is
acceptable
to
your
business
right.
So
just
log
information
is
not
it's
not
that
much,
but
it
depends
on
the
tracer
implementation,
so
the
so.
If
no
tracers
out
there
are
good
for
you.
A
You
can
write
your
own
traitor
just
login,
so
there
are
open
trace
implementations
where
you
can
specify
same
playing
strategy,
for
instance,
so
you
want
to
log
only
ten
percent
of
the
requests,
or
only
request
that
reach
a
certain
code
path
and
so
on.
So,
but
there
is
an
overhead
I
guess
it's
not
that
expensive,
but
it
all
depends
on
the
use
case.
A
A
as
far
as
I
remember-
it's
not
it's
not
blocking,
because
it
just
puts
on
a
queue,
and
then
there
is
a
batch
trace
publisher
that,
on
the
background
and
another
thread
sends
data
to
rate
them
server,
so
that
specific
call
is
not
blocking,
but
that
doesn't
mean
it's
not
doing
something
right,
so
it
might
run
out
of
memory.
Frances.
A
A
Like
and
react
that
with
others
and
do
some
actions,
for
instance,
increase
the
number
of
my
yeah,
so
no
I'm,
so
answering
for
a
pirate
game
specifically
know
is
they're
talking
for
open
tracing
notice.
It's
on
the
scope
of
open
tracing
right,
so
you
could
have
a
system
that
is
reading
your
data
and
acting
on
that
so
I
guess.
The
important
part
here
is
that
you
have
access
to
the
data
to
the
Roger
right
exactly.
A
A
So
you
do
need
some
part
of
that
on
the
application
scope,
because
so
there
are
two
things
here.
So
one
is
that
the
framework
part
in
the
communication
part
and
that
could
be
in
a
sidecar
deployment
or
it
could
be
any
proxy.
So
you
have
a
proxy
in
front
of
your
application
and
that
process
is
doing
the
tracing
tracing.
But
again
this
is
an
infra
monitoring,
the
infra
right.
So
you
can
extract
much
business
information
on
that.
A
So
there
there
is
value
in
building
this
information
inside
your
application,
your
business
code.
So
in
your
business
called
you
a
notice
saying
this
is
something
that
I
care
about.
This
is
an
operation
that
I
don't
care,
or
this
is
does
that
answer.
I
mean
there
are
some
discussions
there
on
doing
sidecar
deployments
or
sidecar
yeah
deployment.
You
are
so
you
deploy
your
application
in
your
application.
Instead
of
contacting
an
external
one,
directly
contacts,
the
sidecar,
which
then
like.
A
Right
so
I
would
say
which
opens
racing.
The
possibilities
are
unlimited,
so
you
can
write
your
own
tracer.
That
does
that,
and
it
is
something
that
have
been
discussed
in
for
opens
racing.
So
the
thing
is
there
are
so
many
things
to
do
right
now,
right
so
to
instrument
all
the
frameworks
out
there,
and
so,
if
you
read
a
use
case
that
is
important
to
you,
then
join
the
community.
Absolutely
yep.
A
Right,
each
tracer
has
a
different
technique.
Usually
what
it's
done
is
you
ambit
some
context,
information
into
the
metadata
to
your
downstream
downstream.
So
if
you
are
doing
HTTP
call,
for
instance,
then
you
add
HTTP
header
with
the
transaction
ID,
for
instance,
or
the
request
ID,
or
something
like
that,
if
you're
doing
a
JMS
or
a
vertex
code,
you
just
call
vertex
Collins
code
you'll.
Add
that
as
a
vertex
message
metadata,
you
have
to
do
that
somehow
right
so.