►
From YouTube: OpenShift Commons Briefing #133: Bringing Microservices Under Control Steve Ross-Talbot (Estefet)
Description
Controlling Chaos: Bringing Microservices Under Control
Guest Speaker: Steve Ross-Talbot is the Chief Technology Officer at Estafet, a specialist integration consultancy based in London, UK and in Sofia, Bulgaria.
As our services shrink in size so the interactions between them become the dominant architectural norm and coupled with compensating transactions for eventual consistency so the complexity multiplies. Never has then been a more urgent need to bring such chaos under control. And this is what OpenTracing and Distributed Monitoring though tools like Zipkin, Hawkular, Prometheus and Jaeger bring.
In this session, we shall see how to leverage OpenTracing and Distributed Monitoring.
A
B
Okay,
thanks
for
coming
to
this
OpenShift
commons
briefing
was
now
controlling
chaos
and
mike
services.
My
name
is
steve
ross
talbert
and
from
a
company
called
esta
FA,
where
I'm
the
CTO
and
today
what
I
want
to
do
is
talk
about
some
of
the
challenges
of
building
microservice
architectures,
some
of
the
design
choices
that
we
face
and
some
of
the
new
technology.
B
That's
that
has
come
to
the
fore
in
the
last
two
or
three,
maybe
four
years,
but
starting
to
find
real
success
in
how
we
deliver
micro
services
over
a
period
of
time
and
dealing
with
the
chaos
that
often
meant
to
so
what
I'm
going
to
do
is
walk
you
through
some
basic
stuff
for
the
high
level
bit
of
history.
How
do
we
get
here?
B
B
So
the
challenge
really
is
that
how
do
we
make
order
of
the
chaos
that
that
is
useful
when
we
start
building
micro
services,
because
if
you're,
building
small
things
and
deploying
them
continuously,
it's
pretty
hard
to
track
everything
and,
as
you
go
above,
10
micro
services,
it
starts
to
feel
like
visa.
This
little
icon
here
of
strings
intertwined
and
what
we
really
want
is.
We
want
some
kind
of
order.
B
They
went
to
a
micro
services
architecture,
they
turned
off
their
old
j2ee
legacy,
style,
architecture
and,
and
the
thing
didn't
work
they
expected
so
a
lot
of
people
weren't
able
to
get
money
out
of
the
bank,
and
this
has
caused
a
lot
of
consternation
and
what
I've
heard
fairly
directly
is
that
they
tried
everything
to
try
and
dynaTrace
their
try
that
dynamics
they've
tried
every
monitoring
tool
available
on
the
planet.
They've
got
Splunk,
but
it
wasn't
really
giving
them
what
they
wanted.
B
What
they
really
needed
was
to
understand
the
transactional
trace
through
the
distributed
architecture,
so
a
passes
to
be
passes
to
see
who
passes
to
D
and
E,
and
then
then,
all
that
comes
back
so
trying
to
make
that
order
from
the
chaos
that
ensues
is
essentially
what
Jaeger
and
open
tracing
gives
us
a
shot
at
and
as
you'll
see
as
we
go
through.
This
becomes
ever
more
important
in
the
world
of
Micra
services.
So
probably
do
that.
B
I
just
want
to
talk
a
little
bit
about
history,
I'm,
pretty
old
by
comparison
to
most
of
my
colleagues
and
I'm
much
new
at
the
end
of
my
career
than
the
beginning
and
I've
seen
a
lot
over
the
history
of
computer
science.
It
wasn't
quite
there
when
Island
cheering
was
there,
but
I
was
born
perhaps
about
five
years
after
Alice
he
he
he
died.
So
if
we
tracked
back
to
the
1960s,
which
is
kind
of
when
I
started
really,
it
was
a
world
replete
with
mainframes.
B
B
And
then
the
desire
for
structure
comes
in
when
we
see
Algol
60,
initially
another
eight,
and
then
we
get
really
simple
languages
that
come
to
the
fore
like
the
C
programming
language
that
led
to
the
rise
of
UNIX
operating
system
and,
of
course,
in
the
1980s
that
we
get
to
want
to
pcs.
With,
with
the
Macs
coming
out
in
force
and
mini-computers,
we
get
Ethernet
and
this
more
distributed
model
of
computing.
That
involves
sort
of
client-server
and
yeah.
A
B
Of
you
that
around
in
the
80s
and
programming
I'm
sure
will
remember
all
sorts
of
things
like
core
Burren
and
message.
Buses
and
all
of
this
stuff
and
and
methodologies
changed
a
little
bit.
We
get
SSA
diem,
we
get
chunks
and
Jack's
and
structured
programming,
various
others.
More
talk,
arise
and
C++
arise.
B
Those
than
the
small
talk
community
would
probably
issue.
The
C++
community
is
not
being
truly
oo
because
it
was
kind
of
like
C
on
acid
or
steroids.
However,
you
want
to
look
at
it
when
a
small
talk
was
much
more
of
a
proper
object-oriented
language,
in
the
sense
that,
rather
than
being
fascinated
by
polymorphism,
it
was
much
more
fascinated
by
interaction
which
comes
back
in
the
world
of
micro-services.
B
B
W3C
is
starting
to
emit
standards
like
like
there's
no
tomorrow
as
its
Oasis
and
all
the
others.
We
get
the
rise
of
grids
and
virtualization
laptops
all
over
the
shop
and
racks
and
and
and
the
birth
of
things
like
service-oriented
architecture
as
an
idiom
and
and
really
the
birth
of
agile
is
a
generic
delivery
method
and
scrum
alongside
it,
and
we
get
additional
languages
largely
driven
because
of
the
web
tier
and
with
JavaScript
and
Python,
because
you
know
data
and
actually
automating
things.
B
Scripting
becomes
kind
of
an
interesting
place
to
be
because
we're
becoming
more
distributed.
So
control
of
the
distributed
pain
that
we're
executing
things
on
and
delivering
into
becomes
a
little
more
tricky
so
in
languages
like
Python
and
come
to
the
fore.
So
when
we
look
at
the
2010
and
then
to
where
we
are
now
2018
we've
got
the
birth
of
cloud.
B
Nothing
much
has
changed,
I
mean
micro-services,
you
might
consider
to
be
to
some
extent
a
methodology,
but
really
is
predicated
on
two
things:
one
is
agile
delivery
and
the
other
is
this
sort
of
boundless
cloud-based
execution
that
allows
you
to
scale
to
internet
size
and
meanwhile,
in
the
languages
domain
we
get
the
birth
of
go
oddly
enough.
Openshift
and
yoga
are
both
written
and
go,
and
we
get
now
very
recently
the
birth
of
something
like
rolling
as
a
really
really
good
smart
contracts,
language
for
blockchain,
which
may
have
other
applications.
B
So
the
world
that
we
live
in
is
got
infinitely
more
complex,
much
harder
to
deal
with
in
terms
of
being
able
to
point
to
where
your
application
is
running
and
pretty
much,
all
we
can
do
is
to
sort
of
a
look
at
the
application
as
a
distributed
system
in
terms
of
its
resource
consumption
rather
than
semantically.
What
is
it
actually
doing?
You
know
what
is
it
doing,
an
international
payment
and
what
are
the
constituent
parts
of
doing
such
a
a
transaction?
So,
on
the
one
side
it's
become
easy
to
deploy.
B
We
don't
have
to
wait
too
long
for
an
environment,
but
on
the
other
side,
it's
become
very
tricky
to
manage
the
complexity
that
ensues,
because
things
are
just
much
more
connected,
so
microservices
as
an
example.
So
if
we
talk
now
about
an
example
to
illustrate
so
look
the
concepts
of
distributive
monitoring
generally,
when
you
do
a
micro
sees
delivery
if
you're
coming
in
at
the
beginning,
you're
starting
to
gather
story
cards.
So
in
the
example
that
I'm
going
to
show
our
business
domain
is,
how
do
we
manage
projects
print
or
is
tasks
stand?
B
How
do
we
look
at
the
burned
ends,
so
you
see
a
set
of
story
cards
as
a
manager.
I
want
to
be
able
to
see
a
burn
down
chart
of
my
sprints.
This
is
fairly
typical
and
we're
using
this
example,
specifically
because
it's
probably
familiar
to
most
of
you
listen
to
this
this
webcast.
So
the
thing
about
the
microservices,
of
course,
is
on
the
other
side.
Is
we're
looking
for
independence
of
deployability,
so
we
don't
want
to
have
to
deploy
everything
in
one
go.
We
want
to
be
able
to
pick
and
choose
what
we
upgrade.
B
What
we
change
as
and
when
we
we
need
to-
and
we
don't
really
want
to
have
to
have
loads
of
stage
gates,
so
we
want
it
more
automated
into
production.
We
want
to
protect
ourselves
from
change
with
service
contracts.
We
want
to
decentralize
decision-making
as
to
who
owns
the
Sprint's
versus
who
owns
the
tasks
versus
who
owns
whichever
Micra
service,
and
to
do
that
we
would
establish
cross-functional
teams
and
that
would
include
the
a
product
manager
who
represents
the
users
and
we
deliver
this
in
an
agile
way.
B
So
this
is
just
an
example
of
four
story
cards
that
drove
this
particular
example.
So
when
we
look
at
the
example,
we
start
to
you,
imagine
you
in
an
in
Section
the
first
sort
of
week
or
two
of
the
project's
life.
You
might
start
jotting
on
the
board.
Well,
you
know
one
of
the
fundamental
business
domain
objects,
so
we
might
say,
hey,
look,
we've
got
a
project.
Projects
are
made
up
of
Sprint's,
sprints
has
stories,
stories,
have
tasks
and
stories
have
acceptance
criteria.
B
So
every
time
you
have
a
sprint,
there
might
be
one
or
more
stories.
Each
story
may
have
several
tasks
and
they
have
a
status
which,
for
a
story
or
a
sprint,
might
be
whether
is
whether
it's
been
started,
whether
it's
been
completed,
whether
it's
not
started
yet
and
those
become,
if
you
like,
the
state
behavior
of
the
the
business
domain
model
objects,
so
I
might
have
something
that
looks
very
much
akin
to
this
and,
as
we
start
to
sort
of
look
at
how
we
would
actually
build
that
as
microservices
we'll
see.
B
First
of
all,
how
would
you
set
it
up
if
you
need
that
independence
of
deployability-
and
this
is
what
open
shift
is
particularly
good
at
promoting,
so
it
makes
it
easy
to
build
these
microsomes
'as.
We
look
at
how
to
set
that
up
on
the
on
the
pin
in
github.
You
actually
establish
a
github
repository
for
each
and
every
one
of
the
micro
services
so
that
you
have
that
independence,
because,
at
the
end
of
the
day,
OpenShift
we'll
work
on
the
basis
of
the
get
hooks.
B
So
when
you
commit
any
one
of
these,
if
I
change
the
esta
FA
Mike
services,
scrum
API,
sprint
micro
service,
independently
of
all
the
others,
the
get
hook
would
be
registered
with
open
shift
and
it
would
pick
up
the
changes
and
automatically
go
through
a
CD
CI
pipeline
and
so
that
you
can
see
what
was
going
on
live
as
the
thing
starts
to
get
deployed.
So
the
project
setup
becomes
quite
important
in
terms
of
how
you
how
you
achieve
the
independence
of
the
boiler
ability.
B
If
you
get
this
wrong,
then
your
independence
of
deployability
will
not
be
as
as
high.
So
when
you
think
about
these
mike
rizzo
cystic
these
domain
objects
and
then
the
micro
services
that
might
represent
this
architecture
shows
how
we've
structured
the
project.
So
what
we
went
for
was
a
basic
UI
that
so
you
can
interact
with
the
data.
B
We
may
pass
messages
to
inform
one
particular
micro
services,
the
state
change
in
our
micro
service.
So
in
this
way
it
becomes
easier
to
carve
up
the
architecture
and
understand
who
owns
what
data
and
who
is
interested
in
what
data,
which
is
a
very
different
thing,
and
then
at
the
bottom
of
this,
we've
got
sort
of
a
some
sort
of
database,
which
is
progress
in
our
instance,
and
then
they
have
all
their
own
logical
partitions
or
they
might
have
one
database
per
Mike.
Rizzo
is
physically
and
and
that's
kind
of
power.
B
One
might
look
at
the
microsomes
is
example.
So
the
way
that
this
works
is
that,
for
is
that
you,
you
end
up,
writing
taking
stuff
from
your
JIRA
and
end
up.
Writing
your
Java,
your
JavaScript.
You
go
whatever
language
you're
using
for
your
micro
services,
committing
that
in
the
github
and
then
your
QA
guys
are
writing
solar,
cube
scripts
and
cucumber
bits
and
selenium
scripts
and
your
DevOps
guys
writing
the
answer
ball
to
actually
implement.
B
If
you
like,
the
life
cycle
of
how
you're
going
to
do
your
build,
deploy,
test,
validate,
run
and
lifecycle
from
the
point
at
which
you
start
cutting
code
to
the
point
at
which
it
is
deployed
and
runs
in
in
an
environment
and
all
that
gets
into
github.
Somehow
ansible
is
responsible
for
to
some
extent
orchestrating
how
different
things
get
get
put
into
open
shift
in
which
particular
containers
or
pods,
and
they
go
into
and
the
github
once
all
that's
established,
the
github
hooks
are
just
used
to
enable
source
to
image
that
everything
can
run.
B
So
what
we
going
to
look
at
specifically
is
we
don't
have
the
java
script
and
go
stuff
we're
going
to
concentrate
on
the
Java
github
to
docker
ansible
side,
rather
than
look
at
all
of
the
tests,
because
if
you
just
download
this,
you
can
see
the
test
run
in
yourself
and
pop
them
out
of
the
github
butt
and
see
how
they
constructed
too.
So
if
you
go
to
the
github,
you'll
see
cucumber
tests,
you'll
see
some
selenium
tests
and
so
on
and
so
forth.
B
So
what
I'm
going
to
show
you
first
is
a
small,
a
small
movie
of
just
how
we
get
this
project
up
and
running.
So
this
small
movie,
if
I
press
play
what
you've
got
here,
is
what
we're
seeing
is,
is
one
of
the
pods
being
deployed.
So
this
is
the
answer
script
to
deploy.
You
can
see
loads
of
stuff,
going
up
on
the
screen
and
and
with
with
within
that
within
this
particular
smoothie
segment,
we're
seeing
everything
from
starting
mini
shift
from
scratch.
This
would
be
true
with
open
shift.
B
B
And
and
within
that
you
can
look
within
open
shift
and
see
all
of
the
projects,
so
you
can
see
that
there's
lots
of
option
allottee
with
an
open
shift
itself.
You
have
to
be
careful
about
your
your
IP
address
and
the
port
number,
because
when
you
actually
download
this
and
and
play
with
the
example
which
includes
open
tracing
in
iaeger,
you
need
to
actually,
as
you
can
see,
I
did
this
for
Centrica
before
you.
You
need
to
make
sure
that
the
VARs
match
the
IP
address
and
the
port
number.
B
B
We
can
see
that
the
pods
already
up,
it's
already
got
an
endpoint
address
and
there's
a
project
API
that's
up,
and
you
can
see
that
at
the
moment
the
project
burned
down
is
being
deployed.
So
this
is
a
real
DD
CI
pipeline
with
all
of
those
tests
being
run
as
if
it's
a
real
project
that
somebody's
really
paying
for
rather
than
a
small
project.
B
Just
to
illustrate
a
small
point,
and
that's
because
in
doing
this,
what
we
wanted
to
do
from
a
nest
of
a
perspective
was
to
share
how
you
do
this
end-to-end
and
the
role
that
open
tracing
and
Jager
can
play
subsequently.
So
if
you
have
no
code,
I've
been
tracing
in
Yeager's,
not
of
any
use
to
you,
because
there
is
no
code
to
instrument.
There
is
no
code
to
look
at,
but
the
minute
you
have
stuff
and
you're
starting
to
deploy
it.
B
That's
when
this
stuff
comes
into
its
own,
but
I'm
gonna,
I'm,
just
gonna
move
on
to
the
next
step.
The
next
slide
and
look
at
what
happens
when
you
have
a
bunch
of
micro-services.
Well,
how
do
you
decompose
them?
Because
what
the
temptation
is
often
is
to
look
at
them
and
think,
oh
well,
these
three
mics
OSes
share
kind
of
the
same
notion
of
it
objects.
B
They
will
make
that
mate
and
share
the
database
so
often
within
the
bounded
context
that
you
use
you'll
have
a
shared
database,
and
this
is
the
classic
mistake
in
building
mic
services,
because
what
you've
got
at
this
point
is
schema.
Looking,
so
the
blue
ones.
You
can't
change
the
schema
without
impacting
potentially
any
of
the
other
blue
ones,
and
the
red
ones
are
the
same,
so
they're
bound
to
the
schema
which
makes
independence
of
deployability
rather
than
a
nonsense,
because
you
can't
really
you
can't
really
do
it
very
well.
B
So
the
alternative
is
to
move
to
an
interaction,
centric
model
where
everyone
has
their
own
store,
and
that
requires
you
to
think
a
little
bit
more
because
now
you've
got
to
think
about
what
data
do.
I
am
I'm
responsible
for
what
data's
away,
update
versus
what
data
I'm
interested
in
so,
for
example,
in
our
sprint
example,
a
project
is
interested
in
everything.
That's
within
a
project
object,
we
might
consider
it
that
way
and
a
project
may
have
sprints
and
the
Sprint
is
also
interested
potentially
in
projects.
B
So
we
can
have
an
asynchronous
notification
between
them,
though,
that
project's
look
after
projects
and
sprints.
Look
after
Sprint's
now
there's
some
challenges
in
doing
that,
which
is
how
we
synchronize
things,
though,
in
order
to
do
that
synchronization,
it
means
we've
got
to
emit
an
event,
though
this
is
an
event
with
an
architecture
and
that's
all
well
and
good.
B
But
now
something
goes
wrong
where
we're
forced
to
look
at
the
lock
files
of
every
single
one
of
these
micro
services
and
try
and
figure
out
what
went
wrong
as
something
went
wrong
and
that
what
can
go
wrong
is
many
fold.
It
could
be
that
single
Mike
reverses
come
to
sleep
for
some
reason,
and
so
it
needs
stimulating
to
come
back
up.
It
could
be
that
there's
timeout
Cascades,
as
you
add
more
scale
terms,
have
fallen
through
the
micro
services
and
timeouts
is
a
problem
that
come
back
to
because
Jaeger
really
helps
to
them.
B
But
it's
challenging
many
of
us
I'm
sure
of
log
files.
Look
put
them
on
that
in
the
boardroom
of
the
of
a
building
that
we're
working
in
and
had
several
of
other
of
our
colleagues
with
post-it
notes.
Looking
at
the
log
files
trying
to
find
a
transactional
trace
in
order
to
determine
the
root
cause
of
a
problem,
so
with
data
centric
architectures,
it
is
easier
to
do
that.
But
you
lose
an
awful
lot
in
terms
of
your
ability
to
change
in
rapidly,
and
so
it
has
a
negative
business
impact.
B
So
we
have
to
work
harder
in
an
interaction
centric
model
to
get
this
right
and
that
working
it
and
making
it
it.
That
also
makes
it
harder
to
understand.
What's
actually
going
on
and
as
we
know,
teams
don't
always
stay
together,
and
so
there's
also
a
need
to
be
able
to
discover
there's
structural
properties
of
some
distributed
system,
in
the
same
way
that
if
we
have
a
single
application,
we
might
put
it
through
to
bug
you
kind
of
want
to
do.
B
The
same
thing,
though,
that
that
loss
of
control
is
what
kind
of
leads
to
chaos,
because
we
can't
really
see
the
structural
properties
that
need
to
be
preserved
and-
and
that's
really,
the
fundamental
aspects
of
jäger
that
I'm
going
to
show
you
presently
and
at
the
end
of
the
day.
As
we
said,
you
can't
bring
order
to
what
you
don't
understand.
So
understanding
correctness
becomes
a
real
problem
and
requires
you
to
tick
off
and
look
at
the
log
files
in
this
way.
B
Here
we
go
so
here's
project,
six,
five,
nine,
oh
and
within
that
you
can
see
that
there's
a
number
of
them
I'll
just
go
back
a
second
and
hold
it
there.
Then
what
we've
got
they
go
back,
so
we've
got
a
number
of
within
this
project:
six,
five,
nine!
Oh
we've
got
an
axis
print,
which
is
number
three
and
within
there
within
that
sprint
there
are
a
number
of
other
sprints.
B
Then
there's
sprint
number
three
which
is
active,
and
this
is
completed
sprints
here,
which
is
sprint
number
one
and
sprint
number
two,
and
you
can
see
that
there's
a
whole
bunch
of
stories
associated
with
with
this
project
which
have
a
bunch
of
story
points.
So
when
we,
when
we
click
through
and
have
a
look
look
deeper,
we
can
look
at
story
538,
which
is
within
the
same
project
and
see
that
that's
completed
and
when
we
click
on
it,
we
can
see
you
precisely
what
tasks
there
are
on.
B
You
can
see
that
all
of
the
tasks
are
completed
and
that
there's
a
bunch
of
acceptance
criteria.
Don't
what
we
do
know
is
that
we
do
know
that
when
a
a
story
is
complete,
it's
because
in
part
it's
tasks
are
all
complete,
and
so
that's
where
you
get
the
two
completed,
so
you
get
completed
status
for
the
story
completed
status
for
all
the
tasks.
Then
what
we're
going
to
do
just
to
invoke
an
here
and
really
and
see
what
happens
when
things
go
wrong
with
the
microcircuits?
B
Is
we're
going
to
add
a
task
to
a
a
story
and
therefore
a
sprint
and
therefore
a
project
story
and
that's
already
been
completed,
which
you
shouldn't
be
allowed
to
do
because
it's
been
completed
so
when
we
try
and
add
a
task
here,
what
comes
back
to
the
UI
and
I
have
to
say
it's
not
particularly
helpful
and
I
have
seen
projects
really
do
it
this
way
and
then
realize
there's
a
problem
and
the
issue
when
there
is
a
problem
of
this
nature.
So
this
is
submitting
the
task
there's
something
that's
already
closed.
B
The
issue
that
we've
got
here
is
there
was
miscommunication
amongst
the
team,
and
so
what
we
get
is
some
horrible
error
like
this.
Is
a
white
label
error
page
that
says:
hey
look,
I,
don't
know
what's
going
on
here
got
nothing
to
tell
you,
but
it
failed,
and
sometimes
we
see
404s
and
500.
So
the
worst
thing
that
we
we
can
possibly
see
as
a
status
500
just
being
thrown
back
in
us
in
this
raw
form.
B
So
if
this
was
a
real
project-
and
we
were
doing
this,
we
would
try
and
figure
out
well
what
happened?
How
did
that
happen?
Where
do
we
need
to
go
and
fix
it,
and
this
is
just
a
simple
case.
This
is
just
the
simple
case,
so
oops,
let
me
get
back
in,
but
what
I'm
going
to
do
now
is
is
have
a
look
at
how
we
might
use
Jaeger
and
open
tracing
to
change
this
picture.
So
the
beauty
of
open
tracing
and
the
ager,
so
open
tracing
is
a
standard.
B
This
is
what
most
of
us
would
have
gone
through,
10
to
15
years
ago,
much
harder
now
with
micro
services,
because
they've
got
lots
of
log
files
and
what
you
want
to
do
is
pick
out
the
actual
lines
within
the
log
files
that
match
and
therefore
deliver
in
effective
business
process
trace.
And
this
is
the
idea
behind
Jaeger
and
open
tracing.
You
could
use
sitting
in
a
similar
way
that
open
tracing
is
the
real
enabler
Yeager's
a
way
of
displaying
it.
B
So
this
then
becomes
really
easy,
because
all
those
events
that
happen-
the
JMS-
calls
that
the
sends
and
receives
in
a
JMS
context
or
caused
to
the
database
gets
shown
you
a
Jaeger
trace
and
we'll
have
a
look
at
a
real
one
in
a
minute
and
and
and
that's
the
idea.
So
what
we're
going
to
do
now
is
have
a
look
at
the
actual
example
that
we
just
saw
so
when
we
saw
the
micros
is
running
no.
B
So
what
we're
doing
here
is
finding
the
traces,
though
we'll
find
the
traces
here
and
you'll
see
that
there's
a
basic
UI
that
has
what
it
says,
ten
spans.
So
a
span
is
like
a
trace,
though
this
is
kind
of
like
a
trace
through
the
microservice.
So
what
we
were
doing
so
in
this
instance
we're
looking
at
actually
how
how
we
created
a
project.
So
a
new
project,
though
listed
this,
would
be
akin
to
a
developer,
trying
to
understand.
What's
going
on,
as
opposed
to
necessarily
diagnosing
on
a
problem.
B
In
the
first
instance,
and
what
you
get
is
a
span
route,
if
you
like,
at
the
top
and
within
that,
you
can
see,
calls
that
are
made
to
other
micro-services
and
what
happens
in
in
those
microservices
too,
but
very
handy,
and
we
can
expand
and
we
can
contract.
So
when
we
look
at
the
the
new
submit
in
particular
so
submitting
a
project,
it
kind
of
gets
much
more
interesting
as
to
how
this
works.
Then
what
we've
got
here
is
at
the
top.
B
You
can
see
that
new
project
submit
is
there
so
when
you
press
the
new
project
button
on
the
UI,
this
is
what
gets
in.
This
is
what
this
is,
what
results
from
an
instrumentation
perspective,
so
we
can
see
that
there's
that
line
under
new
project
submit
underneath
that
what
happens
is
it?
Does
a
post
to
the
project,
create
the
new
project
and
within
the
project
microservice?
What's
it,
what
it's
going
to
do
is:
is
it's
it?
B
Well,
it's
not
what
it's
going
to
do
it's,
what
it's
done,
which
is
the
important
thing
so
what
it
did
was
it
issued
a
query
to
get
the
next
project?
Id
sequence.
Number,
it
then
sent
a
JMS
tend
to
anyone,
that's
interested
in
the
fact
that
I'm
creating
this
new
project
and
then
it
updated
the
database
that
add
the
project
into
the
database.
So
that's
the
order
that
things
went
in
so
we
can.
B
We
can
have
a
look
at
all
of
the
detail
of
this
and
what
Yaeger
is
giving
us
is
a
contextualized
view
of
all
of
this,
which
gets
really
interesting,
particularly
when
there's
errors
and
unequally
Yeager's,
giving
us
the
filings
on
all
of
this.
So
I
said:
I'd
come
back
to
timeouts
and
that's
kinda
how
I'm
going
to
do
it,
though
it's
kind
of
interesting
that
we
can.
We
can
look
in
more
details,
so
you
know
we
can
see,
for
instance,
the
query
going
up
going
off
here.
B
We
can
see
what
query
is
sense
of
what
query
comes
from
what
the
results
of
that
query
are
and
when
we
look
at
me
you
know
the
update
we
can
do
the
same
and
with
the
JMS.
What
gets
really
interesting
is
you
can
see
the
topic
that
it
was
sent
sent
on?
You
see
the
details
of
that
and
you
can
see
all
the
details
of
the
message
that
was
sent.
So
this
is
saying:
hey
look,
you
know,
I
created
a
new
project
and
here's
the
details
of
the
project
for
anyone.
B
That's
interested,
and
this
is
how
we
use
an
event-driven
architecture
which
historically
are
very,
very
difficult
to
debug.
This
makes
it
fairly
trivial
to
debug,
but
not
only
can
we
see
the
send,
we
can
see
they
receive,
though
you
know
we
can
see
the
send
on
the
one
side.
We
can
see
the
sprint
API
then
getting
a
message,
because
it's
subscribing
to
new
projects,
because
when
a
project
is
created,
it's
going
to
the
sprint.
B
The
sprint
micro
service
will
automatically
create
a
sprint
for
the
project,
and
so
it
that's
why
it
needs
to
know
the
project
IDs.
So
when
it
gets
the
message,
hey
I've
created
it
a
new
project.
It
says
all
I
need
to
create
a
sprint
for
that,
so
it
creates
a
new
and
you
sprint
for
that
and
then
so
it
cascades
everything
down.
So
you'll
see
the
same
pattern
in
the
left-hand
side
of
the
Jaeger
console.
Here.
B
Somebody
puts
their
finger
up
to
the
wind
and
sees
its
way
the
wind
is
blowing
and
says
I'll
set
mine
at
30
seconds
or
four
or
five
seconds
and
the
person
next
to
them
say
what
did
you
say
your
words
at
the
melt
set
it
the
same.
The
problem
is:
is
that
under
load,
when
you
set
that
in
that
way,
you'll
get
a
cascade
failure
of
timeouts
which
are
really
hard
to
diagnose,
because
you
can't
really
pull
everything
together,
but
in
this
in
this
example
with
Jaeger,
we
can
see
all
of
that
timing
information.
B
So
we
could
do
an
induction
of
failing.
We
can
induce
the
failure
and
see
the
impact
in
Jaeger
and
therefore
adjust
the
time
outs
before
we
actually
have
to
release
this
into
full
production
or
switch
our
a/b
testing
or
whatever
it
is
we
might
be
doing
so.
This
is
one
of
the
fundamental
powers
behind
behind
Jaeger,
in
that
it
gives
you
this
ability
to
trace,
through
everything
that
happens,
including
asynchronous
eventing,
using
Jamison's
and
recedes
and
all
for
the
cost
of
a
single
line
of
code
in
a
in
a
micro
service.
B
So
that's
kind
of
that's
kind
of
how
it
works
at
a
high
level
by
moving
olden.
Let's
have
a
look
at
how
we
get
diagnose
this
error,
though,
in
the
example
that
we
had,
we
added
a
task
to
a
sprint,
to
a
story
to
a
sprint,
to
a
project
that
we're
this
that
where
the
sprint
was,
was
actually
closed,
it
was
completed.
So
what
going
to
do
now
is
we're
going
to
we're
gonna.
Look
we're
going
to
do
some
find
some
fine,
Tracey
and
and
have
a
look
for
the
earth.
B
We
have
to
take
the
tags
off,
so
we
take
the
tags
off
to
do
an
open
search
and
then
we
find,
as
you
can
see
here,
you
can
immediately
see
sort
of
center
of
the
screen.
Aha,
my
basic
UI
has
two
errors
against
it.
Interesting,
let's
have
a
dive
now
we're
not
seeing
all
the
spans
we're
only
seeing
the
50
limit
result.
B
There
are
many
other
transactions
that
occurred
before
this
and
we
can
do
interesting
searches
which
I'll
show
you
in
a
minute,
but
let's
first
dive
into
the
error
and
see
where
did
that
error
occur.
So
when
we,
when
we
the
beauty
the
real
beauty
of
Jaeger
is
it
gives
us
effectively
not
just
a
distributed
debug
of
you
but
contextualize
logging,
so
we
don't
have
to
worry
about
which
log
we're
looking
at
it
comes
as
part
of
the
context
of
the
micro
services
that
we
look
at.
B
So
when
we
drill
into
as
you
can
see
here,
we've
got
our
basic
UI,
calling
the
tasks
API
to
add
the
create
the
task
which
it's
not
allowed
to
do,
and
we
can
see
a
little
icon.
That
suggests,
as
an
arrow,
which
is
the
relation
circle.
So
when
we
click
into
these,
we
can
see
actually
what
happened
so
rather
than
us
look
at
the
basic
UI
log
file
and
then
look
at
another
look
for
another
log
phone.
So
we
we
get
to
the
end
of
the
chain.
B
We
can
just
see
it
straight
away,
and
so
it's
takes
huge
amounts
of
time
and
you'll
see
here
that
you
know
it's
crying
out
at
us.
You
can
see
the
message
here
cannot
add
tasks.
The
completed
story,
I'll
underneath
you've
actually
got
the
log
file
statuaries
and
that's
situated
within
the
span,
which
makes
it
really
easy
to
debug
things,
and
we
can
look
up
and
down
that
and
see
what
should
have
happened.
B
So
this
is
really
a
huge
benefit
to
a
developer
when
they're
actually
developing
stuff
and
something
goes
wrong,
but
you're
able
to
use
Jaeger
in
effectively
as
a
distributed
debugger
that
allows
you
to
rectify
quite
complex
problems,
much
faster
than
you
would
otherwise
be
able
to
do
and
in
an
Esther
face.
This
has
now
become
de
rigueur.
This
is
what
we
do
when
we
do
mike
services.
So
we
don't
write
mike
services
without
open
tracing
and
without
the
ability
to
then
have
those
open
tracing
events
reported
back
through
Jaeger
wall.
B
Our
developers
today
use
this,
and
this
has
had
a
major
impact
on
their
lives,
makes
it
much
more
pleasant.
So
if
I,
if
I
sort
of
speed
this
up
a
bit,
we'll
have
a
look
at
see
how
we
can
we
can
change
the
we
can
change
the
the
number
of
traces
we
see
so
the
number
of
spans
we
see
by
changing
it
from
the
last
hour
to
the
last
twelve
hours
or
whatever.
B
How
if
we
want
to
do
it
once
my
apologies
I'll,
do
that
again,
oh
yeah,
so
we
can
change
the
M
and
we
can
change
the
the
banks
slightly
alright.
So
the
key
here
is
that
the
error
tag
there's
an
error
tag.
That's
set
to
true.
This
is
done
automatically
within
within
the
Spring
Framework.
You
can
have
other
tags
too,
so
you
don't
just
have
to
have
the
error
tag.
B
That's
that's
kind
of
like
a
built-in
tag,
but
the
beauty
is
is
that
you
can
find
all
of
the
traces
where
error
is
true
and
that's
in
under
the
tags
bit
up
here
where
it
says
error,
:,
true
now,
I
might
have
another
tag
that
says:
NIF
it
or
I
might
have
another
tag
that
says
GDP,
L
and
I
can
add.
My
own
would
have
called
baggage
tags
to
my
transactions.
B
So
the
sort
of
changes
that
you've
got
to
do
to
implement
this,
a
really
really
really
simple
I,
cannot
emphasize
how
how
simple
it
is
to
do
the
first
step,
the
real
skill
comes
in
understanding
how
to
use
baggage
tags
in
an
effective
way
and
the
real
skill
comes
from.
How
are
you
put
your
mic
receivers
together?
In
the
first
place
you
can,
you
can
get
the
best
kind
of
scaling
dynamics
and
the
best
independence
of
deployability
and
therefore
the
better
velocity
to
to
incorporate
change
over
time,
but
to
do
it
in
spring.
B
So
if
you
were
just
using
spring
there's
a
single
change
right,
so
that's
you
can
see
the
change
in
front
of
you
in
the
middle
of
the
screen
on
line
24,
there's
pretty
much
a
single
change,
so
you
could
imagine,
certainly
writing
a
script
that
would
go
through
all
your
for
your
existing
spring
based
Mike
services
and
adding
this
to
it
and
then
deploying
it
with
open
tracing
and
yoga.
But
as
I
say,
that's
the
first
step,
and
you
should
definitely
do
that.
B
But
the
second
step
is
really
to
send
sit
back
and
think
about
how
else
can
I
use
this
because
there's
way
more
power
and
in
using
open
tracing
in
iaeger,
then
simply
as
a
distributed
debugger
and
we'll
come
to
that
in
a
second
and
here's
a
another.
So
you've
got
the
the
basic
UI
Mike
service
and
you've
got
the
eight
the
microsomes
api
board
as
well.
So
there
is
only
one.
There
is
only
one
line
really
that
changes
which
is
on
line
25
in
both
instances.
B
So
it's
just
the
configuration
telling
it
it's
the
spring,
the
Sprint
board
or
its
two
basic
UI,
so
I'm
just
going
between.
So
it's
pretty
simple,
so
the
caches.
So
this
is
how
this
is
what
a
span
looks
like,
so
we
saw
it
as
as
line
and
stacked
on
top
of
the
other,
but
actually
there's
a
routes
fan.
So
the
basic
UI
has
the
reach
span
and
then
there
may
be
other
spans
of
the
basic
UI.
So
each
Micra
service
may
induce
a
new
span.
B
B
I
can
separate
spans,
so
there's
more
engineering
to
do
if
you
want
to
get
the
benefits
of
Jaeger
and
there's
some
skill
to
be
brought
to
the
table
in
terms
of
how
you
separate
your
spans
out
per
micro
service
and
that's
something
that
that
we
do
esta
fail
all
the
time.
So
this
is
kind
of
how
we
might
see
that
same
span
if
it
were
portrayed
into
or
if
we
we
saw
it
rendered
in
itself.
B
So
you
got
spell
a
which
is
the
longest,
and
then
you
got
span
be
fancy
which
are,
if
you
like,
peers
of
each
other's
bodies
inside
span
be,
and
then
you've
got
e,
F,
G
and
H,
and
this
is
how
the
spans
might
look
the
key
thing
what's
actually
in
a
span.
What's
that
the
actual
key
data
that
allows
us
to
make
the
causal
links
between
things
is
quite
it's.
B
It's
a
simple
list,
such
it's,
an
operation
name,
it's
a
start,
start
time
stamp
and
the
finish
time,
something
that's
how
you
get
the
telling
means
it's
a
set
of
zero,
more
key
values
span
tags
and
analogues,
though
that's
where
the
log
stuff
comes
from
and
a
span
context,
and
then
you
might
have
zero,
more
causally
related
spans
and
then
plus
baggage
tags,
Plus
baggage
tags
onto
that,
but
effectively
the
key
value
pairs
is
the
fourth
attribute
in
that
list.
That's
where
error
is
becomes
a
built-in,
so
that
happens
automatically
if
you're
using
spring.
B
So
when
we
look
at
baggage
tags
as
key
value
pairs.
So
if
I
had
a
a
baggage
tag
somewhere,
I
might
see
a
key
value
pair
added
to
the
span
of
C,
which
gets
propagated
to
E
and
propagated
to
F,
G
and
H.
So
this
is
why,
when
you
start
adding
baggage
bags
to
your
open
tracing
world,
you
have
to
be
conscious
of
that.
If
you
were
to
add
a
gigabyte
value,
that's
probably
not
a
good
idea,
because
that's
going
to
propagate
through
your
network,
I
think
your
network
will
go
very
slowly.
B
Don't
use
it
very
carefully
and
think
very
carefully
about
why
it
is.
You
would
want
to
use
the
baggage
tags
we
have
found
use
for
it,
certainly
for
things
like
identifying
mythic
transactions,
identifying
gdpr
transactions,
and
things
like
that.
So
it
has
a
relationship
oddly
enough
to
the
world
of
compliance.
B
There's
many
opportunities
in
in
this
world
of
open
tracing
Jaeger
and
we're
a
testimony
working
quite
a
lot
with
Red
Hat
and
the
the
Jaeger
team
within
Red
Hat
that
Gary
Brown
leads
and
we
working
with
a
bunch
of
academics
to
do
some
really
cool
stuff.
So
some
of
the
stuff
behind
dopant
racing
in
iaeger
came
from
to
some
extent
came
from
a
paper
that
received
an
ACM
best
paper
if
the
last
decade
award
in
January,
and
so
it's
usually
about
ten
years
when
something
comes
into
production.
B
So
this
is
about
right
from
the
way
that
that
generally
academic
excellence
has
reached
business
our
world
of
business.
So
it's
about
that.
Ten
years
now-
and
there
are
many
opportunities-
some
of
them-
it's
not
just
distribute
monitoring
for
developers,
so
understanding
legacy.
So
we
we
can
consider,
given
that
we
know
what
a
what's
needed
for
a
span,
what
the
attributes
are.
We
could
consider
thinking
of
the
legacy
system.
B
That's
not
been
instrumented
at
all
in
terms
of
its
log
files
and
therefore
unpick
it
and
turn
it
into
open
tracing
events
and
see
how
it
looks
in
iaeger
might
be
useful.
Certainly
it
would
provide
us
with
enough
upfront
knowledge
to
at
least
understand
some
of
the
key
transactions
equally
process,
lineage
forum
for
method,
two,
so
understanding
that
a
transaction
is
truly
compliant.
So
if
you
imagine
your
baggage
tags,
the
missing
to
all
having
a
tag
that
says
method
two
and
then
within
the
method,
two
at
the
end
of
it.
B
If
we
add
to
the
end
of
the
word
method,
two
additional
aspects
to
a
particular
business
process
that
we're
interested
in.
If
you
add
that
as
part
of
your
development
story,
then
you
become
automated
in
terms
of
how
you
can
show
compliance
permitted
to
or
the
same
would
be
true
of
any
kind
of
compliance
process
that
that
you
want
to
put
there,
and
that
gives
us
a
whole
bunch
of
other
things
that
we
can
potentially
do
with
open
tracing
and
with
Yaeger.
That
thus
far
have
yet
to
be
turned
into
true
products.
B
But
it
is
something
that
that
we're
working
on
with
the
Yaeger
team
and
with
with
the
academic
institutions
that
we've
been
working
with
over
the
last
couple
of
years.
So
there's
many
opportunities,
so
I
kind
of
just
were
summarized
as
to
you
know
what
would
be
talking
about
so
hopefully
giving
you
some
guidelines
as
to
how
to
build
good
Micra
services.
There's
a
lot
out
there
on
on
Google
and
on
YouTube
that
tell
you
how
to
do
bounded
context.
They
tell
you
how
to
that.
B
If
you
wished
and
equally
we
wanted
to
be
able
to
have
you
run
it
on
a
single
box
or
in
an
Amazon
or
as
your
environment
doesn't
matter.
As
always.
Open
shift
is
the
consolidating
platform
as-a-service,
it
all
works,
and
we
want
it.
Also
to
give
you
a
view
as
to
how
open
tracing
in
Jaeger
really
helps
to
diagnose
problems
and
show
you
some
additional
use
cases
there
might
be
a
value,
so
I
hope
we've
covered.
B
All
of
that
I
do
want
to
say
thank
you
to
for
Diane
for
organizing
this
say,
thank
you
to
Gary
Brown
for
continuing
to
work
with
us.
After
all
these
years
and
I
want
to
say
thank
you
in
particular
to
Dennis.
Williams
who's
been
my
right-hand
man
on
putting
this
presentation
and
the
demo
together.
Please
come
to
our
github.
If
you
want
to
contact
me,
it's
very
easy.
B
A
Steve,
thank
you
very
much
for
a
wonderful
presentation
and
really
practical
good
advice
on
using
open
tracing
in
Yaeger.
I
love
the
part
about
debugging
with
open
tracing,
and
it's
just
that
it's
become
sort
of
the
de
facto
way
of
things
working
in
a
testified
and
I.
Think
a
lot
of
other
places
will
start
using
it
as
well.
I
encourage
everybody
to
take
a
look
at
the
week
load
that
he's
got
down
there
as
well
as
take
a
look
at
the
the
Jager
repo
and
the
information
around
Jager
as
well.
A
B
A
Hat
working
on
it,
but
there's
a
good
crew
of
folks
that
are
helping
make
Jaeger
sort
of
the
de
facto
standards,
which
is
pretty
wonderful.
But
if
you
have
questions
now
would
be
the
time
to
ask
them
I'm
not
seeing
any
in
the
chat.
However,
though,
if
not
I
will
post
this
video
up
on
YouTube
shortly
along
with
the
slides
and
there
were
a
few
reference
links
as
well,
including
the
github
here4s,
to
thoughts,
wonderful
demo
and
I
will
let
everybody
get
on
with
their
days.
A
But
thanks
again,
Steve
and
I
look
forward
to
seeing
you
in
person
at
the
upcoming
London
gathering
on
June
love,
June,
January,
thirtieth
and
2019,
when
we
have
our
London
UK
based
regional,
open,
shaped
Commons
gathering
I'm
hoping
Gary
was
on
stage
last
time
on
one
of
the
panels.
But
like
a
showcase
and
more
of
this
that
event,
you
will.