►
Description
Keynote: Service Meshes and Observability - Ben Sigelman, Co-founder & CEO, Lightstep
Service mesh technology facilitates the discovery, interconnection, and authentication of microservices. While it’s straightforward to use a service mesh to measure peer performance, actually explaining the behavior of transactions in a microservices deployment requires distributed tracing.
In this keynote, Ben will explain why distributed tracing is important, where the service mesh comes into play, and how OpenTracing makes it all elegant and portable. We will illustrate these concepts with a live, audience-interactive demo, and provide guidance for those who want to add these technologies to their own microservice deployments.
About Ben Sigelman
Ben is a cofounder at LightStep, a company that makes complex microservice applications more transparent and reliable. Previously, Ben spent nine years at Google where he ate lots of snacks and designed several large (~1M-process) distributed systems. The most significant of these were Dapper, an always-on distributed tracing system; and Monarch, a high-availability timeseries collection, storage, and query system.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Good
afternoon
or
evening,
as
the
case
IV
I'm
been
single
men,
I
am
one
of
the
co-creators,
the
open
and
tracing
standard
and
I'm,
also
a
co-founder
and
the
CEO
at
light
step
where
we
deliver
insights
about
complex
production
software
systems.
So
I'm
here
to
talk
about
the
service
mesh
and
in
particular
how
we
can
make
better
sense
of
a
service
mesh
architecture.
I
will
also
be
talking
about
open
tracing
a
bit
observability
how
to
make
things
more
operationally.
Sane
and
I'm
I'm
really
excited
to
talk
about
it,
so
I'll
just
get
started.
A
This
is
a
picture
of
a
lot
of
birds,
lots
of
them
there,
hundreds
or
thousands
of
them.
This
is
a
murmuration
of
starlings
and
I
chose
this
picture
because
I
think
it's
evocative
of
the
vision
that
we
have
for
micro
services.
You
have
independent
actors
that
are
working
in
concert.
Each
one
is
sentient
and
has
its
own
agenda,
but
taking
together
their
greater
than
the
sum
of
their
parts
and
honestly
it's
beautiful,
like
the
idea
of
micro
services
is
beautiful.
I
was
at
this
conference
last
year
and
I.
A
A
A
This
is
how
it
feels
you're
on
stage
and
you're
micro-service
employment
you're
worried
that
it's
going
to
you
know,
have
a
bowel
movement
on
you,
while
you're
on
stage
so
operating
microservices,
it's
very
challenging,
very
challenging,
stop
it
and
it's
it's
for
a
lot
of
reasons.
Some
of
them
are
fundamental
things
that
we're
never
going
to
get
past.
Software
engineering
is
hard.
Everyone
here
has
a
lot
of
job
security
if
they're
on
that
side
of
the
house
as
a
result,
but
some
of
them
are
not
such
good
reasons.
A
I
think
that
we
often
find
herself
wanting
to
build
application
software
and
yet
what
what
we're
actually
doing
is
writing
the
same
code
over
and
over
again
to
do
something?
That's
actually
like
a
hard
computer
science
problem
and
should
be
factored
into
some
kind
of
common
interface
and
yeah.
You
know
we're
doing
it
manually,
so
cerebus
mesh
can
help
with
discovery
of
services.
Interconnection
of
the
services
circuit,
braking
load,
balancing
that
sort
of
thing.
These
are
hard
problems.
You
don't
want
to
solve
them
yourself.
Service
mesh
can
help
with
security
and
authentication.
A
You
want
to
make
sure
that
you
discover
the
wolf
without
a
lot
of
effort
or
a
custom
code,
and
that's
something
that
is
near
and
dear
to
all
of
our
hearts
as
well.
We
need
to
build
secure
things.
Security
should
be
built
in
from
the
firmaments
and
that's
what
service
mesh
can
offer
us.
What
I'm
really
here
to
talk
about
is
transactions,
though,
so
this
is
evocative
of
what
it
feels
like
to
look
at
a
transaction
and
the
typical
microservices
architecture.
A
Each
piece
each
service
is
written
in
a
slightly
different
way,
maybe
a
slightly
different
language
or
a
very
different
language,
different
idioms,
different
frameworks,
and
when
they
don't
work,
they
don't
work
in
spectacular
fashion.
It's
often
incredibly
difficult
to
figure
out
what
happened,
and
yet
we
have
to
do
that,
like
the
the
transactions
that
don't
work
are
the
most
important
they're,
often
stranger
than
fiction.
When
you
actually
do
the
post-mortem
and
figure
it
out,
but
you
want
to
do
you
want
to
understand
that
explanation
rapidly,
and
that
can
be
a
difficult
thing.
A
So
you
know
what
about
this.
This
is
my
favorite
tweet
of
all
time.
We
replaced
our
monolith
with
mica
sources
so
that
every
outage
could
be
more
like
a
murder
mystery.
This
is
this
is
something
that
that
we
need
to
move
away
from
is
an
industry,
it's
a
really
bad
feeling.
I've
had
it
I'm
sure
everyone
here
has
ever
been
on
calls
had
this
feeling
of
like
oh
gosh,
like
I,
don't
know,
what's
going
on
and
I
need
to
figure
it
out
quickly.
A
Micro-Services
should
not
be
getting
in
your
way,
and
yet
sometimes
it
feels
like
that's
exactly
what
they're
doing
so.
The
canonical
answer
to
that
problem
of
understanding
transactions
is
distributed.
Tracing
distributed
tracing
is
about
telling
stories
about
specific
transactions
across
distributed
systems
across
service
boundaries,
and
this
could
be
micro
services.
It
could
be
serverless,
it
could
just
be
a
bunch
of
monoliths
that
are
all
talking
to
each
other.
It
doesn't
matter
that
idea
is
that
you
need
to
stitch
things
together
and
develop
a
unified
picture
and
a
unified
explanation
for
the
behavior
of
individual
transactions.
A
What
you
see
here
is
on
the
left
is
Jaeger,
which
is
another
CN
CF
project
on
the
right
is
my
companies.
This
trace
viewer
both
of
these
are
open,
trace
and
compatible
they're,
actually
open
tracing
native
Zipkin
also
works
at
open
tracing
many
other
awesome.
Vendors
I
have
a
lot
of
respect
for
New
Relic
data
dog.
They
are
also
coupled
to
open
tracing
and
the
reason
why
we're
doing
this
is
because
we
shouldn't
have
every
developer
having
to
manually
integrate
into
distributed
tracing.
There's
too
many
touch
points.
A
There's
too
much
diversity
of
language
too
much
diversity
of
framework
to
have
everyone
integrate
with
everything
and
open
tracing
is
a
lingua
franca
for
describing
the
behavior
of
transactions
and
making
that
possible.
So
when
the
service
mesh,
you
know,
became
a
thing
really
quite
rapidly
and
has
taken
off
it's
a
natural
point
of
integration
for
something
like
hope
and
tracing.
So
let's
talk
about
transaction
tracing
without
a
service
mesh,
you've
got.
You
know
four
services.
A
A
There
are
many
points
of
integration
here
again,
if
you're
dealing
with
many
languages,
that's
a
lot
of
work
and
open
tracing
can
help
a
certain
amount
here,
but
gosh
it
would
be
nice
if
at
least
those
touch
points
between
services
could
be
factored
out,
and
yet
that's
exactly
what
the
service
mesh
gives
us.
So
too
many
points
of
integration
in
a
sort
of
smash.
It
looks
like
it
starts
off
the
same.
You
have
your
services,
but
now
you
have
these
side
cars
that
run
a
lot.
A
You
know
I,
either
on
the
way
in
or
the
way
out
on,
ingress
or
egress
from
your
services
and
service
meshes
at
level.
Seven
proxies,
so
it's
completely
reasonable
to
integrate
with
application
level
things
like
distributed
tracing
systems.
So
now
our
connections
go
through
the
service
mesh.
The
service
mesh
integrates
the
open,
tracing
and
open
tracing
integrates
sorry,
sorry
slide
problem
open
tracing,
integrates
with
many
different
vendors
and
open
source
tracing
systems,
as
well
as
other
tools.
A
A
Take
a
screenshot
of
one
earlier.
If
you
do
it
this
way,
this
is
not
what
we
ordered.
We
wanted
a
trace
that
has
a
lot
of
structure
and
tells
a
story
about
a
transaction.
We
wanted
something
that
looks
like
this,
where
you
have
a
shared
timing,
diagram
that
shows
many
different
services
and
how
they
interact,
but
somehow
we
had
this
instead.
So
that's
that's
not
what
we
wanted.
We
did
integrate
with
every
RPC,
but
this
isn't
what
we
wanted.
So
you
know
what
happened:
we're
not
so
victorious
after
all,
so
observing
every
RPC
is
great.
A
I
would
say
it's
necessary
and
service
mesh
gives
us
an
incredible
incredible
lever
to
do
that
and
I'm
incredibly
excited
about
it
personally.
I've
seen
what's
happened
at
lift
with
envoy,
they
were
able
to
stand
up
tracing
across
the
entire
system
of
hundreds
and
hundreds
of
services
with
it,
with
a
configuration
change.
Its
profound
honestly,
it's
incredible
to
watch,
but
you
do
need
to
tie
things
together.
Just
observing
the
edges
alone
is
not
sufficient
and
there's
a
couple
of
ways
to
do
this.
The
documentation
will
say
something
like.
A
Oh,
you
know
you
need
to
forward
the
header
from
ingress
to
egress
and
that
that
is
true
like
that
works.
It
can
be
easy
for
a
simple
service
if
your
services
are
more
complex,
that
they
have
queuing
and
be
in
the
middle
of
them
or
if
there's
some
kind
of
fan
out
fan
and
behavior,
it
gets
a
little
bit
harder
in
certain
languages
is
even
harder
than
that,
and
this
is
the
problem
that
open
tracing
was
designed
to
solve.
A
Open
tracing
was
designed
to
describe
that
sort
of
behavior
so
that
no
matter
what
sort
of
tool
or
what
sort
of
solution
you're
pursuing
you
can
see.
All
of
that,
so
we
can
use
open
tracing
both
of
the
service
mesh
layer
and
within
the
process
to
connect
these
dots
in
a
way.
That's
truly
end
n,
so
it's
you're
using
service
mesh
for
a
single
integration
point
at
every
RPC
boundary
and
reusing
open
tracing
within
the
process
to
propagate
all
of
this
state,
and
this
really
does
work
and
is
actually
a
really
wonderful
thing
in
practice.
A
The
cool
thing
here
is
that,
as
an
application
developer,
the
somewhat
cool
thing
is
that
you
haven't
coupled
yourself
to
a
particular
downstream
tracing
system
or
observability
tool.
The
really
cool
thing
is
that
you're,
not
writing
a
ton
of
code
yourself
to
integrate
into
tracing
and
if
you're
running
micro
services,
and
you
want
to
debug
things
quickly.
I
think
you
need
it.
I,
don't
think
it's
a
I'm,
biased,
but
I.
Don't
think
it's
something!
That's
optional,
I!
Think
it's
a
necessity,
and
you
know
when
we
integrate
this
open
tracing.
A
We
get
more
detailed
with
the
processes
as
well,
which
will
help
root
cause
analysis.
So
I
think
this
is
getting
a
little
bit
dry
even
saw
some
of
yawn.
The
birds
have
stopped
it's
not
as
interesting
anymore,
so
we
need
to
apply
this
to
something
in
poor,
something
with
food
coloring,
something
that
we
can
really
get
excited
about.
A
So
let's
talk
about
donuts,
so
I
want
to
introduce
donut
zone
I
like
doing
demos
involving
donuts
donuts
owns
their
motto,
is
move
fast
and
bake
things.
We
are
offering
a
Dass
donuts
as
a
service
scaling
really
fast
running
into
performance
problems
built
with
service
mesh
using
envoy
in
this
case,
and
also
built
with
open
tracing
and
so
yeah.
This
is
done.
Its
own
I'm,
not
gonna,
spend
a
lot
of
time
describing
architecture
here.
I,
don't
think
it's
necessary
for
the
purposes
of
this
presentation,
but
suffice
to
say
there
are
several
services.
A
There's
a
controller,
there's
a
service
that
fries
Donuts
there's
a
service
that
applies
toppings,
whether
they
be
sprinkles
or
chocolate
or
cinnamon,
and
there
are
external
dependencies
on
brain
Powell,
the
popular
payment
system
and
a
number
of
clients
web
clients
that
that
order,
donuts
and
restock
things.
So
at
this
point,
I
think
it's
time
for
a
live
demonstration.
I
will
go
over
here
for
that
all
right.
So
first
I'm
just
gonna
show
donut
salon.
This
is
a
slightly
same
idea.
A
different
domain.
A
A
If
you
go
to
the
bottom
here,
you
can
restock
donuts,
so
I
I
can
add
more
chocolate,
donuts
and
see
the
numbers
go
up
or
I
can
order
some
more
okay
great,
so
you
can
order
your
donuts
and
then
there's
some
asynchronous
JavaScript
and
XML
some
Ajax
here
that
allows
you
to
see
which
Donuts
you've
received
and
Oh
someone's
already
content
here
in
order
donuts,
you
Rascals,
you
can't
trust
anybody
these
days.
I
didn't
say
to
do
that
anyway.
So
I
can
join
here
and
in
Jaeger.
A
I
can
say:
I
want
to
see
things
that
took
at
least
half
a
second
and
indeed
here's
a
here's,
a
trace
showing
one
of
those
particular
requests.
This
one
resulted
in
error
because
we'd
run
out
of
inventory
due
to
whomever
an
audience
is
sneakily
going
to
this
domain
on
your
own,
but
you
can
see
a
propagating
across
this.
This
simple
distributed
system
as
we
top
the
donuts
and
fry
them,
and
so
on
and
so
forth.
A
So
you
know
it's
cool
I
do
actually
want
to
encourage
some
audience
participation
so,
but
I
want
you
get
a
doughnut
dot
zone
instead
of
Donuts
ilan
dot
zone
domains
have
not
been
fully
tapped
like
the
dot-com,
so
I
was
able
to
get
a
donut
domain,
even
if
it
was
a
dot
zone
instead
of
a.com.
So
I
encourage
you
to
go
there
and
order
things
on
your
phone.
I'll
I'll
do
the
same.
A
I
think
it
looks
like
there's
too
many
people
ordering
so
I
would
also
recommend
going
to
the
restocking
and
restocking
some
Donuts,
pretty
fun.
People
are
probably
already
kind
of
going
crazy.
I
can
see.
The
numbers
here
are
going.
Nuts
I
will
admit
this
was
load
tested
in
a
somewhat
insufficient
way,
given
the
number
of
people
in
the
room
here.
So
if
it
falls
over
I
apologize
but
yeah,
maybe
we
should
stop.
A
It's
been
a
good
run.
Everybody
Wow,
that's
a
lot
of
donuts.
It's
still
up
and
still
up.
This
is
great
news:
okay,
well,
I'm!
Okay!
Let's
stop!
Let's
really
stop!
So.
Let's
look
at
some
of
the
latest
examples
of
orders
for
sprinkled,
doughnuts
I
can
yeah
bring
these
up
well.
These
are
all
these
are
examples
of
doughnuts
that
we've
we've
run
out
of
doughnuts
in
these
cases.
A
But
that's
okay.
You
see
a
lot
of
errors
here
because
of
donut
exhaustion.
I
think
I
actually
have
an
example
of
a
donut
that
that
actually
did
come
through.
This
is
a
trace
from
just
just
before
the
talk.
These
traces
are
all
legitimate,
but
there
are
no
more
donuts
think
that's
what's
happening
there.
This
is
a
trace
showing
this
timing.
Diagram.
Actually
I
wanted
to
show
what
this
looks
like.
If
we
just
have
the
service
mesh,
it's
it's.
It's
definitely
useful.
A
You
can
see
the
propagation
from
the
browser
into
the
front-end
proxy
and
through
the
frying
process,
as
well
as
the
doughnut
topping
process,
but
that's
really
all
we
get
to
see.
We
can
look
at.
You
know
URLs
and
things
like
that,
but
it's
not
possible
to
really
get
to
the
bottom.
What's
going
on,
all
we
know
is
that
that's
slow
if
I
look,
for
example,
at
this
trace
that
involves
the
interior
of
the
process.
Not
only
have
we
stitched
things
together,
but
we've
also
added
more
detail
in
this
case.
A
The
mutex
library
actually
gives
information
about
how
many
waiters
were
in
front
of
us
for
a
mutex
lock.
In
my
mind,
this
sort
of
thing
well,
in
fact,
I
know
this
from
our
own
internal
debugging.
At
light
step,
this
can
be
really
invaluable
to
understand
that
the
slowness
is
coming
from
contention
around
a
particular
shared
resource.
A
We
can
go
and
and
look
for
examples
of
traces
that
had
you
know,
20
waiters
in
front
of
us,
instead
of
instead
of
just
four
and
and
see
where
the
latency
comes
from,
and
indeed
you
can
see
when
you
have
to
wait
for
20
users
in
front
of
you
on
the
mutex
that
ends
up
dominating
the
end-to-end
latency
for
these
requests.
So
in
my
mind,
I
don't
want
to
go
into
like
great
detail
about
debugging
this
fake
app,
but
I
do
want
to.
Hopefully
it
comes
through
that
these
sorts
of
analyses
are
fairly
trivial.
A
A
These
support
open
tracing,
which
means
that
you
know
any
open,
trace,
incompatible
project
or
vendor
can
just
plug
into
these
things
directly
and
one
of
the
things
I
get
really
excited
about,
and
about
CNC
F
actually
and
about
this
sort
of
integration
is
that
envoy,
sto
linker
D,
conduit
nginx.
These
things
are
exciting
technologies.
Open
tracing
is
also
an
exciting
technology,
but
when
you
combine
them,
they're
actually
greater
than
the
sum
of
their
parts.
A
This
this
family
of
projects
they're
not
just
independently
useful,
and
it
goes
back
to
my
flock
of
birds
in
the
very
beginning,
I
think
the
the
sum
is
greater
than
parts
in
this
case,
and
it's
something
that's
really
exciting
to
me
as
a
developer
and
as
this
technologist
to
see
how
these
things
make
each
other
better
and
when
you
combine
them.
It's
it's
even
more
powerful,
more
powerful,
still
and
I
just
think.
That's
exciting!
A
Speaking
of
feel-good
messages
and
preschool
and
stuff
like
that,
we're
going
to
have
some
small
group
discussions
are
more
friendly
to
QA
than
something
like
this
that
relate
to
both
open
tracing
and
service,
meshes
tomorrow
from
350
until
5:10.
There's
a
salon
for
open
tracing
I
would
love
people
to
attend,
there'll
be
a
bunch
of
us
there.
Matt
Klein
from
envoy
will
be
there
as
well,
and
we
can
talk
about
the
sort
of
material
in
this
talk.
A
How
people
get
basic
questions
answered
as
well,
as
you
know,
just
have
a
kind
of
support
group
about
deploying
this
kind
of
stuff
in
production
and
and
the
types
of
problems
that
we're
all
facing
Jaeger
and
an
awesome.
Cn
CF
project
that
is
a
distributed.
Tracing
system
is
having
their
salon
tomorrow,
as
is
linker
D,
and
the
Envoy
salon
was
fantastic
that
was
earlier
today.
Unfortunately,
so
you
can't
go
to
that
one.
A
If
you
didn't
already
go
but
I
hope
you
went,
I
heard
it
was
a
fantastic
and
oh,
that's
it
so
yeah
I
I
wanted
to
thank
everyone
for
listening.
It's
it's
been
a
great
pleasure
to
be
here
and
I'm
really
excited
about
the
rest
of
this
incredible
conference
and
yeah
with
that
I.
Thank
you
and
I
will
plug
the
ethernet
jack
in
for
the
next
presenter.