►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Tutorial: Service Mesh for the Developer Workflow - Christian Posta, Solo.io & Nic Jackson, Hashicorp (Limited Available Seating; First-Come, First-Served Basis)
Please bring your laptop fully charged as we will have limited charging stations available in the room. Service mesh is often presented as a solution for network engineering and system operability, increasing security, reliability, and observability. However, service mesh is also an incredibly useful tool for developers, and understanding how to leverage this technology can dramatically simplify your day to day workflow. By leveraging free and open-source tools and a scenario-based approach, we will illustrate how a service mesh can help with application resilience, observability, and debugging. By the end of this workshop you will understand: How to use metrics and distributed tracing effectively Reliability patterns like retries, timeouts, and circuit breaking How to leverage Canary deployments How you can effectively debug distributed systems The cloud-native, open-source technology used in this tutorial include: Envoy Prometheus Gloo shot Consul Service Mesh Loop Squash Open Census
https://sched.co/Uaeb
A
Well,
thank
you
all
for
coming.
I
think
this
is
going
to
be
we're
excited
to
deliver
this
workshop.
This
is
something
we've
been
thinking
about
for
quite
a
while,
and
we
think
it's
very
valuable
and
we're
pleased
to
see
you
all
think
along
the
same
lines,
we're
going
to
be
talking
about
service
mesh
and
how
that
applies
to
developer
workflows.
What
are
the
challenges
and
the
pains
that
a
developer
might
have,
regardless
of
the
language
or
the
framework
that
they're
using,
and
what
can
the
service
mesh
do
to
help
alleviate
some
of
that
pain.
B
So,
thank
you
again.
I'm
I'm
this
guy
here
my
my
hair's
kind
of
growing
back
to
the
length
of
my
photograph.
It
was
either
that
or
I
had
to
get
a
new
headshot,
but
I
I
work
at
hash,
co-op
in
in
developer
relations,
but
I
am
a
developer.
That
is
my
background.
I've
worked
20
years
building
and
designing
software
systems,
so
I've
literally
done
everything
that
you're
doing
now.
I
feel
your
pain.
We
have
shared
shared
anguish
around
stuff
and
hopefully
we're
gonna
solve
a
lot
of
those
things.
And,
of
course,
my
my.
A
Name
is
Christian
posta
I
am
a
field
CTO
at
a
start-up
called
solo
dial,
where
we
build
tooling
to
help
people
be
successful
with
service
mesh.
For
those
of
you,
I
think
earlier
that
raised
their
hand
and
said
they
are
running
service
mesh
in
production
and
maybe
for
those
who
are
getting
their
first
few
steps
into
service
mesh
land.
You
might
notice
that
it's
a
little
complicated,
and
so
it's,
although
we're
trying
to
alleviate
some
of
some
of
that
but
I
to
come
from
a
developer
background.
B
A
So
this
is
roughly
what
we're
going
to
do.
We're
going
to
give
you
a
an
environment
that
you're
gonna
set
up.
You
can
run
it
locally
on
your
own
laptops.
If
you
have
access
to
a
doctor
daemon,
that's
all
you
need.
That's
the
only
requirement
that
in
a
web
browser,
we
will
also
give
you
an
alternative
setup
where
you
can
just
go
to
a
online
web
training
style
environment
where
you
can
just
run
everything
from
the
web
browser
with
no
anything
running
on
your
laptop.
A
What
it's
like
to
make
changes
to
your
services
and
bring
them
out
into
production,
and
then
you
know
when
things
start
to
go
wrong
as
they
do,
then
how
do
we
go
in
and
debug
them
and
do
this
in
a
distributed
environment,
and
do
this
safely
go
to
this
link?
This
is
the
gateway
to
the
tutorial
here,
no
pun
intended.
Okay
go
to
this
link.
This
will
give
you
the
the
guides
for
the
tutorial.
It
will
give
you
the
links
to
get
to
instruct
environment
that
you
can
run
through
through
the
web.
A
B
We
built
the
the
kind
of
the
documentation
around
this
to
be
like
a
traditional
resource
that
you
can
follow
step
by
step.
We're
we're
kind
of
like
not
gonna,
read
that
verbatim
today,
but
it's
intended
so
that
you
can
kind
of
look
at
this.
You
can
watch
what
we're
doing.
You
can
kind
of
see
these
concepts,
but
you
can
can't
go
away,
have
a
little
think
and
you
can
kind
of
maybe
practice
as
a
mole
play
around
with
with
those
things
at
your
leisure,
because
you
must
be
pretty
beat
by
now.
B
A
If
you
build
in
these
cloud
environments,
where
the
networks
are
less
reliable,
then
we
may
have
thought
previously
and
the
infrastructure
is
elastic
and
the
you
know
it's
constantly
changing
things
are
failing
and
so
forth.
You
you
have
to
build
your
applications
to
live
in
this
environment
where
things
will
not
always
be
there,
I
think
in
the
past
we
have
this.
We
accepted
this
premise
that
the
database
will
never
go
down.
A
The
network
will
never
be
slow
and
when
it
did,
our
applications
didn't
respond
very
well
to
that
in
a
service
of
stall
architecture
and
cloud
infrastructure.
You
have
to
make
that
a
first-class
thing
that
you've
solved
for,
and
we
were
writing
things
in
application.
Libraries
things
like
Netflix:
oh
it's
a
square,
putting
it
into
our
code.
We
were,
you
know,
entangling
our
business
logic
with
this
stuff,
that
was
more
networking,
things
like
service
discovery
and
retries
and
client-side
load,
balancing
and
so
forth,
with
the
service
mesh.
A
Now
the
service
mesh
usually
gets
talked
about
in
terms
of
well.
We
can
solve
problems
like
network
security,
identity
on
the
at
the
application
level,
encryption
for
the
transport
layer
and
so
on.
We
can
get
telemetry
to
plug
into
our
monitoring
systems
for
operational
concerns.
We
can
do
traffic
routing,
for
you
know
the
releasing
new
applications
and
routing
around
errors
and
and
that
kind
of
thing,
but
the
service
mesh
brings
this
abstraction.
A
Where
we,
you
know,
as
the
traffic
comes
in,
it
allows
us
to
do
these
various
operational
things
and
those
are
absolutely
important
for
any
one
of
the
developers
who
are
in
here,
and
you
write
your
code.
If,
if
you
write
your
code
in
such
a
way
that
you
don't
really
have
responsibility
for
it,
you
just
give
it
to
the
next
person
to
try
to
run
it.
A
Then
you're
gonna
do
some
interesting
things
and
the
people
on
the
other
side
of
the
wall
are
gonna,
say
well
now
how
the
hell
do
I
run
this
thing,
service
match
kind
of
bridges
that
that
wall
or
that
divide,
so
traffic
comes
in
you.
Can
you
can
get
the
operational
aspects
of
it,
but
from
a
developer's
perspective,
I
need
time
outs,
I
need
retries,
I
need
circuit,
braking,
I
need
to
somehow
participate
in
distributed,
tracing
I
need
to
debug
things
and
and
the
the
combination
of
these
different
control
points
and
observability
points
in
the
network.
B
I'm
christian
kind
of
touched
very,
very
briefly
and
mentioned
this
word
like
data
plain
proxy,
the
the
kind
of
the
important
thing
around
service
matches
the
architecture,
so
we're
we're
here
to
talk
to
you
about
patterns.
Yes,
we're
winning
kind
of
use
some
specific
tools,
but
the
key
takeaway
that
you're
gonna
get.
Is
that
doesn't
really
matter
which
service
match
you're,
using
which
kind
of
toolset
you're
using
the
patterns
and
the
practices
are
going
to
be
the
same?
So
how?
B
How
kind
of
does
a
service
mesh
work
so
kind
of
in
the
indus
or
dove
the
the
guys
of
console?
You
have
a
control
plane.
The
control
plane
is
the
thing
that
allows
you
to
kind
of
centrally
store
your
configuration.
It
allows
you
to
manage
your
service,
catalog
and
sto.
This
is
like
etsy
d
things
like
this,
but
the
important
thing
for
the
developer
is
that
you
have
the
data
point,
so
the
data
plane
and
your
application
are
one-to-one
relationship
when
traffic
flows
in
or
out
of
your
application.
It
flows
through
the
data
plane.
B
So
the
proxy
data
plane
proxy
that
kind
of
two
terms
you'll
hear
pretty
much
interchangeable.
We
we
have
that
located
in
in
kubernetes
world,
it's
a
side
card
container,
so
you
won't
directly
inject
it
into
your
pods
back.
It's
something
that
the
the
service
mash
will
do
for
you,
but
it's
there
and
it
exists
the
the
kind
of
the
configuration
it
differs
ever
so
slightly
from
service
managed
to
service
mash.
But
but,
as
I
say,
the
concepts
are
broadly
the
same.
So
we
we're
kind
of
we're.
B
Gonna
look
at
console
today,
but
literally
I
hope,
you're,
gonna,
walk
away
and
you're
gonna
be
able
to
apply
this,
regardless
of
which,
which
tool
you're
using
so
here's
a
simple,
a
simple
sort
of
example.
So
I've
got
two
services
here:
I've
got
a
web
service
and
well
innocent
since
I've
got
a
database.
B
B
So
my
my
proxy
is
configured
by
the
control
plane,
so
I
kind
of
make
a
configuration
to
state
that
web
is
allowed
to
talk
to
database,
that
kind
of
allowable
security
rule
web
talks.
A
proxy
of
a
local
host
local
host
then
forwards
the
request
on
to
the
proxy
for
the
database,
which
then
forwards
the
request
on
to
the
end
application
all
the
database
service,
and
that
happens
completely
transparently.
B
B
So
what
do
you
get?
Well,
you
pretty
much
get
application.
Resiliency
I
am
pretty
much
for
free.
You
can
configure
things
like
timeouts.
You
know
like
cloud.
Networking
is
as
wonderful
as
it
is.
It
occasionally
goes
wrong.
You
have
problems
with
with
kind
of
transient
failure,
so
you
need
to
be
able
to
do
things
like
automatic
retries
service
discovery
is
a
really
kind
of
broad
concept
and
something
if
you've
been
using
kubernetes
for
a
while,
you'll
be
very,
very
used
to.
B
But
you
know
you've
got
to
kind
of
picture
this
as
a
problem
that
an
application
needs
to
know
the
location
of
another
application
if
you've
got
50
instances,
because
you're
horizontally
auto
scaling
those
across
a
cluster
which
has
5,000
nodes
like
how
are
you
gonna
manage
that
IP
address
location?
How
do
you
manage
that
load
balancing?
So
you
know
again
it's
a
sort
of
a
construct
which
which
you're
gonna
get
for
free.
It's
something
you
just
don't
even
need
to
think
about.
B
Importantly
for
application
development
is
distributed
tracing,
so
the
ability
to
be
able
to
see
a
request
flow
through
multiple
services
being
able
to
see
things
like
latency
timing.
Information
like
that
being
able
to
see
logging
information
and
being
able
to
see
error
messages
and
have
that
correlated
into
a
single
trace
graph
is
is
incredible.
It
really
really
helps
you
to
kind
of
diagnose
problems
to
be
able
to
discover
whether
or
the
weather
or
issues
like
MT
LS.
This
I
think
is
the
unsung
hero
of
service
mesh.
What
does
it
mean
for
you
as
a
developer?
B
Well,
it
means
you
don't
have
to
deal
with
Mt
LS,
so
if
you're
a
go
programmer,
if
you
want
to
implement
MT
LS
into
your
service
well,
the
first
thing
is:
you've
got
to
be
able
to
configure
your
HTTP
server
using
TLS
certificates.
Those
tell
certificates
are
got
to
come
from
somewhere
operations
problem
fair
play,
but
you
know
you've
still
got
to
make
that
configuration.
B
B
B
A
That's
that's
so
we
had
a
solo,
that's
the
lens
through
which
we're
looking.
How
do
we
narrow
in
on
the
difficult
parts
of
operating
a
service
mesh
and
a
micro
services,
style
architecture
and
and
solve
those
problems?
So
one
of
the
things
that
the
NIC
was
just
talking
about
is
the
ability
to
debug
the
first
open
source
project.
That's
that
solo
did
was
a
project
called
squash,
I'm,
just
curious
kind
of
an
odd
name
after
a
vegetable.
A
How
many
people
have
heard
of
squash
is
kind
of
curious,
okay,
a
couple
and
so
squash
is
a
multi-language
distributed
debugger.
So,
if
you're
going
to
build
your
services,
you
know
build
your
applications
and
services
distribute
over
multiple
different
deployments.
How
do
you,
and
maybe
even
multiple
languages?
How
do
you
connect
up
to
them
and
do
a
more
familiar,
step-by-step
debug
from
your
IDE,
because
you're
used
to
you
know
as
a
developer,
working
from
your
IDE?
How
do
you
do
the
step
by
step
debugging
across
potentially
multiple
services,
potentially
across
multiple
languages?
A
Now,
that's
that's
something
that
developers
leverage
in
their
own
environments
on
their
laptops,
maybe
in
a
development
and
shared
development
environment,
but
once
you
start
to
get
into
ist
UAT
production,
all
this
you
don't
get
to
just
connect
your
debugger
up
to
those
environments
right
now.
What
we're?
A
What
we're
going
to
show
you
today
and
as
part
of
the
tutorial,
is
in
a
in
a
mesh
environment,
the
types
of
things
that
we
can
do
like
that,
because,
because
the
requests
are
going
through
the
the
data
plane
through
the
different
proxies
is
we
can
see
those
requests.
Now
we're
going
to
follow
a
similar
pattern
to
what
open
telemetry
did,
whereas
requests
are
coming
through
we're
going
to
attach
some
correlation
IDs
to
it
and
build
distributed
tracing
traces
out
of
these
spans
that
we
collect
that's.
What
open
till
on
the
tree
will
do
now.
A
We'll
open
telemetry
is
really
just
grabbing
things
in
the
headers.
That's
that's
the
challenge.
Talking
to
it'll
it'll
help
you
identify
what
the
hops
were
and
where
the
slowdown
might
have
been,
but
that's
that's
about
where,
where
it
ends.
So
with
this
with
this
approach
that
we're
taking-
and
it
will
show
you
today
is
because
we
can
capture
because
the
proxies
in
the
data
plane
and
can
capture
the
requests
as
you
go
through,
the
requests
are
going
through
the
different
graph
different
services.
A
Things
are
responding
and
if
there's,
if
there's
an
issue
or
if
everything
works,
fine,
then
we'll
we'll
capture
those
requests,
but
then
throw
them
away.
We're
not
interested
in
them,
because
everything
turned
out
fine.
But
if
things
did
not
turn
out
fine,
then
the
requests
that
we've
captured
in
the
context
and
the
and
you
know
the
the
order
in
which
the
requests
propagated
through
the
system.
We
can
capture
them,
save
them
and
then
take
those
requests
and
replay
them
in
a
different
environment.
A
One
where
you
can
connect
up
your
debugger,
and
so
that's
the
workflow
that
we're
gonna
look
at
today.
We're
going
to
look
at
deploying
our
applications
and
the
service
mesh
using
the
mesh
to
identify
issues
and
help
us
figure
out
where
the
issues
are
we're
going
to
add,
distributed.
Tracing
we're
gonna,
look
at
the
mesh
from
being
able
to
bring
out
new
deployments
new
new
versions
of
our
applications
using
canary
and
then
we're
going
to
look
at
debugging.
Hopefully
that
works
for
everyone
again,
here's
the
here's,
the
link
to
the
environment.
A
B
A
So
what
we're
gonna
do
now
is
give
a
little
bit
of
time
to
click
on
and
boot,
up
the
instruct
environment
or
download
a
tool
here
called
shipyard.
If
I
zoom
in,
can
you
see
it
all
right?
A
tool
here
called
shipyard
that
will
assist
in
setting
up
the
environment
through
through
docker,
we'll
give
that
a.
C
B
But
test
stressing
is
sort
of
like
a
certain
cloud.
I
won't
remain
mention
which
one
but
they're
pretty
awesome.
Don't
your
favoritism,
the
in
strict
environments,
pretty
cool
what
it
does.
Is
it
it?
It
allows
you
to
kind
of
build
these
interactive
workshops
and
what
we
do
is
we
can
actually
through
the
instructor,
which
is
you
know,
great
tool.
Spin
up
these
environments,
the
behind
the
scenes,
like
virtual
machines,
are
actually
getting
spun
up
which
have
been
imaged.
B
B
D
B
Okay,
so
in
Visual
Studio
code,
if
you
want
to
get
to
terminal,
you
can
either
use
the
the
menu
option,
which
I
think
is,
in
the
top
left
hand,
side
there's
like
a
little
burger
menu
or
you
can
use
control
a
little
ya
till
ya
and
command
command
control.
P
will
give
you
the
like
the
quick
actions
and
you
can
you
can
search
through
to
do
that.
C
C
A
D
B
So
Visual
Studio
code-
it's
it
literally,
is
just
Visual
Studio
code.
So
if
you
do
I,
usually
I
use
vim,
so
I
I
can't
remember
what
the
color
of
the
command
is.
You
can,
if
you,
if
you
see
up
on
the
left
there,
where
you've
got
the
menu,
you
can
find
terminal,
hey
so
burger
menu
terminal
and
then
just
new
terminal
and
it'll
do
a
little
pop-up
terminal
inside
of
es
code.
For
you.
D
B
B
B
D
A
So
if
you
have
the
instruct
page
up,
which
I
don't
yet
then
will
want
to
follow
the
instructions
to
install
the
various
we
have
to
install
the
example
services,
we're
gonna
need
to
install
a
an
API
gateway
and
then
we'll
go
from
there.
The
service
mash
is
already
installed
as
part
of
the
image
that
we're
using.
B
So
if
you've
got
so,
that's
there's
a
bunch
of
tools.
If
you
want
to
use
mini
cube,
there's
a
bunch
of
tools
you
need
to
install
so
like
console,
Service
mesh
can
be
installed
with
a
helmet
art
which
is
pretty
easy.
But
then
you
wear
the
tools
that
we're
gonna
use
today.
Are
we're
gonna
be
using
the
squash
CTL,
a
loop,
CTL,
Gloup,
Gloup
CTL,
and
a
couple
of
other
sort
of
a
couple
of
other
tools
to
install
which
might
kind
of
like
take
a
look?
C
B
A
C
D
D
B
D
C
C
C
A
A
C
C
B
C
A
C
B
B
C
D
A
D
C
A
C
C
C
B
B
So
one
of
the
things
that
we
we
kind
of
want
to
look
at
is
is
the
tracing,
so
we're
gonna
look
at
this
more
in
in
sort
of
depth
in
a
in
a
little
in
a
little
while,
but
if
we
kind
of
take
a
look
at
tracing-
and
we
just
click
on
web
here-
and
we
just
go
to
find
traces,
so
I
can
see
a
trace.
So
what
is
a
trace?
B
Well,
it
trace
is
that
that
kind
of
flow
of
all
of
the
communication
traffic
between
my
services,
so
I
can
see
that
web
handles
an
upstream,
that's
a
spam.
It
then
calls
an
upstream.
So
it's
an
action.
Another
span
what
we
can
see
here,
where
we
see
this
localhost
1991,
is
a
tracer
to
automatically
inject
for
us
by
the
the
control
plane.
So
these
first
two
I
manually
create
in
code.
C
A
A
So
once
you
have
the
services
deployed
and
you
have
Jager
exposed,
so
you
can
see
the
distributed
tracing.
You
can
send
some
traffic
through
if
you
have
access
to
that
web
service
through
the
web
tab.
The
next
thing.
What
we
want
to
do
is
we're
going
to
set
up
the
glue.
Api
gateway
glue
is
a
as
a
gateway
built
on
Envoy
proxy
and
that
will
tie
in
with
any
service
match,
including
console
service
much
to
allow
traffic
in
through
the
cluster
through
an
ingress
points.
A
So
if
you
follow
this
well,
this
will
install
all
the
glue
proxy
and
and
it's
control
plane
and
the
interesting
thing
about
glue
and
why
we're
using
it
here
is
because
this
will
allow
us
to
to
use
the
loop
functionality
that
I
mentioned
so
in
this
particular
tutorial.
What
we're
going
to
do
is
capture
the
messages
that
fail
when
they
come
through
the
Gateway,
so
that
that'll
be
later
down
into
the
debugging
part
of
the
tutorial,
but
for
right
now
just
set
up
the
the
Gateway
itself,
and
you
know
once
it
gets
deployed.
A
The
glue
Caudle
command
should
be
on
your
on
the
on
the
web,
the
terminal
that
you
have
in
the
console
and
then
from
there.
You
can
create
the
route
that
needs
to
go
into
the
glue
routing
table
to
allow
traffic
to
get
into
the
web
service,
and
then,
ultimately,
when
you
call
it,
you
should
be
able
to
see
the
adjacent
response.
C
B
Once
we've
we've
got
everything
set
up,
so
we've
got
our
our
application
installed
now
I
kind
of
touched
on
this
earlier,
but,
like
I,
mentioned
that
there
are
four
services,
but
we're
only
seeing
three
of
them
here.
So
why
is
that?
Well,
we've
got
to
look
at
at
the
distributed
tracing,
so
what
is
distributed
tracing?
What
does
it
mean
for
you
as
an
application
developer?
So
we,
like
you,
know,
service
mesh.
It
solves
all
your
it's
perfect.
Everything
is
magic.
You
get
everything
out
of
the
box.
B
B
Well
you
don't
get
all
of
it
out
of
the
box
because
of
what
a
trace
is
so
a
trace
is
a
collection
of
individual
spans.
A
span
is
kind
of
like
a
unit
of
work
inside
your
application
code,
so
in
this
instance,
I've
got
call
upstream
that
a
unit
of
work
that
I've
defined
as
a
span
I've
got
handle
requests.
That's
like
an
inbound
span,
which
is
created
when
an
HTTP
handler
is
called
now.
B
B
What
information
is
giving
me
as
a
developer
is
I
can
see
all
sorts
of
things
such
as
the
request,
ID
HTTP
method,
kind
of
stuff.
That's
really
easy
to
kind
of
dig
into
gives
me
information
about.
What's
going
on
with
the
application,
I
can,
of
course,
use
this
when
something
goes
wrong,
but
why
am
I
not
seeing
the
paint
service
and
the
currency
service
so
I'm,
seeing
payments
here,
but
I
don't
see
anything
else,
I,
don't
see
any
of
the
the
inbuilt
spans.
The
reason
I
don't
see.
B
Just
saying
that
well,
it's
just
HTTP
I
can
see
that
in
my
in
my
in
my
code,
through
these,
this
metadata
that's
available
to
me
and
then
I'm
calling
this
finish,
because
the
span
defines
the
duration
around
a
kind
of
a
piece
of
work.
So
in
Ingo,
if
you're
not
a
go
program,
a
defer
means
execute
this
at
the
end
of
everything's
done
so
I
could
have
put
that
at
the
bottom,
but
this
is
a
kind
of
the
go
way
of
doing
it.
So
then,
what
am
I
doing
so
I
make
a
request
so
I'm.
B
I'm
not
creating
any
sort
of
things
around
there,
but
this
is
where
the
kind
of
the
trace
is
going
dead
and
the
trace
is
going
dead,
because
when
it's
coming,
things
are
coming
inbound,
I'm,
not
picking
up
the
parent
ID
when
things
are
going,
outbound
I'm,
not
setting
the
parent
ID.
So
how
do
we
do
that?
Well,
it's
pretty
straightforward.
B
So
what
we
do
again
we're
going
to
kind
of
use
open
tracing
is
that
in
the
the
span
for
an
HTTP
request,
something
which
is
outside
the
cold
creation
of
the
spans
I
need
to
be
able
to
define
these
HTTP
headers.
So
the
presence
of
these
HTTP
headers
is
what's
going
to
allow
me
to
associate
my
my
span
with
a
parent
and
by
setting
these
on.
An
outbound
request
is
going
to
allow
the
upstream
service
to
do
the
same
Association
in
terms
of
the
world
of
open
tracing.
B
Wire
context,
open
tracing,
obviously
supports
a
number
of
different
protocols,
not
just
Zipkin,
but
the
the
kind
of
the
generic
nature
is
that
in
your
code,
regardless
of
what
you're
doing,
if
you're
using
the
open
tracing
library
you've
got
this
kind
of
conventional
approach,
what
I'm
also
doing
is
I'm
just
writing
some
some
sort
of
debug
information
to
myself.
So
if
I
can't
create
this
span
context
based
on
the
presence
of
HTTP
headers
I'm,
just
gonna
write
that
out
to
a
log
message:
shouldn't
shouldn't
happen,
but
you
know
you
never
you
never
know
so.
B
Where
do
I
do
that?
I'm
doing
this
in
the
beginning
of
my
application
here
so
I'm
setting
that
wire
context,
I've
got
my
error
message
so
now,
when
I
create
my
span
instead
of
just
creating
a
flat
span
like
this,
what
I'm
going
to
do
is
I'm
going
to
create
a
span
with
the
context,
so
I
can
replace
this
line
here.
B
Still
doing
my
log
fields,
but
you
can
see
here,
I've
got
the
this
option
here
now.
I
just
need
to
do
my
my
import
because
again
go
everything
is
sort
of
package
oriented
so
I'm
just
going
to
import
the
the
X
package
which
all
of
this
contact
extraction
comes
from
and
I
can
add
that
into
there
right.
B
B
B
B
Got
my
new
version
there's
now
running
inside
of
inside
a
coupe,
so
I
can
I
can
run
this
a
few
times
just
to
get
a
little
bit
more
sort
of
trace
information
putting
into
here,
but
now
look
at
my
trace.
Instead
of
that
eighths
pounds,
I've
now
got
15
spans
and
the
reason
that
I've
got
the
15
is
because
I've
added
that
correlation
of
the
parent
ID
in
my
application
code.
This
is
like
a
really
important
thing.
The
service
match
does
a
lot
for
me.
B
What
I've
done
there,
though,
is
I,
haven't
really
taken
a
very
quality
quality
approach
to
doing
the
deployment
I've
just
thrown
that
at
production.
Now,
of
course,
I
got
my
unit
tests.
I've
got
things
like
that
to
be
able
to
test
what's
what's
going
on,
but
will
it
work
in
production
and
that's
the
kind
of
the
the
kind
of
the
key
question?
B
So
one
of
the
things
about
working
in
production
is
that
we're
application
might
be
tested
and
might
a
functional
test
might
have
unit
tests?
One
of
the
things
you've
got
to
think
about
is
what
are
the
configuration
differences?
Is
there
anything
on
the
server?
That's
different
from
your
local
environment?
The
answer
is
most
likely
yes
kind
of
a
lot
of
bugs,
no
matter
how
many
sort
of
things
that
you
put
around
them.
You
can
still
find
that
you
deploy
something
to
fraud
and
it
just
fails.
B
So
what
you
want
to
be
able
to
do
is
to
deploy
things
to
pop
to
production.
Softly
I
want
to
be
able
to
have
two
versions
of
my
application
running
in
production.
I
want
to
have
the
old
version.
The
new
version
and
I
want
to
be
able
to
send
a
very
kind
of
gradual
amount
of
increasingly
gradual
amount
of
traffic
to
the
new
version
until
I'm.
Confident
that
everything
is
is
working
as
it.
It
should
kind
of
concept
behind
that
is
probably
heard
before
a
canary
deployment.
B
I'm
gonna
use
this
Bluegreen
situation,
so
the
way
that
that
works
internally
is
the
service.
Mash
is
going
to
automatically
manage
that
traffic
routing
I'm,
just
going
to
control
and
say:
hey
I
want
10%
going
to
version
1
and
I
won
90%
going
to
to
version
2
because
you're
using
this
data
plane,
you
don't
need
to
put
any
application
code
to
be
able
to
control
this
process.
It's
all
of
this.
Smarts
is
driven
through
the
configuration
of
the
service
match,
so
my
web
service,
it
makes
an
outbound
call.
B
So
how
do
I
do
well?
I
got
a
hypothetical
scenario,
so
I
was
just
using
the
standard.
Golang
HTTP
interface
platform
team
have
come
along
and
they
throw
on
a
spanner
in
the
works
platform.
Team
are
saying
default.
No
good.
You've
got
to
use
our
specific
package.
Fine,
so
they've
called
its
sleepy
service,
so
I'm
gonna
make
another
modification
to
to
my
application.
I'm
gonna
replace
the
the
default
HTTP
client.
B
B
B
B
Just
pushing
and
direct
to
the
cube,
so
I
don't
have
to
manually
upload
that
to
get
hub
and
test
the
Wi-Fi
further
than
Wi-Fi
has
nothing
good.
Today,
all
right,
that's
pushed
so
next
step.
I'm
blue,
greening!
Now
right,
I've
got
a
canary,
so
the
thing
around
doing
a
Bluegreen
is
that
I,
don't,
whereas
previously
I
just
replaced
the
the
version
inside
my
kubernetes
deployment.
I,
don't
want
to
do
that.
I
want
two
different
versions
of
the
application
running
at
exactly
one,
the
same
time
so
I'm
to
do
that,
I'm,
going
to
create
two
deployments.
B
I've
got
my
blue
deployment,
which
is
here
and
I've,
got
my
green
deployment,
which
is
here
so
how
do
I
differentiate
between
a
blue
and
a
green
deployment?
Is
it
on
the
deployment
name?
No,
in
effect,
what
I'm
actually
doing
is
I'm
deploying
everything
with
the
same
service
identity.
Everything
is
just
going
to
be
classified
as
payment
now,
where
things
are
differing.
Is
that
I'm
actually
going
to
be
able
to
get
the
service
match
to
use
metadata,
which
is
associated
with
the
service
to
be
able
to
do
this
traffic
splitting?
B
So
you
can
see
here,
I'm
defining
some
service
tags,
I'm,
giving
the
identity
service
tag
of
this,
the
green
to
v3
and
the
matter
version
of
3.
If
you
look
at
the
blue,
we've
got
a
v2
and
2.
So
pretty
much.
Everything
remains
the
same
so
before
I.
Deploy
that,
though,
to
to
kubernetes,
then
what
I've
got
to
do
is
I've
got
to
look
at
the
configuration
because
I
if
I
just
deploy
this
straightaway
what's
going
to
happen,
is
kubernetes
is
gonna
start
round-robin
in
to
both
my
new
version
and
my
old
version.
B
I'm
gonna
get
an
equal
distribution
of
traffic
between
the
old
and
new,
and
that's
not
what
I
one
I
want
to
be
able
to
tightly
control
that
so
to
be
able
to
do
that.
What
I'm
going
to
do
is
I'm
going
to
use
a
configuration,
the
service
resolver.
This
is
specific
to
console
if
you're,
using
SDO
and
say
or
link
addi,
a
very
sort
of
specific
construct
exists.
But
let's
look
at
what
we're
doing
here
so
I'm,
giving
it
a
name,
it's
kind
of
like
a
virtual
service,
calling
it
payment
now
I'm,
defining
these
subsets.
B
So
I'm
creating
a
subset
of
a
service
called
blue.
The
blue
service
instances
are
the
ones
which
are
going
to
have
a
metadata
aversion
to
the
green
instances,
are
going
to
have
a
metadata
of
version
3
the
important
part
for
us
in
order
to
ensure
that
we're
only
sending
traffic
to
the
existing
version
until
we're
ready
is
by
using
the
default
subset.
So,
unless
I
configure
any
traffic
splitting
all
traffic
is
just
going
to
go
to
to
blue,
so
I'm
going
to
apply
that.
B
B
B
B
Then
what
I
can
can
see
is
I've
got
this
service.
It's
called
payment,
even
though
I've
got
these
two
deployments:
payment
deployment,
green
and
deployment
blue,
but
they're
they're,
actually
just
a
single
identity.
The
thing
that's
different
is
I.
Have
this
metadata
associated,
which
eath
with
each
of
the
service
instances
only
got
the
two
service
instances
there,
but
you
can
see.
B
So
what's
the
next
step?
Well,
the
next
step
is
that
what
I
need
to
do
to
be
able
to
start
splitting
my
traffic
is
another
configuration
I
want
to
define
a
service
splitter,
so
I've
got
my
new
configuration
element.
I'm
gonna
do
some
traffic
splitting
again
it's
got
a
name
of
payment
because
it
relates
to
the
payment
service
and
I'm.
Defining
these
splits,
splits
I'm,
just
gonna
do
straight
50:50.
For
the
purposes
of
a
demo
in
a
real
world
production,
environment,
you're
gonna
use
much
much
much
much
lower
numbers.
B
How
low
you're
going
to
use
this
lower
number
that'll.
Allow
you
to
get
meaningful
information
to
determine
if
the
new
version
of
your
services
using
is
working
and
that's
going
to
depend
on
the
service
utilization.
The
number
of
instances
you
cluster
and
all
sorts
of
things
like
that,
but
for
this
we're
gonna
do
5050
we're
signing
50%
of
traffic
to
blue
and
with
sign
50%
of
traffic
to
green.
In
the
same
way,
I'm
just
gonna
write
that
to
the
control
plane.
B
B
B
Again,
I
can
dig
through
this
information,
but
what
I
need
to
do
is
I
need
to
start
debugging
this.
Ordinarily,
if
you
were
running
in
production,
what
would
you
do?
Well,
the
thing
that
you
would
do
is
you
would
roll
that
straight
back.
You
would
identify
that
there's
an
error
occurred,
and
you
would
reverse
that
you
would
set
your
traffic
splitter
to
be
0%
for
the
blue
version.
Oh
sorry,
for
the
green
version
of
your
service
and
you
just
back
it
out
for
the
again
for
the
purpose
of
the
workshop.
A
C
A
New
deployment
we're
able
to
look
at
the
tracing
and
we're
able
to
see
okay.
This
is
this
is
erroring
out
now.
Well
we're!
If
we're
in
a
production
environment,
we're
not
going
to
be
able
to
connect
up
directly
to
our
news
service
to
be
able
to
debug
it
we're
going
to
need
to
do
some
additional.
You
know.
Will
we
have
additional
tools
to
be
able
to
do
that
for
us,
but
before
we
get
to
that,
I
want
to
familiarize
you
with
the
squash
debugger.
So
I
mentioned
this
earlier.
Squash
is
a
distributed
multi
language
debugger.
A
What
it
does
is
when
you're
running
in
kubernetes-
and
you
want
to
attach
your
IDE
to
a
single
service
or
any
number
of
services.
You
need
to
get
the
debug
port
open
on
the
application.
You
need
to
spin
up
the
debugger.
You
need
to
attach
all
of
the
the
piping
to
be
able
to
port
forward
it
to
your
local
machine
so
that
you
can
use
your
IDE
and
all
of
that
complicated
stuff
that
happens
inside
of
kubernetes.
A
That
is
what
squash
takes
care
of
squash
takes
care
of
the
plumbing
finding
out
exactly
what
process
on
what
host
and
what
container
and
starting
up
the
debugger,
attaching
it
to
that
port,
forwarding
of
it
back
and
and
so
forth.
So
that's
that's
what
squash
does
and
that's
what
we're
going
to
take
a
look
at
here.
So
in
the
tutorial
under
this
section
we
are
going
to
attempt
to
run
squash
from
within
the
IDE.
So
we
have
the
vs
code
packaged
up
and
ready
to
go
from
within
vs
code.
A
A
A
So
from
here
we're
going
to
select
the
namespace
for
the
application
that
we
want
to
debug.
We
can
see
now
that
there
is
a
listing
of
the
various
services
through
various
other
means.
We
figured
out
okay
in
this
general
area,
maybe
the
the
new
deployment.
The
green
deployment
was
something
that
we
should
take
a
closer
look
at.
We
can.
We
can
filter
down
click
on
that
now
inside
the
deployment
pod.
The
service
mesh
is
also
running.
A
We
see
the
console
or
envoy
running
in
this
case
right
next
to
the
the
payment
service,
but
we
want
to
debug
the
payment
service.
So
let's
click
on
that
and
then.
Lastly,
this
is
a
ja.
Our
Goliath
we're
gonna,
attach
the
the
dal
debugger
to
it
and
we'll
do
that
and
give
it
a
second.
What
this
is
gonna
do
is
spin
up
a
a
pod
within
kubernetes
that
is
based
on
the
delve
debugger
it's
going
to
find
out,
which
container
is?
A
A
There
we
go
now:
we've
attached,
we've
switched
to
debug
mode,
and
now
what
we're
gonna
do
is
we're
going
to
set
a
breakpoint
and
cross
our
fingers
there.
Now
it
looks
good
and
now
what
we're
gonna
do
is
we're
gonna
call
into
our
service,
and
we
saw
that
it
fails
if
I
click
it
again
that
one
so
we're
doing
the
50-50
split
from
the
canary
right,
but
this
one
this
call
succeeded.
A
If
we
come
back
to
code,
we
should
see
that
the
debugger
caught
that
when
the
request
came
into
this
failing
service
squash
and
dealt
together
caught
it
and
brought
it
back
to
the
IDE
now
from
the
IDE,
we're
just
using
the
regular
golang
debugger,
and
we
can
we
can
step
over
our
calls
here
and
the
service
is
running
remotely
remember
this
is
you
know
we're
able
to
step
through
and
we
see?
Okay
that
looks
all
right.
A
A
We'll
get
to
that
part,
and
then
we
try
to
step
over
that
one
I
didn't
oh
I,
didn't
do
the
request
step
over
that
one,
and
then
it
sits
in
Hanks.
So
we're
able
to
step
through
the
code
evaluate
the
call
stack,
evaluate,
evaluate
the
context
variables,
and
then
we
got
to
a
point
where
another
thing
just
is
a
black
hole
right,
it
just
sinks,
and
it's
and
it's
it's
not
returning
now.
A
If
we
do
wait
long
enough,
the
sleepy
client
will
relent
and
we
will
be
able
to
recover
and
continue
on,
but
we
can
see
all
right.
This
is
this
is
a
problem.
This
is
an
issue
we
need
to.
We
need
to
address
this
there,
so
news
on
now.
This
is
here
going
to
stop
the
debugging
for
a
second
now.
This
is
this
is
fine.
If
we're
running
in
a
local
environment,
maybe
your
local,
kubernetes
or
but
if
you're
running
in
production,
then
this
is
not
necessarily
something
that
the
platform
team
is
going
to.
A
Allow
you
to
do
to
connect
your
developer
laptop
up
into
production.
So
what
we're
interested
in
doing
is
finding
a
way
where
we
can
reliably
recreate
that
issue
with
the
same
version
of
the
services
that
are
running
in
production
in
a
different
environment
so
that
we
can
safely
debug
in
that
environment.
So,
if
we
scroll
down
into
so
when
you
go
up,
go
back
and
do
this
for
yourself,
you
can
go
through
the
guide
and
step
by
step
with
with
these
images
here
to
help
you
debug
the
sleepy
client.
A
What
we
want
to
do
is
use
a
tool
called
loop
and
loop
is
a
capture
and
replay
mechanism
for
unvoyage
based
gateways.
So
now
in
our
demo
here,
what
we're
gonna
do
is
we're
gonna
set
up
the
API
to
the
glue
API
gateway
which
has
loop
in
it,
or
the
capability
for
the
Gateway
to
capture
from
from
with
an
envoy
and
then
request
that
fail.
What
we're
going
to
try
to
do
is
capture
them
and
then
replay
them
alright.
So
the
first
thing
that
we're
going
to
do
is
we're
going
to
install
loop.
A
So
loupe
is
up
and
running,
clue
is
coming
up,
we'll
give
that
a
second
to
to
come
up,
and
now
we
have
loop,
running
and
and
loop
running
in
the
in
the
kubernetes
cluster.
Here
means
whenever,
when
a
request
is
going
through
envoy
and
we've
enabled
loop
to
do
the
capture
on
certain
requests,
ones
that
failed
are
the
more
interesting
ones
and
I'll
show
you
how
we
do
that
in
a
second.
But
when
Envoy
captures
the
request,
then
it
will
send
it
to
the
loop
server,
which
will
then
store
it
and
cut
and
categorize
it.
A
Especially
if
there's
multiple
requests,
part
of
a
single
request
thread
coming
in
loop
will
do
the
well.
How
do
we
put
it
right
in
the
right
order
and
that
kind
of
stuff,
but
for
right
now
we're
gonna
is
pretty
simple
when
a
request
comes
in,
it,
fails
we're
gonna
capture
and
send
it
to
loop
loop,
we'll
hold
onto
it
and
from
there
we'll
be
able
to
play
it.
So
if
we
come
back
here,
everything's
up
and
running
nicely
so
now,
the
next
step.
A
A
A
A
So
we're
what
the
the
goal
is
to,
if
you
have
we're
gonna
try
to
piggyback
on
open
tracing.
So
if
you
have
open
tracing
correlation,
IDs
and
so
forth,
then
we'd
be
able
to
figure
it
out
that
way,
we've
toyed
with
adding
other
headers
to
do
correlation
IDs.
But
you
know,
if
that's
just
going
to
be
another
another
issue,
so
ideally
we're
going
to
be
able
to
do
this
on
top
of
an
existing
correlation
system.
Otherwise,
we'll
have
to
you
know,
have
to
force
the
developers
to
include
additional
metadata
to
do
that.
A
So
we're
able
to
grab
the
raw
requests
now
when
it
gets
to
loop.
We
and
we
don't
haven't
built
the
UI
and
the
management
capability
for
this
yet,
but
the
idea
is
that,
once
we
grab
that
raw
request
that
we
can
screen
out
any
very
environment,
specific
headers,
where
you
need
to
or
content
where
you
need
to
write,
because
if
you're
passing
path,
production,
passwords
and
tokens
and
that
kind
of
stuff
you
might
want
to
mask
those
or
inject
different
ones,
for
when
you're,
in
a
different
environment,.
A
So
you
can
so
I
mean
I'm,
saying
we
we
have
but
I
know.
We've
talked
with
certain
customers
who
said
there
are
environment,
specific
things
that
you
cannot
take
out
of
production,
but
but
we
will
grab.
We
will
have
access
to
the
full
raw
requests,
including
all
of
the
production
Johnson.
So
for
that
get
past
and
you're
able,
you
are
able
to
take
that
if
that's
something
that
you're
willing
to
to
do.
Yes,.
C
A
So
the
vs
code
debug.
So
what
basically
is
what
squash
does
is
if
you
have
a
java
application
or
a
golang
application,
and
we're
really
close
to
no
js'
support.
One
of
our
customers
asked
specifically
for
that.
We're
able
to
you
can
just
use
the
regular
vs
code
debugger
and
do
java
golang
nodejs
all
within
a
single.
You
know
ide
and
what
it
does
is
on
remote
kubernetes.
A
So
the
way
that
I
showed
you
right
now,
what
it
does
is
it
attaches
up
to
the
process
and
attaches
the
debugger
and
pipes
it
all
the
way
back,
but
for
a
lot
of
people
who
actually
use
this.
That's
too
insecure
for,
even
even
though
it's
in
the
development
fire.
If
it's
in
a
shared
environment
where
they
have,
they
don't
want
just
random
other
teams
to
be
able
to
connect
up
into
their
system
what
they
do
is
they
they
specify
a
CR.
A
So
we
use
a
CR
D
to
say
that
this
debugging
is
allowed
to
happen
for
this
particular
user
and
we're
able
to
leverage
kubernetes
our
back
to
do
that.
So
it's
when
we
start
to
get
into
this
the
more
secure
way
of
debugging.
That's
that's
where
and
then
propagating
that
plumbing
and
all
the
way
back
to
the
ID.
That's
where
the
value
of
squash
starts
to
become
more
apparent,
I
mean
if
you
have
the
delve
debugger,
you
know
you're
able
to
open
the
root
the
the
ports
and
so
forth
and
use
vs
code
remote.
A
A
So,
for
example,
you
have
kubernetes
health
checks
and
liveness
checks
and
so
forth
and
then,
when,
when
squash
runs
into
it,
whether
or
not
it
gets
it
gets
killed
because
it
because
Karina
doesn't
doesn't
return,
I
think
the
answer
to
to
that
is
I.
Don't
I
haven't
seen
that
bite,
but
that's
a
valid
point.
We're
gonna
have
to
come
up
with
a
way
to
solve
that,
but
the
users
that
we've
seen
debug
their
applications
like
this
they're
not
doing
it
in
production
right
so
requests
are
not
actually
we're,
not
impacting
the
users
in
lower
environments.
C
C
A
A
B
A
It's
it's
on
Boyd
base,
so
an
API
gateway
built
on
a
non-void
that
uses
the
tap
filter.
We
can
do
it
with
ISTE.
Oh,
we
can
with
the
sto
sidecar
proxies.
We
can
do
it
with
the
console.
Sidecar
proxies
right
now.
The
tap
filter
that
we're
using
is
it
we've
pushed
it
upstream,
but
it's
not
in
the
master
yet,
but
that
is
that
is
something
that
will
be
coming
so
that
any
any
distribution
that
uses
envoy
will
should
be
able
to.
A
We
should
be
able
to
leverage
that
to
pull
back
yeah
all
right,
I
think
we're
supposed
to
end
at
3:55.
I.
Think
right.
All
right
well
definitely
take
a
look
at
the
guide.
You
know,
I've
run
it
locally
on
docker
in
your
own,
on
your
own
Internet
and
reach
out
to
us.
If
you
have
any
any
thoughts
or
questions
or
things
didn't
work
out
right,
yeah.