►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Debugging Applications on Kubernetes - Michelle Noorali & Radu Matei, Microsoft
Once upon a time, using a debugger was a normal part of development. Then along came the cloud, and suddenly, a tool developers took for granted completely disappeared. We used to have a single place like an IDE of choice to add some breakpoints, step through the code, and hopefully solve the issue. In a distributed world where there are many moving parts that can’t always be run on a single machine, debugging an application consisting of multiple services has become significantly harder with microservices written in a variety of languages that come with their own toolchain spread across multiple teams and repositories. In this demo-led session, we’ll discuss the current state of debugging applications in Kubernetes and provide multiple methods and solutions for debugging so that we can leverage all of the goodness of microservices and the ease of use of existing debugging workflows.
To learn more: https://sched.co/GrRd
A
A
Working
in
the
world
of
Ruby
on
Rails
and
JavaScript,
I'm
gonna
add
some
fun
facts:
I
really
love
tennis,
I'm,
really
bad
at
it.
I
love
coffee
but
gives
me
anxiety,
so
I
should
not
drink
it.
So
much
and
I've
excelled
I've
learned
how
to
excel
at
packing
travel
travel
luggage
for
the
past
few
years,
but
I
really
don't
want
to
travel
anymore
stuff.
That's
my
life
right
now,
hello.
B
Everyone
I'm
Radu
I'm
up
software
engineer
at
Microsoft,
I'm,
a
crow
maintainer
of
draft
Brigade,
then
the
new
cloud
native
application,
bundled
projects,
I'm,
a-goin
type
script,
developer
and
almost
rusts
developer
and
I
work,
mostly
on
open
source
cloud
native
developer,
tooling,
I,
absolutely
love,
music,
I'm,
a
lousy
piano
and
guitar
player
and
I
love
bicycles.
So.
B
It's
worth
taking
a
step
back
and
try
to
remember
what
was
like
before
we
were
in
the
world
of
microservices,
specifically
when
we're
developing
large
monoliths,
especially
when
we
were
developing,
for
example,
dotnet
or
old
Java
applications.
We
used
to
have
one
place
where
the
application
was
running,
where
I
could
see
the
all
the
logs
of
my
application,
where
I
could
start
up
build.
My
application,
put
a
breakpoint
and
evaluate
the
state
of
my
requests
right,
and
this
is
true:
for.net
and
java
applications,
but
and
I.
A
But
for
the
most
part
we
are
no
longer
writing
monoliths.
We're
writing.
Micro
services
and
containers
and
kubernetes
have
become
this
clear
platform
of
choice
for
developing
and
deploying
them,
and
this
has
allowed
us
to
break
down
these
incredibly
complex
solutions
into
smaller,
easier
to
understand
components
that
are
independently
deployable
and
loosely
coupled.
B
As
a
as
an
industry,
we've
been
very
good
at
developing
solutions,
for
example,
for
centralized
logging,
distributed
tracing
and
lots
of
other
tools
that
allows
us
to
understand
the
lifecycle
of
a
request
and
how
to
understand
the
problem
of
our
request,
but
breaking
down
to
how
do
we
understand
the
problem
right
now?
How
would
people
in
the
audience
would
go
about
understanding
what
what
the
problem
is
with
a
pod
keep
control
logs
right?
B
That's
the
first
thing
we
do
when
we
understand
that
something's
not
working
right,
but
unless
the
problem
is
in
the
specific
part
that
we're
trying
to
read
the
logs
from
it's,
not
gonna,
be
of
much
use
for
us.
If
the
problem
is
in
the
in
grass
or
in
the
service
or
in
somewhere
else,
along
the
line
of
our
request
problem,
probably
the
there's
a
broken
secret
right.
We
have
no
knowledge
of
understanding
that
and
that's
where
a
distributed,
tracing
or
logging
comes
in
right.
B
Tools
like
Ravana,
allows
me
to
trace
a
particular
request
through
all
of
its
lifecycle,
and
on
top
of
that,
the
industry
has
also
been
working
on
a
lot
of
tooling
for
debugging.
For
example,
both
Google
cloud
platform
and
Azure
offer
paid
services
to
do
debugging
on
top
of
kubernetes.
The
thing
is
both
of
them
only
work
on
their
respective
cloud
platforms,
we're
looking
for
something
that
is
open-source
and
works
on
AV
kubernetes
cluster
out
there
now
I
cannot
recommend
the
squash
debugger
enough
and
I
with
that.
Take
a
look
at
it.
B
A
But
you
might
be
wondering
like
why
do
I
want
to
debug
on
cue
Brunei's
at
all?
Well
we're
at
a
point
where
there
are
too
many.
We
get
to
a
point
where
there
are
too
many
services
we
have
to
run
locally.
You
might
also
have
some
security
constraints
that
don't
let
you
do
things
outside
of
your
cluster,
so
some
in
cluster
dependencies
liking,
you
might
need
access
to
private
api's
and
data.
You
might
have
IP
restrictions
or
things
of
that
nature
and
also
debugging
on
kubernetes
lets
us
get
closer
to
a
production
like
environment.
A
A
B
A
B
And
this
is
where
the
actual
fun
ends,
because
so
actually
debug
an
application
on
kubernetes
in
the
true
sense
of
debugging.
You
have
to
first
of
all
build
your
application
for
debugging.
That
might
mean
including
debugging
symbols
in
your
binary,
starting
the
application
process
with
some
flags
like
that
inspect
for
nodejs,
or
any
other
way
that
you
build
an
application
for
debugging
in
your
platform,
then
you
have
to
actually
build
a
container
image
using
the
binary
just
built.
You
have
to
push
the
container
image
to
a
container
registry.
B
Those
are
a
lot
of
steps
right
and
the
tools
that
make
it
easier
to
solve
some
steps
of
the
the
entire
process,
such
as
casing
synchronizes,
your
local
file
system,
with
the
file
system
inside
a
pod.
There's
telepresence,
which
redirects
traffic
from
your
cluster
to
your
local
machine
for
specific
service,
there's
drafting
scaffold
which
rebuild
the
container
image,
push
it
to
a
registry
and
then
update
the
deployment
and
both
of
us
we've
been
working
on
drafts
for
quite
a
while.
So
Michelle
do
you
want
to
show
a
bit
of
Draft?
Okay.
A
B
B
A
Okay,
so
in
the
draft
repository
there's
an
examples
directory
and
there's
an
example,
called
example
go
and
in
this
example,
you'll
find
a
simple
web
server
in
the
main
taco
file,
and
this
just
listens
on
port
8080
and
says
hello.
Coupe
con
remember
to
do
your
laundry.
That
should
say
remember
to
do
your
laundry
as
soon
as
you
get
home,
because
I
always
forget,
so
we
are
going
to
do
a
draft
up
and
what
this
is
going
to
do.
Is
it's
going
to
build
our
container
image?
A
It's
going
to
release
it
to
arts,
gonna
push
it
to
a
container
registry,
and
then
it's
going
to
deploy
this
application
to
kubernetes
using
a
helmet
right
now,
if
you'll
see
if
you'll
remember
or
if
you
noticed
there
are
some
more
files
in
my
directory,
there's
a
docker
file
which
tells
you
how
to
build
your
image.
There's
a
chart
structure
which
contains
some
home
charts
to
deploy
your
application.
There
is
a
draft
tamil
file.
This
is
some
metadata
to
help
draft
deploy
your
application.
These
all
come
from
draft
packs
which
we'll
look
at
later.
A
Alright
but
I'm
gonna
do
a
draft
connect
and
what
this
does
is
it
gives
you
a
local
tunnel
to
your
application,
using
kubernetes
port
forwarding
in
the
backend
and
we're
gonna
go
to
our
browser
and
I'm
gonna
shut
off
slack
here,
okay
and
and
yeah.
It
says
hello.
Cout
can
remember
to
do
your
laundry
okay,
we're
gonna,
go
back
to
our
app
we're
gonna
control,
C
out
of
that
we're
going
to
edit
our
application
is
the
font
okay
for
the
font
size
mask
not
so
great.
A
Then
grab
this
URL.
You
can
specify
what
port
you
want
to
connect
to,
but
this
just
gives
us
a
randomly
generated.
Fine,
so
our
application
updated
and
that's
great,
but
you
might
be
thinking
like
hey.
You
still
have
to
have
like
a
docker
file
on
a
home
chart
and
dracto
tamil
file,
and
that's
like
a
lot,
and
it
is
so
I'm
gonna
go
to
this
example.
Ruby
application
and
I'm
gonna
delete
my
darker
file
and
drop
down
the
file
on
my
charts
and
now
I
just
have
my
application.
A
If
I
wrote
a
draft
create
here,
it's
going
to
detect
the
language
of
the
application
and
it's
going
to
grab
something
called
a
draft
pack
which
is
essentially
just
a
directory
with
some
files
in
it.
It's
the
files
are
the
docker
file
in
the
charts
and
the
draft
AMA
file,
and
it's
gonna
place
it
in
my
in
my
directory
of
my
source
code
and
then
draft
can
use
that
on
a
draft
off.
A
You
might
be
thinking
well,
I,
don't
need
auto
language
detection
and
that's
that's
cool
too,
because
that's
really
just
nice
magic
default
behavior
to
have.
If
you
have
some
custom
docker
files
that
that
you
use
for
your
organization,
you
can
create
a
draft
pack
for
yourself
and
then
you
can
run
a
draft
create
and
then
override
the
default
behavior
using
the
pack
flag
with
some
custom,
Ruby
pack
that
you
made
and
that
will
have
your
custom
configuration
for
your
dog
or
file
and
your
kubernetes
manifests
in
there
all
right.
A
Well,
but
it's
ok
great,
oh,
that
worked
a
really
bad
luck
with
those
kinds
of
things.
Alright,
so
just
a
recap:
real,
quick.
That
was
a
few
different
things.
We
did
a
draft
up
which
built
and
pushed
our
image
and
then
deployed
her
app,
and
we
did
a
draft
connect
which
gave
us
some
local
tunnels
on
the
exposed
container
ports
in
the
kubernetes
application,
and
all
this
was
powered
by
draft
packs
which
are
great
for
scaffolding.
Your
doctor
filed
and
Cooper
kubernetes
configuration.
A
So
that's
all
well
and
good,
but
let's
do
something
a
little
more,
a
little
more
interesting,
a
little
more
complex.
So
what
I
have
here
is
an
application
with
an
angular,
client
front-end
and
a
node,
back-end
and
I've
already
deployed
these
actually
I'll
show
you
the
pods
that
are
running
I've
deployed
these
just
to
save
a
little
bit
of
time,
using
draft
up
and
all
of
that,
but
I'm
gonna
do
a
draft
connect
in
my
angular
client
front-end.
A
B
A
Laundry
we
really
like
clean
clothes
and
then
after
I
do
some
laundry
I'm
gonna
have
I'm
gonna
need
something
else
to
do,
but
first
I
want
to
tell
you
a
little
thing
that
I've
done
to
my
cluster,
which
is
I've
installed.
Something
called
linker
D,
which
is
this
simple
service
smash.
Basically
I
installed
the
CLI
on
my
local
system
and
then
I
installed
the
control
playing
using
like
one
command,
and
you
can
see
the
control
plane
running.
A
A
It
injects
a
link
or
D
sidecar
and
updates
your
deployment
manifest
okay.
So
now
we
have
our
application
pods.
We
have
our
linker
D
sidecars.
Why
does
that
matter?
Okay?
That
matter
is
because
you
get
this
like
nice
little
dashboard
with
with
information
about
your
requests
and
responses.
So
when
you
have
one
service,
it
might
be
a
little
easier
to
see.
A
What's
going
on,
but
when
you
have
lots
of
services
you
need
a
way
to
be
able
to
track
the
requests
and
responses
and
and
what
you
know,
codes
you
get
back
and
and
all
of
that,
so
in
a
refresh
this
guy
we're
in
their
top
section
this
aggregates
those
requests
and
responses
and
I'm
just
gonna
head
start
start
recording
I'm
going
to
refresh
this
one
more
time,
I'm
going
to
check
some
stuff
off
I've
done
everything.
What
else?
What
what
do
you
all
need
to
do
when
you
get
home
from
coop
con
laundry
you're
right?
A
Oh
my
gosh,
so
much
laundry
should
be
had
great.
What
kind
of
detergent
you
use?
Okay,
all
right!
So
this
down
here
this
down
here
is
a
button
I'm
going
to
click
it
I'm
going
to
have
after
everything
you
know,
I
need
something
to
do
so.
I'm
gonna
get
something
to
do
and
look
at
that.
I
have
an
error.
So
if
I
go
back
to
linker
D
to
the
linker
D
dashboard,
then
I
will
see
these
requests
and
responses
like
popups.
So
I
can
even
see
here
that
the
request
came
from
my
angular
client.
A
It's
going
to
my
node
back-end
and
then
it's
all
the
paths
that
I'm
hitting
some
to
statistics
around
Cal
and
time
and
the
success
rate.
And
what
really
gets
me
is
this
little
red
right
here.
It
looks
like
when
I
hit,
slash
random,
which
was
the
path
that
we
hit
when
we
click
click
that
button
I
I
didn't
get
a
success.
A
B
I
have
the
exact
same
setup
on
my
cluster
as
well,
which
means
I
have
a
kubernetes
cluster
I
have
helm
configured
on
my
cluster
and
I.
Have
both
the
node.js
backends
deployed
in
my
angular
front-end
and
I'm
have
the
same
application
and
I'm
refreshing
the
the
app
and
I'm
trying
to
press
this
button,
and
yes,
this
is
a
button.
This
is
my
the
extent
of
my
web
development
skills,
ready.
B
At
this
point,
what
I
can
do
is
open
vs
code
is
that
large
enough?
Okay,
so
I
can
open
vs
code
with
the
source
code
for
the
node.js
backend
and
because
this
is
a
draft
application
and
I
have
the
Visual
Studio
code,
kubernetes
extension
installed.
What
I
can
do
is
start
debugging
by
pressing
f5
and
at
this
point,
vs
code
will
start
building
the
application
using
draft.
So
it's
doing
a
draft
up
which
builds
the
container
image,
pushes
it
to
a
container
registry
and
then
deploys
the
application
on
the
cluster.
B
It
waits
for
the
pods
to
be
in
running
state,
then
we'll
port
forward
all
application
ports
locally
on
my
machine
on
localhost,
including
the
debugger
port
for
nodejs,
and
then
it
will
attach
the
node.js
debugger
on
my
process.
That's
running
inside
my
kubernetes
cluster.
So
all
of
this
is
powered
by
the
vs
code,
kubernetes
extension,
which
I
really
recommend.
You
have
a
look
at
if
you
use
vs
code
and
at
this
point
the
debugger
seems
to
be
attached
on
a
process-
that's
running
inside
my
kubernetes
cluster.
B
So
if
I
go
back
on
the
application
and
I
just
refresh
it
in
theory,
I
should
be
able
to
hit
a
breakpoint
from
the
node
from
the
angular
client,
whose
request,
because
making
a
request
to
the
nodejs,
micro-service
and
I
catch
that
using
your
breath
point
that
I
set
up
earlier
in
vs
code.
So
at
this
point
what
I
can
do
is
see
all
of
the
information
for
the
request
in
the
response
evaluate
variables
have
a
full
debug
console
so
request.
That
host,
for
example,
is
to
do
API,
no
debug.
B
And
right
now,
in
theory,
I
should
be
able
to
debug
the
problem
that
we're
seeing
on
the
give
me
something
to
do
button.
So
if
I
press
that
oh
I
just
catch
the
breakpoint
that
I
set
up
earlier-
and
apparently
this
is
the
issue
of
the
problem,
it's
throwing
a
new
error.
Every
time,
I
press
that
button.
It's.
B
B
Want
to
get
a
random
to-do
item
with
our
application.
Apparently
some
other
team
wrote
up
a
go
micro
service
that
just
gives
you
a
random
to-do
item
right,
so
hopefully
I
can
be
able
to
use
the
micro
servers
they
just
built
and
let's
build
the
application
again,
which
this
does
again
doc
draft
up.
Does
a
docker
build
and
docker
push
then
deploys
the
application
using
helm,
then
forwards
all
ports
and
attaches
the
debugger
and
look.
B
B
B
B
A
B
Quite
the
opposite
of
magic,
it's
plumbing,
so
the
thing
is:
we're
not
doing
anything
but
take
a
tool
that
is
able
to
iterate
on
your
application
so
continuously
deploy
your
application
in
a
cluster
then
forward
the
ports
and
attach
a
debugger
in
theory,
you
could
be
able
to
use
any
tool
and
change
any
tool
that
we're
using
in
our
workflow
with
your
own.
So
the
thing
is:
there's
not
one
single
tool
that
allows
you
to
have
a
workflow
like
this.
B
That's
why
we're
trying
to
put
together
a
workflow
that
allows
you
to
essentially
build
your
application
for
debugging,
create
the
image
push
the
image
redeploy,
attach
a
debugger
and
so
on
and
so
on,
and
you
can
definitely
do
that
manually.
But
if
you're
doing
that
five
times
in
a
row,
you've
already
wasted
15
minutes
only
deploying
the
application
and
attaching
a
debugger.
So.
B
This
is
probably
the
most
asked
questions
that
we
get
user
when
we
show
this
thing.
The
easy
way
we
try
to
solve
this
at
the
moment
is
whenever
we
redeploy
the
application
and
try
to
attach
the
debugger,
we
simply
started
with
one
instance
right,
so
this
is
not
something
that
will
solve
all
of
your
problems
and
we're
actively
looking
into
integrating
with
a
service
mesh
to
redirect
traffic
to
a
single
instance,
but
this
is
currently
working
development.
What.
B
That
is
definitely
something
it's
my
opinion.
First
of
all,
I
definitely
not
recommend
using
something
like
this
in
production.
First
of
all,
if
you
have
only
have
one
instance
and
you're
blocking
it,
no
other
clients
can
get
requests
out
of
your
innocence.
So
it's
definitely
not
something
that
we
recommend.
B
So
just
to
recap,
what
we've
done
here
with
we
have
an
opinionated
workflow
with
open
source
and
pluggable
components.
So
just
as
I
said
earlier,
if
you
have
another
tool
that
you
want
to
use
to
redeploy
your
application
or
we
have
another
tool
to
forward
your
ports
locally
or
another
ID
for
debugging,
you
can
definitely
mix
and
match
all
of
the
tools
that
you
want.
A
So
if
you
want
to
get
involved,
you're
welcome
to
check
out
the
draft
code
base.
It's
on
github,
slash
user,
slash
draft
check
out
the
BS
code.
Extension
linker
gee,
we're
big
fans
of
the
user
experience
on
the
developer
experience
there
and
we
highly
recommend
watching
Edith's
talk
on
squash
from
coop
Canasta
in
2017.
A
B
As
next
steps
for
the
things
that
you've
seen
so
far,
number
one
on
our
list
is
performance
improvements.
Whenever
you
build
container
image,
so
the
trick
that
we've
did,
we
done
we've
done
so
far.
Is
we
didn't
actually
update
the
application
that
much
so
what
we're
trying
to
do
is
improve
the
performance
when
building
application
to
only
build
the
top
layer
that
change
three
application.
B
The
next
thing
is
we're
looking
into
building
a
draft
plugin
for
linker
D,
so
that
whenever
you
spawn
an
application
in
your
cluster,
we
automatically
inject
the
sidecar
for
linker
D,
so
that
your
traffic
automatically
shows
in
the
linker
D
dashboard.
Another
thing
that
we're
actively
trying
to
do
is
come
up
with
best
practices
for
configuring
draft
PACs,
as
well
as
for
well-known
and
popular
front-end
and
back-end
experiments
and
examples
as
well
we're
looking
at
pluggable
builders
and
deployers
for
draft.
B
As
I
said
at
the
beginning,
squash
is
using
in
cluster
components
to
attach
debugger
to
processes.
Draft
all
it's
doing
is
its
building
the
container
image,
it's
adding
the
debug
information
inside
a
container
or
through
a
sidecar
container.
So
it's
not
using
squash.
It's
not
using
implants
or
components
at
all.
B
B
A
B
So
the
question
is
how
you
would
handle
something
like
this
in
a
multiple
person
team.
Well,
it
depends
on
how
your
team
works
and
how
they
use
kubernetes
clusters,
so
you
can
definitely
have
one
cluster
per
team
or
have
one
namespace
per
person
in
the
team
or
one
namespace
per
person
or
per
team.
If
there
is,
it
boils
down
to
how
the
team
works.
In
theory,
this
this
allows
you
to
debug
the
same
application
from
the
exact
same
deployment
from
two
instances
with
no
problem.
B
So
the
question
is:
if
you
can
use
this
to
attach
to
something
that's
already
running,
the
answer
is
yes:
behind
the
scenes
we're
using
the
vs
code,
debug
process
of
any
debugger,
that's
supported
by
vs
code,
and
that
can
do
remote
debugging
can
be
used.
And
yes,
it's
something
on
a
feature
list
that
we
have
to
support
right
now,.