►
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Effortless Profiling on Kubernetes - Eden Federman, Verizon
Finding performance issues in applications is a hard problem. It's even harder when those applications run on Kubernetes. The needed profiling tools are usually not included in the container image and require a new deployment in order to be enabled. This session covers the best practices and the different tools available to profile Java, Go, Python, and Ruby applications.
A
So,
hey
everybody!
Thank
you
for
coming
to
my
talk
and
staying
so
late.
So
today
we're
going
to
talk
about
effortless
profiling
on
kubernetes
a
little
bit
about
myself,
so
my
name
is
eden.
I
am
currently
an
engineer
at
yahoo.
I
have
been
coding
for
almost
15
years
now,
I'm
really
into
everything,
observability
and
profiling.
A
A
So
profiling
is
the
act
of
analyzing,
the
performance
of
applications
in
order
to
improve
poorly
performance
section
of
code
and
simple
words
is
basically
profiling
is
the
process
you
do
when
you
have
some
slow
running
application,
and
you
want
to
understand
why
this
application
runs
so
slowly,
one
of
the
most
popular
ways
to
visualize
a
profile
to
present
the
profile
is
by
generating
something
that
is
called
flame
graph
flame
graphs.
Look
something
like
that:
it's
multiple
stack
traces
stacked
on
top
of
each
other.
A
Basically,
the
y-axis
is
the
stock
depth.
It's
basically
the
stack
choice
itself.
The
x-axis
is
the
sample
population.
Basically,
the
wider.
The
flame
is
and
the
longer
a
method,
a
call
took,
the
color
is
usually
used
in
order
to
differentiate
between
different
type
of
of
method.
Calls.
For
example,
here
green
can
be
java
code,
red
can
be
c
code
and
orange
can
be
some.
I
don't
know
kernel
space
code
and
the
order
is
usually
not
that
important
frame
graphs
are
usually
ordered.
Alphabetically.
A
A
So
those
two
pain
points
are
only
getting
harder
when
dealing
with
applications
running
on
top
of
kubernetes.
So
let's
say
we
want
to
add
a
profiler
to
our
container
image.
We
need
to
prof
to
modify
the
image
itself.
We
need
to
include
the
relevant
profiler
binary
and
then
modify
the
application
code
after
we
did
all
those
changes.
We
have
to
build
the
modified
image,
push
it
to
some
registry
and
deploy
it
to
our
cluster.
A
A
The
way
it
works
is
that
cube,
ctl
flame
is
basically
a
facade
for
choosing
the
best
profiler
for
every
task.
So
if
we
are
trying
to
profile
a
golden
application,
it
will
choose
an
ebpf
based
profiler.
If
we
try
to
do
a
profile
for
java
or
any
jvm
based,
it
will
choose
async
provider
for
python.
It
will
be
pi
spy
and
for
ruby
will
be
rbspy
and
many
more
languages
are
coming
soon.
A
A
This
way
we
can
inject
a
profiler
into
a
running
container,
a
running
pod
without
having
to
do
any
code,
modifications
or
any
restarts
so
enough
with
presentations.
Let's
do
some
profiling,
so
in
order
to
demonstrate
this
tool,
this
profiling,
I'm
going
to
use
a
micro
services
demo
from
google
cloud.
It's
like
a
demo
of
an
e-commerce
app
which
contains
like
10
micro
services
in
five
different
programming
languages
and
even
some
reddish
database.
A
We
are
going
to
focus
on
two
main
applications,
one
in
java
and
one
in
python.
The
java
one
is
the
ad
service,
the
application
that
is
responsible
for
displaying
advertisements
and
in
python
it's
the
recommendation
service.
It's
the
application
that
is
responsible
for
showing
recommendations
on
products.
So
just
to
give
you
like
a
an
idea.
What
is
this
application?
It's
simple
e-commerce
site.
You
can
buy
products
okay
to
the
demo,
so
I
already
have
this
application
deployed
here
locally
on
my
mini
cube
cluster.
A
So,
let's
start
with
doing
the
profiling
first
manually
and
then
we'll
try
to
improve
this
process
using
cubesitl
flame.
So
the
first
thing
we
need
to
do
is
to
modify
the
container
images
to
include
profiler.
So,
let's
start
with
the
recommendation
service,
I'm
going
to
go
ahead
and
modify
the
image
so
the
I
already
did
the
research
and
choose
and
chose
to
use
pi
spy
profiler,
because
it's
a
python
application.
There
are
many
other
profiles,
profilers
you
can
use
and
to
add
it.
A
To
simply
add
it
to
the
container
image,
I'm
going
to
add
a
pip
install
pi
spy
command,
and
I
already
have
scaffold
running
in
the
background
and
scaffold
is
like
a
local
ci
cd
that
automatically
watches
all
the
files
that
I'm
changing
and
it's
going
to
build
the
image
for
me
and
push
it
to
your
registry
and
deploy
it
to
the
cluster.
Once
I
will
save
this
docker
file.
So
let's
go
ahead
and
do
that,
let's
watch
him
do
his
thing.
A
A
A
A
A
For
java,
I
already
did
the
research
and
I
chose
to
to
run
it
with
async
profiler,
and
this
time
we
are
going
to
add
it
to
the
container
image
a
little
bit
differently.
We're
going
to
do
wget
download
the
latest
release
from
github
extract
it,
the
data
gzip
file
and
move
it
to
slash,
opt
slash.
Profiler
again,
I
will
save
this
file
and
scaffold
will
build
the
the
image
for
me
and
deploy
to
the
cluster.
A
A
A
A
The
third
step.
The
third
step
is
to
actually
execute
into
the
container
and
invoke
the
profiler.
There
are
two
main
issues
with
this
step:
first
executing
to
into
the
container
and
running
this
command
actually
assumes.
There
is
some
a
shell
shell
on
the
container
on
the
image,
which
is
not
always
true,
for
example,
if
you're
using
a
really
tiny
image
like
distroless
or
images
like
that,
you
don't
always
have
shell
in
your
container
image.
So
this
whole
manual
process
might
not
even
work,
but
let's
say
you
do
have
a
shell.
A
You
still
have
to
execute
into
the
into
the
container
and
run
some
kind
of
command.
Every
profiler
have
a
different
api.
You
need
to
understand
and
read
the
documentation
and
know
what
are
the
command
line
arguments
and
what
every
argument
does
and
after
you
did
all
this,
you
still
have
to
go
ahead
and
copy
the
file
over
to
your
machine.
So
you
can
examine
it
and
open
it
and
visualize
it
in
some
kind
of
browser,
for
example,
so,
overall,
not
a
very
friendly
process.
A
A
This
way
we
can
see
all
the
con
on
the
airpods
that
created
in
real
time,
and
it
will
help
us
to
understand
better
how
qctl
frame
works
behind
the
scenes,
and
here
I'm
going
to
actually
execute
the
commands.
So,
as
I
said,
area
cube
city
flame,
it's
basically
a
cube,
ctl
plugin.
So
after
you
install
it,
you
simply
use
it
like
any
other
sub
command
in
cube
ctrl.
So
you
do
cuba,
city
or
flame.
A
So
once
we
run
it,
you
can
see
that
a
new,
a
profiler
pod
is
created
on
the
same
node.
It
will
start
running
it
will
attach
to
the
running
ad
service,
invoke
the
profiler
automatically
for
us
and
we'll
terminate
when
it's
finished,
and
the
file
is
also
copied
over
to
our
machine,
and
the
other
cool
thing
is
that
ad
service
you
can
look
at
the
age
did
not
restart
it
at
all.
A
A
Again
from
10
seconds,
and
let's
call
it
wreck
to
the
svg
again,
we
can
see
that
the
new
profiler
part
is
created
on
the
same
node,
with
the
relevant
profiler
inside.
It
will
attach
to
the
recommendation
service,
run
the
profiler
and
terminate
at
the
end
and
again
we
can
open
the
same
image,
the
same
flame
graphs
and
understand
exactly
what
our
recommendation
service
is
currently
doing.
A
A
A
A
A
They
recently
added
a
new
feature,
called
compile
ones
run
everywhere,
which
makes
running
ebpf
applications
much
more
easier.
You
don't
have
to
have
all
those
linux
headers
inside
your
node
or
container
and
the
third.
The
third
point
is
a
continuous
profiling
tools.
There
are
more
and
more
tools
that
try
to
make
profiling
a
continuously
running
process
with
a
flame
graphs
that
are
generated
continuously
and
always
on.