►
From YouTube: Conprof - Profiling in the cloud-native era - Matthias Loibl, Polar Signals & Kemal Akkoyun, Red Hat
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Conprof - Profiling in the cloud-native era - Matthias Loibl, Polar Signals & Kemal Akkoyun, Red Hat
For years Google has consistently been able to cut down multiple percentage points in their fleet-wide resource usage every quarter, using techniques described in their “Google-Wide Profiling” paper. Ad-hoc profiling has long been part of the developer’s toolbox to analyze CPU and memory usage of a running process, however, through continuous profiling, the systematic collection of profiles, entirely new workflows suddenly become possible. Matthias and Kemal will start this talk with an introduction to profiling with Go and demonstrate via Conprof - an open-source continuous profiling project - how continuous profiling allows for an unprecedented fleet-wide understanding of code at runtime. Attendees will learn how to continuously profile Go code to help guide building robust, reliable, and performant software and reduce cloud spend systematically.
https://www.parca.dev/
A
A
B
Hello:
everyone,
my
name
is
kamagapkuin,
I'm
a
senior
software
engineer
at
older
signals
as
well.
I
just
recently
joined
previously.
I
was
working
for
reddit
besides
working
on
cool
things
in
polar
signals,
I'm
also
a
maintainer
of
thomas
project,
prometheus
client
go
and
cue,
prometheus
and
now
parker.
B
B
A
B
B
So
today
we
actually
focus
on
the
sampling,
profilers
and
because
it's
more
feasible
to
build
upon
the
processes
that
we
have
today,
so
how
it
works
is
it's
actually.
Let's
say
we
want
to
profile
a
process
for
10
seconds
we
collect
and
aggregate
all
of
the
data
and
then
periodically.
We
just
collect
this
data
from
the
processes
process
and
this
since
the
samples
snapshots
like
the
net,
not
that
infrequent
and
that
frequent
for
the
cpus.
This
has
really
low
overhead
to
check
this
data.
There
is
also
a
nice
paper
from
google.
B
You
can
check
out
in
the
video.
So
today
we
gonna
specifically
talk
about
a
little
bit
about
p.
Prof
p.
Proof
itself
is
a
profiler.
It
has
like
cross
language
support
and
that's
why
we
kind
of
picked
that
so
p
prof
a
tool
from
google
and
it
actually
specified
an
open
format
in
form
of
protograph,
and
it's
like
reads
a
collection
of
profiles
from
the
processes.
It
can
either
use
an
http,
endpoint
or
it
can.
B
It
can
use
a
local
file
in
formatted
protocol
product,
and
this
tool
is
also
using
dot
visualizations
using
graphics
you.
It
can
actually
visualize.
I'm
sure,
like
you,
have
seen
a
couple
of
demos
how
you
can
use
the
keyboard.
People
has
a
lot.
A
lot
of
language
supports
some
of
language
supports
already,
and
some
of
them
is
like
really
good,
like
gold,
because
it's
also
from
google
and
some
of
them
is
just
really
getting
there.
But
since
prof
is
an
open
format
and
you
can
actually
just
like
any
language.
B
Runtime
can
actually
support
this.
So,
let's
dive
into
code
and
see
these
things
in
action
in
the
left.
You
see
a
code
piece
and
it's
just
doing
some
iterations
and
doing
some
function
calls
and
when
this
program
is
running,
we
are
like
doing
a
sampling
profiling
and
from
the
samples
we
kind
of
end
up
something
like
in
the
right
way.
This
format
in
the
right
is
called
folded,
stack,
trace,
it's
par
from
brandon
greg.
B
But
it's
like
a
lot
of
redundant
information
that
same
folded,
stack
trace,
can
be
interpreted
in
the
keeper
format
as
aggregation
of
the
locations
and
how
many
times
we
actually
seen
that
step
choice.
In
order
to
understand
this
p
pro
format
more
in
that
you
can
actually
check
out
a
blog
post
from
matthias,
which
is
already
in
the
polar
signals
blog
you
can
have
the
link
inside
the
slides.
B
So
we
already
talked
about
people
can
actually
fetch
the
profiles
using
an
http
endpoint.
You
can
in
this
slide,
you
can
see
how
you
can
actually
instrument
your
application
to
actually
expose
that
profiling
endpoints.
It
is
super
simple
for
go
itself,
it's
already
built
in
the
runtime
standard
library.
You
can
just
like
get
the
net
http
prof
library
and
you
can
just
register
those
handlers,
and
you
already
have
those
endpoints
exposed
from
your
process.
B
B
You
need
to
have
something
to
actually
fetch
that
from
the
exposed
end
points
for
because
of
those
reasons
like
there
are
a
lot
of
hombre
workflows
and
it's
not
like
actually
automated,
or
there
are
some
scripts
laying
around
specific
to
your
organization,
and
you
just
like
use
them.
So
in
order
to
fix
those
things
which
is
the
actual
topic
of
our
talk,
is
continuous
profile.
A
A
We
don't
know
so
what
we
would
really
love
is
we
want
to
use
prof
to
create
the
the
profile
samples
we
want
to
sample
every
so
often
we
want
to
do
it
with
little
a
little
overhead
due
to
the
sampling
as
kamal
explained,
and
then
additionally,
we
hope
to
now
get
the
profiles
right
before
these
unkills
happen,
and
all
of
that
should
be
done
automatically
rather
than
by
hand.
We
don't
want
the
menu
of
the
toilet,
so
yeah
enter
continuous
profiling
again
just
visualizing.
We
want
to
do
this
like
over
time.
A
A
A
B
Parka
is
the
new
open
source
project,
it's
a
continuation
of
homepro,
as
matthias
already
mentioned,
it's
a
neutral.
It
has
a
mutual
government's
organization
model
and
like
since
we
just
recently
built
that
we
still
waiting
for
the
contributions.
So
if
you
are
into
this
or
check
it
out
and
please,
contributions
are
welcome.
B
Parker
itself,
heavily
inspired
by
prometheus,
like
all
of
us,
are
kind
of
coming
from
the
prometheus
community.
That's
why
we
like
carry
over
a
couple
of
things
that
work
for
prometheus,
such
as,
like
single
studies,
statically
statically,
linked
to
the
binaries,
to
ease
up
the
operational
work.
We
carry
over
the
multi-dimensional
label
model,
which
you
can
actually
share
these
labels
with
as
prometheus,
and
we
can
just
like
script
and
populate
all
those
labels
in
the
park
itself
as
well.
It's
using
the
same
service
discovery
and
it
has
like
new
cool
built-in
storage
as
well.
A
B
This
is
one
of
the
major
differences
between
compro
and
parka.
Now
the
agent
is
and
proposed
another
way
to
discover
the
workloads
in
your
running
system.
It's
using
ebpf
and
it's
kind
of
try
to
find
off
the
c
groups
that
is
running
on
your
system
and
from
those
c
groups
is
trying
to
understand
where
the
cpu
and
memory
or
the
io
reso
resources
are
being
used.
It
captures
the
current
stack
and
traces
x
many
times
per
second
and
created
an
analysis
of
that.
B
You
don't
need
to
change
any
code.
You
just
need
to
drop
the
park
agent
in
your
nodes
and
then
it
can
start
scraping
the
profiling
data
and
send
it
over
to
the
parka
backend
itself
so
to
check
out
this
a
high
level
overview
of
this
profiler
it
just
like
discovers
c
groups
from
the
target
provider.
This
could
be
any
container
runtime.
B
Docker
or
cryo,
and
then
from
that
it
kind
of
loads,
ebpf
a
little
ebpf
program
per
c
group
and
and
start
collecting
data
from
that
it
reads
that
data
right
into
the
bpf
map
and
from
that
vpnf
map
it
transforms
data
to
pro
approve
format
and
then
extract
the
symbol,
the
symbolization
information
and
then
send
those
data
over
to
server
and
then
flashes
the
buffer
and
starts
doing
the
same
thing
again.
This
is
a
low
frequency
event
and
that's
why,
like
it,
doesn't
have
a
lot
of
overhead
for
running
systems.
B
B
So
let's
have
a
look
at
the
allocated
space
and
in
here
we
can
see
that
over
an
hour
like
how
parker
is
allocating
bytes.
So
we
can
pick
a
profile
in
nearly
an
hour
ago
and
we
can
see
that
like
which
function
is
allocating
more
bytes
or
whatnot.
But
all
in
all
this
single
profile
doesn't
give
a
lot
of
information
to
us.
What
we
can
do
is
we
can
actually
compare
the
profiles
so.
B
A
profile
from
an
hour
ago
and
more
recent
one,
and
now
we
can
actually
compare
these
profiles.
This
is
getting
a
diff
of
those
profiles
and
it's
showing
like
how
much
batch
we
are
allocating
in
which
function
and
the
it's
all
color
coded
and
the
more
red
indicates
it's
more
bites.
Actually,
the
arcade
one
thing
we
can
also
see
check
out
is
the
cpu
samples
from
the
agent
itself.
B
A
Okay,
that
was
a
fantastic
demo,
so
let's
talk
about
the
storage
and
what
really
makes
parka
parker
and
why
we
didn't
continue
with
controv
as
a
project
right.
So
previously,
controv
itself
just
got
the
profiles
and
took
the
profiles
as
they
came
in
individually
gzipped
and
stored
them
in
a
slightly
modified,
prometheus
time
series
database,
where
we
could
store
bite,
slices
and
not
float
64
anymore,
and
that
was
fine
to
get
like
a
proof
of
concept
out
of
the
door.
But
the
compression
was
great.
A
So
during
a
talk
at
prom
con
earlier
this
year,
I
explained
how
we
improved
the
compression
by
basically
storing
the
uncompressed
profiles
and
then
in
groups
compressing
them
together
which
got
us
between
like
20
and
50
percent
of
improvements.
But
it
wasn't
that
much,
and
it
was
only
like
this
usage
to
begin
with
and
we
wanted
to
improve
on
computation
and
throughput
and
everything.
So
we
needed
something
better
still
and
that's
why
we
created
parker's
tcb
and
that
is
written
from
scratch.
A
So
it
is
still
inspired
by
prometheus
as
again
the
same.
That
was
true
for
the
service
discovery,
and
the
scraper
is
true
for
the
storage
and
it
uses
a
separate
meta
metadata
to
store
things.
It
handles
stack
traces
in
the
storage
as
a
first
class
citizen
and
has
different
chunk
encodings
for
specific
various
data
and
to
kind
of
like
walk
you
through
the
entire
architecture.
A
B
Set
from
the
service
discovery,
we
have
already
built
in
parker
itself
from
that
we
get
the
row
profile
and
we
just
like
using
p
proof
itself
parse
and
validate
that
data
from
that
data.
We
start
building
a
memory
representation
of
that
in
parker
itself,
so
the
profiles
track
or
profiling.
The
memory
looks
like
this.
The
in
this
format,
the
samples,
mapping
and
location
and
functions
are
the
relevant
things
that
we
are
actually
looking
for.
B
So
using
that
data
schema,
then
we
got
an
actual
data
in
it.
It
looks
like
this.
We
have
a
sample
value
and
attached
metadata
information
in
it
from
that
data.
We
built
a
model
in
the
memory
and
we
ingest
the
metadata
in
our
freshly
built
metastore,
and
by
doing
so
we
eliminate
the
redundant
information
redundant,
meta
information
that
we
have
in
the
profiles
right
now.
This
metadata
store
is
implemented
in
sqlite
and
in
memory
sqlite.
B
So
after
we
script
that
metadata
from
this
people
of
profile,
we
ended
up
with
a
bunch
of
location,
ids
and
corresponding
sample
values.
To
that
location
ids,
but,
as
you
can
see,
those
location
ids
actually
builds
a
tree.
That's
actually
what
we
do
and
we
build
a
tree
out
of
those
location
ids
and
we
store
those
sampled
values
as
nodes
in
the
tree,
and
this
is
how
we
actually
store
the
data
in
this.
In
the
packet
storage.
A
So
we
can
use
run
length
encoding
for
those
to
to
really
store
as
little
data
as
we
have
to
okay
cool.
Now
that
we've
taken
care
of
storing
or
creating
and
getting
the
series
and
then
storing
the
timestamp
duration
and
periods,
what
are
we
actually
doing
with
the
profile
tree
with
the
individual
stack
tracers,
and
how
do
we
store
them
over
time
right?
So,
as
kamai
said
earlier,
we
do
get
this
profile
tree
with
the
stack
trace
and
and
the
corresponding
values,
and
we
take
those
as
a
struct
and
those
those
profile.
A
Can
differ
what
this
value
belongs
to
really
and
we
can
use
those
to
then
walk,
walk
the
tree
and
ingest
the
values
like,
so
we
basically
walk
the
tree,
and
then
we
see
for
every
location
key
that
we
see.
We
store,
for
example,
for
the
zeroth
key,
which
is
always
the
root.
We
start
on
the
cumulative
value
for
g6,
and
then
we
walk
the
tree
with
the
stack
and
install
the
other
values.
A
I'm
only
looking
at
the
cumulative
values,
the
flat
values
are
kind
of
ignored,
but
it's
the
same
for
them
as
well.
Next
script,
another
profile
tree
gets
created
and
we
want
to
append
that.
So
again,
we
get
the
same
in
this
case,
the
same
profile
tree
and
we
just
take
a
look
at
the
individual
keys
and
append
to
those
chunks.
The
values
now
gets
a
bit
more.
A
We
actually
get
a
profile
tree
where
one
node
is
missing,
so
we
append
all
the
the
values
to
the
keys
that
we've
seen
and
we
really
just
ignore
appending
the
the
anything
to
to
the
key
that
didn't
exist
and
again
another
edge
case
or
another
use
case.
Is
we
see
another
stack
trace
that
didn't
show
up
earlier
and
because
remember,
we
only
start
the
timestamps
once
we
need
to
work
off
of
the
index
for
duration
periods,
but
also
all
these
values.
A
We
can't
just
like
store
them,
the
timestamps
and
the
values
together,
but
we
need
to
work
off
the
index.
So
we
need
to
ingest
the
11
here
for
the
third
index
and
we
do
that
by
just
appending
three
zeros
and
then
appending
it
right
there.
The
index
three
now,
you
might
think,
might
be
thinking.
Well,
if
we're
going
to
read
back
this,
what
about
the
the
missing
values
right
there
and
that
is
called
sparseness?
A
A
It
stores
all
the
metadata,
uniquely
in
the
sqlite
database
right
now,
and
then
we
append
this
to
a
tsdb
which
is
still
a
lot
like
prometheus,
but
it
works
on
top
of
the
profile
tree
and
then
every
chunk
for
the
different
types
of
values
also
works.
Just
like
prometheus,
okay
cool.
Now
that
we've
seen
how
to
store
profiles
in
parker's
tsdb.
How
do
we
actually
query
things
again?
A
We
do
use
grpc
and
we
can
send
a
query
request
next
to
the
mode
which
specifies
if
we
want
a
single
profile
or
a
diff
or
a
merge
type.
We
do
need
to
give
it
a
single
profile
in
our
case,
which
we
are
interested
in,
and
the
single
profile
which
you
can
see
at
the
very
bottom
takes
a
time
as
a
timestamp
and
a
query.
So
what
does
it
look
like?
A
The
time
is
actually
just
a
unix
timestamp
and
the
query
is
just
like
you
know
from
prometheus
almost
like
a
prompt,
ui
query
where
you
give
some
label
matches
and
that
will
match
all
the
series
that
that
you're
looking
for
and
then
in
memory.
The
tsdb
is
creating
a
wrapper.
The
query
that
will
look
five
minutes
earlier
from
the
timestamp.
You
want
five
minutes
later
and
then
we'll
select
all
the
series
based
on
the
selector
that
that
you've
given
and
then
we're
in
the
first
for
loop.
A
A
A
For
that
reason,
okay,
now
that
we've
talked
all
about
the
parka
tsdb.
What's
on
the
roadmap
for
parker,
we
want
to
have
persistence
on
disk.
We
want
to
be
able
to
persist
the
data
that
we
have.
We
want
to
improve
querying.
We
want
to
be
able
to,
for
example,
only
query
sub
stack
traces
where
we
ignore
everything
else
in
the
tree
and
then
once
we've
done
all
the
math,
we
can
return
that
symbolize
it
and
show
it.
That
would
be
fantastic.
A
Additionally,
we
can
still
improve
the
symbolization.
There's
lots
of
work
to
be
done
in
the
meta
store.
The
sql
parts
are
are
also
still
ready
for
improvements,
but,
most
importantly,
we
want
really
you
to
get
involved
with
parker
and
the
community
for
you
to
try.
Parker,
open
issues
open
for
requests
just
contribute,
and
we
would
really
love
to
create
a
community
about
foreign
performance,
like-minded
people
based
on
parka.