►
From YouTube: Low-Overhead Tracing Using eBPF for Observability into Kubernetes Apps and Services - Gaurav Gupta
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Low-Overhead Tracing Using eBPF for Observability into Kubernetes Apps and Services - Gaurav Gupta, SAP Labs (Intermediate Skill Level)
In the Kubernetes world, there is a need to build observability around apps and backing services deployed in Pods. The observability should allow to capture on demand low level metrics at a very low overhead. The proposal is to use ebpf as the tracing technology to capture details at kernel and user level, and generate on demand flamegraphs, heat maps for applications & backend services. The in-kernel BPF JIT compiler, bpf program verifier and in-kernel storage maps implementation allow users to run the programs in kernel that can never interfere with the kernel negatively, without much overhead. These features make eBPF programs safe to run in production, and allow admins and engineers to collect crucial data from systems for performance analysis and monitoring. Link: https://docs.google.com/presentation/d/1tnhlVqddPQ7wvhxp0_YiqCrowjB29RqV3sK2U7dKPug/edit?usp=sharing
About Gaurav
Gaurav is a Software Engineer at SAP Labs, working with the Cloud Platform team. Gaurav is currently working on design and implementation of Service Brokers for SAP's Cloud Foundry and Kubernetes Cloud Platforms. He also has experience in Linux Kernel and User space profiling and tracing on these Cloud Platforms.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
A
So
what
is
BPF
for
BPF
is
part
of
the
Linux
kernel
and,
as
we
see
here,
BPF
is
a
Linux
kernel,
virtual
machine
which
runs
inside
the
kernel
and
which
helps
user
to
attach
handlers
or
programs
to
K
probes.
You
probes
or
trace
points
and
perfo
vents,
so
user
rights
BPF
program
in
the
user
space
and
compiles
it
to
BPM,
bytecode
and
loads
into
the
BPF
kernel
machine
kernel
virtual
machine
through
verifier
verifier
checks
for
the
validity
of
the
BPF
program
that
is
submitted
by
the
user.
A
There
are
certain
things
you
are
not
allowed
to
do.
In
BPF
programs,
for
example,
you
can
turn
loops
in
BPF
program
and
to
read
any
of
the
kernel
variables
you
need
to.
First
copy
it
to
the
BPF
stack
so
that
there
are
no
security
holes
so
in
in
in
the
code
also,
it
has
its
own
jet
compiler,
which
makes
it
really
efficient
to
run
in
the
production
environment,
so
it
reduces
its
own
overhead
when
this
program
is
run
in
today's
functions
of
kernel
or
the
user.
A
Space
BPF
system
also
provides
kernel,
Maps
now
BPF
maps
in
which
data
can
be
aggregated
into
the
kernel
itself,
which
can
then
be
read
from
the
user
space
or
synchronously
it
into
a
particular
interval
of
times.
So
that
way,
you
have
that
way.
The
not
all
the
data
is
copied
to
the
users
user
space.
Only
the
summary
of
the
data
can
be
copied
to
the
user
space.
Further,
reducing
the
overhead
of
the
tracer
tools.
A
A
This
here
is
a
link
provided
or
is
a
page
on
Cuba
native
blog,
which
which
talks
about
different
EVPs
projects
and
how
they
can
be
used
in
Cuba
notice.
This
is
a
screenshot
from
V
scope.
Here
we
see
that
they
are
different
part
I
own.
The
edges
that
we
see
here
are
the
connections
between
different
parts.
These
edges
can
also
be
drawn
using
Eva
TFO
program
by.
A
A
One
of
the
tools
that
we'll
be
talking
about
is
BCC
and
in
the
in
the
image
that
we
saw
earlier.
Bcc
is
the
user
space
component,
which
provides
us
the
front-end
libraries
through
which
VPS
program
from
the
higher-level
languages
can
be
compiled
into
vpf
programs.
It
also
provides
helper
functions
to
read
the
maps
or
the
data
that
is
aggregated
from
vpf
code
into
the
user
space
ascend
honestly.
A
A
Today
we
will
be
focusing
on
security
aspect
and
the
observability
aspect
of
it.
So
using
EVP,
F
programs.
We
can
track
new
processes
that
are
getting
spawned
in
the
system
and
also
we
can
monitor
the
PT
write
it
order
TTY
or
the
pts
devices
that
that
are
open
when
some
user
takes
a
shell
login,
for
example,
and
for
observability.
A
A
This
is
a
typical
example
of
BCC
code
snippet,
which
is
taken
from
one
of
the
tools
available
in
BCC
toolchain,
which
is
called
func
latency.
So
here
we
see
that
we
are
attaching
a
que
probe
to
the
function.
Vfs
read,
we
register
a
handler
trace,
func
entry,
which
is
called
whenever
VFS
read,
is
called.
This
function
gets
the
PID
and
the
timestamp
at
which
this
was
called
and
stores
it
in
a
associative
array
into
the
BPF
map,
which
is
in
the
kernel
itself.
A
This
tool
also
registers
a
return
probe
and
the
handler
is
trace
func
return,
which
is
called
whenever
the
VFS
read
returns.
What
this
function
does
is
that
it
it
it
takes
the
PID.
It
looks
up
that
PID
into
the
storage
map,
which
was
which
was
saved
by
the
entry
function
and
it
finds
out
the
Delta,
and
then
it
has
the
latency
of
the
function
that
was
seen
by
VFS
read,
ended
further
stores
in
another
map
provided
by
BPF,
which
can
be
read
from
the
bcc
is
user
comparing
synchronously
after
every
particular
interval.
A
time
interval.
A
A
The
vertical
columns
that
we
see
here
are
individual
stack
traces,
so
each
column
here
from
the
top
is
a
stack
trace
and
the
x-axis
is
nothing,
but
these
stack
traces
are
just
sorted
alphabetically,
but
the
width
of
each
column
shows
the
or
is
proportional
to
the
time
taken
or
the
time
a
particular
stack
was
scheduled
of
the
CPU
so
for
on
the
CPU
frame
graph,
the
width
will
denote
how
much
time
it
it
spent
on
the
CPU
and
for
foreign
of
CPU
flame
graph.
It
denotes
how
much
time
it
spent
of
the
CPU.
A
The
thread
to
also
was
she
dueled
out,
mostly
because
sleep
was
called,
and
through
this
we
can
easily
visualize
that
which
part
of
the
code
which
thread
was
is
not
optimized.
Here
we
see
the
garbage
collection
stack,
also
here
too
so.
Details
like
this.
We
can
get
from
BPF
tools
with
a
very
low
overhead
of
its
own.
A
Evp
F
programs
can
be
attached
to
see
groups
one
of
the
projects
from
Ken
woke
season,
8,
which
attaches
EVP
F
programs
to
containers
or
pods,
and
it
gathers
the
network,
starts
from
the
pod
and
then
pushes
it
to
prometheus,
which
can
so.
This
is
a
great
way
of
visualizing
networks,
now
network
states
per
part
with
minimal
overhead.
A
A
A
What
I
have
for
the
demo
is
a
cluster
that
is
deployed
using
gardener
and
for
the
first
demo,
what
we'll
be
doing
is
we'll
be
generating
of
CPU
time
flame
graph
for
the
processes
running
on
the
host.
So,
for
example,
tom
has
a
post-race
instance
running
in
a
node
in
in
inside
a
pod,
and
he
is
using
another
web
app
to
use
that
Postgres
instance
in
another
part,
and
he
sees
that
there
is
some
lag
or
latency
in
in
in
the
system.
So
he
goes
to
bob
from
operations
and
then
he
asked
him.
A
Ok,
there
is
some
problem
with
the
instance.
How
do
I
figure
out
where
the
problem
is?
Bob
looks
at
the
dashboard
and
sees
okay.
You
know
there
is
a
CPU
high,
CPU
utilization
or
or
the
process
or
particular
process
is
consuming
this
much
of
CPU,
but
still
doesn't
even
answer
that
what
is
wrong
with
the
process
running,
why
it
is
taking
so
much
of
CPU
or
details
like
that.
A
A
A
A
Thank
You
vision,
okay,
so,
but
so
here,
I
have
two
terminals
in
one
terminal:
I'll
be
logging
into
the
PG
Postgres
instance,
and
running
a
benchmark
tool,
PG
bench
and
in
in
second
terminal
I
will
be
running
the
the
of
CPU
time
tool
that
will
collect
the
of
CPU
time
and
generate
the
flame
graph
for
it.
Correspondingly.
A
So
what
I
will
do
is
I'll
start,
so
this
is
the
URL
that
is,
that
is
the
load,
balancers
load
balancer
service
URL,
which
and
the
off
CPU
time
tool
is
exported.
On
this
end,
point
I
will
run
this
tool
for
10
second
duration
and
expect
the
output
in
the
flame
graph
format
and
on
another
terminal
from
the
Postgres
instance,
I
have
launched
the
load
for
Postgres
and,
let's
see
what
this
of
CPU
time
flame
graph
looks
like
so
here,
okay,
so
I
was.
A
So
what
what
of
CPU
time
to
this
tool
does?
Is
that
it
traps,
or
it
races
the
scheduler
process
of
the
Linux
kernel
and
whenever
the
scheduler
schedules
out
to
any
process,
it
saves
it
stack
and
saves
the
time
at
which
it
was
scheduled
out
and
whenever
that
process
is
scheduled
back
into
the
system.
A
It
calculates
the
time
difference
at,
for
which
this
stack
was
off
the
CPU
and
it
keeps
aggregating
that
data
in
the
kernel
and
which
is
read
from
the
user
space,
and
that
data
is
fed
to
the
flame
rough
tool
that
I
talked
about
earlier,
and
then
it
sends
the
data
to
this
back
to
the
requester.
So
here
we
see
that
the
Postgres
process
spent
most
of
the
time
of
the
CPU,
because
it
was
running
this
function,
PQ
flush.
So
details
like
this,
we
can
gather,
and
similarly
there
is
a
Java
part
that
is
also
running.
A
A
So
what
I
want
is
that
whenever
someone
logs
into
the
node
of
cuban
it
is
node,
I
I
generate
an
alert.
Maybe
some
siren
goes
off
somewhere.
Maybe
I
get
an
SMS
or
a
call.
So
what
I'll
be
showing
here
is
that
I'll
try
to
log
in
in
the
node
and
what
I
will
be
expecting
is
a
call
on
my
phone
that
says
that
there
was
a
login
attempt
on
the
node
so
yeah.
Let's
do
that.
A
Yeah
here
it
is
so
when
I
pick
this
up,
what
it
says
is
so
here
it
was
I
I
hope
it
was
audible.
So
these
kind
of
full
alerts
can
be
generated.
What
it
does
is,
whenever
someone
logs
in
it
the
handler
is
called,
which
submits
and
even
to
the
user
space
and
the
user-space
component,
handles
that
event
or
synchronously
makes
the
call
to
the
number
provided
pre-programmed
number
provided
in
the
in
the
tool
chain.
A
Now
this
tool
can
also
be
used
to
log
what
the
horse
is
doing,
or
so
someone
has
was
able
to
login
into
the
system.
Now
I
want
to
log
what
that
user
was
doing
on
on
that
on
that
pts
device
that
it
opened
so
for
that
I
can
I
can
run
this
another
tool,
tty
snow
for
n
log,
whatever
was
happening
on
that
particular
character
device,
so
I'll
run
it
for
20
minutes
duration,
but
this
can
as
well
be
triggered
when
someone
was
able
to
login
into
the
system.
A
Something
and
after
the
duration
of
20
20
seconds
after
the
duration
of
20
seconds,
I
see
that
whatever
the
user
was
doing
in
after
taking
the
shell
terminal
is
duplicated
or
I
get
the
log
of
what
that
user
was
seen.
Now
this
can
be
utilized
to
replay
what
the
user
did
and
what
the
effects
were
of
his
actions.
A
A
A
A
A
B
Try
to
do
something
similar
about
kind
of
streaming
back
the
session
and
but
I
couldn't
work
out
how
or
where
to
hook
into
the
system,
to
firstly
know
when
a
new
session
has
started
and,
secondly,
what
to
used
to
track
and
the
session.
So
what
kind
of
M
BPF
folks
are
using
to
track
the
session
itself
and
keep
streaming
the
the
output
back?
Okay,.
A
So
I
believe
the
functions
name
is
TTY
right
from
the
kernel
yeah
so
and
it
gets
all
the
data
that
is
being
written
on
the
terminal
and
that
data
you
gather
either
in
the
in
the
kernel
maps
or
you
submit
an
event
for
it,
which
can
then
be
handled
from
the
user
space
component
and
can
further
be
saved
in
data
structure
which
can
be
returned
back
so
I
believe.
Tty
right
is
the
correct
function,
but.