►
From YouTube: eBPF for Security Observability - Liz Rice, Isovalent
Description
eBPF is a kernel technology that brings superpowers to cloud native tooling. You might already be using eBPF, in the form of CNCF projects like Cilium or Falco, lower level tools like bpftrace or even in the form of seccomp profiles.
Website: https://isovalent.com/
Organized by @Microsoft @kubermatic7173 @SysEleven
Thanks to our sponsors @CapgeminiGlobal , @gardenio, @sysdig, @SUSE @anynines , @redhat , nginx, serve-u
A
Don't
worry
if
you
don't
know
what
evpf
is
yet.
You
seem
well
so
I've
really
been
immersing
myself
in
evpf,
because
it's
a
really
great
foundation
for
instrumenting
applications,
and
particularly
in
the
cloud
native
world,
so
today
I'm
going
to
talk
about
how
we
can
use
ebpf
for
security
and
specifically
security
observability,
so
I
should
probably
start
by
so
actually,
let's,
let's
see
if
this
works
put
your
hands
up.
If
you
are
familiar
with
EVPs
I
can
only
see
a
section
of
you,
but
we'll
see.
A
So,
let's
make
sure
we're
all
on
the
same
page
about
what
we
mean
when
we
talk
about
the
kernel,
it's
the
part
of
the
operating
system
that
interfaces
with
Hardware
and
if
we're
developers,
we
usually
write
code
for
user
space,
but
whenever
our
user
space
applications
want
to
do
anything
that
involves
Hardware
in
any
way,
whether
that's
accessing
memory
accessing
files
sending
a
message
over
the
network.
All
these
things
writing
something
to
the
screen.
All
these
things
require
the
kernel
to
do
these.
A
On
behalf
of
user
space,
the
kernels
also
looking
after
coordinating
different
user
space
applications
running
simultaneously
in
in
their
own
processes
and
the
interface
between
user
space
applications
and
the
kernel
is
called
system
calls
usually
as
developers.
We
don't
have
to
worry
about
the
system
core
will
low
level
interface
because
we
use
higher
level
programming
languages
that
have
the
kind
of
hide
the
details
of
system
calls
for
us,
but
underneath
the
covers
are
user.
A
Space
applications
are
making
requests
over
this
system
core
interface
all
the
time
now
with
ebpf,
we
can
run
custom
code
in
the
kernel
and
attach
them
to
events.
So
whenever
this
event
that
we've
chosen,
whenever
it
happens,
it
triggers
our
evpf
program
to
run
and
I
think
to
make
this
concrete,
we
should
look
at
an
example.
So,
if
I'd
bring
up
the
right
screen,
I've
got
a
hello
world
example.
A
Here,
let's
make
this
say
hello
Berlin
and
brought
a
little
bit
of
a
conflict
here
between
making
the
code
readable
and
making
the
well
fitting
everything
on
the
screen.
So
this
part
here
it's
it's
C
code,
and
this
is
the
program
that
I'm
going
to
run
inside
the
kernel.
All
it's
going
to
do
is
Trace
out
a
message
and
then
this
C
code
is
embedded
in
a
string
inside
a
user
space,
a
python
application.
A
My
python
application
is
going
to
load
my
evpf
program
into
the
kernel
and
it's
going
to
associate
it
with
the
system.
Call
Exec
ve.
This
is
a
system
call
that's
used
whenever
we
are
new
when
we
run
a
new
program
and
on
the
virtual
machine
that
I'm
using
here,
there's
a
lot
going
on
so
we'll
see
a
lot
of
exactly
events
being
triggered.
Hopefully,
so
I
have
to
do
this
as
root
and
there
we
go.
We
already
started
to
see
lots
of
messages
being
triggered.
A
A
So
what
I
want
to
sort
of
take
away
from
that
is
well,
partly
that
I
just
loaded
that
program
I
just
ran
it
and
it
started
reacting
immediately
to
exec
ve
event
and
I
also
want
you
to
be
aware
that
we're
getting
this
contextual
information
about
that
process,
we
can
see
what
the
executable
was.
What
the
process
ID
was
time
stamp
and
other
information
that
relate
to
to
that
system
call.
A
So
a
typical
tracing
tool
would
have
a
user
space
part
that
loads
an
evpf
program
and
attaches
it
to
whatever
event
it
is
that
we
want
to
trace.
We
want
to
collect
information
about
and
whenever
that
event
happened,
our
program
would
write
some
events
or
increment
a
counter
and
store
that
result
in
what's
called
an
evpf
map,
and
a
map
is
a
data
structure
that
we
can
share
between
the
kernel
and
user
space
or
between
multiple
ebtf
programs
running
in
the
kernel.
A
There
are
dozens
of
these
command
line.
Applications
that
can
give
us
metrics
or
information,
observe,
what's
happening
in
all
sorts
of
different
aspects
of
the
system,
whether
we're
talking
about
file
systems
or
applications
or
networking
there
are
or
performance.
We
can
use
EVPs
to
collect
data
about
all
of
these
different
aspects
of
a
system.
A
But
it's
important
to
remember
that
there
is
only
one
kernel
per
host
per
virtual
machine
or
bare
metal
machine.
If
we're
using
bare
metal
doesn't
matter
how
many
containers
we're
running
on
that
host,
there
is
only
one
kernel
that
they
all
share
so
whenever
our
containers
want
to
do
anything
interesting
like
reading
and
writing
for
files
or
receiving
messages
from
a
network
whenever
kubernetes
wants
to
create
new
containers,
all
of
these
things
requires
support
from
the
kernel.
A
A
And
even
better,
we
can
get
the
contextual
information
about
the
kubernetes,
pods,
namespace,
node,
etc,
etc.
We
can
get
this
kubernetes
identity
information,
as
well
as
the
the
event
information,
so
I'm
going
to
show
an
example
of
inspective
gadgets,
which
this
is
a
similar.
It's
using
an
Epps
program
very
similar
to
what
we
just
saw
in
open
snow.
It's
going
to
show
us
trials
being
opened,
but
this
time
we're
going
to
get
information
about
the
kubernetes
identity,
that's
responsible,
so
I've
kicked
it
off
and
I'm.
A
A
So
just
flip
out
there,
okay,
so
Inspector
Gadget
is
an
example
of
a
command
line
tool
that
can
give
us
or
can
use
ebpf
to
give
us
event.
Information
combined
with
kubernetes
information,
and
there
are
lots
of
other
tools
in
the
cloud
native
landscape
that
use
ebpf
and
present
information
in
different
ways.
A
One
example
is
the
pixie
project,
which
runs
again
taking
inspiration
from
that
BCC
project
and
all
of
those
different
command
line
tools.
It's
running
very
similar
evpf
programs
on
your
kubernetes
cluster
and
gathering
information,
presenting
it
to
you
in
a
visual
way.
This
is
a
an
example
of
a
flame
graph
showing
how
CPU
is
being
used
across
an
entire
cluster.
A
A
A
Well,
for
me,
I
think
the
difference
is
policy,
so
a
policy
will
tell
us
what
is
the
difference
between
events
that
we
expect
to
see
and
events
that
are
an
indication
that
maybe
something
suspicious
is
happening.
Maybe
something
malicious
is
happening.
Maybe
an
attack
is
taking
place,
so
the
policy
will
tell
us
the
difference
between
good
events
that
we're
not
particularly
interested
in,
because
they're
expected
to
happen
and
malicious
events
that
we
do
want
to
be
told
about,
but
to
really
call
it
observability.
We
need
not
just
to
know
that
those
events
happened.
A
We
also
need
to
be
able
to
ask
questions
of
that
data.
We
need
to
be
able
to
dig
into
understanding
how
that
malicious
behavior
happened.
What
which
of
our
applications
have
been
affected?
Are
there
components
that
we
need
to
stop
or
upgrade
or
or
in
some
way,
act
on
and
in
order
to
really
call
it
observability?
A
So
really
the
difference
here
is
having
a
policy
to
compare
it
to
the
kind
of
activities
that
we
care
about
from
a
security
perspective
are
exactly
the
kind
of
things
that
the
colonel
gets
involved
with
we're
interested
in
spotting
malicious
access
to
data,
whether
that's
in
files
or
memory.
We
want
to
know
if
network
activity
happens,
that
we
don't
expect
like
a
DDOS
attack
or
or
sending
data
to
to
an
unexpected
destination.
We
want
to
know
when
executables
are
run,
that
we
don't
expect.
A
So
when
we
make
a
system
call,
remember
that's
the
interface
that
user
space
applications
use
to
ask
the
kernel
to
do
things
on
its
behalf,
and
we
could
use
ebpf
to
observe
the
entry
to
that
system
call
and
make
decisions
on
whether
or
not
it's
a
malicious
event
or
not.
But
there
is
a
problem
with
this
approach.
If
we
just
naively
look
at
system
calls
it's
totally
fine
for
general
observability
purposes,
but
if
we're
really
serious
about
security,
there
is
a
time
of
checks,
a
time
of
use
issue.
A
This
was
very
well
described
in
a
couple
of
talks
last
year
and
the
the
gist
of
this
is
when
you're,
just
based
application
makes
a
system
call
and
it
passes
in
a
parameter.
Some
of
those
parameters
will
be
pointed
and
the
Cardinal
is
going
to
have
to
copy
the
data
from
that
pointer
into
kernel
space
to
before
it
acts
on
it,
and
there
is
a
window
of
opportunity
for
an
attacker
to
modify
those.
A
A
If
you've
used
tools
like
app
armor
or
SC,
Linux
they're,
using
this
API
to
be
informed
about
invent
events
and
make
decisions
on
whether
or
not
they
are
in
or
out
of
policy,
and
this
API
is
stable.
It's
not
going
to
change
from
one
version
of
the
kernel
to
the
next
and
it's
a
safe
place
to
make
Security
checks.
A
Instead
of
you
using
a
kernel
module
behind
that
API,
we
used
ebpf
programs
and
that's
exactly
what
an
initiative
called
BPF
LSN
does,
but
this
has
all
the
benefits
of
the
LSM
API
in
that
it's
a
stable
and
safe
place
to
make
Security
checks
combined
with
the
benefits
of
being
able
to
load
ebpf
programs
dynamically.
We
don't
have
to
restart
processes
to
get
observability
of
them.
A
A
So
this
is
a
really
great
approach
to
building
serious
security
tools.
This
is
just
an
example,
I
think
in
the
interest
of
time.
I
won't
show
this
in
code,
but
this
is
an
example
of
a
an
LSM
hook
in
code,
and
you
can
see
that
passed
into
our
BPF
program.
Here
we
get
a
data
structure
called
path.
This
is
the
Kernel's
representation
of
a
file
that
we're
about
to
try
and
change
the
permissions
on,
and
we
can.
A
A
A
So
the
question
that
we
asked
ourselves
in
the
psyllium
project
is
we
really
need
to
use
a
an
API
that
is
declared
a
stable
or
are
there
other
parts
of
the
Linux
kernel
that
are
sufficiently
stable
and
that
have
existed
in
the
kernel?
For
a
long
time,
we
can
attach
evpf
programs
to
any
function
within
the
kernel.
A
We
can
attach
it
to
all
kinds
of
different
events,
and
that
includes
the
entry
to
or
exit
from
or
any
Trace
point
in
any
function
across
the
30
million
lines
of
code
that
make
up
the
Linux
kernel,
and
we
know
that
those
30
million
lines
of
code
don't
change
from
one
release
to
the
next.
There
are
huge
amounts
of
the
kernel
that,
although
they're
not
guaranteed
to
be
stable,
they
actually
are
pretty
stable.
They
have
proven
over
history,
not
to
change
very
much.
A
So
what
we've
done
in
the
psyllium
project?
We
have
a
new
member
of
the
Salim
family
called
tetragon
that
uses
the
knowledge
that
we
we
have
amongst
psyllium
developers
of
kernel
functions
that
are
sufficiently
stable,
that
haven't
changed
for
ages,
they're
unlikely
to
change
anytime
seen
and
we
can
use
those
functions
as
places
to
hook
our
ebpf
programs
into
plus.
We
also
already
have
the
kubernetes
context
from
other
parts
of
the
Cillian
project.
A
We
called
it
tetragon
because
there
is
a
family
of
bees
and
you've
probably
noticed
that
bees
figure
pretty
heavily
in
ebtf
logos
and
so
on.
There's
a
family
of
bees
called
tetragonist
get
and
there's
a
really
nice
quote
on
Wikipedia
about
how
this
particular
family
of
these
are
very
small.
They
build
unobtrusive
nests
a
bit
like
how
ubpf
programs
are
small
and
lightweight,
and
this
bee
family
produces
large
amounts
of
honey
and
it's
not
seen
as
a
threat
to
humans.
A
A
So
let
me
bring
up
my
kubernetes
cluster
again
and
stop
Inspector
Gadget,
but
the
start
touch
gone.
So
actually,
let
me
oops
keyboard's
not
behaving
or
my
phone's,
not
behaving
I'll
start
by
just
showing
some
raw
logs,
because
what
this
shows
is
we
get
a
lot
of
information
about
events
generated
on
the
system.
We
can
see
here.
A
In
fact,
there's
the
Inspector
Gadget
that
was
originally
developed
by
Kim
folk
has
generated
an
event
here,
that's
pretty
hard
to
read,
so
we
have
a
a
tool
called
touch
gun
that
we
can
use
to
kind
of
format.
This
output,
not
sure
I,
spelled
that
right.
I
did
say
right,
okay,
so
this
should
now
be
following
events
that
match
my
policy
and
if
I
try
to
edit
a
file
in
the
ETC
directory.
A
A
A
If
we
take
that
further
and
we
combine
it
with
network
information,
we
can
actually
see,
for
example,
if
a
network
connection
is
made
to
a
suspicious
looking
destination,
we
can
find
out
what
was
the
executable
involved
in
opening
that
network
connection
and
what
pop
was
it
running
in
and
when
did
it
happen
and
what
was
the
image
involved
and
so
on
and
so
forth?
This
kind
of
forensics
is
really
useful
for
understanding
how
a
security
event
took
place,
but
can
we
use
ebpf
to
prevent
these
malicious
events
from
happening?
A
A
And
then
you
send
information
to
user
space
and
the
user
Space
Program
take
some
amount
of
time
to
act
on
that
notification.
There
might
be
enough
time
for
the
malicious
act
to
do
something.
Serious
data
might
have
had
time
to
be
exfiltrated.
The
attacker
might
have
persisted
a
threat
onto
your
hard
drive.
Bad
things
could
have
happened
in
even
this
small
window.
A
What
we're
doing
in
tetradon
is
actually
sending
a
kill
or
we
have
the
option
to
send
a
Kill
from
the
kernel
synchronously,
and
it
prevents
at
that
event
from
completing
it
can
prevent
a
system
call,
for
example,
from
completing
so
I'm
gonna
just
change
my
policy,
so
I've
actually
got
two
different,
very
similar
policy
file,
the
difference
being
that
one
of
them
kills
when
it
detects
a
right
to
a
file
in
the
Etsy.
Well,
Etsy
directory,
so
I'll
just
apply
the
tool
version.
A
A
A
A
A
We
can
include
contextual
information
and
data
about
the
kubernetes
identities
involved
and
with
Advanced
knowledge
of
how
the
kernel
works
and
how
EVS
can
be
leveraged.
We
can
even
do
runtime
enforcement
from
the
panel
so
with
that
I'll
say.
Thank
you,
I
hope
it's
given
a
little
bit
of
an
insight
into
why
I
think
ubf
is
super
powerful.
A
We've
got
a
couple
of
reports
that
dive
in
a
bit
more
detail
both
to
the
general
topic.
I
mean
yeah
and
more
specifically,
security
observability
that
are
available
from
the
isobality
website
for
price
of
your
contact
details,
but
there's
also
plenty
more
details,
plenty,
more
information
on
the
selenium
project
website
and
on
the
evpf
community
websites.