►
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Deep Dive: Runtime Security With Falco in Userspace - Loris Degioanni, Sysdig
In this talk Loris will discuss the tradeoffs for using different backend drivers to access systemcall information for cloud native security with Falco. We learn about eBPF, kernel modules, and ptrace(2). Loris will discuss other approaches considered and the performance impact with other solutions such as LD_PRELOAD. Loris shares deep knowledge and expertise in the matter and draws on years of contributing to wireshark, and creating sysdig and falco.
https://sched.co/cpBB
A
Talk
about
a
tool
called
pd
which
puts
these
theories
in
practice
and
then
leaves
some
space
for
for
questions.
First
of
all
me,
my
name
is
loris
de
jouani.
I
am
the
founder
and
cto
of
a
company
called
csd
which
provides
tools
for
security
and
visibility
for
cloud
native
environments
based
on
kubernetes.
A
I
am
also
one
of
the
original
creators
of
falco
in
2016.,
I'm
still
an
active
contributor
to
the
tool
and
before
falco
I
spent
a
lot
of
time
in
open
source
and
networking.
In
particular.
I
was
one
of
the
original
contributors
to
to
wildshark
the
open
source
packet
network
analyzer.
You
find
me
on
twitter
at
laurie
station
all
right.
A
Falco
leverages
the
kernel
of
the
operating
system
to
collect
information
and,
in
particular,
is
based
on
some
technologies
for
collecting
data.
One
of
them
is
called
ebpf
and
stands
for
enhanced
berkeley,
filter
and
we'll
talk
a
little
bit
about
it
later,
but
allows
falco
to
essentially
collect
system
calls
from
the
from
the
operating
system.
A
Very
importantly,
falco
is
designed
to
be
modular
and
consumable.
So
falco
is
a
component
that
we
designed
to
be
very
tightly
embedded
in
the
cloud
native
ecosystem
and
in
the
cloud
native
stack
of
tools
and
is
designed
to
interact
and
interface
with
these
tools
to
be
part
essentially
of
the
pipeline.
A
We
got
accepted
by
cncf
as
assembled
sandbox
project
in
at
the
end
of
2018
and
around
six
months
ago,
in
december
of
2018,
we
were
promoted
to
incubation.
It's
a
big
focus
of
us
to
make
sure
that
falco
is
a
a
project
that
is
a
broad
audience,
but
in
a
way
that
is
completely
open.
It's
community
community
oriented
and
it's
by
driven
in
is
driven
by
the
guidelines
at
the
core
of
of
cncf.
A
Let's
go
a
little
bit
into
architecture
at
the
high
level.
The
firework
architecture
is
relatively
simple:
there
are
inputs,
there
are
rules,
there's
an
engine
and
there
are
outputs,
so
inputs,
of
course,
can
be,
as
I
was
mentioning
kernel.
Events
in
the
form
of
system
calls,
but
also
container
metadata,
kubernetes
events,
kubernetes
metadata
information
and
so
on.
A
Rules
allow
you
to
specify
essentially
conditions
on
these
inputs
and
are
evaluated
at
the
edge
by
farco
engines
that
are
running
on
every
machine
and
they
provide
outputs
that
can
be,
you
know,
can
go
to,
syslog
can
go
to
standard
output,
can
go
to
grpc
or
web
books
can
go.
For
example,
there's
an
integration
for
prometheus,
so
there's
a
bunch
of
ways
essentially
to
consume
falco
outputs.
A
Most
of
the
people
tend
to
send
the
outputs
coming
from
falco
to
through
seeds,
log
or
through
some
other
mechanism
to
a
sim
tool
that
then
can
collect
all
of
this
kind
of
of
stuff
in
a
centralized
place
and
falco,
of
course
puts
a
lot
of
richness
in
these
events
and
a
lot
of
context
and
then
from
the
same
tool.
You
can
essentially
create
dashboards
or
investigate
or
alert
or
do
whatever
you
want.
A
Some
rule
examples,
as
you
can
see,
falco
is
designed
with
the
syntax
that,
as
a
the
goal
of
being
as
lean
and
as
simple
as
possible,
so
maybe
compare
compared
to
other
security
kubernetes
tools
like,
for
example,
oppa
falco
is
a
slight,
a
less
expressive
syntax,
but
also
a
very
simple
syntax
that
allows
you
to
create
sophisticated
stuff
with
one
liners
that
are
pretty
easy
to
grasp
and
pretty
easy
to
to
manage
and
create
new
ones.
For
example,
a
shell
is
running.
A
Container
namespace
change,
so
this
rule,
for
example,
is
trying
to
detect
a
container
trying
to
escape
its
its
name.
Space
is
its
gel
and,
as
you
can
see
in
this
one,
we
look
at
the
certainness
system
call
and,
and
so
on.
You
know,
rules
can
be,
can
become.
You
know
more
and
more
sophisticated,
but
at
the
basic
they
trying
to
be
simple
expressive
and
powerful
at
the
same
time.
A
Okay,
now
that
we've
seen
what
kind
of
rules
we
can
express
and
how
it
works,
let's
look
at
how
falco
collects
the
data,
because
the
whole
value
and
richness
of
a
tool
like
falco
and
the
ability
to
run
at
runtime
and
inform
you
of
of
granular
changes
and
actions
in
your
infrastructure
depend
on
the
ability
to
collect
the
best
data
in
particular,
falco
is
uses.
The
kernel
of
the
operating
system
to
collect
system
calls.
So
here
in
this
diagram,
we
have
a
simplified
diagram
of
a
of
a
machine.
A
In
this
slide,
we
have
a
simplified
diagram
of
a
machine
with
the
kernel
of
the
operating
system,
and
we
have
three
containers
running
on
the
on
this
machine.
With
three
different
container
run
times
you
deploy
falco
by
running
another
container
on
the
machine.
Only
one
new
container.
You
typically
do
this
through
a
demon
set,
for
example,
in
kubernetes,
by
telling
kubernetes
on
a
on
each
of
my
hosts.
I
want
a
falco
container
only
one
if
it
goes
down
bring
it
back
again.
A
You
click
enter
cube
cattle
and
essentially
pushed
falco
everywhere
when
falco
starts
on
the
machine,
it
detects
the
version
of
the
operating
system
and
it
deploys
instrumentation
at
the
edge
between
the
containers
and
the
operating
system
itself.
This
is
important
because
falco
is
not
based.
The
the
the
traditional
instrumentation
for
falco
is
not
based
on
cycles.
A
Once
this
is
deployed
in
the
corner
of
the
operating
system.
Typically
under
the
form
of
any
bpf
module,
falco
is
able
to
collect
every
single
system
call.
So,
let's
first
talk
a
little
bit
about
ebpf.
As
I
was
saying
before,
bpf
stands
for
enhanced
packet
filter
and
it's
essentially
a
virtual
machine
that
runs
in
the
kernel
of
the
operating
system
and
allows
you
to
inject
scripts
in
the
operating
system
to
do
a
bunch
of
different
things.
A
So
it's
a
beautiful
way
to
essentially
extend
and
include
functionality
in
the
kernel
that
allows
a
tool
like
falco
to
operate.
At
that
point,
the
eppf
scripts
allow
falco
to
capture
system
calls,
as
I
was
saying
system
calls,
are
anything
you
know,
reading
or
writing
to
disk
opening
files,
establishing
network
connections
or
receiving
network
connections
and
servers
interface,
communication
executing
commands.
This
is
all
system
calls.
So,
on
a
really
super
granular
way,
falco
is
able
to
collect
all
of
this
information
and
give
you
essentially
notifications.
If
something
goes
wrong.
A
Kernel-Based
instrumentation
is
very
powerful
and
the
reason
why
we
chose
it
is
the
original
way
to
instrument.
Falco
is
because
it's
accurate,
it's
in
the
kernel,
so
you
you
don't
really
bypass
it
and
and
it
really
collects
everything
it's
extremely
efficient,
so
it
slows
down
minimally
the
target
containers
and
the
target
applications
at
the
same
time.
It's
scalable,
because
you
only
need
to
deploy
one
thing
and
then
you
can
have
one
container
running
on
the
machine
or
a
thousand
containers
running
on
the
machine.
This
orientation
point
is
always
one
you
know.
A
While
when
you
have
side
cars,
essentially
the
overhead
of
the
instrumentation
tends
to
be
linear.
With
the
number
of
containers
that
you're
running
the
problem
with
the
kernel-based
instrumentation
is,
there
are
some
places
where
kernel
instrumentation
is
not
available
anymore.
I
mentioned
in
fargate,
for
example,
so
falcon
needs
access
to
the
kernel.
The
collection
system
that
I
described
requires
a
kernel
access,
but
there
are
places
like,
for
example,
fargate
in
aws
or
microsoft.
Hyper-V
containers
where
this
access
is
not
available
anymore,
because
essentially
the
same
machine
could
host
multiple
containers
from
multiple
users.
A
So
it's
not
safe
to
give
some
one
of
these
users
access
to
the
kernel.
This
could
create
issues
for
the
other
users
that
are
that
are
in
a
multi-tenant
environment,
sharing
the
same
machine.
So
how
can
we
address
this?
We
need
to
find
a
solution
that
collects
the
data
with
the
same
granularity,
but
it
does
not
access
the
corner
of
the
operating
system.
A
A
So
if
we
replace
lipsy
in
one
of
your
containers,
then
we're
able
to
install
some
instrumentation
that
can
send
the
data
to
falco
the
preload
is
the
advantage
of
being
efficient,
pretty
easy
to
implement,
but
as
the
huge
disadvantage
that
it
requires
essentially
replacement
of
dynamic
libraries
in
an
application
which
myself
is
a
little
bit
dirty,
but
even
more
importantly,
it's
limited
because
it
doesn't
support
the
whole
classes
of
applications
that
just
don't
use
lip
c
and,
and
they
don't
link
dynamic
libraries,
see
statically
linked
programs.
Have
this
feature
they
they
don't.
They
use
dynamic.
A
A
So
another
approach
is
using
ptrace
ptrace
is
a
functionality
in
the
linux
kernel
that
allows
deploying
essentially
something
inside
the
container
that
can
essentially
detect
every
single
system
call
by
going
and
getting
access
to
the
monitored
application.
Ptrace
is:
what's
you
it's?
What's
at
the
base
of
tools
like
a
gdb
or
or
s-trays,
so
just
by
looking
at
what
gdp
can
do,
you
can
get
a
hint
of
what
p
trace
can
do?
It
can
freeze
a
target
process,
it
can
install
breakpoints,
it
can
inspect
the
memory
of
the
target
process.
A
You
can
also
see
every
single
system
call
and
the
arguments
of
the
system
call
so
by
leveraging
ptrace
it's
possible
to
essentially
create
a
a
tool
which
I
call
pde
call
it
pd
gear
that
can
collect
the
system,
calls
and
send
them
to
funk
benefits
of
phrase.
Are
it's
accurate
because
you
still
go
through
the
kernel,
so
I
used
to
say
the
kernel
never
lies,
because
there's
there's
it's
hard
to
bypass
something
that
is
collected
at
the
kernel
level.
A
A
If
you
think
about
that,
let's
take
a
look,
so
I'm
switching
to
my
aws
console
and,
as
you
can
see
here,
I
have
a
couple
of
tasks
and
I'm
I
have
one
that
is
called
falco
trace,
ssh,
so
let's
go
inside
it
and,
let's
gather
the
public
ip.
This
is
essentially
a
container
that
is
running
ubuntu
and
let
me
ssh
into
it.
A
A
B
A
This
container
has
been
instrumented
with
the
technique
that
they
just
described,
so
we
have
pdig
running
inside
this
container
using
ptrace
to
collect
information.
So
I
can
do
some
simulated,
malicious
activity
here
like
I
can
change
a
system
binary,
slash
being
then
slash
cat
and
now
I'm
going
into
my
console.
Actually
let
me
refresh
and,
as
you
can
see
sure
enough,
I've
been
notified
that
the
system
binary
has
been
changed
inside
this
container,
so
we
get
essentially
granular
run
time
system
call
level
visibility
through
falco
on
fargate,
thanks
to
ptrace-based.
A
One
more
thing
that
I
want
to
talk
about
is
performance,
so,
as
I
was
saying,
the
issue
with
ptrace
is
that
it
tends
to
be
relatively
slow
because
every
time
there's
a
system
called
every
time
the
application
talks
to
the
kernel
of
the
operating
system.
The
system
call
needs
to
go
up
and
go
back
to
user
space
and
go
and
be
delivered
essentially
to
a
separate
process
running
inside
the
the
machine.
This
process
by
the
way
needs
to
be
essentially
loaded
in
the
container.
In
my
demo,
I
loaded
it.
A
It
is
as
the
entry
point
of
of
the
container
so
that
then
everything
inside
the
container
would
be
captured
essentially,
and
by
do
and
doing
that,
creates
a
little
bit
of
an
overhead.
There
are
things
that
can
be
done
to
accelerate
this,
because
otherwise,
without
optimizations
p
trades
would
just
be
as
a
technique
too
slow
to
work
effectively
in
production
on
your
containers.
So
we
do
multiple
things
to
make
this
faster.
A
One
example
is
the
use
of
a
second
bpf
when
ptrace
has
been
enabled,
for
example,
on
fargate
in
the
environment,
where
I
give
the
demo.
Another
thing
that
was
enabled
by
by
amazon
was
a
second
vpf
which
allows
essentially
to
inject
little
simple,
bpf
scripts
that
can
selectively
capture
only
specific
system
calls,
for
example,
falco
needs
to
know
when
a
file
is
open
or
when
a
connection
is
established,
but
falco
doesn't
currently
look
at
the
payloads
for
performance
reasons
so
through.
Second,
we
can
selectively
capture
and
do
this
path.
A
There
are
also
other
optimizations
that
we're
doing,
but
those
go
a
little
bit
deeper
and
might
be
the
subject
of
a
future
kubecon
talk
for
the
moment,
I'm
just
giving
you
a
few
links
if
you
want
to
play
with
this,
in
particular,
everything
about
falco
is
on
falco.org
and
anything
that
I
showed
on.
This
demo
is
already
available
for
the
community
under
the
falco
organization
and
can
be
essentially
downloaded
from
github.
This
is
where
you
download
and
and
install
falco
from
github-
and
this
is
where
you
find
pdig
on
github.
A
B
A
So
if
you
go
either
to
the
falco
website
or
to
the
falco
blog,
which
you
find
from
the
website
as
well,
you
will
find
several
reference
architectures
to,
for
example,
integrate
falco
with
elastic
for
for
message,
collection
or
with
the
event
pipelines
on
aws
or
lambda
or
or
stuff
like
that.
So
yes,
there
are
many
another
one,
for
example,
that
the
detect
mention
is
prometheus.
We
have
exporters
for
falco
that
can
convert
far
colors
into
prometheus.
A
A
In
order
to
have
yes,
this
kind
of
functionality
embedded
into
just
kubernetes
when
you,
when,
when
you
get
it
for
sure
kubernetes
itself,
is
making
a
lot
of
progress
in
terms
in
terms
of
just
getting.
You
know
better
included
from
the
security
point
of
view,
but
there's
definitely
space.
I
see
you
know,
especially
the
runtime
security
that
falco
does
to
be.
A
A
One
thing
that,
for
example,
I
can
mention
that
we
have
announced
recently
is
the
integration
with
file
between
falco
and
openshift
through
the
reddit
acm
2.0.
So
red
hat
now
is
an
operator
that
makes
it
possible
to
essentially
win
the
thing
with
a
single
click
to
just
have
falco
deployed
in
your
cluster
and
configured
essentially
by
the
flow
by
default.