►
From YouTube: Panel - Klustered: eBPF Edition - David Flanagan, Duffie Cooley, Loris Degioanni, & Marga Manterola
Description
Panel - Klustered: eBPF Edition - Moderated by David Flanagan, Pulumi; Duffie Cooley, Isovalent, Loris Degioanni, Sysdig, & Marga Manterola, Microsoft
In this special edition Klustered, David sends our worthy opponents from Isovalent, Microsoft, and Sysdig through a series of eBPF challenges. Wielding only their tools of choice, Cilium, Inspektor Gadget, and Sysdig/Falco, the teams must unravel the 3 breaks across their bare metal Kubernetes clusters to get their applications working.
During this panel we’ll show clips of each episode while discussing how eBPF changes the game for debugging, networking, and security.
We welcome your questions on eBPF and the tools above throughout the session, so come have a giggle and enjoy the show.
A
So
our
kind
of
almost
final
session
of
the
day
is
going
to
be
a
really
rather
interesting
panel.
Now
how
many
of
you
here
have
watched
raw
code
and
in
particular
clustered
before
we
got
some
people
who
are
familiar
with
clustered?
So
it's
a
really
great,
like
youtube,
show
where
people
break
kubernetes
clusters
and
people
try
to
figure
out
how
those
clusters
were
broken
and
fix
them.
A
A
A
B
Yeah,
my
name
is
royce
dejani,
I'm
the
cto
and
founder
of
sysdig
sysdig
is
a
company
that
does
security
and
visibility
for
modern
cloud-based,
containerized
applications,
I'm
also
one
of
the
early
contributors
to
wireshark
the
network
analyzer
and
one
of
the
also
creators
of
falco
the
open
source
tool.
So
a
career
in
open
source
and
kernel
hacking.
C
E
Hello,
hello,
my
name
is
david
flanagan,
I
go
by
raw
code
and
I'm
excited
to
have
this
panel,
so
we
have
slides.
Okay,
we're
good.
So
let
me
fill
you
in
a
little
bit
for
the
people
that
are
not
familiar
with.
What
clustered
is
I
often
call
it
the
worst
idea
I
ever
had
having
people
break
kubernetes
clusters
and
then
live
streaming.
Fixing
them
is
extremely
stressful,
but
also
a
very
valuable
learning
experience
and
we
have
two
different
versions
of
clustered.
E
One
is
solo
edition,
where
it's
just
one
community,
member
and
another
community
member,
and
we
also
experiment
with
teams,
editions
and
I'm
always
looking
for
new
victims,
so
feel
free
to
reach
out
to
me.
I
also
want
to
thank
equinix
merrell.
They
provide
the
hardware
for
the
three
clusters
that
we
are
going
to
want
to
get
fixed
today,
so
we've
got
teams.
We've
already
did
on
the
introduction
so
we'll
skip
by
this,
and
I
want
to
just
say
before
we
show
the
videos
that
this
is
not
easy.
E
So
I
want
to
thank
everyone
involved,
the
breakers
and
the
fixers
like
being
given
a
cluster,
and
you
have
no
idea
what
is
wrong
with
it
is
intimidating
and
stressful,
but
they
all
absolutely
smashed
it
so
well
done
and
all
of
the
tools
that
we're
going
to
see
today
to
debug
and
fix
these
clusters
are
open
source.
So
this
is
stuff
that
you
can
play
with
and
experiment
in
your
own
time
as
well.
C
C
C
Okay,
so
we're
starting
off
by
just
you
know,
taking
a
look
at
what's
happening
inside
the
cluster,
we're
just
trying
to
see
what's
happening,
and
we
can
see
that
initially
we
see
that
there
is
a
posit,
isn't
up
and
running
and
we're
trying
to
figure
out
like
why
that
pod
isn't
running
and
we
find
a
very
important
init
container.
That
is
basically
keeping
it
from
from
running
by
trying
to
make
a
connection
to
google.com
as
a
pre-check
for
the
container.
So
until
that
init
container
happens,
it
won't
start
up.
F
D
C
So
we
were,
you
know
our
the
goal
of
our
of
our
efforts
is
to
make
it
so
that
that
container,
that
we're
referencing
here
is
able
to
come
up
and
that
we're
able
to
see
like
a
success
image
on
the
website
is
being
hosted
by
this
container.
And
so
first
we
try
to
figure
out
like
what
was
happening
with
the
with
google.com,
and
we
eventually
came
to
the
conclusion
that
what
was
happening
was
that
someone
had
actually
put
google.com
into
the
etsy
hosts
file,
pointing
at
like
some
other.
C
You
know
url,
and
so
it
was
never
going
to
actually
resolve
because
it
was
okay.
It
was
against
some.
I
think
it
was
against
localhost
or
something,
and
so
that
was
actually
probably
one
of
the
more
creative
dns
problems
that
I've
seen
in
a
while,
because
basically
it
was
just
intercepting
it
since
coordinates
was
using
the
hosts
file
to
determine
what
the
what
the
result
of
google.com
was.
C
And
at
that
point
we
started
moving
on
to
the
second
set
of
problems
and
the
second
set
of
problems.
Actually,
I
think
we're
still
talking
about
the
dns
one
here,
so
we're
trying
to
figure
out
like
what
the
problem
was,
and
we
had
a
policy
that
would
allow
for
us
to
communicate
to
google.com,
but
for
some
reason
it
wasn't
resolving,
and
so
we
were
trying
to
figure
out
why
it
wasn't
resolving
and
the
reason
was
because
of
the
hosts
file,
which
was
very
clever.
F
A
J
A
J
A
F
J
I
I
G
C
H
I
I
F
C
C
C
That
when
traffic
within
this
namespace
happens,
when
something
tries
to
reach
the
service
postgres
in
the
default
namespace
on
port
5432
and
it's
tcp
traffic,
we're
going
to
redirect
that
traffic
to
a
different
backend
using
a
local,
a
local
endpoint
selector
that
matches
labels
app
fluent
d.
So
basically
in
ebpf,
we
can
be
like
don't
send
it
to
the
postgres
database.
Instead,.
G
E
F
E
C
Yeah
I
mean
like
what's
interesting,
is
like
that
variety
of
products
actually
describes
the
value
in
ebpf
across
a
variety
of
different,
like
verticals.
If
you
will
right
so
with
tetragon
when
we
started
talking
about
tetragon.
One
of
the
amazing
things
that
we're
talking
about
is
the
ability
to
gather
context
about
what's
happening
at
the
application
layer
and
present
it
to
you
in
a
way
that
you
can
use
it
to
understand.
C
What's
actually
happening,
you
know
like
which
given
process
is
making
what
connect
call,
what
system
calls
are
they
making
and
those
sorts
of
things?
And
then,
if
you
look
at
the
way
that
we're
using
psyllium
we're
using
psyllium
to
leverage
ebpf
to
solve
a
lot
of
the
networking
use
cases
like
how
do
we
handle
load
balancing
when
we
replace
cube
proxy
we're
talking
about
using
psyllium
to
be
effectively
like
that
internal
load,
balancer
that
every
kubernetes
cluster
has
right,
and
so
it's
a
variety
of
different,
really
important
use
cases
for
ebpf.
E
Awesome
what
I
really
loved
about
our
session
was
the
developer
experience
which
we
haven't
got
in
that
six
minutes,
but
the
videos
will
be
available,
but
there
was
the
hubble
ui.
There
was
a
hubble
cli.
There
were
grafana
dashboards
that
were
shipped
and
even
the
use
of
the
selium
editor
for
network
policies.
C
Maybe
you've
misdefined
a
network
policy,
maybe
you're,
you
know
typed
the
wrong
url.
You
actually
wanted
to
see
like
some
kind
of
an
xnx
domain
that
you
you
know
that
it
is
actually
dns,
but
like
some
of
the
this
visibility,
I
think
is,
is
pretty
killer.
It's
one
of
the
it's
one
of
the
big
differences.
H
So
yeah
here
we
have
the
cluster.
So
let's
see
the
nodes
that
we
have
before
going
on
so
yeah,
okay,
so
yeah.
Let's
check
the
pots
that
are
running
there
so
yeah.
He
seems
that
the
parts
of
the
control
plane
of
the
the
capsule
name
space
are
all
running
and
good.
So
yeah,
that's
good
for
us
and
yeah.
We
have
this
spot.
That
is
in
crash
the
back
of
the
state.
So
probably
this
is
the
one
that
we
have
to
fix.
Now
we
can
get
the
lock
so
get
the
pot.
The
name
of
the
pot.
H
H
D
H
H
E
H
Oh,
so
we
have
the
top
contest.
Capability
drop
all
yeah
this.
This
is
not
good.
We
are
running
the
pod
without
any
probability,
of
course,
this
is
gonna
fail.
What
we
need
to
understand
right
now
is
what
are
the
capabilities
that
are
required
by
this
spot,
so
yeah?
I
think
we
can
use
the
capabilities
gadget
to
get
a
list
of
of
the
capabilities
that
this
spot
is
trying
to
use.
So
we
have
capacity
and
gadgets
capabilities.
H
B
H
E
G
H
H
B
E
E
H
E
E
All
right,
so
that
was
quite
a
tricky
one.
We
dropped
all
the
capabilities.
We
added
a
custom
site
comp
profile
that
was
missing
one
syscall
and
I
gave
them
a
pod
with
no
idea
what
port
it
was
running
on.
So
all
right,
so
debugging
kubernetes
is
a
huge
challenge
for
all
teams,
but
what
I
thought
was
amazing
there
is.
You
really
didn't
look
at
the
ammo
to
see
if
there
was
a
container
port.
You
just
listed
all
the
ports
that
the
container
was
using
and
I
thought
that
was
a
very,
very
cool.
E
D
Right
yeah,
so
we
also
have
a
second
profile
advisor.
That
is
just
for
that.
So
you
can
run
the
advisor
like
in
capture
mode
first
to
see
all
the
syscalls
that
your
pod
is
making
and
then
just
tell
it
to
generate
the
profile,
and
it
will
generate
a
profile
with
just
the
ciscos
that
your
pod
is
using
and
you
will
have
a
matching
profile.
You
don't
need
to
handcraft
it
because,
like
it
just
generates
a
profile
for
you.
E
Awesome.
Thank
you
very
much.
We'll
just
do
questions
at
the
end
and
we'll
move
on
to
the
the
next
one,
which
is
the
systig
video,
and
I
just
want
to
say,
like
I
worked
with
breakers
from
all
of
these
companies
and
I've
got
the
cystic
breakers
are
really
wanting
to
make
you
sweat
like
they.
They
really
wanted
to
cause
you
some
trouble.
So
exactly.
E
B
B
B
B
E
E
F
E
E
B
K
B
C
E
F
B
E
Interesting,
they
put
a
lot
of
effort
into
that.
So
that
was
a
lot
of
fun
and
we
see
mutating
web
hooks
a
lot
on
cluster.
They
they
are
very
opaque,
they're
difficult
to
track
down.
So
it
was
actually
really
cool
to
see
falco
with
psychic
and
the
ui
actually
give
you
visibility
into
what
was
happening
and
that
which
was
kind
of
cool.
E
So
something
that
I
often
see
is
that
security
is
still
kind
of
considered
like
a
day
two
problem,
but
now
that
we
have
tools
like
falco
coming
in
and
becoming
adopted
and
there's
ability
to
share
rules,
and
do
you
see
a
shift
changing
there
where
people
are
starting
to
maybe
work
on
security?
Earlier.
B
Yeah
for
sure
what
we're
seeing
with
falco
saw
falco
is
to
attempts
to
accomplish
two
goals.
One
is
the
use
of
modern
techniques
like
ebpf
to
capture
data
that
is
as
granular
and
as
real
time
as
possible
from
the
operating
system
and
from
other
sources
as
well
nowadays,
and
the
other
one
is
creating
sort
of
a
simple
policy
language
that
people
can
use
really
in
a
community
way.
B
One
point
of
view,
I
would
say
we're
going
toward
the
direction
from
the
other
point
of
view.
It's
also
constantly
a
challenge,
because,
because
security
is
always
it's
always
environment,
dependent
right,
so
building
something
that
works
out
of
the
box,
for
everybody
is
the
dream
and
definitely
with
every
iteration
of
security
tools,
we're
getting
better
and
better,
but
I
don't
think
we're
there
yet
with
something
that
just
works
out
of
the
box.
You
know
for
everybody
and
it's
secure
by
default.
E
Okay
right,
I'm
going
to
throw
this
out
to
all
of
you
now
and
maybe
we
can
have
a
bit
of
a
conversation.
But
I
think
what's
really
cool
here
is
that
we've
seen
ebpf
applied
to
networking
security
debugging?
You
know
it's
almost
a
superpower
that
anybody
can
learn
and
harness
and
use
so
do
use,
have
any
tips
for
people
here
that
want
to
adopt
ebpf
and
then
let's
try
and
tackle
that
as
consumers
of
ebpf
tools,
but
also
as
being
eppf
developers.
B
I
don't
know
I
can
go
first
in
my
opinion,
so
ebpf
is
a
super
cool
technology,
incredibly
powerful,
and
I
recommend
to
anybody
to
learn
more
about
it.
It's
such
a
powerful
swiss
army
knife,
but
at
the
same
time,
you
don't
need
to
be
like
a
compiler
expert,
a
kernel
expert,
to
take
advantage
of
this.
C
Yeah,
I
completely
agree
with
that
with
that
direction.
I
think
that
you
know.
Even
earlier
today
we
saw
a
talk
about
leveraging,
pixi
and
and
some
of
the
some
of
the
capabilities
of
just
leveraging
bpf
trace
right
to
understand,
what's
actually
happening
on
your
systems,
I
mean
as
a
system,
I'm
an
old
systems
administrator
and
I've
worked
in
networking
for
years
and
years.
C
D
Yeah,
I
agree.
I
yeah
I'm
not
sure
I
have
much
to
add,
but
earlier
today
there
was
a
plug
for
lee's
talk
from
last
year,
so
for
those
that
were
not
there,
that's
like
a
good
introductory
talk
and
I
think
the
bpf
trace
is
also
a
great
tool
to
getting
acquainted
with
like
what
are
the
different
trace
points
that
are
available.