►
From YouTube: CNL: Troubleshooting eBPF Data Plane in K8s cluster
Description
Cloud Native Live: Troubleshooting eBPF Data Plane in Kubernetes cluster for networking and security
eBPF for networking in a Kubernetes cluster brings some advantages such as performance, lower resource utilization, shorter routing path, etc. However, due to the compiler nature of an eBPF program, it is not a trivial task to troubleshoot an issue. In this session, we will discuss the open-source tools that are embedded in Project Calico that you can use to gain insight into the inner working of the kernel and associated eBPF programs for networking and security.
https://github.com/frozenprocess/Tigera-Presentations/tree/master/2023-08-21-CNCF-Troubleshooting-eBPF-data-plane-cluster-for-networking-and-security
A
Hello,
everyone
Welcome
to
Cloud
native,
live
where
we
dive
into
the
code
behind
Cloud
native
I'm,
Andy,
talvasto
and
I'm,
a
cncf
Ambassador
as
well
as
I,
lead
marketing
at
vision
and
I
will
be
your
host
Tonight.
So
every
week
we
bring
a
new
set
of
presenters
to
Showcase
how
to
work
with
Cloud
native
Technologies
and
as
always,
they
will
build
things.
They
will
break
things
and
they
will
answer
all
of
your
questions.
B
B
A
A
B
B
B
We
can
dive
into
the
stuff,
so
hello,
everyone,
my
name
is
Reza
I'm,
a
developer
Advocate
at
tigera
Tiger
is
the
company
behind
open
source
project
Calico,
where
we
do
everything
in
terms
of
security,
networking
and
the
cloud
native
scene,
I'm
always
eager
to
learn
new
stuff
and
open
to
suggestions.
So
if
you
got
an
idea
or
you
want
to
explore
anything,
let's
connect
up,
you
can
usually
find
me
in
Calico
users
like
I,
believe
if
you're
watching
this
from
LinkedIn,
you
should
be
able
to
click
on
my
profile
as
well.
B
I'm,
pretty
easy
to
find
so
for
this
presentation,
I'm
gonna
talk
about
a
little
I'm
gonna
talk
a
little
bit
about
Cloud
networking
and
then
give
you
a
brief
overview
of
Calico
abpf
advantages.
Then
talk
about
troubleshooting,
because
without
talking
about
these
fundamentals,
it
would
be
very
difficult
to
explain
why
we
need
to
build
new
tools
to
troubleshoot
evpf
anyways
yeah.
A
B
So
evpf
stands
for
extended
vertically
packet
filtering.
However,
at
this
point
it's
not
really
about
packet
filtering,
it's
the
sort
of
a
virtual
machine
that
runs
inside
a
kernel,
I
guess
in
a
couple
of
months
or
years
in
Windows
as
well,
that
can
directly
execute
commands
or
interact
with
the
kernel.
B
It's
offers
a
lot
of
performance
and
you
can
basically
do
a
lot
of
the
stuff
that
are
very
expensive
in
terms
of
resources
with
it
that
that
are
very
expensive
if
you
want
to
do
it
in
other
ways,
so
as
evpf
so
project
Calico,
if
you
like
to
connect
with
us,
these
are
our
social
handles
project.
Calico
is
all
about
Cloud
networking
and
security.
We
got
a
lot
of
like
channel
members
and
a
lot
of
contributors.
B
So
let's
talk
about
Cloud
networking.
First
now,
modern
applications
are
mostly
hosted
on
the
cloud.
These
things,
oh
by
the
way
I
forgot
to
mention,
if
you
got
any
questions,
please
ask
any,
would
help
with
the
questions
and
read
them
for
me
and
train
decisions.
There
is
also
a
q
a
at
the
end
so
depending
on
how
the
stream
goes
and
how
the
demo
is
working
out,
we
might
have
time
to
answer
your
questions
in
the
beginning
or
in
the
middle
of
the
stream
or
at
the
end.
B
If
I
don't
know
the
answer
which,
by
my
experience,
is
most
of
the
time
I'll
try
to
find
one
of
Calico
developers
and
Engineers,
who
know
a
lot
more
about
these
sort
of
stuff.
To
answer
your
question,
that
being
said,
the
best
way
to
get
answers
about
kubernetes
and
everything
Cloud
native,
is
that
project
slack
Channel.
B
There
are
a
huge
number
of
community
members
and
Engineers
who
are
always
present
in
these
project
slide
channels
that
are
that
be
very
happy
to
answer
all
your
questions
with
that
being
said,
modern
applications
are
mostly
hosted
on
the
cloud
these
days.
Cloud
computing
allows
companies
of
all
size
to
offer
their
services
without
investing
in.
B
And
data
centers
on
top
of
that
cloud,
computing
offers
unparalleled
flexibility
and
scalability
that
can
quickly
be
turned
attuned
to
your
needs.
Oh,
but
this
magical
experience
is
possible
by
utilizing
the
power
of
cloud.
Networking
kubernetes
is
one
of
the
main
contenders
in
running
a
cloud
environment
on
its
own.
Kubernetes
is
a
modular
architecture,
modular
orchestrator
that
gives
you
the
required
setup
to
run
and
scale
your
workloads
in
this
presentation.
B
B
A
B
It
it's
there.
The
other
part
of
cloud
networking
is
the
cluster
internal
networking
in
a
kubernetes
cluster.
This
is
managed
by
your
clusters.
Cni
plug
this
cni.
The
cni
uses
a
software-defined
networking
or
sdn
to
provide
connectivity
between
your
workloads
and
of
either
internal
or
external
resources.
Well,
there
are
some
standard
features
that
most
cnis
provide.
Each
one
has
a
unique
take
on
networking.
A
B
B
A
B
Evpf
data
plane
offers
capabilities
such
as
complete
proxy
replacement
source,
IP
preservation
and
DSR,
and
couple
others.
If
you're
running
a
hybrid
environment
with
your
kubernetes,
you
can
use
Calico
for
Windows,
which
is
based
on
Microsoft,
hns
or
host
network
service
and
can
deliver
networking
security
to
your
windows.
Nodes
VPP
is
the
newest
data
plane
for
Calico.
It
was
an
external
contribution
from
Cisco,
currently
in
its
better
phase,
which
accelerates
the
networking
experience
by
utilizing
the
power
of
the
user
space
programming.
Now,
why
I'm
talking
about
these
in
an
evpf
screen?
B
Well,
this
is
the
catch.
So
evpf,
as
I
mentioned,
is
a
technology
that
runs
inside
the
kernel.
So
everything
that
you
try
to
establish
is
directly
injected
into
the
kernel
at
the
lowest
level.
Vpp
is
at
the
user
Space
level,
which
is
at
the
highest
level
possible
for
a
program
to
run
now
you
might
think
well
well,
the
kernel
is
faster
or
you
might
think
the
user
space
is
faster.
Well,
the
thing
is,
depending
on
your
scenario:
each
one
can
have
different
performance
and
what
is
important
is
to
check
everything
out.
B
Do
your
due
diligence
and
figure
out
which
one
works
for
your
scenario,
in
fact,
ebpf
and
SNS
are
some
of
the
technologies
that
we
use
in
our
commercial
offerings
too.
So
keep
short
we're
going
to
explore
on
the
standard
data
point
first
now
the
standard
data
plane
is
built
on
the
Linux
networking
stack,
it
utilizes
the
power
of
iptables
to
communicate
with
the
kernel
and
pushes
Network
boundaries
as
much
as
possible.
B
Their
customers,
evpf
data
plane,
is
another
data
plane,
as
I
mentioned,
utilizes
the
power
of
evpf
programs
to
directly
interact
with
the
Linux
kernel,
implementing
Network
for
your
cluster
now
standard
Linux
data
plane
as
a
height
throughput,
and
it
was
designed
for
10
gigabit
links.
Policy
enforcement
is
very
fast
and
kubernetes
services
are
fast
in
it.
B
A
B
Anyone
can
change
its
Behavior
to
some
extent
now
we're
getting
to
the
parts
that
will
be
included
in
the
troubleshooting,
so
standard
data
plane
is
really
easy
to
troubleshoot.
In
fact,
anyone
can
face
a
problem
and
search
it
in
a
stack,
Overflow
or
any
other
forums,
to
figure
out
how
it
can
be
saved,
how
it
can
be
solved.
B
But
with
ebpf
it's
a
little
bit
more
difficult,
because
again,
everything
is
a
binary
or
as
a
program
that
runs
inside
the
kernel,
so
evpf
data
plane
is
designed
for
40
gigabit
links
or
more.
It
has
a
higher
throughput,
while
consumption
or
its
resource
consumption
is
less
than
standard
standard
data
plane.
Keep
in
mind
that
this
is
not
something
that
all
evpf
data
points
might
share.
Everything
is
different
again,
the
broad
the
thing
is
ebpf
on
its
own.
B
B
It
offers
a
faster
policy
enforcement
since
evpf
policies
are
compiled
as
EV
fit
programs
and
are
injected
directly
into
the
Linux
kernel.
Kubernetes
services
are
faster
since
evpf
data
plane
implements
them
by
ebpf
programs
inside
the
Linux
kernel.
So
you
don't
have
to
go
through
netslink
and
I
to
iptable
and
with
Calico
ebpf
data
plane.
You
have
more
flexibility
to
control
or
manipulate
traffic.
Since
you're
directly
interacting
with
the
kernel,
so
I
talked
about
the
difficulty
of
troubleshooting
now
troubleshooting
evpf
programs,
as
I
said,
can.
A
B
Since
evpf
is
a
fresh
idea,
it
lacks
a
of
the
amount
of
resources
or
the
questions
that
were
asked
for
other
Technologies,
like
the
iptables.
You
might
find
yourself
a
little
bit
discouraged,
but
don't
be
because
each
project
that
uses
ebpf,
usually
ships,
troubleshooting
tools
that
can
peek
into
their
evpf
programs
to
give
you
some
sort
of
output
or
some
sort
of
information
or
insight.
B
The
thing
that
is
important
is
to
know
that
all
these
evpf
programs
that
are
injected
into
the
kernel
are
binary
files.
So,
if
you
are
trying
to
use
evpf
make
sure
that
you
first
go
to
the
troubleshooting
page
of
the
project
that
you
are
choosing
and
figure
out
what
utility
can
be
used
today,
I'm
gonna
show
you
how
Calico
BPF
troubleshooting
works.
B
B
A
B
B
This
plot,
or
this
fluid
of
requests,
aims
to
exhaust
the
victim's
capacity
to
respond,
rendering
their
services
inaccessible
to
legitimate
clients
to
legitimate
clients.
Now
that
we
know
how
the
attack
Works,
let's
explore
the
defense
measures
that
we
can
employ
with
both
the
standard,
Linux
data
plane
and
the
ebpf
data.
B
B
In
such
a
scenario
where
an
attacker
is
targeting
your
environment
with
a
Dos
attack,
the
earliest
point
that
you
can
deny
these
you
know
these
volume
of
requests
is
the
iptables
wrong.
Now
it
is
important
to
note
that,
with
this
approach,
depending
on
the
size
of
the
attack,
there
can
be
a
noticeable
impact
on
your
note.
This
is
because
these
packets
must
be
must
be
processed
by
the
Linux
networking
stack
before
they
can
be
identified
and
eventually
dropped.
B
So,
let's,
when
illegitimate
user
tries
to
contact
the
ebpf
data,
plane
will
take
the
request
and
send
it
to
the
process,
that's
running,
so
this
is
something
that
is
ideal
now.
What
happens
when
the
Dos
attack
happened
when
a
Dos
attack
starts
well,
XDP
integration
allows
you
to
interact
with
these
packets
at
several
points
in
your
environment.
B
In
some
cases,
Calico
can
directly
load
its
ebpf
programs
into
your
network
card
and
drop
malicious
packets
before
they
can
even
hit
the
kernel.
Keep
in
mind
that
these
this
requires
a
network
card
that
supports
XDP
integration
and
not
all
network
cards
are
capable
of
doing
this.
This
method
is
used
referred
to
as
loaded
XDP
in
cases
where
the
offloaded
XDP
is
not
supported.
The
network
driver
card
might
provide
the
capability
to
load
the
xtp
programs
as
part
of
the
initial
receive
path
of
the
traffic.
B
This
method
is
referred
to
as
native
XDP
and
again
not
every
network
cards
are
capable
of
supporting
it.
If
both
cases
are
not
available
in
your
environment,
Calico
will
automatically
use
the
kernel
to
run
its
xtp
programs,
which
is
a
great
way
to
run
XDP
programs
on
generic
Hardware.
That
does
not
provide
specific
support
for
XMP.
B
A
A
A
A
B
B
B
A
B
Oh,
so
multi-pass
is
an
application
that
allows
you
to
run
virtual
machines
in
like
seconds
it's
something
that
I
usually
use
to
spin
up
Laboratories
to
run
my
experiments,
it's
as
you
can
see.
It
allows
you
to
use
data
I,
think
it
was
Data
user
or
something
which
is
Linux
startup
command
for
an
attend,
Linux
startup
script
for
unattended
install.
B
B
To
install
Calico,
the
recommended
way
is
Tiger
operator
now,
if
this
cluster,
that
you
have
access
to
throughout
the
URL
advice,
that
Annie
was
kind
enough
to
share,
is
configured
without
Coop
proxy,
so
usually
cool
proxy
is
what
configures
iptable
in
your
kubernetes
environment
that
allows
the
resources
that
you
have
inside
your
cluster
to
be
able
to
talk
to
API
server
other
resources,
Outside
Inside,
so
everything
happens
within
with
cool
products.
Now,
if
we
take
a
look
at
the
operator
here,.
B
At
some
point
it
would
tell
us
something:
is
wrong:
I
think
it's
going
to
be
like
a
minute
or
a
sec
30
seconds
so
here,
as
you
can
see,
the
operator
is
trying
to
communicate
with
this
IP
address.
We
don't
know
what
is
at
the
moment
and
it
can't
so
if
I
come
to
my
services,
it's
like.
Aha,
it's
the
API
server.
Now,
as
I
mentioned
in
ebpf
mode
Calico
data
plane
can
fully
replace
coup
products.
B
Now
how
you
can
do
it
is
you
need
to
create
a
config
map
with
the
four
new
tiger
operator
to
let
it
know
where
the
kubernetes
host
actually
or
lies?
So,
if
you
look
here,
I
have
a
environment
variable.
This
environment
variable
is
giving
me
back
the
IP
address
of
my
control
plane
and
that's
about
it.
So
now.
B
Knows
what's
going
on
now,
the
operator
would
restart
itself
and
check
for
the
installation,
so
for
troubleshooting,
most
of
the
time,
if
nothing
is
coming
up,
just
go
ahead
figure
out.
What
is
the
log
is
saying
it
could
be
tiger
operator
or
any
sort
of
Operator
just
skim.
The
logs
describe
the
Pod.
It
will
give
you
a
hint
of
where
to
look
at
now.
First
problem
was
easy
to
the
so
now
that
now
that
we
have
the
kubernetes
API
server
configured.
B
I
can
go
ahead
and
check
the
logs
again
now
this
time
as
you
can
see,
after
the
find
it
after
the
operator
finds
the
OS
finds
the
stuff
it
goes
into.
Communicating
with
the
API
server
gets
some
information
that
it
requires
and
immediately
stops.
It
stops
for
the
installation
confident,
so
the
operator
would
continuously
look
for
your
installation
resource
now.
What
is
an
installation
resource?
It
is
the
instruction
that
tells
the
operator
in
which
way
you
want
to
install
Calico
and
how
you
want
to
configure
it
up
for
this
demo.
B
A
B
B
A
B
B
B
A
B
As
variables
to
an
extent,
you
can
change
this
configuration
and
it
tells
you
a
lot
of
information.
You
can
prep
for
error
and
figure
out
what
is
not
working.
There
is
an
explicit
thing
that
says
the
data
plane.
Something
is
not
working,
but
how
you
can
figure
out
if
BPF
is
running
well,
you
again
check
the
logs
for
Calico
node
and
it
tells
you
BPF
modes
enabled
you're
all
set.
B
Oh
one
thing
to
mention
here:
Calico
uses
unprivileged
ppf,
so
it
there
was
an
exploit
some
moons
ago
where
people
could
be
still
possible,
but
it
depends
on
how
you
configure
your
BPF
environment.
For
instance,
Calico
only
uses
unprivileged,
BPF
programs.
So
no
matter
what
happens,
nobody
can
use
BPF
programs
in
order
to
exploit
your
environment.
B
B
B
While
everybody
might
use
a
a
same
hook,
they
will
they
can
use
it
in
different
ways
or
they
can
utilize
it
in
different
ways.
So
it
will
be
pretty
difficult
to
write
a
generic
tool
that
can
give
you
unique
insights.
It
is
possible
to
write
something
that
gives
you
a
general
performance
or
general
error.
Sort
of
thing
like
you
can
use
TC.
For
instance,
TC
will
allow
you
to
run
your
generic
BPF
program
and
get
some
what's
it
called
some
art
books
about
what
is
happening.
B
Tc
also
gives
you
a
way
to
like
hook
into
interfaces,
but
we'll
come
back
to
that.
So,
let's
say
I,
don't
know
what
BPF
is.
I
have
no
idea
how
to
program.
I
enabled
the
ebpf
data
plane
because
of
its
performance
and
I
want
to
troubleshoot
it
now.
What
is
the
solution
for
me?
Well,
one
thing
that
you
can
do
is
shipped
with
your
Cal
code:
node.
There
is
an
application
called
Calico
note
yeah,
not
surprisingly,
but
it
has
an
option.
B
Dash
BPF
now
dash
BPF
has
pre-prepared
BPF
programs
that
you
can
use
to
figure
out
what
is
going
inside
your
cluster.
So,
for
instance,
I
can
use
the
Contours
to
figure
out
what
is
the
Ingress
egress
or
XDP
metrics.
If
I
have
any
xdps,
if
you
launch
the
we
environment
that
I
shared
with
a
QR
code,
the
xtp1,
you
can
actually
use
these
commands
to
figure
out
what
is
going
on,
how
your
how
those
policies
are
being
used,
and
you
can
use
this
information
to
your
advantage,
either
Performance
Tuning
or
troubleshooting.
B
B
B
B
B
B
Dump
that
you
can
do
so,
these
will
give
you
what
is
happening
in
terms
of
your
pods
accessing
the
internet
by
the
way,
if
these
outputs
are
not
familiar
are
not
familiar,
don't
worry,
I'll
share
a
link
to
the
documentation
which
these
are
explained
there
way
better
than
how
I
can
explain
it.
It's
explained
by
the
clever
Engineers
who
build
these
tools
I'm
just
trying
to
talk
about
it.
So
there
are
routes
that
you
can
figure
out.
A
B
Interesting
for
you,
guys
is
pulsing.
Now,
if
you
do
a
policy
dump
you'll
get
an
error.
This
is
because,
when
you
want
to
do
a
policy
dump,
you
need
to
tell
what
interface
and
what
hook
now
the
hook
is
either
Ingress
for
the
incoming
packets
or
egress
for
the
outgoing
packets
or
XDP,
the
one
that
I
mentioned
for
dos
attacks.
Now
you
can
use
all
but
I
would
say
an
all
is
usually
used
when
you
want
to
just
get
a
feeling
of
what
is
going
on
anyway.
A
B
B
A
A
B
A
B
B
B
B
B
If
you're
talking
about
calico
evpf
data
plane,
that
is
also
open
source
too,
you
can
go
change,
it
run
it
in
your
own
environment,
hopefully
submit
the
changes
for
us
too.
So
we
can
use
your
awesome
contributions,
and
can
you
use
it
for
testing?
Yes,
you
can
also
use
it
for
production
too.
It
depends
on
again
you
and
your
environment.
A
B
B
A
B
A
B
B
A
B
Available
so
Calico
iterative
data
point
relies
in
Calico,
node
and
Calico
node
is
highly
available.
So
if
something
happens,
it
will
not
drop
your
connections.
Everything
goes
smoothly,
but
it
like,
what's
it
called
try
to
recuperate
on
the
background
and
comes
back
so
again,
BPF
networking
happens
with
BPF
programs
that
are
injected
into
your
kernel.
That's
about
it!
If
the
BPF
data
plane
crashes,
the
programs
are
in
there
and
doing
their
work.
B
A
So
many
great
questions
which
is
always
lovely
to
see,
so,
let's
wrap
it
up.
So
thank
you.
Everyone
for
joining
the
latest
episode
of
cloud
native
live.
It
was
great
to
have
a
session
about
troubleshooting
EPP
every
data
plane
in
kubernetes,
cluster
and
I
really
love
the
audience
interaction
today
and
the
questions
really
quick
stuff.
Everyone.
Thank
you
so
much
and
as
always,
we
bring
you
bring
you
the
latest
Cloud
native
code
every
week,
so
tune
in
in
the
coming
weeks.