►
From YouTube: Opening the Black Box: Understanding and Troubleshooting Calico’s eBPF Data Plane - Tomas Hruby
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Opening the Black Box: Understanding and Troubleshooting Calico’s eBPF Data Plane - Tomas Hruby, Tigera
Troubleshooting Kubernetes issues is hard enough in a standard Linux environment using well-proven tools. It’s even harder with the novel and ever more popular eBPF data planes. Some of the standard tools like iptables-save or conntrack no longer help, as they were built for the legacy environment based on Linux network stack and iptables. However, eBPF bypasses large parts of the node’s host network namespace for better performance and scalability. In this session, Tomas will show how to inspect and troubleshoot the eBPF mode of Calico Open Source, one of the most widely adopted container networking and security solutions, to help users better understand the eBPF mode. This will give users the skills to dive in and diagnose issues in their Kubernetes deployments. It will also help the community to contribute code to the Calico Open Source project.
B
A
A
Ebpf
data
plane
how
it
differs
and
about
the
tools
about
how
we
can
troubleshoot
policies,
how
to
get
statistics
and
login
out
of
the
data
plane.
This
talk
is
not
meant
to
be
a
page
for
any
observability
features,
but
it's
really
meant
to
reflect
on
the
feedback
from
the
community.
We
got
about
how
we
can
improve
on
on
the
project.
A
Lots
of
users
or
colleague,
users
are
switching
from
iptables
to
ebpf,
but
that
switch
is
not
always
as
smooth
as
the
users
would
hope
for
Hope,
for
because
the
environment
has
changed
dramatically,
so
kalika
is
probably
I.
Don't
have
to
introduce
it
that
much
but
project
Calico
is
a
cni
for
kubernetes
and
also
works
in
other
environments.
So
it's
responsible
for
networking
and
provides
Network
policy
model.
The
model
is
richer
than
what
kubernetes
natively
support,
but
particularly
completely
implements
that
kubernetes
Network
policy
model
as
well.
A
A
Geico
is
essentially
a
multi-platform
project.
It
has
it
supports
multiple
data
planes.
It
naturally
runs
on
iptables.
It
supports
the
evpf
data
plane
and
that
I'm
going
to
talk
about
here
today,
but
also
there
are
other
data
planes
like
the
VPP
contributed
by
Cisco
and
colleague,
also
runs
on
Windows
and
Hood
gas.
We
can
also
leverage
some
ebpf
on
Windows.
A
The
data
plans
differ
depending
on
users
needs
in
the
IP
tables.
You
have
pretty
much
everything
provided
by
the
kernel
and
Calico
just
programs.
The
rules,
while
any
BPF
kernel,
provides
the
environment,
but
the
data
plan
is
actually
provided
by
project
Calico.
The
code
is
injected
in
the
kernel
and
the
code
comes
with
Calico
itself.
A
A
The
cool
thing
is
that
we
can
add
new
features
like
the
fact
that,
if
you,
if
you're
client,
is
accessing
your
clusters
through
node
Port,
that
we
can
preserve
the
client's
IP
all
the
way
to
the
destination
pod,
which
helps
the
users
to
write
more
sensible
and
more
robust
Network
policies,
something
that
wasn't
possible
with
iptables.
It
allows
us
to
implement
DSR,
which
helps
with
performance
and
in
general.
A
Ebpf
provides
us
with
lots
of
flexibility
so
that
we
can
tweak
existing
kubernetes
features
or
when
there
is
a
need
or
users
have
a
need
for
a
specific
feature
that
is
not
supported
by
the
mainstream
kubernetes.
Then
it's
possible
to
extend
it,
but
let's
first
start
with
iptables
and
the
good
old
days
in
those
days.
It
is
a
super,
well
understood,
data
plane,
but
that
data
plane
don't
necessarily
scale
as
well,
and
it's
not
very
extendable.
A
A
So
when
you
dump
the
ipt
rule,
this
kind
of
human
readable,
that's
great,
so
the
service
is
implemented
by
Q
proxy
and
the
policies
implemented
by
Calico
or
any
other
project
for
networking,
a
kind
of
in
plain
size,
so
you
can
dump
it
and
with
some
F4
you
can
read
it
and
understand
it
right.
So
this
is
kind
of
what
people
were
used
to
Linux
does
the
rest
of
the
heavy
lifting
it
does
routing
it,
handles
the
protocols,
edge
cases
and
so
on
and
forth
so
forth.
A
A
The
nodes,
Network
Stacks,
largely
bypassed
the
Q
proxy
that
comes
with
kubernetes
now
useless
because
it
implements
the
services
in
iptables
and
in
the
host
Nazareth
stack
and
now
the
package
don't
hit
it
anymore
and
the
policies
are
implemented
in
ebpf
code
and
that
may
may
refer
to
ebpf
Maps,
but
again,
unlike
in
the
IP
tables
world,
where
you
could
dump
the
rules
and
kind
of
read
it.
That's
not
really
straightforward.
So.
A
A
We
have
users
that
say:
I
don't
see
collections
in
my
contract.
It
is
difficult
to
verify
that
the
services
are
programmed
correctly
right.
All
of
that
was
in
plain
size
in
iptables
world.
Some
users
may
even
have
tools
that
use
the
use,
the
Linux
contract
and
so
on
for
their
third-party
tools,
and
these
environments
only
break.
A
A
A
The
first
map
is
a
set
of
front
ends
that
point
to
the
map
with
a
set
of
backends
right,
and
you
can
use
a
colleague
of
BPF,
not
dumb
command
to
list
this
in
human
readable
form,
so
it'll
print
you,
which
service
IP
is
there,
will
sort
of
protocol
it
uses,
will
sort
of
port
and
how
many
there
are
backends
how
many
local
back-ends
there
are
on
the
indented
lines.
You'll
get
a
list
of
the
backends
which
the
service
is
pointing
to
with
the
service
ID
and
the
backend
ID
only
slide.
A
A
The
next
thing
you
would
like
to
check
is
whether
you're
understanding
of
what
is
in
the
cluster
matches
the
understanding
of
the
data
plane
itself
right.
So
another
thing
we
can
dump
is
routes,
it's
not
quite
a
route
in
like
the
IP
route,
sensor
things,
but
it
describes
what
a
certain
side
you're
within
the
cluster
means.
So
on
the
right
left
hand
side,
you
will
see
the
slider
and,
on
the
right
hand,
side,
you
will
see
the
explanation
in
what
the
cider
means.
A
So
if
you
verify
that
and
if
that
matches
your
expectation,
the
next
thing
is
to
look
at
the
connections
themselves.
The
ebpf
data
plane
has
been
mentioned
before
implements
services
on
its
own,
because
the
Q
proxy
is
kind
of
bypassing
this
world.
If
you
have
needs
to
do
all
the
netting
and
for
that,
it
needs
to
maintain
connection
tracking,
so
it
can
do
the
reverse
translation,
and
it
also
maintains
the
connections
so
that
it
knows
what
was
allowed
into
the
system
and
what
not.
A
In
general,
we
have
a
single
entry
per
connection
that
maintains
all
the
core
properties,
and
in
case
there
is
a
address
translation.
We
have
a
second
entry
that
refers
to
the
one
that
maintains
most
of
the
properties.
This
renders
the
kernel
contract
pretty
much
Irrelevant.
This
is
kind
of
the
one
of
the
most
important
things
that
users
are
being
confused
with
that
there
are
some
of
the
two
contracts.
A
So
when
you
use
the
Calico
BPF
tool
to
dump
the
contract,
you'll
pretty
much
see
a
line
per
entry.
Unfortunately,
here
is
rap
because
it
will
fit
the
slide.
So
in
this
simple
case,
where
there
is
no
netting,
the
entry
just
shows
you
that,
what's
the
connection,
that's
the
key
and
was
the
type
of
the
entry
here.
It
says
that's
a
simple
one
and
it
tells
you
other
properties
of
the
connection
like
here.
It's
a
TCP
connection.
A
It
tells
you
that
it's
been
open
and
it
was
gracefully
closed
and
it
tells
you
also
how
long
ago
these
things
happen.
That's
important
for
garbage
garbage
collecting
this
these
connections
in
the
more
complex
case
on
the
services
in
play,
you
would
also
see
an
extra
entry
which
is
keyed
by
the
predeen
topple.
So
you
see
that
the
part
is
connected
to
service
and
that
entry
carries
the
reverse
key.
That
points
to
the
post,
DNA
translation
and
that
points
to
another
entry
that
maintains
the
information
I
have
described
on
the
previous
slide.
A
You
can
you
can
carry
on
I've
mentioned
a
couple
times
that
Little's
contract
is
not
important
anymore.
It's
irrelevant,
it's
not
quite
entirely.
True,
there
are
scenarios
where
doing
things
outside
of
ebpf
is
just
simpler
than
doing
it
in
ppf.
Typical
scenario
is
the
marsh
parade.
Calico
allows
you
to
tag
IP
pool
for
not
outgoing,
so
if
you're
sending
stuff
outside
of
your
cluster,
then
the
traffic
outside
of
a
cluster
will
have
the
cluster's
IP
or
one
of
the
classes
IP.
A
It
is
possible
to
do
it
in
ebpf,
but
it's
not
entirely
straightforward.
It's
inherently
difficult
to
allocate
a
non-conflict
install
Sports,
so
why
not
to
let
those
packets
go
through
the
IP
tables
and
through
the
host
Network
stack
and
let
the
kernel
does
do
what
it
does
pretty
well,
we
just
instruct
the
connections
to
go
that
way
and
let
the
Linux
track
those
connections.
Nevertheless,
they
connection
is
also
tracked
in
ebpf
connection
tracking.
We
must
do
that
because
we
have
to
tell
what
should
happen
to
those
packets
on
those
on
that
connection.
B
A
We
were
talking
so
far
about
connectivity
in
your
cluster,
but
if
you're
using
project
collica,
you
probably
are
most
interested
in
network
policies.
Let
us
also
change
dramatically
before
Calico
compiled
the
information
from
the
cluster
with
the
policies
and
injected
it
into
the
iptables
kind
of
into
a
single
point.
While
now
that
information
is
split
and
the
policy
or
part
of
the
policy
is
attached
to
individual
endpoints
and
individual
devices
within
the
system,
so
it's
kind
of
broken
up.
A
These
policies
are
implemented
in
in
the
program
and
that
program
May
point
to
ebpf
maps,
but
from
the
perspective
of
user
it
is
essentially
a
binary
blob.
You
can
dump
the
policy,
you
can
disassemble
it
and
if
you
have
intimate
knowledge
of
the
data
structures,
you
may
actually
get
something
out
of
it
and
that's
the
complaint
that
we
got
not
because
the
users
would
need
to
like
debug
that
it's
more
like
in
the
zero
trust,
poly
security
world.
The
the
users
want
to
only
trust
what
they
see
so
in
recent
release.
A
What
rule
of
the
policy
is
specifically
implemented
and
some
more
human
readable
form
of
how
the
rules
actually
implemented
and
for
those
really
interested?
There
is
also
a
dump
of
the
assembly
code
more
in
the
human
readable
form,
with
some
extra
annotations
that
allow
you
to
also
inspect
which
state
and
how
that
state
is
actually
being.
A
A
A
So
we
put
it
in
what
we
call
IPC,
we
kind
of
borrowed
the
name
from
the
IP
tables
world,
because
it
is
essentially
a
set
of
ips
and
the
policy
Dom
shows
you,
which
selector
is
implemented
by
by
what
IP
said
and
the
Calico
BPF
allows
you
to
go:
dump
the
IP
set
and
verify
whether
the
IP
set
matches
your
expectations
the
net.
The
next
thing
we've
added
recently
is
a
set
of
counters.
A
These
set
of
counters
are
per
interface
and
allow
you
to
see
the
obvious
things
how
many
packets
went
through
the
interface?
How
many
were
accepted
by
a
policy?
How
many
were
dropped
by
the
policy,
but
also
allows
you
to
see
other
things.
Packets
made
make
a
draw
because
of
Errors
because
of
unexpected
events.
These
counters
should
be
zero.
So
if
something
doesn't
work
in
your
system-
and
you
see
that
these
counters
are
not
zero,
that's
kind
of
a
Smoking
Gun
and
you
can
go
and
investigate
one
of
the
most
important
counters.
A
A
A
So
essentially,
we
use
kind
of
like
a
pcap
TCP,
dump
style,
fillers
and
expressions.
So
people
should
be
familiar
with
that.
Allow
us
to
specify
on
what
interface
at
waterhook
we
would
like
certain
packets
be
logged
and
rest.
Not
so
certain
packets
may
take
the
slow
pass
while
the
rest
is
taking
the
fast
bars.
A
So
you
should
have
a
minimal
impact
and
it
should
be
possible
to
use
that
in
production
environments,
not
just
some
staging
environments,
we
kind
of
Leverage
here
the
power
of
the
pcap
which
can
emit
the
which
can
compile
the
CCB
Dam
Expressions
into
BPF
code.
Unfortunately,
that's
the
ancient
classic
ppf,
which
has
some
similarities
with
the
ebpf,
but
has
to
be
translated
into
the
ebpf
first,
that's
actually,
what
happens
if
you
use
TCP
dump
and
the
code
is
injected
into
kernel
kernel
also
translates
it
into
ebpf.
A
First,
those
cylinders
are
prepended
only
if
we
are
in
the
debug
mode,
and
both
paths
has
to
execute
the
same
policy
so
that
when
we
debug
in
in
production,
the
system
is
not
compromised
and
believe
me,
it
is
much
easier
to
do
it
this
way
than
when
we
use
the
J
trace
or
similar
tools
in
the
hfip
tables,
because,
as
I
said
before,
that
we
had
a
fire
hose,
we
really
want
to
have
a
fine-grained
login
here.
So
we've
added
an
option:
every
can
specify
a
list
of
devices
and
expressions.
A
We
also
have
some
special
values
like
you
can
you
can
run
the
filter
for
all
the
devices
in
your
system.
You
can
run
it
only
for
hosts
and
points
or
vocal,
endpoints
and
Calico
always
supported
the
configuration
which
was
either
Global
or
local
to
a
node.
So
you
can
really
specifically
select
what
you
are
interested
within
your
system.
A
So
let
me
walk
you
through
some
of
the
examples
of
the
login
here
on
the
first
line.
On
the
left
hand
side,
you
see
that
each
line
is
prepended
by
the
name
of
the
device
where
the
log
comes
from
and
also
indicates
in
this
case,
with
the
I
as
Ingress
who,
by
use
also
CE
for
Libras
and
x,
for
xzp
they'll.
Surprisingly,
we
first
have
to
parse
the
package,
so
you
get
some
information
about
what
sort
of
packet
is
flowing
through
the
system
we
have
to
do
contract
lookup.
A
In
this
case
we
have
a
Miss.
Therefore
this
is
the
first
packet
of
a
connection,
and
in
that
case
we
have
to
see
whether
it
matches
a
certain
service.
In
this
case
it
doesn't
match
any
service.
We
go
to
that
mess
and
therefore
there
is
a
plane
connection
and
we
have
to
pass
it
through
a
policy.
We
have
a
host
endpoint
without
the
policy,
so
we
accept
the
the
packet
we
have
to
create
a
contract
entry
and
the
last
thing
we
have
to
decide
where
the
packet
goes.
A
A
As
translation,
you
would
see
lines
like
these
in
the
first
batch
with
trying
to
match
it
against
the
service
and,
unlike
in
the
example
before
we
have
a
hit
in
the
first
table,
we
got
a
service
with
a
certain
ID
and
we
picked
a
ordinal
for
the
for
the
back
end.
Then
we
go
and
read
the
back
end
and
Report,
which
backend
we
selected
later
on.
We
have
to
create,
let
me
create
a
contract.
A
We
also
have
to
create
a
second
entry
which
I've
mentioned
before
and
as
we
flow
through
the
rest
of
the
program,
we
also
have
to
actually
execute
the
DNA,
so
we
have
to
patch
the
destination
IP
and
the
destination
Port
into
the
packet
update,
checksums
and
all
this
all
this
stuff.
And
then
we
carry
on
with
what
you
see
on
the
previous
slide.
A
So
with
that
I'd
like
to
conclude
that
ebpf
is
a
great
tool,
it's
super
flexible
and
powerful
and
I
love
working
with
that
and
kudos
to
everyone
who
contributed
to
the
ebpf
system
in
Linux,
but
it
also
disturbs
how
users
were
used
to
work
with
the
systems.
It
disturbs
how
users
used
to
think
about
the
flow
of
the
package
through
the
system,
and
we
need
to
have
a
good
understanding
of
that
good
tooling,
and
we
have
to
actually
do
quite
a
bit
of
teaching.
So
is
that
just
UC
BPF
enjoy
it?
D
D
A
E
When
you
say
that
it
disturbs
the
flow
of
packets
as
we
knew
it,
what's
the
meaning
of
that,
it's
just
because
we
are
just
hooking
into
the
functions
in
the
kernel
or
for
something
else.
Could
you
repeat
the
beginning
of
the
question
when
you
say
that
it
disturbs
the
flow
of
packet?
What
does
it
mean.
A
Yeah
great
question:
it
really
means
that
the
packet
may
go
from
a
device
to
device
and
doesn't
follow
the
rest
of
the
past
that
it
otherwise
would
take
through
the
kernel.
So
if
you
hook
into
TC
and
then
it's
forward
to
another
another
device
that
just
keeps
Lots
in
the
host
network
network
stack
just
takes
a
different
path.
D
A
D
A
Yeah
I
think
that's
where
the
feedback
from
the
from
the
community
came
in
the
first
place,
so
we
had
users,
it
was
like:
hey,
I,
have
a
I,
have
a
contract
which
is
kind
of
half
open
or
I,
have
a
connection
and
I
I,
don't
see
it
in
contract.
So
that's
where
the
confusion
came
from
and
that's
what
we
are
trying
to
address
here.