►
From YouTube: eBPF Data Plane Deep Dive - Chris Tomkins, Tigera
Description
Are you always curious? Then let’s take the lid off a cluster running the Calico eBPF data plane and see what’s going on in there.
You will learn:
* The theory of a packet walk through a cluster running the Calico eBPF data plane
* How to see the real thing on a cluster running Calico eBPF
* How to use available tools for diagnostics or to gain visibility of Calico’s eBPF data plane
A
So
today's
obsession
is
christmas,
mince
pies,
which
I
really
like,
which
don't
contain
any
meat
strangely,
and
I'm
always
looking
to
to
learn
and
share
and
connect.
So
you
know
get
in
touch
and
I'll
share
my
contact
details
at
the
end
of
the
presentation
so
yeah,
let's,
let's
let's
chat
so
hopefully
you
know
what
calico
is,
but
if
not,
then
the
project
calico
community
and
tigera
develop
and
maintain
calico,
which
is
an
open
source.
Networking
and
network
security
solution
for
containers,
virtual
machines
and
host-based
workloads.
A
Sorry,
all
three
of
my
monitors
just
informed
me
that
they
they're
going
to
turn
off
in
a
minute
to
save
power,
which
I
really
don't
want
them
to
do.
So
just
fixing
that
so
here's
some
details.
We
have
more
than
6
000
people
in
our
slack
more
than
150
contributors
and
more
than
a
million
nodes
are
powered
by
calico
every
day.
A
A
A
So,
from
a
background
knowledge
point
of
view
on
such
a
short
session,
I
wasn't
able
to
go
into
ebpf
itself
or
to
to
teach
you
how
to
build
a
calico
ebpf
cluster.
The
reason
I
haven't
done
either
of
those
things
is
because
there's
a
lot
of
really
great
documentation
out
there
about
both
of
those
things.
A
So
if
you
are
unclear
what
eppf
does
then,
then
obviously
come
back
to
this
session
after
after
you've
learned
about
that.
But
I
think
you'll
pick
up
quite
a
lot
here
as
well,
and
our
documentation
does
a
really
good
good
job
of
giving
a
basic
example
of
how
to
build
an
ebpf
cluster,
it's
surprisingly
straightforward,
to
convert
a
vanilla
cluster
to
our
edpf
data
plane.
A
If
you
want
to
try
this
on
your
own
cluster,
you
can
just
follow
along
and
see
what
I
do
today.
But
if
you,
if
you
want
to
to
build
your
own
ebpf
cluster,
then
you
can
convert
any
cluster
to
an
ebpf
cluster
and
you
you
just
need
to
go
to
docs.projectcalico.org
and
those
three
links
there.
I
think
we'll
probably
be
sharing
a
recording
for
the
session
later.
Those
three
links
there
will
take
you
to
the
resources
that
will
tell
you
how
to
do
that.
A
So
I
won't
take
too
long
to
dwell
on
this,
but
why
dive
deep
at
all
a
cluster
usually
just
works
right.
So
why
do
take
the
time?
Well,
it
enables
you
to
diagnose
your
own
problems
and
to
get
a
deeper
understanding
of
how
it
works
and
it
improves
support
as
well
commercial
or
open
source
in
in
an
open
source
scenario.
A
A
A
Now
this
is
really
important.
This
at
this
blue
mount
point
here
is
really
important,
because
if
you
don't
have
that
mount
point
set,
the
bpf
cluster
will
continue
to
work.
But
when
the
data
plane
restarts,
you
will
get
a
longer
than
necessary
service
interruption.
So
I've
been
discussing
internally
with
the
team,
whether
we
may
change
this
to
become
a
hard
requirement
that
if
this
file
system
is
not
mounted
then
then
maybe
we
should
not
spin
up
the
ebpf
data
plane.
A
A
So
the
short
answer
to
your
question
jerome
is
it
depends,
but
you
will
need
to
have
met
the
requirements
defined
here.
So
as
long
as
you
have
that
file
system
mounted
and
you're
running
a
new
enough
kernel
version
and
the
headers
are
built
into
into
your
distribution,
then
it
will
probably
work.
If
you
want
to
be
100
sure
it
will
work,
then
it
needs
to
be
one
of
these
districts.
A
So
some
other
basics
quickly.
Before
we
dive
deeper,
you
need
to
have
a
coup
proxy,
either
fully
disabled
or
in
the
event
that
you
don't
have
coop
proxy
disabled.
You
need
to
you
need
to
use
this
configuration
flag,
bpf2
proxy
iptables,
cleanup
enabled
set
to
false
and
the
out
of
those
two
settings.
The
preferred
one
is
to
disable
coup
proxy
and
the
reason
you
just
don't
can't
do
that
in
all
scenarios
is
because.
A
A
A
Into
more
detail
a
theoretical
packet
flow,
so
what
we're
seeing
here
is
not
our
ebpf
data
plane.
Yet
what
we're
seeing
here
is
the
packet
flow
in
net
filter
through
a
single
linux
host,
and
this
includes
a
kubernetes
node.
So
this
diagram
is
courtesy
of
yan
engelhart.
This
is
an
amazing
diagram,
but
what
you'll
notice
is
that
there's
a
large
section
in
the
middle
with
the
input,
path,
forward,
path
and
output
path
and
the
majority
of
the.
A
However,
bpf
programs
are
attached
in
different
places,
they're
attached
here
at
the
xdp
ebpf
hook
here
at
the
ingress
q,
disc
and
here
at
the
egress
q
disk.
So
we're
able
to
attach
ebpf
programs
in
the
linux
kernel
at
the
start
of
the
flow
and
at
the
end
of
the
flow,
but
not
in
the
middle.
So
all
of
the
bpf
processing
happens
either
at
the
start
or
the
end
of
the
flow,
and
you
can
see
ebps
can
sidestep
the
whole
net
filter
flow.
A
A
No,
it
is
not,
and
that
is
an
attachment
point
for
attaching
an
ebpf
program
to
to
do
arbitrary
work.
Now
the
class
act.
Q
disk
is,
what's
called
a
no
op
q
disk,
which
means
that
it
doesn't
perform
any
action
at
all.
It's
simply
an
attachment
point
if
you
wish
to
attach
your
programs,
so
the
programs
are
attached
at
are
compiled
at
build
time
and
the
per
node
agent
felix.
The
calculate
node
agent
attaches
the
policy
programs
at
runtime.
A
So
at
ingress
you
can
see
that
the
packet
is
received,
and
then
it
goes
through
the
if
we
go
back
essentially
we're
seeing
this
part
of
the
diagram
zoomed.
In
now,
some
bbh
programs
can
be
attached
at
the
xdp
hook
here,
but
the
majority
of
the
ebpf
policy
happens
here
attached
at
the
tc
ingress
tree
of
q
disks
and
the
class
act
q
disk.
Now
this
leads
back
to
a
previous
point.
A
Work
alongside
the
coup
proxy
functionality
and
it's
it's
difficult
to
interleave
the
ebps
program,
functionality
and
the
coup
proxy
functionality
and,
as
a
result,
it
was
necessary
to
replace
the
proxy
function.
Functionality
in
kubernetes
with
the
ebpf
functionality,
essentially
to
rewrite
that
functionality
in
ebpf,
fortunately
rewriting
that
coup
proxy
functionality
in
ebpf
actually
allowed
our
dev
team
to
make
some
improvements.
A
A
A
A
And
the
traffic
is
still
routed
normally
through
iptables
and
the
fib
just
checking
my
notes
to
see.
If
there's
anything
else,
I
should
point
out
the
ingress
and
egress
programs
are
attached
at
cali
data
and
tunnel
interfaces.
Other
interface
types
are
handled
as
exceptions.
I
should
make
that
point.
Yeah
ipnip
tunnel
interfaces
and
wireguard
interfaces
are
handled
as
exceptions.
A
A
A
A
A
A
Sorry
calculate
vpf
runs
within
the
calico
node
pod,
so
we
run
it
by
using
cooperative
exec
in
the
calculator
name
system,
namespace
the
name
of
the
calico
node
that
we're
interested
in
and
then
the
command
and
I'll
give.
You
live
examples
of
this.
So
this
command
as
with
so
many
kubernetes
commands
it
doesn't
really
roll
off
the
tongue.
But
but
when
you
see
it
live,
hopefully
it
will
stick.
A
A
A
A
Okay,
I
flew
through
that
because
I
wanted
to
have
plenty
of
time
to
demonstrate
this
on
a
live
cluster.
As
a
result,
we've
actually
ended
up
with
enough
time.
I
think
I
need
about
20
minutes
and
we
have
just
over
that.
So
we
have
time
if,
if
anyone
wants
to
go
back
over
any
of
that
content,
for
me
to
speak
in
more
detail
or
has
any
questions
now
would
be
an
okay
time
for
that.
A
A
A
Obviously,
in
my
home
isp
changes
it's
ip,
I'm
connecting
to
a
tcp
connection
to
a
gcp
load,
balancer
again
that
ip
is
different
because
I've
I've
torn
down
this
environment
and
rebuilt
it,
but
the
porta
direct
port
is
the
same
and
we'll
actually
see
load.
Bouncing
forwarding
me
to
a
node
port
on
one
of
the
nodes.
A
That
program
will
make
a
forwarding
decision
and
it
will
refer
to
the
ebpf
maps,
then
I'll
be
vxlan
tunnelled
across
to
the
node.
That's
serving
the
workload
which
is
answering
on
port
8080
and
then
the
ingress
and
egress
q
disks
that
we
showed
that
processing
will
happen
here
on
the
physical
interface
of
the
node,
as
well
as
on
the
pods
of
ether
interface
pair
and
then
we'll
see.
A
direct
server
returned
to
the
client
so
notice
that
that
return
traffic
doesn't
go
back
through
the
ingress
node
as
it
conventionally
would,
with
crew
proxy.
A
A
A
A
A
A
We
don't
need
to
check
these
permissions,
particularly
it's
a
little.
It
seems
misleading
that
this
says
none,
but
that's
fine!
That's
correct!
If
you're
familiar
with
mail,
I
used
to
know
how
to
read
this,
but
I
can't
quite
recall
anymore,
but
the
key
point
is
this
is
fine.
This
is
what
you're
expecting
to
see
the
bad
outcome
would
be
that
you
have
no
is
that
you
have
zero
lines
of
output.
A
So
if
you
still
have
q
proxy
running
on
your
cluster,
alongside
the
ebpf
data
plane,
things
will
continue
to
work,
but
you'll
see
very
high,
cpu
utilization
and
that's
because
coup
proxy
and
the
bbf
data
plane
are
fighting
for
fighting
over
the
ip
tables
rules.
So
we
need
to
make
sure
coup
proxy
is
disabled.
So
the
first
way
we
can
do
that
is
to
simply
look
at
the
output,
and
we
should
see
that
we
have
no
q
proxy
running
and
I'll
show
you
how
we
disable
it.
A
In
kubernetes,
you'll
be
familiar
with
the
concept
of
a
demon
set
which
specifies
that
you
want
to
have
a
workload
run
on
every
node
and
that's
how
coop
proxy
conventionally
runs
so
there's
in
the
cube
system.
Namespace
there's
a
demon
set
called
qproxy
and
you
can
see
that
it's
not
running,
which
is
what
we
want
and
that's
the
case
because
we've
applied
this
non-calico
equals
true
node
selector.
A
A
A
A
A
A
A
A
But
of
course,
we
just
said
that
we
need
to
disable
coop
proxy,
so
we
can't
disable
coup
proxy
and
have
felix
use
coup
proxy
to
talk
to
the
kubernetes
api.
So
the
crux
of
it
is:
we
need
to
change
the
configuration
of
calico
to
tell
it
to
talk
directly
to
the
real
workload
endpoint
of
the
api
rather
than
talking
to
this
cluster
ip.
A
A
A
Now
the
tigera
operator's
job
is
to
deploy
calico's
per
node
agent
and
to
bring
it
into
conformance
with
a
target
configuration.
So
what
happens?
Is
we
create
this
kubernetes
services
endpoint
and
we
specify
the
real
kubernetes
service
host
and
port
and
as
soon
as
felix,
the
per
node
calico
agent
sees
that
configuration
it
will
restart
the
daemon
and
the
kubernetes
the
calcodemon
and
speak
directly
to
the
kubernetes
api,
at
which
point
we
can
stop
the
q
proxy
service?
A
A
A
A
B
A
A
A
So
so
that
quick
command
lets
us
see,
for
example,
the
ip
sets,
but
it
will
also
let
us
see
all
kinds
of
other
ebpf
maps
and
that
lets
us
see
what's
happening
on
the
node.
So
let's
see
some
other
cool
examples
of
that,
this
command
looks
really
crazy,
but
again
it's
it's
actually
a
variation
on
the
same
command.
A
A
A
A
A
A
A
A
So
all
it
remains
to
do
is
to
have
a
quick
look
using
tc
like
this.
I
mentioned
tc
earlier
show
the
q
disk
q
disks
for
that
device,
and
we
can
see
that
the
class
app
q
disk,
which
is
the
no
op
q
disk.
Like
I
mentioned,
we
can
see
that
it's
got
five
drops,
which
means
that,
as
well
as
the
legitimate
traffic
that
I've
been
sending,
there's
been
five
attempts
to
connect
to
that
port.
A
A
Go
on
this
there's
a
chance.
This
may
not
work.
I've
noticed.
Google
chrome
do
some
really
interesting
things
on
my
phone,
where
it
doesn't
connect
out
properly,
but
let's
give
it
a
go.
Also,
my
phone's
trying
to
connect
to
that
url
and
failing
because
it's
my
phone
is
on
4g
and
it's
not
coming
from
the
same
source
type
address.
A
A
I
get
the
debug
logs
now
this
looks
crazy
and
the
reason
there's
so
much
going
on
is
because,
if
I
ctrl
c
that
you
can
see
that
the
destination
port
is
port,
22
ssh,
so
all
this
traffic
is
actually
itself
being
evaluated.
It's
evaluating
the
ssh
session
that
I'm
connecting
to
the
host
fire.
But
that
being
said,
these
these
logs
are
incredibly
verbose,
which
is
why
you
shouldn't
turn
this
on
on
the
production
cluster.
A
A
Cool,
so
oh
mario's
got
a
question
I'll
just
answer
that
in
one
second,
so
mary
just
before
I
answer
your
question
just
to
give
to
come
back
to
this
final
slide.
I
just
wanted
to
to
give
you
these
contact
details,
so
you
can
make
a
note
of
them.
This
is
my
personal,
twitter,
linkedin
and
slack,
and
so
on.
So
I'm
very
happy
to
you
know
chat
if
any
of
you
would
like
to
so,
you
ask
what's
actually
stored
in
the
bpf
maps.
A
A
B
A
A
A
A
So
jerome,
you
asked
a
really
great
question:
I'm
going
to
publish
a
blog
post,
hopefully
in
the
next
few
days,
about
building
a
bgp
and
kubernetes
and
calico
iptables
data
plane
cluster
in
minikube.
On
my
laptop,
I
had
quite
a
lot
of
success
with
doing
that,
and
I
tried
adding
ebpf
to
that
cluster
and
it
didn't
work
because
the
mini
cube
iso
doesn't
contain
the
necessary
bpf,
headers
and
so
on,
even
though
it
is
running
a
late
enough
kernel.
A
Failing
that,
if
you
look
at
the
ebpf
course
that
I've
released
in
the
last
few
days,
if
you
search
for
ccol2
ebpf,
which
is
the
calico
ebpf
course,
you'll
find
that
there's
some
instructions
in
there
about
how
to
enable
ebpf
data
plane
on
a
vanilla,
kubernetes
cluster
on
vms
and
in
my
course
I
described
doing
that
in
gcp.
But
there's
no
reason
that
you
couldn't
do
it
on
your
local
host.
A
A
A
A
Mario,
I
see
your
question
about
how
ip
roots
coexist
with
the
bpf
maps.
I'm
going.
To
be
honest,
I
don't
think
I
understand
it
in
enough
detail
to
give
a
definite
answer.
Now.
Perhaps
we
can
talk
on
calculus
slack,
because
I
don't
want
to
give
the
wrong
information
and
I'm
not
quite
sure
how
that
how
the
relationship
works
so
yeah.
Maybe
we
can
take
that
one
offline
francis.
Let's
have
a
look.
A
Yes,
that's
right,
francis
correct
an
ebpf
map,
so
francis
is
making
the
point
that
an
ebpf
map
can
store
any
kind
of
data.
It's
just
that
the
calico
data
plane
chooses
to
store
the
these
part,
these
pieces
of
information
so
but
this
also
relates
to
mario's
point
if
it's
storing
roots
in
the
edpf
maps,
how
does
that
relate
to
the
to
the
normal
routing
table?
And
the
answer
is,
to
be
honest,
I'm
not
totally
sure.