►
Description
Featuring Lawrence Gadban. Istio’s new ambient mesh mode is designed to alleviate challenges associated with the long-standing sidecar-based approach. However, moving from sidecars to a node-level agent fundamentally changes the data path for traffic flowing within the service mesh. As part of the initial release of ambient mesh, a working implementation of the necessary networking configuration was created. Still, there is room for improvement, especially when considering the exciting Linux kernel technology eBPF. In this session we will explore how traditional Istio networking works, what changes were necessary for ambient mesh, and finally we will dig into how eBPF can be used to further enhance ambient mesh.
A
A
So
in
this
session
we're
going
to
be
talking
about
enhancing
ambient
mesh
with
ebpf.
We
just
heard
a
great
summary
of
Ambien
and
kind
of
the
state
of
it
in
istio,
and
so
now
we'll
look
at
how
we
can
kind
of
use
ebpf
the
exciting
new
technology
that
is,
you
know,
going
to
be
everywhere
this
week,
how
we
can
use
that
to
help
with
ambient
so
quickly.
My
name
is
Lawrence
godbon
I'm,
a
software
architect
at
solo.
A
I
o
I've
been
here
for
quite
a
while
now,
almost
three
years,
and
so
recently,
I've
been
working
primarily
on
ebpf,
so
I've
had
some.
You
know
time
to
get
very
kind
of
in
the
weeds
with
it
and
also
how
it's
going
to
relate
to
ambient.
So,
as
we
just
heard
in
the
previous
talk,
kind
of
just
quickly
go
over
kind
of
the
state
of
istio
and
ambient,
so
the
current
sidecar
model
of
istio,
it's
widely
used
very
battle
tested.
Several
people
are
running
in
production.
You
know
massive
scale.
A
So
let's
quickly
take
a
look
at
the
sidecar
model,
kind
of
just
set
the
stage
so
I'm
sure
you've
seen
this
in
some
form
or
fashion,
where
you
have
kind
of
two
pods
talking
to
each
other
in
a
sidecar
mesh
and
on
the
left.
We
have
the
sleep,
a
sleep
application
on
the
right.
We
have
a
pet
store,
and
so
we
want
to
make
a
call
from
sleep
to
the
pet
store.
And
so
what
will
happen
is
we'll
make
a
curl
or
whatever
it
is.
A
That
request
will
get
kind
of
redirected
to
the
sidecar
proxy
living
in
the
sleep
pod
and
then,
as
it
goes
to
pet
store.
There
is
also
a
sidecar
proxy
and
pet
store
that
will
accept
the
request
and
then,
ultimately,
it
will
end
up
at
the
pet
store
application,
and
so
the
way
that
this
actually
is
implemented.
You
may
have
seen
this
diagram
before
it's
a
really
good,
it's
from
a
really
good
article,
by
Jimmy
song,
on
kind
of
the
whole
sidecar
injection
model
and
what
the
actual
details
are
of
how
this
is
implemented.
A
Essentially,
what
this
is
showing
is
that
there's
quite
a
bit
of
iptables
rules
necessary
in
each
pod
in
order
to
get
this
redirection
to
work
and
the
way
that
that
works
is,
you
may
have
seen
this
before.
This
is
kind
of
the
the
net
filter
in
Linux.
This
kind
of
the
flow,
the
diagram
of
how
that
flow
works,
as
well
as
just
the
general
networking
path
in
Linux.
So
the
iptables
rules
configure
at
various
stages
in
this
diagram.
A
So
in
other
words,
we
have
a
pod
and
that
pod
ultimately
has
its
own
network
name.
Space
and
kind
of
there
are
two
ways
of
doing
this
in
today's
model,
with
the
with
in
a
knit
container,
which
I
believe
is
the
default
right
now,
where
you'll
get
a
sidecar
injected,
you
also
get
an
init
container
injected
and
that
init
container
will
create
the
iptables
rules
necessary
for
this
redirection
and
those
iptable
rules
will
be
created
in
the
pods
Network
namespace.
A
Only
so
then
the
rest
of
your
application
spins
up
and
then,
as
you
make
Network
as
Network
traffic,
happens
inside
of
that
pod.
Those
iptables
rules
kick
in
in
order
to
redirect
the
traffic
that
we
need.
So
you
know
if
we
have
two
pods
on
the
same
node
or
even
if
they're,
on
different
nodes.
The
key
is
that
those
iptables
rules
are
in
the
pods
network
name
spaces,
and
they
only
interact
with
the
traffic.
That's
in
in
that
pod.
A
So
with
Ambien,
we
kind
of
just
saw
a
high
like
a
high
level
overview
of
what
the
networking
looks
like
in
in
ambient
it
can.
It
requires
significant
changes
from
the
current
model
of
redirection,
and
so,
let's
take
a
look
at
how
it's
implemented
today.
So
in
this
this
diagram
we
see
the
same
sleep
in
pet
store
application.
We
have
them
running
on
two
separate
nodes
and
we
have
the
Z
tunnels
running
on
the
separate
nodes
that
act
as
the
that's.
A
We
need
to
redirect
it
to
the
Z
tunnel
for
for
the
node
that
the
Pod
is
on
and
then
the
then,
as
the
traffic
goes
to
the
pet
store,
the
Z
tunnel
on
the
pets
on
the
no
the
pet
store
is
on
needs
to
intercept
that
traffic,
and
then
we
will
route
it
to
the
actual
pet
store
application,
and
so,
unfortunately,
it's
not
as
simple
as
just
changing
the
arrows
like
in
this
diagram.
There's
a
lot
of
complex
networking
changes
that
had
to
be
made
to
make
this
work.
A
So
how
can
we
achieve
that,
especially
if
we
don't
get
a
chance
of
kind
of
interacting
with
the
workload
instance
again,
one
of
the
benefits
of
ambient
is
that
you
don't
have
to
actually
modify
the
pods
themselves.
You
can
just
say
that
you
want
this
pod
to
be
part
of
the
ambient
mesh
and
it
will
work
you
know
without
having
to
change
that
pod,
where
you
don't
have
to
bounce
it
or
anything.
A
So
one
this.
This
is
going
to
be
a
quick
example
of
how
it
works,
because
it
is,
you
know,
pretty
complicated,
so
we
have
a
node
and,
as
part
of
the
ambient
networking
you
there's
no
more
a
knit
container
or
anything
that
will
will
do
this.
You
have
to
use
the
cni,
and
so
the
istio
cni
plugin
has
a
mode
essentially
for
ambient,
and
so
when
that,
when
that
pod,
that
represents
the
cni
spins
up,
it's
also
a
demon
set.
So
it
will
be
on
every
node
when
it
spins
up
it
does
a
few
things.
A
It
creates
a
set
of
iptables
rules,
similar
to
kind
of
what
the
the
istio
knit
container
was
doing.
It
will
set
up
a
what's
called
an
IP
set,
which
is
how
we
keep
track
of
various
IPS.
That
are,
you
know,
for
example,
pods
in
the
mesh.
We
need
to
know
what
those
IPS
are
and
then
it
will
also
set
up
a
set
of
routing
tables
which
the
kernel
will
use
as
for
basically
we're
using
policy
based
routing.
A
We
also
have
the
Z
tunnel,
so
that's
also
the
demon
set
that
runs
on
every
node,
so
that
will
spin
up
that
pod
will
spin
up,
and
in
this
example,
we
have
the
the
Z
tunnel
actual
application
container,
and
then
there
is
kind
of
the
standard
container
networking
which
we'll
talk
a
little
bit
more
in
detail
in
a
second
where
it
will
have
its
own
interface
and
that
inner
that
network
interface
will
be
connected
to
potentially
a
vet
pair.
That's
in
the
host
Network
namespace
and
then
there's
also
a
gene.
A
A
set
of
geneve
tunnels
set
up
that
actually
allow
the
packet
to
get
to
where
it
needs
to
go
in
the
Z
tunnel.
So
today
the
Z
tunnel
is
implemented
by
Envoy
that
that
is
potentially
going
to
change
in
the
future,
but
the
the
key
is
that
the
envoy
listener
needs
to
know.
You
need
to
be
able
to
route
to
the
correct
Envoy
listener,
for
whether,
for
example,
you're
hijacking,
an
outbound
connection
or
an
inbound
connection,
and
so
after
every
so
in
order
to
to
set
that
up.
A
The
Z
Channel
also
has
its
own
IP
tables,
rules
that
are
configured
in
the
Z
tunnels,
networking
space
as
well
as
a
set
of
also
routing
tables
in
the
Z
tunnels,
Network
namespace,
so
combined.
You
kind
of
have
this
set
of
iptables
rules
in
the
host
Network,
as
well
as
in
the
Z
tunnel
network
namespace,
and
you
can.
These
are
all
going
to
combine
to
allow
us
to
achieve
the
redirection
we
need
so
going
back
to
our
sleep
pod
example.
A
So
in
this
case
we
have
the
Sleep
container
on
this
node
and
we
also
have
the
similar
kind
of
f
F0
to
a
vet
pair
setup,
where
you
know
you
send
traffic,
so
the
sleep
pod
will
send
traffic
out
of
its
F0
interface.
Go
into
the
network
namespace
through
the
ve
pair,
that's
assigned
to
it,
and
so,
if
we
want
to
do
a
curl
to
the
pet
store,
a
packet
is
going
to
be
sent
out,
basically
right.
A
So
if
this
is
the
first
packet,
this
will
be
the
sin
packet
of
the
TCP
handshake,
and
so
in
this
case
this
packet
sent
out
it
will
go
out
of
the
vet
pair
for
sleep
and
then
kind
of
all.
This
magic
happens
where
we
have
an
iptables
rule
in
the
host
Network
namespace.
That
will
determine
that
this
IP
The
Source
IP,
is
in
an
IP
set.
A
We
know
this
is
part
of
the
ambient
mesh,
so
we're
going
to
actually
Mark
the
packet
and
by
marking
the
packet,
then
we
can
use
a
routing
table
to
say
if
this
packet
has
this
Mark,
then
we're
going
to
redirect
it
through
the
istio
out
tunnel,
so
that
It
ultimately
ends
up
at
the
correct
listener
in
Envoy,
and
so
that's
that's
literally
for
the
first
hop
of
that's
the
first
place
that
one
packet
goes
and
so
kind
of
the.
The
main
point
is
that
it's
a
lot
of
really
like
Arcane
magic.
A
It's
things
that
are
very
low
level,
Linux
networking
you've
all
our
Chief
Architect
worked
on
this
and
he
was
talking
to
him.
I
got
the
sense
that
it
was
challenging
to
say
the
least,
to
get
it
together.
So
the
ultimate
result
is
that
it's
something
that's
difficult
to
read
difficult
to,
maintain
and
really
we're
trying
to
convince
the
the
network
stack
to
do
something
that
it
wasn't
designed
to
do,
and
so
how
can
we
improve
that?
A
So
that's
where
ebpf
comes
in
so
ebpf
Linux
technology
that
allows
you
to
kind
of
extend
the
kernel
on
the
Fly.
It's
very
there's
a
lot
of
hype
surrounding
it
and
it's
gaining
steam
really
quickly
in
the
cloud
native
World.
Aside
from
the
hype,
though,
there
there's
a
lot
of
value
to
it
there.
It
has
a
lot
of
benefits
in
things
like
networking
which
we're
talking
about
today.
Another
example
that
I
called
out
is
even
today
with
a
sidecar
model.
A
There
is
kind
of
the
concept
of
sidecar
acceleration,
and
this
is
you
know
there
are
several
Community
projects
open
source.
Community
projects
like
mayor
bridge
is
one
we
have
something
in
glue
mesh
that
does
this,
and
several
other
platforms
also
have
sidecar
acceleration,
which
essentially
uses
evpf
to
shorten
the
network
path
for
traffic
going
to
and
from
side
cars.
A
This
is
probably
like.
It
seems
like
this
is
going
to
become
table
Stakes
for
a
sidecar
based
mesh,
and
so
I
wanted
to
call
it
out
as
something
that
today,
people
are
using
evpf
to
solve.
There's
also,
there's
also
places
in
security
and
observability.
That
evpf
has
a
lot
of
value
too,
since
EVPs,
so
ebpf
originated
from
classic
BPF,
which
was
essentially
designed
for
packet
filtering.
A
So
if
you've
ever
used,
TCP
dump
that
originally
was
written
with
classic
ppf
as
a
way
of
doing
very
quick
packet
filtering
and
the
goal
there
was,
you
need
to
do.
A
lot
of
processing
as
packets
are
flowing
through
the
network
stack
and
that's
in
the
kernel,
so
you
can't
you
wouldn't
be
able
to
have
the
performance
you
need
if
you
were
essentially
copying.
A
We
can
programmatically
Define
our
Network
rules,
and
so,
if
you
think
about
it,
what
we
really
can
do
is
we
essentially
can
program
our
own
network
stack
and
so
in
the
case
of
ambient,
we
can
program
the
kernel
to
do
exactly
what
we
want
it
to
do.
Rather
than
trying
to
use
all
these
different
tools
to
kind
of
piece
together
something.
A
A
So
we
have
a
node
on
that
node
as
a
cubelet,
a
container
runtime
and
a
cni
plug-in
that's
kind
of
like
the
basis
of
creating
pods
on
a
given
node.
In
kubernetes,
so
the
cubelet
will
see
that
there
is
a
pod
to
be
run
on
this
node,
and
so
the
cubelet's
job
is
to
actually
create
what
you
need
to
do
to
create
this
pod.
So
the
first
thing
it
will
do
is
it
will
talk
to
The,
Container
runtime.
You
know
whatever
container
runtime
that
is
in
this
case.
A
I
have
an
example
of
container
D
container
D
will
then
create
all
of
the
Linux
Network
namespaces.
It
will
spin
up
whatever
containers
are
in
the
Pod.
Essentially
you
end
up
with
a
pod
that
has
the
containers
you
need
inside
of
it,
then
the
container
runtime
is
going
to
pass
it
off
to
the
cni
plugin,
which
will
actually
wire
up
the
the
interfaces
that
the
Pod
will
use,
and
so,
in
this
example,
kind
of
similar
to
what
we
saw
earlier
in
the
sleep
pod.
A
There
is
an
f-zero
inner
network
interface
that
basically
all
of
the
network,
that's
how
you
have
Network
traffic.
So
inside
of
the
sleep
pod,
when
you
use
the
network
you're
going
to
go
through
this
F0
interface,
which
in
this
example
is
a
Beth
pair
or
a
virtual
ethernet
pair,
which
allows
you
to
get
packets
from
one
network
namespace
to
the
other,
so
you
send
a
packet
through
F0
in
sleep.
A
It
is
that's
on
that
note,
okay,
so
why
do
we
care
about
kubernetes
networking
with
ebpf?
So
if
we
go
back
to
this
diagram
of
kind
of
the
general
networking
flow
and
net
filter,
there's
kind
of
two
pieces
at
the
beginning
at
the
end,
at
the
end
of
this
flow,
that
I
highlighted
the
Ingress
and
egress,
and
so
these
are
essentially
the
TC
Ingress
and
TC
egress
hook.
A
Points
of
the
networking
stack
and
you
can
see
that
these
actually
are
before
and
after
pretty
much
all
of
the
layers
of
processing
that
happens
in
the
kernel
and
so
TC
stands
for
Traffic
Control,
it's
another
one
of
those
kind
of
core
subsystems
of
the
Linux
networking
stack
what's
interesting
about.
That
is
that
we
can
attach
ebpf
programs
to
those
hook
points
so
at
the
Ingress
and
egress
of
TC.
A
We
can
attach
an
ebpf
program,
and
that
means
that
when
we
can
interact
directly
with
those
container
network
interfaces
which
is
kind
of
the
core
concept
of
kubernetes
networking,
we
can
attach
our
own
evpf
programs.
We
have
a
consistent
place
to
attach
evpf
programs
and
we
can
do
things
either
before
or
after
the
network
stack
has
has
had
this
chance
to
do
whatever
it
needs
to
do,
and
then
with
the
cni
we
actually
also
get
a
life
cycle.
A
A
This
is
actually
a
pretty
common
pattern,
so
psyllium's
data
path,
kind
of
uses
this
as
one
of
its
core
components,
there's
a
lot
of
things
implemented
at
the
TC,
Ingress
and
Ingress
layer
in
psyllium.
And
so,
if
we
go
back
to
our
diagram,
you
can
see
here
where
we
have
the
cni.
We
have
the
Z
tunnel
on
a
node.
A
We
have
our
sleep
container
and
now,
at
this
vet
pair
in
the
host
Network
namespace,
we
have
this
TC
ebpf
program,
and
so
what
that
allows
us
to
do
is
let's
go
back
to
the
same
example
where
you
want
to
make.
You
know,
requests
to
the
pet
store
that
that's
going
to
ultimately
result
in
a
packet
being
sent
out
as
that
packet
is
flowing
through
the
vep
interface
and
the
host
Network
namespace.
A
We
have
our
ebpf
program
there,
and
so
what
we
can
do
is
kind
of
programmatically
do
something
there's
you
know
some
pseudocode
here
on
the
right
where,
if
this
packet
has
a
mark
for
some
reason,
we
need
to
do
something
special
with
it
or
we
don't
want
to
do
anything
with
it.
We
can
just
return
and
let
the
network
stack
take
care
of
it,
but
otherwise,
in
the
case
that
you
know
it's
not
marked,
which
is
kind
of
signifying
that
this
is
traffic
from
an
application
container.
A
We
can
use
something
like
a
ebpf
helper
to
do
a
redirect.
We
can
like
explicitly
code
it
to
do
that.
So
we
can
say
what
I
want
to
do
is
based
on
the
fact
that
there's
no
Mark
I
want
to
redirect
this
to
the
outbound
tunnel
of
the
Z
tunnel,
and
so
it
allows
us
to
kind
of
build
up
in
code
exactly
what
we
need
to
do
at
every
step
of
the
network
flow
and
so
in
general,
that
ambient
networking
is
it
is
complex
because
of
what
we're
trying
to
do.
A
But
the
goal
is
that
the
solution
doesn't
need
to
be
complicated
as
well,
and
so,
rather
than
trying
to
convince
the
net
the
kernel
to
do
you
know
something
that
it
doesn't
know
what
it's
doing
or
doesn't
want
you
to
do
that.
We
can
actually
program
it
to
do
exactly
what
we
want
and
we
can
kind
of
essentially
encode
tell
it
tell
the
kernel
what
ambient
networking
is,
and
this
is
how
I
want
you
to.
A
So
I
will
call
out
that
ebpf
is
not
Silver
Bullet.
It's
very
there's
a
lot
of
new
skills
necessary,
not
just
for
writing
it,
but
also
for
debugging
it
it's
challenging,
but
we
kind
of
see
that
the
pros
outweigh
the
cons,
especially
with
how,
with
the
speed
of
adoption
and
progression
that
evpf
exists
today,
I'm
sure
that
you
know
throughout
this
entire
week
you'll
see
lots
and
lots
of
talks
of
evpf.
A
A
Yet
actually
so
Adam
from
our
team
is
actually
giving
a
talk
about
this
tomorrow
and
Aiden
is
giving
talk
today
at
EBP
update
on
getting
HTTP
metrics
using
evpf
without
the
need
of
any
kind
of
proxy
okay,
and
that's
it
if
you're
interested
I
will
also
be
happy
to
talk
in
the
Halls
about
this
but
yeah.
That's
all
I
have
thank
you.