►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Simplifying and Making the Network Programmable with Kubernetes and SRv6 - Daniel Bernier, Bell Canada & Louis De Los Santos, Isovalent
SRv6 is being rapidly deployed by some of the largest names in the networking industry because it has the potential to simplify existing enterprise, telco, and provider networks while also introducing the idea of “programming the network as a computer.” Networking teams are now looking for ways to bridge the gap between SRv6 and Kubernetes to treat containers as ordinary clients on their networks. This talk will discuss how Cilium and its eBPF data plane was extended to support telco networking requirements in a cloud native way by integrating SRv6 and Kubernetes. It will start with an overview of the SRv6 use cases and network architecture at Bell Canada. We will then demo how Cilum’s SRv6 integration can be utilized with BGP to build a multi-cluster L3VPN topology over native IPv6.
A
A
So
I
will
first
start
with
a
problem
statement,
because
when
you
do
this
and
people
ask
me
why
we
want
to
do
this,
there's
there
needs
to
be
a
problem
statement.
The
logic
is
around.
How
do
we
make
our
Network
more
tuned
to
the
application?
We
normally
do
things
with
encapsulation.
We
do
some
kind
of
networking
tricks,
but
we
don't
look
at
what
we're
trying
to
do
is
actually
connect
applications.
A
So
I
did
I
brought
just
three
statements
which
I
liked
I
ran
out.
Do
we
some
of
the
problems
around
the
applications?
For
example,
Network
tweet
in
2030,
from
the
it
UT
explained
that
one
of
the
gaps
we
had
in
the
existing
internet
right
now
was
the
network
doesn't
really
connect
well
really
well,
with
the
the
applications
another
one
which
is
famous
I,
think
from
it
was
in
medium.
A
They
just
really
want
to
build
their
applications
and
the
concept
of
what
this
network
for
them
is
kind
of
overly
complex
for
no
reason
and
I
really
like
this
talk,
because
in
a
utopian
World,
We
would
like
this
to
be
happening.
We'd
like
to
be
able
to
have
a
simple
declarative
and
developer
friendly
way
to
do
the
application,
networking
and
evolving
towards
a
network
specification
or
a
way
to
expose
those
networking
Concepts
in
a
meaningful
manner,
from
the
the
further
developers
not
really
how
to
do
a
layer.
Two
Constructor
layer,
3
constructs.
A
So
basically,
what
we
want
is
a
an
application,
developer
or
coder
to
be
able
to
expect
the
specifications.
Some
magic
will
happen
in
the
network
and
try
and
make
that
happen.
And
how
do
we
make
that
developer,
friendly,
so
crash
course
into
srv6
Network
programming?
Some
people
might
have
their
ears
ears
bleeding
afterwards,
but
this
is.
This
comes
out
directly
from
the
ITF.
It
is
a
standard,
it's
an
RFC
which
is
a
8986
which
allows
to
create
network
programs
using
or
basically
policies
composed
of
instructions
and
behaviors
or
behaviors
that
we
call
them.
A
Network
programs
have
a
view
which
are
network-wide.
You
build
a
network
program
which
is
end-to-end
from
a
telecom
perspective,
but
your
instructions
are
locally
defined.
You
build
out
your
code,
and
this
is
the
one
that
does
the
the
instructions
it's
built
based
on
standard
IPv6
hit
leverages
IPv6
as
a
common
denominator
and
transport,
and
basically,
what
you
do
is
you
connect
those
small
instructions
together
to
create
that
end-to-end
program?
A
It
uses
a
source,
routing
Paradigm,
meaning
that
at
the
source
of
the
network
or
the
source
of
the
the
domain
of
segment
rotting,
you
actually
encode
all
the
places
you
want
to
up
in
your
network
and
it
will
give
you
the
application.
The
the
the
actual
program
you
wanted
to
to
adhere
to
this
is
done
to
either
an
extension
header,
which
is
also
a
standard,
a
RFC
8754
or
through
basic
use
of
IPv6
other
IP
address
hit.
A
Those
instructions
are
identified
or
encoded
into
what
we
called
path,
segments
or
segment
segment
IDs,
which
are
represented
into
an
IPv6
format
and
there's
two
ways
of
doing
so.
One
which
does
the
base,
which
was
the
initial
one,
which
is
a
using
128
bit
of
the
IPv6,
address
to
be
able
to
identify
segments.
A
The
the
newer
one
which
is
compressed
Sid
is
how
to
use
some
chunks
of
16
bits
out
of
that
under
28-bit
to
create
those
segments,
the
the
latter
part
being
more
friendly
for
towards
Asic,
but
they
actually
both
support,
are
the
same
architecture.
So,
basically,
you
can
encode
a
path
into
either
a
list
of
108-bit
segments
in
an
extension
Adder
of
IPv6
of
just
another
IP
Adder.
If
you
only
want
to
have
one
instruction
and
only
one
other
IP
Adder,
if
you
want
to
have
multiple
small
instructions,
that's
a
compressed
version.
A
So
you're
gonna
tell
me
what
the
heck
does
it
do
with
applications.
So,
basically
you
do
your
instructions.
Instructions
are
packet,
processing
programs.
If
you
want
to
do
when
you
treat
the
packet
can
be
done
in
P4.
Can
it
be
done
in
C?
Can
it
be
done
in
evpf,
for
example,
and
it's
all
you
do
what
instructions
you
want
your
packet
to
be
processed
on
and
you
implement
them
a
multiple
port,
as
I
said
before
they
are
locally
dependent.
A
So
I
can
actually
create
a
code
that
nobody
in
the
world
will
ever
know
what
it
does.
But
it's
my
code
and
actually
it's
customized
to
me
they
can
be
standardized.
So
some
of
those
instructions
are
standard
how
to
do
encapsulation
of
Layer
Two
how
to
Decap
and
do
a
lookup
in
iptable
V4,
those
kind
of
things
and
some
can
be
totally
user
defined.
So,
for
example,
a
network
function
vendor
would
want
to
create
something
which
is
srv6
compliant,
could
decide.
A
I
re
I
do
a
complete
instruction
code
into
my
box
and
my
function,
and
it
still
it
would
still
be
a
sorry
six
compliant.
Even
though
I
haven't
not
gone
to
the
ITF
and
standardized
it
so
to
my
logic,
and
if
I
do
the
summarization
of
Sr
segment
rotting,
V6
I
call
it
the
one
protocol
to
rule
them
all,
because
once
you've
done
it
well,
you
don't
have
to
go
back
to
any
standards
to
build
a
rebuild
and
be
have
acknowledgment.
A
It's
all
integrated
once
you've
done
this
once
you
have
created
all
your
say,
your
your
your
segments,
then
you
create
your
Sr
policy
or
your
end-to-end
program,
and
then
you
steer
traffic
based
on
any
kind
of
characteristics.
You
want
to
do
either
logical
physical
interfaces
or
five
tuples
and
all
those
things
and
the
actually
how
you
will
build.
You
will
build
your
network,
your
you
build
your
your
network
program.
So
what
is
interesting
is
from
a
single
logic
of
a
policy
of
a
single
address.
A
Bit
I
can
represent
any
networking,
constructs
layer,
2,
layer,
3,
functionalis
firewall.
Everything
can
be
identified
this
way.
So
that
brings
me
back
to
the
dreaded
Telco
networking
use
case.
This
is
how
we
actually
have
to
deal
with
networking
right
now
in
the
communities
world.
Most
of
our
applications
need
to
talk
to
multiple
domains.
We
don't
know
to
do
it
easily
in
in
communities.
We
create
an
insane
amount
of
complexity
with
maltus
multi-cni.
Try
to
do
those
kind
of
works.
A
We
still
haven't
figured
out
how
to
connect
my
kubernetes
cluster
to
multiple
environments
and
the
reality
is
networked.
Operators
even
Enterprises
have
that
problem
so
and
before
that,
we
didn't
really
think
about
this,
but
I
think
it
became
into
a
spotlight
when
Telco
started
to
think
about
using
kubernetes,
which
was
the
first
thing
that
happened,
but
so
what
I'm
trying
to
look
at
and
when
we
look
what
we
engage
with
these
element
team
is:
how
could
we
do
something,
a
bit
more
disruptive
and
looking
at
new
ways
of
addressing
their
problem?
A
This
is
where
we
went
and
did
srv6
VPN
or
layer,
3
VPN,
into
cilium
how
to
Leverage,
The
construct
of
cilium
and
the
policies
egress
policies
to
be
able
to
be
able
to
have
a
pod
single
interface
single
default
routing
table
nothing
complex,
no
forwarding
rules,
notes,
insanities
or
IP
static
routes
to
be
able
to
attach
the
right.
The
right
networking
constructs
when
required
and
use
the
logic
of
bgpv
apnv4
for.
B
A
For
the
layer,
3
VPN
advertisement
to
be
able
to
create
those
policies
dynamically,
so
from
a
developer
perspective,
you
just
assign
a
vrf.
The
rest
becomes
really
quite
simple.
Once
you've
done
this.
Any
pattern
is
now
possible
because
I
found
a
way
to
hook
my
network,
my
my
application
into
a
network.
My
network
can
be
stretched
across
multiple
domains.
Multiple
Technologies,
because
I
can
still
use
the
earlier
constructs
of
the
network
with
like
gateways
and
net
devices.
Whatever
we
want,
we
can
also
introduce
service
change.
A
We
can
actually
insert
functions
or
different
devices,
either
physical
or
virtual
inside
that
policy,
meaning
my
cluster
can
now.
My
program
can
now
reach
beyond
the
communities
cluster
and
be
able
to
attach
to
to
the
network
or
to
have
the
services
so
in
essence,
I'm
making
my
network
more
programmable
by
combining
a
strat,
the
strengths
of
VPN,
vvpf
and
srv6.
How
do
I
do
it?
You
build
a
simple
instruction
using
evpf
Code,
whether
it's
in
encapsulation
decapsulation,
any
kind
of
packet
processing
engine.
You
link
those
segments
to
those
instructions
to
a
srv6
segment
ID.
A
Basically,
you
give
them
an
IPv6
address.
You
build
the
policies
out
of
those
out
of
those
IPv6
addresses
those
segments
which
creates
a
segment
list,
and
then
you
find
a
way
to
steer
the
traffic
in
any
way
or
form
into
that
policy
and
be
all
you
now
have
a
network
program,
simple
encapsulation,
supporting
overly
underly
and
be
able
to
do
complex
service
training
which,
in
the
in
in
realm
I,
can
actually
have
an
application.
Doing
Network
functions
at
the
application,
Level
still
being
able
to
be
tied
to
a
program
which
is
also
support.
A
My
underlay
traffic
engineering,
for
example,
Network
slicing.
If
you
look
about
the
5G
transport,
all
of
that
all
those
kind
of
things,
including
servers,
training
in
a
simple,
simple
way
using
the
the
srv6
protocol
and
the
flexibility,
psyllium
and
adpf,
give
me
that's.
Basically
the
crash
course
of
why
we
went
at
that
approach
to
support
our
networking
requirements.
I
will
pass
the
mic
to
to
Lewis.
We
can
explain
the
details
of
How
It's
been
implemented.
C
Hey
everyone
yeah,
my
name
is
Lewis
I'm,
a
Staff
software
engineer
at
I
surveillance
and
for
this
part
of
the
talk,
I'm
gonna
go
in
into
detail
on
how
we
achieved
the
POC
of
a
popular
srv6
feature,
L3
VPN,
which
was
someone
explained
already.
So
the
first
concept
that
we
needed
to
achieve
was
pod
vrf
membership.
C
The
first
map
that
we
have
is
the
vrf
map,
and
this
takes
the
pods
IP
address
and
the
destination
cider
and
Maps
it
to
a
vrf
ID.
So
this
gives
us
the
notion
of
vrf
pod
vrf
membership.
Next
we
have
a
sid
map,
which
is
a
simple
mapping
between
srv6
SIDS
and
the
vrf's,
which
the
SIDS
act
as
locators.
For
and
finally,
we
have
a
policy
map.
The
policy
map
takes
the
destination
cider
and
the
vrf,
which
we've
identified
the
traffic
belonging
to
and
then
provides
you
a
sid.
C
This
really
allows
us
to
encode
multiple
SIDS
for
a
particular
vrf
okay.
So
what
does
that
actually
look
like
on
the
data
path
side?
The
next?
The
next
concept
that
we
have
to
achieve
is
traffic
encapsulation
right.
So
the
idea
here
is
that
you
label
a
pod
and
that
pod
that
label
determines
the
vrfs
that
the
Pod
could
potentially
be
in
when
the
Pod
begins.
C
Egressing
traffic,
the
psyllium
data
path,
will
identify
which
vrf
it
would
belong
to,
and
then
it
performs
a
policy
lookup
inside
the
policy
map
which
I
just
covered
to
identify
which
Sid
should
actually
encap
the
outgoing
traffic.
So
at
the
actual
data
path
level,
what
are
we
doing
so
here?
I
have
a
packet
flow
diagram
and
I'm
going
to
start
at
the
top
left.
C
When
a
pod
sends
traffic,
we
Traverse
the
egress,
the
V
device
and
we
hit
the
host
networking
stack
from
there
we'll
do
a
fib,
lookup
and
we'll
find
the
native
device
on
the
native
device.
We
attach
a
tcegress
ebpf
program
if
you're
not
super
familiar
with
the
terms
TC
egress
or
TC
ingresses
they're,
more
or
less
cues,
which
you
can
hook
ebpf
programs
onto
as
the
packet
traverses,
either
direction
of
the
interface.
C
So
let
me
go
through
the
actual
logic
flow
of
what
happens
when
SRV
srv6
traffic
is
egressing
well
perform
a
look
up
into
the
IP
cache
map,
so
I
didn't
cover
this
map
much.
But
it's
a
mapping
of
IP
addresses
that
cilium
knows
about.
It,
gives
us
metadata.
For
instance,
it
will
tell
us
that
the
destination
address
is
it
a
IP
address
in
our
cluster
or
not?
If
it's
not
in
our
cluster,
then
we're
going
to
actually
go
the
in-cap
path
from
there.
C
The
traffic
will
do
a
vrf
lookup
on
the
traffic,
we'll
take
the
source,
IP
address
and
the
destination
address,
and
then
look
up
the
vrf
ID
again
we'll
take
the
vrf
ID
and
then
we'll
do
a
destination
along
with
the
destination
address
and
do
a
lookup
in
the
policy
map.
The
policy
map
will
provide
us
the
Sid.
It
will
in-cap
the
traffic
and
then
the
srv6
becomes
the
outer
packet.
The
srv6
Sid
becomes
the
outer
packet
and
then
the
inner
packet
is
the
original
pod
traffic.
C
So
now
the
next
step
we'd
have
to
deal
with
is
traffic
decapsulation.
So
in
this
packet
flow
diagram
we
first
have
a
native
device,
each
zero
and
incoming
to
the
psyllium
data
path.
We
see
that
there
is
a
Sid
when
the
psyllium
data
path
determines
that
this
could
be
a
sid
we're
going
to
do
a
lookup
in
the
Sid
map
and
determine
if
we
know
about
this
did,
did
we
find
this
from
the
control
plane?
Did
we
allocate
it
ourselves
if
we
did
we
Decap
the
packet?
C
So
again?
What
does
that
look
like
on
the
data
path?
Here
we
have
the
eth
device
hanging
off
external
network,
and
packet
is
now
ingressing.
Packets
are
ingressing.
Now
we
again
attach
a
ebpf
program
on
the
TC
Ingress
this
side
and
we
perform
a
sid
lookup
with
the
Sid
map.
And
then,
if
it
is
a
destination
said,
we
know
we
Decap
it,
and
then
we
send
the
packet
the
inner
packet
to
RX.
That
might
be
more
data
path.
Processing.
C
Normally
that's
just
goes
to
the
pod,
so
L3
VPN
usually
also
includes
a
bgp
aspect
and
in
this
case
psyllium
can
act
as
a
bgp
speaker
and
it
can
actually
learn
about
Upstream
PE
VPN
networks.
This
way,
so
when
psyllium
is
acting
as
a
bgp
speaker,
we
have
a
concept
called
the
bgp
control
plane
and
it
can
peer
with
Upstream
PES
and
the
advertisements.
C
Likewise,
for
the
decapsulation
side,
again
using
our
bgp
control
plane,
we're
able
to
understand
a
vrf
has
been
created
and
we
can
allocate
a
sid
for
this
vrf
and
then
we
can
advertise
that
forward
to
Upstream
pe's
when
they
learn
about
it.
They'll
understand
that
they
can
encapsulate
traffic
in
a
particular
Sid,
send
it
to
us
we'll
Decap
it
and
then
send
it
off
to
a
pod.
C
C
C
C
C
Okay,
so
we're
taking
an
initial
look
here
at
frr,
and
this
explains
the
FR
configuration
that
we
have.
This
is
pe0,
so
this
is
explaining
that
the
routing
table
of
frr
has
two
ndt4
routes.
These
ndt4
routes
tell
frr
that
when
it
sees
those
SIDS
we're
interested
in
the
vrf
zero
routing
table
here,
the
routing
entry
there
and
the
Sid
of
the
b200100
when
it
sees
that
Sid
Ingress
it's
going
to
decapsulate
it
and
send
it
off
to
vrf0.
That's
our
initial
FR
configuration.
C
C
C
Sid
allocation,
which
is
highlighted
there
and
then
the
resulting
routing
table
100
on
frr,
indicates
that
we
have
learned
about
the
allocated
Sid
and
that
when
we
see
egressing
traffic
to
10,
100
24
we're
going
to
encapsulate
it
in
the
Sid
which
psyllium
allocated
and
sent
forward
over
the
bgp
control
plane,
the
10100-24,
that's
our
pod
cider.
So
that's
going
to
get
the
traffic
back.
C
C
And
okay,
so
now
we
have
an
echo
request
and
an
echo
reply.
What
we
can
see
here
is
the
outer
IP
address
of
the
request.
You
see
our
allocated
Sid
and
you
see
that
the
destination
is
the
vrf0
on
frr
side,
the
Sid,
which
locates
that
you
can
see
that
it
is
an
encapsulated
packet
and
the
inner
IPS
are
our
pod.
Ip
address
101097
and
the
destination
is
vrf
zeros
10301.
C
C
Likewise,
the
return
traffic
is
now
coming
from
our
host,
our
frr
node
at
B,
one
colon
colon
one
and
it's
destined
to
our
srv6
Sid
on
the
psyllium
side.
So,
ideally,
what
happens
is
that
when
psyllium
sees
this
Sid,
it's
going
its
data
path
is
going
to
be
set
up
in
such
a
way
that
it
will
Decap
that
Sid
pull
out
the
inner
IP
address
of
101097
and
deliver
that
forward
to
pod
one,
and
that
is
currently
occurring.
C
Demonstrated
there
and
then
the
last
interesting
thing
to
note
is
just
looking
at
CE
ones
right,
so
ce1
is
all
the
all
the
way
on
the
other
side
of
the
customers
Network
it's
past
the
PES
and
it's
over
hanging
off
frrs
vrf,
zero
interface.
What
we
see
here
is
the
encapsulated
traffic
right.
So
this
is
just
a
a
proof
saying:
okay,
yeah
we
did
the
decapsulation,
we
don't
see
any
IPv6
addresses.
We
basically
have
end-to-end
L3
VPN
as
a
proof
of
concept
working,
so
yeah.
C
A
B
A
B
End-To-End
observability,
but
what's
happening
between
ebpf
and
srv6,
is
Hubble
taking
srv6
data,
or
do
you
sort
of
do
debugging
between
evpf
and
srv6,
if
you're
trying
to
figure
out
end
to
end
or
what
was
the
state
of
a
route
at
one
point
or
if
which
labels
were
used,
which
V6
addresses
were
used
for?
What
at
one
point
like?
How
do
you
think
about
that?
End-To-End
operability,
debugging?
B
C
Currently,
I
don't
think
there
is
a
integration
with
this
POC
and
Hubble
currently,
but
the
traffic
flow
would
follow
the
normal
data
path
flow.
So,
given
that
we
can
see
the
traffic
and
pull
out
labeled
information,
then
we
do
have
the
control
plane,
which
has
all
the
smarts
of
what's
going
on
right.
So
we
should
be
able
to
pick
apart
that
information
and
be
intelligent
about
how
we
show
the
traffic
observation.
B
Right,
you
know
so
so
going
a
few
days
before,
not
just
the
real
time.
You
know
thinking
about
that,
but
how
you
would
look
at
what
the
state
was
a
few
days
ago,
if
you're
trying
to
debug
a
ticket
or
some
question
about
connectivity,
you
know
something
that
might
not
have
been
working
but
didn't
wasn't
detected
in
real
time.
Yeah.
A
So
you,
you
will
add
the
information
out
of
a
bull
for
the
data
path,
the
rotting
plane.
You
can
actually
get
the
stream
Telemetry
or
the
stream
data
about
the
bgp
routes.
You
can
get
time,
stamping
on
how
you
your
state
of
your
route,
work
I,
think
it's
not
part
of
bubble,
but
the
correlation
can
be
done
quite
quite
easily.