►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Developing Edge with Kubernetes - Dejan Bosanac & Ted Ross, Red Hat
While its original intent was to solve cloud-centric architecture challenges, Kubernetes can be used as a platform for developing Edge and IoT applications. The session will take a look at this topic from the developer's point of view. It'll focus on special aspects of developing services for IoT and Edge environments, like autonomy, locality and secure communication to name some. What Kubernetes features can be used to address IoT and Edge aspects of developing microservices? What other projects in the ecosystem can developers use to achieve their goals? We'll try to answer those questions and demonstrate the flexibility of the platform by demoing an Edge application consisting of services deployed into various environments (like Edge clusters and Edge nodes) working autonomously and communicating securely with each others and the central cloud services.
https://sched.co/UaXg
A
A
Environment
people
usually
talk
about
infrastructure
and
platform,
and
everything
like
that.
But
today
we
want
to
focus
more
on
a
developer
experience
and
developer
angle
or
two
to
the
edge
computing.
So
just
just
a
quick
intro
intro
to
the
topic
itself.
So
we
want
you,
we
don't
want
to
waste
too
much
time
with
time
on
it.
So
to
me,
edge
is,
is
a
extension
of
the
cloud.
So
it's
not
something
that
that
you
know,
depending
on
who
you
talk
to.
A
There
are
different
things
that
we
need
to
do
to
consider,
for
example,
in
the
provider,
edge
environment,
network
bandwidth
and
and
network
latency,
usually
is
not
a
problem,
while
the
more
you
go
to
the
left
of
this
diagram,
you
will
see
that
the
resource
is
becoming
more
constraint
in
the
in
the
compute
sense,
but
also
in
the
in
the
network
sense.
So
so
the
net
repair,
then
we
become
more
expensive,
Andrew
and
less
really
reliable
and
that
some
of
the
things
that
that
we
need
to
that.
A
We
need
to
think
about
when
we
try
to
develop
services
for
the
edge.
Also,
this
diagram
trying
trying
to
put
put
a
little
bit
more
sensing
into
why
we
want
to
do
edge
computing
and
what?
What
is
the
cost
in
trying
to
do
that
and
like
with
everything
in
engineering
there's
there's
the
trade
of
trade-offs,
always
right.
A
So
the
closer
we
put
our
computer's
resources
to
the
edge
we
get
some
benefits
of
less
route,
relying
on
the
Bedouin
bandwidth,
increasing
the
latency
of
data
processing
and
achieving
greater
rezulin,
see
and
of
our
offer
edge
side
right.
But
we
lose
economic
economies
of
the
scales,
which
is
one
of
the
main
reasons
why
we
do
our
cloud
computing
in
the
first
place
right.
So
so
the
the
management
overhead
and
the
cost
of
the
control
plane
is
becoming
more
expensive.
A
The
more
we
go
to
the
to
the
to
the
edge
so
having
all
that
when
we
think
about
edge
computing,
they're,
basically,
three
main
categorical
of
challenges.
There,
our
first
one,
is
infrastructure
challenge,
so
so
how
we
actually
physically
put
our
or
compute
nodes
on
the
edge
edge
sides
right.
So
it's
not
secure
data
centers,
which
we
don't
have
a
lot
of
experienced,
IT
personnel
to
do
to
rule
it
and
the
other
level
is
a
control,
plane
challenges.
A
So
once
we
have
this
compute
computing
infrastructure
there,
how
we
actually
put
our
root
loads
on
those
edge
nodes
right
and
and
finally,
once
our
our
services
are
running
on
the
edge
sides
and
on
the
cloud
the
question
is:
is
you
know
how
we
develop
applications
for
these
distributed
environment,
how
these
services
communicate
with
the
cloud
and
with
each
other?
And
today,
in
this
talk,
we
wanna
focus
mostly
on
the
third
third
part
of
these.
A
These
challenges,
so
for
all
four
for
all
applications.
And-And-And-And
service
is
running
running
on
the
edge
there.
Their
two
main
main
challenges
remaining.
As
you
said,
resources
are
always
gonna,
be
a
problem
because
once
we're
on
the
edge,
we
lose
the
alto
scaling
the
magic
of
our
centralized
central,
centralized
clouds
well,
meaning
that
we
can't
easily
add
more
nodes
and
and
more
more
computing
power
as
we
need
it
right.
A
So
we
need
to
be
careful
of
of
how
we,
how
we
prioritize
our
services
and
and
how,
how
we
do
a
capacity
planning
for
for
these
sites
and,
as
I
already
told,
told
soul,
so
network
is
usually
limited,
usually
expensive,
and
and
we
need
to
make
sure
that
that
that
the
services
are
being
able
to
communicate
with
each
other.
So
when
you
take
all
this
in
consideration,
there
are
a
couple
of
things
that
it
is
always
always.
A
That
we
need
to
take
take
care
kind
of
when
we're
thinking
about
the
tip,
deploying
our
micro
services
on
the
edge.
So
first
is
sorry
about
it.
Deployment
itself
other
is,
is
the
node
resources.
Third
is:
how
are
we
going
to
do
to
the
communication
and
and
finally,
the
the
security
is
a
big
concern,
because
in
some
of
these
use
cases
the
edge
sites
are
are
running
outside
of
the
physically
secured
data
centers
so
usually
in
in
in
on
the
infrastructure?
That's
that's
unattended,
so
so
we
need
to
have
our
tools
as
well.
A
That
will
provide
us
a
proper
guarantees
that
we
are
running
actually
the
services
that
that
we
that
we
think
we
want.
We
want
to
run
right.
So,
having
all
this
in
mind,
we
need
to
figure
out
how
we're
gonna,
how
we
gonna
use
existing
toolkit
that
we
have
in
the
cloud
native
space
to
actually
do
this
job
and
but
there's
a
lot
of
hype
in
in
each
computer.
These
days
and
and
everything
is
labeled
as
a
Zed.
But
for
me
something
too
to
be
called
a
edge.
A
A
So
first
thing
that
comes
to
mind
is
how
we,
how
we
solve
the
deployment
problem
and
and
that's
where
get
get
ups
best
parties
practices
can
play
the
role
right
first.
First
of
all,
we
want
to
have
our
environments
to
be
operated
by
by
by
people
that
are
not
highly
trained,
IT
specialists
right,
so
it
needs
to
be
as
simple
as
possible
and
and
these
websites
are
you
running
in
the
private
networks.
So
so
we
don't.
A
So
all
our
gamal
definitions
and
all
our
secrets
should
be
hosted
in
the
in
the
gate
and
by
using
that
we
can
use
the
same
developer
developer
workflow
by
by
you
know,
doing
the
poor
request,
branching
and
caed
testing
to
manage
these.
These
this
configuration
once
we
have
that
we
can
here,
when
some
kind
of
the
operator
running
running
on
the
air
site,
basically
pulling
the
changes
from
the
get
in
and
applying
up
applying
those
those
environment.
A
What
we
have
in
this
toolkit
today,
I
think,
should
should
work
pretty
well
in
most
of
the
use
cases.
So
the
flux
is
pretty
pure
popular
get
get-ups
operator
and
we
can
use
our.
You
know
the
tools
that
we
use
for
other
stuff
like
like
Helmand
fabricate,
to
actually
create
those
environment
configurations
and
or
once
we
want
to
store
some
secrets
in
the
gate.
A
I
think
the
sealed
secrets
project
could
be
a
good,
the
good
fit
to
keep
keep
encrypted
versions
versions
of
the
secrets
in
energy
in
the
repositories
which
can
be
encrypted
only
by
the
by
the
by
the
proper
operator
on
the
on
the
edge
side.
So
the
others.
The
other
thing
is
how
we're
going
to
deal
with
with
with
resource
constraints.
So
we
want
to
make
sure
that
that
that
our
our
services
are
as
small
as
possible
and
using
using
the
the
resources
and
CPU
and
memory
as
less
as
possible.
A
So
I
know
this
is
not
not
a
typical
Java
crowd,
but
we,
if
we
want
to
be
successful
in
developing
edge,
we
need
to
we
need
to
have.
We
need
to
have
a
solution
for,
for,
for
you
know
all
the
languages
and
attract
developers
for
all
the
environments.
So
one
of
these
examples
is
the
power
course
project
which
provides
a
cloud
native
native
Java
using
growl
vm
to
come
to.
A
Basically,
compile
our
applications
to
native
applications,
which
drastically
reduces
reduces
the
the
memory
footprint
and
the
startup
time
of
of
these
micro
services,
and
it's
good
in
a
cloud
native
way.
But,
as
I
said,
like
some
things
are
not
even
more
emphasized
for
the
edge
of
environments,
because
because
we
don't
have
unlimited
resources
there
and
beyond
that.
Even
if
our
resources
are
are
small
and
efficient.
There's
always
a
problem
of
of
not
having
enough
of
them
on
on
the
edge
and
that's
where
kubernetes
provide
us
with
its
priorities.
A
A
B
Next
day
on,
I'm
gonna
actually
switch
gears
a
little
bit
and
take
a
deeper
dive
into
the
into
the
question
of
networking
and
cluster
interconnect,
and
just
a
quick
word
about
the
state
of
tcp/ip
and
the
Internet
and
the
weight
you
know
the
Weig
to
this
serves
us,
as
you
know,
the
way
that
the
internet
is
constructed.
It's
and
the
way
that
it
is
able
to
scale
so
large
is
that
it's
divided
into
public
cloud,
public
sub
networks
and
private
sub
networks,
and
that's
enabled
by
network
address
translation
and
NAT
routers.
B
So
there
are
millions
upon
hundreds
of
millions
of
private
clouds,
of
which
I'm
sorry
private
networks,
of
which
you
know
they
may
be
sharing
many
of
the
same
addresses
and
the
orientation
of
this
network
is
vertical,
where
you
have
server
clients
in
those
private
networks
and
big
expensive
servers
up
in
the
public
network.
So
the
internet
is
built
for
a
client-server
world,
but
the
thing
that
we're
going
into
right
now,
the
more
modern
development
techniques
that
we've
been
talking
about.
You
know
hybrid
cloud,
micro
services,
architecture,
server,
list
etc
is
distinctly,
not
client-server.
B
We
want
to
be
able
to
put
servers
services
anywhere
and
we'd
like
them
to
be
able
to
talk
to
each
other.
So
you
know
in
this.
In
this
illustration,
I've
got
two
hosts
in
private
sub
networks,
both
of
which
share
the
address
10.10
dot
v.
Clearly,
these
two
hosts
cannot
talk
to
each
other
or
address
each
other
directly.
B
They
can
interact
through
you
know,
chat
or
maybe
Google
Docs
or
some
such
thing,
but
it's
going
to
be
the
software
that
enables
that
is
actually
designed
in
such
a
way
that
it's
talking
using
some
sort
of
protocol
going
through
some
public
service.
So
this
is
exacerbated
in
the
world
of
edge
computing,
where
we
may
have
lots
of
small
private
subnets
and
we
really
want
to
have
a
way
of
giving
developers
the
ability
interact
more
directly
through
that.
So
with
that
little
piece
of
background
I'm
gonna
we're
gonna
introduce
an
idea
that
we're
working
on.
B
This
is
a
new
thing
that
we
are
working
on
in
actually
in
an
open
source
project
called
scuppered
IO
and
it's
the
virtual
application
network,
where
we
are
going
to
talk
about
services.
So
here
I've
got
a
set
of
services
and
I've
drawn
one
client,
but
of
course
there
could
be
many
instead
of
dealing
with.
You
know,
IP
addresses,
because
we
can't
address
those
services
using
IP.
We
want
to
address
them
in
such
in
a
more
application
specific
way.
B
So
in
this
case
and
this
particular
network
I've
introduced
a
router
into
each
of
my
locations,
think
of
this
as
networks
or
clouds
or
kubernetes
clusters
that
may
be
in
different
locations
and
tcp/ip
is
used
for
what
it's
good,
for,
which
is
the
connectivity.
The
vertical
connectivity
getting
everything
hooked
together,
but
the
application
network
allows
me
to
talk
about
these
services
with
a
name
called
service
and
then
allowing
me
then
to
distribute
my
application
traffic.
You
know
if
that
service
is
anycast
load
balance
address,
then
the
load
from
the
client.
B
So
some
of
the
implications
and
I'll
do
this
quickly,
because
time
is
of
the
essence.
We've
got
some
security
implications
when
you
have
a
network
where
you're
addressing
things
at
that
level
where
access
control
is
expressed
in
terms
of
the
application
or
the
business.
Rather,
the
business
logic
I'll
talk
to
talk
about
as
the
business
resolution.
The
network
itself
is
locked
down.
The
connections
between
those
routers
are
mutual
TLS
and
best
practices.
We
recommend
is
actually
using
a
dedicated
certificate
authority
so
that
those
are
very
locked
down
and
the
cross
cluster
applications.
B
So
if
I've
got
a
service
in
one
cluster
being
accessed
by
a
client
in
a
different
cluster,
there's
no
exposure
that
that
service
is
not
exposed
using
kubernetes,
networking
or
openshift
routes
or
anything
of
the
like
it's
tunneled.
So
the
public
exposure
is
then
limited.
Only
to
the
ingress
is
that
you
create
in
your
application
for
getting
things
in
from
the
outside
world.
There's
also
the
notion
of
a
trusted
or
an
untrusted
edge,
where
we
can
actually
say
that
that
router
and
the
edge
well,
we
don't
really
trust
it.
B
So
we're
going
to
apply
our
our
policy
actually,
where
the
router
hooks
into
the
network,
the
edge
to
fix
the
network
and
that
actually
simplifies
the
development
of
the
code
on
that
edge
itself,
because
it's
being
enforced
at
a
higher
level
and
then
from
a
management
and
monitoring.
You
know
the
metrics
are
also
collected
at
that
same
resolution,
so
you
have
counters
on
the
actual
services
and
the
actual
business
objects,
and
this
processes
that
you
are
running.
B
B
The
emphasis
here
is
to
make
it
easy
to
set
up
networks.
We
want
to
be
able
to
set
up
networks,
not
requiring
any
advanced
networking.
You
know
it
software-defined,
networking,
no
VPNs
or
tunnels
or
IPSec
firewall
rules.
A
developer
need
not
have
elevated
or
admin
privileges,
a
cluster
access
and
there's
no
issue
with
overlapping
addresses
address
space.
So
it's
a
very
applicable
to
the
edge
computing
world.
There's
no
single
point
of
failure:
it
uses
a
redundant
topology
and
it
supports
multiple
protocols
for
your
applications,
including
GG,
RPC
or
HTTP,
TCP,
messaging
protocols,
etc.
B
You
know
it's
particularly
if
you
have
like
satellites
going
over
buoys
and
oceans
and
things
like
that,
where
you
are
disconnected
for
some
for
some
time:
I'm
edge
integration,
here's
a
case
where
I've
got
you
know
four
different
cloud
locations,
two
of
them
are
in
private
edges,
and
two
of
them
are
in
public
clouds
and
I
have
a
redundant
topology
set
up
in
this
case.
In
this
picture,
I've
actually
got
arrow
heads.
B
The
arrow
heads
are
only
significant
in
the
direction
of
the
establishment
of
connectivity,
so
the
edges
connect
upward
through
their
firewalls
to
the
public
cloud,
because
it
can
do
that.
We
can't
connect
backwards.
Without
doing
you
know
special
tunneling,
but
once
the
connections
are
established,
the
arrow
heads
are
no
longer
significant
and
traffic
can
flow
and
by
you
know,
both
directions
over
these
lines
so
that
these
services
and
these
edge
sites
can
then
talk
to
each
other
seamlessly.
B
Here's
basically
the
same
picture,
but
in
this
case
I've
got
clients
in
both
edge
sites
using
their
local
services.
So
here's
a
case
where
there
we're
enforcing
or
using
locality,
clients
use
their
local
services
where
possible,
but
if
those
services
either
go
down
or
they
become
backed
up
and
the
latency
goes
up
and
this
network
can
spill
over
to
the
services
in
the
public
cloud
and
it
can
even
spill
over
to
the
services
in
other
edge
sites
and
and
also
in
a
more
messaging
oriented
use
case.
B
Where
we've
got
you
know,
we've
got
a
network
where
we
absolutely
must
be
able
to
produce.
We've
got
a
very
high
degree
of
requirement
for
availability
for
producing
messages,
and
here's
where
you
know
where
you
can
actually
have
multiple
sites.
You
can
have
routers
and
multiple
brokers
running
where
you
know
the
probability
of
having
one
up
that
can
access
accept.
Your
deliveries
is
increased,
so
I've
left
a
little
bit
time
and
we're
going
to
do
a
demonstration
and
I'm
gonna
actually
show
you
a
picture
of
this
demo
before
we
start
it.
B
So
this
is
just
a
quick
picture
of
what
it
is
that
we're
going
to
eat
attempting
live
here.
We've
got
a
core
network,
that's
actually
AWS.
It's
running
on
the
u.s.
West
availability
zone.
The
one
on
the
right
is
a
private
cloud.
It's
actually
running
on
a
server
in
raleigh-durham
someplace
and
the
one
on
the
left
is
going
to
be
on
my
laptop.
So
let's
see
how
this
goes.
B
These
are
already
connected,
but
on
my
laptop
I
have
basically
and
what
amounts
to
an
empty
namespace
right
now,
there's
just
nothing
in
here,
so
I'm
gonna
use
the
scupper
tool,
I'm
just
gonna
say
scupper
in
it,
and
this
says
you
know
enable
scupper
for
my
for
my
namespace
and
when
I
created
the
one
in
the
public
cloud,
I
actually
generated
a
connection
token
and
I
have
made
that
connection
token
available
to
this
laptop
by
whatever
means
is
necessary.
They
both
happen
to
have
been
run
from
the
same
one.
So
it's
in
my
file
system.
B
So
now,
if
I
do
a
scupper
connect,
do
it
was
us
West
and
that's
the
connection.
Token
token
is
a
little
yellow
file
that
contains
the
information
needed
to
set
up
this
connection
to
the
network.
So
this,
if
it
all
works
well,
is
going
to
create,
is
going
to
connect
my
namespace
it's
going
to
introduce
a
router
and
there
it
is
on
my
laptop's
namespace
and
it's
going
to
make
a
connection
to.
B
B
So
there's
a
couple
of
services
that
were
created
for
the
purpose
of
doing
this
networking
now
I'm
gonna
I've
got
a
tunnel
an
SSH
tunnel
over
to
my
rdu
server
and
I
can
also
do
the
same
thing
here.
Keep
control
get
services
but
I
have
a
service
over
here.
That's
that's!
That's
my
edge
service.
This
is
part
of
my
application.
Now
I
didn't
even
mention.
This
I
should
have
probably
said
this,
but
we
were
we're
showing
what
is
like
a
electric
car
charger
charging
station
where
these
stations
are
all
edge
locations.
B
They
have
certain
amount
of
resources
and
they're
being
managed
by
a
headquarters.
So
so
this
edge
service
is
the
one
that
is
providing
this
this
this
charging
service.
Now,
if
I
were
to
do
a
you,
see,
I've
got
right
here,
so
I'm
gonna
do
a
curl,
it's
just
a
straight
up:
HTTP,
REST
type
service
and
since
I'm
on
that
same
cluster
I'm
cheating
here
in
that
I'm,
not
using
an
ingress
or
a
route,
but
I've
got
the
address
available,
and
this
is
the
edge
service
address
on
port
8080
and
I'm
gonna.
B
Ask
it
you
know:
what's
the
status
of
my
slot,
so
I've
got
ten
slots
and
they're
all
free
right
now,
so
I
can
then
go
in
here
and
post
now,
I'm,
not
doing
anything
fancy
here.
This
is
this
is
just
one
one
cluster
with
a
little
simple
web
service.
That's
running
and
I'm
gonna
ask
it
to
reserve
slot
four
and
see
if
I
typed
it
right
so
slot
four
is
now
reserved
for
you.
B
Sons
could
be
done
remotely
because
maybe
the
station
on
that
right
now
is
all
filled
and
I
want
to
reserve
something
nearby.
So
back
here
on,
my
laptop
I
would
like
to
be
able
to
access
that
service.
But
it's
not
running
here.
It's
running
in
raleigh-durham
on
some
remote
thing.
You
know
remote
network
that
I
don't
have
access
to.
So
what
I'm
going
to
do
over
here
is
I
am
going
to
annotate
if
I'm
lucky
I've
got
it
in
my
okay,
so
I'm
going
to
do
a
cute.
B
Keep
control,
annotate,
I'm,
gonna
annotate,
that
service
with
ask
upper
specific
annotation.
Now
I'm
gonna
just
tell
her
I
want
to
proxy
that
by
HTTP.
Now,
when
I
come
back
to
my
laptop
and
I,
have
a
look
at
the
services?
I
have
a
service
of
the
same
name:
edge
service,
rdu,
zero.
This
is
the
same.
B
There's
no
pot
here
running
the
pot
is
running
over
are
to
you,
but
now
I
have
a
proxy
service
that
is
located
elsewhere
in
the
network
that
allows
me
to
have
access
to
it,
and
this
is
what
the
the
application
layer
networking
is
doing,
so
that
the
address
of
the
service
is
actually
edge.
Service
are
to
use
zero
and,
if
I
curl
to
that
see,
if
I
can
type
coherently
get
HTTP
and
I'll
grab
that
address.
B
All
right,
so
let
me
see
if
I
can
get
my
now
you'll
notice.
Also
in
this,
this
is
actually
the
router
console,
and
it
has.
It
has
a
few
little
visualizations
for
traffic,
so
it'll
out
it
actually
just
showing
that
flow
of
that
traffic
when
it
occurs.
So
since
I'm
running
this
thing
out
of
the
laptop
and
it's
being
serviced
over
rdu,
this
is
actually
showing
that
well
the
network,
since
there's
no
connection
between
the
two.
It
has
to
actually
go
by
our
public
headquarters.
To
get
there.
A
A
We
have
we
handle
just
a
little
bit
of
time.
Just
just
to
you
know
scratch
the
surface
of
all
this
all
this
discussion.
But
but
you
know,
if
you
want
to
try
to
start
developing
applications,
it's
not
all
about
infrastructure,
it's
not
all
about
bathroom.
We
need
to
start,
you
know,
solving
problems
for
developers
and
how
we're
gonna
address
those
challenges.
So
we
try
to
provide
some
of
the
guidance
for
four
for
three
free
or
aspects
also
how
we're
gonna
deploy
those
micro
services
to
the
edge.
A
So
you
use
your
some
of
the
get-ups
existing
tools.
Try
always
to
make
sure
that
your
service
sizes
are
as
small
as
possible
for
for
the
edge
environments,
so
try
to
find
the
appropriate
framework
for
the
language
are
using
and
try
to
to
use
or
to
use
the
existing
tool
kit
for
prioritization
kinetise
gives
us.
And
finally,
you
know
try
to
think
a
lot
about
the
networking
considerations.
A
You
know,
how
are
we
going
to
connect
those
those
services
and
and
in
the
you
know
how
we're
gonna
provide
like
like
a
uniform,
uniform
experience
for
developing
applications
in
the
edge
so
yeah?
We
don't
have
a
much
more
time
so
there's
a
Cuban
ethnicity,
an
edge
working
group
that
discuss
all
the
things
related
to
to
you
to
to
these
kind
of
topics.
C
B
B
Observing
and
instrumenting
the
you
know
the
connection
between
HTTP
based
services
and
there's
there's
some
overlap.
Obviously,
between
what
we're
doing
and
what
they're
doing
our
our
focus
is
really
it's
a
it's.
A
different
approach
to
multi
cluster
communication
and
our
focus
is
right.
Now
is
really
on
just
the
communication
aspect,
because
we
believe
that
there's
a
lot
of
difficulty
involved
in
non
star
non
full
mesh
types
of
topologies
when
you're
talking
about
multi
cluster,
particularly
when
there
are
private
private
networks
involved.
B
D
I
have
a
follow-up
Shin
do
what
the
previous
question
was:
it's
not
about
East
you,
but
there
is
a
network
mesh
which
also
has
at
least
some
of
the
use
cases
similar
where
you
can
connect
I,
think
they
use
obvious
over
that
you
can
have
any
kind
of
connectivity
within
private
network
or
outbound
or
whatever.
So
just
the
question
is:
do
you
use
that?
Do
you
plan
to
use
that
or
how
do
you
relate
to
that.
B
B
A
E
F
B
Yeah,
it's
using
tcp/ip
to
do
that
vertical.
You
know
to
a
router
to
a
router
connection,
but
as
long
as
there
is
a
total
connection,
the
paths
cross
network
what's
actually
happening
is
these
routers
are
doing,
are
doing,
link
state
routing
at
the
application
level.
So
it
actually
is
using
a
no
SPF
like
protocol
to
determine
what
the
topology
is
and
what's
the
best
paths
are,
so
you
can
have
redundancy
it'll
react
to
changes
and.