►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Clusters All the Way Down: Crazy Multi-cluster Topologies - Matt Caulfield, Oort
What happens when your application outgrows a single cluster? How do you scale from one cluster to dozens without sacrificing high availability or performance? How do you structure your application to actually take advantage of multi-cluster environments? In this session, you'll learn about multi-cluster topologies: different options for structuring your application architecture to naturally benefit from multiple Kubernetes environments. Whether you're building a new application from scratch or scaling up an existing deployment, this session will equip you with best practices, rules of thumb, and the right mental model for successfully using Multi-Cluster Kubernetes.
To Learn More: https://sched.co/GrWW
A
Good
afternoon,
everybody
is
the
last
session
of
the
day
on
the
Multi
cluster
track,
so
anybody
been
in
here
for
the
whole
day
doing
all
the
multi
cluster
sessions.
No,
all
right!
That's
fine,
who
here
has
more
than
one
cluster
in
production:
okay,
quite
a
few!
What
if
you,
who
has
more
than
five
all
right,
10
fewer
50,
cool,
100,
Wow,
all
right,
so
whether
you
raised
your
hands
at
100
or
you
didn't
raise
your
hand
at
all.
This
talk
is
meant
for
you.
This
is
all
about.
A
You
know
how
we
deal
with
multiple
clusters
and
scaling
an
application
from
one
cluster
to
two
to
ten
to
a
hundred
and
all
the
way
up
to
a
thousand
and
that's
a
lot
of
ground
to
cover
in
half
an
hour.
Essentially
so
we're
gonna
talk
at
a
pretty
high
level.
There
are
a
lot
of
talks
today
in
this
room
that
have
gone
into
the
lower
levels
about
okay.
How
do
we
do
networking
between
two
clusters?
A
I
want
to
talk
from
an
architecture
perspective
if
you
are
building
an
application
that
requires
more
than
one
cluster
either
today
or
in
the
near
future.
Then
I'm
hoping
I
can
answer
some
questions
for
you,
or
at
least
tell
you
what
questions
to
ask.
So
if
you
sign
up
for
this
talk
last
week,
you
probably
saw
Cisco
on
on
the
schedule.
I'm
gonna,
Cisco
anymore,
I've
been
there
for
10
years
and
I'm.
A
Thank
you
I'm
now
at
ort
ort,
it's
short
for
Oort
cloud,
a
co-founder
and
CEO
on
with
my
my
good
friend
Adam
in
the
back
of
the
room,
we're
building
solutions
around
cloud
native
edge
computing.
But
this
is
not
a
sales
pitch
and,
although
I
am
the
CEO,
I
will
refrain
from
going
into
details.
If
you
want
to
hear
about
ort
feel
free
to
come
up
after
I'll
put
the
email
address
on
the
last
slide.
I
would
love
to
talk
you
about
it
all
right.
A
So,
what's
the
deal
with
the
name
clusters
all
the
way
down,
so
that
comes
from
Turtles
all
the
way
down.
If
you
imagine
that
your
application
is
the
world
and
the
turtle
is
the
cluster
in
your
application,
assuming
the
cluster,
most
of
us
just
sit
on
one
cluster
today,
but
if
you
imagine
the
need
for
many
clusters,
you
can
always
imagine.
A
A
A
My
wife
sent
me
a
message
saying
that
she
had
taken
the
kids
to
go
visit,
a
neighbor's
house
and
they
were
giving
away
a
red-eared
slider,
which
is
an
aquatic
turtle,
so
that
was
two
years
ago
fast
forward
to
a
couple
weeks
ago,
I
come
home
early
from
work,
walk
into
the
kitchen
and
see
the
picture
that
you
see
before
you,
which
is
nanny
on
the
floor.
She's
an
aquatic
turtle.
A
Remember
this
so
she's
supposed
to
be
a
tank
of
water,
all
dried
out
on
the
floor
kind
of
limping
along,
and
the
hope
is
that
we
get
to
somewhere
in
between
your
cluster
of
limping
along
supporting
one
application
and
your
turtle
cruising
through
space
on
a
pile
of
other
turtles.
So
somewhere
in
there.
You
know
we
hope
to
find
some
kind
of
balance.
A
Today.
Kubernetes
is
great
for
cluster
management.
We
all
know
that,
but
we
all
know
that
Cooper
needs
itself
is
really
meant
for
multi
cluster
management.
It's
it's
a
great
tool
for
cluster
management
and
does
one
thing
and
does
one
thing
really
well?
Well,
maybe
more
than
one
thing,
but
for
multi
cluster
there,
a
bunch
of
other
issues,
whether
you're
at
two
clusters
or
one
hundred
clusters.
A
A
Why
there's
a
kind
of
a
howl
response?
It
is
it
scalability
just
having
two
data
centers
nationally,
actually
give
you
any
additional
scalability
and
I
pass
it
probably
not.
The
reason
why
you're
doing
is
that
is,
that
is
for
catastrophic
failure
right.
It's
for
high
availability.
You
finally
got
that
customer
who's
gonna
write
you
the
check,
but
before
they
do
that
they
need
high
availability,
enterprise-grade
availability,
great.
A
So
some
of
you
might
be
angry
at
this
point
because
I'm
conflating
clusters
with
regions
or
cells-
that's
fine.
You
can
be
angry,
but
we'll
come
back
to
this
later
so
for
now
bear
with
me
the
assumption
that
one
cluster
from
one
region-
okay,
high
availability,
one
the
two:
what
are
the
issues?
So
what
is
so
hard
about
this?
Why
can't
we
just
do
this
out
of
the
box?
Well,
for
one
your
application
probably
wasn't
designed
for
two
clusters.
It
was
designed
for
one
region
right,
that's
what
you
did
out
of
the
box.
A
That
was
easy.
What
do
you
do
now?
Knowing
you
have
two
clusters
which
parts
of
your
application
are
going
to
run
on
us
East
and
which
parts
are
going
to
run
on
us
west?
So
one
strategy
you
could
take-
and
this
is
a
pretty
basic
strategy-
might
be
just
fine
from
going
from
one
to
two
clusters.
Is
cloning
everything
same
microservices
that
you
have
in
US
East
can
run
just
as
well
in
u.s.
West,
and
so
you
have.
You
know
something
that
looks
like
this
and
for
one
to
two.
A
You
know
this
person
perfectly
reasonable
strategy
and
we'll
see
where
that
breaks
down
as
we
as
we
go
along,
but
but
for
now
let's
go
with
it
issue
number
two:
when
you
go
to
two
clusters
is
okay:
now
your
data
isn't
just
in
one
data
center.
It's
actually
in
multiple
data
centers.
So
are
you
putting
every
database
record
in
both?
Are
you
doing
full
replication?
Are
you
using
an
object
store
using
a
database,
so
this
question
about?
A
A
If
you're
running
in
a
cloud
platform
AWS,
you
can
rely
on
the
AWS
toolbox,
so
each
one
of
these
services
like
s3,
for
example,
if
you,
google,
s3
geo
redundancy
you'll,
get
answers
for
different
options
for
replicating
data
between
region,
same
thing
for
dynamo,
DB
or
any
of
these
other
ones.
There's
always
some
strategy
there.
That
needs
to
be
investigated
then
same
thing.
For
Google,
you
know
they
have
a
similar
offering
where
you
can
use
their
cloud
storage
and
then
rely
on
them
for
to
your
redundancy.
A
So
it's
really
an
architectural
choice
about
okay,
are
you
going
to
depend
on
a
cloud
provider
or
you're?
Gonna
pretend
depend
on
the
C&C,
f
and
and
and
their
toolbox
so
which
toolbox
you're
gonna
pick
that's
first
and
then
second,
how
are
you
gonna
use
it
for
geo
redundancy
and
your
answer
for
one.
A
The
two
data
centers
might
be
different
from
two
to
ten,
all
right,
moving,
quite
a
long
load
balancing,
so
some
people
might
argue
that
load,
balancing,
isn't
aprama
here,
you're,
not
trying
to
balance
load
between
data,
centers
you're
really
trying
to
route
traffic
to
the
closest
data
center.
So
this
is
an
issue
now
for
your
multi
cluster
application.
So
how
is
traffic
shared
across
multiple
clusters?
How
do
you,
if
I'm
standing
here
in
Seattle?
How
does
my
laptop
and
know
to
hit
us
west
rather
than
us
east?
A
And
you
know
again:
we
can
reach
into
the
AWS
toolbox
and
pull
out
their
global
load
balancer
or
reach
into
the
the
Google
loop
tool
box
and
pull
out
their
load
balancer
or
you
can
build
something
custom.
You
know
built
on
core
D&S
or
you
could
go,
buy
a
solution
from
one
of
the
global
load.
Balancing
companies,
the
the
opposites
are
out
there,
but
suffice
to
say
you
need
an
answer
for
all
three
of
these
problems,
these
issues
to
get
the
two
clusters,
and
are
these
the
only
three
issues?
A
Then
you
solve
no,
of
course
not,
but
these
are
the
three
biggest
things.
So
how
do
you
design
your
application
topology?
How
do
you
break
down
your
microservices
and
decide
what
runs
where
and
how
do
you
decide
how
your
data
is
shared
and
then,
third,
how
do
you
decide
where
your
traffic
is
shared?
So
just
you
know
really
basic
building
blocks
from
going
to
from
one
or
two?
No,
let's
make
it
harder.
How
do
you
go
from
two
clusters,
the
tenth
and
before
we
do
that
before
the?
How
come
back
to
the?
Why?
A
Why
are
we
going
from
two
to
ten
clusters?
What's
the
point?
Well,
usually
when
this
is
happening,
it's
for
a
reason
that
your
your
applications
going
global
right.
You,
your
application,
popular
outside
of
your
country.
You
need
to
have
global
presence,
and
you
might
imagine
that
your
product
owner
comes
along
and
says
hey.
You
know
our
applications
operating
great
in
Seattle,
but
the
latency
and
Asia
is
killing
us
and
the
folks
in
Europe
hate
having
their
data
in
American
locations.
A
So
we
got
to
do
something
about
this,
so
the
two
really
driving
forces
here
are
really
user
experience
to
go
from
two
to
ten,
its
user
experience
and
its
security
and
and
it's
data
sovereignty.
So
let's
go
back
to
our
issues.
What
is
the
application?
Topology
look
like?
Well?
Could
you
clone
everything
again,
you
could
clone
everything
again.
Should
you
have
a
full
Metro
application
where
you
share
every
single
database
record
and
every
single
object
in
every
location?
The
obvious
answer
to
these
questions
is
no.
A
So
what
is
the
answer,
and
so
I
would
pause
it,
and
this
is
the
first
concept
that
I'm
bringing
in
is
a
hierarchical
topology
and
when
we
say
crazy,
multi
cluster
topologies
I'd.
It's
a
bit
of
a
hyperbole.
We're
really
for
referring
to.
Is
these
hierarchical
topologies,
where
you're
designing
that
something
that
looks
more
or
less
like
a
tree
and
maybe
has
multiple
roots,
but
it's
more
or
less
like
a
tree,
and
if
the
roots
of
that
tree
are
your
data
and
at
the
leaves
of
that
tree
are
your
users.
A
So,
let's
take
an
example,
we
could
take
those
clusters
and
rather
than
cloning,
everything
instead
arrange
them.
Something
like
this.
So
we
break
up
the
world
divide
up
the
world
into
Americas,
Europe
and
Asia
Pacific,
and
then
each
of
those
locations
have
an
active
data
center,
where
your
data
is
maybe
have
a
stand
by
data
center
that
can
even
failover
to
and
then
any
other
data
centers
in
those
regions.
A
They
can
be
more
cash
only
or
stateless
data
centers
for
your
micro
services
and
deciding
ok,
which
regions
or
which
the
data
centers
should
be
active,
which
should
be
standby
and
which
should
be
stateless,
that
those
are
really
important
design
decisions
to
make,
as
you
scale,
from
2
up
to
10
traffic
routing
remains
a
problem
and
the
solutions
remain
largely
the
same.
You
need
a
global
load
balancing
solution
and
then
there
are
new
issues
that
we
didn't
talk
about
with
two
clusters,
so
service
discovery.
A
A
Some
folks,
in
the
you
know,
cloud
native
architecture,
space
posit
that
message
queues
are
the
right
way
to
decouples
and
that's
reasonable
and
you
can
use
something
like
Nats
to
create
a
globally
scalable
decoupled
message
to
for
doing
pub/sub
of
information
from
one
micro
service
to
another,
or
you
can
draw
again
on
the
google
or
the
or
the
amazon
toolbox
in
just
an
aside
here,
I
don't
mean
to
exclude
the
Microsoft
toolbox.
I
I
know
that's
just
as
valid,
but
there
are
only
so
many
slides
and
so
much
time.
A
Sorry
what's
another
issue
when
we
get
to
10,
so
it
might
have
been
the
case
that
for
two
clusters
you
were
just
fine
having
a
separate
login
monitoring
stack
for
each
of
those
clusters,
and
somebody
had
to
look
at
two
tabs
on
their
browser
and
was
a
pain
in
the
neck.
But
it
was
manageable.
Now
you're,
a
10
tho,
that's
no
longer
cool!
You
need
to
start
aggregating
your
metrics
and
your
logs.
You
need
to
start
filtering
them.
So
what
are
your
options
there?
A
A
A
You
have
tools
like
VPC
from
Amazon
and
Google
that
can
span
multiple
clouds,
and
then
you
can
finagle
things
like
calico
or
fly
annal
to
create
a
overlay
network,
but
for
the
most
part,
I
would
posit
or
suggest
that
you
you
should
avoid
trying
to
combine
networks
across
regions
like
this.
If
you
can
and
treat
each
region
more
like
a
autonomous
unit
with
its
own
ingress
capabilities.
Okay,
so
let's
call
it
ten
complete.
You
know
again,
of
course
those
aren't
all
the
issues,
but
they
all
do.
A
We
have
time
for
today,
let's
go
to
100
and
again.
Why
and
before
we
answer
why
I
want
to
go
back
to
that
assumption
that
we
had
earlier.
So
what
if
you
wanted
more
than
one
cluster
per
region,
maybe
that's
a
reason
why
you're
going
to
100
and
I
would
say
that
the
reasons
for
doing
that
and
we'll
get
back
to
the
scaling,
the
application
to
100
locations.
A
But
why
might
you
have
more
than
one
cluster
per
region
and
I
was
going
to
show
it
to
Paula
G
for
each
one
of
these,
but
you
can
kind
of
imagine
in
your
head.
So
one
reason
might
be
you're
isolating
either
teams
or
resources
from
each
other.
Maybe
you
have
a
special
set
of
hardware
that
has
special
security
characteristics
or
special
hardware
characteristics.
You
just
don't
want
it
to
be
part
of
the.
A
You
know
the
unwashed
masses
same
thing
for
so
grid
grid
computing,
another
example
where,
rather
than
having
one
giant
cluster
that
you
scale
up,
instead
of
scaling
up,
you
start
scaling
out
right.
You've
really
nailed
your
configuration
for
your
hardware
and
your
kubernetes
instances
for
a
rack
of
servers
and
then,
rather
than
going
and
back
to
the
drawing
board,
when
you
go
to
two
racks
or
four
racks
or
a
tracks.
A
Instead,
you
plop
down
more
clusters
that
can
be
a
little
bit
more
scalable
and
you
know
if
and
when
Federation
v2
takes
off
it
might
also
be
by
viable
and
then
a
third
one.
So
this
is
from
personal
experience
at
Cisco
is
we
typically
sell
or
Cisco
had
typically
sold
the
customers
who
had
more
resources
or
were
running
at
a
higher
scale
than
Cisco
themselves
could
test
at?
So
we
would
build
a
product
for
a
single
rack
of
kubernetes.
A
We
would
package
that
product
with
kubernetes,
but
then
when
we
went
to
sell
it
and
they
wanted
to
run
it
on
four
racks,
because
I
was
their
scale
requirement.
We
wouldn't
say
good
luck.
Scaling
up
to
four.
We've
never
tried
it.
That
would
not
go
well.
Instead,
we
would
say
place
four
of
these
next
to
each
other,
okay
back
to
100.
So
why
100
well
for
the
most
part,
it's
around
performance
and
scalability
we're.
You
know
same
sort
of
reasons
as
ten
clusters,
but
let
me
give
you
an
example
to
make
it
more
concrete.
A
I,
don't
know
how
many
people
read
the
Dropbox
blog,
but
they've
had
some
really
nice
articles
around
their
edge
network
as
somebody
who's
an
enthusiast
and
edge
computing.
These
are
these
are
really
interesting.
So
if
you
look
at
the
article
from
October,
they
kind
of
describe
why
and
how
they've
gone
from
a
few
pops
points
of
presence
up
to
20
pops
and
how
they've
done
that
and
the
reasons
are
really
for
them.
It's
all
about
scale.
A
It's
about
the
amount
of
data
that
needs
to
come
in
and
out
of
their
infrastructure,
they're,
synchronizing
files
constantly
and
moving.
You
know
tons
of
bits
every
second
from
our
clients
or
phones
or
our
PCs,
up
to
the
cloud
and
back
down
again
synchronizing
those
files.
So
when
you
hit
Dropbox
scale
or
greater,
you
sort
have
the
need
for
more
than
just
a
handful
of
pops
and
the
same
issues
apply.
These
issues
don't
go
away
just
cause,
you
have
more,
they
get
exacerbated,
but
for
the
most
part,
the
first
six
that
we
talked
about.
A
They
can
largely
be
solved
with
the
same
solutions
as
we
use
to
solve
those
problems
at
10.
Is
it
perfect?
No,
but
it's
close.
What
are
the
new
issues
when
we
hit
100
I'd
say
you
could
maintain
10,
see
ICD,
you
know
deployment
parts
of
the
pipeline,
but
when
you
get
to
100
it
gets
a
little
bit
unwieldy
so
having
a
unique
C
ICD
pipeline
to
point
each
cluster.
A
Rather,
if
you
consider
this
so
consider
the
concept
of
resource
propagation,
rather
than
going
to
each
cluster
and
saying
deploy,
deploy,
deploy,
deploy
and
making
sure
that
all
those
are
cut
into
in
sync,
using
your
CI
CD
tools
use
this
concept
of
resource
propagation,
where
you
deploy
to
a
single
root
cluster
or
root
clusters,
and
allow
the
system
to
propagate
that
out
from
those
root
clusters,
and
so,
if
you've
looked
at
Federation
v2,
which
is
very,
very
different
from
Federation
v1.
So
if
that's
what's
in
your
head,
you
know
reset
on
that.
A
Okay,
another
issue
security.
So
obviously
security
was
an
issue
for
one
cluster,
but
I
would
argue
that
the
issues
at
one
cluster
are
similar
to
ten
clusters,
but
a
hundred
clusters.
We
now
run
into
this
new
security
problem,
which
is
that
I
can't
trust
100
clusters,
I,
don't
care
who
they
came
from
or
where
they
are
I,
can't
trust,
100
clusters
I
might
be
able
to.
You
could
convince
me
to
trust
10
clusters,
but
100
is
a
step
too
far.
A
So
the
solution
here
is
is
not
exactly
simple,
but
you
need
to
limit
trust
and
access
to
just
your
core
clusters.
So
to
do
this,
it's
important
to
identify
and
what
are
the
crown
jewels
of
your
over
your
application?
Where
is
the
most
important
data?
Never
put
you
know
your
root
certificates
at
the
edge
or
in
in
these
hundred
clusters
always
consider
what
belongs
in
those
core
trusted
clusters
and
those
hundred
semi
trusted
clusters.
You
have
some
level
of
trust,
otherwise
you
wouldn't
run
code
there,
but
you
need
to
decide.
A
So,
as
you
propagate
applications
out
to
these
clusters
and
all
far
corners
of
the
world,
you
also
want
to
be
propagating
out.
The
images
that
underlie
those
containers,
and
so
using
tools
like
harbor
or
taking
image
registry
and
setting
up
in
a
caching
topology
is
actually
something
that
we've
done
in
the
past
in
order
to
get
better
scalability
out
of
image
deployment
and
artifact.
Caching,
all
right.
So,
let's
say
that
hundred
is
complete
again.
If
you
were
to
do
this,
no
guarantee
is
that
taught
me
these
issues
would
be
enough,
but
good
enough
for
now.
A
A
It's
really
at
this
point
about
being
on
premise:
it's
really
about
these
things,
latency
so
actually
having
a
really
low
latency
between
a
device
and
the
back-end,
that's
serving
it
or
something
that
produces
a
lot
of
bandwidth
or
something
that
requires
a
high
level,
security
and
privacy
or
something
that
needs
survivability
right.
You
think
that's
survivability
requirements,
you
think
about
like
a
hospital
right.
A
A
So
if
you
imagine
like
okay,
I
have
a
video
camera,
that's
monitoring
a
robot
in
a
factory,
I
need
really
low,
latency
and
a
high
level
of
bandwidth
and
security
and
privacy
and
resiliency,
for
that
particular
modality,
and
a
lot
of
this
starts
to
look
like,
and
this
might
just
be
me,
but
everything
to
me
it
looks
like
education.
Sorry,
this
looks
a
lot
like
edge
computing.
There's,
no
way
that
those
thousand
clusters
are
public
cloud
regions
there
just
aren't
that
many
of
them.
You
at
this
point
are
no
longer
just
doing
cloud
computing.
A
You
are
doing
cloud
computing
and
edge
computing
and
there's
a
whole
set
of
requirements
and
tools
to
help
you
do
that
and
also
a
ton
of
gaps
currently
in
that
space
and
one
of
the
issues
that
we
think
about
it.
Org
is
scarcity,
so
one
of
the
characteristics
of
edge
computing
isn't
just
being
at
the
edge
which
kind
of
a
nebulous
term
anyway,
it's
also
being
in
a
resource-constrained
environment,
where
you
don't
have
a
limitless
font
of
compute
resources
available
to
you,
like
you
do
in
AWS
us
east.
Instead,
you
have
a
smaller
set.
A
Maybe
it's
just
those
three
knucks
right.
Maybe
it's
half
Iraq,
but
resources
are
scarce
and
you
need
to
deal
with
it.
So
it
does
that
mean
that
you
should
clone
every
micro
service
and
run
them
everywhere,
or
should
we
be
more
intelligent,
so
I
bring
up
this
final
concept,
which
is
multi-dimensional,
auto
scaling,
so
we
we
all
know
about
scaling
up
and
scaling
down.
That's
one
of
those
things
that
Cooper
needs
is
is
really
good
at
you
can
scale
up
the
number
of
pods.
A
You
can
scale
down
the
number
of
pods,
but
what
about
scaling
out
and
scaling
back?
So,
if
you're
using
that
number
of
clusters,
do
you
need
to
run
every
micro
service
in
every
location
all
the
time?
Or
could
you
actually
use
some
type
of
metric
to
decide?
Okay,
I'm
seeing
a
ton
of
traffic
coming
to
no
sched
com
from
Seattle,
because
you
know
8,000
people
just
flew
in
there
and
they're
accessing
the
sched
website.
Maybe
we
could
place
some
of
our
resources.
A
Maybe
there
you
know
the
front-end
server
or
authentication
pieces
out
in
the
edge,
that's
in
the
Seattle
location,
and
so
you
now
no
longer
have
to
operate
in
one
dimension:
scaling
up
and
down.
You
can
now
operate
in
two
dimensions:
I
can
just
guy
decide,
ok,
more
load,
I
can
either
scale
up
where
I
am
or
I
could
actually
scale
out
closer
to
the
user.
So
it
gets
interesting.
A
Alright,
a
lot
more
issues
at
1,000,
but
let's
just
call
that
complete
because
you
know,
as
I
said,
we
don't
have
a
ton
of
time.
What
about
1,000
to
10,000
and
I?
Don't
yet
have
an
example
for
this,
and
if
somebody
does,
you
know
please
come
out
after
I'd
love
to
talk
to
you
about
it.
But
to
me
this
is
more
of
a
future
and
we
still
need
to
identify.
You
know
why
why
10,000
might
be
necessary
and
what
kind
of
tools
would
be
necessary?
A
Ok,
so
let's
bring
it
back,
wrap
it
up
over
the
past
25
minutes,
we've
gone
from
one
cluster
to
210
to
one
hundred
to
a
thousand.
We've
talked
about
these
at
cursory
level.
I'll
be
the
first
to
admit.
This
is
a
fairly
high-level
talk.
I
haven't,
given
you
the
details
about
how
to
do
a
lot
of
this,
just
the
you.
What
and
the
what
questions
to
ask
so
I'm?
A
Sorry
for
that,
but
we've
made
it
from
1
to
1,000,
and
this
is
just
summarizes,
so
we
talked
about
application
topology
and
that's
probably
one
of
the
most
important
ones
along
with
number
two.
When
you're
designing
your
application,
you
need
to
think
about
the
topology,
not
just
within
the
cluster
of
what
microservices
talked
to
what
other
micro
services.
But
how
do
you
take
those
micro
services
when
you
go
global
and
distribute
them
to
many
clusters,
same
thing
with
the
data?
Where
does
the
data
live?
What
parts
of
the
data
live
where?
How
is
it
replicated?
A
What's
your
consistency
story,
what's
your
availability
story
for
the
data
traffic
routing?
There
are
some
tried-and-true
methods
there
service
discovery,
as
I
said
you
know,
maybe
a
steal
will
catch
up
there,
but
maybe
in
the
shorter
term,
use
something
like
gnats
or
Amazon's,
queuing
service
logging
and
monitoring.
We
talked
about
things
like
Prometheus
networking.
We
talked
about
V
pcs
for
CI
CD.
We
talked
about
the
concept
of
maybe
adopting
Federation
Federation
v2
for
security.
We
talked
about
whittling
down
the
number
of
clusters
that
you
trust
image
distribution.
A
We
talked
about
actually
creating
a
caching
hire
it's
almost
like
a
CDN
for
image,
distribution
and
then,
finally,
for
scarcity,
we
talked
about
this
concept
of
multi-dimensional,
auto
scaling
and
along
the
way,
we
kind
of
developed
these
concepts
of
hierarchical
topology.
This
is
probably
the
most
important
one
when
you're
thinking
about
applications
that
span
the
clusters
always
think
about
hierarchy.
A
What's
at
the
root
of
my
tree,
what's
that
data?
What's
up
the?
What
are
the
leaves
of
the
tree
and
what's
in
between
resource
propagation?
So,
rather
than
touching
many
points
in
the
network,
instead
we're
touching
one
point
and
propagating
out
and
then
multi-dimensional,
auto
scaling.
I
can
now
scale
out
and
back
in
my
heart,
a
hierarchy
and
not
just
up
and
down
within
a
single
cell
and
then
finally,
you
know
rule
of
thumb
every
step
along
the
way.
A
The
question
we
asked
was
why
why
do
we
go
from
1
to
2
2,
to
10
10,
to
110,
to
1,000?
And
unless
you
can
answer
that
question
that
why
question
with
absolute
certainty,
or
at
least
some
level
of
certainty,
you
probably
don't
want
to
take
that
jump
so
never
use
more
clusters
than
you
need,
and
unless
you
can
answer
that,
why
question?
Maybe
you
don't
need
it?
A
Okay,
so
a
couple
minutes
left
here,
I
hope
I've
been
selling
you
a
little
bit
more
Multi
cluster
confidence
so
that
as
you're
scaling
from
1
to
1,000,
you
feel
good
about
what
issues
you'll
need
to
address.
As
I
promised
I'll
tell
you
a
tiny
a
little
bit
about
ort.
So
if
you're
interested
you
can
go
to
or
tae-oh
feel
free
to
give
us
your
email,
we
won't
spam.
You've
got
the
Terms
of
Service
on
there.
We
are
currently
in
stealth
mode
and
right
now
just
providing
consulting
around
edge
computing.
A
So
how
do
you
go
from
1
to
1,000
clusters
for
low
latency
use
cases
and
for
high
bandwidth
use
cases?
So
if
that's
a
special
interest
to
you
or
if
you're,
particularly
interested
in
how
do
you
go
from
1
to
2,
to
10
to
100
to
1,000
I'd
love
to
talk
to
you?
You
can
email
us
here
at
a
hello,
a
torque
that
IO,
if
you
like
to
schedule
some
time
with
me
or
my
team
I'll
be
available
all
day
tomorrow.
A
A
The
internet
of
things
for
10,000
clusters,
yeah
I,
suppose
I-
could
see
that.
So
the
comment
was,
you
know
the
answer
for
why
10,000
clusters
might
be
obvious
and
that
might
be
the
Internet
of
Things
we're
in
every
home
in
every
location.
In
every
telephone
pole
you
now
have
a
cluster
serving
you
know,
serving
data
or
processing
data.
Other
questions,
yes,
ma'am.
A
Right,
how
do
you
deal
with
a
single
point
of
failure
problems?
So
it's
important
to
treat
every
cluster
is
expendable.
Essentially,
so
you
might
have
ten
clusters,
but
you
can't
rely
on
any
single
one
of
them.
Any
single
one
could
could
go
down
and
that
kinda
has
to
be
built
into
your
architecture.
There's
another
question
up
here.
A
So
in
general,
at
least,
what
we've
discovered
is
that
running,
like
meta
clusters,
are
trying
to
take
a
single
cluster
and
span
it
across
multiple
regions.
This
has
never
gone
well
for
us,
so
we've
avoided
it.
If
somebody
gets
that
working
I
know,
there's
people
who
try
to
make
it
look
like
clusters
span
multiple
regions,
it's
interesting,
but
it's
it's
not
the
approach
that
we've
taken
anybody
else.
Yes,.
A
Right
right
so,
for
example,
for
like
data
sovereignty,
where
you
might
want
to
keep
it
in
one
particular
region,
yeah,
so
I.
Imagine
that
we
need
to
be
managed
kind
of
at
that
database
layer.
You
need
policies
at
that
database,
saying
okay,
these
set
of
Records.
We
need
to
stay
in
this
location.
I.
Imagine
that
so
say
you
need
to
keep
something
in
Germany.
You
almost
need
to
have
two
data
centers
or
clusters
in
Germany
for
the
reasons
of
failover,
so
data
pinning
almost
always
would
necessitate
not
just
one
but
but
to
anybody
else.
A
So
think
the
question
was:
how
do
you
deal
with
synchronizing
deployments
of
applications
across
clusters
of
deployment
versions
right?
So
the
hope
is
that
Federation
v2
can
can
solve
some
of
those
problems
for
you
where
you,
as
you're,
rolling
out
a
new
versions
of
your
resource
templates.
Those
are
kind
of
slowly
propagating
out
to
the
network,
but
knowing
what's
actually
been
deployed
and
having
observability
I,
don't
know
the
answer
to
that.
That's
a
good
question.
Other
other
questions.
A
Yeah
yeah
yeah,
so
one
of
the
nice
things
about
resource
propagation
is
that
you're
kind
of
forced
to
into
a
model
where
you
need
to
start
at
least
from
a
common
root.
You
can
do
the
templating
thing
and
that
could
get
out
of
hand,
but
at
least
you're
starting
from
the
defaulters
start
from
having
the
same
thing
on
every
cluster,
so
hoping
that
that
mental
model
of
resource
propagation
helps
solve
that
problem.
A
A
Yes,
good
point
so
we're
kind
of
oversimplifying
here-
and
this
goes
back
to
like
service
discovery-
is
that
yeah?
You
probably
have
like
a
default
topology,
but
you
almost
need
to
decoupled
the
layers
of
that
topology.
So
it's
not
quite
a
tree.
It's
more,
like
you,
probably
have
two
things
in
the
route.
Then
a
group
of
multiple
things
of
the
mid
layer.
Then
a
group
of
like
a
hundred
things
at
the
at
the
edge,
and
it's
really
like
an
ending
to
any
relationship.
A
A
How
do
you
operate
the
thousand
clusters,
so
good
question?
I
I
totally
glossed
over
that.
So
how
do
you
get
to
the
point
of
having
1,000
clusters
so
we
had
built
the
tool
around
you
know
deploying
and
managing
across
lots
of
clusters.
There
are
other
companies
out
there
who
are
doing
that,
but
m'naghten.
A
A
Okay,
you're
asking
how
you
handle
multiple
cloud
providers-
yeah,
okay
right.
So
a
lot
of
the
assumptions
I
made
here
were
that
you're
either
using
AWS
or
using
Google
you're,
not
using
both
right.
But
if
you
know,
if
you
were
here
early
in
day-
and
you
saw
some
like
the
cross
plane
stuff
from
upbound-
you-
you
see
like
the
need
for
working
across
many
different
cloud
providers
that
gets
hard.
You
almost
need
a
layer
of
abstraction
and
to
do
the
global
load.