►
From YouTube: Orchestrating Interconnected Apps Across Geographically Distributed Kubernetes... - John Belamaric
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Orchestrating Interconnected Apps Across Geographically Distributed Kubernetes Clusters - John Belamaric, Google
Speakers: John Belamaric
Imagine deploying a set of complex, interconnected workloads across a fleet of geographically distributed Kubernetes clusters. How do you decide where to run each workload? How do we specialize the configs for each site? How do we make sure those configs conform to our policies? How do we deliver the configs to the right clusters, and make sure they don't drift? What happens when we add a site - how do we know which interconnected workloads need to be reconfigured? How do we know what to change in each of those workloads? Do we just need to change Kubernetes manifests, or do the configuration files of the workloads themselves need to be changed? How do we do that? Can we really automate all this? Linux Foundation’s Nephio project (https://nephio.org) uses Kubernetes-based automation to solve these problems with an extensible platform for large scale, multi-site workload orchestration and configuration management. Come learn how we’re doing it!
A
A
A
A
But
you
know,
there's
other
use
cases
as
well:
retail
Edge,
where
you
have
tens
of
thousands
of
of
Stores
Factory
Automation.
In
fact,
there's
enough
demand
for
this
that
the
various
Cloud
providers
and
others
have
been
coming
up
with
sort
of
new
technologies.
They
call
multi-access
Edge
compute,
which
is
basically
a
rack.
You
can
drop
in
any
sort
of
Pop
anywhere
on
your
own,
your
own
site,
your
own
data
center,
your
own
location
or
in
one
of
their
pops,
and
you
wire
it
up
to
the
cloud
and
you
can
use.
A
You
can
use
your
Cloud
apis
to
to
manage
that
compute
resource
out
at
the
edge.
This
is
really
really
cool.
Actually,
because
one
of
the
sort
of
great
Revolutions
of
cloud
is
that
you
get
this
separation
between
the
capacity
provisioning
and
the
hardware
provisioning
of
that
capacity
and
the
consumption.
A
So
we
have
API
on-demand
driven
consumption,
which
means
other
people
like
Cloud
providers
can
build
that
capacity,
and
you
can
just
you
know,
rent
it
and
consume
it
as
you
need
it,
and
so
to
be
able
to
take
that
from
just
the
data
center
to
the
edge
and
spread
out
across
the
world.
That
same
consumption
model
creates
enormous
opportunities
for
cloud
providers
and
for
our
customers
as
well
as
many
many
other
people,
but
what
it
does
also
that
people
don't
really
talk
too
much
about
is.
A
A
Well,
let's
think
about
the
decisions
we
have
to
make
when
we
decide
to
deploy
deploy.
You
know
even
one
application,
much
less
a
set
of
interconnected
applications,
as
the
title
of
the
talk
is
I
mean
you
have
to
decide.
First
of
all,
where
do
we
run
clusters?
Because,
just
because
we
say
just
because
we
say
we
have
Edge
sites
doesn't
mean
they're,
all
Edge
clusters.
Do
you
want
to
put
a
cluster
at
every
single
edge?
A
You
say
what,
if
it's
a
tiny
little
Edge
site
with
one
one
or
two
nodes
at
it,
you
want
a
whole
cluster
there
or
not.
You
want
a
control
plane
there,
maybe
not
once
you've
decided
where
to
put
those
clusters.
How
do
you
decide
where
to
put
which
workloads
when
you
pick
the
place
you
want
to
put
a
workload.
How
do
you
decide?
A
How
does
how
do
you
specialize
the
configuration
for
that
particular
site,
because
if
it's
a
one
node
site
or
a
10,
000
node
site,
you
may
want
to
configure
your
workload
a
little
bit
differently
and
even
if
it's
not
just
the
IP
addresses
things
like
that?
May
change
across
those
things.
So
you
can
see
that
there's
just
so
many
problems
that
aren't
addressed
by
hey
just
stick
a
rack
out
there
and
hook
it
up
to
the
cloud.
It
leaves
a
lot
a
lot
more
problems
to
solve.
A
To
make
it
a
little
more
concrete,
this
is
a
picture
of
a
sort
of
a
very
hand,
wavy
simplified
picture
of
what
you
might
what
you
might
have
out
there
in
the
field.
If
you
are
a
a
Telco,
for
example,
so
at
the
very
far
Edge
the
left
side,
that
is,
you
have
radio
towers.
Each
of
those
radio
towers
has
a
cabinet
sitting
at
the
base,
has
a
little
bit
of
power
going
to
it
and
has
a
handful
of
machines
in
it.
A
So
that's
hundreds
of
thousands
of
edge
sites
there,
maybe
a
couple
miles
from
each
of
those
radio
towers.
You
know
covering
a
certain
area.
You've
got
a
small
building
with
a
couple
thousand
square
feet,
and
you
know
a
bunch
of
racks
in
there
that
can
handle
that.
Maybe
a
little
bit
farther
away
you've
got
a
bigger
data
center
that
can
hold
more.
And
finally
you
know
maybe
three
or
four
dozen
in
the
world
there
are
these
giant
Cloud
regions
with
acres
and
Acres
of
machines.
A
You
can
you
can
see
that
that
decision
of
where
do
I
run
the
workload
isn't
just
about
putting
it.
You
know
just
just
using
capacity
it's
about.
Why
does
it
need
to
be
that
close
to
the
to
the
end
user?
What
are
the
latency
requirements,
for
example,
that
the
user
needs
to
interact
with
that
workload
really
really
at
a
very
low
latency
basis?
And
what
is
the
cost
of
that?
Those
Tower
the
base
of
the
tower?
A
You
know
it's
a
very
limited
capacity
there,
and
so
it's
essentially
very
high
cost,
and
if
you
need
to
make
a
change,
you
know
you've
got
thousands
of
them
out
there,
hundreds
of
thousands
and
it's
very
expensive
to
do
so,
focusing
in
a
little
bit
more
on
workloads.
The
two
boxes
on
the
right,
you'll
see
UPF,
AMF
SMF.
It
doesn't
actually
matter
what
those
mean
so
I'm
not
even
going
to
tell
you.
But
what
is
important
here
is.
A
These
are
different
workloads
needed
for
a
5G
core
and
they
have
a
relationship
with
one
another.
So
the
the
the
UPF
is
going
to
run
a
whole
bunch
of
them
closer
a
little
bit
closer
to
the
to
the
user.
And
then
this
thing
called
the
SMF
is
going
to
sit
in
the
middle
and
they're.
All
going
to
talk
to
the
SMF,
so
this
is
when
I,
when
I
go
from,
we
talked
about
single
work
a
little
bit.
We
go
to
interconnected
workloads,
they
have
to
talk
to
each
other.
A
They
have
to
talk
to
each
other
across
these
tiers
of
sites
and
not
just
across
sites
but
across
tiers
of
sites
and
those
those
workloads.
Don't
just
have
to
have
interconnection
but
they're
actually
interrelated.
If
you
double
the
number
of
upf's,
you
have
you're
going
to
need
to
increase
the
memory
and
CPU
of
the
SMF,
where
you're
going
to
have
to
you're
going
to
have
to
maybe
provision
another
replica
or
something
in
another
place.
So
this
is
what
I
mean
when
I
say.
A
Imagine
yourself
deploying
a
set
of
applications
that
are
interconnected
or
interrelated
across
this,
so
it
you
know
it's
it's
pretty
challenging
putting
it
in
user
terms,
an
example
of
where
you
might
actually
do
that
and
we'll
sort
of
drill
into
the
details
of
the
problems
we
face
when
we
do.
This
is
one
one
thing
that
the
telcos
want
to
do
is:
is
private
5G,
so
they
want
to
be
able
to
roll
out
a
more
secure
network.
A
That's
just
for
your
company,
so
an
example
might
be
I've
got
these
20
000
trucks
in
the
Northeast
us
they're
driving
all
over
the
place,
and
they
need
to
send
Telemetry
data
back,
so
I
can
know
where
they
are
and
I
I
know
whether
they're
functioning
properly
and
maybe
I'll
use
it
like
a
CB
radio
system
too.
So
there's
certain
latency
requirements
for
that.
There
shouldn't
bandwidth
requirements,
but
they're
pretty
low,
and
so
you
know
this.
A
Theoretically,
if
we
can
do
it
properly
should
be
cheaper
than
using
the
the
general
purpose
and,
of
course,
more
secure,
so
just
to
roll
out.
You
know
we
want
our
customers
to
come
to
us
with
that
sort
of
statement
of
what
they
want,
and
then
we
have
to
figure
out
from
there.
How
do
we
Implement
that?
So
just
to
do
that
for
day
one
we
go
through
some
more
of
those
problems.
I
talked
about
in
the
abstract
in
a
more
practical
way
or
a
more
concrete
way.
A
We
need
to
identify
what
are
the
sites
in
the
Northeast.
What
types
of
sites
are
there
are
these
Edge
sites?
Are
they
Cloud
regions
which
workload
will
I
run
in
each
one?
What
are
the
infrastructure?
How
do
I
configure
the
nodes,
because
you
know
typically
there's
a
region
reason
we're
running
this
workload
at
the
edge?
It's
not
because
it's
just
a
generic
workload,
but
it
has
some
special
thing
it
wants
to
do,
which
may
include
a
special
sensor
and
then
maybe
a
GPU
or
a
dpu
that
processes
the
data
from
that
sensor.
A
Those
things
are
often
not
super
cloud
native
and
not
super
fungible,
and
so
you
actually
end
up
having
to
reboot
nodes
and
things
to
prepare
the
node
or
in
Telco
cases.
You
have
to
set
up
special
networking
so
often
there's
a
preparation
step
for
for
the
node.
So
there's
just
all
of
these
problems
we
have
in
day
one
and
even
if
we
figure
all
that
out
and
we
deploy
all
that
we
get
all
that
done.
A
We
have
day
two
to
worry
about
so
more
complexity.
We
have
to
monitor,
make
sure
that
things
stay.
The
way
that
we've
we've
sort
of
asked
them
to
be
make
sure
there's
no
configuration
drift.
We
have
to
handle
changes.
The
topology
the
customer
calls
up
says
Hey
every
time
my
trucks
drive
up
I-95,
this
far
I
lose
track
of
them,
and
so
we
need
to
build
out
more
capacity
up
that
Corridor,
and
that
means
we're
adding
more
upfs,
and
that
means
we
need
a
bigger
SMF.
A
So
we
have
to
be
able
to
track
all
of
that
and
and
figure
all
of
that
out
and
if
that's
not
enough,
it
does
get
worse.
So
every
single
one
of
these
things
I've
talked
about
the
topology,
the
cluster
creation,
the
workload
configuration
the
workload
manifests
in
kubernetes.
All
of
those
are
done
by
different
systems,
and
you
know
in
your
topology
might
just
be
a
spreadsheet.
That
says
here
are
the
sites
I
have
here's
the
capacity
available
I'm
going
to
put
clusters
in
these
different
places?
A
Then
you
have
to
call
cloud
provider
API
standstantiate
those
clusters.
Then
you
have
to
call
the
kubernetes
API
server
on
each
of
those
clusters
to
insert
a
workload
there.
So
every
single
one
is
done
by
a
different
different
system.
You
may
have
vendor
specific
systems,
so
so
you
know,
if
you
think
about
I
said,
like
the
UPF
and
the
SMF
to
know
about
each
other.
Well,
in
this
particular
case,
it's
a
Telco
thing.
Telcos,
don't
use
DNS,
because
we
all
know
that
you
know
outages
are
always
DNS
and
being
a
DNS
person.
A
I
can
say
that
safely,
but
the
the
you
know,
so
they
don't
actually
don't
use
DNS.
They
they
they
they
fix.
Ip
addresses
in
in
these
places
to
just
to
take
a
little
bit
of
the
complexity
out,
and
so,
but
that
means
you
need
some
Central
Source.
These
aren't
cluster
IPS
they're,
not
local,
to
a
cluster.
A
It
it.
It
needs
to
have
a
list
of
these
IP
addresses,
so
how
do
we
do
that?
We
do
it
with
some
vendor
proprietary
management
system
that
probably
SSH
ssh
is
into
the
workload
not
into
the
node
into
the
actual
workload
and
makes
changes
and
runs
commands
against
it.
So
tons
of
complexity,
tons
of
of
difficulty
all
right.
So
what
do
we
do
about
it?
Why
are
we?
Why
am
I
here?
A
So
we
were
faced
with
this
question
from
our
our
customers,
not
just
the
Telco
but
retail
Edge
and
other
customers,
and
so
we,
you
know,
we
sort
of
gave
it
a
lot
of
thought
and
we
came
up
with
three
basic
ideas
for
how
we
can
dramatically
reduce
the
complexity
and
hopefully
take
this
seemingly
intractable
problem
and
make
it
make
it
solvable
or
tractable.
The
first
is
to
stop
all
this
nonsense.
A
Is
that
there's
just
too
much
configuration
for
people
to
understand
or
manage,
and
so
we
need
to
make
that
configuration
processable
by
machines,
but
not
just
sort
of
like
in
a
in
a
very
shallow
way,
but
we
need
to
be
able
to
build
automations
that
sort
of
deeply
understand
that
configuration
and
can
do
things
like
understand
that
when
you
scale
up
this
particular
workload,
it's
going
to
have
an
impact
on
this
configuration
of
this
other
workload.
So
in
order
to
do
that,
the
machines
have
to
understand
the
configuration
in
a
pretty
pretty
deep
way.
A
All
right,
those
are
the
insights
or
the
theory.
How
do
we
actually
make
it
happen?
Well,
the
first
two
we
have
decided
we
can
address
with
kubernetes
everywhere,
so
this
doesn't
mean
that
we're
going
to
rewrite
all
vendors
software
to
be
natively
kubernetes,
because
that
will
never
happen
or
it
will
take
20
or
30
years.
So
we're
not
going
to
try
and
do
that.
But
what
it
does
mean
is
that
we
need
a
layer.
A
We
need
a
an
API
that
represents
those
configurations
within
kubernetes
resources,
and
then
we
need
actuators
or
controllers,
for
you
know
actually
applying
that
configuration
on
the
lower
layer.
So
this
is
not
some
great
Insight
of
ours.
This
is
what
everybody
here
has
probably
been
doing
for
four
or
five
years
with
operators
and
crds.
But
it's
really
more
about
that.
A
We're
going
to
apply
this
consistently
across
all
of
these
layers
and
we're
going
to
make
sure
that
the
you
know
that
that
that
intent-based
sort
of
declarative
management
is
is
the
style
we
we
use
and
specifically
picking
kubernetes,
not
just
because
we
wanted
to
be
able
to
come
to
kubecon,
but
because
you
know
it's
a
Well
understood,
extensible
framework
for
building
these
kind
of
platforms.
That
has
at
least
you
know,
some
mileage
under
its
belt
and
a
great
ecosystem.
A
The
third
bullet
we're
addressing
with
something
we
call
configuration
as
data.
So
this
is
a
new
approach
to
configuration
management
and
you,
you
may
have
played
with
it
some
customized
as
sort
of
our
first
foray
into
this,
but
it's
got
its
challenges
and
its
limitations
and
we're
trying
to
make
that
better
through
some
new,
tooling
and
Technology.
But
this
is
a
sort
of
recognition
that
existing
configuration,
Management,
templating
Systems,
like
Helm,
have
some
they
have
their
place
and
they
have
a
certain
role
but
they're
not
effective
at
the
level
of
scale.
A
We're
talking
about
and
part
of
the
reason
is
that
every
single
Helm
chart
is
effectively
creating
a
new
API.
The
values
file
itself
is
a
new
API,
that's
specific
for
that
individual
package.
It's
really
really
hard
to
write
automations
that
have
to
deal
with
bespoke
interfaces
for
every
single
thing
they
want
to
automate.
In
fact,
it's
not
just
hard,
it's
really
pointless
and
impossible.
A
So
the
idea
here
is
based
on
four
essential
principles.
One
is
represent
the
configuration
in
well-defined
structured
data
model
and
we
use
krm
kubernetes
research
model
for
that
which
shouldn't
shock
anybody
here.
The
other
is,
is
use
a
git
Ops
model
so
put
that
into
a
version
store
prior
to
the
live
State,
and
the
reason
for
this
is
that
it
enables
one
it
enables
things
like
undo
or
or
rolling
back,
which
is
important,
but
it
also
gives
us
a
A
pre-actuation
or
a
pre.
A
Another
main
point
is
that
tools
operate
on
the
config
and
those
tools
are
reusable,
so
this
one
I
tend
to
think
of
like
like.
If
you
look
at
templating
systems,
what
they
do
is
they
intermingle
your
code
and
your
configuration?
They
treat
the
configuration
as
text
and
the
code
is
sort
of
wrapped
all
around
it
and
it
you
feed
in
some
inputs
and
it
spits
out
some
hopefully
well-formed
yaml
in
the
end,
that
doesn't
work
very
well
at
scale
and
it
doesn't
make
reusable
pieces
of
code
and
testable
pieces
of
code.
A
So
I
tend
to
think
of
it
a
little
bit
like
the
journey.
The
industry
made
from
writing
individual
bespoke
programs
with
bespoke
data
structures
in
order
to
produce
you
know
your
TPS
report
in
Cobalt
or
whatever
you
had
to
write
some
very
specific
program
that
processed
a
very
specific
data
structure
and
output,
something
but
then,
eventually
somebody
said
hey.
You
know,
let's
make
streams
in
Unix.
Well,
every
everything
will
just
be
a
stream
and
then
they
said,
hey,
let's
put
a
little
structure
on
that.
Let's
make
line
breaks
now,
all
of
a
sudden.
A
We
have
you
know,
line
editors
and
wc
and
all
these
tools
that
can
operate
on
any
stream
of
data
that
follows
this
very
simple
structure.
You
add
column
breaks
in
there
and
all
of
a
sudden,
you
can
build
a
relational
database
management
system
on
top
of
this
very
simple
structure.
So
just
a
little
bit
of
structure
can
enable
highly
reusable
tools
and
in
a
similar
way,
configuration
as
data
says,
let's
just
put
a
little
bit
of
structure.
A
Last
main
point
on
configuration
is
data
and
I
could
talk
about
configurations.
Data
for
hours,
so
I
have
to
be
careful
not
to
run
out
of
time.
Is
that
the
clients
interact
via
apis,
so
we
put
an
API
layer
on
top
of
that
storage,
instead
of
them
directly
interacting
with
storage.
This
allows
for
much
better
concurrency
and
sort
of
consistent
transformations
of
that
configuration
by
these
independent
actors.
So
I'll
I'll
give
an
example
later
about
the
independent
actors.
So
that's
that's
our
approach.
A
Foundation
project
called
nepheo,
and
we
got
a
bunch
of
our
friends
in
the
industry
to
join
us
in
this
effort
which
we
launched
back
in
April,
we
had
about
two
dozen
and
now
I
think
there's
60
or
70
different
companies.
Telcos
are
a
lot
of
the
representation,
mostly
because
they
operated
a
scale
that
they're
running
into
these
problems.
A
lot
of
these
problems,
you
don't
hit
until
you
hit
scale,
so
everything
seems
fine,
and
then
you
try
to
get
to
a
certain
level
of
scale
in
your
your
toast.
A
A
So
the
idea
and
the
goal
of
nephew
is
to
implement
this
Vision.
We
just
talked
about
at
a
very
high
level.
We
do
this
and
I
do
say:
I'm
pretty
low
on
time
to
leave
any
any
time
for
questions,
but
at
a
very
high
level.
What
we
do
is
we
take
and
we
build
an
orchestration
cluster.
This
is
one
management
cluster.
That
kind
of
has
a
bunch
of
it
contains
these
apis
that
sit
on
top
of
the
on
top
of
the
storage
layer.
A
That
manages
the
configuration
and
we
use
that
those
apis
to
implement
kubernetes
controllers
that
can
operate
on
top
of
those
those
configurations.
So
these
configurations
we
I'll,
use
the
word
package
a
lot.
A
package
is
a
little
bit
like
a
Helm
chart,
except
that
it's
all
krm
inside
there's
no
templates
that
might
be
ill-formed
or
anything
like
that.
So
it's
all
sort
of
machine
processable,
because
it's
it's
krm.
A
So
that's
the
sort
of
platform
level
architecture
we
need
in
order
to
solve
this
problem,
but
we
also
need
sort
of
content
for
that
prop
that
that
platform
to
process
and
by
content
I
mean
kind
of
the
the
packages
or
the
models
that
represent
the
things
we
want
to
create
so
make
that
a
little
bit
more
concrete.
This
diagram
shows
three
sort
of
swim
Lanes
vertically,
along
like
this,
the
very
left
hand,
side
is
infrastructure.
So
the
idea
is,
we
need
a
model
for
infrastructure.
A
We
need
kept
packages
that
represent
a
cluster
that
represent
a
network
and
we
need
to
be
able
to
run
them
through
that
configuration
process
and
realize
them
so
we're
not
going
to
produce
in
nephew
a
new
actuators
for
clusters.
We
already
have
cluster
API
and
we
have
kubernetes
config
connector,
which
is
good
for
Google
Cloud.
We
have
ack
for
aw,
U.S
Azure
has
another
thing:
we
have
cross
plane.
A
We
have
a
number
of
of
up
of
ecosystem
tools
that
play
that
role,
but
what
we
do
is
provide
linkages
to
our
automations
for
those
different
tools.
So
that's
the
idea.
The
middle
swim
Lane
are
the
worklets.
Remember
we
talked
about
you've
got
to
deploy
a
workload
across
a
set
of
clusters.
First,
you
have
to
deploy
the
set
of
clusters.
A
That's
the
first
swimlin.
Then
you
have
to
deploy
those
workloads.
That's
the
second
swim
Lane
in
ordinary
Edge
applications.
Those
are
just
kubernetes
resources,
ordinary
kubernetes
resources
in
Telco.
We
can
take
it
a
little
further
because
there's
sort
of
these
special
models
of
what
a
UPF
and
AMF
is
and
SMF
and
different
vendors
can
Implement
them
but
effectively.
That's
that's
the
content
again
that
this
platform
needs
to
process.
The
third
one
is
application
config.
This
is
sort
of
farther
out,
but
the
idea
is
what
we
talked
about
earlier.
A
The
application
has
to
know
has
to
be
reconfigured.
Potentially
when
a
new
workload
that's
related
to
it,
comes
up.
We
need
a
way
to
actuate
that
through
krm,
but
there's
a
lot
of
variability
in
how
that's
done
so
we're
that's,
that's
a
couple
years
out
Okay,
so
that
is
sort
of
the
that
and
the
previous
slider
sort
of
the
the
platform.
The
content,
how
we
move
platform
through
move
content
through
a
nephew
platform.
A
But
what
does
the
user
interaction
look
like
because
I
know
this
is
pretty
abstract
and
can
be
hard
not
hard
hard
to
grasp.
So,
from
a
user
point
of
view,
you
know
we
sort
of
tend
to
think
of
it
as
a
few
different
types
of
users,
one
type
of
user
is
that
that
isv,
independent
software
vendor
who's
got
some
packaged
software
that
they
that
they
make
and
they
sell
that
they
want
customers
to
be
able
to
deploy
on
their
Edge
sites.
A
Another
one
is
an
organizational
platform
team
or
in
the
Telco
case,
it's
like
a
network
designer.
They
may
want
to
take
that
that
off-the-shelf
package
they
clone
it
into
their
environment
and
they
customize
it
for
their
particular
organization.
They
add
some
policies,
they
add
some
configuration
that
points
to
some
Central
Services.
A
They
configure
the
IM
that
they
use
things
like
that
that
that
apply
to
any
anything
running
in
their
organization
and
then
you've
got
your
app
teams
or
your
deployers
they're,
going
to
further
specialize
that
package
by
cloning
it
to
their
environment,
so
so
we're
using
multiple
git
repositories
to
clone
this
package.
But
it's
not
just
a
fork
like
in
GitHub.
A
This
is
actually
done
in
a
special
way,
and
this
is
part
of
the
capped
functionality
for
configurationist
data,
where,
when
we
clone
that
we
keep
track
of
the
upstream-
and
this
is
super
important,
because
we're
keeping
track
of
the
Upstream-
not
just
like
it's
in
that
spot,
but
the
actual
tagged
revision
of
that
upstream-
and
this
is
how
we
we
start
to
address
day
two.
So
you
can
imagine
you
you
take
the
package
software,
you
clone
it
to
your
local
environment.
You
create
your
own
version
of
it.
A
A
Automations
I'll
take
one
more
minute
to
describe
one
more
little
bit
of
that
automation.
So
after
that
app
team
deploys
that
one
of
the
things
I
talked
about
earlier
was
we
really
want
humans
and
machines
to
be
able
to
interact
with
the
configuration
to
make
it
all
come
as
one
I'll
be
a
complete
configuration
and,
and
so
to
make
that,
like
really
specific
I,
don't
want
a
giant
script
that
takes
every
possible
input
and
tries
to
create
10
000
instances
of
a
package
across
10
000
environments.
A
So
these
three
pieces
of
information
are
very
when
we
fan
out
that
the
variance
of
that
package
across
those
10
000
clusters,
those
three
pieces
of
information-
are
going
to
vary,
or
at
least
two
of
them
are
going
to
vary
for
each
one
of
those
instances
of
that
package.
But
we've
actually
created
separate
instances
of
those
those
packages.
A
Those
configurations
we're
not
we're
not
taking
a
a
values,
file
and
spraying
a
bunch
of
10
000
API
servers
with
ephemeral
configuration
data,
we're
actually
storing
in
git
the
actual
configuration
that's
going
to
be
for
that
particular
cluster
and
the
there's
a
controller
there's
a
controller
sitting
in
the
in
the
management
cluster.
That's
watching
for
new
revisions
of
that
contain
that
resource
that
says,
I
need
an
IP
address
and
that
controller
can
pick
up.
That
configuration
see.
Oh,
you
need
an
IP
address
and
those
are
the
three
parameters.
I'm
going
to
go.
A
Talk
to
my
ipam
system.
I
have
a
central
IP
address
management
system
in
my
organization
that
has
an
API
and
this
controller
is
going
to
go
talk
to
it.
It's
going
to
allocate
an
IP
address
out
of
it,
and
it's
going
to
inject
it
in
that
configuration,
and
it's
going
to
do
that.
10
000
times
with
10
000
different
IP
addresses
across
10
000
clusters,
and
so
now
I
was
able
to
build
that
automation.
A
A
B
Hello,
hey
thanks
by
the
way
great
talk.
What
I
was
asking
about
is
at
one
point
in
the
talk
you
mentioned:
users
can
interact
with
these
rollouts
too,
so
something
like
Argo
for
syncing
configuration
would
work.
How
would
what
would
that
look
like
in
this
setup,
like
you
had
the
configuration
sync
yeah
so.
A
So
I'm
not
really
talking
about
other
workflows
here,
I'm
just
talking
about
the
the
get
Sinker
piece
so
essentially
and
I'm,
not
an
Argo
expert,
so
I
can't
say
in
in
detail.
I
do
know
that
some
of
our
customers
want
to
use
that
in
place
of
config
sync
and
that
it
should
be
able
to
serve
that
role.
But
nephew
is
new,
we're
still
building
it.
Several
of
our
community
members
have
expressed
interest
in
that
as
well,
so
I'm
sort
of
looking
to
them
to
build
that
integration,
but
you
know
it
essentially.
A
The
final
completed
configuration
gets
pushed
into
a
git
repository,
but
it
has
to
get
somehow
from
the
git
Repository
to
the
actual
API
server
and
and
so
we
need
a
git
sync
our
agent
to
do
that
and
that's
the
role
I
would
see
it
playing.
There
may
be
other
Integrations
possible
as
well
as
well.
Yes,
oh
okay
over
there
and
then
in
the
black
sweater,
and
then
the
blue
sweater.
A
That
that's
a
good
question,
so
that's
something
we're
looking
again
I'll
pump
that
to
the
community
we're
looking
to
the
community
to
solve
that
there
is
a
desire
to
implement
some
of
the
standards
apis.
Personally,
in
my
my
opinion
that
well
like
so
so,
I've
talked,
for
example,
to
folks
from
some
of
the
service,
orchestration
companies
and
and
they're
like
well
yeah.
Just
whatever
API
you
have
is
fine.
A
It
can
be
a
kubernetes
API,
so
they
you
know
they
don't
really
care
right,
they're,
going
to
build
an
integration
anyway,
they're
happy
to
just
build
it
through
kubernetes
API.
So
we
did
talk
early
on
in
the
community
about
building
some
interfaces
that
provide
standards-based
apis
and,
if
necessary,
that's
something
we
can
do,
but
really
we're
Downstream
of
all
of
those
right
we're
southbound
of
all
of
those.
So
you
know
we
may
need
to
present
some
standards
base
up
north
north
API.
D
A
We
do
exactly
the
same
thing.
You
know
we
like
to
learn
lessons
from
other
people,
so
we
we
package
revisions
have
conditions
once
a
certain
set
of
conditions
are
met,
which
is
definable
by
the
package
author.
Then
we
can
go
to
a
publishable
state
once
we're
in
a
publishable
state.
Then
we
can
we'll
actually
end
up
on
the
cluster,
so
we
can
either
have
a
rollout
controller
that
toggles
the
publishable
state,
or
we
can
have
a
rollout
controller
that
toggles
the
get
Sinker
configuration
in
the
edge
cluster
to
base
to
Twitter.
A
E
A
Absolutely
so,
in
fact
porch,
which
is
the
API
layer,
it's
the
it's
the
configuration
it's
sort
of
kept
as
a
service
kept
as
a
CLI
tool.
Porch
is
the
API
surface
we
put
on
top
of
Git.
It
also
can
can
write
to
LCI
so
actually
for
a
lot
of
these
Edge
type
of
workloads.
That
is
likely
we'll
end
up
using
oci.
It's
just
it's
a
subtlety.
Most
people,
don't
don't
get
so
I
just
to
talk
about
githubs,
but
yeah,
absolutely
good
question
all
right.