►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Kubernetes The Database - Jonathan Owens & Maryum Styles, New Relic
In the operations world, one of the hardest problems is keeping track of your inventory: Which machines belong to which teams? Which machines are in service? How long have they been there? At New Relic, the ability to keep track of a massive inventory that runs across multiple providers quickly became an unbearable task so much so that it required designing a completely new central {tracking?} system that could scale with a large infrastructure. In this talk, you’ll learn how Jonathan Owens and Maryum Styles used the Kubernetes API server to jump-start this design and create a unified infrastructure description service. They will share how they defined resources, created controller services, and dramatically decreased the process of manual updates.
To learn more: https://sched.co/GrYP
A
We're
a
public
company
you
get
to
take
a
look
at
our
safe
harbors
slide.
It's
worth
clarifying
that
this
is
not
a
talk
about
running
databases
on
kubernetes.
There
are
many
excellent
talks
about
that
here.
What
this
is
actually
about
is
running
kubernetes
components
themselves,
much
like
a
database.
In
our
case,
we
use
it
to
store
database
of
our
hosts
inventory
on
our
different
cloud
providers
and
hardware
providers
from
around
the
world.
First,
a
small
question:
what's
on
your
kitchen
counter
right
now,
you
probably
haven't
been
home
for
a
few
days.
A
You
might
not
remember
exactly
you
might
have
a
cleaner
kitchen
than
I
do,
but
when
it
comes
to
answering
questions
about
what
you
have,
it
gets
harder
when
that
thing
is
even
further
away
again:
you're-
probably
not
close
to
your
kitchen
right
now,
but
what
if
you
had
a
rental
property
say
in
Germany?
Could
you
tell
us
what
was
on
that
counter?
Could
you
tell
us
what
your
rental?
What's
your
tenants
we're
doing
in
there
not
so
fun,
so
we
had
a
similar
problem
on
our
team
at
New
Relic.
A
A
A
We
picked
DCOs
and
mezzos
at
a
time
when
it
was
stable
and
served
as
well,
and
it's
a
pretty
decently
sized
deployment
we
have
about
a
thousand
machines,
most
of
which
are
hardware
and
it's
running
and
now
in
two
regions
on
three
different
infrastructure
providers
again,
including
our
own
bare-metal
data
center,
and
the
purpose
of
all
this,
of
course,
is
to
run
user
services,
make
them
available
and
make
them
scale.
So
the
way
we
actually
manage
this
thing
is
a
lot
of
ansible
in
terraform.
A
Before
we
added
this
inventory
concept
and
used
the
kubernetes
components,
there
was
a
lot
of
well.
We
got
to
startup
machines
with
terraform
tara
forms
going
to
talk
directly
to
the
infrastructure
provider
once
those
machines
are
created
by
the
infrastructure
provider.
We
turn
around
and
take
ansible.
Ask
those
infrastructure
providers
again
for
the
things
that
we
just
made
then
go
and
orchestrate
the
changes
on
the
machines
that
we
can
put
them
to
work
for
all
this
to
function.
A
We
need
a
total
network
connectivity
across
these
components
so
from
the
orchestrating
machine
that
was
running
ansible
and
jenkins.
That
might
be
a
laptop
that
might
be
another
machine
in
the
CI
system.
We
would
need
to
connect
over
a
VPN
to
the
infrastructure
provider
to
talk
to
it
about
what
it
built
and
then
also
over
that
same
VPN
to
the
actual
machines,
so
that
we
could
actually
do
work
on
them.
A
funny
thing
happened
on
the
way
to
scale
we
added
a
European
region
last
year.
A
So
all
of
a
sudden,
this
gets
awkward
right,
which
VPN
do
I
beyond
do
I
have
to
be
on
a
separate
one,
to
look
at
the
European
ones
versus
the
North
American
ones,
and
then,
when
we
talk
to
the
people
that
were
running
this
project,
they
described
a
future.
That
looks
a
little
bit
like
this.
This
made
us
a
little
uncomfortable
as
it
should
you
that's
a
lot
of
lines.
That's
a
lot
of
networks.
A
We
can't
really
operate
comfortably
in
a
world
where
every
single
time
we
want
to
touch
something,
there's
some
central
point
that
has
to
work
on
it
and
we
have
to
figure
out
just
which
cluster
do
we
talk
to
at
all.
So
wouldn't
it
be
nice
if
these
things
could
be
more
independent,
wouldn't
it
be
nice
if
we
could
just
sort
of
push
this
intelligence
closer
to
the
edge?
A
So
we
looked
hard
at
the
problem
and
realized
that
the
problem
in
fact
was
not
one
of
connectivity.
It
wasn't
essential
for
us
to
talk
to
all
these
machines
directly,
all
the
time
we
just
needed
to
know
what
was
there.
We
had
an
inventory
problem
and
not
necessarily
a
connectivity
problem.
If
we
can
find
out
what's
there
and
what's
going
on
and
have
that
be
updated
as
a
reflection
of
what's
real
on
the
ground,
then
we
can
turn
back
around
and
go
over
the
whatever
VPN
is
appropriate
to
region
and
clusters.
A
Where
there's
something
interesting
going
on
and
actually
make
the
changes,
this
turned
out
to
be
a
pretty
good
idea,
the
more
we
thought
about
it
and
well.
What
else
could
we
do
with
this?
We
could
actually,
if
we
had
a
place
where
the
state
of
the
clusters
was
easily
available
and
always
up
to
date,
we
can
hang
more
parts
off
of
it.
We
can
work
a
straight
monitoring
off
of
this
thing.
A
A
It
makes
this
TF
state
file
and
it's
really
useful
because
it
lets
you
look
at
all
the
things
that
terraform
did
and
what
state
it
believes
the
world
to
be
in,
but
it
only
updates
when
you
actually
run
a
plan
or
when
you
there's
a
resynchronization
operation,
you
can
do
and
that
only
happens
on
demand.
It's
not
really
live
on
top
of
that.
A
It's
just
a
file,
so
you
can
find
it
in
s3,
but
you
have
to
like
read
and
write
this
entire
file
when
you
want
to
do
anything
so,
just
really
clumsy,
we
looked
at
commercial
datacenter
inventory
management
tools.
This
would
be
something
like
the
vice
42
or
n
light.
These
are
marketed
for
towards
people
that
are
running
their
own
data,
centers,
which
we
are
but
they're
really
deeply
concerned
with
things.
What's
the
Machine,
what
rack
is
it
in?
What
are
your
PD
use?
A
How
many
hard
drives
do
you
have
on
hand
and
not
really
anything
higher
level
than
that?
And
we
really
wanted
the
ability
to
express,
like
I've,
got
a
cluster
and
here's
what's
important
to
me
about
a
cluster
and
here's
its
regen,
and
it
wasn't
really
gonna
extend
into
that
world.
As
I
said
we're
running
on
DC
OS,
so
you
might
be
asking
yourself
well
what
about
that?
Doesn't
that
have
anything
you
could
use
here?
Sadly,
the
answer
is
no.
Do
you
cos
because
it's
built
on
mezzos
is
really
built
on
a
responsive
pattern.
A
Mesos
itself
waits
for
offers,
it
waits
for
agents,
it
waits
for
things
to
happen
in
the
infrastructure
and
then
asks
for
things
to
happen
when
those
have
occurred.
There's
not
really
a
declarative
world
inside
DCs
and
all
the
components
that
come
with
it,
and
that
was
really
what
we
wanted.
We
wanted
this
pattern
of
declaring
state,
so
we
looked
at
what
didn't
work.
What
did
we
learn
from
that?
And
what
do
we
actually
need?
Well,
it
needs
to
be
a
single
source
of
truth.
We
can
tell
that
much.
That's
the
point
of
this
whole
thing.
A
That's
why
we
don't
wanna
have
all
those
lines
we
wanted
the
ability
to
update
it
from
anywhere.
So
you
know
the
European
region
and
then
future
regions
or
future
separate
networks
could
just
go
and
talk
to
this
thing
without
having
to
tunnel
to
it.
It
needed
to
be
compatible
with
the
ansible
inventory.
So
we
want
to
make
this
one
step
at
a
time.
A
We
want
to
change
the
inventory,
but
we're
not
yet
ready
to
change
all
the
orchestration,
and
so
we
just
need
to
be
able
to
plug
this
thing
into
the
ansible
script
and
have
it
go.
It
needs
to
be
production
quality.
Of
course,
if
we're
gonna,
hang
monitoring
and
other
things
off
of
it
and
then
going
back
to
the
thing
we
learned
from
the
DCIM
tools,
we
want
the
ability
to
put
higher
level
abstractions.
This
is
a
cluster.
A
This
is
a
log
host
there's
what's
going
on,
that,
I
want
to
have
happen
more
so
than
just
like
where's,
my
stuff,
so
being
operators
of
a
cluster
and
being
in
this
space
we
started
reading
all
kinds
of
websites,
including
the
kubernetes
one,
and
if
you
look
at
the
kubernetes
website,
you
might
find
a
list
something
like
this,
and
you
know
we
looked
at
that
and
went
well.
We
have
most
of
these
things.
Okay,
that's
cool!
A
You
kind
of
need
those
things
to
run
a
cluster,
but
there's
one
thing
on
here
that
DCOs,
notably
really
doesn't
have-
and
that's
this
centralized
configuration
thing
that
it
sounds
pretty
neat.
How
could
we
get
just
that?
Well,
it
turns
out
you
totally
can
so.
The
API
server
is
one
of
the
first
things
that
you
provision
when
you
build
a
cluster
and
it's
the
thing
that
holds
the
declarative
state
of
all
the
objects
that
represent
the
clusters
behavior,
but
it
can
do
a
little
more
than
that.
A
So
what
would
we
need
to
do
to
make
an
inventory
system
that
would
consisted
entirely
of
CR
DS?
Well,
we
would
need
a
basic
object
structure,
so
we
need
to
tell
the
API
server
here.
Are
the
objects
I
want
to
talk
about
things
like
houses,
things
like
clusters?
That's
just
a
blob
of
llamo
you'd.
Give
that
a
thing.
We
need
something
that
lives
inside
these
remote
clusters
to
talk
to
the
infrastructure
providers
and
see
what
do
you
have?
What
hosts
are
under
management?
What
has
changed
and
then
we
just
need
an
adapter.
B
So
we
need
a
piece
of
code
that
would
query
our
provider
API.
It's
a
push
data
into
our
cube
server.
We
run
a
pretty
naive,
kubernetes
controller
that
would
manage
this
process,
and
then
we
call
them
fetchers
because
they
fetch
data.
They
also
push
data.
We
still
call
them
fetchers,
so
we're
gonna
go
over
how
they
work.
This
is
the
overview
and
we
start
by
making
our
API
call
to
our
provider.
We
have
one
fetcher
per
provider
and
the
we
are
our
providers,
our
AWS
software,
and
then
we
also
have
our
data
center.
B
We
take
our
provider
response
and
store
that
in
a
ghost
truck
internally,
so
we
can
work
with
it.
We
then
think
that
struck
and
convert
into
a
host
object.
Our
host
object
is
a
custom
resource
in
kubernetes.
Customer
expressions
are
really
awesome
because
of
how
you
to
extend
the
kuving's
framework
with
their
own
custom
restarts
you
get
to
use
the
same
powerful
api's
along
with
the
crud.
You
get
a
caching,
you
get
informers,
you
get
Watchers
and
you
can
take
all
of
that
and
do
amazing
things
in
order
to
create
a
custom
resource.
B
You
can
follow
these
steps.
I
mean
start
by
doing
your
custom
resource
definition
mo
file.
You
then
create
like
a
ghost
truck
that
you
use
in
your
code.
You
update
the
register
file
with
your
new
object
name,
and
then
you
run
your
cogeneration,
your
code
generations,
where
the
real
magic
happens.
That's
where
you
get
all
of
the
files
I
will
do
the
Informer's
the
like
create
the
updates,
the
deletes.
All
of
that
is
included
with
you,
and
then
you
apply
your
CRD
file
to
your
API
server.
So
that
way
the
API
server
says
I
hostess.
B
Almost
I
could
look
at
a
host,
see
idml
file.
This
is
what
ours
looks
like
it's
pretty
minimal,
says:
hey
server,
look
out
for
a
custom
resource
definition
and
look
it's
going
to
be
a
host,
and
so
it
says
like
hey.
This
is
what
a
host
is
gonna
look
like
well
TRD
file,
so
you
have
the
option
to
use
validation
and
validation
can
say,
hey
for
some
fields.
We
don't
want
to
have
any
numeric
characters.
We'd
had
chosen
to
use
that
and
we
chose
to
just
with
the
option.
B
This
is
what
our
host
looks
like
and
go
and
it
sort
of
programming
with
internally
and
the
type
meta
stores,
the
kind
of
API
version
we
saw
earlier,
and
then
our
object,
meta
stores
and
annotations
and
labels
which
we
look
into.
You
have
respect,
which
represents
the
desired
state
of
yourself
object
and
then
your
status,
which
represents
the
status
status
of
your
host
object.
B
We
use
our
labels
and
annotations
a
lot
for
our
inventory
service.
Labels
are
really
awesome.
You
can
use
them
to
search
and
filter.
For
example,
if
you
were
like
hey
I
want
to
see
all
of
my
hosts
that
are
log
machines
that
are
running
in
AWS.
You
can
say
you
can
use
the
command
line
or
the
API
request
to
do
that.
Annotations
are
really
great.
We
using
the
store
data
about
the
hosts
that
we
then
use
in
downstream
systems
or
in
scripts.
B
So
now
we
got
our
host
object.
We
can
store
it
in
our
database
via
an
update
or
create
API
call
to
our
cube.
Api
start
right
and
it
is
going
to
a
real
database.
It's
going
to
at
CD.
One
of
the
things
you
have
to
make
sure
you
do
is
like,
if
you're
using
an
update,
call
you
can
possibly
clobber
your
data
and
so
on
our
case,
we
use
patch
to
make
sure
that
we
don't
drop
any
fields
in
our
struct
object.
B
We
then
use
ansible
to
add
our
certs
and
then
use
install
cube,
API
and
the
controller
manager
so
know
about
the
note
about
the
certs.
Our
certs
are
added
by
hand
and
currently,
if
we
have
any
human
or
a
new
service
that
wants
to
talk
to
her
cube,
cube,
API
server.
We
need
to
approve
the
new
certs
and
that's
a
manual
process
or
something
we
hope
to
automate
in
the
future,
and
the
only
reason
we're
installing
the
controller
manager
is
because
it
allows
you
to
the
cert
approval
process.
B
If
you
heard
a
kubernetes
the
hard
way
by
Kelsey
Hightower
we're
doing
kubernetes
the
easy
way,
because
we're
only
doing
the
first
three
steps,
we're
not
to
point
any
pods
or
any
workloads
we're
not
worrying
about
any
pod
networking.
So
if
you're
interested
in
playing
around
with
this
model,
I
would
say
it's
pretty
simple
and
really
easy.
B
So
now,
how
do
we
use
our
data?
So
this
is
what
a
host
looks
like
when
when
it
gets
the
command
line
and
we
can
seal
it
in
the
yellow
file.
Let
me
take
this
data
and
we
can
transform
it
into
JSON
for
instable
and
to
do
this
process.
We
use
a
Python
script
that
goes
through
makes
api
calls
for
a
cube
api
server
and
then
it'll.
We
take
that
and
spit
it
into
a
JSON
file.
B
A
You
Miriam,
so
we
found
this
pattern
to
be
already
really
useful
right.
We've
spent
a
lot
of
time
on
how
we
use
it
for
inventory,
but
that
was
really
just
like
the
first
thing
as
soon
as
we
had
this
place,
where
we
could
easily
define
declarative
desired
States
for
our
cluster
a
whole
new
set
of
opportunities
opened
up.
A
A
Well,
that's
just
like
kubernetes
with
extra
steps.
Right
like
this
is
a
big
hassle.
So
instead
we
get
to
do
that
dynamically.
We
were
able
to
lift
up
all
that
monitoring
logic
into
the
API
server.
So
now,
as
soon
as
the
inventory
changes
the
host
objects
in
the
API
server,
the
monitoring
server
can
pick
that
up
right
off
the
data
plane
because
they're
listening
for
new
hosts
and
go
reconfigure
itself
right
on
the
monitoring
container
way.
Faster.
No
poll
requests
required
no
gate
connectivity.
A
It's
just
much
simpler
to
think
about
really
anything
that
needs
updates
about
what
goes
where
can
do
this?
You
know,
monitoring
is
a
great
start.
We
really
want
to
go
further
than
that.
So
you
know
it's
not
yet
declarative.
The
host
objects
that
we
have
are
just
really
status,
there's
not
really
a
spec
that
we
rely
on
for
much
yet
and
creating
a
new
one
only
happens
when
a
new
one
actually
appears
in
the
infrastructure
provider.
So
terraform
is
still
making
direct
calls
to
the
infrastructure.
A
There's
no
controller
for
building
new
instances
like
a
cluster
autoscaler
might
do.
Similarly,
ansible
has
to
just
like
go
reach
directly
out
to
these
machines.
It's
not
running
right
on
the
machines,
though
we
have
managed
to
actually
build
on
this
a
little
bit
and
we
now
have
a
component
that
lives
on
the
actual
hosts
that
monitors
for
when
changes
need
to
be
a
run
by
ansible
and
they
can
go
ask
for
ansible
to
be
run
on
them.
So
it's
like
a
little
half
step
towards
having
self-managed
clusters
and
we
expect
to
continue
to
evolve
that.
A
Please
ask
for
more
details
after
if
you're
curious
or
contact
us
online.
So
this
is
how
he
came
to
think
about
kubernetes.
You
need
a
lot
of
parts
to
make.
Kubernetes
go
all
those
features
you
saw
earlier
each
one
of
those
is
like
an
entire
software
project
in
and
of
itself
it's
like
a
hamburger
right.
You
need
a
lot
of
stuff
to
make
a
hamburger.
A
You
need
something
from
the
bakery
something
from
the
ranch,
something
from
the
dairy,
something
from
the
garden
and
they
all
come
together
and
make
something
sort
of
greater
than
the
sum
of
the
parts.
But
you
know
if
you
just
take
the
G's
G's
good,
just
on
its
own,
and
you
know
we
took
this
sort
of
cheesy
part
of
kubernetes,
this
sort
of
thing
that's
delicious
on
its
own
and
made
a
sandwich
out
of
it.
You
know
and
still
cheese
in
there,
it's
just
it.
It's
sort
of
differently
shaped
when
you
eat
it
all
together.
A
So
you
know
we
encourage
you.
It
was
so
valuable
for
us
to
look
at
these
parts
of
kubernetes.
Look
at
the
opinions
about
how
distributed
systems
work
that
are
baked
into
the
code
that
are
encoded
in
it
and
then
think
about
how
you
can
apply
that
to
your
own
systems.
We've
struggled
for
years
with
this
pattern
of
how
do
we
converge
lots
of
stuff
and
the
folks
that
build
kubernetes
have
too,
but
they've
come
up
with
ideas
that
are
better
than
we
and
we've
had
a
lot
of
enjoyment,
applying
them
to
our
world.
Thank
you.
A
A
A
Ideas
being
thrown
around
include
things
all
the
way
up
to
and
including
converting
all
of
the
core
kubernetes
objects
to
CR
DS
themselves
that
get
loaded
when
you
install
kubernetes
or
even
generally,
just
teasing
apart
more
of
the
API
component
from
kubernetes
core
objects,
you
know,
I,
don't
think
we're
running
at
cross-purposes
with
what
they're
trying
to
do
with
the
API
server
at
all
and
the
more
we
talk
to
them
about
it.
The
more
kind
of
useful
things
are
emerging
from
that
dialogue,
so
yeah.
We
think
this
will
totally
go.
Yes,.
A
A
A
How
would
that
go,
and
it
gave
us
an
opportunity
to
learn
the
system
we've
seen
other
teams
inside
New
Relic,
look
at
what
we've
done
here
and
go
I
want
that,
and
so
we've
already
got
a
team
using
it
to
manage
the
load
balancers
any
place
where
I
want
to
declare
state
in
some
flexible
way
and
then
serve
that
and
reify
that
in
in
actual
machines,
it's
proven
to
be
pretty
useful.
I,
don't
see
any
reason
why
you
couldn't
apply
this
to
non
containerized
systems.
A
Is
this
open
source?
No,
we
would
like
to
I
think
the
fetchers,
especially
for
things
like
AWS
and
IBM
cloud
are
pretty
like
you
could
make
those
generic
in
a
meaningful
way
right
now
we're
working
with
another
team
inside
New
Relic
to
get
them
to
use
this
as
well
for
their
large
cluster,
orchestration
and
I.
Think,
as
part
of
that
process
will
look
at
teasing
that
apart
because
I
don't
there's
no
reason
why
we
couldn't
make
these
open
they're
pretty
pretty
powerful.
So,
yes,
we're
working
on
that.
A
B
A
So
the
question
was:
how
do
we
deal
with?
Are
we
thinking
of
replacing
this
terraform
flow
or
we're?
Terraform
still
has
to
go
and
reach
out
and
make
things
without
being
queued
by
changes
to
the
CRT
that
reflect
the
question
so
because
of
our
migration
plans
off
DCOs
I.
Don't
think
we're
gonna
make
that
change
on
this
system,
but
when
we
start
building
kubernetes
clusters,
we
absolutely
want
to
do
that
and
so
we're
trying
to
get
really
plugged
in
with
the
cluster
API
folks,
because
they're
talking
about
a
design
for
managing
kubernetes
clusters.
A
That
is
totally
this
idea.
You
make
a
cluster
object,
then
you
make
controllers
that
create
that
actual
cluster
in
the
real
world.
That's
the
world.
We
want
to
operate
in
because
we
expect
at
our
scale
to
have
many
clusters
and
I
don't
want
to
manage
many
clusters
with,
like
a
whole
bunch
of
pull
requests
all
over
the
codebase
I
want
to
make
just
an
object
and
an
API
server
and
how
to
go
so
yeah.
That's
that's
where
we
want
to
be
I,
hope
to
come
back
next
year
and
be
like
check
it
out.
A
B
A
The
cluster
API
folks
have
a
whole
set
of
CRTs
that
they've
defined
that
are
pretty
close
to
what
we're
doing
here.
So
they
define
a
machine
object
and
a
machine
set.
We're
gonna
look
real
hard
at
that,
because
you
know
we
may
have
to
straddle
two
worlds.
We
may
have
a
host
inventory
that
has
both
machines
that
are
running
DCOs
and
also
machines
that
are
running
kubernetes
and
those
may
take
different
representations.
Honestly,
we
don't
know
we'll
see
as
for
what's
hard
about
moving
off
DCOs.
A
I
think
in
our
case
it's
not
actually
going
to
be
that
hard,
because
all
we
run
on
DCOs
are
stateless
services
and
those
are
pretty
easy
to
migrate
to
kubernetes,
I
think,
what's
going
to
be
hard
is
is
pulling
on
the
stateful
services
which
we
do
want
around
kubernetes
and
we're
very
early
in
that
process.
Still
so
we're
not
certain.
Yet
we've
made
it
easier
on
ourselves,
it's
not
shown
in
the
architecture
diagrams,
but
we
do
have
a
piece
of
software
that
sits
in
the
middle.
A
So
when
teams
deploy
their
services
that
go
through
a
piece
of
software
called
Grand
Central,
that's
what
actually
talks
to
DCOs
and
creates
the
applications.
We
expect
that
same
flow
to
happen
when
they
deploy
to
kubernetes
for
the
most
part,
and
so
we
just
changed
that
part
in
the
middle,
the
grand
central
part
and
have
it
just
deploy
to
another
cluster
and
have
that
go
which
makes
it
easy
for
the
service
owners
and
hard
for
us,
because
we
have
to
make
sure
that
both
of
those
things
work
well.
A
B
There
aren't
any
conflicts
because
they're
not
really
talking
to
each
other.
The
fetchers
are
deployed
in
DC
waves,
but
they're
just
communicating
on
the
cube
API
server
via
API
request,
it's
just
over
it
like
HTTP
request,
so
there's
not
really
any
like
fighting.
That
would
happen.
There's
no
situate
space
for
that
yeah.
A
We're
only
running
like
just
the
small
part
of
kubernetes
right,
there's,
no
CNI,
there's
no
cubelet,
there's!
No!
You
know
service
objects,
no
IP
mapping,
and
so
all
the
parts
of
kubernetes
that
sort
of
do
interesting,
work
that
might
be
in
conflict
with
DCOs
simply
aren't
running,
and
when
we
stand
up
communities
clusters
we
expect
them
to
be
on
separated
equipment.
We're
not
gonna
try
to
like
co-locate
those
I
think
that
would
end
in
tears
all
right.
One
more
question!
Oh
yes,
sorry.
A
B
Guess
it
depends
right,
cuz
we're
having
AWS
and
we
have
our
data
center.
So
for
our
data
center
we
would
just
change
the
role
like
in
the
API
calls
and
then
one
cube,
API
queries
that
provider
I
wouldn't
know
all
of
that
was
terraformed,
yeah
I
guess
you
would
just
it
would
end
up
in
AWS
the
API.
Somehow
and
that's
API
would
be
I
guess
the
API
is
really
a
source
of
truth
and
then
cubic
cubic.
Lee
I
saw
like
leaning
on
that
and
that's
where
it's
going
with
this
data.
A
Of
the
intelligence
about
what
a
host
does
in
our
system
is
determined
by
the
host
name,
so
that
gets
set
on
creation
of
a
host
and
then
almost
every
tool
that
sort
of
is
downstream
of
that
is
like
oh
you're,
one
of
these
I'm
gonna.
Do
that
thing
with
you
and
that's
done
not
for
any
particularly
good
architectural
reasons,
but
it
you
could
conceptually
it's
the
same
as
like
putting
a
bunch
of
labels
on
it,
which
is
how
that's
often
done
all
right.
That's
all
the
time
we
have.