►
From YouTube: kubectl apply, and The Dark Art of Declarative Object Management [I] - Aaron Levy, CoreOS
Description
kubectl apply, and The Dark Art of Declarative Object Management [I] - Aaron Levy, CoreOS
kubectl apply is a powerful and commonly used command meant for declaratively managing your applications. However, even if you are using this command today, you may still be surprised by how it functions. In this talk we will go over the inner-workings of the kubectl apply command, and discuss patterns for successfully managing your applications using declarative object management techniques.
We will cover areas such as the interplay between imperative commands (set, scale, edit, etc.) and declarative object management. The different types of patch strategies, and how object merges are calculated. As well as pros/cons of the various approaches, and some subtle gotchas you might run into.
When you leave this talk it will make sense when you describe that your application is managed as a declarative base, with replicas driven imperatively, by an autoscaler that is declaratively configured. In other words, Kubernetes magic.
About Aaron Levy
Aaron Levy is a software engineer at CoreOS, working on all things Kubernetes. He is also the lead maintainer of Bootkube, a kubernetes-incubator project that enables launching self-hosted kubernetes clusters.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Everyone
thank
you
for
coming.
You
know
almost
last
session
on
the
last
day,
so
I
appreciate
it.
Thank
you.
My
name
is
Aaron
Levie
and
today
I'm
here
to
talk
about
coop
control,
apply
and
the
dark
art
of
declarative
object
management.
If
any
of
you
saw
Bryan
grants
talk
earlier
this
morning.
Apparently
it
Skoob
control
is
the
correct
way
of
saying
it.
A
I
have
a
habit
of
saying
coops
ETL,
so
I
will
do
my
best
to
say
Cube
control
through
this
presentation,
but
let's
get
started
so
why
the
dark
art
and
part
of
it
is
because
if
you
do
Harry
Potter
references,
your
talks
get
accepted,
but
also
because
coops
ETL
apply
may
not
actually
behave.
How
you
expect
it
to
my
original
understanding
of
COO
control
apply.
Behavior
was
essentially
okay,
it's
taking
configuration
that
I
have
locally
and
it's
applying
it
to
the
cluster,
and
that
was
about
it.
A
And
if
you
look
at
the
help
text
of
the
apply
command,
it
says
something
similar
apply
a
configuration
to
a
resource.
This
resource
will
be
created
if
it
doesn't
exist
yet
perfect
talks
over
there
isn't
a
whole
lot
else
to
go.
There,
but
in
actuality,
there's
a
ton
of
behavior
underneath
coop
control,
apply
and
I
personally
didn't
really
understand,
most
of
it
and
so
putting
the
the
abstract
together.
A
For
this
talk,
there
was
a
ton
of
stuff
that
I
wanted
to
get
into,
and
then
what
I
realized
in
in
putting
this
presentation
together
is
that
there's
just
actually
too
much
to
cover
in
30
minutes.
So
I'll
do
my
best,
but
there's
going
to
be
some
homework
at
the
end
to
kind
of
dig
further
in
so
when
I
started
using
apply
more
heavily.
A
Most
of
these
were
my
fault,
because
I
didn't
actually
understand
the
underlying
behavior
of
apply
and
how
to
use
it
correctly
and
so
I
actually
began
digging
into
a
deeper,
because
I
wanted
to
be
able
to
predict
the
outcome
of
using
this
command
and
I
and
I
clearly
wasn't
able
to
so
what
I
wanted
to
know
was
you
know
how
our
fields
actually
calculated,
how
bout
the
patches
themselves
generated
and
then
how
do
I
predict?
What
the
final
object
will
actually
look
like
and
then
is
this?
A
Is
this
functionality
client-side
is
every
side
is
a
little
bit
of
both.
So
what
does
COO
control
apply?
Actually,
do
this
I
kind
of
pulled
and
paraphrased
from
some
other
documentation?
This
isn't
actually
the
help
text
for
the
command,
but
when
invoked,
it
does
a
three-way
diff
between
the
previous
configuration
that
you
had
the
input
that
you're
providing
and
the
current
configuration
of
the
object
itself,
and
then
it
uses
those
three
states
to
determine
how
to
modify
the
resource,
and
then
it
applies
those
changes
and
doesn't
overwrite
other
changes
to
properties
that
you
haven't
specified.
A
So
we'll
dig
into
this
a
bit
more.
So
those
three
states
that
I
just
mentioned
that
we're
calculating
kind
of
what
we're
going
to
do.
The
first
is
the
object,
configuration
file
and
you
could
think
of
this.
As
you
know,
if
you
have
a
git
checkout
of
your
actual
object
manifests.
You
know
that
you're
going
to
operate
on
locally
and
then
raghu
control
apply
on
those
actual
files.
Then
you
have
your
live
object
configuration
this
is
the
state
of
the
object
as
it
exists
in
your
cluster.
A
So
if
you
were
to
say,
coop
control
get
a
deployment.
Whatever
you
see
down,
that's
the
live
object.
What
we're
talking
about
and
then
the
last
applied
configuration.
This
is
the
state
that
the
last
time
you
invoked
apply
what
it
looked
like
at
that
time
and
between
these
three
states.
That's
what
you
use
to
kind
of
compute,
what's
actually
going
to
happen
when
you're
running
this.
A
So
if
we
look
at
an
example
of
this,
our
base
objects
a
pretty
simple
deployment.
If
we
were
to
run,
could
control
apply
on
this?
What
would
actually
end
up
in
the
live
object
in
the
cluster
is
that
it
would
inject
that
state
that
you're
pushing
into
the
cluster
into
that
special
annotation
into
that
last
applied
configuration
and
then
the
rest
of
your
object
exists
normally,
so
let's
go
over
an
example
of
adding
a
field
to
an
object.
A
A
This
is
an
annotation
that
we've
kind
of
pulled
out
just
to
make
it
easier
to
describe-
and
this
is
part
of
the
live
object
itself
and
then
the
live
object
is
just
the
rest
of
that
manifest
and
so
between
these
three
states
we're
going
to
determine
the
actions
that
we're
going
to
take,
and
in
this
case
we
decided
well,
we
already
have
this
object
in
the
cluster
and
the
live.
The
live
object.
We
want
to
add
a
new
field
to
it.
A
We
want
to
add
the
min
ready
seconds
to
it,
and
so
we're
going
to
you
know
on
our
local
copy
of
that
manifest
we're
going
to
add
that
field,
we're
gonna,
run,
cout
control
apply,
and
then
it's
going
to
determine
what
action
we're
going
to
take
based
on
these
other
states,
and
so
you
see
well,
it
exists
in
my
local
configuration,
but
it
doesn't
exist.
The
last
time
I
tried
to
apply
it
and
it
doesn't
exist
in
the
live.
A
Configuration
so
the
action
that
I'm
actually
going
to
take
here
is
I
want
to
add
that
field.
So
we
see
after
we
execute
this.
That
first
of
all,
the
live
object
reflects
what
we
wanted
it
to.
We've
said
we
want
to
add
this
field
and
then
last
supplied
is
now
set
to
what
we
just
pushed
into
the
cluster.
So
the
next
time
we
do
an
action,
we're
able
to
compute
the
next
state
that
we
want,
and
so
we
can
see
that
from
you
know
an
example
of
now
we
want
to
actually
remove
a
field.
A
So
in
this
case
we
see
that
in
our
live
object,
we
still
have
that
field
that
we
added
previously,
but
now
we
decide.
Actually
we
don't
want
that
field
in
it.
So
what
we
do
is
we
modify
our
local
object.
You
know
there
are
check
out
of
our
manifest
and
we
say
all
right.
Actually,
let's,
let's
delete
that
field
from
it
and
now,
when
we
run
Q
control
apply
what
it's
going
to
do.
Is
it's
going
to
look
and
say?
A
Well,
it
doesn't
exist
in
my
local
state,
but
it
does
exist
in
the
live
State
and
the
last
time
I
ran
apply.
It
existed
there
as
well,
and
so
this
is
actually
a
deletion
event.
It
used
to
exist
now
it
doesn't,
let's
get
rid
of
it,
and
so
we
can
see
after
running
it.
All
that
happens
is
the
live
object.
That
field
is
removed.
The
last
applied
annotation
is
updated
to
just
match
what
we
just
pushed
into
the
cluster,
so
we
can
create
objects
with
apply,
we
can
add
and
remove
fields.
What
else
why?
A
Why
do
we
want
to
use
apply?
So
another
really
powerful
aspect
of
using
apply
is
the
ability
to
kind
of
preserve
and
enforce
different
fields,
and
so,
when
I
say
enforce,
that
means
that
your
local
object
configuration
the
fields
that
you're
putting
in
there.
The
things
that
you're
populating
you're
essentially
saying
these
are
things
that
I
want
to
enforce
when
I
run,
apply,
I
want
to
set
them
to
these
values,
but
you
can
also
emit
certain
fields
from
that
and
not
have
them
controlled.
So
you
could
apply
the
object
and
not
update.
A
What's
on
your
live
object,
and
so
that
allows
those
other
fields
to
potentially
be
controlled
by
other
aspects
of
the
system?
A
really
good
example
of
this
is
auto
scaling
components,
so
an
example
would
be
you
know.
I
want
to
be
able
to
update
things
about
my
my
object.
I
want
to
you
know
change
labels
on
it.
I
want
to
change
the
containers
that
are
inside
of
it,
but
I
want
an
auto
scaler
based
on
some
some
metrics
to
be
scaling
this
up
and
down.
A
So
when
I
run
apply,
I,
don't
want
to
know
how
many
copies
of
it
are
running.
I
just
want
something
else
to
be
managing
that,
and
so
that's
an
example
of
how
we
would
say:
I
want
enforce
certain
fields,
but
I
actually
want
to
ignore
and
preserve
others.
So
it's
go
over
an
example
of
this,
so
what
we
see
is
in
our
live
object.
A
Replicas
is
actually
exists
within
the
object,
and
we
say
we
have
three
replicas
of
this,
but
that
same
field
doesn't
exist
in
our
local
object
and
it
doesn't
exist
in
the
last
applied
annotation.
So,
whenever
we've
pushed
updates,
this
object.
We've
said
nothing
about
replicas,
because
we
don't
want
to
manage
it
as
part
of
this.
This
piece
of
object
management.
We
want
something
else
to
deal
with
it.
So
then
let's
say
that
the
autoscaler
is
just
running.
A
Coupe
control
scale
commands
separately
from
us,
and
so
we
say
you
know
scale
my
app
deployment
up
to
replicas
the
the
count
to
five.
So
we
just
went
from
three
to
five
still
our
local
object
and
the
last
applied
annotation
don't
actually
change,
because
this
is
just
something
that
we're
essentially
preserving
and
ignoring
on
the
live
object
when
we're
calculating
any
kind
of
patches.
So,
let's
say
later,
you
know
I
want
to
deploy
a
new
version
of
my
application
and
so
I
check
out
my
copy
of
these
objects
and
I
say
well.
A
I
want
to
deploy
version.
Two
of
this,
so
I
update
my
local
copy
and
I'm
gonna
run
coupe
control
apply
and
what
ends
up
happening
is
that
in
my
live
object,
that
field
is,
is
updated
and
I
don't
touch
the
replicas
I.
Don't
need
to
know
about
them.
It
doesn't
matter
to
me
in
this
workflow
I
want
the
autoscaler
to
be
dealing
with
that.
So
we've
seen
how
coops
do
a
cou
control
apply,
I
can
add,
update
and
remove
and
also
preserve
fields
in
force
fields.
So
how
is
this
stuff
being
calculated?
A
How
are
we
deciding
that
some
fields
should
be
left
alone
and
others
should
be
updated?
Some
should
be
cleared,
there's
several
ways.
The
merge
actually
takes
place
so
for
primitive
values.
Things
like
strings
integers,
boolean,
x'
example.
Fields
of
this
would
be
like
the
image
field
is
a
string
or
the
replicas
is
an
integer.
The
actions
that
we
can
take
on
that
are
essentially
out
of
clear
or
replace
it
for
things
like
maps
and
objects.
Examples
of
these
would
be
like
labels
or
metadata,
or
maybe
the
deployment
spec.
A
The
action
on
this
is
actually
to
merge
the
elements
or
to
merge
the
subfields
of
the
objects
themselves
and
then.
Lastly,
we
have
lists-
and
in
this
case
like
you,
might
have
in
a
pod,
a
list
of
containers
or
you
might
have
a
list
of
ports
or
you
might
have
a
list
of
arguments
provided
to
a
container,
and
the
action
here
actually
depends
quite
a
bit
so
I
want
to
the
other.
A
Two
are
decently
well
understood,
but
I
want
to
jump
into
list,
because
this
can
get
a
bit
more
complex,
so
several
strategies,
one
is
just
that
the
list
gets
replaced.
The
entire
list
just
gets
replaced
and
another
is
that
you
can
actually
merge
the
elements
of
a
list
of
objects
and
we'll
go
into
both
of
these.
So
another
example.
This
is
a
list
of
primitive
values,
just
the
args
that
we're
passing
to
a
container,
and
in
this
case
all
three
of
our
states
are
slightly
different.
A
You
know
our
our
local
object,
our
desired
state
that
we're
going
to
be
pushing
into
the
cluster
says.
The
argument
should
be
a
and
C,
but
the
live
object
actually
has
a
B
and
D
and
the
last
time
we
applied
this.
It
was
an
V
when
we
run
qu
control
apply.
What's
going
to
happen,
is
we're
going
to
take
our
desired
state
that
local,
that
local
config
and
it's
actually
just
going
to
replace
args
wholesale?
There
is
no
merging
it's
not
we're
going
to
take
the
live
state
and
and
mix
them
together.
A
It
just
replaces
it
completely.
So
that's
an
example
of
just
that
one
option
now:
what
about?
If
we
have
a
list
of
more
complex
objects,
and
how
do
we
can
merge
them
together?
An
example
of
this
would
be
a
list
of
containers,
so
containers
being
a
more
complex
object,
and
our
goal
in
this
case
in
this
local
object
we
can
see,
is
that
we
want
to
be
able
to
add
a
sidecar
container
to
our
existing
application.
A
So
initially
we
just
had
just
the
app
itself
as
one
container,
and
we
can
see
that
by
the
last
time
we
applied
this
configuration
this
last
supplied.
The
only
container
that's
listed
is
app
the
live
object,
but
we
see
that
same
thing
app
exists.
There
we
also
see
another
field
has
been
added
kind
of
out-of-band.
This
is
one
of
those
fields
where
we're
not
managing
it
in
our
oblah.
In
our
object
configuration,
but
it
exists
there
in
live
and
now
we're
updating
to
a
new
desired
State.
So
our
local
object,
we
said.
A
Alright,
we
want
this
new
sidecar
container
to
exist.
So
when
we
apply
that
what
we
actually
see
is
that
the
sidecar
container
is
added
to
our
live
object
as
we
intended,
but
also
args
is
preserved,
which
didn't
exist
in
our
local
and
our
local
config,
and
so
what
you
see,
there's
we're
not
simply
taking
the
entire
list
of
containers
and
then
applying
it
and
replacing
it
on
the
on
the
live
object,
we're
actually
walking
into
each
of
those
objects
and
updating
the
individual
elements
and
updating
the
individual
fields
and
merging
them
together.
A
So,
let's
take
a
little
bit
more
complex
example,
so
we
just
saw
that
happen
with
containers.
But
let's
look
at
that
for
a
list
of
Toleration
x',
so
again
a
list
of
complex
objects.
We
should
expect
that
we
would
be
able
to
merge
these
as
an
action,
so
we're
gonna
step
through
this
kind
of
piece
by
piece
into
what
are
the
expected
actions
that
we
would
take
by
evaluating
these
three
different
states.
So
the
first
state
being
you
know
the
local
object.
A
This
is,
you
know
we
are
going
to
try
and
push
some
desired
state
into
the
cluster
and
and
what's
going
to
happen
there,
so
we
see
there's
a
toleration
with
a
key
of
Bath's
and
if
we
look
at
the
last
time
we
applied
this
configuration
to
the
object,
it
didn't
exist
there
and
if
we
look
at
the
live
object,
it
also
doesn't
exist
there.
So
my
expected
outcome
from
this
is
this:
is
we
should
add
that
toleration?
That's
the
action
that
should
come
out
of
this,
then
next.
A
If
we
look
at
the
last
supply
the
last
time
we
ran
this,
we
actually
told
it
that
we
wanted
a
toleration
with
a
key
of
foo
and
we
also
see
that
that
still
does
exist
in
the
live
configuration
but
that
no
longer
exists
in
our
desired
local
object
configuration
so
that
translates
to
me
is
okay.
Actually
we
should
be
deleting
this.
It
used
to
exist.
We
used
to
apply
it,
we
no
longer
do
let's
get
rid
of
it
and
then.
A
Lastly,
we
have
this
key
of
bar
that
exists
in
the
libel
project
and
this
isn't
specified
in
our
local
config.
So
again,
it's
not
something
that
we
want
to
manage
or
enforce,
and
so
essentially
we're
just
going
to
ignore
that
that
value
and
it
should
just
still
exist.
So
my
expected
outcome
of
this
would
be
that
we
have
two
toleration
x'.
We
have
a
toleration
for
a
kiev
bar
anna
toleration
of
baz.
You
know
we
saw
this.
A
We
containers
this
should
be
my
expected
outcome
of
merging
more
complex
objects,
but
instead
what
happens
is
when
we
run
apply.
It
actually
just
sets
the
live
state
to
exactly
what
we
had
in
our
local
config.
So
it
actually
does
a
replacement,
it's
not
merging
the
objects
and
then
our
last
supplied
is
also
updated
as
well.
So
we
expected
to
see
the
Toleration
merge
they
weren't
what
you
know.
Why
did
this
happen?
A
We
saw
the
exact
completely
different
behavior
on
something
similar,
so
I
want
to
briefly
go
over
the
ways
that
patches
are
calculated
here,
so
the
two
that
apply
actually
makes
use
of
our
json
merge
patch,
which
is
an
actual
spec.
You
can
go
read
about
the
exact
behavior
of
it,
but
in
this
case
it's
something
called
strategic,
merge
patch,
which
is
custom
to
kubernetes,
and
it
deals
with
handling
how
to
merge
lists
of
complex
objects
in
when
you're
running
applied.
A
So
with
a
strategic,
merge
patch,
you
can
treat
a
list
much
like
a
map
and
you
can
merge
specific
elements
of
that
list
based
on
a
predefined
patch,
merge,
key
and
then
based
on
that
key
you
can
individual
elements
can
either
be
added
or
update
or
removed,
and
this
key
itself
is
actually
defined
on
a
per
field
basis,
and
this
exists
in
the
kubernetes
source
code.
So,
generally,
you
have
to
look
this
up
directly.
You
can
look
in
the
source
code
itself
to
find
it
or
any
api
reference.
This
is
also
bubbled
up.
A
You'll
see
these
these
patch
merge,
keys
and
I'll
show
an
example
of
this.
So
if
we
were
to
dig
into
the
code-
and
we
were
to
look
at
this-
is
just
a
stub
of
what
the
pod
spec
looks
like,
we
would
see
that
as
part
of
the
pod
spec,
we
have
a
list
of
containers
and
there's
this
metadata
associated
with
it.
That
says
the
match,
merge
key
is
named,
and
so
what
we
saw
before
was
that
we
had
these
two
containers.
A
One
of
the
containers
had
a
name
of
app
and
one
of
the
containers
had
a
name
of
sidecar,
and
so
we
were
able
to
enter
the
list
and
merge
and
look
at
the
states
and
say
well.
The
live
object
has
and
a
container
with
name
app
and
my
local
configuration
the
name
of
app.
Let's
merge
those
objects
because
of
the
same
one,
even
though
they're
their
list,
but
if
we
look
at
toleration
z--,
it
doesn't
have
that
metadata,
and
so
we
know
that.
Well,
it
doesn't
know
how
to
inspect
the
complex
object.
A
It's
just
going
to
replace
the
entire
list
wholesale.
But
if
we
look
at
another
one
like
volumes,
we
also
again
see
that
that
metadata
added.
So
we
would
know
well
if
we
wanted
to
update
volumes,
we
know
they
would
be
merged
together.
It
wouldn't
just
be
replaced,
and
this
is
you
know,
important
to
know,
because
it's
it's
changing
the
behavior
of
what
your
local
configuration
should
be
in
one
case,
you're,
going
to
always
need
to
specify
every
Toleration
that
you
want
and
it
has
to
be
managed
wholesale.
A
So
a
little
bit
of
complexity
here
in
like
the
actual
behavior,
is
not
super
consistent
in
some
cases
and
there's
there's
more
complexity
underneath
this
so
another
one
that
I
want
to
mention
is
actually
defaulting
fields
of
objects.
So
an
example
of
this
is
a
deployment
object.
For
example,
even
though
you
might
have
a
very
simple
manifest
itself
when
you
apply
that
that
object
to
the
cluster,
the
API
server
is
actually
going
to
default
certain
fields
for
you,
and
so,
if
you
don't
specify
replicas,
then
it's
going
to
default
to
a
count
of
one.
A
If
you
don't
specify
an
update
strategy,
it's
going
to
default
to
a
rolling
update
strategy
and
in
some
cases
these
defaulting
values
can
actually
cause
some
problems
when
you're
interacting
with
that
object
with
apply.
So
an
example
of
this,
is
you
know
the
local
objects,
pretty
simple:
we
don't
define
a
replicas,
we
don't
define
a
update
strategy,
but
when
we
actually
apply
this
into
the
cluster,
those
fields
are
created
and
so
they're
defaulted.
A
So
we
actually
will
see
in
the
live
object
if
we
were
to
just
say,
cout
control
get
replicas
of
one,
and
then
the
strategy
is
going
to
be
rolling
update
and
then,
as
part
of
rolling
update,
there's
more
subfields
that
are
defaulted
like
Max,
surged
and
max
unavailable.
This
becomes
problematic
because,
let's
say
that
at
some
point
in
time
you
say
well:
I
want
to
change
what
the
update
strategy
is.
I
want
to
change
it
to
recreate
instead
of
rolling
update,
so
the
the
process
would
be.
Will
I
just
add
that
field?
A
A
So,
even
though
we're
changing
the
type
there's
other
subfields,
like
Mac,
Serge
and
Mac's
unavailable
and
the
rolling
update
key
itself
that
haven't
changed,
and
so
you
would
actually
this
would
fail
if
you
were
to
try
and
apply
this
configuration,
so
it
can
kind
of
cause
some
problems,
and
it
takes
some
little
bit
of
foresight
and
thinking
to
be
like
well.
If
I
might
want
to
manage
these
fields,
I
should
probably
explicitly
define
them
rather
than
relying
on
defaulted
fields.
A
A
So,
coming
back
to
this
example
of
the
my
app
deployment
in
this
case,
we're
gonna
have
two
users
that
are
interacting
with
this
object,
and
one
thing
I
just
want
to
know
is
that
we're
not
going
to
specify
the
replicas
in
this
again
we're
gonna,
say
an
auto
scaler
is
going
to
manage
that
we
don't
want
it
as
part
of
our
local
configuration,
so
the
first
user
they're
going
to
initially
create
this
application.
Coupe
control
apply,
creates
it
over
time.
It
might
be
scaled
up
manually.
A
It
might
be
scaled
up
by
the
autoscaler
KooKoo
control
scale.
My
app
set
the
replicas
to
3
later
you
know,
we
want
to
bump
it
to
a
new
version,
and
so
we
get
our
check
out.
We
change
the
image
from
Oh
No
OH
OH,
and
then
we
push
that
into
the
cluster.
Workflow
works
fine,
everything's
great
later,
a
second
user
needs
to
add
a
volume,
but
for
whatever
reason
they
don't
have
the
same
local,
checkout
and
they're
just
like.
Well,
it's
a
quick
change.
I
just
need
to
make
this
right
now.
A
I'm,
just
gonna
get
a
live
copy
of
the
act
of
the
object
itself.
You
know
it's
it's.
What
the
current
state
is,
I
may
as
well
just
grab
it
so
you'd,
say:
coop
control
get
deployment,
we
output
it
to
a
local
file,
and
then
we
edit
that
file
directly
and
we
add
the
volumes
and
then
you
know,
user
to
things
Aaron
said
use,
apply
so
I'm
going
to
apply
that
manifest
to
the
cluster.
A
Everything
works,
fine,
it's
you
know
it
has
any
volume,
nothing
changed
later
say
a
week
from
then
user
ones
needs
to
bump
the
image
version
again.
They
you
know
check
out
their
manifest.
They
changed
the
image
version
they
commit
it.
They
apply
it
to
the
cluster.
What
just
happened
is
that
user,
one
inadvertently
reset
the
replicas
back
down
to
one
and
they
removed
the
volume
and
they
removed
other
fields
as
well.
So
if
we
look
at
how
that
actually
happened,
what
what
we're
looking
at
is
this
last
applied.
A
This
middle
column
is
essentially
the
outcome
of
what
user
two
did
and
what
they
did
is
they
pulled
a
live
copy
of
the
the
object
which
contained
all
fields,
whether
they
were
originally
specified
or
not,
so
it
contained
replicas
that
contain
the
volumes
it
contained
the
update
strategy,
everything
they
pulled,
that
local
they
made
their
edits
and
then
they
said,
apply,
which
means
that
all
of
those
fields
now
exist
in
the
last
applied
annotation.
So
now,
they're
managed
now
they're
enforced.
A
So
now,
when
user
one
comes
back-
and
this
is
their
view-
essentially
they
their
local
object
doesn't
have
any
of
those
fields.
It
doesn't
specify
replicas,
it
doesn't
spy
to
specify
the
volume
and
they
just
say,
apply,
and
so
these
all
turn
into
deletion
events,
because
they
say
well.
The
last
time
you
apply
this
a
volume
was
present,
a
rolling
update
was
present,
a
replica
count
was
present
and
now
they're.
Not
so
we
should
delete
all
those
fields.
When
you
delete
the
replica
field,
it
defaults
back
to
one.
A
We
delete
the
volumes
that
were
added
by
user
two,
and
none
of
this
was
intended.
So
the
important
thing
to
note
here
is
user
ones,
workflow
didn't
change
at
all.
They
did
the
exact
same
thing
every
single
time
they
had
a
source
config,
they
modified
it
and
they
used
apply.
And
what
happened,
though,
is
even
though
their
workflow
didn't
change.
The
there
didn't
get
the
expected
outcome
and
you
could
think
of
user
1
is
actually
being
a
CI
CD
system.
A
There
is
no
behavior
change,
it
is
told
this
is
the
tag
that
you
should
check
out
and
then
you
should
apply
those
manifests
of
the
cluster,
but
inadvertently.
Another
user
has
actually
changed
that
workflow.
So
it's
pretty
important
to
kind
of
consider
that
mixing
these
styles
of
object
management
can
lead
to
these
unintended
outcomes.
A
So
brief
list
of
recommendations
based
on
some
of
these
things,
one
in
general,
you
probably
don't
want
to
actually
define
replicas
as
part
of
your
objects,
because
it's
probably
better
managed
separately
when
you're
making
updates
to
your
object.
You
probably
don't
want
no,
how
many
replicas
are
actually
running
in
the
cluster
there's
times
where?
Maybe
it's
going
to
be
three
copies?
That's
never
going
to
change!
A
A
Well,
how
many
replicas
do
you
have
right
now,
so
that
I
can
update
this
local
manifest
so
that
when
I
push
it
back
to
the
cluster,
that's
preserved,
and
this
also
goes
to
other
fields
that
you
might
want
externally
managed,
there's
some
other
ones
where
it's
like
the
load:
balancer
IP,
for
example,
on
a
load
balancer
type
service.
This
isn't
something
that
you
are
managing.
This
is
something
that
cloud
provider
code
is
managing.
A
These
can
lead
to
unintended
consequences
unless
you're
really
sure
of
what
you're
doing
so
looking
into
the
actual
behavior
and
knowing
it's
okay,
that
I
set
this
particular
field
because
I
know
it's
not
going
to
be
managed.
Declaratively
in
this
configuration
that
I
have
locally
and
then
I
know
time
wise,
which
actually
I
talked
way
faster.
This
time,
but
things
we
didn't
get
to
cover
but
I
do
want
to
briefly
go
over.
Is
that
there's
also
cou
control
apply
prune
and
this
gets
into
declarative
object
deletion
itself?
A
Later
you
decide
well
object,
C
is
deprecated,
I
want
it
removed
the
cluster
rather
than
you
having
to
decide.
Oh
I've
got
to
go,
run,
coop
CTL,
delete
objects,
see
you
just
remove
it
from
that
directory
and
you
run
the
exact
same
command
again.
You
just
say:
coop
control
apply
prune
and
it
looks
at
the
state
of
the
cluster
in
the
live
state
of
the
cluster,
and
it
says
well,
I
see
this
object,
see
and
I
don't
see
it
in
the
local
configuration.
A
So
that's
a
deletion
event,
but
this
is
something
that
you
should
be
careful
about
using
definitely
look
deeply
into
its
behavior,
because
you
could
inadvertently
delete
a
whole
bunch
of
objects
that
you
actually
don't
want
to
another
really
interesting
one
is
you
can
set
the
override
flag
to
false
when
you're
running
could
control
apply?
So
the
typical
behavior
is
to
set
this
to
true,
and
that
means
that
you're
generally
a
desired
outcome.
A
Is
that
your
local
configuration
state,
the
local
file,
is
what
you
want
to
end
up
in
your
live
object,
but
you
can
actually
have
it
calculate
whether
this
might
have
been
a
conflict
that
the
live
changed
underneath
you
from
the
state
of
the
last
time
that
you
applied
from
the
state
that
you
have
locally
and
it
can
report
back
in
some
cases.
That's
nice,
because
it's
kind
of
a
signal
that
hey
wait.
A
So
there's
some
helper
commands
sub
commands
of
apply
so
that
you
can
view
set
or
edit
that
annotation
again
be
aware
that
when
you're
changing
this
annotation
you're
changing
behavior
when
you
the
next
time
you
apply
to
this
and
then
similarly
the
patch
command
itself,
the
underlying
functionality
of
apply
is
that
it's
calculating
patches
and
applying
them
against
objects
in
the
cluster.
You
can
also
do
that
yourself.
You
can
actually,
you
know,
formulate
a
patch
and
apply
it
against
the
object
itself
outside
of
apply
as
well.
A
So
I
did
have
a
pop
quiz
to
see
if
we
actually
understood
the
three
different
states
and
how
they
go
through,
but
instead
I'll
just
give
homework.
The
first
link
is
the
most
important.
This
goes
into
everything
that
I
just
talked
about,
plus
a
lot
more
depth,
plus
different
styles
of
object
management,
so
the
imperative
style
object,
management,
big,
shout
out
to
Phillip,
whit
Rock,
who
led
the
documentation
effort
on
this
and
everyone
who
worked
on
it.
It's
incredibly
helpful
before
this
existed.
It
was
kind
of
a
guessing
game,
at
least
for
me.
A
A
So
this
is
getting
into
some
of
the
implementation
details
of
how
apply
functions
and
how
it's
communicating
with
the
API
server
and
how
you
can
extend
the
kind
of
patch
strategies
that
are
implemented
but
really
interesting
information
and
read
into
that
like
to
kind
of
see
where
the
future
of
this
stuff
is
going
and
then
another
really
interesting
document
is
Bryan,
Grant's,
declarative
application
management.
So
again,
how
we're
talking
here
about
objects?
A
It's
taking
these
kinds
of
concepts
in
expanding
into
how
could
we
generically
manage
generically
and
declaratively
manage
applications,
so
it
would
be
multiple
objects
and
services
and
control,
config
maps
and
deployments,
taking
these
same
kind
of
concepts
of
these.
These
patches
in
these
overlays
on
objects
and
then
just
using
all
native
kubernetes
to
manage
applications
themselves
really
long,
really
really
interesting,
and
then,
as
part
of
that,
there's
another
document
about
the
issues
that
are
related
to
that
effort.
A
But
a
lot
of
these
issues
have
to
do
with
behaviors
of
coop
CTL,
apply
and
so
they're
applicable
if
you're
kind
of
digging
into
the
behavior-
and
you
want
to
know
some
of
the
nuances
or
some
of
the
edge
cases
of
the
problems.
This
is
discussing
those
states
so
that
they
can
kind
of
be
put
on
the
roadmap
and
addressed
anyway.
That's
all
I
have
thank
you
so
much.
B
B
So
the
question
is
like
basically
about
the
mechanics,
how
cube
City
works.
So
the
question
is
whether
it
supports
custom
x
like
CD
types
or
custom,
a
Passover
like
Service
Catalog
types,
for
example,
or
if
you,
if
I,
build
my
own
a
book,
whether
it
be
a
server
or
CD
controller,
with
defining
my
own
types,
will
I
be
able
to
annotate
my
types
this
way
or
are
they
built
into
the
cube
city?
I
mean
basically
like
whether
there
is
a
metadata
API,
including
this
API,
which
passes
this
metadata
back
to
cube
CTO
right.
So.
A
Is
anyone
from
API
machinery,
and/or
sig
CLI
in
the
room,
because
I'm
not
sure
if
those
mechanics
are
exposed
to
things
like?
If
you
would
be
able
to
import
that
into
your
own
objects,
you
could
I
mean.
Essentially
all
that's
happening
underneath
is
like
patch
is
being
calculated
and
then
being
communicated
to
the
server.
So
as
long
as
it
matched
those
same
semantics,
I
believe
so,
but
I'm
not
positive
on
them.
B
Yeah,
so
the
question
is
just
because
I
have
seen
with
these
some
changes
like
there
was
a
stream
of
changes
to
make
it
less
core
specific
into
serial
to
move
it
back
to
the
API
server
to
pass
the
metadata
from
service,
so
there
with
API,
aggregator
and
other
things
cubes.
They
would
support
like
generically
such
things
like
that.
But
I've
seen
the
cases
where,
when
you
every
Gator,
for
example,
some
API
mention
of
your
stuff
edit.
The
support
for
that,
but
because
cube
CTL
haven't
been
changed
to
use
that
meta
information.
A
So
I
from
just
some
discussions
that
I've
had
it
sounds
like
the
near-term
direction.
One
option
is
that
start
with
moving
all
of
the
the
calculation
logic
into
the
client
side,
so
that
you
would
actually
calculate
the
final
state
of
the
object
and
instead
of
using
a
patch
request,
you
actually
just
use
a
put
request,
and
so
it
would
be
okay,
let's
do
all
calculation
locally,
get
the
final
state
of
the
object
and
push
that
to
the
cluster
versus
calculate
the
patch
object
locally
and
then
communicate
that
with
the
patch
tile
to
the
API
server.
A
So
that
would
be
a
first
step
and
if
it
was,
if
that
was
the
case,
then
on
your
own
custom
types,
you
would
just
do
the
same
kind
of
thing
like
you
would
put
the
end
state
that
you
desired
then
pass
that
point.
Another
thing
that
I've
heard
discussed
is
implementing
this
logic
on
the
API
server
side,
except
all
you,
as
a
user
doing
or
providing
those
three
states
explicitly
to
the
API.
So
you're
saying
this
is
the
local
object.
A
This
is
the
the
last
applied
and
then
the
server
obviously
has
the
the
live,
and
then
all
it
does
is
it
calculates
all
of
it
on
server
side
and
it
doesn't
know
any.
There
is
no
patch
communication
at
all.
It's
just
calculating
it
locally,
but
that's
part
of
that
kind
of
v2
discussion.
There's
a
couple.
People
I
can
try
and
connect
you
with.
It
would
probably
be
able
to
answer
your
question
a
bit
better
yeah.
Thank
you.
Yeah.
D
If
I
understand
this
correctly,
if
you're
running
on
the
slide,
where
you
have
the
replicas
that
were
not
previously
defined
and
are
now
getting
them
and
you're
getting
an
error,
I
believe
it's
what
it
is
you're
changing
the
the
update
strategy
right.
It
is
so
could
you
work
around
this
error
by
somehow
updating
last
applied
weather
by
weather
by
patch
or
apply
or
the
sub
command,
to
include
the
update
strategy
in
the
last
applied
and
then
change
it
absolutely.
A
That
would
be
the
exact
way
that
you
would
get
yourself
out
of
this
situation,
and
so
that
that
applies
sub-command
of
cubes
detail
apply
edit
last
supplied.
You
could,
then
it
would
pull
it
up
in
an
editor,
and
you
could
add
those
fields
into
the
last
supplied
so
that
next,
when
you
went
and
applied
this,
it
would
know
that
it
was
actually
deleting
all
of
those
subfields.
So
that
would
be
exactly
how
you
get
yourself
out
of
that
situation
and.
D
A
Changed
over
time,
I
don't
know
if
it's
more
helpful
now
once
upon
a
time
it
was
kind
of
confusing
but
I.
Think
now
it
actually
like
that.
It's
you
know.
This
field
is
incompatible.
I,
think
it
says
something
along
the
lines
of
like
strategy
type
recreate,
isn't
compatible
with
rolling
update
or
something
of
that
sort,
but
it's
still
not
immediately
clear
you're
like
okay
cool
I'm,
trying
to
change
it.
So
how
do
I
get
out
of
that
state
which
it's
not
immediately
clear?
A
C
You
yep
so
two
questions.
One
is:
is
there
a
mechanism
to
do
like
a
dry
run
or
a
preview
to
see
what
it
would
change
before
applying
the
patch
and
second
question,
given
what
Kelsey
said
about,
like
you
know,
maybe
we
shouldn't
be
giving
developers
Cube
could
alter
run
in
the
first
place.
Is
it
a
good
idea
to
use
this,
given
all
the
semantics
in
your
opinion?
Well,.
A
So
so
to
answer
your
second
question:
first,
if
you
think
of
like
from
Kelsey's
presentation
that
that
was
actually
a
CI
CD
system,
underneath
it
I'm
pretty
sure
he's
probably
running
applies.
So
you
know
you
have
your
manifest
and
it's
tweaking
the
image
field
and
then
a
CI
CD
system
is
going
to
apply
that
or
it's
going
to
patch
it.
The
concern
is
what,
if
that
user
too
comes
along
and
then
does
it.
Your
CI
CD
system
is
completely
correct
and
it's
doing
the
right
thing
over
and
it's
repeatable
and
yeah.
A
The
user
shouldn't
probably
be
doing
anything
else.
It's
that,
like
okay,
well
being
knowledgeable,
that
there's
real
reasons
why
you
shouldn't
be,
you
know
doing
that
directly
or
you
need
to
have
very
like
clear
understanding
of
it.
So
I
actually
agree
like
in
that
approach.
It's
just
also
worrying
that
someone
with
the
best
intentions
could
still
mess
up
that
system
and
then
repeat,
your
first
question.
Sorry
is.
C
A
To
my
knowledge,
it
would
take
a
couple
round
trips
for
it
to
actually
like
say
this
would
be
what
we
would
have
done,
but
I
think
that
if
either
the
code
moved
all
the
way
to
the
client
and
the
only
thing
that
it
was
doing
was
a
put
that
would
be
much
easier
to
do.
Dry
runs
on
or,
conversely,
if
it
was
an
implemented
server
side,
but
that
code
at-least
of
how
it
takes
those
three
states
and
computes
a
final
state
that
at
least
would
get
us
closer,
I.
Think
yeah
thanks.