►
From YouTube: Kubectl Apply 2019: Defense Against the Dark Arts - Phillip Wittrock & Jennifer Buckley, Google
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Kubectl Apply 2019: Defense Against the Dark Arts - Phillip Wittrock & Jennifer Buckley, Google
Kubectl Apply (loved and hated by users) is the canonical way to manage Kubernetes Resources, and the building block for systems performing continuous delivery. Originally written in 2015, the inner workings of Apply have been referred to as the “dark arts” and have been accused of “making peoples eyeballs bleed”. The Apply maintainers took this feedback, and embarked on a journey to completely rearchitect Apply, this time without unleashing a primordial evil. In 2019 Apply received its first major architectural overhaul, addressing long standing issues and bringing a cornucopia of new functionality to users and platform developers. Phillip Wittrock and Jenny Buckley will review both the changes in architecture and the new functionality that will usher in a delivery renaissance for Kubernetes.
https://sched.co/MPZ1
A
Kubernetes
is
a
declarative
system,
but
that
declarative
system
is
primarily
managed
through
imperative.
Api
Slyke
create
update,
delete
and
patch
and
lots
of
users
wanted
a
more
declarative
way
to
manage
their
clusters
by
defining
a
local
state
and
running
something
to
push
that
state
into
a
cluster.
A
A
A
You
might
just
think
that
we
could
naively
replace
the
state
in
the
cluster
with
the
users
local
state
when
applies
run,
but
that
doesn't
actually
work,
because
the
control
plan
is
going
to
add
some
state
of
its
own.
Every
time
you
make
a
request.
For
example,
when
you
create
a
service,
a
cluster
IP
field
is
added
that
allows
your
cluster
IP
to
be
known
and
if
you
just
naively,
replace
the
state
in
the
cluster
with
the
local
state.
That
field
is
going
to
get
wiped
out
and
that's
not
good,
and
there
are
other
fields.
A
Besides
this
one
and
they're
set
by
admission,
plug-ins
and
web
hooks
and
defaulting.
So,
instead
of
sending
a
replace
request,
we're
going
to
do
a
patch
and
that's
going
to
preserve
those
fields
that
we
don't
know
about,
and
the
patch
is
generated
from
the
local
state
in
the
system.
The
last
applied
state
and
also
the
current
live
state
in
the
cluster.
A
A
B
Okay,
so
what
Jenny
just
finished
talking
about
is
challenges
when
you're
trying
to
take
the
resource
config
for
one
resource
and
push
those
changes
into
the
cluster
there's
another
set
of
challenges
with
declarative
application
management,
which
is
about
the
authoring
of
the
resource
config
itself.
So
the
vision
is
that
users
can
author,
all
their
all
their
config.
Declaratively
put
it
in
this
nice
directory.
You
can
write
to
control,
apply
on
that
directory
and
it
will
just
magically
make
the
resources
appear
in
the
cluster
right.
If
they
need
to
be
created,
it
will
create
them.
B
If
they
need
to
be
updated,
it
will
update
them,
and
if
you
change
those
resources,
then
it
just
gets
pushed
so
there
are
actually
a
couple
challenges
with
this.
The
first
is
around
config
Maps
and
secrets,
so
config
maps
and
secrets
are
oftentimes
derived
from
some
other
source
of
truth.
The
secret
is
really
comes
from
some
key
file.
The
config
map
may
come
from
some
environment
file,
and
so
what
a
user
wants
to
do
is
change
the
environment
file
or
the
key
file
and
then
have
that
pushed
into
the
cluster.
B
Another
challenge
is
when
you
push
a
config
map
or
secret
change
to
actually
consumed
by
pods
and
those
pods
might
consume
it
as
environment
variables,
whereas
command-line
arguments
and
pushing
that
config
map
or
secret
change
doesn't
necessarily
do
anything
to
those
pods.
And
so
you
won't
see
that
change
reflected
right
and
what
music
the
way
users
are
used
to
rolling
out
changes
to
deployments,
for
instance,
as
they
push
a
change
to
the
image
or
something
like
this,
and
they
need
to
get
a
nice
rolling
update.
B
So
what
did
we
tell
users
to
do?
Well,
one
recommendation
we
gave
is
create
a
new
config
map,
a
new
secret
when
you
make
changes
those
files
right
and
so
now
you
change
your
deployments
to
point
to
those
new
config
maps
and
new
secrets
right
and
then
you
actually
get
a
rolling
update,
because
there
is
a
change
to
the
deployment
itself,
so
this
is
actually
just
really
easy,
not
impaired
at
all.
You
just
write
these
imperative
scripts
to
regenerate
these
things,
use
the
hash
suffix
flag
to
make
sure
you
get
a
unique
name.
B
B
Another
issue
has
to
do
with
metadata
and
in
fields
that
are
set
across
many
resources
and
a
common
desire
is
to
perhaps
set
an
annotation
across
a
collection
of
resources
where
all
those
resources
are
part
of
a
single
release,
and
you
want
to
annotate
them
with
release
and
you
don't
want
to
go
update
every
single
resource
file.
You
just
want
to
update
one
place
that
says:
here's
the
new
annotation
labels
are
similar
to
this
and
and
especially
namespaces
right.
B
If
you
have
a
collection
of
resources
that
you
know
what
you
want
to
go
into
the
same
rese
namespace
like
services
that
prefer
to
pause
from
deployments,
you
don't
want
to
have
the
possibility
of
introducing
this
error
where
you
leave
the
namespace
off
one,
but
not
the
other,
or
you
copy
and
pasted
one
that
have
the
namespace
set
to
a
different
thing
right.
This
is
just
a
way
of
introducing
air
into
your
system,
and
so
there
was
the
in
flag
introduced
a
coop
control
apply.
B
So
this
allowed
you
to
apply
a
set
of
resources
to
a
single
namespace,
but
that
data
isn't
captured
declaratively
anymore.
It
exists
in
some
script,
some
other
place
right,
and
so,
if
you
just
look
at
the
resource
config
that
may
be
checked
in
to
get
where
these
things
actually
getting
applied.
Well,
you
have
to
look
someplace
else.
B
So
what
do
we
decide
to
do?
We
recognize
that
there's
two
parts
of
this
problem.
One
part
of
the
problem,
is
about
taking
a
single
resource,
pushing
updates
into
a
cluster
for
that
single
resource,
the
complexity
surrounding
that
the
ability
for
other
tools
and
controllers
and
other
systems
to
take
advantage
of
that
logic
rather
than
a
single
resource
basis.
B
A
second
problem
was
about
users
actually
managing
their
resource,
config,
declaratively,
right
and,
and
how
can
they
do
updates
to
secrets
and
do
updates
to
config
maps
and
Express
this
higher-level
logic
that
we've
already
put
into
coop
control
as
imperative
commands.
So
we
already
know
that
users
want
high-level
access
to
this
sort
of
stuff,
but
we
haven't
provided
a
nice
mechanism
from
the
declarative
way,
I.
A
Okay,
so
for
the
first
of
those
projects,
what
did
we
change
in
server
and
client
relationship?
We
used
to
require
a
client
that
wanted
to
use
the
supply
operation,
to
maintain
a
last
applied
configuration
annotation
and
to
do
some
client-side
dipping
and
to
generate
a
complicated
strategic,
merge,
patch
request
and
in
practice
no
one
was
doing
that
implementation
themselves.
They
would
just
exec
out
to
coop
control
ply
and
it
caused
lots
of
problems
that
we
already
talked
about.
A
A
You
can
just
send
your
configuration
for
a
single
resource
directly
to
the
server
and
it
will
understand
what
you
intended
to
change
and
instead
of
you
managing
that
annotation,
the
cluster
is
going
to
manage
a
new
field
called
the
manage
fields
in
metadata
that
will
keep
track
of
the
last
user
to
update
each
field
in
the
object.
So,
by
moving
all
this
work
and
logic
into
the
server,
it
allows
tools,
besides
cube
control
and
controllers,
now
to
take
advantage
of
this
workflow
and
have
greater
consistency
between
them.
A
So
this
was
pretty
good.
It
worked
pretty
well,
people
could
see
what
was
going
to
happen
when
they
ran
apply
without
it
actually
changing
their
clusters
configuration,
but
it
wasn't
perfect
and
the
reason
is
that
the
server
actually
changes
your
request
before
it
gets
committed
to
the
storage.
It
has
admission,
plug-ins
and
user-defined
web
hooks
and
also
defaulting
that
all
change
your
resource
before
it
gets
committed.
A
So
when
we
move
dry
run
to
the
server
as
a
query,
parameter
on
all
the
modifying
endpoints
like
create
or
patch
or
delete,
what
we
found
is
that
it's
more
accurate
than
it
was
when
we
did
it
on
the
client
and
when
we
combined
the
apply
endpoint
with
the
dry
run
parameter.
It
creates
a
more
accurate
dipping
tool,
cube
control,
diff
and
that's
going
to
you,
allow
users
to
see
exactly
what's
going
to
change
on
the
cluster,
the
next
time
they
run
apply,
and
so
they
won't
have
any
surprises
or
unexpected
changes
to
their
cluster.
A
And
the
last
thing
we
changed
is
those
implicit
patch
directives
that
the
server
sets
on
the
patches
that
you
send
are
being
replaced
by
a
more
declarative
structure
that
is
defined
for
the
schema
of
the
resource
and
it's
useful
for
things.
Besides
cube
control
apply
now
too
it's
useful
for
CRD
field
pruning
and
different.
B
Okay,
so
now
what
did
we
do
on
the
client
side
to
address
the
challenges
with
authoring
the
config?
So
we
came
up
with
a
new
declarative
file,
and
this
is
a
meta
resource
that
isn't
actually
applied
itself
to
the
cluster
but
describes
how
to
apply
resources
right
in
the
the
coop
config
file
that
describes
how
to
talk
to
a
cluster
is
another
example
of
something
like
this,
and
so
what
I've
shown
here
is
part
of
a
customization
EML
file,
and
so
the
first
thing
we
did
is
well.
B
B
So
the
deployments
and
the
staple
sets
and
the
different
workloads
that
went
to
consume
this
config
map
can
do
so.
Referring
to
that
app
name
right,
the
app
config
name
and
then
customize,
because
it's
schema
where-
and
it
knows
what
a
reference
to
a
secret
looks
like,
and
it
knows
what
a
reference
to
a
config
map
looks
like
and
because
it
has
a
visible
view
into
all
of
the
resources
that
are
being
customized.
B
It
can
go
then
change
the
references
to
those
right,
and
so
it
can
substitute
the
app
config
name
with
the
hash,
suffix
name
right,
and
so
this
allows
you
to
get
that
rolling,
update
behavior.
That
would
kind
of
been
asking
users
to
manually
do
the
second
entry
is
for
a
secret.
This
is
very
similar
to
a
config
map.
It
looks
basically
the
same
but
generates
secrets,
and
in
this
case
it
generates
a
TLS
secret
from
a
couple
files.
B
B
B
The
second
piece
of
the
authoring
piece
that
we
were
trying
to
address
was
about
these
kind
of
meta
level,
transformations
right,
certain
best
practices
around
applying
labels,
applying
annotations,
putting
things
in
namespaces.
We
tell
users
to
do
it,
but
we
didn't
make
it
easy
for
them
right.
So
how
can
we
make
it
easy
for
them
declaratively?
C
B
That's
a
good
excuse!
Now,
as
I
was
saying,
the
namespace
is
now
a
field.
You
can
specify
declaratively
right,
so
someone
looking
at
this
yam
will
config
can
look
at
okay,
here's
the
resources
that
are
being
applied,
here's
the
bases
that
are
being
pulled
in
and
here's
the
namespace
this
stuff
is
going
into.
B
The
second
pieces
are
round
labels
and
annotations
are
Oh,
name
prefix.
So
another
common
pattern
actually
is
for
users
to
have
some
naming
convention
around
their
apps
right.
You
may
want
all
your
dev
resources
to
have
the
dev
prefix
names
or
you
may
want
your
progress
sources,
tab.
The
prod
name,
prefix,
alright,
and
this
can
make
it
easier
just
to
look
at
a
resource
and
say
yes.
B
So
this
was
kind
of
a
fun
new
feature.
We
introduced
the
others,
our
annotations
and
labels,
the
annotations,
it's
exactly
what
you'd
expect
you
specify
top
level
annotations
and
then
it
will
apply
it
to
all
the
resources
and
the
labels
similarly
get
applied
to
all
the
resources,
but
they
also
get
applied
to
the
selectors
and
the
labels
of
the
pod
templates,
and
so
this
allows
you
to
define,
like
top-level
wiring,
across
all
the
resources
in
your
project.
B
A
Alright,
so
you
might
think
from
what
we
told
you
that
this
is
a
solved
problem
and
there's
nothing
left
to
work
on,
but
in
2020
we
have
a
few
areas
that
we
still
plan
to
improve
and
one
of
those
is
pruning.
What
pruning
means
is,
if
you
apply
with
a
resource
in
your
directory
once
and
then
apply
again
with
that
resource
missing
from
your
directory,
and
you
specified
pruning
that
the
resource
should
be
deleted
from
the
cluster.
A
So
we
have
something
like
this
already,
but
we
still
want
to
extend
it
to
support
deleting
created,
config,
Maps
and
secrets.
Another
thing
we
want
to
do
is
to
support
the
cube
control,
apply
blocking
on
the
status
of
the
resource
being
created.
So
when
you
apply
a
deployment
to
update
the
image,
the
request
is
going
to
complete
before
the
image
has
been
fully
rolled
out
to
all
of
the
replicas,
and
this
is
the
intended
behavior.
A
You
can
check
the
status
of
the
deployment
yourself
to
see
if
it's
been
fully
rolled
out,
but
some
users
might
want
to
have
a
blocking
cube
control
apply
request
so
that
they
can
be
sure
that
their
changes
have
fully
propagated,
and
we
don't
just
want
to
support
this
for
built-in
types
like
deployments.
We
really
need
this
to
work
for
CR
DS
as
well,
so
we'll
need
some
mechanism
for
CRD
authors
to
define
which
fields
in
their
in
their
status
are
blocking
and
what
values
should
be
waited
for.
B
Finally,
some
other
interesting
things
that
we're
gonna
be
working
on
are
around
how
we
develop
the
CLI
tools
themselves.
Today,
we're
kind
of
in
this
weird
state,
where
customizes
and
actually
a
separate
repo
the
code
for
customize,
was
built
in
a
criminai
SIG's
repo
cout
control
still
in
the
main
repo,
and
the
reason
we
did
customize
is
a
separate
repo
was
because
we
wanted
to
be
able
to
get
user
feedback
quickly
and
address
it
and
experiment
with.
Is
it
possible
to
actually
build
something
outside
of
kubernetes
kubernetes
for
a
long
time
it
was
like?
B
Can
we
actually
split
up
the
model
repo,
and
this
was
kind
of
a
chance
to
see
if
we
could-
and
it
did
have
some
nice
things?
As
a
result,
we
were
able
to
get
feedback
from
users
who
could
actually
take
build
of
customize
and
then
make
changes
and
then
push
a
new
build
within
two
weeks
and
get
more
feedback
right,
and
that
worked
out
nicer
than
a
three-month
release
cycle
right,
where
it's
a
lot
harder.
B
We
actually
want
to
move
to
control
outside
the
main
repo.
This
has
been
something
that
we
have
been
trying
to
do
for
about
two
years
and
it
has
been
a
massive
task
that
I
think
is
harder
than
anyone
expected
it
to
be
and
burned
out
a
lot
of
maintainer
x'
people
have
moved
off
a
bit,
but
we're
very
close
today
and
it's
going
to
move
into
something
called
the
staging
directory,
and
then
we
firmly
believe,
despite
other
people,
saying
it's
impossible,
that
we
will
move
it
into
its
own
cupen,
jewelry
PO.
B
B
It's
funny
because
it's
true
we've
talked
about
a
high
level
about
where
cou
control
apply
came
from
many
of
the
challenges:
users
we're
encountering
kind
of
how
we
worked
around
them
in
the
short
term
and
the
guidance
we
gave
to
users
and
then
how
we
try
to
look
at
those
challenges
and
figure
out.
What
can
we
do
to
address
them?
B
B
D
E
E
B
B
The
last
model
of
customization
is
hard
with
templating
right.
How
can
a
user
a
templating
is
great
like
to
have
an
abstraction
with
a
set
of
knobs
right,
but
then,
when
someone
wants
to
break
through
that
abstraction,
are
you
able
to
break
through
it
to
the
lower
level
right
and
how
easy
is
that
right
and
that
what
we
were
looking
at
is
how
can
we
enable
that
right,
and
so
that
was
actually
done
through
this
notion
of
patches,
which
I
didn't
get
a
chance
to
talk
about?
B
But
if
you
look
at
the
way
the
API
machinery
will
looks
with
controllers
and
you
look
at
the
original
patch
logic
for
coop
control
apply.
One
of
the
mechanics
was
we
generate
like
these
were
schema
where
and
we
able
to
generate
patches
and
overlay.
This
config
onto
that
config
without
having
a
notion
of
the
whole
thing
right
so
like
Jenny
was
saying
the
cluster
IP.
How
do
we
just
override
this
set
of
fields
without
blasting
the
custom
cluster
IP,
or
these
sorts
of
things
right,
and
so
we
looked
at,
we
said.
B
Actually,
we
can
provide
kind
of
like
arbitrary
last-mile
customers
a
using
that
exact
same
mechanic,
but
instead
of
applying
a
patch
to
the
resource
in
the
API
server
write
a
human
patching
over
a
robot.
We
can
actually
have
a
human
patch
over
something
written
by
another
human
on
the
client-side,
so
I
think
that's
one
of
the
lessons
learned
and
I
think
Vicki
classiest
has
done
some
talks
on
the
ways
customizing
helm
can
be
used
together
to
complement.
B
I'm
so
glad
you
asked
this
question:
yes,
we
actually
are
doing
some
things
and
that
are
really
exciting
around
just
that
we're
planning
on
making
every
every
sort
of
generator
pluggable.
So
it's
not
just
secret
generators,
it's
not
just
config
maps
but
to
actually
allow
users
to
generate
arbitrary
objects
and
do
arbitrary
transformations
right.
If
you
look
at
the
architecture
of
this,
it's
general,
it's
just
about
how
do
I
have
a
high
level
generation
of
a
couple
objects
and
then
how
can
I
do
some
transformation
of
looking
at
the
collect
total
collection?
B
That's
in
early
alpha
right
now
and
I.
Think
there's
gonna
be
a
lot
of
back-and-forth
on,
what's
the
best
way
of
developing
that
in
a
way
that
we're
able
to
allow
the
broader
ecosystem
of
tools
to
take
advantage
of
it
and
allow
them
to
maybe
plug
themselves
into
coop
control
more
directly
through
possibly
building
plugins.
But
yes,
there's
there's
already
actually
an
alpha
ability
to
both
do
just
secret
generation
and
then
there's
the
ability
to
do
arbitrary
generation
of
resources.
There's
alpha
as
well.
Thank
you.
C
Hey
so
quick
question
now
that
we
have
customized
integrated
and
we
have
the
keeps
TL
DIF
ordinary,
any
recommended
ways
to
so
we're
basically
using
the
Hamill's
from
upstream
you're,
not
losing
help,
and
is
there
any
dark
heart
to
use
the
cube
secured
if
to
auto-generate
the
differences
between
the
upstream
or
how
do
you
recommend
staying
close
to
upstream
iam
was
using
chip,
steel,
different
customized?
Is
there
any
recommendation
so.
B
B
One
way
of
doing
that
is
to
actually
use
the
patching
mechanism
I
described
earlier.
It
it
kind
of
depends
on
on
the
specifics
of
your
environment
and
setup.
If
you
are
trying
to
do
something
very
simple
like
we
want
to
use
the
upstream,
but
we
want
this
one
arbitrary
field
changed
or
this
set
of
arbitrary
fields
change.
B
You
can
actually
use
a
patch
to
just
not
actually
modify
the
source,
and
so
you
can
just
look
at
that
patch
and
you
can
see
these
are
the
things
I've
changed
from
upstream
without
having
to
do
it
every
base.
So
that's
that's
one
approach.
If
you
are
just
taking
a
another
yamo
file
and
from
some
get
repo
and
changing
it
then
maybe
get
diff
is
enough
for
you
that
case.