►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Krane: A Developer-Centric Deploy Tool - Daniel Turner & Katrina Verey, Shopify
Have you ever shipped changes to a Kubernetes app and found yourself wondering what actually happened? Krane is an open-source command-line tool created to solve this problem: it helps developers, especially those who may be new to Kubernetes, deploy with confidence. Krane translates Kubernetes’ asynchronous convergence process into a clear pass/fail result for each deploy. It detects unsuccessful rollouts and shows developers the information they need to take corrective action. Krane also helps ensure dependencies are rolled out in a sane order, it natively supports custom resources, it allows developers to run scripts as part of their deploys, and more! Come find out what Krane can do, learn how its design makes it resilient and scalable, and discover how it may help your organization provide a better developer experience for Kubernetes apps.
https://sched.co/UaaC
B
A
A
A
It's
real
functionality
from
there
we'll
take
a
look
inside
of
the
guts
of
it
and
actually
look
at
the
code,
and
our
goal
here
is
to
help
you
get
a
better
mental
model
of
what's
going
on
and
since
we're
open-source,
if
you
ever
want
to
contribute,
you'll,
have
a
leg
up
and
getting
started,
and
then
finally,
we'll
talk
about
some
lessons
learned.
This
project
is
over
three
years
old.
A
Now
we've
had
internal
contributors
at
Shopify,
we've
had
external
contributors
in
the
community
and
we
integrate
with
kubernetes
and
we've
had
some
things
really
well
and
we've
made
some
mistakes
along
the
way
and
we're
hoping
to
share
those
so
that
all
of
you
can
learn
from
our
experiences
so
to
kick
it
off.
Here's
Katrina.
B
So
what
exactly
is
crane
fundamentally
crane
is
an
open-source
deploy
tool
that
we
built
specifically
for
kubernetes,
it's
a
tool
that
we
use
a
lot
at
Shopify.
Our
metrics
show
that,
on
a
normal
business
day,
we
invoke
crane
over
2000
times.
Things
haven't
always
been
this
way.
Of
course,
cranes
story
starts
back
in
early
2016
when
Shopify
started
investigating
the
possibility
of
running
our
apps
on
kubernetes.
B
After
extensive
experimentation,
we
decided
to
move
forward
with
an
internal
platform
as
a
service
that
would
run
all
of
our
applications
from
the
smallest
experiments
right
through
to
our
business,
critical
rails
monolith.
We
wanted
our
new
platform
to
be
self-service,
so
one
of
the
things
that
we
had
to
work
out
was
how
to
enable
our
developers
to
deploy
to
it
themselves.
B
Maybe
the
most
obvious
thing
to
do
is
to
give
developers
a
tool
that
simply
runs
good
control,
apply
for
them
right
in
our
first
iteration.
That's
exactly
what
we
did
and,
of
course
it
worked
beautifully
for
initiating
changes.
However,
keep
control
appliers
success
is
more
the
equivalent
of
okay
I'll.
Try,
then
you
equivalent
to
your
changes,
worked.
So
all
lixo
sects
success
status
really
means
is
that
the
API
server
accepted
your
desired
state,
not
that
your
desired
state
has
been
or
will
ever
be
reached.
B
We
knew
that
once
we
started
onboarding
non-experts,
this
was
not
going
to
cut
it.
What
we
wanted
was
a
deploy
tool
that
would
empower
our
developers
to
play
confidently
to
kubernetes,
eventually,
convergence
system.
This
tool
could
not
assume
any
kubernetes
expertise
and
it
had
to
give
our
developers
the
information
they
would
need
to
recognize
and
solve
problems
independently.
B
At
the
time
we
didn't
find
any
tools
that
were
trying
to
do
this,
notably
queue
control,
rollout
itself
didn't
even
exist
and,
although
helm
is
already
getting
popular,
the
feedback
that
it
was
giving
at
the
time
wasn't
much
more
than
you
got
from
cute
control
apply.
So
we
decided
to
build
crane
and
make
this
our
mission.
B
We
wrote
crane
and
ruby,
and
at
first
it
was
actually
just
a
little
script
inside
our
higher-level
deploy
tool,
which
is
called
shipit'.
Incidentally,
ship.
It
is
also
open
source.
So
you
can
check
that
out.
If
you're
interested
as
our
script
grew,
we
realized
that
it
had
independent
value
and
we
extracted
it
to
a
ruby
gem
with
its
own
CLI.
Originally
that
gem
was
named
after
literally
the
file
that
has
had
been
called
in
our
in
other
deploy
tool
so
had
the
gun
event,
uninventive
name
of
criminales
deploy
this
month.
B
So
this
here
is
what
a
the
most
basic
invocation
of
crane
looks
like
you
have
crane
deploy
the
name
of
your
namespace,
the
name
of
the
context
that
namespace
is
in
and
then
the
list
of
files
or
directories
that
contain
the
set
of
resources
that
you
want
to
have
in
that
namespace
already.
This
tells
you
something
important
about
the
model
that
we
chose
for
crane.
Fundamentally,
we're
managing
a
namespace
for
you.
This
means
we're
going
to
add,
update
and
remove
resources
to
make
the
live.
B
As
I
mentioned.
Our
focus
is
on
developer
experience
and
we
laid
the
foundations
for
that
by
translating
what's
happening
in
kubernetes
into
an
accurate
pass/fail
result.
This
means
that
we
do
not
declare
success
until
the
desired
state
has
actually
been
achieved
and,
conversely,
we
try
to
detect
when
convergence
is
doomed
as
quickly
as
possible
to
give
developers
the
chance
to
act
on
it
when
we
give
developers
that
result,
especially
if
it's
a
failure
results,
we
want
to
make
sure
that
they
can
take
action
and
fix
things.
B
So,
for
example,
if
the
problem
is
with
a
pod,
we're
going
to
show
them
a
sample
of
its
logs
and
events,
we
also
do
a
lot
behind
the
scenes
to
make
deploys
go
more
smoothly.
For
example,
we
do
some
deploy
sequencing.
One
example
of
that
is
that
we
validate
everything
before
we
even
attempt
to
deploy,
which
is
a
good
idea
right,
and
we
also
sequence
some
resources
before
others.
For
example,
it
can
be
very
important
to
deploy
things
like
a
quota
before
the
things
that
the
quota
is
supposed
to
constrain.
B
A
So
I
know
I
promised
you
live
demos,
but
I'll
be
honest,
I'm,
not
that
brave
these
demos
were
live
when
I
recorded
them,
but
well
that's
stretching
the
word
live,
isn't
it
so?
The
first
demo
is
really
straightforward:
we're
going
to
play
a
basic
web
app
something
we
do
a
lot
of
Shopify
demo.
Sorry,
the
deploy
has
four
phases.
We
begin
by
validation
after
that
we're
actually
to
check
the
initial
state
of
the
cluster.
A
Make
sure
we
know
where
we
came
from
I
will
actually
do
the
deploy
the
thing
we
said
we're
going
to
do
and
then
we'll
use
some
smart
and
rollout
logic
to
monitor.
What's
going
on
and
to
be
able
to
assert
confidently
that
this
was
success
or
failure,
the
devil
is
gonna.
Go
quick!
So
don't
worry
about
that.
If
you
don't
catch
it
all
the
first
glimps
I
promise
I'll
break
it
apart
after
it
runs.
So
let's
do
it.
A
So
you
begin
with
the
command
we
saw
earlier
crane
deploy
test,
app
test
cluster
and
the
resources
from
there
we
validate
it,
the
cluster
state.
We
actually
do
the
deploy
and
now
we're
waiting
for
it
to
verify.
Big
success,
it's
exactly.
We
wanted
to
see.
Ok
so
now,
as
promised
we're
breaking
it
apart.
So
in
the
beginning
we
promised
validation.
So
we
start
by
validating
the
command
line.
Flags.
You
arguments
you
passed
right.
Does
this
cluster
exist?
Does
the
namespace
exist
in
the
cluster?
How
about
those
resources?
Are
they
valid
llamo?
A
Are
they
valid
kubernetes
yeah?
Well,
let's
make
sure,
by
the
way
when
I
said
simple
web
app,
this
is
about
as
simple
as
I
could
come
up
with
a
deployment
a
service
in
an
ingress,
so
we
check
for
their
existence.
Now.
This
is
the
first
time
we
deployed
it
till
you're
not
found.
This
is
not
particularly
helpful,
you're
all
sitting
here,
like
yeah.
A
Of
course,
it's
not
there
Danny
for
the
first
deployment,
but
where
this
becomes
really
powerful
as
later
on,
when
things
go
wrong,
you
can
say
it's
crashing
was
a
crashing
before
or
why
isn't
that
resource
there
what's
going
on?
So
this
is
a
nice
historical
step.
After
that
we
actually
do
the
deploy.
So
by
default,
we'll
deploy
all
resources
with
cube,
CTL
apply,
but
we
can
use
other
verbs
per
resource
kind.
A
So
if
there's
a
bug
or
some
other
issue
we
can
create,
we
can
replace,
we
can
force
replace
if
we
need
to
and
then
finally,
the
things
we're
most
proud
of
is
the
smart
rollout
logic.
So
we
first
just
you
know,
check
to
make
sure
that
cube,
see
tail
apply,
succeeded.
But
after
that
we
need
to
make
sure
that
other
resources
were
successful.
So
for
a
deployment,
let's
make
sure
the
number
of
replicas
you
asked
for
the
number
that
are
updated
and
available
for
a
service
if
you're
using
pod
selectors.
A
Let's
make
sure
those
pod
selectors
actually
select
something
right.
It's
no
good
to
say
the
ingress
is
fine.
The
deployments
fine,
but
oh
yeah,
your
service,
that's
connected
to
nothing
out
there
right
like
without
the
whole
pipeline
to
work,
but
this
is
a
super
simple
demo
right
and
things
get
more
complicated.
Your
apps
get
bigger,
say:
I
want
some
dynamic
configuration
with
a
config
map,
so
in
this
demo
we'll
see
the
same
things
you
saw
before,
but
there'll
be
a
new
step.
This
is
the
priority
resource
deployment
step
where
we
actually
get
more
deploy
sequencing.
A
So
again,
this
will
run
pretty
quickly.
Oh
I
should
actually
show
you
the
config
map
basic
config
map,
nothing
too
crazy.
Here,
I
said
well,
kick
this
off
it'll
run
pretty
quick,
so
do
the
validation
and
the
validation
failed.
How
did
I
fail?
A
pre-recorded
demo
am
I
crazy.
Well,
okay,
so
it
failed.
There's
a
new
resource
result
here:
right
failure,
that's
new
I!
Guess:
I'll!
Take
this
opportunity
to
tell
you
what
we
do
on
failures.
So
first
we
give
you
some
helpful
error.
Messages
from
gypsy
tale,
apply
like
dadoes,
isn't
filled
and
config
map.
A
Oh
there's
a
typo
in
there
I'll
fix
that
in
a
second.
The
other
thing
we
do
is
actually
give
you
the
template
now
in
this
demo,
giving
the
templates
silly
right.
There's
one
config
map
in
one
directory.
I
know
where
to
find
it.
But
if
you
use
this
as
part
of
a
larger
pipeline,
where
you
render
your
resources
dynamically
or
dynamically,
determine
oh
you're,
gonna
render
or
and
generate
well
actually
seeing.
The
final
thing
here
is
a
lot
better
than
having
to
sort
of
piece
it
together.
A
Okay,
we
got
through
validation.
This
time
got
to
the
priority
resource,
so
it
went
out
and
now
we're
just
validating
that
everything
else
is
happy.
Cool
success
Oh
be
really
embarrassing
to
do
that
twice.
Wouldn't
it
okay,
so
breaking
this
apart
here,
we're
gonna
validate.
What's
in
the
cluster
and
the
config
Maps
not
found
right,
we
failed
validation.
We
didn't
try
and
get
it
there.
So
historically,
it's
not
there.
The
other
piece
is
the
priority.
A
A
A
The
correct
config
map
for
other
things,
I'm
gonna
call
state
modifiers,
are
back
network
policies,
resource
quotas,
right,
there's,
a
sane
result.
You
expect
if
your
resource
quota
says
ten
pods
and
your
planets
as
a
thousand
sensibly.
You
want
ten
pods.
If
you
do
these
in
the
wrong
order,
you're
not
gonna,
get
what
you
want
and
then
finally,
a
class
of
things
will
call
tasks
and
we
represent
these
as
bear
pods,
and
if
you
use
a
rails
app
or
any
sort
of
web
app.
A
You
often
know
that
before
you
start
serving
web
sites,
you
want
to
get
your
static
assets
uploaded
to
a
CDN,
your
JavaScript,
your
CSS
and
images.
So
we
use
a
pod
to
do
acid
uploading.
This
gets
phased
before
we
actually
start
serving
anything
and
it's
perfect
for
us
I.
You
can
also
stick
other
tasks
in
here.
Whatever
your
heart
desires,
okay,
but
this
config
back
was
super
contrived
right
like
it
doesn't
really
buy
us
anything
anything.
So,
let's
kill
it
and
we
do
that.
A
A
So,
what's
new
here
is
the
pruning,
like
we
said
as
soon
as
we
deleted
the
config
map
and
deployed
it
got
pruned
from
the
namespace
and
as
promised,
we
manage
your
namespace.
We
keep
everything
in
sync,
but
I
keep
talking
about
deploy
commands.
But
if
you
member
the
title
slide
of
the
talk,
it
was
Krane
not
Krane,
deploy
and
the
reason
for
that
is
we're
not
one
command,
we're
actually,
five,
so
we've
seen
deploy
a
lot.
We
have
a
similar
command
called
global,
deploy
to
help
you
manage
the
global
namespace
or
not
names
based
resources.
A
We
have
restart
it's
super
great
when
deployments
do
rolling
updates
right
this
smooth,
continuous
availability,
but
sometimes
you
don't
have
to
deploy.
You
just
want
to
restart
things.
So
we've
added
that
functionality
we
have
Krane
run
in
the
same
way
that
there
are
a
lot
of
tasks
associated
with
deploys.
You
want
to
do.
Sometimes
you
have
tasks
that
are
outside
of
deploys.
If
you
know
rails,
these
would
be
things
like
rake
tasks,
I'm
pretty
sure.
Php
has
something
equivalent
and
probably
every
framework
and
then
finally
a
render
of
tasks.
A
So
some
historic
information
we
used
to
dynamically
render
ARB
as
part
of
the
deploy,
and
this
meant
that
you
didn't
know
you
were
deploying
until
you
actually
deployed
it
sometimes,
and
that
also
meant
that
you
couldn't
check
your
templates
in
CI
because
they
weren't
the
ammo.
They
were.
You
know,
Ruby's
ERP
language,
so
by
breaking
the
renderer
out
into
its
own
command
that
let
us
have
a
continuous
integration
that
could
also
verify
sensible
things
about
your
templates.
So,
let's
see
another
command,
maybe
restart
this
time.
It
looks
a
little
different.
A
We
don't
actually
up
to
say
what
we
want
to
restart.
Will
dynamically
find
it
will
do
the
restart
and
then
we'll
continue
to
use
our
smart
logic
to
make
sure
it
successfully
goes
out.
So
this
one
also
goes
by
pretty
quick
I
promise
we'll
break
it
apart,
so
bang
type
the
command
how'd
it
find.
What
you
want
have
it
do
the
restart
smart
logic
goes
and
then
we
get
text.
Read
text
gives
me
a
bad
feeling.
Okay,
let's
see
where
this
goes
yep,
it
didn't
succeed.
A
So
the
first
thing
is:
we
have
a
new
result
status
timed
out,
so
we
saw
a
success.
We
saw
failure
time
down
to
something
in
the
middle
right.
Kübra
is
eventually
convergent,
so
maybe
a
timed
out
status
will
converge
to
success
or
maybe
they'll
never
get
to
failure.
Maybe
I'll
sit
here
forever,
but
if
you're
familiar
with
a
progress
deadline,
seconds
four
deployments
use
this
idea
that,
like
at
some
point,
you
got
to
make
a
decision
and
just
decide
it's
not
right.
So
I
cranked
this
way
down.
A
So
we
want
to
be
sitting
here
forever,
like
three
seconds
in
reality.
Right,
like
I,
don't
think
anyone's
ever
set
their
PDS
to
three
but
again,
I.
Don't
want
to
keep
us
here
for
three
minutes.
So
if
you
stare
at
a
screen-
and
so
we
failed
so
let's
actually
see
what
other
information
we
might
feel
to
determine
well
yeah,
busybox,
bad
tag,
I
wonder
why
I
can't
pull
that
image.
A
So
that's
one
of
things.
We
do
up
all
the
relevant
events.
We
also
grab
logs
from
your
container.
So
if
you're
not
familiar
with
rails,
this
is
basically
the
start
of
a
stack
trace
and
basically
it
means
this
pods
crashing
on
startup.
Now
this
demo
is
super
contrived
right,
I
think
most
of
you
figured
out
like
how
did
he
get
to
the
state?
A
Right
was
working,
the
last
employee
and
so
in
between
here
I
screwed,
some
things
up
in
the
system,
and
if
you
were
a
developer,
you
would
had
a
failed
deploy
but
I
suspect
as
a
developer.
You've
never
ever
deployed
something
had
it
fails,
it
may
be
a
restart,
we'll
fix
it.
I'll.
Try
that
right!
No
one
ever
does
that
the
other
thing
to
highlight
here
and
why
this
is
a
little
bit
contrived,
is
that
we
have
fast
failure,
detection.
We
expect
developers
to
build
images
before
they
deploy
them.
A
So
if
we
ever
actually
discovered
this
event,
you
know
can't
find
an
image
tag.
We
just
immediately
fail
their
deploy.
Now
we
don't
immediately
fail
it.
We
take
more
than
three
seconds
to
do
it,
but
if
I
had
set
a
reasonable
PDS
we'd
see
failure
here,
but
then
I
couldn't
highlight
timed
out
conditions
without
having
you
waiting
forever.
So,
oh
yeah,
so
on
a
happier
note
though
one
other
can
talk
about
is
the
dynamic
discovery,
and
hopefully
it's
not
off
the
top
of
the
slide.
A
And
you
might
have
a
my
sequel
pod
and
your
right
name
space
and
you
think
yeah
I
don't
want
to
restart
those,
so
we
actually
use
the
annotations
straw.
Let
you
describe
what
you
want
to
be
start
when
you
were
starting
your
app
okay,
think
it's
more
and
more
contrived
I
keep
having
to
like
walk
back
and
back
dice
at
the
beginning.
The
other
thing
that's
contrived
about
this
is
I.
Did
it
from
my
laptop,
but
this
is
part
of
our
CD
tooling
right.
A
It
turns
out
that
you
can
run
crane
from
a
laptop,
but
we
don't.
We
give
you
access
to
the
Ruby
library,
so
you
can
do
that.
But
if
our
mm
deploys
a
day,
I
think
they're
all
done
through
our
CD
pipeline.
So
I
want
to
show
you
what
that
looks
like
for
a
realistic
holistic
view
of
how
crane
is
used.
So
this
is
ship
it.
This
is
specifically
the
restart
page
for
one
of
our
real
apps.
Well,
it's
a
test
app,
it's
a
real
app,
the
page
tells
them
the
user
or
the
developer.
A
A
It
looks
like
interactive
terminal,
but
it's
not
we're
just
doing
the
same
logging
we've
seen
before
on
a
black
background
this
time,
but
it
allows
the
developer
to
see
what's
happened
and
to
send
links
to
other
people
saying
like
hey
I'm
in
the
success
case,
they
don't
just
send
a
link,
but
in
the
failure
case
they
can
send
a
link
to
someone
else
to
evaluate.
What's
going
on
so
to
peek
behind
the
scenes.
Here's
Katrina.
B
What
I'm,
gonna
do
first,
is
go
over
some
of
the
key
classes
that
are
involved
in
and
deploy.
The
main
point
is
called
deploy,
tasks
and
its
main
method
is
called
run
and
if
you
were
to
use
crane
directly
as
a
gem,
this
is
what
you
would
be
calling
deploy.
Task
is
responsible
for
that
sequence
of
phases
that
you
saw
in
the
demos,
and
it
does
this
by
coordinating
other
more
specialized
objects,
such
as
resource
deployer
resource
deployer,
basically
adds
some
smarts
around
cube
control
commands.
B
Another
important
class
is
called
resource
watcher
and
this
one
is
responsible
for
determining
when
the
deploy
is
actually
done
and,
finally,
at
the
heart
of
crane,
we
have
kubernetes
resource
classes.
Career
90s
resource
is
actually
a
super
class
and
we
implement
subclasses
of
it
for
all
the
different
resource
kinds.
B
B
What
we're
doing
here
is
pulling
the
most
important
part
of
the
work
happens
in
the
middle
method,
their
sink
resources
and
that's
where
we're
talking
to
the
API
server
and
I'm
gonna,
get
back
to
that
in
a
minute
after
we
talk
to
the
API
server,
we
tell
the
user
about
what
we
just
saw
so
did
any
resources
just
complete,
did
they
just
succeed,
fail
or
timeout?
And
what
are
we
still
waiting
for?
We
let
them
know
about
that
as
well.
B
Finally,
we
keep
looping
and
then
once
there's
nothing
left
to
watch
or
when
we
hit
the
global
timeout,
which
is
something
that
you
can
configure.
We
exit
as
I
mentioned.
Sync
resources
is
the
most
important
thing
here.
Andrey's
are
syncing
is
a
really
key
concept
for
us
to
understand
inside
the
sync
resource
method,
we're
going
to
give
each
resource
instance.
B
So
that's
an
instance
of
that
kubernetes
resource
class
or
subclasses,
but
that
set
represents
all
the
resources
that
you
want
in
your
namespace
or
the
subsets
that
we're
still
watching
we're
going
to
give
each
one
a
chance
to
gather
information
from
the
kubernetes
api
server.
Otherwise
it
doesn't
have
any
access
to
that
API
server,
and
there
are
two
reasons
that
we
so
strictly
control.
When
these
API
calls
are
made.
B
The
first
one
is
perhaps
the
most
obvious
we're
doing
it
for
performance
reasons
if
you're
fetching
hundreds
of
resources
or
even
thousands
from
a
large
cluster,
this
can
be
very
expensive.
So
you
really
don't
want
to
make
any
more
calls
than
you
have
to.
The
second
reason
is-
maybe
less
obvious,
but
it's
also
very
important
we're
doing
this,
because
it
makes
our
determinations
consistent
with
kerbin
at
ease.
As
you
know,
it
it's
impossible
to
make
a
determination.
B
That's
going
to
be
valid
indefinitely,
because
kubernetes
behind
the
scenes
is
trying
to
converge
basically
forever,
and
there
are
many
processes
in
play
that
can
cause
the
state
to
change
over
time.
So
what
we're
actually
trying
to
do
here
is
create
an
accurate
snapshot
representing
a
specific
point
in
time.
So
that's
what
we're
really
doing
during
the
sync
we're
taking
that
snapshot.
B
B
So
here's
a
real
example
of
one
of
the
kubernetes
resource
classes.
This
is
actually
the
entirety
of
the
config
map.
Class
configuration
objects
are
the
simplest
ones
to
implement,
because
success
for
them
just
means
that
they
were
accepted
by
the
API
server
and
they're
available
for
consumption.
So
you
can
see
here
that
the
deploy
succeeded
for
a
config
map
is
just
confirming
that.
Last
time
we
looked
yes,
this
actually
existed,
and
you
can
also
notice
that
deploy
failed.
We
don't
have
any
node
and
deploy
failure
states
for
this
kind.
B
This
next
example
isn't
a
real
class,
but
it's
representative
of
how
the
more
complex
logic
looks
in
in
the
more
complex
resource
classes.
So
essentially,
the
sync
method
is
taking
a
snapshot
of
the
live
state
of
that
resource
and
then,
when
we
later
call
deploy,
succeeded
and
deploy,
failed,
we're
using
that
cached
data
to
inspect
the
state
and
determine
whether
it
matches
our
predefined
expectations
regarding
success
and
failure.
B
So,
if
you're
interested
in
contributing
to
crane
these
resort
classes
are
a
really
great
way
to
get
started,
we've
implemented
many
kinds
over
the
years,
but
there
are
so
many
left
to
do.
In
particular,
we
only
recently
added
support
for
non
namespace
resources
via
our
new
global
deploy
task,
so
we
could
definitely
use
help
losing
modeling
more
of
those.
B
Another
way
that
you
can
help
is
contributing
your
expertise
on
failure
modes
that
you've
seen
in
the
wild
to
help
improve
our
fast
failure
logic,
in
other
words,
that's
augmenting
those
deploy
failed
methods
if
you're
less
familiar
with
Ruby
or
kubernetes,
we
can
always
use
help
with
documentation
and,
of
course,
we
always
appreciate
receiving
bug
reports.
If
you
have
another
idea,
please
open
an
issue
and
let
us
know.
B
The
first
lesson
learned
is
about
being
opinionated
about
your
mission,
as
Danny
alerted
alluded
to
this
earlier.
But,
as
you
know,
our
mission
is
poised,
not
template
management,
but
when
we
started
our
developers
needed
a
way
to
inject
the
latest
image
tag
into
their
deployments,
and
we
decided
to
solve
that
problem
by
integrating
a
renderer
right
into
our
deploy
command.
We
defaulted
to
using
a
template
language
language
that
would
be
very
familiar
to
our
developers.
Prb
ear
B
is
essentially
Ruby,
and
that
means
there's
hardly
anything.
It
can't
do.
B
Naturally,
over
time
our
developers
took
full
advantage
of
this,
adding
loops,
conditionals
and
dare
I,
say
API
calls
right
into
their
templates.
What
we've
experienced
is
that,
as
templates
become
increasingly
complex,
leaving
rendering
to
deploy
time
becomes
increasingly
dangerous.
My
recommendation
is
that
if
you
use
ARB
or
any
other
complex
templating
to
do
something
non-trivial,
you
commit
the
result
ahead
of
time
or,
at
the
very
least,
render
invalidate
your
templates
in
CI.
B
The
second
mistake
I
want
to
talk
about
is
related
to
the
importance
of
fulfilling
the
expectations
that
you
set.
As
you
now
know,
crane
claims
to
manage
a
namespace
for
you,
and
it's
always
been
that
way.
However,
our
pruning
was
originally
based
on
a
whitelist,
which
meant
that
we
weren't
actually
pruning
a
bunch
of
the
less
common
namespace
resource
types.
B
This
meant
that
whenever
we
adopted
more
kinds
internally,
we
would
go
through
these
really
painful
audits
to
make
sure
it
was
safe
to
add
a
kind
to
our
like
whitelist
that
hadn't
been
there
for
years.
We
would
also
have
to
then
post
messages
in
our
change
log
warning
company
other
companies
that
they
had
to
do
the
same
and
that
frankly,
really
sucks.
When
we
decided
to
go
with
the
whitelist
approach
at
the
beginning,
we
thought
we
were
being
safe.
B
B
A
There
are
mistakes
and
they're
just
hard
problems,
things
that
I'm
sure
you've
all
experienced.
So
the
first
one
to
talk
about
is
timeouts.
So
in
our
restart
demo
we
saw
a
timeout
and
that
one
was
obviously
the
users
fault,
but
there's
a
lot
of
situations
where
it
won't
be.
If
your
container
builder
system
builds
large
images
and
then
you
have
a
node
that
has
to
download
multiple
large
images
at
the
same
time,
you're
gonna
have
right
contention
and
that's
gonna
be
slow,
maybe
pause.
A
Startup
is
gated
by
connecting
to
a
slow
external
resource,
may
be
a
cluster
issues
who
knows,
but
inevitably
developers
will
hit
Riis
timeouts
so
the
first
time
this
happens,
developers
like
oh,
this
is
different.
Let's
investigate,
let's
power
up
the
logging
system,
the
portal
system,
let's
get
to
the
bottom
of
what's
going
on
and
by
the
time
they
figure
it
out
could
be
needs.
Eventual
convergence
has
converged,
they
have
a
successful
deployed
like
oh
I
was
to
knowing
they
had
retried
the
deployment
it's
idempotent.
A
It
succeeds
success
somewhere
around
four
or
five
times
investigating
they
get
upset,
they
stop
trying
to
investigate
and
they
just
mash
retry
I've
seen
people
hit
that
button
like
six
times
it's
shocking,
but
this
was
our
fault.
It
wasn't
theirs.
Right,
like
we
didn't
gave
them
no
way
to
know
like
oh
this
timed
out
and
it
was
slow
versus
there's
a
failure.
So
we
added
some
nice
icons
to
our
CD
system.
We
always
had
the
you
know:
green
everything's,
good,
but
never
Li.
A
The
red
X
meant
timed
out
or
failed,
and
we
said
okay,
let's
add
one
more
and
we
did
that
with
this
nice
black
clock.
So
now,
red
X
means
investigate
now
something's
wrong.
The
clock
means
okay.
We
wouldn't
like
you
to
investigate,
but
if
you
mash
retry
we'll
look
the
other
way
we
did.
Those
are
the
error.
Codes
by
default.
Success
is
zero,
one
is
failure,
and
then
we
defined
an
error
code
for
timeout
and
there's
no
consensus
on
this.
A
Like
we
looked
at
a
bunch
of
tools
and
we
picked
270
but
I'm
sure
everybody
has
their
own
opinion
about
what
error
code
timeouts
should
have
been.
The
other
tricky
problem
is
actually
with
cube,
CTL
apply.
So
if
you
haven't
seen
it
before
there's
this
last
applied
configuration
notation
and
it
only
exists
on
the
server
right.
It's
not
in
your
resources
and
so
for
developers
out
of
sight
out
of
mind.
So
it
looks
like
the
there.
A
It's
a
JSON
blob
is
the
value
of
this,
and
it's
essentially
the
last
thing
you
tried
to
apply,
but
this
doesn't
like.
Okay.
This
is
fine.
It's
nice
to
know
what
that
is,
but
it
gets
used
in
some
interesting
ways.
The
first
is
pruning.
If
that
annotation
is
available
on
a
resource,
it's
considered
eligible
for
pruning.
So
this
is
exactly
what
you
want,
though,
right
you're
like
you've,
deleted
the
thing
from
the
file
it
used
to
be
on
the
cluster
you
deploy
to
cut,
prove
success,
but
during
incidents.
Well,
that's
when
everything
goes
out.
A
The
window
right
humans
start
applying
things
and
forget
to
check
them
into
repositories.
They
start
editing.
Things
go
wrong
and
you
don't
notice
that
you
next
deploy
where
you
accidentally
prune
something
you
really
wish
you
hadn't
and
then
you're
back
in
that
first
incident
from
the
day
ago.
The
other
fun
one
is
the
three-way
merge,
so
I
think
my
mental
model
when
I
start
with
kubernetes
was
there's,
live
State
on
the
cluster
and
there's
this
thing
here
and
keep
CTL
applied
patches
the
live,
State
and
we're
all
happy,
but
that's
actually
not
the
real
model.
A
There's
the
three-way
merged
I
was
mentioning
so
essentially
kubernetes
considers
ownership
of
various
fields
based
on
those
values.
So
where
does
this
happen?
Well,
we
started
all
of
our
applications
with
a
static
count
of
replicas
right
when
you're
first
getting
going.
Adding
auto
scaling
is
just
one
more
complexity.
You
don't
need,
but
eventually
you
get
big
and
you
need
to
control
costs.
You
need
to
make
sure
you
scale
up
in
the
middle
the
night
or
page
an
operator,
and
you
start
adding
HPA's,
and
these
things
are
fantastic.
A
I
think
we
all
love
them,
but
you
need
to
remove
the
static
replica
count
from
your
deployment
NFS.
So
you
deploy
you,
don't
reset
to
some
random
number
and
then
every
PHP
to
kick
back
in,
and
so
if
your
mental
model
is
well
I
just
delete
that
line
and
do
a
patch
and
because
it's
a
patch
everything's
fine,
you
totally
miss
the
fact
that
the
three-way
merge
says:
oh
no,
no,
no!
You
used
to
own
that
field,
and
now
you
don't
so
you
want
the
default
and
the
default
one
and
yeah.
A
B
B
Another
thing
that
has
been
working
well
for
us
is
the
realization
that
there
are
a
lot
of
transient
errors
in
systems
like
these
and,
most
of
the
time,
they're
not
worth
bringing
to
the
attention
of
developers.
There
are
a
lot
of
places
in
our
code
where
we're
able
to
ignore
errors.
We
get
back
from
the
API
server
and
just
try
again
later,
notably
in
the
next
polling
cycle.
B
Another
pattern
that
we
really
like
is
using
annotations
directly
on
resources.
As
a
mechanism
for
letting
developers
communicate
resource
specific
settings
to
us.
We
actually
saw
one
of
these
in
the
demos
when
Danny
showed
you
the
annotation,
that
you
can
add
to
your
deployments,
to
tell
our
crane
restart
command
to
target
those
ones
by
default.
B
Another
tip
on
the
theme
of
customization
is
about
logging,
since
our
default
output
is
very
developer
focused,
we
want
to
keep
it
simple
and
clear.
This
means
that
when
we
want
to
use
output
to
debug
crane
itself,
our
default
output
pretty
useless.
So
to
address
this,
we
use
an
environment
variable
to
enable
debug,
debug
level
logging,
and
that
gives
us
all
kinds
of
extra
information,
including
about
all
the
API
calls
we're
making.
B
My
final
tip
is
that,
if
you're
running
a
open
source,
kubernetes
tool
definitely
consider
making
an
official
supported
versions
list
and
setting
up
a
CI
cluster
for
each
one
of
those
versions.
Our
test
suite
does
dozens
and
dozens
of
deploys
to
each
of
our
supported
versions
and
I
can't
tell
you
how
many
bugs
that
has
caught
and
how
much
reassurance
that
has
provided
us.
A
So
to
wrap
up
crane
is
our
developer.
Centric
deploy
tooling
it's
highly
scalable.
We
have
namespaces
with
thousands
of
pods
hundreds
of
services
and
hundreds
of
deployments.
If
you're
saying
we
need
something
bigger
than
that.
Let
us
know
we
want
a
new
challenge.
It's
open
source,
if
you
found
this
exciting
we'd
love
to
have
you
contribute
that
to
have
you
use
it.
It
comes
really
exciting.
A
We're
hiring
you
can
contribute
from
the
inside
and
with
that,
we're
easy
to
find
we're
on
github
at
Shopify,
slash
crane,
where
the
kübra
days
slack
channel
as
Crane,
and
we
have
a
ton
of
developers-
and
this
is
all
I-
could
fit
on
a
slide,
so
I
want
to
thank
them
and
all
the
rest
of
you
as
well
as
been
here
for
coming
up
with
crane
this
our
new
name
and
with
that
I'd
love
to
take
questions.
Thank
you.
Thank
you
very
much.