►
Description
Continuous Delivery with Kubernetes at Box [I] - Greg Lyons, Box
Deploying and managing applications with Kubernetes can be challenging. Organizing configuration across multiple environments, rolling out changes incrementally, safely killing or rolling back failed deployments - these are just a few difficulties that organizations face when running containers in production.
At Box, we've dealt with these issues and more, at the scale of thousands of servers across multiple data centers and public cloud providers. In this talk, we'll share how we set up a continuous delivery pipeline with Jenkins, Docker, Artifactory, and Kubernetes to test, build, and release our software rapidly and reliably. We'll discuss how our pipeline reduces time to ship to production, provides greater visibility into the deployment process, and empowers our engineers to deploy quality code with confidence.
About Greg Lyons
Greg is a software engineer at Box, where he works on tooling for running microservices with Kubernetes. He built and open-sourced kube-applier, a containerized service for deploying Kubernetes apps with declarative configuration.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
So
for
those
of
you
who
aren't
familiar
with
box,
we
are
a
cloud
content
management
platform
which
means
that
we
make
it
easy
for
companies
and
organizations
to
store
all
of
their
content
securely
in
the
cloud
access
it
from
anywhere
share
and
collaborate
so
that
our
customers
can,
you
know,
work
together
and
do
their
work
as
effectively
as
possible.
So
to
take
a
quick
look
at
what
kubernetes
looks
like
it
box
we're
currently
running
four
clusters,
three
of
which
are
in
production.
A
This
point
we
have
about
80
services
running
in
production,
which
really
represents
a
pretty
significant
chunk
of
our
core
mission:
critical
code,
the
powers
box.
So
at
this
point
we
are
pretty
heavily
reliant
on
kubernetes
and
our
services
running
on
kubernetes
to
power
our
product
for
the
over
80,000
organizations
around
the
world
that
depend
on
us.
So,
given
that
context,
when
we
need
to
make
changes
to
our
services
that
are
running
in
kubernetes,
we
want
to
do
so
in
a
way.
That's
safe,
we're
very
much
in
the
business
of
not
breaking
things
for
our
customers.
A
I'd.
Imagine
many
were
in
that
same
business
as
well.
So
that's
what
I'm
here
to
talk
about
today,
the
problem
of
how
can
we
make
changes
to
our
services
running
on
kubernetes
in
a
way
that's
safe
and
easy.
So
when
I
say
changes
to
a
service
running
on
kubernetes
I
could
be
talking
about
a
bunch
of
different
things.
You
might
be
talking
about
a
change
to
a
source
code.
If
you
have
an
app
written
and
say
go
or
Java
you're
adding
a
new
feature.
A
This
might
be
a
change,
some
sort
of
third-party
package
that
you're
pulling
in
you
might
upgrade
a
version
of
some
rpm
that
you're
pulling
down
from
the
internet.
This
might
be
some
config
parameters
like
an
environment
variable
or
a
command
line
flag.
I'm
gonna
stop
there,
but
really
the
list
could
keep
going
on.
A
Suddenly
the
whole
site
is
down
so
that
may
sound
familiar
to
some
of
you
you're,
not
alone,
but
there
is
hope-
and
there
are
some
things
we
can
do
better
to
avoid
that
so
with
our
services
that
we're
running
now
in
kubernetes
are
really
trying
to
do
things
the
right
way.
So,
when
I
talk
about
the
right
way,
what
do
I
mean?
There
are
a
couple
key
principles:
I
want
to
highlight,
first,
that
are
guiding
our
whole
change
control
process.
A
First,
one
is
that
we
want
our
change
rollouts
to
be
incremental,
so
that
means
that
we
roll
out
our
changes
a
little
bit
at
a
time
so
that,
if
something
is
going
to
break,
we
only
break
it
for
a
small
subset
of
our
system
instead
of
crippling
our
whole
system
at
once.
So
what
does
that
look
like
for
us
in
practice?
Well,
most
you
are
probably
familiar
with
you
know.
A
We
have
a
production
environment
that
handles
our
live
traffic
and
before
we
make
our
changes,
all
the
way
to
a
production
environment
were
going
to
roll
it
out
through
some
intermediate
environments,
so
we're
first
gonna
roll
out
to
a
dev
environment,
a
staging
environment
and
a
production
environment.
But
that's
not
even
the
whole
story
within
each
of
those
environments.
We
also
roll
out
incrementally.
So
we
do
a
canary
deployment
rolling
out
to
a
small
subset
of
the
instances
in
that
environment.
A
A
The
next
sprint
so
I
want
to
talk
about,
is
automation
so
that
whole
incremental
workflow
sounds
great
for
risk
mitigation,
but
it
sounds
pretty
terrible
if
you're
gonna
have
to
do
all
that
yourself
no
wants
to
sit
there
and
you
know
click
a
button
deploy
to
dev
canary
sit
there
and
babysit.
It
make
sure
everything's.
Okay,
okay,
ready
to
deploy
click,
a
button
to
deploy.
Do
your
main
deployment
in
dev
everything
looks
good
click,
a
button
to
deploy
a
staging
click.
A
A
button
to
deploy
a
prod,
you're
gonna
have
to
sit
there
and
babysit
your
whole
rollout
process.
That's
probably
your
whole
afternoon,
maybe
your
whole
day,
but
we
want
to
be
software
engineers,
not
monkeys,
pushing
buttons.
So
we
want
as
much
that
to
be
automated
as
possible,
so
we
can
focus
on
real,
more
important
work.
So
what
that
looks
like
for
us
is
between
our
start
of
our
rollout
process
and
our
finish
we
want
to
have
as
much
of
that
in
the
middle
automated
as
possible.
A
We're
gonna
have
to
have
some
sort
of
probably
manual
intervention
in
the
beginning,
and
we
really
don't
want
to
be
have
to
sitting
and
watching
that
whole
process.
We
want
as
much
of
it
to
take
care
of
itself
as
possible.
Another
benefit
that
automation
gets
us
is,
if
we're
minimizing
the
interface
for
our
engineers
to
have
manual
interaction
with
the
system,
we're
minimizing
the
risk
of
one
of
our
engineers
accidentally
making
mistake
fact
is:
there
are
some
things
that
computers
are
better
than
us
at.
One
of
those
is,
you
know,
simple,
repeated
processes.
A
A
computer
is
much
more
likely
to
accidentally
forget
a
step
in
the
rollout
process
than
you.
Our
last
principle,
I
want
to
highlight
is
that
we
want
our
change
rollout
process
to
be
declarative
first.
So
for
those
of
you
who
aren't
familiar
with
declarative
versus
imperative
models
in
a
declarative
model,
you
make
a
change
to
a
system
by
specifying
a
desired
end
state
of
the
system,
we're
in
an
imperative
model.
A
You
make
a
change
by
specifying
a
set
of
steps
that
you
expect
to
achieve
that
desired
state,
and
one
of
the
benefits
we
get
from
a
declarative
model
is
that
when
we
write
out
our
desired
system
configuration,
we
can
actually
write
that
out
as
code
and
check
it
into
version
control.
So
that
means
we
can
have
a
version
history
of
the
system
of
the
state
of
our
system
at
any
point
and
the
configuration
we
wrote
to
get
to
that
state.
A
So
what
that
looks
like
for
us
in
kubernetes
means
that,
if
we're
running
our
rolling
on
our
app
to
kubernetes,
we're
gonna
need
some
kubernetes
api
objects
to
make
that
work.
So
we
might
need
a
deployment
to
manage
pods.
We
might
need
a
service
object
to
load
balance
across
those
pods.
We
may
need
a
namespace
to
run
our
app
in.
We
want
all
of
those
kubernetes
api
objects
to
be
defined.
Declaratively
some
you've,
probably
all
seen
JSON
and
yellow
files
that
describe
your
apps
config.
A
A
So
to
get
a
quick
look
at
what
our
whole
change
lifecycle
looks
like
I'm
gonna
give
this
big
diagram
think
it's
gonna
be
a
little
confusing
at
first,
so
I'm
going
to
kind
of
breeze
through
it,
but
then
I'm
going
to
go
through
each
of
the
different
components,
in-depth
and
we'll
revisit
it
later.
So
if
this
whole
first
part
doesn't
make
sense,
hopefully
it'll
make
sense
by
the
end.
So,
as
you
can
see,
we
have
a
bunch
of
different
clusters
that
we're
running.
A
A
So,
as
we
said,
we
want
there
to
be
one
minimal
manual
entry
point
for
kevin
to
roll
out
his
changes.
That
entry
point
is
going
to
be
as
app
repo,
so
this
is
where
a
source
code
lives
or
as
docker
file
live
some
other
config
as
well.
We
want
really
that
to
be
the
point
where
Kevin's
gonna
make
his
change
and
we're
gonna
start
the
rest
of
the
rollout
process.
A
So
when
Kevin
makes
a
commit
to
that
app
repo
and
pushes
it
it's
gonna
kick
off
a
git
hook
that
will
trigger
the
start
of
a
Jenkins
pipeline.
So
if
those
of
you
aren't
familiar
with
Jenkins,
it's
an
open
source,
automation
tool,
it
makes
it
really
easy
to
do.
Things
like
build.
Testing
and
deploying
Jenkins
is
really
going
to
be
the
driver
of
the
whole
rest
of
our
automated
rollout
process,
we're
gonna
kind
of
hand
over
the
reins
at
this
point.
A
So,
as
we
said,
we
want
our
changes
to
be
declarative,
which
means
that
if
we
want
to
affect
our
storage
service
as
it's
running
on
our
kubernetes
clusters,
we're
gonna
need
to
modify
the
actual
manifests
that
describe
what
those
apps
look
like
or
what
that
app
looks
like.
So
in
our
system,
we
don't
actually
directly
have
our
Jenkins
pipeline
modify
that
app.
Those
app
manifests.
A
We
actually
have
a
layer
of
indirection
here
we
have
a
templating
system
in
the
way
that
our
templating
system
works
is
that
our
Jenkins
pipeline
is
going
to
inject
certain
parameters
into
our
templates.
That's
going
to
generate
what
the
actual
final
manifest
look
like,
it'll
make
more
sense
in
a
sec
when
we
look
at
it
more
in
depth,
but
suffice
to
say
for
now
that
our
Jenkins
pipeline
modifies
some
parameters
in
our
templating
system
there,
and
then
our
templates
are
written
in
JSON
it,
which
is
a
JSON
templating
language.
We
find
super
useful.
A
We're
then
going
to
run
the
JSON
a
command-line
tool
to
regenerate
our
actual
app
manifest.
So
great
we've
made
our
change
all
the
way
to
our
final
declarative,
spec.
We
say
this
is
what
we
want
our
app
to
or
what
this
is,
what
we
want
our
app
to
look
like
on
the
cluster.
The
last
gap
that
we
have
to
bridge
is
how
do
we
actually
reflect
those
changes
in
the
cluster?
So
we've
actually
built
a
service
to
handle
this?
For
us,
we
call
it
cube.
A
Applier
and
we've
opened
sources
as
well,
and
I'll
talk
a
little
bit
more
about
how
cube
applier
works
later,
but
for
now
just
know
that
cube
applier
serves
the
purpose
of
keeping
that
app
manifest
repo
in
sync,
with
the
actual
state
on
the
clusters,
so
to
look
at
the
app
repo
a
little
more
in-depth
as
the
first
component,
we
have
our
storage
service
as
the
root
name.
We
have
our
source
code,
Kevin's
gonna
have
a
docker
file
as
well.
A
It
describes
how
to
package
this
app
into
a
container
image
and
we
have
a
Jenkins
file,
so
Jenkins
file
is
actually
a
really
cool
way
to
specify
the
steps
that
make
up
your
Jenkins
pipeline
and
you
can
check
that
in
alongside
the
rest
of
your
code.
So,
as
we
said,
there
are
a
lot
of
different
types
of
changes
that
Kevin
cares
about
not
just
source
code
changes,
not
just
changes
to
his
docker
file.
You
may
want
to
change
from
his
kubernetes
configuration
as
well.
A
Maybe
like
he's
running
different
amounts
of
replicas
and
different
environments,
and,
as
we
said,
we
want
this
app
repo
to
be
sort
of
a
single
point
for
that
and
our
manifests
live
elsewhere.
So
how
can
Kevin
modify
those
manifest
changes?
Well,
we
started
rolling
out
a
set
of
parameters
that
Kevin's
able
to
modify
from
within
his
actual
app
repo.
What
that's
going
to
look
like
is
they're
gonna,
be
based
on
our
environment
and
the
track
either
Manor
canary.
A
So
in
this
example,
little
truncated
version
of
our
file,
you
can
see
within
the
dev
environment
and
the
canary
deployment.
We
may
only
be
wondering
running
one
replica
that
only
needs
four
CPUs,
but
in
one
of
our
main
production
deployments
we
may
be
running
20
20
replicas
that
require
more
CPUs.
So
we
want
Kevin
again
to
be
able
to
make
those
changes
to
his
app
within
the
app
repo
and
not
really
have
to
worry
about
the
rest
of
the
automated
system.
A
So,
as
we
said,
when
Kevin
makes
the
change
this
app-
let's
say
in
this
example
he's
making
a
source
code
change.
He
gets
it
reviewed
by
his
team.
He's
gonna
make
a
commit
push
it
to
his
master
branch.
That's
gonna
kick
off
our
Jenkins
pipeline,
so
the
first
part
of
our
Jenkins
pipeline
is
a
bunch
of
steps
that
I've
bundled
under
pre
kubernetes.
A
This
is
less
interesting
stuff
for
this
talk,
but
I'll
go
through
a
quick
we're
going
to
run
some
tests,
an
analysis
for
things
like
code
quality
code
coverage
run
some
unit
tests
make
sure
it
does
what
we
expect
it
to
do.
We
have
some
security
checks
as
well.
We're
gonna
build
Kevin,
a
new
docker
image,
so
he's
made
a
change
to
his
source
code.
We're
gonna
need
to
roll
out
a
new
actual
image
of
his
app.
We
use
docker.
A
Then
we're
gonna
push
that
image
up
to
artifactory,
which
will
be
used
for
our
container
registry
blur.
They
could
use
anything
here
and
then,
as
I
said,
we're
gonna
start
rolling
out
to
our
environments,
so
we
now
need
to
start
pushing
this
image
change
out
or
actual
clusters,
as
I
said,
we're
gonna
roll
out
first
to
a
dev
environment
and
within
that
dev
environment,
we're
gonna
do
a
canary
deployment.
First,
we're
gonna
have
some
automated
testing
and
monitoring
of
that
canary
things.
A
Look
good,
we're
then
gonna
roll
out
to
the
rest
of
our
instances
on
dev.
Do
some
testing
and
monitoring
make
sure
everything
looks
good
if
everything
is
okay
in
dev
Jenkins
is
then
gonna
start
rolling
out
to
staging,
and
our
staging
rollout
process
is
going
to
be
pretty
much
the
same
as
our
dev
environment,
we're
gonna
deploy
to
canary
before
we
deploy
to
Maine.
A
Finally,
if
everything
looks
good
in
staging
we're
ready
to
roll
out
to
production,
but
some
of
you
astute
audience,
members
may
have
noticed
that
the
title
of
this
talk
is
continuous
delivery,
not
continuous
deployment,
so
different
people
have
different
definitions,
but
for
us
what
that
means
in
a
continuous
deployment
system,
all
changes
are
automatically
rolled
all
the
way
out
to
production.
We're
in
a
continuous
delivery
system,
which
is
what
we're
using
all
changes,
are
candidates
to
be
rolled
out
to
production
but
aren't
necessarily
rolled
out.
A
They
require
some
sort
of
manual
approval,
so
continuous
deployment
requires
you
to
have
a
guarantee
with
your
with
your
service
owners
that
their
code
is
ready
to
be
rolled
all
the
way
up
to
production.
At
any
point,
we
don't
currently
have
that
back
guarantee
with
our
service
owners,
but
we'd
like
to
get
there,
but
for
now
we
require
some
manual
approval
to
roll
out
to
production
right
now.
A
The
main
way
that
we
have
that
implemented
for
engineers
is
they'll,
get
a
slack
notification
that
says
that
their
deployment
is
ready
to
roll
out
to
production
and
they,
then
they
can
just
click,
approve
and
then
go
back
to
whatever
they
were
doing
and
Jenkins
is
going
to
handle
rolling
out
to
prod
and
again,
our
prodigal
up
process
looks
very
similar
to
our
Devon
staging
as
well.
So
this
whole
process
that
makes
up
the
Jenkins
pipeline,
as
we
said,
is
defined
in
a
Jenkins
file
that
lives
in
Kevin's
app
repo.
A
A
lot
of
our
pipelines
for
our
different
apps
tend
to
look
very
similar
because
we
want
them
to
follow
this
whole
incremental
process.
So
in
order
to
have
everyone
duplicating
the
same
exact
Jenkins
file,
we've
actually
written
a
Jenkins
pipeline
library
internally.
That
basically
allows
people
to
reuse
some
of
these
common
steps
like
deploying
to
a
certain
environment
as
building
blocks,
so
they
can
get
up
and
running
pretty
quickly,
so
I
very
much
glossed
over
the
actual
deployment
mechanism,
thus
far
I've
pretty
much
just
said.
Okay,
we
deployed
a
canary
and
deploy
to
Maine.
A
So
now
we're
gonna,
look
at
what
that
deployment
mechanism
actually
looks
like
as
I
hinted
at
it
as
something
to
do
with
templates
and
manifests
so
I'm
gonna
try
to
make
that
a
little
more
clear.
So
if
we
work
backwards,
we
have
Kevin's
app
deployed
in
a
bunch
of
different
environments,
dev
staging
production
and
there's
going
to
be
some
configuration
that
differs
across
those
different
environments.
So,
for
example,
I
get
as
I
showed.
In
that
example,
we
might
have
different
replica
counts
in
our
different
environments.
A
So
that
means
we're
gonna
need
different
app
manifests
for
each
of
those
environments,
we're
gonna
Devlin
staging
one
broad
one.
These
are
all
written
in
JSON,
but
a
lot
of
that
config
is
going
to
be
shared
and
duplicated
and
we
don't
want
to
have
to
have
Kevin
write
that
same
config
every
time.
So
what
he's
gonna
do
is.
Instead
we're
gonna
write
a
template.
That's
able
to
kind
of
centralize
a
lot
of
that
shared
config
and
we're
able
to
actually
inject
parameters
into
that
config
or
into
that
template.
A
So,
as
you
saw
earlier,
I
had
that
set
of
parameters
like
the
replicas
CPUs.
So
these
parameters
here
are
going
to
be
a
superset
of
those
ones
that
exist
in
Kevin's
own
repo.
So
these
are
going
to
be
pretty
much
everything
that
differs
across
environments,
because
there
may
be
some
things
that
he
can't
even
control
from
his
repo
that
we
want
Jenkins
to
be
able
to
modify
so
as
it
I'll
explain
in
a
sec
when
I
get
back
to
Jenkins.
A
A
Half
of
of
this
set
up
what
our
Jenkins
pipeline
is
going
to
do
say,
for
example,
we're
rolling
out
a
new
image
tag
and
we're
rolling
it
out
to
the
dev,
Canary
environment
and
track
Jenkins
pipeline
is
going
to
write
the
new
image
tag
into
our
parameters
for
that
environment
and
then
Jenkins
is
going
to
run
our
JSON.
It
commands
taking
the
template
and
the
parameters
as
input
and
is
going
to
regenerate
all
of
our
actual
manifest
files.
A
A
We
have
an
Apps
section.
This
is
going
to
be
where
our
templates
live.
So
pretty
much
the
left
side
of
this
we
have
under
our
storage
service
section.
We
have
our
app
J
sonnet,
which
is
our
template,
and
then
we
have
the
parameters
that
are
injected.
We
have
a
library
section,
that's
gonna
have
things
like
the
shared
sidecar
code,
so
we
have,
as
you
can
see,
we
have
a
smart
stack.
Libs
on
a
template.
Live
sonnet
is
just
a
convention
within
json
it
that
for
shared
library
code,
that's
only
imported.
A
You
want
to
have
that
with
a
lip.
Sonnet
extension
doesn't
really
make
much
of
a
difference,
and
then,
finally,
we
have
our
release
subdirectory
and
this
release
subdirectory
is
what's
going
to
have
all
the
actual
app
manifests
that
are
generated,
so
we
don't
want
to
modify
those
manually.
We
just
want
them
to
be
regenerated
based
on
whatever
is
specified
in
the
templates
and
the
parameter
is
injected
into
those
templates.
So
this
release
directory
really
serves
as
the
whole
state
of
our
system,
and
we
want
that
to
reflect
what's
on
the
cluster
at
any
given
time.
A
So
if
you
look
at
the
bottom,
the
files
that
were
modified
here
I
hope
people
can
kind
of
see
that,
but
we've
modified
both
the
parameters
file
under
the
storage
service,
section
of
app,
so
we've
modified
a
parameter
that
was
injected
and
that's
also
resulted
in
our
app
manifest
being
modified
under
release.
So
our
commit
history-
and
this
repo
is
going
to
have
the
changes
that
were
actually
written
by
Jenkins
and
how
it
actually
affected
the
cluster,
and
we
also
have
some
links
to
source
code
change
the
Jenkins
build
associated
with
this
change.
A
So
now
that
we've
modified
our
manifests
again,
we
want
those
to
reflect
the
actual
state
on
the
cluster.
How
does
that
work?
Well,
as
I
said,
we've
written
a
service
called
Cuba
plier
to
help
us
do
that,
so
a
Cuba
plier
works
is
it
runs
on
the
AR
cluster,
just
like
any
other
service.
So,
as
you
can
see
here,
we
have
an
API
server.
We
have
Cuba
plier,
we
have
our
storage
service
and
I've
also
thrown
in
a
couple
other
example,
services
that
we
run
upload
proxy
and
download
proxy.
A
So,
each
of
these
services
we
want
to
have
defined
by
a
declarative,
app
manifest
that
lives
under
deployment
config
and
the
way
the
Cuba
plier
works
is
Cuba.
Plier
is
going
to
continuously
pull
that
repo
for
changes
and
if
Cuba
plier
sees
that
there
has
been
a
change
to
one
of
the
files,
for
example,
storage
service
now
has
a
new
image
tag.
Cuba
plier
is
going
to
run
a
cube.
A
Ctl
apply
command
to
the
API
server,
so
those
of
you
who
aren't
familiar
with
cube
CTL
apply
it's
basically
the
single
declared
declarative
operator
that
says
here's.
What
I
want
my
object
to
look
like
in
this
file
and
the
API
server
will
handle,
handle,
making
it
a
reality.
So,
if
we've
update
our
image
tag,
API
server
will
handle
rolling
out
the
actual
new
version
of
storage
service,
with
a
new
image
tag.
So
I
mentioned
that
cube.
A
Applier
is
continuously
pulling
for
changes,
but
we
also
have
it
running
on
a
loop
so
that,
even
if
nothing
changes
there's
been
no
commits
to
deployment
config,
we
have
cube,
apply
or
run,
do
a
full
run
through
all
of
the
app
manifests
under
the
repo
every
X
number
of
minutes.
We
do
something
like
10
or
20,
depending
on
our
environment,
but
essentially
what
that
gives
us
is
a
guarantee
that
whatever
is
actually
in
the
repo
is
whatever
is
the
state
of
the
cluster
at
a
given
time.
A
So
if
someone
goes
in
and,
for
example,
manually
modifies
one
of
their
deployments
in
the
cluster,
they
might
use
the
cube.
Ctl
scale
command
change
the
number
of
replicas
to
be
different
from
what's
defined
in
their
app
manifest.
We
always
want
the
app
manifest
to
be
the
source
of
truth,
so
cube
applier,
even
though
no
changes
have
been
made,
it'll
periodically
run
through
that
whole
thing
and
it'll
override
the
manual
change
that
was
made
and
resynchronized
the
state
of
the
system
with
the
actual
with
the
repo.
A
So
again,
this
gives
us
a
better
guarantee
that
whatever's
in
the
repo
is
versioned
history,
of
whatever
the
state
of
our
system
is
again
Cuba
pliers,
also
an
open-source
project.
It's
available
on
github,
so
I'll
have
some
links
at
the
end
to
that
and
if
other
people
might
find
it
useful
as
well.
It
really
allows
us
to
kind
of
abstract
away
this
process
and
we
get
to
use
sort
of
the
repo
as
a
real
interface
for
interacting
with
the
cluster
and
not
have
to
worry
about
actually
manually
running
those
cubes
ETL
apply
commands
themselves.
A
Some
of
you
might
be
I'll.
Get
your
questions
at
the
end,
I'm
happy
to
talk
more
after
some
of
you
might
be
for
more
familiar
with
like
push
sorta
models.
Where,
like
you
know,
you
may,
when
you
make
a
change,
your
CI
is
just
gonna
run
the
cube.
Ctl
apply
command
itself,
as
I
said.
One
of
the
things
we
like
about
this
model
is
that
it
gives
us
kind
of
a
stronger
guarantee
that
whatever's
actually
in
the
repo,
is
whatever's
actually
in
the
cluster
to
given
time,
because
we're
continuously
looping
through
and
ensuring
it's
synchronized.
A
All
right
so
to
zoom
out
for
a
little
bit
we
have.
Each
of
our
clusters
is
going
to
be
running
its
own
instance
of
Cuba
plier.
Each
of
those
instances
is
going
to
be
looking
at
a
specific
subdirectory
within
our
release
that
within
our
release
directory
that
has
again
all
of
the
app
manifest
for
that
specific
environment.
A
So
one
of
the
really
cool
things
that
this
gets
us
is
that
if,
for
example,
we
have
a
outage
in
one
of
our
whole
clusters,
so
one
of
our
production
clusters
goes
down.
We
have
a
version
history
of
the
entire
state
of
the
system.
At
any
point,
so
we
can
easily
spin
up
another
cluster
to
replace
it
run
a
new
instance
of
Cuba
plier
and
pointed
at
the
same
directory
we're
using
before,
and
we
can
spin
up
all
of
our
apps
exactly
as
they
existed
before.
A
So
when
you
don't
really
have
a
centralized,
a
centralized
store
of
your
state
at
any
time
say
each
of
their
apps
only
rollout
with
their
own
pipeline
and
apply
their
own
manifests.
Your
cluster
goes
down.
It
can
be
hard
to
make
sure
you
kick
off.
All
of
the
pipelines
to
bring
up
their
instances
exactly
as
they
were
so
again.
This
was
gives
us
a
pretty
good
disaster
recovery
model
for
that
sort
of
thing.
A
So
to
zoom
back
you
get
back
to
our
what
our
change
life
cycle
looks
like,
hopefully,
it'll
make
a
little
more
sense.
Now
we
have
our
app
repo
where
we
want
to
make
our
changes.
It's
going
to
kick
off
a
Jenkins
pipeline,
that's
going
to
handle
rolling
out
to
our
different
environments,
we're
gonna
roll
out
to
those
different
environments
by
modifying
parameters
injected
into
our
templates
they're,
going
to
use
our
templates.
A
We
generate
the
actual
app
manifests
that
described,
but
we
want
our
cluster
to
look
like
and
then
Cuba
plier
is
going
to
handle,
making
sure
that
those
app
manifests
are
a
reality
in
the
cluster.
So
as
a
challenge
to
this
model,
I
want
to
introduce
Joe.
Joe
is
also
here
at
cube.
Con
Joe
is
a
box
engineer
on
our
service
discovery
team,
so
Joe
as
I
mentioned,
a
lot
of
our
apps
run.
A
So
in
our
current
system,
as
I
said,
we
have
everything
centralized
under
that
deployment,
config
repo,
so
we
still
do
have
an
entry
point
for
Joe
to
essentially
make
his
change
in
that
centralized
shared
library
that
everyone
pulls
from
and
then
Joe
can
have
his
own
pipeline
that
actually
regenerates
everyone
else's.
Json
files
regenerates
their
app
manifest
to
pick
up
his
change
and
he
can
roll
out
all
their
apps
for
them.
We
don't
really
love
that
system
because
it
kind
of
takes
away
the
control
from
the
service
owner
themselves.
A
If
they're
expecting
that
all
their
changes
are
being
rolled
out
through
their
pipeline
and
they're
modifying
that
suddenly
we're
deploying
their
service
without
their
knowledge,
they
might
not
have
any
record
that
the
fact
that
their
last
time,
their
service
was
deployed
is
different
from
whatever
they
have
in
their
pipeline
history.
So
we'd
really
like
to
do
is
have
our
app
owner
is
actually
able
to
kick
off
all
of
their
pipelines
when
we
need
to
make
a
change
and
roll
that
change
out
through
their
pipeline.
A
That's
gonna
require
us
to
be
doing
continuous
deployment,
so
we
want
Joe
to
make
a
change
to
that
shared
library
and
then
have
a
big
red
button
where
he
can
kick
off
everyone's
pipelines
all
at
once,
so
that
they
pick
up
the
change
they
have
their
own
history
of
that
deployment
in
their
own
pipeline.
So
that's
one
reason
we'd
like
to
get
to
get
to
newest
deployment
is
so
that
we
can
roll
out
changes
to
all
the
apps
all
the
way
to
production
at
once.
A
We
have
a
greatly
reduced
risk
of
production
outages
so
because
we're
rolling
out
things
incremental
II,
it's
a
lot
lot
decreased
risk
that
something
that's
bad
is
gonna,
make
its
way
all
the
way
out
to
our
main
production
deployment
and
cause
some
real
problems,
because
we
have
a
version
history
of
the
state
of
our
system.
At
any
point,
we
have
a
really
good
audit
trail
for
when
we
need
to
debug
things
or
rollback,
or
someone
makes
a
change
lends
up
affecting
the
whole
system.
A
We
can
kind
of
roll
back
our
entire
state
to
whatever
it
was
before.
Then,
we've
increased
our
engineering
productivity
because
our
engineers
can
focus
on
more
important
problems
than
just
sitting
in
babysitting
their
deployments,
waiting
for
it
to
roll
out
all
day.
If
we
do
require
manual
intervention,
we
have
to
come
up
in
a
easy
way
that
where
you
can
get
back
to
doing
whatever
you
were
doing.
A
Lastly,
our
engineers
have
greater
confidence
in
making
changes,
so
changes
can
be
scary,
but
if
we
give
them
a
process
that
makes
them
feel
confident
deploying
their
code,
we
want
our
engineers
to
feel
innovative
and
feel
like
they're
able
to
make
some
changes
and
build
some
cool
things.
So
that's
all
I've
got
I've
got
a
couple
links
here.
A
If
you
want
to
read
some
more
about
some
of
the
blog
post
we've
written
on
this
topic
and
we're
also
going
to
be
working
on
some
more
content,
blog
post
and
an
open-source
repo
as
well,
to
kind
of
make
a
more
practical
example
of
what
this
looks
like
outside
of
box.
So
that
happy
to
open
it
up
to
any
questions
that
people
have.
A
A
Yeah
I
don't
have
a
great
example
for
that
specific
case,
but
there
are
some
similar
cases
that
we
have
tried
to
solve.
So,
for
example,
say:
you're,
changing
an
environment,
variable
that's
used
by
your
service
in
your
source
code,
you're,
going
to
make
that
change
both
in
your
source
code
and
you're,
also
going
to
need
to
change
the
environment.
A
Variable
that's
passed
in,
and
the
kubernetes
config
that's
one
of
the
reasons
we're
trying
to
bring
those
parameters
into
the
app
repo
so
that
we
can
actually
do
an
atomic
commit
that
changes
both
the
source
code
and
the
environment.
Variable
that's
passed
in
as
far
as
a
change
to
a
database
schema
that
we
don't
really
have
any
insight
into
in
our
kubernetes
config
that
one's
pretty
tough
I,
don't
think
we
have.
A
group
I
have
a
great
answer
for
that
in
our
current
framework,
but
happy
to
talk
more
after
about
it.
B
A
When
so,
if
someone
makes
a
manual
change
to
one
of
our
service
running
in
our
cluster,
so
we
do
have
some
access
controlled
so
that
we
can
limit
what
people
can
change.
But
if
someone
does
make
a
change
to
something
they
do
have
access
to,
we
do
have
some
auditing
in
place
as
far
as
like
cube,
cuddle
commands
that
are
being
run
on
the
cluster.
We
have
logs
of
that
available
again
for
the
most
part,
we'd
like
things
to
be
rolled
out
through
our
declarative
process,
but
we
have
tried
to.
A
A
A
Yep
we
do,
we
do
have
config
Maps,
we
do
have
secrets
secrets,
are
kind
of
a
tough
one
to
roll
out
with
this
process
and
that
you
don't
really
want
to
specify
your
you
know:
secrets
declaratively
in
a
file
as
they're
exactly
expected
to
look.
We
were
doing
some
work
on
an
encrypted
secrets
project
that
we
kind
of
lost
track
on,
but
hopefully
can
get
back
on
board
and
maybe
open
source.
Essentially
we
want
you
to
be
able
to
declaratively
specify
your
secrets
in
a
way
that's
safe.
A
For
now
our
secrets
are
largely
configured
manually,
but
we
do
have
config
maps
as
well
that
we're
using
so
yeah
so
I,
don't
supposed
to
so
smart
stack.
We
use
as
our
sort
of
service
discovery
mechanism
of
like
which
services
need
to
talk
to
which
other
services
smart
stack.
We
use
some
config
maps
to
pass
in
the
information
that
it
needs,
but
our
smart
stack
config
is
also,
as
I
said.
There's
some
code,
that's
shared,
but
each
of
the
apps
are
going
to
have
some
things
that
differ
among
their
source.
A
Our
smart
stack,
shared
library,
isn't
just
like
a
drop
in
function.
That
does
everything
for
you
you're
going
to
pass
in
some
information
about
which
other
services
you
need
to
talk
to
it
just
kind
of
abstracts
away
some
of
the
shared
code.
I,
don't
know
if
that
answers
your
question
I'm
happy
to
talk
more
afterwards
about
it.
Yep.
A
So
the
question
was:
can
Kevin
or
Joe
modify
their
Jenkins
files
that
they
can
deploy
directly
to
production?
So
we
do
have
some
restrictions
on
our
actual
Jenkins
machines.
As
far
as
like
which
all
right,
we
do
have
some
restrictions
within
our
Jenkins
infrastructure.
As
far
as
like
what
you're
doing
and
like
you
can't
just
have
what
we
don't
want
you
to
have
a
Jenkins
file
that
can
just
arbitrarily
do
whatever
steps
it
wants.
A
That's
one
of
the
reasons
why
we
really
focus
on
having
that
pipeline
library
that
kind
of
gives
people
building
blocks
that
they
can
use.
If
I
don't
know
if
you're
talking
about
like
a
malicious
case
or
like
an
accidental
case
again,
we
want
to
try
to
limit
the
accidental
cases
where
people
are
doing
something
that
they
shouldn't
be
doing.
We
want
to
make
it
easy
for
them
to
just
kind
of
follow
process.
We've
laid
out.
A
A
Yeah,
sorry,
so
the
question
there
was:
why
do
we
have
our
multiple
environments
specified
within
sorry,
just
distinct
within
a
file
system
like
as
subdirectories
I,
think
versus
having
different
branches
like
a
branching
model,
where
we
have
a
dev
branch
and
a
staging
branch?
I
don't
have
a
great
answer
for
that.
I
was
not
part
of
that
historical
choice,
yep.
A
So
our
access
model
I
did
very
much
gloss
over
essentially
the
way
we
implemented
it
a
while
back
when
there
wasn't
a
lot
of
you
know.
We've
been
running
kubernetes
for
about
three
years
now
since,
like
0.11,
so
some
of
these
changes
are
historical
changes
based
on
the
way
kubernetes
was
in
early
States
and
we
haven't
totally
caught
up
in
some
ways.
A
So
as
far
as
clients,
we
try
to
match
those
the
groups
that
have
access
to
certain
namespaces
with
an
LDAP
system
that
we
have,
and
so
we
want
to
ensure
that
you
know
we
have
some
sort
of
you
know,
production
auditing
system
for
who
has
access
to
what
and
again
we
also
in
order
to
make
those
changes
to
make
changes
to
access
control.
We
need
to
bounce
the
API
server,
so
that
is
going
to
some.
That's
something
that's
going
to
require
the
attention
of
our
admins.
A
So
the
question
was:
are
we
seeing
the
need
to
move
our
app
manifests
into
our
the
service
repos?
This
has
been
a
going
battle
for
us.
We
do
like
the
idea
of
having
our
service
owners
able
to
manage
their
configuration
alongside
the
rest
of
their
code.
It
does
get
a
little
dicey
for
some
of
the
shared
libraries
and
the
things
that
we
want
to
be
able
to
generate
from
a
centralized
place.
So
we
want
our
smart
stack
developers
to
have
some
kind
of
centralized
entry
point
that
everyone
pulls
from.
A
So
that's
why
we've
started
to
pull
out
some
of
those
parameters
into
the
app
repo
without
pulling
the
whole
canonical
source
of
truth
manifest.
Another
thing
we
get
from
having
all
of
those
app
manifests
in
one
place
is
that's
like
our
single
source
of
truth
of
what
the
cluster
looks
like
if
we
have
those
kind
of
dispersed
through
all
our
different
app
repos
across
our
system.
A
Again,
we
don't
really
have
a
clear
picture
of
like
what
each
app
is
running
at
any
point,
so
it's
possible
we
might
be
able
to
just
move
the
whole
app
manifest
out
to
someone's
repo
we'd
still
probably
have
a
copy
of
it
somewhere.
That
has
everything
centralized
and
have
that
kind
of
be
the
canonical
source
of
truth.
It's
it's
an
ongoing
debate
for
us.
That's
pretty
interesting!