►
From YouTube: A Staging Environment Per Pull Request: Building Our Own Controller! [B] - Kamal Marhubi, Heap
Description
A Staging Environment Per Pull Request: Building Our Own Controller! [B] - Kamal Marhubi, Heap
Services like TravisCI and CircleCI make it easy to run your test suite on every pull request. But sometimes you just want to kick the tires of a complete system with your changes applied. Kubernetes let's us do far better than an ad hoc staging environment: launch an entire environment in the cluster for each pull request! In this talk, we'll walk through the design of a system for declaratively specifying staging environments. Along the way we'll take a look behind the curtain and learn about the beautifully simple control loop architecture that underlies all Kubernetes components.
About Kamal Marhubi
Kamal Marhubi loves programming, systems, and ringing bells.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
So
yeah
I'm,
I'm,
Kemal
and
I
work
at
a
place
called
heap.
It's
a
user
analytics
company
and
I'm
here
today
to
talk
to
you
about
having
a
staging
environment
for
every
floor
request,
but
more
than
that
specifically
about
how
to
use
kubernetes
to
build
a
system.
So
here
at
cube
con
I.
Imagine
a
lot
of
us
are
very
interested
in
running
different
kinds
of
things:
on
top
of
kubernetes
run
everything
on
top
of
grouper
Nettie's
right.
A
So,
like
everyone
around
our
storage
on
kubernetes,
there's
a
talk
right
now
about
rook,
which
is
some
set
kubernetes
thing.
And
then
you
have
block
stores
and
file
systems
and
there's
linker
D
and
bringing
more
things
so
that
you
can
run
your
externally
facing
websites
getting
their
ingress
all
in
the
magical,
kubernetes
universe
and
there's
all
this
service
mesh
talk
and
then
so,
if
we're
not
talking
about
running
things
on
kubernetes
we're,
then
instead
talking
about
making
things
run
more
smoothly
on
kubernetes.
So
this
kind
of
thing
encompasses
metrics
and
logging
and
tracing
and
monitoring.
A
We
want
all
of
our
magical,
kubernetes
stuff
to
be
as
magical
as
possible
and
when
it's
not
magical,
we
want
to
be
woken
up
and
told
about
it
really
quickly.
So
we
can
go
back
to
making
a
magical
again,
but
the
way
I
want
to
frame
kubernetes
in
this
talk
a
little
bit
is
to
switch
from
being
something
that
you
run
software
on
and
that
you
spend
a
lot
of
effort
making
it
possible
to
run
software
on
smoothly
to
being
something
that
you
actually
build.
A
Something
on
and
I'll
go
into
a
little
bit
as
we
go
through
the
talk
on
what
I
mean
by
this.
But
the
really
key
thing
I
want
to
give
you
here.
Is
that
a
little
bit
not
a
little
bit
of
knowledge
about
how
kubernetes
works?
How
the
underpinnings
are
put
together
can
go
a
really
long
way
to
building
something
really
cool
with
relatively
little
effort.
A
So
the
just
to
frame
this
I
said
I
work
at
heap,
I
started
there
in
November,
and
shortly
after
starting
I
saw
a
proposal
go
by
for
building
a
system
where
we
can
easily
see
what
the
app
looks
like,
based
on
a
pull
request.
So
we
don't
have
to
like
deploy
to
some
staging
environment
or,
like
trust
the
developers
like
say
that
you
know
hey.
It
looks
great
trust
me
on
my,
like
local
docker
stuff,
it's
cool,
we're
like
well
it'd,
be
nice.
A
If
we
could
actually
have
a
URL
that
we
can
give
to
people
to
poke
around
and
see
what
changed.
So,
someone
proposed
this
and
it
had
kind
of
a
let's.
Do
this
I
don't
know
how
feeling
to
it?
There
were
a
lot
of
unknowns,
and
so
some
of
those
unknowns
were
like
how
are
we
gonna
handle
like
having
six
of
these
different
environments
running
at
once?
How
are
the
port's
not
going
to
step
on
top
of
each
other?
A
How
are
we
gonna
be
able
to
look
at
logs
for
all
of
these
things
like
they're,
running
somewhere,
I?
Think
the
you
know,
one
idea
was
like:
let's
we
already
have
dr.
Campos
setup:
let's
run
it
in
elastic
container
service
which
takes
docker
compose
config.
How
do
we
make
sure
we're
not
just
leaving
a
bunch
of
stuff
running
and
wasting
our
money
in
ec2,
and
then
how
do
we
expose
us?
A
These
staging
deploys
that
we
can
see
them,
so
we
can
pass
them
to
your
teammate
to
someone
on
customer
success
who
wants
to
see
what's
going
on
with
the
change
that
you're
making
and
well
kubernetes
has
like
reasonably
good
answers
to
all
of
these
right.
How
do
we
have
ports
not
step
on
each
other?
How
do
you
have
containers
not
step
on
each
other's
toes
with
the
ports?
Well,
there's
the
whole
networking
model
container.
I
p
per
container.
You
use
the
ports
you
want
to
use.
A
You
don't
have
to
have
some
kind
of
complicated,
dense
logging
at
a
very
basic
level.
You
can
get
it
from
the
command
line,
cubes
ETL
logs.
You
can
get
it
from
the
built-in
UI.
This
doesn't
require
any
setup
and
then,
if
you
want
to
be
complicated,
you
know,
there's
probably
like
five
other
talks
right
now
about
how
to
go
on
do
more
clever
stuff.
With
logs
tearing
down
cube,
CTL
delete
L,
you
know,
delete
everything
with
a
given
label.
A
It's
all
gone
blown
away,
no
worry
that
you're
leaving
stuff
running
around
and
how
to
get
it
so
that
you
can
pass
that
URL
to
someone
well,
the
ingress
stuff
is
perfect
for
this
you
did
create
an
ingress
resource
and
then
that
will
set
it
up
so
that
your
load
balancer
and
your
cloud
provider
or
your
nginx
front-end,
has
a
virtual
host
or
whatever
to
point
back
to
this
exact
service.
So
this
is
like
really
cool
and
I
was
like.
A
Okay,
kubernetes
has
got
all
of
these
things,
so
I'd
like
to
say,
I
actually
went
and
built
the
whole
thing
we
didn't
we're.
A
small
team,
like
the
whole
thing,
is
not
necessary
for
the
size
of
our
team,
yet
it
was
kind
of
a
proposal
that
was
put
out
there,
as
this
would
be
something
that
be
good
to
have
like
inside
the
next
year
or
so,
as
our
team
gets
bigger,
but
I
still
throw
together
a
proof
of
concept
and
I'm
gonna.
A
A
This
is
an
excellent.
This
is
like
a
perfect,
commit
message
for
this
update
index.html.
Everyone
will
know
exactly
what
I'm
doing
and
why
so
I'm
just
going
to
go
ahead
and
create
the
pull
request
and
down
here
there's
a
little.
It's
got
my
face
on
it.
That's
cool
little
cube
review
thingy,
and
this
has
a
link
to
pr6
cube
reviewed.
A
This
takes
a
little
while
the
the
link
gets
set
up
before
the
thing
is
actually
running,
but
the
URL
there
is
pr6
cube,
review,
Kamal,
nginx
Kemal,
nor
who
beat
cube
reviewed,
ka
desa,
come
out
of
that
cloud.
There
we
go
I
just
wanted
to
get
the
part
where
it's
come.
All
delight.
I
bought
that
just
for
this
I
thought
it
was
great.
A
A
This
is
in
US,
East,
I,
think
US
central
one
anyway,
it's
somewhere
Google
container
engine
somewhere
else,
and
this
is
publicly
on
the
internet
from
tour
requests,
and
it's
just
like
it's
there
so
demo
time,
demo
God
smiled
on
me,
I'm
really
happy
so
there's
kind
of
a
lot
going
on
there
right,
like
you
know
we
take.
Maybe
we
take
this
for
granted,
like
you,
click
create
a
pull
request
and
then
Travis
does
all
the
stuff
you've
set
up
your
integration
and
all
that
or
using
circles.
Cia
there's.
A
If
you
know
anyone
working
at
any
of
these
places,
there's
a
lot
of
stuff
going
on
behind
the
scenes
to
make
that
work.
I.
Think
it's
really
cool
that,
like
basically
I,
was
able
to
throw
this
together
in
a
couple
of
days.
Obviously
it's
not
robust.
You
can
try
and
make
pull
requests.
You
will
probably
break
it.
A
If
you
do,
please,
let
me
know:
I'm
I
know
it's
broken
and
not
very
robust,
but
I
would
like
to
knowing
exactly
which
ways,
but
I
was
able
to
throw
this
together
in
a
couple
of
days
and
like
my
level
of
go,
is
Yolo
I
don't
know
go.
This
was
like
all
of
the
go
I've
ever
done
in
my
life
was
building
this
plus
and
other
hundred
lines,
or
so
so.
A
This
is
like
I,
don't
know
a
lot
of
copy
paste,
slapping
together,
stuff
out
of
kubernetes
repositories
and
examples,
and
it
works,
and
it's
cool
and
I
think
that
the
reason
it
is
able
to
work
and
be
cool
is
not
that
I'm,
a
genius
or
anything
like
that.
It's
just
that
I,
like
was
really
excited
about
kubernetes
and
read
a
lot
of
the
documentation
and
the
underpinnings.
A
So
I
want
to
talk
a
little
bit
about
what's
going
on
in
it
and
yeah
just
coming
back
to
that,
a
little
bit
of
this
knowledge
about
how
this
stuff
fits
together
can
go
a
long
way.
You
know
two
days
to
that
little
thing,
so
this
I
want
to
talk
about
how
the
kubernetes
architecture
works
and
how
it
actually
makes
makes
your
workloads
run.
So
this
is
a
muffin
tray,
it's
not
it's
not
services
or
anything.
This
is
a
muffin
tray,
and
so
you
want
to
be
baking.
A
Muffins
you're,
probably
gonna
put
them
in
your
oven.
This
is
your
oven
and
your
oven
has
a
thermostat
and
you
dial
in
190
Celsius
375
Fahrenheit,
for
this
is
probably
slightly
too
hot
for
muffins.
But
that's
fine,
because
this
is
a
kubernetes
talk,
I'm,
not
a
baker
but
yeah.
So
you
dial
in
the
level
you
want.
You
set
the
desired
state
on
your
oven
and
then
the
oven
adjusts
the
heating
elements
or
the
gas
fire
thingies
to
maintain
it.
So
if
it's
too
hot
it'll
turn
them
off,
it's
too
cold.
It'll
turn
them
on.
A
So
is
this
control
loop,
there's
a
thermometer
or
bimetallic
strip
or
whatever
there's
something?
That's
measuring
the
temperature
and
that's
giving
feedback
back
to
the
heating
element.
It
turns
them
off
if
it's
too
cold
and
then
that
feeds
back
and
increases
the
temperature
and
it's
too
hot
and
it
turns
it
off.
So
is
this
control
loop
is
what's
underpinning
the
thermostat
and
also
control
absorb
underpin,
basically
everything
in
kubernetes
the
way
the
cubelet
works.
The
way
the
scheduler
works,
the
way
the
ingress
controller
works.
A
All
of
these
things
are
built
on
this
same
idea
of
control
loops
having
a
desired
state
observations
of
the
real
state
and
working
on
bringing
them
closer
to
each
other.
So
the
cubelet,
for
example,
is
got.
It
got
darker
that
it's
it's
able
to
affect
it's
able
to
create
containers
and
it's
also
observing
from
docker.
A
So
if
the
cubelet
knows
it's
supposed
to
have
seven
containers
running-
and
it
only
sees
four-
then
it
will
create
the
extra
three
and
then
it
will
observe
from
darker
that
okay
now
there
are
seven
running
and
they're
supposed
to
be
seven
running,
that's
good
and
then,
if
some
for
some
reason
you
decide
to
tear
down
some
work,
some
workload,
then
all
of
a
sudden.
Now,
instead
of
seven
containers
on
the
supposed
to
be
running
on
that
node
they're
only
supposed
to
be
six.
Well,
then
the
cubelet
will
see.
A
Oh
I'm
running
I've
got
seven
nodes
in
docker,
but
I'm
only
supposed
to
have
six
and
it
will
delete
the
one.
That's
not
supposed
to
be
there
so
again,
there's
a
desired
state
which
is
weak,
which
exact
containers
are
supposed
to
be
running
on
the
node
and
there's
the
real
estate,
which
is
which
containers
are
actually
running
in
in
docker
at
the
moment,
and
it's
just
reading
from
and
writing
to
this
state
or
effecting
change
to
make
it
come
closer.
A
The
ingress
controller,
which
is
what
I
use
to
get
that
really
long,
URL
wildcard
DNS
entries,
are
cool.
You
can
just
like
have
really
really
long
stupidly,
nested
domains
with
one
line
of
DNS,
but
what
that
goes
to
is
nginx
running
in
my
kubernetes
cluster
and
when
you
create
a
new
service
and
give
it
an
ingress
resource
which
is
telling
kubernetes
to
expose
it
so
that
someone
can
visit
it
from
outside
of
the
cluster,
then
what
that
comes
down
to
is
a
curly
brace
block
somewhere
in
an
engine
X
config
file.
A
It's
like
server,
hostname,
pr6
dot.
You
know
that
really
long
hostname
and
any
other
like
path
setup
that
you
have
and
then
it
points
to
the
back
end,
which
is
the
kubernetes
service,
is
supposed
to
talk
to.
So
when
you
create
an
ingress
that
results
in
adding
some
config
to
the
nginx
to
the
nginx
containers,
and
then
when
you
remove
one,
then
it
observes
that
oh,
the
config
is
out
of
sync
with
what
I'm
expecting
there
are
not
enough.
A
It's
how
we
dial
in
the
temperature
in
this
case
it's
how
we
run
of
this
garish,
yellow
background
with
blue
writing
on
website
and
in
order
for
us
to
be
able
to
go
a
step
of
beyond
what
is
just
provided
by
the
queue
bernardi's
resources.
Kubernetes
actually
gives
us
a
way
to
define
our
own
resources.
So
these
are
called
third-party
resources,
they're
an
extension
of
some
kind
but
they're
a
way
to
say
so.
You
know
resources
in
kubernetes
the
ones
we're
familiar
with
pod
service
deployment.
A
These
are
all
you
know
the
thing
that
goes
after
kind
in
the
in
the
ammo.
We
can
actually
define
our
own
so
for
this
demo,
I
defined
my
own,
which
is
a
review
deployment
resource
and
the
review
deployment.
Is
the
data
that's
necessary
about
the
pull
request
in
order
to
download,
build
and
run
that
the
code?
A
So
now
we
can
actually
create
our
own
custom
review
deployments
and
submit
them
into
the
kubernetes
api
server
and
they'll
have
meaning
that
so
from
the
api
service
point
of
view,
that's
just
a
blob
but
we're
able
to
request
and
modify
and
watch
and
do
all
of
the
things
that
the
api
server
makes
available
with
that
resource,
so
to
come
back
to
the
actual
architecture.
For
this
very
very
simple
example,
when
you
click
create
a
pull
request,
github
sends
a
web
hook
post
to
this
repo
listener,
which
is
just
a
little
service.
A
A
This
is
the
review
review
deployment
resource,
so
we
pull
out
the
information
we
want
and
then
post
it
to
the
API
server,
the
kubernetes
api
server.
We
have
taken
the
data
about
the
pull
request
and
created
a
review
deployment
and
stuck
that
into
the
into
the
cluster,
and
so
the
the
parts,
the
different
resources
that
we're
like
writing
and
creating
within
kubernetes.
For
this
are
there's
this
review
deployment
our
custom,
one
we're
gonna
need
to
build
and
push
at
that
commit.
A
So
there's
going
to
be
a
pod,
that's
just
launched
into
into
the
cluster
and
then
after
it's
been
built
and
pushed
there's
another
pod
to
run
it
and
then
there's
some
services
and
ingresses,
and
I'm
not
going
to
go
into
like
how
these
things
are
written.
It's
just
basically
templates
that
situation
like
I
defined
what
I
wanted
you
know.
I
have
a
five
line:
bash
script
that
runs
up
docker,
build
and
pushes
it
somewhere
and
just
substitute
in
the
argument
of
like
the
commit
ID
and
the
the
source
to
get
it
from
and
so
on.
A
A
A
Turning
on
and
off
the
heater
or
creating
or
deleting
a
pod,
then
you
could
also
write
the
kubernetes
controller,
like
it's
not
drastically
different
from
a
tiny
little
I.
Don't
know.
If
I
once
tried
to
write
a
thermostat
for
a
sous-vide,
cooking
thingy
and
it's
like
it's
not
a
lot
of
code,
it's
like
you,
know,
20
lines
of
C
or
whatever
it's
like
read
the
thermometer
turn
on
the
heating
element
to
turn
off
the
heating
element.
A
Here,
it's
listen
for
new
things
from
kubernetes,
create
things
delete
things
it's
just
as
straightforward
and
conceptually
it's
like
all
it's
there
I
will
say,
don't
create,
don't
write
a
thermostat
like
these
things
could
burn
your
house
down
at
the
very
least.
If
you
do
write
one
and
you
do
burn
your
house
down,
I
didn't
tell
you
to
do
it,
I
told
you
not
to
do
it.
It
says
so
so.
A
A
You
know
I
came
up
with
this.
That's
not
that
great,
but
this
platform
is
really
powerful.
I
don't
have
to
store
I,
don't
have
to
have
a
postgrads
or
anything
to
store
the
the
state.
For
all
of
these
things,
it's
all
managed
by
kubernetes
for
me,
and
I
really
hope
that
you'll
think
about
different
things
that
you
can
do
by
taking
this
like
really
complex
engine.
That
has
a
kind
of
like
relatively
nice
interface
that
is
kubernetes
and
come
up
with
other
cool
things
that
are
built
on
top
of
that
so
yeah.
A
A
A
What
I
wrote
didn't,
but
there's
no
reason
that,
like
that
would
be
an
extension
like
in
two
days,
I
didn't
get
there
in
four
days.
I
probably
would
have
gotten
there.
Like
that's
kind
of
you
know
it's
not
necessary.
Just
to
repeat
the
question
is
what
would
happen
when
you
update
the
PR,
because
you
know
like
frankly,
those
it's
it.
A
A
So
I
actually
translated
the
yamo
into
the
question
is
how
I
templating
in
order
to
just
send
things
into
kubernetes
I,
actually
translated
the
llamo
in
yeah
the
Amal
in
to
go
literal
structures
like
structures
as
a
literal,
because
I
was
doing
this
thing
go
and
then
just
defined
the
values
in
there
I
didn't
use
a
templating
engine.
I
was
going
to,
but
I
ended
up.
Finding
that
easier.
A
A
So,
in
this
case,
it's
it's
not
about
the
repositories,
but
about
that
particular
pull
request.
I
keep
the
state
of
what's
going
on
for
that
pull
request,
which
is
have
we
built
it?
Yet?
What
was
the
image
that
we?
What
was
the
name
for
the
image
we
created
out
of
it
and
stuff
like
that?
Yeah?
There
was
a
question
for
the
back
somewhere
music,
oh
yeah,.
A
A
May
or
may
not
work
anyway,
there's
a
wild
card
record
for
one
of
the
subdomains
on
this
may
be
the
whole
kamala
log
cloud
and
then,
when
you
make
an
HTTP
request,
the
host
header
includes
what
you're
actually
trying
to
do
and
that's
what's
being
like
used
by
nginx
I'm
kind
of
cheating,
but
it's
like
what
any
shared
host
does:
I'm,
not
cheating
more
than
like
dreamhost
or
anyone
else.
Yeah.
A
A
A
A
There
I
haven't
figured
out
what
my
answer
would
be,
but
like,
however,
you
orchestrate
that,
at
the
moment,
like
possibly
like
I,
think
the
apps
couger
net,
that
apps
resource
is
a
way
to
handle
multiple
workloads
or
something
or
you
could
just
have
like.
Instead
of
a
template
for
one
pod,
a
template
for
many
pods
and
kind
of
plug
things
in
yourself,
it
would
be
some.
A
Ideally,
it
would
be
something
generic,
but
an
easy
way
would
be
to
like
encode
your
your
application
as
a
bunch
of
templates
and
then
create
the
different
resources
for
that,
but
yeah
the
for
this
actually
be
of
any
use
to
anyone
really
is
probably
going
to
have
to
be
able
to
handle
multiple
containers
because
most
like
reasonable
applications
are
at
least
going
to
have.
You
know
a
data
store
of
some
kind
yeah.