►
From YouTube: Are You About to Break Prod? Acceptance Testing with Ephemeral Environ... Erin Krengel & Sean Holung
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Are You About to Break Prod? Acceptance Testing with Ephemeral Environments - Erin Krengel, Pulumi & Sean Holung, Nordstrom
How confident are you that the changes you’re about to make won’t break production? In a world of Continuous Delivery, we need to be prepared for the fact that our code is going to production. K8s makes it easy to quickly deploy applications, so building pipelines with robust quality gates is vital. There’s a lot of emphasis on this, yet how to create a solid deployment strategy isn’t clearcut. Erin and Sean will demonstrate a pattern for acceptance testing complex architectures, which verifies a K8s app properly interacts with its infrastructure. Leveraging ephemeral environments, these tests will validate as well as document the app’s business and functional requirements. Utilizing infrastructure as code and K8s Jobs, they will demo how to create a comprehensive acceptance test suite that allows you to continuously deploy to production.
https://sched.co/UaWE
A
So
before
we
get
started,
I
just
want
to
remind
you
all
that
the
slides
are
available
online
and
all
the
code
will
be
on
get
lab
as
well
for
your
reference
after
the
question.
We're
all
here
to
ask
ourselves
is:
how
confident
are
you
that
you're
not
about
to
break
production
and,
along
with
that
comes
the
question
house
are
in?
Are
you
that
the
changes
you're
about
to
make
do
exactly
what
you
expect?
They
meet
all
the
requirements
set
forth
by
your
p.m.
engineering
manager,
etc.
A
B
The
services
we
built
today
often
contain
a
large
complex
of
web
of
dependencies.
Our
services
need
to
talk
to
databases,
storage,
buckets,
connect
to
message
brokers.
We've
got
to
make
sure
that
we
have
all
of
our
permissions
configured
correctly
and
much
more.
There
are
a
lot
of
moving
pieces
to
make
everything
happen.
A
Well,
Sean
and
I
were
on
the
order
management
team
at
Nordstrom.
We
were
faced
with
a
challenge
of
how
to
continuously
deliver
an
event-based
architecture
with
confidence.
It
was
a
critical
component
to
her
ecosystem
and
we
needed
to
be
able
to
be
certain.
We
weren't
going
to
break
anything.
We're
super
excited
to
be
here
with
you
all
today,
sharing
with
you
some
of
our
experiences
testing
enterprise
software.
B
So
before
we
get
started,
let's
just
go
over
some
quick
fundamentals
when
it
comes
to
testing,
so
our
unit
tests
are
going
to
focus
on
testing
those
internal
implementation.
Details
of
our
application,
so
does
this
function,
I
wrote
behave
the
way
that
I
expect
acceptance
test.
On
the
other
hand,
which
you
may
some
times
here,
referred
to
as
integration
test
or
a
functional
test,
these
tests
are
going
to
test
your
entire
service
along
with
its
infrastructure
and
make
sure
if
they're
behaving
the
way
that
we
expect.
B
A
When
testing
a
traditional
micro
service,
the
setup
is
pretty
straightforward.
We
have
some
code
that
calls
our
service
that's
being
tested
and
then
validates
the
response.
We
often
see
this
being
done
in
a
lower
environment.
That's
used
for
testing
the
challenge
with
testing
environments.
Is
that
often
become
a
bottleneck
for
deployments.
B
A
Does
Sean,
but
were
these
are
ephemeral
environments
to
do
just
that?
An
ephemeral
environment
is
a
short-lived
environment
that
we're
gonna,
spin
up
for
testing
and
then
teardown
short-lived
is
subjective.
So
for
testing
we're
going
to
want
this
to
be
quick
if
we're
debugging
or
troubleshooting,
we
may
leave
that
environment
up
for
a
little
bit
longer.
The
bounds
of
our
environment
are
also
subjective
to
maintain
agility,
we're
going
to
want
to
make
sure
we're
only
testing
the
particular
micro
service
we're
trying
to
deploy.
A
That
means
we're
not
gonna
want
to
spin
up
every
micro
service
or
a
piece
of
infrastructure
we
haven't,
but
we
may
need
to
spin
up
some
extra
pieces
of
infrastructure
to
be
able
to
properly
validate
our
service.
For
example,
if
our
service
is
writing,
an
outbound
message
to
a
pub
sub
topic
will
likely
need
a
subscription
to
pull
from
that
topic
and
validate
the
outbound
message
was
sent.
B
Next,
we're
going
to
want
to
make
sure
that
our
infrastructure
is
templatized
so
on.
This
will
allow
us
to
associate
specific
tests
and
environments
with
our
with
the
course
or
with
their
corresponding
resources.
So
here
you
can
see
that
we're
doing
that
by
just
leveraging
that
variable
that
we
could
figure
it
in
the
previous
slide
and
we're
using
it
when
we
go
to
define
our
infrastructure
next,
we're
going
to
want
to
make
sure
that
both
our
application,
as
well
as
our
tests
are
configurable.
B
So
these
extra
resources
are
that
we
need
to
deploy,
for
our
tests
are
going
to
we're
going
to
use
them
to
make
validations
against
our
resources
that
are
running
a
product
or
that
we'd
be
the
point.
And,
lastly,
we're
going
to
want
to
make
sure
that
we
clean
up
our
resources
at
the
end
of
every
test
run,
and
this
will
make
our
environment
ephemeral.
A
Each
service
is
its
own,
unique
snowflake,
and
only
you
have
the
context
to
make
the
right
call
for
your
service.
We're
gonna
want
to
balance
the
speed
of
our
test
with
how
closely
our
formal
environment,
mimics
production,
a
kubernetes
cluster,
takes
significantly
longer
to
spin
up
than
a
pub
subtopic
to
save
time
as
well
as
money,
we
may
choose
to
run
all
of
our
tests
against
one
database
instance,
whereas
in
production
will
likely
want
our
own
dedicated
database
instance.
A
A
B
So
today,
we've
put
together
a
simple
demo
for
you
all
to
demonstrate
the
effectiveness
of
integrating
acceptance
testing
with
the
ephemeral
environments
into
your
deployment
process.
So
our
manager
has
tasked
us
with
developing
a
service
that
receives
a
message
and
upon
receiving
this
message,
we'll
go
ahead
and
take
that
message
and
place
it
in
a
storage
bucket
and
then,
if
that
succeeds,
we're
gonna
publish
a
notification
that
about
this
message
that
other
downstream
services
can
subscribe
to
by
publishing
that
message
on
a
pub
sub
topic.
B
So
whenever
we
go
about
developing
a
new
service,
we
want
to
make
sure
that
we
established
a
solid
deployment
strategy
up
front.
This
is
gonna,
ensure
that
we're
building
in
quality
from
day
one.
So
before
we
get
into
the
details
of
our
service,
let's
go
ahead
and
take
a
look
at
what
our
pipeline
looks
like
so,
for
today
we're
gonna
be
using
gitlab
CI
for
all
of
our
pipelines
and
the
demo
that
you'll
see.
Today.
Of
course,
you
can
apply
any
of
these
concepts
and
ECI
CD
system.
B
B
B
A
Before
we
continue,
I
want
to
quickly
introduce
Pulu
me
a
bit
more
since
it's
the
tool
we'll
be
using
the
provision
all
over
infrastructure
in
the
rest
of
the
pipeline.
Paluma
is
an
open
source.
Infrastructure
is
code
tool
that
allows
you
to
use
a
variety
of
programming
languages
to
provision
your
infrastructure.
B
So,
coming
back
to
our
pipeline
immediately
following
our
acceptance
stage,
we've
got
our
preview
stage,
so
the
plumie
CLI
comes
with
a
command
called
luma
preview,
and
this
is
essentially
just
going
to
do
a
dry
run
of
all
the
infrastructure
that
we're
going
to
provision
before
we
actually
go
to
provision
it.
So
it's
going
to
give
us
an
idea
of
what
resources
are
going
to
be
created,
updated
or
deleted
before
we
actually
go
to
make
those
changes.
B
So
many
teams
like
to
incorporate
this
into
their
code
review
process
because
it
gives
them
a
chance
to
validate
all
the
infrastructure.
That's
going
to
be
changed
and
all
the
resources
they're
going
to
be
changed
before
they
actually
move
that
it's
a
prod
and
the
final
stage
in
our
pipeline
is
Pulu
me
up.
So
this
is
the
stage
that's
actually
going
to
go
ahead
and
deploy
our
infrastructure,
as
well
as
our
service
application
to
prod,
assuming
that
all
the
other
previous
stages
in
our
pipeline
of
passed,
of
course,.
A
A
So
are
in
our
main,
go
file
and
you
can
see
we
have
configuration
that
we're
taking
in
we
have
this
helper
called
get
configuration
value.
This
is
a
helper
function
that
just
reads
an
environment
variable
and
if
it's
not
present,
it
shuts
down
our
application.
This
is
because
everything
we
do
today
is
gonna,
be
configuration
driven
and
if
those
variables
are
not
set,
we've
done
something
wrong
and
we
want
our
application
to
shut
down.
These
are
also
what's
going
to
allow
our
application
to
talk
to
different
resources
and
different
environments.
A
A
A
We
then
write
the
message
to
our
bucket
and
return
the
object
name
that
object
name
is
used
to
create
a
new
message,
alert
which
is
marshaled
in
two
bytes
and
those
bytes
are
used
to
write
to
the
pub
sub
topic
as
well
as
respond
to
our
client,
and
we
were
still
responding
with
a
status
accepted.
Okay.
So
now
that
we
have
basic
understanding
of
our
application,
let's
go
back
and
look
at
our
architecture,
so
this
diagram
is
a
little
bit
overly
simplified.
A
There's
actually
many
more
resources
that
we
need
to
create.
To
make
this
all
happen.
We
need
service
account
provision
with
the
correct
permissions.
We
need
to
get
a
key
for
that
service
account
which
make
into
a
kubernetes
secret
and
in
volume
out
to
our
deployment,
we're
also
going
to
have
a
kubernetes
service
with
a
cluster
IP
to
load
balancer
application,
so
we'll
use
paluma
to
spin
up
all
of
these
resources
and
every
produce
in
a
bowl
manner.
A
A
We
then
have
some
application
specific
configuration,
so
we
have
our
and
which
is
our
name
of
our
environment
and
so
in
production.
This
would
be
like
prod
or
production
for
one
of
our
thermal
environment.
This
is
gonna,
be
like
test
1,
2
3,
and
then
we
also
taken
our
docker
tag,
and
this
is
so
that
we
make
sure
we're
always
using
that
same
immutable
image.
A
We
then
have
our
global
stock
reference,
and
this
is
those
longer-lived
pieces
of
infrastructure
that
we
talked
about
earlier.
We
have
a
kubernetes
cluster
and
a
namespace
we've
set
up.
We
also
have
a
service
account
for
our
acceptance
test,
and
this
is
what
our
acceptance
test
will
use
to
make
validations
against
the
GCP
resources.
A
We
also
have
this
conditional
statement
below,
which
is
the
service
account
for
our
application
in
a
testing
environment.
We're
going
to
use
our
test
service
account
and
in
production
we're
going
to
want
to
use
our
production
service
account.
This
is
how
we
can
easily
make
deviations
between
our
environments
using
configuration.
A
We
then
set
up
a
name
variable,
and
this
name
variable
is
what
we're
going
to
use
to
name
all
of
our
pieces
of
infrastructure.
You
see
we,
we
prepend
the
environment
name
to
it,
and
this
allows
us
to
easily
identify
all
of
our
test
resources
as
well
as
make
sure
that
we
can
spin
up
multiple
ephemeral
environments
at
once,
so
Shawn
can
be
running
a
pipeline.
I
can
be
running
a
pipeline,
and
since
we
each
have
our
own
unique
pipe
on
an
ID
they're,
not
gonna
conflict.
A
A
So
we've
provisioned
the
correct
permissions
on
our
service
account,
and
now
we
need
to
get
a
key
and
we
use
that
key
to
create
a
kubernetes
secret.
This
is
the
first
kubernetes
resource
we
look
at
so
I
just
want
to
take
a
second
to
talk
about
it
a
little
bit
more.
So
it
looks
very
similar
to
a
kubernetes
manifest
file
that
you
may
be
familiar
with
the
structure.
A
All
of
these
are
the
same
since
we're
using
a
programming
language
such
as
typescript,
we're
gonna,
get
some
type
safety
auto-completion
things
like
that,
and
we're
also
going
to
be
able
to
reference
that
service
account
key
that
we
just
created
above
so
if
we
were
using
something
else
to
provision
our
infrastructure,
we
may
need
a
lot
of
bash
magic
to
make
this
happen,
but
here
it's
as
easy
as
just
referencing
a
variable
okay,
and
then
we
have
our
deployment.
Once
again,
our
deployment
is
looks
very
similar
to
a
kubernetes
manifest
file.
A
This
is
just
going
to
actually
be
running
our
application.
You'll
see
we
get
some
things
like
reuse
and
drying
out
our
code
by
creating
those
a
playable
above
which
we
can
reference
throughout
our
deployment,
as
well
as
in
our
kubernetes
service
that
you'll
see
below
some
other
things
to
call
out
about
this
deployment.
A
We
have
that
docker
tag
that
we
talked
about
earlier.
We
have
environment
variables
that
were
setting
for
our
deployment,
and
we
can
reference
that
topic
and
bucket
that
we
just
created.
This
is
what's
gonna,
configure
the
Go
application
that
we
saw
earlier,
and
it's
also
was
going
to
make
sure
that
we're
always
talking
to
the
correct
resources
in
the
correct
environment
and
lost
them
to
call
up
here
is
that
we
have
that
volume
amount
where
we
can
just
reference.
A
The
secret
last
resource
we
have
to
create
here
is
our
service,
and
you
can
see
where
we
referencing
those
app
labels
that
cluster
provider
again,
and
so
this
is
the
same
pluming
program
that
will
run
for
a
few
more
environments
as
well
as
production.
Let's
go
ahead
and
take
a
look
at
how
we're
gonna
accept
and
test
this.
A
So,
since
our
service
is
not
publicly
exposed,
we're
gonna
use
a
kubernetes
job
to
run
our
acceptance
test.
This
is
a
great
technique
to
use
when
you're
running
CI
from
some
external
runners,
such
as
get
lab,
CI
or
Travis,
our
exception
test
will
run
in
the
job
and
if
our
acceptance
tests
pass,
then
our
job
will
pass
in
our
cept
infest
fail.
Then
our
drought
will
fail.
This
is
also
what
we'll
actually
be
doing.
The
validation
to
make
sure
service
is
operating
correctly.
Let's
go
ahead
and
take
a
look
at
the
go
code.
A
That
is
our
acceptance
test,
so
we
have
a
subdirectory
called
acceptance.
We
have
this
acceptance
underscore
test
go
file.
You
see
that
we're
once
again
taking
configuration
in
everything
we
do
is
configuration
driven.
So
we
want
to
make
sure
our
acceptance
tests
are
validating
against
the
correct
resources.
A
We
then
have
our
happy
path
test
and
we
set
up
our
clients
to
talk
to
our
GCP
resources,
and
then
we
create
a
test
message
and
at
this
point
everyone's
a
little
bit
concerned
because
we've
looked
at
go
code,
we've
looked
at
type
script
code
and
you're
wondering.
Are
we
about
to
look
at
some
Ruby
code
as
well?
A
A
The
last
thing
we
have
to
do
is
clean
up
the
object,
and
this
is
because
we
want
to
be
able
to
cleanly
teardown
our
formal
environment,
and
we
also
have
a
side
path
test
which
we
post
a
message
warning
marshal
and
validate
that
we
get
back
that
400
okay.
So
let's
go
back
and
look
at
our
architecture.
One
more
time.
A
So
we've
already
looked
at
how
we
use
paluma
to
spin
up
all
those
blue
pieces
of
infrastructure,
and
now
we
need
to
conditionally
provision
six
more
pieces
of
infrastructure.
Where
need
are
our
job,
we're
going
to
need
correct
permissions,
secret,
etc?
So,
let's
go
look
at
halloumi
and
see
how
we
are
polluting
code
to
see
how
we've
provisioned
that
conditioning
the
bottom
of
our
index
dot
TS
file,
so
this
was
that
type
to
a
file
that
we
walked
through
earlier
and
the
right
under
that
service.
A
We
have
our
conditional
statement
and
we're
using
a
program
in
language.
It's
as
easy
as
an
if
statement.
If
our
environment
starts
with
test,
then
set
up
our
acceptance
test,
and
this
doesn't
just
set
them
up.
It
actually
is
going
to
also
run
the
job
since
production
doesn't
start
with
test.
It
won't
set
up
our
acceptance
test
for
prod.
Let's
click
in
and
see
what
that
looks
like.
A
A
And
then
we
create
our
kubernetes
job,
so
a
Cooper
new
job
is
actually
going
to
be
what's
running
our
acceptance
tests,
some
things
to
call
out
here,
you
see
that
dr.
tag
again,
you
see
those
configuration
values
easily
being
passed
in
and,
at
the
very
end,
you'll
see
the
dependency
on
our
our
deployment.
So
this
means
that
we
will
Pulu
me
we'll
wait
until
our
deployment
is
complete
before
it
spins
up
the
job.
B
Awesome
our
pipeline
has
passed.
Our
application
is
deployed
to
production
and
businesses
out,
throwing
a
party
that
is
celebrating
all
the
new
features
that
have
been
deployed
and
our
customers
are
extremely
happy
and
of
course,
we
were
able
to
do
this
by
leveraging
acceptance
testing
with
the
ephemeral
environments
that
gave
us
that
confidence
to
move
quickly.
A
A
This
means
the
peripheral
behavior
of
our
application
is
going
to
remain
identical,
but
we're
gonna
have
to
change
both
our
application
and
your
infrastructure.
Luckily,
we've
set
ourselves
up
for
success
by
creating
a
robust
pipeline
and
we'll
be
able
to
move
quickly
and
make
this
change.
So
let's
go
ahead
and
look
at
the
PR
that
we
opened
to
make
this
happen,
so
you
can
see
that
we're
removing
our
dependency
on
pub/sub
for
application.
A
A
B
B
We
can
see
that
we're
setting
up
those
configurable
values
that
we've
been
talking
about
during
the
presentation
and,
of
course
we
ran
Pulu
me
up-
and
here
you
can
see
that
we're
skipping
the
preview,
because
this
is
just
for
our
acceptance
test,
so
we're
just
going
to
go
ahead
and
just
spin
up
all
the
infrastructure.
So,
of
course,
this
command
deployed
our
service
application,
spun
up
all
of
our
ephemeral
infrastructure
and,
of
course,
ran
that
acceptance
test
job.
So
if
we
keep
scrolling
here,
let's
see.
B
B
So
we
can
see
it
failed
here.
It
failed
when
we
try
to
publish
that
message
to
pub/sub
and
it
looks
like
it
expected
a
value,
but
it
got
nil.
So
it
seems
like
our
message:
payload
was
empty
and
we
went
to
publish
so
we
can
see
a
few
more
failures
occurring
here.
These
all
seem
to
be
related
to
the
failure.
Above
so
we
can
keep
scrolling
and
we
think
looks
good
and
at
the
end
we
are
running
our
paluma
destroy
command.
B
A
Push
to
change
that
fixes
this
and
we'll
look
at
the
different
layers
commit
that
was
pushed,
so
we
set
payload
format
to
none
when
any
interview
JSON
it's
easy
to
miss
things
and
code
reviews.
You
think
you've
done
things
correctly,
but
until
you've
actually
spun
up
an
environment
and
run
code
through
it.
It's
often
difficult
to
know
that
things
are
working
as
they
should.
A
So
we
see
it's
all
green,
which
is
awesome.
You
see
that
the
stages
are
identical
to
those
stages
that
Shawn
walked
us
through
earlier
on
our
merge
request.
The
difference
here
is
that
we
don't
do
that.
Pulu
me
up
to
production,
because
it's
a
merge
request.
We
want
to
make
sure
we're
validating
everything
correctly.
So
we've
done
our
code
review.
We
know
our
acceptance.
Tests
are
working,
let's
go
ahead
and
look
at
our
plumie
preview
and
see
actually
before
we
do
that.
A
Can
we
scroll
to
the
top
I
just
want
to
show,
so
this
pipeline
took
seven
minutes
and
55
seconds
we
often
see
somewhere
between
six
minutes
and
eight
minutes
for
a
pipeline
to
run
on
a
merge
request.
This
is
pretty
fast
to
be
building
our
acceptance
test,
our
service
running
our
acceptance
test
and
during
our
preview,
let's
go
ahead
and
take
a
look
at
our
plumie
preview.
A
So,
as
part
of
our
merge
request
review
process,
we
can
look
and
validate
that
our
infrastructure
changes
are
the
ones
that
we
expect.
So
we
scroll,
we
see
our
plumie
preview
and
then
we
can
see
a
lot
of
the
pieces
of
infrastructure
or
remaining
identical.
The
difference
is
that
we
have
a
topic.
I
am
binding
that
we're
adding
we're,
adding
a
notification.
A
A
B
Using
ephemeral
environments
with
our
acceptance
test
makes
developers
even
more
productive
infrastructure
is
a
critical
component
of
our
application.
We
need
to
be
validating
that
our
application
is
correctly
interacting,
with
its
defendant
infrastructure,
being
able
to
run
these
tests
before
merging,
to
master
and
being
able
to
run
multiple
tests
at
once
gives
us
those
quick
feedback
loops
and,
most
importantly,
we
encourage
all
of
our
engineers
to
use
automated
testing.