►
Description
Speakers: Isaac Mosquera and Kate MacAleavey
What does culture have to do with continuous delivery? Everything. Companies implement new tools to accelerate delivery but don’t account for the necessary beliefs that underpin a successful transformation. Culture work is the competitive differentiator for digital transformation, expediting the adoption of CI/CD tools. Continuous delivery embodies the values strongly correlated with high performance: trusting and empowering developers to make deployment decisions, prioritizing safety in service of velocity, building resilient systems with chaos engineering, replacing controls with context by providing as much information as possible to support decisions, and fostering an experimental culture by creating a safe-to-try, safe-to-learn environment. As more organizations undergo digital transformation it’s critical to work on the associated cultural changes. More companies are finding that it’s a combination of new technologies and new behaviors that truly unlock innovation.
A
Thank
you.
I
think
the
way
that
we
wanted
to
start
the
webinar
today
was
really
kind
of
thinking
about
the
last
time
that
you
have
deployed
something,
and
if
you
just
kind
of
close
your
eyes
and
think
about,
when
was
the
last
time
you
released
something
into
production
and
what
were
those
feelings
like?
A
A
How
you
feel,
when
you
are
performing
some
sort
of
technical
activity
is
a
lot
of
what
we're
going
to
talk
about
today,
because
the
way
that
certain
people
feel
in
certain
processes
and
in
certain
organizations
when
they
release
code
is
very
different
than
many
other
organizations,
depending
on
what
the
repercussions
are
for
failure.
What
the
process
is
to
get
something
out
the
door.
How
do
you
gain
confidence
in
the
process?
How
do
you
assess
risk
right
and,
if
you
think
about
a
software
delivery
process,
what
we're
doing
along
each
individual
step
is
gaining
confidence.
B
In
a
little
bit
more
context,
so
isaac
is
currently
the
cto
at
armory
that
you
know
uses
spinnaker
to
well.
They
implement
spinnaker
to
enterprise
companies
and
I
used
to
work
at
armory,
but
I'm
an
executive
coach
and
I
also
do
culture
transformation
work
as
a
consultant,
and
so
that's
initially
how
I
actually
started
to
work
with
armory
back
when
we
were
engaged.
B
A
So
the
first
thing
that
we
really
want
to
talk
about
is
complexity
and
how
the
world
is
changing
in
not
only
just
the
the
the
digital
world,
but
also
how
our
organizations
are
are
actually
organized
right,
how
our
companies
are
organized,
how
humans
are
put
together
and
to
put
together
to
work
and
and
the
complexity
for
on
the
digital
side
and
on
the
technical
side
is
increasing
exponentially
and
the
reason
why
is,
or
one
of
the
main
reasons
why
is,
is
microservices.
So
what
had
happened
in
the
past?
A
Is
we
used
to
just
grab
a
handful
of
developers?
Maybe
20
30
100
developers,
and
we
used
to
work
on
this
thing
called
the
monolith
and
the
monolith
was
just
one
application
and
we
could
reason
about
that
application
and
its
architecture,
and
we
had
that
three-point
architecture
that
we
would
go
from.
A
You
know
api
gateway
to
the
application
to
the
database,
and-
and
it
was
rather
simple
but
as
the
internet
grew
and
as
the
number
of
applications
on
the
internet
started
to
grow,
and
the
number
of
things
that
we
wanted
to
do
it
digitally
started
to
grow.
That
became
untenable.
We
couldn't
have
a
thousand
developers
working
on
a
single
monolith,
because
there
was
too
many
bottlenecks
and
it
became
too
slow
to
do
so.
So
what
did
we
do?
A
There's
too
many
things
that
could
go
wrong
and
for
you
to
isolate
just
single
points
of
failures,
it's
too
challenging,
and
so
our
world,
our
world
is
changing
and
the
complexity
that
comes
along
with
it
means
that
we
have
to
think
differently
about
how
we
deliver
code
into
production.
What
used
to
be
this?
One
application
that
we
could
put
onto
a
handful
of
servers
is
now
literally
thousands
of
applications
built
by
thousands
of
engineers
put
onto
hundreds
of
thousands
of
servers
right
and,
and
so
now
we
have
to
think
of
of
things
differently.
B
So
I
think
it's
also
helpful
to
understand.
Like
the
more
I
started,
to
work
at
armory
and
I
worked
with
isaac
in
understanding
kind
of
you
know,
I
was
brought
into
work
on
the
cultural
components
of
the
organization
to
really
help
move
it
from
a
commanding
control
environment
to
a
more
like
free
and
autonomous
environment.
B
That
was
one
process,
but
the
second
process
was
really
learning
about
digital
transformation,
as
well
as
continuous
delivery
and
seeing
where
people
were
really
stuck
and
where
clients
and
customers
were
really
trapped
and
so
isaac,
and
I
were
finding
out
that
the
language
that
we
actually
used
and
the
theories
that
we
use
were
identical,
whether
it
was
culture,
transformation
or
continuous
delivery,
and
so
I
think
that's
like
a
fascinating
point.
So
just
keep
that
in
mind,
because
what
we're
trying
to
do
today
is
kind
of
tie
these
things
together
and
really
bring.
B
You
know
the
software
component
into
the
psychological
opponent
really
both
ways.
So
this
was
really
helpful.
When
I
first
learned
about
the
monolith
and
microservices,
I
was
like
wait
a
minute.
I
was
like
okay
like
we're
working
on
that
culturally
as
well,
and
so
I
also
want
to
quickly
define
culture
and
the
way
we
define
it.
So
culture
really
is:
how
do
we
do
everything
this
isn't
about,
like
your
benefits
or
your
perks?
This
is
literally
like
how
do
you
show
up
together?
How
do
you
do
meetings?
B
How
do
you
make
decisions?
How
do
you
let
people
go
like
literally
it's
the
the
how
of
your
work,
so
I
think
it's
important
to
get
really
clear
on
that,
so
zooming
out
really
thinking
about
what
complicated
versus
complex
is
when
it
comes
to
systems
theory.
This
is
really
where
a
lot
of
our
work
is
predicated
upon
and
so
like
early
on
organizations
and
systems
were
very
much
complicated
in
the
way
that
it's
a
causal
system.
B
So
if
you
think
of
a
car
engine
right,
if
you
do
x,
you're
going
to
get
y
like
causal
means
like
you,
have
a
direct
impact
and
you
know
exactly
what
it
is
and
it
can
be
measured,
and
so
the
hierarchy
of
an
organization
was
actually
built
because
we
used
they
used
to
be
complicated
systems
back
in
the
manufacturing
days.
And
so
that's
why
the
hierarchy
actually
was
established.
And
so
similarly
right,
if
you
think
about
like
that,
is
the
monolith.
B
But
now
the
environment
and
context
has
changed
completely
and
we
now
have
a
complex
environment,
meaning
that,
instead
of
something
that's
causal,
we
have
something:
that's
not
causal,
so
you
think
of
a
car
engine,
and
now
you
think
of
like
weather,
you
can
try
to
predict
it.
You
can
try
to
control
it.
You
can
try
to
hope
it
into
doing
something
like
you're
not
going
to
actually
get
anywhere
with
that
right.
So
how
do
you
actually
manage
and
deal
with
a
complex
system
and
very
similar
to
what
isaac
is
talking
about
with
microservices?
B
You
can't
use
the
same
practices
in
theory
that
you're
using
in
a
monolith
right,
you're.
Actually
going
to
be
really
restricted,
so
what's
important
here
is
understanding
that
using
the
right
kind
of
applications
with
the
right
context,
so
rules
really
strict
rules
make
sense
when
something
is
complicated.
B
Just
like
think
of
the
rule
book,
you
have
in
your
organization
that
restricts
you
from
doing
things
and
is
telling
you
exactly
how
to
do
something.
It
doesn't
allow
for
you
to
be
nimble
or
flexible,
or
to
really
respond
to
your
environment,
which
is
hyper
critical
to
actually
thriving
right
now,
and
that's
very
much
in
respect
and
reflects
what's
going
on
with
microservices.
A
Yeah
and
to
I
guess,
to
give
an
example
of
like
I
like
how
you
broke
down
the
complicated
versus
complex,
an
engine
in
a
weather,
and
if
you
think
about
weather
systems,
you
can
actually
provide
the
same
input
into
any
weather
prediction
system
and
actually
end
up
with
different
results,
and
how
that
relates
to
microservices
is
your
same.
Input
can
go
through
the
same
api
and
go
through
some
of
the
same
components
but
it'll
land
on
different
computers,
it'll
land
on
different
nodes.
A
You
could
end
up
with
with
different
results
that
internal
api,
that
is
three
or
four
layers
down,
could
fail.
It
could
be
400.
Second
milliseconds
delayed
right,
so
you
don't
always
actually
get
the
same
response
in
in
the
way
that
we
used
to
with
systems.
In
fact,
actually
now
with
the
cloud,
we
actually
expect
systems
to
fail
right
and-
and
this
is
again
popularized
by
by
google
very
long
ago
what
they
said.
What
what?
A
As
you
mentioned,
like
building
organizations
like
we
were
built
to
do
this
tops-down
hierarchy
to
build
things
like
engines
and
control
the
entire
supply
chain
right,
because
the
way
to
win
historically
in
business
has
always
been
control.
What
your
organization
does
and
then
the
supply
chains
below
it,
but
now
in
this
world
right
so
now
we
have
sure
we
have
the
cloud.
A
But
if
you
look
inside
of
the
cloud-
and
the
number
of
you
know
with
the
the
number
of
microservices
you
have
also,
you
know
you
have
java,
javascript
c,
plus
plus
golang-
you
have
so
many
different
languages,
so
many
different
variables
and
to
apply
these
rules
that
you
were
mentioning
on.
Developers
only
only
serves
to
kind
of
slow
us
down,
but
the
benefit
now
is
is
now
diminished,
because
we
we
have
so
many
differences
in
each
layer
of
our
stack,
whether
it
be
in
our
supply
chain
or
our
technology
chain.
B
And
I
love
that
isaac
is
bringing
that
up,
because
really
the
the
basis
of
a
lot
of
our
work
is
actually
looking
at
like
the
old
patterns
and
behaviors
that
come
along
with
this
work.
So
you
know
when
we
look
at
customers
and
clients
and
we
listen
to
them
complaining
about
their
continuous
delivery
process
or
trying
to
transform
as
much
as
they
may
be.
Speaking
about
the
software
they're,
really
talking
about
people,
issues
and
behavioral
issues
right
like
it's
actually
a
lot.
B
It's
a
lot
of
learning
and
a
lot
of
change
that
has
to
happen
moving
from
a
monolith
to
microservices
or
moving
into
the
cloud
and
what
we're
finding
is.
Companies
are
not
really
honoring
that
process
and
they're
getting
so
frustrated
because
transformations
like
this
are
extremely
complex,
but
at
the
end
of
the
day,
these
are
people,
and
this
is
behavior
change
and
if
you
don't
honor
those
kind
of
like
invisible,
critical
components
like
you're
going
to
get
stuck
and
we
saw
it
over
and
over
and
over
again.
B
So
the
more
we
started
to
really
address
that
the
easier
things
started
to
get
as
far
as
adoption
goes,
and
so
I'd
love
isaac
to
share
an
example
of
actually
a
customer
that
we
were
working
with
where
they
were.
You
know
they
were
trying
to
change.
They
were
trying
to
get
to
a
zero
percent
downtime
with
their
customer
and
failure,
but
it
had
repercussions
so
I'd
love
isaac
to
talk
about
that.
A
Yeah,
so
let's,
let's
relate
this
back
to
software
delivery
or
continuous
delivery
and
like
a
pattern
that
kate
you
and
I
saw
was
okay,
we
want
the
benefits
of
microservices.
We
want
to
move
fast.
We
want
the
teams
to
be
distributed
and
essentially
kind
of
self-sufficient,
but
at
the
same
time
we
want
all
the
software
delivery
patterns
to
be
exactly
the
same,
and
we
want
them
all
to
go
out
the
exact
same
pipeline
in
the
way
that
we
used
to
do
it
in
the
data
center.
A
You
have
the
overhead
of
microservices,
but
you're
constraining
everything
to
go
through
a
single
pipeline.
So
you
get
none
of
the
benefits.
The
reason
I
went
back
to
this
architecture
diagram
imagine
trying
to
release
all
of
these
services
at
one
time:
you're,
not
releasing
a
thousand
services
altogether
in
tandem.
Additionally,
how
do
you
test
this?
A
How
do
you
build
systems
that
you
know
what
you're
going
to
push
out
the
door
is
going
to
be
reliable
and
get
that
you
know?
Okay,
you
were
mentioning
100
reliability
right,
but
at
its
basis,
at
its
core
of
microservices.
The
idea
is
that
you
right
getting
back
to
what
google
and
others
have
figured
out
a
long
time
ago-
is
that
in
fact,
when
you're
breaking
things
down
is
that
you
are
expecting
failure
and
that
a
single
failure
won't
take
out
the
system.
A
So
you
have
to
think
differently
because
if
you
think
in
the
old
way,
you're
never
going
to
get
to
this
speed
that
these
industry
giants
have
gotten
to
and
so
you're
going
to
have
to
start
breaking
things
down,
not
only
at
the
micro
service
level,
but
your
organizational
level
right
stop
start
thinking
about
right,
netflix,
although
they
use
spinnaker
predominantly
that
choice
is
not
mandated
on
anybody.
They
can
use
their
own
deployment
tools
to
get
things
out
the
door.
A
They
don't
have
to
go
through
spinnaker,
but
they
do
so
because
it's
easier
now
are
there
cases
in
which
they
don't
use,
spinnaker,
of
course,
right
and
and
and
because
they're
able
to
do
that,
certain
teams
are
able
to
move
faster,
where
spinnaker
may
not
be
able
to
support
their
use
case
yet,
or
maybe
they
try
to
build
into
spinnaker
to
help
support
their
use
cases.
So
the
idea
here
is
making
sure
that
the
services
can
run
independently
of
each
other
and
are
just
loosely
coupled.
That
word
is
loosely
coupled.
B
So
bringing
that
into
like
the
behavioral
and
organizational
cultural
parts
of
things
right
like
loosely
held,
so
we
think
of
like
principles
instead
of
having
strict
rules
right,
so
it's
kind
of
like
85
percent
of
the
time
80
of
the
time.
This
is
like
what
we're
trying
to
get
to
we're,
allowing
for
flexibility
to
happen,
because
we
understand
the
environment
is
extremely
complex
right,
so
you
think
about
maybe,
like
your
hr
policies,
how
you
use
money
in
your
company
how
transparent
you
might
be.
B
Some
companies
are
like
very
strict,
but
the
problem
with
it
being
strict.
Is
it's
not
flexible
right?
It
doesn't
allow
for
creativity,
it
doesn't
allow
for
innovation
and
oftentimes
in
innovation.
It
happens
by
accident
right.
You
notice
like
a
pattern
that
you
weren't
actually
paying
attention
to
before,
and
so
it's
really
like
how
do
you
honor
and
think
about
your
organization
as
well
as
your
software
delivery
in
both
complex
systems,
you
can't
hold
it
too
tightly,
but
what
comes
with
that?
B
Which
is
what
isaac
and
I
were
really
picking
up
with
with
the
customer,
was
like
the
amount
of
fear
and
like
the
the
fear
that
comes
along
with
that
and
then
the
control
that
people
really
really
want,
and
how
do
we
kind
of
help
people
ease
themselves
into
that
process?
Where
you
get
more
comfortable
with
uncertainty,
I
think,
obviously,
with
covet
happening.
It's
kind
of
like
oh
wow,
there's
extreme
uncertainty,
and
maybe
we
never
had
it
in
the
first
place,
but
man
we're
going
to
try
to
control
it
as
much
as
we
can.
B
It's
a
natural
instinct
to
try
to
control
these
things,
but
the
more
you
try
to
do
that
in
both
processes
and
culture,
transformation
and
continuous
delivery.
It
gets
worse,
things
actually
stop
and
they
halt
and
some
people
take
great
pride
in
it.
And
so
we
had
a
customer
sit
there
talking
about
the
single
window,
the
single
deployment
window
and
they
were
like.
B
Oh,
we
have
like
zero
percent
downtime
now
and
we
were
like
and
what
happened,
your
innovation
and
they
looked
at
us
blank
like
blankly-
and
this
is
a
very
large
company-
that's
actually
known
for
innovation.
They
weren't
even
measuring
it,
and
so
it's
fascinating
to
see
like
it.
They
were
more
focused
on
having
this
perfect
zero
percent
downtime
than
finding
a
risk
that
they
were
comfortable
with
and
that
it
could
be
known
for
and
actually
designed
around.
A
Yeah
and
interesting
on
that
point,
kate
right.
We
see
this
with
like
change
review
boards
and
it's
been
proven
that
change
review
boards
don't
actually
help
with
the
reliability
of
deployments
being
or
being
safer.
That's
in
the
accelerate
book
by
nicole
forsgren
and
others
so
that
you
know
that's
a
study.
That's
been
well
done,
but
it
feels
good
right.
A
It
feels
good
to
be
able
to
control
the
situation
and
and
part
of
that
like
what
they
did
there
in
that
particular
pattern
was
they
took
hundreds
of
microservices
and
they
only
allowed
one
to
be
deployed
at
a
time
which
again
just
defeats
the
purpose
of
microservices.
You
might
as
well
just
go
back
to
the
monolith
and
there's
nothing
wrong
with
that
either.
A
A
So
then,
you
have
to
look
higher
and
if
you
look
higher
in
that
hierarchy
means
the
things
that
you
really
want
to
do
is
make
an
impact,
and
if
you
know,
if
you're
dealing
with
systems
that
don't
allow
you
to
deploy
and
people
who
won't
allow
you
to
deploy,
then
you'll
go
you'll.
Look
for
that
other
opportunity
that
will
allow
you
to
make
that
impact.
A
B
And
this
problem
isn't
just
for
engineers
right,
like
everybody,
wants
to
make
an
impact,
and
so
many
companies,
just
employees,
feel
controlled.
They
feel
restricted.
They
feel
like
there's
a
ton
of
red
tape,
I'm
sure
so
many
people
listening
to
this
can
like
nod
their
head.
We've
all
been
there.
It's
not
nobody
really
wants
to
be
controlled.
We
want
autonomy.
We
want
to
be
really
making
sure
that
our
work
is
being
implemented
and
that
our
efforts
are
being
used
right
and
so
really
thinking
about
even
like
in
an
organization
and
culture.
B
How
do
you
start
distributing
power?
How
do
you
start
distributing
responsibility
and
that's,
what's
so
cool
about
microservices
like
that's
exactly
what
it
does
and
it
makes
it
safer
and
faster
right,
and
so
many
people
are
concerned
with
safety
and
speed
as
they
should
be,
but
like
your
organization,
if
you
think
about
it,
if
it's
in
a
strict
hierarchy,
it's
not
actually
that
safe
and
it's
definitely
not
very
fast
and
you're
going
to
need
that
moving
forward.
B
No
matter
what-
and
so
you
know,
I
was
speaking
to
a
bunch
of
cios
a
few
weeks
ago,
and
I
just
asked
blankly
like
what,
if
you
know
what,
if
the
seven
of
you
all
caught
copied
right
now
like
what?
How
dependent
are
people
on
you?
Do
your
teams?
Are
your
teams
capable
of
running
without
you,
you
know,
and
they
all
just
look
like
pretty
terrified
and
they're,
like
you
know
like
we're.
B
What
kind
of
backups
do
we
have
and
so
again
like
if
you're
treating
your
software
delivery
process
like
this
use
those
same
theories
and
bring
them
to
your
organization?
How
do
you
distribute
responsibility
and
power,
and
this
is
a
great
time
to
do
it,
since
people
are
distributed
physically
now
and
think
about
what
dependencies
are
there
and
then
how
do
you
build
up
those
backup
systems
so
that
no
matter
what
happens,
you
can
actually
get
better
and
stronger,
and
you
don't
have
to
worry
so
much
about
that.
A
And
you
know
we
talk
about
things
like
metrics
and
logging
all
right,
but
kate
was
using
words
like
sensing
and
responding,
and
we
do
this
all
the
time
actually
and
and
it's
a
pretty
critical
aspect
to
dealing
with
this
complexity,
which
is
how
do
you
understand
what's
happening
inside
of
this
complex
system?
And
today
we
deal
with
that
through
three
main
ways.
A
A
So
logging
is
great
for
data
that
has
high
cardinality
right
things
that
are
very,
very
unique
things
like
user,
ids
or
transaction
ids,
and
it's
good
to
be
able
to
use
logging
when
you're
trying
to
find
that
needle
in
a
haystack,
you're
trying
to
say
okay,
this
one
transaction
continues
to
fail.
Why
does
it
continue
to
fail
right
so
you're
going
to
want
to
use
something
like
logging?
A
It's
a
little
bit
of
laziness
here.
There's
two
problems
with
that.
The
first
problem
is
that
you're
not
really
able
to
get
the
fidelity
that
you
need
on
the
aggregation
side,
because
the
aggregation
is
not
meant
for
not
meant
to
be
used
inside
of
logging.
Although
companies
like
splunk
and
elasticsearch,
allow
you
to
do
aggregation,
it's
not
as
as
I
think
it
doesn't
have
that
resolution
that
you
need,
on
the
the
other
reason
not
to
do.
That,
of
course,
is
when
you're
pushing
all
these
things
to
a
logging
system.
A
It
creates
a
lot
of
additional
costs
and
then,
of
course,
on
the
tracing
side
is
really
really
important.
Now,
when
we're
dealing
with
these
complex
systems,
because
you
want
to
be
able
to
understand
how
did
a
request
make
it
through
the
system
and
in
doing
so
you're
able
to
understand
which
network
partitions
it
went
through
or
how
did
it
go
through
all
of
the
thousands
of
microservices?
How
did
the
request
make
it
all
the
way
through
and
which
systems
are,
are
slowing
you
down
and
being
able
to
just
answer?
A
In
terms
of
how
do
you
know
what
you
are
pushing
into
production
is
good
or
not
right
you.
If
you
don't
have
these
metrics,
then
you
won't
be
able
to
know.
Like
is
the
thing
that
I
deployed
good
and
so
many
times
I
see
engineers
taking
so
then
that
becomes
actually
let
me
go
back
that
becomes
the
basis
of
software
delivery
from
software
delivery.
We
get
into
the
performance
of
engineering
teams
right
so
many
times.
A
We
I
I
see
engineers
responding
without
sensing
or
taking
whatever's
immediately
in
front
of
them
and
just
trying
to
solve
for
it
without
really
trying
to
sense-
and
I
think
one
of
the
things
kate
that
you've
taught
me
on
the
people
side,
which
is
to
stop
responding
so
immediately
so
sit
back.
Take
in
some
signals,
take
in
more
signals
than
you
feel
comfortable
with
than
acting,
and
I
think
that's
that's
kind
of
that's
really
important.
A
I
think
on
the
on
the
people
side,
so
we'll
get
into
that
here
in
a
second,
let's
go
into
why
these
bases
of
the
metrics
become
so
important.
So
you
know
going
back
back
to
google.
What
they
realized
was
that
there
was
an
operations
team
and
they
could
not
up.
They
could
not
scale
their
operations
team
to
deal
with
all
the
thousands
of
microservices
and
what
they
said
was
okay.
Well,
if
we
can't
scale
the
operations
team
to
deal
with
this
complexity.
A
Well,
the
developers
are
going
to
have
to
kind
of
do
this
on
their
own
as
well,
but
we
will
provide
them
help
with
the
help
of
sres
site
reliability.
Engineers
who
know
the
operational
side,
but
will
programmatically
help
them
solve
their
problems
and
and
what
they
ended
up
doing
was
coming
up
with
this
framework
called
sli,
slos
and
slas,
and
the
sli
and
slos
so
slis
are
service
level
indicators
right.
A
These
are
things
that
we
get
out
of
metric
systems
for
the
most
part,
things
like
air
accounts
and
cpu
and
response
time,
and
then
the
slo
are
the
thresholds
that
we're
looking
for
right
and
what
this
allowed
us
to
do
is
to
not
respond
every
time
a
deployment
fails,
but
only
respond
when
a
deployment
fails.
That
causes
us
to
break
our
slo,
and
that
is
when
we're
going
to
change
our
behavior
right,
and
I
think
this
is
really
important.
A
A
So
the
basis
of
all
this
is
is
the
metrics,
and
then
this
came
from
these
are
higher
level
performance
metrics
that
came
from
the
book
accelerate
that
is
now
being
popularized
as
kind
of
like
your
performance
metrics
again.
The
basis
of
these
are
all
the
metrics
that
we
can
get
from
either
systems
or
machines
and
our
deployment,
so
change,
failure,
percentage
and
lead
time
is
all
based
on.
How
often
can
we
deploy
and
deployment
frequency?
How
often
can
we
actually
do
deployments
based
on
the
work
that
we've
accomplished?
A
How
many
you
know
what,
if
something
were
to
fail?
How
quickly
can
we
end
up
rolling
back
using
perhaps
a
deployment
system
like
spinnaker
and
and
responding
to
these
are
important,
but
you
have
to
be
able
to
measure
them
first
and
then
we
can
now
we
can
start
talking
about
okay
once
we
have
all
this
data,
how
do
we
kind
of
respond
to
them
and
how
do
we
treat
them
in
the
cultural
world?
B
Yeah
from
a
broader
perspective,
I
mean,
I
think,
the
engineering
teams
out
there
are
very
used
to
running,
but
the
most
important
thing
really
to
change
behavior
is
to
be
aware
of
it
just
to
observe
it.
So
I
always
recommend
running
mini
retros
wherever
you
can,
so
it
can
be
the
end.
Like
the
last
five
minutes
of
any
meeting,
I
simply
ask
a
team
like
what
did
you
notice?
B
What
did
you
notice?
There's
no
judgment
in
the
question,
but
it's
really
like
hey,
let's
just
become
aware
of
our
behavior
and
the
answers
might
be
like
hey
like
I
know,
shelley
spoke
for
like
90
of
the
time
or
like
hey.
I
noticed
that
we
had
three
hours
and
really
only
needed
two
or
that
you
know
20
people
were
on
video
and
then
like
10
people
weren't.
B
So
this
is
just
to
start
getting
clear
like
what
are
we
observing
and
noticing
and
then
also
like
what
do
we
want
to
actually
design
for
so
often
times?
We
don't
even
notice
the
patterns
and
behaviors
that
we're
playing
out
and
so
using
retrospectives
everywhere
and
anywhere,
whether
it's
marketing,
whether
it's
customer
success
engineering.
It's
important
to
just
think
about
that
process,
because
then
you
can
ask
like
what
do
we
want
to
do
differently?
You
know
or
like
what
went
well,
what
do
we
want
to
improve
on
it's
that
small
iterative
process?
B
You
actually
can
start
to
change
and
transform
what
you're
doing,
but
again
most
companies
don't
stop
to
think
about
it.
A
lot
of
people
think
culture
is
really
like
sitting
on
the
the
shoulders
of
the
leaders,
maybe
the
ceo
and
like
maybe
the
hr
team
is
supposed
to
implement
it.
But
we
really
argue
that
culture
is
everyone's
responsibility
and
everyone
needs
to
be
working
on
it.
B
So
people
in
every
department
need
to
get
used
to
running
retros
and
contributing
similar
to
microservices
right
like
you're,
not
just
like
this
one
thing:
it's
multiple
people
contributing
which
actually
increases
the
complexity
as
well
as
the
beauty
and
so
you're
wanting
to
build
a
culture
that
actually
technically
culture
already
exists.
Really,
if
it's
the
space
between
people,
how
do
you
honor
that?
And
how
do
you
really
ask
people
to
show
up
and
contribute?
So
I
think
using
observation
is
really
critical,
because
it's
not
supposed
to
be
about
judgment.
B
A
B
Also,
I
think
it
clears
up
assumptions
and
stories
so
even
in
particular,
when
we
were
working
with
this
very
large
customer
and
they
had
their
zero
percent
downtime
and
the
the
engineer
running
that
was
extremely
proud.
You
know
they
had
done
something
for
the
company.
I
was
very
much
needed,
but
when
sitting
around
with
other
people
and
asking
like,
how
does
everyone
feel
about
this,
his
assumption
we
could
tell
by
how
he
was
responding
was
like,
of
course,
everyone's
really
excited
and
stoked.
But
then
other
engineers
started
to
speak
up
and
they
weren't
exact.
B
They
weren't
happy
about
it,
and
the
story
to
them
wasn't
actually
necessarily
a
positive
one,
because
it
was
deeply
limiting
their
work
and
actually
making
their
lives
hell.
So
it
was
fascinating
to
kind
of
unearth
this
assumption
in
this
story,
where
it
was
just
assumed
right
that,
like
zero
percent,
downtime
was
what
everybody
wanted,
and
it
wasn't
necessarily
the
case.
So
I
think
that's
also
the
power
of
that
process.
A
Yeah
yeah,
it's
definitely,
I
think,
a
harder
lesson
to
learn
as
a
leader,
because
you
just
you
just
innately,
want
to
act.
You
inactively
want
to
help
people
immediately
and
do
the
work
immediately,
but
I've
you
know
I
myself
have
shot
myself
in
the
foot
too
many
times
by
just
acting
quickly
before
just
sitting
back
and
responding
and
gathering
all
the
information
for
longer
periods
of
time
than
you,
you
would
feel
comfortable
for,
and
you
know,
I
think,
that's
something
that
we
as
an
engineering
industry.
A
Our
software
industry,
can
be
a
lot
better
at
you
know
just
sitting
back
and
sensing
and
gathering
all
the
data,
and
in
fact,
actually
we
see
this.
I
see
this
all
the
time,
even
when
we
deal
with
outages
right
when
we
deal
with
outages.
What's
the
first
thing
you
want
to
do
you
want
to
get
the
system
back
up?
Well,
how
do
you
get
the
system
back
up?
Well,
you
tinker
with
it.
A
You
you
poke
and
prod
it
and
you
keep
you
keep
trying
and
iterating
and
just
like
get
the
system
up,
get
the
system
up,
which
is
not
the
right
thing
to
do.
The
right
thing
to
do
is
to
sit
back
and
get,
and
it's
and
it's
it
feels
so
uncomfortable
because
you're
just
like
I
want
to
fix
the
system,
I
want
to
fix
it,
but
it
feels
so
uncomfortable
to
sit
back
and
gather
gather
the
data.
B
I
love
I
love
that
isaac
says
this
because
I
always
say
with
a
client
right,
like
I'm,
really
comfortable
with
your
discomfort,
like
the
amount
of
discomfort,
that's
actually
required
in
these
transformational
processes.
It's
like
you
have
to
start
moving
slowly,
so
you
can
actually
really
build
a
strong
foundation,
just
move
really
quickly
and
there's
so
much
discomfort.
I
love
that
isaac
brings
this
up
because
every
leader
is
like.
Oh,
my
god,
oh
like
this,
this
is
terrible
right.
People
want
to
respond
out
of
fear
and
they're,
often
acting
because
they
are
uncomfortable.
B
They
just
want
to
get
out
of
discomfort
instead
of
like
holding
that
breath
for
a
second
asking,
more
questions.
Seeing
what's
going
on
right,
I
think
it's
so
critical.
So
for
all
of
you,
even
like
listening
think
of
those
times
that
you're
making
decisions
and
ask
yourself
simply
am
I
making
this
just
because
I'm
uncomfortable
like
what
is
the
speed?
What
is
the
urgency
behind
this?
Are
people
gonna
die,
or
are
people
just
irritated
right
now,
and
I
want
out
of
this
space
because
that's
a
really
important
difference
and
oftentimes.
It's
very
overlooked.
A
Yeah
and
it's
hard
it's
hard
because
you
know
especially
you
know
if
you're
dealing
with
you
know
a
vp.
That
is
one
of
two
things
either.
You
know
yelling
at
you,
because
the
system
is
down
or
yelling
at
you,
because
the
feature
didn't
didn't
go
out
the
door
right.
You
know
your
first
reaction
is
like
I
need
to.
I
need
to
push
this
out
the
door.
Actually
I
you
know
what's
really
interesting.
This
is
a
pattern
that
I've
seen
in
software
delivery
where
they
will
do
a
canary.
A
It's
like
okay,
this
this
doesn't
seem
right.
This
is
wrong,
but
continue
to
push
because
of
those
pressures
that
the
rest
of
the
business
has
on
them
right
and
that's
really
interesting.
There's
no
there's
no
real
system
that
we
can
build
to
overcome
that
that
pressure
that
you
get
from
from
the
business
that
that
will
help
alleviate
this.
A
It's
the
it's,
the
emotional
response
and
the
stress
and
the
fear
and
and
and
the
distress
that
these
engineers
go
through,
because
they
maybe
have
missed
a
date
or
there's
an
outage
that
they
have
to
deal
with
right.
Being
woken
up
at
three
o'clock
in
the
morning
on
a
saturday
is
not
a
fun
thing
to
do.
B
A
Yeah
absolutely
so
this
is
one
I
felt
like
this
word
is
the
word
that
really
kind
of
at
least
opened
my
eyes
to
like
we're,
actually
talk.
You
and
I
are
talking
about
the
same
thing,
because
you
were
talking
about
resiliency
and
I
was
talking
about
resiliency,
but
when
I
was
talking
about
resiliency,
I
was
talking
more
about
chaos,
engineering
right.
But
if
you
look
at
the
foundations
of
chaos
engineering,
what
are
we
trying
to
accomplish?
It
turned
out.
A
It
was
actually
the
same
thing,
kate
that
you
were
trying
to
accomplish
when
you're
dealing
with
a
complex
system
of
thousands
of
different
systems,
microservices
communicating
with
each
other
trying
to
predict
failure
is
impossible
or
trying
to
predict
failure
before
you're
in
production
is
impossible.
There
are
just
way
too
many
edge
cases,
and
you
won't
really
know
how
your
system
will
respond
to
a
system
to
a
system
that
you
communicate,
which
is
three
hops
away
from
you,
it's
just
impossible,
and
so
what
netflix
and
amazon
and
others
did
a
while
back?
They
realized.
A
But
what
you
figured
out
at
that
point
is
that
your
system
is
not
able
to
deal
with
the
data
center
failure
and
with
and
I've
seen
this
happen
before,
where
s3
failed.
There
was
an
epic
failure
back
in,
I
don't
know,
maybe
like
2012
or
something
like
that,
and
it
took
out
most
of
the
internet,
because
most
of
the
internet
was
dependent
on
aws
s3
and
nobody
had
ever
tested
a
multi-region.
A
Their
multi-region
strategy
right,
whereas
netflix's
and
others
are
testing
this
every
single
day.
Shutting
down
data
centers
all
the
time
and
then
figuring
out
and
then
just
just
observing,
going
back
into
sensing.
What
failed
right?
You
can't!
You
can't
get
to
a
level
this
level
of
complexity
with
microservices
without
taking
the
prior
step
of
sensing
and
responding
and
then
understanding
what
happens
when
you
remove
a
dependency
right.
What
happens
when
this
system
that
you
talk
to
goes
away
and
one
of
those
things
that
we've
heard
from
netflix?
A
A
That's
all
based
on
a
machine
learning
algorithm
that
optimizes
the
videos
for
you.
What
happens
when
that
system
is
down?
Actually
it
reverts
back
to
a
static
feed,
so
you,
as
a
viewer,
you'll
notice
that
there's
probably
some
differences,
but
you
won't
have
this
like
complete
failure
that
you
won't
be
able
to
watch
a
movie
right
or
a
show
that
you
want
to
watch
you'll,
be
able
to
kind
of
see
that,
because
they've
tested
this
system,
what
happens
when
this
machine
learning
system
goes
down?
A
Oh,
this
fails
right
and
there's
a
bunch
of
other
scenarios
that
you
have
to
test
for
so
network
latency.
What
happens
when
cpu
spikes?
How
does
your
system
deal
with
the
auto
scaling
events
going
up
and
down
on
number
of
knowns?
How
do
you
reroute
traffic
right?
So
they
inject
these
failures,
and
if
we
get
into
kind
of
this
chaos
engineering
loop,
the
idea
is:
go
from
a
steady
state,
make
a
hypothesis
run
an
experiment,
verify
and
then
improve
it.
So
a
hypothesis
again
would
be
something
like
when
this
system
is
down.
A
A
B
Yeah,
the
like
the
experiments,
I
always
ask
people
to
do
it's
usually
leadership
teams
and
I'll
just
say
stop
by
stopping
right
like
what
can
you
stop
doing
so?
I've
been
at
a
customer
or
a
client
before,
and
they
have
a
three
hour
meeting
with
the
entire
executive
team,
which
costs
like
over
half
a
million
dollars
and
when
I
would
listen
to
the
meeting
and
be
like,
is
anybody
really
getting
a
lot
out
of
this
and
watching
how
people
are
responding
and
then
thinking
about
how
much
it
actually
costs?
B
You
know
the
experiment
was
like
hey:
let's
stop
this
meeting
for
a
week,
let's
see
what
happens
right
like
what
rerouting
has
to
happen
with
your
team
below
you,
what
did
they
do,
and
so
we
did
that
for
maybe
like
three
or
four
weeks
and
they
were
like.
Oh,
the
executive
summaries
got
shorter
and
we
were
able
to
distribute
information
more
quickly
and
my
team
actually
knew
what
was
going
on
because
they
weren't
waiting
for
me
to
like
get
everything
from
this
one
meeting
it
turned
out.
B
That
meeting
was
like
not
important
and
didn't
need
it,
but
like
it
seemed
like
a
dependency.
Another
way
I
look
at
dependencies
is
also
like
leadership
teams
and
managers
like
right
now.
What
I
see
is
the
bottleneck
and
a
huge
problem
is
that
leaders
are
decision,
makers
and
they're
executors,
but
again,
if
you're,
the
only
one
doing
it
like.
B
If
you
leave
your
job
tomorrow
or
something
happens
to
you,
what's
going
to
happen
right
and
a
lot
of
power
is
held
at
the
highest
parts
of
an
organization,
that's
actually
really
unhealthy,
because
something
inevitably
does
happen,
and
it's
like
is
everything
halted?
How
many
times
have
you
heard
like?
Oh,
we
have
to
wait
for
so-and-so
to
come
back
from
like
a
hawaii
or
this
vacation
or
something
happen
that
did
take.
You
know
a
family
member
or
like
a
life
event
which
is
supernatural
but
like
when
things
halt
in
progress.
B
Because
of
one
person,
you
have
a
very
unhealthy
dependency
and
I
see
that
most
pop
up
with
leaders,
and
so
I
help
people
understand
like
how
do
you
empower
other
people?
It's
one
of
the
biggest
things
is:
I
teach
leaders
and
directors
like
how
do
you
coach
people
so
that
they
have
the
ability
to
think
for
themselves
and
make
decisions
without
you,
so
if
you're
able
to
step
away,
they
should
be
able
to
function
completely
and
perfectly
with
the
accountability
and
decision
making
rights
without
you
being
there.
A
How
will
the
organization
respond
right
and
taking
this
back
into
software
delivery
right?
If
isaac
is
your
release,
manager
and
isaac
gets
hit
by
a
bus
tomorrow?
Will
the
company
be
able
to
release
software
right
and
the
only
way
that
you
can
actually
test
this
out
is
to
remove
isaac
from
the
release
management
position
and
see
what
happens
right?
Have
isaac
go
away
for
three
weeks?
Can
your
company,
you
know
back
to
sensing?
Don't
respond
yet
just
sense,
just
observe
what
happens
when
isaac
goes
away
right?
A
Will
we
be
able
to
release-
and
this
happened
to
me
at
my
last
company-
we,
you
know
an
engineer
had
left
and
then
a
week
had
passed
and
then
we
realized
that
we
couldn't
deploy
the
main
website.
That
literally
happened
to
us
right,
and
that
is
a
sick.
You
know
right,
so
that
was
really
bad.
We
had
to
bring
back
that
engineer
to
help
us
kind
of
redeploy
the
website,
but
that
is
a
crit.
A
You
know
that
meat
that
showed
that
we
were
not
resilient
to
to
one
of
the
most
simplest
activities
and
I
think,
there's
probably
a
lot
of
companies
that
fall
under
this
I've
I
mean.
I
know
there
are
I've
spoken
to
them.
Where
I
go
well,
tell
me
about
your
release
process.
Tell
me
how
things
get
out
the
door
and
get
completed
now
and
inevitably
those
there.
You
know
60
70
percent
of
the
time.
A
There's
this
there's
this
role
called
a
release
manager
and
this
person
is
doing
a
bunch
of
manual
activity
to
get
things
out
the
door,
and
if
that
person-
and
I
asked
him-
I
said
what
happens
if
this
person
isn't
there?
Oh
we'll
wait
for
that
person
to
come
back
right.
Well,
you
know,
okay,
you
know
what
happens
when
that
person's
quit.
We'll
transfer
the
knowledge,
then.
C
A
B
C
A
B
A
You
know
so
so
fear
and
risk
at
the
at
the
core
of
everything
that
we
do
is
this
right.
All
we
are
trying
to
do
is,
I
think,
avoid
risk
right,
avoid
downtime,
avoid
failure,
because,
because
with
that
there's,
like
fear
of
repercussions,
what
are
the
fears
that
your
your
company
has
created
based
on
their
culture
and
that
and
as
we
talked
about
earlier,
that
dictates
how
your
software
delivery
pipeline
looks.
A
It
dictates
how
often
you
deploy,
because
if
there's
a
you
know,
if
I'm
going
to
get
fired
for
deploying
something
that
fails
well,
then
I'm
probably
not
going
to
deploy
things
right,
because
I
don't
want
to
be
fired.
I
don't
want
to
deal
with
the
repercussions
of
that
and
at
the
core
of
everything
that
we
do.
Is
it
comes
down
to
this
right
and
and
reducing
risk
and
reducing
that
fear,
and
if
we
can
do
that,
I
think
we'll
make
a
greater
kind
of
work.
A
Environments
we'll
be
more
innovative,
we'll
break
create
better
software
systems,
and
it
comes
down
to
this,
which
is,
if
you
want
to
be
successful,
you
are
going
to
fail
a
bunch.
You
want
to
fail
a
bunch
because
it
means
that
you're,
innovating
and
trying
things,
but
the
environment
has
to
be
open
for
failure.
What
do
you?
What
do
you
want
to
add
to
this
kate.
B
I
think
like
recognizing,
for
people
just
to
stop
and
think
about
like
what
are
you
actually
scared
of
like
verbalize
it,
because
once
you
do,
I
think
you'll
actually
realize
how
non-threatening
it
actually
is
or
how
people
can
actually
have
a
conversation
around
it.
But
you
want
to
find
the
space
that
you
can
play.
B
And
I
think
it's
just
important
to
keep
thinking
about
when
you're
doing
software
delivery,
when
you're
doing
digital
transformation
using
those
same
principles
and
theories
and
bringing
it
back
to
your
people
and
your
organization.
Because,
again,
you
want
to
be
building
resiliency
everywhere
and
anywhere
and
like
really
making
people
more
engaged
in
their
jobs
again
and
at
the
end
of
the
day.
That's
what
we
really
care
about,
because
it's
an
amazing
thing
to
watch
and
see.
A
Yeah,
and-
and
the
other
thing
too
is,
I
think,
is
like
lean
into
those
fears,
if
you're
a
leader
lean
into
those
fears,
if
your
team
is
afraid
to
deploy,
have
them
deploy
10
times
a
day.
No
changes
like
a
no
op
change
right,
but
have
them
push
the
deploy
button.
Have
everybody
push
the
deploy
button
right,
get
comfortable
with
the
things
that
you
are
afraid
to
do
right.
A
Yeah
yeah
right,
maybe
the
more
you
do
it
you,
the
more
you
know
getting
back
into
this
loop
right,
the
more
you
do
it,
the
more
you
will
understand
the
more
that
you'd
be
able
to
improve.
So
what
people
have
done
with
deployments
is:
oh,
my
god,
they're
scary,
let's
not
deploy
where
there's
a
there's,
a
a
bunch
of
companies.
That
said,
oh,
my
god,
it's
scary,
let's,
let's
deploy
10
times
more,
so
we
don't
aren't
scared
of
doing
it
right.
A
The
more
you
do,
something
the
less
scared
you're
going
to
be
the
more
you're
going
to
understand
how
to
do
it,
and
and
and
even
even
with
that
kind
of
sensing
and
learning
how
to
fail.
It,
you
know,
is
also
like
an
experience.
You
know
going
through
that
of
like
that.
That
meeting
that
you
were
talking
about
kate
is
like
it
felt
very
uncomfortable
to
go
like.