►
From YouTube: Trusting Mission Critical Infrastructure: Building Reliable CD P... Fernando Freire & Isaac Mosquera
Description
For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/
Trusting Mission Critical Infrastructure: Building Reliable CD Platforms is About More than Building Features - Fernando Freire, Armory & Isaac Mosquera, AWS
Critical infrastructure products like Spinnaker form the basis of many technology strategies, increasing the velocity of innovation while reducing the risk of outages. Failure of these systems erodes trust, limiting a team’s ability to delegate responsibility. When deploys fail due to operational constraints or bugs in the platform, it becomes difficult for teams to reliably deploy, encouraging alternative solutions that detract from delivering business value. Much like software infrastructure, failure in more traditional domains is no less worrisome. For example, without trust in a system of laws, it’s difficult to facilitate basic interactions between parties. In this session, we’ll explore the relationship between critical infrastructure across domains and propose a set of strategies for building resilient CD platforms that prioritize trust over building features.
A
Well,
thank
you
all
for
coming
today.
I
hope
you
all
got
in
okay
yesterday
and
you
all
flew
safely.
I
know
a
lot
of
us
were
delayed,
so
today
we're
going
to
talk
about
something
that
we're
all
very
familiar
with,
but
we
do
struggle
to
define
it.
We
struggle
to
encourage
it.
We
struggle
to
just
like
understand
what
it
really
means,
and
yet
it's
a
part
of
our
every
single
day
lives,
whether
it
be
our
relationships
with
other
human
beings
or
our
relationships
with
systems.
So
my
name
is
isaac
mascara.
A
We
have
about
30
years
of
platform,
development
experience,
and
the
one
thing
that
we
have
learned
in
these
years
is
that
the
features
themselves
that
make
up
the
platform
are
somewhat
irrelevant,
because
what's
more
important
than
those
features
is
actually
the
trust
that
we
have
with
the
teams
that
use
those
features.
Are
the
teams
themselves,
the
devops
teams,
the
leadership
and
how
we
view
the
people
around
us.
Ultimately,
that
is
what
drives
those
features
and,
depending
on
how
we
have
that
trust.
Those
features
manifest
differently
and
the
users
end
up
using
the
platform
differently.
B
But
it's
the
combination
of
all
three
of
these
things
that
are
going
to
make
you
more
effective
in
the
way
that
you
roll
out
your
cd
strategy
in
your
organizations.
So
trust
is
that
that
bedrock
that
we
need
to
rely
on
in
order
to
make
more
complex
interactions
right,
have
more
complex
experiences
with
our
users.
B
B
B
You
need
to
be
able
to
demonstrate
that
competency
on
some
kind
of
written
tests
and
an
actual
driving
test
with
a
dmv,
and
then
you
need
to
be
able
to
yeah
demonstrate
that
in
in
person
and
and
we
we
do
that
in
in
software
as
well
right,
it's
not
just
the
the
dmv
right.
It's
not
just
these,
these
real
world
things
if
you're
building
a
continuous
delivery
platform,
whether
it's
spinnaker,
whether
it's
jenkins,
whether
it's
any
you
know,
custom
internal
tooling.
B
A
So
the
second
part
is
empathy.
So
we'll
talk
a
little
bit
about
empathy,
it's
a
little
bit
trickier
than
competency
and
reliability
right,
because
we
don't
talk
about
that
very
much
as
engineers
or
folks
in
I.t
talk
a
lot
more
about
the
first
and
third
one,
but
the
second
one
is
actually
pretty
critically
important
because
it
manifests
itself
in
those
features
that
we
were
talking
about
earlier,
and
so
empathy
is
the
ability
to
care
about
somebody
else's
situation,
and
so
so
this,
I
think,
we're
all
familiar
with
the
at
here.
A
Who
uses
slack
first
hands:
do
we
all
use
slack?
Okay,
so
we're
all
familiar
with
this
one
right,
so
this
is
showing
a
lack
of
empathy.
What
this
shows
is
that
I
care
about
my
needs
being
met
way
before
you
and
that
I
don't
even
care
about
what
you're
doing
right.
I
need
information
right
now,
right,
here's
another
one
that
is
outside
of
the
it
world,
but
I
really
like
this
example,
because
the
first
example
was
a
negative
example.
Here's
a
positive
example
of
empathy.
Now
this
isn't
a
trick
that
says:
don't
buy
this
jacket.
A
This
is
a
patagonia
ad
that
ran
some
time
way
back.
I
don't
know
when,
and
this
ran
before
black
friday
and
they
they
meant
it
they,
they
don't
want
you
to
buy
that
jacket.
In
fact,
they
want
you
to
stay
home
and
they
don't
want
you
to
buy
things
on
black
friday
because
they
want.
They
believe
that
reusing
recycling
will
leave
earth
a
inhabitable
earth
for
our
children
right.
So
they
have
empathy
for
people
who
aren't
even
born
yet
now
it's
not
all
altruistic,
of
course,
right.
A
B
The
last
pillar
that
we're
going
to
be
focusing
on
today
is
reliability.
It's
our
ability
to
do
something
consistently
over
time,
so
think
about
when
you're
using
your
deployment
systems.
Is
it
deploying
successfully
five
out
of
ten
times
seven
out
of
ten
times
or
nine
out
of
ten
times,
or
is
it
doing
it
ten
out
of
ten
times?
A
So
one
of
the
things
that
we
observed
is
that
we
build
systems
of
enforcement
instead
of
enablement,
and
so
we
know
what
these
systems
are
like,
who
has
ever
heard
in
it
of
a
change
control
board
like
the
ccrv
right.
Those
are
not
systems
of
enablement.
Those
are
systems
of
enforcement.
There
are
rules
that
we
must
enforce
onto
people
and
I
think
we
could
all
agree
that
no
one's
really
happy
to
go
through
that
that
process
right,
but
there's
other
processes
that
are
good
again,
showing
a
positive
example
right.
A
A
Now,
if
you
look
at
the
systems
of
enforcement
right
when
there's
lack
of
trust,
I
don't
trust
you
to
do
the
things
that
you
need
to
do.
I
don't
trust
you
to
take
care
of
security
or
architecture,
and
you
build
things
like
a
ccrb.
What
ends
up
happening
is
that
you
start
incentivizing
the
wrong
behavior
start
batching
up
deployments
right
star.
Maybe
we
don't
break
things
into
monoliths.
B
B
So
you
start
to
minimize
the
probability
of
failure
when
the
equation
is
really
one
of
two
things
right:
it's
the
probability
of
failure
and
the
impact
of
that
failure.
So
if
you
have
your
website
going
down,
you
know,
let's
use
an
example.
Well
we're
not
going
to
use
this
example.
We're
going
to
talk
about
a
separate
example.
Let's
say:
you're
running
a
cloud
service
that
has
three
nines
of
availability
and
maybe
one
time
out
of
every
three
years.
It
goes
down
and
everybody's
relying
on
that
service.
B
If
that
service
goes
down,
the
impact
of
failure
is
going
to
be
pretty
large,
even
though
the
probability
of
that
failure
was
pretty
small.
But
if
you
spend
all
of
your
engineering
effort
trying
to
minimize
that
probability,
the
impact
is
still
going
to
make
it
really
painful.
So
we'll
talk
later,
when
we
start
talking
about
strategies
for
fostering
trust,
we'll
talk
about
how
you
can
minimize
impact,
rather
than
minimizing
the
probability
of
that
failure.
A
So
the
other
thing
that
happens
when
we,
when
we
don't
have
trust
as
we
left,
we
suffer
from
low
engagement
right.
So
how
many
times
have
you
all
heard
answered
a
survey
about
your
deployment
strategy
or
your
cd
platform?
The
devops
team
sends
one
out:
have
you
ever
sent
one
out
or
have
ever
received
a
survey
from
a
centralized
platform
by
show
hands
yeah
so
and
how
many
times
as
a
result
of
that
survey,
has
nothing
changed
like
have
you?
Do
you
think
that
things
change
when,
when
you
get
this
survey?
A
No,
they
typically
don't,
and
that
is
the
gap,
the
trust
gap
that
we
have
right.
This
happens
all
the
time.
I
work
with
a
lot
of
organizations
where
they
send
out
those
surveys,
and
then
these
developers
and
development
teams
feel
unheard
right
and
then
they
have
low
engagement
and
you
and
user
retention,
meaning
they're,
not
wanting
to
use
the
platform
right
and
when
you
take
the
power
away
from
them.
B
And
so
a
consequence
of
that
is
leading
to
introducing
overhead
right
if
you
have
low
engagement,
you're
going
to
start
to
build
systems
that
gate
access
to
resources,
you're
going
to
start
making
sure
that
you
prevent
bad
behavior
from
happening
rather
than
working
with
your
teams
to
empower
the
kind
of
behavior
that
you
want
to
see
so
think
about
a
time
when
you've
wanted
to
explore
a
new
piece
of
infrastructure,
a
new
tool.
Right.
Let's
say
you
wanted
to
adopt.
B
A
But,
no
matter
how
hard
you
try,
people
will
find
a
way
nature
finds
a
way.
People
will
find
a
way
around
all
of
the
constraints,
the
enforcements,
the
lack
of
trust,
everything
you
build
right
if
people
want
to
get
their
job
done
and
need
to
get
their
job
done
and
your
tool
is
in
the
in
their
way
they'll
go
around
your
tool
and
guess
what
they're
going
to
build
their
own
systems,
and
now
you
end
up
with
three
new
problems
right.
The
first
is
in
order
to
get
them
off
of
their
system.
A
You
have
to
understand
what
they
built
in
the
first
place,
so
now
you're
spending
time
learning
what
they
built.
The
second
time
is
you
have
to
debate
the
merits
of
what
they
built.
Maybe
they
actually
built
something
better
than
you
right.
Let's
assume
that
they
didn't,
though,
for
this
this
case,
and
the
third
is
now
you
have
to
bring
them
back
into
your
platform
right,
so,
no
matter
how
hard
you
want
to
constrain
and
and
reduce
other
people
to
your
system
right,
it's
just
creating
a
lot
more
work
for
you.
A
B
Thank
you
very
much
yeah.
So
we've
talked
a
little
bit
about
you
know
what
trust
is
and
what
happens
when
it
starts
to
break
down.
So
let's
figure
out
ways
that
we
can
enable
trust-
and
the
first
thing
to
call
out
is
that
you
can't
build
trust.
Trust
is
given
between
two
parties,
so
the
best
that
we
can
do
is
enable
environments
and
situations
where
trust
can
happen
and
and
can
foster
over
time.
B
So
before
we
dive
into
strategies.
We
want
you
to
know
that
everything
that
we're
going
to
put
on
here
will
be
on
the
slides.
You
can
check
it
out
later.
We'll
include
a
couple
of
links
as
well
for
some
practical
examples,
so
you
can
take
this
back
to
your
teams,
but
overall,
the
the
high
level
themes
to
keep
in
mind.
Is
that
not
all
of
the
strategies
that
we're
going
to
call
out
are
going
to
work
for
your
organization.
B
You
know,
keep
in
mind
the
size
of
your
team,
the
number
of
teams
that
you're
supporting
and
what
your
risk
tolerance
is
right.
So
if
you
have
a
zero
tolerance
policy
of
of
production
going
down-
or
if
you
have
you
know
some
other
constraint-
that
you're
working
with
some
of
these
strategies
are
going
to
work
better
or
worse
and
and
the
best
advice
that
we
can
give
is
to
choose
the
smallest
combination
of
things
to
foster
trust
right,
start
small
and
build
up
over
time.
B
A
So
the
first
thing
is
being
able
to
communicate
and
establish
clear
values.
We
see
this
a
lot
where
you
know
we
have
a
value
of
being
honest
or
doing
the
best
you
can.
Those
aren't
neces,
those
are
values
for
sure,
but
they
don't
really
tell
developers
how
to
behave
right
and
most
of
the
time,
the
developers
or
even
the
devops
folks
building
those
platforms
have
no
idea
what
they're,
what
the
trade-offs
are:
they're
making
trade-offs
in
a
vacuum
without
really
understanding
the
values
of
leadership.
A
So
so,
if
you're,
in
a
leadership,
position
establishing
and
communicate,
these
values
is
really
important.
Typically,
it's
kind
of
like
even
overstatements,
so
maybe
build
versus,
buy
right
right
or
buy
over
build
or
open
source
versus
vendors
or
vendors
versus
open
source,
whatever
it
may
be,
just
make
sure
that
it's
a
very
clear
kind
of
trade-off
decision
that
you're,
making
and
you're
communicating
to
your
organization
right.
So
we
all
remember,
facebook's,
you
know,
go
fast
and
break
things
or
build
fast
and
break
things
right.
A
That
was
a
very
clear
kind
of
value
system
and
saying:
hey,
it's
actually:
okay,
if
we
lose
a
little
bit
of
reliability
as
long
as
we're
moving
faster
right,
and
so
these
all
these
kind
of
messages
help
communicate
to
your
organization
why
decisions
are
being
made,
because
I
can
tell
you
that
there's
nothing
more
frustrating
than
having
a
decision
being
made
that
you
disagree
with,
and
you
have
no
idea
why
it
was
made
that
way.
So,
the
quicker
you
can
establish
these
values
with
your
teams
and
have
buy-in
the
rest
will
come
a
lot
easier.
B
So
the
next
way
that
we
can
enable
trust
is,
by
leveraging
other
trustworthy
sources
right.
So
this
might
be
a
little
bit
counter-intuitive.
How
can
I
borrow
somebody
else's
trust,
but
it
can
work
when
implemented
in
the
right
way,
right
so
think
about
all
of
the
open
source
communities
and
open
source
tools
that
you
use
today
that
you
have
trust
in.
You
have
trust
that
those
communities
are
competent,
they're
empathetic
to
their
users,
needs
and
they're
reliable.
A
Yeah,
so
when
you
provide
structured
feedback
loops
ways
to
get
back
and
enable
your
users
to
have
a
voice
right,
it's
pretty
important
because
it
a
lot.
It
forces
you
to
sit
down
and
think
about
how
people
are
gonna,
have
that
voice
and
give
and
be
part
of
your
development
process.
So
it's
really
important
to
have
this
structure
feedback
loop
and
not
just
kind
of
like
an
open
question
of
like
well
tell
me
what
to
do
next,
but
instead
go
through
your
value
system.
Go
through.
A
Are
you
meeting
your
goals
with
your
users
and
understand
where
you're,
actually,
where
the
gaps
are?
Where
do
you
need
to
iterate
next
right
and
it
goes
both
ways
too
right,
because
on
the
other
side,
the
users
of
your
platform
also
need
to
be
able
to
be
empathetic
and
to
be
able
to
communicate
clearly.
So
it's
an
opportunity
for
them
to
communicate
clearly
with
you
through
a
structured
feedback,
loop.
B
And
then
the
culmination
of
those
a
couple
of
strategies
is
in
defining
a
clear
roadmap
right.
So
if
you
have
a
clear
value
system,
you
provided
a
structured
way
for
people
to
tell
you
about
their
needs.
Then
you
can
demonstrate
that
empathy
and
that
competency
by
communicating
that
roadmap
out
right.
So
if
you
know
that
you
have
three
sets
of
teams
that
you're
trying
to
onboard
and
they're
all
using
different
technologies,
you
need
to
find
a
way
within
that
value
system
to
communicate.
B
B
They're
able
to
move
within
that
space
within
that
value
system
and
that
kind
of
transparency
increases
the
the
reliability
that
you're
going
to
have
those
people
come
back
to
you
and
come
tell
you
their
needs
in
the
future
right,
even
if
they
may
not
like
the
answer
ahead
of
time,
then
the
next
thing
that
we
want
to
talk
about
is
minimizing
that
impact
of
failure
right.
So
if
we
think
back
to
risk
right
risk
is
that
composition
or
the
product
of
the
probability
of
failure
occurring
with
the
impact
of
that
failure?
B
So
if
our
strategy
so
far
has
been
to
minimize
the
the
probability
of
that
failure
occurring,
we
should
instead
try
and
minimize
the
the
impact
right
so
find
ways
that
you
can
work
with
your
teams
to
test
out
new
ideas
within
your
platform.
So
if
you
have
a
team
that
wants
to
explore
a
brand
new
technology,
maybe
a
function
as
a
service
platform,
you
want
to
work
with
them
to
create
a
pilot
use
case.
Have
a
hypothesis
is
this
going
to
increase
our
our
deployment
velocity?
B
Is
this
going
to
increase
our
ability
to
bring
new
product
to
market
great?
Let's
test
that
hypothesis
out
and
let's
structure
it
in
a
way
that
we
can
learn
from
it
and
then
incorporate
it
into
our
roadmap
and
that
way
we're
not
defining
a
multi-quarter
strategy
where
we're
going
to
support
functions
as
a
service
and
then
we're
going
to
roll
it
out
to
the
organization.
Then
we're
going
to
say
great,
go
ahead
and
use
it
right,
because
at
that
point
the
probability
of
failure
is
pretty
high
and
the
impact
is
also
pretty
high.
B
So
the
one,
the
the
simplest
thing
that
you
could
do
to
get
started
today
as
well,
is
to
implement
a
contributing.md
file
right.
So
if
it's
a
single
project
or
you
have
multiple
projects,
this
is
the
best
way
to
start
incorporating
your
teams
into
your
cd
strategy,
so
give
your
users
a
way
to
contribute
back,
even
if
you're,
a
small
team.
B
This
actually
works
best
when
you're
a
small
team,
because
you
can
define
exactly
the
ways
in
which
you
want
people
to
come
and
provide
input
to
the
the
platform
that
you're
building
there's
a
couple
of
very
simple
strategies.
You
can
add
this
file
to
your
repository
and
then
you
can
also
demonstrate
empathy
and
reliability
by
consistently
engaging
with
those
pull
requests
right.
So
if
you
have
people
that
are
contributing
to
your
project,
then
they're
adding
functions
as
a
service
they're,
adding
support
for
containers,
they're,
adding
support
for
you
know
the
the
latest
and
greatest
vm.
B
You
know
product
that
gets
released
great.
You
can
start
to
engage
with
them
understand
their
needs
incorporate
that
into
the
platform,
and
now
you
have
invested
users
right,
they're,
putting
skin
in
the
game
and
they're
more
likely
to
trust
the
platform,
because
they've
had
a
hand
in
building
it
as
well.
A
So
one
of
the
last
ones
here
separating
out
controls
from
best
practices.
What
we
see
in
a
lot
of
platforms
is
that
they
combine
both
right.
One
is
actually
a
need
and
one
is
actually
a
want.
Best.
Practices
are
a
desire
right
where
controls
regulations,
those
are
a
need.
We
need
to
abide
by
the
rules
and
the
laws
that
we
have
so
many
times.
We
see
platform
teams,
take
best
practices
and
combine
them
with
controls
in
order
for
them
to
get
adopted
right.
A
A
What
else
canary
right
right
when
we
can
separate
out
those
things
from
the
compliance
area
right
and
let
developers
use
the
best
tools
that
they
can
to
to
actually
perform
delivery
and
deployments
from
those
controls.
Then
we
can
have
an
honest
conversation
about.
Is
this
even
really
a
best
practice,
because
many
times
what
we
end
up
doing
is
making
everybody
in
a
large
organization
use
a
tool
that
is
the
best
practice
for
one
group,
but
is
absolutely
not
a
best
practice
for
another
group.
How
we
use
those
vms
how
we
use
those
containers?
A
How
we
use
functions
as
a
service
is
very,
very
different,
so
we
can't
just
constrain
them
all
to
the
same
thing,
even
though
the
controls
and
the
compliance
is
always
the
same
right,
compliance
rules,
don't
change
per
the
infrastructure
that
we
use.
So
we
have
to
be
very,
very
diligent
in
this
area.
Otherwise,
users
will
get
upset.
Will
actually,
you
know
have
all
of
the
impact
that
we
had
the
negative
impact
that
we
had
at
the
very
beginning
of
this
conversation.
B
So,
at
the
end
of
the
day,
right
trust
is
the
combination
of
these
three
items.
Competency,
empathy,
reliability.
If
you
haven't
heard
us
say
it
enough
you're
going
to
hear
us
say
it
probably
a
few
more
times
but
we've,
given
you
a
number
of
strategies
that
you
can
implement
with
your
teams
today
to
help
build
these
three
pillars
up
and
encourage
you
to
have
a
stronger
cd
strategy
within
your
organization.
B
B
Yeah,
so
no,
no,
no
so
yeah
use
that
example
from
open
source
and
apply
it
internally
to
your
organization.
There's
actually
a
really
neat
organization
called
the
intersource
commons,
which
is
basically
a
way
of
applying
open
source
principles.
Inside
of
your
organization,
you
don't
have
to
go
the
full
nine
yards
and
implement
like
a
full,
open
source
strategy
internally,
but
adopting
techniques
from
the
inner
source
commons
are
a
great
way
to
start
collaborating
with
more
teams
within
your
organization.
A
Yeah,
just
to
add
on
to
that,
like
the
one
thing
that
you
you'll
want
to
kind
of
google
and
is
inner
source,
there's
a
lot
of
content
out
there
for
intersourcing
with
inside
of
your
organization
or
private
repo,
and
then
why
this
is
so
important
is
because,
as
a
centralized
team,
it's
hard
to,
I
mean
if
you're
dealing
with
hundreds
of
thousands
of
developers,
you
can't
scale
to
that
size
on
your
own.
It's
just
impossible.
A
So
having
this
contributing
md
allows
you
to
create
a
process,
a
scalable
process
that
invites
everybody
in
so
you
kind
of.
Have
you
know
it's
two
things
one.
It
helps
you
get
your
job
done,
but
two.
It
helps
foster
that
communication
and
trust
in
a
way
that
you
you
couldn't
otherwise,
so
in
a
scalable
way.
That's
really
what
we're
after
here
scalability.
A
A
And
I
think
what
we're
seeing
now
more
so
than
ever
before
in
open
source
community
is,
is
that
middle
point?
We
actually
probably
should
have
moved
it
to
the
front,
because
empathy
now
is
something
that
we're
seeing
as
a
critical
part
of
open
source
communities,
but
I
think
that's
the
way
you
go
to
evaluating
now.
None
of
us
can
predict
the
future.
Absolutely
things
go
away
right
and
open
source
projects
go
away,
but
the
more
that
they
can
demonstrate
those
three
areas,
the
better
the.
A
I
believe
that
the
the
higher
likelihood
that
they'll
stick
around
and
I
and
I
see
a
lot
of
that
in
especially
in
the
linux
foundation-
a
lot
of
those
projects
there
they're
really
trying
to
foster
that
the
empathy
part
how
we
collaborate
and
work
with
each
other
on
particular
projects
is
really
really
important
and
that's,
ultimately,
what
will
make
it
survive
and
then,
of
course,
who's.
A
part
of
that
community
right,
that's
the
competency
part
are
these
people
who
have
demonstrated
you
know,
expertise
in
the
particular
area.
A
B
Oh,
that's
a
great
question.
So
we'll
repeat:
it
are
there
case
studies
where
some
of
these
techniques
have
been
used
within
organizations,
I'm
going
to
fall
back
again
to
the
inner
source
commons
project.
That
really
is
a
great
resource
for
understanding
how
organizations
have
leveraged
this
stuff
successfully
internally,
and
then
we
can
post
a
few
more
resources
with
the
slides
that
talk
a
little
bit
more
about
some
of
the
other
techniques
that
we
were
mentioning
today.