►
From YouTube: DevOps Patterns and Antipatterns for Continuous Software Updates - Kat Cosgrove, JFrog
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
DevOps Patterns and Antipatterns for Continuous Software Updates - Kat Cosgrove, JFrog
So, you want to update the software for your user, be it the nodes in your K8s cluster, a browser on user’s desktop, an app in user’s smartphone or even a user’s car. What can possibly go wrong? In this talk, we’ll analyze real-world software update failures and how multiple DevOps patterns, that fit a variety of scenarios, could have saved the developers. Manually making sure that everything works before sending update and expecting the user to do acceptance tests before they update is most definitely not on the list of such patterns. Join us for some awesome and scary continuous update horror stories and some obvious (and some not so obvious) proven ideas for improvement and best practices you can start following tomorrow.
https://sched.co/Zelk
A
Hey
y'all
thanks
for
coming
to
see
me.
This
is
devops
patterns
and
anti-patterns
for
continuous
software
updates
a
little
bit
about
me.
First,
my
name
is
kat
cosgrove
and
I'm
a
developer
advocate
at
jfrog.
Before
that
I
was
an
engineer
on
the
iot
team
here
and
if
you
want
to
get
a
hold
of
me,
I
am
on
twitter
at
dixie3flatline
or
you
can
just
shoot
me.
An
email
twitter
is
faster,
though
I
am
not
on
that
zero
inbox
technology.
A
A
Obviously
there's
some
software
that
doesn't
really
require
updates,
for
example,
bc.
It's
a
command
line,
calculator,
you
type
two
plus
two
and
you
get
four.
This
kind
of
software
is
not
updated
very
frequently,
as
you
can
see
here,
but
frankly,
I
am
not
sure
what
these
updates
were,
but
you
can
see
that
it
hasn't
been
updated
for
20
years
and
that's
probably
okay.
This
is
an
exception,
though
most
software
is
updated
and
it's
updated
frequently
for
a
lot
of
different
reasons.
A
The
first
reason
software
is
updated
is
simply
us
we're
users
and
we
want
shiny
new
features,
lots
of
them.
We
are
super
greedy.
We
want
them
frequently.
We
want
them
now.
Yesterday
is
better
honestly.
If
I
could
get
it,
you
might
relate
to
this
a
little
bit
if
you
had
a
phone
back
in
the
90s
one
of
those
nokias
with
snake.
A
A
We
didn't
get
smartphones
until
the
2000s
and
the
early
ones
were
still
really
really
clunky
about
updates
and
moving
data
from
one
phone
to
the
next.
I
remember
having
to
buy
a
data
cable
to
move
over
from
a
samsung
galaxy
to
the
pixel,
2
and
silent
app
updates
are
also
relatively
new.
This
transition
was
driven
entirely
by
users.
A
A
Think
of
it
like
stopping
distances
when
you're
learning
how
to
drive
two
components,
make
up
the
stopping
distance,
your
thinking,
distance
or
the
time
it
takes
you
to
recognize
that
there's
a
problem
and
react
to
it
by
slamming
on
the
brakes
and
the
actual
braking
distance,
which
is
the
time
it
takes
to
stop
the
car
once
you've
reacted
and
hit
the
brake
pedal
to
reduce
the
stopping
distance.
You
have
to
reduce
both
components:
better
materials,
to
make
better
brake
pads,
better
pavement
more
advanced
tires
and
to
reduce
the
thinking
distance.
That's
a
little
bit
harder.
A
It's
similar
with
security
vulnerabilities
there's
the
time
it
takes
to
identify
that
a
problem
exists,
the
time
it
takes
to
fix
it,
which
is
often
just
updating
your
dependencies
and
the
time
it
takes
to
deploy
the
fix.
This
is
where
continuous
updates
become
a
big
deal,
because
the
faster
you
can
update
the
better
off
you'll
be
a
couple
of
examples
you
might
remember
about
three
years
ago
there
was
an
exploit
that
effectively
shut
down
hospitals
in
the
uk
after
ransomware
took
over
their
networks.
A
It
was
identified
immediately,
which
is
great,
but
the
fix
was
upgrading
the
operating
systems.
Now
these
hospitals
were
still
using
windows
xp,
which
had
reached
its
end
of
life
by
then
and
was
no
longer
being
supported
with
security
updates,
so
the
ransomware
was
able
to
take
advantage
of
an
unpatched
flaw
and
completely
take
over
it,
took
literally
years
to
deploy
the
fix,
in
this
case,
upgrading
every
single
computer
in
every
hospital
from
windows
xp
to
something
a
little
bit
more
modern.
A
This
had
actually
been
present
for
months,
though
it
was
unexploited.
It
took
them
two
months
to
find
and
upgrade
all
usages
of
the
vulnerable
version
and
deploy
the
fix,
and
I
know
we're
only
talking
about
vulnerabilities
in
enterprise
environments
and
they're,
going
to
be
slower
to
update
just
like
by
their
very
nature,
and
you
can
say
well
in
our
environment.
This
isn't
the
case.
We
are
young,
we
use
all
of
the
cool
technology,
we
know
what
we're
doing.
We
know
how
to
update
immediately.
So
this
would
never
happen
to
us.
A
That's
great
and
all,
but
it's
not
really
enough.
Last
year,
spectre
and
meltdown
went
public.
People
mostly
talked
about
meltdown
because,
like
10
lines
of
javascript
could
expose
passwords
from
different
parts
of
memory
and
that's
terrifying,
but
I
would
argue
that
spectre
is
the
more
dangerous
of
the
two.
A
The
problem
with
spectre
is
that,
while
most
malware
exploits
coding
errors
or
just
generally
poor
design
spectre
exploits
good
design.
Its
attacks
work
by
observing
how
well-designed
applications
affect
the
memory
and
what
decisions
the
cpu
makes
based
on
certain
conditions
in
the
code
from
there.
They
assume
that
other
software
reasoning,
the
same
way,
will
lead
to
the
same
decisions.
A
A
We
need
to
identify
the
problem
as
fast
as
possible
and
we
need
to
deploy
the
fix
as
fast
as
possible
and
we
need
to
update
as
fast
as
possible.
That's
what
this
is.
The
talk
is
all
about
updating
as
quickly
as
we
can
on
the
bright
side,
we're
actually,
in
surprisingly
good
shape
when
it
comes
to
continuous
updates.
The
state
of
devops
report,
which
many
of
you
may
be
familiar
with-
surveys,
more
than
30
000
organizations
annually
on
their
practices,
they're
looking
to
measure
a
few
different
metrics
the
lead
time.
A
How
long
it
takes
to
go
from
code
to
production,
the
frequency
of
deployments,
what
percent
of
deployments
fail
and
need
to
be
rolled
back
and
how
long
it
takes
to
recover
from
one
of
those
failures
based
on
those
responses.
Organizations
are
grouped
into
categories
by
performance
and
only
about
20
of
them
have
what
would
be
considered
elite
performance
to
qualify
as
an
elite
organization.
You
need
to
be
able
to
deploy
multiple
times
per
day.
A
All
of
these
translate
to
dear
god.
Please
please,
please,
release
faster.
It
is
not
really
a
new
concept,
so
you
want
to
update
faster,
great
cool
there's,
no
doubt
that
it
leads
to
a
better,
more
secure
product,
but
it
can
also
go
terribly
terribly
wrong
and
here's
how
our
first
example
is
of
course,
java
three
years
ago,
mark
reinolds
announced
that
java
would
be
updating.
More
often
gone
were
the
days
where
a
java
update
takes
a
year
or
a
year
and
a
half
to
release.
Instead,
it
would
be
every
six
months
that
was
java.
A
Eight
and
five
versions
have
come
out
since
then
so
how's
that
going.
Let's
look
at
the
state
of
developer
ecosystem
report
by
jetbrains
from
2019.
they
survey
developers
and
look
for
a
variety
of
trends,
but
in
particular
we
care
about
the
results
on
which
versions
of
java
are
being
used
in
production.
A
83
percent
of
people
are
still
on
java,
8.,
five
versions
old
yikes,
so
we
just
made
a
case
for
faster
updates
java
announced
that
they
were
going
to
update
more
frequently
why
the
hell
are
so
many
people
still
on
java,
eight
to
understand
what
happened.
We
need
to
understand
how
we
actually
decide
to
update.
A
A
This
is
an
easy
question
with
things
that
don't
matter
much
like
the
netflix
app
on
your
smart
tv
or
the
twitter
app
on
your
phone.
If
there
are
risks,
we
don't
say
no
immediately.
We
ask
if
we
trust
the
update.
Now
this
one
is
a
little
bit
harder.
There
aren't
really
a
lot
of
companies
or
vendors
that
we
just
kind
of
blindly
trust,
but
in
the
early
2010s
we
did
tend
to
blindly
trust
apple,
so
I
will
use
them
as
an
example.
A
Back
then,
if
there
was
an
update
for
mac
os,
you
most
likely
just
went
for
it,
even
though
it
should
absolutely
be
considered
a
high
risk
update.
This
isn't
as
true
now
most
people,
I
know,
waited
months
to
upgrade
to
catalina
and
did
the
same
thing
with
high
sierra.
So
why
don't
we
trust
our
vendors
anymore?
A
A
The
competing
hypothesis
is
increasing
complexity.
You
can
see
here
that
the
number
of
artifacts
involved
is
growing
pretty
rapidly,
which
is
making
things
way
way
way
more
complicated,
using
agile
as
a
starting
point.
We
see
ci,
add
a
little
bit
of
complexity
and
then
cd
rolls
up
and
helps
us
out
a
lot,
but
also
adds
some
complexity.
A
Infrastructure's
code
does
the
same
micro
services
enter
the
playing
field,
docker
becomes
a
thing,
serverless
enters
the
ring
and
then
the
rising
popularity
of
iot
makes
things
way
more
complex.
There's
another
problem
at
play.
Here,
though,
it's
not
just
the
code,
it's
the
data.
We
have
a
lot
of
it.
This
is
a
prediction
of
the
global
data
sphere
by
2025..
A
Now
this
report
is
from
seagate
and
obviously
they
have
a
dog
in
that
race,
but
it's
still
today,
we
think
of
petabytes
as
a
lot
of
data
and
we're
rapidly
moving
into
exabytes
and
zettabytes
right
now,
it's
mind-boggling,
but
by
2025
we're
looking
at
175
zettabytes
of
data.
What
this
means
is
that
staging
environments
kind
of
need
to
die.
It
doesn't
make
economic
sense
anymore.
To
replicate
your
production
environment
for
staging
big
companies
are
already
testing
in
production.
A
A
We
have
another
question
in
our
decision
tree
now:
can
we
verify
that
the
update
is
going
to
be
worth
the
work
involved?
If
we
can't,
we
just
don't
update,
we
skip
that
version.
If
we
can,
we
have
to
spend
time
going
through
user
acceptance,
testing
and
verification,
and
this
is
a
problem
for
us
as
software
providers,
because
users
do
not
want
to
deal
with
all
of
this.
So
we
have
to
make
sure
that
every
update
includes
so
many
appealing
features
they'll
be
willing
to
deal
with
it.
A
A
balance
has
to
be
struck,
features
that
we
want
versus
the
cost
of
accepting
the
update.
Can
we
get
around
this
somehow?
Is
there
like
a
cheat
code?
Does
the
konami
code
work
here?
I
wish
it
did,
but
it
does
not.
We
do
have
a
cheat
code,
though
it's
auto
updates.
This
only
works
for
things
that
aren't
high
risk.
Like
your
browser
or
your
twitter
app.
I
would
wager
that
most
of
you
have
no
idea
what
version
of
chrome
you're
running
these
are
low
risk,
so
they
happen
without
notifying
you.
A
A
A
Onhub
was
a
router
from
google.
Eventually,
google
wi-fi
one
of
the
really
appealing
features
was
its
self-improving
system
today
kind
of
it's
just
an
over-the-air
update.
It
was
a
cool
concept
back
then
lots
of
slick
marketing
attached
to
it,
but
one
day
in
2017
they
released
a
bad
update.
These
updates
are
automatic.
A
So
the
problem
here
was
that
an
update
went
catastrophically
wrong
in
a
way
that
prevented
an
over-the-air
fix
from
reaching
the
device.
In
this
example,
it's
not
really
harmful.
It's
kind
of
just
a
little
bit
inconvenient,
but
if
you
think
about
it,
that
could
that
could
be
really
really
dangerous
for
some
edge
devices.
There
are
devices
out
there
in
some
very,
very
isolated
places
that
would
be
expensive,
impractical
or
maybe
outright
impossible
to
get
to.
A
A
If
you
changed
your
desktop
resolution
to
something
incompatible
with
your
monitor,
the
whole
screen
would
go
black
as
an
answer
to
this
problem
windows
would
require
you
to
confirm
that
you
want
to
keep
the
new
settings
if
you
didn't
click,
yes,
after
a
certain
amount
of
time,
it
would
assume
that
you
couldn't,
because
your
screen
is
black
and
you
can't
see
anything
and
revert
for
you.
Local
rollbacks
today
are
a
little
bit
more
complicated
than
just
remembering
a
previous
resolution
setting.
Obviously,
but
it's
the
same
idea.
A
Physical
recalls
are
very
expensive.
You
have
the
pr
cost
of
issuing
one
to
begin
with,
but
you
also
have
a
cost
in
actual
money
to
rectify
the
issue
at
hand,
even
bigger
problem,
you
can't
actually
force
the
user
to
do
it.
They
have
to
take
a
day
off
work
and
drive
the
car
into
a
service
center
and
get
a
rental,
that's
a
pain.
A
So
what
is
the
difference
between
over-the-air
updates
and
continuous
over-the-air
updates?
Let's
look
at
another
example:
one
of
the
problems
with
tesla's
autopilot
was
phantom.
Braking
the
conditions
are
good
nothing's
in
the
way
or
on
the
road
weather
is
nice,
but
out
of
the
blue,
the
car
slams
on
the
brakes,
it
sounds
like
a
serious
problem
right
and
it
took
a
while
to
fix,
but
why?
A
That's
the
fix
for
the
phantom
breaking
people
didn't
get
this
bug
fixed
quickly
because
they
waited
until
the
chess
game
was
ready
and
bundled
the
fix
in
with
that
is
the
chess
game
as
important
as
fixing
phantom
braking
kind
of
depends.
Really,
if
you
actually
have
the
autopilot
feature
and
you
use
it,
then
no,
it's
absolutely
not
fixing
the
phantom
brake
takes
priority,
but
if
you
don't
have
autopilot
and
you
just
really
like
playing
chess,
while
you
wait
for
your
car
to
charge,
then
it
probably
is
more
important
than
the
phantom
braking
fix.
A
The
problem
here
is
that,
if
you
do
batch
updates
features
that
are
important
to
one
set
of
users
have
to
wait
on
features
that
are
irrelevant
to
them,
but
important
to
a
separate
set
of
users
and
that
sucks.
Instead,
if
you
implement
continuous
updates,
this
problem
goes
away.
You
can
deliver
the
features
as
they
come
out.
A
You
might
ask
yourself
why
the
hell
is
she
talking
about
iot
and
mobile
development,
I'm
a
cloud
native
developer.
This
is
kubecon
and
we're
mostly
cloud
native
developers
here.
How
is
this
relevant?
The
thing
is
our
problems
as
cloud
native
developers
and
software
providers
kind
of
pale
in
comparison
to
the
struggles
of
iot
developers,
and
we
can
learn
from
them.
A
They
don't
control
the
availability
of
the
target
device.
They
don't
control
the
state
of
the
target.
They
may
not
know
the
version
of
the
target
they
may
not
have
access
to
the
device,
so
our
lives,
doing
server
side
updates
are
way
easier
than
the
lives
of
those
solving
problems
for
iot,
and
yet
we
still
manage
to
screw
it
up.
Like
repeatedly.
A
A
Night
capital
was
a
trading
firm
that
specialized
in
automated
trading
on
the
new
york
stock
exchange.
They
decided
to
roll
out
a
new
system
and
for
some
reason
they
decided
to
reuse
some
old
apis,
but
some
of
the
endpoints
had
changed.
Somehow
one
of
the
eight
servers
was
not
updated.
New
clients
sent
requests,
but
one
of
those
machines
contained
old
code
with
an
api
that
behaves
differently.
Engineers
decide
to
roll
back
the
working
code
from
those
seven
updated
servers
increasing
the
load
on
the
one
server
running
old
code.
A
There
was
also
no
monitoring,
no
alerts
and
no
debugging,
so
they
had
to
sit
there
trying
to
figure
out
what's
going
on
in
real
time,
losing
money
minute
over
minute
over
the
course
of
the
42
minutes.
This
was
live.
They
lost
over
400
million
dollars
and
went
out
of
business
as
a
result,
absolutely
horrifying.
A
So
what
could
have
fixed
this?
The
problem
here
is
that
humans
are
just
really
bad
at
re,
repetitive
tasks.
Those
servers
were
all
being
updated
manually,
but
humans
get
tired
or
bored
or
distracted
or
they're
depressed
or
somebody's
just
having
a
bad
day,
and
they
aren't
working
at
100.
It's
normal.
You
can't
expect
anybody
to
be
perfect
all
the
time.
A
So
the
answer
is
to
just
automate
everything:
computers
don't
get
bored,
they
also
weren't
updating,
frequently
not
deploying
regularly
makes
every
infrequent
update,
feel
more
stressful,
which
humans
don't
really
handle
well,
and
it
leads
to
mistakes.
The
answer
here
is
to
just
update
more
frequently
get
better
at
it.
It
takes
practice,
make
it
a
habit
that
way
the
chances
of
you
making
a
mistake,
because
you're
anxious
go
down
pretty
radically.
A
Another
problem
is
that
they
reused
some
state
on
the
target
machine.
Now
this
could
be
a
good
decision
or
a
bad
one.
It
kind
of
depends,
but
regardless
you
need
to
know
the
state
of
the
device
that
you're
updating,
reverting
an
update
might
require
reverting
the
state.
You
have
to
know
what
will
happen.
A
A
Cloudflare
fights,
ddos
attacks
with
routing
rules.
These
routing
rules
are
deployed
frequently
to
combat
attacks,
and
in
this
case
a
deployment
of
a
single
misconfigured
rule
caused
the
outage,
and
the
culprit
was
something
nobody
really
likes
dealing
with
regex
in
this
case
it
spiked
cpu
usage
to
100.
Their
status
page
displayed
the
affected
region
as
earth,
which
is
cute
but
kind
of
a
problem.
A
The
problem
here
was
that
they
released
a
bug
that
affected
all
users.
Instead,
they
should
have
released
to
a
small
number
of
users.
First,
that
way,
if
a
problem
occurs,
you
can
just
stop
the
release
and
either
revert
or
patch
the
affected
users.
Your
potential
blast
radius
for
a
problem
is
radically
reduced.
A
Some
problems
are
also
really
hard
to
trace
by
relying
exclusively
on
user
feedback.
By
and
large,
it
is
hard
to
get
users
to
give
you
information
that
actually
helps
you
nail
down
the
problem,
I'm
guilty
of
this.
You
are
guilty
of
this.
We
all
do
it.
Nobody
is
perfect,
don't
pretend
you
are,
and
the
solution
here
is
to
just
always
implement
tracing
monitoring
and
logging
in
your
services
and
applications.
So
you
don't
have
to
rely
on
a
user
to
find
your
problem
for
you.
A
A
Not
every
platform
supports
that,
though,
I
think
neither
apple
app
store
nor
google
play
support
rollbacks.
You
would
have
to
take
the
previous
version
in
that
case,
rename
it
as
the
next
version
and
submit
it
for
review
again.
So
we
need
an
alternative
feature.
Flags
can
help
you
out
here.
You
embed
two
versions
of
the
affected
feature
in
the
app
and
enable
or
disable
them
with
api
calls.
So
if
a
feature
update
causes
problems
for
a
set
of
users,
you
just
use
feature
flags
to
disable
it
for
that
user.
Pretty
easy.
A
Another
dramatic
example
is
moviepass
rip
in
order
to
complete
an
app
update
they
shut
down
for
several
weeks.
They
initially
didn't
even
say
how
long
they
were
going
to
be
shut
down
and
obviously
moviepass
had
other
problems.
It
arguably
wasn't
sustainable
from
the
very
beginning,
and
this
probably
was
not
their
death
knell,
but
it
certainly
didn't
help
and
it
never
really
should
have
happened.
A
We
need
zero
time
updates
if
you
shut
down
for
five
weeks
to
perform
an
update,
your
users
are
not
gonna,
wait
for
you.
Why
should
they
there's
no
reason
it's
an
unreasonable
ask?
Instead,
your
updates
should
be
small
and
they
should
be
frequent.
They
should
never
ever
ever
cause
downtime
for
the
user
just
to
put
out
an
update.
A
A
A
This
has
been
devops
patterns
and
anti-patterns
for
continuous
software
updates.
I
hope
I
was
able
to
teach
you
something
valuable
or
at
least
keep
you
entertained
for
a
while.
Again,
my
name
is
kat
cosgrove
and
thanks
for
having
me,
I
hope
you
enjoy
the
rest
of
kubecon
and
don't
hesitate
to.
Let
me
know
if
you
have
any
questions
once
again,
you
can
reach
out
to
me
on
twitter
at
dixie3flatline.