►
Description
Reducing Burnout with Continuous Remediation - Anton Weiss, Canarian
Speakers: Anton Weiss
Continuous Delivery and global spread are turning information systems into a beautiful messwork of partially degraded autonomous components. Incidents in such systems are abound and managing these incidents becomes an impossible task for mere humans. It sometimes feels like plugging holes in a cloud with your bare fingers in an attempt to stop the rain. In order to build the resilient systems of the future (and reduce the operators' burnout) we need smart continuous remediation. The talk will present some of the more interesting developments in this field across the industry.
For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/
A
A
whole
conference
about
continuous
delivery
seriously
isn't
continuously
delivering
software
in
2021
a
no-brainer
already
releasing
dozens
times
a
day,
easy
peasy
piece
of
cake
right.
But
what?
If
something
breaks?
Well,
no
problem,
we
have
all
of
them.
Observability
tools
right
logs,
metrics,
we'll
find
the
problem.
We
have
the
dashboards,
we
have
the
alerts
and
we'll
fix
it
like
one.
Two.
B
A
Could
you
sense
the
sarcasm
in
what
I
just
said?
This
is
an
important
question
by
the
way.
So,
if
you
could
identify
the
sarcasm,
please
send
a
plus
to
the
session
chat
and
if
you
couldn't,
please
send
a
minus,
I'm
pretty
sure
you
could.
We
folks
in
tech
are
very
good
at
sarcasm
and
we
like
sarcasm
understanding.
Its
subtleties
requires
sophistication,
so
it
makes
us
feel
smart
and
amused
at
the
same
time.
A
A
A
Burnout
have
you
ever
been
emotionally
burned
out
at
work?
Well,
according
to
statistics,
if
you've
been
in
this
industry
for
10
years
or
more
there's
a
70
higher
than
70
percent
chance
that
you've
been
burned
out
in
one
way
or
another,
I've
been
there.
I
was
also
one
of
those
who
burn
out
really
hard
took
me
more
than
a
year
of
therapy
to
fully
recover.
A
I
don't
want
even
to
go
there
now,
let's,
instead
talk
about
what's
causing
burnout
according
to
christina
maslock,
who
is
one
of
the
top
experts
on
this
subject
globally,
there
are
a
number
of
contributing
factors
with
the
main
contributing
factors
being
overload
too
much
work,
perceived:
lack
of
control,
lack
of
fairness,
lack
of
appropriate
reward,
toxic
or
eliminating
community
and
finally,
values
mismatch.
Now,
admittedly,
most
of
the
factors
contributing
to
burnout
are
social
factors
and
that's
of
ultimate
importance.
A
We
cannot
even
hope
to
solve
the
burnout
problem
by
technology
alone.
If
you
have
a
toxic
team
or
unfair
treatment.
Technology
won't
help
here,
but
still
this
is
a
technology
talk
and
I
do
believe
that
technology
can
help
us
with
at
least
two
of
the
factors
contributing
to
burnout,
namely
overload
and
perceived
lack
of
control.
B
A
You're
also,
probably
paying
for
some
kind
of
an
observability
tool
to
push
your
logs
and
metrics
to
and
your
maybe
have
some
dashboards
set
up.
So
you
have
something
to
stare
at
when
things
break,
and
you
know
they
will
break
because,
as
you
continue
building
the
entropy
keeps
creeping
in
the
tag.
Debt
starts
looming,
new
services
get
added
manual,
fixes,
get
applied
and
never
copied
to
code.
A
The
manifesto
of
chaos,
engineering,
the
so-called
principles
of
chaos,
engineering,
tell
us
in
non-ambiguous
terms.
Our
systems
are
too
complex
for
us
to
reason
about,
and
unpredictable
crap
will
invariably
happen
and
guess
what,
in
large
part,
this
lack
of
certainty.
This
unpredictability
is
caused
by
the
change
that
we
introduce
and
the
rate
of
uncertainty
grows
proportionally
to
the
rate
of
change,
so
continuous
deployment
causes
instability
in
our
systems.
A
What
else
is
new?
You
already
knew
that
right.
That's
probably
why
you
are
not
doing
continuous
deployment.
That
is
the
main
barrier.
Isn't
it
why
the
angry
face?
Did
it?
Did
it
touch
on
a
painful
subject?
Did
I
rub
salt
on
your
wounds,
and
you
know
what
the
funny
thing
is
we
keep
lying
to
ourselves
by
saying
that
we
can
lower
the
level
of
uncertainty
by
investing
more
in
rci.
A
A
For
one
thing,
if
you
haven't
been
doing
testing
for
10
years,
there's
not
a
very
high
chance
of
you
starting
doing
testing
tomorrow
with
a
high
enough
level
of
certainty
that
will
allow
you
to
deploy
to
production
right,
but
even
if
you're
saying
to
yourself
one
day,
we'll
have
enough
testing
and
will
be
able
to
deploy
to
production
with
certainty.
I
have
one
message
for
you:
that's
the
only
way
to
actually
test
things
in
production
is
well
by
testing
production.
A
A
So
if
you
do
some
kind
of
a
smoke,
automated
testing
of
your
deployments
in
production,
please
send
a
yes
to
the
session
chat
and
maybe
some
details
about
what
you're
using
for
that.
How
you're
doing
that,
and
if
you're
not
doing
that,
then
let's
start
thinking,
why?
What
are
the
barriers
right?
Okay,
let's
continue,
and
this
brings
us
to
the
topic
of
resilience.
A
You
see
if
we've
been
studying
the
subject
of
building
resilience
and
adaptivity
into
our
systems.
For
a
long
time
a
number
of
patterns
have
evolved,
but
basically
it
all
comes
down
to
the
system
being
able
to
autonomously
do
two
things:
one
identify
a
problem
and
two
deal
with
the
problem
today.
All
this
is
the
work
of
human
operators,
so
we
define
what
normal
is
and
then
we
collect
logs
and
metrics
to
help
us
make
sure
that
the
system
behaves
the
way
it
should
and
if
we're
really
knowledgeable.
A
All
this
is
tons
of
work,
of
course,
and
it's
also
usually
done
as
an
afterthought.
Unless
you
have
a
very
strong
and
experienced
and
importantly
non-overloaded
sre
team,
and
how
many
companies
today
can
boast,
they
have
that.
So
let's
say
we've
identified.
The
problem
now
comes
the
part
of
dealing
with
the
problem.
This
would
involve
analyzing
all
of
the
options
of
getting
back
to
normal
and
choosing
the
best
the
most
appropriate
option
for
the
situation
at
hand.
A
A
A
world
in
which
each
time
a
system
changes
each
time,
there's
a
new
deployment,
a
configuration
change,
an
infrastructure
upgrade.
There
is
an
automated
verification
process
that
gets
triggered
and
verifies
that
the
system
still
behaves
correctly,
and
if
there
is
a
problem,
there
is
also
an
automated
remediation
process
that
gets
kicked
off.
That
starts
iterating
on
all
of
the
possible
solutions
for
the
problem
at
hand,
and
it
can
find
and
apply
the
best
solution.
Such
a
solution
could
be
aerobic,
but
it
doesn't
have
to
be.
A
We
all
know
that
the
automated
rollback
is
seldom
the
best
option,
so
instead
it
could
be
a
reconfiguration
of
some
kind,
a
resource,
reallocation,
you
name
it
by
the
way,
I'm
not
making
this
stuff
up.
These
self-adaptivity
self-healing
properties
are
only
present
in
the
systems
we're
running
today.
A
We
can
also
see
this
in
networking
with
all
kinds
of
resilience
patterns
such
as
exponential
back
offs
fallback
servers,
bulkheading,
etc
and,
of
course,
the
reconciliation
patterns
we
find
in
kubernetes
also
have
these
self-healing
properties.
They
also
implement
this,
albeit
in
a
very
dumb,
primitive
way.
So
there's
a
replica
set
it
identifies
that
there
are
more
replicas
than
needed
or
not
enough
replicas,
and
it
reconciles
this.
A
Actually
that
and
then
we
have
the
aliveness
and
readiness
and
startup
probes
that
all
verify
the
health
of
our
pods
and
make
sure
that
the
deployment
progresses
in
a
safe
incremental
way
and
then
for
more
complex
scenarios.
There's
the
operators
pattern,
of
course,
which
adds
a
management
layer
that
automates
the
operation
of
complex
stateful
distributed
systems,
and
this
includes
such
self-adaptivity
self-healing
scenarios
as
failover
resharding.
B
A
B
A
B
A
A
Therefore,
we
need
not
only
continuous
verification,
but
also
continuous
remediation
processes
in
place,
and
we
are
already
starting
to
see
the
first
glimpses
of
this.
In
many
of
the
projects
presented
at
this
conference
in
this
foundation,
for
example,
argo
rollouts
brings
us
such
abstractions,
useful
abstractions
as
experiments
and
analysis
that
allow
us
to
verify
the
system
state
to
identify
problems
and
to
experiment
with
alternative
versions
and
configurations.
A
Captain
provides
another
interesting
approach,
called
actually
remediation,
which
allows
us
to
define
a
problem
and
then
define
a
number
of
actions
that
we
can
apply
when
a
problem
is
identified,
with
actions
being
customizable
via
action
providers
that
we
can
create
as
needed.
Now,
an
attentive
observer
will
probably
notice
that
all
this
somehow
intersects
with.
A
They
all
enable
humans
to
do
resilience,
instead
of
enabling
the
systems
to
do
resilience
in
an
autonomous
way,
and
the
key
to
that
is,
as
you
can
guess,
machine
learning
right.
The
system
has
to
be
able
to
learn
its
own
behavior,
to
learn
the
behavior
of
its
components,
to
learn
the
behavior
of
the
system
as
a
whole
to
analyze
the
semantics
and
the
impact
of
change
and
to
iterate
on
possible
solutions
to
a
problem
until
the
slo
is
achieved.
A
This
is
the
way
forward,
and
this
is
what
we're
building
at
canarian
and
if
I
got
you
curious
I'll,
be
happy
to
take
your
questions
feel
free
to
challenge
me
in
every
possible
way
and
let's
see
how
resilient
I
am,
how
well
I
can
handle
unpredictable
unexpectable
adversities
that
come
my
way,
see
you
in
the
q
a
area
bye.
So
after
finishing
the
talk
I
realized
I
didn't
introduce
myself
so
in
case
you
do
want
to
talk
to
me
and
ask
questions
here.
A
Basically,
we
do
devops
and
cloud
native,
and
I'm
also
the
host
of
the
show
called
devops
shorts,
the
link
to
which
I'll
be
providing,
and
today
I'm
also
one
of
the
co-founders
of
an
exciting
startup
called
canarian,
which
is
building
the
very
technologies
that
I
was
talking
to
you
about.
In
the
talk
see
you
in
the
q
a
area
by
this
time.
It's
really.
A
A
A
A
A
A
A
Actually
it's
not
only
us.
The
whole
industry
is
looking
into
that
direction,
and
so
I
was
recently
looking
at
what
opsamacks
who
are
sponsoring
the
conference
have
to
offer
and
they're
also
talking
about
continuous
verification,
so
they
have
that
system
called
autopilot
and
still
need
to
see
what
it
offers.
But
it's
built
upon
spinnaker
we're
not
building
a
pot
spinnaker.
A
Criteria,
because
this
is
the
the
hard
issue-
this
is
the
hard
issue
with
all
kinds
of
progressive
delivery
controllers.
Today,
whoever
tried
to
define
cannery
rolls
for
their
services
realizes
that
the
hard
part
there
is
defining
the
query
that
will
provide
good
criteria
for
your
service
health
and
we're
looking
into
improving
the
usability
of
this.
First
of
all,.