►
From YouTube: Life Beyond the Commit - Deepak Giridharagopal, Puppet
Description
Life Beyond the Commit - Deepak Giridharagopal, Puppet
The rate of change in our apps and infrastructure has risen dramatically as we've gotten more sophisticated in how we build things. And oh how sophisticated we've become! We have infrastructure that's programmable and dynamic, that we can quickly tweak to meet our needs. And atop that, we've built elaborate applications that are distributed, containerized, and microservice-ified. CD and GitOps let us sustain this level of complexity without everything collapsing. We're now in an era where the plumbing is so advanced that the state and scale of our applications can change in the blink of an eye, faster than the speed of a git commit. Our tools must adapt, becoming much more event-oriented and responsive if we hope to keep things under control. This session will discuss self-healing systems, event-driven automation, and how we can apply CD principles to ongoing systems management.
For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/
A
Hi
and
welcome
to
life
beyond
the
commit
I've
been
wanting
to
give
this
give
a
talk
on
this
subject
for
a
while
now
and
what
better
time
than
at
cdcon
right.
So
without
further
ado,
let's
light
this
candle,
my
name
is
deepak
and
I
am
cto
at
puppet
and
puppet.
Some
of
you
might
be
familiar
with
puppet
the
open
source
tool
that
does
configuration
management,
but
puppet
incorporated
not
only
builds
products
that
do
configuration
management
but
holistically.
A
So
with
that,
let's
get
it
started,
I'm
going
to
try
something
different
with
this
talk
and
start
with
the
conclusion
and
then
spend
the
rest
of
the
talk
substantiating
the
rationale
that
lies
behind
that
conclusion,
and
the
conclusion
is
that
self-healing
infrastructure
is
inevitable
and
I
think
it's
inevitable
for
two
major
reasons
that
I'll
get
into
more
detail.
As
the
talk
goes
on.
First,
reliability
is
non-negotiable,
I'm
not
sure
reliability
ever
was
really
negotiable,
but
certainly
in
2021,
with
expectations
around
services
being
extremely
high.
I
think
reliability
is
non-negotiable.
A
Now,
more
than
ever
and
two
we
can
manage
things
at
a
much
higher
level
of
abstraction.
Now
than
probably
at
any
point
in
history,
I'll
talk
a
little
bit
more
about
those
as
the
talk
goes
on,
but
for
the
time
being,
I'd
like
us
to
all
go
back
to
december
of
2012.,
so
in
late
december
of
2012
netflix
had
a
pretty
severe
outage
that
was
caused
by
an
underlying
outage
in
aws,
and
that
outage
was
specifically
relating
to
elastic
load
balancer.
A
The
line
that
I've
highlighted
here
from
netflix's
own
blog
about
the
incident,
I
think,
summarizes
it
pretty
well
netflix
streaming-
was
impacted
on
christmas,
eve,
2012
by
problems
in
the
aws
elastic
load,
balancer
service,
that
routes
network
traffic
to
the
network
service
to
the
netflix
services
that
support
streaming,
that's
kind
of
a
bummer
and
for
those
of
us
who
are
actually
hoping
to
watch
something
on
christmas
eve
in
2012.
That
was
absolutely
a
bummer.
A
The
thing
I
like
about
this
incident
is
that
in
operation
circles
many
of
us
would
consider
netflix
the
top
of
the
heap
in
terms
of
companies
that
really
understand
how
to
do
operations
correctly,
but
even
they
are
not
immune
to
the
occasional
operational
problem
that
affects
their
uptime.
It
affects
their
reliability
right
and
to
point
out
with
more
detail
how
they
are
not
the
only
one.
A
So
this
is
not
the
kind
of
thing
that
is
isolated
to
just
people
that
don't
know
how
to
do
operations
right.
It's
something
that
can
affect
us
all,
and
this
was
just
the
start
of
it.
If
we
look
at
just
azure
as
an
example,
we
can
see
a
number
of
different
incidents
that
started
to
play
out
over
time.
A
A
A
Instead
what
they
did
was
they
started
to
look
more
deeply
at
automated
recovery
and
remediation
systems
in
an
article
I've
taken
a
screenshot
of
here.
It
goes
into
some
of
the
root
cause
analysis
around
that
initial
azure
outage
that
I
mentioned,
and
I
thought
the
highlighted
section
was
really
interesting.
It
says
some
of
the
nodes
didn't
have
node
protection
turned
on
the
monitoring
system
for
detecting
this
kind
of
problem
had
a
defect
resulting
in
no
alarms
no
escalation.
A
On
top
of
this,
a
transition
to
a
new
primary
node
triggered
a
reaction
that
led
to
an
incorrect
formatting
of
other
nodes.
So
I
think
this
is
very
interesting
because
in
the
aftermath
of
this,
I
think
that
microsoft
ended
up
with
a
number
of
both
cultural
as
well
as
technological
improvements,
many
of
which
are
listed
in
this
snippet
that
they
started
to
apply
a
greater
degree
of
rigor
around
in
order
to
make
their
systems
more
self-regulating
and
more
autonomous.
Even
in
the
face
of
failure.
A
Several
years
later,
netflix
introduced
an
entire
system
that
they
built
to
do
nothing
but
event-driven
diagnostics
and
remediation
again,
it's
a
self-healing
system
that
was
designed
to
improve
the
overall
reliability
of
everything
that
they
do,
and
I
think
this
is
a
really
important
point.
I
would
like
to
make
these
self-healing
systems
end
up.
A
Complementing
these
companies
existing
ci
cd
systems
in
the
netflix
example,
the
tool
that
they
ended
up,
building
called
winston
was
complementary
to
spinnaker,
which
is
their
ci
cd
system
right
and
I
think
to
understand
why
it's
useful
to
complement
ci
cd
systems
with
these
new
kind
of
autonomous
operation
systems.
I
think
we
need
to
talk
about
something
first,
that
I
call
the
automation,
surface
area,
every
application,
every
team's
infrastructure
every
estate.
A
They
all
have
an
automation,
surface
area,
and
by
that
I
mean
the
set
of
components
involved
in
all
of
your
development
operations
and
security
workflows.
This
surface
area
represents
all
the
processes
that
tie
absolutely
everything
together
and
keep
things
running.
In
essence,
it
represents
all
the
things
you
could
in
theory
automate.
A
So
for
many
teams,
the
operating
system
is
a
significant
part
of
their
automation.
Surface
area,
so
engineers
are
automating
workflows
that
involve
direct
os
level
interactions
across
their
entire
infrastructure.
Like
manipulating
file
content,
installing
packages,
configuring
user
accounts,
setting
up
firewall
rules
and,
more
so
in
this
world.
Configuration
management
tools
like
say
puppet
are
great
and
they
could
easily
wrangle
the
complexity
of
an
os
and
do
it
safely
and
securely
at
scale.
A
A
The
operating
system
itself
represents
a
smaller
percentage
of
the
overall
automation,
surface
area
relative
to
components
and
services
developers
interact
with
via
apis,
and
these
apis
operate
at
a
higher
level
than
the
operating
system.
So
they
present
us
with
some
really
great
abstractions
for
controlling
all
kinds
of
aspects
of
our
infrastructure
and
the
applications
that
run
on
top.
A
All
these
tasks
manually
introduces
a
ton
of
room
for
error
even
when
they
are
properly
documented,
and
we
all
know
how
often
these
processes
are
properly
documented
right.
So
between
responding
to
things
like
service
down
incidents
and
rolling
backfill
deployments
and
securing
cloud
resources,
the
the
struggle
is
real
for
many
solving.
These
problems
involves
gluing
together,
a
patchwork
of
existing
scripts
broke,
bespoke
in-house
apis
and
third-party
services.
In
order
to
actually
get
anything
done.
A
A
I
think
that,
in
order
to
get
there,
we
must
capture
what
I
call
intention
as
code
or
put
differently
when
this
thing
happens.
Here
is
what
we
must
do.
Fundamentally,
that
statement
is
a
huge,
huge
part
of
operations
as
a
field,
most
ops,
folks,
whether
you're
a
systems
administrator,
whether
you're
a
developer
that
happens
to
be
managing
production
systems
or
you're,
a
site,
reliability,
engineer
or
a
devops
engineer.
A
You're
constantly
asking
yourself
and
thinking
about
when
this
thing
happens.
Here
is
what
needs
to
happen
as
a
result,
and
I
think
if
we
could
capture
the
answers
to
those
questions,
that
is
the
intention.
That
is
what
we
are
trying
to
accomplish,
and
if
we
could
capture
that
as
code,
then
I
think
we
would
be
in
a
fundamentally
different
place
as
an
industry
with
respect
to
how
to
operate
all
of
our
systems
as
they
as
they
get
more
complex
or
operating
them
at
scale
or,
most
importantly,
operating
them
in
a
more
self-regulating
unattended
way.
A
For
those
of
you
paying
attention
that
would
have
been
the
capital
one
security
breach,
let's
see
or
when
our
intrusion
detection
system
detects
something
fishy.
On
a
compute
instance
like
say,
I
don't
know,
someone's
mining
bitcoin,
then
orchestrate
a
reconfiguration
of
the
network
to
quarantine
the
system
and
notify
the
security
team
via
jira
or
servicenow
or
whatever.
So
again,
you
know
when
this
thing
happens.
Here
is
what
we
must
do
so
this
list,
I
think,
can
go
on
and
on
and
on,
but
ultimately
it
all
comes
back
to
that
same
sort
of
pattern.
A
It's
not
about
the
tools
that
you
are
using
in
and
of
themselves.
It's
not
about
how
many
things
off
the
cncf
landscape
page
that
you're
bringing
to
bear
to
solve
your
problems.
That's
only
one
part
of
the
puzzle,
the
rest
of
it
is
how
you
tie
these
things
together
to
solve
these
big
operations,
problems
that
you
have.
A
So,
let's
put
that
phrase
up
on
the
screen
and
then
I
think
an
interesting
exercise
would
be
to
try
to
break
this
down
at
a
technical
level
to
figure
out
how
we
can
attack
this
problem
technologically,
and
I
think
we
can
start
by
remixing
a
lot
of
the
core
concepts
that
we
already
know
and
love
from
the
continuous
delivery
world
one
place.
We
can
start
with
is
with
events
aka
the
when
this
happens,
part
modern
applications
generate
all
kinds
of
events,
mostly
noise,
but
with
important
signals.
A
These
events
can
be
things
like
inputs
for
monitoring
systems,
security
scanners,
cloud
events,
pub
sub
messages,
state
changes
and
ticketing
systems,
incoming
slack
messages,
etc.
Remember
these
are
all
events
produced
by
things
that
aren't
just
super
techy
or
infrastructure
related.
It's
any
kinds
of
events
coming
from
anything,
that's
part
of
your
overall
automation,
surface
area,
so
it
shouldn't
surprise
anybody
that
things
like
ticketing
systems
or
communication
tools
show
up
there,
because
those
are
critical
parts
of
an
ops
person's
stack,
more
or
less
in
terms
of
solving
problems
right.
A
So
it's
one
thing
to
just
say:
we
need
to
have
smarter
systems
that
receive
all
these
events
right,
but
it's
another
thing
I
think
to
actually
do
it
in
practice
and
that's
because
I
think
the
implementation
of
this
idea
is
significantly
more
complicated
than
it
seems
on
the
surface.
Even
for
something
really
simple
like
web
hooks
say
that
you
want
to
have
an
event-driven
automation.
A
You
want
to
have
a
automated
response
based
on
events
that
were
coming
from
two
vendors
signals
right,
like
maybe
one
is
from
your
cloud
provider
and
another
is
from
your
monitoring
system
and,
let's
say
both
of
those
are
very
simple
events
in
that
they
are
web
hooks.
Well,
okay,
let's
walk
through
what
it
would
actually
take
in
order
to
support
that
right.
A
A
I
think
all
these
things
kind
of,
add
and
add
an
ad
to
the
layers
of
complexity
required
just
to
do
something
very
basic
like
I
just
want
to
launch
a
script.
When
I
get
a
couple
of
web
hooks
that
come
in
and
then
imagine
how
much
more
complex
it
would
be,
if
you
don't
just
want
to
launch
a
script,
and
you
want
to
do
something
incredibly
sophisticated.
A
I
think,
if
you
take
all
these
things
together,
it
creates
a
bit
of
an
unfair
burden
on
operations.
Folks,
the
difficulty
required
in
building
systems
that
are
both
robust
and
actually
help
you
automate
other
stuff.
It
can
be
hard
and
I
don't
think
it's
reasonable-
to
expect
operations
folks
to
build
all
of
these
systems
themselves.
I
think
this
is
a
place
where
the
cd
community
could
definitely
help
to
give
an
example
about
how
tools
can
help
solve
these
problems
in
relay
the
tool
that
we
just
launched.
A
We
can
automatically
ensure
that
your
container
is
launched
when
we
receive
these
web
hooks
and
we
can
automatically
route
all
the
traffic
to
your
custom
logic
as
necessary
on
demand
in
a
way
where
you
don't
have
to
worry
about
the
uptime
of
those
kinds
of
services.
So
I
think
that's
a
useful
pattern
that
probably
a
lot
of
other
cd
tools
could
think
about,
as
we
start
to
expand
our
idea
of
what
kinds
of
events
we
want
to
receive
in
order
to
enable
the
maximum
amount
of
automation
that
is
theoretically
possible
to
solve
ops
problems.
A
So
if
we
break
down
the
second
part
of
when
this
happens,
do
the
following
to
do
the
following
part.
I
think
one
way
that
we
can
address
that
technologically
is
through
workflows,
which
I
would
say
are
a
more
generalized
abstraction
over
the
concept
of
a
pipeline,
and
I
don't
need
to
explain
this
part
to
all.
I
hope,
because
pipelines
and
workflows
are
kind
of
the
cornerstone
of
the
entire
continuous
delivery
universe.
A
What
we
couple
the
triggers
to-
and
you
know
the
web
hook
example
that
I
gave
earlier
is
an
example
of
a
relay
trigger,
which
is
the
when
this
happens.
Part
and
the
workflow
that
you
see
on
the
screen
is
part
of
our
workflow
engine,
which
is
actually
built
on
tekton,
which
is
a
cd
foundation
project.
So
big
shout
out
to
all
of
the
techton
people.
A
We
want
to
do
a
couple
of
different
things,
some
of
which,
in
parallel
like
creating
tickets,
that
represent
the
nature
of
that
incident,
but
we
also
want
to
do
things
like
create
a
war
room
inside
of
slack,
invite
all
the
right
people,
especially
the
folks
that
are
on
call
and
then
ultimately
do
some
notifications
there
and
then.
Lastly,
we
want
to
take
any
forensic
information
and
augment
the
pagerduty
incident,
and
then
we
can
post
that
back
to
the
incident
response
system
for
later
analysis.
A
So
that's
just
one
example
of
an
operational
thing,
nothing
to
do
with
cd.
This
is
about
when
our
incident
response
system
detects
that
an
incident
has
occurred.
What
do
we
need
to
do,
but
things
like
for
us,
jira,
slack,
pagerduty
itself,
they're,
all
part
of
our
automation,
surface
area,
they're,
all
the
tools
that
we're
critical
parts
of
our
stack
in
order
to
actually
respond
to
problems.
So
I
think
that's
just
one
example
of
how
you
can
apply
a
workflow
to
automating
more
of
these
operational
processes
where,
in
many
cases,
it's
not
technological
at
all.
A
So
for
teams
like
that
presenting
them
with,
for
example,
a
wizard,
no
code
kind
of
experience
for
crafting
their
automation.
Suddenly
that
automation
is
off
on
an
island,
their
intention
is
not
captured
as
code,
and
it
is
therefore
not
it
doesn't
get
the
benefits
that
the
rest
of
their
infrastructure
management
tool
chain
gives
them.
If
it's
not
represented
as
code,
you
can't
peer
review
it.
You
can't
check
it
into
revision
control.
You
can't
do
a
get
blame
to
see
how
these
things
have
evolved
over
time.
A
A
A
You
know
read
like
or
spend
weeks
upon
weeks
figuring
out
how
to
code
up
their
automation
in
the
form
of
a
kubernetes
operator
or
controller,
or
something
like
that.
I
think
that
is
it's
a
bridge
too
far
right
and
I
think
it
makes
this
problem
a
little
more
challenging
to
solve,
because
I
think
it's
harder
to
make
a
complex
solution
easier
for
people
to
digest.
But
I
think
this
is
a
hugely
valuable
focus
area
for
any
of
us
in
the
cd
community
that
are
looking
at
trying
to
apply
cd
principles
to
these
other.
A
These
other
very
related
problem
areas,
and
I
think
a
big
part
of
that
is
like
how
do
we
put
these
capabilities
in
the
hands
of
the
absolute
largest
number
of
people
possible
that
are
part
of
that
target
audience.
So,
personally,
I'm
not
super
interested
anymore
in
building
tools
for
the
most
elite
of
the
elite.
A
The
one
percent
of
the
one
percent
of
ops
folks
out
there
I'd
rather
build
tools
that
work
for
the
99
of
folks
in
it,
and
I
think
that's
something
I
would
encourage
all
of
us
to
think
a
lot
more
about
easy
extensibility
of
these
kinds
of
tools
without
requiring
advanced
programming,
I
believe,
is
a
really
huge
deal.
So
I
think
for
those
of
us
who
are
building
these
kinds
of
tools,
it's
worth
thinking
about.
A
What
is
that
unit
of
abstraction
for,
say,
building
a
step
that
you
can
put
inside
of
a
workflow
or
building
an
event
handler?
Does
that
require
that
you're,
a
javascript
programmer?
Does
that
require
that
you're
an
expert
go
programmer,
or
can
you
just
put
a
container
in
there
and
that
could
have
a
bash
script
inside
of
it?
One
of
those
is
clearly
more
democratic
quote,
unquote
in
terms
of
its
accessibility
and
the
others.
A
Cd
covers
the
beginning,
and
I
think
self-healing
systems
can
cover
the
end
and
I
think
we're
going
to
need
both
types
of
tools,
because,
ultimately
I
don't
really
care
what
tool
you
end
up
using
for
this
I
mean
you
know.
I'd
obviously,
love
you
to
use
mine,
but
it's
okay,
no
matter
which
one
you
choose,
because
the
important
thing
is
that
I
think
the
ops
community
and
the
cd
community.
A
You
know
that
we
need
to
work
together
to
get
to
a
place
where
our
systems
end
up
more
reliable
and
where
we
can
spend
less
time
getting
paged
in
the
middle
of
the
night
to
fix
stuff
and
more
time
relaxing
and
after
all,
I
think,
we've
all
earned
some
relaxation,
don't
you
so
that's
all
I've
got
so
I'm
debuck,
and
this
has
been
life
beyond
the
commit.
So
please
check
out
relay.sh
where
we
implement
many
of
the
ideas
from
this
talk
in
the
form
of
an
actual
product.