►
From YouTube: GitOps Guide to the Galaxy (E39) | GitOps Guide to SRE
Description
Want to know how SREs use GitOps? In this episode we'll discuss, becoming an SRE, use of GitOps and more. Join Hilliary and special guest Jonny Rickard, co-host of Ask an OpenShift Admi, for an open Q&A.
Learn GitOps https://red.ht/3saqlzS
What is GitOps? https://www.redhat.com/en/topics/devops/what-is-gitops?sc_cid=7013a000002qG1BAAU
Try Red Hat OpenShift: https://red.ht/3sMaW7O
Follow us:
Hilliary Lipsig https://twitter.com/Int2Caffeinate
Jonathan Rickard https://twitter.com/jrocktx1
OpenShift https://twitter.com/openshift
Subscribe to Red Hat's YouTube channel: https://www.youtube.com/redhat/?sub_confirmation=1
What is GitOps Guide to the Galaxy?
Every other Thursday at 3pm ET host Christian Hernandez and guests dig in to discuss everything in the GitOps universe, from end-to-end CICD pipelines to creating Git workflows.
About OpenShift:
Red Hat OpenShift is an open source container application platform based on the Kubernetes container orchestrator for enterprise application development and deployment.
OpenShift on redhat.com https://red.ht/3LGCIeD
OpenShift on GitHub https://github.com/openshift/
#RedHat #GitOps #Kubernetes
A
B
B
I'm
like
I
don't
even
know
how
to
top
that
truthfully.
I
might
just
like
message
him
and
be
like
hey.
Can
I
just
can
I
just
use
yours?
Can
I
just
exactly
I'm
not
as
creative
anyway?
Thank
you
folks
for
joining
today's
episode.
Is
the
get
ups
guide
to
sre
right
for
anybody
who
missed
the
big
news?
Christian
hernandez
has
left
red
hat
after
nearly
eight
years
he's
moved
on
to
head
up.
I
think
devrel
at
code
fresh,
so
now
I'm
running
this.
B
This
show
and
driving
this
train
for
better
or
for
worse,
and
so
I
thought
this
was
a
really
important
episode
topic
to
do
as
as
sort
of
my
first
photo
solo
show,
and
we
have
johnny
here
from
ask
an
openshift
admin
joining
me
and
I'll.
Let
him
introduce
himself
for
folks
who
don't
know
him
real,
quick.
B
Great
thanks,
so
thanks
willie,
so
oh
hey,
christian's
in
the
chat,
hey
christian
right,
so
I
you
know,
I've
been
around
for
a
few
episodes.
Christian
knows
me,
of
course,
very
very
well,
so
I
think
you
all
kind
of
took
it
on
his
word
that
I
you
know,
know
something
about
something,
but
today's
episode
get
ups
guide
to
sre.
B
I
thought
was
a
really
important
episode
for
folks
to
get
to
know
both
me
as
as
a
you
know,
host
more
personally
and
how,
like
my
relation
to
gitops
a
little
bit
more,
what
is
the
word?
I'm
looking
for
language
is
failing
me,
but
basically,
what
is
the
sre
relationship
with
git
ups
right,
because
it's
a
different
perspective
and
johnny
you-
and
I
were
kind
of
chatting
about
this
before
the
show
right.
B
So
as
we
go
through
today's
today's
episode,
I'm
gonna
be
talking
a
little
bit
about
those
those
types
of
things
and
johnny's
here,
basically,
so
that
I'm
not
alone
in
my
fishbowl,
so
that
I
have
somebody
to
interact
with
he's,
got
a
lot
of
questions
queued
up
to
ask
me
so
that
we
can
kind
of
keep
this
conversation
rolling.
There's
not
a
technical
demo.
Today,
I'm
not
planning
on
showing
anything
particularly
interesting.
B
A
B
What
are
we
gonna
queue
up
the
music
in
the
lyrics?
I
don't
even
know
if
I
could
sing
that
one
without
like
eating
it
christian
wants
your
fedora
shirt.
He
should
have
thought
about
that
before
he
left
so
anyway.
If
you
want,
if
you
haven't
already
make
sure
you
like
subscribe
and
share,
the
stream
is
going
to
be
changing
a
little
bit.
B
There's
still
a
couple
pre-planned
episodes
that
christian,
I
had
mapped
out
that
I
will
continue
with
and
then
from
there
as
I
kind
of
work
through
my
long-term
strategy
and
direction
for
this
stream,
we'll
start
getting
more
into
get
ups
referees
and
and
get
us
for
different
types
of
personas
right.
So
that's
those
are
some
things
that
christian
I
had
strategized
about.
B
I
believe
yeah,
I
believe,
that's
currently
available
and
that's
of
course
published
through
o'reilly,
so
that
is
that
that's
kind
of
our
our
guest
lineups
for
the
for
the
next
few
weeks
or
a
few
episodes.
Of
course,
we're
still
continuing
every
other
week
and
then
from
there
we'll
start
moving
into
a
kind
of,
I
guess
a
slightly
new
direction
which,
like
I
said
like
subscribe,
I
hope
you're
going
to
join
me
for
that,
and
I
hope
that
you'll
enjoy
it.
B
A
Yeah
yeah,
so
I
I
think
that,
like
the
whole
sre
concept
in
good
ops
is,
is
pretty
awesome
because,
like
we're
talking
about
before
the
show,
you
know
like
for
me
I'll
spin
up
a
cluster
I'll,
deploy
an
application
to
that
cluster
and
then
call
it
right
and
there's
no
real
care
and
feeding
or
anything
like
that
to
it.
So
I
think
it's
interesting
that,
like
all
right,
I
think
I'm
more
interested
in
how,
like
you
approach
the
long-term
support
of
that
application.
So
I
know
like
when
you're
on
asking
openshift
admin.
A
You
were
saying
that
there's
like
this
onboarding
stuff,
that
you
guys
do
to
bring
an
app
onto
the
managed
service
and
stuff
like
that,
but
like
what
all
is
that,
like
once
you
get
the
application
like?
What
are
you
doing
to
make
sure
that
that
thing
stays
alive?
Is
it
like?
Are
you
like
monitoring
and
then
are
you
plugging
all
that
stuff
in
through,
like
argo,
like
how's,
that
gonna
work.
B
Right,
yes,
so
the
the
the
way,
monitoring
and
learning
ties
into
get
ops
is
a
little
bit
dependent
on
which
sre
team
you're,
looking
at
and
there's
a
few
reasons
for
that
right.
So
a
lot
of
the
openshift
sre
and
the
layered
products
on
openshift
sre
group
we're
looking
after
clusters
that
actually
we
like
red
hat,
doesn't
own
these
right.
These
are
customer
owned,
openshift
clusters,
so
we're
not
spending
their
compute.
B
B
B
Yeah,
we
don't
actually
use
ansible
a
lot.
My
team
uses
ansible
and
I'll
segue
into
that
right,
quick,
so
the
way
my
team
actually
uses
ansible
is
for
chaos
engineering.
So
to
how
do
you
repeatably
train
essaries?
Well,
if
you're
me
right,
you
looked
to
operationalizing
that
getting
that
in
his
code,
you
left
to
get
ops
for
that
right.
So
right
now,
at
like
phase
one
of
this
product
project
we're
doing
something
called
combat.
B
I
talk
about
it.
Sometimes
I
I
spoke
about
this
when
I
went
on
the
dynatrace
pure
performance
podcast
a
few
weeks
ago.
You
know
they
had
come
guested
here
and
then
asked
me
to
reverse
guest
for
them.
So
I
did.
It
was
great,
so
I
say
omelet
so
when
we,
when
we're
doing
our
our
chaos
engineering
games,
we're
basically
kind
of
doing
a
capture,
the
flag
style
game
right.
We've
got
three
teams
red
blue
green
and
we
stand
up
a
copy
of
a
managed
service
and
standalone
infrastructure.
So
it's
not
it's!
B
It's
not
touching
any
potential
production
even
including,
like
standing
up
fresh
copies
of
any
interdependencies.
So
we
stand
that
up.
Red
team
goes
in
and
they
get
to
sabotage
it
and
so
the
way
that
our
red
team
players
have
actively
sabotaged
things
is
all
documented
and
then
we're
turning
those
into
ansible
playbooks
so
that
we
can
repeat
successful
red
team
scenarios
for
new
team
members
who
are
on
boarding.
B
So
then
you
can
learn
the
so
then
you
can
learn
the
kind
of
the
way
to
fix
the
problem
as
a
new
person,
without
necessarily
having
have
to
like
hope.
It
shows
up
in
production
or
hope
that
that
revisits,
a
live
like
active
player
scenario,
so
we're
basically
whatever
we
do
live
in
the
I
we
kind
of
call
them
the
ad
hoc
games.
B
B
Thank
you
christian,
and
so
we
are
not
only
that
we're
looking
to
even
take
that
a
step
further
and
turn
that
into
an
ansible
operator
to
to
not
only
go
through
and
break
things.
But
then,
when
dealing
with
operators,
which,
if
you
change
the
state
of
an
application,
the
operator
will
revert
it
back
to
the
expected
state,
you
can
actually
create
a
situation
of
of
two
operators
fighting
with
each
other,
which
is
a
real
thing
that
can
happen
right.
B
B
Every
decision
that
an
sre
makes
during
an
incident
is
going
to
be
data.
Driven
what
do
my
slos
tell
me?
I
should
be
prioritizing
in
this
service.
What
does
your
service
service
definition
tell
me
that
you
have
promised
customers
so
in
a
situation
where
real
situation
I
have
persistent
volumes
are
over
full
there's,
there's
really
no
no
way
to
get
at
them
or
back
them
up
or
whatever
it's
a
job
application
heaps
are
also.
You
know
overloading
it's
it's
nothing.
B
Nothing
is
going
very,
very
well
at
this
point
in
time
and
I'm
looking
at
okay.
What
am
I
supposed
to
prioritize
right
now?
I
have
slos
around
the
service
uptime
and
the
service
responsiveness,
both
of
which
I
have
alerts
for
because
we're
in
it
we're
in
breach
of
those
objectives,
which
means
we're
at
risk
of
breaching
our
service
level
agreements,
the
things
we've
promised
customers
and
they
pay
us
on
those
based
on
those
promises.
B
So
I
have
a
service
definition
that
explicitly
says
we're
not
responsible
for
backing
up
your
data.
You
better
do
it
yourself
right,
so
I
look
at
these
things
and
I
say:
well,
I
have
to
prioritize
based
on
data
deleting
this
data
and
getting
the
service
the
rest
of
the
service
restored
right.
So
you
do
this,
and
this
was
this
was
this
was
a
real
scenario,
not
a
real
customer.
B
So,
like
we
got
real,
that's
that's
very
lucky
that
we
caught
it
in
that
type
of
a
phase
right,
because
these
types
of
things
can
happen
with
real
customers
too.
So
we
do
this
and
then,
in
the
post-mortem
of
that
which
we
present
to
the
business
unit,
they
go,
oh
god,
what
an
oversight!
The
data
continuity
is
extremely
important
to
this
service
would
be
better
to
leave
the
other
parts
of
the
service
down
as
long
as
the
data
could
be
preserved,
right
cool
now
we
know
that
we
learned
that
lesson
through.
B
You
know
testing
our
own
software
for
having
long-lived
operational
implementations
of
our
software
internally
eating
our
own
dog
food.
So
you
know
that
was.
That
was
very
fortunate
for
us.
Another
really
common
mistake
is
sometimes
tenants
come
to
us
and
so
I'll
I'll.
Ask
them
for
things
that
they're,
maybe
not
really
thinking
about
log4j,
is
a
great
example
of
something
unplanned.
B
You
should
plan
for,
because
log4j
happened
and
when
log4j
happened,
red
hat,
shut
down
a
lot
of
things
and
some
of
those
things
were
external
facing,
and
so
when
we,
when
we
talk
to
them,
I
have
to-
and
they
say,
okay,
we
want
to
put
out
this
product,
and
these
are.
This-
is
the
component
list
for
this
product.
I
say:
okay,
I
see
a
lot
of
open
source
upstream
components.
B
Do
you
have
maintainers
on
your
team
for
those
components
so
that
you
can
respond
to
cves,
and
if
the
answer
is
no,
they
have
to
go,
get
maintainer
status
or
find
someone
at
red
hat.
Who
is
a
maintainer
who's
willing
to
commit
to
doing
something
in
case
of
a
cve,
because
we
have
to
be
able
to
respond
to
really
critical,
bugs
and
really
critical
security
vulnerabilities
in
upstream
software?
B
So
those
are.
Those
are
two
things
that
that
tenants
aren't
always
necessarily
really
thinking
about
because
kind
of,
like
you
said
they
kind
of
tend
to
come
from
a
background
of
shipping
and
forget
it
box
products,
typically
so
running
things
as
a
service
where,
where
you're
you
know
constantly
supposed
to
be
constantly
improving,
it
continuously
improving
it
even
we'll
go
with
that,
so
that
those
are
some
of
those
perspectives
that
can
be
missed,
as
teams
make
that
cultural
change
and
then
finally
I'll
actually
ask
them.
So
what
does
your
ci
cd
look
like?
B
How?
What's
your?
What's
your
upgrade
strategy
and
how
are
you
making
sure
you're
highly
available
during
upgrades
and
I've
had
the
answer
be?
Okay,
so
well,
if
sree
goes
in
and
does
such
and
such
and
such
and
such
and
it's
like
a
six
step
process,
then
that's
how
you
do
an
upgrade-
and
I
said
great
sre
is
not
your
sysadmin.
We
will
not
be
doing
your
upgrades
you're
going
to
have
to
go
automate
that
and
we
will
not
take
the
pager
over
until
you
do
so.
B
B
So
I
need
to
be
able
to
guarantee
certain
things
that
I
know
make
operating
fleet.
Basically,
like
we
said
in
the
openshift
admin
episode,
you
need
to
be
treating
your
osd
clusters
like
cattle,
because
I
am
right.
So
what
allows
me
to
operate?
My
osd
clusters
is
cattle,
one.
The
fact
that
all
of
our
services
are
operator
driven
and
two
that
we
have
we
force
certain
maturity
models
we
force.
A
So
then,
like
so
say,
if
I
have
an
application
that
I'm
wanting
to
you
know
bring
on
to
manage
service
like
I've
gone
through
the
adoption
process
or
the
onboarding
process,
and
all
that
right
and
and
the
s3
team
at
red
hat
said
all
right:
hey
I'm
good
to
go
now.
If
I
want
to
make
an
upgrade,
not
an
upgrade
business
like
a
enhancement
or
future
enhancement
or
whatever,
I
would
still
develop
that
low
and
then
go
through.
B
We
have
regular
syncs
with
our
tenants
so
between
things
like
program
calls
and
just
like
sre
eng,
qe
syncs,
where
we
we
talk
about
like
so
those
types
of
things
are
discussed
and
then
sre
basically
provides
consulting
and
guidance
during
these
synchronous,
synchronization
meetings
where
we
say
yeah,
that's
that's
not
major
enough
that
we're
gonna
care
or
yeah.
Okay,
have
you
thought
through
how
that's
going
to
change?
You
know
how
do
we
when
we
respond
to
an
alert
there
should
be
an
sop,
a
standard
operating
procedure?
B
This
is
a
a
document
typically
stored
in
git,
that
is,
it
should
be
copy
pasteable
right,
it's
called
the
3am
standard
at
3am.
When
your
pager
goes
off
for
your
service
and
I'm
the
sre
responding.
I
should
be
able
to
copy
and
paste
from
this
sop
and
solve
the
alert.
So
have
you
thought
about
how
this
is
going
to
change
the
stops?
B
Have
you
updated
this
appropriately,
and
we
also
have
this
in
feature
jiras
with
the
engineer,
tenant
engineering
teams
that
they
adopt
a
template,
and
it
basically
includes
all
of
these
checklist
items,
for
their
features,
are
the
softs
updated?
Are
the
documentation
updated,
you
know,
has
the
sre
team
been
informed?
So
typically,
you
know
they
should
be
able
to
add
features
without
having
to
like
really
go
through
a
whole
review
process.
B
But
when
there's
this
extremely
large
architectural
change
that
will
that
comes
up,
we
typically
are
involved
in
the
planning
of
that
as
well
and
again,
we
use
these
think
meetings
in
these
program
calls
to
to
coordinate
that
planning
and
coordinate
the
execution
on
these
really
major
changes,
because
they
will
change
the
way
we
operate.
The
software
and.
B
You're
not
just
surprised
of
how
that's
going
to
work
and
understand
deeply
how
that's
going
to
work,
but
we
need
to
be.
You
know
handing
that
over
across
it
we're
a
globally
distributed
team.
I'm
responsible
for
I
don't
remember:
I'd,
have
to
count
10
engineers
in
four
geographic
regions
as
a
global
technical
lead,
it's
my
job
to
make
sure
we're
all
aligned.
A
Gotcha
so
then,
so,
though,
like
once,
you
guys
take
it
on,
are
you
are
you
doing
like,
like
you
said,
you're
using
like
gitlab
ci,
and
you
know
github
actions
and
jenkins,
and
all
that,
so
I'm
just
I'm
assuming
you're
just
hooking
that
in
as
a
run
and
then
are
you
doing
like
blue
green
deployments
or
you're
carrying
it
out
like
how
are
you
using
like?
What's
the
what's
the
flow
to
get
it
out
into
production
after
that.
B
You
know
the
requirement
that
staging
should
be
very
production-like
and
there'll
be
a
burn-in
period,
so
stuff
rolls
out
to
the
production-like
staging
and
typically
it
actually
uses
the
production,
osd
openshift
dedicated
and
then
there's
a
burning
period
where
it
has
to
run
with
its
with
its
burn
in
parameters
for
24
hours
or
potentially,
more
and
without
alerting.
So
during
that,
during
that
last
kind
of
phase,
like
the
burn
in
phase
that
can
generate
real
alerts
that
will
actually
raise
sre.
So
it's.
B
It's
a
real
smoke
test
yeah,
and
so
that's
that's
how
that's
how
my
team
does
it?
The
other
sre
teams
kind
of
can
do
some
things
differently.
Red
blue
green
deployments,
absolutely
is
is
one
of
the
strategies
it
a
little
bit
depends
on
the
team
and
the
service
model
as
well,
because
the
managed
services
are
not
all
following
the
exact
same
model.
B
So
do
red
hat
sre
use
dedicated
project
managers
for
the
onboarding
process,
or
do
you
rotate
the
responsibility?
So
it's?
How
do
you
it's
a
little
bit
of
both
right?
So
there's
dedicated
program
managers
and
project
managers
that
actually
belong
to
the
tenant
engineering
teams,
and
then
we
have
project
managers
for
the
end
pro
and
program
managers
on
the
sre
side,
so
they're
interacting
together
and
then
there's
typically
still
one
person,
one
sre
who's
who's
been
given
the
responsibility
of
shepherding
the
onboarding
through.
B
So
it's
our
job
we
go
and
part
of
part
of
our
value.
Add
is
our
sre
consulting
right,
so
I
don't
just
ask
people
to
throw
slos
and
slides
over
the
wall
at
me,
my
team
will
sit
down
with
them
or
somebody
from
my
team
will
sit
down
with
them
and
help
them
think
through
what
is
a
good
slo?
What
is
a
good
sign
we'll
give
that
consulting
yeah?
What's.
B
Service
level,
objective
and
service
level
indicator
a
service
level.
Objective
is
basically
how
what
are
we?
What
do
we
want
to
make
sure
that
we
are
doing
you
know
we
want
to
make
sure
that
our
service
is
available
99.95
of
the
time.
We
want
to
make
sure
that
our
our
api
is,
you
know,
responding
at
you
know,
rates
of
better
than
200
milliseconds
for
a
response.
B
You
know
at
least
99
of
the
time
right
like
and
then
that's
that
percentage
is
over
a
certain
amount
of
time
based
on
math,
which
I'm
not
great
at
and
so,
and
a
service
little
level
indicator
is
how
you,
how
you
measure
whether
or
not
you're
meeting
your
objective.
So
if
some
sometimes
that's
one
to
one,
the
indicator
might
be
the
objective,
and
sometimes
that's
not.
A
B
Service
level
objective,
so
my
some
sre
teams
go
only
as
far
as
caring
about
service
level,
objective,
there's,
another
layer,
it's
a
service
level
agreement
and
that's
actually
the
contract
that
involves
like
real
money
with
end
customers.
Since
my
team
is
an
sre
team
for
workload
or
for
services
that
are
available
to
end
red
hat
customers,
we
also
have
to
care
about
our
sla.
So
I
have
to
be
aware
of
what
all
of
those
things
are
for
a
service
and
and
be
able
to
kind
of
keep
that
in
mind.
B
Of
that
and
and
the
red
hat
had
observatorium
stack,
which
I
don't
know
if
that
means
anything
to
anybody
outside
myself,
and
I
don't
even
think
I
have
enough
time
on
the
stream
to
explain
that,
but
basically
it's
where
we
aggregate
data
around
alerting
and
and
indicators
that
tell
us,
you
know
how
what
is
our
error
budget?
How?
How
close
are
we
meeting
our
objectives?
Are
we
at
risk
of
not
meeting
our
objectives
and
then,
of
course,
you
have
to
know
how
does
your
objective
actually
map
back
to
the
sla?
B
I
always
recommend
that
you
should
aim
to
do
better
than
what
you've
promised
right.
So,
if
my
agreement
is
I'm
providing
you
three
nines
of
uptime,
I
should
be
aiming
on
my
own
internal
side,
objective
size
for
three
and
a
half
nines
right
to
make
sure
that
by
the
time
my
pager
goes
off,
I
should
be
able
to
get
in
and
fix
it
before
the
customer
notices.
There
was
a
problem
not
because
we've
reached
our
agreements.
A
Gotcha,
so
how
much
like
not
not
the
other
sra
teams,
you
know
I
mean
I'm
sure
they're,
probably
doing
their
own
thing,
but
like
how
much
is
your
team
using
get
ops
to
to
onboard
and
maintain,
and
you
know
essentially
remove
customers,
I
guess
when
they
fall
out
of
contract
or
whatever
like?
How?
How
are
you
using
the
tools
to
do
all
that.
B
Right
so
my
my
my
team
is
actually
interested
in
an
interesting
space.
We
don't
own
any
of
the
git
ops
infrastructure,
but
we
have
to
use
it
that
the
other
teams
use
we
have
to
be.
We
have
to
understand
what
it
does
and
what
is
aware.
So
when
we
say
we
onboard
a
customer,
let's
talk
about
what
a
customer
means
for
my
team
right.
A
customer
is
both
the
red
hat
and
customer
using
the
service
and
the
tenant
engineering
team
offering
the
managed
service
right.
B
I
have
to
make
sure
that
I'm
kind
of
taking
care
of
both
both
groups
right.
They
have
very
separate
interests
and
very
different
things
that
they
need
for
me.
So
if
I'm
off-boarding
a
customer,
typically
a
red
hat
customer,
I
don't
really
know
if
they
come
and
go
truthfully
all
of
the
ingress
and
egress
of
end
customers
is
handled
through
the
openshift,
dedicated
and
add-on
or
managed
service
flows.
So
that's
that's!
That's
pretty
good!
B
I
don't
have
to
do
anything
like
that,
but
if
I
did
need
to
off-board
a
tenant,
then
yes,
there
are
some.
There
are
some
get
ops
files
that
I
have
to
go
and
adjust
and
get
and
and
get
rid
of
some
things.
And
similarly,
though,
when
we
add
a
tenant,
there's
some
get-offs,
you
know
there's
some
files
that
we
go
and
we
add
to
to
to
kind
of
create
the
definitions.
They
do
a
lot
of
it
themselves.
We
do
some
other
pieces,
then
we
talk
work
through
again.
B
I
say
that
oh,
we
have
very
specific.
Are
back
so
we
actually
define
when
we're
we're,
defining
our
access,
based
on
the
presence
of
this
managed
service
right,
because
I
can
only
access
the
fleet
that
has
the
managed
services
I
care
about,
because
restricted
are
back.
I
also
have
to
define
what
my
expected
actions
are
going
to
be.
Can
I
delete
things
in
namespaces
here
right?
Typically?
No,
so
that's
that's
what
we
do
and
then,
when
we're
when
we're,
if
we're
off-boarding,
then
we
actually
go
through.
B
We
actually
simply
just
remove
or
deprecate
all
those
all
those
access
files.
We
also
do
some
configuration
changes
as
part
of
our
management
of
our
services.
So
if
we
need
to
add,
let's
like,
for
example,
there's
some
some
parts
of
a
service
that
requires
a
persistent
volume
and
if
we
need
to
increase
the
size
of
that
persistent
volume,
we
actually
can
do
that
with
get
ops.
Similarly,
if
we
needed
we,
I
don't
think
we
would
ever
decrease
it,
but
should
we
theoretically
need
to
decrease
it?
We
could
do
the
same.
B
It's
yeah,
it's
a
lot
of
moving
parts.
I
will
say
this
to
to
manage
this
and
again
we
never
know
what
size
our
fleet
is.
So
having
all
of
these,
these
very
highly
standardized
ways,
and
then
you
know
the
rigorous
onboarding
process
to
make
sure
that
things
are
standardized.
It
kind
of
doesn't
matter
what
people
are
doing
with
their
implementation
of
that
managed
service.
I
don't
have
to
care.
I
know
the
topology
of
that
managed
service
on
the
openshift
dedicated
cluster.
I
know
the
way
the
components
interact.
B
B
You
know,
and
we
actually
do,
reviews
on
these
things
too.
So
you
know
we
missed
it
too
right.
Both
both
sides
missed.
It
just
happens.
If
these
things
happen,
we're
we're
human.
So
I
got
an
alert
on
a
weekend
for
a
component
and
when
I
go-
and
I
log
in
and
I
take
a
look
at
the
you
know-
the
end
customer's
cluster
and
I
look
at
the
component-
I'm
like
this
is
healthy.
There's,
nothing
wrong
here
and
the
alert
self
resolves
okay.
So
that
happened
all
in
around
five
minutes.
B
Typical
response
time
must
be
within
15
minutes.
So
I'm
already
like
you
know,
I've
got
I'm
like
well,
it
be
like
well
ahead
of
my
potential
risks
of
error
budget,
so
I
go
ahead
and
I'm
looking
at
general
cluster
health
at
this
point,
I'm
checking
to
make
sure
is
there
anything
weird
with
the
api
server
which
would
cause
this
component
to
show
as
unavailable?
Am
I
you
know?
Is
there
any
potential
capacity
issues
with
ncd,
and
all
of
that
is
looking
looking
good
right?
B
Okay,
so
something
I
mean
things
these
are:
these
are
liveness
probes.
They
do
go
out
and
kind
of
through
the
internet.
It
could
have
just
been
a
dns
error
right
or
a
little
blip
in
networking.
So
I'm
gonna
go
ahead
and
you
know
sign
out
of
the
cluster
at
this
point
when
the
alert
fires
again.
Okay,
so
that's
not
a
blip,
there's
something
real
where
we
don't
what
what?
What
is
this
real
thing
that
could
be
happening.
B
Not
you
know
that
they're
failing
I'm
like
oh
they're,
not
even
in
like
the
same
name
space
as
the
component,
I'm
getting
alerted
on
right.
Well,
it
turns
out
the
liveness
probes
requires
going
through
it's
an
sso
authentication
implementation,
so
it
require
it
requires
authenticating
to
that
to
finish
the
liveness
probe,
so
it
never
it
wasn't.
It
wasn't
finishing
the
loop.
It
would
occasionally
finish
the
loop
because
one
of
the
pods
was
up
and
then
down
and
then
up
and
down
the
other
pod
was
never
coming
online.
B
B
You
know
we
make
a
a
root
cause
analysis
document,
an
rca
that
says
this
is
what
happened
and
why
it
happened,
and
this
is
what
we
learned
and
what
you
know
we
found
out,
and
so
we're
continuously
improving
the
process
of
of
our,
including
improving
our
the
services
that
we
run
by
opening
rfes.
Whenever
something
happens
in
production.
A
B
So
when
the
customer
is
actually
impacted,
yes,
theoretically,
this
customer
could
have
been
impacted,
but
we
were
so.
We
were
still
well
within
our
our
budget
right
because,
like
I
said
remember,
I
always
have
at
least
15
minutes
to
respond
to
alert,
and
I
responded
the
first
time
in
under
five,
so
we
were
so
well
within
our
error
budget
that
I
was
able
to
fix
those
pods
by
rerolling
that
stateful
set.
You
know.
Basically,
I
do
have
an
I
do.
We
did
do
an
rca
if
the
customer
ever
said
hey,
we
noticed
this.
B
B
Like
the
actual,
like
mission,
critical
workloads
were
all
fine,
it
was
this
one
authentication
piece
used
for
literally
this.
One
thing
it
was
really
literally
just
used
for
the
health
check
honestly
is,
is
how
how
how
low
impact
that
was
had
their
entire
service
been
down
for
five
minutes,
I
would
have.
We
would
have
opened
a
proactive
case
with
them
and
sent
them
the
rca.
A
Nice,
so
so
you're
talking
earlier
about
your
about
creating
an
ansible
operator
for
a
part
of
your
chaos,
engineering
and
stuff
like
that.
But
like
do
you
plan
on
using
the
operator
for
more
of
your
sr
reaction
type
things,
or
is
it
really
just
going
to
be
to
you
know
essentially
arm
your
your
next,
your
next
batch
of
engineers?
It's.
B
Really
to
arm
the
next
batch
of
engineers,
oh,
we
do
have
automation.
It's
actually
get
ops,
adjacent
automation
that
that
we
can
use
to
resolve
alerts
on
production
clusters,
but
red
hat
we're
very
highly
regulated
in
how
we
access
production
workloads.
So
you
can
automate
that
and,
like
I
said,
we're
not
allowed
to
spend
customer
compute.
A
B
B
I
don't
know
that
those
boards
are
going
to
be
super
useful
to
you.
Typically,
though,
one
of
the
things
that
we
look
at
it
one
of
the
things,
a
characteristic
of
the
board
is
that
we
have
kind
of
a
holistic
view
of,
like
typically
in
slo.
Our
slos
are
measured
over
like
28
days
like
how
many
out
of
these
28
days
period
like
how.
A
B
A
A
A
Yeah
I
was,
I
was
actually
reading
about
dashboards
and
it's
like
the
one
thing
that
covanta,
like
was
just
they
kept
saying
on
the
site
and
its
best
practices
was
make
sure
that
the
dashboard
is
telling
the
story
that
you
want
to
hear
you
know.
So
I
don't
don't
have
all
this
like
extraneous
data
on
this
thing.
Right,
don't
include
everything
and
your
3
a.m
test
right,
where,
if
you
have
to
go,
look
at
it
at
3am
and
you're
like
what
am
I
looking
at
yeah,
it's
too
much
at
that
point.
So.
B
Exactly
it's
actually
better
to
have
like
logically
broken
down
dashboards,
so
smaller
dashboards.
So,
like
I
said,
the
slo
dashboard
is
a
different
dashboard.
Then
it's
like
an
overall
slo.
It's
like
a
different
dashboard
than
the
like,
there's
a
dashboard
that
can
show
me
how
many
kafka
instances
exist
right.
B
B
So
if
you've
never
done
something
like
that
before,
I
would
say,
go
get
involved
in
a
capture
the
flag
game,
you're
going
to
to
really
learn
things.
You've
never
had
to
learn
you're
going
to
have
to
look
up
things
that
you
would
have
never
necessarily
thought
to
look
up
and
so
playing
some
capture.
The
flag
games
is
a
really
great
way
to
kind
of
get
used
to
the
concept
of
responding
to
to
utter
chaos
like
something
I'm
never
going
to
expect.
That's
what
I
modeled
my
games
off
of
their
chaos.
B
A
So
dwayne
dwayne
was
on
the
asking
openshift
admin
and
yesterday
right,
and
so
he
was
he's,
he's
learning
he's
trying
to
get
he's
trying
to
get
his
feet
wet
and
trying
to
like
dive
into
a
bunch
of
things.
So
I
can
see
where
he's
going
with
this
yeah.
I
think
I
think
he
probably
tackled
it
pretty
well.
B
B
First
is
really
cool
because
we're
working
on
there
we
go
we're
working
on
open
sourcing,
trainings
for
basically
like
sre
and
and
general
operations
and
so
forth,
and
I
will
give
this
to
stephanie
to
to
put
into
the
the
chat
and
is
it
show
off
my
keyboard
no
someday.
I
will
show
off
my
rainbow
my
rainbow
led
keyboard,
but
not
today,
so
we
will
be
pushing
actually
a
an
open
source
version
of
the
chaos
engineering
game
that
we
call
affectionately
internally
called
combat.
B
That's
that's.
My
version
we'll
be
pushing
an
open
source
version
of
that
sometime,
hopefully
sometime
this
year,
maybe
next
year,
so
we're
working
on
open
source,
every
open
sourcing.
What
we
consider
our
best
practices
at
red,
hat
and
so
operate
first
is
a
good
like
keep
an
eye
on
it.
You'll
see
them
on
twitter
too.
You
should
definitely
follow
them
on
twitter
and
the
folks.
The
folks
at
red
hat,
who
are
running
operate
first
are
super
fantastic.
A
B
Yeah
there's
something
called
sre
university
that
I've
been
working
on
for
like
the
last
year,
which
is
like
an
open
source
like
sre
training
and
a
self-assessment
like
what
you
know.
What
do
I
need
to
learn
in
order
to
round
myself
out
to
be
an
sre?
Those
are
some
efforts
that
I'm
involved
in
that
will
eventually
be
showing
up
on
the
operate
first
cloud.
A
B
B
And-
and
one
of
those
is
because,
if
we're
doing
this
an
ansible,
then
we
can
make
an
operator
and
makes
our
lives
a
little
bit
better.
It
also
forces
us
to
have
better
domain
knowledge
on
the
types
of
technologies
we
support
because
learning
it
up
learning
operators
is
actually
very
interesting.
I
mean
they're
very
similar
to
like
a
standard
like
linux.
B
Linux
demon
like
if
you're
familiar
with
that
concept
and
operator
makes
a
lot
of
sense,
but
you
know
the
way
it
watches
things,
how
many
things
it
watches,
how
operator-based
workloads
scale?
What
makes
for
a
good
operator.
You
know
what
is
the
purpose
of
operators,
which
is
to
pretend
to
be
human,
getting
getting
a
little
bit
closer
to
that
through
our
implementation
of
chaos,
engineering.
It
was
a
little
bit
important
because
of
the
way
it
forces
the
sre
team
to
again
sort
of
eat
our
own
dog
food.
A
Yeah
yeah,
it's
I
think
conceptually
operators
have
always
made
sense
to
me
but
like
when
I,
when
I
jumped
in
like
the
the
sdk
trying
to
write
one
I
was
like.
I
should
not
have
started
with
go
so
I
was
trying
to
like
back
out
and
you
know
go,
do
something
different
like
home
seems
like
it's
a
little
bit
easier
and
obviously
ansible
is
a
lot
easier
since
I'm
familiar
with
it.
So.
B
Yeah,
no,
and
so
so
truthfully,
there's
the
the
you
can
start
with
home
charts
and
then
do
a
helm
operator
some
automation
there.
We
recently
tried
it
on
my
own
team
and
actually
didn't
work
perfectly,
so
you
still
have
to
do
some.
Some
copy
paste
efforts
at
the
point
you're
doing
those
kind
of
copy
paste
efforts.
The
operator
sdk
is
ansible
under
the
hood
right.
That's
all
it
is
it's.
A
Yeah
and
dan
mester
like,
if
you
guys
don't
know
who
dan
master
is
he's,
I
think,
he's
a
product
manager
for
quay,
but
he
used
to
be
part
of
the
operator
sdk.
I
think
that
they're,
I
think,
they're
working
on
like
a
hybrid
operator,
where
you
can,
you
can
work
on
like
one
language
and
then
move
over.
You
can
start
and
go
and
then
go
to
helm
or
start
in
helm
and
then
move
to
go
or
actually
have
a
combination
of
both
in
the
same
operator,
which
will
be
super
nice.
B
A
B
A
B
B
It's
going
up
we're
getting
we're
getting
some
some
more
folks
as
well
so,
but
we
have
a
mixture
on
our
team
and
part
of
it
is
just
the
history
of
our
team.
About
50
of
my
team
comes
from
an
ops
background
and
50
of
my
team
comes
from
a
like
an
engineering
background.
So
when
I'm
making
decisions
on
like
what
technologies
we're
going
to
use,
I'm
trying
to
make
sure
that
we're
sticking
with
technologies
that
we
can
all
contribute
to,
and
so
ansible
and
ansible
operators
for
us
is
was
a
good
decision.
B
A
Yeah
I
was
going
to
ask
that
about
that.
Is
your
team
mostly,
like
you
know
software
engineers,
or
is
it
something
else,
because
something
else
I
was
reading
was
saying
that
generally
an
sre
team
is
software
engineers,
but
then
that's
really
like
a
like
a
vanilla
statement
right
because
software
engineering
isn't
necessarily
writing
code.
It's
it's.
It's
really
solving
problems
right,
it's
it
doesn't.
B
Exactly
so
I've
I've,
so
I've
talked
to
a
lot
of
I've,
talked
to
sres
at
google.
The
way
they
do
it
over
there
is
they
typically
well
the
way
they
told
me
they
do
it.
I
mean.
Let
me
make
sure
I
represent
that
correctly.
The
way
they
told
me
they
do
it
over
there
is
that
they
have
about
an
80
20
split
80
people
with
a
software
development
background
and
20
people
with
pure
operational
background
with
maybe
some
scripting,
and
it
a
little
bit
depends
on
the
team.
B
B
There's
various
like
kind
of
ways
that
that
can
kind
of
all
all
come
down
and
a
little
bit
comes
down
to.
What's
the
team
really
working
on
what
do
they
need
to
be
doing
what,
whether
their
chief
considerations
and
that's
kind
of
what
they're
going
to
hire
for
at
the
end
of
the
day,
though
we're
all
everybody
that
hire
is
hired,
needs
to
feel
comfortable,
doing
operational
tasks
like
we
all
have
to
understand
operations.
B
B
B
A
A
A
Your
team
is
like
it
sounds
like
your
team's
pretty
mature
right
but
like
if,
if
there
was
another
team
coming
on
like
how
do
you
take
what
you
know
now
right
as
far
as
like
get
ops
and
and
using
argo
cd
and
using
ci
cd
tools
and
stuff
like
that,
like?
How
would
you
on
board
a
brand
new
sre
team
other
than
the
chaos
engineering
right
to
like
get
into
this
flow
of
all
right?
Here's
don't
ever
let
me
catch
you
on
the
cluster
hand,
jamming
something
out.
B
You
know
what
bringing
up
an
entire
team
is
a
very
different
question
than
bringing
up
individuals
and
we
act
not
myself,
but
there
is
a
team
that
that
has
been
being
brought
up
more
recently
than
mine
and
they
just
experienced
like
all
of
the
pain
points
you
were
saying
right,
so
the
first
office
again
everybody's
our
back-
is
very
restricted.
So
that's
that's
really
helpful,
because
it
makes
sure
that,
as
a
team,
we're
sticking
to
the
things
that
we're
supposed
to
stick
to
in
terms
of
our
allowed
actions.
B
If
we,
for
whatever
reason,
must
go
outside
of
our
bounds
and
break
glass,
there
is
paperwork
we
have
to
do
paper.
There
is
paperwork
involved
in
that
right
that
will
get
logged
that
will
get
flagged
by
prosec.
That
will
get
you
know
somebody
is
going
to
come
knock
on
my
door.
What
did
you
do?
We
can
see
what
you
did.
Why
did
you
do
it?
Why
was
it
important?
Why
was
it
justified?
You
have
to
give
us
all
these
things,
because
we
need
to
keep
it
in
case.
The
customers
need
it
right.
B
So
there's
just
kind
of
some
fundamental
training
that
we
put
everybody
through
of
like
this
is.
This
is
what
your
our
back
is.
This
is
why
your
our
back
is.
This
is
the
importance
of
the
security.
This
is,
you
know,
blah
blah
blah
and,
of
course,
we
all
have
done
these
extensive
background
checks.
That's
proved
that
we
know
we're
we're
likely
to
be
good
to
be
trusted.
B
It's
actually
the
way
we've
done
it
is
they
sent
somebody
who
was
incredibly
senior
over
with
the
new
manager
for
the
new
team
as
the
technical
lead
to
help
build
up
the
team
as
this
person
with
a
lot
of
experience,
a
lot
of
experience,
not
just
a
you
know,
not
just
an
sre
but
at
red
hat
as
a
whole,
and
so
he's
one
of
the
other
global
technical
leads
he's
a
peer
of
mine.
We,
you
know
talk
all
the
time
like
way
smarter
than
I
am.
B
I
swear
and
so
that
that's
basically
what
happened
is
so
the
way
red
hat
does
their
management
structure,
I
think,
is
really
key
for
how
why
this
is
kind
of
able
to
work,
at
least
in
the
sre
group.
Our
technical
managers,
like
our
engineering
managers,
aren't
actually
managing
the
tech
right,
they're,
managing
the
people,
their
career
paths,
making
sure
that
they're,
you
know
accomplishing
certain
things
they're
supposed
to
accomplish.
Keeping
on
like
the
performance
of
development,
you
know
raises
salaries
bonuses,
all
all
those
considerations.
B
What
they
are
not
making
decisions
on
are
technical
decisions
that
are
the
job
of
the
technical
lead.
So
as
my
job
as
the
technical
lead,
I'm
the
person
who
makes
I
have
the
final
say
in
the
technical
decisions
for
my
team,
I
keep
us
on
the
path
towards
the
northern
star.
I
keep
us
aligned
with
the
greater
organization
as
a
whole
and
in
all
of
that
technical
management
is
done
through
the
technical
leads
and
then
we're
informing
the
managers
of.
B
What's
going
on
we're,
you
know,
keeping
everybody
apprised
of
like
this
is
our
project's
work.
This
is
how
it's
going
to
be
work.
This
is
how
it
needs
to
be
broken
up
and
then,
ideally,
the
the
managers
and
the
technical
leads
on
like
a
regional
level
right,
I'm
so
I'm
global
and
I
have
four
geographic
regions
so
on
a
regional
level,
then
they
sit
down
and
they
figure
out
who,
on
the
team,
is
going
to
actually
do
actually
the
implementation
pieces
so
like
for
me.
B
B
A
That's
awesome
and
it's
good
to
have
pain
points
like
that
too
right,
because
you
learn
as
a
group,
you
learn
how
you
work
as
a
team
right
and
you
learn
like
okay.
This
is
what
it's
like
when
it
sucks
you
know
like.
I.
I
need
to
remember
this
feeling
so
that
way
we
don't
get
back
here
and
then
you
just
keep
growing
from
there.
You
know,
so
I
think
that
that's
it's
the
the
growing
pains
or
the
the
the
good
pains
of
of
being
new.
B
Yeah
yeah,
and
so
the
thing
is,
is
what
we?
What
we're
not
doing
is
it's
not
one
sre
team
per
service,
it's
one
sre
team
per
many
services,
so
we're
also
constantly
trying
to
calculate
what
is
our
team's
capacity
to
take
on
new
services
to
take
on
new
tooling
to
take
on
new
technical
debt,
and
that's
why
we
have
the
layered
products.
Sre
team
is
a
group
called
sres
or
sre
services,
and
so,
like
I
said
so,
there's
three
technical
leads.
B
Three
functional
teams
with,
like
sort
of,
I
guess
you'll,
say
specializations
in
terms
of
the
managed
services
that
they
run
and
one
of
the
biggest
things
that
we're
doing
is
trying
to
make
sure
when
we're
scaling
that
we're
scaling
in
a
way
that
allows
us
to
almost
step
into
each
other's
shoes.
So
where
the
way
that
we're
doing
onboarding
is
we're
trying
to
make
that
extremely
uniform
across
all
the
teams.
So
everybody's
making
some
changes,
there's
an
initiative
that
I
actually
just
published.
B
The
first
slide
deck
for
is
the
disaster
planning
office
hours.
I
think
I
mentioned
it
earlier
where
anybody
from
our
team
can
go
and
present
this
slide
deck.
It
has
a
bunch
of
rcas
they're,
not
just
allowed
to
present
their
own
teams
rcs.
They
have
to
present
from
all
of
the
teams,
to
prospective
tenants
to
help
them
think
learn,
learn
about
disasters
and
start
thinking
through
their
own
disaster
planning
and
strategies,
and
how
do
they
want
their
service
handled
during
a
problem
during
a
catastrophic
incident
where
data
could
be
lost
or
so
forth?
B
That's
another
part
where
the
the
the
the
synchronization
between
the
technical
leadership
and
the
management
is
really
important,
because
keeping
everybody
on
the
path
to
the
northern
star
in
the
same
way
is
also
a
process
thing,
which
is
a
lot
of
a
people
manager
thing
and,
of
course
that's
where
our
our
project
managers
and
program
managers
also
really
save
us
because
they
keep
us
like.
I
say
they
keep
us
honest.
A
B
Your
most
important
skill
is
your
ability
to
to
manage
your
stress
because
you're
going
to
hit
things
that
can
be
very
stressful.
As
an
sre
they're
going
to
be
impacting
life
customers,
they
might
even
have
revenue
consequences
and
you
you
can't
really
let
those
things
get
through.
You
have
to
be
able
to
to
manage
your
stress,
hit
a
zen
and
and
just
get
into
the
problem
and
solve
it
as
best
you
can.
You
know
not
without
panicking.
B
A
B
B
B
So
when
I
get
to
our
hope
night's
question
about,
we
don't
take
responsibility
because
for
backups,
that's
not
exactly
accurate,
so
there's
a
degree,
so
basically
within
the
openshift
dedicated
ecosystem
itself.
For
example,
if
you
have
one
of
the
layered
products,
has
you
know
rds
instances,
postgres
redis
and
those
have
a
backup
cycle?
B
You
know
we're
we're
going
to
make
best
effort
to
keep
your
data
safe
and
and
and
continuous
in
openshift
dedicated
itself,
but
it's
there's
and
and
a
lot
of
the
layered
products.
But
it's
sort
of
you
know
remember
that
things
can
happen.
Things
can
go
wrong.
We
can
never
promise
100,
because
we
don't
control,
aws
or
gcp,
or
you
know
anything
else
right.
We
don't
control
dns.
So,
even
if
your
data
is
there,
but
you
can't
get
it
and
you
need
it,
you
should
have
something
somewhere
else.
B
If
you
can
right
so
these
types
of
things,
data
data
loss
is
very
very
uncommon.
I
am
aware
of
some
instant
some
situations
where
the
platform
mastery
team
has
had
to
restore
clusters
from
backups,
but
those
those
backups
do
exist.
They
just
are
there
there's
the
potential
for
them
to
be
imperfect
right,
because
the
backup
is
from
the
point
of
time
of
the
backup,
not
necessarily
the
point
of
time
of
the
failure,
so
customers
should
always
use
their
best
judgment.
You
should
always
have
multiple
layers
of
redundancy.
B
That's
that's
just
really
best
best
practices
in
general
and
so
yeah.
So
again,
if
it's
a
layered
product
that
has
a
has
it
already
essences,
there's
a
backup
process
in
there.
Openshift
itself
has
backup
processes.
If
anything
fails
with
those
backup
processes,
sre
will
go
in
and
do
those
backup
processes
to
make
sure
those
backups
exist,
but
again
dns
happens
then
you're
just
you're
not
going
to
have
access
to
your
data
until
dns
is
restored,
and
these
these
things
do
happen.
B
Yeah,
I
think,
did
I
answer
that
question
satisfactorily
so
yeah
when
we
say
that
when
I
was
talking
earlier
about
the
service
that
didn't
have
data
continuity
planned.
This
was
not.
This
was
ephemeral
data
in
a
persistent
volume,
not
something
that's
actually
put
into
like
truly
persistent
storage,
in
that
it's
it's
an
rds
instance,
or
an
s3,
bucket
or
or
some
kind
of
of
database.
That
actually,
you
know,
has
the
ability
to
stick
around
a
little
bit
better.
A
So
do
you
set
up
log
forwarding,
so
it's
so!
If,
if
you
have
a
customer
that
wants
to
to
upload
their
apps
or
offload
their
logs
out
of
the
cluster
right,
do
do
you
help
them
set
that
up
or
is
that
something
that
they
would
have
to
do
on
their
own
to
set
up
the
forwarder?
So
that
way,
it
just
kind
of
dumps
everything
out.
B
B
A
B
Red
hat
sre,
yes,
exactly
so,
and
then
again
again
it
depends
a
little
bit
on
the
s3
team
right,
so
the
platform
sre
team
they
may
they
typically
try
to
be
hands
off
customer
workloads.
That's
that's
policy!
If
there's,
there's
really
something
going
wrong,
the
customer
doesn't
understand
they
may
out
of
you,
know
kind
of
their
hearts,
help
them
out
a
little
bit
there.
Typically
speaking,
though,
anything
that
the
customer
is
doing
or
installing
on
their
cluster,
that's
not
a
red
hat,
managed
service
or
part
of
the
managed,
open,
citizenship,
layer
itself.
B
A
A
B
Yeah,
I
I
know
somebody
I
mean
I
could.
I
could
maybe
maybe
convince
one
of
the
other
people
within
sres.
One
of
the
other
global
leads
used
to
be
a
helm
maintainer,
and
I
he's
like
blogged
about
helm
and
I
would
call
him
a
helm
power
user.
I
might
might
could
convince
him
to
come
on
for
an
episode
like
that.
But
we'd
have
to
have
it
at
a
different
time
because
he's
in
israel
so
yeah,
maybe
but
he's
he's
really
busy.
A
A
B
A
A
I
think
it's
actually
like
learning
home
is
what
it's
called
andrew
block
is
the
author
and
austin
dewey,
I
believe,
is
the
co-author,
or
maybe
they
both
have
individual
books
and
then
our
hope,
nine
said
something
about
like
backups
are
all
yes.
Backups
are
definitely
important,
but
testing
and
restoring
is
probably
more
important
than
the
actual
backup
itself
did.
You
want
to.
B
Chime
in
on
that
a
little
better,
I
mean
yeah,
I
don't
know
so.
Sre
does
nothing
without
a
sock
without
a
standard
operating
procedure
right
we
test
the
stops.
We
test,
we
test
every
stop
as
part
of
the
onboarding
process.
We
test
the
stops.
So
if
anything
changes
with
how
those
works
they
get
tested
again
and
we
actually
work
with
the
quality
engineering
team
to
test
our
stops
too.
That
way,
they're
making
sure
that
we're
getting
stops
that
make
sense
and
work
and,
like
I
said
we
can
copy
paste
at
3am.
B
Call
it
the
3am
standard.
I
don't
remember
if
I
originated
that
just
started
saying
it
so
either
somebody
said
it
and
I
started
saying
it,
but
now
I
say
the
3am
standard
all
the
time.
I'm
pretty
sure
my
first
manager
at
red
hat
is
the
is
the
person
who
said
three
called
it:
the
3
a.m
standard,
but
yeah,
it's
the
3
a.m,
standard!
I
don't
I'm
not
a
smart
human
at
3am.
You
don't
want
me
making
independent
decisions
at
3am.
You
you
want
to
give
me
a
stop.
B
B
B
We
did
this
really
clever
server
rotation
to
promote
the
production
and
was
swapping
dns
instead
of
actually
promoting
promoting
builds
again
so
like
I
know
that
he
knew
better
on
those
on
those
backups
like,
though
we
yeah,
he
was
both
serious.
I
mean
that's
true,
but
we
also
had
what
kind
of
it
was
a
database
backup.
We
had
a
whole
database
administrator.
B
B
But
I
never
mind
anyway.
My
mind
is
a
little
bit
blown
shame
on
you,
you
knew
better,
and
so
that
said
we
we
are
at
the
top
of
the
hour,
just
like
I'm
still
working
on
my
greeting
I'm
kind
of
still
working
on
my
exit,
my
catchphrase
right,
you
know
christian
always
said
if
it's
not
in
get
it's
always
only
a
rumor,
I'm
going
to
go
ahead
and
just
remind
you
to
choose
your
technical
debt
wisely,
because
s3
will
remind
you
that
you
need
to
pay
it.