►
From YouTube: Ask an OpenShift Admin (E70) | SRE is not sysadmin
Description
Architecting, deploying, and supporting infrastructure - and the applications deployed on top - are jobs commonly associated with system administrators for the last 40+ years. As applications have become more complex and scalable, and the amount of work needed to maintain and support the infrastructure and platforms those applications are built on has increased, we've had to evolve skillsets to continue to be effective. In the last decade or so, the merging of application/developer and administrator/operations roles has often been referred to as DevOps, where we - the administrators - often become a part of the AppDev teams.
However, even more recently a new skillset, and moniker, has become popular: Site Reliability Engineering, or SRE. This role is a full merging of the two skill sets, but rather than bringing operations knowledge and skills to AppDev teams, SRE brings a software engineering approach to operations.
Today's episode we are joined by Hilliary Lipsig, SRE and co-host of the GitOps Guide to the Galaxy, to discuss the different approaches and skills for SRE compared to traditional administrators. We'll be diving into how that applies to OpenShift and especially how the SRE role applies to managing a large number of OpenShift clusters with a relatively small number of personnel.
Want to learn more?
Try Red Hat OpenShift: https://red.ht/3uVxkya
Follow us:
Andrew Sullivan https://twitter.com/practicalandrew
Jonathan Rickard https://twitter.com/jrocktx1
Hilliary Lipsig https://twitter.com/Int2Caffeinate
OpenShift https://twitter.com/openshift
Subscribe to Red Hat's YouTube channel: https://www.youtube.com/redhat/?sub_confirmation=1
About Ask an OpenShift Admin:
Do you have questions? We want to answer them! Every Wednesday at 11am ET, hosts Andrew Sullivan and Jonathan Rickard sit down to answer your questions about Red Hat OpenShift.
About OpenShift:
Red Hat OpenShift is an open source container application platform based on the Kubernetes container orchestrator for enterprise application development and deployment.
OpenShift on redhat.com https://red.ht/3rYz5ZS
OpenShift on GitHub https://github.com/openshift/
#RedHat #OpenShift #Kubernetes #ContainerPlatform
A
B
B
B
C
B
C
A
B
Yeah
nice,
it's
funny-
I
you
know
so
today's
wednesday,
so
the
holiday
here
in
the
u.s
was
two
days
ago
monday
right
and
my
son,
who
is
14,
getting
ready
to
be
in
high
school.
He
was
asking
me:
oh,
you
don't
have
to
work
and
oh
we're
getting
ready
for
summer
vacation
and
you
know.
Oh,
I
should
get
a
job
and
I'm
like
you're
14.
B
B
It's
that's
when
a
lot
of
people
are
either
like
said,
hillary
you're
on
call
in
case
something
goes
wrong
or
I
johnny
you
and
I,
when
we
worked
together
many
years
ago,
right,
there's
a
lot
of
times
where
it's.
You
know
that
week
between
christmas
and
new
year's.
That's
when
all
the
updates
and
upgrades
happen,
because
that's.
A
B
Yeah,
that's
right,
yeah,
so
yeah,
it's
a
never
dull
moment
around
here
right,
even
though
it's
early
late
spring,
not
quite
early
summer.
I
guess
I
saw
on
the
the
weatherman
this
morning
said
it
was
the
meteorological
start
to
summer,
which,
coincidentally,
is
the
same
as
the
start
of
hurricane
season
here
on
the
east
coast,
that's
right
and
the
gulf
coast.
I
don't
want
to
exclude
you
there
johnny,
but
all.
B
C
C
Yeah
it's
true,
though
I.
How
often
has
an
earthquake
destroyed
my
house
zero
times
zero
times.
Has
an
earthquake
destroyed
my
house.
This
happens
roughly
once
every
100
years,
depending
on
which
fault
line
you're
on
right,
so
technically
we're
overdue
for
a
big
one.
They
say
that
forever.
They
don't
really
know
so.
You
have
to
have
your
zombie
apocalypse.
Kit
is
actually
just
the
california
earthquake
survival
kit
right,
everybody
has
one.
If
you
don't,
you
need
to
get
one
okay,
west.
B
C
A
C
Yes,
so,
but
we
have
all
these
things
ready
to
go
right
in
case
the
sky
falls.
The
earth
crumbles.
What
have
you?
But
you
know
it's
basically,
then
you've
you've
done
the
best
you
can
can
to
prepare,
and
you
can't
worry
about
it
more
because
if
you
sit
there
and
you
stress
about
it
every
single
day
of
your
life,
you
would
go
insane.
So
we
don't,
we
just
ignore
it.
It
doesn't
exist,
it's
not.
We
are
not
democracies.
There
is
no
sword
hanging
over
our
bed
right.
B
Yeah,
it's
real
see.
I
look
at
it
as
most
of
those
things
are
fairly
predictable.
You
know
hurricanes.
We
tend
to
know
about
at
least
a
week,
if
not
two
ahead
of
time,
whereas
earthquakes
they
just
happen,
but
I
I
recently
saw
a
thing
that
made
me
think
of
of
californians
and
earthquakes
and
it
was
some
internet
meme,
that's
been
around
for
a
long
time,
I'm
sure
of
like
you
know
I
was
somebody
asked
in
eod
an
explosive
ordinance
disposal
guy.
You
know
so
the
bomb
squad
you
know.
B
Oh,
you
know,
isn't
that
job
stressful.
You
know,
don't
you
worry
about
if
you
do
the
wrong
thing
and
and
the
response
was
along
the
lines
of
like
either
I'm
right
or
it's
suddenly
not.
My
problem.
Yeah,
like
earthquakes,
are
kind
of
that
way
right.
If
it's
gonna
be
that
earthquake.
Well,
it's
just
not
yeah.
It's
problem
solved.
I.
A
I
knew
I
knew
an
old
air
force
guy.
He
was
an
air
force
bomb
tech,
eod
guy
right
and
he
was
just
he
was
always
had
this
calm,
demeanor
and
I
didn't
understand,
and
then
you
know
for
the
line.
Then
he
told
me
he's
like
he
was
like
yeah.
I
was
a
bomb
tech
in
afghanistan.
I
was
like
yeah,
oh
well,
okay.
I
guess
the
level
of
like
anxiety
that
you
have
is
it's
completely
different
than
mine
yeah.
I
just
don't
worry
about
the
little
things
at
all.
Yeah.
B
C
C
C
B
Not
just
old
I'm
fat,
I'm,
I
got
no
hope.
My
kids
are
way
more
athletic,
all
right,
so
yeah.
Thank
you,
hillary,
a
little
bit
of
backgrounds
on
today's
stream,
so
johnny
and
I
it
was
a
holiday
weekend
and
all
that
other
stuff.
We
we
both
kind
of
zoned
out
and
suddenly
it
was
like
monday
afternoon
and
we
realized
oh
crap.
We
don't
have
a
topic
for
this
week,
and
so
we
were.
B
We
were
chatting
on
slack
amongst
the
four
of
us,
the
three
of
us
here
in
christian
who's,
the
the
other
co-host
for
get
ops
guide
of
the
galaxy
and
hillary
had
this
great
suggestion.
So
hillary.
Whenever
we
see
you
next,
we
we
owe
you
a
beverage
of
your
choice
and
a
huge
thank
you
for
being
willing
to
join
us
and
talk
about
something
that
it's
actually
something
that
we
probably
should
have
talked
about
a
long
time
ago.
B
So
I
do
appreciate
you
bringing
it
up
offering
to
join
us
and-
and
I'm
excited
about
today's
topic,
so
great
all
right.
So
I
I
see,
we've
got
a
few
questions
already
happening
over
in
chat,
we'll
keep
track
of
those.
I
do
want
to
take
a
couple
of
minutes
just
to
go
through
quickly.
Go
through
our
top
of
mind
topics.
So
it's
been
a
relatively
quiet
week.
You
know,
I
guess
a
holiday
will
do
that
so
johnny,
I
don't
know.
A
A
C
Yeah
yeah
there
is,
if
you
are
running
openshift,
you
should
definitely
make
sure
to
upgrade
to
the
latest
versions.
There
is
a
bug
that
can
result
in
corruption
of
xcd
in
anything,
I'm
not
sure
which
ocp
version
is
an
osd,
it's
anything
before
4.9.3,
so
it
should
be
the
same
ocp
version.
So
that's
like
the
non-managed
openshift.
B
I
think
that's
scd
version,
not
openshift
version
or
anything
like
that,
where
it
would
effectively
have
a
silent
corruption
on
one
of
the
nodes
as
a
result
of
it
getting
out
of
sync-
and
I
want
to
say
I
know
for
sure
it
was
4.10.9
and
I
want
to
say
4.9.28
were
the
versions
where
they
inserted
the
mitigation,
which
is
basically
to
have
it,
do
a
a
check
beforehand,
and
so
that
way,
if
it
corrupted
it
would
no
longer
be
a
silent
corruption.
It
would
be
hey.
B
I'm
corrupted,
I'm
not
going
to
rejoin
the
cluster
here
and
then
okay,
and
then
I
think
I
remember
before
the
last
couple
of
weeks
and
summit
and
all
that,
like
we
were
on
the
cusp
of
having
a
fix
in
etcd,
3.5.3
or
4
or
whatever
the
next
version
was.
But
I
don't
know
if
that
made
it
into
openshift.
Yet
so.
B
Awesome,
yes
yeah,
I'm
always,
I
think
I
put
on
twitter
a
while
ago,
like
you
know,
hills,
I'm
willing
to
die
on
and,
like
always
running
the
latest
z
stream
is
one
of
those
yeah.
I'm
pretty
adamant
about
you
know.
Z
streams
are
important.
These
streams
is
where
we,
you
know
we
we
fix
all
of
the
bugs.
We
do.
All
of
those
you
know
apply
security
patches.
All
of
those
things
happen
in
z,
streams.
C
B
B
Yeah,
that's
that's
fair,
we'll
talk
about
that
in
just
a
moment.
So
a
couple
of
other
things.
I
said
that
there
was
no
no
releases
or
anything
like
that.
There
was
a
relatively
it's
small,
but
it's
kind
of
important
released
in
my
opinion.
So
let
me
let
me
share
my
screen
real
quick.
I
want
this
guy
yeah.
I
want
that
that
one.
B
So
I
I
quickly
deployed
just
a
4.10.11
cluster.
It's
not
the
current
version,
as
you
can
tell
it's
just
the
version
of
openshift
install
I
had
installed
and
what
I
want
to
talk
about
is
the
general
availability
of
the
web
terminal
operator.
So
this
is
one
that's
been
around
for
quite
a
while,
probably
a
year,
maybe
even
two
years,
just
in
tech
preview
status
and
what
it
does
is
it
offers
you
or
gives
you
the
ability
to
open
a
terminal
directly
inside
of
your
browser.
So
let's
go
here
and
go
to
operator
hub.
B
We
want
web
terminal,
so
we
can
see
it
no
longer
says
tech
preview
up
in
the
corner,
we'll
hit,
install
and
take
all
the
defaults
and
what's
going
to
happen
here,
is
it'll,
deploy
the
operator
and
then
up
here
in
the
corner,
I'll
get
a
little
another
icon
up
here.
That
looks
surprisingly
like
a
here.
It
looks
like
this.
The
little
carrot
underscore
it
always
reminds
me
of
the
powershell
yeah
it
does
it
really.
A
B
B
Let's
see
here
so
other
things
yeah
see
all
right.
So
it's
installed
and
now
we'll
come
back
up
here
to
I
guess
I
got
to
refresh
the
browser
after
a
second
it'll
deploy
the
thing
and
it'll
be
visible
up
there,
we'll
come
back
to
that
in
just
a
second
oftentimes.
The
operator
deploying
does
not
equate
to
the
service
being
available,
which
hopefully
we
we
know
that
now
so
just
a
couple
of
other
things.
So
one
in
this
blog
post
here
and
hopefully
stephanie.
B
I
don't
know
if
you're
posting
links
to
things
we're
all
three
of
these
are
in
the
notes
document
that
we
have
so
I
saw
this
blog
post
come
across
my
my
feed.
I
recently
installed
tiny
tiny
rss
which
I've
been
getting.
It
reminds
me
of,
of
the
google
rss
right.
The
the
google
feeds
stays,
so
I'm
recollecting
all
the
all
the
blogs,
all
the
things
to
read.
B
So
if
anybody
has
any
suggestions
please,
I
would
appreciate
that
you
can
post
them
here
in
chat
or
you
can
send
them
over
in
social
media
or
email,
andrew.sullivan,
redhat.com,
so
anyways,
a
great
blog
post
here.
So
ian
talks
about
how
to
do
a
get
ops
type
of
method
for
creating
and
protecting
at
cd
backups.
B
So
this
is
something
that
I
I
think
we
universally
encourage
folks
to
do,
create
and
protect
those
lcd
backups.
But
I
think
we
also
want
folks
to
understand
that
etcd
backups
are
not
a
full
disaster
recovery
type
of
solution
right,
if
my
whole
cluster
is
lost,
it's
not
the
best
option
for
standing
up
a
whole
new
cluster
in
the
recovery
site,
and
then
you
know
going
and
restoring
std
onto
that
new
cluster.
B
It
it
may
work
but
you're
going
to
have
a
lot
of
of
problems
and
a
lot
of
issues
doing
that.
I
don't
remember
if
I've
said
it
here
on
the
stream
before
johnny.
Maybe
maybe
you
remember
a
lot
of
times.
I
think
we
forget
to
decouple
application
disaster
recovery
from
openshift
disaster
recovery,
and
I
especially
when
we're
talking
about
virtualized
environments
right.
We
think
you
know
vsphere
rev
openstack.
All
of
that
you
know
we
think.
B
Oh,
if
I
just
restore
the
openshift
virtual
machines,
everything
will
come
back
but
right,
there's
there's
a
whole
nother
layer.
Those
applications
need
to
be
accounted
for
kind
of
completely
separately
and
the
way
that
I
usually
talk
to
a
virtualization
admin
about
it
is,
you
know,
think
about
your
vsphere
environment.
If
something
happened
in
the
primary
environment,
do
you
just
do
a
backup
of
the
esxi
host
and
then
restore
it
onto
a
new
esxi
host?
B
And
you
know
oh
yeah,
all
the
vms
will
just
magically
come
back
and
all
the
stuff
inside
no
right,
you
have
to
restore
vsphere,
and
then
you
restore
the
vms
that
are
inside
of
there.
Well,
we
do
the
same
thing
with
openshift,
you
restore
openshift,
and
then
you
restore
the
applications
that
are
deployed
on
top
of
openshift.
A
B
C
So
it's
the
type
of
sre
I
am-
is
actually
really
important.
For
this
conversation
right
there
are
two
distinct
types
of
sres
at
red
hat.
You
have
the
sres
who
are
responsible
for
openshift
dedicated
itself.
That's
not
what
I
do
right.
I
I
know
how
to
do
a
lot
of
that
stuff,
probably
most
of
it.
If
not
all,
I
won't
say
all
because
I'm
sure
they
know
some
interesting
tricks.
I
don't
because
I've
I
just
never
had
to
do
it
right,
I'm
a
layered
product,
sre
or
a
managed
service
sre.
C
So
when
you
look
at
things
like
the
red
hat,
open
api
manager
or
you
look
at
the
red
hat,
open
red
hat
openshift
streams
for
apache
kafka
rosac,
it's
managed
kafka.
Okay,
that's
what
it
is
when
you
look
at
that.
That
is
what
I'm
a
reliability
engineer
for.
That
is
what
my
team
manages.
One
of
the
same
group.
Different
teams
manages
quay
right
at
io
right,
that's
what
we
manage,
so
my
expertise
is
actually
the
apps
that
run
on
openshift.
C
How
is
your
application,
interacting
with
your
platform,
right,
the
openshift,
dedicated
platform,
and
so
a
lot
of
what
I
do
has
to
do
with
exactly
exactly
that
right?
How
are
we
making
sure
that
this
service
is
running
correctly
on
openshift,
and
how
are
we
making
sure
that
this
service
is
recoverable
with
openshift,
and
I'm
completely
I
mean
I'm
not
100,
removing
open
shift
itself
from
my
considerations,
but
by
and
large
I
don't
actually
go
that
deep
into
it,
because
I'm
I'm
at
the
application
layer.
I
don't
actually
necessarily
care
what
the
applications
do
either.
B
Yeah,
it's
a
question
that
you
know
I
I've
been
meaning
to
get
christian
on
for
a
long
time
and
you're,
just
as
good
as
christian
for
for
asking
or
talking
about
the
question
of
like
get
ops.
I
think
in
particular
sort
of
personifies
or
exemplifies
this
concept
of
the
openshift
cluster
itself
is
disposable.
B
B
You
can
use
both
at
the
same
time,
if
you
so
choose,
but
deploy
a
whole
new
cluster
right
relay
out
the
configuration
and
redeploy
the
applications
and
it
kind
of
all
comes
back,
so
it
it's
one
of
those
really
interesting
things
for
me
around
openshift,
kubernetes
containers
and
the
the
what
I'll
call
the
the
old
mindset.
I
say
old
mindset,
because
it's
a
mindset
that
I've
had
for-
and
you
know
20
years
and
I
struggle
with
this
whole
thing
of
no.
I
I
want
to
love
my
clusters.
I
want
them
to
be
healthy.
B
A
This
goes
back
to
that
conversation.
You
remember
a
few
months
ago
we
got
asked.
I
can't
remember
his
name
now
but
like
we
got
asked
offline
about
like
the,
why
aren't
clusters
openshift
clusters
in
that
pets
versus
cattle
conversation
yeah?
You
know
what
I
mean,
so
we
we've
always
been
talking
about
apps,
and
then
we
started
talking
about
vms
like
that
and
then
he
was
like
well.
C
Yeah,
if
you're
not
talking
about
clusters
versus
cattle,
I
don't
know
how
to
help.
You
like
you
should
be
there.
So
there's
an
interesting
question.
I
want
to
address
it.
Real
quick
does
hillary's
team
use
operators
for
these
managed
services
or
something
else,
and
the
answer
is
yes,
any
managed
service
that
red
hat
puts
out
must
be
a
level
four
operator.
So
if
you
look
at
what
is
operator
hub
dot,
io,
I'm
the
worst.
C
C
We
also
have
collections
of
scripts
and
so
forth
and
when
one
versus
the
other
comes
into
play
depends
on
what
type
of
managed
service
it
is
where
it
lives
if
it's
living
on
red
hat,
owned
infrastructure
or
actually
end
customer
owned
clusters,
because
I
sre
services
that
do
both
so
that
that
all
depends.
But
but
these
are
operators,
if
you've
seen
my
helm
and
back
again
talk.
I
talk
about
these
operators
and
some
of
the
some
of
the
ways
that
an
operator
can
burn
you.
B
Actually,
I
would
just
submitted
one
to
kubecon
here
in
north
america,
or
jointly
submitted
one
with
some
some
peers
at
a
partner,
and
we
can
so
if,
if
we
can
stephanie
share
my
my
screen
real
quick.
So
if
we
go
to
operator
or
framework
dot
io,
so
this
is
the
site
that
you
were
referring
to.
I
think
hillary
and
if
we
go
to
the
sdk
documentation
and
then
the
documentation,
link
and
then
overview,
we
have
this
nice
little
chart
of
operator
maturity
or
operator
capability
level.
B
So
just
for
reference
this
level,
one
two,
three
four
and
we
have
basic,
install
upgrades
lifecycle,
deep
insights
if
you've
ever
wondered
like
when
you
come
over
here
to
operator
hub-
and
you
select
here
I'll
pick
one
at
random-
that's
what
these
capability
levels
map
to
so
the
these
things
right
here
are
these
things
and
that's
that's!
What
it's
referring
to
right
is
what
are
the
set
of
capabilities?
What
are
the
set?
B
C
B
C
So,
whatever
we're
doing
we're
working
more
efficiently,
when
we've
audited
away
automated
away,
what
sre
calls
the
toil
right?
There's
those
mundane
repeatable,
easily
automated
tasks
that
you
have
to
do
a
lot
that
are
not
necessarily
very
difficult,
a
very
high,
lift
or
or
require
the
human
thought
process
to
execute
on
there's
a
question
in
the
chat
about
integration,
engineers
and
sres,
and
I
think
this
this
person's
asked
this
twice
now,
so
I
want
to
address
it,
and
the
answer
is
no.
C
I'm
not
saying
that
they're
the
same,
I'm
also
saying
that
when
I
spent
11
years
as
quality
engineer,
90
of
my
job
is
effectively
the
same
as
an
sre.
So
when
you
say
integration
engineer
is
actually
going
to
depend
a
little
bit
on
the
company.
The
the
big
thing
to
know
about
somebody
who
is
a
reliability
engineer
the
the
thing
that
we
do
that
really
stands
apart
is
we
are
looking
holistically.
C
I
actually
sometimes
say:
sre
is
a
quality
function,
we're
looking
holistically
at
these
apps
that
we're
asked
to
to
ensure
the
uptime
for
right?
We
are
looking
at
what
are
we
telling
the
customer
they're
getting
from
this
app
and
we're
looking
holistically
at
how
we
make
sure
the
app
can
deliver
on
those
promises
with
or
without
us
right
so
in
both
in
both
scenarios.
C
So
I
care
about
things
like
integration
points.
I
I
care
about
things
like
sysadmin,
I
I
have
had
to
run
netcat
to
debug
a
production
issue
right,
like
that
happens,
like
a
lot
of
system
and
level
things
happen,
code,
debugging,
actual
software
engineering
things
happen
right,
it's
it's
like.
Sometimes
it's
one
part
it
one
part
system
and
one
part
software
engineer,
and
sometimes
it's
all
three
in
the
same
day,
but
that's
not
what
sets
sres
aside
from
from
the
other
professions.
What
sets
those
threes
aside
from
the
other
professions?
C
Is
that
experience
and
understanding
of
everything
that
can
go
wrong
in
production
and
taking
that
and
turning
it
around
and
driving
the
maturity
model
of
the
software?
We
support
to
force
it
to
be
better,
to
make
it
more
reliable
to
to
insist
on
change,
and
I've
said
this.
I
I
went
on
a
dynatrace,
pure
performance
podcast
a
while
ago,
and
I
told
them
this.
This
is
our
like
secret,
open
source
contribution
right.
Our
names
are
not
on
this.
C
We
don't
write
this
code,
but
we
are
sitting
there
and
we
are
working
with
these
engineering
teams
that
are
providing
these
products
and
we're
telling
them
hey.
This
is
how
you
can
improve
things.
This
is
you
know
how
you
can
change
the
way
your
your
scaling
model
works
to
to
be
more
highly
available
and
and
more
reliable
right,
and
so
we
we
have
this
very
close
relationship
with
engineering,
or
we
try
to
have
this
very
close
relationship
with
engineering
to
get
them
to
really
understand
not
how
their
software
works,
but
how
their
software
runs.
B
B
No,
no,
that's
fine!
It's
you
know
the
joy
of
working
with
another
stream
host.
Is
you
know
exactly
how
all
these
transitions
work
and
all
that
so
thank
you
so
yeah.
So
for
anybody
who
is
watching,
please
don't
hesitate
to
ask
any
questions.
I'll.
Do
my
normal
reminder
here
of
whatever
it
is.
That's
on
your
mind,
anything
openshift
related
feel
free
to
ask
that
in
the
chat
we'll
do
our
best
to
answer
those
questions.
B
We,
if
it's
related
today's
topic,
of
course,
we'll
discuss
it
with
hillary
and
and
get
answers
as
best
as
possible.
If
it's
other
technical
things
don't
hesitate
to
ask
those
we'll
do
our
best
to
answer
those
too.
If
we
can't
find
the
answer,
we
we
generally
know
where
to
go
and
ask
the
answer
inside
a
red
hat,
whether
it's
product
management
or
engineering
or
any
other
teams,
and
we'll
certainly
do
that
and
follow
up
either
in
a
blog
post
or
in
a
follow-up
stream
all
right.
B
B
So
we've
heard
there's
system
administrators,
there's
devops
engineers,
there's
sres
right
there
there's
these
three
core
descriptors
for
people
who
are
either
responsible
for
the
infrastructure
or
the
platform
or
the
application
or
various
mixtures
of
all
three
of
those.
So
andrew's,
layperson
description
of
each
of
those
is
effectively
a
system
administrator
right.
An
operations
team
is
responsible
for
basically
everything
that
isn't
the
application.
B
I
want
to
make
sure
that
you
know
vsphere
or
whatever
it
is
that
I'm
using
is
up
and
running
the
most
efficiently
and
and
effectively
write
high
availability,
all
those
things
as
possible,
and
then
we
had
devops
and
devops
to
me
was
I'll,
say
teaching
operations,
people
how
to
be
software
engineers
right
so
bringing
an
operations
approach
to
software
engineering
and
sre
is
sort
of
the
inverse.
I
think
a
lot
of
times.
It
brings
software
engineering
to
operations,
and
I
I
I
agree
very
much
with
your
very
squishy
or.
C
So,
like
I
said
earlier
right,
11
years
in
quality
engineering,
and
I
did
a
lot
of
production
support,
I
did
a
lot
of
sre
work
right.
So
what
what
sre
has
done
is
it's
codified,
a
set
of
responsibilities
that
was
typically
shared
across
a
several
different
people
right
in
in
past
lives?
It
was
like
myself,
this,
the
you
know,
sysadmin
and
the
lead
developer,
who
had
production
keys,
and
it
would
be
one
or
you
know
all
of
us
going
in
and
working
on
production
issues.
C
Basically,
instead
of
the
lead
developer,
having
production,
keys
or
the
head
of
quality
having
production,
keys-
or
you
know
the
person
in
charge
of
devops
or
assist
administration
or
or
what
have
you
having
the
production
keys,
you
have
the
the
sre
group
that
has
that.
So
what
is
devops
devops
is
about
helping
promote
helping
developers
to
not
have
to
worry
about
the
infrastructure
so
much
it
allows
developers
to
just
think
about
the
code
and
not
the
infrastructure.
C
C
I
know
right
oh
gosh
anyway,
but
a
typical
full
stack
developer
had
to
understand
everything
that
was
happening
in
linux,
apache,
mysql
and
php
right.
When
you
add
in
devops,
and
you
work,
you
give
somebody
the
ability
to
create
tooling
to
help
developers
integrate
with
those
their
code
with
those
pieces
a
little
bit
better
right.
You
get
someone
to
handle
the
database
administration,
ideally
right
that
allows
people
to
be
a
little
bit
more
specialized
a
little
bit
more
focused.
C
It
actually
allows
you
to
deliver,
deliver
faster,
better,
more
reliable
code
right
when
you
start
splitting
out
these
considerations,
so
devops
really
was
about
how
do
we
help
get
the
developers
delivering
code
in
a
consistent,
reliable
manner,
and
we
will
worry
about
the
infrastructure
and
everything
under
the
hood.
They
just
have
to
basically
use
our
tooling
and
write
their
code
right
and
that's
a
lot
of
what
devops
does
and
a
lot
devops
is
still
a
really
important
role
for
that
I'm
an
sre.
That's
not
what
I
do.
C
I
make
sure
the
developers
are
doing
it
like,
I
make
sure
they
have
ci
cd,
but
I'm
not
sitting
there.
Fine
tooth
comb
like
going
through
how
it's
done,
how
it's
implemented
and
if
it
works
like
your
stuff,
should
work.
That's
on
you,
that's
not
my
job
right,
devops
is
still
a
really
important
role.
Devops
may
or
may
not
have
had
production
keys
and
been
doing
production
support.
It
typically
was
not
if
there's
something
wrong
with
the
infrastructure
sure,
but
typically
not
their
job.
Right.
C
The
I
mean
this
is
admin
like
you
said,
like
that's
the
person
who
can
tell
you
everything,
that's
happening
in
your
server
right.
The
server
is
like
they
know
it.
Like
the
back
their
hand,
they
know
where
everything
lives,
how
it's
done
any
little
quirks
about
the
operating
system.
The
server
is
running
like
that.
That's
your
cis
admin
right.
They
really
are
going
to
know
what
is
happening
on
that
bare
metal
right.
You
have
to
know
that,
and
so
what
was
the
third
job
family
you
mentioned?
C
And
then
I
guess
the
sra
right,
it's
the
s3
teams
are
look.
I
spent
a
lot
of
a
lot
of
us
actually
come
from
things
like
quality
engineering
or
devops,
or
actually
more
scripting
backgrounds
and
operations,
heavy
backgrounds
than
actually
software
engineering
backgrounds.
Now
that
actually
depends
a
little
bit
on
which
team
you're
talking
to
who's,
doing
the
hiring
with
what
they
think
some
of
the
other
teams
have
have
sort
of
an
inverse
of
that.
My
team's,
a
very,
is
like
50
50..
C
C
B
C
That's
what
my
team
dates
back
to
is
speed,
henry,
not
me
by
the
way,
I'm
newer
than
that,
but
my
team
dates
back
to
feed
henry
and
then
there
was
something
called
rh
map
which
was
like
a
managed
mobile
application
development
platform
and
then
rh
mac
went
into
rhmi
and
rhmi
turned
into
today's
rome.
So,
like
more
than
five
years
greater
than
five
years,
I
think
it's
more
like
seven
years
of
of
operating
apps
at
red
hat
my
team
has,
and
so
because
of
that
we're
very
senior
heavy.
C
A
C
B
C
So
right,
so
we
we
have
this.
These
two
sets
of
backgrounds
and
a
good
sre
team
is
typically
balanced,
like
that
you're,
not
hiring
developers
with
an
interest
in
operations,
you're,
not
hiring
operations
with
an
interest
in
development,
you're
hiring
people
who
enjoy
production
support.
I
say
we're
the
adrenaline
junkies
of
the
software
development
world.
Yesterday
I
had
alerts,
go
off
and
hit
my
entire
fleet.
At
one
time
I
got
hammered
like
30
alerts
are
going
off
on
my
phone,
I'm
the
person
on
the
pager
right
and,
like
my
response,
is
like
cool.
A
B
B
Yeah
it
I
I
do
think
it
takes
a
special
personality
to
deal
with
that
type
of
pressure.
Like
most
of
the
time
when
I
caused
a
widespread
outage
it
it
like
that
the
key
phrase
was
there.
I
caused
right
and,
like
I
can't
tell
you
how
many
times
I
I
accidentally
deleted
the
wrong
data
store
inside
of
like
vsphere,
and
you
know:
why
did
these
40
vms
just
suddenly
disappear?
B
Wait!
Oops
and
you
know
you
go
and
fix
the
problem,
and
you
know
this
was
back
in
the
when
I
was
a
junior
admin.
I
say
I
can't
tell
you
how
many
times
I
know
exactly
how
many
times
it
was
two
so
yeah,
it's
one
of
those
like
there's
some
degree
of
oh.
No,
something
is
wrong.
It
is
a
widespread
something
it
is
having
significant
impact
and
not
panicking,
which
is
a
good
thing
right
yeah.
So
so
there's
a
couple
of
questions
here.
B
C
Yeah,
if
I
didn't
ask
answer
that
satisfactory,
let
me
know
and
I'll
come
back
to
it,
there's
one
from
like
way
back.
We
didn't
actually
go
back
to
johnny
reminded
us,
the.
How
does
esri
help
improve
disaster
recovery
and
that's
a
really
great
question
and
the
answer
is
kind
of
what
I
talked
about
with
the
onboardings
and
saying
hey.
You
know:
hey
we're
going
to
do
this
review
and
you
have
to
prove
these
things.
C
So
we
actually
talked
to
the
tenants
during
their
onboardings
about
their
disaster
recovery
strategies
and
there's
several
different
ways
that
when
a
disaster
strikes,
sre
makes
decisions
right.
We
have
to
make
data
driven
decisions,
so
I
have
a
learning
and
monitoring
the
alerting
and
the
monitoring
tells
me
what
to
prioritize
in
the
application.
C
So
and
then
the
other
thing
is
the
service
definition.
I
actually
have
to
know
what
we've
promised
customers.
What's
our
sla,
some
some
sre
groups
only
care
about
the
service
level
objective
right,
which
is
like
what
are
we?
What
are
we?
How
are
we
trying
to
measure
that
we're
doing
well,
but
because
I'm
dealing
with
applications
that
are
facing
and
end
customers
are
purchasing?
C
C
What
will
go
wrong
with
my
software,
because
that's
that's
the
sra
expertise
is.
We
can
tell
you
what
will
probably
go
wrong
and
help
you
plan
accordingly.
So
a
lot
of
the
disaster
recovery
stuff
is
actually
done.
Some
of
its
tooling.
There
is
tooling,
of
course,
it's
actually
done
during
the
onboarding,
so
working
with
the
tenants
to
make
them
to
make
them
think
about
it
sooner
and
come
into
being
a
managed
service
with
a
plan
because
at
3
a.m
and
everything
is
on
fire.
I
need
a
sop.
B
I
think
there's
something
that
you're
you
mentioned
it
before
and
I
feel
like
you're
glossing
over
it
here,
which
is
you
also
have
that
relationship
with
the
engineering,
the
app
dev
teams?
Yes,
to
drive
that
awareness
back
into
the
product
itself,
so
that
it's
not
an
afterthought.
It's
not
made
more
difficult.
Ideally,
the
application
helps
with
that
whole
process,
and
you
know
removing
those
what
I'll
call
barriers
to
availability.
B
A
C
B
C
I
will
say
that
red
hat
is
working
on
putting
out
some
open
source
s3
training
a
little
bit
of
a
spoiler
alert.
You
can
sort
of
start
seeing
things
on
operate.
First
about
this.
You
will
see
content
that
comes
from
myself
and
others.
On
this
I
would
say
like
look
at
the
training
as
an
sre,
I'm
typically
considered
a
jack
of
all
trades
right
I
mean
I
do
get
ops.
I
support
get
ops
infrastructure
as
well
as
an
sre.
I
do
ansible.
I
do
you
know
programming.
I
have
to
know
I.t
stuff.
C
Really
do
you
really
do
it's?
Okay,
it's
okay,
to
have
like
a
niche
like
set
skill
but
yeah,
and
then
of
course
the
thing
is
is
like.
How
do
you
respond
to
pressure?
You
know,
like
I
said
I
got
hammered
yesterday
for
over
an
hour
of
constant
alerts,
that
something
is
going
wrong
with
my
entire
fleet.
C
C
Right,
but
you
know
what
the
customers,
the
customers,
don't
know,
because
what,
when
do,
when
do
our
alerts
go
up?
Our
alerts
don't
go
off
when
we've
slipped
our
agreement,
all
right.
Let's
our
alerts
go
up
when
we
could
be
at
risk
of
slipping
our
agreements
and
that's
when
essa
reacts.
So
by
the
time
our
the
customer
would
know,
we've
already
fixed
it.
That's
like
that's
the
nirvana,
that's
the
goal,
so
that
you
know
that.
C
That's
really
that's
really
so
wanting
to
go
from
a
system
to
an
sre,
you're
you're,
basically
going
to
need
to
get
familiar
with
some
of
the
tools
that
are
kind
of
considered,
devops
oriented
and
then
you're
going
to
need
to
basically
have
the
opportunity
to
do
production
support,
not
to
say
that
you
won't
find
a
group.
That's
willing
to
train
you,
but
that's
that's
one
of
the
things
that
we
typically
ask
during
the
interviews
like
hey.
C
What
experience
do
you
have
with
production
support,
because
you
will
get
a
pager
alert,
it
will
be
at
3am.
You
will
not
be
happy
about
that,
and
you
will
still
need
to
do
the
thing
and
and
and
know
how
to
respond
and
know
how
to
escalate.
If
things
are
well
outside
your
control,
like
sometimes
the
pager
alerts,
you
because
amazon
has
done
something
terrible
and
you
can't
actually
fix
that
right.
B
So
you
made
a
comment
earlier
about
how
your
team
is
kind
of
senior
heavy,
meaning
that
it's
it's
largely,
you
know
more
senior
people.
Do
you
find
that
that's
common,
like
it's
hard
to
hire,
somebody
who
is
a
junior
or
a
you
know
brand
new
to
the
industry
to
be
an
sre,
because
you
do
need
that
production
support
experience.
C
It's
it
is
fairly
common
and
it's
it's
exactly
that
I,
my
group
is
actually
my
particular
team.
All
of
our
hiring
is
is
intended
to
be
not
senior.
At
this
point
we
would
like
to
hire
people
who
may
not
have
this
experience
and
teach
them
and
upskill
them
now.
The
trick
is,
how
do
you
teach
somebody
to
have
production
experience
right?
I
actually
have
devised
a
chaos
engineering
game
for
this.
It
is
called
combat.
It's
kubernetes.
Chaos
combat.
It
is
a.
It
is
a
red
blue,
green
team.
It's
it's.
C
You
know,
you'll
be
familiar
with
this
kind
of
concept,
I'm
sure
so
you
have
this.
We
stand
up
a
copy
of
the
infrastructure
that
we
would
be
supporting
with
the
applications
that
we
would
be
supporting
or
an
application
that
we
would
be
supporting
and
before
the
game
starts,
the
red
team
has
to
go
in
and
break
it.
So
now
it
doesn't
matter
and
engineering
plays
qe
plays
cee
plays
and
sre
plays.
C
So
we
have
these
groups
and
you
rotate
through
the
teams,
so
you're,
not
always
engineering
as
engineering,
sometimes
you're
engineering
as
the
separator,
sometimes
you're
engineering
as
the
sre
sre
team,
and
so
like
that.
You
know
this
we're
doing
like
the
second
round
of
this
right
now
we're
trying
to
run
it
quarterly,
and
so
it
forces
people
to
to
learn
a
different
perspective
about
supporting
the
the
infrastructure
in
production
and
it
actually
practices
real
incidents.
So
we
go
in
and
we,
the
threat
team
goes
in
and
break
things.
Actual
alerts
are
firing
right.
C
We
do
actual
escalations
to
engineering
or
other
like
srep
the
platform
sre
group,
just
like
it
was
a
real
production
incident
and
all
the
other
sd
groups
do
something
similar
as
well.
This
is
just
the
way
my
team
does
it,
and
so
we
we
run
this
game
and
it
it
and
create
real,
real
incidents
that
you
have
to
respond
to
and
try
to
make
sure
that
we're
within
the
slos
and
slas
and
everything
and
so
forth.
So
we
actually
that's
what
we're
using
for
trading
and
then
we're
working
on
phase.
C
Two
of
this
is
actually
turning.
This
into
ansible
playbooks,
they
create
the
environment,
and
then
they
go
through
and
break
it
in
various
scenarios
that
help
the
people
learn
how
to
debug,
because
that's
the
way
to
practice,
we
practice
production
support
that
way,
and
so
that's
one
of
the
reasons
we
feel
very
confident,
aside
from
being
generally
senior
heavy
one
of
the
reasons
we
feel
pretty
confident
bringing
in
people
who
have
not
been
esseries
before
to
be
yesterdays
with
our
group
is
because
we
we
feel
confident
in
our
ability
to
train
them.
C
C
Yes
exactly
so
it
is
absolutely
that
and
it
just
sre
is
a
great
job.
You
know
and
like
I
said
like,
I
did
it
as
I
did
it
as
quality
engineer.
If
I
can,
you
know
just
be
very
frank
with
you
as
a
quality
engineer,
I
didn't
make
anywhere
near
the
amount
of
money
I
do
as
an
sre
and
the
work
was
exactly
as
hard,
so
this
industry
needs
to
do
better
by
quality
engineers,
just
fyi
as
a
rule,
seriously
underappreciated,
but
it's
just
it's
a
really
good
job.
B
Yeah,
so
I'm
not
to
cut
you
off
or
in
that
conversation
I'm
just
reading
back
through
questions
here
yeah,
so
I
I
see
shubham,
I
think,
asked
developing.
You
know
two
projects,
one
developing
ansible
playbooks
and
the
other
managing
kubernetes
infrastructure
for
dev
teams.
Does
that
mean
that
you're,
a
devops
engineer
or
an
sre?
And
I,
like
our
hope,
nine's
responsive?
You
know
why
not
both,
I
think,
a
lot
of
times.
We
focus
on
this
job
title.
B
Has
these
responsibilities
and
reality
is
it's
quite
messy
and
we
end
up
with
a
lot
of
different
roles
and
responsibilities,
and
you
know
I
would
argue
that
many
of
us
are
who
call
ourselves
a
system
administrator.
You
know.
Maybe
our
job
title
is
system
administrator,
we're
probably
doing
devops
and
sre,
and
maybe
even
sometimes
you
know,
engineering
right
software
engineering
type
roles.
B
You
know.
I
know
that
there's
a
number
of
folks
who
have
contributed
pull
requests
to
various
software
projects
that
they
use
right.
I'm
a
sysadmin!
I
encountered
this
problem,
here's
the
the
solution
for
it,
here's
how
to
to
fix
it.
Ironically,
that's
how
the
poison
pill
operator
came
about
right,
hey.
We
somebody
needed
the
ability
to
quickly
recover.
You
know
from
a
lost
node
in
less
than
the
five
minutes
that
we're
you
know
that
the
default
scheduler
takes
so
it's
here.
I
may
I
made
this
thing.
B
B
Yeah
I'll
add
on
to
that
of
hillary.
You
mentioned
you
know
one
of
the
big
things
where
one
of
the
big
metrics
with
sre
teams
is
slas
and
you
you
can't
just
have
an
arbitrary
sla.
You
know
so
in
order
to
be
able
to
define
your
sla,
you
have
to
understand
all
of
those
things
and
what
happens
so
you
know
it's
how
long
you
know
disaster
happens
whatever
it
is.
So
how
long
does
it
take
me
to
get
a
cluster
back
up?
B
Well,
if
it's
an
hour,
then
you
have
to
factor
that
into
your
sla
and
if
your
sla
needs
to
be
less
than
an
hour,
then
you
have
to
figure
out.
You
know
what
are
the
things
that
I
can
change
to
improve
that
you
know.
Maybe
it's
pre-staging
certain
parts
of
you
know
new
cluster
deployment.
So
that
way
it's
faster.
Maybe
it's
you
know
more
involved
or
more
thorough
automation
in
order
to
make
things
faster.
B
I
I've
always
taken
the
approach
of
the
amount
of
time
it
takes
is
the
amount
of
time
it
takes
and
either
we
have
to
work
around
that
through
automation
and
other
things
or
through
the
slas
that
we
define
but
yeah.
If,
if
you
don't
know
what
that
is,
then
you
can't
really
effectively
make
those
plans.
A
Yeah
and
sean
nelson
made
a
good
point
and
I'll
just
quickly
say
what's
up
to
sean,
but
he
says
the
metrics
based
process.
Mapping
could
help
help
measure
the
long
or
the
the
process
or
how
long
the
process
can
take
to
create
that
product
cluster.
But
at
the
end
of
the
day,
right,
it's
still
it's
a
maturity
thing
you
have
to
have.
A
You
know
you
have
to
know
which
metrics
you
need
to
collect
and
then
how
to
actually
interpret
those
metrics
and
then
make
a
decision
based
off
of
that
right
and
yeah.
It's
definitely
there's
the
tools.
Are
there
it
just
it
takes
it
takes
time
to
get
there
and
it's
it's
just.
It's
organizational
maturity.
At
the
end
of
the
day,.
B
A
And
then
so,
just
going
back
to
the
the
sre
from
sysadmin
type
of
conversation
that
we're
having
earlier
lester
was
saying
that
you
know,
obviously
we
have
to
bring
up
junior
people.
So
lester
made
a
comment
about
bringing
up
junior
people
from
from
the
ranks
so
that
way
they
can
get
the
experience.
C
Yeah
again,
though,
when
we
go
back
to,
I
was
talking
about
the
the
chaos
engineering
game.
Right
we
make
that's
exactly.
The
goal
of
the
game
is
to
make
sure
that
we
can
stay
with
confidence
that,
before
I
put
these
people
on
the
pager
that
they
have
that
and
some
of
what
you
get
during
the
the
interview
process.
You
know
you
typically
get
people
with
a
couple
years
experience,
but
we've
absolutely
hired
somebody
straight
out
of
college
and
she
was
amazing.
She
was.
She
was
absolutely
incredible.
So
really
it's
it's!
C
How
how
well
do?
I
think
I
can
work
with
this
person
and
how
well
do
I
think
this
person-
and
I
can
work
together-
is
also
based
on
things
like.
Do.
I
think
that
they're
going
to
complement
my
skill
set
not
do
they
mirror?
My
skill
sets,
I
think,
they're
going
to
complement
my
skill
set.
Do
I
think
that
I
can
help
them
grow,
I'm
the
technical
lead
right.
So
when
I'm
looking
at
somebody,
I
love
to
find
people
who
know
things.
C
B
B
Because
I
I
I
would
agree,
I
I
so
it's
funny,
so
I
used
to
work
for
a
storage
company
and
one
of
the
co-founders
is
a
a
well-known
or
was
a
well-known
speaker
before
he,
you
know
retired
and
did
whatever
co-founders
do
after
they're
gone
yeah
and
I
remember
sitting
in
new
higher
orientation
right.
So
you
know
it's
this
like
two
day
thing
and-
and
you
have
you
know
lunch
with.
B
You
know
senior
execs
on
day
two
and
all
this
other
stuff,
and
he
was
sitting
at
the
table
and
one
of
the
other
folks
at
the
table
asked
like.
Oh,
how
did
you
become
such
a
great
speaker
and
like
this
is
after
he'd,
given
us
like
an
hour-long
presentation
talking
about
how
he
was
like
a
super
introvert
and
all
this
other
stuff?
And
so
she
asked
him
like.
How
did
you
become
such
a
good
and
well-known
public
speaker
and
like
completely
dead
pan,
like
just
immediate
response
practice,
yep.
B
And
to
me
like,
basically,
you
know
people
talk
a
lot
about
how
you
know.
Oh
talents,
you
know
that
person
you
know
playing
an
instrument,
they
have
so
much
talent,
talent
gets
you
started
and
gets
you
so
far.
It's
the
investment
in
time
and
practice,
and
you
know
all
of
those
things,
and
it
applies
with.
Basically
any
skill,
troubleshooting
included,
yep.
C
B
Because
I've
I
encountered
a
lot
of
folks,
you
know
when
I
was
a
practicing
administrator
full-time.
You
know,
junior
folks
and
it's
you
know
I'll,
never
be
able
to
know,
learn
how
to
do
that.
I'll,
never
be
able
to
do
this.
I
don't
know
how
you
learned
that
skill
or
how
you
did
this.
It's
literally
just
do
it.
I.
C
And
yeah,
so
I'm
actually
fun
fact:
I'm
self-taught
right!
I
didn't
go
to
college
at
all,
and
so
I
can
tell
you
I
you
can
absolutely
teach
yourself
the
skills
right,
you
absolutely
can
you
can
learn
them
and
you
can
and
you
can
practice
them
and
you
know
do
things
like
look
for
capture
the
flags
right
capture.
The
flag
games
are
a
really
great
way
to
learn
troubleshooting
skills,
because
a
lot
of
that
will
come
into
troubleshooting
and
other
odd
things.
C
B
I
didn't
know
that
that's
kind
of
cool
yeah-
that's
cool
thinking
about
it.
It
makes
sense,
particularly
for
you
know,
with
osd
and
and
I'm
sure
I
know,
there's
sri
teams
for
the
other
managed
services
as
well.
That
makes
sense.
You
know
a
lot
of
our
customers
can't
right
right,
like
regulatory
things
can't
give
cluster
admin
or
willy-nilly.
C
No,
if
I
have
to
use
cluster
admin,
I
will
receive
some
compliance
paperwork
that
I
have
to
do
from
from
prosec
and
infosec
and
explain
what
did
I
touch
and
I
mean
they
know
what
I
touched.
They
can
see
it
right,
it's
logged,
but
what
did
I
touch?
Why
did
I
touch
it?
Why
was
it
necessary?
I
have
to
there's
a
whole
piece
of
paperwork
that
we
have
to
do.
It
has
to
go
through
various
layers
of
approval
to
demonstrate
that
what
I
did
really
needed
doing
so
yeah.
B
Yeah
yeah,
that's
really
cool
yeah,
our
hope,
nine.
Ninety
percent
of
troubleshooting
is
having
seen
it
before.
I
I
don't
know
if
it's
as
high
as
ninety
percent,
I
think
a
lot
of
it
and
when
I
used
to
interview
administrators
like
hiring,
you
know
unix
solaris,
administrators,
hiring,
linux
administrators.
B
B
What's
all
of
the
you
know,
specific
parameters
and
how
to
use
them
for
the
ls
command,
because
there's
like
400
of
them
rather
it's
well,
I
think
I'd
start
by
doing
a
man
of
this,
and
if
that
didn't
you
know
have
the
answer,
then
I
would
go
and
look
here
and
you
know
and
like
I
want
to
know
that
process
and
I
think
troubleshooting
is
very
similar
of
you.
Maybe
there
is
some
experience
or
I
would
say
there
is
definitely
some
experience
of.
I
can
jump.
B
C
A
A
So
you
have
to
be
able
to
kind
of
like
look
at
it
holistically
and
say:
okay,
well,
here's
the
different
things
that
could
affect
it
and
then
just
kind
of
be
creative
with
it
and
then,
and
that
I
think,
helps
when
you
have
like
a
deeper
problem,
because
you
don't
get
stuck
on
okay
like
trying
to
go,
you
know
linearly
linearly,
trying
to
go
linear.
You
know
along
your
line
or
your
process
going.
A
C
Some
other
problem
that
causes
a
problem
in
that
thing,
and
so
you
actually
what
you're
getting
pinged
on
is
the
cascaded
effect,
but
you
have
to
be
able
to
trace
it
all
the
way
back,
and
so,
if
I
just
you
know,
got
stuck
on
me
well,
this
is
the
alert.
This
is
the
error.
This
is
what
should
fix
it
and
it's
still
not
working
right
that
wouldn't
be
like
the
most
useful
so
like
it
is
understanding
again.
B
C
Actually
it's
in
our
onboarding
packet
to
go
to
certain
meetings
and
meet
certain
people,
believe
it
or
not.
Yes,
part
of
our
onboarding
is
hey
new
person
meet
engineering
engineering,
new
person
right
like
it.
Actually
it
actually
is
in
our
onboarding.
I
know
you
were
joking,
but
that
is
part
of
the
onboarding
is
to
go
meet
the
people
you
need
to
have
relationships
with
yeah.
B
No,
I
we
do
the
same
thing
on
our
tech,
marketing
team,
so
yeah
our
hope.
Nine
yeah.
Thank
you
for
the.
Let
me
google
that,
for
you
link,
I
I
I
I
pull
that
out
occasionally
when,
when
somebody
has
asked
me
the
same
question
like
40
times
and
you
know
or
or
I've
had
other
people
on
my
team,
where
people
will
send
you
know
like
each
person
on
the
team,
the
same
question
to
see
what
responses
they'll
get
and
then
they'll
pick
their
favorite
one
yeah.
That's.
B
C
I
touched
it
briefly
and
let
me
answer
this
because
this
is
going
to
depend
on
the
organization,
some
organizations,
the
sre
teams,
find
a
bug
in
production
and
they
actually
go
all
the
way
through
and
fix
the
production
bug,
and
they
need
to
be
able
to
code.
For
that.
That
is
not
how
my
sre
team
does
it.
C
I
s
I
support
apps
in
golang,
ruby,
java,
there's
python
in
there
I
we
we
are
language
agnostic
in
what
we
support,
so
we
don't
fix
production
code
bugs
we
identify
them,
we'll
file
the
jiras
and
tickets
for
them,
but
we
don't
actually
fix
them.
So
with
that
whether
how
how
much
coding
you
really
need
depends
on
what
type
of
sre
organization
you're
going
into
and
whether
or
not
you're
going
to
be
expected
to
fix
bugs
as
part
of
the
sre
team.
C
I
don't
like
that
model
personally,
because
I
think
that
you
get
like
way
too
stuck
into
one
one
kind
of
coding
thing
and
then
it
again
it
depends
on
the
team
again.
So,
even
though
s3s
at
red
hat,
don't
fix
code
bugs
like
that,
some
of
our
teams
do
more.
Like
software
traditional
software
development
stuff
than
others,
my
team
I
mentioned
again
has
very
like
50
50
split
lots
of
people
with
a
system
in
background.
So
when
we're
typically
doing
tooling
and
automation,
we
use
a
lot
of
bash.
C
We
use
some
python
occasional
goaling
this
or
that
and
ansible
playbooks
right,
because
those
are
those
are
yaml
driven
and
we
all
know
how
to
link
yaml
in
our
sleep.
At
this
point,
so
you
know,
we
typically
ask
that
someone
has
the
ability
to
script
in
python
when
joining
my
team
and
we're
looking
for
things
like
okay,
can
you
do
scripting,
but
did
you
also
handle
for
like
errors
right?
C
B
Yeah
yeah,
you
don't
need
to
be
a
software
engineer,
but
you
do
need
to
I
I
I
think
being
having
the
ability
to
read
code
and
understand.
What's
happening
is
important.
I
I
fall
into
one
of
those
things
like.
I
can't
write
a
lick
of
c,
but
I
can
read
it
and
I
can
tell
you
generally:
what's
going
to
be
going
on
assemblies
on
a
whole
nother
language
or
a
whole
other
level.
I
I
got
no
hope
there,
but
yeah
most
languages
and
my
son
again
he's
14
right.
B
He
has
an
interest
in
these
things,
he's
been
doing
coding,
classes
and
I've
been
very
adamant
with
him
of
learn.
The
fundamentals
right
learn.
What
a
loop
is
learn.
How
logic
you
know
is
handled,
learn
all
these
things
and
the
language
is
really
unimportant
after
that,
it's
just
syntax
yeah.
No,
you
want
to
write
in
pearl
and
use
all
the
keys
on
your
keyboard
by
all
means.
I
love
pearl.
Oh,
I
wrote
a
lot
of
pearl.
I.
A
C
A
C
C
It
wouldn't
be
human
readable
or
at
all,
so
the
thing
I
love
about
pearl
is
pearl
is
a
language
that
taught
me.
I
can
learn
anything
in
17
hours.
I
learned
pearl
for
a
job
interview.
I
learned
pro
for
a
job
interview
and
I
had
to
do
live.
I
had
to
do
a
coding
project
in
pearl
over
a
course
of
a
weekend
and
then
I
had
to
go
in
and
do
the
coding
stuff
live.
C
I
did
not
get
that
job
because
I
learned
pearl
for
the
interview
and
it
kind
of
showed,
but
I
I
learned
pearl
in
17
hours
of
straight
up
working
on
learning
pearl
and
I
honestly
like
that
was
the
one
I
was
like
I
can.
I
can
actually
do
this.
I
can.
I
can
learn
anything,
my
app
that
I
submitted
by
the
way
had
unit
tests
in
everything
because
unit
tests
are
life,
and
I
will
that's
my
hill
I'll
die
on
that
one
by
the
way
but
yeah
I
love
pearl.
C
A
B
C
B
B
So
yeah
pearl
is
one
of
those
and
I
think
python
falls
into
that
category.
It's
kind
of
universal
right.
You
can
do
basically
anything
with
it.
Sometimes
it's
easier
than
others.
I
would
argue
that
at
this
point,
despite
the
prolific
number
of
modules
in
cpan,
perl
is
more
difficult
than
some
other
options
out
there,
but
that
doesn't
mean
it's
less
less
capable
right.
People.
A
C
C
C
A
B
Yeah
we,
I
know
we
had
a.
We
had
a
lot
of
love
for
pearl.
That's
why
I
learned
it.
That's
how
I
got
started
in
jamal
new
to
openshift
most
problems
of
face
when
deploying
are
privileged
containers.
What's
your
tip
for
it
yeah,
so
that's
pretty
common.
Actually,
so
openshift
does
not
allow
root
in
containers
or
privileged
containers
by
defaults.
B
B
You
know
in
your
container
definitions,
you
assign
a
user
id,
that's
not
root
if
you're
accessing
things
right,
keep
in
mind
that
ui
uids
are
randomly
assigned
when
the
pod
is
started.
So
if
you're
accessing
you
know,
say
an
nfs
export
or
something
like
that
you're,
if
it's
tied
to
a
specific
uid,
you
know,
access
to
those
files
is
tied
to
a
specific
uid,
not
a
group
id,
because
you
can
set
the
supplemental
group
id
you're
you're,
going
to
have
a
bad
time,
yeah
I'll
see.
B
If
we
can
dig
up
that
that
link
jamal.
If
you'd
like
you
can
send
me
a
message,
so
you
can
send
me
an
email,
andrew.sullivan
redhat.com
or
on
social
media,
practical
andrew
on
twitter
reddit,
all
of
the
places.
Thank
you
stephanie
for
popping
up
my
my
contact
information
there.
So
yeah.
If
you
reach
out
I'll
I'll
shoot
you
back
that
the
link
along
with
any
other
information
that
we
have,
because
it
is
a
fairly
common
thing
that
comes
up.
B
So
it
is
we're
approaching
quarter
after
12
here
on
the
east
coast,
which
is
well
past
coffee
time
on
the
west
coast
for
hillary.
So
if
anybody
has
any
questions,
if
there's
anything
that
is
still
outstanding
in
your
mind,
if
we
missed
a
question
for
example-
hopefully
we
didn't,
but
it
does
happen.
Please
go
ahead
and
get
those
in
here
we'll
address
those
before
we
end
our
stream.
B
B
Know
it's
I
I
have
never
done
it
have
I
I
might
have
done
it
completely
by
myself
once
I
don't
think.
I
did,
though,
because
I
think
I've
always
had
stephanie
or
you
or
chris
short,
when
he
was
doing
when
he
was
here.
I
don't
think
I've
ever
done
it
by
myself.
So
yeah,
it's
a
lot
going
on
so
yeah.
We
won't
be
streaming
next
week,
the
week
after
we
will
be
back
much
like
this
week.
I
don't
know
the
topic
yet,
but
we
do
have
a
really
good
list
of
things.
B
This
time.
Last
week
last
week
we
had
like
nothing
and,
like
I
said
earlier,
hillary
build
us
out.
So
please,
whatever
platform
you're
on
subscribe,
you
can
follow
us
on
social
media
and
as
soon
as
we
nail
down
or
cement
the
I'm
using
a
lot
of
construction
analogies
today.
Our
topic
for
two
weeks
from
today
we'll
be
sure
and
share
that.
So
that
way
you
can
join
us.
There's
lots
of
interesting
stuff.
That's
that's
going
on
as
well.
B
B
B
B
I
miss
my
wife
does
the
same
thing:
occasionally
she'll
bring
bring
beverages
anyways.
So
thank
you
so
much
to
to
johnny
for
joining
me
today
for
to
stephanie
for
helping
us
out
hillary
again.
Thank
you
so
much
for
joining
us.
This
has
been
super
fun.
Super
enlightening.
I've
had
the
opportunity.
We
have
a
lot
of
really
smart
people
yourself
included
here
at
red
hat,
who
not
just
practice
devops
in
sre,
but
who,
who
are
also
like
well
known
and
and
really
good
at
pontificating
about
it.
B
So,
john
willis,
you
know
andrew
clay,
shafer
they're,
both
here
at
red
hats.
I
I
would
definitely
encourage
you
to
check
out
the
stuff
that
they
published
on
the
topic
as
well.
They're
they're
way
more
knowledgeable
on
it
than
I
could
ever
hope
to
be.
What's
that
thing
about
they've
forgotten
more
about
it
than
I'll
ever
know
so,
yeah
with
that,
I
think
I
said
all
my
thank
yous.
I
think
I've
reminded
folks
of
everything
that
I
needed
to
remind
you
of
don't
hesitate
to
reach
out.
C
C
Any
parting
shots:
oh
man,
that's
like
inviting
it
I'll
I'll
I'll
leave
you
you
poor
old
men
alone.
We
do
have,
as
mentioned
earlier.
We
have
tomorrow
at
3
p.m.
Eastern
12
p.m.
Pacific
get
ups
guide
to
the
galaxy
with
christian
hernandez
and
myself.
Oh
I'm
terrible.
I
believe
our
topic
is
argo,
cd
notifications.
C
So
that's
like
a
very
like
hard
turn
from
what
we
discussed
here
today,
but
it
should
be
a
good
episode
and
I
hope
folks
that
will
join
us
for
that,
and
then
we
can
just
give
christian
during
his
practical
demo
a
lot
of
a
lot
of
grief
about
the
fact
that
he's
not
doing
it
in
pearl
it'll
be
fun.
Join
me
for
that.
A
All
right,
awesome,
yeah,
no
hillary,
thank
you
for
coming
on.
This
was
awesome.
I
had
a
great
time
I
I
know
like
it's
fun,
cutting
up
with
you
and
you
know
so
it
was
great
having
you
come
on
and
you
know
really
bail
us
out
so
for
all
the
viewers
make
sure
that
you
go
to
youtube
or
wherever
you're
watching
from
and
like
and
subscribe.
So
that
way
we
can
kind
of
keep
these
things
going
and
stephanie,
as
always,
you're
awesome.
Thank
you
so
much
for
everything
and.