►
From YouTube: SLO Driven Engineering: from Dev to Prod
Description
Google’s SRE Book popularized the concept of Service Level Objective (SLO) and the SLO-driven approach. But what does it really mean to make SLO driven decisions? How can we generate observability and synchronize teams around joint SLOs? And how can we automate SLOs and integrate them into the software release pipeline?
In this episode I’ll host Andreas Grabner. We’ll discuss the SRE practices, and how to automate SLO from dev all the way to prod. We’ll talk about the open source efforts to standardize the process under the Continuous Delivery Foundation, and about Keptn, the new CNCF open source project that promises to help with this automation.
Andreas Grabner (@grabnerandi) has 20+ years of experience as a software developer, tester and architect and is an advocate for high-performing cloud scale applications. He is a contributor and DevRel for the CNCF open source project keptn (www.keptn.sh). Andreas is also a regular contributor to the DevOps community, a frequent speaker at technology conferences and regularly publishes articles on blog.dynatrace.com or medium. In his spare time you can most likely find him on one of the salsa dancefloors of the world.
The podcast episodes are available for listening on your favorite podcast app and on this YouTube channel.
We live-stream the episodes, and you’re welcome to join the stream here on YouTube Live or at https://www.twitch.tv/openobservability.
Have you got an interesting topic you'd like to share in an episode? Reach out to us and submit your proposal at https://forms.gle/9LDkYCmegyS5D8Li7
A
Hello,
everyone
and
welcome
to
another
episode
of
open
observability
talks,
I'm
your
host,
tutan
horowitz
and
here
at
open,
observability
talks.
We
talk
about
anything
devops,
observability
and
open
source.
I'd
like
to
thank
our
sponsors.
Log.Io.
The
cloud
native
observability
platform
luxo
uses
the
best
of
breed
open
source
projects
such
as
prometheus
elasticsearch,
open
search,
jaeger
and
so
on.
Thank
you
about
that
and
for
those
joining
the
live
stream
on
youtube
or
twitch,
then
feel
free
to
ask
questions
and
comments
on
the
chat.
A
It
will
make
things
much
more
interesting
and
we'll
definitely
try
and
reach
out
and
address
some
of
the
comments
and
questions,
and
with
that,
let's
move
on
to
today's
episode
and
today,
we'll
discuss
sre
practices
and
how
to
establish
slos
as
a
core
foundation
of
our
operations,
I'll
discuss
that
with
andy
grabner
who's,
a
devops
activist
at
the
dynatrace
and
he's
also
a
contributor
to
the
cncf
open
source
project
captain
for
automating
slos.
So
let
me
add
andy
to
the
stream
hello
andy
good
morning.
A
A
A
That's
impressive,
especially
in
our
in
our
industry
and
you've,
seen
our
industry
evolving.
I
guess
throughout
this
these
14
years
from
the
you
know,
traditional
system
days
to
to
today's
sre
and
devops
practices
and
observability
and
so
on.
So
I
think
it'd
be
best
place
to
to
start.
Maybe
if
you
can
give
us
your
perspective
on
what's
sre
and
how
it's
different,
maybe
from
devops
and
the
traditional
sysadmin
role
that
we've
come
to
know.
B
And
again,
what
I'm
saying
here
is
is
probably
my
opinion
and
everybody
has
their
own
opinions
and
is
entitled
to
their
own
opinions,
which
is
also
great,
because
this
allows
us
to
have
discussions
like
this,
because
if
I
would
have
the
final
truth,
then
that's
it
and
then
we
don't
need
any
of
these
podcasts
anymore
to
educate
people.
So
from
my
perspective,
and
things
have
changed
also
in
my
perspective,
over
the
years
since
I've
been
involved
in
this
area,
I
think
sre
such
reliability.
B
While
this
is
great
making
their
or
executing
their
actions
based
on
data
that
they
have
from
the
opposibility
platform,
I
think
for
sres,
it's
very
important
to
also
influence
the
architectures
that
are
built
so
therefore,
with
their
knowledge
on
how
systems
behave
and
also
with
that
knowledge
on
certain
technology
sticks,
they
need
to
influence
and
kind
of
mentor.
Architects,
software
engineers,
so
that
they
really
build
systems
that
they
are
observable
b,
can
also
provide
apis.
B
A
I
think
it's
interesting
what
you
said,
because
that's
also
part
of
the
transition
that
is
the
one
that
I've
experienced
in
the
industry.
The
emphasis,
for
example,
on
automation,
has
grown
significantly,
probably
much
more
so
than
14
or
even
10
years
ago.
Have
you
seen
this
this
shift
within
the
industry
throughout
these
these
years.
B
A
A
Yeah,
it's
interesting,
and
actually
I
had
a
very
interesting
episode
with
with
a
senior
sre
at
google
here
on
the
show
last
year
and
his
name
is
ramon
and
he
said
that,
for
example,
sre
at
google
is
is
a
separate
organization
and
we
discussed
that
we
discussed
also
different
organizations.
Like
I
don't
know
facebook
in
which
it's
structured
differently.
It's
not
a
separate
organizations.
Obviously
they're
pros
and
cons
to
each
I'm
wondering
how
do
you
see?
Where
do
you
see
sr
replaced
with
the
inside
the
org
chart.
B
Yeah
again
this,
I
don't
think
there's
a
one
size
fit
all.
I
think
it's
a
mis.
I
think
it's
great,
that
we
have
organizations
like
google
and
facebook
and
they
tell
us
how
they
run
things,
but
they
have
a
different
history
and
that
they're
grown
different
and
they're.
Just
a
different
organization
like
a
financial
institution
that
has
been
around
for
a
hundred
years
and
50
years
ago,
started
with
the
mainframe
and
now
has
evolved.
B
So
I
think
it's
a
little
it's
a
little
hard
to
just
make
the
comment
and
say
this
is
the
way
as
a
reason
sorry
need
to
be
done,
because
google
does
it
so,
and
everything
else
is
wrong.
So
I
I
believe
it
really
depends
on
the
organization.
What
does
not
change
is
what
an
sre
role
should
do
and
and
and
what
the
the
outcome
of
asterisks.
B
As
I
mentioned
earlier,
you
need
to
make
sure
in
a
world
that
we
live
today
in
2022,
where
pretty
much
everything
depends
on
software
being
available
and
running
perfectly
at
all
the
time.
We
need
to
make
sure
it's
an
organization
that
we
put
everything
in
place,
that
this
can
be
accomplished,
and
I
want
to
highlight
one
one
story
that
was
recently
brought
up
to
me.
I
was
fortunate
enough.
B
When
you
go
to
the
grocery
store
and
you
are
a
less
privileged
family,
when
you
don't
have
enough
income,
you
get
these
called
ebt
cards.
These
are
cards
that
people
I
give
amount
from.
The
government
get
a
certain
money
and
then
you
buy
it
like
a
debit
card
and
the
team
that
he
worked
with
was
responsible
that
these
transactions
actually
go
through
and
the
systems
behind
the
seams
do.
B
B
They
are
swiping
the
ebt
card
and
if
some
things
then
don't
work,
they
have
to
walk
home
without
feeding
their
family,
and
I
think
this
is
a
this
is
was
for
me
a
very
compelling
moment
when
he
said
he
remembers
when
they
were
sitting
in
a
situation
like
in
a
war
room
kind
of
scenario,
but
they
were
all
relaxed.
But
then
the
cto
came
in
put
the
fist
on
the
table
and
says
guys.
And,
ladies
and
gentlemen,
we
know
we
are.
B
You
are
responsible
for
feeding
people,
and
if
your
systems
are
not
reliable,
then
we
have
a
problem
as
a
as
an
organization,
but
more
so
that
the
society
were
impacting.
So.
This
is
why,
however,
you
are
structured
in
your
organization,
whether
you
have
a
dedicated
sov
team
to
provide
can,
let's
say,
observability
and
automation
as
a
service
to
the
development
teams,
whether
they're,
mentoring
or
whether
they're
baked
in.
I
think
you
need
to
figure
this
out
for
your
organization.
B
A
Yeah,
I
definitely
agree
on
the
mission
statement.
I
just
found,
at
least
from
my
experience,
that
the
yoga
structure
does
matter
and
the
organizational
culture
on
devops.
It's
it's
a
it's
a
long-standing
discussion
about
the
cultural
aspects
of
devops.
So
in
order
to
enable
the
impact
in
order
to
create
the
synergy
between
the
teams
in
charge
of
the
applications,
the
teams
in
charge
of
the
infrastructure
and
the
pieces
of
the
monitoring
you
know
it
is,
it
is
a
valid
discussion.
I
think
in
the
different
aspects
on
that
I.
B
Then
we
all
need
to
do
everything
it
takes
to
get
an
idea
as
fast
as
possible
to
our
end
users
and
get
the
feedback
and
make
sure.
If
something
is
breaking,
then
we
all
get
together.
So,
in
the
end,
as
an
organization,
we
need
to
figure
out
how
we
get
the
right
things
at
the
right
time
to
the
right
people
and
then
you
know,
take
the
consume
the
feedback
and
then
fix
things
or
improve
things.
B
B
B
The
way
I
I
explain
it
these
days
and
again,
this
has
also
changed
over
the
years,
because
I
learn,
I
see
things
and
then
I
adapt
also
the
way
I
advocate
for
things
I
see
devops
first
of
all,
devops
alone
is
a
very
loosely
defined
term,
because
it
can
mean
a
lot
of
things
and
not
enough
what
I,
what
we
try
to
do
is
use
the
term
devops
platform
engineers.
I
think
we
see
a
lot
of
devops
teams
they're.
Really.
B
B
You
know
how
can
we
get
stuff
from
development
all
the
way
into
production
by
providing
a
platform
that
includes
continuous
integration,
continuous
delivery,
automated
testing
automatic
quality
control
and
and
getting
stiff
stuff
into
production
kind
of
day
everything
until
day,
zero
like
deploying
it,
and
then
sres
is
really
what
I
mentioned
earlier.
They
are
using
the
automation,
the
observability
as
new
stuff
gets
into
production.
B
To
make
sure
we
get
the
insights,
we
can
react
on
problems
to
keep
the
system
reliable
and
stable
so
kind
of
day,
one
and
day
two
operations,
and
so
there's
obviously
they're
working.
They
have
to
work
together
because,
in
the
end,
what
combines
them?
What
connects
them
is
that
the
observability,
the
slice
and
slos
that
define
whether
a
system
runs
stable
in
production.
It
should
not
just
be
monitored
in
production,
as
I
mentioned
earlier,
they
need
to
figure
out.
B
How
can
the
devops
engineers
the
devops
platform
engineers
into
the
platform
that
they're
building
already
bake
observability
in
that
key
slis
key
indicators,
performance,
behavior,
architectural
behavior,
resiliency
behavior-
can
already
be
monitored
and
measured
in
your
test
environments,
and
I
think
these
these
things
need
to
be
taken.
Care
of.
B
This
is
why
there's
the
overlap
and
and
they
work
hand
in
hand,
but
I
think
they
just
have
maybe
different
backgrounds
and
different
different
experiences
on
what
what
does
it
take
to
get
stuff
into
production
and
what
does
it
take
to
keep
it
in
production
safe?
But
in
the
end,
you
know
they
have
to
work
together.
A
Yeah,
that's
for
sure,
so
you
started
mentioning
slis
and
slos
and
other.
So,
let's
talk
in
the
title
of
the
episode
that
I
chose
is
centered
around
the
slo,
the
service
level
objective,
and
I
I
find
that
many
treat
it
as
a
technical.
Let's
call
it
sre
metric,
but
it
ties
directly
to
the
business
objectives
of
the
company,
at
least,
if
you,
if
you,
if
you
do
it
right
so
can
you
tell
us
what's
good
slo?
As
far
as
you
see
it.
B
I
wanna
start
with:
what's
the
peres,
hello
and
badass
hello
is,
and
I
have
to
I
had
it.
Somebody
challenged
me
on
twitter
two
weeks
ago,
when
I
posted
a
video
on
kind
of
seven
slos
to
start
with,
and
then
this
person
responded-
and
I
know
him
personally
and
he
sometimes
has
an
interesting
attitude
on
social
media.
He
basically
said:
stop
stop
this
bs,
stop
educating
people
about
slos,
because
it
doesn't
scale
right,
defining
an
slo
in
every
single
service.
B
B
You
need
to
figure
out
what
is
your
business
goal
and
that's
exactly
what
we
said
earlier
right.
I
mean
if
I
come
back
to
the
story
with
kelsey
hightower.
The
business
goal
is
that
these
systems
are
always
on
that.
These
systems
needs
to
provide
a
certain
performance
because
if
they
don't
perform,
if
it,
if
I
swipe
the
card-
and
it
takes
five
seconds
10
seconds
eventually-
I
will
have
a
backup
in
people
that
are
standing
in
line.
It
doesn't
scale
anymore.
Then
the
supermarkets
need
to
figure
out.
B
You
know
how
to
scale
the
alliance
or
they
go
to
a
different
vendor.
So
I
think
availability
is
a
big
thing.
Performance
is
a
big
thing
and
also
accuracy
right.
So
if
I
swipe
the
card
and
the
card
is
valid,
then
I
should
get
a
hundred
percent
valid
responses
and
it
shouldn't
just
fail
at
some
point,
but
I
think
what's
what
a
good
slos
to
start
with
is
really
figuring
out.
B
What's
the
business
value
that
your
service
provides
either
to
an
end
user
or
to
a
consumer
of
your
api,
because
not
everything
is
customer
facing
some,
some
services
are
obviously
consumed
by
other
services
and-
and
I
think
these
are
good
slis
to
use.
Then
you
define
your
objective
like
always
available
a
hundred
percent.
If
that's
possible
response
time,
let's
say
has
to
respond
within
50
milliseconds
or
100,
milliseconds
and
accuracy.
B
It
needs
to
be
it
needs
to
respond,
successful
responses
to
successful
queries
and
then
the
next
thing
is,
you
need
to
break
it
down.
What
does
this
mean?
And-
and
this
is
where
I
disagree
a
little
bit-
I
think,
with
the
discussion
that
the
two
of
us
had
on
linkedin,
where
you
said
slos,
don't
need
to
be
defined
on
every
service.
I
agree,
but
you.
B
It's
like
breaking
down,
it's
breaking
down
an
slo
into
its
individual
pieces
in
performance
engineering,
because
this
is
where
worked
a
lot.
We
call
this
performance
budget
to
break
down
a
budget.
We
have
on
the
top
into
individual
buckets
and
then
every
team
has
a
certain
bandwidth
where
they
operate
in,
and
I
think
this
is
where,
where
slos
are
great
to
to
define
the
slos
on
the
individual
api
interfaces.
A
I
definitely
agree-
and
I
think
what
you
said
in
terms
of
the
the
key
slos
and
concentrating
on
on
performance
on
accuracy
on
availability.
These
are
you
know.
Basic
tenets
of
performance
engineering
have
been
even
before
establishing
the
sre
practices.
I
think
I
remember
them
ten
plus
years
ago,
but
definitely
our
core
tenants
for
for
sre.
You
mentioned
my
my
article,
my
blog
post
on
sre,
in
the
micro
service
era.
A
I
think
my
point
there
was
was
very
much
around
and
I
think
by
the
way,
the
conversation
with
the
gentleman
from
google
sre
was
very
much
aligned
with
that
is
that
people
have
grown.
Sres
have
grown
accustomed
to
monitoring
the
the
binaries
that
they
produce
and
with
more
monolithic
applications
where
the
binary
is
is
a
significant
chunk
of
the
application,
if
not
the
entire
application,
or
at
least
the
entire
tier.
I
don't
know
all
the
business
tier
from
a
three-tier
application,
or
something
like
that.
A
The
part
that
is
important,
ultimately,
is
the
the
user
experience
and
the
user's
flows
and
what
the
user
ultimately
is.
It
impacts
the
the
ultimate
user
experience
and
that
should
derive
again
in
the
monolithic
era.
It
was
one-to-one
because
the
the
binary
also
represented
the
full
flows.
Typically
now,
I
think
people
need
to
step
back.
A
Look
go
back
to
the
architecture
diagram
that
you
mentioned
before
in
the
design
phase
derive
the
slos
from
the
end
to
end
user
flows
and
then
start
seeing
which
slos
serve
to
measure
that,
and
I
agree
with
you,
you
need
to
break
down
different
steps
because
you
need
to
attribute
the
the
latency
or
the
performance
to
each
and
every
step
of
the
way,
but
in
a
way
that
is
aggregated.
That
is
manageable
and
definitely
not
that
can
tie
back
directly
into
the
into
the
main
floor.
A
B
B
This
was
easier
to
break
down
because
then
you
said
certain
things
in
the
front
end.
When
I'm
loading
a
page,
I
know
they
have
my
images
and
my
javascript
classical
steve
sauders.
If
you
remember
him
from
web
performance
optimization,
he
had
the
80
20
rule
and
he
said
80
of
the
stuff
happens
in
the
front
and
in
the
browser
for
certain
transactions.
It's
the
other
way
around,
because
the
heavy
lifting
is
in
the
back
end.
But
in
the
end
it's
front
end
then
there's
some
business
logics
running
somewhere
and
that's
the
database.
B
And
now
obviously
we
have.
If
you
look
at
this,
we
have
the
front
end.
We
have
a
lot
of
middle
pieces,
a
lot
of
different
services
that
are
kind
of
aligned
with
individual
business
use
cases,
and
then
they
call
more
back-end
services.
But
in
the
end
we
still
have
a
nice
layering,
and
we
just
I
think
we
can
use
the
power
of
slos
to
enforce
our
budgets
and
that's
really
what
what
this
is
all
about.
B
B
A
I
think
you
gave
just
to
make
things
a
bit
less
abstract
to
the
users
yeah
you
gave
in
our
earlier
chat
a
very
good
example
of
a
customer
that
you
escorted,
that
did
the
very
nice
breakdown,
with
your
guidance
on
from
the
high
level
business
objectives,
all
the
way
down
to
the
technical
sre
metrics.
Can
you
maybe
share
some
of
that
as
an
example
process.
B
Yeah,
that's
a
good
one,
and
and
thanks
for
reminding
me
about
this
discussion
because
I
wasn't
in
a
similar
situation,
they
came
to
me
and
said
help
us
define
our
hundreds
of
slos
and
I
said
well,
let's
first
start
with
what's
your
business
objective
and
that
customer
was
a
financial
institution.
So
what
do
they
have?
B
They
have
a
mobile
app
and
they
actually
had
their
old
financial
mobile
app
and
now
they
have
a
new
mobile
app
and
then
I
said
who
owns
this
app
and
then
somebody
raised
their
hand-
and
I
said,
what's
your
goal
with
this
app
and
I
said:
well,
we
need
to
get
adoption
of
the
new
mobile
app
because
that's
our
future,
where
we
are
competitive
with
our
competitors.
We
need
to
get
the
adoption
of
this
app
up
to
75
by
the
end
of
the
year.
I
said:
that's
a
great
business
objective.
B
So
then
I
said
so
basically
the
next
goal
is:
you
need
to
get
the
rating
up,
because
it's
it's
obviously
not
nice,
to
look
at
the
two-star
rating,
and
then
he
said
yes,
that
would
be
a
good
objective.
We
want
to
get
to
four
and
a
half
star,
which
kind
of
is
like
90
right,
because
in
slos
we
always
try
to
talk
about
percentage
numbers,
and
then
I
said
what
are
the
most
complaints
about
people
from
people
and
they
said
it's
crashes.
B
I
said
so.
That
means
this
is
your
first
kind
of
link
from
your
business
objectives
to
your
technical
metrics.
You
need
to
get
these
crashes
down,
which
means
you
have
bad
quality
right
now.
So
the
objective
to
the
development
team
should
be
crash
rates
on
supported
devices,
which
is
also
very
important
right,
because,
if
you
just
say
crash
rate
crashes
can
be
something
different
for
me
than
for
you,
and
then
we
said
okay,
so
we
have
the
kind
of
the
end
user
perspective,
already
three
metrics
adoption
or
adoption
of
the
new
app.
B
So
how
many
users
are
moving
from
the
old
to
the
new
one?
The
app
rating
on
the
store
and
then
the
crashes,
and
then
we
said
now
behind
the
mobile
app
behind
every
mobile,
app,
there's
somewhere
back
in
service,
and
we
said
the
first
service
that
you
interact
with
is
the
authentication
service
that
authenticates
you
as
a
user,
and
if
I
want
to
use
this
app,
I
want
to
log
in,
and
I
want
to
as
fast
as
possible
say.
Yes,
I'm
loading
your
account
details
or
transactions.
B
So
we
then
said:
okay
from
a
technical
level,
for
the
authentication
service,
we
need
high
availability,
because
otherwise
your
mobile
app
will
respond
back
with
an
error
to
the
user.
Sorry
cannot
log
you
in
right
now,
not
good.
It
also
has
to
be
fast,
so
performance
is
important
and
it
needs
to
be
successful
in
handling
the
request.
That
means,
if
I'm
logging
in
with
the
right,
username
and
password,
then
I
should
be
logged
in
and
therefore
coming
back
to
the
example
earlier.
B
A
I
I
really
like
the
example,
although
I
hear
it
a
second
time
around
and
I
still
enjoy
it
because
I
think
it
really
clearly
outlines.
What's
how
slos,
I
think
should
be
addressed.
So
it's
an
excellent
example-
and
I
think
you
also
one
of
the
things
that
I
find
with
slos,
that
is
very
useful-
is
to
manage
the
the
the
let's
say,
the
tension
between
the
different
roles
and
personas,
because
you
have
the
you
have
the
developer.
A
Obviously
you
have
the
the
product,
you
have
the
sre
and
it's
it's
the
legitimate
areas
of
contention
between
them.
But
when
you
govern
that
with
slos,
it's
easy
to
regulate
it's
easy
to
balance
the
cost
versus
the
reliability
versus
the
velocity.
You
know
sre1
stability
just
for
an
example,
the
more
stable
it
is,
the
easier
their
job
is.
But
then
again,
developers
want
to
release
and
product
want
to
release
innovation
faster
with
higher
velocity,
more
more
value
and
so
and
then
how
do
you
balance
that?
B
From
my
experience
exactly
and
it's
the
same
thing
with
I'm
not
sure
if
you're
familiar
with
the
concept
of
okrs
objective
key
results
or,
however,
you
call
it
in
your
organization,
it's
the
same
thing.
You
have
a
certain
goal
in
your
organization
right,
you
want
to
become
the
number
one
in
your
market
or
let's
say
you
want
to
get
a
thousand
paying
customers
by
the
end
of
the
year
and
then
you
break
it
down.
And
what
does
this
really
mean?
B
How
can
you
get
there
and
then
you
can
figure
out
how
you,
as
in
my
case,
andy
grabner?
How
can
I
contribute
to
this,
even
though
I'm
just
somewhere
three
levels
down
the
hierarchy,
but
I
know
that
if
I'm
writing
an
authentication
service
that
works
and
is
available-
and
I
talk
with
the
sre
team
so
that
they
can
give
me
some
insights
on
how
I
can
make
this
more
robust
and
more
resilient.
Then
I
can
build
something
that
contributes
to
the
overall
goal
and
that's
really
what
it
is.
B
A
A
Obviously
to
everyone,
but
let's
say
product,
for
example,
is
naturally
more
exposed
to
the
high
end
goals
and
others,
but
there's
no
engineering
that
usually
many
organizations
treat
it
as
a
back-end
function
in
terms
of
exposure
to
the
to
the
high-level
goals.
I
think
that
that
is
for
me.
It's
it's
an
anti-pattern.
I
think
that
actually
relaying,
that
transparently
gets
them
very
eager
and
very
very
adamant
in
getting
these
features
done
and
also
focusing
on
the
right
areas,
because
engineers
like
perfection
in
every
point.
A
But
if
you
focus
on
the
impact
on
each
in
each
point,
then
they
can
focus
the
efforts
on
their
in
the
places
that
matter
most.
So
so
that's
that's
interesting
and
maybe
another
question
related
to
that
is
so
who
owns
slos,
who
and
who's
in
charge
of
the
monitor
and
who's
in
charge
of
the
remediation.
B
Coming
back
to
the
example
I
mentioned
earlier
from
this
financial
institution,
I
think
owning
owning
means
who,
who
is
who
has
an
interest
in
getting
this
metric
and
bringing
it
to
the
right
value?
So
what
I
mentioned
earlier,
the
adoption
is
a
business
metric
business
owner.
The
is
it
available
is
a
metric
that
site
reliability.
B
Engineers
can
impact
as
well
as
development,
can
impact,
because
why
sres
can
make
sure
they
are
provisioning,
the
right
failover
strategy
they
are
making
sure
that
their
other
remediation
is
put
in
right
infrastructure
is
there
to
actually
support
the
load.
On
the
other
side,
the
engineers
can
obviously
work
on
the
on
the
aspects
of
quality
so
that
the
system
is
just
by
default
available
or
at
least
higher
available.
B
B
In
my
perspective,
the
people
I
work
with
with
the
the
s
the
sre
function
within
an
organization
because
sres
right,
a
lot
of
their
work,
is
around
automating
stuff
based
on
observability
data
and
the
slis.
The
indicators
are
coming
from
your
prometheus
from
your
apm
solution,
whatever
it
is
right
from
your
locking
from
from
from
your
logging
system,
I
mean
so
I
I
see
a
lot
of
organization,
it's
the
sre
role,
whether
this
is
a
dedicated
sre
team
or
whether
this
is
part
of
another
observability
team.
B
But
it's
part
of
it
depends
on
the
organization,
but
I
think
they
are
responsible
to
provide
this,
and
then
they
need
to
enable
the
individual
stakeholders
to
a
define
their
sli's.
What
is
the
metric
that
they
want
help
them
to
get
it
to
get
this
metric,
define
it
as
slo
the
objective
and
then
also
what
should
the
action
be
in
case?
B
This
slo
is
not
met,
but
I
think
the
sre
team
is
then
the
enabler
to
the
development
teams
to
the
business
teams
to
to
let
them
define
this
in
a
very
easy
way
and
easy
means
for
developers.
It
could
be
a
yaml
file
somewhere
in
a
git
repository
for
the
business.
It
might
be
a
point-and-click
solution
because
they
might-
and
I
don't
want
to
say
the
business
is
not
technically
capable
of
doing
gambles.
But
I
think
you
want
to
make
it
appealing
to
the
to
the
stakeholder
and
how
to
define
and
maintain
these
slices.
A
B
So
you
can
declare
what
you
want
to
be
executed
for
a
particular
problem
and
then
we
take
care
of
it.
Now
who
is
responsible
for
this?
I
think
the
sre
would
be
the
one
that
brings
in
this
type
of
automated
remediation
into
the
platform,
the
user
of
it.
Like
the
definition
of
these
run
books,
we
call
them
automation,
sequences
or
remediation
sequences.
B
Some
of
them
will
be
done
by
the
s3
right
because
you
want
there's
certain
things
from
the
infrastructure
perspective
like
spinning
up
a
new
node
scaling
up,
changing
load,
balancing
routes
and
stuff
like
this
is
something
that
the
ns3
will
bring
to
the
table.
On
the
other
side,
a
development
team
may
define
their
own
rules
and
say:
hey
we're
running
some
experiments.
B
We
have
a
new
feature
flag
and
if
we
turn
the
feature
flag
on
and
then
we
are
breaking
our
slos
with
that,
we
want
the
order
remediation
to
just
switch
the
feature
flag
off,
and
I
want
this
to
happen
automatically,
and
I
don't
want
to
be
woken
up
at
two
o'clock
in
the
morning.
So
this
is
why
I
think
the
what
I
see
is
the
it's.
A
B
That's
a
good
question,
I
think
in
every
in
any
organization,
mature
or
knowledge,
not
mature.
I
assume
you
have
somebody
that
is
responsible
for
your
infrastructure,
that
is
responsible
for
keeping
your
digital
services
up
and
running,
and-
and
I
think
it
would
be-
they
would
be
benefiting
the
most
if
they
would
invest
in
something
like
observability
in
measuring
your
slos
and
then
also
investing
in
all
the
remediation,
because
I
guess
in
less
mature
organization.
This
right
now
means
there's
a
lot
of
things
that
happens
manually
because
there's
less
automation
space
in
in
in
in
place.
B
B
Do
you
maybe
need
to
hire
new
people
that
you
bring
in
that
already
have
this
expertise
and
with
hiring
I
mean
either
hiring
people
that
didn't
work
for
you
and
there's
also
no
shame
in
just
saying
hey.
I
need
an
external
help
from
from
a
consulting
company
that
gets
me
started
and
and
then
you
figure
out
if
this
new
role
then
fits
in
your
current
team
structure
or
whether
at
some
point
it's
time
to
to
have
a
dedicated
history
team
that,
then,
you
know,
takes
care
of
observabilities.
A
Sounds
good,
thank
you
for
the
question
and
to
the
audience
on
the
live
stream
on
youtube
and
twitch.
If
you
have
more
questions,
feel
free
to
type
them
in
the
chat
and
we'll
try
to
address
them.
So
thanks
all
for
the
question
and
going
back
to
you
mentioned
captain
the
the
new
project.
Essentially
it's
a
project
that
was
developed
by
at
dynatrace
right
as
and
then
it
was
open
sourced
and
they
contributed
to
the
cncf.
A
A
B
So
the
the
motivation
was
was
interesting
because
what
we
have
done
over
the
years
we
always
try
to
advocate
to
our
customers.
How
can
you
bake
observability
into
your
current
delivery
process
and
into
your
current
remediation
process
and
before
we
started
with
captain
right,
we
always
showed
them
hey.
What
do
you
have?
B
Oh,
you
have
jenkins
and
you're
using
this
test
tool,
and
then
you
have
ansible
for
deployment
and
for
configuration
management,
and
so
we
said,
okay,
you
know
we
can
show
me
your
jenkins
pipeline
and
now
we're
building
in
your
changes
pipeline,
not
only
the
deployment
after
the
deployment.
We
do
some
sanity
checks
reaching
out
to
observability
platform.
Then
we
call
your
testing
tool.
B
Chain,
deployment,
testing,
evaluation,
configuration
change
and,
in
the
middle,
also
reach
out
to
observability
platform,
to
then
make
better
data-driven
decisions,
and
this
is
where
captain
came
about.
We
said
we
understand
that
there
are
certain
processes
that
organizations
have
deployment
tests
evaluate
promote,
for
instance,
but
they
may
have
different
tools
right
in
there.
In
this
case
they
have
jenkins
for
or
they
use
jenkins
and
the
other
one
they
use
helm.
B
So
what
we
did
we
separated
process
from
tooling
in
captain
you
can
define
a
process,
we
call
them
automation,
sequences
where
you
just
say
I
want
deploy
test,
evaluate,
promote
and
then
we
have
an
event
driven
model,
which
means
you
can
have
one
or
multiple
tools
subscribing
to
one
or
multiple
events,
and
captain
is
then
basically
the
event
driven
orchestrator
just
to
enforce
the
process
and
is
connecting
to
your
tools
through
loosely
coupled
events
and
between
the
individual
steps.
We
also
connect
to
your
observability
platform
to
also
ask
for
hey
observability
platform.
B
B
Exactly
it
allows
it
allows
two
things
it
allows.
It
allows
skating
as
part
of
your
devops
delivery
process.
Remember
I
talked
about
what
is
devops
to
devops,
provide
develop
platform,
engineers
kind
of
provide
the
mechanisms
to
get
stuff
into
production,
but
you
can
use
then
this
data
to
as
gates
and
for
sres,
for
your
order,
remediation
use,
slos
to
figure
out
hey,
I'm
in
run
book
step
number
one
or
we
call
them
remediation
sequence
steps.
We
execute
that
first
step.
The
system
is
still
red,
it's
execute
the
next
step.
It
is
still
red.
B
Let's
execute
the
third
step.
Now
the
system
is
back
to
a
healthy
state,
so
that
means
on
the
one
side
we
use
it
for
gating.
On
the
other
step,
we
use
it
for
validating
and
also
steering
the
remediation
process,
because,
just
as
a
human
being
would
do
right,
you
have
your
run
book
and
you
say:
click
this
button
restart
and
then
watch
your
dashboards
if
it
goes
green
after
five
minutes
enough.
Now
it
didn't
solve
the
problem.
B
Let's
do
the
next
step
and
the
last
step
might
be
call
the
engineers
at
two
o'clock
in
the
morning,
and
so
we
automate
the
whole
thing
as
well,
and
therefore
we
use
slis
and
slos,
as
as
kind
of
for
your
release.
Validation
for
your
quality,
for
your
gating
is
releases
shift
into
prod,
and
then
we
use
them
to
really
control
and
and
and
observe
the
the
impact.
Hopefully,
the
positive
impact
of
your
remediation
strategy.
B
I
mean
you
can
fortunately
it's
it's,
it's
all
public.
We
have
a
the
majority
to
be
honest
with
you
right.
I
think
we
have
about
80
or
90
percent
of
the
co-maintainers
from
dynasties,
because
we
just
started,
but
we
have
people
from
microsoft
with
people
from
citrix.
We
have
people
from
real
dolmen
from
neotus,
so
that
I
think
you
know
the
cncf
stats
that
you
can
see
who
is
contributing
to
it,
but
on
the
one
side
it's
contributing
and
the
other
ones
are
users.
B
They
just
went
on
the
webinar
with
me
recently
and
they
showed
how
they
are
using
captain
as
part
of
their
release,
validation
process,
so
exactly
using
that
these
slis
and
slos
after
every
new
release
that
is
getting
pushed
through
their
existing
pipeline
and
kepnes,
then
pulling
in
the
data
in
their
case
on
from
dynatrace
and
say
good
release
to
go
all
the
way
to
production
for
all
the
australians
out.
There
were
not
good.
B
So
that's
a
great
story.
Other
stories
are
from
volusion.
I
remember
them
very
vividly
as
well,
because
they
also
were
in
the
webinar
with
me
recently.
They
are.
They
started
with
captain
and
and
to
automate
the
complete
end-to-end
delivery
workflow
with
blue-green
and
canary
deployments,
and
the
strength
of
captain
here
is
that
they
can
easily
decide
because
it's
an
event-driven
model
which
tool
they
want
to
use
for
the
delivery
piece.
A
And
a
platform
such
as
that
again,
the
richness
of
integrations
obviously
is
a
key
and-
and
I
know
that
you
have
quite
a
rich
suite
of
integrations-
so
that's
that's.
I
think
something
that
will
promote
the
project
very
much
and
another
point
that
is
related
and
very
relevant.
Is
your
work
and
the
work
under
the
the
continuous
delivery
foundation,
the
cdf
congratulations
for
being
elected
as
a
as
an
ambassador
in
the
cdf
for
2022.
A
B
B
You
will
find
tools
like
jenkins
that
have
been
instrumental
over
the
last
years
in
in
really
bringing
continuous
delivery
to
where
it
is
today,
and
it's
really
a
group
that
wants
to
drive
certain
standards
forward,
and
this
is
where,
where,
with
the
interest
from
our
side,
also
comes
in
the
coming
back
to
the
eventing
model
that
I
mentioned
earlier
right,
we
didn't
want
to
just
come
up
with
our
own
events,
because
then
it's
again
proprietary.
What
we
wanted
to
do.
B
The
great
thing
is:
if
we
all
agree
on
a
standard,
then
it
will
be
less
painful,
zero
painful,
hopefully
in
the
future,
to
integrate
a
tool
into
your
orchestration,
because,
right
now,
every
tool
has
their
own
api
on
how
you
call
it
and
has
their
own
way.
You
call
it
and
the
result
that
comes
back
now
what
we
are
doing
with
captain.
B
We
have
brought
up
our
initial
event
standard
and
for
the
first
tool
integrations,
we
have
built
kind
of
our
own
kind
of
proxy
that
takes
our
event
and
then
converts
it
into
a
call
of
that
target
tools,
api
and
waits
for
it's
done.
We
call
this
a
captain
service.
Now
this
is
a
nice
way.
It
works.
We
also
have
a
way
to
easily
call
external
tools
by
just
specifying
apis
through
a
web
book
service.
B
However,
where
we
want
to
get
to
is
we
just
want
to
make
sure
that
the
tools
in
the
future
deployment,
testing,
evaluation,
monitoring
tools,
notification
tools,
they
natively
support
the
cd
event
standard,
which
means
in
the
future.
If
I
am
signing
up
for
a
new
tool
and
this
tool
supports
that
standard,
I
can
just
say:
hey
youtube.
You
are
now
subscribing
to
your
favorite.
Orchestrator
could
be
captain,
maybe
in
the
future
some
something
else
also
comes
along,
but
now
you're
connecting
you're,
subscribing
and
and
there's
no
additional
overhead.
A
A
Exciting
also
posted
on
the
on
the
chat
here
for
our
viewers
on
twitch
and
youtube,
the
link
to
the
github
page
for
the
cdf
and
specifically
the
sig
events
very
interesting
activity,
and
I
I'm
a
I'm
a
strong
believer
in
in
open
standards.
I
think
this
is
the
way
to
converge,
especially
in
things
such
as
that
that
are
orchestrators
by
by
nature.
A
B
A
To
converge
and
be
able
to
create,
I
think
it's
it's
been
seen
very
very
nicely,
for
example,
with
metrics,
with
open,
metrics
and
prometheus.
Having
like
a
standard
in
which
you
know,
it's
been
exposed,
the
exposition
format
and
then
it's
very
easy
to
plug
in
prometheus
compatible
frameworks
to
just
pull
the
scrape
the
metrics
and
present
them,
and
it
creates
decoupling
and
and
standardization.
I
think
something
very,
very
nice
can
be
similarly
here
with
the
ci
cd
tools
and
similar
the
last
point
before
we
run
out
of
time.
A
B
Yeah
thanks
for
allowing
me
to
make
some
ads
here,
so
pure
performance
spelled
as
one
word
pure
performance.
You
can
find
it
on
your
favorite
podcast
app.
It
was.
We
started
it.
I
think,
three
or
four
years
ago
now,
with
one
of
my
colleagues,
we
are
both
performance
engineers
by
heart,
brian
wilson.
B
He
was
actually
a
performance
engineer
in
the
previous
company
before
he
joined
dynabase,
and
I
have
performance
also
in
my
heart,
because
before
dynamesh
I
was
also
working
for
a
performance
testing
company,
and
we
said
you
know
we
want
to
explore
the
new
option
of
podcasting.
It
is
new
for
us
to
educate
people
on
what
are
common
performance
problems.
What
what
do
people
run
into?
Why
do
systems
fail,
and
what
can
we
learn
from
it
and
what
can
we
share?
B
It
was
also
very
much
inspired
by
a
couple
of
folks
that
we've
known
over
many
years,
it's
the
guys
from
perth
bytes,
so
there's
another
podcast
called
perf
bytes
they've
been
doing
podcasting
longer
than
we
have
and
they
also
had.
You
know
lots
of
great
stories
and
at
some
point
they
we
met
them
at
a
conference
and
then
they
said
hey.
Why?
Don't
you
just
do
your
own
podcast
and
focus
on
your
performance?
B
We
had
gene
kim,
who
is,
I
think,
a
known
entity
when
it
comes
to
devops.
We
had
him
on
the
show
two
or
three
times
talking
about
the
the
phoenix
project
and
the
devops
handbook
and
and
so
on,
and
it
was
really
and
one
one
that
I
was
more
recently
was
with
an
nana
from
tech
world
with
nana
she's
kind
of
the
a
youtube
star
these
days,
because
she's
been
since
the
pandemic
started.
I
created
a
lot
of
great
educational
content
on
cloud
native
and
observability
on
ci
cd
on
youtube.
B
B
A
Sounds
great,
I
will
definitely
come
pass
by
the
booth
at
cubecon,
looking
forward
to
it
in
may
another,
by
the
way,
a
very
interesting
event
the
same
month
earlier
that
month
in
may
is
slo
conf,
which
is
also
very
interesting
and
very,
very
focused
on
this
area
that
we
talked
about
extensively
today.
The
slo
service
level
objectives
going
to
be
there
also
online.
I
know
that
you're
going
to
be
there
so.
B
B
A
A
It
was
released
actually
last
week
march
8th.
I
think,
if
I'm
not
mistaken
and
actually
found
it
quite
insightful,
I'm
just
looking
quickly
to
share
here
with
the
ones
on
the
stream,
although
of
course,
we'll
show
for
the
ones
that
are
on
the
links
and
on
the
audio.
Only
we'll
definitely
share
the
rest
of
that.
But
what
I
wanted
to
note
several
interesting
observations
from
that
one
is
about
the
platforms
themselves,
as
you
can
see
the
popular
projects
that
are
being
used
for
observability.
A
I
wasn't
a
too
much
of
a
surprise
for
me,
but
I
think
it's
a
good
validation.
We
can
see
here
the
prometheus
leading
the
chart
with
86
percent
and
following
it
open,
telemetry
and
fluent
d
and
jager.
All
of
these
are
are
actually
also
a
project
that
we've
been
discussing
here
at
open,
observability
talks,
podcast
very
interesting
to
see,
and
also
we
can
see
tunnels
that
we've
been
discussing
and
and
open
tracing
that
you
know
merged
into
open,
telemetry
and
cortex.
So
it
all
comes
down
to.
A
B
I'm
not
surprised
at
all,
because
this
is
my
main
in
my
in
my
day,
work
and
both
of
us
while
we
are
in
the
observability
space
in
my
day,
work.
I
see
exactly
these
these
frameworks
and
these
projects
showing
up,
and
this
is
why
it's
also
so
important
that
these
are
supported.
Also
by
vendors
like
us,
and
I
can
talk
about,
I
guess
for
you
as
well.
You
are
supporting
and
then
probably
also
contributing
engineering
resources.
We
do
the
same
from
adan
from
a
dyna
trace
perspective.
A
That
definitely
and
and
other
interesting
insights
from
that
that
I
found
at
least
is
beyond
the
best
of
breed
stock-
is
about
the
challenges
which
is
usually
what
is
more
interesting
to
look
forward
so
maybe
going.
This
is
specifically
about
the
projects,
but
even
before
the
projects
about
challenges
in
observability,
I
think
the
number
one
is
about
the
number
of
engineers
and
teams
that
are
using
multiple
tools.
This
this
sprawl,
as
I
call
it
is,
is
a
really
recurrent
pain.
This
is
actually
some
that
have
been
last
week.
A
I
I
was
at
the
devops
conference
and
I
during
my
talk
there,
I
also
posted
the
poll
to
the
audience,
and
it
came
out
with
this
very
same
result
about
the
high
number
of
of
two
that
are
being
used
for
for
fetching
telemetry
for
analyzing
for
storing.
I
think
this
is
a
significant
problem
that
keeps
on
popping
up
in
the
in
our
ecosystem
and
another
one
we
talked
about
before
about
this
projects
themselves
and
what
challenges
people
have
with
with
the
projects
themselves.
A
We
see
number
one
is
the
complexity
just
plain
too
complex
to
understand
or
run,
which
is
still
a
challenge,
all
the
projects
I
think
are
facing
that
I
see
that
in
open
telemetry.
I
see
that
with
with
prometheus
and
many
others,
a
lot
of
work
is
being
done
to
try
and
make
it
more
accessible,
but
still
I
think
this
is
something
that
we
as
a
community
need
to
need
to
look
into
that,
and
we
see
also
you
know,
lack
of
supporting
documentation
which
obviously
feeds
into
that
complexity
and
similar
related
topics.
A
Another
point
that
I
found
interesting
is
is
a
question
that
was
asked
about
what
what
prevents
what
concerns
prevent
people
from
adopting
or
or
extending
adoption,
and
I
saw
the
integration
as
as
I
highlighted
it
here,
it's
number
one
again
because
of
so
many
tools
and
so
many
systems
we
talked
about
about
it
a
bit
also
in
the
context
of
captain.
These
needs
to
address
the
different.
A
B
I
think
you
just
mentioned
it
right.
We're
really
trying
one
of
the
things
we're
solving
is
exactly
this
here,
because
we
are
the
integration
hub,
a
standardized
integration
up
between
all
of
your
tools
and
your
observability
and
kind
of
connecting
all
of
your
tools,
basically
to
the
backbone,
which
is
your
observability,
and
if
people
are
having
challenges
with
the
complexity
of
of
of
the
current
systems,
the
tool
sprawl,
I
mean
this
is
exactly
what
what
we
are
solving
with
captain
or
trying
to
solve
with
captain
here.
A
Yeah
so
very
interesting,
I
will
share
the
link
to
the
survey
to
the
micro
survey
also
with
with
the
forum,
so
you
can,
I
definitely
recommend
checking
it
out.
I
don't
think
we
have
too
much
time
for
additional
news
and
breaks,
but
this
is
definitely
something
that
I
wanted
to
spend
some
time
on.
So
with
that
I'd
like
to
thank
you
again,
andy
for
for
joining
me.
Okay,.
B
A
I'd
like
to
thank
all
the
audience
for
joining
us
here
on
the
on
the
live
stream
and
on
the
on
the
podcast.
As
always,
all
the
episodes
are
available
on
the
on
your
favorite
podcast
apps
or
on
youtube,
so
you
can
find
on
the
videocast
format
as
well.
If
you
are
listening
to
this
on,
a
podcast
app
do
know
that
we
also
stream
the
episodes
live
on
twitch
and
youtube.
A
So
do
follow
us
to
see
when
the
next
live
stream
is
and
then
you
can
join
and
ask
questions
and
put
your
comments
and
get
more
engaged
with
the
conversation.
You
can
find
all
the
details
at
openobservability.io
and
you
can
also
follow
us
on
twitter
at
openobserve
or
me
directly
at
horowitz
h-o-r-o-v-I-t-s.