►
From YouTube: Using SLOs for Continuous Performance Optimizations of Your K8s Workloads - Andreas Grabner
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Using SLOs for Continuous Performance Optimizations of Your K8s Workloads - Andreas Grabner, Dynatrace
Moving to k8s doesn’t prevent anyone from bad architectural decisions leading to performance degradations, scalability issues or violating your SLOs in production. In fact – building smaller services running in pods connected through service meshes are even more vulnerable to bad architectural or implementation choices. To avoid any bad deployments, the CNCF project Keptn provides automated SLO-based Performance Analysis as part of your CD process. Keptn automatically detects architectural and deployment changes that have a negative impact to performance and scalability. It uses SLOs (Service Level Objectives) to ensure your services always meet your objectives. The Keptn team has also put out SLO best practices to identify well known performance patterns that have been identified over the years analyzing hundreds of distributed software architectures deployed on k8s. Join this session and learn what these patterns are and how Keptn helps you prevent them from entering production.
A
This
already
tells
you
that
I'm
not
a
comedian,
otherwise
I
would
be
paid
for
better
jokes.
My
name
is
andy
grabner
or
andreas
carmen,
but
please
call
me
andy.
It's
always
easier.
I
always
say
that
only
my
mom
calls
me
andreas.
If
I
did
something
rude,
my
I'm
originally
from
austria.
I
actually
live
in
austria,
so
I
also
want
to
say
thanks
for
actually
allowing
me
to
come
here,
and
I
know
a
lot
of
international
speakers
that
couldn't
make
it
to
kubecon.
A
I
was
one
of
the
lucky
ones
that
got
an
exception
to
travel
here.
I've
been
in
performance
engineering
for
the
last
20
years
and
I
want
to
share
a
couple
of
things
that
I've
learned
in
my
life
as
a
performance
engineer,
especially
around
application
performance
around
distributed
performance,
but
not
about
just
like
how
do
you
do
performance,
testing
or
performance
engineering,
but
really
how
we
automate
all
of
this-
and
this
is
where
captain
comes
into
play
right.
Captain
is
a
cncf
sandbox
project.
We
are
hopefully
soon
reaching
incubator
status
and
yeah.
A
I
want
to
share
with
you
what
I've
learned
and
how
I
hope
I
can
make
your
life
easier
in
case
you're,
really
interested
in
building
and
optimizing
or
building
better
performing
systems
and
optimizing
your
systems
all
the
links
and
how
you
reach
me
and
how
you
can
follow
up
on
captain
on
the
slide
now.
One
thing
that
I've
learned
over
my
last
20
years
is
when
I
do
performance
analysis.
A
I
always
look
at
performance
patterns
and
I
want
to
start
with
this
before
I
go
into
the
core
part
of
this
talk,
which
is
how
we
can
automate
performance
analysis
based
on
slo
service
level
objectives.
Why
are
performance
pattern
patterns
important?
Because
this
is
what
I
do
in
my
life
when
I
analyze
application
performance?
I
typically
look
at
things
like
distributed
traces
to
understand
the
architecture,
especially
in
distributed
systems.
A
Hopefully
I
mean
for
the
people
in
the
room
distributed.
Traces
is
something
we
are
hopefully
all
aware
of
who
is
looking
at
distributed?
Traces?
Yes,
so
this
is
actually
a
small
distributed
trace.
It
starts
with
the
front
end
load
balancer
with
some
legacy
systems,
some
micro
services,
some
databases.
Some
third
party,
some
load,
balancers
and
so
on.
This
is
a
from
a
company
in
germany
called
step
stone.
A
I
did
a
presentation
with
them
a
couple
of
years
ago,
where
they
talked
about
how
they
moved
to
a
container-based
environment
and
how
kind
of
their
architecture
changed,
and
the
reason
why
this
is
important
is
you
have
to
understand
your
architecture?
You
have
to
understand
how
components
talk
to
each
other
to
identify
very
important
patterns
that
tell
you
if
this
architecture
can
actually
scale
or
not.
A
Now
the
number
one
pattern
that
I
always
find-
and
this
is
nothing
new,
but
I
have
to
repeat
it
because
I
see
it
every
day.
That's
the
n,
plus
one
query
pattern.
What
you
see
here
is
a
distributed,
trace
a
single
distributor
trace
from
one
of
my
friends
who,
15
years
ago,
he
started
his
own
company
building,
an
e-commerce
store,
very
small
for
very
small
shops
over
the
years
it's
scaled
and
scaled.
Now,
his
e-commerce
shop
is
very
popular
in
in
europe.
Unfortunately,
when
you
look
at
this
distributed
trace,
it
shows
this
very
interesting
pattern.
A
This
is
a
trace
for
the
search
feature
of
his
e-commerce
platform
when
you
search
what
his
initial
implementation
did,
give
me
the
number
of
product
categories
and
then
he's
iterating
through
all
the
product
categories
and
for
every
product.
He
was
then
querying
the
database
by
looping
through
a
list
of
ids
and
therefore
ending
up,
as
you
can
see
here
with
26
000
database,
calls
to
execute
a
very
simple
query
against
the
database
to
figure
out
which
products
in
the
product
catalog
match
a
certain
criteria.
A
This
was
not
a
problem
in
the
beginning
when
he
sold
his
software
to
very
small
shops,
but
this
became
a
problem
with
organizations
that
had
thousands
of
articles
in
the
catalog
and
the
same
pattern
is
also
apparent
in
architectures,
where
one
service
calls
another
one.
This
is
again
distributed,
trace,
visualization,
just
showing
you
the
cascading
effect
of
one
service
calling
the
another
recursively,
and
this
goes
on
and
on
and
on.
This
is
not
a
good
sign,
and
it's
very
it's
visually
very
easy
to
understand.
A
When
you
look
at
this
that
there's
obviously
something
maybe
not
wrong,
but
something
you
definitely
need
to
look
into
now.
This
talk
is
not
about
more
performance
patterns
that
you
may
already
know,
or
maybe
not
already
know,
but
I
think
performance
problem
patterns
are
very
important
because
if
we
understand
patterns
we
know
how
we
can
detect
them
visually
and
hopefully,
when
we
can
detect
them
visually,
we
can
also
figure
out.
How
can
we
detect
them
by
looking
at
key
metrics
like
how
many
database
statements
are
called?
A
How
many
calls
do
I
make
to
another
service?
What's
the
payload?
These
are
a
couple
of
these
problem
patterns
and
also
the
metrics
to
look
into.
If
you
want
to
know
more,
I
did
over
the
last
couple
of
years
several
talks
where
I
talked
specifically
about
all
these
patterns
giving
examples
the
links
on
the
bottom.
Please
take
us
for
people
in
the
room,
of
course
take
a
screenshot,
but
this
the
links
are
also
in
the
slides
and
they
will
be
shared.
A
So
my
point
is
understand
your
patterns
and
then
derive
your
metrics
from
it,
which
really
gets
me
to
once.
We
know
metrics
on
how
to
detect
bad
patterns.
We
can
then
automate
the
analysis
and
the
automation
is
brought
to
you
today
by
captain.
That's
the
cncf
sandbox
project
that
I've
been
working
on
for
the
last
couple
of
years
and
the
reason
why
we
built
captain
well,
there
were
many
different
reasons,
but
it
was.
It
was
in
the
end.
A
Really
we
wanted
to
save
people
from
building
a
lot
of
automation
that
they
then
have
to
maintain,
and
so
we
wanted
to
provide
a
better,
better
alternative,
because
you
should
not
build
and
maintain
large
scripts
for
automating
things.
So
coming
back
to
the
use
case
of
performance,
because
I
would
love
more
people
to
automate
performance
analysis
into
the
delivery
pipeline,
so
hopefully
you're
not
sure
if
you're
running
performance
tests-
yes,
yes
or
not,
but
what
I
see
a
lot
of
people
are
held
back
in
including
performance
tests
in
every
build.
Why?
A
Because
many
that
are
executing
tests,
they
have
beautiful
dashboards,
their
grafana
dashboards
or
their
their
whatever
load,
runner
dashboards
or
whatever
tool
they're
using,
and
while
these
dashboards
are
pretty
to
look
at
it's
really
hard
to
analyze
them
unless
you're
really
expert
and
within
a
second.
You
know
if
this
is
good
or
not,
so
this
doesn't
scale,
even
if
you
already
deploy
automatically
in
test
automatically,
but
analyzing
manually
is
hard.
A
How
does
this
work
in
captain
now
with
I
talk
about
slo
service
level
objectives,
the
way
this
works
in
captain
is
you
define
a
list
of
slis
right
slices,
nothing
else
than
the
metric?
I
have
an
example
here:
five
metrics
response
time,
query
failure
rate
and,
and
a
couple
of
more
in
the
bottom,
even
an
open
security
vulnerability.
A
If
you,
if
you
have
a
tool
that
provides
you,
the
number
of
open
vulnerabilities
a
great
metric
to
have
so
you
specify
what
you
normally
look
at,
then
in
captain
you
specify
your
target,
your
slos,
where
we
allow
you
to
do
two
things:
either
warning
or
pass
meaning.
You
expect
a
certain
metric
to
have
a
certain
value
where
we
go
a
step
further.
You
cannot
only
specify
a
certain
let's
say
response
time.
What
does
it
say
here?
Response
time
should
be
faster
than
100
milliseconds?
Then
it's
green
or
250.
Then
it's
warning.
A
If
you
look
down
to
the
test
step,
login
response
time,
we're
actually
combining
it,
you
can
say
test
step
login.
This
particular
function
that
we
are
testing
should
be
faster
than
150
milliseconds,
but
also
should
not
get
slower
by
more
than
10
percent
to
the
previous
good
builds.
So
we
can
also
do
regression
detection
automatically
metrics.
A
So
you
specify
the
metric.
You
specify
your
slos
and
you
specify
your
overall
target
so
build
number.
One
comes
along
you're.
Pushing
your
pipeline
captain
automatically
reaches
out
to
your
prometheus
to
your
apm
tool
of
choice.
Pulls
in
the
metrics,
compares
them
against
your
objectives,
grades,
everything
and
then
you
get
a
score
between
0
and
100.
A
We
are
normalizing
this
to
100
points
in
total.
You
can
specify
weights
if
you
want
to,
but
you
can
also
specify
individual
metrics
as
key
metric
or
key
sli.
So
build
number
one
looks
good.
Build
number
two
comes
along.
The
developer
may
have
made
some
changes.
What
do
we
see?
Oh
things,
getting
slower,
plus
we
all
of
a
sudden
make
more
data
more
calls
to
the
back.
A
End
service
like
a
recursive
call,
maybe
even
and
therefore
be
getting
penalized,
so
we're
getting
less
points
and
therefore
immediately
feedback
that
this
bill
should
not
be
go
all
the
way
through,
because
we
would
end
up
making
twice
as
many
calls
to
the
back-end
service.
Once
this
goes
into
production,
I
get
immediate
feedback.
Maybe
I
fix
the
thing
build
number
three
comes
along.
It
looks
like
I
fixed
the
performance
problems.
I
brought
the
number
of
service
calls
back
to
one,
but
all
of
a
sudden,
I've
introduced
the
security
vulnerability.
A
This
is
also
part
of
performance
right
because
if
it
doesn't
secure,
it's
not
secure,
then
who
cares
about
the
best
performance
so
again
getting
penalized
so
that,
in
the
best
case
scenario,
build
number
four
comes
along?
Everything
is
green
again,
this
is
the
way
captain
works.
This
is
the
way
we
are
allowing
you
to
define
what
metrics,
what
indicators
are
important,
what
you
want
to
compare
it
against,
but
instead
of
you
doing
it
manually,
we
do
this
fully
automatically
for
you
a
little
bit
behind
the
scenes
with
captain.
A
When
you
kick
off
the
evaluation,
we
have
a
so-called
lighthouse
service
and
the
lighthouse
service
is
triggered
by
the
slo
definition.
You
define
your
slos
in
the
ammo
that
you
put
next
to
your
source
code
in
your
source
code
ripple.
You
can
see
a
little.
I
know
it's
a
little
small,
maybe
from
the
back
but
objectives
error
rate.
Jvm
memory
number
of
database
calls
where
you
specify.
What
do
you
expect
these
metrics
to
be,
but
here's
the
important
thing.
A
A
You
can
build
a
service
that
then
delivers
the
data
in
every
service
like,
for
instance,
we
have
integrations
with
prometheus,
with
dyna
trace
with
neoload
with
refund.
You
would
then
define,
or
whoever
is
responsible
for
observability
and
monitoring
in
your
organization.
You
then
define
how
do
you
actually
query
this
particular
piece
of
data
like
how
do
you
get
error
rate?
How
do
you
get
the
number
of
database
calls,
so
it's
separated
objectives
and
how
you
get
the
data
is
completely
separation
of
concerns,
because
typically
two
different
teams
are
responsible
for
that.
A
A
So
this
is
the
core
component
of
captain,
this
slo
validation.
To
give
you
one
example:
one
of
our
early
adopters
christian,
he
was
responsible
for
maintaining
gitlab
pipelines,
and
his
task
was
to
also
include
an
slo
validation.
Every
time
the
gitlab
pipeline
deploys
in
tests
and
now
he's
just
making
a
call
to
captain
where
captain
is
doing
all
the
evaluation
for
him
saving
a
lot
of
time,
because
otherwise
he
would
have
built
all
this
logic
in
his
gitlab
title
and
himself.
A
So
this
is
the
core
component
that
allows
us
to
analyze
slos
very
easily
and
get
this
into
existing
pipelines.
But
the
thing
is
captain.
We
wanted
to
solve
a
much
bigger
problem,
but
the
core
is
this
data-driven
approach,
where
we
always
go
towards
slos
and
analyze?
How
your
system
is
currently
doing.
A
What
we
want
to
really
solve
with
captain
is
the
following
thing:
a
quick
show
of
hands
who
is
using
jenkins
in
your
organization,
okay,
who
is
using
gitlab
perfect,
who
is
using
powershell
scripts
for
deployment
right
and
or
these
are
all
great
tools.
But
the
thing
is
they're
all
doing
certain
things
pretty
well,
but
what
we
have
observed
is
your
classical
pipeline
has
a
hard
coded
set
of
steps.
Hard
coded
integrations
with
your
tools.
A
Even
sometimes
you
have
configuration
in
your
pipeline.
I've
seen
jenkins
pipelines
also
internally
in
our
organization,
where
you
have
your
yaml
files,
your
your
deployment,
definitions,
files,
hard
coded
somewhere
in
the
jenkins
pipeline.
This
doesn't
scale.
This
becomes
very
ugly,
very
fast.
So
what
we
said
we
want
to
solve
this
problem
because
we
want
to
make
it
like.
We
want
to
make
the
life
easier
for
people
that
need
to
automate
more
things
into
their
pipelines.
So
what
we
do
with
captain
we
are
ripping
apart
the
hard-coded
process
and
tooling,
on
the
left
side.
A
You
can
specify
we
call
it
ship
in
a
shipyard
file,
automation,
sequences,
so
you
specify
what
tasks
you
want
to
execute.
You
want
to
execute
on
the
right
side.
You
have
your
two
links
completely
separated
the
configuration
all
the
configuration
is
maintained
in
the
git
repository
and
then
we're
using
eventing
to
connect
the
two
things
together.
A
So,
to
give
you
an
example,
because
we're
talking
about
performance
engineering
here,
if
you
wanna
use
captain
for
the
complete
sequence
for
performance
engineering,
you
want
to
deploy
a
new
artifact.
You
want
to
test
it
and
you
want
to
have
it
evaluated,
so
you
specify
deploy
test
and
evaluation
without
any
tool
context,
and
in
my
example,
then
I
can
say
captain
trigger
performance
in
a
certain
stage,
and
I
give
it
some
additional
metadata
like
what
image
do
I
want
to
test?
A
What
captain
then
does
it
sends
a
cloud
event
out
to
the
world
and
says
who
is
there
in
my
environment
that
can
deploy
this
particular
artifact
in
staging
with
a
blue-green
deployment
strategy?
I
may
have
helm
doing
it.
I
may
have
an
argo
pipeline.
I
may
call
gitlab,
I
may
call
it
jenkins
pipeline.
This
is
all
event
driven
and
our
tools
can
then
subscribe
to
the
event.
To
then
do
the
actual
action
once
it's
done.
Sending
it
back
next
thing
is
test.
A
Captain
sends
an
event.
I
need
somebody
that
can
execute
a
performance
test
in
staging.
Today
you
may
use
jmeter
the
nice
thing
about
this
is
if
your
organization
decides
next
next
year,
you're
going
to
a
different
tool,
you
don't
need
to
update
any
of
your
automation
code.
The
only
thing
you
need
to
do
is
you
turn
off
the
subscription
for
the
event
for
geometer
and
you
just
let
your
new
tools
subscribe
to
the
cloud
events
they're
all
standardized.
A
A
A
A
You
can
also
use
obviously
captain
for
the
complete
and
trained
automation.
So
it's
not
just
constrained
to
one
thing.
So
captain
orchestrates
all
of
this,
for
you
on
the
right
side.
You
see
all
the
tools,
we're
not
replacing
any
tool.
We
want
you
to
use
the
tool
of
choice
for
a
particular
task,
there's
so
many
great
tools
out
there
I
mentioned
earlier
gitlab
jenkins,
they're,
all
great
tools,
if
you
have
invested
in
this,
don't
throw
them
away,
but
don't
build
additional
automation
code.
I
always
say
friends,
don't
let
friends
build
their
own
automation.
A
I
know
it's
a
bold
statement
on
the
bottom,
but
it
says
90,
less
custom,
automation,
code,
that's
what
we've
seen
from
people
like
christian.
I
brought
up
earlier.
It's
event
and
data
driven
it's
based
on
open
standards
and
all
the
configuration
is
stored,
maintained
and
versioned
in
git
now
captain
in
the
real
world.
How
does
captain
look
like
outside
of
my
little
environment?
Here?
I
am
from
austria,
it's
also
a
famous
song
in
austria,
but
it's
also
why
I'm
very
proud
of
this
year.
A
Every
time
they
push
their
new
services,
the
new
builds
for
their
jenkins
pipelines.
They
have
captain
automatically
validates
that
release
based
on
their
slos.
Another
cool
example,
because
we
are
talking
about
performance
engineers.
Here
I
have
a
youtube
video
on
the
on
the
top:
it's
a
little
small
but
mike
kobush.
She
works
for
ndic
national
association
for
insurance
companies,
he's
a
big
load,
runner
expert
and
is
now
using
also
jmeter,
and
he
says
when
he
runs
this
large-scale
load
test
against
the
replications.
A
It
takes
him
typically
30
to
60
minutes
to
analyze
the
results
with
captain.
It
takes
him
one
minute.
That
means
every
week
he
has
a
couple
of
more
hours
to
go
fishing,
that's
what
he
told
his
cto
and
I
haven't
have
him
on
record
on
this
video.
Another
cool
use
case
is
from
vitelity.
They
are
a
large
insurance
company,
what
they
do.
They
have
a
central
software
organization,
they're,
building
the
services,
and
then
they
deploy
it
to
the
different
countries.
So
it's
the
same
software,
but
with
different
feature
flags.
A
They
turn
on
different
things
that
are
country
specific
and
what
they're
doing
they're
testing
every
single
configuration
every
single
tenant
and
then
captain
gives
them
feedback
on
whether
this
configuration
is
good
and
what
you
can
see
here.
The
quality
gates
are
all
red,
so
probably
they
need
to
invest
a
little
bit
more
in
quality,
they're,
really
cool,
though
it's
building
great
software.
B
A
A
A
So
next
slide
says:
let's
wrap
it
up.
I
want
to
wrap
it
up
with
the
slides,
but
then
I
know
I'm
a
little
fast.
Actually
I
wanna
then
also
show
you
a
little
bit
of
captain
so
that
you
see
it's
not
just
screenshots,
but
my
advice
for
you.
This
talk
is
about
optimizing
performance
based
on
slos.
I
want
to
just
recap
how
I
started
my
presentation
understand
the
applications
you're
responsible
for
whether
you're
testing
that
your
performance
engineer
look
at
distributor,
traces,
understand
the
patterns.
A
Do
these
metrics
really
tell
you
after
test
that
something
is
wrong
and
if
they
do,
then
you
can
convert
these
metrics
behind
the
dashboard
into
an
sli
and
then
slow
definition
in
captain,
and
then
you
can
have
captain
automate
that
analysis
completely
for
you
and
then
you
want
to
bake
it
into
your
existing
two
landscape
and
have
captain
automate
all
of
this
fully
well.
Every
time
when
you
trigger
a
pipeline
also
think
about
what
I
what
I
said
earlier.
Captain
is
not
just
there
for
performance
analysis.
Captain
can
really
orchestrate
any
type
of
process.
A
Not
only
are
we
because
we
don't,
I
always
get
the
question,
so
is
captain
it's
another.
Argo
is
captain.
Another
jenkins
is
captain.
What
is
captain
just
doing
delivery
right?
No
captain
is
bringing
your
data-driven
orchestration
of
automation,
sequences.
Whether
this
is
around
anything
you
need
to
do
in
delivery
or
in
operations.
We
have
a
lot
of
users,
also
using
us
for
other
remediation
meaning.
When
there's
a
problem
in
production,
they
then
trigger
a
captain
sequence.
A
Where
captain
first
notifies
people,
then
maybe
scales
up
scales
down,
does
whatever
whatever
it
needs
to
happen.
In
the
context
of
the
problem,
then
evaluates
against
the
slos,
it's
the
critical
part
and
based
on
that
decides,
was
the
system
brought
back
to
a
healthy
state?
If,
yes,
great,
if
not
execute
the
next
section
so
be
reminded
of
that,
so
cap
is
a
lot
of
things.
I
only
had
a
couple
of
minutes
here
to
talk
about
it
today
with
the
focus
on
performance
engineering.
A
If
you
want
to
learn
more,
I
had
a
couple
of
presentations
at
different
conferences.
You
will
find
a
lot
of
material
about
captain
and
we
are
obviously
welcoming
every
new
community
member.
We
have
a
lot
of
cool
stuff
on
the
captain
website.
If
we
go
to
captain.sh,
then
you'll
find
all
the
information
you
need
to
get
started.
We
also
have
a
live
captain
installation,
a
demo
installation
that
you
can
click
through,
which
I'm
going
to
show
you
in
a
second
you
can
sign
up
for
the
slack
you
can
install.
You
can
download
it.
A
A
Got
ten
minutes
even
well
again,
I'm
going
to
do
amazing
demos,
so
this
is
actually
the
the
captain.
Public
demo
live
environment,
and
this
is
the
one
you
get
to
yourself.
So
if
you
go
to
captain.sh,
you
can
click
on
explore
live
demo.
That
brings
you
to
a
little
tutorial.
That
explains
all
the
projects
that
are
on
here
we're
working
very
closely
with
litmus
which
was
a
great
partner
for
us,
and
it
explains
the
different
scenarios.
And
then
you
have
to
link
to
captain.public.demo.captain.sh.
A
In
this
case
I
have
a
hello
service
and
you
can
see
here
that
there
was
a
build
number
zero
one
one
deployed
and
what
I
see
in
the
default
view,
because
this
is
the
most
important
thing
as
a
service
owner.
What's
the
quality
of
the
service,
you
see
this
heat
map
that
I
showed
you
earlier
right
and,
and
you
see
on
the
bottom,
I'm
not
sure
if
it's
how
easy
it
is
to
see,
but
on
the
heat
map
on
the
top.
That's
the
score.
A
I
only
have
two
metrics
here
slo's
that
are
evaluated,
but
you
can
go
back
in
history.
If
I
scroll
down
a
little
bit,
I
see
the
individual
values
I
also
see
against
which
value
it
was
compared
to
in
case
you
decide
you
want
to
compare
also
with
previous
releases,
so
that's
really
neat.
Most
importantly,
this
is
just
the
thing
that
is
most
important.
Are
we
green?
Are
we
yellow
or
are
we
red
behind
the
scenes
that
much
more
happened?
A
A
The
test
was
using
a
combination
of
jmeter
to
run
some
load
and
litmus
so
the
nice
thing
about
an
event-driven
system
like
this
is,
you
can
say
I
want
test
to
be
executed
and
then,
depending
on
your
projects,
depending
on
your
event
subscription
you
can
have
my
one
or
many
tools
do
something
in
this
case,
I'm
running
load
with
jmeter
and,
at
the
same
time,
litmus
kicks
in
with
with
chaos,
engineering,
and
once
these
are
done,
then
captain
triggers
the
evaluation
right.
So
how
does?
How
can
you
define
this
right?
A
A
Initially,
when
you
create
a
project,
you
start
with
a
so-called
shipyard
file,
and
this
is
really
kind
of
your
sequence
definition.
So
I
have
a
sequence
that
is
called
delivery.
That
does
a
deployment
a
test
and
an
evaluation.
As
you
can
see
here,
there's
no
hard
coded
tool
integration
because
there
are
no.
We
don't
reach
out
to
a
tool
with
a
hardcoded
integration.
Captain
will
work
history.
This
and
we'll
send
events
to
then
trigger
the
right
tools.
A
Bless
you
and
then
what's
very
important.
Is
captain
also
automatically
keeps
branches
for
every,
as
we
call
it
stage
where
you
cannot.
You
can
group
your
your
all
your
automation,
sequences
into
so-called
stages,
and
then
here
you
can
put
in
the
actual
files
that
then
the
individual
tools
need
because
obviously
help
needs
a
helm.
Chart
chain
meter
needs
a
geometer
script.
Litmus
needs
the
chaos
experiment
for
prometheus.
You
need
to
think
you
need
to
define
the
queries,
but
this
is
all
organized
in
git
all
right.
A
So,
coming
back
to
my
little
example
here,
if
I
go
back
last
thing,
I
want
to
show
you
when
I
triggered
this
sequence,
you
can
either
trigger
it
through
the
captain.
Cli
command
line
interface
or
through
the
api
for
every
step.
Captain
sends
a
so-called
cloud
event
right.
This
is
the
stuff
we're
currently
standardizing.
A
So
in
this
case,
captain
sends
deployment
triggered
and
then
whatever
tool,
you've
subscribed.
To
that
event
can
then
say
hi,
it's
me
I
want
to
do
it.
Then
they
send
a
starter
event
and
once
they're
done,
they
send
a
finished
event
again.
These
are
all
cloud
events
with
additional
information
about
what
did
they
do?
How
did
they?
You
know
in
this
case?
A
You
have
your
subscriptions
here
on
the
on
the
so-called
uniform
page,
so
you
see
which
tools
are
actually
subscribed
in
for
a
particular
project
for
a
particular
stage,
all
right,
so
I
just
got
designed
it's
time
to
wrap
up,
because
we
have
questions
hopefully
and
yeah.
What
I
want
to
say
if
you
are
interested
in
captain
with
a
booth,
we
have
a
captain
move
in
the
pavilion
I
think
tomorrow
and
like
tonight
also
tomorrow
after
two
and
on
friday.
A
I
also
want
to
say
a
big
thank
you
to
diner
trace
diner
choice
is
my
employer
and
diner
trace
is
the
big
sponsor
of
this
project
and
I
want
to
say
thanks
for
investing
in
open
source
and
giving
people
a
chance
to
work
on
this
and
yeah.
If
you
are
interested
also
in
that
we
also
have
a
booth
down.
I
think
it's
g11,
but
I
now
I
want
to
open
up
the
question
in
case.
There
are
questions.
If
you
have
questions
in
the
audience,
then
over
there
hey.
A
Okay,
so
the
question
was
specific:
how
do
we
compare
to
argo
analysis
because
they
also
pull
in
some
metrics?
I
don't
want
to
say,
there's
no
overlap,
but
what
we
try
to
be
is
we
try
to
be
not
a
tool
just
for
delivery
right.
That's
what
I
want
to
say
earlier
with
captain.
You
can
automate
sequences
that
you
need
to
automate
in
delivery,
but
also
in
operations
and
what
we
do.
We
always
pull
in
data
through
our
sli
and
solo
definition
that
then
allow
the
orchestrator
to
decide.
Do
we
go
on
or
not?
A
If
you
are
happy
with
argo
and
your
argo
analysis
and
that's
fine
with
you,
then
it's
good
right.
If
you
can,
with
argo
orchestrate
the
deployment,
your
tests,
your
evaluation
and
then
you
notify,
and
if
you
switch,
you
want
to
switch
from,
let's
say
one
tool
to
another
and
you
are
and
that's
fine
and
then
stick
with
it.
What
I
showed
you
today
is
a
separation
of
concerns
between
process
definition.
You
can
define
whatever
process
and
the
tooling
is
separated
the
magic
behind
the
scenes
is
that
they
are
connected
through
events.
A
A
I
have
a
test
step
because
that's
what
I
want
to
do
in
one
project,
I
made
a
functional
test
in
a
second,
I
made
a
load
test,
and
maybe
next
year
your
company
says
we
need
to
add
security
tests
to
every
process.
Every
project.
The
only
thing
you
have
to
do
is
subscribe
to
the
test
event
and
then
automatically
your
security
tool
kicks
in.
So
these
are
some
of
the
considerations
that
we
have
put
into
the
architecture
of
captain
and
it's
not
constrained
to
delivery.
A
Even
though
we
talk
a
lot
about
delivery,
automated
remediation
production,
you
define
your
sequence.
What
you
want
to
do
when
a
problem
strikes
in
production
right.
Do
you
want
to
first
notify
people?
Then
you
want
to,
I
don't
know
scale
up
then
you
want
to
validate.
Then
you
do
the
next
step,
so
you
can
define,
we
call
them
remediation
run
books,
but
they
are
orchestrated
in
the
same
way
as
you
saw
for
the
delivery
use
case
yeah,
but
see
it
for
yourself
right.
B
A
Spin
up
the
entire
thing
and
run
all
the
screen
against
that
yeah.
So
the
question
is:
what's
the
best
practice
for
microservice
architectures?
Do
you
rather
try
to
test
microservices
individually,
or
do
you
deploy
everything
and
run
everything
at
once?
I
think
the
the
best
practice
do
both
right
if
it's
possible.
That
means,
if
you
have
a
micro
service
and
you
can
run
it
in
isolation.
That
means
you
can
mock
away
the
dependencies,
or
at
least
you
can
deploy
an
environment
where
all
the
dependencies
are
also
just
there
and
you
can
run
tests
against
it.
A
Then
this
is
the
ideal
state
because
remember
my
number
one
problem
pattern
I
had
today
it
was
the
m
plus
one
query
problem.
If
your
microservice
has
always
made
one
call
to
the
database
and
all
of
a
sudden,
it
makes
five
calls
to
the
database,
then
you
don't
need
to
deploy
the
whole
app
in
a
big
environment,
because
you
already
know
something
has
changed.
That
will
have
a
make
a
huge
impact
later
on,
so
these
patterns
they
can
most
of
them,
can
be
found
just
by
testing
it
in
isolation.
B
A
Yes,
so
the
question
is:
can
you
share
tests
with?
Yes,
I
showed
you
earlier
that,
for
instance,
the
media
files
that
were
stored
in
git,
so
you
can
either
store
them
on
the
main
branch.
Then
it's
available
for
every
service
in
every
stage
you
can
store
it
on
a
branch
level.
Then
it's
available
for
every
service
in
that
brand.
That's
in
that
stage
or
you
can
define
it
on
the
per
stage.
So
it's
like
an
inheritance
level
that
we
have
there.
A
Say
it
again
new
relic
if
captain
supports
new
relic
and
bitbucket,
so
I
would
love
if
captain
would
support
new
relic,
it
doesn't
do
it
right
now.
We
have
a,
however,
a
kid
issue
out
there
ready
for
new
relic
for
data
talk,
and
I
think
also
in
stannah.
We
would
just
would
you
know
it's
an
open
source
project.
That's
why
if
you
have
new
relic,
if
you
have
datadog
or
any
other
data
providers,
then
you
know
join
us
and
and
and
help
us
implement
it's
just
it's
very
simple.
B
A
So
yeah-
maybe
I
quickly
so
the
nice
thing
about
this
is
so
captain
sandbox
right
all
of
our
all
of
our
new
services
start
in
captain
sandbox
right.
So
we
have
jenkins
library,
artifact
blah
blah
synthesis.
So
all
of
the
integrations,
if
you
want
to
get
started,
there's
just
a
service,
a
captain
service
template.
So
this
is
just
a
go
template
that
you
clone,
and
then
you
just
put
in
your
let's
see.
There's
a
lot
of
I
mean
it's.
A
Exactly
so
that
service
you
deployed
at
service
in
kubernetes,
let's
say
on
your
target
server
and
that
in
in
the
configuration
of
that
service,
you
say
I
want
to
subscribe
to
this
event,
and
then
this
really
then
becomes
the
service
that
then
obviously
calls
let's
say
the
new
relic
api
or
whatever
else
you
want
to
integrate.
Now
we
are
currently
working
with
the
cdf
foundation
in
a
special
interest
group
to
completely
standardize
all
of
our
cloud
events,
which
means
once
this
standard
is
through.
Then,
companies
like
new
relic
and
others.