►
Description
In many organizations up to 80% of pipeline execution time is spent in manual build validation steps. How can we reduce that? One option is applying Google's SRE (Site Reliability Engineering) practices by automating SLI (Service Level Indicators) & SLO (Service Level Objectives) based build validation. This method has proven to detect problematic issues in production and also allows us to automatically approve or reject builds being pushed through our pipelines. The side benefit is that once code passes our pipelines we have already been validating it with SLIs/SLOs relevant for production as well.
This is a presentation by Andreas Grabner, Dynatrace. In this session you learn the basics of picking good SLIs & SLOs and how to extract them from your monitoring tools. After this session you will be able to start implementing this integration yourself with Jenkins. To give you a jump start you will be introduced to the open source project Keptn (www.keptn.sh) which provides automated SLI/SLO-based quality gates. Then we'll talk about Keptn Jenkins Shared Library which integrates Jenkins and Keptn with just a couple of function calls.
References:
* Slides: https://www.slideshare.net/grabnerandi/jenkins-online-meetup-automated-sli-based-build-validation-with-keptn
* Keptn Jenkins Shared Library: https://github.com/keptn-sandbox/keptn-jenkins-library
* Meetup page: https://www.meetup.com/Jenkins-online-meetup/events/270861119/
A
Thank
you
so
much
all
right.
So
let
me
share
my
screen
here,
thanks
for
thanks
for
the
venue,
thanks
Jenkins
community,
for
inviting
me
yeah,
it's
just
some
additional
information
on
what
all
it
just
said.
But
if
anybody
wants
to
reach
out
to
me,
Twitter
is
one
option.
If
you're
not
on
Twitter,
you
might
be
on
LinkedIn.
So
here's
my
LinkedIn
link,
you
can
see
Grabner
andy
with
an
I'
in
the
end.
That's
always
very
important
for
me.
A
There's
a
lot
of
angels
with
the
why
I
would
say
I'm
the
one
with
the
eye
you
can
find
me
LinkedIn
as
well.
In
case
you
wanna,
reach
me
by
email.
You
can
just
use
my
full
first
name
dot,
my
last
name
at
dynaTrace
calm
because,
as
you
can
see,
I
couldn't
be
and
have
been
working
for
the
energies
for
twelve
years
and
hopefully
for
many
more
years
to
come.
So
you
can
definitely
reach
me
on
andreas
top,
grab
net
diner
system
as
well.
In
case
you
don't
like
Twitter
or
LinkedIn.
Now.
A
Today's
topic
is
about
a
project
that
we've
been
working
on
for
more
than
a
year
now
the
project
is
called
captain
and
in
case
you
wonder
what
captain
stands
for.
Captain
is
the
German
phonetic
for
a
captain,
all
right,
so
we
or
I
and
the
more
the
large
part
of
the
captain
team
we're
from
Austria.
So
our
native
language
is
German,
even
though
I
think,
if
you
ask
German
people,
they
probably
say
the
Austrian
is
not
really
German.
A
It's
a
strange,
weird
dialect
that
we
have,
and
we
have
many
of
them,
even
though
we're
very
small
country
but
anyway.
So
what
captain
does
captain
provides
a
lot
of
different
things?
A
lot
of
automation,
use
cases
on
things
that
we
have
seen
in
our
own
organization.
Also,
the
people
we
work
with
as
power
on
a
part
of
the
continuous
delivery
lifecycle,
there's
different
things.
So,
if
you
are
interested
in
learning
more
about,
captain
obviously
go
to
the
captain
web
site.
Go
to
the
github
repo
check
out
the
tutorials.
Follow
us
on
Twitter
you
can.
A
You
can
find
all
the
information
today.
I
want
to
focus
on
a
particular
capability
of
captain,
which
is
automating
quality
gates,
based
on
the
concept
that
Google
has
kind
of
spearheaded,
at
least
on
the
terminology,
perspective
around
a
slice
and
SLO
service
level,
indicators
and
service
level
objectives.
Now,
before
I
go
into
the
the
presentations
and
also
live
demos
and
discussions,
they
get
the
questions
that
you
have
in
the
polling.
A
These
are
questions
that
did
you
know
I'm
all
against
me.
You
know
what
kind
of
questions
can
we
ask
in
the
polls
and
for
me,
it's
really
interesting
to
understand.
Have
you
been
familiar
with
the
term?
It's
a
licensed
ellos.
Are
you
using
this
red
in
your
company,
or
is
it
completely
new?
What
I'm
showing
you
today
really
works?
Well,
if
you
integrate
this
in
your
pipeline,
where
you
have
automated
tests,
so
the
question
for
me
is
also
you
know:
how
does
do
you
have
actually
tests
as
part
of
your
pipeline?
A
A
You
know
we
are
here
today
with
Jenkins
on
kubernetes,
so
captain
runs
on
kubernetes,
whether
it's
you
know
any
type
of
path,
kubernetes
that
the
cloud
vendors
put
out
your
own
kubernetes
or
openshift,
but
the
question
that
I
also
have
to
you
as
a
community.
If
you
are
in
charge
of
your
Jenkins
pipelines,
do
you
actually
have
access
to
kubernetes?
Could
you
be
able?
A
Are
you
able
to
install
a
tool
like
captain
on
a
kubernetes
cluster,
or
is
this
something
you
cannot
do
for
whatever
reason
we
want
to
get
a
little
feeling
and
which
deployment
models
we
need
to
support
in
the
future?
All
right,
so
let
me
get
started.
There's
a
lot
of
things.
I
wanted
to
do
both
from
an
educational
perspective,
but
also
live
demos.
I
have
two
major
use
cases
prepared,
but
they're
all
centered
around
the
SLO
based
quality
gates,
so
the
first
one
is
really.
Why
do
we
do
all
this?
A
It's
around
automating
build
approvals
and
I
want
to
first
kind
of
come
up
with
a
problem
statement.
So
what
is
the
thing
that
we
are
seeing
in
our
organization,
but
also
with
some
of
the
people
we
work
with?
You
know
you
may
have
your
Jenkins
pipelines
and
you
do
your
build
your
deploy.
You
run
your
tests,
but
then
very
often,
there's
a
there's,
a
manual
approval
stage.
Where
then
somebody
needs
to
look
at
results
and
in
case
you
have
some
unit
tests
or
maybe
functional
tests.
You
already
at
least
have
some
data.
A
Where
you
can
say
you
know
this
is
a
good
build
or
not
a
good
build,
but
then,
if
there
is
a
problem,
this
is
a
deal-breaker,
yes
or
no
right
is
a
hard
call
that
called
the
kidiots.
Oh
no,
so
that's
already
good,
if
you
at
least
have
some
some
functional
tests.
Now,
if
you
are
like
me,
I've
been
working
in
performance
engineering
for
the
past
20
years.
So
performance
is
very
dear
to
my
heart
performance
scalability
reliability.
A
Then
the
next
question
is
well.
Can
we
can
we
add
more
performance
metrics
to
the
mix,
so
we
actually
get
a
better
understanding?
Is
this
a
good
build
where
bad
will
will
this
build
break
the
next
environment
and
I
know,
there's
a
lot
of
great
tools
and
integrations
with
Jenkins
already
available
their
performance
plugins
that
can
pull
in
data
from
different
testing
tools,
but
still
the
problem
at
that
at
least
I
see
from
that
from
the
integrations
that
I
know
there's
a
lot
of
mineral
comparisons.
A
You
can
look
at
charts,
you
can
look
at
graphs,
but
then
is
this
really
better
or
worse
right?
If
you
look
at
it
at
these
two
charts
now
do
I
know.
Is
the
current
build
better
than
the
other
one?
You
know
it
takes
time
to
figure
it
out.
So
it's
already
great,
though,
if
you
have
function
tests
and
maybe
some
low
tests,
the
next
thing
from
a
maturity
perspective
that
we
see
is
people
that
are
adding
monitoring
data
to
the
mix,
so
in
case
you're
deploying
a
build.
A
If
you
run
your
test
and
then
you
also
have
you
know
either
Prometheus
as
a
monitoring
option
for
you
or
an
APM
tool
like
dynaTrace
new
relic
data
dog
ape
the
name
is
there's
a
lot
of
tools
out.
There
I
obviously
put
dynaTrace
on
it,
because
there's
the
company
I
work
for,
but
still
you
know,
any
type
of
monitoring
tool
will
do
that.
A
The
great
thing
about
this
is:
yes,
we
get
even
more
data,
but
often
the
people
that
we
talk
to
they
say
well
now
we
have
even
more
data
and
we
don't
even
know
what
the
data
means
and
what
to
tailor
to
look
at,
and
also
some
of
these
tools
that
don't
tell
you
that,
from
which
particular
test
and
from
which
particular
builds,
does
this
data
actually
come
from.
So
yes,
it's
great
to
have
more
data,
but
itself
it
still
prolongs
the
process
of
approving
the
build.
A
So
this
is
a
challenge
that
we
want
to
solve
Ren.
So
what
was
the
inspiration
on
how
we
want
to
solve
this
problem?
The
first
inspiration
came
as
I
said
earlier,
from
Google's
essary
practices.
So
in
case
you
have
not
heard
about
a
sorry,
it's
a
site,
reliability,
engineering,
there's
a
lot
of
great
material
out
there,
but
essentially
it
is
targeted
on
three
pillars.
A
The
one
concept
is
service
level
indicators
or
a
slice,
and
really
this
is
it's
just
defining
a
particular
metric
that
you
wanna
extract
and
based
on
evaluation
on
so,
for
instance,
it
could
be
the
error
rate
of
a
let's
say,
a
login
test.
If
you
have,
if
you
deploy
and
run
a
test-
and
you
check
your
login,
you
log
out
your
Add
to
Cart,
then
you
can,
as
a
Nestle
I
define
I,
want
to
know
the
error
rate
when
I
run
my
test,
how
many
arrows
do
I?
Have
that's
an
ESL
I.
A
The
next
thing
on
top
of
that
is
the
SLO
or
the
service
level
objective.
So
here
it's
kind
of
like
a
binding
contract
where
you
say
my
objective
for
a
particular
metric
is
a
certain
target
right,
let's
say:
login
error
rate
of
a
new-build.
If
we
execute
these
set
of
tests
has
to
be
less
than
2%.
Another
metric
could
be
the
the
response.
Time
should
be
faster
than
100
millisecond
for
a
certain
API,
so
these
are
my
soul.
Oh
self,.
A
A
So
these
are
kind
of
you
know,
concepts
that
Google
has
brought
out
and
really
what
I
can
encourage
you
to
do.
If
you
want
to
read
up
on
it,
there's
a
great
Google,
Cloud,
YouTube
video,
it's
called
slic
solos
SLA
and
there's
here's
the
youtube
link,
but
you
can
easily
find
it,
but
basically
what
they
say
and
I'm
not
kind
of
you
know
extracted
this
information
from
them
is
SL
eyes.
Drive
SLO
switch
in
for
Mesa
lays
now.
The
question
is:
this
is
typically
done
in
a
production
environment.
A
So
all
these
concepts
have
typically
been
used
in
production,
which
is
great.
But
if
we
are
pushing
stuff
into
production,
intend
view
we
are
validating
metrics
and
objectives.
Why
are
we
not
using
the
same
concept
early
on
in
development
in
CI
CD
to
enforce
these
I
solos
already,
because
we
already
know
what
is
what
is
expected
later
on,
and
this
is
kind
of
also
what
inspired
us
to
build,
what
I'm
going
to
show
you
another
thing
and
another
implementation
that
I've
seen
actually
comes
from
from
our
organization.
A
So,
as
I
said,
I
worked
for
dynaTrace
in
Thomaston
mara
he's
one
of
our
chief
performance,
architects.
What
he's
been
doing
for
years
is
every
time
he
gets
in
you
build
a
daily
build.
He
deploys
it
with
Jenkins
into
an
environment,
he
monitors
it.
Obviously
we
monitor,
you
know
our
tool
with
our
own
tool,
so
we
get
some
monitoring
data
and
every
day
he
runs
continuous
load
and
then
he
validates.
How
does
this
build?
Compare
to
the
other
and
I
thought
he
will
hear
a
very
interesting
concept.
He
called
it.
A
The
performance
signature,
it's
kind
of
how
less
the
performance
or
does
the
quality
look
like,
and
essentially
he
didn't
told
me
he
let
me
look
behind
the
scenes
and
he
said
you
know
Andy
what
I'm
doing
after
every
build
after
we
get
a
new
deployment
and
we
ran
our
continuous
tests
on
it.
Then
I
am
looking
at
multiple
metrics
is
a
kind
of
DSL
eyes,
memory,
error
rate
and
so
on
for
the
time
frame
of
the
test,
then
I'm
comparing
them
and
validating
him
against
thresholds.
A
Either
we
have
hard-coded
s
thresholds
where
we
know
we
cannot
cross
that
limit
or
victress
compared
with
previous
builds
for
regression
detection
and
then
what
we
do.
We
look
at
all
of
these
metrics
and
we
basically
calculate
and
overall
quality
status
of
that
particular
build.
That
comes
in
and
that's
what
he
calls
the
performance
signature,
and
so
we
thought
this
is
actually
cool.
We've
been
doing
this
in
turn
already
integrated
into
Jenkins.
A
It
was
custom,
build
it
was
home
built
and
while
they
worked
wait
for
us,
we
said
how
can
we
take
this
and
put
it
into
now
the
open
source
project
kept
and
so
that
everyone
out
there
can
leverage
it?
So
that's
why
what
we
built
remember.
Captain
is
a
larger
project,
but
one
of
the
capabilities
is
the
quality
gate
capability.
So
the
way
this
works,
captain
quality
gates
can
talk
to
different
data
sources,
whether
it's
your
monitoring
tools,
your
testing
tools
or
any
other
tool.
A
It's
extendable
through
an
event-based
mechanism,
so
you
can
ride
your
own
extensions.
The
most
important
thing
to
is,
you
define
a
slice,
so
your
service
level
indicators.
So
what
are
the
metrics
that
are
important
for
you
when
you
push
a
build
through
your
pipeline?
Typically,
this
is
stuff
that
your
performance
engineers
would
define
where
your
your
team
leads,
your
architects
would
define.
They
say
these
are
metrics
that
are
important,
and
here
is
a
name,
and
here
is
the
query
that
they
ended
the
individual
tool
that
can
that
can
deliver
that
metric
understands.
A
A
You
can
also
then
specify
the
total
passing
score
now.
The
way
this
works.
If
you
have
captain
in
place,
you
basically
say:
hey:
captain
I
want
to
start
my
evaluation,
because
I
do
just
when
your
champions
pipeline,
you
deployed,
you
ran
your
tests
and
then
you
can
say
captain
for
the
last
30
minutes
for
this
particular
application.
Here
are
my
eyes
and
here's
my
solos.
Now
you
go
off
and
pull
the
data
from
these
tools.
A
One
of
many
tools
pulling
the
data
like
these
are
the
SLI
valleys
that
may
come
in
then
captain
will
compare
it
against
your
SLO.
So
what
are
your
objectives
and
then
every
individual
metric
gets
to
score
between
between
one
points
of
one
point:
if
it's
everything
is
good,
half
a
point
if
it
is
in
warning
range
with
zero
points,
if
it's
failing
and
then
captain
calculates
ok,
how
many
points
did
we
achieve?
A
A
Any
type
of
tool
that'll
give
you
a
little
better
example,
also
over
build
thread.
If
you
have
an
SLI
document
where
you
say
these
are
my
metrics
response
time,
95th
percentile
failure,
rate
response
time
of
my
login
tests
number
of
database
calls
that
my
login
test
is
executing.
So
you
specify
the
s
slice,
then
you
specify
your
SLO.
So
what
are
your
objectives?
What
have
you
set
out
to
achieve
for
the
engineering
team
and
again
this
is
a
combination
of
dynamic
values
with
where
you
can
compare
with
previous
builds.
A
Previous
baselines
were
static
values
if
you
have
hard-coded
thresholds
and
you
define
the
overall
gore
score
but
overall
goal
now,
if
build
number
one
comes
along-
and
you
say:
captain
do
the
evaluation
for
my
project
for
this
service
you
will
see
later
on,
captain
is
organized
in
projects
and
services
and
stages,
but
he
said
this
is
the
time.
Then
captain
goes
off
for
you
pools
and
the
data
compares
it.
In
this
case
everything
is
green.
You
get
100%
build
number
two
comes
along
in
my
example.
A
To
two
metrics
are
not
good,
so
you
getting
penalized
overall
75%,
that's
warning.
Build
number
three
comes
along,
the
first
two
problems
have
been
fixed,
but
now
the
number
of
database
calls
that
that
trend,
that
one
of
the
services
that
you're
testing
has
been
going
from
three
to
six,
but
you
didn't
allow
that
increase.
So
now
you
getting
penalized,
which
means
now
we're
down
at
62,
point
five
percent
right,
which
is
a
bad
build,
so
you
can
stop
the
pipeline
and
then
build
for
comes
along.
Everything
is
green
again,
so
this
is
the
idea.
A
What
this
really
does
it
automates
a
lot
of
the
stuff
where
you
would
normally
go
off
to
different
tools
and
pull
in
all
the
metrics
and
then
compare
them
now.
This
is
the
way
we
explain
it
in
in
a
PowerPoint
with
a
table,
and
this
is
how
it
looks
in
the
captain's
bridge.
So
captain
itself
not
only
has
an
API
where
you
can.
You
know,
trigger
the
evaluation
and
get
a
result
back
as
Jason,
but
we
also
have
a
UI
that
visualizes
hey.
This
is
every
single
SLI
and
like
every
row
here
is
an
SLI.
A
Every
column
is
an
evaluation
run
and
then
on
the
top
you
see
the
overall
score.
So
it's
easy
to
see
how
individual
builds
have
done
from
a
quality
perspective
and
kind
of
taking
this
number
and
aggregating
it
to
you
know
s
warning
will
fail
and
then
we
also
have
a
chart
view.
So
you
know
as
a
performance
engineer
that
I
am
and
by
heart,
I,
also
like
to
look
at
the
at
numbers
and
trends,
so
you
can
also
then
look
at
okay.
A
A
Remember
earlier,
I
talked
about
the
manual
approvals,
at
least
what
we've
seen
you
know.
It
just
takes
time
and
it's
manual,
so
what
we
can
now
do
with
Jenkins
and
captain,
and
we
have
a
champions
library
that
if
you
have
captain
installed,
then
you
have
a
champions
library,
if
can
make
all
these
rest
calls
to
kappa
because
kappa
the
rest
interface,
we
can
trigger
it
and
get
the
results.
A
The
first
thing,
though-
and
this
is
kind
of
a
little
bit
outside
of
captain-
but
it's
important-
it's
the
best
practice-
remember
one
of
the
things
I
say
it.
If
you're,
if
you
have
it
an
environment
we
deploy,
and
then
you
run
different
tests.
I
think
one
thing
we
also
need
to
solve
is
whatever
tool
you
have
in
the
backend
that
is
collecting
logs
or
metrics.
A
A
This
was
the
time-
and
this
was
the
moment
in
time
when
I
executed
this
particular
test,
because
then
you
can
also
get
more
specific
metrics
back
from
the
monitoring
tool,
so
taking
is
important,
but
then
from
Jenkins
now
I
just
say:
captain
here
is
my
SLI
here
is
my
SLO.
Captain
goes
off
pulls
in
all
the
metrics
from
the
the
monitoring
tool
or
your
log
analytics
tool
residence
that
you
could
still
look
at
the
dashboards,
but
it's
automating
that
it
is
then
validating
them.
A
It
is
then,
giving
you
the
option
to
obviously
look
at
the
heat
map,
but,
more
importantly,
you
also
can
automatically
pull
in
the
result,
which
means
you
can
fully
automate
your
pipeline.
So
if
a
successful
score
comes
back,
you
can
let
the
champions
build
pass
or
if
a
bad
result
comes
back,
you
can
let
it
fail-
and
this
is
now
completely
automated
right.
So
instead
of
spending
30
to
60
minutes
on
looking
at
different
reports,
it's
just
it's
just
you
know
fully
automated
and
so
that's
the
other
point
a
lot
of
time
savings
alright.
A
So
how
does
it
work?
Let
me
just
quickly
go
through
this
setting
up
captain
there's
a
lot
of
tutorials
now
I'm
going
to
do
it
today.
I'll
show
you
how
I
do
it
with
dynaTrace,
but
we
also
have
tutorials
out
there
for
Prometheus
and
other
tools,
but
in
a
nutshell,
captain
itself
runs
on
some
flavor
of
kubernetes.
So
that
means
you
need
some
type
of
kubernetes.
We
also
have
a
cool
installation
option
for
Ubuntu
Linux,
where
we
ever
install
micro
kubernetes.
A
So
you
really
just
need
a
Linux
machine
and,
in
my
case,
I
need
a
dynaTrace
tenon,
because
I'm
fetching
data
from
a
monitoring
tool,
but
then
really
what
it
is.
You
install
you
download
the
captain
CLI.
Then
you
say:
captain
install,
there's
two
installation
options
where
you
can
get
the
full
feature
set
of
captain
or
just
the
feature
set
for
quality
gates,
but
essentially,
what
kept
me
installs
for
you
in
your
kubernetes
cluster
is
different
components
that
captain
comes
with
the
ones
that
we
are
interested
in
is
the
evaluation
component.
A
Also,
what
you
might
be
interested
in
optional
is
notification,
so
captain
is
an
event-driven
system
and
you
can
notify
different
tools
like
like
slick
and
stuff,
like
that
I
also
Central
get
ops.
Captain
comes
with
a
git
repo
pre-installed,
because
every
time
you
are
giving
captain
and
us
a
lie
or
a
solo,
it
will
also
version
control
it
now
itself.
Captain
exposes
an
API,
endpoint
and
also
a
bridge,
that's
the
UI.
So
that's
what
we
basically
do
is
a
captain
install.
A
A
Man
and
every
time
when
we
create
a
project
and
add
resources,
then
a
Kip
will
automatically
push
this
into
the
into
its
github
repository.
So
it's
all
key
top
spaced.
All
right
now,
the
first
thing
is
so
you
need
to
install
kit,
it's
a
one-time
thing
now.
In
order
for
this
obviously
to
work,
you
need
a
system
there
is
that
value
that
when
you
deploy
something
and
you
run
tests,
you
can
also
pull
data
out
of
it.
A
So
in
my
case,
what
I
have
I
have
a
CI,
CT
environment
where,
where
I
deploy
you
know
my
menu
builds,
this
can
be
any
type
of
environment.
In
my
case,
I
have
installed
my
monitoring
tool
on
it.
I
mean
it's
an
agent-based
solution.
Dynatrace
automatically
get
infrastructure
insights,
then
every
time
I,
for
instance,
use
Jenkins
for
deploy
whether
I
deploy
a
docker
container.
Whether
this
is
a
kubernetes
cluster
and
I.
Do
a
cube,
CTL
apply
or
whether
I
deploy
a
Java
Web
whatever
it
is.
A
The
monitoring
tool
will
automatically
pick
this
up,
so
I
already
get
metrics,
and
then
this
is
the
same
type
that
I'm
using
I
always
say:
I'm,
not
very
proud
of
it,
because
it's
not
pretty,
but
it
does
a
good
job
in
explaining
what
we're
doing
here,
then
the
monitoring
tool
also
gives
you
insights
into
what's
happening
within
that
thing,
and
it
gives
you
things
like
response
time
failure
rates.
It
gives
you
memory
information
and
all
that
stuff.
All
these
s
lies
and
metrics
that
are
important.
Now.
A
The
last
thing
is,
you
obviously
need
to
have
some
tests,
because
when
you
deploy,
you
want
to
run
some
tests,
and
this
is
the
critical
point
here
where,
whatever
test
you
use
make
sure
to
look
at
what
monitoring
tool
the
log
analytics
tool
you
have.
So
you
can
combine
these
so
in
my
case,
I
will
be
using
a
custom,
developed
a
curl
script
and
also
chain
meter
later
on,
and
they
have
basically
I
just
I
just
integrated
them.
So
everytime
jmeter
execute
a
test
as
part
of
the
HTTP
request.
A
It
adds
a
little
token
to
it
and
says
hey.
This
is
my
home
page
call.
This
is
when
I
check
version.
This
is
going
to
do
an
echo.
This
is
going
to
do
invoke,
so
these
four
are
for
different
use
cases
of
my
sample
app,
but
as
part
of
the
request,
it
is
adding
that
context,
so
my
monitoring
tool
is
picking
this
up
and
the
monitoring
tool
can
then
later
give
me
metrics
on
what
the
response
time
of
home
page
version
echo
invoke.
A
What's
the
memory
usage
of
home
page
version,
echo
invoke
how
many
database
calls
were
made
by
each
individual
thing,
and
these
are
not
meaningful
metrics
that
we
are
interested
in
right,
especially
as
part
of
our
work
that
we
want
to
do
here.
We
do
not
only
want
to
look
at
response
time
and
failure
rate
I
think
in
the
more
distributed
world
we're
going.
A
We
need
to
look
at
metrics,
like
the
number
of
database
calls
that
are
made
the
number
of
exceptions
that
are
thrown
the
number
of
service
calls
that
are
made
to
the
backend,
because
then
we
can
also
detect
architectural
regressions,
meaning
if
your
developers
pushing
and
you
build
and
whatever
that
build
us.
It's
super
fast,
but
all
of
a
sudden
it
makes
fifty
thousand
new
database
calls
to
the
back-end.
A
That's
something
you
want
to
flag,
and
this
is
why
whatever
monitoring
or
log
analytics,
so
you
use
in
the
end
figure
out
how
you
can
connect
testing
tools
with
it
all
right
now.
If
we
have
this
right
and
if
you
have
data
that
is
interesting,
for
you
form
your
monitoring
tool
from
a
log
analytics
tool,
then
you
can
take
these
metrics
and
instead
of
Luke
having
it
in
a
nice
dashboard.
A
We
can
then
convert
them
into
remember
these
SLI
documents
and
SLO
files,
so
that,
instead
of
just
looking
at
it
at
a
dashboard
and
then
still
manually
doing
it,
we
can
let
cap
and
do
the
work
and
so
I'm
going
to
show
you
some
examples
later
on
right.
But
the
idea
is
right
once
we
have
done
this.
Yes,
we
can,
after
every
build
manually,
go
into
all
these
reporting
tools
and
all
that
stuff
and
then
manually
compare
it.
A
But
this
is
the
reason
why
we
build
captain,
because
it's
about
extracting
data
comparing
it
and
then
giving
you
a
result
all
right.
So
now
I'll
come
to
my
first
demo,
the
first
demo.
This
is
part
of
the
you
can
see
the
kids
up
link
on
top
get
up,
calm,
slash,
captain
sandbox,
slash,
kept
Jenkins
tutorial,
there's
a
couple
of
of
Jenkins
pipelines
that
you
can
just
run
and
the
one
that
I'm
going
to
show.
A
You
is
a
this
Kip
and
quality
gate
evaluation
and
when
I
run
it
what
it
does,
it
just
reaches
out
to
a
particular
environment
that
I
have
running.
There
is
already
under
load
and
it
just
pulls
back
metrics,
and
then
it
gives
me
the
result
and
then
hopefully,
I
will
see
something
like
this.
And
so
let
me
do
this:
let's
go
over
so
here
my
equipment,
quality,
gait,
evaluation,
I,
have
filled
with
parameters
and
again
these
are
just
very
focused
pipelines.
A
You
can
obviously
build
this
into
your
pipelines,
but
here
I'm,
saying
I
have
a
captain
project
that
is
called
GQ
project
I
have
a
stage
and
services.
This
is
the
way
captain
is
structured
in
project
stages
and
services.
I
can
also
specify
what
type
of
monitoring
tool
I
want
to
use:
I'm
choosing
dynaTrace
what
type
of
SLI
so
I
have
pre-configured
a
couple
of
SLI
definitions
and
then
I'm
saying
what
time
frame
do
I
want
to
evaluate.
A
So
in
this
case
the
system,
the
most
simplest
use
case,
where
I'm
lazy
I
don't
want
to
go
to
different
results,
different
different
different
tools
and
then
pulling
the
metrics
but
I
let
cap
and
do
this
for
me.
So
while
this
is
running
quickly,
you
can
obviously
look
at
the
source
code
as
well.
If
you
go
to
that
Jenkins
tutorial,
but
what
the
pipeline
actually
does.
A
There's
a
cap
in
its
stage
where
my
captain
Jenkins
library,
let's
also
and
get
you
just
call
in
it,
you
give
it
the
project
name,
the
stage
name,
you
give
it
the
name
of
the
service
in
the
monitoring
tool,
and
then
the
library
will
automatically
make
sure
that
your
captain
installation
has
this
project
created.
If
not,
it
will
create
the
project,
it
will
also
automatically
configure
monitoring.
So
everything
is
fully
automated.
You
don't
even
need
to
touch
captain.
A
It
will
create
the
service,
as
I
said,
and
it
will
then
also
upload.
My
SL
is
in
my
solos
and
also
my
monitoring
specific
configuration.
This
all
happens
out
of
my
Jenkins
pipeline.
This
means
these
files
SL
isin
as
solos.
In
my
case,
they
come
from
our
source
code,
repo.
That
means
SL.
Aizen
is
SLOS.
A
They
sit
next
to
your
source
code
next
to
your
tests,
because
they
also
belong
together,
but
now
I
hand
it
off
to
cap
and
then
say
kept
before
this
build
I
want
you
to
use
these
files,
then
I'm
triggering
the
evaluation.
Does
it
library
function
is
called
st.
start
evaluation
event
for
a
particular
timeframe,
let's
say
600
seconds,
and
then
what
captain
will
do?
It
will
take
that
timeframe.
A
It
will
reach
out
to
the
different
monitoring
tools
and
it
will
basically
execute
these
queries
that
we
specified
is
in
the
correct
context,
and
it's
also
pacing
in
the
right
timeframe
right.
It's
transforming
these
queries
and
then
it's
pulling
back
the
metric
and
then
at
the
end.
I
can
also
because
this
is
an
asynchronous
function
depending
on
the
tool.
It
may
take
half
a
minute
a
minute.
A
If
the
data
is
there
for
that
time
frame,
then
you
can
say:
wait
for
for
evaluation
done
so
you're
polling
and
then
whatever
comes
back
I
can
either
let
the
pipeline
succeed
or
I
can
let
it
fail.
So
if
I
switch
back
now,
I
can
actually
see
the
pipeline
failed.
Why
I
didn't
fail
if
I
move
over
captain
score
was
60
result
fail
now?
Why
is
that?
Let's
have
a
look
at
this.
Alright
I
can
click
on
it.
So
these
are
the
artifacts
there's
a
there's.
A
lot
of
time.
A
In
my
example,
I
put
everything
out.
There's
a
captain
HTML,
and
this
is
something
I
want
to
improve,
there's
a
link
to
the
captain's
bridge,
so
the
captain's
bridge
is
now
the
UI
and
the
nice
thing
is
I
have
a
link
here.
That
directly
gets
me
to
this
particular
run.
So
every
time
I
execute
something
in
captain
I
get
a
context
that
he
we
call
it
the
captain
context
and
this
context
that
he
keeps
everything
in
place
that
belongs
to
this
execution.
So
with
this
link
I
get
to
hey.
This
was
a
captain
quality
evaluation.
A
A
It
did
the
evaluation,
and
here
I
see
all
the
results
and,
as
you
can
see
here,
these
are
all
my
old
results
and
I
can
go
back
in
history
and
it
seems
my
build
is
really
crappy,
but
it's
basically
data
that
comes
back
that
is
pulled
in
from
this
is
the
app
that
I
have
running
right.
So
this
is
the
this
is
the
one
after
that
running
here
is
the
monitoring
tool
behind
the
scenes.
That
is
monitoring
this
particular
service.
So
let
me
just
show
you:
this
is
my
service
that
I've
take
with
evil
service.
A
Here
we
go
so
again,
I
could
go
in
here
and
click
through
all
the
reports
and
get
the
data,
but
why
right?
This
is
what
we've
automated.
So
it
means.
Where
are
we
here?
We
are
I,
see
the
results.
Every
single
SLI,
with
the
actual
value
why
it
failed
and
I
can
also
switch
over
to
the
chart.
So
now
I
can
write.
A
So
this
is
you
can
you
can
click
on
all
these
individual
columns
here,
metrics
and
then
you
can
see,
however,
individual
metrics
kind
of
evolve
over
time,
because
if
you
want
to
see
trends,
cool
thing
go,
this
is
you
know
fully
fully
automated
the
next
use
case.
I
want
to
quickly
show
you
and
I
know
we
have
about
probably
10
more
minutes.
A
Let
me
go
here
the
next
one.
So
what
their
first
use
case
was
you
have
an
environment
you
have
deploy.
You
have
run
some
tests
from
your
pipeline
and
you
know
which
time
frame
the
test
ran
and
I
basically
kept
and
go
often
in
the
last
at
time
frame.
The
second
one
part
of
the
tutorial
is
I
actually
built
in
a
little
testing
script.
A
So
this
is
a
very
simple
pipeline
that
just
executes
curl
comments
in
a
loop
against
different
URLs
of
my
app
very
simple
Pullman's
load
testing
tool
as
I
always
call
it,
but
it
does
the
job.
So
in
this
case,
right,
if
you
have
a
pipeline
where
you're
also
running
test,
is
part
of
your
pipeline,
let
me
go
back.
This
is
the
it's
called
simple
load.
Testing
with
captain
quality
gates.
I
can
do
build
with
parameters,
so
here
I
specify
again
captain
project
stage
and
service
the
pipeline
will
automatically
create
this
construct.
A
I,
say
monitoring
now:
I
want
to
switch
to
a
different
SLI
proof
test
and
I
will
show
the
SLS,
because
now
this
will
run
a
little
longer.
I
basically
say
what
is
the
URL
I
want
to
test?
This
is
my
sample
app.
My
simple
load
testing
tool.
I
can
pass
in
here
some
some
endpoints.
It
should
test
and
the
load
test
should
run
for,
let's
say
3
minutes
go
off
so
now
what
my
pipeline
is
doing.
It's
initializing
captain.
It
is
then
running
the
test.
A
A
You
remember
earlier:
I
had
SLI
basic
and
proof
test,
so
I
have
to
accessorize
in
my
tutorials
one
is
just
basic.
These
are
three
Metro
five
metrics
throughput
error
rate
respond
time,
so
these
are
the
logical
names
and
here's
the
query
behind
the
scenes
that
gets
executed
against
the
tool
that
you
are
get
the
data
from
basic
means,
just
the
basic
metrics,
the
proof
test
metric.
This
is
now
extended
by
particular
metrics
for
for
met
metrics,
that
give
me
data
from
a
particular
test
run,
so
I
still
have
my
file
from
the
top.
A
What's
the
overall
throughput
overall
error
rate
and
so
on,
but
now
I
have
what's
the
response.
Time
of
my
test
case
invoke
my
echo
my
version,
my
homepage
right
because
I'm
executing
these
different
tests-
and
so
you
can
add
as
many
service
level
indicators
here
as
long
as
your
tool
can
also
extract
that
data,
and
this
is
where
at
least
and
dynaTrace
perspective.
I
can
say.
A
Dynatrace
give
me
the
response
time
of
a
test
case
where
the
test
step
was
version
or
homepage
or
give
me
the
number
of
service
calls,
because
my
service
is
making
bacon
service
calls
to
other
services.
So
I
can
get
the
number
of
bacon
service
calls.
It
makes
and
pull
in
the
metrics,
and
so
you
define
what
metrics
you
want
for
your
particular.
You
know
project
or
service
or
pipeline
stage,
and
then
you
also
need
the
s.
Ellos,
remember
the
SLO,
so
those
are
showed
you
in
the
slides
earlier
where
you
list.
This
is
my
eyes.
A
This
is
amazing,
eyes
my
eyes,
so
you
can
actually
say:
hey
captain
I
not
only
want
you
to
pull
the
data
out,
but
I
also
want
you
to
compare
them
against
a
pass
and
warning
criteria.
Now
here's
the
cool,
some
cool
things.
What
I
didn't
mentioned
earlier?
You
can
also
weight
different
metrics,
so
you
can
say:
error
rate,
for
instance,
is
more
important
than
others
by
default,
every
se
like
it's
a
weight
of
one,
but
you
can
change
the
weight.
Also,
what
you
can
do
you
can
do
something
like
this.
It's
called
key
SLI.
A
True,
if
you
say
key
SLI
true,
that
means
this
is
a
key
SLI.
If
this
fails,
the
whole
test
fails.
So
maybe
you
have
a
metric
that
is
so
important
that
if
this
fails,
everything
should
fail
alright.
So
this
is
also
there
yeah
and
I
have
the
basic
version
and
I
have
the
proof
test
version
right,
so
you
can
see
here
it's
just
this
contains
mometrix.
A
Also
you
can
you
can
just
reference
an
SLI
without
having
person,
then
captain
is
just
with
showing
you
the
value.
Why
would
you
do
this?
Well,
maybe
you
don't
yet
know
what
a
good
threshold
is
right.
So
that's
the
option
here,
good,
let's
go
back
to
my
pipeline,
so
the
tests
are
running
now
and
they'd
see
it
earlier,
took
three
minutes
and
then
it
triggers
the
quality
gate
and
it
takes
about
two
minutes
until
all
the
tests,
the
data
comes
back
instead
of
waiting.
A
Here,
let
me
just
go
back
to
the
previous
run
from
from
build
number
nine.
Let's
do
this
because
execute
it
the
same
thing
just
earlier
to
save
some
time
all
right.
It
gives
me
in
here
so
this
was
build.
Number
nine-
and
you
can
now
see
here
we're
getting
all
of
these
metrics,
so
it's
it
seems
this
was
actually
a
pretty
good
build
except
the
throughput
didn't
work,
but
overall
I
received
92
out
of
100
points.
So
that's
why
I
was
considered
a
good
build.
I
can
get
all
the
results
here
for
every
single
metric.
A
It's
also
pretty
cool
and
yeah
I.
Think
that's
fine.
So
last
thing
I
want
to
show
you
before.
We
then
open
it
up
for
Q&A,
oh
yeah,
it's
just
in
my
slides
that
I'll
I'm
also
going
to
share
you.
I
I,
just
explain
how
my
Jenkins
pipelines
actually
called
the
library.
So
you
know
it's
it's
just
in
this
case.
I
do
not
keep
any
this
before.
A
The
current
timestamp
is
the
end
timeframe
and
then
the
start
time
that
was
there
was
stored
with
the
mark
evaluation
time
from
so
you
have
the
exact
time
frame
that
should
be
evaluated,
but
the
rest
is
the
same
goes
in
and
then
and
then
here
we
go
perfect
all
right.
Let's
see
last
use
case,
they
don't
want
to
show
you
because
you
know
I'm
a
performance
enthusiast.
A
One
thing
that
we
also
addressed
with
captain
is:
we
enable
you
to
make
performance
testing
very
easy
to
come
up
with
a
performance
driven
culture
as
we
call
it
so
I'm
sure
some
of
you
out
there
and
I
know
some
I
show
some
names
earlier
in
the
attendee
list.
There
are
some
performance
experts,
so
if
you
want
to
integrate
performance
testing
in
your
pipeline,
there's
a
lot
of
questions
you
have
to
answer,
you
know:
where
do
we
run
at
what
type
of
tests
to
be
run?
A
Where
do
we
store
the
data
and
so
on
and
so
forth?
And
and
that's
all
you
know
very
hard
questions
some
harder
than
others
and
it's
you
can
all
build
your
own
testing
tools.
You
can
all
use
commercial,
so
we
can
do
it
build
it
yourself,
like
do,
there's
a
lot
of
do
with
yourself
material
out
there,
but
what
we
also
said
with
captain.
We
also
want
to
work
a
straight
text
test
execution,
so
one
thing
that
captain
can
do
it
is:
it
can
trigger
other
testing
tools.
A
We
have
one
integration,
for
instance,
with
new
loads.
They
provide
a
cloud-based
service
and
also
service
where
they
can
launch
their
load
generators
in
kubernetes,
but
we
also
have
an
integration
with
jmeter,
and
this
is
what
I
want
to
show
you
now
is,
if
you
want
to
integrate
testing,
but
you
don't
have
thought
about
testing
for
structure
it
for
load
testing,
but
you
want
to
run
some
simple
chain:
Meter
scripts,
for
instance,
then
you
can
change
your
Jenkins
pipeline
from
the
way
it
is
on
top
right.
A
Instead
of
trying
to
figure
out
how
to
stand
up
your
test
infrastructure
execute
the
tests.
The
only
thing
you
need
to
do
is
you
need
to
tell
captain
hey
captain
I
just
deployed.
My
app
here
is
my
URL
of
my
app.
Now
you
go
off
and
you
run
the
test
scripts.
The
only
thing
you
need
to
give
captain
is
the
test
script
itself,
then
again,
whatever
tool
it
is,
and
then
what
captain
will
actually
do.
A
It
will
execute
the
tests
after
the
tests
report
back
to
captain
that
the
tests
are
complete,
it
will
then
go
through
its
regular
process
of
querying
the
slis,
comparing
them
against
your
solos
and
giving
you
back
the
result
right.
So
that's
kind
of
the
overview
here
all
right.
So
let
me
show
you
this
as
well.
I
have
another
pipeline
captain
performance
as
a
service.
That's
the
way,
I
call
it
and
it's
it's
again.
It
is
just
screenshots
on
how
it
should
look
like.
So
let's
try
this.
A
And
then,
in
this
case
again,
I
always
reference
a
kept
project.
Now,
in
this
case,
I
wanted
to
again.
Dynatrace
is
monitoring
tool
here,
I'm
using
jmeter,
because
that's
an
integration
that
comes
with
captain
and
what
captain
can
also
do
you
can
app,
you
can
give
it
the
chain
meter
scripts
and
you
can
define
different
workloads,
so
I
want
to
run
a
quick
test.
I
think
performance.
A
10
is
10
virtual
users,
very
quick,
but
I
basically
say
please
here
is
my
URL
go
off
run
this
particular
test
use
these
as
allies
for
evaluation
and
then
wait
60
60
minutes
in
maximum
until
the
results
are
completely
done,
we'll
let
the
pipeline
fail
anyway,
and
that's
it.
So,
let's
try
this
out
and
let's
see
how
it
runs.
A
A
There
is
a
tutorials
and
samples.
I
also
have
another
video
linked
here
and
all
the
different
use
cases
that
he
basically
just
saw
important
because
I
know
a
lot
of
you
are
interested
in
in
the
actual
Jenkins
files.
So,
for
instance,
the
the
quality
gate
here
is
the
captain
evaluation
Jenkins
file
and
it's
it
is
really
straightforward.
I'm,
including
my
my
my
captain
library,
then
I
have
my
captain
object
here
and
oh
there's
probably
other
ways
to
do
this
as
well.
A
But
I
include
my
captain
library
and
then
what
I
do
is
a
captain
in
it
I
give
it
a
project
name,
the
service
name,
the
stage
and
the
monitoring
tool.
These
are
all
parameters
that
I've
all
of
my
pipeline
parameters
and
then
what
I
do
I
give
captain
all
the
necessary
files.
I
say:
captain
captain
edge
resource.
So
here
is
my
dynaTrace
configuration
file,
my
sli
in
maya,
solo
and,
and
then
this
is
my
a
to
initialize
captain
and
then
in
my
trigger
quality
gate.
I
just
say:
hey
captain
st.
A
star
devaluation
event
where
you
can
give
it
a
start
time
and
an
end
time,
and
my
library
supports
different
different
ways.
You
can
either
space
if
a
full
time
stamp.
If
you
note
start
an
end,
you
can
only
specify
a
start
time
frame.
Then
it
will
go
from
start
to
now.
You
can
also
specify
just
numbers
where
you
can
say:
let's
say
60
minutes
in
the
past,
until
0
and
so
now
or
60
minutes
until
30
minutes,
then
it's
kind
of
a
like
a
time
window
in
minutes.
A
This
also
works,
and
the
nice
thing
is
what
you
get
back
is
the
kept
in
context.
It's
the
unique
ID
that
then
allows
you
also
to
query
later
on
what
data
status,
and
it
also
allows
you
to
open
up
the
bridge
right.
The
kept
in
context
is
also
something
that
allows
you
to
go
into
the
deep
link
and
then
at
the
end,
wait
for
result.
I
just
have
a
function
here.
A
It
is
called
wait
for
evaluation
done
where
you
could
say:
hey
wait
until
a
maximum
time
out,
because
sometimes
it
can
you
know
maybe
something
happens
and
you
don't
want
to
wait
forever.
So
you
can
specify
timeout
and
you
can
also
say,
set
build
with
all
true
in
this
case.
This
function
will
also
set
the
build
result
of
the
change
in
spy
plan.
You
know
successful,
failed
or
or
in
stable,
depending
on
the
result.
That
comes
back.
So
this
is
this.
A
Alright,
as
you
can
see,
these
tests
may
run
a
little
longer,
but
I
hope
I've
shown
you
pretty
much
most
of
the
things
I
wanted
to
show
you
just
to
complete
the
slides
and
then
very
happy
and
to
open
it
up
for
Q&A
the
in
the
slides.
Just
what
I
showed
you
in
the
walkthrough
I
always
explain
what
type
of
calls
I
make
from
Jenkins
to
the
captain
library
to
initialize
the
project
to
upload
the
right
resources
in
this
case,
for
when,
when
captain
should
also
execute
my
my
tests,
I
have
to
upload
my
test.
A
Scripts,
obviously
like
in
this
case
g-meter
and
then,
instead
of
saying,
evaluate
I,
want
captain
to
actually
do
the
test.
So
a
doesn't
there's
a
special
function
that
is
called
sense,
deployment
finished
event,
I
basically
say:
captain
I
have
deployed
my
app
is
deployed.
Here's
the
URL.
Now
you
go
off
and
do
your
thing.
This
is
where
then,
captain
triggers
jmeter
execute
the
tests
based
on
the
workload
configuration
I
gave
it.
It
also
sends
events
to
the
monitoring
tools
is
also
a
nice
integration.
A
Kept
North
automatically
sends
event
to
dynaTrace,
to
Prometheus
or
to
other
tools
informing
that
tool
about
what
is
actually
happening.
Then
the
SLI
magic
happens
again
right
in
short
and
then
at
the
end
you
get
the
result
back.
Okay,
so
this
is
I
think
this
is.
This
should
show
you
how
this
can
be
done.
All
the
different
use
cases
for
the
the
chamber
that
you
media
integration.
A
We
also
built
the
option
of
not
only
running
one
jmeter
script,
but
you
can
upload
multiple
scripts
and
then
you
can
also
specify
through
workloads
what
type
of
workload
you
want
to
execute.
Remember
in
my
drop
down
box,
I
specified
performance,
50
or
performance
10.
So
cap
will
then
execute
whatever
I've
specified
here
and
and
pass
in
things
like
loop
count
number
of
virtual
users.
So
these
are
parameters
that
go
into
the
script
yeah
and
to
wrap
it
up.
A
If
you're
interested
the
champions
shared
library
right,
that's
the
one
that
my
tutorial
is
using
is
all
on
github
and
it's
hopefully
you
know
easy
to
use
on
the
github
page.
If
you
look
at
that
github
page,
then,
where
is
it?
Here's
the
kid
library
I
tried
to
to
do
a
reasonably
good
job
of
showing
some
examples
on
how
this
all
works?
How
did
the
quality
get
evaluation?
How
do
the
performance
testing
also?
How
do
the
progressive
delivery,
because
captain
can
do
much
more?
A
So
there's
a
lot
of
examples
in
here
and
in
order
for
Jenkins
to
know
to
which
captain
to
talk
to
right
now,
I'm
expecting
global
properties,
global
environment
variables
with
these
names,
this
is
obviously
something
we
want
to
improve
on.
I
also
Inc
allow
credentials,
but
that's
it
yeah
and
that's
actually
the
current
plans.
So
we
want
to
support
credentials
versus
the
environment
variables.
A
We
want
to
potentially
create
a
champions
plugin
to
also
visualize
some
of
the
data
from
captain
in
inch
Incans
directly,
like
the
heat
map
be
great,
and
we
also
want
to
provide
some
callbacks,
because
one
of
the
capabilities
that
captain
also
has
kept
them
can
orchestrate
the
complete
end-to-end
delivery.
I
can
do
deployment,
they
can
do
testing,
we
can
do
evaluation
and
for
the
individual
pieces
we
can
even
say
captain
for
deployment.
Call.
A
This
Jenkins
pipeline
for
testing
do
this
in
for
evaluation,
obviously
do
this,
and
if
you
are
calling
a
Jenkins
pipeline
from
Captain
to
do
a
particular
test,
and
that
pipeline
also
needs
to
report
back
to
captain.
There
is
a
ready
way.
There
is
a
service
out
there
that
connects
these
two,
but
we
want
to
make
it
even
easier
and
with
that
I
want
to
say
thanks
for
listening,
I
really
hope.
There's
a
lot
of
questions
now,
and
hopefully
this
was
useful
and
yeah
olenka
mark
I
wonder
hand.
It's
pick
over.
B
To
you,
yeah
thanks
a
lot
for
a
presentation
yeah.
It
was
definitely
useful
and
it's
great
to
see
performance
testing
at
scale.
So
I
was
using
Jenkins
g-meter
and
other
tools
for
awhile
well,
one
before
kubernetes
was
created
and
one
before
other
tools
were
created
so
I.
If
anyone
who
has
any
questions,
please
use
some
kony
again,
we
can
grant
those
permissions.
If
you
want
to
ask
and
yep
I
have
a
few
questions
say
in
unless
mark
has
questions.
First,.
C
I
have
I
have
questions
in
mind.
I
have
a
page
full
of
scribbles
that
I
took
well
address,
was
talking
thanks,
very
much
Andres,
so
so,
but
if
there,
if
there
are
participants,
I'm
happy
to
defer
to
the
participants,
I
just
don't
see
any
in
the
participant
Q&A
right
now
so
I'd
like
to
go
with
mine
or
like
are
you
okay,
if
I
ask
yeah
just
abut
all
right,
so
so
Andres
I'm
I'm
deeply
deeply
interested
in
how
to
accelerate
these
kinds
of
evaluations.
C
Are
there
any
insights
you
could
share
on
things
you've
learned
as
you've
developed
captain
as
you've
interacted
with
it?
How
do
you
accelerate
the
throughput
of
getting
these
kinds
of
results?
They
seem
like
performance
benchmarks,
are
probably
expensive
and
hard
to
do
and
hard
to
repeat
what?
What
guidance
would
you
offer
us
for
how.
A
To
accelerate
that's
a
cool,
that's
a
good
point,
so
I
agree
with
your
performance
benchmarks
for
some
organizations
of
depending
on
the
project,
it's
hard,
but
one
thing
that
I've
hinted
to
it's
not
only
the
response
time
and
the
throughput,
but
it
might
be
some
basic
metrics
that
you
can
also
look
at
when
you
just
run
your
functional
tests
so,
for
instance,
a
metric
that
I
love
because
I've
seen
so
many
applications
fail.
Is
things
like
how
many
database
calls
do
we
see
after
we
execute
this
particular
functional
test?
How
many
log
files?
A
A
But
the
thing
is,
you
need
to
look
at
the
metric
and
you
need
to
get
this
metric,
but
the
nice
thing
with
this
now
is
keptin
all
make
sure
that
you
automatically
get
this
metric
from
whatever
tool.
Obviously
that
provides
it
and
automatically
compares
it
stores
it
for
you
in
history.
You
see
trends,
but
if
you
don't
so
the
way
enter
equation.
If
you
don't
yet
have
a
performance
based
practice.
A
Well,
first
of
all
get
started
with
it,
because
it's
not
that
hard
to
rather
chip
me
the
test
or
what
I
did
my
my-my-my
Pullman's
load
testing
tool?
It's
a
curl
script
come
on.
Everybody
can
write
a
while
loop
with
some
curls
there's.
No
just
you
look
at
my
example.
What
I
will
read,
but
if
you
have
some
API
tests
and
functional
tests,
I'm
pretty
sure
there's
some
of
these
I
call
them.
A
C
D
A
The
service
we
have
here
this
will
basically
just
run
j
meter
from
a
so
it
deploys
j
meter
in
a
container
in
in
kubernetes,
where
captain
runs
and
then
just
runs
it
from
there.
I
would
encourage
anybody
out
there
to
help
us
extend
that
that
service
today,
maybe
you
know,
reach
out
and
and
run
it
from
from
different
from
different
locations
or
I
know.
A
There
are
solutions
out
there
that
can
already
deploy
a
meter
in
different
places,
and
then
you
just
have
an
API
call
that
you
need
to
make
from
captain
in
order
to
trigger
it
and
but
right
now
the
current
thing
that
comes
with
captain
itself.
It
just
deploys
a
container
and
then
runs
your
team.
You
script
in
that
container,
it's
what
it
does.
A
B
A
Don't
do
random
reporting
rather
remember
what
captain
really
does
captain
isn't?
Work
is
greater
so
with
captain
we
are
triggering
different
things
like
we
are
triggering.
You
know
somebody
needs
to
deploy
something.
Somebody
needs
to
run
a
test.
Somebody
needs
to
evaluate
so
for
evaluation.
The
built
in
captain
quality
gate
comes
in
for
test
execution.
You
can
use
the
Qi
meter
service,
but
you
can
also
trigger
whatever
else
you
want.
So,
for
instance,
one
others.
One
of
the
things
here
is
the
generic
execute
to
service
of
captain.
A
This
is
also
a
cool,
very
I
would
say
easy-to-use
thing
where
you
can
either
define
an
HTTP
file
or
a
shell
script
that
gets
then
executed
based
on
a
certain
captain
event.
So,
for
instance,
you
can
say
every
time
captain
makes
a
configuration
change.
Please
call
this
shell
script
because
you
want
to
do
something
or
if
captain
sends
internally
a
star
test
event.
You
can
execute
this.
A
So
that
means
you:
can
it's
you're
very
flexible
with
what
you
do,
but
remember
a
captain
is
event-driven
and
then
one
thing
we
do
is
with
the
quality
gate
where
we
pull
in
metrics
from
different
days
versus.
But
we
don't
do
it
life.
You
could
build
a
component
that
does
it
life,
but
I
believe
that
this
is
the
job
of
your
monitoring
tools
and
your
log
equation
tools
and
in
your
tests,
yeah
I
think
that's
a
different
job.
A
B
A
I
mean
obviously
again
I'm
a
little,
not
biased
here,
but
I'm
lucky
I
have
dynaTrace,
and
so
all
the
data
ends
up
in
damaged
ways
anyway.
So
I,
my
only
data,
says
I
need
for
my
stuff.
It's
dynaTrace,
because
I
get
after
the
log,
metrics
I
get
the
infrastructure
metrics
the
process.
Metrics
the
the
test,
matrix
I,
get
everything
in
one
tool,
but
we
build
captain
agnostic
because
we
wanted
to
me
know:
there's
a
lot
of
people
out
there
that
need
different
data
sources
and
that's
just
the
real
world,
and
so.
C
A
So
I
can
encourage
you
to
if
you're
interested
in
captain,
then
I
can
encourage
you
on
the
tutorials
page
stories
of
captains
or
the
sage
in
general.
The
concepts,
though,
of
performance
engineering,
so
I've,
been
doing
performance
engineering
for
many
many
years.
So
there's
access.
There
is
a
trace.
I
just
wrote
a
a
guy
to
automated
a
story
driven
performance
engineering
which
again
I
know
it's.
You
see
stylish
ways,
but
it
explains
all
the
concepts
of
SL
eyes
and
SL
O's
and
and
how
you
can
start
building
a
necessary
driven
mindset.
A
So
I
think
that
could
be
an
interesting
to
start
and
another
one
that
I
wrote
actually
three
years
ago
is
trades
of
a
performance
engineer
in
2020,
I
wrote
this
2017,
I,
believe
and
I
believe
it
still
holds
true.
So
what
does
it
need
to
to
kind
of
start
with
the
performance
movement
within
a
company
and
there's
a
lot
of
other
great
resources
out
there
as
well?
I
really
love.
A
My
friends
had
proof
bytes
Pervez
podcast
mark
Tomlinson,
James
boolean
Co.
They
are
great.
They've
been
talking
about
performance
engineering
for
many
years.
Alexander
Patel
Co
is
one
of
the
the
people
that
I
also
know
he's
been
writing
a
lot
of
blogs.
Also
I
have
another
podcast.
It's
called
pure
performance.
A
B
A
So
I
gotta
tell
you
that
you
know
my
I've
been
I've
been
trained
as
a
as
a
software
engineer
when
I
started
so
I
think
engineering
and
coding
is
easy.
For
me.
It
always
gets.
It
always
takes
getting
used
to
a
particular
type
of
language
in
a
particular
type
of
you
know,
terminology
obviously,
overall
I
found
it
pretty
simple
and
straightforward
I
mean
I,
did
a
lot
of
trials
and
errors
and
all
thread.
Why
does
this
stupid
thing
not
compile?
What's
what's
wrong
again?
Where
did
I
miss
something?
A
Well,
overall,
it
was.
It
was
super
easy
there's.
So
many
I
found
the
tutorial
I.
Don't
remember,
I,
don't
remember
the
tutorial
that
I
found,
but
it
was
a
tutorial
kind
of
like
my
first
Jenkins
pipeline
library
and
and
I
just
took
it
and
started
from
there
I
think
from
a
troubleshooting
perspective.
It
might
be
sometimes
a
little
easier
so
to
not
work
with
too
much
log.
Outputs
to
figure
out
where,
like
I,
don't
know
like
a
debugging
capability,
would
sometimes
be
cooler.
Maybe
something
like
this
exists.
I
don't
know
because
I
am
you
know.
A
B
A
B
A
A
B
So
in
principle,
you
have
to
write
a
plug-in.
At
the
same
time,
there
are
plugins
like
a
port
plug-in
or
something
like
that
which
take
data,
and
can
you
visualize
this
data?
Okay,
yeah
also
a
plug-in,
for
example,
plot
plug-in
just
a
plot:
okay,
yeah,
so
plot
uses
a
bin
upload
to
the
data.
So
obviously
you
can't
awesome
our
visualization
there,
but
it's
just
something
straight
forward:
yeah
there
are
other
plugins
which
do
more
with
regards
for
graphing
and
right
now,
yet
maybe
you've
heard
about
warnings
and
G
plug-in.
So
it's.
C
B
In
basically
will
gates
over
analysis
and
reporting
tools.
At
the
moment,
there
is
a
lot
of
enhancements
away
in
terms
of
digitization
and
reporting
tools.
So
last
week
we
had
ux
harvest
and
we
have
not
was
presenting
how
to
do
but
reporting
with
jenkins,
basically
using
standard
javascript
libraries,
and
they
believe
that
it
would
be
possible
to
create
plugins,
which
the
intake,
generic
camels
or
json
or
whatever,
and
build
fancy
a
charts
like
this.
We
already
have
yeah.
A
C
Bad
thanks
so
now
I
had
assumed
in
order
to
get
throughput
that
you
were
probably
running
many
of
these
tests
in
parallel.
If
they
are
running
in
parallel,
how
do
you
associate
a
change
in?
How
do
you
associate
something
at
the
backend
like
database?
Hits
to
your
which
front-end
test
caused,
or
do
you
not
is
it
I
do
typically
don't
run
parallel
around.
A
You
do
run
parallel
tests,
and
this
is
let
me
just
quickly
go
to
my
excuse
me
my
Qi
meter
here.
So
this
is
what
I
mentioned.
The
beginning
is
tagging
of
requests.
So
when
jmeter
executes
that
requests
to
a
URL,
then
what
I
have
done?
I
have
a
pin
shell
preprocessor,
where
I'm,
adding
an
additional
HTTP
header
on
every
request
in
Ted
header,
I
call
it
external
phase.
A
Test
includes
name
value,
pairs,
Ellison
load,
script,
name,
TSN,
test,
step,
name,
ltn,
loyal
test
name,
so
there's
a
script
name,
the
step
name
to
test
name
and
the
virtual
user.
So
I
can
be
I,
basically
add
context.
Data
to
every
request,
I
send
to
the
application
I
my
case,
my
app
is
monitored
by
dynaTrace
and
we
do
distributed
tracing.
So
that
means
what
I
can
say.
A
I
can
say
on
the
diagnostic
tools
and
I
can
now
say:
I
want
our
half
it's
a
number
of
it's
a
number
of
calls
to
our
services,
not
split
by
request
name,
because
these
are
URLs
right.
So
here
on
the
bottom,
you
see
the
URLs
API
blah
blah,
there's
a
lot
of
stuff,
but
now
I
can
say:
I
wanna,
have
it
split
by
the
request
edge
requested
to
build
it's
the
test,
step
name
and
I
only
wanna
have
it.
A
A
So
if
I,
if
I
look,
if
I
show
you
what
this
what's
behind
the
scenes
and
again
whether
it's
dynaTrace
or
I,
don't
know
you
really
get
dynamics
data
talk,
I'm
sure
they
do
it
in
a
similar
way.
But
in
our
case,
I,
look
at
every
single
request
that
my
monitoring
tool
captured
and
on
every
request
we
call
them
prepares
our
end-to-end
trace
I
can
go
on
every
single
request
and
and
and
in
my
case
read
so
this
is
the
stuff
that
manager
is
extracted.
A
For
me,
this
is
information
that
the
load
testing
tool
put
on
the
requests.
So
this
is
the
external
trace
test
error,
but
dynaTrace
is
extracting
the
individual
pieces
and
therefore
I
have
this
metadata
on
every
trace.
But
then
dynaTrace
also
gives
me
the
ability
to
say
well,
I,
don't
I'm
not
interested
in
every
trace,
I'm
interested
in
metrics,
based
on
traces
with
particular
piece
of
metadata
and
now.
This
is
then
also
the
data
that
I
can
put
on.
My
charts
right,
like
in
this
case
I,
have
my
dashboard
that
says
top
CPU
consuming
test
step.
A
So
look
at
all
the
requests,
look
at
the
CPU
consumption,
but
split
it
by
a
test
name
or
top
pick
and
database
club
topic
and
calls,
and
now,
as
we
have
a
metric
I,
can
also
put
these
metrics
into
my
it's
alive.
With
my
cell
I,
here's,
my
cell
I
number
of
service
calls
all
in
a
particular
test
case.
This
is
the
way
I
curate
from
dynaTrace,
and
this
is
also
how
captain
can
automatically
query
it
and
then
compare
it
with
previous
builds.
B
A
A
You
have
process
with
tool
integrations
all
bundled
in
in
your
pipeline
code,
and
then
you
start
from
one
type
and
then
you
copy
it
over
and
you
have
10
15
200
different
copies,
and
the
problem,
though,
really
is
that
you
don't
really
have
a
separation
of
concern.
You
don't
have
the
process
clearly
defined
somewhere
and
the
tools
that
should
do
this.
A
So
what
kept
the
escaping
is
an
event-based
orchestration
plane
or
actually
we
like
to
call
it
now,
a
choreography
engine
where
captain
you
in
again,
if
you
look
at
the
tutorials
the
way,
the
way
captain
works,
you
define
a
so
called
shipyard
file.
So,
instead
of
defining
pipelines
and
stages
and
then
writing
pipeline
code,
you
say:
I
have
two
stages:
I
have
a
staging
stage
and
I
have
a
production
stage
in
staging
I
want
to
do
a
direct
deployment
strategy
and
I
want
to
run
performance
tests
in
production.
A
I
want
to
do
a
Bluegreen
deployment
and
I
also
want
around
performance
tests
and
I
also
have
a
remediation
strategy.
So
this
is
on
the
one
side
where
you
define
the
process,
and
then
you
have
another
option
where
you
can
say:
ok,
which
tools
should
then
actually
be
called
or
triggered
when
there's
a
new
artifact
to
be
deployed,
so
kepner's,
really
an
event-driven
orchestration
engine
where
you
can
give
it
an
artifact
whether
it
is
a
container
or
a
jar
file
and
then
based
on
that
process
definition.
A
It
then
internally
triggers
events,
and
these
events
are
then
picked
up
by
tools
that
can
do
a
particular
job.
It
could
be.
I
have
maybe
a
Jenkins
pipeline
that
can
deploy
my
java
app
into
certain
environment.
Then
I
can
use,
for
instance,
the
captain
Jenkins
service,
keep
Jenkins
service
and
get
up
to
call
a
particular
Jenkins
pipeline.
A
Alright,
so
I
can
say:
I
want,
for,
let's
say,
a
deployment
for
a
configuration
change
event.
When
somebody
wants
to
change
the
configuration
you
build,
then
please
call
this
particular
Jenkins
pipeline.
When
this
pipeline
is
done,
then
kapnos.
Okay,
now
deployment
is
done.
That's
great.
What's
the
next
thing
I
need
to
do.
Okay,
I
need
to
run
tests
so
now
it
sends
an
event
Hey
who
can
execute
performance
tests
and
then
maybe
my
Qi
meter
script
picks
it
up,
and
then
the
G
meter
runs
the
tests.
So
it's
decoupling.
A
First
of
all,
actions
of
processes
from
tools
and
the
beneficiary
I
think
is
a
developer,
but
you
can
still
use
all
your
existing
tools
that
you
wanted
to
individual
builds.
So
we're
not
replacing
anything
which
is
work
is
trading
it.
This
is
for
continuous
delivery
and
for
operations.
This
is
where
the
remediation
strategy
come
in.
We
all
have
a
lot
of
people
that
are,
you
know,
running
large
systems
in
production
and
when
a
problem
happens,
when
they
can
notify
been
alert,
they
want
to
execute
certain
actions,
and
it's
also
an
event-driven
model.
A
Problem
comes
in
based
on
the
problem.
Execute
a
certain
task,
wait
for
the
tool
to
say:
did
it
work?
Yes,
no
really
evaluates
that
it
worked.
If
not
do
something
else,
so
captain
also
heads
and
in
an
orchestration
engine
for
auto
remediation.
This
is
what
captain
does
overall
and
the
piece
that
I've
shown
you
today
is
one
sliver.
It's
the
the
quality
gate,
evaluation
that
we
use
between
stages,
and
it
is
the
test
execution
we
believe
for
DevOps.
There
was
the
question:
how
can
it
help
in
devups?
A
We
believe
it's
the
it's
it
it's
the
the
answer.
It
allows
you
to
write
the
it
allows
you
to
deliver
software
in
a
way
that
your
developers
actually
writing.
Software.
You
developers
write
event-driven
micro
service,
the
writing,
componentized
services,
that
work
is
traded
through
events,
yet
your
pipelines
are
large
micro,
lytic,
monolithic
pieces
of
code
and
I.
Think
captain
tries
to
help
you
to
you
know,
use
your
bits
and
pieces
that
you
have
maybe
ever
champions
pipeline
for
deploying
for
testing
and
for
doing
something
else,
but
then
it
orchestrates
the
whole
thing
even
driven.
C
Thank
you,
hey
is,
is
evaluation
of
a
pull
request
in
this
kind
of
environment
or
a
potential
change
any
different
in
the
captain
environment
than
then
doing
work
off
of
the
master
ranch.
Are
there
things
that
are
unique
to
thinking
about
many
pull,
requests
arriving
that
cause
you
to
configure
captain
differently
or
to
have
it
think
differently
about
the
problem.
A
So
pull
requests
right.
If
you
want
to
integrate
captain
with
your
with
your
get
ups
with
your
pull
requests,
then
you
would
again
make
a
call
to
cap
and
then
trigger
captain
as
part
of
the
city,
at
the
of
a
pull
request.
Validation,
like
the
quality
gate.
Do
you
need
to
think
about
it
differently?
Based
on
branch?
Is
a
master?
A
I
would
say
you
could
model
it
in
cap
based
on
stages,
so
we
have
the
concept
of
a
cap
project
and
then
different
stages,
and
then,
instead
of
having
def
stage
production,
you
could
say
I
want
to
integrate
it
with
my
deaf
environment,
but
I
allow
every
developer
to
have
their
individual
feature
branch
and
then
a
full
request
to
come
in
and
I
want
to
separate.
The
data
I
may
have
kept
internally
organize
each
branch
in
in
a
stage.
So
much
of
this
make
sense
problem.
A
Let
me
show
you
something
here:
I
have
a
project
here:
it's
the
simple
node
project
and
right
I
have
staging
in
prod,
so
you
can
have
in
enter
and
deliver
environment.
You
have
multiple
stages
and
if
you
want
to
have
multiple
branches
you
go,
probably
one
option
would
be
to
have
you
take
the
stage
concept
in
captain
and
apply
it
to
different
branches,
and
then
you
have
it
clearly
separated
you
see
all
the
data
there,
all
the
quality
gates
that
happen.
A
This
branch,
which
is
this
branch,
the
other
option,
would
be
to
use
the
tags
and
the
labels
remember
what
we
had
in
my
was
here
right.
If
I
go
over
I
every
time
when
I
called
captain
I
gave
it
labels,
so
you
can
clearly
differentiate
the
data
with
hey.
This
was
triggered
by
this
build.
This
was
triggered,
but
this
pull
request
on
this
particular
range,
so
you
can
also
separate
it
through
the
labels
through
metadata
here.
A
D
B
A
The
area
so
familiar
with
the
concepts,
but
not
using
them
right
now,
mm-hmm
yeah,
that's
consistent
in
what
I've
seen
in
the
past
there's
only
a
small
group
of
people
that
are
really
from
using
it
deliberations,
work
right
now,
partially
automate
I
like
to
fully
automate
it.
It's
a
33
percent.
It's
great!
A
Hardly
any
performance
tests
or
no
tests,
even
that's
interesting,
and
what
I'm
really
happy
about
is.
Yes,
we
have
existing
kubernetes
cluster
to
deploy
it
to
and
there's
only
one,
no
answer,
because
we
have
received
feedback
from
some
people.
That
said,
while
they
have
kubernetes
clusters,
some
people
might
not
be
able
to
deploy
something
into
these
clusters,
but
I
assume
the
audience
here
that
are
maintaining
and
managing
pipelines
and
obviously
Jenkins
on
in
the
combination
with
kubernetes.
B
D
B
A
And
if
anybody
wants
to
follow
up
right
again,
here's
my
material
I'm
also
very
happy
to
to
go
into
a
direct
conversation.
If
you
want
to
know
how
this
works.
Also,
we
have
a
select
channel
I
forgot
to
mention
this
selected
captain
daughter's
age
I
would
be
great
if
people
could
sign
up
slick.
Captain
Doris.
B
They
were
shaving
up
communication
and
also,
if
you
develop
pipe
magnetic
heated
pipelines
for
captain
and
Jenkins,
and
if
you
decide
to
develop
a
plugin
just
let
the
community
know,
because
we
have
a
lot
of
use
cases
to
share
all
circa
nice
regular
meetings.
For
example,
we
have
special
meaning
for
cloud
native
seed,
which
is
focused
also
on
various
integrations
with
components
environment.