►
From YouTube: Scaling performance test execution & analysis
Description
Roman Ferstl (Triscon-IT) joins this Keptn user group to share their lessons learned and best practices on how to scale performance testing.
music by Roman Heuser - Spirit's Creek
Some questions that Roman answered as part of Q&A:
- You mention memory leak testing - based on your experience - do you need large-scale tests that run for hours - or can we also detect memory leaks as part of delivery like you show today with Keptn?
- You talk about reusing functional tests - can you elaborate? Does it mean you just drive that functional testing tool with many “virtual users”? Or are you converting these scripts to the load testing tool script language?
- What are your best practices for defining the thresholds for your performance metrics you showed in your slide? Do you define them as a performance team? Or is this something that comes from the development teams?
- On your load testing dashboard you have a lot of metrics - do you have best practices on which metrics to include when running these tests? Do you have any template dashboards for different type of applications?
Meeting notes: https://docs.google.com/document/d/1Om9pj16hGKP_w2vUaH-7Cp0ffEIj-Oe3IezeVCpFYAM/edit#
Learn more: https://keptn.sh
Get started with tutorials: https://tutorials.keptn.sh
Join us in Slack: https://slack.keptn.sh
Star us on Github: https://github.com/keptn/keptn
Follow us on Twitter: https://twitter.com/keptnProject
A
A
So
it's
kind
of
carnival
the
last
day
of
carnival
where
we
typically
go
out
and
party
like
crazy.
Today,
the
party
is
going
to
be
a
little
smaller
and
a
little
more
virtual,
but
still
we
have
friends
over
roman
christian,
you
again
and
I'm
sure
some
attendees
over
there
as
well
fabian
and
some
others,
so
lai
lai
or
I
don't
know
what
else
we
should
say.
A
And
I'm
also
very
glad
I
mean
roman
we've
been
doing
a
couple
of
presentations
over
the
last
couple
of
weeks.
You've
always
done
an
amazing
job,
but
we
always
had
to
say,
and
now
we
have
these
exciting
slides
that
have
been
heavily
influenced
by
christian.
Who
is
now
also
with
us
here
and
that's
great
to
have
you.
A
Beautiful
really
phenomenal
cool.
I
want
to
just
kick
it
off
quickly,
so
captain
user
group.
This
is
really
about
the
captain
community,
talking
about
how
they're,
using
captain
to
solve
certain
use
cases
or
how
they're
using
it
to
make
their
lives
better
or
to
make
the
lives
better
of
others
that
they
help
with
captain
and
with
their
other
services
that
they
do.
So
it's
a
way
to
show
captain
to
the
community
and
and
explore
different
use
cases.
A
The
meeting
doc
so
feel
free
to
put
in
your
names-
and
I
will
also
especially
later
on
when
we
get
to
q
and
a
do
some
moderation
here,
but
without
further
ado
roman,
you
have
been
using
captain
you
and
your
organization
twist
on
you've
been
using
captain
over
the
last
couple
of
months
and
it's
just
phenomenal
what
you've
been
doing,
especially
around
performance
engineering,
and
so
I
really
want
to
just
now
sit
back
and
let
you
talk
a
little
bit.
I
may
interject
from
time
to
time.
B
Feel
free
to
interrupt
me
anytime.
Yes,
as
you
already
mentioned,
captain
made
also
our
life
a
lot
easier
and
today
we're
gonna
focus
entirely
about
topics
around
performance
testing.
So
we
are.
I'm
gonna
introduce
this
in
a
couple
of
slides
in
more
detail,
but
we
are
all
about
performance,
engineering
and
performance,
testing
and
apm,
and
today
I'm
going
to
talk
about
how
to
scale
performance
tests
by
automating.
B
First
of
all,
let
me
quickly
introduce
us
so
to
say:
triscon,
I'm
roman
from
triscon
check
us
out
on
linkedin
make
sure
to
reach
us
out.
If
you
have
any
questions
about
performance
testing,
we
are
entirely
dedicated
to
this
topic.
We
do
this
for
a
lot
of
customers
and
we
also
data
trace
partners.
We
implement
apm
solutions
and
we
offer
a
lot
of
workshops
and
trainings
if
you
want
to
get
a
hand
on
practice
on
captain
hands-on
practice.
B
In
a
two
to
three
days
workshop,
we
can
build
some
awesome,
automated
quality
gates,
for
instance,
if
you're
listening
to
this
and
you
you
may
not
be
a
performance
tester.
I
want
to
just
briefly
discuss
in
one
slide.
I
know
it's
not
a
lot,
but
I
want
to
mention
it
why
it
is
so
important
to
perform
performance
tests,
so
there's
basically
three
pillars
that
you
cover
with
performance
tests.
One
is,
of
course,
the
stability
topics,
so
you
want
to
avoid
crashes.
B
B
So
the
stability
issue
is
really
important
to
uncover
something
nasty
like
memory,
leaks
and
stuff
like
that
and
also
performance
tests,
as
it
says
in
the
word,
it's
about
performance.
It's
about
speed.
You
wanna
sure
you
meet
your
slas
that
it's
fast
enough.
You
wanna
provide
the
best
experience
for
your
users,
but
this
is
actually
only
one
aspect
of
performance
tests
and
the
third
one
is
the
resource,
the
resource
consumption,
which
is
actually
heavy
topic.
It's
becoming
more
and
more
important.
If
you
move
to
the
cloud
also
performance
tests
are
really
important.
B
It
makes
a
lot
of
sense
if
you
move
to
the
clouds
to
run
performance
tests
before
and
after
you
do
it
to
make
sure
that
you
have
the
best
configuration,
save
some
money
and
that
it
actually
works,
and
also
this
is
really
brand
new
stuff.
You
also
want
to
check
out
more
legacy.
Things
like
mainframe
calls.
We
have
this
a
lot
and
we
see
that
customers
do
a
lot
of
outsourcing.
Our
clients
of
ours
do
a
lot
of
outsourcing
and
the
people
who
are
sitting
in
the
development
teams.
B
So
these
are
the
things
that
you
may
want
to
to
think
about
when
you
hear
about
performance
tests.
We
do
this
for
a
lot
of
customers
and
I
brought
you
a
story.
If
you
want
to
check
out
what
customer
it
is
and
learn
more
about
the
customer
himself,
I'm
talking
about
you
can
watch
our
breakout
session
that
I
had
with
andy
together
at
the
perform.
We
talked
a
lot
about
that
a
little
bit
more
in
detail
there,
but
in
principle
I
brought
the
story
here.
B
We
have
a
big
insurance
company
and
we
managed
in
a
couple
of
years
to
increase
the
test
throughput
and
the
apps
that
we
support
with
performance
tests
by
a
factor
of
10
to
15
in
terms
of
tests,
and
we
dare
to
taking
the
step
to
have
a
self
service
so
really
to
get
to
the
point
where
we
reach
performance
test
as
a
service
and
to
enable
other
teams,
but
before
we
we
go
this
way,
what
are
actually
the
challenges
that
every
performance
engineer
is
facing?
I
brought
some
of
these
as
well.
B
So
usually,
if
you
start
with
the
topic,
you
don't
start
with
a
huge
team.
So
if
you're
a
performance
engineer,
it
might
be
the
case
that
you
see
yourself
as
a
one-man,
show
fighting
against
a
a
lot
of
inquiries
and
trying
to
to
deal
with
the
workload.
So
I
have
been
there
it.
It
makes
totally
sense
to
put
more
awareness
on
this
topic
to
get
more
resources.
But
what
can
you
do
if
you
are?
B
The
one
main
show
what
helped
us
a
lot
to
cover
the
needs
and
to
cover
the
work
is
to
to
go
full
into
the
topic
of
automation,
and
I'm
going
to
talk
about
that.
How
you
can
help
yourself
to
deal
with
all
this
amount
of
tests
you
you
may
want
to
do
and
script.
Maintenance
is
also
a
big
challenge.
So
a
lot
of
time
in
performance
testing
is
actually
spent
not
for
getting
the
valuable
results
are
out
there
and
performing
an
analysis
to
gain
more
insights.
A
lot
of
time
is
still
spent
in
script
maintenance.
B
We
don't
gain
anything
about
it
except
time
for
actually
being
able
to
run
your
test,
and
you
want
to
make
this
as
small
as
possible
this
amount
of
time
and
that's
why
I'm
gonna
also
talk
about
the
reusability
of
tests
and
another
part
I
mentioned
it
already
is
the
restart
analysis.
So
you
want
to
be
as
efficient
as
possible
in
the
resort
analysis
and
we
are
leveraging
dynatrace
for
the
most
awesome
root
cause
analysis
out
there
to
have
immediate
feedback.
B
Once
we
run
our
performance
test
to
have
the
information
that
the
performance
test
has
been
run
in
visible
there
in
your
apm
tool,
so
that
others
also
know
oh
a
performance
test
was
run.
This
is
why
I
see
increased
load
on
some
applications
and
stuff
like
that
and
how
this
is
actually
gonna.
Look
in
practice.
I'm
gonna
show
you
in
a
couple
of
minutes.
B
So
what
was
this
story
about
the
customer
I
already
mentioned
when
we
arrived
there
in
2016,
they
were
already
doing
performance
tests,
which
is
awesome
if
you're
doing
performance
tests.
It's
awesome,
you
keep
doing
it,
it's
really
important
and
they
were
doing
it
with
more.
Like
a
legacy
approach,
an
older
tool,
visual
studio
web
test
framework-
in
the
meantime,
it's
not
supported
in
like
from
microsoft
anymore,
and
we
suggested
to
switch
to
a
more
elaborate
tool.
B
Maybe
so
the
first
thing
you
want
to
think
about
is:
how
can
you
integrate
other
test
automation
tools
that
may
exist
in
your
company
already
into
your
load
test
tool
and
save
a
lot
of
time
for
generating
and
maintaining
your
scripts?
There
we
used,
for
instance,
the
neoload
selenium
integration
and
build
our
own
selenium
framework,
and
the
third
part
I
mentioned-
and
we
also
did
this
at
this
customer
side-
was
to
become
more
efficient
by
using
dynatrace
for
root
cause
analysis.
It
helps
you
to
pinpoint
the
bottleneck
as
fast
as
possible.
B
You
don't
need
to
run
tests
over
and
over
again
and
changing
some
things
in
the
system
like
load
balancing
to
become
to
narrow
down
the
problem,
because
you
have
full
stack
monitoring
all
the
time
across
the
entire
landscape
of
your
rit
and
by
combining
all
these
measures.
We
ended
up
three
years
later,
with
10
times
more
tests
or
15
times
more
tests
and
10
times
more
applications
that
we
were
supporting
and
we
didn't
use
10
times
more
people
and
that's
the
effective
point
here.
B
So
it's
actually
something
that
you
can
do
to
get
some
free
resources
to
take
the
next
step,
and
this
is
the
step
that
I
want
to
talk
about
now.
It's
the
step
of
automating.
The
entire
process
also
the
result
analysis
and
we
did
a
proof
of
concept
there
in
2020
by
using
captain
in
a
combination
with
new
load
and
dyno
trace,
and
we
build
an
automated
quality
gate
and
before
I
gonna
talk
about
that,
I
want
to
talk
about.
Why
has
this
not
happened
before
so
far?
B
There
is
like
a
gap
from
here
to
there
because
we
see
automated
unit
tests,
of
course,
also
end-to-end
tests,
mostly
on
interface
level
apis,
and
we
see
this
also
on,
of
course,
browser
automation,
so
functional
tests,
but
actual
load
tests
or
performance
tests
are
almost
never
automated.
Why
is
this
the
case?
If
we
look
at
the
nature
of
a
performance
test,
it's
actually
quite
clear,
because
the
decision
is
a
lot
harder
for
a
unit
test
or
an
end
to
end
test.
It's
simple
right.
You
have
a
browser
clicking
through
your
application.
B
If
something
unexpected
happens,
the
field
is
not
there
or
something
it
simply
fails.
You
can
put
in
simple
validations
there,
so
it's
a
true
false
decision
actually
same
goes
for
unit
tests
in
the
performance
tests.
As
we
mentioned
there
is
these
three
pillars
of
three
aspects
of
performance
testing:
it's
the
performance
itself,
so
you
care
about
response
time,
the
response
times.
You
also
care
about
error
rates
and
stability,
of
course,
and
you
in
addition,
you
care
about
the
infrastructure
you're
using,
so
you
care
about
the
resource
consumption,
for
instance.
B
This
could
be
how
many
parts
you
use.
Cpu
usage
mainframe
calls
database
calls
whatever
it's
hundreds
of
metrics,
that
you
want
to
look
at
to
be
sure
that
everything
is
smooth
when
you
hit
production,
and
this
is
the
the
reason
why
the
decision
is
not
so
simple
anymore
and
the
the
entire
approach
is
to
combine
all
these
metrics
back
into
a
final
discord
and
make
this
decision
simple
again,
and
this
is
where
captain
comes
into
place
here.
B
B
I've
seen
andy
that
you
changed
it
recently
now
it's
called
the
cloud
native
application
life
cycle
orchestration
and
I'm
gonna
talk
about
the
definition
that
we
see
that
what
we
see
for
captain,
but
I
think
there
is
a
lot
of
use
cases
out
there.
Of
course,
there's
other
remediation
use
cases
etc.
But
as
I
mentioned
today,
we
care
about
the
automated
quality
gates,
since
it's
the
one
that
we
use
most
often
in
our
point
of
view,
captain
can
do
a
lot
of
things
and
it
couldn't
can
be
used
for
a
lot
of
things.
B
We
see
it
that
in
principle,
you
can
tell
captain
that
something
happened.
You
can
send
events
to
captain
and
it
can
be
anything
and
also
captain
can
trigger
events
and
it
can
trigger
anything.
And
since
you
can
build
your
own
sli
provider,
you
could
probably
even
do
smart
home
things
with
it,
but
to
pick
out
one
use
case
in
terms
of
the
continuous
delivery
we
try
to
to
build
a
sketch
here
to
see
where
this
is
going,
and
this
is
the
one
of
the
most
awesome
use
cases.
I
think
so.
B
So
we
are
basically
in
the
world
of
ci
cd
pipelines
or
it's
event
driven
and
this
container
or
this
service
will
go
to
production
and
in
the
best
case
you
have
your
production,
monitored
with
dynatrace,
and
so
you
have
an
ai,
like
davis
sitting
on
his
app
and
telling
you
oh
with
this
new
version.
Something
is
wrong,
something
is
worse
and
you
don't
reach
your
goals
and
it
has
negative
impact
on
your
user
experience.
B
A
And
I
just
had
to
stay
quiet
because,
first
of
all,
I
love
hearing
you
talk
because
you
obviously
have
an
creative,
great
experience,
and
this
all
comes
from
your
customer
engagements,
also
how
you,
over
the
years
kind
of
figured
out
how
to
automate
performance.
If
you
go
back
to
the
previous
slide-
and
I
know
something
where
we
need
to
also
give
christian
some
credit
for
the
animation,
but
christian
here
I
have
a
little
extra
homework
for
you.
I
really
like
when
the
ship
goes
from
left
to
right
kind
of
delivering
it.
A
One
of
the
things
captain
does
as
part
of
delivery
is
doing
automated
performance
testing
and
some
chaos
engineering.
So
you
could
even
visualize
this
as
if,
as
the
ship
is
kind
of
traversing
the
ocean,
you
could
have
like
a
wind
blowing
in
and
then
some
some
high
waves.
So
basically
really
the
captain
is
really
trying
to
figure
out
and
battle
test
the
container
that
is
on
the
way
from
dev
to
production.
If
we
can
withstand
any
type
of
chaotic
situation,
it
just
came
to
mind
because
I
know
visual.
A
B
A
B
A
B
Think
of
that
yeah
I
actually
have
a
suggestion.
It's
awesome
that
you
mention
it
and
if
we
we
go
now
to
the
next
slide,
so
keep
this
picture
in
mind
so
magically.
We
will
shrink
this
picture
and
put
it
to
the
upper
right
side
of
the
next
slide.
Now
we
already
prepared
this,
because
we
see
this
use
case
now,
the
automated
quality
gates
it
could
be
between
both
sides
right.
So
it's
what
you
say
you.
B
B
So
we
have
application
version,
one
or
service
version,
one
some
version
in
production,
and
we
wanna
update
this.
So
we
need
to
develop
version
2
in
this
case
and
of
course,
once
we
develop
it,
we
want
to
also
do
quality
assurance
and
run
some
tests,
and
maybe
there
is
unit
tests
already
around
on
component
level,
also
maybe
performance
tests,
and
you
want
to
proceed
to
the
next
stage.
B
You
could
fully
integrate
this
in
a
cicd
pipeline
without
human
interaction
needed
anymore.
I
would
not
suggest
to
go
this
like
instant
immediately
into
a
full
integration
approach.
Have
your
automation,
your
script
evaluation,
your
final
score
around
a
couple
of
times
to
to
make
sure
it
makes
sense
and
then
go
and
integrate
it
in
real,
build
pipelines,
but
this
actually
works,
and
hopefully
you
are
able
to
fix
your
performance
issues
in
the
development
stage.
Proceed
automatically
through
the
automated
quality
gates
to
the
testing
stage
and
finally
hit
the
production.
B
So
we
mentioned
this
a
couple
of
times.
I
think
most
of
the
users
here
heard
about
sli
slos
slas
already,
so
I
don't
want
to
spend
more
than
a
minute
with
this.
Just
as
a
quick
reminder,
this
is
the
language
of
captain.
So
to
say,
this
is
something
that
google
came
up
with
and
we
need
to
define
these
metrics
that
I
talked
about.
It
has
been
there
like,
always
if
you're
a
performance
engineer.
This
is
the
performance
counters
that
you
have
been
gathering.
It
can
be
response
time.
It
can
be
cpu.
B
It
can
be
anything
you
want
like
you
need
to
define
your
metric
as
an
sli,
and
you
need
to
define
how
you
want
to
measure
it
in
terms
of
what
statistical
method
and
you
want
to
define
some
goals
and,
for
instance,
this
would
be
the
slo.
It
could
mean
that
a
response
time
must
be
less
than
a
certain
value
in
a
certain
period
of
time,
for
instance,
here
less
than
400,
milliseconds
or
30
days,
and
if
you
take
these,
this
is
so
to
say
the
groundwork
for
building
something
greater.
B
Of
course
we
care
about
how
to
find
to
define
our
metrics
and
our
goals,
so
this
is
already
something
that
goes
a
little
bit
into
captain
and
how
it
works,
but
it's
still
a
few
from
afar,
but
the
nice
thing
is
that
you
actually
don't
need
to
understand.
What's
what's
what's
below
the
surface
right,
you
can
just
use
it,
but
the
way
it
works
is
you
can
trigger
the
captain
api
from
anywhere.
It
could
be
jenkins,
build
pipeline.
B
It
could
be
azure
devops,
it
could
be
simple
command
line
whatever
you
need
to
trigger
an
event
and
tell
captain
that
in
test
is
running,
and
there
is
a
lighthouse
service
which
is
quite
central
in
captain
and
managing
something
like
the
sli
providers
or
handling
the
communication
when
event
comes
in
and
so
the
sli
providers
are
can
be.
For
instance,
dynatrace
or
or
neo
load,
it's
actually
in
our
use
case,
where
you
take
your
metrics
from
so
I'm
gonna
talk
about
that
in
a
minute
more
how
it's
the
the
way.
B
If
you
wanna
use
your
metrics
from
dynatrace,
it's
can
be
also
prometheus.
It
can
be
actually
anything
since
you
can
also
write
your
own
soi
provider,
but
for
these
tools
it's
already
there.
What
you
end
up
with
is
actually
the
heat
map
in
the
captain's
bridge.
So
you
can
of
course,
analyze
the
goal
if
it's
met
for
each
individual
metric,
which
is
super
important,
so
don't
get
me
wrong.
You
don't
end
up
with
just
the
score
and
you
don't
know
what
went
wrong,
but
the
score
is
low.
You
have
the
details
still
available.
B
Of
course,
this
is
the
first
site
where
you
can.
If
something
is
red,
have
a
look
at
the
bridge
see
what
what
goals
are
not
met
on
what
metrics?
But
we're
gonna
see
this
in
practice
in
a
couple
of
minutes,
so
in
principle
it
was
built
on
yaml
basis.
So
if
you're
a
load
test
engineer,
you
probably
have
in
your
apm
tool
or
anywhere
else
build
some
mechanism
that,
after
the
test,
you
have
your
graphs,
you
have
your
metrics,
you
see
how
the
test
was
was
run
you
zero
response
times.
B
You
see
your
cpu
times
everything
you
care
about,
and
the
idea
is
to
tell
now
captain
these
are
the
metrics
that
I
want
to
evaluate,
and
captain
speaks
here
in
the
language
of
yaml.
So
you
have
to
define
your
slis,
of
course,
and
your
slos
so
to
say
again
what
you
want
to
measure
and
what
is
the
goal,
and
this
is
actually
in.
In
our
view,
it
was
quite
maintenance
heavy,
as
by
running,
I
would
say,
minor
load
tests.
Even
we
ended
up
with
yamos
that
consists
of
500
lines
of
code.
B
B
So
if
you
want
to
go
this
way-
and
you
have
somewhere,
your
dashboards,
for
instance
in
dynatrace,
would
be
perfect
case,
because
all
you
need
is
actually
provide
additional
informations
into
the
name
of
your
charts
for
captain
and
all
these
yaml
stuff
that
is
needed
to
actually
perform.
The
evaluation
on
the
captain's
side
is
generated
automatically
so
because
the
metrics
that
you
care
about
are
the
ones
that
you
were
looking
always
in
in
any
case
here,
for
instance,
failure
rates
and
stuff
like
that.
B
But
I
am
also
I
don't
want
to
talk
about
this
too
much,
because
I
have
already
also
prepared
a
demo
for
you
to
keep
it
a
little
bit
more
practice
oriented
and
before
we
get
to
the
demo
part.
I
want
to
talk
more
a
lot
about
the
awesome
integrations
of
load,
test
tools
like
new
load
and
dynatrace,
because
it
helps
you
a
lot
actually
and
it's
quite
simple.
B
If
you're
near
load
user
make
sure-
and
your
dyno
trace
is
also
in
place,
make
sure
to
hit
this
checkbox
and
simply
tell
where's
your
tenant
where's,
the
energy
is
hosted,
and
what
will
happen
automatically
is
that
new
load
requests
will
be
tagged,
so
they
will
be
easier
to
find
in
dynatrace,
and
also
here
you
will
see
them
as
request
attributes
automatically.
They
also
become
renamed,
so
you
won't
be
able
to
mix
up
your
load
test
requests
with
actual
requests
coming
from
elsewhere.
B
So
this
is
an
awesome
feature
to
simply
look
at
what
you
have
generated
and
filter
out
all
the
rest.
Since,
to
be
honest,
you
won't
have
production
equivalent
test
systems
just
for
your
own
for
running
load
tests.
I've
never
seen
this
and
also
nice,
is
that
all
the
services
and
applications
that
are
so
to
say
under
stress
during
your
load
test.
They
get
a
information
here
that
that
load
test
has
been
performed
automatically
out
of
the
box.
B
If
you
hit
the
checkbox
and
also
you
get
a
link
back
to
the
results,
so
if
someone
else
is
looking
at
the
service
wondering
why
the
load
has
so
much
increased
and
he's
analyzing
something
completely
different,
he
will
see
immediately.
Oh,
there
was
a
load
test,
all
right-
I
I
don't
have
to
care
about
this
anymore.
It's
super
important
to
keep
things
simple,
and
another
part
is-
and
this
is
quite
new-
it
came
also
a
blast
here.
B
You
can
define
a
special
tag,
let's
call
it
a
magic
tag
in
a
new
load
and
you'd
simply
tag
the
entry
application
in
dynatrace
and
it
means
by
entry
application.
I
mean
the
the
service
in
in
the
front
that
you
are
going
to
put
under
stress
and
all
the
services
in
the
back
end.
That
also
are
important
for
your
load
test.
For
instance,
database
calls
back-end
services.
B
B
In
this
case,
it's
called
triscontestingturf,
it's
our
demo
application.
You
will
have
a
peak
at
this
just
in
a
minute
in
the
demo
part,
and
it
consists,
as
you
can
see
here.
Maybe
let's
keep
that
in
mind
already
very
simple
web
front
end.
I'm
gonna
show
it
to
you
and
it's
an
api.
It's
separated
in
the
back
end,
so
to
say,
and
the
api
at
the
back
end
calls
some
database.
A
B
A
A
I
don't
know
defining
how
a
testing
tool
can
integrate
with
the
monitoring
tool
without
maybe
necessarily
them,
knowing
each
other.
You've
brought
up
the
excel
in
a
slow
discussion,
and
I
know
that
jurgen
is
also
on
the
call
who
recently
started
a
long
discussion
on
twitter
regarding
how
can
we
standardize,
slis
and
slos
so
that
more
people
more
tools
can
benefit
from
the
same
definition
and
yeah?
It's
it's.
A
B
B
I
see
a
huge
connection
between
sre
and
the
entire
performance
engineering
discussion,
since,
as
you
mentioned
it,
you,
you
want
to
care
more
about
defining
goals
on
a
on
a
smaller
scale,
on
a
service
scale,
so
but
yeah,
maybe
not
to
keep
the
audience
waited
too
much.
It
was
a
little
bit
of
teaser.
Now,
let's,
let's
go
to
the
demo
part
and
also
discuss
this
further
at
the
end.
B
So
before
showing
it
to
you,
I'm
going
to
explain
what's
going
to
happen
when
I
hit
a
button
to
trigger
a
pipeline
because
there's
actually
a
lot
of
things
going
on,
but
since
we
automated
everything,
all
you
will
see
is
a
final
score
right,
so
I
have
to
explain
all
the
awesome
things
that
are
happening
there
and
in
this
case
andy
knows
this
slide
already,
and
I
once
mentioned
it.
A
pipeline
is
awesome.
It's
like
the
symphony
and
many
things
are
playing
together
perfectly
in
the
middle
of
it.
B
B
B
B
B
So
here
you
see
already
our
dashboard.
I'm
gonna
explain
this
further
and,
as
you
see
here,
we
have
a
lot
of
links
in
our
dashboard.
The
first
thing
is
gonna
me
gonna,
be
me
calling
the
pipeline
triggering
it
here
from
azure
devops,
I'm
gonna
say
run
pipeline
and
as
soon
as
it
spins
up,
I'm
gonna
switch
back
and
talk
more
about
the
dashboard
we
built.
B
So
this
is
already
a
so
to
say,
final
pipeline.
We
have
our
near
load
project
and
the
api
isn't
all
configured
we
simply
say
run.
We
could
trigger
this
from
anywhere.
It
could
be
build
driven
or
whatever.
Now
I
will
kick
off
the
load
test
manually
from
this
pipeline,
so
we
kind
of
we
can
have
a
look
here.
So
all
these
things
that
I
just
mentioned
are
going
to
be
visible
here
in
the
console
as
well.
Now
neoload
will
will
spin
up
and
nearly
web
will
be
informed
to
run
the
test,
etc.
B
Let's
give
it
a
little
bit
time
switch
back
to
the
dashboard.
So,
what's
the
important
thing
here,
we
have
a
lot
of
things
here
visible,
but
we
have
these
global
goals
telling
us
when
the
test
should
pass
or
fail.
Building
on
this,
depending
on
the
score,
you
can
build
your
test
in
a
comparison
form
where
you
say
I
want
to
compare
it
with
the
last
run.
I
want
to
compare
it
with
the
last
10
runs
or
whatever
you
need,
and
you
you
have
no
restrictions
there.
B
That's
the
important
part
you
want
to
take
home
with,
and
here
we
have
front
end
response
times,
which
is
awesome,
so
you
can
baseline
and
you
can
do
this
in
your
load
test
time
anyway
right.
So
this
is
something
that
the
load
test
tool
offers,
but
what
the
load
test
tool
doesn't
offer
is
backend
response
times
or
number
of
database
calls.
So
you
can
actually
take
into
account
many
more
metrics
that
you
can
have
together
in
a
different
way
elsewhere
and
combine
it
into
this
final
score
and
the
entire
automation.
B
So
we
have
an
application
here
to
make
it
a
little
bit
more
easy
to
for
you
to
follow
it's
a
simple
application.
It's
called
the
testing
turf,
it's
a
demo
application
actually
that
we
built
for
recruiting
purposes
and
for
these
purposes
and
yeah
there
will
be
a
recruiting
challenge
available
if
you're
able
to
break
the
site,
you'll
get
an
interview
with
us
sure
to
check
it
out
in
our
web
page
in
a
couple
of
weeks.
If
you
have
interest
in
working
with
us,
so
what
this
application
is
good
for.
It's
just
for
demo
purpose.
B
B
B
B
New
load
web
is
already
running
the
test.
I
was
able
to
see
it
here,
so
our
pipeline
mechanism
was
already
working
and
we
see
that
the
load
test
is
already
running
and
we
designed
it
to
only
run
for
a
couple
of
minutes
here,
which
is
enough
to
do
some
proper
baselining
on
a
small
scale,
and
we
see
we
are
facing
already
some
errors
here
which
do
not
occur
under
load
like
everything
seems
to
be
awesome.
I'm
still
here
right.
It
can
click
everything
it's
working,
so
why
is
the
load
test
running
on
arrows
right?
B
B
B
You
have
a
link
to
the
captain's
bridge,
so
we
thought
this
could
be
some
dash
ops
approach
and
it
is
in
this
way
you
want
to
have
meta
information
available
for
anyone
who
looks
at
the
dashboard
to
also
make
sense
of
it,
but
it's
it's
also
important
that
this
is
the
part,
that's
gonna.
What
I
want
to
mention
this
is
the
part.
B
B
So,
let's
head
back
to
our
pipeline
here,
you
see
the
captain.
Event
is
already
so
it's
already
the
final
event
so
to
say
so.
We
are
waiting
for
captain
to
evaluate
the
test
and
we
see
this
was
red.
We're
gonna
head
back
to
our
bridge
to
see
what
went
wrong
here
in
new
load
web.
Actually,
I
already
see
should
already
see
that
the
test
is
over.
I
see
this
here
and
if
I
don't
have
a
look
at
this,
I
don't
have
to
look
at
this
right.
B
We
see
here
now
two
minutes
ago,
we've
finished
our
test
and
captain
has
evaluated
it
and
we
see
that
our
score
is
not
good
enough.
Actually,
the
score
would
be
good
enough,
but
some
key
metrics
that
always
need
to
fulfill
some
certain
goal,
like
the
error
rates
that
we
really
really
care
about.
If
something
is
goes
wrong,
we
want
this
test
to
fail,
even
if
the
final
score
is
still
good.
B
So
here
we
see
that
we
had
some
failure
rates
increased
on
the
api
level
and
if
I
now
want
to
know
what
went
wrong,
of
course,
I
can
simply
go
back
to
neoload
web.
We
I
can
have
a
look
in
diner
trace.
It
would
be
really
really
simple
to
find
out
in
this
case.
What
went
wrong
here,
for
instance,
I'm
going
to
show
it
in
your
load
web.
We
have
it
on
the
events,
so
the
requests
that
are
failing
and
we
see,
although
everything
was
still
performing-
I
was
clicking
through
this
right.
You
saw
it.
B
It
was
still
awesome,
but
we
have
here
a
concurrency
problem
so
as
soon
as
two
people
are
hitting
this
register
button.
In
the
same
second,
it
goes
boom,
and
this
is
not
something
that
we
just
exotically
came
up
with
it's
something
that
we
see
very,
very,
very
often
also
different
customer
stages,
and
we
implemented
this
in
a
way.
Of
course,
this
is
on
purpose
in
it.
It's
a
demo
application,
but
we
implemented
this
in
a
way
of
using
a
static
dictionary
and
you're
gonna
see
this
here.
It
looks
like
this.
A
B
A
A
A
But
to
you,
but
to
your
point
with
the
the
concurrency
problem
right,
we
see
this
all
the
time
as
as
impacting
applications
and
the
thing
is,
you
don't
even
need
a
whole
lot
of
load
to
test
your
service
or
your
application
against
these
concurrency
issues,
and
this
is
also
a
question
that
that
often
comes
up.
Do
I
really
can
I
actually
find
performance
and
scalability
issues
in
applications
as
part
of
my
delivery
pipeline
with
very
fast
tests
like
you
shown,
or
is
this
all
just
marketing
blah
blah
and
you?
A
B
And
it's
awesome
that
you
mention
it
and,
if
you're
more
interested
in
this
kind
of
tests,
we're
going
to
have
an
entire
session
about
concurrency
testing
at
the
near
lord
pacquiao
event,
I'm
gonna
put
another
reference
at
this.
At
the
end
of
the
presentation,
I
think
it's
in
the
23rd
of
march
make
sure
to
check
this
out.
You're
absolutely
right.
This
is
something
that
you
can
only
discover
in
performance
tests
and
you
don't
need
ten
thousands
of
users.
B
B
B
B
So
this
is
a
highly
scalable
approach
and
the
best
thing
that
comes
with
it
is
that
you
enable
others
to
run
your
test
like
if
you
think,
really
hard
about
all
the
metrics
that
that
need
to
be
considered
in
the
load
test.
Others
don't
have
to
be.
Do
it
anymore.
They
can
just
look
at
the
score
and
come
back
to
you
if
it's
red
and
if
they
don't
understand
it,
and
you
will
explain
them
how
this
is
the
because
this
metric
and
it
comes
from
there.
B
But
if
it's
green,
no
one
needs
to
care
about
it
anymore.
That
is
the
idea
about
behind
about
that
and,
of
course,
it
helps
you
to
build
this
early
to
catch
more
problems
before
they
hit
production.
That's
the
entire
idea
right
you!
We
don't
want
to
have
these
nasty
concurrency
things,
for
instance,
in
production
and
what
we
showed
here.
I
want
to
want
to
also
stress
this.
It
was
for
an
end-to-end
application
for
a
front
end.
You
may
want
to
start
with
a
micro
service.
B
It's
the
best
thing
here
to
start
with
this
small
load
test
on
microservice
scale,
and
then
you
can
think
about
also
evaluating
bigger
load
tests,
where
you
have
many
test
cases
on
the
front
end
with
it,
but
it's
more
complex,
then.
So,
if
you
start,
this
approach
start
with
a
microservice,
and
this
is
the
last
slide,
I
think
already
where
we
want
to
give
you
a
little
guideline.
B
If
you're
not
ready
yet
for
automated
quality
gates,
it
doesn't
make
sense
to
pull
this
off
so
when
we
say
think
about
where
you're
standing
is
think
about
how
much
time
you
spend
for
script
maintenance,
if
this
is
still
the
most
time
of
the
most
amount
of
time
in
your
process
of
performing
performance
tests,
you
want
to
change
that.
So
you
can
change
that
by
thinking
about
other
tools
that
exist
out
there,
which
may
be
better.
But
another
approach
is
to
reuse
your
functional
tests.
B
So
this
is
a
really
important
thing
to
lower
your
maintenance
time.
Another
thing
is
that
you
want
to
utilize
your
apm
tools,
such
as
dynatrace
for
your
test
analysis,
if
you
don't
have
leveraged
all
this
nice
integration
that
we
showed
before
just
as
a
quick
reminder.
By
doing
so,
we
oh
these
were
the
only
measures
yeah,
we
didn't
do
anything
else
and
we
ended
up
there
with
already
10
times
more
apps,
that
we
were
supporting
and
15
times
more
tests.
B
So
if
you
don't
have
the
time
for
automating
the
performance
test,
because
there's
so
much
to
do
think
about
automating
the
the
steps
that
you
do
each
day
to
improve
from
there,
it
will
help
you
a
lot
and
if
you've
reached
this
point
right
so
well,
I
actually
am
quite
happy
with
my
maintenance
time.
It's
all
good.
I
used
a
lot
of
automation.
There
then
you're
ready
for
automating
the
final
step
of
the
result.
Analysis
and
you
want
to
start
with
one's
project,
one
microservice.
B
You
want
to
automate
the
test
analysis
and
you
wanna
talk
to
your
developers.
You
don't
want
to
tell
them
now.
I
built
the
performance
test
I
integrated
in
the
cicd
pipeline
and
it
breaks
their
build
each
time
they
they
try
to
deploy
something
or
or
it
breaks
their
pipeline.
You
don't
want
to
do
this.
You
want
to
first
make
sure
that
your
score,
your
metrics,
make
sense
that
your
test
design
is
stable
in
anything
that
they
say
that
this
is
awesome.
A
B
B
A
A
What
hit
me
again
was
your
second
to
last
slide
with
like
your
initiatives
for
2021
this
one
here,
because
this
is
also
so
true
for
captain,
because
we
have
a
lot
of
people
that
would
like
to
use
captain
quality
gates,
but
then
very
often
they
say
well,
the
thing
is
we
don't
yet
have
either
tests
at
all
or
we
have
tests
that
are
constantly
fading.
Therefore,
my
quality
gates
are
not
even
you
know,
stable
enough,
because
I
don't
have
the
pre-work
done
so
I
you
know.
B
Exactly
and
yeah,
if
I
would
summarize
it
it's
first
thing-
is
to
look
around
yourself.
What
you
are
doing,
try
to
see
if
you
can
improve
from
there
look
around
what
others
are
doing,
try
to
improve
from
that,
and
then
you,
if
you're
good
to
go,
then
then
you,
this
is
the
most
important
part
and
thanks
for
making
the
intervention
in
it
again,
you
want
to
talk
to
the
people
and
you
make
you
wanna.
B
You
want
them
to
have
this
solution
right.
You
want
to
have
a
communication
bridge
to
the
people
building
the
apps.
If
you
are
the
performance
tester,
not
building
the
apps
yourself,
you
want
them
to
care
about
your
performance
test,
and
this
is
the
perfect
way
to
build
a
communication
bridge
to
them
since,
as
soon
as
it's
easy
for
them,
they
will
accept
this
as
a
valid
solution.
B
Yes,
that
is
actually
a
really
good
question,
so
I,
the
the
honest
answer,
is
that
I
wouldn't
go
for
discovering
a
memory
leaks
in
an
automated
quality
approach
right
away.
First
of
all,
you
want
to
start
with
the
small
baseline
tests,
where
you
do
always
exact
the
same
things
in
the
short
time
to,
and
you
don't
want
to
have
your
pipelines
being
run
forever
right.
B
So
this
is
why
it's
awesome
to
start
with
a
micro
service
where
you
can
say:
okay,
I
call
it
a
thousand
of
times
in
in
five
minutes,
and
then
you
end
up
always
with
the
same
results.
Is
this
enough
to
discover
a
memory
leak?
No,
usually
not.
What
we
do
is
when
I
started
with
load
testing.
The
legacy
approach
is
that
we
have
an
application.
We
need
to
like
run
12
hours
of
load
tests
to
to
see
the
potential
memory
leaks
there.
B
And
yes,
if
it's
a
really
slow,
decaying
memory
leak,
you
and
you
wanna
your
application
to
eventually
actually
crashing
the
test.
This
is
the
way
to
go,
but
it's
also
important
that
you
know
how
does
a
memory
leak
look
like
in
your
matrix
and
if
you
are
aware
that
it's
mostly
heap
sizes,
you
could
actually
just
say-
and
you
could
build
this
into
captain
and
say
well,
I'm
gonna
run
it
for
10
minutes
and
if
I'm
gonna
build
up,
I
don't
know
100
megabytes
of
heap
and
after
the
test
it's
not
cleared
up.
B
So
you
wait
a
little
bit
after
your
test,
then
you
could
say
well,
it
shouldn't
behave
like
this.
If
you
have
some
caching
mechanisms
implemented-
and
this
is
like
a
feature
of
course-
you
don't
want
to
break
your
build
from
this
because
it
works
as
design
there
right.
You
would
have
to
run
longer
tests,
but
this
is
the
way.
B
This
is
the
important
thing
if
you
see
in
your
first
load
tests
that
the
heap
is
like
the
clear
horizon
on
an
ocean
surface
and
it
doesn't
change
yeah
like
perfect,
horizontal
yeah,
you
want
to
you,
want
to
put
the
metric
in
there.
You
and
you
want
to
put
a
goal
on
this,
that
it
doesn't
change
from
one
release
to
the
other,
and
if
this
is
the
case,
you
could
already
estimate
a
memory
leak
in
a
smaller
scale
test
which
doesn't
run
for
hours,
cool.
A
That's
an
awesome
question
yeah.
No,
that's
it!
You
had
a
the
load
testing
dashboard
that
you
had.
Is
this
something
that
you
share
with
others?
Are
there
some
best
practices?
Do
you
do
we
have
like
a
set
of
metrics
and
dashboards
that
you
always
use
for,
let's
say
a
particular
type
of
application,
because
the
question
comes,
what
can
we
reuse
and
what
can
we
also
give
people
as
a
best
practices
as
how
do
we
start
like
which
metric
do
we
choose,
which
metrics
do
we
choose
for
a
particular
application?
B
So
the
answer
is
yes,
there
is
a
certain
things
that
you
always
want
to
care
about,
and
it's
response
times.
Usually
it's
it's
error
rates,
so
failure
rates
requests
that
fail,
and
you
want
to
put
goals
on
this
right.
So
hopefully,
the
the
error
rates
should
be
zero
right,
and
this
is
something
that
you
always
want
to
consider.
Also
maybe
database
calls,
but
this
is
already
where
it
becomes
so
is
my
application
doing
database
calls?
B
You
don't
want
to
hit
hundred
percent
and
response
times
and
error
rates,
but
you
always
in
each
case
want
to
talk
to
the
people
who
did
the
architecture
or
understand
the
entire
thing
to
make
sure
that
the
best
metrics
are
in
your
evaluation.
This
is
something
that
it
will
always
suggest.
The
metrics.
B
B
And,
for
instance,
what
we
do
is
at
the
customer's
side,
they're
doing
a
lot
of
mainframe
stuff
I
mentioned
so
you
maybe
you
want
to
put
the
number
of
calls
that
you're
doing
to
the
mainframe
or
that's
going
through
the
mainframe.
Through
your
background,
we
want
to
put
this
in
a
metric
and
you
want
to
say
well
from
one
release
to
another.
We
don't
want
to
do
10
times
more
since
you're,
going
to
pay
probably
10
times
more
right,
yeah.