►
From YouTube: Keptn Quality Gates with Argo and Jenkins for Distributed Load Testing – Lessons Learned from Intuit
Description
In this Keptn User Group we have Sumit Nagal who leads the Reliability Engineering Team at Intuit. In his latest blog he gave insights into why and how his team provides Distributed Load Testing using Argo in Kubernetes (DISTRO) to his organization.
We will learn how they integrated Keptn Quality Gates to automate performance test analysis for every tested microservices. He will discuss the challenges and their solutions for dynamic environments, dynamic SLIs and SLOs and dynamic work loads.
Join this User Group as the goal is to not only hear from Sumit but also engage in an active discussion about how Keptn might also help you with your challenges.
Meeting notes: https://docs.google.com/document/d/1Om9pj16hGKP_w2vUaH-7Cp0ffEIj-Oe3IezeVCpFYAM/edit#heading=h.yxcezq7qmm3m
Learn more: https://keptn.sh
Get started with tutorials: https://tutorials.keptn.sh
Join us in Slack: https://slack.keptn.sh
Star us on Github: https://github.com/keptn/keptn
Follow us on Twitter: https://twitter.com/keptnProject
A
Here
we
go
hi
everyone
and
welcome
to
this
captain
user
group.
It's
first
time
in
this
format,
and
we
are
very
excited
to
have
a
great
agenda
for
today,
since
we
are
trying
out
this
new
format.
I
want
to
let
you
know
that
these
meetings
here
also
recorded.
We
will
put
those
meetings
also
on
our
captain.
Youtube
channel
captain
is
a
cncf
project,
so
we're
also
following
the
cncf
code
of
contact.
A
That
basically
means
please
be
nice
to
each
other
in
the
chat
and
in
the
q,
a
section
and
in
our
captain
community
in
general.
For
today
we
have
a
packed
agenda.
I
want
to
say,
and
the
first
agenda
item-
it's
not
on
the
agenda
directly,
but
I
would
like
to
ask
you:
please
also
put
your
name
on
the
smd
list,
so
we
can
keep
track
a
little
bit
of
who
is
joining
our
user
groups
and
who
is
part
of
the
captain
community.
A
So
you
all
know
that
we
have
the
captain
slack
and
our
twitter
channel,
but
we
also
want
to
establish
these
meetings
and
we
want
to
keep
track
a
little
bit
of
who
is
joining
and
who
actively
wants
to
also
to
contribute
to
this
to
these
meetings.
I
will
then
again
post
the
link
to
this
document
in
the
chat.
So
you
can
also
add
your
name.
A
So
thank
you
so
much
for
joining
and
congratulations
on
the
community
rockstar
award.
We
are
right
now,
in
the
background,
preparing
the
pr
that
you
also
officially
named
community
rock
stars
of
captain,
but
I
want
to
briefly
hand
over
maybe
to
roma,
to
you
and
your
team.
Maybe
you
can
talk
briefly
about
your
success
with
captain
and
how
you're
using
captain
and
what
what
you
did
to
conjugate
to
captain
to
to
receive
this
award.
B
B
C
A
D
A
A
This
is
great,
so
thanks
so
much
again
for
everyone
that
just
joined
our
captain
music
group,
I
will
just
post
the
link
to
this
document
in
a
second.
Please
add
your
name
also
on
the
snd
list
and
with
this
I'm
already
handing
over
to
andy
and
sumit,
and
I'm
really
excited
to
hear
their
story
on
captain
with
argo
with
jenkins
for
distributed
little
testing.
C
Exactly
well,
I
think,
first
of
all
concrete
from
my
side
as
well:
it's
not
a
rock
star.
It's
a
rock
band
until
we
kind
of
celebrate
today
and
yeah,
it's
perfect
it's!
It
was
a
pleasure
so
far
working
with
you
guys
on
this
and
I'm
sure
there
will
be
many
more
success
stories
coming
out.
So
I'm
I'm
hoping
to
see
you
soon
back
on
on
the
rockstar
board
or
user
groups
webinars
whatever
it
is,
we'll
do
our.
B
C
Captain
is
one
part
of
it,
but
you
will
see
today
what
he
and
his
team
have
built
is
amazing
in
terms
of
automating
performance
testing
in
a
large
distributed
way,
which
means
I
would
actually
like
to
hand
over
to
sumit,
and
maybe
you
can
share
your
screen
and
because
I
know
you
have
some
slides
prepared
and
a
demo
as
well.
So
let's
see
if
the
slide
sharing
works,
perfect.
D
D
C
For
those
that
are
online
watching,
if
you
have
any
questions,
I
will
play
the
moderator
and
also
sumit.
I
may
inject
with
a
question
or
two
because
obviously
I
know
your
history,
because
we've
been
working
together
over
a
couple
of
years,
more
recently
on
captain,
but
if
questions
come
in
then
you
know
also
try
to
if
it
makes
sense
push
it
over
to
your
life.
Otherwise,
we'll
have
dedicated
time
at
the
end.
D
D
This
is
a,
I
think,
andy
share
that
like
how
we
are
using
today.
So
we
are
right
now
using
one
single
cluster,
two
captain
project,
50
plus
services
and
most
of
the
services,
what
we
have
actually
onboarded.
They
are
automated
way.
We
have
not
manually,
go
and
automated
through
jenkins,
declarative
pipeline.
We
onboarded
that
and
we
created
a
dynamic
slo.
I
will
talk
a
little
bit
more
about
that,
and
then
we
have
various
kind
of
loads
so
how
we
are
using
jenkins
to
switch
dynamically
based
on
the
load.
D
I
will
share
a
few
of
our
code
base
to
go
more
on
that
now.
We
do
have
a
lot
of
automation,
but
this
actually
help
in
do
without
any
manual
integration,
end-to-end,
automation-
and
this
is
self-service.
So
we
assume
anyone
who
is
able
to
add
the
code
for
the
relevant
project
and
service
for
the
sli
slo.
Everything
will
go
handy,
they
don't
need
to
come
and
do
that.
So
it's
a
user
base
get
commit.
D
They
make
the
change,
put
everything
in
the
git
and
then
the
jenkins
declarative
pipeline
will
pick
it
up
and
it
we
use
our
go
or
workflow
which
definitely
help
in
the
distributed
load.
Testing
cost.
So
a
little
bit
about
my
team,
I
was
leading
the
team
of
three
three
domain.
One
is
the
chaos
engineering,
the
second
one
is
the
performance
and
then
the
third
one
is
the
infrastructure.
Where
this
whole
thing
is
coming,
we
are
doing
distributed,
load
testing
on
chaos
as
well
as
performance.
D
I
have
recently
presented
on
cubecon,
where
I
talk
in
brief
about
that
chaos,
workflows
and
how
we
are
doing
in
performance
testing
and
distributed
load.
We
are
using
the
same
concept.
Actually
we
introduced
that
in
performance.
First
and
later
we
have
actually
implemented
on
the
chaos.
This
is
about
my
platform.
We
have
thousands
of
developers
more
than
200
clusters,
many
name
space
and
more
than
200
services.
Already
onboarded
on
that.
C
Okay,
so
me
just
a
quick
question:
if
you
can
quickly
go
back
to
this,
I
have
a
from
a
chaos,
engineering
perspective
and
also
jurgen
is
working
closely
with
the
litmus
team,
so
we
do
have
elite
most
integration.
I
know
we
talked
about
this
and
by
the
way,
people
that
don't
know
you
have
just
yesterday.
C
I
think
literally
just
came
back
from
india
you're
now
back
on
in
the
us
and
and
you
it
seems,
you're
juggling
a
lot
of
different
projects
like
chaos,
engineering
performance
engineering
and
an
amazing
platform
that
your
developers
use
it
into
it.
It's
it's
amazing
what
you
do
and
it's
it's
great,
that
you
contribute
back
to
open
source
and
even
find
the
time
to
talk
with
us
about
this.
This
is
really
cool.
D
Thanks
a
lot
andy,
in
fact,
the
captain
and
the
litmus
chaos
integration.
I
was
the
when
I
was
presenting
in
the
litmus
chaos
I
actually
shared
that
we
are
integrated
with
the
captain
and
that's
how
this
whole
communication
started
and
karthik
and
they
work
with
the
captain
community
leaders
and
did
that.
So
this
is
the
place
where
we
come
in
handy
and
it.
This
is
an
overall
platform
which
is
helping
all
our
services
to
onboard
and
provide
the
highly
scalable
stable
solution.
D
So
let
me
just
take
a
step
back
and
let's
talk
about
what
is
distributed,
load
infrastructure
and
I
think
there
are
a
few
folks.
They
are
already
there
and
they
might
be
dealing
something
or
similar
that
so
we
typically
used
to
have
a
common
jenkins
infrastructure,
which
is
a
shared
infrastructure
through
which
we
usually
create
the
loader
test.
Loader
we
call,
and
then
you
have
certain
commercial
tool,
also
came
like
the
front
line
or
the
some
something
like
that,
which
also
help
in
doing
that.
D
But,
logically
speaking,
we
don't
have
any
dedicated
infrastructure,
which
is
just
running
the
path,
so
we
usually
be
sharing
a
junkies
infrastructure
which
is
doing
our
build
pipeline,
functional
test
and
and
then
it
become
a
very,
very
messy,
because
if
you
run
the
path
test,
it
will
take
longer
time.
Security
is
a
challenge
because
any
time
you
go
and
interact
with
that,
you'll
probably
need
to
take
care
of
that.
D
How
you
are
interacting
with
that
specific
end,
point
cost
is
pretty
high
and
as
well
as
time
few
of
the
puff
tests
they
take
longer
time
and
then
they
hold
lot
many
resources
and
there
was
no
natural
way
of
kubernetes
integration
and
the
this
challenge.
How
we
have
solved
is
that
so
we
came
up
with
the
problem
statement
that
we
need
a
some
solution
which
is
scalable,
which
is
a
container
native
and
has
to
have
a
self-service
build
on
the
open
source
tool.
D
So
this
was
the
main
problem
statement
when
we
started
on
this
and
how
we
achieve
at
this
presentation
is
all
about
that.
So
little
bit
inside
about
this
distributed
load
infrastructure.
What
we
are
able
to
do
that
is
stands
for
input,
kubernetes
services
platform
and,
as
we
were
doing
validation
of
this
platform,
we
started
building
this
infrastructure.
So
that
means
that
we
are
eating
our
own
dog
food
up
front
which
later
we
have
shared
with
our
customer
and
that
there
are
many
teams
has
actually
started
using.
D
We
recently,
I
think,
month
or
two
months
back.
We
are
open
source
also,
so
it's
readily
available
if
anyone
wanted
to
consume
on
kubernetes.
The
second
and
important
part
is
that
lot
of
load
testing
tools.
They
are
not
the
container
native
approach,
we
have
a
test
container,
but
we
really
don't
have
anything
which
is
a
container
native
most
of
the
things.
What
you
see
that
you
get
the
code
from
git
or
you
get
the
code
from
some
of
the
other
versioning
system.
D
You
use
your
one
of
the
load
distribution,
either
jenkins
or
front
line,
or
whichever
or
new
load,
and
then
you
go
and
trigger
that.
So
this
is
the
the
insight
we
have.
So
we
really
don't
have
that
and
then
most
of
the
time
you
have
infra
as
a
code.
So
right
now,
if
you
look
for
the
self
service,
you
go
and
modify
get
and
then
you
get
a
trigger,
but
there
is
no
way
you
can
make
those
changes
dynamically
that
I
already
put
a
code
now,
just
based
on
the
traffic
pattern.
D
I
increased
from
10
tps
to
10
000,
tps,
5,
notes
to
50
notes,
something
like
that
is
not
there.
You
usually
have
to
go
and
code,
it's
not
very,
very
self
intuitive
and
then
the
cost,
if
you
use
any
commercial
tool
as
well
as
any
of
the
jenkins
infrastructure,
also
it
require
lot
of
cost.
So
we
use
ergo
workflow,
which
is
the
backbone
of
this
and
that
take
care
of
the
all
the
resourcing
and
job
related
stuff.
D
This
is
based
on
the
gatling
right
now
it's
supporting
gatling,
but
it
supports
geometer.
Also.
It
is
based
on
the
ergo
workflow
right
now
we
are
already
integrated
with
litmus
chaos.
Captain
is
the
one
which
we
are
today
talking
about
that
and
the
distro
is
the
project
which
already
being
keeping
everything
in
in
tandem,
and
this
has
a
lot
of
reporting
capability.
We
use
operation
data
lake,
we
call
it
odl
and
then
it
has
a
splunk
integration.
D
Last
but
not
least,
we
have
a
lot
of
collaboration
and
monitoring
tool,
but
what
we
have
done
is
that
we
created
a
pave
road
so
that
any
new
team
comes
they
onboard
to
this
platform
and
they
get
everything
out
of
the
box.
So
we
have
two
use
cases.
One
use
case
is
that
any
team
who
has
a
service
or
engineer
they
wanted
to
run
the
test,
and
the
second
one
is
that
for
us,
the
platform
or
sre
team
or
reliability
engineer
we
wanted
to
validate
the
platform
so
to
do
that,
based
on
the
specific
need.
D
We
made
lot
of
changes
and
we
got
lot
of
requirement
to
make
that
so.
First
and
foremost
is
that
I
should
be
able
to
test
against
any
application.
It
should
be
self-service
and
then
I
should
be
able
to
scale
based
on
my
need.
I
really
don't
need
to
come
and
say
that
provision
this
instance
or
I
need
to
go
and
add
more
virtual
users.
Something
like
that.
I
want
everything
to
be
dynamic.
As
long
as
I
have
kubernetes
cluster,
I
have
a
name
space.
I
have
resources
available.
D
D
So
at
a
very
high
level.
This
infrastructure
is
like
everything
is
in
code,
so
application
service
code
is
there
and
that
will
create
a
service
image
like
that.
We
create
a
test
code
also
as
part
of
the
get,
and
we
create
a
test
image,
and
all
this
work
is
happening
by
this
infra
code,
which
is
tying
with
the
service
image,
as
well
as
the
test
image.
We
get
all
these
images
like
as
docker
container
images
and
set
it
up
on
one
of
the
name
space
and
from
those
namespace.
D
These
test
stuffs
will
make
a
call
to
the
service
endpoint
and
they
push
the
data
through
one
they
they
we
do
have
monitoring
and
we
push
the
data
to
the
data
lake
and
we
use
amazon
s3
because
we
are
heavily
rely
on
aws
now,
just
the
same
infrastructure.
If
you
go
one
level
down
like
how
actually
it's
been
realized,
it's
realized
is
that,
like
you,
have
application
or
service
code
which
is
creating
a
container
image
which
can
be
triggered
by
a
jenkins
spy
and
your
ci
integration
function
and
performance
test.
D
Your
test
code
is
also
being
part
of
the
your
ci
process.
So
when
you
build
your
service
code,
you
build
a
test
container
code
also
and
that
test
container
will
also
be
published.
Now
this
infra
code,
which
we
have
this,
will
pick
this
container,
which
is
the
test
container
using
the
argo
yamal
base
configuration
and
it
will
execute
on
one
of
the
kubernetes
namespace
now
to
execute
anything
on
the
kubernetes
namespace.
You
need
two
thing.
I
think
I
have
shared
in
the
the
distro
project
that
how
how
you
can
do
that.
D
So
you
need
argo
workflow
so
that
you
can
view
that
so
or
go
ui
to
view
that
and
a
workflow
which
actually
trigger
that
stuff
and
based
on
this
container,
it
will
spawn
off
these
workflow
parts,
which
is
just
using
this
container
image
and
through
ergo
workflow
account.
It
will
target
any
of
the
name
space
in
a
same
cluster
or
outside
cluster.
So
right
now
we
operate
both
the
option.
D
You
can
have
the
clust
target,
the
same
cluster
in
a
same
name,
space
in
a
cluster
or
you
can
go
outside
and
then
all
this
data
we
because
when
you're
running
the
distributed
load,
testing
each
pod
will
generate
a
test
and
it
will
create
a
report.
So
it's
really
important
that
you
need
to
aggregate
the
report
and
to
do
the
aggregation.
D
We
have
a
certain
step
mentioned
and
when
we,
when
I
mention
this
funnel,
that
means
that
we
are
aggregating,
consolidating
and
reporting.
So
I
will
show
that
so
anything
if
you
want
to
take
out
from
this
presentation,
is
this
slide
that,
however,
everything
is
being
happening.
So
everything
is
in
code.
So
we
have
a
gatling
script
which
is
being
checked
in
the
gate,
and
you
have
infra
code,
which
is
a
yaml
code,
which
is
nothing
but
doing
what
a
container
test
container.
D
This
has
been
triggered
by
jenkins
jenkins
will
interact
with
the
kubernetes
cluster
by
cube
context,
and
this
has
a
very
limited
permission
just
to
submitting
a
workflow
once
you
submit
a
workflow
or
you
submit
through
cronjob.
If
you
don't
want
to
use
the
jenkins
scheduled
one,
then
what
will
happen
is
that
the
workflow
pod
will
initiate
this
workflow
part
once
it
will
initiate.
It
will
do
three
action.
It
will
just
go
and
create
a
pod
description
budget
so
that
workflow,
where
we
are
actually
executing
it,
will
remain
in
the
system.
D
It
execute
the
test
and
it
aggregate
the
test,
and
then
it
pushed
the
data
to
the
s3
bucket.
So
these
are
the
four
steps
we
usually
do
as
part
of
the
workflow.
Now
as
part
of
the
service
account
when
we
are
actually
executing
it
requires
certain
role
and
role
binding.
Those
three
things
are
actually
given
here,
and
this
is
the
one
which
have
the
information
that
where
and
how
we
will
go
and
hit
the
end
point.
So
before
I
jump
in.
D
D
So
go
here
input,
and
here
we
put
all
the
parameter
like
how
much
load
you
want
how
long
you
want.
What's
the
study
time,
what's
the
ramp
up
time,
how
you
want
to
configure
the
captain
which
kind
of
load
you
want
every
anything
and
everything
we
go
and
we
kept,
we
put
it
there.
So
here
these
are
the
thing
and
then
I
just
enable
the
captain
and
then
I
can
go
and
proceed
so
it
will
take
five
to
ten
minutes
time
before.
C
C
D
D
C
D
So
now
this
is
a
one
of
our
gatling
test
which
is
actually
running.
This
gatling
test
is
a
small
crud
operation
for
a
given
service
where
we
go
and
create
a
specific
resource,
and
then
we
go
and
manipulate
that
so
read
is
more
about
that.
Getting
that
and
create
is
more
about
creating
that
resource.
So
this
is
a
small
hello
world
service,
which
is
a
kind
of
we
are
giving
for
all
our
onboarding
team
as
a
paid
role.
D
D
And
by
the
time
we
will
finish,
we'll
probably
have
some
result
coming.
So
behind
this
thing,
what
will
happen?
You
will
get
this
kind
of
workflow
where
you
can
define
your
load
generator
dynamically.
So
here
in
the
earlier
case,
I
just
define
one
workload
distributor,
but
you
can
go
and
define
that.
I
need
one
two,
three,
four,
five,
the
same
time.
You
can
also
define
that
how
you
want
your
throughput,
you
want
one
tps
to
tps
three
tps
or
even
ten
or
ten
thousand
tps.
D
C
D
Yes,
so
behind
the
scene,
what
we
do
is
that
you
we
will
ask
that
you
just
tell
the
10
000
tps
behind.
We
will
take
care
that
you
need
a
three
parts
or
five
parts
or
ten
parts.
We
will
distribute
that
we
know
the
capacity
of
each
port,
how
much
it
can
handle.
We
know
for
a
sure
and
then
based
on
that
and
again
those
all
are
configurable.
D
So
in
your
yaml
you
go
and
configure
that
how
you
want
the
distribution
to
go
and
it
will
it
will
give
the
aggregated
result
so
10
000
tps
it
will
distribute
on
six
one
tps,
one
one
pod
is
there
and
that's
what
you
have
seen
here
like
when
I
just
executed.
I
just
executed
only
one
part
so
now
here
by
this
time
you
have
everything
has
been
actually
executed.
You
can
look
into
the
output
of
the
artifact.
You
could
see
that
run
those
who
are
familiar
with
the
gatling.
D
D
So
it
will
come
something
like
that:
it
will
go
and
run
the
test
initialize
the
captain.
It
will
make
the
captain
quality,
get
call
post
the
event
and
then
it
fetch
that
event
and
make
the
call
whether
it's
based
on
sli
and
slo
how
things
are
going
so
now.
This
is
a
good
segue
to
start
like
why,
when
everything
was
working
till
that
everything
was
working,
we
were
able
to
do
the
load.
Testing
download
the
artifact
archive
then
then
comes
why,
after
that,
what
was
missing.
D
So
what
we
found
is
that
this
is
fine,
but
this
is
not
end-to-end
automation.
We
are
able
to
get
the
gatling
report.
We
can
still
make
some
manipulation
and
say
that
yeah,
the
response
time
is
this
or
status
is
this
we
can
pass
or
fail,
but
our
system
is
huge
right.
We
have
everything
dynamically,
so
it's
end-to-end,
automation,
kind
of
stuff
is
missing,
so
someone
need
to
validate.
They
still
need
to
go
and
look
into
the
gatling
report
and
yeah.
D
This
is
the
thing
and
then
you
go
to
the
server
side
and
say
that
yeah,
this
response
time
is
a
fine.
So
still
some
manual
work
is
required
and
then
the
other
part
is
that
the
scale
is
very
dynamic.
So
sometimes
you
need
a
10
tps
sometime.
You
need
10
000
tps,
so
it
is
really
important
that
how
you
can
manipulate
the
scale
and
then
with
the
manual
stuff
it
become
very
challenging.
There
is
no
native
way
we
were
running
on
the
jenkins
and
we
were
looking
into
that.
D
D
There
comes
the
captain,
so
we
found
that
captain
has
a
sli
slo
base
information
and
we
already
have
very
good
clarity
that
what
is
our
sli
for
a
one
one
hour
test
or
a
four
hour
test
or
a
two
days
test.
So
we
and
again
one
other
thing
which,
which
is
very
interesting
for
us-
that
we
want
the
the
load
pattern
to
be
very
dynamic.
So
sometimes
we
wanted
to
run
on
kubernetes
cluster.
On
one
specific
service
sometime,
we
went
on
a
10
different
services.
D
Sometimes
we
went
on
50
different
services
and
there
was
a
few
scenarios,
which
is
a
rare,
which
you
do
for
releases
that
we
need
in
one
cluster
200
namespaces
same
time
execution.
So
those
are
some
very,
very
interesting
challenge,
which
is
making
everything
a
dynamic.
Now,
on
top
of
that,
the
other
challenge
is
that
the
distribution
timing
is
also
changed,
like
small,
medium
and
low
heavy
and
extra
large
kind
of
execution.
D
So
we
opted
for
that,
and
here
you
could
see
that
we
onboarded
all
our
services
on
a
given
cluster
dynamically
using
a
declarative
pipeline.
So
we
won't
go
one
by
one
and
onboard
that
now
the
result
is
that
you
have
the
run
coming
and
then,
whichever
run
we
are
doing,
we
can
have
initialize
the
captain,
get
the
quality
and
get
the
result
later
point
of
time
we
converted,
on
top
of
what
the
captain
team
has
created
the
declarative
pilot
we
created
our
own
pipeline
and
show
the
result
now
why
we
pick
captain.
D
I
think
that's
also
a
important
aspect,
so
the
automation.
So
now,
when
you
go
and
execute
you
don't
have
any
interaction.
It's
a
complete
hundred
percent
automation
from
starting
to
end.
You
just
need
to
specify
what
kind
of
pattern
you
need
so
here,
if
you
could
say
that
load
type
has
been
defined
as
a
small,
so
this
is
a
small
load
type.
I
should
be
dynamically
pick
my
sli
slo
based
on
the
small
load,
and
I
should
be
able
to
find
that
what
what
is
the
end-to-end
pass
of
this
quality
data?
D
The
everything
is
a
code.
So
if
you
want
to
modify,
we
just
go
and
modify
our
slas
based
on
sli
slo
on
the
code,
and
then
everything
will
remain.
There
is
no
manipulation
require
later
point
of
time.
We
can
handle
the
small,
medium
high
load
vertical
and
horizontal.
I
talk
about
that.
One
service,
10
services,
50
services,
200
services,
everything
is
in
a
declarative
pipeline.
I
think
that's
a
very
important
thing.
D
Captain
has
given
declarative
pipeline
and
we
are
working
only
on
the
declarative
pipeline
and
I
will
show
the
code
how
this
whole
thing
is
working
and
then
everything
is
a
pure
road,
so
any
team
who
on
board
what
they
need,
is
jenkins
pipeline
and
in
the
jenkins
pipeline
we
have
those
two
libraries
shared
and
with
those
two
library
everything
is
available
behind
the
scene.
We
already
defined
the
sli
slo.
So
what
they
need
is
that
in
case
they
need
to
change
their
sli
slo,
then
to
go
and
change
the
sli
slo
rest.
D
Everything
will
remain
same
and
how
we
have
done.
I
think
that's
a
teaser.
Is
that,
like
right
now
we
are
using
one
cluster
from
one
cluster,
which
is
we
have
hosted
the
captain.
Everyone
go
and
interact
with
the
context,
this
cluster.
We
have
use
ingress
and
route
and
end
point,
so
we
expose
the
bridge
as
well
as
an
api.
We
use
our
intuit
artifactory
in
place
of
taking
from
open
source
or
the
docker.
D
We
download
that
in
our
artifact
and
use
that
image
security
and
compliance
of
so
that
we
ensure
that
these
images
are
compliant.
Everything
is
being
integrated
in
junkies
and
I
will
talk
in
a
while,
and
the
self
service
of
sli
slo,
for
mainly
for
the
load,
distribution
and
dynamic,
is
there
so
with
that.
D
Let
me
just
quickly
show
one
part,
which
is
that
how
this
whole
thing
is
happening,
so
this
is
one
jenkins
pipeline
we
have,
and
here,
if
you
could
see
that
we
have
defined
the
captain
library
and
we
have
our
own
library.
So
as
part
of
this
captain
library,
we
do
a
lot
of
work
which
is
more
towards
that.
How
we
are
running
the
this
is
the
input
part
which
you'll
see
in
that
when.
C
D
D
So
like
that,
we
have
for
chaos
also
the
same
thing
so
captain
these
are
that
main
function.
We
have
written,
and
I
will
tell
you
why
we
written
our
own
functions.
So
here
you
see
that
behind
the
scene
we
will
go
and
we'll
make
the
actual
call
to
the
three
core
function
of
the
library.
So
why
we
did
this.
We
did
this
because
we
want
everything
to
be
self
service.
So
what?
What
do
you
mean
by
self
service?
So
we
have
defined
all
the
configuration
on
one
config
file.
D
So
if
I
have
10
environment
or
I
have
10
different
services,
what
I
need
to
go,
I
will
go
and
define
this
parameter,
which
is
telling
that
what
I
wanted
to
do
so
I
will
go
and
say
that
I
need
number
of
parts
number
of
tps
ramp
up
time
and
there
I
say
that
all
this
parameter,
what
I
need
for
the
captain,
also
being
provided
that
which
project,
which
performance,
environment
or
stage
I
want
and
load
type
equal
to
small.
Now
my
jenkins
library
read
this
configuration
and
then
it
will
be.
D
D
There
are
five
different
configuration
and
that's
what
I
mentioned
that
your
as
the
your
slas
are
varying
right.
So
when
you
are
doing
10,
000
tps
your
response
time,
validation
is
different
for
tp99
or
tp95
and
the
same
way
when
you
have
a
small
scale.
So
here,
if
you
see
that
this
is
a
small
scale
where
I
have
defined,
that
criteria
is
a
little
bit
relaxed.
D
But
again
it's
sorry
it's
more
stringent,
but
when
it's
the
higher
the
criteria
is
the
throughput
is
pretty
high.
So
that
way,
just
with
the
throughput
you
could
see
that
for
small
load.
I
know
that
I
have
5000.
Transaction
is
good
enough
for
heavier
load.
I
need
a
5
million
transaction
so
that
that
way
I
can
just
vary,
and
I
don't
need
to
go
and
change
these
things
manually.
I
just
keep
those
things
and
through
parameter.
I
can
pass
those
values.
D
D
Is
so
you
could
see
that
this
is
the
complete
heat
map
and
where
we
defined
that
95
percentile,
throughput
error
rate,
the
percentile
50th
and
all
the
other
response
times
so
now,
initially
we
had
some
hiccups,
we
do
have
some
outlier,
but
now
everything
is
kind
of
stable.
Again,
this
is
for
demo,
so
things
has
to
be
stable,
but
on
the
real
scenarios,
these
values
change
and
the
moment
we
see
that
we
fail
that
and
the
quality
gate
will
say
that
this
is
a
failure.
D
So
to
show
that,
like
how
other
other
way
we
can
integrate,
we
have
so
if
you
see
that
bridge
captain
this
is
the
place
where
we
have
actually
hosted
everything
and
we
have
swagger
api
also
available.
So
if
someone
wanted
to
try
it
out
that
they
can
go
and
execute
that
this
is
the
latest
execution
and
you
will
get
a
specific
context
and
on
that
specific
con.
D
And
then
here
you
will
get
the
full
information
now,
based
on
this
information,
which
is
coming
from
your
s
line
slo,
you
know
what
to
if
there
is
a
problem,
what
to
work
on,
otherwise
everything
will
go
smooth,
so
this
is
the
way
that
that's
a
demo.
I
I
talk
about
that
now
in
intuit,
we
have
all
the
data
going
to
the
wavefront,
so
we
do
have
the
app
dynamics.
D
We
do
have
dynamics,
but
as
we
want
everything
to
be
in
a
single
source,
there
is
another
work
we
we
started,
though
we
are
still
not
finished
that
so
we
initially
did
with
the
wavefront
integration.
We
wanted
to
make
a
another
sli
of
wavefront
and
define
all
those
sli
and
slos
and
fetch
that
information
litmus
chaos
is
another
thing
which
we
started
and
I'm
glad
that
that
is
all
I
think
it's
been
finished
and
we
still
need
to
take
the
latest
one.
D
D
This
is
the
project
where
how
we
are,
or
anyone
wanted
to
use
this
distributed,
load
testing
on
a
kubernetes,
how
they
can
do
that.
This
has
been
shared.
I
think
a
couple
of
months
back-
and
this
is
a
blog
I
have
written-
which
talks
about
that
if
someone
wanted
to
start
from
scratch
and
wanted
to
do
the
distributed
load
testing
in
that,
in
that,
I
briefly
talk
about
that
how
all
this
integration
works,
how
the
captain
works,
as
well
as
how
ergo
workflow
and
the
jenkins
pipeline
will
go.
So,
mr.
D
Yeah,
so
this
is
a
good
starting
point
and
the
biggest
problem.
What
we
saw
is
mainly
for
the
cost.
I
think
cost
is,
if,
if
you
think
about
any
of
the
performance,
tooling,
the
hosting
cost
and
the
licensing
cost
is
pretty
high
and
based
on
your
uses,
you
can
definitely
optimize
that,
and
then
this
is
a
more
kubernetes
native
way
of
implementation,
and
captain
is
also
a
cncf
project
and
being
hosted
in
a
community.
So
it
become
a
smooth
integration
without
thinking
too
much
about
that.
D
So
again,
all
those
things
again.
It
failed
because
I
have
a
small
bug
where
I'm
not
able
to
fetch
again
and
again,
because
I'm
on
a
pretty
old
version-
and
we
were
me
and
andy-
was
trying
to
fixing
that,
but
we
were
not
able
to
fix
that.
But
here
you
could
see
that
all
this
information
about
how
the
captain
is
invoking
in
the
post
event
and
then
getting
that
event
in
a
looping
manner
and
then
providing
that
information.
D
D
You
know
we
do
have
some
information
about
the
95
percentile
and
that,
but
again,
as
I
mentioned,
that
if
you
look
for
dynamic,
if
you
look
for
everything
on
a
different
different
load,
this
50th
thing
is
being
hard
coded
right.
So
it
cannot
go
and
then
it
will
only
look
for
the
certain
parameter
which
gatling
supports
right
like
it
cannot
go
the
cpu
percentile.
D
D
So
we
pull,
we
pull
the
data
from
two
places,
one
from
dynadress
and
the
other
one
is
a
local
prometheuses,
as
I
mentioned
that
we
are
still
mix
of
that.
We
want
a
single
source
of
truth,
so
we
wanted.
We
wanted
that
to
be
wavefront
integration
so
that
everything
will
be
picked
from
the
wavefront,
and
this
is
the
test
we
actually
executed
and.
D
D
Yes,
yes,
I
think
this
is
something
which
we
shared
long
back
and
last
performance
performed
2019
yeah
so
and
we
do
have
some
handy
stuff
going
about
the
chaos
workflow.
So
now
we
are
working
on
the
workflows
for
the
chaos
also,
and
this
also
need
to
be
integrated,
because
right
now
we
run
the
perf
test
and
then
we
can
also
run
the
chaos
test,
and
this
is
the
workflow
we
have
and
then,
after
that,
we
can
still
have
a
chaos
specific
sli,
slo
defined,
which
we
actually
have.
That.
D
C
Yeah
I
got
a
I
got.
I
got
a
question
for
you.
So,
first
of
all,
thanks
for
this
I
hope
you
will
find
soon
time
to
upgrade
captain
to
the
latest
version.
I
know
you've
just
been
traveling
to
india
for
the
last
couple
of
weeks
and
you
have
a
lot
of
other
things
going
on,
but
the
great
thing
is
with
the
latest
version:
you
will
the
heat
map
that
we've
seen-
and
you
said
you
know.
Sometimes
you
get
some
messy
data.
C
One
of
the
one
of
the
latest
features
in
the
heat
map
is
that
you
can
actually
invalidate
results.
Yeah,
I
think
that's
going
to
be
nice
and
it's
also
needed,
because
sometimes
you
just
run
tests
that
you
know
are
just
not
great
and
so
there's
a
lot
of
new
capabilities
in
the
latest
version.
We
should
check
it
out.
D
C
Also
talking
about
performance,
one
of
our
engineers
today
just
made
a
major
breakthrough
on
the
performance
of
the
captain
ui
when
we
query
a
large
scale
of
data,
so
you
will
be
very
happy
when
up
when
you
upgrade.
One
of
the
questions
that
came
in
is
talking
about
slis
and
slos.
You
mentioned
that
you
have
different
slos
for
your
different
workloads.
C
Now
yeah,
you
have
any
best
practice,
because
I
think
people
are
struggling.
At
least
the
people
that
I
talk
with
they're
struggling
a
lot
in
figuring
out.
What
are
the
best
sli's
and
slos.
Is
this
something
that
you,
where
you
say,
just
go
with
the
traditional
red,
metrics
or
response
time,
error
rate
and
throughput?
And
then
you
look
at
your
monitoring
data
in
production
and
figure
out
what
it
should
be
or
how
what's
the
best
approach?
What
can
you
tell
people.
D
Yes,
so
I
I
followed
the
what
captain
is
suggesting,
but
we
do
have
certain
sla
defined
for
our
services.
We
follow
three
concept.
One
is
the
error
rate.
We
don't
want
anything
which
is
beyond
point
zero,
one
as
a
error
rate.
We
want
our
response
time
response.
Time
is
something
which
we
have
actually
nailed
it
down
very
clearly,
so
we
define
percentile
99
percent
and
95
percent
and
90
and
percent
is
50..
So
what
that
mean
is
that
we
have
five
different
level
of
the
response.
Time
validation.
D
I
think
this
is
one
of
the
needle
mover
for
us
with
the
status
and
this
we
are
actually
ensuring
that
the
load
distribution
is
across
the
board.
We
don't
see
any
outlier
anytime.
We
see
anything
major
on
the
error.
We
should
see
that
is
it
because
of
the
infrastructure
platform
or
a
cloud
side,
so
that
actually
help
us
a
lot
to
figure
it
out
that,
but
I
think
keeping
sli
slo
for
standard
that
yeah.
C
Okay,
another
thing
that
I
would
love
to
see
and
again
this
is
something
maybe
we
can
work
together
in
the
future.
I
know
you're
currently
using
jenkins
to
trigger
your
test
and
then
once
the
test
is
done,
you
trigger
captain
captain
itself,
orchestrates
and
automates
processes.
So
actually
it
would
be
nice
to
see
an
integration
where
captain
can
actually
take
care
of
the
of
this
process.
Workflow,
where
captain
can,
for
instance,
also
trigger
this
draw,
and
after
this
choice
done
it
then
automatically
falls
into
the
quality
gate
evaluation.
D
C
D
I
think
again,
performance
testing
and
performance
execution
are
too
like
it's
a
vast
thing
for
our
need.
We
were
looking
only
for
performance
test,
we
were
not
and
we
we
have
a
pretty
clear
distinction
that
we
are
looking
only
for
study,
time
and
the
ramp
up
time,
so
we
want
ramp
up
and
the
study.
So
this
is
a
very
important
aspect
for
us
now.
If
you
have
a
different
way
of
performance
execution,
you
probably
need
to
valid
vary
your
sli
in
slo
for
our
use.
C
What
is
the,
what
is
the
difference
of
all
of
these
like
for
each
service
is
behaving
differently
right,
and
that
means
your
developers.
Then,
if
they
test
service
a
b
and
c
is,
do
you
educate
them?
Do
you
mentor
them?
Do
you
sit
down
with
them
to
then
define
the
slos
that
are
good
for
their
service
based
on
these
in
different
workloads,
or
how
does
they
work.
D
D
D
So
if
you
see
that
the
parameters,
what
we
are
defined,
this
is
just
for
one
service
and
one
environment
like
that.
You
have
various
environments.
So
here
this
is
the
environment.
Here
there
is
a
different
environment.
This
is
another
different
environment,
so
this
is
another
environment.
You
see
that
so
now,
for
one
environment,
it
is
going
for
five
parts
with
the
1200
as
a
stick
for
someone.
It
is
a
very
small
for
someone.
It's
a
pretty
lame.
D
Also
so
like
that's
the
way
how
we
do
do
the
distribution,
so
we
have
various
distribution,
you
put
it
in
the
config
and
then
you
do
that
so
again,
as
you
have
to
define
that
that
what
is
your
sla
and
then
you
put
it
in
that
and
based
on
that,
you
pick,
the
load
type
is
large
now
so
now
I
will
use
for
this
one
as
a
large
sli
slo
compared
to
the
other
one,
where
I
use
a
small.
So
here
in
a
qa,
it's
just
a
small
one
for
e2e.
C
D
Nope,
I
think
there
are.
There
are
a
lot
of
positive
on
that.
This
actually
helps
us
a
lot.
I
think
I
don't
have
that
one
of
the
big
execution
where
we
actually
executed
200
parallel
load,
distributed
test,
I
think,
which
actually
created
more
than
50
000
tps
load
across
the
board
and
all
those
things
we
are
able
to
validate
via
so
here.
This
is
a
one
big
execution
yeah.
So
it's
a
40,
000,
tps,
40,
000
tps
on
a
given
cluster,
is
a
pretty
pretty
huge
mark.
C
D
A
A
If
you
have
not
added
your
name,
I
tried
to
add
a
couple
of
names
that
I
saw
in
the
participant
list.
If
you
want
to
add
your
name
or
your
details,
please
do
so.
I
will
share
the
link
to
this
document
again
in
the
chat
and
also,
if
you
have
any
ideas
on
future
agenda
items.
If
you
want
to
comment
on
it,
so
I
will
allow
also
the
microphone.
So
maybe,
in
the
last
couple
of
minutes
we
can
just
have
a
discussion
if
you
want.
A
So
if
you
want
to
open
up
your
mic,
you
don't
have
to
and
yeah
we.
It
was
the
first
time
trying
this
format.
So
if
you
have
any
comments
on
this,
if
you
like
it,
if
you
don't
like
it,
just
let
us
know
also
around
the
future
agenda
items.
If
you
want
to
present
something
in
our
captain
user
group.
Let
us
know
if
you
want
to
hear
more
about
the
topic
that
we
suggested
here.
A
Also,
let
us
know
by
by
just
commenting
on
it,
and
then
we
can
try
to
make
it
work
to
present
you
these
topics,
and
we
can
have
discussions
on
this
also
sweet.
If
you
have
the
link,
if
you're
able
to
share
it
with
the
captain
community,
please
paste
it
in
here
the
link
to
the
slides
I've
already
added
the
blog.
That
would
be
cool.
I
think
it's
a
really
great.
It
was.
A
Chat
perfect,
thank
you,
and
otherwise
we
will
let
you
know
when
we
host
the
next
captain
user
group.
We
were
thinking
about
doing
this
on
a
monthly
cadence,
probably
on
tuesdays,
but
we
will
let
you
know
in
our
captains
like
and
also
on
the
twitter
account
that
we
have,
and
we
were
happy
to
to
see
you
there
as
well.
A
D
A
A
But
for
today
thanks
everyone
for
joining,
we
do
host
our
captain
develop
meeting
tomorrow,
5
pm
central
european
time,
so
basically
the
same
time
as
today,
if
you're
interested
in
building
integrations
with
captain
if
you're
interested
in
the
latest
developments
on
captain.
Please
join
us
tomorrow
at
5
pm
central
european
time,
and
you
will
find
all
the
the
links
also
in
the
on
the
captain
community
space
on
github.com
captain
slash
community.
Congratulations
again,.
C
C
Yeah
and
as
for
brian
becoming
a
member
of
the
user
group,
this
is
all
open.
We
will,
you
know
we
organize
ourselves
through
through
git
here
and
also
we
are
on
twitter
at
captain
project,
so
follow
us
there
and
you
will
see
when
we
have
new
announcements,
new
events
and
on
the
community
page
on
git.
You
also
find
the
link
to
the
google
calendar,
so
you
will
see
which
events
we
have
when.
A
And
I
would
just
share
with
the
that
was
cool,
very
interesting
article.
That
was
also
mentioning
argo
city
plus
captain,
so
I
think
it's
a
great
fit
for
today
it's
one
of
the
five
open
source
projects
to
watch
in
2021.
So
maybe
it's
a
nice
read
for
the
evening
and
with
that
thanks
so
much
everyone
and
see
you
next
time.