►
From YouTube: Cloud-Native Chaos Engineering WG Weekly Sync Up - September 17 2021 | CNCF TAG App Delivery
Description
Check out the recording from the Cloud Native Chaos Engineering Working Group Weekly Sync Up from September 17th, 2021
For more information check out the WG Charter: https://docs.google.com/document/d/1scr9uuvG1g1xpIHPs3314FqeFufE31ustTVnRMrX3gI/edit?usp=sharing
B
B
B
A
B
So
on
the
agenda
today
we
have
our
first
user
discussion
or
user
presentation
on
chaos,
so
we
wanted
to
get
started
with
inviting
members
from
the
chaos
engineering,
space,
people
who
are
end
users,
who
are
vendors
and
maintenance
of
kiosk
projects
which
probably
not
listed
in
this
group
to
start
talking
about
their
experiences,
chaos,
engineering.
So
this
goes
into
our
learnings
and
makes
its
way
into
the
white
paper
as
part
of
the
first
discussion
of
the
kickoff.
B
I
think
this
was
a
little
bit
of
a
short
notice
to
get
michael
in.
So
I'm
not
sure
if
we
have
a
demo
and
things
like
that,
we
will
go
through
our
slide
deck
template
that
we
created
and
go
through
the
questions
there
and
let
michael
speak
his
mind
on
you
know
how
he
sort
of
went
about
tackling
those
problems
or
issues
with
that.
I
think
that's
the
agenda
for
today.
B
Apart
from
that,
I
don't
really
have
any
other
item
I
know
has
been
working
on
some
security
related
documentation
and
they
also
sync
with
the
siam.
I
think
I
might
join
this
call
in
some
time
who
has
been
working
on
an
initial
tutorial
for
chaos,
engineering
which
will
go
into
the
paper.
There
is
some
progress
on
that.
B
B
C
B
C
I
joined
a
number
of
teams,
we
didn't
have
separation
between
production
and
non-production.
Basically
I
mean
same
people
would
maintain
production
and
same
people
would
develop
the
system
and
implement
code
as
well
yeah.
So
I
lost
this
gap
between
administration
and
development
in
that
time
and
it
nicely
fits
into
the
what
is
called
devops
nowadays.
C
There
are
no
other,
like
practical
ways
to
make
sure
system
is
resilient
and
reliable,
and
that's
when
we
discussed
this
chaos
engineering
scene,
because
it
was
everyone
knew
about
it,
because
it
was
sort
of
big
bank
when
netflix
introduced
that
chaos
monkey
sync
yeah.
On
the
other
hand,
everyone
was
quite
scared
of
chaos.
Engineering
because
I
guess
in
the
beginning
it
was
marketed
in
a
little
bit
wrong
way.
People
said
that
we
need
to
break
production.
You
know,
if
you
talk
to
some
managers,
they
say
they
don't
want
to
break
production
outright
right.
C
C
C
Strange
image
you
know
and
yeah,
but
at
the
end
of
the
day
I
after
proper
explanation,
people
understood
that
actually
chaos
being
an
opposite
of
reliability
right,
it's
it's
working
like
vaccine,
so
you
need
to
do
it
in
controlled
way
to
be
to
understand.
The
system
can
withstand
real
problems,
real
diseases,
real
issues,
and
so
I
worked
a
lot
on
that
as
a
devops
consultant
from
july
2020
to
july
2021,
exactly
one
year
and
now
I
joined
another
company,
let's
say
biggest
one
of
the
biggest
telecom
companies
in
australia.
C
B
Great
intro,
you
mentioned
a
couple
of
things.
We
sort
of
doing
this
user
discussion
a
little
bit
more
of
a
question
and
answer
style
being
the
first
one.
I
think
we
can
always
let
michael
come
back
for
closer,
look
and
demo
and
probably
structure
the
newer
presentations
little
differently,
but
just
to
go
ahead.
You
mentioned
a
couple
of
points.
B
C
Yes,
exactly
exactly
exactly
because
you
know,
certainly
not
everyone
is
a
good
engineer
or
it's
not
expected
to
be
a
good
engineer
right.
Some
people
are
just
like
a
team
managers
and
yeah.
They
know
this
like
caves
engineering.
Oh
it's
something
like
innovative
and
risky.
You
know
in
mind
of
in
the
mind
of
many
people,
so
ideally
to
make
it
clear
for
people
we
want
to
add
always
something
like
resilience,
testing
this
case
engineering
or
ensure
reliability,
risks
engineering.
C
Yeah
yeah
yeah
because
yeah,
if
you
yeah,
if
you
ask
layman
in
the
street,
you
know,
do
you
need
chaos
like
no?
No,
I
don't
so
do
you
want
a
reliability
like
yes,
of
course,
so
yeah
I
guess
it's.
It
should
be
ideally
like
immediate
second
step
in
during
the
introduction
of
this
important
engineering.
Subject.
B
A
B
B
But
what
were
the
challenges
that
you
faced
when
you
got
started?
We
talked
about
production
and
non-production
earlier
it
was
sort
of
mixed,
and
then
there
was
a
very
scientific
practice
that
came
up.
People
started
making
a
couple
of
ops
as
a
function.
They
had
staging
environments,
production,
etc.
B
C
Yeah
so
one
of
the
challenges
yeah.
First
of
all,
we
normally
do
it
in
pre-production
or
staging
environment
or
even
in
qa
environment-
not
production.
At
this
point,
so
I
guess
the
chaos
engineering
practices
organization
should
be
extremely
mature
to
expand
it
into
production.
It.
I
would
say
it's
it's
it's.
There
are
not
many
companies
in
the
world
who
can
afford
that
you
know,
but
actually
practically
sits.
In
my
mind
this
in
staging
environment
or
pre-production
environment.
C
Where
it's,
you
know,
you
still
can
potentially
break
environment
and
you
are
not
expected
to
interfere
with
quality
assurance
testing.
So
if
you
look
into
the
artificial,
you
know
classical
environment
where
there
are
developers
sandboxes,
there
is
some
integration
server.
There
is
environment
for
qa,
where
some
extra
testing
is
happening,
including
manual,
then
we
have
some
pre-production
staging
environment,
which
we
normally
don't
touch
much.
C
It
can
be,
for
example,
organized
in
a
way
that
the
development
environment
at
some
predef
predefined
times
like
after
hours,
can
be
subject
to
automated
chaos.
Engineering
based
resilience
testing,
it's
also
a
good
choice.
It's
exactly
actually
what
we
did
at
in
my
current
workplace,
because,
due
to
product
project,
project,
budget
and
complexities,
we
have
from
the
one
non-production
environment
and
we
just
need
to
agree
with
other
team
members
at
which
point
we
can
conduct
our
chaos.
C
Engineering
based
resilience,
testing
there
yeah
so
challenge
is
to
coordinate
and
apply
this
important
resilience
testing,
because
the
goal
of
the
test
is
to
break
right.
Good
tests
should
do
whatever
possible
to
break
the
system,
so
yeah
it
is,
can
be
in
contrary
to
like
immediate
users
of
the
system
in
any
environment
right.
C
Somebody
wants
some
endpoint
to
function
for
functional
testing,
for
example,
and
it
can
interfere
with.
You
know,
stress,
testing
or
chaos,
engineering
based
testing
so
cartonation
of
time,
and
it
can
be
also
noted
that,
in
addition
to
time
captivation
we
want
to
coordinate
budget
right
so
normally
having
dedicated
environment
for
this
sort
of
resilience.
Testing
is
like
too
costly.
B
That
leads
us
to
the
next
set
of
questions
you
already
touched
upon
it,
michael
so
the
persona
that
is
actually
practicing
the
chaos
engineering.
You
mentioned
that
at
some
point
you
would
like
to
have
some
automated
resilience
on
sandbox
environments
and
then
do
things
in
a
very
careful
way
on
the
staging
environment,
because
there
are
other
users
of
that
system,
so
your
brain
coordination
etc.
B
B
I
I
I
understand
that
you
mentioned
that
that's
what
you're
doing
your
current
workplace
but
generally,
is
that
something
that
depends
upon
the
culture
of
the
organization.
Is
that
the
domain
that
drives
it
or
is
it
just
the
freedom
that
management
gives
or
is
it
an
individual
choice?
How
does
it
generally
work?
Who
is
the
person
that's
doing?
Chaos.
C
C
Without
this
deep
knowledge
and
this
deep
knowledge,
it
exists
in
minds
of
like
devops
people
right,
not
classical
programmers.
So
it's
certainly
ideally
fits
in
between
development
and
operations.
So
I
would
even
say
if
we
have
devops
term,
we
have
also
def
sec,
ops,
right
development,
security
and
operations.
C
B
B
That's
a
very
interesting
perspective,
thanks
michael
so
you
mentioned
that
one
of
the
challenges
or
the
major
things
that
you
had
to
look
out
for
was
budgetary
constraints
and
time
coordination,
because
the
people
using
the
system
that
also
brings
up
another
couple
of
points.
One
is
multi
tenancy
or
you
know,
doing
chaos
and
shared
environments.
B
Sometimes
you
have
a
large
staging
cluster,
which
many
people
are
using.
Many
developers
might
be
using
it.
They
may
be
just
a
disc
folks
or
using
some
name
spaces
in
their
communities
clusters
or
even
in
non-communities
environments.
It
could
be
a
staging
system
that
testbed
that
are
using.
So
did
you
face
any
challenges
in
you
know
having
to
reduce
your
blast
radius
to
a
specific,
you
know
boundary
when
you
were
doing
your
experiments
was.
C
B
C
C
It
can
be
a
design
option
for
like
larger
systems
where
chaos
engineering
is
being
applied
by
like
teams
which
are
some
of
which
are
some
of
the
tenants.
You
know
in
this
if
you
have,
for
example,
tenants
10
tenants
in
such
a
system,
and
one
team
has
an
engineer
who
can
do
case
engineering
yeah.
I
can
imagine
that,
but
I
again,
I
don't
think
there
are
many
such
environments
in
the
world.
C
So
normally,
if
you
speak
about
the
platform,
normally
there
are
folks
in
the
platform
team
who
can
conduct
those
tests
or
they
asked
to
sit
up
now.
There
are
you
know
if
you
can
see
the.
On
the
other
hand,
if
you
can
see
the,
for
example,
aws
right
people
who
are
working
in
aws
are
tenants
of
aws
and
they
can
set
up
case
engineering
for
their
cloud
products
right.
A
B
Great,
the
other
question
was
around:
what
were
the
aids
you
needed
around
the
core
chaos
and
flood
structure?
You
have
the
test
bed.
You
also
have
some
chaos
tooling,
with
you
to
carry
out
experiments,
but
what
are
the
other
aids
that
you
sort
of
employed
as
part
of
your
experimentation
process?
Was
there
any
cicd
tools
that
you
used?
Were
there
any
observability
platforms?
C
They
are
happening
in
an
automated
way,
because
this
way
we
are
excluding
humans,
error
and
it
can
be
reproduced.
So
automation
in
this
regard
is
very
important,
but,
speaking
of
like
it
can
be
triggered
manually,
though
right,
nobody
cares
actually
in
practice
that
it
should
be
part
of
like
ci
process,
I
mean
normally
functional
tests.
Unit
tests
are
being
executed
in
ci,
but
those
if,
if
you
want
you
know
reliability
pipelines,
they
can
be
triggered
once
a
week,
for
example,
according
this
agreement
manually,
but
implementation
of
those
pipelines
should
be
automated
right.
B
C
B
C
Yeah
yeah
in
my
practice,
because
chaos
engineering
is
engineering
right,
so
people
really
really
want
to
define
themselves.
What
is
what
is
correct
and
what
is
incorrect
so,
basically
it
it's
quite
essential
to
have
ability
to
script
verdicts
of
what
is
expected
and
unexpected,
because
everyone's
case
is
special
unless
it
is
extremely
trivial.
C
So
basically,
teams
want
to
define
the
great
criteria
of
reliability
or
success
of
failures
themselves,
and
it
comes
to
tooling
so
something
like
json
data
should
be
available,
but
the
final
decision
should
be.
We
should
able
to
implement
some
simple
scripting,
which
will
say
if
this,
and
that
is
okay
or
higher
than
particular
values
and
test
passed.
Otherwise,
it's
not
yeah
speaking
of
observability,
so
full
logs
are
extremely
important
and
everyone
asks
like,
where
are
the
locks
of
the
whole
pipeline?
C
B
You
mentioned
about
pipelines,
and
you
mentioned
about
automated
runs
that
are
set
up
based
on
some
agreements.
Michael,
I
have
also
participated
in
game
days
where
there's
more
than
one
person
carrying
out
an
experiment.
You
have
you
have
people
from
different
teams
coming
together.
You
have
a
scenario
and
a
hypothesis
in
mind
that
you
do
manually
is.
C
C
B
B
C
Yeah
yeah
yeah,
normally
from
especially
from
my
team
manager
or
director
of
development,
point
of
view,
resilience,
testing,
stress
tests
and
load
testing
chaos,
engineering.
It
fits
into
the
same
bucket
as
security.
It's
like
non-functional
requirement,
so
for
them
it's
convenient
and
you
know
it's
justified
to
see
it
all.
As
like
non-functional
ensure
that
system
is
good.
C
You
know
in
non-functional
way,
so
security
is
no
not
broken
so
similar
to
how
we
want
that
durian
stress
or
chaos
effect.
Our
requests
are
being
served
for
for
the
users.
In
the
same
way,
we
want
to
ensure
that
when
there
is
disruption,
security
is
not
breached
right
so
again
like
in
real
life,
those
aspects
they
are
not
separated,
they
always
coexist,
and
I
guess
if
so
chaos
engineering
doesn't
exist
by
itself.
That's
the
primary
point,
I
guess
of
my
speech.
C
There
are
always
aspects
like
security
like
being
able
to
serve
requests.
At
some
rate.
Those
questions
are
always
asked,
and
basically
this
you,
you
know
you
are
not
able,
basically
to
define
in
advance
how
system
should
behave
during
chaos
disruption.
What
you
can
do,
though,
is
to
provide
some
information
insights
into
the
system,
to
let
this
particular
team
decide.
C
Is
it
good
or
not
in
general,
from
from
the
context
of
that
system
in
non-functional
way,
it
can
involve
security
questions
and
can
involve
number
of
requests
served.
It's
not
able
to.
We
are
not
able
to
predict
it
in
advance
right.
What
we
can
do
is
to
give
them
people
good
apis,
tooling
or
indicators,
to
use
to
draw
the
conclusion
about
quality
of
the
system.
At
the
end
of
the
day,.
C
B
About
how
did
we
make
sure
that
the
systems
and
platforms
were
not
compromised
during
the
chaos
experimentation
and
where
we
did,
you
have
any
detection
mechanism
that
did
detect
in
a
security
breaches
if
there
were
any
and
if
there
was
any
auto
response
built
etc?
Would
you
like
to
comment
on
that.
C
C
So,
for
example,
if
that
component,
which
is
performing
actual
authentication,
is
responsible
for
that,
it's
unavailable
due
to,
for
example,
network
failure.
Then
the
outcome
of
the
request
should
be
fail.
It
should
not
go
through
right,
rather
than
be
allowed
so
basically
again
like
in
practice,
it's
extremely
hard
to
separate
those
aspects.
So
people
especially
like
high
level
managers
or
architects.
They
look
at
always
at
the
system
in
a
holistic
way
right.
C
So
kerosene
engineering
is
like
universal
interes.
We
can
speak
about
application
of
disruption
or
effect
chaotic
effect.
Yes,
it
it's
like
more
or
less
unified
right,
but
when
we
come
to
measurement
and
conclusions
part,
it
should
be
very
flexible
to
allow
context-aware
specialization,
depending
on
the
particular
system
context.
A
Yeah
so
michael,
my
my
view
of
security
is
where
everything
is.
Security
has
code
right
and
everything
is
automated
detection
and
auto
response
to
that
detection
is
automated
too
that's
my
ideal
scenario,
but
in
most
enterprises
we
haven't
reached
that
full
state
of
automation,
right
where
you
will
get
alerts.
A
There
is
some
automation,
maybe
in
organizations
but
not
everything,
is
automated
and
then
the
whole
manual
sock
operations.
I
I
don't
believe
in
detection
right
it's
hard
to
get
the
confidence
and
the
resiliency
of
security
in
any
platform
today.
So
chaos,
engineering
from
that
perspective
is
very
interesting,
but
I
was
looking
for
some
security
use
cases
that
you
might
have
used
in
the
past.
You
know
that
you
can.
You
are
able
to
share.
C
C
Yeah
yeah.
Yes,
so
we
didn't
specifically
focus
on
security
chaos,
but
I
can
imagine
that
it's,
it
depends
on
the
age
of
the
project
again,
so
it
happened
to
me
that
in
the
both
places
I
was
building
the
platform
from
the
almost
very
beginning
and
those
like
things
like
security
chaos,
it
it's
an
advanced
practice
which
normally
is
like
day
two.
B
You
did
mention
I
used
case
by
way
of
example,
a
few
minutes
back,
I
think,
michael,
where
you
injected
some
network
loss
against
an
authentication
module
and
made
it
not
reachable
and
checked
that
your
requests
are
not
going
through.
I
think
that
was
the
expectation
that
we
had
probably
a
basic
level
security
chaos.
C
C
Yes,
yes,
yes,
so
yeah,
I
would
say
you
know
again
it
it.
It
would
be
very
practical
for,
like
people
who
are
working
on
improvement
of
this
practice
of
chaos,
engineering
to
give
a
lot
of
disruption,
effects
to
the
community
and
give
a
lot
of
easily
consumable
in
automation
like
indicators,
values
and
json.
If
you
want
and
suggest
how
to
use
them,
but
leave
with
that,
ideally
leave
as
much
flexibility
as
possible
to
draw,
in
conclusion,
to
particular
to
particular
teams
and
particular
engineer
engineers
who
are
users
of
that,
because
envisioning
in
advance.
B
B
You
already
have
given
us
an
answer
in
a
way
to
the
last
question
on
the
screen
as
to
what
you
would
like
to
see
from
the
chaos
engineering
community
in
cncf,
more
disruptions,
like
you,
said,
and
more
ways
to
indicate
those
destructions
and
methods
to
validate
whether
we
see
expected
behavior
or
not
just
to
cover
a
few
more
things
before
we
end
in
the
organizations
and
projects
where
you
did
implement
chaos,
engineering
practices
were
there
any
kpis
that
were
set
up
generally.
How
are
these
kpis
defined?
B
C
C
And
another
bit
is
provisioning
of
stress
number
of
replicas,
for
example
in
kubernetes,
will
our
configuration
of
cluster
autoscaler
scale
out
and
scale
in
back
when
we
don't?
We
remove
this
slot
and
it's
important
to
test
it
in
separation
like
similar
to
unit
tests
just
provisioning
number
of
replicas.
Without
all
these
requests
to
isolate
potential
problems
with
configuration
of
cluster
autoscaler
like
42
configuration
options
of
it.
C
C
C
C
So
it's
hard
to
predict
like
what's
needed
so
again.
Speaking
of
like
development
of
frameworks
for
ks
engineering,
I
guess
the
focus
should
be
on
visibility
into
the
components
like
system
components
and
various
effects
of
inducing
those
issues
in
disruptions,
because
it's
it
can
be
generified
to
good
extent.
B
B
B
B
B
So
we
will
have
that
discussed
in
our
next
call
how
we
are
putting
things
up.
Meanwhile,
if
you
would
like
to
invite
folks
from
the
community
and
from
the
industry
to
come
and
present
about
chaos,
engineering
that
would
be
very
appreciated,
so
feel
free
to
bring
your
contacts
in
for
this
session.
The
idea
would
be
to
have
at
least
four
sessions
like
this
and
have
a
working
draft
by
the
cubic
on
time.