►
From YouTube: Make Cloud Native Chaos Engineering Easier - Deep Dive into Chaos Mesh - Cwen Yin, PingCAP
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Make Cloud Native Chaos Engineering Easier - Deep Dive into Chaos Mesh - Cwen Yin, PingCAP
Chaos Mesh is one of the most popular open-source chaos engineering platforms, with the goal of making chaos engineering easier and more accessible. In this session, Cwen will provide a brief overview of Chaos Mesh. He will then dive deep into how Chaos Mesh is implemented and its concrete practices. Cwen will also guide the audience through how to better apply observability to a chaos engineering Workflow, in order to improve the efficiency and effectiveness of chaos experiments. In the end, he will share how the team’s latest exploration around chaos engineering and their plan to evolve Chaos Mesh.
A
A
A
Force
can
happen
anytime
anywhere
else
and
in
any
way,
so
many
phones
can't
be
avoided.
We
can
avoid
many
force,
such
as
the
unstable
network
and
the
disk
failures,
and
so
on
so
and
just
the
writing
test
and
the
debugging
for
this
issue
is
very
hard
when
we
developed
a
distributed
with
tidb.
We
also
have
the
same
issues
and
we
want
to
find
a
solution
to
improve
the
distributed
base
stabilities.
A
So,
finally,
we
find
kelsey
and
urine-
and
here
is
our
official
or
official
definitions
of
the
chaos
and
diaries
it
says.
Chelsea
nearing
is
about
breaking
sins
in
control
in
control
and
stacking
environment
and
the
source
of
well-planned
experiments
in
order
to
build
conditions
in
your
application
to
withstand
turbulent
conditions
and
by
these
definitions.
The
key
word
is
experiments.
A
The
other
keyword
is
a
control
controller
and
the
role
plan,
because
chelsea
engineering
is
not
about
breaking
things
randomly
without
a
purpose,
because
we
use
chaosmet
to
find
the
potential
problems
and
rather
than
to
use
it
to
affect
the
normal
uses
of
the
systems,
normal
uses
of
the
applications
and
so
on
and
the
beginning
of
my
journeys
of
the
kelsey
nuri.
I
also
made
some
problems.
A
A
Is
very
difficult
for
for
me
to
measure
and
schedule
so
many
klc
permits,
and
there
is-
and
also
we
find
it's
also
difficult
to
inject
some
low
levels
failures
on
this
database,
because
so
many
tools,
such
as
pc
tables
fields
and
so
on.
These
two
couldn't
be
used
in
kubernetes
environment
directly,
so
we
need
a
tools
that
can
provide
some
low
levels
for
ejection,
such
as
flight
systems
and
kernel
levels,
and
so
on
and
and
then
times
about
three
years
ago.
A
A
What
kills?
What
is
scale
smith?
The
answer
is
actually
changes
all
the
time
and
it
is
relaxed
related
to
the
evolution
of
the
chaos
niche
and
the
beginning
of
the
encouragement
journey
on
cosmetic
goal
was
simple.
We
just
want
to
have
a
full
injection
and
measurement
tools
that
can
grow
well
in
kubernetes
environment,
and
these
tools
can
provide
the
low
level
for
injections
and
and
first
we
just
use
this
to
to
test
tadp
after
we
did
do
this
for
a
while.
We
realize
then
this
tools
can
be
a
general
tools.
A
You
can
use
it
to
define
your
applications,
data
checkings
and
also
we
have
a
more
various
times
of
the
four
injections
and
so
on,
and
during
these
journeys,
cosmet
also
injects
joining
the
cncf
and
and-
and
then
this
year
we
also
saved
as
a
immigration
project
of
the
cncf
and
the
projects,
evolves
and
project
bombs.
Classmates
also-
and
the
goal
of
the
councilman
is,
is
clear
and
is
our
make
calcium
error
ether
is,
is
our
goal,
and
also
this
goal
also
on
guidance
directly.
A
Next,
oh
next,
let's
dig
deep
into
chaos
mesh
and
here-
and
this
is
the
handkerchief
architecture
you
can
use-
you
can
see-
you
can
use
the
chaos
match
and
continues
plugging
because
transmit
all
case
times
and
workflow
times
objects
just
design
and
and
use
the
crd
to
define
this
case.
Science
and
such
as
body
cares,
network
health,
stress
skills,
hour,
chaos,
workflow
and
so
on,
and
so
and
if
you
are
familiar
with
the
kubernetes
and
you're
familiar
with
the
operator
mode,
you
will
be.
A
You
will
be
familiar
with
chaos
mesh
and
then
because
classmate
is
a
very
in
the
architecture
is
very
easy
and
very
simple.
You
can
see
that
classmates
include
four
components.
The
first
one
is
cast
dashboard,
it's
a
web
ui
for
user
to
managers
and
of
the
scales
experiments,
and
the
second
is
the
chaos
control
members
is
it.
It
is
the
curve,
components
and
uses
it
is
used
to
schedule
and
to
measure
the
health
premise,
and
also
this
commander
also
includes
a
workflow
engine.
A
Our
net
native
workflow
engines
and
the
customer
customer
is
executed,
securities
components
and
will
be.
It
will
be
drawn
here.
A
demon
set
and
will
be
deployed
in
every
kubernetes
nodes,
and
this
is
ksd.
Cassidy
is
a
same
to
the
current
theme.
It's
also
excused
bonus,
but
these
components
will
be
deployed
on
a
lot
of
daily
notes
and
also
you
can
inject
rejects
and
these
components
to
the
case
match,
and
then
you
will.
A
And
this
sorry
and
this
picture
is
the
this
picture
shows
up,
shows
the
whole
workflow
of
the
characters,
and
you
can
see,
then
this
workflow
can
be
depended
in
three
parts,
and
here
is
the
first
part,
and
this
here
is
the
second
part-
and
here
is
the
third
part
in
the
first
part
and
is
about
the
user's
user
pass.
The
users
can
use
a
kuber
command
to
apply
this
command
or
use
the
cast
dashboard
to
input
your
to
apply
your
class
experiments
or
workflows,
or
your
all
kills
workflows.
I
know
I
know.
A
A
I
I
have
introduced
this
there
and
customer
is
his
cutest
component
and
there
and
it
will
be
wrong
here.
Demon
said
and
will
be
deployed
in
each
comedic
nodes,
and
then
these
components
of
simon
will
receive
a
requirement
from
the
case
control
manager,
and
then
there
will
be
enter
to
enter
the
target
post
some
network
namespace
or
enter
the
pid
namespace.
A
A
Next
chaosmet
define
multiple
crd
times
based
on
different
four
times.
I
have.
I
also
introduced
this
and
the
previous
slides
such
as
this
is
a
portable
put.
Chaos
and
the
network,
health,
and
so
on-
stress,
scales,
overflow
physics,
physical
machines
and
so
on,
and
here
is
a
simple-
is
an
example
of
the
podcast.
A
It's
very
in
this.
This
divide
defines
his
jump
files
and-
and
it
is
very
simple-
this
scales,
experiments,
simulates,
the
random
scale,
a
random
ti
q
report
will
be
killed
and
here
a
level
selector
will
select
all
every
post
with
the
resist
levels
with
this
level
and
the
mode
this
represents,
the
kelsey
minimum
will
be,
will
select
one
randomly
from
the
output
than
the
means
that
meet
the
selectors,
and
here,
if
this
is
the
best
case,
object,
if
you
want
to
define
some
schedule,
rulers
for
chaos,
experiments
you
can
use,
you
can
use.
A
The
schedule
object
to
define
this
and
it's
rules
and
the
this
schedule
object
is
like
the
the
use,
the
schedule
rules.
It
lacks
the
current
job
in
kubernetes
and
this
this
schedules
define
the
broadcast
experiments
with
clone
rules,
and
this
object
will
create
a
new
postcare
of
the
hourly
file
meanings.
A
And
when
you
apply
this
skills
experiments,
we
can
check
the
readout
on
calc
dashboard,
and
also
you
can
you
can
you
can
check
the
results
on
your
application
monitors
and
like
this
in
this
the
the
moment
of
the
applications
like
this,
you
can
find
this
will
apply.
This
schedule
objects,
you
will
find
the
qps
will
be
dropped
to
will
resolve
and
the
incident
is
returned
to
normal
and
will
be
this,
and
this
behavior
will
be
happens.
Every
five
minutes.
A
Next
chaos
mesh
also
includes
a
negative
workflow
engines.
This
is
decided
for
chaosmatch
and
also
used
to
design
the
case
scenarios
to
measure
a
group
of
the
cal
experiments
and
the
standard
checks
of
the
application
here
is
also
an
example
of
the
workflow
object
which
match
this
news
method
list
contains
three
parts:
workflow
names,
the
entries,
the
entries.
A
The
entries
is
the
entries
of
the
whole
workflow
and
also
include
a
timeless
set
of
the
templates
and
the
present
classmates
spot
five
times
of
templates
in
serials
parallels
and
chaos
suspects
and
the
tasks
in
this.
These
five
times
of
the
templates
and
servers
and
the
parallels
templates
define
the
rules
to
run
the
real
times
no
task
and
the
test
families
close
timeless
is
used
to
define
a
specific
skills
action
which
will
be
translated
and
translated
to
different
test
times
such
as
such
as
such
as
polar
chaos
and
network
cares,
and
so
on.
A
Task
templates
is
used
to
define
some
custom
tasks
such
as
we
can
use
this
to
define
the
status
check,
task
and
so
on,
and
here
is
a
very
simple,
very
simple
examples
and
test
match,
and
this
workflow
is
very
simple
and
it
is
used
for
user
to
define
your
chaos
and
workflow,
and
you
also
can
define
your
workflow
on
cashbaseboard,
so
you
can
define
your
workflows
and
your
jump
files.
You
also
can
define
your
workflow
on
cassette
dashboard
by
the
other
forms
and
so
on.
A
And
killsmith
also
defined
a
multiple
selector
for
users
to
define
the
scope
of
the
carry
experiments,
including
a
namespace,
selector
level.
Selector,
annotation,
selector,
node,
selector
and
so
on.
Here
is
a
selected
example.
This
selector
represents
the
scope
of
the
characters,
is
the
the
poles
with
this
level
and
also-
and
you
also
can
set
these
levels
on
cast
dashboard,
and
here
you
can
see
the
namespace
levels,
selectors
and
level
selectors
and
annotations
level,
selectors
and
so
on.
A
A
Users
can
create
a
specific
service
account
with
the
limited
author
authorization
such
as
you
can
create
the
main
rules.
You
can
create
a
view,
viewer
rules
and
you
can
use
and
then
use
the
tokens.
Then
this
coconut
was
generated
by
the
service
account
and
then
you
you
can
use
the
token
to
login
secure
dashboard,
and
then
you
will
use
these
users
and
this
user
will
be
have
the
limited
authorizations.
A
A
Okay,
next
is
a
demo
time
for
most,
the
user
can
try
cal
smith
easily
and
quickly
curse
match
pro
website
provider
in
interactive,
interactive
tutorials,
and
this
tutorial
is
contributed
by
casemate
communities,
members,
unisa
and
also
it's
built
on
cataclysm,
and
this
time
I
will.
I
will
use
the
in
interactive
tutorial
to
show
this
demo.
A
A
Yeah,
you
can
see
that
this
tutorial
will
introduce
and
introduce
some
information
about
your
smash,
and
you
can
next.
You
can
click
here
to
start
this
demo
yeah.
Maybe
they
may
need
some
time
to
prepare
some
environment,
and
in
this
demo
a
tesla
kubernetes
cluster
will
be
created
by
carcoder,
and
here
you
come
in
a
moment.
A
And
here
is
the
the
step
of
these
two.
The
tutorials
may
have
six
six
steps.
You
can
follow
these
tutorials
to
start
your
first
skills
experiments
here
and
here
is
a
share
command.
You
can
delete,
and
here
you
this
command
will
show
the
media's
clustering
information,
and
here
you
can
see
the
class
in
the
community
cluster
versions
and
also-
and
here
you
can
clean
this
amount-
to
check
the
hem
version,
because
we
need
user
hem
to
install
chaos
match
on
this.
On
this
kubernetes
cluster
yeah.
A
A
A
A
Uploader
is
running,
and
here
and
yeah
you
can
see.
The
customer
dashboard
is
open
and
we
can
access
this
by
this
link,
and
here
and
the
first
we
will
have
overviews
of
the
test
match.
They
include
your
hand,
you
you
have
how
many
experiments,
how
many
schedule
workflow
is
running
and
here
is
a
workflow.
A
For
this,
for
you
for
in
this
demo,
we
we
also
need
a
a
target
application
and,
and
for
this
demo
I
will
install
application
recorders
web
show
and
this
web
application
is
a
web.
Your
web
ui
have
a
web
ui.
You
can
see
the
network
delay
from
this
port
to
the
system,
port
and
this
network
layer
will
be
recorded
on
this
application
and,
first
and
and
and
this
this
command
resists
this
command
to
install
this
application.
A
Yeah
and
it
applies
the
web-
show
deployment.
This
application
is
running
deployment,
and
here
is
the
together
to
install
the
service,
for
we
can
access
the
this
application,
and
here
use
this
command
to
check
the
application
status
we
and
we
need.
We
actually
need
to
make
sure
the
application
is
running
and
then
to
tools
to
continue
the
next
steps.
A
A
A
A
You
can
see
open
this
yeah
at
the
moment.
Yeah.
You
can
see
that
this
application
will
require
the
network
link
between
the
the
application
port
to
the
from
the
application
port
to
the
system
mode.
You
can
see
that
the
lender
see
is
about
one
millisecond
and
then
next
I
will
start
a
network
house
to
inject
the
10
millisecond.
A
And
here
yeah,
you
can
see
this,
you
can
click
here
to
to
to
check
the
well
the
the
network
house,
con
definition
you
can
see,
then
this
this
network
has
this.
This
jump
file
define
a
network
house
and
the
action
is
delay
is
selector,
is
a
web
show
and
the
length
c
is
10
millisecond
and
this
represents.
We
will
inject
10
milliseconds
on
our
target
application,
and
also
you
can
use
this
selector
to
make
sure
this
selector
will
be
select
our
the
target
port.
A
You
can
make
sure
this
like
this
yeah
you
can
see,
then
we,
this
lecture
will
be
select.
The
web
show
port
and
a
week,
and
here
for
for
for
the
for
same
time
we
can.
We
just
use
the
apply
to
apply
this,
to
apply
this
jump
file
to
kubernetes
clusters
and
there,
and
then
they
will
be
just
the
network
delay
on
our
target
application.
A
Yeah
you
can
see.
Then
google
fly
you
can
see.
Then
this
network
house
will
be
created
and
and
then
you
you
can
use
command
to
check
this
yeah.
You
can
leave
this.
This
network
account
has
been
created
on
kubernetes
cluster
and
also
you
can
check
the
results
on
the
web.
Ui
on
the
web
show
ui.
You
can
see
this
yeah,
you
can
see
the
network
delay
turned
to
the
10
millisecond
and,
and
this
represents
the
chaos
experiment-
is
work.
A
This
command
will
be
added
annotations
if
annotation
is
web,
show
network
else.
Delay
will
be
and
end
this
annotation
with
a
network
object
and
the
the
annotation
is
this
experiments
calc
match
then
top
on
org
and,
and
here
is
annotation
and
you
you
can
check
the
readout
you
can
see,
you
can
see
that
the
normal
and
the
the
network
delay
has
is
turned
to
normal,
and
also
you
can
check
the
results
and
check
the
calcium
primers
and
the
two
of
the
the
calc
experiments
on
cal
state,
full,
it's
okay!
A
You
can
see
that
the
this,
and
this
case
remains,
is
supposed,
and
also
you
can
click
here
to
get
the
detailed
information,
and
here
we
will
show
the
events
about
this.
Skeleton
premise
on
cast
dashboard-
and
here
is
definition
here-
is
the
basics
and
basic
information,
and
also
you
can
studies
and
then
restart
it,
and
also
you
can
achieve
it,
and
and
it's
oh,
it's,
okay,
and
if
you
want
to
restart
this,
you
also
can
use
to
remove
the
annotations.
A
A
A
And
also
you
can
just
use
a
good
delete
and
to
delete
this
network
house
yeah.
You
can
see
check
the
results
on
the
workshop
application
yeah.
You
can
see
that
the
network
delay
turn
to
normal
and
also
you
can
check
the
result
on.
You
can
check
the
health
experiments
on
cal's
dashboard.
Oh,
it's,
okay!
You
can
see
that
the
the
network
health
has
been
delayed
when
you,
when
you
deliver
the
experiments,
you
deliver
the
schedule
or
workflows
that
will
be
achieved.
A
A
Okay
and
also
next,
we
also
have
other
demos
about
the
schedule,
object
and
also
about
the
podcast
demos
and
so
on.
If
you
are
interested
about
test
match,
you
can
try
things
and
it's
very
easy
and
very
simple:
you
just
open
skillsmatch
website
and
follow
the
interactive
tutorials
and
you
you
can
try
scale,
smash
and
also
you
can
start
your
first
case
and
there
neurons
of
foster
care,
experiments
and
very
cool.
It's
very
cool,
okay.
Next,
let's
turn
to
our
slides.
A
Okay,
next
is
about
the
future
plans
in
the
future.
We
will
continue
to
focus
on
use
of
use
and
of
the
of
the
ability,
such
as
we
plan
to
provide
a
more
comprehensive
data,
inspection
mechanisms,
and
we
will
provide
a
a
more
simple
and
more
powerful
report
and
also
we
improve
the
availability
provide
by
the
even
logs
and
matrix
and
so
on
about
security.
A
We
also
plan
to
develop
a
new
component
to
foster
coverage,
calc
improvements
which,
which
are
used
to
program
the
chaos
experiments
being
out
of
control,
and
also,
we
also
plan
to
provide
a
plugin
to
extend
the
company's
skills
times.
Users
can
use
this
plugin
to
develop
their
own
test
time,
such
as
ruby
and
quest
ready
scales
and
so
on.
We
also
will
build
a
test
hub
for
users
to
share
their
own
case
times
and
to
share
their
workflow,
which
can
be
reduced.
A
The
cost
of
the
new
users
on
calcium
permits
next
is
about
some
chaos,
mesh
communities
and
some,
and
maybe
some
research
you
can
find
on
our
communities.
I
think
customer
website
is
a
good
place
for
users
to
style
your
case
to
style,
chelsea
and
theories,
and
you
can
find
the
document
on
our
calendar
website
and
they
guide
the
user
to
how
to
install
customers
and
how
to
quickly
start
their
own
calculations.
A
In
addition,
you
also
can
find
some
user
user
cans
use
use
case
on
transmit
blog
and
also-
and
here
is
also
our
cosmetics-
youtube
channel.
You
can
find
some.
It
is
also
good
research.
We
have
a
monthly
community
meetings
and
there
there
are,
there
will
be.
We
will
be
uploads
the
videos
on
this
channel.
You
can,
you
can
find
the
meetings
record
and
you
can
find
this
and
also
you
also
can
join
the
chaos
match.
I
joined
the
cncf
killsmith
channel
and
the
whole
community
is
very
active
here.
A
We
also-
and
here
is
our
github
occupy
repo,
and
we
also
welcome
you
to
create
an
issue
on
the
chaos
match,
get
high
repo
or,
if
you,
if
you
are,
if
you
want
to,
if
you
have
some
future
requests,
you
also
can
create
an
issue
on
this
great
repo,
and
also
you
can
just
remove
appear,
pull
required
to
help
us
to
improve
this
project,
and
kel
smith
is
still
still
young.
Everyone
is
welcome
to
participate
in
evolution
of
the
chaos
match,
and
I
welcome.
I
welcome
all
people
to
join
our
community.