►
From YouTube: What’s New In Chaos Mesh And Deep Dive Into Multi Clusters Support - Ed Huang & Chao Zheng, PingCAP
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
What’s New In Chaos Mesh And Deep Dive Into Multi Clusters Support - Ed Huang & Chao Zheng, PingCAP
Chaos Mesh is one of the most popular open-source chaos engineering platforms, with the goal of making chaos engineering easier and more accessible. In this session, In addition to a brief overview of Chaos Mesh, Cwen will provide an update about the latest feature developments, and deep into how to support chaos experiments on multiple Kubernetes clusters. In the real case, chaos experiments across multiple clusters are often needed, to achieve it and make it easy, chaos mesh supports this feature natively. In this session, Cwen will introduce how to design and implement this feature and introduce typical real world use cases.
A
Welcome
all
of
you,
of
course,
to
to
you
know
in
this
room
to
yeah.
Today
we
are
going
to
give
the
you
know
like
the
the
what's
new
in
chaos
match
and
also
I
want
to
share
a
little
bit
history
about
cashmere.
You
know
this
drone
is
the
maintenance
check
of
maintain
a
chart
of
cash
mesh,
but
technically
I'm,
not
a
maintainer
I
would
say,
but
I'm.
The
kind
of
like
the
creator
of
cast
mesh
and
I
would
say
Carousel
smash
is,
is
a
project
that
is
created
by
the
accident
yeah.
A
So,
first
of
all,
I
want
to
do
a
really
quick
survey.
Who
is
any
of
you
are
using
cast
mesh
in
production?
You
can
raise
your
hand
only
one
I
know
one
okay,
cool
I
would
say:
I
can
see
the
great
potential
of
this,
this
open
source
project
so
actually,
before
introducing
you
know,
the
newest
update
of
our
cash
mesh
I
want
to
give
a
little
bit
about
the
history
about
chaos
mesh.
A
Well,
you
know
we
are
come
from
the
the
company
called
pin
cap,
which
is
the
a
distributed
database
company.
We
are
seven
years
ago.
We
started
to
started
to
build
a
you
know:
open
source
version
of
Google
spanner,
which
is
which
is
idb.
It
is
highly
distributed
massive
scale
distributed
system
and
provide
the
full
feature
stickers.
So
that
means
it's
super
complicated
disability
system.
A
So
that's
it's
really
hard
to
build,
but
I
think
the
even
harder
problem
is
that
how
to
test
the
the
you
know
a
distributed
system
like
this
I
think
it
is
even
harder
than
you
know
to
build
a
you
know,
distributed
database
in
internally.
I
I
have
a
calculate
the
the
code,
the
of
the
hour,
you
know
like
unit
test
and
the
test
case.
I
I
think
the
the
the
the
line
of
code
of
our
you
know.
Integration
test
or
test
case
is
like
10
times
larger
than
the
actual
code
base
of
our
database.
A
So
that's
that's
what
we
said
the
how
to
test
the
distributed
system
is
really
really
hard
and
a
good
way
to
test
the
you
know.
Distributed
system
is
actually
you
know,
deletes
your
your
file
in
in
your
in
your
disk,
I
I
bet.
Some
of
you
may
may
know
this
TV
show
like
Silicon
Valley
and
it
is
accidentally
accidentally
deletes
the
the
the
file
that
I
think
that's
a
good
way.
Maybe
that's
a
good
way
to
test
the
distribute
assistant
and
another.
You
know
very
interesting.
A
You
know
Speech
is
given
by
Foundation
DB
and
in
2014
in
this
in
exchange
Loop.
This
this
talk
is
actually
really
really
good,
and
this
talk
is
long
before
that
Netflix
introducing
the
idea
about.
You
know,
chaos
engineering,
so
it's
like
I
think
eight
years
ago
and
before
we
even
create
a
pin
cap
and
I
I
I
watched
that
that
that
talk,
I
sing,
I
learned
a
lot
from
this.
A
This
talk
because
in
this
talk
this
guy
introducing
the
very
interesting
concept
means
deterministic
simulation
for
the
you
know
for
the
discrete
assistant,
so
that
guy
is
basically
want
to
build
the
you
know:
deterministic
Simulator,
for
all
of
the
to
emulate
all
of
the
complicated
status
of
your
your
system
and
to
reproduce
the
the
bug.
100,
reproduce
the
the
the
bug
but
I
think,
okay,.
A
I
think
it's
not
a
the
deterministic
testing
for
the
you
know.
Large-Scale
distributed
system
is
not
that
realistic
so,
but
on
the
other
hand,
I
think
increase
the
the
probability
of
reproducing
bug
by
you
know
inject
the
failure
to
the
the
the
system
is
is
doable,
and
so
we
are
not
going
to
create
100
reproduce
rate
for
for
your
system.
Just
like
unit
test
I
think
unit
test
is
not
unit.
A
Tests
are
not
looking
for
you
know
a
lot
of
you
know
Pro
box
in
complicated
distributed
system,
but
I
think
the
fourth
injection,
which
is
like
chaos,
engineering,
is
more
reasonable
way
to.
You
know,
create
a
high
quality
system
on
a
high
quality
system
in
the
distributed
world.
So
that's
the
beginning
of,
and
we
take
a
look
at.
A
We
we
called
sharinger
if
you're
familiar
with
quantum
physics,
you
know
what
I
mean
about
this
name.
It's
basically
create
a
container
based
for
the
injection
system,
for
you
know
for
for
our
database,
so
it
is
only
designed
for
for
for
for
tidy
B,
but
I.
Think
that's
the
you
know,
and
then
we
open
source
this
this
project
at
changing
the
name
as
the
cash
mesh.
That's
the
beginning
of
cash
mesh.
A
A
B
In
the
following
slides
and
an
assert,
one
is
multi-kubernetes
support
which
allow
users
to
launch
chaos
workflow
on
multi-cluster,
using
one
Global
controller,
okay.
So
the
first
feature
I
would
like
to
talk
about
is
the
Azure
chaos,
so
SN
M
imply.
This
feature
allows
us
to
run
Chaos
workloads
on
top
of
the
Azure
clouds.
So
before
2.4
we
have
already
support
gcp
chaos
in
AWS
chaos.
We
have
already
received
the
Manning
positive
feedback
from
a
community.
So
very
naturally
we
wanted
to
extend
the
cloud
provider
feature
sets
by
implementing
this
Azure
chaos.
B
So
the
underlying
idea
of
the
Azure
chaos
is
very
similar
to
gcp
chaos
and
AWS
chaos,
which
authenticated
the
chaos
mesh
controller
manager
to
access
the
control
to
access
and
control.
Azure
Cloud
resource
under
the
user's
account
some
simple
Azure,
some
simple
chaos
actions.
We
provided
our
Virtual
Machine
Stop,
which
allow
user
to
stop
specific
virtual
machine
instance.
B
So
nowadays
we
know
the
public
cloud
provider
provide
very
reliable
service,
but
some
unexpected
instance
can
still
happen,
sometimes,
like
the
whole
rack
of
servers,
May
temporarily
out
of
use
due
to
fire
or
power
outage
and
a
VM
stop.
Can
help
us
to
simulate
this
situation
and
the
next
next
one
is
VM
restart.
This
one
is
a
slightly
different
between
a
slightly
different,
because
sometimes
the
virtual
machine
may
keep
restarting
during
a
period
of
time
because
of
Hardware
issue
or
resource
scheduling
problem.
B
This
Behavior
can
add
extra
pressure
to
distribute
system
that
require
coordination
between
peer
nodes.
For
example,
if
you
are
using
Thai
KV
as
the
backend
storage,
if
a
server
keep
restarting,
you
will
have
a
peer,
keep,
leaving
and
rejoin
a
network
which
will
add
extra
synchronization
overheads
to
the
whole
system.
The
third
one
is
the
disk
detaching
detaching.
Most
of
the
virtual
machine
instance
on
the
cloud
nowadays
are
using
a
Unified.
B
B
Here
is
cloud
key
secret
and
apply
this
to
the
kubernetes
cluster.
Then
we
can
Define
Azure
chaos,
workloads
which
will
point
to
the
secret.
We
just
created-
and
the
defined
action
we
wanted
to
take
here-
is
the
VM
restart.
Finally,
we
apply
this
workload
to
the
Azure,
to
the
kubernetes
and
in
the
chaos.
Mesh
controller
will
load
it
and
authenticate
in
a
perceived
action.
B
The
second
feature
I
wanted
to
introduce
is
the
block.
Chaos
in
early
years
of
kubernetes,
most
of
users
are
using
kubernetes
to
build
status
application,
but
nowadays
we
are
seeing
increasing
demand
of
building
stable
application
on
top
of
kubernetes,
so
for
stable
application.
What's
the
top
priority,
it
is
saving
the
date
correctly
and
efficiently.
Therefore,
staple
and
high
performance
storage
is
very
critical
and
ensuring
application
can
still
provide
servers
with
temporarily.
I
o
performers
degradation
is
even
more
critical.
B
To
least
to
to
this
end
we
Implement
block
chaos.
You
may
be
a
little
bit
curious
because
before
2.4
we
actually
have
an.
I
o
related
chaos,
workload
called
IO
chaos,
which
can
also
be
used
to
simulate
the
scenario
of
unexpected
I
o
latency
on
block
device.
Then
why
we
introduce
a
new,
I
o
related
feature,
the
I
o
chaos
work
works
well
in
most
of
the
case,
but
it
may
come
with
a
side
effect
when
simulating
a
case
of
accessing
huge
amount
of
files
to
implement
the
io
chaos.
B
We
run
a
chaos
sidecar
container
inside
the
target
Parts
using
p-trace
to
replace
the
file
descriptor
of
the
target
process
and
then
take
control
of
the
file
descriptor
by
adding
extra
latency
to
the
I
o
operation.
This
mechanism
works
well
in
most
of
the
case,
but
if
the
process
is
opening
huge
amount
of
BIOS,
like
thousands
or
tens
of
thousands
file,
then
the
P
trays
will
post
the
target
process,
and
that
is
some
side
effects
side
effect.
We
don't
want
to
import
so
to
better
simulate
the
I
o
chaos
scenario
without
importing
side
effect.
B
We
implement
the
block
chaos
in
version
2.4,
so
Blockhouse
simulate
the
case
of
extra.
I
o
latency
on
block
device
by
implementing
a
kernel.
I
o
scheduler
and
offering
two
key
features.
First
is
the
I
o
delay,
which
allow
users
to
specify
the
latency
of
the
block
device
and
on
the
second
one
which
is
currently
under
intensive
development,
is
the
limit
I
o
Ops,
which
allow
user
to
specify
the
iops
of
the
block
device.
B
So
how
does
block
chaos
work
internally,
First
We
Take
a
look
at
how
a
normal
I
o
request
will
be
handled
on
Computing
nodes.
After
the
application
sending
a
read
or
write
requested
to
the
file
system,
the
file
system
will
then
send
the
corresponding
block.
I
o
requested
to
the
kernel
request
the
queue
elevator,
so
this
request
queue.
Elevator
contains
multiple,
I
o
scheduler
from
the
kernel
and
then
a
block.
I
o
request
will
go
through
all
kernel.
B
I
o
scheduler
be
sent
to
the
hardware
request
queue
passed
to
the
driver,
insert
it
into
the
nbmeq,
which
is
the
short
of
non-volatile,
Memory,
Express
and
finally
be
right
into
the
disk.
As
I
mentioned,
we
implement
the
kernel.
I
o
scheduler
called
ioem,
which
will
be
added
into
the
request:
queue
elevator.
So
when
a
block
IO
reach
the
ioem,
instead
of
passing
it
to
the
next
scheduler,
the
ioem
will
hold
it
for
a
specific
time
and
then
pass
it
to
the
next.
I
o
scheduler,
which
simulates
iOS
latency
scenario.
B
So
here
is
a
brief
example
of
how
it
works.
We
start
with
the
right
request.
Chaos
is
cool
to
the
specific
block
sector
and
ioem
scheduler
capture.
The
request
hold
it
for
three
millisecond
and
the
password
to
the
hardware
queue
compared
to
the
iok
US.
These
features
simulate
the
I
o
throughput
decreasing
in
a
more
natural
way
without
posing
the
process.
B
The
third
feature
I
would
like
to
talk
about
is
physical
machine
chaos.
If
you
are
familiar
with
chaos
mesh,
then
you
may
probably
know
that
chaos
mesh
are
usually
used
as
a
kubernetes
plugin,
which
allow
users
to
launch
chaos,
experiments
along
with
Target
kubernetes
workloads,
but
lyric,
some
Legacy
workloads
that
just
cannot
be
run
in
container
and
cannot
be
easily
migrated
to
container
or
kubernetes
environment.
B
So
some
users
may
need
to
manage
your
hybrid
cluster,
consisting
of
notes
managing
my
by
kubernetes
running
containers,
nodes
managed
by
hypervisor
running
virtual
machine
and
the
nodes
managed
by
themselves
as
a
physical
nodes
to
better
support
this
case
we
implement
the
physical
machine
chaos
which
extend
the
chaos
mesh
framework
from
kubernetes
to
physical
nodes
or
virtual
machines
to
use
the
physical
machine
chaos.
We
need
to
first
set
up
chaos
D
on
each
virtual
on
each
physical
machine.
B
Klst
is
a
demon
process
receiving
commands
from
a
central
chaos
controller
in
a
kubernetes
and
launching
across
rounding
chaos.
Experiments
KLC
can
be
run
as
a
row
command
or
assistant
service
on
a
Target
physical
machine.
Some
common
action
of
physical
machine
chaos
are
simulating
a
process
flow
like
process,
Interruption,
simulate,
Network
flow,
like
network
drop
or
throating
or
network
bandwidth
simulating
host
default
like
host
the
shutting
down,
and
it
also
provides
some
application
or
runtime
specific
features
like
jvm
chaos
or
radius
chaos.
B
So
those
are
some
new
chaos
workloads
you
can
try
out
in
the
next.
I
would
like
to
talk
about
some
three
generic
features.
We
added
to
the
chaos
match
framework
since
2.4,
so
the
first
generic
features
I
would
like
to
discuss.
Is
the
status
check
in
workflow
so
to
get
the
best
results
when
running
chaos
workflow,
we
sometimes
launch
the
chaos
workflow
in
a
real
production
environment,
along
with
the
real
application
of
workloads.
However,
sometimes
this
application
or
workloads
are
so
critical
that
we
don't
want
it
to
break
them.
B
B
We
also
improve
the
web
workflow
web
UI
to
faciliate
the
process
of
defining
a
chaos,
workflow
I
believe
most
of
you
are
like
me,
like
you,
have
years
of
experience
of
using
kubernetes
and
quite
comfortable
of
writing.
Yaml
people
say
kubernetes,
engineer
or
yamo
engineer,
but
if
we
try
to
define
a
large
chaos
workflow,
including
multiple
stage,
then,
which
can
easily
lead
to
hundreds
lines
of
yaml
file
and
I,
believe
writing
hundreds
lines
of
yaml
file
can
easily
get
lost.
B
B
Okay,
so
first
we
need
to
decide
what
kind
of
experiments
we
wanted
to
run.
Here.
We
choose
the
serial,
we
name
it
set
a
deadline.
Then
we
will
have
a
blank
workspace
and
we
can
drag
and
drop
this
action.
We
wanted
to
run
in
the
workspace
we
Define
the
namespace
and
all
other
related
fields,
and
then
we
click
the
submit
button
and
then
zombie
the
workflow
Button,
as
we
can
see
that
the
system
will
generate
chaos,
experiment
yamo
in
the
backhand
so
feel
free
to
try
out
this
feature.
B
So
last
but
not
least,
let's
talk
about
multi-cluster
support
for
chaos
mesh,
so
we
keep
receiving
requests
from
Community
users
that
they
are
wondering
about.
If
chaos
mesh
can
support,
running
chaos,
experiments
across
multiple
kubernetes
clusters,
so
just
one
single
Global,
Entry
I,
think
this
is
a
very
reasonable
request.
Considering
that
many
users
are
building
large
distributed
systems
or
SAS
platform
that
need
to
be
deployed
across
multiple
kubernetes
clusters
located
in
different
geographical
regions,
and
sometimes
they
wanted
to
run
a
global-wide
chaos.
Experiments
like
maybe
simulate
the
apocalypse
situation
so
launching
chaos.
B
Mesh
experiments
on
each
kubernetes
cluster
separately
is
very
inconvenient,
as
you
need
to
log
in
different
clusters,
set
up
chaos
mesh
framework
on
each
of
them,
run
chaos,
mesh
workflow
and
keep
your
eyes
on
multiple
chaos,
mesh
dashboard
to
monitoring
all
the
cluster
so
to
release
the
burden
of
managing
multiple
cluster.
We
developed
the
multi-cluster
chaos,
chaos,
mesh
support
or
you
can
call
it
a
chaos
Federation.
The
basic
idea
is
we're
connecting
multiple
kubernetes
cluster
on
a
chaos
mesh
level.
B
We
will
have
a
central
controller
which
will
be
assigned
as
the
row
coordinator,
and
we
will
assign
an
agent
row
to
for
all
other
chaos
mesh
controllers.
So
when
setting
up
the
environment,
we
only
need
to
set
up
the
coordinator
once
and
then
register
each
member
cluster
to
this
coordinator
by
applying
a
yaml
file
like
this,
so
we
will
first
encode
the
member
cluster
that
could
be
configured
into
a
secret
here
like
cluster
blah
blah
could
be
config
apply.
The
yaml
file
the
coordinator
will
load.
B
B
Once
we
successful
successfully
set
up
the
environment
and
build
up
the
connection
between
a
coordinator
and
agents,
the
rest
of
the
steps
are
very
straightforward:
we
Define
the
chaos
mesh
experiment,
as
user
weighs
only
two
different.
First,
we
need
to
specify
in
the
Target
remote
cluster
and
submitting
the
experiment
to
the
coordinator.
Instead
of
the
agent
here
is
a
simple
yaml.
As
we
can
see,
it
is
just
like
a
normal
chaos.
Mesh
yaml
file
with
one
actual
field,
remote
cluster,
once
the
workflow
is
submitted
to
the
coordinator.
B
It
will
then
create
the
workflow
on
a
Target
cluster
and
the
syncing.
The
workflows
data
is
back
to
itself
using
this
users
can
control
and
observe
chaos,
mesh
experiments
across
multiple
kubernetes
clusters,
using
one
Central
controller
in
the
dashboard
okay.
So
that's
all
the
major
features
I
would
like
to
cover
in
today's
talk
so
for
the
future
plan.
There
are
several
things
on
our
roadmap.
First,
though,
we
wanted
to
improve
the
usability
of
the
system.
B
We
wanted
to
add
more
inspection,
sorry,
inspection
and
the
reports,
and
we
also
wanted
to
improve
the
debuggability
of
the
system
like
adding
more
reach,
obserability
results,
and
we
wanted
to
support
more
status,
check
types
as
I
just
mentioned,
like
Prometheus
and
day
dogs,
and
we
wanted
to
develop
a
new
framework.
That
is
cool
because
we
wanted
to
allow
user
to
extend
chaos
mesh
framework
themselves
by
defining
their
own
chaos.
Mesh
workflow
and,
of
course,
is
what
is
currently
under
development.
B
B
C
B
Okay,
okay,
so
I
think
in
most
of
case.
If
we
are
just
use
the
chaos
mesh,
then,
since
we
are
running
container
right,
so
the
things
mostly
is
just
injecting
to
The
Container,
so
it
will
not
leak
to
the
physical
machine,
but
in
other
cases,
as
I
just
mentioned,
we
have
a
physical
machine
chaos
so
that
one
you
need
to
be
careful.
So
if
you
inject
something
to
the
physical
machine,
then
that
may
be
something
yeah.
E
F
E
E
D
A
D
F
D
D
B
So
your
concern
is:
what
about
the
chaos
controller
itself
crashed
during
around
time
right,
so
chaos
controller?
Normally,
you
are
deployed
as
a
container
right.
So
if
you
it's
Crash,
you
definitely
will
want
to
restart
it
right
and
we
have
a
crd
to
record
the
status
of
each
chaos
workflow.
So
after
you
restart,
you
still
read
the
latest
status
from
the
crd
and
you
can
recover
in
the
resume
your
workflow
yeah.
F
B
F
B
G
B
Yeah,
so
your
question
is:
what
is
status
check
used
for
right?
So
sometimes,
when
you're
doing
Chaos
mesh
you,
you
may
want
to
add
stress
to
a
large
cluster
right,
so
sometimes
I
mean
as
I
just
mentioned.
We
don't
want
it
to
run
it
in
production
environment,
but
sometimes
maybe
we
wanted
to
run
it
in
production
environment,
but
we
don't
want
to
break
the
actual
application
or
workloads.
They
are
very
critical.
So
we
have
this
status
check
to
keep
checking
the
application
and
the
workloads.
A
Will
use
a
example
to
explain
that,
for
example,
if
you
are
running
the
database
distributed
database,
the
unit
text,
the
testing
is
like
keep
a
stable
workload
and
you
will
have
the
QPS
and
TPS
status.
And
then
you
do
the
following
Junction,
because
you
you,
you
know
the
system
will
be
automatically
recovered,
but
if
the
the
system
after
the
fourth
injection
they
are
not
recovered.
You.