►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Transparent Chaos Testing with Envoy , Cilium and BPF - Thomas Graf, Isovalent
Chaos testing has become a prerequisite for successful operations by testing chaotic, unexpected situations before they occur in production. Chaos testing is still done manually to a large extent. This talk will demonstrate how to use Envoy in combination with Cilium and BPF to introduce chaos such as service unavailability, latency and random rate limiting into any Kubernetes environment in a completely transparent manner. To achieve this, we will make use of Cilium and BPF to transparent adjust the networking behavior and use Go extensions for Envoy to customize chaos behavior by returning HTTP error codes for specific events, limit request throughput to a trickle and to simulate network segmentation between availability zones.
https://sched.co/MPcG
Join us for KubeCon + CloudNativeCon in San Diego November 18 - 21. Learn more at https://bit.ly/2XTN3ho. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Welcome
everybody.
The
next
talk
on
the
operation
struck
it's
by
Thomas
graph.
He
is
the
creator
of
psyllium
and
the
CTO
I
forgot,
the
name
I,
so
violent
right,
I
got
all
right.
Okay.
So
this
is
this
about
transportation,
with
Celia
nany
BPF
EVP
F
is
a
really
cool
technology,
I'm
already
familiar
with
it
on
Falco,
and
this
looks
very
promising,
so
I
hope
you
enjoy
it.
Thank
you.
B
So
hello,
my
name
is
Thomas
Groff
thanks
for
intro.
Some
of
you
may
be
wondering
why
am
I
speaking
about
chaos,
testing
I'm,
usually
talking
about
networking,
BPF
security
and
topics
like
this?
This
talk
is
kind
of
coming
out
of
our
own
chaos,
testing
that
we
have
been
doing
and
we
figured
if
we
have
tooling
available
that
we
write
ourselves.
Why
not
try
and
use
that
for
chaos
testing.
B
So
what
I'm
sharing
with
you
today
is
how
we
do
some
of
our
chaos
testing
using
the
components
that
are
listed
on
the
slide,
so
on
voice
Solomon
BPF.
Some
of
you
may
be
wondering
how
are
they
kind
of
connected
and
how
does
that
lead
to
chaos,
testing
and
what
is
actually
transparent.
Chaos
testing.
That's
what
we'll
dive
into
today.
B
So
the
stack
we'll
build
is
the
following
one:
we
will
use
what
we
call
go
extensions
for
Envoy
if
you
have
never
heard
about
that.
This
will
do
a
quick
introduction,
but
in
a
nutshell,
it's
the
ability
to
run
go
code
as
part
of
Envoy,
so
you
can
extend
and
customize
envoy
and
run
go
code
instead
of
writing
C++
code.
There's
several
other
extension
mechanisms
as
well.
B
We
are
using
go
extension
in
this
case,
we'll
be
using
envoy,
will
do
a
quick,
deep
dive
into
that
as
well,
we'll
be
using
solely
below
envoy
Asti
CNI
in
the
arcing
plugin
and
the
still
open,
Lansing
plug-in,
and
we
will
use
eb
PF
as
the
powerful
technology
in
the
kernel
that
allows
us
to
do
a
lot
of
this
transparently
and
efficient.
I
will
dive
into
all
of
those
details,
but
I
wanted
to
show
you
the
full
stack
of
what
we're
using
up
front.
B
So
these
are
the
components
that
were
using
to
do
chaos
testing
today.
So
who
has
been
doing
chaos
testing
many
hands
great?
Who
knows
what
chaos
test
again
is,
but
has
never
done
it.
Many
more
hands
I'm,
not
sure
whether
that's
good
or
bad.
So
if
you
look
up
the
definition,
it's
like
it
looks
something
like
this
and
it's
coming
a
bit
from
the
original
cast
testing
talk.
B
I
came
out
of
the
of
net
of
Netflix
chaos,
testing
monkeys
and
it's
actually,
if
you,
if
you're
very
specific,
it
means
to
entails
testing
in
production
I'm,
not
sure.
If
everybody
has,
this
understanding
has
to
be
in
production,
but
in
a
nutshell,
it
means
you're,
introducing
chaos
into
your
infrastructure
to
better
understand
failure
modes
right
now,
knowing
what
chaos
testing
is,
who
has
been
doing
unintentional
chaos,
testing
many.
C
B
That's
that's
what
I
fear,
so
what
we
specifically
focus
on
today
is
called
fault.
Injection
and
fault
injection
is
one
subset
of
chaos
testing,
and
it
basically
means
that
we
inject
faults
into
software
where
a
fault
is
not
occurring
normally,
so
the
service
is
actually
operating
regularly
and
we
want
to
inject
a
fault
to
simulate
an
outage
or
two
similar
simulate
a
service
failure,
and
so
on.
That's
what
we
that's.
What
we
focus
on
today,
fault
injection
is
really
important
as
you
can.
B
B
Typically,
if
we
simplify
it,
we
have
services
or
micro
services
talking
to
each
other,
and
this
is
a
very
simple
example
where
one
service
is
doing
a
rushed
API
call
to
another
service
and
is
making
a
put
two
slash
or
some
API
front
one,
and
because
that
service
is
operating
normally
it
returns.
200.
Okay,
right,
we
fault
injection.
B
What
we
want
to
get
to
is
to
this,
that
we
can
change
what
is
being
returned
and,
for
example,
return
a
503
application
error,
or
we
would
like
to
delay
the
response
so,
for
example,
to
simulate
the
application
or
the
service
is
taking
longer.
We
may
even
want
to
corrupt
some
of
the
payload.
There
is
many,
many
more
fault
injection
strategies.
B
This
is
a
very
simple
example,
but
probably
the
most
common
that
you
simply
want
to
simulate
that,
like,
let's
say
a
go
panic
in
your
service
code
and
you
get
the
five
over
three,
because
your
API
Handler
panicked
and
recovered
and
returned
the
503.
So
this
is
what
we'll
we'll
focus
on.
What
do
we
require
for
two
kind
of
do
fault
injection.
First
of
all,
we
need
a
man-in-the-middle.
Weiss
is
important
because
we
need
something
in
between
the
services
that
will
change
what
is
being
returned.
B
So
this
is
our
changing
things
on
the
fly,
because
we
don't
want
to
modify
the
application
to
actually
return
that
failure.
We
want
to,
on
behalf
of
the
application
change
the
return
code
we
need
to.
We
need
ability
to
actually
simulate
a
failure.
So
we
need
ability
to
translate
or
turn
a
200
okay
into
a
500
500
for
application
error
or
a
five
or
three
or
a
four,
oh
four
or
whatever,
whatever
you're
testing.
B
B
We
want
10%
of
the
requests
to
actually
fail,
but
if
it's
failing
all
the
time
it
actually
it's
probably
easier
to
handle
it
only
if
it's
only
like
1,
if
1%
probability
that
the
response
is
delayed
by
500
milliseconds,
that's
a
much
harder
problem
to
solve
than
if
all
responses
are
always
getting
delayed.
So
we
need
a
probability
aspect
or
a
probability
multiply.
B
So
yes,
inject
failure,
50
percent
of
the
times,
then
we
want
transparency
so
and
we're
getting.
This
is
maybe
a
bit
new
to
some
of
you.
We
want
to
be
able
to
do
this
completely
transparent,
so
we
have
running
applications,
unmodified
running
applications,
they're
running
a
production
or
running
in
your
staging
environment.
Let's
simulate
failures
in
that
running
environment
and
see
how
the
applications
behave,
how
your
infrastructure
behaves
all
the
parts
restoring
or
is
auto
scaling
working
and
so
on,
like
loop,
lensing
failing
over
and
so
on.
B
All
of
these
things
you
want
to
test-
and
you
want
to
do
this
transparently,
so
avoid
need
to
change
or
restore
services
and
then
the
last
piece
you
need
visibility
right.
If
you
have
a
50
percent
chance
of
doing
a
fault
injection,
you
actually
need
to
know
whether
that
fault
injection
happened
or
not,
because
the
service
could
actually
fail
for
real.
B
So
if
you
get
a
5
of
3
application
error,
you
want
to
understand
whether
was
done
by
the
Kaos
testing
framework
or
whether
the
application
really
crashed
or
really
misbehaved
or
whether
the
delay
was
was
done
by
our
testing
or
whether
it
was
done
by
the
application
itself.
So
what's
the
failure,
simulated
or
real?
B
So
the
first
and
biggest
component
we
need
is
envoy
envoy
is
a
service
and
etch
proxy
and
I.
Think
many
of
you
have
heard
about
this
either
for
a
while
or
this
cup,
this
cube
cone.
It
has
understanding
of
layer,
7
protocols,
it
can
also
do
TCP
or
elf
or
mode
only,
but
the
primary
purpose
of
envoy
is
to
act
as
a
proxy
and
to
understand
application
protocols
and
then
perform
various
services
or
functions
such
as
advanced
lopa,
lancing
path,
based
routing
or
layer,
7
or
outing.
Cannery
releases,
retries
circuit,
breaking
reg
limits.
B
All
of
these
things
envoy
as
a
proxy
can
perform.
It
can
also
provide
security
services,
authorization,
mutual
TLS
and
so
on.
It
can
provide
observers
observability.
This
is
where
a
service
match
is,
is
I,
think
very
important,
or
is
it's
very
promising?
Giving
your
visibility
in
all
of
the
API
calls
in
all
of
the
service
communication
that
is
going
on
are
giving
you
Tracy
and
giving
you
metrics
and
so
on,
and
then
envoy
is
also
extendable.
B
We're
looking
into
go
extensions
today,
but
there's
also
a
web
assembly
support
coming
up,
there's
newer,
so
there's
multiple
ways
to
actually
extend
and
customize
envoy
and
air
logic
that
you
need,
but
is
not
in
the
base
envoy
distribution.
That's
a
very
quick
intro
into
envoy.
If
you
want
to
learn
more
about
envoy,
there's
many
good
introductions
to
envoy
online
on
youtube.
B
B
We
want
to
extend
that
event
and
actually
get
to
the
next
point
and
why
this
is
important
is
because
we
want
to
eventually
get
to
the
point
that
we
can
no
longer
not
just
simulate
a
generic
service
failure,
but
that
we
can
actually
simulate
an
application-specific
service
failure.
So
if,
for
example,
your
services
are
doing
some
sort
of
building
transaction,
then
you
want
to
simulate
the
trend
that
the
billing
transaction
failed.
You
note
you'll
only
want
to
return
it
like
a
common
five.
Four
three.
B
You
actually
want
to
have
the
transaction
fail,
for
example,
so
you
want
to
also
be
able
to
do
application
specific
failures
in
orders,
and
for
this
you
need
the
proxy
that
is
parsing.
The
request
to
understand
the
actual
application.
Payload
like
what,
like
the
the
REST
API,
calls
that
you
make-
and
this
is
a
perfect
useful
case
for
go
extensions,
because
by
running
go
extensions,
we
can
use
the
standard
HTTP
parser
of
:,
which
gives
us
the
standard,
HTTP
request
and
response
types,
and
we
can
bring
in
all
the
existing
goal.
B
B
This
is
a
typical
example
why
you
use
envoy
or
what
you
can
use
envoy
for,
and
this
is
also
the
mode
that
we
use
for
our
chaos
testing
later
on,
then,
the
next
component
we
need
is
solium
and
the
reason
why
we
need
cinema
specifically
is
for
the
transparency
aspect,
as
we
will
see
later.
In
this
case,
I
will
give
you
a
quick
intro
into
what
cilium
dots
and
what
solium
stands
for.
B
First
of
all,
the
unique
aspect
about
Salaam
is
that
it
is
based
on
EB
PF
technology,
as
we
saw
an
intro,
so
it's
based
on
and
will
do
a
very
quick
intro
on
VP
f
as
well,
for
those
that
have
never
heard
about
it,
but
it's
based
on
next-generation
technology
called
EPF
in
the
kernel
which
allows
it
to
be
very
efficient
and
transparent
and
secure.
So
them
there's
a
lot
of
things.
It
does.
First
of
all
networking
it
has
its
own
psyllium
CNI
plugin,
but
you
can
also
run
it
on
top
of
other
CNET
plugins.
B
You
can
also
run
it
on
top
something
like
flannel
or
calico,
or
AWS
di
or
lyft
V,
PCC
and
I.
So
we're
compatible
with
others
in
eyes,
and
you
can
you
don't
necessarily
have
to
use
ours,
yep
Naiya
plugin
you
can.
You
can
also
run
it
on
top
of
others.
Then
we
implement
cuban
Edit
services.
We
have
a
full
implementation,
which
is
very
scalable.
We
can
scale
to
ten
thousands
of
number
of
services.
We
implement
network
policies,
cuban
ettus
network
policies.
B
We
also
extend
them,
so
we
have
additional
functionalities,
such
as
dns,
aware
policies
where
you
can
say
I'm
reaching
out
of
my
cluster
I,
want
to
allow
I
want
to
allow
all
connections
to
go,
calm
or
I
want
to
allow
all
connections
to
start
my
core
calm
and
so
on.
So
you
can,
you
can
define
the
inner
space
policies,
we
can
do
layer,
seven
of
our
policies.
We
can
do
service,
name
policies
and
so
on,
so
we
have
extended
functionality.
B
That
is
not
yet
in
the
Cuban
Atlas
network
policy,
spec
that
is
currently
in
our
custom
resource
definition
and
we're
working.
If
the
signal
org
team
to
get
that
into
the
standard
and
and
fully
supported
by
Auto
Center
plugins,
we
do
identity
based
enforcement.
I
will
not
go
into
all
of
the
details
here.
That
will
be
too
long,
but
it's
definitely
a
change
from
IP
address
or
IP
based
enforcement
into
identity
based
enforcement.
We
can
do
multi
cluster.
We
can
do
encryption,
we
have
not
native
envoy
and
is
to
integration.
This
is
some
of
it.
B
What
we
see
today
or
in
this
talk,
particularly
on
way
integration
as
part
of
this
that's
what
we
specifically
focus
on
today
is
the
transparent
envoy
injection.
So
we
can
get
an
envoy
proxy
in
between
two
services
or
in
front
of
a
service
in
a
completely
transparent
way,
without
the
service
to
actually
know
without
the
service
to
even
need
to
run
anything
inside
of
the
pot.
So
it's
completely
transparent
to
both
the
service,
the
pod
and
also
the
all
other
system
components
and
then
a
bit
more
advanced.
We
can
accelerate
service
measure,
sidecar,
use
cases.
B
If
you're
running
in
SEO
service,
mesh
or
another
service
match
we
can.
We
can
accelerate
that
communication
and
in
the
future
this
is
not
released
yet
we'll
be
able
to
do
transparent,
SSL
visibility,
that's
kind
of
the
Salomon
show
if
you
want
to
learn
more
about
solium,
just
hit
me
up
after
the
talk
I'm
happy
to
provide
more
details,
whoever
is
interested
and
then
the
last
piece
of
intro
EVP
F,
that's
kind
of
the
last
big
component
we
need.
This
is
super
exciting
we're
seeing
a
ton
of
new
e
BPF
usage
in
this
year.
B
Eb
PF
itself
is
actually
all
vp
f
itself
is
pretty
old,
but
most
recently
we
kind
of
the
need
for
what
we
call
a
virtual
machine
that
allows
to
program.
The
kernel
has
has
become
very
obvious
and
there's
been
a
ton
of
effort
that
has
gone
into
this.
So
what
is
this
EP
PF
thing?
In
a
nutshell,
it's
actually
very
simple:
it
means
that
you
can
run
a
piece
of
logic,
a
piece
of
code
on
certain
events
in
the
inside
of
Kronik.
B
I've
put
a
couple
of
examples
on
the
slide
here,
for
example,
you
can
do
a
run,
a
be
PF
program
whenever
you
send
or
receive
packets
from
a
network
device.
This
is
what
Saleem
does
on
the
networking
side.
You
can
run
a
BPF
program
whenever
a
system
call
is
being
made
like
read
or
connect.
This
is
how
SEC
comp
works
right.
This
is
how
some
of
the
container
runtime
protection
mechanisms
based
on
vpf
work.
You
can
run
it
on
block
IO
right.
B
This
is
some
of
how
the
tracing
and
metrics
work
they
actually
look
at
every
time
you,
your
system,
writes
into
a
block
idea
of
all
bio
device.
It
runs
a
small
program
and
captures
some
samples
and
you
can
run
it
for
so
called
trace
points.
For
example,
every
time
the
kernel
lost
a
TCP
retransmission
every
time
that
event
happens,
we
can
run
a
program,
so
it's
really
making
our
core
Linux
Colonel
programmable
without
requiring
to
change
it,
and
it's
done
so
in
a
completely
secure
and
efficient
manner,
which
makes
this
perfect.
B
So
we
can
load
a
program.
It
runs
out
native
execution
speed,
it
is
secure,
so
it's
basically
perfect
roll.
We
can
we
can.
We
can
extend
and
and
and
customize
our
Linux
kernel
and
the
like,
add
more
functionality
without
ever
having
to
change
it
or
made
having
to
restore
it
and
so
on.
This
is
why
so
many
people
are
super
excited
about
what
PPF
will
bring
in
the
coming
years.
B
So
what
exactly
is
transparent
chaos,
testing
and
we'll
go
through
an
example
to
make
this
clear,
I
kind
of
put
the
whole
stack
that
we
introduced
and
put
it
together.
So
simple
example:
service
a
talking-to
service,
be
there
both
on
two
different
nodes.
They
could
also
be
in
the
same.
No,
but
let's
assume
they're
on
two
different
notes:
it
doing
an
HTTP
request.
B
You
will
talk
to
each
other
and
the
first
thing
that
you
will
need
in
this
case
is
you
need
a
C
and
I
plug
in
F
chosen
Solem
in
this
case
could
be
any
other
CNF
plug-in
as
well.
The
CNF
plug-in
will
assure
that
one
part
on
one
node
can
talk
to
another
part
on
another
node,
that's
pretty
much
their
purpose
office,
dir
plug-in
that
doesn't
give
us
any
injection.
There
is
no
one
well
yet
nothing
Todd's
can
talk.
Now
we're
introducing
unroyal
and
we're
running.
B
B
It
will
accept
those
connections,
it
will
pass,
it
will
read
on
those
connections
and
it
will
then
pass
both
the
data
and
the
HTTP
requests
through
a
filter
chain
where
we
can
do
filtering,
and
after
filtering
has
happened,
it
will
go
through
the
proxy
layer
where
we
will
basically
decide
where
do.
I
have
to
send
this
all
right
and
we'll
decide
how
to
forward
this
and
so
on,
and
then
on
top
of
that
were
running
the
go
extensions.
B
So
the
Co
extensions
are
a
filter
of
Envoy
that
allow
us
to
load
go
code
up
here,
while
keeping
envoy
running
and
every
time
the
action
is
receiving
data.
That
data
will
go
to
the
NGO
extension
and
our
NGO
extension
can
do
something
with
that
data
and,
as
we'll
see
we'll
do
fault
injection
with
that.
B
The
reason
why
this
is
called
transparent
injection
is
because
this
Envoy
injection
here
they're
running
envoy
in
the
middle
will
send
the
demo
this
happens
without
the
user
or
without
anybody
actually
noticing
Salim
will
bring
up
envoy,
run
it
and
then
using
vpf
magically
or
magically
it's
it's
not.
It's
networking,
but
it
will
look
like
it's.
It's
magically
redirecting
all
traffic
subject
to
our
fault
injection
policy
to
go
through
envoy,
go
through
the
NGO
extension
and
and
perform
whatever
fault
injection
we
need.
B
So
that's
the
when
we
want
to
talk
about
transparency
on
transparent,
curves
testing.
It
means
that
you
can
have
existing
parts
running
and
we
can
then
go
and
do
fault
in
to
actually
get
envoy
into
the
middle,
so
without
doing
actual
sidecar
injection
or
without
running
something
anywhere.
We
can
simply
instruct
solium
to
do
transparent
and
transparent
chaos.
Testing.
B
Yes,
so
this
is
a
simple
example
of
kind
of
how
the
configuration
looks
like
so
they're
the
go
extension
runs
and
I
will
go
through
this
example.
Very
simple:
this
is
the
solium
custom
resource
definition
for
a
network
policy,
and
you
will
see
a
couple
of
things
here.
You
will
see
that
we
have
a
endpoint
selector.
This
is
basically
the
pod
label
selector
and
we'll
select.
What
are
the
parts
where
this
fault
injection
policy
should
apply,
and
in
this
case
it's
all
parts.
B
B
So,
let's
recap
kind
of
the
requirements
we
saw
on
how
we
solve
them.
So
women
in
middle
is
US
envoy,
quite
simply
ability
to
simulate
fault,
injection
and
probability
aspect.
That's
the
co.
Extension
chaos,
transparency,
kind
of
loading
it
on
demand
for
existing
parts,
that's
cilium,
+,
eb,
PF,
visibility,
HTTP,
headers
and
envoy.
We
will
see
the
HTTP
headers
in
the
demo,
what
I
mean
specifically
by
that
and
yes,
let's
go
and
dig
in
because
I've
been
talking
for
a
long
while
and
I
think
we
should
just
demo.
B
B
So
this
is
actually
just
many
cubes
or
very
simple
cluster.
The
only
thing
I
deployed
here
on
top
is
sodium,
so
you
see
the
cilium
power
running
as
a
daemon
set.
This
is
the
one
normally
cube,
so
it's
not
much
there,
but
that's
the
only
thing
we
deployed
on
top
of
mini
cube.
Otherwise
this
is
a
standard
mini
cube,
so
I
can
go
on
and
I
can
run
curl
to
the
death
store,
and
this
is
actually
just
as
you
will
see.
This
is
just
HTTP
min,
so
it
will
return.
B
Some
HTML
I
can,
for
example,
curl
two
headers
and
headers
that
headers
will
return.
What
are
the
headers
we
provided
in
the
request?
I
can
run
curl
with
I,
so
the
curl
will
tell
me
what
other
HTTP
headers
that
were
returned
and
the
HTTP
code.
So
so
far
no
fault
injection
this
services
returning
200,
okay,
you
can
see
this
here
is
the
phone
big
enough
or
should
I
good
awesome.
So
there's
no
fault
injection
going
on
this.
Everything
is
working
perfectly.
B
So
how
can
we
do
what
which
is
talked
about,
which
is
fault
injection,
so
we
can
look
at
I'm
actually
in
the
wrong
repositories.
Good
I
want
to
be
here
so
this
repos
public.
So
you
can
try
all
of
this
yourself
afterwards.
I
have
two
simple
example:
here:
let's
look
at
the
first
one,
so
this
first
one
is
very
similar
to
what
we
saw
on
the
slide.
It's
a
policy
that
says
rewrite
the
status
and
rewrite
it
to
504
application
error.
Let's
actually
increase
the
probability
a
bit,
so
we're
more
likely
to
hit
it.
B
B
All
right
so
I
applied
this
policy.
That's
all
I
did
let's
rerun
curl,
so
we
just
got
lucky
in
the
first
hit.
It
actually
rewrote
the
application
for
the
status
code,
two
five,
four
four
application
or
we
didn't
change
the
application
at
all.
But
all
of
a
sudden
it
looks
like
the
application
is
failing.
So
that's
running
a
couple
of
more
times
hit
it
again
hit
it
again.
Let's
see
when
when
are
we
unlucky
so
far,
it's
always
hitting
the
80%.
Now
it
didn't
hit
it
so
we're
implementing
that
probability
aspect.
B
B
All
right-
and
let's
run
it
again
in
this
case,
it
should
actually
pretty
much
always
a
return
200
and
it
does
all
right.
What
else
can
we
do
so
we
can.
We
can
delete
the
policy
this
guy
I'm,
just
deleting
all
of
them,
because
I
only
have
one
loaded.
I
can
run
it
again.
In
this
case
it
will
always
return
200,
okay,
again
so
I
can
in
check
this.
Take
it
out
again,
it's
completely
transparent
and
your
pods
don't
even
know
that
this
is
happening
it's
completed
on
from
the
outside.
B
B
It's
instant
again,
so
we
can
do
delay,
we
can
do
rewrite
status,
and
so
we
can
do
many
more
things.
So,
quick
demo,
like
how
simple
this
is
from
a
usability
perspective.
What's
even
more
important
in
this
case
is
that
this
is
all
gold
code.
So
if
you
need
additional
functionality,
it's
absolutely
trivial
to
extend
this
and
write
whatever
additional
logic.
You
need,
because
it's
all
going
HTTP
parsing
code,
like
almost
everybody.
It's
not
like
DPF
low-level
code,
is
not
C++
absolutely
trivial.
To
extend
this.
B
If
you
want
to
get
started
and
basically
repeat
what
I
just
did
very
simple
as
well
I
put
all
of
the
examples
into
a
git
repo,
so
you
can
get
clone
that
you
can
create
a
cube
notice
cluster.
You
can
use
your
existing
one
in
this
case,
I'm,
just
using
mini
cube,
very
simple:
you
install
solely
on
one
command
and
solemn
is
installed,
that's
just
the
daemon
set.
This
is
part
of
the
chaos
testing
examples
repo,
but
you
can
also
follow
the
standard
same
installation
path.
You
deploy
your
app
in
this
case.
B
It's
a
def
store
and
the
Falcon,
and
then
you
apply
the
fault
detection
policy.
That's
all
you
need
to
do
so
very
simple.
You
can
try
this
out
using
your
existing
apps
or
you
can
use
this
for
development
purposes
on
mini
cube.
You
can
run
this
in
your
staging
clusters.
You
can
run
this
in
your
product
testers.
If
you
want
and
so
on,
to
go
into
some
more
examples.
B
You
can
combine
the
the
probabilities,
so
you
can,
for
example,
have
a
policy
that
says
with
a
probability
of
80%
delay
by
50
milliseconds
and
then
with
an
additional
probability
of
20%
delay
for
one
second,
so
it's
not
just
one
rule
or
like
you
can
actually
combine
them,
which
is
kind
of
a
bit
of
a
more
realistic
example,
because
you
can
create
a
distribution
that
you
are
more
likely
to
see.
You
can
also
rewrite
or
match
on
path
method
on
the
status
code,
so
you
can
basically
say
only
rewrite
the
HTTP
response.
B
If
the
request
was
a
get
to
/foo,
so
you
can
become
very
specific
on
which
responses
you
want
to
want
to
where
you
want
to
inject
false.
So
you
can
not
only
just
match
on
the
poor
labels
you're
matching
on
or
the
selected
services.
You
can
also
apply
filters
on.
What
are
the
requests
that
you
want
to
inject
failures
into
now?
Obviously,
you
can
load
as
many
of
these
policies
as
you
want.
B
No
I've
only
loaded
one,
you
can
load
as
many
as
you
want,
of
course,
with
that
I'm
happy
to
take
questions
before
we
get
to
that.
All
of
this
is
open
source.
The
limits
open
source,
the
link
to
the
gate
up.
Is
there
the
link
to
the
examples
with
all
the
chaos
testing.
Scripts
and
policies
are
up
there.
We
have
a
slack
channel.
B
D
D
B
Service
there's
a
gas
law
container,
so
the
delay
is
caused
at
the
packet.
The
request
is
currently
in
envoy
and
the
response
has
been
parsed,
but
it's
basically
held
in
there.
So
the
Envoy
is
maintaining
a
connection
to
both
services
and
the
response
is
in
the
exactly
the
same
as
if
you
do
envoy
rate-limiting,
it's
holding
the
request.
It's
not
actually
holding
packets
like
Network
packets,
it's
it's
parsing
the
full
response
holding
that
for
like
one
second
and
then
sending
it
back.
C
B
You
what
the
demo
I
did
today
this
I
brought
this
chaos
testing,
clogging
like
one
day
and
I'm
using
:
net
HTTP,
so
it
will
support
any
version
of
HTTP,
but
the
co
extension
plugging
is
on
data
level
and
the
plug-in
I'm
using
is
done
in
a
way
that
you
can
now
hook
in
any
any
parcel.
You
have
that's
using
the
goal
line,
io
reader
interface.
You
can
just
plug
it
in
so
you
can
parse
whatever
protocol.
Even
if
your
proprietary
protocols
can
all
plug
it
in
the
demo,
I
did
was
HTTP
specific,
so.
E
B
So
the
convoy
is
actually
not
running
at
all
until
we
injected
the
first
network
policy.
So
when
we
did
the
first
apply
if
the
policy
envoy
will
start
it
and
at
that
point
the
filter
like
let's
go
back
to
the
policy,
so
only
the
connections
that
are
matching
this
selector
here,
so
all
connections.
So
that's
only
those
will
go
through
envoy
and
then,
when
you
unload
this
policy,
no
connections
will
go
through
envoy
again.
So
only
then
neither
does
it
go
through
envoy
and
then
I
think.
B
F
Maybe
the
question
is
going
in
the
same
direction
because
you
are
using
the
cilium
network
policy,
which
is
special
to
psyllium
and
provides
the
sto
in
to
add
even
more
integration.
What
would
happen
if
I
would
have
a
network
policy
and
I
would
add
an
additional
psyllium
Network
policy
for
the
chaos
testing?
Yes,.
B
It
will
respect
both
so
all
cubanelles,
never
polished
in
all
network
policy
or
so-called
whitelist
is
or
whitelisting
so
there's
no
denial
reject
in
convenors
policy,
so
you
cannot
even
have
a
conflict,
but
the
both
will
be
respected.
You
may
not.
The
only
reason
why
we
need
some
network
policy
is
because
there
is
this
part
here.
This
is
a
this
is
just
a
gold
egg
map
and
allows
us
to
parse
arbitrary
key
value
pairs
into
the
go
extension,
so
you
can
basically
pass
whatever
configuration
you
need
for
your
call
extension.
B
G
B
Can
run
this
with
Sto?
All
you
have
to
do
is
this.
The
co
extension
is
an
envoy
filter
envoy
or
is
tu,
has
its
own?
We're
kind
of
not
for
version
of
envoy,
choose
to
add
our
patch
to
add
our
filter,
because
the
sto
proxy
does
not
have
the
the
go
extension
filter
in,
but
as
soon
as
you
add
that
and
sto
sidecar
proxy
becomes
capable
of
running
Co
extensions.
This
will
just
work
in
the
pilot
managed
sidecar
as
well.
This
is
the
the
go
extension
patch
on
top
is
very
small.
C
B
Point
the
overhead
is
coming
because
Co
is
doing
memory
a
bit
different
and
it
has
it
there's
every
time
you
switch
contexts,
it's
not
really
a
context
which
it's
not
kernel
user
space,
but
there's
a
memory
context
switch
between
C++
and
Co.
That
will
that's
this
10%,
but
that's
not
per
packet
or
anything
like
this
is
every
time
it's
usually
one
call
per
request,
and
then
you
get
the
data
for
the
entire
HTTP
request
unless
it's
really
really
really
large.
What
we
measured
in
the
worst
case
is
about
ten
percent.