►
From YouTube: Komrade: an Open-Source Security Chaos Engineering (SCE) Tool for... Aaron Rinehart & Matas Kulkovas
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Komrade: an Open-Source Security Chaos Engineering (SCE) Tool for K8s - Aaron Rinehart, Verica.io & Matas Kulkovas, Cast.ai
Security Chaos Engineering (SCE) is an emerging discipline that serves as a foundation for proactively discovering system weaknesses before they become an opportunity for a malicious actor. The goal of SCE experiments is to move security toward continuous recalibration and increased confidence by deriving a more realistic understanding of how well security practices perform under expected conditions. This new technique of instrumentation proactively injects security turbulent conditions or faults into systems to determine the conditions by which our security will fail so that we can fix it before it causes customer pain. During this session, the speakers will dive into SCE as a discipline as well as showcase a demo of 'komrade', the 1st Open-Source Tool for running SCE experiments on Kubernetes.
A
Good
afternoon
and
welcome
to
the
last
day
here
at
kubecon,
we've
got
some
exciting,
I'm
aaron
reinhart.
This
is
matas.
Now
we
have
an
exciting
talk
to
to
bring
to
you
and
today
we're
gonna
talk
about
a
new
open
source
tool.
It's
an
exciting
announcement.
We
actually
chose
kubecon
in
particular
to
announce
the
tool
the
tool
was
written
some
time
ago,
but
we
we
chose
kubecon
to
actually
debut
and
announce
the
first
ever
open
source
tool
for
security,
kiosk
engineering
and
we're
gonna.
B
A
So,
like
I
said,
I'm
aaron
reinhart,
I'm
the
cton
co-founder
at
baraca.io,
I'm
also
known
as
being
the
sort
of
the
creator,
the
person
behind
security
chaos
engineering.
I
wrote
the
first
open
source
tool
in
the
space
about
five
years
ago.
I'm
also
the
co-author,
with
kelly
shortridge
on
the
o'reilly
books
on
the
topic.
If
anyone's
interested
in
getting
a
copy
of
the
rally
book,
please
see
me
after
the
talk
and
I'll
be
happy
to
get
you
a
copy
of
it.
B
A
Like
I
said,
there
are
several
books
on
some
I've
been
talking
about
several
topics
today,
but
in
particular
we're
going
to
I'm
going
to
I'm
going
to
go
over
what
chaos
engineering
is
just
give
you
a
basis
of
what
that
is
also.
But
there
are
several.
There
are
several
o'reilly
books
that
you
can
download
and
dive
deeper
in
chaos,
engineering
and
security,
kiosk
engineering
mata.
So
this
tool
curvis,
will
be
documented
in
the
the
animal
book
that
comes
out
later
this
year.
A
So
so,
so
what
are
we
going
to
talk
about
that?
So
today,
we're
going
to
talk
about
the
nature
of
complexity
and
modern
software,
we're
going
to
talk
about
chaos,
engineering,
we're
going
to
talk
about
its
application
to
security,
we're
going
to
talk
about
use
cases
and
we're
going
to
talk
about
we're
gonna,
give
a
demo
of
of
curvis
as
well
as
how
to
get
started
and
how
to
start
applying
these
concepts
when
you
get
back
to
back
home.
A
A
So
the
crux
of
what
we're
addressing
here
is
that
in
the
world
of
cyber
security,
even
modern
engineering,
when
it
comes
to
outages,
breaches
and
incidents,
you
know
no
matter
how
how
sophisticated
we
seem
to
be
getting
where
we
don't
seem
to
be
getting
a
lot
better
at
preventing
them
or
are
or
are.
It
seems
that
outages
and
instances
are
happening
more
and
more
often,
I'm
gonna
talk
about
why
I
think
that
is-
and
why
is
this-
I
mean:
what
are
we
doing
something
wrong?
A
Well,
not
necessarily
we're
doing
lots
of
things
right.
Well,
the
problem
is,
is
that
our
systems
have
fundamentally
evolved
beyond
our
human
ability
to
mentally
model
their
behavior.
It's
it's
hard
for
humans
to
simplify
and
abstract
thou
hundreds
of
thousands
of
things,
and
it's
very
difficult
in
in
in
today's
modern
computing
environment.
A
A
So
where
does
all
this
complexity
come
from?
So
you
see
you
see
things
on
here
that
are
things
we're
trying
to
adopt?
What
you're,
trying
to
do,
devops,
ci
cd,
auto
canary
circuit
breakers
service,
meshes
right.
These
are
all
these
are
all
great
things
right.
Well,
so
the
nation
of
software
is
software,
never
decreases
the
complexity.
You
can't
actually
decrease
it.
If
you
have
a
complex
piece
of
software,
you
want
to
make
it
simple.
A
So,
furthermore,
so
on
the
right
here,
you
see
the
new
osi
stack.
It's
software
software
software
right
software
is
officially
taking
over
news
flash
if
you're
at
a
software
conference-
and
you
know
this
is
the
this-
this
has
brought
lots
of
advantages,
but
also
it's
brought
a
new
era
of
complexity
that
we
have
to
manage
as
operators
so
like.
I
said
software
only
ever
increases
in
complexity.
It
doesn't
decrease
complexity.
A
So
there's
a
paper
written
in
1985
called
no
silver
bullet.
It's
it's
about
the
nature
of
where
complexity
comes
from.
In
software.
There
are
two
different
schools
of
thought:
there's
accidental
complexity
and
essential
complexity,
so
essential
complexity
comes
from
things
like
conway's
law,
where
organizations
are
destined
to
design
computer
systems
that
reflect
the
way
they
operate
as
a
business.
What
I
mean,
what
that
means
is,
is
that
you
can't
actually
change
the
complexity
without
changing
the
business.
A
You
inherited
that
and
it
found
its
way
into
software
right,
so
you
have
to
change
the
business
to
actually
change
that
complexity.
Accidental
complexity
comes
from
the
ways
in
which
we
build
software,
and
but
really
you
know,
you
can't
actually
remove
complexity
and
software.
Like
I
said
you
know
it's,
it's
not
necessarily
about
trying
to
make
this
more
simple.
It's
trying
to
provide
further
context
and
navigate
the
complexity.
A
So,
like
I
said
so,
chaos
engineering
we'll
talk
about
is
not
not
breaking
things
in
production.
By
the
way,
don't
do
that
you'll
get
fired
right.
This
is
don't
tell
your
boss,
you
want
to
do
that.
It's
about
we're,
trying
to
proactively
fix
things
and
we're
in
part
of
the
ability
to
do
that
is
navigating
the
inherent
complexity.
A
Okay.
So
when
we
design
systems
as
engineers
as
human
beings,
we
we
we
forget
that
how
messy
systems
engineering
really
is
because
I
used
to
be
the
chief
security
architect
at
united
health
group,
the
largest
healthcare
company
in
the
world-
and
I
have
a
data
architect
and
a
solutions
architect
coming
to
me
for
the
same
system
and
they
have
two
different
diagrams
right.
But
that's
the
data
architect
had
a
different
mental
model
of
what
they
believe
the
system
was,
so
the
solutions
architect
right.
Well,
neither
one
of
them
are
wrong.
A
They're,
both
right
but
they're,
more
right
when
you
overlay
them
together.
Does
that
make
sense?
So
it's
it's
we
so
in
the
beginning
of
designing
a
system
right.
So
we
the
plan,
is
so
simple,
so
clear
we
have.
You
know:
we've
got
our
resources,
we
kind
of
know
the
timeline.
We've
got
our
repo,
you
know,
we've
got
our
docker
images,
we've
got
staging
prod,
you
know,
and
we
always
we
have-
that
nice
3d
diagram
of
the
system
right
well.
In
reality,
the
system
almost
never
ever
looks
like
this
right.
A
We
have
to
refactor
or
we
there's
an
expired
cert,
because
we
forgot
to
change
it
or
update
it
right
or
you
know
we
a
sydney
decker
when
the
world's
experts
in
safety
engineering
in
the
world
of
resilience,
engineering,
which
is
where
all
this
stuff
comes
from,
is
likes
to
call
this.
The
slow
drift
into
failure
right
is
our
system
slowly
evolves
to
a
state
where
we
no
longer
recognize
it.
A
So,
in
the
end,
the
point
of
behind
all
this
is
our
systems
have
fundamentally
become
more
complex
and
messy
than
we
remember
them
being.
So
what
does
all
this
stuff
have
to
do
with
security
right?
I'm
getting
I'm
building,
I'm
building
we're
building
suspense
we're
getting
there.
So
cyber
security
is
a
context,
dependent
discipline.
Okay.
A
So
so
I
was
a
software
engineer
for
most
of
my
career
before
I
got
into
cyber
security
and
as
an
engineer,
you
know,
you
need
the
flexibility
convenience
to
change,
something
you're,
not
sure
one
you're,
not
sure
if
you
can
actually
do
what
you're
being
asked
to
do
so,
you're,
it's
a
process
of
trial
and
error,
you're
trying
to
figure
it
out.
You
don't
know
what
permissions
you
need
or
what
ports
you
need
open.
A
You
don't
know
those
things
right,
so
you
kind
of
need
the
flexibility
to
make
changes
and
because
the
software
engineer,
your
job
is
to
deliver
value
to
customer
via
product
via
software
right
and
but
your
content.
Software
engineers
are
constantly
changing
the
environment.
They're
constantly
trying
to
achieve
business
value
right
well,
security
is
context
dependent.
You
must
know
what
you're
trying
to
secure
in
order
to
know
what
needs
to
be
secured
about
it
right
you
have
to.
I
mean
you
have
to
know
that
first
right.
A
Well,
so
what
that
the
problem
with
that
is,
is
that
we
go
forth.
We
often
build
based
on
a
context
at
a
state
in
time
we
build
that
like
we
build,
we
deploy
a
runtime
security
tool
or
configuration
rules.
We
do
that,
based
on
that
context,
what
happened?
What
happens
is
that
engineers
never
stopped
changing
the
system,
they're
still
moving
towards
business
value.
At
that
point
in
time
we
built
the
security
we
built
the
security
based
on
the
context
we
understood,
but
the
drift
is
occurring
if
that
makes
sense.
A
So
with
chaos
engineer
for
security,
we're
introducing
the
conditions
by
which
we
expect
the
security
to
function.
So
we
know
proactively
that
oh,
my
gosh,
my
security
doesn't
actually
work
the
way
I
thought
it
did,
or
it
no
longer
is
effective
at
doing
what
I
designed
it
to
do.
We
learned
that,
before
an
adversary
can
take
advantage
of
a
fundamental
issue,
a
bit
of
visibility,
so
in
terms
of
chaos
engineering,
so
I
like
to
like
kind
of
break
down
in
the
term
world
of
instrumentation
testing.
You
know
what
I
love
about.
Chaos.
A
Engineering
in
particular,
is
that
you
know
it's
fundamental
science
and
engineering.
All
science
and
engineering
revolves
around
testing
instrument
instruments.
Instrumentation
data
measurement
and
that's
what
we're
trying
to
achieve
here,
but
in
terms
of
what
we
do
in
software,
so
there's
a
little
bit
of
a
difference
between
testing
and
experimentation.
A
Experimentation
is
really
where
the
chaos
engineering
fits
in,
so
testing
real
testing.
We're
trying
to
verify
or
validate
something
we
already
know
to
be
true
are
false.
It's
sort
of
a
binary
thing
in
the
world
of
security.
So
it's
what
we
we
know
we're
looking
for
before
we
go
looking
for
it,
so
the
world
of
security.
A
That's
things
like
see
a
cbe,
an
attack
pattern
and
a
signature,
whereas
experimentation
in
chaos,
engineering
we're
trying
to
derive
new
information
that
we
previously
did
not
know
the
more
the
unknown
space
of
the
system,
and
we
do
that
by
introducing
the
conditions
we
expect
the
system
to
successfully
operate
under.
We
never
introduce
chaos
engineering
to
introduce
chaos.
Chaos,
energy
is
not
about
chaos,
creating
chaos,
it's
actually
about
creating
order,
and
I'm
going
to
expand
to
that
in
a
minute,
so
chaos
engineering.
A
So
the
original
netflix
definition
is
the
discipline
of
experimentation
on
distributed
systems
in
order
to
build
confidence
in
your
ability
to
withstand
tribal
conditions.
So
another
way
of
saying
this
is
it's
the
practice
of
proactively
introducing
the
failure
into
a
system
to
try
to
determine
the
conditions
by
which
the
system
will
fail
before
it
fails
we're
this
is
a
proactive
discipline,
we're
trying
to
understand
and
verify
and
validate
and
build
confidence
that
the
system
can
actually
handle
140
milliseconds
of
light
circuit.
A
I
figured
so
so
that
chaos
engineering
was
sort
of
the
genesis
of
of
cass
monkey
was
sort
of
the
genesis
of
chaos
engineering
at
netflix,
and
so
what
I
want
to
do
is
kind
of
maybe
tell
some
things
you
maybe
not
know,
do
not
know
about
cast
monkey,
so
chaos
monkey
was
sort
of
came
about
about
2008,
2009
and
netflix,
so
netflix
was
moving
during
their
cloud
transformation.
Remember
netflix,
wasn't
always
in
the
cloud
during
their
cloud
transformation
aws.
What
was
happening
was.
A
Is
amis
amazon,
machine
images
just
were
poof
just
disappearing?
There's
no
explanation
why
it
was.
It
was
a
feature
right.
Well,
netflix
was
trying
to
build
a
very
large
streaming
streaming
service.
You
know,
customers
are
not
going
to
like
it.
If
the
service
goes
down
when
they're
trying
to
consume
it.
So
what
they
did
was
is
they
went
forth
and
they
built
their
system.
They
said:
okay,
we're
gonna,
build
our
systems
to
be
resilient
to
this
problem,
so
when
they
did
that
they
needed
a
way
to
actually
verify
that
that
logic.
A
So
so,
if
you
think
about
things
like
retries
circuit,
breakers,
security,
detective
and
preventative
kind
of
things
or
failovers
that
we
design
that
logic
at
a
point
in
time
to
operate
under
certain
conditions,
so
so
when
x
happens,
y
will
be
triggered
right.
What
happens
is
remember.
Engineers
are
constantly
changing
the
system
and
we
almost
never
actually
exercise
that
code
path
until
the
problem
occurs.
While
engineers
have
been
changing
system
a
lot
that
says
that
may
not
actually
still
do
what
you
need
to
do
so
with
chaos.
A
A
I
said
it
was
born
during
a
netflix's
cloud
transformation,
but
it
it
what's
what
it
does
is
so
pseudo-randomly
during
business
hours
it
will
terminate
in
ami,
and
it
does
that
so
engineers
can,
like,
I
said,
validate,
that
their
their
logic
actually
functions
so
casey
rosenthal
by
co-founder,
and
also
the
creator
of
chaos.
Engineering
at
netflix
likes
to
say
that
you
know
be
very
careful
about
breaking
things
on
purpose:
we're
not
trying
to
do
that.
Do
not
leave
here
today
and
with
the
concept
of
breaking
things.
A
That
is
not
what
we're
doing
right,
we're
proactively
fixing
them
it's
about.
Like
I
said
it's
about
continuously
verifying
the
system
and
creating
order
and
yeah,
and
I'm
pretty
sure,
casey
loves
to
tell
people
this.
I'm
pretty
sure
that
whenever
I'm
breaking
things,
I
probably
won't
have
a
job
for
very
long.
A
So
who's
doing
chaos,
engineering,
so
many
companies.
I
can't
even
I
can't
even
count
anymore-
I
mean
everyone,
I
mean
people
are
at
various
stages
of
maturity.
People
are
adopting
the
concepts
and
implementing
tools
and
practices,
but
this
is
becoming
more
and
more
of
a
standard
practice
and
in
the
future
you
really
have
no
choice.
We
have
because
our
systems
have
become
so
large,
they're
changing
so
fast.
We
need
a
way
to
instrument
the
system
post
deployment.
This
chaos
engineering
is
not
a
build
process,
testing
or
instrumentation.
A
So
security,
kiosk
engineering
news
flash
it's
not
a
lot
different,
it's
the
same
thing
as
chaos
engineering.
So
the
point
is
we're
trying
to
verify
that
we
that
our
security
actually
works
so
hope.
As
an
engineer,
I
don't
believe
in
two
concepts
as
an
injury,
I
don't
believe
in
hope
or
luck,
right
and
hope
has
never
been
an
effective
strategy.
I
mean
it
works
in
star
wars,
but
it
doesn't
really
work
in
engineering.
A
We
believe
in
data
measurement
and
instrumentation,
and
so
what
we're
trying
to
do
with
the
like,
I
said
before,
is
the
with
the
security
kiosk
engineering,
so
we're
trying
to
proactively
understand
where
our
gaps
are
in
our
security
before
an
adversary
can
take
advantage
of
the
fact
we
just
don't
see
it
so
some
use
cases
in
the
o'reilly
books.
There's
a
it's
much
deeper.
A
I
would
try
to
give
them
the
best
guidance
possible,
but
I
was
never
sure
if
they
understood
me
correctly
or
if,
if
the,
if
they
actually
configured
it
correctly,
they
placed
it
correctly
because
engineering
is
a
very
opinionated
and
specific
discipline
and
what
I
needed
was
a
way
to
ask
the
computer
questions
and
that's
the
so
that's
that's
where
I
started
so.
Chaos.
Engineering
for
security
and
ks
engineer
in
general
is
great
for
instant
response
for
improv,
actually
identifying.
A
You
know,
gaps
in
your
observability
and
as
well
as
every
chaos
experiment,
whether
security
or
it's
availability
based.
They
all
have
compliance
value.
If
you
think
about
it
right
so
you're,
basically
proving
whether
the
technology
worked.
The
way
you
said
it
did
so
make
sure
you
keep
your
your
output
of
your
experiments
in
you
know,
because
that
could
be
used
for
as
an
audible
artifact.
A
How
does
it
work
so
we're
gonna
get
to
that?
That's
he's
gonna
talk
about,
but
I
started
a
united
health
group
by
writing
a
tool.
You
could
actually
go
to
the
github
repo.
Still
it's
somewhat
deprecated,
I'm
not
there
anymore.
I've
been
there
for
a
long
time.
They
wrote
a
different
tool
that
they
use,
but
to
this,
but
you
can
still
go
to
repo
kind
of
see
how
experiments
are
written.
A
We
needed
an
example
that
a
security
engineer
that
a
network
engineer
a
software
engineer
the
executive
that
people
could
generally
understand.
Well,
we've
been
we've
been
solving
for
misconfigured
poor
changes
for
like
35
years
right,
but
some
reason
it
still
happens
right.
It's
not
because
it's
it's
not
because
anybody
intentionally
did
something
wrong
or
malicious
mistakes
happen.
Some
in
if
you're,
not
a
network
engineer
flow
network
flow
is
not
a
very
intuitive
thing
actually,
and
it
happens,
so
people
make
mistakes
right
well.
A
So,
at
when
I
was
at
united
states
united
health
group,
this
was
a
problem
we
thought
we
had.
100
solved
is
that
this
is
something
we
we.
We
felt
so
confident
that
if
this
occurred,
we
would
detect
it.
We
prevented
we'd,
stop
it
and
we
had
you
know,
there's
and
we
had
it
covered
well,
so
so
we'll
be
so
this
so
the
example
main
example
in
the
repo
for
chaos.
Singer
is
this
tool
called
this.
A
This
misconfigured
port
injection,
and
so
we
started
doing
this
on
all
of
our
aws
instances
at
united
health
group.
It
was
happening,
was
about
60
of
the
time
the
firewall
actually
caught
it,
but
there's
40
times
it
didn't.
That
was
not
something
we
expected.
We
were
very
new
to
the
cloud
due
to
aws
at
the
time,
and
you
know,
but
what
we
found
out
was
remember.
This
was
proactive.
There
was
no
outage.
There
was
no
incident,
there
was
no
breach.
We
proactively
realized.
Oh
my
gosh.
A
This
doesn't
really
work,
so
we
you
know.
So
what
we
found
out
was
there's
a
drift
issue
between
our
commercial
software
and
our
non-commercial
software
firewall
instances
so
no
problem,
we
proactively
found
it
and
we
fixed
it
before
there
was
pain.
So
that
was
the
first
thing
we
learned.
The
second
thing
we
learned
was:
is
our
cloud
native
configuration
management
tool
caught
and
blocked
to
change?
100
of
the
time
something
we
didn't
even
think
about
was
actually
better
more
effective
than
what
we
expected.
That
was
the
second
thing.
A
A
If
s
nat
is
part
of
it,
you
could
be
maybe
an
hour,
and
in
that
case
you
know
when
I
was
at
without
a
health
group
about
one
during
the
busiest
time
of
year,
one
minute
of
downtime
was
over
a
million
dollars
for
one
minute,
all
that,
so
all
that
pain
never
has
to
occur.
If
we
proactively
verify
the
system
is
doing
what
we
expect.
B
So
the
curve
is
mentioned
previously:
it's
an
open
source
tool,
it's
a
security
cast
engineering
tool
for
kubernetes,
and
what
am
I
specifically
talking
about
so
the
low
hanging
fruit
targets
are
cis
benchmark
benchmarks
for
kubernetes,
for
example,
api
server
configurations,
master
and
worker
nodes,
there's
also
cubelet
configuration
and
parameters
which
are
relevant
for
security
networking.
It's
always
interesting.
B
A
B
Only
relevant
for
those
who
use
self-managed
kubernetes
but,
for
example,
there's
this
small
parameter
called
rbac.
Do
you
know
what
would
happen
to
your
cluster
if
you
removed
it?
You're
welcome
drive
my
tool.
Also,
you
can
do
the
same
with
each
cubelet
in
your
cluster.
Fortunately,
there
was
this
dynamic
cubelet
configuration
feature.
Apparently
I
was
the
only
user
so
with
one
two:
four:
they
deprecated
and
removed
it,
but
we're
still
going
to
be
demoing
it.
B
Maybe
it's
useful
to
someone
and
how
does
the
experiment
look
like,
so
you
choose
an
experiment,
something
you
want
to
test
something
you're
sure
that
is
not
something
you're
secure
about.
When
you
choose
the
experiment,
you
what
the
pod
does
is
backups
your
previous
configuration
for
the
parameter
or
configuration
you're
testing.
If
it's
applicable.
B
It
then
applies
its
payload.
That
changes
the
existing
configuration.
Then
it
tries
to
validate,
did
the
did
it
apply
if
it
did
that
enters
ready,
state
and
then
you're
you're
free
to
verify.
Am
I
seeing
what's
happening
when
you
decide
you've
had
enough,
you
restored
the
previously
backed
up
configuration
and
you
you
go
on
as
you
did
before.
B
B
So
I
previously
ran
qbench
and
if
you
notice
the
it
passed
like
everything's
fine,
then
we
apply
one
of
the
experiments.
Well,
we
apply
the
experiment
part
with
the
parameter
four
one:
five
I
wanna
misconfigure.
I
wanna
test
that
so
you
can
see
that
the
pod
enters
ready
state.
It's
called
the
yeah,
the
chaos
pod
and
it
reports
that
I'm
currently
misconfiguring
cis415
run
queue
bench
again:
q
bench
backwards,
query
and
in
the
logs
we
can
check
again
and
four
run.
Five
will
be
invalidated,
so
you
know
you.
B
Can
you
see
that
it's
actually
doing
its
job?
It's
it's
not
passing
for
everything.
Then
we
can
terminate
the
experiment
pod
and
it
will
restore
the
previously
backed
up
file
permissions
and
it
will
pass
again
so
the
nice
thing
about
that
is,
you
know.
Well,
of
course,
there's
always
risks.
I
mean
something.
You
know
electricity
going
down
during
the
middle
of
the
experiment,
but
yeah
you
don't
have
to
back
it
up
yourself
and
that's
a
nice
thing.
B
You
can
see
the
experiment
we're
back
to
the
initial
state
regarding
the
parameter
four
one:
five,
the
next
next
experiment,
it's
relevant
for
cube,
api
server
and
we're
good.
It's
again,
part
of
cis
benchmark
configurations.
It's
regarding
the
anonymous
authentication
parameter,
but
yeah
you
can
work
well.
The
tool
curve
is,
can
work
with
any
parameter,
one
setting
updating
removing
existing
parameters.
B
My
personal
favorite
is
authorization
modes
in
this
case
we're
just
using
anonymous
authentication
and
we
create
a
pod
that
misconfigures
the
api
server.
It
takes
a
while
because
the
api
server
has
to
restart,
and
as
mentioned
it's
only
relevant
for
self-managed
kubernetes,
but
yeah
you
can
see
in
the
logs
it's
supplying
one
to
one.
B
Then
we
it
validates
itself
by
checking
the
process
process
running
on
the
master
node
and
seeing
whether
you
know
the
misconfigured
parameters
are
what
we're
experimenting
with
and
yeah.
That's
how
it
can
confirm,
but
it's
not
just
doing
some
random
thing.
You
can
also
see
in
the
manifest
that
an
arm
synthetication
has
been
added.
B
B
So
as
we
can
see,
I
chose
a
parameter
called
event
record
qps.
It's
a
easily
quantifiable
value.
It's
easy
to
see
like
the
changes
in
here
and
we're
acquiring
the
cubelet
configuration
for
for
its
the
offers
parameters.
We
can
see
that
I
want
to
record
qps
it's
set
to
five
right
now,
we're
just
choosing
this
parameter
and
applying
a
cube
limits,
configuration
experiment
with
it
with
it.
B
When
it
can
verify
that
it's
been
applied,
it
starts
reporting
that
event
record
qps
parent
is
running.
We
can
verify
manually
that
it
has
been
changed
now.
What
you
would
do
is
you
would
if
this
is
something
that
you
want
to
be
aware
of
continuously,
you
would
go
check
your
your
observability
stack.
Whoever
knows
someone
is
changing
the
configuration.
Can
I
see
that
it's?
Of
course
I
don't
record
qps.
Maybe
it's
not
something
a
hacker
would
do,
but
as
a
poc,
it
is
an
interesting
test.
B
B
What
we
do
is
core
dns
spoofing-
or
we
just
add
the
new
domain
name
to
the
core
file
in
the
coordinates
config
map
and
in
one
of
the
pods
in
the
cluster
I'm
trying
to
crawl
for
google,
it's
the
http
version,
so
we
get
through
the
result
that
we
want.
We
then
apply
a
corianis
configuration
experiment,
I've
configured
it
to
so
that
all
queries
to
google
go
to
yahoo.com
and
once
once
the
pod
enters
the
ready
state.
It
takes
a
while
corey
and
his
it's.
B
B
And,
as
you
can
see,
the
experiment
is
applied
and
once
we
go
into
the
the
previously
pod,
we
receive
a
different
response,
so
it
might
be
interesting
to
someone
who's
like
who
want
to
verify
that
any
changes
in
the
core
file
they
there
will
be
a
variable.
I
mean
it
doesn't
take
much
for
something
bad
to
happen.
When
we
delete
again
like
like
with
the
others,
we
revert
to
initial
state
takes
a
while.
That's
that's!
Why
we're
not
doing
a
live
demo
and
yeah?
We
can
confirm
that
we
can
yeah
return
to
previous
state.
B
The
last
experiment
that
I'm
I'm
going
to
demo.
It's
yeah,
it's
it's
relevant
to
dns
poofing.
Also
thanks
to
nerd
choco
from
cyberark
he's
been.
How
did
I
turn
the
audio
for
this
okay
yeah?
It's
like
I've
used
the
poc
that
he
wrote
on
on
his
blog,
it's
about
dns
proofing
and
networks,
with
l2
bridges,
and
basically,
what
we
do
is
it's
arp
spoofing,
plus
dns
poofing
on
pods,
so
we
have
a
regular
pod
who
yeah?
Who
is
communicating
some
service?
B
B
B
In
this
case
we
use
I'm
using
a
flannel
cni,
so
it's
probably
not
relevant
for
everyone,
but
anyone
who
uses
that
and
wants
to
experiment
feel
free
once
we
delete
the
pod.
As
always.
In
this
case
we
didn't
what
we
did
was
we
backed
up
the
arp
table
and
it
takes
awhile
until
it
gets
restored.
So
it's
a
different
kind
of
backup,
but
once
we
get
the
once,
we
finish
termination,
we
return
to
the
initial
state
and
we
can
resolve
google.com
as
intended
previously
and
yeah.