►
From YouTube: Visualise Cluster Connectivity with Goldpinger; Smash it with PowerfulSeal - Mikolaj Pawlikowski
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Visualize Cluster Connectivity with Goldpinger; Smash it with PowerfulSeal - Mikolaj Pawlikowski, Bloomberg
Let's talk about two open source tools to test and troubleshoot your Kubernetes setup: Goldpinger and PowerfulSeal. Goldpinger is a Kubernetes networking troubleshooting tool. Deploy it on your clusters and get a pretty graph of connectivity between all your nodes, Prometheus metrics to generate alerts from, and more. We will be introducing the tool and sharing it with the community during the conference! PowerfulSeal brings Chaos Engineering to your Kubernetes clusters. It will destroy things following your policies, so that you can discover and fix problems before real outages happen. We published PowerfulSeal at KubeCon + CloudNativeCon North America 2017 in Austin, and we'll be giving you an update on the project's momentum. Learn more about it at https://github.com/bloomberg/powerfulseal. Come to our talk and gain two new Kubernetes superpowers!
To learn more: https://sched.co/GrX6
A
A
Ok,
that's
good!
So
my
name
is
Miko
I
pavlikov
scheme
I
flew
here
from
London
for
Bloomberg
office
in
London,
so
that
means
that
my
biological
clock
is
roughly
a
quarter
to
nine.
So
if
I
fall
asleep
in
flight,
don't
hold
this
against
me.
I'm,
going
to
talk
to
you
about
two
friends
of
mine,
Goldfinger
and
powerful
CEO
that
are
hopefully
going
to
become
friends
of
yours
tune
by
the
end
of
this
talk.
So
the
idea
for
the
next
30
minutes
is
to
first
make
a
bold
statement
that
distributed
systems
are
hard.
A
It's
going
to
be
my
hypothesis
and
then
talk
about
Goldfinger
setting
it
up
using
it
and
powerful
CEO
with
a
quick
introduction
to
chaos.
Engineering
for
those
of
you
who
haven't
heard
of
that,
even
though
I
think
it's
pretty
difficult
to
pass
the
cube
come
without
hearing
about
it.
Now
we're
going
to
conclude
and
have
some
bonus
section
with
answers
that
you
can
use
to
sell
it
to
your
manager.
So
before
all
of
that,
let
me
just
quickly
do
a
shout
out
to
my
team
back
in
London
this
all
some
people
make
what's
called
DDP.
A
It's
a
microservices
platform
that
we
built
on
top
of
kubernetes.
That
has
some
interesting
QC
features
I'm
not
going
to
go
into
details.
We
gave
a
talk
years
ago
on
cube,
come
so
if
you're
interested
you
can
get
that
all
right.
So,
let's
start
with
the
show
of
hands
who
thinks
that
distributed
systems
are
easy.
A
All
right,
that's
good!
Otherwise,
I
have
to
stop
the
talk
here.
I
think
they're
particularly
hard,
and
if
you
are
anything
like
me
and
you
have
a
bad
day
and
you
go
to
Google,
uncle
Google
does
provide
some
solace
to
this
is
a
screenshot
49
million
reasons
to
that's
difficult
and
I'm.
Seeing
some
confused
faces
the
public
confused,
because
it's
already
old
crome
you
are
that
was
December
last
year.
If
you
take
a
new
screenshot,
we
are
the
87
million
results
and
it
doesn't
stop
there.
A
I
was
September
if
we
took
one
last
week
we
already
had
201
million
reasons,
so
don't
feel
bad.
We
are
all
in
the
same
boat,
so
I,
don't
know
what
you
guys
do
your
free
time
in
the
evenings,
but
one
of
my
favorite
hobbies
is
to
extrapolate
completely
relevant
data.
So
if
we
do
just
that
in
a
almost
scientific
way
on
my
graph,
we
can
clearly
see
that
reaching
roughly
four
gazillion
reasons
soon
enough
on
the
x-axis
to
strike
a
slightly
more
serious
poster.
I
like
this
quote
from
Lampert
at
the
paraphrase.
A
It's
basically
saying
that
you
know
in
a
distributed
system
were
a
computer,
we
didn't
know,
existed,
messed
up
your
own
computer
and
that's
really
the
essence
of
it.
I
don't
think
there
is
a
code
that
makes
gives
more
justice
to
it,
because,
essentially,
everything
is
plotting
against
you
right.
The
networking
is
particularly
plotting
against
you.
All
the
time
fails,
hardware
fails,
environments,
diverge
and
consistency
is
either
hard
or
expensive
or
both.
A
So
the
failure
really
is
a
mother
of
one
and
it's
not
an
if
and
that's
probably
like
the
biggest
mind
shift
that
one
has
to
take
in
order
to
be
able
to
deal
with
that.
So
if
you
are
struggling
with
this
one
and
not
an
eff
problem
or
you
have
to
convince
someone
like
your
manager,
hypothetically
here
is
an
example
of
what
you
can
use
so
a
good
example
in
this
setup,
for
me
has
three
criteria.
It
has
to
have
an
abbreviation.
A
It
has
to
have
some
mental
arithmetics
that
can
be
done
in
the
head
of
a
person.
It
has
to
fit
in
three
bullet
points,
so
this
should
check
all
these
three
boxes.
So
if
we
take
a
reliable
set
of
reliable
servers,
let's
say
the
median
time
between
failure
of
ten
years
check
for
the
abbreviation
and
we
take
4,000
of
these
servers.
We
are
already
at
averaging
more
than
one
failure
dating
right,
easy
arithmetic
for
your
manager
check
and
we
don't
need
web-scale
for
that.
We
don't
need
Google
scale.
A
We
just
need
four
thousand
servers
to
average.
That
and
averages
are
also
quite
horrible,
because
that
means
that
we're
probably
going
to
have
good
days
and
I
have
terrible
days
where
everything
goes
wrong.
So
basically,
the
bottom
line
here
is
that
the
fault,
tolerant
design
is
inevitable
and
I.
Don't
think
I
need
to
convince
anyone
of
that.
Even
though
we
have
this
conversations
daily,
there's
a
little
link
there
that
I
recommend
are
looking
at,
but
I
won't
spend
more
time
on
that.
I
think
we're
on
the
same
page.
A
So
this
is
all
a
little
bit
gloomy
and
a
little
bit
depressing,
but
the
good
news
is
that
it's
a
conference,
so
there
should
be
good
goodies
and
I
have
two
goodies
for
you,
the
IP
type
of
goodies,
not
t-shirts,
one
of
them
is
called
Goldfinger
and
the
other
one
is
powerful.
So
to
give
you
a
little
bit
of
background,
powerful
seal
and
Goldfinger
are
both
pieces
of
software
they've
been
running
at
Bloomberg
here
for
about
two
years
and
I've
proved
extremely
useful.
A
Gulping
girl
is
my
best
friend
like
quite
honestly,
we
bonded
so
much
that
if
it
came
to
me
today
and
said,
listen
it's
good
time
to
inverse
in
Bitcoin
I'd,
be
like
okay,
Goldfinger
I.
It
looks
bad,
but
I'll
trust
you
I
trust.
You
no
questions
asked
so
Goldfinger
does
one
thing:
it
basically
things
other
gold
fingers
that
it's
going
to
ask
you
burn.
It
is
to
give
them
all
it
all
the
pods
and
it's
going
to
ping
them
periodically
and
provide
you
with
visibility.
A
So
the
visibility
kind
of
the
thing
that
you
interact,
probably
with
the
most,
is
in
the
form
of
this
UI.
This
is
a
small
graph,
more
likely
you're
going
to
see
graphs
that
look
more
like
this
and
basically
shows
you
all
possible
connections
between
the
nodes.
So
it's
going
to
call
all
its
neighbors
and
ask
the
neighbors
to
call
all
their
neighbors
to
produce
this
complete
graph
for
you.
This
is
good
for
good.
A
For
debugging-
and
this
is
probably
one
of
the
screens
that
I
stare
at
a
lot
during
my
days-
but
probably
the
mo
more
useful
and
more
interesting-
is
that
it
comes
with
a
primitives
matrix
out
of
the
box,
so
the
primitives
matrix
are
this
thing
on
the
right,
the
graph
on
a
dashboard?
You
can
actually
go
ahead
and
grab
it
from
the
repo
and
kind
of
drop
on
your
cluster
and
also
a
possibility
to
set
up
alert
manager
alerts.
A
If
you
are
running
on
primitives
and
quite
frankly,
you
should
so
setup
is
quite
simple
to
him.
It's
take
matter
of
acute
care
to
apply
the
thing
on
the
right
is
a
direct
screenshot
from
the
readme
on
the
github
repo,
and
that's
really
all
you
need
to
do
to
get
started
with
that.
So
we
open
sourced,
we
publish
it
as
open
service
lishi
last
week,
so
it's
brand-new.
A
A
While
giving
a
demo
we're,
obviously
by
definition,
I
can't
type,
so
this
is
Goldfinger
on
github,
and
this
is
the
it
describes
all
the
steps
for
building
installation,
and
this
is
the
mo
that
I
was
talking
about
it's
a
demon
set
with
a
service
I'm
going
to
go
ahead
and
try
to
say
okay.
So
this
was
my
mini
cube.
I
ported
that
on
mini
cube,
so
that
I
don't
have
to
rely
on
the
network
and
I
have
checked
out
the
repo
with
the
example
on
the
service
account
and
just
so
that
you
can
see.
A
This
is
literally
all
there
is
to
it.
The
one
interesting
bit
is
probably
that
I'm
injecting
hostname
as
spike
a
node
name,
so
that
in
the
graphs
and
metrics
I
can
actually
see
the
real
host
name
so
that
it's
easier
to
understand
things
but
I'm
just
going
to
go
ahead
and
apply
that
before
there.
Let
me
see,
let
me
watch
the
pod
so
that
you
can
see
what's
coming
up,
I'm
gonna
go
ahead,
a
ply,
create
a
service
and
create
a
sea
demon
set,
and
it's
already
come
up.
So
let
me
show
you
the
UI.
A
A
So
I
did
cheat
a
little
bit
and
I
set
myself
a
deployment,
a
deployment
that
produces
basically
more
gold
finger-post
for
you,
so
that
I
have
something
to
think
so,
if
I
scale
that
to
ten
and
I
wait
for
for
them
to
come
up
at
the
end
of
creating
they
started
running,
and
if
I
refresh
this
a
couple
of
times,
I
should
be
showing
up
the
earlier.
So
that's
how
what
the
graph
looks
like
you
can
go
ahead.
You
can
click
around.
You
can
kind
of
see.
A
What's
going
on
in
debugging
and
see
the
view
from
from
a
particular
node
and
like
I
mentioned,
the
metrics
are
available
right
here,
alright,
so
that,
hopefully
was
less
than
three
minutes.
Let
me
go
back
to
the
presentation
so,
like
I
said,
the
gold
finger
is
now
open.
Service
I
can
go
and
grab
one
and
hopefully,
by
the
end
of
this
presentation,
you're
going
to
have
it
installed
in
your
clusters.
A
A
Let's
take
a
look
at
the
other
part
of
the
presentation
about
Kaos
engineering,
so
who
hasn't
heard
of
Kaos
engineering
the
audience
here
all
right,
I
see
exactly
zero
hands.
So
that's
good.
That
means
that
you've
all
went
have
gone
to
principle
of
cast
at
org,
and
you
seen
the
document
and
you
know
where
it
is
right,
but
just
in
case
some
of
you
haven't
raised
their
hand,
I'm
going
to
make
it
easy,
and
just
summarize
that
in
three
bullet
points.
A
So
basically
the
idea
is
to
increase
confidence
in
a
distributed
system
by
purposefully,
introducing
failure
and
this
way
being
able
to
detect
bugs
that
other
layers
of
testing
haven't
caught
and
in
this
in
this
mindset
we
build
powerful,
CEO.
So
powerful
CEO
is
our
chaos
engineering
tool
that
we
have
open
source
last
year
at
the
version
1.0.
Today
we
have
version
2.0.
It
was
initially
inspired
by
chaos
monkey.
Then
it
evolved
kind
of
a
life
of
its
own.
A
But
the
main
difference
really
is
that
powerful
skills
designed
to
speak
kubernetes,
it's
kind
of
qnt
static,
occurs
engineering
tool.
So
it's
very
easy
to
work
with
the
abstractions
that
you
bring
any
provides.
It
comes
with
interactive
mode.
It
comes
with
a
few
more
scenarios.
We're
going
to
look
at
that.
The
setup
is
pretty
straightforward.
A
It
needs
to
talk
to
your
Cuban
G's
API,
so
it
needs
either
serve
public
cube,
config
and
that
needs
to
be
able
to
execute
sudo
the
double
kill
on
the
nodes
of
versus
H,
and
the
reason
for
that
is
that
if
you
just
delete
pods
through
kubernetes,
you're
kind
of
giving
it
headstart
and
if
you
crush
the
processes
themselves,
then
you
kind
of
test
in
the
whole
loop,
because,
as
far
as
good
range
is
concerned,
it's
going
to
show
up
as
a
crash
all
right.
So
then
modes,
powerful
skills.
A
Since
version
1.0
has
developed
four
different
modes
and
we
have
labeled
demo,
interactive
and
autonomous
mode.
So
in
label
mode
you
kind
of
start
the
seal
and
then
you
go
ahead
to
your
communities
and
you
mark
the
pods
as
targets.
So
these
are
the
things
that
are
available.
You
mark
it
as
sill
slash
enabled
to
true
to
actually
enable
powerful
Co
to
kill
it.
And
then
you
have
tweaks
and
things
like
kill
probability
or
days
of
the
week
and
start
and
time
to,
for
example,
I.
A
A
So
I'm
back
here
to
my
cluster
I,
have
all
this
Goldfinger
nodes
running
iPods
running?
So
why
don't
we
given
some
hard
time?
One
thing
is
that
a
powerful
CEO,
if
you
want
to
test
it
out
of
the
box,
supports
mini
cube.
So
we
have
things
like
make
label,
so
you
can
basically
just
go
ahead
and
do
that
target
on
the
mag
file.
It's
going
to
run
with
defaults
that
can
that
work
with
mini
cube,
so
you
can
see
that
it
started.
But
it's
not
doing
particularly
much.
A
No
ports
are
dying
because
I
haven't
marked
any
so
I'm
going
to
go
ahead
and
I'm
going
to
mark
some
with
my
cube
cattle
I'm
going
to
do
cube
cattle
label,
I'm
gonna
select
deployment
yep,
it's
just
a
random
selector
that
I
did
I'm
gonna
set
the
end
time
to
midnight
because
I
actually
am
in
London
as
far
as
this
is
concerned.
So
let
me
do
label
enable.
A
Enable
true
proof-
and
if
we
go
back
to
our
powerful
CEO
and
we
see
that
it's
actually
quite
aggressive,
so
it's
basically
killing
everything
well
killing
welcome
every
few
seconds-
and
this
is
a
nice
occasion
to
go
ahead
and
take
a
look
at
what
the
graph
looks
like
when
things
don't
go
so
well,
so
I'm
cleaned,
I'm
kind
of
randomly
so
I'm
expecting
every
time,
I
reload
the
graph
to
be
slightly
different.
But
you
can
see
that
this
gives
you
an
overview.
A
A
It's
one
of
our
latest
additions
that
will
probably
be
deprecated
in
biggest
hipster
went
away,
but
the
idea
here
is
that
you
point
powerfully
at
hipster
and
it's
going
to
use
the
metrics
to
kind
of
try
to
figure
out
what
is
important
on
your
cluster.
What's
using
CPU
what's
using
rum
and
then
you
can
go
and
just
let
it
run
and
kill
things
for
you
a
little
bit
randomly
the
next
mode
is
interactive,
and
this
is
the
first
mode
that
we
actually
wrote
for
this
year.
It
basically
gives
you
like
console
who
are
with
nice.
A
Tab-Completion
features.
You
can
go
ahead
and
you
can
work
on
things,
so
it
lets
you
take
nodes
up
and
down.
I
can
delete
them.
If
you
want,
you
can
filter
them
on
kinds
of
things
and
for
the
pod
you
can
list
no
spacers
deployments
all
the
kinds
of
things
that
you
would
do
manually
with
keep
cattle
in
one
place
and
you
can
go
ahead
and
kill
them.
I'm,
not
gonna.
Do
that
what
is
probably
the
most
important
mode
is
the
autonomous
mode,
and
this
is
when
you've
already
tested
it.
A
You've
played
it
out,
kick
the
tires
and
you
have
an
application
that
you
want
to
actually
see
if
it
survives
through
some
SLS
that
you
expected
to
so
in
autonomous
mode,
powerful
Co
works
with
policies.
A
policy
is
a
simple
llamo
file
that
consists
of
basically
two
arrays
of
things
of
scenarios.
One
is
for
nodes
and
one
is
for
pods
and
they
both
follow
the
same
schema.
Where
you
much
an
initial
set
of
things,
then
you
filter
through
that
and
then
you
action.
So
whether
these
are
nodes
or
pods,
you
can
match
them.
A
We're
going
to
go
through
an
example
and
just
in
a
second
you
duplicate
that,
then
you
filter
out
the
things
that
you
don't
want
and
then
you're
action
on
all
the
things
that
are
remaining
so
to
go
through
the
example
from
the
repo.
It
looks
something
like
this
so
for
the
matters
and
the
top
one
is
for
the
pods
you
can
match
by
just
listing
and
namespace
listing
deployment
or
doing
some
more
funky,
with
labels,
free
style
and
for
nodes.
A
On
the
other
hand,
you
can
list
by
things
like
name
IP,
group,
availabilities
and
state
the
kind
of
things
that
you
will
expect,
then
in
the
filter
face
you
can
filter
again
by
the
various
properties,
and
you
can
do
things
with
randomness.
So
the
usual
things
that
you
would
do
will
be
take
a
random
sample
of
a
particular
size
or
a
particular
percentage
of
whatever
group
it's
resolved
to
and
then
finally,
you
just
add
actions
which
for
nodes
are
start/stop
and
execute.
Execute
will
come
in
handy.
A
If,
for
example,
you
want
to
test
things
that
don't
run
on
kubernetes
but
affect
your
cluster.
So
if
you
wanted
to
go
ahead
and,
for
example,
restart
your
flannel
D,
a
demon
or
stalker
or
whatnot,
you
can
just
execute
it
like
this
and
for
pods.
What
you
do
is
you
basically
go
and
kill
them.
So,
if
you
so
when
you're
running
with
autonomous
mode,
it
comes
with
a
couple
of
nice
features
like
again.
A
It
comes
with
built-in
primitives
metrics
so
that
you
can
plot
things
and
you
can
alert
and
then-
and
it
also
comes
with
a
URI-
that
I
should
actually
probably
just
show
you.
So
if
we
go
ahead
and
we
take
an
example
from
the
repo
I
have
copied
it
here-
and
this
is
a
very
minimal
conflict
that
basically
between
every
between
1
and
30
seconds
goes
goes
ahead.
It
takes
all
the
parts
from
the
default.
Namespace
takes
a
random
sample
of
that
and
kills
it
with
probability
of
77%,
so
starting
that
is
pretty
easy.
A
It's
again
available
through
the
mini
cube
autonomous
mode.
If
you
just
start
that
these
are
the
options
that
are
specifying
and
my
powerful
skill
is
up,
I
should
be
able
now,
with
a
little
bit
of
luck
to
go
through
the
UI.
So
the
UI
provides
you
with
visibility
into
what's
going
on,
it's
going
to
stream
logs
here,
for
you
about.
What's
going
on
in
particular,
cannot
kill.
Container
interesting.
I
can
start
stop
things.
I'm
gonna
go
ahead
and
stop
it.
You
can
go
and
play
with
the
configuration
so
inside
the
different
scenarios.
A
You
can
go
and
tweak
the
variables
see.
If
that
improves
it,
you
can
add
matters.
You
can
basically
modify
all
the
config
that
you
can
do
through
llamó
and
then
you
can
either
export
it
to
llamo
and
go
ahead
and
check
it
in
the
repo
somewhere
or
you
can
also
override
it
on
disk.
If
you
happen
to
be
writing
this
locally
and
finally,
you
can
also
just
go
ahead
and
do
things
and
kind
of
click
and
play
away.
So
I'm
writing
this
with
no
cloud
or
meaning
cube.
A
A
A
All
right
back
here,
so
the
CEO
s
has
been
out
for
a
year
now
we
have
had
some
pretty
amazing
feedback
from
the
open
source
community
we've
had
people
ranging
from
hedge
funds
to
fortune
100
companies
actually
using
us,
but
we
would
like
more
so
if
any
of
you
happen
to
be
using
it
or
applying
to
use
it,
please
do
come
and
share
your
experience
with
us.
That
will
help
us
are
the
features
and
help
us
know
how
the
cares,
engineering
effort,
which
way
it's
going
forward.
A
We
would
love
more
clouds,
drivers
anywhere
from
Google
cloud
here.
Ok,
no,
like
so
I,
guess
no
Google
cloud
driver
for
us
and
the
stars
on
Reaper.
That's
you're,
obviously
the
most
important
metric
of
any
open-source
product.
So
these
are
the
friends
that
hopefully
are
going
to
be
for
you
in
the
difficult
times,
I'm
just
going
to
wrap
it
up
with
a
bonus
of
kind
of
selling.
All
of
those
to
your
manager,
because
there
are
like
four
repeated
questions
that
constantly
kind
of
show
up.
So
the
first
one
is:
why
break
it?
A
If
it
actually
works
right
in
in
some
people's
mind
and
the
answer
to
that
is
kind
of
what
we
covered
right,
if
the
failure
is
not
an
if
it's
a
1,
then
it's
better
to
have
this
one
as
soon
as
possible,
so
that
the
cost
of
fixing
it
is
smaller.
The
next
one
is,
you
see
some
kind
of
variant:
ok,
yeah
cool,
but
it's
production
there.
A
So
so,
let's
not
do
that
and
I
found
that,
even
in
our
case
the
easiest
way
was
to
kind
of
you
ease
ourselves
into
that
with
some
more
aggressively
set
up
in
dev
and
testing,
and
then
slowly
getting
a
production
and
usually
it's
taiko
yeah,
but
we
have
enough
trouble
already.
This
one
is
a
difficult
one,
but
it
can
be
helped
with
okay.
Let's
just
do
it
during
the
work
time
when
everybody's
in
and
we
don't
break
things
when
people
are
asleep.
A
And
finally
it's
like
okay,
we
don't
have
expertise,
and
with
this
one
you
don't
really
have
this
excuse
anymore.
You
just
got
two
tools
to
help
you
with
that,
and
they
are
pretty
easy
to
start
with
and,
quite
frankly,
it's
good
fun
too
so
I'm
gonna
wrap
it
up
here
and
my
conclusion
is
still
don't
worry
about.
The
next
outage
just
make
your
own
right,
preferably
in
production,
preferably
every
day
and
embrace
the
inevitable
failure
and
befriend
the
seal
and
indulge
Goldfinger
alright.
Thank
you
very
much.
A
B
A
So
that's
for
Goldfinger
yep,
so
obviously,
if
your
networking
layer
doesn't
allow
them
to
communicate
you're
going
to
have
to
add
that
so
that's
going
to
be
dependent
on
your
networking
stock.
So
the
question
was:
do
we
need
to
do
anything
about
policies
for
Goldfinger's
to
talk
to
each
other
and
our
questions.
B
A
B
B
A
Know
that's
a
very
good
point,
so
the
question
is
about
basically,
apart
from
just
a
single
paying
binary,
whether
I
could
connect
or
not
whether
it
supports
any
insight
into
amount
of
packet
loss
or
bigger
payloads.
The
answer
is
currently
doesn't,
but
I
will
be
very
straightforward
to
add
social
chat.
A
A
A
A
Yeah
so
we're
basically
doing
a
little
overhaul
of
powerful
CEO.
So
currently
the
setting
is
that
you
have
two
powerful
agent
graph
outside
and
it
doesn't
require
anything
to
run
on
any
of
these
nodes.
We're
working
right
now
on
adding
a
demon
kind
of
to
run
on
oldest
nodes
and,
as
part
of
that
will
be
support
for
using
different
runtime
containers,
and
they
also
be
support
for
actually
doing
a
cube
car
to
delete
pod
for
people
who
don't
want
to
give
any
kind
of
SSH
access.