►
Description
Come join Zac and Fang, where we'll look at the https://github.com/K8sbykeshed/k8s-service-lb-validator test suite - and how it can be used to rapidly ascertain cluster networking health in a world of divergent service proxy implementations for K8s, and learn about sonobuoy plugin design !
A
B
Okay,
cool
hi
guys
my
name
is
zach.
I'm
excited
to
be
here
in
so
many
ways
for
a
brief
self
introduction.
Previously,
I
have
worked
on
work
as
an
engineer
on
platforming
for
a
field
for
for
a
few
years
range
for
a
range
farm
deploy,
icd
automation,
monitoring
stuffs,
I'm
currently
working
in
the
tanzu
infrared
team,
working
on
the
kubernetes
networking
stuffs,
yeah
and
also,
I
think,
maybe
other
people
can
do
self-introduction
like.
C
B
Yeah
and
also
very
welcome
vlad
coming
here
previously
when
they
are
doing
this
project
doing
the
service
validator
kubernetes
service
manager.
We
got
a
lot
of
help
from
blood
to
run
this
web
data
on
using
using
sony
boy.
So
by
that
we
can
just
run
this
product
in
any
kubernetes
cluster.
Very
easy
awesome.
D
I
should
introduce
myself
then,
since
you
mentioned
me,
my
name
is
vladimir
vavian,
I'm
engineer
at
vmware
working
on
open
source,
stuff,
sona
boy.
I
think
we're
is
gonna
be
discussed
today,
which
is
one
of
the
projects.
I've
worked
on,
so
definitely
just
happy
to
be
here
and
add.
Whatever
I
can
add
to
the
discussion.
A
Hey:
what's
up
arun,
I
see
people
are
starting
to
roll
in
looks
like
lou
braun
is
here,
hey
lebron
june
jen
is
here
all
our
friends
june,
jan
over
from
the
andrea
team
and
pera?
Is
here
and
I
know
pear,
had
an
interesting
question
we'll
get
to
your
question
in
a
little
bit
and
june.
Jen
can
help
answer
it
pair
about
the
ovs
nsx
stuff,
okay,
cool,
all
right
yep.
We
see
zach
I'll
turn
your
screen
on.
B
B
A
B
Okay,
this
this
is
one
kubernetes
ide,
which
I
use
to
visualize
the
components
inside
of
any
cognitive,
kubernetes
cluster,
which
is
very
useful,
I'm
not
sure
people
using
it,
but
I
want
to
share
this
yeah.
So
so,
if
you
open
lens
it
can,
if
you
have
the
good
config
file
on
your
local,
you
can
just
visualize
any
thing
inside
of
the
cluster.
B
A
E
D
E
A
B
A
A
B
Yeah,
okay,
thanks
for
joining
yeah,
so
using
lens
you
can
very
easy
to
to
to
to
to
see
what's
inside
of
the
cluster,
how
many
paws?
What's
a
config
map
deployments
going
on
staple
sets,
replica
sets
these
kind
of
things
yeah,
so
current
the
current
current
cluster
is
very
empty.
Actually,
so
this
is
just
for
testing,
so
I
have
three
four
four
parts:
four
nodes
running
and
there
are
some
pausing
in
there
yeah
try
it.
B
It's
very
useful
and
very
easy
to
use
how's
that
compared
to
octane
this,
this
one
is
id,
so
you
need
to
install
it
on
your
local
machine,
so
it's
not
opened
in
the
in
the
in
the
browser.
Oh
I
see
and
also
you
can
add
more
clusters
here.
If
you
have
more
different
good
config
files,
so
easy
to
switch
different
ones,
okay,
cool!
We
can
go
to
the
next
one.
B
Yeah
cool
yeah,
there's
article
about
five
combinations
trend,
trends
to
watching
2022.,
yeah,
so
kubernetes
as
a
platform
for
everything
I
I
do
want
to
call
out
the
second
one,
actually
so
kubernetes
and
machine
learning,
so
we
can
run
so.
Kubernetes
is
very
suitable
for
to
run
machine
learning,
workload,
workloads
on
the
on
the
crime
or
hybrid
or
public
clouds.
B
A
B
I'm
very
I'm
not
very
specialized
in
machine
learning
workloads
so
yeah
it.
I
just
know
the
features
about
it
to
run,
to
run
the
workflows
very
easily
on
kubernetes.
So
so
you
don't
need
to
set
up
the
clusters
by
yourself.
It
can
do
automations
for
you
yeah
cool
yeah,
maybe
maybe
people
with
machine
learning
knowledge
you
can
can.
F
B
B
Know
yeah:
I
think
this
is
a
good
question.
We
can
check
it
out
together
later,
I
think
should
be
for
both,
so
it
can
so
it
can
easily
to
automate
you
to
to
start
up
the
cluster
and
then
run
run
the
binaries
of
machine
learning
into
the
clusters:
cool,
cool,
cool;
okay,
let's
go
to
the
next
one,
there's
a
question
for
andrea.
B
B
The
reason
why
we
recreated
this
tool
is
as
kubernetes,
gets
more
and
more
popular
and
it
gets
applied
to
different
cloud
providers
on
prime
hybrid
multicloud
on
different
networking
devices
like
switches
and
lots
of
core
networking
modules
involved
and
the
service
proxy
is
evolving
quickly.
So
currently
we
have
many
different
sync
eyes,
like
catechol,
entria,
celium
and
those
running
on
different
os,
like
linux
and
windows,
and
also
we
have
different
dns
like
coordinates
and
coupe
dns
different
service,
mesh,
linker
d,
stu
and
even
called
proxies
two.
B
I
think
I
think
it's
three
so
but
my
ipps
and
a
happy
table
which
makes
makes
this
very
hard
to
target
and
identify
networking
figures.
So
sometimes,
when
you
run
the
networking
e3
test,
it's
very
hard
to
target
the
exact
networking
problem
of
the
cluster.
So
you
don't
know
so
when
there
is
a
networking
issue,
you
don't
know
it's
because
of
pause
inside
of
the
same
node
has
a
connection
issue
or
pause
from
different
nodes
has
a
connection
issue.
B
A
Yeah
and
I'm
just
going
to
interject
there
that,
like
you
know,
we
found
that
there's
not
a
lot
of
exercising
of
the
service
proxy.
When
you
run
k8's
conformance
tests,
there's
a
little
bit,
there's
a
few
tests
like
5
or
6
out
of
300.
That
use
it
like,
I
think,
there's
one
of
them
as
a
session
affinity
test.
A
What
you
have
is
that
your
proxy
might
be
down
on
one
node,
but
not
on
another
and
things
work
on
in
one
place
and
not
on
the
other
and
so
on
and
so
forth.
You
might
have
slowness
with
that
cd
or
your
api
server,
and
so
we
always
see
this
with
customers
where
they
have
our
network
down
and
they
think
it's
dns
or
they
think
it's
services
or
they
think
it's
that
you
know.
A
Node
ports
aren't
working
correctly
or
whatever,
but
actually
it
might
have
something
to
do
with
pod
readiness
or
it
might
have
to
do
something
with
the
particular
node
that
they're
probing
things
from
like
one
particular
known
might
be
down,
and
so
it's
like
you
get
into
this
game
of
whack-a-mole,
where
you're
trying
to
test
three
things.
At
the
same
time,
the
network,
the
topology
of
the
cluster
and
a
bunch
of
other
stuff.
A
Yeah,
that's
all
you
ever
see.
Right
is
a
timeout.
So
so
that's
like
I
mean,
did
the
initial
prototype
of
this
demo.
Did
it
signed
network
about
a
year
ago
and
then
zach
has
actually
made
it
like
production,
ready
and
and
powered
it
by
sono
buoy?
Now
so
zach,
can
you
zoom
in
a
little
bit,
so
we
could
see.
What's
going
on
on
your
screen,
there
a
little
bit,
yes
sure
sure
yeah
beautiful,
all
right.
B
Yeah.
We
can
think
about
this
product
as
a
sphere
which
compares
into
any
kubernetes
cluster
and
identify
the
networking
obstacles
so
yeah
so
for
the
visualization.
I
think
we
can
jump
into
into
this
metric
first,
so
we
can
see
each
part
if
they
can
connect
to
other
parts.
So
in
the
metrics,
if
it's
okay
or
if
it's
fail,
then
it
implies
the
connectivity.
B
Yeah,
so
currently,
so
currently
we
are
in
the
in
the
project
ci.
We
support
different
scenarios
and
proxies
accommodation,
networking,
networking
solutions
combination,
so
we
support
entria
proxy
and
the
chemical
with
cbpf
and
also
cool
proxy
in
ip
table
modes,
good
proxy
in
ipvs
mode,
and
also
sodium.
A
B
Yeah
yeah
and
we
have
developed
a
very
convenient
tool
to
to
aggregate
the
reports
from
different
ais.
We
can
see
if
the
testing
pass
on
different
networks,
networking
solutions
so
in
in
in
cdm.
We
can
pass
different
tasks
which
test
field
and
also
our
entry,
and
also
on
good
proxy,
which
test
passed
and
which
test
failed.
B
C
B
Yeah
we
can,
we
can
correct
later
after
showing
the
product
together
to
audience.
So
so,
when
we
are
trying
to
test
in
the
existing
cluster
when
we
want
to
use
this
project
actually
we
will
not
not
touch
the
existing
pause
and
service
in
the
cluster.
We
know
when
we
run
the
project.
We
will
inject
the
testing
part
into
arsenals
in
the
cluster
and
create
and
create
the
services
we
want
to
test
in
the
cluster
like
cluster
ip
node
ports,
external
name
even
node
part
local,
these
services
yep.
A
Pod
stays
stationary,
we
spin
up
all
the
pods
and
then
we
move
the
services
in
and
move
them
out
right
and
the
nice
thing
about.
That
is
it's
fast
right
because
you
can
boom
boom
boom
boom
pole
over
and
over
again
and
you're,
not
sitting
there
and
exercising
at
cd,
and
all
these
other
things
that
you
don't
care
about
right,
you're,
not
testing.
How
long
you
don't
have
this
weird
two-minute
latency
between
tests,
so
all
these
running
about
ten
ten
five
minutes
at
this
point
right,
zach,
like
you've,
got
like
we've
got.
A
B
B
B
Okay,
cool
so
inside
of
the
project
when
the
pause
and
service
are
set
up.
How
do
we
do
the
test
we
set
in
in
each
of
the
tasks
we
set,
the
it's
a
cluster
ip
service
or
node
port
service
to
to
to
support,
and
we
we
enable
pod
to
connect
with
each
other
using
agent,
using
the
connect
using
using
the
connect
command
to
to
reach
out
to
each
other
and
see
if
the
connectivity
is
is
successful
or
not,
and
then
we
generate
the
metrics.
B
A
C
A
C
H
A
B
B
Oh
yeah,
okay,
let's
move
on
yeah.
So
after
we
run
the
test,
yeah
you'll
see
some
it's
a
validation,
metric
matrix.
It's
it
implies
the
connectivity
between
pause.
So
we
can
easily
see
if
the
connectivity
is
successful
or
not.
So
there
will
be
three
parts
expected
observe
and
the
comparison
it's
just
like
the
idea
of
kubernetes
right,
the
intent
and
the
reality
of
the
state,
so
the
intent
state
should
be.
B
B
We
have
the
same
logic
for
all
the
other
for
other
test
coverage
for
different
for
different
features
like
cluster
ip
session
affinity,
node,
port
traffic,
local
yeah
and
headless
label
hairpin
host
network.
So
we
use
this
methodology
to
test
all
the
features
and
to
generate
some
metrics,
so
the
user
can
easily
identify
the
problems
inside
of
the
cluster
which
functionality
is
not
working.
I.
A
Have
a
war
story
here:
we
just
got
out
of
a
long
customer
escalation
around
headless
services.
Recently
they're
really
tricky
because
there's
no
cluster
ip,
and
so
you
hit
a
you,
hit
a
headless
service
and
core
dns
is
load,
balancing
you
to
the
pods
and
so
being
able
to
diagnose
those
and
test
them,
especially
on
like
legacy.
A
B
Yeah
and
when
you
want
to
test
against
any
cluster,
it's
very
easy
to
run.
If
you
have
the
complete
file
on
your
local
and
if
you
can
run
group
cattle
on
your
local
to
fashion
information
forms
cluster,
you
can
always
run
this
project
and
you
can
get
better
reports
on
your
local
machine
with
visa
metrics,
okay
cool.
B
So,
for
the
remaining
part
of
this
demo,
I
will
go
through
the
code
structure,
go
through
the
repo
and
also
show
you
how
to
run
with
soundboy,
and
also
I
will
show
you
one
one
interesting
finding
of
with
this
project.
So
we
have
reproduced
one
ask
one
upstream
issue
to
which
is
about
the
ib
table
and
ipvs
modes
in
cool
proxy
is
different
behave
differently
for
session
affinity,
service,
okay,
cool,
so
moving
on,
let's
have
a
quick
code
walkthrough,
instead
of
the
ripple,
I
need
help
from
fong.
B
A
So
thanks,
while
you're
switching
big
thanks
to
vlad
and
eleanor
and
john
for
supporting
us
in
trying
to
move
this
over
to
being
a
sono
buoy
plugin,
that's
important
to
us,
because
we
want
to
be
able
to
give
this
to
people
to
run
in
a
way,
that's
coherent
with
the
way
that
they
normally
run
conformance
tests.
You
know
like
for
us
if
we've
got
customers
and
we
tell
them
to
run
conformance
if
they
think
something's
wrong
and
we
they
we
use
sonobuoy
for
that.
So
they
you
know
between
buoy
and
crash
the.
E
And
it
I
just
say
that
the
direction
I've
got
dropping
sex.
I
want
to
get
this
in
the
direction
that
you're
doing
going
using
sauna
buoy
for
automated
cluster
validation
is
exactly
what
we're
hoping.
The
future
of
direction
of
sauna.
Buoy
will
be.
Sanabo
is
known
for
conformance
testing
and
that's
excellent,
but
we
think
it
has
a
wider
use
as
cluster
validation,
so
yeah
and
I.
B
F
F
So
you
can
just
say
you
know,
run
you
know
santa
buoy
run
mode,
networking
checks
or
something
like
that,
and
once
we
fine
tune
that
list
of
tests
it'll
be
easier
to
run
those
and
you
can
run
those
at
the
same
time
as
you're
running
these.
Or
unless
are
these
tests,
particularly
disruptive?
Or
can
you
run
other
tests
in
parallel?
At
the
same
time,.
C
C
C
A
A
B
C
Yep,
okay,
yeah
all
right
nice.
In
the
end,
I
wanna
wanna
show
you
how
the
the
test,
the
test
case,
is
actually
written
and
so
remember
the
first.
We
are
using
the
end
to
end
framework
here
and
the
first
things
is
to
do
the
setup.
Every
test
case
goes
to
the
setup
and
we're
where
we
have
all
the
nodes
and
we
are
adding
part
to
each
node
so
for
if
we
have
four
nodes
and
after
the
text
setup
function
is
run,
we
will
expect
four
different
parts
here.
C
C
C
That
means
the
service
have
an
end
point
to
its
corresponding
part,
and
after
that
is
done,
we
have
set
up
all
the
test
fixture
and
we
are
ready
to
assert
the
reachability
behavior
and
for
this
particular
particular
case
the
since
we
are
using
end-to-end
frameworks,
the
test.
The
assertion
is
here
right
and
we
are
creating
a
new
reachability
matrix.
I
A
D
C
K
H
A
All
right,
so
so,
by
the
way,
one
other
thing
fong
is
that
you
know
some
of
y'all
will
notice
these
from
the
e2es
upstream.
So
we've
taken
bits
from
the
e2es
upstream:
that's
how
we
do
the
probe
function
in
the
other
e2es
for
like
network
policies
and
stuff,
so
he's
like
borrowed
that
and
mashed
them
up
with
other
stuff
here.
So
some
of
you,
some
of
this
stuff,
might
look
familiar
to
all,
and
that's
that's
probably
why.
C
C
A
So
when
they,
when
we
actually,
we
might
want
to
show
them
what
the
invocation
is
from
the
sonaboli
side.
Since
most
of
people
here
are
probably
going
to
be
users,
even
though
we
would
love
to
have
contributors,
and-
and
we
should
talk
about
that
like
how
people
can
help.
But
like
can,
we
show
them
the
sona
bowie
invocation
and
and
how
they
actually
get
that
visual
table
readout
that
a
meme
was
talking
about
out
of
it.
C
All
right,
I
think
we
said
I
can
summarize
like.
We
have
entities
that
represent
different
resources
to
be
test
and
we
have
all
the
tests.
I
we
we
exploited
the
antenna
frameworks
and,
and
we
have
in
the
main.task,
we
have
everything
set
up.
And
finally,
we
are
running
the
assertions
during
in
the
assess
functions.
C
B
B
A
B
A
A
B
Yeah-
let's
get
to
this
part,
sunnyboy
yeah,
very,
very
simple-
to
sound
about
team.
Actually,
so
I've
got
especially
vlad
have
gotta
through
how
to
set
this
up
and
help
help
us
to
understand
how
to
use
onboard.
So
currently
we
can
run
the
project
into
any
kubernetes
cluster.
If
you
have
the
config
file
on
your
local
machine,
so
we
can
use
so
we
can
use
sound
by
command
to
inject
two
parts
inside
of
the
cluster,
so
the
first
one
is
solid
board.
The
second
one
is
the
plugin
itself.
B
A
B
B
You
can
see
there
are
many
testing,
so
there
are
certain
parts
and
nodes
inside
of
this
cluster.
So
currently
we
want
to
test
the
stability,
the
networking
stuff
inside
of
the
cluster,
so
this
is
my
local
machine.
This
is
my.
This
is
my
local
machine.
I
just
have
the
group
config
file
on
my
local
and
I
can
use
the
group
ctl
quick
cuddle
command
to
get
the
resource
in
the
cluster
okay.
B
F
E
B
A
A
It
works
and
x
doesn't
mean
it
doesn't
work
and
like
when
you
do
things
like
node
port
local
tests.
It's
interesting
because
you
know
you'll
see
x's,
but
it's
working
because
you,
you
just
don't
expect
certain
things
to
forward
traffic
and
other
ones
to
to
work
right
because
you'll,
if,
if
the
pod
is
on
another
node
you're,
not
gonna,.
B
G
H
C
H
C
B
Running
inside
of
the
cluster,
and
they
are
doing
the
task
and
after
the
test
finish
see,
will
be
decommissioned
after
the
test
finish,
we
can
also
retrieve
the
report.
I
will
show
you
later
when
it
finish
yeah,
let's
let
it
run
and
in
the
meantime,
in
the
meantime,
let
me
show
you
how
to
how
to
run
local.
A
B
D
So
the
I
think,
the
question
that,
or
the
issue
that
john
wanted
to
bring
up
before
he
left
was
the
use
of
raw
output
as
opposed
to
a
structured
output
for
the
test
results,
so
I'm
guessing
you're
doing
raw
so
that
you
can
build
a
table
as
and
lay
out
the
column.
The
layout
that
you
have
with
the
table
is
that
is
that
why.
A
D
Know
I
mean
you
know
without
jumping
into
each
framework.
You
can
because
e2e
framework
is
basically
goes
testing
framework,
that's
all
it
uses,
and
with
that
you
can
output
json.
If
you
wanted
to
capture
it
in
a
structured
manner,
but
what
I
was
talking
about
is
actually
more
specifically
for
for
sona
boy.
A
C
Oh
yeah,
I
I
don't
think
yeah,
so
the
output
here
is
so
like
the
formatted
output.
Here
is
only
from
the
validator.
We
are
only
passing
the
the
done
file
once
it's
finished
and
pass
that
the
the
finish,
the
signal
to
the
sono
boy.
We're
also
wonder
if
there's
some
structural
way
that
songboy
can
accept
results
like
like
a
json
file
or
something
yeah
yeah.
C
D
A
D
D
B
Yeah
yeah
yeah:
that's
that's
thanks
for
adding
it
yeah
we
do
need
to
actually.
Currently
we
are
just
parsing
the
the
the
output
file
and
to
identify,
if
the
result
passed
or
not,
we
did
not
include
the
metrics
into
the
result
of
soundboard
result,
but
but
the
people
can
still
check
the
the
output,
the
log
of
the
testing,
so
you
can
also
see
the
metrics
in
the
log
okay
cool.
B
D
B
Okay
cool,
so
so
we
can
use
a
command
make
sonobiotree
to
repeat
the
result
of
the
testing
and
the
current
one
is
passed.
The
current
the
current,
the
current
one
is
passed.
We
saw,
we
can
see
the
detailed
log
in
your
local
machine
and
you
can
see
yeah,
oh
so
yeah
yeah,
it
seems
it
seems
it
seems,
like
a
result.
Folder
it's
in
the
dark,
folder.
A
E
B
B
D
A
B
B
B
Yeah,
okay
cool.
We
have
right.
We
have
used
this
tool
to
test
against
the
the
cluster,
which
is
very
easy
to
run
because
the
use
makes
on
the
boy
run
and
make
sure
nobody
retrieve
to
get
back
to
get
back
the
result
yeah.
I
think
we
have
done
the
demo
of
this
project,
but
I
still
want
to
include
one
thing
which
which
is
very
interesting
funding
using
this
project
before
I
work
on
it.
B
B
So
when
you
have
a
session
affinity
cluster
ip
service,
if
you
have
the
same
client
to
reach
to
the
class
cluster
ip
service
with
session
infinity,
the
result,
the
behavior
is
different
in
cool
proxy
ip
table
mode
and
ipvs
mode.
So
in
ip
table
mode,
the
session
affinity
is
attached
to
the
specific
port.
A
B
B
We
expect
the
same
target
pause
for
all
spots,
but
it's
different.
So
this
time
we
have
different
targets,
so
each
part
goes
to
different
parts
target
pod
using
port
81
yeah.
This
is
iptable.
This
is
its
behavior,
but
if
we
are
using
iqes,
but
if
we
are
using
ipvs
when
we
establish
the
session
affinity
using
one
of
the
poor
ports
and
then
same
client
to
reach
out
to
the
same
class
id
service
using
another
port,
they
will
still
go
to
the
same
target
pod.
B
A
A
B
A
E
E
B
A
H
A
B
A
B
B
A
B
A
My
mother-in-law
got
me
this
hat.
Do
you
all
like
it?
This
is
my
hat
for
the
show
he's
very
professional
as
zach
would
say:
okay
good
show
zach.
That
was
that's
really
that's.
I
really
appreciate
you
coming
on
and
doing
the
show
and
thanks
a
meme
for
mentoring
and
helping
these
folks
build
this
out
and
fong
han
thanks
for
not
sleeping
for
a
month,
while
you're
doing
another
job
and
helping
zach
along
the
way.
A
Yeah
yeah
nice
work,
nice
work.
So
next
we
need
to
yeah.
It's
like.
Let's
see
what
we
can
do
to
get
this
more
deeply
integrated
into
andreas
cni,
I
enter
a
ci,
somehow
yeah.
I
don't
know
june
jen
if
you're
still
on
the
show.
What
do
you
think
you
think
if
we
filed
an
issue,
we
could
start
running
this
as
part
of
andrea's
ci
jobs
and
outputting
the
results.
I
think
that
would
be.
A
We
would
really
like
to
do
that
if
it's
we're
planning
on
trying
to
get
this
integrated
with
the
sig
network
jobs
as
well.
I
mean
demoed
this
about
a
year
ago,
and
tim
hawkins
was
on
board
with
that,
but
we
we
always
like
to
put
things
in
andrea
first,
if
we
can
just
because
it's
a
really
good
proving
ground
before
we
go
full-on
upstream.
A
B
A
B
Okay,
cool
yeah-
let's
let's
put
a
here
thanks
for
thanks
all
for
joining
and
spending
time
with
us,
the
bell
theater
will
get
better
and
along
the
way
we
hope
we
can
get
your
help.
I
just
submit
pr's
issues
or
open
discussion
in
the
github
repo
for
this
project.
We
spend
time
and
we
open
source
it.
I
hope
it's
useful
to
the
community
and
last,
if
you
like
this
session,
please
click
the
like
button
subscribe
to
the
channel
and
even
turn
on
the
ring
bell
of
this
channel.