►
Description
Whether production just went down and you need answers fast or you’re trying to find which deployment is using this secret that's stashed in your development environment, searching for an artifact in one, let alone hundreds, of Kubernetes clusters can be a headache! Knowing where to look and what to look for on a single cluster is an acquired skill, but what do you do at the multicluster scale? Join us as we talk about an incredible open source multicluster search tool with some of the team behind the project - Disaiah Bennett, Jorge Padilla, and Xavier Dharmaiyan. Multicluster search aims to simplify the object discovery process for the SRE, developer, and everyone in-between - all at the multicluster scale.
A
Hello
and
welcome
to
the
cloud
multiplier
your
show
for
all
things
distributed
cloud.
I
am
your
host
co-host,
gurney
buchanan.
I
am
here,
as
always
with
my
co-host
joy,
deep
and
we
are
actually
joined
by
the
wonderful
multi-cluster
search
team
from
red
hat.
So
these
are
the
three
folks
behind
a
lot
of
the
multi-cluster
search.
A
A
I
will
say
to
everyone
on
stream,
I'm
hoping
to
change
out
the
board
games
in
the
background
next
week,
that'll,
hopefully
become
a
regular
thing.
So,
two
weeks
from
now,
maybe
some
different
board
games
will
show
up
so
there's
my
little
hobby
for
the
week.
Jordy
got
any
hobbies
for
us
this
week.
We're
going
to
keep
changing
this
up
until
something
sticks.
B
B
A
new
video
we
did
well
that
says,
probably
that
says,
that's
probably
one
of
the
reasons
right
and
the
only
other
thing
I
can
report
on
has
been
awfully
hot
here
at
south
bay
in
l.a
area
and
where
I
live,
we
had
to
turn
on
the
ac
for
a
few
days.
Every
hour
I
mean
for
for
a
few
hours
for
a
couple
of
days
and
last
two
weeks
that
says
it
all.
A
A
C
A
D
Hi,
my
name
is
xavier
damian.
I
work
for
such
as
part
of
the
acm
team.
My
technical
background
must
be,
you
know
analytics,
so
I
do
scale
analytics
for
such.
So,
if
you
do
not
see
me
at
working,
you
me,
you
can
see
me
at
the
soccer
field
or
you
can
see
me
at
practicing
for
the
half
marathon,
that's
coming
up
in
november
in
downtown
raleigh.
So
if
you
are
one
of
them,
who's
running
in
downtown
raleigh
come
say
hi
to
me.
D
B
E
Yeah
sure
thing
so,
hello,
everyone,
my
name,
is
a
bennett,
I'm
one
of
the
engineers
on
the
search
and
observability
squad.
So
in
terms
of
hobbies,
I'm
not
running
light
xavier
and
I
don't
have
any
kids
yet,
but
I
started
getting
into
building
pcs.
You
can't
see
it
here,
but
off
screen
is
a
very
beautiful
big
pc
that
I'm
building
and
hopefully
I'll
get
it
ready
soon.
So
I'm
glad
to
be
here.
Whole
loop,
water,
cooling,
yes,.
E
A
Oh
gosh,
he
went,
he
went
all
in
that's
amazing.
Well,
we
are
here
with
episode,
eight.
So
a
little
bit
of
background
before
we
get
into
these
wonderful
folks
to
talk
about
our
topic.
Today,
we've
talked
on
the
show
about
a
variety
of
things
we
have
a
bunch
of.
We
actually
have
three
shows
scheduled
out
from
here
already
a
little
bit
of
a
little
bit
of
background,
but
we've
talked
about
hypershift.
A
We
talked
about
growing
your
fleet,
we've
talked
about
building
and
automating
your
fleet,
with
with
ansible
we've
talked
about
observing
your
fleet.
We've
talked
about
rolling
out
upgrades,
we've
talked
about
some
of
the
managed
services
and
analytics
that
can
help.
You
run
your
fleet
in
a
reliable
way,
and
today
I
think
we're
going
to
be
doing
one
of
honestly,
my
favorite
deep
dives,
the
search
folks
from
the
multi-cluster
search
team
here
are
going
to
talk
to
us
about
the
multi-cluster
search
component
in
rackham
and
in
general.
A
The
open
source,
multi-cluster
search
project
they've
also
been
involved
in
some
other
open
source
projects
that
are
really
cool.
Maybe
we'll
talk
about
that
a
bit,
but
this
is
really
for
me.
A
devops
sre
someone
who
runs
some
infrastructure
and
multiple
clusters.
This
is
really
a
dream,
so
I
think
I
forget
who
said
they're
going
to
kick
us
off,
but
do
you
want
to
give
us
the
primer
on
multi-cluster
search.
C
Sure
so,
like
gurney
said
we
have
seen
before
in
the
show
that
lacmo
helps
you
to
create
a
fleet
of
clusters.
It's
very
easy.
You
go
and
create
10
clusters.
You've
got
created
constant
clusters.
So
now,
after
that,
how
do
you
manage
those
clusters?
No,
we
can
go
traditional
way,
go
to
cli
into
each
of
the
clusters
and
do
your
thing
there.
C
Really
search
comes
into
play
now
we
collect
data
from
all
those
clusters,
we're
monitoring
the
clusters
and
bringing
data
pretty
much
near
real
time,
bring
all
that
into
our
acm
hot
cluster
and
from
there
we
provide
you
ways
to
to
search
for
your
resources.
Like
everything
is
in
one
place,
you
go
and
search
for
some
conditions
like,
for
example,
you
want
to
see
your
trading
pods.
You
want
to
see
all
the
assets
that
have
a
specific
label.
C
You
can
see
results
very
quickly
across
all
your
posters.
Now
it's
super
fast
because
we
are
doing
the
indexing
pretty
much
continuously
in
real
time.
So
now,
like
you,
could
have
automation
that
goes
to
the
cluster
and
brings
up
data
again.
Another
fact
that
that
we
collect
that
beforehand
makes
it
super
fast.
So
goal
is
that
you
can
access
that
data,
not
only
for
problem
termination.
C
You
know
in
the
future
we
can
we're
planning
to
do
more
things
like
you
know,
analyze
the
data
analytics.
On
top
of
it
see
how
how
things
come
flowing
into
the
system,
how
things
change,
and
now
we
can
build
on
top
of
it,
because
it's
now
it's
on
a
database,
that's
more
variable
and
we
can
use
that
data
in
many
different
ways.
C
C
After
it's
failing
see
the
assets
that
are
not
checking
for
stances
and
all
that,
and
then
you
want
to
probably
compare
again
another
cluster
see
if
the
problem,
if
all
the
clusters
are
showing
the
same
signature
of
the
problem,
you
can
do
that
by
running
some
research
queries
and
really
figure
out.
If
it's
only
one
cluster
out
in
the
problem,
all
your
fleet
hit
by
the
same
issue
and
then
the
running
ui.
You
can
even
see
the
logs
of
the
pods
try
to
do
some
problem
termination
from
there.
C
A
D
C
So
more
like
high
level
data
now
imagine
if
you
do
oc
get
pods
like
at
the
least
level
everything
on
the
command
on
the
occli
you'll
see
like
the
columns
of
the
high
level.
So
that's
what
we
collected
now,
because
we
want
to
index
a
small
amount
of
data.
We
don't
want
to
limit
everything,
because
then
we
lose
the
benefit
of
this
really
we're,
building,
search
base
and
speed
and
getting
the
data
that
you
need
so.
D
B
B
A
B
E
E
So
when
you
first
access
acm,
you
will
be
brought
to
the
show
you
the
different
features
that
you
can
access
within
the
console.
But
what
we
want
to
focus
on
today
is
the
search
feature:
go
to
the
left
hand,
navigation,
drop
down
and
select
the
search
page
now.
One
thing
that
jorge
did
mention
was
problem.
E
Determination
and
search
actually
provides
three
different
templates
or
suggested
search
templates
for
users
to
use
once
they
access
the
search
page
and
the
one
that
I
wanted
to
focus
on
today
was
actually
the
unhealthy
pause,
and
this
will
show
our
users
a
quick
insights
into
the
health
of
any
pods
that
have
a
unhealthy
status.
So
what
we're
going
to
do
we're
going
to
go
ahead
and
drill
down
into
this
tile.
E
E
One
thing
that
jorge
mentioned
is
that,
when
you're
accessing
the
data
from
search,
all
the
data
is
going
to
be
brought
into
a
centralized
place,
so
you'll
be
able
to
see
the
resources
from
the
hub
cluster,
along
with
the
managed
clusters
that
you're
managing
so
from
here,
we
can
see
that
our
managed
to
live
cluster
or
airline
application,
and
one
thing
I
should
note
is
that
this
airline
application
is
actually
deployed
across
our
fleet
of
clusters.
But
for
this
managed
cluster
you
can
see
that
it's
within
the
crash
move
back
off
state.
E
So
one
unique
feature
that
jorge
mentioned
was
actually
going
into
the
resource
or
the
pod
resource
and
viewing
the
logs
that
are
available.
So
we
can
drill
down
into
that
pod
resource
and
from
the
top
navigation
you
can
see
the
logs
link.
So
if
we
click
on
that,
we'll
see
that
there's
a
few
issues
that
are
occurring
within
our
container,
so
we
are
running
a
postgres
database
image
so
right
now
what
we
would
need.
It
seems
that
we
have
a
issue
with
the
environment
variables
that
we
are
setting
within
the
container.
E
E
E
So
what
we're
going
to
search
for
is
the
related
deployment
for
this
airline
application,
so
we're
going
to
go
ahead
and
drill
down
into
that
and
if
we
were
to
scroll
down
we're
going
to
take
a
look
at
the
environment
variables
that
are
set
for
the
container,
so
we
can
see
that
we
have
our
postgresql
user,
our
pass
and
our
database,
so
we
see
that
we
do
have
them
set,
but
there's
still
an
issue
within
the
environment.
So
let's
go
ahead
and
take
a
look
at
one
of
the
airline
applications.
That's
actually
working
and
remember.
E
E
E
So
what
I
want
to
go
to
next
is
to
search
for
the
related
deployment
and
we're
going
to
focus
on
the
one
that's
set
for
the
manage
one
live
cluster,
just
to
do
a
quick
comparison,
so
I'll
open
that
up
within
a
separate
tab
and
I'm
going
to
open
the
manage
to
live
cluster
yeah
I'm
over
the
deployment.
So
we
can
go
ahead
and
see
that
these
are
the
environment
variables
that
we
were
looking
at
before
and
we're
going
to
go
ahead
and
compare
those
to
the
one
that's
within
our
working
deployment.
E
So
here
we
can
see.
We
have
postgresql
user
password
and
database,
so
the
issue
that's
occurring
here
is
that
we
are
using
the
wrong
environment
variable
name,
so
there
will
always
be
an
error.
That's
throwing
or
that's
going
to
get
thrown
for
that
appointment.
So
let's
go
ahead
and
edit
this
appointment
and
we're
going
to
go
ahead
and
save
those
changes
and
we'll
go
ahead
and
scroll
back
down
just
to
verify
that
this
has
been
updated
and
as
you
can
see
that
this
is
updated
now.
E
So
let's
go
back
to
our
search
page
and
we're
going
to
go
back
and
look
at
the
related
pods
and,
as
you
can
see,
our
managed
to
live
cluster
power
for
the
airline
application
is
now
running
and
just
to
double
check.
We
can
go
ahead
and
look
back
into
the
logs
where,
before
we
are
seeing
that
there
is
an
environment
variable
issue
and
now
we're
seeing
that
the
logs
are
not
showing
the
error.
E
B
C
B
B
Perfect
awesome,
this
is
awesome
and
you
know
this
crash
loop
back
off.
I
have
a
series,
I
I
have
a
fleet
of
clusters
and
somewhere
on
some
corner
of
earth
or
some
several
corners
of
our
things
are
crashing
and
we
are
doing
it
from
one
place.
We
are
finding
out,
okay,
not
this
time,
you
showed
a
log,
but
I
am
assuming
in
other
cases
where
it
might
not
be
in
logs
in
the
ammo,
we'll
capture
the
events
right
and
we
can
see.
B
C
B
D
A
B
B
C
It
depends
if
you
did
the
labels
right,
but
in
this
case
no,
like
let's
say
we
didn't
do
labels
right.
We
searched
for
just
the
multi-cluster
hub
is
what
installs
our
acm.
So
we
right,
we
see
the
cr.
We
see
all
the
nine
things
that
it
created
directly
expands
that
one
of
it
is.
I
chose
seven
subscriptions
which
that
will
lay.
B
B
A
So
I
have
alert
manager
set
up
so
alert
manager,
pings
me
and
yelps
and
says:
hey
your
cluster's
down,
but
now
I
want
to
see
what
the
difference
is,
and
I
had
two
terminal
windows
with
two
different
coupe
coupe
config
set
just
furiously,
trying
to
figure
out
what
was
different,
and
this
is
one
place
where
I
can
side
by
side
them
super
easily.
That
would
have
saved
some
time
and
the
other
important
question
is
so
that
so
you
pull
up
the
ammo
you
edit,
it
live.
A
E
Awesome
and
now
the
next
scenario
that
I
want
to
show
you
guys
is
something
that
jorge
also
referenced
to
earlier
with
just
in
regards
to
our
back.
So
if
a
user
say,
for
instance,
if
a
department
is
allowing
a
user
to
access
one
cluster
and
only
want
to
see
that
information
for
one
cluster
instead
of
viewing
the
resources
for
say
for
say,
for
instance,
are
managed,
one
live
cluster,
we
can
actually
restrict
them
to
only
see
data
from
the
cluster
that
their
role
is
being
assigned
to.
So
we
have
a
user.
A
E
D
E
E
Guys
here
and
let's
check
out
the
af
people,
airline
app
and,
as
you
can
see,
if
we
were
to
view
the
related
pods,
the
only
part
that's
disappointing
is
our
managed
to
live
cluster
pod.
So
before,
when
we
were
signed
in
as
the
coop
admin,
we
were
able
to
see
the
vocal
cluster
pods
along
with
the
manage
one
live
cluster
pods.
So
this
is
just
something
that
hcm
once
are
back
is
enabled
search,
will
filter
that
data
based
upon
the
user
role
and
the
data
will
be
returned
properly
back
to
the
user.
A
So
this
this
opens
up
that
possibility
not
just
for
our
back
boundaries
for
permissions,
but
maybe
you
have
you
know
dev
stage
and
prod,
and
you
need
a
developer
to
be
able
to
see
stage
and
dev,
but
you
don't
want
them
to
touch
prod.
You
can
give
them
the
same
surface,
that
in
in
the
same
interface
that
an
sre
is
going
to
see
to
interface.
This
is
really
good,
because
you
can
give
someone
I
I
can
teach
someone
how
to
use
what
I'm
familiar
with.
C
C
C
Okay,
so
yeah,
I
have
both
one
here,
so
I
can
use
a
graphql
api.
I
already
exposed
the
route.
It's
just
very
simple,
oc
create
route
that
we
have
documented
on
on
our
documents.
So
it's
just
one
command
and
after
that
you
will
need
to
get
the
wordpress
token,
which
I
already
set
up
on
this
postman.
But
from
there
now
you
can
write
your
graphql
queries
so
I'll
show
you
an
example
of
pretty
much
what
we
were
saying
on
the
ui.
C
C
I
get
my
list
of
spot,
here's
the
airline
application
and
I
went
the
query
now.
I
requested
the
items
and
I
also
requested
the
relationship
just
like
we
showed
in
the
ui
and
the
cool
thing
with
graphql
is
that
you
can
decide
what
you
request.
So
that's
the
big
benefit.
So,
on
the
query
itself,
you
can
decide
to
just
run
the
query
and
get
the
item
so
just
from
the
query
and
get
the
relationships,
we
have
ways
to
filter
the
relationship.
C
A
A
A
So,
instead
of
doing
that,
I
could
just
point
it
to
the
search
host
and
then
request
it
all
from
that
one
source
make
one
request
with
one
boundary
of
exposure,
for
you
know
one
key
to
rotate
if
something
gets
leaked
rather
than
having
a
key
to
nine
different
clusters
and
get
all
that
data
from
one
place.
That
is
truly
wonderful.
D
C
C
A
Protect
that
makes
sense.
Yeah.
That's
that's
wonderful.
My
my
next
obvious
question
is
so
you've
shown
us
what
what
only
strikes
me
as
a
huge
amount
of
work
in
progress,
especially
with
how
robust
the
ui
and
the
api
are
with
graphql
and
the
performance
is
wild.
The
two
ones
are
where:
how
do
we
get
here?
Where?
How
long
has
this
project
been
going?
Where
can
people
find
this
project
in
the
open
source
space?
If
someone
wants
to
look
at,
you
know
peep
at
the
code
and
see
how
it
works.
D
I
think
I
have
my
screen
up,
so
it's
much
more
easier,
so
it's
there
on
the
link
that
we
posted.
So
we
got
everything
in
this
link
up
above,
that's
all
for
them.
The
message
that
we
posted,
where
we
have
documented
out
the
current
architecture
and
also
the
repositories
that
are
in
a
part
of
this
entire
project.
So
you
are
welcome
to
contribute,
make
use
of
it.
D
D
So
we
have
a
collector
process
that
runs
the
collector,
which
is
responsible
for
picking
up
all
the
resources
in
the
manus
cluster
and
then
once
we
picked
up,
we
use
the
rest,
api
and,
and
that
is
running
in
the
hub
cluster
right,
so
the
hub
hosts
all
the
indexing
process.
The
managed
closest
job
is
to
just
send
the
data
to
the
hub.
D
So
once
the
hub
gets
the
data,
its
job
is
to
index
it
and
then
store
synthetic
in
a
data
store,
as
we
just
saw
that
the
apb
exports
an
api
graphql
based
api,
where
the
clients
can
you
know,
make
use
of
the
api
and
then
render
it
to
their
ui
as
they
make
make
an
access
to
an
ai
com,
processor
and
whatnot
right.
So
that's
overall
flow
of
things.
But
if
you
want
to
go
through
the
details-
and
you
can
go
through
that.
A
Awesome,
that's.
That
is
amazing,
the
other.
The
other
important
question
is
we
talked
about
scale.
A
lot
jorge
t
is
something
about
hundreds
of
clusters.
What
what
does
the
scale
look
like
for
search?
What
you
showed
us
today?
How
many
clusters
can
I
throw
at
this
collector
before
things
start
slowing
down
yeah.
C
C
We
grew
to
a
thousand
now
we're
targeting
like
2
000
managed
clusters,
but
now
it
it's
ever
growing.
You
know
it's
not
just
the
amount
of
match
clusters.
It's
the
amount
of
data
in
those
managed
clusters
keep
growing
with
every
new
release
of
kubernetes
of
openshift.
So
it's
definitely
a
growing
target
and
we
are
trying
to
get
ahead
of
it.
We
have
done
major
re-architecture
to
go
to
a
more
performance
database
and
bring
things
more
to
the
scale
that's
needed.
C
C
A
B
C
Yeah
besides
scale
that
that
will
always
be
there.
The
other,
bigger
thing
that
we
have
is
not
like.
We
currently
have
the
current
state
of
every
cluster,
but
like
everything,
is
flowing
through
our
search
collectors
and
search
database.
So
why
not
keep
a
history
of
that
like
that?
We
think
that
that
will
help
a
lot.
This
is
most
times
you
have
a
problem,
it's
because
something
changed
and
we
have
visibility
into
that.
We
just
don't
exploit
it.
C
Yet
that's
the
next
thing
that
we're
looking
to
exploit
now
like
how
do
we
keep
a
history
of
your
resources,
what
change
and
when
and
we
can
use
our
search
database
to
to
answer
questions
such
as
well,
these
spots
started
failing
one
of
the
related
resources
to
it
changed
recently,
so
maybe
we
will
find
that
a
confit
map
recently
changed
and
the
body
is
picking
up
a
different
configuration
and
that's
causing
it
to
fail.
That's
something!
C
That's
not
as
easy
to
see
just
looking
at
your
resources
to
debug
that
you
need
to
go
through
different
things
and
figure
out
all
the
different
resources,
but
since
we
have
that
connection,
since
we
know
when
things
change,
we
can
start
becoming
smarter
later.
You
know
even
not
in
analytics
and
stuff
that
let's
say
well,
this
will
change
start
failing
around
this
time.
We
can
go
what
else
change
around
that
time
and
help
you
with
the
with
the
actual
process
of
debugging
and
fixing
your
problem.
A
Yeah,
I
can
imagine
getting
it's
a
dream
to
get
an
alert
that
tells
you
what
changed
to
not
have
to
search
for
what
changed
and
say.
This
is
what
it
was
before.
I
found
it
in
in
in
our
get
out
sweep
of
exactly
what
changed,
but
to
have
an
alert
fire
and
say
this
pod
is
in
a
crash
loop
back
off
its
deployment
changed
three
minutes
ago
from
this
to
this,
and
now
the
problem
isn't
what
happened
and
what
changed
it's?
How
did
it
change
and
how
do
I
fix
it?
It
gets.
A
D
One
more
thing
right
so
as
we
are
showing
this
picture,
so
one
of
the
things
that
we
are
exploring
is
that,
right
now
we
are
only
looking
at
the
communities
resources.
We
are
not
tapped
into
the
events
that
are
generated
by
the
kubernetes,
so
that's
something
that
we
are
exploring.
If
you
can
bring
values
by
you
know
watching
the
events
and
then
that
can
help
us
in
terms
of
our
problem.
Determination.
B
A
Yeah-
and
I
I'm
guessing
that
feedback
is
more
than
welcome,
I
I
will
toss
up.
We
have
an
email
address
for
reaching
out
after
the
show.
If
anyone
sees
this
and
has
any
ideas,
I'm
I'm
can
pass
it
on
to
jorge
and
xavier
and
isaiah,
because
I'm
I'm
sure
they're
happy
to
hear
how
you
might
use
this.
This
is
incredible.
A
A
A
Let
me
find
it
in
this
nice
schedule.
It's
the
13th
today,
next
week,
next
stream,
we
should
be
talking
about
micro
shift,
we'll
be
looping
in
there
with
that
team,
and
I
feel
like
that
might
be
an
interesting
synergy
with
today's
presentation
because,
as
jorge
just
said,
their
target
is
eventually
going
to
be
what
seems
to
be
collecting
search
data
from
the
cell
towers
that
your
you
know,
provider
is
running
and
that's
exactly
what
microshift
is
here
to
do
a
lot
of
those
far
edge
scenarios.