►
From YouTube: CDS Reef: Orchestrator
Description
Apologies for the legibility of this recording.
The Ceph Developer Summit for Reef is a series of planning meetings around the next release and some community planning.
Schedule: https://ceph.io/en/news/blog/2022/ceph-developer-summit-reef/
A
This
is
the
brief
planning
the
orchestrator.
We
have
our
initial
topics
here
in
this
other
pattern,.
A
So
the
topics
that
are
on
there
right
now
are
the
things
that
the
team
sort
of
come
up
with
we're
talking
last
weekly
about
some
things
we
knew
we
wanted
to
get
in.
So
we
have
those
as
sort
of
starting
points,
we'll
discuss
those
ones
and
then,
if
anyone
has
any
other
topics
that
want
us
to
consider
trying
to
get
in
for
for
our
we'll
do
that.
B
A
B
A
Beautiful
yeah
I
can,
I
can
even
actually
link
in
the
other
pad,
and
I
think
about
it.
The
pull
requests
go,
find
it
real,
quick,
okay,
yeah.
So
it
was
mostly
done
and
then
what
ends
up
happening
is
we
got
too
close
to
quincy
release
and
we
didn't
think
we're
gonna
get
it
in
for
the
release
and
it
seemed
like
too
big
of
a
change
to
be
introducing
in
a
minor
release,
and
I
got
pushed
back.
A
All
right,
yeah
I'll,
share
my
screen,
so
we
can
just
look
at
the
there.
A
Yeah
so
we'll
start
with
going
through
this
list
and
then
any
other
things
want
to
bring
up
for
what
we
should
do
for
reef
we'll
do
that,
after
starting
with
the
agent
stabilization
stuff
so
overview
that
people
who
don't
know
what
that
is,
the
agent
is
essentially
a
non-containerized
demon
that
would
be
running
on
on
each
host.
That
fadium
would
automatically
deploy.
A
If
you
like
a
naval
setting
and
essentially
the
point,
is
that
it
speeds
up
our
our
metric
gathering
so
like
they
developed
the
demons
from
the
hosts
the
devices
all
those
things
we
have
to
gather
those
and
right
now.
The
way
it
works
is
part
of
our
service
loop,
every
single,
like
10
minutes
or
so
it'll
go
and
we'll
refresh
these
things.
A
So
that's
the
sh
into
the
house
and
gather
them,
and
it's
pretty
slow
once
you
get
to
a
large
number
of
hosts,
especially
because
just
in
general
gathering
the
state
of
all
the
demons
can
be
slow
and
the
set
volume
commands
we
have
to
run
like
get
all
the
devices
is
slow.
A
We
have
this
whole
agent
feature
where
it's
it'll
gather
it
for
us
and
it'll.
Send
it
over
http
to
the
manager,
and
so
we
don't
have
to
spend
so
much
time
in
the
server
loop
doing
that
and
it
seems
to
make
it
so
we
can
get
those
metrics
more
frequently
and
it
makes
the
server
run
faster,
so
we
can
get
to
the
actual
operations
quicker
instead
of
having
to
spend
so
much
time
refreshing
all
of
that
information.
A
A
I
think
it's
if
the
host
comes
offline
and
goes
back
up,
there
is
some
issue
where
the
manager
would
report
the
agent
is
down
when
it
wasn't
really
down
just
general,
a
bunch
of
smaller
things
that
have
to
be
sort
of
worked
into
it.
A
We
have
to
handle.
So
we
want
to
do
that
stuff,
then,
for
the
additional
features,
I
think
we
want
to
do
stuff,
like
you
know,
deploy
demons
with
it,
because
it
has
a
copy
of
this
fading
binary,
and
so
theoretically,
it
should
be
able
to
do
all
the
sort
of
actions
we
would
do
on
the
host
normally
like
deploying
demons
and
gathering
information
and
all
that
stuff.
A
A
That
was
just
one
of
the
big
things
you
wanted
to
do
for
r
is
try
to
actually
incorporate
this
a
bit
more
and
get
it
properly
tested
and
everything,
because
right
now,
it's
sort
of
experimental
and
we
just
have
it
disabled
by
default,
and
we
do
have
some
tests
for
tautology
at
least,
but
it
needs
to
be
flushed
out.
It
also
needs
to
be
documented.
There's
a
there's,
no
documentation
either.
D
Yeah,
so
that's
one
thing
we
could
possibly
add
to
the
list.
If
you
want
to
just
to
containerize
it,
it
would
probably
need
to
be
a
privileged
container,
but
once
your
privilege
container
can
always
reach
outside
of
its
container
context
and
it
might
simplify
the
packaging
and
delivery.
D
I'll
admit:
I'mma
containerize,
all
the
things
sort
of
person,
but
yeah.
We
can
discuss
it
as
a
individual
topic
later,
all
right
yeah.
It's
obviously.
A
We
can
at
least
come
up
or
talk
about
the
idea
that
once
these
other
sort
of
things
about
it
are
are
set
up
properly,
it
could
be
a
container
that
could
work.
So
those
are
the
main
things
for
that.
This
one
something
we
were
sort
of
planning
to
do.
We
wanted
to
actually
do
originally
as
part
of
quincy.
A
A
C
A
Oh
yeah
yeah
for
the
detecting
offline
host.
C
A
Yeah,
because
right
now
we're
when
the
agent
was
on
the
one
we
were
taking
offline
hosts
with
it
was
they
say,
just
didn't
report.
We
would
have
that
in
there,
which
is,
I
guess,
sort
of
similar
to
a
heartbeat
and
we
wanted
a
more
explicit
heartbeat.
So
there's
a
normal
reporting
and
there's
a
heartbeat
that
should
come
in
no
matter
what,
if
we
don't
get
that,
then
we
know
that
it's
offline,
so
we
want
to
do
something
like
that
as
well.
That
is
some
point
here.
A
All
right
that
was
all
I
had
for
this
topic
didn't
have
anything
else.
I
want
to
talk
about
with
the
the
agent
for
now.
A
No
all
right
I'll
move
on
to
the
next
thing
we
have
on
here,
which
is
the
serve
loop
transparency,
so
right
now
the
way
that
a
lot
of
the
seth
adm
sort
of
notifications
work
or
when
something
goes
wrong.
Is
it
has
these
events,
because
there's
about
the
more
this
topic
than
the
other
one,
but
they
go
together,
there's
demon
events
and
there's
service
events
and
then
there's
obviously
there's
just
the
random
logging
that
the
server
loop
does.
A
We
found
that
all
these
things
are
sort
of
hard
for
people
to
access
they're,
not
really
like
intuitive,
like
something
goes
wrong
in
the
server
it
might
log
an
event
or
like
demon
manager
whatever,
and
you
specifically
go
in.
You
run
something
like.
A
Something
like
that
and
then
it
would
have,
and
if
there
were
any
events
for
that
demon,
they
hear
they
went
wrong.
They
would
show
up
there,
but
we
found
people
like
don't
aren't
going
to
check
that
by
default,
especially
because
there's
a
lot
of
services
a
lot
of
demons.
A
So,
even
though
we
have
this
information
stored
somewhere
hard
to
find
currently,
and
so
we
want
to
have
better
ways
of
explaining
this
sort
of
information
with
people,
so
on
top
of
just
the
events
for
the
demons
and
the
services,
also
just
in
general,
what
the
servlet
is
doing
so
say
like
we
were
talking
earlier
about
how
we
have
to
like
refresh
all
the
demons
and
we
have
to
professional
devices
and
things.
And
then
we,
after
that,
we
can
go,
do
like
x,
y
and
z,
really
what
those
things
are.
A
What
we're
doing
in
the
background
should
there
should
be
some
way
for
the
user
to
see
what
that
is.
Look
it
up
for
a
lot
of
the
debugging
cases.
A
So
like
say,
if
you
had
something
that
was
hanging
like
we
had
that
issue
before,
where
there
was
some
stuff
volume
command
hanging
that
we
were
trying
to
run
at
least
be
able
to
see
like
oh,
this
is
what
it's
trying
to
do
right
now.
This
must
be
hanging
somewhere,
make
the
debugging
a
bit
easier
and
also
some
sort
of
history
to
these
things.
A
So
you
could
just
see
these
are
the
actions
that
it
ran,
saying
the
last
serve
loop
when
I
ran
through
they
did
these
things
that
could
be
useful
for
debugging,
because
then
you
could
see
if
something
that
you
wanted
to
happen
isn't
happening.
Then
this
you
know
why.
A
So
you
want
to
have
stuff
like
that.
I'm
thinking
some
cli
commands
that
give
you
again
the
history
of
what
server
was
doing
recently,
just
like
short
snippets,
maybe
something
to
say
what
we're
doing
currently
stuff
like
that
and
then
some
way
to
make
the
demon
the
service
events
more
visible.
A
Maybe
we
just
need
some
like
a
cli
command
just
for
viewing
events
in
general
and
we'll
just
list
out
all
the
known
events
we
have
recently.
So
we
see
all
the
errors
that
have
happened
for
all
the
demons
and
then
you
can
sort
through
that
to
find
what
you're,
looking
for,
rather
than
having
to
specifically
go
look
for
a
certain
demon
in
the
check.
If
that
demon
has
an
event,
it's
just
easier
just
to
look
through
the
existing
events
that
happen.
A
Yes,
that's
one
of
the
big
topics.
I
guess
these
two
points
sort
of
go
together
is
we
want
to
make
the
debugging
and
everything
easier.
A
We
found
that
people
have
some
trouble
finding
some
of
the
problems
when
things
do
go
wrong.
F
All
right
does
that
have
any
yeah:
where
do
the
self-loop
logs?
Where
are
they?
Why?
Why
can
I
currently
find
them?
Are
they
in
the
self
manager
logs.
A
Yeah,
I
think
they
all
go
under
the
stuff
manager
logs
currently
normally,
when
I
look
at
them
I'll,
usually
watch
them
live
or
I'll
just
do
like.
If
you
do
like
a.
A
Glass
I'll
do
something
like
this:
there's
a
command
and
that'll
just
show
the
last
like
it
doesn't
have
to
do
too
much.
I
think,
there's
a
cap
on
it,
but
it'll
show
the
last
x
number
of
debug
logs
this
f50m
spit
out,
which
will
usually
be
all
the
stuff
from
the
serve
loop.
You
can
see
them
there
and
then
also.
A
I
think
they
go
into
the
manager
they're,
just
in
general
like
what
I'm
talking
about
here,
though,
obviously
the
logs
will
still
exist,
but
we
want
to
have
something
a
bit
more
concise,
it's
easier
to
look
at
and
see
like
what
the
circle
is
doing
in
general,
so
like
the
debug
logs,
if
you
want
like
a
more
detailed
view
of
what's
going
on,
but
just
I
think
that
overview
would
be
super
helpful.
A
So
maybe
we
even
want
like
an
easier
command
to
get
our
last
number
of
debug
vlogs.
That
was
possible.
We
could
hold
a
certain
number
of
them
good
people
as
well.
F
A
Yeah,
I
think
stuff
item
does
have
its
own
log
level,
but
again,
I
think
the
problem
isn't
necessarily
setting
the
log
level.
Is
just
people
don't
know
where
to
find
our
logs?
We
had
to
do
something
to
make
them
a
bit
more
transparent.
F
A
Yeah,
I
think
there
is
like
a
config
setting.
I
don't
remember
exactly
my
head,
but
it's
like
versus
f80m
slash
what
it
is.
A
And
the
energy
might
be
fall
under
just
the
generic
manager
log
level,
that's
what
I
was
thinking
of
yeah,
so
I
might
not
just
be
part
of
one
of
the
other
ones,
but
we
should
probably
maybe
even
consider
adding
one
of
those
just
specifically
for
cepheidium.
If
that's
something
that
actually
have
is
a
new
thing
to
do,.
A
But
yeah
we'll
add
that
to
the
list
of
things
we
want
to
do
when
making
this
more
visible.
I
have
some
way
to
more
customize
specifically
set
videom's
logs,
as
you
set
the
level
easier
and
you
can
find
them
a
bit
easier
as
well.
A
A
Hey,
I
didn't
have
anything
else
on
us
or,
I
guess
yeah.
I
want
to
open
the
floor
now.
If
anyone
has
anything
you
want
to
say
about
the
the
logging
or
the
transparency
in
the
server
loop
or
anything,
maybe
something
they've
run
into
that's
a
bit
hard
to
find
or
any
ideas
for
how
we
do
the
debugging
or
anything
like
that.
F
Yeah
with
respect
to
our
service
events,
I
mean
we
can
filter
by
our
service
name
and
service
type.
I
do
that
and
I
try
to
figure
out
more
logs
for
nfs
servers,
for
example,
and
maybe
we
need
better
documentation
upstream
for
the
conflict
settings
that
you
mentioned
adam
or
maybe
it's
already.
There.
A
F
F
Yeah,
we
need
to
make
sure
that
each
service-
you
know
you
mentioned
that
in
each
one
of
those
servers
that
you
can
use
these
safar
commands
and
filter
by
service
name
and
service
type.
Yeah.
A
Yeah,
I
know
I
don't
know
I
think
they're
in
the
troubleshooting
section
somewhere,
but
I
think
they
should
be
a
bit
more
visible
as
well.
I
think
that's
probably
part
of
the
problem
why
people
seem
to
not
find
these
is
they're
not
documented
in
like
some
more
forward-facing
spots,
where
they
probably
should
be
because
they're
kind
of
important
we
should
do
that
as
well.
I
still
add
that
points
here.
A
A
Another
movement-
this
is
a
another
topic
on
here-
that's
actually
pretty
much
in
the
same
vein
of
transparency
and
everything
I
hit
the
wrong
button.
Sorry,
I
have
fires
the
previous.
B
One
I
put
my
video
on,
I
said
it
on
muting.
Just
a
curiosity
question:
is
there
a
way
to
make
any
of
the
you
know,
make
everything
that
comes
back
sort
of
actionable.
You
know
try
to
try
to
bring
it
down
into
like
you
know,
go
do
this
to
take
care
of
the
problem
versus
you
know,
you
know
just
leaving
it
sort
of
opaque.
You
know
what
I
mean.
Make
everything
try
to
try
to
do
it
in
such
a
way
that
everything
has
an
action.
You
know
this
happened.
B
A
I
mean
in
theory
as
long
as
we,
I
guess
the
hard
part
would
be
knowing
what
to
do
in
that
situation,
but
we
definitely
if
we
didn't
know
what
to
do,
give
a
recommendation,
because
the
way
we
do
it
works
right
now
is
we
just
sort
of
log
an
event,
and
so
we
could
also
just
easily
log
some
other
like
about
the
event
there.
B
Yeah,
because,
typically
in
the
past
I
mean
to
make
a
you
know
product,
you
know
easy
to
use
and
usable
I
mean
you've
really
got
to
try
to
get
most
things
actionable,
and
you
know
if
you
fall
into
like
a
one
percent
category
of
things,
that
you
know
you
just
throw
your
hands
up
and
call
support.
I
mean
that's,
not
bad,
I
mean
you
know
it's
better
than
probably
where
we're
at
today
a
lot
of
our
actions
today,
a
lot
of
the
events
that
occur
just
like
you
scratch
your
head.
B
E
You
enter
some
command
and
you
get
an
error
code
and
it
comes
back
and
it
says
you
need
to
do
x
before
you
do
y
or
something
like
that.
Is
that
what
you're
talking
about
yeah
yeah,
you
know
a
good
example
to
me
to
that
is
if
you
use
command
line,
get
git
has
a
habit
of
telling
you
exactly
what
you
need
to
do
when
you
do
something
wrong.
E
I
would
have
to
agree
with
that
the,
but
the
problem
with
then
goes
being
sure.
You
know
what
the
user
is
trying
to
do.
Yeah
yeah,
and
that
is
the
problem
where
you
know,
there's
an
error,
but
you
don't
know
what
the
user
is
trying
to
do.
Unless
you
came
back
and
said
you
either
do
a
b
or
c
yeah
yeah.
Something
like
that.
I
don't
know,
but
I
do
agree
that
some
helpful
hints
on
air
would
be
nice.
F
I
think
one
of
the
complexities
is
that
some
of
the
commands
to
the
deployment
of
daemons
asynchronously,
for
example,
when
you
create
an
fs
cluster
or
you-
create
a
cfs
volume,
the
the
command
returns,
but
it
shoots
off
asynchronous
deployments.
F
A
Yeah,
that
was
one
of
the
things
I
kind
of
wanted
with
this
point
with
the
history
and
everything.
Maybe
we
could
like
log,
not
log
but
like
something
more
easy
to
access
than
logs,
like
that.
We
just
actually
tried
to
deploy
this
demon
like
this
many
seconds
ago
or
whatever.
D
So
one
thing
I
would
recommend-
maybe
not
copying
but
kind
of
taking
for
inspiration
when
I
debug
problems
with
kubernetes
the
cuddle
describe
command
is
extremely
helpful.
It
brings
a
lot
of
useful
information
about
the
state
of
a
resource
in
one
place,
so
that
might
solve
the
problem
that
you
were
just
bringing
up
there,
which
is
like.
What's
the
state
of
this
asynchronous
thing,.
A
Yeah
we
can
take
a
look
at
that.
I
don't
know.
A
B
But
if
you
do
have
a
context
of
something
at
a
you
know
a
larger
level
that
you're
trying
to
accomplish
and
it
fires
off
a
whole
bunch
of
asynchronous
events.
I
mean
somehow
you
do
have
to
keep
track
of
that
know
the
history
and
you
know
at
the
end
of
the
day,
if
that
larger,
you
know
context
fails,
then
you
know
because
of
one
of
these
other
asynchronous
events
you
did,
then
I
would
think
that
somehow
you've
got
to
make
that
you
know
connection.
I
mean
again.
B
If
that's
all
part
of
the
conversation
I
agree
100
I
mean
you
know
you
absolutely
have
to
be
able
to
do
that
and
hopefully
make
sense
of
it,
and
then
from
that
you
know
what
is
your?
What's
your
course
of
action.
As
a
result,
I
mean,
depending
on
maybe
availability
model.
Maybe
there's
absolutely
nothing.
You
have
to
do
let
the
system
recover
eventually
or
if,
if
the
service
didn't
this
thing,
you
tried
to
do
a
larger
context
doesn't
work.
Then
you
have
to
you
know.
B
A
Yeah,
I
think
all
these
things
sort
of
tie
together
where
we
need
to
be
able
to
describe
the
asynchronous
service
or,
what's
going
on
there,
we
need
to
be
able
to
properly
display
errors
when
things
do
go
wrong,
and
then
we
want
to,
if
possible,
give
some
recommendations
for
how
to
fix
it
and
sort
of
do
all
those
things
together
and
improve
the
ux
a
lot,
because
you
could
see
like
the
history
of
what
happened.
C
Do
we
have
somehow
something
to
get
the
history
of
comments
that
were
shoot
in
the
sephie
dm.
A
A
It
would,
I
guess,
as
long
as
they're
only
orchestrator
commands
like
we
can't
it's
kind
of
hard
to
do
it
for
and
it's
not
orchestrator
command
like
they
just
change
the
log
level
or
just
do
some
like
random
radios
gateway
admin
thing.
We
can't
really
see
that,
but
we
could
try
to
track
which
orange
commands
they
ran.
If
we
thought
that
was
like
a
thing,
people
would
want
know
what
they
ran.
I
guess
it
could
be
useful
actually
if
they
are
need
debugging
and
we
want
to
see
what
they
did
yeah
exactly.
D
Rather
than
the
commands,
maybe
log
state
changes
like
up
to
some
maybe
limit,
because
you
don't
necessarily
want
to
capture
everything,
but
I
don't
think
you
want
to
log
random,
say:
read-only
commands.
I've
used
a
system
that
did
that
in
the
past,
and
it
was
just
noise.
A
Yeah,
I
think,
there's
already
for
our
decorators
for
the
clx.
I
think
there
already
is
like
a
read
versus
the
right
one.
I
don't
know
if
they've
all
been
applied
correctly,
but
there
is
something
in
place
for
like
differentiating
them.
A
So
maybe
we
could
at
least
look
there
see
that
on
the
our
right
ones,
particular
we
could
maybe
log
those
somewhere.
So
we
know
what
people
are
doing.
They
could
share.
I
Generics
of
audit
log
that
exists
for
general
stuff
commands,
I'm
not
sure
if
the
zfdm
commands
are
showing
up
there
as
well,
but
that
would
be
a
good
place
to
include
if
it's
not,
if
they're,
not
there
already.
A
All
right,
yeah,
that's
a
little
less.
A
B
C
B
B
Is
that
something
that
would
be
helpful
or
is
that
something
that's
just
a
little
little
much.
Are
you
playing
like
all
the
commands?
They
ran,
not
all
necessarily,
but
a
group
of
you
know.
I
mean
like
a
almost
like
a
history
kind
of
you
know.
You
know
click
kind
of
thing.
I
don't
know
I
just.
Is
it
the
right
level
to
do
it?
Do
you
do
it
at
a
higher
level?
I
don't
know
I'm
just
bringing
that
up.
Is
that
how
is
it
helpful
at
the
at
at
this
level?
C
A
A
Yeah,
maybe
at
least
stephanie
might
not
be
the
place
you
want
to
do
that
he's
like
we
could
get
like
the
history
of
the
commands
really
easily
there.
Then
maybe
people
could
use
it
for
making
their
automation
or
for
doing
the
recreation.
I
don't
know
if
you
want
to
implement
something
that
actually
that
videom
does
the
recreates
the
commands
or
whatever.
C
A
Right
well,
yeah.
That
was
a
lot
of
good
stuff.
I'm
gonna
move
forward
with
that
yeah.
I
mentioned
a
look
at
more
of
this
generic
logging
stuff
and
make
sure
this
is
fdm.
Related
stuff
is
all
in
there.
I'll
have
to
give
that
a
closer
look.
A
If
we
could
get
some
way
to
get
those
out
and
just
use
them
at
least
get
all
those
fdm
related
ones.
Maybe
we
can
store
them
somewhere
temporarily
or
whatever
just
so.
We
can
have
an
easier
time
debugging
when
something
goes
wrong,
figure
out
what
people
are
up
to
I'll,
be
good.
A
All
right
don't
have
anything
else.
I
want
to
say
on
this
sort
of
debugging
stuff.
Basically,
this
whole
section
here
we
were
just
talking
about
where
to
view
failures.
What
should
we
be
tracking?
What
people
are
doing
and
things
like
that.
I
Maybe
out
of
the
loop
here
but
but
I
know
there
was
some
kind
of
events
framework
that
the
cypherdm
was
using
but
to
try
to,
but
these
are
now
like
what
had
happened
in
the
past.
That's
still
useful
for
failure
tracking
in
this
kind
of
sensor.
A
Yeah,
so
we
were
talking
about
a
little
bit
earlier.
It's
like
one
of
these
ones.
So,
basically,
when
you
look
at
like
an
rgps
or
hls
output,
and
if
you
as
long
as
you
give
it
a
format
that
isn't
like
the
plain
one,
if
that
event
or
if
that
demon
or
service
has
had
an
event
happen
like
something
has
gone
wrong
there,
it'll
it'll
show
up,
we
were
saying
earlier,
is
that
the
only
problem
with
that
is
people
tend
not
to
define
them
or
not?
Look
there
like.
A
It
might
just
be
like
too
hard
to
find,
and
also
just
like
the
level
of
granularity
where
you
have
to
specifically
go
out
of
your
way.
To
like
look,
I
want
to
check
if
something
happened
with
this
service
might
be
too
hard.
We
might
need
some
more
general
way
of
finding
all
these
events,
so
they
are
there,
but
it's
just
they
don't
seem
to
be
super
visible
right
now
we
wanted
to
increase
the
visibility
as
part
of
this
okay.
A
All
right,
let's
move
on
to
the
next
thing
we
have
on
here,
so
this
one
again,
it's
actually
just
in
the
same
vein
of
what
we
were
just
talking
about,
which
is
an
upgrade
history.
A
So
one
of
the
things
that
hasn't
been
great
with
the
way
the
upgrade
works
is
when
it
ends
the
whole,
like
upgrade
status,
that
you
can
kind
of
use
to
track,
how
it's
going
the
whole
time
just
goes
away,
and
it
says
it's
not
in
progress
anymore,
but
if
we
had
an
upgrade
history
sort
of
thing
going,
then
we
could,
as
you
after
you
see
an
upgrade,
no
longer
says
it's
in
progress
anymore.
Instead
of
just
hoping
that
it's
all
worked,
and
it's
all
good,
you
could
check
the
upgrade
history.
E
A
This
would
be
like
a
command
line
thing,
because
the
idea
is,
we
just
have
like
a
log
of
say,
like
something
pretty
quick
like
we
just
completed
a
upgrade
of
like
these
demons
to
this
version.
At
this
time
you
could
see
sort
of
what's
happened
recently,
yeah,
I'm
sure
this
stuff
could
be
found
in
the
logs
already.
A
But
again,
it's
just
super
hard
to
find
right
now,
so
we
want
to
have
a
command
line
to
just
say:
you'd
say
if
you've
done
multiple
upgrades
as
long
as
there's
not
like
too
many
of
them,
you
could
just
store
them
somewhere
and
that
way
you
could
see
what
upgrades
have
happened
in
the
cluster.
A
So
if
we're
just
trying
to
make
things
that
happen,
asynchronously
more
visible
because
upgrade
is
one
other
one
of
those
things
that
we
do
in
the
background
you
have
to
fire
it
off
and
then
it
just
goes
and
somebody's
nicer
just
have
some
better
tracking
of
what
happened
with
those
things,
and
there
is
some
work
already
going
on,
like
I
said,
it'll
end
up
in
r
at
some
point,
let's
plan
to
go
into
earlier
versions
as
well
to
make
the
upgrade
you
can
split
it
up.
A
But
even
though
it
doesn't,
it
doesn't
take
away
from
the
usefulness
of
having
a
history
like
this,
where
you
could
have
some
way
of
seeing
all
the
upgrades
that
happened.
So
you
know
like
when
this
was
upgraded
and
when
it
finished
and
everything
that
could
be
useful
for
people.
A
So
does
that
have
anything
you
want
to
say
about
that
upgrade
history?
What
could
be
useful
to
be
there?
What
things
should
be
tracked?
Anything
like
that.
A
Yeah,
that's
sort
of
the
idea
we
were
going
for
is
just
some
way
to
see
what
those
things
were.
J
A
Yeah
it'll
tell
you
what
type
of
demon
it's
upgrading,
but
it
won't
tell
you
which
one
in
particular
yep,
but
then
that's
what
I've
noticed.
We
could
even
add.
I
don't
think
I
have
a
section
in
here
or
just
the
upgrade
status.
A
A
C
A
Few,
it's
not
either
something
that's
going
to
happen
super
frequently.
Typically,
I
I
I
like.
E
A
I
mean
it
could
be
for
for
either
it
just
wasn't.
Even
very
generic.
E
Yeah,
I'm
more
apt
to
upgrade
within
a
you
know
within
let's
say
nautilus
or
something
like
that.
Then
I
am,
I
wouldn't
very
often
go.
I
mean
a
major
branch
jump.
I
mean
that
would
be
something
that
you.
I
would
think
that
would
be
every
couple
years,
but
within
the
branch
upgrades
are
more
common
when
bug
fixes
come
out
and
you
upgrade.
A
A
E
C
E
A
Yeah
I
mean,
I
don't
think
it's
gonna
be
some
like
very
much
information
it'll
probably
be
like
a
few
lines,
upgrade
I'm
not
sure
exactly
the
technical
side,
where
we're
gonna
put
it
in
everything
yet,
but
it
shouldn't
be
too
much.
So
I
don't
think
it
will
matter
where
it
is
really.
A
Love
to
see
yeah,
but
most
of
those
tests
about
here,
just
sort
of
what
needs
to
be
in
there.
How
about
like
how
far
back
you
want
to
go.
That's
another
thing,
because.
A
Is
it
useful
to
know
back
like
three
or
four
upgrades?
You
only
need
to
know
the
last
like
two
like
stuff,
like
that,
like
high
level
stuff.
A
Yeah
all
right
how
many
else
want
to
say
about
the
upgrade
history.
A
All
right-
and
we
just
talked
about
the
status
improvements,
so
we
go
to
the
binary
factoring.
I
mentioned
this
earlier.
There's
a
pull
request
here.
It's
linked.
Anyone
wants
to
look
at
it.
It's
been
outstanding
for
a
pretty
long
time,
now,
sort
of
what
happened
with
that
is.
A
It
was
almost
done
close
to
the
quincy
release,
but
we
sort
of
agreed
to
something
that
we
wouldn't
be
able
to
get
it
done
in
time
frequency
release
and
it
was
too
big
of
a
change
to
be
a
minor
release
thing
because
it
would
sort
of
change
how
you
get
access
to
the
binary
and
everything,
and
so
we
wanted
that
to
be
something.
A
That's
only
done
on
a
major
release,
so
we
sort
of
pushed
it
back,
and
so
basically,
what
it
allows
to
do
is
right
now
the
binaries
is
one
really
large
file
like
8
000
lines
or
so
right
now,
and
it
makes
it
hard
to
work
with
a
lot
of
the
time
it's
sort
of
stuck
in
that
form
for
sort
of
technical
reasons.
We
reuse
it
right
now,
but
the
work
in
in
this
pull
request
here
was
giving
us
a
way
to
split
it
up
and
still
make
use
of
it
properly.
D
A
D
A
A
Yeah
yeah,
so
the
idea
was
try
to
get
back
to
that
sort
of
late
in
the
year
and
see
if
we
can
get
it
really
ready
like
right
before
sort
of
r
comes
out
and
sort
of
get
that
in
and
then
do
some
refactoring
like
the
month
or
two
before
the
r
release,
and
then
we
can
have
our
sort
of
come
out
with
this
new
refactored
version
and,
however,
we're
going
to
distribute
that
and
everything
will
be
something
new
for
that
release
and
then,
like
q
and
an
earlier
we'll
just
use
the
old
sort
of
system
of
just
the
the
one
big
script
file.
A
That
was
the
idea.
There's
something
we've
been
trying
to
do
for
a
while.
Now
this
goes
back
to
I
think
when
pacific
was
the
newest,
I
don't
know
even
when
before
pacific
came
out,
I
think
we
already
saw
some
discussion
being
able
to
refactor
this
there's
just
something
that's
never
been
able
to
get
in
one
reason,
another
thing
and
it's
kind
of
hard,
because
it's
a
big
change
that
it
sort
of
needs
to
be
a
major
release.
Thing.
A
A
Don't
really
go
into
that
stuff
right
now.
That
is
something
we
definitely
plan
to
do
for
r,
so
they
might
have
anything
they
want
to
say
about
the
binary
factoring.
I
need
that
this
is
kind
of
like
a
random
topic.
That's
more
technical.
A
And
that
case
we
can
keep
going
here,
and
so
this
point
here
is
more
of
a
generic
points
about
just
continuing
to
sort
of
do
what
something
I'm
supposed
to
do,
which
is
simplify,
workflows,
make
them
a
bit
easier,
and
so
this
was
just
sort
of
a
starting
list
on
things
we
plan
to
do
that,
for
so
we
have
here
setting
the
monitoring
stack
images,
anyone
who's
changed
their
monitoring,
stack
images
by
default
or
from
the
defaults
sort
of.
A
I
know
this
process
fdm,
where
you
have
to
sort
of
change
the
config
option
and
then,
if
you
use
the
config
option,
you
have
to
then
go
like
redeploy
the
demon.
So
it's
not
like
a
super
complex
process
or
anything,
but
it's
one
of
those
things
that
it
would
be
pretty
straightforward
to
automate.
Just
have
one
command
said
like
I
want.
A
My
monitoring
stack
this
like
my
prometheus
demons
on
this
image,
then
we
should
just
be
able
to
handle
that
for
you
and
set
the
image
and
make
sure
the
demons
get
redeployed
and
all
that
so
there's
no
real
reason
for
us
to
force
you
to
do
all
these
extra
steps.
We
should
be
able
to
sort
of
automate
that
for
you,
so
that's
something
we
want
to
do.
A
One
of
those
processes
when
automate
and
rw
multi-site
stuff
is
something
that
almost
happened
a
long
time
ago,
and
then
it
didn't
happen,
something
we're
going
to
want
to
bring
back
up.
A
Hopefully
now
that
we
have
some
more
time
to
look
into
it,
just
having
some
way
to
make
the
deployment
of
rw
multi-site
easier,
fm
does
have
the
power
to
run
those
like
windows,
gateway,
admin,
commands
and
everything.
A
So
like
say,
all
those
steps
were
like
setting
up
the
realm
in
the
zones
and
all
that,
if
we
gave
it
with
a
good
format
for
like
a
proper
yaml,
you
could
provides
to
specify
what
everything
you
need
for
multi-sites,
theoretically
cfdm
code
sort
of
handle
that,
for
you,
make
sure
those
things
get
created
correctly
and
set
up
some
of
that
for
you.
A
So
that's
one
of
the
workflows
you
want
to
automate
we'll
probably
be
trying
to
work
with
the
rgw
team
and
get
some
of
that
stuff
in
as.
B
Well,
you
know
my
question:
why
cli
and
just
not
work
with
dashboard
and
put
it
in
dashboard,
add
value
there.
You
know
I
wouldn't
do
it
in
the
cli
personally,
but
I
know
this
I
I
know
that's
probably
not
a
popular
statement.
H
A
Yeah
I
mean
I
don't
want
to
do
this
as
like
an
alternative
to
the
dashboard
I
want
to.
I
was
honestly
if
it
worked
properly
in
the
cli,
then
the
dashboard
could
even
make
use
of
that
to
some
degree.
It's
not
like
it's
anything
we
do
in
this
fdm
can't
dashboard
can't
make
use
of
it
all.
So
I
feel
like
having
it
here
wouldn't
take
away
from
that,
but.
B
E
A
Getting
that
in
I
know
some
people
have
wanted
that
yeah
and
again
we
tried
to
do
it
before
and
we
hadn't
gotten
it
working.
I
think
it
got
removed
a
while
ago
and
then
even
again,
even
if
we
got
this
started
working
here,
that
sort
of
work
that
could
help
get
it
working
the
dashboard
as
well.
A
It's
not
like
it
would
only
help
percept
video
and
we
could
sort
of
either
they
can
make
use
of
it
directly
or
we
could
sort
of
I'll
be
sort
of
the
ideas
from
one
place
to
another
if
we
figure
out
a
good
format
for
doing
that
stuff.
A
So
I
think
maybe
there
could
even
be
some
collaboration
there
when
setting
this
stuff
up
to
make
sure
it's
working
in
both
places.
So
yeah
again
it's
something
we
want
to
do
so
I'll
open
it
up
again.
If
you
want
to
say
anything
about
either
these
workflows
any
comments
on
these
in
particular,
or
if
there's
anything
else,
you
think
that
it
would
be
nice
if
it
was
automated.
That's
not
currently
automated
something
like
that.
So
anyone
wants
to
say
that
stuff.
A
It
doesn't
sound
like
we
have
anything
there,
so
in
that
case
yeah
over
the
course
of
the
next
few
months
as
well.
If
anyone
comes
up
with
anything
and
reach
out
to
me
on,
like
the
rc
channel
or
anything
that
says,
focus
triggers
one
or
anywhere
else
and
pose
those
ideas
there,
and
we
can
see
if
we
can
look
into
adding
stuff
like
that.
A
A
Right-
and
so
the
last
point
we
have
on
here
so
far,
is
this
disconnected
environment
stuff,
so
we
have
some
basic
documentation
and
whatever
for
how
you
handle
disconnected
environments,
but
we
at
this
point
is
basically
just
that
we
want
to
give
this
a
bit
more
attention
over
the
our
release.
Do
a
bit
more
testing
around
it's
in
environments
and
all
that
I
know
we
have
some
documents
that
tell
you
you
should
accept
a
private
registry
and
whatever,
and
there
might
be
like
a
conflict
option
or
two.
A
We
tell
you
to
change,
but
there's
not
a
lot
of
like
testing
around
us,
and
I
I
think
maybe
we
could
there's.
Maybe
one
of
these
things
again
we
can
automate
certain
parts
of
it
and
we
have
on
here
is
the
server
sub
points.
This
came
up.
A
Didn't
really
want
to
do
that,
but
for
the
other
things
in
general,
making
sure
that
as
long
as
you
have
the
registry
set
up
that
have
a
discontent
environment
that
bill
is
built
on
top
of
that
private
registry
is,
is
a
smooth
process,
because
we
know
a
lot
of
people
use
these
disconnected
environments.
So
it
would
be
good
to
make
sure
all
that
stuff
works.
Well.
A
Yeah,
so
there's
not
a
lot
of
specifics
there
right
now,
it's
more
of
a
big
idea.
I
guess
what
you
want
to
do
so.
Does
anyone
have
any
comments
on
that
one?
They
want
to
say
disconnected
environments,
things
that
they've
found
that
are
difficult
with
cepheidium
dispensed
environments
or
things
that
can
be
improved
there.
They
have
used
them.
A
All
right,
it
doesn't
sound
like
it,
so
in
that
case
yeah
something
we're
doing
anyway.
So
if
you
guys,
you
do
use
protective
stuff,
you
look
out
for
that
stuff
in
our
hopefully
have
a
few
small
improvements
making
that
a
bit
smoother,
but
that
was
the
last
topic
we
had
come
up
with
so
far.
A
B
Totally,
okay,
just
just
an
overall
comment:
it's
it
this!
This
is
actually
an
interesting
list
because
it
looks
like
it's
more
about
stabilization.
There's
not
a
you
know,
not
nothing,
not
anything
major
new.
I
guess
I'd
say
it's
all
about.
You
know,
refinement
stabilization,
making
the
product
more
usable.
I
think
it's
all
good.
You
know
it's!
So
that's
I!
I
think
you
know
a
good
sign
of
where
we're
at
one
of
those
things.
I
guess
would
be
missing
in
my
mind
from
that
kind
of
stabilization.
B
You
know
we
don't
have
anything
about
performance
and
scale
here.
You
know
there's
work.
We
need
to
do
there
to
try
to
drive
that
to
a
different
level.
You
know
and
see
where
we
can
go.
What
happens
when
you
get
to
huge
note,
you
know,
note
counts
huge,
you
know,
drive
counts,
I
mean
whatever
I
mean
that
the
only
thing
is
it's
another
stabilization.
B
B
I
know
we
have
a
little
bit
of
the
overall
assessment
about
where
we're
at
I
mean,
like
you,
know,
there's
nothing
like
shockingly
major
new
year.
It's
like
you
know
it's
all
about
polishing
refinement.
You
know
making
it
more
a
more
stable,
better
product
I
mean.
So
I
guess
that's
that's
sort
of
the
interesting
point
as
well.
I
think.
H
E
Yeah
exactly
yeah,
I
will
admit
that
you
know
I'm
topped
out
right
and
I'm
I'm.
I'm
gonna
call
me
or
myself
a
more
small
guy,
because
I
got
popped
out
at
about
40
data
nodes,
but
certainly
some
sort
of
performance
metrics.
I
don't
know.
A
Yeah
how
the
trouble
has
been
recently
is:
we've
been
trying
to
do
some
testing
scheduling
and
you
need
like
the
hardware
to
do
stuff
like
that.
So
on
a
couple
occasions,
we've
been
able
to
get
some
good
hardware
for
that
stuff
that
there
was
the
posi
lab
got
access
to
for
a
while.
A
E
Real
world
clusters
to
test
with
is
yes,
that's
an
absolute
problem.
I
will
say
my
biggest
deployment
is
at
a
customer
site
and
I
can't
mess
with
the
customer
system.
E
It's
an
operational
system,
so
yeah
and
I
can't
afford
you
know
I'll,
say
to
duplicate
their
system.
A
Yeah,
but
that's
something
we
have
gotten
some
work
on
and
there
has
been
a
fourth
range
says
now,
there's
a
presentation
on
it.
I
don't
know
that
preservation
come
out
yeah,
it's
I
don't
remember,
there's
a
whole
set
of
slides
and
everything
on
our
scale
testing
and
all
this.
J
A
J
Yeah,
I
think
the
it
has
a
clear
section
for
what
we
are
done
with
gibba
pausi
and
the
other
stuff.
That
paul
also
did
it's
going
to
be
ready.
Once
quincy
release
goes
out.
A
Okay,
yeah,
so
there
I
guess
there
will
be
a
presentation,
I
guess
around
the
quincy
release
of
the
test.
We've
done
so
far
with
the
scale,
performance
and
sort
of
what
we're
able
to
achieve
currently
and
when
we
start
running
the
problems
and
those
things,
but
it
is
something
definitely
yeah
once
things
are
functionally
stable.
A
The
performance
at
large
scales
is
something
we
want
to
look
at
for
scifidm
and
the
problem
is
always
finding
consistent
ways
to
test
it
and
everything,
and
also,
I
think,
part
of
the
scale
performance
stuff
ends
up
being
not
stepping
specifically
but
just
the
manager
itself.
I
know
when
you
did.
The
policy
testing
sage
said
a
bit
of
refactoring
on
the
locks
lock
handling
in
in
the
manager,
so
maybe
some
changes
there
as
well.
A
If
you
want
to
prove
performance
at
that
point,
so
it's
all
things
left
to
start
looking
into
as
things
get
a
bit
more
stable,
yeah
that'll
be
good,
there's,
definitely
a
point:
we
should
start
looking
at
well.
If
we're
going
to
be
talking
about
stabilization
and
stuff,
then
yeah
scale
performance
is
something
that
should
become
a
big
topic.
A
All
right
does
that
mean
anything
else
yeah.
I
know
we're
sort
of
nearing
the
end
of
timeslot.
I
believe
so.
K
Yeah,
I
have
got
a
question
it's
about.
I
remember
we
talked
about
this
a
while
ago.
Is
there
any
update
on
the
scheduling
of
service
or
demon
scheduling
for
for
safety
right
now,
it's
kind
of
a
static
right.
So
I
remember
we
identified
some
piece
of
work
for
the
dashboard
too,
so
basically
to
somehow
balance
the
resources
or
the
diamonds
on
the
services
across
the
cluster
right.
K
Scheduling
yeah,
maybe
it
was
with
sebastian
yeah,
basically
right
now:
it's
we
have
the
placement
right,
which
is
kind
of
a
static.
You
can
still
use
the
count
and
the
labels,
which
yeah
ensure
some
kind
of
high
availability,
but
for
avoiding
hotspot
and
distributing
a
lot
of
of
the
different
services
across
the
cluster.
I
think
we
remember.
We
were
talking
about
some
kind
of
smart,
dynamic
scheduler
for
safety,
so
basically
based
on
cpu
and
memory,
it
would
try
to
balance
the
services.
A
A
A
Yeah,
basically,
the
idea
was
just
that
all
things
equal
like
if
there's
multiple
hosts
we
could
choose
from
where
to
put
something
we
would
actually
check
their
resources
and
everything
and
make
our
decisions
based
off
of
that
but
yeah,
I
don't
honestly.
I
don't
think
a
lot
of
work
has
happened
so
far.
Is
this
something
something
we
can
start
looking
into
for
r,
because
there
was
something
we
planned
some
time
ago
to
start
looking
at,
and
I
think
we
have
the
means
of
gathering
all
the
host
metrics.
A
Now
we
have
that
gather
facts
and
everything
that
we
didn't
have
like
a
year
ago.
That
could
all
be
used
to
make
these
sort
of
calls.
So
it's
definitely
something
that
we
can
start
looking
into
for
r
but
yeah.
If,
I'm
being
honest,
I
don't
think
much.
Work
has
happened
on
that
anytime
recently,
but
I
know.
A
All
right,
thanks
for
just
though
alright
somebody
have
anything
else
to
talk
about
last
minute
or
so.
A
All
right,
in
that
case
thanks
everybody
for
coming.
We
have
our
sort
of
a
good
list
of
topics
here.
I
think
sort
of
some
good
discussion
yeah.
So
that's
our
our
our
planning.
A
Yeah
thanks
everyone
for
coming
and
enjoy
your
day,
see
you
guys
later.