►
From YouTube: WG Data Protection Meeting 20200909
Description
No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).
A
All
right
today
is
wednesday
september
9th,
and
this
is
a
kubernetes
data
production
working
group
meeting.
Let
me
share
with
you
guys
my
screen.
This
web
client
is
so
hard
to
use.
A
A
Great,
the
couple
of
discussions
happened
in
the
past
two
weeks.
Some
of
them
I
was
part
of-
and
we
discussed,
the
major
thing
we
discussed
is
to
cover
the
existing
very
representative
representative
applications
and
how
they
conduct
snapshot
and
and
the
backup-
and
we
worked
around
some
of
those
applications.
Me
particularly
worked
on
google,
there,
the
kafka
one
and
they
are
mysql
and
the
others
today.
The
first
item
is
to
go
through
that.
What
do
we
have?
A
One
of
the
big
purposes
that
we
put
all
these
use
cases
into
our
kind
of
white
paper
and
try
to
provide
from
or
perspective
what
do
we
think
those
how
those
applications
should
be
working
and
that's
one
and
the
other
aspect
is
really
to
see
whether
the
up
the
container
notifier,
aka
application
execution
hook
can
support
those
scenarios.
A
So
that's
the
second
items
and
I
will
go
through
what
or
what
are
all
the
updates
on
the
container
notifier.
I
think
we
have
made
to
a
point
more
or
less.
We
can
we're
ready
for
finding
a
cap.
A
The
last
things
we
are
doing
at
this
moment
are
to
adjust
the
kept
document
slightly
and
then
once
we
have
that
done,
I
think
xin.
Our
plan
is
to
file
the
cap
as
soon
as
possible.
Right
and
then
we
will
open
the
discussion
to
the
group.
Any
one
of
you,
john
tom.
You
want
to
get
started.
C
Yeah,
let
me
let
me
grab
the
document
that
we
were
working
off
of
give
me
a
moment.
Sure.
C
C
C
C
So
here
can
everyone
see
my
screen?
Okay,
yep
great.
So
this
is
the
dock.
We're
talking
about
quiesce
hooks.
C
I
think
the
the
approach,
the
high
level
approach
that
we
wanted
to
take
was
to
look
at
different
applications
and
see
what
kind
of
hooks
that
they
require
to
take
application,
consistent,
backups
and
so
really.
Our
motivation
here
is
to
help
drive
the
primitives
we
need
in
kubernetes
and
specifically,
in
this
context,
the
container
notifier
api.
C
There
are
obviously
many
approaches
to
backup
applications,
so
we
decided
not
to
make
this
an
exhaustive
list
here
right.
I
think
there's
other
sections
of
the
main
white
paper
that
we'll
add
that
will
kind
of
go
into
that
little
more
detail.
There's
also
many
applications
that
provide
their
own
data
protection.
C
It
can
be
things
that
are
outside
of
kubernetes.
You
know
one
example
would
be.
You
know
cloud
data
services
like
like
rds,
for
example:
maybe
they
have
their
own
backup
mechanisms.
We
decided
to
not
include
that
in
this
section
specifically
and
is
very
application-specific
things.
So
an
example
here
would
be
kafka
which
shawn
and
I
looked
at
there.
C
The
other
thing
we
we,
I
think,
wanted
to
talk
about
was
the
scope
of
these
hooks.
So
we
wanted
to
figure
out.
You
know
what
what
kind
of
commands
we'd
actually
be
running
in
these
hooks.
You
can
imagine
very,
very
general
commands
you
know
doing
anything
from
issuing
arbitrary
apis
to
applications
or
you
can
make
them
very
specific.
You
know
you
can
have
very
specific
volume
freeze
on
freeze,
you
can
make
it
very
specific
to
an
application.
You
know
like
flush
tables
with
reedlock,
for
example.
C
We
kind
of
went
through
we'll
go
through
this
later,
but
we
figure
out
what
what
examples
we
wanted.
We
discussed
the
different
mechanisms,
so
this
is
kind
of
you
know.
I
think
more,
what's
well
outlined
in
the
container
notifier
design
dock
that
we
have
going
so
I'll
skip
over.
That
section.
C
We
did
discuss
execution
controls,
but
I
think
again
that
will
fall
into
the
domain
of
the
container
notifier
right.
If
we,
if
we
figure
out
some
applications,
we'll
need
specific
controls
around
retries
or
timeouts
or
those
kind
of
things
you
know,
I
think
it'll
have
to
be
part
of
the
container
notifier
api
as
well.
C
So
after
that,
we
really
just
categorize
the
different
types
of
of
databases,
and
then
we
kind
of
went
into
specifics
from
there,
and
so
I
think
we
we
had
different
sections.
So
you
know
we
can
start
with
kafka,
because
that's
the
one
that
sean
and
I
worked
on
and
then
we
can
jump
into
the
other
ones,
but
just
really
quickly.
At
a
high
level.
We
talk
about
relational
databases,
so
things
like
mysql
and
postgres.
We
talked
about
time
series
databases
so
in
flux,
db
from
etfs.
C
That
kind
of
thing-
and
I
think
we're
shown
to
look
at
that
one.
We
also
talked
about
key
value
stores
like
redis.
C
This
one
was
assigned
to
me,
but
I
didn't
get
to
that.
We
talked
about
kafka,
so
sean
and
I
like
that
one
distributed
database
like
mongodb,
so
it's
in
here
this
is
great.
Actually
I
realize
everyone
updated
this.
This
document
with
with
the
hooks
so
sean
and
I
worked
in
a
separate
document,
but
we'll
we'll
go
into
that.
C
So
with
that,
let's
go
into
the
specific
databases.
Maybe
each
person
can
talk.
I
can
present
still
if
that's
easier,
but
maybe
each
person
who
worked
on
each
section
can
talk
about
what
it
takes
to
back
them
up
and
we
can
talk
about
how
that
applies
to
the
various
hooks.
So
I
guess
fung.
Do
you
want
to
talk
about
my
sequel
here.
D
Yeah,
I
can
talk
briefly
about
my
sequel,
so
basically
the
the
workflow
is
almost
every
every
database
would
be
similar
in
term
that
first
you
have
to
quiet
the
database
operation
in
one
way
or
another.
Then
you
back
up
all
the
resistant
volume
being
used
by
that
application
in
the
approach
that
we
are
using.
We
just
take
a
snapshot
of
the
pvc,
but
some
other
application.
D
They
might
use
some
other
way
and
then
we
undergoes
right.
Those
are
the
general
steps
for
almost
all
the
database
that
we
interact
with
then,
but
mysql
will
have.
A
special
thing
is
that
we,
if
you
connect
it
to
a
mysql
server
and
run
a
lock
and
as
soon
as
you
disconnecting
from
that
the
lock
will
be
automatically
released,
so
you
have
to
do
a
little
bit
trick
there.
So
if
you
look
into
my
ps
from
queer
second
you're
gonna
see
that
I
propose
a
streak
there.
D
So
that
means
that
you,
instead
of
running
across
command
you
script
and
that
script
you're
going
to
be
pushing
into
a
background
with
the
sleep
so
that
it
will
keep
the
session
running.
So,
even
when
you,
the
container
notifier
command
exit,
the
skip
will
be
running
in
the
background
and
then
you
go
on
and
the
controller
will
go
on
and
take
a
snapshot
and
in
the
unquest,
which
is
another
script,
to
searching
for
the
the
script,
the
queer
script
and
kill
it.
D
That's
that
is
just
the
general
workflow
on
for
that
one
and
then,
after
you
know,
queers
and
queers,
you
go
to
a
phase
of
cleanup,
so
we
clean
up
the
you
also
because
we
is
using
container
notifier
in
this
case
right
so
for
each
of
the
queens
and
enquires.
We
issues
a
notification,
then,
in
the
cleanup
we
have
to
delete
these
notification
objects.
D
If,
in
case
of
the
right,
you
cannot
guarantee
that
everything
will
be
successful
in
case
of
the
failure
you
might
have
to
do
some
actually
an
additional
cleanup
like
if
you,
let's
just
say
you,
have
three
snapshots
and
only
two
of
them
success.
You
cannot
go
on
and
back
up
it
right,
so
you
have
to
somehow
go
back
and
clean
up
the
tool
that
you
already
create
so
that
it
will
not
occupy
the
take
over.
Take
the
resource
from
the
system,
so
that
is
with
my
sequel.
D
C
And
I
guess
what's
interesting
here
from
a
requirement
perspective,
is
that
you'll
have
to
run
the
the
hooks
in
parallel
with
the
snapshot
right.
D
Yeah
yeah,
that's
the
this
is
how
you
can
guarantee
up
consistent
for
my
sequel.
Otherwise
you
know
my
sql
operation
is
still
going
on
and
so
on
and
so
forth.
The
the
quiz
is
very
simple:
it's
a
flush
out
table
with
the
read
lock.
So
as
soon
as
you
lock
that,
then
you
know,
customer
user
can
continue
reading,
I
think
from
the
database
as
well,
but
they
were
not
able
to
write
to
it
so,
which
is
said
for
the
app
consistent
from
my
point
of
view.
C
C
With
the
container
notifier
proposal,
having
that
kind
of
run
in
parallel.
B
Yeah,
I'm
not
sure
this
is
this
one's
a
little
weird,
but
I
did
do
a
prototyping
with
mysql
for
using
the
execute
the
execution
hook.
The
cra
approach
yeah
yeah
this
one.
No,
we
haven't
really
get
to
the
coding
part
yet,
but
for
the
initial
prototyping
we
are
going
to
do
exact
anyway.
So
so
that's
going
to
be
similar,
we're
not
going
to
start
inside
the
couplet
when
we're
doing
prototyping
yeah.
D
Yeah,
I
do
not
have
implement
using
notifier
container
file
yet
because,
obviously,
it's
not
available
for
us
to
use,
but
I'm
using
port
exact
from
valero
and
it
works.
B
Yeah
so
yeah,
that's
what
I
was
using.
So
I'm
saying
that
for
the
prototyping
for
content
notification
that
we
will
also
be
using
exact
approach.
Actually,
so
it's
going
to
be
similar,
yeah
yeah,
but
if
once
we
really
switch
to
equivalent
I'm
not
not
sure
but
because
cubelet
internally,
it's
also
doing
an
exact
so
yeah,
I
don't
know
if
there's
any
difference
there.
If
you
know
if
we
do
poc,
it
works
with
a
controller
approach
that
doesn't
work
with
that
that
we
have
to
see.
B
D
F
Jing,
I
didn't
understand
the
issue
here.
Look,
it
seems
as
far
as
the
steps
but
quieting
taking
pvc
snapshots
and
on
quiet
thing,
these
are
all
done.
Serially
what's
different
is
that
the
semantics
of
quiet
is
different
for
my
sequel,
in
a
sense
that
the
application
is
not
completely
quiet,
it
allows
read
access,
but
that's
harmless
as
far
as
backup
is
concerned,
so
I
think
the
workflow
is
quite
actually
general
generic
can
be
applied
to
other
applications.
It's
just
that
the
semantics
of
quies
may
be
different
for
different
applications.
F
B
Yeah,
I
think,
for
the
application
from
the
application
point
of
view.
I
was
just
I'm
just
not
quite
sure,
I'm
not
that
familiar
with
cube
lady,
I'm
just
since.
Eventually
we
wanted
to
do
the
swinging
people,
I'm
not
sure
if
there's
any
anything
extra
there.
That's
what
I'm
trying
to
say
since
I
haven't
really
coding
directly
inside
kublet
and
then
the
plc
we're
going
to
do
even
with
the
container
notification
is
going
to
be
in
a
separate
controller.
B
To
start
with,
so
we'll
have
to
check
with
the
signal
they
probably
you
know
understand
more
from
this
point
of
view.
A
B
A
A
H
A
E
B
Because,
normally
you
would
have
the
external
controller
will
have
to
check
the
status
of
the
notification,
so
in
this
case
I'm
not
sure
if
q
or
whatever
cubelet
or
this
other
controller
will
be
able
to
update
the
status.
Yet
so
that's,
I
think,
that's
the
that's
the
only
thing
so
you
know.
E
B
When
I
was
doing
prototyping,
this
was
some
time
ago
with
the
execution
hook.
I
basically
just
do
it
immediately
right,
because
you,
if
you
are
keep
running
that
and
you
wouldn't
know,
then
then
that
says,
cubelet
or
something
or
execution
controller
extreme
controller.
It's
not
going
to
update
the
notification
status
and
then
so
for
this
external
controller.
What
how
is
it
going
to
know
that
it's
really.
A
B
Notification
status,
but
then,
in
this
case,
if
it
keeps
running
it's
not
going
to
return
status,
I
think
that's
the
that's
the
problem
here
with
this
particular,
I
think,
let's
not
just
to
get
stuck
on
this
first
one.
I
think
that
mexico
is
very
special.
We
can
come
back
and
talk
about
this.
Can
we
maybe
move
on
to
this
other
databases?
We
can
come
back.
I
think
mexico
is
really
special.
D
We
deal
with
is
the
most
complicated.
B
D
The
detail
offline
to
and
to
show
the
exact
status
how
you
know
that
is
successful
it
it.
We
can
do
that.
It's
just
that.
I
do
not
show
all
the
detail
here.
B
C
Yeah,
that
sounds
good,
and
even
if
you
close
your
session,
you
still,
you
can
still
take
a
best
effort
snapshot
afterwards,
which
we'll
at
least
have
all
the
data
up
to
the
date
when
you're
in
the
flush
tables
right,
which
I
think
is
what
happens
with
this.
I
Can
anyone
hear
me
well
yeah,
yeah,
okay,
so
what
I
did
was
I
researched
three
different
time
series
databases,
no
db
prometheus
and
influx
db
and
at
a
high
level
what
I
realized.
What
I
felt
was.
These
databases
are
relatively
newer
to
you
know
my
sequel,
postgres
and
other
relational
or
object
storage,
disputed
databases
that
I've
been
around
for
a
while,
and
these
databases
are
so
mature
that
they
are
providing.
I
You
know,
backup
and
restore
utilities
which
are
not
only
just
qsing
the
application,
but
actually
creating
the
backup
object
for
the
user
as
well
right.
So
as
a
result
of
that,
you
know
the
data
protection,
the
data
protection
challenge
is
greatly
simplified
from
a
vendor
perspective.
It's
as
easy
as
running
those
commands
for
taking
a
backup-
and
you
know
doing
a
restore
and
pointing
the
database
to
that
restored
archive
or
whenever
you
are
restoring
the
contents,
it's
as
simple
as
that,
so
we
can
go
in
order.
I
What
I've
done
here
is
kind
of
looked
at
each
of
these
databases
differently.
I've
kind
of
looked
at
you
know
chopped
out
each
database
in
these
four
segments.
We're
talking
about
the
overview,
you
know
what
is
the
backup
strategy
to
use?
What
are
the
resource
strategies
to
use
and
a
set
of
commands
that
will
give
you
the
full
backup.
You
know
full,
restore
incremental,
backup,
incremental,
restore,
and
you
know
any
kind
of
transactional
log
copying
or
journal
copying
of
that
nature.
I
So,
speaking
of
newer
db,
this
one
has
all
the
bells
and
whistles
you
know
to
do
you
know
offline
backups
if
you
want
to
shut
down
your
database
and
take
a
backup
or
if
you
want
to
do
it
online,
while
the
database
is
running,
it
supports
doing
a
full
backup
of
the
entire
database,
incremental
backup
of
the
database
and
also
copying
of
the
transactional
logs.
So
all
are
just
single
line.
You
know
cli
options
that
you
can
run
against
the
database
and
creates
all
that
for
you
from
a
strategy
perspective.
I
I
E
B
I
Okay,
so
yeah,
so
all
the
full
backups
incremental
backups
and
the
journal
copies
are
online
activities.
I
What
happens
is
there
is
a
directory
that
is
created
where
all
these
items
get
placed,
and
you
know,
depending
upon
the
command,
that
you
use,
whether
it's
a
command
for
a
full,
incremental
or
journal,
and
then,
similarly,
when
you
restore
it,
you
just
have
to
point
to
that
backup
set
where
these
items
are
placed
or
point
to
an
individual,
full
backup
or
an
incremental
backup
and
the
target
cluster
or
you
know,
the
restored
application,
would
restore
it
and
start
using
that
data
for
its
activities
or
for
its
configuration
moving
forward.
I
So
very
straightforward
didn't
find
any
challenges
in
terms
of
you
know
what
is
required,
the
documentation,
you
know
the
documentation
is
pretty
large,
but
I've
kind
of
summarized
it
in
terms
of
this
is
how
a
successful
backup
and
restore
should
happen.
C
How
do
you
map
this
to
kubernetes?
So
if
we,
the
primitives,
we
have
right
now
are
essentially
long
snapshots
and
then
we're
talking
about
adding
the
s
hooks.
So
I
think.
I
The
difference
that
I
see
is
you
know
with
traditional
databases,
we
would
qs
it
and
then
we
would
take
a
snapshot.
You
know
here.
I
would
say
that
we
would
probably
just
depend
upon
the
application
to
you
know:
do
the
qs
and
create
the
copy
of
the
backup
we
would,
you
know,
take
a
snapshot
of
the
persistent
volume
still
and
then
you
know
the
hooks
or
whatever
the
injection
commands
are.
I
So
I
think
if
we
are
it's
because
it
is
happening
at
the
file
level,
there
is
you
know.
Definitely
you
know
there
are
these
pointers
and
everything
which
they
are
using
to
just
create
that
snapshot
object
and
from
an
api
perspective.
If
I
understand
the
the
question
correctly,
I
don't
think
there
should
be
any
difference,
because,
as
long
as
we
are
capturing
the
pv,
you
know
using
any
kind
of
snapshotting
technology
via
csi
or
something
we
should
be
able
to
capture
all
the
file
level
items
without
any
issues.
B
But
what
other
sequence
then?
Who
is
creating
those
api
objects.
I
The
api
of
you
mean
the
api.
B
I
So
basically,
what
is
happening
is
you
know
when
we
are
taking
the
full
backup
full
restores.
You
know
it
is
creating
the
backup
object
and
putting
it
on
the
persistent
volume
itself
that
was
used
by
the
database
right.
So
as
long
as
you
take
that
persistent
volume
and
you
map
it
to
another
instance
of
the
database-
and
you
know
make
that
instant
point
to
the
backup
object
after
restoring
it,
it
should
be
good
enough
to
come
up.
I
No,
I
wouldn't
say
it's
double
because
you
know
it
because
it's
a
file
level
thing:
it
must
be
using
pointers
and
it
you
know
it
doesn't.
It
doesn't
increase
the
capacity
doubly
or
something
like
that.
It
does
use
space,
efficient
mechanisms
underneath
the
covers.
So
while
there
is
some
amount
of
data
that
will
be
created,
it's
not
going
to
double
it
right
off
the
bat.
I
So
when
you
are
doing
a
restore,
you
are
still,
I
mean
there
is
a
restore
command
that
needs
to
work
on
the
backup
object
to
restore
the
contents
into
a
format
that
the
newer
or
the
target
database
can
understand.
I
Yes,
so
that's
why
you
need
to
map
it.
The
real
data
is
not
the
consistent
data
right.
The
consistent
data
is
the
backup
object
that
you
created.
I
So
while
both
of
them
are
on
the
same
tv,
you
will
want
to
change
like
you
know,
you
can
delete
the
original
data
if
you
need
to
and
just
map
to
the
you
know
the
backed
up
data
and
do
it
that
way,
but
you
technically
do
not
need
the
original
data
after
you
do
the
restore
okay
right
and
the
limited
example
that
I
have
next
is
exactly
that.
You
know
where
the
where
the
snapshot,
you
know
the
steps
and
everything
pretty
much
the
same.
I
There
is
a
snapshot,
copy
or
snapshot
command
that
you
run,
which
creates
that
snapshot
on
the
same
pv,
where
the
prometheus
data
database
is
pulling
all
its
information
from
now
when
you're
restoring
it.
All
you
need
to
do
is
map
that
part
storage,
dot,
time
series
database
path
to
that
snapshot
that
you
had
just
collected.
I
So
it's
as
it
seems
I
mean
it's
very
straightforward,
because
you
know
all
the
qs
inconsistency.
Everything
is
taken
care
of
by
the
application
that
is
putting
it
into
the
same
pv
that
you
were
already
using
and
from
a
backup
restore
perspective.
It's
about
mapping
it
to
the
right,
consistent
object
upon
restore.
A
Okay,
where
are
the
queer
scene
hooks
executed.
I
When
you
say
where
are
the
qsync
hooks
from
the
applications
perspective
or
from
the
you
know,
from
our
perspective
from
kubernetes
from
kubernetes,
there
are,
there
is
no
specific
hooks.
I
would
say
that
are
technically
required
here
other
than
the
fact
that
sorry,
there
is
no
qsing
piece
of
a
hook
required
here
other
than
the
fact
that
you
should
be
able
to
hook
into
these
cli
libraries
or
the
command
api
libraries
that
these
applications
provide.
B
I
I
So
I
mean,
as
I
went
through
these
you
know,
that
was
my
understanding.
You
know
it
seems
like
these
newer
databases
have
seen
the
challenges,
the
problems
that
have
been
faced
with
all
the
other
databases
that
we've
been
dealing
with
so
far
and
they've
just
provided.
You
know
very
simple,
straightforward
ways
to
do
these
things
and
the
newer
db.
You
know
the
full
backup
incremental
backup
journal
copy,
resembles
a
lot
of
a
microsoft
sql
server
in
terms
of
how
they
provide
their
backup
capabilities
as
well.
I
Not
really,
I
can
do
some
more
digging
around
it,
but
but
yeah
I'm
not
100
sure
I
would
assume
it's.
You
know
just
cbt
kind
of
stuff
and
they
may
have
their
own
handlers
to
figure
out
what
the
change
blocks
was.
I
Yes,
yes,
so
what
I'll
do
is
I'll
update
the
cli
command
reference
as
well,
but
yeah?
You
need
to
provide
the.
Are
you
talking
about
the
backup
or
are
you
talking
about
the
restore.
A
A
Okay,
so
when
you
take
when
you,
whenever
you
run
this
command,
you
will
create
the
database.
We
will
create
an
incremental
or
full
backup
right.
Then,
actually
the
size
might
be
much
bigger
than
the
database
itself,
because
you
might
end
up
running
this
comment
multiple
times
no
correct.
I
Yes,
yes,
so
those
are
things
that
we
definitely
need
to
keep
track
of,
and
the
other
thing
that
I
did
not
add
over
here
is
you
know.
The
new
db
and
influx
db
also
provide
policies.
Okay,
so
they
have
their
own
policy
schedules
that
can
manage
these
incremental
backups
delete
the
oldest
one.
You
know
when
the
latest
one
is
taken
and
those
other
things,
so
the
user
could
leverage
those
items.
I
So
right
now,
as
I
said,
the
backup
directory
is
the
directory
or
a
folder
structure
on
the
persistent
volume
itself,
but
you
can
configure
additional
controllers
to
map
it
to
an
external
s3
repository
or
something
as
well
so
to
I
I'm
not
sure
if
I'm
answering
your
question
correctly,
no.
A
No,
no,
the
the
those
data
is
right
how
many
pvp
uses
they
have.
A
I
A
I
I'm
assuming
that
it
would
be,
it
would
be,
placing
it
on
some
master
controller
object.
Where
you
know
everything
is
stored
together.
I
can
take
in
that
for
you
and
figure
it
out
yeah,
it's
a
good
question.
I
did
not
look
into
it
yeah.
I
think.
F
I
A
good
point,
that's
a
good
point
and
I
think
for
influx
db.
If
we
scroll
scroll
a
bit
up
down-
and
I
think
that's
exactly
what
they're
also
doing-
they
basically
create
these
three
objects.
You
know
they
you,
you
can
have
multiple
shards,
but
it
just
creates
a
single
shard
object
for
as
part
of
the
backup
item
that
it
creates.
E
I
Yes,
yes
yeah,
I
I
looked
into
there's
no
single
way
of
just
qsing
the
data
and
then
taking
snapshots
of
the
pvs.
You
know
this
ability
to
do
snapshotting,
plus
dump
by
the
application,
was
the
only
method
that
I
saw
across
these
three
databases.
I
I
mean
it's
it's
more
about.
You
know
if,
if
you're
not
using
a
centralized
data
protection
solution
and
if
you're
just
using
the
application,
you
know
pieces
to
protect
it,
then
that
would
definitely
make
sense.
I
think
the
question
or
the
strategy
from
from
a
data
protection
solution
perspective
should
be
having
the
ability
to
run
these
api
commands.
I
believe
you
know
before
we
take
a
backup.
F
I
Yes,
yes,
and
especially
with
what
the
thing
that
we
need
to
keep
in
mind,
is
you
know
once
we
if
we
are
moving
these
applications
from
one
side
to
another
side
and
we
are
qsing
it?
You
know
the
application
is
still
in
a
qs
state
when
we
are
going
to
be
restoring
it
so
upon
the
restore.
Also,
there
needs
to
be
a
nqs
command
that
might
need
to
be
executed.
I
Yeah
yeah
you've
got
to
look
at
it
at
two
levels:
right:
the
applications
where,
like
you
know
these
time,
series
and
cd
that
provide
everything
built
in
versus
you
know,
databases
like
mysql
and
so
on,
which
just
provided
up
to
the
qs
import.
I
think
that's
how
the
high
level
you
know
workflows
would
be
differentiated.
In
my
opinion,.
F
I
So
everything
follows
the
concept
of
a
backup
set.
So
if
you're
running
an
incremental
backup
and
you're
pointing
to
that
slash,
dem
hot
copy
directory
and
that
hot
copy
directory
was
a
backup
set,
then
it
will
use
the
data
within
that
directory
to
do
the
change
block,
tracking
and
figure
out
what
the
incremental
pieces
are
that
need
to
go
into
that
particular
directory.
You
can
have
multiple
directories
for
the
same
kind
of
backup
as
well,
so
it
just
follows
the
hierarchy,
structure
of
a
backup
set
and
everything
gets
put
into
that
backup
set.
F
I
I
see
yeah
overall
I
mean
I
would
say
that
definitely
simplifies
life
from
you
know:
recovery,
migration,
perspective
and
so
on,
like
if
we,
if
just
another
point
I'll,
add
on
in
flux
db,
they
also
have
the
capability
of
doing
continuous
disaster
recovery.
So
you
could
have
two
influx
db
instances
and
you
can
just
start
keep
importing
data
from
one
instance
into
the
other
instance
and
provide
a
time
into
it.
J
A
Cover
can
we,
if.
A
G
Yeah
I
can,
I
can
go
through
it
quickly,
yeah,
so
I
spent
the
time
to
look
at
mongodb
and
for
mongodb,
as
opposed
to
my
sequel,
where
you
have
to
keep
that
station
running
during
the
the
choirs
command.
Mongodb
provides
the
a
similar
command
to
run
the
choirs
and
enquires,
but
for
this
one
you
don't
have
to
keep
the
session
running
so
once
you
run
the
quiz
command.
This
is
the
db
fsync
lock.
G
G
You
know
the
status
of
the
the
command
and
also
it
keeps
track
of
the
lock
count
so
that,
for
every
lock
that
you
run,
mongodb
will
keep
the
account
and
it
will
keep
incrementing
that
for
every
requires
for
every
dbf
sync
log
command
that
you
run
so
there
should
be
a
corresponding
unlock
command
that
will
decrement
that
log
count
and
for
the
for
the
database
to
be
unlocked.
That
log
count
has
to
be
zero.
G
G
So
for
solid
cluster
you
have.
The
data
is
distributed
among
the
the
shads
and
before
you
take
the
backup,
you
first
have
to
stop
the
balancer,
because
if,
if
there
is
any
data
migration,
that's
happening
between
the
shards
during
the
backup,
then
you
can.
You
might
end
up
with
an
inconsistent
backup.
So
it's
very
important
to
stop
the
balancer
before
the
backup
and
that's
what
I
guess.
C
Yeah,
so
it's
interesting
here
for
the
for
the
from
the
perspective
of
the
container
notifier
is
that
you
have
to
do
some
a
little
bit
of
coordination
between
what
happens
on
each
replica.
It
seems
like
for
char,
you
have
to
execute
commands
on
each
tropical
first
and
then
you
can
do
the
fsync
and
then
the
snapshot.
G
Right,
actually,
it
seems
you
just
need
to
connect
to
from
the
documentation
you
just
connect
to
any
of
the
the
mongos
and
you
can
run
the
the
stop
balancer
command
and
then
so.
This
will
stop.
The
cluster
balancer
and
mongodb
also
provides
two
functions
that
you
can
use
to
query
the
status
of
the
the
balancer.
This
is
the
you
can
you
can
use
the
get
balance
state
and
there's
another
one.
There
is
balancer
running,
so
this
one
will
enable
you
to
to
verify
the
status
of
the
balancer.
G
So
once
you
verify
that
the
balancer
is
disabled,
you
need
to
backup
the
config
database.
I
think
this
is
where
the
mongodb
stores
the
metadata
for
the
shaded
cluster
and
then
so
it
depends
now.
There
are
some
options
where,
if
you
choose
precision,
you
you'll
need
to
run
the
choice
command
so
lock,
all
the
you
know
all
basically
lock
the
entire
cluster
you
you
need
to
run
the
the
db
fsync
lock
command
on
all
the
I
believe
for
each
replica.
G
This
is
prior
to
doing
the
previous
snapshot,
so
you
lock,
basically
the
entire
cluster.
Alternatively,
you
can
just
run
the
the
lock
command
on
the
second
remember,
which
shard
a
replica
set.
This
is
in
case
you
don't
want.
You
know
you
don't
have
to
have
your
all
the
whole
cluster
locked.
A
G
Right
so
this
will
be
run
on
the
shell,
so,
for
example,
with
the
pod
exec
on
the
shell
of
the
you
know,
the
mongodb
container.
G
Right
yeah,
I
tested
this
with
I've
done
this
with
using
the
valero
that,
like
found
mentioned
development
exactly
so
this
one.
I
was
about
to
get
it
running
with
the
pod
exec.
But
yes,
if
you
shell,
if
you
open
a
shell
to
one
of
the
containers,
then
you
can
you
can
down
this
same
command.
G
Right
so
from
the
deployments
I
tried,
I
yeah,
I
saw
there's
just
one
pvc
per
shard,
so
each
shot
is
it's
a
stateful
set
and
you
cannot.
You
love
like
the
primary
member
and
any
number
of
secondaries
for
each
shot
right.
I
And
the
the
snapshotting
and
everything
can
happen
like
in
parallel
for
each
of
those
pvcs
right.
G
Right
so
I
think
on
step,
so
once
you
acquire
the
you
know
depending
if
it
requires
the
whole
application
or
each
secondary
member,
then
the
pvcs
need
to
be
snapshotted
in
parallel.
I
think
just
to
make
sure
that
you
know
they
are
consistent.
J
Yeah,
I
have
one
question:
when
you
do
sync
log
right
have
sync
log,
so
that
means
all
right
operation
will
stop.
J
To
the
right
operation,
if,
during
the
back
up
period
the
if
anybody
is
right
trying
to
write
what
will
happen
to
those
right.
G
They'll
be
cured
or
yeah,
it
will
be
cued
yeah,
it
will
be
cured
and
then,
once
you
unlock
that's
when
now
the
right
will
be
committed.
J
G
J
G
J
In
general,
in
general,
mongodb
also
suggests
to
take
backup
using
ops
manager.
They
have
a
one
tool
right
and
it
takes
the
back
one
s3
or
any
other
devices,
and
you
can
restore
point
in
time.
Recovery,
I'm
not
sure.
By
taking
snapshot,
can
we
restore
pointing
time
recovery.
E
C
Al
is
like
a
logical
level
dump
and
then
it
uses
the
ops
log
to
do
point
in
time.
I
think
which.
C
Journal
basically,
okay,
I
from
my
experience
the
logical
dumps
for
are
very
slow
for
any
reasonably
sized
database.
We,
you
know,
we
only.
We've
only
been
able
to
use
volume
snapshots,
but
that's
you
know,
that's
my
anecdotal
experience.
A
J
No,
but
if
you
use
ops
manager
right,
they
have
the
backup
vrs,
they
call
backup
recovery
service,
they
do
a
snapshot.
Take
this
snapshot
and
every
15
minutes
one
hour,
and
then
they
took
the
op
log.
Is
there
upload
it
takes
the
incremental
changes,
so
you
can
restore
to
any
point
in
time
recovery.
It
creates
a
time.
A
A
J
F
G
Oh
yeah,
I
could
check
this
so
yeah
mongodb,
it
provides
there's
an
there's,
a
call
I
think
db
is
master,
which
would
return
true.
If
that
specific
instance
is
the
is
the
the
primary
sorry,
and
so
you
can
use
that
call
to
check
if
that
instance
is
the
primary
node
and
then
for
this
one
for
the
approximate
point
in
time,
this
is
from.
Actually
I
I
linked
down
there
there,
the
documentation
for
for
the
from
the
manual
for
our
mongodb
recommends
taking
a
backup
of
the
shared
cluster.
F
G
It's
okay,
if
you
don't
know
the
last
yeah,
I'm
not
sure
yeah.
Okay,
thank
you.
I
can
look
into
it
and
also
you
know
one
thing:
the
mongodb
documentation.
It's.
I
found
it
to
be
very
comprehensive,
so
I
tried
to
link
some
of
them
down
there.
You
can
also
look
look
into
it
after
the
meeting.
A
Thank
you
all
right.
Sheen.
We
only
got
five
minutes
left.
We
won't
be
able
to
go
through
the
execution
hooks,
but
this
is
all
good
discussions.
E
A
Still
got
kapka,
we
will
make
it
a
short
time.
C
Yeah,
I
I
mean
sean,
you
do
most
heavy
lifting
here,
but
I
can
give
a
quick
summary.
So
kafka
is
interesting
because
it's
pretty
tough
to
actually
do
data
protection
for
kafka.
Kafka
has
multiple
broker
nodes.
Each,
which
has
is
primary
for
some
subset
of
the
topics
which
is
kind
of
similar
to
the
shards
in
in
manga.
We
were
just
talking
about
backing
it
up
is,
is
really
a
challenge
because
you
really
have
to
take
down
the
brokers,
there's
not
kind
of
a
simple
quiet
hook
that
you
can.
C
You
can
execute
really
it's
kind
of
a
stop
the
service
itself.
The
other
thing
is
there's
kind
of
two
imagine
data
services
you
need
for
this.
One
is
kind
of
the
thing
that
serves
the
actual
topics
and
messages
and
the
other
one
is
zookeeper.
So
zookeeper
contains
two
kind
of
important
bits
of
information.
One
bit
is
the
consumer
offsets.
So
if
I
have
a
client,
I
actually
do
have
some
server-side
state
that
tracks
kind
of
where
I
am.
C
It
also
contains
topology
information
for
who
is
who's
currently
the
leader
for
for
the
various
among
the
brokers
for
various
topics.
So
we
did
kind
of
mock
up
what
backups
of
this
would
look
like
in
kubernetes?
C
It's
pretty,
I
would
say
it's
definitely
best
effort.
You
know
this.
This
would
be
possible
to
execute,
but
it
wouldn't
involve
some
downtime
right.
In
fact,
when
you
freeze
and
unfreeze
the
zookeeper
volumes,
you
would
not
be
able
to
make
progress
in
your
your
client-side
offsets.
Your
excuse
me,
your
consumer
offsets
stored
in
zookeeper,
which
is
problematic
right
that
that
doesn't
go
with
one
of
the
goals
here
was
to
have
as
little
downtime
as
possible,
and
then
you
could
just
take
the
normal
volume
snapshots
right
on
a
per
broker
basis.
C
I
think
our
recommendation,
for
this
is
basically
that
you
know
this
is
similar
to
what
we
looked
at
with
nuo,
where
maybe
it's
not
a
great
fit
for
this,
this
type
of
quiesce
unquest
hook.
You
know
you
might
need
some
kind
of
application
integration
to
get
this.
This
correct.
A
Yeah,
I
also
want
to
bring
the
topic
to
the
community.
It
is
interesting.
The
kafka
is
one
of
the
great
examples
where
there
are
actually
multiple
applications
within
this
offering
so
kafka
itself
depends
on
zookeeper
or
if
you
treat
your
keeper
to
keep
yourself
as
a
notification
right
the.
What
makes
things
complex
is
the
zookeeper
itself
can
serve
cop
cap,
but
it
also
can
serve
others.
A
So
we
put
this
thing
together
again
with
the
best
effort,
because
at
least
based
on
very
limited
research
tom-
and
I
haven't
found
the
effective
way
of
doing
this
in
kubernetes
environment,
and
that's
one.
The
other
one
which
is
also
interesting,
is
that,
as
previously
mentioned
applications,
they
are
already
pretty
rich
tools
around
those
applications
to
do
either
a
in
place,
backup
or
a
snapshot
within
the
environment
or
a
synchronizedly
copy.
The
data
to
another
cluster,
that's
actually
most
of
the
data.
A
Most
of
the
end
users
achieve
dr
in
these
days,
for
example,
mirror
maker
2
is
a
very
kind
of
you
know.
Popular
tool
that
is
widely
used
in
the
cochlear
community
to
asynchronous
replicate
make
the
synchronized
replication.
A
You
get
some
drawback
from
that
and
more
interesting,
more
interestingly,
I
think
for
almost
all
these
distributed
systems,
they
offer
very
high
availability
guarantees
right
and
some
kafka.
Can
you
even
do
a
stretch,
cut
cluster
kafka,
which
means
that
your
your
cover
can
run
in
two
different
data
centers.
Even
if
you
lost
one
data
center,
you
still
have
the
the
other
one
up
and
running
without
affecting
your
business.
A
So
in
this
case,
with
all
these
complications
feeding
in
here,
I'm
not
sure
we
are
ready
to
propose
anything
in
this
community
for
this
very
complex
applications
per
se.
A
One
thing
I
I
think
I
took
away
from
today's
session
is
that
maybe
it's
not
a
very
bad
idea
to
look
into
those
typical
applications
supporting
tools
to
take
backup
in
this
case,
if
those
tools
already
supporting
application,
consistency,
snapshot
or
backup
functionality,
us
as
a
community,
maybe
can
just
wrap
or
control
on
top
of
it
and
utilize.
A
Those
functions
directly
without
you
know
even
in
working
web
hook
to
do
the
choirs
and
the
web
hook,
becomes
a
mechanism
to
issue
a
signal
and
that
actually
goes
well
for
the
concept
of
container
nullifier
yeah.
That's
pretty
much
what
I
have
today.
I
think
I
will
sync
up
with
shin
net
on.
I
think
I
really
appreciate
all
the
effort
everyone
put
into
this.
It's
a
lot
of
effort
in
a
very
short
period
of
time
shin.
Unfortunately,
we
don't
have
enough
time
to
go
through
the
our
agenda
today.
I
guess.
B
A
Yeah
and
alice
wanted
himself
to
cover
the
xcd
operator
from
openshift.
A
Maybe
we
we
can
sing
offline
on
that
as
well.
Thanks
everyone
for
today's
meeting
any
last
minute
questions
or
suggestions.
C
I
think
we'll
have
to
figure
out.
We
still
want
to
follow
up
and
maybe
have
more
meetings
and
figure
out.
What's
left
here,
yeah,
maybe
xing
and
sean.
Maybe
you
can
drive
that.
A
Yeah,
I
will
I
will
set
up
a
following
on
to
the
on
to
this
discussion.
Anyone
if
you
are
interested
in
joining
that
follow-up
meeting,
we're
going
to
mostly
focus
on
what
we
discussed
today
in
more
detail.
Please
send
me
or
send
an
email
all
right.
Thank
you.
Everyone
have
a
race,
nice
trip.
Thank
you.