►
Description
March 17, 2020
Agenda
Status Updates
[ashish]
More CSI work and getting couple of test and cleanup PRs merged.
Preparing for a talk on Velero with Nolan
Brought up issues from last week in upstream volumesnapshot working group meeting. More discussions will happen on these.
VolumeSnapshotRef.UID should be cleared on volumesnapshot deletion
More discussions on this in the data protection working group
Relation between VolumeSnapshotContent and VolumeSnapshot should be 1:many
Effectively won’t fix as there are reasonable workarounds
Lifecycle of VolumeSnapshot Kubernetes objects should be decoupled from that in the storage system
More discussions on this will happen in the issue.
[steve]
shipped v1.3.1
finalized Azure blob chunking code
starting to look at two-stage snapshot/backup issue again (https://github.com/vmware-tanzu/velero/issues/1519)
[carlisia]
CLI PR good to review: https://github.com/vmware-tanzu/velero/pull/2202 (look at the TODO list, working on the last 2 items)
[Nolan]
Prepping talk with Ashish
More work on https://github.com/vmware-tanzu/velero/pull/2323 for uploading CSI resources to object storage
Allows us to show these snapshots in the client output & do imports w/ the backups if we need
Discussions w/ Ashish on the snapshot issues he mentioned
[brito-rafa]
new labels for the backups #2342
Examples:
velero.io/source-cluster-k8s-major-version=1
velero.io/source-cluster-k8s-minor-version=14
velero.io/source-cluster-k8s-version=v1.14.10
Discussion Topics
[steve] add plugins/ top-level dir to backup storage locations to allow plugins to write arbitrary data to bucket – https://github.com/vmware-tanzu/velero/issues/2344
[Mayank] restoreItemAction plugin to clear/update ‘selected-node’ in PVC, which are created by scheduler for dynamic provisioning. Code https://github.com/mynktl/velero-plugin/commit/a03a74004bf0482537649ae4b390cbd2ee9e2ac5
[Antony] How can we control ordering of resource backups and restores?
We don’t currently have the mechanisms to do this in Velero
StatefulSets don’t have proper support; we need to special case them, but it’s not a generalized solution
Antony to file an issue and we can brainstorm solutions
[Sam Lucidi] Custom CA support
Working on how to get certificate bundle on the pods
Both plugins and restic need access to it, but have slightly different paths to that data.
A
For
now
I'll
deal
with
this
later
all
right
looks
like
we're.
Recording
so
welcome
everyone.
It's
the
valar
community
meeting,
March
17
2020,
hope
everyone's
doing
all
right,
staying
safe,
all
right!
Well,
it's
let's
just
jump
into
it,
so
we've
got
some
status
updates
to
start
and
then
a
few
few
different
discussion
topics,
I
know
shishas.
First,
here
on
the
status
updates.
I
know
he
was
you
had
to
run
to
a
personal
appointment,
so
I
don't
think
he'll
be
available.
I
know
some
of
the
stuff
he
has
on
here
around
the
CSI
snapshot.
A
Work
Nolan
is
going
to
cover
a
little
bit
as
well,
so
I
think
we'll
just
skip
over
his
topics
for
now
and
Nolan
can
cover
with
what
he
has
information
on
so
yeah
I'm
up
next.
So
last
week,
I
got
1.3.1
shipped
on
Tuesday,
with
a
small
bug
fix
for
backups.
I
haven't
heard
any
further
issue
reports
from
that,
so
hopefully
things
are
going
all
right.
There
I
also
finalized
the
blob
chunking
code
for
Azure
that
I
talked
about
last
week.
I
got
finished
up,
testing
and
pushed
that
out
for
review.
A
So
I
think
that's
ready.
Parlays
had
one
comment
on
the
documentation
which
I've
addressed
so
should
be
ready
for
another
look
and
hopefully
emerge
there
and
then
I've
also
started
looking
at
the
issue
that
we
were
discussing
last
year,
around
kind
of
having
a
two-stage
process
for
snapshots,
where
the
first
step
is
creating
the
snapshot,
the
local
snapshot
and
then
the
second
step
is
some
form
of
either
waiting
for
the
snapshot,
data
to
replicate
off
to
durable
storage
or
actually
triggering
that
upload
of
data.
A
So
we
had
a
contributor
from
open
EBS
who
had
started
to
put
together
a
design
doc
there,
but
I'm
sort
of
picking
that
up
again
and
hoping
to
at
least
get
some
thoughts
down
on
paper
and
start
to
get
input
from
the
relevant
folks.
There
so
I'll
be
continuing
to
work
on
that
this
week,
I'm
also
on
community
support
this
week.
So
that's
keeping
me
keeping
me
a
little
bit
busy
that
hasn't
been
too
burdensome
so
far,
quick.
B
A
Yeah
potentially
I
mean
I
really
haven't
gotten
into
it
much
yet,
but
you
know
I
think,
there's
a
potential
that
some
of
that
code
lives
in
Valero
and
there's
some
sort
of
interface
or
plug-in
mechanism
to
allow
folks
to
plug
in
specific
data
movers.
But
I
would
sort
of
like
to
consider
the
the
full
spectrum
from
you
know.
A
Just
a
very
passive
model
where
we're
just
maybe
pulling
a
plug
in
to
you,
know,
ask
if
the
snapshots
been
replicated
yet
kind
of
all
the
way
to
the
other
side
of
the
spectrum,
where
Valero
is
very
actively
contribute,
vents,
yeah
and
I
hope
hope
to
get
input
from
from
anyone
who's
interested
there.
And
you
know
we
definitely
need
to
consider
both
the
kind
of
public
cloud
use
cases
as
well
as
the
the
more
on-prem
use
cases,
because
the
models
are
definitely
different.
C
A
B
I
am
working.
She
should
also
has
also
been
working
on
getting
restores
working
with,
so
he
had
demoed
backups
in
stores.
Last
week
we
merged
the
restore
work.
This
yesterday
and
I
know
he's
got
some
proposals
for
cross
names
base
restores
that
will
not
work
right,
the
same
way
as
namespace,
remapping
and
folio.
Currently
we
have
a
separate
issue
for
that.
We're
gonna
do
in
the
separate
PR
and
then
I
am
working
on
a
separate
PR.
B
That's
in
floral
core
to
upload
the
CSI
resources
associated
with
a
backup
into
objects
store
so
that
we
can
show
those
snapshots
and
client
output.
So
if
you
do
velaro
backup
subscribe,
you
can
see
them,
and
possibly
so
we
can
do
imports
on
backup
sync.
So
when
you
sync
in
the
literal
backups,
you
can
sync
in
volume.
B
A
B
I
need
to
I
need
to
set
up
an
environment
similar
to
a
shishas.
That's
the
code,
all
that
I
have
all
compiles,
but
I
haven't
tested
actually
like
generating
a
backup
and
getting
with
with
the
updated
code.
I
haven't
tested,
getting
the
full
backup
generating
CSI
objects,
which
would
then
get
shipped
up
to
object,
store
so
I,
don't
know
that
that
actually
functions,
yeah,
okay,.
D
A
Alright
I
think
we
can
move
on
to
the
discussion
topics.
Then
I've
got
the
first
one
here
so
yesterday,
I
was
talking
about
Dave
Smith
ochita
about
the
vSphere
plug-in
that
the
his
team
has
been
working
on,
and
one
thing
that
they
are
interested
in
doing
is:
writing
some
data
to
the
basically
into
object,
storage
and
he's
sort
of
trying
to
figure
out
what
the
most
appropriate
place
to
put
that
data
is
so
this
is.
A
It
comes
out
of
a
volume,
snapshot
or
plugin,
and
so
what
I
talked
about
with
him
was
adding
a
new
top-level
directory
to
the
bucket
that
Valero
uses
right
now
we
have
within
that
bucket.
We
have
backups,
we
have
restores,
we
have
rustic
and
I
think
we
have
metadata
I,
think
those
are
the
four
directories.
So
we
talked
about
adding
a
fifth
directory.
That's
allowed
they're
called
plugins,
which
would
basically
be
a
place
where
any
plug-in
could
then
create
a
further
subdirectory
and
write
whatever
arbitrary
data.
It's
interested
in
writing
there.
A
The
the
reason
that
there's
actually
a
code
change
required
here
if
we
want
to
support
this,
is
because
we
have
some
validation
code
at
velaro
server,
startup
time
that
essentially
looks
at
the
contents
of
the
bucket
or
the
you
know,
the
prefix
within
the
bucket
that
you're
that
you
have
configured
to
your
backup,
storage
location
and
it
it
basically
errors
out,
if
you
have
any
directories
in
there
that
that
don't
match
the
set
of
allowed
directories.
So
if
we
want
to
do
allow
this
plugins
directory
we'd
have
to
make
I
think
it's
a
one
line.
A
B
A
It's
not
something
I
remember,
but
okay,
double
checking,
I,
guess
the
you
know
the
other.
The
other
thing
to
add
here
is
that
I
think
that
in
this
case,
the
data
that's
gonna
be
uploaded
is
is
somewhat
similar
to
kind
of
the
rustic
repository
data,
where
you
can't
just
very
clearly
separate
it
and
put
you
know
an
individual
file
within
a
specific
backup
subdirectory,
because
that
would
be
another
option
in
some
cases
this
is
more
kind
of
a
general
collection,
a
bit
of
data
and
metadata
that
spans
across
various
backup.
A
Not
necessarily
I
think
it
would
be
up
to
you
know
up
to
the
plugin
author
to
decide
how
to
organize
the
data,
but
you
know
what
I
was
envisioning
here
was
so
we'd
have
the
plugins
top-level
directory.
Then
you
know
for
this
one,
which
is
a
vSphere
plug-in
there.
There
basically
be
a
subdirectory
under
plugins
called
you
know,
peace,
fear,
volume,
snapshot
or
whatever
you
know,
whatever
name,
and
then
under
that
the
plugin
author
can
can
organize
things.
However,
they
want.
B
B
Tangentially
I
think
Andy
and
back
had
one
day,
I
think
he
was
sitting
there
saying
I
want
to
be
able
to
cross
communicate
between
my
plugins
and
he
had
opened
up
an
issue
saying
he
wanted
to
have
like
Redis
to
in
the
deployment
to
talk
between
all
the
plugins,
but
that's
just
kind
of
sitting
than
the
backlog.
We
never
really
did
anything
with
that.
So
I
don't
see
any
any
problems
with
this,
provided
we
make
I,
don't
know
how
we
enforce
it,
but
plugins
should
only
work
in
their
directory
or
in
their
prefix.
A
Yeah
and
I
just
to
respond
to
that
I,
don't
know.
If
there's
you
know
any
really
any
way
that
we
can
enforce.
That
I
mean
you
know
right
now,
the
plug-in
could
you
know
could
modify,
could
basically
the
plug-in
is
gonna
have
access
to
the
bucket,
and
so
you
know
the
plug-in
uses
the
same
I'm
credentials
to
connect
to
it,
and
so
you
know
they
can
change
data
in
in
kind
of
any
way
right.
B
Unless
we
only
exposed
it
through
the
velaro
persistence,
so
we
have
our
persistence
layer,
the
package,
if
we
exposed
it
through
that
we
could
make
it
a
little
bit
smarter
and
say:
hey
your
your
your
this
plug-in
trying
to
access
some
other
prefix.
You
can't
do
that,
but
that
would
require
some
extra
architecting.
C
A
Not
currently,
no
okay,
you
know
so
so
sort
of
as
a
parallel.
The
download,
the
velaro
backup
download
command
also
does
not
rustic
data,
which
some
I
I
think
we've
had
enhancement
requests
from
some
folks
saying
that
it
should,
but
as
of
right
now,
you
know
valera
backup
download
is
basically
limited
to
metadata
in
kubernetes
IMO,
as
opposed
to
actual
snapshot,
contents
or
backup
contents
volume,
backup,
contents,
I
guess,
ok,.
A
E
A
A
Ok,
yeah
I
mean
I,
think
I
haven't
had
a
chance
to
look
through
this
in
detail
and
the
ID.
You
know
based
on
what
I
understand
so
far,
it
seems
like
something
that
that
would
be
up.
Streamable
I
mean
it
says
it's
you
know,
I
think
of
it
as
kind
of
fairly
similar
fairly
analogous
to
our
storage
classroom
mapping
plug-in
which
we
have
entry.
E
A
Yeah
I
mean
it's,
you
know
it's,
it
can
certainly
be
packaged
up
in
its
own
plugin
image
and
and
basically
be
optionally
added
that
way,
I
mean
we.
You
know
we
do
have
a
certain
set
of
plugins
that
are
in
the
Valero
repo
and
those
are
generally
just
ones
that
we
think
are.
You
know
commonly
useful
applicable
to
pretty
much
any
user
of
the
lair.
A
A
F
G
Yeah,
let
me
just
go
okay,
so
I
had
a
question
regarding
the
so
right
now,
we've
been
doing
some
work
with
pre
and
post
hook
to
take
backups
of
lissa
like
if
you
have
stateful
database
is
running
on
your
cabinets.
Is
we
found
that
we
can
make
use
of
the
pre
and
post
oak?
Let's
say
to
to
run
some
choirs
commands
before
the
backup
and
then
do
the
inquires
with
the
the
post
hook?
So
we
find
right
now
we're
trying
to
see
how
we
can
have
some.
G
So
let's
say
if
you're
running
a
you
know,
multi
cluster
database
deployments
with
they
say
master
and
couple
of
slaves.
We
want
to
have
an
ordered
way
in
which
we
we
have
a
little
do
the
backup
of
the
pod,
so
that
in
case
they
say
we
might
want
to
first
take
a
backup
of
the
the
PVCs
that
are
being
used
by
listen
like
the
master
pod.
So
in
that
case,
we
want
to
have
a
little
is
execute
the
the
backup
of
that
pod
fast
before
the
other
pods.
So,
basically,
my
question
was
with
Valero.
G
A
Yeah,
that's
a
good
question:
I
mean
off
the
top
of
my
head
I.
Don't
think
we
really
have
the
mechanisms
that
we
would
need
to
control
order
at
that
at
that
level.
You
know
we
have
some
ordering
in
terms
of
the
resource
types,
but
if
you're
saying
I
want
to
backup,
you
know
this
pod
before
this
other
product
and
I
can't
think
of
any
way
that
you
could
sort
of
that.
You
could
do
that
through
the
existing
mechanism.
So
it
would
probably
you
know,
require
an
enhancement
to
Valero
to
be
able
to
support
that
I.
G
A
F
G
That
we
found,
if
you
have
like
a
single
cluster
deployment,
just
a
master
or
just
one
node.
We
know
that
works
okay
for
us,
but
just
for
the
multi-class,
the
deployments
is
when
we
are
trying
to
see,
we
can
have
more
control
of
the
order.
So
right
now,
it's
still
you
know
very
early
on
and
still
looking
into
a
certain.
We
can
do
this.
A
That
sounds
good
and
I'd
say
we're.
You
know
we're
definitely
interested
to
hear
kind
of
the
results
of
as
you're
trying
this
with
some
different
workloads.
Seeing
you
know
what
works
and
where
you're
running
into
issues,
we'd
love
to
kind
of
have
that
knowledge
so
feel
free
to
share.
If
you
can,
okay.