►
Description
March 24, 2020
Status Updates
[ashish]
Working on CSI
Discussion on VolumeSnapshotRef.UID should be cleared on volumesnapshot deletion
May have to rethink the restore scenario as there are some logic that depends on how volumesnapshotcontents are named.
Coming w/ alternative approach here, plan for the week
BackupItemAction plugin to for CSI volumesnapshots, created outside of Velero.
WIP PR #31
//TODO: link issue here
Building a catalog of SOPs for backing up and restoring datastores. Help us prioritize during discussion:
mySQL: Single pod/ HA?
Cassandra
MariaDB
MS SQL?
PostgreSQL
MongoDB
[brito-rafa]
In progress code to backup multiple API Group versions (instead of only preferred APIGroups)
Table of preferred group version and supported groups
Quick demo of horizontalautoscaler item: v1 versus v2beta2
Collect any ideas and feedback from the community
[steve]
did some CSI snapshot testing on Azure
lot of support & reviews
gonna have to rebase https://github.com/vmware-tanzu/velero/pull/2317 at a quiet time
[carlisia]
CLI revamp design doc ready for another round of reviews: https://github.com/vmware-tanzu/velero/pull/2202
Testing PR to update the Velero Helm chart to version 3: https://github.com/vmware-tanzu/helm-charts/pull/81
[nolan]
Need to reply to reviews on https://github.com/vmware-tanzu/velero/pull/2323
Also need to set up CSI test beds, didn’t actually run this code yet
Last week was mostly spent doing internal presentation stuff
Testing and documenting Helm 2 chart behavior with Velero
Releases restored before Tiller aren’t “adopted”
Discussion Topics
[Cyril] Cassandra backups and missing features
https://github.com/vmware-tanzu/velero/issues/2361
[Mayank] restoreItemAction plugin to clear/update ‘selected-node’ in PVC, which are created by scheduler for dynamic provisioning.
Code https://github.com/mynktl/velero-plugin/commit/a03a74004bf0482537649ae4b390cbd2ee9e2ac5
steve: :+1: to PR into core (detailed code review still to be done)
[Nolan] Kubernetes 1.18 comes out today, if anyone has bandwidth to try Velero v1.3.1 with it, that would be awesome!
A
Hi
everyone
and
welcome
to
another
episode
of
the
Valero
community
meeting,
slash
open
a
discussion
today.
We've
got
a
bunch
of
things
and
we
want
to
handle
our
usual
and
I'm
going
to
start
with
sharing
my
screen.
So
you
all
could
see
the
agenda
here
and
then
we
have
Ashish
first
and
he's
not
on
yet
so
we'll
wait
for
him
to
give
his
update.
So
we'll
move
over
to
Rito
Rafa.
B
B
I
think
I
spoke
last
week
about
it
now,
I'm
tackling
a
little
bit
longer
conversation
about
API
group
versions
and
I'm,
not
gonna,
go
into
what
API
groups
is
I,
probably
gonna
write
a
blog
post
about
it,
but
what
I
wanna
share
quickly
share
my
screen.
If
I
may
about
the
ticket
that
I'm
using
to
do
this
proposal,
the
ticket
is,
there's
one
two,
two
five
one
who
can
see
the
details.
The
person
who
created
I
think
I
like
the
way
the
the
person
is
very
well.
B
Each
API
group
has
versions
and
depending
on
your
deployment,
you
have
a
preferred
version
of
your
API
group.
As
far
as
I
can
tell
it's
not
configurable
the
preferred
version,
you
can
disable
groups,
but
you
cannot
tell
cherry-pick
which
ones
are
preferred
version
why
this
is
important,
because
Valero
backs
up
the
preferred
version
today.
B
But
if
you
look
at
the
API
discover
API,
if
you
do
another
call,
you
get
entire
list
of
the
items
across
multiple
group
groups,
whatever
the
cluster
support,
so
I
did
a
preliminary
code
change
in
what
is
showing
right
now
and
what
I'm
put
on
the
screen.
It's
a
typical
backup
using
the
patch
that
I'm
working
on
if
you're
familiar,
has
the
directory
structure
of
the
backup.
So
you
have
resources
and
then
you
have
the
directories
of
the
resource
lists.
B
It's
the
resource
and
group
right.
I'm
gonna
use
this
this
one
as
an
example,
horizontal
pod
out
scalar,
which
is
part
of
the
group
out
scaling
which,
by
the
way,
we
can
classify
API
groups
in
two
major
categories:
the
ones
that
are
native
to
kubernetes,
part
of
the
API
server,
that's
code,
embedded
to
the
API
server
and
another
group
of
API
groups
that
are
part
of
CR
DS
that
you,
when
you
extend
kubernetes,
you
create
our
own
group
for
Valero,
we're
concentrating
on
the
kubernetes
for
this
testing
and
encoding
I'm
using
the
native
kubernetes
examples.
B
B
Here
now,
I
have
a
sub-directory
v1
v2
beta
1
V
2
beta
2,
and
they
point
to
the
same
object:
okay
and
how
I
know
that's
the
same
object
when
I
double-click
here
you
know
the
UID
of
the
object
across
all
those
versions
is
the
same.
The
format
which
is
different
because
V
2
beta
2
has
extra
fields
then
V
1,
which
is
being
backed
up,
is
the
preferred
so
that
I'm
explaining
this
to
the
community
to
number
one
tell
what,
where
am
I
right
and
collect
thoughts
concerns
ideas
about
how
to
tackle
this?
B
B
Jason,
this
object
being
backed
up.
It's
out,
scaling
v1
in
the
horizontal
scaling
and
have
everything
above
v1.
However
I
when
I
created
that
that
object,
if
you
see
you're,
less
apply
configuration
I
created,
it's
auto
scaling,
v2
beta,
don't
be
scared,
we're
not
losing
any
information
today
right
because
the
the
actual
logic
for
the
new
code
I
mean
the
new
objects.
Right
here
is
a
notation,
but
moving
forward
would
be
great
to
have
all
the
versions
of
the
objects
inside
of
the
backup.
B
When
you
restore
is
going
to
be
in
a
different
version
of
kubernetes,
it
happens
not
to
support
the
preferred
version
that
you
backed
up,
but
because
we're
gonna
have
some
versions
ahead.
We're
gonna
have
some
convergence
and
get
be
able
to
restore
all
the
objects.
I
think
I'm.
Thinking
ahead
right
now,
I
have
like
20
groups,
part
of
kubernetes
native
groups,
write
20
depending
off
your
deployment
can
have
more
than
that.
B
I
think
when
it'll
I
saw
OpenShift
is
40
groups,
but
think
a
couple
years
from
now,
when
we're
gonna
probably
have
hundreds
of
groups
and
that's
where
we're
gonna
setting,
you
know,
there's
foundation
to
be
able
to
backup
and
restore
those
structures
regardless
the
version
of
the
kubernetes.
You
running
gonna
stop
for
a
second
and
ask
if
anyone
has
a
question
about
it,
I.
C
B
A
D
Yeah
I,
don't
don't
have
a
big
update
this
week,
I
was
on
community
support
last
week,
a
lot
of
PR
reviews
to
do
both
from
the
core
team
and
from
community
members,
so
that
kept
me
pretty
busy.
I
did
spend
some
time
getting
my
getting
an
environment
set
up
for
testing
with
the
CSI
snapshot,
work
that
Ashish
and
Nolan
I've
been
working
on
so
I
did
that
in
Azure,
I
was
able
to
get
an
environment
up
and
running
pretty
easily.
D
If
folks
have
access
to
that
and
want
to
try
it
out,
there
I
basically
spun
up
an
aks
cluster
using
kubernetes
1.17
and
then
essentially
just
followed
the
instructions
in
the
azure
CSI
repo
which
walk
you
through.
Getting
the
CSI
drivers
stood
up
and
then
adding
all
the
the
snapshot
controllers
as
well.
So
I
did
some
testing
there
submitted
one
or
two
pr's
based
on
on
what
I
found
that
fix
a
couple
of
minor
issues.
But
now,
generally
things
are
looking
good.
D
You
know
that
doesn't
mean
we're
done,
but
for
the
things
that
have
been
done,
things
are
looking
good.
So
probably
will
continue
to
do
some
more
testing
there
and
it's
just
a
good
way
for
me
to
stay
engaged
with.
What's
going
on
there
beyond
that,
I
have
PR,
that's
been
open
for
a
couple
weeks,
2317,
which
was
doing
some
refactoring
of
where
we
have
validation
code
for
our
backup,
API
objects,
I
think
I'm.
D
A
E
E
E
This
is
a
scenario
where
you,
where,
as
an
application
user,
it
is
possible
to
use
the
CSI
drivers
to
take
volume,
snapshot
yourself.
We
also
want
to
be
able
to
back
that
up,
so
I
have
a
PR
that
is
work
in
progress
to
back
those
up
as
well.
So,
though,
those
are
the
two
updates
on
the
stuff
that
I've
been
working
on
after.
F
E
E
So
that's
what
this
PR
is
basically
interesting.
The
other
thing
was
yeah.
The
other
thing
that
I've
been
working
on
other
than
CSI
is
kind
of
trying
to
build
a
catalogue
of
standard
operating
procedures
for
backing
up
and
restoring
data
stores
and
I
wanted
to
kind
of
reach
out
to
the
community
to
see
what
you
are
all
up
to
and
kind
of
get
a
prioritized
list
of
the
data
stores
that
we
want
to
tackle
with
first
and
see
how
useful
it
is
I,
know.
E
E
C
Let
us,
let's
figure
out
which
ones
you
think
is
the
most
important
give
them
the
option
to
order
it
so
I
think
the
survey
link
allows
you
to
create
an
ordering
query
and
then
ask
them
if
anyone
is
interested
to
help
us
on
any
of
these
deliver
this
solution,
maybe
we'll
get
a
community
member
into
haba.
So
this
I
think
this
would
engage
them
a
lot
more.
A
G
G
If
anyone
from
the
community
would
like
to
review,
this
will
be
great
in
addition
to
us,
and
if
anybody
thinks
that
they
could
use
some
help,
some
walkthrough
help
just
ping
me
I'll,
be
glad
to
hop
on
a
zoom
and
do
that,
and
that
also
goes
for
the
team.
We
can
review
this
together.
If
you
think
it
is
easy
to
to
go
back
to
and
wrap
your
your
head
around
it
so
I'd
be
glad
to
do
that
any
time
and
I
was
actually
actually
asking
myself.
Why
are
we
doing
this?
G
For
example,
if
there
is
no,
if
there
is
an
invalid
backup,
storage
location,
develop
server
will
continue
running,
which
is
today.
There
is
a
lot
of
things
there
there's
a
lot
of
dependency
or
me
try
to
split
up.
So
it's
it's.
It's
CLI
changes,
but
also
changes
on
on
the
side
of
a
little
behaves,
which
I
think
a
lot
of
us
think
I
think
our
thinks
it's
an
improvement,
at
least
as
a
little
core
team
which
have
we
have
been
talking
about
this
right
and
the
other
things.
G
I
am
testing
a
PR,
that's
up
from
a
community
contributor
to
make
the
developer
help
chart
compatible
with
helm
three.
So
right
now
it's
only
comparable
with
old
help
to
and
before
when
I
started.
Looking
into
this
attire,
we
would
have
to
maintain
the
helm
two
charts
side
by
side
with
the
helm,
three
charts,
but
from
our
I
researched
recently.
G
G
D
Carly
see
just
so
you
know,
I
did
start
reading
through
the
the
doc
on
the
install
and
convict's
tough.
Yesterday,
like
you
said,
there
is
a
lot
there,
so
it's
I'm
reading
through
it
a
few
times
to
kind
of
page
it
all
back
into
my
brand
I,
do
think
it
probably
would
be
helpful
to
have
another
zoom
session
at
some
point
in
the
coming
days,
but
I
think
it'd
be
good.
D
G
C
C
Part
of
our
deliverables
to
also
I'm
assuming
the
documentation
to
tell
people
like
you
know
if
they
had
a
script
that
looked
like
this.
What's
a
new
script,
gonna
look
like,
for
example,
like
the
changes
I'm
assuming
they
can
go
through
a
documentation
and
find
everything
that
changed
but
like
how
are
we
gonna
have
folks
transition
from
before
to
the
we
don't
have
to
discuss
it
here,
but
thinking.
G
C
G
G
We
were
document
things
as
best
we
can
and
keeping
proven
as
we
find
that
people
are
having
a
hard
time.
Understanding.
One
thing
for
sure
is
that
I
think
we
so
part
of
this
of
this
of
this
change
that
we're
going
to
have
a
memo
files
as
output
from
each
command
that
can
be
run
using
customize,
so
we're
making
sure
that
it's
compatible
with
the
customize
usage
and
we're
going
to
have
example,
I
propose
we
have
a
folder
with
example.
Files
of
what
these
yellow
files
would
look
like
and
actually
I
do
have.
F
So
pull
requests
23
23.
That
was
something
I
need
to
get
back
to
this
week.
It's
taking
its
changes,
2/4
Valero,
to
take
the
CSI
snapshot,
objects
that
are
being
generated
by
CSIS
changes
and
actually
uploading
them
to
the
object
storage
so
that
we
can
use
them
for
other
things
like
getting
details
in
the
client
and
doing
syncs,
and
things
like
that.
F
So
Steve
did
a
did.
A
review
and
I
need
to
go
back
and
respond
to
that,
but
the
last
week
was
mostly
done
spent
doing
an
internal
presentation
prep,
so
I
need
to
get
back
to
this
clean
it
up
and
do
some
more
testing
on
it.
So
I'm,
basically
I,
think
I'm
at
the
same
same
same
spot,
I
was
at
last
update.
F
Let's
see
what
was
my
other
update,
something
else
I've
I
started
doing
yesterday
was
testing
and
documenting
the
helm
to
behavior
with
Valero,
so
something
we've
been
aware
of
for
a
little
bit,
but
didn't
really
have
documented
I.
Have
an
issue
for
this
I
know:
I'll
link
it
in
here
shortly
was,
if
you
have
helm,
2
charts
deployed
and
you
back
them
up
with
Valero,
and
then
you
do
a
restore
with
Valero.
F
Your
application
will
come
back
and
be
running,
but
the
helm
functionality
will
be
broken
so
like.
If
you
do
helm
list
or
helm
status,
it
won't
work
and
through
experimentation.
Yesterday,
what
it
looks
like
is
hell,
tiller,
doesn't
grab
the
releases,
the
releases
are
starting
to
Nick
maps
and
tiller
does
not
grab
the
config
maps
that
are
present
before
it
starts.
So,
if
you
restore
tiller
first
then
restore
the
config
maps
that
represent
the
releases.
Then
it
all
works.
F
Fine,
so
in
my
just
manual,
restores
just
like
doing
it
back
up
with
a
full
cluster
and
restoring
things
piece
by
piece
like
I
kind
of
this
is
what
I
came
up
with
and
without
deep
diving
into
their
docks,
but
it
looks
like
that.
Order
is
what
you
need
to
do
to
get
home
to
working,
given
the
deprecation
of
helm,
to
I'm-
probably
just
going
to
document
this,
and
if
someone
wants
to
make
a
make
a
plug-in
that
goes
and
detects
a
config
map
then
goes
and
gets
to
tiller
deployment.
F
They're
welcome
to,
and
figures
out,
restore
item
action
that
goes
and
restores
the
tiller
to
deployment
first.
That
would
be
great,
but
for
now
I
think
I'm
just
going
to
document
this,
and
then
the
next
thing
will
be
to
take
a
helm
three
chart
for
the
same
application
and
dry
it
and
see
if
it
displays
similar
behavior
I,
don't
think
it
will,
because
elm
tree
doesn't
have
tiller.
F
C
It's
it's
kind
of
weird
to
know.
You
know
help
like,
for
example,
some
some
versions
of
open
50
and
support
helmet
all
because
of
Taylor.
So
I
was
very
celebrated
in
the
community
that
when
Taylor
went
away,
however,
it
takes
a
long
time
for
IT,
admins
and
infrastructure
to
move
to
the
latest
and
greatest
home
actually
just
has
to
graduate
in
ciencia
for
now
today,
today,
actually,
so
you
know,
you
can
think
that
people
will
move
to
home
3
how
many
of
them
have
moved.
I
just
takes
a
while.
It
really
does
I
would.
F
G
F
F
F
A
H
E
E
I
I
Is
the
second
one
I
think
the
less
I
just
pasted
it
into
the
notes?
Perfect,
okay,
okay,
now
see,
yeah
I
can
see
it's
like
yes,
so
I'm
working
on
Casca,
which
is
a
kubernetes
operator
for
Cassandra.
It
allows
you
to
make
to
create
a
cluster
we
just
by
just
creating
a
custom
resource,
and
you
can
do
many
operations
like
like
adding
nodes,
getting
down
triggering
some
actions
on
the
different
parts.
I
But
the
issue
that
we
have
is
that
we
can't
back
do
any
backups
right
now,
so
we're
looking
to
two
solutions,
the
first
one
being
the
lair.
Oh
and
the
second
one
being
cane
and
cane,
is
more
just
Cassandra
focus.
Only
so
I
wanted
to
give
a
shot
to
Valero
yeah,
but
they
should
have.
Today
are
the
one
I
listed,
say
yeah
and
in
short,
it's
that
I
cannot
specify
what
I
wanna
back
up
for
example.
I
So
we
have
like
I
can
snapshot
the
whole
data
volume,
which
is
a
local
persistent
volume
and
it
works
fine,
but
I
cannot
specify
what
I
want
to
back
up.
I,
looked
into
annotations,
I
did
a
manual
rustic,
backup,
it's
a
on
premise
and
it
works
as
expected
if
we
had
like
the
annotations
available,
but
but
then
there
are
other
issues
like
hooks
not
being
able
to
see
what
those
I
mean.
Any
information
regarding
the
current
backup
like
what
are
we?
I
So
I
know
it's
a
lot
of
requirements,
but
yeah
I
want
you
to
bring
this
up
and
know
if
this
is
something
that
others
were
able
to
do,
I
don't
know
we
work
around
or
any
other
solutions.
If
this
is
something
that
you
had
in
mind
for
any
other
databases
or
oh,
no,
yeah
I
wanted
to
bring
this
up.
I
feel
like
it
makes
sense
and
that
very
always
a
great
tool.
I
I
would
like
to
be
able
to
use
video,
because
that
way,
I
can
use
very
off
not
only
for
Cassandra,
but
anything
else,
as
opposed
to
the
other
solution
that
we
have.
That
is
only
Cassandra
focused
and
we
don't
work
with
any
other
solution.
I
mean
it
does
the
job
and
can
I
mean
it's
it's
a
simple
tool
and
can
do
more
than
what
vero
can
do
today.
Just
for
Cassandra,
but
yeah
Barrow
seems
to
be
like
a
more
global
general
solution
for
backups
I.
E
Yes,
so
when
yeah
I
would
say
yes
and
no,
what
we
wanted
to
do
was
stack
up
whole
of
Cassandra,
just
from
a
community's
perspective
like
how
do
you,
what
are
the
best
best
practices
to
do
we
to
back
up
Cassandra
say:
do
you
want
to
use
less
stick
you
want
to
use
the
volume
snap,
shutter
plugins
or
how
else
to
back
that
up
and
like
if,
if
there
are
C
or
D
so
that
require
that
work
with
the
data
store?
How
do
we
back
that
up?
E
The
reason
why
I
asked
to
open
this
issue
was
to
kind
of
see
what
everybody
else
was
up
to
like.
Is
this
a
requirement
for
most
people
using
Cassandra,
so
III
think
the
last
we
were
talking
about
this
in
the
slap
channel
was
to
use
rustic
and
let
Chris
take
scrape
just
part
of
the
file
system
instead
of
the
whole
file
system.
Is
that
Christ
yeah.
I
I
Same
for
yeah,
you
said
there
are
ways
to
backup
Cassandra,
so
you
can
use
a
snapshot,
but
it's
best
to
have
the
same
snapshot
down
on
all
the
nodes.
At
the
same
time,
and
even
if
Cassandra
files
are
immutable,
you
don't
want
those
files
to
disappear.
When
you
start
the
backup,
so
I
don't
know
exactly
how
rustic
works
internally
if
it
keeps
handles
on
the
files,
just
in
case
they're
deleted,
but
if
it
doesn't
yeah,
we
don't
want
to
lose
them.
So
there
is
a
tool
on
Cassandra
to
create
snapshots.
D
D
That's
that's
not
necessarily
too
difficult
to
do.
We
have
stayed
away
from
doing
sort
of
individual
file
recovery.
That's
something
that
we've
avoided
in
the
past,
and
that's
that's
not
to
say
that
we
couldn't
revisit
that
decision,
but
when
it
has
come
in
from
the
past
week,
we've
stayed
away
from
it
and
I
guess.
The
other
thing
is
kind
of
I.
D
It's
so
it's
it's
in
our
backlog,
somewhere,
I
think
we're
waiting
for
some
design
work
to
be
done
there,
but
that
would
that
would
definitely
cover
kind
of
part
of
the
needs.
Here
so
yeah,
it's
a
I
mean
I,
don't
have
like
a
clear
answer,
but
you
know
some
of
the
things
we
have
thought
about.
Some
we
haven't,
but
we
probably
need
to
do
a
little
bit
more
analysis
here
to
kind
of
figure
out
what
we
want
to
do.
F
One
thing
I'll
toss
out
and
I
don't
know
if
this
is
the
right
way
to
do
it,
but
it's
something
that
came
to
mind
for
me
is
especially
with
rustic.
It
may
make
more
sense
to
do
a
two-step
process,
where
you
use
the
casandra's
snapshotting
tool
to
send
the
snapshots
to
some
location.
That
rustic
then
backs
that
up
instead
of
backing
up
the
cassandra'
files
themselves.
I
don't
know
if
that's
betters,
but
that's
kind
of
why
that
came
to
mind
as
you
were,
describing
how
this
was
working.
F
I
I
So
that's
why
I
had
in
mind
just
tell
rustic
to
just
back
at
the
right
location,
I
mean
everything
can
be
done.
I
just
felt
that
you
know
being
able
to
just
target
those
files
will
make
it
just
faster
and
would
wouldn't
need
a
new
volume.
Another
volume,
just
you
know,
to
temporarily
store
those
files.
F
I
F
Yeah
I
think
the
only
thing
then
is
like
Steve,
said:
the
granularity
and
and
I
get.
Why
why
you
say
like?
If
you
only
deleted
one
one
table,
you
don't
want
to
wait
for
the
whole
restore,
but
so
far
Valero
in
terms
of
PDS
hasn't
really
done
anything
beyond
restoring
the
PV.
The
granularity
has
just
been
at
the
pv
layer.
I
E
A
A
D
D
J
F
L
The
describe
just
shows
tells
you
the
number,
but
doesn't
numerate
them
in
the
output,
and
you
got
to
look
at
the
logs
for
the
backup
to
see
those
errors
and
restore
seems
to
do
the
opposite,
actually
lists
the
errors
there,
and
sometimes
those
errors,
don't
actually
show
up
in
the
log
at
all
wondering
if
doesn't
have
thought
about,
maybe
trying
to
make
those
to
kind
of
behave
in
a
similar
way.
You
know
giving
kind
of
more
information
in
both
cases,
yeah.
D
That's
a
discrepancy,
that's
been
around
for
a
while,
and
we
we
have
talked
about
it.
I've
talked
about
I
think
primarily
making
restore
work
more
like
backup,
so
basically
just
putting
everything
in
the
log,
but
potentially
also
making
some
CLI
improvements
so
that
you
know
whether
it's
backup
or
restore
you
can.
D
You
can
potentially
just
display
all
of
the
error
messages
from
the
log
directly
in
the
the
output
of
describe
so
I
think
that
that
would
kind
of
be
the
best
of
both
worlds,
because
it's
you
know
it's
a
consistent
experience
on
both
sides,
but
also
easily
accessible
through
the
CLI.
Is
that
does
that
sound.
L
Like
is
that,
for
me
personally,
it's
very
convenient,
see,
there's
a
boots
in
describe
it's
right
there
and
a
bolero
described
for
the
log
file,
but
at
the
same
time,
I
run
into
cases
where
you
know.
Users
are
looking
at
this
and
they
don't
necessarily
have
the
velaro
in
a
client
installed,
because
they're,
just
using
bolero
was
kind
of
a
component,
and
you
know
some
of
that'll
work
through
getting
them
to
install
the
velaro
client.
So
they
can
look
at
the
output
there.
L
F
Like
this,
this,
like
I
I,
was
doing
an
internal
presentation
and
I
was
like
stepping
through
the
the
describe
output
on
both,
and
this
actually
like
hit
me
in
the
face,
because
I
was
like
wait.
A
second
as
I'm
stepping
through
both
of
these,
they
aren't
the
same
and
yeah
I
would
like
to
unify
them
and
I
kind
of
agree
that,
having
having
that
summary
of
like
hey,
there's
some
errors
in
the
backup.
What
were
those
errors?
I
personally
would
like
to
see
them.
Bubbled
up
yeah.
L
In
describe
and
I
noticed
that
that
difference
for
a
while
it
didn't
surprise
me
recently
was
realizing
that
the
restore
errors
is
actually
the
opposite
of
just
right
there
in
the
summary
and
they're
in
you,
know,
they're
on
the
file
storage.
But
if
you
actually
look
at
the
pod
logs
you're,
not
seeing
them
at
all
in
some
cases
right.