►
Description
Jan 12, 2021
Status Updates
dave
- e2e tests running in Azure
- Working on e2e tests running against all clouds as install target
nrb
Out most of last week due to family emergency
Reviewed https://github.com/vmware-tanzu/velero/issues/3191 for the cause of throttling log messages
Increase --client-burst & --client-qps if you see this. We’ll get this increased in v1.6
Opened an issue for some of Velero’s upcoming rearchitecture, https://github.com/vmware-tanzu/velero/issues/3229
Covering community support w/ Carlisia this week
bridget
Community support last week
Working on multiple credentials support
Need to make a decision on approach to take
Need to follow up on some PRs that have been updated
carlisia
- This week will be PR reviews and addressing code reviews on my PRs
- Sharing community support with Nolan
- PTO this Thu and Fri
Discussion Topics
dave
Azure issues with memory limit and plug-in failure. Hard to debug some plug-in failures.
nrb - There is an issue w/ Azure plugin where object storage chunk uploading uses the maximum memory of the Velero pod.
nrb - Adding ~10 log messages into the Backup CR seems useful, especially when it can’t upload to object store
Dylan
Ability to selectively patch resources on per restore basis.
Use case is Secrets & CRs
User wants to continually patch Secrets vs replace it
Operators watching CRs, they wanted to make sure the operators were going to keep working
Current design of plugins is very much all or nothing.
Plugins should be able to allow mutations or patching
Give post hooks the backup manifest to help
When patching, this is a merge operation. That may result in a merged, incorrect object at the end
Can we add a way to download individual JSON from a backup to then re-submit with Kubectl?
This exists with the velero backup download command, but you need to take it out of a tarball
Can we modify the restoreitemaction plugin API to return an action?
The plugin could identify the relevant fields and return a patch object & the patch action
Action - Alay and Dylan to work on design doc
Phuong: (phuong.n.hoang@dell.com) invite to design meeting for Velero plugin timeout 9am PST Friday Jan 15th. This is the Zoom meeting: https://Dell.zoom.us/j/91078486282?pwd=Yk9sU1UwT0NScXE2Q2oyRGV3K002QT09
This is for timing out plugins generically
Enhance the plugins to use the Context for timeouts
Please add to https://groups.google.com/g/projectvelero
carlisia
hit a temporary snag with using GH Discussion: https://github.com/github/feedback/discussions/3105
is HackMD getting slow again? Time to archive?
nrb
Current v1.6.0 status: https://github.com/vmware-tanzu/velero/projects/4
Testing is moving along
Need to get multiple credential support design approved
bridget
Can we do a v1.5.3 release in the meantime before 1.6?
nrb: :thumbsup:
We need to include some patches like properly deleting when DeleteItemActions are missing, and fixing include/exclude plugin logic (I don’t have a full list)
Contributor Shoutouts
@justenwalker/Bryson Shepherd (on Slack): fix: missing results in ListObjects/CommonPrefixes
@eyenx: [velero] feat(schedule): add annotation templating
A
B
Yeah
the
dangers
of
being
the
first
one
in
the
hack,
md
yeah,
so
I've
been
working
on
getting
the
ed
tests
running
against
azure
ran
into
some
difficulties,
just
getting
things
working
on
azure
I'll
go
into
those
in
the
discussion
topics,
but
those
now
work
and
I'm
working
on
getting
everything
to
work
versus
all.
The
different
clouds
is
actually
running
in
the
clouds
getting
installed
running
there
and
then
we'll
start
pulling
it
into
the
cicd
pipeline.
C
So
picking
up
with
yesterday,
we
did
some
community
bug
report
triaging,
and
I
reviewed
this
issue
3191,
which
folks
may
have
seen
some
throttling
issues
in
in
logs.
Basically,
there's
client
go
logging
that
if
client
law
throttles
you,
you
can
increase
the
I'm
not
even
sure
how
to
determine
it's
like
this
weird
bucket
token
bucket
structure
that
it
uses.
C
But
if
you
get
these
throttling
requests
took
over
a
second
I'm
gonna
tag,
abby
to
add
some
documentation
for
this
and
then
we'll
also
upgrade
increase
the
defaults
on
these
values.
They're
just
set
really
low
and
looking
around,
we
can
set
them
much
higher.
So
yeah
did
some
investigation
on
that.
It
looks
like
a
fairly
fairly
small
fix
from
code.
Just
took
a
little
bit
to
dig
into
that.
C
C
This
issue
is
a
essentially
an
epic
and
a
very
long
term
focused
thing
and
not
anything
that
we're
gonna.
It's
not
something
we're
acting
on
in
the
next
three
months.
Necessarily,
but
maybe
I
see
dave
wagging
his
head,
but
this
is
just
just
letting
people
know
kind
of
the
direction.
We're
heading
we're.
Looking
at
adding
a
data
movement
layer
is
what
we're
calling
it
for
generic
file
system
level
data
copy
and
the
idea
would
be.
C
C
It's
not
going
to
be
a
thing
that
we
just
drop
in
it'll,
be
something
we
get
to
along
the
way,
but
just
letting
folks
know
that's
out
there
and
one
of
the
things
we
hope
it
would
enable
is
things
like
taking
a
snapshot
and
then
also
doing
file
system
copy
off
of
that
snapshot.
So
you
could
do
things
like
rustic
or
file
system
level,
backup
off
of
a
snapshot
versus
one
or
the
other.
So
just
letting
people
know
this
this
we're
starting
to
plan
for
a
2.0
re-architecture.
C
C
D
D
So
I
think
I'd
like
for
us
just
to
kind
of
come
to
consensus
so
that
we
can
definitively
kind
of
start
and
make
some
progress
on
this,
given
that
we
want
to
get
it
in
for
one
sixth-
and
apart
from
that,
oh
sorry,
aside
from
that,
yeah
there's
been
a
few.
There
were
a
few
pr's
that
I
reviewed
and
that
I
requested
changes
on
and
there
have
been
updates
to
those.
So
I
owe
some
follow-up
reviews
to
some
folks,
so
I
will
try
and
get
on
that
as
soon
as
possible.
D
So,
for
that
I
would
like
the
maybe
I'll
I'll
just
set
up
like
a.
I
don't
know
what
folks
preferences
are,
whether
we
just
set
up
a
call
just
to
kind
of
revisit
where
we
were
with
it
before
the
break,
and
that
might
just
be
like
the
easiest
way
to
get
everyone
the
opportunity
for
us
all
to
chat
on
it,
and
so
I'll.
Do
that
and
that's
kind
of
that's
all.
I
really
need
from
it.
D
I
know
carly's
already
made
some
some,
I
mean
to
start
she'd,
put
up
a
pr
for
try
to
set
their
credentials
from
the
cli
and
but
we
kind
of
need
to
make
the
decision
on
once
we
get
them
from
the
user.
Where
do
they
go
from
there?
So
that's
kind
of
what
we
need
to
discuss
so
I'll
set
up
a
call
with
folks
and
we'll
go
from
there.
E
B
Yeah
so
so
I
was
trying
to
get
the
indian
tests
running
on
azure,
and
this
was
my
first
time
actually
working
with
azure.
So
everything
was
kind
of
new
for
me.
So
I
got
I
was
able
to
get
like.
Was
it
aks
up
and
running
pretty
pretty
easily?
And
then,
when
I
started
going
through
the
plug-in
install
there
were
just
a
whole
bunch
of
different
directions.
You
could
go
in
and
it
took
me
a
while
to
to
figure
that
out-
and
I
was
so
so
one
thing
would
be.
B
B
Yeah,
I
don't
think
anybody
else
is
using
as
sure
so
so
there
is
that
and
then
I
finally
got
the
plug-in
to
the
point
where
it
actually
ran
or
it
was.
You
know
I
had
to
hold
my
config
setup
and
then
it
would
crash
at
the
end
and
I
get
a
backup
failed,
but
no
nothing
in
the
backup
log.
That
indicated
why
it
failed,
and
I
finally
looked
into
the
the
pod
logs
and
there's
an
indication
there
that
you
know
of
the
error.
B
F
C
B
C
B
The
memory
limit
on
valero,
but
that's
a
separate
but
but
the
other
issue
I
had
was-
I
couldn't
even
get
to
the
point
where
I
knew
that
was
happening
right
and
as
a
user
as
a
relatively
stupid
user,
I'm
going
it
failed.
There's
no
log
messages,
because
what
the
heck
right
and
there's
actually
even
a
message
that
cometh
that
comes
up
that
says,
plug-in
exited,
but
it's
debug
level
I'm
trying
to
figure
out
where
it
actually
gets
output
from
so
there's
some
room
to
improve
the
login.
B
I'm
going
to
dig
into
it
more
and
just
kind
of
dig
through
this
case
and
do
that.
But
I
think
we
could
do
some
things
like
when
we
mark
a
backup
has
failed.
We
should
really
either
relog
at
the
end
of
the
log
would
be
okay
if
we
could
put
it
into
the
the
cr.
That
would
be
handy
just.
Why
did
you
fail.
C
C
B
Think,
even
like
general
failures,
the
failures
may
not
happen
at
the
end
of
the
log,
so
you
see
backup
failed
and
then
you
have
to
go
and
dig
through
the
log
to
figure
out
what
actually
caused
it
to
fail
right,
and
so
I'd
like
to
have
that
information.
Since
we
have
that
information,
we
know
why
we
failed.
We
have
the
list
of
errors
that
caused
us
to
fail.
We
should
put
it
somewhere
where
it's
easy
to
find.
C
B
C
And
we
show
that
and
describe
you
know
decorate
that
in
client.
Somehow
that
says:
hey
you
know
we
can't
print
them
all.
Something
really
went
wrong,
go
and
go
look
in
the
logs,
but
for
a
case
like
this,
it's
super
handy
because
we
can't
even
have
a
log
because
we
can't
get
the
object
store
so
yeah,
I'm
not.
B
I'll
put
in
a
pr
for
that
then
and
and
I'll
pull
something
together.
Then
we
can
see
if
we
want
to
put
that
in
and
then
the
yeah,
the
and
the
indian
tests
need
to
capture
logs
as
well,
which
I
think
bridget.
We
could
probably
work
on
that
together
as
part
of
the
valero
debug
command,
we're
talking
about,
because
we
should
be
getting
the
same
information
anyway.
So
we
run
the
tests.
B
That
sounds
good
yeah,
and
so
then
I
want
to
bump
up
the
limit
on
valero
overall.
So
the
memory
limit,
I
think,
512
seem
to
work,
but
we
really
need
to
run
some
bigger
tests
and
stuff
in
there,
but
I
think
it'll
be
good
if
we
have
the
end
end
test
running
right
now.
The
indian
test
is
relatively
small
scale.
C
C
B
C
H
Hey,
I
was
just
curious.
I
was
looking
through
the
previous
issues
on
valero
and
I
didn't
I
could
have
sworn
I've
seen
something
about
this
come
up
in
the
past,
but
I
couldn't
find
it.
We
had
a
folk
ask
around.
They
essentially
wanted
a
server
configuration
that
would
force
valero
to
patch
the
resources
on
the
destination
cluster.
H
To
me,
that
seemed
a
little
scary,
obviously,
but
they
had
a
pretty
solid
use
case
for
it
specifically
around.
I
believe
it
was
secrets
now
I
ideally
I'm
sure
you'd
want
to
do
that
with
like
a
label
selector
or
something,
but
has
this
come
up
in
the
past?
I
know
that
valero
explicitly
takes
the
approach
not
to
patch
existing
resources
on
the
cluster,
but
is
there
an
issue
around
it
that
I'm
not?
I
couldn't
find
that's
kind
of
my
only
question.
Maybe
you
guys
have
thought
about
this
in
the
past.
E
B
Of
work
on
that
we've
been
looking
at
different
patch
mechanisms,
and
somebody
had
the
question
come
out
a
little
while
ago
and
our
preferred
response
was
hey.
Why
don't
you
write
like
a
restore
item
action
to
do
your
specific
thing
but
restore
item
actions
are
pretty
heavy
weight,
especially
when
they
live
outside
the
core
code,
so
we've
been
looking
at.
Can
we
make
a
kind
of
a
generic
one
that
might
execute
a
script
or
something,
but
that's
very
focused
on
searching
for
and
replacing
patching
resources?
C
Okay,
yeah
we've
had
we've
had
we've
had
similar,
if
not
exactly
the
same
requests.
Where
folks
have
asked,
can
we
like
force
override
like
if
a
resource
exists,
can
we
just
like
pave
over
it,
delete
it
and
recreate
it?
So
I
think
this
is
different
in
that
you're
asking
to
patch
it
to
make
it
match,
instead
of
deleting
it
to
make
it
match.
E
H
Me,
but
my
understanding
was
that
there
was
two
solid
use
cases
they
had.
One
was
that
they
had
an
application
that
was
reading
secrets
and
when
they
were
essentially
secrets,
kind
of
constantly
being
mutated
and
so
they're
using
it
as
a
migration
tool.
So
they
wanted
to
migrate.
Whatever
the
data
on
the
source
side
of
the
destination
side,
they
didn't
want
to
skip
over
the
secret
if
already
existed,
they
want
to
continuously
patch
it
with
the
newest
data.
That
was
one
and
then
the
second
was
custom
resources.
H
Source
clusters,
cr
definition
and
the
destination
side,
and
they
wanted
to
patch
those
resources.
So
you
know
it
was
a
little
scary,
because
if
we
enable
this
for
them,
you
know
openshift's
got
a
whole
set
of
default
secrets
and
stuff
that
you
know
we
don't
want
to
patch
so
like
enabling,
on
this
on
a
resource
level,
seemed
a
little
scary
to
us,
but
we
kind
of
said:
hey,
maybe
there's
a
way
you
can
just
set
a
flag
on
the
player
server.
C
Yeah
so
the
operator
case
watching
crs
is
kind
of
a
general
thing
that
we've
been
trying
to
figure
out.
We
have
that
issue
with
cappy.
We
have
that
issue
with
some
vmware
operators
that
if
the
operator
doesn't
know
how
to
handle
an
external
program
like
valero
or
conveyor
or
whatever,
writing
writing
those
crs,
then
the
operator
might
do
the
wrong
thing,
so
I
can
see
why
they
might
want
to
patch
it.
H
F
F
My
very
beginning,
ideas
have
a
a
whole
cup
like
instead
of
a
beyond
restore
item
like
a
new
theater
item,
where
you
don't
necessarily
store
the
original
item,
but
you
make
an
action
and
you
return.
Something
else
in
this
case
would
be
a
patched
object
or
even
a
completely
different
object,
but
definitely
something
will
be
like
another
hook
on
the
plug-in
level.
C
C
F
F
H
B
E
C
Yeah,
other
than
other
than
having
those
post
hooks
like
go
download
the
backup
resource
list,
which
is
possible,
because
that
is
part
of
backup
describe
dash
dash
details.
But
that's
I
mean
that's
hacky,
but
it's
also
that
doesn't
give
them
stuff
like
did
it
actually
try
to
restore
and
fail?
Because
it's
already
there,
which
is
what
you
would
want
to
do
in
a
patching
scenario
right
now,.
H
If
so,
if
I
were
to
open
issue
around
this,
I
think
the
concrete
example
that
I
was
given
around
the
secrets
that
I
think
makes
the
most
applicable
sense
would
be
like
a
credential
secret.
So
let's
say
siri
connects
credentials
to
a
database
and
that
cigar
gets
corrupted,
so
someone
overwrites
it
by
accident,
and
then
they
want
to
run
our
store
to
get
the
secret
back
into
its
original
spot
and
then
make
sure
you
know
all
the
other
application
specific
stuff
is
up.
H
F
F
C
C
A
lot
of
a
lot
of
people
have
wanted
the
behavior
of
where
we
delete
all
objects
and
then
restore
all
objects
to
force
it
to
match
the
backup
I'm
more
comfortable
with
a
patch
yeah.
I
agree.
I
agree,
though
I
don't
think
we
can
patch
everything
because,
like
patching,
a
pv
isn't
logical
so
because,
especially
because
those
are
those
are
not
mutable,
so
there
may
be
constraint
external
constraints
put
on
us
here,
but
for
secrets
that
makes
sense.
J
C
J
File
system,
restore
idea
or
the
the
individual
file
restore
right,
like
you,
don't
get
the
ability
to
say
actually
merge
these
files
right
like
right,
which
is
what
patch
would
do
like
patch,
actually
merges
to
the
existing
thing.
With
the
backed
up
thing
in
some
ways,
you
almost
want
to
say
either
take
the
newly
backed
up
thing
or
keep
the
old
thing
for
a
given
file
not
and
like
overwrite
it
rather
than
like,
merging
them
together.
C
B
So
I'm
thinking
one
thing
we
might
provide
is
a
way
to
extract
individual
resources
or
all
the
resources
as
yammels
out
of
the
backup,
and
then
you
know
the
user
can
pick
and
choose
and
apply
them
using
cubecto,
because,
like
this
case
like
I
want
to
apply
credentials
applying
like
a
gross
level
thing
of
like.
Oh,
let's
overwrite
credentials,
it's
like
that's
really
scary
to
me.
F
F
Creating
a
new
hook
maybe
can
get
away
with
all
these
unique
cases
put
in
this
basket.
This
new
hook,
where
you
can
pretty
much
do
whatever
you
want,
you
can
instead
of
restore
yes
or
no,
you
can
patch
it.
You
can
mutate,
you
can
read
again
and
make
another
action,
but
this
is
pretty
much
open
up
for
you
to
have
a
go
client
in
in
into
this
manipulation.
G
That
actually
actually
have
another
alternative,
and
just
just
trying
to
summarize
the
problem
we
are
solving
here
is
that
we
don't
want
all
or
nothing.
We
want
an
ability
to
stay
pick
and
choose
between
clients.
So
in
some
ways
we
want
x
and
extensibility
of
the
system.
So
can
we
do
it
on
the
restore
item
plugin
instead
of
just
returning
an
item?
It
will
also
return
an
action
along
with
it.
J
G
H
C
Yeah,
I
I
think
I
do
like
that
approach
with
the
caveat
that
it
since
it's
a
new
return,
it's
an
additional
return
field.
It
may
break
additional
existing
restore
item
actions,
so
I'm
not
opposed
to
it.
We
just
need
to
handle
backwards,
compatibility
yeah.
That
seems
like
a
good
middle
ground
to
me,
but
I
wouldn't
I
wouldn't
say
no
to
a
design
document
on
this
to
allow
allow
that
functionality.
G
I
I
Yeah,
so
I
think
we
talked
about
this
long
time
ago
that
I
blocked
the
ball,
because
I
got
sucked
into
other
issues
at
work.
Basically,
we
have.
We
want
to
enhance
the
valid
plugin
to
handle
camera,
so
I
have
a
feedback
from
the
previous
design
proposal
and
about
you
know
having
the
housing
will
come
out
through
the
the
to
the
executor
incremental
of
the
plugin
so
that
it
can
come
out
and
clean
up
accordingly.
I
I
Implement
that
time
out
and
what
should
we
do
and
so
on
and
so
forth?
So
I
want
to
invite
you
know
our
invite
people
here
to
come
to
this
meeting
and
we
can
talk
in
detail.
I
post
my
email
there.
So
if
anyone
who
wants
to
be
invited
in
the
meeting,
please
email
me,
I
will
invite
and
I
also
have
a
zoom
link
there.
It
will
be
this
friday
at
9
00
a.m.
Pacific
time,
so
we
can
talk
in
detail
there.
I
A
I
I
think
the
design
that
I
intention
to
see
for
any
plug-in
you
need
to,
because
I
I
think
that
originally
my
focus
is
only
for
myself
for
my
company
right.
It's
only
do
I'm
interested
only
in
the
backup,
but
the
way
I
look
at
this
is
that
it
it.
If
you
enhance
the
the
little
plug-in
time-out
it
should
be
applicable
to
anyone
right
to
anyone
who
are
who
have
any
needs
to.
I
You
know
work
with
this
hammer,
because
I
also
see
that
the
restore
also
now
might
have
some
time
out,
or
maybe
the
snapshot.
There's
a
volume
snapshot
plug-in
also.
I
don't
know
if
anyone's
interested
in
it
yeah,
but
I
I
think
I
actually
I
I
anticipated
that
people
might
be
interested.
So
why
don't
we
make
this
available
to
all
the
plugin
types
and
then
it's
up
to
the
implementer,
whether
they
decide
to
implement
that
function
or
not
right,
we
might
have
to
make
sure
that
it
is
network
compatible.
I
A
E
I
just
wanted
to
let
everybody
know.
We
had
a
little
snag
with
using
the
discussion
github
discussion
to
for
support
because
they
turned
off.
They
had
a
bug
and
they
had
to
turn
off
the
transferring
of
each
issues
to
discussion
topics,
and
so
we,
I
don't
think
we
should
make
a
fuss
about
it
or
announce
that
we're
using
that
until
that
is
resolved,
they're
working
on
it,
though,
but
I
don't
know
when
that's
gonna
be
in
effect,
like
whatever
fix
they
have
to
put
in.
E
D
C
So
we
are
moving
forward
with
the
testing
where
we
really
need
to
pick
it
up
is
with
getting
the
multiple
credential
provider
or
multiple
credentials.
Design
approved
that's
kind
of
our
our
big
ticket
thing
aside
from
tech
debt,
the
various
other
things
in
the
list
are
ordered
by
priority
from
top
to
bottom
and
that
that
stuff
needs
to
maybe
cut
in
order
to
work
for
time.
Some
of
it
could
go
into
a
point.
Release
folks
are
certainly
welcome
to
grab
some
items
off
of
this
backlog.
C
We
do
have
some
teams
here
at
vmware
that
are
going
to
be
helping
I'm
going
to
have
a
more
prepared
review
of
of
this
board
next
week,
but
I'm
I'm
gonna
integrate
going
over
this
release
each
week
in
the
community
meeting
yeah.
This
is
definitely
something
we
we
need
to
be
taking
a
look
at
more
regularly
and
letting
people
know
where
we
are
so.
The
the
big
ticket
items
we
have
in
motion
right
now
are
the
the
the
end-to-end
testing
and
the
credentials.
D
This
was
just
a
follow-up
just
because
I
know
there's
no
point
about
the
one
six
release:
there
were
a
few
pull
requests
that
have
been
merged
recently
and
there
are
a
few
bugs
related
to
that
which
mean
you
wanted
to
get
in
for
one
five.
Three
and
some
folks
have
been
asking
about
them.
So
I
was
wondering
if
we
could
take
that
on.
D
I
don't
know
if
there's
anything
specific,
that's
blocking
us
from
doing
this
or
whether
that's
I'm
happy
to
kind
of
just
take
this
on
and
try
and
gary
site.
I
also
wanted
to
follow
up
it's
one
of
the.
The
pr
is
one
of
the
contributor
shout
outs
below,
but
I
was
also
I
just
wanted
to
ask
as
well
about
patch
releases
for
plugins.
C
So
to
answer
the
questions
about
1.523,
my
intention
was
to
get
them
out,
get
that
out
last
week
or
this
week
the
the
main
blocker
there
is
all
pretty
much
all.
Excuse
me
all
the
stuff
that
I
wanted
to
put
in.
There
is
closed,
so
I
need
to
go
back
and
review
exactly
what
items
to
put
in
there,
but,
like
I
mentioned
your
pr
with
properly
deleting
when
I
think
it
was
delete.
Item
actions
are
missing.
C
You
know,
that's
hitting
the
vsphere
plug-in
team
and
then
raphael
wanted
a
fix
for
include,
exclude,
plugins
and
there's
some
others
that
we
need
to
get
included,
and
I
don't
know
what
all
of
them
are
to
answer
the
the
plug-in
release.
Generally,
we've
been
grouping
them
together,
but
there's
no
reason
like
if
there
is
a
fatal
flaw
in
one
of
them.
There's
no
reason
why
we
couldn't
release
a
plug-in
out
of
step.
D
C
D
D
The
backups
using
the
blair
cli
and
what
was
happening
behind
the
scenes
was
the
azure
plugin
can
return
up
to
5000
blobs
in
a
get
request,
but
it
might
not
always
return
them
and
you're
supposed
to
paginate
through
the
results
and
the
azure
plug-in
was
not
paginate
through
the
results,
which
meant
that
the
backup
sync
controller
thought
that
there
were
some
items
that
were
missing
and,
as
a
result,
was
deleting
the
associated
custom
resources
in
the
cluster.
So
the
backups
were
still
existing
in
storage,
but
flair
thought
that
they
didn't
exist
anymore.
D
So
that
was
why
this
was
a
pretty
serious
issue
for
for
them.
I
mean
like
there
was
no
loss
of
data,
but
it
was
just
it
impacted
the
experience
so
yeah,
so
justin,
walker
and
bryson.
They
debugged
this
and
justin
submitted
a
pr.
So
that's
one
that
I
think
would
be
a
really
good
candidate
for
a
patch
release
for
the
azure
plugin.