►
From YouTube: Velero Community Meeting/Open Discussion - Oct 27, 2020
Description
Oct 27, 2020
Status Updates
carlisia
Suport-a-thon last week
Experimented with an internal tool to boot up an AWS EC2 cluster
Tested and documented
Neat (but dangerous) OSS tool to decommision AWS infrastructure: https://github.com/rebuy-de/aws-nuke
Sample config file: https://gist.github.com/carlisia/c687505ff9b89b429557f897659e1c49
PR reviews + Helm
ashish
Support-a-thon last week
Community support this week
Have you heard about our Velero Office hours?
this opens another hackmd page
dsmithuchida
Support-a-thon last week
Working on basic e2e test, mostly working
nrb
Support-a-thon last week
Revisting our bug triage process
Released v1.4.3 and v1.5.2 last week with a security fix - please upgrade when you can.
Blocker - several design PRs are waiting on other maintainers to review
Discussion Topics
Rafael @brito-rafa , Frankie @codegold79 and Jenting @jenting - Migration doc: https://github.com/vmware-tanzu/velero/issues/2551 . About K8s API Groups: http://www.rafaelbrito.com/2020/04/kubernetes-api-groups.html
Alay: Provide more visibility on reasons for PartialFailure by reporting the errors on velero CR
The CRs don’t provide errors directly, just he number
CRs are limited to 1.5mb usually due to etcd
There’s a mismatch between restore & backup log formats, making just parsing difficult
This could be a subresource backed by a web hook, not a CRD
Community Shoutouts
@a-mccarthy: fix minio code samples
@cvhariharan: Add warning to velero version cmd.
@doughepi: Adding fix for restic init container index on restores.
@jenting: [velero] Restrict release on upstream repo
@mukulgit123: [velero] Correct syntax of iam role pod annotation in values.yaml
@machine424: [velero] Add priorityClassName in the values.yaml file to make it ‘official’
A
B
We
all
worked
on
that
and
it
was
basically
we
triaged
and
categorized
and
investigated
what
we
tried
and
categorized
all
the
support
issues
that
were
open
and
investigated
some
and
we're
getting
a
much
better
process
to
handle
that
I'm
talking
about
support
issues,
and
so
that
happens
I
on
friday,
I
started
looking
into
an
internal
tool.
Well,
I
started
looking
for
like
a
the
fastest
way
to
bring
up
a
aws
cluster,
an
ec2
cluster
and
got
some
a
bunch
of
options
then
turned
out
I
found
out.
B
We
have
an
internal
tool,
it's
basically
a
jenkins
job
that
that
triggers
this
tool
and
the
tool
uses
tkg
to
put
up
a
cluster,
and
I
just
finished-
I
mean
I
ran
it.
A
bunch
of
times
ran
into
all
kinds
of
issues
kept.
The
bug
in
it
wrote
out,
wrote
out
the
documentation
so
I'll
sync
up
with
the
with
the
internal
team
later.
If
anybody
wants
to
use
this,
so
it's
pretty
cool,
you
just
press
a
button
pretty
much
and
40
minutes
later
boom.
You
have
a
cluster,
I
love
it.
B
I
know
there
are
a
lot
of
options
that
does
exactly
this,
but
this
I
don't
have
to
type
anything.
I
just
press
literally
press
a
button,
it's
beautiful
and
it
works,
and
a
part
of
that
is
this
neat
tool,
but
very
dangerous
warning
warning.
If
people
don't
know
you
can
use
this
with
the.
If
you
have
a
dc2
cluster,
you
can
use
this
to
decommission
your
stuff
or
delete
anything
anything
not
for
production,
so
you've
been
warned,
but
this
is
really
neat,
so
just
run
a
command
and
everything
gets
decommissioned.
B
B
B
B
D
C
From
work
for
a
couple
of
weeks
and
came
back,
we
had
the
support,
and
now
I'm
on
community
support
this
week,
so
I
am
kind
of
trying
to
make
more
progress
on
our
supportathon
progress
that
we
already
made
and
also
address
any
new
incoming
stuff.
So
I
don't
have
much
updates
on
that.
I'm
not
sure.
If
people
here
know
we
have
been
having
our
office
hours,
we
have
had
a
couple
of
office
hours.
I
think
this
week
was
our
third.
C
A
Yeah
for
the
valera
office
hours.
We
have
that
on
the
the
schedule
for
the
distribution
list,
I
believe
and
yeah
we'll
we'll
make
some
more
noise.
We
wanted
to
try
it
out,
see
how
it
worked
out.
It's
been
working
pretty
well,
we've
had
a
few
people
join
in
and
ask
questions,
and
and
so
on
so
yeah.
Please
join
us
when,
when
we
have
these
we'll
make
some
more
noise
in
the
slack
channels
as
well.
So
people
know
that
they're
they're
happening
during
those
days.
E
E
E
We've
got
approval
to
open
source
that
so
I'm
going
to
be
pushing
that
out
as
an
open
source
project,
hopefully
this
week
and
that's
something
we
can
use
both
with
valero
but
pretty
much
any
anything
where
you
want
to
create
data
and
verify
it.
And
ideally
the
the
verification
process
tells
you
what
went
wrong
rather
than
simply
exploding
like
it
does,
with
a
lot
of
regular
applications.
E
F
Yeah
so
was
also
participating
in
this
portathon
last
week.
We're
revisiting
our
bug
triage
process,
we're
going
to
get
into
a
more
regular
cadence
with
reviewing
the
the
bugs,
and
this
portathon
was
part
of
an
effort
to
kind
of
get
to
a
more
level,
setting
a
new,
a
new
baseline
for
starting
that
so
we're
beginning
that
this
week
also
last
week
released
version
one
v,
1.4.3
and
v
1.5.2
both
included
a
security
fix
for
properly
binding
restored,
persistent
volumes
and
persistent
volume
claims.
F
There
was
a
race
condition
on
on
busy
clusters
where
the
pv
would
get
restored
first,
and
if
there
was
a
persistent
volume
claim
that
the
pv
satisfied
it
could
get
claimed
by
something
else,
both
one
four
three
and
one
five
two
fix
that,
and
they
both
have
some
other
fixes
in
them
as
well,
and
then
just
a
reminder
for
the
maintainers.
There's
several
design
prs
out
there.
That
could
use
additional
comments.
I
need
to
get
in
on
a
couple
of
those
as.
A
F
Yeah,
I
know
scott's
been
waiting
for
a
while
on
the
restore
item,
action
for
additional
items
to
be
ready.
2867
I
need
to
get
on.
I
need
to
look
at
the
the
top
two,
the
timeout
for
plugins
and
the
restore
progress
reporting.
I
know
I
think
dylan
asked
about
3016
if
it
could
get
included
in
version
1.6.
A
G
G
I'm
gonna
explain
very
quickly
what
api
groups
and
I
put
those
links
on
the
on
the
dock
at
the
same
link
inside
of
the
ticket,
and
basically
this
is
gonna
help
as
well
have
the
little
to
support
different
crds
out
there
right.
So,
every
time
we
create
this,
a
crd
or
even
a
native
kind
in
kubernetes,
it's
name
a
kind
that
belongs
to
a
resource
list
that
belongs
an
api
group
and
api
groups
have
different
versions.
G
Okay,
so
time
goes
by.
Kubernetes
got
new
versions.
Api
groups
evolve
today
we
have
valero
to
the
ability
to
take
a
backup
of
those
kinds
in
different
versions,
and
if
you
go
here
on
our
output
on
our
documentation,
how
we
take
the
backup
and
the
format,
if
you
pass
a
specific
parameter,
you
can
take
multiple
you're
gonna
get
multiple
versions
of
the
kind
right.
That's
today,
and
this
is
an
example
about
one
of
my
favorite
api
groups
out
scaling.
G
If
you
don't
do
anything,
you
just
always
you're
always
gonna
get
the
preferred
version,
but
with
this
parameter
you
can
at
least
take
backups
of
all
the
versions
that
given
cluster
support,
I'm
gonna
hand
over
to
frankie
how
we
are
proposing
to
tackle
this
problem.
We
coded
as
part
of
the
literal
how
to
take
the
backup
which
is
half
of
the
conversation.
G
The
remaining
half
of
this
conversation
is
during
the
restore
how
we're
gonna
pick
those
versions
to
be
restored.
Remember
there's
two
key
concepts.
Each
cluster
has
the
preferred.
Each
cluster
has
a
preferred
version
for
each
api
group
and
an
array
of
the
versions
that
cluster
support
for
that
api
group.
G
H
H
So
I
have
the
hpa,
auto
scaling
backup
here
that
I
have
done
without
the
feature
flag,
enabled
the
api
groups
feature
flag
and
it
only
gives
me
the
preferred
version,
but
if
with
it
enabled,
you
can
see
that
I
have
multiple
versions,
I
have
preferred
v2,
which
is
the
v1
v2
beta1
and
v2
beta2.
So
this
is
now
with
the
feature
flag,
enabled
in
the
backup
that
was
accomplished
using
this
pr
this
by
fulfilling
this
issue.
Closing
this
to
the
shoe
2373..
H
H
All
right,
in
that
case,
I'll
go
on
with
I'll
go
on
to
the
the
three
priorities
we
selected.
The
first
priority
is
what
happens
normally,
and
that
is
that
our
our
target
preferred
version.
Is
our
source
preferred
version
or
our
target
preferred
version?
Is
our
source
a
supported
version?
That
means
we
can
use
party
one
is
that
the
target
preferred
version
can
be
used.
H
Priority
two
is:
if
the
target
preferred
version
cannot
be
used,
then
the
source
preferred
version
can
be
used.
That
is
priority.
Two
and
priority
three
means
we're
going
to
choose
a
common
supported
version
between
the
target
and
the
source
clusters,
and
we
will
choose
the
latest
version
if
there's
more
than
one
all
right.
So
let's
go
into
the
concrete
examples
for
the
highest
priority
case
priority
one
we
have.
H
H
H
H
H
H
G
C
C
G
Yeah,
like
if
the
user
there's
any
input,
we
can
definitely
put
some
in
this
design
an
option
to
have
a
user
input,
but
we
have
to
remember
that
those
versions
is
per
resource
list
which
can
be
you
know
the
painting
of
the
backup.
If
it's
a
cluster
level,
backup
can
be
like
even
hundreds
right
of
the,
but
we
have
to
be
smart.
How
we're
gonna
have
this
a
user
input
to
cherry
pick
for
a
given
crd
right?
We
can,
but
definitely
we
can
make
a.
G
C
Yeah,
so
I'm
thinking
about
the
scenario
where
the
you're
you're
choosing
based
on
version
numbers,
but
there
may
be
some
breaking
changes
in
the
crd
itself,
which
makes
it
impossible
to
restore
at
which
point
the
user
may
want
to
take
control
of
the
wheel
and
say
no.
This
is
exactly
the
version
that
I
want
and
I'm
going
to
handle
the
migration
to
a
newer
version
myself.
So
we
want
to
give
the
users
that
control-
and
the
other
thing
that
I
was
like,
I
think
we
can
use.
C
F
C
G
Okay,
so
one
thing
that
we
are
working
towards
is
actually
creating
a
crd
testing
crd
with
the
web
hooks,
because
all
all
this
translation
between
supported
version
preferred
version,
it
happens
by
magic
by
the
web
hooks
right.
So
we
are
we
created
crd,
but
then
we
realized
that
oh,
no,
not
only
creating
a
crds,
but
we
have
to
code
a
controller
that
has
a
web
hook
to
make
those
conversions.
So
that's
what
I'm
working
on
once
we
have
this
prototype.
I
It
seems
that
the
valero
crs
currently
just
report
the
backup
and
restores
here
just
report
the
number
of
errors
and
don't
actually
report
errors
themselves.
So
just
wanted
to
bring
this
up
for
discussion
and
understand
the
thought
process
of
is.
Was
there
any
work
done
to
actually
report
the
errors
on
the
cr
itself?
F
So,
actually,
errors
used
to
go
directly
on
the
the
crs
and
what
we
found
was
that
actually
overflowed
the
space
on
fcd
for
a
given
object,
so
usually
objects.
Kubernetes
objects
get
about
1.5
megabytes
of
space
on
ftd,
so
we
were
not
able
to
put
all
the
errors
on
on
a
cr
because
put
that
full
log
there
for
a
big
cluster
with
a
lot
of
resources,
then
that
won't
fit.
F
But
this
is
a
great
point,
because
the
only
place
they're
in
right
now
is
in
the
log,
and
you
basically
have
to
download
and
parse
the
log
you
can.
You
can
enable
json
logging
to
make
it
a
little
bit
easier
to
parse
in
machine
format,
but
right
now
that's
kind
of
it's
like
a
universal
switch,
so
it's
either
all
json
for
yourself
for
the
user
and
any
using
any
consuming
programs
or
it's
in
a
more
human
friendly
format.
J
I
was
going
to
say
related
to
that
the
personal
logs.
One
thing
I
run
into
is
the
inconsistency
currently
between
backup
and
restore
also
adds
to
the
challenge
there.
If
you're,
actually,
looking
at
the
results
of
say,
you
know,
valero
get
logs
because
for
backups
valero
logs
includes
the
errors
but
described
as
not,
and
for
restorers
describe
includes
the
areas,
but
logs
does
not.
I
Yeah
so
one
of
the
things
that
we
were
discussing-
and
I
think
scott
raised
this-
is
we
we
know
that,
although
described
does
help
in
some
sort
of
error
in
some
sort
of
parsing,
would
it
be
possible
to
have
that
as
an
exported
function?
So
maybe
we
could
reuse
that
and
we
can
make
a
programmatic
use
of
that
in
our
code
base.
I
So
that's
one
thing
I
wanted
to
point
out,
and
second
thing
was
that
upstream.
I
know
that
there
are
issues
regarding
making
some
resources
on
cr
custom
resource
possible.
So
would
you
think
that
this
is
one
of
the
use
case
for
a
sub
resource
like
logs,
so
let's
say,
valero
backup
and
restore,
have
sub-resource
logs,
which
can
be
implemented
with
custom
logic
that
would
parse
the
logs
and
not
have
us
dump
in
the
cr
or
maintain
a
custom
utility
just
for
parsing
those.
So.
F
So
the
the
function
so
the
the
way
the
numbers
are
added
to
a
custom
resource
for
like
restore
or
the
backup
is
it's
not
actually
parsing
the
log?
It's
just
taking
a
length
of
the
log
messages,
so
it's
it's
not
parsing
as
they
come
through.
So
there's
no
function
for
parsing
to
export
right
now
we
just
say
we
have
a
construct.
I
forget
what
it's
called.
F
J
F
H
J
We
still,
this
may
still
need
to
be
an
on-demand
kind
of
thing,
but
I
guess
part
of
the
thing
is:
we
would
like
to
be
able
to
pull
it
from
our
migration
application,
because
one
of
the
things
is
that
you
know
one:
the
our
migration
application
kind
of
talks
to
multiple
clusters
and
our
user
interface
talks
to
that
cluster.
So
we
wouldn't
want
the
ui
to
have
to
go
to
the
valero.
You
know
on
whatever
cluster.
J
I
Yeah,
I
I
I
wanted
to
just
spend
a
couple
of
minutes
to
share
what
I
understand
of
customs
sub
resource,
so
I
think
what
would
happen
with
the
customs
of
resource
is
that
it
would.
We
would
not
hit
the
same
hcd
cap
there,
because
the
way
folks
are
talking
about
customs
of
resources
is
that
you
would
have
your
own
implementation
of
that
resource
via
a
web
hook.
I
So
essentially,
what
we
could
do
is
we
could
imagine
what
valero
described
does
in
order
to
parse
the
logs
we'd
have
the
exact
same
functionality
as
a
webhook
and
exposed
by
a
log
sub
resource,
so
this
would
become
a
native
cube
way
of
doing
things.
So
I
know
this
is
a
lot
of
work,
and
this
involves
support
upstream
as
well.
So
I
I
just
wanted
to
point
out
that
this
could
this
be
a
valid
use
case
for
that
kind
of
implementation?
F
We
wouldn't
want
them
to
be
pure
custom
resources,
and
I
don't
know
enough
about
sub
resources
to
know
if
it's
a
good
use
of
it.
Yet
I
thought
some
resources
were
limited
to
status
and
something
else,
but
if
you
can
make,
if
you
can
make
custom
ones,
that
may
be
an
option
and
I
and
you
you
aren't
the
only
ones
that
would
like
to
use
this,
like,
I
think
we've
had
folks
at
dell
say
hey.
I
want
to
know
why
this
thing
failed.
So
I
can
like
remediate
it
or
you
know.
Is
this
a?
F
I
I
I
I
don't
think
this
will
all
depend
how
it
is
implemented
and,
okay,
I
think
more
immediate
short
term
goal
would
be
to
understand
if
we
can
do
things
with
the
existing
describe
functionality
and
maybe
have
a
common
programmatic
way
of
exposing
errors
in
the
current
thing
itself
and
long
term.
We
can
have
this
discussion.
I
just
wanted
to
bring
it
up.
So
if
we
have
all
the
we
have
all
the
context
needed
for
this.
F
I
F
I
I
think
they,
I
think,
that's
the
distinction
between
group
version
kind
and
group
version
and
resource
like
the
group
version
resource
is
an
endpoint,
whereas
kind
represents
actual
serializable
data
structure.
So
I
I
imagine
this
queue
like
the
sub
resource
log,
being
a
gvr
group
version
resource,
so
you
would
only
have
an
endpoint
and
then
that
endpoint
can
be
redirected
to
the
webhook
custom
implementation.
F
I
Yeah
yeah,
I
I
think
the
main
answer
I
was
looking
for.
I
already
got
out
of
this
discussion
as
to
why
they
are
not
part
of
the
the
backup
and
restore
resource
itself.
So
that's
super
helpful
and
going
forward.
I
think
we
might
come
up
with
a
very
programmatic
way
to
expose
this
short
term
yeah.
You
will
know
we
will
have
a
design
docker
outdoor
and
stuff
like
that,
going
forward
so
cool,
we'll
circle.
Back
on
to
this
discussion.
F
I
I
D
F
Yeah
so
first
off,
we've
got
abby
mccarthy
here
from
vmware
fixing
some
of
our
documentation
up,
so
this
fixes
some
of
the
mineo
code
code.
Examples.
Thank
you
very
much
for
fixing
these
up.
Abby
always
appreciate
documentation,
fixing
and
clarification
this
next
one.
I
don't
know
how
to
pronounce
this
name.
Hari
haran
excuse
me,
filed
an
issue
and
fixed
and
fixed
the
issue,
so
the
valero
client
will
now
report.
F
If
you're,
you
have
a
mismatch
between
your
server
and
your
client,
which
is
super
useful
for
catching
drift.
So
thank
you
very
much
hariharan
next.
I
think
this
is
piper
piper
authority.
So
there
was
some
discussion
about
restoring
some
istio
resources
with
valero
on
the
latest
version
of
istio
and
istio
mutating
web
hooks
actually
break
some
assumptions
on
both
sides,
so
piper
actually
fixed
some
issues
with
our
expectations
on
the
rustic
weight
container
being
the
first
in
a
container.
F
So
this
now
fixes
it
so
the
rustic
weight
container
can
be
anywhere
and
we'll
wait
for
it.
So
thank
you
very
much
piper
for
catching
that
and
fixing
it,
and
I
think
the
rest
of
these
are
helm.
So
this
one
genting
caught
an
issue
that
we've
seen
in
valero
itself,
where
the
ci
would
run
against
forks,
and
so
he
fixed
that
issue
on
the
helm
chart.
So
ci
doesn't
run
against
your
fork.
It
only
runs.
F
F
Yeah,
it's
a
it's
a
common
trip
up
with
github
actions.
It's
I
think
their
their
documentation
is
not
super
clear
on
on
this
and
we
had
it
for
a
while
before
we
figured
out
what
was
going
on
and
then
we've
got,
it
looks
like
some
syntax
change
from
mccool
for
cleaning
up
the
im
annotation
on
our
values.