16 Mar 2021
March 16, 2021
Status Updates - items working towards roadmap
carlisia
Theme: unblocking items for the v1.6 release.
Description Status
#3489 [velero] Use Credential from BSL for restic commands (release-blocker) reviewed PR ✅
#3559 [velero] Add E2E test for multiple credentials (release-blocker) reviewed PR ✅
#3527 [velero] Prefer conditional waiting over magic sleep (release-blocker) reviewed PR ✅
#3583 [velero] Restore progress reporting bug fix (release-blocker) reviewed PR ✅
#3568 [velero] Update upgrade docs (release-blocker) PR merged 🆗
#3584 [velero] Improve GH Action PR assign + labeling PR open for review 📬
#3591 [velero] Improve plugin release instructions PR open for review 📬
AWS / GCP / Azure [plugins] Update changelogs PRs merged 🆗
Issue 3493 Initial scoping out of work for packaging with Carvel Meeting ☎️
-Next:
- push tags and run tests against v1.2.0 of the 3 Velero maintainted plugins
- plan the rest of the week and start planning/organizing for v1.7
- PR reviews
bridget
Focusing on 1.6 release blockers
Multiple credentials work is complete
dave
E2E tests with volume snapshots
Release candidate when E2E tests running/passing on AWS & vSphere
RC tests with latest/earliest K8S supported, Azure, AWS, Kind, vSphere, GCP
Manual tests pass
Discussion topics
carlisia
Welcome Eleanor, who officially starts this week! (PM for Velero)
e2e tests for Velero upgrades?
Contributor Shoutouts PRs
@codegold79: Restore progress reporting bug fix
Contributor Shoutouts Velero Q&A
None this week. We do have unanswered questions.
Status Updates - items working towards roadmap
carlisia
Theme: unblocking items for the v1.6 release.
Description Status
#3489 [velero] Use Credential from BSL for restic commands (release-blocker) reviewed PR ✅
#3559 [velero] Add E2E test for multiple credentials (release-blocker) reviewed PR ✅
#3527 [velero] Prefer conditional waiting over magic sleep (release-blocker) reviewed PR ✅
#3583 [velero] Restore progress reporting bug fix (release-blocker) reviewed PR ✅
#3568 [velero] Update upgrade docs (release-blocker) PR merged 🆗
#3584 [velero] Improve GH Action PR assign + labeling PR open for review 📬
#3591 [velero] Improve plugin release instructions PR open for review 📬
AWS / GCP / Azure [plugins] Update changelogs PRs merged 🆗
Issue 3493 Initial scoping out of work for packaging with Carvel Meeting ☎️
-Next:
- push tags and run tests against v1.2.0 of the 3 Velero maintainted plugins
- plan the rest of the week and start planning/organizing for v1.7
- PR reviews
bridget
Focusing on 1.6 release blockers
Multiple credentials work is complete
dave
E2E tests with volume snapshots
Release candidate when E2E tests running/passing on AWS & vSphere
RC tests with latest/earliest K8S supported, Azure, AWS, Kind, vSphere, GCP
Manual tests pass
Discussion topics
carlisia
Welcome Eleanor, who officially starts this week! (PM for Velero)
e2e tests for Velero upgrades?
Contributor Shoutouts PRs
@codegold79: Restore progress reporting bug fix
Contributor Shoutouts Velero Q&A
None this week. We do have unanswered questions.
- 9 participants
- 30 minutes
9 Mar 2021
March 9, 2021
Status Updates - items working towards roadmap
nrb
Not a lot of progress on code items, mostly working on process stuff and internal builds.
bridget
BSL and Plugin PRs merged for multiple credentials
Restic PR now ready for review: PR 3489
Working on E2E test for multiple credentials
carlisia
Last week: community support + PR reviews of release blockers
This week: PR reviews + review of upload progress design + resolving support issue from last week (restore gets stuck restoring an nfs-pv using restic - (https://github.com/vmware-tanzu/velero/issues/3450)
dave
vSphere stuff
Updated ROADMAP.md with 1.7.0 items (https://github.com/vmware-tanzu/velero/blob/main/ROADMAP.md)
This week - community support
ashish
E2E tests flakyness:
Issue https://github.com/vmware-tanzu/velero/issues/3532
PR https://github.com/vmware-tanzu/velero/pull/3527
Restic upgrade:
https://github.com/vmware-tanzu/velero/pull/3528
Discussion topics
Eleanor Millman will be our new PM starting March 15, 2021
Nolan: Go over release-blockers for v1.6.0
Nolan: Velero Roadmap update for v1.7.0
We will be moving to date-based releases, probably close to Kubernetes release dates.
This would be a 4 month cadence, and give us more realistic timeframes; what do folks think about this?
No matter what timeframe we pick, we’ll be shipping what’s included at that time and only stopping for critical bugs.
Leave comments on https://github.com/vmware-tanzu/velero/pull/3551
Nolan: Working on a Velero vision for the future. Please leave a comment on this hackmd if you’d like to share your thoughts.
Contributor Shoutouts PRs
@jenting: Add jenting as reviewer
@jenting: Add jenting as reviewer
@jenting: [velero] Helm v2 deprecated
@jenting: [velero] Update chart maintainer
Contributor Shoutouts Velero Q&A
Status Updates - items working towards roadmap
nrb
Not a lot of progress on code items, mostly working on process stuff and internal builds.
bridget
BSL and Plugin PRs merged for multiple credentials
Restic PR now ready for review: PR 3489
Working on E2E test for multiple credentials
carlisia
Last week: community support + PR reviews of release blockers
This week: PR reviews + review of upload progress design + resolving support issue from last week (restore gets stuck restoring an nfs-pv using restic - (https://github.com/vmware-tanzu/velero/issues/3450)
dave
vSphere stuff
Updated ROADMAP.md with 1.7.0 items (https://github.com/vmware-tanzu/velero/blob/main/ROADMAP.md)
This week - community support
ashish
E2E tests flakyness:
Issue https://github.com/vmware-tanzu/velero/issues/3532
PR https://github.com/vmware-tanzu/velero/pull/3527
Restic upgrade:
https://github.com/vmware-tanzu/velero/pull/3528
Discussion topics
Eleanor Millman will be our new PM starting March 15, 2021
Nolan: Go over release-blockers for v1.6.0
Nolan: Velero Roadmap update for v1.7.0
We will be moving to date-based releases, probably close to Kubernetes release dates.
This would be a 4 month cadence, and give us more realistic timeframes; what do folks think about this?
No matter what timeframe we pick, we’ll be shipping what’s included at that time and only stopping for critical bugs.
Leave comments on https://github.com/vmware-tanzu/velero/pull/3551
Nolan: Working on a Velero vision for the future. Please leave a comment on this hackmd if you’d like to share your thoughts.
Contributor Shoutouts PRs
@jenting: Add jenting as reviewer
@jenting: Add jenting as reviewer
@jenting: [velero] Helm v2 deprecated
@jenting: [velero] Update chart maintainer
Contributor Shoutouts Velero Q&A
- 10 participants
- 48 minutes
2 Mar 2021
March 2, 2021
Status Updates - items working towards roadmap
bridget
Multiple credentials support
Adding docs and E2E test
Need one more approval on PR 3442, this will unblock PR 3489
nrb
Not a lot of progress on roadmap last week, dealing with internal TKG builds
ashish
Working on making e2e tests as part of our release pipeline.
dave
Discussion on upload progress monitoring
Working on reviewing outstanding feature backlog
Discussion Topics
What was the outcome/progress from last week’s discussion topics?
Findings for what happens when a resource is in multiple API groups when processed by a custom plugin - new documentation merged in #3498
v1.7.0 rough roadmap
Velero debug support, based on crashd
Trying to get CSI snapshot support to GA
Plugin versioning
Snapshot upload progress
Manifest data structure
Distroless container base image to reduce attack surface
Velero Carvel installation
More detailed test plan; get more thorough in what we’re actually looking for out of tests
Contributor Shoutouts (PRs and Discussions)
@codegold79: Update in-code documentation to show resources can be specified with group name
@slavina-rumenova: Adding check if disk description is empty
@justbert: Add documentation around minimum required permissions
@tareqhs: [velero] customize client-qps and client-burst
@peterekhator: [velero] feat: add nodeselector for restic
@cyril-corbon: [velero] feat: add annotations for velero deployment and restic daemonset
@codegold79: [Velero] Add velero component label and update docs
Status Updates - items working towards roadmap
bridget
Multiple credentials support
Adding docs and E2E test
Need one more approval on PR 3442, this will unblock PR 3489
nrb
Not a lot of progress on roadmap last week, dealing with internal TKG builds
ashish
Working on making e2e tests as part of our release pipeline.
dave
Discussion on upload progress monitoring
Working on reviewing outstanding feature backlog
Discussion Topics
What was the outcome/progress from last week’s discussion topics?
Findings for what happens when a resource is in multiple API groups when processed by a custom plugin - new documentation merged in #3498
v1.7.0 rough roadmap
Velero debug support, based on crashd
Trying to get CSI snapshot support to GA
Plugin versioning
Snapshot upload progress
Manifest data structure
Distroless container base image to reduce attack surface
Velero Carvel installation
More detailed test plan; get more thorough in what we’re actually looking for out of tests
Contributor Shoutouts (PRs and Discussions)
@codegold79: Update in-code documentation to show resources can be specified with group name
@slavina-rumenova: Adding check if disk description is empty
@justbert: Add documentation around minimum required permissions
@tareqhs: [velero] customize client-qps and client-burst
@peterekhator: [velero] feat: add nodeselector for restic
@cyril-corbon: [velero] feat: add annotations for velero deployment and restic daemonset
@codegold79: [Velero] Add velero component label and update docs
- 7 participants
- 48 minutes
23 Feb 2021
Feb 23, 2021
Status Updates - items working towards roadmap
bridget
Multiple credentials support
BSL support in PR 3442
Restic support in draft PR 3489
Plugin support for AWS, Azure, GCP
Currently running tests and need to write docs
Added fix for restic restores when using namespace mapping: PR 3475
Released Azure plugin v1.1.2
carlisia
Addressed code reviews for the DownloadRequest controller - https://github.com/vmware-tanzu/velero/pull/3004
Looking at an issue with a capi cluster
PR reviews
nrb
Reviewing blockers for v1.6.0
Continuing to refine roadmap out to v1.9+
Testing an update to restic v0.12.0
dave
Working on road map
Open review/discussion for Upload Progress monitoring design doc
https://github.com/vmware-tanzu/velero/pull/3416
Zoom - Feb 25, 2021 9:00 AM Pacific - https://VMware.zoom.us/j/94331975442?pwd=V2Rmci9kMjlOMWlHQUZqSkZKQ0ttdz09&from=addon
ashish
Finishing up plumbing work for Velero e2e tests on internal platform.
Discussion Topics
Any rotating topics we want to add here?
What was the outcome/progress from last week’s discussion topics?
nrb: Started a design proposal for a backup manifest data structure
@codegold79 / @brito-rafa: Findings for what happens when a resource is in multiple API groups when processed by a custom plugin. Related issue: https://github.com/vmware-tanzu/velero/issues/3491
Plugin authors should specify fully qualified names in the ResourceSelector type
Action: Document ResourceSelector type
Action: Resources that have multiple views with shared UUIDs - define how to handle these properly on restore
Example: Deployments which move versions from v1.15 - v1.16
Example: customresourcedefinitions.apiextensions.k8s.io v1beta1 - v1 introduced different fields, but the object was the same one
dave: Using an outbound proxy - Any experience from the group on best practices?
Contributor Shoutouts
@MadhavJivrajani: added documentation for how velero handles encryption
@aslafy-z: fix typo in item_hook_handler
@aristosvo: Security configuration Azure Storage Account with AKS
Status Updates - items working towards roadmap
bridget
Multiple credentials support
BSL support in PR 3442
Restic support in draft PR 3489
Plugin support for AWS, Azure, GCP
Currently running tests and need to write docs
Added fix for restic restores when using namespace mapping: PR 3475
Released Azure plugin v1.1.2
carlisia
Addressed code reviews for the DownloadRequest controller - https://github.com/vmware-tanzu/velero/pull/3004
Looking at an issue with a capi cluster
PR reviews
nrb
Reviewing blockers for v1.6.0
Continuing to refine roadmap out to v1.9+
Testing an update to restic v0.12.0
dave
Working on road map
Open review/discussion for Upload Progress monitoring design doc
https://github.com/vmware-tanzu/velero/pull/3416
Zoom - Feb 25, 2021 9:00 AM Pacific - https://VMware.zoom.us/j/94331975442?pwd=V2Rmci9kMjlOMWlHQUZqSkZKQ0ttdz09&from=addon
ashish
Finishing up plumbing work for Velero e2e tests on internal platform.
Discussion Topics
Any rotating topics we want to add here?
What was the outcome/progress from last week’s discussion topics?
nrb: Started a design proposal for a backup manifest data structure
@codegold79 / @brito-rafa: Findings for what happens when a resource is in multiple API groups when processed by a custom plugin. Related issue: https://github.com/vmware-tanzu/velero/issues/3491
Plugin authors should specify fully qualified names in the ResourceSelector type
Action: Document ResourceSelector type
Action: Resources that have multiple views with shared UUIDs - define how to handle these properly on restore
Example: Deployments which move versions from v1.15 - v1.16
Example: customresourcedefinitions.apiextensions.k8s.io v1beta1 - v1 introduced different fields, but the object was the same one
dave: Using an outbound proxy - Any experience from the group on best practices?
Contributor Shoutouts
@MadhavJivrajani: added documentation for how velero handles encryption
@aslafy-z: fix typo in item_hook_handler
@aristosvo: Security configuration Azure Storage Account with AKS
- 10 participants
- 51 minutes
19 Feb 2021
Feb 9, 2021
Status Updates - items working towards roadmap
nrb
Helping implement credential support on volume snapshot locations
Opened a PR to close issues if they’re stale. (PRs will not be closed as stale)
Opened a PR for labeling issues and adding a PR template
Working with Dave to solidify a roadmap for the next few versions
Reviews of items defined as release blockers.
This list is incomplete, but these are the PRs we’re prioritizing
Aim to get a pre-release of v1.6.0 out by end of February
Playing with “crashd” to get info about a Velero install within a cluster.
Getting some sort of bundling would be helpful so we can put the Velero crashd scripts in our repo.
Would be part of a solution to velero debug or something similar
bridget
Working on multiple credential support
Blocked on #3190 and #3409 (working off these branches locally)
Will push branches for support in BSLs and VSLs once the above are merged
Work on restic support is still ongoing
carlisia
Updated DownloadRequest PR, ready for review
Wrapping up addressing reviews for the BSL credentials field PR
Next: review PRs related to the multiple credentials change
After: PR review of the e2e tests
dave
Community support last week
Upload progress design doc - https://github.com/vmware-tanzu/velero/pull/3416
Will schedule community design review on upload progress
E2E tests running automatically
Discussion Topics
shawn-hurley (Shawn Hurley)/dave (Dave Smith-Uchida) - Have the Velero CLI use the active namespace in the current context (https://github.com/vmware-tanzu/velero/issues/3194)
This has links to multitenancy design - if Velero watches for backups & restores in different namespaces, this becomes important
Nolan owes architecture diagrams
Rahul Rathi - Ways to capture the volume size corresponding to every snapshot contained in a backup
Currently there is no place where Velero captures the snapshot sizes
Can the velero VolumeSnapshotter API add a way to report the size of the snapshot and the size of the volume?
Dave - snapshots could have dependencies
This is why there’s a request for the original volume size
Nolan
Does the CSI API expose this?
There’s a restoreSize field there
We’d need to add another field on the CSI objects, which is viable
How would this work with restic
Could use PV’s size to start with
Also helps on restore
Dave - Can we get use cases?
May need a dry run
Give users an idea of how much data is going to be restored?
This would be hard to use across clouds
Requirements in priority
Volume size (in Velero and CSI API)
The VolumeSnapshotters are eventually going to be deprecated, so we’ll need to make sure it gets into CSI
CSI and vSphere snapshots are a different path - they’re BackupItemActions
We have the PV YAML in the backup, and can scrape it
Can we get a design doc?
Frankie (codegold79)/Rafael (brito-rafa) - Update on PR #3133 (Enable API Groups Feature)
Contributor Shoutouts
@a-mccarthy: remove FAQ pages
@mike1808: Add colors to describe command
@shellwedance: Modify InitContainer checking function that potentially incurs error
@aslafy-z: [velero] feat: allow servicemonitor to be deployed in another namespace
Status Updates - items working towards roadmap
nrb
Helping implement credential support on volume snapshot locations
Opened a PR to close issues if they’re stale. (PRs will not be closed as stale)
Opened a PR for labeling issues and adding a PR template
Working with Dave to solidify a roadmap for the next few versions
Reviews of items defined as release blockers.
This list is incomplete, but these are the PRs we’re prioritizing
Aim to get a pre-release of v1.6.0 out by end of February
Playing with “crashd” to get info about a Velero install within a cluster.
Getting some sort of bundling would be helpful so we can put the Velero crashd scripts in our repo.
Would be part of a solution to velero debug or something similar
bridget
Working on multiple credential support
Blocked on #3190 and #3409 (working off these branches locally)
Will push branches for support in BSLs and VSLs once the above are merged
Work on restic support is still ongoing
carlisia
Updated DownloadRequest PR, ready for review
Wrapping up addressing reviews for the BSL credentials field PR
Next: review PRs related to the multiple credentials change
After: PR review of the e2e tests
dave
Community support last week
Upload progress design doc - https://github.com/vmware-tanzu/velero/pull/3416
Will schedule community design review on upload progress
E2E tests running automatically
Discussion Topics
shawn-hurley (Shawn Hurley)/dave (Dave Smith-Uchida) - Have the Velero CLI use the active namespace in the current context (https://github.com/vmware-tanzu/velero/issues/3194)
This has links to multitenancy design - if Velero watches for backups & restores in different namespaces, this becomes important
Nolan owes architecture diagrams
Rahul Rathi - Ways to capture the volume size corresponding to every snapshot contained in a backup
Currently there is no place where Velero captures the snapshot sizes
Can the velero VolumeSnapshotter API add a way to report the size of the snapshot and the size of the volume?
Dave - snapshots could have dependencies
This is why there’s a request for the original volume size
Nolan
Does the CSI API expose this?
There’s a restoreSize field there
We’d need to add another field on the CSI objects, which is viable
How would this work with restic
Could use PV’s size to start with
Also helps on restore
Dave - Can we get use cases?
May need a dry run
Give users an idea of how much data is going to be restored?
This would be hard to use across clouds
Requirements in priority
Volume size (in Velero and CSI API)
The VolumeSnapshotters are eventually going to be deprecated, so we’ll need to make sure it gets into CSI
CSI and vSphere snapshots are a different path - they’re BackupItemActions
We have the PV YAML in the backup, and can scrape it
Can we get a design doc?
Frankie (codegold79)/Rafael (brito-rafa) - Update on PR #3133 (Enable API Groups Feature)
Contributor Shoutouts
@a-mccarthy: remove FAQ pages
@mike1808: Add colors to describe command
@shellwedance: Modify InitContainer checking function that potentially incurs error
@aslafy-z: [velero] feat: allow servicemonitor to be deployed in another namespace
- 13 participants
- 57 minutes
16 Feb 2021
Feb 16, 2021
Status Updates - items working towards roadmap
carlisia
Addressing code reviews for the DownloadRequest controller
PR reviews
bridget
Community support last week
Multiple credentials support
Draft PR for BSL support
Working on Restic support
Will continue on plugin support
Would like to release the Azure plugin this week
To release PR 87
dave
Finishing up multi-cloud E2E tests PR
Worked with Nolan on roadmap
nrb
Finished up some github actions
Community Shoutout
@codegold79: Use label to select Velero deployment in plugin cmd
Status Updates - items working towards roadmap
carlisia
Addressing code reviews for the DownloadRequest controller
PR reviews
bridget
Community support last week
Multiple credentials support
Draft PR for BSL support
Working on Restic support
Will continue on plugin support
Would like to release the Azure plugin this week
To release PR 87
dave
Finishing up multi-cloud E2E tests PR
Worked with Nolan on roadmap
nrb
Finished up some github actions
Community Shoutout
@codegold79: Use label to select Velero deployment in plugin cmd
- 7 participants
- 10 minutes
2 Feb 2021
Feb 2, 2021
Status Updates - items working towards roadmap
carlisia:
Addressing code review for:
https://github.com/vmware-tanzu/velero/pull/3004
https://github.com/vmware-tanzu/velero/pull/3190
PR reviews
bridget:
Focusing on PR reviews
Continuing on multiple credentials support
Will open more PRs once dependent PRs are reviewed/merged
dave
Community support
Upload progress tracking design
jonas
New website layout: https://github.com/vmware-tanzu/velero/pull/3279
Discussion Topics
pranavgaikwad (Pranav Gaikwad)
Need help building Velero CLI
Community Shoutouts
None this week (lots of PRs pending reviews)
Status Updates - items working towards roadmap
carlisia:
Addressing code review for:
https://github.com/vmware-tanzu/velero/pull/3004
https://github.com/vmware-tanzu/velero/pull/3190
PR reviews
bridget:
Focusing on PR reviews
Continuing on multiple credentials support
Will open more PRs once dependent PRs are reviewed/merged
dave
Community support
Upload progress tracking design
jonas
New website layout: https://github.com/vmware-tanzu/velero/pull/3279
Discussion Topics
pranavgaikwad (Pranav Gaikwad)
Need help building Velero CLI
Community Shoutouts
None this week (lots of PRs pending reviews)
- 7 participants
- 21 minutes
26 Jan 2021
Jan 26, 2021
Status Updates - items working towards roadmap
nrb:
Reviewing Frankie’s PR for version negotiation at restore.
Reviewing PRs that multi-creds work depends on.
Tech debt items for v1.6.0 will be trimmed from the board this week and moved to v1.7.0
bridget:
Continuing work on multiple credentials support
Reviewing PRs
dave
e2e tests across multi-cloud
e2e test for multiple namespaces
Discussion Topics
Ritesh Patel from Nirmata will demo Kyverno
Pranav Gaikwad (@pranavgaikwad) Restore Progress feedback. Should the excluded resources be included in the count of items restored? Not included as per design.
Have a question? You can ask in the Discussion Q&A
Community Shoutouts
@MadhavJivrajani: Raise logging level for PV deletion timeout
@a-mccarthy: Update docs to clarify backup location and relationship with other data
@cpanato: [velero] Helm2 fix
@cpanato: add stable repo to the new location
Status Updates - items working towards roadmap
nrb:
Reviewing Frankie’s PR for version negotiation at restore.
Reviewing PRs that multi-creds work depends on.
Tech debt items for v1.6.0 will be trimmed from the board this week and moved to v1.7.0
bridget:
Continuing work on multiple credentials support
Reviewing PRs
dave
e2e tests across multi-cloud
e2e test for multiple namespaces
Discussion Topics
Ritesh Patel from Nirmata will demo Kyverno
Pranav Gaikwad (@pranavgaikwad) Restore Progress feedback. Should the excluded resources be included in the count of items restored? Not included as per design.
Have a question? You can ask in the Discussion Q&A
Community Shoutouts
@MadhavJivrajani: Raise logging level for PV deletion timeout
@a-mccarthy: Update docs to clarify backup location and relationship with other data
@cpanato: [velero] Helm2 fix
@cpanato: add stable repo to the new location
- 8 participants
- 42 minutes
19 Jan 2021
Jan 19, 2021
Status Updates
carlisia
Almost done with addressing reviews for Download Request (https://github.com/vmware-tanzu/velero/pull/3004)
bridget
Released v1.5.3
Helped out with some internal build processes
Working on POC for multiple credentials
Need to review outstanding PRs
nrb
Internal build updates for Velero v1.5.x series (build changes taking a while)
Reviewed timeout design w/ Phuong
My notes: https://hackmd.io/rGQuMa3XRDm7BW-CVVLAsg
dave
E2E tests running in AWS, Azure clouds
Working on next tests, CI/CD integration
Discussion Topics
carlisia
We are using GH Discussions for community support now (and other topics): https://github.com/vmware-tanzu/velero/discussions
bridget
Multiple credentials POC demo
Velero PR: https://github.com/vmware-tanzu/velero/pull/3299
AWS plugin PR: https://github.com/vmware-tanzu/velero-plugin-for-aws/pull/69
phuong: quick update on the plugin timeout design
Step 1. Add golang contexts into the plugin interfaces
Uses new WithContext variations to each fuction for backwards compatibility
Step 2. Determine timeouts & cancellation behavior
Plugin versioning: https://github.com/vmware-tanzu/velero/issues/3285
Helps with modifying plugin interfaces without requiring everyone upgrading in lock step
High level: looking for a pod-level timeout that covers ALL PVs connected.
Nolan’s notes: https://hackmd.io/rGQuMa3XRDm7BW-CVVLAsg
Recording of the meeting: https://Dell.zoom.us/rec/share/EtlVa0vDa_fM0pzxArc9Izbq1LJwNpc1v5bfWcIwjCGzC4KDkjBqkudCWAGZqqI.e01a98H7-RO0y2_q Passcode: xH.vY3D5
Nolan & Dave will be doing a presentation on Velero at the Data Protection Working Group on Wednesday 1/26, 9am Pacific time
#wg-data-protection on the kubernetes slack
Status Updates
carlisia
Almost done with addressing reviews for Download Request (https://github.com/vmware-tanzu/velero/pull/3004)
bridget
Released v1.5.3
Helped out with some internal build processes
Working on POC for multiple credentials
Need to review outstanding PRs
nrb
Internal build updates for Velero v1.5.x series (build changes taking a while)
Reviewed timeout design w/ Phuong
My notes: https://hackmd.io/rGQuMa3XRDm7BW-CVVLAsg
dave
E2E tests running in AWS, Azure clouds
Working on next tests, CI/CD integration
Discussion Topics
carlisia
We are using GH Discussions for community support now (and other topics): https://github.com/vmware-tanzu/velero/discussions
bridget
Multiple credentials POC demo
Velero PR: https://github.com/vmware-tanzu/velero/pull/3299
AWS plugin PR: https://github.com/vmware-tanzu/velero-plugin-for-aws/pull/69
phuong: quick update on the plugin timeout design
Step 1. Add golang contexts into the plugin interfaces
Uses new WithContext variations to each fuction for backwards compatibility
Step 2. Determine timeouts & cancellation behavior
Plugin versioning: https://github.com/vmware-tanzu/velero/issues/3285
Helps with modifying plugin interfaces without requiring everyone upgrading in lock step
High level: looking for a pod-level timeout that covers ALL PVs connected.
Nolan’s notes: https://hackmd.io/rGQuMa3XRDm7BW-CVVLAsg
Recording of the meeting: https://Dell.zoom.us/rec/share/EtlVa0vDa_fM0pzxArc9Izbq1LJwNpc1v5bfWcIwjCGzC4KDkjBqkudCWAGZqqI.e01a98H7-RO0y2_q Passcode: xH.vY3D5
Nolan & Dave will be doing a presentation on Velero at the Data Protection Working Group on Wednesday 1/26, 9am Pacific time
#wg-data-protection on the kubernetes slack
- 6 participants
- 39 minutes
12 Jan 2021
Jan 12, 2021
Status Updates
dave
- e2e tests running in Azure
- Working on e2e tests running against all clouds as install target
nrb
Out most of last week due to family emergency
Reviewed https://github.com/vmware-tanzu/velero/issues/3191 for the cause of throttling log messages
Increase --client-burst & --client-qps if you see this. We’ll get this increased in v1.6
Opened an issue for some of Velero’s upcoming rearchitecture, https://github.com/vmware-tanzu/velero/issues/3229
Covering community support w/ Carlisia this week
bridget
Community support last week
Working on multiple credentials support
Need to make a decision on approach to take
Need to follow up on some PRs that have been updated
carlisia
- This week will be PR reviews and addressing code reviews on my PRs
- Sharing community support with Nolan
- PTO this Thu and Fri
Discussion Topics
dave
Azure issues with memory limit and plug-in failure. Hard to debug some plug-in failures.
nrb - There is an issue w/ Azure plugin where object storage chunk uploading uses the maximum memory of the Velero pod.
nrb - Adding ~10 log messages into the Backup CR seems useful, especially when it can’t upload to object store
Dylan
Ability to selectively patch resources on per restore basis.
Use case is Secrets & CRs
User wants to continually patch Secrets vs replace it
Operators watching CRs, they wanted to make sure the operators were going to keep working
Current design of plugins is very much all or nothing.
Plugins should be able to allow mutations or patching
Give post hooks the backup manifest to help
When patching, this is a merge operation. That may result in a merged, incorrect object at the end
Can we add a way to download individual JSON from a backup to then re-submit with Kubectl?
This exists with the velero backup download command, but you need to take it out of a tarball
Can we modify the restoreitemaction plugin API to return an action?
The plugin could identify the relevant fields and return a patch object & the patch action
Action - Alay and Dylan to work on design doc
Phuong: (phuong.n.hoang@dell.com) invite to design meeting for Velero plugin timeout 9am PST Friday Jan 15th. This is the Zoom meeting: https://Dell.zoom.us/j/91078486282?pwd=Yk9sU1UwT0NScXE2Q2oyRGV3K002QT09
This is for timing out plugins generically
Enhance the plugins to use the Context for timeouts
Please add to https://groups.google.com/g/projectvelero
carlisia
hit a temporary snag with using GH Discussion: https://github.com/github/feedback/discussions/3105
is HackMD getting slow again? Time to archive?
nrb
Current v1.6.0 status: https://github.com/vmware-tanzu/velero/projects/4
Testing is moving along
Need to get multiple credential support design approved
bridget
Can we do a v1.5.3 release in the meantime before 1.6?
nrb: :thumbsup:
We need to include some patches like properly deleting when DeleteItemActions are missing, and fixing include/exclude plugin logic (I don’t have a full list)
Contributor Shoutouts
@justenwalker/Bryson Shepherd (on Slack): fix: missing results in ListObjects/CommonPrefixes
@eyenx: [velero] feat(schedule): add annotation templating
Status Updates
dave
- e2e tests running in Azure
- Working on e2e tests running against all clouds as install target
nrb
Out most of last week due to family emergency
Reviewed https://github.com/vmware-tanzu/velero/issues/3191 for the cause of throttling log messages
Increase --client-burst & --client-qps if you see this. We’ll get this increased in v1.6
Opened an issue for some of Velero’s upcoming rearchitecture, https://github.com/vmware-tanzu/velero/issues/3229
Covering community support w/ Carlisia this week
bridget
Community support last week
Working on multiple credentials support
Need to make a decision on approach to take
Need to follow up on some PRs that have been updated
carlisia
- This week will be PR reviews and addressing code reviews on my PRs
- Sharing community support with Nolan
- PTO this Thu and Fri
Discussion Topics
dave
Azure issues with memory limit and plug-in failure. Hard to debug some plug-in failures.
nrb - There is an issue w/ Azure plugin where object storage chunk uploading uses the maximum memory of the Velero pod.
nrb - Adding ~10 log messages into the Backup CR seems useful, especially when it can’t upload to object store
Dylan
Ability to selectively patch resources on per restore basis.
Use case is Secrets & CRs
User wants to continually patch Secrets vs replace it
Operators watching CRs, they wanted to make sure the operators were going to keep working
Current design of plugins is very much all or nothing.
Plugins should be able to allow mutations or patching
Give post hooks the backup manifest to help
When patching, this is a merge operation. That may result in a merged, incorrect object at the end
Can we add a way to download individual JSON from a backup to then re-submit with Kubectl?
This exists with the velero backup download command, but you need to take it out of a tarball
Can we modify the restoreitemaction plugin API to return an action?
The plugin could identify the relevant fields and return a patch object & the patch action
Action - Alay and Dylan to work on design doc
Phuong: (phuong.n.hoang@dell.com) invite to design meeting for Velero plugin timeout 9am PST Friday Jan 15th. This is the Zoom meeting: https://Dell.zoom.us/j/91078486282?pwd=Yk9sU1UwT0NScXE2Q2oyRGV3K002QT09
This is for timing out plugins generically
Enhance the plugins to use the Context for timeouts
Please add to https://groups.google.com/g/projectvelero
carlisia
hit a temporary snag with using GH Discussion: https://github.com/github/feedback/discussions/3105
is HackMD getting slow again? Time to archive?
nrb
Current v1.6.0 status: https://github.com/vmware-tanzu/velero/projects/4
Testing is moving along
Need to get multiple credential support design approved
bridget
Can we do a v1.5.3 release in the meantime before 1.6?
nrb: :thumbsup:
We need to include some patches like properly deleting when DeleteItemActions are missing, and fixing include/exclude plugin logic (I don’t have a full list)
Contributor Shoutouts
@justenwalker/Bryson Shepherd (on Slack): fix: missing results in ListObjects/CommonPrefixes
@eyenx: [velero] feat(schedule): add annotation templating
- 10 participants
- 51 minutes
5 Jan 2021
Jan 5, 2021
Status Updates
carlisia
- Back from 2 weeks PTO
- This week will be PR reviews and addressing code reviews on my PRs
bridget
Catching up after PTO
On community support this week
Will pick up work on multiple credentials support again
PR #2403 needs another review
nrb
Also catching up after PTO - need to catch up on opened issues and outstanding PRs
Discussion Topics
carlisia
are we ready to start with the GitHub Discussion? I started a list of potential categories here: https://hackmd.io/OFFTp_RIRmOQ7xwhxkv0yA
Status Updates
carlisia
- Back from 2 weeks PTO
- This week will be PR reviews and addressing code reviews on my PRs
bridget
Catching up after PTO
On community support this week
Will pick up work on multiple credentials support again
PR #2403 needs another review
nrb
Also catching up after PTO - need to catch up on opened issues and outstanding PRs
Discussion Topics
carlisia
are we ready to start with the GitHub Discussion? I started a list of potential categories here: https://hackmd.io/OFFTp_RIRmOQ7xwhxkv0yA
- 4 participants
- 19 minutes
15 Dec 2020
Dec 15, 2020
The Velero maintainer team at VMware will be on holiday between December 19, 2020 to January 4, 2021.
Status Updates
nrb
On community support this week
EOY tasks, planning for next year
Code reviews
Maintainers, please give the following PRs a look:
DownloadRequest controller to kubebuilder
Set custom certificates on a BSL
carlisia
PRs + PR reviews
wip: Adding reference to a credential to the BSL
Secrets stuff
wip: review Frankie’s PR (Restore API group version by priority): https://github.com/vmware-tanzu/velero/pull/3133
bridget
Community support first half of this week
Starting vacation Wed/Thurs
Continuing to work on multiple credentials proposal
PR reviews
About to submit a PR for delve debugging in Tilt
dsmithuchida
Submitted talk proposal for Kubecon EU on Backup/restore of operator driven apps
Started on design for upload progress and data movement tracking
Discussion Topics
carlisia
JenTing Hsiao as core maintainer [APPROVED!]: https://github.com/vmware-tanzu/velero/pull/3188
If you’d like to join us, take a look at our Governance document
What if we optimize the behavior of the sync controller in regards to instantiating a backup store: https://github.com/vmware-tanzu/velero/pull/3187
Combine the behavior of sync & BSL controller
Could we have this done in a design doc?
Reduces the amount of requests to object storage
Reduces instantiations of object storage plugins
Reconciling the server default BSL with the user configured default BSL: https://github.com/vmware-tanzu/velero/issues/3173
nrb
New GitHub Processes
Using Ginkgo
Frankie:
Uses BDD descrptions, which isn’t necessarily used
Doesn’t feel “Go-native”
Carlisia
controller-runtime uses Ginkgo’s BeforeTest entrypoints to do setup.
Dave
Used e2e tests cause we were using it elsewhere
Thought there were benefits to it that haven’t manifested
Contributor Shoutouts
@matheusjuvelino: issue: add flag to the schedule cmd to configure the useOwnerReferencesInBackup option #3176
@matheusjuvelino: Owner reference in backup when created from schedule
@imjokey: Tencent S3 Compatible Support Docs
@nemeth: doc: Use inline markdown links in tables
@yusufgungor: Preserve nodePort support with --preserve-nodeports flag
@jenting: feat: support configure BSL CR to indicate which one is the default
The Velero maintainer team at VMware will be on holiday between December 19, 2020 to January 4, 2021.
Status Updates
nrb
On community support this week
EOY tasks, planning for next year
Code reviews
Maintainers, please give the following PRs a look:
DownloadRequest controller to kubebuilder
Set custom certificates on a BSL
carlisia
PRs + PR reviews
wip: Adding reference to a credential to the BSL
Secrets stuff
wip: review Frankie’s PR (Restore API group version by priority): https://github.com/vmware-tanzu/velero/pull/3133
bridget
Community support first half of this week
Starting vacation Wed/Thurs
Continuing to work on multiple credentials proposal
PR reviews
About to submit a PR for delve debugging in Tilt
dsmithuchida
Submitted talk proposal for Kubecon EU on Backup/restore of operator driven apps
Started on design for upload progress and data movement tracking
Discussion Topics
carlisia
JenTing Hsiao as core maintainer [APPROVED!]: https://github.com/vmware-tanzu/velero/pull/3188
If you’d like to join us, take a look at our Governance document
What if we optimize the behavior of the sync controller in regards to instantiating a backup store: https://github.com/vmware-tanzu/velero/pull/3187
Combine the behavior of sync & BSL controller
Could we have this done in a design doc?
Reduces the amount of requests to object storage
Reduces instantiations of object storage plugins
Reconciling the server default BSL with the user configured default BSL: https://github.com/vmware-tanzu/velero/issues/3173
nrb
New GitHub Processes
Using Ginkgo
Frankie:
Uses BDD descrptions, which isn’t necessarily used
Doesn’t feel “Go-native”
Carlisia
controller-runtime uses Ginkgo’s BeforeTest entrypoints to do setup.
Dave
Used e2e tests cause we were using it elsewhere
Thought there were benefits to it that haven’t manifested
Contributor Shoutouts
@matheusjuvelino: issue: add flag to the schedule cmd to configure the useOwnerReferencesInBackup option #3176
@matheusjuvelino: Owner reference in backup when created from schedule
@imjokey: Tencent S3 Compatible Support Docs
@nemeth: doc: Use inline markdown links in tables
@yusufgungor: Preserve nodePort support with --preserve-nodeports flag
@jenting: feat: support configure BSL CR to indicate which one is the default
- 7 participants
- 1:03 hours
8 Dec 2020
Dec 8, 2020
Status Updates
carlisia
Community support last week
Addressed PR reviews for the Tilt Setup (https://github.com/vmware-tanzu/velero/pull/3119)
Mainly did PR testing and reviews
bridget
Spent quite a bit of time on PR reviews
Experimenting with Delve/Tilt debugging
Working on initial steps for supporting multiple credentials
Mostly experimental work based on design doc
dsmithuchida
Working on Distributed Data Generator on Azure
nrb
PR reviews
Working on board processes
Discussion Topics
Rafael @brito-rafa and Frankie @codegold79 - Restore from multiple backed up API Group Versions. A WIP PR #3133 has been submitted that follows the corresponding design doc.
nrb - Restore progress and waiting for additional items?
ballpark of release timeframe of version 1.6 and/or 1.5.3 ?
Community Shoutouts
Swanand Shende for helping folks out in our upstream Slack
Status Updates
carlisia
Community support last week
Addressed PR reviews for the Tilt Setup (https://github.com/vmware-tanzu/velero/pull/3119)
Mainly did PR testing and reviews
bridget
Spent quite a bit of time on PR reviews
Experimenting with Delve/Tilt debugging
Working on initial steps for supporting multiple credentials
Mostly experimental work based on design doc
dsmithuchida
Working on Distributed Data Generator on Azure
nrb
PR reviews
Working on board processes
Discussion Topics
Rafael @brito-rafa and Frankie @codegold79 - Restore from multiple backed up API Group Versions. A WIP PR #3133 has been submitted that follows the corresponding design doc.
nrb - Restore progress and waiting for additional items?
ballpark of release timeframe of version 1.6 and/or 1.5.3 ?
Community Shoutouts
Swanand Shende for helping folks out in our upstream Slack
- 8 participants
- 30 minutes
1 Dec 2020
Dec 1, 2020
Status Updates
carlisia
Community support this week
Internal presentation yesterday
Finished the Tilt setup for Velero (PR: https://github.com/vmware-tanzu/velero/pull/3119)
PR reviews + starting on CLI v1.6 issues
bridget
Updated design for multiple credentials support
Needs further thought on the impact on upgrades and Helm
Updated the docker actions used in CI
Spent some time on community support last week over Thanksgiving
nrb
Bereavement & holiday last week
PR reviews yesterday
Discussion Topics
carlisia: Tilt setup demo
Community Shoutouts
@sseago: Design doc for RestoreItemAction wait for AdditionalItems to be ready
@pranavgaikwad: restore progress design doc
@funkycode & Michael Ketslah: pass annotations from scheduler to created backup
@shellwedance: Modify function name typo
@MadhavJivrajani: Add instructions to clone repo for examples
@jenting: feat: add delete sub-command for backup-location
@dsu-igeek: Add an E2E test framework to test Velero across cloud platforms
@damoon: [velero] fixed dnsPolicy position in yaml definition
@jaredallard: [velero] Support setting dnsPolicy on restic daemonset and velero deployment
Status Updates
carlisia
Community support this week
Internal presentation yesterday
Finished the Tilt setup for Velero (PR: https://github.com/vmware-tanzu/velero/pull/3119)
PR reviews + starting on CLI v1.6 issues
bridget
Updated design for multiple credentials support
Needs further thought on the impact on upgrades and Helm
Updated the docker actions used in CI
Spent some time on community support last week over Thanksgiving
nrb
Bereavement & holiday last week
PR reviews yesterday
Discussion Topics
carlisia: Tilt setup demo
Community Shoutouts
@sseago: Design doc for RestoreItemAction wait for AdditionalItems to be ready
@pranavgaikwad: restore progress design doc
@funkycode & Michael Ketslah: pass annotations from scheduler to created backup
@shellwedance: Modify function name typo
@MadhavJivrajani: Add instructions to clone repo for examples
@jenting: feat: add delete sub-command for backup-location
@dsu-igeek: Add an E2E test framework to test Velero across cloud platforms
@damoon: [velero] fixed dnsPolicy position in yaml definition
@jaredallard: [velero] Support setting dnsPolicy on restic daemonset and velero deployment
- 6 participants
- 55 minutes
10 Nov 2020
Nov 10, 2020
Status Updates
carlisia
My birthday is tomorrow, yeay!
I’ll. be on GoTime live today at 12PM PT: https://changelog.com/live/26m
On community support
Tons of PRs to review
Working on a Velero presentation
Storj/Tardigrade interview about Velero: https://www.youtube.com/watch?v=ldWF7cmNEFc
ashish
Bug fix PRs:
🐛 Use namespace and name to match PVB to Pod restore
🐛 Do not run ItemAction plugins for unresolvable types for all types
Pick up on the e2e tests. Release item for v1.6
PR reviews:
Design PRs
Sync restore of additional items from restoreItemAction plugins.
Proposal looks good overall.
Will take another pass before signing off.
Appreciate your patience.
Also working on the presentation with Carlisia.
nrb
Reviewed proposal with Rafael’s team on the API migration work
DockerHub is limiting pulls from free accounts. This may limit your ability to pull Velero images, especially from CI if it doesn’t cache. We’ll likely be moving Velero into VMware’s newly created public repo at some point in the future to remove the restrictions.
KubeCon NA is next week! Bridget and I will be doing a Velero Q&A at 4pm ET, Wednesday November 18th.
Zoom link
bridget
Mostly catching up on things after PTO
Reviewing design docs
Would like to get involved with multiple credentials work.
dsmithuchida
First pass at e2e tests - https://github.com/vmware-tanzu/velero/pull/3060
Released Distributed Data Generator (Kibishii) test tool - https://github.com/vmware-tanzu-labs/distributed-data-generator
Discussion Topics
Rafael @brito-rafa and Frankie @codegold - Updates to Choosing API Group Version to Restore Design Doc: PR #3050
Review priority cases (with examples with an academic API Group): https://github.com/brito-rafa/k8s-webhooks/tree/master/examples-for-projectvelero
FYI: Changing priority system to match that of K8s more closely
Phuong Hoang: Timeout for BackupItemPlugin design https://github.com/vmware-tanzu/velero/pull/3018 Adding timeout to ExecuteRequest may cause incompatibility with prevous version of plugins.
Action: Phuong to add a meeting for this
Scott/Nolan Addtional items wait design doc
https://github.com/vmware-tanzu/velero/issues/1970 is also related; we cannot currently restore a namespace by itself, or create RestoreItemActions for a namespace object.
Pranav Restore progress status design doc
Handling multiple credentials/secrets
Community Shoutouts
@invidian: Fix various typos found by codespell
@a-mccarthy: Add custom 404 page to website
Preview of the 404 page
@mynktl: fixing ‘velero.io/change-pvc-node-selector’ plugin to fetch configmap using plugin name
Status Updates
carlisia
My birthday is tomorrow, yeay!
I’ll. be on GoTime live today at 12PM PT: https://changelog.com/live/26m
On community support
Tons of PRs to review
Working on a Velero presentation
Storj/Tardigrade interview about Velero: https://www.youtube.com/watch?v=ldWF7cmNEFc
ashish
Bug fix PRs:
🐛 Use namespace and name to match PVB to Pod restore
🐛 Do not run ItemAction plugins for unresolvable types for all types
Pick up on the e2e tests. Release item for v1.6
PR reviews:
Design PRs
Sync restore of additional items from restoreItemAction plugins.
Proposal looks good overall.
Will take another pass before signing off.
Appreciate your patience.
Also working on the presentation with Carlisia.
nrb
Reviewed proposal with Rafael’s team on the API migration work
DockerHub is limiting pulls from free accounts. This may limit your ability to pull Velero images, especially from CI if it doesn’t cache. We’ll likely be moving Velero into VMware’s newly created public repo at some point in the future to remove the restrictions.
KubeCon NA is next week! Bridget and I will be doing a Velero Q&A at 4pm ET, Wednesday November 18th.
Zoom link
bridget
Mostly catching up on things after PTO
Reviewing design docs
Would like to get involved with multiple credentials work.
dsmithuchida
First pass at e2e tests - https://github.com/vmware-tanzu/velero/pull/3060
Released Distributed Data Generator (Kibishii) test tool - https://github.com/vmware-tanzu-labs/distributed-data-generator
Discussion Topics
Rafael @brito-rafa and Frankie @codegold - Updates to Choosing API Group Version to Restore Design Doc: PR #3050
Review priority cases (with examples with an academic API Group): https://github.com/brito-rafa/k8s-webhooks/tree/master/examples-for-projectvelero
FYI: Changing priority system to match that of K8s more closely
Phuong Hoang: Timeout for BackupItemPlugin design https://github.com/vmware-tanzu/velero/pull/3018 Adding timeout to ExecuteRequest may cause incompatibility with prevous version of plugins.
Action: Phuong to add a meeting for this
Scott/Nolan Addtional items wait design doc
https://github.com/vmware-tanzu/velero/issues/1970 is also related; we cannot currently restore a namespace by itself, or create RestoreItemActions for a namespace object.
Pranav Restore progress status design doc
Handling multiple credentials/secrets
Community Shoutouts
@invidian: Fix various typos found by codespell
@a-mccarthy: Add custom 404 page to website
Preview of the 404 page
@mynktl: fixing ‘velero.io/change-pvc-node-selector’ plugin to fetch configmap using plugin name
- 12 participants
- 58 minutes
27 Oct 2020
Oct 27, 2020
Status Updates
carlisia
Suport-a-thon last week
Experimented with an internal tool to boot up an AWS EC2 cluster
Tested and documented
Neat (but dangerous) OSS tool to decommision AWS infrastructure: https://github.com/rebuy-de/aws-nuke
Sample config file: https://gist.github.com/carlisia/c687505ff9b89b429557f897659e1c49
PR reviews + Helm
ashish
Support-a-thon last week
Community support this week
Have you heard about our Velero Office hours?
this opens another hackmd page
dsmithuchida
Support-a-thon last week
Working on basic e2e test, mostly working
nrb
Support-a-thon last week
Revisting our bug triage process
Released v1.4.3 and v1.5.2 last week with a security fix - please upgrade when you can.
Blocker - several design PRs are waiting on other maintainers to review
Discussion Topics
Rafael @brito-rafa , Frankie @codegold79 and Jenting @jenting - Migration doc: https://github.com/vmware-tanzu/velero/issues/2551 . About K8s API Groups: http://www.rafaelbrito.com/2020/04/kubernetes-api-groups.html
Alay: Provide more visibility on reasons for PartialFailure by reporting the errors on velero CR
The CRs don’t provide errors directly, just he number
CRs are limited to 1.5mb usually due to etcd
There’s a mismatch between restore & backup log formats, making just parsing difficult
This could be a subresource backed by a web hook, not a CRD
Community Shoutouts
@a-mccarthy: fix minio code samples
@cvhariharan: Add warning to velero version cmd.
@doughepi: Adding fix for restic init container index on restores.
@jenting: [velero] Restrict release on upstream repo
@mukulgit123: [velero] Correct syntax of iam role pod annotation in values.yaml
@machine424: [velero] Add priorityClassName in the values.yaml file to make it ‘official’
Status Updates
carlisia
Suport-a-thon last week
Experimented with an internal tool to boot up an AWS EC2 cluster
Tested and documented
Neat (but dangerous) OSS tool to decommision AWS infrastructure: https://github.com/rebuy-de/aws-nuke
Sample config file: https://gist.github.com/carlisia/c687505ff9b89b429557f897659e1c49
PR reviews + Helm
ashish
Support-a-thon last week
Community support this week
Have you heard about our Velero Office hours?
this opens another hackmd page
dsmithuchida
Support-a-thon last week
Working on basic e2e test, mostly working
nrb
Support-a-thon last week
Revisting our bug triage process
Released v1.4.3 and v1.5.2 last week with a security fix - please upgrade when you can.
Blocker - several design PRs are waiting on other maintainers to review
Discussion Topics
Rafael @brito-rafa , Frankie @codegold79 and Jenting @jenting - Migration doc: https://github.com/vmware-tanzu/velero/issues/2551 . About K8s API Groups: http://www.rafaelbrito.com/2020/04/kubernetes-api-groups.html
Alay: Provide more visibility on reasons for PartialFailure by reporting the errors on velero CR
The CRs don’t provide errors directly, just he number
CRs are limited to 1.5mb usually due to etcd
There’s a mismatch between restore & backup log formats, making just parsing difficult
This could be a subresource backed by a web hook, not a CRD
Community Shoutouts
@a-mccarthy: fix minio code samples
@cvhariharan: Add warning to velero version cmd.
@doughepi: Adding fix for restic init container index on restores.
@jenting: [velero] Restrict release on upstream repo
@mukulgit123: [velero] Correct syntax of iam role pod annotation in values.yaml
@machine424: [velero] Add priorityClassName in the values.yaml file to make it ‘official’
- 10 participants
- 48 minutes
20 Oct 2020
Oct 20, 2020
Status Updates
carlisia
Community support status:
done: triaged all recent (couple weeks) issues
wip: triaging older issues and starting on the other repos
PR reviews
tackle my broken test for https://github.com/vmware-tanzu/velero/pull/3004 (DownloadRequest migration to kubebuilder)
nrb
Fixed https://github.com/vmware-tanzu/velero/issues/2570, plan to get v1.5.2 & v1.4.3 out this week.
Planning our support-a-thon to reduce the number of open support issues this week, Wednesday to Friday
v1.6 checkup!
bridget
Made some changes to #2993 based on comments
Discovered a problem with approach in newest changes so need to revisit that approach and modify tests
Helped Ashish with internal release processes last week
Reviewing design docs
dsmithuchida
PTO last week
Working on e2e tests
Discussion Topics
carlisia
Should we add this to our list of supported plugins: https://github.com/Lirt/velero-plugin-swift/
How do we handle requests for a new plugin: https://github.com/vmware-tanzu/velero/issues/2988
Add them to https://github.com/vmware-tanzu/velero/tree/main/site/content/plugins/list
Also https://github.com/vmware-tanzu/velero/blob/main/site/content/docs/main/supported-providers.md
We are hiring a senior Product Manager to lead Velero: https://twitter.com/carlisia/status/1318304806784573440 (spread the word please!)
Jonas got a promotion, congratulations!! Well deserved title of Head of Community Management
nrb
What do folks think about moving from ZenHub to GitHub Projects?
Example: https://github.com/orgs/projectcontour/projects/2
Advantages:
Integrated (doesn’t need a 3rd party site/plugin)
Can create multiple project boards for different things
Disadvantages:
GitHub issues aren’t automatically added to the board (maybe GitHub Actions exist for this?)
https://github.com/marketplace/actions/github-project-automation
No concept of epics
One time switch cost
v1.6 priorities
Implementing e2e testing
Continue migrating to kubebuilder/controller-runtime
Enable the use of custom credential providers
Upgrade to restic v0.10.0
Phuong
Backup in parallel https://github.com/vmware-tanzu/velero/issues/2888: maybe just backing up pods in parallel
Enhance Backup with plugin timeout (https://github.com/vmware-tanzu/velero/issues/2922): pull request submitted https://github.com/vmware-tanzu/velero/pull/3018
dymurray
Follow up from previous meeting, restore progress design proposal: https://github.com/vmware-tanzu/velero/pull/3016
Community Shoutouts
@betta1: Fix BSL controller to avoid invoking init() on all BSLs regardless of…
@gliptak: Bump Go to 1.15
Arianit Uka for reporting https://github.com/vmware-tanzu/velero/issues/2570
Pranav Gaikwad for https://github.com/vmware-tanzu/velero/pull/3016
Status Updates
carlisia
Community support status:
done: triaged all recent (couple weeks) issues
wip: triaging older issues and starting on the other repos
PR reviews
tackle my broken test for https://github.com/vmware-tanzu/velero/pull/3004 (DownloadRequest migration to kubebuilder)
nrb
Fixed https://github.com/vmware-tanzu/velero/issues/2570, plan to get v1.5.2 & v1.4.3 out this week.
Planning our support-a-thon to reduce the number of open support issues this week, Wednesday to Friday
v1.6 checkup!
bridget
Made some changes to #2993 based on comments
Discovered a problem with approach in newest changes so need to revisit that approach and modify tests
Helped Ashish with internal release processes last week
Reviewing design docs
dsmithuchida
PTO last week
Working on e2e tests
Discussion Topics
carlisia
Should we add this to our list of supported plugins: https://github.com/Lirt/velero-plugin-swift/
How do we handle requests for a new plugin: https://github.com/vmware-tanzu/velero/issues/2988
Add them to https://github.com/vmware-tanzu/velero/tree/main/site/content/plugins/list
Also https://github.com/vmware-tanzu/velero/blob/main/site/content/docs/main/supported-providers.md
We are hiring a senior Product Manager to lead Velero: https://twitter.com/carlisia/status/1318304806784573440 (spread the word please!)
Jonas got a promotion, congratulations!! Well deserved title of Head of Community Management
nrb
What do folks think about moving from ZenHub to GitHub Projects?
Example: https://github.com/orgs/projectcontour/projects/2
Advantages:
Integrated (doesn’t need a 3rd party site/plugin)
Can create multiple project boards for different things
Disadvantages:
GitHub issues aren’t automatically added to the board (maybe GitHub Actions exist for this?)
https://github.com/marketplace/actions/github-project-automation
No concept of epics
One time switch cost
v1.6 priorities
Implementing e2e testing
Continue migrating to kubebuilder/controller-runtime
Enable the use of custom credential providers
Upgrade to restic v0.10.0
Phuong
Backup in parallel https://github.com/vmware-tanzu/velero/issues/2888: maybe just backing up pods in parallel
Enhance Backup with plugin timeout (https://github.com/vmware-tanzu/velero/issues/2922): pull request submitted https://github.com/vmware-tanzu/velero/pull/3018
dymurray
Follow up from previous meeting, restore progress design proposal: https://github.com/vmware-tanzu/velero/pull/3016
Community Shoutouts
@betta1: Fix BSL controller to avoid invoking init() on all BSLs regardless of…
@gliptak: Bump Go to 1.15
Arianit Uka for reporting https://github.com/vmware-tanzu/velero/issues/2570
Pranav Gaikwad for https://github.com/vmware-tanzu/velero/pull/3016
- 10 participants
- 59 minutes
13 Oct 2020
Oct 13, 2020
Status Updates
carlisia
DownloadRequest migration to kubebuilder/runtime-controller: https://github.com/vmware-tanzu/velero/pull/3004
Testing https://github.com/vmware-tanzu/velero/issues/2991 (BSL controller loop always invokes init() regardless of backup location’s ValidationFrequency)
PR reviews
nrb
Code for https://github.com/vmware-tanzu/velero/issues/2570, but need to test against clusters now.
Working on community issue support process with Carlisia to help improve response time on GitHub issues.
Need to get the Azure and CSI plugin updated and out in the next day or so.
Noticed PRs that go from draft to open aren’t getting processed by the review Action; may look into it, or just file an issue.
bridget
PR for deleting backups when cloud resources have been deleted (https://github.com/vmware-tanzu/velero/pull/2993). Need to follow up on comments from @nrb.
Discussion Topics
Phuong
enhance BackupItemActionPlugin with timeout https://github.com/vmware-tanzu/velero/issues/2922
Aspects to this are in the wait for AdditionalItems to be ready
[nrb] Can we make timeouts work for both?
[nrb] Are the timeouts per plugin, per plugin type, or universal?
sseago
https://github.com/vmware-tanzu/velero/pull/2867 was for a specific CR in OpenShift
[nrb] there’s a use case for the TAS Postgres operator now too
dymurray
Supporting restore progress reporting https://github.com/vmware-tanzu/velero/issues/21
Tackle restores similar to the backups right now as a first phase
Current issue is on restore we don’t have a fully accurate number of items. For example, when using --include-namespaces, we don’t know the number of to-be-restored resources ahead of time.
Getting a 2 pass restore system would help immensely here, since we would pre-process and match resources before sending them to the API server. This would give us an overall count to report for progress.
Antony
https://github.com/vmware-tanzu/velero/issues/2991:
BSL controller initializes backup storage locations almost every second
Significant issue for object stores where init has expensive calls (e.g API calls)
Any chance we could get a hotfix? When does v1.5.2 ship?
Contributor Shoutouts
@mickkael: Allow Timezone change in the container
@jenting: [velero] Add helm annotation for velero.io/v1 schedule CRD
Status Updates
carlisia
DownloadRequest migration to kubebuilder/runtime-controller: https://github.com/vmware-tanzu/velero/pull/3004
Testing https://github.com/vmware-tanzu/velero/issues/2991 (BSL controller loop always invokes init() regardless of backup location’s ValidationFrequency)
PR reviews
nrb
Code for https://github.com/vmware-tanzu/velero/issues/2570, but need to test against clusters now.
Working on community issue support process with Carlisia to help improve response time on GitHub issues.
Need to get the Azure and CSI plugin updated and out in the next day or so.
Noticed PRs that go from draft to open aren’t getting processed by the review Action; may look into it, or just file an issue.
bridget
PR for deleting backups when cloud resources have been deleted (https://github.com/vmware-tanzu/velero/pull/2993). Need to follow up on comments from @nrb.
Discussion Topics
Phuong
enhance BackupItemActionPlugin with timeout https://github.com/vmware-tanzu/velero/issues/2922
Aspects to this are in the wait for AdditionalItems to be ready
[nrb] Can we make timeouts work for both?
[nrb] Are the timeouts per plugin, per plugin type, or universal?
sseago
https://github.com/vmware-tanzu/velero/pull/2867 was for a specific CR in OpenShift
[nrb] there’s a use case for the TAS Postgres operator now too
dymurray
Supporting restore progress reporting https://github.com/vmware-tanzu/velero/issues/21
Tackle restores similar to the backups right now as a first phase
Current issue is on restore we don’t have a fully accurate number of items. For example, when using --include-namespaces, we don’t know the number of to-be-restored resources ahead of time.
Getting a 2 pass restore system would help immensely here, since we would pre-process and match resources before sending them to the API server. This would give us an overall count to report for progress.
Antony
https://github.com/vmware-tanzu/velero/issues/2991:
BSL controller initializes backup storage locations almost every second
Significant issue for object stores where init has expensive calls (e.g API calls)
Any chance we could get a hotfix? When does v1.5.2 ship?
Contributor Shoutouts
@mickkael: Allow Timezone change in the container
@jenting: [velero] Add helm annotation for velero.io/v1 schedule CRD
- 10 participants
- 54 minutes
6 Oct 2020
Oct 6, 2020
Status Updates
dsmithuchida
Working on e2e testing and ginkgo framework
bridget
Couple of outstanding PRs that I’d like reviewed (#2951, #2969)
Been working on some bugs:
Backup deletion fails if cloud resources already deleted #2980
This is a regression in behaviour from v1.4.2 due to the introduction of DeleteItemActions. We can introduce a fix, but how should we handle the case where there are DeleteItemActions to perform but the resources are deleted?
Unable to install Velero using go get #2966
We can fix this by upgrading the version of CSI external-snapshotter but it requires updating the version of Kubernetes modules we’re using to 0.19.0. What is the impact of upgrading these libraries?
nrb
Not a whole lot of updates; was on semi-PTO last week.
Intending to do a “support-a-thon”/bug smash session when the whole team is back
Discussion Topics
Waiting for Additional Items to restore
Issue #2980 - deleteitemactions regression
Happens all the time, but what do we do when there’s DeleteItemActions but no backup tarball?
Since there’s no way to really know what to match the plugins against, log that there was no tarball and continue, skipping the plugins.
Community Contributions
@sseago: restore proper lowercase/plural CRD resource
@svcua: [velero] Bump Velero version to v1.5.1
@xUnholy: [velero] fix bug using workload identity issue #161
@nirav-chotai: [Velero] Fixing restic daemonset yaml
Status Updates
dsmithuchida
Working on e2e testing and ginkgo framework
bridget
Couple of outstanding PRs that I’d like reviewed (#2951, #2969)
Been working on some bugs:
Backup deletion fails if cloud resources already deleted #2980
This is a regression in behaviour from v1.4.2 due to the introduction of DeleteItemActions. We can introduce a fix, but how should we handle the case where there are DeleteItemActions to perform but the resources are deleted?
Unable to install Velero using go get #2966
We can fix this by upgrading the version of CSI external-snapshotter but it requires updating the version of Kubernetes modules we’re using to 0.19.0. What is the impact of upgrading these libraries?
nrb
Not a whole lot of updates; was on semi-PTO last week.
Intending to do a “support-a-thon”/bug smash session when the whole team is back
Discussion Topics
Waiting for Additional Items to restore
Issue #2980 - deleteitemactions regression
Happens all the time, but what do we do when there’s DeleteItemActions but no backup tarball?
Since there’s no way to really know what to match the plugins against, log that there was no tarball and continue, skipping the plugins.
Community Contributions
@sseago: restore proper lowercase/plural CRD resource
@svcua: [velero] Bump Velero version to v1.5.1
@xUnholy: [velero] fix bug using workload identity issue #161
@nirav-chotai: [Velero] Fixing restic daemonset yaml
- 6 participants
- 27 minutes
22 Sep 2020
Sept 22, 2020
Status Updates
nrb
Documenting plugin release process
Need to get Azure and CSI plugins released
Stop trying to push docker images on forks
Reviews on main repo.
Thinking on ways to make Velero more concurrent/faster as part of v1.6; no concrete suggestions yet, I owe a high level goals doc per discussion last week.
Maintainers please take a look at https://github.com/vmware-tanzu/velero-plugin-for-csi/pull/70; needed for the next version of the CSI plugin
carlisia
Last week worked on v1.5.1 release
Reviews
Piling up: more reviews, GH issues + Helm chart
working on the CLI side of the download request migration to kubebuilder/controller-runtime.
bridget
Worked on some improvements to our release scripts following the 1.5.1 release
Now looking at internal build processes
dsmithuchida
Resource blocking in vSphere plug-in for Project Pacific internal resources
OpenSource project approvals for GVDDK (currently part of Astrolabe) and Data Generator (Kibishii) test tool
Discussion Topics
nrb: Defining prerelease to GA timeframes
From last week:
Prereleases ~1 week between each?
RC to GA, ~2 weeks?
Phuong - 3 month release cycle is acceptable for them, waiting for features on that timeframe is reasonable. Longer is too much.
The RC was helpful to integrate and test with their product. It meant updating to the actual release was just removing some characters.
1 week seemed reasonable to them, but if they hit bugs it may not be enough time.
Dylan - Red Hat’s Konveyor lags a little. They want to support older Kubernetes releases like v1.7 due to supporting OpenShift 3. (this isn’t determined for upstream Velero)
Konveyor does OpenShift 3-4 and 4-4 migration.
Red Hat also has OADP for OpenShift data protection, and their Velero fork is used there, too.
Red Hat likes the 3 month release cycle, especially for OADP. Backwards compatibility in Konveyor is trickier.
For OADP, they tried the RC for basic tasks.
poojita: Recover OpenShift’s native resource: DeploymentConfig
Velero restore error seen has been captured below:
Velero failed to restore namespace frank3. {“namespaces”:{“frank3”:[“error restoring imagetags.image.openshift.io/frank3/httpd-example:latest: ImageTag.image.openshift.io \“httpd-example:latest\” is invalid: spec: Required value: spec is a required field during creation”]}}. Velero restore ‘51685651-5369-51e1-88df-8977874919ca-2020-09-16-08-53-48-frank3’ failed: {\n “phase”: “PartiallyFailed”,\n “warnings”: 6,\n “errors”: 1\n}."
Dylan - There’s an OpenShift plugin that can help restore these CRDs on vanilla Velero installs. https://github.com/konveyor/openshift-velero-plugin
For ImageTag, it skips restore of these
Plugin recreates ImageStream, which then recreates the ImageTag.
ImageTag is a new, undocumented resource in OpenShift v4.4.
The plugin is used in the context of the migration product (Konveyor) and data protection (OADP).
Can file GitHub issues on that repo if you have issues/questions.
OADP bundles Velero and the AWS plugin on OpenShift. The benefit here is that it backs up the images to the S3 bucket.
Plugin can’t do it by itself right now. OADP sets up a image repo that the plugin doesn’t orchestrate right now
Alay - Plugin dependencies are challenging
If you deploy Velero by itself, without a wrapper, there are challenges in connecting plugins to external systems such as StatefulSets or Deployments.
Dave: Difference between the vSphere plugin and OADP is that OADP is at what point in the lifecycle they’re active.
May be able to extend plugins via the Velero app operator that VMware is working on.
RH’s requirements:
Before the plugin runs, ensure the dependency is healthy
During/after backup, ensure the dependency is healthy.
If it’s not, short circuit the operation instead of trying the full backup operation and failing.
Narashima - Took a backup w/ Velero and tried to restore it to another cluster. ReplicaSets/Deployments are getting duplicated in the new cluster.
kubectl get shows multiple entries.
Shoutouts
Slow week for contributions, but thanks to mikkael for the PR to allow users to change the container’s timezone
Status Updates
nrb
Documenting plugin release process
Need to get Azure and CSI plugins released
Stop trying to push docker images on forks
Reviews on main repo.
Thinking on ways to make Velero more concurrent/faster as part of v1.6; no concrete suggestions yet, I owe a high level goals doc per discussion last week.
Maintainers please take a look at https://github.com/vmware-tanzu/velero-plugin-for-csi/pull/70; needed for the next version of the CSI plugin
carlisia
Last week worked on v1.5.1 release
Reviews
Piling up: more reviews, GH issues + Helm chart
working on the CLI side of the download request migration to kubebuilder/controller-runtime.
bridget
Worked on some improvements to our release scripts following the 1.5.1 release
Now looking at internal build processes
dsmithuchida
Resource blocking in vSphere plug-in for Project Pacific internal resources
OpenSource project approvals for GVDDK (currently part of Astrolabe) and Data Generator (Kibishii) test tool
Discussion Topics
nrb: Defining prerelease to GA timeframes
From last week:
Prereleases ~1 week between each?
RC to GA, ~2 weeks?
Phuong - 3 month release cycle is acceptable for them, waiting for features on that timeframe is reasonable. Longer is too much.
The RC was helpful to integrate and test with their product. It meant updating to the actual release was just removing some characters.
1 week seemed reasonable to them, but if they hit bugs it may not be enough time.
Dylan - Red Hat’s Konveyor lags a little. They want to support older Kubernetes releases like v1.7 due to supporting OpenShift 3. (this isn’t determined for upstream Velero)
Konveyor does OpenShift 3-4 and 4-4 migration.
Red Hat also has OADP for OpenShift data protection, and their Velero fork is used there, too.
Red Hat likes the 3 month release cycle, especially for OADP. Backwards compatibility in Konveyor is trickier.
For OADP, they tried the RC for basic tasks.
poojita: Recover OpenShift’s native resource: DeploymentConfig
Velero restore error seen has been captured below:
Velero failed to restore namespace frank3. {“namespaces”:{“frank3”:[“error restoring imagetags.image.openshift.io/frank3/httpd-example:latest: ImageTag.image.openshift.io \“httpd-example:latest\” is invalid: spec: Required value: spec is a required field during creation”]}}. Velero restore ‘51685651-5369-51e1-88df-8977874919ca-2020-09-16-08-53-48-frank3’ failed: {\n “phase”: “PartiallyFailed”,\n “warnings”: 6,\n “errors”: 1\n}."
Dylan - There’s an OpenShift plugin that can help restore these CRDs on vanilla Velero installs. https://github.com/konveyor/openshift-velero-plugin
For ImageTag, it skips restore of these
Plugin recreates ImageStream, which then recreates the ImageTag.
ImageTag is a new, undocumented resource in OpenShift v4.4.
The plugin is used in the context of the migration product (Konveyor) and data protection (OADP).
Can file GitHub issues on that repo if you have issues/questions.
OADP bundles Velero and the AWS plugin on OpenShift. The benefit here is that it backs up the images to the S3 bucket.
Plugin can’t do it by itself right now. OADP sets up a image repo that the plugin doesn’t orchestrate right now
Alay - Plugin dependencies are challenging
If you deploy Velero by itself, without a wrapper, there are challenges in connecting plugins to external systems such as StatefulSets or Deployments.
Dave: Difference between the vSphere plugin and OADP is that OADP is at what point in the lifecycle they’re active.
May be able to extend plugins via the Velero app operator that VMware is working on.
RH’s requirements:
Before the plugin runs, ensure the dependency is healthy
During/after backup, ensure the dependency is healthy.
If it’s not, short circuit the operation instead of trying the full backup operation and failing.
Narashima - Took a backup w/ Velero and tried to restore it to another cluster. ReplicaSets/Deployments are getting duplicated in the new cluster.
kubectl get shows multiple entries.
Shoutouts
Slow week for contributions, but thanks to mikkael for the PR to allow users to change the container’s timezone
- 11 participants
- 50 minutes
8 Sep 2020
Sept 7, 2020
Status Updates
carlisia
RC this week
On community support
Nolan is on PTO
Working on the download request migration
ashish
1.5 release testing
restore hooks PR:
testing
PR 2804 reviews
getting that merged hopefully today.
bridget
Reviewing PRs for the 1.5 release
Discussion Topics
Phuong: Timeout for plugins. For app consistent backup, the pod will be quiesced while snapshot being taken. If the PVC plugin taking the snapshot and that plugin take a long time (say because it fails and retries), the application will be blocked for a long time which is unwanted behavior.
Contributor Shoutouts
@jenting: fix EnableAPIGroupersions output log format
@pawanpraka1: fix: rename the PV if VolumeSnapshotter has modified the PV name
@arminc: [velero] Adding imagePullSecrets to the chart
Status Updates
carlisia
RC this week
On community support
Nolan is on PTO
Working on the download request migration
ashish
1.5 release testing
restore hooks PR:
testing
PR 2804 reviews
getting that merged hopefully today.
bridget
Reviewing PRs for the 1.5 release
Discussion Topics
Phuong: Timeout for plugins. For app consistent backup, the pod will be quiesced while snapshot being taken. If the PVC plugin taking the snapshot and that plugin take a long time (say because it fails and retries), the application will be blocked for a long time which is unwanted behavior.
Contributor Shoutouts
@jenting: fix EnableAPIGroupersions output log format
@pawanpraka1: fix: rename the PV if VolumeSnapshotter has modified the PV name
@arminc: [velero] Adding imagePullSecrets to the chart
- 7 participants
- 24 minutes
1 Sep 2020
Sept 1, 2020
Status Updates
carlisia
pto last Friday
addressing code reviews
going to review/test PRs for 1.5
nrb
Planning for Q4 w/ VMware
Some planning for v1.6
reviewing PRs
v1.5.0 delayed (in discussion topics)
Welcoming Dave!
Community support this week
On vacation next week, will transfer responsibilities over to folks prior
bridget
Finally transitioned over to the Velero project
Getting up to speed
Reviewing PRs for 1.5
Discussion Topics
nrb: v1.5.0 has been delayed. We’ve got a list of release blocker PRs that we’d like to get through, and will not add more code changes unless there’s a critical bug/security fix. The PRs are in relative priority.
Timeframe - ~2 weeks
Testing will be more automated starting in v1.6, but we don’t have a defined e2e test suite right now, just whatever is in make ci
nrb: VMware will be having a company holiday on Friday, September 4 and Monday, September 7 will be US Labor Day.
carlisia: do we want to halt the Velero server in this case: “Invalid value for --disable-controllers flag provided” (https://github.com/vmware-tanzu/velero/blob/976484553062262324cc1304607ea00841262db8/pkg/cmd/server/server.go#L822)
Action Item: Carlisia to open an issue to discuss this and document the issue
phuong:
quick update on the backup cluster resources
Able to back up clustorerole and pv in the context of a namespace, don’t need to modify the selection logic
backing up PVC in parallel
Taking backups of pods in logical group in a serial manner could lead to drift
This is parallelization w/in a backup job, not parallelization of all jobs
Currently this would be VolumeSnapshotter plugins - may not be a resource contention issue
This is the main issue here - use case is for Cassandra
How would this work with the CSI plugin? Since it’s BackupItemAction, the interface is way different
At the Kubernetes API server level, would we want to specify parallelism per resource type?
CRDs may need this, too
Need to be careful to not overflow resource limits
Don’t want to overwhelm writing to the backup tarball
Once we get the body, that could be funneled to the tarball serially
Action Item: Phuong to make an issue to continue discussion
sseago: posted design doc for RestoreItemAction “wait for additional item to be ready” (https://github.com/vmware-tanzu/velero/pull/2867)
To add timeouts, we could use https://velero.io/docs/v1.4/custom-plugins/#plugin-configuration
brito-rafa: will work with jenting on supporting multiple API Groups (https://github.com/vmware-tanzu/velero/issues/2551)
Contributor Shoutouts
@a-mccarthy: point contributors to style guide
@a-mccarthy: add hugo default TOC
@a-mccarthy: add new table shortcode
@a-mccarthy: update docs to match style guide
@a-mccarthy: Update release checklist to include more info around blog posts and r…
@c3y1huang: [velero] Fix helm 3.3.0 lint issue
Status Updates
carlisia
pto last Friday
addressing code reviews
going to review/test PRs for 1.5
nrb
Planning for Q4 w/ VMware
Some planning for v1.6
reviewing PRs
v1.5.0 delayed (in discussion topics)
Welcoming Dave!
Community support this week
On vacation next week, will transfer responsibilities over to folks prior
bridget
Finally transitioned over to the Velero project
Getting up to speed
Reviewing PRs for 1.5
Discussion Topics
nrb: v1.5.0 has been delayed. We’ve got a list of release blocker PRs that we’d like to get through, and will not add more code changes unless there’s a critical bug/security fix. The PRs are in relative priority.
Timeframe - ~2 weeks
Testing will be more automated starting in v1.6, but we don’t have a defined e2e test suite right now, just whatever is in make ci
nrb: VMware will be having a company holiday on Friday, September 4 and Monday, September 7 will be US Labor Day.
carlisia: do we want to halt the Velero server in this case: “Invalid value for --disable-controllers flag provided” (https://github.com/vmware-tanzu/velero/blob/976484553062262324cc1304607ea00841262db8/pkg/cmd/server/server.go#L822)
Action Item: Carlisia to open an issue to discuss this and document the issue
phuong:
quick update on the backup cluster resources
Able to back up clustorerole and pv in the context of a namespace, don’t need to modify the selection logic
backing up PVC in parallel
Taking backups of pods in logical group in a serial manner could lead to drift
This is parallelization w/in a backup job, not parallelization of all jobs
Currently this would be VolumeSnapshotter plugins - may not be a resource contention issue
This is the main issue here - use case is for Cassandra
How would this work with the CSI plugin? Since it’s BackupItemAction, the interface is way different
At the Kubernetes API server level, would we want to specify parallelism per resource type?
CRDs may need this, too
Need to be careful to not overflow resource limits
Don’t want to overwhelm writing to the backup tarball
Once we get the body, that could be funneled to the tarball serially
Action Item: Phuong to make an issue to continue discussion
sseago: posted design doc for RestoreItemAction “wait for additional item to be ready” (https://github.com/vmware-tanzu/velero/pull/2867)
To add timeouts, we could use https://velero.io/docs/v1.4/custom-plugins/#plugin-configuration
brito-rafa: will work with jenting on supporting multiple API Groups (https://github.com/vmware-tanzu/velero/issues/2551)
Contributor Shoutouts
@a-mccarthy: point contributors to style guide
@a-mccarthy: add hugo default TOC
@a-mccarthy: add new table shortcode
@a-mccarthy: update docs to match style guide
@a-mccarthy: Update release checklist to include more info around blog posts and r…
@c3y1huang: [velero] Fix helm 3.3.0 lint issue
- 11 participants
- 54 minutes
25 Aug 2020
August 25, 2020
Status Updates
stephanie
building the priority/stack rank list for the 1.6 release
this will be published after we complete planning next week for the community to refer to and will include additional details that compliment our current product roadmap
ashish
Community support this week.
Misc work for the 1.5 release:
Pre-release is out. Try it out in your non-prod environments
Bug fix for server version not being correctly displayed as output of velero version.
Updating some docs
Getting PRs reviewed and merged.
Testing out the release
Will be taking another look at the exec restore hooks PR.
carlisia
community support last week
ServerStatusRequest PR (migration to kubebuilder): https://github.com/vmware-tanzu/velero/pull/2838
wip: DownloadRequest migration to kubebuilder
nrb
released v1.5.0-beta.1
testing the beta - please try it out in your environments, too!
planning for v1.6
Owe a review on the restore hooks exec work
Discussion Topics
nrb: We aim to have v1.5.0 out by September 1
Let us know what Velero code you depend on
Defining printer columns
Do we need to get the exact same values as Velero’s CLI in kubectl printer columns?
Not necessarily - they can be raw values, but we can use webhooks for calculations.
Phuong updating go modules
1.5.0-beta.1 generating CRs having issues
The error has an issue with missing a context argument - related to upgrading to Kubernetes v1.18
ERROR] generated/clientset/versioned/typed/applicationtemplate/v1beta1/applicationtemplate.go:73:5: not enough arguments in call to c.client.Get().Namespace(c.ns).Resource("applicationtemplates").Name(name).VersionedParams(&options, scheme.ParameterCodec).Do
[ERROR] have ()
[ERROR] want (context.Context)
Contributor Shoutouts
@a-mccarthy: add index files to api types folder
@ipochi: docs: add metadata to resource-filtering.md
@bgagnon: Override logrus.ErrorKey when json logging is enabled
@jenting: doc: fix feature flag EnableCSI on velero backup describe command only
Status Updates
stephanie
building the priority/stack rank list for the 1.6 release
this will be published after we complete planning next week for the community to refer to and will include additional details that compliment our current product roadmap
ashish
Community support this week.
Misc work for the 1.5 release:
Pre-release is out. Try it out in your non-prod environments
Bug fix for server version not being correctly displayed as output of velero version.
Updating some docs
Getting PRs reviewed and merged.
Testing out the release
Will be taking another look at the exec restore hooks PR.
carlisia
community support last week
ServerStatusRequest PR (migration to kubebuilder): https://github.com/vmware-tanzu/velero/pull/2838
wip: DownloadRequest migration to kubebuilder
nrb
released v1.5.0-beta.1
testing the beta - please try it out in your environments, too!
planning for v1.6
Owe a review on the restore hooks exec work
Discussion Topics
nrb: We aim to have v1.5.0 out by September 1
Let us know what Velero code you depend on
Defining printer columns
Do we need to get the exact same values as Velero’s CLI in kubectl printer columns?
Not necessarily - they can be raw values, but we can use webhooks for calculations.
Phuong updating go modules
1.5.0-beta.1 generating CRs having issues
The error has an issue with missing a context argument - related to upgrading to Kubernetes v1.18
ERROR] generated/clientset/versioned/typed/applicationtemplate/v1beta1/applicationtemplate.go:73:5: not enough arguments in call to c.client.Get().Namespace(c.ns).Resource("applicationtemplates").Name(name).VersionedParams(&options, scheme.ParameterCodec).Do
[ERROR] have ()
[ERROR] want (context.Context)
Contributor Shoutouts
@a-mccarthy: add index files to api types folder
@ipochi: docs: add metadata to resource-filtering.md
@bgagnon: Override logrus.ErrorKey when json logging is enabled
@jenting: doc: fix feature flag EnableCSI on velero backup describe command only
- 6 participants
- 27 minutes
18 Aug 2020
August 18, 2020
Status Updates
ashish
Mostly preparing for 1.5 release and addressing any outstanding items, doc fixes and PR reviews.
I owe @areed another round of review on the exec restore hooks PR. Going to do that today.
During investigation of Issue #2789 added some documentation to our wiki on setting up standalone MinIO object storage using custom ca cert and self-signed certificates with https to use with Velero
nrb
Worked on adding the DeleteItemActions plugin type in https://github.com/vmware-tanzu/velero/pull/2808 and https://github.com/vmware-tanzu/velero/pull/2815
Reviews
Looking to review release process w/ Abbie in light of the v1.5 beta release
carlisia
On community support this week
Lots of PR reviews last week
wip (ServerStatusRequest to kubebuilder): https://github.com/vmware-tanzu/velero/compare/main...carlisia:c-status-request
Discussion Topics
Phuong Hoang: avoid backup PersistentVolumes not referenced by included namespaces when enable --include-cluster-resources
Use --exclude-resources=persistentvolumes?
We still want to include the PersistentVolumes being used by PVCs belong to the namespace being backed up but not including other PersistentVolumes
Plugin additional items may ignore this, but we can’t remember
Also want to include other cluster-scoped resources like ClusterRoleBindings
Sounds like we want OR selector logic support
One idea - use --include-resources and specify exactly what you want
Limitation: you have to specify exactly what you want
Action item: Phuong will investigate a proposal to enhance item selection.
stephanie
Velero 1.6 release planning (starting around v1.5 beta release)
We’ll share this around Sept 1 in ZenHub & GitHub
Will update roadmap.md, too
Between now and then, please :+1: any issues that you’re interested in seeing in the v1.6 release
Let us know what Velero code you depend on
Ashish: Velero Office Hours are coming, exact dates & times to be announced
Contributor Shoutouts
@a-mccarthy: add note about windows support
@runzexia: add kindfor func to get apiresource from gvk
@jenting: [velero] bump velero version to v1.4.2
@jenting: fix: pull request template not working
@cpanato: Update GH-pages branch
@cpanato: actions: update helm actions to use release 1.0.0
@jenting: Add Github issue template
@jenting: fix: custom CA not working
Status Updates
ashish
Mostly preparing for 1.5 release and addressing any outstanding items, doc fixes and PR reviews.
I owe @areed another round of review on the exec restore hooks PR. Going to do that today.
During investigation of Issue #2789 added some documentation to our wiki on setting up standalone MinIO object storage using custom ca cert and self-signed certificates with https to use with Velero
nrb
Worked on adding the DeleteItemActions plugin type in https://github.com/vmware-tanzu/velero/pull/2808 and https://github.com/vmware-tanzu/velero/pull/2815
Reviews
Looking to review release process w/ Abbie in light of the v1.5 beta release
carlisia
On community support this week
Lots of PR reviews last week
wip (ServerStatusRequest to kubebuilder): https://github.com/vmware-tanzu/velero/compare/main...carlisia:c-status-request
Discussion Topics
Phuong Hoang: avoid backup PersistentVolumes not referenced by included namespaces when enable --include-cluster-resources
Use --exclude-resources=persistentvolumes?
We still want to include the PersistentVolumes being used by PVCs belong to the namespace being backed up but not including other PersistentVolumes
Plugin additional items may ignore this, but we can’t remember
Also want to include other cluster-scoped resources like ClusterRoleBindings
Sounds like we want OR selector logic support
One idea - use --include-resources and specify exactly what you want
Limitation: you have to specify exactly what you want
Action item: Phuong will investigate a proposal to enhance item selection.
stephanie
Velero 1.6 release planning (starting around v1.5 beta release)
We’ll share this around Sept 1 in ZenHub & GitHub
Will update roadmap.md, too
Between now and then, please :+1: any issues that you’re interested in seeing in the v1.6 release
Let us know what Velero code you depend on
Ashish: Velero Office Hours are coming, exact dates & times to be announced
Contributor Shoutouts
@a-mccarthy: add note about windows support
@runzexia: add kindfor func to get apiresource from gvk
@jenting: [velero] bump velero version to v1.4.2
@jenting: fix: pull request template not working
@cpanato: Update GH-pages branch
@cpanato: actions: update helm actions to use release 1.0.0
@jenting: Add Github issue template
@jenting: fix: custom CA not working
- 7 participants
- 28 minutes
11 Aug 2020
August 11, 2020
Status Updates
carlisia
PTO last week
Reviewing PRs
Back to the last of migrating ServerStatusRequest to kubebuilder
ashish
Community support last week
Got to a bunch of PRs and GH issues.
Restore Hooks:
PR for Init container restore hooks PR 2787
PR to fix API tyes PR 2793
Fix to pass the --default-volumes-to-restic flag from schedules to the backup options. PR 2776
Setup CRD validation CI in GitHub actions PR 2805
Currently setup to run against kubernetes version 1.16.9, 1.17.0 and 1.18.4
AI: Switch to latest of each minor release
Restructuring build broke the EntryPoint for the restic-restore helper. PR 2802
PRs in the CSI plugin repo:
Updating the CI in the repo to push images with latest tag. PR 65
Better handling of restore when VolumeSnapshot resources exist but the associated workloads got deleted PR 64
nrb
Community support this week
Working on DeleteItemAction implementation
Design doc
Trying to get GitHub Actions working to automatically assign reviewers on PRs when opened. (https://github.com/vmware-tanzu/velero/pull/2807)
Aiming to have a 1.5 beta out by August 21st at the latest.
stephanie
We are aiming to have 1.6 release planning start following the 1.5 beta at the end of August
refer to our roadmap for items currently planned for December 2020 which corresponds with the 1.6 release
Velero security policy published
Discussion Topics
Introducing Bridget!
Let us know what Velero code you depend on
Velero-plugin-for-CSI PR 1902 KEP to add validation hooks on the VolumeSnapshot API objects.
Took a look at it and these don’t seem to violate our assumptions/expectations
Would love to have more 👀 on it
What versions of Kubernetes does Velero support?
Alay: Red Hat supports all the way back to Kube v1.7, will bump up to v1.9. Uses Velero v1.3
An LTS
Contributor Shoutouts
@alexcpn: Update minio.md
@bgagnon: Always use groupResource.String() when logging (fixes #2795)
@doughepi: Add the ability to set the allowPrivilegeEscalation property on the Restic restore helper via plugin ConfigMap
@RobReus: Reverting change on 1.4 docs and re-applying to main docs
@skriss: update CI badge on README
@a-mccarthy: update docs to include cpu/memory defaults for restic
@mtritabaugh: Add resource filtering page
@RobReus: Refactor image builds to use buildx for multi arch image building
@cfreemoser: Fixed wrong environment variable in Readme
@sabvente: Remove extra indentation from README
Status Updates
carlisia
PTO last week
Reviewing PRs
Back to the last of migrating ServerStatusRequest to kubebuilder
ashish
Community support last week
Got to a bunch of PRs and GH issues.
Restore Hooks:
PR for Init container restore hooks PR 2787
PR to fix API tyes PR 2793
Fix to pass the --default-volumes-to-restic flag from schedules to the backup options. PR 2776
Setup CRD validation CI in GitHub actions PR 2805
Currently setup to run against kubernetes version 1.16.9, 1.17.0 and 1.18.4
AI: Switch to latest of each minor release
Restructuring build broke the EntryPoint for the restic-restore helper. PR 2802
PRs in the CSI plugin repo:
Updating the CI in the repo to push images with latest tag. PR 65
Better handling of restore when VolumeSnapshot resources exist but the associated workloads got deleted PR 64
nrb
Community support this week
Working on DeleteItemAction implementation
Design doc
Trying to get GitHub Actions working to automatically assign reviewers on PRs when opened. (https://github.com/vmware-tanzu/velero/pull/2807)
Aiming to have a 1.5 beta out by August 21st at the latest.
stephanie
We are aiming to have 1.6 release planning start following the 1.5 beta at the end of August
refer to our roadmap for items currently planned for December 2020 which corresponds with the 1.6 release
Velero security policy published
Discussion Topics
Introducing Bridget!
Let us know what Velero code you depend on
Velero-plugin-for-CSI PR 1902 KEP to add validation hooks on the VolumeSnapshot API objects.
Took a look at it and these don’t seem to violate our assumptions/expectations
Would love to have more 👀 on it
What versions of Kubernetes does Velero support?
Alay: Red Hat supports all the way back to Kube v1.7, will bump up to v1.9. Uses Velero v1.3
An LTS
Contributor Shoutouts
@alexcpn: Update minio.md
@bgagnon: Always use groupResource.String() when logging (fixes #2795)
@doughepi: Add the ability to set the allowPrivilegeEscalation property on the Restic restore helper via plugin ConfigMap
@RobReus: Reverting change on 1.4 docs and re-applying to main docs
@skriss: update CI badge on README
@a-mccarthy: update docs to include cpu/memory defaults for restic
@mtritabaugh: Add resource filtering page
@RobReus: Refactor image builds to use buildx for multi arch image building
@cfreemoser: Fixed wrong environment variable in Readme
@sabvente: Remove extra indentation from README
- 8 participants
- 39 minutes
4 Aug 2020
August 4, 2020
Status Updates
ashish
Community support this week
Have a list of PRs to review
New issues that need triage
Working on restore hooks
PR 2787 to implement the init container restore hooks.
Need to run this through tests before removing WIP.
nrb
Out sick late last week
Working on DeleteItemAction design. Please take a look!
Investigating options on project management and release cadence to get issues handled more quickly, as well as get fixes out more regularly.
carlisia
PTO!
Working on migrating ServerStatusRequest to kubebuilder
Discussion Topics
Let us know what Velero code you depend on
Andrew Reed - need any help with exec restore hook?
How do we mitigate people being on PTO? Slackening the reviewer constaints?
Storj persistent volume support
Goal: back up any Kubernetes PV to Storj object store
rclone/restic support?
treating volume as a block device?
vSphere plugin currently has a data mover - https://github.com/vmware-tanzu/velero-plugin-for-vsphere
github.com/openebs/velero-plugin has a data mover too
Let’s start a thread on the Google Group
Contributor Shoutouts
@jenting: Add a flag cacert for velero backup-location create
@jenting: fix: helm upgrade did not work correctly on BSL and VSL configuration
@MrFreezeex: Fix Helm 2 CRD generation
@alikhtag: [impovement] Add podLabels to the helm chart
@kyounger: fix: Restic daemonset SA always set, like velero deployment
@fredgate: Add log configuration to Restic
@cesarokuti: add schedule labels
@ckotzbauer: add affinity option to velero deployment
Status Updates
ashish
Community support this week
Have a list of PRs to review
New issues that need triage
Working on restore hooks
PR 2787 to implement the init container restore hooks.
Need to run this through tests before removing WIP.
nrb
Out sick late last week
Working on DeleteItemAction design. Please take a look!
Investigating options on project management and release cadence to get issues handled more quickly, as well as get fixes out more regularly.
carlisia
PTO!
Working on migrating ServerStatusRequest to kubebuilder
Discussion Topics
Let us know what Velero code you depend on
Andrew Reed - need any help with exec restore hook?
How do we mitigate people being on PTO? Slackening the reviewer constaints?
Storj persistent volume support
Goal: back up any Kubernetes PV to Storj object store
rclone/restic support?
treating volume as a block device?
vSphere plugin currently has a data mover - https://github.com/vmware-tanzu/velero-plugin-for-vsphere
github.com/openebs/velero-plugin has a data mover too
Let’s start a thread on the Google Group
Contributor Shoutouts
@jenting: Add a flag cacert for velero backup-location create
@jenting: fix: helm upgrade did not work correctly on BSL and VSL configuration
@MrFreezeex: Fix Helm 2 CRD generation
@alikhtag: [impovement] Add podLabels to the helm chart
@kyounger: fix: Restic daemonset SA always set, like velero deployment
@fredgate: Add log configuration to Restic
@cesarokuti: add schedule labels
@ckotzbauer: add affinity option to velero deployment
- 8 participants
- 32 minutes
28 Jul 2020
July 28, 2020
Status Updates
carlisia
Community support this week
PTO next week
Bringing the Helm repo up to date with v1.4 + more Velero Helm TLC
Continue work on the ServerStatusRequest migration to kubebuilder
ashish
Restore hooks:
Adding constants PR#2750
Updating RestoreSpec PR#2761
Backing up pod volumes using restic by default:
PR 2611
Excluding volumes mounting secrets and config maps
Opening Issue 2770. Support for label selector to choose pods to which the defaultVolumesToRestic should be applied. :thumbsdown: / :thumbsup: in the issue
PR reviews
nrb
Working on some CI improvements around assignment/reviewers, but ran into issues with how GitHub Actions expose secrets
PR reviews
Intending to get DeleteItemAction design doc updated today or tomorrow
Was on community support last week
Discussion Topics
Welcome JenTing Hsiao (https://github.com/jenting) as an official maintainer of the Velero Helm chart!
Let us know what parts of the Velero API you use - https://github.com/vmware-tanzu/velero/issues/2654
Contributor Shoutouts
@areed: Add hooks fields to restore context
@thejasbabu: Add Start and Completion Timestamp to Restore Status
Helm Chart
@jenting: fix: helm upgrade did not work correctly on BSL and VSL configuration
@jenting: helm 2,3 CI test + Upgrade CRDs to v1.4
@MrFreezeex: Fix Helm 2 CRD generation
@cesarokuti: add schedule labels
@tomahawk28: Helm 2,3 supports CRD at the same time
@ckotzbauer: add affinity option to velero deployment
Status Updates
carlisia
Community support this week
PTO next week
Bringing the Helm repo up to date with v1.4 + more Velero Helm TLC
Continue work on the ServerStatusRequest migration to kubebuilder
ashish
Restore hooks:
Adding constants PR#2750
Updating RestoreSpec PR#2761
Backing up pod volumes using restic by default:
PR 2611
Excluding volumes mounting secrets and config maps
Opening Issue 2770. Support for label selector to choose pods to which the defaultVolumesToRestic should be applied. :thumbsdown: / :thumbsup: in the issue
PR reviews
nrb
Working on some CI improvements around assignment/reviewers, but ran into issues with how GitHub Actions expose secrets
PR reviews
Intending to get DeleteItemAction design doc updated today or tomorrow
Was on community support last week
Discussion Topics
Welcome JenTing Hsiao (https://github.com/jenting) as an official maintainer of the Velero Helm chart!
Let us know what parts of the Velero API you use - https://github.com/vmware-tanzu/velero/issues/2654
Contributor Shoutouts
@areed: Add hooks fields to restore context
@thejasbabu: Add Start and Completion Timestamp to Restore Status
Helm Chart
@jenting: fix: helm upgrade did not work correctly on BSL and VSL configuration
@jenting: helm 2,3 CI test + Upgrade CRDs to v1.4
@MrFreezeex: Fix Helm 2 CRD generation
@cesarokuti: add schedule labels
@tomahawk28: Helm 2,3 supports CRD at the same time
@ckotzbauer: add affinity option to velero deployment
- 6 participants
- 25 minutes
21 Jul 2020
July 21, 2020
Status Updates
ashish
Getting Release 1.4.2 released and testing the release.
PR reviews and issue investigation
PR to add metrics from restic backup 2719
nrb
PR reviews
Breakdown of restore hooks work
Merged insensitive language PRs
Added design doc for DeleteAction plugins
I have a couple small PRs ready for review: https://github.com/vmware-tanzu/velero/pull/2735 & https://github.com/vmware-tanzu/velero/pull/2705
carlisia
Community support last week
We now have a BSL controller
Wrapping up conversion of serverStatusRequest to use controller-runtime
Fixing a bug on the verify script
We started filling out our FAQ page: https://github.com/vmware-tanzu/velero/wiki/FAQ
stephanie
restore hooks product requirements reviews (https://github.com/vmware-tanzu/velero/pull/2699)
Discussion Topics
stephanie
discussion restore hooks use case #7 data quiescing (https://github.com/vmware-tanzu/velero/pull/2699/files#diff-1e4163f56a0d34eb56295c1901279753)
dave
DeleteAction discussion
Started as a high level delete, but wanted a DeleteItemAction
Act on items in the backup tarball.
Options:
Pull down the whole tarball and look through it
Add index of things that BackupItemActions created
Reminder to let us know if you’re using any Velero code at https://github.com/vmware-tanzu/velero/issues/2654
Community Shoutouts
Phuong for testing out the main branch and reminding us we need to get update instructions documented
@kathpeony: Add backupValidationFailureTotal to metrics
@fvsqr: StorageGrid compatibility
@laverya: k8s 1.18 import
@bryanro92: [FIX] add --bucket param to workload identity install
@stephanwehr: add support for incremental snapshots of Azure disks
Status Updates
ashish
Getting Release 1.4.2 released and testing the release.
PR reviews and issue investigation
PR to add metrics from restic backup 2719
nrb
PR reviews
Breakdown of restore hooks work
Merged insensitive language PRs
Added design doc for DeleteAction plugins
I have a couple small PRs ready for review: https://github.com/vmware-tanzu/velero/pull/2735 & https://github.com/vmware-tanzu/velero/pull/2705
carlisia
Community support last week
We now have a BSL controller
Wrapping up conversion of serverStatusRequest to use controller-runtime
Fixing a bug on the verify script
We started filling out our FAQ page: https://github.com/vmware-tanzu/velero/wiki/FAQ
stephanie
restore hooks product requirements reviews (https://github.com/vmware-tanzu/velero/pull/2699)
Discussion Topics
stephanie
discussion restore hooks use case #7 data quiescing (https://github.com/vmware-tanzu/velero/pull/2699/files#diff-1e4163f56a0d34eb56295c1901279753)
dave
DeleteAction discussion
Started as a high level delete, but wanted a DeleteItemAction
Act on items in the backup tarball.
Options:
Pull down the whole tarball and look through it
Add index of things that BackupItemActions created
Reminder to let us know if you’re using any Velero code at https://github.com/vmware-tanzu/velero/issues/2654
Community Shoutouts
Phuong for testing out the main branch and reminding us we need to get update instructions documented
@kathpeony: Add backupValidationFailureTotal to metrics
@fvsqr: StorageGrid compatibility
@laverya: k8s 1.18 import
@bryanro92: [FIX] add --bucket param to workload identity install
@stephanwehr: add support for incremental snapshots of Azure disks
- 9 participants
- 60 minutes
14 Jul 2020
July 14, 2020
Status Updates
(ashish)
Community support last week
Fix for capturing version of CRD before version remap plugin:
PR 2683 (https://github.com/vmware-tanzu/velero/pull/2683)
Merged last week and is in this weeks 1.4.2 release
PR reviews:
VolumesnapshotProgress design proposal PR
merged last week. 🎉
Ordered Resource backup design proposal
Couple of passes at the
Fixed Issue 17 with PR 60 in the velero-plugin-for-csi repo
allows users of the CSI plugin to keep regular volumesnapshotclass separate from that used by Velero. Primarily to allow different DeletionPolicy.
(carlisia)
Community support
Blocker (BSL controller): https://github.com/vmware-tanzu/velero/pull/2674
(nrb)
v1.4.2 is out!
Includes fixes from CRD
Going to try to help catch up on community support this week, particularly on GH issues/PRs.
Get master to main transition done this week
Discussion topics
Emmet
cacert integration w/ restic & rancher
https://github.com/vmware-tanzu/velero/issues/2675
Contributor Shoutouts
@a-mccarthy: add style guide file
@firethestars: Added Arm Multi-arch Make Logic
@ehashman: Fix missing quotes that are breaking page render
Status Updates
(ashish)
Community support last week
Fix for capturing version of CRD before version remap plugin:
PR 2683 (https://github.com/vmware-tanzu/velero/pull/2683)
Merged last week and is in this weeks 1.4.2 release
PR reviews:
VolumesnapshotProgress design proposal PR
merged last week. 🎉
Ordered Resource backup design proposal
Couple of passes at the
Fixed Issue 17 with PR 60 in the velero-plugin-for-csi repo
allows users of the CSI plugin to keep regular volumesnapshotclass separate from that used by Velero. Primarily to allow different DeletionPolicy.
(carlisia)
Community support
Blocker (BSL controller): https://github.com/vmware-tanzu/velero/pull/2674
(nrb)
v1.4.2 is out!
Includes fixes from CRD
Going to try to help catch up on community support this week, particularly on GH issues/PRs.
Get master to main transition done this week
Discussion topics
Emmet
cacert integration w/ restic & rancher
https://github.com/vmware-tanzu/velero/issues/2675
Contributor Shoutouts
@a-mccarthy: add style guide file
@firethestars: Added Arm Multi-arch Make Logic
@ehashman: Fix missing quotes that are breaking page render
- 7 participants
- 23 minutes
7 Jul 2020
July 7, 2020
Status Updates
nrb
Community support last week, still playing catch up on GH issues though
Review blitz yesterday, but did not get them all
End in sight for our internal build process work, hope to give more time to this open source work!
Restore hooks meeting yesterday, thanks everyone for attending!
carlisia
BSL controller PR: https://github.com/vmware-tanzu/velero/pull/2674
Worked on adding the BSL validation as webhooks; talked to @nrb and decided we won’t do it
Did and doing PR reviews
Finished reviewing the volume backup/restore progress design doc
Next: converting ServerStatusRequest to controller-runtime
ashish
Community support this week:
PR to fix a CRD backup regression in 1.4. PR Issue
Tested against "v1.16.9" "v1.17.0" "v1.18.4"
Will be included in the release for 1.4.1
Working on internal build setup. Should mostly be done, need a few finishing touches
PR to fix the file permissions on the manifests in the backup tarball.
0755 to 0644
PR 2685
Discussion topics
Phuong Demo of Resource ordering
nrb: master branch will be changing to main. We need to work on making sure docs redirects will work, but it looks like in-flight PRs will not be redirected unless GitHub introduces some sort of PR redirection feature. https://github.com/vmware-tanzu/velero/pull/2677 is tracking that work.
vitta: PR 2543
Contributor Shoutouts
@danielthrasher: updated acceptable values on cron schedule for day of the week from 0-7 to 0-6
@mauilion & team: Builder Image handling
@tbatard & team: Add linters
Everyone who attended the restore hooks meeting yesterday!
Status Updates
nrb
Community support last week, still playing catch up on GH issues though
Review blitz yesterday, but did not get them all
End in sight for our internal build process work, hope to give more time to this open source work!
Restore hooks meeting yesterday, thanks everyone for attending!
carlisia
BSL controller PR: https://github.com/vmware-tanzu/velero/pull/2674
Worked on adding the BSL validation as webhooks; talked to @nrb and decided we won’t do it
Did and doing PR reviews
Finished reviewing the volume backup/restore progress design doc
Next: converting ServerStatusRequest to controller-runtime
ashish
Community support this week:
PR to fix a CRD backup regression in 1.4. PR Issue
Tested against "v1.16.9" "v1.17.0" "v1.18.4"
Will be included in the release for 1.4.1
Working on internal build setup. Should mostly be done, need a few finishing touches
PR to fix the file permissions on the manifests in the backup tarball.
0755 to 0644
PR 2685
Discussion topics
Phuong Demo of Resource ordering
nrb: master branch will be changing to main. We need to work on making sure docs redirects will work, but it looks like in-flight PRs will not be redirected unless GitHub introduces some sort of PR redirection feature. https://github.com/vmware-tanzu/velero/pull/2677 is tracking that work.
vitta: PR 2543
Contributor Shoutouts
@danielthrasher: updated acceptable values on cron schedule for day of the week from 0-7 to 0-6
@mauilion & team: Builder Image handling
@tbatard & team: Add linters
Everyone who attended the restore hooks meeting yesterday!
- 6 participants
- 50 minutes
30 Jun 2020
June 30, 2020
Status Updates
carlisia
BSL controller to handle validation + update BSL status phase - PR ready for review (this changes the behavior of the Velero server): https://github.com/vmware-tanzu/velero/pull/2674/
WIP: add webhook validation to BSL create/update
PR reviews owed
ashish
on community support last week
worked on some internal build setup and testing of the build artifacts
Opened this issue last week Issue 2673
Currently working on some load testing for restic
nrb
Was out for the end of last week for family health issues, didn’t get everything I wanted done.
Reviews!
Updates on docs, getting Jason S up to speed for implementing new commands
On community support this week
Discussion topics
[nrb] Moving code from backup & restore controllers out into the pkg/{backup/restore} directories
Helps with kubebuilder/concurrency
[stephanie] Scheduling a backup and restore focus meeting Monday July 6th, more info to come here
Contributor shoutouts
Vishnu Itta for the volume snapshotter progress design doc!! Let’s get that merged this week (for real!)
Marc Campbell for the restore hooks design doc!
Phuong for the work on the ordering design doc!
Status Updates
carlisia
BSL controller to handle validation + update BSL status phase - PR ready for review (this changes the behavior of the Velero server): https://github.com/vmware-tanzu/velero/pull/2674/
WIP: add webhook validation to BSL create/update
PR reviews owed
ashish
on community support last week
worked on some internal build setup and testing of the build artifacts
Opened this issue last week Issue 2673
Currently working on some load testing for restic
nrb
Was out for the end of last week for family health issues, didn’t get everything I wanted done.
Reviews!
Updates on docs, getting Jason S up to speed for implementing new commands
On community support this week
Discussion topics
[nrb] Moving code from backup & restore controllers out into the pkg/{backup/restore} directories
Helps with kubebuilder/concurrency
[stephanie] Scheduling a backup and restore focus meeting Monday July 6th, more info to come here
Contributor shoutouts
Vishnu Itta for the volume snapshotter progress design doc!! Let’s get that merged this week (for real!)
Marc Campbell for the restore hooks design doc!
Phuong for the work on the ordering design doc!
- 8 participants
- 40 minutes
23 Jun 2020
June 23, 2020
Status Updates
nrb
Lots of reviews, but sorry if I didn’t get to yours
Lots of internal work too
carlisia
On community support this week
kubebuilder PR still getting reviews: https://github.com/vmware-tanzu/velero/pull/2561
PR reviews
Adding tests to: Add BSL controller - https://github.com/vmware-tanzu/velero/pull/2617
ashish is out
Worked on backing up volumes with restic by default
Discussion Topics
nrb Restore hooks - let’s have a seperate call for this to address some concerns about using init containers.
Main use case seems to be quiescing/unquiescing databases
Will need to use some sort of tool to send this to a backup PV, and a tool to take it from the backup PV to a runtime PV
nrb/carlisia - Public/private APIs
Let us know what you rely on!
This will be phased, not all at once
In 2.0, the current Go client for Velero will be replaced by a controller-runtime client. The informers/listers will be replaced by a client manager.
Vishnu (Enable restic by default - https://github.com/vmware-tanzu/velero/pull/2611) and (Progress on backup/restore by plugins - https://github.com/vmware-tanzu/velero/pull/2543)
Contributor Shoutouts
@adamrushuk: Update basic-install and release-instructions documentation
@JessicaGreben: add a supported provider: Storj object storage
@adamrushuk: Update basic-install.md: Add windows cli installation option via chocolatey
@tbatard: Update Jekyll to 4.1.0
Status Updates
nrb
Lots of reviews, but sorry if I didn’t get to yours
Lots of internal work too
carlisia
On community support this week
kubebuilder PR still getting reviews: https://github.com/vmware-tanzu/velero/pull/2561
PR reviews
Adding tests to: Add BSL controller - https://github.com/vmware-tanzu/velero/pull/2617
ashish is out
Worked on backing up volumes with restic by default
Discussion Topics
nrb Restore hooks - let’s have a seperate call for this to address some concerns about using init containers.
Main use case seems to be quiescing/unquiescing databases
Will need to use some sort of tool to send this to a backup PV, and a tool to take it from the backup PV to a runtime PV
nrb/carlisia - Public/private APIs
Let us know what you rely on!
This will be phased, not all at once
In 2.0, the current Go client for Velero will be replaced by a controller-runtime client. The informers/listers will be replaced by a client manager.
Vishnu (Enable restic by default - https://github.com/vmware-tanzu/velero/pull/2611) and (Progress on backup/restore by plugins - https://github.com/vmware-tanzu/velero/pull/2543)
Contributor Shoutouts
@adamrushuk: Update basic-install and release-instructions documentation
@JessicaGreben: add a supported provider: Storj object storage
@adamrushuk: Update basic-install.md: Add windows cli installation option via chocolatey
@tbatard: Update Jekyll to 4.1.0
- 10 participants
- 54 minutes
16 Jun 2020
June 16, 2020
Status Updates
steve:
community support & reviews
nrb:
Mostly focused on internal build tooling last week
Intending to catch up on community review this week
ashish:
Focussing on internal build setup
Finishing up on PR 2611
community support this week
carlisia
wip: add kubebuilder/controller-runtime - https://github.com/vmware-tanzu/velero/pull/2561
wip: add BSL controller w/ controller-runtime - https://github.com/vmware-tanzu/velero/pull/2617
Discussion Topics
[nrb] Deletion plugins?
Would we want to take this opportunity to also do pre-backup plugins to? I don’t have a use case right now, more asking because it’s an opportunity to do so
Next steps: Design doc
Use labels, not configmap
[nrb] Restore hooks design should be close to merging. Maintainers, please take another pass!
[duffie] Linters for Velero, possibly also image handling
[Vishnu] Restic snapshot specific differential size and https://github.com/vmware-tanzu/velero/pull/2543
Problem: How much object storage did a given backup consume?
Is there a way to know differential size from the previous snapshot?
This would help to prune object storage
Restic does deduplication; if I have 10 backups and I delete backup 6, how much do I get back?
Could we add a field to progress to tell report this?
Restic’s format design documents: https://restic.readthedocs.io/en/latest/100_references.html#design
[tony] Update to Jekyll 4.1.0, https://github.com/vmware-tanzu/velero/pull/2625
Status Updates
steve:
community support & reviews
nrb:
Mostly focused on internal build tooling last week
Intending to catch up on community review this week
ashish:
Focussing on internal build setup
Finishing up on PR 2611
community support this week
carlisia
wip: add kubebuilder/controller-runtime - https://github.com/vmware-tanzu/velero/pull/2561
wip: add BSL controller w/ controller-runtime - https://github.com/vmware-tanzu/velero/pull/2617
Discussion Topics
[nrb] Deletion plugins?
Would we want to take this opportunity to also do pre-backup plugins to? I don’t have a use case right now, more asking because it’s an opportunity to do so
Next steps: Design doc
Use labels, not configmap
[nrb] Restore hooks design should be close to merging. Maintainers, please take another pass!
[duffie] Linters for Velero, possibly also image handling
[Vishnu] Restic snapshot specific differential size and https://github.com/vmware-tanzu/velero/pull/2543
Problem: How much object storage did a given backup consume?
Is there a way to know differential size from the previous snapshot?
This would help to prune object storage
Restic does deduplication; if I have 10 backups and I delete backup 6, how much do I get back?
Could we add a field to progress to tell report this?
Restic’s format design documents: https://restic.readthedocs.io/en/latest/100_references.html#design
[tony] Update to Jekyll 4.1.0, https://github.com/vmware-tanzu/velero/pull/2625
- 13 participants
- 1:03 hours
9 Jun 2020
June 9, 2020
Status Updates
ashish:
Allowing all pod volumes to be backed up using restic by default, without requiring annotation on the pod. PR 2611
Exclude: host path volumes, volumes mounting default service account tokens and volumes excluded through “exclude” annotation
Review and discussion of the volume snapshot progress proposal
Focussing on internal build setup.
steve
on community support this week
updating & testing a community-contributed PR to support aad-pod-identity for Azure (https://github.com/vmware-tanzu/velero/pull/2602, https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/pull/51) (thanks @gitirabassi for the code!)
log a warning instead of an error if an additional item doesn’t exist during backup (https://github.com/vmware-tanzu/velero/pull/2595)
nrb
More reviewing on reporting progress on volume snapshotting
https://github.com/vmware-tanzu/velero/pull/2610
Plan to follow up on KubeBuilder PR review this week
Also want to follow up on the restore hooks PR
Can other maintainers look at the tagging scripts PR?
Been doing work on some internal projects the last week
carlisia
BSL resource conversion to kubebuilder PR is ready for review: https://github.com/vmware-tanzu/velero/pull/2561
Wrapping up new BSL controller, just need to add tests
Discussion Topics
Vishnu:
Thanks to maintainers for review comments on this PR https://github.com/vmware-tanzu/velero/pull/2543. Would like to discuss on next step and pending items from me related to this PR.
steve: finalize naming & CR lifecycle (who creates?)
Phuong:
We would like to propose a change to allow caller to specify an order to backup items of the same type. More generic than https://github.com/vmware-tanzu/velero/issues/1917 which only for PVC.
In backup spec, have a map of type to strings of specific names (e.g. PVC - names of PVCs)
Next step - submit a design doc PR based on https://github.com/vmware-tanzu/velero/blob/master/design/_template.md
steve: restore hooks design PR
@marccampbell will you be able to address open comments/questions?
Contributor Shoutouts
@cblecker - when creating a backup from a schedule, allow a default name (https://github.com/vmware-tanzu/velero/pull/2569)
@vitta - Thanks for being super responsive on PR 2543!
Status Updates
ashish:
Allowing all pod volumes to be backed up using restic by default, without requiring annotation on the pod. PR 2611
Exclude: host path volumes, volumes mounting default service account tokens and volumes excluded through “exclude” annotation
Review and discussion of the volume snapshot progress proposal
Focussing on internal build setup.
steve
on community support this week
updating & testing a community-contributed PR to support aad-pod-identity for Azure (https://github.com/vmware-tanzu/velero/pull/2602, https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/pull/51) (thanks @gitirabassi for the code!)
log a warning instead of an error if an additional item doesn’t exist during backup (https://github.com/vmware-tanzu/velero/pull/2595)
nrb
More reviewing on reporting progress on volume snapshotting
https://github.com/vmware-tanzu/velero/pull/2610
Plan to follow up on KubeBuilder PR review this week
Also want to follow up on the restore hooks PR
Can other maintainers look at the tagging scripts PR?
Been doing work on some internal projects the last week
carlisia
BSL resource conversion to kubebuilder PR is ready for review: https://github.com/vmware-tanzu/velero/pull/2561
Wrapping up new BSL controller, just need to add tests
Discussion Topics
Vishnu:
Thanks to maintainers for review comments on this PR https://github.com/vmware-tanzu/velero/pull/2543. Would like to discuss on next step and pending items from me related to this PR.
steve: finalize naming & CR lifecycle (who creates?)
Phuong:
We would like to propose a change to allow caller to specify an order to backup items of the same type. More generic than https://github.com/vmware-tanzu/velero/issues/1917 which only for PVC.
In backup spec, have a map of type to strings of specific names (e.g. PVC - names of PVCs)
Next step - submit a design doc PR based on https://github.com/vmware-tanzu/velero/blob/master/design/_template.md
steve: restore hooks design PR
@marccampbell will you be able to address open comments/questions?
Contributor Shoutouts
@cblecker - when creating a backup from a schedule, allow a default name (https://github.com/vmware-tanzu/velero/pull/2569)
@vitta - Thanks for being super responsive on PR 2543!
- 11 participants
- 36 minutes
2 Jun 2020
June 2, 2020
Status Updates
michael
Discuss the community roles in Velero Governance
nrb
Added a draft PR for automating our release process
Reviewed some community PRs, specifically reporting progress on volume snapshotting
Working on some internal Velero-related projects
On community support this week
ashish
Moving all our plugin repos to use go modules
Decommission Travis and switch to using github actions for our CI and release
Have you tried the new CSI volume snapshotting feature in 1.4. This blog post should help you get started.
Reviewed some PRs on the restore hook and the volume snapshot progress reporting proposal added comments.
stephanie
discuss up-vote and down-vote on feature requests, and announcement of ‘go-live’ for that capability later today
carlisia
community support and catching up on PRs last week
Addressing code reviews for https://github.com/vmware-tanzu/velero/pull/2561 (Convert manifests + BSL api client to kubebuilder)
jonas
Introducing Abbie, Tony, Duffie, Matyas, and Mike
Discussion Topics
michael
Discuss the community roles in Velero Governance
Action Item - Carlisia to prep a PR for a reviewer role. Let her know if you’re interested in being a reviewer!
carlisia
Modernize the Velero code base/tests
abbie
Documentation style guide
kevin leffew
Kubernetes backup to the decentralized cloud via Velero and StorJ
https://twitter.com/kleffew94/status/1262382842086141953
https://forum.storj.io/t/kubernetes-backup-via-velero/6879
Status Updates
michael
Discuss the community roles in Velero Governance
nrb
Added a draft PR for automating our release process
Reviewed some community PRs, specifically reporting progress on volume snapshotting
Working on some internal Velero-related projects
On community support this week
ashish
Moving all our plugin repos to use go modules
Decommission Travis and switch to using github actions for our CI and release
Have you tried the new CSI volume snapshotting feature in 1.4. This blog post should help you get started.
Reviewed some PRs on the restore hook and the volume snapshot progress reporting proposal added comments.
stephanie
discuss up-vote and down-vote on feature requests, and announcement of ‘go-live’ for that capability later today
carlisia
community support and catching up on PRs last week
Addressing code reviews for https://github.com/vmware-tanzu/velero/pull/2561 (Convert manifests + BSL api client to kubebuilder)
jonas
Introducing Abbie, Tony, Duffie, Matyas, and Mike
Discussion Topics
michael
Discuss the community roles in Velero Governance
Action Item - Carlisia to prep a PR for a reviewer role. Let her know if you’re interested in being a reviewer!
carlisia
Modernize the Velero code base/tests
abbie
Documentation style guide
kevin leffew
Kubernetes backup to the decentralized cloud via Velero and StorJ
https://twitter.com/kleffew94/status/1262382842086141953
https://forum.storj.io/t/kubernetes-backup-via-velero/6879
- 9 participants
- 49 minutes
26 May 2020
May 26, 2020
Status Updates
steve
finalizing things for v1.4; one bug-fix (thanks @ashish for catching) - https://github.com/vmware-tanzu/velero/pull/2560
bug in using custom CA cert for restic restores: https://github.com/vmware-tanzu/velero/issues/2562
spent some time brainstorming on multi-tenancy
ashish
1.4 testing using cluster-api as the workload to backup
Working on a blog post about get started with using CSI snapshotting. PR#2553.
nrb
1.4 testing last week
Was off Thursday/Friday
Planning to do the v1.4.0 release today, pending bug fixes
carlisia
on community support this week
PR out for initial kubebuilder migration: https://github.com/vmware-tanzu/velero/pull/2561
short video on how I use Goland for PR reviews: https://www.loom.com/share/a60b72518f5c453b97e797f6a5ba39f8
need to do lots of PR reviews
Discussion Topics
stephanie
we are looking to add an up-vote/down-vote capability to feature requests in the velero backlog as a way to more easily quantify what features the community is more interested in as a whole
we’ve filed the issue here: https://github.com/vmware-tanzu/velero/issues/2574
it could look something like this
we will be working to get this implemented and open this as an opportunity to provide quick feedback on features that you are most interested in - let us know what you think
Status Updates
steve
finalizing things for v1.4; one bug-fix (thanks @ashish for catching) - https://github.com/vmware-tanzu/velero/pull/2560
bug in using custom CA cert for restic restores: https://github.com/vmware-tanzu/velero/issues/2562
spent some time brainstorming on multi-tenancy
ashish
1.4 testing using cluster-api as the workload to backup
Working on a blog post about get started with using CSI snapshotting. PR#2553.
nrb
1.4 testing last week
Was off Thursday/Friday
Planning to do the v1.4.0 release today, pending bug fixes
carlisia
on community support this week
PR out for initial kubebuilder migration: https://github.com/vmware-tanzu/velero/pull/2561
short video on how I use Goland for PR reviews: https://www.loom.com/share/a60b72518f5c453b97e797f6a5ba39f8
need to do lots of PR reviews
Discussion Topics
stephanie
we are looking to add an up-vote/down-vote capability to feature requests in the velero backlog as a way to more easily quantify what features the community is more interested in as a whole
we’ve filed the issue here: https://github.com/vmware-tanzu/velero/issues/2574
it could look something like this
we will be working to get this implemented and open this as an opportunity to provide quick feedback on features that you are most interested in - let us know what you think
- 7 participants
- 21 minutes
19 May 2020
May 19, 2020
Status Updates
[ashish]
Community support last week.
Quite a few restic related issues/ mostly questions.
Migrating our plugin repos to use go modules:
velero-plugin-for-microsoft-azure
velero-plugin-for-gcp
velero-plugin-for–aws
Moving our CI from travis to Github Actions:
Started with velero-plugin-for-microsoft-azure. Once this is merged other plugins and velero core should be straight forward.
Testing 1.4 release
Testing CSI in Azure
More testing this week.
Started on blog post for CSI velero integration. PR this week.
Update our helm charts to support Velero feature flags
Started working on a Octant plugin for velero over the weekend.
Not fully functional just- scaffolding.
more to com
PRs, ideas and suggestions welcome!
[carlisia]
k8s Slack #velero channel now has a @velero-maintainers user group
Migration of BSL CRD to kubebuilder:
All CRD manifests, and generation of manifests ported to kubebuilder (simpler to do all of them imo, but optional)
BSL api client and code generation ported to kubebuilder
Current status:
basic CRUD works on BSL and backup
need to fix failing tests
steve
on community support this week
getting ready for 1.4 release
1.5+ planning
stephanie
1.5 release planning
velero roadmap planning
nrb
1.4 testing
Blog post for v1.4.0
Some reviews
Discussion Topics
vitta
Would like to have a high level walk through of this PR https://github.com/vmware-tanzu/velero/pull/2543 and the procedure of next steps
cblecker
Non-existent items should not cause backup to be PartiallyFailed
areed / marccampbell
Discussion of RestoreHooks proposal @ https://github.com/vmware-tanzu/velero/pull/2465
Governance Model review
I may be gone to at another meeting by the time this comes up. folks can view the PR and comment.
PR at https://github.com/vmware-tanzu/velero/pull/2541
Status Updates
[ashish]
Community support last week.
Quite a few restic related issues/ mostly questions.
Migrating our plugin repos to use go modules:
velero-plugin-for-microsoft-azure
velero-plugin-for-gcp
velero-plugin-for–aws
Moving our CI from travis to Github Actions:
Started with velero-plugin-for-microsoft-azure. Once this is merged other plugins and velero core should be straight forward.
Testing 1.4 release
Testing CSI in Azure
More testing this week.
Started on blog post for CSI velero integration. PR this week.
Update our helm charts to support Velero feature flags
Started working on a Octant plugin for velero over the weekend.
Not fully functional just- scaffolding.
more to com
PRs, ideas and suggestions welcome!
[carlisia]
k8s Slack #velero channel now has a @velero-maintainers user group
Migration of BSL CRD to kubebuilder:
All CRD manifests, and generation of manifests ported to kubebuilder (simpler to do all of them imo, but optional)
BSL api client and code generation ported to kubebuilder
Current status:
basic CRUD works on BSL and backup
need to fix failing tests
steve
on community support this week
getting ready for 1.4 release
1.5+ planning
stephanie
1.5 release planning
velero roadmap planning
nrb
1.4 testing
Blog post for v1.4.0
Some reviews
Discussion Topics
vitta
Would like to have a high level walk through of this PR https://github.com/vmware-tanzu/velero/pull/2543 and the procedure of next steps
cblecker
Non-existent items should not cause backup to be PartiallyFailed
areed / marccampbell
Discussion of RestoreHooks proposal @ https://github.com/vmware-tanzu/velero/pull/2465
Governance Model review
I may be gone to at another meeting by the time this comes up. folks can view the PR and comment.
PR at https://github.com/vmware-tanzu/velero/pull/2541
- 11 participants
- 52 minutes
12 May 2020
May 12, 2020
The v1.4.0-beta.1 release is out! Please try it out in your non-critical environments and give us feedback.
Status Updates
[carlisia]
Lotsa PR reviews
Moving our BSL CRD to a kubebuilder CRD + controller (Vince Prignano has been offering guidance)
Next:
move BSL logic out of the controller
add handlers to create/update
[ashish]
You can now supply feature flags to Velero using the Velero CLI. Read more about it here
Getting CSI integration with Velero ready to ship. Take a look at the CSI page on our website
Backup syncing: include CSI API objects
Backup deletion: cleaning up storage snapshots on backup deletion.
Set up Github actions to build and push container images for velero-csi-plugin. Hoping to do this for all our repos.
steve
finalizing things for v1.4.0-beta.1
starting to look at 1.5 planning
nrb
Writing the v1.4 blog post
Catching up reviews that I pushed out in order to get the beta out; particularly want to get a look at the restore hooks document this week
Will be testing v1.4.0-beta.1 against a few different platforms and versions of Kubernetes to verify the overall package
Some docs PRs to get in for the GA release
Discussion Topics
stephanie
Review community feature reqeust for backing up restic volumes by default
Detailed product requirements are available and we will be providing a review of the three main use cases during this meeting.
You can find the product requirements here: https://drive.google.com/file/d/1i5IQfuOtsHhGuYiiTrCZ6zjVhtKWLRqV/view?usp=sharing
And, sample annotations to illustrate the main product use cases are here (these are also included in the UI/UX section of the requirements doc): https://gist.github.com/skriss/dccc494aea67a47473e184c18aac9af6
vitta
Backup/Restore progress for operations performed by velero plugins
In similar lines, if plugin have their own CRs (similar to PVB, PVR), how to get those CRs backed up from source cluster and get them synced at destination cluster regularly?
Finalizers on Backup CRs to protect data from getting deleted
The v1.4.0-beta.1 release is out! Please try it out in your non-critical environments and give us feedback.
Status Updates
[carlisia]
Lotsa PR reviews
Moving our BSL CRD to a kubebuilder CRD + controller (Vince Prignano has been offering guidance)
Next:
move BSL logic out of the controller
add handlers to create/update
[ashish]
You can now supply feature flags to Velero using the Velero CLI. Read more about it here
Getting CSI integration with Velero ready to ship. Take a look at the CSI page on our website
Backup syncing: include CSI API objects
Backup deletion: cleaning up storage snapshots on backup deletion.
Set up Github actions to build and push container images for velero-csi-plugin. Hoping to do this for all our repos.
steve
finalizing things for v1.4.0-beta.1
starting to look at 1.5 planning
nrb
Writing the v1.4 blog post
Catching up reviews that I pushed out in order to get the beta out; particularly want to get a look at the restore hooks document this week
Will be testing v1.4.0-beta.1 against a few different platforms and versions of Kubernetes to verify the overall package
Some docs PRs to get in for the GA release
Discussion Topics
stephanie
Review community feature reqeust for backing up restic volumes by default
Detailed product requirements are available and we will be providing a review of the three main use cases during this meeting.
You can find the product requirements here: https://drive.google.com/file/d/1i5IQfuOtsHhGuYiiTrCZ6zjVhtKWLRqV/view?usp=sharing
And, sample annotations to illustrate the main product use cases are here (these are also included in the UI/UX section of the requirements doc): https://gist.github.com/skriss/dccc494aea67a47473e184c18aac9af6
vitta
Backup/Restore progress for operations performed by velero plugins
In similar lines, if plugin have their own CRs (similar to PVB, PVR), how to get those CRs backed up from source cluster and get them synced at destination cluster regularly?
Finalizers on Backup CRs to protect data from getting deleted
- 12 participants
- 32 minutes
5 May 2020
May 5, 2020
Status Updates
steve
updated container base images to ubuntu:focal
fixed a CRD restore issue related to unstructured conversion functions
fixed an issue with unresolvable includes/excludes
updated plugin server code to ignore unrecognized flags, and added special handling for --features flag in Velero
reviews
carlisia
continuing with the new BSL: controller (https://github.com/vmware-tanzu/velero/pull/2490); New: added ValidationFrequency / LastValidationTime, adding validation on BSL create/update events, using status/phase to validate backups
ashish
CSI work:
Deleting CSI objects and cleaningup storage snapshots on backup deletion
Syncing in CSI objects during backup sync-in. Mainly to ensure they are deleted on backup deletion/ expiry.
Testing.
Fixup in persisting backup here
PR reviews
Investigate why we are unable to install ca-certificate package on ARM builds
nrb
Almost done with CRD back up work in https://github.com/vmware-tanzu/velero/pull/2478. Want to grab a few more samples today if I can
specifically want to get prometheus objects as mentioned in https://github.com/vmware-tanzu/velero/issues/2370
Reviewing CSI work, need to finish up the describe output too
On community support this week
Will be handling the v1.4 release
Will be on Crossplane’s TBS YouTube series Thursday with Carlisia
Discussion Topics
[carlisia]
Want to add a Tilt setup for rapid development for 1.5. Here’s an example from the cluster api team: https://cluster-api.sigs.k8s.io/developer/tilt.html
Contributor Shoutouts
@jaygridley - [Azure] add support for static storage account keys with restic (https://github.com/vmware-tanzu/velero/pull/2455)
@jones-paul1525 - [AWS] update AWS SDK to enable some new auth methods (https://github.com/vmware-tanzu/velero-plugin-for-aws/pull/41)
@dunglas - [Helm chart] Add support for secret env vars (https://github.com/vmware-tanzu/helm-charts/pull/99)
@austinbv - [Helm chart] Add documentation for having separate providers (https://github.com/vmware-tanzu/helm-charts/pull/100)
@hasheddan [Velero core] Reviewing #2490 for CRD handling
Status Updates
steve
updated container base images to ubuntu:focal
fixed a CRD restore issue related to unstructured conversion functions
fixed an issue with unresolvable includes/excludes
updated plugin server code to ignore unrecognized flags, and added special handling for --features flag in Velero
reviews
carlisia
continuing with the new BSL: controller (https://github.com/vmware-tanzu/velero/pull/2490); New: added ValidationFrequency / LastValidationTime, adding validation on BSL create/update events, using status/phase to validate backups
ashish
CSI work:
Deleting CSI objects and cleaningup storage snapshots on backup deletion
Syncing in CSI objects during backup sync-in. Mainly to ensure they are deleted on backup deletion/ expiry.
Testing.
Fixup in persisting backup here
PR reviews
Investigate why we are unable to install ca-certificate package on ARM builds
nrb
Almost done with CRD back up work in https://github.com/vmware-tanzu/velero/pull/2478. Want to grab a few more samples today if I can
specifically want to get prometheus objects as mentioned in https://github.com/vmware-tanzu/velero/issues/2370
Reviewing CSI work, need to finish up the describe output too
On community support this week
Will be handling the v1.4 release
Will be on Crossplane’s TBS YouTube series Thursday with Carlisia
Discussion Topics
[carlisia]
Want to add a Tilt setup for rapid development for 1.5. Here’s an example from the cluster api team: https://cluster-api.sigs.k8s.io/developer/tilt.html
Contributor Shoutouts
@jaygridley - [Azure] add support for static storage account keys with restic (https://github.com/vmware-tanzu/velero/pull/2455)
@jones-paul1525 - [AWS] update AWS SDK to enable some new auth methods (https://github.com/vmware-tanzu/velero-plugin-for-aws/pull/41)
@dunglas - [Helm chart] Add support for secret env vars (https://github.com/vmware-tanzu/helm-charts/pull/99)
@austinbv - [Helm chart] Add documentation for having separate providers (https://github.com/vmware-tanzu/helm-charts/pull/100)
@hasheddan [Velero core] Reviewing #2490 for CRD handling
- 6 participants
- 29 minutes
28 Apr 2020
April 28, 2020
Status Updates
steve
finalized backup progress PR
fixed issue with unresolvable includes/excludes
reviews
[stephanie] - update on where we are currently with the velero 1.4 release
https://github.com/vmware-tanzu/velero/milestone/23
https://github.com/vmware-tanzu/velero-plugin-for-csi/milestone/2
[carlisia]
Adding tests to the new BSL controller (https://github.com/vmware-tanzu/velero/issues/1967)
Recorded a tutoria to show the build/run/test workflow for Velero: https://www.youtube.com/watch?v=Aiz9M4aVFdA
On community support this week
[nrb]
Was out sick late last week, not a ton of updates from me. Spent yesterday replying to things I’d missed
Need to get 2373 reviewed and merged this week so we can get CRD restore fixes started
Will also make a test of using the v1beta1 client approach
Will be on Brett Fisher’s YouTube channel this Thursday at 1pm ET - https://youtu.be/pRgap975b5E
[ashish]
Community support this past week
Reviewing some PRs and getting CSI closer to the beta ship
Discussion Topics
[steve] demo of backup progress
[stephanie] - can we document how to execute backup and recovery in an airgapped environment?
[nrb] - this is partially documented at https://velero.io/docs/v1.3.2/on-premises/#on-premises-environments, though it’s incomplete since there’s a lot of possible options in airgapped/on-prem environments
[from timh] Perhaps we can reuse some of the docs over at Sonobuoy
[carlisia] Question: should we (RE)consider not crashlooping for when there’s no BSL? This issue was posed here: https://github.com/vmware-tanzu/velero/issues/1967#issuecomment-620208525
[marccampbell] Custom Restore Hooks proposal: https://github.com/vmware-tanzu/velero/pull/2465
Contributor Shoutouts
@raghavendrabhat - clarify wording in velero restore describe for included namespaces (https://github.com/vmware-tanzu/velero/pull/2449)
@mynktl - skip VolumeSnapshotLocation validation for backups when --snapshot-volumes=false (https://github.com/vmware-tanzu/velero/pull/2450)
Status Updates
steve
finalized backup progress PR
fixed issue with unresolvable includes/excludes
reviews
[stephanie] - update on where we are currently with the velero 1.4 release
https://github.com/vmware-tanzu/velero/milestone/23
https://github.com/vmware-tanzu/velero-plugin-for-csi/milestone/2
[carlisia]
Adding tests to the new BSL controller (https://github.com/vmware-tanzu/velero/issues/1967)
Recorded a tutoria to show the build/run/test workflow for Velero: https://www.youtube.com/watch?v=Aiz9M4aVFdA
On community support this week
[nrb]
Was out sick late last week, not a ton of updates from me. Spent yesterday replying to things I’d missed
Need to get 2373 reviewed and merged this week so we can get CRD restore fixes started
Will also make a test of using the v1beta1 client approach
Will be on Brett Fisher’s YouTube channel this Thursday at 1pm ET - https://youtu.be/pRgap975b5E
[ashish]
Community support this past week
Reviewing some PRs and getting CSI closer to the beta ship
Discussion Topics
[steve] demo of backup progress
[stephanie] - can we document how to execute backup and recovery in an airgapped environment?
[nrb] - this is partially documented at https://velero.io/docs/v1.3.2/on-premises/#on-premises-environments, though it’s incomplete since there’s a lot of possible options in airgapped/on-prem environments
[from timh] Perhaps we can reuse some of the docs over at Sonobuoy
[carlisia] Question: should we (RE)consider not crashlooping for when there’s no BSL? This issue was posed here: https://github.com/vmware-tanzu/velero/issues/1967#issuecomment-620208525
[marccampbell] Custom Restore Hooks proposal: https://github.com/vmware-tanzu/velero/pull/2465
Contributor Shoutouts
@raghavendrabhat - clarify wording in velero restore describe for included namespaces (https://github.com/vmware-tanzu/velero/pull/2449)
@mynktl - skip VolumeSnapshotLocation validation for backups when --snapshot-volumes=false (https://github.com/vmware-tanzu/velero/pull/2450)
- 10 participants
- 59 minutes
21 Apr 2020
April 21, 2020
Status Updates
[stephanie] Introducing Velero’s new PM!
[nrb]
Outlined our approach to versioning the backup file format https://velero.io/docs/master/output-file-format/#output-file-format-versioning (PR)
Finished up PR 2323, Velero master will now upload CSI snapshot information for use by client-side commands (such as velero backup --details)
Started in on 2448, which will add the client-side info.
Investigated issue 2383, where CRDs are reporting errors about fields being duplicated in versions yesterday.
This is a result of the “fix” for v1beta1 to v1 CRD restores in Velero v1.3.0. More in discussion time.
Will be on the Brett Fisher show next week talking about Velero
[steve]
was on community support
CSI reviews & testing
draft PR & testing for backup progress
[carlisia]
wip: bsl/vsl/velero readiness: https://github.com/vmware-tanzu/velero/issues/1967
presented Velero to Brazilian meetup
[ashish]
CSI bug fixes and testing
Community support this week
Sharing a link to the survey we sent out 3w ago. If you’ve not had a chance to look at this, please do. Survey is available here.
Discussion Topics
[marccampbell] Willingness / timeline for custom restore hooks? https://github.com/vmware-tanzu/velero/issues/2116
[mayank] patch release for plugin velero.io/change-pvc-node-selector
I had a discussion about this on slack with Steve. It won’t be a good idea to have a patch release for the above plugin since it’s not a bug-fix.
[nrb] Possible fixes for 2383
if the object is truly a v1beta1 CRD, re-fetch it using a v1beta1 client in the plugin
Use the API server’s internal representation & conversion functions to get the correct version on restore. If we keep this in the backup tarball, this has ramifications for the backup file format
Essentially 3 versions of CRDs in Go code:
v1beta1.CustomResourceDefinition
v1.CustomResourceDefinition
apiextensions.CustomResourceDefintion - this one is internal to the API server, and is version-independent.
Merge #2373 and try v1 CRD, if it fails, try applying v1beta1
Action items:
Get #2373 merged
Try 1 & 2 for v1.4
investigate 2 for long term implementation. Problem with it is that it may trigger a Velero v2 as the internal CRD representation isn’t directly restorable
To prevent this kind of thing in the future: document regression test scenarios, stand up e2e kind tests, and begin automating.
[carlisia] - Tilt
Demo of running Tilt with Velero
Walkthrough of the setup
Contributor Shoutouts
@hasheddan for pointing out the CRD conversion functions!
@TomaszKlosinski for providing the elasticsearch & kibana CRD issues
Helm
@tareqhs - added quotes to extraEnvVars to support special chars
@yurinnick:
for making our chart Helm 3 compatible!
allowing multiple backup/snapshot providers
fixed timeout chart-testing parameter in the chart
prioritize location provider over global provider
@ahmadhajali - enable specifying docker image by digest in addition to tag
@limed fixing toYaml generation
Status Updates
[stephanie] Introducing Velero’s new PM!
[nrb]
Outlined our approach to versioning the backup file format https://velero.io/docs/master/output-file-format/#output-file-format-versioning (PR)
Finished up PR 2323, Velero master will now upload CSI snapshot information for use by client-side commands (such as velero backup --details)
Started in on 2448, which will add the client-side info.
Investigated issue 2383, where CRDs are reporting errors about fields being duplicated in versions yesterday.
This is a result of the “fix” for v1beta1 to v1 CRD restores in Velero v1.3.0. More in discussion time.
Will be on the Brett Fisher show next week talking about Velero
[steve]
was on community support
CSI reviews & testing
draft PR & testing for backup progress
[carlisia]
wip: bsl/vsl/velero readiness: https://github.com/vmware-tanzu/velero/issues/1967
presented Velero to Brazilian meetup
[ashish]
CSI bug fixes and testing
Community support this week
Sharing a link to the survey we sent out 3w ago. If you’ve not had a chance to look at this, please do. Survey is available here.
Discussion Topics
[marccampbell] Willingness / timeline for custom restore hooks? https://github.com/vmware-tanzu/velero/issues/2116
[mayank] patch release for plugin velero.io/change-pvc-node-selector
I had a discussion about this on slack with Steve. It won’t be a good idea to have a patch release for the above plugin since it’s not a bug-fix.
[nrb] Possible fixes for 2383
if the object is truly a v1beta1 CRD, re-fetch it using a v1beta1 client in the plugin
Use the API server’s internal representation & conversion functions to get the correct version on restore. If we keep this in the backup tarball, this has ramifications for the backup file format
Essentially 3 versions of CRDs in Go code:
v1beta1.CustomResourceDefinition
v1.CustomResourceDefinition
apiextensions.CustomResourceDefintion - this one is internal to the API server, and is version-independent.
Merge #2373 and try v1 CRD, if it fails, try applying v1beta1
Action items:
Get #2373 merged
Try 1 & 2 for v1.4
investigate 2 for long term implementation. Problem with it is that it may trigger a Velero v2 as the internal CRD representation isn’t directly restorable
To prevent this kind of thing in the future: document regression test scenarios, stand up e2e kind tests, and begin automating.
[carlisia] - Tilt
Demo of running Tilt with Velero
Walkthrough of the setup
Contributor Shoutouts
@hasheddan for pointing out the CRD conversion functions!
@TomaszKlosinski for providing the elasticsearch & kibana CRD issues
Helm
@tareqhs - added quotes to extraEnvVars to support special chars
@yurinnick:
for making our chart Helm 3 compatible!
allowing multiple backup/snapshot providers
fixed timeout chart-testing parameter in the chart
prioritize location provider over global provider
@ahmadhajali - enable specifying docker image by digest in addition to tag
@limed fixing toYaml generation
- 7 participants
- 48 minutes
14 Apr 2020
April 14,2020
Status Updates
[carlisia]
reviewing + testing of Helm PRs
Broke down new install CLI work into issues, see epic: https://github.com/vmware-tanzu/velero/issues/2419
wip: bsl/vsl/velero readiness: https://github.com/vmware-tanzu/velero/issues/1967
presenting/demo’ing Velero at a CNCF (online) meetup: https://www.meetup.com/Cloud-Native-Talks-Brazil/events/269858787/
[steve]
community support this week
lotta PR reviews
did some testing with WIP CSI plugins, reported a couple bugs
bunch of cleanup on the https://github.com/vmware-tanzu/velero-plugin-example repo, hoping to apply similar changes to the AWS/Azure/GCP ones too
continued on backup progress idea
[nrb]
CSI upload PR should be done & ready for review now
Tackling some CRD issues this week, mainly https://github.com/vmware-tanzu/velero/issues/2383 & https://github.com/vmware-tanzu/velero/issues/2370
Started working on https://github.com/vmware-tanzu/velero/issues/1970, but ran into https://github.com/vmware-tanzu/velero/issues/2429
Discussion Topics
[nrb] When do we increment the backup format version? Refer to this comment
PR that added backup version: https://github.com/vmware-tanzu/velero/pull/1117
AI - Nolan to document when we increment backup format versions (most likely following semantic versioning)
Current PR changes are backwards compatible, so they’re a patch version in semantic versioning parlance
[dymurray/sseago] Race condition with AdditionalItmes https://github.com/vmware-tanzu/velero/issues/1350
We haven’t seen others asking for this feature, but willing to let the RH team lead with a design doc on this
[nrb] I see this as a generically useful tool, +1 to RH taking the lead here
[dymurray/sseago] Restoring PVs that already exist (related https://github.com/vmware-tanzu/velero/issues/2170)
[nrb] Is this a specific case of https://github.com/vmware-tanzu/velero/issues/1977?
data-only restores https://github.com/vmware-tanzu/velero/issues/504
Could the PV rename mechanism work here? I was thinking of the StorageClass remapping
[carlisia] should “liveness” mean “ready”, and should we use a liveness check to check for readiness: https://github.com/vmware-tanzu/velero/pull/2382
Kubernetes definitions of readiness/liveness - https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
[marccampbell] Willingness / timeline for custom restore hooks? https://github.com/vmware-tanzu/velero/issues/2116
[mayank] patch release for plugin velero.io/change-pvc-node-selector
Contributor Shoutouts
@duyanghao - docs on community project to automatically annotate all volumes for restic backup
Status Updates
[carlisia]
reviewing + testing of Helm PRs
Broke down new install CLI work into issues, see epic: https://github.com/vmware-tanzu/velero/issues/2419
wip: bsl/vsl/velero readiness: https://github.com/vmware-tanzu/velero/issues/1967
presenting/demo’ing Velero at a CNCF (online) meetup: https://www.meetup.com/Cloud-Native-Talks-Brazil/events/269858787/
[steve]
community support this week
lotta PR reviews
did some testing with WIP CSI plugins, reported a couple bugs
bunch of cleanup on the https://github.com/vmware-tanzu/velero-plugin-example repo, hoping to apply similar changes to the AWS/Azure/GCP ones too
continued on backup progress idea
[nrb]
CSI upload PR should be done & ready for review now
Tackling some CRD issues this week, mainly https://github.com/vmware-tanzu/velero/issues/2383 & https://github.com/vmware-tanzu/velero/issues/2370
Started working on https://github.com/vmware-tanzu/velero/issues/1970, but ran into https://github.com/vmware-tanzu/velero/issues/2429
Discussion Topics
[nrb] When do we increment the backup format version? Refer to this comment
PR that added backup version: https://github.com/vmware-tanzu/velero/pull/1117
AI - Nolan to document when we increment backup format versions (most likely following semantic versioning)
Current PR changes are backwards compatible, so they’re a patch version in semantic versioning parlance
[dymurray/sseago] Race condition with AdditionalItmes https://github.com/vmware-tanzu/velero/issues/1350
We haven’t seen others asking for this feature, but willing to let the RH team lead with a design doc on this
[nrb] I see this as a generically useful tool, +1 to RH taking the lead here
[dymurray/sseago] Restoring PVs that already exist (related https://github.com/vmware-tanzu/velero/issues/2170)
[nrb] Is this a specific case of https://github.com/vmware-tanzu/velero/issues/1977?
data-only restores https://github.com/vmware-tanzu/velero/issues/504
Could the PV rename mechanism work here? I was thinking of the StorageClass remapping
[carlisia] should “liveness” mean “ready”, and should we use a liveness check to check for readiness: https://github.com/vmware-tanzu/velero/pull/2382
Kubernetes definitions of readiness/liveness - https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
[marccampbell] Willingness / timeline for custom restore hooks? https://github.com/vmware-tanzu/velero/issues/2116
[mayank] patch release for plugin velero.io/change-pvc-node-selector
Contributor Shoutouts
@duyanghao - docs on community project to automatically annotate all volumes for restic backup
- 9 participants
- 1:02 hours
7 Apr 2020
April 7, 2020
Status Updates
[nrb]
On community support this week
Reviews
Pushed up an example of readiness probes for Velero at https://github.com/vmware-tanzu/velero/pull/2382
Talked to Steve and Ashish about revisiting the Velero CSI plugin data flow/data model
Continued testing with CSI integration at https://github.com/vmware-tanzu/velero/pull/2323. Will likely require some changes given we’re revisiting our data model
Seeing some more CRD issues crop up, though not all around versioning - https://github.com/vmware-tanzu/velero/issues/2370 & https://github.com/vmware-tanzu/velero/issues/2383
Looks like some schema restriction errors that need to be investigated further
Worked with Rafael and Carlisia on a presentation for his work to the data protection working group
[steve]
released v1.3.2
discussion/reviews on CSI integration
install/config UX design review
investigating ways to report backup progress (https://github.com/vmware-tanzu/velero/issues/20)
PSA: looks like we’re hitting https://github.com/kubernetes/kubernetes/issues/86149 - manifests as a ~15-second delay at the start of every backup
[carlisia]
Addressed code reviews on CLI PR - close to wrapping that and starting to implement
Fixed the yaml/kustomize examples
[ashish]
Have a few PRs in the pipeline for CSI. Close to opening this up for wider review in the community. This is coming soon.
Survey that we announced last week is now live and available here. Call to community action!
Discussion Topics
[carlisia] DEMO of deploying Velero using Kustomize - Files: https://github.com/carlisia/velero/tree/c-cli-design/design/CLI/PoC
[brito-rafa]
Handling multiple API Group versions during restore (related to PR #2373)
Default (today)
Restore Source Cluster Preferred Version
Target-Version Mode (proposed - feature flag?)
Triggered by detection the target cluster runs a different K8s version from the source backup cluster (see PR #2346 collecting cluster k8s version during backup). Restore objects based on target cluster apiGroup preferred version.
Next step: Please look at the PR, and then Rafael will put together a design proposal
Contributor Shoutouts
@rochfordk - docs to clarify backup TTL (https://github.com/vmware-tanzu/velero/pull/2399)
@mansam - add --cacert flag to CLI commands (https://github.com/vmware-tanzu/velero/pull/2364)
@mynktl - add selected-node remapping plugin (https://github.com/vmware-tanzu/velero/pull/2377)
Status Updates
[nrb]
On community support this week
Reviews
Pushed up an example of readiness probes for Velero at https://github.com/vmware-tanzu/velero/pull/2382
Talked to Steve and Ashish about revisiting the Velero CSI plugin data flow/data model
Continued testing with CSI integration at https://github.com/vmware-tanzu/velero/pull/2323. Will likely require some changes given we’re revisiting our data model
Seeing some more CRD issues crop up, though not all around versioning - https://github.com/vmware-tanzu/velero/issues/2370 & https://github.com/vmware-tanzu/velero/issues/2383
Looks like some schema restriction errors that need to be investigated further
Worked with Rafael and Carlisia on a presentation for his work to the data protection working group
[steve]
released v1.3.2
discussion/reviews on CSI integration
install/config UX design review
investigating ways to report backup progress (https://github.com/vmware-tanzu/velero/issues/20)
PSA: looks like we’re hitting https://github.com/kubernetes/kubernetes/issues/86149 - manifests as a ~15-second delay at the start of every backup
[carlisia]
Addressed code reviews on CLI PR - close to wrapping that and starting to implement
Fixed the yaml/kustomize examples
[ashish]
Have a few PRs in the pipeline for CSI. Close to opening this up for wider review in the community. This is coming soon.
Survey that we announced last week is now live and available here. Call to community action!
Discussion Topics
[carlisia] DEMO of deploying Velero using Kustomize - Files: https://github.com/carlisia/velero/tree/c-cli-design/design/CLI/PoC
[brito-rafa]
Handling multiple API Group versions during restore (related to PR #2373)
Default (today)
Restore Source Cluster Preferred Version
Target-Version Mode (proposed - feature flag?)
Triggered by detection the target cluster runs a different K8s version from the source backup cluster (see PR #2346 collecting cluster k8s version during backup). Restore objects based on target cluster apiGroup preferred version.
Next step: Please look at the PR, and then Rafael will put together a design proposal
Contributor Shoutouts
@rochfordk - docs to clarify backup TTL (https://github.com/vmware-tanzu/velero/pull/2399)
@mansam - add --cacert flag to CLI commands (https://github.com/vmware-tanzu/velero/pull/2364)
@mynktl - add selected-node remapping plugin (https://github.com/vmware-tanzu/velero/pull/2377)
- 10 participants
- 60 minutes
31 Mar 2020
March 31, 2020
Status Updates
[nrb]
Testing https://github.com/vmware-tanzu/velero/pull/2323 with review feedback
Adds the --features flag into the plugin framework for plugins now, still bugs to chase down (informer cache issues)
Led to https://github.com/vmware-tanzu/velero-plugin-for-gcp/pull/23
I created https://github.com/nrb/velero-csi-env based on Ashish’s script for setting up his host path driver
Reviewed https://github.com/vmware-tanzu/velero/pull/2373
[steve]
lot of reviews (CSI, cacert, Azure storage keys)
started looking at two-stage snapshot + backup process again (more discussion below)
[carlisia]
Community support this week + PR reviews + triaging new issues
Tested new Helm 3 chart with helm2 and helm3, PR merged.
Might write a blog post about how to connect Velero with a service
Community: please review the CLI install/config redesign PR: https://github.com/vmware-tanzu/velero/pull/2202
[ashish]
CSI
Update on VolumeSnapshotRef.UID issue
Discussion w/ CSI folks on the usage of secrets in volumesnapshotclass and volumesnapshotcontents
WIP
Opened Issue 2371: Which VolumeSnapshotContents to include in the backup
Building a catalog of buleprints/workflows for backing up and restoring stateful applications/databases. Survey to go out shortly.
[brito-rafa]
Will quickly screenshare and show an example of how a backup tar ball does look like with all API Groups and versions (as PR #2373 )
Discussion Topics
Seems there is a high interest in backing up volumes by default. Should/could we prioritize this? ]
Here’s a request to add an operator to our docs: https://github.com/vmware-tanzu/velero/issues/2375
Here’s a related issue: https://github.com/vmware-tanzu/velero/issues/605
Here’s another: https://github.com/vmware-tanzu/velero/issues/1871
[steve] two-stage snapshot + backup process design (https://github.com/vmware-tanzu/velero/issues/1519)
Problem Statement
Velero does not wait/check for snapshots to be made durable/restorable
A backup marked as completed may still be at risk if the snapshot data is not made durable
A restore may fail if its backup’s snapshots are not yet ready to be restored from
Things to Consider
Do we want to solve this for Velero snapshots? CSI snapshots? Both?
There’s potentially a difference between “durable” and “ready to restore” that needs to be modeled
With EBS/GCP, the snapshot can’t be restored until it’s been made durable by replicating the data to object storage, so they’re effectively the same thing
With the vSphere plugin and maybe the OpenEBS plugin, a local snapshot can be restored from, even if it hasn’t yet been made durable
Should Velero actively drive the upload process, or should it passively check for upload status as reported by an external component?
We don’t want to block the Velero backup queue while waiting for snapshot data to be made durable
We need to keep the timespan between pre- and post-hooks as short as possible, i.e. data replication should take place outside of hook execution
Since it may take a significant amount of time to make a snapshot durable, we need to be able to handle pod restarts, network interruptions, etc. gracefully.
[nolan] v1.3.2 release this week
https://github.com/vmware-tanzu/velero/pull/2350 for plugin dir in object storage
Anything else?
[Mayank] can we upgrade from 1.0.0 to 1.3.1? https://kubernetes.slack.com/archives/C6VCGP4MT/p1585647294073200
[Dylan] Concurrent Backup/Restores
https://github.com/vmware-tanzu/velero/issues/487
Contributor Shoutouts
@mansam for adding support for custom cert bundles (https://github.com/vmware-tanzu/velero/pull/2353, https://github.com/vmware-tanzu/velero-plugin-for-aws/pull/34 and more)
@jaygridley - Azure: support using static storage keys (https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/pull/32)
Shoutout to Jonas for adding search to our docs!
Helm chart
@yurinnick for making the chart Helm 3 compatible (https://github.com/vmware-tanzu/helm-charts/pull/81)
@yurinnick for fixing timeout chart-testing parameter (https://github.com/vmware-tanzu/helm-charts/pull/85)
Status Updates
[nrb]
Testing https://github.com/vmware-tanzu/velero/pull/2323 with review feedback
Adds the --features flag into the plugin framework for plugins now, still bugs to chase down (informer cache issues)
Led to https://github.com/vmware-tanzu/velero-plugin-for-gcp/pull/23
I created https://github.com/nrb/velero-csi-env based on Ashish’s script for setting up his host path driver
Reviewed https://github.com/vmware-tanzu/velero/pull/2373
[steve]
lot of reviews (CSI, cacert, Azure storage keys)
started looking at two-stage snapshot + backup process again (more discussion below)
[carlisia]
Community support this week + PR reviews + triaging new issues
Tested new Helm 3 chart with helm2 and helm3, PR merged.
Might write a blog post about how to connect Velero with a service
Community: please review the CLI install/config redesign PR: https://github.com/vmware-tanzu/velero/pull/2202
[ashish]
CSI
Update on VolumeSnapshotRef.UID issue
Discussion w/ CSI folks on the usage of secrets in volumesnapshotclass and volumesnapshotcontents
WIP
Opened Issue 2371: Which VolumeSnapshotContents to include in the backup
Building a catalog of buleprints/workflows for backing up and restoring stateful applications/databases. Survey to go out shortly.
[brito-rafa]
Will quickly screenshare and show an example of how a backup tar ball does look like with all API Groups and versions (as PR #2373 )
Discussion Topics
Seems there is a high interest in backing up volumes by default. Should/could we prioritize this? ]
Here’s a request to add an operator to our docs: https://github.com/vmware-tanzu/velero/issues/2375
Here’s a related issue: https://github.com/vmware-tanzu/velero/issues/605
Here’s another: https://github.com/vmware-tanzu/velero/issues/1871
[steve] two-stage snapshot + backup process design (https://github.com/vmware-tanzu/velero/issues/1519)
Problem Statement
Velero does not wait/check for snapshots to be made durable/restorable
A backup marked as completed may still be at risk if the snapshot data is not made durable
A restore may fail if its backup’s snapshots are not yet ready to be restored from
Things to Consider
Do we want to solve this for Velero snapshots? CSI snapshots? Both?
There’s potentially a difference between “durable” and “ready to restore” that needs to be modeled
With EBS/GCP, the snapshot can’t be restored until it’s been made durable by replicating the data to object storage, so they’re effectively the same thing
With the vSphere plugin and maybe the OpenEBS plugin, a local snapshot can be restored from, even if it hasn’t yet been made durable
Should Velero actively drive the upload process, or should it passively check for upload status as reported by an external component?
We don’t want to block the Velero backup queue while waiting for snapshot data to be made durable
We need to keep the timespan between pre- and post-hooks as short as possible, i.e. data replication should take place outside of hook execution
Since it may take a significant amount of time to make a snapshot durable, we need to be able to handle pod restarts, network interruptions, etc. gracefully.
[nolan] v1.3.2 release this week
https://github.com/vmware-tanzu/velero/pull/2350 for plugin dir in object storage
Anything else?
[Mayank] can we upgrade from 1.0.0 to 1.3.1? https://kubernetes.slack.com/archives/C6VCGP4MT/p1585647294073200
[Dylan] Concurrent Backup/Restores
https://github.com/vmware-tanzu/velero/issues/487
Contributor Shoutouts
@mansam for adding support for custom cert bundles (https://github.com/vmware-tanzu/velero/pull/2353, https://github.com/vmware-tanzu/velero-plugin-for-aws/pull/34 and more)
@jaygridley - Azure: support using static storage keys (https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/pull/32)
Shoutout to Jonas for adding search to our docs!
Helm chart
@yurinnick for making the chart Helm 3 compatible (https://github.com/vmware-tanzu/helm-charts/pull/81)
@yurinnick for fixing timeout chart-testing parameter (https://github.com/vmware-tanzu/helm-charts/pull/85)
- 13 participants
- 1:07 hours
24 Mar 2020
March 24, 2020
Status Updates
[ashish]
Working on CSI
Discussion on VolumeSnapshotRef.UID should be cleared on volumesnapshot deletion
May have to rethink the restore scenario as there are some logic that depends on how volumesnapshotcontents are named.
Coming w/ alternative approach here, plan for the week
BackupItemAction plugin to for CSI volumesnapshots, created outside of Velero.
WIP PR #31
//TODO: link issue here
Building a catalog of SOPs for backing up and restoring datastores. Help us prioritize during discussion:
mySQL: Single pod/ HA?
Cassandra
MariaDB
MS SQL?
PostgreSQL
MongoDB
[brito-rafa]
In progress code to backup multiple API Group versions (instead of only preferred APIGroups)
Table of preferred group version and supported groups
Quick demo of horizontalautoscaler item: v1 versus v2beta2
Collect any ideas and feedback from the community
[steve]
did some CSI snapshot testing on Azure
lot of support & reviews
gonna have to rebase https://github.com/vmware-tanzu/velero/pull/2317 at a quiet time
[carlisia]
CLI revamp design doc ready for another round of reviews: https://github.com/vmware-tanzu/velero/pull/2202
Testing PR to update the Velero Helm chart to version 3: https://github.com/vmware-tanzu/helm-charts/pull/81
[nolan]
Need to reply to reviews on https://github.com/vmware-tanzu/velero/pull/2323
Also need to set up CSI test beds, didn’t actually run this code yet
Last week was mostly spent doing internal presentation stuff
Testing and documenting Helm 2 chart behavior with Velero
Releases restored before Tiller aren’t “adopted”
Discussion Topics
[Cyril] Cassandra backups and missing features
https://github.com/vmware-tanzu/velero/issues/2361
[Mayank] restoreItemAction plugin to clear/update ‘selected-node’ in PVC, which are created by scheduler for dynamic provisioning.
Code https://github.com/mynktl/velero-plugin/commit/a03a74004bf0482537649ae4b390cbd2ee9e2ac5
steve: :+1: to PR into core (detailed code review still to be done)
[Nolan] Kubernetes 1.18 comes out today, if anyone has bandwidth to try Velero v1.3.1 with it, that would be awesome!
Status Updates
[ashish]
Working on CSI
Discussion on VolumeSnapshotRef.UID should be cleared on volumesnapshot deletion
May have to rethink the restore scenario as there are some logic that depends on how volumesnapshotcontents are named.
Coming w/ alternative approach here, plan for the week
BackupItemAction plugin to for CSI volumesnapshots, created outside of Velero.
WIP PR #31
//TODO: link issue here
Building a catalog of SOPs for backing up and restoring datastores. Help us prioritize during discussion:
mySQL: Single pod/ HA?
Cassandra
MariaDB
MS SQL?
PostgreSQL
MongoDB
[brito-rafa]
In progress code to backup multiple API Group versions (instead of only preferred APIGroups)
Table of preferred group version and supported groups
Quick demo of horizontalautoscaler item: v1 versus v2beta2
Collect any ideas and feedback from the community
[steve]
did some CSI snapshot testing on Azure
lot of support & reviews
gonna have to rebase https://github.com/vmware-tanzu/velero/pull/2317 at a quiet time
[carlisia]
CLI revamp design doc ready for another round of reviews: https://github.com/vmware-tanzu/velero/pull/2202
Testing PR to update the Velero Helm chart to version 3: https://github.com/vmware-tanzu/helm-charts/pull/81
[nolan]
Need to reply to reviews on https://github.com/vmware-tanzu/velero/pull/2323
Also need to set up CSI test beds, didn’t actually run this code yet
Last week was mostly spent doing internal presentation stuff
Testing and documenting Helm 2 chart behavior with Velero
Releases restored before Tiller aren’t “adopted”
Discussion Topics
[Cyril] Cassandra backups and missing features
https://github.com/vmware-tanzu/velero/issues/2361
[Mayank] restoreItemAction plugin to clear/update ‘selected-node’ in PVC, which are created by scheduler for dynamic provisioning.
Code https://github.com/mynktl/velero-plugin/commit/a03a74004bf0482537649ae4b390cbd2ee9e2ac5
steve: :+1: to PR into core (detailed code review still to be done)
[Nolan] Kubernetes 1.18 comes out today, if anyone has bandwidth to try Velero v1.3.1 with it, that would be awesome!
- 12 participants
- 50 minutes
17 Mar 2020
March 17, 2020
Agenda
Status Updates
[ashish]
More CSI work and getting couple of test and cleanup PRs merged.
Preparing for a talk on Velero with Nolan
Brought up issues from last week in upstream volumesnapshot working group meeting. More discussions will happen on these.
VolumeSnapshotRef.UID should be cleared on volumesnapshot deletion
More discussions on this in the data protection working group
Relation between VolumeSnapshotContent and VolumeSnapshot should be 1:many
Effectively won’t fix as there are reasonable workarounds
Lifecycle of VolumeSnapshot Kubernetes objects should be decoupled from that in the storage system
More discussions on this will happen in the issue.
[steve]
shipped v1.3.1
finalized Azure blob chunking code
starting to look at two-stage snapshot/backup issue again (https://github.com/vmware-tanzu/velero/issues/1519)
[carlisia]
CLI PR good to review: https://github.com/vmware-tanzu/velero/pull/2202 (look at the TODO list, working on the last 2 items)
[Nolan]
Prepping talk with Ashish
More work on https://github.com/vmware-tanzu/velero/pull/2323 for uploading CSI resources to object storage
Allows us to show these snapshots in the client output & do imports w/ the backups if we need
Discussions w/ Ashish on the snapshot issues he mentioned
[brito-rafa]
new labels for the backups #2342
Examples:
velero.io/source-cluster-k8s-major-version=1
velero.io/source-cluster-k8s-minor-version=14
velero.io/source-cluster-k8s-version=v1.14.10
Discussion Topics
[steve] add plugins/ top-level dir to backup storage locations to allow plugins to write arbitrary data to bucket – https://github.com/vmware-tanzu/velero/issues/2344
[Mayank] restoreItemAction plugin to clear/update ‘selected-node’ in PVC, which are created by scheduler for dynamic provisioning. Code https://github.com/mynktl/velero-plugin/commit/a03a74004bf0482537649ae4b390cbd2ee9e2ac5
[Antony] How can we control ordering of resource backups and restores?
We don’t currently have the mechanisms to do this in Velero
StatefulSets don’t have proper support; we need to special case them, but it’s not a generalized solution
Antony to file an issue and we can brainstorm solutions
[Sam Lucidi] Custom CA support
Working on how to get certificate bundle on the pods
Both plugins and restic need access to it, but have slightly different paths to that data.
Agenda
Status Updates
[ashish]
More CSI work and getting couple of test and cleanup PRs merged.
Preparing for a talk on Velero with Nolan
Brought up issues from last week in upstream volumesnapshot working group meeting. More discussions will happen on these.
VolumeSnapshotRef.UID should be cleared on volumesnapshot deletion
More discussions on this in the data protection working group
Relation between VolumeSnapshotContent and VolumeSnapshot should be 1:many
Effectively won’t fix as there are reasonable workarounds
Lifecycle of VolumeSnapshot Kubernetes objects should be decoupled from that in the storage system
More discussions on this will happen in the issue.
[steve]
shipped v1.3.1
finalized Azure blob chunking code
starting to look at two-stage snapshot/backup issue again (https://github.com/vmware-tanzu/velero/issues/1519)
[carlisia]
CLI PR good to review: https://github.com/vmware-tanzu/velero/pull/2202 (look at the TODO list, working on the last 2 items)
[Nolan]
Prepping talk with Ashish
More work on https://github.com/vmware-tanzu/velero/pull/2323 for uploading CSI resources to object storage
Allows us to show these snapshots in the client output & do imports w/ the backups if we need
Discussions w/ Ashish on the snapshot issues he mentioned
[brito-rafa]
new labels for the backups #2342
Examples:
velero.io/source-cluster-k8s-major-version=1
velero.io/source-cluster-k8s-minor-version=14
velero.io/source-cluster-k8s-version=v1.14.10
Discussion Topics
[steve] add plugins/ top-level dir to backup storage locations to allow plugins to write arbitrary data to bucket – https://github.com/vmware-tanzu/velero/issues/2344
[Mayank] restoreItemAction plugin to clear/update ‘selected-node’ in PVC, which are created by scheduler for dynamic provisioning. Code https://github.com/mynktl/velero-plugin/commit/a03a74004bf0482537649ae4b390cbd2ee9e2ac5
[Antony] How can we control ordering of resource backups and restores?
We don’t currently have the mechanisms to do this in Velero
StatefulSets don’t have proper support; we need to special case them, but it’s not a generalized solution
Antony to file an issue and we can brainstorm solutions
[Sam Lucidi] Custom CA support
Working on how to get certificate bundle on the pods
Both plugins and restic need access to it, but have slightly different paths to that data.
- 9 participants
- 29 minutes
10 Mar 2020
March 10, 2020
Agenda
Status Updates
ashish
CSI Integration
Use Velero to backup and restore using the v1beta1 Snapshot APIs
Backup and restores- in-cluster restores and cluster migration. Live demo of this later.
Created some issues:
Upstream external snapshotter:
VolumeSnapshotRef.UID should be cleared on volumesnapshot deletion
Relation between VolumeSnapshotContent and VolumeSnapshot should be 1:many
Lifecycle of VolumeSnapshot Kubernetes objects should be decoupled from that in the storage system
Velero:
Ensure restored workloads can be backed up
Ensure backup and restore of volumesnapshots created outside of velero work
Repeated backup of the same CSI workload should not leave orphan resources
I’ll add more details into the issues in the Velero repo.
See our progress in the velero-plugin-for-csi repo and feel free to jump in and contribute.
carlisia
Was out last week
On support rotation this week
Catching up on PR reviews
Pushed an update to the CLI PR containing yaml and kustomize files that properly create/update a velero deployment (https://github.com/vmware-tanzu/velero/pull/2202/commits/bda5099153c588477c80494f1782147108031b51)
jonas
Plugins page is live!
https://velero.io/plugins/
nolan
PR #2232, fixing CRD int/float conversions on backup.
Thank you to guyguy333 and NissesSenap for the reports, data, and testing!
Tracked down nodePort validation error (issue #2308), will require some more thinking
Started working on uploading CSI artifacts to object storage
Need to review Unprioritized backlog to help build roadmap, missed that last week
steve
finalized refactoring of restore/discovery code (https://github.com/vmware-tanzu/velero/pull/2248)
working on refactoring to move logic out of controllers
merged https://github.com/vmware-tanzu/velero/pull/2299
WIP https://github.com/vmware-tanzu/velero/pull/2317
worked on implementing blob chunking in azure plugin: https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/pull/30
lots of reviews
Discussion Topics
v1.3.1 patch release
https://github.com/vmware-tanzu/velero/pull/2322
nrb and skriss will try to get v1.3.1 out today
Important for users of v1.3.1 to upgrade, but it’s largely based on what CRDs users have; if there are numeric fields, they will see this bug
Update from Dylan Murray - actively working on https://github.com/vmware-tanzu/velero/issues/2332 before tackling client & install issues
Contributor Shoutouts
guyguy333 and NissesSenap for reports, data, and testing for issue 2319/PR 2322
betta1 for the detailed reproduction case on issue 2308
Agenda
Status Updates
ashish
CSI Integration
Use Velero to backup and restore using the v1beta1 Snapshot APIs
Backup and restores- in-cluster restores and cluster migration. Live demo of this later.
Created some issues:
Upstream external snapshotter:
VolumeSnapshotRef.UID should be cleared on volumesnapshot deletion
Relation between VolumeSnapshotContent and VolumeSnapshot should be 1:many
Lifecycle of VolumeSnapshot Kubernetes objects should be decoupled from that in the storage system
Velero:
Ensure restored workloads can be backed up
Ensure backup and restore of volumesnapshots created outside of velero work
Repeated backup of the same CSI workload should not leave orphan resources
I’ll add more details into the issues in the Velero repo.
See our progress in the velero-plugin-for-csi repo and feel free to jump in and contribute.
carlisia
Was out last week
On support rotation this week
Catching up on PR reviews
Pushed an update to the CLI PR containing yaml and kustomize files that properly create/update a velero deployment (https://github.com/vmware-tanzu/velero/pull/2202/commits/bda5099153c588477c80494f1782147108031b51)
jonas
Plugins page is live!
https://velero.io/plugins/
nolan
PR #2232, fixing CRD int/float conversions on backup.
Thank you to guyguy333 and NissesSenap for the reports, data, and testing!
Tracked down nodePort validation error (issue #2308), will require some more thinking
Started working on uploading CSI artifacts to object storage
Need to review Unprioritized backlog to help build roadmap, missed that last week
steve
finalized refactoring of restore/discovery code (https://github.com/vmware-tanzu/velero/pull/2248)
working on refactoring to move logic out of controllers
merged https://github.com/vmware-tanzu/velero/pull/2299
WIP https://github.com/vmware-tanzu/velero/pull/2317
worked on implementing blob chunking in azure plugin: https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/pull/30
lots of reviews
Discussion Topics
v1.3.1 patch release
https://github.com/vmware-tanzu/velero/pull/2322
nrb and skriss will try to get v1.3.1 out today
Important for users of v1.3.1 to upgrade, but it’s largely based on what CRDs users have; if there are numeric fields, they will see this bug
Update from Dylan Murray - actively working on https://github.com/vmware-tanzu/velero/issues/2332 before tackling client & install issues
Contributor Shoutouts
guyguy333 and NissesSenap for reports, data, and testing for issue 2319/PR 2322
betta1 for the detailed reproduction case on issue 2308
- 11 participants
- 56 minutes
3 Mar 2020
March 03, 2020
Agenda
Status Updates
[steve]
shipped v1.3.0!
GitHub release
blog post
getting go 1.14/modules PR ready (https://github.com/vmware-tanzu/velero/pull/2214)
working on moving code out of controllers, into packages – starting with backup & restore
https://github.com/vmware-tanzu/velero/pull/2299
more PRs to come
will finalize refactoring of restore/discovery code (https://github.com/vmware-tanzu/velero/pull/2248)
[nolan]
On community support this week
Worked through my review backlog yesterday
Reviewed Xing Yang’s ExecutionHook KEP upstream
Going to pick up some CSI work this afternoon - namely syncing VolumeSnapshots & VolumeSnapshotContents into object storage so we can use it in client commands and syncing them into clusters
Will be on Bret Fisher’s YouTube show talking about Velero on 4/30.
[ashish]
Was on community support last week.
Discovered a missing feature of importing snapshots with the csi-hostpath-plugin and implemented that in this PR Discover existing snapshots on disk int he csi-hostpathplugin repo.
Making progress on the CSI integration work.
CSI blogpost that I wanted to put together already exists at here
(carlisia’s on PTO)
[jonas] Finishing up the plugins page
Discussion Topics
[nolan] Backlog grooming
Michael has asked us to review our “Unprioritized backlog” in ZenHub and form a rough roadmap for 1.5 and beyond
Tag @skriss and/or Michael for any issues you’d like us to take a look at in the next 6-9 months
Contributor Shoutouts
Helm/Velero chart: https://github.com/vmware-tanzu/helm-charts/pulls?q=is%3Apr+is%3Aclosed
Agenda
Status Updates
[steve]
shipped v1.3.0!
GitHub release
blog post
getting go 1.14/modules PR ready (https://github.com/vmware-tanzu/velero/pull/2214)
working on moving code out of controllers, into packages – starting with backup & restore
https://github.com/vmware-tanzu/velero/pull/2299
more PRs to come
will finalize refactoring of restore/discovery code (https://github.com/vmware-tanzu/velero/pull/2248)
[nolan]
On community support this week
Worked through my review backlog yesterday
Reviewed Xing Yang’s ExecutionHook KEP upstream
Going to pick up some CSI work this afternoon - namely syncing VolumeSnapshots & VolumeSnapshotContents into object storage so we can use it in client commands and syncing them into clusters
Will be on Bret Fisher’s YouTube show talking about Velero on 4/30.
[ashish]
Was on community support last week.
Discovered a missing feature of importing snapshots with the csi-hostpath-plugin and implemented that in this PR Discover existing snapshots on disk int he csi-hostpathplugin repo.
Making progress on the CSI integration work.
CSI blogpost that I wanted to put together already exists at here
(carlisia’s on PTO)
[jonas] Finishing up the plugins page
Discussion Topics
[nolan] Backlog grooming
Michael has asked us to review our “Unprioritized backlog” in ZenHub and form a rough roadmap for 1.5 and beyond
Tag @skriss and/or Michael for any issues you’d like us to take a look at in the next 6-9 months
Contributor Shoutouts
Helm/Velero chart: https://github.com/vmware-tanzu/helm-charts/pulls?q=is%3Apr+is%3Aclosed
- 6 participants
- 24 minutes
25 Feb 2020
Feb 25, 2020
Agenda
--------
Status Updates
--------
[steve]
community support last week
shipped v1.3.0-beta.2 with a couple more fixes
updating v1.3 blog post
[carlisia]
Slow progress with the CLI design/PoC
Caught up on PRs that requested my review
Intention for the week: go over other helm+velero PRs
Stretch: go over issues and give feedback
On PTO next week
[nolan]
Got PR #2264 finished
Reviewed Ashish’s PR for CSI updates
Will be at KubeCon EU
Need to follow up on https://github.com/kubernetes/kubernetes/issues/87675 from Scott Seago
[ashish]
Working on integration of CSI snapshot APIs into the csi-plugin-for-velero.
This PR implements backup support for volumes backed by CSI drivers.
This was demo’d for the team internally and we will have a demo for the community in the coming weeks.
Will be continuing work on the restore scenario.
On community support this week
Also want to get started on a blogpost about setting up an environment that supports CSI external-snapshotter APIs.
Discussion Topics
--------
@dymurray Restic Restore Failures with OpenShift (https://github.com/vmware-tanzu/velero/issues/1981)
@dymurray Custom CA bundle design update (https://github.com/vmware-tanzu/velero/pull/2259)
Contributor Shoutouts
--------
@jenting - back up PVs to correct location with restic (https://github.com/vmware-tanzu/velero/pull/2244)
Helm/Velero chart: https://github.com/vmware-tanzu/helm-charts/pulls?q=is%3Apr+is%3Aclosed
Agenda
--------
Status Updates
--------
[steve]
community support last week
shipped v1.3.0-beta.2 with a couple more fixes
updating v1.3 blog post
[carlisia]
Slow progress with the CLI design/PoC
Caught up on PRs that requested my review
Intention for the week: go over other helm+velero PRs
Stretch: go over issues and give feedback
On PTO next week
[nolan]
Got PR #2264 finished
Reviewed Ashish’s PR for CSI updates
Will be at KubeCon EU
Need to follow up on https://github.com/kubernetes/kubernetes/issues/87675 from Scott Seago
[ashish]
Working on integration of CSI snapshot APIs into the csi-plugin-for-velero.
This PR implements backup support for volumes backed by CSI drivers.
This was demo’d for the team internally and we will have a demo for the community in the coming weeks.
Will be continuing work on the restore scenario.
On community support this week
Also want to get started on a blogpost about setting up an environment that supports CSI external-snapshotter APIs.
Discussion Topics
--------
@dymurray Restic Restore Failures with OpenShift (https://github.com/vmware-tanzu/velero/issues/1981)
@dymurray Custom CA bundle design update (https://github.com/vmware-tanzu/velero/pull/2259)
Contributor Shoutouts
--------
@jenting - back up PVs to correct location with restic (https://github.com/vmware-tanzu/velero/pull/2244)
Helm/Velero chart: https://github.com/vmware-tanzu/helm-charts/pulls?q=is%3Apr+is%3Aclosed
- 9 participants
- 31 minutes
18 Feb 2020
Feb 18, 2020
Agenda
-------
Status Updates
[steve]
returning from 4-day weekend
on community support this week
[carlisia]
Have yaml files that get installed with kustomize working
Had a meeting to go over kutomize and questions:
Next step: getting updates to be patched w/ kustomize
Need to catch up with PR reviews today
[nolan]
Two CRD proposals up on PR https://github.com/vmware-tanzu/velero/pull/2264
Returning to CSI testing with Ashish this week
[ashish]
making progress on the velero plugin for CSI
PRs to switch to go modules
PR incoming for the backup flow.
Release from plugin repos to switch from nobody:nobody to nobody:nogroup. Current beta release of Velero is incompatible with the existing releases of the plugins in certain linux distros.
Discussion Topics
------------------
[steve] v1.3.0 status
need to finalize approach for https://github.com/vmware-tanzu/velero/pull/2264
[m2] Discuss cleanup request https://github.com/vmware-tanzu/velero/issues/1977
[m2] Update on custom CA support https://github.com/vmware-tanzu/velero/pull/2259
Agenda
-------
Status Updates
[steve]
returning from 4-day weekend
on community support this week
[carlisia]
Have yaml files that get installed with kustomize working
Had a meeting to go over kutomize and questions:
Next step: getting updates to be patched w/ kustomize
Need to catch up with PR reviews today
[nolan]
Two CRD proposals up on PR https://github.com/vmware-tanzu/velero/pull/2264
Returning to CSI testing with Ashish this week
[ashish]
making progress on the velero plugin for CSI
PRs to switch to go modules
PR incoming for the backup flow.
Release from plugin repos to switch from nobody:nobody to nobody:nogroup. Current beta release of Velero is incompatible with the existing releases of the plugins in certain linux distros.
Discussion Topics
------------------
[steve] v1.3.0 status
need to finalize approach for https://github.com/vmware-tanzu/velero/pull/2264
[m2] Discuss cleanup request https://github.com/vmware-tanzu/velero/issues/1977
[m2] Update on custom CA support https://github.com/vmware-tanzu/velero/pull/2259
- 7 participants
- 28 minutes
11 Feb 2020
Feb 11, 2020
Agenda
Status Updates
---------------
[steve]
shipped v1.3.0-beta.1, working on blog post
put up draft PR with refactor of restore code related to CRDs
some testing
[carlisia]
community support last week
updated CLI revamp PR: https://github.com/vmware-tanzu/velero/pull/2202#issuecomment-577229282
POC for installing with kustomize
[nolan]
On community support this week
Moved last week, was out
Back to working on CRDs, troubleshooting #2249. Will go more in depth in discussion topics
Need to work with Ashish on CSI support
[ashish]
Update on memory leak #2069
Fixed with Steve’s change and v1.3.0-beta.1 has the fix. Please consider upgrading.
CSI update
Using CSI hostpath-plugin driver to validate Velero
Successfully setup an environment with hostpath-plugin driver using beta snapshot APIs
Blogpost?
[jonas]
Implemented an ADOPTERS.md file for public user statements
Working on a /plugins page
Discussion Topics
------------------
[carlisia] CRD/api/k8s versioning issues: https://github.com/vmware-tanzu/velero/labels/versioning
Visual summary of #2249
[dymurray] Add design doc for Custom CA support for S3 BSLs
Contributor Shoutouts
----------------------
@markrity - docs update to point to new Helm chart repo (https://github.com/vmware-tanzu/velero/pull/2209)
@jnaulty - add Restore API docs (https://github.com/vmware-tanzu/velero/pull/2240)
@mynktl - allow VolumeSnapshotter plugin to rename PV via SetVolumeID function (https://github.com/vmware-tanzu/velero/pull/2216)
@jenting - ongoing beta testing, bug-fixing
@betta1 - Thank you for providing information to narrow down memory leak
Agenda
Status Updates
---------------
[steve]
shipped v1.3.0-beta.1, working on blog post
put up draft PR with refactor of restore code related to CRDs
some testing
[carlisia]
community support last week
updated CLI revamp PR: https://github.com/vmware-tanzu/velero/pull/2202#issuecomment-577229282
POC for installing with kustomize
[nolan]
On community support this week
Moved last week, was out
Back to working on CRDs, troubleshooting #2249. Will go more in depth in discussion topics
Need to work with Ashish on CSI support
[ashish]
Update on memory leak #2069
Fixed with Steve’s change and v1.3.0-beta.1 has the fix. Please consider upgrading.
CSI update
Using CSI hostpath-plugin driver to validate Velero
Successfully setup an environment with hostpath-plugin driver using beta snapshot APIs
Blogpost?
[jonas]
Implemented an ADOPTERS.md file for public user statements
Working on a /plugins page
Discussion Topics
------------------
[carlisia] CRD/api/k8s versioning issues: https://github.com/vmware-tanzu/velero/labels/versioning
Visual summary of #2249
[dymurray] Add design doc for Custom CA support for S3 BSLs
Contributor Shoutouts
----------------------
@markrity - docs update to point to new Helm chart repo (https://github.com/vmware-tanzu/velero/pull/2209)
@jnaulty - add Restore API docs (https://github.com/vmware-tanzu/velero/pull/2240)
@mynktl - allow VolumeSnapshotter plugin to rename PV via SetVolumeID function (https://github.com/vmware-tanzu/velero/pull/2216)
@jenting - ongoing beta testing, bug-fixing
@betta1 - Thank you for providing information to narrow down memory leak
- 10 participants
- 50 minutes
4 Feb 2020
Feb 4, 2020
Agenda
Status Updates
----------------
Steve
prepping for v1.3.0-beta.1 release - will be pushed out later today
need to see if we should release new versions of the cloud provider plugins
working on some refactoring in the restore code related to restoring CRDs
Ashish
Coming back from community support
Checking in on https://github.com/vmware-tanzu/velero/issues/2069
Have some troubleshooting steps
Carlisia
On community support
CLI install redesign - https://github.com/vmware-tanzu/velero/pull/2202#issuecomment-577229282
Nolan on PTO
Discussion Topics
------------------
[carlisia] Drop CRDs manifest but install CRDs by velero install? https://github.com/vmware-tanzu/helm-charts/issues/44
[jonas] Would it make sense to create an ADOPTERS.md file, where people and organizations can PR themselves in? Similar to Linkerd and Harbor?
[carlisia] How to best deprecate CLI cmds, add/remove the word new?: https://github.com/vmware-tanzu/velero/pull/2202#issuecomment-577229282
[carlisia]
velero [resource] [operation] [flags]
OR
velero [operation] [resource] [flags]
https://github.com/vmware-tanzu/velero/pull/2202#issuecomment-581961860
[dymurray] Unfortunately need to miss the community meeting today but hoping to have a design doc PR up by EOD tomorrow for https://github.com/vmware-tanzu/velero/issues/1027
Contributor Shoutouts
----------------------
@jenting - prune unknown fields on Velero CRDs (https://github.com/vmware-tanzu/velero/pull/2187)
@cpanato - allow velero restore create --from-schedule to work for schedules in other clusters (https://github.com/vmware-tanzu/velero/pull/2218)
@cpanato - use GA topology labels on PVs by default (https://github.com/vmware-tanzu/velero/pull/2219)
@koehn - fix panic in restic status updating for empty output (https://github.com/vmware-tanzu/velero/pull/2231)
@shaneutt - add arm/arm64 container builds (https://github.com/vmware-tanzu/velero/pull/2227)
Agenda
Status Updates
----------------
Steve
prepping for v1.3.0-beta.1 release - will be pushed out later today
need to see if we should release new versions of the cloud provider plugins
working on some refactoring in the restore code related to restoring CRDs
Ashish
Coming back from community support
Checking in on https://github.com/vmware-tanzu/velero/issues/2069
Have some troubleshooting steps
Carlisia
On community support
CLI install redesign - https://github.com/vmware-tanzu/velero/pull/2202#issuecomment-577229282
Nolan on PTO
Discussion Topics
------------------
[carlisia] Drop CRDs manifest but install CRDs by velero install? https://github.com/vmware-tanzu/helm-charts/issues/44
[jonas] Would it make sense to create an ADOPTERS.md file, where people and organizations can PR themselves in? Similar to Linkerd and Harbor?
[carlisia] How to best deprecate CLI cmds, add/remove the word new?: https://github.com/vmware-tanzu/velero/pull/2202#issuecomment-577229282
[carlisia]
velero [resource] [operation] [flags]
OR
velero [operation] [resource] [flags]
https://github.com/vmware-tanzu/velero/pull/2202#issuecomment-581961860
[dymurray] Unfortunately need to miss the community meeting today but hoping to have a design doc PR up by EOD tomorrow for https://github.com/vmware-tanzu/velero/issues/1027
Contributor Shoutouts
----------------------
@jenting - prune unknown fields on Velero CRDs (https://github.com/vmware-tanzu/velero/pull/2187)
@cpanato - allow velero restore create --from-schedule to work for schedules in other clusters (https://github.com/vmware-tanzu/velero/pull/2218)
@cpanato - use GA topology labels on PVs by default (https://github.com/vmware-tanzu/velero/pull/2219)
@koehn - fix panic in restic status updating for empty output (https://github.com/vmware-tanzu/velero/pull/2231)
@shaneutt - add arm/arm64 container builds (https://github.com/vmware-tanzu/velero/pull/2227)
- 7 participants
- 38 minutes
28 Jan 2020
# Jan 28, 2020
## Agenda
- ### Status Updates
- Nolan (have to leave early)
- Working CRD restore issues
- I think all we have left is the int64/float64 conversion on some CRDs raised by Scott Seago on https://github.com/vmware-tanzu/velero/pull/1937
- CSI update
- Currently AWS, GCP, and Azure aren't up-to-date on their CSI drivers, partially due to the newer v2.0.0 images requiring Kubernetes v1.17
- We're going to work on implementing Velero against the v2.0.0 codebase and v1beta1 CRDs and help update their drivers where we can, using the demo hostpath driver for now.
- Ashish (also have to leave early)
- Same update as Nolan on CSI
- Path forward:
- Test Velero against the hostpath plugin.
- Currently figuring out the deployment of a CSI driver:
- All the various sidecars that need to be used
- What should a deployment of the CSI driver look like:
- This information is a bit muddled/ fragmented across the various drivers.
- On community support this week:
- Getting caught-up on PRs and open issues
- [carlisia]
- All caught up on PR reviews except 1 for the helm-chart and 1 for the aws plugin
- Updated the resources page with latest videos/blog posts/podcast about Velero. New entries are welcome if anyone knows of resources not yet included.
- Contributed to discussions and revisions of a submission for VMware internal conference on a Velero related topic, submission is today
- Participated in the DP WG meeting, great start
- Back to updating CLI design doc after this meeting!
- Steve
- lots of PR reviews
- working with @nrb on sorting out CRD backup/restore issues
- ### Discussion Topics
- [steve] do we want to release a v1.3 soon-ish with the CRD fixes and the other [~70 commits](https://github.com/vmware-tanzu/velero/compare/v1.2.0...master) in master since v1.2, and push the CSI integration to v1.4 (can still keep to an end of March timeframe if feasible)?
- thinking here is that it's easier to release everything in master than cherrypicking a few things, and we have a decent # of commits built up already, which I don't really see a reason to hold back
- [Antony B] - What about https://github.com/vmware-tanzu/velero/issues/2069 ? Is that still 1.3 or 1.4?
- We'll review and see if we can fit it in for 1.3 or have to move it.
- [carlisia] Do we want to move https://github.com/vmware-tanzu/velero/pull/2214 (migrate to go 1.13 + modules) to "Ready for review"?
- This doesn't look like it'd be part of Velero: https://github.com/vmware-tanzu/velero/pull/2119 ([design] Proposal of triggering backups based on Kubernetes events)
- ### Contributor Shoutouts!
- @cpanato - CLI auto-completion docs (https://github.com/vmware-tanzu/velero/pull/2208)
- @Sh4d1 - bump restic version (https://github.com/vmware-tanzu/velero/pull/2210)
## Agenda
- ### Status Updates
- Nolan (have to leave early)
- Working CRD restore issues
- I think all we have left is the int64/float64 conversion on some CRDs raised by Scott Seago on https://github.com/vmware-tanzu/velero/pull/1937
- CSI update
- Currently AWS, GCP, and Azure aren't up-to-date on their CSI drivers, partially due to the newer v2.0.0 images requiring Kubernetes v1.17
- We're going to work on implementing Velero against the v2.0.0 codebase and v1beta1 CRDs and help update their drivers where we can, using the demo hostpath driver for now.
- Ashish (also have to leave early)
- Same update as Nolan on CSI
- Path forward:
- Test Velero against the hostpath plugin.
- Currently figuring out the deployment of a CSI driver:
- All the various sidecars that need to be used
- What should a deployment of the CSI driver look like:
- This information is a bit muddled/ fragmented across the various drivers.
- On community support this week:
- Getting caught-up on PRs and open issues
- [carlisia]
- All caught up on PR reviews except 1 for the helm-chart and 1 for the aws plugin
- Updated the resources page with latest videos/blog posts/podcast about Velero. New entries are welcome if anyone knows of resources not yet included.
- Contributed to discussions and revisions of a submission for VMware internal conference on a Velero related topic, submission is today
- Participated in the DP WG meeting, great start
- Back to updating CLI design doc after this meeting!
- Steve
- lots of PR reviews
- working with @nrb on sorting out CRD backup/restore issues
- ### Discussion Topics
- [steve] do we want to release a v1.3 soon-ish with the CRD fixes and the other [~70 commits](https://github.com/vmware-tanzu/velero/compare/v1.2.0...master) in master since v1.2, and push the CSI integration to v1.4 (can still keep to an end of March timeframe if feasible)?
- thinking here is that it's easier to release everything in master than cherrypicking a few things, and we have a decent # of commits built up already, which I don't really see a reason to hold back
- [Antony B] - What about https://github.com/vmware-tanzu/velero/issues/2069 ? Is that still 1.3 or 1.4?
- We'll review and see if we can fit it in for 1.3 or have to move it.
- [carlisia] Do we want to move https://github.com/vmware-tanzu/velero/pull/2214 (migrate to go 1.13 + modules) to "Ready for review"?
- This doesn't look like it'd be part of Velero: https://github.com/vmware-tanzu/velero/pull/2119 ([design] Proposal of triggering backups based on Kubernetes events)
- ### Contributor Shoutouts!
- @cpanato - CLI auto-completion docs (https://github.com/vmware-tanzu/velero/pull/2208)
- @Sh4d1 - bump restic version (https://github.com/vmware-tanzu/velero/pull/2210)
- 9 participants
- 35 minutes
2 Jul 2019
In this meeting we cover the road to 1.1, and Carlisia does a great lengthy demo of several Velero features, check it out!
As always, for updated community info head over to https://velero.io/community and follow the links.
As always, for updated community info head over to https://velero.io/community and follow the links.
- 7 participants
- 42 minutes
18 Jun 2019
During this meeting we discuss the CSI prototype plugin, read-only storage locations, and restic improvements.
Keep a tab on the community calls here: https://github.com/heptio/velero-community
Keep a tab on the community calls here: https://github.com/heptio/velero-community
- 6 participants
- 38 minutes
4 Jun 2019
Topics of today:
- v1.1+ planning
- restic improvements
- testing improvements
- v1.1+ planning
- restic improvements
- testing improvements
- 7 participants
- 31 minutes
21 May 2019
Velero 1.0 has been released!
Version 1.0 release blog is published here: https://velero.io/velero-1.0-has-arrived/
Version 1.0 release blog is published here: https://velero.io/velero-1.0-has-arrived/
- 4 participants
- 3 minutes
7 May 2019
The Velero team talks about the v1.0.0-beta.1 release, rustic integration moving to "beta", stabilizing the Helm chart and more.
- 4 participants
- 13 minutes
16 Apr 2019
In this community meeting, the Velero team talks about the first alpha of version 1.0.0, super exciting!
- 6 participants
- 34 minutes
2 Apr 2019
Velero Community Meeting - April 2nd, 2019
First Community meeting as Velero!
This meeting covers work that's been done in the past few months, the roadmap for 1.0, and community efforts and contributions.
Meeting information and notes can be found here:
https://github.com/heptio/velero-community
First Community meeting as Velero!
This meeting covers work that's been done in the past few months, the roadmap for 1.0, and community efforts and contributions.
Meeting information and notes can be found here:
https://github.com/heptio/velero-community
- 7 participants
- 51 minutes
27 Nov 2018
This design session will cover our plans to drive Ark to 1.0. Everyone is welcome to join to give us feedback on what they'd like to see in 1.0. The focus for 1.0 will be bug fixes, correctness, and API stability and forward compatability.
https://kubernetes.slack.com/messages/ark-dr
https://kubernetes.slack.com/messages/ark-dr
- 9 participants
- 57 minutes
9 Oct 2018
Join us here and on Slack for an open question and answer session with the Heptio Ark Developers. All users and expertise levels are welcome! https://kubernetes.slack.com/messages/ark-dr
Full notes at: https://github.com/heptio/ark-community/blob/master/meetings/2018/qa/1009/README.md
Full notes at: https://github.com/heptio/ark-community/blob/master/meetings/2018/qa/1009/README.md
- 7 participants
- 34 minutes
27 Sep 2018
This session will cover design specifics for the snapshot locations feature in Ark. Everyone is welcome to join us here and in #ark-dr on the Kubernetes slack.
- 6 participants
- 39 minutes
13 Jun 2018
Feel free to bring your questions to our development team as they answer questions live from the audience. All skill levels are welcome. We will also cover new features in development
- 4 participants
- 51 minutes