VMware Velero Community Meetings, 22 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Velero Community Meeting / Open Discussion - Sept 22, 2020

Description

Sept 22, 2020

Status Updates
nrb
Documenting plugin release process
Need to get Azure and CSI plugins released
Stop trying to push docker images on forks
Reviews on main repo.
Thinking on ways to make Velero more concurrent/faster as part of v1.6; no concrete suggestions yet, I owe a high level goals doc per discussion last week.
Maintainers please take a look at https://github.com/vmware-tanzu/velero-plugin-for-csi/pull/70; needed for the next version of the CSI plugin
carlisia
Last week worked on v1.5.1 release
Reviews
Piling up: more reviews, GH issues + Helm chart
working on the CLI side of the download request migration to kubebuilder/controller-runtime.
bridget
Worked on some improvements to our release scripts following the 1.5.1 release
Now looking at internal build processes
dsmithuchida
Resource blocking in vSphere plug-in for Project Pacific internal resources
OpenSource project approvals for GVDDK (currently part of Astrolabe) and Data Generator (Kibishii) test tool

Discussion Topics
nrb: Defining prerelease to GA timeframes
From last week:
Prereleases ~1 week between each?
RC to GA, ~2 weeks?
Phuong - 3 month release cycle is acceptable for them, waiting for features on that timeframe is reasonable. Longer is too much.
The RC was helpful to integrate and test with their product. It meant updating to the actual release was just removing some characters.
1 week seemed reasonable to them, but if they hit bugs it may not be enough time.
Dylan - Red Hat’s Konveyor lags a little. They want to support older Kubernetes releases like v1.7 due to supporting OpenShift 3. (this isn’t determined for upstream Velero)
Konveyor does OpenShift 3-4 and 4-4 migration.
Red Hat also has OADP for OpenShift data protection, and their Velero fork is used there, too.
Red Hat likes the 3 month release cycle, especially for OADP. Backwards compatibility in Konveyor is trickier.
For OADP, they tried the RC for basic tasks.
poojita: Recover OpenShift’s native resource: DeploymentConfig
Velero restore error seen has been captured below:
Velero failed to restore namespace frank3. {“namespaces”:{“frank3”:[“error restoring imagetags.image.openshift.io/frank3/httpd-example:latest: ImageTag.image.openshift.io \“httpd-example:latest\” is invalid: spec: Required value: spec is a required field during creation”]}}. Velero restore ‘51685651-5369-51e1-88df-8977874919ca-2020-09-16-08-53-48-frank3’ failed: {\n “phase”: “PartiallyFailed”,\n “warnings”: 6,\n “errors”: 1\n}."
Dylan - There’s an OpenShift plugin that can help restore these CRDs on vanilla Velero installs. https://github.com/konveyor/openshift-velero-plugin
For ImageTag, it skips restore of these
Plugin recreates ImageStream, which then recreates the ImageTag.
ImageTag is a new, undocumented resource in OpenShift v4.4.
The plugin is used in the context of the migration product (Konveyor) and data protection (OADP).
Can file GitHub issues on that repo if you have issues/questions.
OADP bundles Velero and the AWS plugin on OpenShift. The benefit here is that it backs up the images to the S3 bucket.
Plugin can’t do it by itself right now. OADP sets up a image repo that the plugin doesn’t orchestrate right now
Alay - Plugin dependencies are challenging
If you deploy Velero by itself, without a wrapper, there are challenges in connecting plugins to external systems such as StatefulSets or Deployments.
Dave: Difference between the vSphere plugin and OADP is that OADP is at what point in the lifecycle they’re active.
May be able to extend plugins via the Velero app operator that VMware is working on.
RH’s requirements:
Before the plugin runs, ensure the dependency is healthy
During/after backup, ensure the dependency is healthy.
If it’s not, short circuit the operation instead of trying the full backup operation and failing.
Narashima - Took a backup w/ Velero and tried to restore it to another cluster. ReplicaSets/Deployments are getting duplicated in the new cluster.
kubectl get shows multiple entries.

Shoutouts
Slow week for contributions, but thanks to mikkael for the PR to allow users to change the container’s timezone

A

Good morning, good afternoon, good evening, everyone and welcome to the valero community meeting and open discussion today is september, 22nd, 2020 and per usual we're going to start out with some status updates. First up, we have nolan.

B

Yeah, so um I've got a couple prs out the first one, a little bit larger is pulling our our plug-in release process out of our brains and putting it on uh virtual paper. um This applies for all of our plug-ins, except for the vsphere plug-in, uh which isn't um maintained by a different team inside of vmware, who has a lot more vsphere expertise, um but um I think I think the majority of the process is documented. There, I'm still trying to figure out why um some internal links are not rendering in hugo.

B

For some reason uh short codes aren't working. um I tried a bunch of different stuff according to their docs, I'm not getting it to work, so I may just abandon it and reference the page without an anchor, um but uh was documenting that so we can get the releases out following 1.5.1.

B

um The most important plugins that I think we need to get out are azure and csi. I know those both have some fixes that need to get out um the next pr I have there is um uh it's a small fix, but it's been annoying. I know some folks have been hit by this and I get I get emails for this too.

B

um When you push to your main branch on your fork, the build process would try to push to docker hub and nobody has permissions to push docker hub, except for this vmware tonzo slash valero uh rebo. So um I made this change to explicitly um skip that step so that people aren't getting bogus failures, so this has been tested against my fork and shouldn't have any bearing on changes in actual in this valero repo.

B

um Besides that I've been trying to get get into reviews and issues on the main repo um I've been doing community support as well. um I do owe some uh reviews on the helm, repo. I know I've gotten some pings on that.

B

um I've also been brainstorming on uh ways to make valero, either more concurrent or faster as part of the v16 1.6 release. I don't have any concrete suggestions on this right now.

B

I know we talked a little bit about defining high-level goals last week, um so I owe that document still, um but I'm looking for ways to possibly make things faster without a complete overhaul or uh what we can do for um ways to make it more concurrent, basically looking for variety of implementation solutions, not just one um and then jumping, I guess back up to the top. um This is a request for the maintainers to look at this pr um from ashish.

B

Oh yeah, I need to get the semicolon off of there. um This is basically updating uh to use uh some new rustic apis that were included in 1.5.1 um in the csi repo. So if we can get this reviewed and merged, then we can include it in a new release for the csi plugin.

B

And I think that's most of what I have.

A

Awesome any uh questions, comments for nolan.

A

All right uh moving on to.

C

Calicia hello, so last week we still spent. um We worked as a group, one one 1.5.1 release.

C

There were reviews, and I am aware that I give more reviews and github issues and definitely help chart work in reviews, and I'm also trying to get this download request migration out.

C

I have to finish the cli side of that. That's it for me,.

A

All right any questions or comments for kalisia.

B

Any any help needed on the um the download request. One.

C

No not so far, okay, but maybe.

A

All right uh next up, we have bridget.

D

Hi everyone. um So, as it's already been mentioned, um I also helped out a little bit with the 1.5.1 release and primarily there were a few issues um that we encountered during the release process, with some of the documentation and also with some of the scripts, and so I have some pr's open um to help fix up some of those issues that were fined I've.

D

Now I've been spending some time looking at some of the internal build processes that we have um at vmware, um just some other um things that we're going to have to do within the 1.6 time frame just to improve um some things internally. um So I'm going to be looking at that and automating some of those processes, because I think right now, it's all very um manual is my current understanding.

D

um With regards to some of the the things that I've been piling up, carly you mentioned in nolan as well, and with the uh reviews and github issues and things um I haven't been doing a lot of the triage on the issues yet so I think I will start trying to get them involved there, where I can so ping me if there's any help needed I'd, be happy to help.

D

A

Sounds good and we're also doing something fun on thursday.

D

Yes, we are, uh I didn't know if you were going to mention that later jonas or uh I'm looked up far during the dock. Yet, um yes, I'm going to be helping out with uh the upcoming uh office hours we're going to be starting on thursday, so I'll have some prep work to do. For that um sorry, I'm I haven't looked to see if there's a a discussion topic for that. No.

A

There's no discussion topic there, so I I just thought I'll mention it so yeah on thursday, at 1 pm, eastern 10 a.m, pacific and other times on in central and mountain time. Of course, um we're gonna host the first ever valero office hours, uh so the office hours is a little different than the community meetings, so the community meetings where we do status, updates and discussion topics here.

A

um You all know, what's happening in these meetings, so in the office hours, it's more an open forum for you as users or contributors to valero, where you can ask questions like how do I do x in valero, or how do I configure my backups on this specific cloud provider? How do I do this? How do I do that, and also how do I start contributing to valero?

A

uh How can I, how can I um improve my my local dev environment um when, when I want to help out and contribute to valero, so things like that more open forum, and we will have a few topics each week uh that might get punted to to the next week? Depending on how many questions and how many discussions we have um so, the the plan here is to start at one and we'll finish up uh at 3 p.m.

A

If there aren't any questions or anything like that after, like an hour, we'll we'll cut a little short, but anyone can join in and just ask questions. As I said, it's an open forum. So please join in and chat with us at 1pm on thursday.

D

Looking forward to seeing some of you, there.

A

Yeah, it's going to be super fun, so it's going to be run by bridget and ashish.

A

Any questions on the office.

E

B

All right, this is- I I think you may have mentioned this to jonas sorry, but this is bi-weekly right.

A

Yes, so we're doing it uh twice each month uh so on the second and the fourth thursday. uh If you are a member of the valero google group, you should have gotten an invite. If you haven't gotten an invite, uh please let me know: um we've been, we've been having some issues getting invites through in the google group. So if you haven't gotten an invite yet please ping me on slack and I will make sure that you get one.

A

All right uh next up, we have dave.

E

Hey so, um let's see so working on the vsphere plugin and one of the things we're doing is blocking some of the resources that are internal to the project pacific supervisor clusters. I put that code in basically there's some resources that if you set them without the um the system actually being aware of upsetting them, uh they break things so we're just blocking those from backup and restore because you can't really restore them um and.

F

E

Kicking off a couple of open source projects, um these are existing projects that we're trying to bring out so gvddk is the wrapper or the vsphere uh virtual disk development kit. It's a tool kit for uh extracting snapshot data and uh restoring volumes.

E

um We've got the code, it's already open sourced, but it's been part of the asteroid project and we're kicking it out as its own separate project. It's probably not uh hugely interesting and then, um as part of our testing effort. uh We have a data generator, that's a synthetic test for uh backup restore, so it will generate data and then um verify data. So we use that as part of our internal testing, we're going to start pulling that out uh into the valero and end test. So we're open sourcing that.

A

That is great, more open source, goodies.

B

And dave does the the the vsphere plug-in test currently used could be issued right now right.

E

They do among other things but yeah, so the kibishi is um just basically generates a bunch of files knows what's in them and then can verify them afterwards, and we can do that scale out. So you can run n replicas. You know to generate a hundred gig or a terabyte each and scaling up and down it's all. It's a kubernetes app.

B

Okay and it's for the that's for the contents of a persistent volume, not necessarily the contents of the kubernetes api cd.

E

Yes- and I think that's actually something we need to talk about is how to um verify the contents of the api server. I was going to check and see if we already had some stuff there, but.

G

Yeah not not right now, um but so I think we should add that under this, as long as we've got this test tool, we'll put that in as part of it.

A

Cool sounds good, any other questions, comments for dave.

A

All right and we're moving on to the discussion topics. uh No one you got the first one.

B

Yeah, so this was a follow-up. From last week um we had the discussion about um spacing between pre-releases, the rc and ga or nrc nga. um I I put um some sample timelines that were thrown out last week, um but the actual discussion uh we want to keep the discussion and decisions uh documented here in this issue.

B

um Excuse me, like I, put in this comment: I'd like it for us to make a decision by this friday uh september 25th for the one six 1.6 time frame, um that's not necessarily binding forever and ever, but it's what we'll go with for one six, so that we can try it and iterate.

B

um My uh my thoughts were were looking at roughly a three-month cycle. um Pre-Releases don't have a set time to start within there, um but it's it's about when we're starting to get uh features roughly complete, but maybe they have some bugs um and we should be releasing. We should feel free to release beta's uh as we alphas are betas as we need uh so I'm I don't want to set an arbitrary restriction on just one uh because we should be.

B

uh I think we should feel free to release them in response to feedback uh both bugs and people. Saying: hey, wait! A second. This feature doesn't work uh like this design is flawed.

B

And then for uh spacing, I think beta is one to two uh one week between uh and then once an rc is cut, I'm I'm open to discussion, but one to two weeks, uh the final final one to to be determined, uh but once an rc is cut, we can say one to two weeks until the ga is released. So just summarizing my position there um looking for feedback from other folks, especially people not on the core maintainer team, uh like what works for you and.

B

If you could uh share that here and we can get it captured in the issue as well for posterity that'd, be.

A

A

So any any comments on this: do you think it sounds like a good.

A

A

Not a lot of discussions for a discussion topic.

C

I I agree with what nolan proposed and I just added my comment. There awesome.

E

Yeah, I guess the question for the community members is: does this like three-month release cycle? Is this something that works for you or you know? um Would we want something different.

H

um From the right, this is from, uh I think it's good as well. First, because we also have about three release cycle in our company and, for example, if we I'm waiting for some feature that we want to enhance the valero like recently, we have a order resource right. We only need to wait for that three months so that we can incorporate that into our product a bit longer than that. We will have to push our feelings and waiting for that too long. So three months seem to be matching with what my cycle of releases.

H

I'm just speaking from my my personal experience, I don't know if any other company they have the same. You know psycho release, but three months seem to be reasonable. For me,.

B

Yeah, that's exactly the sort of feedback we're looking for um not to well yeah to put put folks on the spot like um dylan or alay, or anybody else from red hat. Like I know you, you folks build on valero a lot, um and I noticed uh conveyor is still on. 1.3 um is three.

F

Months too much for you or is yeah, uh I can jump in there. We actually just upgraded one for so um it's kind of weird. If you look just at the version strings in our conveyor fork, because um we we have a release, our release. Cadence is actually in one three currently, so you may be seeing one threes in places where it's actually um referencing our conveyor release, um but we did just upgrade our valero four to one four.

F

We are moving a little bit slower because we do have a um for example. We actually met this morning to talk about um what was the timeline for starting to get one five tested and merged in, and um we have a pretty weird requirement that we are supporting some pretty old kubernetes versions. So um we just have to do some pretty extensive testing to make sure um we can actually, especially like okay controller runtime.

F

um I I don't think you guys go back to support one seven with the control road time library and we have that requirement today. um That may change, obviously, but um so so we're on one for now and I think the plan for us is um we really wanna start testing one five um with all of our testing suites as soon as possible, since we're kind of concerned about that one.

F

So um it did take us a while to go from one three to one four, but I think the change to one five will be faster for us.

B

Yeah that um that kubernetes release dependency is uh worth mentioning and that that's something we're discussing internally um just for transparency. This is something that valero is probably going to sink to the um vmware tons of kubernetes grid um decisions on, but that's not set in stone yet, um but that's uh that was a good reminder. I didn't remember that uh conveyor went back to 1.7, but I did know that at one point I just forgot it.

F

Yeah, I mean really because we have so many um folks using openshift three, which is just they're, still running old versions. I had no desire to upgrade so um we have a pretty far back dependency.

B

Tree and conveyor um conveyor is meant to be a uh open shift three to openshift four migrator. Is that correct, or is that a misclassification um that.

F

Is how the project started? We now um want to support four to four migrations, um but mainly yes, it's meant to do through, um probably will we're trying to hopefully extend it to support a bunch of four to four use cases, so um it could change, but that's accurate for now yeah, okay, we also um we also consume.

F

So we have this other project that um it's currently called oadp. It's essentially a data protection solution for openshift and we use valero for that, um so the conveyor valero fork. It's actually used not just for um sorry, I'm throwing a few different things out here, but there's a migration tool which is meant to migrate workloads from three to four. um Then we also want to just support, backups or stores on openshift itself, um so that conveyor, valero fork is used there as well.

F

So um that's definitely helped drive us to stay on the newer valero versions, but we also just allow customers, since it's just an upstream community project right now. We allow folks to use the valero images too. So we have oedp up to one five and stuff.

E

E

A

That's helpful.

F

So I I mean I guess, to bring this uh back to your original question. The three-minute three-month recycle is actually we we like it, especially with the adp. Since it's an upstream thing, we want to stay with the newest, you know lightest and greatest with valero. It just gets trickier for us with the migration toolkit um with the version dependencies. So um I don't think I think three months is fine. I guess I'm curious how much part it is for you guys if you, if you find that to be too fast, but.

B

I just personally, I think, three months, even six months in an enterprise setting is super aggressive uh and it's not valero that it's not unique to valero um upstream kubernetes. Has this problem too. um Openstack had the problem, um so uh it's not unique and it's not new, but in in enterprise. I, like enterprise, like you said, you have customers using openshift three kubernetes 1.7 in in in the enterprise world kubernetes one seven is fairly new, um based based on uh their their reckoning of time.

B

You know it's newer than mainframes, um so, um but that's good to know. I I I think um my perspective is, I think three months is good for getting new features out uh and in this cloud native world, it's all of us are helping our customers adjust.

B

What what their appetite is for upgrading is gonna come down to negotiation there. That vmware has has uh customers that want to stay on older stuff too. It's the nature of enterprise.

F

Yeah that makes sense yeah, I'm happy to you, know, chat more about our use case. If it's interesting and related to this, just let me know.

B

Okay, well, those those data points are helpful for me. um If, if folks, if there aren't any other data points for people to add, then I think I'm I'm willing to move on. I I just wanted to get a little bit more feedback from folks outside the team.

H

I just wanted to add a little bit. I just we in the last few days. The 1.5 is really helpful for us in terms of integrating the release 1.5.1 into our product, because when we have that uh release 1.5.0 rc1, we can, um you know, take out all the compiling problems with upgrading and so on and so forth to play it. So when you guys release the 1.5.1, I can only switch one five one. You know one one small text in the palm fire. That's it I mean everything goes smooth.

H

So I really appreciate that rc1 release, at least for my personal experience with it.

E

What's a good time frame for you, then, between you know between the rc release and the time we we go, ga so we're thinking one or two weeks. At this point.

H

The one week seem to be reasonable from my point of view, because we have all the time needed to drop to smooth out all the roughness.

H

However, if we hit some bugs in the rc-1- and there might not be enough time.

E

H

Fix it in the village, so that is something that I'm concerned, but but very fortunate that we do not hit any problem with this rc1. So we can do the 1.5.1 very smoothly.

B

And dylan did did: um did your team use the rc uh with the one five.

F

um So for oedp I have installed the one five rc's um images for my valero install, but I can't say that I'm doing complex stuff with it, it's more just hey, install valero and back up a wordpress app for a demo. um We have not pulled in one five at all for our migration um product, so.

B

Okay, that's fair all right and the rc's the rc was kind of a new thing in one five. So that's totally fair.

A

All right any other comments on this one.

A

Okay, I think we can move to the next one. uh We have pogita here. Who has a question you want to unmute yourself.

I

Yes, uh hey everyone. um This is pojita from dell technologies, so we recently um found out that um when we have uh an open shift native resource deployment config in a project um which has uh involved one replication controller and we are trying to perform a restore to a new namespace, we happened to get this error from malado, which seemed to trigger a thought whether we support this resource uh with respect to valero restore and the valero version that we used was 1.3.2.

I

So I just wanted to bring this up in this forum and then ask if we support this resource for restore.

A

Anyone wanna chime in.

B

I'm actually trying to find um I'm not sure if this is a valero error or if this is something that could be resolved by an open shift plugin. So I mentioned conveyor.

F

So I can actually jump into your normal if you want yeah, okay,.

B

F

We've dealt with this so um image tag is a new resource in openshift44 um and it's something that we had to basically add some object into the openshift plugin. I would recommend, if you're using vlair with openshift, we have tried to get to a point now um I'll paste it in the chat. um We do have an openshift plugin that the locations change a couple times, so I'm not sure if I still haven't updated one on the blur docs itself, but I pasted it in the zoom chat.

F

This plugin is meant to be used with um vanilla, valero installs on openshift, and then we also use it from our migration product as well, but um this does have an image tag plugin in here um for the current version of it. I believe we simply skip the restore of it, because um image tag is essentially a secondary virtual resource in the cluster.

F

um So what we do instead is we recreate the image stream tag resource which automatically gets the image tag resource to get recreated, so it's just kind of a weird little workaround we did, but um image tag is not even really documented. Yet it's like it was like a new resource added in 4-4 that is not really being used yet so um we simply added a restore plugin to skip it. um But if you are interested in you know especially restoring deployment configs, we have a deployment, config plug-in that should do the job.

F

um It's not very complicated. It's basically just handles the name, space mapping um and then checks. If you had any internal image references and it does a swap for you um so yeah I mean we're looking for people to test this out. We really only used it in the context of oedp and our migration um products. So um if you are interested in running low on openshift, I think this would get the job done for you.

I

Sure, uh thanks I'll take a look at it.

H

Thank you, and uh can we contact you apply to um continue on that uh all right I mean.

B

I think you cut out their phone.

H

F

There a question: sorry.

H

I'm gonna miss it. Oh I just say.

F

H

Contact you offline to continue on it because positive.

C

H

Are on on the same team at emc, their vmc here and we we are, we might uh contact when you contact you off by and continue on that openshift, better operating.

F

Yeah, no, absolutely um you can contact me on the community slack or um if you have an issue with that plugin itself. Specifically, you can file a github issue there too, and we'll respond there. Whatever works all right. Thank you. I'll contact you, via slack.

H

Right all right! Okay, thank you. Thank you.

J

Oh, hey guys, uh I'm alive from reddit um really quickly just wanted to um ask a question: um can we probably use oadp here because by default it will come with uh the plug-in installations right.

F

Yeah um sure yeah we can bring that up real quick. um So let me let me get another link for you. If you are interested in using the letter on openshift, we did try to package it up in an easy to use operator. um So here is another link in the chat. um This is this thing odp I keep talking about. Essentially it allows you to install this operator, it's published on the community operators in openshift.

F

So if you're using openshift4 and you've enabled the community operator source, you can basically just click on odp install it create the expose the valero api itself.

F

um But then you create a custom resource that will go and install the openshift plugin, we'll install valero with the object, plugin and aws plugin by default, um and then it's it's fairly pretty easy to get started with valero using this mechanism. But if you're already comfortable installing it, you can just add the plug-in yourself as well.

F

The the benefit of using something like this odp operator is, um if you want to back up an application on openshift that has internal images, so um images that exist in the internal registry, then um this operator is going to set up all the bits needed to actually get the image contents in the s3 bucket as well.

F

So that's kind of why I'd push you in this direction, but if you already have a workflow with the valero installed that you know how that works, you know this will work on that as well.

F

But please reach out to me. If you have any questions about any of this, I'm happy to kind of talk through this on slack.

F

I'm a dui murray on slack two fedex.

A

Awesome, thank you dylan and thank you fujita for for bringing this up.

A

All right does anyone have any other discussion topics that they want to bring up on the meeting today.

B

uh I have a real, quick question for dylan or in la does oadp. um I'm sorry. Does the valero openshift plug-in um back up the images to the s3 bucket? If you do not use it with oadp or does it uh not? Is it not able to do that? It is not able to do.

F

That and the reason for that is, um we basically have to stand up a temporary registry pod in the cluster that is backed by the s3 bucket um and that orchestration has to happen outside of the plugin. So um we will simply skip the internal image backup, if you don't have that registry enabled. So if you do it outside odp, you just get everything except the image. Backups.

F

Well, let me clarify that's how it's supposed to work if it doesn't work that way, please let me know.

J

Well, well, the learning is um looking that up. um I we've been uh working with the uh plugins on transferring the images, and uh our experience is that um this is kind of an improvement area for well paler plugins in general, because there is no way for us currently to specify a dependency of a plugin and have on the plugin code handle it.

J

Fortunately, in all our products, we have uh other agents that can deploy this dependency for us, so we get away by uh depending on them, but, uh for instance, if you are deploying valero um by itself, you no longer have that other agents to install those dependencies and that's where this fails.

B

So I think I'm I'm not quite understanding that. So if, if you're saying, if the plug-in depends on say other pods or daemon sets or staple sets, then um deploying it with valero by itself uh you're having challenges doing that. Is that what you're saying.

J

Yes, so yeah, so the way we uh orchestrate this is, we would set up a registry deployment um that will dump the images from your from the openshift internal registry into the um into the deployment that we have set up. So that's how backup will work uh and on the restore it will be the other way down from our registry. It would be dumped into the um the openshift registry.

B

Right: okay, I'm I'm curious about this because it sounds like this might be similar to how the vsphere plug-in works. um The vsphere plug-in has a separate, I believe, demons. I might be wrong there. It might be.

E

We have all kinds of bad things going on.

B

Yeah, but it has, it has uh what what it's called uh data manager that comes like in sits beside valero, but uh we do have I'm trying to think what the installation process for that is like it's. Basically, that yaml has to be installed side by side in the plug-in.

E

uh It happens in the init container right now. Okay, so uh when we init when, when the init container runs, we uh install the plug-in itself, we copy the plug-in over to the plug-ins directory and then we go ahead and we create the yamls for the data manager and now the backup driver.

E

But those are strictly things that we use internally and they're very tied inside the plug-in, whereas this sounds more like we're.

C

E

Taking care of like kubernetes uh images and stuff, which uh you may even need to be able to get them before you boot, valero.

J

um No, I I I'm not sure if I'm able to follow um how we should we would be able to get that dependency before we install error right.

E

That's what I'm saying yeah so we we are, our dependencies are internal to us, so we need these things for us to be able to restore, but valero has to be installed first, in order for our plug-in to get called, so our stuff is strictly moving uh volume data. At this point.

B

Okay, that makes sense yeah. I was wondering where the where the difference was between the vsphere plug-in and like this 080p um stuff, but it sounds like it's it's the point at which you're trying to to inject into the life cycle.

E

Yeah now we we are developing this operator that uh installs valero and that's really for uh initially getting things into the project, pacific environment, but we're planning to open source this, and I think, we're planning to make this a more general valero um feature, and I wonder if uh it makes sense to have plug-ins to the operator.

E

Maybe that would be helpful for this, so we can consider that in the future.

B

Yeah I I yeah. That may be um sorry I just wanted to. I wanted to dig into that just a little bit and see if, if there was any way, we could um pull these things together, but it doesn't sound like at the.

B

J

Yeah, so from our perspective, the um for our dependencies, we have two uh distinct um requirements. Right first is before the plugin runs. uh We need to make sure that the dependency is healthy and also during a backup and restore. We need to make sure that the dependency is healthy um if during uh a backup or a restore, the dependency is uh not healthy.

J

uh We kind of, like short circuit short circuit, that uh instead of timing out, we are planning on um just declaring that this migration or key store backup resource will not succeed, because we know that the dependency is not healthy. It would be awesome if we could get this requirements into um valero itself, so that, um like we can, um you know valero can stand alone as a standalone product valero can handle these things. um Yeah.

B

Okay, yeah. That makes sense so like uh to use an example: that's in in valero um like if, if rustic isn't even running and you need to have rustic in the back of, we should fail the backup because rustic's not running so. This is similar. This is a similar situation.

J

Yes, I think so.

B

Okay, that's that's useful to know.

K

Can I ask one question regarding this: volatile backup.

H

K

Oh, I took entire cluster back up by using only row, so while restoring this cluster into this new cluster means migration, so I'm getting replicas, it's repeated as well as the deployment apps also got repeated things.

B

K

Did you repeat the question, so um I used only row to take cluster migration from one cluster to another cluster.

K

While restoring this backup, what I took entire cluster backup, I get some replicasets and deployments getting repeated.

K

So if I have a replica sets of four it may some duplicate replicas, it's got created by using polyureous backup and one more question I have to ask regarding this valero is that valero will handle this deployment process. What I used in my cluster means, if I will take entire clusters back up. If the migration is going to happen, it valero will uh will have the same deployment process to bring up this bring up the cluster.

B

So for the first one, so what you're saying is you deploy it and you you do the migration and the deployment and replica set comes back with like more pods than what it originally had.

K

No, I don't get any more parts, but replicas it's it's showing that duplicates.

B

Oh okay, so there's more than one entry when you do like uh cube control, get.

K

Replicasets as well as deployment absolute, so I can get duplicates here in deployment apps, I used config maps as well as routing routing image, means uh config micro services and routing micro services. So in that scenario the deployments also will get repeated and the same thing replica set also will get a duplicates.

K

It means finally, the deployment apps and there are replicas at apps getting duplicated.

K

And I'm getting the book validation failed. Maybe it didn't follow this uh same deployment process, while migration.

K

H

K

In email itself, I didn't get any response.

B

Oh, did you post this in the google group.

K

Yeah in google, a email gmail, I think you can.

B

K

B

Yeah, okay, I'll take a look at the google group message um and look at the errors in specific, because I'm not I'm not sure I can answer this live. um But I I will take a look at the google message and respond to today.

K

Okay, send email itself.

K

Yep, I will do that sure.

K

I'll come up with the same question next week and with this complete error log, what I got today, I just removed my entire cluster because of this repetends. I will show you that in next week,.

B

Okay, well, hopefully, we can take the discussion to the mailing list be easier to to uh troubleshoot this with actual logs and output. Okay, sure.

A

Sounds good, and this is also something that could be taken up during the office hours that we have on thursday at 1 p.m. Eastern. So, if, if you want to bring it up, then you can also do that and then we can take a little deeper look into.

A

It all right uh any uh any last discussion items anyone wants to add in for.

A

Today, all right, then we'll dive into shout outs, and we only have one today, nolan.

B

Yeah, so we didn't uh um didn't, have a whole lot of prs that got merged across the projects from uh external contributors in the last week or so. But I wanted to send out a thanks to uh mikhail okay, michael uh for this, quick pr that um it's not merged yet. But uh he submitted this pr for um injecting the tze data package uh into the.

B

Container image that that's the word and allowing the user to set uh their um their time zone. So uh thank you for submitting that.

A

Awesome and uh with that I'm gonna close up the meeting for today. Thank you. Everyone for joining have a fantastic rest of the week. Please join us on thursday for the first ever valero office hours uh and uh we'll see you all next week have a good folks.

D

B