Kubernetes Storage SIG Miscellaneous, 27 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Storage - Volume Populator API Design Meeting 20210427

Description

Kubernetes Storage Special-Interest-Group (SIG) Volume Populator API Design Meeting - 27 April 2021

Meeting Notes/Agenda: -

Find out more about the Storage SIG here: https://github.com/kubernetes/community/tree/master/sig-storage

A

There we go okay, hello and welcome. This. Is the uh sig storage weekly volume populators meeting?

A

um So I uh I have a few things to report for the the last week and um a few questions to ask for you guys and then I can open it up for anyone who wants to add something else to the agenda, um so I have been working towards getting the alpha releases out, which is going to involve uh getting all the you know, code, review, comments, updated um and getting, but more importantly, getting release tools integrated into the various branches and I kind of screwed up the way I did release tools with um with one of the repos.

A

uh So I've been sort of untangling that that mess- and I think it's it's basically, I have a solution now. um So I just have to get. You know, push a bunch more prs into each repo and get the release, tools, updated and then do the release. And then we can move on to uh api reviews for 122, which we found out yesterday. uh The schedule for and it's incredibly tight right shang.

A

It's like we have two weeks until the next feature: freeze or yeah, or enhancements, freeze, yeah, so uh there's basically almost no time to uh to get the apis reviewed again, so I'm gonna be contacting uh tim in parallel with all of this work um to get at least get something on the calendar, so we can hopefully get his uh his nod for getting this merged, we're getting the the kept changes merged and then we can move into 122.

A

which, which we also heard yesterday, um is going to be a a four month long release. Unlike you know the original plan, I think, which was three months long, so I think we've permanently moved now into four-month releases. So at least we have some predictability around that.

B

Yeah, I think, there's more time to like code, a code merged, but the enhancement that time period seems to be shorter than before for some reason, maybe just because they made the decision later or something.

A

Yeah, I think that there was a sort of gap week between the end of 121 and the beginning of win 22, while they were maybe maybe even two weeks while they were figuring out the schedule, change and they've only lengthened the whole schedule by four weeks. So if, if two of those weeks goes to code development and one of those weeks goes to testing, there's only you know an extra week to lengthen the enhancements period, so I totally get why they did what they did um makes sense to me.

A

um But uh this this brings me to uh the main problem I'm facing. So I have a lot of pr's. I want to merge, um as I was going through and doing some of the cleanup around the volume populator library I discovered, you know, I'm missing a bunch of logging there's some obvious performance enhancements I could make. Is there's just like a bunch of changes that I wanted to to slam in, and I don't know if I can expect rapid code reviews.

A

um So what I? What I'm wondering is.

B

A lot of I saw- I was just going through this before this meeting. I there are two pr's on the volume data data source validator, so I reviewed one and the other one is the release tools. I thought you need to do. You need to make changes in the release tools ripple. First, I'm not quite sure what is the status, because.

A

B

A

Have a so so I have, I haven't, pushed all my pr's. Yet um what I wanted to discuss with with this team is um for projects that are sort of at their early alpha stages. um Do we ever relax the um sort of the the merge requirements and just like uh allow things to merge quickly or.

B

uh That didn't happen with the snapshot at least out there for a very long time. You.

A

Know what we're doing with cozy, for example, the cozy guys are beavering away, but.

B

They do yeah, but they do have a lot of they do. Have people reviewing them right. It's not like you, don't review, then you just to get them in and also that's like the it's like provisional. They have the uh right. So it's a little different. um Oh yeah,.

A

So this is what I wanted to sort out is: do we want to relax the code review guidelines and, if not- and it sounds like the answer is no, then I need to to have at least a few people that I can sort of ping for rapid code reviews for small things and just sort of get them in um because, like.

B

Small things should be fine, I mean it's just the big ones, like the you know, like your initial, commit that huge that.

A

B

Takes a long time right so um so I think I reviewed the. I read the one right. I still need to do the other one. Okay, so.

A

The volume data source.

B

uh You have a you, you address the comment. I just reviewed that one okay, um thank you yeah, so I need to review the this other that this other one, but it's because it's big. So that would take some time. But if it's like released to us, then you should really pin like patrick. Did you ping him to take a look.

A

I just have to push the branch. The branch is going to have a series of commits, because what I had to do is I had to delete the whole release tools directory then yeah.

B

So you can separate your it'll be better if the you get each pr smaller. That will be easier for people to look at so don't don't mix your release. Two changes with your like functional code changes.

A

Right right, but, like I'm saying even the release tool changes is like a five commit series, because the first commit deletes everything that was there. The second commit pulls into new release tools and the third commit well actually the pulling a new release. Tools involves two commits: there's a base commit and a merge commit and then there's another commit that like adds the the cloudbuild.sh and cloudbuilt.eml files. So there's like there's a series but yeah like that's a self-contained release, tools, update um that can merge by itself.

A

uh There was um uh so so on the other one, the the lib volume populator. um I have like a series of pr's that I haven't pushed yet that mutually depend on each other, because I implement one and then I implement another. Then I implement another.

A

Are we allowed to push multiple pr's that are mutually dependent, so they have to merge in a certain order. Yeah.

B

Yeah, that's actually uh yeah, especially if your pr is too big. It's actually good to do that way, so people can look at each one just to separate separate another ways you do you you can do is to if you separate them into separate, commits in the same pr that's another way, but I mean it may be difficult to update. Though the address come like if you, when you address comments, then then they probably will be you know, then how do you uh uh keep them separate and that that that's the challenge of myself?

B

I think it's probably easier if you just just making them into different pr's, I think that's actually easier. So we can just put.

A

It first, but you can have a you, can have a pr with like uh three commits yeah.

B

What I'm saying.

A

Is then, when you address the comments, that's a little.

B

Messy that that was that.

A

Right, I'm hoping that these are gonna be no-brainers that are easy to merge uh and there won't be any comments, I'm hoping because yeah they're, just it's just adding log statements where there are none. Oh.

B

Yeah, those should be fine, because just.

A

Logs and should be fine.

B

Yeah, but if you separate them they'll be you know.

A

Okay, so so so shing. If, if you're willing to volunteer to do like relatively quick turnaround, reviews on small pr's, I will yeah.

B

Small price- I can quick- I just uh revealed this morning that I merged it. I hope it's not. You know.

A

I'll I'll push a bunch of them up then, because I have them sitting in my workspace. I was trying to figure out how to do this, but if yeah, if you just want to push up small individual, pr's and you'll review them, um then we can uh sort of get get the repos in the shape they need to be in to actually do an alpha release quickly um and and then uh yeah.

B

Because sometimes like, for example, like the snapshotter, when we actually asked several people to review the like the controller itself just took too long, that's why it takes a long time for that thing to get merged like and there's the api reviews and for the crds, and then there was a controller review. So it takes a really long time for the initial pr's to get merged to just, but I think the words at least the crd is not not that complicated. It's uh it's easier that way at least, um but I do need to.

B

I didn't- need to look at your initial commit for the for the link.

A

Yeah, okay, um all right.

C

A

C

Comment like uh uh hello, um so I know like I don't have the authority to approve or do something like that. But if you, if uh I'm happy to actually do the cosmetic reviews, if that helps so yeah, that really.

B

Helps actually we, I think it's good to have more eyes right. You know, even if you you know, if you don't have uh you can you can add a label, something if you just say look good to me that actually helps right. So I see okay, there's more people looking at it, it's actually good yeah.

C

And that's where and that's where I wouldn't get. The hang of you know how the things are so oh yeah yeah.

A

C

Anyone do the lgtm.

A

Isn't that a open to all.

B

A

B

Of a six, are you a member of a kubernetes six? Then you can do lgtm yeah.

A

I should just are you a member of kubernetes csi.

B

A

D

B

Okay, yeah, that's okay! If he just write down lgtm, then.

A

Yeah yeah yeah. We would appreciate any eyes on these changes. Okay, so um we unfortunately don't have a dedicated channel where I can just sort of ping everyone but I'll put comments. Maybe in this storage channel yeah.

B

Then you can then maybe other people you know who want to volunteer to review. They can also yeah it's good to get more people.

A

That's good yeah, I'll I'll I'll, put links in the sig storage channel and I'll I'll tag you shang and ashitosh um for reviews.

B

Also, there's also like a pr review section, but I don't know who is actually paying attention to that. The separate it's like for all the kubernetes vrs and it's actually just a separate channel for that.

B

I think part of it is more relevant.

A

I mean the product, the product readiness, one.

B

No, no there's a channel called the pr reviews, so they're actually saying suggest you to actually go there to to pin people, but I mean um oh well.

D

B

Like people in work, the 640 may not be there. So that's that's the only challenge.

A

Yeah, I imagine that's that's for people that need direction, that they have a change, that they want to propose and they have no idea who to talk to about it. Maybe those people will help to help directly.

B

Those are like, usually they suggest you to pin people there. However, I mean yeah, I found I mean I'm not paying much attention to that channel at all. So unless you people just pin like if they just put my name there, then of course it doesn't matter which channel that is, but otherwise I mean. How do I pay attention to that? There's like all over. This is like all kubernetes right, so yeah yeah.

A

But we have, we have a million channels and slack is, is a horrible way of communicating, but it's it's sort of it's it's good enough. I guess it's just not great um okay, so uh thank you, shing. Thank you, ashtosh. I will uh work on flinging all my pr's in there to try to get them rapidly merged and get get uh things in releasable shape uh in parallel with trying to schedule an api review with tim uh as soon as possible.

A

um Who uh so so, aside from tim and me- and uh I don't know shane, do you want to be in an api review meeting or should we try to be yeah.

B

Yeah I'd like to be in the api review meeting as well: yeah, okay,.

A

Okay, so so the mutual scheduling might start to get hairy the more people we are.

B

Yeah, that's that's the thing you probably want to get team. You know just make sure you got tim because I think from the storage side it's you know easier. So definitely yeah.

A

Yeah so I'll work on that and then also.

B

I don't I'm not sure like this, I'm not. I don't know about his schedule. Like the you know, cube kinds next week, so I don't know if he's going to get tired because of that, because a lot of people like me like next week will be difficult, um but I don't know about him. So I'm just saying: that's that's why I'm saying this uh this uh new release schedule is very bad like one week after cubekind's actually really bad timing.

A

Yeah I had forgotten about kubecon.

B

Yeah, but you can check check with him. I'm.

A

Not sure you know.

B

A

I'll see, if I can get on the schedule for this week, then we still have wednesday thursday friday. I guess um maybe he's got a slot um and and this this should be a quick one, because we've talked about it at length. Three months, yeah.

B

I think it's really because he has this uh uh concern about the like. Is that relaxing the data source thing right was that I forgot what was excited was that.

A

It's it's gonna, be a a backwards, incompatible change; okay, because if you, if you put a core kubernetes object in the data source field of a pvc today, it just ignores it. Oh.

B

Okay, so we want to basically fail.

A

B

A

That's the backwards, incompatible change and.

D

I think everyone agrees like.

A

Yes, that is what.

B

We should do.

A

B

We just need to.

A

Get get people to say to actually agree to that, so, okay, so this is this- adds even more urgency to to get on tim's calendar for me so I'll make that my top priority. um As for as for this group, the other thing I wanted to bring up today is around so release tools.

A

I understand you know when you're releasing a container. It has all the machinery to like you know, do a tag and and push yeah.

B

So those should be okay, um so there is a mat. There is some are automatic and then there is also some menu things that you have to do afterwards. um So when you get there, I can show you how, but I mean I don't know if you are probably need certain permissions to do that. I.

A

I've, never I've never been through that path before so, I'm gonna have to read all those docs, but but yeah.

B

So you get your first thing down first yeah and then there is a few things like the image we have to kind of promote it. uh We actually have to submit a pr um in the okay.

A

So this is separate.

B

Yeah there are just so many different ripples. Sometimes I, like I just go blank. I don't know which one is that so it's a there's, a different repo that we have to submit a pr so that the so the images will be promoted because, right now what happens is so I think yeah. So I think it should be because as soon as long as your image get viewed right but there's a release choice, you can automatically get those field and then oh did you set up the in the test info?

B

Did you get those set up? You need those there's a lot.

A

Of stuff that I need to do mechanically, but my specific question is about the library libraries don't produce any artifacts. There's no container.

B

Oh, you don't have to okay right so.

A

So what is the process for releasing a library.

B

Oh okay, so, okay, maybe then we can look at the like for because we do have that client library for for the snapshot. So we just.

A

B

It I think, what do we do.

A

We don't have any images.

B

A

What what the one I was thinking about, in particular, was the the volume populator library that's right.

B

A

External provisioner library, the external provisioner library, so external provisioner, a bunch of its code, is actually in a library. That's in a separate repo right.

B

Oh yeah yeah that I yeah, so they release it and I haven't looked at how they but like, for example, I think it's going to say they tag it, but that you don't have like images, but you still tag it. People can just.

A

So my question is: do you even need release tools to do that? If it's a library does release tools, do anything? Because because there's no make.

B

Files, I think I think that one still has released towards that particular one. I believe so because you still need to run all those ci's and everything right. So this it's not just for building the images, but also uh running all of those tests. I think you still have to run those tests in there.

A

Okay, well we're not going to have a lot of tests with the populator stuff right off the bat right.

B

Well, that's actually that one is in under six, so yeah, okay, you know what I I'm not sure actually probably need to either. We need to check check that out or we need to ask.

A

B

If not, you had.

A

She would know, and michelle.

B

Should know more than me about this, you know english, so you can ask her because patrick may be maybe a little late for him he's probably still wrong. So why.

A

This doesn't need to be done today. I just I needed to know like, but first of all, if, if I should be trying to merge release tools into that repo, so it sounds like yeah.

B

Yeah, I think I think that's unique. Okay, I think it looks like that one is the uh actually that one is like uh what is matt matt wong is the one who is doing the release so and the young okay young also did the release for that one. So maybe, but if I, if you look at what they released, they just release the you know the source code. Those are automatically those there's, no images, but they tag it so, but that one also, uh let's see yeah, that one.

B

Oh this one does not have release to us. Okay, I thought that has released was. I was I'm just checking that one. This one does not have these tools? Okay, oh so! Maybe that's easier, then yeah, so you probably should actually check with it. Maybe she'll ask young because a mad one I'm not sure uh how to get hold of him. I'm just looking at that repo. I don't see, I don't see, release tools there, but then oh, but they do have an e2e interesting. They do have a test e to e.

B

I'm not sure what this one is. This one.

A

Okay, we're looking this is a sig storage.

B

A

External provisioner is that right.

B

It's a it's it's in! uh Okay! Let me put it in the chair chat here.

A

Well, this one right, I just pulled it up on the screen. Yeah.

B

Yeah yeah yeah, so I don't. I don't see, release tools there so, but they do have some each like test e3. So I don't know what they what they run. But you can. We can take a look of the you know. They have some pr's out right. So, let's see what is wrong.

A

This is very simple. This is like uh okay,.

B

So this shell script, so they do have okay, so you just you know what what you do you go to the test info and see what type of a ci set up there. So there's cis, if you see there's a there's test, there's a there's, a build and there's a unit. There are two tests there. Why is the unit test? The other is the external provision of beauty. Just look at the pr's. The reason appears there.

B

There are two two jobs there, so I think you probably just need to set up something similar, but at least there's no looks like you don't need resource in this repo.

B

You know what I mean you just did you see the did you see the take a take, a look of the pr on that repo.

A

Yeah this is this is one of the pr's yeah.

B

Yeah, so if you look at it, there are like, I only see two drops. Why is the uni test and then there's a there's, a beard? How come you don't see the.

A

I don't see anything.

B

Yeah, the first one: uh how come you? Oh? Maybe you didn't log in.

A

Oh because I'm not.

B

A

B

If you log in you'll see there are, there are a couple jobs, so I think you need to set up something.

A

All similar, let's here's, what firefox is where I'm logged in? Oh, okay, so yeah you got to be logged in to see that.

B

Yeah yeah so so go to test info and see how this one is set up, because under there's.

A

A separate repo for test infra or.

B

You are going to set that up anyway for uh for your other repo anyway right, there's a test there's a test in for that you need to set up your jobs. I thought we. If you check your notes, you should have that in your notes. Somewhere. I thought we talked about this one.

A

Yes, yes, I have links to to the guides that I have not yet read. um I I will read those, but I'm just I'm.

B

Saying that you can point to something for each repo, you need to set up separate jobs.

A

Okay in a separate repo, not not in the repo itself.

B

It's it's in the test. It's called test info, okay,.

A

It's a it's a kubernetes.

B

Testing first thing: okay,.

A

So I I really need to read those guides. Yeah.

B

And then go there and you can see, I think uh you will find drop set under kubernetes csr and this is under kubernetes six. So right so uh I don't know if there's anything similar on the kubernetes csi, but yeah just just look at this one. I think it should be similar anyway.

A

Okay, but but it looks like whatever tests are run by by the the sig stores live external provisioner, they're, minimal, right, yeah.

B

Yeah definitely this one looks uh much so that that's much easier, that's good. At least it's simpler than this other one. You probably just set up something that you need to do the same thing as the other.

A

Well, I want to start off with, like one sanity test, that just to prove that.

B

A

And then expand it over time. Of course,.

B

Yeah, I think, for the first one you probably just you know just set up the uni test. I think that'll be good. Yeah, okay,.

A

Okay, so yeah this, this one will be my model. Is this the the live external provisioner for the for the library because yeah it doesn't produce any artifacts or binaries? So I guess release tools isn't necessary here, but you still need some sort of makefile. That does something so we'll figure out some verifies.

B

Yeah and there's actually a lot of uh yeah, you don't have anyone help you it's a lot of work just to get all of this thing set up. I mean well.

A

So so sod and you are the ones that are helping me and and if you'll remember.

B

um I'm not having a review. I'm saying that, like the coding part, looks like you, don't have anyone help you with like.

A

B

Thing set up and there's a lot.

A

Of work, as I'm.

B

A

Is why I was resistant to having any new repos way back when.

B

A

Definitely a pain, but we made this decision so we're I'm just going.

B

A

All and then once I know it all then it'll be easy um yeah, so I'm so so so you're you're seeing me learn as we go um and and yeah it's just uh it's hard to do wonder under the gun of the schedule, um but um certainly well. Even if we don't get a release out by the alpha release out by the feature, freeze or the enhancements freeze, we can still get the apis approved for 122 and then do an alpha release and a beta release both in the yeah.

B

Yeah, because this is the yeah- this is the out of a tree so yeah. This is powerful thing do you, but you also need to make some changes in tree right for the for that. Yes,.

A

Yes, and and that pr has been sitting there since january.

B

Oh okay, so you have to peel that. Okay, that's good yeah!.

A

That that pr is written and ready to merge. In fact, I think, there's.

B

A

There's one for the backwards, incompatible change and there's one to flip the feature gate.

B

A

Those those are both ready. Okay, the only additional changes would be if, if there was any feedback from from the api review, saying no, no, we want to change something. Then I have to update those pr's but yeah. Those are just those were ready to go it's just out of tree stuff and the api approval. That's been holding us back so, okay, uh so that's, I think um what I wanted to cover today. um Thank you for all the information shang. It's very helpful.

B

A

For people volunteering to review uh did anyone else have the topic for today or a question.

C

I lost my participant list here. It is. um I do have a little update then, like last week I told that I'll go through the code to write. Some tests uh actually did not get a chance with the havoc going in india. So I'll just again look back into that. But the question is: is the test thing a priority like? I see that there are a lot of things that needs to be done in terms of release tools.

A

Oh, oh, the the automated uh tests for um unit tests, the the unit test for which repo was it that was lacking it was it the the validity.

C

I think it was. It was for validator.

A

Yeah so volume data source validator is going to produce artifacts, so it is going to have release tools and we are going to. We do want to have like proud tests that run when people uh push new pr's to to try to catch regressions before they merge um so yeah. I'm just.

B

Loading up the.

A

Code here give me a second: my ide is slow.

A

Oh, no, the the the validator has has some unit tests. So let me yank this over here yeah, the the validator has this package data source validator and it.

B

A

It has some test cases here and then the test directory there's there's no tests for maine, um but there's not much code here.

C

That's why uh I mean I went to this and I couldn't figure out what else needs to be there? Maybe I lack understanding of the project, so yeah.

A

This is a pretty simple project, so this one I don't know how much test we can do. The other one is.

A

The volume populator library this one because there's no artifacts um and there's just an example, uh then.

A

Unit tests are going to be especially helpful here in the populator machinery directory, which is the actual library um you have controller.go, which is it's 760 lines, um and, and it's got this one enormous function called sync pv, that's half the file about 300 lines. So probably what needs to happen is sync. Pv probably needs to get refactored to be more testable because having a giant 300 line function is not great for unit testing.

A

And then the rest of this probably well, I don't know the rest of this- is basically just boilerplate controller setup. So um if you wanted to take stabs at a at refactoring some of this and making it testable, I would appreciate it I mean I I haven't. I I'm looking for help because I just haven't thought about it right. I have a million things on my to-do list and I know that there's no tests here and I know that something needs to be done about it.

A

There's also no logging in any of this, so, like one of my pr's, is to add a bunch of logging statements here, um there's some performance enhancements. I was gonna I was gonna put in to uh so that the watchers don't watch quite so much stuff. um There's a few other kinds of small enhancements like that. I was gonna make and then I was gonna break it. I was gonna start trying to break it apart to make it more testable um and yeah.

A

If anyone wants to just like help, uh it would be, it would be great, but I like I said I haven't thought about what needs doing and what doesn't need doing so it would need to be someone who can just come in and be like. This is what needs. This is what we need to do.

A

um So, if you, if you have ideas about how to like refactor this and add tests, um go ahead and submit a pr if that seems like too much I'll get to it eventually,.

C

Sure I'll just.

A

C

This repo and I'll.

A

C

To see, if there's anything that I can do about that and if I'll have questions.

A

We also need need to think about like end-to-end tests so like I have. I have my scripts that I run which so I build hello populator, which uses the library and I set up a kubernetes cluster, and I create some instances of of hello objects and I watch you know uh the pvcs get populated and I have pods that run to validate that happen, and those are all shell scripts that I have in a completely separate project like those.

A

Maybe I need to move in here, um but but then the question is like what kind of framework is necessary to to run those tests like in in prow, um because, like I have I have my way of setting up a kubernetes cluster and installing this stuff, but like in in prow.

A

We would need something: that's able to build the the example and then set up a kubernetes cluster and then install the example in the kubernetes cluster and and do all that stuff- and I just you know, that's a whole bunch of test stuff that I've never done before um like I well, I I've done it, but I don't know how to do it. In prow and in the infrastructure that that would be the new thing for me um so yeah.

A

If someone wants to figure out that kind of stuff as well, that would be really cool.

C

Yeah I mean at this point of time, like I don't have anything but I'll try to go through once I go through and learn and I'll understand that if there is anything I can do about that, so okay.

A

Well, that would be great yeah, thank you um and, while you're at it you should. You should join the uh the kubernetes csi group, um so.

C

A

A

There was something that came out of one of the. uh I should mention this because this just happened when I was in a cozy meeting and we were talking about finalizers and how to reliably add and remove them, and I was using my pop my live volume, populator example. This one right here ensure finalizer as an example of how to how I do this and we found a bug during that meeting um in my implementation. So I have a patch that fixes that too. I.

B

Think that that's the one that I reviewed at the emerge, I believe I hope it's merged. Okay, I put the labels on.

A

You did oh okay, I can't even remember which ones I pushed and which ones.

B

This one is, this is the only pr on that particular repo. I think, and then you have two pr's on this other ripple. One is release tools. I reviewed the other one where you address the comments.

A

Okay, okay, yeah! I just I need to go through and get all my pr's straight, because I got a bunch of them and so I'll appreciate uh quick, small reviews shang, because yeah most of these are going to be like five lines: 10 lines, little quickies, um okay, all right uh anything else, then from anyone.

D

C

D

A

D

Some update on the use case that we talked about earlier so.

A

With the database initialization.

D

uh No actually, we were. I was working on a demo to demonstrate this feature, one of the opening base user. So the requirement we got like there are some application uh which maintain the replication uh by themselves. So storage level replication is not required, uh but uh application can maintain the pool and they can add node or delete node. They can maintain the replication by themselves.

D

So one of the user was asking to have a solution uh in which you can he can create a volume using some seed data. So basically he was using cassandra and he wanted to create a add, a new node of cassandra and that node should contain some volume and that volume is present in some remote location.

D

So uh I'm going to demo him in the future and maybe, if any more point I get from that user, I can discuss with you and okay.

A

I'd like to understand this a little more clearly because it sounds very interesting, um so is it? Is it the case that they have like a cassandra database in a separate kubernetes cluster and they're, trying to create a new copy of it in a different kubernetes cluster, and so they want a way to basically create the volume with a copy of the data.

A

D

So so they have, uh they have. They have a cassandra instance and they want to add a new node to cassandra and when we add a new node to cassandra, cassandra.

B

D

Data copy to more than one place, so there is a performance degradation if we rebuild everything from the scratch. If we just add.

B

D

Without sheet data, it will take more more time to come up.

D

So what the teacher was asking some solution in which he can create a new node with some seed data and that data is present somewhere in some remote.

B

D

That data is present, you just need to pull the data and add a new cassandra node on his cassandra cluster.

A

Yeah, I unfortunately don't know anything about cassandra, so I that part doesn't make sense to me, but um but no it sounds. It sounds like a good use case. um I. What I'm trying to understand is like is the data that you want to seed an exact copy of some other pv, that's somewhere or is it some like? Where does it come from.

C

Was it like this that you have cassandra deployed in a kubernetes cluster?

C

Let's say it is a stateful set uh right now having three replicas and then you want to add the fourth replica, just for example right so, for example, all three replicas they have some data into it and when the fourth replica comes up, we actually want the seed data to be you know, populated by the volume population is. This is my understanding right.

A

Okay, but where will this data come from? That's what I'm trying to understand, yeah.

D

Yeah that uh seed data actually kind of uh backup or something they have set up for that cassandra instance and they post that seed data to some remote location from the cluster.

B

So is that, like.

D

B

A snapshot of that.

D

Using some time interval yeah.

B

I mean, did you take I mean you take a snapshot or you use some cons uh cassandra's own way of like a date down for something. What what is the method.

D

Yeah, that is cassandra's own way of.

B

D

A

Okay- I I guess yeah, I don't know if I cassandra to know if how how this would work in practice, but um but yeah like the whole. The whole area where populators are meant to shine is when you want to be able to have volumes that contain stuff that you don't already have in another volume or another snapshot, because because we have ways to clone volumes with volume, we have ways to take snapshots of volumes and then clone those to new volumes.

A

But but there's all these other cases where you have data, that's either outside the cluster entirely or or it's something where you don't want to have you want to be able to dynamically, generate it and not just have a a bunch of volumes and a bunch of snapshots to clone from, and that that's where this feature is intended to really shine um so yeah. I I guess yeah when you have that demo ready. Please please consider presenting it to this group, so we can understand how it works.

A

I I I'm just a dummy about cassandra, so I don't understand like what kind of replication it does or what kind of c data it needs, but um hopefully a demo would make that obvious and we could figure it out.

B

Like do you have to take uh like create a um blank pvc first and then override that with those data, or um I'm just wondering how you.

A

Think that that's what the populator would do right, it would start with an empty.

B

A

And fill it with data.

B

Right so I'm just wondering like uh okay, so let's just put the populator away for a minute is like manually when you do. I don't know what like how how far you are with your demo like manually when you do this type of thing. Do you only just create the pvc first and then try to?

B

I don't know what is the term? So you have those you have the process of getting that data back using the cassandra's way. Is that the steps or completely wrong.

B

Hello, did you get my question.

D

uh So, uh actually that uh cassandra use case is uh for one one of the open piece user that that user has requested, for I also don't have much idea on cassandra what I am doing, I am just presenting him a demo solution where I can pull the data that is present in s3.

A

D

A

Question was like manually: how would you do this without a populator?

A

Do you create an empty volume and then get an s3 client and pull the data out of s3 into the volume yeah.

D

A

D

A

D

Yes, so my container image contained the s3 cli and it the cr has the configuration of the bucket and it has the access key and everything it can. It will pull the data using populator.

B

A

That makes sense to me: does that make sense to you shane.

B

Yeah, okay: I was also thinking about like how to uh restore this type of uh data. Using you know the the you know the dump uh yeah, so it looks like you have to have a pvc first and then you can run that command. You cannot run that without a volume. I think that's what I'm just trying to do.

A

Yeah yeah, I have a, I have a prototype that I made like probably a year and a half ago. Well, I I can't remember how long ago, um of of some sort of like a backup and restore uh implementation, that that relies on some sort of populator that push data into s3 and pulled data out of s3, and that was that was when I realized that we needed to have populators is when I wrote that prototype I was like.

A

Oh, we need something to do this this this pulling of the data and putting it into an empty pvc that that's transparent and so that that's why I started working on all this to begin with so yeah it totally makes sense. um Okay, uh anything else. Are we done?

A

What's that once going twice? Okay, thank you guys! um Thank you, shane uh and we'll. uh I guess I guess we'll keep the meeting on the calendar for next week, even though it's kubecon, I am not doing kubecon personally, um but I I'm going to attempt to get a meeting with tim before then, if at all possible, um the sooner the better I can get on his calendar.

A

So uh if I'm able to get a meeting with tim I'll I'll, let you guys know so.

D

A

Everyone can attend. Thank you.

B

Thank you, bye.

A

All right, bye, guys.