Kubernetes Data Protection Working Group, 19 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes Data Protection WG - Bi-Weekly Meeting 20210519

Description

Kubernetes Data Protection WG - Bi-Weekly Meeting - 19 May 2021

Meeting Notes/Agenda: -

Find out more about the Data Protection WG here: https://github.com/kubernetes/community/tree/master/wg-data-protection

Moderator: Xing Yang (VMware)

A

Hello, everyone today is, may 19 2021. This is the kubernetes data protection, one group meeting. So today uh we have a few things on. The uh shenzhen gave a presentation at kubecon about the data production group and we got some questions, so we will first discuss this question that we got there and then we'll go over some caps.

A

Okay, so let's get started is uh martin here. Yes, I'm here hi welcome.

A

You want to talk about your question.

B

Well, the question is already known because it's actually about external data, so that data that is not inside your cluster.

B

It might be more complicated or simpler if you think about cloud provider provided resources, so databases provided by a cloud provider that you access from your cluster, for example,.

B

A

Okay, so so currently, uh how do you, let's say: how do you protect your workloads right right now,.

B

Okay, you go ahead so when you use managed databases from a cloud provider, the databases are backup for you, but that doesn't ensure that you can restore your application to a consistent state. So that's the big problem. Oh.

A

Really, okay, oh, I thought I thought there was. I was actually just reading this because I'm not really familiar with how this works. I was just reading this. uh This is a you know: amazon rds. It says that you can also perform a user initiated backups, I'm not sure. If that helps so you're saying you are actually using this automatic one. um Yes,.

B

Right now we use well, we don't use aws, we use uh azure.

A

B

um I work for the dutch government and the dutch government has a special agreement with micros with only microsoft, so we can only measure and azure isn't as complete in its services as um amazon is, unfortunately, oh you're saying it does not have this way for you to. You know I think you can trigger. I think you can trigger external backups, but what you can't do is store the backups outside of your environment.

B

Okay, you have to do standard uh backups like uh well. If you use a postgres database, there are tools to make postgres backups, but that's actually the thing you don't want when you use managed databases.

A

So you can so it's it, it's actually store. This is for you. You can't really like say manage where you want to store the database. That's what you're saying.

B

A

Okay, uh oh, we actually have an ad here. I wonder if ad actually works for microsoft. Do you have any comment on this.

C

uh On the effect of, um we can't store our backups outside of our environment. No, I I don't really, I'm not a super expert on the backup end of things, but um I have contact with folks who are and can certainly provide this feedback to them.

A

Okay, yeah, maybe uh yeah, maybe ad- can.

C

Yeah martin, I'm happy to have a conversation with you offline and learn more about um the the differences here. I'm certainly you know some of our backup folks have a good idea about that, but we can at least provide uh some additional feedback to them.

B

Okay, that sounds fine yeah.

A

So I wonder if this is okay, so you're talking as you, I don't, but I don't know if this problem is really soft if we are using aws that I'm not sure, because I don't really have experience with this- I'm not sure if anyone else on this call have any experience um dealing with this uh cloud manager, service.

D

Yeah, I can jump in here.

A

D

What we do is so I'm tom from castin uh customer by veeam. What we do is we treat it like a snapshot, so we treat the cloud provider apis like a snapshot and then what we do is we do a logical we. You can go through and do a restore of that snapshot separately and then do an export of that data using a logical dump.

D

So you you mentioned postgres. So, for example, you can do postgres. Dump that extracts data logic level. Then you can put it kind of into any backup repository you want. um There are not for many databases, you're exactly right, they're, not good apis to get data out into external systems, except through kind of the the database protocol itself. Right.

A

Oh okay, so this is actually okay, so use this. You can solve this problem. uh No.

E

No, no! No! What cloud provider have you.

A

So uh looks like a jew, maybe there's us looks like there's some issues with you, but you try to visit others or what is this specific.

D

Yeah I mean this is kind of the general approach we take from many things right, even with um you know, filers, for example,.

E

D

And then export data, uh it takes kind of the same approach where you do a local, restore and then export it, um but we yeah the apis for aws I mean I'm pretty familiar with I'm a little bit less familiar with the backup apis for azure, but with aws. They have a pretty rich suite of backup apis that you can take snapshots of your systems.

D

You can do things on schedules, and so what we typically do is we leverage that to take just snapshots, but then you can export data to move it between different regions or in our you know, in some cases, even different cloud providers or on-prem as well.

A

A

Okay, does that sound good to you, martin.

B

Sure it's a great start, um but I think we have to look into um so most cloud providers today have a way of creating cloud resources from kubernetes, so you can see them as kubernetes resources and what would be great if we could extend that api to also include things like backup and snapshots, and things like that right now. I think none of them really have that they only allow you to create a database or a database server from these apis.

A

When you're saying kubernetes what you're saying they actually have their own crds defined yeah, you use okay, all of.

B

Them well way back. We had what was it called again.

B

It came from the um uh cloud foundry guys.

A

B

From pivotal, uh they started a way to use resources in uh their environment.

B

uh This was uh reused by first, I think, google, um but this was not a kubernetes native way of providing kubernetes resources for external for cloud resources, so they started to develop a operator to do the same thing.

B

So now, all the cloud providers provide operators that allow you to create cloud resources from within the cluster, so you can create a database. You can well. You can actually create vms from kubernetes resources.

B

Basically, lot of the things you can do with the cloud provider.

B

Create an s3 bucket from within your kubernetes cluster, as if it was a kubernetes resource.

A

So so those operators does it does it normally. They probably also provide a backup functionality themselves right, as far as I know, they don't. But oh okay, I'm not aware of it. Okay,.

F

I believe these uh operators are basically mappings of the apis from the cloud providers right. Basically, it's taking all the let's say the aws cli or the aws api and then mapping it as kubernetes calls within the cluster. So it should be possible to also.

B

Add the api for the backups.

D

Yeah, martin, I think um this is kind of what our product does actually. um Okay,.

E

So I can send you some material.

D

Or do a demo if you feel like we can follow up offline? If that would help.

A

Yeah, we maybe you know some of those we can even uh have a session in this group. I think it's it's interesting yeah, um because I'm not I'm not sure. I thought I thought this is a at least right now. I thought we don't have a way. So until I just hear about this, today uh looks like there is a way, actually um so yeah, so I think maybe yeah. Maybe we can uh so tom. Maybe we can uh have a session here as well if of course, free free.

A

If you want to talk to modern offline, but.

D

Yeah, absolutely no I'd love to give a demo. This is kind of what we talked about with the application integrations, because I think you need a combination of cloud apis and yeah logical dumps, and so I can yeah. You can sign me up for something.

A

A

A

Okay, do you have anything else about this topic? I see that in modern. You also, I think that you added some comments to our white paper uh just trying to include this one in our scope, so I think yeah, let's see uh after tom's demo and see what what we should do, because I was not quite sure how to handle this yet uh yeah. So, let's take a look.

A

Okay, thanks! Thanks for your feedback.

D

Yeah great questions thanks.

A

Okay, uh do we have anything else on this topic anywhere else?

A

If not, we move on to the next uh okay.

G

So yeah, could you give me sharing power? Oh.

A

Sorry, let's see.

A

All right, yeah, so uh yeah so ben is going to give an update on the volume populator cap.

C

A

Yours I'll stop sharing.

G

It actually, let me cancel your sharing yeah.

A

Can you see that.

G

Yeah yeah, it said: do you want.

E

A

G

The screen sharing- and I said yes, it just did okay.

A

G

We uh we merged an update to the volume populators kepp last week, but only after a significant design change, um and I wanted to sort of socialize here what what the design changes and what the plan is uh to sort of stave off any confusion about this, uh because I think when people first hear about it, they're like what the heck so um so, uh in short, the the plan is we're gonna, be replacing the data source field with a new field called data source ref kept 1495 was the original volume data source cap.

G

It's been updated for 122., here's a link to it. The reason for this is that uh it was not possible to proceed with the original plan and, at the same time guarantee that no workload anywhere on earth would be broken.

G

Now we had been confident that, like the probability of anyone's workload being broken was like very very low, but the current stance of the kubernetes api team is, it has to be zero. um We can't even risk it um so so we can put this new plan um with the help of the uh the kubernetes api folks. um So I'm gonna outline what the plan is over.

G

All the different releases, so in 122 we're going to introduce a new alpha field, called data source ref, it's going to be controlled by the existing any volume data source, feature gate, we're going to add backwards, compatibility logic to cube api server such that you know. If, if you have clients using the new field and clients using the old field, everything interoperates in exactly the way you would expect without the potential to break anything that already exists.

G

So there's a lot of thought went into how to do this right and the kep spells it out. If you're curious about the details, we'll document this as a new alpha field and in this time frame, I expect to update the existing volume populators implementations to use the new field so that you'll have to um uh switch on this feature gate in a kubernetes cluster to actually test the volume populator. But that's how it is today already so so that's not going to be too new.

G

It's just more code to rate the plan is in 123, then this feature would move to beta. The feature gate would default to on at that point and we would update the docs.

A

Excuse me ben: do you have to have a slide that shows exactly how that field look like, because I'm I don't know if everyone realized what this one means for putting snapchat. I.

G

Don't have a slide, but it looks. I mean it's, it's it's a carbon copy of data source. You.

A

G

A

Your cab later, okay,.

G

Everything in the field is identical to the data source field. It just has a different name, and it has. It has different semantics such that, if you put garbage into it, unlike today, where your garbage just gets ignored, you will the garbage will either be accepted or you'll, get a rejection, error and it'll throw out your pvc so so you'll all you'll never have a situation where the user input will be ignored.

G

You'll either get a success or an error. And then, if you get a success, uh you know the the results will depend on exactly what the data source is and whether the popular is installed and all the current considerations that you have with the old design um so and and I'm happy to go into detail on on on how this is going to work.

G

If we want to discuss it, but I just wanted to lay out the initial plan um so so in the 1283 time frame, this would mean the data or volume populators would be usable on production clusters, because the feature gate would be turned on, and that would be the release coming in the winter time. I think um if all goes well in 124, this new feature would move to ga.

G

What that means is the feature gate. You can't disable it anymore. It would just become baked into kubernetes.

G

We would deprecate the data source field, but it would continue to work as before, um as part of the deprecation pvcs submitted with data source fields would start to get api warning saying, like hey, you know I'll accept this request, but by the way you're using a deprecated field, you should consider using the new field, and you know the warning would sort of spell out the change that we had made um and at this point we would update the kubernetes documentation to stop mentioning data source and just tell people use this instead going forward.

G

Now, even if new users did use the data source, ref old clients will continue to work and be compatible and nothing will break. I want to emphasize that, um but uh but like this has to be the the path forward to sort of get past the existing bad design with the data source field in the 125 time frame, we would probably be switching the external provision or sidecar to start using the new field.

G

We would probably have to do a version bump of that sidecar, because this is a backwards, incompatible change to to use a new field which would mean that that version of the external provisioner sidecar that would happen next spring about a year from now would depend on kubernetes 124 later, and the data source field would continue to work for old clients.

G

Even if you're using an updated, provisioner side card again, there would be no compatibility break by making this change. You would just be giving you a new sidecar that wouldn't be getting deprecation warnings when it send apis to the kubernetes server.

G

So many releases later I want to emphasize data source field still works. uh You know so.

A

When you say it's your, you can still use this to create a new.

G

Yeah, what it would mean is is people who were using like yaml from today to create a pvc from a snapshot. It would work exactly the way it does today, people that were using side cars from today that that are working on the data source field. They would continue to work the same as well. They might receive deprecation warnings, but, like you, can ignore those so.

A

When you do a okay, so you can create and you get it should you it's going to be converted right. So it's going to be converted to draft to data source.

E

A

When you do a get of the other source, you can still.

G

A

Both because you get that.

G

Yeah yeah well.

A

Actually we're not going to get that by itself, though right. This is just like one field, so you're going to get a pvc you'll.

G

Get a whole pvc and it'll have a.

A

Downstream field.

G

A

It's going to have two you're saying it's going to show up.

G

It'll have two copies of the data: yeah it'll just be a little ugly, but but we'll tell people, don't don't look at that old field? It's it's. It's legacy cruft, but you know we. Obviously we have to keep it for backwards, compatibility all the way up to you know one dot whatever um and then someday kubernetes 2.0. We would remove the data source field, uh but this will probably never happen. um So this is just to emphasize that you know if there is ever a 2.0.

G

That's when we would remove the field, but no one anticipates there ever being a 2.0 of kubernetes.

A

G

So so that's that's the high level plan of the next. You know four releases and how we would roll out this new field, but, like I said that the details of the exact semantics of the new field are spelled out in the updated cap.

A

Can you show uh how that api looked like? I just want to remind everyone, not sure if folks realize what this mean yet.

G

Yeah, there's not actually an example of it.

A

Maybe you just go to maybe just go to that data source in kubernetes. Apis type go, maybe maybe just.

G

Do you happen to know what the search.

A

G

Want do you want to hunt it down and share.

A

Let me open it, let me find it yeah.

A

You have not submitted any pr yet right right now. It's just.

G

Right, the cap is approved for 122., so api guys are happy with this, and they see this as the future to sort of get out of the mess we're in with the existing data source field. um I need to write, write the code and and get it approved by the api guys and the storage and us the storage.

G

You know people we all need to make sure that we're on the same page about this, but um but but the the understanding is that the way that the existing data source field was implemented, we made some mistakes along the way and we shouldn't have been ignoring user input starting at around 115.

G

You know we, we should have stopped ignoring user input when those features went to beta and we didn't and because we didn't we're stuck with this weird semantic- that it's possible, some clients depend on and and therefore we can't get rid of it.

A

Okay, so maybe I will share, uh I don't know if this isn't that clear, it's just just showing how the api looks like.

A

Maybe this is not that clear? Well, so this is basically what ben was saying is right now this is the this. Is the persistent volume claims back um here? We have a data source so currently this data source can either be a volume snapshot so that you, when you create a volume from snapshot right.

A

This is the data source or it can be a pvc when you are doing a cloning, uh create a pvc file on pvc or does any modern data source, any volume data source that your brand has been working on for the voting population. So.

G

What we'll do here is we'll we'll copy this. This field data source, so line line 459 will get cloned and there'll be a second field with the same type called data source. Ref, we'll probably update the comments to remove any reference to custom resources in the old field, because we won't support them and those will all be moved to the new field and and then there will be compatibility between the two.

G

So it's such that, if, if a client specifies one the contents will get copied to the other, you know in the way that you would expect, and so clients that are consuming either field will always see the right values and it will be impossible to have different values for those two fields.

G

They'll either they'll always either be blank or have the same contents.

A

Right so this is a so I wonder. Maybe people don't really use this field for for restore, become restored because, or at least not everybody? No, it's not for every story system. Okay,.

G

So basically, this is for them, that's how you create snapshots, and this is how you clone snapshots and clone from existing right.

A

Right so this is yeah yeah. So this is the example right. This is just the source field. When you create a pvc, you specify data source field. Now this is the example you take the one in snapshot, but in future um ben is going to add another one called data source ref well,.

G

Yeah, so what will happen is if you submit this exact yaml to a future kubernetes server like 1.24 or 1.23.

G

It will accept this yaml, but the object that gets created will have a second copy of the data source. Called data source ref like so data source, will be left alone. There'll be another field called data source ref with the same contents.

G

Therefore, if an old client looks at it, it'll see the data source that it's expecting and if a new client looks at it, it'll also see the new field and then- and if you specify this yaml with data source ref instead of data source, the same thing would happen. It would populate the data source ref field, with what you wanted. Oh.

A

Does it actually go back if you yeah.

G

Yeah it'll copy you back too yeah. The crucial thing is, if you specify something other than a pvc or a snapshot today in the data source field, what happens? Is it just gets ignored? Like you never specified anything, you get an empty volume. If you try to do that with the data source ref field, it will either accept it and and preserve the contents, and then you'll get a pvc with that data source set or if it's invalid, you will get.

G

The whole thing will get rejected, but under no circumstances will you get an empty volume if you put something in the data source field in the data source, ref field of a new pvc after we've implemented this code and that's the semantic that we wanted to have all along with the data source field, and we we somehow ended up not getting because if someone puts something in the data source field, clearly they they don't intend to get an empty volume, but but they do today.

A

G

A

Then yeah and also this so so we'll start the deprecation notice, probably in.

G

Yeah, when this feature reaches g8, which is 124 at the earliest, it could be.

A

G

It could be later if things go, would go a different path.

A

A

Right, uh okay, is this uh clear to everybody I just want? I thought it's important for for him to talk about this. When I first saw this one, I was a little surprised and worried that what will happen to a to this, uh the absolute field, but I think it's okay.

G

Yeah, even the api guys admitted that, like this is a little gross but like we had a long list of other alternatives, and you can go back and look at the discussion in the cap if you're curious, all the other alternatives were worse than this, so so this is the best we were able to come up with.

G

A

uh So I wonder, actually do people use this field doing restore- maybe maybe not necessarily actually using this.

G

A

Field will pass.

G

The data source field.

A

Right because uh yeah, when you do back on risk, do you actually I mean not? Maybe not not all the time, maybe sometimes.

G

Provider they can use.

A

G

Populators use this field and external provisioners use this field right.

A

I know I'm talking about like backup software. I was just talking about it. Yeah like we don't use this, we don't we don't do it this way, but I mean if it's like cloud provider. Okay, you can totally just use this.

G

If there is, if there's backup software today, I mean there's no support for any kind of backup api built into kubernetes. Yet so anything that does backups is built on top of pvcs, so they create empty pvcs or they take existing pvcs and they shove data into them through the normal mechanisms.

G

They don't use the data source field. um We are hoping someday to have a built-in backup restore api in kubernetes. That would use this, but um but that's a that, that's sort of would depend on having the volume populator's mechanism working at all.

A

But I think it's uh for cloud providers. Actually they can just use this field. Yeah I mean, depending on the storage system. That's what I mean. So if it's a cloud provider, you can totally use this field too.

G

If they've implemented their restore process as a volume populator, they would do it that way if they.

A

Yeah, I'm I'm talking about right now. Existing right, yeah.

F

G

They're, if they're using something other than a volume populator, they would not use that field today ever because it wouldn't do it.

A

No, no, no, I know there are. I know there are people who actually use this already, but it's just of course that's only for cloud providers, but anyway, I was just trying to see if there were any feedbacks uh from this group. That was that's why I was asking this uh because I does not. I didn't hear anyone. This was surprised by this.

A

D

No, I'm sorry or not, as I say, I don't think we use this, but I'm just confirming.

A

Okay, do you not use it, but I think for cloud providers you actually could use this right. You don't need to use this field, so if it is like uh for like aws ebs, you can just use this directly. Can you.

A

You don't either.

G

I don't believe so I um I I will mention that, like you know,.

A

I'm just asking like other other providers, I don't know because I know some people actually, I know like there are other backup vendors who actually uh uh just leverage this well. Okay, actually, I know balara actually use part of this for one of the plugins.

D

uh So I just checked: we actually do use this, so I'm going to follow up there.

G

Okay, did you use it for your backup process for your store process, restore.

A

Of course we saw it's not that restore right. This is a pretty pvc.

G

I I was just going to mention that netapp has a backup and restore product and we of course, use snapshots as part of our backup process, but when we do a restore you just take your pvc and shove data into it. There's no.

A

Yeah, that's normally yeah! That's because netapp is also the same right, not not a cloud provider. I'm just saying like, for example, if you look at the velara, vallar has a velara plug-in for csi. uh If you use that one uh for aws ebs uh believers- maybe I don't know if that's.

G

Are they running modified versions of kubernetes, because what I'm getting at is kubernetes will delete whatever you put in that field, if it's not a snapshot.

A

No, no, no, it's I'm saying is they are going to create a pvc from from a1 snapshot using exactly this. This type.

A

Yeah yeah backup. Yes, they use that so, but that's just for that, but then that's not going to work for uh for other storage systems. Well, the snapshot is actually local right. So it's just not not right. That's what I'm just saying! That's why I was asking if anyone actually is already has already been using, I'm just thinking that some uh backup software actually uses in for some code path. That's.

G

All I'm trying to say if you're representing your backup as a kubernetes snapshot, then of course you would do it that way and- and you can absolutely do that.

D

Yeah, so what we do here is actually we use this as part of backup. We take a snapshot of the volume okay and if we need to extract it and push it to a backup repository, we then do a restore using this mechanism and then extract data if needed. That's you know that's kind of one workflow.

A

Okay, yeah, so so that you just want so that now you hear what ben is talking about right so just to be aware, some changes coming for this field.

D

Definitely yeah yeah thanks ben.

A

Okay, uh anything else, any other questions regarding this change.

D

Can you share those slides, then? uh Are they in here.

G

uh I I I can, although uh it's on a netapp server and the the link will self-destruct in 30 days, so yeah I'll, put the link there and make sure to.

A

uh Maybe you can just to make that, like a google slides or something.

G

Yeah, it's not it's, not a google doc or no. No.

A

G

It's it's a sharepoint. That's that's why it's weird! Oh sharepoint,.

E

F

G

On a sharepoint um I can, I can share it absolutely it's just it'll only be good for 30 days. I can copy it out if you did. Thank you.

D

A

Why don't we just make that a google doc, google slides.

G

A

G

With google slides and sharing oh.

A

You're, not okay, you know, if you, if you, why don't we provide that here, then I can uh convert that to your google slide. Okay, thank you! Okay! So that's this first one uh and then I just want to give a quick update on the container notified cap.

A

uh So this cup um we actually uh got an approval from the front team api reviewer, so which is good, so we made some progress that we actually uh uh so he actually gave a look good to me uh so right now we are waiting for review from signaled, um but we, you know, passed the merged airline, so this is going to be uh 1.23, but we want to get this cub merged early. I don't want to wait until the merge deadline yeah. So that's this one.

A

And and then uh for cozy, uh I just want to give a quick update so that cap, um it's a it's a very big cap, so we didn't uh meet the we didn't make the merge deadline, but then we realized that all the development work for cozy actually happened out of three, so we don't have to um be restricted by this uh merge deadline. Actually, so we can actually continue to work on this uh continue with the api review uh still trying to get this one in 1.22 timeframe.

A

So that's the that's the status for cozy yeah, because this uh this is a really really big cab and did not get time to resolve all their concerns. From april reviewers last week,.

A

Okay, uh anything else.

A

That's all I have here, hey tom, I actually missed your your talk. You want to give a uh just to talk about what is this? Let's talk about.

D

Yeah definitely so we added ransomware to our product, but this was obviously a vendor neutral talk, so we just talked about how to build this kind of on your own, um and so what we we described is how to create immutable backups in s3, um using the different, uh advanced, locking apis that the s3 protocol provides. It. We've also tested it with many different providers that have the s3 protocol and it seems to work with the ones we've tested, um but obviously like different.

D

If you you have your own version of s3, maybe you don't have to run all the locking apis, um but I think it went pretty well with a good attendance, some good feedback, um but yeah it's it's this. This is kind of a mix of a fairly technical talk, so we dive into the exact apis. You need to use um if you want to back up, have immutable backups in s3.

A

And- and this is a uh okay, this is s3 compatible okay, so this is only for s3, uh so those are like s3 specific apis that you're using there.

D

They are yeah, and so, for example, um you know we have our own storage engine in uh in s3 um kind of similar rustic, and so we've added these support for understanding these apis and be able to restore, and so with that you can get a protection from ransomware, because once you write and if you treat the blobs immunity, you cannot, you cannot overwrite them or delete them, and so you can have permanent backups that will last for some retention period.

D

uh If you use s3 to target um you know we added the star rs3 storage engine, you could do something similar just on your own with, if your backup is an individual blob, but you could also do this with something like rustic yeah. The second half goes into the actual kind of apis.

A

A

D

Yeah, it was very timely because we, you know obviously ransomware has been in the news recently yeah.

A

D

Yeah but the apis industry are pretty cool actually like they give you everything you need to build this out.

A

D

Edging, how was I I couldn't tell, but how was uh cn cdm cloud native data management.

A

uh Yeah, it was yeah, it was actually uh went well. Well, I didn't attend all of them. The time is just.

F

A

So I pretty much just like attended my own sessions. I think I listened to a couple afterwards yeah, but I think the attendance I think it's like there are 600 people all together.

A

Yeah um yeah, so I think uh it overall definitely went well.

D

Yeah we got some good feedback. People seem to really like it, so I think we'll probably do it again. Yeah.

A

A

um Anything else anyone want to talk about anything else here.

A

Coupon or other things.

D

Coupon cfps for north america are due on the 23rd.

A

D

Monday or sunday I forgot yeah yeah end of this week this week. That's right.

A

And hopefully that will be in person so we'll see how that goes.

G

Yeah, it seems, like people have already said it will be.

A

Yeah yeah, that's the plan right. I was just hoping that uh nothing. uh Nothing weird happens before that. But right now, at least in the u.s seems like it's going going into the positive direction. So everybody is uh getting vaccines.

A

A lot of people are getting it. So that's good.

D

The rumor I heard was that the mayor of l.a has to make a call sometime in june.

A

Oh, for whatever the.

D

A

Confirmation or for whether we we can go ahead with the in-person conference. That's.

D

Right yeah, but I think.

A

Some of these things fall.

D

Into plan that people are already booking things up, and you know planning for it to be in person.

A

Yeah, so I think california is doing well right. What's the percentage you know like the percentage of people who got vaccinated, I think they're doing relatively well.

G

A

G

By the time kubecon happens, I don't think vaccinations in the united states will be a problem, but people coming from out of the country. I could see there being issues even even by the fall.

E

Well, there's there's a huge topic or a huge sort of debate. I guess ultimately around whether or not in-person attendance should be gated on the basis of establishing that you you've been vaccinated and that gets into a lot of hairy issues and the board has a working group that will be touching upon that. But you know another factor is that certain corporations, you know as a from a corporate events policy just basically aren't doing anything in person this calendar year.

E

So it's it's not a foregone conclusion that there will be a hybrid event, but but yeah the cncf marketing folks are planning for both contingencies.

A

Oh okay, so it's still not not decided yet.

E

Not not yet but, like I said, planning ongoing for both both both scenarios. It does get a little hairy. I mean it could be as straightforward as if you can stop show me some sort of authenticated vaccination come on in and if not or if you're, not comfortable or whatever attend the virtual event, but none of that's been decided yet.

A

Or would that be like I thought if you're getting a flight, you need to get a test done right. That.

E

Varies from one civil authority and one airline to the next on how that's handled.

A

A

G

But more problematic is, is people in other countries might not even have access to vaccines in the town.

E

Yeah and then they.

G

E

Yeah, well so I mean at the minimum, there will be a virtual component, so you know if there is a hybrid event. They'll still be a virtual attendance option. Oh.

A

You're saying that you can so let's say we we have that in person, but then other people can actually kind of watch, watch it or kind of participate.

E

That that's the way that the planning is proceeding currently.

A

Okay, okay, yeah, probably we'll see uh whether olympics can happen. First, maybe that's an indicator.

A

Anything else, if not that's it, for today, thanks everyone thank.

C

You thank you bye.

A

C