Ceph Ceph Tech Talks, 1 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Ceph Tech Talk: Persistent Bucket Notifications

Description

Ceph Tech Talk Schedule: https://ceph.io/ceph-tech-talks/
Ceph Tech Talk Playlist: https://www.youtube.com/playlist?list=PLrBUGiINAakM36YJiTT0qYepZTVncFDdc

A

Hello, everybody and welcome to another ceph tech talk. I'm joined today by evolve, who's going to be walking us through persistent bucket notifications, and, as I understand this is uh it provides us. You know a additional endpoint to like our our publish and um subscription kind of model module rather- and this is coming and uh the pacific release that we're looking for coming out this month. So thank you all for taking the time to share with us this new feature and I'll. Let you at the stage now thank you.

B

Thank you mike okay, so I'm going to really uh go really quickly through bug notifications in general, so because this is really building on that uh bucktifications are a mechanism to let external systems know what we're doing on the redis gateway. So whenever we put an object, we delete an object. We copy, uh we don't get certification for fetching or getting an object or listing a bucket or something like that, but any modification to that we do to an object. um You can create uh identification.

B

This could be sent currently to http and atp, or rabbit and q endpoints, and to kafka endpoints and in the future. We're gonna, we're gonna, add some more at endpoints this uh these options. So um this is something that was interesting, introducing narulos and is also an octopus, and it is working fine, um but it has. It has a couple of drawbacks.

B

So, first of all I want to talk about the real reliability of the bug notifications. The way it's done is that notifications are sent synchronously with the op. So, whenever you put an object into a bucket, then after you successfully put the object into the bucket, then then, um before we reply to the client that the object was successfully put, we also create a notification which is like lots of metadata on on whatever was done there and send it to whatever endpoint that was configured.

B

So if you configure your kafka message broker as your endpoint, then notifications are going to be sent there whenever the broker gonna act, which means say to you that now I own this message, I have my reliability and persistency in whatever mechanisms. Only then we reply back to the client saying yeah, your put object. Operation was successful.

B

This gives reliability in the sense that if the rgw crashed in the middle of the process, um then we're not going to send back the ax or the the notifications would not send, but also the client didn't get a successful completion of whatever they did, which means that they're going to retry or do whatever they want, but they would know that that the notification was not sent.

B

So it gives certain level of reliability. What it doesn't do is that it doesn't handle the case where the end point has issues. So let's say that the kafka cluster is down, or maybe connectivity to the kafka cluster is is, is not reliable. Then we haven't, we don't have a way to solve that. um We, we would log an error, but we can't even reply with an error to the client, because we don't guarantee atomicity, because we cannot roll back whatever we did to the object.

B

Let's say that we deleted the object and successfully deleted the object, and then we send the notification, try to send notification to kafka saying the object was deleted, but this failed for some reason. Then we cannot reply that that the operation failed because the object was deleted. Only the notification part of it that failed.

B

uh So this is one aspect that is not well covered by the the model of synchronous, certifications.

B

Another problem- and this is really something that actually happened in the field and was the trigger for adding the uh perceived notifications- is uh what if the end point is down but or is very slow.

B

But you don't really know about that, and the only way for you to figure out that the input is down is using a timeout mechanism.

B

So if you send something- and you immediately get a reply back saying- you know, the end point is down the kafka cluster is down, they wrap them. Q server is down, then everything is pretty much fine, but if you don't get a plot and you're kind of waiting for some certain amount of time until timeout expires, and only then you're gonna know that things didn't go then, because of this, this synchronicity of everything with the operation- that's gonna, really slow down the rgw and gonna actually bring it to complete halt.

B

So in order to tackle those two issues, we said, okay, we need to introduce asynchronous notifications, which means that.

B

You're, sending something but you're not waiting for the reply um and and you're gonna reply immediately back uh to the client and make sure that the message would would arrive to the endpoint later on now. Just doing that is actually even possible with the existing model you you do have in the model uh in the synchronous notification, you have an option to say it's really fire and forget.

B

I don't really care so much whether notification is going to reach the its destination is not going to reach the destination and I'm not going to wait for the ax. So in this case it will just work. It doesn't matter if the input is down or not nothing's going to slow you down, you're, going to sell notification, and you know hope things would work and if it doesn't work you don't know about that, but this doesn't work as well, because sometimes the reliability of notifications is really critical.

B

So if you want to have something which is asynchronous on one hand, but also reliable, on the other hand, you need a mechanism that would support that, and these are exactly persistent notifications. So this is the premise. So this is the reason for for why we we added this.

B

uh I tried to summarize that here in the table saying what a regular versus certification, what kind of failures, they're gonna support um and what kind of guarantees they're gonna they're gonna give and, um as I said um in direct notifications, I didn't have a way to uh I didn't have atomicity. So I couldn't say: okay, the the operation was successful, but identification was failed.

B

So you know I'm going to roll back the operation, no, that doesn't work um and the way that I did that I just did the identification at the end of the process. So after I knew everything was successful, but if notification failed, there was no way to roll back. So there's no atomicity between the the radius operations and the notification. Sending you know to tackle that with specifications.

B

The mechanism for sending notification was, what's called a two-phase, commit queue, which means that you're going to first reserve something on the queue making sure that we're almost sure that you will be able to send a notification later on, and you do that before you start the whole process of radius operations and when you finish the radius operations at the end of the process, when you're sure that everything worked fine from redis perspective, then you're going to commit whatever you reserved on the queue and that's going to actually send the notification out.

B

So these two phases gives pretty good guarantees, because if you were able to reserve some something on the queue it means you have space on the queue and if you weren't able to reserve something like you, which means that something bad happens on the other side of the queue. So there are lots of problems or disconnects, or things are down, so you can't really push into the queue. So you would bail out early on before you did anyway. This operation, you would bail out and say this operation is a failure.

B

I cannot do the notification, so I'm not going to try to do the radius operation parts of it so so this is why the two-phase commit creates some kind of atomicity. It's not a hundred percent guarantee because the queue made may be uh not full, so you reserve something on the keys to settle.

B

You do the radius operations and they're also successful when you try to commit, maybe you had a disconnect with the all the osds or some other problem uh that are more rare and, in this case, there's no way to roll back, because you already did your radius operations, but this shouldn't happen. I mean this is some internal problem with the with the step cluster uh for the cases whether you have connectivity problems or some issues with the end point that would be covered by this model of reservation and then committing.

B

um If you made the reservation and for some reason your raiders operation failed, then you just canceled the reservation, so you won't just take space on the queue and other um uh ops that want to send notifications could to use the space on the queue and and send the messages out.

B

So this guarantees automatically but, as I said, everything is is asynchronous, which means that when I'm setting notification, I'm just putting stuff into the queue. Well, maybe another another word about this queue. This queue is a radius-backed object, so it also survives crashes, 100 failures and whatever, because it's backed by the radius object. So if the rgw crashed or if your all the rgw crashed or your your hardware really doesn't work anymore, then everything is backed in redis object. So um other.

B

Algebra can come up again or other rgws can just send the notifications later on. So this gives really reliability in all aspects both of the rgw aspects and gives reliability, even if your endpoint has problems.

B

um Maybe a small small comment here is that um even if you're raider skate, I mean when you're ready to get a crash. There could be reservations that were taken on the cube, but not committed or cancelled. Yet, and in order for that not to accumulate and eventually fill up your queue, then there's like a periodic cleanup for preservations that are there for too long they're just going to be deleted.

B

um So that's that's the the mechanisms that uh tackle the problem of um atomicity.

B

Now, as I said, um the whole process is asynchronous, which means that I'm just pushing stuff into the, and I need to set it over, especially for the cases where you have disconnect problem with your kafka cluster, or you have other issues there and for that the only way to actually guarantee delivery or really solve the original problem of reliability of your endpoint is to have a refry mechanism and your retirement.

B

The referral mechanism is based on just the fact that everything is skewed and cued in a persistent radius object, because some other rgw could be then not necessarily the one that actually pushed stuff into the queue.

B

Is the one that reads from the queue and send messages now those sends are just like. The original sounds completely synchronous, though you you take from the queue you try to send and um whenever you get the ack you're gonna clear that from the queue, if you didn't get the ack, you don't clear that from the queue which means that next time I'm gonna go through the queue again and whatever was that was not act, I'm going to resend it and retry it and so on.

B

If I have big problems and and I'm not managing to send anything, the queue would have built up and accumulated eventually, uh you know, would fill the entire size of the queue and then that would be signaled to the uh rgw's doing the operations by the fact that this queue is full. So in this case I would just fail the operation immediately, because the queue full I need to send notification.

B

But I cannot because I cannot push new notifications, because, on the other hand, I cannot really send them and get axe for them and and therefore uh the problem can trickles back and push back to the client. But in this case it doesn't push back to the client in a way that would uh slow down the client or anything.

B

It will just immediately fail any operation that requires notifications into an endpoint or a destination or a queue which is full the indication the queue is full is immediate, and even though I would get errors, which is what I want to get in this case, I won't get the slowdown of the of the rgw. So this is how the this mechanism tried to tackle the problem.

B

um Now, I'm going to you know, explain a little bit about the the cues and the topics and how they're managed so in in regular bug notifications. um We usually have something called a topic. A topic is really a definition of an endpoint, and um so let's say a certain ip address of a of a kafka cluster or cluster or web server or anything.

B

I want to send notifications to and then there's something called notification, which is maybe not the best name, because it's not the notifications that you're sending this is the definition of configuration identification and really this is what is tying uh tying together.

B

The bucket from one side and the topic from the other side and together this is a notification configuration which means that whenever I'm getting certain events on a on a bucket, I want to send to this topic or to this endpoint and- and this is like how the the all the bug notification mechanism work.

B

Now, whenever I'm defining a topic, I can say that this topic is a persistent topic. It is a property of the topic itself, so um I have two topics to the same: let's say cluster the same endpoint. One of them would be persistent one non-persistent, the regular one.

B

Whenever I'm going to get the op, I'm going to synchronously send the notification to this kafka cluster, the the persistent one uh behaves a little differently. So when I'm creating the topic and I'm marking that as persistent the first thing, I'm doing, I'm creating the cue. So I'm creating a cue- and this is the um the radius object or the queue the two-phase commit queue that later I'm going to use. In order to send notifications to this persistent, this endpoint via persistent topic.

B

um Now it is important to understand that you're going to have a cooper topic, and this is critical for the case where um let's say you have multiple endpoints, some of them can be good. Some of them can be bad. Some of them can be down some. Some of them can be slow and you don't and you don't want the slow one or the bad ones to impact the other ones. So um it could be that one of your uh kafka, entry points or brokers is down, but the other one is good.

B

So only this specific here is going to build up and fill up and eventually fail, but the other one is going to keep on sending. So there is some separation and isolation there.

B

So I'm creating this um this topic, and as a result, this cue and what's going to happen, is whenever I'm going to have an operation on a bucket that has a notification configuration that ties to this queue. Then I'm gonna push the notification to the queue and the rgw is getting the operation. This is the only thing they're going to do just push stuff into the queue now I need somebody to pop stuff into from the queue and send it over now.

B

Anyone can push stuff into the queue there's no requirements on guarantees or any sorts, but the person that is popping from the queue and sending to the endpoint has to be only one, and this is to try to prevent reordering now reordering is not guaranteed to be prevented because of retries you can always have reordering and whatever system that sits on the other side has to be able to handle duplications and reordering and everything, but they have to be able to handle them.

B

Only in when problems happen, they shouldn't be able they shouldn't be needed or required to handle duplications and reordering on a regular basis. This is why we want to guarantee that, in a regular operation, the system there's no reordering and duplication. So uh for that purpose you need a single rgw at a certain point in time that pulls stuff from the queue and send it over. So we using object locks for that.

B

So whenever a new topic is created arbitrarily, one of the um redis gateway is going to see that, because there's a list of topics in the in the object see that and try to take ownership of this topic.

B

um It will do that using an object, a serious log, and if it is successful, then it became the owner of this topic and it will kind of own the queue start to read from the queue and send to it to the end point. If it, if it failed to take the lock, it means that somebody else owns this queue, which is good because somebody else handles that now the queue or the lock over the queue has to be renewed, and this is for the case where rgbs go down.

B

So if one of the rgw's owned couple of queues and served them, which we emptied them and send it over uh this, this rgb can crash and go down, and then we need to make sure that, after a while, the lock would expire. So some other rgw would try to log in success and be successful. All of a sudden, because the uh the lock that the the crashed rtw expired and then the rw, the new rgw, would be the new owner of the queue and would start to empty it and send messages over.

B

So this is where the mechanism, how the mechanism works. So uh once you own the queue you renew your lock, you know periodically and if you don't renew that it means that you're down and somebody else can take the lock and start serving this cue. uh This is how we guarantee that to be uh no reordering or duplications on a regular basis and on the other side. On the other hand, there'll always be an rgw that manages a queue and serves it.

B

um When you delete a topic, because you don't need this topic anymore or you decommission the kafka cluster and you want to take down the topic, it also deletes the queue um and there'll be no new notifications being set for this. One.

B

What else so? Yeah I mean in this guest? um There are a couple of commands that can help you figure out which uh rgw owns, which queue this could be helpful uh for the purpose of of debunking, like a large system. Would that have lots of organelles and so on. um The the kind of gotcha that exists there is that currently, the identification of the um of the rgw that is owning the queue so the then this is part of the redis command figure out.

B

The queue is based on the either the redis client id uh or the this address identifier, and you don't have a good way to get those except looking at the um at the log file at the rgw log file. So uh we probably need to improve on that, but now the the only way to kind of uh associate, a certain cue with, which is the rgw that serves it and activate and send the notifications from it is, is by running this command and then matching that to whatever you see in the rgw log.

B

So this is pretty much what it all. What I wanted to uh to talk about when it comes to persistentifications, um mainly, you know, give some hints about debugging and how the mechanism work. If somebody is using them and things are not working or not working, as as they expect them to work.

B

So I'll be happy to answer questions or if you want really to explain about other things,.

C

Sure I'll have a quick question. uh You guys, so you are in the end uh you had mentioned about the two times: two lock-in timer periods, one was 90 seconds and one was 30 seconds is.

B

C

Configurable, or is it hard coded.

B

I think it's hard coded uh this I mean.

C

B

Pretty easy to add that to configuration, but I think it's currently hard coded so 30 seconds is the time where I renew.

B

So, if I own a queue that I'm going to renew my ownership, every 30 seconds and 90 seconds, this is the timeout for uh the lock to expire, which means that if I'm down then at most 90 seconds after I'm down another rgw gonna pick up on this uh queue and and start to to own it and you know, serve it. um So what would be the concern like? Why would we need to change that? I mean it could be good reasons. I'm just asking.

C

So I was just thinking if a user has only two rgws in their cluster and both of them have.

C

I can both both these rgbws acquire lock on the same object, apparently not right.

B

No, no, I even try to kind of prevent all kind of potential, so the the lock is a cls action, which means this is something running inside the osd. So on a certain object, there's only one osd that can do any operation. This is completely atomic.

B

So you know if somebody is trying to take the lock on a certain object on a certain osd that only this code can run there. No other code can run there at the same time on this osd for this object. So this is why the locking mechanism is is guaranteed to work.

B

I also create you know a little bit of of jitter in those uh timeouts to make sure that there's no too much race conditions at the same time, exactly if the both rgw started at the same time, I don't want them to their cycles to be completely in sync, so I'm editing some. So it's not exactly 90 seconds. I added some g to there so to make sure that it doesn't really match.

C

Okay, but if one rgw goes down which had that lock, the second one have will have to wait for 90 seconds right before it can yes, so if a user has that problem, so if and they want to customize that and probably tweak it down to- let's say 45 seconds right so would.

B

It be a use case to yeah could be yeah. It could be, I mean, depending depending on the rates and especially if there are like only two rgws. This could be a valid concern too, because I mean what can happen is that if the rates are really high, then during those 90 seconds the queue can fill up and then all of a sudden you will start to get errors, and it's like you know- and you just waited for 90 seconds for no good reason, because you could have picked up on this queue in 10 seconds.

B

Maybe so um like I don't want. Like the reason I took those zombies because I don't want like false positives. I don't want, because you know one rgw is really busy with doing something for a couple of seconds and didn't have the chance to renew. Then I don't want it to lose the ownership for no good reason, but 90 seconds could be a could could be decreased. I think I mean you can submit a an r3 in the attack in the tracker for making this timeout configurable or even making the hard-coded numbers smaller.

C

Sure I I think making it configurable would make sense, depending on the use case and the number of rgb views they have. I will add a tracker.

B

Yeah that'd be great.

A

So maybe you could speak on uh like future development in terms of this feature, you mentioned uh currently right now to see the q's ownership you have. It pulled up right now in terms of uh running this particular command. uh What is the plan if any for quincy.

B

Yeah yeah, so yeah I'd be happy to talk about some future stuff with the back notifications. So, first of all, um yes, I mean having to debug that myself kind of really proved to me that the uh we need a little bit of tooling here. um You know trying to to match something uh by scanning a log file. It really doesn't doesn't work so well so uh having a command. That really tells you, which one owes what uh it's gonna be great and that's gonna, be uh shortly added.

B

uh We also thinking or when you're thinking this actually kind of almost done and in in ready state that um we're going to add more endpoint types, so um as part of the 2020 gsoc project that we were part of there was a student that added uh aws endpoints. So um the student added, a lambda endpoint and an sns input, so sns is like kafka only it's an aws service and the lambda. This is like the serverless uh function of aws, so you can.

B

uh Instead of sending notification to kafka or to write mq like in an on-premises case, you can send notification to to an aws sns that would do whatever they want with it or you can even trigger an aws lambda with it. So this is almost ready, hopefully pretty soon.

B

um uh So that's that's another, and we're also talking about some other uh options uh I mean in there might be we might be adding uh so we're supporting uh mqp um the rabbit and q version of amqp, which is the the more common one, uh but there's also an mqp 1.0 version that is supported by activemq, so both the apache project and red hat products, support that and- and we're going to add this as another endpoint there. um So these are there's a couple directions that we have there.

B

I mean there are always fixes and enhancements to our to the implementation of communication with kafka and other things. um But overall, this is where it's currently currently headed. uh Another thing that is kind of on a back burner, but might also be in quincy, is um to send so, as I said, magnification center, where the object is put or deleted or copy, and so on.

B

um We also want to solidification uh when an object is deleted due to a life cycle policy, but this is currently not supported, but this would be a very nice enhancement in the sense that you know if somebody is interested in deletion of objects, life cycle deletion is also deletion of objects. So this is another use case, and this is also going to be added, hopefully in the near future, uh to as a notification trigger.

B

And that's it, I don't have any, but I mean this is this feature is quite really picking up in the past years or so and the more it picks up the more feedback we're going to get we're going to add more features and and make make the the users of this feature as happy as we can so there'll be other enhancements for sure yeah.

A

And thanks for mentioning uh the google summer of code, uh I did put a link into chat in case people want to take a look at that. I noticed uh as well that you have yeah those two projects. The next level advanced message queue protocol project as well as going gnats. So that's pretty fun.

B

Just one word so um that the going that is an example of of how maybe we wanted to add notifications for endpoints that are not that important. I mean like one-offs, so the the idea there is that you know you have your like nas, it's like a kafka that it's a it's a cloud native uh like a cncf project, whatever it's not very commonly used, so I won't go and put that in our code base. But um you know, maybe somebody really wants to identification to it.

B

I mean so we want to enable that, via like a scripting mechanism, that is more um kind of uh ad hoc uh integration point than the writing system, putting that in our code base and so on. So that's been an example of something like that.

A

And just I apologize for my ignorance with this feature in particular, but is this uh something that might get surfaced up to be configurable like in the cf dashboard as a operator usf cluster.

B

So uh there is work uh to add bucket notifications to rook.

B

It's uh it should be, I'm not 100 sure where it stands out right now. I was involved in the design process and we're going to add that to the crd. So whenever you define a bucket, you can also define, I mean you'll, be able to define topics and then tie the topics to the bucket buckets via notifications, and that would be part of work. So yes, currently sometimes defining uh magnification could be a little bit cumbersome.

B

I mean you have to write a couple commands, um although you can use the standard tooling for that, so the aws cli tool is and border three. Both they support uh the topics and everything. So um it's not like you have to craft your own rest client for that you can use standard tool, but still riding a yaml and pushing that into uh into uh an open shift or kubernetes cluster is easier, so um the the design for adding that into the uh bucket or bucket claim. I mean there are a couple of things there.

B

Crd was was done, and hopefully this will be implemented, I'm not sure when. So I'm not really in the loop for the implementation. There.

A

So for the operators they're tuning in uh what exactly for enabling this in pacific, what steps do they have to take? uh You did mention the crds inside of rook, but I don't know if that's actually necessary.

B

I mean if, if very I mean, if, if this is, if this is just you just running a step cluster, then everything is supported via standard interfaces, so you can use the aws cli you. You can't really use the s3cmd, because the topic and stuff those are not s3 commands. Those are sns commands. Those are the standard, aws tool set commands, but they're not s3 commands, so you need the aws cli or both three, both the it's the same thing pretty much um and and uh we support like we work with them.

B

Both um I'm gonna put the link. I have.

A

I had somebody who linked uh the actual docs. Would people be able to find that in the latest stocks.

B

Well, in the docks, we're kind of more generic we're not talking about specific toolings, but.

B

Here we have explanations about how to actually do that, specifically for both three and aws cli. So um even like we, we extended and changed the interface a little bit like we have more than the standard anywhere support, but we also provide a file like it's a json file, which is an extension, and if you put this file in place, then you're gonna, you're, gonna automatically get all of our extensions to the standard tool.

B

You don't need to change one line of code in the in python or in this in the tool and using this file you can actually um get all of our extensions and modifications to the to the interface, so it shouldn't be too bad. I mean, if you don't need to craft your own messages or or curl based client or whatever. You just use uh this uh aws tool uh all border three, if you use python and and things should be just working, so it shouldn't be too bad.

A

Well, uh so if people wanted to get involved uh and join you for future development in quincy, uh how may they get in contact with you, I'm assuming on irc and the seth million was but.

B

Yeah I can paste my email I wanna see my handle. Is s okay and.

A

And I'll put these all in the video later, so people can.

B

Just find them on there, and I can I mean you can also put my my red hat email. That's also great.

A

I could take care of that. Okay, no problem.

B

A

All right all right, well great! Thank you so much for taking the time I understand you know you're, taking especially after hours now you're in your own evening, so I hope you do enjoy the rest of your evening have a relaxing time, and I thank everybody as well for joining us and tuning in for this neat feature inside of pacific, and I would also like to remind people that we have ourself a user survey that is ending april 2nd, so please fill it out.

A

I think that would also help us in certain initiatives like this, as well as understanding how people are using stuff, so all right until the next uh stuff tech talk for next month. Thank you, everybody for joining us, and that ends everything so.

B

A

B

For the pleasure being here.