Ceph Ceph Tech Talks, 24 May 2023

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Ceph Tech Talk: RGW Lua Scripting Code Walkthrough

Description

Presented by: Yuval Lifshitz

Join us monthly for Ceph Tech Talks: https://ceph.io/en/community/tech-talks/

Ceph website: https://ceph.io
Ceph blog: https://ceph.io/en/news/blog/
Contribute to Ceph: https://ceph.io/en/developers/contrib...
What is Ceph: https://ceph.io/en/discover/

A

Hey everyone welcome to uh Seth Tech talk, it's been quite some time since we've had a tech talk, so thank you all for offering to give us some content.

A

If anyone is interested in giving a tech talk, please reach out to me I'm Mike at stuff.foundation, and you can probably find my emails on the Seth mailing list as well.

A

But let me go ahead and introduce you Paul. uh It's going to be speaking on Lewis scripting in the rgw, and this will be the Code walkthrough. So you off. Will you please.

B

Hi thanks Mike, um so yeah. That's what I'm going to do I'm going to do a Code, walkthrough um I'm not going to give too much background or demo. uh The little scripted on the rgw um I have a couple of presentations about that. uh So um I had a talk in open source Summits in Austin last summer, where there's a full explanation and demoing of the real scripting and um and the other resources as well.

B

But this is really focusing on the Code walkthrough and the reason is um that uh the way I envisioned the scripting feature is that it should grow so I'll explain in a second, but it has, it can grow in two Dimensions. One is um add more places where we put lower hooks So, currently, there's only a few places, but if people think there are other places that could benefit from more lower Hooks and I want to show how the newer hooks are done.

B

So people can do that in other places as well, and the second thing is even if a lower hook exists, maybe people want to expose more data or to allow more actions on the data that is exposed so I'm going to show how you add more Fields make the fields writable, for example, all kinds of things like that that again, this is a work in progress, so the the more people know how to do that uh to be more more effective.

B

um So, um let's start the first thing I'm going to show like the presentation just have like two slides. So it's mostly looking at the code, but um I want to give like a very quick uh map of the different files. So you would know which files has to do with Lua and which files kind of uh do what functions with Lua. So I guess the most important files are the files that are actually implementing the data access to the in the different contexts.

B

So, as I said, uh Aloha, it's a scripting language, but it has to run in a specific context. It it can't just run and touch whatever it wants, um and you have to expose from your Superfast code, um the values and the the data fields to this specific context.

B

um So the most important files are the rgw lower request, and this is what exposed the S3 request context. We have two concepts: we have a pre-request and post request. uh We have the rgw Lua data filter, and this is also kind of in the request context, but it's slightly different. It is for put and get operations. You get access to the actual payload, while in the request you don't have access to the payload, you have access to all the fields of the request itself.

B

um There is a little background context. This is something between the POC. It's not completely mature I'll explain that in a second, um so those are the important files that actually do the heavy lifting of the implementation of.

C

B

um And there are also files that kind of support the whole the whole operation, so um in rgw admin this is Advanced Gateway command line interface. uh This is where you put the scripts, and you can also do package management if you have, if you want to have the um low package dependencies that use in a script. For example, you want to use Json in your lure code, then you need to get the Json package.

B

Rgw lure is what actually implements this script and package management and because they, both the the package names and the actual content of the script, like the actual text of the scripts they're stored in radius object. So we have the rgw cell radius file, where we actually Implement um putting those scripts into radius objects. Rgw cell. This is the high level interface, because now we have The Zipper architecture, where you can store things not necessarily in reders, but only where this implements uh The Lure bits.

B

So this is for the script management um and last but not least, we have the um rgw process and rgw app, so rgw processes where the uh pre and post request context uh scripts are being invoked. This is what really calls the hooks and rgw op is what calls the data hooks for the put and get data scripts. uh The background um context is being invoked from well, I mean it has its own threads or invoke it invoked. It invokes itself, uh but the whole thing is being set up.

B

uh This special thread is being set up in azurew app main. uh This is another thing that is important in app main. This is what installs The Lure packages that you configured by the Admin CLI um and- um and that's that's pretty much it. So this is really a high level kind of mapping of the different files that um that are touching Lua.

B

um We we do plan of adding more functionality with the script management, so you can have more than one script per context, for example, things like that, but this is really the the core development of Lua. Of course, you're you're welcome to system as well, but what I usually see is, uh as contributions from the community is, as I said, adding more hooks, adding more Fields, so that would probably go uh other in the um lure request file or in a new file.

B

If you want to add a new place for Lua hooks, um so the Code walkthrough is going to go through to all. Hopefully all of these areas um they I'll start with the script and package management.

B

um Then I'll explain a little bit about the package. Loading uh then we'll go through the request context, the data context and the background context. So that's pretty much what I have for the presentation? Okay, so let's start with the um the script management. So um please, let me know if you, if uh you want me to increase the font, but uh if this is okay, then I'll stick to this.

B

um So we have a couple of new command in redis, Gateway uh admin, CLI, uh there's a script put. This is what really uploads the script and what it does you give it a file. So the file is just a lure file that you want to upload. It reads the file, it does something called verify. This is very basic verification. So what verify?

B

Does it make sure that you don't have any syntax errors, but it doesn't mean that there are no errors in the script and the main reason and I'll show that later is because all the parsing of the actual lure is lazy.

B

So it means that if you have a really ugly syntax error, then then the verify function would would catch that, but if you're kind of accessing a field that doesn't exist or trying to write to a field that is a read-only field or something like that, then the verify function cannot cannot catch this because we're not actually building the entire uh structure of fields, because we have no idea if we're actually going to use them or not. So again, it's lazy parsing. So if you have some, you know garbage field name, but it's under some.

B

If else condition that doesn't happen, then the law script would work. Fine there'll be no failure. So this is why the verify cannot actually verify the full correctness of the script. um But if you have some basic synthetics or it would verify that and and return an error say, something is wrong. um So this is the verify function. So if everything is fine and and you um you- you provided the correct context, so you have to give a right context to the whole thing.

B

um Then we have something here called The, Lure manager. This is the the cell interface, that kind of abstracts the writing to the radius object, um and so you get the manager. And then you call the the right script and we can see we can see here- and this is the the rgwr file that the only thing that the right script does is that it's it's accessing this cell lure manager that is supposed to have something called a put script.

B

uh The only the only um cell manager that implements the putscript is the radius one. So you put that in a in a radius object, and uh this is fairly simple, so it just views the system object where it stores the actual lure code or Lua script. It's just a text file that is stored in a in a way this object. So this is fairly fairly simple and straightforward.

B

um This is the put begin, your script, you have another command for getting a script and the last comma command is to delete the script. So there's nothing special here, I won't explain too much is just uh storing the whole thing in the uh in those um redis objects.

B

um The commands that are kind of uh maybe more interesting are the commands for the package management. So, um as I said in your Lua scripts, we allow, uh for you know, using external packages, and this is very useful. Like click, if you want to send something from Lua, you need a socket package. If you want to do some Json processing, you need a Json package or any package that you want. There are different examples for how people are using packages and the way the packages are working. Is that well a Lua package.

B

Oh, we use a little package manager called Lua rocks and a little package is a whole bunch of things. So a lower package is usually a couple of Lua files together with some C files and make files- something like that.

B

So, usually most, if you look at most Lua packages when lower rocks, tries to install them, it compiles the C code and copies over some Lua files, which are just text files, and because of that, we can't really store a lower package um inside SEF, somehow like we did with the with the little script, without just text files, because it's compiled code- and we have no idea like where you're going to run that.

B

So um the only thing that we do, when you add package or remove a package, is we add or remove the package name? So when you call the admin select command and tell it to add package, we do write something into where this object. What you're? Just writing the name of the package that you want to add um and the same. The same goes for the remove, so we're actually not um not adding or removing anything we're just writing the name of the packages to a file. So that's um that's about the the management.

B

You would also see that everything has to do with lower packages is protected by Nick staff, and this is could be macroed out in cmake. The reason is that packages they could be risky, so uh you may decide that you know if you have your own uh version of of red, skater or saf, and you want to support Lua. But you don't want people to add packages that would kind of allows the lower code to get out of the sandbox and do all kind of risky things.

B

Then you may you may want to disable that completely. So this is why we have those if deaths um and other than that I mean all the commands of the remove package, these packs and so on. They follow kind of the same, the same concept and you you would see here um like the the ad package and the remove package in the cell radius file.

B

It it just uh updating the um well we're using the the omap keys to for the packages, we're not really storing that inside an object, so it is easier to just uh search for them and remove and add them as those are just uh kind of attributes. So these are the functions that actually Implement that Invaders.

B

um So this is about the script management and the package management. Next thing, I'm going to have a look at is um okay. So how do you actually do the packages themselves, uh so in rgw app main, you would see uh the code that does that and what it does is again protect it. Under this kind of, if death there is a command to do, install packages, so install packages happens only on the startup of the um of the redis Gateway, which I agree. It's it's pretty ugly.

B

um It means that if you want to add or remove package, it would only take effect. The next time you restart the rgw and.

B

The reason for that is that the installation of the package is uh it's something pretty heavy I mean in the future. We probably would fix that and allow for a specific command to go into this installation, but overall you we cannot just install a package when somebody puts a new package, because uh that would this actually requires it invokes the lower rocks command.

B

That can cause uh compilation and all kind of things to happen uh so that that has to be done with care um currently only on Startup, but it probably would be fixed in the future.

B

um So you don't need to restart the rgw for that to take effect. There's also something here that has to do with uh um isolation of those packages so usually lure rocks when you, when you use it, maybe we'll go through this file, so so you see the actual limitation.

B

um So usually, um when, when you install something using lower rocks, it would install in some canonical place like I know, slash, opt or Slash use or something, um but we want this installation to be isolated just for this specific rgw. So, for example, we have multiple objectives running on the same host or if you have lower running on the host, for whatever things that you want, we don't want to, but.

C

A

You evolve your uh cutting out right now.

B

Sorry so maybe I'll, just repeat um yeah information of the packages is done in a specific place for this specific rgw, because we don't want to interfere with any packages of Lua that were installed on the machine. We want to to allow for multiple rgws, that running on the same machine, Tunes install pack, ages and.

B

We're looking for for lower rocks and then here is the kind of the magic happening so for each package: We Are, invoking The Lure rocks uh file and where um installing we can have a specific version of the of the pack and the minus three command is telling what is a specific path where this is being installed and then later on. We also want to see if there are issues and so on.

B

So we kind of reading the standard, error and standard out from the Blue Rocks command and um kind of clicking the information and putting that back into our logs. So if, on some reason the installation failed, then people would have the information of how to debug that so I think this covers uh all the bits that are needed for the uh um management. So we know how to how the code works for managing scripts.

B

We know how the code works for managing packages and what actually is happening when we add or remove a package now what we do here because of the adding and the removal. So everything is kind of easier. Whenever we're calling this code on the rgw startup, we'll remove all packages that were ever installed in this specific directory and we're reinstalling all the new packages, um all the packages that that we have configured.

B

This is how, like the removal and everything works, easy um if we would like to do that in a more Dynamic and and efficient way, uh we'll have to think of how to do that. um But when it's happening on Startup, this is probably okay.

B

um um I don't know, maybe I'll I'll take questions now before I move to the actual uh implementation. This is more about the management of the Dual Scripts.

B

Okay, maybe if you have a question, we can do that at the end as well. Okay, so now we're getting to the uh more interesting thing. So um this is the pre and post uh request um context, and this is the the context most people that are using the Lewis scripting on the RW are using.

B

um So how do you do that in rgw process? Rgw process is actually what's kind of hooks or ties together. The front end the Beast front end with the actual implementation of the different S3 operations, and this is where we are doing this so before we actually do the operation. We have the pre-request context, and this is before the operation is being executed, and this is why you don't have access to some things here.

B

So you have access to everything that came into the rgw, but you don't have access to things that you're fetching from the actual object or bucket or anything that is actually stored in Raiders. So this is why people- sometimes they write code in the pre-request context, but they don't have access to all kind of things, and this is this is why it is happening because this is before we executed the request or the operation and the the whole process here is really pretty straightforward. We read the script for this pre-request context.

B

If there's no script, there's nothing to do like this is the common case where the there's no script. um If there is script, we're calling it The execute command, and um this is in the prequest and in the post request. It's pretty much the same, so this is after the operation is done, so everything is done. Everything is complete. This is before we reply. Whatever we built with the reply, so you still have a chance here to change things with the reply.

B

So if, for example, you want to change the error code of the reply or change something some attributes in the reply or whatever, then you can still do that because the script runs before we actually do the reply, but after the operation is done. So there's no point in kind of saying: oh, this put request, failed and I didn't put anything because the put request already finished here, so you cannot undo whatever the operation uh was was already doing, but you can change the actual value that you're applying.

B

So and again the processor is simple: you read the script. If the script exists, you call execute. So, let's see what execute does.

B

Okay, so this is the execute function and we'll go through that in a little bit of details. So the first thing that the execute function does it creates a new Lua State. This is what creates the Lua VM. So this is important to understand. We instantiate the Lua VM for each request. So this is why there'd be no locking or race conditions or anything between the different low VMS, because we have a separate VM for each request.

B

um It does have some overhead, but the whole point about Lua or the beauty of Lua is that the VM is small in size and very, very fast to be created. I don't have the exact numbers, but it is. It is pretty fast. The next thing after creating the VM is we want to allow the standard Library. So what we do here and I think this is the lower utils.

B

Is that um we're opening all the standards lure libraries, but we're disabling some of them so because some of the libraries they just makes no sense in the context of running a Lua script on the request, for example, part of the standard library is loading Lua scripts I. Don't think we want to load the Lua script inside Google script.

B

um The the debug commands that again they don't make sense here, because you have to tie them to a c function.

B

We don't want people to um be able to exit, because this would just call exit and crash the redis Gateway and in the future uh we may uh have like some some um configuration flags that allows the standard library to be even more restricted, for example, the entire OS Library. We just disable exit, but you can do currently. You can invoke uh bash script from Lua now, whether it makes sense or not.

B

This is really debatable, uh but the good thing about this function is that you can control that and we may decide later on to kind of make it even more sandboxed and then it is today and not allow for other things. Of course, there are the standard libraries that always needed, like math libraries string, manipulation, libraries stuff like that. Of course everybody would need that, but this would not break the sandboxing of the of the uh of the luo VM So.

B

Currently we allow some breakage with the what they call the OS Library, um but we we may restrict that, furthermore, in the future, um okay, so we did that we set the package path if you're using some packages. This is the path that, where we installed the packages when the release gate, we started um and we create some action. So the actually, the only action that we allow currently in Lua, is to write something to the web log. So this is the create debug action and um we also allow um invoking the Ops log action.

B

So this is our own action, and this is just tying to an existing stuff um and there's some something about the background. I'll touch that in a minute, the most important function here is the create metadata meta table function and I'll. Explain our the overall concept that I have here with the the way it's structured, because this is again where you probably would like to add your access to fields and so on.

B

um So I'll touch that in a second um and then we actually, uh the the string is what actually runs the script and we have some error handling. So that's the execute function here. um So what does the create metadata meta table does?

B

So, let's have a look here.

B

B

The idea that we have here is that, like in Lua, you have tables- and you have uh meta table so table is usually where you store your data and make the tables those are like um allowing for polymorphic behavior on the tables. So what happens when you access the table, write stuff to them, read stuff from them. Iterate them count the side and so on.

B

The thing is that we don't want to copy data around, and the data exists in our C plus plus structures like we have uh in the rgw. We have C plus structure, holding all the information that you need, so we don't I. We don't want to copy them over to some Lua table, so our tables are actually empty. There's nothing! There's absolutely nothing in our in our implementation that holding data in any lower table. The only thing that we have are meta tables.

B

Why we need that because we're using the polymorphic behavior of the meta tables in order to kind of um allow the the Lua code to access the the or actually, the data in that we store in the in the C plus structures.

B

So when you, when this create meta table function, is being invoked um it, it does create a table just because you need a table for every meta table, but it doesn't use the table for anything um and the way that uh because I did not want to use like some inheritance hierarchy and so on so you're actually creating a meta table that is implementing the actual access to the different fields.

B

And what you do here, you just have the create meta table, function, templatized with a table, so there's no inhabitants um and you can actually, as long as you implement the correct functions, then then you're, good um and I'll show how it works. So, let's say that you want to give access to some field in the request, or you have to create the meta table for this field and then at the right place. You create the meta table.

B

You invoke this function and you pass it the um the value of the class that implements different ways you access them. The app values is like the different functions in this meta table class um are actually closures, so you have to kind of uh uh the it's like the like. You have the the capture variables in the in the C plus Lambda.

B

You have what they call up values here, so you you capture the actual references to the C object and then when people are evoking the different functions, they have the specific uh pointers already kind of in their hands. So this is why they can they can use them. So I'll show that in in a second. um So what do these does it? Do? uh It takes all those app values. Those are usually pointers to C plus structures. It creates a new table.

B

uh If it is a top level table, it gives us a name other than other than that there's no there. No, the tables are, don't have any names uh because they're just like um uh you know, the structure is like you, have something dot something dot. Something in all. Those fields that are kind of concatenated by dots uh are the different tables, but um they don't have their own top level.

B

Names only request is the only top level name that we have so this is why we give it a name but other than that we don't give it a name. Even so, we create a new meta table and then we start to implement the different callbacks or different uh kind of polymorphic hooks of the of the The Meta table. So the first thing is index index is where you have something dot or brackets in Lua and because the the table itself is empty.

B

So if you, if you access the table via a DOT or via brackets, it would not find anything so it will go to The Meta table and see what it needs to do to do there. And what we do here is that, first of all, we pass in all this captured values the the up values, and then we pass it the actual closure, and we know that the closure has to be called index closure.

B

So, in whatever class that you created here and you passed to this uh templatized function, this class, it has to have a function called index closure, and this function will get all those app values and would be invoked when somebody is accessing the table with brackets or with dots and pass it the index that is passed after the like inside the bracket. So after the dot and would also have like the app values stored somewhere, so you have the index, you have a new index.

B

New index is same as index only for um like writable values like if you want to write something into the table. uh Pairs is the way to you. You iterate over the tables and Len is when you do the Lua hash sign, which gives you the length of a table. So you need to implement all those closures or not I mean you don't have to like. If something doesn't have a length, then you don't implement it or you implement it and give it an error or something.

B

And then, if somebody invoke a lengths on a field that doesn't have a meaning for length, then it will just give an error to lower and the last thing that this method, uh this create meta table does is tie the table uh with the meta table. This is what it took so I understand. This is pretty abstract, so we'll go to see an actual example of that, and that would probably make it simpler to understand.

B

um So the first meta table that we create is the request make table now. This is a top level one, so it actually the table has a name but other than that. The fact that it has a name it's very similar to the other ones. So, let's look at the implementation, so I said that I don't do any inheritance. I do inherit it inherit from an empty meta table.

B

The only reason is that, if you're lazy to to um to implement some of the functions that doesn't do anything, so if you, if we look at what this one is doing the like, the the only reason we have this is that it implements all the functions that I assume that exist like the the new. Well, almost all of them. The new index closure, the first closure and the lens closure, and it gives an error if you invoke them. Otherwise, I'll have to implement that in many places.

B

The reason I don't Implement here the new closure. uh Sorry, the index closure is because you need that in each and every table or thing that you. Otherwise, why do you even create that? So um if you forgot to implement the like the writable one or the iteratable one or the length one, then I assume that you didn't want that and I'll just give an error. If somebody writes to a code that access it. But if you forgot to implement the index closure, then you have a bug in your code. You need to implement that.

B

Otherwise it's meaningless, um so the top level. One called request. Sorry.

B

The top level one called called request- and you can see this the name here um and uh you cannot iterate it. You cannot change values to it and so on. So um this is why the only closure that is implemented here is the index closure, and here you can see a long list of things, that of fields a subfields. So it means that um whenever you're, writing lurequest dot something it has to be one of those fields.

B

So if you write request dot, rjw op then put string just pushes the op name as a string, and then you can do whatever you want that in Lua, if you wrote requests, don't decoded URI, then I have this up value right. This is like what is captured in the closure, so I have this app value, which is uh the request State, and this is kind of a huge six structures, C plus structure, that is holding pretty much everything about the request.

B

So one of the things here is the coded URI and you can go on and see that uh whenever I'm, exposing something which is just a string or an integer, it's pretty straightforward. Sometimes, what I'm exposing is something that is itself a table, so itself has dots and and can have more more level of of hierarchy. So whenever I'm exposing this I cannot just do the lower push which just puts a value somewhere in Lua I need to create a new meta table. So what I actually push here?

B

I push a new meta table so or a new table with a meta table that, if somebody is, for example, writing request.generic attributes dot something it would know how to how to do that. um Generally attributes, it's kind of a map, so it's slightly more complex but I can take a simpler example. So let's say what that you write request.bucket.

B

So, what's going to happen here, I'm going to create a table that has nothing in it and this table is going to have a meta table that knows how to access the different things that has to do with the bucket. So we can have a look here and again.

B

It inherits from this empty meta table just to implement the things that are not implemented, and it's pretty much the same thing you see well, the bucket also have this up value. The app value here is again the request, State um and kind of one of the reasons it is. The request.

B

State and not the bucket is the app value because sometimes I don't have the bucket, but I still want to have the bucket name, um and this is why um I'm using like, for example, here, like especially like in pre-request context, um you don't have access to the bucket object at that point, but you still know what the bucket name, because it's coming in the request, so I'm checking whether if the bucket is empty, I'm taking the name from from here um and uh you can get the the Canon.

B

The name like you cannot get all kind of things from the bucket here um and some of the things by themselves are also another level of another level of uh of a meta table. For example, if you have something like request.bucket.user, then it goes another level, because user itself or held holds more uh more fields in it. So you can see here that the user has a tenant and an ID, and this can go on to whatever levels of hierarchies that that you want.

B

um So, if you know, I won't go through, of course, through all the all the fields and all the things in this uh in this place, but you can see that uh uh the way that you would like to expose new Fields is, you know, know how to access them from the C code.

B

Pass the CIS structures, as pointers as up values, to the functions and create a new meta table um kind of class, and you know plug it in the right place that you want that it makes sense to access um I'm going to show uh two more things here. One would be around um tables that are or field that are writable and the other one would be around things that are iteratable. It's iterate rule whatever it is: okay, one second and Okay. So let's see new index closure.

B

What's this one okay, so the bucket meta table actually does a lot of writing. But, as you can see, the bucket has lots and lots of fields in it. As we've seen before, but only one of them you can actually override.

B

So the thing that you can override is the name so for the the whole thing to be writable, then I need to implement the new index closure. So if somebody is doing a request.bucket, DOT name equals something. Then it would jump here like this. Is the The Meta table mechanism, the polymorphic Behavior Lua is allowing and we jump to this function and again it would take the the uh the app value or the the value that they're captured in the closure, and if somebody is calling uh bucket.name, then I'm allowing this this to be overweight.

B

um If I'm writing something like bucket request.bucket dot, end time and trying to override that, then it would fail, because if I'll compare the index and it's not name, then it's a other, an unknown field or you're trying to write into um something that is not writable.

B

So this is an example for implementation of of writing values or over, like here I'm, actually overriding the actual value and- and this is useful like, for example, if you want to to steer messages to different places, to different buckets than the original message, or something like that, and then you can do it here.

B

uh Pers closure: um this is a little more tricky. So the way that that you do uh iteration on a table in Lua you, you write like a four command and you you have like a special keyword called Pairs and you put inside the parenthesis. The name of the uh the table that you want to iterate over and what is resulting from the pairs are key value Pairs and that you should get like inside your to inside your photo.

B

If you can get a key value that iterates over the entire structure, so uh we have all kind of places where we have um uh like. You know simple Plus Code. We have something that is iteratable like we have maps and and other things. So this is a grand map.

B

um So again, in order to get that you, you have to go pretty deep, so um there is a ACL. So this is part of the request, so you have a request of let's say: user ACL, and this creates a table of ACL and in this ACL ACL itself is not iteratable. But if you do acl.grants, then the grants you can iterate over them because I have the pairs closure and the way that currently it's done, it's called a stateless iterator.

B

The way it's it is done so, every time you do an iteration, you pass the new key in value to the to the next Call of the iteration and then um I'm searching for them I'm, getting the values returning them. Sorry.

B

B

Sorry and the next iteration is taking um the key, and um you find you find that the last key you you move ahead, one um one value to the next iterator and then you pass the the next key to the next iteration. Until you reach the end and then you you return um like nil, and then Lua knows that the for Loop needs to end because we got the then the nailed key. So um this is the mechanism.

B

It's a fairly robust mechanism, because um you just keep on looking up new keys. um It doesn't work well with multi-maps, so this needs to be fixed in the motor map and it's highly inefficient because you do you do you do find every time. So you get like the next key and you increment the next iterator. But you forgot about the iterator and you just return the next key and and look up the next key.

B

So it's a very inefficient way to iterate a table so I'll be fixing that um I'll need to somehow uh create the iterator as an app value. So I can pass it to the next invocation, uh because the only thing that Lua is passing to me is a string key.

B

um It cannot pass me an actual iterator, so I need to store the iterator as an app value and and manage it somehow so, but that I'll fix that hopefully soon, uh but currently, this is how, if you want to iterate something over a table, so you have a structure which is actually uh let's say, an STD map or something like that. Then this is the way to implement that you have to just copy that from the uh from the examples that we have here.

B

um So that's uh that's about the the request context. I know it was, uh you know, lots of things to consume, but hopefully uh next time you look at the code, you get a better understanding and of course, if you have more questions, please shoot me. um What I've seen is that in the past people added stuff like or made something writable um and um the change itself is pretty straightforward and simple, so um it shouldn't be too bad to um to add to add things.

B

Okay, uh the next thing I'm going to show- and hopefully do that uh much more briefly is- um is the uh the data context. So the data context is invoked from the rgw up, so I allow to trace data only in put object and get object, so both put objects to get object. The way that law works is as a filter now not to be confused with the zipper filters or something um so the the put object and get object.

B

They have a filtering mechanism where you know currently it's it's used for uh compression and encryption So currently I'm, also adding the the lower the lower stuff there and um you can see where this is invoked. So um in the case of the, where am I the put object, um I'm invoking the Lua script before data is compressed and encrypt and encrypted so I'm.

B

Putting that as the last filter, because the way it works it's the last filter runs first, so I'm putting that last and in the case of the um the get object, I do the opposite: I put in the lower filter first, because um I want it to run last uh because I want that to work on the decompressed and decrypted data. So that's where I put that in the rgw op and what it does when it comes to kind of what kind of fields and so on it exposes.

B

It is much much simpler than what we've seen in the request, so it exposes a field or a table called Data. uh It's um it's! The the data is actually an iterateable table. That is just the kind of the the buffer list and you can iterate it byte by byte.

B

um It also have a request called uh sorry it. You also have access to the request. So you can you can access the request itself uh and also the background, but we'll we'll put that aside for now. um If you see the execute function, it's pretty similar, although much simpler than the execute function of the um of the request context.

B

So you have the the data field, the request and I also have the offset Shield I'm, not sure what it is, but it is something that is available there, so I just added it, but the important one, of course, is the the data field. The data field is implemented in this class, so hmm um the uh you know you can access the data using the brackets and those are just offsets like it's just a huge array, so data brackets you know 15 is the 15th byte.

B

You have to make sure that you've passed the right indexes because Lewis starts coming from one and we of course start counting from zero, and you also have like the the iteration one, because you can actually iterate the data so you're going to iterate that byte by byte- and um this is how we expose it. The the key here is just the index like just the running counter of what you're returning and the value is the byte value of um uh the back value of what's inside there.

B

So um and of course we have the length. So this is the the whole thing about the iterating, the data. Now it's not iterating the entire object, because it's just iterating whatever is coming inside the request. So whatever is an in-flight inside the memory of the archw, so not actually iterating an entire um S3 or array. This object, just whatever is coming in the request.

B

um What you can do with it I'm not sure I have uh in this open source. Summit I had a demo showing um how I do entropy calculation on data entering the redness Gateway and trying to implement some kind of a algorithm. It's really a demo, not something too real to detect malware based on the the level of uh encryption of the data that is incoming.

B

um So, for example, you can do this entropy calculation, just by traversing, the the this buffer is this data coming into the rgw.

B

um Last but not least, is the background context. So I'll do that really quickly, but the background context has two things: it has, first of all a context by itself. It's just a thread that is periodically running um running a law, script that is put in the background context and because it's just a special thread, it doesn't have access to anything almost so it's kind of useless.

B

However, unless you just want to do something periodically and I don't know in in Lua, but there's also the like a global table that we call the rgw table and the background context does have access to this table, and the thing is that all contexts have access to the table and, unlike what I said about that, every VM is its own kind of sandbox world, with no access to any other VM. This is actually a mechanism to allow um communication between different dual Scripts or different invocations of the same dual script.

B

So, for example, you want to count how many times you ran a Lua script in the pre-request context. There's no way to do that, because if you do that in Lua, then this information is lost when the luo VM dies and the lovian dies when the when the operation ends and you send the response back unless you put that in this rgw table. So this is a global table that survives.

A

They evolve, you cut out again.

B

That's right: sustainable they're, all synchronized of this mutas, which is very inefficient, um but it is, it does give you something um the table. Of course you can read and write from it, so it has the index closure.

B

It also have an increment and decrement atomic actions. The reason is that, because it's a shared table, if you try to read increments right from Lua, then there's no atomicity. That is guaranteed because it can be reading one value incremented that Lua by the time you wrote the value.

B

There was some other instance of a Lua script that already changed the value, so we added Atomic operations that are done atomically inside C, plus that does the increment in decrement, but other than that you can set and get values and, as I said, it's really a POC level because of this Global lock, uh it does have some nice features.

B

It is using like when you, when you write or read, values from the table, the values could be um I mean the values doesn't have to be.

B

um The key has to be a string, but the value could be a string or an integer or something else, and we have some nice mechanism to do that, but I won't get into that now because I think we're running out of time. So this is pretty much the end of the this. This presentation.

B

uh We have like five minutes for questions. If you have some questions and and if not feel free to shoot me emails or ask me questions on slack or anything.

A

D

To put questions in chat.

A

A

It's either um Tom Bennett.

D

Yeah, this is uh obviously a very interesting um talk and um yeah it's. We are certainly um you know for implementation. We are we're going to be using. uh We were quite quite extensively. um I I just wanted to check uh I I, didn't I I mean obviously I know about the uh the requests um uh through Chris field that you can access. um I, didn't know much about the data and the uh the background context with the uh what's it the Red House gateway table is there?

D

um Is there any documentation anywhere that kind of just um or example, code somewhere that that uses this? So we can so I can kind of just get a feel for how to how to use it.

B

I'm just checking the the documentation, like the official uh Upstream dock I'm, just gonna paste that so I see that I have a I've mentioned the background and the data context.

B

But maybe I didn't write the proper documentation for.

A

B

Field, let me check.

B

Yeah, so you can see the the background context. There's a you know. It is documented, maybe not not well enough, but I have uh something called background context section there, data context um and I have the global rgw table explaining about this table and even mentioning of the increment and decrement functions. Yeah.

D

Okay, um I've um I've, just I've just seen that now, okay.

B

Yeah, um maybe it needs more more example. Oh I even have an example I see. So if you scroll all the way down, I've even wrote my uh object, entropy function, that is using the data context in order to calculate the entropy of the the buffer list. So you can even see that cool.

D

Thanks yeah! That's! That's! That's! That's! That's awesome! Okay!.

B

A

D

Yeah I had a follow-up question. If there aren't any other well, actually just a comment, if there aren't any other questions, sure yeah, so it's actually um because you're talking about so something that we want to do is we we we want to manipulate requests coming in and when somebody does request that you don't want them to do um we we want to be able to return an error as opposed to complete the request um and and obviously I mean from your talk.

D

You know you've confirmed that you can't do that, but um we've we've decided. We want to do that. So uh uh um I thought you might be interested that that what we do is we. um We actually just manipulate the storage class, um which is a very simple way to um so. We we manipulate the storage class and then uh return an error message with some bits of information.

D

um So what we do there is we we create a storage costs or in this simple example, we had set something called unavailable, um and so that's our storage class that is defined inside the redos Gateway. So um we were very easily able to do things like um like um stop delete. You know if we went to like a a worm style, um Red House Gateway, where we we write stuff, but you never want to delete it.

D

So we want to absolutely be guaranteed that if anybody comes in and tries to delete stuff that they can't um and I mean it's, it's amazing for with a couple lines of code, how we could very easy to manipulate the request coming in and say, I'm sorry, you can't actually do this um and then guarantee that no matter what happens, we can't you can't. Actually um you can't actually delete data. um So it's it's already proved for our use case and I know.

D

We are a very particular use case, um a very powerful tool to manipulate the requests coming in and um and obviously then look at requests going out. um So it's yeah, it's been been extremely useful, um yeah I've, you know I certainly think we'll have um as we as we move on and develop more code. We'll certainly have more interesting questions, but this has been extremely useful.

D

It's been a great kind of look in and and see kind of um that when we start actually wanting to understand the stuff where we can, where we can look um so one other question is, is where and where in the where in safety actually store the um uh the bits and pieces that make up. um You know the uh when you come when you, when you execute the lure rocks uh where in SEF is that is that actually stored.

B

Okay, so the the thing is: um there's maybe I'll I'll.

A

B

That so it's probably here, okay yeah! So when when the rgw is starting, then I'm. Invoking this uh lurex install and minus minus three is the lure rocks path, and this is where the the whole thing is being installed so uh and the path is a very specific one.

B

um It's like specific to the rgw I, don't know exactly, but you can see it in the code. What the what the default path it! You can change it. It's a it's a config variable, so you can change your path to something else um and um yeah. There were a couple of issues that you guys found in containerized deployments when it came to the Rocks uh I, hopefully fixed, that I had a couple of PRS to fix that, but yeah I I, don't remember what the default path is, but maybe it's here somewhere.

D

Yeah yeah, so so that's that's the path in the on the file system, uh so as more interested in where it's actually being you know, as.

B

Well, so, instead from not destroying I'm, not installing the I'm, not storing the packages, as I said, the packages are compiled, C code and couple of low functions so I'm, not installing the packages or storing the packages itself at all. The only thing I'm storing is the name of the package in in uh and I. You can see it here. What is the.

D

B

Yeah so I'm installing the name of a package if you yeah. So this is the ad package. There's a radius object and um I store it as a an omap attribute for each package. So.

C

B

Package is a no map attribute. This is how I did that.

C

D

Yeah thanks: okay.

B

C

How's it guys um thank you, you're well, um I've just got a question regarding the the lower rocks and and the whole packaging, um or maybe it's a more of a suggestion, but wouldn't wouldn't it make more sense or would it make sense to to package all of the lure rocks outside of uh the rgw and just upload a big tarot file that gets kind of unturned in a certain place, and then you run from there.

D

C

Say I I, don't really know, I mean the times that I've used lower rocks I've got burnt with it, um but you kind of have a main.lua which would be the code that the rgw that your C code will call and all the packages that it needs. I know that I mean there's certain linked libraries and stuff that you might need in reference. That could be a problem um but yeah just asking.

B

Yeah so I think it should be doable, I mean even in the current case, so um you'll have to install it where, where we're looking for it so I, when you add a package, the only thing it does, it adds us to this radius object or this omap value, and then it is using lurex to install it in a certain location, I there's nowhere in the code that I'm checking that in your Lua script, you imported a package that you added to the list. So in theory, you can do your own Packaging.

B

This tar file go put that in the right place. You have to put it in the right place like in the where, where the rgw is looking for it and then don't add any packages in the command line.

B

Oh no, no you're, right I'm, deleting everything there so to allow for external packages, kind of mode we'll have to I'll, have to change it right, I see yeah so right, so the yeah the install package of script. What is that the first thing it does? It removes everything it calls removal.

B

um So for that to actually work, I'll have to have like a a flag saying don't do the removal and then it's like external packages, it's your business.

B

um So if you open a Tracker, this is really a very simple fix to do, and then we can have this external mode where we're not using the rgw to install packages or anything. And it's really up to you to do all the work.

C

Yeah cool I'll catch you offline about that. It's just a thought that came to me.

B

Yeah I think it makes perfect sense yeah because I agree sometimes I mean there's a reason why most of the code in this install package function is grabbing the STD error messages coming out of Lorax and exposing them to the user. This is because many many cases the lower rocks, fails for all kind of reasons and and um yeah you may say you know the don't I don't need your favors I'll package.

B

Everything make sure that it works on my system and you'll just use it, and- and this is a this- is a valid, failed approach here.

D

Yeah thanks I mean just to add what what Martin said. I think that you know the context is that we have containers that we we run a containerized, um deploy deployment of of Quincy and uh it's obviously in our desk gateways.

D

We need to Now package in GCC and make um because every time we restart the Lewis every time we start the Gateway, it's it's completing UK container and we then need to so we always have to make sure that we have all the build packages which is which is not ideal, um but I mean we're loving it at the moment, because it's I mean. Obviously we want to use this stuff, but um but I think Martin. If I'm correct. That's the that's the context.

B

D

B

um It makes sense, although, although I've asked the the owner of the safe container to add all the bits that were missing so in in the next version of Something of it should be there already installed like uh make and and GCC and Lua rocks. Everything should be installed there, so it should. It should be okay, uh but but I agree, I mean what you suggested. Regardless is a good suggestion, and I should probably do that.

D

Yeah great thanks sure.

A

Any other questions.

A

Okay, I think you've all for providing us content for our Tech talks and thank you, everybody for joining us both on uh jitsy and the live stream as well, and this recording will be put up uh relatively soon.

B

Thank you. Thanks, Mike see you guys.