Status SWARM Orange Summit, 1 Aug 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: The encryption implementation

Description

In this presentation from Day 3 of the #SwarmOrangeSummit, Balint Gabor presentation came as a continuation of David’s talk and centred on the implementation of encryption into Swarm with a thorough technical walkthrough.

A

So hello, my name is Valentine I'm part of the swarm team, since December and I will talk about how I implemented the encryption, but Daniel had a presentation before lunch into this form good days and as the Daniel mostly talked about hi it works. My.

A

It was not easy for me to decide what should be a useful topic for this talk because I wanted something technical about implementing it, but like I guess most of the audience doesn't have any personal, hands-on experience on the swarm code. I did not want to just I, don't know brought some code, although I did not want to talk about how it works, because that's already covered I mean I'm, also not a cryptography expert. So why should I talk about that?

A

So I wanted to make a little bit more general point about the experience of starting to contributing to a big complex project and, and how can you add, the new functionality to a code which is way too complex and big for you to refactor? Although you have problems with it and yeah, you know you are I'm pretty sure that all of you have been in this situation that you have to add the feature to a code, and then you realize that that code is convoluted. So you have like two options.

A

One is to add your feature to it and increase the complexity and negative, inverse or refactor the thing. But probably you don't have. Oh sorry,.

A

Actually I think I have an app for that.

A

Or it's not not instant, okay and then I will just try to be more active in my computer. So so you have the option to either to refactor it and yeah. You don't have time for that, so you cannot do that then. Maybe I should just add, but that's the complexity. So what should I do and I think there is a general pattern here. What you can do in these cases, and that is to get this- is the code but is way too complex and you. You cannot really understand that.

A

You know that you have to write things into it and one of the options, if you cannot retract or the whole thing, is just to separate small parts which are responsible for something and try to extract those like okay. This is something what I understand I could obstruct, and this is something which I also understand is just a very small part of the whole responsibility.

A

This is another something let's extract these and then I have three components which I like and then there is still something which is nasty and complex, and you don't understand, but at least it's smaller and and it's separated and then you can add extra defined boxes for your stuff. So basically, this must be. This was the strategy. What I tried to follow when I, when I added the encryption to experiment? Why why it was hard to explain this I have to talk a little bit about this form code.

A

So how chunk retrieval work in this form? Well, there are all these outer layers which are actually not complex and I. Don't really want to talk about them like there is an HTTP interface and then we have to think about how to change the API. So it's at an extra the extra information, whether you want to enter it or not the the thing what you wanna upload or on the retrieval part.

A

You had to accept these long references reaching to the decryption key that was easy, but inside what I found that there is this thing called DPA, which has a retrieve and store function. Basically, this is this form basic form. Functionality like you, have a key which is the hash, and then you get the content or you have the content. You want to store it and you want to get back the key.

A

So it was kind of clear that, but you have to happen here that that the store function will has to have an extra parameter, whether you want to encrypt or not, and the retrieve function is easy because you don't have to change the the API there if it's encrypted, you have to decrypt it for sure.

A

But here you can see this thing called chunker and- and that was that was something complex, and it was clear from this that probably I have to touch that. There is also this thing which will be important later. This is the retrieved channel and the store channel. These Argo channels and the chunker itself puts information there onto these channels and then this same BPA object is responsible to retrieve or store chunks. When something arrives on those channels. Yeah yeah, it's a distributed, pre image archive.

A

Yeah, so that was like one of the barriers to understand this code that actually, even last week, I asked Victor what is DPA I forget yeah, so the problem is the chanc are so what is the chancre? So the chunker is the component which is responsible to split and join your content into this 4k chunk and and build the tree, but is already described by several people, including the onion in the talk.

A

So this is the content, and then you split it into chunks, it hashes all the chunks and, and then these intermediate chunks they contain the hash or the reference of the of the child chunks and also all the chance always for the first. Eight bytes contain the span, which is the size of the whole subtree. Under this chunk.

A

So basically, everything what's happening here is done by the chancre. So what is in one chunk? As I said, there is a span which is the size of the subtree, and there is data. If it's a leaf chunk, then it's actually the date of the content, what you want to store or retrieve in the intermediate chunks, those are the references of the children chunks and then we encrypt as you as you saw in the encryption presentation of Daniel. Then we encrypt.

A

First of all, we have to add extra random bytes to the chunk. If the content is not 4k this way we can hide the even the length of the content, so that's the padding and then we separately have to encrypt the span and the data, and we just concatenate those two batteries together and then we have an encrypted chunk.

A

Just a note that, because the span is also encrypted, it's not solid anymore, so in the first eight bytes, so the nine cryptic chunk is not the size of the subtree. The original length is hidden.

A

So, in the encrypted case, this these three looks a little bit different because, like the chunks are encrypted and and these intermediary chunks they contain not just as reference is they contain that just hashes, but the hash is always concatenated with the encryption key for that chunk and I want to emphasize this, because maybe it's not good that all the chunks are encrypted with a different, angry bianchi.

A

So from this, it's clear that the branching ratio of these three is different in the activated case, because the references are longer and the intermediate chunks are still just 4k, so they can contain only half as much references. That's why this tree here it has a the deficit, is bigger for this example yeah. So.

A

What is a chunk, and here happened the first problem, because my original naive idea was that Oh encryption will be easy, because basically, nobody has to care about whether the content encrypted or not, when I save it into a chunk, which is an object somewhere. The chunk will encrypt its content by itself, because all the chunks are separately and shifted, and here I bumped into this issue, that this is the chunk in the code, and if you know the go language then do you know that that in go weather field is public or not?

A

It's defined by whether the first character is copied lower or upper case, because as data which is the actual content chunk, including this including the span and the and the data, is capital, it's a public field and I have to tell you that overall, the swarm code is as data is written manually like the first eight bytes are indexed. The rest is indexed right. Everybody is adding to this thing, so it's unfortunately, the chunk itself does not encapsulate its structure.

A

Well, it doesn't have an and well-defined API and because of this, it's not easy to add the encryption on the chunk level, because I would have to modify all the code which accesses change. Chunks and I can tell you that it's everywhere, like in the network, layer, storage, everything and I, for example, I- did not wanna touch anything in the network layer, because this is this- should just affect storage, so I had to give that up and then I. Then it was trivia that it has to be implanted on the chunk Oliver.

A

But what's the problem with the chunker, the problem with the chunker that it's very complex, so everything what I described in this one big chunk of code? It it splits chunks, each joint strength. It built this whole tree, that's pretty complex code by itself. It also does all the hashing it instantiate. This chunk objects feel there this as data and also it it puts them on the retrieve and store channels for the VPS.

A

So if I would just just to add something on to this code right and it implement, it tries because we have two different genders. So, just to add this functionality on to this already, like I, don't know the thousand lines of code. I don't know, I did not want to do that. So what's the solution, so let me again show the overall architecture, so we had the BPA that caused called the join or a split on the chunker and then the chunker splits or joins a chunk hash.

A

Is the data instances the chunk objects and pushes them on the chatter for the BPA to retrieve or store it, and then in the DPA there are workers were instantiated at constantly listening on these channels and retrieve and and store chunks. So the chunker in this case is the blob on my first slide and I wanted to extract stuff from it, and it was clear that, for example, hashing is something which can be extracted. Maybe I could eliminate the whole chunk instantiation and this channel based communication, which is also hard to debug and complex.

A

So these were my goals, so I I did not honor that just put the encryption decryption to the chunker I could not use the chunk object easily, as I said, because of the public as data, and it was also a goal for me that I wanted to write a separate encryption package which just stopped, which is 100% independent from the swarm code. I did not want to rely on the chunk type or or the chunk or any other type.

A

So this is what I done. I created an encryption package and, as you see it's not on chunks, but on byte slices, it's it's. It does not depend on any others form packages and it can just receive a byte slice and the key and encrypt it or the other way around, and it can be initialized with all the parameters. What we need like how long is the default chunk size because we had to add the random bytes to it and the initial counter for the for the counter mode, encryption and the ND hash function.

A

So this is the encryption package. This is the interface of the encryption package and the other idea was to to separate this whole, storing and retrieving and and hashing and chunky instantiation from the chunker for this I created two very simple interfaces: a putter and the gutter, a putter, but what a putter can do, it can put chunks into a store. A gutter can get chunks, it seems like it has some complex tiles, but if you check these out just bite slices, so this is also independent from other parts of this worm code.

A

You can put a bite slice and then it store it is hashed and stored, or you can give a hash and then the appropriate chunk data is given you back without, depending on Shanker chunk or anything and I actually made an implementation of this put putter and gutter code hasher store the this hash. Our store is a putter and the gutter at the same time, and what's awesome in it that you can give it an encryption object which implements the encryption interface I showed you earlier, and then it does the encryption. So you can.

A

You can provide a chunker with a putter which not just put the chunk but also encrypts it. This will make that dpi and the chunk array simpler as I, will show you it hides and execute all the encryption decryption. So you just have all different kind of assure stores and their hash restores, which encrypt and decrypt and assure stores which do not and as I said it doesn't depend on chunks. So the DPI will be much simpler from now. There are no workers to to get and receive chunks it just what it does.

A

For example, in the store case there is an extra parameter, whether you want to encrypt or not, and then it just instantiates a hash or store. You can stay to the hash rest or whether you want to encrypt or not. You can give a hash function. You can give a lecture a chance to to to put somewhere the data, and then you can call the chancre in a way that you you give them this putter, but you instantiate it and and the chancre it's field doesn't have to care about hashing or anything.

A

It just calls the put so this will simplify the chancre. Why.

A

This will simplify the chunker, so there is no more chunk. Dependence is. The chunk type is not in the chancre anymore. There is no hashing. Shanker doesn't even know that things are hashed, it doesn't know anything about encryption, but it happens that it just receives a putter and it was a put. So a lot of things were actually removed from the Juncker code. I can show you an example. This is how the process chunk function looked like.

A

You can see that that there is a hash sure which does hashing, and then it instantiates chunk, object and then somewhere here it puts it into a channel to store it and wait until it's stored. So this code now looks like this. It just goes I put on the putter and put put a chunk data into it. I.

A

Also added another simplification, so the the chunker was a viewed object because it was an object, but it really did not store anything. Basically, it was more just a module, it had a split and the joint function, and you could call this speed and joint function, probably multiple times on different channel data, and it did not cause any problems the so you could use the chancre like this, that there is a chunker and then you call split, split, split split and you did not have to wait for the first speed to be ready.

A

Instead of that, I wanted to do something like this, that one chunk of object is responsible for just one split I had a good reason for that, because I wanted to eliminate a lot of very complex function, calls inside the chancre which had a lot of parameters because they could not save anything on the chunk or object instance. But this looks really like Java code. So I like are you wrapped it into a split function and in the speed function, implementation I, actually instantiate a chunker and then just throw it away.

A

So we before that the chunker had functions like this prepared chunks. You know you have to pass down all the information because they are not on the object and the state would Shanker. But I it's got like this, because all the other data is just on the object itself, so it's very simpler.

A

So actually it did not look like this, but finally, it looked like this, so I just had to go on a join for a split function without instantiating a chancre.

A

So let me summarize, this is how it looked. There was a DPA, it called joined or split on the chunker, which I did a lot of things. Instead, this is the old architecture. Instead now it looks like this, so there is a DPA it instantiates, hash, first or with or without encryption, depending on the. Actually we request the hashes store. Encrypts object is the data, if necessary it hash. Is the data, creates the chunk, object that actually stores them in the Chancellor or retrieves them from the chunk store.

A

So actually the hash sure store can have two encryption objects to help instantiate it from this new encryption package. One is responsible for the span encryption, the other is responsible for the data encryption and then the DPA can call the chancre as before, but the chancre puts and gets the chunk data and as all the extra tasks around that through the hash sure store, so the encryption is really not something which is Juncker has to care about. Finally, this is the new architecture. This is how it can compare.

A

It looks more complex, but the individual components are simpler, so I think it's a better code. So with this way, I managed to make the chunker code nicer, although I did not fully grasp how it works. I just got the parts which I could get and and extracted it, and I really suggest this technique for you when you touch complex code.

A

Thank you for your attention.

A