IPFS IPFS Camp 2022 - Libp2p Day, 2 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: DOS Defense - Do’s and Don’ts - Max Inden

Description

This talk was given at IPFS Camp 2022 in Lisbon, Portugal.

A

um I want to give a very quick talk, and this is really just food for thought um on how to prevent dust attacks, denial of service attacks, uh I'll phrase this as the do's and don'ts um there's, really amazing awareness for the topic. I'm, not gonna, give you uh a solution to this all right toss. What is it denial of service attack, um denalo service attack?

A

The idea is, as an attacker I want to attack someone for that Target to no longer be able to serve in the purpose it was initially put up for defending and against denial of service attacks is super hard if you go on school scholar.google.com or if you just read the gospels of paper or any any of those papers out there like it's. It's super hard right- and this is especially very hard in the peer-to-peer world, as identities are cheap right.

A

An attacker can do a DDOS so targeting from multiple angles and then in addition, they can generate as many identities as they want and it's very hard to prevent that and in addition, many times peer-to-peer is actually happening, not on very beefy systems, where you control everything and you can throw a lot of money against the hardware, but much rather smaller servers or even yeah cell phones or laptops and so on.

A

um Denial of service attacks, um I would say the number one way to do this is to attack or to exhaust a scarce resource. Now the idea of unbounded CPU, the idea of unbounded memory and the idea of, for example, unbounded file descriptors, that's all a lie right, so uh we need to somehow prevent yeah those resources to be exhausted. I think that's quite obvious to everyone cool okay, so this is obviously relevant in general. This is relevant in the peer-to-peer world and this is also especially relevant in lipidopy and I.

A

With this talk, I just want to raise awareness for this and make you all as post users and implementers um keep an eye out for this all right. The dues uh one single, easy rule: um it's the solution to everything, I guess uh bound everything right. No, you don't have unmounted memory. No, you don't have unbounded amount of sockets. uh No, no, no right bound. Everything. Everything that is unbounded is an attack Vector. Now, obviously you can this.

A

This sounds dogmatic right and this is not possible and, for example, which I'm mostly working on definitely doesn't live up to that. But let's strive for it right.

A

There are two ways on once: we bounded everything. Let's say um what we can do once we exceed a bound right once there is an item that exceeds a Bound, for example, on the number of items you can put in a channel uh the the size of a message, uh the amount of buffering you want to do, and so on one item one way is drop the item right. If there's a message coming in and I currently can't handle it um drop it.

A

That's not good um for various reasons, and then the other one is enforced pressure. So the idea of if the consumer cannot handle the item right at the moment, don't read from the producer and thus slow down the producer.

A

So you have a whole slide on that uh yeah back pressure. The idea is, you have a you, always have a producer and a consumer. You can basically boil down every computation into those two. You want the consumer in case it's slow to slow down the producer. The nice side effects is that can improve resource utilization. I'm saying can right. Quite often, batching is a good idea, and this also can improve latency. As you keep buffer small um yeah as a nice side effect, but this is really on Dust prevention here.

A

Cool okay, so so a couple of don't slides and again there's not. Obviously we want to bound everything but yeah, that's quite hard. um So I have a couple of slides on code. Examples and I would like you all to figure out what the problem with that is.

A

um This is hard to read easy to read: okay, to read, okay, to read, yeah, otherwise, come forward. I can't make this bigger.

A

Okay, all right, I'll walk you through it. um We do networking here right. This could be in lipidoe, but this could also be used using lipido B building your own protocol on top of Flippy to be, um we have some bytes that represent the message. Thanks prefix uh you have. We do a uvi, encoding, anti-invariant integer, um it's just a message length. Then we allocate um a buffer for the length and then we read the message into the buffer.

A

All right go for it.

A

You're not gonna, send me the message length.

B

So I lie about the size of my message: yeah and then I only send you n minus 1 bytes, so you're stuck waiting for the end byte.

A

Yeah, very cool, okay, so now I have a co-routine or rust task or a thread just hanging there never reading anything. So we need a Time bound on this. What else can I do here as an attacker.

C

um I'm gonna send you a many terabyte long message or B U size, Max.

A

You're gonna send me that message.

D

uh I'll pipe Dev, you random, okay,.

A

So you can do even better than sending me a very large message: how can you do even better? This might require some reverse knowledge for the other, non-rust folks, I'm. Sorry, all right in case no one knows I'll give Elena this chance. But Elena knows all the solutions already I think angry.

A

B

It was a I I, tell you that I will send one terabyte, but don't send the terabyte yeah.

A

And then I'll allocate and the operating system hopefully will kill me or stop. Okay, no rocket science right, no rocket science. I just want to plant this into your brains right whenever you code watch out for these things cool all right next one.

A

um This is my go skills. It's probably something bad here, I think we don't do semicolons.

A

um We have a for Loop. We receive a request from a channel right and then we spawn the go routine handling that channel any issues with this.

E

If you get a bunch of incoming requests at the same time, there's no limit on how many go routines, you're going to end up with and.

A

Yeah, it's not going to be good, very good, cool, again no rocket science right, but yeah we've all seen this so um go channels. First off awesome: they have back pressure right, they have a limit, they are bounded, wonderful and if you read from them and then spawn to go routine, you just broke back pressure across your system. Right because maybe go routines are cheap, but maybe me as an attacker I can send requests faster than you can handle them. So eventually, you'll run out of memory, yeah, yeah, okay, cool same game and rust.

A

um Rust does have unbounded channels, so that's even more fun, but we'll we'll not have them. Here. We require read a request from a channel right and then we spawn a task, but this also applies. If we spawn a thread right uh or we spawn something to a remote note like it's all the same right, we break back pressure and that will eventually be an attack vector, okay, cool um any comments to this one again, quite obvious.

A

The unbounded channel so I want to plan something especially into rust, Engineers here. If you use an unbounded Channel, you need to somehow externally have a mechanism to bound what's happening in their channel right or you run a small CLI tool that no one cares about in terms of back pressure. Fine use an unbounded Channel, but in all other cases you probably don't want this cool uh on go um yeah. You have bounded channels, really cool, oh no, in the JS ecosystem channels, and are they bounded by default? Okay, so um I'll explain this one.

A

We have a vector of request. We want to handle later on right. We can't handle them right now we have a vector. We have an incoming request again coming from the channel, and then we push the request into the vector I'll not put this up as a quiz like I. Think it's obvious right, yeah, an unbounded channel is the same like an unbounded, vector or anything.

A

You need something that watches out that that Vector, um sorry Vector array, collection, whatever language you use, is not running out of bounds right, okay, cool I think those are all my examples. Not very sophisticated uh yeah, I hope I planted that thought into your brains. The next time you program in general, the next time um you program on anything lipid be related or on top of wwp and yeah. At some point, I'll get a tattoo which says, which will say back pressure, but until then please do back pressure. It's super important cool.

A

Thank you very much. Are there any questions.

D

Okay, so for us the P2P, uh does it do all these things correctly or is it? Are there still some things to be.

A

Solved there so I say bound everything: cool recipe doesn't like. We don't bound everything I think we're decent in many many places and I think we got a lot better. I think there's still a lot of work to be done so.

C

um Cdn providers have tools for statistically detecting DDOS attacks um and I'm wondering if there's been any infrastructure developed for live P2P nodes that allow them the similar level of functionality, maybe not as powerful, but something that allows them to preemptively, reject certain requests before processing them.

A

Yeah I think the closest we get to this is the gossip sub paper, with the many parameters and scoring functions, and so on. You could get rid of the idea of this. You could prevent the Civil attack through some kind of proof of work mechanism or putting it on some chain right.

A

Yeah. It's not ideal. I would love to have the solution. Yeah.

B

You can use Fade to ban a.

A

Couple this is actually a good point, um so on I'll mention this right now on our docs page docs.liby.dale, we have a bigger section on in general dos prevention and there are a bunch of mechanisms again, no Silver Bullet, like it's just engineering work um and then, for example, on Rust specific things we have on Rust Liberty. We have a coding guideline on like how we think we can best enforce back pressure throughout a rust code base. This is very opinionated I'm happy to talk about it. I'm not saying this is the Silver Bullet?

A

No, it's a solution to everything, but yeah a step forward in my eyes,.

F

Getting back to the rust spawning a task yeah the risk version- I, don't I, don't know about go, but if this couldn't back pair should be implemented in a schedule so that you have a finite amount of threads, whereas if you have all them busy, you don't spawn another and you make it so that the scheduler doesn't pick and launches another task. Yeah.

A

I think that would be yeah. That would be perfect so around here you have some some. If, whether the amount of tasks you spawned exceeded and only then sorry inside.

F

The screen inside.

A

F

A

Could do that, but the scheduler doesn't know the scale of your system like okay tasks are cheap right, not not full stack frames right, that's fine cool, but maybe I'm moving each time, I'm moving a one gigabyte buffer into there. So does the scheduler know whether uh like I it can spawn a hundred or a thousand or a million tasks? It doesn't know right, so it doesn't know the scale of how much memory I'm moving into the task. This is not only about memory right.

A

This is also about CPU, like if I can keep your CPU busy while you're not serving any other inbound connections or outbound connections. The remote peer will terminate those connections right. So so many attack vectors here.

E

B

I I think the the lesson here is you don't want to be isomorphic to handlers to requests? Is that a fair statement? I, don't know what isomorphic.

A

One is context.

B

One to one you don't want to spawn a new task per request: yeah.

A

So, for example, invest Liberty. What we do is we spawn one task per connection, but the amount of task connections is limited. You can limit the amount of inbound and outbound connections, um but we only we don't spawn tasks per stream, for example,.

G

Sometimes, in dust protection have this concept of kind of enforcing diversity, for instance, in terms of ips, you don't want to have too many requests in the buffer from the same IP or from IPS that are too similar. Do you have this kind of protections or.

A

Yeah, for example, in class, it's up I think one of the parameters is either autonomous system numbers or IPS that need to be diverse, but then once this is a step forward right, but it shouldn't be the only measure, as we have like once we're in the IPv6 space yeah, you get the slash 64., so yeah cool. Okay. Thank you. Please do back pressure.