Node.js Node.js Interactive North America 2016, 15 Dec 2016

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Take Data Validation Seriously by Paul Milham, WildWorks

Description

Take Data Validation Seriously - Paul Milham, WildWorks

Genetically evolving cellular automata -- that's a mouthful. Let's break it down 1. We will be looking at cellular automata. A cellular automata is a set of units governed with very simple rules. These units make up a complex system, or model. 2. We can train cellular automata to genetically evolve, making themselves better overtime. We can make them adhere to a set of rules that would make the system reach a certain outcome at the end of N generations. We can even use these concepts in the wild. 3. We will be doing it all with JavaScript.

About Irina Shestak
Irina is a node + javascript developer and enthusiast. Aside from enthusiasting about javascript during the workday, she instructs with Ladies Learning Code in Vancouver and explores the great outdoors (i.e. the Rockies).

A

Thank you for coming. Austin is an awesome place. It's a gorgeous place. You've got here. So did you guys see that? Okay, that's literally the only animation in my talk so enjoy it? Okay, so we're gonna talk about data validation today, all right and and to take it seriously. I know a lot of like it's an afterthought a lot of times and that's not not the best attitude to have okay, so a data validation isn't something that people get like excited about right, but it's a little bit like washing your hands right.

A

When people leave the bathroom, they need to wash their hands right. Yeah yeah nod your heads, yeah, okay, so it's it's important and if for our application to we're on the Internet, we have no idea who's shaking our hands right. These HTTP handshakes, we don't know who's shaking our hands. Okay, so this is important.

A

You want to get some data pinkeye, okay, so I work for Wilde works, and this is a I like this company, a lot actually I kind of have to say that, because they paid for me to come here, but nonetheless, it's a good company. We make video games. We make online video games. Okay, like this one Animal Jam. This is what I spend almost all of my time working on, and so these gifts they they pretty well capture. What Animal Jam is about.

A

Okay, so if you have a pet, you have a dog or something like that. You can put clothes on it. You can put pants on it, but they don't really like that from what I've seen. So, if you want to put pants on a cat or a giraffe, come play our game, it's a better, better way to carry out that dream. Okay, so, okay, anyway, the rest of the talk is pretty dry, so we got to start it out right now: okay, oh yeah, yeah!

A

Let's, let's go over the in outline just what we're gonna cover in the talk? Okay, so we're going to talk about some attacks, so why is data validation, important I'll, just give you a sample of a few attacks? Okay, so in data validation and how that leads to security of your application. We're also going to talk about data normalization, which people don't talk a lot about a lot and how that can lead to stability. Okay, we're going to talk about joy. This is a data validation.

A

It does some normalization as well and we'll talk about TN. This is a library we wrote at wild works and then we'll talk about how to integrate it with express and happy. Ok. These are two really popular web frameworks, so hopefully that'll prove useful at the end. I'll take some questions. I'll just do it out in the hall, because I don't I, don't want everyone to get awkward with the silence and everything. But if you ask me a question, you get a free animal jam toy, okay, pretty cool, ok and Google's, giving out awesome stuff.

A

You can't really compete with that, but ok anyway, ok, so my job is basically to keep kids safe. We have lots of kids playing online and we want it to be a safe, friendly environment. We don't want anything like you know, bullying or worst cases like predation. Obviously, but we don't want anything like that happening in our world. Ok, today we're going to talk a little bit about how do we keep our application, safe, ok and so safe from what so. This is where we're in a segue into attacks.

A

Ok, now this is just the truth. A lot of people just want to watch the world burn right. If you write like did you ever write a choose-your-own-adventure game and see or something when you're learning, how to program you're like hey check this out, and somebody puts invalid input in and then text barfs all over your screen, so they're there people just that, want to do that they want to break your application. Ok, so if you don't believe me, you can check out this link.

A

Ok, there's a nice list of things that uh that are extremely horrifying and scary. Ok, so at your leisure, read that to your children at night. Let's, this is a really common one. Okay, so sequel injection don't spend a lot of time on this, but we just want to insert a user into our database here. You can see on the third line. We just want to insert a user right, and this is this is all fine.

A

Well, actually, this isn't fine if this line of code is in your is in your code base you're in trouble anyway, but somebody signs up and their name is Paul. That's great I, like that name, ok, but their email happens to be drop, table, DB user. Ok, that's not good! Okay. You've probably seen this as a joke on xkcd or whatever, but ok so there's also shell injections. This is really similar. This is this is where you you're gonna contact the shell for some some function. You need to run. Ok.

A

In our case, we were contacting PHP to run this script function and that's all fine, but then some jerk says his password ringwraith. Okay, so you can see this is going generally speaking whenever you're like manipulating strings and passing them on to another process that can be that can be trouble. Okay, now read off. Did anyone see the talk this morning by a mmm guy put yarny I, hope I'm, not butchering his name? Okay, if you didn't see that those are really good great talk and you should, you should go back and check that out.

A

He had some really good tips, and this is one thing that he covered is a reg X denial of service. Not everyone wants to just burn your data or steal it right. They might want to take down your service too, and this is one way they can do that now. This looks pretty innocuous right so on the third line here, we're just we're just checking for these. These trailing semicolons right.

A

But if somebody passes in this message at the top, you can see if there happened to be 65535 semicolons in the string they pass in, which can happen. You run this reg X and suddenly it ties everything up for six seconds. Okay, that's what it did on my laptop anyway. This is a big deal. This is your event. Loop is blocked and nothing is coming in your applications, not responsive at this point. Okay, all you did was throw in this little red, X and suddenly everything's broken okay, and this is this is a real vulnerability.

A

Okay, this was in tough cookie, I pulled it from snick. Snick is awesome, check them out too, but be careful with your red X's. Okay, so hopefully I've convinced you that the world is a bad place. That was the goal of my talk. Okay, no come on. Those are pretty funny joke all right anyway, so the security is important. There's things we need to look out for, and security is important. Those are just three sample attacks. Okay, so what can we do about? It? Validation, so we're going to talk about here.

A

Let me back up so with validation. We want to make sure that our data looks like what we're expecting right a lot of times. You just get some some JSON or something like that, but we want to make sure it looks right. This is an image from joy and I really like that, because I mean like these little blobs. That is exactly what types and JavaScript look like. They all look like this okay, but you want to make sure it's the right.

A

Looking blob right and if we do this, then malicious data can't come in okay. This is this is real important. This is the first line of defense in our application. Okay, I mean security comes in layers. There's lots of layers of security in your application. Well, I mean in your network in your application, but as far as the application code code goes, our validation should be the first thing we do. Okay. So let's look at some joy. This is just a simple example of joy.

A

Okay, so we're gonna pull it in and you can see on the third line here. We want to validate this input seriously a string. Okay, we want to make sure that's a string. Okay, it seriously might not be so. We pass in our schema as the second, and this just happens to be a joy object. Okay, and then is this really standard has a callback with an error and a value, so the error is going to be null because is a string and our value is seriously a string. Okay, great.

A

Let's move on okay, so let's test a failure scenario, so we pass in five is five: a string? No okay! So we're gonna get an error in that case, okay, but most of the time you're not validating a single value. Right that, like that right, you have some schema. Okay, so here we define a schema and the schema has a username and a password. So let's walk through this, though, because this is important how this works. So the username is a string.

A

That is also an email and it has to be a wild works, email, okay, so I'm being specific here and it is a required field. I know this field is going to exist when I'm done validating or it will be rejected. Okay, so that's that provides a lot of security and a lot of reliability right there. Okay I also require a password. The password is a string. It needs to be at least six characters long at most 25 characters long and it has to be there.

A

Okay, we want to be strict about what we're requiring in our API okay. So let's validate it, my user name is Paulo. Nolan that wild works- and my password is just oh I- have to go change. My password now, maybe I'll do that later. Okay, anyway, so yeah this passes, but this is what I want to illustrate with. This is that data validation for happy is all in okay. So that means we validate one thing: you have to validate everything, and this is really important for the security and stability of our application.

A

Okay, so we have a user name in our schema, and this is the exact same thing as last slide, just no password, okay, but then I want to validate this object right. So I have my username and my password, but then I go to validate against the schema now and we find out. Justin Bieber is not allowed, which is exactly what we want right. So this is important, because what, if somebody goes in maintenance, they go to update your API route, they add a new parameter or something and they forget to add the validation.

A

Okay, that's a scary thing. That's a potentially an attack vector in this case. It can't happen because it's all in there gonna go to test the endpoint, it's gonna fail, then they'll add the validation and we're good to go okay, so you validate one thing: you have to validate everything. Okay, this means that your developers are not going to forget. They can't forget- and this is really important when it comes to security. You want to force people to pay attention to the validation layer. Okay.

A

So let's talk about normalization a little bit. So joy calls this conversion. If you look it up on their website, but this is kind of being a good citizen. In a way, let's say that our API is accepting a user ID okay, and so we want an integer, but somebody you know somebody calls it and they pass in the string with the number 5 in their user ID, 5 K. You don't necessarily want to reject it right because that's not an integer. Ok, so that's that's a nice thing to do.

A

Ok, and so it helps create a contract with your consumer as well whatever that might be so they know what they can input, and you know what you're gonna get, even if they send the string 5 in you know, you're gonna get an integer 5 after the data is normalized, ok and it goes deeper than that we'll get into that later. Ok, so this! This is why that's important so, let's say we're expecting a number. You can see the second parameter that were expecting a number, but then we pass in this string.

A

One point: nine one: six, okay, that's I mean that doesn't seem like a big deal, but if we didn't normalize the data here on our second line, when we call to fixed we'd, have a crash. Okay and that's stupid, you don't want your application to crash on something like that right, okay, so this is a very simple example of why normalization is important. Another form of it. These defaults. Let's say that they don't pass a value. We can supply one for them.

A

Okay, the thing that's cool about this is: it allows us to make safe assumptions in our code, I mean you want to code defensively, but coding defensively super annoying right.

A

Okay, now, if we have a layer that is ensuring we have certain certain types on our values, we can make safe assumptions if we don't have to put like a billion if null checks everywhere, okay, which is nice okay, so TN is a library that we wrote at wild works because we couldn't really find anything at the time that fit our needs and I'm not telling you to run out and start using TN in your project.

A

I think it has high level ideas that are really valuable and it has some best practices and what I mean by best practices, practices that I consider best in there. So it's good, so a declarative, syntax I think this is great, because it's very easy to read. You can just go check it out. They're, just plain old JavaScript objects, Capo, Joe's and asynchronous. So this is important because if you want to do any sort of validation that hooks into a database or anything else asynchronously, this is going to be really helpful.

A

Okay, we can convert data into models, so we accept a user ID. We can convert it right into a data model, okay, our user model, or they could just pass the user's name or something any unique. Identifier like that, we can take it. We can map it into a model directly. Okay, it's very strict, okay, that's where you can find it and as as a know, if you want to stick to joy, they have custom. Validators were added recently as well.

A

Okay, so, let's just run through this really really quick, because it's kind of the same thing the parameters are swapped, so the first one is the schema. The second is the the value you want to validate. So here we just have an animal and the animal should be a string. Okay, we have an animal, that's a kangaroo. That is a string great okay, so our callback accepts air. It'll it'll has two parameters and that's is valid and result. Okay, so you're gonna say yes or no did this pass and the result?

A

If yes is going to be your normalized data, that's the day you don't you want to use. This is safe data to pass along to your application that you can make assumptions about. If it's not it's gonna tell you what's wrong. Okay, it'll say you know, I mean in this case. It's just gonna be like animal should be a string, it's not okay, but it gets a little more advanced.

A

If you have, you know, arrays of objects with nested values, that's really nice for tracking down bugs, but so in this case let's say the data we pass in is null, so our animal is null okay. Now we consider that not a string, because you can't make safe assumptions about null like you'd, have to put an if null check on there. It could lead to a crash okay. So we reject that value. If you wanted to accept nulls for values, you can do that, but in this case we didn't want to.

A

So let's look at something a little more strict, okay. So in our our schema here we're accepting an animal and it has to be a string, but we only want kangaroos or Tigers. Okay. This is animal Jim, so only kangaroos are Tigers, okay and sparkles cuz. You always want your your animal to Sparkle, okay, and you can see default values in here anyway. We send in Tiger great that's a tiger, but the important part here is our result.

A

You can see this default value was added to our our object, so it's safe to make the assumption that, because we have a default value there, it's going to be present. We don't have to check for it. We can just go ahead and use it. Okay, so I talked a little bit about model mapping earlier. This is really powerful in that we can take input values and we can directly map it onto a model, and so that way you can just go ahead and use your model.

A

In this case we talked to that we have a user UID. This is a user, unique identifier, so anything that uniquely identifies a user. They can pass into your API. It's going to look up the data and then it's going to pass back in this result: the user model okay, so we can just go ahead and access the user's name or whatever we want at that point. Okay, so that's data normalization and this provides a lot of reliability in your application. Like safe assumptions are a really really cool thing.

A

You know and it helps clean up your code. It helps prevent a lot of these stupid bugs where you're. Just like oh I, didn't you know, I didn't have a null check there or you know. I thought this would be a string, but it's a number all that stuff. Okay. So let's move fast.

A

Okay, so express everyone uses it. It doesn't have any validation built in yeah. Okay, the bullet point doesn't have any purpose. Okay. So anyway, you know what expresses so, let's look at how we use it with joy. Okay, so pretty much here, we're just declaring a route, and then we have a schema, validates against right away when it comes in- and you know we send it back so, okay, it's pretty much the same thing with Tina.

A

Okay right, when we get the parameters we validate them and send it on okay, the problem with this is we're still relying on the developer to remember or the maintainer who ever to update at this point or when they add a new route where we're relying on them to actually add the validation code in there. Okay, that's that's an issue right, so let's talk about how we can use route middleware and Express to fix this.

A

Okay, so TN helps facilitate this a little bit and it'll hook right into you and it'll update your attach this parameter called safe data. Okay, saying you know, that's safe to use, you can go ahead and use that okay, there's also one for joy called celebrate. You can check that out.

A

Okay and happy is a lot different than Express. Okay, it's got a lot more stuff to it. Okay, lots of lots of garbage in there. Okay happy is really cool I. Actually, like happy a lot. So let's look at how we declare a route and happy so we this looks pretty standard right. Well, we just say it's: post we've got a path reply. This handler, just you know, sends back hey mmm great, that's great whatever okay, but the cool part here is.

A

We can just attach this validate property until we want to validate the payload and what to validate in it and from that point on, we can our data safe. If somebody updates this route and adds another parameter to the payload, they have to update this or it'll fail. Okay, that's cool, okay, so just a few takeaways that I hope we get from this just high level. Okay, okay, we want to force data validation, yeah.

A

If you have a system where it's like, like we don't want this to become Doc's right, nobody maintains Doc's, so we want make sure that there is a mechanism in place that forces people to keep this up to date. Okay, we want to make sure the shape of our data is what we're expecting in a powerful way. I've found to make sure this happens is the rule, no validation, no data, that's how it works. Okay, I, don't care if you're in development mode, whatever you don't validate the data, you don't get it.

A

This forces the developer to update the data, validation, make sure he's set. He or she is accepting what he wants, while in development. Okay, if you do this, you're gonna know what's coming into your application, we're not going to have mystery values, show up okay and as far as normalization goes, we also want to make sure our data is all normalized. We know the shape of our data, so we can make safe assumptions about it, so it's always going to have a consistent shape and it's going to be reliable.

A

We're gonna get rid of all those stupid bugs right, we're not going to have all these dumb gotchas and if null checks everywhere. One final thing is: we've just talked about data coming into our API so far, but what about data leaving our API? Okay? Have you thought about that? Okay, not a lot of people do. Okay, that's I was supposed to do this mind blown but anyway, okay, so that's pretty cool. So what this can do is prevent leaking information.

A

So we've seen a lot of attacks where someone gets something bad into your database and then the real damage is when it comes back out, they're, either stealing information or they're, sending information down to another client where it shouldn't be. But if we're getting this data on the way out, even if they're successful in the initial attack, it can't happen.

A

So, if you're sending back data you're not expecting to send back, you view that you have malformed data, something has gone wrong and you shouldn't be returning that data anyway, so I have heartbleed up here in law quotes right. So a heartbleed wasn't you know that was something different, um but we don't want to leak uninitialized memory or anything like that.

A

If we're not expecting the shape of data, we see on the way out, we shouldn't be handing it, and this is also cool, because it's a really easy way to see what your API is returning. You know like what is this endpoint get a hand back? You don't have to like trace the code through or look up your models or anything like that. It's just gonna be like okay. Here's!

A

My validation map- this is what it passes back: okay and that's cool for your client application as well, because they have a reliable contract that they can depend on coming out of your API okay. So thanks.