Ethereum Foundation DevCon 6 - Security Track, 12 Oct 2022

Previous Meeting

⏯

youtube image

►

From YouTube: Web3 vs Web2 Security: Same or Different? | Devcon Bogotá

Description

Visit the https://archive.devcon.org/ to gain access to the entire library of Devcon talks with the ease of filtering, playlists, personalized suggestions, decentralized access on Swarm, IPFS and more.
https://archive.devcon.org/archive/watch/6/web3-vs-web2-security-same-or-different/

Web3 security is typically associated with smart contract security. The biggest Web3 hacks have however involved traditional Web2 vulnerabilities and attack vectors. This panel proposes to debate on the similarities and differences between Web3 vs Web2 security with some leaders in this space towards the goal of highlighting the current status, historical lessons from Web2 security and future challenges for a safer Ethereum ecosystem.

Speaker(s): Mudit Gupta, Spencer Macdonald, samczsun, Cory Hardman, Nassim Eddequiouaq, Taylor Monahan
Track: Security
Keywords: Security,Web3,Web2

Follow us: https://twitter.com/efdevcon, https://twitter.com/ethereum
Learn more about devcon: https://www.devcon.org/
Learn more about ethereum: https://ethereum.org/

Devcon is the Ethereum conference for developers, researchers, thinkers, and makers.
Devcon 6 was held in Bogotá, Colombia on Oct 11 - 14, 2022.
Devcon is organized and presented by the Ethereum Foundation, with the support of our sponsors. To find out more, please visit https://ethereum.foundation/

A

A

We love you all right, guys, hey everyone, so uh we're here to talk uh what web3 projects need to know about web 2, cyber security uh and really I think this is going to be a great kind of follow-on. Talk to the bridge security framework. Talk that Patrick just gave.

A

um This is kind of the compliment right, so he went over really some of the smart contract and protocol layer aspects of bridge security. uh Unfortunately, a lot of bridges have been hacked, though not from you know the hardcore cryptography, but from Basics from cyber security, Basics. Okay, the cyber security Basics matter they matter because they have geopolitical significance.

A

So before we get started on the panel, take a look on the screen. There was a great article by CNET that came out a couple days ago and it really talked about how Lazarus Lazarus Group, which is affiliated with North Korea, is seeking to infiltrate this ecosystem by setting up front companies and then having applicants. Job applicants apply to these front companies and then the engineers get fished right. That's actually how the Ronin Bridge attack happened. Okay, so the fundamentals matter- and it doesn't just impact people in this room- the people at this conference.

A

It impacts geopolitics, because then it funds things like North, Korea's new program, or you know transnational criminal organizations and stuff of that nature. Okay. So what do we need to take away from this panel? So a lot of fundamentals from the last two to three decades of web 2.

A

Cyber security, there's Cloud security risks that we have to understand and mitigate a couple with that there's front-end security risks right, there's operational security risks, there's also corporate security risks and then obviously big topic with some of these Bridge hacks has been Key, Management, endpoint detection and response and again management of individual, individual nodes and validators um doesn't mean we need to Discount smart contract security risk. What it means, though, is we need to shift the security conversation in this space from very siled.

A

You know: individual audits to a holistic, end-to-end scope of the Project's entire architecture right and not just their technical architecture, but the people architecture, and that's something that you know. We have some newer entrepreneurs in the space and that's something that, as we kind of get some Battle Scars as an ecosystem, I think we're going to learn time and time again. Okay, so who are the panelists um we'll start left to right? We got uh I, guess we'll start over there, uh Naseem from a16z crypto. We have Taylor from from metamask.

A

We have Corey from openc special guest Hudson from crypto Twitter and then another special guest Moody from also from crypto Twitter, oops, no polygon polygon, says cool.

A

All right guys so we'll start um with kind of a basic question and I'll ask it to NAS first and then the rest of the panel. Now as if you could kind of describe, you know from the context of just normal web3 or, like a you, know, representative web project in the ecosystem. What are the web two parts of other Tech stack and then what are the web? Three parts of the of the tech stack and then what are the security relevant parts of of each yeah.

B

I actually feel like it's a it's, a fairly uh long answer that is required for that, but you know we. We do spend kind of quite a bit of time going from the early days of the company up until kind of like the ongoing updates to um you know, decentralized protocols.

B

I would say that um you know it kind of like starts with the people. It's essentially just like a set of people who just want to you know design a component uh build it, uh deploy it make it available and just like, maintain it over time. So there is kind of like the everything that it that it encompasses.

B

So that means you know you essentially start introducing devices to build the components on you start in involving kind of like the entire kind of, like you know, secure software development lifecycle that is relevant to every single person, kind of like Building Technology in general um and I'm trying to cover cover, like anything that is, you know, kind of common to what to enter through here. um So you're going to have like this part, then you know this.

B

These people need to build an organization right and like with that column, you know kind of like corporate security um as you, as you mentioned earlier kind of. Like anything, you know from like a endpoint, uh you know security standpoint, device, security and and so on.

B

I would say that, like the appsec is also like very uh very similar, you know always kind of, like you know, analysis of your code as you kind of like develop it, and, and obviously you want to iterate uh pretty fast, so you're gonna build your CI. You were going to integrate you're gonna run. You know, security, testing and bond scanning and all the the stuff that we all know in love um doesn't really matter whether you're building for web 2003.

B

Yet at this point, right, um I think that yeah like in my in my mind, everything that that spans over you know, network security, um essentially like network security, uh corporate security, appsec security, engineering, key management and by the way like the key management, is the part that is like highly overlooked. Like a lot of people. Think like okay, it's okay, I can deal with this later. You know I'm just gonna have, like you know like a single key on like a either letter or like some. You know a fairly, not robust.

B

You know key key storage and then kind of like push it out to uh to later.

B

But you know, Key Management is like a fairly complex thing that, like we have to deal with in in web 2., um yeah I would think I would say that, like these are probably like the the the majority of the things that we see and obviously you know comes with, like you know, physical security I think that, like it kind of like ties into you, know key management and the security of the people as well, um because it's not just about the phones.

A

It's also on that note: we've heard stories of people walking around uh certain places with you know: crypto swag and exactly having their phones out and so on. So that probably ties into this exactly as.

B

Well, right, um yeah, and we had like quite a few conversations actually with uh you know various teams of our portfolio that were coming here and you know, because decentralization is a spectrum and, like you, try to kind of, like start, obviously centralized with, like just very few people and kind of like you know, centralize over time.

B

uh There is like a point in time, and we're probably at this point in time like right now, where uh any person that kind of uh gets hacked attacked like physically, can kind of like result in either assets getting in trouble or even like reputational. You know damages that can be created as well.

A

Or potentially, physical access.

B

Physical actions, company or exact protocol, and so I think this, but yeah the repetitional damage is actually something that like is, is highly overlooked as well. It's not just about your protocol. It's like you, know everything that comes around that. Do you guys have your stuff squared away or.

A

Not effectively yeah pretty much anyone else on the panel um good.

C

C

So when I think about web2 versus web3 security, especially right now how early we are with web3 and ethereum and blockchain, there is going to be a need for web to stuff in your stack, uh not even consumer facing necessarily but like who here actually like, doesn't run their project on GitHub and uses something like matter most instead of Discord or slack or other non-hosted you're awesome, whoever that one guy in the back is uh but yeah uh yeah good for you, but if you're, unless you're doing that, like you really need to be very, very careful about where your data is going and keep up with that.

C

Something else is that I think is overlooked. A lot, especially when with web3 and how open it is and how open it is to contribute, are two things uh watching your GitHub uh to make sure that the you know, PRS that are being put in uh are actually from your team members, that you trust and that you've vetted uh Sushi swap had a bad incident where someone joined their join their team and then put something in that like I, think stole keys or something on their web page.

A

Insider thread.

C

Yeah Insider threat, basically, and then. Secondly, um this is something I worked at the ethereum foundation as the org security lead for three years and and something I learned, especially as I was leaving and hadn't reflected back on. There were just a handful, maybe two people in the organization that had Master access to just about everything and that's incredibly bad. uh It's even it was even worse earlier, so just bear.

A

With me, yeah like what happens if those people get run over by a bus right, not.

C

Even that they could literally take down the ethereum website post on the ethereum official Twitter post on the YouTube, pretend that vitalik died crashed the market like.

A

There's there's like a lot of incentive there and there's a montage. Everyone has a price right, yeah.

C

Yeah, so the thing is: if it wasn't that we just really liked and trusted the people doing it like we got lucky, but really there should be different ways and there's you can look up ways that, like people have theorized how to do this best for years separate the people who have access to the most critical parts of your organization and have an audit Trail have like a foolproof audit Trail. They cannot wipe uh and so yeah.

C

That's that's kind of what I think of because it's like web 2 security seemingly has mastered this in some levels, but web3 like forgot about it. So that's what I think! Oh so.

D

On the sushi incident, first of all glad we are not talking about the horse incident, but.

A

Off limits, yeah.

D

It happened a while back, um so what happened? Actually was one of the contractors had push access to the code base. uh There was no PR review approval process and it just got uh they like got angry with something post, malicious code. Nobody noticed it got pushed out and the key takeaway from that is. You should always manage access. People should only have access to things that they really need in a production environment.

D

No single person should have Push access, uh it should always be approval based have at least one or two reviews required and expanding that further, uh really like just minimize access to only things. People really need, for example, like uh access should not be considered as uh like a pride or something so uh co-founders, for example, or founders of a company shouldn't really have access to critical things like your emails servers or whatever, unless they are actually doing something of it, just because they have a higher precedence or whatever doesn't mean they should have access.

D

The thing is like people in People, Are People, no matter what people will get compromised one way or another, so you have to have that second layer of security, where even if someone gets compromised the impact is minimum.

D

So a practical example in the Iranian case, uh the bridge hack, their employee got compromised that that I can understand like if someone a phishing attack he was like it was a very well orchestrated attack, so uh I'll give it to him like okay sure he got caught, but the issue there main issue was that that one single employee had access to everything like that. One single employees access could have trained, 600 million, which should not have happened in production. That was that that was the key problem in that heart. Not that deficient happened.

D

That is gonna happen. You just have to protect from the impacts.

A

D

E

Ahead, all right um so I'll just add that um I think uh if you want to look at the different uh what's missing in the web, three on versus web 2 is just look at the web. 2 security Award chart. uh You know you have your instant response teams, you have your. uh You have a corpsec group. You have a production set group. You have an app set group, you have your uh secured reviewers. You have your vendor security audits. You have your. uh You know.

E

All those roles are there because that's how you achieve security, you.

C

Just have Samsung yeah yeah well.

A

Not right now, but Sam doesn't see it yeah.

E

A

Doesn't scale, though, right.

E

Yeah yeah, yeah and I, and so I think like. If you want to know what you're missing like look at the Delta between your org chart and you know any any well-established web 2 company.

A

So that's an interesting question, then so in the various life's lifestyle, like life phases of a crypto project, what needs to be internal versus external right, so obviously everyone goes out gets an audit. It's kind of a commonplace thing. um I still have audits, right, I'm, the co-founder of an auditing Marketplace. So we love audits, but it's not the the pansia, so I guess core to you. First and the rest of the panel like. When do you need to hire an internal Sizzle, uh we're starting to see later stage projects higher Sizzle?

A

But is this something where you know you raise Venture money and then on your use of funds like employee? Three? Is an internal ciso.

E

uh Well, uh I have a bias that, yes, this place, maybe playing two actually, but uh you know I, think I think an under underappreciated property of security is that it's uh it is not something that's added on later. It is an emergent property of your design, and so, if you don't start from the beginning, thinking about security you're not going to have security at the end uh and I. Think that, like so yes I would I would argue that you should have them early.

F

Sorry, yes, I would like 100 agree with that, because um security is really about like the whole system and it's like the social system. It's the people system. It's it's! How you interact with your community, uh it's how you hire it's, who you hire it's, how you vet them.

F

um It literally touches like everything that you do um and if any one piece is like not secure. That's an entry point and the thing about specifically like these crazy hacking groups that are coming out of North, Korea and other locations is that uh let's say that they just get access to your slack right. So it's like your internal slack. You got like 20 employees right.

F

They will then use that just that access and they will sit there for like literally months and they will read everything that you do and they will understand how you operate.

A

Not creepy At All by the.

F

Way, it's super crazy uh and they will use the information they glean there to then like escalate to fish. You Spearfish you via email or Spirit issue via telegram or LinkedIn DMS or whatever they, whatever you use, they're gonna get you and, and they just over the course of like months and even years, they'll, just like keep going and going going, um and so it's like these little things that we don't take seriously because you don't have anyone on your team early. That's thinking about security!

F

That is the problem right and the solution is not to get a freaking security, audit, I'm, sorry, I.

A

Love security audience no, no to be clear, like it's a it's a secondary check, yeah! That's for just a secondary check! Yeah.

F

A

Are not going to solve your internal security problems for you, projects have to solve that yeah.

F

And if you especially, if you hire like a good security person early on, they will uh indoctrinate your team as it grows and they will uh design your team right. Your org chart your organization, your products, how you work right.

F

They will help design that from the get-go and therefore every layer of your sack as it gets bigger um as the responsibility gets spread out across more people, as you scale up as the financial incentives get greater you're just gonna, be in a way better position and I I really cannot emphasize enough like what Corey said like you cannot add Security on after, like it's not a thing, you can't be like oh yeah, we'll just be lazy now and then we'll like fix it later.

F

That's not how it works, like security is a core part of your entire organizational design, your product design and your system, design.

D

Make this a little spicy! So oh.

A

D

I think like these guys, are just securing their jobs right now, but in.

A

Terms of I mean to be fair, we all are but.

D

You know yeah, but like just being practical, if you are just starting out I, don't think a second employee needs to be the security guy, but.

A

D

Needs to happen is your first developer needs to have a security mindset, your first it hire. That will probably happen when you are 10 to 25 people. That needs to have some security background. How.

A

Do you qualify that, like the.

D

Security mindset when you're high- so that's a great question. Yes, it's not something I can like uh like Define qualities or something. But if you look for me, if I look at someone's code the way they are writing it, the way they are documenting it. I can tell like it's just intuitive that this guy is thinking about security.

D

uh If you just talk to someone like look, uh ask them their priorities, uh see how well tested their code is uh even the code, syntax and everything it will tell you uh how serious they are about security, uh so yeah. Your first developer have someone with a security mindset. uh Let them train others, then your it. Guy probably will be your next time. Get someone with price security experience, but does it need to be a dedicated security people and then will you scale Beyond? Maybe 20 people?

D

That's when you should start looking into a dedicated security person for your own before that? Just have internal people with security mindset, plus hiring or not hiring, but like just Contracting external service companies to provide you with security help external companies. They don't only do audits. They will help you with internal security as well, because the thing is, if your org is below 20 people, the security guy won't really be utilized fully.

D

Unless you are someone like FTX with like 10 uh 10x Engineers, that's pushing the whole code out, then it's a different story, but talking about normal logs yeah security, people can come in a bit.

A

Later maybe the 20 metric is there for projects at the application layer, yes, okay, obviously l2s and I.

F

Just want to chime in on that that the point that you made about like um the security mindset like what is the security mindset, um I, think one reason why web3 sort of struggles with this security mindset a lot of times is that we are like so generally optimistic and like Visionary and forward thinking, and we are thinking about the potential and we are like thinking that nothing is impossible, and so so much of our time is spent in this sort of idealistic future world that we're trying to build and that's necessary, like we have to be that in order to build these crazy things that we're building, uh however, um like I, would say that I have a security mindset, but I would also say that I don't spend most of my time.

F

Thinking about, like the potential of the future I spend a majority of my time. Thinking about like how we're gonna it up.

A

So that's like the security mindset on that note, how do you ensure compliance, then right, I think it's something we talked about before is like getting people to actually do the fundamentals, even simple things like MFA being enabled right or um Corey would talk about this. You know endpoint detection and response actually being used, and if people aren't using that, then you kick them off the network. How do you, how do you get people to actually do the basics so.

F

Like when we were starting up right like and I I, Grew, From, Me and My co-founder to 20, and now we've got uh like sort of over 100 on the core team. uh We got like there's a thousand at consensus right now, um it's a when we had like two people, three people and started bringing people on the thing that we did from the gecko was we. uh We had good documentation about like what was required and what was expected of our employees.

F

uh It talked about like what they needed to do and it need they needed to do it across, like all of their accounts, like there's no personal, professional split in crypto. So, like you have your personal Twitter like that needs to be security if you're gonna be a part of our company.

F

um The other thing is a lot of these services like GitHub. You can force like if you want to be a contributor to my repo, like you have to have your MFA on. uh We bought everyone, uh multiple, like all the Little Hardware doodads right. Everyone has a ledger. Everyone has a tresler. Everyone has uh all all of them: the Google ones, the the the yeah, the Yuba keys and the Google one, the Titan one.

F

C

F

That one was pretty slick, uh still go back to my ubiky. All the time, though, um and these things right from the get-go like like the first employees, no matter what, um then, as you grow, you like your procedures, are sort of in place, um but then your early employees also like establish this like floor of like this is what's expected right and and that's the culture like that's how you establish the culture like nobody is going to step out of line.

F

One is already there right and- and you call each other out like when you see someone being a little bit lazy. When you see someone doing something, that's a little bit. You know even.

B

The founders right I mean it starts.

F

B

Wonders honestly.

A

B

Like the like, the the First Security, people that need to be hired are like literally the founders because, like I mean they need to have like this adversarial mindset like from the get-go as Madrid was saying. It's like I feel, like almost like web 2 versus web 3 is kind of like the wrong kind of like comparison, because, like we're really and just the the business of um security, critical software, we had this conversation yesterday with Corey. It's like you know.

B

It's like everyone here is writing code like whenever we you know, unchain code is essentially like ring zero. In a kernel like there is no safety net. You know besides, like whatever you're gonna put in place. Essentially, no one is going to do essentially like the security for you, and so it's just like coming with this adversarial mindset as a founder and just like thinking through, like everything that is going to go wrong, just like I was saying is like the most important thing. So probably like starts from the top.

B

The the culture and the DNA of the company needs to be built around that like there is no way for your company to be successful if they don't have that, because, like the the Founders need to kind of be behind every single one of these decisions where, like obviously, security, always comes at a cost of other things. Right like it's, always the the security, it doesn't live kind of like on its own right, like in a corner.

B

It actually always is a balance between you know, convenience or, like you know, product features and like user experience versus you know the safety of of doing these actions. So I think that, like just uh just having the founders uh kind of like come with, this mindset like right away, is the most important thing.

A

Is this something that VCS in their due diligence process should include like asking the founders about their views on security.

B

um I think that, like we, don't want to be uh too prescriptive about, like the mindset that people have before uh before initially like investment, but it's more like as soon as like we meet with them. We kind of like do the rundown of everything and we kind of like provide.

B

You know insight into all the threats that they're exposed to and kind of like try to really if they don't have it already like this mindset uh just make sure that, like after a discount do the first conversation they're already kind of like think about like everything that that is going to go wrong, because it's only a matter of time right, you were saying it's like it's a matter of price and time. Essentially, so that's that's like how we how we approach this interesting.

A

Kind of switching from the project perspective to the user perspective. You brought this up before what the projects need to do to educate end users on their security, hygiene.

D

Exactly so, that's one thing: this ecosystem needs to understand that, uh at the end of the day, security is really about users, even if your protocol is secure, but nobody knows how to properly use it and everyone just keeps getting compromised. Then you have failed, like you create, uh like you, use the public private key architecture, but you never tell anyone how to secure their private key everyone just copy paste that on Google everyone gets compromised. You have failed your mission, so user education is super.

D

Important metamask here is doing a great job at it, but we need a lot more of it and to get yeah and to get that like. We need to really understand like in this space. We need to understand one thing that it is uh not always about getting the uh like getting your mindset uh very crypto, focused or whatnot. You can be a crypto Anarchist, but you have to be practical in the sense stop telling like 60 or 60 year old grandmas to manage their own keys.

D

They are much better off using a custodial Service, uh not your keys, not your coins, everyone's favorite logo, but if you don't know how to manage your private Keys, it doesn't matter. If your key is yours or not, it will just get compromised.

D

So unless you know what you're doing it's often better to just go with a service product just a custodial product, uh rather than trying to learn what you don't know, uh this is why, even in traditional Finance like earlier in the days you saw like bonds and everything were paper based, but eventually everything got digitized and everyone just uses a custodian.

D

Nobody really ever buys shares on anything directly anymore. Wherever you go, you are buying them through a platform through a custodial platform, because traffic has understand that it's it's not ex like. If you want to scale, you can't ask everyone to just manage their own keys or secure their whole things. You have to make it much easier, but what we in web 3, what we need to do is give users the choice.

D

So if someone is tech savvy enough, if they want to run their own node, if they want to manage their own Keys, uh they should be able to do that. But we shouldn't be like shaming others for using infuro or other products or like using custodial products. We just have to change that mindset and educate people on both ends.

E

I, uh uh so one thing I would add to uh the ux which 100 it's it's ux is such a big uh part of uh security and helping people do the right thing, uh but I think an unappreciated part of it. Ux is. um uh Is that it's actually a um you know it's a team game. It's it's every one of us in this room every product, every project uh that works on web3, uh the uh the the ux is not just how good your ux is it's. How consistent is your ux with everyone else?

E

uh You know the way, users uh the way they learn. What is safe. What is unsafe is consistency uh when they, uh otherwise they have no prayer of going on uh to a website and knowing if this is going to scam them or not, because they're gonna have a whole new ux and every single time, and maybe this website just does a little bit differently and there's no way of telling uh just you know using your.

E

um You know your lizard brain on in fact right when you're on that website, uh what your, what you should or shouldn't be doing. um You know I. um uh My favorite uh example of this is uh logging in uh you know, on a d app, uh you connect your wallet and you know some websites change and, like start start, treating giving you data about like what's associated with your wallet, uh some don't yet or they'll, immediately challenge you some will.

E

Let you click around and then you then challenge you um and you don't you don't have this concept of, like am I authenticated or am I, not like how like where? Where is that? What is it? What is expected of me and all of a sudden, you click something you just pop up asking you to sign something, and you know who knows what that something is. Hopefully it's science ethereum, but you know- um and you know, compare that Tool uh passwords.

E

uh You know users have three decades of experience with passwords and knowing how to log into a website. It's consider every website's exactly the same. uh They they know exactly what the experience is. They know where they know what authenticated state is versus an unsending estate like that. Is you you we're con we're in Ground Zero with user education, and it's not just how good your product is?

E

It's how good we can be as an ecosystem at being consistent and showing the same experience over and over again teach users, the right moments um and I think that's something. We really need to internalize and start pushing more as standards.

C

Yeah I was just like I was exactly actually what I was gonna say, but I was also going to show Taylor's project because they might not.

C

uh If you want to look at good user documentation and education around every little thing, go to my cryptos facts and their like documentation, it's so good because it like will walk you through, like the mempool it'll, walk you through what a private key is and all these other websites that are built on ethereum, assume prior D5 knowledge assume prior Key, Management knowledge and, if you're, really building for people outside of the ecosystem and including other crypto ecosystems that have dissimilar key structures, and things like that.

C

You should really spend just a little bit of time, invest in a tech, Rider and or even just point to other people's pages that give that education for the users and like, if that sign like sign this message. Box pops up I mean there's so many people who don't realize that doesn't cost money. That is not a transaction that is signing a message from your public from your keys. Keep here so like like we're just so far behind.

C

In that, because the protocol layer can't be the ones to teach that users aren't running CLI, Geth and learning that from scratch anymore. That's where I learned it, but that was way long ago, where it's very different now uh so yeah just look at really good documentation and try to emulate that in your project,.

F

Yeah I want to emphasize like the amount of docs that I've written, but uh it's worth it because um you will be surprised at like how much people want to learn and so, like there's sort of like this track, where like we, can make the ux better and we could give users more choices that better serve them like. We should definitely do all of those things.

F

But if something is like the current way of doing it- and it's not clear- and it's not obvious- take that opportunity to tell the user just like literally tell them- don't treat them like they're idiots right treat them like they're, smart, capable adults and tell them what's happening because they will carry that knowledge and then they will actually pass that knowledge along to other people. And it's uh it's quite remarkable.

F

You would think like the docs I've written like what what what comes with that nobody reads: docs like it's weird, when my words like show up in other people's, like Tweets in other people's documents, right people, reference myself like I'm, like wow. There's people really reading this. um It's it's amazing. I also want to say, like back to what Corey was talking about. We do need to work together. Better uh I'm told nadav is in the audience, love you nadab nadab called us out recently like now.

A

We're getting spicy finally,.

F

Open Sea called metamaskow because there is so much use or loss happening right now, um and a lot of the user loss is actually literally with users of openc who are using metamask, and we should be working together better to uh make sure that every single thing that we're doing and showing the users is uh way clearer and we should be in lockstep so that when openc releases, a new sighting mechanism or a new way of handling something metamask can actually like.

F

You know not show gibberish literally not show gibberish, please anything but gibberish, um and so we have some like exciting things in the works that, like, hopefully, we'll uh we'll get into production soon, that that will ultimately serve not just metamask users, not just openc users. Right it'll serve everyone in the ecosystem, but specifically right now. It will prevent the loss that we're seeing and we are seeing an immense amount of loss.

A

What's the way to solve it, is it just point to point interactions between like the metamask and Open Season? Do we need, like industry working groups.

C

A

I know: that's a can of worms. A lot of people don't like that.

C

We have industry working groups, they fall off time and time again, because the new shiny thing comes.

A

Out but now tokenizer.

C

Man, just like that's just so meditative economically tokenized, the incentives to economically.

A

There's the incentives repositories.

C

Of this layered.

A

C

But yeah no I I, you might be able to answer a better tail, but all the groups I've seen some of them have worked really great. Some of them floundered because they're the teams, especially the ones starting out, but they have a big impact- are small. They don't have the time to commit to going to a bi-weekly, Zoom meeting to look into to that and then there's a lot of competing.

C

um What am I trying to say there was like for a while. There was even competing ERC standards. You know a few years ago up until now, with even the basics of how we're doing messaging and signing of transactions across a theory. That's a very interesting history to look back on, but uh to fully answer it.

C

I guess yeah just pretty much direct communication with the teams like if something's happening, just telegram them and then don't just leave them for a few days like literally get back to them soon, especially if it's like money loss, security related that kind of that's where you need to have a dedicated person, who's doing at least security, infra or ciso or whoever who could be like. Oh, let me grab my lead, devil think this is in 10 minutes. It.

A

All comes back to the point of someone on the team, hopefully from the founders side to NASA's. Point earlier has to own security right yeah. One.

B

Thing: yeah, oh good, okay, one thing I'd like to add, is kind of like on this conversation. I, think that, like we should learn from a lot of the the great kind of like uh ux, efficient security that, like that has been put out. I think that, like apple, is probably like top.

B

You know like the the kind of like top company in in this space that, like has been not inspections like in the tech space in general, uh in terms of like uh ux security, and you know, if you take something as simple, you know: I I used to work there and um in the security team and kind of like saw a bit like the mindset from like internally, and you can see.

B

For example, like you know, um logging in, for example, like used to be like unlocking your phone used to be pin-based, and you know we all know that, like people just like hate to remember things, so what did people do? You know when they were allowed to bypass it? They bypassed it right. It was kind of like I, think for only 14 of the people used to have uh depend on, and everyone else had just like the fully or even before the passwords right password.

B

You can exactly one two, three ABC or whatever exactly yeah, and then the moment they switched to face ID this that actually Rose to over 90 percent.

B

So you know it it like, if you're thinking about kind of like the mechanism to just like make security, easy and just like flow with the the ux people will just like come along with you in the process and so I think that, like again we're discussing this with because yesterday I think that, like the the you know, the the user interface can do a lot more for us as of today. You know when we're thinking about um when we're thinking about making.

B

Obviously, like educated decision, you know, metamask and and other wallets are starting to to put like great features in order to kind of like show you, you know what are you essentially achieving by submitting this transaction to the network? What is what is going to happen essentially uh ahead of time, but something that is super interesting is like. There is more to be done. You know, like you, have uh approvals for your nfts. These are permissions right.

B

Similarly, for, like your uh fungible token allowances that could be managed actually directly within the wallet, the wallet should be able to do that. Just like your phone actually manages your permissions. You know like giving access to you know, cameras and like Bluetooth and like location services and so on, like all of these things are things that we can learn from.

B

Just you know great user experiences on, like you know, mobile devices and so on uh and and I'm really hopeful that we're going to see more of these things as part of wallets and similarly for, like you, know, fishing when we see like a lot of people, that kind of like get scammed, sign the wrong like set approval for all and get kind of, like all of their wallet, trained, even kind of like wallet management should be possible like within the within the wallet itself.

B

Like you know, think about uh you know your wallet essentially just like spinning off, like you know, generating a new address on the Fly. uh You know providing essentially really like allowing you to just like send funds like minimal amount of funds to just like pay for the gas fees for the operation and not have access to any other asset.

B

That is part of kind of, like your main address right, like just trying to think about, like all of these things, that don't necessarily need to show any complexity to the user, which is like help them in the journey, essentially of like there's.

C

A startup doing that right. Do you all know about this? There's like a thing where, when you launch a metamask transaction, there's a second window and it takes the byte code and puts it into human English, so.

A

There's uh hexagate that's working on that.

C

There's a second uh redefine we Define yep.

A

C

E

So something I would add um so 100 agree, uh I, think all that stuff's, amazing and I think um you know we're going back to you know this whole thing sort of web two right, uh I, think one of the things we can also take away from the web 2 world is like the depth of security controls that have been implemented.

E

um You know, no one is um you know going back to right now we're kind of running and bring zero, um and uh when you, when you sign you sign with your wallet um the equivalent of an operating system. Like um you know, adding those kind of controls would be awesome ways to put some sandboxing some capability based access controls and extra reduction mechanisms.

E

um But you know there's more to um there's more layers. You can add right, um I. Think one of the biggest areas in my mind right now is um uh and I've learned this from talking to victims of phishing attacks. Is they don't? They often don't even know where they signed the malicious transaction at um they they're they're at a loss um and um there's no introspectability uh we're missing.

E

um You know we're missing logs we're missing uh and I think that we're also missing like a really like, like key linkage for like on-chain and off chain uh transactions. So not.

A

Only do we need monitoring login on the project side yeah. We need to log in on the on the user side.

E

As the user side yeah, we, if we're going to trust you we're going to say your key, your your uh your Authority. You also owe we also owe them every tool they need to be able to do that correctly, and that logging and monitoring is a big part of that.

F

Yes, um specifically, one thing that we've like experienced um at metamask that openc has called us out on and continues to call us out on is um for the longest time there were transactions and the transactions have like Financial value, and it's pretty easy to, like you know, like teach users like this is a thing and like there might be some bad things that happen depending on.

F

If you click this button and then um now a lot more stuff is just off chain right or it's not on chain, until after a certain period of time, if the order is matched or whatever right, um and because of this, like super early assumption that, like everything, is, there's a key and then you have transactions and the transactions go on a chain. Because of that super early assumption that we made. uh We did not see this coming we're now, a like just a signature right, just like an off chain.

F

Signature can actually lose you like all of your nfts or all of your tokens or allow permission somewhere else right and so now we're like playing catch up while open sees like sprinting ahead like making sure that it's usable making things scalable we're playing catch-up to try to like map back and like in reality, we should have never ever treated a transaction like an on-chain transaction signature differently than a message signature, and because of that very early design. Choice like at the core architecture level.

F

We are now like remapping everything and it's like uh it's just a it's a stupid amount of work to be honest, and it takes way longer than it should, and it's especially painful when users are are losing money because of this every day, right like when we are hearing about celebrities losing their nfts. Because of this thing that, like we know, should be better. That is. That's horrible.

F

um I also want to comment that, like um my talk tomorrow, I'm gonna be diving into like some of the early early choices of like wallets, but also of the protocol itself uh and diving into like a lot of the stuff. We're talking about here, because I think that the the biggest sin of it all is that it all traces back to this like private key and this private key, gives full authority over everything.

F

Right, like that's the world that we've like like started with and like fundamentally I'm so hopeless like it is so broken like. Why is the single thing that you can't change? You can't do anything with the thing that grants like so much permission and I.

F

Think that, like um we are now seeing that, like, at the token level, we're trying to like Implement permissions and restrictions, but ideally like that, would actually live like out the very core layer right where, like the first thing, the thing that holds your stuff is actually like a default like you don't do anything, and the first thing you have to do is Grant the permission, and then you can take the action.

F

um So yeah come to my talk tomorrow, uh we'll we'll dive into this. But you know I think that this is why I really want to emphasize like have conversations with like the people that are around you learn from your users. Listen to them like the most valuable thing that I've ever done is have literal one-on-one conversations with people, who've lost all their money and like just ask questions and let them talk, because you will learn so much about where your product is failing um and you will be able to serve them so much better.

F

If you carry those stories with you, it's not super fun, but it is like the most valuable thing that I think I've ever done uh to this day.

D

You agree with that uh on on the bed: dot private key search, I completely agree that refers back to the point. I was saying like uh theoretical security is one thing: uh people have built products that are secure in the best case, but if they are not practically secure, if nobody knows how to use their product, if nobody knows how to manage their keys and so on by just failing at your goal and then the other thing on monitoring and alerting.

D

That's another super important bit so not just about users but about your product as well. If your Bridge loses 600 million, you shouldn't have to wait six days before you realize that so definitely always have.

A

And that was because of no monitoring.

D

Yes, yeah someone literally message them on desk or hey something's, not right so yeah anyway, you should. You should have proper, alerting and monitoring, but yeah also don't go too overboard. Don't start logging, private keys of users. You have to find the right balance there, uh but all of that aside, I want to like take a step back and like say we just like.

D

Never talk really talked about the title, which is web 2 versus web 3 same or different, because I think we all just assume they are the same web3 just sprinkles a few things on top of webto security. But if it wasn't clear like at least in my opinion, they are basically the same things. Spencer here has been trying to guide us towards a bunch of I, think 20 questions or something he prepared if we have managed to get through, maybe three no just wait and turn on random spree yeah, but yes, great job.

A

Yeah speaking of a curveball question, so uh we were sharing more stories outside um yeah, there's, a reason why we're not going with the prep questions uh so Taylor you're. You were sharing one about response because you know, obviously, after you do monitoring to logging. Oh crap, something happened right.

A

Maybe you were.

F

A

Do you do next right? So what are.

F

A

Learned for that so.

F

um This piece is decentralized and a lot of the teams are there for fully remote. uh We use telegram, we use slack, we use Discord, we use signal. We use like there's like 8 000 chat, apps I, don't know about you but like for me. uh My notifications are basically like off because otherwise I'll just be like constantly inundated with information and I, wouldn't be able to think.

F

um However, if something happens like how do you let that person know that something happened?

F

So, for example, this one time Hudson was giving a talk and you got Sim swap while on stage uh and The Sims offers actually talked to his wife uh and I think told her that he was kidnapped because they're, because they're Sim, sloppers, um and because me and husband, have like a relationship that goes back and because Twitter, like immediately alerted that Hudson was, uh was Sim swapped and was like damning asking for money because of that like I had the actual sort of like in real life Network and the actual phone numbers in place to be able to like contact right same thing has happened with my own team, like I, don't know having each of those phone numbers and making like old school phone calls that break through, like the Do Not Disturb mode is like critical.

F

Almost every single, like incident response includes like tracking down the person who has the person's phone number. Who has the critical access to stop the thing.

C

Yeah do two things number one I think both of you are on a list so like. If I call you I go through Do, Not, Disturb mode right, exactly.

F

A

Here so like I have that set up yeah we have that set up, yeah yeah so like.

C

If one of us, because another time, someone posted something on my Twitter, because I had like an API key out there anyways. The second thing is when I got Sim swapped and as soon as it got on there, I I walked. I went back home when I was everything was fixed. There were 10, very critical group chats that just said, Hudson Sim swap don't listen, anything he said: Hudson swim stock, don't listen to anything. He said and my startup at the time open, Innovations immediately shut out all my access cleared.

C

It cleared it completely the ethereum foundation in less than 10 minutes after the Sim swap happened. Clear had a round table discussion, a live, War, Room Zoom call and got all my access taken away in.

A

Minutes, did you guys have a plan for that or just kind.

C

Of uh we did have so by that time, so I talked earlier about how not great the EF stuff was by that time, we'd improved so much. We had a whole incident response plan ready for this for.

B

A

C

We had a dedicated CSO role, type deal that we have kept over the years and years and years, and we have a great one right now who replaced me uh after I left or not before I before I left, the EF and I was doing other things, but either way yeah.

C

That was one of the scariest things because, like the cops were called, my spouse is crying one more thing, and this is something just not related to this exactly, but um something I'm really proud of little it for my spouse, wouldn't I finally got a hold of her. The first person after resetting my passwords and I, went to the like. You know the phone store, I called them and I said hey. This is Hudson and they were crying and they were like I'm here with a police officer.

C

I know this sounds like Hudson I'm, going to give you two questions that only you would know and I was like. Thank you I've taught. You upset, oh my God, so uh so the day to do it was me, and everything worked out from there yeah.

E

Yeah I I just wanted uh like really high. At one point uh you had there, which uh well is like having the plan uh but I, think a key is practicing the plan.

A

Rehearsing doing.

E

Tabletop simulations making sure that uh uh what you know documenting where the failures were white, what was hard and smoothing it over before you actually need to do it in uh in real life. uh Actually, one of my favorite stories of this was uh we had a simulation where we had to disconnect our Corp from our prod Network previous company, um and we had one room that we could get into, but that room kept getting smaller and smaller, because you know we never need that room uh and then, when we in our simulation we did.

E

uh We realized there's only one ethernet Jack and we had no way of getting all of us into that room, to be able to connect to prod and to keep it up, keep everything up and then like so we're like yeah. So it's just one of those like there's. It's always the thing you don't think about that. Fails you um in an incident, and you want to know that before you actually before it counts.

A

Something uh you know: sweating training versus bleeding and War yeah.

B

Another another thing: I think that, like is important to to take into account, is that um a lot about um incidents response and like Incident Management is about time right like time works against you, the person is you know someone or like some entity, some group already kind of like perform like found the issue and kind of like try to execute it, so your role is also to kind of like expand as much as possible, the time window that you have where time can kind of work for you right, it's kind of like and, and you know we kind of like always discuss these things.

B

Well, like you know, uh governance protocols and, like you know, uh Bridge security, it's it's about, like you, know, time locks and just like freezing funds for like a specific amount of time, so that, like you, actually give yourself some kind of like breathing rooms and kind of like your teams to kind of like go over the logs go over like everything that happened understand exactly like is like. Is this something legit or not? If this is not legit, you know.

B

Obviously, if you have other mechanism in place, great otherwise kind of like get ready for you know, kind of like freezing assets across you know, chains and so on. So I think that just like making time work for you is also one of the most critical things that you that you can do and uh a lot of people have been kind of like theorizing about this.

B

uh It's um how much time essentially like the uh bad guys need to spend in your trenches um in order to kind of, like you know, get the assets and then get away with it.

B

So that's that's something that is very, very important and obviously you know uh will never uh emphasize enough that the need to kind of like you know, baking time whenever you have like a new governance proposal and just like, uh whenever you have like specific thresholds or like limits that are hit, you know making sure that um these things like take time and that you have like some some freezing periods where you can actually like cancel operations.

E

And just want to hit on this point again. uh This is a this all the stuff layers together right, you hold.

B

E

Your systems, you may uh you you make sure you patch systems and you find them uh and you've audit them and then but then, whenever things do happen, you time box along the attacker has in your systems that we have. You can quickly respond like these all the stuff layers and that's how you get a great security, defense and depth on.

D

That one little bit so with incident response, I agree time is very critical. So what everyone needs to do is have an a year plan.

D

Have a disaster recovery plans literally have playbooks of every scenario you can think of of anything that can go wrong, have playbooks of what to do so that, even if, like your out sake, someday and some other security or someone else in the article doesn't really know as much about the product can just see your playbook follow the exact steps and be all good and like you need to practice these playbooks as well. It's not just about having the content out there.

D

You need to practice all of these playbooks and everyone should be aware, like what needs to happen. When needs to happen, how to follow the Playbook and only then you will be able to properly respond in a timely fashion uh and another bit I would say like coming back to the swim swap thing so there I wanted to add a little thing. So switching swaps, in fact not just Sims wife, but any product where the support department of the company can uh like do whatever.

D

With your access or wherever, like you, don't have control on your product, which is most of the webto products, don't depend on either get something like highly secure or don't depend on the cheapest one. So for Sims, almost all of the public providers like they can just same swap any of their support engineer or uh whatever can same, swap you so never use mobile numbers for 2fa or recovery or anything permission. What.

A

Do you think of a funny? That's the one uh supposedly non-sen swappable cell phone provider, I'm.

D

Not Lumen into them too much but they're.

B

Like a regular I mean they're like legal requirements for them to like allow specific operations and like porting, you know like you're, lying to something else, so like so long as like this operation is like required by law from from these providers. There is an entry point right, and it's like this is just like a matter of like how much Social Engineering you can do like people are saying that, like Google 5 was actually not sim swappable, because there is no one to talk to on the customer service, but you know it it could happen.

B

You know like no one is kind of safe, despite because even the more secure cell.

A

Phone providers.

B

Yeah, exactly like you know, you want to aim for. Obviously you know the regular kind of like non-sms to fa base but like if you're, if it is mandatory, go for, like you know, Google Voice number, for example like there's no sim to swap essentially so you know go for for services that do not have like the same associated associated to it. Yeah.

F

And back to, like the whole stack right and like layering, this on top right, get the most secure, Sim and then have nothing like.

D

To have nothing.

F

On that Sim right or like only the the two services that like actually have to be on that Sim and like have that video number that isn't really your primary number right. Like all of these things, layer and like that's.

A

And then you're not signing up for like marketing databases, yeah.

F

For this conference- and this is like this is why we talk about like the culture right, we talk about the organization, we talk about the people and like how important that is from the ground up. It's like every little decision that you make has to be the most secure one right and so that you have all the different layers on top of each other, so that it's less likely that things happen that are really bad. But even if those things happen, you can mitigate the loss, you can respond to loss quickly.

F

uh We are being we're.

A

F

Ushered off the stage, thank you for.

A

Coming love, you.

D

All thanks. Everyone.