Cloud Native Computing Foundation Cloud Native Rust Day EU 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Keyhouse: Production-ready Key Management Service in Rust - Maxwell Bruce & Sergey Shekyan

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Keyhouse: Production-ready Key Management Service in Rust - Maxwell Bruce & Sergey Shekyan, ByteDance

In this talk, Max and Sergey from ByteDance security team will share their experience of building a production-ready key management system using Rust. It uses Spire to establish mutual trust with the customers and other components, which adds up to the already unique combination of being the only OSS production-ready KMS written in Rust. The system is nearly self-contained, with the exception of the storage system, which is ETCD at the moment. Compared to go its predecessor which is written in Go, keyhouse uses much less memory and CPU cores to secure a large amount of user data. Keyhouse will be open sourced on Github very soon.

Acknowledgment:
We thank Yu Ding as a 3rd author. He will not be a presenter but he contributed significantly to keyhouse.

Yu Ding is a security researcher and engineer. His research interest is security issues around Intel SGX and building security-critical systems. He is a zealot of security oriented software development and dedicated to build a memory safe world.

A

Hi everyone um we will be talking talking about key house, a key management system, that's ready for production and um we work for a company called by dance. It was founded 90, roughly nine years ago. We have a lot of products, most famous one is tick tock and a couple of words about ourselves. So building leads trusted. Computing can secure coding initiatives at by dance and his big supporter of rust. He is maintaining apache t clave, which is an open source, uh universal, secure computing platform and before that he worked at baidu on similar problems.

A

Max who is with us today is a very quick learner and he is the primary developer on key house. So without him this project wouldn't exist. He brings his rust skills and like in general, computer science, skills and security skills to the table and previously max worked on some anti-automation problems and me sergey. I work on primarily work on like web security problems and which includes anti-automation and some browser security, apis and so on. So none of us had a prior key management system uh development experience. So that's kind of more interesting.

A

So what is key house?

A

It's an open source project well soon to be open source project completely written in rust and um at by then security and privacy are top concerns, and uh just a few words how we started doing this is that there was an existing key management system that we were integrating started integrating with zti with the with our new authentication uh mechanism in our in our company cloud and while looking at the code, several refactoring initiatives came to our attention and at some point, critical mass of changes was so big that was so massive that um we decided to rewrite everything from scratch.

A

And while there was no massive trust projects in the company, we were allowed to management allowed us to experiment with rust for a bit and maybe rewrite some some small components in rust and see if uh performance benefits or any other benefits, overweight, uh steep learning curve and like we can, we can benefit from it and yeah it was.

A

It went so good that um within a couple of weeks, we wrote a minimum minimum viable product and we were amazed with the performance and scalability and like memory safety guarantees that we were allowed basically or we were. We got a support to implement everything from scratch. 100 in rust and.

A

B

Yeah, so our our rust deployment, uh you know this project has has since matured at bike dance and we're deployed in production um all across the world. Now um so you know we can say this project has internally been a success, and part of our open source initiative is turning this into a success for everybody else um yeah. So thanks sergey for telling us uh stuff about why rust. um So I'm max I'll be going into some of the design considerations in key house um so starting off here with our core components.

B

um Everything in keyhouse is designed around kind of like a monolith uh kind of end product. In that you know you ship, like just one service out that provides key house, and then you you'd balance that out amongst many pods or or whatever it would be horizontally scaled, but just one category of service.

B

uh The reason being is just minimizing chances of failure, minimizing um any kind of like potential dependency problems and uh just overall trying to maintain as much simplicity as possible because, as it turns out, key management systems tend to be dependent on by a lot of critical services. We really really need that top reliability and that top um top like just reliability and redundancy, so we just don't have any room for failure.

B

So a lot of our design considerations go with that in mind, um and with that in mind, as a monolith, we have uh kind of like two halves to our program. uh So we have our data plane, which is what our sdks talk to, um and then we also have our control plane, which is what, like you uh or an operator administrator, would talk to to.

B

You know, create new keys and and do other administration operations um uh and in general you could say, are the each component of keyhouse design to be very easily hot swappable at the build phase, and um we did this kind of with a uh a service as a library approach where you would import key house almost as a library and then fill in whatever implementations you need to integrate with your own corporate infrastructure, the idea being okay, you know as bike dance.

B

If this was somebody else's open source project and we have all the specific infrastructure requirements we need to fulfill. How can we get this open source project to mesh with our own internal systems? um And so we tried to make it an approach so that we could both answer that question for us, but also answer that question for everybody else, um and so that's why we landed this kind of hot swappable uh system.

B

So, for example, like control plane, authorization can be defined um by any system like some kind of sso internal to a company, uh just standard jbt tokens directly through spire spiffy, uh zero test infrastructure.

B

um You know anything works uh and, on a similar note, uh like exact crypto, algorithm support um and even like the back end itself like we were probably, we will primarily be talking about ncd here, but uh you could swap out back ends of other needs of rows, um and this extends to a lot of other components of keyhouse as well uh speaking of fcd uh etcds are is, if you're not familiar with it, a back-end data store and we'll be talking more about that shortly.

B

um But it's ultimately like the the root source of all uh like back-end storage for t-house cd itself doesn't have support for spiffy spire zti, which is what we use for identity and attestation and ultimately authorization.

B

We wrote another project called spire proxy that allows us to uh con ls is to provide the kind of like a a wrapper over at cd in order to um uh provide like that that spiffy spire attestation authorization we'll be talking more about spiffy later on in the presentation.

B

um Also, generally speaking, kms uh key house we'll also be talking to uh nhsm hardware security modules, a hardware to trust we'll also be talking about that more later on. um So let's talk more about etcd, so etcd, as I said before, it's our primary, authoritative uh backend data source um every keyhouse instance talks that cd all the time. Of course, we do have failure like prevention. We have like kind of failovers in place and stuff, uh that's all, ultimately up to how you want to deploy things.

B

um We use spiffy ids as a form of identity uh between ncd and keyhouse, uh so we're not relying on like username password schemes or anything like that, um and you can set that up with other backend stores as well, but, uh like I said before, we'll be focusing on ncd um and uh in general uh ncd, if you're not already familiar with it, it is a it's a very simple key value store with its primary selling points being high.

B

Consistency, guarantees very persistent, very reliable uh load distribution, but it is distributed like generally speaking, you plug like five to seven nodes, um so uh it's relatively scalable uh for what we need, and it also um you know it's also relatively easy to create replicas of both with fcd and as we'll be talking about right now, uh with re-replicas and with caching. So uh I guess in the last slide, I talked about a little bit here.

B

um We do also create read replicas of fcd in order to deploy with the same cluster data set across multiple regions around the world, um and uh you know part of that is we have this really interesting kind of cache infrastructure in key house where each keyhouse instance right so each like pod running keyhouse effectively becomes a replica of that cd in and of itself a read replica that is- um and you know in uh you know, we kind of like more theoretical terms.

B

This ends up being a kind of like a right through evictionless cache and that, due to the the generally low size of the entire data set, like I think, our entire production network, uh you know it doesn't grow above a couple megabytes uh for every you know encryption key. There is because crypt keys just aren't that big. So what we end up doing is we actually just cache everything all the time in every keyhouse instance, which provides amazing like failure, uh you know, like you know, just handling failures, really well. uh Etsy goes down.

B

Key house just keeps trucking like it's no big deal. um Of course, we deny any rights during that time, but uh and that's why we're right through is to deny rights when uh that cd can't confirm anything uh and in general, key house actually relies a lot on the kind of consistency guarantees that std provides in order to synchronize its operations, um and so that's a it's a very useful. uh You know future provided by fcd um and uh yeah.

B

This has given us great performance gains across the board and ultimately, if you think about it, it makes it such that any given request that key house has to handle does not have any direct network dependencies, which is both at least other than the request itself, of course, um and so this results in uh just like a really reliable, um like request serving uh latency and in general, just high reliability of serving requests.

B

um We talked a little bit about it before, but you know what is spiffy if you're not already familiar with it. It's a attestation and identity framework. uh It's an open standard, open source, implementations and standard. um We mostly center around spire at uh key house and bike dance.

B

um Our own internal deployment is using spire and, if you're not already familiar with it, ultimately, what the usage of spiffy inspire provides is some kind of spiffy, verifiable identity document or spin generally in the form of an x 509 certificate uh signed by spire through some form or uh an es256 jwt uh token containing a spiffy id, whereas spiffy id is just a uri of a specific form in keyhouse, we place a a generalized form of this and that you have like spiffy uh and then you have like a trust domain, uh and then you have a series of key value pairs uh which denotes some property, the intention being that you can use wild cards to uh be able to more generically refer to parts of a spiffy id um so yeah.

B

How does key house specifically use spiffy? uh So in general, everything in keyhouse is protected with mutual tls, and uh you know we're using spiffy to communicate identity through this use of mtls, so, like sdks, will confirm the identity of the keyhouse server.

B

It's talking to the server will confirm the identities of sdks, if applicable- um and you know, on top of that, the key house will also use these identities to authorize resources in uh you know we'll talk more about the specific groups as being authorized later, but the customer key is uh that uh a given request that the data plane is accessing uh gets authorized to these spiffy idea identities as well, so those uris we looked at before um and all of our integrations of this uh due to the general low uh integration of spire and spiffy in rus.

B

So far in open sourcing, we actually wrote our own implementation called spire workload rs that will also be getting open source at the same time as key house when that gets open sourced in the coming months.

B

So uh let's talk more about our key hierarchy, so in general, like k, key house, it really revolves around a lot about envelope encryption and we also put a lot of uh you know: effort into hardware route of trust, so our master key, generally speaking, is stored at our at our. In our you know our hsm, and it is, you, know, really hardened down. It's in you know a lockdown cluster um of like hsn modules, with any implementation defined like backup system and access control outside the keyhouse's hands, um and uh you know to avoid.

B

Generally speaking, one would expect that to have a very like uh low throughput for encryption, like you're, not going to send a bunch of stuff into your hsm, uh so we use an intermediate key that is directly encrypted by the hsm, um and this gets stored in fcd uh at rest encrypted, um and this is used to encrypt all other data in ncd.

B

Generally speaking, so like customer keys, uh as our number one example we'll talk more about those later, but those are really like our meat and buns uh and that's what a lot of like access, control and users kind of interact with data keys are uh what we use an sdk level to uh it's kind of like the end use key. uh You can think about it. That way, so we made a master key.

B

um These are generally manually rotated uh by any, like you know, presumably have some kind of like uh operation in place uh to to create a new master key internally, um and then uh you would just have keyhouse uh updated to uh start. You know uh migrating all of the existing encrypted customer keys and intermediate keys around uh this new master key.

B

So our intermediate keypad.

A

uh The only time you would need to access, hsm or any other like hardware route of trust, implementing storage uh will be at the startup when you need to decrypt uh intermediate key, that's in at rest or when you rotate intermediate key.

B

Yeah, so uh this ends up being that in our current defaults, where we rotate intermediate keys every 24 hours and assuming you don't restart keyhouse uh very often that results in one encrypt in uh uh like and decrypt operations, one per pod and then one encrypt operation to create a new intermediate key uh and then each pod would have to decrypt the intermediate key on boot.

B

uh That's kind of like the general uh load on the hsm, that's expected um yeah, so our intermediate keys uh we've heard a little bit about these, but uh these are, uh as I said, uh are rotated once per day um and are encrypted at rest by the the master key in the hsm.

B

um There can be more than one floating around uh due to the way that keyhouse uh migrates customer keys when we rotate intermediate keys, intermediate keys actually get stored alongside or they get copied alongside customer keys encrypted at rest. uh This way that in the when we rotate an intermediate key, um you know we can do it reliably uh with each pod, taking an arbitrary chunk of those customer keys and uh re-encrypting them, with new intermediate keys um in a highly reliable way, um taking advantage of fcd's consistency to coordinate.

B

uh So our customer keys- these are you know, are real meat and bones. um These are what access control lists live on. These also have an assigned purpose like okay. These are for encrypting secrets, these are for encrypting data um and uh they never leave the keyhouse uh it's kind of like the the lowest layer key uh that is in key house server itself. um Generally speaking, what happens with these keyholes? These customer keys is an administrator goes through, creates one sets an access control list, and then we get to the data key.

B

So what happens is if you think about it. From the sdk perspective, sdk wants to encrypt some data. uh Sdk requests a data key from keyhouse keyhouse uses an uh a one, a one time, password scheme kind of like two factor authentication to uh generate a data key for a given day to uh make caching uh a much more reasonable thing to do, because we want to keep the number of data keys floating around for a given time period low.

B

um Generally speaking, we do one data key per day per customer key, um and so what uh keyhouse will do is it'll return. The encrypted uh data key an envelope encrypted by the customer key and the unencrypted data key to the sdk, which will then encrypt the original payload and store alongside the encrypted data key alongside it, along with metadata that tags it along to its owning customer key um and uh yeah. It's you know, ends up being this really clean implementation, where uh the data never gets sent to keyhole.

B

So it minimizes your bandwidth usage, um but we also uh avoid sending the customer key to the client and also minimizing the number of requests upstream to keyhouse. This also provides us a lot of redundancy, so in the event that the keyhole server is down, uh if the sdk can't fetch another data key, I mean it's not great, but you know you can have some leeway where like say, we try to fetch a new data key every six hours, but if it fails well, we can wait another six hours.

B

It only rotates every 24 hours anyways um and you know. Of course you want to put strict limits on that to avoid any kind of problems, but it allows for a lot of extra fault, tolerance um and just improving that that uh you know reliability guarantee.

B

um So we talked a little bit about this before but secrets. So not everything can really fit into the neat box. That is like a customer key data, key setup. um So say you have some persistent key. You need to use for some other purpose. Like a you know, an hmac key, an immutable private key for some pki stuff, um some credentials for some sensitive service.

B

um You know you can store these in a keyhouse secret um and they get stored in that cd um uh buy a data key that doesn't leave uh key house in a given customer key. That's the that is mutually exclusively assigned to only be able to encrypt secrets, and not other data will uh uh be able to have any number of these uh associated with it.

B

um So key rings so key rings. This is not a type of key is rather a category or a grouping of customer keys, mostly used for operator and admin or slash administrator uh ease of use and convenience.

B

um You can use these to to kind of group together customer keys and like say, share to another operator, um and it's mostly all, therefore, is it's just a form of organization, because I mean, if you have one customer key per use case as you should it's falling good security principles, then you can end up with quite a few of them, and then you know you know it's nice to grip them together into some like coherent groupings, say like x, team or x service or whatever.

B

So uh you know, we talked a little about operators and how we're using access controllers to the customer level. This is kind of like a full diagram of what authorization looks like uh in keyhouse. So you have some administrator. You know they have access through some implementation defined uh mechanism, you know say like sso or whatever uh they have ownership of some set of keyrings or access to some sort of keyrings, um and then each keyring had uniquely owned some number of customer keys under it.

B

Like I said before, it's a grouping, then each customer key has its own unique access, control lists which define how it works at the data plane level, and so the operators can then go into customer keys. They have indirect control of and set up a access control to whatever spiffy authenticated entities like service person, whatever your spiffy ids represent uh and can set it up to uh so they have whatever specific access is necessary, reading secrets, writing secrets, uh encrypting data, etc.

B

So uh next steps- uh currently he house, is not open source. However we're working to open source it now, we've been working with some security. Vendors to uh you know, make sure everything's tip-top shape uh and we're also working to open-source our spire workload project uh inspire proxy as well, which is mostly just an offshoot of spyware club um and uh yeah those we expect those to be open source in the next few months.

B

We also want to increase the number of integration for generic or open source implementations of uh various components of keyhouse, say like different, open source metric servers uh like log different ways to to gobble up logs um control, plane authorization schemes, perhaps team, different background stores. So just maximizing our uh kind of like ready to go on a lot of different open source, more open source uh stacks, um and also we want to add support for asymmetric keys right now.

B

Keyhouse is really really tailored for what we need internally and we've really designed it around what our internal uh customers at bikecans like need for their different projects, uh and one of those things just hasn't been. uh You know, asymmetric key is stored in uh keyhouse.

A

Store asymmetric key as a secret yeah.

B

You can sort as a secret, but like say automatic rotation or setting up pki infrastructures is not something currently innately supported by keyhouse. As some special feature like we have for customer keys, uh it fits in pretty well, but we just haven't implemented that um it's not something we want to add in, but yeah in general. A lot of our design decisions have been primarily motivated by what we need to get this rolling inside.

B

uh I think we mentioned a little bit before, but key house is deployed in production, now bike dance and is now you know serving all the people on tech talk.

A

Yeah, we can't really give out any concrete numbers, but uh several pods of key house are serving tens of thousands of services. Yeah.

B

uh Yeah, that's all we have here uh thanks everyone, uh and I think now we have some time for questions. Thank you so much.